ForexBot2/backtest/research/panel_v2_2026-09-17.md

Agent panel v2 — probabilities anchored to base rates — 2026-09-17

Why. The live review of 2026-09-04 → 09-17 (signal_log_review_2026-09-17.md, §5 and §7.1) found the panel's verdict was tilted the wrong way: the macro persona's stance was the verdict, its only input was headlines (a one-day-lagging echo of price, news_sentiment_value_2026-09-04.md), opposed setups reached their first target more often than agreed ones (54% vs 42%), three of four personas voted the same way on nearly every setup (technical 79% support, risk 79% oppose, contrarian 65% oppose), and the stated confidence was uncalibrated. The chair had already been shown anti-informative on 2026-09-03. Nothing versioned the prompts, so no change could be measured.

What shipped (code, all advisory-only; the panel still never runs inside analyze_from_dataframes and no gate reads it).

Piece	Where
Every persona returns p_win (0-100, the chance the trade closes profitably under its trailing exit) plus one reason_for and one reason_against; stance is derived symmetrically (≥ 60 support, ≤ 40 oppose)	agent_panel.py _TAKE_CONTRACT, derive_stance, _validate_take (v1 stance+confidence replies still accepted and mapped to a pseudo-probability)
Verdict = equal-weight mean of the personas' p_win, quorum 3 of 4, no LLM aggregator (verdict_source: "mean", the new default; macro and chair kept for comparison)	mean_verdict, run_panel, app_config.py
A code-computed context block in every payload: stop / target in ATR and R, cost-to-risk, the class base rate from the validated books, prior move 4 h / 24 h in ATR, distance from the 24 h extreme and its age, move_started_at, largest bar of the last 3 in ATR, signal-bar body, event timing against the class's expected hold, penalty tags with the book's verdict on them	_price_context, _base_rate, agent_panel_base_rates.json (built by scripts/build_panel_base_rates.py from the MD5-verified books)
Macro persona: no news doctrine; judges whether news is NEW relative to move_started_at; receives the ATR-normalised candles (as does the contrarian)	_MACRO_SYSTEM, _panel_jobs
Technical persona: indicator agreement declared uninformative; judges higher-timeframe conflict and key levels. Risk: only setup-specific risks versus the class. Contrarian: the engine's own warnings are not evidence	_TECH_SYSTEM, _RISK_SYSTEM, _CONTRARIAN_SYSTEM
PANEL_PROMPT_VERSION logged with every verdict and folded into the payload hash; per-persona probabilities and reasons, the exact per-persona payloads and the context logged (panel_prompt_version, panel_probability, panel_probs_json, panel_payloads_json, panel_context_json)	signal_log.py additive migration, update_panel, log_panel_update
Live frames handed to the add-ons through report._frames in analyze_pair (never in the core)	signal_aggregator.py
Plain-language "AI estimate" line on Telegram, CLI and dashboard; legacy sources render as before	telegram_bot._panel_estimate_lines, main.py, static/dashboard.js
Scorecard: Brier, log-loss, calibration bands, discrimination, constant-voter rate, ρ(p, R), counterfactual books, split by prompt version	analyze_signal_log.probability_scorecard (printed by --stored-candles)
Offline replay harness: rebuilds each persona's payload for every resolved logged setup (exact payloads when logged, else from the stored blobs, interned news and the kept Oanda cache, events evaluated as of the signal bar), runs the current prompts, scores them against the outcome and against the logged v1 takes and the base-rate predictor	scripts/panel_replay.py (--dry-run, --max-calls)
Backtester export strategy tag fixed to the generator's own stamp (label-only; every gold ATR setup was exported as GENERIC)	backtest/gpu_backtest.py; books re-pinned, see below

Tests: test_all.py Module 35 (15 tests) plus the extended Module 23 parity test; suite RESULTS: 521 passed, 0 failed, 521 total.

Reference books re-pinned (label-only)

The five books were regenerated with the committed code (stashed) and with the fix, on the same cache. Dropping the strategy column, every book is byte-identical to its previous pin; trades, R and PF are unchanged. New labels: M15 W2 174 XAU_ATR / 757 GENERIC; M5 W2 32 / 411 / 338 XAU_CP; M15 OOS 1 / 354; H1 W3 57 / 196; H4 W3 53 / 61.

Book	old MD5	new MD5
M5 W2	29120a1ee32979431d062f210d41a9d1	49f8e602fc6ae769713c94ef7b257810
M15 W2	d185f099482a07f29502cb5a371a9bbf	b4aa07e33e17e23774c486a6900d673b
M15 OOS	6f8f24b49597ced28f50a19e08c3472f	9fd624fb387c36052122bd1830cf9c32
H1 W3	3b45d12bade823178b267c13bc698458	7b848c917dcb5c371ecbb9f8f7292278
H4 W3	e75adae13c839b15736f69c0114e6aa5	186e677fbbd53cca735c94094c23ce05

Backend: NumPy CPU fallback, GPU cache fallbacks: 0 on all ten runs.

Base-rate table

agent_panel_base_rates.json: 45 classes (34 exact) from the relabelled M5 W2, M15 W2, H1 W3 and H4 W3 books; key timeframe|strategy|grade|direction, fallbacks tf|*|*|direction and tf|*|*|*; the exact class is used only with n ≥ 30. Examples: M15 all 931 trades win 60.5%, median hold 30 bars; M15 SELL 296 / 54.4%; M15 XAU_ATR A SELL 78 / 50.0%, median hold 6 bars; M5 XAU_CP B BUY 321 / 51.7%; H1 all 253 / 64.4%.

Pre-registered acceptance (from the approved plan)

On the resolved logged setups (37 today), a new prompt version ships only if: (1) no persona's modal stance is above 65%; (2) Brier ≤ the logged v1 takes and ≤ the base-rate-only predictor; (3) the skip-OPPOSE counterfactual does not remove a net-positive bucket. Outcome = advisory R > 0 at $0.65. Predictors compared on the same events: base-rate only (the class win rate for every trade), the logged v1 takes (stance × confidence → pseudo-probability; 27 events carry them), and the new prompts run live through DeepSeek on the rebuilt payloads (4 calls per event).

Replay round 1 (draft prompts)

predictor	n	Brier	log-loss	mean p	modal stance	constant %	R skip ≤ 40
base-rate only	37	0.263	0.719	54.8	neutral	81	−7.15 (nothing skipped)
v1 logged, mean	27	0.270	0.738	48.2	neutral	48	−4.00
v1 logged, macro	27	0.299	0.855	56.5	support	44	−2.82
v1 logged, technical	27	0.371	1.011	71.7	support	78	−6.47
v1 logged, risk	27	0.343	0.947	25.7	oppose	78	+0.03
v2 draft, mean	37	0.257	0.707	51.3	neutral	95	−7.15 (nothing skipped)

Like-for-like on the 27 events with v1 takes: Brier v2 0.257, v1 mean 0.270, v1 macro 0.299, base rate 0.259, coin flip 0.250. Rank correlation of p with R: v2 −0.05, v1 mean +0.14, v1 macro +0.36, base rate +0.16 (n = 27; the ±0.35 confidence band swallows all of them).

Reading: the draft is calibrated (it sits 3.5 points under the base rate with a spread of 1.7 points) and no longer directional, but it carries no rank information: winners averaged p = 50.9, losers 51.7. The old macro verdict had some ordering (+0.36) and terrible calibration (82% confidence on the three losing FOMC shorts); the new one has calibration and no ordering. Criterion (2) passes, (3) is vacuous (no OPPOSE issued), (1) fails in the opposite sense to the one it was written for: the personas cluster on neutral around the base rate instead of voting a constant direction. That is hedging, not bias.

A flaw in the draft prompts, found from the per-event reasons. The five lowest probabilities went to the Sep 10 and Sep 14 shorts that won (+1.31, +1.05, +0.70R), because the technical and contrarian prompts told the personas to lower p when "the move has already run far" or "a big candle just printed". That instruction encoded the live fortnight's pattern, which the two-year books contradict (stretched entries are the system's best trades, 2026-08-04; the big-candle gate was re-rejected in the review: those buckets are +32R and +60R). The final prompts remove that instruction, state the book's finding in the context legend and in both personas, and limit downward moves to higher-timeframe conflict, key levels, event timing inside the expected hold, news newer than the move, and a stop far wider than the class norm.

Replay round 2 (final prompts, the text that ships)

Same 37 events, same rebuilt payloads, the final prompt text (the technical and contrarian personas told that a long prior move or a big candle is not a reason by itself; downward moves limited to higher-timeframe conflict, key levels, event timing, news newer than the move, and a stop far wider than the class norm).

predictor	n	Brier	log-loss	mean p	modal stance	directional %	R skip ≤ 40	R ≥ 60 only
base-rate only	37	0.263	0.719	54.8	neutral	19	−7.15	−3.07
v1 logged, mean	27	0.270	0.738	48.2	neutral	52	−4.00	−1.66
v1 logged, macro	27	0.299	0.855	56.5	support	56	−2.82	−2.16
v2 final, mean	37	0.262	0.717	53.8	neutral	8	−7.15	−1.98
v2 final, macro	37	0.259	0.712	53.4	neutral	11
v2 final, technical	37	0.270	0.734	54.2	neutral	16
v2 final, risk	37	0.258	0.709	53.3	neutral	11
v2 final, contrarian	37	0.262	0.718	54.1	neutral	11

Like-for-like on the 27 events with v1 takes: Brier v2 final 0.260, v1 mean 0.270, v1 macro 0.299, base rate 0.259. Rank correlation of p with R: v2 final +0.13 (round 1 −0.05), v1 mean +0.14, v1 macro +0.36. Calibration of the v2 mean: 34 events in the 40–60 band averaged p = 53 and won 47%; the 3 events at 60–61 won 1. The final personas sit 1.0 point under the base rate (sd 1.8, no event moved 5 points or more); winners averaged p = 53.1, losers 54.3.

Acceptance: (2) passes — Brier below the logged v1 takes and equal to the base-rate predictor (0.262 vs 0.263; 0.260 vs 0.259 on the common 27); (3) passes vacuously (no OPPOSE issued); (1) fails as written (92% neutral) and passes as re-specified below (directional share 8%). The correction removed the chase penalty: the winning Sep 10 / Sep 14 shorts moved from 45–47 to 49–51, and the reasons against now name what the books do not already price (a release due in 20 minutes, price sitting on a support level) instead of "the move already ran far".

Cost: 2 × 148 DeepSeek calls (both rounds), inside the shared daily cap of 500.

Decision and what it means for the alert

The shipped verdict is a probability anchored to the validated base rate that moves only for facts the books do not already price. On this sample it is calibrated and uninformative: the line will usually read "about a 50% chance … (usual for this kind of setup: 55%)". That is the honest state of the evidence — the 65-feature sizing model was a clean null too — and it is strictly better than the previous line, which asserted 82% confidence in the wrong direction. The information the line adds is the base rate itself, the biggest concrete worry (event inside the hold, news newer than the move), and each persona's reason. It never gates; the user guide says so.

Re-specified criterion (1) for future prompt versions: the share of directional stances (support + oppose) on one side must not exceed 65% — neutral clustering around a calibrated base rate is not the pathology the criterion targets.

Persona set (owner decision, same day)

The owner's rule: a persona that does not give correct guidance is removed, to save confusion, tokens and log space. Evidence per persona:

persona	v1 (27 outcomes)	v2 final (37 outcomes)	verdict
macro	Brier 0.299, rank correlation with R +0.36, 88% / 25% win split on 2026-09-03	Brier 0.259, ρ −0.06	the only voice that ever showed ordering, and the only one holding facts the engine lacks (news) → kept
technical	support on 79%, Brier 0.371 (worst)	Brier 0.270 (worst), ρ −0.20	re-reads the indicators the signal was built from → removed
risk	oppose on 79%, Brier 0.343	Brier 0.258, ρ −0.02	its inputs (stop in ATR, cost, event timing) are now computed in the context block → removed
contrarian	oppose on 65%, Brier 0.322	Brier 0.262, ρ −0.05	oppose by construction, no information → removed

Shipped as agent_panel.personas (default ["macro"]; app_config validates the names, chair needs two or more, macro source needs the macro persona). One DeepSeek call per panel instead of four (three quarters of the panel's token spend), panel_payloads_json about 4 KB instead of about 14 KB, panel_takes_json one entry, no persona votes line in the alert. The mean verdict with one persona is that persona's probability (quorum = min(3, personas)). The full panel remains one config line away for research replays (scripts/panel_replay.py --personas macro,technical,risk,contrarian).

Next evaluation

• Every live v2 verdict logs its exact payloads, so scripts/panel_replay.py can re-score any future prompt on identical evidence. At 100 resolved v2 outcomes (about five weeks): the --stored-candles probability scorecard by prompt version. At 300: fit a probability-band → observed-win-rate table on the first half by time, verify on the second, ship it as a numpy lookup; only then revisit persona weights (owner rule).
• Do not iterate prompts on the current 37 events again: any further gain there is in-sample.
• Deployment: the live box needs the new code (both the bot and the dashboard daemon, which has run 2ac7301 since Sep 10); no config change is required (verdict_source defaults to mean); the additive migration runs on first open.

Commands: venv/bin/python scripts/build_panel_base_rates.py --book 5min=<m5_w2.csv> --book 15min=<m15_w2.csv> --book 1h=<h1_w3.csv> --book 4h=<h4_w3.csv> --out agent_panel_base_rates.json; venv/bin/python scripts/panel_replay.py --db logs/signals.db --dry-run; venv/bin/python scripts/panel_replay.py --db logs/signals.db --max-calls 160 --out replay.csv; venv/bin/python analyze_signal_log.py --db logs/signals.db --stored-candles.