Harwinder Sekhon
ad14ea4fed
Agent panel v2: personas return p_win anchored to the book base rates with a code-computed decision context; verdict = equal-weight mean (verdict_source "mean"); only the macro persona runs live (agent_panel.personas ["macro"]); prompt version and exact payloads logged; offline replay harness scripts/panel_replay.py; base-rate table agent_panel_base_rates.json (scripts/build_panel_base_rates.py). Backtester `strategy` column now carries the generator's own stamp (label-only; five reference books re-pinned, all 62 other columns identical). News corpus (after evaluating the owner's ML-NEWS scikit model against DeepSeek: golden set 39/78 vs 78/78, out-of-sample agreement 64% against a 90% self-agreement ceiling; verdict: fallback only, not a replacement): - NewsSentiment.xau_score + archive column xau_score keep DeepSeek's gold score for every headline (930 of the first 1,546 DeepSeek rows had none); an existing newsdata.db gains the column in place on first open - headlines naming no tracked currency are scored and archived (currencies_json []) but never returned to the platform - _apply_llm_sentiment sends every new headline in its cycle, 25 per call; a failed call ends the cycle's sending Dashboard publisher never writes a database: main() calls app_config.set_db_writes_for_process(False), honoured by signal_log_enabled() and news_archive_enabled() (owner decision: the bot is the single writer). Test suite: the hermetic app config now points both writers at a suite-private temp dir — the RSS fixture tests were writing fixture headlines into the real logs/newsdata.db — with a Module 32 test pinning it. 529 tests.
182 lines
14 KiB
Markdown
182 lines
14 KiB
Markdown
# Agent panel v2 — probabilities anchored to base rates — 2026-09-17
|
||
|
||
**Why.** The live review of 2026-09-04 → 09-17 (`signal_log_review_2026-09-17.md`, §5 and §7.1)
|
||
found the panel's verdict was tilted the wrong way: the macro persona's stance was the verdict,
|
||
its only input was headlines (a one-day-lagging echo of price, `news_sentiment_value_2026-09-04.md`),
|
||
opposed setups reached their first target more often than agreed ones (54% vs 42%), three of four
|
||
personas voted the same way on nearly every setup (technical 79% support, risk 79% oppose,
|
||
contrarian 65% oppose), and the stated confidence was uncalibrated. The chair had already been
|
||
shown anti-informative on 2026-09-03. Nothing versioned the prompts, so no change could be measured.
|
||
|
||
**What shipped (code, all advisory-only; the panel still never runs inside
|
||
`analyze_from_dataframes` and no gate reads it).**
|
||
|
||
| Piece | Where |
|
||
|---|---|
|
||
| Every persona returns `p_win` (0-100, the chance the trade closes profitably under its trailing exit) plus one `reason_for` and one `reason_against`; stance is derived symmetrically (≥ 60 support, ≤ 40 oppose) | `agent_panel.py` `_TAKE_CONTRACT`, `derive_stance`, `_validate_take` (v1 stance+confidence replies still accepted and mapped to a pseudo-probability) |
|
||
| Verdict = equal-weight mean of the personas' `p_win`, quorum 3 of 4, no LLM aggregator (`verdict_source: "mean"`, the new default; `macro` and `chair` kept for comparison) | `mean_verdict`, `run_panel`, `app_config.py` |
|
||
| A code-computed `context` block in every payload: stop / target in ATR and R, cost-to-risk, the class **base rate** from the validated books, prior move 4 h / 24 h in ATR, distance from the 24 h extreme and its age, `move_started_at`, largest bar of the last 3 in ATR, signal-bar body, event timing against the class's expected hold, penalty tags with the book's verdict on them | `_price_context`, `_base_rate`, `agent_panel_base_rates.json` (built by `scripts/build_panel_base_rates.py` from the MD5-verified books) |
|
||
| Macro persona: no news doctrine; judges whether news is NEW relative to `move_started_at`; receives the ATR-normalised candles (as does the contrarian) | `_MACRO_SYSTEM`, `_panel_jobs` |
|
||
| Technical persona: indicator agreement declared uninformative; judges higher-timeframe conflict and key levels. Risk: only setup-specific risks versus the class. Contrarian: the engine's own warnings are not evidence | `_TECH_SYSTEM`, `_RISK_SYSTEM`, `_CONTRARIAN_SYSTEM` |
|
||
| `PANEL_PROMPT_VERSION` logged with every verdict and folded into the payload hash; per-persona probabilities and reasons, the exact per-persona payloads and the context logged (`panel_prompt_version`, `panel_probability`, `panel_probs_json`, `panel_payloads_json`, `panel_context_json`) | `signal_log.py` additive migration, `update_panel`, `log_panel_update` |
|
||
| Live frames handed to the add-ons through `report._frames` in `analyze_pair` (never in the core) | `signal_aggregator.py` |
|
||
| Plain-language "AI estimate" line on Telegram, CLI and dashboard; legacy sources render as before | `telegram_bot._panel_estimate_lines`, `main.py`, `static/dashboard.js` |
|
||
| Scorecard: Brier, log-loss, calibration bands, discrimination, constant-voter rate, ρ(p, R), counterfactual books, split by prompt version | `analyze_signal_log.probability_scorecard` (printed by `--stored-candles`) |
|
||
| Offline replay harness: rebuilds each persona's payload for every resolved logged setup (exact payloads when logged, else from the stored blobs, interned news and the kept Oanda cache, events evaluated as of the signal bar), runs the current prompts, scores them against the outcome and against the logged v1 takes and the base-rate predictor | `scripts/panel_replay.py` (`--dry-run`, `--max-calls`) |
|
||
| Backtester export `strategy` tag fixed to the generator's own stamp (label-only; every gold ATR setup was exported as `GENERIC`) | `backtest/gpu_backtest.py`; books re-pinned, see below |
|
||
|
||
Tests: `test_all.py` Module 35 (15 tests) plus the extended Module 23 parity test; suite
|
||
`RESULTS: 521 passed, 0 failed, 521 total`.
|
||
|
||
## Reference books re-pinned (label-only)
|
||
|
||
The five books were regenerated with the committed code (stashed) and with the fix, on the same
|
||
cache. Dropping the `strategy` column, every book is byte-identical to its previous pin; trades, R
|
||
and PF are unchanged. New labels: M15 W2 174 `XAU_ATR` / 757 `GENERIC`; M5 W2 32 / 411 / 338
|
||
`XAU_CP`; M15 OOS 1 / 354; H1 W3 57 / 196; H4 W3 53 / 61.
|
||
|
||
| Book | old MD5 | new MD5 |
|
||
|---|---|---|
|
||
| M5 W2 | `29120a1ee32979431d062f210d41a9d1` | `49f8e602fc6ae769713c94ef7b257810` |
|
||
| M15 W2 | `d185f099482a07f29502cb5a371a9bbf` | `b4aa07e33e17e23774c486a6900d673b` |
|
||
| M15 OOS | `6f8f24b49597ced28f50a19e08c3472f` | `9fd624fb387c36052122bd1830cf9c32` |
|
||
| H1 W3 | `3b45d12bade823178b267c13bc698458` | `7b848c917dcb5c371ecbb9f8f7292278` |
|
||
| H4 W3 | `e75adae13c839b15736f69c0114e6aa5` | `186e677fbbd53cca735c94094c23ce05` |
|
||
|
||
`Backend: NumPy CPU fallback`, `GPU cache fallbacks: 0` on all ten runs.
|
||
|
||
## Base-rate table
|
||
|
||
`agent_panel_base_rates.json`: 45 classes (34 exact) from the relabelled M5 W2, M15 W2, H1 W3 and
|
||
H4 W3 books; key `timeframe|strategy|grade|direction`, fallbacks `tf|*|*|direction` and `tf|*|*|*`;
|
||
the exact class is used only with n ≥ 30. Examples: M15 all 931 trades win 60.5%, median hold 30
|
||
bars; M15 SELL 296 / 54.4%; M15 `XAU_ATR` A SELL 78 / 50.0%, median hold 6 bars; M5 `XAU_CP` B
|
||
BUY 321 / 51.7%; H1 all 253 / 64.4%.
|
||
|
||
## Pre-registered acceptance (from the approved plan)
|
||
|
||
On the resolved logged setups (37 today), a new prompt version ships only if: (1) no persona's
|
||
modal stance is above 65%; (2) Brier ≤ the logged v1 takes and ≤ the base-rate-only predictor;
|
||
(3) the skip-OPPOSE counterfactual does not remove a net-positive bucket. Outcome = advisory R > 0
|
||
at $0.65. Predictors compared on the same events: base-rate only (the class win rate for every
|
||
trade), the logged v1 takes (stance × confidence → pseudo-probability; 27 events carry them), and
|
||
the new prompts run live through DeepSeek on the rebuilt payloads (4 calls per event).
|
||
|
||
## Replay round 1 (draft prompts)
|
||
|
||
| predictor | n | Brier | log-loss | mean p | modal stance | constant % | R skip ≤ 40 |
|
||
|---|---|---|---|---|---|---|---|
|
||
| base-rate only | 37 | 0.263 | 0.719 | 54.8 | neutral | 81 | −7.15 (nothing skipped) |
|
||
| v1 logged, mean | 27 | 0.270 | 0.738 | 48.2 | neutral | 48 | −4.00 |
|
||
| v1 logged, macro | 27 | 0.299 | 0.855 | 56.5 | support | 44 | −2.82 |
|
||
| v1 logged, technical | 27 | 0.371 | 1.011 | 71.7 | support | 78 | −6.47 |
|
||
| v1 logged, risk | 27 | 0.343 | 0.947 | 25.7 | oppose | 78 | +0.03 |
|
||
| v2 draft, mean | 37 | 0.257 | 0.707 | 51.3 | neutral | 95 | −7.15 (nothing skipped) |
|
||
|
||
Like-for-like on the 27 events with v1 takes: Brier v2 0.257, v1 mean 0.270, v1 macro 0.299,
|
||
base rate 0.259, coin flip 0.250. Rank correlation of p with R: v2 −0.05, v1 mean +0.14, v1 macro
|
||
+0.36, base rate +0.16 (n = 27; the ±0.35 confidence band swallows all of them).
|
||
|
||
Reading: the draft is calibrated (it sits 3.5 points under the base rate with a spread of 1.7
|
||
points) and no longer directional, but it carries no rank information: winners averaged p = 50.9,
|
||
losers 51.7. The old macro verdict had some ordering (+0.36) and terrible calibration (82%
|
||
confidence on the three losing FOMC shorts); the new one has calibration and no ordering. Criterion
|
||
(2) passes, (3) is vacuous (no OPPOSE issued), (1) fails **in the opposite sense to the one it was
|
||
written for**: the personas cluster on neutral around the base rate instead of voting a constant
|
||
direction. That is hedging, not bias.
|
||
|
||
**A flaw in the draft prompts, found from the per-event reasons.** The five lowest probabilities
|
||
went to the Sep 10 and Sep 14 shorts that won (+1.31, +1.05, +0.70R), because the technical and
|
||
contrarian prompts told the personas to lower p when "the move has already run far" or "a big
|
||
candle just printed". That instruction encoded the live fortnight's pattern, which the two-year
|
||
books contradict (stretched entries are the system's best trades, 2026-08-04; the big-candle gate
|
||
was re-rejected in the review: those buckets are +32R and +60R). The final prompts remove that
|
||
instruction, state the book's finding in the context legend and in both personas, and limit
|
||
downward moves to higher-timeframe conflict, key levels, event timing inside the expected hold,
|
||
news newer than the move, and a stop far wider than the class norm.
|
||
|
||
## Replay round 2 (final prompts, the text that ships)
|
||
|
||
Same 37 events, same rebuilt payloads, the final prompt text (the technical and contrarian
|
||
personas told that a long prior move or a big candle is not a reason by itself; downward moves
|
||
limited to higher-timeframe conflict, key levels, event timing, news newer than the move, and a
|
||
stop far wider than the class norm).
|
||
|
||
| predictor | n | Brier | log-loss | mean p | modal stance | directional % | R skip ≤ 40 | R ≥ 60 only |
|
||
|---|---|---|---|---|---|---|---|---|
|
||
| base-rate only | 37 | 0.263 | 0.719 | 54.8 | neutral | 19 | −7.15 | −3.07 |
|
||
| v1 logged, mean | 27 | 0.270 | 0.738 | 48.2 | neutral | 52 | −4.00 | −1.66 |
|
||
| v1 logged, macro | 27 | 0.299 | 0.855 | 56.5 | support | 56 | −2.82 | −2.16 |
|
||
| **v2 final, mean** | 37 | **0.262** | 0.717 | 53.8 | neutral | 8 | −7.15 | −1.98 |
|
||
| v2 final, macro | 37 | 0.259 | 0.712 | 53.4 | neutral | 11 | | |
|
||
| v2 final, technical | 37 | 0.270 | 0.734 | 54.2 | neutral | 16 | | |
|
||
| v2 final, risk | 37 | 0.258 | 0.709 | 53.3 | neutral | 11 | | |
|
||
| v2 final, contrarian | 37 | 0.262 | 0.718 | 54.1 | neutral | 11 | | |
|
||
|
||
Like-for-like on the 27 events with v1 takes: Brier v2 final 0.260, v1 mean 0.270, v1 macro
|
||
0.299, base rate 0.259. Rank correlation of p with R: v2 final +0.13 (round 1 −0.05), v1 mean
|
||
+0.14, v1 macro +0.36. Calibration of the v2 mean: 34 events in the 40–60 band averaged p = 53
|
||
and won 47%; the 3 events at 60–61 won 1. The final personas sit 1.0 point under the base rate
|
||
(sd 1.8, no event moved 5 points or more); winners averaged p = 53.1, losers 54.3.
|
||
|
||
Acceptance: (2) passes — Brier below the logged v1 takes and equal to the base-rate predictor
|
||
(0.262 vs 0.263; 0.260 vs 0.259 on the common 27); (3) passes vacuously (no OPPOSE issued); (1)
|
||
fails as written (92% neutral) and passes as re-specified below (directional share 8%). The
|
||
correction removed the chase penalty: the winning Sep 10 / Sep 14 shorts moved from 45–47 to
|
||
49–51, and the reasons against now name what the books do not already price (a release due in 20
|
||
minutes, price sitting on a support level) instead of "the move already ran far".
|
||
|
||
Cost: 2 × 148 DeepSeek calls (both rounds), inside the shared daily cap of 500.
|
||
|
||
## Decision and what it means for the alert
|
||
|
||
The shipped verdict is a probability anchored to the validated base rate that moves only for
|
||
facts the books do not already price. On this sample it is calibrated and uninformative: the line
|
||
will usually read "about a 50% chance … (usual for this kind of setup: 55%)". That is the honest
|
||
state of the evidence — the 65-feature sizing model was a clean null too — and it is strictly
|
||
better than the previous line, which asserted 82% confidence in the wrong direction. The
|
||
information the line adds is the base rate itself, the biggest concrete worry (event inside the
|
||
hold, news newer than the move), and each persona's reason. It never gates; the user guide says
|
||
so.
|
||
|
||
Re-specified criterion (1) for future prompt versions: the share of *directional* stances
|
||
(support + oppose) on one side must not exceed 65% — neutral clustering around a calibrated base
|
||
rate is not the pathology the criterion targets.
|
||
|
||
## Persona set (owner decision, same day)
|
||
|
||
The owner's rule: a persona that does not give correct guidance is removed, to save confusion,
|
||
tokens and log space. Evidence per persona:
|
||
|
||
| persona | v1 (27 outcomes) | v2 final (37 outcomes) | verdict |
|
||
|---|---|---|---|
|
||
| macro | Brier 0.299, rank correlation with R **+0.36**, 88% / 25% win split on 2026-09-03 | Brier 0.259, ρ −0.06 | the only voice that ever showed ordering, and the only one holding facts the engine lacks (news) → **kept** |
|
||
| technical | support on 79%, Brier 0.371 (worst) | Brier 0.270 (worst), ρ −0.20 | re-reads the indicators the signal was built from → removed |
|
||
| risk | oppose on 79%, Brier 0.343 | Brier 0.258, ρ −0.02 | its inputs (stop in ATR, cost, event timing) are now computed in the context block → removed |
|
||
| contrarian | oppose on 65%, Brier 0.322 | Brier 0.262, ρ −0.05 | oppose by construction, no information → removed |
|
||
|
||
Shipped as `agent_panel.personas` (default `["macro"]`; `app_config` validates the names, `chair`
|
||
needs two or more, `macro` source needs the macro persona). One DeepSeek call per panel instead of
|
||
four (three quarters of the panel's token spend), `panel_payloads_json` about 4 KB instead of about
|
||
14 KB, `panel_takes_json` one entry, no persona votes line in the alert. The mean verdict with one
|
||
persona is that persona's probability (quorum = min(3, personas)). The full panel remains one
|
||
config line away for research replays (`scripts/panel_replay.py --personas macro,technical,risk,contrarian`).
|
||
|
||
## Next evaluation
|
||
|
||
- Every live v2 verdict logs its exact payloads, so `scripts/panel_replay.py` can re-score any
|
||
future prompt on identical evidence. At 100 resolved v2 outcomes (about five weeks): the
|
||
`--stored-candles` probability scorecard by prompt version. At 300: fit a probability-band →
|
||
observed-win-rate table on the first half by time, verify on the second, ship it as a numpy
|
||
lookup; only then revisit persona weights (owner rule).
|
||
- Do not iterate prompts on the current 37 events again: any further gain there is in-sample.
|
||
- Deployment: the live box needs the new code (both the bot and the dashboard daemon, which has
|
||
run `2ac7301` since Sep 10); no config change is required (`verdict_source` defaults to
|
||
`mean`); the additive migration runs on first open.
|
||
|
||
Commands: `venv/bin/python scripts/build_panel_base_rates.py --book 5min=<m5_w2.csv> --book
|
||
15min=<m15_w2.csv> --book 1h=<h1_w3.csv> --book 4h=<h4_w3.csv> --out agent_panel_base_rates.json`;
|
||
`venv/bin/python scripts/panel_replay.py --db logs/signals.db --dry-run`; `venv/bin/python
|
||
scripts/panel_replay.py --db logs/signals.db --max-calls 160 --out replay.csv`;
|
||
`venv/bin/python analyze_signal_log.py --db logs/signals.db --stored-candles`.
|