agent_panel: ATR-normalized candle + indicator-value enrichment, hardened prompts, verdict logging

backtest/research/ai_signal_enrichment_2026-08-15.md

@@ -0,0 +1,116 @@
+# AI signal enrichment — analysis + build record (2026-08-15)
+
+Owner questions:
+1. If the AI panel also received indicator values and (limited) candle data, would the system be more accurate?
+2. If AI reviewed every candle alongside the local score system, would prediction accuracy justify the token burn — or is there a middle ground?
+
+## What the panel already received (pre-change baseline)
+
+The four personas (`agent_panel.py`) already got a *distilled* picture:
+per-indicator signal+confidence, MTF signals/scores, key levels,
+exhaustion count, full trade-setup math, warnings, headlines/events/
+sentiment. Missing: raw indicator **values** and any **candle** data.
+
+Cost guardrails already in place: 5 requests/panel, `max_panels_per_day`
+20 (≤100 requests/day), 3600 s reuse-TTL, per-bar exact cache, shared
+500 req/day DeepSeek cap, static prompts on the ~50× prefix cache.
+
+## Evidence used
+
+- **LLM is strong at text, weak at raw numerics** (measured in-repo,
+  `llm_news_eval_2026-08-06.md`): headline sentiment 100% (78/78) vs
+  keyword scorer 31%; but ~11% of event-direction calls misread close
+  numeric comparisons until code pre-computed the comparison
+  (`vs_forecast`) and the model only mapped it to doctrine.
+- **The book any filter would act on** (M5 canonical,
+  `oanda_m5_12mo_065_nopb_20260814.csv`): 577 trades, +106.82R,
+  PF 1.51, win rate 52.9%; A-grade 59.5%, B-grade 51.4% (+67.8R of the
+  total), C-grade 43.8% (−0.23R, 16 trades). Average edge ≈ 0.185R/trade.
+
+## Answer 1 — enrich the existing panel? YES (cheap, safe)
+
+- A compact candle is ~15–25 tokens; 20 primary-TF + 10 confirm-TF bars
+  ≈ 500–1,500 tokens per persona call → cents/day at the 20-panel cap.
+  **Tokens are not the constraint; the request cap is.**
+- Measured after the build (dry-test wire log): enriched technical
+  payload with 20 candles ≈ **~380 tokens**; replies ≈ 25–35 tokens
+  under the new 15-word note cap.
+- Caveats: (a) the panel is advisory-only — richer payloads improve the
+  human's context, not the book's P&L; (b) raw prices are the model's
+  weakest input, so candles are sent **ATR-normalized** (code does the
+  arithmetic, LLM reads the shape) — the same pattern that fixed the
+  event-direction misreads.
+
+## Answer 2 — AI on every candle? NO
+
+- **Feasibility**: M5+M15+H1+H4 ≈ 414 bars/day → one call/bar nearly
+  exhausts the shared 500/day cap (news needs it too); a panel per bar
+  ≈ 2,070/day = 4× over. Most of that spend reviews HOLD bars.
+- **Parity (rule 4)**: an AI gate cannot be replayed by the backtester
+  (news off; no recorded verdicts) — every pinned book becomes
+  non-comparable, and model-version drift silently mutates the strategy.
+- **Edge**: the engine already compresses 380 bars into 9 tuned
+  indicators + regime + MTF validated over 5 years; an LLM's generic
+  prior on raw candles won't beat fitted thresholds, and the margin it
+  would filter is ~0.185R/trade. B-grade holds +67.8R at 51.4% win —
+  a blunt veto is net-negative.
+
+## The middle ground (adopted)
+
+1. **Enrich payloads** (built today, see below) — near-zero cost, no
+   rule-1 exposure.
+2. **Log panel verdicts to signals.db** (built today) — creates the
+   "does OPPOSE predict losers?" dataset with zero behavior change.
+3. **After 4–8 weeks**: join logged verdicts to outcomes; simulate an
+   OPPOSE-veto against the pinned book offline. Only then consider any
+   gating, as a full rule-1 revalidation.
+
+## Build record (2026-08-15, owner-approved)
+
+- `agent_panel.py`
+  - Technical persona payload now carries indicator **values** (units
+    legend added to the prompt) and **ATR-normalized candles** per TF:
+    `{"ref_close", "atr", "bars": [[o,h,l,c]…]}` oldest-first, each price
+    as `(price − ref_close) / atr`, 1 decimal. Window: 20 bars primary /
+    10 per confirmation TF.
+  - New config keys (validated in `app_config.py`, in
+    `config.json.example`): `candles_enabled` (default true),
+    `candle_bars_primary` (20), `candle_bars_confirm` (10).
+  - Panel-private provider singleton (`_dp`, patchable in tests);
+    TTL cache keeps fetches near-free; any failure → candles silently
+    omitted, panel unaffected.
+  - Prompt hardening: training data declared stale and OFF-LIMITS;
+    "if a fact is not in the JSON, treat it as unknown"; thin evidence
+    → neutral < 30 (unchanged); note cap 25→**15 words**, rationale
+    40→**30 words** (output-token savings).
+  - `attach_panel` now calls `signal_log.log_panel_update(report)` after
+    every successful attach (fresh, exact-cache, and reuse-TTL paths).
+- `signal_log.py`
+  - `log()` returns the rowid; `log_signal` stashes it on the report as
+    `_signal_log_rowid`; new `log_panel_update()` fills the 4 panel
+    columns on that row post-hoc. No rowid (backtest replay, logging
+    disabled, other process) → silent no-op. Fail-safe contract kept.
+- Tests: 381 → **388 passed** (7 new: candle normalization math, payload
+  values/candles, context windows + ATR gating + disable flag, attach
+  injection (technical persona only), prompt-contract assertions,
+  panel row update, rowid hand-off + no-op paths). Existing attach tests
+  stub `ap._dp` to stay off the network.
+- **Live dry test** (`panel_payload_drytest_2026-08-15.py`, fabricated
+  numbers only, 13 real DeepSeek requests total): **7/7 probes pass** —
+  coherent-bullish → support@70; bearish-vs-BUY → oppose@85; genuinely
+  mixed → neutral@30; empty payload → neutral@20 (contract); no invented
+  numbers in any note; all notes ≤ 15 words; synthesizer AGREE with a
+  21-word rationale. Raw wire log:
+  `backtest/research/panel_payload_drytest_2026-08-15.jsonl`.
+  - Notable: an early P3 variant with H1 SELL against a BUY brief got
+    oppose@60 — *correct* per the prompt (higher-TF disagreement is a
+    listed conflict), so the probe was tightened to a genuinely mixed
+    scenario. Documented so nobody "fixes" the prompt for it later.
+
+## What this deliberately did NOT change
+
+- No gate reads panel fields; the panel remains advisory-only.
+- No backtest-path change (`analyze_from_dataframes` untouched; parity
+  test still locks panel fields empty in replay).
+- No verdict-based filtering — that decision waits for the signals.db
+  dataset (4–8 weeks) and an offline veto replay against the pinned book.