ForexBot2/backtest/research/ai_signal_enrichment_2026-08-15.md

AI signal enrichment — analysis + build record (2026-08-15)

Owner questions:

1. If the AI panel also received indicator values and (limited) candle data, would the system be more accurate?
2. If AI reviewed every candle alongside the local score system, would prediction accuracy justify the token burn — or is there a middle ground?

What the panel already received (pre-change baseline)

The four personas (agent_panel.py) already got a distilled picture: per-indicator signal+confidence, MTF signals/scores, key levels, exhaustion count, full trade-setup math, warnings, headlines/events/ sentiment. Missing: raw indicator values and any candle data.

Cost guardrails already in place: 5 requests/panel, max_panels_per_day 20 (≤100 requests/day), 3600 s reuse-TTL, per-bar exact cache, shared 500 req/day DeepSeek cap, static prompts on the ~50× prefix cache.

Evidence used

• LLM is strong at text, weak at raw numerics (measured in-repo, llm_news_eval_2026-08-06.md): headline sentiment 100% (78/78) vs keyword scorer 31%; but ~11% of event-direction calls misread close numeric comparisons until code pre-computed the comparison (vs_forecast) and the model only mapped it to doctrine.
• The book any filter would act on (M5 canonical, oanda_m5_12mo_065_nopb_20260814.csv): 577 trades, +106.82R, PF 1.51, win rate 52.9%; A-grade 59.5%, B-grade 51.4% (+67.8R of the total), C-grade 43.8% (−0.23R, 16 trades). Average edge ≈ 0.185R/trade.

Answer 1 — enrich the existing panel? YES (cheap, safe)

• A compact candle is ~15–25 tokens; 20 primary-TF + 10 confirm-TF bars ≈ 500–1,500 tokens per persona call → cents/day at the 20-panel cap. Tokens are not the constraint; the request cap is.
• Measured after the build (dry-test wire log): enriched technical payload with 20 candles ≈ ~380 tokens; replies ≈ 25–35 tokens under the new 15-word note cap.
• Caveats: (a) the panel is advisory-only — richer payloads improve the human's context, not the book's P&L; (b) raw prices are the model's weakest input, so candles are sent ATR-normalized (code does the arithmetic, LLM reads the shape) — the same pattern that fixed the event-direction misreads.

Answer 2 — AI on every candle? NO

• Feasibility: M5+M15+H1+H4 ≈ 414 bars/day → one call/bar nearly exhausts the shared 500/day cap (news needs it too); a panel per bar ≈ 2,070/day = 4× over. Most of that spend reviews HOLD bars.
• Parity (rule 4): an AI gate cannot be replayed by the backtester (news off; no recorded verdicts) — every pinned book becomes non-comparable, and model-version drift silently mutates the strategy.
• Edge: the engine already compresses 380 bars into 9 tuned indicators + regime + MTF validated over 5 years; an LLM's generic prior on raw candles won't beat fitted thresholds, and the margin it would filter is ~0.185R/trade. B-grade holds +67.8R at 51.4% win — a blunt veto is net-negative.

The middle ground (adopted)

1. Enrich payloads (built today, see below) — near-zero cost, no rule-1 exposure.
2. Log panel verdicts to signals.db (built today) — creates the "does OPPOSE predict losers?" dataset with zero behavior change.
3. After 4–8 weeks: join logged verdicts to outcomes; simulate an OPPOSE-veto against the pinned book offline. Only then consider any gating, as a full rule-1 revalidation.

Build record (2026-08-15, owner-approved)

• agent_panel.py
  • Technical persona payload now carries indicator values (units legend added to the prompt) and ATR-normalized candles per TF: {"ref_close", "atr", "bars": [[o,h,l,c]…]} oldest-first, each price as (price − ref_close) / atr, 1 decimal. Window: 20 bars primary / 10 per confirmation TF.
  • New config keys (validated in app_config.py, in config.json.example): candles_enabled (default true), candle_bars_primary (20), candle_bars_confirm (10).
  • Panel-private provider singleton (_dp, patchable in tests); TTL cache keeps fetches near-free; any failure → candles silently omitted, panel unaffected.
  • Prompt hardening: training data declared stale and OFF-LIMITS; "if a fact is not in the JSON, treat it as unknown"; thin evidence → neutral < 30 (unchanged); note cap 25→15 words, rationale 40→30 words (output-token savings).
  • attach_panel now calls signal_log.log_panel_update(report) after every successful attach (fresh, exact-cache, and reuse-TTL paths).
• signal_log.py
  • log() returns the rowid; log_signal stashes it on the report as _signal_log_rowid; new log_panel_update() fills the 4 panel columns on that row post-hoc. No rowid (backtest replay, logging disabled, other process) → silent no-op. Fail-safe contract kept.
• Tests: 381 → 388 passed (7 new: candle normalization math, payload values/candles, context windows + ATR gating + disable flag, attach injection (technical persona only), prompt-contract assertions, panel row update, rowid hand-off + no-op paths). Existing attach tests stub ap._dp to stay off the network.
• Live dry test (panel_payload_drytest_2026-08-15.py — script removed 2026-09-06, raw replies kept in the .jsonl — fabricated numbers only, 13 real DeepSeek requests total): 7/7 probes pass — coherent-bullish → support@70; bearish-vs-BUY → oppose@85; genuinely mixed → neutral@30; empty payload → neutral@20 (contract); no invented numbers in any note; all notes ≤ 15 words; synthesizer AGREE with a 21-word rationale. Raw wire log: backtest/research/panel_payload_drytest_2026-08-15.jsonl.
  • Notable: an early P3 variant with H1 SELL against a BUY brief got oppose@60 — correct per the prompt (higher-TF disagreement is a listed conflict), so the probe was tightened to a genuinely mixed scenario. Documented so nobody "fixes" the prompt for it later.

What this deliberately did NOT change

• No gate reads panel fields; the panel remains advisory-only.
• No backtest-path change (analyze_from_dataframes untouched; parity test still locks panel fields empty in replay).
• No verdict-based filtering — that decision waits for the signals.db dataset (4–8 weeks) and an offline veto replay against the pinned book.