ForexBot2/backtest/research/llm_news_eval_2026-08-06.md

DeepSeek news-analysis validation — 2026-08-06

Verdict: headline sentiment scoring is excellent; event-direction calls are directionally sound but ~11% of them misread the numeric comparison, and the failure concentrates on Unemployment Claims / close K-M-suffixed values.

Method (owner-directed, token-lean): independent expert re-analysis of a stratified sample of DeepSeek's recorded answers, not a full statistical backtest. Harness: backtest/research/llm_news_eval.py (replays releases through the REAL _llm_event_direction / _llm_headline_scores — real prompts, payloads, validation; raw wire traffic logged). 135 of the 586-event corpus were replayed before the full run was stopped by owner request (artifacts retained in llm_news_eval_2026-08/, resume-safe).

Part A — event gold-direction (28-event stratified sample, Jan–Jun 2024)

Reviewer agreement with DeepSeek: 24/28 (86%). All 4 disagreements are DeepSeek errors, in exactly two failure modes:

1. Numeric comparison misread (3/28 ≈ 11%) — the doctrine logic is right but the premise is backwards:
   • JOLTS 8.79M vs f 8.84M (a miss) → DS: "Job openings above forecast" → bearish (should be bullish)
   • Claims 202K vs f 217K (fewer claims) → DS: "Claims higher than forecast" → bullish (should be bearish)
   • Claims 202K vs f 209K — same backwards read, same wrong call Both Unemployment Claims events in the sample were misread the same way — looks systematic on that event type, possibly an anchoring prior that "claims rose". All misreads involve close values with K/M suffixes.
2. Direction field contradicts its own reason (1/28) — Empire State −43.7 vs f −4.9: reason text says "gold may rise", gold_direction says bearish.

Where the numbers were read correctly, hawkish/dovish→gold mapping was consistently correct (CPI/PPI/NFP/ADP/GDP/PMI/wages/unemployment-rate all handled properly, including inverted metrics when read right).

Price cross-check (1h horizon, cluster-confounded, indicative only): reviewer calls 22/28 with the subsequent move, DeepSeek 20/28.

Part B — headline sentiment (78-headline golden set)

DeepSeek matched the reviewer's expected gold-direction label on 78/78 (100%) — including all 8 negation traps ("signals no rate hike", "no rate cuts until…") and all 6 inverted-metric phrasings ("claims hit two-year high"). The keyword scorer it overrides got 24/78 (31%). Repeat-batch and shuffled-order probes recorded (headlines_replay.jsonl).

Implications

• The headline-sentiment override (the main live LLM surface) is validated — a large, measured upgrade over keywords.
• The event-direction enrichment ("— gold bullish/bearish" appended to surprise descriptions) is right ~85–90% of the time but can state a factually backwards comparison in the displayed reason. It is display-only (never gates signals), so exposure is a possibly-misleading Telegram/CLI line, not trades.
• Hardening SHIPPED same day (owner-approved): the payload now carries a code-computed vs_forecast field ("above/below/in line with forecast", from EconomicEvent.surprise) and the prompt tells the model to trust it. First iteration fixed the arithmetic but exposed a masked doctrine error (model read "claims below forecast" correctly, then called it a weaker labor market — and regressed the unemployment-rate case the same way), so the prompt's inverted-metrics line was expanded to an explicit rule plus a worked claims example. Verification: 18/18 — all 4 original misreads fixed, 11/11 no-regression cases unchanged, 3/3 fresh probes on the side the prompt example does NOT cover (claims 245K vs f 220K → bullish, unemployment 4.1% vs 3.9% → bullish, JOLTS 9.10M vs 8.80M → bearish). Suite green at 346 tests, incl. a new payload assertion in Module 19.
• Production observations found while building the harness (not fixed): _llm_event_cache key omits forecast → a repeated actual re-serves a direction computed for a different forecast; a "neutral" reply is indistinguishable from a failure in the return value.

Reproduce

venv/bin/python backtest/research/llm_news_eval.py events --limit 5   # smoke
venv/bin/python backtest/research/llm_news_eval.py events --determinism 20
venv/bin/python backtest/research/llm_news_eval.py headlines
venv/bin/python backtest/research/llm_news_eval.py report             # offline

Corpus: 586 high-impact USD surprises 2024-01→2026-07 (calendar.parquet, NFP-verified UTC); prices: Oanda M15 mids. Full-run cost ≈ cents. The report subcommand (hit rates vs always-bullish/doctrine/FF-betterWorse baselines, McNemar, cluster view) is built and untested at scale — run it after completing the replay if a statistical answer is ever wanted.