Harwinder Sekhon
858c12280a
Validation (llm_news_eval.py harness + 78-headline golden CSV + verdict doc): headline scoring 78/78 vs keyword baseline 24/78 — validated; event gold-direction calls misread close K/M numeric comparisons ~11% of the time (both claims cases in sample, JOLTS 8.79M vs 8.84M). Fix: event payload now carries code-computed "vs_forecast" (from EconomicEvent.surprise) and the prompt gains an explicit inverted- metrics rule + worked claims example — iteration 1 (field alone) fixed the arithmetic but exposed a masked inversion error. 18/18 on live re-verification (4 misreads fixed, 11 regression cases clean, 3 fresh both-side probes). /status: new "News & AI" section — per-source health for the calendar chain (which of FF/MyFXBook served), each RSS feed, and DeepSeek (daily count, last error kind), written passively by the existing fetch paths under _cache_lock. Failures WARN once on transition (repeats DEBUG, recovery INFO) so the rotating log carries outage stories without 10-min spam. test_all.py: +5 Module 19 tests (351 total); also carries the Module 22 fixed-RR-TP tests from the prior research session (shared file). Replay artifacts dir gitignored like commit_ladder. Claude-Session: https://claude.ai/code/session_017mbXs9HRawZhD1uZ3ze7zQ
87 lines
4.8 KiB
Markdown
87 lines
4.8 KiB
Markdown
# DeepSeek news-analysis validation — 2026-08-06
|
||
|
||
**Verdict: headline sentiment scoring is excellent; event-direction calls are
|
||
directionally sound but ~11% of them misread the numeric comparison, and the
|
||
failure concentrates on Unemployment Claims / close K-M-suffixed values.**
|
||
|
||
Method (owner-directed, token-lean): independent expert re-analysis of a
|
||
stratified sample of DeepSeek's recorded answers, not a full statistical
|
||
backtest. Harness: `backtest/research/llm_news_eval.py` (replays releases
|
||
through the REAL `_llm_event_direction` / `_llm_headline_scores` — real
|
||
prompts, payloads, validation; raw wire traffic logged). 135 of the 586-event
|
||
corpus were replayed before the full run was stopped by owner request
|
||
(artifacts retained in `llm_news_eval_2026-08/`, resume-safe).
|
||
|
||
## Part A — event gold-direction (28-event stratified sample, Jan–Jun 2024)
|
||
|
||
Reviewer agreement with DeepSeek: **24/28 (86%)**. All 4 disagreements are
|
||
DeepSeek errors, in exactly two failure modes:
|
||
|
||
1. **Numeric comparison misread (3/28 ≈ 11%)** — the doctrine logic is right
|
||
but the premise is backwards:
|
||
- JOLTS 8.79M vs f 8.84M (a miss) → DS: "Job openings **above** forecast" → bearish (should be bullish)
|
||
- Claims 202K vs f 217K (fewer claims) → DS: "Claims **higher** than forecast" → bullish (should be bearish)
|
||
- Claims 202K vs f 209K — same backwards read, same wrong call
|
||
Both Unemployment Claims events in the sample were misread the same way —
|
||
looks systematic on that event type, possibly an anchoring prior that
|
||
"claims rose". All misreads involve close values with K/M suffixes.
|
||
2. **Direction field contradicts its own reason (1/28)** — Empire State
|
||
−43.7 vs f −4.9: reason text says "gold may rise", `gold_direction` says
|
||
`bearish`.
|
||
|
||
Where the numbers were read correctly, hawkish/dovish→gold mapping was
|
||
consistently correct (CPI/PPI/NFP/ADP/GDP/PMI/wages/unemployment-rate all
|
||
handled properly, including inverted metrics when read right).
|
||
|
||
Price cross-check (1h horizon, cluster-confounded, indicative only):
|
||
reviewer calls 22/28 with the subsequent move, DeepSeek 20/28.
|
||
|
||
## Part B — headline sentiment (78-headline golden set)
|
||
|
||
DeepSeek matched the reviewer's expected gold-direction label on
|
||
**78/78 (100%)** — including all 8 negation traps ("signals no rate hike",
|
||
"no rate cuts until…") and all 6 inverted-metric phrasings ("claims hit
|
||
two-year high"). The keyword scorer it overrides got **24/78 (31%)**.
|
||
Repeat-batch and shuffled-order probes recorded (headlines_replay.jsonl).
|
||
|
||
## Implications
|
||
|
||
- The headline-sentiment override (the main live LLM surface) is validated —
|
||
a large, measured upgrade over keywords.
|
||
- The event-direction enrichment ("— gold bullish/bearish" appended to
|
||
surprise descriptions) is right ~85–90% of the time but can state a
|
||
factually backwards comparison in the displayed reason. It is display-only
|
||
(never gates signals), so exposure is a possibly-misleading Telegram/CLI
|
||
line, not trades.
|
||
- **Hardening SHIPPED same day (owner-approved):** the payload now carries a
|
||
code-computed `vs_forecast` field ("above/below/in line with forecast",
|
||
from `EconomicEvent.surprise`) and the prompt tells the model to trust it.
|
||
First iteration fixed the arithmetic but exposed a masked doctrine error
|
||
(model read "claims below forecast" correctly, then called it a *weaker*
|
||
labor market — and regressed the unemployment-rate case the same way), so
|
||
the prompt's inverted-metrics line was expanded to an explicit rule plus a
|
||
worked claims example. Verification: **18/18** — all 4 original misreads
|
||
fixed, 11/11 no-regression cases unchanged, 3/3 fresh probes on the side
|
||
the prompt example does NOT cover (claims 245K vs f 220K → bullish,
|
||
unemployment 4.1% vs 3.9% → bullish, JOLTS 9.10M vs 8.80M → bearish).
|
||
Suite green at 346 tests, incl. a new payload assertion in Module 19.
|
||
- Production observations found while building the harness (not fixed):
|
||
`_llm_event_cache` key omits forecast → a repeated `actual` re-serves a
|
||
direction computed for a different forecast; a "neutral" reply is
|
||
indistinguishable from a failure in the return value.
|
||
|
||
## Reproduce
|
||
|
||
```
|
||
venv/bin/python backtest/research/llm_news_eval.py events --limit 5 # smoke
|
||
venv/bin/python backtest/research/llm_news_eval.py events --determinism 20
|
||
venv/bin/python backtest/research/llm_news_eval.py headlines
|
||
venv/bin/python backtest/research/llm_news_eval.py report # offline
|
||
```
|
||
|
||
Corpus: 586 high-impact USD surprises 2024-01→2026-07 (calendar.parquet,
|
||
NFP-verified UTC); prices: Oanda M15 mids. Full-run cost ≈ cents. The
|
||
`report` subcommand (hit rates vs always-bullish/doctrine/FF-betterWorse
|
||
baselines, McNemar, cluster view) is built and untested at scale — run it
|
||
after completing the replay if a statistical answer is ever wanted.
|