# DeepSeek news-analysis validation — 2026-08-06

**Verdict: headline sentiment scoring is excellent; event-direction calls are
directionally sound but ~11% of them misread the numeric comparison, and the
failure concentrates on Unemployment Claims / close K-M-suffixed values.**

Method (owner-directed, token-lean): independent expert re-analysis of a
stratified sample of DeepSeek's recorded answers, not a full statistical
backtest. Harness: `backtest/research/llm_news_eval.py` (replays releases
through the REAL `_llm_event_direction` / `_llm_headline_scores` — real
prompts, payloads, validation; raw wire traffic logged). 135 of the 586-event
corpus were replayed before the full run was stopped by owner request
(artifacts retained in `llm_news_eval_2026-08/`, resume-safe).

## Part A — event gold-direction (28-event stratified sample, Jan–Jun 2024)

Reviewer agreement with DeepSeek: **24/28 (86%)**. All 4 disagreements are
DeepSeek errors, in exactly two failure modes:

1. **Numeric comparison misread (3/28 ≈ 11%)** — the doctrine logic is right
   but the premise is backwards:
   - JOLTS 8.79M vs f 8.84M (a miss) → DS: "Job openings **above** forecast" → bearish (should be bullish)
   - Claims 202K vs f 217K (fewer claims) → DS: "Claims **higher** than forecast" → bullish (should be bearish)
   - Claims 202K vs f 209K — same backwards read, same wrong call
   Both Unemployment Claims events in the sample were misread the same way —
   looks systematic on that event type, possibly an anchoring prior that
   "claims rose". All misreads involve close values with K/M suffixes.
2. **Direction field contradicts its own reason (1/28)** — Empire State
   −43.7 vs f −4.9: reason text says "gold may rise", `gold_direction` says
   `bearish`.

Where the numbers were read correctly, hawkish/dovish→gold mapping was
consistently correct (CPI/PPI/NFP/ADP/GDP/PMI/wages/unemployment-rate all
handled properly, including inverted metrics when read right).

Price cross-check (1h horizon, cluster-confounded, indicative only):
reviewer calls 22/28 with the subsequent move, DeepSeek 20/28.

## Part B — headline sentiment (78-headline golden set)

DeepSeek matched the reviewer's expected gold-direction label on
**78/78 (100%)** — including all 8 negation traps ("signals no rate hike",
"no rate cuts until…") and all 6 inverted-metric phrasings ("claims hit
two-year high"). The keyword scorer it overrides got **24/78 (31%)**.
Repeat-batch and shuffled-order probes recorded (headlines_replay.jsonl).

## Implications

- The headline-sentiment override (the main live LLM surface) is validated —
  a large, measured upgrade over keywords.
- The event-direction enrichment ("— gold bullish/bearish" appended to
  surprise descriptions) is right ~85–90% of the time but can state a
  factually backwards comparison in the displayed reason. It is display-only
  (never gates signals), so exposure is a possibly-misleading Telegram/CLI
  line, not trades.
- **Hardening SHIPPED same day (owner-approved):** the payload now carries a
  code-computed `vs_forecast` field ("above/below/in line with forecast",
  from `EconomicEvent.surprise`) and the prompt tells the model to trust it.
  First iteration fixed the arithmetic but exposed a masked doctrine error
  (model read "claims below forecast" correctly, then called it a *weaker*
  labor market — and regressed the unemployment-rate case the same way), so
  the prompt's inverted-metrics line was expanded to an explicit rule plus a
  worked claims example. Verification: **18/18** — all 4 original misreads
  fixed, 11/11 no-regression cases unchanged, 3/3 fresh probes on the side
  the prompt example does NOT cover (claims 245K vs f 220K → bullish,
  unemployment 4.1% vs 3.9% → bullish, JOLTS 9.10M vs 8.80M → bearish).
  Suite green at 346 tests, incl. a new payload assertion in Module 19.
- Production observations found while building the harness (not fixed):
  `_llm_event_cache` key omits forecast → a repeated `actual` re-serves a
  direction computed for a different forecast; a "neutral" reply is
  indistinguishable from a failure in the return value.

## Reproduce

```
venv/bin/python backtest/research/llm_news_eval.py events --limit 5   # smoke
venv/bin/python backtest/research/llm_news_eval.py events --determinism 20
venv/bin/python backtest/research/llm_news_eval.py headlines
venv/bin/python backtest/research/llm_news_eval.py report             # offline
```

Corpus: 586 high-impact USD surprises 2024-01→2026-07 (calendar.parquet,
NFP-verified UTC); prices: Oanda M15 mids. Full-run cost ≈ cents. The
`report` subcommand (hit rates vs always-bullish/doctrine/FF-betterWorse
baselines, McNemar, cluster view) is built and untested at scale — run it
after completing the replay if a statistical answer is ever wanted.