# Handoff: the local news model and ForexBot2 (2026-09-27)

**Status: IMPLEMENTED later the same day** (`news_model.py`, `news_model` config section, Module 38,
`deploy/README-news-model.md`). Two decisions were added during implementation: `primary` mode writes
nothing to `logs/newsdata.db` except the DeepSeek audit sample, and both processes load the model. The
authoritative record is the status note at the top of `docs/news_model_integration_spec_2026-09-27.md`
and the CLAUDE.md ledger row; this file is kept as the historical handoff and can be deleted.

Read this before touching headline scoring. It records what exists, what the owner decided, and what
was left to build at the time of writing.

## What exists

The owner's news model lives in a separate project, `~/Development/ML-NEWS`
(git `ssh://git@192.168.0.11:222/harry/News-ML-Decoder.git`). On 2026-09-27 it became a
DeBERTa-v3-small two-head transformer, exported to ONNX for CPU:

- Question 1: does this headline move the gold price? Question 2: if so, up or down?
- Trained to copy DeepSeek's gold score on 17,245 headlines: the Sep 8–22 `newsdata.db` archive plus a
  2019–2026 Reuters / CNBC / MarketWatch crawl that DeepSeek labelled (560 calls).
- Golden set 72/78. DeepSeek 78, the keyword scorer 24, the 2026-09-17 TF-IDF model 39.
- Agreement with DeepSeek on unseen live-feed days about 72% (DeepSeek agrees with itself about 90%).
- 19 ms per headline on one CPU thread, 9 ms on four. Needs only `onnxruntime`, `tokenizers`, `numpy`.
- Bundle: `ML-NEWS/models/xau_transformer/{model.onnx (271 MB), tokenizer.json, meta.json}`.
- Full results: `~/Development/ML-NEWS/eval/RESULTS_2026-09-27.md`.

It passed the four promotion criteria fixed in advance on 2026-09-22 (golden ≥ 55/78, distractors ≥ 6/8,
unseen-days agreement ≥ 62.8%, latency ≤ 20 ms). This supersedes the 2026-09-17 verdict "a fallback,
not a replacement" in the CLAUDE.md ledger.

## What the owner decided (2026-09-27)

1. **Use it as the primary headline scorer, with no DeepSeek calls for headline scoring.** DeepSeek
   stays for everything else: calendar-event direction (`_llm_event_direction`), the news summary, the
   dashboard's on-demand summary endpoint.
2. **Model files live outside the ForexBot2 tree**, for example `/home/Forex/models/xau_transformer`,
   copied by hand, path set in `config.json`. Never in git. ForexBot2 keeps deploying by `git pull`.
3. **No shadow mode.** The archive already stores every headline with its score, so offline replay
   covers the comparison.
4. Accepted consequence: with DeepSeek off for headlines, the archive stops receiving fresh DeepSeek
   labels, which is what ML-NEWS retrains on. An optional config knob (`deepseek_audit_per_day`,
   default 0) can label a few headlines a day for the archive only.

## What is left to build

Everything. The specification is `docs/news_model_integration_spec_2026-09-27.md` in this repo
(identical to `ML-NEWS/docs/FOREXBOT2_INTEGRATION_SPEC.md`). In one screen:

| File | Change |
|---|---|
| `requirements.txt` | add `onnxruntime`, `tokenizers` (no "torch"/"transformers" anywhere in the file; a test greps for them) |
| `news_model.py` (new) | vendored copy of ML-NEWS `news_transformer.py` + a load-once singleton that never raises; onnxruntime imported lazily |
| `app_config.py` | optional `news_model` section: `path`, `threads`, `mode` (`off`/`fallback`/`primary`), `deepseek_audit_per_day`; `NEWS_MODEL_MODE` env override |
| `news_engine.py` | `_apply_model_sentiment` + a mode switch at the call site (~line 945); model score goes into `xau_score` and the XAU entry only, never into USD |
| `news_archive.py` | store `teacher_model` for `model` rows (the `model` rank already exists) |
| `test_all.py` | 11 tests listed in the spec |
| `deploy/README-news-model.md` (new) | ops for both the bot and the dashboard daemon |
| `config.json` (live) | the new section |

Rules that must survive any implementation:

- The model has a gold head only. **Never write its score into a non-XAU currency**:
  `get_pair_sentiment` computes `XAU_mean − USD_mean` and the signal aggregator acts on it.
- A headline the model calls not-relevant shows 0.0, whatever the raw score; the raw score still goes
  to `xau_score` for training data.
- Never write model results into `_llm_sent_cache`; that cache decides what is re-sent to DeepSeek.
- If the model fails to load, `primary` falls back to the DeepSeek path for that cycle, not to keywords.
- Both the bot and `dashboard_publisher.py` build a `NewsEngine`; each process that loads the model
  holds about 0.6 GB. Decide per process with the env override.
- News never enters signals. CLAUDE.md §7 applies unchanged; no book is affected by any of this.

## How to verify when it is built

`venv/bin/python test_all.py` green; one news cycle in `primary` mode archives only `score_source='model'`
rows with `xau_score` set and zero DeepSeek headline calls; renaming the model directory away still
starts both processes and yields `llm` rows; `mode="off"` reproduces today's rows exactly.

## Re-verify before editing

Every file:line number in the spec was checked on 2026-09-27 against `6af1237` and drifts with each
commit. Check them first.