ForexBot2/NEWS_MODEL_HANDOFF_2026-09-27.md

Handoff: the local news model and ForexBot2 (2026-09-27)

Status: IMPLEMENTED later the same day (news_model.py, news_model config section, Module 38, deploy/README-news-model.md). Two decisions were added during implementation: primary mode writes nothing to logs/newsdata.db except the DeepSeek audit sample, and both processes load the model. The authoritative record is the status note at the top of docs/news_model_integration_spec_2026-09-27.md and the CLAUDE.md ledger row; this file is kept as the historical handoff and can be deleted.

Read this before touching headline scoring. It records what exists, what the owner decided, and what was left to build at the time of writing.

What exists

The owner's news model lives in a separate project, ~/Development/ML-NEWS (git ssh://git@192.168.0.11:222/harry/News-ML-Decoder.git). On 2026-09-27 it became a DeBERTa-v3-small two-head transformer, exported to ONNX for CPU:

• Question 1: does this headline move the gold price? Question 2: if so, up or down?
• Trained to copy DeepSeek's gold score on 17,245 headlines: the Sep 8–22 newsdata.db archive plus a 2019–2026 Reuters / CNBC / MarketWatch crawl that DeepSeek labelled (560 calls).
• Golden set 72/78. DeepSeek 78, the keyword scorer 24, the 2026-09-17 TF-IDF model 39.
• Agreement with DeepSeek on unseen live-feed days about 72% (DeepSeek agrees with itself about 90%).
• 19 ms per headline on one CPU thread, 9 ms on four. Needs only onnxruntime, tokenizers, numpy.
• Bundle: ML-NEWS/models/xau_transformer/{model.onnx (271 MB), tokenizer.json, meta.json}.
• Full results: ~/Development/ML-NEWS/eval/RESULTS_2026-09-27.md.

It passed the four promotion criteria fixed in advance on 2026-09-22 (golden ≥ 55/78, distractors ≥ 6/8, unseen-days agreement ≥ 62.8%, latency ≤ 20 ms). This supersedes the 2026-09-17 verdict "a fallback, not a replacement" in the CLAUDE.md ledger.

What the owner decided (2026-09-27)

1. Use it as the primary headline scorer, with no DeepSeek calls for headline scoring. DeepSeek stays for everything else: calendar-event direction (_llm_event_direction), the news summary, the dashboard's on-demand summary endpoint.
2. Model files live outside the ForexBot2 tree, for example /home/Forex/models/xau_transformer, copied by hand, path set in config.json. Never in git. ForexBot2 keeps deploying by git pull.
3. No shadow mode. The archive already stores every headline with its score, so offline replay covers the comparison.
4. Accepted consequence: with DeepSeek off for headlines, the archive stops receiving fresh DeepSeek labels, which is what ML-NEWS retrains on. An optional config knob (deepseek_audit_per_day, default 0) can label a few headlines a day for the archive only.

What is left to build

Everything. The specification is docs/news_model_integration_spec_2026-09-27.md in this repo (identical to ML-NEWS/docs/FOREXBOT2_INTEGRATION_SPEC.md). In one screen:

File	Change
requirements.txt	add onnxruntime, tokenizers (no "torch"/"transformers" anywhere in the file; a test greps for them)
news_model.py (new)	vendored copy of ML-NEWS news_transformer.py + a load-once singleton that never raises; onnxruntime imported lazily
app_config.py	optional news_model section: path, threads, mode (off/fallback/primary), deepseek_audit_per_day; NEWS_MODEL_MODE env override
news_engine.py	_apply_model_sentiment + a mode switch at the call site (~line 945); model score goes into xau_score and the XAU entry only, never into USD
news_archive.py	store teacher_model for model rows (the model rank already exists)
test_all.py	11 tests listed in the spec
deploy/README-news-model.md (new)	ops for both the bot and the dashboard daemon
config.json (live)	the new section

Rules that must survive any implementation:

• The model has a gold head only. Never write its score into a non-XAU currency: get_pair_sentiment computes XAU_mean − USD_mean and the signal aggregator acts on it.
• A headline the model calls not-relevant shows 0.0, whatever the raw score; the raw score still goes to xau_score for training data.
• Never write model results into _llm_sent_cache; that cache decides what is re-sent to DeepSeek.
• If the model fails to load, primary falls back to the DeepSeek path for that cycle, not to keywords.
• Both the bot and dashboard_publisher.py build a NewsEngine; each process that loads the model holds about 0.6 GB. Decide per process with the env override.
• News never enters signals. CLAUDE.md §7 applies unchanged; no book is affected by any of this.

How to verify when it is built

venv/bin/python test_all.py green; one news cycle in primary mode archives only score_source='model' rows with xau_score set and zero DeepSeek headline calls; renaming the model directory away still starts both processes and yields llm rows; mode="off" reproduces today's rows exactly.

Re-verify before editing

Every file:line number in the spec was checked on 2026-09-27 against 6af1237 and drifts with each commit. Check them first.