harry/ForexBot2
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

Local news model: ML-NEWS XAU transformer scores headlines (news_model.py, mode primary)

Browse Source 
The owner's DeBERTa-v3-small two-head ONNX model (golden set 72/78, ~72%
agreement with DeepSeek on unseen days) replaces DeepSeek for headline
scoring. DeepSeek keeps event direction, the summary and the dashboard
endpoint.

- news_model.py: vendored ML-NEWS runtime (lazy onnxruntime/tokenizers,
  so the bot starts without them) + load-once, never-raising singleton.
- news_engine: _enrich_headline_scores mode switch (off/fallback/primary),
  _apply_model_sentiment (gold head only: xau_score + XAU entry, never
  USD; not-relevant shows 0.0; never touches _llm_sent_cache),
  _audit_with_deepseek (archive-only labels, deepseek_audit_per_day).
- Owner decision: primary mode writes nothing to logs/newsdata.db except
  audit copies — the model's output is not training data.
- app_config: news_model section (path/threads/mode/deepseek_audit_per_day)
  + NEWS_MODEL_MODE env override; validated at boot.
- news_archive: model rows carry teacher_model xau_transformer@<trained_at>.
- deploy/README-news-model.md (ops for bot + dashboard daemon), unit gets a
  commented NEWS_MODEL_MODE=off line; requirements + onnxruntime, tokenizers.
- test_all.py Module 38 (15 tests, one against the real bundle). Suite 581.
- CLAUDE.md, spec status note, handoff marked implemented.

Books unaffected: news never enters signals; no signal file touched.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
This commit is contained in:
Harwinder Sekhon
2026-09-27 21:02:42 -04:00
parent 6af1237c2d
commit 0c56b4e155
13 changed files with 1966 additions and 34 deletions
Download Patch File Download Diff File Expand all files Collapse all files
docs/news_model_integration_spec_2026-09-27.md
+535
View File
@@ -0,0 +1,535 @@
# Specification: run the ML-NEWS XAU transformer inside ForexBot2
Status: **IMPLEMENTED 2026-09-27** (ForexBot2 branch `sql-integration`, on top of `6af1237`; suite
Module 38). The text below is the specification as written before the implementation; file:line
numbers refer to `6af1237` and have drifted. Where the implementation differs, this list wins:
- **`primary` mode does not archive the cycle's rows.** Owner decision during implementation: once the
local model scores, `logs/newsdata.db` stops growing, because the model's own output is not training
data. Only the DeepSeek audit copies (`deepseek_audit_per_day`, shipped 0) are written. `off` and
`fallback` archive as before (a `fallback` model row carries `model_teacher`). §5.4 and §9 read
accordingly.
- `_audit_with_deepseek` **returns** the `llm` copies; there is no `_pending_audit_rows` attribute.
The mode switch of §5.2 lives in `NewsEngine._enrich_headline_scores(scored, mode)`, which returns
those copies, so the call site and the tests share one function.
- `config.json.example` ships `"mode": "off"`; the live server sets `primary`
(`deploy/README-news-model.md`).
- §8 step 0 was wrong: the dashboard daemon never writes the archive in any mode
(`set_db_writes_for_process(False)`), so its DeepSeek rows are not "free audit labels". Both
processes load the model (owner decision); `NEWS_MODEL_MODE=off` stays as the memory fallback.
- `warm_up_in_background()` is implemented (module-level once flag; a `NewsEngine` is built per scan).
- `news_model.get_news_model()` also treats a missing `path` as a load failure (one warning).
- `requirements.txt` ranges: `onnxruntime>=1.19,<2`, `tokenizers>=0.22,<1`. The live box runs
Python 3.9, where only 1.19.2 / 0.22.2 install; verified on 2026-09-28 to score the bundle
identically to the dev box's 1.30.0 / 0.23.2 (golden pair, four probe headlines).
- ML-NEWS `news_transformer.py` received the §3.4 edits in the same session; the vendored copy differs
only in `load_model` having no default directory.
Repos:
- **ML-NEWS** `~/Development/ML-NEWS` (git `ssh://git@192.168.0.11:222/harry/News-ML-Decoder.git`).
Produces the model. Nothing in ForexBot2 imports from it.
- **ForexBot2** `~/Development/ForexBot2`, live at `/home/Forex/ForexBot2`, deployed by `git pull`.
Consumes the model.
---
## 1. Context and goal
### 1.1 What the model is
`ML-NEWS/models/xau_transformer/` holds a fine-tuned `microsoft/deberta-v3-small` with two heads,
exported to ONNX:
| File | Size | Purpose |
|---|---|---|
| `model.onnx` | 271 MB | encoder + heads; int8 embedding table, fp32 compute |
| `tokenizer.json` | 8 MB | HF `tokenizers` fast tokenizer |
| `meta.json` | 4 KB | thresholds, `trained_at`, `relevance_threshold`, `export_checks`, training stats |
Inputs: one headline, ≤ 64 tokens. Outputs per headline (from `news_transformer.score_batch`):
```python
{"headline": str, "p_relevant": float, "p_bullish": float,
"relevant": bool, # p_relevant >= meta["relevance_threshold"] (0.2 for the shipped bundle)
"direction": "bullish"|"bearish",
"label": "bullish"|"bearish"|"neutral", # neutral when not relevant
"score": float} # p_relevant * (2*p_bullish - 1), in [-1, 1]
```
`score` is on the same scale as DeepSeek's `xau` score that ForexBot2 already stores in
`xau_score`. The model was trained to copy DeepSeek (`deepseek-v4-flash`, prompt sha1 `d32e04c6…`, the
same prompt as `news_engine._LLM_SENTIMENT_PROMPT_SHA1`) on 17,245 headlines. Quality on
2026-09-27: golden set 72/78 (DeepSeek 78, keyword scorer 24), agreement with DeepSeek on unseen live
days ~72%, CPU latency 19 ms/headline on 1 thread, 9 ms on 4. Full numbers: `ML-NEWS/eval/RESULTS_2026-09-27.md`.
**The model scores gold only.** It has no head for USD, EUR or anything else. This drives most of
the rules in §5.
### 1.2 How ForexBot2 scores headlines today
`news_engine.py`, inside `NewsEngine._fetch_news_sentiment_raw` (called via
`_cached_fetch("_headlines_cache", …)` at :822, which holds `NewsEngine._cache_lock`, :629-647):
1. RSS items become `NewsSentiment` objects (:891-910) with the **keyword scorer** filling
`sentiment_score` and `sentiment_by_currency[cur]` for each currency in `currencies_mentioned`,
`score_source="keyword"`, `xau_score=None`.
2. `scored = headlines + unmatched` (:944); `self._apply_llm_sentiment(scored)` (:945, in try/except).
`_apply_llm_sentiment` (:1032-1065) sends headlines not in `NewsEngine._llm_sent_cache` to DeepSeek
in batches of `_LLM_BATCH_CAP = 25` via `self._llm_headline_scores(texts)` (:1009) and, on success,
overwrites `sentiment_score`, `sentiment_by_currency` (XAU gets the gold score, every other currency
the overall score), sets `xau_score`, `score_source="llm"`. A failed batch stops sending; those
headlines stay keyword-scored and **stay out of the cache**, so they are re-sent next window.
3. `news_archive.archive_headlines(scored, teacher_model=…, teacher_prompt_sha1=…)` (:958-964).
The archive UPSERT (news_archive.py:203-260) keeps the higher-ranked source:
`_SOURCE_RANK = {"unknown": 0, "keyword": 1, "model": 2, "llm": 3}` (:107). `"model"` is already
reserved for "a trained local artifact" (news_engine.py:223-227). `build_row` (:323-354) stores
`teacher_model`/`teacher_prompt_sha1` only when `source == "llm"`.
4. Downstream consumers of the scores: `get_fundamental_context` (drops |score| <
`NEUTRAL_HEADLINE_THRESHOLD = 0.1`, :256), `get_pair_sentiment` (:553-557, computes
`XAU_mean − USD_mean`), `signal_aggregator.py:1677/1723` (acts on that when
`sentiment_affects_signals`), `agent_panel.py`, the dashboard.
Two processes run this code: the bot (`venv/bin/python telegram_bot.py`, CLAUDE.md:30) and the
dashboard daemon (`dashboard_publisher.py:401-403` builds its own `NewsEngine`, `:503` calls
`get_fundamental_context()`; unit `deploy/gold-dashboard.service`). They share `config.json` and the
archive DB.
### 1.3 Decisions taken by the owner (2026-09-27)
- **Mode `primary`, no DeepSeek for headline scoring.** The model scores every headline. DeepSeek stays
for everything else it does (`_llm_event_direction` for calendar surprises, the news summary, the
dashboard's on-demand `news_summary.php`).
- **Model files live outside the ForexBot2 tree**, path in `config.json`.
- Accepted consequence: the archive stops receiving fresh DeepSeek labels, so the ML-NEWS weekly
retrain and the "replay and compare" check lose their data source. Mitigation kept optional:
`deepseek_audit_per_day` (§4) labels a few headlines a day for the archive only. Default 0.
### 1.4 Non-goals
- Removing the DeepSeek client or its non-headline uses.
- Scoring non-gold currencies with the model.
- Changing ML-NEWS training. (The one ML-NEWS change is §3.4, a runtime option.)
- Training from price moves.
---
## 2. Target layout on the live server
```
/home/Forex/ForexBot2/ # git pull as today
news_model.py # NEW, vendored runtime (§3)
requirements.txt # + onnxruntime, tokenizers (§3.1)
app_config.py, news_engine.py, news_archive.py, test_all.py # edited (§4, §5, §6, §7)
config.json # + "news_model" section (§4)
deploy/README-news-model.md # NEW ops checklist (§8)
deploy/gold-dashboard.service # maybe + Environment=NEWS_MODEL_MODE=off (§8)
/home/Forex/models/xau_transformer/ # NEW, outside git, copied by hand
model.onnx tokenizer.json meta.json
```
Memory: each process that loads the model holds ~0.5–0.7 GB RSS for the ONNX session. Two processes
→ ~1.5 GB. `free -m` before deciding whether the dashboard daemon also runs it (§8 step 0).
---
## 3. `news_model.py` (new, ForexBot2)
### 3.1 Dependencies
Add to `requirements.txt`:
```
onnxruntime>=1.30,<2
tokenizers>=0.23,<1
```
Pin to the versions that verified the bundle in ML-NEWS `.venv` (check `pip show` there before
writing). `tokenizer.json` compatibility across older `tokenizers` releases is not guaranteed.
**Do not write the words "torch" or "transformers" anywhere in requirements.txt**, not even in a
comment: `test_all.py:11437-11462` greps the file for them and fails.
### 3.2 Source
Copy `ML-NEWS/news_transformer.py` → `ForexBot2/news_model.py`. Keep verbatim: `clean_headline`,
`_sigmoid`, `load_model`, `score_batch`, `score_news`, constants `MAX_LENGTH = 64`,
`RELEVANT_MIN = 0.25`, `RELEVANT_MAX_IRRELEVANT = 0.10`. Drop the CLI `main` or keep it, either is fine.
Add a docstring line: `Vendored from ML-NEWS news_transformer.py at commit <sha>; bundle trained_at
<meta.json trained_at>.`
**Imports.** `onnxruntime` and `tokenizers` must stay imported *inside* `load_model` (as in
news_transformer.py:58-59). Module top imports only `numpy`, `json`, `logging`, `os`, `pathlib`,
`threading`. Reason: `news_engine.py` will `import news_model` at its top; if onnxruntime is missing on
a box, the bot must still start.
### 3.3 Singleton
```python
_lock = threading.Lock()
_bundle: Optional[dict] = None
_failed: bool = False # tried and failed; don't retry, don't re-log
def get_news_model() -> Optional[dict]:
"""The loaded bundle, or None. Loads once per process. Never raises."""
global _bundle, _failed
if _bundle is not None or _failed:
return _bundle
with _lock:
if _bundle is not None or _failed:
return _bundle
try:
import app_config
path = app_config.news_model_path()
threads = app_config.news_model_threads()
b = load_model(path, threads=threads)
logger.info("news model loaded from %s (trained_at %s, threshold %s, %d threads)",
path, b["meta"].get("trained_at"), b["meta"].get("relevance_threshold"), threads)
_bundle = b
except Exception as e: # ImportError, FileNotFoundError, bad JSON, ORT errors
logger.warning("news model unavailable, DeepSeek/keyword path stays: %s: %s",
type(e).__name__, e)
_failed = True
return _bundle
def reset_for_tests() -> None: # clears _bundle/_failed; used by test_all.py
```
Optional: `warm_up_in_background()` that calls `get_news_model()` in a daemon thread at
`NewsEngine.__init__`, so the first news cycle isn't blocked by the ~3–5 s session creation under
`_cache_lock`. Nice to have, not required.
### 3.4 Two changes inside `load_model` (also apply to ML-NEWS `news_transformer.py` so the copies stay identical)
```python
options = ort.SessionOptions()
if threads:
options.intra_op_num_threads = max(1, min(int(threads), os.cpu_count() or 1))
options.inter_op_num_threads = 1
# ORT intra-op threads spin-wait after each run by default; in a mostly idle
# bot process that is 4 cores at 100% between cycles.
options.add_session_config_entry("session.intra_op.allow_spinning", "0")
```
### 3.5 Contract `score_batch(bundle, headlines: list[str]) -> list[dict]`
Unchanged from ML-NEWS. Empty or whitespace-only headline raises `ValueError` from `clean_headline`;
callers pass already-non-empty `NewsSentiment.headline`, but §5 still wraps per-batch in try/except.
Batch size: `_apply_model_sentiment` sends all of a cycle's headlines in one call (typically 30–150);
`score_batch` pads to the longest in the batch, ≤ 64 tokens, so one call of 150 is ~1 s on 4 threads.
---
## 4. Configuration
### 4.1 `config.json` section (add to `config.json.example` too)
```json
"news_model": {
"path": "/home/Forex/models/xau_transformer",
"threads": 4,
"mode": "primary",
"deepseek_audit_per_day": 0
}
```
| Key | Type | Default if absent | Meaning |
|---|---|---|---|
| `path` | str | required when mode ≠ off | directory with the three bundle files |
| `threads` | int ≥ 1 | 4 | ORT intra-op threads (measured: 1→19 ms, 4→9 ms, 16→13 ms) |
| `mode` | `"off"`/`"fallback"`/`"primary"` | `"off"` | see §5.2 |
| `deepseek_audit_per_day` | int ≥ 0 | 0 | primary mode only: headlines/day to also send to DeepSeek, archive-only |
Whole section absent → `mode="off"` → today's behaviour exactly.
Environment override: `NEWS_MODEL_MODE=off|fallback|primary` wins over config.json. Purpose: the
dashboard daemon's systemd unit can turn the model off for that process only. Pattern to copy:
`NEWS_ARCHIVE_ENABLED` in `app_config.py:574-583`.
### 4.2 `app_config.py` edits (follow the `news_archive` section, :397-406, accessors :574/:588)
1. `_KNOWN_TOP` (:86-92): add `"news_model"`. **Required**; `_check_keys(raw, _KNOWN_TOP, "top level")`
at :181 makes boot fail on an unknown top-level key.
2. `_KNOWN_NEWS_MODEL = {"path", "threads", "mode", "deepseek_audit_per_day"}` next to
`_KNOWN_NEWS_ARCHIVE` (:113).
3. `AppConfig` dataclass (~:147): `news_model_settings: Dict[str, Any] = field(default_factory=dict)`.
4. In `load_app_config`, after the `news_archive` block:
```python
nm = raw.get("news_model", {})
_check_keys(nm, _KNOWN_NEWS_MODEL, '"news_model"')
if "mode" in nm and nm["mode"] not in ("off", "fallback", "primary"):
raise AppConfigError('config.json: news_model.mode must be "off", "fallback" or "primary"')
if "threads" in nm and (not isinstance(nm["threads"], int) or isinstance(nm["threads"], bool) or nm["threads"] < 1):
raise AppConfigError("config.json: news_model.threads must be a positive integer")
if "deepseek_audit_per_day" in nm and (not isinstance(nm["deepseek_audit_per_day"], int) or nm["deepseek_audit_per_day"] < 0):
raise AppConfigError("config.json: news_model.deepseek_audit_per_day must be a non-negative integer")
if nm.get("mode", "off") != "off" and not isinstance(nm.get("path"), str):
raise AppConfigError("config.json: news_model.path is required when mode is not off")
app.news_model_settings = dict(nm)
```
5. Accessors:
```python
def news_model_mode() -> str: # env NEWS_MODEL_MODE overrides; default "off"
def news_model_path() -> Optional[str]
def news_model_threads() -> int # default 4
def news_model_audit_per_day() -> int # default 0
```
`NewsConfig` in `config.py:169` is **not** touched: tests construct it bare as `NewsConfig()`, and it
holds feed/threshold settings, not paths. `news_engine.py` reads the new settings through `app_config`
the same way it reaches DeepSeek through `get_llm_client()`.
---
## 5. `news_engine.py` changes
### 5.1 New method `_apply_model_sentiment`
Place next to `_apply_llm_sentiment` (:1032). Runs under `_cache_lock` like it.
```python
def _apply_model_sentiment(self, headlines: List[NewsSentiment]) -> int:
"""Score with the local XAU transformer every headline that is still
keyword-scored. Returns the number scored. Never raises.
The model has a gold head only. It writes xau_score always, XAU's
per-currency score when XAU is mentioned, and the overall score only
when the headline names no other currency. USD/EUR/... entries keep
their keyword score: get_pair_sentiment subtracts USD from XAU and
must not see the same number on both sides.
"""
bundle = news_model.get_news_model()
if bundle is None or not headlines:
return 0
todo = [h for h in headlines if h.score_source == "keyword"]
if not todo:
return 0
try:
results = news_model.score_batch(bundle, [h.headline for h in todo])
except Exception as e:
logger.debug(f"news model scoring failed, keyword scores stand: {e}")
return 0
for h, r in zip(todo, results):
raw = float(r["score"]) # archive value, unmodified
shown = raw if r["relevant"] else 0.0 # display value, see rule D
h.xau_score = raw
if "XAU" in h.currencies_mentioned:
h.sentiment_by_currency["XAU"] = shown
if set(h.currencies_mentioned) <= {"XAU"}: # [] or ["XAU"]
h.sentiment_score = shown
h.score_source = "model"
return len(todo)
```
Rules, with the reason each exists:
- **A. Never write the model score into a non-XAU currency.** `get_pair_sentiment` (:553-557) forms
`XAU_mean − USD_mean`; `signal_aggregator.py:1677/1723` acts on it. The DeepSeek path puts its
*overall* score into USD; the model has no overall score, so USD keeps the keyword number.
- **B. `sentiment_score` from the model only when the headline names no other currency.** The archive's
`score` column is `sentiment_score`, and the ML-NEWS replay eval reads it; for a gold-only headline it
should be the model's number, for "Fed hikes, dollar jumps" it stays keyword so the row is honest
about what the model did and didn't score. `score_source="model"` therefore means "XAU score is the
model's" — amend the dataclass comment at :223-227 to say so.
- **C. `xau_score` is always the raw signed score.** That column exists for training data
(:229-237); keep it unfiltered.
- **D. Display fields get 0.0 when `relevant` is False.** A headline the model calls not-relevant can
still carry |score| up to ~0.2 from a confident direction head; `NEUTRAL_HEADLINE_THRESHOLD = 0.1`
(:256) would let 0.15 through as "leaning bearish". Zeroing the shown value makes "neutral" mean
neutral. (0.1 happens to equal the model's irrelevant band `RELEVANT_MAX_IRRELEVANT`, so the two
notions agree.)
- **E. Do not touch `NewsEngine._llm_sent_cache`.** That cache decides what is re-sent to DeepSeek. In
`fallback` mode a headline that failed DeepSeek must stay uncached so it is retried next window and
its archive row upgrades `model` → `llm` through the rank rule (news_archive.py:203-260).
- **F. Only `score_source == "keyword"` rows are touched.** Never overwrite an `llm` row in memory.
### 5.2 Call site (:944-947)
Replace
```python
scored = headlines + unmatched
try:
self._apply_llm_sentiment(scored)
except Exception as e:
logger.debug(f"LLM sentiment failed, keyword scores stand: {e}")
```
with
```python
scored = headlines + unmatched
mode = app_config.news_model_mode()
try:
if mode == "primary":
n = self._apply_model_sentiment(scored)
if n == 0 and news_model.get_news_model() is None:
# model unavailable this process: degrade to DeepSeek, not to keywords
self._apply_llm_sentiment(scored)
else:
self._audit_with_deepseek(scored) # no-op when audit_per_day == 0
elif mode == "fallback":
self._apply_llm_sentiment(scored)
self._apply_model_sentiment(scored) # only what DeepSeek left keyword-scored
else:
self._apply_llm_sentiment(scored)
except Exception as e:
logger.debug(f"headline sentiment enrichment failed, keyword scores stand: {e}")
```
`import news_model` and `import app_config` at the module top of news_engine.py (news_model's top
imports only numpy, §3.2).
Mode semantics:
| mode | DeepSeek for headlines | model | when a step fails |
|---|---|---|---|
| `off` | as today | never | keyword scores stand (today) |
| `fallback` | first, as today | on rows DeepSeek left keyword-scored | keyword scores stand |
| `primary` | never (except audit) | on all rows | model missing → DeepSeek path for the cycle; model raises mid-batch → keyword |
### 5.3 Audit sample (primary mode, optional feature, default off)
```python
_audit_day: str = "" # class attrs, like the caches
_audit_sent_today: int = 0
def _audit_with_deepseek(self, headlines):
cap = app_config.news_model_audit_per_day()
if cap <= 0 or not get_llm_client().available:
return
today = datetime.now(timezone.utc).date().isoformat()
if NewsEngine._audit_day != today:
NewsEngine._audit_day, NewsEngine._audit_sent_today = today, 0
room = cap - NewsEngine._audit_sent_today
if room <= 0:
return
cand = [h for h in headlines if h.score_source == "model"
and hashlib.sha1(h.headline.encode()).hexdigest() not in NewsEngine._llm_sent_cache]
batch = cand[:min(room, self._LLM_BATCH_CAP)]
if not batch:
return
scores = self._llm_headline_scores([h.headline for h in batch])
if scores is None:
return
NewsEngine._audit_sent_today += len(batch)
copies = []
for h, (overall, xau) in zip(batch, scores):
NewsEngine._llm_sent_cache[hashlib.sha1(h.headline.encode()).hexdigest()] = (overall, xau)
c = copy.copy(h) # the displayed object keeps the model score
c.sentiment_score = overall
c.sentiment_by_currency = {cur: (xau if cur == "XAU" else overall) for cur in h.currencies_mentioned}
c.xau_score = xau
c.score_source = "llm"
copies.append(c)
self._pending_audit_rows = copies # archived by the call site right after `scored`
```
The call site then archives `scored` first and `self._pending_audit_rows` second (same
`teacher_model`/`teacher_prompt_sha1` as today's DeepSeek rows); the rank rule upgrades those rows
to `llm`, the in-memory objects the bot displays keep the model's numbers. One batch per cycle at
most, ≤ `cap` per UTC day. This is the only place DeepSeek is called for headlines in primary mode.
### 5.4 Archive call (:958-964)
Pass, in addition to today's kwargs:
```python
model_teacher=news_model.teacher_tag() # "xau_transformer@<meta.trained_at>" or None
```
where `teacher_tag()` reads the loaded bundle's meta (None if not loaded). See §6.
### 5.5 Dataclass comment
`NewsSentiment.score_source` comment (:223-227): add `"model" = the local XAU transformer scored
xau_score (and XAU's per-currency score); other currencies remain keyword-scored.`
---
## 6. `news_archive.py` changes
No schema change. `_SOURCE_RANK` (:107) and `_STORED_RANK` (:203-206) already rank `model` = 2.
- `archive_headlines(headlines, *, teacher_model=None, teacher_prompt_sha1=None, origin="live",
model_teacher=None)` — new kwarg, passed through to `build_row`.
- `build_row` (:323-354):
```python
"teacher_model": (teacher_model if source == "llm"
else model_teacher if source == "model" else None),
"teacher_prompt_sha1": (teacher_prompt_sha1 if source in ("llm", "model") else None),
```
For a model row `teacher_prompt_sha1` is the DeepSeek prompt the model was distilled from (the same
constant the bot already passes), which is the truthful provenance.
- Effect: a retrain is visible in the data (`teacher_model` changes), and the UPSERT's
`COALESCE(excluded.teacher_model, headlines.teacher_model)` (:253-260) only applies when the new
rank ≥ stored rank, so an `llm` row's teacher is never replaced by a model tag.
---
## 7. Tests (`test_all.py`)
The suite is a plain script: `@test(name)` (:51-64) runs each function at definition time and
tallies; the file ends with `sys.exit(1 if FAIL > 0 else 0)`. Run: `venv/bin/python test_all.py`.
Stubbing patterns in use: instance-level `ne._llm_headline_scores = lambda texts: …` with
`lc._client = _StubLLM({...})` and `finally` cleanup (:5196-5215); module-level swap
`saved = _ne_mod.get_llm_client; _ne_mod.get_llm_client = stub; … finally restore` (:11391-11424).
Add, using the same patterns and a fake bundle (`news_model.score_batch` swapped on the module):
1. **model scores keyword rows only.** Two `NewsSentiment`s, one `score_source="llm"`; after
`_apply_model_sentiment`, the llm one is byte-identical, the keyword one has `score_source="model"`,
`xau_score` set, `sentiment_by_currency["XAU"]` = score, `["USD"]` unchanged (rule A).
2. **overall score rule (B).** `currencies_mentioned=["XAU"]` → `sentiment_score` = model score;
`["USD","XAU"]` → `sentiment_score` unchanged; `[]` → model score.
3. **relevance zeroing (D).** stub returns `relevant=False, score=-0.18` → `sentiment_by_currency["XAU"]
== 0.0`, `xau_score == -0.18`.
4. **cache untouched (E).** `_llm_sent_cache` empty before and after.
5. **failure isolation.** stub raises → all rows still `keyword`, method returns 0, no exception.
`news_model.get_news_model()` returning None → returns 0.
6. **primary degrades to DeepSeek.** `get_news_model` → None, `_llm_headline_scores` stubbed → rows
end up `llm`.
7. **audit sample.** `deepseek_audit_per_day=3`, 10 model rows → exactly 3 sent, displayed objects
still `model`, `_pending_audit_rows` has 3 `llm` copies; second call same day sends 0.
8. **config.** unknown key in `news_model` → `AppConfigError`; `mode="banana"` → error; section
absent → `news_model_mode() == "off"`; `NEWS_MODEL_MODE=off` env beats `"mode": "primary"`.
9. **archive.** `build_row` of a `model` headline with `model_teacher="xau_transformer@T"` stores it;
an `llm` row stores the DeepSeek teacher; existing rank tests still pass.
10. **import hygiene.** update the docstring of the test at :11437-11462 (it says numpy-only); add
an assertion that `requirements.txt` lists `onnxruntime` and `tokenizers`; keep its "no torch /
transformers" checks.
11. **integration, skipped when the bundle is absent** (`os.environ.get("NEWS_MODEL_DIR")` or the
configured path missing → print SKIP): load, `score_news(b, "Fed cuts rates, signals more
easing")["label"] == "bullish"`, `score_news(b, "Red Lobster endless shrimp losses")["label"] ==
"neutral"`, latency of 20 calls < 1 s.
`test_all.py:3675-3690` loads `config.json.example` for pair output only; adding the section there is
safe.
---
## 8. Ops checklist → `deploy/README-news-model.md` (new)
Covers **both** processes: the bot (`telegram_bot.py`) and the dashboard daemon
(`deploy/gold-dashboard.service`).
0. `free -m`. Each model-loading process needs ~0.7 GB headroom. If only the bot should score with
the model, add `Environment=NEWS_MODEL_MODE=off` to `deploy/gold-dashboard.service` (the dashboard
then keeps using DeepSeek for its headlines; those `llm` rows land in the shared archive and act
as free audit labels).
1. `cd /home/Forex/ForexBot2 && git pull && venv/bin/pip install -r requirements.txt`.
2. `mkdir -p /home/Forex/models/xau_transformer` and copy `model.onnx`, `tokenizer.json`, `meta.json`
from the ML-NEWS machine (rsync, ~280 MB). Verify `sha256sum model.onnx` matches the source.
Owner: the account the bot runs as; read-only is enough.
3. Add the `news_model` section to `config.json` (§4.1) with `"mode": "primary"`. Leave the
`deepseek` section unchanged.
4. Smoke test without starting the bot:
```
venv/bin/python -c "import news_model as m; b=m.get_news_model(); print(m.score_news(b,'Gold jumps as dollar slides'))"
```
Expect `label: bullish`, and the "news model loaded" log line.
5. `venv/bin/python test_all.py` — green (integration test runs because the dir exists).
6. Restart the bot; restart the dashboard unit. Check:
- log: one "news model loaded" line per process, no "news model unavailable";
- `top`: RSS ~+0.6 GB per process, idle CPU ~0% between scans (spinning disabled);
- archive: new rows have `score_source='model'`, `xau_score` set, `teacher_model`
`xau_transformer@…`; DeepSeek daily counter for headlines stays at 0 (audit off).
7. Rollback: set `"mode": "fallback"` or `"off"` (or the env var), restart. No data migration needed.
8. Retrain cadence: when ML-NEWS ships a new bundle, copy the three files over the old ones and
restart both processes. `meta.json.trained_at` changes, so the archive shows which model scored which
rows. If `deepseek_audit_per_day` is set, ML-NEWS's `eval/eval_archive.py --scorer transformer` on a
copy of the live archive gives the model-vs-DeepSeek agreement on those rows.
---
## 9. Acceptance criteria (end-to-end)
- With `mode="primary"` and DeepSeek enabled, one live news cycle archives only `model` rows (plus
≤ `deepseek_audit_per_day` `llm` rows), and the cycle's added latency is < 1 s for ~100 headlines.
- With the model directory renamed away, both processes start, log one warning each, and the cycle's
rows are `llm` (DeepSeek path), not `keyword`.
- With `onnxruntime` uninstalled in a scratch venv, `python -c "import news_engine"` succeeds.
- A headline naming USD and XAU ends with `sentiment_by_currency["USD"]` equal to the keyword value
and `["XAU"]` equal to the model value.
- `mode="off"` reproduces today's archive rows exactly (`llm`/`keyword`, same teacher fields).
- `venv/bin/python test_all.py` exits 0.
## 10. Known limits to state in the README
- The model copies DeepSeek's opinion (72% agreement on unseen live days, 72/78 golden); it does not
learn from price. DeepSeek itself is ~90% self-consistent.
- Gold only. Every other currency's sentiment remains the ~31%-accurate keyword scorer in primary
mode, which is what those currencies get today whenever DeepSeek is unavailable.
- With audit off, no new DeepSeek labels accrue; the ML-NEWS retrain loop needs either
`deepseek_audit_per_day > 0`, the dashboard left on DeepSeek, or a separate labelling run.