# Specification: run the ML-NEWS XAU transformer inside ForexBot2

Status: **specification only, not implemented.** Written 2026-09-27 so a coding agent can implement it
later without this conversation. Every file:line reference was checked against the repos on that date;
re-verify before editing, the numbers drift.

Repos:
- **ML-NEWS** `~/Development/ML-NEWS` (git `ssh://git@192.168.0.11:222/harry/News-ML-Decoder.git`).
  Produces the model. Nothing in ForexBot2 imports from it.
- **ForexBot2** `~/Development/ForexBot2`, live at `/home/Forex/ForexBot2`, deployed by `git pull`.
  Consumes the model.

---

## 1. Context and goal

### 1.1 What the model is
`ML-NEWS/models/xau_transformer/` holds a fine-tuned `microsoft/deberta-v3-small` with two heads,
exported to ONNX:

| File | Size | Purpose |
|---|---|---|
| `model.onnx` | 271 MB | encoder + heads; int8 embedding table, fp32 compute |
| `tokenizer.json` | 8 MB | HF `tokenizers` fast tokenizer |
| `meta.json` | 4 KB | thresholds, `trained_at`, `relevance_threshold`, `export_checks`, training stats |

Inputs: one headline, ≤ 64 tokens. Outputs per headline (from `news_transformer.score_batch`):

```python
{"headline": str, "p_relevant": float, "p_bullish": float,
 "relevant": bool,            # p_relevant >= meta["relevance_threshold"]  (0.2 for the shipped bundle)
 "direction": "bullish"|"bearish",
 "label": "bullish"|"bearish"|"neutral",   # neutral when not relevant
 "score": float}              # p_relevant * (2*p_bullish - 1), in [-1, 1]
```

`score` is on the same scale as DeepSeek's `xau` score that ForexBot2 already stores in
`xau_score`. The model was trained to copy DeepSeek (`deepseek-v4-flash`, prompt sha1 `d32e04c6…`, the
same prompt as `news_engine._LLM_SENTIMENT_PROMPT_SHA1`) on 17,245 headlines. Quality on
2026-09-27: golden set 72/78 (DeepSeek 78, keyword scorer 24), agreement with DeepSeek on unseen live
days ~72%, CPU latency 19 ms/headline on 1 thread, 9 ms on 4. Full numbers: `ML-NEWS/eval/RESULTS_2026-09-27.md`.

**The model scores gold only.** It has no head for USD, EUR or anything else. This drives most of
the rules in §5.

### 1.2 How ForexBot2 scores headlines today
`news_engine.py`, inside `NewsEngine._fetch_news_sentiment_raw` (called via
`_cached_fetch("_headlines_cache", …)` at :822, which holds `NewsEngine._cache_lock`, :629-647):

1. RSS items become `NewsSentiment` objects (:891-910) with the **keyword scorer** filling
   `sentiment_score` and `sentiment_by_currency[cur]` for each currency in `currencies_mentioned`,
   `score_source="keyword"`, `xau_score=None`.
2. `scored = headlines + unmatched` (:944); `self._apply_llm_sentiment(scored)` (:945, in try/except).
   `_apply_llm_sentiment` (:1032-1065) sends headlines not in `NewsEngine._llm_sent_cache` to DeepSeek
   in batches of `_LLM_BATCH_CAP = 25` via `self._llm_headline_scores(texts)` (:1009) and, on success,
   overwrites `sentiment_score`, `sentiment_by_currency` (XAU gets the gold score, every other currency
   the overall score), sets `xau_score`, `score_source="llm"`. A failed batch stops sending; those
   headlines stay keyword-scored and **stay out of the cache**, so they are re-sent next window.
3. `news_archive.archive_headlines(scored, teacher_model=…, teacher_prompt_sha1=…)` (:958-964).
   The archive UPSERT (news_archive.py:203-260) keeps the higher-ranked source:
   `_SOURCE_RANK = {"unknown": 0, "keyword": 1, "model": 2, "llm": 3}` (:107). `"model"` is already
   reserved for "a trained local artifact" (news_engine.py:223-227). `build_row` (:323-354) stores
   `teacher_model`/`teacher_prompt_sha1` only when `source == "llm"`.
4. Downstream consumers of the scores: `get_fundamental_context` (drops |score| <
   `NEUTRAL_HEADLINE_THRESHOLD = 0.1`, :256), `get_pair_sentiment` (:553-557, computes
   `XAU_mean − USD_mean`), `signal_aggregator.py:1677/1723` (acts on that when
   `sentiment_affects_signals`), `agent_panel.py`, the dashboard.

Two processes run this code: the bot (`venv/bin/python telegram_bot.py`, CLAUDE.md:30) and the
dashboard daemon (`dashboard_publisher.py:401-403` builds its own `NewsEngine`, `:503` calls
`get_fundamental_context()`; unit `deploy/gold-dashboard.service`). They share `config.json` and the
archive DB.

### 1.3 Decisions taken by the owner (2026-09-27)
- **Mode `primary`, no DeepSeek for headline scoring.** The model scores every headline. DeepSeek stays
  for everything else it does (`_llm_event_direction` for calendar surprises, the news summary, the
  dashboard's on-demand `news_summary.php`).
- **Model files live outside the ForexBot2 tree**, path in `config.json`.
- Accepted consequence: the archive stops receiving fresh DeepSeek labels, so the ML-NEWS weekly
  retrain and the "replay and compare" check lose their data source. Mitigation kept optional:
  `deepseek_audit_per_day` (§4) labels a few headlines a day for the archive only. Default 0.

### 1.4 Non-goals
- Removing the DeepSeek client or its non-headline uses.
- Scoring non-gold currencies with the model.
- Changing ML-NEWS training. (The one ML-NEWS change is §3.4, a runtime option.)
- Training from price moves.

---

## 2. Target layout on the live server

```
/home/Forex/ForexBot2/                       # git pull as today
    news_model.py                            # NEW, vendored runtime (§3)
    requirements.txt                         # + onnxruntime, tokenizers (§3.1)
    app_config.py, news_engine.py, news_archive.py, test_all.py   # edited (§4, §5, §6, §7)
    config.json                              # + "news_model" section (§4)
    deploy/README-news-model.md              # NEW ops checklist (§8)
    deploy/gold-dashboard.service            # maybe + Environment=NEWS_MODEL_MODE=off (§8)
/home/Forex/models/xau_transformer/          # NEW, outside git, copied by hand
    model.onnx  tokenizer.json  meta.json
```

Memory: each process that loads the model holds ~0.5–0.7 GB RSS for the ONNX session. Two processes
→ ~1.5 GB. `free -m` before deciding whether the dashboard daemon also runs it (§8 step 0).

---

## 3. `news_model.py` (new, ForexBot2)

### 3.1 Dependencies
Add to `requirements.txt`:
```
onnxruntime>=1.30,<2
tokenizers>=0.23,<1
```
Pin to the versions that verified the bundle in ML-NEWS `.venv` (check `pip show` there before
writing). `tokenizer.json` compatibility across older `tokenizers` releases is not guaranteed.
**Do not write the words "torch" or "transformers" anywhere in requirements.txt**, not even in a
comment: `test_all.py:11437-11462` greps the file for them and fails.

### 3.2 Source
Copy `ML-NEWS/news_transformer.py` → `ForexBot2/news_model.py`. Keep verbatim: `clean_headline`,
`_sigmoid`, `load_model`, `score_batch`, `score_news`, constants `MAX_LENGTH = 64`,
`RELEVANT_MIN = 0.25`, `RELEVANT_MAX_IRRELEVANT = 0.10`. Drop the CLI `main` or keep it, either is fine.
Add a docstring line: `Vendored from ML-NEWS news_transformer.py at commit <sha>; bundle trained_at
<meta.json trained_at>.`

**Imports.** `onnxruntime` and `tokenizers` must stay imported *inside* `load_model` (as in
news_transformer.py:58-59). Module top imports only `numpy`, `json`, `logging`, `os`, `pathlib`,
`threading`. Reason: `news_engine.py` will `import news_model` at its top; if onnxruntime is missing on
a box, the bot must still start.

### 3.3 Singleton
```python
_lock = threading.Lock()
_bundle: Optional[dict] = None
_failed: bool = False          # tried and failed; don't retry, don't re-log

def get_news_model() -> Optional[dict]:
    """The loaded bundle, or None. Loads once per process. Never raises."""
    global _bundle, _failed
    if _bundle is not None or _failed:
        return _bundle
    with _lock:
        if _bundle is not None or _failed:
            return _bundle
        try:
            import app_config
            path = app_config.news_model_path()
            threads = app_config.news_model_threads()
            b = load_model(path, threads=threads)
            logger.info("news model loaded from %s (trained_at %s, threshold %s, %d threads)",
                        path, b["meta"].get("trained_at"), b["meta"].get("relevance_threshold"), threads)
            _bundle = b
        except Exception as e:                     # ImportError, FileNotFoundError, bad JSON, ORT errors
            logger.warning("news model unavailable, DeepSeek/keyword path stays: %s: %s",
                           type(e).__name__, e)
            _failed = True
        return _bundle

def reset_for_tests() -> None:  # clears _bundle/_failed; used by test_all.py
```

Optional: `warm_up_in_background()` that calls `get_news_model()` in a daemon thread at
`NewsEngine.__init__`, so the first news cycle isn't blocked by the ~3–5 s session creation under
`_cache_lock`. Nice to have, not required.

### 3.4 Two changes inside `load_model` (also apply to ML-NEWS `news_transformer.py` so the copies stay identical)
```python
options = ort.SessionOptions()
if threads:
    options.intra_op_num_threads = max(1, min(int(threads), os.cpu_count() or 1))
    options.inter_op_num_threads = 1
# ORT intra-op threads spin-wait after each run by default; in a mostly idle
# bot process that is 4 cores at 100% between cycles.
options.add_session_config_entry("session.intra_op.allow_spinning", "0")
```

### 3.5 Contract `score_batch(bundle, headlines: list[str]) -> list[dict]`
Unchanged from ML-NEWS. Empty or whitespace-only headline raises `ValueError` from `clean_headline`;
callers pass already-non-empty `NewsSentiment.headline`, but §5 still wraps per-batch in try/except.
Batch size: `_apply_model_sentiment` sends all of a cycle's headlines in one call (typically 30–150);
`score_batch` pads to the longest in the batch, ≤ 64 tokens, so one call of 150 is ~1 s on 4 threads.

---

## 4. Configuration

### 4.1 `config.json` section (add to `config.json.example` too)
```json
"news_model": {
  "path": "/home/Forex/models/xau_transformer",
  "threads": 4,
  "mode": "primary",
  "deepseek_audit_per_day": 0
}
```

| Key | Type | Default if absent | Meaning |
|---|---|---|---|
| `path` | str | required when mode ≠ off | directory with the three bundle files |
| `threads` | int ≥ 1 | 4 | ORT intra-op threads (measured: 1→19 ms, 4→9 ms, 16→13 ms) |
| `mode` | `"off"`/`"fallback"`/`"primary"` | `"off"` | see §5.2 |
| `deepseek_audit_per_day` | int ≥ 0 | 0 | primary mode only: headlines/day to also send to DeepSeek, archive-only |

Whole section absent → `mode="off"` → today's behaviour exactly.

Environment override: `NEWS_MODEL_MODE=off|fallback|primary` wins over config.json. Purpose: the
dashboard daemon's systemd unit can turn the model off for that process only. Pattern to copy:
`NEWS_ARCHIVE_ENABLED` in `app_config.py:574-583`.

### 4.2 `app_config.py` edits (follow the `news_archive` section, :397-406, accessors :574/:588)
1. `_KNOWN_TOP` (:86-92): add `"news_model"`. **Required**; `_check_keys(raw, _KNOWN_TOP, "top level")`
   at :181 makes boot fail on an unknown top-level key.
2. `_KNOWN_NEWS_MODEL = {"path", "threads", "mode", "deepseek_audit_per_day"}` next to
   `_KNOWN_NEWS_ARCHIVE` (:113).
3. `AppConfig` dataclass (~:147): `news_model_settings: Dict[str, Any] = field(default_factory=dict)`.
4. In `load_app_config`, after the `news_archive` block:
   ```python
   nm = raw.get("news_model", {})
   _check_keys(nm, _KNOWN_NEWS_MODEL, '"news_model"')
   if "mode" in nm and nm["mode"] not in ("off", "fallback", "primary"):
       raise AppConfigError('config.json: news_model.mode must be "off", "fallback" or "primary"')
   if "threads" in nm and (not isinstance(nm["threads"], int) or isinstance(nm["threads"], bool) or nm["threads"] < 1):
       raise AppConfigError("config.json: news_model.threads must be a positive integer")
   if "deepseek_audit_per_day" in nm and (not isinstance(nm["deepseek_audit_per_day"], int) or nm["deepseek_audit_per_day"] < 0):
       raise AppConfigError("config.json: news_model.deepseek_audit_per_day must be a non-negative integer")
   if nm.get("mode", "off") != "off" and not isinstance(nm.get("path"), str):
       raise AppConfigError("config.json: news_model.path is required when mode is not off")
   app.news_model_settings = dict(nm)
   ```
5. Accessors:
   ```python
   def news_model_mode() -> str:          # env NEWS_MODEL_MODE overrides; default "off"
   def news_model_path() -> Optional[str]
   def news_model_threads() -> int        # default 4
   def news_model_audit_per_day() -> int  # default 0
   ```
`NewsConfig` in `config.py:169` is **not** touched: tests construct it bare as `NewsConfig()`, and it
holds feed/threshold settings, not paths. `news_engine.py` reads the new settings through `app_config`
the same way it reaches DeepSeek through `get_llm_client()`.

---

## 5. `news_engine.py` changes

### 5.1 New method `_apply_model_sentiment`
Place next to `_apply_llm_sentiment` (:1032). Runs under `_cache_lock` like it.

```python
def _apply_model_sentiment(self, headlines: List[NewsSentiment]) -> int:
    """Score with the local XAU transformer every headline that is still
    keyword-scored. Returns the number scored. Never raises.

    The model has a gold head only. It writes xau_score always, XAU's
    per-currency score when XAU is mentioned, and the overall score only
    when the headline names no other currency. USD/EUR/... entries keep
    their keyword score: get_pair_sentiment subtracts USD from XAU and
    must not see the same number on both sides.
    """
    bundle = news_model.get_news_model()
    if bundle is None or not headlines:
        return 0
    todo = [h for h in headlines if h.score_source == "keyword"]
    if not todo:
        return 0
    try:
        results = news_model.score_batch(bundle, [h.headline for h in todo])
    except Exception as e:
        logger.debug(f"news model scoring failed, keyword scores stand: {e}")
        return 0
    for h, r in zip(todo, results):
        raw = float(r["score"])                       # archive value, unmodified
        shown = raw if r["relevant"] else 0.0         # display value, see rule D
        h.xau_score = raw
        if "XAU" in h.currencies_mentioned:
            h.sentiment_by_currency["XAU"] = shown
        if set(h.currencies_mentioned) <= {"XAU"}:    # [] or ["XAU"]
            h.sentiment_score = shown
        h.score_source = "model"
    return len(todo)
```

Rules, with the reason each exists:

- **A. Never write the model score into a non-XAU currency.** `get_pair_sentiment` (:553-557) forms
  `XAU_mean − USD_mean`; `signal_aggregator.py:1677/1723` acts on it. The DeepSeek path puts its
  *overall* score into USD; the model has no overall score, so USD keeps the keyword number.
- **B. `sentiment_score` from the model only when the headline names no other currency.** The archive's
  `score` column is `sentiment_score`, and the ML-NEWS replay eval reads it; for a gold-only headline it
  should be the model's number, for "Fed hikes, dollar jumps" it stays keyword so the row is honest
  about what the model did and didn't score. `score_source="model"` therefore means "XAU score is the
  model's" — amend the dataclass comment at :223-227 to say so.
- **C. `xau_score` is always the raw signed score.** That column exists for training data
  (:229-237); keep it unfiltered.
- **D. Display fields get 0.0 when `relevant` is False.** A headline the model calls not-relevant can
  still carry |score| up to ~0.2 from a confident direction head; `NEUTRAL_HEADLINE_THRESHOLD = 0.1`
  (:256) would let 0.15 through as "leaning bearish". Zeroing the shown value makes "neutral" mean
  neutral. (0.1 happens to equal the model's irrelevant band `RELEVANT_MAX_IRRELEVANT`, so the two
  notions agree.)
- **E. Do not touch `NewsEngine._llm_sent_cache`.** That cache decides what is re-sent to DeepSeek. In
  `fallback` mode a headline that failed DeepSeek must stay uncached so it is retried next window and
  its archive row upgrades `model` → `llm` through the rank rule (news_archive.py:203-260).
- **F. Only `score_source == "keyword"` rows are touched.** Never overwrite an `llm` row in memory.

### 5.2 Call site (:944-947)
Replace
```python
scored = headlines + unmatched
try:
    self._apply_llm_sentiment(scored)
except Exception as e:
    logger.debug(f"LLM sentiment failed, keyword scores stand: {e}")
```
with
```python
scored = headlines + unmatched
mode = app_config.news_model_mode()
try:
    if mode == "primary":
        n = self._apply_model_sentiment(scored)
        if n == 0 and news_model.get_news_model() is None:
            # model unavailable this process: degrade to DeepSeek, not to keywords
            self._apply_llm_sentiment(scored)
        else:
            self._audit_with_deepseek(scored)         # no-op when audit_per_day == 0
    elif mode == "fallback":
        self._apply_llm_sentiment(scored)
        self._apply_model_sentiment(scored)           # only what DeepSeek left keyword-scored
    else:
        self._apply_llm_sentiment(scored)
except Exception as e:
    logger.debug(f"headline sentiment enrichment failed, keyword scores stand: {e}")
```
`import news_model` and `import app_config` at the module top of news_engine.py (news_model's top
imports only numpy, §3.2).

Mode semantics:

| mode | DeepSeek for headlines | model | when a step fails |
|---|---|---|---|
| `off` | as today | never | keyword scores stand (today) |
| `fallback` | first, as today | on rows DeepSeek left keyword-scored | keyword scores stand |
| `primary` | never (except audit) | on all rows | model missing → DeepSeek path for the cycle; model raises mid-batch → keyword |

### 5.3 Audit sample (primary mode, optional feature, default off)
```python
_audit_day: str = ""          # class attrs, like the caches
_audit_sent_today: int = 0

def _audit_with_deepseek(self, headlines):
    cap = app_config.news_model_audit_per_day()
    if cap <= 0 or not get_llm_client().available:
        return
    today = datetime.now(timezone.utc).date().isoformat()
    if NewsEngine._audit_day != today:
        NewsEngine._audit_day, NewsEngine._audit_sent_today = today, 0
    room = cap - NewsEngine._audit_sent_today
    if room <= 0:
        return
    cand = [h for h in headlines if h.score_source == "model"
            and hashlib.sha1(h.headline.encode()).hexdigest() not in NewsEngine._llm_sent_cache]
    batch = cand[:min(room, self._LLM_BATCH_CAP)]
    if not batch:
        return
    scores = self._llm_headline_scores([h.headline for h in batch])
    if scores is None:
        return
    NewsEngine._audit_sent_today += len(batch)
    copies = []
    for h, (overall, xau) in zip(batch, scores):
        NewsEngine._llm_sent_cache[hashlib.sha1(h.headline.encode()).hexdigest()] = (overall, xau)
        c = copy.copy(h)                     # the displayed object keeps the model score
        c.sentiment_score = overall
        c.sentiment_by_currency = {cur: (xau if cur == "XAU" else overall) for cur in h.currencies_mentioned}
        c.xau_score = xau
        c.score_source = "llm"
        copies.append(c)
    self._pending_audit_rows = copies        # archived by the call site right after `scored`
```
The call site then archives `scored` first and `self._pending_audit_rows` second (same
`teacher_model`/`teacher_prompt_sha1` as today's DeepSeek rows); the rank rule upgrades those rows
to `llm`, the in-memory objects the bot displays keep the model's numbers. One batch per cycle at
most, ≤ `cap` per UTC day. This is the only place DeepSeek is called for headlines in primary mode.

### 5.4 Archive call (:958-964)
Pass, in addition to today's kwargs:
```python
model_teacher=news_model.teacher_tag()   # "xau_transformer@<meta.trained_at>" or None
```
where `teacher_tag()` reads the loaded bundle's meta (None if not loaded). See §6.

### 5.5 Dataclass comment
`NewsSentiment.score_source` comment (:223-227): add `"model" = the local XAU transformer scored
xau_score (and XAU's per-currency score); other currencies remain keyword-scored.`

---

## 6. `news_archive.py` changes

No schema change. `_SOURCE_RANK` (:107) and `_STORED_RANK` (:203-206) already rank `model` = 2.

- `archive_headlines(headlines, *, teacher_model=None, teacher_prompt_sha1=None, origin="live",
  model_teacher=None)` — new kwarg, passed through to `build_row`.
- `build_row` (:323-354):
  ```python
  "teacher_model": (teacher_model if source == "llm"
                    else model_teacher if source == "model" else None),
  "teacher_prompt_sha1": (teacher_prompt_sha1 if source in ("llm", "model") else None),
  ```
  For a model row `teacher_prompt_sha1` is the DeepSeek prompt the model was distilled from (the same
  constant the bot already passes), which is the truthful provenance.
- Effect: a retrain is visible in the data (`teacher_model` changes), and the UPSERT's
  `COALESCE(excluded.teacher_model, headlines.teacher_model)` (:253-260) only applies when the new
  rank ≥ stored rank, so an `llm` row's teacher is never replaced by a model tag.

---

## 7. Tests (`test_all.py`)

The suite is a plain script: `@test(name)` (:51-64) runs each function at definition time and
tallies; the file ends with `sys.exit(1 if FAIL > 0 else 0)`. Run: `venv/bin/python test_all.py`.
Stubbing patterns in use: instance-level `ne._llm_headline_scores = lambda texts: …` with
`lc._client = _StubLLM({...})` and `finally` cleanup (:5196-5215); module-level swap
`saved = _ne_mod.get_llm_client; _ne_mod.get_llm_client = stub; … finally restore` (:11391-11424).

Add, using the same patterns and a fake bundle (`news_model.score_batch` swapped on the module):

1. **model scores keyword rows only.** Two `NewsSentiment`s, one `score_source="llm"`; after
   `_apply_model_sentiment`, the llm one is byte-identical, the keyword one has `score_source="model"`,
   `xau_score` set, `sentiment_by_currency["XAU"]` = score, `["USD"]` unchanged (rule A).
2. **overall score rule (B).** `currencies_mentioned=["XAU"]` → `sentiment_score` = model score;
   `["USD","XAU"]` → `sentiment_score` unchanged; `[]` → model score.
3. **relevance zeroing (D).** stub returns `relevant=False, score=-0.18` → `sentiment_by_currency["XAU"]
   == 0.0`, `xau_score == -0.18`.
4. **cache untouched (E).** `_llm_sent_cache` empty before and after.
5. **failure isolation.** stub raises → all rows still `keyword`, method returns 0, no exception.
   `news_model.get_news_model()` returning None → returns 0.
6. **primary degrades to DeepSeek.** `get_news_model` → None, `_llm_headline_scores` stubbed → rows
   end up `llm`.
7. **audit sample.** `deepseek_audit_per_day=3`, 10 model rows → exactly 3 sent, displayed objects
   still `model`, `_pending_audit_rows` has 3 `llm` copies; second call same day sends 0.
8. **config.** unknown key in `news_model` → `AppConfigError`; `mode="banana"` → error; section
   absent → `news_model_mode() == "off"`; `NEWS_MODEL_MODE=off` env beats `"mode": "primary"`.
9. **archive.** `build_row` of a `model` headline with `model_teacher="xau_transformer@T"` stores it;
   an `llm` row stores the DeepSeek teacher; existing rank tests still pass.
10. **import hygiene.** update the docstring of the test at :11437-11462 (it says numpy-only); add
    an assertion that `requirements.txt` lists `onnxruntime` and `tokenizers`; keep its "no torch /
    transformers" checks.
11. **integration, skipped when the bundle is absent** (`os.environ.get("NEWS_MODEL_DIR")` or the
    configured path missing → print SKIP): load, `score_news(b, "Fed cuts rates, signals more
    easing")["label"] == "bullish"`, `score_news(b, "Red Lobster endless shrimp losses")["label"] ==
    "neutral"`, latency of 20 calls < 1 s.

`test_all.py:3675-3690` loads `config.json.example` for pair output only; adding the section there is
safe.

---

## 8. Ops checklist → `deploy/README-news-model.md` (new)

Covers **both** processes: the bot (`telegram_bot.py`) and the dashboard daemon
(`deploy/gold-dashboard.service`).

0. `free -m`. Each model-loading process needs ~0.7 GB headroom. If only the bot should score with
   the model, add `Environment=NEWS_MODEL_MODE=off` to `deploy/gold-dashboard.service` (the dashboard
   then keeps using DeepSeek for its headlines; those `llm` rows land in the shared archive and act
   as free audit labels).
1. `cd /home/Forex/ForexBot2 && git pull && venv/bin/pip install -r requirements.txt`.
2. `mkdir -p /home/Forex/models/xau_transformer` and copy `model.onnx`, `tokenizer.json`, `meta.json`
   from the ML-NEWS machine (rsync, ~280 MB). Verify `sha256sum model.onnx` matches the source.
   Owner: the account the bot runs as; read-only is enough.
3. Add the `news_model` section to `config.json` (§4.1) with `"mode": "primary"`. Leave the
   `deepseek` section unchanged.
4. Smoke test without starting the bot:
   ```
   venv/bin/python -c "import news_model as m; b=m.get_news_model(); print(m.score_news(b,'Gold jumps as dollar slides'))"
   ```
   Expect `label: bullish`, and the "news model loaded" log line.
5. `venv/bin/python test_all.py` — green (integration test runs because the dir exists).
6. Restart the bot; restart the dashboard unit. Check:
   - log: one "news model loaded" line per process, no "news model unavailable";
   - `top`: RSS ~+0.6 GB per process, idle CPU ~0% between scans (spinning disabled);
   - archive: new rows have `score_source='model'`, `xau_score` set, `teacher_model`
     `xau_transformer@…`; DeepSeek daily counter for headlines stays at 0 (audit off).
7. Rollback: set `"mode": "fallback"` or `"off"` (or the env var), restart. No data migration needed.
8. Retrain cadence: when ML-NEWS ships a new bundle, copy the three files over the old ones and
   restart both processes. `meta.json.trained_at` changes, so the archive shows which model scored which
   rows. If `deepseek_audit_per_day` is set, ML-NEWS's `eval/eval_archive.py --scorer transformer` on a
   copy of the live archive gives the model-vs-DeepSeek agreement on those rows.

---

## 9. Acceptance criteria (end-to-end)

- With `mode="primary"` and DeepSeek enabled, one live news cycle archives only `model` rows (plus
  ≤ `deepseek_audit_per_day` `llm` rows), and the cycle's added latency is < 1 s for ~100 headlines.
- With the model directory renamed away, both processes start, log one warning each, and the cycle's
  rows are `llm` (DeepSeek path), not `keyword`.
- With `onnxruntime` uninstalled in a scratch venv, `python -c "import news_engine"` succeeds.
- A headline naming USD and XAU ends with `sentiment_by_currency["USD"]` equal to the keyword value
  and `["XAU"]` equal to the model value.
- `mode="off"` reproduces today's archive rows exactly (`llm`/`keyword`, same teacher fields).
- `venv/bin/python test_all.py` exits 0.

## 10. Known limits to state in the README
- The model copies DeepSeek's opinion (72% agreement on unseen live days, 72/78 golden); it does not
  learn from price. DeepSeek itself is ~90% self-consistent.
- Gold only. Every other currency's sentiment remains the ~31%-accurate keyword scorer in primary
  mode, which is what those currencies get today whenever DeepSeek is unavailable.
- With audit off, no new DeepSeek labels accrue; the ML-NEWS retrain loop needs either
  `deepseek_audit_per_day > 0`, the dashboard left on DeepSeek, or a separate labelling run.