ForexBot2/deploy/README-news-model.md

# Deploying the local news model (live server)

Since 2026-09-27 ForexBot2 can score news headlines with the owner's ML-NEWS gold transformer
instead of DeepSeek. The model runs on the CPU inside the bot process (and inside the dashboard
daemon), needs two extra Python packages and a 280 MB model folder that lives **outside** the git
tree. DeepSeek keeps its other jobs: the direction of a calendar surprise, the news summary and the
dashboard's on-demand summary endpoint.

What the model is, in one paragraph: a DeBERTa-v3-small transformer with two heads (does this
headline move gold? if so, up or down?) exported to ONNX, trained to copy DeepSeek's gold score on
17,245 headlines. On the 78-headline golden set it scores 72 (DeepSeek 78, the old keyword scorer
24); on unseen live days it agrees with DeepSeek about 72% of the time (DeepSeek agrees with itself
about 90%). It scores one headline in about 10–20 ms on one CPU thread. It is **gold only**: every
other currency's headline sentiment stays with the keyword scorer in `primary` mode.

Owner decisions this deployment implements:

- Mode `primary`: the model scores every headline; **no DeepSeek calls for headline scoring**.
- **The news archive stops growing.** In `primary` mode the model's own output is not archived (it is
  not training data). Only the optional DeepSeek audit sample (`deepseek_audit_per_day`, shipped 0)
  writes rows. `logs/newsdata.db` stays as it is; nothing is migrated or deleted.
- Both processes (bot and dashboard daemon) load the model so Telegram and the dashboard show the same
  headline scores. Each ONNX session holds about 0.6 GB of RAM.

## 0. Check memory

```bash
free -m
```

Each process that loads the model needs about 0.7 GB of headroom; two processes about 1.5 GB. If the
box cannot spare that, keep the model on the bot only: uncomment the `Environment=NEWS_MODEL_MODE=off`
line in `deploy/gold-dashboard.service` (step 5). The dashboard then keeps scoring its headlines with
DeepSeek and its scores may differ from the bot's. The dashboard process never writes the archive in
any mode, so this choice does not affect training data.

## 1. Code and dependencies

```bash
cd /home/Forex/ForexBot2
git pull
venv/bin/pip install -r requirements.txt      # adds onnxruntime and tokenizers
```

`requirements.txt` must never contain the training stack (no torch, no transformers); the test suite
checks that. The live box runs Python 3.9, where pip resolves onnxruntime 1.19.2 and tokenizers
0.22.2; the bundle of 2026-09-27 scores identically there and on the dev box (1.30.0 / 0.23.2).

## 2. Copy the model files (outside git)

On the machine that holds ML-NEWS (`~/Development/ML-NEWS/models/xau_transformer/`), copy exactly
three files. Skip the `torch/` and `torch_final/` subfolders; they are training checkpoints.

```bash
ssh Forex@LIVE 'mkdir -p /home/Forex/models/xau_transformer'
rsync -av --progress \
    ~/Development/ML-NEWS/models/xau_transformer/{model.onnx,tokenizer.json,meta.json} \
    Forex@LIVE:/home/Forex/models/xau_transformer/
```

Verify on the live box:

```bash
sha256sum /home/Forex/models/xau_transformer/model.onnx
# bundle of 2026-09-27: 9e7a64d3ae1df335059ce829cafcf6e4544b85a4f72cc88923c2e42c48e975a0
sha256sum /home/Forex/models/xau_transformer/tokenizer.json
# bundle of 2026-09-27: 045f93f80eb9ad18e559647563fb827dea59d93258fdb8b922de82bc53666d16
python3 -c "import json; print(json.load(open('/home/Forex/models/xau_transformer/meta.json'))['trained_at'])"
# 2026-09-27T23:15:30.403442+00:00
```

The files should be owned by (or readable by) the account the bot runs as. Read-only is enough.

## 3. Configure

Add one block to `/home/Forex/ForexBot2/config.json` (any position; `config.json.example` shows it).
Leave the `deepseek` block as it is.

```json
"news_model": {
  "path": "/home/Forex/models/xau_transformer",
  "threads": 2,
  "mode": "primary",
  "deepseek_audit_per_day": 0
}
```

(`threads` is clamped to the core count; the live box has 2 cores, hence 2. Measured there: load
0.8 s, 16 ms per headline, 633 MB resident for the session.)

| Key | Meaning |
|---|---|
| `path` | folder with `model.onnx`, `tokenizer.json`, `meta.json`; required when `mode` is not `off` |
| `threads` | ONNX threads per process. Measured: 1 thread 19 ms per headline, 4 threads 9 ms, 16 threads 13 ms. 4 is right for a 4–8 core box |
| `mode` | `off` = today's DeepSeek/keyword path, nothing loads. `fallback` = DeepSeek first, the model only for headlines DeepSeek failed on. `primary` = the model scores everything, DeepSeek never sees a headline (except the audit sample) |
| `deepseek_audit_per_day` | `primary` only: headlines per UTC day also sent to DeepSeek and archived as `llm` rows (one call of up to 25 per cycle at most). They are training labels for the next ML-NEWS retrain; the bot never displays them. 0 = no DeepSeek headline calls at all |

The environment variable `NEWS_MODEL_MODE=off|fallback|primary` overrides `mode` for one process. It
exists so the dashboard daemon's systemd unit can pick its own mode.

A config error (unknown key, bad mode, missing path) stops the bot at boot with a clear message, like
every other `config.json` mistake.

## 4. Smoke test without starting the bot

```bash
cd /home/Forex/ForexBot2
venv/bin/python -c "
import news_model as m
b = m.get_news_model()
print(m.score_news(b, 'Gold jumps as dollar slides'))
print(m.score_news(b, 'Red Lobster endless shrimp losses'))"
```

Expect a log line `news model loaded from /home/Forex/models/xau_transformer (trained_at 2026-09-27…,
threshold 0.2, 4 threads)`, then `label: bullish` for the first headline and `label: neutral` for the
second. If instead you see `news model unavailable, DeepSeek/keyword path stays: …`, the message names
the cause (missing file, missing package, bad JSON). The bot would still start in that state and
would score with DeepSeek for the whole process lifetime; fix the cause and restart.

Then run the suite. With the model folder configured, the integration test loads the real bundle:

```bash
venv/bin/python test_all.py        # expect RESULTS: … passed, 0 failed
```

## 5. Restart both processes

Bot: restart it the way it is run on this box (nohup / screen / its unit). Dashboard:

```bash
sudo systemctl restart gold-dashboard
```

If the dashboard must not load the model (step 0), first uncomment the
`Environment=NEWS_MODEL_MODE=off` line in `deploy/gold-dashboard.service`, then
`sudo cp deploy/gold-dashboard.service /etc/systemd/system/ && sudo systemctl daemon-reload` before
the restart.

## 6. What to check afterwards

- **Logs** (`logs/telegram.log`, `logs/dashboard.log`): exactly one `news model loaded` line per
  process, no `news model unavailable` line. If the level is DEBUG you also see
  `news archive: primary mode, 0 DeepSeek audit rows stored, N model/keyword rows not archived` every
  news cycle.
- **`top`**: each process about 0.6 GB more resident memory than before; CPU near 0% between scans
  (thread spinning is disabled in the runtime).
- **DeepSeek spend**: the daily request count no longer moves with the headline count; only the event
  direction, the summary and the panel use it.
- **Archive frozen** (the intended state with audit 0):

  ```bash
  sqlite3 -readonly logs/newsdata.db "SELECT MAX(first_seen), COUNT(*) FROM headlines;"
  ```

  The newest `first_seen` stops advancing after the restart. With `deepseek_audit_per_day` > 0 you
  see at most that many new rows a day, all `score_source = 'llm'`.
- **Telegram / dashboard**: headline scores still appear; the wording and layout are unchanged, only
  the numbers now come from the model. A headline the model calls irrelevant to gold shows as neutral.

## 7. Rollback

Set `"mode": "fallback"` (DeepSeek first, the model only fills gaps and the archive grows again) or
`"mode": "off"` (exactly the pre-2026-09-27 behaviour) in `config.json`, or export
`NEWS_MODEL_MODE=off` for one process, and restart. No data migration in either direction.

## 8. Retraining cadence

When ML-NEWS ships a new bundle, copy the three files over the old ones (step 2), check the new
`trained_at`, and restart both processes. Rows written in `fallback` mode carry
`teacher_model = xau_transformer@<trained_at>`, so the archive shows which bundle scored them.

With `deepseek_audit_per_day` at 0 the archive receives no new DeepSeek labels, so the next retrain
needs one of: a positive audit value for a few weeks, a spell in `fallback` mode, or a separate
labelling run in ML-NEWS on freshly crawled headlines (`eval/eval_archive.py --scorer transformer` on a
copy of the live archive gives the model-versus-DeepSeek agreement on whatever labelled rows exist).

## Known limits

- The model copies DeepSeek's opinion; it has not learned from price moves. DeepSeek itself is only
  about 90% consistent with itself, and the model agrees with it about 72% of the time.
- Gold only. In `primary` mode every other currency's headline sentiment is the keyword scorer (24 of
  78 on the golden set), which is what those currencies got whenever DeepSeek was unavailable before.
- News never enters signals (CLAUDE.md section 7). No trade, grade or backtest book changes with any of
  this; the model only changes the headline lines people read.