Local news model: ML-NEWS XAU transformer scores headlines (news_model.py, mode primary)
The owner's DeBERTa-v3-small two-head ONNX model (golden set 72/78, ~72% agreement with DeepSeek on unseen days) replaces DeepSeek for headline scoring. DeepSeek keeps event direction, the summary and the dashboard endpoint. - news_model.py: vendored ML-NEWS runtime (lazy onnxruntime/tokenizers, so the bot starts without them) + load-once, never-raising singleton. - news_engine: _enrich_headline_scores mode switch (off/fallback/primary), _apply_model_sentiment (gold head only: xau_score + XAU entry, never USD; not-relevant shows 0.0; never touches _llm_sent_cache), _audit_with_deepseek (archive-only labels, deepseek_audit_per_day). - Owner decision: primary mode writes nothing to logs/newsdata.db except audit copies — the model's output is not training data. - app_config: news_model section (path/threads/mode/deepseek_audit_per_day) + NEWS_MODEL_MODE env override; validated at boot. - news_archive: model rows carry teacher_model xau_transformer@<trained_at>. - deploy/README-news-model.md (ops for bot + dashboard daemon), unit gets a commented NEWS_MODEL_MODE=off line; requirements + onnxruntime, tokenizers. - test_all.py Module 38 (15 tests, one against the real bundle). Suite 581. - CLAUDE.md, spec status note, handoff marked implemented. Books unaffected: news never enters signals; no signal file touched.
This commit is contained in:
Harwinder Sekhon
@@ -0,0 +1,535 @@
|
||||
# Specification: run the ML-NEWS XAU transformer inside ForexBot2
|
||||
|
||||
Status: **IMPLEMENTED 2026-09-27** (ForexBot2 branch `sql-integration`, on top of `6af1237`; suite
|
||||
Module 38). The text below is the specification as written before the implementation; file:line
|
||||
numbers refer to `6af1237` and have drifted. Where the implementation differs, this list wins:
|
||||
|
||||
- **`primary` mode does not archive the cycle's rows.** Owner decision during implementation: once the
|
||||
local model scores, `logs/newsdata.db` stops growing, because the model's own output is not training
|
||||
data. Only the DeepSeek audit copies (`deepseek_audit_per_day`, shipped 0) are written. `off` and
|
||||
`fallback` archive as before (a `fallback` model row carries `model_teacher`). §5.4 and §9 read
|
||||
accordingly.
|
||||
- `_audit_with_deepseek` **returns** the `llm` copies; there is no `_pending_audit_rows` attribute.
|
||||
The mode switch of §5.2 lives in `NewsEngine._enrich_headline_scores(scored, mode)`, which returns
|
||||
those copies, so the call site and the tests share one function.
|
||||
- `config.json.example` ships `"mode": "off"`; the live server sets `primary`
|
||||
(`deploy/README-news-model.md`).
|
||||
- §8 step 0 was wrong: the dashboard daemon never writes the archive in any mode
|
||||
(`set_db_writes_for_process(False)`), so its DeepSeek rows are not "free audit labels". Both
|
||||
processes load the model (owner decision); `NEWS_MODEL_MODE=off` stays as the memory fallback.
|
||||
- `warm_up_in_background()` is implemented (module-level once flag; a `NewsEngine` is built per scan).
|
||||
- `news_model.get_news_model()` also treats a missing `path` as a load failure (one warning).
|
||||
- `requirements.txt` ranges: `onnxruntime>=1.19,<2`, `tokenizers>=0.22,<1`. The live box runs
|
||||
Python 3.9, where only 1.19.2 / 0.22.2 install; verified on 2026-09-28 to score the bundle
|
||||
identically to the dev box's 1.30.0 / 0.23.2 (golden pair, four probe headlines).
|
||||
- ML-NEWS `news_transformer.py` received the §3.4 edits in the same session; the vendored copy differs
|
||||
only in `load_model` having no default directory.
|
||||
|
||||
Repos:
|
||||
- **ML-NEWS** `~/Development/ML-NEWS` (git `ssh://git@192.168.0.11:222/harry/News-ML-Decoder.git`).
|
||||
Produces the model. Nothing in ForexBot2 imports from it.
|
||||
- **ForexBot2** `~/Development/ForexBot2`, live at `/home/Forex/ForexBot2`, deployed by `git pull`.
|
||||
Consumes the model.
|
||||
|
||||
---
|
||||
|
||||
## 1. Context and goal
|
||||
|
||||
### 1.1 What the model is
|
||||
`ML-NEWS/models/xau_transformer/` holds a fine-tuned `microsoft/deberta-v3-small` with two heads,
|
||||
exported to ONNX:
|
||||
|
||||
| File | Size | Purpose |
|
||||
|---|---|---|
|
||||
| `model.onnx` | 271 MB | encoder + heads; int8 embedding table, fp32 compute |
|
||||
| `tokenizer.json` | 8 MB | HF `tokenizers` fast tokenizer |
|
||||
| `meta.json` | 4 KB | thresholds, `trained_at`, `relevance_threshold`, `export_checks`, training stats |
|
||||
|
||||
Inputs: one headline, ≤ 64 tokens. Outputs per headline (from `news_transformer.score_batch`):
|
||||
|
||||
```python
|
||||
{"headline": str, "p_relevant": float, "p_bullish": float,
|
||||
"relevant": bool, # p_relevant >= meta["relevance_threshold"] (0.2 for the shipped bundle)
|
||||
"direction": "bullish"|"bearish",
|
||||
"label": "bullish"|"bearish"|"neutral", # neutral when not relevant
|
||||
"score": float} # p_relevant * (2*p_bullish - 1), in [-1, 1]
|
||||
```
|
||||
|
||||
`score` is on the same scale as DeepSeek's `xau` score that ForexBot2 already stores in
|
||||
`xau_score`. The model was trained to copy DeepSeek (`deepseek-v4-flash`, prompt sha1 `d32e04c6…`, the
|
||||
same prompt as `news_engine._LLM_SENTIMENT_PROMPT_SHA1`) on 17,245 headlines. Quality on
|
||||
2026-09-27: golden set 72/78 (DeepSeek 78, keyword scorer 24), agreement with DeepSeek on unseen live
|
||||
days ~72%, CPU latency 19 ms/headline on 1 thread, 9 ms on 4. Full numbers: `ML-NEWS/eval/RESULTS_2026-09-27.md`.
|
||||
|
||||
**The model scores gold only.** It has no head for USD, EUR or anything else. This drives most of
|
||||
the rules in §5.
|
||||
|
||||
### 1.2 How ForexBot2 scores headlines today
|
||||
`news_engine.py`, inside `NewsEngine._fetch_news_sentiment_raw` (called via
|
||||
`_cached_fetch("_headlines_cache", …)` at :822, which holds `NewsEngine._cache_lock`, :629-647):
|
||||
|
||||
1. RSS items become `NewsSentiment` objects (:891-910) with the **keyword scorer** filling
|
||||
`sentiment_score` and `sentiment_by_currency[cur]` for each currency in `currencies_mentioned`,
|
||||
`score_source="keyword"`, `xau_score=None`.
|
||||
2. `scored = headlines + unmatched` (:944); `self._apply_llm_sentiment(scored)` (:945, in try/except).
|
||||
`_apply_llm_sentiment` (:1032-1065) sends headlines not in `NewsEngine._llm_sent_cache` to DeepSeek
|
||||
in batches of `_LLM_BATCH_CAP = 25` via `self._llm_headline_scores(texts)` (:1009) and, on success,
|
||||
overwrites `sentiment_score`, `sentiment_by_currency` (XAU gets the gold score, every other currency
|
||||
the overall score), sets `xau_score`, `score_source="llm"`. A failed batch stops sending; those
|
||||
headlines stay keyword-scored and **stay out of the cache**, so they are re-sent next window.
|
||||
3. `news_archive.archive_headlines(scored, teacher_model=…, teacher_prompt_sha1=…)` (:958-964).
|
||||
The archive UPSERT (news_archive.py:203-260) keeps the higher-ranked source:
|
||||
`_SOURCE_RANK = {"unknown": 0, "keyword": 1, "model": 2, "llm": 3}` (:107). `"model"` is already
|
||||
reserved for "a trained local artifact" (news_engine.py:223-227). `build_row` (:323-354) stores
|
||||
`teacher_model`/`teacher_prompt_sha1` only when `source == "llm"`.
|
||||
4. Downstream consumers of the scores: `get_fundamental_context` (drops |score| <
|
||||
`NEUTRAL_HEADLINE_THRESHOLD = 0.1`, :256), `get_pair_sentiment` (:553-557, computes
|
||||
`XAU_mean − USD_mean`), `signal_aggregator.py:1677/1723` (acts on that when
|
||||
`sentiment_affects_signals`), `agent_panel.py`, the dashboard.
|
||||
|
||||
Two processes run this code: the bot (`venv/bin/python telegram_bot.py`, CLAUDE.md:30) and the
|
||||
dashboard daemon (`dashboard_publisher.py:401-403` builds its own `NewsEngine`, `:503` calls
|
||||
`get_fundamental_context()`; unit `deploy/gold-dashboard.service`). They share `config.json` and the
|
||||
archive DB.
|
||||
|
||||
### 1.3 Decisions taken by the owner (2026-09-27)
|
||||
- **Mode `primary`, no DeepSeek for headline scoring.** The model scores every headline. DeepSeek stays
|
||||
for everything else it does (`_llm_event_direction` for calendar surprises, the news summary, the
|
||||
dashboard's on-demand `news_summary.php`).
|
||||
- **Model files live outside the ForexBot2 tree**, path in `config.json`.
|
||||
- Accepted consequence: the archive stops receiving fresh DeepSeek labels, so the ML-NEWS weekly
|
||||
retrain and the "replay and compare" check lose their data source. Mitigation kept optional:
|
||||
`deepseek_audit_per_day` (§4) labels a few headlines a day for the archive only. Default 0.
|
||||
|
||||
### 1.4 Non-goals
|
||||
- Removing the DeepSeek client or its non-headline uses.
|
||||
- Scoring non-gold currencies with the model.
|
||||
- Changing ML-NEWS training. (The one ML-NEWS change is §3.4, a runtime option.)
|
||||
- Training from price moves.
|
||||
|
||||
---
|
||||
|
||||
## 2. Target layout on the live server
|
||||
|
||||
```
|
||||
/home/Forex/ForexBot2/ # git pull as today
|
||||
news_model.py # NEW, vendored runtime (§3)
|
||||
requirements.txt # + onnxruntime, tokenizers (§3.1)
|
||||
app_config.py, news_engine.py, news_archive.py, test_all.py # edited (§4, §5, §6, §7)
|
||||
config.json # + "news_model" section (§4)
|
||||
deploy/README-news-model.md # NEW ops checklist (§8)
|
||||
deploy/gold-dashboard.service # maybe + Environment=NEWS_MODEL_MODE=off (§8)
|
||||
/home/Forex/models/xau_transformer/ # NEW, outside git, copied by hand
|
||||
model.onnx tokenizer.json meta.json
|
||||
```
|
||||
|
||||
Memory: each process that loads the model holds ~0.5–0.7 GB RSS for the ONNX session. Two processes
|
||||
→ ~1.5 GB. `free -m` before deciding whether the dashboard daemon also runs it (§8 step 0).
|
||||
|
||||
---
|
||||
|
||||
## 3. `news_model.py` (new, ForexBot2)
|
||||
|
||||
### 3.1 Dependencies
|
||||
Add to `requirements.txt`:
|
||||
```
|
||||
onnxruntime>=1.30,<2
|
||||
tokenizers>=0.23,<1
|
||||
```
|
||||
Pin to the versions that verified the bundle in ML-NEWS `.venv` (check `pip show` there before
|
||||
writing). `tokenizer.json` compatibility across older `tokenizers` releases is not guaranteed.
|
||||
**Do not write the words "torch" or "transformers" anywhere in requirements.txt**, not even in a
|
||||
comment: `test_all.py:11437-11462` greps the file for them and fails.
|
||||
|
||||
### 3.2 Source
|
||||
Copy `ML-NEWS/news_transformer.py` → `ForexBot2/news_model.py`. Keep verbatim: `clean_headline`,
|
||||
`_sigmoid`, `load_model`, `score_batch`, `score_news`, constants `MAX_LENGTH = 64`,
|
||||
`RELEVANT_MIN = 0.25`, `RELEVANT_MAX_IRRELEVANT = 0.10`. Drop the CLI `main` or keep it, either is fine.
|
||||
Add a docstring line: `Vendored from ML-NEWS news_transformer.py at commit <sha>; bundle trained_at
|
||||
<meta.json trained_at>.`
|
||||
|
||||
**Imports.** `onnxruntime` and `tokenizers` must stay imported *inside* `load_model` (as in
|
||||
news_transformer.py:58-59). Module top imports only `numpy`, `json`, `logging`, `os`, `pathlib`,
|
||||
`threading`. Reason: `news_engine.py` will `import news_model` at its top; if onnxruntime is missing on
|
||||
a box, the bot must still start.
|
||||
|
||||
### 3.3 Singleton
|
||||
```python
|
||||
_lock = threading.Lock()
|
||||
_bundle: Optional[dict] = None
|
||||
_failed: bool = False # tried and failed; don't retry, don't re-log
|
||||
|
||||
def get_news_model() -> Optional[dict]:
|
||||
"""The loaded bundle, or None. Loads once per process. Never raises."""
|
||||
global _bundle, _failed
|
||||
if _bundle is not None or _failed:
|
||||
return _bundle
|
||||
with _lock:
|
||||
if _bundle is not None or _failed:
|
||||
return _bundle
|
||||
try:
|
||||
import app_config
|
||||
path = app_config.news_model_path()
|
||||
threads = app_config.news_model_threads()
|
||||
b = load_model(path, threads=threads)
|
||||
logger.info("news model loaded from %s (trained_at %s, threshold %s, %d threads)",
|
||||
path, b["meta"].get("trained_at"), b["meta"].get("relevance_threshold"), threads)
|
||||
_bundle = b
|
||||
except Exception as e: # ImportError, FileNotFoundError, bad JSON, ORT errors
|
||||
logger.warning("news model unavailable, DeepSeek/keyword path stays: %s: %s",
|
||||
type(e).__name__, e)
|
||||
_failed = True
|
||||
return _bundle
|
||||
|
||||
def reset_for_tests() -> None: # clears _bundle/_failed; used by test_all.py
|
||||
```
|
||||
|
||||
Optional: `warm_up_in_background()` that calls `get_news_model()` in a daemon thread at
|
||||
`NewsEngine.__init__`, so the first news cycle isn't blocked by the ~3–5 s session creation under
|
||||
`_cache_lock`. Nice to have, not required.
|
||||
|
||||
### 3.4 Two changes inside `load_model` (also apply to ML-NEWS `news_transformer.py` so the copies stay identical)
|
||||
```python
|
||||
options = ort.SessionOptions()
|
||||
if threads:
|
||||
options.intra_op_num_threads = max(1, min(int(threads), os.cpu_count() or 1))
|
||||
options.inter_op_num_threads = 1
|
||||
# ORT intra-op threads spin-wait after each run by default; in a mostly idle
|
||||
# bot process that is 4 cores at 100% between cycles.
|
||||
options.add_session_config_entry("session.intra_op.allow_spinning", "0")
|
||||
```
|
||||
|
||||
### 3.5 Contract `score_batch(bundle, headlines: list[str]) -> list[dict]`
|
||||
Unchanged from ML-NEWS. Empty or whitespace-only headline raises `ValueError` from `clean_headline`;
|
||||
callers pass already-non-empty `NewsSentiment.headline`, but §5 still wraps per-batch in try/except.
|
||||
Batch size: `_apply_model_sentiment` sends all of a cycle's headlines in one call (typically 30–150);
|
||||
`score_batch` pads to the longest in the batch, ≤ 64 tokens, so one call of 150 is ~1 s on 4 threads.
|
||||
|
||||
---
|
||||
|
||||
## 4. Configuration
|
||||
|
||||
### 4.1 `config.json` section (add to `config.json.example` too)
|
||||
```json
|
||||
"news_model": {
|
||||
"path": "/home/Forex/models/xau_transformer",
|
||||
"threads": 4,
|
||||
"mode": "primary",
|
||||
"deepseek_audit_per_day": 0
|
||||
}
|
||||
```
|
||||
|
||||
| Key | Type | Default if absent | Meaning |
|
||||
|---|---|---|---|
|
||||
| `path` | str | required when mode ≠ off | directory with the three bundle files |
|
||||
| `threads` | int ≥ 1 | 4 | ORT intra-op threads (measured: 1→19 ms, 4→9 ms, 16→13 ms) |
|
||||
| `mode` | `"off"`/`"fallback"`/`"primary"` | `"off"` | see §5.2 |
|
||||
| `deepseek_audit_per_day` | int ≥ 0 | 0 | primary mode only: headlines/day to also send to DeepSeek, archive-only |
|
||||
|
||||
Whole section absent → `mode="off"` → today's behaviour exactly.
|
||||
|
||||
Environment override: `NEWS_MODEL_MODE=off|fallback|primary` wins over config.json. Purpose: the
|
||||
dashboard daemon's systemd unit can turn the model off for that process only. Pattern to copy:
|
||||
`NEWS_ARCHIVE_ENABLED` in `app_config.py:574-583`.
|
||||
|
||||
### 4.2 `app_config.py` edits (follow the `news_archive` section, :397-406, accessors :574/:588)
|
||||
1. `_KNOWN_TOP` (:86-92): add `"news_model"`. **Required**; `_check_keys(raw, _KNOWN_TOP, "top level")`
|
||||
at :181 makes boot fail on an unknown top-level key.
|
||||
2. `_KNOWN_NEWS_MODEL = {"path", "threads", "mode", "deepseek_audit_per_day"}` next to
|
||||
`_KNOWN_NEWS_ARCHIVE` (:113).
|
||||
3. `AppConfig` dataclass (~:147): `news_model_settings: Dict[str, Any] = field(default_factory=dict)`.
|
||||
4. In `load_app_config`, after the `news_archive` block:
|
||||
```python
|
||||
nm = raw.get("news_model", {})
|
||||
_check_keys(nm, _KNOWN_NEWS_MODEL, '"news_model"')
|
||||
if "mode" in nm and nm["mode"] not in ("off", "fallback", "primary"):
|
||||
raise AppConfigError('config.json: news_model.mode must be "off", "fallback" or "primary"')
|
||||
if "threads" in nm and (not isinstance(nm["threads"], int) or isinstance(nm["threads"], bool) or nm["threads"] < 1):
|
||||
raise AppConfigError("config.json: news_model.threads must be a positive integer")
|
||||
if "deepseek_audit_per_day" in nm and (not isinstance(nm["deepseek_audit_per_day"], int) or nm["deepseek_audit_per_day"] < 0):
|
||||
raise AppConfigError("config.json: news_model.deepseek_audit_per_day must be a non-negative integer")
|
||||
if nm.get("mode", "off") != "off" and not isinstance(nm.get("path"), str):
|
||||
raise AppConfigError("config.json: news_model.path is required when mode is not off")
|
||||
app.news_model_settings = dict(nm)
|
||||
```
|
||||
5. Accessors:
|
||||
```python
|
||||
def news_model_mode() -> str: # env NEWS_MODEL_MODE overrides; default "off"
|
||||
def news_model_path() -> Optional[str]
|
||||
def news_model_threads() -> int # default 4
|
||||
def news_model_audit_per_day() -> int # default 0
|
||||
```
|
||||
`NewsConfig` in `config.py:169` is **not** touched: tests construct it bare as `NewsConfig()`, and it
|
||||
holds feed/threshold settings, not paths. `news_engine.py` reads the new settings through `app_config`
|
||||
the same way it reaches DeepSeek through `get_llm_client()`.
|
||||
|
||||
---
|
||||
|
||||
## 5. `news_engine.py` changes
|
||||
|
||||
### 5.1 New method `_apply_model_sentiment`
|
||||
Place next to `_apply_llm_sentiment` (:1032). Runs under `_cache_lock` like it.
|
||||
|
||||
```python
|
||||
def _apply_model_sentiment(self, headlines: List[NewsSentiment]) -> int:
|
||||
"""Score with the local XAU transformer every headline that is still
|
||||
keyword-scored. Returns the number scored. Never raises.
|
||||
|
||||
The model has a gold head only. It writes xau_score always, XAU's
|
||||
per-currency score when XAU is mentioned, and the overall score only
|
||||
when the headline names no other currency. USD/EUR/... entries keep
|
||||
their keyword score: get_pair_sentiment subtracts USD from XAU and
|
||||
must not see the same number on both sides.
|
||||
"""
|
||||
bundle = news_model.get_news_model()
|
||||
if bundle is None or not headlines:
|
||||
return 0
|
||||
todo = [h for h in headlines if h.score_source == "keyword"]
|
||||
if not todo:
|
||||
return 0
|
||||
try:
|
||||
results = news_model.score_batch(bundle, [h.headline for h in todo])
|
||||
except Exception as e:
|
||||
logger.debug(f"news model scoring failed, keyword scores stand: {e}")
|
||||
return 0
|
||||
for h, r in zip(todo, results):
|
||||
raw = float(r["score"]) # archive value, unmodified
|
||||
shown = raw if r["relevant"] else 0.0 # display value, see rule D
|
||||
h.xau_score = raw
|
||||
if "XAU" in h.currencies_mentioned:
|
||||
h.sentiment_by_currency["XAU"] = shown
|
||||
if set(h.currencies_mentioned) <= {"XAU"}: # [] or ["XAU"]
|
||||
h.sentiment_score = shown
|
||||
h.score_source = "model"
|
||||
return len(todo)
|
||||
```
|
||||
|
||||
Rules, with the reason each exists:
|
||||
|
||||
- **A. Never write the model score into a non-XAU currency.** `get_pair_sentiment` (:553-557) forms
|
||||
`XAU_mean − USD_mean`; `signal_aggregator.py:1677/1723` acts on it. The DeepSeek path puts its
|
||||
*overall* score into USD; the model has no overall score, so USD keeps the keyword number.
|
||||
- **B. `sentiment_score` from the model only when the headline names no other currency.** The archive's
|
||||
`score` column is `sentiment_score`, and the ML-NEWS replay eval reads it; for a gold-only headline it
|
||||
should be the model's number, for "Fed hikes, dollar jumps" it stays keyword so the row is honest
|
||||
about what the model did and didn't score. `score_source="model"` therefore means "XAU score is the
|
||||
model's" — amend the dataclass comment at :223-227 to say so.
|
||||
- **C. `xau_score` is always the raw signed score.** That column exists for training data
|
||||
(:229-237); keep it unfiltered.
|
||||
- **D. Display fields get 0.0 when `relevant` is False.** A headline the model calls not-relevant can
|
||||
still carry |score| up to ~0.2 from a confident direction head; `NEUTRAL_HEADLINE_THRESHOLD = 0.1`
|
||||
(:256) would let 0.15 through as "leaning bearish". Zeroing the shown value makes "neutral" mean
|
||||
neutral. (0.1 happens to equal the model's irrelevant band `RELEVANT_MAX_IRRELEVANT`, so the two
|
||||
notions agree.)
|
||||
- **E. Do not touch `NewsEngine._llm_sent_cache`.** That cache decides what is re-sent to DeepSeek. In
|
||||
`fallback` mode a headline that failed DeepSeek must stay uncached so it is retried next window and
|
||||
its archive row upgrades `model` → `llm` through the rank rule (news_archive.py:203-260).
|
||||
- **F. Only `score_source == "keyword"` rows are touched.** Never overwrite an `llm` row in memory.
|
||||
|
||||
### 5.2 Call site (:944-947)
|
||||
Replace
|
||||
```python
|
||||
scored = headlines + unmatched
|
||||
try:
|
||||
self._apply_llm_sentiment(scored)
|
||||
except Exception as e:
|
||||
logger.debug(f"LLM sentiment failed, keyword scores stand: {e}")
|
||||
```
|
||||
with
|
||||
```python
|
||||
scored = headlines + unmatched
|
||||
mode = app_config.news_model_mode()
|
||||
try:
|
||||
if mode == "primary":
|
||||
n = self._apply_model_sentiment(scored)
|
||||
if n == 0 and news_model.get_news_model() is None:
|
||||
# model unavailable this process: degrade to DeepSeek, not to keywords
|
||||
self._apply_llm_sentiment(scored)
|
||||
else:
|
||||
self._audit_with_deepseek(scored) # no-op when audit_per_day == 0
|
||||
elif mode == "fallback":
|
||||
self._apply_llm_sentiment(scored)
|
||||
self._apply_model_sentiment(scored) # only what DeepSeek left keyword-scored
|
||||
else:
|
||||
self._apply_llm_sentiment(scored)
|
||||
except Exception as e:
|
||||
logger.debug(f"headline sentiment enrichment failed, keyword scores stand: {e}")
|
||||
```
|
||||
`import news_model` and `import app_config` at the module top of news_engine.py (news_model's top
|
||||
imports only numpy, §3.2).
|
||||
|
||||
Mode semantics:
|
||||
|
||||
| mode | DeepSeek for headlines | model | when a step fails |
|
||||
|---|---|---|---|
|
||||
| `off` | as today | never | keyword scores stand (today) |
|
||||
| `fallback` | first, as today | on rows DeepSeek left keyword-scored | keyword scores stand |
|
||||
| `primary` | never (except audit) | on all rows | model missing → DeepSeek path for the cycle; model raises mid-batch → keyword |
|
||||
|
||||
### 5.3 Audit sample (primary mode, optional feature, default off)
|
||||
```python
|
||||
_audit_day: str = "" # class attrs, like the caches
|
||||
_audit_sent_today: int = 0
|
||||
|
||||
def _audit_with_deepseek(self, headlines):
|
||||
cap = app_config.news_model_audit_per_day()
|
||||
if cap <= 0 or not get_llm_client().available:
|
||||
return
|
||||
today = datetime.now(timezone.utc).date().isoformat()
|
||||
if NewsEngine._audit_day != today:
|
||||
NewsEngine._audit_day, NewsEngine._audit_sent_today = today, 0
|
||||
room = cap - NewsEngine._audit_sent_today
|
||||
if room <= 0:
|
||||
return
|
||||
cand = [h for h in headlines if h.score_source == "model"
|
||||
and hashlib.sha1(h.headline.encode()).hexdigest() not in NewsEngine._llm_sent_cache]
|
||||
batch = cand[:min(room, self._LLM_BATCH_CAP)]
|
||||
if not batch:
|
||||
return
|
||||
scores = self._llm_headline_scores([h.headline for h in batch])
|
||||
if scores is None:
|
||||
return
|
||||
NewsEngine._audit_sent_today += len(batch)
|
||||
copies = []
|
||||
for h, (overall, xau) in zip(batch, scores):
|
||||
NewsEngine._llm_sent_cache[hashlib.sha1(h.headline.encode()).hexdigest()] = (overall, xau)
|
||||
c = copy.copy(h) # the displayed object keeps the model score
|
||||
c.sentiment_score = overall
|
||||
c.sentiment_by_currency = {cur: (xau if cur == "XAU" else overall) for cur in h.currencies_mentioned}
|
||||
c.xau_score = xau
|
||||
c.score_source = "llm"
|
||||
copies.append(c)
|
||||
self._pending_audit_rows = copies # archived by the call site right after `scored`
|
||||
```
|
||||
The call site then archives `scored` first and `self._pending_audit_rows` second (same
|
||||
`teacher_model`/`teacher_prompt_sha1` as today's DeepSeek rows); the rank rule upgrades those rows
|
||||
to `llm`, the in-memory objects the bot displays keep the model's numbers. One batch per cycle at
|
||||
most, ≤ `cap` per UTC day. This is the only place DeepSeek is called for headlines in primary mode.
|
||||
|
||||
### 5.4 Archive call (:958-964)
|
||||
Pass, in addition to today's kwargs:
|
||||
```python
|
||||
model_teacher=news_model.teacher_tag() # "xau_transformer@<meta.trained_at>" or None
|
||||
```
|
||||
where `teacher_tag()` reads the loaded bundle's meta (None if not loaded). See §6.
|
||||
|
||||
### 5.5 Dataclass comment
|
||||
`NewsSentiment.score_source` comment (:223-227): add `"model" = the local XAU transformer scored
|
||||
xau_score (and XAU's per-currency score); other currencies remain keyword-scored.`
|
||||
|
||||
---
|
||||
|
||||
## 6. `news_archive.py` changes
|
||||
|
||||
No schema change. `_SOURCE_RANK` (:107) and `_STORED_RANK` (:203-206) already rank `model` = 2.
|
||||
|
||||
- `archive_headlines(headlines, *, teacher_model=None, teacher_prompt_sha1=None, origin="live",
|
||||
model_teacher=None)` — new kwarg, passed through to `build_row`.
|
||||
- `build_row` (:323-354):
|
||||
```python
|
||||
"teacher_model": (teacher_model if source == "llm"
|
||||
else model_teacher if source == "model" else None),
|
||||
"teacher_prompt_sha1": (teacher_prompt_sha1 if source in ("llm", "model") else None),
|
||||
```
|
||||
For a model row `teacher_prompt_sha1` is the DeepSeek prompt the model was distilled from (the same
|
||||
constant the bot already passes), which is the truthful provenance.
|
||||
- Effect: a retrain is visible in the data (`teacher_model` changes), and the UPSERT's
|
||||
`COALESCE(excluded.teacher_model, headlines.teacher_model)` (:253-260) only applies when the new
|
||||
rank ≥ stored rank, so an `llm` row's teacher is never replaced by a model tag.
|
||||
|
||||
---
|
||||
|
||||
## 7. Tests (`test_all.py`)
|
||||
|
||||
The suite is a plain script: `@test(name)` (:51-64) runs each function at definition time and
|
||||
tallies; the file ends with `sys.exit(1 if FAIL > 0 else 0)`. Run: `venv/bin/python test_all.py`.
|
||||
Stubbing patterns in use: instance-level `ne._llm_headline_scores = lambda texts: …` with
|
||||
`lc._client = _StubLLM({...})` and `finally` cleanup (:5196-5215); module-level swap
|
||||
`saved = _ne_mod.get_llm_client; _ne_mod.get_llm_client = stub; … finally restore` (:11391-11424).
|
||||
|
||||
Add, using the same patterns and a fake bundle (`news_model.score_batch` swapped on the module):
|
||||
|
||||
1. **model scores keyword rows only.** Two `NewsSentiment`s, one `score_source="llm"`; after
|
||||
`_apply_model_sentiment`, the llm one is byte-identical, the keyword one has `score_source="model"`,
|
||||
`xau_score` set, `sentiment_by_currency["XAU"]` = score, `["USD"]` unchanged (rule A).
|
||||
2. **overall score rule (B).** `currencies_mentioned=["XAU"]` → `sentiment_score` = model score;
|
||||
`["USD","XAU"]` → `sentiment_score` unchanged; `[]` → model score.
|
||||
3. **relevance zeroing (D).** stub returns `relevant=False, score=-0.18` → `sentiment_by_currency["XAU"]
|
||||
== 0.0`, `xau_score == -0.18`.
|
||||
4. **cache untouched (E).** `_llm_sent_cache` empty before and after.
|
||||
5. **failure isolation.** stub raises → all rows still `keyword`, method returns 0, no exception.
|
||||
`news_model.get_news_model()` returning None → returns 0.
|
||||
6. **primary degrades to DeepSeek.** `get_news_model` → None, `_llm_headline_scores` stubbed → rows
|
||||
end up `llm`.
|
||||
7. **audit sample.** `deepseek_audit_per_day=3`, 10 model rows → exactly 3 sent, displayed objects
|
||||
still `model`, `_pending_audit_rows` has 3 `llm` copies; second call same day sends 0.
|
||||
8. **config.** unknown key in `news_model` → `AppConfigError`; `mode="banana"` → error; section
|
||||
absent → `news_model_mode() == "off"`; `NEWS_MODEL_MODE=off` env beats `"mode": "primary"`.
|
||||
9. **archive.** `build_row` of a `model` headline with `model_teacher="xau_transformer@T"` stores it;
|
||||
an `llm` row stores the DeepSeek teacher; existing rank tests still pass.
|
||||
10. **import hygiene.** update the docstring of the test at :11437-11462 (it says numpy-only); add
|
||||
an assertion that `requirements.txt` lists `onnxruntime` and `tokenizers`; keep its "no torch /
|
||||
transformers" checks.
|
||||
11. **integration, skipped when the bundle is absent** (`os.environ.get("NEWS_MODEL_DIR")` or the
|
||||
configured path missing → print SKIP): load, `score_news(b, "Fed cuts rates, signals more
|
||||
easing")["label"] == "bullish"`, `score_news(b, "Red Lobster endless shrimp losses")["label"] ==
|
||||
"neutral"`, latency of 20 calls < 1 s.
|
||||
|
||||
`test_all.py:3675-3690` loads `config.json.example` for pair output only; adding the section there is
|
||||
safe.
|
||||
|
||||
---
|
||||
|
||||
## 8. Ops checklist → `deploy/README-news-model.md` (new)
|
||||
|
||||
Covers **both** processes: the bot (`telegram_bot.py`) and the dashboard daemon
|
||||
(`deploy/gold-dashboard.service`).
|
||||
|
||||
0. `free -m`. Each model-loading process needs ~0.7 GB headroom. If only the bot should score with
|
||||
the model, add `Environment=NEWS_MODEL_MODE=off` to `deploy/gold-dashboard.service` (the dashboard
|
||||
then keeps using DeepSeek for its headlines; those `llm` rows land in the shared archive and act
|
||||
as free audit labels).
|
||||
1. `cd /home/Forex/ForexBot2 && git pull && venv/bin/pip install -r requirements.txt`.
|
||||
2. `mkdir -p /home/Forex/models/xau_transformer` and copy `model.onnx`, `tokenizer.json`, `meta.json`
|
||||
from the ML-NEWS machine (rsync, ~280 MB). Verify `sha256sum model.onnx` matches the source.
|
||||
Owner: the account the bot runs as; read-only is enough.
|
||||
3. Add the `news_model` section to `config.json` (§4.1) with `"mode": "primary"`. Leave the
|
||||
`deepseek` section unchanged.
|
||||
4. Smoke test without starting the bot:
|
||||
```
|
||||
venv/bin/python -c "import news_model as m; b=m.get_news_model(); print(m.score_news(b,'Gold jumps as dollar slides'))"
|
||||
```
|
||||
Expect `label: bullish`, and the "news model loaded" log line.
|
||||
5. `venv/bin/python test_all.py` — green (integration test runs because the dir exists).
|
||||
6. Restart the bot; restart the dashboard unit. Check:
|
||||
- log: one "news model loaded" line per process, no "news model unavailable";
|
||||
- `top`: RSS ~+0.6 GB per process, idle CPU ~0% between scans (spinning disabled);
|
||||
- archive: new rows have `score_source='model'`, `xau_score` set, `teacher_model`
|
||||
`xau_transformer@…`; DeepSeek daily counter for headlines stays at 0 (audit off).
|
||||
7. Rollback: set `"mode": "fallback"` or `"off"` (or the env var), restart. No data migration needed.
|
||||
8. Retrain cadence: when ML-NEWS ships a new bundle, copy the three files over the old ones and
|
||||
restart both processes. `meta.json.trained_at` changes, so the archive shows which model scored which
|
||||
rows. If `deepseek_audit_per_day` is set, ML-NEWS's `eval/eval_archive.py --scorer transformer` on a
|
||||
copy of the live archive gives the model-vs-DeepSeek agreement on those rows.
|
||||
|
||||
---
|
||||
|
||||
## 9. Acceptance criteria (end-to-end)
|
||||
|
||||
- With `mode="primary"` and DeepSeek enabled, one live news cycle archives only `model` rows (plus
|
||||
≤ `deepseek_audit_per_day` `llm` rows), and the cycle's added latency is < 1 s for ~100 headlines.
|
||||
- With the model directory renamed away, both processes start, log one warning each, and the cycle's
|
||||
rows are `llm` (DeepSeek path), not `keyword`.
|
||||
- With `onnxruntime` uninstalled in a scratch venv, `python -c "import news_engine"` succeeds.
|
||||
- A headline naming USD and XAU ends with `sentiment_by_currency["USD"]` equal to the keyword value
|
||||
and `["XAU"]` equal to the model value.
|
||||
- `mode="off"` reproduces today's archive rows exactly (`llm`/`keyword`, same teacher fields).
|
||||
- `venv/bin/python test_all.py` exits 0.
|
||||
|
||||
## 10. Known limits to state in the README
|
||||
- The model copies DeepSeek's opinion (72% agreement on unseen live days, 72/78 golden); it does not
|
||||
learn from price. DeepSeek itself is ~90% self-consistent.
|
||||
- Gold only. Every other currency's sentiment remains the ~31%-accurate keyword scorer in primary
|
||||
mode, which is what those currencies get today whenever DeepSeek is unavailable.
|
||||
- With audit off, no new DeepSeek labels accrue; the ML-NEWS retrain loop needs either
|
||||
`deepseek_audit_per_day > 0`, the dashboard left on DeepSeek, or a separate labelling run.
|
||||
Reference in New Issue
Block a user