The owner's DeBERTa-v3-small two-head ONNX model (golden set 72/78, ~72%
agreement with DeepSeek on unseen days) replaces DeepSeek for headline
scoring. DeepSeek keeps event direction, the summary and the dashboard
endpoint.
- news_model.py: vendored ML-NEWS runtime (lazy onnxruntime/tokenizers,
so the bot starts without them) + load-once, never-raising singleton.
- news_engine: _enrich_headline_scores mode switch (off/fallback/primary),
_apply_model_sentiment (gold head only: xau_score + XAU entry, never
USD; not-relevant shows 0.0; never touches _llm_sent_cache),
_audit_with_deepseek (archive-only labels, deepseek_audit_per_day).
- Owner decision: primary mode writes nothing to logs/newsdata.db except
audit copies — the model's output is not training data.
- app_config: news_model section (path/threads/mode/deepseek_audit_per_day)
+ NEWS_MODEL_MODE env override; validated at boot.
- news_archive: model rows carry teacher_model xau_transformer@<trained_at>.
- deploy/README-news-model.md (ops for bot + dashboard daemon), unit gets a
commented NEWS_MODEL_MODE=off line; requirements + onnxruntime, tokenizers.
- test_all.py Module 38 (15 tests, one against the real bundle). Suite 581.
- CLAUDE.md, spec status note, handoff marked implemented.
Books unaffected: news never enters signals; no signal file touched.
Agent panel v2: personas return p_win anchored to the book base rates with a
code-computed decision context; verdict = equal-weight mean (verdict_source
"mean"); only the macro persona runs live (agent_panel.personas ["macro"]);
prompt version and exact payloads logged; offline replay harness
scripts/panel_replay.py; base-rate table agent_panel_base_rates.json
(scripts/build_panel_base_rates.py). Backtester `strategy` column now carries
the generator's own stamp (label-only; five reference books re-pinned, all
62 other columns identical).
News corpus (after evaluating the owner's ML-NEWS scikit model against
DeepSeek: golden set 39/78 vs 78/78, out-of-sample agreement 64% against a
90% self-agreement ceiling; verdict: fallback only, not a replacement):
- NewsSentiment.xau_score + archive column xau_score keep DeepSeek's gold
score for every headline (930 of the first 1,546 DeepSeek rows had none);
an existing newsdata.db gains the column in place on first open
- headlines naming no tracked currency are scored and archived
(currencies_json []) but never returned to the platform
- _apply_llm_sentiment sends every new headline in its cycle, 25 per call;
a failed call ends the cycle's sending
Dashboard publisher never writes a database: main() calls
app_config.set_db_writes_for_process(False), honoured by signal_log_enabled()
and news_archive_enabled() (owner decision: the bot is the single writer).
Test suite: the hermetic app config now points both writers at a
suite-private temp dir — the RSS fixture tests were writing fixture
headlines into the real logs/newsdata.db — with a Module 32 test pinning it.
529 tests.
1. A backfill pass silently wiped teacher identity from live rows.
The provenance compare is `>=`, and backfill rows are llm-scored but
carry no teacher_model/teacher_prompt_sha1 (neither source DB recorded
one), so a bare assignment blanked what the live hook had stored.
Re-running the backfill — e.g. after importing the live server's
signals.db, which is the next planned step — would have erased it from
every live row. COALESCE now keeps a stored identity when the incoming
one is NULL, while a genuinely newer teacher still replaces it.
Timestamps clamp for the same reason: backfill stamps historical
values, so first_seen only moves earlier and last_seen only later.
2. A retried batch could be applied twice. executemany does not roll
itself back, so a batch that half-landed and then hit a lock was
re-applied whole by _with_lock_retry, double-counting seen_count on
the rows that already landed. Writes now run in an explicit BEGIN
IMMEDIATE with a rollback, which also takes the write lock up front so
contention fails before any row is applied.
3. The archive hook could break a news fetch. archive_headlines swallows
its own failures, but its ARGUMENTS are evaluated first, so a raising
get_llm_client() escaped that guard. Guarded at the call site too:
archiving is a side effect and must never cost us the headlines we
just fetched.
Both 1 and 2 were verified to reproduce against 301bf90 before fixing
(teacher_model -> None; seen_count 3 instead of 2 on the rows that
landed pre-failure) and are now regression tests.
Also fixed two weak tests of my own: the disabled-by-config test never
pointed NEWS_ARCHIVE_PATH at its temp dir, so its assertion passed
regardless of behaviour and the write went to the real archive; and the
neutral-headline test asserted the headline was archived without showing
it is dropped downstream, which is the entire reason the hook sits where
it does. Positional binding of the provenance rank is now asserted rather
than trusted, since a miscount would shift every parameter silently.
483 -> 486 tests.
Claude-Session: https://claude.ai/code/session_01Bzv4ETVapFNauFDQht7QtA
Headlines only ever survived as a side effect of signal logging, and
logs/signals.db was rotated around 2026-09-04 — destroying the 3-week,
4,054-headline corpus news_sentiment_value_2026-09-04.md was written on.
logs/ is gitignored, so git could not recover it and nobody noticed for
four days. There is no way to backfill a fetch window once it passes.
news_archive.py writes to its OWN database (logs/newsdata.db) so a
signal-log rotation cannot take the news with it. One row per distinct
headline (sha256 key) with the text, description, link, publisher,
publish time, the score, and — new — which scorer produced that score.
Provenance is the load-bearing part. DeepSeek scores 78/78 on the
2026-08-06 golden set; the keyword scorer it overrides scores 24/78
(31%), and that is what silently stands on no key, disabled, daily cap,
timeout, HTTP/parse failure, or the >25-new-headlines-per-cycle batch
overflow. Until now both landed in the same column, indistinguishable,
so a model trained on the mix would inherit the keyword scorer's errors.
NewsSentiment.score_source now records it, mirrored into signals.db via
signal_log._headline_to_dict, and teacher_model/teacher_prompt_sha1 keep
labels made under a changed prompt from being pooled with older ones.
Writes are one atomic UPSERT per row, upgrade-only: an llm score always
replaces a keyword one (the batch-overflow tail gets labelled on the
next window in a long-running process — verified 13/13), a keyword score
never overwrites an llm one, and re-seeing a headline always counts. A
read-then-write pair lost increments when the bot, the dashboard
publisher and the tests first saw a headline together, and the WAL
switch itself needed _with_lock_retry; both are regression-tested with
three concurrent writers.
The hook sits at the end of _fetch_news_sentiment_raw, BEFORE
_is_neutral_headline runs in get_fundamental_context. Archiving
downstream of that ±0.1 filter is why signals.db holds almost no neutral
headlines and would have taught a trained scorer that "neutral" barely
exists.
scripts/backfill_news_archive.py seeds it from both DB shapes on disk,
read-only and idempotent, recomputing provenance by comparing each
stored score against the keyword scorer (neither source recorded it):
1,016 distinct headlines, 846 DeepSeek-labelled, growing ~1,400/week.
Nothing in the signal core changed. Both news gating flags stay False,
no gate reads any of this, and the backtester never builds a NewsEngine
so replays never archive (test-locked). 468 -> 483 tests, including a
new guard that AST-walks live code for sklearn/scipy/lightgbm imports —
that check did not exist before.
Do not train yet: ~800 labels gives a holdout too small to grade
(SE +/-0.032). Target is ~4,000 deduplicated headlines over >=3 weeks.
Claude-Session: https://claude.ai/code/session_01Bzv4ETVapFNauFDQht7QtA
Harwinder Sekhon