1. A backfill pass silently wiped teacher identity from live rows.
The provenance compare is `>=`, and backfill rows are llm-scored but
carry no teacher_model/teacher_prompt_sha1 (neither source DB recorded
one), so a bare assignment blanked what the live hook had stored.
Re-running the backfill — e.g. after importing the live server's
signals.db, which is the next planned step — would have erased it from
every live row. COALESCE now keeps a stored identity when the incoming
one is NULL, while a genuinely newer teacher still replaces it.
Timestamps clamp for the same reason: backfill stamps historical
values, so first_seen only moves earlier and last_seen only later.
2. A retried batch could be applied twice. executemany does not roll
itself back, so a batch that half-landed and then hit a lock was
re-applied whole by _with_lock_retry, double-counting seen_count on
the rows that already landed. Writes now run in an explicit BEGIN
IMMEDIATE with a rollback, which also takes the write lock up front so
contention fails before any row is applied.
3. The archive hook could break a news fetch. archive_headlines swallows
its own failures, but its ARGUMENTS are evaluated first, so a raising
get_llm_client() escaped that guard. Guarded at the call site too:
archiving is a side effect and must never cost us the headlines we
just fetched.
Both 1 and 2 were verified to reproduce against 301bf90 before fixing
(teacher_model -> None; seen_count 3 instead of 2 on the rows that
landed pre-failure) and are now regression tests.
Also fixed two weak tests of my own: the disabled-by-config test never
pointed NEWS_ARCHIVE_PATH at its temp dir, so its assertion passed
regardless of behaviour and the write went to the real archive; and the
neutral-headline test asserted the headline was archived without showing
it is dropped downstream, which is the entire reason the hook sits where
it does. Positional binding of the provenance rank is now asserted rather
than trusted, since a miscount would shift every parameter silently.
483 -> 486 tests.
Claude-Session: https://claude.ai/code/session_01Bzv4ETVapFNauFDQht7QtA
Headlines only ever survived as a side effect of signal logging, and
logs/signals.db was rotated around 2026-09-04 — destroying the 3-week,
4,054-headline corpus news_sentiment_value_2026-09-04.md was written on.
logs/ is gitignored, so git could not recover it and nobody noticed for
four days. There is no way to backfill a fetch window once it passes.
news_archive.py writes to its OWN database (logs/newsdata.db) so a
signal-log rotation cannot take the news with it. One row per distinct
headline (sha256 key) with the text, description, link, publisher,
publish time, the score, and — new — which scorer produced that score.
Provenance is the load-bearing part. DeepSeek scores 78/78 on the
2026-08-06 golden set; the keyword scorer it overrides scores 24/78
(31%), and that is what silently stands on no key, disabled, daily cap,
timeout, HTTP/parse failure, or the >25-new-headlines-per-cycle batch
overflow. Until now both landed in the same column, indistinguishable,
so a model trained on the mix would inherit the keyword scorer's errors.
NewsSentiment.score_source now records it, mirrored into signals.db via
signal_log._headline_to_dict, and teacher_model/teacher_prompt_sha1 keep
labels made under a changed prompt from being pooled with older ones.
Writes are one atomic UPSERT per row, upgrade-only: an llm score always
replaces a keyword one (the batch-overflow tail gets labelled on the
next window in a long-running process — verified 13/13), a keyword score
never overwrites an llm one, and re-seeing a headline always counts. A
read-then-write pair lost increments when the bot, the dashboard
publisher and the tests first saw a headline together, and the WAL
switch itself needed _with_lock_retry; both are regression-tested with
three concurrent writers.
The hook sits at the end of _fetch_news_sentiment_raw, BEFORE
_is_neutral_headline runs in get_fundamental_context. Archiving
downstream of that ±0.1 filter is why signals.db holds almost no neutral
headlines and would have taught a trained scorer that "neutral" barely
exists.
scripts/backfill_news_archive.py seeds it from both DB shapes on disk,
read-only and idempotent, recomputing provenance by comparing each
stored score against the keyword scorer (neither source recorded it):
1,016 distinct headlines, 846 DeepSeek-labelled, growing ~1,400/week.
Nothing in the signal core changed. Both news gating flags stay False,
no gate reads any of this, and the backtester never builds a NewsEngine
so replays never archive (test-locked). 468 -> 483 tests, including a
new guard that AST-walks live code for sklearn/scipy/lightgbm imports —
that check did not exist before.
Do not train yet: ~800 labels gives a holdout too small to grade
(SE +/-0.032). Target is ~4,000 deduplicated headlines over >=3 weeks.
Claude-Session: https://claude.ai/code/session_01Bzv4ETVapFNauFDQht7QtA
Harwinder Sekhon