news archive: keep every headline + its DeepSeek score in its own DB

Headlines only ever survived as a side effect of signal logging, and
logs/signals.db was rotated around 2026-09-04 — destroying the 3-week,
4,054-headline corpus news_sentiment_value_2026-09-04.md was written on.
logs/ is gitignored, so git could not recover it and nobody noticed for
four days. There is no way to backfill a fetch window once it passes.

news_archive.py writes to its OWN database (logs/newsdata.db) so a
signal-log rotation cannot take the news with it. One row per distinct
headline (sha256 key) with the text, description, link, publisher,
publish time, the score, and — new — which scorer produced that score.

Provenance is the load-bearing part. DeepSeek scores 78/78 on the
2026-08-06 golden set; the keyword scorer it overrides scores 24/78
(31%), and that is what silently stands on no key, disabled, daily cap,
timeout, HTTP/parse failure, or the >25-new-headlines-per-cycle batch
overflow. Until now both landed in the same column, indistinguishable,
so a model trained on the mix would inherit the keyword scorer's errors.
NewsSentiment.score_source now records it, mirrored into signals.db via
signal_log._headline_to_dict, and teacher_model/teacher_prompt_sha1 keep
labels made under a changed prompt from being pooled with older ones.

Writes are one atomic UPSERT per row, upgrade-only: an llm score always
replaces a keyword one (the batch-overflow tail gets labelled on the
next window in a long-running process — verified 13/13), a keyword score
never overwrites an llm one, and re-seeing a headline always counts. A
read-then-write pair lost increments when the bot, the dashboard
publisher and the tests first saw a headline together, and the WAL
switch itself needed _with_lock_retry; both are regression-tested with
three concurrent writers.

The hook sits at the end of _fetch_news_sentiment_raw, BEFORE
_is_neutral_headline runs in get_fundamental_context. Archiving
downstream of that ±0.1 filter is why signals.db holds almost no neutral
headlines and would have taught a trained scorer that "neutral" barely
exists.

scripts/backfill_news_archive.py seeds it from both DB shapes on disk,
read-only and idempotent, recomputing provenance by comparing each
stored score against the keyword scorer (neither source recorded it):
1,016 distinct headlines, 846 DeepSeek-labelled, growing ~1,400/week.

Nothing in the signal core changed. Both news gating flags stay False,
no gate reads any of this, and the backtester never builds a NewsEngine
so replays never archive (test-locked). 468 -> 483 tests, including a
new guard that AST-walks live code for sklearn/scipy/lightgbm imports —
that check did not exist before.

Do not train yet: ~800 labels gives a holdout too small to grade
(SE +/-0.032). Target is ~4,000 deduplicated headlines over >=3 weeks.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Bzv4ETVapFNauFDQht7QtA

news_engine.py

@@ -23,6 +23,7 @@ from typing import Dict, List, Optional, Tuple
 
 import requests
 
+import news_archive
 from config import NewsConfig
 from llm_client import get_llm_client
 
@@ -102,6 +103,13 @@ Examples:
 """
 
 
+#: Identity of the labelling prompt, stored on every archived LLM score.
+#: If the prompt changes, scores before and after are answers to different
+#: questions and must not be pooled into one training set.
+_LLM_SENTIMENT_PROMPT_SHA1 = hashlib.sha1(
+    _LLM_SENTIMENT_SYSTEM.encode("utf-8")).hexdigest()
+
+
 _RELEASE_VALUE_RE = re.compile(r"(-?\d+(?:\.\d+)?)\s*([kmbt])?", re.IGNORECASE)
 _RELEASE_MULTIPLIERS = {"k": 1e3, "m": 1e6, "b": 1e9, "t": 1e12}
 
@@ -212,6 +220,12 @@ class NewsSentiment:
     link: Optional[str] = None         # article URL (Google News = redirect)
     description: Optional[str] = None  # plain text, tags stripped, ≤500 chars
     publisher: Optional[str] = None    # Google News <source> text, else None
+    # Which scorer produced sentiment_score / sentiment_by_currency:
+    # "keyword" (the built-in scorer, ~31% on the 2026-08-06 golden set),
+    # "llm" (DeepSeek, 78/78 on the same set) or "model" (a trained local
+    # artifact). Without it the two are indistinguishable in the archive
+    # and a model trained on the mix inherits the keyword scorer's errors.
+    score_source: str = "keyword"
 
 
 @dataclass
@@ -902,6 +916,16 @@ class NewsEngine:
         except Exception as e:
             logger.debug(f"LLM sentiment failed, keyword scores stand: {e}")
 
+        # Archive BEFORE any downstream filtering. get_fundamental_context
+        # drops every headline inside ±NEUTRAL_HEADLINE_THRESHOLD, so a
+        # corpus built from what the context carries holds almost no
+        # genuinely neutral headlines and would teach a trained scorer
+        # that "neutral" barely exists. Never raises.
+        news_archive.archive_headlines(
+            headlines,
+            teacher_model=getattr(get_llm_client(), "_model", None),
+            teacher_prompt_sha1=_LLM_SENTIMENT_PROMPT_SHA1)
+
         return headlines
 
     def _analyze_headline_sentiment(
@@ -1003,6 +1027,7 @@ class NewsEngine:
                 cur: (xau if cur == "XAU" else overall)
                 for cur in h.currencies_mentioned
             }
+            h.score_source = "llm"
 
     def _llm_event_direction(self, event: EconomicEvent) -> Optional[str]:
         """Gold direction for a released surprise, e.g. "bearish: strong