news archive: keep every headline + its DeepSeek score in its own DB

Headlines only ever survived as a side effect of signal logging, and
logs/signals.db was rotated around 2026-09-04 — destroying the 3-week,
4,054-headline corpus news_sentiment_value_2026-09-04.md was written on.
logs/ is gitignored, so git could not recover it and nobody noticed for
four days. There is no way to backfill a fetch window once it passes.

news_archive.py writes to its OWN database (logs/newsdata.db) so a
signal-log rotation cannot take the news with it. One row per distinct
headline (sha256 key) with the text, description, link, publisher,
publish time, the score, and — new — which scorer produced that score.

Provenance is the load-bearing part. DeepSeek scores 78/78 on the
2026-08-06 golden set; the keyword scorer it overrides scores 24/78
(31%), and that is what silently stands on no key, disabled, daily cap,
timeout, HTTP/parse failure, or the >25-new-headlines-per-cycle batch
overflow. Until now both landed in the same column, indistinguishable,
so a model trained on the mix would inherit the keyword scorer's errors.
NewsSentiment.score_source now records it, mirrored into signals.db via
signal_log._headline_to_dict, and teacher_model/teacher_prompt_sha1 keep
labels made under a changed prompt from being pooled with older ones.

Writes are one atomic UPSERT per row, upgrade-only: an llm score always
replaces a keyword one (the batch-overflow tail gets labelled on the
next window in a long-running process — verified 13/13), a keyword score
never overwrites an llm one, and re-seeing a headline always counts. A
read-then-write pair lost increments when the bot, the dashboard
publisher and the tests first saw a headline together, and the WAL
switch itself needed _with_lock_retry; both are regression-tested with
three concurrent writers.

The hook sits at the end of _fetch_news_sentiment_raw, BEFORE
_is_neutral_headline runs in get_fundamental_context. Archiving
downstream of that ±0.1 filter is why signals.db holds almost no neutral
headlines and would have taught a trained scorer that "neutral" barely
exists.

scripts/backfill_news_archive.py seeds it from both DB shapes on disk,
read-only and idempotent, recomputing provenance by comparing each
stored score against the keyword scorer (neither source recorded it):
1,016 distinct headlines, 846 DeepSeek-labelled, growing ~1,400/week.

Nothing in the signal core changed. Both news gating flags stay False,
no gate reads any of this, and the backtester never builds a NewsEngine
so replays never archive (test-locked). 468 -> 483 tests, including a
new guard that AST-walks live code for sklearn/scipy/lightgbm imports —
that check did not exist before.

Do not train yet: ~800 labels gives a holdout too small to grade
(SE +/-0.032). Target is ~4,000 deduplicated headlines over >=3 weeks.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Bzv4ETVapFNauFDQht7QtA

app_config.py

@@ -85,7 +85,7 @@ _SCANNABLE_TFS = ("M5", "M15", "H1", "H4")
 
 _KNOWN_TOP = {"telegram", "data_providers", "scanning", "pairs", "log_level",
                "oanda", "deepseek", "agent_panel", "market_outlook",
-               "violent_watch", "signal_log"}
+               "violent_watch", "signal_log", "news_archive"}
 _KNOWN_OANDA = {
     # "account_id" is TOLERATED but ignored: it belonged to the auto-
     # execution daemon removed 2026-09-03 (last commit 31002cb). The candle
@@ -106,6 +106,7 @@ _KNOWN_MARKET_OUTLOOK = {"enabled", "telegram", "max_per_day", "candles_bars"}
 _KNOWN_VIOLENT_WATCH = {"enabled", "alert_line", "headsup",
                         "max_headsup_per_day", "headsup_min_level"}
 _KNOWN_SIGNAL_LOG = {"enabled", "path", "log_blocked_scans"}
+_KNOWN_NEWS_ARCHIVE = {"enabled", "path"}
 _KNOWN_TELEGRAM = {"token", "chat_id", "channel_id", "alert_min_quality"}
 _KNOWN_SCANNING = {"pairs", "timeframes"}
 _KNOWN_PAIR_KEYS = {"blocked_hours_utc"}
@@ -153,6 +154,7 @@ class AppConfig:
     violent_watch_settings: Dict[str, Any] = field(
         default_factory=lambda: dict(_VIOLENT_WATCH_DEFAULTS))
     signal_log_settings: Dict[str, Any] = field(default_factory=dict)
+    news_archive_settings: Dict[str, Any] = field(default_factory=dict)
     log_level: str = "INFO"
 
 
@@ -387,6 +389,17 @@ def load_app_config(path: str = CONFIG_PATH) -> AppConfig:
             "config.json: signal_log.log_blocked_scans must be true or false")
     app.signal_log_settings = dict(slog)
 
+    narch = raw.get("news_archive", {})
+    _check_keys(narch, _KNOWN_NEWS_ARCHIVE, '"news_archive"')
+    if "enabled" in narch and not isinstance(narch["enabled"], bool):
+        raise AppConfigError(
+            "config.json: news_archive.enabled must be true or false")
+    if "path" in narch and (not isinstance(narch["path"], str)
+                            or not narch["path"]):
+        raise AppConfigError(
+            "config.json: news_archive.path must be a non-empty string")
+    app.news_archive_settings = dict(narch)
+
     level = raw.get("log_level", app.log_level)
     if level not in _VALID_LOG_LEVELS:
         raise AppConfigError(
@@ -538,6 +551,34 @@ def signal_log_path(app: Optional[AppConfig] = None) -> str:
     return path
 
 
+# ── News-archive resolution helpers (env → config.json → default) ────
+
+def news_archive_enabled(app: Optional[AppConfig] = None) -> bool:
+    """Default ON — the headline corpus is training data for a local
+    scorer and cannot be backfilled once a fetch window has passed. It
+    lives in its own DB so a signals.db rotation cannot take it with it
+    (which is how three weeks of news were lost on 2026-09-04)."""
+    env = os.environ.get("NEWS_ARCHIVE_ENABLED")
+    if env:
+        return env.strip().lower() in ("1", "true", "yes", "on")
+    app = app or get_app_config()
+    return bool(app.news_archive_settings.get("enabled", True))
+
+
+def news_archive_path(app: Optional[AppConfig] = None) -> str:
+    """Absolute path to the SQLite headline corpus. A relative config
+    value resolves against the repo root so CWD never matters."""
+    env = os.environ.get("NEWS_ARCHIVE_PATH")
+    if env:
+        path = env
+    else:
+        app = app or get_app_config()
+        path = str(app.news_archive_settings.get("path", "logs/newsdata.db"))
+    if not os.path.isabs(path):
+        path = os.path.join(os.path.dirname(os.path.abspath(__file__)), path)
+    return path
+
+
 # ── PlatformConfig overlay ────────────────────────────────────────────
 
 def apply_pair_overrides(