Harwinder Sekhon
301bf90f86
Headlines only ever survived as a side effect of signal logging, and logs/signals.db was rotated around 2026-09-04 — destroying the 3-week, 4,054-headline corpus news_sentiment_value_2026-09-04.md was written on. logs/ is gitignored, so git could not recover it and nobody noticed for four days. There is no way to backfill a fetch window once it passes. news_archive.py writes to its OWN database (logs/newsdata.db) so a signal-log rotation cannot take the news with it. One row per distinct headline (sha256 key) with the text, description, link, publisher, publish time, the score, and — new — which scorer produced that score. Provenance is the load-bearing part. DeepSeek scores 78/78 on the 2026-08-06 golden set; the keyword scorer it overrides scores 24/78 (31%), and that is what silently stands on no key, disabled, daily cap, timeout, HTTP/parse failure, or the >25-new-headlines-per-cycle batch overflow. Until now both landed in the same column, indistinguishable, so a model trained on the mix would inherit the keyword scorer's errors. NewsSentiment.score_source now records it, mirrored into signals.db via signal_log._headline_to_dict, and teacher_model/teacher_prompt_sha1 keep labels made under a changed prompt from being pooled with older ones. Writes are one atomic UPSERT per row, upgrade-only: an llm score always replaces a keyword one (the batch-overflow tail gets labelled on the next window in a long-running process — verified 13/13), a keyword score never overwrites an llm one, and re-seeing a headline always counts. A read-then-write pair lost increments when the bot, the dashboard publisher and the tests first saw a headline together, and the WAL switch itself needed _with_lock_retry; both are regression-tested with three concurrent writers. The hook sits at the end of _fetch_news_sentiment_raw, BEFORE _is_neutral_headline runs in get_fundamental_context. Archiving downstream of that ±0.1 filter is why signals.db holds almost no neutral headlines and would have taught a trained scorer that "neutral" barely exists. scripts/backfill_news_archive.py seeds it from both DB shapes on disk, read-only and idempotent, recomputing provenance by comparing each stored score against the keyword scorer (neither source recorded it): 1,016 distinct headlines, 846 DeepSeek-labelled, growing ~1,400/week. Nothing in the signal core changed. Both news gating flags stay False, no gate reads any of this, and the backtester never builds a NewsEngine so replays never archive (test-locked). 468 -> 483 tests, including a new guard that AST-walks live code for sklearn/scipy/lightgbm imports — that check did not exist before. Do not train yet: ~800 labels gives a holdout too small to grade (SE +/-0.032). Target is ~4,000 deduplicated headlines over >=3 weeks. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Bzv4ETVapFNauFDQht7QtA
75 lines
1.7 KiB
JSON
75 lines
1.7 KiB
JSON
{
|
|
"telegram": {
|
|
"token": "123456789:REPLACE_WITH_YOUR_BOT_TOKEN",
|
|
"chat_id": "REPLACE_WITH_OWNER_CHAT_ID",
|
|
"channel_id": "-100REPLACE_WITH_CHANNEL_ID_OR_EMPTY_STRING",
|
|
"alert_min_quality": "C"
|
|
},
|
|
"data_providers": {
|
|
"active": "oanda",
|
|
"streaming": "tiingo",
|
|
"twelve_data": {
|
|
"api_key": "REPLACE_WITH_YOUR_TWELVE_DATA_API_KEY"
|
|
},
|
|
"oanda": {},
|
|
"tiingo": {
|
|
"api_key": "REPLACE_WITH_YOUR_TIINGO_API_KEY_OR_EMPTY"
|
|
}
|
|
},
|
|
"scanning": {
|
|
"pairs": ["XAU/USD"],
|
|
"timeframes": ["M5", "M15", "H1", "H4"]
|
|
},
|
|
"pairs": {
|
|
"XAU/USD": {
|
|
"blocked_hours_utc": [0, 1, 2, 10, 11, 13, 16, 21]
|
|
}
|
|
},
|
|
"oanda": {
|
|
"api_token": "REPLACE_WITH_API_TOKEN",
|
|
"base_url": "https://api-fxpractice.oanda.com"
|
|
},
|
|
"deepseek": {
|
|
"api_key": "REPLACE_WITH_YOUR_DEEPSEEK_API_KEY_OR_EMPTY",
|
|
"model": "deepseek-v4-flash",
|
|
"enabled": true,
|
|
"timeout_seconds": 12,
|
|
"daily_request_cap": 500
|
|
},
|
|
"agent_panel": {
|
|
"enabled": false,
|
|
"telegram": true,
|
|
"dashboard": false,
|
|
"min_quality": "B",
|
|
"max_panels_per_day": 20,
|
|
"reuse_ttl_seconds": 3600,
|
|
"candles_enabled": true,
|
|
"candle_bars_primary": 20,
|
|
"candle_bars_confirm": 10,
|
|
"verdict_source": "macro"
|
|
},
|
|
"market_outlook": {
|
|
"enabled": false,
|
|
"telegram": false,
|
|
"max_per_day": 12,
|
|
"candles_bars": 20
|
|
},
|
|
"violent_watch": {
|
|
"enabled": false,
|
|
"alert_line": true,
|
|
"headsup": true,
|
|
"max_headsup_per_day": 12,
|
|
"headsup_min_level": "hot"
|
|
},
|
|
"signal_log": {
|
|
"enabled": true,
|
|
"path": "logs/signals.db",
|
|
"log_blocked_scans": true
|
|
},
|
|
"news_archive": {
|
|
"enabled": true,
|
|
"path": "logs/newsdata.db"
|
|
},
|
|
"log_level": "INFO"
|
|
}
|