The owner's DeBERTa-v3-small two-head ONNX model (golden set 72/78, ~72%
agreement with DeepSeek on unseen days) replaces DeepSeek for headline
scoring. DeepSeek keeps event direction, the summary and the dashboard
endpoint.
- news_model.py: vendored ML-NEWS runtime (lazy onnxruntime/tokenizers,
so the bot starts without them) + load-once, never-raising singleton.
- news_engine: _enrich_headline_scores mode switch (off/fallback/primary),
_apply_model_sentiment (gold head only: xau_score + XAU entry, never
USD; not-relevant shows 0.0; never touches _llm_sent_cache),
_audit_with_deepseek (archive-only labels, deepseek_audit_per_day).
- Owner decision: primary mode writes nothing to logs/newsdata.db except
audit copies — the model's output is not training data.
- app_config: news_model section (path/threads/mode/deepseek_audit_per_day)
+ NEWS_MODEL_MODE env override; validated at boot.
- news_archive: model rows carry teacher_model xau_transformer@<trained_at>.
- deploy/README-news-model.md (ops for bot + dashboard daemon), unit gets a
commented NEWS_MODEL_MODE=off line; requirements + onnxruntime, tokenizers.
- test_all.py Module 38 (15 tests, one against the real bundle). Suite 581.
- CLAUDE.md, spec status note, handoff marked implemented.
Books unaffected: news never enters signals; no signal file touched.
Agent panel v2: personas return p_win anchored to the book base rates with a
code-computed decision context; verdict = equal-weight mean (verdict_source
"mean"); only the macro persona runs live (agent_panel.personas ["macro"]);
prompt version and exact payloads logged; offline replay harness
scripts/panel_replay.py; base-rate table agent_panel_base_rates.json
(scripts/build_panel_base_rates.py). Backtester `strategy` column now carries
the generator's own stamp (label-only; five reference books re-pinned, all
62 other columns identical).
News corpus (after evaluating the owner's ML-NEWS scikit model against
DeepSeek: golden set 39/78 vs 78/78, out-of-sample agreement 64% against a
90% self-agreement ceiling; verdict: fallback only, not a replacement):
- NewsSentiment.xau_score + archive column xau_score keep DeepSeek's gold
score for every headline (930 of the first 1,546 DeepSeek rows had none);
an existing newsdata.db gains the column in place on first open
- headlines naming no tracked currency are scored and archived
(currencies_json []) but never returned to the platform
- _apply_llm_sentiment sends every new headline in its cycle, 25 per call;
a failed call ends the cycle's sending
Dashboard publisher never writes a database: main() calls
app_config.set_db_writes_for_process(False), honoured by signal_log_enabled()
and news_archive_enabled() (owner decision: the bot is the single writer).
Test suite: the hermetic app config now points both writers at a
suite-private temp dir — the RSS fixture tests were writing fixture
headlines into the real logs/newsdata.db — with a Module 32 test pinning it.
529 tests.
Owner decision 2026-09-15. Gone: market_outlook.py, the post-H1-scan Telegram
outlook DM, main.py --outlook, the outlooks table DDL/migration/writer in
signal_log.py, the market_outlook config block + MARKET_OUTLOOK_ENABLED, and
test Module 26. Last commit containing the feature: 2836024.
- app_config: "market_outlook" stays in _KNOWN_TOP as a TOLERATED, ignored
legacy key (oanda.account_id precedent) with one warning, so a live
config.json that still carries the block keeps booting.
- signal_log: existing DBs keep their orphan outlooks table and rows (no
DROP TABLE); fresh DBs never create it.
- tests: Module 26's 4-thread concurrent legacy-DB migration test (the
suite's only one) moved to Module 31; new test for the tolerated block.
519 -> 509 tests, 34 -> 33 modules, 0 failed.
- docs: CLAUDE.md usage/config/file-map lines and a dated state note.
Verification: W2 books (2024-09-05 -> 2026-09-05, $0.65, NumPy CPU, 0 cache
fallbacks) byte-identical before/after — M5 29120a1ee32979431d062f210d41a9d1
(781 / +146.07R / PF 1.52), M15 d185f099482a07f29502cb5a371a9bbf
(931 / +146.14R / PF 1.43).
Headlines only ever survived as a side effect of signal logging, and
logs/signals.db was rotated around 2026-09-04 — destroying the 3-week,
4,054-headline corpus news_sentiment_value_2026-09-04.md was written on.
logs/ is gitignored, so git could not recover it and nobody noticed for
four days. There is no way to backfill a fetch window once it passes.
news_archive.py writes to its OWN database (logs/newsdata.db) so a
signal-log rotation cannot take the news with it. One row per distinct
headline (sha256 key) with the text, description, link, publisher,
publish time, the score, and — new — which scorer produced that score.
Provenance is the load-bearing part. DeepSeek scores 78/78 on the
2026-08-06 golden set; the keyword scorer it overrides scores 24/78
(31%), and that is what silently stands on no key, disabled, daily cap,
timeout, HTTP/parse failure, or the >25-new-headlines-per-cycle batch
overflow. Until now both landed in the same column, indistinguishable,
so a model trained on the mix would inherit the keyword scorer's errors.
NewsSentiment.score_source now records it, mirrored into signals.db via
signal_log._headline_to_dict, and teacher_model/teacher_prompt_sha1 keep
labels made under a changed prompt from being pooled with older ones.
Writes are one atomic UPSERT per row, upgrade-only: an llm score always
replaces a keyword one (the batch-overflow tail gets labelled on the
next window in a long-running process — verified 13/13), a keyword score
never overwrites an llm one, and re-seeing a headline always counts. A
read-then-write pair lost increments when the bot, the dashboard
publisher and the tests first saw a headline together, and the WAL
switch itself needed _with_lock_retry; both are regression-tested with
three concurrent writers.
The hook sits at the end of _fetch_news_sentiment_raw, BEFORE
_is_neutral_headline runs in get_fundamental_context. Archiving
downstream of that ±0.1 filter is why signals.db holds almost no neutral
headlines and would have taught a trained scorer that "neutral" barely
exists.
scripts/backfill_news_archive.py seeds it from both DB shapes on disk,
read-only and idempotent, recomputing provenance by comparing each
stored score against the keyword scorer (neither source recorded it):
1,016 distinct headlines, 846 DeepSeek-labelled, growing ~1,400/week.
Nothing in the signal core changed. Both news gating flags stay False,
no gate reads any of this, and the backtester never builds a NewsEngine
so replays never archive (test-locked). 468 -> 483 tests, including a
new guard that AST-walks live code for sklearn/scipy/lightgbm imports —
that check did not exist before.
Do not train yet: ~800 labels gives a holdout too small to grade
(SE +/-0.032). Target is ~4,000 deduplicated headlines over >=3 weeks.
Claude-Session: https://claude.ai/code/session_01Bzv4ETVapFNauFDQht7QtA
Removed (all recoverable from git history):
- llm_provider_switch_plan.txt: 2026-08-08 DeepSeek/Anthropic multi-provider plan that
was never implemented (llm_client.py is DeepSeek-only, no `anthropic` anywhere)
- docs/superpowers/specs/2026-07-21-centralized-config-design.md: spec of the
config.json migration that merged the same day; its plan sibling was already
deleted in b936d44. app_config's docstring pointer goes with it.
- scripts/compare_oanda_tiingo.py: Phase 0 recent-tail smoke test, superseded by the
Phase 1/2 study scripts (result recorded in the test plan)
- backtest/research/fetch_oanda_m15.py + oanda_td_divergence.py + its by-hour CSV:
the 2026-07-24 Oanda-vs-Twelve-Data study, superseded by the 2026-08-14 feed
divergence report (which keeps a one-line citation)
- backtest/research/burst_profile.py and hour_analysis.py: superseded by burst_map.py
and book_report.py
- backtest/research/panel_payload_drytest_2026-08-15.py: one-off live DeepSeek probe;
its .jsonl evidence stays
Added to git: the three 2026-08-14 research reports (feed divergence, re-pin
verification, signal parity), their CSV, and docs/data_source_change_test_plan.md -
CLAUDE.md cites them as standing evidence and their driver scripts were already
committed. The test plan no longer refers to auto-execution.
Hygiene:
- .gitignore: drop oanda_state.json (daemon removed 2026-09-03) and the vanished
backtest/trailgrid dirs; ignore .coverage
- docs/architecture_diagram.py only writes the PNG when run as a script (it used
to plot on import); test box refreshed to 461 tests / 31 modules; PNG regenerated
(the committed one still drew the removed daemon)
- backtest/instructions_for_backtest.txt: manual-journal head replaced by the
actual reporting checklist; the 2026-09-06 addendum is unchanged
- doc drift: README/publisher list all six published files; news_summary.php says
4 h (not 24 h) and drops the pre-id sha1 fallback whose window closed;
analyze_signal_log lists six lenses and --scan-outcomes; agent_panel cost model
says 4 requests by default; tiingo comment cites the NY-anchored weekend;
Module 21 title no longer says "execution lifecycle"
Suite: 461 passed, 0 failed.
Claude-Session: https://claude.ai/code/session_0156YhogLytzj9P2iH1skGbL
- telegram_bot._format_scan_summary: multi-pair "auto-scan summary" leftover;
production sends per-report format_signal_message. Its only caller was a test.
- telegram_bot.scan_keyboard: thin wrapper only a test called; main_menu_keyboard
already builds from _scan_button_rows directly.
- telegram_bot.AUTOSCAN_CONFIRMATIONS was byte-identical to TF_CONFIRMATIONS;
the candle loop now reads the one map (app_config comment updated).
- SignalLogger.news_for: one-line wrapper over the module-level read_news that
every real reader uses; the test now calls read_news like its neighbours.
- pair_signals.get_supported_pairs: PAIR_PRODUCERS is the registry; the test
lists it directly.
Suite: 461 passed, 0 failed (two helper-only tests removed).
Claude-Session: https://claude.ai/code/session_0156YhogLytzj9P2iH1skGbL
The candle provider needs only oanda.api_token + base_url; account_id belonged to the
auto-execution daemon removed 2026-09-03 (31002cb). Removed from config.json.example and
the unused oanda_account_id() accessor deleted; the key stays in _KNOWN_OANDA so existing
config.json files keep loading. CLAUDE.md env-var list updated. Test locks all three.
Suite: 464 passed.
Claude-Session: https://claude.ai/code/session_01RWV2yLt3YXHHGVYHSQU2EE
New violent_watch.py assesses each scan from the cached Forex Factory week events, the
UTC clock and the report's own indicator states, and stamps SignalReport.watch_line
post-hoc (telegram alerts, CLI, dashboard). A standalone heads-up goes to the owner on
the fastest scanned timeframe when the level is "hot", once per UTC hour per trigger
set, capped per day. Config block `violent_watch` (enabled False, alert_line, headsup,
max_headsup_per_day, headsup_min_level); env VIOLENT_WATCH_ENABLED.
Rule chosen from a 3-year bar-by-bar sweep (backtest/research/watch_rule_eval.py,
recorded in sizing_campaign_2026-09.md): hot = high-impact USD release within 30 min
(before or after) OR a hot clock slot (weekday 12-14 UTC, 01 UTC, Sunday 22-23 UTC)
with at least one price-extreme reading (outside Bollinger bands, RSI <30/>=70, outside
Keltner); ~8.6% of bars, next-bar odds 3-5x normal, ~4 hot hours/day. watch = clock
slot alone or two price-extreme readings (~2x). Context readings (ADX rising, extreme
volatility, transitioning, MACD acceleration) are phrases only - letting them raise the
level flagged 27% of bars at ~2x and would have sent ~13 heads-ups/day (rejected).
Direction sentence: DOWN when bearish readings dominate (70-78% on both TFs), UP on M5
only. Never a gate or exit: every open-trade protection tested lost money.
Test-locked out of analyze_from_dataframes / engine / backtester. Module 30 (18 tests).
Suite: 453 passed. Guide PDFs not rebuilt.
Claude-Session: https://claude.ai/code/session_01RWV2yLt3YXHHGVYHSQU2EE
agent_panel.verdict_source defaults to "macro": the macro take's stance is the
verdict, the synthesizer call is skipped (4 LLM calls per panel instead of 5);
"chair" restores the four-voice consensus + OPPOSE guard. Verdict source is
cached per mode and logged (panel_verdict_source). 406 tests.
Claude-Session: https://claude.ai/code/session_01FC4s6VCURFte4dub4xKxYL
Deletes oanda_daemon.py, oanda_executor.py, their 28 tests and the daemon-only
config keys (oanda.lot_size/quality_gate/max_hold_bars). The Oanda data
provider and credentials stay — it is the live candle feed. 402 tests.
Claude-Session: https://claude.ai/code/session_01FC4s6VCURFte4dub4xKxYL
New signal_log.py logs each SignalReport to logs/signals.db via a
fail-safe hook in SignalAggregator.analyze_pair — the live-only entry
point used by the bot, daemon, CLI and dashboard. Backtester calls
analyze_from_dataframes directly, so live==backtest parity is preserved
(test-locked). Rows carry signal/grade/price/spread, MTF + trade-setup +
reasons/warnings JSON, and news-at-scan-time (calendar events + scored
headlines JSON via NewsEngine class-level caches).
Config: signal_log block (enabled, path; env SIGNAL_LOG_ENABLED/PATH),
source tagged per consumer (telegram/oanda_daemon/cli/dashboard).
381 tests pass (24 modules).
Three display/warning-only upgrades to the news engine, all behind the
existing keyword path as fallback — signals are never gated on LLM
output (event_risk_caps_signals / sentiment_affects_signals stay False
per the 2026-07 audit; backtester use_news=False untouched):
- Headline sentiment: one batched call per 600s cache window (capped at
25 new headlines, sha1-cached so repeats are never re-sent) overrides
the keyword scores per currency with the gold-inverse doctrine encoded
in the prompt.
- Release surprises: get_event_risk_level's description now says which
way the print pulls GOLD ("— gold bearish: strong jobs lift yields"),
fixing the documented numeric-only gap; risk level and windows
unchanged.
- Market summary: 1-2 sentence macro blurb on SignalReport.news_summary,
rendered in Telegram alerts (📰) and the CLI (🧠).
New llm_client.py: requests-only OpenAI-compatible client (no SDK), one
attempt per call, UTC-day request cap, singleton so the cap survives
per-scan NewsEngine reconstruction. Config: deepseek block in
config.json + DEEPSEEK_API_KEY env; absent key = fully inert.
Live-measured API behavior baked in: V4-Flash is a hybrid thinking model
— requests send thinking:disabled (reasoning otherwise blew the timeout
and could exhaust max_tokens before the answer); prompts are static
module constants so DeepSeek's prefix cache prices repeat calls at the
cache-hit rate; scores use compact [s, xau] pairs; timeout 12s.
Tests 273 → 286 (new Module 19 + deepseek config tests).
Claude-Session: https://claude.ai/code/session_01Nsd8HcrV67GbCZXxZEn1Ma
The monitor has been dead since the Oanda feed switch (Oanda's provider
does not stream). New config.json scalar data_providers.streaming selects
a tick-stream provider independent of the candle feed; telegram_bot
resolves it via _get_streaming_dp() (separate instance, candle cache
stays on the active provider) and falls back to the active provider when
unset — behavior is byte-identical to before when the key is absent.
TiingoDataProvider grows the two sanctioned streaming overrides:
supports_streaming=True and iter_ticks() on wss://api.tiingo.com/fx
(mid price only — the no-spread rule extends to the stream). Two
deliberate divergences from the TD reference implementation, both
because Tiingo authenticates inside the socket: server "E" messages
raise TiingoDataError (a swallowed auth error would end the generator
cleanly and reconnect-hammer with zero backoff), and on_connect fires on
the first received message rather than after the subscribe send. Field
order of quote frames was live-verified: ask is data[7], not data[6] as
the docs table suggests (data[6] is askSize).
make_data_provider(config, name=None) now takes an explicit provider
name; keyless streaming providers disable the monitor with one warning
instead of a traceback-per-minute loop.
Tests 266 → 273 (WS parser, ticker round-trip, streaming config scalar,
explicit-name dispatch, streaming-dp resolution, monitor auto-disable).
Claude-Session: https://claude.ai/code/session_01Nsd8HcrV67GbCZXxZEn1Ma
One class per provider (TwelveData, Oanda, Tiingo), all subclassing
BaseDataProvider which owns cache, validation/cleaning, freshness +
boundary guarantees, fetch_range pagination, and streaming stubs —
subclasses implement only _fetch_raw. config.json data_providers.active
selects the live feed; data_provider.py / oanda_provider.py stay as
back-compat shims. Module 16 provider-abstraction tests (253 total green).
Claude-Session: https://claude.ai/code/session_01Mc7XKYyPySVXiUpcN82QEK
New independent trading daemon (oanda_daemon.py) that places orders on
an Oanda v20 practice account instead of sending Telegram alerts.
Oanda-only market data (oanda_provider.py, mid candles + real bid/ask
spread) through the unchanged signal pipeline; order execution and
trade management via oanda_executor.py (raw requests, no new deps).
Book parity with the backtester: no broker-side TP (exits = SL/TRAIL/
TIMEOUT only), bar-close-only trail ratchet with each closed bar
processed exactly once, 5-bar per-TF entry cooldown, structural SL
from the setup. Accepted divergence: one position total (skipped
signals logged for book reconciliation).
Safety/ops: fail-fast boot on rejected credentials (transient outages
continue), market orders never retried (double-fill risk), entry lock
against aligned-boundary races, restart recovery + adoption via
oanda_state.json, hourly heartbeat, asyncio exception handler, every
failure path logged. Prices formatted at instrument precision.
Config: env OANDA_* -> config.json "oanda" section (validated in
app_config). Tests: Module 15 (30 tests), suite 240 green.
Claude-Session: https://claude.ai/code/session_01VUnCqjWBuvKRSjLwHeB9BJ
Tiingo's free /tiingo/fx/top publishes only a two-tier indicative
spread for gold ($0.10 liquid hours / $0.70 rollover+news; verified on
273 production samples + live probes) — mid ± a posted increment, not
executable depth, so it can never validate the $0.30 backtest cost
assumption. Removed one day after shipping: tiingo_provider.py, the
sampler/quote plumbing in telegram_bot.py, the channel_spread_detail
config knob and tiingo_api_key resolver, and Module 14's Tiingo tests
(221 -> 210). CLAUDE.md documents the verdict: do not re-add a spread
display from Tiingo's free FX feed.
Claude-Session: https://claude.ai/code/session_01BoZKi67Ct6QTeiFJmybXSw
Five-agent audit of the signal path; all fixes are delivery/measurement
side. Regression-proven behavior-neutral on validated signals: the 5yr
M15 replay reproduces the pinned book exactly (+210.65R PF 1.35
WR 57.8% MaxDD 33.90R). Tests 215 -> 221.
- TradeSetup carries trail_arm_r/trail_dist_r stamped from the producing
config; Telegram trail plan reads them (was showing default 0.6/0.75
instead of the validated M15 arm 0.70 / M5 dist 0.85)
- Alert dedup: fingerprint = (signal, quality, direction); entry movement
compared against last-ALERTED entry/risk (old SL-dist bucket churned on
ATR drift -> near-duplicate alerts every candle)
- Fresh fetches enforce the closed-bar boundary check with one 3s grace
retry, then warn (feed lag no longer silently analyzes the prior bar)
- Backtest max DD includes the zero baseline (opening losing streaks
were undercounted; pinned M15 DD unchanged)
- Replay loads the regime-filter TF when the H4-ADX gate is on (was a
silent no-op for M5/M15 primaries)
- clear_cache under _cache_lock; dead ROC zero-cross branches removed
from both engines (provably unreachable); RSS dates via
parsedate_to_datetime (named zones no longer stamped UTC); event risk
grades from the soonest event; .env inline comments stripped;
auto-scan skips NEUTRAL like the breach path; report_to_dict honors
the weight-0 skip rule and emits strategy/trail params; PB funnel
counts spread_gate/accepted; session-filter fail-open paths log loudly
- Docs corrected to match validated code (H&S neckline = lower peak,
exhaustion anchors farthest tracked swing) with re-validation warnings
Claude-Session: https://claude.ai/code/session_01R5soRUeRBi8kHMB99csrJy
Harwinder Sekhon