From b0b71e12254892e6824036b375cfd3bcfa60a472 Mon Sep 17 00:00:00 2001 From: Harwinder Sekhon Date: Tue, 15 Sep 2026 20:15:06 -0400 Subject: [PATCH] backtest cleanup: drop four research/one-time scripts; books and stale cache moved out (owner decisions 2026-09-15) Discussion-first cleanup of backtest/ and scripts/. Nothing in the signal core or the backtester changed. Removed (last commit containing them: 931b265): - backtest/diagnose_filters.py + its two Module 18 tests - scripts/shadow_scan.py (Phase 4 shadow scan, never run) - scripts/backfill_news_archive.py, scripts/migrate_signal_log_news.py (one-time scripts; the interning test keeps its live-path half, the prune test goes) Kept: the replay-recording stack (replay_record.py, replay_scorecard.py, ff_calendar_scraper.py, cache/calendar.parquet, Module 33). Local gitignored data was moved to ~/Development/ForexBot2_trash_2026-09-15 (346 MB) for the owner to delete: all 35 pinned backtest/*.csv books, the three research output folders, the BTC/ETH cache snapshots, the Tiingo parquets and 13 uncited Oanda windows. The three research-folder ignore rules are dropped with the folders. Verification: all five gate books regenerate offline from the pruned cache with the recorded MD5s (M5 W2 29120a1e, M15 W2 d185f099, M15 OOS 6f8f24b4, H1 W3 3b45d12b, H4 W3 e75adae1; NumPy CPU, 0 cache fallbacks). Suite 506 passed, 0 failed, 33 modules. Co-Authored-By: Claude Fable 5.1 --- .gitignore | 3 - CLAUDE.md | 13 +- backtest/diagnose_filters.py | 627 ----------------------------- scripts/backfill_news_archive.py | 230 ----------- scripts/migrate_signal_log_news.py | 177 -------- scripts/shadow_scan.py | 192 --------- test_all.py | 94 +---- 7 files changed, 9 insertions(+), 1327 deletions(-) delete mode 100644 backtest/diagnose_filters.py delete mode 100755 scripts/backfill_news_archive.py delete mode 100644 scripts/migrate_signal_log_news.py delete mode 100644 scripts/shadow_scan.py diff --git a/.gitignore b/.gitignore index 3ee7566..b430a41 100644 --- a/.gitignore +++ b/.gitignore @@ -29,6 +29,3 @@ backtest/*.csv backtest/__pycache__/ .env.* logs/ -backtest/research/sizing_2026-09/ -backtest/research/replay_2026-09/ -backtest/research/cycle_2026-09-10/ diff --git a/CLAUDE.md b/CLAUDE.md index 89cca04..e082b68 100644 --- a/CLAUDE.md +++ b/CLAUDE.md @@ -12,7 +12,7 @@ Every numeric threshold in `config.py` and `pair_signals/xauusd.py` was backtest ```bash python test_all.py # QA suite, no network (mocked); prints its own count - # (509 tests / 33 modules as of 2026-09-15) + # (506 tests / 33 modules as of 2026-09-15) python main.py --pair XAUUSD --timeframe H1 --detail # CLI signal report (--json for full dump, # --panel for the advisory LLM agent panel; the violent-move watch # line appears when the `violent_watch` config block is enabled; @@ -68,18 +68,18 @@ Signal currency everywhere is `SignalStrength` (−2..+2). DataFrames are UTC tz - **`engine.py`** — `TechnicalEngine`: 9 scored indicators (ema_trend, rsi, macd, bollinger, adx, roc, keltner, divergence, candlestick) + ATR as context. Regime classification (trending_up/down, `trending` = mixed/direction-UNKNOWN, transitioning, ranging) drives per-indicator weight multipliers and regime-aware RSI reinterpretation. Compute order matters (ATR first; MACD always computed even when not a pillar — divergence needs the histogram). Private series ride on the result as `_underscore` attrs; scorers/formatters skip anything `_`-prefixed or weight-0. - **`signal_aggregator.py`** — MTF agreement, diminishing-decay penalty system, exhaustion hard block, quality grading, then an ordered chain of post-grade gates (session, XAU SELL quality/vol/hour, regime blocks, M5 noise gates), then three setup generators: XAU ATR (A+/A), generic swing-based, plus the M5 pullback (`XAU_PB`) and chart-pattern (`XAU_CP`, `chart_patterns.py`) entry paths. Step 7 lives in `_attach_core_setup`, which also stamps BUY/SELL on any shipped setup whose `final_signal` was still NEUTRAL (grade C at confidence 45–50, or gate 6c's B — the signal thresholds need ≥ 50) and repairs the pre-gate action text: the backtester always traded those setups (the replay never reads `final_signal`) while Telegram's alert gate rejected NEUTRAL first (2026-09-13; 5/91 trades, +3.7R of +22.2R, in the H1 12-mo book); a final pass after the PB/CP archetypes (`_drop_stale_spread_gate_vetoes`, 2026-09-08) moves a core generator's `SPREAD GATE:` veto to `cleaned_warnings` whenever a later generator shipped a setup (metadata only — books unchanged; the exported `warnings` text column can differ; the prefix is the `_SPREAD_GATE_PREFIX` constant shared by the emitters and both cleanups). The PB/CP generators run AFTER the gate chain and force quality to "B"; the CP path deliberately accepts grade-F composites and re-checks only session/day/big-candle (via warning prefixes) plus its own SELL/vol/RSI gates — it does NOT re-check exhaustion or the M5 min-score/MTF gates. **This is validated behavior, not a bug** (2026-08-04 audit: 226/586 trades in the shipped M5 12mo book are XAU_CP, +38.9R, 104 of them exhaustion-flagged and net-positive) — changing it is a rule-1 revalidation event. `TradeSetup.strategy` selects trailing params; trail params are stamped on the setup from the *producing* config so per-TF overrides reach formatters. - **`pair_signals/`** — per-pair config factories. **Always use the factory (`produce_signal` / `make_config`), never bare `PlatformConfig()`** — the factory applies the config.json overlay first, then the validated per-TF deltas (M5 hour-9 block, M15 trail arm 0.70, M15 SELL-hour block, M5 SELL-in-transitioning block, H1 all-shorts block (2026-09-03), M15 RSI regime-multiplier × 0.5 via `technical.regime_weights_override` (2026-09-09), etc.; evidence in the docstring). A test enforces factory ≡ inline-config equivalence. Confirmation map: M5→[M15,H1], M15→[M5,H1], H1→[M15,H4], H4→[H1,D1]. -- **`signal_log.py` / `advisory_outcome.py`** — SQLite signal-history logger (stdlib only). Hooked into `SignalAggregator.analyze_pair` only (the live entry point — the backtester never writes here, preserving parity), so every live scan from every surface (Telegram `telegram`, CLI `cli`, dashboard `dashboard`) lands one row in `logs/signals.db`: signal/grade/price, MTF + trade-setup + reasons JSON, full per-TF indicator snapshots, the last 14 closed OHLCV candles per fetched TF, and the news at scan time (event risk + calendar events + scored headlines, re-fetched via the NewsEngine class-level caches). Config.json `signal_log` block (`enabled`, `path`, `log_blocked_scans`; env `SIGNAL_LOG_ENABLED`/`SIGNAL_LOG_PATH`/`SIGNAL_LOG_BLOCKED_SCANS`). **Fail-safe by contract** — a logging failure can never break a scan. Panel fields arrive post-hoc: `log_signal` stashes the rowid and `agent_panel.attach_panel` writes verdict + creation/cache provenance + payload hash onto the same row. **Advisory outcomes (2026-09):** EVERY row with a trade setup is seeded at log time (not only panel rows); same-setup re-logs within 60 min from any surface (`setup_key` = tf|direction|entry|stop|strategy) are `DUPLICATE` rows pointing at the canonical row via `advisory_event_id`, so sum `advisory_r_multiple` over `RESOLVED` rows only. The versioned model advances from later stored closed-candle snapshots (next-open reanchoring, intrabar peak with close-only ratchet, SL-first, scheduled weekend/21:00-UTC closures allowed, 300-bar timeout, $0.30/$0.65 views); when a snapshot no longer overlaps the outcome's last bar (either gap reason) it rebuilds the stream from every stored snapshot of that TF; a gap the database cannot fill resolves at the locked trailing stop as a FLOOR (`exit_reason=GAP_LOCKED`, `result.floor=true`, `advisory_outcome.resolve_locked_floor`) when the stop had already crossed the entry, and stays `UNRESOLVABLE` otherwise (2026-09-13 — rows 915/920 of the Sep-8 shorts were +1.14R/+0.84R floors the scorecard had been dropping). It is a cleaned-mid evidence model, not execution P&L. **Post-gate truth columns:** `gate_reason` (the warning that removed the setup), `actionable` (grade ≥ C AND setup — the `action` text is written BEFORE the gate chain and can say "STRONG SELL — High-quality setup" on a D row), `cleaned_warnings_json` (warnings a shipped setup superseded — the XAU_CP acceptance's stripped vetoes and, since 2026-09-08, a stale core spread-gate veto; non-empty is NOT a CP marker, use `setup_json.strategy`), `bar_range_median` (= the misnamed `spread` column: median High-Low, NOT a broker spread), `scan_kind` (`blocked` weekend/day early returns carry no news/candles; `failed` = ANY required frame — primary, confirmation, or the regime-filter TF when that filter is on — failed to fetch: price 0, no signal, no analysis run (2026-09-13; was primary-only, and a lost confirmation frame used to run the scan with an unearned −15 MTF penalty) — readers skip it like `blocked`), `config_fingerprint` (gate config hash — surfaces config drift between processes). **News is content-addressed:** `headlines_json`/`events_json` are interned into `news_snapshots` (they were 73% of a 667 MB DB); read through the `signals_with_news` view or `signal_log.read_news`. Readers must open with `mode=ro`, never `immutable=1` (blind to WAL rows). **Weight-research columns (2026-09):** `bar_ts` (the primary bar LABEL the scan closed on — same string as `indicators_json`, indexed with `timeframe`), `engine_fingerprint` (hash of the SCORING config: indicator params, thresholds, the regime weight table in force, TFs — `SignalAggregator.engine_config_fingerprint`, the gate fingerprint's twin), `code_version` (short git SHA), `penalty_tags_json`/`penalty_total`/`momentum_fading`/`exhaustion_count`, and the post-hoc `watch_level`/`watch_kinds`/`watch_line` written by `signal_log.update_watch` from `violent_watch.attach_watch` (level and kinds are stored even when the display line is off). Every `indicators_json` entry also carries `eff_weight` (base weight × regime multiplier = the weight the composite actually used; `engine.regime_weight_table` is the single source of truth) and `wscore` (its contribution) — null for skipped indicators (weight 0, `_`-prefixed); **divergence and candlestick entries exist only when they fire, so absence is "no vote", not a zero vote**. **Table `scan_outcomes`:** model-free forward moves per scan (`fwd_1h/4h/24h` in $ and ATRs, `mfe_24h_atr`/`mae_24h_atr`, `burst_1h` = a ≥3×ATR bar within the hour, `bars_used`, `complete`), measured from the row's own last closed bar on the stored candle stream; a horizon stays NULL until it has elapsed, and `complete` needs the full 24 h with no gap over 3 bars beyond the scheduled weekend/17:00-NY closures (`advisory_outcome._has_continuity`). `SignalLogger.label_scan_outcomes` runs opportunistically at the end of `log()` (at most once per 60 s per process, 150 rows and ~2 s per pass, idempotent — a row is rewritten only when its numbers change, but its `labeled_at` is bumped so it rotates to the back; an incomplete row older than `SCAN_LABEL_MAX_AGE_H` = 72 h (measured from the newest logged row) is FINAL and never anchors the shared bar stream, and `_stream_since` warns when it hits its row cap — before 2026-09-13 the Labor-Day pile anchored the stream and the 4,000-row cap would have blinded every later row), so no daemon is needed; `scripts/label_scan_outcomes.py` is the backlog driver (`--dry-run` counts read-only). Readers must dedup by the FIRST row per (`timeframe`, `bar_ts`) — several surfaces log the same bar. Tools: `analyze_signal_log.py` (`--stored-candles` DB-only replay with event/opportunity/one-position books + persona scorecard, `--news` news-sentiment lead/lag + extremes + outcome buckets, `--parity` telegram-vs-dashboard divergence report, `--scan-outcomes` forward-move + hit-rate table by TF×signal/grade with a NEUTRAL baseline and burst rate by watch level), `scripts/backfill_signal_outcomes.py` (seed/dedup/advance legacy rows), `scripts/label_scan_outcomes.py` (label/relabel scan outcomes), `scripts/migrate_signal_log_news.py` (intern legacy blobs; `--vacuum` with both writers stopped). +- **`signal_log.py` / `advisory_outcome.py`** — SQLite signal-history logger (stdlib only). Hooked into `SignalAggregator.analyze_pair` only (the live entry point — the backtester never writes here, preserving parity), so every live scan from every surface (Telegram `telegram`, CLI `cli`, dashboard `dashboard`) lands one row in `logs/signals.db`: signal/grade/price, MTF + trade-setup + reasons JSON, full per-TF indicator snapshots, the last 14 closed OHLCV candles per fetched TF, and the news at scan time (event risk + calendar events + scored headlines, re-fetched via the NewsEngine class-level caches). Config.json `signal_log` block (`enabled`, `path`, `log_blocked_scans`; env `SIGNAL_LOG_ENABLED`/`SIGNAL_LOG_PATH`/`SIGNAL_LOG_BLOCKED_SCANS`). **Fail-safe by contract** — a logging failure can never break a scan. Panel fields arrive post-hoc: `log_signal` stashes the rowid and `agent_panel.attach_panel` writes verdict + creation/cache provenance + payload hash onto the same row. **Advisory outcomes (2026-09):** EVERY row with a trade setup is seeded at log time (not only panel rows); same-setup re-logs within 60 min from any surface (`setup_key` = tf|direction|entry|stop|strategy) are `DUPLICATE` rows pointing at the canonical row via `advisory_event_id`, so sum `advisory_r_multiple` over `RESOLVED` rows only. The versioned model advances from later stored closed-candle snapshots (next-open reanchoring, intrabar peak with close-only ratchet, SL-first, scheduled weekend/21:00-UTC closures allowed, 300-bar timeout, $0.30/$0.65 views); when a snapshot no longer overlaps the outcome's last bar (either gap reason) it rebuilds the stream from every stored snapshot of that TF; a gap the database cannot fill resolves at the locked trailing stop as a FLOOR (`exit_reason=GAP_LOCKED`, `result.floor=true`, `advisory_outcome.resolve_locked_floor`) when the stop had already crossed the entry, and stays `UNRESOLVABLE` otherwise (2026-09-13 — rows 915/920 of the Sep-8 shorts were +1.14R/+0.84R floors the scorecard had been dropping). It is a cleaned-mid evidence model, not execution P&L. **Post-gate truth columns:** `gate_reason` (the warning that removed the setup), `actionable` (grade ≥ C AND setup — the `action` text is written BEFORE the gate chain and can say "STRONG SELL — High-quality setup" on a D row), `cleaned_warnings_json` (warnings a shipped setup superseded — the XAU_CP acceptance's stripped vetoes and, since 2026-09-08, a stale core spread-gate veto; non-empty is NOT a CP marker, use `setup_json.strategy`), `bar_range_median` (= the misnamed `spread` column: median High-Low, NOT a broker spread), `scan_kind` (`blocked` weekend/day early returns carry no news/candles; `failed` = ANY required frame — primary, confirmation, or the regime-filter TF when that filter is on — failed to fetch: price 0, no signal, no analysis run (2026-09-13; was primary-only, and a lost confirmation frame used to run the scan with an unearned −15 MTF penalty) — readers skip it like `blocked`), `config_fingerprint` (gate config hash — surfaces config drift between processes). **News is content-addressed:** `headlines_json`/`events_json` are interned into `news_snapshots` (they were 73% of a 667 MB DB); read through the `signals_with_news` view or `signal_log.read_news`. Readers must open with `mode=ro`, never `immutable=1` (blind to WAL rows). **Weight-research columns (2026-09):** `bar_ts` (the primary bar LABEL the scan closed on — same string as `indicators_json`, indexed with `timeframe`), `engine_fingerprint` (hash of the SCORING config: indicator params, thresholds, the regime weight table in force, TFs — `SignalAggregator.engine_config_fingerprint`, the gate fingerprint's twin), `code_version` (short git SHA), `penalty_tags_json`/`penalty_total`/`momentum_fading`/`exhaustion_count`, and the post-hoc `watch_level`/`watch_kinds`/`watch_line` written by `signal_log.update_watch` from `violent_watch.attach_watch` (level and kinds are stored even when the display line is off). Every `indicators_json` entry also carries `eff_weight` (base weight × regime multiplier = the weight the composite actually used; `engine.regime_weight_table` is the single source of truth) and `wscore` (its contribution) — null for skipped indicators (weight 0, `_`-prefixed); **divergence and candlestick entries exist only when they fire, so absence is "no vote", not a zero vote**. **Table `scan_outcomes`:** model-free forward moves per scan (`fwd_1h/4h/24h` in $ and ATRs, `mfe_24h_atr`/`mae_24h_atr`, `burst_1h` = a ≥3×ATR bar within the hour, `bars_used`, `complete`), measured from the row's own last closed bar on the stored candle stream; a horizon stays NULL until it has elapsed, and `complete` needs the full 24 h with no gap over 3 bars beyond the scheduled weekend/17:00-NY closures (`advisory_outcome._has_continuity`). `SignalLogger.label_scan_outcomes` runs opportunistically at the end of `log()` (at most once per 60 s per process, 150 rows and ~2 s per pass, idempotent — a row is rewritten only when its numbers change, but its `labeled_at` is bumped so it rotates to the back; an incomplete row older than `SCAN_LABEL_MAX_AGE_H` = 72 h (measured from the newest logged row) is FINAL and never anchors the shared bar stream, and `_stream_since` warns when it hits its row cap — before 2026-09-13 the Labor-Day pile anchored the stream and the 4,000-row cap would have blinded every later row), so no daemon is needed; `scripts/label_scan_outcomes.py` is the backlog driver (`--dry-run` counts read-only). Readers must dedup by the FIRST row per (`timeframe`, `bar_ts`) — several surfaces log the same bar. Tools: `analyze_signal_log.py` (`--stored-candles` DB-only replay with event/opportunity/one-position books + persona scorecard, `--news` news-sentiment lead/lag + extremes + outcome buckets, `--parity` telegram-vs-dashboard divergence report, `--scan-outcomes` forward-move + hit-rate table by TF×signal/grade with a NEUTRAL baseline and burst rate by watch level), `scripts/backfill_signal_outcomes.py` (seed/dedup/advance legacy rows), `scripts/label_scan_outcomes.py` (label/relabel scan outcomes). The one-time news-interning migration script was removed 2026-09-15 (last commit with it `931b265`). - **`news_engine.py`** — Forex Factory calendar (MyFXBook RSS fallback) + RSS keyword sentiment with gold-inverse keywords. Class-level caches (fresh engine per scan). Optional DeepSeek enrichment via `llm_client.py` (requests-only, no SDK): batched headline sentiment overriding keyword scores, gold-direction on release surprises, and a market-summary blurb (`SignalReport.news_summary`) — **display/warning-only, keyword fallback on any failure, inert without a key**; prompts are static module constants (DeepSeek prefix-cache pricing) and requests send `thinking: disabled` (V4-Flash reasoning otherwise blows the timeout). By XAU-tuned default, event risk and sentiment **warn but do not gate signals** (`event_risk_caps_signals=False`, `sentiment_affects_signals=False`) — the LLM does not change that. **Headline pool (2026-09-04):** only headlines ≤ 4 h old with a parsable publish time (`headline_max_age_seconds=14400`, undated items dropped), and `get_fundamental_context` drops headlines whose every relevant score is inside ±0.1 (`_is_neutral_headline`, `NEUTRAL_HEADLINE_THRESHOLD`) before the per-currency mean — so `news_sentiment` from that date is a mean over directional headlines only (bolder values; log comparisons must split at the deploy date). The market summary prompt writes for someone who has never traded (no acronyms), sends headlines newest-first with the platform's own gold direction, and caps at 400 chars. **Verdict 2026-09-04** (`backtest/research/news_sentiment_value_2026-09-04.md`): `news_sentiment` is a trailing-24h echo of price (r≈0.67 with the past day, ≈0.03 with the next; flips 44 h after tops / 16 h after bottoms) — both gating flags stay False; re-check with `analyze_signal_log.py --stored-candles --news`. -- **`news_archive.py`** — durable headline corpus in its OWN database (`logs/newsdata.db`), separate from `signals.db` so a signal-log rotation cannot take the news with it (that is exactly how the 3-week corpus behind `news_sentiment_value_2026-09-04.md` was lost — 4,054 distinct headlines, unrecoverable, `logs/` is gitignored). One row per distinct headline keyed on `sha256(headline)`, carrying the text, description, link, publisher, publish time, the score, `score_source` (`keyword`/`llm`/`model` — a new `NewsSentiment` field, also mirrored into `signal_log._headline_to_dict`), and `teacher_model`/`teacher_prompt_sha1` so labels made under a changed prompt are never pooled with older ones. **Writes are one atomic UPSERT per row and upgrade-only**: an `llm` score always replaces a `keyword` one (the >25/cycle batch-overflow tail gets its DeepSeek label on the next fetch window in a long-running process — verified: 13/13 upgraded) but a `keyword` score can never overwrite an `llm` one; re-seeing a headline always bumps `seen_count`. A read-then-write pair loses increments when the bot, the dashboard publisher and the tests first see a headline together, and the WAL switch itself needs `_with_lock_retry` — both are regression-tested with 3 concurrent writers. Hooked at the END of `news_engine._fetch_news_sentiment_raw`, **before** `_is_neutral_headline` runs in `get_fundamental_context`: archiving downstream of that ±0.1 filter is why `signals.db` holds almost no neutral headlines and would teach a trained scorer that "neutral" barely exists. Never raises (fail-safe by contract, like `signal_log`). The backtester never builds a `NewsEngine`, so replays never archive (test-locked). Seeded by `scripts/backfill_news_archive.py`, which reads both DB shapes read-only and **recomputes** provenance (stored score vs the keyword scorer: differs → `llm`, exact non-zero match → `keyword`, both zero → `unknown`) because neither source recorded it. Purpose: training data for a local numpy-only headline scorer to replace the 24/78 keyword fallback — see the note below. +- **`news_archive.py`** — durable headline corpus in its OWN database (`logs/newsdata.db`), separate from `signals.db` so a signal-log rotation cannot take the news with it (that is exactly how the 3-week corpus behind `news_sentiment_value_2026-09-04.md` was lost — 4,054 distinct headlines, unrecoverable, `logs/` is gitignored). One row per distinct headline keyed on `sha256(headline)`, carrying the text, description, link, publisher, publish time, the score, `score_source` (`keyword`/`llm`/`model` — a new `NewsSentiment` field, also mirrored into `signal_log._headline_to_dict`), and `teacher_model`/`teacher_prompt_sha1` so labels made under a changed prompt are never pooled with older ones. **Writes are one atomic UPSERT per row and upgrade-only**: an `llm` score always replaces a `keyword` one (the >25/cycle batch-overflow tail gets its DeepSeek label on the next fetch window in a long-running process — verified: 13/13 upgraded) but a `keyword` score can never overwrite an `llm` one; re-seeing a headline always bumps `seen_count`. A read-then-write pair loses increments when the bot, the dashboard publisher and the tests first see a headline together, and the WAL switch itself needs `_with_lock_retry` — both are regression-tested with 3 concurrent writers. Hooked at the END of `news_engine._fetch_news_sentiment_raw`, **before** `_is_neutral_headline` runs in `get_fundamental_context`: archiving downstream of that ±0.1 filter is why `signals.db` holds almost no neutral headlines and would teach a trained scorer that "neutral" barely exists. Never raises (fail-safe by contract, like `signal_log`). The backtester never builds a `NewsEngine`, so replays never archive (test-locked). Seeded 2026-09-08 by a one-time backfill script (removed 2026-09-15, last commit with it `931b265`) that read both DB shapes read-only and **recomputed** provenance (stored score vs the keyword scorer: differs → `llm`, exact non-zero match → `keyword`, both zero → `unknown`) because neither source recorded it. Purpose: training data for a local numpy-only headline scorer to replace the 24/78 keyword fallback — see the note below. - **`agent_panel.py`** — ai-hedge-fund-style advisory panel: 4 DeepSeek persona calls (macro/technical/risk/contrarian, concurrent on a panel-private pool) + 1 synthesizer producing an **ADVISORY-ONLY** consensus (`AGREE/NEUTRAL/OPPOSE` + confidence + rationale) written onto `SignalReport.panel_*` fields. Attached **post-hoc** by consumers (`telegram_bot` alert path, `main --panel`, `dashboard_publisher`) — never runs inside `analyze_from_dataframes` (backtest parity, test-locked) and no gate reads its fields. Persona payloads are role-filtered: macro gets high-impact events plus only code-scored directional news; technical gets strategy/pattern, primary indicators, MTF, levels, and ATR-normalized candles; risk gets setup strategy, strategy cost assumption, trailing semantics, volatility/events/warnings (never the median candle range as spread); contrarian gets directional news, setup, timestamped latest-closed price, indicators, reasons, and warnings. Spend is bounded by exact cache, full-evidence-identity TTL reuse, and `max_panels_per_day`; changed evidence never inherits a direction-only cached verdict (test-locked — which also means TTL reuse effectively never fires live because price/candles change every bar; the daily cap is the real spend bound). Verdict provenance logs creation time, `MISS`/`EXACT_HIT`/`TTL_REUSE`, and a SHA-256 payload hash. Prompts forbid outside knowledge and outputs remain short. **OPPOSE guard (2026-09):** the 3-week signal-log scorecard showed the contrarian/risk/technical personas oppose almost every trend entry (chair said OPPOSE 73% of the time; opposed trades made +30R, PF 2.6) while the macro persona was the only informative voice (macro=support → 88% win). `_apply_oppose_guard` therefore downgrades a chair OPPOSE to NEUTRAL unless macro opposes, or technical opposes while macro is not supportive; the synthesizer prompt states the same rule. Re-score with `analyze_signal_log.py --stored-candles` (persona scorecard) before changing it again. **Verdict source (2026-09-03, owner decision):** all four personas still run and are logged (`panel_takes_json`), but `agent_panel.verdict_source` defaults to `"macro"` — the alert verdict is the macro persona's stance (support/neutral/oppose → AGREE/NEUTRAL/OPPOSE) and the synthesizer call is skipped (4 LLM calls per panel instead of 5); `"chair"` restores the four-voice synthesizer + OPPOSE guard. Re-score personas monthly with `analyze_signal_log.py --stored-candles`; weights are not to be tuned before ~300 graded outcomes with a held-out half. - **`violent_watch.py`** — **display-only** violent-move watch, dark by default (`violent_watch` config block: `enabled`, `alert_line`, `headsup`, `max_headsup_per_day`, `headsup_min_level`; env `VIOLENT_WATCH_ENABLED`). Flags when a "violent bar" (range ≥ 3× prior ATR14) is unusually likely and stamps `SignalReport.watch_line` **post-hoc** via `attach_watch` (telegram alert path, `main.py`, `dashboard_publisher`) — same contract as the agent panel: never imported by `signal_aggregator`/`engine`/`backtest/`, never raises, never read by a gate (test-locked, Module 30). Triggers: a high-impact **USD** calendar release within ±30 min (hot), hot clock slots (weekday 12–14 UTC, weekday 01 UTC, Sunday reopen 22–23 UTC), and **price-extreme** indicator readings (outside the Bollinger bands, RSI <30 / ≥70, outside the Keltner channel). Level = hot when the calendar fires or a clock slot plus ≥1 price extreme; watch on a clock slot alone or ≥2 price extremes. Context readings (ADX slope >3, volatility "extreme", regime "transitioning", MACD STRONG_BUY/STRONG_SELL) are printed but **never raise the level** — the first draft that let them count flagged 27% of bars "hot" at only ~2× odds; the shipped rule flags 8.6% at 3–5× (evidence: `backtest/research/sizing_campaign_2026-09.md` addenda). Direction is called DOWN on bearish readings (70–78% on M5 and M15) and UP only on M5. Telegram also sends a standalone heads-up on the fastest scanned TF (`_HEADSUP_TF`), rate-limited per UTC hour+trigger-set and per day; it explicitly tells the owner **not** to touch open trades — every protection tested lost money, the stop is the protection. - **`telegram_bot.py`** — owner-locked bot + subscriber channel; candle-boundary auto-scan loops (M5/M15/H1/H4, boundary+10 s), alert quality threshold + fingerprint dedup (committed only after a successful send; the fingerprint is signal+direction — grade is NOT part of it, a grade UPGRADE re-alerts, an A↔B flicker does not), 📰 LLM market-context line on alerts when available, WebSocket level-breach monitor fed by the streaming provider (`data_providers.streaming`, falling back to the active provider; auto-disabled when the resolved provider can't stream or has no API key — Oanda does not stream, Tiingo does). The breach handler reuses this bar's M15 report (`_cached_or_scan`, keyed by closed-bar epoch) instead of rescanning — it used to log 3.5 M15 rows per candle. **Alerts require a trade setup (2026-09-08):** `_is_alert_candidate` (shared by the scan and breach paths) needs non-neutral + grade ≥ threshold + `trade_setup`; a signal every generator declined used to go out anyway (a B-grade no-setup SELL under a "costs would eat the edge" warning). **Sends are logged:** every delivered alert / breach alert / violent-move heads-up writes an `… sent:` INFO line carrying the signal-log row id, and a no-setup candidate writes `alert skipped: … (gate reason)` — before this only failures were logged and a review had to reconstruct sends from panel rows. Rotating log: `logs/telegram.log`, one file per server-local day × 30 backups (was 3-h × 240 until 2026-09-08; the old `_HH`-suffixed files on the live box are not matched by the daily cleanup and need one manual delete). - **`dashboard_publisher.py` / `static/dashboard.html`** — web dashboard as a **static publisher** (live LAMP server): standalone daemon scans at candle closes via `produce_signal` and atomically writes `snapshot.json`/`news.json`/`dashboard.html` to `$DASHBOARD_OUT_DIR` (Apache serves them; page is public by owner decision). **Hard independence rule: never imports, and is never imported by, `telegram_bot`** — confirmation map and boundary math are deliberate local copies, so a web-side failure can only mean a stale page. Due-TF classification uses the RAW wake time (flooring wake−buffer skips every H1/H4 scan — 2026-08-02 bug). News panel reads `NewsEngine.get_fundamental_context()` plus the additive `get_week_events()` accessor (display-only; no signal path calls it); as a separate process it has its own news caches and DeepSeek daily cap. News/calendar rows are clickable (detail dialog); headlines carry `id` = sha1(headline) in news.json, and exactly one on-demand endpoint exists: `static/news_summary.php` (installed by `install_static`), which serves a DeepSeek gold-impact summary per headline — whitelist-gated against news.json ids, cache-forever in `ai/`, own fail-closed daily cap (150/day, file-based, separate from the publisher's), key at `/etc/gold-dashboard/deepseek.key`, never reads repo config. Deployment checklist: `deploy/README-dashboard.md`. -- **`backtest/gpu_backtest.py`** — bar-by-bar replay through the real `analyze_from_dataframes` (news off). Realism model (do not regress): next-bar-open entries re-anchored to the raw price path, round-trip `--spread` charged per trade, trade resolution on RAW bars (cleaning applies only to signal-side data), same-bar SL+TP tie → SL first, gap fills at bar open, trail ratchet at bar close only. Indicators are precomputed once per TF over the whole cached history (CuPy GPU or NumPy fallback — check the `Backend:` log line); the per-TF warm-up pad before `--start` is `_history_pad_days` (62 days for M5/M15/H1, 234 for H4, 1400 for D1 — the same 1000-bar seed the live floor guarantees; 2026-09-10, no trade in any book moved); replay fans out over worker processes and the 5-bar cooldown is applied in an ordered pass afterwards, so parallel and sequential books are identical. Parquet cache in `backtest/cache/` keyed by provider/symbol/TF/window. `backtest/*.csv` are gitignored pinned trade books from past campaigns — used for manual before/after comparison. `backtest/instructions_for_backtest.txt` lists what every backtest report must record. +- **`backtest/gpu_backtest.py`** — bar-by-bar replay through the real `analyze_from_dataframes` (news off). Realism model (do not regress): next-bar-open entries re-anchored to the raw price path, round-trip `--spread` charged per trade, trade resolution on RAW bars (cleaning applies only to signal-side data), same-bar SL+TP tie → SL first, gap fills at bar open, trail ratchet at bar close only. Indicators are precomputed once per TF over the whole cached history (CuPy GPU or NumPy fallback — check the `Backend:` log line); the per-TF warm-up pad before `--start` is `_history_pad_days` (62 days for M5/M15/H1, 234 for H4, 1400 for D1 — the same 1000-bar seed the live floor guarantees; 2026-09-10, no trade in any book moved); replay fans out over worker processes and the 5-bar cooldown is applied in an ordered pass afterwards, so parallel and sequential books are identical. Parquet cache in `backtest/cache/` keyed by provider/symbol/TF/window. Pinned trade books are no longer kept on disk (owner decision 2026-09-15): every gate MD5 and its command line is recorded here or in the ledgers, so regenerate a book from the cache when a trade-level diff is needed (`backtest/*.csv` stays gitignored for that). `backtest/instructions_for_backtest.txt` lists what every backtest report must record. ## Rules for making changes (read before editing signal logic) -1. **Never change a tuned value without re-validation.** Acceptance rule: positive total R in BOTH train and validation halves of a ≥12-month window, PF ≥ 1.3, at realistic cost (XAU $0.65 round trip headline; retain $0.30 for TD-era cross-book comparability), trailing exits, next-open fills. `round_trip_cost` stays $0.30 because it also drives the live TP spread gate; changing it is a separate rule-1 event. Marginal edges must additionally pass the 2024-06→2025-03 out-of-sample window — within-year splits alone have approved configs that later failed OOS. **Owner preference (2026-09-05): backtest windows go back at most 3 years** (the 5-year books are historical reference only); the 3-year baseline books (2023-09-05→2026-09-05, $0.65: M5 857/+161.3R/PF 1.53, M15 1086/+148.7R/PF 1.37, H1 255/+70.5R/PF 1.98, H4 113/+22.7R/PF 1.63) live in the gitignored `backtest/research/sizing_2026-09/` and are regenerated with the commands in `sizing_campaign_2026-09.md`. +1. **Never change a tuned value without re-validation.** Acceptance rule: positive total R in BOTH train and validation halves of a ≥12-month window, PF ≥ 1.3, at realistic cost (XAU $0.65 round trip headline; retain $0.30 for TD-era cross-book comparability), trailing exits, next-open fills. `round_trip_cost` stays $0.30 because it also drives the live TP spread gate; changing it is a separate rule-1 event. Marginal edges must additionally pass the 2024-06→2025-03 out-of-sample window — within-year splits alone have approved configs that later failed OOS. **Owner preference (2026-09-05): backtest windows go back at most 3 years** (the 5-year books are historical reference only); the 3-year baseline books (2023-09-05→2026-09-05, $0.65: M5 857/+161.3R/PF 1.53, M15 1086/+148.7R/PF 1.37, H1 255/+70.5R/PF 1.98, H4 113/+22.7R/PF 1.63) are regenerated with the commands in `sizing_campaign_2026-09.md` (their on-disk copies were removed 2026-09-15; the current-HEAD figures are in the 2026-09-10 note). 2. **Per-timeframe overrides live in the pair factory**, never in global defaults — the same knob has opposite effects on different TFs (proven repeatedly: `tp_floor_rr`, hour-9 block, trail params). Document validation evidence in the factory docstring. 3. **Pair isolation is a hard invariant.** XAU gates in the aggregator are display-name-gated so a future pair port cannot inherit gold-calibrated logic by accident. 4. **Live == backtest parity.** If you add a live gate, add it to the replay path in the same commit; all time filters evaluate bar-close time in both paths. @@ -101,7 +101,7 @@ Signal currency everywhere is `SignalStrength` (−2..+2). DataFrames are UTC tz - Oanda v20 is the live feed (`data_providers.active: "oanda"`). The 2026-08-04 re-pin made the 27 `backtest/oanda_*.csv` books the baseline; all TD-era books are superseded. (2026-08-14) The re-pin M5 book `oanda_m5_12mo_065_nopb_20260814.csv` (+106.82R PF 1.51) is itself SUPERSEDED by the 2026-09-03 book in the entry below; the M15 reference `oanda_m15_12mo_065.csv` (397 / +73.93R / PF 1.53) is SUPERSEDED on 2026-09-09 by the 2-year book `oanda_m15_2yr_065_20260909.csv` (931 / +146.14R / PF 1.43; RSI × 0.5 override shipped — see the 2026-09-09 entries below). See `backtest/research/repin_verification_2026-08-14.md`. - Cost reporting is dual: $0.65 is the measured-Oanda realistic headline ($0.65-$0.72 round trip), while $0.30 remains for cross-book comparability and as the live TP spread-gate input. Raising `round_trip_cost` from $0.30 would move the TP floor from about $9 to $19.50 and requires full rule-1 revalidation. The $0.65 figure comes from the 2026-07-24 Dukascopy tick study (9M ticks over 12 months; its tooling was removed 2026-09-06 — the conclusion lives here and in memory). - Twelve Data is signal-incompatible with Oanda (Phase 1: about 19% M5 bar-direction disagreement and a systematic ~$1 high bias; see `backtest/research/feed_divergence_2026-08-14.md`). On an Oanda candle outage, stop scanning rather than switching to TD candles. -- Tiingo is streaming-only (`data_providers.streaming: "tiingo"`), not a designated candle fallback. Oanda-Tiingo signal parity is documented in `backtest/research/signal_parity_2026-08-14.md`; Phase 4's two-week live shadow scan (`scripts/shadow_scan.py`) remains a manual operational step. +- Tiingo is streaming-only (`data_providers.streaming: "tiingo"`), not a designated candle fallback. Oanda-Tiingo signal parity is documented in `backtest/research/signal_parity_2026-08-14.md`; the Phase 4 shadow-scan script was removed 2026-09-15 without ever being run (last commit with it `931b265`). - Human-facing docs in `docs/` (`user_guide.html`, `beginner_guide.html`) must be kept in sync when signal behavior changes. There are NO PDF builds in the tree (checked 2026-09-10; earlier notes about PDFs lagging the HTML are moot). The backtester and the replay recorder read/write Parquet through `pyarrow`, which is installed in the dev `venv/` but deliberately absent from `requirements.txt` (live code never touches Parquet). - 2026-09-03 signal-log campaign shipped two new per-TF gates (`backtest/research/signal_log_campaign_2026-09.md`): M5 `sell_blocked_regimes=["transitioning"]` and H1 `sell_min_quality=[]` (all H1 shorts blocked). Current-HEAD M5 canonical book is now `backtest/oanda_m5_12mo_065_nopb_20260903.csv` (557 / +112.21R / PF 1.56, halves +55.9/+56.4); H1 reference `backtest/oanda_h1_12mo_065_20260903.csv` (91 / +22.2R / PF 1.79; 5yr +49.1R PF 1.46; OOS +16.7R PF 1.74). - 2026-09-06 stake-sizing campaign (`backtest/research/sizing_campaign_2026-09.md`): a walk-forward "second opinion" stake multiplier (meta-labelling on 65 decision-time features, 3-year Oanda books at $0.65) FAILED the pre-registered rule on M5 (clean null) and M15 (lift inside the label-shuffle null, explained by BUY-vs-SELL). Nothing that changes numbers shipped. Behaviour-neutral tooling remains: `position_sizing.build_features` + `--export-features` (default export stays 63 columns), `SignalReport.penalty_tags/penalty_total/momentum_fading`, `TradeSetup.stake_multiplier` (None; reserved) (the campaign's research scripts were removed 2026-09-06; git history has them). Stage-B facts: M5 stretch measures are POSITIVE in both halves (do not re-propose exhaustion sizing on M5); M15 exhaustion tag is weakly negative (≈ +3R/yr, noise); bursts cluster at 12–15 UTC and 01 UTC bar-close and do NOT hurt entries. @@ -115,3 +115,4 @@ Signal currency everywhere is `SignalStrength` (−2..+2). DataFrames are UTC tz - 2026-09-10 improvement cycle 1 (`backtest/research/cycle_2026-09-10.md`; 501 tests; no signal-core change): HEAD baseline re-established on the 3-yr window at $0.65 (M5 870/+153.6R/PF 1.50, M15 1273/+186.6/1.39, H1 253/+70.0/1.98, H4 114/+21.7/1.59) and the three pinned MD5s (M5 2-yr, M15 2-yr, M15 OOS) reproduced exactly. Robustness: the cost ladder trips no red flag (every TF keeps PF ≥ 1.3 with positive halves at $0.80; at $1.30 M5 PF 1.36, M15 1.29, H1 1.89, H4 1.53); the intraday edge is a fat right tail (top 5% of trades carry 94% of M5 R and 96% of M15 R, versus 48% on H1; M5 2024 was flat at +1.0R over 102 trades). Pre-registered bucket screen: exactly one qualifying bucket — M15 grade-A SELLs (230 / −14.4R, negative in all three years and OOS; the 2026-09-03 5-yr run said the opposite) — parked as the only cycle-2 candidate. **C1 REJECTED:** H1 divergence multiplier → 0 in every regime (the parked 2026-09 candidate on its now-sanctioned vehicle): 3-yr ΔR +1.0, second half −1.5R, PF 1.98 → 1.89 (its OOS book 88/+21.9/PF 1.92 is trade-identical to the old `disabled_indicators` run, so a 0.0 regime multiplier ≡ disabling for the composite). **C2:** the M15 RSI × 0.5 ship sits on a stable plateau (k = 0.25 / 0.75 keep 78% / 71% of its 2-yr gain, both OOS-positive; R is monotone in k across 0…1.5) — no other k proposed. **Seeding parity SHIPPED:** live fetches floor at `MIN_LIVE_BARS = 1000` (H4 380 → 1000, D1 300 → 1000; measured ema_trend flips 2/267 H4 and 1/313 D1 bars at the old depths, none ≥ 600; the live D1 bar of 2026-09-10 itself read SELL at 300 bars and NEUTRAL at 1000/5000) and the backtester pad targets the same depth (`_history_pad_days`: 62 / 234 / 1400 days) — M5/M15 3-yr books byte-identical, H1/H4 trade-identical with only window-start float noise in the exported columns, so no re-pin. Also fixed in these notes: no PDF docs exist; `pyarrow` is a dev-venv-only dependency of the backtester/recorder. - 2026-09-13 bug audit (plan file `~/.claude/plans/you-are-python-software-joyful-teacup.md`; 519 tests / 34 modules; the five reference books — M5 W2 `29120a1ee32979431d062f210d41a9d1`, M15 W2 `d185f099482a07f29502cb5a371a9bbf`, M15 OOS `6f8f24b49597ced28f50a19e08c3472f`, H1 W3 `3b45d12bade823178b267c13bc698458`, H4 W3 `e75adae13c839b15736f69c0114e6aa5` (the H1/H4 values are the post-pad books, `cycle_2026-09-10/w3_{1h,4h}_pad.csv`) — byte-identical before and after every change, 0 GPU cache fallbacks). Verified sound, no change: composite = Σwscore/Σeff_weight × 50 on 400/400 DB rows; scan-outcome labels reproduce later scan prices to 0.000 on 844 rows; outcome-model exits reproduced on rows 578/1372/1500; Oanda H4/D1 alignment is UTC; no look-ahead in engine/backtester/chart_patterns; no cross-bar state; parquet caches hold only holiday gaps. Fixed (all delivery/measurement side, books unchanged): (1) NEUTRAL-signal C/B setups now alert (see `_attach_core_setup`); (2) a missing confirmation frame fails the scan (`scan_kind='failed'`, warning `DATA UNAVAILABLE:`); (3) `pd.Timestamp.utcnow()` (deprecated, removed in pandas 4) replaced in `providers/base.py`/`engine.py`, `requirements.txt` pins `pandas<4`, `numpy<3`; (4) advisory outcomes keep locked-profit floors and the rebuild fires on both gap reasons; (5) scan-outcome labeler max-age/anchor bound; (6) the daily settlement pause (Mon–Thu 17:00 NY) is a scheduled closure for the freshness gates — `_is_daily_maintenance`/`_is_scheduled_closure`/`_scheduled_closed_seconds` in `providers/base.py`, the expected label walks back only past labels whose WHOLE span lies in a closure (a D1 bar labelled Sunday 00:00 UTC and an H4 bar labelled Sunday 20:00 exist), and the bot/dashboard skip scans whose just-closed label sits in the pause (61 false "stale" rows in 3 nights before); (7) `fetch_range` retries a failed page in place (3 failures on one window raise) and logs an ERROR for interior gaps of >3 unscheduled bars (holidays look identical — verify, never fatal); (8) `_replay_span` returns the per-span GPU cache-miss count, `run_backtest` sums/logs it (must be 0) and writes `cache_fallbacks` to the `--record` manifest; (9) `regime_weight_table` raises on a partial `regime_weights_override` (every regime × all nine indicators); (10) Telegram boundary loops go through `_cached_or_scan`, `_remember_scan` files a report under the bar it analysed (label + duration), a breach inside 20 s after an M15 boundary waits for the loop's scan (15 duplicate M15 rows in 5 days before), and the rotating file handler is attached in `main()` only (importing the module in tests wrote fixture alerts into `logs/telegram.log`); (11) dashboard `catch_up_timeframes` re-scans any TF whose newest closed boundary is newer than its last scanned one (missed while a cycle overran or the process was down). Semantics recorded, NOT changed (rule 1): `blocked_hours_utc` is applied to the bar LABEL hour in the core (6b) AND the bar CLOSE hour in the live wrapper/backtester, so H1 can only trade bars closing 04–09, 15, 18–20 UTC and H4 only bars closing 08:00/12:00 — parity holds and every book was validated that way. Post-deploy: `scripts/backfill_signal_outcomes.py --reseed-unresolvable` floors the two Sep-8 shorts on the live DB; the `--stored-candles` scorecard (`analyze_signal_log.replay_stored`) still calls `advance_outcome` without the floor step (owner decision pending). - 2026-09-15 market outlook REMOVED (owner decision; plan `~/.claude/plans/you-are-quant-software-logical-cake.md`): `market_outlook.py`, the post-H1-scan Telegram 🔭 DM, `main.py --outlook`, the `outlooks` table DDL/migration/writer in `signal_log.py`, the `market_outlook` config block + `MARKET_OUTLOOK_ENABLED`, and test Module 26 are gone; the last commit containing the feature is `2836024`. A legacy top-level `market_outlook` block in config.json is TOLERATED and ignored with one warning (kept in `_KNOWN_TOP`, the `oanda.account_id` precedent) so the live bot/dashboard keep booting — delete the block from the live config.json at deploy. Existing `logs/signals.db` files keep their orphan `outlooks` table and rows (nothing reads or drops it; drop by hand during a weekend VACUUM if wanted); fresh DBs never create it. Module 26's 4-thread concurrent legacy-DB migration test (the suite's only one) moved to Module 31. Nothing in the signal path changed: the W2 books (2024-09-05→2026-09-05, $0.65, NumPy CPU, 0 cache fallbacks) are byte-identical before/after — M5 `29120a1ee32979431d062f210d41a9d1` (781 / +146.07R / PF 1.52), M15 `d185f099482a07f29502cb5a371a9bbf` (931 / +146.14R / PF 1.43). Suite 509 tests / 33 modules. +- 2026-09-15 backtest-directory cleanup (owner decisions, discussion-first; nothing in the signal core or backtester changed): REMOVED `backtest/diagnose_filters.py` (+ its 2 Module 18 tests), `scripts/shadow_scan.py` (never run), `scripts/backfill_news_archive.py` and `scripts/migrate_signal_log_news.py` (one-time scripts; the interning test kept its live-path half, the prune test went) — last commit containing all four is `931b265`. The replay-recording stack (`replay_record.py`, `replay_scorecard.py`, `scripts/ff_calendar_scraper.py`, `cache/calendar.parquet`, Module 33) STAYS. Local data (gitignored, so git has none of it) was MOVED, not deleted, to `~/Development/ForexBot2_trash_2026-09-15/` (346 MB) for the owner to delete: all 35 pinned `backtest/*.csv` books, the three research output folders (`cycle_2026-09-10/`, `replay_2026-09/`, `sizing_2026-09/`), the BTC/ETH `cache/snapshots/` (167 MB, unmerged crypto branches), the 4 Tiingo parquets and 40 uncited Oanda parquets (13 scratch/duplicate windows). The cache keeps 37 Oanda files (W3 `20230705`, W2 `20240705`, OOS `20240331`, M5-12mo `20250613`, M15/H1-12mo `20250603`, 5-yr `20210531`, 7.6-yr `20181031`, the Aug-14 parity window `20250814`, and their D1/H4 pads) + `calendar.parquet`. Verification: all five gate books regenerate offline from the pruned cache with the recorded MD5s — M5 W2 `29120a1e…`, M15 W2 `d185f099…`, M15 OOS `6f8f24b4…`, H1 W3 `3b45d12b…`, H4 W3 `e75adae1…` (NumPy CPU, 0 cache fallbacks, no network). Suite 506 tests / 33 modules. Pinned books are no longer kept on disk: regenerate from the cache when a trade-level diff is needed. diff --git a/backtest/diagnose_filters.py b/backtest/diagnose_filters.py deleted file mode 100644 index a8815c2..0000000 --- a/backtest/diagnose_filters.py +++ /dev/null @@ -1,627 +0,0 @@ -""" -ForexBot2 — Filter Rejection Diagnostic -======================================== -Replays the same backtest loop as gpu_backtest.py but instead of -collecting trades, counts how many M15 bars are rejected by each -gate in the signal pipeline. - -Produces a summary table showing where potential trades are being -lost, helping tune filter thresholds. - -Usage: - python backtest/diagnose_filters.py --pair XAUUSD --days 365 - python backtest/diagnose_filters.py --pair XAUUSD --days 365 --sample-every 1 -""" - -from __future__ import annotations - -import argparse -import logging -import sys -import time -from collections import Counter, defaultdict -from datetime import datetime, timedelta, timezone -from pathlib import Path -from typing import Dict, List, Optional, Tuple - -import numpy as np -import pandas as pd - -# Ensure project root on sys.path. -PROJECT_ROOT = Path(__file__).resolve().parent.parent -if str(PROJECT_ROOT) not in sys.path: - sys.path.insert(0, str(PROJECT_ROOT)) - -from config import ( # noqa: E402 - PAIR_INFO, - PlatformConfig, - SignalStrength, - Timeframe, -) -from signal_aggregator import SignalAggregator, SignalReport # noqa: E402 - -# Reuse GPU backtest infrastructure (data loading, indicator pre-compute, etc.) -from gpu_backtest import ( # noqa: E402 - BAR_SECONDS, - GPUTechnicalEngine, - PAIR_ALIASES, - IndicatorCache, - fetch_history, - precompute_indicators, - _is_xau_blocked_by_calendar, -) -from providers import make_data_provider # noqa: E402 - -logger = logging.getLogger("diagnose_filters") - - -# ═════════════════════════════════════════════════════════════════════ -# OUTCOME CLASSIFICATION -# ═════════════════════════════════════════════════════════════════════ - -def classify_bar( - report: SignalReport, - is_xau: bool, -) -> Tuple[str, List[str]]: - """Classify a bar's outcome and list which filters killed it. - - Returns: - (outcome, [filter_names]) - - Outcomes: - "TRADE" — quality C+ and trade_setup exists - "QUALITY_OK_NO_SETUP" — quality C+ but trade_setup is None - "LOW_QUALITY" — signal reached BUY/SELL but quality D or F - "NEUTRAL" — signal stayed NEUTRAL (score didn't reach +/-30) - - Filter names are extracted from report.warnings. - """ - filters_hit: List[str] = [] - - # Known filter tags, matched against the exact warning strings the - # aggregator emits (refreshed 2026-07-30 for gates 6d-6i). ORDER - # MATTERS: most-specific prefix first, and each warning maps to at - # most ONE tag (break on first hit) — substring families like - # "M5 FILTER: SELL quality" vs "M5 FILTER" and "TREND EXHAUSTION" - # vs "EXHAUSTION" would otherwise double-count. - filter_tags = [ - # M5 noise gates (6i, _apply_m5_filters) - ("M5 FILTER: |score|", "M5_SCORE"), - ("M5 FILTER: no MTF agreement", "M5_MTF"), - ("M5 FILTER: no confirmation TF", "M5_CONF"), - ("M5 FILTER: SELL quality", "M5_SELL_QUALITY"), - ("M5 FILTER: SELL blocked — RSI", "M5_SELL_RSI"), - ("M5 FILTER: '", "M5_VOL"), - ("M5 FILTER", "M5_OTHER"), - # XAU SELL gates (6e/6g/6g2/6g3) - ("SELL FILTER: quality", "SELL_QUALITY"), - ("SELL FILTER: volatility", "SELL_VOL"), - ("SELL FILTER: regime", "SELL_REGIME"), - ("UTC entry blocked for shorts", "SELL_HOUR"), - # Hard gates - ("MTF OPPOSITION", "MTF_OPPOSITION"), - ("EXHAUSTED TREND FILTER", "EXHAUSTED_TREND"), - ("BIG CANDLE FILTER", "BIG_CANDLE"), - ("WEEKEND FILTER", "WEEKEND_FILTER"), - ("SESSION FILTER", "SESSION_FILTER"), - ("DAY FILTER", "DAY_FILTER"), - ("LOW ATR FILTER", "LOW_ATR_FILTER"), - ("REGIME FILTER", "REGIME_FILTER"), - # Soft penalties / context warnings - ("TREND EXHAUSTION", "TREND_EXHAUSTION"), - ("EXHAUSTION", "EXHAUSTION_PENALTY"), - ("NO PULLBACK", "NO_PULLBACK_PENALTY"), - ("Weak pullback", "WEAK_PULLBACK_PENALTY"), - ("Extended — RSI", "RSI_EXTENDED_PENALTY"), - ("Counter-trend", "COUNTER_TREND_PENALTY"), - ("EXTREME volatility", "EXTREME_VOL_PENALTY"), - ("NEAR EMA200", "NEAR_EMA200_PENALTY"), - ("FAR FROM EMA200", "FAR_EMA200_PENALTY"), - ("EVENT RISK", "EVENT_RISK"), - ("Timeframes CONFLICT", "MTF_CONFLICT"), - ] - - for w in report.warnings: - for tag, name in filter_tags: - if tag in w: - filters_hit.append(name) - break - - # Determine outcome - non_neutral = report.final_signal not in (SignalStrength.NEUTRAL,) - quality_ok = report.signal_quality in ("A+", "A", "B", "C") - - if quality_ok and report.trade_setup is not None: - return "TRADE", filters_hit - elif quality_ok and report.trade_setup is None: - return "QUALITY_OK_NO_SETUP", filters_hit - elif non_neutral and not quality_ok: - return "LOW_QUALITY", filters_hit - else: - return "NEUTRAL", filters_hit - - -# ═════════════════════════════════════════════════════════════════════ -# SCORE DISTRIBUTION TRACKER -# ═════════════════════════════════════════════════════════════════════ - -class ScoreTracker: - """Tracks score/confidence distributions for analysis.""" - - def __init__(self): - self.scores: List[float] = [] - self.confidences: List[float] = [] - self.abs_scores: List[float] = [] - # How many bars had score in various ranges - self.score_bins = Counter() - self.conf_bins = Counter() - # Track "near miss" bars — score close to thresholds - self.near_miss_score: int = 0 # score 20-29 (almost BUY/SELL) - self.near_miss_conf: int = 0 # confidence 45-54 (almost passing) - - def record(self, score: float, confidence: float): - self.scores.append(score) - self.confidences.append(confidence) - self.abs_scores.append(abs(score)) - - # Score bins - a = abs(score) - if a < 10: - self.score_bins["0-9"] += 1 - elif a < 20: - self.score_bins["10-19"] += 1 - elif a < 30: - self.score_bins["20-29"] += 1 - self.near_miss_score += 1 - elif a < 40: - self.score_bins["30-39"] += 1 - elif a < 50: - self.score_bins["40-49"] += 1 - else: - self.score_bins["50+"] += 1 - - # Confidence bins - if confidence < 30: - self.conf_bins["0-29"] += 1 - elif confidence < 45: - self.conf_bins["30-44"] += 1 - elif confidence < 55: - self.conf_bins["45-54"] += 1 - self.near_miss_conf += 1 - elif confidence < 65: - self.conf_bins["55-64"] += 1 - elif confidence < 75: - self.conf_bins["65-74"] += 1 - else: - self.conf_bins["75+"] += 1 - - -# ═════════════════════════════════════════════════════════════════════ -# MAIN DIAGNOSTIC LOOP -# ═════════════════════════════════════════════════════════════════════ - -def run_diagnosis( - pair: str, - start: datetime, - end: datetime, - sample_every: int = 4, - use_cache: bool = True, - primary_tf: Timeframe = Timeframe.M15, - confirm_tfs: Optional[List[Timeframe]] = None, - warmup_bars: int = 250, -) -> None: - if confirm_tfs is None: - confirm_tfs = [Timeframe.H1, Timeframe.H4] - - pair_info = PAIR_INFO[pair] - display = pair_info["display"] - is_xau = display == "XAU/USD" - - platform_cfg = PlatformConfig( - symbol=pair, - display_name=display, - primary_timeframe=primary_tf, - confirmation_timeframes=list(confirm_tfs), - ) - - provider = make_data_provider(platform_cfg) - - # ── Load data ── - timeframes = [primary_tf] + list(confirm_tfs) - pad_days = max(60, warmup_bars // 4) - fetch_start = start - timedelta(days=pad_days) - - dfs_full: Dict[Timeframe, pd.DataFrame] = {} - for tf in timeframes: - df = fetch_history( - provider, pair, tf, - start=fetch_start, end=end, - use_cache=use_cache, - ) - if df.empty: - raise RuntimeError(f"No data for {pair} {tf.value}") - dfs_full[tf] = df - - # ── Pre-compute indicators ── - t0 = time.perf_counter() - caches: Dict[str, IndicatorCache] = { - tf.value: precompute_indicators(dfs_full[tf], platform_cfg.technical, gpu_log=True) - for tf in timeframes - } - logger.info(f"Indicator pre-compute: {time.perf_counter() - t0:.2f}s") - - # ── Build aggregator with GPU engine ── - aggregator = SignalAggregator(platform_cfg) - aggregator.tech_engine = GPUTechnicalEngine(platform_cfg.technical, caches) - - # ── Walk primary TF ── - primary_df = dfs_full[primary_tf] - bt_start_ts = pd.Timestamp(start) - bt_end_ts = pd.Timestamp(end) - - first_eligible = max( - warmup_bars, - int(primary_df.index.searchsorted(bt_start_ts)), - ) - last_pos = min( - int(primary_df.index.searchsorted(bt_end_ts, side="right")), - len(primary_df), - ) - - primary_dur = pd.Timedelta(seconds=BAR_SECONDS[primary_tf]) - - # Counters - total_bars_in_window = 0 - total_bars_sampled = 0 - outcome_counts = Counter() - filter_counts = Counter() - # Filters that killed bars that were otherwise directional (non-NEUTRAL score) - directional_filter_counts = Counter() - # Pre-pipeline rejections (XAU calendar filters applied before analyze_from_dataframes) - pre_pipeline = Counter() - - # Track signal directions - signal_dir_counts = Counter() - quality_counts = Counter() - - # Score tracker - tracker = ScoreTracker() - - # Track "would-have-been" trades: bars where the primary score was - # directional but got killed by a specific filter - filter_kill_breakdown = defaultdict(Counter) - - logger.info( - f"Diagnosing {display} {primary_tf.value}: " - f"bars {first_eligible}-{last_pos} (every {sample_every}th bar)" - ) - - t_loop = time.perf_counter() - for pos in range(first_eligible, last_pos, sample_every): - total_bars_in_window += 1 - ts = primary_df.index[pos] - decision_time = ts + primary_dur - - # ── XAU calendar pre-filter (same as gpu_backtest.py) ── - if is_xau: - block = _is_xau_blocked_by_calendar(ts, platform_cfg) - if block: - pre_pipeline[block] += 1 - continue - - total_bars_sampled += 1 - - # ── Build sliced dfs (same logic as gpu_backtest.py) ── - slice_dfs: Dict[Timeframe, pd.DataFrame] = {} - for tf in timeframes: - full = dfs_full[tf] - bar_dur = pd.Timedelta(seconds=BAR_SECONDS[tf]) - cutoff = decision_time - bar_dur - cut = int(full.index.searchsorted(cutoff, side="right")) - if cut == 0: - continue - slice_dfs[tf] = full.iloc[:cut] - - if primary_tf not in slice_dfs: - continue - - # ── Run the full signal pipeline ── - report: SignalReport = aggregator.analyze_from_dataframes( - symbol=pair, - dfs=slice_dfs, - context=None, - use_news=False, - ) - - # Record score/confidence - tracker.record(report.final_score, report.final_confidence) - - # Signal direction - signal_dir_counts[report.final_signal.name] += 1 - quality_counts[report.signal_quality] += 1 - - # Classify - outcome, filters_hit = classify_bar(report, is_xau) - outcome_counts[outcome] += 1 - - for f in filters_hit: - filter_counts[f] += 1 - - # If the primary score was directional (>= 15 or <= -15), the bar - # had some potential — track what killed it - if abs(report.primary_score) >= 15: - for f in filters_hit: - directional_filter_counts[f] += 1 - - # For bars that got quality D/F but had a directional signal, - # track which filter warnings appeared - if outcome in ("LOW_QUALITY", "QUALITY_OK_NO_SETUP"): - for f in filters_hit: - filter_kill_breakdown[outcome][f] += 1 - - elapsed = time.perf_counter() - t_loop - - # ═════════════════════════════════════════════════════════════════ - # PRINT RESULTS - # ═════════════════════════════════════════════════════════════════ - - bar = "=" * 80 - print(f"\n{bar}") - print(f" FILTER REJECTION DIAGNOSTIC — {display}") - print(f" {start.date()} to {end.date()} | {primary_tf.value} " - f"(sampled every {sample_every}th bar)") - print(f"{bar}") - - # Scale counts if sampling - scale_note = "" - if sample_every > 1: - scale_note = f" (sampled {total_bars_sampled} of ~{total_bars_in_window * sample_every} eligible bars)" - print(f"\n Total bars in backtest window: {total_bars_in_window}") - if pre_pipeline: - for reason, count in sorted(pre_pipeline.items()): - print(f" Pre-pipeline rejection ({reason}): {count}") - print(f" Bars analyzed: {total_bars_sampled}{scale_note}") - print(f" Analysis took: {elapsed:.1f}s " - f"({total_bars_sampled / max(elapsed, 0.001):.0f} bars/s)") - - # ── Outcome breakdown ── - print(f"\n{'-' * 80}") - print(" OUTCOME BREAKDOWN") - print(f"{'-' * 80}") - outcome_order = ["TRADE", "QUALITY_OK_NO_SETUP", "LOW_QUALITY", "NEUTRAL"] - outcome_labels = { - "TRADE": "Actual trades (quality C+ & setup exists)", - "QUALITY_OK_NO_SETUP": "Quality C+ but NO trade setup generated", - "LOW_QUALITY": "Directional signal but quality D/F", - "NEUTRAL": "NEUTRAL (score didn't reach +/-30 or conf < 55)", - } - for o in outcome_order: - count = outcome_counts.get(o, 0) - pct = count / max(total_bars_sampled, 1) * 100 - label = outcome_labels.get(o, o) - print(f" {count:6d} ({pct:5.1f}%) {label}") - - # ── Signal direction breakdown ── - print(f"\n{'-' * 80}") - print(" SIGNAL DIRECTION BREAKDOWN") - print(f"{'-' * 80}") - for name in ["STRONG_BUY", "BUY", "NEUTRAL", "SELL", "STRONG_SELL"]: - count = signal_dir_counts.get(name, 0) - pct = count / max(total_bars_sampled, 1) * 100 - print(f" {count:6d} ({pct:5.1f}%) {name}") - - # ── Quality grade breakdown ── - print(f"\n{'-' * 80}") - print(" QUALITY GRADE BREAKDOWN") - print(f"{'-' * 80}") - for grade in ["A+", "A", "B", "C", "D", "F"]: - count = quality_counts.get(grade, 0) - pct = count / max(total_bars_sampled, 1) * 100 - print(f" {count:6d} ({pct:5.1f}%) Grade {grade}") - - # ── Filter hit counts (all bars) ── - print(f"\n{'-' * 80}") - print(" FILTER HIT COUNTS (all bars, a bar can hit multiple filters)") - print(f"{'-' * 80}") - if filter_counts: - for name, count in filter_counts.most_common(): - pct = count / max(total_bars_sampled, 1) * 100 - print(f" {count:6d} ({pct:5.1f}%) {name}") - else: - print(" (no filters triggered)") - - # ── Filters that killed directional bars (primary_score >= 15) ── - print(f"\n{'-' * 80}") - print(" FILTERS KILLING DIRECTIONAL BARS (|primary_score| >= 15)") - print(f"{'-' * 80}") - if directional_filter_counts: - for name, count in directional_filter_counts.most_common(): - print(f" {count:6d} {name}") - else: - print(" (none)") - - # ── Filter breakdown for LOW_QUALITY bars ── - if filter_kill_breakdown.get("LOW_QUALITY"): - print(f"\n{'-' * 80}") - print(" FILTERS ON LOW_QUALITY BARS (had BUY/SELL but quality D/F)") - print(f"{'-' * 80}") - for name, count in filter_kill_breakdown["LOW_QUALITY"].most_common(): - print(f" {count:6d} {name}") - - # ── Filter breakdown for QUALITY_OK_NO_SETUP bars ── - if filter_kill_breakdown.get("QUALITY_OK_NO_SETUP"): - print(f"\n{'-' * 80}") - print(" FILTERS ON QUALITY_OK_NO_SETUP BARS (quality C+ but no trade setup)") - print(f"{'-' * 80}") - for name, count in filter_kill_breakdown["QUALITY_OK_NO_SETUP"].most_common(): - print(f" {count:6d} {name}") - - # ── Score distribution ── - print(f"\n{'-' * 80}") - print(" SCORE DISTRIBUTION (|score|)") - print(f"{'-' * 80}") - for bin_name in ["0-9", "10-19", "20-29", "30-39", "40-49", "50+"]: - count = tracker.score_bins.get(bin_name, 0) - pct = count / max(total_bars_sampled, 1) * 100 - marker = " <-- near miss (almost BUY/SELL)" if bin_name == "20-29" else "" - print(f" {count:6d} ({pct:5.1f}%) |score| {bin_name}{marker}") - - # ── Confidence distribution ── - print(f"\n{'-' * 80}") - print(" CONFIDENCE DISTRIBUTION") - print(f"{'-' * 80}") - for bin_name in ["0-29", "30-44", "45-54", "55-64", "65-74", "75+"]: - count = tracker.conf_bins.get(bin_name, 0) - pct = count / max(total_bars_sampled, 1) * 100 - marker = " <-- near miss (almost passing)" if bin_name == "45-54" else "" - print(f" {count:6d} ({pct:5.1f}%) confidence {bin_name}{marker}") - - # ── Near misses ── - print(f"\n{'-' * 80}") - print(" NEAR MISSES") - print(f"{'-' * 80}") - print(f" {tracker.near_miss_score:6d} Bars with |score| 20-29 (threshold is 30)") - print(f" {tracker.near_miss_conf:6d} Bars with confidence 45-54 (threshold is 55)") - - # ── Summary statistics ── - if tracker.scores: - scores_arr = np.array(tracker.scores) - conf_arr = np.array(tracker.confidences) - print(f"\n{'-' * 80}") - print(" SCORE/CONFIDENCE STATISTICS") - print(f"{'-' * 80}") - print(f" Score: mean={np.mean(scores_arr):+.1f} " - f"median={np.median(scores_arr):+.1f} " - f"std={np.std(scores_arr):.1f} " - f"min={np.min(scores_arr):+.1f} " - f"max={np.max(scores_arr):+.1f}") - print(f" Confidence: mean={np.mean(conf_arr):.1f} " - f"median={np.median(conf_arr):.1f} " - f"std={np.std(conf_arr):.1f} " - f"min={np.min(conf_arr):.1f} " - f"max={np.max(conf_arr):.1f}") - - # ── Blocked hours analysis (for XAU) ── - if is_xau: - blocked = set(platform_cfg.xau_strategy.blocked_hours_utc) - all_hours = set(range(24)) - open_hours = sorted(all_hours - blocked) - print(f"\n{'-' * 80}") - print(" XAU HOUR/DAY CONFIG") - print(f"{'-' * 80}") - print(f" Blocked hours: {sorted(blocked)}") - print(f" Open hours: {open_hours}") - print(f" Allowed days: {platform_cfg.xau_strategy.allowed_weekdays} " - f"(0=Mon..4=Fri)") - print(f" Open fraction: {len(open_hours)/24:.0%} of hours x " - f"{len(platform_cfg.xau_strategy.allowed_weekdays)}/5 weekdays = " - f"{len(open_hours)/24 * len(platform_cfg.xau_strategy.allowed_weekdays)/5:.0%} of trading time") - - # ── Interpretation / key takeaways ── - print(f"\n{bar}") - print(" KEY TAKEAWAYS") - print(f"{bar}") - - neutral_pct = outcome_counts.get("NEUTRAL", 0) / max(total_bars_sampled, 1) * 100 - low_q_pct = outcome_counts.get("LOW_QUALITY", 0) / max(total_bars_sampled, 1) * 100 - trade_count = outcome_counts.get("TRADE", 0) - - print(f" - {neutral_pct:.1f}% of bars never reach a directional signal (NEUTRAL)") - print(f" - {low_q_pct:.1f}% of bars get a BUY/SELL but confidence/quality too low") - print(f" - {trade_count} bars produce actual trades") - - if filter_counts: - top3 = filter_counts.most_common(3) - print(f" - Top rejection filters:") - for name, count in top3: - print(f" {name}: {count} bars hit") - - if tracker.near_miss_score > 50: - print(f" - SIGNIFICANT: {tracker.near_miss_score} bars had |score| 20-29 " - f"(lowering threshold from 30 to 25 would capture many)") - - if tracker.near_miss_conf > 50: - print(f" - SIGNIFICANT: {tracker.near_miss_conf} bars had confidence 45-54 " - f"(lowering threshold from 55 to 50 would capture many)") - - print(f"\n{bar}\n") - - -# ═════════════════════════════════════════════════════════════════════ -# CLI -# ═════════════════════════════════════════════════════════════════════ - -def _resolve_pair(s: str) -> str: - s_up = s.upper().strip() - if s_up in PAIR_ALIASES: - return PAIR_ALIASES[s_up] - if s_up in PAIR_INFO: - return s_up - raise SystemExit(f"Unknown pair '{s}'. Valid: {', '.join(sorted(PAIR_ALIASES))}") - - -def _resolve_tf(s: str) -> Timeframe: - for tf in Timeframe: - if tf.value == s: - return tf - raise SystemExit(f"Unknown timeframe '{s}'") - - -def main(argv: Optional[List[str]] = None) -> int: - p = argparse.ArgumentParser( - description="Diagnose which filters reject the most bars in the backtest pipeline", - formatter_class=argparse.ArgumentDefaultsHelpFormatter, - ) - p.add_argument("--pair", required=True, help="XAUUSD | GOLD") - p.add_argument("--days", type=int, default=365, help="Days of history") - p.add_argument("--start", help="ISO date YYYY-MM-DD (overrides --days)") - p.add_argument("--end", help="ISO date YYYY-MM-DD (defaults to today)") - p.add_argument("--primary-tf", default="15min", help="Primary TF") - p.add_argument("--confirm-tfs", default="1h,4h", help="Comma-separated confirmation TFs") - p.add_argument("--warmup-bars", type=int, default=250) - p.add_argument("--sample-every", type=int, default=4, - help="Analyze every Nth bar (1=all, 4=25%% sample for speed)") - p.add_argument("--no-cache", action="store_true", help="Skip parquet cache") - p.add_argument("--log-level", default="INFO") - args = p.parse_args(argv) - - logging.basicConfig( - level=args.log_level.upper(), - format="%(asctime)s [%(levelname)s] %(name)s: %(message)s", - ) - - pair = _resolve_pair(args.pair) - primary_tf = _resolve_tf(args.primary_tf) - confirm_tfs = [_resolve_tf(s.strip()) for s in args.confirm_tfs.split(",") if s.strip()] - - end = ( - datetime.fromisoformat(args.end).replace(tzinfo=timezone.utc) - if args.end else datetime.now(timezone.utc) - ) - if args.start: - start = datetime.fromisoformat(args.start).replace(tzinfo=timezone.utc) - else: - start = end - timedelta(days=args.days) - - logger.info( - f"Diagnosing {PAIR_INFO[pair]['display']} | " - f"Window: {start.date()} -> {end.date()} | " - f"Sample: every {args.sample_every}th bar" - ) - - run_diagnosis( - pair=pair, - start=start, - end=end, - sample_every=args.sample_every, - use_cache=not args.no_cache, - primary_tf=primary_tf, - confirm_tfs=confirm_tfs, - warmup_bars=args.warmup_bars, - ) - - return 0 - - -if __name__ == "__main__": - sys.exit(main()) diff --git a/scripts/backfill_news_archive.py b/scripts/backfill_news_archive.py deleted file mode 100755 index ed1f954..0000000 --- a/scripts/backfill_news_archive.py +++ /dev/null @@ -1,230 +0,0 @@ -#!/usr/bin/env python3 -""" -Seed logs/newsdata.db from the headline blobs still on disk. -============================================================== - -The durable headline corpus (``news_archive.py``) only started collecting -when its hook shipped. Everything before that survives only inside signal -logs, in two different shapes: - -* ``logs/signals.db`` — modern schema, headlines interned into - ``news_snapshots`` (kind ``headlines``) and referenced by hash. -* an older orphan DB (default ``~/signals.db``) — pre-interning, blobs - inline in ``signals.headlines_json``, short-form item keys with no - ``description``, ``link``, ``publisher`` or ``score_by_currency``. - -Both are opened READ-ONLY. The script is idempotent: re-running it adds -nothing new and never downgrades a score. - -Provenance is RECOMPUTED, not trusted -------------------------------------- -Neither source records which scorer produced its ``score`` — that flag -only exists from this change onwards. So each headline is re-scored with -the live keyword scorer and compared: - -* stored score differs from the keyword score -> ``llm`` (DeepSeek wrote it) -* exact match on a non-zero score -> ``keyword`` -* both zero -> ``unknown`` (indistinguishable) - -Rows are stamped ``origin='backfill'`` because they come from the -post-filter pool: ``get_fundamental_context`` drops headlines inside -±0.1 before they ever reach a signal row, so genuinely neutral headlines -are under-represented in this history. Rows collected live by the hook -in ``news_engine`` do not have that bias and are stamped ``'live'``. - -Usage ------ - venv/bin/python scripts/backfill_news_archive.py --dry-run - venv/bin/python scripts/backfill_news_archive.py - venv/bin/python scripts/backfill_news_archive.py --legacy-db /path/to/old.db -""" - -from __future__ import annotations - -import argparse -import json -import os -import sqlite3 -import sys -from collections import Counter -from typing import Any, Dict, Iterator, Tuple - -sys.path.insert(0, os.path.dirname(os.path.dirname(os.path.abspath(__file__)))) - -import app_config # noqa: E402 -import news_archive # noqa: E402 -from config import NewsConfig # noqa: E402 -from news_engine import NewsEngine # noqa: E402 - - -def _open_ro(path: str) -> sqlite3.Connection: - """Read-only, but NOT immutable=1 — that flag is blind to WAL rows.""" - return sqlite3.connect(f"file:{os.path.abspath(path)}?mode=ro", uri=True) - - -def _has_table(con: sqlite3.Connection, name: str) -> bool: - row = con.execute( - "SELECT 1 FROM sqlite_master WHERE type='table' AND name=?", - (name,)).fetchone() - return row is not None - - -def iter_blobs(path: str) -> Iterator[Tuple[str, str]]: - """Yield (headlines_json, seen_at) from either DB shape.""" - con = _open_ro(path) - try: - if _has_table(con, "news_snapshots"): - yield from con.execute( - "SELECT body, first_seen FROM news_snapshots" - " WHERE kind='headlines' ORDER BY first_seen") - cols = {r[1] for r in con.execute("PRAGMA table_info(signals)")} - if "headlines_json" in cols: - order = "logged_at" if "logged_at" in cols else "id" - yield from con.execute( - "SELECT headlines_json, " - + ("logged_at" if "logged_at" in cols else "''") - + " FROM signals WHERE headlines_json IS NOT NULL" - f" ORDER BY {order}") - finally: - con.close() - - -class _Row: - """Duck-typed stand-in for NewsSentiment, so news_archive.build_row - flattens a stored dict exactly the way it flattens a live headline.""" - - __slots__ = ("headline", "source", "timestamp", "currencies_mentioned", - "sentiment_score", "keywords", "sentiment_by_currency", - "link", "description", "publisher", "score_source") - - def __init__(self, item: Dict[str, Any], score_source: str): - self.headline = item.get("headline") - self.source = item.get("source") - self.timestamp = item.get("timestamp") - self.currencies_mentioned = item.get("currencies") or [] - self.sentiment_score = item.get("score") - self.keywords = item.get("keywords") or [] - self.sentiment_by_currency = item.get("score_by_currency") or {} - self.link = item.get("link") - self.description = item.get("description") - self.publisher = item.get("publisher") - self.score_source = score_source - - -def classify(engine: NewsEngine, item: Dict[str, Any]) -> str: - """Recover which scorer produced this stored score.""" - text = item.get("headline") - if not isinstance(text, str) or not text.strip(): - return "unknown" - try: - stored = float(item.get("score") or 0.0) - except (TypeError, ValueError): - return "unknown" - keyword = engine._analyze_headline_sentiment(text) - if abs(stored - keyword) > 1e-9: - return "llm" - if abs(stored) < 1e-9: - return "unknown" # both zero — the two scorers are indistinguishable - return "keyword" - - -def collect(paths, engine: NewsEngine): - """Newest wins per headline: later blobs carry upgraded LLM scores.""" - best: Dict[str, Tuple[str, Dict[str, Any], str]] = {} - stats = Counter() - for path in paths: - if not os.path.exists(path): - print(f" skip (not found): {path}") - continue - blobs = 0 - for body, seen_at in iter_blobs(path): - try: - items = json.loads(body) - except Exception: - stats["bad_blob"] += 1 - continue - blobs += 1 - for item in items: - if not isinstance(item, dict): - continue - text = item.get("headline") - if not isinstance(text, str) or not text.strip(): - continue - source = classify(engine, item) - prior = best.get(text) - if prior is None or ( - news_archive._SOURCE_RANK.get(source, 0) - >= news_archive._SOURCE_RANK.get(prior[0], 0)): - best[text] = (source, item, seen_at or "") - stats["items"] += 1 - print(f" {path}: {blobs} blobs") - for source, _, _ in best.values(): - stats[source] += 1 - return best, stats - - -def main() -> int: - default_legacy = os.path.join(os.path.expanduser("~"), "signals.db") - ap = argparse.ArgumentParser(description=__doc__.split("\n")[1]) - ap.add_argument("--db", default=app_config.signal_log_path(), - help="modern signal-log DB (default: config path)") - ap.add_argument("--legacy-db", default=default_legacy, - help=f"older inline-blob DB (default: {default_legacy})") - ap.add_argument("--out", default=app_config.news_archive_path(), - help="archive DB to write (default: config path)") - ap.add_argument("--dry-run", action="store_true", - help="count only, write nothing") - args = ap.parse_args() - - engine = NewsEngine(NewsConfig()) - print("Reading headline blobs (read-only):") - best, stats = collect([args.db, args.legacy_db], engine) - - print(f"\n headline items scanned : {stats['items']}") - print(f" distinct headlines : {len(best)}") - print(f" llm (DeepSeek) : {stats['llm']}") - print(f" keyword : {stats['keyword']}") - print(f" unknown (both zero) : {stats['unknown']}") - if stats["bad_blob"]: - print(f" unparseable blobs : {stats['bad_blob']}") - - if args.dry_run: - print("\n--dry-run: nothing written.") - return 0 - if not best: - print("\nNothing to write.") - return 0 - - rows = [] - for text, (source, item, seen_at) in best.items(): - row = news_archive.build_row( - _Row(item, source), - teacher_model=None, # not recorded by either source - teacher_prompt_sha1=None, - origin="backfill", - first_seen=seen_at or None, - last_seen=seen_at or None) - if row is not None: - rows.append(row) - - archive = news_archive.NewsArchive(args.out) - try: - written = archive.store(rows) - finally: - archive.close() - print(f"\nWrote {written} rows to {args.out}") - - con = _open_ro(args.out) - try: - print("\nArchive now holds:") - for source, count in con.execute( - "SELECT score_source, COUNT(*) FROM headlines" - " GROUP BY score_source ORDER BY COUNT(*) DESC"): - print(f" {source:8s} {count}") - finally: - con.close() - return 0 - - -if __name__ == "__main__": - raise SystemExit(main()) diff --git a/scripts/migrate_signal_log_news.py b/scripts/migrate_signal_log_news.py deleted file mode 100644 index 4791e30..0000000 --- a/scripts/migrate_signal_log_news.py +++ /dev/null @@ -1,177 +0,0 @@ -#!/usr/bin/env python3 -""" -migrate_signal_log_news.py — move inline news blobs into news_snapshots. - -Legacy rows store the full RSS headline list (~33 KB) and calendar events -(~1.2 KB) inline on EVERY scan row; the same blob repeats ~5× because the -news caches outlive a candle. This script interns each distinct blob once -(content hash), stamps `headlines_hash` / `events_hash` on the row and -NULLs the inline columns. Readers use the `signals_with_news` view or -`signal_log.read_news`. - -Idempotent and resumable (batched transactions). Run --vacuum with BOTH -live writers (telegram bot, dashboard publisher) stopped — VACUUM needs -exclusive access and ~1× the final size of free disk. - - python scripts/migrate_signal_log_news.py --dry-run - python scripts/migrate_signal_log_news.py - python scripts/migrate_signal_log_news.py --vacuum # bots stopped -""" -import argparse -import hashlib -import os -import sqlite3 -import sys -from datetime import datetime, timezone - -sys.path.insert(0, os.path.dirname(os.path.dirname(os.path.abspath(__file__)))) - -import app_config # noqa: E402 -from signal_log import SignalLogger # noqa: E402 - - -def _hash(text: str) -> str: - return hashlib.sha256(text.encode("utf-8")).hexdigest() - - -def migrate(con: sqlite3.Connection, batch: int = 500) -> dict: - """Intern legacy inline blobs; returns counters. Caller owns the - connection (schema must already exist — open a SignalLogger once).""" - stats = {"rows": 0, "headlines_interned": 0, "events_interned": 0} - now = datetime.now(timezone.utc).isoformat() - while True: - rows = con.execute( - "SELECT id, headlines_json, events_json FROM signals" - " WHERE (headlines_json IS NOT NULL AND headlines_hash IS NULL)" - " OR (events_json IS NOT NULL AND events_hash IS NULL)" - " ORDER BY id LIMIT ?", (batch,)).fetchall() - if not rows: - break - con.execute("BEGIN IMMEDIATE") - try: - for rowid, headlines, events in rows: - sets, params = [], [] - if headlines is not None: - h = _hash(headlines) - cur = con.execute( - "INSERT OR IGNORE INTO news_snapshots" - " (hash, kind, body, first_seen, last_seen)" - " VALUES (?, 'headlines', ?, ?, ?)", - (h, headlines, now, now)) - stats["headlines_interned"] += cur.rowcount - sets += ["headlines_hash=?", "headlines_json=NULL"] - params.append(h) - if events is not None: - h = _hash(events) - cur = con.execute( - "INSERT OR IGNORE INTO news_snapshots" - " (hash, kind, body, first_seen, last_seen)" - " VALUES (?, 'events', ?, ?, ?)", - (h, events, now, now)) - stats["events_interned"] += cur.rowcount - sets += ["events_hash=?", "events_json=NULL"] - params.append(h) - params.append(rowid) - con.execute( - f"UPDATE signals SET {', '.join(sets)} WHERE id=?", - params) - stats["rows"] += 1 - con.commit() - except Exception: - con.rollback() - raise - return stats - - -def prune_orphans(con: sqlite3.Connection) -> int: - cur = con.execute( - "DELETE FROM news_snapshots WHERE hash NOT IN" - " (SELECT headlines_hash FROM signals WHERE headlines_hash IS NOT NULL" - " UNION SELECT events_hash FROM signals WHERE events_hash IS NOT NULL)") - con.commit() - return cur.rowcount - - -def prune_news_older_than(con: sqlite3.Connection, days: int) -> int: - """Drop the news reference (not the row) from rows older than N days, - then delete snapshots nothing references any more.""" - # logged_at is ISO8601 with a 'T' and a +00:00 offset; SQLite's - # datetime() returns 'YYYY-MM-DD HH:MM:SS'. Comparing them as raw - # strings put every row from the cutoff CALENDAR DAY on the wrong - # side ('T' > ' '), so a full day never pruned. Normalise to the - # same shape first — all logged_at values are UTC (see _iso()). - cur = con.execute( - "UPDATE signals SET headlines_hash=NULL, events_hash=NULL" - " WHERE substr(replace(logged_at, 'T', ' '), 1, 19)" - " < datetime('now', ?)", (f"-{int(days)} days",)) - con.commit() - prune_orphans(con) - return cur.rowcount - - -def main(argv=None) -> int: - ap = argparse.ArgumentParser(description=__doc__.splitlines()[1]) - ap.add_argument("--db", default=None) - ap.add_argument("--dry-run", action="store_true") - ap.add_argument("--batch", type=int, default=500) - ap.add_argument("--prune-news-older-than", type=int, default=None, - metavar="DAYS") - ap.add_argument("--prune-orphans", action="store_true") - ap.add_argument("--vacuum", action="store_true", - help="checkpoint + VACUUM (all writers must be stopped)") - ap.add_argument("--no-backup", action="store_true") - args = ap.parse_args(argv) - path = args.db or app_config.signal_log_path() - if not os.path.exists(path): - print(f"no database at {path}") - return 1 - size0 = os.path.getsize(path) - if args.dry_run: - con = sqlite3.connect(f"file:{os.path.abspath(path)}?mode=ro", uri=True) - n = con.execute( - "SELECT COUNT(*) FROM signals WHERE headlines_json IS NOT NULL" - " OR events_json IS NOT NULL").fetchone()[0] - d = con.execute( - "SELECT COUNT(DISTINCT headlines_json) FROM signals" - " WHERE headlines_json IS NOT NULL").fetchone()[0] - print(f"{n} rows carry inline news blobs ({d} distinct headline " - f"blobs); file size {size0 / 1e6:.1f} MB") - con.close() - return 0 - if not args.no_backup: - backup = f"{path}.pre-news-{datetime.now(timezone.utc):%Y%m%d}" - if not os.path.exists(backup): - src = sqlite3.connect(path) - dst = sqlite3.connect(backup) - src.backup(dst) - dst.close() - src.close() - print(f"backup written: {backup}") - lg = SignalLogger(path) # applies the additive schema - lg.close() - con = sqlite3.connect(path) - con.execute("PRAGMA busy_timeout=10000") - try: - print("migrate:", migrate(con, batch=args.batch)) - if args.prune_news_older_than is not None: - print("rows de-referenced:", - prune_news_older_than(con, args.prune_news_older_than)) - if args.prune_orphans: - print("orphan snapshots removed:", prune_orphans(con)) - if args.vacuum: - try: - con.execute("PRAGMA wal_checkpoint(TRUNCATE)") - con.execute("VACUUM") - except sqlite3.OperationalError as exc: - print(f"VACUUM failed ({exc}) — stop the bot and dashboard " - "publisher first, then re-run with --vacuum") - return 2 - finally: - con.close() - print(f"size {size0 / 1e6:.1f} MB -> {os.path.getsize(path) / 1e6:.1f} MB" - + ("" if args.vacuum else " (run --vacuum to reclaim space)")) - return 0 - - -if __name__ == "__main__": - sys.exit(main()) diff --git a/scripts/shadow_scan.py b/scripts/shadow_scan.py deleted file mode 100644 index 516e15b..0000000 --- a/scripts/shadow_scan.py +++ /dev/null @@ -1,192 +0,0 @@ -#!/usr/bin/env python3 -""" -Phase 4 — live shadow scan (data source change test) -===================================================== -Runs TWO data providers side-by-side at every candle close (M5/M15/H1) -and logs how their signals diverge, so the feed switch can be validated -under real fetch depths / caching / boundary timing for ≥2 weeks. - -Unlike the history studies, this uses the LIVE fetch path (fetch_ohlcv, -forming-candle drop, freshness checks) for both providers, but calls -`analyze_from_dataframes(use_news=False)` directly so the (provider- -independent) news/DeepSeek budget is not burned on shadow scans. The -weekend filter is applied here (skip the whole cycle), and the session/ -day/hour gates run inside analyze_from_dataframes as in the backtest. - -No orders, no Telegram, no signal-log writes — output is a side-by-side -CSV plus a rolling agreement tally. - -Usage: - venv/bin/python scripts/shadow_scan.py --a oanda --b tiingo \ - --tfs M5,M15,H1 --out logs/shadow_oanda_tiingo.csv --once - # production (2+ weeks): - nohup venv/bin/python scripts/shadow_scan.py --a oanda --b tiingo \ - --out logs/shadow_oanda_tiingo.csv >> logs/shadow_scan.log 2>&1 & - -CSV columns (one row per TF per boundary): - ts, tf, a_, b_ for direction/grade/setup/tag/entry/sl/tp1, - a_price, b_price, a_spread, b_spread, a_warnings, b_warnings, - dir_agree, grade_agree, setup_agree -""" - -import argparse -import csv -import sys -import time -from datetime import datetime, timezone -from pathlib import Path - -sys.path.insert(0, str(Path(__file__).resolve().parent.parent)) - -from config import PlatformConfig, Timeframe, XAUUSD -from pair_signals.xauusd import make_config -from providers import make_data_provider -from providers.base import _is_forex_weekend -from signal_aggregator import SignalAggregator, SignalReport - -_TF_LOOKUP = {"M5": Timeframe.M5, "M15": Timeframe.M15, "H1": Timeframe.H1, - "H4": Timeframe.H4} -_TF_MINUTES = {Timeframe.M5: 5, Timeframe.M15: 15, Timeframe.H1: 60, - Timeframe.H4: 240} -_CONFIRMS = {Timeframe.M5: [Timeframe.M15, Timeframe.H1], - Timeframe.M15: [Timeframe.M5, Timeframe.H1], - Timeframe.H1: [Timeframe.M15, Timeframe.H4], - Timeframe.H4: [Timeframe.H1, Timeframe.D1]} - -FIELDS = ["direction", "grade", "setup", "tag", "entry", "sl", "tp1"] - - -def seconds_until_boundary(interval_minutes: int, buffer: int = 10) -> float: - now = datetime.now(timezone.utc) - elapsed = (now.minute % interval_minutes) * 60 + now.second \ - + now.microsecond / 1e6 - return interval_minutes * 60 - elapsed + buffer - - -def due_timeframes(wake: datetime, tfs) -> list: - minutes = wake.hour * 60 + wake.minute - floored = minutes - (minutes % 5) - return [tf for tf in tfs if floored % _TF_MINUTES[tf] == 0] - - -def tag_of(report) -> str: - fb = report.bullish_reasons[0] if report.bullish_reasons else "" - fbe = report.bearish_reasons[0] if report.bearish_reasons else "" - if "XAU PULLBACK" in fb or "XAU PULLBACK" in fbe: - return "XAU_PB" - if "XAU CHART PATTERN" in fb or "XAU CHART PATTERN" in fbe: - return "XAU_CP" - if "XAU STRATEGY" in fb: - return "XAU" - return "GENERIC" - - -def scan(provider, symbol, tf) -> SignalReport: - confirms = _CONFIRMS[tf] - cfg = make_config(tf, confirms) - agg = SignalAggregator(cfg) - agg.data_provider = provider - dfs = {} - for t in [tf] + confirms: - dfs[t] = provider.fetch_ohlcv(symbol, t) - return agg.analyze_from_dataframes(symbol, dfs, use_news=False) - - -def extract(provider, report) -> dict: - s = report.trade_setup - return { - "direction": report.final_signal.value, - "grade": report.signal_quality, - "setup": 1 if s is not None else 0, - "tag": tag_of(report) if s else "", - "entry": round(s.entry_price, 3) if s else "", - "sl": round(s.stop_loss, 3) if s else "", - "tp1": round(s.take_profit_1, 3) if s else "", - "price": round(report.current_price, 3), - "spread": round(provider.get_spread(XAUUSD), 3), - "warnings": " | ".join(report.warnings)[:200], - } - - -def main(): - ap = argparse.ArgumentParser() - ap.add_argument("--a", default="oanda") - ap.add_argument("--b", default="tiingo") - ap.add_argument("--tfs", default="M5,M15,H1") - ap.add_argument("--out", default="logs/shadow_scan.csv") - ap.add_argument("--once", action="store_true", - help="one immediate scan cycle, then exit") - args = ap.parse_args() - - tfs = [_TF_LOOKUP[s.strip()] for s in args.tfs.split(",")] - pa = make_data_provider(PlatformConfig(), args.a) - pb = make_data_provider(PlatformConfig(), args.b) - - out_path = Path(args.out) - out_path.parent.mkdir(parents=True, exist_ok=True) - new_file = not out_path.exists() - tally = {"cycles": 0, "dir": 0, "grade": 0, "setup": 0} - - with open(out_path, "a", newline="") as fh: - w = csv.writer(fh) - if new_file: - w.writerow(["ts", "tf"] + [f"a_{x}" for x in FIELDS + ["price", "spread", "warnings"]] - + [f"b_{x}" for x in FIELDS + ["price", "spread", "warnings"]] - + ["dir_agree", "grade_agree", "setup_agree"]) - - while True: - if args.once: - wake = datetime.now(timezone.utc) - else: - wait = seconds_until_boundary(5, buffer=10) - time.sleep(wait) - wake = datetime.now(timezone.utc) - - if _is_forex_weekend(): - if args.once: - print("weekend — nothing to scan") - return - continue - - for tf in due_timeframes(wake, tfs): - try: - ra = scan(pa, XAUUSD, tf) - except Exception as e: - print(f"[{wake:%H:%M:%S}] {args.a} {tf.value} FAILED: {e}") - continue - try: - rb = scan(pb, XAUUSD, tf) - except Exception as e: - print(f"[{wake:%H:%M:%S}] {args.b} {tf.value} FAILED: {e}") - continue - - ea, eb = extract(pa, ra), extract(pb, rb) - da = ea["direction"] == eb["direction"] - ga = ea["grade"] == eb["grade"] - sa = ea["setup"] == eb["setup"] - tally["cycles"] += 1 - tally["dir"] += da; tally["grade"] += ga; tally["setup"] += sa - - row = [wake.isoformat(), tf.value] - row += [ea[x] for x in FIELDS + ["price", "spread", "warnings"]] - row += [eb[x] for x in FIELDS + ["price", "spread", "warnings"]] - row += [int(da), int(ga), int(sa)] - w.writerow(row) - fh.flush() - - print(f"[{wake:%Y-%m-%d %H:%M:%S}] {tf.value:<5} " - f"{args.a}={ea['direction']:+d}/{ea['grade'] or '-':>2} " - f"{args.b}={eb['direction']:+d}/{eb['grade'] or '-':>2} " - f"dir={'OK' if da else 'FLIP'} " - f"| Δprice {abs(ea['price']-eb['price']):.2f} " - f"| agree dir {tally['dir']}/{tally['cycles']} " - f"grade {tally['grade']}/{tally['cycles']} " - f"setup {tally['setup']}/{tally['cycles']}") - - if args.once: - print("shadow scan complete (--once)") - return - - -if __name__ == "__main__": - main() diff --git a/test_all.py b/test_all.py index c467c5b..83ba3bd 100644 --- a/test_all.py +++ b/test_all.py @@ -4544,10 +4544,10 @@ def _(): # ══════════════════════════════════════════════════════════════════════ -# MODULE 18: --set research override + diagnose_filters tag freshness +# MODULE 18: --set research override # ══════════════════════════════════════════════════════════════════════ -print("\n═══ Module 18: --set override + diagnose tags ═══") +print("\n═══ Module 18: --set override ═══") @test("--set: nested float/bool/JSON-list overrides apply with typing") @@ -4603,42 +4603,6 @@ def _(): assert _parse_args(["--pair", "XAUUSD"]).set == [] -@test("diagnose_filters: every filter tag has a live emitter (staleness guard)") -def _(): - import inspect - import diagnose_filters - import signal_aggregator as sa - src = inspect.getsource(sa) - tag_src = inspect.getsource(diagnose_filters.classify_bar) - # Extract the tag strings from the ordered list in classify_bar. - import re - tags = re.findall(r'\("([^"]+)",\s+"[A-Z0-9_]+"\)', tag_src) - assert len(tags) >= 25, f"tag extraction broke: found {len(tags)}" - for tag in tags: - assert tag in src, f"filter tag '{tag}' has no emitter in signal_aggregator.py" - - -@test("diagnose_filters: one tag per warning, specific prefix wins") -def _(): - import diagnose_filters as d - from signal_aggregator import SignalReport - - class _R: - pass - r = _R() - r.warnings = [ - "M5 FILTER: SELL quality A capped — high-conviction M5 shorts enter at extremes", - "SELL FILTER: 15:00 UTC entry blocked for shorts", - "TREND EXHAUSTION — price extended >3.5 ATR past swing high", - ] - r.final_signal = SignalStrength.SELL - r.signal_quality = "D" - r.trade_setup = None - outcome, hits = d.classify_bar(r, True) - assert outcome == "LOW_QUALITY" - assert hits == ["M5_SELL_QUALITY", "SELL_HOUR", "TREND_EXHAUSTION"], hits - - # ══════════════════════════════════════════════════════════════════════ # MODULE 19: DeepSeek news enrichment (llm_client + news_engine seams) # ══════════════════════════════════════════════════════════════════════ @@ -8827,7 +8791,6 @@ def _(): def _(): import sqlite3 import signal_log as slog - from scripts.migrate_signal_log_news import migrate, prune_orphans fd, path = tempfile.mkstemp(suffix=".db") os.close(fd) try: @@ -8859,19 +8822,6 @@ def _(): "SELECT id, headlines_resolved FROM signals_with_news").fetchall()) assert json.loads(view[legacy_id])[0]["headline"] == "legacy" assert json.loads(view[r2])[0]["headline"] == "Fed signals rate hike" - # migration moves the legacy blob and is idempotent - stats = migrate(lg._conn) - assert stats["rows"] == 1 and stats["headlines_interned"] == 1 - assert migrate(lg._conn)["rows"] == 0 - row = lg._conn.execute( - "SELECT headlines_json, headlines_hash FROM signals WHERE id=?", - (legacy_id,)).fetchone() - assert row[0] is None and row[1] - assert json.loads(slog.read_news(lg._conn, legacy_id)[1])[0][ - "headline"] == "legacy" - lg._conn.execute("DELETE FROM signals WHERE id=?", (legacy_id,)) - lg._conn.commit() - assert prune_orphans(lg._conn) == 2, "legacy headline + event blobs" lg.close() finally: slog.reset_loggers_for_tests() @@ -9127,46 +9077,6 @@ def _(): "a real evaluator fault must be visible in the log" -@test("migrate: prune_news_older_than prunes the cutoff calendar day") -def _(): - import sqlite3 - import importlib.util - spec = importlib.util.spec_from_file_location( - "_mig_test", os.path.join(os.path.dirname(os.path.abspath(__file__)), - "scripts", "migrate_signal_log_news.py")) - mig = importlib.util.module_from_spec(spec) - spec.loader.exec_module(mig) - - fd, path = tempfile.mkstemp(suffix=".db") - os.close(fd) - try: - con = sqlite3.connect(path) - con.execute("CREATE TABLE signals (id INTEGER PRIMARY KEY," - " logged_at TEXT, headlines_hash TEXT, events_hash TEXT)") - con.execute("CREATE TABLE news_snapshots (hash TEXT PRIMARY KEY," - " body TEXT)") - now = datetime.now(timezone.utc) - # Genuinely older than the 30-day cutoff, but on the SAME calendar - # day as it — the case the raw string compare put on the wrong side - # ('T' sorts after ' '). - boundary = (now - timedelta(days=30)).replace( - hour=0, minute=0, second=1, microsecond=0).isoformat() - con.executemany( - "INSERT INTO signals VALUES (?,?,?,?)", - [(1, boundary, "h1", "e1"), (2, now.isoformat(), "h2", "e2")]) - con.executemany("INSERT INTO news_snapshots VALUES (?,?)", - [("h1", "{}"), ("e1", "{}"), ("h2", "{}"), ("e2", "{}")]) - con.commit() - n = mig.prune_news_older_than(con, 30) - assert n == 1, f"the cutoff-day row must prune, got {n}" - kept = con.execute( - "SELECT headlines_hash FROM signals ORDER BY id").fetchall() - assert kept == [(None,), ("h2",)], kept - con.close() - finally: - os.unlink(path) - - # ══════════════════════════════════════════════════════════════════════ # MODULE 29: stake-multiplier sizing — features + record-keeping # ══════════════════════════════════════════════════════════════════════