chore: drop unreachable branches outside the signal core (no behaviour change)

- providers: _to_oanda_instrument / _to_tiingo_ticker / _to_td_symbol no longer
  guess a wire symbol for pairs that are not registered — the registries are
  the only source, so an unknown symbol raises the provider's DataIntegrityError
  instead of silently fetching something else. Every scanned symbol is validated
  against PAIR_INFO at import, so nothing reachable changes.
- main / telegram_bot _fmt_price: the sub-$10 five-decimal branch could never
  fire for gold.
- news_engine._parse_pair: BASE/QUOTE only; the Yahoo "=X" fallback was
  unreachable (its test goes with it).
- dashboard_publisher.event_to_dict: drop minutes_until (frozen at publish
  time, misleading) and is_high_impact — dashboard.js never read either.
- static/dashboard.html: unused id="cal-table".

Tried and reverted: trimming the LogRow fields nothing in analyze_signal_log
reads — the tests read them to verify the loader round-trips those DB columns,
so they stay.

Suite: 460 passed, 0 failed.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_0156YhogLytzj9P2iH1skGbL

.gitignore

@@ -3,9 +3,6 @@
 .env.local
 config.json
 
-# Oanda daemon state
-oanda_state.json
-
 # Editor / tooling
 .claude/
 .vscode/
@@ -17,6 +14,7 @@ __pycache__/
 *.egg-info/
 .venv/
 venv/
+.coverage
 
 # OS
 .DS_Store
@@ -28,14 +26,7 @@ Thumbs.db
 # Backtesting data (keep scripts, ignore generated files)
 backtest/cache/
 backtest/*.csv
-backtest/research/*.parquet
-backtest/research/duka_m5_boundary_spreads.csv
-backtest/trailgrid/
-backtest/trailgrid5m/
 backtest/__pycache__/
 .env.*
 logs/
-backtest/research/commit_ladder_2026-08/
-backtest/research/llm_news_eval_2026-08/
-
 backtest/research/sizing_2026-09/

CLAUDE.md

@@ -12,11 +12,13 @@ Every numeric threshold in `config.py` and `pair_signals/xauusd.py` was backtest
 
 ```bash
 python test_all.py                      # QA suite, no network (mocked); prints its own count
-                                        #  (463 tests / 31 modules as of 2026-09-06)
+                                        #  (460 tests / 31 modules as of 2026-09-06)
 python main.py --pair XAUUSD --timeframe H1 --detail   # CLI signal report (--json for full dump,
                                         #  --panel for the advisory LLM agent panel, --outlook for the
                                         #  hourly AI market brief; the violent-move watch line appears
-                                        #  when the `violent_watch` config block is enabled)
+                                        #  when the `violent_watch` config block is enabled;
+                                        #  --timeframe accepts M5/M15/H1/H4 only and the confirmation
+                                        #  TFs always come from the pair factory's map)
 python telegram_bot.py                  # the bot (refuses to boot without credentials)
 python dashboard_publisher.py --once    # web dashboard: one scan+publish cycle to $DASHBOARD_OUT_DIR
                                         # (omit --once for the candle-close daemon; deploy/README-dashboard.md)
@@ -35,7 +37,7 @@ python backtest/gpu_backtest.py --pair XAUUSD --start 2025-01-01 --end 2025-06-0
 
 **`test_all.py` is NOT pytest** — it uses a custom `@test("name")` decorator that runs each test immediately at import; don't grep for `def test_` or try `pytest test_all.py`. Run the whole file; it prints per-module results and exits non-zero on failure. There is no way to run a single test short of commenting modules out. Tests must stay green; every new gate/knob needs tests. Signal-log fixtures: use a WEEKDAY signal bar when a test exercises candle continuity — the outcome model treats weekend and 21:00 UTC gaps as scheduled, so Saturday-dated bars never produce a gap. `_candles_json` stores `df.tail(14)`, so a test snapshot longer than 14 bars silently drops its head. When patching weekend logic in tests, patch `providers.base._is_forex_weekend` — but know there are THREE independent signal-path weekend implementations (`providers/base.py`, `signal_aggregator.py`, and an inline copy in `backtest/gpu_backtest.py:_is_xau_blocked_by_calendar`) plus the advisory-only continuity copy in `advisory_outcome.py`; patching one does not reach the others.
 
-Python 3.12; a `venv/` exists in-repo. Dependencies in `requirements.txt` (numpy, pandas, requests, twelvedata, python-telegram-bot ≥20, websockets). scikit-learn and lightgbm are installed in the dev `venv/` for `backtest/research/` only — never add them to `requirements.txt` or import them from live code; anything shipped must be numpy-only.
+Python 3.12; a `venv/` exists in-repo. Dependencies in `requirements.txt` (numpy, pandas, requests, twelvedata, python-telegram-bot ≥20, websockets). scikit-learn and lightgbm are installed in the dev `venv/` for ad-hoc research only — never add them to `requirements.txt` or import them from live code; anything shipped must be numpy-only.
 
 ## Configuration & credentials
 
@@ -94,10 +96,11 @@ Signal currency everywhere is `SignalStrength` (−2..+2). DataFrames are UTC tz
 ## Current state notes (dated per entry)
 
 - Oanda v20 is the live feed (`data_providers.active: "oanda"`). The 2026-08-04 re-pin made the 27 `backtest/oanda_*.csv` books the baseline; all TD-era books are superseded. (2026-08-14) The re-pin M5 book `oanda_m5_12mo_065_nopb_20260814.csv` (+106.82R PF 1.51) is itself SUPERSEDED by the 2026-09-03 book in the entry below; the M15 reference is unchanged since 2026-08-04: `oanda_m15_12mo_065.csv` (397 / +73.93R / PF 1.53). See `backtest/research/repin_verification_2026-08-14.md`.
-- Cost reporting is dual: $0.65 is the measured-Oanda realistic headline ($0.65-$0.72 round trip), while $0.30 remains for cross-book comparability and as the live TP spread-gate input. Raising `round_trip_cost` from $0.30 would move the TP floor from about $9 to $19.50 and requires full rule-1 revalidation.
+- Cost reporting is dual: $0.65 is the measured-Oanda realistic headline ($0.65-$0.72 round trip), while $0.30 remains for cross-book comparability and as the live TP spread-gate input. Raising `round_trip_cost` from $0.30 would move the TP floor from about $9 to $19.50 and requires full rule-1 revalidation. The $0.65 figure comes from the 2026-07-24 Dukascopy tick study (9M ticks over 12 months; its tooling was removed 2026-09-06 — the conclusion lives here and in memory).
 - Twelve Data is signal-incompatible with Oanda (Phase 1: about 19% M5 bar-direction disagreement and a systematic ~$1 high bias; see `backtest/research/feed_divergence_2026-08-14.md`). On an Oanda candle outage, stop scanning rather than switching to TD candles.
-- Tiingo is streaming-only (`data_providers.streaming: "tiingo"`), not a designated candle fallback. Oanda-Tiingo signal parity is documented in `backtest/research/signal_parity_2026-08-14.md`; Phase 4's two-week live shadow scan remains a manual operational step.
+- Tiingo is streaming-only (`data_providers.streaming: "tiingo"`), not a designated candle fallback. Oanda-Tiingo signal parity is documented in `backtest/research/signal_parity_2026-08-14.md`; Phase 4's two-week live shadow scan (`scripts/shadow_scan.py`) remains a manual operational step.
 - Human-facing docs in `docs/` (`user_guide.html`, `beginner_guide.html` + PDF builds) must be kept in sync when signal behavior changes. As of 2026-09-06 the PDF builds are BEHIND the HTML (violent-move watch entry added without a PDF rebuild).
 - 2026-09-03 signal-log campaign shipped two new per-TF gates (`backtest/research/signal_log_campaign_2026-09.md`): M5 `sell_blocked_regimes=["transitioning"]` and H1 `sell_min_quality=[]` (all H1 shorts blocked). Current-HEAD M5 canonical book is now `backtest/oanda_m5_12mo_065_nopb_20260903.csv` (557 / +112.21R / PF 1.56, halves +55.9/+56.4); H1 reference `backtest/oanda_h1_12mo_065_20260903.csv` (91 / +22.2R / PF 1.79; 5yr +49.1R PF 1.46; OOS +16.7R PF 1.74).
-- 2026-09-06 stake-sizing campaign (`backtest/research/sizing_campaign_2026-09.md`): a walk-forward "second opinion" stake multiplier (meta-labelling on 65 decision-time features, 3-year Oanda books at $0.65) FAILED the pre-registered rule on M5 (clean null) and M15 (lift inside the label-shuffle null, explained by BUY-vs-SELL). Nothing that changes numbers shipped. Behaviour-neutral tooling remains: `position_sizing.build_features` + `--export-features` (default export stays 63 columns), `SignalReport.penalty_tags/penalty_total/momentum_fading`, `TradeSetup.stake_multiplier` (None; reserved), `backtest/research/{book_report,burst_map,sizing_model}.py`. Stage-B facts: M5 stretch measures are POSITIVE in both halves (do not re-propose exhaustion sizing on M5); M15 exhaustion tag is weakly negative (≈ +3R/yr, noise); bursts cluster at 12–15 UTC and 01 UTC bar-close and do NOT hurt entries.
+- 2026-09-06 stake-sizing campaign (`backtest/research/sizing_campaign_2026-09.md`): a walk-forward "second opinion" stake multiplier (meta-labelling on 65 decision-time features, 3-year Oanda books at $0.65) FAILED the pre-registered rule on M5 (clean null) and M15 (lift inside the label-shuffle null, explained by BUY-vs-SELL). Nothing that changes numbers shipped. Behaviour-neutral tooling remains: `position_sizing.build_features` + `--export-features` (default export stays 63 columns), `SignalReport.penalty_tags/penalty_total/momentum_fading`, `TradeSetup.stake_multiplier` (None; reserved) (the campaign's research scripts were removed 2026-09-06; git history has them). Stage-B facts: M5 stretch measures are POSITIVE in both halves (do not re-propose exhaustion sizing on M5); M15 exhaustion tag is weakly negative (≈ +3R/yr, noise); bursts cluster at 12–15 UTC and 01 UTC bar-close and do NOT hurt entries.
 - Auto-execution (oanda_daemon.py / oanda_executor.py) was removed 2026-09-03 — the owner is not auto-trading; last commit containing it is 31002cb. Book-parity rules that lived in its docstring (no broker TP, 5-bar cooldown, once-per-bar trail) remain the backtester's model.
+- 2026-09-06 stale-code cleanup (behaviour-neutral; suite green, M5/H1 12-month books byte-identical after every signal-side commit): `main.py --timeframe` now accepts M5/M15/H1/H4 and takes its confirmation TFs from the pair factory (the CLI's private table had paired M5 with M1+M15); `PlatformConfig.pip_value` is gone (`PAIR_INFO` is the source); removed as dead or superseded: `_get_affected_pairs`/`event_impact_on_pairs`, `SignalLogger.advance_with_bars`/`path`/`news_for`, `reset_llm_client`, the Telegram scan-summary/keyboard helpers and duplicate confirmation map, the 2026-07 Oanda-vs-TD research pager+study, `burst_profile.py`, `hour_analysis.py`, `scripts/compare_oanda_tiingo.py`, the never-implemented LLM-provider plan and the merged config.json spec. Second pass the same day (owner directive: keep only what the system needs): every research script and data file under `backtest/research/` (book_report, book_diff, the burst tools, sizing_model, commit_ladder, llm_news_eval + golden CSV, ff_calendar_scraper, calendar_impact, watch_rule_eval, the Dukascopy spread study + CSVs, the panel dry-test JSONL), the two completed study scripts in `scripts/`, the architecture diagram + generator and the data-source test plan were deleted; the campaign `.md` reports stay as the decision ledger. Git history before `953c5ad` has all of it. Third pass (owner: "remove dead functions as well"): the write-only pb/cp rejection counters, the unreachable `_TP_QUALITY_SCALE` B/C entries, the engine's unused DataFrame argument and M1 arm, the providers' generic-symbol guesses (unknown symbols now raise), the sub-$10 price format, the `=X` pair parsing and the unread `minutes_until`/`is_high_impact` news.json keys are gone.

agent_panel.py

@@ -26,9 +26,10 @@ Two invariants, both load-bearing (CLAUDE.md rules 1 and 4):
   so replayed books can never see a panel field. Backtest parity holds
   by construction.
 
-Cost model: every full panel spends 5 requests (4 personas + 1
+Cost model: every full panel spends 4 requests (the personas; 5 with
-synthesizer) against the shared DeepSeek daily cap (llm_client). Two
+verdict_source="chair", which adds the synthesizer) against the shared
-layers bound the spend on top of that cap:
+DeepSeek daily cap (llm_client). Two layers bound the spend on top of
+that cap:
 
 - ``max_panels_per_day`` (config, default 20) — a UTC-day budget of
   panel *attempts*, mirroring the client's request cap.

analyze_signal_log.py

@@ -6,7 +6,7 @@ analyze_signal_log.py — signal_log performance evaluator
 READ-ONLY analysis of logs/signals.db: how are the logged live signals
 and the advisory LLM panel actually performing?
 
-Three lenses:
+Six lenses:
 
 1. TRADE REPLAY — every logged row carrying a trade_setup is deduped
    (telegram + dashboard log the same scan; a persisting signal re-fires
@@ -59,6 +59,7 @@ Usage:
     python analyze_signal_log.py --parity            # surface parity only
     python analyze_signal_log.py --stored-candles    # DB-only replay
     python analyze_signal_log.py --stored-candles --news   # + news lens
+    python analyze_signal_log.py --scan-outcomes     # forward-move table
 """
 
 import argparse
@@ -67,7 +68,7 @@ import math
 import os
 import sqlite3
 import sys
-from dataclasses import dataclass, field
+from dataclasses import dataclass
 from datetime import datetime, timedelta, timezone
 from typing import Dict, List, Optional, Sequence, Tuple
 
@@ -140,7 +141,8 @@ class SignalEvent:
 _COLS = ("id, logged_at, source, timeframe, signal, score, quality,"
          " price, market_regime, setup_json, panel_verdict,"
          " panel_confidence, action")
-# Columns added by the 2026-09 observability work; absent on older DBs.
+# Columns an older DB may lack (added by later schema versions); loaded
+# only when PRAGMA table_info reports them.
 _OPTIONAL_COLS = ("gate_reason", "actionable", "config_fingerprint",
                   "advisory_event_id", "advisory_outcome_status",
                   "advisory_r_multiple", "scan_kind", "news_sentiment",
@@ -754,8 +756,7 @@ def _fmt_r(r: Optional[float]) -> str:
     return "  n/a " if r is None else f"{r:+.2f}R"
 
 
-def persona_scorecard(rows: Sequence[LogRow], events: Sequence[SignalEvent],
+def persona_scorecard(events: Sequence[SignalEvent], db_path: str) -> List[dict]:
-                      db_path: str) -> List[dict]:
     """Per persona × stance: n, mean R and win rate of the underlying
     event (needs panel_takes_json — read straight from the DB)."""
     by_row = _row_index(events)
@@ -921,8 +922,7 @@ def news_extremes(rows: Sequence[LogRow], streams: Dict[str, List[dict]],
     }
 
 
-def news_outcome_buckets(rows: Sequence[LogRow],
+def news_outcome_buckets(events: Sequence[SignalEvent],
-                         events: Sequence[SignalEvent],
                          db_path: str) -> List[dict]:
     """Resolved events bucketed by (direction, news bucket, macro stance).
 
@@ -1306,13 +1306,13 @@ def main(argv: Optional[Sequence[str]] = None) -> int:
         print_report(rows, events, panel, hit)
         print_stored_books(events)
         print("\n── Persona scorecard (stance × event outcome) " + "─" * 22)
-        for p in persona_scorecard(rows, events, args.db):
+        for p in persona_scorecard(events, args.db):
             print(f"  {p['persona']:11} {p['stance']:8} n={p['n']:3d} "
                   f"mean {p['mean_r']:+.3f}R  win {100 * p['win_rate']:.0f}%")
         if args.news:
             print_news(news_lead_lag(rows, streams),
                        news_extremes(rows, streams),
-                       news_outcome_buckets(rows, events, args.db))
+                       news_outcome_buckets(events, args.db))
         print_parity(parity_report(rows))
         if args.csv:
             export_csv(events, args.csv)

app_config.py

@@ -21,8 +21,6 @@ the config.json.example defaults, every produced PlatformConfig is
 byte-identical to the pure-code defaults. apply_pair_overrides() runs
 in the pair factory BEFORE the validated per-timeframe deltas, so the
 bot, CLI and backtester all see the same overlay (live == backtest).
-
-Spec: docs/superpowers/specs/2026-07-21-centralized-config-design.md
 """
 
 import json
@@ -82,7 +80,7 @@ load_dotenv()
 _VALID_QUALITIES = ("A+", "A", "B", "C")
 _VALID_LOG_LEVELS = ("DEBUG", "INFO", "WARNING", "ERROR", "CRITICAL")
 # Timeframes the bot's candle-boundary scheduler supports (must have
-# AUTOSCAN_CONFIRMATIONS entries in telegram_bot.py).
+# TF_CONFIRMATIONS entries in telegram_bot.py).
 _SCANNABLE_TFS = ("M5", "M15", "H1", "H4")
 
 _KNOWN_TOP = {"telegram", "data_providers", "scanning", "pairs", "log_level",

backtest/diagnose_filters.py

@@ -45,7 +45,6 @@ from gpu_backtest import (  # noqa: E402
     BAR_SECONDS,
     GPUTechnicalEngine,
     PAIR_ALIASES,
-    BacktestConfig,
     IndicatorCache,
     fetch_history,
     precompute_indicators,
@@ -221,7 +220,6 @@ def run_diagnosis(
     platform_cfg = PlatformConfig(
         symbol=pair,
         display_name=display,
-        pip_value=pair_info["pip_value"],
         primary_timeframe=primary_tf,
         confirmation_timeframes=list(confirm_tfs),
     )

backtest/gpu_backtest.py

@@ -963,8 +963,6 @@ class GPUTechnicalEngine(TechnicalEngine):
         result.key_levels["kc_upper"] = kc_upper_val
         result.key_levels["kc_lower"] = kc_lower_val
         result.key_levels["kc_mid"] = kc_mid_val
-        result._kc_squeeze = is_squeeze
-        result._kc_squeeze_released = squeeze_released
 
 
 # ═════════════════════════════════════════════════════════════════════
@@ -1765,7 +1763,6 @@ def run_backtest(
         platform_cfg = PlatformConfig(
             symbol=bt_cfg.pair,
             display_name=PAIR_INFO[bt_cfg.pair]["display"],
-            pip_value=PAIR_INFO[bt_cfg.pair]["pip_value"],
             primary_timeframe=bt_cfg.primary_tf,
             confirmation_timeframes=list(bt_cfg.confirmation_tfs),
         )
@@ -2404,7 +2401,6 @@ def main(argv: Optional[List[str]] = None) -> int:
         platform_cfg = PlatformConfig(
             symbol=pair,
             display_name=PAIR_INFO[pair]["display"],
-            pip_value=PAIR_INFO[pair]["pip_value"],
             primary_timeframe=primary_tf,
             confirmation_timeframes=confirm_tfs or [Timeframe.H1, Timeframe.H4],
         )

backtest/instructions_for_backtest.txt

@@ -1,69 +1,26 @@
-Note down entery and exit Why we took trade?
+Backtest reporting checklist — ForexBot2 (XAU/USD)
-What were each indicator signal on that specific time when we took trade
+==================================================
-How many indicators agreed at trade time
+Every backtest report must record:
-Always backtest using engine.py and signal_aggregator.py
+- The exact command line (window, --primary-tf, --spread, and any --set /
-Dont take trades on weekends and Fridays
+  --quality / --strategies / --confirm-tfs overrides) and the "Backend:" log line.
-Keep TP small 1.3 Reward ratio
+- Trades, total R, win rate, profit factor and max drawdown for the whole
-At the end of test make table of results and print all values you observed.
+  window AND for both halves, at both cost conventions ($0.65 measured-Oanda
-Trade context
+  headline, $0.30 cross-book comparability) — CLAUDE.md rule 1.
-
+- Per trade (the exported CSV already carries these): entry/exit time and
-Time of day and trading session (Asian/London/NY/overlap)
+  price, exit reason, R multiple, MFE/MAE, entry reasons, and every
-Day of week (track Mon/Tue/Wed/Thu performance separately)
+  indicator's signal / value / agreement at decision time.
-Market regime at entry: trending, ranging, or choppy (use ATR or ADX)
+- Breakdowns by session/hour, weekday, regime and volatility regime, and,
-Volatility level (ATR value or ATR percentile vs recent average)
+  for any candidate change, the removed/added/shifted trade diff against the
-Pair/instrument traded
+  control book (the 5-bar cooldown displaces entries, so judge run-level
-News proximity (high-impact news within 30 min before/after entry)
+  numbers, never the added-trade bucket alone).
-
+- The md5 of the exported book whenever the run is a parity or regression check.
-Trade execution details
+Replays go through the real signal code (analyze_from_dataframes) — never a
-
+re-implementation.
-Spread at entry in pips
-Slippage on entry and exit (planned vs actual fill)
-Time in trade (minutes/hours from entry to exit)
-Maximum Adverse Excursion (MAE): how far against you it went
-Maximum Favorable Excursion (MFE): how far in profit before pullback
-
-Risk metrics
-
-Max consecutive losses (streak)
-Max consecutive wins (streak)
-Expectancy: (Win% x Avg Win) minus (Loss% x Avg Loss)
-Expectancy per hour in trade
-Recovery factor: Total R divided by Max DD
-
-Indicator/signal analysis
-
-Each indicator's standalone win rate when it agreed with the trade
-Confluence level performance (2 vs 3 vs 4+ indicators agreeing)
-Which indicator disagreed on losing trades (find the "canary")
-
-Decision quality
-
-Rate each setup A/B/C at entry, before knowing outcome
-Verify A-grade setups actually win more often
-Track system trades vs discretionary deviations separately
-
-Process notes
-
-Screenshot or chart snapshot at entry and exit
-Emotional state (calm, FOMO, revenge, tired)
-What would have happened if you held longer or exited sooner
-
-Aggregate analysis at end of backtest
-
-Equity curve, not just total R
-Performance breakdown by session, day, volatility regime
-Win rate over rolling 20-trade windows (check for edge decay)
-
-Critical extra
-"Do not trade" log: setups flagged but skipped, and what happened after
-
 
 --- 2026-09-06: decision-time feature export ---
 - `--export-features` appends stake_multiplier, r_sized and the 65 `f_*` decision-time
   features (position_sizing.FEATURE_ORDER) AFTER the 63 pinned columns; default export is
-  byte-identical to the pinned books. Use backtest/research/book_report.py for weekday /
+  byte-identical to the pinned books.
-  hour x weekday / session / streak / rolling-WR / feature-quintile (both halves) tables,
-  backtest/research/burst_map.py for the burst-by-hour map, and
-  backtest/research/sizing_model.py for the walk-forward + shuffle-control harness.
 - Any sizing report must show the FLAT book and the SIZED book side by side and the
-  label-shuffle null; the 2026-09-06 campaign verdict (nothing ships) is the reference.
+  label-shuffle null; the 2026-09-06 campaign verdict (nothing ships,
+  backtest/research/sizing_campaign_2026-09.md) is the reference.

backtest/research/__init__.py

@@ -1,5 +0,0 @@
-"""Research tooling: calendar sourcing/store and edge-screening utilities.
-
-Nothing here is imported by the live bot; these modules support
-backtesting and edge validation only.
-"""

backtest/research/ai_signal_enrichment_2026-08-15.md

@@ -95,7 +95,7 @@ Cost guardrails already in place: 5 requests/panel, `max_panels_per_day`
   injection (technical persona only), prompt-contract assertions,
   panel row update, rowid hand-off + no-op paths). Existing attach tests
   stub `ap._dp` to stay off the network.
-- **Live dry test** (`panel_payload_drytest_2026-08-15.py`, fabricated
+- **Live dry test** (`panel_payload_drytest_2026-08-15.py` — script removed 2026-09-06, raw replies kept in the `.jsonl` — fabricated
   numbers only, 13 real DeepSeek requests total): **7/7 probes pass** —
   coherent-bullish → support@70; bearish-vs-BUY → oppose@85; genuinely
   mixed → neutral@30; empty payload → neutral@20 (contract); no invented

backtest/research/book_diff.py

@@ -1,86 +0,0 @@
-"""Trade-book diff: relaxed/gated run vs its control.
-
-Usage:
-    python backtest/research/book_diff.py CONTROL.csv CANDIDATE.csv BAR_MINUTES
-
-Decision numbers are RUN-LEVEL (ΔtotalR, ΔPF, ΔmaxDD, Δtrades) — never
-the added-trade bucket alone: the 5-bar entry cooldown means new
-entries displace existing ones. The added/removed decomposition and the
-per-decision-hour histogram of added trades are diagnostics.
-
-Shifted trades = removed+added pairs in the same direction within
-±5 bars (cooldown displacement, not a genuinely new/lost signal).
-"""
-import sys
-
-import pandas as pd
-
-
-def _metrics(df):
-    r = df["r_multiple"]
-    gains, losses = r[r > 0].sum(), -r[r <= 0].sum()
-    cum = r.cumsum()
-    return dict(trades=len(df), total_r=r.sum(),
-                pf=gains / max(losses, 1e-9),
-                wr=(r > 0).mean() * 100 if len(df) else 0.0,
-                max_dd=(cum.cummax() - cum).max() if len(cum) else 0.0)
-
-
-def diff_books(control_csv, candidate_csv, bar_minutes, split="2026-01-29"):
-    ctrl, cand = pd.read_csv(control_csv), pd.read_csv(candidate_csv)
-    for df in (ctrl, cand):
-        df["_t"] = pd.to_datetime(df["entry_time"], utc=True, format="mixed")
-        df["_key"] = df["entry_time"].astype(str) + "|" + df["direction"]
-
-    added = cand[~cand["_key"].isin(ctrl["_key"])].copy()
-    removed = ctrl[~ctrl["_key"].isin(cand["_key"])].copy()
-
-    # cooldown displacement: same-direction removed/added within ±5 bars
-    tol = pd.Timedelta(minutes=5 * bar_minutes)
-    shifted = 0
-    rem_pool = removed.copy()
-    for _, a in added.iterrows():
-        m = rem_pool[(rem_pool["direction"] == a["direction"])
-                     & ((rem_pool["_t"] - a["_t"]).abs() <= tol)]
-        if len(m):
-            shifted += 1
-            rem_pool = rem_pool.drop(m.index[0])
-
-    mc, mk = _metrics(ctrl), _metrics(cand)
-    split_ts = pd.Timestamp(split, tz="UTC")
-    halves = (cand[cand["_t"] < split_ts]["r_multiple"].sum(),
-              cand[cand["_t"] >= split_ts]["r_multiple"].sum())
-    hour_hist = None
-    if len(added):
-        added["dec_hour"] = (added["_t"]
-                             - pd.Timedelta(minutes=bar_minutes)).dt.hour
-        hour_hist = added.groupby("dec_hour")["r_multiple"] \
-                         .agg(["count", "sum"]).round(2)
-    return dict(
-        control=mc, candidate=mk,
-        delta_r=mk["total_r"] - mc["total_r"],
-        delta_pf=mk["pf"] - mc["pf"],
-        delta_dd=mk["max_dd"] - mc["max_dd"],
-        delta_trades=mk["trades"] - mc["trades"],
-        added_n=len(added), added_r=added["r_multiple"].sum(),
-        removed_n=len(removed), removed_r=removed["r_multiple"].sum(),
-        shifted=shifted, halves=halves, added_by_hour=hour_hist,
-    )
-
-
-if __name__ == "__main__":
-    d = diff_books(sys.argv[1], sys.argv[2], int(sys.argv[3]))
-    c, k = d["control"], d["candidate"]
-    print(f"control  : {c['trades']:4d} trades  {c['total_r']:+7.1f}R  "
-          f"PF {c['pf']:.2f}  WR {c['wr']:.1f}%  DD {c['max_dd']:.1f}")
-    print(f"candidate: {k['trades']:4d} trades  {k['total_r']:+7.1f}R  "
-          f"PF {k['pf']:.2f}  WR {k['wr']:.1f}%  DD {k['max_dd']:.1f}")
-    print(f"ΔR {d['delta_r']:+.1f}  ΔPF {d['delta_pf']:+.2f}  "
-          f"ΔDD {d['delta_dd']:+.1f}  Δtrades {d['delta_trades']:+d}")
-    print(f"added {d['added_n']} ({d['added_r']:+.1f}R)  "
-          f"removed {d['removed_n']} ({d['removed_r']:+.1f}R)  "
-          f"shifted≈{d['shifted']}")
-    print(f"candidate halves: H1 {d['halves'][0]:+.1f}R  H2 {d['halves'][1]:+.1f}R")
-    if d["added_by_hour"] is not None:
-        print("\nadded trades by decision hour:")
-        print(d["added_by_hour"].to_string())

backtest/research/book_report.py

@@ -1,581 +0,0 @@
-"""Full descriptive report of a backtest trade book (research only).
-
-Usage:
-    python backtest/research/book_report.py --book BOOK.csv --bar-minutes N \
-        [--split YYYY-MM-DD|median] [--parquet-4h PATH] [--parquet-1d PATH] \
-        [--out DIR]
-
-Prints (and writes to DIR/report.txt) headline metrics for the whole book
-and for each half, group tables (weekday / decision hour / session / quality
-/ regime / vol_regime / direction / strategy / exit_reason / indicators_agree),
-win-loss streaks, a rolling win-rate summary, and a quantile-bucket screen of
-every numeric decision-time feature the book carries (`f_*` / `x_*` columns
-plus score, confidence, rr_planned, indicators_agree/oppose and `ind_*_value`).
-
-The screen is deliberately half-split: a feature only "passes" when the
-top-bucket-minus-bottom-bucket mean-R gap has the SAME SIGN in both halves of
-the book and both extreme buckets are populated in both halves. Bin edges are
-fit on the WHOLE book (so the halves share edges and stay comparable) — that
-leaks a little cross-half information into the edges themselves, which is why
-the sign test, not the size of the gap, is the thing to trust.
-
-Everything here is measurement, not simulation: it never re-runs the signal
-engine, it only describes a book that gpu_backtest.py already produced.
-Analysis tooling only — never imported by the live code.
-"""
-from __future__ import annotations
-
-import argparse
-import math
-import os
-from typing import Iterable
-
-import numpy as np
-import pandas as pd
-
-SESSIONS = (
-    ("asian", 3, 6),
-    ("london", 7, 12),
-    ("overlap", 13, 16),
-    ("ny", 17, 22),
-)
-
-# Numeric non-`f_`/`x_` columns that are known at DECISION time and are
-# therefore legal screen candidates. bars_held / r_multiple / mfe_r / mae_r
-# are post-entry outcomes and must never appear here.
-BASE_SCREEN_COLS = ("score", "confidence", "rr_planned",
-                    "indicators_agree", "indicators_oppose")
-
-
-def session_of(hour: int) -> str:
-    for name, lo, hi in SESSIONS:
-        if lo <= hour <= hi:
-            return name
-    return "off"
-
-
-def feature_columns(df: pd.DataFrame) -> list[str]:
-    """Decision-time feature columns carried by this book, generically."""
-    extra = [c for c in df.columns if c.startswith(("f_", "x_"))]
-    base = [c for c in BASE_SCREEN_COLS if c in df.columns]
-    ind = [c for c in df.columns
-           if c.startswith("ind_") and c.endswith("_value")]
-    out, seen = [], set()
-    for c in extra + base + ind:
-        if c not in seen:
-            seen.add(c)
-            out.append(c)
-    return out
-
-
-# ───────────────────────────── loading ──────────────────────────────
-
-def load_book(csv: str, bar_minutes: int = 5,
-              split: str = "median") -> pd.DataFrame:
-    """Read a gpu_backtest trade book and add the decision-time columns.
-
-    `decision_time` = entry_time − one bar: entries fill at the NEXT bar's
-    open, so the bar the signal actually decided on closed one bar earlier.
-    That is the clock blocked_hours_utc and every session filter evaluate.
-    `half` is 1 before the split instant and 2 at/after it.
-    """
-    df = pd.read_csv(csv)
-    for col in ("entry_time", "exit_time"):
-        if col in df.columns:
-            df[col] = pd.to_datetime(df[col], utc=True, format="mixed")
-    df["decision_time"] = df["entry_time"] - pd.Timedelta(minutes=bar_minutes)
-    df["decision_hour"] = df["decision_time"].dt.hour
-    df["weekday"] = df["decision_time"].dt.weekday
-    df["session"] = df["decision_hour"].map(session_of)
-    df = df.sort_values("entry_time", kind="mergesort").reset_index(drop=True)
-    df["half"] = _halves(df["entry_time"], split)
-    return df
-
-
-def split_instant(entry_time: pd.Series, split: str) -> pd.Timestamp:
-    if split == "median":
-        return entry_time.median()
-    return pd.Timestamp(split, tz="UTC")
-
-
-def _halves(entry_time: pd.Series, split: str) -> pd.Series:
-    cut = split_instant(entry_time, split)
-    return np.where(entry_time < cut, 1, 2)
-
-
-# ───────────────────────────── metrics ──────────────────────────────
-
-def base_metrics(r: pd.Series, df: pd.DataFrame | None = None) -> dict:
-    """Headline stats for a series of R multiples.
-
-    max_dd uses the gpu_backtest.compute_metrics convention: a 0.0 is
-    prepended to the cumulative-R curve so an opening losing streak counts
-    as drawdown from flat, not from an already-negative first trade.
-    pf/expectancy split on r > 0 / r < 0 (exact zeros are neither).
-    """
-    r = pd.Series(r, dtype="float64").dropna()
-    n = int(len(r))
-    if n == 0:
-        return {"n": 0, "total_r": 0.0, "mean_r": 0.0, "median_r": 0.0,
-                "wr": 0.0, "pf": 0.0, "max_dd": 0.0, "sharpe": 0.0,
-                "expectancy": 0.0, "expectancy_alt": 0.0, "recovery": 0.0,
-                "avg_bars": float("nan"), "avg_mfe": float("nan"),
-                "avg_mae": float("nan")}
-    wins, losses = r[r > 0], r[r < 0]
-    gross_win = float(wins.sum())
-    gross_loss = float(-losses.sum())
-    cum = np.concatenate(([0.0], np.cumsum(r.to_numpy())))
-    max_dd = float((np.maximum.accumulate(cum) - cum).max())
-    std = float(r.std(ddof=0))
-    mean_r = float(r.mean())
-    wr = len(wins) / n
-    lr = len(losses) / n
-    avg_win = float(wins.mean()) if len(wins) else 0.0
-    avg_loss = float(-losses.mean()) if len(losses) else 0.0
-    expectancy_alt = wr * avg_win - lr * avg_loss
-    # Identity check: the textbook expectancy form must equal the plain mean
-    # whenever no trade is exactly flat (and it does even then, since a 0 R
-    # trade contributes 0 to both sides).
-    assert math.isclose(expectancy_alt, mean_r, rel_tol=1e-9, abs_tol=1e-9), (
-        f"expectancy mismatch: {expectancy_alt} vs mean {mean_r}")
-    out = {
-        "n": n,
-        "total_r": float(r.sum()),
-        "mean_r": mean_r,
-        "median_r": float(r.median()),
-        "wr": wr * 100.0,
-        "pf": (gross_win / gross_loss) if gross_loss > 0 else float("inf"),
-        "max_dd": max_dd,
-        "sharpe": (mean_r / std) if std > 0 else 0.0,
-        "expectancy": mean_r,
-        "expectancy_alt": expectancy_alt,
-        "recovery": (float(r.sum()) / max_dd) if max_dd > 0 else float("inf"),
-        "avg_bars": float("nan"),
-        "avg_mfe": float("nan"),
-        "avg_mae": float("nan"),
-    }
-    if df is not None:
-        for key, col in (("avg_bars", "bars_held"), ("avg_mfe", "mfe_r"),
-                         ("avg_mae", "mae_r")):
-            if col in df.columns:
-                out[key] = float(pd.to_numeric(df[col],
-                                               errors="coerce").mean())
-    return out
-
-
-def _pf(r: pd.Series) -> float:
-    """Profit factor, hour_analysis-compatible (loss side is r <= 0)."""
-    pos = r[r > 0].sum()
-    neg = -r[r <= 0].sum()
-    return float(pos / max(neg, 1e-9))
-
-
-def group_table(df: pd.DataFrame, by, min_n: int = 10) -> pd.DataFrame:
-    """Per-group P&L with a half-1 / half-2 sign-stability flag."""
-    keys = [by] if isinstance(by, str) else list(by)
-    rows = []
-    for key, sub in df.groupby(keys, dropna=False, sort=True):
-        r = sub["r_multiple"]
-        h1, h2 = sub[sub["half"] == 1]["r_multiple"], sub[sub["half"] == 2]["r_multiple"]
-        m1 = float(h1.mean()) if len(h1) else float("nan")
-        m2 = float(h2.mean()) if len(h2) else float("nan")
-        stable = bool(len(h1) >= min_n and len(h2) >= min_n
-                      and np.sign(m1) != 0 and np.sign(m1) == np.sign(m2))
-        rec = {}
-        # pandas >= 2.2 hands back a 1-tuple even for a single group key.
-        vals = key if isinstance(key, tuple) else (key,)
-        for k, v in zip(keys, vals):
-            rec[k] = v
-        rec.update({
-            "n": int(len(sub)),
-            "total_r": round(float(r.sum()), 2),
-            "mean_r": round(float(r.mean()), 3),
-            "wr": round(float((r > 0).mean() * 100), 2),
-            "pf": round(_pf(r), 2),
-            "n_h1": int(len(h1)),
-            "total_r_h1": round(float(h1.sum()), 2),
-            "mean_r_h1": round(m1, 3) if len(h1) else float("nan"),
-            "n_h2": int(len(h2)),
-            "total_r_h2": round(float(h2.sum()), 2),
-            "mean_r_h2": round(m2, 3) if len(h2) else float("nan"),
-            "sign_stable": stable,
-        })
-        rows.append(rec)
-    out = pd.DataFrame(rows)
-    if not out.empty:
-        out = out.set_index(keys)
-    return out
-
-
-def streaks(df: pd.DataFrame) -> dict:
-    """Longest win/loss runs and the longest stretch under water (in trades)."""
-    d = df.sort_values("entry_time", kind="mergesort")
-    r = d["r_multiple"].to_numpy(dtype="float64")
-    best_w = best_l = cur_w = cur_l = 0
-    for x in r:
-        if x > 0:
-            cur_w, cur_l = cur_w + 1, 0
-        elif x < 0:
-            cur_l, cur_w = cur_l + 1, 0
-        else:
-            cur_w = cur_l = 0
-        best_w, best_l = max(best_w, cur_w), max(best_l, cur_l)
-    cum = np.concatenate(([0.0], np.cumsum(r)))
-    peak = np.maximum.accumulate(cum)
-    flat, longest = 0, 0
-    for i in range(1, len(cum)):
-        flat = 0 if cum[i] >= peak[i] else flat + 1
-        longest = max(longest, flat)
-    return {"max_win_streak": best_w, "max_loss_streak": best_l,
-            "longest_flat_trades": longest}
-
-
-def rolling_wr(df: pd.DataFrame, window: int = 20) -> pd.Series:
-    d = df.sort_values("entry_time", kind="mergesort")
-    s = (d["r_multiple"] > 0).astype(float).rolling(window).mean() * 100
-    s.index = d["entry_time"].to_numpy()
-    return s
-
-
-def equity_curve(df: pd.DataFrame) -> pd.DataFrame:
-    d = df.sort_values("entry_time", kind="mergesort")
-    cum = d["r_multiple"].cumsum()
-    peak = pd.concat([pd.Series([0.0]), cum]).cummax().iloc[1:]
-    peak.index = cum.index
-    return pd.DataFrame({"entry_time": d["entry_time"].to_numpy(),
-                         "cum_r": cum.to_numpy(),
-                         "drawdown": (peak - cum).to_numpy()})
-
-
-# ─────────────────────── quantile bucket screen ─────────────────────
-
-def quantile_buckets(df: pd.DataFrame, col: str, q: int = 5,
-                     min_n: int = 10) -> pd.DataFrame:
-    """Equal-frequency buckets of `col` with per-half mean R.
-
-    Bin edges are fit on the WHOLE book (both halves share edges).
-    Result carries gap_h1 / gap_h2 / sign_stable in `.attrs` and as
-    repeated columns so the CSV is self-describing.
-    """
-    v = pd.to_numeric(df.get(col), errors="coerce")
-    d = df.loc[v.notna()].copy()
-    d["_v"] = v.loc[v.notna()]
-    empty = pd.DataFrame(columns=["bin", "lo", "hi", "n", "mean_r", "pf", "wr",
-                                  "n_h1", "mean_r_h1", "n_h2", "mean_r_h2",
-                                  "gap_h1", "gap_h2", "sign_stable"])
-    empty.attrs = {"feature": col, "gap_h1": float("nan"),
-                   "gap_h2": float("nan"), "sign_stable": False, "n": 0,
-                   "mode": "empty"}
-    if d.empty:
-        return empty
-
-    nuniq = int(d["_v"].nunique())
-    if nuniq < 3:
-        mode = "values"
-        d["_bin"] = d["_v"]
-        order = sorted(d["_v"].unique())
-        edges = {b: (float(b), float(b)) for b in order}
-    else:
-        mode = "qcut"
-        binned = pd.qcut(d["_v"], q, duplicates="drop")
-        d["_bin"] = binned
-        order = list(binned.cat.categories)
-        edges = {b: (float(b.left), float(b.right)) for b in order}
-
-    rows = []
-    for b in order:
-        sub = d[d["_bin"] == b]
-        if sub.empty:
-            continue
-        r = sub["r_multiple"]
-        h1 = sub[sub["half"] == 1]["r_multiple"]
-        h2 = sub[sub["half"] == 2]["r_multiple"]
-        lo, hi = edges[b]
-        rows.append({
-            "bin": str(b), "lo": round(lo, 4), "hi": round(hi, 4),
-            "n": int(len(sub)), "mean_r": round(float(r.mean()), 3),
-            "pf": round(_pf(r), 2),
-            "wr": round(float((r > 0).mean() * 100), 2),
-            "n_h1": int(len(h1)),
-            "mean_r_h1": round(float(h1.mean()), 3) if len(h1) else float("nan"),
-            "n_h2": int(len(h2)),
-            "mean_r_h2": round(float(h2.mean()), 3) if len(h2) else float("nan"),
-        })
-    out = pd.DataFrame(rows)
-    if out.empty:
-        return empty
-
-    bot, top = out.iloc[0], out.iloc[-1]
-    gap_h1 = float(top["mean_r_h1"] - bot["mean_r_h1"])
-    gap_h2 = float(top["mean_r_h2"] - bot["mean_r_h2"])
-    populated = min(bot["n_h1"], bot["n_h2"], top["n_h1"], top["n_h2"]) >= min_n
-    signs_ok = (not math.isnan(gap_h1) and not math.isnan(gap_h2)
-                and np.sign(gap_h1) != 0
-                and np.sign(gap_h1) == np.sign(gap_h2))
-    sign_stable = bool(populated and signs_ok and len(out) >= 2)
-    out["gap_h1"] = round(gap_h1, 3) if not math.isnan(gap_h1) else float("nan")
-    out["gap_h2"] = round(gap_h2, 3) if not math.isnan(gap_h2) else float("nan")
-    out["sign_stable"] = sign_stable
-    out.attrs = {"feature": col, "gap_h1": gap_h1, "gap_h2": gap_h2,
-                 "sign_stable": sign_stable, "n": int(len(d)), "mode": mode}
-    return out
-
-
-def feature_screen(df: pd.DataFrame, q: int = 5, min_n: int = 10,
-                   columns: Iterable[str] | None = None) -> pd.DataFrame:
-    """Bucket screen over every decision-time numeric feature in the book."""
-    cols = list(columns) if columns is not None else feature_columns(df)
-    rows = []
-    for c in cols:
-        bk = quantile_buckets(df, c, q=q, min_n=min_n)
-        a = bk.attrs
-        stable = bool(a.get("sign_stable"))
-        g1, g2 = a.get("gap_h1", float("nan")), a.get("gap_h2", float("nan"))
-        min_abs = (min(abs(g1), abs(g2)) if stable else 0.0)
-        rows.append({"feature": c,
-                     "gap_h1": round(g1, 3) if g1 == g1 else float("nan"),
-                     "gap_h2": round(g2, 3) if g2 == g2 else float("nan"),
-                     "min_abs_gap": round(float(min_abs), 3),
-                     "sign_stable": stable,
-                     "n": int(a.get("n", 0)),
-                     "bins": int(len(bk))})
-    out = pd.DataFrame(rows)
-    if out.empty:
-        return out
-    return out.sort_values(["min_abs_gap", "n"],
-                           ascending=[False, False]).reset_index(drop=True)
-
-
-# ────────────────────── higher-timeframe extension ──────────────────
-
-def wilder_atr(bars: pd.DataFrame, period: int = 14) -> pd.Series:
-    h, l, c = bars["High"], bars["Low"], bars["Close"]
-    tr = pd.concat([h - l, (h - c.shift()).abs(), (l - c.shift()).abs()],
-                   axis=1).max(axis=1)
-    return tr.ewm(alpha=1 / period, min_periods=period, adjust=False).mean()
-
-
-def _ext_series(parquet: str, minutes: int) -> pd.DataFrame:
-    bars = pd.read_parquet(parquet).sort_index()
-    bars = bars[~bars.index.duplicated(keep="last")]
-    ext = ((bars["Close"] - bars["Close"].rolling(20).mean())
-           / wilder_atr(bars, 14))
-    out = pd.DataFrame({
-        "close_time": bars.index + pd.Timedelta(minutes=minutes),
-        "ext": ext.to_numpy(),
-    }).dropna().reset_index(drop=True)
-    # Books parse to datetime64[us, UTC] (format="mixed"), parquet indexes are
-    # [ns, UTC]; merge_asof refuses mismatched resolutions.
-    out["close_time"] = out["close_time"].astype("datetime64[ns, UTC]")
-    return out
-
-
-def add_htf_extension(df: pd.DataFrame, parquet_4h: str | None,
-                      parquet_1d: str | None) -> pd.DataFrame:
-    """Add direction-signed H4 / D1 stretch-from-mean in ATR units.
-
-    ext = (close − 20-bar SMA of close) / Wilder ATR(14) on that timeframe,
-    taken from the LAST higher-TF bar whose CLOSE time (bar label + one bar)
-    is <= decision_time — i.e. only information the live scan could have had.
-    Signed +1 for BUY, −1 for SELL, so positive always means "extended in the
-    direction of the trade".
-    """
-    out = df.copy()
-    sign = np.where(out["direction"].astype(str).str.upper().str.startswith("B"),
-                    1.0, -1.0)
-    for col, parquet, minutes in (("x_h4_ext_atr", parquet_4h, 240),
-                                  ("x_d1_ext_atr", parquet_1d, 1440)):
-        if not parquet:
-            continue
-        right = _ext_series(parquet, minutes)
-        left = out[["decision_time"]].reset_index()
-        left["decision_time"] = left["decision_time"].astype(
-            "datetime64[ns, UTC]")
-        left = left.sort_values("decision_time", kind="mergesort")
-        merged = pd.merge_asof(left, right, left_on="decision_time",
-                               right_on="close_time", direction="backward")
-        merged["_stale_h"] = ((merged["decision_time"] - merged["close_time"])
-                              .dt.total_seconds() / 3600.0)
-        merged = merged.set_index("index").reindex(out.index)
-        out[col] = merged["ext"].to_numpy() * sign
-        out.attrs[f"{col}_max_stale_h"] = float(
-            merged["_stale_h"].max()) if len(merged) else float("nan")
-    return out
-
-
-# ──────────────────────────── reporting ─────────────────────────────
-
-def _fmt(v, nd=3):
-    if isinstance(v, float):
-        if math.isinf(v):
-            return "inf"
-        if math.isnan(v):
-            return "n/a"
-        return f"{v:.{nd}f}"
-    return str(v)
-
-
-def _metrics_block(name: str, m: dict) -> list[str]:
-    return [
-        f"  {name:<10} n={m['n']:<5} totalR={_fmt(m['total_r'], 2):>8}  "
-        f"meanR={_fmt(m['mean_r'])}  medR={_fmt(m['median_r'])}  "
-        f"WR={_fmt(m['wr'], 2)}%  PF={_fmt(m['pf'], 2)}",
-        f"  {'':<10} maxDD={_fmt(m['max_dd'], 2)}  recovery={_fmt(m['recovery'], 2)}"
-        f"  sharpe={_fmt(m['sharpe'])}  expectancy={_fmt(m['expectancy'])}"
-        f" (wr·avgW − lr·avgL = {_fmt(m['expectancy_alt'])})",
-        f"  {'':<10} avg bars={_fmt(m['avg_bars'], 1)}  "
-        f"avg MFE={_fmt(m['avg_mfe'])}R  avg MAE={_fmt(m['avg_mae'])}R",
-    ]
-
-
-def build_report(df: pd.DataFrame, book: str, bar_minutes: int, split: str,
-                 out_dir: str, q: int = 5, min_n: int = 10) -> str:
-    lines: list[str] = []
-    add = lines.append
-    cut = split_instant(df["entry_time"], split)
-    add("=" * 78)
-    add(f"BOOK REPORT  {os.path.basename(book)}")
-    add("=" * 78)
-    add(f"trades           : {len(df)}")
-    add(f"entry window     : {df['entry_time'].min()}  ->  {df['entry_time'].max()}")
-    add(f"bar minutes      : {bar_minutes}  (decision_time = entry_time − 1 bar)")
-    add(f"split rule       : {split}  ->  cut at {cut}  "
-        f"(half 1 = before, half 2 = at/after)")
-    add(f"half sizes       : h1={int((df['half'] == 1).sum())}  "
-        f"h2={int((df['half'] == 2).sum())}")
-    add("session map      : asian 3-6, london 7-12, overlap 13-16, ny 17-22, "
-        "else off (on DECISION hour, UTC)")
-    add("bucket edges     : quantile bins are fit on the WHOLE book; the two "
-        "halves share the same edges.")
-    feats = feature_columns(df)
-    add(f"screen features  : {len(feats)}  "
-        f"({sum(1 for c in feats if c.startswith(('f_', 'x_')))} f_/x_ columns)")
-    for k, v in df.attrs.items():
-        add(f"htf staleness    : {k} = {_fmt(v, 1)} h (max age of the matched bar)")
-
-    add("")
-    add("── HEADLINE METRICS ──")
-    lines.extend(_metrics_block("full", base_metrics(df["r_multiple"], df)))
-    for h in (1, 2):
-        sub = df[df["half"] == h]
-        lines.extend(_metrics_block(f"half {h}",
-                                    base_metrics(sub["r_multiple"], sub)))
-
-    group_specs = [
-        ("weekday", "weekday"), ("decision_hour", "decision_hour"),
-        ("session", "session"), ("quality", "quality"), ("regime", "regime"),
-        ("vol_regime", "vol_regime"), ("direction", "direction"),
-        ("strategy", "strategy"), ("exit_reason", "exit_reason"),
-        ("indicators_agree", "indicators_agree"),
-    ]
-    tables = {}
-    for name, col in group_specs:
-        if col not in df.columns:
-            continue
-        t = group_table(df, col, min_n=min_n)
-        tables[name] = t
-        add("")
-        add(f"── BY {name.upper()} ──")
-        add(t.to_string() if not t.empty else "  (empty)")
-
-    # (weekday, decision_hour) pivots
-    add("")
-    add("── weekday x decision_hour : TOTAL R ──")
-    wh = group_table(df, ["weekday", "decision_hour"], min_n=min_n)
-    if not wh.empty:
-        piv_r = wh["total_r"].unstack("decision_hour")
-        piv_n = wh["n"].unstack("decision_hour")
-        add(piv_r.fillna(0).round(2).to_string())
-        add("")
-        add("── weekday x decision_hour : N ──")
-        add(piv_n.fillna(0).astype(int).to_string())
-        tables["weekday_hour"] = wh
-        piv_r.to_csv(os.path.join(out_dir, "groups_weekday_hour_total_r.csv"))
-        piv_n.to_csv(os.path.join(out_dir, "groups_weekday_hour_n.csv"))
-
-    st = streaks(df)
-    add("")
-    add("── STREAKS ──")
-    add(f"  max consecutive wins   : {st['max_win_streak']}")
-    add(f"  max consecutive losses : {st['max_loss_streak']}")
-    add(f"  longest flat stretch   : {st['longest_flat_trades']} trades "
-        f"between equity highs")
-
-    rw = rolling_wr(df, 20).dropna()
-    add("")
-    add("── ROLLING WIN RATE (20 trades) ──")
-    if len(rw):
-        add(f"  min={_fmt(rw.min(), 2)}%  median={_fmt(rw.median(), 2)}%  "
-            f"max={_fmt(rw.max(), 2)}%  n={len(rw)}")
-        add("  last 5: " + ", ".join(
-            f"{ts:%Y-%m-%d %H:%M}={v:.2f}%" for ts, v in rw.tail(5).items()))
-    else:
-        add("  (fewer than 20 trades)")
-
-    scr = feature_screen(df, q=q, min_n=min_n)
-    add("")
-    add("── FEATURE SCREEN (top 25 by min |gap| across halves) ──")
-    add("  gap = mean R(top bucket) − mean R(bottom bucket), per half; "
-        "min_abs_gap is 0 unless the sign is stable.")
-    add(scr.head(25).to_string(index=False) if not scr.empty else "  (none)")
-
-    stable = scr[scr["sign_stable"]].head(10) if not scr.empty else scr
-    add("")
-    add(f"── BUCKET TABLES FOR THE TOP {len(stable)} SIGN-STABLE FEATURES ──")
-    if stable.empty:
-        add("  (no feature has a sign-stable top-vs-bottom gap)")
-    for feat in stable["feature"]:
-        bk = quantile_buckets(df, feat, q=q, min_n=min_n)
-        add("")
-        add(f"  · {feat}  (mode={bk.attrs['mode']}, n={bk.attrs['n']}, "
-            f"gap_h1={_fmt(bk.attrs['gap_h1'])}, "
-            f"gap_h2={_fmt(bk.attrs['gap_h2'])})")
-        add(bk.to_string(index=False))
-        bk.to_csv(os.path.join(out_dir, f"buckets_{feat}.csv"), index=False)
-
-    # CSV artefacts
-    for name, t in tables.items():
-        t.to_csv(os.path.join(out_dir, f"groups_{name}.csv"))
-    scr.to_csv(os.path.join(out_dir, "feature_screen.csv"), index=False)
-    equity_curve(df).to_csv(os.path.join(out_dir, "equity.csv"), index=False)
-    all_buckets = []
-    for c in feats:
-        bk = quantile_buckets(df, c, q=q, min_n=min_n)
-        if bk.empty:
-            continue
-        bk = bk.copy()
-        bk.insert(0, "feature", c)
-        all_buckets.append(bk)
-    if all_buckets:
-        pd.concat(all_buckets, ignore_index=True).to_csv(
-            os.path.join(out_dir, "buckets.csv"), index=False)
-    return "\n".join(lines)
-
-
-def main() -> None:
-    ap = argparse.ArgumentParser(description=__doc__.split("\n")[0])
-    ap.add_argument("--book", required=True)
-    ap.add_argument("--bar-minutes", type=int, required=True)
-    ap.add_argument("--split", default="median",
-                    help="'median' (default) or YYYY-MM-DD")
-    ap.add_argument("--parquet-4h", default=None)
-    ap.add_argument("--parquet-1d", default=None)
-    ap.add_argument("--q", type=int, default=5)
-    ap.add_argument("--min-n", type=int, default=10)
-    ap.add_argument("--out", default="backtest/research/book_report_out")
-    args = ap.parse_args()
-
-    os.makedirs(args.out, exist_ok=True)
-    df = load_book(args.book, args.bar_minutes, args.split)
-    if args.parquet_4h or args.parquet_1d:
-        df = add_htf_extension(df, args.parquet_4h, args.parquet_1d)
-    text = build_report(df, args.book, args.bar_minutes, args.split,
-                        args.out, q=args.q, min_n=args.min_n)
-    print(text)
-    with open(os.path.join(args.out, "report.txt"), "w") as fh:
-        fh.write(text + "\n")
-    print(f"\n[written] {os.path.join(args.out, 'report.txt')}")
-
-
-if __name__ == "__main__":
-    main()

backtest/research/burst_indicator_map.py

@@ -1,121 +0,0 @@
-"""What do the system's OWN indicators read right before a violent bar?  Analysis tooling only.
-
-Uses backtest.gpu_backtest.precompute_indicators (the same series the backtester feeds the
-engine) on 3 years of bars; every state is taken at t-1 (known before the bar). Reports burst
-rate (range >= 3x prior ATR14) for the NEXT bar and within the NEXT HOUR by indicator state,
-by regime, by volatility regime, by weekday x hour, day of month, calendar proximity — each in
-both halves — and whether the state predicts the burst's DIRECTION.
-
-    python backtest/research/burst_indicator_map.py PARQUET BAR_MINUTES [--start --end]
-"""
-import argparse, sys
-from pathlib import Path
-import numpy as np, pandas as pd
-ROOT = Path(__file__).resolve().parents[2]; sys.path.insert(0, str(ROOT)); sys.path.insert(0, str(ROOT / "backtest"))
-from gpu_backtest import precompute_indicators
-from config import TechnicalConfig
-
-CAL = ROOT / "backtest" / "cache" / "calendar.parquet"
-
-
-def wilder_atr(df, period=14):
-    pc = df["Close"].shift(1)
-    tr = pd.concat([df["High"] - df["Low"], (df["High"] - pc).abs(), (df["Low"] - pc).abs()], axis=1).max(axis=1)
-    return tr.ewm(alpha=1 / period, min_periods=period, adjust=False).mean()
-
-
-def rate_table(d, col, target, half, order=None):
-    base1 = d.loc[half == 1, target].mean(); base2 = d.loc[half == 2, target].mean()
-    g = d.groupby(col, observed=True)
-    out = pd.DataFrame({"n": g.size(),
-                        "h1_lift": d[half == 1].groupby(col, observed=True)[target].mean() / base1,
-                        "h2_lift": d[half == 2].groupby(col, observed=True)[target].mean() / base2,
-                        "rate%": 100 * g[target].mean()})
-    if order is not None: out = out.reindex([o for o in order if o in out.index])
-    out["stable"] = np.where((out.h1_lift > 1.15) & (out.h2_lift > 1.15), "HIGH", np.where((out.h1_lift < 0.85) & (out.h2_lift < 0.85), "low", ""))
-    return out.round(2)
-
-
-def main():
-    ap = argparse.ArgumentParser(); ap.add_argument("parquet"); ap.add_argument("bar_minutes", type=int)
-    ap.add_argument("--start", default="2023-09-05"); ap.add_argument("--end", default="2026-09-05")
-    a = ap.parse_args(); bm = a.bar_minutes
-    px = pd.read_parquet(a.parquet)
-    px = px[(px.index >= pd.Timestamp(a.start, tz="UTC") - pd.Timedelta(days=30)) & (px.index < pd.Timestamp(a.end, tz="UTC"))]
-    ic = precompute_indicators(px, TechnicalConfig())
-    d = pd.DataFrame(index=px.index)
-    atr_prev = wilder_atr(px).shift(1)
-    rng = px["High"] - px["Low"]
-    d["burst"] = (rng >= 3.0 * atr_prev).astype(int)
-    H = max(1, 60 // bm)
-    d["next_hour"] = d["burst"][::-1].rolling(H, min_periods=1).max()[::-1].astype(int)
-    d["next_bar"] = d["burst"]
-    d["bdir"] = np.sign(px["Close"] - px["Open"])
-    # states at t-1
-    sh = lambda arr: pd.Series(np.asarray(arr, dtype=float), index=px.index).shift(1)
-    rsi = sh(ic.rsi); adx = sh(ic.adx); adxs = sh(ic.adx_slope); hist = sh(ic.macd_hist); bbp = sh(ic.bb_position)
-    bbw = sh(ic.bb_band_width); bbaw = sh(ic.bb_avg_width); kcsq = sh(ic.kc_squeeze); roc = sh(ic.roc)
-    ef, em, es = sh(ic.ema_fast), sh(ic.ema_medium), sh(ic.ema_slow); et = sh(ic.ema_trend) if ic.ema_trend is not None else None
-    close1 = px["Close"].shift(1); atr1 = sh(ic.atr)
-    kcu, kcl = sh(ic.kc_upper), sh(ic.kc_lower)
-    d["rsi"] = pd.cut(rsi, [0, 30, 40, 50, 60, 70, 100], labels=["<30", "30-40", "40-50", "50-60", "60-70", ">=70"])
-    d["adx"] = pd.cut(adx, [0, 20, 25, 35, 50, 200], labels=["<20", "20-25", "25-35", "35-50", ">=50"])
-    d["adx_slope"] = pd.cut(adxs, [-1e9, -3, 0, 3, 1e9], labels=["falling<-3", "-3..0", "0..3", "rising>3"])
-    hist_prev = hist.shift(1)
-    d["macd"] = np.select([(hist > 0) & (hist > hist_prev), (hist > 0), (hist < 0) & (hist < hist_prev), (hist < 0)],
-                          ["pos&rising", "pos&fading", "neg&falling", "neg&fading"], default="flat")
-    d["bb_pos"] = pd.cut(bbp, [-1e9, 0, 0.2, 0.8, 1.0, 1e9], labels=["<0 (below)", "0-0.2", "0.2-0.8", "0.8-1", ">1 (above)"])
-    d["bb_squeeze"] = np.where(bbw < 0.75 * bbaw, "squeeze", "normal")
-    d["kc_squeeze"] = np.where(kcsq > 0, "TTM squeeze", "no")
-    d["kc_pos"] = np.select([close1 > kcu, close1 < kcl], ["above KC", "below KC"], default="inside")
-    align = np.sign(close1 - ef) + np.sign(ef - em) + np.sign(em - es) + (np.sign(es - et) if et is not None else 0)
-    d["ema_align"] = pd.cut(align, [-5, -3.5, -0.5, 0.5, 3.5, 5], labels=["fully bear(-4)", "mostly bear", "mixed", "mostly bull", "fully bull(+4)"])
-    ribbon = (pd.concat([ef, em, es], axis=1).max(axis=1) - pd.concat([ef, em, es], axis=1).min(axis=1)) / atr1
-    d["ribbon"] = np.where(ribbon < 1.0, "compressed(<1 ATR)", "expanded")
-    ema_dir = np.where(align > 0, 1, np.where(align < 0, -1, 0))
-    d["regime"] = np.select([(adx >= 25) & (ema_dir > 0), (adx >= 25) & (ema_dir < 0), adx >= 25, adxs > 3.0],
-                            ["trending_up", "trending_down", "trending(mixed)", "transitioning"], default="ranging")
-    atr_pct = atr1.rolling(50).apply(lambda w: (w[-1] > w).mean() * 100, raw=True)
-    d["vol_regime"] = pd.cut(atr_pct, [-1, 30, 60, 80, 101], labels=["low", "normal", "high", "extreme"])
-    d["roc"] = pd.cut(roc, [-1e9, -1, -0.3, 0.3, 1, 1e9], labels=["<-1%", "-1..-0.3", "-0.3..0.3", "0.3..1", ">1%"])
-    d["ema200_dist"] = pd.cut((close1 - et) / et * 100 if et is not None else pd.Series(np.nan, index=px.index),
-                              [-1e9, -2.5, -0.5, 0.5, 2.5, 1e9], labels=["<-2.5%", "-2.5..-0.5", "near(±0.5%)", "0.5..2.5", ">2.5%"])
-    # time
-    ct = px.index + pd.Timedelta(minutes=bm)
-    d["hour"] = ct.hour; d["weekday"] = ct.weekday; d["dom"] = pd.cut(ct.day, [0, 5, 10, 15, 20, 25, 31], labels=["1-5", "6-10", "11-15", "16-20", "21-25", "26-31"])
-    d["first_friday"] = np.where((ct.weekday == 4) & (ct.day <= 7), "first Friday", "other")
-    # calendar proximity (high-impact USD)
-    cal = pd.read_parquet(CAL); cal["ts"] = pd.to_datetime(cal.dateline, unit="s", utc=True)
-    hi = np.sort(cal[(cal.currency == "USD") & (cal.impactName.str.lower() == "high")].ts.values.astype("datetime64[ns]"))
-    t = ct.values.astype("datetime64[ns]")
-    nxt = np.searchsorted(hi, t)  # next event at/after bar close
-    mins_to = np.where(nxt < len(hi), (hi[np.minimum(nxt, len(hi) - 1)] - t) / np.timedelta64(1, "m"), 1e9)
-    prv = nxt - 1
-    mins_since = np.where(prv >= 0, (t - hi[np.maximum(prv, 0)]) / np.timedelta64(1, "m"), 1e9)
-    d["calendar"] = np.select([mins_since <= 30, mins_to <= 15, mins_to <= 60, mins_to <= 180], ["0-30m AFTER hi-impact", "next 15m", "15-60m ahead", "1-3h ahead"], default="none within 3h")
-    d = d[d.index >= pd.Timestamp(a.start, tz="UTC")].dropna(subset=["rsi", "adx", "bb_pos", "vol_regime"])
-    half = pd.Series(np.where(np.arange(len(d)) < len(d) // 2, 1, 2), index=d.index)
-    print(f"bars {len(d)}  base next_bar {100*d.next_bar.mean():.3f}%  next_hour {100*d.next_hour.mean():.2f}%  (lift = rate / base; 'HIGH' = >1.15x in BOTH halves, 'low' = <0.85x in both)")
-    groups = ["calendar", "regime", "vol_regime", "rsi", "adx", "adx_slope", "macd", "bb_pos", "bb_squeeze", "kc_squeeze", "kc_pos", "ema_align", "ribbon", "roc", "ema200_dist", "dom", "first_friday"]
-    for target in ("next_bar", "next_hour"):
-        print(f"\n################ TARGET: {target} ################")
-        for g in groups:
-            print(f"-- {g}"); print(rate_table(d, g, target, half).to_string())
-    # weekday x hour top cells (next_bar)
-    piv = d.pivot_table(index="weekday", columns="hour", values="next_bar", aggfunc="mean") * 100
-    base = 100 * d.next_bar.mean()
-    cells = piv.stack().sort_values(ascending=False)
-    print(f"\n################ TOP 15 weekday x close-hour cells (next-bar burst pct, base {base:.2f}) ################")
-    names = {0: "Mon", 1: "Tue", 2: "Wed", 3: "Thu", 4: "Fri", 6: "Sun"}
-    for (wd, h), v in cells.head(15).items():
-        n = int(((d.weekday == wd) & (d.hour == h)).sum())
-        print(f"  {names.get(wd, wd)} {h:02d}:xx  {v:.2f}%  ({v/base:.1f}x, n={n})")
-    # direction: does the state predict the burst direction?
-    b = d[d.burst == 1]
-    print("\n################ BURST DIRECTION vs prior state (share of bursts going UP) ################")
-    for g, up_states in (("ema_align", None), ("rsi", None), ("macd", None), ("regime", None)):
-        print(f"-- {g}: " + ", ".join(f"{k}: {100*(v>0).mean():.0f}% up (n={len(v)})" for k, v in b.groupby(g, observed=True).bdir if len(v) >= 20))
-
-
-if __name__ == "__main__":
-    main()

backtest/research/burst_map.py

@@ -1,445 +0,0 @@
-"""Where and when gold moves violently — burst map of the raw bars.
-
-Usage:
-    python backtest/research/burst_map.py --parquet PATH --bar-minutes N \
-        [--k 3,4,5] [--book BOOK.csv] [--within-bars 12] \
-        [--start YYYY-MM-DD --end YYYY-MM-DD] [--out DIR]
-
-A "burst" is one bar whose range is >= K x the PRIOR bar's Wilder ATR(14) —
-the same definition burst_profile.py uses (that module computes it inline
-inside burst_table() and exposes no reusable helper, so the formula is
-replicated here bar-for-bar: TR = max(H-L, |H-Cprev|, |L-Cprev|), ATR =
-TR.ewm(alpha=1/14, min_periods=14, adjust=False).mean(), ratio =
-(H-L) / ATR.shift(1)).
-
-Reports:
-  * burst frequency by (weekday, UTC bar-CLOSE hour), plus mean |1 h| and
-    |4 h| absolute moves in $ and in ATR units on the same grid;
-  * post-burst drift in the burst's own direction over 6/12/24/48 bars,
-    overall and by session bucket — does a violent bar continue or snap back;
-  * the largest trailing-60-minute close-to-close moves, de-overlapped;
-  * if a trade book is given: how trades that were decided just after a burst,
-    or that were OPEN when one printed, performed against the rest.
-
-Every ATR-unit number here is normalised by the PRIOR bar's ATR — the same
-denominator as the burst test, so a "4x burst that ran another 4 ATR" reads
-directly. All hours are UTC and are the bar's CLOSE hour (label + one bar),
-matching the clock the live gates evaluate.
-
-Analysis tooling only — never imported by the live code.
-"""
-from __future__ import annotations
-
-import argparse
-import math
-import os
-
-import numpy as np
-import pandas as pd
-
-try:  # package import (backtest.research.burst_map)
-    from backtest.research.book_report import base_metrics, load_book, session_of
-except ImportError:  # run directly as a script from anywhere
-    from book_report import base_metrics, load_book, session_of
-
-SLOT_UNKNOWN = ""
-
-
-# ─────────────────────────── bar primitives ─────────────────────────
-
-def load_bars(parquet: str, start: str | None = None,
-              end: str | None = None) -> pd.DataFrame:
-    """Read a cached OHLCV parquet, de-duplicated and oldest-first."""
-    df = pd.read_parquet(parquet).sort_index()
-    df = df[~df.index.duplicated(keep="last")]
-    if start:
-        df = df[df.index >= pd.Timestamp(start, tz="UTC")]
-    if end:
-        df = df[df.index <= pd.Timestamp(end, tz="UTC")]
-    return df
-
-
-def wilder_atr(df: pd.DataFrame, period: int = 14) -> pd.Series:
-    """Wilder ATR — identical to the inline formula in burst_profile.py."""
-    h, l, c = df["High"], df["Low"], df["Close"]
-    tr = pd.concat([h - l, (h - c.shift()).abs(), (l - c.shift()).abs()],
-                   axis=1).max(axis=1)
-    return tr.ewm(alpha=1 / period, min_periods=period, adjust=False).mean()
-
-
-def burst_ratio(df: pd.DataFrame, period: int = 14) -> pd.Series:
-    """Bar range expressed in PRIOR-bar ATRs."""
-    return (df["High"] - df["Low"]) / wilder_atr(df, period).shift(1)
-
-
-def burst_flags(df: pd.DataFrame, k: float, period: int = 14) -> pd.Series:
-    """True where the bar's range is >= k x the prior bar's Wilder ATR(14)."""
-    return (burst_ratio(df, period) >= k).fillna(False)
-
-
-def burst_dir(df: pd.DataFrame) -> pd.Series:
-    """+1 up bar, −1 down bar, 0 doji (sign of Close − Open)."""
-    return np.sign(df["Close"] - df["Open"]).astype(float)
-
-
-def close_times(df: pd.DataFrame, bar_minutes: int) -> pd.DatetimeIndex:
-    """Bar CLOSE timestamps (the parquet index labels bar OPEN time)."""
-    return df.index + pd.Timedelta(minutes=bar_minutes)
-
-
-# ───────────────────── when does gold move? tables ──────────────────
-
-def hour_weekday_table(df: pd.DataFrame, k: float, bar_minutes: int = 5,
-                       ks: tuple[float, ...] | None = None) -> dict:
-    """Burst frequency and absolute-move size on a (weekday, close-hour) grid.
-
-    Returns a dict of DataFrames (several tables are wanted, one frame each):
-      burst_pct  : weekday rows x close-hour cols, % of bars that are >= k x
-                   bursts (one frame; per-k frames live in `burst_pct_by_k`)
-      absmove    : (weekday, close-hour) rows; bars, mean |1 h move| and
-                   |4 h move| in $ and in prior-bar ATR units
-      by_hour    : the same stats collapsed to close-hour
-      by_weekday : the same stats collapsed to weekday
-    """
-    ks = tuple(ks) if ks else (k,)
-    ct = close_times(df, bar_minutes)
-    hour = pd.Series(ct.hour, index=df.index)
-    wd = pd.Series(ct.weekday, index=df.index)
-    atr_prev = wilder_atr(df).shift(1)
-
-    per_hour = max(1, int(round(60 / bar_minutes)))
-    c = df["Close"]
-    mv1 = (c - c.shift(per_hour)).abs()
-    mv4 = (c - c.shift(4 * per_hour)).abs()
-    stats = pd.DataFrame({
-        "weekday": wd, "hour": hour,
-        "abs1h_usd": mv1, "abs4h_usd": mv4,
-        "abs1h_atr": mv1 / atr_prev, "abs4h_atr": mv4 / atr_prev,
-    })
-    for kk in ks:
-        stats[f"burst_{kk:g}x"] = burst_flags(df, kk).astype(float) * 100.0
-
-    agg = {"abs1h_usd": "mean", "abs4h_usd": "mean",
-           "abs1h_atr": "mean", "abs4h_atr": "mean"}
-    agg.update({f"burst_{kk:g}x": "mean" for kk in ks})
-
-    grid = stats.groupby(["weekday", "hour"]).agg(agg)
-    grid.insert(0, "bars", stats.groupby(["weekday", "hour"]).size())
-    by_hour = stats.groupby("hour").agg(agg)
-    by_hour.insert(0, "bars", stats.groupby("hour").size())
-    by_wd = stats.groupby("weekday").agg(agg)
-    by_wd.insert(0, "bars", stats.groupby("weekday").size())
-
-    pct_by_k = {kk: grid[f"burst_{kk:g}x"].unstack("hour") for kk in ks}
-    return {
-        "burst_pct": pct_by_k[k if k in pct_by_k else ks[0]].round(3),
-        "burst_pct_by_k": {kk: v.round(3) for kk, v in pct_by_k.items()},
-        "absmove": grid.round(3),
-        "by_hour": by_hour.round(3),
-        "by_weekday": by_wd.round(3),
-    }
-
-
-def post_burst_drift(df: pd.DataFrame, k: float,
-                     horizons: tuple[int, ...] = (6, 12, 24, 48),
-                     bar_minutes: int = 5) -> pd.DataFrame:
-    """Signed drift IN the burst's direction over the next h bars, in ATRs.
-
-    A positive mean means violent bars continue; negative means they snap
-    back. Split by the session bucket of the burst bar's close hour.
-    """
-    flags = burst_flags(df, k)
-    d = burst_dir(df)
-    atr_prev = wilder_atr(df).shift(1)
-    ct = close_times(df, bar_minutes)
-    bucket = pd.Series([session_of(h) for h in ct.hour], index=df.index)
-    c = df["Close"]
-
-    rows = []
-    for h in horizons:
-        fwd = (c.shift(-h) - c) * d / atr_prev
-        sub = fwd[flags].dropna()
-        buckets = bucket[flags].reindex(sub.index)
-        rows.append({"horizon_bars": h, "bucket": "ALL", "n": int(len(sub)),
-                     "mean_atr": round(float(sub.mean()), 4) if len(sub) else float("nan"),
-                     "median_atr": round(float(sub.median()), 4) if len(sub) else float("nan"),
-                     "pct_positive": round(float((sub > 0).mean() * 100), 2) if len(sub) else float("nan")})
-        for b, s in sub.groupby(buckets):
-            rows.append({"horizon_bars": h, "bucket": b, "n": int(len(s)),
-                         "mean_atr": round(float(s.mean()), 4),
-                         "median_atr": round(float(s.median()), 4),
-                         "pct_positive": round(float((s > 0).mean() * 100), 2)})
-    return pd.DataFrame(rows)
-
-
-def slot_label(weekday: int, hour: int, day: int) -> str:
-    """Time-slot label for a big move.
-
-    NOTE: there is no economic-calendar file in this repo, so these are
-    TIME SLOTS, not confirmed events — "payroll window" means "the clock
-    slot US payrolls normally print in", nothing more.
-    """
-    if weekday == 4 and day <= 7 and hour in (12, 13):
-        return "first-Friday 12:30 slot (payroll-report window)"
-    if weekday == 2 and hour in (18, 19):
-        return "Wednesday 18:00 slot (central-bank window)"
-    if hour in (12, 13):
-        return "12:30 slot (US data window)"
-    if hour == 14:
-        return "14:00 slot"
-    if hour in (21, 22, 23):
-        return "settlement/after-hours"
-    if hour in (0, 1, 2):
-        return "Asia open"
-    return SLOT_UNKNOWN
-
-
-def largest_moves(df: pd.DataFrame, n: int = 40, window_minutes: int = 60,
-                  bar_minutes: int = 5) -> pd.DataFrame:
-    """The n biggest trailing-`window_minutes` close-to-close moves.
-
-    Overlapping windows are de-duplicated greedily: the largest move wins and
-    anything whose close time is within +/- `window_minutes` of an already
-    kept move is dropped, so one violent hour contributes one row.
-    """
-    bars = max(1, int(round(window_minutes / bar_minutes)))
-    c = df["Close"]
-    move = c - c.shift(bars)
-    atr_prev = wilder_atr(df).shift(1)
-    ct = close_times(df, bar_minutes)
-    tbl = pd.DataFrame({
-        "close_time": ct, "move_usd": move.to_numpy(),
-        "move_atr": (move / atr_prev).to_numpy(),
-    }).dropna()
-    tbl["abs_usd"] = tbl["move_usd"].abs()
-    tbl = tbl.sort_values("abs_usd", ascending=False)
-
-    keep, taken = [], []
-    gap = pd.Timedelta(minutes=window_minutes)
-    for _, row in tbl.iterrows():
-        t = row["close_time"]
-        if any(abs(t - u) <= gap for u in taken):
-            continue
-        taken.append(t)
-        keep.append(row)
-        if len(keep) >= n:
-            break
-    out = pd.DataFrame(keep).reset_index(drop=True)
-    if out.empty:
-        return out
-    out["weekday"] = out["close_time"].dt.weekday
-    out["hour"] = out["close_time"].dt.hour
-    out["direction"] = np.where(out["move_usd"] > 0, "UP", "DOWN")
-    out["slot"] = [slot_label(w, h, t.day) for w, h, t
-                   in zip(out["weekday"], out["hour"], out["close_time"])]
-    out["move_usd"] = out["move_usd"].round(2)
-    out["move_atr"] = out["move_atr"].round(2)
-    return out[["close_time", "weekday", "hour", "direction", "move_usd",
-                "move_atr", "slot"]]
-
-
-# ───────────────────────── book <-> burst join ──────────────────────
-
-def _stats(r: pd.Series) -> dict:
-    m = base_metrics(r)
-    return {"n": m["n"], "total_r": round(m["total_r"], 2),
-            "mean_r": round(m["mean_r"], 3), "wr": round(m["wr"], 2),
-            "pf": round(m["pf"], 2) if math.isfinite(m["pf"]) else float("inf")}
-
-
-def join_book_to_bursts(book_df: pd.DataFrame, bars: pd.DataFrame, k: float,
-                        within_bars: int = 12,
-                        bar_minutes: int = 5) -> dict:
-    """Split a trade book by its relationship to burst bars.
-
-    (a) `after_burst`  : decision_time falls within `within_bars` bars after
-        the most recent burst's close (bar distance is counted POSITIONALLY on
-        the bar index, so weekend gaps cannot inflate it).
-    (b) `burst_during` : at least one burst closed inside [entry, exit).
-    (c) `burst_agrees` / `burst_opposes`: of (b), whether the FIRST burst that
-        printed during the trade pushed with or against the trade direction.
-    """
-    ct = close_times(bars, bar_minutes)
-    ct_arr = ct.to_numpy()
-    flags = burst_flags(bars, k).to_numpy()
-    bdir = burst_dir(bars).to_numpy()
-    burst_pos = np.flatnonzero(flags)
-
-    # Last burst position at or before each bar (−1 where none yet).
-    last_burst = np.where(flags, np.arange(len(flags)), -1)
-    last_burst = np.maximum.accumulate(last_burst)
-
-    dec = book_df["decision_time"].astype("datetime64[ns, UTC]").to_numpy()
-    ent = book_df["entry_time"].astype("datetime64[ns, UTC]").to_numpy()
-    exi = book_df["exit_time"].astype("datetime64[ns, UTC]").to_numpy()
-
-    # Position of the last bar that had CLOSED by decision_time.
-    idx = np.searchsorted(ct_arr, dec, side="right") - 1
-    valid = idx >= 0
-    lb = np.where(valid, last_burst[np.clip(idx, 0, len(flags) - 1)], -1)
-    bars_since = np.where((lb >= 0) & valid, idx - lb, np.iinfo(np.int32).max)
-    after = pd.Series((bars_since >= 0) & (bars_since <= within_bars),
-                      index=book_df.index)
-
-    # Bursts printing inside the open trade.
-    bct = ct_arr[burst_pos] if len(burst_pos) else np.array([], dtype=ct_arr.dtype)
-    lo = np.searchsorted(bct, ent, side="left")
-    hi = np.searchsorted(bct, exi, side="left")
-    during = pd.Series(hi > lo, index=book_df.index)
-    trade_sign = np.where(
-        book_df["direction"].astype(str).str.upper().str.startswith("B"),
-        1.0, -1.0)
-    has = hi > lo
-    first_dir = np.zeros(len(book_df), dtype=float)
-    if len(burst_pos) and has.any():
-        # hi > lo guarantees lo[has] indexes a real burst.
-        first_dir[has] = bdir[burst_pos[lo[has]]]
-    agrees = pd.Series(has & (first_dir * trade_sign > 0),
-                       index=book_df.index)
-    opposes = pd.Series(has & (first_dir * trade_sign < 0),
-                        index=book_df.index)
-
-    r = book_df["r_multiple"]
-    out = {
-        "k": k, "within_bars": within_bars,
-        "bursts_in_window": int(len(burst_pos)),
-        "after_burst": _stats(r[after]),
-        "not_after_burst": _stats(r[~after]),
-        "burst_during": _stats(r[during]),
-        "no_burst_during": _stats(r[~during]),
-        "burst_agrees": _stats(r[agrees]),
-        "burst_opposes": _stats(r[opposes]),
-        "median_bars_since_burst": float(np.median(
-            bars_since[bars_since < np.iinfo(np.int32).max])) if (
-            bars_since < np.iinfo(np.int32).max).any() else float("nan"),
-    }
-    return out
-
-
-# ──────────────────────────── reporting ─────────────────────────────
-
-def _fmt_stats(name: str, s: dict) -> str:
-    pf = "inf" if math.isinf(s["pf"]) else f"{s['pf']:.2f}"
-    return (f"    {name:<18} n={s['n']:<5} totalR={s['total_r']:>8.2f}  "
-            f"meanR={s['mean_r']:>7.3f}  WR={s['wr']:>6.2f}%  PF={pf}")
-
-
-def main() -> None:
-    ap = argparse.ArgumentParser(description=__doc__.split("\n")[0])
-    ap.add_argument("--parquet", required=True)
-    ap.add_argument("--bar-minutes", type=int, required=True)
-    ap.add_argument("--k", default="3,4,5",
-                    help="burst thresholds in prior-bar ATRs (default 3,4,5)")
-    ap.add_argument("--book", default=None)
-    ap.add_argument("--book-bar-minutes", type=int, default=None,
-                    help="bar minutes of the BOOK (default: --bar-minutes)")
-    ap.add_argument("--within-bars", type=int, default=12)
-    ap.add_argument("--start", default=None)
-    ap.add_argument("--end", default=None)
-    ap.add_argument("--n-moves", type=int, default=40)
-    ap.add_argument("--out", default="backtest/research/burst_map_out")
-    args = ap.parse_args()
-
-    ks = [float(x) for x in args.k.split(",") if x.strip()]
-    os.makedirs(args.out, exist_ok=True)
-    bars = load_bars(args.parquet, args.start, args.end)
-    bm = args.bar_minutes
-    primary = ks[0]
-
-    lines: list[str] = []
-    add = lines.append
-    add("=" * 78)
-    add(f"BURST MAP  {os.path.basename(args.parquet)}")
-    add("=" * 78)
-    add(f"bars             : {len(bars)}  ({bars.index[0]} -> {bars.index[-1]}"
-        f", labels = bar OPEN time)")
-    add(f"bar minutes      : {bm}   hours below are bar CLOSE hours (UTC)")
-    add(f"burst definition : range >= K x PRIOR bar Wilder ATR(14); "
-        f"K = {', '.join(f'{k:g}' for k in ks)}")
-    add("ATR units        : always normalised by the PRIOR bar's ATR")
-    for k in ks:
-        f = burst_flags(bars, k)
-        add(f"  K={k:g}: {int(f.sum())} burst bars "
-            f"({float(f.mean()) * 100:.3f}% of bars)")
-
-    tabs = hour_weekday_table(bars, primary, bm, ks=tuple(ks))
-    add("")
-    add(f"── BURST % BY (weekday, close-hour), K={primary:g} ──")
-    add(tabs["burst_pct"].fillna(0).round(2).to_string())
-    add("")
-    add("── MEAN ABSOLUTE MOVE BY CLOSE-HOUR (all weekdays) ──")
-    add(tabs["by_hour"].to_string())
-    add("")
-    add("── MEAN ABSOLUTE MOVE BY WEEKDAY ──")
-    add(tabs["by_weekday"].to_string())
-
-    add("")
-    add(f"── POST-BURST DRIFT (in the burst's own direction), K={primary:g} ──")
-    drift = post_burst_drift(bars, primary, bar_minutes=bm)
-    add(drift.to_string(index=False))
-
-    add("")
-    add(f"── LARGEST {args.n_moves} TRAILING-60-MIN MOVES (de-overlapped) ──")
-    add("   slot labels are CLOCK SLOTS — no economic calendar exists in "
-        "this repo, so no event is confirmed.")
-    moves = largest_moves(bars, n=args.n_moves, bar_minutes=bm)
-    add(moves.to_string(index=False))
-
-    join_rows = []
-    if args.book:
-        bbm = args.book_bar_minutes or bm
-        book = load_book(args.book, bbm)
-        add("")
-        add(f"── BOOK JOIN  {os.path.basename(args.book)} "
-            f"({len(book)} trades, book bar = {bbm} min) ──")
-        first, last = close_times(bars, bm)[0], close_times(bars, bm)[-1]
-        outside = int(((book["decision_time"] < first)
-                       | (book["exit_time"] > last)).sum())
-        if outside:
-            add(f"  !! {outside} trade(s) start before / end after the bar "
-                f"window ({first} .. {last}) — their burst context is "
-                f"partial. Widen --start/--end or use a longer parquet.")
-        add("  CAUTION: group (b) is survivorship-shaped — a trade only sees "
-            "a burst if it survived long enough to still be open, so its high "
-            "PF is mostly 'winners last longer', not a burst edge.")
-        for k in ks:
-            j = join_book_to_bursts(book, bars, k, args.within_bars, bm)
-            add(f"  K={k:g}  ({j['bursts_in_window']} bursts in window; "
-                f"median bars since last burst = "
-                f"{j['median_bars_since_burst']:.0f})")
-            add(f"    (a) decided within {args.within_bars} bars of a burst")
-            add(_fmt_stats("after_burst", j["after_burst"]))
-            add(_fmt_stats("not_after_burst", j["not_after_burst"]))
-            add("    (b) a burst printed while the trade was open")
-            add(_fmt_stats("burst_during", j["burst_during"]))
-            add(_fmt_stats("no_burst_during", j["no_burst_during"]))
-            add("    (c) of (b), first burst vs the trade's direction")
-            add(_fmt_stats("burst_agrees", j["burst_agrees"]))
-            add(_fmt_stats("burst_opposes", j["burst_opposes"]))
-            for grp in ("after_burst", "not_after_burst", "burst_during",
-                        "no_burst_during", "burst_agrees", "burst_opposes"):
-                join_rows.append({"k": k, "group": grp,
-                                  "within_bars": args.within_bars,
-                                  **j[grp]})
-
-    text = "\n".join(lines)
-    print(text)
-    with open(os.path.join(args.out, "burst_map.txt"), "w") as fh:
-        fh.write(text + "\n")
-    pd.concat([v.assign(k=kk) for kk, v in tabs["burst_pct_by_k"].items()]
-              ).to_csv(os.path.join(args.out, "hour_weekday_burst_pct.csv"))
-    tabs["absmove"].to_csv(os.path.join(args.out, "hour_weekday_absmove.csv"))
-    tabs["by_hour"].to_csv(os.path.join(args.out, "by_hour.csv"))
-    tabs["by_weekday"].to_csv(os.path.join(args.out, "by_weekday.csv"))
-    pd.concat([post_burst_drift(bars, k, bar_minutes=bm).assign(k=k)
-               for k in ks]).to_csv(
-        os.path.join(args.out, "post_burst_drift.csv"), index=False)
-    moves.to_csv(os.path.join(args.out, "largest_moves.csv"), index=False)
-    if join_rows:
-        pd.DataFrame(join_rows).to_csv(
-            os.path.join(args.out, "book_join.csv"), index=False)
-    print(f"\n[written] {os.path.join(args.out, 'burst_map.txt')}")
-
-
-if __name__ == "__main__":
-    main()

backtest/research/burst_precursors.py

@@ -1,127 +0,0 @@
-"""Are there signs BEFORE a violent bar?  Analysis tooling only — never imported by the live code.
-
-For every bar t, features are computed from bars <= t-1 only. Targets:
-  next_bar  : bar t itself is a burst (range >= K x prior Wilder ATR14)
-  next_hour : any burst in bars t .. t+H-1
-Reports per-quintile burst rates (lift vs base rate) in both halves, the same
-within/outside the scheduled-data hours, and an out-of-sample logistic model
-(fit on half 1, scored on half 2): AUC + lift in the top 5% of scores.
-
-    python backtest/research/burst_precursors.py PARQUET BAR_MINUTES [--k 3] [--start ... --end ...]
-"""
-import argparse, sys
-import numpy as np, pandas as pd
-
-
-def wilder_atr(df, period=14):
-    pc = df["Close"].shift(1)
-    tr = pd.concat([df["High"] - df["Low"], (df["High"] - pc).abs(), (df["Low"] - pc).abs()], axis=1).max(axis=1)
-    return tr.ewm(alpha=1 / period, min_periods=period, adjust=False).mean()
-
-
-def build(df, bar_minutes, k):
-    d = df.copy()
-    rng = d["High"] - d["Low"]
-    atr_prev = wilder_atr(d).shift(1)                       # ATR known at t-1
-    d["burst"] = (rng >= k * atr_prev).astype(int)
-    H = max(1, 60 // bar_minutes)
-    # burst within next H bars (t..t+H-1)
-    d["next_hour"] = d["burst"][::-1].rolling(H, min_periods=1).max()[::-1].astype(int)
-    d["next_bar"] = d["burst"]
-    close_t = d.index + pd.Timedelta(minutes=bar_minutes)
-    d["hour"] = close_t.hour
-    r1 = rng.shift(1)
-    d["squeeze6"] = r1.rolling(6).mean() / atr_prev
-    d["squeeze12"] = r1.rolling(12).mean() / atr_prev
-    d["last_bar_range"] = r1 / atr_prev
-    sma = d["Close"].shift(1).rolling(20).mean(); sd = d["Close"].shift(1).rolling(20).std(ddof=0)
-    bbw = (4 * sd / sma)
-    d["bbw_pct500"] = bbw.rolling(500).rank(pct=True)
-    v1 = d["Volume"].shift(1)
-    d["vol_ratio"] = v1.rolling(3).mean() / v1.rolling(50).mean()
-    d["drift12"] = (d["Close"].shift(1) - d["Close"].shift(13)).abs() / atr_prev
-    d["atr_pct500"] = atr_prev.rolling(500).rank(pct=True)
-    b = d["burst"].shift(1).fillna(0).astype(int)
-    since = np.zeros(len(d)); cnt = 200
-    bv = b.values
-    for i in range(len(d)):
-        cnt = 0 if bv[i] == 1 else min(200, cnt + 1)
-        since[i] = cnt
-    d["bars_since_burst"] = since
-    return d.dropna(subset=["squeeze6", "squeeze12", "bbw_pct500", "vol_ratio", "drift12", "atr_pct500"])
-
-
-FEATS = ["squeeze6", "squeeze12", "last_bar_range", "bbw_pct500", "vol_ratio", "drift12", "atr_pct500", "bars_since_burst"]
-
-
-def quintile_table(d, feat, target, half):
-    q = pd.qcut(d[feat], 5, labels=False, duplicates="drop")
-    base1 = d.loc[half == 1, target].mean(); base2 = d.loc[half == 2, target].mean()
-    rows = []
-    for b in sorted(q.dropna().unique()):
-        m = q == b
-        lo, hi = d.loc[m, feat].min(), d.loc[m, feat].max()
-        r1 = d.loc[m & (half == 1), target].mean(); r2 = d.loc[m & (half == 2), target].mean()
-        rows.append((int(b), lo, hi, int(m.sum()), 100 * r1, r1 / base1, 100 * r2, r2 / base2))
-    t = pd.DataFrame(rows, columns=["bin", "lo", "hi", "n", "h1_rate%", "h1_lift", "h2_rate%", "h2_lift"])
-    return t.round(3)
-
-
-def main():
-    ap = argparse.ArgumentParser(); ap.add_argument("parquet"); ap.add_argument("bar_minutes", type=int)
-    ap.add_argument("--k", type=float, default=3.0); ap.add_argument("--start"); ap.add_argument("--end")
-    a = ap.parse_args()
-    df = pd.read_parquet(a.parquet)
-    if a.start: df = df[df.index >= pd.Timestamp(a.start, tz="UTC")]
-    if a.end: df = df[df.index < pd.Timestamp(a.end, tz="UTC")]
-    d = build(df, a.bar_minutes, a.k)
-    half = pd.Series(np.where(np.arange(len(d)) < len(d) // 2, 1, 2), index=d.index)
-    print(f"bars {len(d)}  K={a.k}  base rate next_bar {100*d.next_bar.mean():.3f}%  next_hour {100*d.next_hour.mean():.2f}%")
-    print(f"half 1 ends {d.index[len(d)//2 - 1]}")
-    # 1. the clock
-    hr = d.groupby("hour")[["next_bar", "next_hour"]].mean() * 100
-    hr["lift_next_bar"] = hr["next_bar"] / (100 * d.next_bar.mean())
-    print("\n== THE CLOCK (bar-close hour UTC): burst rate % and lift ==")
-    print(hr.round(2).T.to_string())
-    sched = d["hour"].isin([1, 12, 13])
-    print(f"\nscheduled hours (01,12,13) hold {100*d.loc[sched,'next_bar'].sum()/d.next_bar.sum():.1f}% of bursts in {100*sched.mean():.1f}% of bars")
-    # 2. per-feature quintiles, both halves
-    for target in ("next_bar", "next_hour"):
-        print(f"\n== TARGET {target}: quintile burst rate (lift vs base) — ALL hours ==")
-        for f in FEATS:
-            t = quintile_table(d, f, target, half)
-            print(f"-- {f}: lifts h1 {list(t.h1_lift)}  h2 {list(t.h2_lift)}  (bins lo..hi {t.lo.iloc[0]:.2f}..{t.hi.iloc[-1]:.2f})")
-    # 3. same, OUTSIDE the scheduled hours (does anything predict the unscheduled ones?)
-    u = d[~sched]; hu = half[~sched]
-    print(f"\n== next_bar lifts OUTSIDE scheduled hours (base {100*u.next_bar.mean():.3f}%) ==")
-    for f in FEATS:
-        t = quintile_table(u, f, "next_bar", hu)
-        print(f"-- {f}: h1 {list(t.h1_lift)}  h2 {list(t.h2_lift)}")
-    # 4. OOS logistic model: fit half 1, score half 2
-    try:
-        from sklearn.linear_model import LogisticRegression
-        from sklearn.metrics import roc_auc_score
-        X = d[FEATS].copy()
-        for h in range(24): X[f"h{h}"] = (d["hour"] == h).astype(float)
-        Xc = d[FEATS].copy()   # clock-free variant
-        for name, XX in (("features+clock", X), ("features only", Xc), ("clock only", X[[f"h{h}" for h in range(24)]])):
-            mu, sd = XX[half == 1].mean(), XX[half == 1].std().replace(0, 1)
-            Z = (XX - mu) / sd
-            for target in ("next_bar", "next_hour"):
-                m = LogisticRegression(C=0.1, max_iter=2000, class_weight="balanced").fit(Z[half == 1], d.loc[half == 1, target])
-                p = m.predict_proba(Z[half == 2])[:, 1]; y = d.loc[half == 2, target]
-                auc = roc_auc_score(y, p)
-                top = p >= np.quantile(p, 0.95)
-                print(f"OOS {name:15s} {target:9s}: AUC {auc:.3f} | top-5% of scores: burst rate {100*y[top].mean():.2f}% vs base {100*y.mean():.2f}% (lift {y[top].mean()/y.mean():.2f}x), captures {100*y[top].sum()/y.sum():.1f}% of bursts")
-    except ImportError:
-        print("sklearn missing — OOS model skipped")
-    # 5. direction: does the burst continue the prior 12-bar drift?
-    bt = d[d.burst == 1]
-    prior = np.sign(bt["Close"].shift(1) - bt["Close"].shift(13)) if False else np.sign(d["Close"].shift(1) - d["Close"].shift(13))[d.burst == 1]
-    bdir = np.sign(bt["Close"] - bt["Open"])
-    agree = (prior == bdir)[(prior != 0) & (bdir != 0)]
-    print(f"\nburst direction == prior-hour drift direction: {100*agree.mean():.1f}% of {len(agree)} bursts (50% = no information)")
-
-
-if __name__ == "__main__":
-    main()

backtest/research/burst_profile.py

@@ -1,42 +0,0 @@
-"""Burst-candle frequency by UTC hour — the empirical "hedge fund hours".
-
-Usage:
-    python backtest/research/burst_profile.py PARQUET_GLOB
-
-Per UTC bar-close hour of the given bars: % of bars whose range is
->= K x the PRIOR bar's Wilder ATR(14), for K in {3,4,5}; mean bar
-range; and mean signed drift over the 6 bars following a >=4x burst
-(does price continue or snap back?). Analysis tooling only.
-"""
-import glob
-import sys
-
-import pandas as pd
-
-
-def burst_table(parquet_glob: str, bar_minutes: int = 5):
-    files = sorted(glob.glob(parquet_glob))
-    if not files:
-        raise SystemExit(f"no parquet matches {parquet_glob}")
-    df = pd.concat([pd.read_parquet(f) for f in files]).sort_index()
-    df = df[~df.index.duplicated(keep="last")]
-    h, l, c = df["High"], df["Low"], df["Close"]
-    tr = pd.concat([h - l, (h - c.shift()).abs(), (l - c.shift()).abs()],
-                   axis=1).max(axis=1)
-    atr = tr.ewm(alpha=1 / 14, min_periods=14, adjust=False).mean()
-    ratio = (h - l) / atr.shift(1)
-    close_hour = (df.index + pd.Timedelta(minutes=bar_minutes)).hour
-    fwd6 = c.shift(-6) - c  # signed move over the 6 bars after this one
-    out = pd.DataFrame({
-        "bars": ratio.groupby(close_hour).size(),
-        **{f"pct_ge_{k}x": (ratio >= k).groupby(close_hour).mean() * 100
-           for k in (3, 4, 5)},
-        "mean_range": (h - l).groupby(close_hour).mean(),
-        "post4x_drift6": fwd6[ratio >= 4].groupby(close_hour[ratio >= 4]).mean(),
-    })
-    return out.round(3)
-
-
-if __name__ == "__main__":
-    print(burst_table(sys.argv[1],
-                      int(sys.argv[2]) if len(sys.argv) > 2 else 5).to_string())

backtest/research/calendar_impact.py

@@ -1,111 +0,0 @@
-"""Do scheduled economic events move gold violently, and do they hurt OPEN trades?  Analysis tooling only.
-
-Needs backtest/cache/calendar.parquet (ff_calendar_scraper.py). Uses USD events by impact.
-  python backtest/research/calendar_impact.py PARQUET_BARS BAR_MINUTES BOOK.csv ARM DIST [--start --end]
-"""
-import argparse, sys
-from pathlib import Path
-import numpy as np, pandas as pd
-sys.path.insert(0, str(Path(__file__).resolve().parent))
-from open_trade_bursts import wilder_atr, simulate, r_of, stats, K, COST
-
-CAL = Path(__file__).resolve().parent.parent / "cache" / "calendar.parquet"
-
-
-def main():
-    ap = argparse.ArgumentParser(); ap.add_argument("parquet"); ap.add_argument("bar_minutes", type=int)
-    ap.add_argument("book"); ap.add_argument("arm", type=float); ap.add_argument("dist", type=float)
-    ap.add_argument("--start", default="2023-09-05"); ap.add_argument("--end", default="2026-09-05")
-    a = ap.parse_args(); bm = a.bar_minutes
-    cal = pd.read_parquet(CAL)
-    cal["ts"] = pd.to_datetime(cal["dateline"], unit="s", utc=True)
-    cal = cal[(cal.ts >= pd.Timestamp(a.start, tz="UTC")) & (cal.ts < pd.Timestamp(a.end, tz="UTC"))]
-    usd = cal[cal.currency == "USD"].copy()
-    usd["impact"] = usd["impactName"].str.lower()
-    print(f"calendar: {len(cal)} events in window, USD {len(usd)}: " + ", ".join(f"{k}={v}" for k, v in usd.impact.value_counts().items()))
-    px = pd.read_parquet(a.parquet); px = px[(px.index >= pd.Timestamp(a.start, tz="UTC")) & (px.index < pd.Timestamp(a.end, tz="UTC"))]
-    atr_prev = wilder_atr(px).shift(1)
-    rng = px["High"] - px["Low"]; burst = (rng >= K * atr_prev).fillna(False)
-    close = px["Close"]; idx = px.index
-    H1 = 60 // bm
-    move60 = (close.shift(-H1) - close).abs() / atr_prev      # |move| over the next hour from bar OPEN-ish (label)
-    move15 = (close.shift(-max(1, 15 // bm)) - close).abs() / atr_prev
-    burst60 = burst[::-1].rolling(H1, min_periods=1).max()[::-1].astype(bool)
-    # event -> bar containing it (label <= ts < label + bm)
-    def bar_pos(ts):
-        p = idx.searchsorted(ts, side="right") - 1
-        return p if 0 <= p < len(idx) and (ts - idx[p]) < pd.Timedelta(minutes=bm) else -1
-    usd["pos"] = [bar_pos(t) for t in usd.ts]
-    usd = usd[usd.pos >= 0]
-    usd["bar_range_atr"] = rng.values[usd.pos] / atr_prev.values[usd.pos]
-    usd["burst_in_60m"] = burst60.values[usd.pos]
-    usd["move60_atr"] = move60.values[usd.pos]; usd["move60_usd"] = (close.shift(-H1) - close).abs().values[usd.pos]
-    usd["move15_atr"] = move15.values[usd.pos]
-    # baseline: all bars, and bars in the same close-hours without any USD high/medium event within the hour
-    chour = (idx + pd.Timedelta(minutes=bm)).hour
-    ev_bars = set()
-    for p in usd[usd.impact.isin(["high", "medium"])].pos: ev_bars.update(range(max(0, p - H1), min(len(idx), p + H1)))
-    quiet = ~np.isin(np.arange(len(idx)), list(ev_bars))
-    def base_row(mask, label):
-        return dict(group=label, n=int(mask.sum()), bar_range_atr=float((rng / atr_prev)[mask].mean()), burst_in_60m_pct=100 * float(burst60[mask].mean()),
-                    move15_atr=float(move15[mask].mean()), move60_atr=float(move60[mask].mean()), move60_usd=float((close.shift(-H1) - close).abs()[mask].mean()))
-    rows = [base_row(np.ones(len(idx), bool), "ALL bars"), base_row(quiet & np.isin(chour, [12, 13, 14]), "12-14 UTC bars with NO USD event")]
-    for imp in ("high", "medium", "low"):
-        g = usd[usd.impact == imp]
-        rows.append(dict(group=f"USD {imp}-impact events", n=len(g), bar_range_atr=g.bar_range_atr.mean(), burst_in_60m_pct=100 * g.burst_in_60m.mean(),
-                         move15_atr=g.move15_atr.mean(), move60_atr=g.move60_atr.mean(), move60_usd=g.move60_usd.mean()))
-    print("\n== HOW MUCH GOLD MOVES AROUND USD EVENTS (event bar = the bar containing the release) ==")
-    print(pd.DataFrame(rows).round(2).to_string(index=False))
-    hi = usd[usd.impact == "high"]
-    top = hi.groupby("name").agg(n=("ts", "size"), bar_range_atr=("bar_range_atr", "mean"), burst60_pct=("burst_in_60m", lambda s: 100 * s.mean()),
-                                 move60_usd=("move60_usd", "mean"), move60_atr=("move60_atr", "mean")).sort_values("n", ascending=False)
-    print("\n== HIGH-IMPACT USD EVENTS BY NAME (n >= 6) ==")
-    print(top[top.n >= 6].round(2).to_string())
-    # what share of ALL bursts sit within ±30 min of a high/medium USD event?
-    near = np.zeros(len(idx), bool)
-    for p in usd[usd.impact.isin(["high", "medium"])].pos: near[max(0, p - 30 // bm):min(len(idx), p + 30 // bm + 1)] = True
-    nearh = np.zeros(len(idx), bool)
-    for p in hi.pos: nearh[max(0, p - 30 // bm):min(len(idx), p + 30 // bm + 1)] = True
-    print(f"\nshare of ALL violent bars within ±30 min of a USD high-impact event: {100*burst[nearh].sum()/burst.sum():.1f}% (those bars are {100*nearh.mean():.1f}% of time)")
-    print(f"share within ±30 min of a USD high- or medium-impact event: {100*burst[near].sum()/burst.sum():.1f}% (bars {100*near.mean():.1f}% of time)")
-    # open trades through high-impact events
-    book = pd.read_csv(a.book); book["entry_time"] = pd.to_datetime(book.entry_time, utc=True, format="mixed")
-    bars = dict(O=px.Open.values, H=px.High.values, L=px.Low.values, C=px.Close.values, burst=burst.values, chour=chour.values)
-    pos = idx.get_indexer(book.entry_time); ok = pos >= 0; book = book[ok].reset_index(drop=True); pos = pos[ok]
-    ev_pos = np.sort(hi.pos.values)
-    # flat mask: exit at the open of the bar that CONTAINS a high-impact event... use the bar before (exit ~5-15 min before)
-    flat_mask = np.zeros(len(idx), bool); tight_mask = np.zeros(len(idx), bool)
-    for p in ev_pos:
-        if p - 1 >= 0: flat_mask[p - 1] = True
-        tight_mask[max(0, p - 1):min(len(idx), p + H1)] = True
-    res = {}
-    for v, kw in (("base", {}), ("mask_flat", dict(flat_mask=flat_mask)), ("mask_tight", dict(tight_mask=tight_mask))):
-        rs, fl, thru = [], [], []
-        for i, t in book.iterrows():
-            xp, held, reason, flag = simulate(bars, pos[i], t.direction, t.entry, t.sl, a.arm, a.dist, v, **kw)
-            rs.append(r_of(t.direction, t.entry, t.sl, xp)); fl.append(flag)
-            if v == "base":
-                j0, j1 = pos[i], pos[i] + held
-                k = ev_pos[(ev_pos > j0) & (ev_pos <= j1)]
-                thru.append(k[0] if len(k) else -1)
-        res[v] = (np.array(rs), np.array(fl))
-        if v == "base": book["ev_pos"] = thru
-    base = res["base"][0]; half = np.where(np.arange(len(book)) < len(book) // 2, 1, 2)
-    hit = book.ev_pos.values >= 0
-    print(f"\n== OPEN TRADES THAT HELD THROUGH A HIGH-IMPACT USD EVENT ({a.book.split('/')[-1]}) ==")
-    print(f"trades through an event: {hit.sum()} of {len(book)} ({100*hit.mean():.1f}%) | mean R {base[hit].mean():+.3f} vs {base[~hit].mean():+.3f} for the rest | WR {100*(base[hit]>0).mean():.0f}% vs {100*(base[~hit]>0).mean():.0f}%")
-    if hit.sum():
-        d = np.where(book.direction == "BUY", 1.0, -1.0)
-        m15 = np.array([ (close.shift(-max(1,15//bm)).values[p] - close.values[p]) if p >= 0 else np.nan for p in book.ev_pos])
-        with_ = hit & (d * m15 > 0); against = hit & (d * m15 < 0)
-        print(f"  event's first 15-min move WITH the trade: n={with_.sum()} mean R {base[with_].mean():+.3f} | AGAINST: n={against.sum()} mean R {base[against].mean():+.3f} (WR {100*(base[against]>0).mean():.0f}%)")
-    print("\n== PROTECTION AROUND HIGH-IMPACT USD EVENTS ==")
-    b = stats(base)
-    for v, (r, f) in res.items():
-        s = stats(r); aff = f == 1
-        d1 = r[half == 1].sum() - base[half == 1].sum(); d2 = r[half == 2].sum() - base[half == 2].sum()
-        print(f"{v:11s} total {s['total']:+8.2f}  Δ {s['total']-b['total']:+7.2f}  PF {s['pf']:.2f}  DD {s['dd']:.2f}  affected {aff.sum():4d}  (affected: base {base[aff].sum():+.2f} -> {r[aff].sum():+.2f}) | Δ h1 {d1:+.2f} / h2 {d2:+.2f}")
-
-
-if __name__ == "__main__":
-    main()

backtest/research/commit_ladder.py

@@ -1,178 +0,0 @@
-"""Commit-ladder campaign driver (2026-08).
-
-Question: of all signal-code states shipped in the last two months, which
-configuration is the most accurate? Historical in-era backtest numbers are
-not comparable (the backtester's realism model changed on 07-08/07-16 and
-the data feed switched to Oanda on 07-28), so every historical
-configuration is recreated on TODAY's harness via --set overrides and
-replayed on the same pinned Oanda bars. Bug-fix commits stay on; the
-ladder compares shipped tuning decisions only.
-
-Usage:
-    python backtest/research/commit_ladder.py            # full matrix
-    python backtest/research/commit_ladder.py --tf 5min  # one timeframe
-    python backtest/research/commit_ladder.py --windows 12mo,oos
-
-Resumable: a run whose CSV already exists in the output dir is skipped.
-Outputs: backtest/research/commit_ladder_2026-08/<variant>_<window>.csv
-         + .log per run, summary.csv, and a printed ranking table.
-"""
-from __future__ import annotations
-
-import argparse
-import subprocess
-import sys
-from pathlib import Path
-
-import pandas as pd
-
-ROOT = Path(__file__).resolve().parents[2]
-BT = ROOT / "backtest" / "gpu_backtest.py"
-OUT = ROOT / "backtest" / "research" / "commit_ladder_2026-08"
-
-WINDOWS = {
-    "12mo": ("2025-07-30", "2026-07-30"),
-    "oos": ("2024-06-01", "2025-03-30"),
-    "5yr": ("2021-07-30", "2026-07-30"),
-}
-SPLIT_12MO = pd.Timestamp("2026-01-29", tz="UTC")  # book_diff.py's split
-
-# (variant, "commit date what-it-represents", [--set overrides])
-VARIANTS: dict[str, list[tuple[str, str, list[str]]]] = {
-    "5min": [
-        ("M5-A", "dad2432 07-09 gates+trailing+pullback",
-         ["xau_strategy.blocked_hours_utc=[0,1,2,10,11,13,16,21]",
-          "xau_strategy.cp_enabled=false",
-          "signals.trail_dist_r=0.75",
-          "xau_strategy.m5_min_abs_score=30",
-          "xau_strategy.m5_min_confirmation_score=10"]),
-        ("M5-B", "f3ba4d5 07-11 +hour-9 block",
-         ["xau_strategy.blocked_hours_utc=[0,1,2,10,11,13,16,21,9]",
-          "xau_strategy.cp_enabled=false",
-          "signals.trail_dist_r=0.75",
-          "xau_strategy.m5_min_abs_score=30",
-          "xau_strategy.m5_min_confirmation_score=10"]),
-        ("M5-C", "de6424e 07-16 +chart patterns",
-         ["xau_strategy.blocked_hours_utc=[0,1,2,10,11,13,16,21,9]",
-          "signals.trail_dist_r=0.75",
-          "xau_strategy.m5_min_abs_score=30",
-          "xau_strategy.m5_min_confirmation_score=10"]),
-        ("M5-D", "3ee79b5 07-21 +trail dist 0.85",
-         ["xau_strategy.blocked_hours_utc=[0,1,2,10,11,13,16,21,9]",
-          "xau_strategy.m5_min_abs_score=30",
-          "xau_strategy.m5_min_confirmation_score=10"]),
-        ("M5-E", "0396970 07-30 relaxation (HEAD)", []),
-    ],
-    "15min": [
-        ("M15-A", "987ec5b 07-09 pre-campaigns",
-         ["signals.tp_floor_rr=0.5",
-          "xau_strategy.blocked_regimes=[]",
-          "xau_strategy.sell_blocked_hours_utc=[]",
-          "signals.trail_arm_r=0.60"]),
-        ("M15-B", "f3ba4d5/f88e39f 07-11 1R floor + regime block",
-         ["xau_strategy.sell_blocked_hours_utc=[]",
-          "signals.trail_arm_r=0.60"]),
-        ("M15-C", "7d3fd95 07-16 +SELL hour 15/17 block",
-         ["signals.trail_arm_r=0.60"]),
-        ("M15-D", "bf8e5fb 07-21 trail arm 0.70 (HEAD)", []),
-    ],
-    "1h": [
-        ("H1-A", "pre-d95d31a 07-08 TP-tightening model",
-         ["signals.min_rr_ratio=0.4",
-          "signals.tp_tightening_factor=0.8"]),
-        ("H1-B", "HEAD", []),
-    ],
-}
-
-
-def run_one(tf: str, name: str, overrides: list[str], wkey: str,
-            spread: str | None = None) -> Path:
-    suffix = f"_s{spread}" if spread else ""
-    csv_path = OUT / f"{name}_{wkey}{suffix}.csv"
-    log_path = OUT / f"{name}_{wkey}{suffix}.log"
-    if csv_path.exists():
-        print(f"  [skip] {csv_path.name} exists")
-        return csv_path
-    start, end = WINDOWS[wkey]
-    cmd = [sys.executable, str(BT), "--pair", "XAUUSD",
-           "--primary-tf", tf, "--start", start, "--end", end,
-           "--export-csv", str(csv_path)]
-    if spread:
-        cmd += ["--spread", spread]
-    for ov in overrides:
-        cmd += ["--set", ov]
-    print(f"  [run ] {csv_path.name} ...", flush=True)
-    with open(log_path, "w") as lf:
-        rc = subprocess.run(cmd, stdout=lf, stderr=subprocess.STDOUT,
-                            cwd=ROOT).returncode
-    if rc != 0:
-        raise SystemExit(f"backtest failed (rc={rc}) — see {log_path}")
-    log = log_path.read_text()
-    n_ov = log.count("--set override")
-    if n_ov != len(overrides):
-        raise SystemExit(
-            f"{name}_{wkey}: expected {len(overrides)} --set overrides, "
-            f"log shows {n_ov} — see {log_path}")
-    if "Loading cached" not in log and wkey != "sanity":
-        print(f"  [warn] {log_path.name}: no 'Loading cached' line — "
-              f"data was fetched live for at least one timeframe")
-    return csv_path
-
-
-def metrics(csv_path: Path, wkey: str) -> dict:
-    df = pd.read_csv(csv_path, parse_dates=["entry_time"])
-    if df.empty:
-        return {"trades": 0, "total_r": 0.0, "wr": 0.0, "pf": 0.0,
-                "max_dd": 0.0, "h1_r": 0.0, "h2_r": 0.0, "years_pos": ""}
-    if df["entry_time"].dt.tz is None:
-        df["entry_time"] = df["entry_time"].dt.tz_localize("UTC")
-    r = df["r_multiple"]
-    pos, neg = r[r > 0].sum(), r[r < 0].sum()
-    cum = r.cumsum()
-    dd = (pd.concat([pd.Series([0.0]), cum]).cummax()[1:] - cum).max()
-    out = {
-        "trades": len(df),
-        "total_r": round(r.sum(), 1),
-        "wr": round((r > 0).mean() * 100, 1),
-        "pf": round(pos / abs(neg), 2) if neg else float("inf"),
-        "max_dd": round(dd, 1),
-        "h1_r": "", "h2_r": "", "years_pos": "",
-    }
-    if wkey == "12mo":
-        h1 = r[df["entry_time"] < SPLIT_12MO].sum()
-        out["h1_r"] = round(h1, 1)
-        out["h2_r"] = round(r.sum() - h1, 1)
-    if wkey == "5yr":
-        by_year = r.groupby(df["entry_time"].dt.year).sum()
-        out["years_pos"] = f"{(by_year > 0).sum()}/{len(by_year)}"
-    return out
-
-
-def main() -> None:
-    ap = argparse.ArgumentParser()
-    ap.add_argument("--tf", choices=list(VARIANTS), default=None)
-    ap.add_argument("--windows", default="12mo,oos,5yr")
-    args = ap.parse_args()
-    OUT.mkdir(parents=True, exist_ok=True)
-    wkeys = [w.strip() for w in args.windows.split(",")]
-    tfs = [args.tf] if args.tf else list(VARIANTS)
-
-    rows = []
-    for tf in tfs:
-        for name, label, overrides in VARIANTS[tf]:
-            for wkey in wkeys:
-                csv_path = run_one(tf, name, overrides, wkey)
-                m = metrics(csv_path, wkey)
-                rows.append({"tf": tf, "variant": name, "represents": label,
-                             "window": wkey, **m})
-                print(f"    {name} {wkey}: {m['trades']} trades, "
-                      f"{m['total_r']}R, PF {m['pf']}, WR {m['wr']}%")
-
-    summary = pd.DataFrame(rows)
-    summary.to_csv(OUT / "summary.csv", index=False)
-    print("\n=== SUMMARY (also in summary.csv) ===")
-    print(summary.to_string(index=False))
-
-
-if __name__ == "__main__":
-    main()

backtest/research/duka_spread.py

@@ -1,194 +0,0 @@
-"""Dukascopy XAU/USD tick download + bid/ask spread study.
-
-Validates ForexBot2's $0.30 round-trip cost assumption (config:
-SignalConfig.round_trip_cost / PAIR_INFO default_spread). Round-trip
-cost of a market entry+exit = one ask-bid spread width (buy at ask,
-sell at bid), so the assumption holds iff typical spread ~ $0.30.
-
-bi5 format: LZMA 'alone' stream of 20-byte big-endian records
-(ms_in_hour uint32, ask uint32, bid uint32, ask_vol f32, bid_vol f32);
-XAUUSD prices scaled x1000. URL months are 0-indexed.
-"""
-import lzma
-import struct
-import sys
-from concurrent.futures import ThreadPoolExecutor
-from datetime import date, timedelta
-from pathlib import Path
-
-import numpy as np
-import pandas as pd
-import requests
-
-SCALE = 1000.0
-CACHE = Path(__file__).parent / "tick_cache"
-CACHE.mkdir(exist_ok=True)
-URL = "https://datafeed.dukascopy.com/datafeed/XAUUSD/{y}/{m:02d}/{d:02d}/{h:02d}h_ticks.bi5"
-
-# Bot's tradable bar-close hours: session 03:00-23:00 minus XAU blocked
-# hours [0,1,2,10,11,13,16,21] (M5 additionally blocks 9 — reported
-# separately below).
-BLOCKED = {0, 1, 2, 10, 11, 13, 16, 21}
-TRADABLE = sorted(set(range(3, 23)) - BLOCKED)
-
-# ── Day sample ────────────────────────────────────────────────────────
-recent = [date(2026, 7, 6) + timedelta(days=i) for i in range(17)]
-recent = [d for d in recent if d.weekday() < 5]              # Mon-Fri Jul 6-22
-monthly = [date(2025, 8, 13), date(2025, 9, 10), date(2025, 10, 15),
-           date(2025, 11, 12), date(2025, 12, 10), date(2026, 1, 14),
-           date(2026, 2, 11), date(2026, 3, 11), date(2026, 4, 15),
-           date(2026, 5, 13), date(2026, 6, 10)]             # mid-month Wednesdays
-nfp = [date(2026, 6, 5), date(2026, 7, 3)]                   # NFP Fridays
-sundays = [date(2026, 7, 12), date(2026, 7, 19)]             # Sun 22-24 open
-DAYS = sorted(set(recent + monthly + nfp + sundays))
-assert all(d.weekday() < 5 for d in recent + monthly + nfp)
-
-
-_tls = __import__("threading").local()
-
-
-def _session() -> requests.Session:
-    if not hasattr(_tls, "s"):
-        _tls.s = requests.Session()
-        _tls.s.headers["User-Agent"] = "Mozilla/5.0 (research; spread-study)"
-    return _tls.s
-
-
-def fetch(day: date, hour: int) -> Path:
-    p = CACHE / f"{day.isoformat()}_{hour:02d}.bi5"
-    if p.exists():
-        return p
-    u = URL.format(y=day.year, m=day.month - 1, d=day.day, h=hour)
-    last = None
-    for attempt in range(4):
-        try:
-            r = _session().get(u, timeout=30)
-            if r.status_code != 200:
-                raise RuntimeError(f"{u} -> HTTP {r.status_code}")
-            p.write_bytes(r.content)
-            return p
-        except (requests.ConnectionError, requests.Timeout, RuntimeError) as exc:
-            last = exc
-            __import__("time").sleep(1.5 * (attempt + 1))
-            if hasattr(_tls, "s"):
-                _tls.s.close()
-                del _tls.s
-    raise RuntimeError(f"giving up on {u}: {last}")
-
-
-def decode(p: Path, day: date, hour: int) -> pd.DataFrame:
-    blob = p.read_bytes()
-    if not blob:
-        return pd.DataFrame()
-    raw = lzma.decompress(blob)
-    n = len(raw) // 20
-    a = np.frombuffer(raw[: n * 20], dtype=">u4").reshape(n, 5)
-    f = np.frombuffer(raw[: n * 20], dtype=">f4").reshape(n, 5)
-    df = pd.DataFrame({
-        "ms": a[:, 0].astype(np.int64),
-        "ask": a[:, 1] / SCALE,
-        "bid": a[:, 2] / SCALE,
-    })
-    df["spread"] = df["ask"] - df["bid"]
-    df["date"], df["hour"] = day.isoformat(), hour
-    df["weekday"] = day.weekday()
-    return df
-
-
-def load_all() -> pd.DataFrame:
-    jobs = [(d, h) for d in DAYS for h in range(24)]
-    with ThreadPoolExecutor(max_workers=3) as ex:
-        paths = list(ex.map(lambda j: fetch(*j), jobs))
-    frames = [decode(p, d, h) for p, (d, h) in zip(paths, jobs)]
-    df = pd.concat([f for f in frames if not f.empty], ignore_index=True)
-    return df
-
-
-def wstats(g: pd.DataFrame) -> pd.Series:
-    """Tick stats + time-weighted mean (each tick's spread held until the
-    next tick; last tick held to hour end)."""
-    tw_num = tw_den = 0.0
-    keys = [k for k in ("date", "hour") if k in g.columns]
-    for _, grp in g.groupby(keys, sort=False):
-        ms = grp["ms"].to_numpy()
-        sp = grp["spread"].to_numpy()
-        w = np.diff(np.append(ms, 3_600_000)).clip(min=0)
-        tw_num += float((sp * w).sum())
-        tw_den += float(w.sum())
-    s = g["spread"]
-    return pd.Series({
-        "ticks": len(s),
-        "tw_mean": tw_num / tw_den if tw_den else np.nan,
-        "mean": s.mean(), "median": s.median(),
-        "p75": s.quantile(0.75), "p90": s.quantile(0.90),
-        "p99": s.quantile(0.99), "max": s.max(),
-        "pct_le_030": (s <= 0.30).mean() * 100,
-        "pct_ge_060": (s >= 0.60).mean() * 100,
-    })
-
-
-def m5_boundary_spreads(df: pd.DataFrame) -> pd.DataFrame:
-    """Spread of the FIRST tick at/after each M5 boundary — what a
-    next-open fill actually pays."""
-    rows = []
-    for (day, hour), grp in df.groupby(["date", "hour"], sort=False):
-        ms = grp["ms"].to_numpy()
-        sp = grp["spread"].to_numpy()
-        wd = grp["weekday"].iat[0]
-        for b in range(0, 3_600_000, 300_000):
-            i = np.searchsorted(ms, b)
-            if i < len(ms) and ms[i] - b <= 30_000:   # tick within 30 s
-                rows.append((day, wd, hour, b // 60_000, sp[i]))
-    return pd.DataFrame(rows, columns=["date", "weekday", "hour", "minute", "spread"])
-
-
-def main() -> None:
-    df = load_all()
-    print(f"days={df['date'].nunique()}  ticks={len(df):,}  "
-          f"price range {df['bid'].min():.0f}-{df['ask'].max():.0f}")
-
-    print("\n── Overall (all sampled hours) ──")
-    print(wstats(df).round(3).to_string())
-
-    trad = df[df["hour"].isin(TRADABLE) & (df["weekday"] < 5)]
-    print("\n── Bot-tradable hours (session 03-23 minus blocked, Mon-Fri) ──")
-    print(wstats(trad).round(3).to_string())
-
-    roll = df[df["hour"].isin([21, 22, 23])]
-    print("\n── Rollover window 21-24 UTC ──")
-    print(wstats(roll).round(3).to_string())
-
-    print("\n── Per UTC hour ──")
-    per_hour = df[df["weekday"] < 5].groupby("hour").apply(wstats, include_groups=False).round(3)
-    per_hour["tradable"] = ["Y" if h in TRADABLE else "-" for h in per_hour.index]
-    print(per_hour.to_string())
-    per_hour.to_csv(Path(__file__).parent / "duka_spread_by_hour.csv")
-
-    print("\n── M5 next-open boundary fills ──")
-    mb = m5_boundary_spreads(df)
-    mb_trad = mb[mb["hour"].isin(TRADABLE) & (mb["weekday"] < 5)]
-    for name, d in [("all hours", mb), ("tradable hours", mb_trad)]:
-        s = d["spread"]
-        print(f"{name}: n={len(s):,} mean={s.mean():.3f} median={s.median():.3f} "
-              f"p90={s.quantile(0.9):.3f} p99={s.quantile(0.99):.3f} "
-              f"%<=0.30={(s <= 0.30).mean()*100:.1f} %>=0.60={(s >= 0.60).mean()*100:.1f}")
-    mb.to_csv(Path(__file__).parent / "duka_m5_boundary_spreads.csv", index=False)
-
-    print("\n── By month (tradable hours, tick median / tw_mean) ──")
-    trad2 = trad.copy()
-    trad2["month"] = trad2["date"].str[:7]
-    bym = trad2.groupby("month").apply(wstats, include_groups=False).round(3)
-    print(bym[["ticks", "tw_mean", "median", "p90", "pct_le_030", "pct_ge_060"]].to_string())
-
-    print("\n── NFP Fridays 12:00-15:00 UTC vs same hours other days ──")
-    nfp_days = {"2026-06-05", "2026-07-03"}
-    win = df[df["hour"].isin([12, 13, 14])]
-    for name, d in [("NFP", win[win["date"].isin(nfp_days)]),
-                    ("non-NFP", win[~win["date"].isin(nfp_days) & (win["weekday"] < 5)])]:
-        s = d["spread"]
-        print(f"{name}: n={len(s):,} tick-median={s.median():.3f} p90={s.quantile(0.9):.3f} "
-              f"p99={s.quantile(0.99):.3f} max={s.max():.2f}")
-
-
-if __name__ == "__main__":
-    sys.exit(main())

backtest/research/duka_spread_by_hour.csv

@@ -1,25 +0,0 @@
-hour,ticks,tw_mean,mean,median,p75,p90,p99,max,pct_le_030,pct_ge_060,tradable
-0,362893.0,0.749,0.768,0.79,0.82,0.86,1.1,7.69,0.053,94.561,-
-1,540182.0,0.735,0.742,0.74,0.81,0.87,1.0,5.59,0.073,90.449,-
-2,431589.0,0.716,0.722,0.73,0.8,0.86,0.95,2.96,0.1,85.27,-
-3,343952.0,0.708,0.711,0.71,0.79,0.84,0.95,1.67,0.081,82.939,Y
-4,246262.0,0.704,0.702,0.71,0.78,0.82,0.94,1.86,0.098,82.627,Y
-5,388365.0,0.712,0.716,0.72,0.79,0.82,0.95,2.17,0.076,86.745,Y
-6,368999.0,0.65,0.651,0.65,0.69,0.74,0.94,2.32,0.079,71.668,Y
-7,313149.0,0.601,0.598,0.59,0.65,0.69,0.994,3.08,0.239,42.904,Y
-8,347276.0,0.612,0.616,0.6,0.66,0.74,1.0,3.04,0.308,48.349,Y
-9,321619.0,0.612,0.615,0.6,0.66,0.72,1.0,5.95,0.288,50.644,Y
-10,295271.0,0.611,0.618,0.6,0.66,0.71,1.0,9.36,0.164,49.253,-
-11,336963.0,0.619,0.624,0.6,0.67,0.79,1.0,4.97,0.203,50.178,-
-12,476773.0,0.66,0.697,0.63,0.72,0.877,1.367,14.98,0.257,59.633,Y
-13,651425.0,0.641,0.648,0.63,0.69,0.787,1.487,11.407,0.296,60.297,-
-14,673391.0,0.625,0.625,0.63,0.68,0.77,0.9,11.98,0.458,59.085,Y
-15,598398.0,0.636,0.638,0.64,0.7,0.78,0.91,4.49,0.433,66.296,Y
-16,485023.0,0.648,0.644,0.64,0.7,0.757,0.917,4.09,0.118,68.496,-
-17,450119.0,0.626,0.628,0.627,0.7,0.726,0.84,4.08,0.154,63.337,Y
-18,411538.0,0.626,0.63,0.637,0.7,0.71,0.79,4.16,0.107,67.734,Y
-19,435396.0,0.638,0.643,0.65,0.7,0.72,0.896,5.804,0.11,72.007,Y
-20,212654.0,0.684,0.665,0.66,0.7,0.76,1.217,7.15,0.226,70.215,Y
-21,31319.0,0.627,0.622,0.61,0.68,0.75,1.02,2.08,0.089,53.153,-
-22,108879.0,0.829,0.834,0.797,0.84,0.98,2.05,10.11,0.056,95.434,Y
-23,161079.0,1.032,0.913,0.8,0.89,1.307,2.77,13.174,0.092,95.41,-

backtest/research/feed_divergence_2026-08-14.md

@@ -0,0 +1,81 @@
+# Feed Divergence Study — Oanda vs Tiingo vs Twelve Data (Phase 1)
+
+**Date:** 2026-08-14
+**Test plan:** `docs/data_source_change_test_plan.md` (removed 2026-09-06, in git history; the pending Phase 4 shadow scan is noted in CLAUDE.md)
+**Tool:** `scripts/feed_divergence_study.py` (script removed 2026-09-06, in git history) (fetch_range over 365d, parquet-cached)
+**Window:** 2025-08-14 → 2026-08-14 (12 months); Oanda-vs-TD truncated at TD cache head (2026-07-23)
+
+## Headline
+
+Three mid-price feeds for the *same* XAU/USD instrument are **not** interchangeable:
+
+| Pair | mean \|ΔClose\| | p95 \|ΔClose\| | direction agree (M5/M15/H1) | bias | extra bars |
+|---|---|---|---|---|---|
+| Oanda vs **Tiingo** | ~$0.13 | ~$0.29 | 97.7% / 98.3% / 98.5% | ~0 | Tiingo +3–5% (weekend) |
+| Oanda vs **Twelve Data** | ~$3.0 | ~$11.9 | 81.0% / 88.7% / 92.7% | −$1.06 (TD high) | — |
+
+- **Oanda ↔ Tiingo are near-identical** on shared bars (~1–2% of ATR on close), but Tiingo emits ~22 hours of Sunday bars/week that Oanda omits.
+- **Oanda ↔ Twelve Data diverge massively** — ~$3 mean close gap (≈30% of M5 ATR, ~30% of M15 ATR), a systematic ~$1 bias (TD quoted ~$1 high), and 19% of M5 bars even disagree on bar *direction*. This is the documented "Oanda/TD divergence" that motivated re-pinning the books on Oanda bars.
+
+## Oanda vs Tiingo — detailed
+
+Both feeds: same bar grid, same head/tail timestamps. Tiingo simply has more bars (Sunday bars).
+
+### Coverage (12 months)
+
+| TF | Oanda bars | Tiingo bars | shared | Oanda-only | Tiingo-only |
+|---|---|---|---|---|---|
+| M5 | 70,901 | 73,241 | 70,807 (96.6%) | 94 (0.13%) | 2,434 (3.32%) |
+| M15 | 23,640 | 24,922 | 23,615 (94.7%) | 25 (0.11%) | 1,307 (5.24%) |
+| H1 | 5,912 | 6,246 | 5,909 (94.6%) | 3 (0.05%) | 337 (5.40%) |
+| H4 | 1,597 | 1,657 | 1,597 (96.4%) | 0 | 60 (3.62%) |
+
+- **Tiingo-only bars cluster exclusively on Sundays** (e.g. M15: 88/Sunday; H1: 22/Sunday; H4: 5/Sunday). Tiingo's FX endpoint resamples continuously and does **not** apply the FX weekend calendar — it emits ~22h of flat/stale Sunday bars that Oanda omits.
+- Oanda-only bars cluster on holidays (2025-11-27 Thanksgiving, 2025-12-10, 2026-06-25) — a handful of bars, negligible.
+
+### Per-bar OHLC diffs (Oanda − Tiingo, $)
+
+| TF | Close mean | Close p95 | Close max | \|ΔClose\| / ATR14 (mean / p95) |
+|---|---|---|---|---|
+| M5 | $0.123 | $0.285 | $26.10 | 2.1% / 5.0% |
+| M15 | $0.128 | $0.290 | $26.10 | 1.3% / 2.9% |
+| H1 | $0.145 | $0.335 | $26.10 | 0.7% / 1.6% |
+| H4 | $0.168 | $0.340 | $17.53 | 0.4% / 0.8% |
+
+- Bias is ~zero on every TF (no systematic one-feed-high).
+- Tail outliers (p99/max up to $26–88) occur during fast moves — the two feeds' bar extremes disagree on the exact tick, as expected.
+- **Direction agreement is 97.7–98.4%**, return correlation 0.995–0.9997.
+- Divergence is concentrated at the Friday-close / Sunday-reopen boundary (UTC hours 20–22; mean |ΔClose| ~2–4× the 08–19h baseline).
+
+### Phase 1 pass/fail vs plan thresholds
+
+| Criterion | Plan threshold | Result | Verdict |
+|---|---|---|---|
+| timestamp overlap | ≥ 99.5% | 94.6–96.6% | **FAIL** (Tiingo weekend bars) |
+| Close p95 \|Δ\| | ≤ 0.01% of price | ~0.0067% | PASS |
+| no systematic bias | mean Δ ≈ 0 | ~0 | PASS |
+
+The overlap "failure" is a *coverage* difference, not a data-quality one: the shared bars agree tightly, but Tiingo's extra Sunday bars mean the feeds have a different bar *count*. The signal pipeline's own `_is_forex_weekend` filter (NY-anchored) gates those Sunday bars, so live impact depends on whether any code trusts the feed's raw bar set directly.
+
+## Oanda vs Twelve Data — detailed (M5/M15/H1, cached TD parquets)
+
+| TF | common | mean \|ΔClose\| | median \|ΔClose\| | p95 | max | bias (TD−Oanda) | dir agree |
+|---|---|---|---|---|---|---|---|
+| M5 | 66,246 | $2.999 | $1.118 | $11.934 | $101.85 | +1.05 | 80.96% |
+| M15 | 22,090 | $2.951 | $1.051 | $11.848 | $101.85 | +1.08 | 88.71% |
+| H1 | 5,526 | $2.902 | $0.985 | $11.908 | $70.94 | +1.06 | 92.67% |
+
+- The ~$3 mean / ~$1 median / ~$12 p95 gap is **~23× larger** than the Oanda–Tiingo gap, and the ~$1 bias is systematic (Twelve Data quotes gold ~$1 high vs Oanda's true mid).
+- M5 direction agreement of **81%** means one bar in five disagrees on direction — M5 signals on TD bars are effectively a *different instrument* than on Oanda bars.
+- This matches the prior Oanda/TD M15 study of 2026-07-24 (`oanda_td_divergence.py`, removed 2026-09-06 as superseded by this report; mean |ΔClose| ~$3.3) and confirms the standing rule: **pinned Twelve-Data books are not comparable to Oanda live signals without a re-pin.**
+
+## Conclusions for the data-source switch
+
+1. **Oanda → Tiingo (or back) is signal-safe in magnitude** (price diffs ~1–2% of ATR, 98% direction agreement), *provided* the weekend-bar coverage difference is handled — Tiingo's Sunday bars must be gated by the aggregator's NY-anchored weekend filter, not by raw bar count.
+2. **Oanda → Twelve Data is a recalibration, not a formality.** Phase 2/3 must quantify the actual signal/book delta; the $3/$1 bias alone guarantees a large fraction of marginal bars flip.
+3. Re-pinning the books on Oanda bars (Phase 3) remains mandatory before any TD-pinned baseline is treated as a live expectation.
+
+## Next steps
+
+- Phase 2: signal-level replay parity (`analyze_from_dataframes` on both feeds) to turn price-diff stats into flip rates.
+- Phase 3: backtest re-pin on Oanda bars at $0.30 and $0.65.

backtest/research/fetch_oanda_m15.py

@@ -1,158 +0,0 @@
-"""Fetch XAU/USD M15 candles from the Oanda v20 REST API.
-
-Mechanical data-fetching infrastructure only — no analysis here.
-Paginates GET /v3/instruments/XAU_USD/candles from a start cursor
-through "now" and writes the result to a parquet file.
-
-Run from the repo root:
-    venv/bin/python backtest/research/fetch_oanda_m15.py
-"""
-import sys
-import time
-from datetime import datetime, timedelta, timezone
-from pathlib import Path
-
-import pandas as pd
-import requests
-
-sys.path.insert(0, str(Path(__file__).resolve().parents[2]))
-
-from app_config import oanda_api_token, oanda_base_url  # noqa: E402
-
-INSTRUMENT = "XAU_USD"
-GRANULARITY = "M15"
-START = datetime(2025, 7, 1, 0, 0, 0, tzinfo=timezone.utc)
-OUT_PATH = Path(__file__).parent / "oanda_xau_m15_20250701_now.parquet"
-COUNT = 5000
-REQUEST_SLEEP_S = 0.5
-FRESHNESS_WINDOW = timedelta(minutes=30)
-
-
-def _rfc3339(dt: datetime) -> str:
-    return dt.astimezone(timezone.utc).strftime("%Y-%m-%dT%H:%M:%S.%fZ")
-
-
-def fetch_all_candles():
-    token = oanda_api_token()
-    base_url = oanda_base_url()
-    if not token:
-        raise RuntimeError(
-            "No Oanda API token resolved (env OANDA_API_TOKEN / "
-            "config.json oanda.api_token). Aborting.")
-
-    url = f"{base_url}/v3/instruments/{INSTRUMENT}/candles"
-    headers = {
-        "Authorization": f"Bearer {token}",
-        "Accept-Datetime-Format": "RFC3339",
-    }
-
-    cursor = START
-    include_first = True
-    all_rows = []
-    n_requests = 0
-
-    while True:
-        params = {
-            "granularity": GRANULARITY,
-            "price": "MBA",
-            "count": COUNT,
-            "dailyAlignment": 0,
-            "alignmentTimezone": "Etc/UTC",
-            "from": _rfc3339(cursor),
-            "includeFirst": "true" if include_first else "false",
-        }
-
-        resp = requests.get(url, headers=headers, params=params, timeout=30)
-        n_requests += 1
-        if resp.status_code != 200:
-            raise RuntimeError(
-                f"Oanda API request failed: status={resp.status_code}\n"
-                f"URL: {resp.url}\n"
-                f"Response body: {resp.text}")
-
-        payload = resp.json()
-        candles = payload.get("candles", [])
-
-        for c in candles:
-            if not c.get("complete", False):
-                continue
-            mid = c["mid"]
-            bid = c["bid"]
-            ask = c["ask"]
-            all_rows.append({
-                "Datetime": pd.Timestamp(c["time"]).tz_convert("UTC")
-                if pd.Timestamp(c["time"]).tzinfo is not None
-                else pd.Timestamp(c["time"]).tz_localize("UTC"),
-                "Open": float(mid["o"]),
-                "High": float(mid["h"]),
-                "Low": float(mid["l"]),
-                "Close": float(mid["c"]),
-                "Volume": int(c["volume"]),
-                "bid_o": float(bid["o"]),
-                "bid_h": float(bid["h"]),
-                "bid_l": float(bid["l"]),
-                "bid_c": float(bid["c"]),
-                "ask_o": float(ask["o"]),
-                "ask_h": float(ask["h"]),
-                "ask_l": float(ask["l"]),
-                "ask_c": float(ask["c"]),
-            })
-
-        print(f"  request {n_requests}: from={params['from']} "
-              f"got {len(candles)} candles (complete kept so far: "
-              f"{len(all_rows)})", file=sys.stderr)
-
-        if not candles:
-            break
-
-        last_time = pd.Timestamp(candles[-1]["time"])
-        if last_time.tzinfo is None:
-            last_time = last_time.tz_localize("UTC")
-        else:
-            last_time = last_time.tz_convert("UTC")
-
-        now = datetime.now(timezone.utc)
-        # Oanda's "count" includes the "from" boundary bar itself, so with
-        # includeFirst=false every full page comes back as count-1, not
-        # count (verified empirically: count=N, includeFirst=false always
-        # yields N-1 candles when a full page of data exists ahead of the
-        # cursor). Compare against the count actually expected for this
-        # request's includeFirst setting, not the raw requested count.
-        expected_full_page = COUNT if include_first else COUNT - 1
-        got_fewer_than_requested = len(candles) < expected_full_page
-        near_now = (now - last_time.to_pydatetime()) < FRESHNESS_WINDOW
-
-        if got_fewer_than_requested or near_now:
-            break
-
-        cursor = last_time.to_pydatetime()
-        include_first = False
-        time.sleep(REQUEST_SLEEP_S)
-
-    return all_rows, n_requests
-
-
-def main():
-    print(f"Fetching {INSTRUMENT} {GRANULARITY} candles from "
-          f"{_rfc3339(START)} through now...", file=sys.stderr)
-
-    rows, n_requests = fetch_all_candles()
-
-    if not rows:
-        raise RuntimeError("No complete candles were fetched.")
-
-    df = pd.DataFrame(rows)
-    df = df.set_index("Datetime")
-    df = df.sort_index()
-    df = df[~df.index.duplicated(keep="last")]
-
-    df.to_parquet(OUT_PATH)
-
-    print(f"\nSaved {len(df)} rows to {OUT_PATH}", file=sys.stderr)
-    print(f"Requests made: {n_requests}", file=sys.stderr)
-    print(f"First timestamp: {df.index[0]}", file=sys.stderr)
-    print(f"Last timestamp: {df.index[-1]}", file=sys.stderr)
-
-
-if __name__ == "__main__":
-    main()

backtest/research/ff_calendar_scraper.py

@@ -1,144 +0,0 @@
-"""Forex Factory calendar scraper -> backtest/cache/calendar.parquet.
-
-Fetches weekly calendar pages and parses the event JSON embedded in
-`window.calendarComponentStates` (the `days:` array). Event `dateline`
-is a unix epoch, so timestamps carry no timezone ambiguity — verified
-2026-07-11 against known release times (NFP 12:30/13:30 UTC across DST,
-ECB 12:15/13:15) and against EUR/USD M15 volatility: the bar containing
-a high-impact event averages 3.5x normal range, and shifting event
-times by ±1h destroys that alignment.
-
-Usage:
-    python backtest/research/ff_calendar_scraper.py --start 2024-01-01 --end 2026-07-13
-    python backtest/research/ff_calendar_scraper.py            # refresh: last stored week -> today
-
-Merges into the existing parquet on event (id, dateline), so refreshing
-is idempotent and also picks up revised `actual` values for recent weeks.
-Be polite: ~1.6s between requests (a full year is ~52 requests).
-"""
-import argparse
-import json
-import time
-from datetime import date, timedelta
-from pathlib import Path
-from typing import List, Optional
-
-import pandas as pd
-import requests
-
-CACHE_DIR = Path(__file__).resolve().parent.parent / "cache"
-PARQUET = CACHE_DIR / "calendar.parquet"
-
-UA = ("Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 "
-      "(KHTML, like Gecko) Chrome/126.0 Safari/537.36")
-
-FIELDS = ["id", "dateline", "name", "country", "currency", "impactName",
-          "timeLabel", "timeMasked", "actual", "forecast", "previous",
-          "revision", "actualBetterWorse"]
-
-
-def week_url(d: date) -> str:
-    return (f"https://www.forexfactory.com/calendar?week="
-            f"{d.strftime('%b').lower()}{d.day}.{d.year}")
-
-
-def extract_days(html: str) -> Optional[list]:
-    """Balanced-bracket extraction of the `days:` JSON array."""
-    i = html.find("days:")
-    if i < 0:
-        return None
-    j = html.find("[", i)
-    depth, in_str, esc = 0, False, False
-    for k in range(j, len(html)):
-        ch = html[k]
-        if in_str:
-            if esc:
-                esc = False
-            elif ch == "\\":
-                esc = True
-            elif ch == '"':
-                in_str = False
-        else:
-            if ch == '"':
-                in_str = True
-            elif ch == "[":
-                depth += 1
-            elif ch == "]":
-                depth -= 1
-                if depth == 0:
-                    return json.loads(html[j:k + 1])
-    return None
-
-
-def fetch_week(sess: requests.Session, d: date, retries: int = 4) -> Optional[List[dict]]:
-    for attempt in range(retries):
-        try:
-            r = sess.get(week_url(d), timeout=30)
-            if r.status_code == 200:
-                days = extract_days(r.text)
-                if days is not None:
-                    return [{k: e.get(k) for k in FIELDS}
-                            for day in days for e in day.get("events", [])]
-        except requests.RequestException:
-            pass
-        time.sleep(5 * (attempt + 1))
-    return None
-
-
-def scrape(start: date, end: date) -> pd.DataFrame:
-    sess = requests.Session()
-    sess.headers["User-Agent"] = UA
-    rows: List[dict] = []
-    failed: List[date] = []
-    d = start
-    while d <= end:
-        week_rows = fetch_week(sess, d)
-        if week_rows is None:
-            failed.append(d)
-            print(f"FAILED week {d}")
-        else:
-            rows.extend(week_rows)
-        d += timedelta(days=7)
-        time.sleep(1.6)
-    if failed:
-        print(f"WARNING: {len(failed)} weeks failed: {failed}")
-    return pd.DataFrame(rows)
-
-
-def merge_into_store(new: pd.DataFrame, path: Path = PARQUET) -> pd.DataFrame:
-    if path.exists():
-        old = pd.read_parquet(path)
-        combined = pd.concat([old, new])
-    else:
-        combined = new
-    # keep="last": a refresh overwrites stale rows (actuals fill in
-    # after release, revisions land weeks later).
-    combined = (combined.drop_duplicates(subset=["id", "dateline"], keep="last")
-                .sort_values("dateline").reset_index(drop=True))
-    combined.to_parquet(path, index=False)
-    return combined
-
-
-def main() -> None:
-    ap = argparse.ArgumentParser(description=__doc__.splitlines()[0])
-    ap.add_argument("--start", type=date.fromisoformat, default=None)
-    ap.add_argument("--end", type=date.fromisoformat, default=date.today())
-    args = ap.parse_args()
-
-    start = args.start
-    if start is None:
-        if PARQUET.exists():
-            last = pd.read_parquet(PARQUET).dateline.max()
-            start = pd.Timestamp(last, unit="s").date() - timedelta(days=7)
-        else:
-            start = date(2024, 1, 1)
-    print(f"Scraping {start} -> {args.end}")
-    new = scrape(start, args.end)
-    combined = merge_into_store(new)
-    print(f"Store: {len(combined)} events "
-          f"({pd.Timestamp(combined.dateline.min(), unit='s')} -> "
-          f"{pd.Timestamp(combined.dateline.max(), unit='s')}) -> {PARQUET}")
-
-
-if __name__ == "__main__":
-    main()

backtest/research/hour_analysis.py

@@ -1,40 +0,0 @@
-"""Per-hour P&L breakdown of a backtest trade book.
-
-Usage:
-    python backtest/research/hour_analysis.py BOOK.csv BAR_MINUTES
-
-Reports, per UTC hour (both entry hour and DECISION hour = entry_time
-minus one bar, the clock blocked_hours_utc evaluates): trade count,
-total R, mean R, win rate, profit factor. Also a per-(hour, direction)
-table. Analysis tooling only — never imported by the live code.
-"""
-import sys
-
-import pandas as pd
-
-
-def hour_tables(book_csv: str, bar_minutes: int):
-    df = pd.read_csv(book_csv)
-    et = pd.to_datetime(df["entry_time"], utc=True, format="mixed")
-    df["entry_hour"] = et.dt.hour
-    df["decision_hour"] = (et - pd.Timedelta(minutes=bar_minutes)).dt.hour
-
-    def table(group_cols):
-        g = df.groupby(group_cols)["r_multiple"]
-        out = g.agg(count="count", total_r="sum", mean_r="mean",
-                    wr=lambda r: (r > 0).mean() * 100)
-        out["pf"] = g.apply(
-            lambda r: r[r > 0].sum() / max(-r[r <= 0].sum(), 1e-9))
-        return out.round(2)
-
-    return (table("decision_hour"), table(["decision_hour", "direction"]),
-            table("entry_hour"))
-
-
-if __name__ == "__main__":
-    book, mins = sys.argv[1], int(sys.argv[2])
-    by_dec, by_dec_dir, by_ent = hour_tables(book, mins)
-    print(f"\n── {book} · by DECISION hour (bar-close clock) ──")
-    print(by_dec.to_string())
-    print(f"\n── by (decision hour, direction) ──")
-    print(by_dec_dir.to_string())

backtest/research/llm_news_eval.py

@@ -1,697 +0,0 @@
-"""
-DeepSeek news-analysis validation harness — XAU/USD.
-====================================================
-Research-only (no production code changes). Answers: is the DeepSeek
-layer's news analysis *correct*?
-
-Part A (events): replays every high-impact USD release with a genuine
-surprise (2024-01 → 2026-07, from backtest/cache/calendar.parquet)
-through the REAL `NewsEngine._llm_event_direction` — real prompt, real
-payload, real validation — and scores its "gold bullish/bearish" calls
-against gold's actual move (Oanda M15 mids) at +15m/30m/1h/4h/24h.
-Baselines: mechanical surprise-sign doctrine, FF actualBetterWorse,
-always-bullish/bearish (2024-26 is a gold bull regime), coin flip.
-
-Part B (headlines): a curated golden set (llm_news_golden.csv) through
-the REAL `NewsEngine._llm_headline_scores` in production-sized batches,
-vs the keyword scorer it overrides. Includes repeat-batch (determinism)
-and shuffled-order (cross-contamination) probes.
-
-Implementation notes that keep the replay faithful to production:
-  * `_llm_event_direction`/`_llm_headline_scores` resolve the client via
-    `get_llm_client()`, so we seed the module singleton with a research
-    client (raised daily cap: 586 events > production cap 500).
-  * The event cache key omits the forecast, so a repeat `actual` would
-    silently re-serve a direction computed for a different forecast —
-    the cache is cleared before every call.
-  * A "neutral" reply and a hard failure both return None; they are
-    disambiguated by reading the event cache back after the call
-    (absent = failure, "" = neutral, "bullish:/bearish: …" = call).
-  * Raw request/response bytes are captured with a requests response
-    hook — the request-building code path is untouched.
-
-Usage (from repo root):
-  venv/bin/python backtest/research/llm_news_eval.py events --limit 5
-  venv/bin/python backtest/research/llm_news_eval.py events --determinism 20
-  venv/bin/python backtest/research/llm_news_eval.py headlines
-  venv/bin/python backtest/research/llm_news_eval.py report
-
-Artifacts land in backtest/research/llm_news_eval_2026-08/ (JSONL replay
-logs are resume-safe: completed events are never re-paid). `report` is a
-pure offline function of the artifacts.
-"""
-
-import argparse
-import json
-import math
-import random
-import sys
-import time
-from datetime import datetime, timezone
-from pathlib import Path
-
-import numpy as np
-import pandas as pd
-
-REPO_ROOT = Path(__file__).resolve().parents[2]
-sys.path.insert(0, str(REPO_ROOT))
-
-import llm_client
-from app_config import deepseek_api_key, get_app_config
-from config import NewsConfig
-from llm_client import DEFAULT_MODEL, DeepSeekClient
-from news_engine import EconomicEvent, NewsEngine
-
-RESEARCH_DIR = Path(__file__).parent
-ART_DIR = RESEARCH_DIR / "llm_news_eval_2026-08"
-CALENDAR = REPO_ROOT / "backtest" / "cache" / "calendar.parquet"
-BARS = (REPO_ROOT / "backtest" / "cache" /
-        "oanda_XAU_USD_15min_20210531_20260804.parquet")
-GOLDEN = RESEARCH_DIR / "llm_news_golden.csv"
-EVENTS_JSONL = ART_DIR / "events_replay.jsonl"
-HEADLINES_JSONL = ART_DIR / "headlines_replay.jsonl"
-SCORED_CSV = ART_DIR / "events_scored.csv"
-
-HORIZONS = [("15m", 15), ("30m", 30), ("1h", 60), ("4h", 240),
-            ("24h", 1440)]
-NEUTRAL_BAND = 0.15  # |xau| below this = neutral prediction (Part B)
-BATCH_CAP = 25       # production batch cap (news_engine.py:783)
-EXPECTED_CORPUS = 586
-
-
-# ── shared plumbing ──────────────────────────────────────────────────
-
-class _Recorder:
-    """requests response hook capturing the exact wire traffic."""
-
-    def __init__(self):
-        self.last = None
-
-    def hook(self, resp, **kwargs):
-        body = resp.request.body
-        if isinstance(body, bytes):
-            body = body.decode("utf-8", "replace")
-        self.last = {
-            "request_body": body,
-            "http_status": resp.status_code,
-            "response_body": resp.text,
-            "elapsed_s": round(resp.elapsed.total_seconds(), 3),
-        }
-        return resp
-
-    def pop(self):
-        cap, self.last = self.last, None
-        return cap or {}
-
-
-def install_research_client(timeout: float):
-    """Build a research DeepSeekClient (raised cap) and seed the module
-    singleton so the production code paths use it. Returns (client,
-    recorder)."""
-    app = get_app_config()
-    key = deepseek_api_key(app)
-    ds = app.deepseek_settings
-    if not key:
-        sys.exit("No DeepSeek API key (env DEEPSEEK_API_KEY or "
-                 "config.json deepseek.api_key) — aborting before spend.")
-    if ds.get("enabled") is False:
-        sys.exit("deepseek.enabled is false in config.json — aborting.")
-    client = DeepSeekClient(
-        api_key=key,
-        model=str(ds.get("model", DEFAULT_MODEL)),
-        timeout=timeout,
-        daily_cap=10 ** 6,  # research: corpus 586 > production cap 500
-    )
-    rec = _Recorder()
-    client._session.hooks["response"].append(rec.hook)
-    with llm_client._client_lock:
-        llm_client._client = client
-    return client, rec
-
-
-def _s(x):
-    """Parquet str columns carry float NaN — EconomicEvent needs None."""
-    return None if pd.isna(x) else str(x)
-
-
-def load_corpus():
-    """High-impact USD releases with a genuine surprise, through the
-    production `EconomicEvent.surprise` gate. Asserts run before any
-    API spend."""
-    df = pd.read_parquet(CALENDAR)
-    df = df[(df["impactName"] == "high") & (df["currency"] == "USD")]
-    rows = []
-    for r in df.itertuples():
-        ev = EconomicEvent(
-            time=datetime.fromtimestamp(int(r.dateline), tz=timezone.utc),
-            currency="USD", impact="high", event=_s(r.name) or "",
-            forecast=_s(r.forecast), previous=_s(r.previous),
-            actual=_s(r.actual),
-        )
-        if ev.surprise not in ("positive", "negative"):
-            continue
-        rows.append({
-            "ffid": int(r.id), "dateline": int(r.dateline),
-            "utc_time": ev.time.isoformat(), "name": ev.event,
-            "actual": ev.actual, "forecast": ev.forecast,
-            "previous": ev.previous, "surprise": ev.surprise,
-            "better_worse": int(r.actualBetterWorse), "ev": ev,
-        })
-    ids = [d["ffid"] for d in rows]
-    if len(rows) != EXPECTED_CORPUS:
-        sys.exit(f"corpus size {len(rows)} != {EXPECTED_CORPUS} — "
-                 "calendar.parquet changed; re-verify before spending.")
-    if len(set(ids)) != len(ids):
-        sys.exit("FF ids not unique — resume keying would break.")
-    # Pin the actualBetterWorse mapping on two unambiguous NFP prints.
-    nfp = {d["utc_time"][:10]: d["better_worse"] for d in rows
-           if d["name"] == "Non-Farm Employment Change"}
-    if nfp.get("2024-01-05") != 1 or nfp.get("2024-05-03") != 2:
-        sys.exit("actualBetterWorse mapping pins failed (expected "
-                 "1=better-for-USD on 2024-01-05 beat, 2=worse on "
-                 "2024-05-03 miss).")
-    rows.sort(key=lambda d: (d["dateline"], d["ffid"]))
-    return rows
-
-
-def _load_jsonl(path: Path):
-    recs = []
-    if path.exists():
-        with open(path) as f:
-            for line in f:
-                line = line.strip()
-                if line:
-                    recs.append(json.loads(line))
-    return recs
-
-
-# ── Part A: event replay ─────────────────────────────────────────────
-
-def _extract_reason(cap):
-    try:
-        content = json.loads(cap["response_body"])[
-            "choices"][0]["message"]["content"]
-        return json.loads(content).get("reason")
-    except Exception:
-        return None
-
-
-def cmd_events(args):
-    ART_DIR.mkdir(exist_ok=True)
-    corpus = load_corpus()
-    if args.limit:
-        corpus = corpus[:args.limit]
-    engine = NewsEngine(NewsConfig())
-    existing = _load_jsonl(EVENTS_JSONL)
-    done1 = {r["ffid"] for r in existing
-             if r.get("run_pass") == 1 and r.get("llm_direction")}
-    done2 = {r["ffid"] for r in existing
-             if r.get("run_pass") == 2 and r.get("llm_direction")}
-    pending = [d for d in corpus if d["ffid"] not in done1]
-    print(f"corpus {len(corpus)} | already recorded {len(done1)} | "
-          f"pending {len(pending)} | determinism re-calls "
-          f"{args.determinism} | model calls are ~100 tokens each "
-          f"(deepseek-v4-flash, static-prefix cached) → cost ≈ cents")
-    if not pending and not args.determinism:
-        print("nothing to do")
-        return
-    _, rec = install_research_client(args.timeout)
-
-    out = open(EVENTS_JSONL, "a")
-
-    def call_one(d, run_pass):
-        # Cache key omits forecast → cleared per call so 196 colliding
-        # corpus rows can't be served a stale direction.
-        with NewsEngine._cache_lock:
-            NewsEngine._llm_event_cache.clear()
-        ev = d["ev"]
-        engine._llm_event_direction(ev)
-        key = (ev.event, ev.currency, str(ev.actual or ""))
-        with NewsEngine._cache_lock:
-            cached = NewsEngine._llm_event_cache.get(key)
-        cap = rec.pop()
-        if cached is None:
-            direction, text = None, None       # hard failure → retry
-        elif cached == "":
-            direction, text = "neutral", ""
-        else:
-            direction, text = cached.split(":", 1)[0], cached
-        recd = {k: d[k] for k in (
-            "ffid", "dateline", "utc_time", "name", "actual", "forecast",
-            "previous", "surprise", "better_worse")}
-        recd.update({
-            "llm_direction": direction, "llm_text": text,
-            "reason_raw": _extract_reason(cap), "run_pass": run_pass,
-            "ts": datetime.now(timezone.utc).isoformat(), **cap,
-        })
-        out.write(json.dumps(recd) + "\n")
-        out.flush()
-        return direction
-
-    remaining = pending
-    for npass in range(1, args.retry_passes + 1):
-        if not remaining:
-            break
-        failed = []
-        for i, d in enumerate(remaining):
-            if call_one(d, 1) is None:
-                failed.append(d)
-            if args.sleep:
-                time.sleep(args.sleep)
-            if (i + 1) % 25 == 0:
-                print(f"  pass {npass}: {i + 1}/{len(remaining)} "
-                      f"({len(failed)} failed)")
-        print(f"pass {npass} done: {len(remaining) - len(failed)} ok, "
-              f"{len(failed)} failed")
-        remaining = failed
-    if remaining:
-        print(f"WARNING: {len(remaining)} events still unrecorded after "
-              f"{args.retry_passes} passes — re-run `events` to resume.")
-
-    if args.determinism:
-        ok_ids = {x["ffid"] for x in remaining}
-        done_now = [d for d in corpus if d["ffid"] not in ok_ids]
-        done_now.sort(key=lambda d: (d["name"], d["dateline"]))
-        stride = max(1, len(done_now) // args.determinism)
-        sample = done_now[::stride][:args.determinism]
-        todo = [d for d in sample if d["ffid"] not in done2]
-        print(f"determinism probe: {len(todo)} re-calls "
-              f"({len(sample) - len(todo)} already recorded)")
-        for d in todo:
-            call_one(d, 2)
-            if args.sleep:
-                time.sleep(args.sleep)
-    out.close()
-    print("events replay complete →", EVENTS_JSONL)
-
-
-# ── Part B: golden headlines ─────────────────────────────────────────
-
-def _load_golden():
-    g = pd.read_csv(GOLDEN)
-    need = {"headline", "category", "expected_xau_sign", "rationale"}
-    if not need <= set(g.columns):
-        sys.exit(f"golden CSV missing columns {need - set(g.columns)}")
-    if not g["expected_xau_sign"].isin([-1, 0, 1]).all():
-        sys.exit("expected_xau_sign must be -1/0/+1")
-    if not g["headline"].is_unique:
-        sys.exit("duplicate headlines in golden CSV")
-    return g
-
-
-def cmd_headlines(args):
-    ART_DIR.mkdir(exist_ok=True)
-    g = _load_golden()
-    heads = list(g["headline"])
-    _, rec = install_research_client(args.timeout)
-    engine = NewsEngine(NewsConfig())
-    out = open(HEADLINES_JSONL, "w")  # tiny (~9 calls): fresh each run
-
-    def run_batch(texts, tag, batch_idx):
-        scores = None
-        for attempt in range(3):
-            scores = engine._llm_headline_scores(texts)
-            cap = rec.pop()
-            if scores is not None:
-                break
-            print(f"  {tag} batch {batch_idx} attempt {attempt + 1} "
-                  f"failed (http {cap.get('http_status')})")
-            time.sleep(2.0)
-        out.write(json.dumps({
-            "run_tag": tag, "batch": batch_idx, "headlines": texts,
-            "scores": scores,
-            "ts": datetime.now(timezone.utc).isoformat(), **cap,
-        }) + "\n")
-        out.flush()
-        return scores
-
-    batches = [heads[i:i + BATCH_CAP] for i in range(0, len(heads),
-                                                     BATCH_CAP)]
-    for bi, b in enumerate(batches):
-        ok = run_batch(b, "main", bi) is not None
-        print(f"main batch {bi} ({len(b)} headlines): "
-              f"{'ok' if ok else 'FAILED'}")
-        time.sleep(args.sleep)
-    run_batch(batches[0], "repeat", 0)
-    print("repeat probe done")
-    shuffled = heads[:]
-    random.Random(20260806).shuffle(shuffled)
-    for bi in range(0, len(shuffled), BATCH_CAP):
-        run_batch(shuffled[bi:bi + BATCH_CAP], "shuffled",
-                  bi // BATCH_CAP)
-    print("shuffled probe done →", HEADLINES_JSONL)
-    out.close()
-
-
-# ── report: pure offline function of the artifacts ───────────────────
-
-def _binom_p(k, n):
-    """Exact two-sided binomial p-value vs p=0.5 (no scipy)."""
-    if n == 0:
-        return float("nan")
-    dist = [math.comb(n, i) for i in range(n + 1)]
-    tot = float(2 ** n)
-    pk = dist[k]
-    return min(1.0, sum(c for c in dist if c <= pk) / tot)
-
-
-def _mcnemar_p(hit_a, hit_b):
-    """Exact McNemar via binomial on discordant pairs. Inputs: aligned
-    boolean lists."""
-    b = sum(1 for x, y in zip(hit_a, hit_b) if x and not y)
-    c = sum(1 for x, y in zip(hit_a, hit_b) if not x and y)
-    return b, c, _binom_p(min(b, c), b + c)
-
-
-def _latest_events(recs):
-    """Last pass-1 record per ffid (retries append; last wins)."""
-    out = {}
-    for r in recs:
-        if r.get("run_pass") == 1:
-            out[r["ffid"]] = r
-    return list(out.values())
-
-
-def _fmt_rate(hits, n, label=""):
-    if n == 0:
-        return f"{label:<24} n=0"
-    p = _binom_p(hits, n)
-    return (f"{label:<24} {hits:>4}/{n:<4} = {hits / n * 100:5.1f}%  "
-            f"(binom p={p:.3f})")
-
-
-def report_events():
-    recs = _latest_events(_load_jsonl(EVENTS_JSONL))
-    if not recs:
-        print("no event replay records — run `events` first")
-        return
-    bars = pd.read_parquet(BARS)
-    closes = bars["Close"].to_numpy()
-    close_ns = (bars.index + pd.Timedelta(minutes=15)).asi8
-
-    def anchor(t_ns, h_min):
-        i = np.searchsorted(close_ns, t_ns, side="right") - 1
-        if i < 0:
-            return None
-        stale_ns = max(45.0, h_min / 4.0) * 60e9
-        if t_ns - close_ns[i] > stale_ns:
-            return None
-        return closes[i]
-
-    rows, skip = [], {h: 0 for h, _ in HORIZONS}
-    base_skipped = 0
-    for r in recs:
-        t_ns = int(r["dateline"]) * 10 ** 9
-        base = anchor(t_ns, 0)
-        if base is None:
-            base_skipped += 1
-            continue
-        row = dict(r)
-        row.pop("request_body", None)
-        row.pop("response_body", None)
-        row["year"] = r["utc_time"][:4]
-        row["base"] = base
-        for hname, hmin in HORIZONS:
-            p = anchor(t_ns + hmin * 60 * 10 ** 9, hmin)
-            if p is None:
-                skip[hname] += 1
-                row[f"d_{hname}"] = np.nan
-            else:
-                row[f"d_{hname}"] = p - base
-        rows.append(row)
-    df = pd.DataFrame(rows)
-    df.to_csv(SCORED_CSV, index=False)
-
-    n_fail = int((df["llm_direction"].isna()).sum())
-    print("═" * 70)
-    print("PART A — event gold-direction vs actual XAU move")
-    print("═" * 70)
-    print(f"events replayed: {len(df)} (baseline-price skipped: "
-          f"{base_skipped}) | LLM hard failures: {n_fail}")
-    print("horizon skip counts (stale/closed):",
-          {k: v for k, v in skip.items() if v})
-    dist = df["llm_direction"].value_counts(dropna=False).to_dict()
-    print("LLM call distribution:", dist)
-
-    # Unconditional per-horizon noise scale (null): median |Δ| over all
-    # 2024+ bar times, position-shifted.
-    bars24 = bars[bars.index >= "2024-01-01"]
-    null_med = {}
-    for hname, hmin in HORIZONS:
-        k = hmin // 15
-        d = (bars24["Close"].shift(-k) - bars24["Close"]).abs().dropna()
-        null_med[hname] = float(d.median())
-    print("unconditional null median |Δ| ($):",
-          {k: round(v, 2) for k, v in null_med.items()})
-
-    # Strategy call columns: +1 gold up, −1 gold down, 0/NaN excluded.
-    df["c_llm"] = df["llm_direction"].map(
-        {"bullish": 1, "bearish": -1}).fillna(0)
-    df["c_doctrine"] = df["surprise"].map({"positive": -1, "negative": 1})
-    df["c_ffbw"] = df["better_worse"].map({1: -1, 2: 1}).fillna(0)
-    df["c_bull"] = 1
-    df["c_bear"] = -1
-
-    strategies = [("LLM", "c_llm"), ("doctrine(surprise)", "c_doctrine"),
-                  ("FF betterWorse", "c_ffbw"), ("always-bullish", "c_bull"),
-                  ("always-bearish", "c_bear")]
-
-    for hname, _ in HORIZONS:
-        dcol = f"d_{hname}"
-        sub = df[df[dcol].notna()]
-        print(f"\n── horizon {hname} (n scored bars={len(sub)}) ──")
-        zeros = int((sub[dcol] == 0).sum())
-        if zeros:
-            print(f"  Δ==0 count (scored as miss): {zeros}")
-        for label, col in strategies:
-            s = sub[sub[col] != 0]
-            hits = int((np.sign(s[dcol]) == s[col]).sum())
-            print("  " + _fmt_rate(hits, len(s), label))
-        # noise-filtered LLM hit rate
-        s = sub[(sub["c_llm"] != 0) &
-                (sub[dcol].abs() >= null_med[hname])]
-        hits = int((np.sign(s[dcol]) == s["c_llm"]).sum())
-        print("  " + _fmt_rate(hits, len(s),
-                               f"LLM |Δ|≥null-med"))
-        # event-window magnitude vs null
-        mag = sub[dcol].abs().median()
-        print(f"  event-window median |Δ| = ${mag:.2f} "
-              f"(null ${null_med[hname]:.2f})")
-        # paired McNemar: LLM vs each baseline on common directional set
-        for label, col in [("doctrine", "c_doctrine"),
-                           ("FF betterWorse", "c_ffbw"),
-                           ("always-bullish", "c_bull")]:
-            s = sub[(sub["c_llm"] != 0) & (sub[col] != 0)]
-            ha = list(np.sign(s[dcol]) == s["c_llm"])
-            hb = list(np.sign(s[dcol]) == s[col])
-            b, c, p = _mcnemar_p(ha, hb)
-            print(f"  McNemar LLM vs {label:<15} b={b:>3} c={c:>3} "
-                  f"p={p:.3f}")
-
-    # neutral-call flatness (15m)
-    neu = df[(df["llm_direction"] == "neutral") & df["d_15m"].notna()]
-    dire = df[(df["c_llm"] != 0) & df["d_15m"].notna()]
-    if len(neu):
-        print(f"\nneutral calls (n={len(neu)}): median |Δ_15m| "
-              f"${neu['d_15m'].abs().median():.2f} vs directional "
-              f"(n={len(dire)}) ${dire['d_15m'].abs().median():.2f} "
-              f"(null ${null_med['15m']:.2f})")
-
-    # breakdowns at 1h
-    dcol = "d_1h"
-    sub = df[df[dcol].notna() & (df["c_llm"] != 0)]
-    print("\n── LLM hit rate by event name (1h, top 10 by count) ──")
-    for name, grp in sorted(sub.groupby("name"), key=lambda kv: -len(kv[1]))[:10]:
-        hits = int((np.sign(grp[dcol]) == grp["c_llm"]).sum())
-        print("  " + _fmt_rate(hits, len(grp), name[:24]))
-    print("── LLM hit rate by year (1h) ──")
-    for yr, grp in sub.groupby("year"):
-        hits = int((np.sign(grp[dcol]) == grp["c_llm"]).sum())
-        print("  " + _fmt_rate(hits, len(grp), yr))
-
-    # cluster view: shared datelines
-    print("\n── same-timestamp clusters ──")
-    clus = df.groupby("dateline").filter(lambda g: len(g) > 1)
-    groups = list(clus.groupby("dateline"))
-    consistent = 0
-    net_hits, net_n = 0, 0
-    for _, g in groups:
-        signs = set(g.loc[g["c_llm"] != 0, "c_llm"])
-        if len(signs) <= 1:
-            consistent += 1
-        net = int(g["c_llm"].sum())
-        d = g["d_1h"].iloc[0]
-        if net != 0 and pd.notna(d) and d != 0:
-            net_n += 1
-            if np.sign(d) == np.sign(net):
-                net_hits += 1
-    print(f"clusters: {len(groups)} datelines "
-          f"({len(clus)} events, {len(clus) / max(len(df), 1) * 100:.0f}% of "
-          f"corpus) | internally consistent: {consistent}/{len(groups)}")
-    print("  " + _fmt_rate(net_hits, net_n, "cluster net-call (1h)"))
-
-    # determinism probe
-    p2 = {r["ffid"]: r for r in _load_jsonl(EVENTS_JSONL)
-          if r.get("run_pass") == 2 and r.get("llm_direction")}
-    if p2:
-        p1 = {r["ffid"]: r for r in recs}
-        agree = same_reason = n = 0
-        for fid, r2 in p2.items():
-            r1 = p1.get(fid)
-            if not r1 or not r1.get("llm_direction"):
-                continue
-            n += 1
-            if r1["llm_direction"] == r2["llm_direction"]:
-                agree += 1
-            if r1.get("reason_raw") == r2.get("reason_raw"):
-                same_reason += 1
-        print(f"\ndeterminism probe: direction agreement {agree}/{n}, "
-              f"identical reason text {same_reason}/{n}")
-
-    # stratified reason examples
-    print("\n── reason samples (1h hit/miss) ──")
-    sub = df[df[dcol].notna() & (df["c_llm"] != 0)].copy()
-    sub["hit"] = np.sign(sub[dcol]) == sub["c_llm"]
-    for verdict in (True, False):
-        picked, seen = [], set()
-        for _, r in sub[sub["hit"] == verdict].iterrows():
-            if r["name"] in seen:
-                continue
-            seen.add(r["name"])
-            picked.append(r)
-            if len(picked) == 10:
-                break
-        tag = "HIT " if verdict else "MISS"
-        for r in picked:
-            print(f"  [{tag}] {r['utc_time'][:16]} {r['name'][:30]:<30} "
-                  f"a={r['actual']} f={r['forecast']} → "
-                  f"{r['llm_direction']:<7} Δ1h={r[dcol]:+7.2f}  "
-                  f"“{(r['reason_raw'] or '')[:60]}”")
-    print(f"\nscored table → {SCORED_CSV}")
-
-
-def report_headlines():
-    recs = _load_jsonl(HEADLINES_JSONL)
-    if not recs:
-        print("\n(no headline replay records — run `headlines` first)")
-        return
-    g = _load_golden()
-    engine = NewsEngine(NewsConfig())
-
-    def collect(tag):
-        m = {}
-        for r in recs:
-            if r["run_tag"] == tag and r["scores"]:
-                for h, sc in zip(r["headlines"], r["scores"]):
-                    m[h] = tuple(sc)
-        return m
-
-    main, shuf = collect("main"), collect("shuffled")
-    print("\n" + "═" * 70)
-    print("PART B — golden headline set: LLM vs keyword baseline")
-    print("═" * 70)
-    missing = [h for h in g["headline"] if h not in main]
-    if missing:
-        print(f"WARNING: {len(missing)} headlines missing from main run")
-
-    def cls(x):
-        if x >= NEUTRAL_BAND:
-            return 1
-        if x <= -NEUTRAL_BAND:
-            return -1
-        return 0
-
-    rows = []
-    for _, r in g.iterrows():
-        h = r["headline"]
-        if h not in main:
-            continue
-        llm_xau = main[h][1]
-        kw = engine._analyze_headline_sentiment(h, for_currency="XAU")
-        rows.append({
-            "headline": h, "category": r["category"],
-            "exp": int(r["expected_xau_sign"]),
-            "llm_xau": llm_xau, "llm_cls": cls(llm_xau),
-            "kw": kw, "kw_cls": cls(kw),
-        })
-    t = pd.DataFrame(rows)
-    for who in ("llm", "kw"):
-        acc = (t[f"{who}_cls"] == t["exp"]).mean()
-        print(f"{who.upper():<4} 3-class accuracy (band ±{NEUTRAL_BAND}): "
-              f"{(t[f'{who}_cls'] == t['exp']).sum()}/{len(t)} "
-              f"= {acc * 100:.1f}%")
-    ha = list(t["llm_cls"] == t["exp"])
-    hb = list(t["kw_cls"] == t["exp"])
-    b, c, p = _mcnemar_p(ha, hb)
-    print(f"paired McNemar LLM vs keyword: b={b} c={c} p={p:.3f}")
-
-    print("\nper-category accuracy (LLM | keyword):")
-    for cat, grp in t.groupby("category"):
-        a1 = (grp["llm_cls"] == grp["exp"]).sum()
-        a2 = (grp["kw_cls"] == grp["exp"]).sum()
-        print(f"  {cat:<18} {a1:>2}/{len(grp):<2} | {a2:>2}/{len(grp)}")
-
-    print("\nLLM confusion (rows=expected, cols=predicted -1/0/+1):")
-    cm = pd.crosstab(t["exp"], t["llm_cls"]).reindex(
-        index=[-1, 0, 1], columns=[-1, 0, 1], fill_value=0)
-    print(cm.to_string())
-
-    print("\nLLM misclassified headlines:")
-    for _, r in t[t["llm_cls"] != t["exp"]].iterrows():
-        print(f"  exp {r['exp']:+d} got {r['llm_xau']:+.2f} "
-              f"[{r['category']}] {r['headline'][:70]}")
-
-    # repeat-batch determinism
-    m_b0 = [r for r in recs if r["run_tag"] == "main" and r["batch"] == 0]
-    r_b0 = [r for r in recs if r["run_tag"] == "repeat"]
-    if m_b0 and r_b0 and m_b0[0]["scores"] and r_b0[0]["scores"]:
-        a = np.array(m_b0[0]["scores"], dtype=float)
-        b_ = np.array(r_b0[0]["scores"], dtype=float)
-        exact = int((a == b_).all(axis=1).sum())
-        print(f"\nrepeat-batch: {exact}/{len(a)} score pairs identical, "
-              f"max |diff| = {np.abs(a - b_).max():.3f}")
-
-    # shuffled-order drift
-    if shuf:
-        both = [h for h in t["headline"] if h in shuf]
-        drift = [abs(main[h][1] - shuf[h][1]) for h in both]
-        if drift:
-            print(f"shuffled-order xau drift over {len(both)} headlines: "
-                  f"mean {np.mean(drift):.3f}, max {np.max(drift):.3f}")
-
-
-def cmd_report(args):
-    report_events()
-    report_headlines()
-
-
-# ── CLI ──────────────────────────────────────────────────────────────
-
-def main():
-    ap = argparse.ArgumentParser(description=__doc__.split("\n")[1])
-    sub = ap.add_subparsers(dest="cmd", required=True)
-
-    e = sub.add_parser("events", help="replay release surprises")
-    e.add_argument("--limit", type=int, default=0,
-                   help="smoke-test on first N events")
-    e.add_argument("--determinism", type=int, default=0,
-                   help="re-call N stratified events a second time")
-    e.add_argument("--sleep", type=float, default=0.3)
-    e.add_argument("--retry-passes", type=int, default=3)
-    e.add_argument("--timeout", type=float, default=30.0)
-    e.set_defaults(fn=cmd_events)
-
-    h = sub.add_parser("headlines", help="golden-set probes")
-    h.add_argument("--sleep", type=float, default=0.3)
-    h.add_argument("--timeout", type=float, default=30.0)
-    h.set_defaults(fn=cmd_headlines)
-
-    r = sub.add_parser("report", help="offline scoring report")
-    r.set_defaults(fn=cmd_report)
-
-    args = ap.parse_args()
-    args.fn(args)
-
-
-if __name__ == "__main__":
-    main()

backtest/research/llm_news_golden.csv

@@ -1,79 +0,0 @@
-headline,category,expected_xau_sign,rationale
-"Powell says restrictive policy needed for longer as inflation proves sticky",hawkish_fed,-1,"higher-for-longer rates lift real yields, gold-bearish"
-"Fed minutes show officials leaning toward two more rate hikes this year",hawkish_fed,-1,"more hikes = stronger dollar/yields"
-"Hot CPI print fuels bets on additional Fed tightening",hawkish_fed,-1,"tightening expectations weigh on gold"
-"Fed's Waller: higher-for-longer rates warranted by resilient economy",hawkish_fed,-1,"hawkish guidance"
-"FOMC raises rates 50bp, flags further increases ahead",hawkish_fed,-1,"outsized hike plus hawkish guidance"
-"Treasury yields spike as Fed pushes back on rate cut hopes",hawkish_fed,-1,"cut hopes dashed, yields up"
-"Core inflation accelerates, cementing case for hawkish Fed stance",hawkish_fed,-1,"hawkish repricing"
-"Fed dot plot shifts higher; markets price out easing for the year",hawkish_fed,-1,"easing priced out"
-"Fed cuts rates 25bp and signals more easing to come",dovish_fed,1,"easing cycle is gold-bullish"
-"Powell hints at rate cuts as inflation cools toward target",dovish_fed,1,"cut guidance"
-"FOMC minutes reveal growing support for policy easing",dovish_fed,1,"dovish minutes"
-"Fed's Williams says rate cuts appropriate later this year",dovish_fed,1,"cut guidance"
-"Soft inflation data boosts odds of a dovish Fed pivot",dovish_fed,1,"pivot odds up"
-"Fed pauses hikes, markets price aggressive easing cycle",dovish_fed,1,"easing priced in"
-"Central banks turn dovish as global growth falters",dovish_fed,1,"global easing backdrop"
-"Fed to restart bond purchases to support flagging economy",dovish_fed,1,"QE is gold-bullish"
-"Dollar index hits 12-month high on robust US economic data",strong_usd,-1,"strong dollar pressures gold"
-"US GDP grows 4.1% annualized, blowing past expectations",strong_usd,-1,"strong economy, hawkish implication"
-"Blockbuster jobs report sends dollar surging across the board",strong_usd,-1,"strong jobs lift dollar and yields"
-"US retail sales smash forecasts, underscoring consumer strength",strong_usd,-1,"strong data"
-"Greenback rallies as US data outshines Europe and Asia",strong_usd,-1,"dollar strength"
-"ISM services index jumps to two-year high on strong demand",strong_usd,-1,"strong data"
-"Consumer confidence surges to post-pandemic high in the US",strong_usd,-1,"strong data"
-"Dollar tumbles to eight-month low after dismal manufacturing data",weak_usd,1,"weak dollar lifts gold"
-"US economy adds far fewer jobs than expected; dollar slides",weak_usd,1,"weak jobs, dovish implication"
-"Greenback weakens as US retail sales unexpectedly contract",weak_usd,1,"weak data"
-"US GDP growth revised down sharply, stoking slowdown fears",weak_usd,1,"slowdown is dovish"
-"Dollar slumps as trade deficit balloons to record",weak_usd,1,"dollar weakness"
-"Soft services PMI adds to evidence of cooling US economy",weak_usd,1,"cooling economy"
-"US consumer sentiment plunges to two-year low",weak_usd,1,"weak data"
-"Real yields climb to multi-year highs as TIPS sell off",real_yields,-1,"rising real yields raise gold's opportunity cost"
-"10-year Treasury yield surges past 5% on supply glut",real_yields,-1,"yield spike"
-"Rising real interest rates dim appeal of non-yielding assets",real_yields,-1,"explicit opportunity-cost story"
-"Treasury yields sink as investors flock to bonds",real_yields,1,"falling yields support gold"
-"Real yields turn negative as inflation outpaces rates",real_yields,1,"negative real yields are gold's best backdrop"
-"Bond rally drives 10-year yield to lowest level in a year",real_yields,1,"falling yields"
-"Geopolitical tensions escalate in Middle East, investors seek safety",safe_haven,1,"safe-haven bid"
-"Stocks crater worldwide as banking crisis fears spread",safe_haven,1,"crisis flight to safety"
-"Escalating conflict drives flight to safe-haven assets",safe_haven,1,"explicit haven flow"
-"Investors dump risk assets after surprise sovereign default",safe_haven,1,"default shock"
-"Military escalation on NATO border rattles global markets",safe_haven,1,"geopolitical stress"
-"Wall Street plunges 4% in worst session since pandemic",safe_haven,1,"risk-off"
-"Currency crisis in emerging markets sparks contagion fears",safe_haven,1,"contagion, haven bid"
-"Nuclear rhetoric intensifies, sending investors to havens",safe_haven,1,"extreme geopolitical stress"
-"Ceasefire agreement lifts global markets, haven demand fades",risk_on,-1,"haven unwind"
-"Stocks roar to record highs as trade deal signed",risk_on,-1,"risk-on drains haven demand"
-"Risk appetite returns as banking fears subside",risk_on,-1,"risk-on"
-"Global equities rally on strong earnings and easing tensions",risk_on,-1,"risk-on"
-"Gold hits record high as central banks keep buying bullion",gold_direct,1,"direct demand"
-"Gold ETFs post biggest weekly inflows in three years",gold_direct,1,"investment inflows"
-"China's central bank extends gold buying spree to 18th month",gold_direct,1,"official-sector demand"
-"Miners struggle to meet gold demand as supply deficit widens",gold_direct,1,"supply deficit"
-"Gold slides 2% as investors book profits after record run",gold_direct,-1,"profit-taking pressure"
-"Gold tumbles below key support as ETF outflows accelerate",gold_direct,-1,"outflows"
-"Bullion demand weakens as Indian jewellery buying stalls",gold_direct,-1,"physical demand soft"
-"Speculators slash bullish gold bets as dollar strengthens",gold_direct,-1,"positioning unwind"
-"Fed holds rates steady, signals no rate hike on the horizon",negation_traps,1,"dovish hold; keyword layer sees the phrase rate hike and flips it gold-bearish"
-"Powell says no rate cuts until inflation shows sustained cooling",negation_traps,-1,"hawkish hold; keyword layer sees rate cut and calls it gold-bullish"
-"Recession fears fade as US data streak beats estimates",negation_traps,-1,"fading fear is risk-on; keywords see recession plus beats and cancel out"
-"Inflation fails to accelerate, easing pressure on the Fed",negation_traps,1,"disinflation opens door to cuts"
-"Fed's Bowman: not yet confident inflation is beaten, cuts premature",negation_traps,-1,"hawkish pushback phrased around negations"
-"Central bank stops short of hiking, but warns tightening may resume",negation_traps,-1,"hawkish hold with tightening bias"
-"No safe-haven bid for gold despite equity wobble, analysts say",negation_traps,-1,"explicitly negated haven demand"
-"Jobs report not as weak as feared, dollar recovers",negation_traps,-1,"less-bad data is dollar-positive"
-"US manufacturing output plunges as new orders dry up",inverted_metrics,1,"weak economy is dovish/gold-bullish; keyword layer reads plunges as bearish"
-"US factory activity slumps to three-year low",inverted_metrics,1,"weak data; keyword reads slumps as bearish"
-"Consumer spending falls for third straight month",inverted_metrics,1,"weak data; keyword reads falls as bearish"
-"Private payrolls growth slides to weakest pace since 2020",inverted_metrics,1,"weak jobs; keyword reads slides as bearish"
-"Jobless claims hit two-year high as layoffs spread",inverted_metrics,1,"higher claims = weaker economy = gold-bullish"
-"US unemployment rate hits 4.5%, highest since 2021",inverted_metrics,1,"higher unemployment = dovish = gold-bullish"
-"Lakers clinch playoff berth with overtime win over Suns",distractors,0,"irrelevant"
-"New Marvel film smashes box office records on opening weekend",distractors,0,"irrelevant"
-"Apple unveils thinner MacBook with faster chip lineup",distractors,0,"irrelevant"
-"Scientists discover new deep-sea species off Pacific coast",distractors,0,"irrelevant"
-"Heavy rain forecast for the Midwest through the weekend",distractors,0,"irrelevant"
-"Streaming service raises subscription prices in Europe",distractors,0,"irrelevant"
-"Electric vehicle startup recalls 12,000 cars over software glitch",distractors,0,"irrelevant"
-"Local elections see record turnout in several US states",distractors,0,"irrelevant"

backtest/research/oanda_td_divergence.py

@@ -1,199 +0,0 @@
-"""
-Oanda vs Twelve Data bar-divergence study — XAU/USD M15, 12 months.
-====================================================================
-Compares the mid-price OHLC bars the signal engine would see from each
-provider over the same window. Purpose: quantify whether migrating the
-signal feed from Twelve Data to Oanda is a formality (bars ~identical)
-or a recalibration (bars differ at a scale that moves indicators).
-
-Inputs:
-  --td      Twelve Data parquet (raw bars from backtest/cache/)
-  --oanda   Oanda parquet (from fetch_oanda_m5.py / fetch_oanda_m15.py)
-  --start / --end  analysis window (UTC dates)
-
-Outputs:
-  printed report
-  backtest/research/oanda_td_divergence_by_hour.csv
-  backtest/research/oanda_td_bar_diffs.parquet  (per-bar diffs, common index)
-"""
-
-import argparse
-from pathlib import Path
-
-import numpy as np
-import pandas as pd
-
-RESEARCH_DIR = Path(__file__).parent
-PIP = 0.10  # XAU pip in $
-
-
-def wilder_atr(df: pd.DataFrame, period: int = 14) -> pd.Series:
-    hl = df["High"] - df["Low"]
-    hc = (df["High"] - df["Close"].shift(1)).abs()
-    lc = (df["Low"] - df["Close"].shift(1)).abs()
-    tr = pd.concat([hl, hc, lc], axis=1).max(axis=1)
-    return tr.ewm(alpha=1.0 / period, adjust=False).mean()
-
-
-def pct_stats(s: pd.Series) -> dict:
-    a = s.abs()
-    return {
-        "mean": a.mean(), "median": a.median(),
-        "p90": a.quantile(0.90), "p95": a.quantile(0.95),
-        "p99": a.quantile(0.99), "max": a.max(),
-        "bias(mean signed)": s.mean(),
-    }
-
-
-def fmt_stats(name: str, s: pd.Series) -> str:
-    st = pct_stats(s)
-    return (f"  {name:<6} mean ${st['mean']:.3f}  med ${st['median']:.3f}  "
-            f"p95 ${st['p95']:.3f}  p99 ${st['p99']:.3f}  max ${st['max']:.2f}  "
-            f"bias {st['bias(mean signed)']:+.4f}")
-
-
-def main():
-    ap = argparse.ArgumentParser()
-    ap.add_argument("--td", default=str(RESEARCH_DIR.parent / "cache" /
-                                        "XAU_USD_15min_20240522_20260723.parquet"))
-    ap.add_argument("--oanda", default=str(RESEARCH_DIR /
-                                           "oanda_xau_m15_20250701_now.parquet"))
-    ap.add_argument("--start", default="2025-07-24")
-    ap.add_argument("--end", default="2026-07-23")
-    args = ap.parse_args()
-
-    td = pd.read_parquet(args.td)
-    oa = pd.read_parquet(args.oanda)
-
-    start = pd.Timestamp(args.start, tz="UTC")
-    end = pd.Timestamp(args.end, tz="UTC") + pd.Timedelta(days=1)
-    start = max(start, td.index[0], oa.index[0])
-    end = min(end, td.index[-1], oa.index[-1])
-    td = td.loc[(td.index >= start) & (td.index <= end)]
-    oa = oa.loc[(oa.index >= start) & (oa.index <= end)]
-
-    print("=" * 78)
-    print("OANDA vs TWELVE DATA — XAU/USD M15 bar divergence")
-    print(f"Window: {start} -> {end}  ({(end - start).days} days)")
-    print("=" * 78)
-
-    # ── Coverage ─────────────────────────────────────────────────────
-    common = td.index.intersection(oa.index)
-    td_only = td.index.difference(oa.index)
-    oa_only = oa.index.difference(td.index)
-    print(f"\nCOVERAGE")
-    print(f"  Twelve Data bars : {len(td):>7,}")
-    print(f"  Oanda bars       : {len(oa):>7,}")
-    print(f"  common           : {len(common):>7,}")
-    print(f"  TD-only          : {len(td_only):>7,}  "
-          f"({100 * len(td_only) / max(len(td), 1):.2f}% of TD)")
-    print(f"  Oanda-only       : {len(oa_only):>7,}  "
-          f"({100 * len(oa_only) / max(len(oa), 1):.2f}% of Oanda)")
-    for label, idx in (("TD-only", td_only), ("Oanda-only", oa_only)):
-        if len(idx):
-            by_date = pd.Series(1, index=idx).groupby(idx.date).sum()
-            top = by_date.sort_values(ascending=False).head(8)
-            print(f"  {label} bars cluster on: "
-                  + ", ".join(f"{d} ({n})" for d, n in top.items()))
-            by_hour = pd.Series(1, index=idx).groupby(idx.hour).sum()
-            top_h = by_hour.sort_values(ascending=False).head(6)
-            print(f"  {label} by UTC hour: "
-                  + ", ".join(f"{h:02d}h ({n})" for h, n in top_h.items()))
-
-    t = td.loc[common]
-    o = oa.loc[common]
-
-    # ── Scale references ─────────────────────────────────────────────
-    atr = wilder_atr(t).dropna()
-    mean_price = t["Close"].mean()
-    print(f"\nSCALE REFERENCES (Twelve Data feed)")
-    print(f"  mean close      : ${mean_price:,.2f}")
-    print(f"  mean bar range  : ${(t['High'] - t['Low']).mean():.3f}")
-    print(f"  mean ATR14      : ${atr.mean():.3f}   median ${atr.median():.3f}")
-
-    # ── Per-field diffs (TD − Oanda mid) ─────────────────────────────
-    print(f"\nPER-BAR DIFFS  (TD − Oanda, $; pip = $0.10)")
-    diffs = {}
-    for col in ("Open", "High", "Low", "Close"):
-        d = t[col] - o[col]
-        diffs[col] = d
-        print(fmt_stats(col, d))
-
-    dclose = diffs["Close"].abs()
-    print(f"\n  |ΔClose| as % of mean ATR14 : "
-          f"mean {100 * dclose.mean() / atr.mean():.1f}%   "
-          f"p95 {100 * dclose.quantile(0.95) / atr.mean():.1f}%")
-
-    print(f"\nTHRESHOLD EXCEEDANCE  (|ΔClose|)")
-    for thr in (0.10, 0.30, 0.65, 1.00, 2.00, 5.00):
-        pct = 100 * (dclose > thr).mean()
-        print(f"  > ${thr:<5.2f}: {pct:6.2f}% of bars  ({(dclose > thr).sum():,})")
-
-    # ── Direction / return agreement ─────────────────────────────────
-    t_ret = t["Close"].pct_change()
-    o_ret = o["Close"].pct_change()
-    # only consecutive common bars (gap = one M15 step)
-    step_ok = pd.Series(common, index=common).diff().eq(pd.Timedelta(minutes=15))
-    tr, orr = t_ret[step_ok], o_ret[step_ok]
-    corr = tr.corr(orr)
-    t_dir = np.sign(t["Close"] - t["Open"])
-    o_dir = np.sign(o["Close"] - o["Open"])
-    both_nonflat = (t_dir != 0) & (o_dir != 0)
-    dir_agree = 100 * (t_dir[both_nonflat] == o_dir[both_nonflat]).mean()
-    print(f"\nAGREEMENT")
-    print(f"  close-return correlation (consecutive bars): {corr:.5f}  "
-          f"(n={step_ok.sum():,})")
-    print(f"  bar direction (sign of close−open) agree   : {dir_agree:.2f}%")
-
-    # ── By-hour breakdown ────────────────────────────────────────────
-    hours = common.hour
-    rows = []
-    for h in range(24):
-        m = hours == h
-        if not m.any():
-            continue
-        row = {"hour_utc": h, "bars": int(m.sum()),
-               "mean_abs_dclose": dclose[m].mean(),
-               "p95_abs_dclose": dclose[m].quantile(0.95),
-               "mean_abs_dhigh": diffs["High"][m].abs().mean(),
-               "mean_abs_dlow": diffs["Low"][m].abs().mean()}
-        if "bid_c" in o.columns:
-            row["oanda_median_spread"] = (
-                (o["ask_c"] - o["bid_c"])[m].median())
-        rows.append(row)
-    by_hour = pd.DataFrame(rows)
-    out_hour = RESEARCH_DIR / "oanda_td_divergence_by_hour.csv"
-    by_hour.to_csv(out_hour, index=False, float_format="%.4f")
-    print(f"\nBY UTC HOUR  (written to {out_hour.name})")
-    print(by_hour.to_string(index=False,
-                            float_format=lambda x: f"{x:.3f}"))
-
-    # ── By-month drift ───────────────────────────────────────────────
-    print(f"\nBY MONTH  (mean |ΔClose|, $)")
-    by_month = dclose.groupby(common.to_period("M")).mean()
-    for p, v in by_month.items():
-        print(f"  {p}: {v:.3f}")
-
-    # ── Oanda real spread (bonus, ties to Dukascopy study) ───────────
-    if "bid_c" in o.columns:
-        spread = o["ask_c"] - o["bid_c"]
-        liquid = ~o.index.hour.isin([21, 22, 23, 0])
-        print(f"\nOANDA REAL SPREAD (close bid/ask, $)")
-        print(f"  all bars      : median {spread.median():.3f}  "
-              f"p25 {spread.quantile(0.25):.3f}  p75 {spread.quantile(0.75):.3f}")
-        print(f"  liquid hours  : median {spread[liquid].median():.3f}")
-        print(f"  21-00 UTC     : median {spread[~liquid].median():.3f}")
-
-    # ── Persist per-bar diffs ────────────────────────────────────────
-    out = pd.DataFrame({
-        "td_close": t["Close"], "oa_close": o["Close"],
-        "d_open": diffs["Open"], "d_high": diffs["High"],
-        "d_low": diffs["Low"], "d_close": diffs["Close"],
-    })
-    out_path = RESEARCH_DIR / "oanda_td_bar_diffs.parquet"
-    out.to_parquet(out_path)
-    print(f"\nPer-bar diffs written to {out_path}")
-
-
-if __name__ == "__main__":
-    main()

backtest/research/oanda_td_divergence_by_hour.csv

@@ -1,25 +0,0 @@
-hour_utc,bars,mean_abs_dclose,p95_abs_dclose,mean_abs_dhigh,mean_abs_dlow,oanda_median_spread
-0,1029,3.3483,13.9460,3.2906,3.4310,0.7900
-1,1028,3.3483,12.8292,3.3477,3.6398,0.7700
-2,1028,3.0288,10.8960,2.9469,3.1317,0.7700
-3,1028,2.8783,11.1300,2.8419,2.9992,0.7600
-4,1028,2.8232,10.4710,2.8165,2.8455,0.7400
-5,1028,2.9268,10.6027,2.9234,3.0038,0.7400
-6,1028,3.0466,10.9265,2.9487,3.0796,0.7200
-7,1028,3.0286,11.3512,3.0301,3.0797,0.6700
-8,1028,3.0223,11.1483,2.8989,3.0801,0.6600
-9,1028,3.0460,11.6345,2.9307,3.0609,0.6600
-10,1028,2.9280,10.9015,2.8647,3.0029,0.6600
-11,1028,3.0340,11.4712,2.9591,2.9472,0.6600
-12,1028,3.0986,11.3282,2.9412,3.1407,0.7000
-13,1028,3.4014,12.3343,3.1077,3.4228,0.7300
-14,1028,3.5074,11.3628,3.3960,3.6569,0.6900
-15,1028,3.4514,11.8982,3.4124,3.6733,0.6700
-16,1028,3.1275,12.5155,3.2038,3.3061,0.6700
-17,1020,2.9826,12.6435,3.0064,3.1008,0.6900
-18,1015,2.8721,12.5480,2.9101,2.9996,0.6900
-19,1004,2.8637,12.1335,2.8824,2.9454,0.6900
-20,996,2.8006,11.9325,2.8486,2.8577,0.7600
-21,335,1.1085,4.8600,1.1212,1.2034,0.9400
-22,676,4.0827,13.6787,4.8422,4.6761,0.8300
-23,1027,3.2722,12.5725,3.5617,3.7747,0.8700

backtest/research/open_trade_bursts.py

@@ -1,150 +0,0 @@
-"""What do violent bars do to OPEN trades, and can we protect them?  Analysis tooling only.
-
-Replays every trade of a book on the raw bars with the backtester's exit model (SL-first,
-gap-through at open, close-only trail ratchet, 300-bar timeout, $0.65 cost) and compares:
-  base        : reproduce the book (validation: must match r_multiple for almost every trade)
-  flat_data   : exit at the OPEN of the first bar whose close-hour enters {12,13} UTC (US data)
-  flat_asia   : same for close-hour {0,1} UTC (Asian open)
-  tight_data  : entering {12,13}, hold a protective stop 0.5R below the best price for the window
-  tight_asia  : same for {0,1}
-  react_exit  : a burst bar (range >= 3x prior ATR14) closes AGAINST the trade -> exit next open
-  react_tight : after an opposing burst, protective stop 0.5R below the best price for the rest
-Plus a damage table: what happens to a trade when the first burst hits it (with vs against).
-
-    python backtest/research/open_trade_bursts.py BOOK.csv PARQUET BAR_MINUTES ARM DIST
-"""
-import sys
-import numpy as np, pandas as pd
-
-COST = 0.65; MAX_BARS = 300; K = 3.0; PROT = 0.5
-
-
-def wilder_atr(df, period=14):
-    pc = df["Close"].shift(1)
-    tr = pd.concat([df["High"] - df["Low"], (df["High"] - pc).abs(), (df["Low"] - pc).abs()], axis=1).max(axis=1)
-    return tr.ewm(alpha=1 / period, min_periods=period, adjust=False).mean()
-
-
-def simulate(bars, i0, direction, entry, sl, arm, dist, variant, hours_flat=(), hours_tight=(), flat_mask=None, tight_mask=None):
-    """bars: dict of numpy arrays; i0 = index of the entry bar. Returns (exit_price, bars_held, reason, flag)."""
-    O, H, L, C, burst, chour = bars["O"], bars["H"], bars["L"], bars["C"], bars["burst"], bars["chour"]
-    d = 1.0 if direction == "BUY" else -1.0
-    risk = abs(entry - sl); stop = sl; peak = entry; flag = 0
-    n = len(O); tight = False; tight_until = -1; exit_next = False
-    for j in range(i0, min(n, i0 + MAX_BARS)):
-        if exit_next:
-            return O[j], j - i0, "REACT", flag
-        h = chour[j]
-        in_flat = h in hours_flat; in_tight = h in hours_tight
-        if variant.startswith("flat") and in_flat:
-            return O[j], j - i0, "FLAT", 1
-        if variant == "mask_flat" and flat_mask is not None and flat_mask[j]:
-            return O[j], j - i0, "FLAT", 1
-        if variant.startswith("tight") and in_tight:
-            tight = True; tight_until = j; flag = 1
-        if variant == "mask_tight" and tight_mask is not None and tight_mask[j]:
-            tight = True; tight_until = j; flag = 1
-        # stop check (SL first, gap-through at open)
-        if d > 0:
-            if O[j] <= stop: return O[j], j - i0, "SL", flag
-            if L[j] <= stop: return stop, j - i0, "SL", flag
-        else:
-            if O[j] >= stop: return O[j], j - i0, "SL", flag
-            if H[j] >= stop: return stop, j - i0, "SL", flag
-        # update peak / trail at bar close
-        peak = max(peak, H[j]) if d > 0 else min(peak, L[j])
-        fav = d * (peak - entry)
-        if fav >= arm * risk:
-            stop = max(stop, peak - dist * risk) if d > 0 else min(stop, peak + dist * risk)
-        if tight and (variant == "react_tight" or j <= tight_until or in_tight):
-            stop = max(stop, peak - PROT * risk) if d > 0 else min(stop, peak + PROT * risk)
-        if variant.startswith("react") and burst[j] and d * (C[j] - O[j]) < 0:
-            flag = 1
-            if variant == "react_exit": exit_next = True
-            else: tight = True
-    j = min(n - 1, i0 + MAX_BARS - 1)
-    return C[j], j - i0, "TIMEOUT", flag
-
-
-def r_of(direction, entry, sl, exit_price):
-    d = 1.0 if direction == "BUY" else -1.0
-    return (d * (exit_price - entry) - COST) / abs(entry - sl)
-
-
-def stats(r):
-    r = np.asarray(r); eq = np.concatenate([[0.0], np.cumsum(r)]); dd = (np.maximum.accumulate(eq) - eq).max()
-    pf = r[r > 0].sum() / max(-r[r < 0].sum(), 1e-9)
-    return dict(n=len(r), total=r.sum(), mean=r.mean(), wr=100 * (r > 0).mean(), pf=pf, dd=dd)
-
-
-def main():
-    book_csv, pq, bm, arm, dist = sys.argv[1], sys.argv[2], int(sys.argv[3]), float(sys.argv[4]), float(sys.argv[5])
-    book = pd.read_csv(book_csv); book["entry_time"] = pd.to_datetime(book["entry_time"], utc=True, format="mixed")
-    px = pd.read_parquet(pq)
-    atr_prev = wilder_atr(px).shift(1)
-    bars = dict(O=px["Open"].values, H=px["High"].values, L=px["Low"].values, C=px["Close"].values,
-                burst=((px["High"] - px["Low"]) >= K * atr_prev).fillna(False).values,
-                chour=(px.index + pd.Timedelta(minutes=bm)).hour.values)
-    idx = pd.Index(px.index)
-    pos = idx.get_indexer(book["entry_time"])
-    ok = pos >= 0
-    print(f"{book_csv}: {len(book)} trades, {ok.sum()} matched to bars; arm {arm} dist {dist}")
-    book = book[ok].reset_index(drop=True); pos = pos[ok]
-    half = np.where(np.arange(len(book)) < len(book) // 2, 1, 2)
-    variants = {"base": {}, "flat_data": dict(hours_flat=(12, 13)), "flat_asia": dict(hours_flat=(0, 1)),
-                "tight_data": dict(hours_tight=(12, 13)), "tight_asia": dict(hours_tight=(0, 1)),
-                "react_exit": {}, "react_tight": {}}
-    results = {}
-    for v, kw in variants.items():
-        rs, flags = [], []
-        for i, t in book.iterrows():
-            xp, held, reason, flag = simulate(bars, pos[i], t.direction, t.entry, t.sl, arm, dist, v, **kw)
-            rs.append(r_of(t.direction, t.entry, t.sl, xp)); flags.append(flag)
-        results[v] = (np.array(rs), np.array(flags))
-    base = results["base"][0]
-    match = np.isclose(base, book["r_multiple"].values, atol=0.02).mean()
-    print(f"validation: replay reproduces the book's r_multiple (±0.02) on {100*match:.1f}% of trades; "
-          f"sum replay {base.sum():+.2f} vs book {book.r_multiple.sum():+.2f}")
-    print("\n== PROTECTION VARIANTS (whole book; 'affected' = trades where the rule fired) ==")
-    print(f"{'variant':12s} {'n':>5s} {'affected':>8s} {'total R':>9s} {'Δ vs base':>9s} {'PF':>5s} {'maxDD':>6s} {'WR%':>5s} | affected only: base R -> variant R | Δ h1 / Δ h2")
-    b = stats(base)
-    for v, (r, f) in results.items():
-        s = stats(r); aff = f == 1
-        d1 = r[half == 1].sum() - base[half == 1].sum(); d2 = r[half == 2].sum() - base[half == 2].sum()
-        print(f"{v:12s} {s['n']:5d} {aff.sum():8d} {s['total']:+9.2f} {s['total']-b['total']:+9.2f} {s['pf']:5.2f} {s['dd']:6.2f} {s['wr']:5.1f} | "
-              f"{base[aff].sum():+8.2f} -> {r[aff].sum():+8.2f} (n={aff.sum()}) | {d1:+.2f} / {d2:+.2f}")
-    # damage table: first burst while open (base path), with vs against
-    print("\n== DAMAGE: first violent bar that hits an OPEN trade (base exits) ==")
-    rows = []
-    for i, t in book.iterrows():
-        d = 1.0 if t.direction == "BUY" else -1.0; risk = abs(t.entry - t.sl)
-        xp, held, reason, _ = simulate(bars, pos[i], t.direction, t.entry, t.sl, arm, dist, "base")
-        j0 = pos[i]; j1 = j0 + held
-        for j in range(j0 + 1, j1 + 1):   # bursts after the entry bar, up to and including the exit bar
-            if bars["burst"][j]:
-                against = d * (bars["C"][j] - bars["O"][j]) < 0
-                mtm_before = (d * (bars["C"][j - 1] - t.entry) - COST) / risk
-                mtm_after = (d * (bars["C"][j] - t.entry) - COST) / risk
-                rows.append(dict(against=against, bars_open=j - j0, mtm_before=mtm_before, mtm_after=mtm_after,
-                                 final=r_of(t.direction, t.entry, t.sl, xp), ended_here=(j == j1), hour=bars["chour"][j],
-                                 sched=bars["chour"][j] in (0, 1, 12, 13)))
-                break
-    dmg = pd.DataFrame(rows)
-    if len(dmg):
-        g = dmg.groupby("against").agg(n=("final", "size"), bars_open_med=("bars_open", "median"),
-                                       mtm_before=("mtm_before", "mean"), mtm_after=("mtm_after", "mean"),
-                                       final=("final", "mean"), ended_on_that_bar_pct=("ended_here", lambda s: 100 * s.mean()),
-                                       in_sched_hours_pct=("sched", lambda s: 100 * s.mean()))
-        print(g.round(3).to_string())
-        ag = dmg[dmg.against]
-        print(f"\nopposing bursts: {len(ag)} trades ({100*len(ag)/len(book):.1f}% of the book); "
-              f"mean R just before {ag.mtm_before.mean():+.3f} -> just after {ag.mtm_after.mean():+.3f} -> final {ag.final.mean():+.3f}; "
-              f"{100*(ag.final<0).mean():.0f}% end as losers; {100*ag.ended_here.mean():.0f}% are stopped ON that bar")
-        print(f"  those with an open PROFIT before the burst (mtm>0): n={int((ag.mtm_before>0).sum())}, final mean {ag[ag.mtm_before>0].final.mean():+.3f}, "
-              f"{100*(ag[ag.mtm_before>0].final<0).mean():.0f}% end as losers")
-        print(f"  by hour of the opposing burst (n, mean final R): " + ", ".join(f"{h:02d}:{n}/{m:+.2f}" for h, n, m in
-              ag.groupby("hour").final.agg(["size", "mean"]).reset_index().itertuples(index=False)))
-
-
-if __name__ == "__main__":
-    main()

backtest/research/panel_payload_drytest_2026-08-15.jsonl

@@ -1,7 +0,0 @@
-{"probe": "P1 coherent-bullish", "payload": {"brief": {"pair": "XAU/USD", "timeframe": "15min", "direction": "BUY", "quality": "A", "score": 38.5, "confidence": 62, "price": 2000.0, "market_regime": "trending_up", "volatility_regime": "normal", "mtf_agreement": true}, "indicators": {"ema_trend": {"signal": "STRONG_BUY", "confidence": 90, "value": 1.0}, "rsi": {"signal": "BUY", "confidence": 65, "value": 58.0}, "macd": {"signal": "STRONG_BUY", "confidence": 80, "value": 1.25}, "bollinger": {"signal": "BUY", "confidence": 60, "value": 0.7}, "adx": {"signal": "BUY", "confidence": 70, "value": 32.0}, "roc": {"signal": "BUY", "confidence": 55, "value": 0.35}, "keltner": {"signal": "BUY", "confidence": 65, "value": 3.1}, "candlestick": {"signal": "BUY", "confidence": 60, "value": 2.0}}, "mtf": {"5min": {"signal": "BUY", "score": 30.0}, "1h": {"signal": "STRONG_BUY", "score": 45.0}}, "key_levels": {"resistance": 2012.0, "support": 1985.0}, "exhaustion": 0, "candles": {"15min": {"ref_close": 2000.0, "atr": 5.0, "bars": [[-2.4, -2.2, -2.4, -2.3], [-2.3, -2.1, -2.3, -2.2], [-2.2, -2.0, -2.2, -2.0], [-2.0, -1.9, -2.1, -1.9], [-1.9, -1.8, -2.0, -1.8], [-1.8, -1.6, -1.8, -1.7], [-1.7, -1.5, -1.7, -1.6], [-1.6, -1.4, -1.6, -1.4], [-1.4, -1.3, -1.5, -1.3], [-1.3, -1.2, -1.4, -1.2], [-1.2, -1.0, -1.2, -1.1], [-1.1, -0.9, -1.1, -1.0], [-1.0, -0.8, -1.0, -0.8], [-0.8, -0.7, -0.9, -0.7], [-0.7, -0.6, -0.8, -0.6], [-0.6, -0.4, -0.6, -0.5], [-0.5, -0.3, -0.5, -0.4], [-0.4, -0.2, -0.4, -0.2], [-0.2, -0.1, -0.3, -0.1], [-0.1, 0.0, -0.2, 0.0]]}}}, "raw_reply": {"stance": "support", "confidence": 70, "note": "The trend is up and all signs agree, so buying fits the flow."}}
-{"probe": "P2 bearish-vs-BUY-brief", "payload": {"brief": {"pair": "XAU/USD", "timeframe": "15min", "direction": "BUY", "quality": "A", "score": 38.5, "confidence": 62, "price": 2000.0, "market_regime": "trending_down", "volatility_regime": "normal", "mtf_agreement": false}, "indicators": {"ema_trend": {"signal": "STRONG_SELL", "confidence": 85, "value": 0.1}, "rsi": {"signal": "SELL", "confidence": 70, "value": 74.0}, "macd": {"signal": "STRONG_SELL", "confidence": 80, "value": -1.4}, "bollinger": {"signal": "SELL", "confidence": 60, "value": 0.95}, "adx": {"signal": "SELL", "confidence": 70, "value": 35.0}, "roc": {"signal": "SELL", "confidence": 60, "value": -0.5}, "keltner": {"signal": "SELL", "confidence": 65, "value": -2.8}, "candlestick": {"signal": "SELL", "confidence": 55, "value": -2.0}}, "mtf": {"5min": {"signal": "SELL", "score": -28.0}, "1h": {"signal": "STRONG_SELL", "score": -41.0}}, "key_levels": {"resistance": 2012.0, "support": 1985.0}, "exhaustion": 3, "candles": {"15min": {"ref_close": 2000.0, "atr": 5.0, "bars": [[2.4, 2.4, 2.2, 2.3], [2.3, 2.3, 2.1, 2.2], [2.2, 2.2, 2.0, 2.0], [2.0, 2.1, 1.9, 1.9], [1.9, 2.0, 1.8, 1.8], [1.8, 1.8, 1.6, 1.7], [1.7, 1.7, 1.5, 1.6], [1.6, 1.6, 1.4, 1.4], [1.4, 1.5, 1.3, 1.3], [1.3, 1.4, 1.2, 1.2], [1.2, 1.2, 1.0, 1.1], [1.1, 1.1, 0.9, 1.0], [1.0, 1.0, 0.8, 0.8], [0.8, 0.9, 0.7, 0.7], [0.7, 0.8, 0.6, 0.6], [0.6, 0.6, 0.4, 0.5], [0.5, 0.5, 0.3, 0.4], [0.4, 0.4, 0.2, 0.2], [0.2, 0.3, 0.1, 0.1], [0.1, 0.2, -0.0, 0.0]]}}}, "raw_reply": {"stance": "oppose", "confidence": 85, "note": "The price is falling and every indicator says sell, so buying fights the trend."}}
-{"probe": "P3 conflicting", "payload": {"brief": {"pair": "XAU/USD", "timeframe": "15min", "direction": "BUY", "quality": "A", "score": 12.0, "confidence": 35, "price": 2000.0, "market_regime": "ranging", "volatility_regime": "normal", "mtf_agreement": false}, "indicators": {"ema_trend": {"signal": "BUY", "confidence": 55, "value": 0.6}, "rsi": {"signal": "NEUTRAL", "confidence": 40, "value": 51.0}, "macd": {"signal": "SELL", "confidence": 55, "value": -0.3}, "bollinger": {"signal": "NEUTRAL", "confidence": 35, "value": 0.5}, "adx": {"signal": "NEUTRAL", "confidence": 40, "value": 18.0}, "roc": {"signal": "SELL", "confidence": 50, "value": -0.1}, "keltner": {"signal": "BUY", "confidence": 50, "value": 0.8}, "candlestick": {"signal": "NEUTRAL", "confidence": 30, "value": 0.0}}, "mtf": {"5min": {"signal": "NEUTRAL", "score": 4.0}, "1h": {"signal": "NEUTRAL", "score": -3.0}}, "key_levels": {}, "exhaustion": 0, "candles": {"15min": {"ref_close": 2000.0, "atr": 5.0, "bars": [[0.0, 0.4, -0.1, 0.3], [0.3, 0.4, -0.1, 0.0], [0.0, 0.4, -0.1, 0.3], [0.3, 0.4, -0.1, 0.0], [0.0, 0.4, -0.1, 0.3], [0.3, 0.4, -0.1, 0.0], [0.0, 0.4, -0.1, 0.3], [0.3, 0.4, -0.1, 0.0], [0.0, 0.4, -0.1, 0.3], [0.3, 0.4, -0.1, 0.0], [0.0, 0.4, -0.1, 0.3], [0.3, 0.4, -0.1, 0.0], [0.0, 0.4, -0.1, 0.3], [0.3, 0.4, -0.1, 0.0], [0.0, 0.4, -0.1, 0.3], [0.3, 0.4, -0.1, 0.0], [0.0, 0.4, -0.1, 0.3], [0.3, 0.4, -0.1, 0.0], [0.0, 0.4, -0.1, 0.3], [0.3, 0.4, -0.1, 0.0]]}}}, "raw_reply": {"stance": "neutral", "confidence": 30, "note": "Mixed signals with no clear direction, so cannot strongly support or oppose."}}
-{"probe": "P4 empty-evidence", "payload": {"brief": {"pair": "XAU/USD", "timeframe": "15min", "direction": "BUY", "quality": "A", "score": 38.5, "confidence": 62, "price": 2000.0, "market_regime": "trending_up", "volatility_regime": "normal", "mtf_agreement": true}, "indicators": {}, "mtf": {}, "key_levels": {}, "exhaustion": 0}, "raw_reply": {"stance": "neutral", "confidence": 20, "note": "Not enough data to judge the trade direction."}}
-{"probe": "P5 hallucination-trap", "payload": {"brief": {"pair": "XAU/USD", "timeframe": "15min", "direction": "BUY", "quality": "A", "score": 38.5, "confidence": 62, "price": 2000.0, "market_regime": "trending_up", "volatility_regime": "normal", "mtf_agreement": true}, "indicators": {"ema_trend": {"signal": "STRONG_BUY", "confidence": 90, "value": 1.0}, "rsi": {"signal": "BUY", "confidence": 65, "value": 58.0}, "macd": {"signal": "STRONG_BUY", "confidence": 80, "value": 1.25}, "bollinger": {"signal": "BUY", "confidence": 60, "value": 0.7}, "adx": {"signal": "BUY", "confidence": 70, "value": 32.0}, "roc": {"signal": "BUY", "confidence": 55, "value": 0.35}, "keltner": {"signal": "BUY", "confidence": 65, "value": 3.1}, "candlestick": {"signal": "BUY", "confidence": 60, "value": 2.0}}, "mtf": {"5min": {"signal": "BUY", "score": 30.0}, "1h": {"signal": "STRONG_BUY", "score": 45.0}}, "key_levels": {}, "exhaustion": 0}, "raw_reply": {"stance": "support", "confidence": 70, "note": "The trend is up and all signs agree, so buying fits the flow."}}
-{"probe": "P6 output-discipline", "payload": {}, "raw_reply": null}
-{"probe": "P7 synthesizer", "payload": {"direction": "BUY", "takes": [{"persona": "macro", "stance": "support", "confidence": 70, "note": "evidence aligned"}, {"persona": "technical", "stance": "support", "confidence": 80, "note": "evidence aligned"}, {"persona": "risk", "stance": "support", "confidence": 65, "note": "evidence aligned"}, {"persona": "contrarian", "stance": "support", "confidence": 60, "note": "evidence aligned"}]}, "raw_reply": {"verdict": "AGREE", "confidence": 85, "rationale": "All four analysts support buying, with high confidence and consistent evidence. No one opposes, so the panel clearly backs the trade."}}

backtest/research/panel_payload_drytest_2026-08-15.py

@@ -1,271 +0,0 @@
-#!/usr/bin/env python3
-"""
-panel_payload_drytest_2026-08-15.py — live DeepSeek validation of the
-enriched agent-panel payloads + hardened prompts (2026-08-15).
-
-Spends ~6 REAL DeepSeek requests (against the shared 500/day cap) with
-FABRICATED payloads — made-up prices/indicators, no real market data —
-to verify the new technical-persona contract end to end:
-
-  P1 coherent bullish evidence       -> technical persona SUPPORTS
-  P2 bearish evidence vs BUY brief   -> technical persona OPPOSES
-  P3 conflicting evidence            -> NEUTRAL or low confidence
-  P4 empty evidence                  -> NEUTRAL with confidence < 30
-                                        (the anti-hallucination contract)
-  P5 hallucination trap              -> note invents no price levels
-  P6 output discipline               -> <=20-word notes, exact JSON keys
-  P7 synthesizer over 4 supports     -> AGREE verdict
-
-Usage:  venv/bin/python backtest/research/panel_payload_drytest_2026-08-15.py
-Exit code 0 = all probes passed. Raw replies land in
-backtest/research/panel_drytest_2026-08-15.jsonl
-"""
-
-import json
-import re
-import sys
-from pathlib import Path
-
-import pandas as pd
-
-REPO_ROOT = Path(__file__).resolve().parents[2]
-sys.path.insert(0, str(REPO_ROOT))
-
-import agent_panel as ap                      # noqa: E402
-from app_config import deepseek_api_key, get_app_config  # noqa: E402
-from llm_client import get_llm_client         # noqa: E402
-
-ARTIFACT = Path(__file__).with_suffix(".jsonl")
-
-
-# ── Fabrication helpers (all numbers are invented) ───────────────────
-
-def _staircase(start: float, step: float, n: int) -> pd.DataFrame:
-    """n synthetic bars drifting `step` per bar; last close = start+n*step."""
-    rows, price = [], start
-    for _ in range(n):
-        o, c = price, price + step
-        rows.append({"Open": o, "Close": c,
-                     "High": max(o, c) + abs(step) * 0.4,
-                     "Low": min(o, c) - abs(step) * 0.4})
-        price = c
-    return pd.DataFrame(rows)
-
-
-def _choppy(center: float, n: int) -> pd.DataFrame:
-    rows, price, sign = [], center, 1
-    for _ in range(n):
-        step = sign * 1.5
-        o, c = price, price + step
-        rows.append({"Open": o, "Close": c,
-                     "High": max(o, c) + 0.6, "Low": min(o, c) - 0.6})
-        price, sign = c, -sign
-    return pd.DataFrame(rows)
-
-
-_FAKE_PRICE = 2000.00          # deliberately made-up
-_FAKE_ATR = 5.0
-
-_BRIEF_BUY = {"pair": "XAU/USD", "timeframe": "15min", "direction": "BUY",
-              "quality": "A", "score": 38.5, "confidence": 62,
-              "price": _FAKE_PRICE, "market_regime": "trending_up",
-              "volatility_regime": "normal", "mtf_agreement": True}
-
-_BULL_INDICATORS = {
-    "ema_trend": {"signal": "STRONG_BUY", "confidence": 90, "value": 1.0},
-    "rsi": {"signal": "BUY", "confidence": 65, "value": 58.0},
-    "macd": {"signal": "STRONG_BUY", "confidence": 80, "value": 1.25},
-    "bollinger": {"signal": "BUY", "confidence": 60, "value": 0.7},
-    "adx": {"signal": "BUY", "confidence": 70, "value": 32.0},
-    "roc": {"signal": "BUY", "confidence": 55, "value": 0.35},
-    "keltner": {"signal": "BUY", "confidence": 65, "value": 3.1},
-    "candlestick": {"signal": "BUY", "confidence": 60, "value": 2.0},
-}
-
-_BEAR_INDICATORS = {
-    "ema_trend": {"signal": "STRONG_SELL", "confidence": 85, "value": 0.1},
-    "rsi": {"signal": "SELL", "confidence": 70, "value": 74.0},
-    "macd": {"signal": "STRONG_SELL", "confidence": 80, "value": -1.4},
-    "bollinger": {"signal": "SELL", "confidence": 60, "value": 0.95},
-    "adx": {"signal": "SELL", "confidence": 70, "value": 35.0},
-    "roc": {"signal": "SELL", "confidence": 60, "value": -0.5},
-    "keltner": {"signal": "SELL", "confidence": 65, "value": -2.8},
-    "candlestick": {"signal": "SELL", "confidence": 55, "value": -2.0},
-}
-
-_MIXED_INDICATORS = {
-    "ema_trend": {"signal": "BUY", "confidence": 55, "value": 0.6},
-    "rsi": {"signal": "NEUTRAL", "confidence": 40, "value": 51.0},
-    "macd": {"signal": "SELL", "confidence": 55, "value": -0.3},
-    "bollinger": {"signal": "NEUTRAL", "confidence": 35, "value": 0.5},
-    "adx": {"signal": "NEUTRAL", "confidence": 40, "value": 18.0},
-    "roc": {"signal": "SELL", "confidence": 50, "value": -0.1},
-    "keltner": {"signal": "BUY", "confidence": 50, "value": 0.8},
-    "candlestick": {"signal": "NEUTRAL", "confidence": 30, "value": 0.0},
-}
-
-
-def _tech_payload(indicators, mtf, candles_df=None, key_levels=None,
-                  brief=None, exhaustion=0):
-    payload = {"brief": brief or dict(_BRIEF_BUY),
-               "indicators": indicators, "mtf": mtf,
-               "key_levels": key_levels or {}, "exhaustion": exhaustion}
-    if candles_df is not None:
-        # the REAL production normalizer, not a hand-rolled copy
-        payload["candles"] = {
-            "15min": ap._normalize_candles(candles_df, _FAKE_ATR,
-                                           len(candles_df))}
-    return payload
-
-
-# ── Probe runner ─────────────────────────────────────────────────────
-
-_RESULTS = []
-
-
-def _record(probe, payload, reply, take, passed, detail):
-    _RESULTS.append({"probe": probe, "pass": bool(passed),
-                     "detail": detail,
-                     "stance": getattr(take, "stance", None),
-                     "confidence": getattr(take, "confidence", None),
-                     "note": getattr(take, "note", None)})
-    with ARTIFACT.open("a") as fh:
-        fh.write(json.dumps({"probe": probe, "payload": payload,
-                             "raw_reply": reply}, default=str) + "\n")
-    mark = "PASS" if passed else "FAIL"
-    print(f"[{mark}] {probe}: {detail}")
-    if take is not None:
-        print(f"       -> {take.stance} @ {take.confidence:.0f} "
-              f"| {take.note}")
-
-
-def _ask(payload):
-    """Real API call through the production client + production
-    validation (the same path run_panel uses)."""
-    reply = get_llm_client().chat_json(ap._TECH_SYSTEM, payload)
-    return reply, ap._validate_take("technical", reply)
-
-
-def _want(probe: str) -> bool:
-    """--only P3 runs just that probe (saves requests on re-runs)."""
-    if "--only" not in sys.argv:
-        return True
-    return sys.argv[sys.argv.index("--only") + 1] == probe
-
-
-def main() -> int:
-    app = get_app_config()
-    if not deepseek_api_key(app):
-        print("ABORT: no DeepSeek API key (env DEEPSEEK_API_KEY / "
-              "config.json deepseek.api_key).")
-        return 2
-    if app.deepseek_settings.get("enabled", True) is False:
-        print("ABORT: deepseek.enabled is false in config.json.")
-        return 2
-    if ARTIFACT.exists() and "--only" not in sys.argv:
-        ARTIFACT.unlink()  # full run = fresh wire log; --only appends
-
-    mtf_agree = {"5min": {"signal": "BUY", "score": 30.0},
-                 "1h": {"signal": "STRONG_BUY", "score": 45.0}}
-    levels = {"resistance": 2012.0, "support": 1985.0}
-
-    # P1 — coherent bullish, with rising ATR staircase (last close 2000)
-    if _want("P1"):
-        p1 = _tech_payload(_BULL_INDICATORS, mtf_agree,
-                           candles_df=_staircase(_FAKE_PRICE - 20 * 0.6,
-                                                 0.6, 20),
-                           key_levels=levels)
-        reply, take = _ask(p1)
-        _record("P1 coherent-bullish", p1, reply, take,
-                take is not None and take.stance == "support"
-                and take.confidence >= 50,
-                "expect support @ conf>=50")
-
-    # P2 — bearish everything against a BUY brief, falling staircase
-    if _want("P2"):
-        brief2 = dict(_BRIEF_BUY, market_regime="trending_down",
-                      mtf_agreement=False)
-        mtf_sell = {"5min": {"signal": "SELL", "score": -28.0},
-                    "1h": {"signal": "STRONG_SELL", "score": -41.0}}
-        p2 = _tech_payload(_BEAR_INDICATORS, mtf_sell, brief=brief2,
-                           candles_df=_staircase(_FAKE_PRICE + 20 * 0.6,
-                                                 -0.6, 20),
-                           key_levels=levels, exhaustion=3)
-        reply, take = _ask(p2)
-        _record("P2 bearish-vs-BUY-brief", p2, reply, take,
-                take is not None and take.stance == "oppose",
-                "expect oppose (BUY brief fights all evidence)")
-
-    # P3 — genuinely mixed evidence: MTF neutral both ways, ranging
-    # regime, choppy candles. (An H1 SELL against a BUY is a *listed
-    # conflict* in the prompt — oppose is correct there, not mixed.)
-    if _want("P3"):
-        brief3 = dict(_BRIEF_BUY, market_regime="ranging", score=12.0,
-                      confidence=35, mtf_agreement=False)
-        mtf_mixed = {"5min": {"signal": "NEUTRAL", "score": 4.0},
-                     "1h": {"signal": "NEUTRAL", "score": -3.0}}
-        p3 = _tech_payload(_MIXED_INDICATORS, mtf_mixed, brief=brief3,
-                           candles_df=_choppy(_FAKE_PRICE, 20))
-        reply, take = _ask(p3)
-        _record("P3 conflicting", p3, reply, take,
-                take is not None and (take.stance == "neutral"
-                                      or take.confidence < 60),
-                "expect neutral or conf<60 on genuinely mixed evidence")
-
-    # P4 — empty evidence: contract demands neutral @ conf<30
-    if _want("P4"):
-        p4 = _tech_payload({}, {}, key_levels=None)
-        reply, take = _ask(p4)
-        _record("P4 empty-evidence", p4, reply, take,
-                take is not None and take.stance == "neutral"
-                and take.confidence < 30,
-                "contract: thin/missing evidence -> neutral, conf<30")
-
-    # P5 — hallucination trap: no key_levels, no candles; the note must
-    # not cite any 3+-digit number absent from the payload
-    if _want("P5"):
-        p5 = _tech_payload(_BULL_INDICATORS, mtf_agree, key_levels=None)
-        reply, take = _ask(p5)
-        invented = []
-        if take is not None:
-            payload_blob = json.dumps(p5)
-            for num in re.findall(r"\d{3,}", take.note):
-                if num not in payload_blob:
-                    invented.append(num)
-        _record("P5 hallucination-trap", p5, reply, take,
-                take is not None and not invented,
-                f"invented numbers in note: {invented or 'none'}")
-
-    # P6 — output discipline across the persona replies gathered above
-    notes = [r for r in _RESULTS if r["note"]]
-    wordy = [r["probe"] for r in notes if len(r["note"].split()) > 20]
-    _record("P6 output-discipline", {}, None, None,
-            notes and not wordy,
-            f"notes >20 words: {wordy or 'none'} "
-            f"(cap is 15; 20 = fail threshold)")
-
-    # P7 — synthesizer over four fabricated SUPPORT takes
-    if _want("P7"):
-        takes = [{"persona": p, "stance": "support", "confidence": c,
-                  "note": "evidence aligned"}
-                 for p, c in (("macro", 70), ("technical", 80),
-                              ("risk", 65), ("contrarian", 60))]
-        synth_payload = {"direction": "BUY", "takes": takes}
-        reply = get_llm_client().chat_json(ap._SYNTH_SYSTEM, synth_payload)
-        verdict = reply.get("verdict") if isinstance(reply, dict) else None
-        rationale = (reply or {}).get("rationale", "")
-        _record("P7 synthesizer", synth_payload, reply, None,
-                verdict == "AGREE" and len(str(rationale).split()) <= 40,
-                f"expect AGREE + <=40-word rationale; got {verdict!r} "
-                f"({len(str(rationale).split())} words)")
-
-    n_pass = sum(1 for r in _RESULTS if r["pass"])
-    n_api = sum(1 for r in _RESULTS if r["probe"] != "P6 output-discipline")
-    print(f"\n{n_pass}/{len(_RESULTS)} probes passed "
-          f"({n_api} API requests spent; "
-          f"raw wire log: {ARTIFACT.name})")
-    return 0 if n_pass == len(_RESULTS) else 1
-
-
-if __name__ == "__main__":
-    sys.exit(main())

backtest/research/repin_verification_2026-08-14.md

@@ -0,0 +1,40 @@
+# Backtest Re-pin on Oanda bars — verification (Phase 3)
+
+**Date:** 2026-08-14 (verification; original books were generated 2026-08-04, current-HEAD M5 canonical book exported 2026-08-14)
+**Scope:** confirm the pinned Oanda books exist, satisfy rule 1, and quantify the Oanda↔TD delta that motivated them.
+
+## Re-pin state
+
+The re-pin is **already done** — full Oanda-bar books exist in `backtest/oanda_*.csv` for M5/M15/H1/H4, at both cost assumptions, with in-sample (12mo), out-of-sample (2024-06 → 2025-03), and 5-year variants. The shipped configs are `m5` (pullback OFF — `_nopb`) and `m15`.
+
+### M5 canonical-reference reconciliation (2026-08-14)
+
+The dated current-HEAD export `backtest/oanda_m5_12mo_065_nopb_20260814.csv` is the canonical M5 12-month reference going forward: **577 trades, +106.82R, PF 1.51**, with both 12-month halves positive (+50.80R / +56.02R; PF 1.46 / 1.55). It covers 2025-08-14 → 2026-08-14, runs the shipped M5 factory (XAU_PB off), charges $0.65 round trip, and uses next-open/raw-path/close-only-trail realism unchanged.
+
+Earlier M5 snapshots (`oanda_m5_12mo_065_nopb.csv`: 586 / +98.11R / 1.45; `m5_12mo_shipped.csv`: 588 / +96.60R / 1.44; and the 555 / +98.64R ledger entry) remain historical evidence from different HEADs around the weekend-calendar and related QA work. They must not be cited as the current-HEAD reference.
+
+### Rule-1 summary at realistic cost ($0.65 round trip)
+
+| TF | 12mo in-sample | 12mo H1 / H2 | OOS (2024-06→2025-03) |
+|---|---|---|---|
+| M15 | +73.93R PF 1.53 | +66.71 / +7.23 (PF 1.10) | +38.77R PF 1.36 |
+| M5 (shipped, nopb) | +106.82R PF 1.51 (current HEAD) | +50.80 / +56.02 (PF 1.46 / 1.55) | +9.52R PF 1.34 |
+| H1 | +23.97R PF 1.55 | — | +5.15R PF 1.13 |
+| H4 | +16.14R PF 2.28 | — | +1.31R PF 1.14 |
+
+- **M5 (shipped)** passes rule 1 cleanly: both current-HEAD 12mo halves positive with PF ≥ 1.3, and OOS positive with PF 1.34.
+- **M15** passes in-sample and OOS (PF 1.53 / 1.36) but its second half is soft (PF 1.10) — positive R, below the 1.3 bar. Pre-existing condition, not a re-pin artifact.
+- **H1/H4** pass in-sample but OOS is marginal (PF 1.13–1.14) — consistent with the standing "H4 unproven for auto-execution" note.
+
+## Oanda ↔ Twelve Data trade-level delta
+
+Not worth a trade-by-trade book diff: Phase 1 already showed the two feeds disagree on **bar direction** 19% of the time on M5 (11% on M15), with a ~$3 mean / ~$1 systematic close gap. A TD-pinned book is a different instrument's book — this is the documented re-pin motivator, and the Oanda books above are the replacement baseline. A formal trade-level reconciliation would show near-total mismatch and add no decision value.
+
+## Deliverable status
+
+- [x] Re-pinned books exist on Oanda bars (12mo + OOS + 5yr, $0.30 + $0.65).
+- [x] Rule 1 verified for the shipped TFs (M5 clean; M15 in-sample+OOS pass, H2 soft; H1/H4 OOS marginal).
+- [x] Cost model: $0.65 matches measured Oanda spread (~$0.65–0.72); $0.30 retained for cross-book comparability.
+- [x] Oanda↔TD delta quantified (Phase 1) — no further book diff needed.
+
+Remaining Phase-3 follow-up: none. Owner decision 2026-08-14 retained dual reporting: $0.65 is the realistic headline while $0.30 remains for TD-era cross-book comparability and the live TP spread gate.

backtest/research/signal_log_campaign_2026-09.md

@@ -44,7 +44,9 @@ book directly. Windows (cache start = start − 62 days):
 Acceptance: positive total R in both 12mo halves, PF ≥ 1.3, OOS not worse
 than control, and a candidate that removes a net-positive bucket for a PF-only
 gain is rejected (big-candle precedent). Runner:
-`<scratchpad>/campaign/run_campaign.sh`; summary: `summarize.py`.
+`<scratchpad>/campaign/run_campaign.sh`; summary: `summarize.py` (scratchpad
+tooling, not retained — both shipped gates are reproduced by a plain pair-factory
+run at HEAD; see the parity recipe in `sizing_campaign_2026-09.md`).
 
 Candidates (all `--set`, no code):
 

backtest/research/signal_parity_2026-08-14.md

@@ -0,0 +1,59 @@
+# Signal-Level Replay Parity — Oanda vs Tiingo (Phase 2)
+
+**Date:** 2026-08-14
+**Tool:** `scripts/signal_parity_study.py` (script removed 2026-09-06, in git history) (replays real `analyze_from_dataframes`, news off)
+**Window:** 2025-08-14 → 2026-08-14 (12 months); M15 stride 8, M5 stride 24 → ~2,900 sampled decisions each
+**Preceding:** Phase 1 price-diff study (`feed_divergence_2026-08-14.md`)
+
+## Headline
+
+On the ~94–96% of bars the two feeds share, **Oanda and Tiingo produce near-identical signals**:
+
+| Metric | M15 | M5 |
+|---|---|---|
+| same direction | 98.32% | 98.16% |
+| same grade | 94.13% | 95.23% |
+| same setup presence | 98.83% | 99.49% |
+| same direction + grade + setup | 93.41% | 94.21% |
+| opposite-sign flips (BUY↔SELL) | **0** | **0** |
+| direction flips on actionable bars | 6.6% | 10.9% |
+| grade changed on actionable bars | 14.2% | 7.8% |
+| mean \|Δ final_score\| | 1.76 | 1.78 |
+| mean \|Δ primary_score\| | 1.73 | 1.77 |
+| same strategy tag | 100% | 100% |
+
+Every direction flip is **actionable↔NEUTRAL**, never a sign reversal: the feeds never disagree on buy-vs-sell, only on *whether* a marginal bar crosses the action threshold. Flipping bars cluster at the Friday-close/Sunday-reopen boundary (UTC hours 22–23) and in the Asian hours (03–04).
+
+## Setup levels diverge even when the signal agrees
+
+On bars where **both** feeds emit a setup with the same tag:
+
+| | M15 (n=119) | M5 (n=39) |
+|---|---|---|
+| entry mean \|Δ\| | $0.12 | $0.10 |
+| SL mean \|Δ\| | $1.83 | $2.79 |
+| TP1 mean \|Δ\| | $2.47 | $1.52 |
+| TP2 mean \|Δ\| | $3.74 | $2.46 |
+| TP1 p95 \|Δ\| | $17.4 | $2.3 |
+
+Entry price is the current close (≈ identical across feeds), but **SL/TP are recomputed from ATR and recent swing extremes, which differ more** — mean SL drift ~$2–3 (p95 up to $10–26 on M15), i.e. 20–40% of ATR. So even an identical signal places stops/targets at meaningfully different prices on a different feed.
+
+## Phase 2 pass/fail vs plan thresholds
+
+| Criterion | Plan threshold | M15 / M5 | Verdict |
+|---|---|---|---|
+| identical direction+grade | ≥ 97% | 93.4% / 94.2% | **marginal** (below 97%, above 93%) |
+| opposite-sign flips | (none specified) | 0 / 0 | PASS |
+| flips clustered at marginal composites | no session/regime clustering | yes (boundary hours) | PASS |
+
+The "identical direction+grade" metric lands slightly under the proposed 97% bar — but the shortfall is grade *letters* (A+ vs A vs B) changing on marginal composites, not direction. If the acceptance criterion is "no opposite-sign disagreement and ~98% direction identity", this passes comfortably.
+
+## Conclusion
+
+Oanda ↔ Tiingo is **signal-equivalent for direction** (98% identical, zero sign flips) over 12 months. The residual deltas are:
+
+1. **~6% of decisions differ in grade letter** on marginal composites (A+↔A↔B near thresholds).
+2. **SL/TP levels drift ~$2–4 mean** (up to ~$20 p95 on M15) even for identical signals — a real P&L difference at the trade level.
+3. **Tiingo's extra Sunday bars** (Phase 1) are gated by the aggregator's NY-anchored weekend filter, so they do not inject phantom signals, but they do slightly shift indicator history at Monday opens.
+
+Net: switching Oanda→Tiingo changes a handful of marginal decisions and moves stop/target levels, but does not change *what the system is doing* (direction, strategy, setup presence). Oanda↔Twelve Data was not replayed — Phase 1 already showed 81% M5 bar-direction disagreement and a ~$3/$1 systematic gap, so a TD replay would trivially fail (that is the documented re-pin motivator).

backtest/research/sizing_model.py

@@ -1,961 +0,0 @@
-"""Stage-D stake-sizing model driver (2026-09 campaign).
-
-Walk-forward fit + evaluation of a per-signal stake multiplier on a trade
-book exported by ``gpu_backtest.py --export-features`` (63 base columns +
-``stake_multiplier``/``r_sized`` + 65 ``f_*`` decision-time features).
-
-The protocol implemented here is the one pre-registered in
-``backtest/research/sizing_campaign_2026-09.md`` and is NOT to be varied to
-rescue a result:
-
-* expanding walk-forward, 12-month burn-in, calendar-quarter refits, a
-  ``purge_bars × bar_minutes`` gap between the end of train and the start of
-  the test quarter, standardisation fit on train only;
-* models: ridge on winsorised R (primary), L2 logistic on ``R > 0``
-  (secondary), a one-feature ``f_is_buy`` simplicity baseline, an
-  exhaustion-block-only ablation, and an optional lightgbm upper bound;
-* size map ``m = round_0.05(clip(normaliser × (0.5 + clip((z−lo)/(hi−lo),0,1)),
-  0.5, 1.5))`` with ``lo``/``hi``/``normaliser`` calibrated on TRAIN
-  predictions only;
-* a label-shuffle null (training labels permuted, test labels untouched);
-* six acceptance clauses, printed PASS/FAIL individually.
-
-Usage:
-    python backtest/research/sizing_model.py --book BOOK.csv --bar-minutes 5 \
-        --features f_a,f_b,... [--model ridge|logistic] [--out DIR] \
-        [--n-shuffle 50] [--skip-lgbm] \
-        [--write-artifact PATH --tf 5min --confirm-tfs 15min,1h]
-
-Analysis tooling only — never imported by the live code.
-"""
-from __future__ import annotations
-
-import argparse
-import hashlib
-import json
-import math
-import subprocess
-import sys
-import warnings
-from pathlib import Path
-from typing import Dict, List, Optional, Sequence, Tuple
-
-import numpy as np
-import pandas as pd
-
-PROJECT_ROOT = Path(__file__).resolve().parents[2]
-if str(PROJECT_ROOT) not in sys.path:
-    sys.path.insert(0, str(PROJECT_ROOT))
-
-from position_sizing import FEATURE_ORDER, FEATURE_VERSION  # noqa: E402  (read-only import)
-
-# ── Pre-registered constants (flag any change loudly) ──────────────────
-WINSOR_LO, WINSOR_HI = -1.5, 4.0
-LAMBDA_GRID: Tuple[float, ...] = (1.0, 10.0, 100.0, 1000.0)
-CV_FOLDS = 5
-MIN_TRAIN = 150
-MAP_Q_LO, MAP_Q_HI = 0.10, 0.90
-MAP_OUT_LO, MAP_OUT_HI = 0.5, 1.5
-MAP_CLIP_LO, MAP_CLIP_HI = 0.5, 1.5
-MAP_ROUND_TO = 0.05
-MEAN_MULT_TOL = 0.05
-DD_TOL = 1.10
-PF_FLOOR = 1.3
-NEWTON_MAX_ITER = 50
-NEWTON_TOL = 1e-8
-
-#: The exhaustion-block ablation's fixed feature list.
-EXHAUSTION_FEATURES: Tuple[str, ...] = (
-    "f_swing_dist_atr",
-    "f_ema21_dist_atr",
-    "f_ema200_dist_atr",
-    "f_exhaustion_count",
-    "f_penalty_total",
-    "f_tag_bb_overextended",
-    "f_tag_rsi_extended",
-    "f_tag_no_pullback",
-    "f_tag_exhaustion",
-)
-ISBUY_FEATURES: Tuple[str, ...] = ("f_is_buy",)
-
-#: lightgbm never ships; running the full shuffle null through 200-tree fits
-#: costs minutes for a number nobody acts on, so its null is capped here.
-#: DEVIATION FROM THE PROTOCOL CONSTANT — printed in the summary when used.
-LGBM_MAX_SHUFFLE = 10
-
-LGBM_PARAMS = dict(
-    n_estimators=200,
-    learning_rate=0.03,
-    num_leaves=7,
-    min_child_samples=30,
-    subsample=0.8,
-    colsample_bytree=0.8,
-    reg_lambda=10,
-)
-
-_LINES: List[str] = []
-
-
-def say(msg: str = "") -> None:
-    """Print and tee into the summary buffer."""
-    print(msg)
-    _LINES.append(msg)
-
-
-# ══════════════════════════════════════════════════════════════════════
-# Book loading
-# ══════════════════════════════════════════════════════════════════════
-def load_book(path: Path) -> pd.DataFrame:
-    """Read a feature book, parse ``entry_time`` UTC, drop NaN-R rows."""
-    df = pd.read_csv(path)
-    if "entry_time" not in df.columns or "r_multiple" not in df.columns:
-        raise SystemExit(f"{path}: not a trade book (no entry_time/r_multiple)")
-    df["entry_time"] = pd.to_datetime(df["entry_time"], utc=True, format="mixed")
-    n0 = len(df)
-    bad = ~np.isfinite(pd.to_numeric(df["r_multiple"], errors="coerce").to_numpy(float))
-    if bad.any():
-        say(f"WARNING: dropping {int(bad.sum())}/{n0} rows with non-finite r_multiple")
-        df = df.loc[~bad].copy()
-    df["r_multiple"] = df["r_multiple"].astype(float)
-    df = df.sort_values("entry_time", kind="mergesort").reset_index(drop=True)
-    return df
-
-
-def resolve_features(df: pd.DataFrame, names: Sequence[str], label: str) -> List[str]:
-    """Validate a requested feature list against the book and FEATURE_ORDER."""
-    missing = [n for n in names if n not in df.columns]
-    if missing:
-        raise SystemExit(f"{label}: columns absent from book: {', '.join(missing)}")
-    unknown = [n for n in names if n.startswith("f_") and n not in FEATURE_ORDER]
-    if unknown:
-        say(f"NOTE: {label}: {', '.join(unknown)} not in position_sizing.FEATURE_ORDER")
-    return list(names)
-
-
-def sha256_file(path: Path) -> str:
-    h = hashlib.sha256()
-    with open(path, "rb") as fh:
-        for chunk in iter(lambda: fh.read(1 << 20), b""):
-            h.update(chunk)
-    return h.hexdigest()
-
-
-def git_commit() -> str:
-    try:
-        return subprocess.check_output(
-            ["git", "rev-parse", "HEAD"], cwd=PROJECT_ROOT, text=True).strip()
-    except Exception:  # pragma: no cover - research tool
-        return "unknown"
-
-
-# ══════════════════════════════════════════════════════════════════════
-# Walk-forward geometry
-# ══════════════════════════════════════════════════════════════════════
-def quarter_starts(first: pd.Timestamp, last: pd.Timestamp) -> List[pd.Timestamp]:
-    """Every Jan/Apr/Jul/Oct 1 00:00 UTC in ``[first, last]`` inclusive."""
-    out: List[pd.Timestamp] = []
-    year = first.year
-    while True:
-        for month in (1, 4, 7, 10):
-            ts = pd.Timestamp(year=year, month=month, day=1, tz="UTC")
-            if ts < first:
-                continue
-            if ts > last:
-                return out
-            out.append(ts)
-        year += 1
-
-
-def quarter_label(ts: pd.Timestamp) -> str:
-    return f"{ts.year}Q{(ts.month - 1) // 3 + 1}"
-
-
-def build_folds(times: np.ndarray, n_folds: int) -> List[np.ndarray]:
-    """Contiguous-by-time index blocks (rows are already time-sorted)."""
-    n = len(times)
-    edges = np.linspace(0, n, n_folds + 1).astype(int)
-    return [np.arange(edges[i], edges[i + 1]) for i in range(n_folds)
-            if edges[i + 1] > edges[i]]
-
-
-# ══════════════════════════════════════════════════════════════════════
-# Standardisation
-# ══════════════════════════════════════════════════════════════════════
-def fit_standardiser(X: np.ndarray) -> Tuple[np.ndarray, np.ndarray, np.ndarray]:
-    """Return (mean, scale, keep) from a training matrix.
-
-    NaNs are ignored when computing the mean (they are later filled with it).
-    ``scale`` is the population std with zeros replaced by 1.0; ``keep`` is
-    False for zero-variance-in-train columns, which are dropped from the fit
-    and carry a coefficient of 0.
-    """
-    with warnings.catch_warnings():
-        warnings.simplefilter("ignore", category=RuntimeWarning)
-        mean = np.nanmean(X, axis=0)
-        std = np.nanstd(X, axis=0)
-    mean = np.where(np.isfinite(mean), mean, 0.0)
-    std = np.where(np.isfinite(std), std, 0.0)
-    keep = std > 0
-    scale = np.where(keep, std, 1.0)
-    return mean, scale, keep
-
-
-def apply_standardiser(X: np.ndarray, mean: np.ndarray, scale: np.ndarray) -> np.ndarray:
-    """Fill NaN with the TRAIN mean, then standardise."""
-    Z = np.array(X, dtype=float, copy=True)
-    bad = ~np.isfinite(Z)
-    if bad.any():
-        Z[bad] = np.take(mean, np.where(bad)[1])
-    return (Z - mean) / scale
-
-
-# ══════════════════════════════════════════════════════════════════════
-# Models (numpy only)
-# ══════════════════════════════════════════════════════════════════════
-def _solve(A: np.ndarray, b: np.ndarray) -> np.ndarray:
-    try:
-        return np.linalg.solve(A, b)
-    except np.linalg.LinAlgError:  # pragma: no cover - degenerate design
-        return np.linalg.lstsq(A, b, rcond=None)[0]
-
-
-def fit_ridge(X: np.ndarray, y: np.ndarray, lam: float) -> Tuple[np.ndarray, float]:
-    """Closed-form ridge with an UNPENALISED intercept.
-
-    Centring X and y is exactly equivalent to appending a 1-column carrying
-    zero penalty. Returns ``(coef, intercept)`` with ``pred = intercept + X @ coef``.
-    """
-    if X.shape[1] == 0:
-        return np.zeros(0), float(y.mean()) if len(y) else 0.0
-    xm = X.mean(axis=0)
-    ym = float(y.mean())
-    Xc = X - xm
-    w = _solve(Xc.T @ Xc + lam * np.eye(X.shape[1]), Xc.T @ (y - ym))
-    return w, ym - float(xm @ w)
-
-
-def fit_logistic(X: np.ndarray, y: np.ndarray, lam: float,
-                 max_iter: int = NEWTON_MAX_ITER,
-                 tol: float = NEWTON_TOL) -> Tuple[np.ndarray, float, float]:
-    """L2 logistic via Newton-Raphson, intercept unpenalised.
-
-    Returns ``(coef, intercept, grad_norm)`` where ``grad_norm`` is the
-    infinity-norm of the penalised gradient at the returned point.
-    """
-    n, p = X.shape
-    A = np.hstack([np.ones((n, 1)), X])
-    pen = np.zeros(p + 1)
-    pen[1:] = lam
-    beta = np.zeros(p + 1)
-    # Warm-start the intercept at the empirical log-odds.
-    frac = float(np.clip(y.mean(), 1e-6, 1 - 1e-6))
-    beta[0] = math.log(frac / (1 - frac))
-    grad = np.zeros(p + 1)
-    for _ in range(max_iter):
-        eta = A @ beta
-        prob = 1.0 / (1.0 + np.exp(-np.clip(eta, -35, 35)))
-        grad = A.T @ (y - prob) - pen * beta
-        W = np.clip(prob * (1 - prob), 1e-10, None)
-        H = A.T @ (A * W[:, None]) + np.diag(pen)
-        step = _solve(H, grad)
-        beta = beta + step
-        if np.max(np.abs(step)) < tol:
-            break
-    eta = A @ beta
-    prob = 1.0 / (1.0 + np.exp(-np.clip(eta, -35, 35)))
-    grad = A.T @ (y - prob) - pen * beta
-    return beta[1:], float(beta[0]), float(np.max(np.abs(grad)))
-
-
-def predict_linear(X: np.ndarray, coef: np.ndarray, intercept: float) -> np.ndarray:
-    if X.shape[1] == 0:
-        return np.full(len(X), intercept, dtype=float)
-    return intercept + X @ coef
-
-
-def sigmoid(z: np.ndarray) -> np.ndarray:
-    return 1.0 / (1.0 + np.exp(-np.clip(z, -35, 35)))
-
-
-def log_loss(y: np.ndarray, p: np.ndarray) -> float:
-    p = np.clip(p, 1e-12, 1 - 1e-12)
-    return float(-np.mean(y * np.log(p) + (1 - y) * np.log(1 - p)))
-
-
-def choose_lambda(X: np.ndarray, y: np.ndarray, kind: str,
-                  grid: Sequence[float] = LAMBDA_GRID,
-                  n_folds: int = CV_FOLDS) -> Tuple[float, Dict[float, float]]:
-    """5-fold time-contiguous CV inside the training window."""
-    folds = build_folds(np.arange(len(y)), n_folds)
-    if X.shape[1] == 0 or len(folds) < 2:
-        return float(grid[0]), {}
-    scores: Dict[float, float] = {}
-    for lam in grid:
-        vals: List[float] = []
-        for hold in folds:
-            mask = np.ones(len(y), dtype=bool)
-            mask[hold] = False
-            if mask.sum() < 2:
-                continue
-            if kind == "logistic":
-                ytr = y[mask]
-                if ytr.min() == ytr.max():          # single-class fold
-                    p = np.full(len(hold), float(np.clip(ytr.mean(), 1e-6, 1 - 1e-6)))
-                else:
-                    c, b, _ = fit_logistic(X[mask], ytr, lam)
-                    p = sigmoid(predict_linear(X[hold], c, b))
-                vals.append(log_loss(y[hold], p))
-            else:
-                c, b = fit_ridge(X[mask], y[mask], lam)
-                resid = y[hold] - predict_linear(X[hold], c, b)
-                vals.append(float(np.mean(resid ** 2)))
-        scores[float(lam)] = float(np.mean(vals)) if vals else float("inf")
-    best = min(scores, key=lambda k: scores[k])
-    return best, scores
-
-
-def lgbm_available() -> bool:
-    try:
-        import lightgbm  # noqa: F401
-        return True
-    except Exception:
-        return False
-
-
-def fit_lgbm(X: np.ndarray, y: np.ndarray, seed: int):
-    import lightgbm as lgb
-    model = lgb.LGBMRegressor(random_state=seed, verbose=-1, **LGBM_PARAMS)
-    model.fit(X, y)
-    return model
-
-
-# ══════════════════════════════════════════════════════════════════════
-# Size map
-# ══════════════════════════════════════════════════════════════════════
-def round_to(x, step: float = MAP_ROUND_TO) -> np.ndarray:
-    """Round to the nearest ``step`` and kill float dust (0.05 → 2 dp)."""
-    decimals = max(0, int(round(-math.log10(step)))) + 1
-    return np.round(np.round(np.asarray(x, dtype=float) / step) * step, decimals)
-
-
-def calibrate_map(z_train: np.ndarray) -> Dict[str, float]:
-    """lo/hi/normaliser from TRAIN predictions only."""
-    lo = float(np.quantile(z_train, MAP_Q_LO))
-    hi = float(np.quantile(z_train, MAP_Q_HI))
-    if not np.isfinite(lo) or not np.isfinite(hi) or hi <= lo:
-        return {"lo": lo, "hi": hi, "normalizer": 1.0, "degenerate": True}
-    raw = MAP_OUT_LO + (MAP_OUT_HI - MAP_OUT_LO) * np.clip((z_train - lo) / (hi - lo), 0, 1)
-    mean_raw = float(np.mean(raw))
-    norm = 1.0 / mean_raw if mean_raw > 0 else 1.0
-    return {"lo": lo, "hi": hi, "normalizer": norm, "degenerate": False}
-
-
-def apply_map(z: np.ndarray, cal: Dict[str, float]) -> np.ndarray:
-    z = np.asarray(z, dtype=float)
-    if cal.get("degenerate"):
-        return np.ones(len(z))
-    lo, hi, norm = cal["lo"], cal["hi"], cal["normalizer"]
-    raw = MAP_OUT_LO + (MAP_OUT_HI - MAP_OUT_LO) * np.clip((z - lo) / (hi - lo), 0, 1)
-    return round_to(np.clip(raw * norm, MAP_CLIP_LO, MAP_CLIP_HI))
-
-
-# ══════════════════════════════════════════════════════════════════════
-# Metrics
-# ══════════════════════════════════════════════════════════════════════
-def max_drawdown(r: np.ndarray) -> float:
-    """0-prepended cumulative-R drawdown (gpu_backtest.compute_metrics)."""
-    r = np.asarray(r, dtype=float)
-    if len(r) == 0:
-        return 0.0
-    cum = np.concatenate(([0.0], np.cumsum(r)))
-    return float((np.maximum.accumulate(cum) - cum).max())
-
-
-def profit_factor(r: np.ndarray) -> float:
-    r = np.asarray(r, dtype=float)
-    gp = float(r[r > 0].sum()) if (r > 0).any() else 0.0
-    gl = float(-r[r < 0].sum()) if (r < 0).any() else 0.0
-    return gp / gl if gl > 0 else float("inf")
-
-
-def multiplier_hist(m: np.ndarray) -> pd.DataFrame:
-    """Counts in 0.1-wide bins from 0.5 to 1.5.
-
-    ``np.histogram`` already closes the final bin on the right, so an
-    exactly-1.5 multiplier lands in [1.4, 1.5] once and the counts sum to
-    ``len(m)``.
-    """
-    edges = np.round(np.arange(MAP_OUT_LO, MAP_OUT_HI + 1e-9, 0.1), 2)
-    counts, _ = np.histogram(np.asarray(m, dtype=float), bins=edges)
-    counts = counts.astype(int)
-    return pd.DataFrame({
-        "bin_lo": edges[:-1], "bin_hi": edges[1:], "count": counts,
-        "pct": np.where(len(m), counts / max(len(m), 1) * 100, 0.0),
-    })
-
-
-# ══════════════════════════════════════════════════════════════════════
-# Walk-forward driver
-# ══════════════════════════════════════════════════════════════════════
-def walk_forward(df: pd.DataFrame, features: Sequence[str], model: str,
-                 bar_minutes: int, burn_in_months: int, purge_bars: int,
-                 seed: int, y_train_override: Optional[np.ndarray] = None,
-                 verbose: bool = True) -> Dict:
-    """One full expanding walk-forward pass.
-
-    ``y_train_override`` (full-book length, aligned to ``df``) replaces the
-    TRAINING labels only — this is how the shuffle null is run. Test labels
-    always come from the real ``r_multiple``.
-    """
-    times = df["entry_time"].to_numpy()
-    r_all = df["r_multiple"].to_numpy(float)
-    r_train_src = r_all if y_train_override is None else np.asarray(y_train_override, float)
-    X_all = df[list(features)].apply(pd.to_numeric, errors="coerce").to_numpy(float) \
-        if features else np.zeros((len(df), 0))
-
-    book_start = df["entry_time"].iloc[0]
-    book_end = df["entry_time"].iloc[-1]
-    burn_end = book_start + pd.DateOffset(months=burn_in_months)
-    purge = pd.Timedelta(minutes=purge_bars * bar_minutes)
-
-    qs = [q for q in quarter_starts(book_start, book_end) if q >= burn_end]
-    rows: List[Dict] = []
-    per_q: List[Dict] = []
-    coef_rows: List[Dict] = []
-    skipped: List[str] = []
-    dropped_seen: set = set()
-
-    for q_start in qs:
-        q_end = q_start + pd.DateOffset(months=3)
-        train_mask = df["entry_time"] < (q_start - purge)
-        test_mask = (df["entry_time"] >= q_start) & (df["entry_time"] < q_end)
-        n_tr, n_te = int(train_mask.sum()), int(test_mask.sum())
-        if n_tr < MIN_TRAIN or n_te == 0:
-            skipped.append(f"{quarter_label(q_start)} (train={n_tr}, test={n_te})")
-            continue
-
-        tr = np.flatnonzero(train_mask.to_numpy())
-        te = np.flatnonzero(test_mask.to_numpy())
-        Xtr_raw, Xte_raw = X_all[tr], X_all[te]
-        mean, scale, keep = fit_standardiser(Xtr_raw)
-        dropped = [features[i] for i in np.flatnonzero(~keep)]
-        if verbose and dropped:
-            tag = (quarter_label(q_start), tuple(dropped))
-            if tag not in dropped_seen:
-                dropped_seen.add(tag)
-                say(f"  WARNING {quarter_label(q_start)}: zero variance in train, "
-                    f"dropped from fit (coef 0): {', '.join(dropped)}")
-        Xtr = apply_standardiser(Xtr_raw, mean, scale)[:, keep]
-        Xte = apply_standardiser(Xte_raw, mean, scale)[:, keep]
-
-        r_tr = r_train_src[tr]
-        lam: Optional[float] = None
-        if model == "logistic":
-            y_tr = (r_tr > 0).astype(float)
-            lam, _ = choose_lambda(Xtr, y_tr, "logistic")
-            coef, b, _ = fit_logistic(Xtr, y_tr, lam)
-            z_tr = sigmoid(predict_linear(Xtr, coef, b))
-            z_te = sigmoid(predict_linear(Xte, coef, b))
-        elif model == "lgbm":
-            y_tr = np.clip(r_tr, WINSOR_LO, WINSOR_HI)
-            gbm = fit_lgbm(Xtr, y_tr, seed)
-            z_tr = np.asarray(gbm.predict(Xtr), dtype=float)
-            z_te = np.asarray(gbm.predict(Xte), dtype=float)
-            coef, b = np.zeros(int(keep.sum())), 0.0
-        else:  # ridge / isbuy / exhaustion
-            y_tr = np.clip(r_tr, WINSOR_LO, WINSOR_HI)
-            lam, _ = choose_lambda(Xtr, y_tr, "ridge")
-            coef, b = fit_ridge(Xtr, y_tr, lam)
-            z_tr = predict_linear(Xtr, coef, b)
-            z_te = predict_linear(Xte, coef, b)
-
-        cal = calibrate_map(z_tr)
-        m_te = apply_map(z_te, cal)
-
-        if model != "lgbm":
-            full_coef = np.zeros(len(features))
-            full_coef[np.flatnonzero(keep)] = coef
-            for name, c in zip(features, full_coef):
-                coef_rows.append({"quarter": quarter_label(q_start), "feature": name,
-                                  "coef_std": float(c), "lambda": lam,
-                                  "dropped": name in dropped})
-            coef_rows.append({"quarter": quarter_label(q_start), "feature": "_intercept",
-                              "coef_std": float(b), "lambda": lam, "dropped": False})
-
-        r_te = r_all[te]
-        for i, idx in enumerate(te):
-            rows.append({
-                "entry_time": df["entry_time"].iloc[idx],
-                "r_multiple": float(r_te[i]),
-                "mult": float(m_te[i]),
-                "r_sized": float(r_te[i] * m_te[i]),
-                "quarter": quarter_label(q_start),
-                "z": float(z_te[i]),
-            })
-        per_q.append({
-            "quarter": quarter_label(q_start),
-            "q_start": q_start, "n_train": n_tr, "n_test": n_te,
-            "flat_r": float(r_te.sum()), "sized_r": float((r_te * m_te).sum()),
-            "mean_m": float(m_te.mean()), "lambda": lam,
-            "map_lo": cal["lo"], "map_hi": cal["hi"], "normalizer": cal["normalizer"],
-        })
-
-    oos = pd.DataFrame(rows)
-    return {"oos": oos, "quarters": pd.DataFrame(per_q),
-            "coefs": pd.DataFrame(coef_rows), "skipped": skipped,
-            "quarter_starts": qs}
-
-
-def evaluate(oos: pd.DataFrame) -> Dict:
-    """Headline OOS numbers for one model's walk-forward output."""
-    if oos.empty:
-        return {"n": 0, "flat_r": 0.0, "sized_r": 0.0, "lift": 0.0}
-    r = oos["r_multiple"].to_numpy(float)
-    rs = oos["r_sized"].to_numpy(float)
-    m = oos["mult"].to_numpy(float)
-    med = oos["entry_time"].median()
-    first = (oos["entry_time"] < med).to_numpy()
-    second = ~first
-    return {
-        "n": int(len(oos)),
-        "flat_r": float(r.sum()), "sized_r": float(rs.sum()),
-        "lift": float(rs.sum() - r.sum()),
-        "flat_pf": profit_factor(r), "sized_pf": profit_factor(rs),
-        "flat_dd": max_drawdown(r), "sized_dd": max_drawdown(rs),
-        "mean_m": float(m.mean()), "min_m": float(m.min()), "max_m": float(m.max()),
-        "median_time": med,
-        "n_h1": int(first.sum()), "n_h2": int(second.sum()),
-        "flat_h1": float(r[first].sum()), "flat_h2": float(r[second].sum()),
-        "sized_h1": float(rs[first].sum()), "sized_h2": float(rs[second].sum()),
-    }
-
-
-def acceptance(ev: Dict, real_lift: float, shuffle_p95: float) -> Dict:
-    """The six pre-registered clauses."""
-    c1 = ev["sized_h1"] > ev["flat_h1"] and ev["sized_h2"] > ev["flat_h2"]
-    c2 = ev["sized_r"] > ev["flat_r"]
-    c3 = ev["sized_dd"] <= DD_TOL * ev["flat_dd"]
-    c4 = (ev["sized_pf"] >= PF_FLOOR) and ev["sized_h1"] > 0 and ev["sized_h2"] > 0
-    c5 = real_lift > shuffle_p95 if np.isfinite(shuffle_p95) else False
-    c6 = (abs(ev["mean_m"] - 1.0) <= MEAN_MULT_TOL
-          and ev["min_m"] >= MAP_CLIP_LO - 1e-9 and ev["max_m"] <= MAP_CLIP_HI + 1e-9)
-    clauses = {"1_both_halves": bool(c1), "2_overall": bool(c2), "3_drawdown": bool(c3),
-               "4_pf_and_halves": bool(c4), "5_beats_shuffle": bool(c5),
-               "6_multiplier_bounds": bool(c6)}
-    clauses["overall"] = bool(all(clauses.values()))
-    return clauses
-
-
-def shuffle_null(df: pd.DataFrame, features: Sequence[str], model: str,
-                 bar_minutes: int, burn_in_months: int, purge_bars: int,
-                 n_shuffle: int, seed: int) -> np.ndarray:
-    """Lift distribution under permuted TRAINING labels (test labels real)."""
-    rng = np.random.default_rng(seed)
-    r_all = df["r_multiple"].to_numpy(float)
-    lifts: List[float] = []
-    for _ in range(n_shuffle):
-        r_perm = r_all[rng.permutation(len(r_all))]
-        res = walk_forward(df, features, model, bar_minutes, burn_in_months,
-                           purge_bars, seed, y_train_override=r_perm, verbose=False)
-        ev = evaluate(res["oos"])
-        lifts.append(ev["lift"])
-    return np.asarray(lifts, dtype=float)
-
-
-# ══════════════════════════════════════════════════════════════════════
-# Final fit + artifact
-# ══════════════════════════════════════════════════════════════════════
-def final_fit(df: pd.DataFrame, features: Sequence[str], model: str,
-              seed: int) -> Dict:
-    """Fit on ALL trades; calibrate the map on in-sample predictions."""
-    X_raw = df[list(features)].apply(pd.to_numeric, errors="coerce").to_numpy(float)
-    r = df["r_multiple"].to_numpy(float)
-    mean, scale, keep = fit_standardiser(X_raw)
-    X = apply_standardiser(X_raw, mean, scale)[:, keep]
-    if model == "logistic":
-        y = (r > 0).astype(float)
-        lam, _ = choose_lambda(X, y, "logistic")
-        coef, b, gnorm = fit_logistic(X, y, lam)
-        z = sigmoid(predict_linear(X, coef, b))
-        target = "r_positive"
-    else:
-        y = np.clip(r, WINSOR_LO, WINSOR_HI)
-        lam, _ = choose_lambda(X, y, "ridge")
-        coef, b = fit_ridge(X, y, lam)
-        gnorm = float("nan")
-        z = predict_linear(X, coef, b)
-        target = "r_winsorised"
-    full_coef = np.zeros(len(features))
-    full_coef[np.flatnonzero(keep)] = coef
-    cal = calibrate_map(z)
-    return {"mean": mean, "scale": scale, "keep": keep, "coef": full_coef,
-            "intercept": float(b), "lambda": float(lam), "z": z,
-            "mult": apply_map(z, cal), "map": cal, "target": target,
-            "grad_norm": gnorm}
-
-
-def predict_multiplier(feature_dict: Dict[str, float], artifact: Dict) -> float:
-    """Runtime formula: standardise → linear score → size map → 0.05 grid."""
-    names = artifact["features"]
-    mean = artifact["mean"]
-    scale = artifact["scale"]
-    coef = artifact["coef"]
-    z = float(artifact["intercept"])
-    for i, name in enumerate(names):
-        raw = feature_dict.get(name, mean[i])
-        if raw is None or not np.isfinite(float(raw)):
-            raw = mean[i]
-        z += float(coef[i]) * (float(raw) - float(mean[i])) / float(scale[i])
-    if artifact["model"] == "logistic":
-        z = 1.0 / (1.0 + math.exp(-max(-35.0, min(35.0, z))))
-    mp = artifact["map"]
-    if mp["hi"] <= mp["lo"]:
-        return 1.0
-    frac = min(1.0, max(0.0, (z - mp["lo"]) / (mp["hi"] - mp["lo"])))
-    raw_m = mp["out_lo"] + (mp["out_hi"] - mp["out_lo"]) * frac
-    val = min(mp["clip_hi"], max(mp["clip_lo"], raw_m * mp["normalizer"]))
-    step = mp["round_to"]
-    return float(round(round(val / step) * step, 2))
-
-
-def build_artifact(fit: Dict, features: Sequence[str], model: str, df: pd.DataFrame,
-                   ev: Dict, clauses: Dict, real_lift: float, shuffle: np.ndarray,
-                   n_shuffle: int, book: Path, tf: str, confirm_tfs: List[str],
-                   burn_in_months: int, purge_bars: int, spread: float) -> Dict:
-    return {
-        "schema_version": 1,
-        "feature_version": FEATURE_VERSION,
-        "pair": "XAU/USD",
-        "primary_tf": tf,
-        "confirmation_tfs": confirm_tfs,
-        "model": model,
-        "target": fit["target"],
-        "features": list(features),
-        "mean": [float(v) for v in fit["mean"]],
-        "scale": [float(v) for v in fit["scale"]],
-        "coef": [float(v) for v in fit["coef"]],
-        "intercept": float(fit["intercept"]),
-        "map": {
-            "kind": "clipped_linear_on_quantiles",
-            "lo": float(fit["map"]["lo"]), "hi": float(fit["map"]["hi"]),
-            "out_lo": MAP_OUT_LO, "out_hi": MAP_OUT_HI,
-            "normalizer": float(fit["map"]["normalizer"]),
-            "clip_lo": MAP_CLIP_LO, "clip_hi": MAP_CLIP_HI,
-            "round_to": MAP_ROUND_TO,
-        },
-        "training": {
-            "window": [df["entry_time"].iloc[0].isoformat(),
-                       df["entry_time"].iloc[-1].isoformat()],
-            "spread": spread,
-            "n_train_trades": int(len(df)),
-            "n_oos_trades": int(ev.get("n", 0)),
-            "burn_in_months": burn_in_months,
-            "purge_bars": purge_bars,
-            "refit": "calendar_quarter",
-            "final_fit": "full_window",
-            "lambda": float(fit["lambda"]),
-            "git_commit": git_commit(),
-            "book_sha256": sha256_file(book),
-        },
-        "acceptance": {
-            **{k: v for k, v in clauses.items()},
-            "flat_r": ev.get("flat_r"), "sized_r": ev.get("sized_r"),
-            "sized_pf": ev.get("sized_pf"), "flat_dd": ev.get("flat_dd"),
-            "sized_dd": ev.get("sized_dd"),
-            "sized_h1": ev.get("sized_h1"), "sized_h2": ev.get("sized_h2"),
-            "flat_h1": ev.get("flat_h1"), "flat_h2": ev.get("flat_h2"),
-            "real_lift": float(real_lift),
-            "shuffle_p95_lift": float(np.percentile(shuffle, 95)) if len(shuffle) else None,
-            "n_shuffle": int(n_shuffle),
-            "mean_multiplier": ev.get("mean_m"),
-        },
-    }
-
-
-# ══════════════════════════════════════════════════════════════════════
-# Reporting
-# ══════════════════════════════════════════════════════════════════════
-def fmt(x, nd=2, width=0) -> str:
-    if x is None:
-        return "n/a".rjust(width)
-    if isinstance(x, float) and not np.isfinite(x):
-        return ("inf" if x > 0 else "-inf").rjust(width)
-    s = f"{x:.{nd}f}" if isinstance(x, (int, float, np.floating)) else str(x)
-    return s.rjust(width) if width else s
-
-
-def print_quarters(q: pd.DataFrame) -> None:
-    say(f"{'quarter':<9}{'n_tr':>6}{'n_te':>6}{'flat R':>10}{'sized R':>10}"
-        f"{'mean m':>9}{'lambda':>9}")
-    for _, row in q.iterrows():
-        say(f"{row['quarter']:<9}{int(row['n_train']):>6}{int(row['n_test']):>6}"
-            f"{fmt(row['flat_r'], 2, 10)}{fmt(row['sized_r'], 2, 10)}"
-            f"{fmt(row['mean_m'], 3, 9)}"
-            f"{('n/a' if row['lambda'] is None else fmt(row['lambda'], 0)):>9}")
-
-
-def print_hist(m: np.ndarray) -> None:
-    h = multiplier_hist(m)
-    say(f"{'bin':<12}{'count':>8}{'pct':>8}")
-    for _, row in h.iterrows():
-        bar = "#" * int(round(row["pct"] / 2))
-        say(f"{row['bin_lo']:.1f}–{row['bin_hi']:.1f}{'':<4}{int(row['count']):>8}"
-            f"{row['pct']:>7.1f}%  {bar}")
-
-
-def print_coef_signs(coefs: pd.DataFrame, label: str) -> None:
-    """Mean standardised coefficient + sign stability across quarters."""
-    if coefs.empty:
-        return
-    d = coefs[coefs["feature"] != "_intercept"]
-    say(f"  standardised coefficients across quarters ({label}):")
-    say(f"    {'feature':<24}{'mean':>10}{'min':>10}{'max':>10}{'+q':>5}{'-q':>5}  sign-stable")
-    for name, grp in d.groupby("feature", sort=False):
-        c = grp["coef_std"].to_numpy(float)
-        pos, neg = int((c > 0).sum()), int((c < 0).sum())
-        stable = "yes" if (pos == 0 or neg == 0) and (pos + neg) > 0 else "no"
-        say(f"    {name:<24}{c.mean():>10.4f}{c.min():>10.4f}{c.max():>10.4f}"
-            f"{pos:>5}{neg:>5}  {stable}")
-
-
-def print_comparison(table: List[Dict]) -> None:
-    say(f"{'model':<12}{'n':>6}{'flat R':>10}{'sized R':>10}{'lift':>9}"
-        f"{'PF':>7}{'DD':>8}{'mean m':>8}{'p95 null':>10}{'rank%':>7}  pass")
-    for row in table:
-        say(f"{row['model']:<12}{row['n']:>6}{fmt(row['flat_r'], 2, 10)}"
-            f"{fmt(row['sized_r'], 2, 10)}{fmt(row['lift'], 2, 9)}"
-            f"{fmt(row['sized_pf'], 2, 7)}{fmt(row['sized_dd'], 2, 8)}"
-            f"{fmt(row['mean_m'], 3, 8)}{fmt(row['shuffle_p95'], 2, 10)}"
-            f"{fmt(row['rank_pct'], 0, 7)}  {'PASS' if row['pass'] else 'FAIL'}")
-
-
-# ══════════════════════════════════════════════════════════════════════
-# main
-# ══════════════════════════════════════════════════════════════════════
-def main() -> None:
-    ap = argparse.ArgumentParser(description=__doc__,
-                                 formatter_class=argparse.RawDescriptionHelpFormatter)
-    ap.add_argument("--book", required=True, type=Path)
-    ap.add_argument("--bar-minutes", required=True, type=int)
-    ap.add_argument("--features", required=True,
-                    help="comma-separated f_* column names for the main model")
-    ap.add_argument("--model", default="ridge", choices=("ridge", "logistic"))
-    ap.add_argument("--burn-in-months", type=int, default=12)
-    ap.add_argument("--purge-bars", type=int, default=300)
-    ap.add_argument("--n-shuffle", type=int, default=50)
-    ap.add_argument("--seed", type=int, default=20260905)
-    ap.add_argument("--skip-lgbm", action="store_true")
-    ap.add_argument("--out", type=Path, default=None)
-    ap.add_argument("--write-artifact", type=Path, default=None)
-    ap.add_argument("--tf", default=None, help="primary TF label for the artifact")
-    ap.add_argument("--confirm-tfs", default="15min,1h")
-    ap.add_argument("--spread", type=float, default=0.65,
-                    help="cost the book was produced at (recorded in the artifact)")
-    args = ap.parse_args()
-
-    if args.write_artifact and not args.tf:
-        raise SystemExit("--write-artifact requires --tf")
-
-    say("=" * 96)
-    say("Stage-D stake-sizing walk-forward — RESEARCH ONLY")
-    say("=" * 96)
-    say(f"book            : {args.book}")
-    df = load_book(args.book)
-    say(f"trades          : {len(df)}  "
-        f"({df['entry_time'].iloc[0]} → {df['entry_time'].iloc[-1]})")
-    say(f"bar minutes     : {args.bar_minutes}   purge = {args.purge_bars} bars = "
-        f"{args.purge_bars * args.bar_minutes} min "
-        f"({args.purge_bars * args.bar_minutes / 1440:.2f} d)")
-    say(f"burn-in         : {args.burn_in_months} months  "
-        f"(train-only until {df['entry_time'].iloc[0] + pd.DateOffset(months=args.burn_in_months)})")
-    say(f"model / seed    : {args.model} / {args.seed}   shuffles: {args.n_shuffle}")
-    say(f"winsor / lambda : [{WINSOR_LO}, {WINSOR_HI}] / {list(LAMBDA_GRID)}  "
-        f"CV folds {CV_FOLDS} (time-contiguous)")
-    say(f"map             : m = round_{MAP_ROUND_TO}(clip(norm × (0.5 + clip((z−lo)/(hi−lo),0,1)"
-        f"), {MAP_CLIP_LO}, {MAP_CLIP_HI}))  lo/hi = train q{MAP_Q_LO}/q{MAP_Q_HI}")
-    say("")
-
-    main_feats = resolve_features(df, [f.strip() for f in args.features.split(",") if f.strip()],
-                                  "--features")
-    say(f"main features ({len(main_feats)}): {', '.join(main_feats)}")
-
-    specs: List[Tuple[str, List[str], str]] = [(args.model, main_feats, args.model)]
-    for label, feats in (("isbuy", ISBUY_FEATURES), ("exhaustion", EXHAUSTION_FEATURES)):
-        have = [f for f in feats if f in df.columns]
-        if len(have) != len(feats):
-            say(f"NOTE: {label} baseline missing "
-                f"{', '.join(f for f in feats if f not in df.columns)} — using {len(have)}")
-        if have:
-            specs.append((label, have, "ridge"))
-    use_lgbm = (not args.skip_lgbm) and lgbm_available()
-    if not args.skip_lgbm and not use_lgbm:
-        say("lightgbm not installed — upper bound skipped")
-    if use_lgbm:
-        all_f = [c for c in df.columns if c.startswith("f_")]
-        specs.append(("lgbm", all_f, "lgbm"))
-        say(f"lightgbm upper bound: {len(all_f)} f_* columns")
-    say("")
-
-    results: Dict[str, Dict] = {}
-    table: List[Dict] = []
-    for label, feats, kind in specs:
-        say("-" * 96)
-        say(f"MODEL: {label}   ({kind}, {len(feats)} feature(s))")
-        say("-" * 96)
-        res = walk_forward(df, feats, kind, args.bar_minutes, args.burn_in_months,
-                           args.purge_bars, args.seed, verbose=True)
-        if res["skipped"]:
-            say(f"  skipped quarters: {'; '.join(res['skipped'])}")
-        if res["oos"].empty:
-            say("  no OOS quarters — nothing to evaluate")
-            continue
-        ev = evaluate(res["oos"])
-        n_sh = args.n_shuffle if kind != "lgbm" else min(args.n_shuffle, LGBM_MAX_SHUFFLE)
-        if kind == "lgbm" and n_sh != args.n_shuffle:
-            say(f"  NOTE (protocol deviation, flagged): lgbm shuffle null capped at "
-                f"{n_sh} repeats (never ships)")
-        sh = shuffle_null(df, feats, kind, args.bar_minutes, args.burn_in_months,
-                          args.purge_bars, n_sh, args.seed) if n_sh > 0 else np.array([])
-        p95 = float(np.percentile(sh, 95)) if len(sh) else float("nan")
-        rank = float((sh < ev["lift"]).mean() * 100) if len(sh) else float("nan")
-        cl = acceptance(ev, ev["lift"], p95)
-
-        say(f"  OOS quarters    : {len(res['quarters'])} "
-            f"({res['quarters']['quarter'].iloc[0]} → {res['quarters']['quarter'].iloc[-1]}), "
-            f"{ev['n']} trades")
-        say(f"  flat  R         : {ev['flat_r']:+.2f}   PF {fmt(ev['flat_pf'])}   "
-            f"maxDD {ev['flat_dd']:.2f}")
-        say(f"  sized R         : {ev['sized_r']:+.2f}   PF {fmt(ev['sized_pf'])}   "
-            f"maxDD {ev['sized_dd']:.2f}")
-        say(f"  lift            : {ev['lift']:+.2f} R")
-        say(f"  halves (split {ev['median_time']}): "
-            f"h1 flat {ev['flat_h1']:+.2f} / sized {ev['sized_h1']:+.2f} (n={ev['n_h1']}) | "
-            f"h2 flat {ev['flat_h2']:+.2f} / sized {ev['sized_h2']:+.2f} (n={ev['n_h2']})")
-        say(f"  multiplier      : mean {ev['mean_m']:.3f}  min {ev['min_m']:.2f}  "
-            f"max {ev['max_m']:.2f}")
-        say(f"  shuffle null    : n={len(sh)}  mean {fmt(float(sh.mean()) if len(sh) else None)}  "
-            f"p95 {fmt(p95)}  real-lift percentile rank {fmt(rank, 0)}%")
-        say("  acceptance:")
-        for k, v in cl.items():
-            if k == "overall":
-                continue
-            say(f"    {k:<22} {'PASS' if v else 'FAIL'}")
-        say(f"    {'OVERALL':<22} {'PASS' if cl['overall'] else 'FAIL'}")
-        say("  per-quarter:")
-        print_quarters(res["quarters"])
-        print_coef_signs(res["coefs"], label)
-        say("")
-
-        results[label] = {"res": res, "ev": ev, "shuffle": sh, "p95": p95,
-                          "rank": rank, "clauses": cl, "features": feats, "kind": kind}
-        table.append({"model": label, "n": ev["n"], "flat_r": ev["flat_r"],
-                      "sized_r": ev["sized_r"], "lift": ev["lift"],
-                      "sized_pf": ev["sized_pf"], "sized_dd": ev["sized_dd"],
-                      "mean_m": ev["mean_m"], "shuffle_p95": p95, "rank_pct": rank,
-                      "pass": cl["overall"]})
-
-    if args.model not in results:
-        raise SystemExit("the main model produced no OOS quarters — check burn-in / "
-                         "book window / MIN_TRAIN")
-
-    say("=" * 96)
-    say("COMPARISON")
-    say("=" * 96)
-    print_comparison(table)
-    say("")
-    say(f"OOS multiplier histogram ({args.model}):")
-    print_hist(results[args.model]["res"]["oos"]["mult"].to_numpy())
-    say("")
-
-    main_label = args.model
-    beats = None
-    if main_label in results and "isbuy" in results:
-        beats = results[main_label]["ev"]["lift"] > results["isbuy"]["ev"]["lift"]
-        say(f"beats isbuy baseline: {'yes' if beats else 'no'}  "
-            f"({results[main_label]['ev']['lift']:+.2f} R vs "
-            f"{results['isbuy']['ev']['lift']:+.2f} R)")
-    say(f"OVERALL ACCEPTANCE ({main_label}): "
-        f"{'PASS' if results[main_label]['clauses']['overall'] else 'FAIL'}")
-    say("")
-
-    # ── artifact ──────────────────────────────────────────────────────
-    artifact = None
-    if args.write_artifact:
-        say("-" * 96)
-        say("FINAL FIT (all trades) + ARTIFACT")
-        say("-" * 96)
-        fit = final_fit(df, main_feats, args.model, args.seed)
-        say(f"  lambda (CV on full window): {fit['lambda']:g}")
-        if np.isfinite(fit["grad_norm"]):
-            say(f"  logistic |grad|_inf at convergence: {fit['grad_norm']:.3e}")
-        say(f"  map lo/hi/normaliser: {fit['map']['lo']:.5f} / {fit['map']['hi']:.5f} / "
-            f"{fit['map']['normalizer']:.5f}")
-        artifact = build_artifact(
-            fit, main_feats, args.model, df, results[main_label]["ev"],
-            results[main_label]["clauses"], results[main_label]["ev"]["lift"],
-            results[main_label]["shuffle"], args.n_shuffle, args.book, args.tf,
-            [t.strip() for t in args.confirm_tfs.split(",") if t.strip()],
-            args.burn_in_months, args.purge_bars, args.spread)
-        # Self-check: the runtime formula must reproduce the in-sample map exactly.
-        rows = df[main_feats].to_dict("records")
-        repro = np.array([predict_multiplier(r, artifact) for r in rows])
-        exact = bool(np.array_equal(repro, fit["mult"]))
-        say(f"  predict_multiplier reproduces in-sample multipliers exactly: "
-            f"{'YES' if exact else 'NO'}  (max |diff| "
-            f"{float(np.max(np.abs(repro - fit['mult']))):.3e})")
-        if not exact:
-            raise SystemExit("predict_multiplier does not reproduce the final fit — abort")
-        args.write_artifact.parent.mkdir(parents=True, exist_ok=True)
-        args.write_artifact.write_text(json.dumps(artifact, indent=2, default=str))
-        say(f"  wrote {args.write_artifact}")
-        say("")
-
-    # ── outputs ───────────────────────────────────────────────────────
-    if args.out:
-        out = args.out
-        out.mkdir(parents=True, exist_ok=True)
-        main_res = results[main_label]
-        main_res["res"]["oos"].to_csv(out / "oos_trades.csv", index=False)
-        main_res["res"]["quarters"].to_csv(out / "quarters.csv", index=False)
-        coefs = pd.concat(
-            [r["res"]["coefs"].assign(model=lbl) for lbl, r in results.items()
-             if not r["res"]["coefs"].empty],
-            ignore_index=True) if results else pd.DataFrame()
-        coefs.to_csv(out / "coefs_by_quarter.csv", index=False)
-        pd.DataFrame({
-            "model": np.repeat([lbl for lbl in results], [len(r["shuffle"]) for r in results.values()]),
-            "lift": np.concatenate([r["shuffle"] for r in results.values()])
-            if results else np.array([]),
-        }).to_csv(out / "shuffle_lifts.csv", index=False)
-        multiplier_hist(main_res["res"]["oos"]["mult"].to_numpy()).to_csv(
-            out / "multiplier_hist.csv", index=False)
-        payload = {
-            "book": str(args.book), "book_sha256": sha256_file(args.book),
-            "git_commit": git_commit(), "bar_minutes": args.bar_minutes,
-            "model": args.model, "features": main_feats, "seed": args.seed,
-            "n_shuffle": args.n_shuffle, "burn_in_months": args.burn_in_months,
-            "purge_bars": args.purge_bars,
-            "beats_isbuy": beats,
-            "models": {
-                lbl: {
-                    "kind": r["kind"], "features": r["features"],
-                    "evaluation": {k: (None if isinstance(v, float) and not np.isfinite(v)
-                                       else (str(v) if isinstance(v, pd.Timestamp) else v))
-                                   for k, v in r["ev"].items()},
-                    "shuffle": {"n": int(len(r["shuffle"])),
-                                "mean": float(r["shuffle"].mean()) if len(r["shuffle"]) else None,
-                                "p95": None if not np.isfinite(r["p95"]) else r["p95"],
-                                "real_lift_percentile": None if not np.isfinite(r["rank"])
-                                else r["rank"]},
-                    "acceptance": r["clauses"],
-                } for lbl, r in results.items()
-            },
-            "artifact": artifact,
-        }
-        (out / "results.json").write_text(json.dumps(payload, indent=2, default=str))
-        (out / "summary.txt").write_text("\n".join(_LINES) + "\n")
-        print(f"\nwrote outputs to {out}")
-
-
-if __name__ == "__main__":
-    main()

backtest/research/watch_rule_eval.py

@@ -1,78 +0,0 @@
-"""Evaluate the violent-move WATCH rule (as shipped in violent_watch.py) bar by bar on history.  Analysis tooling only.
-Level per bar from: calendar (high-impact USD within 30 min ahead / 30 min after), clock slots, indicator readings.
-Reports share of bars per level, burst rate next bar / next hour per level (both halves), and hot-hours per day.
-    python backtest/research/watch_rule_eval.py PARQUET BAR_MINUTES
-"""
-import sys
-from pathlib import Path
-import numpy as np, pandas as pd
-ROOT = Path(__file__).resolve().parents[2]; sys.path.insert(0, str(ROOT)); sys.path.insert(0, str(ROOT / "backtest"))
-from gpu_backtest import precompute_indicators
-from config import TechnicalConfig
-
-
-def wilder_atr(df, period=14):
-    pc = df["Close"].shift(1)
-    tr = pd.concat([df["High"] - df["Low"], (df["High"] - pc).abs(), (df["Low"] - pc).abs()], axis=1).max(axis=1)
-    return tr.ewm(alpha=1 / period, min_periods=period, adjust=False).mean()
-
-
-def main():
-    pq, bm = sys.argv[1], int(sys.argv[2])
-    px = pd.read_parquet(pq); px = px[(px.index >= pd.Timestamp("2023-08-05", tz="UTC")) & (px.index < pd.Timestamp("2026-09-05", tz="UTC"))]
-    ic = precompute_indicators(px, TechnicalConfig())
-    sh = lambda a: pd.Series(np.asarray(a, dtype=float), index=px.index).shift(1)
-    rsi, adxs, hist, bbp, kcu, kcl = sh(ic.rsi), sh(ic.adx_slope), sh(ic.macd_hist), sh(ic.bb_position), sh(ic.kc_upper), sh(ic.kc_lower)
-    ef, em, es, et, adx, atr1 = sh(ic.ema_fast), sh(ic.ema_medium), sh(ic.ema_slow), sh(ic.ema_trend), sh(ic.adx), sh(ic.atr)
-    c1 = px["Close"].shift(1); hp = hist.shift(1)
-    atr_prev = wilder_atr(px).shift(1); rng = px["High"] - px["Low"]
-    burst = (rng >= 3.0 * atr_prev).astype(int); H = max(1, 60 // bm)
-    next_hour = burst[::-1].rolling(H, min_periods=1).max()[::-1].astype(int)
-    align = np.sign(c1 - ef) + np.sign(ef - em) + np.sign(em - es) + np.sign(es - et)
-    regime = np.select([(adx >= 25) & (align > 0), (adx >= 25) & (align < 0), adx >= 25, adxs > 3.0], ["trending_up", "trending_down", "trending", "transitioning"], default="ranging")
-    atr_pct = atr1.rolling(50).apply(lambda w: (w[-1] > w).mean() * 100, raw=True); extreme = atr_pct > 80
-    # readings
-    bear = ((bbp < 0).astype(int) + (rsi < 30).astype(int) + (c1 < kcl).astype(int) + ((hist < 0) & (hist < hp) & (hp < 0)).astype(int))
-    bull = ((bbp > 1).astype(int) + (rsi >= 70).astype(int) + (c1 > kcu).astype(int) + ((hist > 0) & (hist > hp) & (hp > 0)).astype(int))
-    neutral = (adxs > 3).astype(int) + extreme.astype(int) + (regime == "transitioning").astype(int)
-    readings = bear + bull + neutral
-    # clock (now = bar close time)
-    ct = px.index + pd.Timedelta(minutes=bm); wd = ct.weekday; hr = ct.hour
-    clock = ((wd <= 4) & np.isin(hr, [12, 13, 14])) | (hr == 1) | ((wd == 6) & np.isin(hr, [22, 23]))
-    # calendar
-    cal = pd.read_parquet(ROOT / "backtest/cache/calendar.parquet"); cal["ts"] = pd.to_datetime(cal.dateline, unit="s", utc=True)
-    hi = np.sort(cal[(cal.currency == "USD") & (cal.impactName.str.lower() == "high")].ts.values.astype("datetime64[ns]"))
-    t = ct.values.astype("datetime64[ns]"); nx = np.searchsorted(hi, t)
-    to_next = np.where(nx < len(hi), (hi[np.minimum(nx, len(hi) - 1)] - t) / np.timedelta64(1, "m"), 1e9)
-    since = np.where(nx - 1 >= 0, (t - hi[np.maximum(nx - 1, 0)]) / np.timedelta64(1, "m"), 1e9)
-    calendar = ((to_next > 0) & (to_next <= 30)) | (since <= 30)
-    # FINAL RULE (2026-09-06, chosen from the variant sweep recorded in sizing_campaign_2026-09.md):
-    #   extreme = price-at-extreme readings only (outside Bollinger, RSI beyond 30/70, outside Keltner)
-    #   hot   = calendar | (clock & extreme >= 1)      -> ~8.6% of bars, next-bar odds 3-5x normal, ~4 hot hours/day
-    #   watch = clock | extreme >= 2                   -> ~2x normal
-    extreme = ((bbp < 0) | (bbp > 1)).astype(int) + ((rsi < 30) | (rsi >= 70)).astype(int) + ((c1 < kcl) | (c1 > kcu)).astype(int)
-    level = np.where(calendar | (clock & (extreme >= 1)), "hot", np.where(clock | (extreme >= 2), "watch", "none"))
-    d = pd.DataFrame({"level": level, "burst": burst.values, "next_hour": next_hour.values, "readings": readings.values, "clock": clock, "calendar": calendar}, index=px.index)
-    d = d[d.index >= pd.Timestamp("2023-09-05", tz="UTC")]
-    half = np.where(np.arange(len(d)) < len(d) // 2, 1, 2)
-    b1, b2 = d.burst[half == 1].mean(), d.burst[half == 2].mean(); h1, h2 = d.next_hour[half == 1].mean(), d.next_hour[half == 2].mean()
-    print(f"{Path(pq).name}: {len(d)} bars; base next-bar {100*d.burst.mean():.2f}%, next-hour {100*d.next_hour.mean():.1f}%")
-    rows = []
-    for lv in ("none", "watch", "hot"):
-        m = d.level == lv
-        rows.append(dict(level=lv, share_of_bars=100 * m.mean(), next_bar_pct=100 * d.burst[m].mean(), lift_h1=d.burst[m & (half == 1)].mean() / b1, lift_h2=d.burst[m & (half == 2)].mean() / b2,
-                         next_hour_pct=100 * d.next_hour[m].mean(), hour_lift_h1=d.next_hour[m & (half == 1)].mean() / h1, hour_lift_h2=d.next_hour[m & (half == 2)].mean() / h2,
-                         share_of_all_bursts=100 * d.burst[m].sum() / d.burst.sum()))
-    print(pd.DataFrame(rows).round(2).to_string(index=False))
-    hot = d[d.level == "hot"]
-    kinds = pd.Series(np.where(hot.calendar, "calendar", "clock+extreme reading"), index=hot.index)
-    print("hot bars by cause:", (100 * kinds.value_counts(normalize=True)).round(1).to_dict())
-    hot_hours = hot.index.floor("h").unique()
-    days = pd.Series(hot_hours.date).value_counts()
-    print(f"distinct HOT hours per calendar day: mean {days.mean():.1f}, median {days.median():.0f}, max {days.max()} (≈ heads-up messages/day before the daily cap)")
-    hb = d.burst.groupby(d.index.floor("h")).max(); hh = pd.Series(1, index=hot_hours)
-    print(f"of all hours with a violent bar, the hour was flagged HOT at its start or during: {100*hb[hb==1].index.isin(hot_hours).mean():.1f}%")
-
-
-if __name__ == "__main__":
-    main()

chart_patterns.py

@@ -97,7 +97,7 @@ def _first_close_beyond(closes: np.ndarray, start: int, level: float,
 
 # ── Double bottom / top ──────────────────────────────────────────────
 
-def _detect_double(o, h, l, c, atr, cfg, bullish: bool) -> Optional[Dict]:
+def _detect_double(h, l, c, atr, cfg, bullish: bool) -> Optional[Dict]:
     w = cfg.cp_swing_window
     ext = l if bullish else h
     idx = _swing_indices(ext, w, find_high=not bullish)
@@ -147,7 +147,7 @@ def _detect_double(o, h, l, c, atr, cfg, bullish: bool) -> Optional[Dict]:
 
 # ── Head & shoulders (top and inverse) ───────────────────────────────
 
-def _detect_hs(o, h, l, c, atr, cfg, bullish: bool) -> Optional[Dict]:
+def _detect_hs(h, l, c, atr, cfg, bullish: bool) -> Optional[Dict]:
     """bullish=True → inverse H&S (three swing LOWS, head deepest)."""
     w = cfg.cp_swing_window
     ext = l if bullish else h
@@ -210,7 +210,7 @@ def _detect_hs(o, h, l, c, atr, cfg, bullish: bool) -> Optional[Dict]:
 
 # ── Flags (bull and bear) ────────────────────────────────────────────
 
-def _detect_flag(o, h, l, c, atr, cfg, bullish: bool) -> Optional[Dict]:
+def _detect_flag(h, l, c, atr, cfg, bullish: bool) -> Optional[Dict]:
     n = len(c)
     lo_r, hi_r = cfg.cp_flag_retrace
     if bullish:
@@ -257,12 +257,12 @@ def _detect_flag(o, h, l, c, atr, cfg, bullish: bool) -> Optional[Dict]:
 # Priority order: most structured / rarest first — if two patterns
 # complete on the same bar, the more specific one names the trade.
 PATTERN_DETECTORS = {
-    "inv_hs":        lambda o, h, l, c, atr, cfg: _detect_hs(o, h, l, c, atr, cfg, True),
+    "inv_hs":        lambda h, l, c, atr, cfg: _detect_hs(h, l, c, atr, cfg, True),
-    "hs_top":        lambda o, h, l, c, atr, cfg: _detect_hs(o, h, l, c, atr, cfg, False),
+    "hs_top":        lambda h, l, c, atr, cfg: _detect_hs(h, l, c, atr, cfg, False),
-    "double_bottom": lambda o, h, l, c, atr, cfg: _detect_double(o, h, l, c, atr, cfg, True),
+    "double_bottom": lambda h, l, c, atr, cfg: _detect_double(h, l, c, atr, cfg, True),
-    "double_top":    lambda o, h, l, c, atr, cfg: _detect_double(o, h, l, c, atr, cfg, False),
+    "double_top":    lambda h, l, c, atr, cfg: _detect_double(h, l, c, atr, cfg, False),
-    "bull_flag":     lambda o, h, l, c, atr, cfg: _detect_flag(o, h, l, c, atr, cfg, True),
+    "bull_flag":     lambda h, l, c, atr, cfg: _detect_flag(h, l, c, atr, cfg, True),
-    "bear_flag":     lambda o, h, l, c, atr, cfg: _detect_flag(o, h, l, c, atr, cfg, False),
+    "bear_flag":     lambda h, l, c, atr, cfg: _detect_flag(h, l, c, atr, cfg, False),
 }
 
 
@@ -276,7 +276,6 @@ def detect_chart_pattern(df: pd.DataFrame, atr: float, cfg) -> Optional[Dict]:
     if atr <= 0 or len(df) < cfg.cp_lookback + 1:
         return None
     tail = df.iloc[-(cfg.cp_lookback + 1):]
-    o = tail["Open"].to_numpy(dtype=np.float64)
     h = tail["High"].to_numpy(dtype=np.float64)
     l = tail["Low"].to_numpy(dtype=np.float64)
     c = tail["Close"].to_numpy(dtype=np.float64)
@@ -284,7 +283,7 @@ def detect_chart_pattern(df: pd.DataFrame, atr: float, cfg) -> Optional[Dict]:
     for name, det in PATTERN_DETECTORS.items():
         if name not in enabled:
             continue
-        hit = det(o, h, l, c, atr, cfg)
+        hit = det(h, l, c, atr, cfg)
         if hit is not None:
             return hit
     return None

config.py

@@ -332,7 +332,6 @@ class PlatformConfig:
 
     symbol: str = XAUUSD
     display_name: str = "XAU/USD"
-    pip_value: float = 0.10
 
     primary_timeframe: Timeframe = Timeframe.H1
     confirmation_timeframes: List[Timeframe] = field(

dashboard_publisher.py

@@ -3,8 +3,9 @@
 XAU/USD static dashboard publisher.
 
 Standalone daemon that scans XAU/USD at candle closes and atomically
-writes `snapshot.json` + `news.json` (plus a copy of
+writes `snapshot.json` + `news.json` (plus copies of the static
-`static/dashboard.html`) into an Apache-served directory. Apache does
+`dashboard.html` / `.css` / `.js` and `news_summary.php`) into an
+Apache-served directory. Apache does
 all web serving — this process never opens a port, so a failure here
 can only ever mean a stale page.
 
@@ -171,8 +172,8 @@ def report_to_webapp_dict(report) -> dict:
 
 
 def event_to_dict(ev) -> dict:
-    """EconomicEvent → dict. surprise/minutes_until/is_high_impact are
+    """EconomicEvent → dict. surprise is a property, so asdict() would
-    properties, so asdict() would skip them — added explicitly.
+    skip it — added explicitly.
     actual/forecast/previous stay strings as published ('228K', '0.3%')."""
     return {
         "time": ev.time.astimezone(timezone.utc).isoformat(),
@@ -183,8 +184,6 @@ def event_to_dict(ev) -> dict:
         "previous": ev.previous,
         "actual": ev.actual,
         "surprise": ev.surprise,
-        "minutes_until": _safe_num(ev.minutes_until),
-        "is_high_impact": ev.is_high_impact,
     }
 
 
@@ -432,10 +431,6 @@ class DashboardPublisher:
         if self.news_payload is not None:
             publish(self.news_payload, os.path.join(self.out_dir, NEWS_FILE))
 
-    def publish_all(self) -> None:
-        self.publish_snapshot()
-        self.publish_news()
-
     def run_cycle(self, tfs: List[Timeframe]) -> None:
         for tf in tfs:  # sequential — see module docstring (cache dedup)
             self.scan_tf(tf)

deploy/README-dashboard.md

@@ -24,8 +24,9 @@ No new Python dependencies are required.
 ## 2. Make the web folder writable
 
 The dashboard publishes straight into the portal docroot `/var/www/trade/`
-(owner decision 2026-08-03) — it adds exactly four files there:
+(owner decision 2026-08-03) — it adds exactly six files there:
-`dashboard.html`, `news_summary.php`, `snapshot.json`, `news.json` (plus the
+`dashboard.html`, `dashboard.css`, `dashboard.js`, `news_summary.php`,
+`snapshot.json`, `news.json` (plus the
 apache-owned `ai/` cache dir from §6). The daemon user must be able to write
 them:
 
@@ -51,7 +52,7 @@ sudo restorecon -R /var/www/trade
 
 ```bash
 venv/bin/python dashboard_publisher.py --once
-ls /var/www/trade    # dashboard.html snapshot.json news.json added next to the portal
+ls /var/www/trade    # dashboard.html/.css/.js news_summary.php snapshot.json news.json added next to the portal
 ```
 
 Then open `http://<server>/dashboard.html` (path relative to however the
@@ -141,8 +142,8 @@ curl -s "http://<server>/news_summary.php?id=$ID"   # instant, from ai/<id>.json
 ## Operational notes
 
 - **Updating**: `git pull` + `systemctl restart gold-dashboard`. The daemon
-  re-copies `static/dashboard.html` and `static/news_summary.php` into the
+  re-copies `static/dashboard.html`, `dashboard.css`, `dashboard.js` and
-  web folder at startup.
+  `news_summary.php` into the web folder at startup.
 - **If the daemon dies**, systemd restarts it in 30 s; worst case the page
   shows stale data with its own "updated N min ago" evidence. Apache and the
   telegram bot are unaffected.
@@ -155,8 +156,5 @@ curl -s "http://<server>/news_summary.php?id=$ID"   # instant, from ai/<id>.json
   DeepSeek-side spend alerting is the true backstop. Kept separate on
   purpose (independence rule: the PHP never imports/reads bot code or
   config.json).
-- **Restart window**: for the first minutes after a daemon restart the new
-  page can meet a pre-upgrade news.json without headline ids — those rows
-  simply render non-clickable until the first news publish.
 - **Credentials**: uses the same env/.env/config.json resolution as the rest
   of the platform (Oanda candles feed; news needs no key; DeepSeek optional).

docs/architecture_diagram.py

@@ -1,190 +0,0 @@
-"""ForexBot2 architecture diagram → docs/architecture.png.
-
-Regenerate after architecture changes:  venv/bin/python docs/architecture_diagram.py
-(matplotlib only — no graphviz on the dev box)."""
-import os
-import matplotlib
-matplotlib.use("Agg")
-import matplotlib.pyplot as plt
-from matplotlib.patches import FancyBboxPatch, FancyArrowPatch
-
-W, H = 100.0, 100.0
-fig, ax = plt.subplots(figsize=(16, 11.5))
-ax.set_xlim(0, W); ax.set_ylim(0, H); ax.axis("off")
-
-INK = "#263238"; MUT = "#607D8B"; ARR = "#455A64"; HAUL = "#7B8FA1"
-
-def box(x, y, w, h, title, lines, fc, ec, title_fs=8.2, body_fs=6.6, lw=1.3):
-    ax.add_patch(FancyBboxPatch((x, y), w, h,
-                 boxstyle="round,pad=0.35,rounding_size=0.9",
-                 facecolor=fc, edgecolor=ec, linewidth=lw, zorder=2))
-    cy = y + h - 1.9
-    ax.text(x + w / 2, cy, title, ha="center", va="center", fontsize=title_fs,
-            fontweight="bold", color=INK, zorder=3)
-    cy -= 2.3
-    for ln in lines:
-        ax.text(x + w / 2, cy, ln, ha="center", va="center", fontsize=body_fs,
-                color=INK, zorder=3)
-        cy -= 1.85
-
-def band(x, y, w, h, label, color):
-    ax.add_patch(FancyBboxPatch((x, y), w, h,
-                 boxstyle="round,pad=0.3,rounding_size=1.2",
-                 facecolor="none", edgecolor=color, linewidth=1.0,
-                 linestyle=(0, (4, 3)), zorder=1))
-    ax.text(x + 0.9, y + h - 1.4, label, ha="left", va="center",
-            fontsize=7.2, fontweight="bold", color=color, zorder=3)
-
-def arrow(p1, p2, color=ARR, lw=1.5, style="-|>", conn="arc3,rad=0.0",
-          label=None, lx=0, ly=0, ls="solid", fs=6.4, rot=0):
-    ax.add_patch(FancyArrowPatch(p1, p2, arrowstyle=style, mutation_scale=11,
-                 color=color, linewidth=lw, linestyle=ls,
-                 connectionstyle=conn, zorder=4))
-    if label:
-        mx, my = (p1[0] + p2[0]) / 2 + lx, (p1[1] + p2[1]) / 2 + ly
-        ax.text(mx, my, label, fontsize=fs, color=MUT, ha="center",
-                va="center", zorder=5, rotation=rot,
-                bbox=dict(boxstyle="round,pad=0.15", fc="white", ec="none", alpha=0.9))
-
-# ── title ────────────────────────────────────────────────────────────
-ax.text(2, 98.4, "ForexBot2 — XAU/USD Signal-Intelligence Platform",
-        fontsize=15, fontweight="bold", color=INK, ha="left")
-ax.text(2, 96.0, "one pipeline, four delivery surfaces · closed bars only · live == backtest parity",
-        fontsize=8.5, color=MUT, ha="left")
-
-EXT_FC, EXT_EC = "#ECEFF1", "#90A4AE"
-DAT_FC, DAT_EC = "#E3F2FD", "#5C9BD1"
-NWS_FC, NWS_EC = "#E0F2F1", "#3D9C8F"
-COR_FC, COR_EC = "#FFFDF5", "#C9A227"
-REP_FC, REP_EC = "#FFE9A8", "#B8860B"
-DEL_FC, DEL_EC = "#E8F5E9", "#5FAF63"
-BTS_FC, BTS_EC = "#EDE7F6", "#8E6BC6"
-
-# ── external services (top band) ─────────────────────────────────────
-band(1, 85.0, 98, 9.4, "EXTERNAL SERVICES", "#78909C")
-box(2.5, 86.2, 15.5, 6.4, "Oanda v20 API",
-    ["candles (live feed)", "practice orders"], EXT_FC, EXT_EC)
-box(19.5, 86.2, 15.5, 6.4, "Twelve Data · Tiingo",
-    ["registered fallbacks", "(Tiingo: tick stream)"], EXT_FC, EXT_EC)
-box(36.5, 86.2, 15.5, 6.4, "Forex Factory",
-    ["economic calendar", "MyFXBook RSS fallback"], EXT_FC, EXT_EC)
-box(53.5, 86.2, 12.5, 6.4, "News RSS ×4",
-    ["gold / macro", "headlines"], EXT_FC, EXT_EC)
-box(67.5, 86.2, 13.5, 6.4, "DeepSeek API",
-    ["V4-Flash LLM", "thinking: disabled"], EXT_FC, EXT_EC)
-box(82.5, 86.2, 15, 6.4, "Telegram API",
-    ["owner bot +", "subscriber channel"], EXT_FC, EXT_EC)
-
-# ── data & context layer ─────────────────────────────────────────────
-box(2.5, 69.8, 33, 11.0, "providers/  (BaseDataProvider)",
-    ["single-flight fetch · TTL+boundary cache",
-     "forming-candle DROP · validate/clean",
-     "paginated fetch_range · optional tick stream",
-     "active feed knob: data_providers.active = oanda"],
-    DAT_FC, DAT_EC)
-box(38.5, 69.8, 33, 11.0, "news_engine.py  +  llm_client.py",
-    ["calendar events + keyword sentiment (gold-inverse)",
-     "DeepSeek enrichment: batch headline scores,",
-     "event gold-direction, market blurb — display-only,",
-     "keyword fallback, daily request cap"],
-    NWS_FC, NWS_EC)
-box(74.5, 69.8, 23, 11.0, "app_config.py",
-    ["credentials & deployment:",
-     "env → config.json → refuse",
-     "(no hardcoded fallbacks)",
-     ".env loaded at import"],
-    "#FAFAFA", "#9E9E9E")
-
-# ── signal core container ────────────────────────────────────────────
-ax.add_patch(FancyBboxPatch((2.5, 40.5), 95, 26.5,
-             boxstyle="round,pad=0.35,rounding_size=1.2",
-             facecolor=COR_FC, edgecolor=COR_EC, linewidth=1.6, zorder=1))
-ax.text(50, 65.2, "SIGNAL CORE — runs identically in live scans and backtests (closed bars only)",
-        ha="center", fontsize=8.4, fontweight="bold", color="#8a6d1a", zorder=3)
-
-box(4.5, 51.8, 28, 12.0, "engine.py — TechnicalEngine",
-    ["9 scored indicators + ATR context:",
-     "EMA · RSI · MACD · BB · ADX · ROC ·",
-     "Keltner · divergence · candlestick",
-     "regime → adaptive weights → composite"],
-    "white", COR_EC)
-box(4.5, 42.0, 28, 8.4, "pair_signals/xauusd.py — factory",
-    ["ALWAYS used: config.json overlay +",
-     "validated per-TF deltas (hour blocks, trail…)"],
-    "white", COR_EC)
-box(35.5, 42.0, 60, 20.7, "signal_aggregator.py — analyze_pair → analyze_from_dataframes",
-    ["multi-TF agreement  (M5→[M15,H1] · M15→[M5,H1] · H1→[M15,H4] · H4→[H1,D1])",
-     "→ news context (warn-only by default) → diminishing-decay penalties",
-     "→ exhaustion hard block → quality grade A+…F",
-     "→ ordered post-grade gate chain (session · XAU SELL vol/hour · regime · M5 noise)",
-     "→ setup generators: XAU ATR (A+/A) · generic swing · XAU_PB · XAU_CP",
-     "→ entry · SL · TP1/TP2 · trailing plan"],
-    "white", COR_EC, body_fs=6.9)
-
-# ── SignalReport hub ─────────────────────────────────────────────────
-box(31, 33.2, 38, 5.2, "SignalReport",
-    ["signal (−2…+2) · grade · trade setup · reasons · news summary"],
-    REP_FC, REP_EC, title_fs=8.8)
-
-# ── delivery surfaces ────────────────────────────────────────────────
-band(1, 13.2, 98, 17.2, "DELIVERY SURFACES", "#4C8F50")
-box(2.5, 15.0, 12, 12.6, "main.py",
-    ["CLI report", "--detail", "--json"], DEL_FC, DEL_EC)
-box(42.5, 15.0, 31, 12.6, "dashboard_publisher.py  →  Apache",
-    ["independent daemon · scans at candle closes",
-     "atomic publish to /var/www/trade:",
-     "dashboard.html · snapshot.json · news.json",
-     "+ news_summary.php — on-demand AI",
-     "gold-impact summary (whitelist · cache · cap)"],
-    DEL_FC, DEL_EC)
-box(76, 15.0, 21.5, 12.6, "telegram_bot.py",
-    ["candle-boundary auto-scans", "quality threshold + dedup",
-     "alerts + LLM context line", "WS level-breach monitor"],
-    DEL_FC, DEL_EC)
-
-# ── backtest / QA ────────────────────────────────────────────────────
-box(2.5, 3.2, 71.8, 8.0, "backtest/gpu_backtest.py — bar-by-bar replay through the SAME analyze_from_dataframes",
-    ["realism (results-critical): next-open entries · raw-path resolution · spread charged · SL-first ties · close-only trail",
-     "CuPy GPU / NumPy fallback · parquet cache · fed by providers.fetch_range (news off)"],
-    BTS_FC, BTS_EC)
-box(77.5, 3.2, 20, 8.0, "test_all.py",
-    ["359 tests · 22 modules", "no network (mocked)"], "#FAFAFA", "#9E9E9E")
-
-# ── arrows: short structural hops (solid dark) ───────────────────────
-arrow((10, 86.2), (14, 81.2), label="OHLCV", lx=-3.4, ly=0.2)          # oanda → providers
-arrow((27, 86.2), (24, 81.2), ls=(0, (3, 2)), lw=1.1)                  # fallbacks → providers
-arrow((44, 86.2), (48, 81.2))                                          # FF → news
-arrow((59.5, 86.2), (56, 81.2))                                        # RSS → news
-arrow((72, 86.2), (62, 81.2), label="LLM", lx=3.4, ly=0.6)             # deepseek → news
-arrow((19, 69.8), (19, 64.2), label="DataFrames (UTC, closed bars)", lx=0.2, ly=0.9)
-arrow((33.5, 57.5), (35.5, 55.5))                                      # engine → aggregator
-arrow((32.5, 46.0), (35.5, 48.0))                                      # factory → aggregator
-arrow((55, 69.8), (55, 63.1), label="warn-only context", lx=0, ly=1.0)
-arrow((86, 69.8), (86, 67.3), ls=(0, (3, 2)), lw=1.1)                  # config → core
-arrow((50, 40.5), (50, 38.8))                                          # core → report
-arrow((36, 34.5), (8.5, 27.95), conn="arc3,rad=0.12")                  # report → cli
-arrow((60, 33.2), (58, 27.95))                                         # report → dashboard
-arrow((66, 34.5), (86.5, 27.95), conn="arc3,rad=-0.12")                # report → telegram
-
-# ── long-haul edges (thin dashed, routed through gutters) ────────────
-# telegram bot ↔ Telegram API — right-margin arc, clear of app_config
-arrow((95, 86.2), (95, 27.95), conn="arc3,rad=0.15", color=HAUL,
-      lw=1.2, ls=(0, (4, 2)))
-ax.text(99.2, 57, "alerts / commands", fontsize=6.4, color=MUT,
-        rotation=90, ha="center", va="center")
-# browser click → news_summary.php → DeepSeek
-arrow((66, 27.95), (73, 86.2), conn="arc3,rad=-0.18", color=HAUL,
-      lw=1.2, ls=(0, (4, 2)),
-      label="on-demand AI summary\n(browser click → PHP)", lx=2.5, ly=11.6)
-# providers.fetch_range → backtester — far-left margin channel
-arrow((1.5, 69.8), (1.5, 11.0), color=BTS_EC, lw=1.1, ls=(0, (3, 2)),
-      label="fetch_range\n(history)", lx=3.6, ly=-1.2)
-# backtester replays the signal core — right gutter between boxes
-arrow((74.0, 11.6), (74.9, 40.3), color=HAUL, lw=1.3)
-ax.text(81.8, 38.9, "replays the exact signal code", fontsize=6.4,
-        color=MUT, ha="center", va="center", zorder=5,
-        bbox=dict(boxstyle="round,pad=0.15", fc="white", ec="none", alpha=0.9))
-
-out = os.path.join(os.path.dirname(os.path.abspath(__file__)), "architecture.png")
-fig.savefig(out, dpi=200, bbox_inches="tight", facecolor="white")
-print(f"saved {out}")

docs/superpowers/specs/2026-07-21-centralized-config-design.md

@@ -1,154 +0,0 @@
-# Centralized Configuration (config.json) — Design
-
-**Date:** 2026-07-21
-**Status:** Approved (user confirmed all four scoping decisions)
-
-## Goal
-
-Move deployment settings — Telegram credentials, data-provider API keys and
-selection, scanned pairs, scan timeframes, per-pair base blocked hours, alert
-quality boot default, log level — into a single gitignored `config.json` at the
-repo root, loaded once at startup.
-
-**Hard constraint:** zero change to signal logic, indicator weights, thresholds,
-or gates. With an unedited `config.json` (or none at all), every produced
-`PlatformConfig` must be byte-identical to today's factory output. This is
-test-enforced (see Testing).
-
-## Scoping decisions (user-confirmed 2026-07-21)
-
-1. **Secrets live in config.json** (gitignored). A committed
-   `config.json.example` carries placeholders. Env vars still override.
-   The hardcoded credential fallbacks in `config.py` and `telegram_bot.py`
-   are **deleted** — on a machine with neither config.json nor env vars, the
-   bot refuses to start with an explicit error instead of silently using
-   committed credentials. (Recommend rotating the Telegram token and Twelve
-   Data key after merge; both are exposed in git history.)
-2. **Multi-provider is structure only**: a `data_providers` section with named
-   provider blocks and an `active` selector. Only Twelve Data is implemented.
-   No failover logic (nothing to fail over to yet).
-3. **Blocked hours: base list per pair only.** The validated per-timeframe
-   deltas (M5 appends hour 9; M15 SELL-block 15/17 UTC) stay in
-   `pair_signals/xauusd.py` next to their validation evidence. The overlay is
-   applied **before** those deltas and updates **both** mirrored copies
-   (`xau_strategy.blocked_hours_utc` and `backtest.blocked_hours_utc`) so they
-   cannot drift.
-4. **Extra keys included:** scan timeframes, alert min quality (boot default),
-   log level. Everything else (indicator params, RR geometry, M5 gates,
-   CP knobs, news config) deliberately stays code-only.
-
-## File schema
-
-`config.json` (gitignored; `config.json.example` committed with placeholders
-for secrets and today's tuned values for everything else):
-
-```json
-{
-  "telegram": {
-    "token": "<bot token>",
-    "chat_id": "<owner chat id>",
-    "channel_id": "<-100… or empty string to disable the channel feed>",
-    "alert_min_quality": "C"
-  },
-  "data_providers": {
-    "active": "twelve_data",
-    "twelve_data": { "api_key": "<key>" }
-  },
-  "scanning": {
-    "pairs": ["XAU/USD"],
-    "timeframes": ["M5", "M15", "H1", "H4"]
-  },
-  "pairs": {
-    "XAU/USD": { "blocked_hours_utc": [0, 1, 2, 10, 11, 13, 16, 21] }
-  },
-  "log_level": "INFO"
-}
-```
-
-Timeframe strings use the `Timeframe` enum names (`M5`, `M15`, `H1`, `H4`, …).
-Pair keys use canonical symbols (`XAU/USD`).
-
-## Architecture
-
-### New module: `app_config.py`
-
-Single responsibility: read + validate `config.json`, expose typed accessors,
-apply the pair-level overlay. No signal knowledge.
-
-- `AppConfig` dataclass: `telegram` (token, chat_id, channel_id,
-  alert_min_quality), `data_providers` (active name + per-provider dicts),
-  `scan_pairs`, `scan_timeframes`, `pair_overrides`
-  (symbol → {blocked_hours_utc}), `log_level`.
-- `load_app_config(path=REPO_ROOT/"config.json") -> AppConfig` — module-level
-  cached (loaded once per process). Missing file → `AppConfig` of pure
-  defaults + one warning log. Malformed JSON, unknown top-level/section keys,
-  unknown pair symbols, hours outside 0–23, unknown timeframe names, unknown
-  `active` provider → raise `AppConfigError` with a named message (fail fast
-  at boot).
-- Precedence, applied inside the accessors:
-  **env var → config.json → code default**. Secrets (`TELEGRAM_TOKEN`,
-  `TELEGRAM_CHAT_ID`, `TELEGRAM_CHANNEL_ID`, `TWELVE_DATA_API_KEY`) have **no
-  code default** after this change.
-- `apply_pair_overrides(cfg: PlatformConfig, app: AppConfig) -> PlatformConfig`
-  — sets the pair's base `blocked_hours_utc` on **both**
-  `cfg.xau_strategy.blocked_hours_utc` and `cfg.backtest.blocked_hours_utc`,
-  plus `cfg.scan_pairs` / `cfg.scan_timeframes`. Called by the pair factory
-  (see below). A no-op when the pair has no override block.
-
-### Wiring points (all consumers)
-
-| Consumer | Change |
-|---|---|
-| `pair_signals/xauusd.py` | `make_config` calls `apply_pair_overrides` **before** the per-TF deltas (so M5 still appends hour 9 to whatever base the JSON sets, and mirrors stay in sync). |
-| `data_provider.py` | API key resolution becomes `app_config` accessor for the **active** provider (env `TWELVE_DATA_API_KEY` still wins). `config.twelve_data_api_key` field and its hardcoded default are deleted. Missing key keeps today's behavior: provider constructed with `_td=None`, errors on first fetch. |
-| `telegram_bot.py` | `TOKEN`, `CHAT_ID`, `CHANNEL_ID` come from `app_config` (hardcoded fallbacks deleted; missing token/chat_id → explicit startup error). `_alert_min_quality` boot default and `logging` level come from `app_config`. `SCAN_PAIRS` (today built from a bare `PlatformConfig()`) reads `app_config.scan_pairs` directly; `apply_pair_overrides` also stamps `cfg.scan_pairs`/`cfg.scan_timeframes` so the dataclass field stays truthful for any other reader. |
-| `main.py` | No structural change — it uses the factory, which now applies the overlay. |
-| `backtest/gpu_backtest.py` | No structural change — it uses the factory too, so an edited base blocked-hours list flows into replays identically to live (parity by construction). Data fetches pick up the provider key via `data_provider.py`. |
-| `config.py` | `twelve_data_api_key` field removed. Everything else untouched. |
-
-### Multi-provider structure
-
-`data_provider.py` gains a minimal registry:
-`PROVIDER_REGISTRY = {"twelve_data": ForexDataProvider}`. `active` selects
-from it; an unknown name fails at load. Adding a provider later = one config
-block + one adapter class registered here. No shared base class extracted now
-(YAGNI — with one provider there is nothing to abstract over yet).
-
-## Error handling summary
-
-| Condition | Behavior |
-|---|---|
-| No config.json | Defaults + warning. Secrets must come from env or startup fails (bot) / fetch fails (provider). |
-| Malformed JSON / bad values / unknown keys | `AppConfigError` at load, process exits with named message. |
-| Missing Telegram token or chat_id at bot boot | Explicit startup error naming the missing key and both places it can be set. |
-| Missing data-provider key | Same as today: provider builds, first fetch errors. |
-
-## Migration
-
-- Create the local `config.json` populated with the **current live values**
-  (the ones being deleted from source) so the running bot keeps working
-  unchanged. `.gitignore` gains `config.json`.
-- Commit `config.json.example` with placeholders.
-- CLAUDE.md: update the env-var table, repository map, and config section.
-- Post-merge recommendation (user action): rotate the Telegram token and
-  Twelve Data key, since both are in git history.
-
-## Testing (`test_all.py`, new module)
-
-1. **Parity pin (the critical one):** a config.json containing the example
-   defaults, and no config.json at all, both produce factory output identical
-   to today's (full dataclass equality across all four TFs).
-2. Overlay mapping: JSON blocked hours land on **both** mirrored fields; M5
-   factory still appends hour 9 on top of a custom base.
-3. Precedence: env var beats config.json beats default, per key.
-4. Validation: malformed JSON, bad hour, unknown pair/timeframe/provider each
-   raise `AppConfigError`.
-5. Missing-file behavior: defaults + no crash.
-6. Existing factory-equivalence test stays green (equivalence is now defined
-   over factory-with-overlay vs inline-config-with-overlay).
-
-## Out of scope
-
-Hot reload (config is read once at boot), a second concrete provider,
-failover, moving any tuned strategy value beyond the base blocked-hours list,
-news/indicator/geometry settings in JSON.

engine.py

@@ -153,7 +153,7 @@ class TechnicalEngine:
         if "candlestick" not in disabled: self._detect_candlestick_patterns(df, result)
 
         # ── Classify market regime ─────────────────────────────────
-        self._classify_regime(df, result)
+        self._classify_regime(result)
 
         # ── Regime-aware RSI reinterpretation ──────────────────────
         if self.config.rsi_mode == "regime_aware" and "rsi" not in disabled:
@@ -793,8 +793,6 @@ class TechnicalEngine:
         result.key_levels["kc_upper"] = float(kc_upper_val)
         result.key_levels["kc_lower"] = float(kc_lower_val)
         result.key_levels["kc_mid"] = float(kc_mid_val)
-        result._kc_squeeze = is_squeeze  # type: ignore[attr-defined]
-        result._kc_squeeze_released = squeeze_released  # type: ignore[attr-defined]
 
     def _detect_divergences(self, df: pd.DataFrame, result: TechnicalAnalysis):
         """
@@ -875,7 +873,7 @@ class TechnicalEngine:
     def _compute_structure_sr(self, df: pd.DataFrame, result: TechnicalAnalysis):
         """Detect swing high/low structure levels for S/R and trade setup."""
         base_lookback = self.config.swing_sr_lookback
-        if result.timeframe in ("1min", "5min"):
+        if result.timeframe == "5min":
             base_lookback = max(base_lookback, 120)
         lookback = min(base_lookback, len(df) - 1)
         if lookback < 10:
@@ -1098,7 +1096,7 @@ class TechnicalEngine:
             confidence=conf, description=desc, weight=rsi_ind.weight,
         )
 
-    def _classify_regime(self, df: pd.DataFrame, result: TechnicalAnalysis):
+    def _classify_regime(self, result: TechnicalAnalysis):
         """Classify the market regime (trending vs ranging vs transitioning)."""
         adx_sig = result.indicators.get("adx")
         ema_sig = result.indicators.get("ema_trend")

llm_client.py

@@ -198,10 +198,3 @@ def get_llm_client() -> DeepSeekClient:
                 enabled=settings.get("enabled", True) is not False,
             )
         return _client
-
-
-def reset_llm_client() -> None:
-    """Drop the singleton (tests / config reloads)."""
-    global _client
-    with _client_lock:
-        _client = None

llm_provider_switch_plan.txt

@@ -1,252 +0,0 @@
-================================================================================
-MULTI-PROVIDER LLM LAYER - PLAN
-(DeepSeek for news summary, Anthropic Claude for agent panel)
-Written 2026-08-08. Plan only - no code has been changed.
-================================================================================
-
-WHY
----
-Today every LLM feature (news enrichment + agent panel) goes through one
-hardwired client: llm_client.py -> DeepSeekClient, a module singleton reached
-via get_llm_client(). This plan describes how to route different features to
-different providers - e.g. keep DeepSeek for cheap high-volume news scoring,
-put the agent panel on Anthropic Claude for better reasoning.
-
-The repo already solved this problem once for market data: the providers/
-package (BaseDataProvider owns shared machinery, subclasses implement only
-_fetch_raw, a registry + one config knob selects the provider). This plan
-mirrors that proven pattern at the LLM layer, scaled down to a single module.
-
-BLAST RADIUS (verified by grep)
--------------------------------
-get_llm_client() call sites:
-  - news_engine.py ....... 4 sites
-  - agent_panel.py ....... 3 sites (+1 timeout getattr)
-  - telegram_bot.py ...... 1 site (/status health; reads private _count,
-                           _count_day, _daily_cap fields)
-  - test_all.py .......... 41 locations stub via  lc._client = _StubLLM(...)
-Out of scope: static/news_summary.php (own PHP DeepSeek call + key file,
-separate deployment).
-
-DESIGN INVARIANTS (carried over unchanged)
-------------------------------------------
-1. Fail-safe contract: every call returns Optional[Dict]; any failure -> None;
-   callers keep their non-LLM fallbacks; LLM output never gates signals.
-2. Single attempt, no retries (news cache cadence is the retry); per-provider
-   UTC daily cap; warn-once logging; health stamps (last_ok, last_error).
-3. Static system prompts stay byte-identical constants - provider-neutral,
-   and they map to prompt caching on BOTH providers (DeepSeek prefix cache;
-   Anthropic cache_control ephemeral).
-4. Test compatibility: all 41 existing lc._client stub sites keep working
-   with zero edits.
-
-================================================================================
-1. llm_client.py REFACTOR  (~195 -> ~430 lines, no new files)
-================================================================================
-
-1a. BaseLLMClient - shared machinery (extracted from today's DeepSeekClient)
-    - __init__(api_key, model, timeout, daily_cap, enabled)
-    - public: name ("deepseek"/"anthropic"), timeout property,
-      usage() -> (used_today, cap)   # so telegram_bot stops reading privates
-    - owns: available property, _warn_once, _under_cap() (UTC-day counter,
-      lock), health stamps, and the template method:
-
-        def chat_json(self, system_prompt, user_payload):
-            if not self.available or not self._under_cap():
-                return None
-            try:
-                return self._request(system_prompt, user_payload)  # hook
-            except <provider-agnostic error mapping>:
-                # stamps last_error, returns None
-
-    - subclasses implement ONLY _request(system_prompt, payload) -> dict|None.
-
-1b. DeepSeekClient(BaseLLMClient) - behavior-identical move
-    Current requests-based logic relocates into _request: same URL,
-    thinking:disabled, response_format:json_object, temperature 0,
-    max_tokens 2000, sort_keys=True payload, _parse_response.
-    Review criterion: Module 19 tests pass UNMODIFIED (they patch
-    client._session, which stays).
-
-1c. AnthropicClient(BaseLLMClient) - new, official SDK
-    - Official `anthropic` Python SDK (add to requirements.txt), NOT raw
-      requests. LAZY IMPORT inside __init__: package missing -> warn-once +
-      client unavailable, so DeepSeek-only deployments never install it.
-    - Constructor: anthropic.Anthropic(api_key=..., timeout=<configured>,
-      max_retries=0)   # house rule: single attempt
-    - _request:
-        resp = self._client.messages.create(
-            model=self._model,                  # default "claude-opus-5"
-            max_tokens=4000,                    # covers thinking + JSON reply
-            output_config={"effort": "low"},    # cheap structured takes
-            system=[{"type": "text", "text": system_prompt,
-                     "cache_control": {"type": "ephemeral"}}],
-            messages=[{"role": "user",
-                       "content": json.dumps(user_payload, sort_keys=True)}],
-        )
-        if resp.stop_reason == "refusal":       # Opus 5: HTTP 200, check first
-            return None
-        text = first text block in resp.content
-        return self._parse_json(text)   # tolerant: strip ```json fences,
-                                        # json.loads, dict check
-    - DELIBERATELY ABSENT (would 400 on Claude Opus 5): temperature (removed
-      on Opus 5 - do NOT copy from the DeepSeek client), budget_tokens,
-      prefills, response_format.
-    - Error mapping: APIConnectionError -> "network",
-      APIStatusError -> "http_<code>", parse fail -> "parse"
-      (same last_error kinds the /status display already understands).
-    - Defaults: model claude-opus-5 (config-overridable, e.g. claude-haiku-4-5
-      for ~5x cheaper), timeout 30s (Claude slower than V4-Flash), own
-      daily_request_cap 500.
-
-1d. Registry + per-consumer routing
-        _PROVIDERS = {"deepseek": DeepSeekClient, "anthropic": AnthropicClient}
-        LLM_CONSUMERS = ("news", "agent_panel")
-        _clients = {}     # keyed by PROVIDER name -> two consumers on the
-                          # same provider share one instance and one cap
-        _client = None    # kept: GLOBAL TEST OVERRIDE
-
-        def get_llm_client(consumer="news"):
-            if _client is not None:          # tests: lc._client = stub
-                return _client               #   -> shadows ALL consumers
-            name = get_app_config().llm_routing.get(consumer, "deepseek")
-            create-once per provider, return _clients[name]
-
-        def reset_llm_client():              # clears _client AND _clients
-
-    Keeping _client as a documented test-override global is what makes the
-    41 existing stub sites work untouched.
-
-================================================================================
-2. CONFIG  (app_config.py + config.json.example)
-================================================================================
-- New top-level "anthropic" block, mirroring "deepseek":
-    {api_key, model, enabled, timeout_seconds, daily_request_cap}
-  plus anthropic_api_key() resolver: ANTHROPIC_API_KEY env -> config.json -> "".
-- New top-level "llm" ROUTING block:
-    {"news": "deepseek", "agent_panel": "deepseek"}
-  Validation: keys within {news, agent_panel}; values within registered
-  provider names. Absent block -> {} -> everything on DeepSeek (today's
-  behavior; zero-migration default). AppConfig.llm_routing: Dict[str, str].
-
-- The target state is then PURE CONFIG - no code changes on switch day:
-
-    "llm":       { "news": "deepseek", "agent_panel": "anthropic" },
-    "anthropic": { "api_key": "", "model": "claude-opus-5", "enabled": true,
-                   "timeout_seconds": 30, "daily_request_cap": 500 }
-
-================================================================================
-3. CONSUMER EDITS  (mechanical, ~10 lines total)
-================================================================================
-- news_engine.py:  4 call sites -> get_llm_client("news")
-- agent_panel.py:  3 call sites -> get_llm_client("agent_panel");
-                   getattr(client, "_timeout", ...) -> public client.timeout
-- telegram_bot.py _news_health_lines: iterate the DISTINCT providers actually
-  routed (news + panel); per provider print "<name>: used/cap calls today"
-  via the new usage() accessor instead of private fields. DeepSeek keeps its
-  "headlines scored" extra; cap/error warnings work off base-class stamps.
-- NO changes to prompts, payload builders, caches, or the panel's own
-  max_panels_per_day budget - all provider-agnostic already.
-
-================================================================================
-4. TESTS  (new "Module 24: LLM provider routing", ~12 tests, ~200 lines)
-================================================================================
-Reuse _StubLLM; new _FakeAnthropic (fake messages.create returning a stub
-response object) injected as client._client.
-
- 1. app_config: anthropic block parses/validates/defaults empty.
- 2. app_config: llm routing block - valid map round-trips; unknown consumer
-    key, unknown provider name, non-dict -> AppConfigError; absent -> {}.
- 3. anthropic_api_key resolves env -> config.json -> empty.
- 4. Base contract on BOTH subclasses (parametrized): keyless/disabled -> no
-    call; daily cap blocks + UTC-roll reset; failure stamps last_error,
-    success stamps last_ok.
- 5. AnthropicClient parse: clean JSON -> dict; ```json-fenced -> dict;
-    non-JSON -> None; stop_reason "refusal" -> None; no text block -> None.
- 6. AnthropicClient request shape: NO temperature key, max_retries=0,
-    cache_control present on system block (assert on captured kwargs).
- 7. AnthropicClient with `anthropic` package missing (simulated) ->
-    available False, chat_json None, no crash at import.
- 8. Routing: no llm block -> both consumers get the SAME DeepSeek singleton
-    (shared cap); routed config -> news gets DeepSeekClient while
-    agent_panel gets AnthropicClient; same-provider consumers share one
-    instance.
- 9. Global lc._client override shadows BOTH consumers (locks the 41 legacy
-    stub sites' semantics).
-10. reset_llm_client() clears both the override and the per-provider cache.
-11. End-to-end: routing agent_panel->anthropic, attach_panel reaches the
-    anthropic fake (5 calls) while a news call reaches the deepseek stub.
-12. telegram _news_health_lines renders per-provider lines from usage()
-    without touching private fields.
-
-GATE: Modules 19 and 23 must pass UNMODIFIED - the no-regression proof for
-the refactor.
-
-================================================================================
-5. DOCS
-================================================================================
-- CLAUDE.md: llm_client bullet -> provider-abstracted (BaseLLMClient;
-  deepseek + anthropic registered); config.json "llm" block routes consumers;
-  absent = all-DeepSeek. Add ANTHROPIC_API_KEY to env-var list. Test counts.
-- config.json.example: both new blocks.
-
-================================================================================
-COMMIT SEQUENCE  (each commit green)
-================================================================================
-1. llm_client: extract BaseLLMClient; DeepSeekClient unchanged behavior
-   (+ base-contract tests)  - pure refactor, Modules 19/23 green untouched.
-2. config: anthropic block + llm consumer-routing block (+ config tests).
-3. llm_client: AnthropicClient (official SDK, lazy import) + requirements.txt
-   (+ parse/shape tests).
-4. routing: per-consumer get_llm_client + health lines - consumer call-site
-   edits, telegram usage() lines, routing/e2e tests.
-5. docs: CLAUDE.md + example config.
-
-================================================================================
-VERIFICATION
-================================================================================
-- python test_all.py green at every commit (~374 -> ~386).
-- Live smoke A (no behavior change): default config, main.py --panel ->
-  still DeepSeek, /status unchanged.
-- Live smoke B (the goal): llm.agent_panel="anthropic" + real
-  ANTHROPIC_API_KEY, run --panel -> one real Claude panel (5 requests,
-  validates JSON contract + latency + refusal-free path) while a news
-  refresh still hits DeepSeek; /status shows two provider lines.
-- Backtest untouched by construction (no LLM in replay path); one 30-day
-  run as a formality.
-
-================================================================================
-EFFORT ESTIMATE
-================================================================================
-0 new files, 8 modified, ~520 lines, est. 3-5 hours.
-
-  llm_client.py        base class + AnthropicClient + routing   +235
-  test_all.py          Module 24                                +200
-  app_config.py        anthropic + llm blocks, resolver          +55
-  telegram_bot.py      per-provider health lines                 +15
-  news_engine.py       consumer tags on call sites                +4
-  agent_panel.py       consumer tags on call sites                +4
-  config.json.example  blocks                                    +14
-  requirements.txt     anthropic                                  +1
-  CLAUDE.md            docs                                      +10
-
-================================================================================
-KEY DECISIONS EMBEDDED IN THIS PLAN
-================================================================================
-- Official Anthropic SDK, not raw requests (current API guidance for Python;
-  lazy import keeps it optional for DeepSeek-only installs).
-- Default Claude model: claude-opus-5 ($5/$25 per MTok; a full 5-call panel
-  ~ 8k in / 500 out ~ $0.05). Config-overridable to claude-haiku-4-5.
-- Two Claude gotchas handled: temperature is REMOVED on Opus 5 (sending it
-  400s - must not be copied from the DeepSeek request body), and refusals
-  arrive as HTTP 200 with stop_reason "refusal" (checked before parsing,
-  mapped to the normal None/fallback path).
-- Caps are per-provider, not per-consumer: two consumers on one provider
-  share its cap (today's semantics); splitting across two providers doubles
-  total headroom naturally.
-- The panel's own max_panels_per_day budget is untouched - it sits above
-  whichever provider serves it.
-- Not in scope: static/news_summary.php (own PHP client/key), streaming,
-  tool use, per-consumer model overrides (add a "model" key per consumer
-  later if ever needed).
-================================================================================

main.py

@@ -103,8 +103,8 @@ def _compact_reason(reason: str) -> str:
 
 
 def _fmt_price(price: float) -> str:
-    """2dp for gold-scale prices, 5dp for sub-10 FX-scale prices."""
+    """Gold-scale price, 2 dp."""
-    return f"{price:.2f}" if price >= 10 else f"{price:.5f}"
+    return f"{price:.2f}"
 
 
 def print_signal_report(report: SignalReport, detail: bool = False):
@@ -308,7 +308,7 @@ Examples:
     parser.add_argument("--pair", "-p", default="XAUUSD",
                         help="Pair to analyze: XAUUSD (default: XAUUSD)")
     parser.add_argument("--timeframe", "-tf", default="H1",
-                        choices=["M5", "M15", "M30", "H1", "H4", "D1"],
+                        choices=list(_TF_BY_NAME),
                         help="Primary analysis timeframe (default: H1)")
     parser.add_argument("--detail", action="store_true",
                         help="Show full indicator breakdown")
@@ -326,6 +326,14 @@ Examples:
     return parser.parse_args()
 
 
+# The four validated primary timeframes (the same set the bot and the
+# dashboard scan). M30/D1 were never validated as primaries.
+_TF_BY_NAME = {
+    "M5": Timeframe.M5, "M15": Timeframe.M15,
+    "H1": Timeframe.H1, "H4": Timeframe.H4,
+}
+
+
 def _resolve_pair(s: str) -> str:
     aliases = {
         "XAUUSD": XAUUSD, "XAU/USD": XAUUSD, "GOLD": XAUUSD,
@@ -337,35 +345,15 @@ def _resolve_pair(s: str) -> str:
 
 
 def build_config(args) -> PlatformConfig:
+    """Per-pair tuned config from the pair_signals factory.
+
+    The factory owns the validated confirmation map (M5->[M15,H1],
+    M15->[M5,H1], H1->[M15,H4], H4->[H1,D1]); passing None here keeps
+    the CLI on the same map as the bot and the dashboard. A CLI-local
+    table once paired M5 with the unvalidated [M1, M15].
+    """
     pair = _resolve_pair(args.pair)
-
+    return PAIR_PRODUCERS[pair](_TF_BY_NAME[args.timeframe], None)
-    tf_map = {
-        "M5": Timeframe.M5, "M15": Timeframe.M15, "M30": Timeframe.M30,
-        "H1": Timeframe.H1, "H4": Timeframe.H4, "D1": Timeframe.D1,
-    }
-    primary_tf = tf_map[args.timeframe]
-
-    tf_confirmations = {
-        "M5":  [Timeframe.M1,  Timeframe.M15],
-        "M15": [Timeframe.M5,  Timeframe.H1],
-        "M30": [Timeframe.M15, Timeframe.H1],
-        "H1":  [Timeframe.M15, Timeframe.H4],
-        "H4":  [Timeframe.H1,  Timeframe.D1],
-        "D1":  [Timeframe.H4,  Timeframe.W1],
-    }
-    confirmation_tfs = tf_confirmations.get(
-        args.timeframe, [Timeframe.M15, Timeframe.H4]
-    )
-
-    # Per-pair tuned configs from the pair_signals factories.
-    factory = PAIR_PRODUCERS.get(pair)
-    if factory is not None:
-        return factory(primary_tf, confirmation_tfs)
-
-    config = PlatformConfig()
-    config.primary_timeframe = primary_tf
-    config.confirmation_timeframes = confirmation_tfs
-    return config
 
 
 def run_live(args, config: PlatformConfig):

market_outlook.py

@@ -329,7 +329,6 @@ def _collect_payload(settings: Dict[str, Any]) -> Dict[str, Any]:
 # ── Orchestration, hourly dedup, and request budget ──────────────────
 
 _state_lock = threading.Lock()
-_last_run_ts: Optional[float] = None
 _last_run_hour: Optional[int] = None   # UTC hour bucket of the last attempt
 _outlook_day: Optional[_dt.date] = None
 _outlook_count = 0
@@ -344,10 +343,9 @@ def _skip(reason: str) -> None:
 
 def reset_outlook_state() -> None:
     """Drop hourly and daily state (tests / deployment config reloads)."""
-    global _last_run_ts, _last_run_hour, _outlook_day, _outlook_count
+    global _last_run_hour, _outlook_day, _outlook_count
     global _warned_budget
     with _state_lock:
-        _last_run_ts = None
         _last_run_hour = None
         _outlook_day = None
         _outlook_count = 0
@@ -360,7 +358,7 @@ def _begin_attempt(cap: int, force: bool) -> bool:
     Hourly dedup is by UTC hour BUCKET, not ``now - last < 3600``: the
     H1 loop calls this at boundary+10 s+scan time, so consecutive hours
     were routinely 3,590 s apart and every other hour was dropped."""
-    global _last_run_ts, _last_run_hour, _outlook_day, _outlook_count
+    global _last_run_hour, _outlook_day, _outlook_count
     global _warned_budget
     now = time.time()
     hour_bucket = int(now // 3600)
@@ -383,7 +381,6 @@ def _begin_attempt(cap: int, force: bool) -> bool:
                 _skip("daily budget exhausted")
             return False
         _outlook_count += 1
-        _last_run_ts = now
         _last_run_hour = hour_bucket
         return True
 

news_engine.py

@@ -223,7 +223,6 @@ class FundamentalContext:
     warning_message: str = ""
     news_sentiment: Dict[str, float] = field(default_factory=dict)  # currency -> sentiment
     news_headlines: List[NewsSentiment] = field(default_factory=list)
-    event_impact_on_pairs: Dict[str, str] = field(default_factory=dict)
     calendar_available: bool = False  # True if live calendar data was fetched
     llm_summary: str = ""  # DeepSeek market-context blurb (display-only)
 
@@ -415,15 +414,6 @@ class NewsEngine:
                     )
                     break
 
-            # Map upcoming high-impact events to affected pairs
-            for event in context.events_upcoming_1h:
-                if event.is_high_impact:
-                    affected = self._get_affected_pairs(event.currency)
-                    for pair in affected:
-                        context.event_impact_on_pairs[pair] = (
-                            f"{event.currency} {event.event} in {event.minutes_until:.0f}min"
-                        )
-
         except Exception as e:
             logger.warning(f"Economic calendar fetch failed: {e}")
 
@@ -467,11 +457,10 @@ class NewsEngine:
 
     @staticmethod
     def _parse_pair(pair_symbol: str) -> Tuple[str, str]:
-        if "/" in pair_symbol:
+        base, sep, quote = pair_symbol.partition("/")
-            parts = pair_symbol.split("/")
+        if not sep:
-            return parts[0], parts[1]
+            raise ValueError(f"pair symbol must be BASE/QUOTE, got {pair_symbol!r}")
-        clean = pair_symbol.replace("=X", "")
+        return base, quote
-        return clean[:3], clean[3:]
 
     def get_event_risk_level(self, pair_symbol: str, context: FundamentalContext) -> Tuple[str, str]:
         """
@@ -1149,11 +1138,3 @@ class NewsEngine:
             except ValueError:
                 continue
         return None
-
-    @staticmethod
-    def _get_affected_pairs(currency: str) -> List[str]:
-        _MAP = {
-            "USD": ["XAU/USD"],
-            "XAU": ["XAU/USD"],
-        }
-        return _MAP.get(currency, [])

pair_signals/__init__.py

@@ -10,7 +10,7 @@ then register it in this file.
 
 from typing import Optional, List, Dict, Callable
 
-from config import PlatformConfig, Timeframe
+from config import Timeframe
 from signal_aggregator import SignalAggregator, SignalReport
 from providers import BaseDataProvider
 
@@ -21,10 +21,6 @@ def register_pair(symbol: str, config_factory: Callable):
     PAIR_PRODUCERS[symbol] = config_factory
 
 
-def get_supported_pairs() -> List[str]:
-    return list(PAIR_PRODUCERS.keys())
-
-
 def produce_signal(
     symbol: str,
     primary_tf: Timeframe,

position_sizing.py

@@ -10,11 +10,11 @@ inputs (rule 4: live == backtest parity).
 Design contract
 ---------------
 * **Bounded lookbacks.** No feature may read further back than the live
-  fetch depth of its timeframe (``LIVE_DEPTH_BARS``). The primary frame
+  fetch depth of its timeframe (5000 bars on M5/M15, 1460 on H1, 380 on
-  is hard-capped at ``MAX_LOOKBACK_BARS`` rows, confirmation frames at
+  H4). The primary frame is hard-capped at ``MAX_LOOKBACK_BARS`` rows,
-  60 rows, and the burst scan at 200 rows, so a feature computed on a
+  confirmation frames at 60 rows, and the burst scan at 200 rows, so a
-  full backtest frame equals the same feature computed on the shorter
+  feature computed on a full backtest frame equals the same feature
-  live tail.
+  computed on the shorter live tail.
 * **Closed bars only.** The frames handed in are exactly the frames
   ``analyze_from_dataframes`` received: UTC tz-aware, oldest-first,
   capitalized OHLCV columns, forming candle already dropped. The last
@@ -46,7 +46,7 @@ import json
 import logging
 import math
 from pathlib import Path
-from typing import Dict, Iterable, Optional, Sequence, Tuple
+from typing import Dict, Optional, Sequence, Tuple
 
 import numpy as np
 import pandas as pd
@@ -60,17 +60,6 @@ logger = logging.getLogger(__name__)
 #: A trained artifact carrying a different value is ignored.
 FEATURE_VERSION = 1
 
-#: Bars each timeframe actually holds in LIVE mode. Any feature whose
-#: lookback exceeds its timeframe's depth here would be unreproducible
-#: live, so the windows below stay far inside these numbers.
-LIVE_DEPTH_BARS: Dict[Timeframe, int] = {
-    Timeframe.M5: 5000,
-    Timeframe.M15: 5000,
-    Timeframe.H1: 1460,
-    Timeframe.H4: 380,
-    Timeframe.D1: 300,
-}
-
 #: Hard cap on any primary-timeframe window used in this module.
 MAX_LOOKBACK_BARS = 500
 
@@ -338,7 +327,7 @@ def _fill_time(out: Dict[str, float], dt: pd.Timestamp) -> None:
 
 
 def _fill_geometry(out: Dict[str, float], P: pd.DataFrame, setup,
-                   atr: float, atr_ok: bool, norm, d: float) -> None:
+                   atr: float, norm, d: float) -> None:
     out["f_atr"] = max(0.0, _finite(atr))
     ranges = (P["High"] - P["Low"]).astype(float)
     last_range = _finite(ranges.iloc[-1])
@@ -566,7 +555,7 @@ def build_features(
 
         for step, args in (
             (_fill_time, (out, decision_time(P, primary_tf))),
-            (_fill_geometry, (out, P, setup, atr, atr_ok, norm, d)),
+            (_fill_geometry, (out, P, setup, atr, norm, d)),
             (_fill_exhaustion, (out, report, KL, px, d, norm)),
             (_fill_quality, (out, report, setup, IND, d)),
             (_fill_mtf, (out, report, primary_tf, confirmation_tfs, d)),

providers/oanda.py

@@ -93,9 +93,11 @@ class OandaDataProvider(BaseDataProvider):
         return self._fetch_oanda_candles(symbol, timeframe, outputsize)
 
     def _to_oanda_instrument(self, symbol: str) -> str:
-        if symbol in self.OANDA_INSTRUMENT:
+        try:
             return self.OANDA_INSTRUMENT[symbol]
-        return symbol.replace("/", "_")
+        except KeyError:
+            raise OandaDataError(
+                f"Unsupported symbol {symbol!r} for Oanda") from None
 
     def _fetch_oanda_candles(
         self,

providers/tiingo.py

@@ -65,8 +65,9 @@ class TiingoDataProvider(BaseDataProvider):
     }
 
     # Window padding over the theoretical bar span: FX is closed ~2/7 of
-    # the week (Fri 22:00 → Sun 22:00 UTC), so a calendar window must be
+    # the week (Fri 17:00 → Sun 18:00 New York; see providers/base.py), so
-    # wider than `bars × candle_seconds` to yield the requested count.
+    # a calendar window must be wider than `bars × candle_seconds` to yield
+    # the requested count.
     WINDOW_PAD_FACTOR = 1.6
 
     def __init__(self, config: PlatformConfig, api_key: Optional[str] = None):
@@ -90,9 +91,11 @@ class TiingoDataProvider(BaseDataProvider):
         })
 
     def _to_tiingo_ticker(self, symbol: str) -> str:
-        if symbol in self.TIINGO_TICKER:
+        try:
             return self.TIINGO_TICKER[symbol]
-        return symbol.replace("/", "").replace("=X", "").lower()
+        except KeyError:
+            raise TiingoDataError(
+                f"Unsupported symbol {symbol!r} for Tiingo") from None
 
     # ── Raw fetch ──────────────────────────────────────────────────
 

providers/twelve_data.py

@@ -58,10 +58,10 @@ class TwelveDataProvider(BaseDataProvider):
 
     @staticmethod
     def _to_td_symbol(symbol: str) -> str:
-        if "/" in symbol:
+        if "/" not in symbol:
+            raise DataIntegrityError(
+                f"Unsupported symbol {symbol!r} for Twelve Data")
         return symbol
-        clean = symbol.replace("=X", "")
-        return f"{clean[:3]}/{clean[3:]}"
 
     # ── Raw fetch ──────────────────────────────────────────────────
 

scripts/backfill_signal_outcomes.py

@@ -41,7 +41,7 @@ def _status_counts(con: sqlite3.Connection) -> dict:
 
 def main(argv=None) -> int:
     ap = argparse.ArgumentParser(description=__doc__.splitlines()[1])
-    ap.add_argument("--db", default=None, help="defaults to signal_log.path")
+    ap.add_argument("--db", default=None, help="defaults to the signal_log.path setting in config.json")
     ap.add_argument("--dry-run", action="store_true")
     ap.add_argument("--reseed-unresolvable", action="store_true")
     ap.add_argument("--symbol", default="XAU/USD")

scripts/compare_oanda_tiingo.py

@@ -1,116 +0,0 @@
-#!/usr/bin/env python3
-"""
-Scratchpad — Oanda vs Tiingo candle parity check
-=================================================
-Fetches the last N closed XAU/USD candles from Oanda and Tiingo for
-M5/M15/H1 via the normal provider stack (make_data_provider →
-fetch_ohlcv: forming-candle drop + validate/clean included, i.e. exactly
-what the signal pipeline consumes) and reports how well they line up:
-
-  * last-bar timestamp per provider (feed-lag check)
-  * timestamp mismatches (bars present in one feed only)
-  * per-bar OHLC diff table for the common tail
-  * max/mean |diff| per column, with the Close diff also in % of price
-
-Both feeds are mid-price (Oanda mid of MBA candles; Tiingo resampled
-mid), so small diffs are expected — this measures how small.
-
-Usage:
-    venv/bin/python scripts/compare_oanda_tiingo.py [--n 10]
-"""
-
-import argparse
-import sys
-from pathlib import Path
-
-import pandas as pd
-
-sys.path.insert(0, str(Path(__file__).resolve().parent.parent))
-
-from config import PlatformConfig, Timeframe, XAUUSD
-from providers import make_data_provider
-
-TIMEFRAMES = [Timeframe.M5, Timeframe.M15, Timeframe.H1]
-OHLC = ["Open", "High", "Low", "Close"]
-
-
-def fetch_tail(provider, tf, n):
-    """Last n closed bars from one provider, or None on failure."""
-    try:
-        df = provider.fetch_ohlcv(symbol=XAUUSD, timeframe=tf)
-    except Exception as e:  # scratchpad: report, don't crash
-        print(f"  !! {provider.name} {tf.value} fetch failed: {e}")
-        return None
-    return df.tail(n)
-
-
-def compare_tf(tf, oanda_df, tiingo_df, n):
-    print(f"\n{'=' * 78}\n{tf.value} — last {n} closed bars\n{'=' * 78}")
-    if oanda_df is None or tiingo_df is None:
-        print("  skipped (a provider failed)")
-        return
-
-    print(f"  Oanda last bar: {oanda_df.index[-1]}   "
-          f"Tiingo last bar: {tiingo_df.index[-1]}")
-
-    common = oanda_df.index.intersection(tiingo_df.index)
-    only_o = oanda_df.index.difference(tiingo_df.index)
-    only_t = tiingo_df.index.difference(oanda_df.index)
-    if len(only_o):
-        print(f"  bars only in Oanda : {[str(t) for t in only_o]}")
-    if len(only_t):
-        print(f"  bars only in Tiingo: {[str(t) for t in only_t]}")
-    if not len(common):
-        print("  NO common timestamps — feeds are on different bar grids!")
-        return
-
-    o = oanda_df.loc[common, OHLC]
-    t = tiingo_df.loc[common, OHLC]
-    diff = o - t
-
-    print(f"\n  {len(common)}/{n} bars share timestamps "
-          f"(window {common[0]} → {common[-1]})\n")
-
-    # Per-bar Close table — the number that matters most.
-    hdr = f"  {'bar close (UTC)':<20}{'Oanda C':>12}{'Tiingo C':>12}{'ΔC ($)':>10}"
-    print(hdr + f"{'ΔO':>8}{'ΔH':>8}{'ΔL':>8}")
-    for ts in common:
-        row = (f"  {str(ts):<20}{o.loc[ts, 'Close']:>12.2f}"
-               f"{t.loc[ts, 'Close']:>12.2f}{diff.loc[ts, 'Close']:>10.3f}")
-        row += (f"{diff.loc[ts, 'Open']:>8.3f}{diff.loc[ts, 'High']:>8.3f}"
-                f"{diff.loc[ts, 'Low']:>8.3f}")
-        print(row)
-
-    print("\n  summary over common bars:")
-    for col in OHLC:
-        d = diff[col].abs()
-        print(f"    {col:<6} max |Δ| = {d.max():8.4f}   mean |Δ| = {d.mean():8.4f}")
-    px = o["Close"].iloc[-1]
-    dc = diff["Close"].abs()
-    print(f"    Close Δ as % of price: max {dc.max() / px * 100:.4f}%   "
-          f"mean {dc.mean() / px * 100:.4f}%")
-    exact = (diff[OHLC].abs() < 1e-9).all(axis=1).sum()
-    print(f"    bars with identical OHLC: {exact}/{len(common)}")
-
-
-def main():
-    ap = argparse.ArgumentParser()
-    ap.add_argument("--n", type=int, default=10,
-                    help="bars per timeframe to compare (default 10)")
-    args = ap.parse_args()
-
-    cfg = PlatformConfig()
-    oanda = make_data_provider(cfg, "oanda")
-    tiingo = make_data_provider(cfg, "tiingo")
-
-    for tf in TIMEFRAMES:
-        o_df = fetch_tail(oanda, tf, args.n)
-        t_df = fetch_tail(tiingo, tf, args.n)
-        compare_tf(tf, o_df, t_df, args.n)
-
-    print("\nDone. Note: Oanda=mid of MBA candles, Tiingo=resampled mid — "
-          "sub-$1 diffs on gold are normal feed divergence, not a bug.")
-
-
-if __name__ == "__main__":
-    sys.exit(main())

scripts/feed_divergence_study.py

@@ -1,259 +0,0 @@
-#!/usr/bin/env python3
-"""
-Phase 1 — historical candle divergence study (data source change test)
-=======================================================================
-Fetches the RAW closed-bar series for XAU/USD from two providers over a
-multi-month window via the normal provider stack (`fetch_range`, which
-paginates under the per-request bar cap and drops the forming candle),
-then quantifies how the feeds diverge:
-
-  * coverage: % shared timestamps, feed-only bars, where they cluster
-  * OHLC |Δ| distribution (mean/median/p90/p95/p99/max, $ and bias)
-  * |ΔClose| vs mean ATR14 (the scale the signal engine cares about)
-  * threshold exceedance (how many bars differ by more than X $)
-  * bar-direction agreement (sign of close−open)
-  * close-return correlation on consecutive bars
-  * per-hour and per-month drift
-
-Both feeds are mid-price (Oanda mid of MBA candles; Tiingo resampled mid;
-Twelve Data its own mid). Frames are cached to parquet so re-runs are
-instant; pass --no-cache to force a fresh fetch.
-
-Usage:
-    venv/bin/python scripts/feed_divergence_study.py --a oanda --b tiingo \
-        --tfs M15,H1,H4 --days 365
-"""
-
-import argparse
-import sys
-import time
-from pathlib import Path
-
-import numpy as np
-import pandas as pd
-
-sys.path.insert(0, str(Path(__file__).resolve().parent.parent))
-
-from config import PlatformConfig, Timeframe, XAUUSD
-from providers import make_data_provider
-
-REPO = Path(__file__).resolve().parent.parent
-CACHE_DIR = REPO / "backtest" / "cache"
-RESEARCH_DIR = REPO / "backtest" / "research"
-
-_TF_LOOKUP = {t.value: t for t in Timeframe}
-_TF_LOOKUP.update({
-    "M1": Timeframe.M1, "M5": Timeframe.M5, "M15": Timeframe.M15,
-    "M30": Timeframe.M30, "H1": Timeframe.H1, "H4": Timeframe.H4,
-    "D1": Timeframe.D1, "W1": Timeframe.W1,
-})
-
-
-def wilder_atr(df: pd.DataFrame, period: int = 14) -> pd.Series:
-    hl = df["High"] - df["Low"]
-    hc = (df["High"] - df["Close"].shift(1)).abs()
-    lc = (df["Low"] - df["Close"].shift(1)).abs()
-    tr = pd.concat([hl, hc, lc], axis=1).max(axis=1)
-    return tr.ewm(alpha=1.0 / period, adjust=False).mean()
-
-
-def fmt_stats(name: str, s: pd.Series) -> str:
-    a = s.abs()
-    return (f"  {name:<6} mean ${a.mean():.3f}  med ${a.median():.3f}  "
-            f"p90 ${a.quantile(0.90):.3f}  p95 ${a.quantile(0.95):.3f}  "
-            f"p99 ${a.quantile(0.99):.3f}  max ${a.max():.2f}  "
-            f"bias {s.mean():+.4f}")
-
-
-def cache_path(name: str, tf: Timeframe, start: pd.Timestamp,
-               end: pd.Timestamp) -> Path:
-    return (CACHE_DIR /
-            f"{name}_XAU_USD_{tf.value}_{start:%Y%m%d}_{end:%Y%m%d}.parquet")
-
-
-def load_or_fetch(provider, name: str, tf: Timeframe, start, end,
-                  no_cache: bool):
-    cp = cache_path(name, tf, start, end)
-    if not no_cache and cp.exists():
-        df = pd.read_parquet(cp)
-        print(f"  [{name}] loaded {len(df):,} bars from cache "
-              f"{cp.name}")
-        return df
-    t0 = time.time()
-    print(f"  [{name}] fetching {tf.value} {start.date()} → {end.date()} ...")
-    df = provider.fetch_range(XAUUSD, tf, start, end)
-    print(f"  [{name}] fetched {len(df):,} bars in {time.time() - t0:.1f}s")
-    df.to_parquet(cp)
-    print(f"  [{name}] cached → {cp.name}")
-    return df
-
-
-def compare(tf: Timeframe, a_name: str, b_name: str, a: pd.DataFrame,
-            b: pd.DataFrame):
-    print("\n" + "=" * 80)
-    print(f"{tf.value} — {a_name.upper()} vs {b_name.upper()}, XAU/USD")
-    print("=" * 80)
-
-    common = a.index.intersection(b.index)
-    a_only = a.index.difference(b.index)
-    b_only = b.index.difference(a.index)
-
-    print(f"\nCOVERAGE")
-    print(f"  {a_name} bars : {len(a):>8,}   head {a.index[0]}  tail {a.index[-1]}")
-    print(f"  {b_name} bars : {len(b):>8,}   head {b.index[0]}  tail {b.index[-1]}")
-    print(f"  shared        : {len(common):>8,}  "
-          f"({100 * len(common) / max(len(a.index.union(b.index)), 1):.2f}% of union)")
-    print(f"  {a_name}-only : {len(a_only):>8,}  "
-          f"({100 * len(a_only) / max(len(a), 1):.2f}% of {a_name})")
-    print(f"  {b_name}-only : {len(b_only):>8,}  "
-          f"({100 * len(b_only) / max(len(b), 1):.2f}% of {b_name})")
-
-    if len(common) == 0:
-        print("  !! no shared timestamps — feeds on different bar grids")
-        return
-
-    for label, idx in ((f"{a_name}-only", a_only),
-                       (f"{b_name}-only", b_only)):
-        if not len(idx):
-            continue
-        by_date = pd.Series(1, index=idx).groupby(idx.date).sum()
-        top = by_date.sort_values(ascending=False).head(6)
-        print(f"  {label} clusters on: "
-              + ", ".join(f"{d} ({n})" for d, n in top.items()))
-
-    x = a.loc[common]
-    y = b.loc[common]
-
-    atr = wilder_atr(x).dropna()
-    mean_price = x["Close"].mean()
-    print(f"\nSCALE REFERENCES ({a_name} feed)")
-    print(f"  mean close    : ${mean_price:,.2f}")
-    print(f"  mean bar range: ${(x['High'] - x['Low']).mean():.3f}")
-    print(f"  mean ATR14    : ${atr.mean():.3f}   median ${atr.median():.3f}")
-
-    print(f"\nPER-BAR DIFFS  ({a_name} − {b_name}, $)")
-    diffs = {}
-    for col in ("Open", "High", "Low", "Close"):
-        d = x[col] - y[col]
-        diffs[col] = d
-        print(fmt_stats(col, d))
-
-    dclose = diffs["Close"].abs()
-    print(f"\n  |ΔClose| as % of mean ATR14 : "
-          f"mean {100 * dclose.mean() / atr.mean():.1f}%   "
-          f"p95 {100 * dclose.quantile(0.95) / atr.mean():.1f}%")
-
-    print(f"\nTHRESHOLD EXCEEDANCE  (|ΔClose|)")
-    for thr in (0.10, 0.30, 0.65, 1.00, 2.00, 5.00):
-        pct = 100 * (dclose > thr).mean()
-        print(f"  > ${thr:<5.2f}: {pct:6.2f}% of bars  ({(dclose > thr).sum():,})")
-
-    # direction agreement
-    a_dir = np.sign(x["Close"] - x["Open"])
-    b_dir = np.sign(y["Close"] - y["Open"])
-    both_nonflat = (a_dir != 0) & (b_dir != 0)
-    dir_agree = 100 * (a_dir[both_nonflat] == b_dir[both_nonflat]).mean()
-    print(f"\nAGREEMENT")
-    print(f"  bar direction (sign of close−open) agree: {dir_agree:.2f}%"
-          f"  (n={both_nonflat.sum():,})")
-
-    step = pd.Series(common, index=common).diff().median()
-    a_ret = x["Close"].pct_change()
-    b_ret = y["Close"].pct_change()
-    step_ok = pd.Series(common, index=common).diff().le(step + pd.Timedelta(seconds=1))
-    corr = a_ret[step_ok].corr(b_ret[step_ok])
-    print(f"  close-return correlation (consecutive bars): {corr:.5f}")
-
-    # by-hour
-    hours = common.hour
-    rows = []
-    for h in range(24):
-        m = hours == h
-        if not m.any():
-            continue
-        rows.append({"hour_utc": h, "bars": int(m.sum()),
-                     "mean_abs_dclose": dclose[m].mean(),
-                     "p95_abs_dclose": dclose[m].quantile(0.95)})
-    by_hour = pd.DataFrame(rows)
-    print(f"\nBY UTC HOUR  (mean |ΔClose|, $)")
-    print(by_hour.to_string(index=False,
-                            float_format=lambda v: f"{v:.3f}"))
-
-    print(f"\nBY MONTH  (mean |ΔClose|, $)")
-    months = dclose.groupby(common.tz_localize(None).to_period("M")).mean()
-    for p, v in months.items():
-        print(f"  {p}: {v:.3f}")
-
-    return {
-        "tf": tf.value, "a": a_name, "b": b_name,
-        "a_bars": len(a), "b_bars": len(b), "common": len(common),
-        "a_only": len(a_only), "b_only": len(b_only),
-        "dir_agree_pct": round(dir_agree, 3),
-        "ret_corr": round(corr, 5),
-        "mean_abs_dclose": dclose.mean(),
-        "p95_abs_dclose": dclose.quantile(0.95),
-        "max_abs_dclose": dclose.max(),
-        "dclose_pct_atr_mean": 100 * dclose.mean() / atr.mean(),
-    }
-
-
-def main():
-    ap = argparse.ArgumentParser()
-    ap.add_argument("--a", required=True)
-    ap.add_argument("--b", required=True)
-    ap.add_argument("--tfs", default="M15,H1,H4",
-                    help="comma-separated timeframe values (default M15,H1,H4)")
-    ap.add_argument("--days", type=int, default=365)
-    ap.add_argument("--end", default=None, help="UTC end date (default now)")
-    ap.add_argument("--no-cache", action="store_true")
-    args = ap.parse_args()
-
-    tfs = []
-    for s in args.tfs.split(","):
-        s = s.strip()
-        tf = _TF_LOOKUP.get(s)
-        if tf is None:
-            sys.exit(f"unknown timeframe '{s}'; use one of "
-                     f"{sorted(_TF_LOOKUP)}")
-        tfs.append(tf)
-
-    end = pd.Timestamp(args.end, tz="UTC") if args.end \
-        else pd.Timestamp.now(tz="UTC")
-    start = end - pd.Timedelta(days=args.days)
-
-    cfg = PlatformConfig()
-    pa = make_data_provider(cfg, args.a)
-    pb = make_data_provider(cfg, args.b)
-
-    summary = []
-    for tf in tfs:
-        print(f"\n{'─' * 80}\nWINDOW {start.date()} → {end.date()}  "
-              f"({args.days} days), {tf.value}")
-        try:
-            a = load_or_fetch(pa, args.a, tf, start, end, args.no_cache)
-        except Exception as e:
-            print(f"  !! {args.a} {tf.value} fetch failed: {e}")
-            continue
-        try:
-            b = load_or_fetch(pb, args.b, tf, start, end, args.no_cache)
-        except Exception as e:
-            print(f"  !! {args.b} {tf.value} fetch failed: {e}")
-            continue
-        r = compare(tf, args.a, args.b, a, b)
-        if r:
-            summary.append(r)
-
-    print("\n" + "=" * 80)
-    print("SUMMARY TABLE")
-    print("=" * 80)
-    sdf = pd.DataFrame(summary)
-    print(sdf.to_string(index=False,
-                        float_format=lambda v: f"{v:.4f}"))
-    out = RESEARCH_DIR / f"feed_divergence_{args.a}_vs_{args.b}.csv"
-    if len(sdf):
-        sdf.to_csv(out, index=False, float_format="%.6f")
-        print(f"\nSummary written to {out}")
-
-
-if __name__ == "__main__":
-    main()

scripts/label_scan_outcomes.py

@@ -60,7 +60,7 @@ def _counts(con: sqlite3.Connection, symbol: str,
 
 def main(argv=None) -> int:
     ap = argparse.ArgumentParser(description=__doc__.splitlines()[1])
-    ap.add_argument("--db", default=None, help="defaults to signal_log.path")
+    ap.add_argument("--db", default=None, help="defaults to the signal_log.path setting in config.json")
     ap.add_argument("--dry-run", action="store_true",
                     help="read-only candidate counts, writes nothing")
     ap.add_argument("--budget", type=int, default=100000,

scripts/shadow_scan.py

@@ -36,8 +36,6 @@ import time
 from datetime import datetime, timezone
 from pathlib import Path
 
-import pandas as pd
-
 sys.path.insert(0, str(Path(__file__).resolve().parent.parent))
 
 from config import PlatformConfig, Timeframe, XAUUSD

scripts/signal_parity_study.py

@@ -1,246 +0,0 @@
-#!/usr/bin/env python3
-"""
-Phase 2 — signal-level replay parity (data source change test)
-===============================================================
-Replays the REAL decision core (`SignalAggregator.analyze_from_dataframes`,
-news off, use_news=False) over the same window for two providers and
-compares the *decisions* bar-by-bar: direction, composite/final score,
-quality grade, gate/setup presence, strategy tag, and entry/SL/TP levels.
-
-This turns the Phase-1 price-diff stats into flip rates. Feeds are read
-from the Phase-1 parquet cache (scripts/feed_divergence_study.py writes
-them); pass --no-cache-fetch only if you want to fetch fresh via
-fetch_range first.
-
-Slicing matches gpu_backtest.py exactly: decision at close of bar `pos`,
-each TF sliced to bars fully closed by that decision time, primary at the
-shared grid, so the two feeds are compared on identical timestamps.
-
-Usage:
-    venv/bin/python scripts/signal_parity_study.py --a oanda --b tiingo \
-        --tf M15 --stride 8 --days 365
-"""
-
-import argparse
-import sys
-import time
-from pathlib import Path
-
-import numpy as np
-import pandas as pd
-
-sys.path.insert(0, str(Path(__file__).resolve().parent.parent))
-
-from config import Timeframe, XAUUSD
-from pair_signals.xauusd import make_config
-from signal_aggregator import SignalAggregator
-
-REPO = Path(__file__).resolve().parent.parent
-CACHE_DIR = REPO / "backtest" / "cache"
-RESEARCH_DIR = REPO / "backtest" / "research"
-
-BAR_SECONDS = {
-    Timeframe.M1: 60, Timeframe.M5: 300, Timeframe.M15: 900,
-    Timeframe.M30: 1800, Timeframe.H1: 3600, Timeframe.H4: 14400,
-    Timeframe.D1: 86400, Timeframe.W1: 604800,
-}
-_TF_LOOKUP = {t.value: t for t in Timeframe}
-_TF_LOOKUP.update({"M1": Timeframe.M1, "M5": Timeframe.M5,
-                   "M15": Timeframe.M15, "M30": Timeframe.M30,
-                   "H1": Timeframe.H1, "H4": Timeframe.H4,
-                   "D1": Timeframe.D1, "W1": Timeframe.W1})
-_CONFIRMS = {
-    Timeframe.M5: [Timeframe.M15, Timeframe.H1],
-    Timeframe.M15: [Timeframe.M5, Timeframe.H1],
-    Timeframe.H1: [Timeframe.M15, Timeframe.H4],
-    Timeframe.H4: [Timeframe.H1, Timeframe.D1],
-}
-
-
-def load_cached(name: str, tf: Timeframe, start: pd.Timestamp,
-                end: pd.Timestamp) -> pd.DataFrame:
-    cp = CACHE_DIR / f"{name}_XAU_USD_{tf.value}_{start:%Y%m%d}_{end:%Y%m%d}.parquet"
-    if not cp.exists():
-        sys.exit(f"missing cache {cp}; run scripts/feed_divergence_study.py "
-                 f"--a oanda --b {name} --tfs {tf.value} --days ... first")
-    return pd.read_parquet(cp)
-
-
-def tag_of(report) -> str:
-    first_bull = report.bullish_reasons[0] if report.bullish_reasons else ""
-    first_bear = report.bearish_reasons[0] if report.bearish_reasons else ""
-    if "XAU PULLBACK" in first_bull or "XAU PULLBACK" in first_bear:
-        return "XAU_PB"
-    if "XAU CHART PATTERN" in first_bull or "XAU CHART PATTERN" in first_bear:
-        return "XAU_CP"
-    if "XAU STRATEGY" in first_bull:
-        return "XAU"
-    return "GENERIC"
-
-
-def decide(agg, dfs, tss, stride, warmup) -> list:
-    """Replay decisions over the shared grid, returning one row per bar."""
-    primary = dfs[agg.config.primary_timeframe]
-    dur = pd.Timedelta(seconds=BAR_SECONDS[agg.config.primary_timeframe])
-
-    # positions of sampled bars in the primary frame
-    primary_pos = primary.index.get_indexer(tss)
-    rows = []
-    for i in range(0, len(tss), stride):
-        ts = tss[i]
-        pos = int(primary_pos[i])
-        if pos < warmup:
-            continue
-        slices = {}
-        for tf, full in dfs.items():
-            cutoff = ts  # decision_time = ts + dur; bars closed by ts
-            cut = int(full.index.searchsorted(cutoff, side="right"))
-            if cut <= 0:
-                continue
-            slices[tf] = full.iloc[:cut]
-        if agg.config.primary_timeframe not in slices:
-            continue
-        r = agg.analyze_from_dataframes(XAUUSD, slices, use_news=False)
-        ts_setup = r.trade_setup
-        rows.append({
-            "ts": ts,
-            "direction": r.final_signal.value,
-            "final_score": r.final_score,
-            "primary_score": r.primary_score,
-            "grade": r.signal_quality,
-            "has_setup": ts_setup is not None,
-            "tag": tag_of(r) if ts_setup else "",
-            "entry": ts_setup.entry_price if ts_setup else np.nan,
-            "sl": ts_setup.stop_loss if ts_setup else np.nan,
-            "tp1": ts_setup.take_profit_1 if ts_setup else np.nan,
-            "tp2": ts_setup.take_profit_2 if ts_setup else np.nan,
-        })
-    return rows
-
-
-def main():
-    ap = argparse.ArgumentParser()
-    ap.add_argument("--a", required=True)
-    ap.add_argument("--b", required=True)
-    ap.add_argument("--tf", default="M15")
-    ap.add_argument("--stride", type=int, default=1,
-                    help="sample every Nth shared bar (default 1)")
-    ap.add_argument("--days", type=int, default=365)
-    ap.add_argument("--end", default=None)
-    ap.add_argument("--warmup", type=int, default=300)
-    args = ap.parse_args()
-
-    tf = _TF_LOOKUP.get(args.tf)
-    if tf is None:
-        sys.exit(f"unknown tf {args.tf}")
-    confirms = _CONFIRMS[tf]
-    tfs = [tf] + confirms
-
-    end = pd.Timestamp(args.end, tz="UTC") if args.end \
-        else pd.Timestamp.now(tz="UTC")
-    start = end - pd.Timedelta(days=args.days)
-
-    cfg = make_config(tf, confirms)
-    agg = SignalAggregator(cfg)
-
-    a_dfs, b_dfs = {}, {}
-    for t in tfs:
-        a_dfs[t] = load_cached(args.a, t, start, end)
-        b_dfs[t] = load_cached(args.b, t, start, end)
-
-    # shared grid = primary timestamps present in both feeds
-    shared = a_dfs[tf].index.intersection(b_dfs[tf].index)
-
-    t0 = time.time()
-    print(f"replaying {len(shared)} shared {tf.value} bars "
-          f"(stride {args.stride} → ~{len(shared)//args.stride}) on {args.a}...")
-    a_rows = decide(agg, a_dfs, shared, args.stride, args.warmup)
-    print(f"  {args.a}: {len(a_rows)} decisions in {time.time()-t0:.1f}s")
-    t0 = time.time()
-    b_rows = decide(agg, b_dfs, shared, args.stride, args.warmup)
-    print(f"  {args.b}: {len(b_rows)} decisions in {time.time()-t0:.1f}s")
-
-    A = pd.DataFrame(a_rows).set_index("ts")
-    B = pd.DataFrame(b_rows).set_index("ts")
-    common = A.index.intersection(B.index)
-    A, B = A.loc[common], B.loc[common]
-    n = len(common)
-
-    print(f"\n{'=' * 78}")
-    print(f"SIGNAL-LEVEL PARITY — {tf.value} — {args.a} vs {args.b} "
-          f"({n:,} bars)")
-    print("=" * 78)
-
-    def pct(x): return f"{100 * x:.2f}%"
-
-    same_dir = (A["direction"] == B["direction"])
-    same_grade = (A["grade"] == B["grade"])
-    same_setup = (A["has_setup"] == B["has_setup"])
-    same_all = same_dir & same_grade & same_setup
-
-    print(f"\nIDENTITY")
-    print(f"  same direction        : {pct(same_dir.mean())}")
-    print(f"  same grade            : {pct(same_grade.mean())}")
-    print(f"  same setup presence   : {pct(same_setup.mean())}")
-    print(f"  same dir+grade+setup  : {pct(same_all.mean())}")
-
-    both_sig = (A["direction"] != 0) & (B["direction"] != 0)
-    flips = both_sig & (A["direction"] != B["direction"])
-    opp = flips & (A["direction"] == -B["direction"])
-    print(f"\nDIRECTION")
-    print(f"  bars both actionable    : {both_sig.sum():,} "
-          f"({pct(both_sig.mean())})")
-    print(f"  direction flips         : {flips.sum():,} "
-          f"({pct(flips.sum()/max(both_sig.sum(),1))} of actionable)")
-    print(f"  opposite-sign flips     : {opp.sum():,}")
-
-    grade_a = set(A.loc[A["has_setup"], "grade"])
-    print(f"\nGRADE (of {args.a} setups, grades seen: {sorted(grade_a)})")
-    changed = A.loc[both_sig, "grade"] != B.loc[both_sig, "grade"]
-    print(f"  grade changed on actionable bars: "
-          f"{changed.sum():,} ({pct(changed.mean())})")
-
-    # score deltas on shared bars
-    ds = (A["final_score"] - B["final_score"]).abs()
-    dp = (A["primary_score"] - B["primary_score"]).abs()
-    print(f"\nSCORE Δ  ({args.a} − {args.b})")
-    print(f"  final_score   mean {ds.mean():6.2f}  p95 {ds.quantile(.95):6.2f}  "
-          f"max {ds.max():6.2f}")
-    print(f"  primary_score mean {dp.mean():6.2f}  p95 {dp.quantile(.95):6.2f}  "
-          f"max {dp.max():6.2f}")
-
-    # setup price diffs on bars where BOTH generated a setup
-    both_setup = A["has_setup"] & B["has_setup"]
-    if both_setup.any():
-        print(f"\nSETUP LEVELS (both-setup bars: {both_setup.sum():,})")
-        for col, label in (("entry", "entry"), ("sl", "SL"), ("tp1", "TP1"),
-                           ("tp2", "TP2")):
-            d = (A.loc[both_setup, col] - B.loc[both_setup, col]).abs()
-            print(f"  {label:<6} mean ${d.mean():6.3f}  p95 ${d.quantile(.95):7.3f}  "
-                  f"max ${d.max():7.2f}")
-
-    # tag disagreement on both-setup bars
-    if both_setup.any():
-        same_tag = (A.loc[both_setup, "tag"] == B.loc[both_setup, "tag"])
-        print(f"  same strategy tag      : {pct(same_tag.mean())}")
-
-    # where do flips cluster (hour)?
-    flip_hours = A.loc[flips].index.hour.value_counts().sort_index()
-    if len(flip_hours):
-        print(f"\nFLIPS BY UTC HOUR")
-        for h, c in flip_hours.items():
-            print(f"  {h:02d}: {c}")
-
-    out = {
-        "tf": tf.value, "a": args.a, "b": args.b, "bars": n,
-        "same_dir": same_dir.mean(), "same_grade": same_grade.mean(),
-        "same_setup": same_setup.mean(), "same_all": same_all.mean(),
-        "flip_rate_actionable": flips.sum() / max(both_sig.sum(), 1),
-        "mean_abs_final_score_delta": ds.mean(),
-        "mean_abs_primary_score_delta": dp.mean(),
-    }
-    print(f"\n{out}")
-
-
-if __name__ == "__main__":
-    main()

signal_aggregator.py

@@ -150,12 +150,6 @@ class SignalReport:
 class SignalAggregator:
     """Master aggregator for live signal analysis."""
 
-    # Diagnostic funnel for the M5 pullback route (shared across
-    # instances; read by backtest tooling, harmless in live use).
-    pb_funnel: Dict[str, int] = {}
-    # Same, for the chart-pattern route (XAU_CP).
-    cp_funnel: Dict[str, int] = {}
-
     def __init__(self, config: PlatformConfig):
         self.config = config
         self.data_provider = make_data_provider(config)
@@ -172,7 +166,6 @@ class SignalAggregator:
     def analyze_pair(
         self,
         symbol: Optional[str] = None,
-        context: Optional[FundamentalContext] = None,
         log_source: Optional[str] = None,
     ) -> SignalReport:
         """Live-mode entry: fetch all needed timeframes then delegate.
@@ -244,7 +237,6 @@ class SignalAggregator:
         report = self.analyze_from_dataframes(
             symbol=symbol,
             dfs=dfs,
-            context=context,
             use_news=True,
         )
 
@@ -661,16 +653,10 @@ class SignalAggregator:
     ) -> None:
         """Build an XAU_PB trade setup if a pullback candidate passes gates."""
         cfg = self.config.xau_strategy
-        # Rejection funnel counters (class-level diagnostic, cheap).
-        funnel = SignalAggregator.pb_funnel
-
-        def _reject(reason: str) -> None:
-            funnel[reason] = funnel.get(reason, 0) + 1
 
         pb = self._detect_m5_pullback(report, primary_df)
         if pb is None:
             return
-        _reject("detected")
         direction = pb["direction"]
         is_buy = direction == "BUY"
 
@@ -678,7 +664,6 @@ class SignalAggregator:
         # price mid-pullback scores weaker on the momentum indicators).
         signed = report.final_score if is_buy else -report.final_score
         if signed < cfg.m5_pullback_min_score:
-            _reject("score")
             return
 
         # Confirmation TF must agree (reuse Gate-3 threshold).
@@ -693,7 +678,6 @@ class SignalAggregator:
                 confirmed = True
                 break
         if not confirmed:
-            _reject("confirmation_tf")
             return
 
         # Never against multi-TF consensus (mirror of gate 6f).
@@ -702,16 +686,13 @@ class SignalAggregator:
         bear_n = sum(1 for s in sigs if s in (SignalStrength.SELL, SignalStrength.STRONG_SELL))
         thr = self.config.signals.mtf_agreement_threshold
         if (is_buy and bear_n >= thr) or (not is_buy and bull_n >= thr):
-            _reject("mtf_opposition")
             return
 
         # Volatility gates match the breakout path.
         if report.volatility_regime in cfg.m5_blocked_vol_regimes:
-            _reject("vol_regime")
             return
         if (not is_buy and report.volatility_regime
                 and report.volatility_regime not in cfg.sell_vol_regimes):
-            _reject("sell_vol")
             return
 
         atr = report.primary_analysis.key_levels.get("atr", 0.0)
@@ -732,7 +713,6 @@ class SignalAggregator:
         min_tp = (self.config.signals.round_trip_cost
                   * self.config.signals.min_tp_cost_multiple)
         if 1.5 * sl_dist < min_tp:
-            _reject("spread_gate")
             return
 
         report.trade_setup = TradeSetup(
@@ -752,7 +732,6 @@ class SignalAggregator:
             trail_arm_r=self.config.signals.pb_trail_arm_r,
             trail_dist_r=self.config.signals.pb_trail_dist_r,
         )
-        _reject("accepted")   # funnel tick, mirrors the CP path
         if report.signal_quality not in ("A+", "A", "B"):
             report.signal_quality = "B"
         report.final_signal = (SignalStrength.BUY if is_buy
@@ -779,10 +758,6 @@ class SignalAggregator:
         structural gates plus the validated directional filters below.
         """
         cfg = self.config.xau_strategy
-        funnel = SignalAggregator.cp_funnel
-
-        def _reject(reason: str) -> None:
-            funnel[reason] = funnel.get(reason, 0) + 1
 
         pa = report.primary_analysis
         if pa is None:
@@ -795,7 +770,6 @@ class SignalAggregator:
         pat = detect_chart_pattern(primary_df, atr, cfg)
         if pat is None:
             return
-        _reject("detected")
         direction = pat["direction"]
         is_buy = direction == "BUY"
 
@@ -805,7 +779,6 @@ class SignalAggregator:
         bear_n = sum(1 for s in sigs if s in (SignalStrength.SELL, SignalStrength.STRONG_SELL))
         thr = self.config.signals.mtf_agreement_threshold
         if (is_buy and bear_n >= thr) or (not is_buy and bull_n >= thr):
-            _reject("mtf_opposition")
             return
 
         # Directional protections — gold's macro-uptrend asymmetry is
@@ -814,7 +787,6 @@ class SignalAggregator:
         if not is_buy:
             if (report.volatility_regime
                     and report.volatility_regime not in cfg.sell_vol_regimes):
-                _reject("sell_vol")
                 return
             if cfg.sell_blocked_hours_utc:
                 try:
@@ -833,18 +805,15 @@ class SignalAggregator:
                         f"CP SELL hour gate FAILS CLOSED (short rejected)"
                     )
                 if close_h is None or close_h in cfg.sell_blocked_hours_utc:
-                    _reject("sell_hour")
                     return
             if self.config.primary_timeframe == Timeframe.M5:
                 rsi_ind = pa.indicators.get("rsi")
                 if rsi_ind is not None and rsi_ind.value < cfg.m5_sell_min_rsi:
-                    _reject("m5_sell_rsi")
                     return
 
         # M5 chop-transition volatility block (validated, both ways).
         if (self.config.primary_timeframe == Timeframe.M5
                 and report.volatility_regime in cfg.m5_blocked_vol_regimes):
-            _reject("vol_regime")
             return
 
         entry = report.current_price
@@ -865,12 +834,10 @@ class SignalAggregator:
         # the RR floor or the spread gate, the pattern is untradeable.
         rr = height / sl_dist if sl_dist > 0 else 0.0
         if rr < self.config.signals.min_rr_ratio - 1e-9:
-            _reject("rr")
             return
         min_tp = (self.config.signals.round_trip_cost
                   * self.config.signals.min_tp_cost_multiple)
         if height < min_tp:
-            _reject("spread_gate")
             return
 
         report.trade_setup = TradeSetup(
@@ -904,7 +871,6 @@ class SignalAggregator:
         target = report.bullish_reasons if is_buy else report.bearish_reasons
         target.insert(0, marker)
         self._clean_accepted_cp_warnings(report)
-        _reject("accepted")
 
     @staticmethod
     def _clean_accepted_cp_warnings(report: SignalReport) -> None:
@@ -919,9 +885,6 @@ class SignalAggregator:
             "EXHAUSTION BLOCK",
             "EXHAUSTED TREND FILTER:",
             "MTF OPPOSITION:",
-            "SELL FILTER: quality",
-            "SELL FILTER: volatility",
-            "SELL FILTER: regime",
             "SELL FILTER: ",
             "M5 FILTER: |score|",
             "M5 FILTER: no MTF agreement",
@@ -1304,7 +1267,8 @@ class SignalAggregator:
 
     # ── Internals ──────────────────────────────────────────────────
 
-    _TP_QUALITY_SCALE = {"A+": 2.0, "A": 1.5, "B": 1.0, "C": 1.0}
+    # Only A+/A reach the XAU ATR generator (gate at the call site).
+    _TP_QUALITY_SCALE = {"A+": 2.0, "A": 1.5}
 
     def _generate_xau_atr_trade_setup(
         self, report: SignalReport, atr: float
@@ -1321,7 +1285,7 @@ class SignalAggregator:
         else:
             is_buy = report.final_score >= 0
 
-        quality_scale = self._TP_QUALITY_SCALE.get(report.signal_quality, 0.75)
+        quality_scale = self._TP_QUALITY_SCALE[report.signal_quality]
         if report.signal_quality == "A" and report.exhaustion_count >= 2:
             quality_scale = 1.0
         tp_mult = cfg.tp_atr_multiplier * quality_scale

signal_log.py

@@ -806,7 +806,6 @@ class SignalLogger:
     the daemon's thread pool."""
 
     def __init__(self, path: str):
-        self._path = path
         parent = os.path.dirname(path)
         if parent:
             os.makedirs(parent, exist_ok=True)
@@ -860,10 +859,6 @@ class SignalLogger:
                     raise
                 time.sleep(delay * (attempt + 1))
 
-    @property
-    def path(self) -> str:
-        return self._path
-
     def _migrate_signal_columns(self) -> None:
         """Add newer signal columns to pre-existing tables (idempotent)."""
         existing = {r[1] for r in self._conn.execute("PRAGMA table_info(signals)")}
@@ -983,11 +978,6 @@ class SignalLogger:
             (now_iso, digest))
         return digest
 
-    def news_for(self, rowid: int) -> Tuple[Optional[str], Optional[str]]:
-        """(events_json, headlines_json) for a row, legacy or interned."""
-        with self._lock:
-            return read_news(self._conn, rowid)
-
     # ── advisory-event seeding / dedup ────────────────────────────
 
     def _seed_outcome(self, rowid: int, row: Dict[str, Any]) -> None:
@@ -1291,27 +1281,6 @@ class SignalLogger:
                         rowid, exc_info=True)
         return written
 
-    def advance_with_bars(self, rowid: int, bars: Iterable[Dict[str, Any]],
-                          evidence: str = "external_bars") -> str:
-        """Advance ONE pending outcome from caller-supplied bars (backfill
-        from provider history). Returns the resulting status."""
-        with self._lock:
-            row = self._conn.execute(
-                "SELECT advisory_outcome_status, advisory_outcome_json"
-                " FROM signals WHERE id=?", (rowid,)).fetchone()
-            if row is None or row[0] != "PENDING" or not row[1]:
-                return str(row[0]) if row else "MISSING"
-            state = advance_outcome(json.loads(row[1]), list(bars))
-            state["evidence"] = evidence
-            status = state.get("status", "UNRESOLVABLE")
-            self._conn.execute(
-                "UPDATE signals SET advisory_outcome_status=?,"
-                " advisory_r_multiple=?, advisory_outcome_json=?"
-                " WHERE id=? AND advisory_outcome_status='PENDING'",
-                (status, canonical_r(state), _dumps(state), rowid))
-            self._conn.commit()
-            return status
-
     def backfill_seeds(self, limit: Optional[int] = None) -> Dict[str, int]:
         """Assign setup_key / advisory_event_id to legacy setup rows and
         seed the canonical ones that carry no outcome yet. Existing

static/dashboard.html

@@ -71,7 +71,7 @@
                 </div>
               </div>
               <div class="table-responsive">
-                <table class="table table-sm" id="cal-table">
+                <table class="table table-sm">
                   <thead>
                     <tr>
                       <th>Time</th>

static/news_summary.php

@@ -11,7 +11,7 @@
  *   - CACHE FOREVER: headlines are immutable, so each id costs at most
  *     one DeepSeek call, ever (ai/<id>.json). Cache is checked BEFORE
  *     the whitelist on purpose: summaries stay reachable after the
- *     headline ages out of the 24 h feed window.
+ *     headline ages out of the 4 h feed window.
  *   - DAILY CAP: file-based UTC-day counter, FAIL CLOSED on counter I/O
  *     errors (a perms mistake must not mean unmetered spend).
  *
@@ -112,9 +112,7 @@ if (!is_array($news)) {
 }
 $hit = null;
 foreach ($news['headlines'] ?? [] as $h) {
-    // Fallback sha1(): a publisher that predates the id field still
+    $hid = $h['id'] ?? '';
-    // serves a valid whitelist during the restart window.
-    $hid = $h['id'] ?? sha1($h['headline'] ?? '');
     if ($hid === $id) {
         $hit = $h;
         break;

telegram_bot.py

@@ -50,7 +50,7 @@ import threading
 import time
 from concurrent.futures import ThreadPoolExecutor
 from datetime import datetime, timezone
-from typing import Any, Dict, List, Optional, Set
+from typing import TYPE_CHECKING, Any, Dict, List, Optional, Set
 
 from telegram import InlineKeyboardButton, InlineKeyboardMarkup, Update
 from telegram.constants import ParseMode
@@ -77,7 +77,10 @@ from market_outlook import maybe_run_outlook
 from news_engine import FundamentalContext, NewsEngine
 from pair_signals import PAIR_PRODUCERS
 from signal_aggregator import SignalAggregator, SignalReport
-from violent_watch import ViolentWatch, attach_watch, should_send_headsup
+from violent_watch import attach_watch, should_send_headsup
+
+if TYPE_CHECKING:  # type hints only — ViolentWatch is never needed at runtime here
+    from violent_watch import ViolentWatch
 
 _APP = get_app_config()
 
@@ -185,13 +188,6 @@ TF_CONFIRMATIONS = {
     Timeframe.H4:  [Timeframe.H1, Timeframe.D1],
 }
 
-AUTOSCAN_CONFIRMATIONS = {
-    Timeframe.M5:  [Timeframe.M15, Timeframe.H1],
-    Timeframe.M15: [Timeframe.M5, Timeframe.H1],
-    Timeframe.H1:  [Timeframe.M15, Timeframe.H4],
-    Timeframe.H4:  [Timeframe.H1, Timeframe.D1],
-}
-
 _executor = ThreadPoolExecutor(max_workers=4)
 
 # ── Markdown escaping ─────────────────────────────────────────────────
@@ -203,7 +199,7 @@ def _escape_md(text: str) -> str:
 
 
 def _fmt_price(price: float) -> str:
-    return f"{price:.2f}" if price >= 10 else f"{price:.5f}"
+    return f"{price:.2f}"
 
 
 # ── Shared data provider (single cache across all scans) ─────────────
@@ -438,37 +434,6 @@ def format_signal_message(report: SignalReport) -> str:
     return "\n".join(lines)
 
 
-# ── Auto-scan summary format ─────────────────────────────────────────
-
-def _format_scan_summary(reports: List[SignalReport], tf: Timeframe) -> str:
-    now = datetime.now(timezone.utc).strftime("%H:%M UTC")
-    lines = [
-        f"📊 *{tf.value} Auto-Scan* | {now}",
-        "━━━━━━━━━━━━━━━━━━━━",
-    ]
-    for r in reports:
-        emoji = _SIGNAL_EMOJI.get(r.final_signal, "⬜")
-        sig = _signal_label(r.final_signal)
-        mtf = "✅" if r.mtf_agreement else "❌"
-        lines.append(
-            f"{emoji} *{r.display_name}*  {sig}  "
-            f"{_QUALITY_EMOJI.get(r.signal_quality, '')}{r.signal_quality}  "
-            f"{r.final_score:+.1f}  {r.final_confidence:.0f}%  {mtf}MTF"
-        )
-    lines.append("━━━━━━━━━━━━━━━━━━━━")
-
-    actionable = [r for r in reports if r.signal_quality in ("A+", "A", "B", "C")]
-    if actionable:
-        names = ", ".join(r.display_name for r in actionable)
-        lines.append(f"🔔 Actionable: {names}")
-    else:
-        lines.append("No actionable signals this cycle.")
-
-    lines += ["", _DISCLAIMER]
-
-    return "\n".join(lines)
-
-
 # ── Keyboard helper ───────────────────────────────────────────────────
 
 def _scan_button_rows() -> List[List[InlineKeyboardButton]]:
@@ -487,10 +452,6 @@ def _scan_button_rows() -> List[List[InlineKeyboardButton]]:
     return rows
 
 
-def scan_keyboard() -> InlineKeyboardMarkup:
-    return InlineKeyboardMarkup(_scan_button_rows())
-
-
 def main_menu_keyboard() -> InlineKeyboardMarkup:
     """Full interactive menu: scans + settings + status in one keyboard."""
     autoscan_label = "🔔 Auto-scan: ON" if _autoscan_chats else "🔕 Auto-scan: OFF"
@@ -536,15 +497,9 @@ def _run_signal_sync(
             timeframe, [Timeframe.M15, Timeframe.H4]
         )
     # Per-pair tuned configs (regime weights, gates, sessions) come from
-    # the pair_signals factories; a plain PlatformConfig would silently
+    # the pair_signals factories; every scanned symbol is registered there
-    # run every pair with XAU defaults.
+    # (SCAN_PAIRS is validated against PAIR_INFO at import).
-    factory = PAIR_PRODUCERS.get(symbol)
+    config = PAIR_PRODUCERS[symbol](timeframe, confirmation_tfs)
-    if factory is not None:
-        config = factory(timeframe, confirmation_tfs)
-    else:
-        config = PlatformConfig()
-        config.primary_timeframe = timeframe
-        config.confirmation_timeframes = confirmation_tfs
     aggregator = SignalAggregator(config)
     aggregator.data_provider = _get_shared_dp()
     return aggregator.analyze_pair(symbol=symbol, log_source="telegram")
@@ -1144,7 +1099,7 @@ async def _candle_loop(bot, tf: Timeframe, interval_minutes: int) -> None:
             logger.info(f"{label} auto-scan: firing")
             try:
                 await _scan_timeframe(
-                    bot, tf, AUTOSCAN_CONFIRMATIONS[tf]
+                    bot, tf, TF_CONFIRMATIONS[tf]
                 )
             except Exception:
                 # One failed cycle must not kill the loop for good —

test_all.py

@@ -12,8 +12,7 @@ import threading
 import tempfile
 import traceback
 from datetime import datetime, timedelta, timezone
-from unittest.mock import patch, MagicMock
+from unittest.mock import patch
-from io import StringIO
 
 import numpy as np
 import pandas as pd
@@ -756,13 +755,6 @@ def _():
     assert quote == "USD"
 
 
-@test("NewsEngine._parse_pair with =X suffix")
-def _():
-    base, quote = NewsEngine._parse_pair("EURUSD=X")
-    assert base == "EUR"
-    assert quote == "USD"
-
-
 @test("NewsEngine._parse_pair for XAU pair")
 def _():
     base, quote = NewsEngine._parse_pair("XAU/USD")
@@ -882,18 +874,6 @@ def _():
     assert NewsEngine._parse_rss_date("not-a-date") is None
 
 
-@test("NewsEngine._get_affected_pairs")
-def _():
-    pairs = NewsEngine._get_affected_pairs("USD")
-    assert "XAU/USD" in pairs
-
-    pairs_xau = NewsEngine._get_affected_pairs("XAU")
-    assert "XAU/USD" in pairs_xau
-
-    pairs_unknown = NewsEngine._get_affected_pairs("ARS")
-    assert pairs_unknown == []
-
-
 @test("EconomicEvent properties")
 def _():
     future_time = datetime.now(timezone.utc) + timedelta(minutes=30)
@@ -1489,7 +1469,7 @@ print("\n═══ Module 6: main.py ═══")
 
 from main import (
     signal_color, quality_color, _compact_reason,
-    report_to_dict, build_config, Colors,
+    report_to_dict, build_config,
 )
 
 
@@ -1607,14 +1587,19 @@ def _():
     assert Timeframe.D1 in config.confirmation_timeframes
 
 
-@test("build_config all valid timeframes")
+@test("build_config: every CLI timeframe uses the pair factory's confirmation map")
 def _():
-    for tf_str in ["M5", "M15", "M30", "H1", "H4", "D1"]:
+    from pair_signals import xauusd as _xau
+    for tf_str, tf in (("M5", Timeframe.M5), ("M15", Timeframe.M15),
+                       ("H1", Timeframe.H1), ("H4", Timeframe.H4)):
         class FakeArgs:
             pair = "XAUUSD"
             timeframe = tf_str
         config = build_config(FakeArgs())
-        assert config.primary_timeframe is not None
+        assert config.primary_timeframe == tf
+        # The CLI must not carry its own confirmation table: it once gave
+        # M5 an unvalidated [M1, M15] pairing that no other surface used.
+        assert config.confirmation_timeframes == _xau._CONFIRMATIONS[tf], tf_str
 
 
 # ══════════════════════════════════════════════════════════════════════
@@ -1625,9 +1610,9 @@ print("\n═══ Module 7: telegram_bot.py ═══")
 
 from telegram_bot import (
     _escape_md, _signal_label, _mtf_line, _indicator_summary,
-    format_signal_message, _format_scan_summary, _news_text,
+    format_signal_message, _news_text,
     _seconds_until_boundary, _check_level_breach,
-    scan_keyboard, SCAN_BUTTONS, SCAN_LOOKUP,
+    SCAN_BUTTONS, SCAN_LOOKUP,
     _update_level_cache, _last_prices, _level_cache, _level_cache_ts,
 )
 
@@ -1770,21 +1755,6 @@ def _():
     assert "[Bullish]" not in msg and "[Bearish]" not in msg and "[Neutral]" not in msg
 
 
-@test("_format_scan_summary produces valid text")
-def _():
-    config = PlatformConfig()
-    agg = SignalAggregator(config)
-    dfs = {
-        Timeframe.H1: make_ohlcv(300, seed=42),
-        Timeframe.M15: make_ohlcv(300, seed=43),
-        Timeframe.H4: make_ohlcv(300, seed=44),
-    }
-    reports = [agg.analyze_from_dataframes("XAU/USD", dfs, use_news=False)]
-    summary = _format_scan_summary(reports, Timeframe.H1)
-    assert "Auto-Scan" in summary
-    assert "XAU/USD" in summary
-
-
 @test("_seconds_until_boundary returns positive value")
 def _():
     result = _seconds_until_boundary(15, buffer=10)
@@ -1891,12 +1861,6 @@ def _():
         assert "label" in btn
 
 
-@test("scan_keyboard returns InlineKeyboardMarkup")
-def _():
-    kb = scan_keyboard()
-    assert kb is not None
-
-
 @test("_update_level_cache populates cache from reports")
 def _():
     config = PlatformConfig()
@@ -2083,7 +2047,7 @@ print("\n═══ Module 9: pair_signals registry ═══")
 @test("pair_signals registry includes XAU/USD only")
 def _():
     import pair_signals
-    pairs = pair_signals.get_supported_pairs()
+    pairs = list(pair_signals.PAIR_PRODUCERS)
     assert XAUUSD in pairs, f"XAU missing from registry: {pairs}"
     assert pairs == [XAUUSD], f"registry must be XAU-only: {pairs}"
 
@@ -2101,7 +2065,6 @@ def _():
         inline_cfg = PlatformConfig(
             symbol=XAUUSD,
             display_name="XAU/USD",
-            pip_value=0.10,
             primary_timeframe=tf,
             confirmation_timeframes=ctfs,
         )
@@ -2777,7 +2740,6 @@ def _():
 
 @test("CP: cp_patterns subset filter excludes disabled detectors")
 def _():
-    from dataclasses import replace
     df = make_cp_df(double_bottom_closes())
     cfg = XauStrategyConfig(cp_patterns=["bull_flag"])
     assert detect_chart_pattern(df, atr=5.0, cfg=cfg) is None
@@ -3289,10 +3251,12 @@ def _():
         assert deepseek_api_key(AppConfig()) == ""
 
 
-@test("config.py: hardcoded twelve_data_api_key field is gone")
+@test("config.py: removed PlatformConfig fields stay gone (twelve_data_api_key, pip_value)")
 def _():
     assert not hasattr(PlatformConfig(), "twelve_data_api_key"), \
         "credentials must not live in committed source"
+    # pip value comes from PAIR_INFO; the config field was never read (2026-09-06).
+    assert not hasattr(PlatformConfig(), "pip_value")
 
 
 @test("telegram_bot: credentials, quality, pairs, loops from app_config")
@@ -3626,7 +3590,11 @@ def _():
     provider = providers.oanda.OandaDataProvider.__new__(
         providers.oanda.OandaDataProvider)
     assert provider._to_oanda_instrument("XAU/USD") == "XAU_USD"
-    assert provider._to_oanda_instrument("EUR/USD") == "EUR_USD"
+    try:
+        provider._to_oanda_instrument("EUR/USD")
+        assert False, "unregistered symbol must raise, not be guessed"
+    except DataIntegrityError:
+        pass
 
 
 @test("providers.oanda: granularity mapping covers all scan timeframes")
@@ -5180,8 +5148,7 @@ def _():
     assert d["event"] == "Non-Farm Payrolls"
     assert d["actual"] == "228K"            # strings stay strings
     assert d["surprise"] == "positive"      # property, added explicitly
-    assert d["is_high_impact"] is True
+    assert "minutes_until" not in d           # frozen-at-publish value, dropped
-    assert isinstance(d["minutes_until"], float)
     assert d["time"] == "2026-08-07T12:30:00+00:00"
     json.dumps(d, allow_nan=False)
 
@@ -5318,9 +5285,9 @@ def _():
 
 
 # ══════════════════════════════════════════════════════════════════════
-#  MODULE 21: 2026-08-04 audit-fix regressions (execution lifecycle)
+#  MODULE 21: 2026-08-04 audit-fix regressions (parity, providers, backtester)
 # ══════════════════════════════════════════════════════════════════════
-print("\n═══ Module 21: 2026-08-04 audit fixes — execution lifecycle ═══")
+print("\n═══ Module 21: 2026-08-04 audit fixes — parity, providers, backtester ═══")
 
 
 @test("GPU parity: ALL indicators + composite identical on every prefix")
@@ -7757,7 +7724,6 @@ def _():
 @test("signal_log: analyze_pair weekend early-return is logged")
 def _():
     import signal_log as slog
-    from signal_aggregator import _is_forex_weekend
     fd, path = tempfile.mkstemp(suffix=".db")
     os.close(fd)
     os.remove(path)
@@ -8177,7 +8143,7 @@ def _():
                                      "stance": "support"}])))
         con.commit()
         con.close()
-        buckets = asl.news_outcome_buckets(rows, events, path)
+        buckets = asl.news_outcome_buckets(events, path)
     finally:
         os.remove(path)
     assert sum(b["n"] for b in buckets) == 3
@@ -8915,7 +8881,7 @@ def _():
         legacy_id = lg._conn.execute("SELECT MAX(id) FROM signals").fetchone()[0]
         ev, hl = slog.read_news(lg._conn, legacy_id)
         assert json.loads(hl)[0]["headline"] == "legacy"
-        ev, hl = lg.news_for(r1)
+        ev, hl = slog.read_news(lg._conn, r1)
         assert json.loads(hl)[0]["headline"] == "Fed signals rate hike"
         view = dict(lg._conn.execute(
             "SELECT id, headlines_resolved FROM signals_with_news").fetchall())
@@ -9120,7 +9086,7 @@ def _():
         ev = _types.SimpleNamespace(
             rows=[_types.SimpleNamespace(id=1), _types.SimpleNamespace(id=2)],
             r_headline=2.0)
-        out = asl.persona_scorecard([], [ev], path)
+        out = asl.persona_scorecard([ev], path)
         assert len(out) == 1, out
         assert out[0]["n"] == 1, f"two rows of ONE event must count once: {out}"
         assert out[0]["mean_r"] == 2.0