audit 2026-09-13: eleven delivery/measurement fixes; books byte-identical

Signal path - A shipped setup always names a direction: grade C needs confidence >= 45 (gate 6c grants B at >= 45) while BUY/SELL needs >= 50, so C/B reports reached the alert gate as NEUTRAL with a full setup. The backtester traded them (the replay never reads final_signal) while Telegram rejected NEUTRAL first — 5/91 trades, +3.7R of +22.2R, in the H1 12-month book. _attach_core_setup stamps the setup's direction and repairs the pre-gate action text; no-setup reports stay NEUTRAL. - analyze_pair fails the scan when ANY required frame is missing ("DATA UNAVAILABLE:", scan_kind='failed'); a lost confirmation frame used to run with an unearned -15 MTF penalty. Outcomes and labels - advisory_outcome.resolve_locked_floor: a gap the database cannot fill resolves at the locked trailing stop (GAP_LOCKED, result.floor) when the stop had crossed the entry; the rebuild fires on both gap reasons (rows 915/920: +1.14R / +0.84R floors; resolved sum -2.46R -> -0.48R). - label_scan_outcomes: SCAN_LABEL_MAX_AGE_H = 72 h, stale rows never anchor the shared bar stream, labeled_at bumped on re-examination, _stream_since warns at its row cap (the Labor-Day pile would have blinded scan_outcomes around 2026-09-14). Providers and scheduling - The daily settlement pause (Mon-Thu 17:00 New York) is a scheduled closure for _has_latest_closed_bar (whole-span label walk-back) and _is_data_fresh (closed seconds are not age); bot and dashboard skip scans whose just-closed label sits in the pause (61 false stale rows in three nights). - fetch_range retries a failed page in place (three failures on one window raise) and logs an ERROR for interior gaps of >3 unscheduled bars. - pd.Timestamp.utcnow() replaced in providers/base.py and engine.py; requirements pin pandas<4, numpy<3. - telegram_bot: boundary loops share one scan per bar with the breach handler (_remember_scan keys on the analysed bar; breach defers 20 s after an M15 boundary); the rotating file handler attaches in main() only (the test suite used to write into logs/telegram.log). - dashboard_publisher: catch_up_timeframes re-scans boundaries missed while a cycle overran or the process was down. Backtester and engine guards - _replay_span returns the per-span GPU cache-miss count; run_backtest sums, logs it (must be 0) and writes cache_fallbacks to the manifest. - regime_weight_table raises on a partial regime_weights_override. Verification: 519 tests; M5/M15 2-yr, M15 OOS, H1/H4 3-yr books byte-identical before and after; live signals.db untouched (checked on a copy). CLAUDE.md carries the dated entry and the blocked-hours union semantics that every book was validated under. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_0139xz1J6YccndA2be6yFnEo
research: improvement cycle 1 ledger (baseline, cost ladder, C1 reject, C2 plateau, seeding parity)
2026-09-13 21:42:37 -04:00 · 2026-09-10 08:29:14 -04:00 · 2026-09-10 08:29:04 -04:00 · 2026-09-10 07:55:20 -04:00
14 changed files with 1990 additions and 92 deletions
@@ -31,3 +31,4 @@ backtest/__pycache__/
 logs/
 backtest/research/sizing_2026-09/
 backtest/research/replay_2026-09/
 backtest/research/cycle_2026-09-10/
@@ -348,6 +348,39 @@ def advance_outcome(
        return state
 def resolve_locked_floor(state: Dict[str, Any]) -> Dict[str, Any]:
    """If an UNRESOLVABLE gap outcome had already trailed its stop past the
    entry, finish it at that stop as a FLOOR (the true outcome is at least
    this good) instead of discarding it. Returns the state (mutated)."""
    if state.get("status") != "UNRESOLVABLE":
        return state
    reason = str(state.get("reason", ""))
    # Both gap reasons ("lost continuity" / "contain an internal gap")
    # share this prefix; an invalid seed or evaluator error never floors.
    if not reason.startswith("stored candle snapshots"):
        return state
    progress = state.get("progress")
    setup = state.get("setup")
    if not isinstance(progress, dict) or not isinstance(setup, dict):
        return state
    entry = _finite(progress.get("entry_price"))
    stop = _finite(progress.get("current_stop"))
    risk = _finite(setup.get("stop_distance"))
    exit_at = progress.get("last_processed_bar")
    if (entry is None or stop is None or risk is None or risk <= 0
            or not exit_at):
        return state
    buy = setup.get("direction") == "BUY"
    locked = stop >= entry if buy else stop <= entry
    if not locked:
        return state
    _finish(state, "GAP_LOCKED", str(exit_at), stop)
    state["result"]["floor"] = True
    state["reason"] = (
        f"{reason} — resolved at the locked trailing stop (floor)")
    return state
 def canonical_r(state: Dict[str, Any]) -> Optional[float]:
    result = state.get("result")
    value = result.get("net_r_065") if isinstance(result, dict) else None
@@ -973,6 +973,28 @@ class GPUTechnicalEngine(TechnicalEngine):
 CACHE_DIR = Path(__file__).resolve().parent / "cache"
 def _history_pad_days(tf: Timeframe, warmup_bars: int) -> int:
    """Calendar days of history fetched BEFORE --start for one timeframe.
    Base pad: max(60, warmup_bars // 4) days (62 at the default 250), the
    EMA200/ADX warm-up every book since 2026-07 was built on. Slow TFs
    need more: 62 days of H4 is 372 bars and of D1 only ~44, while the
    live scanner now fetches at least BaseDataProvider.MIN_LIVE_BARS
    (1000) bars on every timeframe (2026-09-10 seeding-parity fix — the
    engine's EMAs use ewm(adjust=False), so the first bar of the frame
    keeps weight (1-2/201)^n in the trend EMA). Target that same depth
    here, converting trading to calendar days with 7/5, so the window
    START is seeded like the rest of the window and like live. The
    intraday pads are unchanged (M5/M15/H1 already hold >= 1488 bars in
    62 days), which keeps their cache windows and books byte-identical;
    only H4 (234 days) and D1 (1400 days) fetch deeper.
    """
    base = max(60, warmup_bars // 4)
    bars_per_day = BaseDataProvider.BARS_PER_DAY[tf]
    seed_days = int(np.ceil(BaseDataProvider.MIN_LIVE_BARS / bars_per_day * 7 / 5))
    return max(base, seed_days)
 def _cache_path(
    symbol: str, tf: Timeframe, start: datetime, end: datetime,
    provider_name: str = "twelve_data",
@@ -1518,6 +1540,10 @@ def _replay_span(span: Tuple[int, int]):
    is applied by the caller in a sequential pass over the returned
    candidates, so parallel and sequential runs produce identical books.
    Must stay behaviorally identical to the original sequential loop.
    Returns (skipped, candidates, bar_rows, cache_fallbacks) — the last
    element is this span's GPU-cache-miss count, which the caller sums so
    a parity hazard inside a worker process cannot stay invisible.
    """
    ctx = _REPLAY_CTX
    primary_df = ctx["primary_df"]
@@ -1528,6 +1554,12 @@ def _replay_span(span: Tuple[int, int]):
    bt_cfg = ctx["bt_cfg"]
    primary_dur = ctx["primary_dur"]
    # One worker process handles several spans, so the engine's cache-miss
    # counter must be per span, not cumulative across them.
    engine = aggregator.tech_engine
    if hasattr(engine, "fallback_count"):
        engine.fallback_count = 0
    skipped: Dict[str, int] = {}
    candidates: List[Tuple[int, Trade]] = []
    is_xau = PAIR_INFO[bt_cfg.pair]["display"] == "XAU/USD"
@@ -1697,8 +1729,9 @@ def _replay_span(span: Tuple[int, int]):
                          if tag == "XAU_PB" else 0.0),
        )))
-    return skipped, candidates, (replay_record.finish_bar_rows(bar_buf)
+    return (skipped, candidates,
-                                 if record_bars else None)
+            (replay_record.finish_bar_rows(bar_buf) if record_bars else None),
            int(getattr(engine, "fallback_count", 0)))
 def _reanchor_entries_to_raw(
@@ -1817,23 +1850,17 @@ def run_backtest(
    flt = platform_cfg.backtest
    if flt.regime_filter and flt.regime_filter_timeframe not in timeframes:
        timeframes.append(flt.regime_filter_timeframe)
    # Pad lookback so earliest TF has enough warmup for EMA200, ADX, etc.
    pad_days = max(60, bt_cfg.warmup_bars // 4)
    fetch_start = bt_cfg.start - timedelta(days=pad_days)
    dfs_full: Dict[Timeframe, pd.DataFrame] = {}
    # Which parquet each TF came from (recorded in --record's manifest so
    # a study can prove two runs read the same bars).
    cache_names: Dict[str, str] = {}
    # Per-TF history start; the raw-path fetch below reuses the primary's
    # so the cleaned and raw frames come from the SAME cache file.
    tf_starts: Dict[Timeframe, datetime] = {}
    for tf in timeframes:
-        # Slow TFs need a longer pad: 60 days of D1 is only ~44 bars — no
+        tf_start = bt_cfg.start - timedelta(
-        # EMA200, truncated swing/divergence windows — while live always has
+            days=_history_pad_days(tf, bt_cfg.warmup_bars))
-        # ≥210 bars (outputsize floor). Target data_provider's min_required
+        tf_starts[tf] = tf_start
        # (210 bars); ×7/5 converts trading days to calendar days. Intraday
        # TFs stay on the base pad, so their books are unchanged.
        bars_per_day = BaseDataProvider.BARS_PER_DAY[tf]
        min_pad_days = int(np.ceil(210 / bars_per_day * 7 / 5)) + 5
        tf_start = bt_cfg.start - timedelta(days=max(pad_days, min_pad_days))
        cache_names[tf.value] = _cache_path(
            bt_cfg.pair, tf, tf_start, bt_cfg.end, provider.name).name
        df = fetch_history(
@@ -1911,14 +1938,28 @@ def run_backtest(
    skipped_reasons: Dict[str, int] = {}
    candidates: List[Tuple[int, Trade]] = []
    bar_chunks: List = []
-    for skips, cands, bars in results:
+    cache_fallbacks = 0
    for skips, cands, bars, fallbacks in results:
        for reason, count in skips.items():
            skipped_reasons[reason] = skipped_reasons.get(reason, 0) + count
        candidates.extend(cands)
        if bars is not None:
            bar_chunks.append(bars)
        cache_fallbacks += fallbacks
    candidates.sort(key=lambda pc: pc[0])
    # The workers' counters die with their processes, so surface the run
    # total here and leave it on the parent engine for callers to read.
    aggregator.tech_engine.fallback_count = cache_fallbacks
    if cache_fallbacks > 0:
        logger.warning(
            f"GPU cache fallbacks: {cache_fallbacks} indicator recomputes "
            f"(one per _compute_* override per missed bar) fell back to the "
            f"live engine's pandas math (different EWM seeding) — parity "
            f"hazard; see GPUTechnicalEngine._note_fallback")
    else:
        logger.info("GPU cache fallbacks: 0")
    # ── Phase 2: entry cooldown — the only sequential rule.
    accepted = _apply_cooldown(candidates, bt_cfg.cooldown_bars)
    trades: List[Trade] = [t for _, t in accepted]
@@ -1939,7 +1980,7 @@ def run_backtest(
    # (news candles) that hit stops in reality. Cache hit → no API call.
    sim_df = fetch_history(
        provider, bt_cfg.pair, bt_cfg.primary_tf,
-        start=fetch_start, end=bt_cfg.end,
+        start=tf_starts[bt_cfg.primary_tf], end=bt_cfg.end,
        use_cache=bt_cfg.use_cache, clean=False,
    )
    _reanchor_entries_to_raw(trades, signal_ts_list, sim_df)
@@ -1969,7 +2010,8 @@ def run_backtest(
                          skipped=skipped_reasons, n_candidates=len(candidates),
                          workers=workers,
                          backend="CuPy GPU" if _GPU_AVAILABLE else "NumPy CPU fallback",
-                          cache_files=cache_names, replay_seconds=replay_seconds),
+                          cache_files=cache_names, replay_seconds=replay_seconds,
                          cache_fallbacks=cache_fallbacks),
            sim_df=sim_df, bar_chunks=bar_chunks or None)
        logger.info(f"--record: wrote run to {bt_cfg.record_dir}")
@@ -1055,6 +1055,9 @@ def write_run(record_dir, *, trades, book_df, bt_cfg, platform_cfg, aggregator,
        "max_hold_bars": bt_cfg.max_hold_bars,
        "fixed_rr_tp": bt_cfg.fixed_rr_tp,
        "workers": run_info.get("workers"),
        # >0 means some bar-analyses fell back to the live engine's pandas
        # math instead of the pre-computed cache (a parity hazard).
        "cache_fallbacks": run_info.get("cache_fallbacks"),
        "engine_fingerprint": engine_fp,
        "gate_fingerprint": gate_fp,
        "technical_config": dataclasses.asdict(platform_cfg.technical),
@@ -0,0 +1,287 @@
 # Improvement cycle 1 — 2026-09-10 (HEAD baseline, robustness challenge, seeding parity)
 ## Purpose
 First cycle under the owner's standing brief (evidence-driven incremental improvement of
 the existing XAU/USD system). No new strategy idea is tested. The cycle (1) re-establishes
 the baseline at HEAD and proves the harness reproduces the pinned books, (2) challenges the
 current edge — cost sensitivity, concentration, and the parameter neighbourhood of the M15
 RSI × 0.5 change shipped 2026-09-09, (3) reruns the one parked candidate with prior rule-1
 evidence (H1 divergence off) through the production vehicle it now has, and (4) measures
 and fixes a live-vs-backtest indicator-seeding gap (H4 fetches 380 bars, D1 300, while the
 backtester seeds from its whole cached history). Everything under "Pre-registered" was
 written before the corresponding run.
 ## Harness
 `venv/bin/python backtest/gpu_backtest.py --pair XAUUSD --spread 0.65 --workers 12
 --primary-tf {5min|15min|1h|4h}`, pair-factory config (no `--quality/--strategies/
 --confirm-tfs`). Windows: **W3** 2023-09-05→2026-09-05 (halves at 2025-03-05; last-12-months
 slice from 2025-09-05 reported descriptively), **W2** 2024-09-05→2026-09-05 (halves at
 2025-09-05), **OOS** 2024-06-01→2025-03-30. Code: git `2ac7301` (branch `sql-integration`).
 Backend: NumPy CPU (CuPy imports but has no CUDA toolkit on this box). Bars: cached
 `backtest/cache/oanda_XAU_USD_{5min,15min,1h,4h}_20230705_20260905.parquet`,
 `_1d_20221110_20260905`, `_{5min,15min,1h}_20240705_20260905`, `_{…}_20240331_20250330`.
 Outputs: `backtest/research/cycle_2026-09-10/` (gitignored). Every candidate is compared
 with a CONTROL produced on this harness in this session. `--set
 technical.regime_weights_override=<JSON>` replaces the WHOLE table, so each candidate
 passes the complete table with one indicator's multipliers changed.
 ## Pre-registered rules (written 2026-09-10 before any run)
 ### Step 0 — reproducibility gate
 The HEAD factory run must reproduce the ledger's MD5s: M5 W2
 `29120a1ee32979431d062f210d41a9d1`, M15 W2 `d185f099482a07f29502cb5a371a9bbf`, M15 OOS
 `6f8f24b49597ced28f50a19e08c3472f`. A mismatch stops the cycle until explained.
 ### Step 1 — diagnostics (descriptive; flags fixed in advance)
 - Cost ladder $0.65 / 0.80 / 1.00 / 1.30 on W3 per TF. Red flag for a TF: at $0.80 PF < 1.3
  or either W3 half ≤ 0.
 - Concentration/stability: share of total R in the top 5% of trades, % of calendar months
  positive, longest losing streak, max DD and R per calendar year. No action threshold.
 - Bucket screen over the recorded W3 trades (direction, quality, regime, strategy,
  entry bar-close hour, weekday, `us_holiday`, `ev_entry_within_30m_hi_usd`,
  `ev_hi_usd_during_hold`, year/quarter; plus M15/H4 SELL-in-transitioning and holiday
  trades). A bucket becomes a cycle-2 candidate ONLY if n ≥ 30, R < 0 in each of the three
  12-month slices AND in OOS, and a mechanism can be stated. No gate is built this cycle.
 ### C1 — H1 divergence multiplier → 0 in every regime (H1-only)
 Vehicle: `--set technical.regime_weights_override=<default table with divergence 0.0 in
 all five regimes>`, primary 1h, W3 + OOS, control = the Step-0 HEAD run.
 PASS only if ALL: (a) ΔR ≥ +5R on W3; (b) ΔR > 0 in both W3 halves; (c) PF ≥ 1.3 and not
 more than 0.05 below control; (d) max DD ≤ control + 2R; (e) trade count within ±25%;
 (f) OOS ΔR ≥ 0. Last-12-months slice reported, not a clause. No rescue, no regime-family
 fine-tuning after seeing the result. A pass is a shipping PROPOSAL for the owner.
 ### C2 — M15 RSI multiplier neighbourhood (stability of the live change)
 Ladder k ∈ {0, 0.25, 0.5 (HEAD), 0.75, 1.0 (pre-2026-09-09), 1.5}, full table with
 rsi × k, on W2, OOS and W3. Reading, with G(k) = R(k) − R(1.0) on W2: **stable plateau** if
 G(0.25) ≥ 0.5·G(0.5) and G(0.75) ≥ 0.5·G(0.5) and the OOS deltas at 0.25 and 0.75 are ≥ 0;
 **spike** if either neighbour retains < 50% of G(0.5). Outcome is a report to the owner;
 no different k is proposed whatever the result.
 ### Step 3 — seeding-depth parity
 3a: on the deepest cached H4/D1 frames (H1 as control), sampled bars over the last two
 years, compare `TechnicalEngine.analyze()` on the trailing N bars with the full-history
 result; `MIN_SEED_BARS` = smallest N with zero indicator-signal flips on both H4 and D1.
 3b: live per-TF fetch floor raised to that depth (no book changes; the backtester is the
 reference). 3c: backtester slow-TF pad raised to the same depth, shipped only if the M5
 and M15 W3 books stay byte-identical and the H1/H4 differences are confined to the
 window-start seeding zone; the H1/H4 re-pin is the owner's decision.
 ## Results
 (filled in as runs complete)
 ### Step 0 — HEAD baseline (W3, $0.65, factory config; recorded with `--record`)
 | TF | trades | total R | WR | PF | max DD | half 1 (→2025-03-05) | half 2 | last 12 mo | SELL bucket | book MD5 |
 |---|---|---|---|---|---|---|---|---|---|---|
 | M5 | 870 | +153.64 | 53.3% | 1.50 | 10.1 | 136 / +16.73 / PF 1.36 | 734 / +136.91 / 1.52 | 593 / +119.65 / 1.57 | 100 / +8.95 / 1.25 | `c2722f9830528658b53854f6bae61b8c` |
 | M15 | 1273 | +186.61 | 59.4% | 1.39 | 19.3 | 556 / +75.87 / 1.37 | 717 / +110.74 / 1.41 | 474 / +81.04 / 1.47 | 404 / −15.18 / 0.91 | `b2cfa9e2e6cd3997f4d5ca9b49cab90e` |
 | H1 | 253 | +70.02 | 64.4% | 1.98 | 4.4 | 115 / +26.26 / 1.79 | 138 / +43.76 / 2.15 | 89 / +24.62 / 1.96 | 0 (all shorts blocked) | `b99111132d9ecb80c2998ed5aee2f3ea` |
 | H4 | 114 | +21.68 | 57.9% | 1.59 | 5.4 | 52 / +4.91 / 1.26 | 62 / +16.77 / 1.94 | 46 / +12.34 / 1.79 | 20 / −0.97 / 0.89 | `4508e7a3402e008ba1d93805094cd981` |
 These equal the numbers CLAUDE.md records for HEAD (M5 870/+153.6/1.50, M15 1273/+186.61/1.39,
 H1 253/+70.0/1.98, H4 114/+21.7/1.59). Engine / gate fingerprints from the manifests:
 M5 `f28b1a8c409d`/`bc0991492340`, M15 `536f84340ea0`/`b2920991cdb8` (new since the RSI
 override), H1 `6efb6d365f01`/`0b67d738dc15`, H4 `ed96f258b10e`/`0c3b51f063f9`.
 `replay_scorecard.py --check` on the four recorded runs: see `step0.out`.
 Per calendar year (R / PF): M5 2023 +2.4/1.47, 2024 **+1.0/1.02** (102 trades — a flat
 year), 2025 +40.5/1.34, 2026 +109.7/1.77; M15 +13.2/1.40, +52.9/1.36, +77.3/1.43,
 +43.2/1.37; H1 +5.5/2.60, +14.7/1.58, +33.7/2.11, +16.1/2.35; H4 +1.2/1.28, +4.6/1.39,
 +7.7/1.74, +8.2/1.78.
 ### Step 3a — seeding-depth measurement (`seed_depth.out`)
 Deepest Oanda caches (H4 12,419 bars, D1 2,617, H1 45,914; 2018→2026-08-06), bars sampled
 every 12th (H4, 267 bars) / 2nd (D1, 313) / 48th (H1, 247) over the last two years;
 `TechnicalEngine.analyze()` on the trailing N bars vs on the full history; identical
 results with raw frames and with frames cleaned by `validate_and_clean` (the frame-wide
 σ of the outlier filter did not add a single flip).
 | TF | N = live depth today | bars with ≥1 indicator flip | which | max composite Δ | N = 600 … 5000 |
 |---|---|---|---|---|---|
 | H4 | 380 | 2 / 267 | ema_trend | 4.66 | 0 flips at every depth |
 | D1 | 300 | 1 / 313 | ema_trend | 9.47 | 0 flips at every depth |
 | H1 | 1460 (control) | 0 / 247 | — | 0.00 | 0 |
 Only the 200-bar EMA trend reading is affected, as the `adjust=False` seeding predicts;
 composite-signal and regime never flipped in the sample. Pre-registered pick = smallest
 zero-flip depth = 600; shipped floor = **1000** (seed weight e^-10, a ×40 margin on a
 sample of a few hundred bars; costs nothing — one request either way). The 2026-08-04
 audit's "~2% of H4 bars" was measured on a different sample; this sample says 0.75%
 (H4) and 0.3% (D1) of bars, with composite errors up to 9.5 points on the ones that flip.
 ### Step 1 — concentration, stability, bucket screen (W3 recorded books, `diag_w3_head.txt`)
 | TF | top 5% of trades carry | months positive | longest losing streak |
 |---|---|---|---|
 | M5 | 94% of total R (44 trades, +144R of +154R) | 62% (23/37) | 8 |
 | M15 | 96% (64 trades, +179R of +187R) | 68% (25/37) | 9 |
 | H1 | 48% (13 trades) | 71% (25/35) | 4 |
 | H4 | 80% (6 trades) | 50% (15/30) | 6 |
 The intraday edge is a fat right tail: trailing exits let a few dozen runners pay for
 everything else (M5 2024 was flat over 102 trades). H1 is the only timeframe whose edge is
 spread across its book.
 Bucket screen (pre-registered criterion: n ≥ 30 and R < 0 in each 12-month slice y1/y2/y3
 AND in the OOS slice of the same book). Exactly one bucket qualifies (besides the trivial
 `exit_reason = SL`):
 - **M15 SELL grade A**: 230 trades, −14.4R, PF 0.86, y1 −7.4 (67) / y2 −0.4 (80) /
  y3 −6.5 (83) / OOS −6.9 (62). Context: M15 SELL overall 404 / −15.2R / PF 0.91 (SELL
  max DD 36R on its own curve); SELL|B 160 / −5.5R (y3 +4.6 — not qualifying);
  SELL|A+ 14 / +4.7R; SELL|trending_down 301 / −10.9R (y2 +0.9 — not qualifying).
  This is the same bucket the 2026-09-03 campaign tested as `sell_min_quality=["B"]`
  (allow only B-grade M15 shorts) and REJECTED: 12mo +0.7R, 5-yr −8.9R with max DD 35.8 →
  41.4R, OOS +8.3R — A-grade M15 shorts paid in 2021–2023. Under the owner's ≤3-year
  window they lose in every slice. Recorded as the ONE cycle-2 candidate from this screen
  (M15 `sell_min_quality` excluding "A" via the factory), to be run against a control with
  the 3-yr DD clause; the 5-yr verdict is the fact on the other side of the scale. Not
  built this cycle (rule: no gate from the screen).
 - Not qualifying / no evidence: holiday-day trades are POSITIVE on M15 (27 / +10.6R) and
  small everywhere; entries within 30 min after a high-impact USD release are positive
  (M15 114 / +9.3R); M15 SELL-in-transitioning (the gate-6c bucket) 75 / −2.2R with y2
  and y3 positive; M15 `vol_regime = high` 291 / +4.1R / PF 1.03 (y3 +12.6); hour 7 UTC
  −4.0R (OOS +2.4). M5/H1/H4 produced no qualifying bucket.
 ### Step 0 — reproducibility gate: PASS
 | book | MD5 at HEAD | ledger MD5 | result |
 |---|---|---|---|
 | M5 W2 | `29120a1ee32979431d062f210d41a9d1` | same | equal |
 | M15 W2 | `d185f099482a07f29502cb5a371a9bbf` | same | equal |
 | M15 OOS | `6f8f24b49597ced28f50a19e08c3472f` | same | equal |
 | H1 OOS (new control, no prior MD5) | `c02eebc9066916d7637bbf5796993f87` | — | recorded |
 `replay_scorecard.py --check`: all eight invariants hold on every recorded W3 run.
 ### Step 3b — live floor shipped (commit `9942736`) and verified on the live feed
 `BaseDataProvider.MIN_LIVE_BARS = 1000`, applied in a new pure helper
 `_live_outputsize(timeframe, lookback_days)`; requested depths now M5 5000, M15 5000,
 H1 1460 (unchanged), **H4 1000 (was 380), D1 1000 (was 300)**. The backtester never calls
 `fetch_ohlcv`, so no book changes. Tests: new Module 16 depth test, Module 21 cache test
 updated from 300 to 1000; suite **500 passed**.
 Live verification 2026-09-10 (real Oanda fetch, same latest closed bar, engine on the
 frame cleaned exactly as live does):
 - H4 (bar 2026-09-10 04:00 UTC): 380 / 1000 / 5000 bars → identical readings
  (composite +15.03, ranging) — no flip on today's bar.
 - D1 (bar 2026-09-09): **300 bars → ema_trend SELL, composite −9.53; 1000 and 5000 bars →
  ema_trend NEUTRAL, composite −8.87.** The old depth was mis-reading the daily trend
  today; the new floor agrees with the converged value.
 ### Step 1 — cost ladder (W3, same trade lists; `--spread` only re-prices; mechanics check: the H1 $1.00 total predicted from the $0.65 book (−0.35 $/trade ÷ risk) equals the rerun to the cent)
 | TF | $0.65 R / PF / DD | $0.80 R / PF / DD (halves) | $1.00 R / PF / DD | $1.30 R / PF / DD | red flag ($0.80: PF < 1.3 or a half ≤ 0)? |
 |---|---|---|---|---|---|
 | M5 | +153.6 / 1.50 / 10.1 | +145.6 / 1.46 / 11.0 (+14.9 / +130.7) | +134.9 / 1.42 / 12.2 | +118.8 / 1.36 / 14.1 | no |
 | M15 | +186.6 / 1.39 / 19.3 | +176.4 / 1.37 / 19.9 (+70.0 / +106.4) | +162.8 / 1.34 / 20.8 | +142.4 / 1.29 / 22.6 | no (PF dips under 1.3 only at $1.30) |
 | H1 | +70.0 / 1.98 / 4.4 | +68.9 / 1.96 / 4.5 (+25.6 / +43.2) | +67.3 / 1.93 / 4.5 | +65.0 / 1.89 / 4.7 | no |
 | H4 | +21.7 / 1.59 / 5.4 | +21.3 / 1.57 / 5.5 (+4.7 / +16.6) | +20.8 / 1.56 / 5.7 | +20.1 / 1.53 / 6.0 | no |
 Doubling the assumed round-trip cost ($0.65 → $1.30) costs M5 23% and M15 24% of total R
 and H1/H4 about 7%; the H1 book is the least cost-sensitive because its stops are the
 widest in dollars. The M5 first half is thin at every price (+8.7R at $1.30 over 136
 trades), which is the 2024 flat year again, not a cost effect.
 ### C1 — H1 divergence multiplier → 0 in every regime: **FAIL (a, b, c) — closed**
 | book | trades | total R | ΔR | half 1 | half 2 | PF | max DD | last 12 mo | OOS |
 |---|---|---|---|---|---|---|---|---|---|
 | H1 control (HEAD, W3) | 253 | +70.02 | | +26.26 | +43.76 | 1.98 | 4.4 | +24.62 (89) | 78 / +16.68 / PF 1.74 |
 | H1 divergence 0 (W3) | 280 | +71.01 | **+0.99** | +28.70 (+2.4) | +42.30 (**−1.5**) | **1.89** | 4.8 | +23.09 (102) | 88 / +21.91 / PF 1.92 (+5.2) |
 Clauses: (a) ΔR +1.0 < +5 FAIL; (b) second half negative FAIL; (c) PF 1.98 → 1.89, drop
 0.09 > 0.05 FAIL; (d) DD +0.4 pass; (e) trades +10.7% pass; (f) OOS +5.2 pass. Calendar
 years: 2024 +14.7 → +19.1, 2025 +33.7 → +34.1, 2026 +16.1 → +12.4 — the gain sits in
 2024, the most recent year gives it back. The OOS candidate book (88 / +21.91 / PF 1.92)
 is exactly the 2026-09-03 campaign's `b12_h1_nosell_nodiv_oos` under
 `disabled_indicators` — so a 0.0 regime multiplier is mechanically the same as disabling
 the indicator for the composite (engine skips only base weight 0, but an effective weight
 of 0 adds nothing to score, weight sum or confidence); the only residual difference is
 display (reasons text still lists the vote). The parked candidate's earlier evidence
 (5-yr +9.6R over no-shorts) was carried by 2021–2023; on the owner's 3-year window it is
 noise with a worse profit factor. Not shipped; do not re-propose without new years.
 ### C2 — M15 RSI multiplier neighbourhood: **STABLE PLATEAU** (the 2026-09-09 ship stands)
 Full regime table with the RSI multiplier × k (k = 1.0 is the pre-2026-09-09 table,
 k = 0.5 is HEAD; every other point is a `--set` run on the same harness).
 | k | W2 trades / R / PF / DD (halves) | G(k) = R − R(1.0) on W2 | OOS trades / R / PF / DD | ΔOOS | W3 trades / R / PF / DD (halves) |
 |---|---|---|---|---|---|
 | 0 (RSI off) | 1091 / +145.46 / 1.35 / 18.5 (+74.1 / +71.3) | +27.8 | 418 / +51.89 / 1.34 / 19.5 | +11.3 | 1490 / +188.14 / 1.33 / 22.7 (+69.3 / +118.8) |
 | 0.25 | 1008 / +139.93 / 1.37 / 20.1 (+66.1 / +73.9) | +22.3 (78% of G(0.5)) | 390 / +59.55 / 1.44 / 17.3 | +19.0 | 1374 / +187.58 / 1.36 / 21.4 (+77.7 / +109.9) |
 | **0.5 (HEAD)** | 931 / +146.14 / 1.43 / 16.1 (+66.1 / +80.0) | **+28.5** | 355 / +60.67 / 1.49 / 17.1 | +20.1 | 1273 / +186.61 / 1.39 / 19.3 (+75.9 / +110.7) |
 | 0.75 | 854 / +137.83 / 1.45 / 14.9 (+47.6 / +90.3) | +20.2 (71%) | 326 / +55.56 / 1.49 / 15.6 | +15.0 | 1176 / +178.81 / 1.42 / 17.2 (+66.9 / +111.9) |
 | 1.0 (old) | 793 / +117.62 / 1.41 / 14.8 (+38.2 / +79.4) | 0 | 300 / +40.59 / 1.38 / 15.5 | 0 | 1092 / +151.90 / 1.38 / 15.5 (+55.7 / +96.2) |
 | 1.5 | 676 / +110.17 / 1.46 / 14.2 (+39.4 / +70.7) | −7.5 | 266 / +37.27 / 1.39 / 18.8 | −3.3 | 949 / +136.20 / 1.39 / 18.8 (+42.6 / +93.6) |
 Reading (rule fixed in advance): G(0.25) = 78% and G(0.75) = 71% of G(0.5), both ≥ 50%;
 OOS deltas +19.0 and +15.0, both ≥ 0 → **stable plateau, not a spike**. The response is
 monotone in k on every window (R falls as RSI's weight rises; trade count falls ~90 per
 0.25 step), PF peaks in the 0.5–0.75 band, and drawdown grows as k → 0 (RSI off has the
 same R as HEAD with PF 1.35 and DD 22.7 on W3). The shipped 0.5 sits inside a broad
 region, on its PF/DD-favourable side. Under this rule no other k is proposed.
 ### Step 3c — backtester history pad targets the same 1000-bar seed (`_history_pad_days`)
 Change: the per-TF warm-up pad now targets `BaseDataProvider.MIN_LIVE_BARS` (1000 bars,
 converted to calendar days with 7/5) instead of 210 bars, so a window's first bars are
 seeded like the rest of the window and like live. Intraday pads are unchanged (62 days
 already hold ≥ 1488 bars), so M5/M15/H1 keep their cache windows; H4 now pads 234 days
 (was 62 → 372 bars) and D1 1400 days (was 299 → ~210 bars for a 200-bar EMA). New caches
 `oanda_XAU_USD_4h_20230114_20260905.parquet` (5,826 bars) and
 `_1d_20191105_20260905.parquet` (2,124 bars); the first H4 chunk (2023-01-14, a Saturday)
 is legitimately empty. The raw-path fetch for trade resolution now reuses the primary
 timeframe's own padded start (one cache file per TF, as before). Test: Module 21
 "history pad seeds slow TFs like live"; suite **501 passed**.
 W3 books before → after, trade-by-trade (`bookdiff.py` with the recorded runs):
 | TF | trades / R before | trades / R after | trades added / removed / R changed | what actually differs |
 |---|---|---|---|---|
 | H1 | 253 / +70.02 | 253 / +70.02 | 0 / 0 / 0 | ONE confirmation reading: 2023-09-20 18:00 (15 days into the window), H4 ema_trend STRONG_BUY → BUY (value 1.0 → 0.5, confidence 90 → 65); that trade's blended `score` moves 0.21; nothing else in 5,761 recorded indicator rows |
 | H4 | 114 / +21.68 | 114 / +21.68 | 0 / 0 / 0 | no signal flips; 20 D1 rows (ADX/RSI/MACD Wilder recursions) differ at the 1e-6…1e-9 level, all Oct–Dec 2023; exported floats differ by ≤ 1.7e-10 (score ≤ 1.4e-9, R ≤ 1e-15) |
 Both differences sit in the window-start seeding zone, exactly where the deeper seed was
 meant to act, and neither moves a trade. The canonical H1 book therefore needs NO re-pin
 (trade content identical); only the CSV bytes differ (H1 `3b45d12bade823178b267c13bc698458`
 vs `b99111132d9ecb80c2998ed5aee2f3ea`, H4 `e75adae13c839b15736f69c0114e6aa5` vs
 `4508e7a3402e008ba1d93805094cd981`), which the parity recipe must state.
 M5 and M15 on the patched harness: **byte-identical** to Step 0 (M5
 `c2722f9830528658b53854f6bae61b8c`, M15 `b2cfa9e2e6cd3997f4d5ca9b49cab90e`) — the pad
 change did not leak into the intraday books. Shipped (commit below); no re-pin needed.
 ## Verdicts
 | item | decision | one-line reason |
 |---|---|---|
 | Step 0 reproducibility | PASS | three ledger MD5s reproduced at HEAD; scorecard invariants hold |
 | Cost ladder | no red flag | every TF keeps PF ≥ 1.3 with positive halves at $0.80; edge survives $1.00 on all four |
 | Concentration | fact recorded | M5/M15: top 5% of trades carry 94–96% of R; H1 48% |
 | Bucket screen | one cycle-2 candidate | M15 grade-A SELLs (230 / −14.4R, negative in y1, y2, y3 and OOS); 5-yr history says the opposite — run it against a control with the DD clause before believing it |
 | C1 H1 divergence → 0 | **REJECT** | ΔR +1.0 (< +5), second half −1.5R, PF 1.98 → 1.89; OOS-only gain |
 | C2 M15 RSI ladder | stable plateau | neighbours keep 78% / 71% of the shipped gain, OOS positive at both; ship stands |
 | 3a/3b live seeding floor | **ACCEPT, shipped** (`9942736`) | measured flips at 380/300 bars, none at ≥ 600; live D1 flipped on 2026-09-10 at the old depth; books untouched |
 | 3c backtester pad | **ACCEPT, shipped** | no trade moved in any book; M5/M15 byte-identical; H1/H4 differ only in window-start numerics |
 ## What NOT to re-propose (added to the ledger's closed list)
 - H1 divergence weight 0 (any vehicle) without new years of data.
 - M15 RSI multiplier other than 0.5 on the strength of a single-point rerun; the ladder is on record.
 - Cost-driven gates: the books are not cost-fragile at $0.80–$1.00.
 ## Shipped
 - `9942736` providers: `MIN_LIVE_BARS = 1000` live floor + `_live_outputsize` (Module 16
  test; Module 21 expectation 300 → 1000). Suite 500.
 - backtester: `_history_pad_days` (62 / 234 / 1400 days for intraday / H4 / D1), raw-path
  fetch reuses the primary's start (Module 21 test). Suite 501. Live H1/H4 scans and the
  books now share the same seeding depth; pinned H1/H4 CSVs are trade-identical but no
  longer byte-reproducible (float noise in exported indicator columns — same status the
  M5/M15 pins have had since 2026-09-07).
 - No signal-core change, no docs change (nothing user-visible moved), both news gating
  flags untouched.
@@ -26,6 +26,11 @@ whichever timeframes just closed. Due-classification uses the RAW wake
 time — flooring 12:00:10 to the 5-minute grid yields the 12:00 boundary
 that just closed. Do NOT subtract the buffer first: 12:00:10 − 15 s
 floors to 11:55 and silently skips every H1/H4 scan (v1 bug, 2026-08-02).
 A cycle can overrun its 5-minute slot (or the process can be down), so
 each wake also catches up the timeframes whose latest closed boundary is
 newer than their last scanned one (`catch_up_timeframes`) — without it the
 page served an hours-old H1/H4 report behind a fresh countdown
 (2026-09-13 audit).
 Scans run sequentially M5→M15→H1→H4 through one shared data provider so
 later timeframes hit the warm boundary cache — that ordering IS the
@@ -46,10 +51,12 @@ import os
 import shutil
 import sys
 import time
-from datetime import datetime, timezone
+from datetime import datetime, timedelta, timezone
 from logging.handlers import TimedRotatingFileHandler
 from typing import Dict, List, Optional
 import pandas as pd
 from agent_panel import attach_panel
 from app_config import get_app_config, violent_watch_enabled
 from config import PlatformConfig, Timeframe, XAUUSD
@@ -230,10 +237,60 @@ def due_timeframes(wake_dt: datetime) -> List[Timeframe]:
    """Timeframes whose candle just closed at the boundary this wake
    belongs to. Takes the RAW wake time (boundary + buffer): flooring to
    the 5-minute grid yields the boundary that just passed. Never
-    subtract the buffer first — see module docstring (v1 bug)."""
+    subtract the buffer first — see module docstring (v1 bug).
    A timeframe whose just-closed bar would carry a label inside gold's
    daily settlement pause is skipped: no such bar exists, so the scan
    would only re-read the previous one (2026-09-13 audit)."""
    minutes = wake_dt.hour * 60 + wake_dt.minute
    floored = minutes - (minutes % 5)
-    return [tf for tf in DASHBOARD_TFS if floored % TF_INTERVAL_MINUTES[tf] == 0]
+    boundary = wake_dt.replace(
        hour=floored // 60, minute=floored % 60, second=0, microsecond=0)
    return [
        tf for tf in DASHBOARD_TFS
        if floored % TF_INTERVAL_MINUTES[tf] == 0
        and not _providers_base._is_daily_maintenance(
            boundary - timedelta(minutes=TF_INTERVAL_MINUTES[tf]))
    ]
 def catch_up_timeframes(wake_dt: datetime,
                        last_boundary: Dict[str, float]) -> List[Timeframe]:
    """Timeframes whose most recent closed boundary (<= wake) is newer than
    the boundary of their last successful scan — i.e. boundaries that were
    missed while a cycle overran or the process was down. A TF never scanned
    is due. Excludes bars whose label sits in the daily settlement pause
    (same rule as due_timeframes).
    `due_timeframes` only ever names the timeframes whose boundary coincides
    with THIS wake, so an overrunning cycle (scan + advisory panel + news can
    take minutes) or a restart left the page serving the previous H1/H4
    report for up to 1 h / 4 h while advertising a fresh next_close_epoch
    (2026-09-13 audit)."""
    minutes = wake_dt.hour * 60 + wake_dt.minute
    out = []
    for tf in DASHBOARD_TFS:
        interval = TF_INTERVAL_MINUTES[tf]
        floored = minutes - (minutes % interval)
        last_closed = wake_dt.replace(
            hour=floored // 60, minute=floored % 60, second=0, microsecond=0)
        # No bar carries a label inside a scheduled closure (settlement
        # pause or weekend), so nothing can have been "missed" there — and a
        # weekend-blocked scan never advances the ledger, so without the
        # weekend half of this test every weekend wake would re-scan
        # M15/H1/H4 for nothing (~1,800 blocked rows per weekend).
        # (Whole-span rule, as in providers.base._has_latest_closed_bar: an
        # H4 bar labelled Sunday 20:00 UTC starts in the weekend but runs
        # through the reopen, so it exists.)
        label = last_closed - timedelta(minutes=interval)
        if (_providers_base._is_scheduled_closure(label)
                and _providers_base._is_scheduled_closure(
                    last_closed - timedelta(seconds=1))):
            continue
        scanned = last_boundary.get(tf.value)
        if scanned is None or scanned < last_closed.timestamp():
            out.append(tf)
    return out
 def seconds_until_boundary(interval_minutes: int, buffer: int = 10) -> float:
@@ -324,6 +381,9 @@ class DashboardPublisher:
        self.out_dir = out_dir
        self.snapshots: Dict[str, dict] = {}
        self._reports = {}
        # tf.value → epoch seconds of the boundary the last successful scan
        # belonged to; feeds catch_up_timeframes (2026-09-13 audit).
        self._last_boundary: Dict[str, float] = {}
        self.news_payload: Optional[dict] = None
        self._dp = None
        self._news_engine: Optional[NewsEngine] = None
@@ -355,7 +415,14 @@ class DashboardPublisher:
            shutil.copyfile(src, dst + ".tmp")
            os.replace(dst + ".tmp", dst)
-    def scan_tf(self, tf: Timeframe) -> None:
+    def scan_tf(self, tf: Timeframe,
                boundary_epoch: Optional[float] = None) -> None:
        """Scan one timeframe and store its snapshot entry.
        `boundary_epoch` is the fallback boundary recorded when the report
        carries no analysis (weekend/blocked or failed scan) or its bar
        label will not parse; normally the boundary comes from the report
        itself (bar label + interval)."""
        started = time.time()
        try:
            report = produce_signal(XAUUSD, tf, self._provider(),
@@ -365,6 +432,19 @@ class DashboardPublisher:
            if violent_watch_enabled():
                attach_watch(report, tf,
                             get_app_config().violent_watch_settings)
            # Which boundary this report belongs to: the bar LABEL it closed
            # on plus one interval. Recorded so a boundary missed by an
            # overrunning cycle or a restart is caught up on the next wake
            # instead of sitting stale (2026-09-13 audit).
            boundary = boundary_epoch
            if report.primary_analysis is not None:
                try:
                    boundary = (pd.Timestamp(report.timestamp).timestamp()
                                + TF_INTERVAL_MINUTES[tf] * 60)
                except Exception:
                    boundary = boundary_epoch
            if boundary is not None:
                self._last_boundary[tf.value] = boundary
            self._reports[tf.value] = report
            self.snapshots[tf.value] = {
                "status": "ok",
@@ -449,9 +529,15 @@ class DashboardPublisher:
            logger.info(f"next M5 boundary in {delay:.0f}s")
            time.sleep(delay)
            try:
-                due = due_timeframes(datetime.now(timezone.utc))
+                now = datetime.now(timezone.utc)
                due = due_timeframes(now)
                logger.info(f"boundary wake — scanning {[t.value for t in due]}")
-                self.run_cycle(due)
+                missed = catch_up_timeframes(now, self._last_boundary)
                extra = [tf for tf in missed if tf not in due]
                if extra:
                    logger.info(f"catch-up — scanning "
                                f"{[t.value for t in extra]} (missed boundary)")
                self.run_cycle(due + extra)
            except Exception:
                logger.exception("cycle failed")
@@ -12,6 +12,7 @@ Design philosophy:
 """
 import logging
 import math
 from dataclasses import dataclass, field
 from typing import Dict, List, Optional, Tuple
@@ -84,15 +85,65 @@ _DEFAULT_REGIME_WEIGHTS = {
 }
 def _validate_regime_override(override: Dict[str, Dict[str, float]]) -> None:
    """Reject a PARTIAL ``regime_weights_override`` (raises ValueError).
    An override REPLACES the whole table, it does not patch it: whatever
    it omits is multiplied by 1.0 in _compute_composite, not by the
    shipped default. A half-filled research table (``--set
    technical.regime_weights_override=<JSON>``) would therefore silently
    run 1.0 where the default said 0.3 or 2.5, with no warning — so every
    regime and all nine scored indicators must be present, with a finite
    multiplier >= 0. The shipped M15 override is derived from the default
    table (pair_signals/xauusd._m15_regime_weights) and passes."""
    problems: List[str] = []
    want_regimes = set(_DEFAULT_REGIME_WEIGHTS)
    want_indicators = set(_TRENDING_WEIGHTS)
    got_regimes = set(override)
    if got_regimes != want_regimes:
        problems.append(
            f"missing regimes {sorted(want_regimes - got_regimes)}, "
            f"unexpected {sorted(got_regimes - want_regimes)}")
    for regime in sorted(got_regimes & want_regimes):
        weights = override[regime]
        if not isinstance(weights, dict):
            problems.append(f"regime '{regime}' is not an indicator -> "
                            f"multiplier mapping: {weights!r}")
            continue
        got = set(weights)
        if got != want_indicators:
            problems.append(
                f"regime '{regime}' missing {sorted(want_indicators - got)}, "
                f"unexpected {sorted(got - want_indicators)}")
        for name in sorted(got & want_indicators):
            mult = weights[name]
            if (isinstance(mult, bool) or not isinstance(mult, (int, float))
                    or not math.isfinite(mult) or mult < 0):
                problems.append(
                    f"regime '{regime}' indicator '{name}' multiplier "
                    f"{mult!r} is not a finite number >= 0")
    if problems:
        raise ValueError(
            "regime_weights_override is partial: " + "; ".join(problems)
            + "; an override replaces the WHOLE table "
              "(see engine._DEFAULT_REGIME_WEIGHTS)")
 def regime_weight_table(config: TechnicalConfig) -> Dict[str, Dict[str, float]]:
    """The regime -> indicator weight multipliers _compute_composite runs
    with (``regime_weights_override`` wins, else the gold-optimized
    defaults above).
    A non-empty override must cover the WHOLE table (every regime, all
    nine scored indicators) — a partial one raises ValueError instead of
    silently multiplying the cells it omits by 1.0.
    Pure and side-effect free: ONE source of truth shared by the composite
    and by signal_aggregator.engine_config_fingerprint. Returns a copy, so
    a caller can never mutate the shipped table."""
    override = getattr(config, "regime_weights_override", None)
    if override:
        _validate_regime_override(override)
    table = override if override else _DEFAULT_REGIME_WEIGHTS
    return {regime: dict(weights) for regime, weights in table.items()}
@@ -119,7 +170,7 @@ class TechnicalEngine:
            return TechnicalAnalysis(
                symbol=symbol,
                timeframe=timeframe,
-                timestamp=str(pd.Timestamp.utcnow()),
+                timestamp=str(pd.Timestamp.now(tz="UTC")),
                confidence=0.0,
            )
@@ -65,6 +65,46 @@ def _is_forex_weekend(now=None) -> bool:
    return False
 def _is_daily_maintenance(now=None) -> bool:
    """Gold's daily settlement pause: Monday-Thursday 17:00-17:59 New York
    (21:00 UTC in US summer, 22:00 in winter). No bar carries a label
    inside this hour. Friday 17:00 is the weekend (see _is_forex_weekend).
    Same rule as advisory_outcome._has_continuity's daily_maintenance."""
    if now is None:
        now = pd.Timestamp.now(tz="UTC")
    ny = now.astimezone(_NY_TZ)
    return ny.weekday() < 4 and ny.hour == 17
 def _is_scheduled_closure(now=None) -> bool:
    """The feed is legitimately silent: weekend or daily settlement pause."""
    return _is_forex_weekend(now) or _is_daily_maintenance(now)
 def _scheduled_closed_seconds(start, end) -> float:
    """Seconds of the [start, end) span that fall inside a scheduled
    closure (weekend or the daily settlement pause).
    Bar age is measured from a bar LABEL, so at 21:30 UTC in summer the
    20:55 M5 bar looks 35 minutes old even though the feed could not have
    printed anything since 21:00. Subtracting the closed span keeps the
    3-candle freshness limit meaningful instead of failing it — and
    logging a false "feed lagging"/"STALE DATA" — every settlement hour
    (2026-09-13 audit: 61 suppressed scans on three nights).
    """
    hour = pd.Timedelta(hours=1)
    closed = 0.0
    t = start.floor("h")
    steps = 0
    while t < end and steps < 24 * 10:
        if _is_scheduled_closure(t):
            closed += max(
                0.0, (min(t + hour, end) - max(t, start)).total_seconds())
        t += hour
        steps += 1
    return closed
 class DataIntegrityError(Exception):
    """Raised when data fails validation checks."""
    pass
@@ -96,6 +136,20 @@ class BaseDataProvider(ABC):
    #: Hard per-request bar cap shared by Twelve Data and Oanda.
    MAX_BARS_PER_REQUEST = 5000
    #: Minimum bars per LIVE fetch on every timeframe (2026-09-10,
    #: indicator-seeding parity with the backtester). The engine's EMAs
    #: use ewm(adjust=False), so the first bar of the fetched frame keeps
    #: weight (1-2/201)^n in the 200-bar trend EMA. With the 60-day
    #: lookback, H4 fetched 380 bars (seed weight 2.3%) and D1 300 (5%)
    #: while the backtester seeds from its whole cached history. Measured
    #: on the deepest Oanda caches (267 H4 / 313 D1 bars sampled over two
    #: years, raw and cleaned frames alike): ema_trend flipped on 2/267 H4
    #: and 1/313 D1 bars at those depths (composite off by up to 9.5
    #: points) and on none from 600 bars up; 1000 leaves a seed weight of
    #: e^-10. M5/M15 (5000) and H1 (1460) already exceed it and are
    #: unchanged. One request either way (cap 5000).
    MIN_LIVE_BARS = 1000
    # Grace wait before the one retry when a fresh response at a candle
    # boundary is missing the just-closed bar (feed latency).
    FRESH_FETCH_RETRY_SECONDS = 3.0
@@ -140,6 +194,12 @@ class BaseDataProvider(ABC):
    # ── Public API ─────────────────────────────────────────────────
    def _live_outputsize(self, timeframe: Timeframe, lookback_days: int) -> int:
        """Bars requested for one live fetch: the lookback in bars, floored
        at MIN_LIVE_BARS (indicator seeding) and capped at the request cap."""
        wanted = int(lookback_days * self.BARS_PER_DAY[timeframe]) + 20
        return max(self.MIN_LIVE_BARS, min(wanted, self.MAX_BARS_PER_REQUEST))
    def fetch_ohlcv(
        self,
        symbol: str = XAUUSD,
@@ -151,11 +211,7 @@ class BaseDataProvider(ABC):
            lookback_days = self.config.data_lookback_days
        cache_key = f"{symbol}_{timeframe.value}"
-        outputsize = max(
+        outputsize = self._live_outputsize(timeframe, lookback_days)
            300,
            min(int(lookback_days * self.BARS_PER_DAY[timeframe]) + 20,
                self.MAX_BARS_PER_REQUEST),
        )
        while True:
            now = time.time()
@@ -260,7 +316,7 @@ class BaseDataProvider(ABC):
        chunks = []
        cursor_end = end
        empty_streak = 0
-        error_streak = 0
+        window_errors = 0
        while cursor_end > start:
            cursor_start = max(start, cursor_end - timedelta(days=days_per_chunk))
            try:
@@ -270,23 +326,31 @@ class BaseDataProvider(ABC):
                )
            except Exception as e:
                # A raising chunk is a PROVIDER ERROR (auth, 429, 5xx,
-                # malformed payload) — not "no data". Three in a row means
+                # malformed payload) — not "no data". RETRY THE SAME
-                # the rest of the history is unreachable: raise instead of
+                # WINDOW: until 2026-09-13 a failure advanced the cursor
-                # returning a silently truncated frame (the backtester
+                # past the window, so one or two NON-consecutive page
-                # would treat it as complete).
+                # failures (Oanda's sporadic 401/5xx, even after the
                # provider's own retry ladder) silently dropped up to
                # days_per_chunk of bars — 15 days of M5 — out of the
                # middle of the frame, and only the HEAD was
                # coverage-checked below, so the backtester cached the hole
                # as complete history forever. Three failures on the SAME
                # window means it is unreachable: raise instead of
                # returning a frame with a hole in it.
                logger.warning(
                    f"Chunk {cursor_start.date()}–{cursor_end.date()} failed: {e}"
                )
-                error_streak += 1
+                window_errors += 1
-                if error_streak >= 3:
+                if window_errors >= 3:
                    raise self.DATA_ERROR(
-                        f"{symbol} {timeframe.value}: {error_streak} "
+                        f"{symbol} {timeframe.value}: {window_errors} "
-                        f"consecutive chunk failures walking back from "
+                        f"consecutive failures fetching window "
-                        f"{cursor_end.date()} — refusing to return "
+                        f"{cursor_start.date()}–{cursor_end.date()} — "
-                        f"truncated history (last error: {e})") from e
+                        f"refusing to return truncated history "
-                cursor_end = cursor_start - timedelta(seconds=1)
+                        f"(last error: {e})") from e
-                continue
+                continue        # re-request the SAME window
            window_errors = 0
            if chunk is None or chunk.empty:
                # Genuinely empty windows (weekends/holidays/asset gaps)
                # are normal; several in a row = start of history.
@@ -297,7 +361,6 @@ class BaseDataProvider(ABC):
                continue
            empty_streak = 0
            error_streak = 0
            chunks.append(chunk)
            # Step back; dedup on concat.
            new_cursor = chunk.index[0] - timedelta(seconds=1)
@@ -332,6 +395,38 @@ class BaseDataProvider(ABC):
                f"{df.index[0]} but {start_ts} was requested — "
                f"~{(df.index[0] - start_ts).days} days missing from the "
                f"head; treat downstream results accordingly")
        # INTERIOR-gap audit (2026-09-13): the head check above is blind to
        # a hole in the MIDDLE of the frame, which is exactly what a
        # dropped page used to leave behind. Missing labels inside a
        # scheduled closure (weekend, daily settlement pause) are normal,
        # so only unscheduled runs are reported — and a real market holiday
        # (Christmas, Good Friday, July 4) looks identical to a lost page,
        # so this is LOUD BUT NOT FATAL and the frame is returned unchanged.
        if len(df) > 400_000:
            logger.info(
                f"{symbol} {timeframe.value}: interior-gap audit skipped "
                f"({len(df)} rows)")
        elif len(df) > 1:
            step = pd.Timedelta(seconds=self.CANDLE_SECONDS[timeframe])
            labels = df.index
            # Missing labels between each consecutive pair; >3 is the floor
            # worth a look (weekends/settlement hours leave 1-2 labels that
            # the NY-anchored closure rule does not cover).
            spans = ((labels[1:] - labels[:-1]) / step).astype("int64") - 1
            gaps = []
            for i in (spans > 3).nonzero()[0]:
                missing = sum(
                    1 for k in range(1, int(spans[i]) + 1)
                    if not _is_scheduled_closure(labels[i] + k * step))
                if missing > 3:
                    gaps.append((labels[i], labels[i + 1], missing))
            if gaps:
                logger.error(
                    f"{symbol} {timeframe.value}: {len(gaps)} interior "
                    f"gap(s) of >3 unscheduled bars in fetched history — "
                    f"first: {gaps[0]}; a page fetch may have been dropped "
                    f"or the market was closed (holiday); verify before "
                    f"trusting this window")
        return df
    def is_data_fresh(
@@ -382,7 +477,8 @@ class BaseDataProvider(ABC):
    # ── Shared internals ───────────────────────────────────────────
-    def _has_latest_closed_bar(self, df: pd.DataFrame, timeframe: Timeframe) -> bool:
+    def _has_latest_closed_bar(self, df: pd.DataFrame, timeframe: Timeframe,
                               now=None) -> bool:
        """Cached data must include the most recent fully closed bar.
        The TTL alone is wall-clock based: a snapshot fetched mid-candle is
@@ -392,12 +488,29 @@ class BaseDataProvider(ABC):
        """
        if df is None or df.empty:
            return False
        if now is None:
            now = pd.Timestamp.now(tz="UTC")
        if _is_forex_weekend(now):
            return True
        candle_seconds = self.CANDLE_SECONDS.get(timeframe, 3600)
        dur = pd.Timedelta(seconds=candle_seconds)
        expected_label = now.floor(dur) - dur
        # No bar ever carries a label inside a scheduled closure, so the
        # newest bar that MUST be present is the last label before it.
        # Without this walk-back every scan from 21:05 to 22:00 UTC in
        # summer evicted the cache, refetched twice and logged a false
        # "feed lagging" (2026-09-13 audit).
        # A label counts as "inside" only when the closure covers the bar's
        # whole span: a D1 bar labelled Sunday 00:00 UTC starts in the
        # weekend but runs through the Sunday reopen, so it exists and
        # must be present (68 such bars in the 3-year Oanda D1 cache).
        steps = 0
        last_second = dur - pd.Timedelta(seconds=1)
        while (_is_scheduled_closure(expected_label)
               and _is_scheduled_closure(expected_label + last_second)
               and steps < 2000):
            expected_label -= dur
            steps += 1
        return df.index[-1] >= expected_label
    def _drop_forming_candle(self, df: pd.DataFrame, timeframe: Timeframe) -> pd.DataFrame:
@@ -427,11 +540,13 @@ class BaseDataProvider(ABC):
                timeframe.value, dropped)
        return df
-    def _is_data_fresh(self, df: pd.DataFrame, symbol: str, timeframe: Timeframe) -> bool:
+    def _is_data_fresh(self, df: pd.DataFrame, symbol: str, timeframe: Timeframe,
                       now=None) -> bool:
        """Check if the last candle is within acceptable age for this timeframe."""
        if df is None or df.empty:
            return False
        if now is None:
            now = pd.Timestamp.now(tz="UTC")
        if _is_forex_weekend(now):
@@ -440,7 +555,10 @@ class BaseDataProvider(ABC):
        last_bar = df.index[-1]
        candle_seconds = self.CANDLE_SECONDS.get(timeframe, 3600)
        max_age_seconds = candle_seconds * 3
-        age_seconds = (now - last_bar).total_seconds()
+        # Scheduled closures (weekend, daily settlement pause) are dead
        # time in which no bar can print — they must not count as age.
        age_seconds = ((now - last_bar).total_seconds()
                       - _scheduled_closed_seconds(last_bar, now))
        if age_seconds > max_age_seconds:
            logger.warning(
@@ -466,7 +584,7 @@ class BaseDataProvider(ABC):
        df = df.sort_index()
        df = df[~df.index.duplicated(keep="last")]
-        now = pd.Timestamp.utcnow()
+        now = pd.Timestamp.now(tz="UTC")
        future_mask = df.index > now + pd.Timedelta(minutes=5)
        if future_mask.any():
            logger.warning(f"{symbol}: Dropping {future_mask.sum()} future timestamps")
@@ -1,5 +1,5 @@
-numpy>=1.24
+numpy>=1.24,<3
-pandas>=2.0
+pandas>=2.0,<4
 requests>=2.28
 twelvedata>=1.2.0
 python-telegram-bot>=20.0
@@ -242,6 +242,36 @@ class SignalAggregator:
            except Exception as e:
                logger.warning(f"Fetch failed for {symbol} {tf.value}: {e}")
        # A missing frame is a FAILED scan, never a degraded one. Losing a
        # CONFIRMATION frame used to let the scan continue with fewer
        # frames: _check_mtf_agreement needs >= 2 signals, so it returned
        # False and the composite took a 15-point mtf_disagreement penalty
        # the market never earned (A/A+ became impossible), while on
        # M15/H1/H4 a B/C setup could still ship on one confirmation with
        # no warning at all. The backtester never sees that state
        # (run_backtest raises on an empty frame), so it was a live-only
        # divergence. 2026-09-13 audit.
        missing = [tf for tf in timeframes if tf not in dfs]
        if missing:
            report = SignalReport(
                symbol=symbol,
                display_name=pair_info["display"],
                timestamp=str(now_utc),
            )
            report.final_signal = SignalStrength.NEUTRAL
            report.signal_quality = "F"
            report.trade_setup = None
            if self.config.primary_timeframe in missing:
                # Legacy text signal_log.gate_reason already recognises.
                report.warnings.append("Primary timeframe data unavailable")
            report.warnings.append(
                f"DATA UNAVAILABLE: {', '.join(tf.value for tf in missing)}"
                " — scan failed"
            )
            report.action_recommendation = "NO TRADE — market data unavailable"
            self._log_signal(report, log_source, dfs, failed=True)
            return report
        report = self.analyze_from_dataframes(
            symbol=symbol,
            dfs=dfs,
@@ -303,7 +333,7 @@ class SignalAggregator:
    def _log_signal(
        self, report: SignalReport, source: Optional[str],
        dataframes: Optional[Dict[Timeframe, pd.DataFrame]] = None,
-        blocked: bool = False,
+        blocked: bool = False, failed: bool = False,
    ) -> None:
        """Persist the scan to the signal-history DB (signal_log.py).
@@ -314,9 +344,11 @@ class SignalAggregator:
        is still written, just without the news JSON blobs. ``blocked``
        marks weekend/day early returns: no news fetch, no snapshots —
        a ~1 KB liveness row (or nothing, per signal_log.log_blocked_scans).
-        A scan whose PRIMARY fetch failed is logged as ``scan_kind``
+        A scan missing ANY required frame (primary or confirmation, or
-        ``failed`` (price 0, no signal): liveness evidence that readers
+        the regime-filter frame when that filter is on) is logged as
-        skip like ``blocked``, so it never pollutes NEUTRAL statistics."""
+        ``scan_kind`` ``failed`` (price 0, no signal): liveness evidence
        that readers skip like ``blocked``, so it never pollutes NEUTRAL
        statistics."""
        context = None
        if not blocked:
            try:
@@ -325,9 +357,9 @@ class SignalAggregator:
                logger.debug("signal log: news context unavailable",
                             exc_info=True)
        kind = "blocked" if blocked else "scan"
-        if not blocked and dataframes is not None:
+        primary = (dataframes or {}).get(self.config.primary_timeframe)
-            primary = dataframes.get(self.config.primary_timeframe)
+        if failed or (not blocked and dataframes is not None
-            if primary is None or primary.empty:
+                      and (primary is None or primary.empty)):
            kind = "failed"
        extra = {"scan_kind": kind}
        for key, fingerprint in (
@@ -1297,6 +1329,28 @@ class SignalAggregator:
                report.trade_setup = self._generate_xau_atr_trade_setup(report, atr)
        if report.trade_setup is None and report.signal_quality in ("A+", "A", "B", "C"):
            report.trade_setup = self._generate_trade_setup(report, pip_value)
        # A shipped setup always names a direction. The signal thresholds
        # (BUY/SELL need confidence >= 50) sit above the C grade floor
        # (>= 45), and gate 6c grants B at >= 45 in "transitioning", so a
        # C/B report could leave here as NEUTRAL with a full setup: the
        # backtester traded it (the replay never reads final_signal) while
        # the Telegram alert gate rejects NEUTRAL first (2026-09-13 audit:
        # 5/91 H1 12-mo book trades, +3.7R of +22.2R, were never alerted).
        # The generators derived the direction from the score sign — the
        # same sign every direction gate above used — so this changes
        # display and alert eligibility only; the trade books are unchanged.
        # A report no generator served stays NEUTRAL, as before.
        if (report.trade_setup is not None
                and report.final_signal == SignalStrength.NEUTRAL):
            report.final_signal = (SignalStrength.BUY
                                   if report.trade_setup.direction == "BUY"
                                   else SignalStrength.SELL)
            # The action text was written before the gate chain with the
            # old signal name; keep it consistent with the header.
            if report.action_recommendation.startswith("NEUTRAL"):
                report.action_recommendation = (
                    report.final_signal.name.replace("_", " ")
                    + report.action_recommendation[len("NEUTRAL"):])
    @staticmethod
    def _drop_stale_spread_gate_vetoes(report: SignalReport) -> None:
@@ -104,6 +104,7 @@ from advisory_outcome import (
    canonical_r,
    initial_outcome,
    normalize_timeframe,
    resolve_locked_floor,
    timeframe_value,
 )
 # Private on purpose — the scan-outcome labeller must apply EXACTLY the
@@ -258,6 +259,7 @@ _GATE_PREFIXES = (
    "SESSION FILTER",
    "Primary timeframe stale",
    "Primary timeframe data unavailable",
    "DATA UNAVAILABLE:",
    "REGIME FILTER",
    "EXHAUSTION BLOCK",
    "EXHAUSTED TREND FILTER",
@@ -278,6 +280,10 @@ SCAN_OUTCOME_WINDOW_H = 24
 # A 24 h window can only be judged once it has elapsed; the extra hour
 # leaves room for the last bar of the window to be stored.
 SCAN_LABEL_MIN_AGE_H = 25
 # An incomplete row older than this is FINAL: its window had a real hole
 # (a feed outage), which no later bar can fill. It is neither re-examined
 # nor allowed to anchor the shared bar stream for the rest of the batch.
 SCAN_LABEL_MAX_AGE_H = 72
 # Missing bars tolerated inside a window ON TOP of the scheduled closures
 # advisory_outcome._has_continuity already allows.
 SCAN_LABEL_MAX_GAP_BARS = 3
@@ -399,6 +405,20 @@ def setup_key(setup: Optional[Dict[str, Any]], timeframe: Any) -> Optional[str]:
    ))
 def _is_gap_reason(reason: Any) -> bool:
    """True when an outcome was left UNRESOLVABLE by a candle gap.
    advance_outcome writes TWO such reasons — "stored candle snapshots
    lost continuity" (no overlap with the last processed bar) and "stored
    candle snapshots contain an internal gap" — and each call site used to
    match only one of them, so the stream rebuild and the locked-floor
    rescue skipped half the gap rows. The substring clauses keep states
    written by older code inside the same net."""
    text = str(reason or "")
    return (text.startswith("stored candle snapshots")
            or "continuity" in text or "gap" in text)
 def gate_reason(
    warnings: Iterable[str],
    cleaned_warnings: Iterable[str],
@@ -1150,12 +1170,19 @@ class SignalLogger:
                    before = _dumps(json.loads(hit[0]))
                    advanced = advance_outcome(json.loads(before), bars)
                    if (advanced.get("status") == "UNRESOLVABLE"
-                            and "continuity" in str(advanced.get("reason", ""))):
+                            and _is_gap_reason(advanced.get("reason"))):
                        union = self._stream_since(
                            symbol, timeframe,
                            json.loads(before)["progress"]["last_processed_bar"])
                        if union:
                            advanced = advance_outcome(json.loads(before), union)
                        # The database could not fill the hole. A trade
                        # whose trail had already locked profit still has a
                        # real answer — its floor — so record that instead
                        # of dropping the row out of the scorecard.
                        if (advanced.get("status") == "UNRESOLVABLE"
                                and _is_gap_reason(advanced.get("reason"))):
                            advanced = resolve_locked_floor(advanced)
                    after = _dumps(advanced)
                    if after == before:
                        self._conn.rollback()
@@ -1194,6 +1221,11 @@ class SignalLogger:
                " WHERE symbol=? AND candles_json IS NOT NULL"
                " AND logged_at >= ? ORDER BY id LIMIT ?",
                (f'$."{key}"', symbol, since, int(max_rows))).fetchall()
            if len(rows) == int(max_rows):
                logger.warning(
                    "signal_log: bar stream truncated for %s %s (key %s)"
                    " since %s — hit the %s-row cap, later bars are missing",
                    symbol, tf, key, since, int(max_rows))
            for (blob,) in rows:
                if not blob:
                    continue
@@ -1212,42 +1244,82 @@ class SignalLogger:
        Candidates are rows that carry indicator snapshots (a real scan)
        and a price, whose signal bar closed more than 25 h ago and which
-        have no COMPLETE label yet; the least recently labelled go first,
+        have no COMPLETE label yet. An incomplete row is re-examined only
-        so a permanently short stream can never starve the rest. Returns
+        while it is younger than SCAN_LABEL_MAX_AGE_H (counted from the
-        the number of rows actually WRITTEN — a row whose numbers did not
+        newest logged row when the database is behind the clock) — beyond
-        change is skipped, which makes repeated passes idempotent and lets
+        that its window had a real hole that no later bar can fill, so the
-        a batch driver stop at 0.
+        label it already carries is final — and a stale row never anchors
        the shared bar stream either (a key that has nothing else due
        starts at the age bound plus one window). Both bounds are
        load-bearing: a handful of permanently incomplete rows (a feed
        outage) sort to the front of every pass, and anchoring the stream
        at their bar pushes its row cap past today, which silently labels
        every genuinely due row NULL and incomplete — the scan_outcomes
        table then goes blind. Returns the number of rows actually WRITTEN
        — a row whose numbers did not change is not rewritten (only its
        labeled_at is bumped, so it rotates to the back of the queue),
        which makes repeated passes idempotent and lets a batch driver
        stop at 0.
        Bounded on purpose: at most ``budget`` candidates are examined and
        the pass stops after ``time_budget`` seconds (default 2 s) so the
        opportunistic call inside log() stays cheap. Per-row failures are
        logged and skipped."""
        moment = now or datetime.now(timezone.utc)
-        cutoff = _utc(moment) - timedelta(hours=SCAN_LABEL_MIN_AGE_H)
+        moment_utc = _utc(moment)
        cutoff = moment_utc - timedelta(hours=SCAN_LABEL_MIN_AGE_H)
        deadline = time.monotonic() + float(
            SCAN_LABEL_TIME_BUDGET if time_budget is None else time_budget)
        written = 0
        with self._lock:
            # Age is measured from the newest row when the database itself
            # is behind the clock (an imported snapshot on the dev box),
            # otherwise every row of a 3-day-old import would be declared
            # final on the first pass. On a live logger the two coincide.
            horizon = moment_utc
            newest = self._conn.execute(
                "SELECT logged_at FROM signals ORDER BY id DESC"
                " LIMIT 1").fetchone()
            try:
                horizon = min(horizon, _utc(datetime.fromisoformat(
                    str(newest[0]).replace(" ", "T"))))
            except (TypeError, ValueError, IndexError):
                pass
            stale_before = horizon - timedelta(hours=SCAN_LABEL_MAX_AGE_H)
            # Deepest a stream may start when every row due for that key is
            # stale: one window before the age bound, so such a row still
            # gets its single final pass without dragging the stream back.
            anchor_floor = stale_before - timedelta(hours=SCAN_LABEL_MIN_AGE_H)
            # Row ids are monotonic in logged_at (a uniform ISO string), so
            # an id bound is the cheap, index-friendly way to leave stale
            # incomplete rows out of the re-examination set. bar_ts mixes
            # separators and must never be compared as a string.
            min_id = self._conn.execute(
                "SELECT MIN(id) FROM signals WHERE logged_at >= ?",
                (stale_before.isoformat(),)).fetchone()[0]
            if min_id is None:
                min_id = 2 ** 62          # nothing recent: re-examine none
            rows = self._conn.execute(
                "SELECT s.id, s.symbol, s.timeframe, s.price,"
                " COALESCE(s.bar_ts, json_extract(s.indicators_json,"
                " '$.\"' || s.timeframe || '\".timestamp')) AS bar,"
                " json_extract(s.indicators_json,"
-                " '$.\"' || s.timeframe || '\".key_levels.atr') AS atr"
+                " '$.\"' || s.timeframe || '\".key_levels.atr') AS atr,"
                " (o.signal_id IS NULL) AS unlabelled"
                " FROM signals s"
                " LEFT JOIN scan_outcomes o ON o.signal_id = s.id"
                " WHERE s.indicators_json IS NOT NULL AND s.price > 0"
-                " AND (o.signal_id IS NULL OR o.complete = 0)"
+                " AND (o.signal_id IS NULL OR (o.complete = 0 AND s.id >= ?))"
                " AND bar IS NOT NULL"
                " ORDER BY COALESCE(o.labeled_at, '') ASC, s.id ASC"
-                " LIMIT ?", (int(budget),)).fetchall()
+                " LIMIT ?", (int(min_id), int(budget))).fetchall()
            # One bar stream per (symbol, timeframe) for the whole batch,
            # built from the OLDEST bar in it — _stream_since is a scan
            # over every row logged since that bar, far too expensive to
            # repeat per candidate.
            due: List[Tuple[Any, ...]] = []
            oldest: Dict[Tuple[str, str], Tuple[datetime, str]] = {}
-            for rowid, symbol, tf, price, bar, atr in rows:
+            for rowid, symbol, tf, price, bar, atr, unlabelled in rows:
                try:
                    stamp = _utc(datetime.fromisoformat(
                        str(bar).replace(" ", "T")))
@@ -1255,8 +1327,16 @@ class SignalLogger:
                    continue
                if stamp > cutoff:
                    continue              # window has not elapsed yet
                # Stale = old by the bar it closed on AND by when it was
                # written (on a live database the two agree within minutes;
                # requiring both keeps a freshly imported or backfilled DB,
                # whose rows all carry today's logged_at, out of it).
                stale = stamp < stale_before and rowid < min_id
                if stale and not unlabelled:
                    continue   # final: a hole no later bar can ever fill
                key = (str(symbol), str(tf))
-                if key not in oldest or stamp < oldest[key][0]:
+                if not stale and (key not in oldest
                                  or stamp < oldest[key][0]):
                    oldest[key] = (stamp, str(bar))
                due.append((rowid, symbol, tf, price, bar, atr, key))
            streams: Dict[Tuple[str, str], List[Dict[str, Any]]] = {}
@@ -1265,8 +1345,13 @@ class SignalLogger:
                    break
                try:
                    if key not in streams:
                        # 3 days of anchor is ~2,100 rows on the live DB;
                        # the cap is a safety net, not the working limit.
                        anchor = oldest.get(key)
                        streams[key] = self._stream_since(
-                            symbol, tf, oldest[key][1])
+                            symbol, tf,
                            anchor[1] if anchor else anchor_floor.isoformat(),
                            max_rows=10000)
                    label = scan_outcome(bar, tf, streams[key],
                                         _finite(atr), price)
                    if label is None:
@@ -1277,6 +1362,14 @@ class SignalLogger:
                        + " FROM scan_outcomes WHERE signal_id=?",
                        (rowid,)).fetchone()
                    if prior is not None and tuple(prior) == values:
                        # Unchanged, so NOT counted as written — but its
                        # queue position moves, or the same short-stream
                        # rows would head every pass for ever.
                        now_iso = datetime.now(timezone.utc).isoformat()
                        self._conn.execute(
                            "UPDATE scan_outcomes SET labeled_at=?"
                            " WHERE signal_id=?", (now_iso, rowid))
                        self._conn.commit()
                        continue
                    self._conn.execute(
                        _SCAN_OUTCOME_INSERT,
@@ -1376,6 +1469,9 @@ class SignalLogger:
                    continue
                before = _dumps(state)
                advanced = advance_outcome(json.loads(before), union)
                if (advanced.get("status") == "UNRESOLVABLE"
                        and _is_gap_reason(advanced.get("reason"))):
                    advanced = resolve_locked_floor(advanced)
                after = _dumps(advanced)
                if after == before:
                    continue
@@ -1393,7 +1489,9 @@ class SignalLogger:
        """Bounded retry for continuity-broken outcomes: re-freeze the
        seed from the row's own setup/indicators (at most ``max_reseeds``
        times per row) so the next resolver pass can walk the now-filled
-        stream. Invalid-seed rows stay terminal."""
+        stream. Invalid-seed rows stay terminal. A gap row that had already
        trailed its stop past the entry is finished at that locked floor
        instead of re-seeded; the returned list holds re-seeded ids only."""
        reseeded: List[int] = []
        with self._lock:
            rows = self._conn.execute(
@@ -1403,8 +1501,21 @@ class SignalLogger:
                " AND advisory_event_id = id").fetchall()
            for rowid, tf, setup_json, ind_json, state_json in rows:
                old = json.loads(state_json) if state_json else {}
-                if "continuity" not in str(old.get("reason", "")) and \
+                if not _is_gap_reason(old.get("reason")):
-                        "gap" not in str(old.get("reason", "")):
+                    continue
                # The locked trailing stop lives ONLY in this progress
                # block, and a re-seed rebuilds the walk from scratch over
                # a stream whose hole is still there (the continuity check
                # is all-or-nothing), so the floor would be lost for good.
                # Finish such a row here; re-seed only what has nothing.
                floored = resolve_locked_floor(json.loads(_dumps(old)))
                if floored.get("status") == "RESOLVED":
                    self._conn.execute(
                        "UPDATE signals SET advisory_outcome_status=?,"
                        " advisory_r_multiple=?, advisory_outcome_json=?"
                        " WHERE id=?",
                        ("RESOLVED", canonical_r(floored),
                         _dumps(floored), rowid))
                    continue
                count = int((old.get("reseed") or {}).get("count", 0))
                if count >= max_reseeds or not setup_json:
@@ -49,9 +49,10 @@ import sys
 import threading
 import time
 from concurrent.futures import ThreadPoolExecutor
-from datetime import datetime, timezone
+from datetime import datetime, timedelta, timezone
 from typing import TYPE_CHECKING, Any, Dict, List, Optional, Set
 import pandas as pd
 from telegram import InlineKeyboardButton, InlineKeyboardMarkup, Update
 from telegram.constants import ParseMode
 from telegram.error import BadRequest
@@ -71,6 +72,7 @@ from app_config import (
    telegram_channel_id, violent_watch_enabled,
 )
 from providers import BaseDataProvider, make_data_provider
 from providers.base import _is_daily_maintenance
 from agent_panel import attach_panel, panel_health
 from llm_client import get_llm_client
 from market_outlook import maybe_run_outlook
@@ -110,8 +112,12 @@ _file_handler = logging.handlers.TimedRotatingFileHandler(
    when="midnight", backupCount=30, encoding="utf-8",
 )
 _file_handler.setFormatter(_log_formatter)
 # Console only at IMPORT: `python test_all.py` imports this module, and an
 # import-time file handler wrote fixture alerts (2027 dates, 3300.00 prices)
 # into logs/telegram.log on whatever box ran the suite. main() attaches
 # _file_handler for real bot runs.
 logging.basicConfig(level=getattr(logging, _APP.log_level, logging.INFO),
-                    handlers=[_console_handler, _file_handler],
+                    handlers=[_console_handler],
                    force=True)
 logging.getLogger("httpx").setLevel(logging.WARNING)
 logging.getLogger("httpcore").setLevel(logging.WARNING)
@@ -545,7 +551,21 @@ def _bar_epoch(tf: Timeframe, now: Optional[float] = None) -> int:
 def _remember_scan(symbol: str, tf: Timeframe, report: SignalReport) -> None:
-    _last_scan[f"{symbol}_{tf.value}"] = (_bar_epoch(tf), report)
+    # File the report under the bar it actually ANALYSED, not the wall
    # clock at completion: a scan that straddles a boundary would land
    # under the new bar, so the boundary loop's own scan of that bar
    # found no cache entry and re-fetched/re-scored/re-logged the same
    # closed bar (15 duplicate M15 rows in five days — 2026-09-13 audit).
    epoch = _bar_epoch(tf)
    if report.primary_analysis is not None:
        try:
            # report.timestamp is the primary bar's OPEN label; the bar
            # closes (= the boundary this scan belongs to) one TF later.
            epoch = (int(pd.Timestamp(report.timestamp).timestamp())
                     + _TF_SECONDS_CACHE.get(tf, 900))
        except (ValueError, TypeError):
            pass   # unparseable label → wall-clock boundary, as before
    _last_scan[f"{symbol}_{tf.value}"] = (epoch, report)
 async def _cached_or_scan(
@@ -767,6 +787,16 @@ def _seconds_until_boundary(interval_minutes: int, buffer: int = 10) -> float:
    return remaining
 def _closed_bar_in_settlement(interval_minutes: int,
                              now: Optional[datetime] = None) -> bool:
    """True when the bar that just closed (label = now - interval) falls in
    gold's daily settlement pause — no such bar exists, so a scan would only
    re-read the previous bar (61 stale rows in the 2026-09 audit DB)."""
    label_time = ((now or datetime.now(timezone.utc))
                  - timedelta(minutes=interval_minutes))
    return _is_daily_maintenance(label_time)
 # ── Violent-move watch (display-only) ────────────────────────────────
 # Per-process heads-up budget: {"day": date, "count": int, "sent": {...}}.
 _watch_state: Dict[str, Any] = {}
@@ -800,7 +830,11 @@ async def _scan_timeframe(
    confirmation_tfs: List[Timeframe],
 ) -> None:
    """Scan all pairs for one timeframe and broadcast results."""
-    tasks = [run_signal(symbol, tf, confirmation_tfs) for symbol in SCAN_PAIRS]
+    # _cached_or_scan, not run_signal: the WS breach handler may already
    # have scanned this closed bar (and vice versa) — one fetch/score/log
    # per bar per TF (2026-09-13 audit).
    tasks = [_cached_or_scan(symbol, tf, confirmation_tfs)
             for symbol in SCAN_PAIRS]
    results = await asyncio.gather(*tasks, return_exceptions=True)
    reports: List[SignalReport] = []
    for i, result in enumerate(results):
@@ -930,6 +964,9 @@ _pending_breach: Dict[str, tuple] = {}
 # A failed breach rescan shrinks the just-set cooldown to this retry
 # window instead of silencing the symbol for the full cooldown (H33a).
 _BREACH_RETRY_SECONDS = 30
 # A breach landing inside this many seconds after an M15 boundary waits for
 # the candle loop's scan (boundary+10s) and reuses it — 2026-09-13 audit.
 _BREACH_DEFER_SECONDS = 20.0
 _cfg = PlatformConfig()
 _WS_COOLDOWN = _cfg.ws_cooldown_seconds
@@ -1003,6 +1040,13 @@ async def _handle_breach(bot, pair_symbol: str, price: float, level_name: str) -
    logger.info(f"Level breach: {display} crossed {label} at {price} — rescanning M15")
    try:
        # A tick a second after an M15 boundary would scan a bar the feed
        # may still be lagging on, and the candle loop (boundary+10s) would
        # then scan the same bar again. Wait for the loop and reuse it.
        elapsed = time.time() - _bar_epoch(Timeframe.M15)
        if elapsed < _BREACH_DEFER_SECONDS:
            logger.debug("deferring breach rescan to the boundary scan")
            await asyncio.sleep(_BREACH_DEFER_SECONDS - elapsed)
        report = await _cached_or_scan(pair_symbol, Timeframe.M15)
        _update_level_cache([report])
    except Exception:
@@ -1152,6 +1196,11 @@ async def _candle_loop(bot, tf: Timeframe, interval_minutes: int) -> None:
            await asyncio.sleep(wait)
            if not _feed_active():
                break
            if _closed_bar_in_settlement(interval_minutes):
                logger.info(
                    f"{label} auto-scan: skipped — bar label inside the "
                    f"daily settlement pause")
                continue
            logger.info(f"{label} auto-scan: firing")
            try:
                await _scan_timeframe(
@@ -1824,6 +1873,13 @@ def main() -> None:
            "and telegram.chat_id in config.json, or the TELEGRAM_TOKEN and "
            "TELEGRAM_CHAT_ID environment variables.")
    # Only a real bot run writes the rotating log file: tests import this
    # module (an import-time handler logged their fixtures into
    # logs/telegram.log) and one of them calls main() to assert the refusal
    # above, so the attach sits after that guard — 2026-09-13 audit.
    if _file_handler not in logging.getLogger().handlers:
        logging.getLogger().addHandler(_file_handler)
    _install_excepthooks()
    logger.info("Starting ForexBot2 Telegram bot…")
Author	SHA1	Message	Date
Harwinder Sekhon	2836024e9d	audit 2026-09-13: eleven delivery/measurement fixes; books byte-identical Signal path - A shipped setup always names a direction: grade C needs confidence >= 45 (gate 6c grants B at >= 45) while BUY/SELL needs >= 50, so C/B reports reached the alert gate as NEUTRAL with a full setup. The backtester traded them (the replay never reads final_signal) while Telegram rejected NEUTRAL first — 5/91 trades, +3.7R of +22.2R, in the H1 12-month book. _attach_core_setup stamps the setup's direction and repairs the pre-gate action text; no-setup reports stay NEUTRAL. - analyze_pair fails the scan when ANY required frame is missing ("DATA UNAVAILABLE:", scan_kind='failed'); a lost confirmation frame used to run with an unearned -15 MTF penalty. Outcomes and labels - advisory_outcome.resolve_locked_floor: a gap the database cannot fill resolves at the locked trailing stop (GAP_LOCKED, result.floor) when the stop had crossed the entry; the rebuild fires on both gap reasons (rows 915/920: +1.14R / +0.84R floors; resolved sum -2.46R -> -0.48R). - label_scan_outcomes: SCAN_LABEL_MAX_AGE_H = 72 h, stale rows never anchor the shared bar stream, labeled_at bumped on re-examination, _stream_since warns at its row cap (the Labor-Day pile would have blinded scan_outcomes around 2026-09-14). Providers and scheduling - The daily settlement pause (Mon-Thu 17:00 New York) is a scheduled closure for _has_latest_closed_bar (whole-span label walk-back) and _is_data_fresh (closed seconds are not age); bot and dashboard skip scans whose just-closed label sits in the pause (61 false stale rows in three nights). - fetch_range retries a failed page in place (three failures on one window raise) and logs an ERROR for interior gaps of >3 unscheduled bars. - pd.Timestamp.utcnow() replaced in providers/base.py and engine.py; requirements pin pandas<4, numpy<3. - telegram_bot: boundary loops share one scan per bar with the breach handler (_remember_scan keys on the analysed bar; breach defers 20 s after an M15 boundary); the rotating file handler attaches in main() only (the test suite used to write into logs/telegram.log). - dashboard_publisher: catch_up_timeframes re-scans boundaries missed while a cycle overran or the process was down. Backtester and engine guards - _replay_span returns the per-span GPU cache-miss count; run_backtest sums, logs it (must be 0) and writes cache_fallbacks to the manifest. - regime_weight_table raises on a partial regime_weights_override. Verification: 519 tests; M5/M15 2-yr, M15 OOS, H1/H4 3-yr books byte-identical before and after; live signals.db untouched (checked on a copy). CLAUDE.md carries the dated entry and the blocked-hours union semantics that every book was validated under. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_0139xz1J6YccndA2be6yFnEo	2026-09-13 21:42:37 -04:00
Harwinder Sekhon	1c293f7d2e	research: improvement cycle 1 ledger (baseline, cost ladder, C1 reject, C2 plateau, seeding parity) backtest/research/cycle_2026-09-10.md: pre-registered rules, HEAD baseline on the 3-year window (three pinned MD5s reproduced), cost ladder and concentration facts, bucket screen (one cycle-2 candidate: M15 grade-A SELLs), C1 H1 divergence->0 REJECTED, C2 M15 RSI-multiplier ladder = stable plateau, seeding-depth measurement and the two shipped parity fixes. CLAUDE.md: live fetch depth, backtester pad, 501 tests, the 2026-09-10 state entry; corrected the docs note (no PDF builds exist) and recorded pyarrow as a dev-venv-only dependency. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01NQDjW8rXYrJR4JDWMWKUij	2026-09-10 08:29:14 -04:00
Harwinder Sekhon	b8284e66aa	backtest: seed slow-TF history to the live 1000-bar floor (_history_pad_days) The per-TF warm-up pad before --start targeted 210 bars, so a 3-year H4 book started with 372 H4 bars and ~210 D1 bars of seeding while the live scanner now fetches at least MIN_LIVE_BARS (1000) on every timeframe. _history_pad_days keeps the 62-day intraday pad (already >= 1488 bars, so M5/M15/H1 cache windows are untouched) and pads H4 by 234 and D1 by 1400 calendar days; the raw-path fetch for trade resolution reuses the primary timeframe's own start (one cache file per TF, as before). Verified on the 3-year books at $0.65: M5 and M15 byte-identical (c2722f98…, b2cfa9e2…); H1 and H4 trade-for-trade identical (253 / +70.02R, 114 / +21.68R) — the only differences are one H4 confirmation reading 15 days into the H1 window and 1e-6…1e-9 numerics in D1 Wilder recursions during Oct–Dec 2023, i.e. the window-start seeding zone. Evidence: backtest/research/cycle_2026-09-10.md. Tests: Module 21 "history pad seeds slow TFs like live". Suite 501 passed. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01NQDjW8rXYrJR4JDWMWKUij	2026-09-10 08:29:04 -04:00
Harwinder Sekhon	9942736762	providers: floor every live fetch at 1000 bars (H4/D1 indicator-seeding parity) The engine's EMAs use ewm(adjust=False), so the first bar of a fetched frame keeps weight (1-2/201)^n in the 200-bar trend EMA. With the 60-day lookback the live H4 frame had 380 bars (seed weight 2.3%) and D1 300 (5%), while the backtester seeds from its whole cached history. Measured on the deepest Oanda caches (267 H4 / 313 D1 bars sampled over two years, raw and cleaned frames): ema_trend flipped on 2/267 H4 and 1/313 D1 bars at those depths (composite off by up to 9.5 points) and on none from 600 bars up. MIN_LIVE_BARS = 1000 leaves a seed weight of e^-10; M5/M15 (5000) and H1 (1460) already exceed it and are unchanged. One request either way. The backtester never calls fetch_ohlcv, so no book changes. Tests: new Module 16 depth test; the Module 21 cache test's 300-bar expectation becomes 1000. Suite 500 passed. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01NQDjW8rXYrJR4JDWMWKUij	2026-09-10 07:55:20 -04:00