providers: keep real news bars (snap-back bad-print rule), Oanda retry ladder, scan_kind=failed

2026-09-07 signal-log inspection follow-up — three fixes that do not depend on sample size.

- providers/base.py: the signal-side 10σ outlier filter is now a bad-print filter. A >10σ
  close-to-close bar is dropped only when the NEXT close snaps back ≥80% of the way
  (OUTLIER_SIGMA / OUTLIER_REVERT_FRACTION); persisting moves and the newest bar are kept.
  The unconditional rule deleted ~36 real M5 news bars/yr in the backtester (whole-frame σ)
  and a different ~24/yr live (fetch-window σ) — e.g. the 2026-09-04 12:30 UTC NFP bar
  vanished from M5+M15 and the 12:35 scan was a byte-identical repeat of 12:30. 3-yr Oanda
  drops are now M5 2 / M15 1 / H1 0.
- providers/oanda.py: transient 401/5xx/network faults retry along RETRY_DELAYS_S=(2,5,10)s
  under RETRY_BUDGET_S=45s. 11 boundary scans in two days had lost their primary frame at
  xx:00:22–:26 UTC; one 2 s retry could not outlast the top-of-hour blip.
- signal_aggregator/_log_signal: a scan whose primary fetch failed logs scan_kind='failed'
  (price 0, no signal); analyze_signal_log.parity_report skips it like 'blocked'.
- test_all.py: 466 tests (+6): retry ladder walks all delays, 5xx blip recovers on the third
  attempt, retry budget stops the ladder, persisting/newest/partially-retraced 10σ bars kept,
  failed-scan row kind, parity skip.
- CLAUDE.md: cleaning rule, retry ladder, failed scan kind, dated current-state note.

Book impact at $0.65 (rule 1 still passes everywhere, both halves positive):
  12mo  M5 557/+112.21/PF1.56 → 560/+108.81/1.54   M15 397/+73.93/1.53 → 398/+74.61/1.53   H1 identical
  3yr   M5 857/+161.3/1.53 → 870/+153.6/1.50   M15 1086/+148.7/1.37 → 1092/+151.9/1.38
        H1 255/+70.5/1.98 → 253/+70.0/1.98   H4 113/+22.7/1.63 → 114/+21.7/1.59
Only trades on the previously-deleted event bars move; the pinned M5/M15 CSVs are no longer
byte-reproducible and should be regenerated at the next re-pin.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01NDnBxJ2y8GRUM6ZNFJs1hZ

providers/base.py

@@ -100,6 +100,14 @@ class BaseDataProvider(ABC):
     # boundary is missing the just-closed bar (feed latency).
     FRESH_FETCH_RETRY_SECONDS = 3.0
 
+    # Signal-side bad-print filter (see _validate_and_clean): a bar is a
+    # candidate when its close-to-close return sits more than
+    # OUTLIER_SIGMA standard deviations from the frame mean, and is
+    # dropped only when the NEXT close snaps back at least
+    # OUTLIER_REVERT_FRACTION of the way to the pre-spike close.
+    OUTLIER_SIGMA = 10.0
+    OUTLIER_REVERT_FRACTION = 0.8
+
     def __init__(self, config: PlatformConfig):
         self.config = config
         self._cache: Dict[str, pd.DataFrame] = {}
@@ -498,20 +506,32 @@ class BaseDataProvider(ABC):
         returns = returns.replace([float("inf"), float("-inf")], float("nan"))
         std_ret = returns.std()
         if std_ret > 0:
+            # A 10σ close-to-close move is NOT bad data by default. On
+            # three years of Oanda XAU bars every such bar was a real
+            # news/reopen move (NFP, CPI, Sunday gap — 2026-09-07 audit:
+            # 155 flagged M5 bars, 7 snapped back), and dropping them hid
+            # the most violent bar of the day from ATR/exhaustion math —
+            # while the backtester (whole-frame σ) dropped a DIFFERENT
+            # set than live (fetch-window σ). A bad print has the one
+            # signature a real move lacks: the NEXT close returns to the
+            # pre-spike level. Drop a candidate only on that observed
+            # snap-back; the newest bar (no successor yet) is kept as
+            # real, and the snap-back bar itself is never a candidate
+            # (its residual against the pre-spike close is small).
             dev = returns - returns.mean()
-            outlier_mask = dev.abs() > 10 * std_ret
-            # A spike bar makes the NEXT bar's return look extreme too (it is
-            # measured against the spike). If the previous bar is flagged and
-            # this bar reverses direction, it is the legitimate snap-back —
-            # keep it, or every spike would punch a two-bar hole in the data.
-            recovery = (
-                outlier_mask
-                & outlier_mask.shift(1, fill_value=False)
-                & (dev * dev.shift(1) < 0)
+            candidate = dev.abs() > self.OUTLIER_SIGMA * std_ret
+            close = df["Close"]
+            spike = close - close.shift(1)
+            residual = close.shift(-1) - close.shift(1)
+            snapped_back = residual.notna() & (
+                residual.abs()
+                <= (1.0 - self.OUTLIER_REVERT_FRACTION) * spike.abs()
             )
-            outlier_mask = outlier_mask & ~recovery
+            outlier_mask = candidate & snapped_back
             if outlier_mask.any():
-                logger.warning(f"{symbol}: Removing {outlier_mask.sum()} extreme outlier bars")
+                logger.warning(
+                    f"{symbol}: Removing {int(outlier_mask.sum())} bad-print "
+                    f"bar(s) (>{self.OUTLIER_SIGMA:.0f}σ move that snapped back)")
                 df = df[~outlier_mask]
 
         min_required = max(