harry/ForexBot2
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

providers: keep real news bars (snap-back bad-print rule), Oanda retry ladder, scan_kind=failed

Browse Source 
2026-09-07 signal-log inspection follow-up — three fixes that do not depend on sample size.

- providers/base.py: the signal-side 10σ outlier filter is now a bad-print filter. A >10σ
  close-to-close bar is dropped only when the NEXT close snaps back ≥80% of the way
  (OUTLIER_SIGMA / OUTLIER_REVERT_FRACTION); persisting moves and the newest bar are kept.
  The unconditional rule deleted ~36 real M5 news bars/yr in the backtester (whole-frame σ)
  and a different ~24/yr live (fetch-window σ) — e.g. the 2026-09-04 12:30 UTC NFP bar
  vanished from M5+M15 and the 12:35 scan was a byte-identical repeat of 12:30. 3-yr Oanda
  drops are now M5 2 / M15 1 / H1 0.
- providers/oanda.py: transient 401/5xx/network faults retry along RETRY_DELAYS_S=(2,5,10)s
  under RETRY_BUDGET_S=45s. 11 boundary scans in two days had lost their primary frame at
  xx:00:22–:26 UTC; one 2 s retry could not outlast the top-of-hour blip.
- signal_aggregator/_log_signal: a scan whose primary fetch failed logs scan_kind='failed'
  (price 0, no signal); analyze_signal_log.parity_report skips it like 'blocked'.
- test_all.py: 466 tests (+6): retry ladder walks all delays, 5xx blip recovers on the third
  attempt, retry budget stops the ladder, persisting/newest/partially-retraced 10σ bars kept,
  failed-scan row kind, parity skip.
- CLAUDE.md: cleaning rule, retry ladder, failed scan kind, dated current-state note.

Book impact at $0.65 (rule 1 still passes everywhere, both halves positive):
  12mo  M5 557/+112.21/PF1.56 → 560/+108.81/1.54   M15 397/+73.93/1.53 → 398/+74.61/1.53   H1 identical
  3yr   M5 857/+161.3/1.53 → 870/+153.6/1.50   M15 1086/+148.7/1.37 → 1092/+151.9/1.38
        H1 255/+70.5/1.98 → 253/+70.0/1.98   H4 113/+22.7/1.63 → 114/+21.7/1.59
Only trades on the previously-deleted event bars move; the pinned M5/M15 CSVs are no longer
byte-reproducible and should be regenerated at the next re-pin.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01NDnBxJ2y8GRUM6ZNFJs1hZ
This commit is contained in:
Harwinder Sekhon
2026-09-07 11:07:29 -04:00
parent f7c244c6b5
commit de36b104d9
7 changed files with 248 additions and 36 deletions
Download Patch File Download Diff File Expand all files Collapse all files
providers/base.py
+31 -11
View File
@@ -100,6 +100,14 @@ class BaseDataProvider(ABC):
# boundary is missing the just-closed bar (feed latency).
FRESH_FETCH_RETRY_SECONDS = 3.0
# Signal-side bad-print filter (see _validate_and_clean): a bar is a
# candidate when its close-to-close return sits more than
# OUTLIER_SIGMA standard deviations from the frame mean, and is
# dropped only when the NEXT close snaps back at least
# OUTLIER_REVERT_FRACTION of the way to the pre-spike close.
OUTLIER_SIGMA = 10.0
OUTLIER_REVERT_FRACTION = 0.8
def __init__(self, config: PlatformConfig):
self.config = config
self._cache: Dict[str, pd.DataFrame] = {}
@@ -498,20 +506,32 @@ class BaseDataProvider(ABC):
returns = returns.replace([float("inf"), float("-inf")], float("nan"))
std_ret = returns.std()
if std_ret > 0:
# A 10σ close-to-close move is NOT bad data by default. On
# three years of Oanda XAU bars every such bar was a real
# news/reopen move (NFP, CPI, Sunday gap — 2026-09-07 audit:
# 155 flagged M5 bars, 7 snapped back), and dropping them hid
# the most violent bar of the day from ATR/exhaustion math —
# while the backtester (whole-frame σ) dropped a DIFFERENT
# set than live (fetch-window σ). A bad print has the one
# signature a real move lacks: the NEXT close returns to the
# pre-spike level. Drop a candidate only on that observed
# snap-back; the newest bar (no successor yet) is kept as
# real, and the snap-back bar itself is never a candidate
# (its residual against the pre-spike close is small).
dev = returns - returns.mean()
outlier_mask = dev.abs() > 10 * std_ret
# A spike bar makes the NEXT bar's return look extreme too (it is
# measured against the spike). If the previous bar is flagged and
# this bar reverses direction, it is the legitimate snap-back —
# keep it, or every spike would punch a two-bar hole in the data.
recovery = (
outlier_mask
& outlier_mask.shift(1, fill_value=False)
& (dev * dev.shift(1) < 0)
candidate = dev.abs() > self.OUTLIER_SIGMA * std_ret
close = df["Close"]
spike = close - close.shift(1)
residual = close.shift(-1) - close.shift(1)
snapped_back = residual.notna() & (
residual.abs()
<= (1.0 - self.OUTLIER_REVERT_FRACTION) * spike.abs()
)
outlier_mask = outlier_mask & ~recovery
outlier_mask = candidate & snapped_back
if outlier_mask.any():
logger.warning(f"{symbol}: Removing {outlier_mask.sum()} extreme outlier bars")
logger.warning(
f"{symbol}: Removing {int(outlier_mask.sum())} bad-print "
f"bar(s) (>{self.OUTLIER_SIGMA:.0f}σ move that snapped back)")
df = df[~outlier_mask]
min_required = max(
providers/oanda.py
+37 -14
View File
@@ -32,7 +32,11 @@ class OandaDataProvider(BaseDataProvider):
name = "oanda"
DATA_ERROR = OandaDataError
RETRY_DELAY_S = 2.0
# Transient-fault retry ladder (see _fetch_oanda_candles): sleeps
# between attempts, plus a wall-clock cap on the whole sequence.
RETRY_DELAYS_S = (2.0, 5.0, 10.0)
RETRY_BUDGET_S = 45.0
RETRY_DELAY_S = RETRY_DELAYS_S[0] # first delay; kept for callers/tests
OANDA_GRANULARITY = {
Timeframe.M1: "M1",
@@ -128,34 +132,53 @@ class OandaDataProvider(BaseDataProvider):
else:
params["count"] = min(count, self.MAX_BARS_PER_REQUEST)
# One retry absorbs Oanda's sporadic spurious 401s (~15-20/day on
# the practice API — the same token succeeds on adjacent requests)
# plus transient 5xx/network faults. A real credential problem
# fails both attempts and raises exactly as before. Other 4xx
# (bad request, 429) are never retried.
# Retry ladder for transient faults: Oanda's sporadic spurious
# 401s (~15-20/day on the practice API — the same token succeeds
# on adjacent requests), 5xx and network errors. Other 4xx (bad
# request, 429) are never retried; a real credential problem
# fails every attempt and raises as before. The 2026-09-07
# signal-log audit found 11 boundary scans in two days that lost
# their primary frame at xx:00:22-:26 UTC: a single 2 s retry
# could not outlast a 10-15 s provider blip at the top of the
# hour, and a scan without its primary frame logs a price-0
# NEUTRAL row. RETRY_BUDGET_S bounds the whole sequence so a scan
# (three fetches) still finishes well inside an M5 bar.
last_err: Optional[OandaDataError] = None
for attempt in range(2):
resp = None
attempts = 1 + len(self.RETRY_DELAYS_S)
started = time.monotonic()
for attempt in range(attempts):
if attempt:
time.sleep(self.RETRY_DELAY_S)
delay = self.RETRY_DELAYS_S[attempt - 1]
if time.monotonic() - started + delay > self.RETRY_BUDGET_S:
logger.warning(
f"Oanda fetch for {instrument}: retry budget "
f"({self.RETRY_BUDGET_S:.0f}s) exhausted after "
f"{attempt} attempt(s) — giving up")
break
time.sleep(delay)
try:
resp = self._session.get(url, params=params, timeout=30)
except requests.RequestException as e:
last_err = OandaDataError(
f"Oanda candle fetch failed for {instrument}: {e}")
last_err.__cause__ = e
resp = None
logger.warning(
f"Oanda fetch for {instrument} attempt "
f"{attempt + 1}/{attempts} failed: {e}")
continue
if resp.status_code == 401 or resp.status_code >= 500:
last_err = OandaDataError(
f"Oanda candle fetch failed for {instrument}: "
f"HTTP {resp.status_code}: {resp.text[:300]}")
if attempt == 0:
logger.warning(
f"Oanda fetch for {instrument} got HTTP "
f"{resp.status_code}; retrying in "
f"{self.RETRY_DELAY_S}s")
logger.warning(
f"Oanda fetch for {instrument} attempt "
f"{attempt + 1}/{attempts} got HTTP {resp.status_code}")
resp = None
continue
break
else:
if resp is None:
raise last_err
if resp.status_code != 200: