harry/ForexBot2
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

audit 2026-09-13: eleven delivery/measurement fixes; books byte-identical

Browse Source 
Signal path
- A shipped setup always names a direction: grade C needs confidence
  >= 45 (gate 6c grants B at >= 45) while BUY/SELL needs >= 50, so C/B
  reports reached the alert gate as NEUTRAL with a full setup. The
  backtester traded them (the replay never reads final_signal) while
  Telegram rejected NEUTRAL first — 5/91 trades, +3.7R of +22.2R, in
  the H1 12-month book. _attach_core_setup stamps the setup's direction
  and repairs the pre-gate action text; no-setup reports stay NEUTRAL.
- analyze_pair fails the scan when ANY required frame is missing
  ("DATA UNAVAILABLE:", scan_kind='failed'); a lost confirmation frame
  used to run with an unearned -15 MTF penalty.

Outcomes and labels
- advisory_outcome.resolve_locked_floor: a gap the database cannot fill
  resolves at the locked trailing stop (GAP_LOCKED, result.floor) when
  the stop had crossed the entry; the rebuild fires on both gap reasons
  (rows 915/920: +1.14R / +0.84R floors; resolved sum -2.46R -> -0.48R).
- label_scan_outcomes: SCAN_LABEL_MAX_AGE_H = 72 h, stale rows never
  anchor the shared bar stream, labeled_at bumped on re-examination,
  _stream_since warns at its row cap (the Labor-Day pile would have
  blinded scan_outcomes around 2026-09-14).

Providers and scheduling
- The daily settlement pause (Mon-Thu 17:00 New York) is a scheduled
  closure for _has_latest_closed_bar (whole-span label walk-back) and
  _is_data_fresh (closed seconds are not age); bot and dashboard skip
  scans whose just-closed label sits in the pause (61 false stale rows
  in three nights).
- fetch_range retries a failed page in place (three failures on one
  window raise) and logs an ERROR for interior gaps of >3 unscheduled
  bars.
- pd.Timestamp.utcnow() replaced in providers/base.py and engine.py;
  requirements pin pandas<4, numpy<3.
- telegram_bot: boundary loops share one scan per bar with the breach
  handler (_remember_scan keys on the analysed bar; breach defers 20 s
  after an M15 boundary); the rotating file handler attaches in main()
  only (the test suite used to write into logs/telegram.log).
- dashboard_publisher: catch_up_timeframes re-scans boundaries missed
  while a cycle overran or the process was down.

Backtester and engine guards
- _replay_span returns the per-span GPU cache-miss count; run_backtest
  sums, logs it (must be 0) and writes cache_fallbacks to the manifest.
- regime_weight_table raises on a partial regime_weights_override.

Verification: 519 tests; M5/M15 2-yr, M15 OOS, H1/H4 3-yr books
byte-identical before and after; live signals.db untouched (checked on
a copy). CLAUDE.md carries the dated entry and the blocked-hours union
semantics that every book was validated under.

Claude-Session: https://claude.ai/code/session_0139xz1J6YccndA2be6yFnEo
This commit is contained in:
Harwinder Sekhon
2026-09-13 21:42:37 -04:00
parent be72f3f350
commit 7936c2fe6e
12 changed files with 1602 additions and 70 deletions
Download Patch File Download Diff File Expand all files Collapse all files
providers/base.py
+122 -20
View File
@@ -65,6 +65,46 @@ def _is_forex_weekend(now=None) -> bool:
return False
def _is_daily_maintenance(now=None) -> bool:
"""Gold's daily settlement pause: Monday-Thursday 17:00-17:59 New York
(21:00 UTC in US summer, 22:00 in winter). No bar carries a label
inside this hour. Friday 17:00 is the weekend (see _is_forex_weekend).
Same rule as advisory_outcome._has_continuity's daily_maintenance."""
if now is None:
now = pd.Timestamp.now(tz="UTC")
ny = now.astimezone(_NY_TZ)
return ny.weekday() < 4 and ny.hour == 17
def _is_scheduled_closure(now=None) -> bool:
"""The feed is legitimately silent: weekend or daily settlement pause."""
return _is_forex_weekend(now) or _is_daily_maintenance(now)
def _scheduled_closed_seconds(start, end) -> float:
"""Seconds of the [start, end) span that fall inside a scheduled
closure (weekend or the daily settlement pause).
Bar age is measured from a bar LABEL, so at 21:30 UTC in summer the
20:55 M5 bar looks 35 minutes old even though the feed could not have
printed anything since 21:00. Subtracting the closed span keeps the
3-candle freshness limit meaningful instead of failing it — and
logging a false "feed lagging"/"STALE DATA" — every settlement hour
(2026-09-13 audit: 61 suppressed scans on three nights).
"""
hour = pd.Timedelta(hours=1)
closed = 0.0
t = start.floor("h")
steps = 0
while t < end and steps < 24 * 10:
if _is_scheduled_closure(t):
closed += max(
0.0, (min(t + hour, end) - max(t, start)).total_seconds())
t += hour
steps += 1
return closed
class DataIntegrityError(Exception):
"""Raised when data fails validation checks."""
pass
@@ -276,7 +316,7 @@ class BaseDataProvider(ABC):
chunks = []
cursor_end = end
empty_streak = 0
error_streak = 0
window_errors = 0
while cursor_end > start:
cursor_start = max(start, cursor_end - timedelta(days=days_per_chunk))
try:
@@ -286,23 +326,31 @@ class BaseDataProvider(ABC):
)
except Exception as e:
# A raising chunk is a PROVIDER ERROR (auth, 429, 5xx,
# malformed payload) — not "no data". Three in a row means
# the rest of the history is unreachable: raise instead of
# returning a silently truncated frame (the backtester
# would treat it as complete).
# malformed payload) — not "no data". RETRY THE SAME
# WINDOW: until 2026-09-13 a failure advanced the cursor
# past the window, so one or two NON-consecutive page
# failures (Oanda's sporadic 401/5xx, even after the
# provider's own retry ladder) silently dropped up to
# days_per_chunk of bars — 15 days of M5 — out of the
# middle of the frame, and only the HEAD was
# coverage-checked below, so the backtester cached the hole
# as complete history forever. Three failures on the SAME
# window means it is unreachable: raise instead of
# returning a frame with a hole in it.
logger.warning(
f"Chunk {cursor_start.date()}–{cursor_end.date()} failed: {e}"
)
error_streak += 1
if error_streak >= 3:
window_errors += 1
if window_errors >= 3:
raise self.DATA_ERROR(
f"{symbol} {timeframe.value}: {error_streak} "
f"consecutive chunk failures walking back from "
f"{cursor_end.date()} — refusing to return "
f"truncated history (last error: {e})") from e
cursor_end = cursor_start - timedelta(seconds=1)
continue
f"{symbol} {timeframe.value}: {window_errors} "
f"consecutive failures fetching window "
f"{cursor_start.date()}–{cursor_end.date()} — "
f"refusing to return truncated history "
f"(last error: {e})") from e
continue # re-request the SAME window
window_errors = 0
if chunk is None or chunk.empty:
# Genuinely empty windows (weekends/holidays/asset gaps)
# are normal; several in a row = start of history.
@@ -313,7 +361,6 @@ class BaseDataProvider(ABC):
continue
empty_streak = 0
error_streak = 0
chunks.append(chunk)
# Step back; dedup on concat.
new_cursor = chunk.index[0] - timedelta(seconds=1)
@@ -348,6 +395,38 @@ class BaseDataProvider(ABC):
f"{df.index[0]} but {start_ts} was requested — "
f"~{(df.index[0] - start_ts).days} days missing from the "
f"head; treat downstream results accordingly")
# INTERIOR-gap audit (2026-09-13): the head check above is blind to
# a hole in the MIDDLE of the frame, which is exactly what a
# dropped page used to leave behind. Missing labels inside a
# scheduled closure (weekend, daily settlement pause) are normal,
# so only unscheduled runs are reported — and a real market holiday
# (Christmas, Good Friday, July 4) looks identical to a lost page,
# so this is LOUD BUT NOT FATAL and the frame is returned unchanged.
if len(df) > 400_000:
logger.info(
f"{symbol} {timeframe.value}: interior-gap audit skipped "
f"({len(df)} rows)")
elif len(df) > 1:
step = pd.Timedelta(seconds=self.CANDLE_SECONDS[timeframe])
labels = df.index
# Missing labels between each consecutive pair; >3 is the floor
# worth a look (weekends/settlement hours leave 1-2 labels that
# the NY-anchored closure rule does not cover).
spans = ((labels[1:] - labels[:-1]) / step).astype("int64") - 1
gaps = []
for i in (spans > 3).nonzero()[0]:
missing = sum(
1 for k in range(1, int(spans[i]) + 1)
if not _is_scheduled_closure(labels[i] + k * step))
if missing > 3:
gaps.append((labels[i], labels[i + 1], missing))
if gaps:
logger.error(
f"{symbol} {timeframe.value}: {len(gaps)} interior "
f"gap(s) of >3 unscheduled bars in fetched history — "
f"first: {gaps[0]}; a page fetch may have been dropped "
f"or the market was closed (holiday); verify before "
f"trusting this window")
return df
def is_data_fresh(
@@ -398,7 +477,8 @@ class BaseDataProvider(ABC):
# ── Shared internals ───────────────────────────────────────────
def _has_latest_closed_bar(self, df: pd.DataFrame, timeframe: Timeframe) -> bool:
def _has_latest_closed_bar(self, df: pd.DataFrame, timeframe: Timeframe,
now=None) -> bool:
"""Cached data must include the most recent fully closed bar.
The TTL alone is wall-clock based: a snapshot fetched mid-candle is
@@ -408,12 +488,29 @@ class BaseDataProvider(ABC):
"""
if df is None or df.empty:
return False
now = pd.Timestamp.now(tz="UTC")
if now is None:
now = pd.Timestamp.now(tz="UTC")
if _is_forex_weekend(now):
return True
candle_seconds = self.CANDLE_SECONDS.get(timeframe, 3600)
dur = pd.Timedelta(seconds=candle_seconds)
expected_label = now.floor(dur) - dur
# No bar ever carries a label inside a scheduled closure, so the
# newest bar that MUST be present is the last label before it.
# Without this walk-back every scan from 21:05 to 22:00 UTC in
# summer evicted the cache, refetched twice and logged a false
# "feed lagging" (2026-09-13 audit).
# A label counts as "inside" only when the closure covers the bar's
# whole span: a D1 bar labelled Sunday 00:00 UTC starts in the
# weekend but runs through the Sunday reopen, so it exists and
# must be present (68 such bars in the 3-year Oanda D1 cache).
steps = 0
last_second = dur - pd.Timedelta(seconds=1)
while (_is_scheduled_closure(expected_label)
and _is_scheduled_closure(expected_label + last_second)
and steps < 2000):
expected_label -= dur
steps += 1
return df.index[-1] >= expected_label
def _drop_forming_candle(self, df: pd.DataFrame, timeframe: Timeframe) -> pd.DataFrame:
@@ -443,12 +540,14 @@ class BaseDataProvider(ABC):
timeframe.value, dropped)
return df
def _is_data_fresh(self, df: pd.DataFrame, symbol: str, timeframe: Timeframe) -> bool:
def _is_data_fresh(self, df: pd.DataFrame, symbol: str, timeframe: Timeframe,
now=None) -> bool:
"""Check if the last candle is within acceptable age for this timeframe."""
if df is None or df.empty:
return False
now = pd.Timestamp.now(tz="UTC")
if now is None:
now = pd.Timestamp.now(tz="UTC")
if _is_forex_weekend(now):
return True
@@ -456,7 +555,10 @@ class BaseDataProvider(ABC):
last_bar = df.index[-1]
candle_seconds = self.CANDLE_SECONDS.get(timeframe, 3600)
max_age_seconds = candle_seconds * 3
age_seconds = (now - last_bar).total_seconds()
# Scheduled closures (weekend, daily settlement pause) are dead
# time in which no bar can print — they must not count as age.
age_seconds = ((now - last_bar).total_seconds()
- _scheduled_closed_seconds(last_bar, now))
if age_seconds > max_age_seconds:
logger.warning(
@@ -482,7 +584,7 @@ class BaseDataProvider(ABC):
df = df.sort_index()
df = df[~df.index.duplicated(keep="last")]
now = pd.Timestamp.utcnow()
now = pd.Timestamp.now(tz="UTC")
future_mask = df.index > now + pd.Timedelta(minutes=5)
if future_mask.any():
logger.warning(f"{symbol}: Dropping {future_mask.sum()} future timestamps")