harry/ForexBot2
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

audit 2026-09-13: eleven delivery/measurement fixes; books byte-identical

Browse Source 
Signal path
- A shipped setup always names a direction: grade C needs confidence
  >= 45 (gate 6c grants B at >= 45) while BUY/SELL needs >= 50, so C/B
  reports reached the alert gate as NEUTRAL with a full setup. The
  backtester traded them (the replay never reads final_signal) while
  Telegram rejected NEUTRAL first — 5/91 trades, +3.7R of +22.2R, in
  the H1 12-month book. _attach_core_setup stamps the setup's direction
  and repairs the pre-gate action text; no-setup reports stay NEUTRAL.
- analyze_pair fails the scan when ANY required frame is missing
  ("DATA UNAVAILABLE:", scan_kind='failed'); a lost confirmation frame
  used to run with an unearned -15 MTF penalty.

Outcomes and labels
- advisory_outcome.resolve_locked_floor: a gap the database cannot fill
  resolves at the locked trailing stop (GAP_LOCKED, result.floor) when
  the stop had crossed the entry; the rebuild fires on both gap reasons
  (rows 915/920: +1.14R / +0.84R floors; resolved sum -2.46R -> -0.48R).
- label_scan_outcomes: SCAN_LABEL_MAX_AGE_H = 72 h, stale rows never
  anchor the shared bar stream, labeled_at bumped on re-examination,
  _stream_since warns at its row cap (the Labor-Day pile would have
  blinded scan_outcomes around 2026-09-14).

Providers and scheduling
- The daily settlement pause (Mon-Thu 17:00 New York) is a scheduled
  closure for _has_latest_closed_bar (whole-span label walk-back) and
  _is_data_fresh (closed seconds are not age); bot and dashboard skip
  scans whose just-closed label sits in the pause (61 false stale rows
  in three nights).
- fetch_range retries a failed page in place (three failures on one
  window raise) and logs an ERROR for interior gaps of >3 unscheduled
  bars.
- pd.Timestamp.utcnow() replaced in providers/base.py and engine.py;
  requirements pin pandas<4, numpy<3.
- telegram_bot: boundary loops share one scan per bar with the breach
  handler (_remember_scan keys on the analysed bar; breach defers 20 s
  after an M15 boundary); the rotating file handler attaches in main()
  only (the test suite used to write into logs/telegram.log).
- dashboard_publisher: catch_up_timeframes re-scans boundaries missed
  while a cycle overran or the process was down.

Backtester and engine guards
- _replay_span returns the per-span GPU cache-miss count; run_backtest
  sums, logs it (must be 0) and writes cache_fallbacks to the manifest.
- regime_weight_table raises on a partial regime_weights_override.

Verification: 519 tests; M5/M15 2-yr, M15 OOS, H1/H4 3-yr books
byte-identical before and after; live signals.db untouched (checked on
a copy). CLAUDE.md carries the dated entry and the blocked-hours union
semantics that every book was validated under.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_0139xz1J6YccndA2be6yFnEo
This commit is contained in:
Harwinder Sekhon
2026-09-13 21:42:37 -04:00
parent 1c293f7d2e
commit 2836024e9d
12 changed files with 1602 additions and 70 deletions
Download Patch File Download Diff File Expand all files Collapse all files
dashboard_publisher.py
+92 -6
View File
@@ -26,6 +26,11 @@ whichever timeframes just closed. Due-classification uses the RAW wake
time — flooring 12:00:10 to the 5-minute grid yields the 12:00 boundary
that just closed. Do NOT subtract the buffer first: 12:00:10 − 15 s
floors to 11:55 and silently skips every H1/H4 scan (v1 bug, 2026-08-02).
A cycle can overrun its 5-minute slot (or the process can be down), so
each wake also catches up the timeframes whose latest closed boundary is
newer than their last scanned one (`catch_up_timeframes`) — without it the
page served an hours-old H1/H4 report behind a fresh countdown
(2026-09-13 audit).
Scans run sequentially M5→M15→H1→H4 through one shared data provider so
later timeframes hit the warm boundary cache — that ordering IS the
@@ -46,10 +51,12 @@ import os
import shutil
import sys
import time
from datetime import datetime, timezone
from datetime import datetime, timedelta, timezone
from logging.handlers import TimedRotatingFileHandler
from typing import Dict, List, Optional
import pandas as pd
from agent_panel import attach_panel
from app_config import get_app_config, violent_watch_enabled
from config import PlatformConfig, Timeframe, XAUUSD
@@ -230,10 +237,60 @@ def due_timeframes(wake_dt: datetime) -> List[Timeframe]:
"""Timeframes whose candle just closed at the boundary this wake
belongs to. Takes the RAW wake time (boundary + buffer): flooring to
the 5-minute grid yields the boundary that just passed. Never
subtract the buffer first — see module docstring (v1 bug)."""
subtract the buffer first — see module docstring (v1 bug).
A timeframe whose just-closed bar would carry a label inside gold's
daily settlement pause is skipped: no such bar exists, so the scan
would only re-read the previous one (2026-09-13 audit)."""
minutes = wake_dt.hour * 60 + wake_dt.minute
floored = minutes - (minutes % 5)
return [tf for tf in DASHBOARD_TFS if floored % TF_INTERVAL_MINUTES[tf] == 0]
boundary = wake_dt.replace(
hour=floored // 60, minute=floored % 60, second=0, microsecond=0)
return [
tf for tf in DASHBOARD_TFS
if floored % TF_INTERVAL_MINUTES[tf] == 0
and not _providers_base._is_daily_maintenance(
boundary - timedelta(minutes=TF_INTERVAL_MINUTES[tf]))
]
def catch_up_timeframes(wake_dt: datetime,
last_boundary: Dict[str, float]) -> List[Timeframe]:
"""Timeframes whose most recent closed boundary (<= wake) is newer than
the boundary of their last successful scan — i.e. boundaries that were
missed while a cycle overran or the process was down. A TF never scanned
is due. Excludes bars whose label sits in the daily settlement pause
(same rule as due_timeframes).
`due_timeframes` only ever names the timeframes whose boundary coincides
with THIS wake, so an overrunning cycle (scan + advisory panel + news can
take minutes) or a restart left the page serving the previous H1/H4
report for up to 1 h / 4 h while advertising a fresh next_close_epoch
(2026-09-13 audit)."""
minutes = wake_dt.hour * 60 + wake_dt.minute
out = []
for tf in DASHBOARD_TFS:
interval = TF_INTERVAL_MINUTES[tf]
floored = minutes - (minutes % interval)
last_closed = wake_dt.replace(
hour=floored // 60, minute=floored % 60, second=0, microsecond=0)
# No bar carries a label inside a scheduled closure (settlement
# pause or weekend), so nothing can have been "missed" there — and a
# weekend-blocked scan never advances the ledger, so without the
# weekend half of this test every weekend wake would re-scan
# M15/H1/H4 for nothing (~1,800 blocked rows per weekend).
# (Whole-span rule, as in providers.base._has_latest_closed_bar: an
# H4 bar labelled Sunday 20:00 UTC starts in the weekend but runs
# through the reopen, so it exists.)
label = last_closed - timedelta(minutes=interval)
if (_providers_base._is_scheduled_closure(label)
and _providers_base._is_scheduled_closure(
last_closed - timedelta(seconds=1))):
continue
scanned = last_boundary.get(tf.value)
if scanned is None or scanned < last_closed.timestamp():
out.append(tf)
return out
def seconds_until_boundary(interval_minutes: int, buffer: int = 10) -> float:
@@ -324,6 +381,9 @@ class DashboardPublisher:
self.out_dir = out_dir
self.snapshots: Dict[str, dict] = {}
self._reports = {}
# tf.value → epoch seconds of the boundary the last successful scan
# belonged to; feeds catch_up_timeframes (2026-09-13 audit).
self._last_boundary: Dict[str, float] = {}
self.news_payload: Optional[dict] = None
self._dp = None
self._news_engine: Optional[NewsEngine] = None
@@ -355,7 +415,14 @@ class DashboardPublisher:
shutil.copyfile(src, dst + ".tmp")
os.replace(dst + ".tmp", dst)
def scan_tf(self, tf: Timeframe) -> None:
def scan_tf(self, tf: Timeframe,
boundary_epoch: Optional[float] = None) -> None:
"""Scan one timeframe and store its snapshot entry.
`boundary_epoch` is the fallback boundary recorded when the report
carries no analysis (weekend/blocked or failed scan) or its bar
label will not parse; normally the boundary comes from the report
itself (bar label + interval)."""
started = time.time()
try:
report = produce_signal(XAUUSD, tf, self._provider(),
@@ -365,6 +432,19 @@ class DashboardPublisher:
if violent_watch_enabled():
attach_watch(report, tf,
get_app_config().violent_watch_settings)
# Which boundary this report belongs to: the bar LABEL it closed
# on plus one interval. Recorded so a boundary missed by an
# overrunning cycle or a restart is caught up on the next wake
# instead of sitting stale (2026-09-13 audit).
boundary = boundary_epoch
if report.primary_analysis is not None:
try:
boundary = (pd.Timestamp(report.timestamp).timestamp()
+ TF_INTERVAL_MINUTES[tf] * 60)
except Exception:
boundary = boundary_epoch
if boundary is not None:
self._last_boundary[tf.value] = boundary
self._reports[tf.value] = report
self.snapshots[tf.value] = {
"status": "ok",
@@ -449,9 +529,15 @@ class DashboardPublisher:
logger.info(f"next M5 boundary in {delay:.0f}s")
time.sleep(delay)
try:
due = due_timeframes(datetime.now(timezone.utc))
now = datetime.now(timezone.utc)
due = due_timeframes(now)
logger.info(f"boundary wake — scanning {[t.value for t in due]}")
self.run_cycle(due)
missed = catch_up_timeframes(now, self._last_boundary)
extra = [tf for tf in missed if tf not in due]
if extra:
logger.info(f"catch-up — scanning "
f"{[t.value for t in extra]} (missed boundary)")
self.run_cycle(due + extra)
except Exception:
logger.exception("cycle failed")