harry/ForexBot2
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

audit 2026-09-13: eleven delivery/measurement fixes; books byte-identical

Browse Source 
Signal path
- A shipped setup always names a direction: grade C needs confidence
  >= 45 (gate 6c grants B at >= 45) while BUY/SELL needs >= 50, so C/B
  reports reached the alert gate as NEUTRAL with a full setup. The
  backtester traded them (the replay never reads final_signal) while
  Telegram rejected NEUTRAL first — 5/91 trades, +3.7R of +22.2R, in
  the H1 12-month book. _attach_core_setup stamps the setup's direction
  and repairs the pre-gate action text; no-setup reports stay NEUTRAL.
- analyze_pair fails the scan when ANY required frame is missing
  ("DATA UNAVAILABLE:", scan_kind='failed'); a lost confirmation frame
  used to run with an unearned -15 MTF penalty.

Outcomes and labels
- advisory_outcome.resolve_locked_floor: a gap the database cannot fill
  resolves at the locked trailing stop (GAP_LOCKED, result.floor) when
  the stop had crossed the entry; the rebuild fires on both gap reasons
  (rows 915/920: +1.14R / +0.84R floors; resolved sum -2.46R -> -0.48R).
- label_scan_outcomes: SCAN_LABEL_MAX_AGE_H = 72 h, stale rows never
  anchor the shared bar stream, labeled_at bumped on re-examination,
  _stream_since warns at its row cap (the Labor-Day pile would have
  blinded scan_outcomes around 2026-09-14).

Providers and scheduling
- The daily settlement pause (Mon-Thu 17:00 New York) is a scheduled
  closure for _has_latest_closed_bar (whole-span label walk-back) and
  _is_data_fresh (closed seconds are not age); bot and dashboard skip
  scans whose just-closed label sits in the pause (61 false stale rows
  in three nights).
- fetch_range retries a failed page in place (three failures on one
  window raise) and logs an ERROR for interior gaps of >3 unscheduled
  bars.
- pd.Timestamp.utcnow() replaced in providers/base.py and engine.py;
  requirements pin pandas<4, numpy<3.
- telegram_bot: boundary loops share one scan per bar with the breach
  handler (_remember_scan keys on the analysed bar; breach defers 20 s
  after an M15 boundary); the rotating file handler attaches in main()
  only (the test suite used to write into logs/telegram.log).
- dashboard_publisher: catch_up_timeframes re-scans boundaries missed
  while a cycle overran or the process was down.

Backtester and engine guards
- _replay_span returns the per-span GPU cache-miss count; run_backtest
  sums, logs it (must be 0) and writes cache_fallbacks to the manifest.
- regime_weight_table raises on a partial regime_weights_override.

Verification: 519 tests; M5/M15 2-yr, M15 OOS, H1/H4 3-yr books
byte-identical before and after; live signals.db untouched (checked on
a copy). CLAUDE.md carries the dated entry and the blocked-hours union
semantics that every book was validated under.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_0139xz1J6YccndA2be6yFnEo
This commit is contained in:
Harwinder Sekhon
2026-09-13 21:42:37 -04:00
parent 1c293f7d2e
commit 2836024e9d
12 changed files with 1602 additions and 70 deletions
Download Patch File Download Diff File Expand all files Collapse all files
signal_log.py
+127 -16
View File
@@ -104,6 +104,7 @@ from advisory_outcome import (
canonical_r,
initial_outcome,
normalize_timeframe,
resolve_locked_floor,
timeframe_value,
)
# Private on purpose — the scan-outcome labeller must apply EXACTLY the
@@ -258,6 +259,7 @@ _GATE_PREFIXES = (
"SESSION FILTER",
"Primary timeframe stale",
"Primary timeframe data unavailable",
"DATA UNAVAILABLE:",
"REGIME FILTER",
"EXHAUSTION BLOCK",
"EXHAUSTED TREND FILTER",
@@ -278,6 +280,10 @@ SCAN_OUTCOME_WINDOW_H = 24
# A 24 h window can only be judged once it has elapsed; the extra hour
# leaves room for the last bar of the window to be stored.
SCAN_LABEL_MIN_AGE_H = 25
# An incomplete row older than this is FINAL: its window had a real hole
# (a feed outage), which no later bar can fill. It is neither re-examined
# nor allowed to anchor the shared bar stream for the rest of the batch.
SCAN_LABEL_MAX_AGE_H = 72
# Missing bars tolerated inside a window ON TOP of the scheduled closures
# advisory_outcome._has_continuity already allows.
SCAN_LABEL_MAX_GAP_BARS = 3
@@ -399,6 +405,20 @@ def setup_key(setup: Optional[Dict[str, Any]], timeframe: Any) -> Optional[str]:
))
def _is_gap_reason(reason: Any) -> bool:
"""True when an outcome was left UNRESOLVABLE by a candle gap.
advance_outcome writes TWO such reasons — "stored candle snapshots
lost continuity" (no overlap with the last processed bar) and "stored
candle snapshots contain an internal gap" — and each call site used to
match only one of them, so the stream rebuild and the locked-floor
rescue skipped half the gap rows. The substring clauses keep states
written by older code inside the same net."""
text = str(reason or "")
return (text.startswith("stored candle snapshots")
or "continuity" in text or "gap" in text)
def gate_reason(
warnings: Iterable[str],
cleaned_warnings: Iterable[str],
@@ -1150,12 +1170,19 @@ class SignalLogger:
before = _dumps(json.loads(hit[0]))
advanced = advance_outcome(json.loads(before), bars)
if (advanced.get("status") == "UNRESOLVABLE"
and "continuity" in str(advanced.get("reason", ""))):
and _is_gap_reason(advanced.get("reason"))):
union = self._stream_since(
symbol, timeframe,
json.loads(before)["progress"]["last_processed_bar"])
if union:
advanced = advance_outcome(json.loads(before), union)
# The database could not fill the hole. A trade
# whose trail had already locked profit still has a
# real answer — its floor — so record that instead
# of dropping the row out of the scorecard.
if (advanced.get("status") == "UNRESOLVABLE"
and _is_gap_reason(advanced.get("reason"))):
advanced = resolve_locked_floor(advanced)
after = _dumps(advanced)
if after == before:
self._conn.rollback()
@@ -1194,6 +1221,11 @@ class SignalLogger:
" WHERE symbol=? AND candles_json IS NOT NULL"
" AND logged_at >= ? ORDER BY id LIMIT ?",
(f'$."{key}"', symbol, since, int(max_rows))).fetchall()
if len(rows) == int(max_rows):
logger.warning(
"signal_log: bar stream truncated for %s %s (key %s)"
" since %s — hit the %s-row cap, later bars are missing",
symbol, tf, key, since, int(max_rows))
for (blob,) in rows:
if not blob:
continue
@@ -1212,42 +1244,82 @@ class SignalLogger:
Candidates are rows that carry indicator snapshots (a real scan)
and a price, whose signal bar closed more than 25 h ago and which
have no COMPLETE label yet; the least recently labelled go first,
so a permanently short stream can never starve the rest. Returns
the number of rows actually WRITTEN — a row whose numbers did not
change is skipped, which makes repeated passes idempotent and lets
a batch driver stop at 0.
have no COMPLETE label yet. An incomplete row is re-examined only
while it is younger than SCAN_LABEL_MAX_AGE_H (counted from the
newest logged row when the database is behind the clock) — beyond
that its window had a real hole that no later bar can fill, so the
label it already carries is final — and a stale row never anchors
the shared bar stream either (a key that has nothing else due
starts at the age bound plus one window). Both bounds are
load-bearing: a handful of permanently incomplete rows (a feed
outage) sort to the front of every pass, and anchoring the stream
at their bar pushes its row cap past today, which silently labels
every genuinely due row NULL and incomplete — the scan_outcomes
table then goes blind. Returns the number of rows actually WRITTEN
— a row whose numbers did not change is not rewritten (only its
labeled_at is bumped, so it rotates to the back of the queue),
which makes repeated passes idempotent and lets a batch driver
stop at 0.
Bounded on purpose: at most ``budget`` candidates are examined and
the pass stops after ``time_budget`` seconds (default 2 s) so the
opportunistic call inside log() stays cheap. Per-row failures are
logged and skipped."""
moment = now or datetime.now(timezone.utc)
cutoff = _utc(moment) - timedelta(hours=SCAN_LABEL_MIN_AGE_H)
moment_utc = _utc(moment)
cutoff = moment_utc - timedelta(hours=SCAN_LABEL_MIN_AGE_H)
deadline = time.monotonic() + float(
SCAN_LABEL_TIME_BUDGET if time_budget is None else time_budget)
written = 0
with self._lock:
# Age is measured from the newest row when the database itself
# is behind the clock (an imported snapshot on the dev box),
# otherwise every row of a 3-day-old import would be declared
# final on the first pass. On a live logger the two coincide.
horizon = moment_utc
newest = self._conn.execute(
"SELECT logged_at FROM signals ORDER BY id DESC"
" LIMIT 1").fetchone()
try:
horizon = min(horizon, _utc(datetime.fromisoformat(
str(newest[0]).replace(" ", "T"))))
except (TypeError, ValueError, IndexError):
pass
stale_before = horizon - timedelta(hours=SCAN_LABEL_MAX_AGE_H)
# Deepest a stream may start when every row due for that key is
# stale: one window before the age bound, so such a row still
# gets its single final pass without dragging the stream back.
anchor_floor = stale_before - timedelta(hours=SCAN_LABEL_MIN_AGE_H)
# Row ids are monotonic in logged_at (a uniform ISO string), so
# an id bound is the cheap, index-friendly way to leave stale
# incomplete rows out of the re-examination set. bar_ts mixes
# separators and must never be compared as a string.
min_id = self._conn.execute(
"SELECT MIN(id) FROM signals WHERE logged_at >= ?",
(stale_before.isoformat(),)).fetchone()[0]
if min_id is None:
min_id = 2 ** 62 # nothing recent: re-examine none
rows = self._conn.execute(
"SELECT s.id, s.symbol, s.timeframe, s.price,"
" COALESCE(s.bar_ts, json_extract(s.indicators_json,"
" '$.\"' || s.timeframe || '\".timestamp')) AS bar,"
" json_extract(s.indicators_json,"
" '$.\"' || s.timeframe || '\".key_levels.atr') AS atr"
" '$.\"' || s.timeframe || '\".key_levels.atr') AS atr,"
" (o.signal_id IS NULL) AS unlabelled"
" FROM signals s"
" LEFT JOIN scan_outcomes o ON o.signal_id = s.id"
" WHERE s.indicators_json IS NOT NULL AND s.price > 0"
" AND (o.signal_id IS NULL OR o.complete = 0)"
" AND (o.signal_id IS NULL OR (o.complete = 0 AND s.id >= ?))"
" AND bar IS NOT NULL"
" ORDER BY COALESCE(o.labeled_at, '') ASC, s.id ASC"
" LIMIT ?", (int(budget),)).fetchall()
" LIMIT ?", (int(min_id), int(budget))).fetchall()
# One bar stream per (symbol, timeframe) for the whole batch,
# built from the OLDEST bar in it — _stream_since is a scan
# over every row logged since that bar, far too expensive to
# repeat per candidate.
due: List[Tuple[Any, ...]] = []
oldest: Dict[Tuple[str, str], Tuple[datetime, str]] = {}
for rowid, symbol, tf, price, bar, atr in rows:
for rowid, symbol, tf, price, bar, atr, unlabelled in rows:
try:
stamp = _utc(datetime.fromisoformat(
str(bar).replace(" ", "T")))
@@ -1255,8 +1327,16 @@ class SignalLogger:
continue
if stamp > cutoff:
continue # window has not elapsed yet
# Stale = old by the bar it closed on AND by when it was
# written (on a live database the two agree within minutes;
# requiring both keeps a freshly imported or backfilled DB,
# whose rows all carry today's logged_at, out of it).
stale = stamp < stale_before and rowid < min_id
if stale and not unlabelled:
continue # final: a hole no later bar can ever fill
key = (str(symbol), str(tf))
if key not in oldest or stamp < oldest[key][0]:
if not stale and (key not in oldest
or stamp < oldest[key][0]):
oldest[key] = (stamp, str(bar))
due.append((rowid, symbol, tf, price, bar, atr, key))
streams: Dict[Tuple[str, str], List[Dict[str, Any]]] = {}
@@ -1265,8 +1345,13 @@ class SignalLogger:
break
try:
if key not in streams:
# 3 days of anchor is ~2,100 rows on the live DB;
# the cap is a safety net, not the working limit.
anchor = oldest.get(key)
streams[key] = self._stream_since(
symbol, tf, oldest[key][1])
symbol, tf,
anchor[1] if anchor else anchor_floor.isoformat(),
max_rows=10000)
label = scan_outcome(bar, tf, streams[key],
_finite(atr), price)
if label is None:
@@ -1277,6 +1362,14 @@ class SignalLogger:
+ " FROM scan_outcomes WHERE signal_id=?",
(rowid,)).fetchone()
if prior is not None and tuple(prior) == values:
# Unchanged, so NOT counted as written — but its
# queue position moves, or the same short-stream
# rows would head every pass for ever.
now_iso = datetime.now(timezone.utc).isoformat()
self._conn.execute(
"UPDATE scan_outcomes SET labeled_at=?"
" WHERE signal_id=?", (now_iso, rowid))
self._conn.commit()
continue
self._conn.execute(
_SCAN_OUTCOME_INSERT,
@@ -1376,6 +1469,9 @@ class SignalLogger:
continue
before = _dumps(state)
advanced = advance_outcome(json.loads(before), union)
if (advanced.get("status") == "UNRESOLVABLE"
and _is_gap_reason(advanced.get("reason"))):
advanced = resolve_locked_floor(advanced)
after = _dumps(advanced)
if after == before:
continue
@@ -1393,7 +1489,9 @@ class SignalLogger:
"""Bounded retry for continuity-broken outcomes: re-freeze the
seed from the row's own setup/indicators (at most ``max_reseeds``
times per row) so the next resolver pass can walk the now-filled
stream. Invalid-seed rows stay terminal."""
stream. Invalid-seed rows stay terminal. A gap row that had already
trailed its stop past the entry is finished at that locked floor
instead of re-seeded; the returned list holds re-seeded ids only."""
reseeded: List[int] = []
with self._lock:
rows = self._conn.execute(
@@ -1403,8 +1501,21 @@ class SignalLogger:
" AND advisory_event_id = id").fetchall()
for rowid, tf, setup_json, ind_json, state_json in rows:
old = json.loads(state_json) if state_json else {}
if "continuity" not in str(old.get("reason", "")) and \
"gap" not in str(old.get("reason", "")):
if not _is_gap_reason(old.get("reason")):
continue
# The locked trailing stop lives ONLY in this progress
# block, and a re-seed rebuilds the walk from scratch over
# a stream whose hole is still there (the continuity check
# is all-or-nothing), so the floor would be lost for good.
# Finish such a row here; re-seed only what has nothing.
floored = resolve_locked_floor(json.loads(_dumps(old)))
if floored.get("status") == "RESOLVED":
self._conn.execute(
"UPDATE signals SET advisory_outcome_status=?,"
" advisory_r_multiple=?, advisory_outcome_json=?"
" WHERE id=?",
("RESOLVED", canonical_r(floored),
_dumps(floored), rowid))
continue
count = int((old.get("reseed") or {}).get("count", 0))
if count >= max_reseeds or not setup_json: