harry/ForexBot2
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

Daily-trend veto on M15 + lot-size / already-open lines (campaign 2026-09-30), and the 2026-10-01 whole-codebase bug review

Browse Source 
Two bodies of work that touch the same files, so one commit.

Direction and setup-size campaign (2026-09-30 → 10-01, owner's RISK-FIRST rule,
written before the runs):
- SIGNAL: M15 daily-trend veto (factory override 7, xau_strategy.daily_trend_veto
  "both"). A finished setup against a clear daily trend (closed D1 bars, Sunday
  stubs dropped; close vs its 50-bar EMA AND vs the close 60 kept days earlier)
  is removed as the last step of the core. One definition
  (signal_aggregator.daily_trend_series) for live, the bar replay and the twin's
  host side; D1 is a required frame of the M15 scan; mirrored in
  cuda_signal_kernels.py K2. W3 824 / +205.12R / PF 1.73 / max DD 12.4R vs
  1105 / +195.79R / 1.49 / 20.9R; 2019 → 2026 1473 / +223.33R / 1.42 / 41.5R vs
  2091 / +192.53R / 1.24 / 64.8R; unseen holdout −4.62R vs −6.63R.
- FAILED, not shipped: the veto on M5 (W3 drawdown 10.1R → 15.2R) and on H1
  (−9.7R), buy-only and sell-only on M15, the wide-stop cap
  (xau_strategy.wide_stop_cap_atr stays 0.0, a research knob).
- New M15 pins: W2 dfc647ee…, OOS e87b6de1…, W3 ad3b0dbd…. M5 / H1 / H4 pins
  unchanged. --set xau_strategy.daily_trend_veto=off reproduces the previous M15
  pins byte for byte. Twin and reference script MD5-equal on every window;
  recorder and feature-export parity re-proved; base-rate table rebuilt.
- DISPLAY (books byte-identical): exposure.py — the lot size for
  trade_size.risk_per_trade (new config.json block, off by default, private to
  the owner) and the count of alerted same-direction plans still open
  (signals.alerted_at, read-only open_alerted_setups); the daily trend in words
  on Telegram, the CLI and the dashboard; signals.daily_trend logged.
- providers/oanda: a date window with no bars is an empty page, not an error
  (a D1 pad starting on a Saturday could not be fetched).
- Reports only, nothing shipped: seven context screens (no state nominated) and
  eight outside data sources (none adds a forecast to gold's own daily trend).
- Independent verification (report section 5.13): every figure reproduced on
  both scripts, scoring identical to the previous commit on all four
  timeframes, no look-ahead. Caveats recorded there: this is a risk filter, not
  a more accurate direction call; nine tenths of the 2019 → 2026 gain is 2022;
  the neighbour clause is thin (a simple 100-day average misses the R line by
  1.2R); "never adds a trade" is an observation, not a guarantee.

Bug review of the whole codebase (2026-10-01). Signal path clean. 17 defects
elsewhere, plus 4 found by an independent review of the fixes; all fixed, each
with a test; no trade book changed:
- telegram_bot: one _deliver_alert for the candle scan and the level break. The
  row is stamped alerted_at BEFORE the send, on a one-worker alert-db pool, and
  un-stamped when no chat received it or the task is cancelled mid-stamp, so
  two alerts at one candle close count each other and no SQLite write runs on
  the event loop. The 5-bar cooldown counts bars of the scan's own frame
  (== gpu_backtest._apply_cooldown; clock time released the 22:00 UTC bar after
  the settlement pause) and is restored from signals.db at start-up. The
  lot-size line goes to the owner's own chat only (a group the owner ran
  /autoscan in used to receive it). /status gains a news-model line and escapes
  failure details.
- signal_log: mark_alerted returns the stamp it wrote; unmark_alerted;
  last_alerted_bars (read-only).
- advisory_outcome: candle continuity is judged bar by bar inside the walk; a
  hole after a trade's exit no longer voids it. The stored-candle replay
  resolves 108 of 111 logged setups instead of 99, all equal to the live
  resolver's R.
- providers/oanda: an empty date window is accepted only when it is 5 days or
  shorter and no malformed candle was skipped; a longer one is retried, then
  refused, instead of being cached as history with a hole.
- news_engine: blank RSS titles skipped; one unreadable headline no longer
  costs the local model its whole batch (checked on the real bundle); model
  health recorded; no DeepSeek audit call in a process that does not archive;
  event-direction cache keyed per release; peek_week_events never fetches.
- llm_client: chat_json never raises and never logs the API key (requests
  quotes the Authorization value in InvalidHeader; error bodies are redacted).
- signal_aggregator: the scan's news block is guarded; the replay path
  (use_news=False) assigns exactly the values it did before.
- agent_panel: PANEL_PROMPT_VERSION v2.2 — same prompt text, corrected input
  (event timing over the cached calendar week, headlines newest first).
  scripts/panel_replay.py replays the logged payload per persona;
  scripts/label_scan_outcomes.py counts with an ISO cutoff.
- test_all.py: no ALTER TABLE … DROP COLUMN (the live box has SQLite 3.34);
  tests that write Parquet print a notice and return without pyarrow.

Verification after the last code change: suite 632 passed, 0 failed (also with
both Parquet engines blocked and FB2_BACKEND=cpu, the live box's view); the
five reference MD5s reproduce on both backtesters, 0 cache fallbacks; Module 37
green; static Python 3.9 / SQLite 3.34 check clean on every touched file.

Evidence: backtest/research/daily_trend_campaign_2026-09-30.md,
context_screens_2026-10.md, new_inputs_2026-10.md; CLAUDE.md sections 2, 4, 6,
8, 9, 10 and 11; pair_signals/xauusd.py docstring (M15 override 7).
This commit is contained in:
Harwinder Sekhon
2026-10-01 10:28:28 -04:00
parent 42025da0b0
commit 2cc5915b00
31 changed files with 6021 additions and 286 deletions
Download Patch File Download Diff File Expand all files Collapse all files
signal_log.py
+220 -5
View File
@@ -84,6 +84,23 @@ or news_engine) so this module stays at the bottom of the import graph.
time-boxed; batch driver: scripts/label_scan_outcomes.py). Readers
must dedup by the FIRST row per (timeframe, bar_ts) — several surfaces
log the same bar.
2026-09-30 daily-trend veto:
* Row column ``daily_trend`` (+1 / 0 / -1 from closed D1 bars: computed
by the core where the veto is on, stamped by the live wrapper for
display where it is off; NULL on blocked / failed scans and when the
daily frame could not be read) and the gate prefix ``DAILY TREND
FILTER`` in ``gate_reason``.
* Row column ``alerted_at`` (UTC time of the Telegram alert for this row;
``mark_alerted``) and the read-only helper ``open_alerted_setups`` —
alerted plans whose advisory outcome is still PENDING — behind the
"already open" line on the alert (exposure.py). Since 2026-10-01 the
stamp is written just BEFORE the send and cleared again
(``unmark_alerted``) when no chat received the alert, so two alerts
sent at the same candle close count each other; ``last_alerted_bars``
reads the newest alerted bar per timeframe so the bot's entry cooldown
survives a restart.
"""
import hashlib
@@ -100,6 +117,7 @@ from typing import Any, Dict, Iterable, List, Optional, Tuple
import app_config
from advisory_outcome import (
MAX_HOLD_BARS,
advance_outcome,
canonical_r,
initial_outcome,
@@ -240,6 +258,14 @@ _ADDITIVE_SIGNAL_COLS = (
("panel_probs_json", "TEXT"),
("panel_payloads_json", "TEXT"),
("panel_context_json", "TEXT"),
# 2026-09-30 daily-trend veto (see module docstring).
("daily_trend", "INTEGER"),
# 2026-09-30: UTC time of the Telegram alert for THIS row (mark_alerted:
# written by the bot just before the send, cleared by unmark_alerted
# when no chat received it). NULL on every row that was scanned but not
# alerted — the "which plans did the reader actually receive" key the
# open-exposure line reads (open_alerted_setups).
("alerted_at", "TEXT"),
)
# Read-side compatibility view `signals_with_news`: legacy rows keep
@@ -262,6 +288,11 @@ _GATE_PREFIXES = (
"Primary timeframe stale",
"Primary timeframe data unavailable",
"DATA UNAVAILABLE:",
# The daily-trend veto removes a setup that every earlier gate had let
# through, so on such a row it IS the reason: listed ahead of the chain
# because the vetoed setup's stale warnings (a cleaned SPREAD GATE, the
# chart-pattern path's cleaned vetoes) are still in the pool.
"DAILY TREND FILTER",
"REGIME FILTER",
"EXHAUSTION BLOCK",
"EXHAUSTED TREND FILTER",
@@ -313,6 +344,7 @@ _SIGNAL_INSERT_COLS = (
"config_fingerprint",
"bar_ts", "engine_fingerprint", "code_version", "penalty_tags_json",
"penalty_total", "momentum_fading", "exhaustion_count",
"daily_trend",
)
_INSERT = (
"INSERT INTO signals (" + ", ".join(_SIGNAL_INSERT_COLS) + ") VALUES ("
@@ -330,6 +362,12 @@ def _finite(x: Any) -> Optional[float]:
return f if math.isfinite(f) else None
def _daily_trend_value(x: Any) -> Optional[int]:
"""The report's daily trend as -1 / 0 / +1, or None when it carries none."""
f = _finite(x)
return None if f is None else int(f)
def _iso(ts: Any) -> Optional[str]:
if ts is None:
return None
@@ -815,6 +853,7 @@ def _report_row(
"momentum_fading": int(bool(getattr(report, "momentum_fading", False))),
"exhaustion_count": int(
_finite(getattr(report, "exhaustion_count", 0)) or 0),
"daily_trend": _daily_trend_value(getattr(report, "daily_trend", None)),
}
@@ -903,6 +942,11 @@ class SignalLogger:
self._conn.execute(
"CREATE INDEX IF NOT EXISTS idx_signals_tf_bar"
" ON signals (timeframe, bar_ts)")
# Partial index: only alerted rows (a few per day), so the
# open-exposure read on every scan never walks the table.
self._conn.execute(
"CREATE INDEX IF NOT EXISTS idx_signals_alerted"
" ON signals (symbol, alerted_at) WHERE alerted_at IS NOT NULL")
self._conn.commit()
def log(
@@ -1106,6 +1150,31 @@ class SignalLogger:
(str(level or ""), kinds_text, str(line or ""), int(rowid)))
self._conn.commit()
def mark_alerted(self, rowid: int,
when_iso: Optional[str] = None) -> Optional[str]:
"""Stamp the time of the alert for this row. The first stamp wins:
a re-alert of the same row keeps the original time. Returns the
stamp when THIS call wrote it and None when the row already had
one (or does not exist), so a caller that must undo its own stamp
can never clear an earlier alert's."""
when_iso = when_iso or datetime.now(timezone.utc).isoformat()
with self._lock:
cur = self._conn.execute(
"UPDATE signals SET alerted_at=?"
" WHERE id=? AND alerted_at IS NULL", (when_iso, int(rowid)))
self._conn.commit()
return when_iso if cur.rowcount == 1 else None
def unmark_alerted(self, rowid: int, when_iso: str) -> bool:
"""Clear the stamp mark_alerted returned (the alert reached no
chat). Only that exact stamp is cleared; True when it was."""
with self._lock:
cur = self._conn.execute(
"UPDATE signals SET alerted_at=NULL"
" WHERE id=? AND alerted_at=?", (int(rowid), str(when_iso)))
self._conn.commit()
return cur.rowcount == 1
def resolve_pending(
self, symbol: str, candles_json: Optional[str], limit: int = 100
) -> int:
@@ -1488,11 +1557,12 @@ class SignalLogger:
old = json.loads(state_json) if state_json else {}
if not _is_gap_reason(old.get("reason")):
continue
# The locked trailing stop lives ONLY in this progress
# block, and a re-seed rebuilds the walk from scratch over
# a stream whose hole is still there (the continuity check
# is all-or-nothing), so the floor would be lost for good.
# Finish such a row here; re-seed only what has nothing.
# The locked trailing stop lives in this progress block. A
# re-seed would rebuild the walk from scratch only to stop
# at the same hole (advance_outcome checks continuity bar
# by bar and keeps the progress up to the hole) and be
# floored at the same stop, so finish such a row here;
# re-seed only what has nothing locked.
floored = resolve_locked_floor(json.loads(_dumps(old)))
if floored.get("status") == "RESOLVED":
self._conn.execute(
@@ -1673,6 +1743,151 @@ def log_watch_update(report: Any, watch: Any) -> None:
exc_info=True)
def _logged_row(report: Any) -> Optional[Tuple["SignalLogger", int]]:
"""(logger, rowid) of the row log_signal wrote for this scan; None for
a report that was never logged or a process with logging off."""
rowid = getattr(report, "_signal_log_rowid", None)
if rowid is None or not app_config.signal_log_enabled():
return None
path = app_config.signal_log_path()
with _loggers_lock:
lg = _loggers.get(path)
return None if lg is None else (lg, int(rowid))
def mark_alerted(report: Any) -> Optional[str]:
"""Stamp `alerted_at` on the row logged for this scan. The bot calls
this just BEFORE it sends the alert, so an alert sent a moment later
on another timeframe already counts this plan, and undoes it with
unmark_alerted when no chat received the message. Same hand-off as
log_panel_update (the private rowid from log_signal). Returns the
stamp this call wrote; None when nothing was written (never logged,
logging off, or the row already carried a stamp). NEVER raises."""
try:
hit = _logged_row(report)
return None if hit is None else hit[0].mark_alerted(hit[1])
except Exception:
logger.warning("signal_log: failed to mark the alerted row",
exc_info=True)
return None
def unmark_alerted(report: Any, stamp: Optional[str]) -> bool:
"""Clear the stamp mark_alerted(report) returned: the alert reached no
chat, so the row must not count as a plan the reader holds. A no-op
without a stamp; only that exact stamp is cleared. NEVER raises."""
if not stamp:
return False
try:
hit = _logged_row(report)
return False if hit is None else hit[0].unmark_alerted(hit[1], stamp)
except Exception:
logger.warning("signal_log: failed to clear the alerted stamp",
exc_info=True)
return False
# A PENDING outcome older than this is not shown as "open": the advisory
# model times a trade out after MAX_HOLD_BARS bars, so an older PENDING row
# is one the resolver never reached (bot outage, candle gap), not a trade
# still running. Calendar allowance: 7/5 for weekends plus three days.
def _open_setup_max_age_seconds(timeframe: Any) -> float:
bar = _TF_SECONDS.get(normalize_timeframe(timeframe), 3600)
return MAX_HOLD_BARS * bar * 1.4 + 3 * 86400
def open_alerted_setups(symbol: str, exclude_key: Optional[str] = None,
now: Optional[datetime] = None,
path: Optional[str] = None) -> List[Dict[str, Any]]:
"""ALERTED trade plans whose advisory outcome is still open, oldest
first: ``[{id, event_id, timeframe, direction, entry, stop_loss,
alerted_at, setup_key}]``, one entry per advisory event.
Read-only (a ``mode=ro`` connection of its own, so the dashboard
process, which never writes this database, can call it). "Open" means
the row's advisory event is still PENDING in the cleaned-mid outcome
model: next-open entry, stop first, close-only trailing. That is the
model's view of a plan the reader MAY have taken, not a broker
position. ``exclude_key`` drops the plan being alerted now from its
own count. NEVER raises: any failure (no database yet, an older schema
without ``alerted_at``) returns an empty list."""
try:
db_path = path or app_config.signal_log_path()
if not os.path.exists(db_path):
return []
now = now or datetime.now(timezone.utc)
con = sqlite3.connect(f"file:{db_path}?mode=ro", uri=True, timeout=2.0)
try:
rows = con.execute(
"SELECT a.id, COALESCE(a.advisory_event_id, a.id), a.timeframe,"
" a.setup_json, a.alerted_at, a.setup_key"
" FROM signals a JOIN signals e"
" ON e.id = COALESCE(a.advisory_event_id, a.id)"
" WHERE a.symbol=? AND a.alerted_at IS NOT NULL"
" AND e.advisory_outcome_status='PENDING'"
" ORDER BY a.id", (symbol,)).fetchall()
finally:
con.close()
out: List[Dict[str, Any]] = []
seen = set()
for rowid, event_id, timeframe, setup_json, alerted_at, key in rows:
if event_id in seen or (exclude_key and key == exclude_key):
continue
try:
age = (now - datetime.fromisoformat(alerted_at)).total_seconds()
except (TypeError, ValueError):
continue
if age > _open_setup_max_age_seconds(timeframe):
continue
setup = json.loads(setup_json) if setup_json else {}
direction = setup.get("direction")
if direction not in ("BUY", "SELL"):
continue
seen.add(event_id)
out.append({
"id": int(rowid), "event_id": int(event_id),
"timeframe": normalize_timeframe(timeframe),
"direction": direction,
"entry": _finite(setup.get("entry")),
"stop_loss": _finite(setup.get("stop_loss")),
"alerted_at": alerted_at, "setup_key": key,
})
return out
except Exception:
logger.debug("signal_log: open alerted setups unavailable",
exc_info=True)
return []
def last_alerted_bars(path: Optional[str] = None
) -> Dict[Tuple[str, str], str]:
"""``{(symbol, timeframe): bar label}`` of the newest ALERTED row per
symbol and timeframe (newest = highest id). The bot seeds its 5-bar
entry cooldown from this at start-up, so a restart does not forget the
last alert. Read-only (``mode=ro``); NEVER raises: no database, an
older schema or a locked file returns ``{}``."""
try:
db_path = path or app_config.signal_log_path()
if not os.path.exists(db_path):
return {}
con = sqlite3.connect(f"file:{db_path}?mode=ro", uri=True, timeout=2.0)
try:
rows = con.execute(
"SELECT s.symbol, s.timeframe, s.bar_ts FROM signals s"
" JOIN (SELECT MAX(id) AS id FROM signals"
" WHERE alerted_at IS NOT NULL"
" GROUP BY symbol, timeframe) m ON m.id = s.id"
).fetchall()
finally:
con.close()
return {(str(symbol), normalize_timeframe(timeframe)): str(bar)
for symbol, timeframe, bar in rows if bar}
except Exception:
logger.debug("signal_log: last alerted bars unavailable",
exc_info=True)
return {}
def reset_loggers_for_tests() -> None:
"""Close and drop all cached loggers (test isolation)."""
with _loggers_lock: