Daily-trend veto on M15 + lot-size / already-open lines (campaign 2026-09-30), and the 2026-10-01 whole-codebase bug review
Two bodies of work that touch the same files, so one commit. Direction and setup-size campaign (2026-09-30 → 10-01, owner's RISK-FIRST rule, written before the runs): - SIGNAL: M15 daily-trend veto (factory override 7, xau_strategy.daily_trend_veto "both"). A finished setup against a clear daily trend (closed D1 bars, Sunday stubs dropped; close vs its 50-bar EMA AND vs the close 60 kept days earlier) is removed as the last step of the core. One definition (signal_aggregator.daily_trend_series) for live, the bar replay and the twin's host side; D1 is a required frame of the M15 scan; mirrored in cuda_signal_kernels.py K2. W3 824 / +205.12R / PF 1.73 / max DD 12.4R vs 1105 / +195.79R / 1.49 / 20.9R; 2019 → 2026 1473 / +223.33R / 1.42 / 41.5R vs 2091 / +192.53R / 1.24 / 64.8R; unseen holdout −4.62R vs −6.63R. - FAILED, not shipped: the veto on M5 (W3 drawdown 10.1R → 15.2R) and on H1 (−9.7R), buy-only and sell-only on M15, the wide-stop cap (xau_strategy.wide_stop_cap_atr stays 0.0, a research knob). - New M15 pins: W2 dfc647ee…, OOS e87b6de1…, W3 ad3b0dbd…. M5 / H1 / H4 pins unchanged. --set xau_strategy.daily_trend_veto=off reproduces the previous M15 pins byte for byte. Twin and reference script MD5-equal on every window; recorder and feature-export parity re-proved; base-rate table rebuilt. - DISPLAY (books byte-identical): exposure.py — the lot size for trade_size.risk_per_trade (new config.json block, off by default, private to the owner) and the count of alerted same-direction plans still open (signals.alerted_at, read-only open_alerted_setups); the daily trend in words on Telegram, the CLI and the dashboard; signals.daily_trend logged. - providers/oanda: a date window with no bars is an empty page, not an error (a D1 pad starting on a Saturday could not be fetched). - Reports only, nothing shipped: seven context screens (no state nominated) and eight outside data sources (none adds a forecast to gold's own daily trend). - Independent verification (report section 5.13): every figure reproduced on both scripts, scoring identical to the previous commit on all four timeframes, no look-ahead. Caveats recorded there: this is a risk filter, not a more accurate direction call; nine tenths of the 2019 → 2026 gain is 2022; the neighbour clause is thin (a simple 100-day average misses the R line by 1.2R); "never adds a trade" is an observation, not a guarantee. Bug review of the whole codebase (2026-10-01). Signal path clean. 17 defects elsewhere, plus 4 found by an independent review of the fixes; all fixed, each with a test; no trade book changed: - telegram_bot: one _deliver_alert for the candle scan and the level break. The row is stamped alerted_at BEFORE the send, on a one-worker alert-db pool, and un-stamped when no chat received it or the task is cancelled mid-stamp, so two alerts at one candle close count each other and no SQLite write runs on the event loop. The 5-bar cooldown counts bars of the scan's own frame (== gpu_backtest._apply_cooldown; clock time released the 22:00 UTC bar after the settlement pause) and is restored from signals.db at start-up. The lot-size line goes to the owner's own chat only (a group the owner ran /autoscan in used to receive it). /status gains a news-model line and escapes failure details. - signal_log: mark_alerted returns the stamp it wrote; unmark_alerted; last_alerted_bars (read-only). - advisory_outcome: candle continuity is judged bar by bar inside the walk; a hole after a trade's exit no longer voids it. The stored-candle replay resolves 108 of 111 logged setups instead of 99, all equal to the live resolver's R. - providers/oanda: an empty date window is accepted only when it is 5 days or shorter and no malformed candle was skipped; a longer one is retried, then refused, instead of being cached as history with a hole. - news_engine: blank RSS titles skipped; one unreadable headline no longer costs the local model its whole batch (checked on the real bundle); model health recorded; no DeepSeek audit call in a process that does not archive; event-direction cache keyed per release; peek_week_events never fetches. - llm_client: chat_json never raises and never logs the API key (requests quotes the Authorization value in InvalidHeader; error bodies are redacted). - signal_aggregator: the scan's news block is guarded; the replay path (use_news=False) assigns exactly the values it did before. - agent_panel: PANEL_PROMPT_VERSION v2.2 — same prompt text, corrected input (event timing over the cached calendar week, headlines newest first). scripts/panel_replay.py replays the logged payload per persona; scripts/label_scan_outcomes.py counts with an ISO cutoff. - test_all.py: no ALTER TABLE … DROP COLUMN (the live box has SQLite 3.34); tests that write Parquet print a notice and return without pyarrow. Verification after the last code change: suite 632 passed, 0 failed (also with both Parquet engines blocked and FB2_BACKEND=cpu, the live box's view); the five reference MD5s reproduce on both backtesters, 0 cache fallbacks; Module 37 green; static Python 3.9 / SQLite 3.34 check clean on every touched file. Evidence: backtest/research/daily_trend_campaign_2026-09-30.md, context_screens_2026-10.md, new_inputs_2026-10.md; CLAUDE.md sections 2, 4, 6, 8, 9, 10 and 11; pair_signals/xauusd.py docstring (M15 override 7). Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
This commit is contained in:
Harwinder Sekhon
+210
-16
@@ -14,6 +14,7 @@ from datetime import datetime, timezone
|
||||
from typing import Dict, List, Optional
|
||||
from zoneinfo import ZoneInfo
|
||||
|
||||
import numpy as np
|
||||
import pandas as pd
|
||||
|
||||
_NY_TZ = ZoneInfo("America/New_York")
|
||||
@@ -155,6 +156,65 @@ def _transitioning_boost(abs_score: float, confidence: float, mtf: bool, mode: s
|
||||
return None
|
||||
|
||||
|
||||
# ── Daily trend (campaign 2026-09-30) ────────────────────────────────
|
||||
# One definition for the live scan, the bar replay and the CUDA twin's host
|
||||
# side (backtest/cudaBasedBacktesting.trend_flags calls daily_trend_series on
|
||||
# the whole D1 frame and indexes it). Every step is causal (the Sunday
|
||||
# filter, the EMA recursion, the shift), so the value on a prefix slice is
|
||||
# bit-identical to the value at the same bar of the full series.
|
||||
DAILY_TREND_MODES = ("off", "both", "buy", "sell")
|
||||
DAILY_TREND_EMA_SPAN = 50 # validated literals (do not edit): the
|
||||
DAILY_TREND_LOOKBACK = 60 # campaign tested exactly this definition
|
||||
_DAILY_TREND_PREFIX = "DAILY TREND FILTER:"
|
||||
|
||||
|
||||
def daily_trend_mode_of(xau_cfg) -> str:
|
||||
"""The validated `xau_strategy.daily_trend_veto`; an unknown value is a hard error."""
|
||||
mode = xau_cfg.daily_trend_veto
|
||||
if mode not in DAILY_TREND_MODES:
|
||||
raise ValueError(
|
||||
f"xau_strategy.daily_trend_veto={mode!r}; expected one of {DAILY_TREND_MODES}"
|
||||
)
|
||||
return mode
|
||||
|
||||
|
||||
def daily_trend_needed(config: PlatformConfig) -> bool:
|
||||
"""True when the scan needs the D1 frame: the veto is on for a gold config."""
|
||||
return (daily_trend_mode_of(config.xau_strategy) != "off"
|
||||
and config.xau_strategy.enabled
|
||||
and config.display_name == "XAU/USD")
|
||||
|
||||
|
||||
def daily_trend_series(d1: Optional[pd.DataFrame]) -> pd.Series:
|
||||
"""Daily trend per CLOSED D1 bar: +1 when the close is above its 50-bar
|
||||
EMA and above the close 60 bars earlier, -1 when below both, else 0.
|
||||
|
||||
`d1` must hold closed bars only (the provider drops the forming candle;
|
||||
the replay slices as-of). Bars labelled Sunday are dropped first: the
|
||||
provider's UTC-aligned daily candles include a 1-2 hour stub for the
|
||||
Sunday reopening, which is not a trading day. The first 60 kept bars
|
||||
read 0 (no 60-bar change yet)."""
|
||||
if d1 is None or len(d1) == 0:
|
||||
return pd.Series([], dtype="int8")
|
||||
kept = d1[d1.index.weekday != 6]
|
||||
close = kept["Close"].astype("float64")
|
||||
ema = close.ewm(span=DAILY_TREND_EMA_SPAN, adjust=False).mean()
|
||||
change = close / close.shift(DAILY_TREND_LOOKBACK) - 1.0
|
||||
up = (close > ema) & (change > 0)
|
||||
down = (close < ema) & (change < 0)
|
||||
state = np.where(up, 1, np.where(down, -1, 0)).astype("int8")
|
||||
return pd.Series(state, index=kept.index)
|
||||
|
||||
|
||||
def daily_trend(d1: Optional[pd.DataFrame]) -> Optional[int]:
|
||||
"""The daily trend at the last closed D1 bar of `d1` (+1 / 0 / -1), or
|
||||
None when the frame holds no usable bar."""
|
||||
series = daily_trend_series(d1)
|
||||
if len(series) == 0:
|
||||
return None
|
||||
return int(series.iloc[-1])
|
||||
|
||||
|
||||
def _is_forex_weekend(now_utc: datetime) -> bool:
|
||||
"""Gold/FX market closed for the weekend: Friday 17:00 → Sunday 18:00
|
||||
NEW YORK time (= Fri 21:00 UTC in US summer, 22:00 in winter —
|
||||
@@ -284,6 +344,17 @@ class SignalReport:
|
||||
penalty_tags: Dict[str, float] = field(default_factory=dict)
|
||||
penalty_total: float = 0.0
|
||||
momentum_fading: bool = False
|
||||
# Daily trend at the last closed D1 bar (+1 up / 0 unclear / -1 down);
|
||||
# None = not computed. Set by the core when the daily-trend veto is on
|
||||
# (_apply_daily_trend_veto); where the veto is off the LIVE wrapper
|
||||
# stamps it for the alert text and the signal log only
|
||||
# (_stamp_daily_trend_for_display) — no gate reads it there.
|
||||
daily_trend: Optional[int] = None
|
||||
# Lot-size and open-exposure lines (advisory/DISPLAY-ONLY). Stamped
|
||||
# post-hoc by surfaces via exposure.attach_exposure — never populated
|
||||
# inside analyze_from_dataframes and never read by any gate.
|
||||
size_line: str = ""
|
||||
open_line: str = ""
|
||||
|
||||
|
||||
class SignalAggregator:
|
||||
@@ -292,6 +363,11 @@ class SignalAggregator:
|
||||
def __init__(self, config: PlatformConfig):
|
||||
self.config = config
|
||||
grading_mode_of(config.signals) # fail at start, not at the first bar
|
||||
daily_trend_mode_of(config.xau_strategy)
|
||||
if not config.xau_strategy.wide_stop_cap_atr >= 0:
|
||||
raise ValueError(
|
||||
f"xau_strategy.wide_stop_cap_atr={config.xau_strategy.wide_stop_cap_atr!r}; "
|
||||
f"expected a number >= 0 (0 = off)")
|
||||
self.data_provider = make_data_provider(config)
|
||||
self.tech_engine = TechnicalEngine(config.technical)
|
||||
self._news_engine: Optional[NewsEngine] = None # lazy
|
||||
@@ -362,6 +438,12 @@ class SignalAggregator:
|
||||
bt = self.config.backtest
|
||||
if bt.regime_filter and bt.regime_filter_timeframe not in timeframes:
|
||||
timeframes.append(bt.regime_filter_timeframe)
|
||||
# The daily-trend veto reads the D1 frame inside the core, so it is
|
||||
# a REQUIRED frame when the veto is on (a failed fetch fails the
|
||||
# scan below, as for any other frame). The backtesters append it
|
||||
# under the same guard (rule 4).
|
||||
if daily_trend_needed(self.config) and Timeframe.D1 not in timeframes:
|
||||
timeframes.append(Timeframe.D1)
|
||||
|
||||
dfs: Dict[Timeframe, pd.DataFrame] = {}
|
||||
stale_tfs = []
|
||||
@@ -465,9 +547,34 @@ class SignalAggregator:
|
||||
f"{symbol}: live session filter errored — filter SKIPPED this scan"
|
||||
)
|
||||
|
||||
self._stamp_daily_trend_for_display(report, dfs)
|
||||
self._log_signal(report, log_source, dfs)
|
||||
return report
|
||||
|
||||
def _stamp_daily_trend_for_display(
|
||||
self, report: SignalReport, dfs: Dict[Timeframe, pd.DataFrame]
|
||||
) -> None:
|
||||
"""Daily trend for the alert text and the signal log on the
|
||||
timeframes where the daily-trend veto is OFF (the core stamps it
|
||||
where the veto is on). Display and record-keeping only: the frame
|
||||
is never added to `dfs`, nothing reads the value back, and a failed
|
||||
fetch leaves it None instead of failing the scan. The scan's own D1
|
||||
frame is used when it has one (H4 confirms on D1); otherwise the
|
||||
provider's D1 frame, cached until the next daily close and
|
||||
refreshed by the M15 scan, which needs it as a required frame.
|
||||
Live wrapper only — the backtesters never come through here."""
|
||||
if report.daily_trend is not None or report.display_name != "XAU/USD":
|
||||
return
|
||||
if report.primary_analysis is None:
|
||||
return
|
||||
try:
|
||||
d1 = dfs.get(Timeframe.D1)
|
||||
if d1 is None:
|
||||
d1 = self.data_provider.fetch_ohlcv(report.symbol, Timeframe.D1)
|
||||
report.daily_trend = daily_trend(d1)
|
||||
except Exception:
|
||||
logger.debug("daily trend (display) unavailable", exc_info=True)
|
||||
|
||||
def _log_signal(
|
||||
self, report: SignalReport, source: Optional[str],
|
||||
dataframes: Optional[Dict[Timeframe, pd.DataFrame]] = None,
|
||||
@@ -541,6 +648,13 @@ class SignalAggregator:
|
||||
bt.session_end_hour],
|
||||
"regime_filter": bool(bt.regime_filter),
|
||||
}
|
||||
# 2026-09-30 knobs enter the hash only when they are in force (the
|
||||
# grading_mode precedent in engine_config_fingerprint): a config
|
||||
# that leaves them off keeps the fingerprint its live rows carry.
|
||||
if xs.daily_trend_veto != "off":
|
||||
payload["daily_trend_veto"] = xs.daily_trend_veto
|
||||
if xs.wide_stop_cap_atr > 0:
|
||||
payload["wide_stop_cap_atr"] = xs.wide_stop_cap_atr
|
||||
return hashlib.sha1(
|
||||
json.dumps(payload, sort_keys=True, default=str).encode("utf-8")
|
||||
).hexdigest()[:12]
|
||||
@@ -1032,6 +1146,14 @@ class SignalAggregator:
|
||||
if height < min_tp:
|
||||
return
|
||||
|
||||
# Wide-stop cap AFTER the gates (xau_strategy.wide_stop_cap_atr,
|
||||
# 0 = off), same contract as the swing generator's.
|
||||
cap_dist = self._wide_stop_cap(report, atr, min_sl_price)
|
||||
if cap_dist is not None and sl_dist > cap_dist:
|
||||
sl_dist = cap_dist
|
||||
sl = entry - cap_dist if is_buy else entry + cap_dist
|
||||
rr = height / sl_dist
|
||||
|
||||
# Printed-target floor AFTER the gates (2026-09-26, see
|
||||
# config.SignalConfig.tp_print_floor_rr): the measured move stays
|
||||
# the gated target; the printed guide is at least the floor.
|
||||
@@ -1215,23 +1337,26 @@ class SignalAggregator:
|
||||
report.mtf_agreement = self._check_mtf_agreement(report)
|
||||
|
||||
# ── 4. Fundamental context (optional) ──────────────────────
|
||||
# News is display and warning only (it never gates), so a failure
|
||||
# anywhere in it must not cost the scan: the report then carries
|
||||
# no news at all, the values the replay path (use_news=False)
|
||||
# always has. All five fields are set together or not at all.
|
||||
news_fields = ("clear", "", 0.0, "", "")
|
||||
if use_news:
|
||||
news = self._get_news_engine()
|
||||
if context is None:
|
||||
context = news.get_fundamental_context()
|
||||
event_risk, event_desc = news.get_event_risk_level(symbol, context)
|
||||
report.event_risk = event_risk
|
||||
report.event_description = event_desc
|
||||
sentiment, sent_desc = news.get_pair_sentiment(symbol, context)
|
||||
report.news_sentiment = sentiment
|
||||
report.news_description = sent_desc
|
||||
report.news_summary = context.llm_summary
|
||||
else:
|
||||
report.event_risk = "clear"
|
||||
report.event_description = ""
|
||||
report.news_sentiment = 0.0
|
||||
report.news_description = ""
|
||||
report.news_summary = ""
|
||||
try:
|
||||
news = self._get_news_engine()
|
||||
if context is None:
|
||||
context = news.get_fundamental_context()
|
||||
event_risk, event_desc = news.get_event_risk_level(symbol, context)
|
||||
sentiment, sent_desc = news.get_pair_sentiment(symbol, context)
|
||||
news_fields = (event_risk, event_desc, sentiment, sent_desc,
|
||||
context.llm_summary)
|
||||
except Exception:
|
||||
logger.warning(
|
||||
f"News context failed for {symbol} — the scan continues "
|
||||
f"without news", exc_info=True)
|
||||
(report.event_risk, report.event_description, report.news_sentiment,
|
||||
report.news_description, report.news_summary) = news_fields
|
||||
|
||||
# ── 5. Spread estimate from primary data (avoids extra API call) ─
|
||||
try:
|
||||
@@ -1471,6 +1596,12 @@ class SignalAggregator:
|
||||
# cost-floor veto no longer describes it.
|
||||
self._drop_stale_spread_gate_vetoes(report)
|
||||
|
||||
# ── 12. Daily-trend veto (opt-in via per-TF factories) ───────
|
||||
# Last on purpose: a FINAL veto on whatever archetype shipped the
|
||||
# setup (core, XAU_PB or XAU_CP), which is exactly what the
|
||||
# campaign replayed. Mirrored in cuda_signal_kernels.py K2.
|
||||
self._apply_daily_trend_veto(report, dfs)
|
||||
|
||||
return report
|
||||
|
||||
# ── Internals ──────────────────────────────────────────────────
|
||||
@@ -1543,6 +1674,57 @@ class SignalAggregator:
|
||||
report.warnings[:] = kept
|
||||
report.cleaned_warnings.extend(stale)
|
||||
|
||||
def _apply_daily_trend_veto(
|
||||
self, report: SignalReport, dfs: Dict[Timeframe, pd.DataFrame]
|
||||
) -> None:
|
||||
"""Step 12: remove a finished setup that goes against a clear daily
|
||||
trend (xau_strategy.daily_trend_veto, off by default).
|
||||
|
||||
Reads the D1 frame from `dfs` (closed bars only: the live wrapper
|
||||
fetches it as a required frame, the replay slices it as-of), stamps
|
||||
report.daily_trend, and follows the gate-chain convention when it
|
||||
fires: grade D, no setup, a warning that names the gate. The final
|
||||
signal and the action text are left as they are."""
|
||||
xcfg = self.config.xau_strategy
|
||||
mode = daily_trend_mode_of(xcfg)
|
||||
if mode == "off" or not xcfg.enabled or report.display_name != "XAU/USD":
|
||||
return
|
||||
state = daily_trend(dfs.get(Timeframe.D1))
|
||||
report.daily_trend = state
|
||||
if state is None:
|
||||
# Mixed case like the regime filter's twin: never a gate prefix.
|
||||
report.warnings.append(
|
||||
"Daily trend filter requested but 1d data unavailable")
|
||||
return
|
||||
setup = report.trade_setup
|
||||
if setup is None or state == 0:
|
||||
return
|
||||
is_buy = setup.direction == "BUY"
|
||||
against = ((is_buy and state < 0 and mode in ("both", "buy"))
|
||||
or (not is_buy and state > 0 and mode in ("both", "sell")))
|
||||
if not against:
|
||||
return
|
||||
report.warnings.append(
|
||||
f"{_DAILY_TREND_PREFIX} {setup.direction} against a "
|
||||
f"{'falling' if is_buy else 'rising'} daily trend — no trade"
|
||||
)
|
||||
report.signal_quality = "D"
|
||||
report.trade_setup = None
|
||||
|
||||
def _wide_stop_cap(self, report: SignalReport, atr: float,
|
||||
stop_floor: float) -> Optional[float]:
|
||||
"""The largest stop distance xau_strategy.wide_stop_cap_atr allows
|
||||
for a swing or chart-pattern setup: max(K x ATR, the setup's own
|
||||
stop floor); None when the cap is off (K = 0) or the pair is not
|
||||
gold. Applied by the generators AFTER their min-RR and spread
|
||||
gates, so the gates keep judging the uncapped geometry and the
|
||||
set of trades cannot change (the printed-target floor precedent)."""
|
||||
xcfg = self.config.xau_strategy
|
||||
k = xcfg.wide_stop_cap_atr
|
||||
if not k > 0 or not xcfg.enabled or report.display_name != "XAU/USD":
|
||||
return None
|
||||
return max(k * atr, stop_floor)
|
||||
|
||||
def _printed_target_floor(self) -> float:
|
||||
"""The floor every generator applies to the PRINTED target 1, as a
|
||||
fraction of the stop: max(tp_floor_rr, tp_print_floor_rr). Applied
|
||||
@@ -2024,6 +2206,18 @@ class SignalAggregator:
|
||||
)
|
||||
return None
|
||||
|
||||
# Wide-stop cap AFTER the gates (xau_strategy.wide_stop_cap_atr,
|
||||
# 0 = off): the gates above judged the swing-anchored stop, so the
|
||||
# same trades ship; only the stop (the risk unit) is pulled in,
|
||||
# never inside the generator's own floor. Mirrored in
|
||||
# cuda_signal_kernels.py K2.
|
||||
cap_dist = self._wide_stop_cap(
|
||||
report, atr, max(atr * sl_atr_floor, min_sl_price))
|
||||
if cap_dist is not None and sl_distance > cap_dist:
|
||||
sl_distance = cap_dist
|
||||
sl = entry - cap_dist if is_buy else entry + cap_dist
|
||||
rr = tp1_distance / sl_distance
|
||||
|
||||
# Printed-target floor AFTER the gates (2026-09-26, see
|
||||
# config.SignalConfig.tp_print_floor_rr): the gates above judged
|
||||
# the swing/ATR target, so the trade is unchanged; only the guide
|
||||
|
||||
Reference in New Issue
Block a user