Daily-trend veto on M15 + lot-size / already-open lines (campaign 2026-09-30), and the 2026-10-01 whole-codebase bug review

Two bodies of work that touch the same files, so one commit. Direction and setup-size campaign (2026-09-30 → 10-01, owner's RISK-FIRST rule, written before the runs): - SIGNAL: M15 daily-trend veto (factory override 7, xau_strategy.daily_trend_veto "both"). A finished setup against a clear daily trend (closed D1 bars, Sunday stubs dropped; close vs its 50-bar EMA AND vs the close 60 kept days earlier) is removed as the last step of the core. One definition (signal_aggregator.daily_trend_series) for live, the bar replay and the twin's host side; D1 is a required frame of the M15 scan; mirrored in cuda_signal_kernels.py K2. W3 824 / +205.12R / PF 1.73 / max DD 12.4R vs 1105 / +195.79R / 1.49 / 20.9R; 2019 → 2026 1473 / +223.33R / 1.42 / 41.5R vs 2091 / +192.53R / 1.24 / 64.8R; unseen holdout −4.62R vs −6.63R. - FAILED, not shipped: the veto on M5 (W3 drawdown 10.1R → 15.2R) and on H1 (−9.7R), buy-only and sell-only on M15, the wide-stop cap (xau_strategy.wide_stop_cap_atr stays 0.0, a research knob). - New M15 pins: W2 dfc647ee…, OOS e87b6de1…, W3 ad3b0dbd…. M5 / H1 / H4 pins unchanged. --set xau_strategy.daily_trend_veto=off reproduces the previous M15 pins byte for byte. Twin and reference script MD5-equal on every window; recorder and feature-export parity re-proved; base-rate table rebuilt. - DISPLAY (books byte-identical): exposure.py — the lot size for trade_size.risk_per_trade (new config.json block, off by default, private to the owner) and the count of alerted same-direction plans still open (signals.alerted_at, read-only open_alerted_setups); the daily trend in words on Telegram, the CLI and the dashboard; signals.daily_trend logged. - providers/oanda: a date window with no bars is an empty page, not an error (a D1 pad starting on a Saturday could not be fetched). - Reports only, nothing shipped: seven context screens (no state nominated) and eight outside data sources (none adds a forecast to gold's own daily trend). - Independent verification (report section 5.13): every figure reproduced on both scripts, scoring identical to the previous commit on all four timeframes, no look-ahead. Caveats recorded there: this is a risk filter, not a more accurate direction call; nine tenths of the 2019 → 2026 gain is 2022; the neighbour clause is thin (a simple 100-day average misses the R line by 1.2R); "never adds a trade" is an observation, not a guarantee. Bug review of the whole codebase (2026-10-01). Signal path clean. 17 defects elsewhere, plus 4 found by an independent review of the fixes; all fixed, each with a test; no trade book changed: - telegram_bot: one _deliver_alert for the candle scan and the level break. The row is stamped alerted_at BEFORE the send, on a one-worker alert-db pool, and un-stamped when no chat received it or the task is cancelled mid-stamp, so two alerts at one candle close count each other and no SQLite write runs on the event loop. The 5-bar cooldown counts bars of the scan's own frame (== gpu_backtest._apply_cooldown; clock time released the 22:00 UTC bar after the settlement pause) and is restored from signals.db at start-up. The lot-size line goes to the owner's own chat only (a group the owner ran /autoscan in used to receive it). /status gains a news-model line and escapes failure details. - signal_log: mark_alerted returns the stamp it wrote; unmark_alerted; last_alerted_bars (read-only). - advisory_outcome: candle continuity is judged bar by bar inside the walk; a hole after a trade's exit no longer voids it. The stored-candle replay resolves 108 of 111 logged setups instead of 99, all equal to the live resolver's R. - providers/oanda: an empty date window is accepted only when it is 5 days or shorter and no malformed candle was skipped; a longer one is retried, then refused, instead of being cached as history with a hole. - news_engine: blank RSS titles skipped; one unreadable headline no longer costs the local model its whole batch (checked on the real bundle); model health recorded; no DeepSeek audit call in a process that does not archive; event-direction cache keyed per release; peek_week_events never fetches. - llm_client: chat_json never raises and never logs the API key (requests quotes the Authorization value in InvalidHeader; error bodies are redacted). - signal_aggregator: the scan's news block is guarded; the replay path (use_news=False) assigns exactly the values it did before. - agent_panel: PANEL_PROMPT_VERSION v2.2 — same prompt text, corrected input (event timing over the cached calendar week, headlines newest first). scripts/panel_replay.py replays the logged payload per persona; scripts/label_scan_outcomes.py counts with an ISO cutoff. - test_all.py: no ALTER TABLE … DROP COLUMN (the live box has SQLite 3.34); tests that write Parquet print a notice and return without pyarrow. Verification after the last code change: suite 632 passed, 0 failed (also with both Parquet engines blocked and FB2_BACKEND=cpu, the live box's view); the five reference MD5s reproduce on both backtesters, 0 cache fallbacks; Module 37 green; static Python 3.9 / SQLite 3.34 check clean on every touched file. Evidence: backtest/research/daily_trend_campaign_2026-09-30.md, context_screens_2026-10.md, new_inputs_2026-10.md; CLAUDE.md sections 2, 4, 6, 8, 9, 10 and 11; pair_signals/xauusd.py docstring (M15 override 7).
2026-10-01 10:28:28 -04:00
parent 42025da0b0
commit 2cc5915b00
31 changed files with 6021 additions and 286 deletions
@@ -45,7 +45,9 @@ if str(PROJECT_ROOT) not in sys.path:
 from config import PAIR_INFO, PlatformConfig, Timeframe  # noqa: E402
 from providers import make_data_provider  # noqa: E402
 from providers.base import BaseDataProvider  # noqa: E402
-from signal_aggregator import SignalAggregator  # noqa: E402  (fingerprints + the two static cleanups only)
+from signal_aggregator import (  # noqa: E402  (fingerprints, the two static cleanups, the daily-trend definition)
+    SignalAggregator, daily_trend_needed, daily_trend_series,
+)
 import position_sizing  # noqa: E402
 from backtest import gpu_kernels as _gk  # noqa: E402
 from backtest import cuda_signal_kernels as ck  # noqa: E402
@@ -318,6 +320,12 @@ def format_warning(code: int, arg: float, ctx: Dict) -> str:
    if n == "rf_blocked":
        return (f"REGIME FILTER: {ctx['rf_tf']} ADX={arg:.1f} < "
                f"{ctx['regime_filter_min_adx']:.0f} — market choppy, skipping signal")
+    if n == "trend_veto":
+        is_buy = arg > 0
+        return (f"DAILY TREND FILTER: {'BUY' if is_buy else 'SELL'} against a "
+                f"{'falling' if is_buy else 'rising'} daily trend — no trade")
+    if n == "trend_unavailable":
+        return "Daily trend filter requested but 1d data unavailable"
    raise ValueError(f"unknown warning code {code}")


@@ -379,6 +387,34 @@ def host_flags(primary_index: pd.DatetimeIndex, platform_cfg: PlatformConfig,
    }


+def trend_flags(primary_index: pd.DatetimeIndex, d1: Optional[pd.DataFrame],
+                bt_cfg: BacktestConfig) -> np.ndarray:
+    """Per primary bar: the daily trend the live code reads on the D1 slice
+    _replay_span hands it (+1 / 0 / -1), or ck.DAILY_TREND_UNAVAILABLE when
+    that slice holds no usable bar.
+
+    The definition is signal_aggregator.daily_trend_series, evaluated once on
+    the whole frame (every step is causal, so the value at a bar equals the
+    value on any prefix ending there). The as-of rule is the replay's: a D1
+    bar is visible once label + 1 day <= the primary bar's close, compared in
+    int64 nanoseconds; the Sunday stubs daily_trend_series drops are skipped
+    by searching its own (kept-bar) index."""
+    n = len(primary_index)
+    out = np.full(n, ck.DAILY_TREND_UNAVAILABLE, dtype=np.int8)
+    if d1 is None or len(d1) == 0:
+        return out
+    series = daily_trend_series(d1)
+    if len(series) == 0:
+        return out
+    primary_ns = primary_index.as_unit("ns").asi8
+    cutoff = primary_ns + (BAR_SECONDS[bt_cfg.primary_tf] - BAR_SECONDS[Timeframe.D1]) * 1_000_000_000
+    labels_ns = series.index.as_unit("ns").asi8
+    k = np.searchsorted(labels_ns, cutoff, side="right")
+    seen = k > 0
+    out[seen] = series.to_numpy()[k[seen] - 1]
+    return out
+
+
 def replay_bounds(primary_df: pd.DataFrame, bt_cfg: BacktestConfig) -> Tuple[int, int]:
    bt_start_ts = pd.Timestamp(bt_cfg.start)
    bt_end_ts = pd.Timestamp(bt_cfg.end)
@@ -421,6 +457,8 @@ def refuse_unsupported(platform_cfg: PlatformConfig, bt_cfg: BacktestConfig) ->
        problems.append(f"technical.ema_mode={t.ema_mode!r}")
    if platform_cfg.signals.grading_mode not in ck.GRADING_MODE_CODES:
        problems.append(f"signals.grading_mode={platform_cfg.signals.grading_mode!r}")
+    if x.daily_trend_veto not in ck.DAILY_TREND_CODES:
+        problems.append(f"xau_strategy.daily_trend_veto={x.daily_trend_veto!r}")
    if problems:
        raise SystemExit("cudaBasedBacktesting.py cannot run this configuration: "
                         + "; ".join(problems) + ". Use backtest/gpu_backtest.py.")
@@ -456,6 +494,11 @@ def analyze_frames(bt_cfg: BacktestConfig, platform_cfg: PlatformConfig,
        rf_cut = cuts[rtf] if rtf != bt_cfg.primary_tf else cp.arange(1, primary_dev["n"] + 1, dtype=cp.int64)
        rf = (dev[rtf]["adx"], rf_cut)
    flags = host_flags(primary_df.index, platform_cfg, bt_cfg)
+    # The daily trend per primary bar (read by K2 only when the veto is on).
+    flags["daily_trend"] = (
+        trend_flags(primary_df.index, dfs_full.get(Timeframe.D1), bt_cfg)
+        if daily_trend_needed(platform_cfg)
+        else np.zeros(len(primary_df), dtype=np.int8))
    first_eligible_pos, last_pos = replay_bounds(primary_df, bt_cfg)
    logger.info(f"Replaying {bt_cfg.pair} {bt_cfg.primary_tf.value}: "
                f"bars {first_eligible_pos}–{last_pos} of {len(primary_df)}")
@@ -508,6 +551,9 @@ def run_backtest_cuda(bt_cfg: BacktestConfig, platform_cfg: PlatformConfig) -> T
    flt = platform_cfg.backtest
    if flt.regime_filter and flt.regime_filter_timeframe not in timeframes:
        timeframes.append(flt.regime_filter_timeframe)
+    # the D1 frame of the daily-trend veto, appended exactly like the reference script
+    if daily_trend_needed(platform_cfg) and Timeframe.D1 not in timeframes:
+        timeframes.append(Timeframe.D1)
    dfs_full: Dict[Timeframe, pd.DataFrame] = {}
    cache_names: Dict[str, str] = {}
    tf_starts = {}
@@ -615,27 +661,30 @@ def _warning_ctx(ctx, pos: int, r2i: Dict, r2f: Dict) -> Dict:


 _RF_CODES = {ck.WARN_CODES["rf_unavailable"], ck.WARN_CODES["rf_no_adx"], ck.WARN_CODES["rf_blocked"]}
+_TREND_CODES = {ck.WARN_CODES["trend_veto"], ck.WARN_CODES["trend_unavailable"]}


 def build_warnings(ctx, pos: int, r2i: Dict, r2f: Dict, codes, args) -> Tuple[List[str], List[str]]:
    """(warnings, cleaned_warnings) of one analysed bar, after the XAU_CP
    cleanup (when a pattern setup was accepted) and the stale spread-gate
-    step (when a setup existed at step 11), with the regime-filter lines
-    appended last like the replay does."""
+    step (when a setup existed at step 11), with the daily-trend line of
+    step 12 and then the regime-filter lines appended last, in the order
+    the live code and the replay write them."""
    wctx = _warning_ctx(ctx, pos, r2i, r2f)
    n = r2i["warn_n"]
    main: List[str] = []
+    trend: List[str] = []
    rf: List[str] = []
    for q in range(n):
        code = int(codes[q])
        text = format_warning(code, float(args[q]), wctx)
-        (rf if code in _RF_CODES else main).append(text)
+        (rf if code in _RF_CODES else trend if code in _TREND_CODES else main).append(text)
    ns = types.SimpleNamespace(warnings=main, cleaned_warnings=[],
                               trade_setup=object() if r2i["strategy_pre_rf"] != 0 else None)
    if r2i["cp_accepted"]:
        SignalAggregator._clean_accepted_cp_warnings(ns)
    SignalAggregator._drop_stale_spread_gate_vetoes(ns)
-    return list(ns.warnings) + rf, list(ns.cleaned_warnings)
+    return list(ns.warnings) + trend + rf, list(ns.cleaned_warnings)


 def build_reasons(ctx, r2i: Dict, r2f: Dict, inds: Dict[str, Dict]) -> Tuple[List[str], List[str]]: