audit: NY-anchored weekend calendar (all 3 copies) + FB2_EMA_LO crash-proofing

Owner decisions from the 2026-08-04 audit report:
- spread-in-exit-triggers: DECLINED (books stay comparable; ledger in
  CLAUDE.md records the measured ~4-6% optimism).
- weekend clock: FIXED. The market week is Fri 17:00 -> Sun 18:00 New
  York time, not Fri/Sun 22:00 UTC — during US DST the Oanda feed stops
  at 21:00 UTC (measured: 159/162 summer weekends). All three weekend
  implementations (providers/base, signal_aggregator, gpu_backtest
  calendar filter) now share the NY-anchored rule; a new test sweeps
  them hourly across both DST transitions and asserts agreement. No NTP
  needed — the host clock was fine, the constant was wrong.
  Ends the weekly false-"STALE DATA / feed lagging" hours (summer Fri
  21-22 UTC, winter Sun 22-23 UTC).
- FB2_EMA_LO (delegated): kept as a research knob but crash-proofed —
  an unparseable value degrades to the tuned default 0.5 with a warning
  instead of killing every scan. Documented in CLAUDE.md.

Book gates: M15 12mo pinned book byte-identical (+78.7111R, 397
trades). M5 12mo drops exactly ONE trade (556 -> 555, +97.56 ->
+98.64R): a phantom whose signal fired on the final summer-Friday bar
(decision 21:00 UTC = NY 17:00 market close) and was "filled" at
Sunday's reopen 49 hours later, -1.08R — an entry live trading can
never take. The backtest no longer contains it; live == backtest
parity improved. M15 unaffected because hour 21 is in its blocked set.

Suite: 341/341 green.

Claude-Session: https://claude.ai/code/session_01UnZkzeFcTu9agFg8BDh2tn
This commit is contained in:
Harwinder Sekhon
2026-08-04 20:21:34 -04:00
parent 5b3f47f3e1
commit 7dbcbc6c82
5 changed files with 129 additions and 28 deletions
+5 -5
View File
@@ -74,12 +74,12 @@ Signal currency everywhere is `SignalStrength` (−2..+2). DataFrames are UTC tz
8. Don't add config fields nothing reads; don't disable indicators in production paths (`disabled_indicators` is for experiments only). 8. Don't add config fields nothing reads; don't disable indicators in production paths (`disabled_indicators` is for experiments only).
9. EUR/USD was removed 2026-07-16 after a no-edge verdict (cost-to-volatility; last commit with EUR support: `7d3fd95`). Do not re-add without new data. BTC/ETH work lives on unmerged branches only. 9. EUR/USD was removed 2026-07-16 after a no-edge verdict (cost-to-volatility; last commit with EUR support: `7d3fd95`). Do not re-add without new data. BTC/ETH work lives on unmerged branches only.
## Known open items from the 2026-08-04 QA audit (owner decisions pending) ## 2026-08-04 QA audit — owner decisions ledger
- **Weekend constant vs Oanda reality**: `_is_forex_weekend` hardcodes Fri 22:00→Sun 22:00 UTC, but Oanda's week is Fri 17:00→Sun 18:00 New York time — during US DST the feed stops at 21:00 UTC (159/162 summer weekends measured) and winter reopen is Sun 23:00. Result: two false-"STALE DATA/feed lagging" hours weekly (summer Fri 21–22, winter Sun 22–23). Fix = NY-anchored calendar in all three copies; books provably unaffected (no bars exist in those hours) but run a book gate anyway. - **Weekend calendar: FIXED (owner-approved 2026-08-04).** All three `_is_forex_weekend` copies are NY-anchored (Fri 17:00 → Sun 18:00 ET; during US DST that is Fri 21:00 UTC — measured on 5yr of Oanda bars). Kills the weekly false-"STALE DATA/feed lagging" hours. Book impact, fully accounted: M15 byte-identical; M5 12mo dropped exactly ONE trade — a phantom whose signal fired on the final summer-Friday bar (close 21:00 UTC) and "filled" at Sunday's reopen 49 h later (−1.08R, live-impossible). M5 12mo reference at HEAD: 555 trades, +98.64R PF≈1.50 @ $0.65 (was 556/+97.56/1.48). A Module 21 test sweeps all three copies hourly across both DST transitions.
- **Spread in exit-trigger detection (rule 6)**: backtest stop triggers use raw mid High/Low; bid/ask-adjusted triggers would change 15/397 trades (−3.55R) at $0.30 and 30/397 (−4.24R) at $0.65 on the pinned M15 12mo book — a ~4–6% systematic optimism. Fixing invalidates all book comparability: conscious re-pin decision only. - **Spread in exit-trigger detection (rule 6): owner DECLINED 2026-08-04.** Backtest stop triggers stay on raw mid High/Low; the measured optimism (~4–6% of total R: −3.55R @ $0.30, −4.24R @ $0.65 on the M15 12mo book) is accepted and books stay comparable. Re-open only as a conscious re-pin campaign.
- **H4 seeding divergence**: live H4 fetches only 380 bars; the backtest sees unbounded history — ~2% of sampled H4 bars flip an indicator signal (composite Δ up to 12.3). M5/M15/H1 are clean at their real fetch depths. H4 is already "unproven for auto-execution". - **`FB2_EMA_LO`: HARDENED 2026-08-04 (delegated).** Research override for the EMA200-proximity band, default = tuned 0.5; an unparseable value now degrades to the default with a warning instead of crashing every scan. Documented here — it is a knob that exists.
- `FB2_EMA_LO` env var is an undocumented knob inside the scoring path (`signal_aggregator.py` EMA200-proximity penalty floor); an unparseable value crashes every analysis. Remove or config-ify only with rule-1 revalidation. - **H4 seeding divergence: parked.** Live H4 fetches only 380 bars vs the backtest's unbounded history — ~2% of sampled H4 bars flip an indicator signal (composite Δ up to 12.3). M5/M15/H1 clean at their real fetch depths. Fold into any future H4 campaign; H4 remains "unproven for auto-execution".
## Current state notes (2026-07-30) ## Current state notes (2026-07-30)
+13 -6
View File
@@ -40,10 +40,13 @@ from dataclasses import dataclass, field
from datetime import datetime, timedelta, timezone from datetime import datetime, timedelta, timezone
from pathlib import Path from pathlib import Path
from typing import Dict, List, Optional, Tuple from typing import Dict, List, Optional, Tuple
from zoneinfo import ZoneInfo
import numpy as np import numpy as np
import pandas as pd import pandas as pd
_NY_TZ = ZoneInfo("America/New_York")
# Ensure project root on sys.path so this script can be invoked from anywhere. # Ensure project root on sys.path so this script can be invoked from anywhere.
PROJECT_ROOT = Path(__file__).resolve().parent.parent PROJECT_ROOT = Path(__file__).resolve().parent.parent
if str(PROJECT_ROOT) not in sys.path: if str(PROJECT_ROOT) not in sys.path:
@@ -1450,18 +1453,22 @@ class BacktestConfig:
def _is_xau_blocked_by_calendar(ts: pd.Timestamp, cfg: PlatformConfig) -> Optional[str]: def _is_xau_blocked_by_calendar(ts: pd.Timestamp, cfg: PlatformConfig) -> Optional[str]:
"""Apply XAU weekend + allowed-weekday filter matching live `analyze_pair`. """Apply XAU weekend + allowed-weekday filter matching live `analyze_pair`.
Forex market closed: Friday 22:00 UTC -> Sunday 22:00 UTC (Sydney open). Market closed Friday 17:00 -> Sunday 18:00 NEW YORK time (= Fri
Must match `_is_forex_weekend` in signal_aggregator.py exactly. 21:00 UTC in US summer, 22:00 in winter — matches measured Oanda
feed behavior; 2026-08-04 audit I43). Must match `_is_forex_weekend`
in signal_aggregator.py exactly (rule 4; Module 21 agreement test).
Note the weekday gates below stay on UTC weekday — they mirror the
live gates, which evaluate UTC weekdays.
""" """
xau = cfg.xau_strategy xau = cfg.xau_strategy
if xau.weekend_filter: if xau.weekend_filter:
wd = ts.weekday() ny = ts.astimezone(_NY_TZ)
hour = ts.hour wd = ny.weekday()
if wd == 4 and hour >= 22: if wd == 4 and ny.hour >= 17:
return "WEEKEND_FILTER" return "WEEKEND_FILTER"
if wd == 5: if wd == 5:
return "WEEKEND_FILTER" return "WEEKEND_FILTER"
if wd == 6 and hour < 22: if wd == 6 and ny.hour < 18:
return "WEEKEND_FILTER" return "WEEKEND_FILTER"
if xau.allowed_weekdays and ts.weekday() not in xau.allowed_weekdays: if xau.allowed_weekdays and ts.weekday() not in xau.allowed_weekdays:
return "DAY_FILTER" return "DAY_FILTER"
+26 -8
View File
@@ -24,6 +24,7 @@ import time
from abc import ABC, abstractmethod from abc import ABC, abstractmethod
from datetime import datetime, timedelta from datetime import datetime, timedelta
from typing import AsyncIterator, Callable, Dict, Optional, Sequence, Tuple from typing import AsyncIterator, Callable, Dict, Optional, Sequence, Tuple
from zoneinfo import ZoneInfo
import pandas as pd import pandas as pd
@@ -31,19 +32,36 @@ from config import PAIR_INFO, PlatformConfig, Timeframe, XAUUSD
logger = logging.getLogger(__name__) logger = logging.getLogger(__name__)
# The gold/FX market week is defined in New York time (see
# _is_forex_weekend); a fixed UTC hour is wrong for half the year.
_NY_TZ = ZoneInfo("America/New_York")
def _is_forex_weekend(now=None) -> bool: def _is_forex_weekend(now=None) -> bool:
"""Forex market closed: Friday 22:00 UTC → Sunday 22:00 UTC.""" """Gold/FX market closed for the weekend.
The market week is anchored to NEW YORK time — Friday 17:00 ET →
Sunday 18:00 ET — not a fixed UTC hour. During US DST the feed
actually stops at 21:00 UTC (measured on 5 years of Oanda bars:
159/162 summer Fridays; winter close 22:00, winter reopen Sunday
23:00 UTC). The old hardcoded Fri 22:00→Sun 22:00 UTC rule produced
two false-"STALE DATA/feed lagging" hours every week (2026-08-04
audit I43, fixed per owner decision).
Kept as three deliberate copies (here, signal_aggregator.py,
gpu_backtest._is_xau_blocked_by_calendar) — a Module 21 test asserts
they agree hour-by-hour across DST transitions.
"""
if now is None: if now is None:
now = pd.Timestamp.now(tz="UTC") now = pd.Timestamp.now(tz="UTC")
wd = now.weekday() ny = now.astimezone(_NY_TZ)
hour = now.hour wd = ny.weekday()
if wd == 4 and hour >= 22: if wd == 4: # Friday
return True return ny.hour >= 17
if wd == 5: if wd == 5: # Saturday
return True
if wd == 6 and hour < 22:
return True return True
if wd == 6: # Sunday
return ny.hour < 18
return False return False
+25 -8
View File
@@ -10,9 +10,12 @@ import os
from dataclasses import dataclass, field from dataclasses import dataclass, field
from datetime import datetime, timezone from datetime import datetime, timezone
from typing import Dict, List, Optional from typing import Dict, List, Optional
from zoneinfo import ZoneInfo
import pandas as pd import pandas as pd
_NY_TZ = ZoneInfo("America/New_York")
from chart_patterns import detect_chart_pattern from chart_patterns import detect_chart_pattern
from config import PlatformConfig, SignalStrength, Timeframe from config import PlatformConfig, SignalStrength, Timeframe
from providers import BaseDataProvider, make_data_provider from providers import BaseDataProvider, make_data_provider
@@ -23,15 +26,19 @@ logger = logging.getLogger(__name__)
def _is_forex_weekend(now_utc: datetime) -> bool: def _is_forex_weekend(now_utc: datetime) -> bool:
"""Forex market closed: Friday 22:00 UTC → Sunday 22:00 UTC (Sydney open).""" """Gold/FX market closed for the weekend: Friday 17:00 → Sunday 18:00
wd = now_utc.weekday() NEW YORK time (= Fri 21:00 UTC in US summer, 22:00 in winter —
hour = now_utc.hour matches measured Oanda feed behavior; 2026-08-04 audit I43).
if wd == 4 and hour >= 22: # Friday after 22:00 Deliberate copy of providers.base._is_forex_weekend — a Module 21
return True test asserts the copies agree across DST transitions."""
if wd == 5: # All Saturday ny = now_utc.astimezone(_NY_TZ)
return True wd = ny.weekday()
if wd == 6 and hour < 22: # Sunday before 22:00 if wd == 4: # Friday
return ny.hour >= 17
if wd == 5: # Saturday
return True return True
if wd == 6: # Sunday
return ny.hour < 18
return False return False
@@ -1253,7 +1260,17 @@ class SignalAggregator:
ema200 = primary.key_levels.get("ema_trend") ema200 = primary.key_levels.get("ema_trend")
if ema200 and ema200 > 0 and report.current_price > 0: if ema200 and ema200 > 0 and report.current_price > 0:
ema_dist_pct = abs(report.current_price - ema200) / ema200 * 100.0 ema_dist_pct = abs(report.current_price - ema200) / ema200 * 100.0
# FB2_EMA_LO: research override for the EMA200-proximity
# penalty band; the tuned default is 0.5. An unparseable
# value must degrade to the default, never crash the scan
# (2026-08-04 audit; hardened per owner delegation).
try:
_lo = float(os.environ.get("FB2_EMA_LO", "0.5")) _lo = float(os.environ.get("FB2_EMA_LO", "0.5"))
except (TypeError, ValueError):
logger.warning(
"FB2_EMA_LO=%r is not a number — using tuned "
"default 0.5", os.environ.get("FB2_EMA_LO"))
_lo = 0.5
if ema_dist_pct < _lo: if ema_dist_pct < _lo:
report.warnings.append(f"NEAR EMA200 ({ema_dist_pct:.2f}%) — chop risk") report.warnings.append(f"NEAR EMA200 ({ema_dist_pct:.2f}%) — chop risk")
penalties.append(("ema200_near", 5)) penalties.append(("ema200_near", 5))
+59
View File
@@ -6397,6 +6397,65 @@ def _():
assert [p for p, _ in _gb._apply_cooldown(cands, 1)] == [10, 12, 15, 21, 24] assert [p for p, _ in _gb._apply_cooldown(cands, 1)] == [10, 12, 15, 21, 24]
@test("weekend calendar is NY-anchored: summer close 21:00 UTC, winter reopen 23:00")
def _():
# 2026-08-04 audit I43, fixed per owner decision: the market week is
# Fri 17:00 -> Sun 18:00 NEW YORK time (measured on 5yr of Oanda
# bars), not a fixed UTC hour.
import providers.base as pbase
T = lambda s: pd.Timestamp(s, tz="UTC")
# US DST 2026: Mar 8 – Nov 1.
assert pbase._is_forex_weekend(T("2026-07-31 21:30")) is True, \
"summer Friday 21:30 UTC = 17:30 ET → market CLOSED (old code said open)"
assert pbase._is_forex_weekend(T("2026-07-31 20:30")) is False
assert pbase._is_forex_weekend(T("2026-01-02 21:30")) is False, \
"winter Friday 21:30 UTC = 16:30 ET → still open"
assert pbase._is_forex_weekend(T("2026-01-02 22:30")) is True
assert pbase._is_forex_weekend(T("2026-01-04 22:30")) is True, \
"winter Sunday 22:30 UTC = 17:30 ET → still closed (old code said open)"
assert pbase._is_forex_weekend(T("2026-01-04 23:30")) is False
assert pbase._is_forex_weekend(T("2026-08-02 21:30")) is True # summer Sun 17:30 ET
assert pbase._is_forex_weekend(T("2026-08-02 22:30")) is False # summer Sun 18:30 ET
assert pbase._is_forex_weekend(T("2026-08-01 03:00")) is True # Saturday
@test("weekend calendar: all three copies agree hourly across DST transitions")
def _():
import providers.base as pbase
import signal_aggregator as sa_mod
import gpu_backtest as _gb
cfg = PlatformConfig()
cfg.xau_strategy.weekend_filter = True
cfg.xau_strategy.allowed_weekdays = [] # isolate the weekend gate
# Sweep spring-forward (2026-03-06..11) and fall-back (2026-10-30..11-04).
for start in ("2026-03-06", "2026-10-30"):
for ts in pd.date_range(start, periods=5 * 24, freq="1h", tz="UTC"):
a = pbase._is_forex_weekend(ts)
b = sa_mod._is_forex_weekend(ts)
c = _gb._is_xau_blocked_by_calendar(ts, cfg) == "WEEKEND_FILTER"
assert a == b == c, f"{ts}: base={a} aggregator={b} backtest={c}"
@test("FB2_EMA_LO: garbage value degrades to tuned default, scan survives")
def _():
os.environ["FB2_EMA_LO"] = "not-a-number"
try:
config = PlatformConfig()
config.primary_timeframe = Timeframe.H1
config.confirmation_timeframes = [Timeframe.M15, Timeframe.H4]
agg = SignalAggregator(config)
dfs = {
Timeframe.H1: make_ohlcv(300, seed=42),
Timeframe.M15: make_ohlcv(300, seed=43),
Timeframe.H4: make_ohlcv(300, seed=44),
}
report = agg.analyze_from_dataframes("XAU/USD", dfs, use_news=False)
assert isinstance(report, SignalReport), \
"analysis must survive an unparseable FB2_EMA_LO"
finally:
os.environ.pop("FB2_EMA_LO", None)
# ══════════════════════════════════════════════════════════════════════ # ══════════════════════════════════════════════════════════════════════
# SUMMARY # SUMMARY
# ══════════════════════════════════════════════════════════════════════ # ══════════════════════════════════════════════════════════════════════