diff --git a/CLAUDE.md b/CLAUDE.md index 479a6c0..90c0509 100644 --- a/CLAUDE.md +++ b/CLAUDE.md @@ -11,7 +11,7 @@ Every numeric threshold in `config.py` and `pair_signals/xauusd.py` was backtest ## Commands ```bash -python test_all.py # QA suite: 302 tests, 20 modules, no network (mocked) +python test_all.py # QA suite: 338 tests, 21 modules, no network (mocked) python main.py --pair XAUUSD --timeframe H1 --detail # CLI signal report (--json for full dump) python telegram_bot.py # the bot (refuses to boot without credentials) python oanda_daemon.py # auto-execution daemon on Oanda practice account @@ -26,7 +26,7 @@ python backtest/gpu_backtest.py --pair XAUUSD --start 2025-01-01 --end 2025-06-0 # --warmup-bars --cooldown-bars --no-cache --export-csv out.csv --detail ``` -**`test_all.py` is NOT pytest** — it uses a custom `@test("name")` decorator that runs each test immediately at import; don't grep for `def test_` or try `pytest test_all.py`. Run the whole file; it prints per-module results and exits non-zero on failure. There is no way to run a single test short of commenting modules out. Tests must stay green; every new gate/knob needs tests. When patching weekend logic in tests, patch `providers.base._is_forex_weekend`. +**`test_all.py` is NOT pytest** — it uses a custom `@test("name")` decorator that runs each test immediately at import; don't grep for `def test_` or try `pytest test_all.py`. Run the whole file; it prints per-module results and exits non-zero on failure. There is no way to run a single test short of commenting modules out. Tests must stay green; every new gate/knob needs tests. When patching weekend logic in tests, patch `providers.base._is_forex_weekend` — but know there are THREE independent weekend implementations (`providers/base.py`, `signal_aggregator.py`, and an inline copy in `backtest/gpu_backtest.py:_is_xau_blocked_by_calendar`); patching the provider one does not reach the other two. Python 3.12; a `venv/` exists in-repo. Dependencies in `requirements.txt` (numpy, pandas, requests, twelvedata, python-telegram-bot ≥20, websockets). @@ -39,7 +39,8 @@ Precedence for every credential/deployment setting: **env var → config.json ### End-to-end pipeline (every scan) ``` -providers/.fetch_ohlcv() per TF: throttle, fetch, drop forming candle, validate/clean, cache +providers/.fetch_ohlcv() per TF: single-flight fetch, drop forming candle, validate/clean, cache + (only Twelve Data rate-throttles; Oanda has no client throttle) → engine.TechnicalEngine.analyze() per TF: 9 scored indicators + ATR context → regime → composite score → signal_aggregator.analyze_pair() live entry point (weekend/day filters, fetch, staleness) └─ analyze_from_dataframes() replayable core: MTF agreement → news → bonuses/penalties → @@ -53,11 +54,11 @@ Signal currency everywhere is `SignalStrength` (−2..+2). DataFrames are UTC tz - **`providers/`** — one class per provider subclassing `BaseDataProvider` (`providers/base.py`), which owns everything the pipeline depends on: TTL+boundary cache, forming-candle drop, freshness check, validate/clean, paginated `fetch_range` (feeds the backtester from any provider), optional tick streaming. Subclasses implement only `_fetch_raw` + symbol/interval maps. Registry + `make_data_provider(config, name=None)` in `providers/__init__.py`; config.json `data_providers.active` selects the live candle feed and the optional `data_providers.streaming` scalar selects a different provider for the tick stream. A test enforces that subclasses don't override shared base methods (`supports_streaming`/`iter_ticks` are the sanctioned streaming override points). Tiingo streams mid-price ticks (`wss://api.tiingo.com/fx`) — do not re-add spread display from its free FX feed (two-tier indicative quotes, removed 2026-07-23); the no-spread rule extends to the WS stream. - **`engine.py`** — `TechnicalEngine`: 9 scored indicators (ema_trend, rsi, macd, bollinger, adx, roc, keltner, divergence, candlestick) + ATR as context. Regime classification (trending_up/down, `trending` = mixed/direction-UNKNOWN, transitioning, ranging) drives per-indicator weight multipliers and regime-aware RSI reinterpretation. Compute order matters (ATR first; MACD always computed even when not a pillar — divergence needs the histogram). Private series ride on the result as `_underscore` attrs; scorers/formatters skip anything `_`-prefixed or weight-0. -- **`signal_aggregator.py`** — MTF agreement, diminishing-decay penalty system, exhaustion hard block, quality grading, then an ordered chain of post-grade gates (session, XAU SELL quality/vol/hour, regime blocks, M5 noise gates), then three setup generators: XAU ATR (A+/A), generic swing-based, plus the M5 pullback (`XAU_PB`) and chart-pattern (`XAU_CP`, `chart_patterns.py`) entry paths. `TradeSetup.strategy` selects trailing params; trail params are stamped on the setup from the *producing* config so per-TF overrides reach formatters. +- **`signal_aggregator.py`** — MTF agreement, diminishing-decay penalty system, exhaustion hard block, quality grading, then an ordered chain of post-grade gates (session, XAU SELL quality/vol/hour, regime blocks, M5 noise gates), then three setup generators: XAU ATR (A+/A), generic swing-based, plus the M5 pullback (`XAU_PB`) and chart-pattern (`XAU_CP`, `chart_patterns.py`) entry paths. The PB/CP generators run AFTER the gate chain and force quality to "B"; the CP path deliberately accepts grade-F composites and re-checks only session/day/big-candle (via warning prefixes) plus its own SELL/vol/RSI gates — it does NOT re-check exhaustion or the M5 min-score/MTF gates. **This is validated behavior, not a bug** (2026-08-04 audit: 226/586 trades in the shipped M5 12mo book are XAU_CP, +38.9R, 104 of them exhaustion-flagged and net-positive) — changing it is a rule-1 revalidation event. `TradeSetup.strategy` selects trailing params; trail params are stamped on the setup from the *producing* config so per-TF overrides reach formatters. - **`pair_signals/`** — per-pair config factories. **Always use the factory (`produce_signal` / `make_config`), never bare `PlatformConfig()`** — the factory applies the config.json overlay first, then the validated per-TF deltas (M5 hour-9 block, M15 trail arm 0.70, M15 SELL-hour block, etc.; evidence in the docstring). A test enforces factory ≡ inline-config equivalence. Confirmation map: M5→[M15,H1], M15→[M5,H1], H1→[M15,H4], H4→[H1,D1]. - **`news_engine.py`** — Forex Factory calendar (MyFXBook RSS fallback) + RSS keyword sentiment with gold-inverse keywords. Class-level caches (fresh engine per scan). Optional DeepSeek enrichment via `llm_client.py` (requests-only, no SDK): batched headline sentiment overriding keyword scores, gold-direction on release surprises, and a market-summary blurb (`SignalReport.news_summary`) — **display/warning-only, keyword fallback on any failure, inert without a key**; prompts are static module constants (DeepSeek prefix-cache pricing) and requests send `thinking: disabled` (V4-Flash reasoning otherwise blows the timeout). By XAU-tuned default, event risk and sentiment **warn but do not gate signals** (`event_risk_caps_signals=False`, `sentiment_affects_signals=False`) — the LLM does not change that. - **`telegram_bot.py`** — owner-locked bot + subscriber channel; candle-boundary auto-scan loops (M5/M15/H1/H4, boundary+10 s), alert quality threshold + fingerprint dedup (committed only after a successful send), 📰 LLM market-context line on alerts when available, WebSocket level-breach monitor fed by the streaming provider (`data_providers.streaming`, falling back to the active provider; auto-disabled when the resolved provider can't stream or has no API key — Oanda does not stream, Tiingo does). Rotating log: `logs/telegram.log`, 3-h rotation × 240 backups. -- **`oanda_daemon.py` / `oanda_executor.py`** — practice-account auto-execution. Book-parity rules (see daemon docstring): no broker-side TP (trailing model — TP1/TP2 advisory), 5-bar per-TF cooldown, trail ratchets at bar close exactly once per bar. Accepted divergence: one position total. +- **`oanda_daemon.py` / `oanda_executor.py`** — practice-account auto-execution. Book-parity rules (see daemon docstring): no broker-side TP (trailing model — TP1/TP2 advisory), 5-bar per-TF cooldown (wall-clock approximation, so it expires over weekends where the backtester would still count bars), trail ratchets at bar close once per *observed* bar (a missed/failed cycle skips those bars entirely — the intra-gap high/low never enters the peak). Accepted divergence: one position total; orders send `positionFill=OPEN_ONLY` so a state/broker divergence cancels the order instead of silently netting. State/broker re-sync runs at boot and after any suspicious fill (`_reconcile_positions`). - **`dashboard_publisher.py` / `static/dashboard.html`** — web dashboard as a **static publisher** (live LAMP server): standalone daemon scans at candle closes via `produce_signal` and atomically writes `snapshot.json`/`news.json`/`dashboard.html` to `$DASHBOARD_OUT_DIR` (Apache serves them; page is public by owner decision). **Hard independence rule: never imports, and is never imported by, `telegram_bot`/`oanda_daemon`** — confirmation map and boundary math are deliberate local copies, so a web-side failure can only mean a stale page. Due-TF classification uses the RAW wake time (flooring wake−buffer skips every H1/H4 scan — 2026-08-02 bug). News panel reads `NewsEngine.get_fundamental_context()` plus the additive `get_week_events()` accessor (display-only; no signal path calls it); as a separate process it has its own news caches and DeepSeek daily cap. Deployment checklist: `deploy/README-dashboard.md`. - **`backtest/gpu_backtest.py`** — bar-by-bar replay through the real `analyze_from_dataframes` (news off). Realism model (do not regress): next-bar-open entries re-anchored to the raw price path, round-trip `--spread` charged per trade, trade resolution on RAW bars (cleaning applies only to signal-side data), same-bar SL+TP tie → SL first, gap fills at bar open, trail ratchet at bar close only. Indicators are precomputed once per TF (CuPy GPU or NumPy fallback — check the `Backend:` log line); replay fans out over worker processes and the 5-bar cooldown is applied in an ordered pass afterwards, so parallel and sequential books are identical. Parquet cache in `backtest/cache/` keyed by provider/symbol/TF/window. `backtest/*.csv` are gitignored pinned trade books from past campaigns — used for manual before/after comparison. `backtest/instructions_for_backtest.txt` lists what every backtest report must record. @@ -73,6 +74,13 @@ Signal currency everywhere is `SignalStrength` (−2..+2). DataFrames are UTC tz 8. Don't add config fields nothing reads; don't disable indicators in production paths (`disabled_indicators` is for experiments only). 9. EUR/USD was removed 2026-07-16 after a no-edge verdict (cost-to-volatility; last commit with EUR support: `7d3fd95`). Do not re-add without new data. BTC/ETH work lives on unmerged branches only. +## Known open items from the 2026-08-04 QA audit (owner decisions pending) + +- **Weekend constant vs Oanda reality**: `_is_forex_weekend` hardcodes Fri 22:00→Sun 22:00 UTC, but Oanda's week is Fri 17:00→Sun 18:00 New York time — during US DST the feed stops at 21:00 UTC (159/162 summer weekends measured) and winter reopen is Sun 23:00. Result: two false-"STALE DATA/feed lagging" hours weekly (summer Fri 21–22, winter Sun 22–23). Fix = NY-anchored calendar in all three copies; books provably unaffected (no bars exist in those hours) but run a book gate anyway. +- **Spread in exit-trigger detection (rule 6)**: backtest stop triggers use raw mid High/Low; bid/ask-adjusted triggers would change 15/397 trades (−3.55R) at $0.30 and 30/397 (−4.24R) at $0.65 on the pinned M15 12mo book — a ~4–6% systematic optimism. Fixing invalidates all book comparability: conscious re-pin decision only. +- **H4 seeding divergence**: live H4 fetches only 380 bars; the backtest sees unbounded history — ~2% of sampled H4 bars flip an indicator signal (composite Δ up to 12.3). M5/M15/H1 are clean at their real fetch depths. H4 is already "unproven for auto-execution". +- `FB2_EMA_LO` env var is an undocumented knob inside the scoring path (`signal_aggregator.py` EMA200-proximity penalty floor); an unparseable value crashes every analysis. Remove or config-ify only with rule-1 revalidation. + ## Current state notes (2026-07-30) - Oanda v20 is the live feed (`data_providers.active: "oanda"`), but the pinned backtest books are still Twelve-Data-pinned and the backtest cost assumption is $0.30 while measured real spreads are ~$0.65–0.72 — re-pin books on Oanda bars before treating feed-switch comparisons as valid (see Oanda/TD divergence study in memory). diff --git a/backtest/gpu_backtest.py b/backtest/gpu_backtest.py index cabd4c7..4f0eb8b 100644 --- a/backtest/gpu_backtest.py +++ b/backtest/gpu_backtest.py @@ -1678,6 +1678,24 @@ def _reanchor_entries_to_raw( ) +def _apply_cooldown( + candidates: List[Tuple[int, Trade]], cooldown_bars: int +) -> List[Tuple[int, Trade]]: + """Ordered cooldown pass over (bar_pos, Trade) candidates: drop any + candidate within `cooldown_bars` bars of the previously ACCEPTED + entry — exactly what a sequential replay loop would have skipped. + Candidates must already be sorted by position (stable merge from the + worker spans).""" + accepted: List[Tuple[int, Trade]] = [] + last_entry_pos = -10_000 + for pos, trade in candidates: + if pos - last_entry_pos < cooldown_bars: + continue + accepted.append((pos, trade)) + last_entry_pos = pos + return accepted + + def run_backtest( bt_cfg: BacktestConfig, platform_cfg: Optional[PlatformConfig] = None, @@ -1805,18 +1823,10 @@ def run_backtest( candidates.extend(cands) candidates.sort(key=lambda pc: pc[0]) - # ── Phase 2: entry cooldown — the only sequential rule. A candidate - # inside the cooldown window of the previously ACCEPTED entry is - # dropped, exactly as the sequential loop skipped those bars. - trades: List[Trade] = [] - signal_ts_list: List[pd.Timestamp] = [] - last_entry_pos = -10_000 - for pos, trade in candidates: - if pos - last_entry_pos < bt_cfg.cooldown_bars: - continue - trades.append(trade) - signal_ts_list.append(primary_df.index[pos]) - last_entry_pos = pos + # ── Phase 2: entry cooldown — the only sequential rule. + accepted = _apply_cooldown(candidates, bt_cfg.cooldown_bars) + trades: List[Trade] = [t for _, t in accepted] + signal_ts_list: List[pd.Timestamp] = [primary_df.index[p] for p, _ in accepted] logger.info( f"Replay complete: {len(trades)} entries in " diff --git a/dashboard_publisher.py b/dashboard_publisher.py index 496241f..17723d6 100644 --- a/dashboard_publisher.py +++ b/dashboard_publisher.py @@ -231,8 +231,11 @@ def seconds_until_boundary(interval_minutes: int, buffer: int = 10) -> float: elapsed_seconds = elapsed_minutes * 60 + now.second + now.microsecond / 1e6 interval_seconds = interval_minutes * 60 remaining = interval_seconds - elapsed_seconds + buffer - if remaining <= buffer: - remaining += interval_seconds + # NOTE: remaining > buffer always (elapsed < interval by construction); + # a call at the exact boundary instant targets the NEXT boundary — + # steady-state loops wake at boundary+buffer, so only a process + # started in the boundary microsecond skips one scan (2026-08-04 + # audit I42; the old `remaining <= buffer` guard was unreachable). return remaining diff --git a/oanda_daemon.py b/oanda_daemon.py index 82797d7..47a7614 100644 --- a/oanda_daemon.py +++ b/oanda_daemon.py @@ -139,8 +139,11 @@ def _seconds_until_boundary(interval_minutes: int, buffer: int = 10) -> float: elapsed_seconds = elapsed_minutes * 60 + now.second + now.microsecond / 1e6 interval_seconds = interval_minutes * 60 remaining = interval_seconds - elapsed_seconds + buffer - if remaining <= buffer: - remaining += interval_seconds + # NOTE: remaining > buffer always (elapsed < interval by construction); + # a call at the exact boundary instant targets the NEXT boundary — + # steady-state loops wake at boundary+buffer, so only a process + # started in the boundary microsecond skips one scan (2026-08-04 + # audit I42; the old `remaining <= buffer` guard was unreachable). return remaining # ── Trade state persistence ─────────────────────────────────────────── diff --git a/telegram_bot.py b/telegram_bot.py index 23ec767..8f38ced 100755 --- a/telegram_bot.py +++ b/telegram_bot.py @@ -103,6 +103,12 @@ logging.basicConfig(level=getattr(logging, _APP.log_level, logging.INFO), force=True) logging.getLogger("httpx").setLevel(logging.WARNING) logging.getLogger("httpcore").setLevel(logging.WARNING) +# Never let DEBUG runs write credentials into the 30-day rotating log: +# the websockets client logger emits the connect URI (Twelve Data puts +# the API key in the query string) and frame contents (Tiingo's auth +# frame carries the key) — 2026-08-04 audit, finding K47. +logging.getLogger("websockets").setLevel(logging.INFO) +logging.getLogger("websockets.client").setLevel(logging.INFO) logger = logging.getLogger("telegram_bot") TOKEN = telegram_token() @@ -667,8 +673,11 @@ def _seconds_until_boundary(interval_minutes: int, buffer: int = 10) -> float: elapsed_seconds = elapsed_minutes * 60 + now.second + now.microsecond / 1e6 interval_seconds = interval_minutes * 60 remaining = interval_seconds - elapsed_seconds + buffer - if remaining <= buffer: - remaining += interval_seconds + # NOTE: remaining > buffer always (elapsed < interval by construction); + # a call at the exact boundary instant targets the NEXT boundary — + # steady-state loops wake at boundary+buffer, so only a process + # started in the boundary microsecond skips one scan (2026-08-04 + # audit I42; the old `remaining <= buffer` guard was unreachable). return remaining diff --git a/test_all.py b/test_all.py index 942f1da..cb3fad4 100644 --- a/test_all.py +++ b/test_all.py @@ -6316,6 +6316,87 @@ def _(): _aio21.run(main()) +def _m21_sim_trade(direction="BUY", entry=2400.0, sl=2390.0, tp1=2410.0, + ts="2026-01-05 10:00"): + import gpu_backtest as _gb + return _gb.Trade(pair="XAU/USD", entry_time=pd.Timestamp(ts, tz="UTC"), + entry_price=entry, sl=sl, tp1=tp1, tp2=tp1 + 10, + direction=direction, pip_value=0.1, risk_pips=100.0, + reward_pips=100.0, rr_planned=1.0, quality="B", + final_score=30.0, final_confidence=60.0, + market_regime="trending_up") + + +def _m21_sim_df(bars, ts="2026-01-05 10:00"): + idx = pd.date_range(ts, periods=len(bars), freq="15min", tz="UTC") + return pd.DataFrame( + {"Open": [b[0] for b in bars], "High": [b[1] for b in bars], + "Low": [b[2] for b in bars], "Close": [b[3] for b in bars], + "Volume": 1.0}, index=idx) + + +@test("backtester: same-bar SL+TP tie resolves to SL (static path)") +def _(): + import gpu_backtest as _gb + t = _m21_sim_trade() + sim = _m21_sim_df([(2400, 2411, 2389, 2400)]) # both barriers inside bar 0 + _gb.simulate_trades([t], sim, max_bars=5, spread=0.0, trail_arm_r=0.0) + assert t.exit_reason == "SL", f"tie must be SL-first, got {t.exit_reason}" + assert abs(t.exit_price - 2390.0) < 1e-9 + assert abs(t.r_multiple - (-1.0)) < 1e-9 + + +@test("backtester: gap through SL fills at the open, not the stop") +def _(): + import gpu_backtest as _gb + t = _m21_sim_trade() + sim = _m21_sim_df([(2385, 2388, 2384, 2386)]) # opens below the 2390 stop + _gb.simulate_trades([t], sim, max_bars=5, spread=0.0, trail_arm_r=0.0) + assert t.exit_reason == "SL" + assert abs(t.exit_price - 2385.0) < 1e-9, "gap must fill at the open" + assert t.r_multiple < -1.0, "gap loss must exceed 1R" + + +@test("backtester: TIMEOUT closes at the last window bar's close, spread charged") +def _(): + import gpu_backtest as _gb + t = _m21_sim_trade() + sim = _m21_sim_df([(2400, 2405, 2395, 2402)] * 3) # never hits either side + _gb.simulate_trades([t], sim, max_bars=3, spread=0.30, trail_arm_r=0.0) + assert t.exit_reason == "TIMEOUT" and t.bars_held == 3 + assert abs(t.exit_price - 2402.0) < 1e-9 + expected_r = (2402.0 - 2400.0 - 0.30) / 10.0 + assert abs(t.r_multiple - expected_r) < 1e-9, \ + "R must be (pnl - round-trip spread) / risk" + + +@test("backtester: trailing stop ratchets at bar close, fills next bar only") +def _(): + import gpu_backtest as _gb + t = _m21_sim_trade() + # bar0: runs to +1R (2410) — arms trail (0.6R=6), stop → 2410-7.5=2402.5; + # bar0 low 2398 must NOT trigger the new stop intra-bar. + # bar1: dips to 2402 → trailed stop 2402.5 hit → TRAIL exit. + sim = _m21_sim_df([(2400, 2410, 2398, 2409), (2409, 2409.5, 2402, 2403)]) + _gb.simulate_trades([t], sim, max_bars=5, spread=0.0, + trail_arm_r=0.6, trail_dist_r=0.75) + assert t.exit_reason == "TRAIL", f"got {t.exit_reason}" + assert t.bars_held == 2, "new stop may only fill from the NEXT bar" + assert abs(t.exit_price - 2402.5) < 1e-9 + assert abs(t.r_multiple - 0.25) < 1e-9 + + +@test("backtester: cooldown pass drops candidates inside the window, in order") +def _(): + import gpu_backtest as _gb + cands = [(10, "A"), (12, "B"), (15, "C"), (21, "D"), (24, "E")] + out = _gb._apply_cooldown(cands, cooldown_bars=5) + assert [(p, t) for p, t in out] == [(10, "A"), (15, "C"), (21, "D")], \ + f"cooldown must measure from the previously ACCEPTED entry, got {out}" + assert _gb._apply_cooldown([], 5) == [] + assert [p for p, _ in _gb._apply_cooldown(cands, 1)] == [10, 12, 15, 21, 24] + + # ══════════════════════════════════════════════════════════════════════ # SUMMARY # ══════════════════════════════════════════════════════════════════════