diff --git a/CLAUDE.md b/CLAUDE.md index becc897..43cb07b 100644 --- a/CLAUDE.md +++ b/CLAUDE.md @@ -4,16 +4,19 @@ This file provides guidance to Claude Code (claude.ai/code) when working with co ## What this is -Signal-intelligence platform for **XAU/USD (gold — the only supported pair)**. It fetches OHLCV from the active data provider (**Oanda v20 since 2026-07-28**; Twelve Data and Tiingo remain registered fallbacks), scores 9 technical indicators per timeframe with regime-adaptive weights, checks multi-timeframe agreement and news risk, grades the signal (A+…F), and generates a full trade setup (entry, SL, TP1/TP2, trailing plan). Delivery: Telegram bot and CLI. A realism-focused backtester replays history through the *exact same* signal code. +Signal-intelligence platform for **XAU/USD (gold — the only supported pair)**. It fetches OHLCV from the active data provider (**Oanda v20 since 2026-07-28**; Twelve Data and Tiingo remain registered fallbacks), scores 9 technical indicators per timeframe with regime-adaptive weights, checks multi-timeframe agreement and news risk, grades the signal (A+…F), and generates a full trade setup (entry, SL, TP1/TP2, trailing plan). Delivery: Telegram bot, CLI, and a static web dashboard. A realism-focused backtester replays history through the *exact same* signal code. Every numeric threshold in `config.py` and `pair_signals/xauusd.py` was backtested or is load-bearing for a backtested result. Validation evidence lives in the factory/module docstrings and in `~/.claude/projects/.../memory/`. ## Commands ```bash -python test_all.py # QA suite: 414 tests, 27 modules, no network (mocked) +python test_all.py # QA suite, no network (mocked); prints its own count + # (463 tests / 31 modules as of 2026-09-06) python main.py --pair XAUUSD --timeframe H1 --detail # CLI signal report (--json for full dump, - # --panel to run the advisory LLM agent panel on-demand) + # --panel for the advisory LLM agent panel, --outlook for the + # hourly AI market brief; the violent-move watch line appears + # when the `violent_watch` config block is enabled) python telegram_bot.py # the bot (refuses to boot without credentials) python dashboard_publisher.py --once # web dashboard: one scan+publish cycle to $DASHBOARD_OUT_DIR # (omit --once for the candle-close daemon; deploy/README-dashboard.md) @@ -24,13 +27,15 @@ python backtest/gpu_backtest.py --pair XAUUSD --start 2025-01-01 --end 2025-06-0 --primary-tf 15min --confirm-tfs 1h,4h # key flags: --spread --quality A+,A,B,C --strategies --workers --max-hold-bars # --warmup-bars --cooldown-bars --no-cache --export-csv out.csv --detail +# --set PATH=JSON (research override of any PlatformConfig leaf; unknown path = hard exit) +# --export-features (append the 65 decision-time f_* columns; default export stays 63 cols) # --fixed-rr-tp N (research-only: hard TP at N×risk, trail off; verdict # 2026-08-05: NOT shipped — backtest/research/fixedtp_2026-08-05.md) ``` **`test_all.py` is NOT pytest** — it uses a custom `@test("name")` decorator that runs each test immediately at import; don't grep for `def test_` or try `pytest test_all.py`. Run the whole file; it prints per-module results and exits non-zero on failure. There is no way to run a single test short of commenting modules out. Tests must stay green; every new gate/knob needs tests. Signal-log fixtures: use a WEEKDAY signal bar when a test exercises candle continuity — the outcome model treats weekend and 21:00 UTC gaps as scheduled, so Saturday-dated bars never produce a gap. `_candles_json` stores `df.tail(14)`, so a test snapshot longer than 14 bars silently drops its head. When patching weekend logic in tests, patch `providers.base._is_forex_weekend` — but know there are THREE independent signal-path weekend implementations (`providers/base.py`, `signal_aggregator.py`, and an inline copy in `backtest/gpu_backtest.py:_is_xau_blocked_by_calendar`) plus the advisory-only continuity copy in `advisory_outcome.py`; patching one does not reach the others. -Python 3.12; a `venv/` exists in-repo. Dependencies in `requirements.txt` (numpy, pandas, requests, twelvedata, python-telegram-bot ≥20, websockets). +Python 3.12; a `venv/` exists in-repo. Dependencies in `requirements.txt` (numpy, pandas, requests, twelvedata, python-telegram-bot ≥20, websockets). scikit-learn and lightgbm are installed in the dev `venv/` for `backtest/research/` only — never add them to `requirements.txt` or import them from live code; anything shipped must be numpy-only. ## Configuration & credentials @@ -47,7 +52,7 @@ providers/.fetch_ohlcv() per TF: single-flight fetch, drop forming → signal_aggregator.analyze_pair() live entry point (weekend/day filters, fetch, staleness) └─ analyze_from_dataframes() replayable core: MTF agreement → news → bonuses/penalties → final signal + quality grade → gates → trade setup → reasons - → SignalReport consumed by telegram_bot.py, main.py, backtester + → SignalReport consumed by telegram_bot.py, main.py, dashboard_publisher.py, backtester ``` Signal currency everywhere is `SignalStrength` (−2..+2). DataFrames are UTC tz-aware, oldest-first, capitalized OHLCV columns. Everything runs on **closed bars only** — the forming candle is dropped, and session/hour filters evaluate **bar-CLOSE time** (bar label + candle duration) identically in live and backtest. @@ -69,7 +74,7 @@ Signal currency everywhere is `SignalStrength` (−2..+2). DataFrames are UTC tz ## Rules for making changes (read before editing signal logic) -1. **Never change a tuned value without re-validation.** Acceptance rule: positive total R in BOTH train and validation halves of a ≥12-month window, PF ≥ 1.3, at realistic cost (XAU $0.65 round trip headline; retain $0.30 for TD-era cross-book comparability), trailing exits, next-open fills. `round_trip_cost` stays $0.30 because it also drives the live TP spread gate; changing it is a separate rule-1 event. Marginal edges must additionally pass the 2024-06→2025-03 out-of-sample window — within-year splits alone have approved configs that later failed OOS. +1. **Never change a tuned value without re-validation.** Acceptance rule: positive total R in BOTH train and validation halves of a ≥12-month window, PF ≥ 1.3, at realistic cost (XAU $0.65 round trip headline; retain $0.30 for TD-era cross-book comparability), trailing exits, next-open fills. `round_trip_cost` stays $0.30 because it also drives the live TP spread gate; changing it is a separate rule-1 event. Marginal edges must additionally pass the 2024-06→2025-03 out-of-sample window — within-year splits alone have approved configs that later failed OOS. **Owner preference (2026-09-05): backtest windows go back at most 3 years** (the 5-year books are historical reference only); the 3-year baseline books (2023-09-05→2026-09-05, $0.65: M5 857/+161.3R/PF 1.53, M15 1086/+148.7R/PF 1.37, H1 255/+70.5R/PF 1.98, H4 113/+22.7R/PF 1.63) live in the gitignored `backtest/research/sizing_2026-09/` and are regenerated with the commands in `sizing_campaign_2026-09.md`. 2. **Per-timeframe overrides live in the pair factory**, never in global defaults — the same knob has opposite effects on different TFs (proven repeatedly: `tp_floor_rr`, hour-9 block, trail params). Document validation evidence in the factory docstring. 3. **Pair isolation is a hard invariant.** XAU gates in the aggregator are display-name-gated so a future pair port cannot inherit gold-calibrated logic by accident. 4. **Live == backtest parity.** If you add a live gate, add it to the replay path in the same commit; all time filters evaluate bar-close time in both paths. @@ -86,13 +91,13 @@ Signal currency everywhere is `SignalStrength` (−2..+2). DataFrames are UTC tz - **`FB2_EMA_LO`: HARDENED 2026-08-04 (delegated).** Research override for the EMA200-proximity band, default = tuned 0.5; an unparseable value now degrades to the default with a warning instead of crashing every scan. Documented here — it is a knob that exists. - **H4 seeding divergence: parked.** Live H4 fetches only 380 bars vs the backtest's unbounded history — ~2% of sampled H4 bars flip an indicator signal (composite Δ up to 12.3). M5/M15/H1 clean at their real fetch depths. Fold into any future H4 campaign; H4 remains "unproven". -## Current state notes (2026-08-14) +## Current state notes (dated per entry) -- Oanda v20 is the live feed (`data_providers.active: "oanda"`). The 2026-08-04 re-pin made the 27 `backtest/oanda_*.csv` books the baseline; all TD-era books are superseded. Current-HEAD M5 canonical book: `backtest/oanda_m5_12mo_065_nopb_20260814.csv`, +106.82R PF 1.51 at $0.65 (both halves positive; PF 1.46 / 1.55). M15 reference: +73.93R PF 1.53. See `backtest/research/repin_verification_2026-08-14.md`. +- Oanda v20 is the live feed (`data_providers.active: "oanda"`). The 2026-08-04 re-pin made the 27 `backtest/oanda_*.csv` books the baseline; all TD-era books are superseded. (2026-08-14) The re-pin M5 book `oanda_m5_12mo_065_nopb_20260814.csv` (+106.82R PF 1.51) is itself SUPERSEDED by the 2026-09-03 book in the entry below; the M15 reference is unchanged since 2026-08-04: `oanda_m15_12mo_065.csv` (397 / +73.93R / PF 1.53). See `backtest/research/repin_verification_2026-08-14.md`. - Cost reporting is dual: $0.65 is the measured-Oanda realistic headline ($0.65-$0.72 round trip), while $0.30 remains for cross-book comparability and as the live TP spread-gate input. Raising `round_trip_cost` from $0.30 would move the TP floor from about $9 to $19.50 and requires full rule-1 revalidation. - Twelve Data is signal-incompatible with Oanda (Phase 1: about 19% M5 bar-direction disagreement and a systematic ~$1 high bias; see `backtest/research/feed_divergence_2026-08-14.md`). On an Oanda candle outage, stop scanning rather than switching to TD candles. - Tiingo is streaming-only (`data_providers.streaming: "tiingo"`), not a designated candle fallback. Oanda-Tiingo signal parity is documented in `backtest/research/signal_parity_2026-08-14.md`; Phase 4's two-week live shadow scan remains a manual operational step. -- Human-facing docs in `docs/` (`user_guide.html`, `beginner_guide.html` + PDF builds) must be kept in sync when signal behavior changes. +- Human-facing docs in `docs/` (`user_guide.html`, `beginner_guide.html` + PDF builds) must be kept in sync when signal behavior changes. As of 2026-09-06 the PDF builds are BEHIND the HTML (violent-move watch entry added without a PDF rebuild). - 2026-09-03 signal-log campaign shipped two new per-TF gates (`backtest/research/signal_log_campaign_2026-09.md`): M5 `sell_blocked_regimes=["transitioning"]` and H1 `sell_min_quality=[]` (all H1 shorts blocked). Current-HEAD M5 canonical book is now `backtest/oanda_m5_12mo_065_nopb_20260903.csv` (557 / +112.21R / PF 1.56, halves +55.9/+56.4); H1 reference `backtest/oanda_h1_12mo_065_20260903.csv` (91 / +22.2R / PF 1.79; 5yr +49.1R PF 1.46; OOS +16.7R PF 1.74). - 2026-09-06 stake-sizing campaign (`backtest/research/sizing_campaign_2026-09.md`): a walk-forward "second opinion" stake multiplier (meta-labelling on 65 decision-time features, 3-year Oanda books at $0.65) FAILED the pre-registered rule on M5 (clean null) and M15 (lift inside the label-shuffle null, explained by BUY-vs-SELL). Nothing that changes numbers shipped. Behaviour-neutral tooling remains: `position_sizing.build_features` + `--export-features` (default export stays 63 columns), `SignalReport.penalty_tags/penalty_total/momentum_fading`, `TradeSetup.stake_multiplier` (None; reserved), `backtest/research/{book_report,burst_map,sizing_model}.py`. Stage-B facts: M5 stretch measures are POSITIVE in both halves (do not re-propose exhaustion sizing on M5); M15 exhaustion tag is weakly negative (≈ +3R/yr, noise); bursts cluster at 12–15 UTC and 01 UTC bar-close and do NOT hurt entries. - Auto-execution (oanda_daemon.py / oanda_executor.py) was removed 2026-09-03 — the owner is not auto-trading; last commit containing it is 31002cb. Book-parity rules that lived in its docstring (no broker TP, 5-bar cooldown, once-per-bar trail) remain the backtester's model.