diff --git a/CLAUDE.md b/CLAUDE.md index f27fc72..8a21bf1 100644 --- a/CLAUDE.md +++ b/CLAUDE.md @@ -12,7 +12,7 @@ Every numeric threshold in `config.py` and `pair_signals/xauusd.py` was backtest ```bash python test_all.py # QA suite, no network (mocked); prints its own count - # (498 tests / 33 modules as of 2026-09-09) + # (499 tests / 33 modules as of 2026-09-09) python main.py --pair XAUUSD --timeframe H1 --detail # CLI signal report (--json for full dump, # --panel for the advisory LLM agent panel, --outlook for the # hourly AI market brief; the violent-move watch line appears @@ -68,7 +68,7 @@ Signal currency everywhere is `SignalStrength` (−2..+2). DataFrames are UTC tz - **`providers/`** — one class per provider subclassing `BaseDataProvider` (`providers/base.py`), which owns everything the pipeline depends on: TTL+boundary cache, forming-candle drop, freshness check, validate/clean, paginated `fetch_range` (feeds the backtester from any provider), optional tick streaming. Subclasses implement only `_fetch_raw` + symbol/interval maps. **Cleaning rule (2026-09-07):** the signal-side outlier filter is a bad-print filter, not a volatility filter — a >10σ close-to-close bar is dropped only when the NEXT close snaps back ≥80% of the way (`OUTLIER_SIGMA`/`OUTLIER_REVERT_FRACTION` on the base class); persisting moves (NFP/CPI/Sunday-gap bars) and the newest bar are always kept. The previous unconditional 10σ rule deleted ~36 real M5 bars/yr in the backtester and a *different* ~24/yr live (window-dependent σ), hiding e.g. the 2026-09-04 12:30 UTC NFP bar from ATR/exhaustion math. Oanda fetches retry 401/5xx/network faults along `RETRY_DELAYS_S` (2/5/10 s) under a 45 s `RETRY_BUDGET_S`; a scan that still loses its primary frame is logged with `scan_kind='failed'`. Registry + `make_data_provider(config, name=None)` in `providers/__init__.py`; config.json `data_providers.active` selects the live candle feed and the optional `data_providers.streaming` scalar selects a different provider for the tick stream. A test enforces that subclasses don't override shared base methods (`supports_streaming`/`iter_ticks` are the sanctioned streaming override points). Tiingo streams mid-price ticks (`wss://api.tiingo.com/fx`) — do not re-add spread display from its free FX feed (two-tier indicative quotes, removed 2026-07-23); the no-spread rule extends to the WS stream. - **`engine.py`** — `TechnicalEngine`: 9 scored indicators (ema_trend, rsi, macd, bollinger, adx, roc, keltner, divergence, candlestick) + ATR as context. Regime classification (trending_up/down, `trending` = mixed/direction-UNKNOWN, transitioning, ranging) drives per-indicator weight multipliers and regime-aware RSI reinterpretation. Compute order matters (ATR first; MACD always computed even when not a pillar — divergence needs the histogram). Private series ride on the result as `_underscore` attrs; scorers/formatters skip anything `_`-prefixed or weight-0. - **`signal_aggregator.py`** — MTF agreement, diminishing-decay penalty system, exhaustion hard block, quality grading, then an ordered chain of post-grade gates (session, XAU SELL quality/vol/hour, regime blocks, M5 noise gates), then three setup generators: XAU ATR (A+/A), generic swing-based, plus the M5 pullback (`XAU_PB`) and chart-pattern (`XAU_CP`, `chart_patterns.py`) entry paths. Step 7 lives in `_attach_core_setup`; a final pass after the PB/CP archetypes (`_drop_stale_spread_gate_vetoes`, 2026-09-08) moves a core generator's `SPREAD GATE:` veto to `cleaned_warnings` whenever a later generator shipped a setup (metadata only — books unchanged; the exported `warnings` text column can differ; the prefix is the `_SPREAD_GATE_PREFIX` constant shared by the emitters and both cleanups). The PB/CP generators run AFTER the gate chain and force quality to "B"; the CP path deliberately accepts grade-F composites and re-checks only session/day/big-candle (via warning prefixes) plus its own SELL/vol/RSI gates — it does NOT re-check exhaustion or the M5 min-score/MTF gates. **This is validated behavior, not a bug** (2026-08-04 audit: 226/586 trades in the shipped M5 12mo book are XAU_CP, +38.9R, 104 of them exhaustion-flagged and net-positive) — changing it is a rule-1 revalidation event. `TradeSetup.strategy` selects trailing params; trail params are stamped on the setup from the *producing* config so per-TF overrides reach formatters. -- **`pair_signals/`** — per-pair config factories. **Always use the factory (`produce_signal` / `make_config`), never bare `PlatformConfig()`** — the factory applies the config.json overlay first, then the validated per-TF deltas (M5 hour-9 block, M15 trail arm 0.70, M15 SELL-hour block, M5 SELL-in-transitioning block, H1 all-shorts block (2026-09-03), etc.; evidence in the docstring). A test enforces factory ≡ inline-config equivalence. Confirmation map: M5→[M15,H1], M15→[M5,H1], H1→[M15,H4], H4→[H1,D1]. +- **`pair_signals/`** — per-pair config factories. **Always use the factory (`produce_signal` / `make_config`), never bare `PlatformConfig()`** — the factory applies the config.json overlay first, then the validated per-TF deltas (M5 hour-9 block, M15 trail arm 0.70, M15 SELL-hour block, M5 SELL-in-transitioning block, H1 all-shorts block (2026-09-03), M15 RSI regime-multiplier × 0.5 via `technical.regime_weights_override` (2026-09-09), etc.; evidence in the docstring). A test enforces factory ≡ inline-config equivalence. Confirmation map: M5→[M15,H1], M15→[M5,H1], H1→[M15,H4], H4→[H1,D1]. - **`signal_log.py` / `advisory_outcome.py`** — SQLite signal-history logger (stdlib only). Hooked into `SignalAggregator.analyze_pair` only (the live entry point — the backtester never writes here, preserving parity), so every live scan from every surface (Telegram `telegram`, CLI `cli`, dashboard `dashboard`) lands one row in `logs/signals.db`: signal/grade/price, MTF + trade-setup + reasons JSON, full per-TF indicator snapshots, the last 14 closed OHLCV candles per fetched TF, and the news at scan time (event risk + calendar events + scored headlines, re-fetched via the NewsEngine class-level caches). Config.json `signal_log` block (`enabled`, `path`, `log_blocked_scans`; env `SIGNAL_LOG_ENABLED`/`SIGNAL_LOG_PATH`/`SIGNAL_LOG_BLOCKED_SCANS`). **Fail-safe by contract** — a logging failure can never break a scan. Panel fields arrive post-hoc: `log_signal` stashes the rowid and `agent_panel.attach_panel` writes verdict + creation/cache provenance + payload hash onto the same row. **Advisory outcomes (2026-09):** EVERY row with a trade setup is seeded at log time (not only panel rows); same-setup re-logs within 60 min from any surface (`setup_key` = tf|direction|entry|stop|strategy) are `DUPLICATE` rows pointing at the canonical row via `advisory_event_id`, so sum `advisory_r_multiple` over `RESOLVED` rows only. The versioned model advances from later stored closed-candle snapshots (next-open reanchoring, intrabar peak with close-only ratchet, SL-first, scheduled weekend/21:00-UTC closures allowed, 300-bar timeout, $0.30/$0.65 views); when a snapshot no longer overlaps the outcome's last bar it rebuilds the stream from every stored snapshot of that TF before giving up. It is a cleaned-mid evidence model, not execution P&L. **Post-gate truth columns:** `gate_reason` (the warning that removed the setup), `actionable` (grade ≥ C AND setup — the `action` text is written BEFORE the gate chain and can say "STRONG SELL — High-quality setup" on a D row), `cleaned_warnings_json` (warnings a shipped setup superseded — the XAU_CP acceptance's stripped vetoes and, since 2026-09-08, a stale core spread-gate veto; non-empty is NOT a CP marker, use `setup_json.strategy`), `bar_range_median` (= the misnamed `spread` column: median High-Low, NOT a broker spread), `scan_kind` (`blocked` weekend/day early returns carry no news/candles; `failed` = the primary fetch failed — price 0, no signal — readers skip it like `blocked`), `config_fingerprint` (gate config hash — surfaces config drift between processes). **News is content-addressed:** `headlines_json`/`events_json` are interned into `news_snapshots` (they were 73% of a 667 MB DB); read through the `signals_with_news` view or `signal_log.read_news`. Readers must open with `mode=ro`, never `immutable=1` (blind to WAL rows). **Weight-research columns (2026-09):** `bar_ts` (the primary bar LABEL the scan closed on — same string as `indicators_json`, indexed with `timeframe`), `engine_fingerprint` (hash of the SCORING config: indicator params, thresholds, the regime weight table in force, TFs — `SignalAggregator.engine_config_fingerprint`, the gate fingerprint's twin), `code_version` (short git SHA), `penalty_tags_json`/`penalty_total`/`momentum_fading`/`exhaustion_count`, and the post-hoc `watch_level`/`watch_kinds`/`watch_line` written by `signal_log.update_watch` from `violent_watch.attach_watch` (level and kinds are stored even when the display line is off). Every `indicators_json` entry also carries `eff_weight` (base weight × regime multiplier = the weight the composite actually used; `engine.regime_weight_table` is the single source of truth) and `wscore` (its contribution) — null for skipped indicators (weight 0, `_`-prefixed); **divergence and candlestick entries exist only when they fire, so absence is "no vote", not a zero vote**. **Table `scan_outcomes`:** model-free forward moves per scan (`fwd_1h/4h/24h` in $ and ATRs, `mfe_24h_atr`/`mae_24h_atr`, `burst_1h` = a ≥3×ATR bar within the hour, `bars_used`, `complete`), measured from the row's own last closed bar on the stored candle stream; a horizon stays NULL until it has elapsed, and `complete` needs the full 24 h with no gap over 3 bars beyond the scheduled weekend/17:00-NY closures (`advisory_outcome._has_continuity`). `SignalLogger.label_scan_outcomes` runs opportunistically at the end of `log()` (at most once per 60 s per process, 150 rows and ~2 s per pass, idempotent — a row is rewritten only when its numbers change), so no daemon is needed; `scripts/label_scan_outcomes.py` is the backlog driver (`--dry-run` counts read-only). Readers must dedup by the FIRST row per (`timeframe`, `bar_ts`) — several surfaces log the same bar. Tools: `analyze_signal_log.py` (`--stored-candles` DB-only replay with event/opportunity/one-position books + persona scorecard, `--news` news-sentiment lead/lag + extremes + outcome buckets, `--parity` telegram-vs-dashboard divergence report, `--scan-outcomes` forward-move + hit-rate table by TF×signal/grade with a NEUTRAL baseline and burst rate by watch level), `scripts/backfill_signal_outcomes.py` (seed/dedup/advance legacy rows), `scripts/label_scan_outcomes.py` (label/relabel scan outcomes), `scripts/migrate_signal_log_news.py` (intern legacy blobs; `--vacuum` with both writers stopped). - **`news_engine.py`** — Forex Factory calendar (MyFXBook RSS fallback) + RSS keyword sentiment with gold-inverse keywords. Class-level caches (fresh engine per scan). Optional DeepSeek enrichment via `llm_client.py` (requests-only, no SDK): batched headline sentiment overriding keyword scores, gold-direction on release surprises, and a market-summary blurb (`SignalReport.news_summary`) — **display/warning-only, keyword fallback on any failure, inert without a key**; prompts are static module constants (DeepSeek prefix-cache pricing) and requests send `thinking: disabled` (V4-Flash reasoning otherwise blows the timeout). By XAU-tuned default, event risk and sentiment **warn but do not gate signals** (`event_risk_caps_signals=False`, `sentiment_affects_signals=False`) — the LLM does not change that. **Headline pool (2026-09-04):** only headlines ≤ 4 h old with a parsable publish time (`headline_max_age_seconds=14400`, undated items dropped), and `get_fundamental_context` drops headlines whose every relevant score is inside ±0.1 (`_is_neutral_headline`, `NEUTRAL_HEADLINE_THRESHOLD`) before the per-currency mean — so `news_sentiment` from that date is a mean over directional headlines only (bolder values; log comparisons must split at the deploy date). The market summary prompt writes for someone who has never traded (no acronyms), sends headlines newest-first with the platform's own gold direction, and caps at 400 chars. **Verdict 2026-09-04** (`backtest/research/news_sentiment_value_2026-09-04.md`): `news_sentiment` is a trailing-24h echo of price (r≈0.67 with the past day, ≈0.03 with the next; flips 44 h after tops / 16 h after bottoms) — both gating flags stay False; re-check with `analyze_signal_log.py --stored-candles --news`. - **`news_archive.py`** — durable headline corpus in its OWN database (`logs/newsdata.db`), separate from `signals.db` so a signal-log rotation cannot take the news with it (that is exactly how the 3-week corpus behind `news_sentiment_value_2026-09-04.md` was lost — 4,054 distinct headlines, unrecoverable, `logs/` is gitignored). One row per distinct headline keyed on `sha256(headline)`, carrying the text, description, link, publisher, publish time, the score, `score_source` (`keyword`/`llm`/`model` — a new `NewsSentiment` field, also mirrored into `signal_log._headline_to_dict`), and `teacher_model`/`teacher_prompt_sha1` so labels made under a changed prompt are never pooled with older ones. **Writes are one atomic UPSERT per row and upgrade-only**: an `llm` score always replaces a `keyword` one (the >25/cycle batch-overflow tail gets its DeepSeek label on the next fetch window in a long-running process — verified: 13/13 upgraded) but a `keyword` score can never overwrite an `llm` one; re-seeing a headline always bumps `seen_count`. A read-then-write pair loses increments when the bot, the dashboard publisher and the tests first see a headline together, and the WAL switch itself needs `_with_lock_retry` — both are regression-tested with 3 concurrent writers. Hooked at the END of `news_engine._fetch_news_sentiment_raw`, **before** `_is_neutral_headline` runs in `get_fundamental_context`: archiving downstream of that ±0.1 filter is why `signals.db` holds almost no neutral headlines and would teach a trained scorer that "neutral" barely exists. Never raises (fail-safe by contract, like `signal_log`). The backtester never builds a `NewsEngine`, so replays never archive (test-locked). Seeded by `scripts/backfill_news_archive.py`, which reads both DB shapes read-only and **recomputes** provenance (stored score vs the keyword scorer: differs → `llm`, exact non-zero match → `keyword`, both zero → `unknown`) because neither source recorded it. Purpose: training data for a local numpy-only headline scorer to replace the 24/78 keyword fallback — see the note below. @@ -100,7 +100,7 @@ Signal currency everywhere is `SignalStrength` (−2..+2). DataFrames are UTC tz ## Current state notes (dated per entry) -- Oanda v20 is the live feed (`data_providers.active: "oanda"`). The 2026-08-04 re-pin made the 27 `backtest/oanda_*.csv` books the baseline; all TD-era books are superseded. (2026-08-14) The re-pin M5 book `oanda_m5_12mo_065_nopb_20260814.csv` (+106.82R PF 1.51) is itself SUPERSEDED by the 2026-09-03 book in the entry below; the M15 reference is unchanged since 2026-08-04: `oanda_m15_12mo_065.csv` (397 / +73.93R / PF 1.53). See `backtest/research/repin_verification_2026-08-14.md`. +- Oanda v20 is the live feed (`data_providers.active: "oanda"`). The 2026-08-04 re-pin made the 27 `backtest/oanda_*.csv` books the baseline; all TD-era books are superseded. (2026-08-14) The re-pin M5 book `oanda_m5_12mo_065_nopb_20260814.csv` (+106.82R PF 1.51) is itself SUPERSEDED by the 2026-09-03 book in the entry below; the M15 reference `oanda_m15_12mo_065.csv` (397 / +73.93R / PF 1.53) is SUPERSEDED on 2026-09-09 by the 2-year book `oanda_m15_2yr_065_20260909.csv` (931 / +146.14R / PF 1.43; RSI × 0.5 override shipped — see the 2026-09-09 entries below). See `backtest/research/repin_verification_2026-08-14.md`. - Cost reporting is dual: $0.65 is the measured-Oanda realistic headline ($0.65-$0.72 round trip), while $0.30 remains for cross-book comparability and as the live TP spread-gate input. Raising `round_trip_cost` from $0.30 would move the TP floor from about $9 to $19.50 and requires full rule-1 revalidation. The $0.65 figure comes from the 2026-07-24 Dukascopy tick study (9M ticks over 12 months; its tooling was removed 2026-09-06 — the conclusion lives here and in memory). - Twelve Data is signal-incompatible with Oanda (Phase 1: about 19% M5 bar-direction disagreement and a systematic ~$1 high bias; see `backtest/research/feed_divergence_2026-08-14.md`). On an Oanda candle outage, stop scanning rather than switching to TD candles. - Tiingo is streaming-only (`data_providers.streaming: "tiingo"`), not a designated candle fallback. Oanda-Tiingo signal parity is documented in `backtest/research/signal_parity_2026-08-14.md`; Phase 4's two-week live shadow scan (`scripts/shadow_scan.py`) remains a manual operational step. @@ -113,4 +113,4 @@ Signal currency everywhere is `SignalStrength` (−2..+2). DataFrames are UTC tz - 2026-09-06 stale-code cleanup (behaviour-neutral; suite green, M5/H1 12-month books byte-identical after every signal-side commit): `main.py --timeframe` now accepts M5/M15/H1/H4 and takes its confirmation TFs from the pair factory (the CLI's private table had paired M5 with M1+M15); `PlatformConfig.pip_value` is gone (`PAIR_INFO` is the source); removed as dead or superseded: `_get_affected_pairs`/`event_impact_on_pairs`, `SignalLogger.advance_with_bars`/`path`/`news_for`, `reset_llm_client`, the Telegram scan-summary/keyboard helpers and duplicate confirmation map, the 2026-07 Oanda-vs-TD research pager+study, `burst_profile.py`, `hour_analysis.py`, `scripts/compare_oanda_tiingo.py`, the never-implemented LLM-provider plan and the merged config.json spec. Second pass the same day (owner directive: keep only what the system needs): every research script and data file under `backtest/research/` (book_report, book_diff, the burst tools, sizing_model, commit_ladder, llm_news_eval + golden CSV, ff_calendar_scraper, calendar_impact, watch_rule_eval, the Dukascopy spread study + CSVs, the panel dry-test JSONL), the two completed study scripts in `scripts/`, the architecture diagram + generator and the data-source test plan were deleted; the campaign `.md` reports stay as the decision ledger. Git history before `953c5ad` has all of it. Third pass (owner: "remove dead functions as well"): the write-only pb/cp rejection counters, the unreachable `_TP_QUALITY_SCALE` B/C entries, the engine's unused DataFrame argument and M1 arm, the providers' generic-symbol guesses (unknown symbols now raise), the sub-$10 price format, the `=X` pair parsing and the unread `minutes_until`/`is_high_impact` news.json keys are gone. - 2026-09-08 news archive shipped (`news_archive.py`, 483 tests; nothing in the signal core changed, both news gating flags stay `False`, books untouched). Motivation: headlines only ever survived as a side effect of signal logging and `signals.db` was rotated around 2026-09-04, destroying the 3-week / 4,054-headline corpus the news study was written on — with `logs/` gitignored, git could not recover it and nobody noticed for four days. Day-one corpus after backfilling both DBs on disk: **1,016 distinct headlines, 846 of them DeepSeek-labelled**, growing ~200/day (~1,400/week). The point of collecting it: distil DeepSeek's headline scoring (78/78 on the 2026-08-06 golden set) into a numpy-only local model so the fallback stops being the keyword scorer (**24/78 = 31%**, which is what silently runs on no key / disabled / daily cap / timeout / HTTP or parse failure / batch overflow). **Do not train yet** — 800-odd labels gives a holdout too small to grade (SE ±0.032); the pre-registered plan is ~4,000 deduplicated headlines over ≥3 weeks, time-split with near-duplicates clustered first (the same story is re-titled by several outlets, so a random split scores memorisation), headline text ONLY (the teacher never sees `description` — `_llm_headline_scores` sends titles), graded against the golden set recoverable with `git show 1cebebf^:backtest/research/llm_news_golden.csv`. When it ships it slots in as DeepSeek → model → keyword and stays display-only. Training happens offline in `venv/`; a Module 32 test now AST-walks the root, `providers/` and `pair_signals/` asserting no `sklearn`/`scipy`/`lightgbm`/`torch` import and that none of them reach `requirements.txt` — that guard did not exist before. - 2026-09-09 replay recording shipped (`backtest/replay_record.py`, `backtest/replay_scorecard.py`, Module 33, 498 tests; nothing in the signal core changed, books byte-identical — `backtest/research/replay_recording_2026-09-09.md`). `--record DIR` stores every ACCEPTED trade's decision evidence in Parquet: each indicator's vote / value / confidence / BASE weight / regime multiplier / EFFECTIVE weight / contribution on the primary AND both confirmation TFs (`indicators.parquet`, long; absence = the indicator did not fire, never a zero row), per-TF regime/composite, penalty tags, setup, outcome, and the Forex Factory calendar events from 4 h before entry to exit (`events.parquet`, all-day rows matched on the New-York date; scalar `ev_*`/`us_holiday` columns on `trades.parquet`); `--record-bars` adds one float32 row per ANALYZED bar with the same votes plus 1 h/4 h/24 h forward moves in ATRs (`bars.parquet`). The manifest carries the engine + gate fingerprints, the regime weight table in force and the book MD5. Calendar source is the gitignored `backtest/cache/calendar.parquet` (15,124 FF events 2023-09-01→2026-09-10, unix-UTC `dateline`; `timeLabel` is display-only); the scraper that built it is back at `scripts/ff_calendar_scraper.py` (resume + idempotent merge, ~1.6 s/request). The 2-year books (2024-09-05→2026-09-05, $0.65) live in `backtest/research/replay_2026-09/{m5,m15}/`: M5 781 / +146.07R / PF 1.52 (halves +28.0/+118.1), M15 793 / +117.62R / PF 1.41 (halves +38.2/+79.4); control-vs-recorded MD5 equal on both. First-look facts are descriptive only (Bollinger opposes 94% of M5 entries; divergence-opposed M5 trades did best again; every indicator's SELL votes were followed by moves against them on both TFs; the bar-level baseline drifts +1.9 ATR/24 h on M5 so BUY votes must be drift-adjusted first) — every candidate change stays a rule-1 rerun with a control book. -- 2026-09-09 weight-change test (`backtest/research/weight_test_2026-09-09.md`, first use of the recorded replay files): a drift-adjusted per-bar screen ranked 108 regime-multiplier candidates per TF; the top 3 per TF (24 h-sign filter) went through the full 2-year replay against same-harness controls plus the 2024-06→2025-03 OOS window under a rule pre-registered before any run. M5: all three MACD candidates FAIL (−10 to −46R; the screen ranked them first — never trust the per-bar screen on M5 without a replay). M15: ema_trend × 2 FAILS (PF 1.32, DD 26R, +55% trades), divergence → 0 in trending FAILS (H1 −3R, DD); **RSI multiplier × 0.5 in every regime PASSES a–f** (2-yr 793/+117.6R/PF 1.41 → 931/+146.1R/PF 1.43, halves +27.9/+0.6, OOS +40.6 → +60.7R PF 1.49, 3-yr +151.9 → +186.6R PF 1.39 but max DD 15.5 → 19.3R and 2026 −4R; added trades are mostly B-grade BUYs). NOT shipped — owner decision pending; if shipped it is an M15-only `regime_weights_override` in the factory (rule 2) plus a re-pin. +- 2026-09-09 weight-change test (`backtest/research/weight_test_2026-09-09.md`, first use of the recorded replay files): a drift-adjusted per-bar screen ranked 108 regime-multiplier candidates per TF; the top 3 per TF (24 h-sign filter) went through the full 2-year replay against same-harness controls plus the 2024-06→2025-03 OOS window under a rule pre-registered before any run. M5: all three MACD candidates FAIL (−10 to −46R; the screen ranked them first — never trust the per-bar screen on M5 without a replay). M15: ema_trend × 2 FAILS (PF 1.32, DD 26R, +55% trades), divergence → 0 in trending FAILS (H1 −3R, DD); **RSI multiplier × 0.5 in every regime PASSES a–f** (2-yr 793/+117.6R/PF 1.41 → 931/+146.1R/PF 1.43, halves +27.9/+0.6, OOS +40.6 → +60.7R PF 1.49, 3-yr +151.9 → +186.6R PF 1.39 but max DD 15.5 → 19.3R and 2026 −4R; added trades are mostly B-grade BUYs). SHIPPED the same day on the owner's decision as an M15-only `technical.regime_weights_override` in `pair_signals/xauusd.py` (M15 override 5: the shipped table with RSI halved in every regime — 0.5 trending family / 0.75 ranging / 0.5 transitioning; derived from the default table, pinned literally by the Module 9 equivalence test) — the factory run without `--set` reproduces the validated candidate book byte-for-byte (parity MD5 in `weight_test_2026-09-09.md`); M15 canonical book re-pinned to `backtest/oanda_m15_2yr_065_20260909.csv`. Live M15 `engine_fingerprint` changes from that deploy; signal-log comparisons across it must split at the deploy time. Suite 499 tests. diff --git a/backtest/research/weight_test_2026-09-09.md b/backtest/research/weight_test_2026-09-09.md index e1ba33c..4459cee 100644 --- a/backtest/research/weight_test_2026-09-09.md +++ b/backtest/research/weight_test_2026-09-09.md @@ -171,3 +171,17 @@ recent one does not, and the 3-year max drawdown is 3.8R deeper (all of it in th (931 / +146.14R / PF 1.43 at $0.65). Not shipped in this campaign. Books and logs: `backtest/research/replay_2026-09/candidates/` and `.../oos/` (gitignored). + +## Shipped (owner decision 2026-09-09, same day) + +M15 override 5 in `pair_signals/xauusd.py`: `technical.regime_weights_override` = the shipped +table with the RSI multiplier halved in every regime (0.5 trending family / 0.75 ranging / +0.5 transitioning), derived from the default table and pinned literally by the Module 9 +factory-equivalence test; one more Module 33 test locks that M5/H1/H4 carry no override. +Parity proof — the factory run with NO `--set` flag reproduces the validated candidate books +byte-for-byte: 2-year MD5 `d185f099482a07f29502cb5a371a9bbf` (931 / +146.14R / PF 1.43) and +OOS MD5 `6f8f24b49597ced28f50a19e08c3472f` (355 / +60.67R / PF 1.49). Canonical M15 book +re-pinned to `backtest/oanda_m15_2yr_065_20260909.csv` (copy of the 2-year factory book). +Docs: `docs/user_guide.html` and `docs/beginner_guide.html` carry a one-line note (PDF builds +remain behind, as before). Suite 499 tests. Live M15 scans get a new `engine_fingerprint` +from the next deploy; signal-log comparisons across it must split at the deploy time. diff --git a/docs/beginner_guide.html b/docs/beginner_guide.html index d3df26f..c5b4dfd 100644 --- a/docs/beginner_guide.html +++ b/docs/beginner_guide.html @@ -138,6 +138,7 @@ The message says: "once price hits 4,159, trail SL $33 behind best price."H1 trades are held longer — often overnight, sometimes over days. That is normal. Your stop loss is working for you while you sleep.
  • H1 is BUY-only. All H1 SELL signals are blocked (since 2026-09-03) because they were a losing side on this timeframe — expect to see BUY signals here, never SELL.
  • M5 also holds back some SELL signals — specifically while the market is in a choppy "transitioning" phase, since those shorts tested poorly.
    • +
  • M15 got a small tune-up on 2026-09-09 — one of its nine indicators (RSI) now counts for half as much, which means a few more M15 signals per week. Nothing changes in how you act on a signal.
  • Never take the same signal on two timeframes. They are usually the same gold move seen at different zoom levels. Taking both doubles your risk, not your profit.
    • diff --git a/docs/user_guide.html b/docs/user_guide.html index 3b76397..9dc17f9 100644 --- a/docs/user_guide.html +++ b/docs/user_guide.html @@ -98,7 +98,7 @@
    If you trade manually from the Telegram alerts, H1 (or M15) is the honest choice. M5's backtested edge assumes near-instant fills that a human copying signals rarely achieves.
    diff --git a/pair_signals/xauusd.py b/pair_signals/xauusd.py index b099158..bc79318 100644 --- a/pair_signals/xauusd.py +++ b/pair_signals/xauusd.py @@ -90,6 +90,23 @@ metric. One clause waived BY OWNER DECISION 2026-07-21: 5yr max DD is accepted in exchange for +20% total R. M5 keeps the global 0.6 arm (same per-TF divergence pattern as tp_floor_rr and the hour-9 block). +M15 override 5 — RSI regime multiplier x 0.5 in every regime +(validated 2026-09-09, 2-year Oanda window 2024-09-05 -> 2026-09-05 at +$0.65, pre-registered rule in backtest/research/ +weight_test_2026-09-09.md): 793/+117.62R PF 1.41 DD 14.8R -> +931/+146.14R PF 1.43 DD 16.1R, fixed halves +27.9/+0.6R vs control; +OOS 2024-06-01 -> 2025-03-30 300/+40.59R PF 1.38 -> 355/+60.67R +PF 1.49; 3-year 1092/+151.90R PF 1.38 -> 1273/+186.61R PF 1.39 (max DD +15.5 -> 19.3R, all in the 2023-24 year; 2026 -4R is the one calendar +year that does not improve). Mechanism: RSI is neutral on ~73% of M15 +entries and votes against the trend when it fires; halving it lets more +trend-following entries clear the score band (210 added trades +26.8R, +mostly B-grade BUYs; 72 removed -7.1R). Candidate came from the +recorded replay screen (replay_recording_2026-09-09.md); the same +screen's M5 picks (MACD-down) all FAILED the replay, and ema_trend x 2 +/ divergence -> 0 on M15 failed too. Canonical M15 book re-pinned to +backtest/oanda_m15_2yr_065_20260909.csv. + M5 override (validated 2026-07-11, full-history replay 2020-03-17 -> 2026-07-11 + 12mo book, $0.30 spread): block hour 9 UTC (post-London- open whipsaw). Hour-9 entries lost -28.9R/143 over 6 years, negative @@ -210,8 +227,8 @@ bollinger all look mis-weighted in-book and all replay worse when oppose vote is a load-bearing quality bar, like ROC). """ -from typing import Optional, List -from config import PlatformConfig, Timeframe +from typing import Dict, Optional, List +from config import PlatformConfig, TechnicalConfig, Timeframe from app_config import apply_pair_overrides _CONFIRMATIONS = { @@ -222,6 +239,19 @@ _CONFIRMATIONS = { } +def _m15_regime_weights() -> Dict[str, Dict[str, float]]: + """Shipped regime weight table with the RSI multiplier halved in + every regime — the M15-only override validated 2026-09-09 (see the + module docstring, M15 override 5). Derived from the default table so + the intent ("RSI x 0.5, everything else as shipped") stays explicit; + the Module 9 equivalence test pins the resulting numbers.""" + from engine import regime_weight_table # local import: keep the factory import-light + table = regime_weight_table(TechnicalConfig()) # a fresh TechnicalConfig has no override -> the shipped table + return {regime: {name: (mult * 0.5 if name == "rsi" else mult) + for name, mult in weights.items()} + for regime, weights in table.items()} + + def make_config( primary_tf: Timeframe, confirmation_tfs: Optional[List[Timeframe]] = None, @@ -272,6 +302,9 @@ def make_config( # Later trailing-stop arm on M15 only — see module docstring. config.signals.trail_arm_r = 0.70 config.signals.trail_dist_r = 0.75 + # RSI regime multiplier × 0.5 in every regime on M15 only — see + # module docstring (M15 override 5, validated 2026-09-09). + config.technical.regime_weights_override = _m15_regime_weights() if primary_tf == Timeframe.H1: # Block every H1 SELL — the SELL quality gate (6e in # signal_aggregator.py) downgrades any actionable-grade short diff --git a/test_all.py b/test_all.py index 2fda1d3..4fa7fbf 100644 --- a/test_all.py +++ b/test_all.py @@ -2078,6 +2078,35 @@ def _(): inline_cfg.xau_strategy.sell_blocked_hours_utc = [15, 17] inline_cfg.signals.trail_arm_r = 0.70 inline_cfg.signals.trail_dist_r = 0.75 + # M15 override 5 (2026-09-09): RSI x 0.5 in every regime — + # LITERAL so a silent default-table change fails loudly here. + inline_cfg.technical.regime_weights_override = { + "trending_up": { + "ema_trend": 2.0, "rsi": 0.5, "macd": 2.0, + "bollinger": 0.5, "keltner": 1.5, "roc": 1.5, + "candlestick": 0.8, "divergence": 1.0, "adx": 0.8, + }, + "trending_down": { + "ema_trend": 2.0, "rsi": 0.5, "macd": 2.0, + "bollinger": 0.5, "keltner": 1.5, "roc": 1.5, + "candlestick": 0.8, "divergence": 1.0, "adx": 0.8, + }, + "trending": { + "ema_trend": 2.0, "rsi": 0.5, "macd": 2.0, + "bollinger": 0.5, "keltner": 1.5, "roc": 1.5, + "candlestick": 0.8, "divergence": 1.0, "adx": 0.8, + }, + "ranging": { + "ema_trend": 0.3, "rsi": 0.75, "macd": 1.5, + "bollinger": 1.5, "keltner": 2.0, "roc": 0.5, + "candlestick": 1.0, "divergence": 2.5, "adx": 0.5, + }, + "transitioning": { + "ema_trend": 1.5, "rsi": 0.5, "macd": 2.5, + "bollinger": 1.0, "keltner": 2.5, "roc": 2.0, + "candlestick": 1.0, "divergence": 2.0, "adx": 1.0, + }, + } if tf == Timeframe.M5: # Documented per-TF overrides (see xauusd.py docstring): # hour 9 UTC blocked, hour 21 UNBLOCKED (2026-07-30 @@ -11861,6 +11890,37 @@ def _(): "the GPU engine must inherit the live composite, never shadow it" +@test("replay record follow-up: M15 factory halves RSI in every regime, other TFs carry no override") +def _(): + # M15 override 5 (validated 2026-09-09): the M15 factory ships a + # regime_weights_override that is the shipped table with the RSI + # multiplier halved everywhere and every other cell untouched. + import engine as _eng + from pair_signals import PAIR_PRODUCERS + default = _eng.regime_weight_table(TechnicalConfig()) + m15 = PAIR_PRODUCERS[XAUUSD](Timeframe.M15, [Timeframe.M5, Timeframe.H1]) + override = m15.technical.regime_weights_override + assert set(override) == set(default), \ + f"M15 override must cover exactly the shipped regimes: {sorted(override)}" + for regime, weights in default.items(): + got = override[regime] + assert set(got) == set(weights), \ + f"{regime}: indicator set changed: {sorted(got)}" + for name, mult in weights.items(): + want = mult * 0.5 if name == "rsi" else mult + assert got[name] == want, \ + f"{regime}.{name}: {got[name]} != {want} (default {mult})" + # The engine really runs with it (regime_weight_table prefers the override). + assert _eng.regime_weight_table(m15.technical) == override + # ...and it is M15-only — same per-TF pattern as every other override. + for tf, ctfs in [(Timeframe.M5, [Timeframe.M15, Timeframe.H1]), + (Timeframe.H1, [Timeframe.M15, Timeframe.H4]), + (Timeframe.H4, [Timeframe.H1, Timeframe.D1])]: + cfg = PAIR_PRODUCERS[XAUUSD](tf, ctfs) + assert cfg.technical.regime_weights_override == {}, \ + f"{tf} must carry no regime weight override" + + # ══════════════════════════════════════════════════════════════════════ # SUMMARY # ══════════════════════════════════════════════════════════════════════