From 9d7f7ff4ec45fc471405812c4a7693c63b2032a1 Mon Sep 17 00:00:00 2001 From: Harwinder Sekhon Date: Wed, 9 Sep 2026 17:26:21 -0400 Subject: [PATCH] research: weight-change test on the recorded replay files (pre-registered; M15 RSI x0.5 passes, not shipped) First use of the --record evidence: a drift-adjusted per-bar screen ranked regime-multiplier candidates, the top 3 per timeframe went through the full 2-year replay against same-harness controls plus the 2024-06..2025-03 OOS window under a rule written before any run. M5: all three MACD candidates fail (-10..-46R). M15: ema_trend x2 and divergence->0 (trending) fail; RSI x0.5 in every regime passes every check (2-yr +117.6 -> +146.1R PF 1.43, OOS +40.6 -> +60.7R PF 1.49, 3-yr +151.9 -> +186.6R) with the caveats recorded (2026 -4R, 3-yr max DD 15.5 -> 19.3R, added trades mostly B-grade BUYs). Nothing shipped; the passing candidate is an owner decision. Claude-Session: https://claude.ai/code/session_01Nc9Ninzyi18QkWezrxeBvB --- CLAUDE.md | 1 + backtest/research/weight_test_2026-09-09.md | 173 ++++++++++++++++++++ 2 files changed, 174 insertions(+) create mode 100644 backtest/research/weight_test_2026-09-09.md diff --git a/CLAUDE.md b/CLAUDE.md index 60509e3..f27fc72 100644 --- a/CLAUDE.md +++ b/CLAUDE.md @@ -113,3 +113,4 @@ Signal currency everywhere is `SignalStrength` (−2..+2). DataFrames are UTC tz - 2026-09-06 stale-code cleanup (behaviour-neutral; suite green, M5/H1 12-month books byte-identical after every signal-side commit): `main.py --timeframe` now accepts M5/M15/H1/H4 and takes its confirmation TFs from the pair factory (the CLI's private table had paired M5 with M1+M15); `PlatformConfig.pip_value` is gone (`PAIR_INFO` is the source); removed as dead or superseded: `_get_affected_pairs`/`event_impact_on_pairs`, `SignalLogger.advance_with_bars`/`path`/`news_for`, `reset_llm_client`, the Telegram scan-summary/keyboard helpers and duplicate confirmation map, the 2026-07 Oanda-vs-TD research pager+study, `burst_profile.py`, `hour_analysis.py`, `scripts/compare_oanda_tiingo.py`, the never-implemented LLM-provider plan and the merged config.json spec. Second pass the same day (owner directive: keep only what the system needs): every research script and data file under `backtest/research/` (book_report, book_diff, the burst tools, sizing_model, commit_ladder, llm_news_eval + golden CSV, ff_calendar_scraper, calendar_impact, watch_rule_eval, the Dukascopy spread study + CSVs, the panel dry-test JSONL), the two completed study scripts in `scripts/`, the architecture diagram + generator and the data-source test plan were deleted; the campaign `.md` reports stay as the decision ledger. Git history before `953c5ad` has all of it. Third pass (owner: "remove dead functions as well"): the write-only pb/cp rejection counters, the unreachable `_TP_QUALITY_SCALE` B/C entries, the engine's unused DataFrame argument and M1 arm, the providers' generic-symbol guesses (unknown symbols now raise), the sub-$10 price format, the `=X` pair parsing and the unread `minutes_until`/`is_high_impact` news.json keys are gone. - 2026-09-08 news archive shipped (`news_archive.py`, 483 tests; nothing in the signal core changed, both news gating flags stay `False`, books untouched). Motivation: headlines only ever survived as a side effect of signal logging and `signals.db` was rotated around 2026-09-04, destroying the 3-week / 4,054-headline corpus the news study was written on — with `logs/` gitignored, git could not recover it and nobody noticed for four days. Day-one corpus after backfilling both DBs on disk: **1,016 distinct headlines, 846 of them DeepSeek-labelled**, growing ~200/day (~1,400/week). The point of collecting it: distil DeepSeek's headline scoring (78/78 on the 2026-08-06 golden set) into a numpy-only local model so the fallback stops being the keyword scorer (**24/78 = 31%**, which is what silently runs on no key / disabled / daily cap / timeout / HTTP or parse failure / batch overflow). **Do not train yet** — 800-odd labels gives a holdout too small to grade (SE ±0.032); the pre-registered plan is ~4,000 deduplicated headlines over ≥3 weeks, time-split with near-duplicates clustered first (the same story is re-titled by several outlets, so a random split scores memorisation), headline text ONLY (the teacher never sees `description` — `_llm_headline_scores` sends titles), graded against the golden set recoverable with `git show 1cebebf^:backtest/research/llm_news_golden.csv`. When it ships it slots in as DeepSeek → model → keyword and stays display-only. Training happens offline in `venv/`; a Module 32 test now AST-walks the root, `providers/` and `pair_signals/` asserting no `sklearn`/`scipy`/`lightgbm`/`torch` import and that none of them reach `requirements.txt` — that guard did not exist before. - 2026-09-09 replay recording shipped (`backtest/replay_record.py`, `backtest/replay_scorecard.py`, Module 33, 498 tests; nothing in the signal core changed, books byte-identical — `backtest/research/replay_recording_2026-09-09.md`). `--record DIR` stores every ACCEPTED trade's decision evidence in Parquet: each indicator's vote / value / confidence / BASE weight / regime multiplier / EFFECTIVE weight / contribution on the primary AND both confirmation TFs (`indicators.parquet`, long; absence = the indicator did not fire, never a zero row), per-TF regime/composite, penalty tags, setup, outcome, and the Forex Factory calendar events from 4 h before entry to exit (`events.parquet`, all-day rows matched on the New-York date; scalar `ev_*`/`us_holiday` columns on `trades.parquet`); `--record-bars` adds one float32 row per ANALYZED bar with the same votes plus 1 h/4 h/24 h forward moves in ATRs (`bars.parquet`). The manifest carries the engine + gate fingerprints, the regime weight table in force and the book MD5. Calendar source is the gitignored `backtest/cache/calendar.parquet` (15,124 FF events 2023-09-01→2026-09-10, unix-UTC `dateline`; `timeLabel` is display-only); the scraper that built it is back at `scripts/ff_calendar_scraper.py` (resume + idempotent merge, ~1.6 s/request). The 2-year books (2024-09-05→2026-09-05, $0.65) live in `backtest/research/replay_2026-09/{m5,m15}/`: M5 781 / +146.07R / PF 1.52 (halves +28.0/+118.1), M15 793 / +117.62R / PF 1.41 (halves +38.2/+79.4); control-vs-recorded MD5 equal on both. First-look facts are descriptive only (Bollinger opposes 94% of M5 entries; divergence-opposed M5 trades did best again; every indicator's SELL votes were followed by moves against them on both TFs; the bar-level baseline drifts +1.9 ATR/24 h on M5 so BUY votes must be drift-adjusted first) — every candidate change stays a rule-1 rerun with a control book. +- 2026-09-09 weight-change test (`backtest/research/weight_test_2026-09-09.md`, first use of the recorded replay files): a drift-adjusted per-bar screen ranked 108 regime-multiplier candidates per TF; the top 3 per TF (24 h-sign filter) went through the full 2-year replay against same-harness controls plus the 2024-06→2025-03 OOS window under a rule pre-registered before any run. M5: all three MACD candidates FAIL (−10 to −46R; the screen ranked them first — never trust the per-bar screen on M5 without a replay). M15: ema_trend × 2 FAILS (PF 1.32, DD 26R, +55% trades), divergence → 0 in trending FAILS (H1 −3R, DD); **RSI multiplier × 0.5 in every regime PASSES a–f** (2-yr 793/+117.6R/PF 1.41 → 931/+146.1R/PF 1.43, halves +27.9/+0.6, OOS +40.6 → +60.7R PF 1.49, 3-yr +151.9 → +186.6R PF 1.39 but max DD 15.5 → 19.3R and 2026 −4R; added trades are mostly B-grade BUYs). NOT shipped — owner decision pending; if shipped it is an M15-only `regime_weights_override` in the factory (rule 2) plus a re-pin. diff --git a/backtest/research/weight_test_2026-09-09.md b/backtest/research/weight_test_2026-09-09.md new file mode 100644 index 0000000..e1ba33c --- /dev/null +++ b/backtest/research/weight_test_2026-09-09.md @@ -0,0 +1,173 @@ +# Weight-change test campaign — 2026-09-09 (does the recorded evidence improve predictions?) + +## Purpose + +First use of the replay-recording files (`replay_recording_2026-09-09.md`): take candidate +changes to the regime weight table suggested by the per-bar evidence, rerun the FULL backtest +with each, and compare against the same-harness control books. This is the "before versus +after" that measures improvement. Everything below the rule was written before any candidate +was run. + +## Harness (identical to the recording campaign) + +`backtest/gpu_backtest.py --pair XAUUSD --start 2024-09-05 --end 2026-09-05 --spread 0.65 +--workers N --primary-tf {5min|15min}` + `--set technical.regime_weights_override=` +(the field replaces the whole table, so every candidate passes the complete table with one +cell family changed). Controls: `backtest/research/replay_2026-09/{m5,m15}/book_control.csv` +(same code, no override). OOS window: `--start 2024-06-01 --end 2025-03-30` (cached +`oanda_XAU_USD_*_20240331_20250330.parquet`), control produced on the same harness. +Backend NumPy CPU. Code: git `af06685`. + +## Pre-registered candidate-selection rule (written 2026-09-09 before the screen ran) + +1. A candidate changes ONE indicator's regime multipliers only (no code, no gate, no + threshold change), on ONE timeframe. Allowed forms: multiplier → 0 in all regimes, + × 0.5 in all regimes, × 2 in all regimes, or the same applied to a single regime family + (trending / ranging / transitioning) when the screen shows the effect is regime-specific. +2. The screen (scratch script over `bars.parquet`, both TFs): for each candidate, recompute + every analyzed bar's PRIMARY composite from the recorded effective weights and + contributions, find the bars that would NEWLY clear ±15 and the bars that would DROP OUT, + and score the candidate by the net drift-adjusted signed 4-hour forward move + (gained bars minus lost bars; drift = the mean unsigned 4-hour move of all analyzed bars + in the same regime). Top 3 per timeframe by that score are run; nothing else. +3. Earlier verdicts on the same knob are recorded next to the candidate but do not veto it + (this harness — 2-year Oanda window, post-2026-09-07 outlier rule — is new). + +## Pre-registered acceptance rule (vs the same-harness control) + +A candidate PASSES only if ALL of the following hold on the 2-year window at $0.65: +- a. ΔR ≥ +5R overall versus the control (smaller differences are noise on ~800 trades); +- b. ΔR > 0 in BOTH fixed halves: 2024-09-05→2025-09-05 and 2025-09-05→2026-09-05; +- c. PF ≥ 1.3 and not more than 0.05 below the control's PF; +- d. max drawdown not more than 2R worse than the control's; +- e. trade count within ±25% of the control's (guards against "fewer trades" artefacts); +- f. OOS window 2024-06-01→2025-03-30: ΔR ≥ 0 versus the OOS control. +Failing candidates are recorded and closed. No rescue: no multiplier fine-tuning after +seeing a result, no second window shopping. A passing candidate is a SHIPPING PROPOSAL for +the owner (rule 2: it would land as a per-timeframe `regime_weights_override` in +`pair_signals/xauusd.py` with tests), not an automatic change. + +## Screen results (scratch script over `bars.parquet`, both TFs; composite reproduced from the stored columns within 3.4e-6 on every bar) + +Drift per regime (mean 4 h move of all analyzed bars, ATR units, M5): trending_up +0.55, +trending_down +0.16, trending +0.35, transitioning +0.64, ranging +0.38 — a strong up-drift +that every BUY vote inherits, hence the per-regime adjustment. + +Standalone votes after drift removal (mean signed 4 h move; positive = the vote pointed the +right way), M5 highlights: + +| indicator | regime family | buy votes | sell votes | reading | +|---|---|---|---|---| +| macd | trending | −0.079 (n 8,717) | −0.154 (n 8,891) | wrong way on BOTH sides; also at 24 h (−0.22 / −0.56) | +| macd | transitioning | +0.273 (1,738) | +0.288 (1,885) | best caller in that regime | +| rsi | transitioning | −1.089 (357) | −0.146 (570) | wrong way, small n | +| bollinger | transitioning | −0.261 (2,668) | −0.337 (2,675) | wrong way | +| bollinger | trending | +0.084 | +0.091 | mildly right | +| ema_trend | ranging / transitioning | +0.33 / +0.42 | ≈0 | buy side right, sell side nothing | +| adx | trending | +0.048 | +0.041 | never fires outside trending | + +Candidate screen (net drift-adjusted 4 h move of gained minus lost bars; the 24 h column must +agree in sign for a candidate to be run). No candidate flips a bar's direction — one +indicator's weight only moves bars in and out of the neutral band. + +M5 top of the board: `macd:zero_trending` +1235 (24 h +4181, agrees; 3,406 gained / 7,339 lost, +touches 23% of directional bars), `macd:double_ranging` +1186 (24 h −21, rejected), +`divergence:zero_all` +1026 (24 h −633, rejected), `bollinger:zero_ranging` +940 (24 h −728, +rejected), `bollinger:zero_all` +800 (24 h −1654, rejected), `macd:zero_all` +705 (24 h +1644, +agrees; touches 50% of directional bars), `macd:half_trending` +689 (24 h +334, agrees). +Bottom: `ema_trend:zero_all` −389, `macd:double_trending` −540, `macd:zero_transitioning` −607. + +M15 top of the board: `ema_trend:double_all` +329 (24 h +1209, agrees), `macd:zero_all` +293 +(24 h −79, rejected), `divergence:zero_all` +264 (24 h −480, rejected), +`divergence:zero_trending` +246 (24 h +390, agrees), `rsi:half_all` +221 (24 h +99, agrees). +Bottom: `macd:double_trending` −337, `macd:double_all` −323. + +Candidates run (rule 2 above, 24 h-sign filter applied): +- M5-1 `macd:zero_trending` — MACD multiplier 2.0 → 0 in trending_up/trending_down/trending. +- M5-2 `macd:zero_all` — MACD multiplier → 0 in every regime. +- M5-3 `macd:half_trending` — MACD multiplier 2.0 → 1.0 in the trending family. +- M15-1 `ema_trend:double_all` — ema_trend multiplier × 2 in every regime. +- M15-2 `divergence:zero_trending` — divergence multiplier 1.0 → 0 in the trending family. +- M15-3 `rsi:half_all` — RSI multiplier × 0.5 in every regime. +Prior verdicts on the same knobs: none for MACD or ema_trend; "divergence-down" on M5 was +rejected 2026-08-04 and "divergence-off" on M15 was declined 2026-09 (R up, 5-yr DD 35.8 → +47.4R); neither is the same change as M15-2 (trending family only). + +Half of each board's top entries fail the 24 h check, some violently (`bollinger:zero_all` on +M5: +800 at 4 h, −1654 at 24 h) — a 4 h-only screen is a coin flip. Scores are sums, so +volume can carry them: `macd:zero_all` scores by removing 16,170 directional bars whose mean +was −0.07, while its gained bars average −0.09. + +## Rerun results (2-year window 2024-09-05→2026-09-05, $0.65; halves fixed at 2025-09-05; OOS 2024-06-01→2025-03-30) + +| candidate | trades | total R | ΔR | H1 ΔR | H2 ΔR | PF | max DD | OOS ΔR | verdict | +|---|---|---|---|---|---|---|---|---|---| +| M5 control | 781 | +146.07 | | | | 1.52 | 10.1 | (86 / +9.47, PF 1.33) | | +| M5-1 macd → 0 in trending | 808 | +105.35 | −40.7 | +1.9 | −42.6 | 1.35 | 17.1 | +0.2 | FAIL a,b,c,d | +| M5-2 macd → 0 everywhere | 699 | +99.79 | −46.3 | +0.5 | −46.8 | 1.38 | 17.2 | −1.8 | FAIL a,b,c,d,f | +| M5-3 macd × 0.5 in trending | 804 | +136.12 | −10.0 | +7.4 | −17.4 | 1.47 | 15.2 | +6.8 | FAIL a,b,c,d | +| M15 control | 793 | +117.62 | | | | 1.41 | 14.8 | (300 / +40.59, PF 1.38) | | +| M15-1 ema_trend × 2 everywhere | 1229 | +148.20 | +30.6 | +10.0 | +20.5 | 1.32 | 26.4 | +0.7 | FAIL c,d,e (+55% trades) | +| M15-2 divergence → 0 in trending | 861 | +127.69 | +10.1 | −3.0 | +13.1 | 1.41 | 18.4 | +3.6 | FAIL b,d | +| **M15-3 rsi × 0.5 everywhere** | **931** | **+146.14** | **+28.5** | **+27.9** | **+0.6** | **1.43** | **16.1** | **+20.1 (355 / +60.67, PF 1.49)** | **PASS a–f** | + +The M5 screen was wrong in every case: the three MACD candidates that scored best on the +per-bar screen (and were the only ones whose 24 h score agreed) lose 10–46R in the full +replay, all of it in the second year. The per-bar composite screen sees threshold crossings; +it does not see that MACD's "wrong-way" votes in trending regimes are doing useful work +downstream (the exhaustion penalties, the M5 min-score gate and the setup generators all sit +between the composite and a trade). Lesson recorded: the screen ranks hypotheses, the replay +judges them, and on M5 the two disagreed 3/3. + +### M15-3 trade diff (control → candidate, 2-year) + +- 721 shared entries: control +124.7R → candidate +119.4R (56 shared trades changed R because + the score moved their grade and therefore the setup distances). +- 210 added trades: +26.8R, mean +0.127R, WR 59.5% — 157 BUY (+32.5R) and 53 SELL (−5.8R); + 156 are B-grade (+19.1R), 201 sit in the 25–40 score band; by regime ranging 63 (+13.1R), + trending_up 88 (+12.1R), transitioning 32 (+2.9R), trending_down 27 (−1.3R). +- 72 removed trades: −7.1R, WR 50%. +- By year: 2024 (Sep–Dec) +19.4 → +26.7R, 2025 +51.9 → +77.4R, 2026 (Jan–Sep) +46.3 → +42.1R. + The 2-year gain is concentrated in 2025; the fixed second half passes rule b by only +0.6R. +- OOS (2024-06→2025-03): 300 → 355 trades, +40.6 → +60.7R, PF 1.38 → 1.49, WR 59.3 → 60.8%. + +Mechanism in plain terms: RSI is neutral on most M15 entries and, when it votes, it votes +against the trend (a mean-reversion reading). Halving its weight lets more trend-following +entries clear the score band, mostly BUYs, in a data set that is a gold bull market +throughout. The OOS window improves in quality (PF, win rate), not only in count, which is +the strongest single fact in the candidate's favour; the year-by-year unevenness is the +strongest fact against. + +### Robustness check on the 3-year window (owner's maximum; 2023-09-05→2026-09-05, cached bars) + +| book | trades | total R | WR | PF | max DD | +|---|---|---|---|---|---| +| M15 control 3-yr | 1092 | +151.90 | 59.4% | 1.38 | 15.5 | +| M15-3 rsi × 0.5, 3-yr | 1273 | +186.61 | 59.4% | 1.39 | 19.3 | +| extra year only (2023-09-05→2024-09-05, never seen by the screen): control | 299 | +33.23 | 56.9% | 1.29 | 14.9 | +| extra year only: candidate | 342 | +39.43 | 56.4% | 1.29 | 19.3 | + +Calendar years, control → candidate: 2023 (Sep–Dec) +10.1 → +13.2R; 2024 +42.6 → +52.9R; +2025 +51.9 → +77.4R; 2026 (Jan–Sep) +47.3 → +43.2R. Three of four years improve; the most +recent one does not, and the 3-year max drawdown is 3.8R deeper (all of it in the 2023-24 year). + +## Verdict + +- **The process works end to end**: recorded evidence → screen → pre-registered rule → full + replay against same-harness controls → one pass, five fails, each with the reason recorded. +- **M5**: all three MACD candidates FAIL (−10 to −46R). Closed. The per-bar screen is not to + be trusted on M5 without a replay; it ranked them first. +- **M15-1 (ema_trend × 2)** FAILS on PF, drawdown and trade count; **M15-2 (divergence → 0 in + trending)** FAILS the first half and the drawdown. Both closed. +- **M15-3 (RSI multiplier × 0.5 in every regime) PASSES the pre-registered rule** (a–f) and the + 3-year robustness check is positive (+34.7R, PF 1.38 → 1.39, extra year +6.2R). It is a + SHIPPING PROPOSAL for the owner, with these facts on the other side of the scale: the 2-year + second half passes by only +0.6R, 2026 is −4R, the added trades are mostly B-grade BUYs in a + bull-market data set (added SELLs lose), and the 3-year max drawdown deepens 15.5 → 19.3R. + If shipped it lands as an M15-only `technical.regime_weights_override` in + `pair_signals/xauusd.py` (rule 2) with the full table (RSI 1.0 → 0.5 in the trending family, + 1.5 → 0.75 in ranging, 1.0 → 0.5 in transitioning), a factory-equivalence test update and a + docs note; the M15 canonical book would then be re-pinned from `candidates/m15_rsi:half_all_2yr.csv` + (931 / +146.14R / PF 1.43 at $0.65). Not shipped in this campaign. + +Books and logs: `backtest/research/replay_2026-09/candidates/` and `.../oos/` (gitignored).