Harwinder Sekhon
be4db8b2a7
Owner-approved 2026-09-09 after the pre-registered weight test (backtest/research/weight_test_2026-09-09.md). M15-only technical.regime_weights_override = the shipped table with rsi x 0.5 (0.5 trending family / 0.75 ranging / 0.5 transitioning), derived in the factory and pinned literally by the Module 9 equivalence test; a Module 33 test locks that M5/H1/H4 carry no override. Evidence, 2-year Oanda window 2024-09-05..2026-09-05 at $0.65: 793/+117.62R PF 1.41 -> 931/+146.14R PF 1.43 (fixed halves +27.9/+0.6R vs control); OOS 2024-06-01..2025-03-30 300/+40.59R PF 1.38 -> 355/+60.67R PF 1.49; 3-year +151.90R -> +186.61R (max DD 15.5 -> 19.3R, 2026 -4R). Parity: the factory run with no --set flag reproduces the validated candidate books byte-for- byte (2-yr MD5 d185f099..., OOS 6f8f24b4...). M15 canonical book re-pinned to backtest/oanda_m15_2yr_065_20260909.csv. Docs carry a one-line note. Claude-Session: https://claude.ai/code/session_01Nc9Ninzyi18QkWezrxeBvB
188 lines
13 KiB
Markdown
188 lines
13 KiB
Markdown
# Weight-change test campaign — 2026-09-09 (does the recorded evidence improve predictions?)
|
||
|
||
## Purpose
|
||
|
||
First use of the replay-recording files (`replay_recording_2026-09-09.md`): take candidate
|
||
changes to the regime weight table suggested by the per-bar evidence, rerun the FULL backtest
|
||
with each, and compare against the same-harness control books. This is the "before versus
|
||
after" that measures improvement. Everything below the rule was written before any candidate
|
||
was run.
|
||
|
||
## Harness (identical to the recording campaign)
|
||
|
||
`backtest/gpu_backtest.py --pair XAUUSD --start 2024-09-05 --end 2026-09-05 --spread 0.65
|
||
--workers N --primary-tf {5min|15min}` + `--set technical.regime_weights_override=<full table>`
|
||
(the field replaces the whole table, so every candidate passes the complete table with one
|
||
cell family changed). Controls: `backtest/research/replay_2026-09/{m5,m15}/book_control.csv`
|
||
(same code, no override). OOS window: `--start 2024-06-01 --end 2025-03-30` (cached
|
||
`oanda_XAU_USD_*_20240331_20250330.parquet`), control produced on the same harness.
|
||
Backend NumPy CPU. Code: git `af06685`.
|
||
|
||
## Pre-registered candidate-selection rule (written 2026-09-09 before the screen ran)
|
||
|
||
1. A candidate changes ONE indicator's regime multipliers only (no code, no gate, no
|
||
threshold change), on ONE timeframe. Allowed forms: multiplier → 0 in all regimes,
|
||
× 0.5 in all regimes, × 2 in all regimes, or the same applied to a single regime family
|
||
(trending / ranging / transitioning) when the screen shows the effect is regime-specific.
|
||
2. The screen (scratch script over `bars.parquet`, both TFs): for each candidate, recompute
|
||
every analyzed bar's PRIMARY composite from the recorded effective weights and
|
||
contributions, find the bars that would NEWLY clear ±15 and the bars that would DROP OUT,
|
||
and score the candidate by the net drift-adjusted signed 4-hour forward move
|
||
(gained bars minus lost bars; drift = the mean unsigned 4-hour move of all analyzed bars
|
||
in the same regime). Top 3 per timeframe by that score are run; nothing else.
|
||
3. Earlier verdicts on the same knob are recorded next to the candidate but do not veto it
|
||
(this harness — 2-year Oanda window, post-2026-09-07 outlier rule — is new).
|
||
|
||
## Pre-registered acceptance rule (vs the same-harness control)
|
||
|
||
A candidate PASSES only if ALL of the following hold on the 2-year window at $0.65:
|
||
- a. ΔR ≥ +5R overall versus the control (smaller differences are noise on ~800 trades);
|
||
- b. ΔR > 0 in BOTH fixed halves: 2024-09-05→2025-09-05 and 2025-09-05→2026-09-05;
|
||
- c. PF ≥ 1.3 and not more than 0.05 below the control's PF;
|
||
- d. max drawdown not more than 2R worse than the control's;
|
||
- e. trade count within ±25% of the control's (guards against "fewer trades" artefacts);
|
||
- f. OOS window 2024-06-01→2025-03-30: ΔR ≥ 0 versus the OOS control.
|
||
Failing candidates are recorded and closed. No rescue: no multiplier fine-tuning after
|
||
seeing a result, no second window shopping. A passing candidate is a SHIPPING PROPOSAL for
|
||
the owner (rule 2: it would land as a per-timeframe `regime_weights_override` in
|
||
`pair_signals/xauusd.py` with tests), not an automatic change.
|
||
|
||
## Screen results (scratch script over `bars.parquet`, both TFs; composite reproduced from the stored columns within 3.4e-6 on every bar)
|
||
|
||
Drift per regime (mean 4 h move of all analyzed bars, ATR units, M5): trending_up +0.55,
|
||
trending_down +0.16, trending +0.35, transitioning +0.64, ranging +0.38 — a strong up-drift
|
||
that every BUY vote inherits, hence the per-regime adjustment.
|
||
|
||
Standalone votes after drift removal (mean signed 4 h move; positive = the vote pointed the
|
||
right way), M5 highlights:
|
||
|
||
| indicator | regime family | buy votes | sell votes | reading |
|
||
|---|---|---|---|---|
|
||
| macd | trending | −0.079 (n 8,717) | −0.154 (n 8,891) | wrong way on BOTH sides; also at 24 h (−0.22 / −0.56) |
|
||
| macd | transitioning | +0.273 (1,738) | +0.288 (1,885) | best caller in that regime |
|
||
| rsi | transitioning | −1.089 (357) | −0.146 (570) | wrong way, small n |
|
||
| bollinger | transitioning | −0.261 (2,668) | −0.337 (2,675) | wrong way |
|
||
| bollinger | trending | +0.084 | +0.091 | mildly right |
|
||
| ema_trend | ranging / transitioning | +0.33 / +0.42 | ≈0 | buy side right, sell side nothing |
|
||
| adx | trending | +0.048 | +0.041 | never fires outside trending |
|
||
|
||
Candidate screen (net drift-adjusted 4 h move of gained minus lost bars; the 24 h column must
|
||
agree in sign for a candidate to be run). No candidate flips a bar's direction — one
|
||
indicator's weight only moves bars in and out of the neutral band.
|
||
|
||
M5 top of the board: `macd:zero_trending` +1235 (24 h +4181, agrees; 3,406 gained / 7,339 lost,
|
||
touches 23% of directional bars), `macd:double_ranging` +1186 (24 h −21, rejected),
|
||
`divergence:zero_all` +1026 (24 h −633, rejected), `bollinger:zero_ranging` +940 (24 h −728,
|
||
rejected), `bollinger:zero_all` +800 (24 h −1654, rejected), `macd:zero_all` +705 (24 h +1644,
|
||
agrees; touches 50% of directional bars), `macd:half_trending` +689 (24 h +334, agrees).
|
||
Bottom: `ema_trend:zero_all` −389, `macd:double_trending` −540, `macd:zero_transitioning` −607.
|
||
|
||
M15 top of the board: `ema_trend:double_all` +329 (24 h +1209, agrees), `macd:zero_all` +293
|
||
(24 h −79, rejected), `divergence:zero_all` +264 (24 h −480, rejected),
|
||
`divergence:zero_trending` +246 (24 h +390, agrees), `rsi:half_all` +221 (24 h +99, agrees).
|
||
Bottom: `macd:double_trending` −337, `macd:double_all` −323.
|
||
|
||
Candidates run (rule 2 above, 24 h-sign filter applied):
|
||
- M5-1 `macd:zero_trending` — MACD multiplier 2.0 → 0 in trending_up/trending_down/trending.
|
||
- M5-2 `macd:zero_all` — MACD multiplier → 0 in every regime.
|
||
- M5-3 `macd:half_trending` — MACD multiplier 2.0 → 1.0 in the trending family.
|
||
- M15-1 `ema_trend:double_all` — ema_trend multiplier × 2 in every regime.
|
||
- M15-2 `divergence:zero_trending` — divergence multiplier 1.0 → 0 in the trending family.
|
||
- M15-3 `rsi:half_all` — RSI multiplier × 0.5 in every regime.
|
||
Prior verdicts on the same knobs: none for MACD or ema_trend; "divergence-down" on M5 was
|
||
rejected 2026-08-04 and "divergence-off" on M15 was declined 2026-09 (R up, 5-yr DD 35.8 →
|
||
47.4R); neither is the same change as M15-2 (trending family only).
|
||
|
||
Half of each board's top entries fail the 24 h check, some violently (`bollinger:zero_all` on
|
||
M5: +800 at 4 h, −1654 at 24 h) — a 4 h-only screen is a coin flip. Scores are sums, so
|
||
volume can carry them: `macd:zero_all` scores by removing 16,170 directional bars whose mean
|
||
was −0.07, while its gained bars average −0.09.
|
||
|
||
## Rerun results (2-year window 2024-09-05→2026-09-05, $0.65; halves fixed at 2025-09-05; OOS 2024-06-01→2025-03-30)
|
||
|
||
| candidate | trades | total R | ΔR | H1 ΔR | H2 ΔR | PF | max DD | OOS ΔR | verdict |
|
||
|---|---|---|---|---|---|---|---|---|---|
|
||
| M5 control | 781 | +146.07 | | | | 1.52 | 10.1 | (86 / +9.47, PF 1.33) | |
|
||
| M5-1 macd → 0 in trending | 808 | +105.35 | −40.7 | +1.9 | −42.6 | 1.35 | 17.1 | +0.2 | FAIL a,b,c,d |
|
||
| M5-2 macd → 0 everywhere | 699 | +99.79 | −46.3 | +0.5 | −46.8 | 1.38 | 17.2 | −1.8 | FAIL a,b,c,d,f |
|
||
| M5-3 macd × 0.5 in trending | 804 | +136.12 | −10.0 | +7.4 | −17.4 | 1.47 | 15.2 | +6.8 | FAIL a,b,c,d |
|
||
| M15 control | 793 | +117.62 | | | | 1.41 | 14.8 | (300 / +40.59, PF 1.38) | |
|
||
| M15-1 ema_trend × 2 everywhere | 1229 | +148.20 | +30.6 | +10.0 | +20.5 | 1.32 | 26.4 | +0.7 | FAIL c,d,e (+55% trades) |
|
||
| M15-2 divergence → 0 in trending | 861 | +127.69 | +10.1 | −3.0 | +13.1 | 1.41 | 18.4 | +3.6 | FAIL b,d |
|
||
| **M15-3 rsi × 0.5 everywhere** | **931** | **+146.14** | **+28.5** | **+27.9** | **+0.6** | **1.43** | **16.1** | **+20.1 (355 / +60.67, PF 1.49)** | **PASS a–f** |
|
||
|
||
The M5 screen was wrong in every case: the three MACD candidates that scored best on the
|
||
per-bar screen (and were the only ones whose 24 h score agreed) lose 10–46R in the full
|
||
replay, all of it in the second year. The per-bar composite screen sees threshold crossings;
|
||
it does not see that MACD's "wrong-way" votes in trending regimes are doing useful work
|
||
downstream (the exhaustion penalties, the M5 min-score gate and the setup generators all sit
|
||
between the composite and a trade). Lesson recorded: the screen ranks hypotheses, the replay
|
||
judges them, and on M5 the two disagreed 3/3.
|
||
|
||
### M15-3 trade diff (control → candidate, 2-year)
|
||
|
||
- 721 shared entries: control +124.7R → candidate +119.4R (56 shared trades changed R because
|
||
the score moved their grade and therefore the setup distances).
|
||
- 210 added trades: +26.8R, mean +0.127R, WR 59.5% — 157 BUY (+32.5R) and 53 SELL (−5.8R);
|
||
156 are B-grade (+19.1R), 201 sit in the 25–40 score band; by regime ranging 63 (+13.1R),
|
||
trending_up 88 (+12.1R), transitioning 32 (+2.9R), trending_down 27 (−1.3R).
|
||
- 72 removed trades: −7.1R, WR 50%.
|
||
- By year: 2024 (Sep–Dec) +19.4 → +26.7R, 2025 +51.9 → +77.4R, 2026 (Jan–Sep) +46.3 → +42.1R.
|
||
The 2-year gain is concentrated in 2025; the fixed second half passes rule b by only +0.6R.
|
||
- OOS (2024-06→2025-03): 300 → 355 trades, +40.6 → +60.7R, PF 1.38 → 1.49, WR 59.3 → 60.8%.
|
||
|
||
Mechanism in plain terms: RSI is neutral on most M15 entries and, when it votes, it votes
|
||
against the trend (a mean-reversion reading). Halving its weight lets more trend-following
|
||
entries clear the score band, mostly BUYs, in a data set that is a gold bull market
|
||
throughout. The OOS window improves in quality (PF, win rate), not only in count, which is
|
||
the strongest single fact in the candidate's favour; the year-by-year unevenness is the
|
||
strongest fact against.
|
||
|
||
### Robustness check on the 3-year window (owner's maximum; 2023-09-05→2026-09-05, cached bars)
|
||
|
||
| book | trades | total R | WR | PF | max DD |
|
||
|---|---|---|---|---|---|
|
||
| M15 control 3-yr | 1092 | +151.90 | 59.4% | 1.38 | 15.5 |
|
||
| M15-3 rsi × 0.5, 3-yr | 1273 | +186.61 | 59.4% | 1.39 | 19.3 |
|
||
| extra year only (2023-09-05→2024-09-05, never seen by the screen): control | 299 | +33.23 | 56.9% | 1.29 | 14.9 |
|
||
| extra year only: candidate | 342 | +39.43 | 56.4% | 1.29 | 19.3 |
|
||
|
||
Calendar years, control → candidate: 2023 (Sep–Dec) +10.1 → +13.2R; 2024 +42.6 → +52.9R;
|
||
2025 +51.9 → +77.4R; 2026 (Jan–Sep) +47.3 → +43.2R. Three of four years improve; the most
|
||
recent one does not, and the 3-year max drawdown is 3.8R deeper (all of it in the 2023-24 year).
|
||
|
||
## Verdict
|
||
|
||
- **The process works end to end**: recorded evidence → screen → pre-registered rule → full
|
||
replay against same-harness controls → one pass, five fails, each with the reason recorded.
|
||
- **M5**: all three MACD candidates FAIL (−10 to −46R). Closed. The per-bar screen is not to
|
||
be trusted on M5 without a replay; it ranked them first.
|
||
- **M15-1 (ema_trend × 2)** FAILS on PF, drawdown and trade count; **M15-2 (divergence → 0 in
|
||
trending)** FAILS the first half and the drawdown. Both closed.
|
||
- **M15-3 (RSI multiplier × 0.5 in every regime) PASSES the pre-registered rule** (a–f) and the
|
||
3-year robustness check is positive (+34.7R, PF 1.38 → 1.39, extra year +6.2R). It is a
|
||
SHIPPING PROPOSAL for the owner, with these facts on the other side of the scale: the 2-year
|
||
second half passes by only +0.6R, 2026 is −4R, the added trades are mostly B-grade BUYs in a
|
||
bull-market data set (added SELLs lose), and the 3-year max drawdown deepens 15.5 → 19.3R.
|
||
If shipped it lands as an M15-only `technical.regime_weights_override` in
|
||
`pair_signals/xauusd.py` (rule 2) with the full table (RSI 1.0 → 0.5 in the trending family,
|
||
1.5 → 0.75 in ranging, 1.0 → 0.5 in transitioning), a factory-equivalence test update and a
|
||
docs note; the M15 canonical book would then be re-pinned from `candidates/m15_rsi:half_all_2yr.csv`
|
||
(931 / +146.14R / PF 1.43 at $0.65). Not shipped in this campaign.
|
||
|
||
Books and logs: `backtest/research/replay_2026-09/candidates/` and `.../oos/` (gitignored).
|
||
|
||
## Shipped (owner decision 2026-09-09, same day)
|
||
|
||
M15 override 5 in `pair_signals/xauusd.py`: `technical.regime_weights_override` = the shipped
|
||
table with the RSI multiplier halved in every regime (0.5 trending family / 0.75 ranging /
|
||
0.5 transitioning), derived from the default table and pinned literally by the Module 9
|
||
factory-equivalence test; one more Module 33 test locks that M5/H1/H4 carry no override.
|
||
Parity proof — the factory run with NO `--set` flag reproduces the validated candidate books
|
||
byte-for-byte: 2-year MD5 `d185f099482a07f29502cb5a371a9bbf` (931 / +146.14R / PF 1.43) and
|
||
OOS MD5 `6f8f24b49597ced28f50a19e08c3472f` (355 / +60.67R / PF 1.49). Canonical M15 book
|
||
re-pinned to `backtest/oanda_m15_2yr_065_20260909.csv` (copy of the 2-year factory book).
|
||
Docs: `docs/user_guide.html` and `docs/beginner_guide.html` carry a one-line note (PDF builds
|
||
remain behind, as before). Suite 499 tests. Live M15 scans get a new `engine_fingerprint`
|
||
from the next deploy; signal-log comparisons across it must split at the deploy time.
|