ForexBot2/backtest/research/weight_test_2026-09-09.md

Weight-change test campaign — 2026-09-09 (does the recorded evidence improve predictions?)

Purpose

First use of the replay-recording files (replay_recording_2026-09-09.md): take candidate changes to the regime weight table suggested by the per-bar evidence, rerun the FULL backtest with each, and compare against the same-harness control books. This is the "before versus after" that measures improvement. Everything below the rule was written before any candidate was run.

Harness (identical to the recording campaign)

backtest/gpu_backtest.py --pair XAUUSD --start 2024-09-05 --end 2026-09-05 --spread 0.65 --workers N --primary-tf {5min|15min} + --set technical.regime_weights_override=<full table> (the field replaces the whole table, so every candidate passes the complete table with one cell family changed). Controls: backtest/research/replay_2026-09/{m5,m15}/book_control.csv (same code, no override). OOS window: --start 2024-06-01 --end 2025-03-30 (cached oanda_XAU_USD_*_20240331_20250330.parquet), control produced on the same harness. Backend NumPy CPU. Code: git af06685.

Pre-registered candidate-selection rule (written 2026-09-09 before the screen ran)

1. A candidate changes ONE indicator's regime multipliers only (no code, no gate, no threshold change), on ONE timeframe. Allowed forms: multiplier → 0 in all regimes, × 0.5 in all regimes, × 2 in all regimes, or the same applied to a single regime family (trending / ranging / transitioning) when the screen shows the effect is regime-specific.
2. The screen (scratch script over bars.parquet, both TFs): for each candidate, recompute every analyzed bar's PRIMARY composite from the recorded effective weights and contributions, find the bars that would NEWLY clear ±15 and the bars that would DROP OUT, and score the candidate by the net drift-adjusted signed 4-hour forward move (gained bars minus lost bars; drift = the mean unsigned 4-hour move of all analyzed bars in the same regime). Top 3 per timeframe by that score are run; nothing else.
3. Earlier verdicts on the same knob are recorded next to the candidate but do not veto it (this harness — 2-year Oanda window, post-2026-09-07 outlier rule — is new).

Pre-registered acceptance rule (vs the same-harness control)

A candidate PASSES only if ALL of the following hold on the 2-year window at $0.65:

• a. ΔR ≥ +5R overall versus the control (smaller differences are noise on ~800 trades);
• b. ΔR > 0 in BOTH fixed halves: 2024-09-05→2025-09-05 and 2025-09-05→2026-09-05;
• c. PF ≥ 1.3 and not more than 0.05 below the control's PF;
• d. max drawdown not more than 2R worse than the control's;
• e. trade count within ±25% of the control's (guards against "fewer trades" artefacts);
• f. OOS window 2024-06-01→2025-03-30: ΔR ≥ 0 versus the OOS control. Failing candidates are recorded and closed. No rescue: no multiplier fine-tuning after seeing a result, no second window shopping. A passing candidate is a SHIPPING PROPOSAL for the owner (rule 2: it would land as a per-timeframe regime_weights_override in pair_signals/xauusd.py with tests), not an automatic change.

Screen results (scratch script over bars.parquet, both TFs; composite reproduced from the stored columns within 3.4e-6 on every bar)

Drift per regime (mean 4 h move of all analyzed bars, ATR units, M5): trending_up +0.55, trending_down +0.16, trending +0.35, transitioning +0.64, ranging +0.38 — a strong up-drift that every BUY vote inherits, hence the per-regime adjustment.

Standalone votes after drift removal (mean signed 4 h move; positive = the vote pointed the right way), M5 highlights:

indicator	regime family	buy votes	sell votes	reading
macd	trending	−0.079 (n 8,717)	−0.154 (n 8,891)	wrong way on BOTH sides; also at 24 h (−0.22 / −0.56)
macd	transitioning	+0.273 (1,738)	+0.288 (1,885)	best caller in that regime
rsi	transitioning	−1.089 (357)	−0.146 (570)	wrong way, small n
bollinger	transitioning	−0.261 (2,668)	−0.337 (2,675)	wrong way
bollinger	trending	+0.084	+0.091	mildly right
ema_trend	ranging / transitioning	+0.33 / +0.42	≈0	buy side right, sell side nothing
adx	trending	+0.048	+0.041	never fires outside trending

Candidate screen (net drift-adjusted 4 h move of gained minus lost bars; the 24 h column must agree in sign for a candidate to be run). No candidate flips a bar's direction — one indicator's weight only moves bars in and out of the neutral band.

M5 top of the board: macd:zero_trending +1235 (24 h +4181, agrees; 3,406 gained / 7,339 lost, touches 23% of directional bars), macd:double_ranging +1186 (24 h −21, rejected), divergence:zero_all +1026 (24 h −633, rejected), bollinger:zero_ranging +940 (24 h −728, rejected), bollinger:zero_all +800 (24 h −1654, rejected), macd:zero_all +705 (24 h +1644, agrees; touches 50% of directional bars), macd:half_trending +689 (24 h +334, agrees). Bottom: ema_trend:zero_all −389, macd:double_trending −540, macd:zero_transitioning −607.

M15 top of the board: ema_trend:double_all +329 (24 h +1209, agrees), macd:zero_all +293 (24 h −79, rejected), divergence:zero_all +264 (24 h −480, rejected), divergence:zero_trending +246 (24 h +390, agrees), rsi:half_all +221 (24 h +99, agrees). Bottom: macd:double_trending −337, macd:double_all −323.

Candidates run (rule 2 above, 24 h-sign filter applied):

• M5-1 macd:zero_trending — MACD multiplier 2.0 → 0 in trending_up/trending_down/trending.
• M5-2 macd:zero_all — MACD multiplier → 0 in every regime.
• M5-3 macd:half_trending — MACD multiplier 2.0 → 1.0 in the trending family.
• M15-1 ema_trend:double_all — ema_trend multiplier × 2 in every regime.
• M15-2 divergence:zero_trending — divergence multiplier 1.0 → 0 in the trending family.
• M15-3 rsi:half_all — RSI multiplier × 0.5 in every regime. Prior verdicts on the same knobs: none for MACD or ema_trend; "divergence-down" on M5 was rejected 2026-08-04 and "divergence-off" on M15 was declined 2026-09 (R up, 5-yr DD 35.8 → 47.4R); neither is the same change as M15-2 (trending family only).

Half of each board's top entries fail the 24 h check, some violently (bollinger:zero_all on M5: +800 at 4 h, −1654 at 24 h) — a 4 h-only screen is a coin flip. Scores are sums, so volume can carry them: macd:zero_all scores by removing 16,170 directional bars whose mean was −0.07, while its gained bars average −0.09.

Rerun results (2-year window 2024-09-05→2026-09-05, $0.65; halves fixed at 2025-09-05; OOS 2024-06-01→2025-03-30)

candidate	trades	total R	ΔR	H1 ΔR	H2 ΔR	PF	max DD	OOS ΔR	verdict
M5 control	781	+146.07				1.52	10.1	(86 / +9.47, PF 1.33)
M5-1 macd → 0 in trending	808	+105.35	−40.7	+1.9	−42.6	1.35	17.1	+0.2	FAIL a,b,c,d
M5-2 macd → 0 everywhere	699	+99.79	−46.3	+0.5	−46.8	1.38	17.2	−1.8	FAIL a,b,c,d,f
M5-3 macd × 0.5 in trending	804	+136.12	−10.0	+7.4	−17.4	1.47	15.2	+6.8	FAIL a,b,c,d
M15 control	793	+117.62				1.41	14.8	(300 / +40.59, PF 1.38)
M15-1 ema_trend × 2 everywhere	1229	+148.20	+30.6	+10.0	+20.5	1.32	26.4	+0.7	FAIL c,d,e (+55% trades)
M15-2 divergence → 0 in trending	861	+127.69	+10.1	−3.0	+13.1	1.41	18.4	+3.6	FAIL b,d
M15-3 rsi × 0.5 everywhere	931	+146.14	+28.5	+27.9	+0.6	1.43	16.1	+20.1 (355 / +60.67, PF 1.49)	PASS a–f

The M5 screen was wrong in every case: the three MACD candidates that scored best on the per-bar screen (and were the only ones whose 24 h score agreed) lose 10–46R in the full replay, all of it in the second year. The per-bar composite screen sees threshold crossings; it does not see that MACD's "wrong-way" votes in trending regimes are doing useful work downstream (the exhaustion penalties, the M5 min-score gate and the setup generators all sit between the composite and a trade). Lesson recorded: the screen ranks hypotheses, the replay judges them, and on M5 the two disagreed 3/3.

M15-3 trade diff (control → candidate, 2-year)

• 721 shared entries: control +124.7R → candidate +119.4R (56 shared trades changed R because the score moved their grade and therefore the setup distances).
• 210 added trades: +26.8R, mean +0.127R, WR 59.5% — 157 BUY (+32.5R) and 53 SELL (−5.8R); 156 are B-grade (+19.1R), 201 sit in the 25–40 score band; by regime ranging 63 (+13.1R), trending_up 88 (+12.1R), transitioning 32 (+2.9R), trending_down 27 (−1.3R).
• 72 removed trades: −7.1R, WR 50%.
• By year: 2024 (Sep–Dec) +19.4 → +26.7R, 2025 +51.9 → +77.4R, 2026 (Jan–Sep) +46.3 → +42.1R. The 2-year gain is concentrated in 2025; the fixed second half passes rule b by only +0.6R.
• OOS (2024-06→2025-03): 300 → 355 trades, +40.6 → +60.7R, PF 1.38 → 1.49, WR 59.3 → 60.8%.

Mechanism in plain terms: RSI is neutral on most M15 entries and, when it votes, it votes against the trend (a mean-reversion reading). Halving its weight lets more trend-following entries clear the score band, mostly BUYs, in a data set that is a gold bull market throughout. The OOS window improves in quality (PF, win rate), not only in count, which is the strongest single fact in the candidate's favour; the year-by-year unevenness is the strongest fact against.

Robustness check on the 3-year window (owner's maximum; 2023-09-05→2026-09-05, cached bars)

book	trades	total R	WR	PF	max DD
M15 control 3-yr	1092	+151.90	59.4%	1.38	15.5
M15-3 rsi × 0.5, 3-yr	1273	+186.61	59.4%	1.39	19.3
extra year only (2023-09-05→2024-09-05, never seen by the screen): control	299	+33.23	56.9%	1.29	14.9
extra year only: candidate	342	+39.43	56.4%	1.29	19.3

Calendar years, control → candidate: 2023 (Sep–Dec) +10.1 → +13.2R; 2024 +42.6 → +52.9R; 2025 +51.9 → +77.4R; 2026 (Jan–Sep) +47.3 → +43.2R. Three of four years improve; the most recent one does not, and the 3-year max drawdown is 3.8R deeper (all of it in the 2023-24 year).

Verdict

• The process works end to end: recorded evidence → screen → pre-registered rule → full replay against same-harness controls → one pass, five fails, each with the reason recorded.
• M5: all three MACD candidates FAIL (−10 to −46R). Closed. The per-bar screen is not to be trusted on M5 without a replay; it ranked them first.
• M15-1 (ema_trend × 2) FAILS on PF, drawdown and trade count; M15-2 (divergence → 0 in trending) FAILS the first half and the drawdown. Both closed.
• M15-3 (RSI multiplier × 0.5 in every regime) PASSES the pre-registered rule (a–f) and the 3-year robustness check is positive (+34.7R, PF 1.38 → 1.39, extra year +6.2R). It is a SHIPPING PROPOSAL for the owner, with these facts on the other side of the scale: the 2-year second half passes by only +0.6R, 2026 is −4R, the added trades are mostly B-grade BUYs in a bull-market data set (added SELLs lose), and the 3-year max drawdown deepens 15.5 → 19.3R. If shipped it lands as an M15-only technical.regime_weights_override in pair_signals/xauusd.py (rule 2) with the full table (RSI 1.0 → 0.5 in the trending family, 1.5 → 0.75 in ranging, 1.0 → 0.5 in transitioning), a factory-equivalence test update and a docs note; the M15 canonical book would then be re-pinned from candidates/m15_rsi:half_all_2yr.csv (931 / +146.14R / PF 1.43 at $0.65). Not shipped in this campaign.

Books and logs: backtest/research/replay_2026-09/candidates/ and .../oos/ (gitignored).

Shipped (owner decision 2026-09-09, same day)

M15 override 5 in pair_signals/xauusd.py: technical.regime_weights_override = the shipped table with the RSI multiplier halved in every regime (0.5 trending family / 0.75 ranging / 0.5 transitioning), derived from the default table and pinned literally by the Module 9 factory-equivalence test; one more Module 33 test locks that M5/H1/H4 carry no override. Parity proof — the factory run with NO --set flag reproduces the validated candidate books byte-for-byte: 2-year MD5 d185f099482a07f29502cb5a371a9bbf (931 / +146.14R / PF 1.43) and OOS MD5 6f8f24b49597ced28f50a19e08c3472f (355 / +60.67R / PF 1.49). Canonical M15 book re-pinned to backtest/oanda_m15_2yr_065_20260909.csv (copy of the 2-year factory book). Docs: docs/user_guide.html and docs/beginner_guide.html carry a one-line note (PDF builds remain behind, as before). Suite 499 tests. Live M15 scans get a new engine_fingerprint from the next deploy; signal-log comparisons across it must split at the deploy time.