ForexBot2/backtest/research/gpu_backend_2026-09-18.md

GPU backend made real — 2026-09-18

Behaviour-neutral change. No signal logic, weight, gate or exit rule moved; the proof is the ten MD5s below (five reference books on the GPU, five with --backend cpu), all equal to the pins in CLAUDE.md section 9. The books therefore carry the same trades, R, win rate, profit factor and drawdown, in both halves and at both cost conventions, as the pinned books do.

1. What was wrong

backtest/gpu_backtest.py probed CuPy at import with a cumsum and swallowed any exception. On the dev box the first kernel compile raised

RuntimeError: Failed to auto-detect CUDA root directory. Please specify `CUDA_PATH` environment
variable if you are using CUDA versions not yet supported by CuPy.

CuPy 14.0.1 from pip (cupy-cuda12x) was installed with only the nvidia-cuda-nvrtc-cu12 wheel; the CUDA-12 pip layout is splayed (no /usr/local/cuda, no nvcc), so CuPy's _get_cuda_path() returned None and NVRTC could not find the headers. Every run since the Oanda re-pin logged Backend: NumPy CPU fallback; every research report in this folder records it. Independently of that, almost nothing in the file was written for the GPU: the EWM precompute was a pure-Python loop, Bollinger was pandas rolling, and the trailing-stop resolver (the exit mode of every reference book) was a per-trade per-bar Python loop. The only CuPy code was a first-hit reduction in the static SL/TP branch, which the reference books never execute.

2. What shipped

• nvidia-cuda-runtime-cu12==12.9.79 installed into the dev venv (headers only, not requirements.txt; same status as cupy and pyarrow).
• backtest/gpu_kernels.py: the backend state (BACKEND, probed lazily so importing the backtester or analyze_signal_log.py creates no CUDA context; the probe compiles a kernel, and a failure keeps its reason), CUDA_PATH pointed at the header wheel when unset, and three CUDA kernels compiled with --fmad=false:
  • ewm_batch: one thread per series, the exact recurrence of _gpu_ewm_alpha (seed at the first non-NaN value, NaN carries the state without advancing the min_periods counter).
  • roll_mean_var: pandas 3.0.2 roll_mean (Kahan add/remove, sign count, consecutive-same- value guard) and roll_var (Welford + Kahan, InvCondTol unstable recompute); std as pandas' zsqrt.
  • resolve_trades: one thread per trade; trailing mode mirrors _simulate_trailing line by line, static mode mirrors the SL-first/TP1/TIMEOUT branch, including the "NaN close after a TIMEOUT" bookkeeping; R and MFE/MAE on the device.
• backtest/gpu_backtest.py: precompute_indicators dispatches to _precompute_indicators_gpu (CuPy elementwise ops in the original order, two EWM launches and one rolling launch per timeframe, one transfer to host at the end, host-only cache asserted before the fork) or to the untouched NumPy path; simulate_trades dispatches to _resolve_trades_gpu or to the untouched CPU paths; pooled device memory is released before the workers are forked; --backend auto|gpu|cpu (env FB2_BACKEND); the Backend: line names the device or the fallback reason.
• test_all.py Module 36 (7 tests): a visible CUDA device must never fall back silently; the three kernels against their CPU references on synthetic edge cases (NaN head, interior NaN, flat window, cancellation spike, gaps, NaN bars, zero risk, per-trade trail overrides); --backend / FB2_BACKEND / cpu_only(); import guards (cupy is now banned from the live roots by the Module 32 AST guard as well).

Why the kernels are sequential per series: EWM and rolling statistics are order-dependent floating point. A parallel scan or a two-pass variance changes the last bits, and the file's own history shows what that does (a cumsum Bollinger shortcut moved bb_position by up to 8e-3 and crossed signal thresholds). The GPU's gain here is the batch across series and across trades, not per-series speed: on a single very long series the sequential kernel is slower than pandas' C loop.

3. Evidence

Environment: NVIDIA GeForce RTX 4070 SUPER, driver 615.71.09 (CUDA 13.4 UMD), CuPy 14.0.1 (cupy-cuda12x), nvidia-cuda-nvrtc-cu12 12.9.86, nvidia-cuda-runtime-cu12 12.9.79, NumPy 2.2.6, pandas 3.0.2, Python 3.12, 16 CPUs. Code: this working tree on branch sql-integration on top of 1f4741b.

3.1 Precompute equality on every cached window

GPU vs CPU precompute_indicators, every IndicatorCache field compared with np.array_equal(..., equal_nan=True):

precompute GPU == CPU on 42/42 windows, 2342009 bars total; gpu 13.71s (incl. cold JIT) vs cpu 40.18s

3.2 Test suite, three ways

Run	Result	Module 36 note
default (GPU)	RESULTS: 536 passed, 0 failed, 536 total	kernels exercised
CUDA_VISIBLE_DEVICES=""	RESULTS: 536 passed, 0 failed, 536 total	NumPy CPU fallback (CUDARuntimeError: cudaErrorNoDevice: ...) printed
FB2_BACKEND=cpu	RESULTS: 536 passed, 0 failed, 536 total	NumPy CPU fallback (forced: ...) printed

3.3 Reference books on both backends

Harness: venv/bin/python backtest/gpu_backtest.py --pair XAUUSD --spread 0.65 --workers 12 --primary-tf <tf> --start <s> --end <e> --export-csv <out> (factory config; no --set, --quality, --strategies, --confirm-tfs or --export-features), plus --backend cpu for the CPU column. GPU cache fallbacks: 0 on every run.

Book	Window	Backend	Precompute	Replay	Trade sim	Wall	MD5	Pinned
M5 W2	2024-09-05 → 2026-09-05	CuPy GPU	1.31 s	112.0 s	0.31 s	114.8 s	49f8e602fc6ae769713c94ef7b257810	✓
M5 W2		NumPy CPU (--backend cpu)	3.74 s	112.6 s	0.22 s	117.0 s	49f8e602fc6ae769713c94ef7b257810	✓
M15 W2	2024-09-05 → 2026-09-05	CuPy GPU	1.29 s	34.6 s	0.33 s	37.2 s	b4aa07e33e17e23774c486a6900d673b	✓
M15 W2		NumPy CPU	3.69 s	34.6 s	0.28 s	39.0 s	b4aa07e33e17e23774c486a6900d673b	✓
M15 OOS	2024-06-01 → 2025-03-30	CuPy GPU	0.63 s	14.0 s	0.05 s	15.6 s	9fd624fb387c36052122bd1830cf9c32	✓
M15 OOS		NumPy CPU	1.70 s	13.9 s	0.09 s	16.1 s	9fd624fb387c36052122bd1830cf9c32	✓
H1 W3	2023-09-05 → 2026-09-05	CuPy GPU	0.61 s	11.3 s	0.07 s	13.0 s	7b848c917dcb5c371ecbb9f8f7292278	✓
H1 W3		NumPy CPU	1.68 s	11.1 s	0.07 s	13.3 s	7b848c917dcb5c371ecbb9f8f7292278	✓
H4 W3	2023-09-05 → 2026-09-05	CuPy GPU	0.20 s	20.0 s (1 worker)	0.13 s	21.2 s	186e677fbbd53cca735c94094c23ce05	✓
H4 W3		NumPy CPU	0.46 s	19.9 s (1 worker)	0.02 s	20.7 s	186e677fbbd53cca735c94094c23ce05	✓

Backend: lines verbatim: CuPy GPU (NVIDIA GeForce RTX 4070 SUPER, cupy 14.0.1) and NumPy CPU fallback (forced: --backend cpu / FB2_BACKEND=cpu). The GPU precompute figures are warm (the kernel cache under ~/.cupy/kernel_cache was already populated; a cold first run adds roughly a second of JIT once per machine). The trade-simulation figures on the GPU include the host↔device transfers and the launch of a kernel that runs a few hundred threads; on these trade counts it is not faster than the CPU loop, which is expected. The before/after picture is exactly as planned: the precompute is about 3× faster, everything else is unchanged, and the replay through the live signal code (the same bars/s on both rows) is where the wall time lives.

3.4 Other checks

• Single worker on the GPU (--workers 1, H1 W3): MD5 7b848c917dcb5c371ecbb9f8f7292278, equal to the 12-worker book (the CUDA context exists in the parent before the fork; the workers never touch it).
• --record pair on the GPU (H4 W3): control 186e677f… == recorded 186e677f… == manifest.book_csv_md5; the manifest records backend = CuPy GPU (NVIDIA GeForce RTX 4070 SUPER, cupy 14.0.1) and cache_fallbacks = 0.
• CUDA_VISIBLE_DEVICES="" ... --backend gpu exits with RuntimeError: --backend gpu requested but CUDA is unavailable: CUDARuntimeError: cudaErrorNoDevice ...; the same run in auto mode logs the warning and Backend: NumPy CPU fallback (CUDARuntimeError: cudaErrorNoDevice: no CUDA-capable device is detected).
• analyze_signal_log.py --stored-candles (which reuses simulate_trades) on the imported logs/signals.db: the report is line-for-line identical on the GPU and with FB2_BACKEND=cpu.
• nvidia-smi --query-compute-apps during the M5 precompute listed the backtester process on the device (184 MiB).
• Scratchpad experiments that preceded the implementation: the EWM kernel bit-identical to _gpu_ewm_alpha on 9 series over the 154k-bar M5 Close (95 ms incl. JIT vs 1.29 s Python); the rolling kernel bit-identical to pandas on all 84 series of the 42 cached windows.
• git diff --stat touches backtest/gpu_backtest.py, test_all.py, CLAUDE.md, backtest/instructions_for_backtest.txt and adds backtest/gpu_kernels.py and this note; engine.py, signal_aggregator.py and requirements.txt are untouched.