backtest: real CuPy GPU backend (kernels are bit-identical CPU twins); books byte-identical on both backends (2026-09-18)
CuPy from pip had never compiled a kernel on the dev box (no CUDA_PATH for the splayed CUDA-12 wheel layout); the probe swallowed the error and every book to date was built on the NumPy path under a "GPU backtester" name. - backtest/gpu_kernels.py (new): lazily probed, never-silent BACKEND state (CUDA_PATH pointed at the nvidia-cuda-runtime-cu12 header wheel; the probe compiles a kernel and keeps the failure reason), --backend auto|gpu|cpu and FB2_BACKEND, and three CUDA kernels compiled with --fmad=false: ewm_batch (the _gpu_ewm_alpha recurrence, one thread per series), roll_mean_var (pandas 3.0.2 roll_mean/roll_var replica, Kahan + Welford + unstable recompute), resolve_trades (trailing and static exits, one thread per trade). - gpu_backtest.py: precompute_indicators and simulate_trades dispatch to the GPU twins or to the untouched NumPy paths; caches are host NumPy and device memory is released before the replay workers fork; the Backend: line names the device or the fallback reason. - test_all.py Module 36 (7 tests): a visible CUDA device must never fall back silently; kernels vs their CPU references on synthetic edge cases; backend selection; import guards (cupy banned from the live roots). Suite 536. - CLAUDE.md, instructions_for_backtest.txt, research note with the evidence. Verification: all five reference MD5s reproduced on the GPU and with --backend cpu (ten runs, GPU cache fallbacks 0); precompute GPU == CPU on all 42 cached windows (2.34 M bars); suite green with GPU, with CUDA_VISIBLE_DEVICES="" and with FB2_BACKEND=cpu; --workers 1 and --record pairs equal. The per-bar replay stays the live Python signal code on CPU workers by design; wall time is roughly unchanged (precompute ~3x faster). Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
This commit is contained in:
Harwinder Sekhon
@@ -0,0 +1,137 @@
|
||||
# GPU backend made real — 2026-09-18
|
||||
|
||||
Behaviour-neutral change. No signal logic, weight, gate or exit rule moved; the proof is the ten
|
||||
MD5s below (five reference books on the GPU, five with `--backend cpu`), all equal to the pins in
|
||||
CLAUDE.md section 9. The books therefore carry the same trades, R, win rate, profit factor and
|
||||
drawdown, in both halves and at both cost conventions, as the pinned books do.
|
||||
|
||||
## 1. What was wrong
|
||||
|
||||
`backtest/gpu_backtest.py` probed CuPy at import with a `cumsum` and swallowed any exception.
|
||||
On the dev box the first kernel compile raised
|
||||
|
||||
```
|
||||
RuntimeError: Failed to auto-detect CUDA root directory. Please specify `CUDA_PATH` environment
|
||||
variable if you are using CUDA versions not yet supported by CuPy.
|
||||
```
|
||||
|
||||
CuPy 14.0.1 from pip (`cupy-cuda12x`) was installed with only the `nvidia-cuda-nvrtc-cu12`
|
||||
wheel; the CUDA-12 pip layout is splayed (no `/usr/local/cuda`, no `nvcc`), so CuPy's
|
||||
`_get_cuda_path()` returned `None` and NVRTC could not find the headers. Every run since the
|
||||
Oanda re-pin logged `Backend: NumPy CPU fallback`; every research report in this folder records
|
||||
it. Independently of that, almost nothing in the file was written for the GPU: the EWM
|
||||
precompute was a pure-Python loop, Bollinger was pandas `rolling`, and the trailing-stop
|
||||
resolver (the exit mode of every reference book) was a per-trade per-bar Python loop. The only
|
||||
CuPy code was a first-hit reduction in the static SL/TP branch, which the reference books never
|
||||
execute.
|
||||
|
||||
## 2. What shipped
|
||||
|
||||
- `nvidia-cuda-runtime-cu12==12.9.79` installed into the dev venv (headers only, not
|
||||
`requirements.txt`; same status as `cupy` and `pyarrow`).
|
||||
- `backtest/gpu_kernels.py`: the backend state (`BACKEND`, probed lazily so importing the
|
||||
backtester or `analyze_signal_log.py` creates no CUDA context; the probe compiles a kernel, and
|
||||
a failure keeps its reason), `CUDA_PATH` pointed at the header wheel when unset, and three CUDA
|
||||
kernels compiled with `--fmad=false`:
|
||||
- `ewm_batch`: one thread per series, the exact recurrence of `_gpu_ewm_alpha` (seed at the
|
||||
first non-NaN value, NaN carries the state without advancing the `min_periods` counter).
|
||||
- `roll_mean_var`: pandas 3.0.2 `roll_mean` (Kahan add/remove, sign count, consecutive-same-
|
||||
value guard) and `roll_var` (Welford + Kahan, `InvCondTol` unstable recompute); std as
|
||||
pandas' `zsqrt`.
|
||||
- `resolve_trades`: one thread per trade; trailing mode mirrors `_simulate_trailing` line by
|
||||
line, static mode mirrors the SL-first/TP1/TIMEOUT branch, including the "NaN close after a
|
||||
TIMEOUT" bookkeeping; R and MFE/MAE on the device.
|
||||
- `backtest/gpu_backtest.py`: `precompute_indicators` dispatches to `_precompute_indicators_gpu`
|
||||
(CuPy elementwise ops in the original order, two EWM launches and one rolling launch per
|
||||
timeframe, one transfer to host at the end, host-only cache asserted before the fork) or to the
|
||||
untouched NumPy path; `simulate_trades` dispatches to `_resolve_trades_gpu` or to the untouched
|
||||
CPU paths; pooled device memory is released before the workers are forked; `--backend
|
||||
auto|gpu|cpu` (env `FB2_BACKEND`); the `Backend:` line names the device or the fallback reason.
|
||||
- `test_all.py` Module 36 (7 tests): a visible CUDA device must never fall back silently; the
|
||||
three kernels against their CPU references on synthetic edge cases (NaN head, interior NaN,
|
||||
flat window, cancellation spike, gaps, NaN bars, zero risk, per-trade trail overrides);
|
||||
`--backend` / `FB2_BACKEND` / `cpu_only()`; import guards (`cupy` is now banned from the live
|
||||
roots by the Module 32 AST guard as well).
|
||||
|
||||
Why the kernels are sequential per series: EWM and rolling statistics are order-dependent
|
||||
floating point. A parallel scan or a two-pass variance changes the last bits, and the file's own
|
||||
history shows what that does (a cumsum Bollinger shortcut moved `bb_position` by up to 8e-3 and
|
||||
crossed signal thresholds). The GPU's gain here is the batch across series and across trades,
|
||||
not per-series speed: on a single very long series the sequential kernel is slower than pandas'
|
||||
C loop.
|
||||
|
||||
## 3. Evidence
|
||||
|
||||
Environment: NVIDIA GeForce RTX 4070 SUPER, driver 615.71.09 (CUDA 13.4 UMD), CuPy 14.0.1
|
||||
(`cupy-cuda12x`), `nvidia-cuda-nvrtc-cu12` 12.9.86, `nvidia-cuda-runtime-cu12` 12.9.79, NumPy
|
||||
2.2.6, pandas 3.0.2, Python 3.12, 16 CPUs. Code: this working tree on branch `sql-integration`
|
||||
on top of `1f4741b`.
|
||||
|
||||
### 3.1 Precompute equality on every cached window
|
||||
|
||||
GPU vs CPU `precompute_indicators`, every `IndicatorCache` field compared with
|
||||
`np.array_equal(..., equal_nan=True)`:
|
||||
|
||||
```
|
||||
precompute GPU == CPU on 42/42 windows, 2342009 bars total; gpu 13.71s (incl. cold JIT) vs cpu 40.18s
|
||||
```
|
||||
|
||||
### 3.2 Test suite, three ways
|
||||
|
||||
| Run | Result | Module 36 note |
|
||||
|---|---|---|
|
||||
| default (GPU) | `RESULTS: 536 passed, 0 failed, 536 total` | kernels exercised |
|
||||
| `CUDA_VISIBLE_DEVICES=""` | `RESULTS: 536 passed, 0 failed, 536 total` | `NumPy CPU fallback (CUDARuntimeError: cudaErrorNoDevice: ...)` printed |
|
||||
| `FB2_BACKEND=cpu` | `RESULTS: 536 passed, 0 failed, 536 total` | `NumPy CPU fallback (forced: ...)` printed |
|
||||
|
||||
### 3.3 Reference books on both backends
|
||||
|
||||
Harness: `venv/bin/python backtest/gpu_backtest.py --pair XAUUSD --spread 0.65 --workers 12
|
||||
--primary-tf <tf> --start <s> --end <e> --export-csv <out>` (factory config; no `--set`,
|
||||
`--quality`, `--strategies`, `--confirm-tfs` or `--export-features`), plus `--backend cpu` for
|
||||
the CPU column. `GPU cache fallbacks: 0` on every run.
|
||||
|
||||
| Book | Window | Backend | Precompute | Replay | Trade sim | Wall | MD5 | Pinned |
|
||||
|---|---|---|---|---|---|---|---|---|
|
||||
| M5 W2 | 2024-09-05 → 2026-09-05 | CuPy GPU | 1.31 s | 112.0 s | 0.31 s | 114.8 s | `49f8e602fc6ae769713c94ef7b257810` | ✓ |
|
||||
| M5 W2 | | NumPy CPU (`--backend cpu`) | 3.74 s | 112.6 s | 0.22 s | 117.0 s | `49f8e602fc6ae769713c94ef7b257810` | ✓ |
|
||||
| M15 W2 | 2024-09-05 → 2026-09-05 | CuPy GPU | 1.29 s | 34.6 s | 0.33 s | 37.2 s | `b4aa07e33e17e23774c486a6900d673b` | ✓ |
|
||||
| M15 W2 | | NumPy CPU | 3.69 s | 34.6 s | 0.28 s | 39.0 s | `b4aa07e33e17e23774c486a6900d673b` | ✓ |
|
||||
| M15 OOS | 2024-06-01 → 2025-03-30 | CuPy GPU | 0.63 s | 14.0 s | 0.05 s | 15.6 s | `9fd624fb387c36052122bd1830cf9c32` | ✓ |
|
||||
| M15 OOS | | NumPy CPU | 1.70 s | 13.9 s | 0.09 s | 16.1 s | `9fd624fb387c36052122bd1830cf9c32` | ✓ |
|
||||
| H1 W3 | 2023-09-05 → 2026-09-05 | CuPy GPU | 0.61 s | 11.3 s | 0.07 s | 13.0 s | `7b848c917dcb5c371ecbb9f8f7292278` | ✓ |
|
||||
| H1 W3 | | NumPy CPU | 1.68 s | 11.1 s | 0.07 s | 13.3 s | `7b848c917dcb5c371ecbb9f8f7292278` | ✓ |
|
||||
| H4 W3 | 2023-09-05 → 2026-09-05 | CuPy GPU | 0.20 s | 20.0 s (1 worker) | 0.13 s | 21.2 s | `186e677fbbd53cca735c94094c23ce05` | ✓ |
|
||||
| H4 W3 | | NumPy CPU | 0.46 s | 19.9 s (1 worker) | 0.02 s | 20.7 s | `186e677fbbd53cca735c94094c23ce05` | ✓ |
|
||||
|
||||
`Backend:` lines verbatim: `CuPy GPU (NVIDIA GeForce RTX 4070 SUPER, cupy 14.0.1)` and
|
||||
`NumPy CPU fallback (forced: --backend cpu / FB2_BACKEND=cpu)`. The GPU precompute figures are
|
||||
warm (the kernel cache under `~/.cupy/kernel_cache` was already populated; a cold first run adds
|
||||
roughly a second of JIT once per machine). The trade-simulation figures on the GPU include the
|
||||
host↔device transfers and the launch of a kernel that runs a few hundred threads; on these trade
|
||||
counts it is not faster than the CPU loop, which is expected. The before/after picture is exactly
|
||||
as planned: the precompute is about 3× faster, everything else is unchanged, and the replay
|
||||
through the live signal code (the same bars/s on both rows) is where the wall time lives.
|
||||
|
||||
### 3.4 Other checks
|
||||
|
||||
- Single worker on the GPU (`--workers 1`, H1 W3): MD5 `7b848c917dcb5c371ecbb9f8f7292278`, equal
|
||||
to the 12-worker book (the CUDA context exists in the parent before the fork; the workers never
|
||||
touch it).
|
||||
- `--record` pair on the GPU (H4 W3): control `186e677f…` == recorded `186e677f…` ==
|
||||
`manifest.book_csv_md5`; the manifest records `backend = CuPy GPU (NVIDIA GeForce RTX 4070 SUPER,
|
||||
cupy 14.0.1)` and `cache_fallbacks = 0`.
|
||||
- `CUDA_VISIBLE_DEVICES="" ... --backend gpu` exits with
|
||||
`RuntimeError: --backend gpu requested but CUDA is unavailable: CUDARuntimeError: cudaErrorNoDevice ...`;
|
||||
the same run in `auto` mode logs the warning and
|
||||
`Backend: NumPy CPU fallback (CUDARuntimeError: cudaErrorNoDevice: no CUDA-capable device is detected)`.
|
||||
- `analyze_signal_log.py --stored-candles` (which reuses `simulate_trades`) on the imported
|
||||
`logs/signals.db`: the report is line-for-line identical on the GPU and with `FB2_BACKEND=cpu`.
|
||||
- `nvidia-smi --query-compute-apps` during the M5 precompute listed the backtester process on the
|
||||
device (184 MiB).
|
||||
- Scratchpad experiments that preceded the implementation: the EWM kernel bit-identical to
|
||||
`_gpu_ewm_alpha` on 9 series over the 154k-bar M5 Close (95 ms incl. JIT vs 1.29 s Python);
|
||||
the rolling kernel bit-identical to pandas on all 84 series of the 42 cached windows.
|
||||
- `git diff --stat` touches `backtest/gpu_backtest.py`, `test_all.py`, `CLAUDE.md`,
|
||||
`backtest/instructions_for_backtest.txt` and adds `backtest/gpu_kernels.py` and this note;
|
||||
`engine.py`, `signal_aggregator.py` and `requirements.txt` are untouched.
|
||||
Reference in New Issue
Block a user