# GPU backend made real — 2026-09-18

Behaviour-neutral change. No signal logic, weight, gate or exit rule moved; the proof is the ten
MD5s below (five reference books on the GPU, five with `--backend cpu`), all equal to the pins in
CLAUDE.md section 9. The books therefore carry the same trades, R, win rate, profit factor and
drawdown, in both halves and at both cost conventions, as the pinned books do.

## 1. What was wrong

`backtest/gpu_backtest.py` probed CuPy at import with a `cumsum` and swallowed any exception.
On the dev box the first kernel compile raised

```
RuntimeError: Failed to auto-detect CUDA root directory. Please specify `CUDA_PATH` environment
variable if you are using CUDA versions not yet supported by CuPy.
```

CuPy 14.0.1 from pip (`cupy-cuda12x`) was installed with only the `nvidia-cuda-nvrtc-cu12`
wheel; the CUDA-12 pip layout is splayed (no `/usr/local/cuda`, no `nvcc`), so CuPy's
`_get_cuda_path()` returned `None` and NVRTC could not find the headers. Every run since the
Oanda re-pin logged `Backend: NumPy CPU fallback`; every research report in this folder records
it. Independently of that, almost nothing in the file was written for the GPU: the EWM
precompute was a pure-Python loop, Bollinger was pandas `rolling`, and the trailing-stop
resolver (the exit mode of every reference book) was a per-trade per-bar Python loop. The only
CuPy code was a first-hit reduction in the static SL/TP branch, which the reference books never
execute.

## 2. What shipped

- `nvidia-cuda-runtime-cu12==12.9.79` installed into the dev venv (headers only, not
  `requirements.txt`; same status as `cupy` and `pyarrow`).
- `backtest/gpu_kernels.py`: the backend state (`BACKEND`, probed lazily so importing the
  backtester or `analyze_signal_log.py` creates no CUDA context; the probe compiles a kernel, and
  a failure keeps its reason), `CUDA_PATH` pointed at the header wheel when unset, and three CUDA
  kernels compiled with `--fmad=false`:
  - `ewm_batch`: one thread per series, the exact recurrence of `_gpu_ewm_alpha` (seed at the
    first non-NaN value, NaN carries the state without advancing the `min_periods` counter).
  - `roll_mean_var`: pandas 3.0.2 `roll_mean` (Kahan add/remove, sign count, consecutive-same-
    value guard) and `roll_var` (Welford + Kahan, `InvCondTol` unstable recompute); std as
    pandas' `zsqrt`.
  - `resolve_trades`: one thread per trade; trailing mode mirrors `_simulate_trailing` line by
    line, static mode mirrors the SL-first/TP1/TIMEOUT branch, including the "NaN close after a
    TIMEOUT" bookkeeping; R and MFE/MAE on the device.
- `backtest/gpu_backtest.py`: `precompute_indicators` dispatches to `_precompute_indicators_gpu`
  (CuPy elementwise ops in the original order, two EWM launches and one rolling launch per
  timeframe, one transfer to host at the end, host-only cache asserted before the fork) or to the
  untouched NumPy path; `simulate_trades` dispatches to `_resolve_trades_gpu` or to the untouched
  CPU paths; pooled device memory is released before the workers are forked; `--backend
  auto|gpu|cpu` (env `FB2_BACKEND`); the `Backend:` line names the device or the fallback reason.
- `test_all.py` Module 36 (7 tests): a visible CUDA device must never fall back silently; the
  three kernels against their CPU references on synthetic edge cases (NaN head, interior NaN,
  flat window, cancellation spike, gaps, NaN bars, zero risk, per-trade trail overrides);
  `--backend` / `FB2_BACKEND` / `cpu_only()`; import guards (`cupy` is now banned from the live
  roots by the Module 32 AST guard as well).

Why the kernels are sequential per series: EWM and rolling statistics are order-dependent
floating point. A parallel scan or a two-pass variance changes the last bits, and the file's own
history shows what that does (a cumsum Bollinger shortcut moved `bb_position` by up to 8e-3 and
crossed signal thresholds). The GPU's gain here is the batch across series and across trades,
not per-series speed: on a single very long series the sequential kernel is slower than pandas'
C loop.

## 3. Evidence

Environment: NVIDIA GeForce RTX 4070 SUPER, driver 615.71.09 (CUDA 13.4 UMD), CuPy 14.0.1
(`cupy-cuda12x`), `nvidia-cuda-nvrtc-cu12` 12.9.86, `nvidia-cuda-runtime-cu12` 12.9.79, NumPy
2.2.6, pandas 3.0.2, Python 3.12, 16 CPUs. Code: this working tree on branch `sql-integration`
on top of `1f4741b`.

### 3.1 Precompute equality on every cached window

GPU vs CPU `precompute_indicators`, every `IndicatorCache` field compared with
`np.array_equal(..., equal_nan=True)`:

```
precompute GPU == CPU on 42/42 windows, 2342009 bars total; gpu 13.71s (incl. cold JIT) vs cpu 40.18s
```

### 3.2 Test suite, three ways

| Run | Result | Module 36 note |
|---|---|---|
| default (GPU) | `RESULTS: 536 passed, 0 failed, 536 total` | kernels exercised |
| `CUDA_VISIBLE_DEVICES=""` | `RESULTS: 536 passed, 0 failed, 536 total` | `NumPy CPU fallback (CUDARuntimeError: cudaErrorNoDevice: ...)` printed |
| `FB2_BACKEND=cpu` | `RESULTS: 536 passed, 0 failed, 536 total` | `NumPy CPU fallback (forced: ...)` printed |

### 3.3 Reference books on both backends

Harness: `venv/bin/python backtest/gpu_backtest.py --pair XAUUSD --spread 0.65 --workers 12
--primary-tf <tf> --start <s> --end <e> --export-csv <out>` (factory config; no `--set`,
`--quality`, `--strategies`, `--confirm-tfs` or `--export-features`), plus `--backend cpu` for
the CPU column. `GPU cache fallbacks: 0` on every run.

| Book | Window | Backend | Precompute | Replay | Trade sim | Wall | MD5 | Pinned |
|---|---|---|---|---|---|---|---|---|
| M5 W2 | 2024-09-05 → 2026-09-05 | CuPy GPU | 1.31 s | 112.0 s | 0.31 s | 114.8 s | `49f8e602fc6ae769713c94ef7b257810` | ✓ |
| M5 W2 | | NumPy CPU (`--backend cpu`) | 3.74 s | 112.6 s | 0.22 s | 117.0 s | `49f8e602fc6ae769713c94ef7b257810` | ✓ |
| M15 W2 | 2024-09-05 → 2026-09-05 | CuPy GPU | 1.29 s | 34.6 s | 0.33 s | 37.2 s | `b4aa07e33e17e23774c486a6900d673b` | ✓ |
| M15 W2 | | NumPy CPU | 3.69 s | 34.6 s | 0.28 s | 39.0 s | `b4aa07e33e17e23774c486a6900d673b` | ✓ |
| M15 OOS | 2024-06-01 → 2025-03-30 | CuPy GPU | 0.63 s | 14.0 s | 0.05 s | 15.6 s | `9fd624fb387c36052122bd1830cf9c32` | ✓ |
| M15 OOS | | NumPy CPU | 1.70 s | 13.9 s | 0.09 s | 16.1 s | `9fd624fb387c36052122bd1830cf9c32` | ✓ |
| H1 W3 | 2023-09-05 → 2026-09-05 | CuPy GPU | 0.61 s | 11.3 s | 0.07 s | 13.0 s | `7b848c917dcb5c371ecbb9f8f7292278` | ✓ |
| H1 W3 | | NumPy CPU | 1.68 s | 11.1 s | 0.07 s | 13.3 s | `7b848c917dcb5c371ecbb9f8f7292278` | ✓ |
| H4 W3 | 2023-09-05 → 2026-09-05 | CuPy GPU | 0.20 s | 20.0 s (1 worker) | 0.13 s | 21.2 s | `186e677fbbd53cca735c94094c23ce05` | ✓ |
| H4 W3 | | NumPy CPU | 0.46 s | 19.9 s (1 worker) | 0.02 s | 20.7 s | `186e677fbbd53cca735c94094c23ce05` | ✓ |

`Backend:` lines verbatim: `CuPy GPU (NVIDIA GeForce RTX 4070 SUPER, cupy 14.0.1)` and
`NumPy CPU fallback (forced: --backend cpu / FB2_BACKEND=cpu)`. The GPU precompute figures are
warm (the kernel cache under `~/.cupy/kernel_cache` was already populated; a cold first run adds
roughly a second of JIT once per machine). The trade-simulation figures on the GPU include the
host↔device transfers and the launch of a kernel that runs a few hundred threads; on these trade
counts it is not faster than the CPU loop, which is expected. The before/after picture is exactly
as planned: the precompute is about 3× faster, everything else is unchanged, and the replay
through the live signal code (the same bars/s on both rows) is where the wall time lives.

### 3.4 Other checks

- Single worker on the GPU (`--workers 1`, H1 W3): MD5 `7b848c917dcb5c371ecbb9f8f7292278`, equal
  to the 12-worker book (the CUDA context exists in the parent before the fork; the workers never
  touch it).
- `--record` pair on the GPU (H4 W3): control `186e677f…` == recorded `186e677f…` ==
  `manifest.book_csv_md5`; the manifest records `backend = CuPy GPU (NVIDIA GeForce RTX 4070 SUPER,
  cupy 14.0.1)` and `cache_fallbacks = 0`.
- `CUDA_VISIBLE_DEVICES="" ... --backend gpu` exits with
  `RuntimeError: --backend gpu requested but CUDA is unavailable: CUDARuntimeError: cudaErrorNoDevice ...`;
  the same run in `auto` mode logs the warning and
  `Backend: NumPy CPU fallback (CUDARuntimeError: cudaErrorNoDevice: no CUDA-capable device is detected)`.
- `analyze_signal_log.py --stored-candles` (which reuses `simulate_trades`) on the imported
  `logs/signals.db`: the report is line-for-line identical on the GPU and with `FB2_BACKEND=cpu`.
- `nvidia-smi --query-compute-apps` during the M5 precompute listed the backtester process on the
  device (184 MiB).
- Scratchpad experiments that preceded the implementation: the EWM kernel bit-identical to
  `_gpu_ewm_alpha` on 9 series over the 154k-bar M5 Close (95 ms incl. JIT vs 1.29 s Python);
  the rolling kernel bit-identical to pandas on all 84 series of the 42 cached windows.
- `git diff --stat` touches `backtest/gpu_backtest.py`, `test_all.py`, `CLAUDE.md`,
  `backtest/instructions_for_backtest.txt` and adds `backtest/gpu_kernels.py` and this note;
  `engine.py`, `signal_aggregator.py` and `requirements.txt` are untouched.