harry/ForexBot2
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

backtest: real CuPy GPU backend (kernels are bit-identical CPU twins); books byte-identical on both backends (2026-09-18)

Browse Source 
CuPy from pip had never compiled a kernel on the dev box (no CUDA_PATH for the
splayed CUDA-12 wheel layout); the probe swallowed the error and every book to
date was built on the NumPy path under a "GPU backtester" name.

- backtest/gpu_kernels.py (new): lazily probed, never-silent BACKEND state
  (CUDA_PATH pointed at the nvidia-cuda-runtime-cu12 header wheel; the probe
  compiles a kernel and keeps the failure reason), --backend auto|gpu|cpu and
  FB2_BACKEND, and three CUDA kernels compiled with --fmad=false: ewm_batch
  (the _gpu_ewm_alpha recurrence, one thread per series), roll_mean_var
  (pandas 3.0.2 roll_mean/roll_var replica, Kahan + Welford + unstable
  recompute), resolve_trades (trailing and static exits, one thread per trade).
- gpu_backtest.py: precompute_indicators and simulate_trades dispatch to the
  GPU twins or to the untouched NumPy paths; caches are host NumPy and device
  memory is released before the replay workers fork; the Backend: line names
  the device or the fallback reason.
- test_all.py Module 36 (7 tests): a visible CUDA device must never fall back
  silently; kernels vs their CPU references on synthetic edge cases; backend
  selection; import guards (cupy banned from the live roots). Suite 536.
- CLAUDE.md, instructions_for_backtest.txt, research note with the evidence.

Verification: all five reference MD5s reproduced on the GPU and with
--backend cpu (ten runs, GPU cache fallbacks 0); precompute GPU == CPU on all
42 cached windows (2.34 M bars); suite green with GPU, with
CUDA_VISIBLE_DEVICES="" and with FB2_BACKEND=cpu; --workers 1 and --record
pairs equal. The per-bar replay stays the live Python signal code on CPU
workers by design; wall time is roughly unchanged (precompute ~3x faster).
This commit is contained in:
Harwinder Sekhon
2026-09-18 10:39:55 -04:00
parent ad14ea4fed
commit 02589cbb88
6 changed files with 1487 additions and 70 deletions
Download Patch File Download Diff File Expand all files Collapse all files
CLAUDE.md
+36 -9
View File
@@ -20,8 +20,10 @@ you must follow before changing anything that affects signals. Section 10 lists
python -m venv venv && venv/bin/pip install -r requirements.txt # Python 3.12; a venv/ already exists on the dev box
cp config.json.example config.json # then fill in the keys (see section 5)
venv/bin/pip install cupy-cuda12x "nvidia-cuda-runtime-cu12==12.9.*" # optional, dev box: backtester GPU backend
# (CuPy from pip needs that header wheel to compile kernels)
venv/bin/python test_all.py # QA suite, no network; prints its own count
# (529 tests / 34 modules as of 2026-09-17)
# (536 tests / 35 modules as of 2026-09-18)
venv/bin/python main.py --pair XAUUSD --timeframe H1 --detail # one signal report in the terminal
# --json full dump, --panel adds the advisory LLM panel,
# --timeframe M5 | M15 | H1 | H4
@@ -41,7 +43,10 @@ realistic Oanda figure is 0.65 and must be passed explicitly), `--quality A+,A,B
`--export-csv out.csv`, `--detail`, `--set PATH=JSON` (research override of any config value; an
unknown path is a hard exit), `--export-features` (appends the 65 decision-time columns; never use
it on a parity run because it changes the file), `--fixed-rr-tp N` (research only, not shipped),
`--record DIR [--record-bars]` (research recorder, see section 9).
`--record DIR [--record-bars]` (research recorder, see section 9), `--backend auto|gpu|cpu` (`auto`
uses the CuPy GPU backend when CUDA works and falls back to NumPy with the reason in the `Backend:`
line; `gpu` refuses to fall back; `cpu` forces the NumPy twins for a parity run; the env var
`FB2_BACKEND` sets the default). Both backends produce byte-identical books (section 9).
`test_all.py` is **not pytest**. It uses its own `@test("name")` decorator that runs each test the
moment the file is imported. Run the whole file; it prints per-module results and exits non-zero
@@ -266,7 +271,8 @@ Arrows show who imports whom (generated from the code on 2026-09-15).
| File | Role |
|---|---|
| `backtest/gpu_backtest.py` | The backtester (CuPy on a GPU, NumPy otherwise; check the `Backend:` log line) |
| `backtest/gpu_backtest.py` | The backtester. The indicator precompute over the full history and the resolution of every trade run on the GPU when CUDA works (`--backend`, `Backend:` log line names the device or the fallback reason); the per-bar replay through `analyze_from_dataframes` is the live Python signal code in forked CPU workers by design and is 80–95% of wall time |
| `backtest/gpu_kernels.py` | CuPy backend state (lazy probe, never silent, sets `CUDA_PATH` to the pip header wheel) and the three CUDA kernels: EWM batch, pandas rolling mean/var replica, trade resolver. Each is a bit-for-bit twin of the NumPy/pandas code kept in `gpu_backtest.py` (`--fmad=false`, one thread per series or trade); the twins are pinned by Module 36 and by the reference MD5s |
| `backtest/replay_record.py` | Opt-in `--record` recorder: why every accepted trade was taken, plus calendar events. Book-neutral by contract |
| `backtest/replay_scorecard.py` | Read-only reader of recorded runs (`DIR --check` verifies its invariants) |
| `scripts/ff_calendar_scraper.py` | Builds `backtest/cache/calendar.parquet` (Forex Factory events) for the recorder |
@@ -281,7 +287,7 @@ Arrows show who imports whom (generated from the code on 2026-09-15).
| `app_config.py` | Loads `config.json` once per process; env var → config.json → refuse; validates keys |
| `config.json.example` | The schema of `config.json` (which is gitignored) |
| `deploy/README-dashboard.md`, `deploy/gold-dashboard.service` | Live-server checklist and systemd unit for the dashboard daemon |
| `test_all.py` | The QA suite (34 modules; Module 35 is the agent-panel v2 module) |
| `test_all.py` | The QA suite (35 modules; Module 36 is the GPU backend parity module) |
---
@@ -359,7 +365,9 @@ day, 30 kept), `logs/dashboard.log`. Read the databases with `mode=ro`, never `i
## 6. Testing
- `venv/bin/python test_all.py` runs everything with mocked network. Expect
`RESULTS: 529 passed, 0 failed, 529 total` (2026-09-17). The suite's hermetic app config points
`RESULTS: 536 passed, 0 failed, 536 total` (2026-09-18). On a box with a visible CUDA device
Module 36 fails if the GPU backend falls back (the reason is in the message); with
`FB2_BACKEND=cpu` or no CUDA the GPU-path tests print a notice and the NumPy twins are tested. The suite's hermetic app config points
both database writers at a suite-private temp dir; before 2026-09-17 the RSS fixture tests wrote
fixture headlines into the real `logs/newsdata.db`, and a Module 32 test now pins the isolation. One Twelve Data interval-map test
reaches the network and can time out on a slow connection; a rerun clears it.
@@ -406,8 +414,9 @@ day, 30 kept), `logs/dashboard.log`. Read the databases with `mode=ro`, never `i
Process rules that go with them: every backtest report follows
`backtest/instructions_for_backtest.txt`; every candidate is compared with a control run on the
same code and cache, never with an old book directly; the `Backend:` line and the "GPU cache
fallbacks: 0" line are part of the evidence; keep `docs/` in sync when behaviour changes.
same code and cache, never with an old book directly; the `Backend:` line (since 2026-09-18 it
names the CUDA device or the fallback reason) and the "GPU cache fallbacks: 0" line are part of
the evidence; keep `docs/` in sync when behaviour changes.
---
@@ -446,7 +455,9 @@ trade-level diff is needed; each run takes seconds to a few minutes.
Harness for every reference book: `venv/bin/python backtest/gpu_backtest.py --pair XAUUSD
--spread 0.65 --workers 12 --primary-tf <tf> --start <start> --end <end> --export-csv <out>`,
factory config, no `--set`, `--quality`, `--strategies`, `--confirm-tfs` or `--export-features`.
All of these run offline from the kept cache windows (NumPy CPU backend, 0 cache fallbacks).
All of these run offline from the kept cache windows (0 cache fallbacks). Since 2026-09-18 the
CuPy GPU backend and `--backend cpu` both reproduce every MD5 below; earlier reports say
`NumPy CPU fallback` because CuPy could not compile kernels on the dev box until then.
| Book | Window | Trades / R / PF | MD5 of the CSV |
|---|---|---|---|
@@ -466,7 +477,8 @@ live names (`XAU` still means `XAU_ATR`).
Other current-HEAD figures at $0.65: M5 W3 (3-year) 870 / +153.6R / PF 1.50; M15 W3 1273 /
+186.6R / PF 1.39; M5 12-month (2025-08-14 → 2026-08-14) 560 / +108.81R / PF 1.54;
H1 12-month (2025-08-04 → 2026-08-04) 91 / +22.2R / PF 1.79. All five MD5s above were
reproduced on 2026-09-15 after the cache prune.
reproduced on 2026-09-15 after the cache prune and on 2026-09-18 on both backends
(`backtest/research/gpu_backend_2026-09-18.md`).
Kept cache windows (`backtest/cache/oanda_XAU_USD_<tf>_<paddedStart>_<end>.parquet`; the padded
start is the window start minus 62 days for M5/M15/H1, 234 for H4 and 1,400 for D1, so the D1
@@ -515,6 +527,20 @@ MD5s. Scorecards from it are in-book statistics, never decisions.
`NewsEngine`.
- Twelve Data candles are not a drop-in replacement for Oanda (section 4).
- `pyarrow` (Parquet) is installed in the dev venv only; live code never touches Parquet.
- CuPy from pip (`cupy-cuda12x`) cannot compile a kernel without the CUDA headers; the
`nvidia-cuda-runtime-cu12` wheel provides them and `backtest/gpu_kernels.py` points `CUDA_PATH`
at it. Until 2026-09-18 that failure was swallowed: every book was built on NumPy while the file
was called a GPU backtester. Now the fallback reason is in the `Backend:` line and Module 36
fails on a box that has a visible GPU but no working backend. `cupy` is banned from the live
roots by the same AST guard as scikit-learn.
- The CUDA kernels are bit-for-bit replicas of the CPU code: one thread per series (EWM, rolling)
or per trade, compiled with `--fmad=false`. Do not "optimise" them with a parallel scan or a
two-pass variance: the last bits change and a threshold can flip somewhere in 500k scans. A
single sequential series is not faster on the GPU than pandas' C loop; the win is the batch
across series and trades, and the replay itself stays on the CPU (same code as live).
- The replay workers are forked after the CUDA context exists and never touch the device (every
cache array is host NumPy, asserted before the fork). Python 3.12 may warn about forking a
multi-threaded process under `-X dev`; that is expected.
- The dashboard's due-timeframe check uses the raw wake time; flooring `wake − buffer` skipped
every H1 and H4 scan once (2026-08-02).
- The agent panel's price context reads the frames `analyze_pair` stashes on `report._frames`.
@@ -535,6 +561,7 @@ One line per validated change or verdict, newest first. Full evidence in the nam
| Date | Decision | Where the evidence is |
|---|---|---|
| 2026-09-18 | GPU backend made real: CuPy had never compiled a kernel on the dev box (`CUDA_PATH` missing for the pip layout), so every book to date was NumPy. SHIPPED: `backtest/gpu_kernels.py` (lazy, loud backend probe; `nvidia-cuda-runtime-cu12` header wheel; CUDA kernels for the EWM batch, the pandas rolling mean/var replica and the trade resolver, each a bit-for-bit twin of the CPU code), `--backend auto/gpu/cpu` + `FB2_BACKEND`, Module 36. All five reference MD5s reproduced on the GPU and with `--backend cpu`; precompute GPU == CPU on all 42 cached windows (2.34 M bars). Wall time roughly unchanged: the replay is CPU by design | `backtest/research/gpu_backend_2026-09-18.md` |
| 2026-09-17 | Local news scorer evaluated (the owner's scikit TF-IDF model in `~/Development/ML-NEWS`, trained on 606 DeepSeek gold labels): golden set 39/78 vs DeepSeek 78/78 vs keywords 24/78; out-of-sample agreement with DeepSeek 64% (DeepSeek agrees with itself 90%); cleaning / negation marking / label weighting move nothing; backfilling the 961 missing gold labels lifts macro-headline agreement 38% → 52% but only reshuffles golden-set errors. Verdict: a fallback, not a replacement. SHIPPED to grow a complete corpus: `xau_score` archived for every headline, no-currency headlines scored and archived (never shown), every new headline sent in its cycle (25 per call). Dashboard process made database-read-only (`set_db_writes_for_process`); the bot is the single writer. Books unaffected (news never enters signals) | `~/Development/ML-NEWS/eval/RESULTS_2026-09-17.md` (evaluation scripts live outside the repo by owner instruction) |
| 2026-09-17 | Live review of two weeks of `signals.db`: live == backtest bar for bar (26/29 trades matched, R within 0.14); the raw advisory −7.15R is cluster re-entries the validated books never take (cooldown view −1.6R, backtester +1.1R on the window, an ordinary fortnight); composite score inversely related to the 24 h move on all timeframes (two-week bear reading, test in-book with `--record-bars`); big-candle gate re-tested and REJECTED again (removes +32R / +60R of net-positive trades). Agent panel v2 SHIPPED: probabilities anchored to book base rates, code-computed decision context, no news doctrine, mean aggregator (`verdict_source: "mean"`), prompt version logged, offline replay harness; owner decision the same day: only the macro persona runs live (`personas: ["macro"]`, 1 call per panel; technical / risk / contrarian carried no information in either evaluation, kept for research); backtester `strategy` tag fixed (label-only, books re-pinned). Delivery findings: dashboard daemon on stale code since Sep 10, bot outage Sep 9 | `backtest/research/signal_log_review_2026-09-17.md`, `backtest/research/panel_v2_2026-09-17.md` |
| 2026-09-15 | Backtest folder cleanup: `diagnose_filters.py`, `shadow_scan.py`, and the two one-time news scripts removed (last commit with them `931b265`); every pinned CSV, the research output folders, the BTC/ETH cache snapshots, the Tiingo caches and 13 uncited Oanda windows moved to `~/Development/ForexBot2_trash_2026-09-15/` (346 MB) pending the owner's delete; all five reference MD5s reproduced afterwards. Suite 506 / 33 | commit `b0b71e1` |