harry/ForexBot2
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

backtest: full-GPU research twin (cudaBasedBacktesting.py + cuda_signal_kernels.py); books byte-identical to gpu_backtest.py (2026-09-18)

Browse Source 
Owner decisions: research only (live scans stay on the CPU code), gpu_backtest.py
stays the base reference, every research flag supported, the one exemption from
instructions item 6 recorded in backtest/instructions_for_backtest.txt.

- backtest/cuda_signal_kernels.py: the CUDA decision stage. K1 = engine.py + the
  GPUTechnicalEngine overrides (one thread per bar), K2 = analyze_from_dataframes
  end to end incl. gates 6b-6i, the four setup generators and the regime filter
  (one thread per primary bar), K3 = position_sizing.build_features, K4 cooldown,
  K5 prefix median, K6 recorder bar rows. Config packer with generated #defines;
  leaf module; --fmad=false; Python evaluation order kept (ternary min/max,
  pairwise sums, Neumaier sum(), exact round(x, 2)).
- backtest/cudaBasedBacktesting.py: same CLI, cache files and writers as the
  reference; the host turns kernel codes into the exact strings; no CPU path.
- test_all.py Module 37 (16 tests): every stage pinned against the live code;
  suite 552 / 552 on the GPU, without CUDA and with FB2_BACKEND=cpu.
- Proof: five reference MD5s, four 6-month acceptance books (M5/M15, $0.65/$0.30),
  M15 OOS --export-features and --record --record-bars tables, ten research-flag
  variants, and the owner's five-year M5 and M15 books, all byte-identical to
  gpu_backtest.py. Wall time 0.8-6.6 s per book against 13-353 s.
- CLAUDE.md sections 1, 3, 6, 7, 9, 10, 11 and the instructions addendum: any
  signal change is now a parity event for the twin.
This commit is contained in:
Harwinder Sekhon
2026-09-18 16:11:47 -04:00
parent 02589cbb88
commit 32ad73c59a
8 changed files with 5602 additions and 6 deletions
Download Patch File Download Diff File Expand all files Collapse all files
CLAUDE.md
+43 -6
View File
@@ -23,7 +23,7 @@ cp config.json.example config.json # then fill i
venv/bin/pip install cupy-cuda12x "nvidia-cuda-runtime-cu12==12.9.*" # optional, dev box: backtester GPU backend
# (CuPy from pip needs that header wheel to compile kernels)
venv/bin/python test_all.py # QA suite, no network; prints its own count
# (536 tests / 35 modules as of 2026-09-18)
# (552 tests / 36 modules as of 2026-09-18)
venv/bin/python main.py --pair XAUUSD --timeframe H1 --detail # one signal report in the terminal
# --json full dump, --panel adds the advisory LLM panel,
# --timeframe M5 | M15 | H1 | H4
@@ -35,6 +35,9 @@ venv/bin/python dashboard_publisher.py --once # one scan +
venv/bin/python backtest/gpu_backtest.py --pair XAUUSD --days 365
venv/bin/python backtest/gpu_backtest.py --pair XAUUSD --start 2024-09-05 --end 2026-09-05 \
--primary-tf 15min --spread 0.65 --workers 12 --export-csv book.csv
venv/bin/python backtest/cudaBasedBacktesting.py --pair XAUUSD --start 2024-09-05 --end 2026-09-05 \
--primary-tf 15min --spread 0.65 --export-csv book.csv # research twin: same flags, every stage on
# the GPU, seconds per book, same MD5 (section 9)
```
Backtest flags worth knowing: `--spread` (round-trip cost in dollars; the default is 0.30, the
@@ -273,6 +276,8 @@ Arrows show who imports whom (generated from the code on 2026-09-15).
|---|---|
| `backtest/gpu_backtest.py` | The backtester. The indicator precompute over the full history and the resolution of every trade run on the GPU when CUDA works (`--backend`, `Backend:` log line names the device or the fallback reason); the per-bar replay through `analyze_from_dataframes` is the live Python signal code in forked CPU workers by design and is 80–95% of wall time |
| `backtest/gpu_kernels.py` | CuPy backend state (lazy probe, never silent, sets `CUDA_PATH` to the pip header wheel) and the three CUDA kernels: EWM batch, pandas rolling mean/var replica, trade resolver. Each is a bit-for-bit twin of the NumPy/pandas code kept in `gpu_backtest.py` (`--fmad=false`, one thread per series or trade); the twins are pinned by Module 36 and by the reference MD5s |
| `backtest/cudaBasedBacktesting.py` | Research-only full-GPU twin of the backtester (2026-09-18): same CLI, same cache files, same 63-column book byte for byte, but the per-bar decision runs in CUDA kernels instead of the live Python code, so a book takes seconds (M5 two-year 2.4 s against 115 s). Reuses the loader, the precompute, the resolver, the metrics and the writers from `gpu_backtest.py`; the host only turns the kernels' numeric codes into the exact strings. No CPU path: `--backend cpu` is refused. Its numbers count as evidence only while the MD5 parity with `gpu_backtest.py` holds (section 7) |
| `backtest/cuda_signal_kernels.py` | The CUDA decision stage: K1 (`engine.py` plus the `GPUTechnicalEngine` overrides, one thread per bar), K2 (`analyze_from_dataframes` end to end with the gates, the setup generators, XAU_PB, XAU_CP, the regime filter and the replay's per-bar rules, one thread per primary bar), K3 (`position_sizing.build_features`), K4 (cooldown), K5 (prefix median), K6 (recorder bar rows); the config packer and the name lists the `#define`s are generated from. A leaf module that never imports the live code; pinned by Module 37 |
| `backtest/replay_record.py` | Opt-in `--record` recorder: why every accepted trade was taken, plus calendar events. Book-neutral by contract |
| `backtest/replay_scorecard.py` | Read-only reader of recorded runs (`DIR --check` verifies its invariants) |
| `scripts/ff_calendar_scraper.py` | Builds `backtest/cache/calendar.parquet` (Forex Factory events) for the recorder |
@@ -287,7 +292,7 @@ Arrows show who imports whom (generated from the code on 2026-09-15).
| `app_config.py` | Loads `config.json` once per process; env var → config.json → refuse; validates keys |
| `config.json.example` | The schema of `config.json` (which is gitignored) |
| `deploy/README-dashboard.md`, `deploy/gold-dashboard.service` | Live-server checklist and systemd unit for the dashboard daemon |
| `test_all.py` | The QA suite (35 modules; Module 36 is the GPU backend parity module) |
| `test_all.py` | The QA suite (36 modules; Module 36 is the GPU backend parity module, Module 37 the CUDA decision-stage parity module) |
---
@@ -365,9 +370,12 @@ day, 30 kept), `logs/dashboard.log`. Read the databases with `mode=ro`, never `i
## 6. Testing
- `venv/bin/python test_all.py` runs everything with mocked network. Expect
`RESULTS: 536 passed, 0 failed, 536 total` (2026-09-18). On a box with a visible CUDA device
`RESULTS: 552 passed, 0 failed, 552 total` (2026-09-18). On a box with a visible CUDA device
Module 36 fails if the GPU backend falls back (the reason is in the message); with
`FB2_BACKEND=cpu` or no CUDA the GPU-path tests print a notice and the NumPy twins are tested. The suite's hermetic app config points
`FB2_BACKEND=cpu` or no CUDA the GPU-path tests print a notice and the NumPy twins are tested.
Module 37 (the CUDA decision stage) adds about 85 s on the GPU because its references replay
two synthetic fixtures through the live signal code bar by bar; without CUDA each of its device
tests prints a notice and only the arithmetic rule, packing, flag and source-guard tests run. The suite's hermetic app config points
both database writers at a suite-private temp dir; before 2026-09-17 the RSS fixture tests wrote
fixture headlines into the real `logs/newsdata.db`, and a Module 32 test now pins the isolation. One Twelve Data interval-map test
reaches the network and can time out on a slow connection; a rerun clears it.
@@ -416,7 +424,13 @@ Process rules that go with them: every backtest report follows
`backtest/instructions_for_backtest.txt`; every candidate is compared with a control run on the
same code and cache, never with an old book directly; the `Backend:` line (since 2026-09-18 it
names the CUDA device or the fallback reason) and the "GPU cache fallbacks: 0" line are part of
the evidence; keep `docs/` in sync when behaviour changes.
the evidence; keep `docs/` in sync when behaviour changes. Any change to `engine.py`,
`signal_aggregator.py`, `chart_patterns.py`, `position_sizing.py`, the recorder's row layout or
the `GPUTechnicalEngine` overrides is also a **parity event for the CUDA research twin**
(`backtest/cudaBasedBacktesting.py`): mirror it in `backtest/cuda_signal_kernels.py` in the same
commit, keep Module 37 green, and regenerate at least the M15 OOS book with both scripts and show
equal MD5s before a number from the twin is used as evidence. Until then its books are not
evidence.
---
@@ -480,12 +494,20 @@ H1 12-month (2025-08-04 → 2026-08-04) 91 / +22.2R / PF 1.79. All five MD5s abo
reproduced on 2026-09-15 after the cache prune and on 2026-09-18 on both backends
(`backtest/research/gpu_backend_2026-09-18.md`).
The CUDA research twin (`backtest/cudaBasedBacktesting.py`, section 3) reproduced all five MD5s
above on 2026-09-18 in 0.8–2.4 s per book, plus the four 6-month acceptance books (2026-03-18 →
2026-09-18, M5 and M15 at $0.65 and $0.30: `86111c36…`, `e70d0252…`, `b8c98083…`, `e66b5fbd…`,
each equal to the reference script's), the M15 OOS `--export-features` CSV (`6239d197…`) and
`--record --record-bars` tables, and ten research-flag variants
(`backtest/research/cuda_backtester_2026-09-18.md`).
Kept cache windows (`backtest/cache/oanda_XAU_USD_<tf>_<paddedStart>_<end>.parquet`; the padded
start is the window start minus 62 days for M5/M15/H1, 234 for H4 and 1,400 for D1, so the D1
and H4 files of one run have different start dates than its M5 file): 3-year `20230705_20260905`,
2-year `20240705_20260905`, OOS `20240331_20250330`, M5 12-month `20250613_20260814`, M15/H1
12-month `20250603_20260804`, 5-year `20210531_20260804`, 7.6-year `20181031_20260807`, the
2026-08-14 parity window `20250814_20260814`, their D1/H4 pads, and `calendar.parquet`.
2026-08-14 parity window `20250814_20260814`, the 6-month CUDA acceptance window
`20260115_20260918` (M5/M15/H1), their D1/H4 pads, and `calendar.parquet`.
The research recorder (`--record DIR [--record-bars]`) writes Parquet files describing every
accepted trade (indicator votes on all three timeframes, penalties, setup, outcome, calendar
@@ -541,6 +563,20 @@ MD5s. Scorecards from it are in-book statistics, never decisions.
- The replay workers are forked after the CUDA context exists and never touch the device (every
cache array is host NumPy, asserted before the fork). Python 3.12 may warn about forking a
multi-threaded process under `-X dev`; that is expected.
- `backtest/cudaBasedBacktesting.py` is a CUDA re-implementation of the decision stage, exempt
from "replays go through the real signal code" only while its book MD5 equals
`gpu_backtest.py`'s. A signal change that is not mirrored in `cuda_signal_kernels.py` makes the
twin drift silently: no test compares it with the live code except Module 37 and the paired
MD5 runs. It has no CPU fallback, refuses `--backend cpu`, and every description, warning and
reason string is produced on the host from the same f-string templates as the live code; the
kernels emit codes and numbers only.
- The kernels keep Python's evaluation order on purpose: `min`/`max` as argument-order ternaries
(never `fmin`/`fmax`), numpy's pairwise summation for `mean`/`nanmean`, pandas' online rolling
and EWM recurrences, Python 3.12's Neumaier-compensated `sum()` for the multi-timeframe mean,
`round(x, 2)` through an error-free product, `atr_percentile = count / 50 * 100` (not
`count * 2`: 15/50·100 = 30.000000000000004 flips "low" to "normal"). A cleaner formulation
flips a threshold somewhere in 500k bars. The as-of alignment compares int64 nanoseconds; a
mixed-unit index once returned slice length 0 for every confirmation frame.
- The dashboard's due-timeframe check uses the raw wake time; flooring `wake − buffer` skipped
every H1 and H4 scan once (2026-08-02).
- The agent panel's price context reads the frames `analyze_pair` stashes on `report._frames`.
@@ -561,6 +597,7 @@ One line per validated change or verdict, newest first. Full evidence in the nam
| Date | Decision | Where the evidence is |
|---|---|---|
| 2026-09-18 | Full-GPU research twin SHIPPED: `backtest/cudaBasedBacktesting.py` + `backtest/cuda_signal_kernels.py` (owner decisions: research only, `gpu_backtest.py` stays the base reference, every research flag supported, the one exemption from instructions item 6 recorded there). Proof: the five reference MD5s, four 6-month acceptance books (2026-03-18 → 2026-09-18, M5 and M15, $0.65 and $0.30), the M15 OOS `--export-features` CSV and `--record --record-bars` tables, and ten research-flag variants, all byte-identical to the reference script; Module 37 (16 tests) pins every stage against the live code. Books take 0.8–2.4 s instead of 13–115 s. Rule added: any signal change is a parity event for the twin | `backtest/research/cuda_backtester_2026-09-18.md` |
| 2026-09-18 | GPU backend made real: CuPy had never compiled a kernel on the dev box (`CUDA_PATH` missing for the pip layout), so every book to date was NumPy. SHIPPED: `backtest/gpu_kernels.py` (lazy, loud backend probe; `nvidia-cuda-runtime-cu12` header wheel; CUDA kernels for the EWM batch, the pandas rolling mean/var replica and the trade resolver, each a bit-for-bit twin of the CPU code), `--backend auto/gpu/cpu` + `FB2_BACKEND`, Module 36. All five reference MD5s reproduced on the GPU and with `--backend cpu`; precompute GPU == CPU on all 42 cached windows (2.34 M bars). Wall time roughly unchanged: the replay is CPU by design | `backtest/research/gpu_backend_2026-09-18.md` |
| 2026-09-17 | Local news scorer evaluated (the owner's scikit TF-IDF model in `~/Development/ML-NEWS`, trained on 606 DeepSeek gold labels): golden set 39/78 vs DeepSeek 78/78 vs keywords 24/78; out-of-sample agreement with DeepSeek 64% (DeepSeek agrees with itself 90%); cleaning / negation marking / label weighting move nothing; backfilling the 961 missing gold labels lifts macro-headline agreement 38% → 52% but only reshuffles golden-set errors. Verdict: a fallback, not a replacement. SHIPPED to grow a complete corpus: `xau_score` archived for every headline, no-currency headlines scored and archived (never shown), every new headline sent in its cycle (25 per call). Dashboard process made database-read-only (`set_db_writes_for_process`); the bot is the single writer. Books unaffected (news never enters signals) | `~/Development/ML-NEWS/eval/RESULTS_2026-09-17.md` (evaluation scripts live outside the repo by owner instruction) |
| 2026-09-17 | Live review of two weeks of `signals.db`: live == backtest bar for bar (26/29 trades matched, R within 0.14); the raw advisory −7.15R is cluster re-entries the validated books never take (cooldown view −1.6R, backtester +1.1R on the window, an ordinary fortnight); composite score inversely related to the 24 h move on all timeframes (two-week bear reading, test in-book with `--record-bars`); big-candle gate re-tested and REJECTED again (removes +32R / +60R of net-positive trades). Agent panel v2 SHIPPED: probabilities anchored to book base rates, code-computed decision context, no news doctrine, mean aggregator (`verdict_source: "mean"`), prompt version logged, offline replay harness; owner decision the same day: only the macro persona runs live (`personas: ["macro"]`, 1 call per panel; technical / risk / contrarian carried no information in either evaluation, kept for research); backtester `strategy` tag fixed (label-only, books re-pinned). Delivery findings: dashboard daemon on stale code since Sep 10, bot outage Sep 9 | `backtest/research/signal_log_review_2026-09-17.md`, `backtest/research/panel_v2_2026-09-17.md` |