Owner decisions: research only (live scans stay on the CPU code), gpu_backtest.py
stays the base reference, every research flag supported, the one exemption from
instructions item 6 recorded in backtest/instructions_for_backtest.txt.
- backtest/cuda_signal_kernels.py: the CUDA decision stage. K1 = engine.py + the
GPUTechnicalEngine overrides (one thread per bar), K2 = analyze_from_dataframes
end to end incl. gates 6b-6i, the four setup generators and the regime filter
(one thread per primary bar), K3 = position_sizing.build_features, K4 cooldown,
K5 prefix median, K6 recorder bar rows. Config packer with generated #defines;
leaf module; --fmad=false; Python evaluation order kept (ternary min/max,
pairwise sums, Neumaier sum(), exact round(x, 2)).
- backtest/cudaBasedBacktesting.py: same CLI, cache files and writers as the
reference; the host turns kernel codes into the exact strings; no CPU path.
- test_all.py Module 37 (16 tests): every stage pinned against the live code;
suite 552 / 552 on the GPU, without CUDA and with FB2_BACKEND=cpu.
- Proof: five reference MD5s, four 6-month acceptance books (M5/M15, $0.65/$0.30),
M15 OOS --export-features and --record --record-bars tables, ten research-flag
variants, and the owner's five-year M5 and M15 books, all byte-identical to
gpu_backtest.py. Wall time 0.8-6.6 s per book against 13-353 s.
- CLAUDE.md sections 1, 3, 6, 7, 9, 10, 11 and the instructions addendum: any
signal change is now a parity event for the twin.
CuPy from pip had never compiled a kernel on the dev box (no CUDA_PATH for the
splayed CUDA-12 wheel layout); the probe swallowed the error and every book to
date was built on the NumPy path under a "GPU backtester" name.
- backtest/gpu_kernels.py (new): lazily probed, never-silent BACKEND state
(CUDA_PATH pointed at the nvidia-cuda-runtime-cu12 header wheel; the probe
compiles a kernel and keeps the failure reason), --backend auto|gpu|cpu and
FB2_BACKEND, and three CUDA kernels compiled with --fmad=false: ewm_batch
(the _gpu_ewm_alpha recurrence, one thread per series), roll_mean_var
(pandas 3.0.2 roll_mean/roll_var replica, Kahan + Welford + unstable
recompute), resolve_trades (trailing and static exits, one thread per trade).
- gpu_backtest.py: precompute_indicators and simulate_trades dispatch to the
GPU twins or to the untouched NumPy paths; caches are host NumPy and device
memory is released before the replay workers fork; the Backend: line names
the device or the fallback reason.
- test_all.py Module 36 (7 tests): a visible CUDA device must never fall back
silently; kernels vs their CPU references on synthetic edge cases; backend
selection; import guards (cupy banned from the live roots). Suite 536.
- CLAUDE.md, instructions_for_backtest.txt, research note with the evidence.
Verification: all five reference MD5s reproduced on the GPU and with
--backend cpu (ten runs, GPU cache fallbacks 0); precompute GPU == CPU on all
42 cached windows (2.34 M bars); suite green with GPU, with
CUDA_VISIBLE_DEVICES="" and with FB2_BACKEND=cpu; --workers 1 and --record
pairs equal. The per-bar replay stays the live Python signal code on CPU
workers by design; wall time is roughly unchanged (precompute ~3x faster).
Agent panel v2: personas return p_win anchored to the book base rates with a
code-computed decision context; verdict = equal-weight mean (verdict_source
"mean"); only the macro persona runs live (agent_panel.personas ["macro"]);
prompt version and exact payloads logged; offline replay harness
scripts/panel_replay.py; base-rate table agent_panel_base_rates.json
(scripts/build_panel_base_rates.py). Backtester `strategy` column now carries
the generator's own stamp (label-only; five reference books re-pinned, all
62 other columns identical).
News corpus (after evaluating the owner's ML-NEWS scikit model against
DeepSeek: golden set 39/78 vs 78/78, out-of-sample agreement 64% against a
90% self-agreement ceiling; verdict: fallback only, not a replacement):
- NewsSentiment.xau_score + archive column xau_score keep DeepSeek's gold
score for every headline (930 of the first 1,546 DeepSeek rows had none);
an existing newsdata.db gains the column in place on first open
- headlines naming no tracked currency are scored and archived
(currencies_json []) but never returned to the platform
- _apply_llm_sentiment sends every new headline in its cycle, 25 per call;
a failed call ends the cycle's sending
Dashboard publisher never writes a database: main() calls
app_config.set_db_writes_for_process(False), honoured by signal_log_enabled()
and news_archive_enabled() (owner decision: the bot is the single writer).
Test suite: the hermetic app config now points both writers at a
suite-private temp dir — the RSS fixture tests were writing fixture
headlines into the real logs/newsdata.db — with a Module 32 test pinning it.
529 tests.
backtest/research/cycle_2026-09-10.md: pre-registered rules, HEAD baseline
on the 3-year window (three pinned MD5s reproduced), cost ladder and
concentration facts, bucket screen (one cycle-2 candidate: M15 grade-A
SELLs), C1 H1 divergence->0 REJECTED, C2 M15 RSI-multiplier ladder =
stable plateau, seeding-depth measurement and the two shipped parity
fixes. CLAUDE.md: live fetch depth, backtester pad, 501 tests, the
2026-09-10 state entry; corrected the docs note (no PDF builds exist)
and recorded pyarrow as a dev-venv-only dependency.
Claude-Session: https://claude.ai/code/session_01NQDjW8rXYrJR4JDWMWKUij
Owner-approved 2026-09-09 after the pre-registered weight test
(backtest/research/weight_test_2026-09-09.md). M15-only
technical.regime_weights_override = the shipped table with rsi x 0.5
(0.5 trending family / 0.75 ranging / 0.5 transitioning), derived in the
factory and pinned literally by the Module 9 equivalence test; a Module 33
test locks that M5/H1/H4 carry no override.
Evidence, 2-year Oanda window 2024-09-05..2026-09-05 at $0.65: 793/+117.62R
PF 1.41 -> 931/+146.14R PF 1.43 (fixed halves +27.9/+0.6R vs control); OOS
2024-06-01..2025-03-30 300/+40.59R PF 1.38 -> 355/+60.67R PF 1.49; 3-year
+151.90R -> +186.61R (max DD 15.5 -> 19.3R, 2026 -4R). Parity: the factory
run with no --set flag reproduces the validated candidate books byte-for-
byte (2-yr MD5 d185f099..., OOS 6f8f24b4...). M15 canonical book re-pinned
to backtest/oanda_m15_2yr_065_20260909.csv. Docs carry a one-line note.
Claude-Session: https://claude.ai/code/session_01Nc9Ninzyi18QkWezrxeBvB
First use of the --record evidence: a drift-adjusted per-bar screen ranked
regime-multiplier candidates, the top 3 per timeframe went through the full
2-year replay against same-harness controls plus the 2024-06..2025-03 OOS
window under a rule written before any run. M5: all three MACD candidates
fail (-10..-46R). M15: ema_trend x2 and divergence->0 (trending) fail;
RSI x0.5 in every regime passes every check (2-yr +117.6 -> +146.1R PF 1.43,
OOS +40.6 -> +60.7R PF 1.49, 3-yr +151.9 -> +186.6R) with the caveats recorded
(2026 -4R, 3-yr max DD 15.5 -> 19.3R, added trades mostly B-grade BUYs).
Nothing shipped; the passing candidate is an owner decision.
Claude-Session: https://claude.ai/code/session_01Nc9Ninzyi18QkWezrxeBvB
Adds an opt-in recorder to the backtester so a replay keeps the evidence each
trade setup was decided on: every indicator's vote, value, confidence, base
weight, regime multiplier, effective weight and contribution on the primary and
both confirmation timeframes, penalties, the setup and outcome, and the Forex
Factory calendar events around the trade (4 h before entry to exit, all-day rows
on the New-York date). --record-bars adds one float32 row per analyzed bar with
the same votes plus 1 h/4 h/24 h forward moves in ATRs. Output is Parquet in a
folder per run with a manifest carrying the engine/gate fingerprints, the regime
weight table in force and the book MD5.
Nothing in the signal core changes: the trade book is byte-identical with
recording on or off (control-vs-recorded MD5 equal on both 2-year runs, M5
781 / +146.07R / PF 1.52, M15 793 / +117.62R / PF 1.41 at $0.65). Module 33
adds 12 tests (498 total): snapshot reproduces the engine's composite, absence
is no vote, calendar join edges/phases/holiday, write/load round-trip,
_replay_span parity with recording on, CLI flags, import and source guards.
Also: backtest/replay_scorecard.py (first-look scorecards, calendar buckets,
in-book and per-bar what-ifs, --check invariants), scripts/ff_calendar_scraper.py
restored from git history (the only way to regenerate the gitignored calendar
file), report in backtest/research/replay_recording_2026-09-09.md.
Claude-Session: https://claude.ai/code/session_01Nc9Ninzyi18QkWezrxeBvB
Owner directive 2026-09-06: Dukascopy and similar research material is irrelevant to
the running system; keep only code and files with a role in system functioning.
Deleted (all in git history before this commit):
- backtest/research/ tooling and data: book_report, book_diff, burst_map,
burst_precursors, burst_indicator_map, open_trade_bursts, calendar_impact,
watch_rule_eval, sizing_model, commit_ladder, llm_news_eval + llm_news_golden.csv,
ff_calendar_scraper, duka_spread + duka_spread_by_hour.csv (the Dukascopy spread
study), feed_divergence_oanda_vs_tiingo.csv, the panel dry-test JSONL, __init__.py
- scripts/feed_divergence_study.py, scripts/signal_parity_study.py (completed studies;
their reports keep the results)
- docs/architecture.png + architecture_diagram.py (developer illustration)
- docs/data_source_change_test_plan.md (Phase 4 shadow scan stays noted in CLAUDE.md)
Kept: the campaign .md reports in backtest/research/ as the decision ledger CLAUDE.md
cites (rule 1 evidence); scripts/shadow_scan.py (pending operational step);
backtest/diagnose_filters.py (test-locked backtester diagnostic).
Local, untracked material removed on this box: .superpowers/, walkforward/, baseline/,
the commit-ladder and LLM-eval result folders, the Dukascopy boundary CSV; the
sizing_2026-09 folder is pruned to the four 3-year baseline books CLAUDE.md names.
.gitignore no longer lists the removed research outputs; the backtest instructions
and CLAUDE.md no longer point at deleted tools.
Suite: 461 passed, 0 failed (nothing imported the removed files).
Claude-Session: https://claude.ai/code/session_0156YhogLytzj9P2iH1skGbL
Removed (all recoverable from git history):
- llm_provider_switch_plan.txt: 2026-08-08 DeepSeek/Anthropic multi-provider plan that
was never implemented (llm_client.py is DeepSeek-only, no `anthropic` anywhere)
- docs/superpowers/specs/2026-07-21-centralized-config-design.md: spec of the
config.json migration that merged the same day; its plan sibling was already
deleted in b936d44. app_config's docstring pointer goes with it.
- scripts/compare_oanda_tiingo.py: Phase 0 recent-tail smoke test, superseded by the
Phase 1/2 study scripts (result recorded in the test plan)
- backtest/research/fetch_oanda_m15.py + oanda_td_divergence.py + its by-hour CSV:
the 2026-07-24 Oanda-vs-Twelve-Data study, superseded by the 2026-08-14 feed
divergence report (which keeps a one-line citation)
- backtest/research/burst_profile.py and hour_analysis.py: superseded by burst_map.py
and book_report.py
- backtest/research/panel_payload_drytest_2026-08-15.py: one-off live DeepSeek probe;
its .jsonl evidence stays
Added to git: the three 2026-08-14 research reports (feed divergence, re-pin
verification, signal parity), their CSV, and docs/data_source_change_test_plan.md -
CLAUDE.md cites them as standing evidence and their driver scripts were already
committed. The test plan no longer refers to auto-execution.
Hygiene:
- .gitignore: drop oanda_state.json (daemon removed 2026-09-03) and the vanished
backtest/trailgrid dirs; ignore .coverage
- docs/architecture_diagram.py only writes the PNG when run as a script (it used
to plot on import); test box refreshed to 461 tests / 31 modules; PNG regenerated
(the committed one still drew the removed daemon)
- backtest/instructions_for_backtest.txt: manual-journal head replaced by the
actual reporting checklist; the 2026-09-06 addendum is unchanged
- doc drift: README/publisher list all six published files; news_summary.php says
4 h (not 24 h) and drops the pre-id sha1 fallback whose window closed;
analyze_signal_log lists six lenses and --scan-outcomes; agent_panel cost model
says 4 requests by default; tiingo comment cites the NY-anchored weekend;
Module 21 title no longer says "execution lifecycle"
Suite: 461 passed, 0 failed.
Claude-Session: https://claude.ai/code/session_0156YhogLytzj9P2iH1skGbL
- backtest/diagnose_filters: unused BacktestConfig import
- pair_signals/__init__: unused PlatformConfig import
- scripts/shadow_scan: unused pandas import
- research burst_precursors / calendar_impact / duka_spread: unused sys / COST / struct
- test_all: MagicMock, StringIO, Colors and two function-local imports that
nothing in the test bodies used
- telegram_bot: ViolentWatch is only a type hint -> imported under TYPE_CHECKING
compare_oanda_tiingo.py and oanda_td_divergence.py are left alone here because
the hygiene commit deletes them.
Suite: 464 passed, 0 failed (unchanged).
Claude-Session: https://claude.ai/code/session_0156YhogLytzj9P2iH1skGbL
Walk-forward "second opinion" stake multiplier (meta-labelling on 65 decision-time
features, 3yr Oanda books @ $0.65) FAILED the pre-registered rule on M5 (clean null)
and M15 (lift inside the label-shuffle null, explained by BUY-vs-SELL). Nothing that
changes numbers ships; the 12-month M5 book re-run is byte-identical to the pinned one.
Behaviour-neutral plumbing:
- position_sizing.build_features (65 features; multiplier path returns 1.0, no artifact)
- SignalReport.penalty_tags / penalty_total / momentum_fading (record-keeping)
- TradeSetup.stake_multiplier = None (reserved)
- gpu_backtest --export-features (default export stays the pinned 63 columns; extras
forced to the tail even when the first trade lacks an indicator)
Research tools (backtest/research): book_report, burst_map, burst_precursors,
open_trade_bursts, calendar_impact, burst_indicator_map, sizing_model, watch_rule_eval.
Findings recorded in backtest/research/sizing_campaign_2026-09.md: M5 stretch measures
are positive in both halves; every open-trade protection against violent bars loses on
both TFs; high-impact USD releases cause 20-31% of violent bars but trades held through
them do better; violent bars come when price is already at an extreme, not from calm.
Module 29 tests (13). Suite: 435 passed.
Claude-Session: https://claude.ai/code/session_01RWV2yLt3YXHHGVYHSQU2EE
Verdict from the 3-week signal log (backtest/research/news_sentiment_value_2026-09-04.md):
news_sentiment is a trailing-24h echo of price (r +0.67 with the past day, +0.03 with
the next; most bullish 2 h after the 08-25 top, bearish 44 h later; most bearish at the
09-02 low, bullish 16 h later). Per-headline LLM labels are correct; the 12 h pool and
the averaging are what destroy the information. Both gating flags stay False.
- news_engine: _LLM_SUMMARY_SYSTEM rewritten for someone who has never traded (no
acronyms, two worked examples); summary payload newest-first with the platform's own
gold direction; reply cap 300 -> 400 chars.
- config: headline_max_age_seconds 43200 -> 14400; RSS items without a parsable
pubDate are dropped (they used to be stamped "now" and never aged out).
- news_engine.get_fundamental_context drops headlines whose every relevant score is
inside +-0.1 (_is_neutral_headline) before the per-currency mean; news_sentiment
from this date is a mean over directional <=4 h headlines (log comparisons split here).
- static: dashboard copy for the 4 h pool; news_summary.php prompt gets the same style rule.
- analyze_signal_log: --news (under --stored-candles) prints sentiment lead/lag with
split halves, sentiment at the window high/low with flip lags, and outcome buckets by
direction x news x macro stance; LogRow gains news_sentiment/event_risk; _row_index shared.
- tests: 406 -> 414 (age window, undated drop, neutral filter, summary prompt/payload/cap,
--news helpers, load_rows round-trip); RSS fixtures now carry a pubDate.
- docs: user/beginner guides, CLAUDE.md, deploy README.
Claude-Session: https://claude.ai/code/session_0169BgFtWjp1SEAa5392Q9jH
Validation (llm_news_eval.py harness + 78-headline golden CSV + verdict
doc): headline scoring 78/78 vs keyword baseline 24/78 — validated;
event gold-direction calls misread close K/M numeric comparisons ~11%
of the time (both claims cases in sample, JOLTS 8.79M vs 8.84M).
Fix: event payload now carries code-computed "vs_forecast" (from
EconomicEvent.surprise) and the prompt gains an explicit inverted-
metrics rule + worked claims example — iteration 1 (field alone) fixed
the arithmetic but exposed a masked inversion error. 18/18 on live
re-verification (4 misreads fixed, 11 regression cases clean, 3 fresh
both-side probes).
/status: new "News & AI" section — per-source health for the calendar
chain (which of FF/MyFXBook served), each RSS feed, and DeepSeek
(daily count, last error kind), written passively by the existing
fetch paths under _cache_lock. Failures WARN once on transition
(repeats DEBUG, recovery INFO) so the rotating log carries outage
stories without 10-min spam.
test_all.py: +5 Module 19 tests (351 total); also carries the Module 22
fixed-RR-TP tests from the prior research session (shared file).
Replay artifacts dir gitignored like commit_ladder.
Claude-Session: https://claude.ai/code/session_017mbXs9HRawZhD1uZ3ze7zQ
Hard TP at N×risk beats the trail on all 2024+ windows but the grid is
monotone to 3R (trend artifact) and M15 rr20 goes negative in 2021+2023
with doubled DD — trailing exits stay. Flag is research-only; its
Module 22 tests land with the next commit (shared test_all.py).
Claude-Session: https://claude.ai/code/session_017mbXs9HRawZhD1uZ3ze7zQ
Added: backtest/research/commit_ladder.py (recreates historical configs
via --set on the fixed harness; 2026-08-03 campaign confirmed HEAD is
the best config on all timeframes). Results stay local like all pinned
trade books.
Claude-Session: https://claude.ai/code/session_01Nsd8HcrV67GbCZXxZEn1Ma
- Delete back-compat shims data_provider.py / oanda_provider.py;
test_all.py now imports providers.* directly (ForexDataProvider name
retired; the two shim-only tests removed, 268 -> 266 tests)
- Delete backtest/research/calendar_store.py (no importer, no entry
point); config.py audit comment reworded
- Delete executed plan doc (2026-07-21 centralized-config) and the two
unbuildable guide PDFs; HTML guides remain the source of truth
- OANDA_MIGRATION_ROADMAP.md status header corrected: phases 1-2
shipped 2026-07-28, feed is live on Oanda; phases 3-6 still unbuilt
- Branches deleted locally + on both remotes: oanda_demo (merged),
web-dashboard, eurusd-modular, reversal-exit-alert (owner-approved
permanent deletion); July stash dropped
Claude-Session: https://claude.ai/code/session_019zR58p8ACW1oJ5QXVUZ5es
Gate 6h (big-candle entry gate): blocks all XAU entries (core, XAU_PB,
XAU_CP) when the primary TF's last 24h contains a bar with range >= K x
prior-bar ATR. Ships default-OFF: validation rejected every threshold
(K 3-6 on M5/M15 12mo) — removed trades were net-POSITIVE at every cell;
M5 quiet-window trades are the only losing bucket. Config knobs
big_candle_atr_multiple/lookback_hours, --big-candle backtest flag,
Module 17 tests.
M5 relaxation SHIPPED (validated 2026-07-30, Oanda bars, $0.30 spread):
m5_min_abs_score 30->25, m5_min_confirmation_score 10->5, hour-21 UTC
unblocked M5-only. Beats control in ALL 6 calendar years of the 5yr
replay (+230.9R/2319 PF 1.26 DD 21.3 vs +168.0R/2048 PF 1.21 DD 27.0),
12mo +107.4R PF 1.36 vs +84.6R PF 1.33, OOS +26.1R vs +17.2R, $0.70
spread +86.9R vs +66.4R. M15: every relaxation retested negative —
config unchanged. Declined-but-open: M5 hour-22 block, M15 hour-16
unblock (both fail 5yr/OOS). Evidence in the factory docstring.
Tooling: generic --set PATH=JSON research override in gpu_backtest
(typo-proof, logged); diagnose_filters filter_tags refreshed for gates
6d-6i with a staleness-guard test (Module 18); research scripts
hour_analysis/burst_profile/book_diff. Suite: 268 tests.
Claude-Session: https://claude.ai/code/session_014jqHpmNYmwYMowF6bPZyeD
Scripts + summary CSVs for the 2026-07-24 studies (real XAU spread ~$0.65
tradable hours; Oanda vs TD bars diverge ~11% of ATR with 10% direction
flips). Large raw data files (parquet, per-boundary samples) gitignored.
Claude-Session: https://claude.ai/code/session_01Mc7XKYyPySVXiUpcN82QEK
Scrapes Forex Factory weekly pages (epoch datelines, actual/forecast/
impact/betterWorse) into backtest/cache/calendar.parquet (gitignored;
rebuild with the scraper). Timestamps verified: NFP/CPI/ECB times exact
across DST, and high-impact event bars average 3.5x ATR with the spike
vanishing at +/-1h shifts. calendar_store provides precise_events and
event_window_mask for screens and counterfactuals.
Harwinder Sekhon