2cc5915b006e7ca1b6175e16ef634da516743c7e
8 Commits
| Author | SHA1 | Message | Date | |
|---|---|---|---|---|
 Harwinder Sekhon
|
2cc5915b00 |
Daily-trend veto on M15 + lot-size / already-open lines (campaign 2026-09-30), and the 2026-10-01 whole-codebase bug review
Two bodies of work that touch the same files, so one commit. Direction and setup-size campaign (2026-09-30 → 10-01, owner's RISK-FIRST rule, written before the runs): - SIGNAL: M15 daily-trend veto (factory override 7, xau_strategy.daily_trend_veto "both"). A finished setup against a clear daily trend (closed D1 bars, Sunday stubs dropped; close vs its 50-bar EMA AND vs the close 60 kept days earlier) is removed as the last step of the core. One definition (signal_aggregator.daily_trend_series) for live, the bar replay and the twin's host side; D1 is a required frame of the M15 scan; mirrored in cuda_signal_kernels.py K2. W3 824 / +205.12R / PF 1.73 / max DD 12.4R vs 1105 / +195.79R / 1.49 / 20.9R; 2019 → 2026 1473 / +223.33R / 1.42 / 41.5R vs 2091 / +192.53R / 1.24 / 64.8R; unseen holdout −4.62R vs −6.63R. - FAILED, not shipped: the veto on M5 (W3 drawdown 10.1R → 15.2R) and on H1 (−9.7R), buy-only and sell-only on M15, the wide-stop cap (xau_strategy.wide_stop_cap_atr stays 0.0, a research knob). - New M15 pins: W2 dfc647ee…, OOS e87b6de1…, W3 ad3b0dbd…. M5 / H1 / H4 pins unchanged. --set xau_strategy.daily_trend_veto=off reproduces the previous M15 pins byte for byte. Twin and reference script MD5-equal on every window; recorder and feature-export parity re-proved; base-rate table rebuilt. - DISPLAY (books byte-identical): exposure.py — the lot size for trade_size.risk_per_trade (new config.json block, off by default, private to the owner) and the count of alerted same-direction plans still open (signals.alerted_at, read-only open_alerted_setups); the daily trend in words on Telegram, the CLI and the dashboard; signals.daily_trend logged. - providers/oanda: a date window with no bars is an empty page, not an error (a D1 pad starting on a Saturday could not be fetched). - Reports only, nothing shipped: seven context screens (no state nominated) and eight outside data sources (none adds a forecast to gold's own daily trend). - Independent verification (report section 5.13): every figure reproduced on both scripts, scoring identical to the previous commit on all four timeframes, no look-ahead. Caveats recorded there: this is a risk filter, not a more accurate direction call; nine tenths of the 2019 → 2026 gain is 2022; the neighbour clause is thin (a simple 100-day average misses the R line by 1.2R); "never adds a trade" is an observation, not a guarantee. Bug review of the whole codebase (2026-10-01). Signal path clean. 17 defects elsewhere, plus 4 found by an independent review of the fixes; all fixed, each with a test; no trade book changed: - telegram_bot: one _deliver_alert for the candle scan and the level break. The row is stamped alerted_at BEFORE the send, on a one-worker alert-db pool, and un-stamped when no chat received it or the task is cancelled mid-stamp, so two alerts at one candle close count each other and no SQLite write runs on the event loop. The 5-bar cooldown counts bars of the scan's own frame (== gpu_backtest._apply_cooldown; clock time released the 22:00 UTC bar after the settlement pause) and is restored from signals.db at start-up. The lot-size line goes to the owner's own chat only (a group the owner ran /autoscan in used to receive it). /status gains a news-model line and escapes failure details. - signal_log: mark_alerted returns the stamp it wrote; unmark_alerted; last_alerted_bars (read-only). - advisory_outcome: candle continuity is judged bar by bar inside the walk; a hole after a trade's exit no longer voids it. The stored-candle replay resolves 108 of 111 logged setups instead of 99, all equal to the live resolver's R. - providers/oanda: an empty date window is accepted only when it is 5 days or shorter and no malformed candle was skipped; a longer one is retried, then refused, instead of being cached as history with a hole. - news_engine: blank RSS titles skipped; one unreadable headline no longer costs the local model its whole batch (checked on the real bundle); model health recorded; no DeepSeek audit call in a process that does not archive; event-direction cache keyed per release; peek_week_events never fetches. - llm_client: chat_json never raises and never logs the API key (requests quotes the Authorization value in InvalidHeader; error bodies are redacted). - signal_aggregator: the scan's news block is guarded; the replay path (use_news=False) assigns exactly the values it did before. - agent_panel: PANEL_PROMPT_VERSION v2.2 — same prompt text, corrected input (event timing over the cached calendar week, headlines newest first). scripts/panel_replay.py replays the logged payload per persona; scripts/label_scan_outcomes.py counts with an ISO cutoff. - test_all.py: no ALTER TABLE … DROP COLUMN (the live box has SQLite 3.34); tests that write Parquet print a notice and return without pyarrow. Verification after the last code change: suite 632 passed, 0 failed (also with both Parquet engines blocked and FB2_BACKEND=cpu, the live box's view); the five reference MD5s reproduce on both backtesters, 0 cache fallbacks; Module 37 green; static Python 3.9 / SQLite 3.34 check clean on every touched file. Evidence: backtest/research/daily_trend_campaign_2026-09-30.md, context_screens_2026-10.md, new_inputs_2026-10.md; CLAUDE.md sections 2, 4, 6, 8, 9, 10 and 11; pair_signals/xauusd.py docstring (M15 override 7). |
||
|
Harwinder Sekhon
|
32ad73c59a |
backtest: full-GPU research twin (cudaBasedBacktesting.py + cuda_signal_kernels.py); books byte-identical to gpu_backtest.py (2026-09-18)
Owner decisions: research only (live scans stay on the CPU code), gpu_backtest.py stays the base reference, every research flag supported, the one exemption from instructions item 6 recorded in backtest/instructions_for_backtest.txt. - backtest/cuda_signal_kernels.py: the CUDA decision stage. K1 = engine.py + the GPUTechnicalEngine overrides (one thread per bar), K2 = analyze_from_dataframes end to end incl. gates 6b-6i, the four setup generators and the regime filter (one thread per primary bar), K3 = position_sizing.build_features, K4 cooldown, K5 prefix median, K6 recorder bar rows. Config packer with generated #defines; leaf module; --fmad=false; Python evaluation order kept (ternary min/max, pairwise sums, Neumaier sum(), exact round(x, 2)). - backtest/cudaBasedBacktesting.py: same CLI, cache files and writers as the reference; the host turns kernel codes into the exact strings; no CPU path. - test_all.py Module 37 (16 tests): every stage pinned against the live code; suite 552 / 552 on the GPU, without CUDA and with FB2_BACKEND=cpu. - Proof: five reference MD5s, four 6-month acceptance books (M5/M15, $0.65/$0.30), M15 OOS --export-features and --record --record-bars tables, ten research-flag variants, and the owner's five-year M5 and M15 books, all byte-identical to gpu_backtest.py. Wall time 0.8-6.6 s per book against 13-353 s. - CLAUDE.md sections 1, 3, 6, 7, 9, 10, 11 and the instructions addendum: any signal change is now a parity event for the twin. |
||
|
Harwinder Sekhon
|
02589cbb88 |
backtest: real CuPy GPU backend (kernels are bit-identical CPU twins); books byte-identical on both backends (2026-09-18)
CuPy from pip had never compiled a kernel on the dev box (no CUDA_PATH for the splayed CUDA-12 wheel layout); the probe swallowed the error and every book to date was built on the NumPy path under a "GPU backtester" name. - backtest/gpu_kernels.py (new): lazily probed, never-silent BACKEND state (CUDA_PATH pointed at the nvidia-cuda-runtime-cu12 header wheel; the probe compiles a kernel and keeps the failure reason), --backend auto|gpu|cpu and FB2_BACKEND, and three CUDA kernels compiled with --fmad=false: ewm_batch (the _gpu_ewm_alpha recurrence, one thread per series), roll_mean_var (pandas 3.0.2 roll_mean/roll_var replica, Kahan + Welford + unstable recompute), resolve_trades (trailing and static exits, one thread per trade). - gpu_backtest.py: precompute_indicators and simulate_trades dispatch to the GPU twins or to the untouched NumPy paths; caches are host NumPy and device memory is released before the replay workers fork; the Backend: line names the device or the fallback reason. - test_all.py Module 36 (7 tests): a visible CUDA device must never fall back silently; kernels vs their CPU references on synthetic edge cases; backend selection; import guards (cupy banned from the live roots). Suite 536. - CLAUDE.md, instructions_for_backtest.txt, research note with the evidence. Verification: all five reference MD5s reproduced on the GPU and with --backend cpu (ten runs, GPU cache fallbacks 0); precompute GPU == CPU on all 42 cached windows (2.34 M bars); suite green with GPU, with CUDA_VISIBLE_DEVICES="" and with FB2_BACKEND=cpu; --workers 1 and --record pairs equal. The per-bar replay stays the live Python signal code on CPU workers by design; wall time is roughly unchanged (precompute ~3x faster). |
||
|
Harwinder Sekhon
|
daa2369ccb |
backtest: --record/--record-bars Parquet recorder + 2-year M5/M15 replay with indicator weights and calendar
Adds an opt-in recorder to the backtester so a replay keeps the evidence each trade setup was decided on: every indicator's vote, value, confidence, base weight, regime multiplier, effective weight and contribution on the primary and both confirmation timeframes, penalties, the setup and outcome, and the Forex Factory calendar events around the trade (4 h before entry to exit, all-day rows on the New-York date). --record-bars adds one float32 row per analyzed bar with the same votes plus 1 h/4 h/24 h forward moves in ATRs. Output is Parquet in a folder per run with a manifest carrying the engine/gate fingerprints, the regime weight table in force and the book MD5. Nothing in the signal core changes: the trade book is byte-identical with recording on or off (control-vs-recorded MD5 equal on both 2-year runs, M5 781 / +146.07R / PF 1.52, M15 793 / +117.62R / PF 1.41 at $0.65). Module 33 adds 12 tests (498 total): snapshot reproduces the engine's composite, absence is no vote, calendar join edges/phases/holiday, write/load round-trip, _replay_span parity with recording on, CLI flags, import and source guards. Also: backtest/replay_scorecard.py (first-look scorecards, calendar buckets, in-book and per-bar what-ifs, --check invariants), scripts/ff_calendar_scraper.py restored from git history (the only way to regenerate the gitignored calendar file), report in backtest/research/replay_recording_2026-09-09.md. Claude-Session: https://claude.ai/code/session_01Nc9Ninzyi18QkWezrxeBvB |
||
|
Harwinder Sekhon
|
c004e3f30b |
chore: keep only what the system needs — drop research tooling, study scripts, diagram, plan
Owner directive 2026-09-06: Dukascopy and similar research material is irrelevant to the running system; keep only code and files with a role in system functioning. Deleted (all in git history before this commit): - backtest/research/ tooling and data: book_report, book_diff, burst_map, burst_precursors, burst_indicator_map, open_trade_bursts, calendar_impact, watch_rule_eval, sizing_model, commit_ladder, llm_news_eval + llm_news_golden.csv, ff_calendar_scraper, duka_spread + duka_spread_by_hour.csv (the Dukascopy spread study), feed_divergence_oanda_vs_tiingo.csv, the panel dry-test JSONL, __init__.py - scripts/feed_divergence_study.py, scripts/signal_parity_study.py (completed studies; their reports keep the results) - docs/architecture.png + architecture_diagram.py (developer illustration) - docs/data_source_change_test_plan.md (Phase 4 shadow scan stays noted in CLAUDE.md) Kept: the campaign .md reports in backtest/research/ as the decision ledger CLAUDE.md cites (rule 1 evidence); scripts/shadow_scan.py (pending operational step); backtest/diagnose_filters.py (test-locked backtester diagnostic). Local, untracked material removed on this box: .superpowers/, walkforward/, baseline/, the commit-ladder and LLM-eval result folders, the Dukascopy boundary CSV; the sizing_2026-09 folder is pruned to the four 3-year baseline books CLAUDE.md names. .gitignore no longer lists the removed research outputs; the backtest instructions and CLAUDE.md no longer point at deleted tools. Suite: 461 passed, 0 failed (nothing imported the removed files). Claude-Session: https://claude.ai/code/session_0156YhogLytzj9P2iH1skGbL |
||
|
Harwinder Sekhon
|
c00ed4471b |
chore: repo hygiene — stale gitignore entries, finished plans, superseded research, doc drift
Removed (all recoverable from git history):
- llm_provider_switch_plan.txt: 2026-08-08 DeepSeek/Anthropic multi-provider plan that
was never implemented (llm_client.py is DeepSeek-only, no `anthropic` anywhere)
- docs/superpowers/specs/2026-07-21-centralized-config-design.md: spec of the
config.json migration that merged the same day; its plan sibling was already
deleted in
|
||
|
Harwinder Sekhon
|
6bf58fd9d4 |
research: 3-year stake-sizing campaign + violent-move studies (nothing changes signals)
Walk-forward "second opinion" stake multiplier (meta-labelling on 65 decision-time features, 3yr Oanda books @ $0.65) FAILED the pre-registered rule on M5 (clean null) and M15 (lift inside the label-shuffle null, explained by BUY-vs-SELL). Nothing that changes numbers ships; the 12-month M5 book re-run is byte-identical to the pinned one. Behaviour-neutral plumbing: - position_sizing.build_features (65 features; multiplier path returns 1.0, no artifact) - SignalReport.penalty_tags / penalty_total / momentum_fading (record-keeping) - TradeSetup.stake_multiplier = None (reserved) - gpu_backtest --export-features (default export stays the pinned 63 columns; extras forced to the tail even when the first trade lacks an indicator) Research tools (backtest/research): book_report, burst_map, burst_precursors, open_trade_bursts, calendar_impact, burst_indicator_map, sizing_model, watch_rule_eval. Findings recorded in backtest/research/sizing_campaign_2026-09.md: M5 stretch measures are positive in both halves; every open-trade protection against violent bars loses on both TFs; high-impact USD releases cause 20-31% of violent bars but trades held through them do better; violent bars come when price is already at an extreme, not from calm. Module 29 tests (13). Suite: 435 passed. Claude-Session: https://claude.ai/code/session_01RWV2yLt3YXHHGVYHSQU2EE |
||
|
harwinder
|
15a7074574 |
Add backtest scripts to version control
Track gpu_backtest.py, diagnose_filters.py, and instructions. Data files (CSVs, parquet cache) remain gitignored. |