Fixes all high+medium findings from the 2026-07-16 full-codebase audit.
High:
- data_provider: cache now rejects snapshots missing the latest closed
bar (_has_latest_closed_bar, weekend-exempt). The wall-clock TTL served
mid-candle snapshots at the next boundary, so live M15/H1 scans ran one
full bar behind and the bar-close session check evaluated the wrong hour.
- signal_aggregator: gate 6e (SELL quality) only acts on actionable
grades. It was re-grading session-blocked F reports to D, which
re-opened the M5 pullback path (entry check: quality != "F") on
session-blocked bars.
- gpu_backtest: port the live EMA-ribbon compression/expansion logic
(+10 conf on expansion) into the GPU engine. Its absence diverged
composite score/conf on 7-9% of bars and flipped signals on ~0.2%.
Medium:
- gpu_backtest: Bollinger precompute uses the exact pandas rolling calls
(cumsum E[x2]-E[x]2 lost precision at gold price levels; bb_position
error up to 7.6e-3 crossed signal thresholds, worst during squeezes).
- gpu_backtest: --confirm-tfs defaults to the pair factory's live
confirmation map instead of a hardcoded 1h,4h.
- gpu_backtest: entries re-anchor to the first RAW bar after the signal
bar, so >10-sigma spike bars dropped by cleaning are no longer skipped
at entry (favorable bias).
- gpu_backtest: per-TF warmup pad — D1 confirms get >=210 bars at window
start (62-day pad gave ~61 bars: no EMA200, unlike live).
- data_provider: thread-safety locks for cache + throttle (shared
provider across the bot's 4-thread pool); outlier filter keeps the
snap-back bar after a spike instead of punching a two-bar hole.
- telegram_bot: alert fingerprint gains a risk-relative entry bucket
(materially-moved setups re-alert) and is committed only after a
successful send (failed sends no longer consume the alert for 1h).
- engine: disabled_indicators=["macd"] now removes the MACD pillar from
scoring (histogram still computed for divergence).
Verification: test_all.py 187/187 (new Module 11: 8 audit-fix
regressions incl. a live-vs-GPU parity sweep with ribbon coverage);
real-data parity sweep over 813 slices of 14mo XAU M15: zero divergence
(pre-fix: 7-9% of bars diverged); both original bug repros no longer
reproduce; cached end-to-end backtest run passes with the live confirm
map. Books shift slightly vs pre-fix runs — comparability note added to
CLAUDE.md; pinned tuned numbers need re-validation under the fixed
backtester.
Fix 1 — mixed-"trending" regime (ADX strong, EMAs unaligned) was
hard-coded bullish in 5 sites: RSI reinterpretation used the uptrend
playbook (RSI 38-55 -> STRONG_BUY conf 75), BUY-only confidence
bonuses, BUY-only extension penalties, no counter-trend penalty.
Now: extremes-only RSI (as transitioning), no directional bonuses,
symmetric extension penalties. 6yr validation: H1 +67.2->+61.9R
(mixed skew 16B/11S -> 4B/7S), H4 +31.6->+27.5R (the 4 bias BUYs),
M15 +152.6->+154.7R, M5 +207.1->+206.1R with maxDD 28.7->24.3R.
The ~8R aggregate give-back is bull-market subsidy from the bug.
Fix 2 — live session filter now applies the EXACT replay-loop
predicate (session window + blocked hours on the signal bar's CLOSE
time, from the bar timestamp not wall clock). Replaces a wall-clock
filter gated on a "XAU STRATEGY" marker that no code writes, which
made live diverge from every validated book.
Fix 3 — delete dead _check_xau_oversold_bounce and
_generate_xau_trade_setup (unreachable; the latter hardcoded BUY and
skipped the spread/RR gates).
Backtester: parallel bar replay via forked workers (--workers,
default auto = cores-2 capped 12). Per-bar evaluation is pure; the
entry cooldown is applied in an ordered phase-2 pass. Proven
byte-identical to sequential on the same window; ~300 -> 1030-1680
bars/s (6yr M5: ~26min -> 5-8min). --workers 1 = sequential path.
Tests: 179 passed (new mixed-regime RSI regression test).
M15 mixed-"trending" regime block (ADX strong, EMAs unaligned =
momentum without direction): -6.8R/24 trades on 12mo, negative both
halves; blocking scores +65.7R/499 PF 1.38 DD 10.4R vs +59.0R/521
PF 1.32 DD 12.6R, better both halves, OOS 2024-06->2025-03 neutral
(-1.4R = the 4 such trades there). M5 never trades this regime.
New XauStrategyConfig.blocked_regimes / sell_blocked_regimes knobs
(default empty) + aggregator gate 6g2; M15 factory opts in.
M5 hour-9 UTC block (post-London-open whipsaw): -28.9R/143 over the
6-year book, negative 5/7 calendar years. Full-history replay with
block: +207.1R vs +178.3R, max DD 28.7R vs 35.5R, 2023 -13.0R ->
breakeven; tuned 12mo book unchanged (139.0 vs 139.3). Hour 9 is
positive on M15 (+1.7R 6yr, +8.6R OOS) so the block is factory
M5-only.
Rejected after validation (documented in xauusd.py + session memory):
transitioning-SELL block (bull-market artifact, +5.7R in 2024 OOS),
M15 trending_down block (M5 cross-TF contradiction), losing-streak
circuit breakers, 30d efficiency-ratio trend gate, and all entry-level
indicator filters (winners == losers at entry on every feature).
Full-history context (Twelve Data 5min starts 2020-03-16): M5 6.3yr
+207R PF ~1.3 post-fix with losing year 2021; M15 +152.6R PF 1.19
DD 49.7R — edge is regime-dependent, chop years are structural.
Tests: 178 passed (new gate test + factory-equivalence updates).
Replaying the live danger/caution gates against the validated 12mo XAU
M5 book showed they suppress 25-35 trades worth +4.6..+7.2R/yr — trades
entered near USD releases outperform the book average (+0.475R vs
+0.256R). New NewsConfig.event_risk_caps_signals (default False =
XAU-tuned) keeps EVENT RISK warnings in alerts but no longer guts
confidence; EUR factory re-enables the caps (evidence is gold-specific).
Pinned XAU M5 regression: 547 trades, +139.94R, trade-identical.
Tests: 177 passed.
The SELL quality gate (6e), SELL volatility gate (6g), M5 noise gates
(6h), ATR trade-setup generator, and M5 pullback archetype are all
gold-tuned but previously keyed only on xau_strategy.enabled or the
timeframe, so they would have silently applied gold constants to any
other pair. Each now also requires display_name == "XAU/USD".
No behavior change for XAU (conditions are tautologically true for the
only registered pair today) — verified by test suite and pinned-window
backtest CSV diff.
Second entry pattern alongside the Keltner-breakout: enter on the
retrace to EMA21/KC-mid in an established trend (EMA >=3/4 aligned,
ADX>=20, retrace 0.8-6.0 ATR, resumption bar closed back in trend
direction, RSI in the pullback zone not the extended zone). Detector
lives in the aggregator so live and GPU-backtest paths share one
implementation. Own gates (composite floor 15, confirmation-TF
agreement, MTF opposition, vol regimes) with a diagnostic funnel.
PB stops anchor to the pullback swing (~$8 vs ~$25 breakout), so PB
trades get their own trail arm (0.8R vs 0.6R global) via per-trade
trail overrides in the backtester and strategy-aware Telegram alerts.
12mo M5 validation: PB alone +38.3R/254 trades, positive both halves
and all four full quarters; core breakout trades undisturbed
(+101.9R vs +105.0 baseline). Combined book: 549 trades (~10.6/wk),
+140.1R, PF 1.78, DD 11.3R vs 311 trades/+105.0R before.
Gates 5-6 in the M5 filter: block 'high' vol regime entries (chop
transition — 61% WR, negative total while low/normal/extreme all
positive) and block SELLs with RSI<25 (dip already extended, mean-
reverts before target). 365d: 419→311 trades, +41→+48.4R, DD 11.2→8.0R.
Replace the fixed RR-0.5 take-profit with a trailing stop: arm at
+0.6R favorable excursion, trail 0.75R behind best price, stop levels
update at bar close (conservative). The RR-0.5 target was the binding
constraint — winners kept running after TP. Whole trail parameter grid
(arm 0.4-1.2 × dist 0.5-1.25) lands +90-103R, a plateau not a spike;
also +91.7R on the pre-gate 419-trade population. TP1 stays advisory;
Telegram alerts include the arm price and trail distance. max_hold
200→300 bars (trailing exits need room; +10R).
12mo M5, same 311 trades: +105.0R total, PF 2.27, DD 7.9R, 61% WR,
train/val +54.6/+50.4, every quarter positive.
ATR was computed after EMAs so ribbon compression detection was dead code
on all timeframes — moved ATR first. Extended M5 swing S/R lookback from
50 to 120 bars. Added four M5-specific filters: min score, MTF agreement,
confirmation TF conviction, and SELL quality cap (A-grade M5 shorts enter
at dip extremes and lose). Fixed backtester blocked-hour check to use bar
close time matching the live bot. Added trade geometry config (spread gate,
SL floor, TP floor RR) and primary_tf mismatch guard in backtester.
Backtest: 1133 trades/+11.5R → 419 trades/+41.0R, PF 1.03→1.37, DD 21→11R.
Train/val split positive in both halves (+19.8R/+21.3R).
Live pipeline:
- Drop still-forming candle before analysis (signals no longer repaint)
- Direction-aware MTF agreement + opposition gate (no trade against 2-of-3 TF consensus)
- SELL volatility gate: block shorts in low/normal vol regimes (config sell_vol_regimes)
- Divergence: 40-bar lookback, price/indicator swings must align within ±2 bars
(fired on 0.5% of trades before, 29% after), trending weight 2.0→1.0
- RSI regime_aware by default; transitioning regime only signals at extremes
- ADX ±DM tie no longer nudges bearish; TP2 clamped beyond TP1
Backtester realism:
- Charge $0.30 round-trip spread (--spread), gap-through SL/TP fills at bar open
- Resolve trades on RAW price path (cache stores raw bars, cleaning runs on load)
- Weekend/session filters use bar-close decision time to match live wall clock
Validated per-step on M15 2025-03-30→2026-05-31 under identical pricing:
+46.9R → +66.8R, win rate 69.9% → 73.0%, SELL side -2.3R → +9.6R.
Tested-and-reverted: ROC weight removal (-13R), BB fade suppression (-11.6R),
candlestick de-weight (-2R) — composite thresholds are co-tuned with weight mass.
The TP floor calculations used raw min_rr (0.4) but after 20% tightening
the effective RR dropped to ~0.32, failing the min_rr check and returning
None for all trade setups. Pre-compensate the floor with min_rr/tf so
post-tightening RR still meets the minimum.
The old `weekday() >= 5` check blocked all of Sunday, but the forex
market reopens Sunday 22:00 UTC (Sydney open). Now uses proper time-based
check so the bot fetches data and generates signals on Sunday evening.
A-grade trades entered exhausted moves with TP too far (0.75R), causing
57.7% WR and +0.25R over 30 days. Store exhaustion_count on SignalReport
and scale TP to B-level (1.0×) when A-grade has 2+ exhaustion tags.
Backtest: A-grade 73.1% WR / +3.25R, B/C unchanged.
The general session filter ran after analyze_from_dataframes() and
wiped out XAU oversold bounce signals at most hours. Now skips the
session filter when the XAU strategy has already fired — the strategy
has its own backtested hour filter.
BUY-only strategy for gold: enters when M15 RSI drops below oversold
threshold while price remains above EMA200 on all three timeframes
(M15, H1, H4). Uses ATR-based TP/SL (0.7x/2.0x) sized from M15 ATR.
Isolated to XAU/USD only — other pairs completely unaffected.
- Drop USD/JPY from scan pairs (zero signals across all timeframes)
- Disable divergence indicator (zero impact in 14-config lab test)
- Parallelize evaluate.py with ProcessPoolExecutor (16 cores)
- Add --pair CLI filter to evaluate.py for single-pair backtests
- Add data freshness guard: stale cache auto-invalidated, primary TF
stale data suppresses signals to prevent trading on old data
- Add /cache command: shows fetch age and last bar per pair/TF
- Add /clearcache command: wipes in-memory cache for fresh API fetch
- Remove BB from WebSocket breach alerts (keep structural levels only)
- Show live prices only for WS-monitored pairs (EUR/USD, XAU/USD)
- Replace arrow emojis with text tags in indicator lines
- Trade setup TP now scales to SL distance ensuring min RR is met when
no viable swing target exists; skip swing targets with poor RR
- Add float tolerance to RR gate to prevent edge-case rejection
- M5 auto-scan runs for level cache only, no Telegram broadcast
- WebSocket limited to EUR/USD + XAU/USD (Grow plan constraint)
- Register bot command menu via set_my_commands on startup
Expand from 2 to 4 pairs (EUR/USD, USD/JPY, GBP/USD, XAU/USD) with
automatic scanning at M15 and H1 candle close boundaries. Scheduler
staggers H1 by 60s to stay within 8 credits/min; daily budget is
480 of 800 allowed. Cache TTL is now candle-duration-aware so cross-TF
confirmation data is reused without extra API calls. Spread estimate
uses primary-TF data instead of fetching M5 separately.
Remove backtest.py, backtest_gpu.py, metrics.py, local CSV data,
Dukascopy loader, and all backtest CLI modes from main.py. Rebalance
engine weights: leading indicators (divergence, RSI, stochastic, BB)
now carry 66% of signal weight vs 34% for lagging (EMA, MACD). Add
ws_twelvedata.py for live tick streaming. Drop pyarrow and tqdm deps.
## What changed
### backtest_gpu.py (new — ~1,300 lines)
Standalone GPU-accelerated backtester completely separate from backtest.py.
Computes all indicator series in one vectorised O(N) pass over the full price
history, then replays positions with a sequential state machine.
Key design:
- VectorIndicators: pandas ewm/rolling (C-optimised) for all 7 indicators
- CuPy GPU backend with automatic NumPy fallback (no hard dependency)
- Two-phase API: prepare_arrays() once, score_and_run() many times
- --optimize flag: 36-combo weight sweep (1 baseline + 7 ablations + 28
multiplier runs) — calls prepare_arrays() once, replays all combos in seconds
- Full CLI: --tf, --local, --start/--end, --regime-filter, --regime-tf,
--session-filter, --optimize, --journal, --journal-csv
O(N) vs O(N^2): backtest.py recomputes all indicators per bar (O(N^2));
backtest_gpu.py computes each indicator series once and reads bar T in O(1).
On 100,000 H1 bars: ~4 seconds vs >10 minutes.
### data_provider.py
Added load_local_csv() method — reads MetaTrader tab-delimited CSV exports
from market_data/ directory. Format: Time|Open|High|Low|Close|Volume|Spread
(spread column dropped on load, timestamps localised to UTC). Applies the same
_validate_and_clean() pipeline as the Twelve Data path. LOCAL_CSV_MAP maps
each Timeframe enum to its filename.
market_data/ files added (MetaTrader exports, UTC):
USDJPY_M15.csv 100,000 bars 2022-05-04 to 2026-05-08
USDJPY_H1.csv 100,000 bars 2010-04-23 to 2026-05-08
USDJPY_H4.csv 25,855 bars 2010-04-22 to 2026-05-08
USDJPY_D1.csv 5,020 bars 2010-04-22 to 2026-05-08
16 years of H1/H4/D1 vs ~6 weeks of H1 on Twelve Data free plan. Use --local
for all backtests; Twelve Data reserved for live signal feed.
### engine.py — optimised indicator weights
36-combo sweep on H4 2022-2026 found MACD and EMA trend are the dominant
signals; RSI and Bollinger are net drag on this trending pair.
Old → new weights:
ema_trend: 1.2 → 3.6 (×3 — strongest directional signal)
macd: 1.1 → 3.3 (×3 — ablating it is the single worst move: -42%)
stochastic: 0.8 → 1.6 (×2 — clean timing, 40% WR at this level)
adx: 1.0 → 0.5 (÷2 — partially redundant with EMA stack)
ichimoku: 1.1 → 0.55 (÷2 — lags too much at H4, hurts in ranging)
rsi: 0.3 → 0.0 (ablated — net drag across all test windows)
bollinger: 0.9 → 0.0 (ablated — counter-trend noise on trending pair)
RSI and Bollinger are still computed and displayed in live output for
reference; weight=0 means they contribute nothing to the composite score.
Combined result vs baseline (H4 2022-2026, local data):
Return: +7.12% → +21.30% (+14.2pp)
Max DD: -9.05% → -8.52% (tighter — gains did not come at expense of risk)
Sharpe: 0.23 → 0.47 (2x improvement)
Sortino: 1.38 → 1.91
Win Rate: 36.3% → 38.1%
### backtest.py
Added 16-thread CPU parallelism for signal pre-computation via
ThreadPoolExecutor. _precompute_all_signals() dispatches each bar to a worker;
pandas/NumPy C extensions release the GIL enabling true parallel execution.
Results are collected and replayed in the main loop, replacing the per-bar
recompute. Also added --regime-tf CLI flag so regime filter timeframe can be
specified independently of the primary TF.
### signal_aggregator.py
EMA200 distance gate: confidence *= 0.55 when price is within 0.5% of EMA200
(chop zone) or beyond 1.5% (overextended). Sweet spot is 0.5-1.5% from EMA200
where trending momentum is cleanest. Strongest single filter found: lifted
cumulative return +13.88% to +35.50% in ablation window.
### ENGINE_LOGIC.md (new)
Full reference document covering: architecture, data pipeline, all indicator
logic, composite scoring formula, optimised weights with rationale, regime
classification, EMA200 gate, MTF alignment, grade system, backtesting results,
filter catalogue, and configuration reference.
Across 2024 trend / 2023 chop / 2026 hostile windows, distance from the
H1 EMA200 was empirically the most discriminative variable. Sweet spot
0.5-1.5% from EMA200; closer = false-breakout chop, farther = chase.
Outside the band, confidence is multiplied by 0.55 so the trade falls
below the B-grade threshold and is rejected.
Cumulative return across the 3 windows: +13.88% baseline -> +35.50%.
Win rate improved in every regime; 2023 chop turned positive (-2.56% to
+14.94%).
Also extends the trade journal with per-timeframe MTF scores, RSI at
entry, and EMA200 distance for future filter mining; surfaces ema_trend
in key_levels so the gate can read it.
Session filter restricts trades to a UTC hour window (default 07-16,
London + NY) — opt-in via --session-filter, with --session-start /
--session-end overrides. Wired into both the backtester (cheap hour
check next to the regime gate) and live mode (downgrades signal to
NEUTRAL/F when current UTC hour is outside the window).
Auto-toggle (--auto-session-filter) tracks rolling mean pnl_pct over
the last N closed trades; when it falls below threshold, the session
filter activates, releasing once expectancy recovers. Default lookback
10 trades, threshold -0.3%.
Trade dataclass extended with diagnostic fields captured at entry:
entry_hour, entry_dow, ADX, ATR%, market_regime, volatility_regime,
mtf_agreement. These let --journal-csv exports surface loss-pattern
clusters when a window underperforms.
Validated on 2024 trend / 2023 chop / 2026 recent windows. Auto-toggle
preserves ~50% of trend edge while still cutting ~25% of recent-regime
losses — pure regime filter is still best on cumulative, so both
session flags ship default-off.
--regime-filter previously affected only backtests. Live mode now also
checks higher-TF ADX after analysis: signals are downgraded to NEUTRAL
with a warning when ADX < threshold, and confirming when above.
This brings live behavior into parity with what the backtester measures.
Harwinder Sekhon
harry