New additive columns on every logged scan: bar_ts (primary closed-bar label, indexed with
timeframe), engine_fingerprint (hash of the scoring config incl. the regime weight table in
force - SignalAggregator.engine_config_fingerprint, twin of the gate fingerprint), code_version
(short git SHA), penalty_tags_json / penalty_total / momentum_fading / exhaustion_count, and
post-hoc watch_level / watch_kinds / watch_line written by violent_watch.attach_watch via
signal_log.update_watch. Each indicators_json entry now also carries eff_weight (base weight x
regime multiplier, the weight the composite actually used) and wscore.
engine: IndicatorSignal.effective_weight / weighted_score are stamped inside _compute_composite
(record-keeping only); regime_weight_table(config) is the single source of truth for the
multiplier table (override honoured). Composite output byte-identical (120-report hash proof).
New table scan_outcomes: model-free forward moves per scan (fwd_1h/4h/24h in $ and ATRs,
mfe/mae over 24h, burst_1h, bars_used, complete) measured on the stored candle stream with the
advisory model's continuity rule; labelled opportunistically at the end of log() (time-boxed,
throttled, idempotent) - no daemon; scripts/label_scan_outcomes.py drives the backlog
(--dry-run is read-only). analyze_signal_log --scan-outcomes: forward move + hit rate by
TF x signal/grade with a NEUTRAL baseline and burst rate by watch level. Readers dedup by the
first row per (timeframe, bar_ts).
Module 31 (10 tests). Suite: 463 passed. Backtester untouched (test-locked).
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01RWV2yLt3YXHHGVYHSQU2EE
Five-agent audit of the signal path; all fixes are delivery/measurement
side. Regression-proven behavior-neutral on validated signals: the 5yr
M15 replay reproduces the pinned book exactly (+210.65R PF 1.35
WR 57.8% MaxDD 33.90R). Tests 215 -> 221.
- TradeSetup carries trail_arm_r/trail_dist_r stamped from the producing
config; Telegram trail plan reads them (was showing default 0.6/0.75
instead of the validated M15 arm 0.70 / M5 dist 0.85)
- Alert dedup: fingerprint = (signal, quality, direction); entry movement
compared against last-ALERTED entry/risk (old SL-dist bucket churned on
ATR drift -> near-duplicate alerts every candle)
- Fresh fetches enforce the closed-bar boundary check with one 3s grace
retry, then warn (feed lag no longer silently analyzes the prior bar)
- Backtest max DD includes the zero baseline (opening losing streaks
were undercounted; pinned M15 DD unchanged)
- Replay loads the regime-filter TF when the H4-ADX gate is on (was a
silent no-op for M5/M15 primaries)
- clear_cache under _cache_lock; dead ROC zero-cross branches removed
from both engines (provably unreachable); RSS dates via
parsedate_to_datetime (named zones no longer stamped UTC); event risk
grades from the soonest event; .env inline comments stripped;
auto-scan skips NEUTRAL like the breach path; report_to_dict honors
the weight-0 skip rule and emits strategy/trail params; PB funnel
counts spread_gate/accepted; session-filter fail-open paths log loudly
- Docs corrected to match validated code (H&S neckline = lower peak,
exhaustion anchors farthest tracked swing) with re-validation warnings
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01R5soRUeRBi8kHMB99csrJy
- Delete test_ws_dual.py (standalone probe, unused) and a stale tracked log
- config.py: drop MarketSession enum, is_jpy_pair, candle_buffer_seconds,
and the never-read legacy SignalConfig/XauStrategyConfig fields
- data_provider.py: drop uncalled get_latest_price/get_spread_estimate
- Remove unused imports (engine, telegram_bot, main, news_engine)
- test_all.py: drop/trim the corresponding defaults assertions (199 tests)
- CLAUDE.md updated to match
Zero behavior change: no production code read any removed field
(verified reference-by-reference); suite green at 199/199.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01VPNF37MP9Kuv62B1VBrZ1b
Fixes all high+medium findings from the 2026-07-16 full-codebase audit.
High:
- data_provider: cache now rejects snapshots missing the latest closed
bar (_has_latest_closed_bar, weekend-exempt). The wall-clock TTL served
mid-candle snapshots at the next boundary, so live M15/H1 scans ran one
full bar behind and the bar-close session check evaluated the wrong hour.
- signal_aggregator: gate 6e (SELL quality) only acts on actionable
grades. It was re-grading session-blocked F reports to D, which
re-opened the M5 pullback path (entry check: quality != "F") on
session-blocked bars.
- gpu_backtest: port the live EMA-ribbon compression/expansion logic
(+10 conf on expansion) into the GPU engine. Its absence diverged
composite score/conf on 7-9% of bars and flipped signals on ~0.2%.
Medium:
- gpu_backtest: Bollinger precompute uses the exact pandas rolling calls
(cumsum E[x2]-E[x]2 lost precision at gold price levels; bb_position
error up to 7.6e-3 crossed signal thresholds, worst during squeezes).
- gpu_backtest: --confirm-tfs defaults to the pair factory's live
confirmation map instead of a hardcoded 1h,4h.
- gpu_backtest: entries re-anchor to the first RAW bar after the signal
bar, so >10-sigma spike bars dropped by cleaning are no longer skipped
at entry (favorable bias).
- gpu_backtest: per-TF warmup pad — D1 confirms get >=210 bars at window
start (62-day pad gave ~61 bars: no EMA200, unlike live).
- data_provider: thread-safety locks for cache + throttle (shared
provider across the bot's 4-thread pool); outlier filter keeps the
snap-back bar after a spike instead of punching a two-bar hole.
- telegram_bot: alert fingerprint gains a risk-relative entry bucket
(materially-moved setups re-alert) and is committed only after a
successful send (failed sends no longer consume the alert for 1h).
- engine: disabled_indicators=["macd"] now removes the MACD pillar from
scoring (histogram still computed for divergence).
Verification: test_all.py 187/187 (new Module 11: 8 audit-fix
regressions incl. a live-vs-GPU parity sweep with ribbon coverage);
real-data parity sweep over 813 slices of 14mo XAU M15: zero divergence
(pre-fix: 7-9% of bars diverged); both original bug repros no longer
reproduce; cached end-to-end backtest run passes with the live confirm
map. Books shift slightly vs pre-fix runs — comparability note added to
CLAUDE.md; pinned tuned numbers need re-validation under the fixed
backtester.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Fix 1 — mixed-"trending" regime (ADX strong, EMAs unaligned) was
hard-coded bullish in 5 sites: RSI reinterpretation used the uptrend
playbook (RSI 38-55 -> STRONG_BUY conf 75), BUY-only confidence
bonuses, BUY-only extension penalties, no counter-trend penalty.
Now: extremes-only RSI (as transitioning), no directional bonuses,
symmetric extension penalties. 6yr validation: H1 +67.2->+61.9R
(mixed skew 16B/11S -> 4B/7S), H4 +31.6->+27.5R (the 4 bias BUYs),
M15 +152.6->+154.7R, M5 +207.1->+206.1R with maxDD 28.7->24.3R.
The ~8R aggregate give-back is bull-market subsidy from the bug.
Fix 2 — live session filter now applies the EXACT replay-loop
predicate (session window + blocked hours on the signal bar's CLOSE
time, from the bar timestamp not wall clock). Replaces a wall-clock
filter gated on a "XAU STRATEGY" marker that no code writes, which
made live diverge from every validated book.
Fix 3 — delete dead _check_xau_oversold_bounce and
_generate_xau_trade_setup (unreachable; the latter hardcoded BUY and
skipped the spread/RR gates).
Backtester: parallel bar replay via forked workers (--workers,
default auto = cores-2 capped 12). Per-bar evaluation is pure; the
entry cooldown is applied in an ordered phase-2 pass. Proven
byte-identical to sequential on the same window; ~300 -> 1030-1680
bars/s (6yr M5: ~26min -> 5-8min). --workers 1 = sequential path.
Tests: 179 passed (new mixed-regime RSI regression test).
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Ported from feature/eurusd-support (e6e09c5), pair-neutral fixes:
- _find_swing_points: all(<empty generator>) is vacuously True, so a value
surrounded by NaNs was reported as a swing point; now requires at least
one valid neighbor on each side
- ROC: guard against zero close at the lookback positions (inf scores)
- ADX: epsilon denominator instead of NaN-replace to avoid NaN cascade
Adds 8 regression tests (Module 9). 150/150 pass.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
ATR was computed after EMAs so ribbon compression detection was dead code
on all timeframes — moved ATR first. Extended M5 swing S/R lookback from
50 to 120 bars. Added four M5-specific filters: min score, MTF agreement,
confirmation TF conviction, and SELL quality cap (A-grade M5 shorts enter
at dip extremes and lose). Fixed backtester blocked-hour check to use bar
close time matching the live bot. Added trade geometry config (spread gate,
SL floor, TP floor RR) and primary_tf mismatch guard in backtester.
Backtest: 1133 trades/+11.5R → 419 trades/+41.0R, PF 1.03→1.37, DD 21→11R.
Train/val split positive in both halves (+19.8R/+21.3R).
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Live pipeline:
- Drop still-forming candle before analysis (signals no longer repaint)
- Direction-aware MTF agreement + opposition gate (no trade against 2-of-3 TF consensus)
- SELL volatility gate: block shorts in low/normal vol regimes (config sell_vol_regimes)
- Divergence: 40-bar lookback, price/indicator swings must align within ±2 bars
(fired on 0.5% of trades before, 29% after), trending weight 2.0→1.0
- RSI regime_aware by default; transitioning regime only signals at extremes
- ADX ±DM tie no longer nudges bearish; TP2 clamped beyond TP1
Backtester realism:
- Charge $0.30 round-trip spread (--spread), gap-through SL/TP fills at bar open
- Resolve trades on RAW price path (cache stores raw bars, cleaning runs on load)
- Weekend/session filters use bar-close decision time to match live wall clock
Validated per-step on M15 2025-03-30→2026-05-31 under identical pricing:
+46.9R → +66.8R, win rate 69.9% → 73.0%, SELL side -2.3R → +9.6R.
Tested-and-reverted: ROC weight removal (-13R), BB fade suppression (-11.6R),
candlestick de-weight (-2R) — composite thresholds are co-tuned with weight mass.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
- Drop USD/JPY from scan pairs (zero signals across all timeframes)
- Disable divergence indicator (zero impact in 14-config lab test)
- Parallelize evaluate.py with ProcessPoolExecutor (16 cores)
- Add --pair CLI filter to evaluate.py for single-pair backtests
- Add data freshness guard: stale cache auto-invalidated, primary TF
stale data suppresses signals to prevent trading on old data
- Add /cache command: shows fetch age and last bar per pair/TF
- Add /clearcache command: wipes in-memory cache for fresh API fetch
- Remove BB from WebSocket breach alerts (keep structural levels only)
- Show live prices only for WS-monitored pairs (EUR/USD, XAU/USD)
- Replace arrow emojis with text tags in indicator lines
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Remove backtest.py, backtest_gpu.py, metrics.py, local CSV data,
Dukascopy loader, and all backtest CLI modes from main.py. Rebalance
engine weights: leading indicators (divergence, RSI, stochastic, BB)
now carry 66% of signal weight vs 34% for lagging (EMA, MACD). Add
ws_twelvedata.py for live tick streaming. Drop pyarrow and tqdm deps.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
## What changed
### backtest_gpu.py (new — ~1,300 lines)
Standalone GPU-accelerated backtester completely separate from backtest.py.
Computes all indicator series in one vectorised O(N) pass over the full price
history, then replays positions with a sequential state machine.
Key design:
- VectorIndicators: pandas ewm/rolling (C-optimised) for all 7 indicators
- CuPy GPU backend with automatic NumPy fallback (no hard dependency)
- Two-phase API: prepare_arrays() once, score_and_run() many times
- --optimize flag: 36-combo weight sweep (1 baseline + 7 ablations + 28
multiplier runs) — calls prepare_arrays() once, replays all combos in seconds
- Full CLI: --tf, --local, --start/--end, --regime-filter, --regime-tf,
--session-filter, --optimize, --journal, --journal-csv
O(N) vs O(N^2): backtest.py recomputes all indicators per bar (O(N^2));
backtest_gpu.py computes each indicator series once and reads bar T in O(1).
On 100,000 H1 bars: ~4 seconds vs >10 minutes.
### data_provider.py
Added load_local_csv() method — reads MetaTrader tab-delimited CSV exports
from market_data/ directory. Format: Time|Open|High|Low|Close|Volume|Spread
(spread column dropped on load, timestamps localised to UTC). Applies the same
_validate_and_clean() pipeline as the Twelve Data path. LOCAL_CSV_MAP maps
each Timeframe enum to its filename.
market_data/ files added (MetaTrader exports, UTC):
USDJPY_M15.csv 100,000 bars 2022-05-04 to 2026-05-08
USDJPY_H1.csv 100,000 bars 2010-04-23 to 2026-05-08
USDJPY_H4.csv 25,855 bars 2010-04-22 to 2026-05-08
USDJPY_D1.csv 5,020 bars 2010-04-22 to 2026-05-08
16 years of H1/H4/D1 vs ~6 weeks of H1 on Twelve Data free plan. Use --local
for all backtests; Twelve Data reserved for live signal feed.
### engine.py — optimised indicator weights
36-combo sweep on H4 2022-2026 found MACD and EMA trend are the dominant
signals; RSI and Bollinger are net drag on this trending pair.
Old → new weights:
ema_trend: 1.2 → 3.6 (×3 — strongest directional signal)
macd: 1.1 → 3.3 (×3 — ablating it is the single worst move: -42%)
stochastic: 0.8 → 1.6 (×2 — clean timing, 40% WR at this level)
adx: 1.0 → 0.5 (÷2 — partially redundant with EMA stack)
ichimoku: 1.1 → 0.55 (÷2 — lags too much at H4, hurts in ranging)
rsi: 0.3 → 0.0 (ablated — net drag across all test windows)
bollinger: 0.9 → 0.0 (ablated — counter-trend noise on trending pair)
RSI and Bollinger are still computed and displayed in live output for
reference; weight=0 means they contribute nothing to the composite score.
Combined result vs baseline (H4 2022-2026, local data):
Return: +7.12% → +21.30% (+14.2pp)
Max DD: -9.05% → -8.52% (tighter — gains did not come at expense of risk)
Sharpe: 0.23 → 0.47 (2x improvement)
Sortino: 1.38 → 1.91
Win Rate: 36.3% → 38.1%
### backtest.py
Added 16-thread CPU parallelism for signal pre-computation via
ThreadPoolExecutor. _precompute_all_signals() dispatches each bar to a worker;
pandas/NumPy C extensions release the GIL enabling true parallel execution.
Results are collected and replayed in the main loop, replacing the per-bar
recompute. Also added --regime-tf CLI flag so regime filter timeframe can be
specified independently of the primary TF.
### signal_aggregator.py
EMA200 distance gate: confidence *= 0.55 when price is within 0.5% of EMA200
(chop zone) or beyond 1.5% (overextended). Sweet spot is 0.5-1.5% from EMA200
where trending momentum is cleanest. Strongest single filter found: lifted
cumulative return +13.88% to +35.50% in ablation window.
### ENGINE_LOGIC.md (new)
Full reference document covering: architecture, data pipeline, all indicator
logic, composite scoring formula, optimised weights with rationale, regime
classification, EMA200 gate, MTF alignment, grade system, backtesting results,
filter catalogue, and configuration reference.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Across 2024 trend / 2023 chop / 2026 hostile windows, distance from the
H1 EMA200 was empirically the most discriminative variable. Sweet spot
0.5-1.5% from EMA200; closer = false-breakout chop, farther = chase.
Outside the band, confidence is multiplied by 0.55 so the trade falls
below the B-grade threshold and is rejected.
Cumulative return across the 3 windows: +13.88% baseline -> +35.50%.
Win rate improved in every regime; 2023 chop turned positive (-2.56% to
+14.94%).
Also extends the trade journal with per-timeframe MTF scores, RSI at
entry, and EMA200 distance for future filter mining; surfaces ema_trend
in key_levels so the gate can read it.
Framework:
- TechnicalConfig.disabled_indicators skips named indicators in engine.analyze
- Optimizer supports disable_indicator override for parallel ablation sweeps
- main.py --ablate runs leave-one-out across all 10 directional indicators
Cleanup applied (validated across 2024 trend / 2023 chop / 2026 recent):
- Drop pivot by default (cumulative +9.5% to disable)
- Drop candlestick by default (+3.4% cumulative)
- Regime-gate ichimoku: 1.3x in trending, 0.2x in ranging (was a chop villain)
- Trim RSI weight 1.0 -> 0.3 (cumulative noise across regimes)
Result: +10.60% on 2024 trend (+33.24% -> +43.84%, Sharpe 2.17 -> 2.77),
+15.71% on 2023 chop (-18.27% -> -2.56%). Recent regime still adverse.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
Harwinder Sekhon
harry