Merge pull request 'Remove EUR/USD support — XAU/USD-only codebase' (#1) from xau-only-chart-patterns into main

Reviewed-on: http://192.168.55.11:3000/harry/ForexBot2/pulls/1

.gitignore

@@ -0,0 +1,27 @@
+# Secrets
+.env
+.env.local
+
+# Editor / tooling
+.claude/
+.vscode/
+.idea/
+
+# Python
+__pycache__/
+*.pyc
+*.egg-info/
+.venv/
+venv/
+
+# OS
+.DS_Store
+Thumbs.db
+
+# Logs
+*.log
+
+# Backtesting data (keep scripts, ignore generated files)
+backtest/cache/
+backtest/*.csv
+backtest/__pycache__/

CLAUDE.md

@@ -0,0 +1,357 @@
+# ForexBot2 — Codebase Reference
+
+Signal-intelligence platform for **XAU/USD (gold — live, validated)**. It fetches OHLCV from Twelve Data, scores 9 technical indicators per timeframe with regime-adaptive weights, checks multi-timeframe agreement and news/calendar risk, grades the signal (A+…F), and generates a full trade setup (entry, SL, TP1/TP2, trailing plan). Delivery: Telegram bot (auto-scanner + subscriber channel) and a CLI. A realism-focused backtester replays history through the *exact same* signal code.
+
+This file is the authoritative developer/agent reference. The human-facing docs live in `docs/` (`user_guide.html` + `beginner_guide.html`, and their PDF builds) — keep them in sync when signal behavior changes. `backtest/instructions_for_backtest.txt` lists what every backtest report must record.
+
+**Read this whole file before changing signal logic.** Every numeric threshold below was either backtested or is load-bearing for a backtested result. Rules for changes are at the bottom ("Rules for making changes").
+
+---
+
+## Repository map
+
+| Path | Lines | Purpose |
+|------|-------|---------|
+| `config.py` | ~374 | All config dataclasses, `PAIR_INFO` registry, per-pair strategy configs. **`PlatformConfig` defaults ARE the tuned XAU config.** |
+| `data_provider.py` | ~347 | Twelve Data SDK wrapper: fetch, rate-limit (55 credits/min), in-memory cache, validation/cleaning, freshness checks |
+| `engine.py` | ~1173 | `TechnicalEngine`: computes all indicators on one DataFrame → `TechnicalAnalysis` (composite score, regime, key levels) |
+| `chart_patterns.py` | ~280 | Chart-pattern detectors for the XAU_CP archetype (pure functions, ATR-unit geometry, first-close-beyond triggers) |
+| `signal_aggregator.py` | ~1447 | `SignalAggregator`: MTF analysis, news integration, gates/penalties, quality grade, trade-setup generation → `SignalReport` |
+| `news_engine.py` | ~680 | Economic calendar (Forex Factory JSON + MyFXBook RSS fallback) + keyword RSS sentiment |
+| `telegram_bot.py` | ~1259 | Async Telegram bot: candle-boundary auto-scanner, WebSocket level-breach monitor, owner lock, subscriber channel |
+| `pair_signals/__init__.py` | 52 | `PAIR_PRODUCERS` registry + `produce_signal()` |
+| `pair_signals/xauusd.py` | 71 | XAU config factory + per-timeframe overrides (docstring = validation record) |
+| `main.py` | ~348 | CLI: `python main.py --pair XAUUSD --timeframe H1 --detail/--json` |
+| `test_all.py` | ~2100 | QA suite: **162 tests in 10 modules**, no live API needed. Must stay green. |
+| `backtest/gpu_backtest.py` | ~2231 | Bar-replay backtester (CuPy→NumPy fallback) using the real `SignalAggregator` |
+| `backtest/diagnose_filters.py` | 609 | Filter-diagnosis tooling |
+| `backtest/research/` | — | Calendar research infra (`ff_calendar_scraper.py`, `calendar_store.py`) + result CSVs |
+| `backtest/*.csv` | — | Verification trade books from past tuning campaigns (`xau_*.csv`, …). **Gitignored, local-only** — used for manual before/after comparison, not by tests |
+| `backtest/cache/` | — | Parquet OHLCV cache (raw bars) |
+| `test_ws_dual.py` | 108 | Standalone WebSocket dual-connection probe |
+| `logs/telegram.log` | — | Bot log (midnight rotation, 30 backups) |
+
+---
+
+## Pair status
+
+- **XAU/USD**: the tuned, validated product — and the ONLY supported pair. All defaults in `PlatformConfig`/`XauStrategyConfig` are its tuned values.
+- **EUR/USD**: REMOVED from the codebase (2026-07-16, branch `xau-only-chart-patterns`). It never had a validated edge: the 2026-07-11/12 verdict found no configuration passing validation on M5, M15, or H1 at a 1-pip round-trip cost; root cause was cost-to-volatility (EUR M15 ATR ≈ 10 pips → 1-pip spread taxes ~10% of ATR; gold's ATR/spread ratio is several times larger). Calendar-surprise and H1-hybrid avenues were also tested and closed. The full EUR implementation and verdict history live in git history (last commit with EUR support: `7d3fd95`). Do not re-add EUR without new data (tick fills or H4/D1 horizons).
+
+---
+
+## End-to-end pipeline (what happens on every scan)
+
+```
+Twelve Data REST (+ WebSocket ticks in the bot)
+  → data_provider.fetch_ohlcv()        per timeframe: throttle, fetch, drop forming candle, validate/clean, cache
+  → engine.TechnicalEngine.analyze()   per timeframe: 9 scored indicators + ATR context → regime → composite score
+  → signal_aggregator.analyze_pair()   live entry point (weekend/day filters, fetch, staleness)
+      └─ analyze_from_dataframes()     replayable core: MTF agreement → news → bonuses/penalties →
+                                       final signal + quality grade → gates (session, SELL, M5, regime) →
+                                       trade setup (3 generators) → reasons
+  → SignalReport                       consumed by telegram_bot.py, main.py, backtest/gpu_backtest.py
+```
+
+Signal currency everywhere is the `SignalStrength` enum: `STRONG_BUY=2, BUY=1, NEUTRAL=0, SELL=-1, STRONG_SELL=-2`.
+
+---
+
+## config.py — every knob
+
+### PAIR_INFO (market constants; key is the canonical symbol `"XAU/USD"`)
+| Field | XAU/USD | Meaning |
+|---|---|---|
+| `pip_value` | 0.10 | price units per pip |
+| `price_precision` | 2 | decimals for SL/TP rounding & display |
+| `default_spread` | 0.30 | round-trip cost charged by the backtester when `--spread` not given |
+
+### TechnicalConfig (indicator parameters)
+EMA 9/21/50/200 (`ema_mode`: `"4ema"` default or `"2ema"` = 50/200 only); RSI 14 (OB 70 / OS 30 / neutral zone 40–60; `rsi_mode`: `"regime_aware"` default or `"classic"`); MACD 12/26/9 (`macd_as_pillar=True` — MACD is a scored indicator, not just divergence input); BB 20/2.0σ; ATR 14; ADX 14 (strong ≥25, very strong ≥50); divergence lookback 40; swing-S/R lookback 50; ROC period 10, threshold 1.0%; Keltner: EMA 20 mid ± 1.5 × ATR(10); ADX slope lookback 5, "emerging" threshold 3.0; `disabled_indicators` list (removes the pillar from scoring; for MACD the histogram is still computed — divergence needs it); `regime_weights_override` dict (replaces the gold regime-multiplier table for experiments).
+
+### SignalConfig (scoring & trade geometry — XAU-calibrated)
+| Field | Default | Meaning |
+|---|---|---|
+| `mtf_agreement_threshold` | 2 | timeframes (of primary + confirmations) that must agree |
+| `min_rr_ratio` | 0.5 | reject setups below this RR |
+| `tp_tightening_factor` | 1.0 | TP scale factor. **Leave at 1.0**: 0.8 was re-tested 2026-07-08 with spread+gap fills and nearly halved total R |
+| `round_trip_cost` | 0.30 | XAU $ round trip |
+| `min_tp_cost_multiple` | 30.0 | spread gate: TP1 distance must be ≥ cost × this (= $9 on XAU). Validated: sub-$10 targets lose |
+| `min_sl_pips` | 80.0 | SL floor in pips ($8 on XAU) |
+| `tp_floor_rr` | 0.5 | TP1 floor as fraction of SL distance (M15 factory overrides to 1.0) |
+| `trail_arm_r` / `trail_dist_r` | 0.6 / 0.75 | trailing exit: once favorable excursion ≥ arm×risk, stop trails dist×risk behind best price (bar-close updates). TP1 becomes advisory. Validated on 12mo M5: +105R vs +48R static. 0 disables |
+| `pb_trail_arm_r` / `pb_trail_dist_r` | 0.8 / 0.75 | trailing params for XAU_PB pullback trades (tighter stops → arm later) |
+| `weights`, `min_confidence`, `cooldown_bars`, `default_risk_pct`, `default_rr_ratio`, `atr_sl_multiplier` | — | **LEGACY / UNUSED** outside tests. The engine has its own weights; backtest cooldown lives in `BacktestConfig` |
+
+### NewsConfig
+Tracked currencies `["USD","XAU"]`; pre-event buffer 30 min, post-event 15 min (45 min after a *surprise* print); `event_risk_caps_signals=False` — **XAU default: event risk warns but does NOT cap confidence** (2026-07-11 audit: the caps suppressed trades worth +4.6..+7.2R/yr; XAU trades near USD releases outperform). Context cache TTL 600 s (must stay < pre-event buffer). 4 RSS feeds (investinglive, FXStreet, Google-News gold query, CNBC economy).
+
+### XauStrategyConfig (all XAU gates; `enabled=True`)
+| Field | Default | Notes |
+|---|---|---|
+| `blocked_hours_utc` | [0,1,2,10,11,13,16,21] | bar-close hours never traded. M5 factory appends 9 (post-London-open whipsaw, −28.9R/143 over 6 yr) |
+| `allowed_weekdays` | [0,1,2,3,4,6] | Mon–Fri + Sun (Sydney open). `weekend_filter=True`: Fri 22:00 → Sun 22:00 UTC closed |
+| `min_atr_threshold` | 1.0 | $1 ATR floor for swing-based setups |
+| `v2_sl_atr_multiplier` | 2.0 | SL ATR floor in the generic setup generator |
+| `sl_atr_multiplier` / `tp_atr_multiplier` | 2.0 / 1.0 | used by the XAU ATR setup generator (A+/A only) |
+| `sell_min_quality` | ["A+","A","B"] | SELLs below B are blocked |
+| `sell_vol_regimes` | ["high","extreme"] | SELLs only in these volatility regimes (quiet shorts fade vs gold's macro uptrend) |
+| `blocked_regimes` / `sell_blocked_regimes` | [] / [] | per-TF factory opt-ins. M15 factory sets `blocked_regimes=["trending"]` (mixed regime: strong ADX, EMAs unaligned — lost −6.8R/24) |
+| `sell_blocked_hours_utc` | [] | bar-CLOSE UTC hours where SELLs are blocked (BUYs unaffected). Per-TF factory opt-in: M15 sets [15, 17] (NY-session shorts, −33.5R/165 over 5 yr, negative 6/6 years both halves; BUYs there +34.3R) |
+| M5 gates | | `m5_require_mtf=True`; `m5_min_abs_score=30`; `m5_min_confirmation_score=10`; `m5_sell_max_quality=["B","C"]` (A-grade M5 SELLs enter at dip extremes and lose); `m5_blocked_vol_regimes=["high"]` (chop transition — 61% WR, negative total); `m5_sell_min_rsi=25` |
+| M5 pullback (XAU_PB) | | `m5_pullback_enabled=True`; min score 15 (lower than gate 1 BY DESIGN — mid-pullback scores weaker); ADX ≥ 20; RSI zones BUY (35,62) / SELL (38,65); retrace depth 0.8–6.0 ATR from the 40-bar swing extreme; band touch within 3 bars. Validated +38.3R/254 trades on 12mo |
+| Chart patterns (XAU_CP) | `cp_enabled=False` | `cp_*` fields: pattern list, lookback 72, swing window 3, recency 20, double tolerance 0.5 ATR, height 1.5–8 ATR, H&S head ≥0.5 ATR / shoulder tol 1.0 ATR, flag pole ≥3 ATR in ≤12 bars, flag 3–12 bars retracing 10–50%. Textbook defaults, NOT tuned. **Default OFF** — M5 factory enables BUY-only (`inv_hs`, `double_bottom`, `bull_flag`); M15/H1/H4 remain disabled (all failed OOS validation 2026-07-16). Bearish patterns (`hs_top`, `double_top`, `bear_flag`) dropped: SELL CP lost −3.21R on OOS, −4.52R on M15 5yr |
+
+Legacy fields kept for config-shape compatibility: `rsi_entry_threshold`, `min_ema200_tfs`, `allow_generic_trades`, `buy_threshold_boost`, `m5_confirmation_bars`, `h4_rsi_gate=False`, `h1_macd_gate=False`, `rsi_zone` — not wired to current gates.
+
+### FilterConfig (`config.backtest` — despite the name it applies LIVE too)
+`session_filter=True`, window 03:00–23:00 UTC, `blocked_hours_utc` mirrors XAU's list (non-XAU pairs read this copy), `allowed_weekdays` Mon–Fri+Sun. `regime_filter=False` (optional H4-ADX≥25 gate, off).
+
+### PlatformConfig (top level)
+`symbol="XAU/USD"`, primary TF H1, confirmations [M15, H4], `data_lookback_days=60`, `scan_pairs=[XAUUSD]` (**the single switch for what the bot scans**), `scan_timeframes=[M5,M15,H1,H4]`, `candle_buffer_seconds=10`, `ws_cooldown_seconds=300`, hardcoded `twelve_data_api_key` fallback.
+
+---
+
+## engine.py — TechnicalEngine
+
+**Input contract:** pandas DataFrame, UTC tz-aware DatetimeIndex, oldest-first, columns `Open High Low Close Volume` (capitalized). `< 30` bars → empty NEUTRAL result. EMA200 needs ≥ 200 bars (otherwise EMA-trend indicator degrades gracefully).
+
+**Compute order matters:** ATR first (EMA-ribbon and candlestick proximity use it) → EMA → RSI → BB → ADX → MACD (always computed, even if not a pillar — divergence needs the histogram) → ROC → Keltner (needs BB levels + MACD histogram) → divergence → structure S/R → candlesticks → regime classification → regime-aware RSI reinterpretation → composite.
+
+Results land in `TechnicalAnalysis.indicators: Dict[str, IndicatorSignal]` (fields: name, value, signal, confidence 0-100, description, weight). Private series ride on the result object as underscore attrs (`_rsi_series`, `_macd_histogram`, `_adx_slope`, `_kc_squeeze`, `_kc_squeeze_released`) and are skipped by the composite scorer and all formatters (skip rule: name starts with `_` or `weight == 0`).
+
+### The 9 scored indicators (+ ATR as context)
+
+| Key | Base weight | Signal logic (exact) |
+|---|---|---|
+| `ema_trend` | 1.0 | 4-EMA mode: alignment score over checks (price>EMA9, 9>21, 21>50, 50>200), normalized to [-1,1]. >0.6 STRONG_BUY (conf ≤90), >0.2 BUY, mirrored sells, else NEUTRAL (40). Fast/medium cross ±10 conf. Ribbon (9/21/50 spread < 1 ATR) = "COMPRESSED"; compressed→expanded = +10 conf. Emits `ema_fast/medium/slow/trend` key levels |
+| `rsi` | 1.5 | Wilder RSI (ewm α=1/14). Classic pass first: ≥70 SELL, ≤30 BUY, ≥60 BUY-momentum (45), ≤40 SELL-momentum (45), else NEUTRAL (30). **Then overwritten by regime reinterpretation** (below) when `rsi_mode="regime_aware"` |
+| `macd` | 1.5 | Histogram = MACD line − signal line. Cross above 0 → BUY (conf 55+); accelerating (hist>prev>0) → STRONG_BUY; mirrored down; positive-but-fading / negative-but-fading → NEUTRAL (35); flat → NEUTRAL (30). Confidence scaled by |hist| vs its 20-bar mean |
+| `bollinger` | 1.5 | Position = (close−lower)/(upper−lower). >1.0 SELL (65), <0.0 BUY (65), >0.8 SELL (45), <0.2 BUY (45), else NEUTRAL (30). Squeeze = width < 0.75 × 20-bar avg width → +10 conf. Emits `bb_upper/lower/mid` |
+| `adx` | 0.5 | Wilder ADX/DI (ties zero BOTH DMs — fixes a systematic bearish nudge). ADX ≥ 25 → BUY/SELL by DI dominance, else NEUTRAL. Conf 40/65/80 by ADX band. ADX slope (5-bar diff): <25 with slope > 3.0 → "TREND EMERGING" +15 conf; also strengthening/weakening adjustments. Sets `_adx_slope` |
+| `roc` | 1.5 | 10-period %change. Positive & rising → BUY (STRONG if >2× threshold=2%); negative & falling → SELL; zero-cross → BUY/SELL (65); else NEUTRAL. **Directionally redundant with MACD/Keltner/EMA (0.8–0.94 corr) but its weight is load-bearing: zeroing it backtested 13R WORSE — composite thresholds are calibrated with it. Do not remove without retuning thresholds** |
+| `keltner` | 1.5 | EMA20 mid ± 1.5×ATR(10). TTM squeeze = BB fully inside KC. Squeeze released (was→not) + MACD hist sign → STRONG_BUY/STRONG_SELL (80); squeeze active → NEUTRAL (60); close above/below channel → BUY/SELL continuation (65); inside → NEUTRAL (30). Emits `kc_upper/lower/mid` |
+| `divergence` | 1.5 | RSI + MACD-hist vs price over 40 bars. Swing points (window 3) must align within ±2 bars to pair up; last two pairs: price LL + indicator HL = bullish, price HH + indicator LH = bearish. Any found → BUY/SELL/NEUTRAL-if-both, conf 70, value = count |
+| `candlestick` | 1.0 | Last bar: Doji (body<10% range), Hammer/Hanging Man (lower wick>2×body), Shooting Star/Inverted Hammer, Engulfing (vs prev bar), Marubozu (body>85%). Majority vote → signal, conf 50; +20 if pattern sits within 0.5 ATR of a swing S/R level |
+| `atr` | **0.0** (context) | Wilder ATR. Volatility regime from percentile vs last 50 ATR values: >80 extreme, >60 high, >30 normal, else low. Emits `atr`, `atr_pct` |
+
+Structure S/R (not scored): swing highs/lows (window 3) over `swing_sr_lookback` (min 120 bars on 1min/5min TFs); up to 3 above and 3 below stored as `swing_high_1..3` / `swing_low_1..3` key levels.
+
+### Market regime classification (exact rules)
+- ADX ≥ 25 **and** EMA-trend signal bullish → `trending_up`; bearish → `trending_down`; EMA mixed → **`trending`** (the "mixed" regime — momentum without directional structure; treated as direction-UNKNOWN everywhere; a past bullish-bias bug here cost −6.8R and is fixed)
+- else ADX slope > 3.0 → `transitioning` (trend birth, ~5–10 bars before ADX crosses 25)
+- else → `ranging`
+
+### Regime-aware RSI reinterpretation (`rsi_mode="regime_aware"`, the default — classic mode backtested worse for gold)
+| Regime | RSI zones |
+|---|---|
+| trending_up | 38–55 STRONG_BUY 75 ("pullback zone, ideal entry"); 30–38 BUY 60; 55–65 BUY 50; >65 NEUTRAL ("extended, wait"); <30 NEUTRAL ("possible reversal") |
+| trending_down | mirrored: 45–62 STRONG_SELL 75; 62–70 SELL 60; 35–45 SELL 50; <35 / >70 NEUTRAL |
+| transitioning / trending (mixed) | ONLY extremes: ≥70 SELL 50, ≤30 BUY 50, else NEUTRAL 30 |
+| ranging | ≥70 SELL 70, ≤30 BUY 70, ≥60 SELL 45, ≤40 BUY 45, else NEUTRAL |
+
+### Composite score
+For each scored indicator: `effective_weight = base_weight × regime_multiplier`; contribution = `signal.value × effective_weight × confidence/100`. Composite = `(Σ contributions / Σ effective_weights) × 50`, clamped to ±100. Confidence = weight-averaged indicator confidence. Composite signal: ≥40 STRONG_BUY, ≥15 BUY, ≤−40 STRONG_SELL, ≤−15 SELL, else NEUTRAL.
+
+**Gold regime multipliers** (defaults in `_compute_composite`; `technical.regime_weights_override` replaces the whole dict):
+
+| Indicator | trending_up/down/trending | ranging | transitioning |
+|---|---|---|---|
+| ema_trend | 2.0 | 0.3 | 1.5 |
+| rsi | 1.0 | 1.5 | 1.0 |
+| macd | 2.0 | 1.5 | 2.5 |
+| bollinger | 0.5 | 1.5 | 1.0 |
+| keltner | 1.5 | 2.0 | 2.5 |
+| roc | 1.5 | 0.5 | 2.0 |
+| candlestick | 0.8 | 1.0 | 1.0 |
+| divergence | 1.0 | 2.5 | 2.0 |
+| adx | 0.8 | 0.5 | 1.0 |
+
+---
+
+## signal_aggregator.py — SignalAggregator
+
+Two entry points:
+- **`analyze_pair(symbol)`** — LIVE: applies XAU weekend + allowed-weekday filters on wall clock, fetches all TFs (primary + confirmations), flags stale data (primary stale → signal suppressed to NEUTRAL/F), delegates to `analyze_from_dataframes`, then applies the live session check on the **bar-CLOSE time** (bar timestamp + candle duration): outside 03:00–23:00 or in blocked hours → NO TRADE. This is the exact predicate the backtester replays, so live == backtest.
+- **`analyze_from_dataframes(symbol, dfs, context, use_news)`** — replayable core; treats each df as the visible history at decision time (caller must slice; no weekend filter here — the backtester applies it).
+
+### Pipeline inside `analyze_from_dataframes` (numbered as in the source)
+1. Primary TF analysis → seeds report (score, confidence, regime, vol regime, price, timestamp = last bar's index label).
+2. Confirmation TF analyses → `mtf_signals` / `mtf_scores` keyed by TF string ("5min", "1h", …).
+3. **MTF agreement**: count BUY-ish vs SELL-ish across ALL TFs (primary included); agreement = count ≥ 2 **in the primary's direction** (two bullish confirms cannot "confirm" a bearish primary).
+4. News (if `use_news`): event risk (`clear`/`caution`/`danger`) + pair sentiment (−1..+1). Backtests pass `use_news=False`.
+5. Spread estimate = median(High−Low) of primary df.
+6. **`_compute_final_signal`** (bonuses → event cap → penalties → thresholds → grade):
+   - Bonuses: MTF agreement → +10 conf and `score = 0.6×primary + 0.4×mean(all TF scores)`; news sentiment aligned (|sent|>0.2) → +5; in a *directional* trend, RSI in pullback zone → +5, BB position favorable → +5 (mixed `trending` regime gets NO directional bonus).
+   - Event risk `danger`: warning always; confidence hard-capped at 30 ONLY if `news.event_risk_caps_signals` (False by default — XAU-tuned).
+   - **Penalty system (diminishing, replaces old additive stacking):** collect, sort by size desc, apply with decay ×1.0, ×0.5, ×0.25…, total capped at 30. Penalties: `mtf_disagreement` 15; `news_conflict` 10; `event_caution` 15 (config-gated); `counter_trend` 15; `extreme_vol` 10; `ema200_near` 5 (price within 0.5% of EMA200 — env `FB2_EMA_LO` overrides); `ema200_far` 5 (>2.5%); `exhaustion` 8 (price >3.5 ATR beyond nearest swing); `no_pullback` 8 (>2.5 ATR from EMA21 in trend) / `weak_pullback` 5 (>2.0); `rsi_extended` 5 (RSI>65 with BUY / <35 with SELL in any strong-ADX regime, both directions in mixed `trending`); `bb_overextended` 8 (BB pos >0.85 BUY / <0.15 SELL).
+   - **Exhaustion hard block**: ≥2 exhaustion-tagged penalties ({exhaustion, no_pullback, rsi_extended, bb_overextended} → `report.exhaustion_count`) AND momentum fading (ROC sign against score, or MACD signal against score) → confidence = 0.
+   - Confidence clamped [0, 95]. **Final signal**: score ≥35 & conf ≥55 STRONG_BUY; ≥25 & ≥50 BUY; mirrored sells; else NEUTRAL.
+   - **Quality grade**: |score|≥40 & conf≥65 & MTF → **A+**; ≥33 & ≥55 & MTF → **A**; ≥28 & ≥50 → **B**; ≥25 & ≥45 → **C**; ≥15 → **D**; else **F**.
+7. Post-grade gates, in order:
+   - **6b session filter** (bar-label hour + weekday; XAU reads `xau_strategy.blocked_hours_utc`/`allowed_weekdays`, others read `config.backtest.*`) → NEUTRAL/F.
+   - **6c transitioning boost**: regime `transitioning` and grade B/C/D → A if |score|≥30 & conf≥50 & MTF; else B if |score|≥25 & conf≥45 (early trend entries).
+   - **6d exhausted-trend cap**: trending regime + exhaustion_count ≥2 + grade C/D → D, setup killed.
+   - **6e XAU SELL quality gate**: sell-direction & grade actionable (A+..C) & not in `sell_min_quality` → D. The actionable guard is load-bearing: without it a session-blocked F report was *upgraded* to D, re-opening the M5 pullback path (its entry check is `!= "F"`).
+   - **6f MTF opposition**: ≥2 TFs pointing against the trade direction → D (never trade against consensus).
+   - **6g XAU SELL volatility gate**: sell-direction & vol regime not in {high, extreme} → D.
+   - **6g2 XAU regime gates**: regime in `blocked_regimes` (M15: mixed `trending`) → D; sell-direction & regime in `sell_blocked_regimes` → D.
+   - **6g3 XAU SELL hour gate**: sell-direction & bar-CLOSE hour (label + candle duration — the entry hour under next-open fills) in `sell_blocked_hours_utc` → D. Per-TF opt-in; M15 factory blocks 15/17 UTC.
+   - **6i XAU M5 noise gates** (`_apply_m5_filters`, primary TF = M5 only): (1) |score| ≥ 30; (2) MTF agreement required; (3) ≥1 confirmation TF with score ≥ ±10 in trade direction (NEUTRAL confirms don't count); (4) SELL grade must be B/C (A-grade M5 shorts lose −0.017R/trade); (5) vol regime `high` blocked; (6) SELL blocked when RSI < 25. Each failure → D, setup killed.
+8. **Trade setup**: XAU + grade A+/A → `_generate_xau_atr_trade_setup` first; any pair + grade A+/A/B/C and no setup yet → `_generate_trade_setup` (swing-based).
+9. Reasons: every non-neutral indicator description → `bullish_reasons`/`bearish_reasons` (the backtester tags strategy from reasons[0] markers).
+10. **XAU_PB pullback** (M5 XAU only, `m5_pullback_enabled`, no setup yet, grade ≠ F): `_try_m5_pullback` — see below.
+11. **XAU_CP chart pattern** (any TF, XAU only, `cp_enabled`, no setup yet, NOT session/day-blocked — weak-composite grade F is allowed BY DESIGN, reversals complete on weak composites): `_try_chart_pattern` — see below.
+
+### Trade setup generators (exact geometry)
+
+**A. `_generate_xau_atr_trade_setup`** (XAU, grade A+/A, ATR>0): TP quality scale A+ 2.0 / A 1.5 (A with exhaustion≥2 → 1.0). TP1 = entry ± `tp_atr_multiplier`(1.0)×scale×ATR; TP2 = 1.5×that; SL = entry ∓ 2.0×ATR. Gates: RR ≥ 0.5; TP1 distance ≥ $9 (spread gate). Prices rounded to 2 dp; pip 0.10 hardcoded — **this generator is XAU-only by construction**.
+
+**B. `_generate_trade_setup`** (swing-based, any pair, grade A+..C): XAU requires ATR ≥ $1 (LOW ATR FILTER). Per-side:
+- SL anchor = nearest swing low below price (BUY) / swing high above (SELL) ∓/± `sl_swing_buffer`×ATR (1.0); fallback (no swing) = (1+buffer)×ATR. SL distance floored at max(ATR × sl_atr_floor, min_sl_pips×pip) — XAU: max(2.0×ATR, $8).
+- TP1 = nearest swing beyond entry ± 0.3 ATR if it's within `tp_swing_cap` ATR (2.5) AND ≥ SL×RR-floor; else entry ± max(`tp_mult`×ATR, SL×RR-floor). tp_mult 1.5, RR-floor = max(min_rr 0.5, tp_floor_rr — 1.0 on M15).
+- TP2 = second swing if within 5 ATR, else entry ± max((tp_mult+1.5)×ATR, SL×floor×1.5); forced beyond TP1 by ≥1 ATR.
+- Gates: RR ≥ min_rr; spread gate TP1 distance ≥ cost×multiple ($9 on XAU).
+
+**C. `_try_m5_pullback` (XAU_PB)**: detection = EMA alignment ≥3 of 4 checks (9>21, 21>50, 50>200, close>200 — mirrored for SELL), ADX ≥ 20, price touched max(EMA21, KC-mid) within last 3 bars, resumption bar closed back in trend direction beyond EMA9, retrace depth 0.8–6.0 ATR from the 40-bar extreme, RSI in pullback zone (BUY 35–62 / SELL 38–65). Then gates: directional score ≥ 15; ≥1 confirmation TF ≥ ±10; not against MTF consensus; vol regime gates (same as breakout path). Geometry: SL = pullback 8-bar extreme ∓ 0.5 ATR (min $8); TP1 = 1.5×risk, TP2 = 2.5×risk; spread gate $9. Sets `strategy="XAU_PB"`, grade min B, marker `"XAU PULLBACK STRATEGY: …"` at reasons[0]. Rejection funnel counters in class attr `SignalAggregator.pb_funnel`.
+
+**D. `_try_chart_pattern` (XAU_CP)**: detection in `chart_patterns.py` — 6 classical patterns (double_bottom/double_top, inv_hs/hs_top, bull_flag/bear_flag), swing-point based (strict local extrema, window `cp_swing_window`=3), all thresholds in ATR units, scanned over the last `cp_lookback`=72 closed bars. Trigger = FIRST close beyond the neckline/boundary since the pattern formed (one breakout → one signal, no repaint); reversal patterns are invalidated by a new extreme beyond the pattern. No composite-score gate BY DESIGN. Gates: MTF non-opposition (6f mirror); SELL patterns respect `sell_vol_regimes`, `sell_blocked_hours_utc` (bar-close hour), M5 RSI floor; M5 blocks `m5_blocked_vol_regimes` both ways; ATR ≥ $1. Geometry: SL = pattern anchor ∓ 0.5 ATR (min $8); TP1 = measured move (pattern height), TP2 = 1.5×height; RR ≥ 0.5; spread gate $9. Sets `strategy="XAU_CP"` (global trailing params), grade min B, marker `"XAU CHART PATTERN: …"` at reasons[0]. Funnel in `SignalAggregator.cp_funnel`.
+
+`TradeSetup` fields: pair, direction, entry, SL, TP1, TP2, RR, risk/reward pips, pip_value, `strategy` ("GENERIC", "XAU_PB", or "XAU_CP" — selects trailing params; XAU_CP uses the global pair).
+
+---
+
+## news_engine.py — NewsEngine
+
+- **Calendar**: Forex Factory free JSON (`nfs.faireconomy.media/ff_calendar_thisweek.json`), 3 retries, honors 429 Retry-After with class-level backoff (≤600 s); fallback MyFXBook RSS. Never fabricates events — unavailable calendar = risk checks skipped that cycle.
+- **Caches are class-level** (a fresh NewsEngine is built per scan): calendar + headlines, TTL = 600 s on success, 60 s on empty result; lock held across fetch so concurrent scans share one request.
+- **Event risk** (`get_event_risk_level`): high-impact event for base/quote within 30 min → `danger`; within 60 min → `caution`; already-released high-impact → `caution` for 15 min after (45 min if actual ≠ forecast — "surprise"). `EconomicEvent.surprise` compares parsed numeric actual vs forecast (handles "%", "K/M/B" suffixes); direction is numeric-only (doesn't know if higher is bullish).
+- **Sentiment**: keyword sets over headlines from 4 RSS feeds, deduped by title (Google News " - Publisher" suffix stripped), last 24 h only. Per-currency scores; **`GOLD_INVERSE_KEYWORDS`** (rate hike/hawkish/tightening/rate cut/dovish/easing/stimulus) flip sign for XAU (hawkish = dollar-bullish = gold-bearish). Currency detection via compiled word-boundary-ish regexes (bare "us"/"eu"/"uk" deliberately excluded). Pair sentiment = base − quote.
+
+---
+
+## data_provider.py — ForexDataProvider
+
+- Twelve Data SDK (`TDClient.time_series(...).as_pandas()`), symbols normalized to `"XAU/USD"` form; renames lowercase columns → capitalized; sorts oldest-first; localizes UTC.
+- **Rate limit**: 55 credits/min sliding window (`_throttle`), blocks with sleep when exhausted.
+- **In-memory cache**: key `{symbol}_{tf}`, TTL = candle duration (M5 300 s … H4 14400 s); a cached frame is evicted and refetched if it fails the freshness check **or the boundary check** (`_has_latest_closed_bar`: the frame must contain the most recent fully closed bar, weekend-exempt — a wall-clock TTL alone would serve mid-candle snapshots at the next boundary and scans would run one bar behind). Cache and throttle are lock-protected (one provider is shared across the bot's thread pool).
+- **`_drop_forming_candle`**: Twelve Data includes the current partial candle; the scanner fires seconds after boundary, so the last bar is dropped if its close time is in the future. Without this every indicator would read a seconds-old stub and signals would repaint.
+- **Freshness** (`_is_data_fresh`): last bar older than 3× candle duration → stale (warns; live path suppresses signal if the PRIMARY TF is stale). Weekends exempt (Fri 22:00→Sun 22:00 UTC).
+- **`_validate_and_clean`**: drop all-NaN rows → ffill (limit 3) → drop NaN closes → sort + dedupe (keep last) → drop future timestamps (>now+5 min) → repair OHLC inconsistencies (High=max, Low=min of O/C/existing) → remove >10σ return outliers (spike bar only — the opposite-direction snap-back bar right after it is kept, else every spike punched a two-bar hole) → warn if < ~210 bars.
+- `outputsize = max(300, min(days×bars_per_day+20, 5000))`; TD caps at 5000/call (the backtester paginates by date).
+- `get_latest_price` / `get_spread_estimate` (median M5 High−Low) helpers.
+
+---
+
+## pair_signals/ — per-pair factories (ALWAYS use these, never bare PlatformConfig for a pair)
+
+`PAIR_PRODUCERS: Dict[symbol, factory]`; factory signature `make_config(primary_tf, confirmation_tfs=None) -> PlatformConfig`. `produce_signal(symbol, tf, data_provider)` builds config → aggregator → `analyze_pair`. Registered: XAU/USD.
+
+**A bare `PlatformConfig()` silently runs any pair with XAU defaults — always go through the factory** (the bot, CLI, and backtester all do).
+
+Default confirmation map (same in the factory, the bot, and backtest usage): M5→[M15,H1], M15→[M5,H1], H1→[M15,H4], H4→[H1,D1]. (Exception: `main.py`'s CLI map uses M5→[M1,M15] — a CLI-only quirk.)
+
+**xauusd.py**: `PlatformConfig()` defaults + per-TF overrides (each validated; full evidence in the module docstring):
+- M5: append hour 9 UTC to blocked hours (6-yr replay: +207.1R vs +178.3R; M15-and-up keep hour 9 — it's positive there). Enable BUY-only chart patterns (`cp_enabled=True`, `cp_patterns=["inv_hs","double_bottom","bull_flag"]`): 12mo +41.8R PF 1.60 (train/val both positive), 5yr +65.2R PF 1.48; bearish CP dropped (OOS −3.21R). M15/H1 CP remain disabled (all failed OOS).
+- M15: `signals.tp_floor_rr = 1.0` (1R minimum targets: +116.6R vs +105.4R, better in train/val and 2024 OOS; **hurts M5** — per-TF only) and `xau_strategy.blocked_regimes = ["trending"]` (mixed regime block: +65.7R vs +59.0R). A transitioning-SELL block was tested and REJECTED (bull-market artifact — failed 2024 OOS).
+- M15: `xau_strategy.sell_blocked_hours_utc = [15, 17]` (SELL-only hour block, validated 2026-07-16 on the 5-yr replay: +176.7R PF 1.31 vs +144.9R PF 1.22, better in all 6 years, both 12-mo halves, and 2024-06→2025-03 OOS; M5 cross-TF pocket confirms; BUYs at those hours are +34.3R so `blocked_hours_utc` would be wrong).
+- The factory must stay equivalent to inline `PlatformConfig(symbol=XAUUSD, …)` + overrides — **enforced by a test in test_all.py**.
+
+---
+
+## telegram_bot.py — the bot
+
+Run: `python telegram_bot.py` (env `TELEGRAM_TOKEN`, `TELEGRAM_CHAT_ID`, `TELEGRAM_CHANNEL_ID`; hardcoded fallbacks exist in the file).
+
+- **Owner lock**: every command/button is wrapped in `_owner_only` — only `TELEGRAM_CHAT_ID` gets responses; others get a refusal. The bot is the owner's control panel; the product is the channel.
+- **Subscriber channel** (`TELEGRAM_CHANNEL_ID`, `-100…`; `""` disables): auto-scan signal alerts + breach alerts also post there (bot needs channel-admin). Monetization = Telegram Stars subscription invite links; Telegram handles membership, the bot has zero payment code. The channel feed keeps running even when the owner toggles DM auto-scan off (`_feed_active()` = DM chats OR channel).
+- **Auto-scan**: boot-starts if `TELEGRAM_CHAT_ID` set (initial M15 scan populates the level cache first). Candle-boundary loops for M5/M15/H1/H4, firing at boundary + 10 s. Each scan runs all `scan_pairs` concurrently, updates the level cache, then broadcasts.
+- **Alert gating**: quality threshold (`_alert_min_quality`, default "C", menu cycles C→B→A) + fingerprint dedup per (symbol, TF) on (signal, quality, direction, entry bucket) with 1-hour re-arm — a persisting identical signal doesn't re-send every candle, but a setup that moved ≥ half its risk distance (new pullback/swing anchor) re-alerts. The fingerprint is committed only **after** a message reaches ≥1 target (`_commit_alert`), so a failed Telegram send doesn't consume the alert.
+- **WebSocket monitor**: subscribes every `scan_pairs` symbol to TD quotes WS; 10 s heartbeat; exponential reconnect (≤60 s). On tick crossing any cached key level (`swing_high/low_1..3`, `ema_trend`, `bb_upper/lower`, `kc_upper/lower`; cache TTL 1 h) → immediate M15 rescan → "LEVEL BREAK" alert if actionable+changed (same gates as auto-scan), 5-min per-pair cooldown.
+- **Commands**: `/start` (menu), `/scan` (buttons: per pair — M5/M15/H1/H4 + Scan All TFs), `/autoscan` (toggle DM copy), `/status` (loops, WS, live prices, level cache, cooldowns), `/cache` (age per pair/TF), `/clearcache`, `/help`.
+- Signal messages include the full indicator breakdown, warnings, trade setup, **trailing-stop management plan** (arm price + trail distance, from Signal/pb params by `strategy`), and a risk disclaimer on every outbound message.
+- Blocking scan work runs in `ThreadPoolExecutor(max_workers=4)`; one shared `ForexDataProvider` across all scans (cache reuse). Logs to `logs/telegram.log`.
+
+---
+
+## main.py — CLI
+
+`python main.py [--pair XAUUSD] [--timeframe M5|M15|M30|H1|H4|D1] [--detail] [--json] [--verbose]`. Uses the pair factory, renders a compact terminal report (decision, setup, context, top-3 reasons per side, warnings) or full JSON via `report_to_dict`.
+
+---
+
+## backtest/gpu_backtest.py — the backtester
+
+Replays history bar-by-bar through the **real `SignalAggregator.analyze_from_dataframes`** — same gates, same setup math. News is disabled (`use_news=False`): backtests are pure technical.
+
+**Realism model (all validated fixes — do not regress):**
+- Entry at the **next bar's open** after the signal bar; SL/TP distances preserved relative to actual entry. Entries are **re-anchored to the raw price path** (`_reanchor_entries_to_raw`): if cleaning dropped a >10σ spike bar right after the signal bar, the trade enters at the spike bar's open, not the bar after it — skipping it was a favorable bias.
+- Round-trip `--spread` subtracted from every trade's PnL before computing R (default = pair's `default_spread`: XAU $0.30). Results before 2026-07-08 (+59.5R baseline era) charged no spread — not comparable.
+- Trade resolution runs against the **RAW price path** (`fetch_history(clean=False)`): the parquet cache stores raw bars; cleaning (which drops >10σ news spikes that hit stops in reality) is applied only to the signal-side data. Caches written before this fix contain cleaned data — refetch to restore spike bars.
+- Same-bar SL+TP tie → SL first (conservative). Gap through SL/TP → filled at the bar open.
+- **Trailing exits** (default on, from `signals.trail_arm_r/trail_dist_r`; XAU_PB trades carry per-trade 0.8/0.75 overrides): stop ratchets at bar close only, so a new stop can only fill from the next bar (conservative). Exit reasons: SL / TRAIL / TP1 (static mode) / TIMEOUT / NO_DATA.
+- Session + weekend + weekday filters evaluate the **decision time = bar close** (`ts + primary_duration`) — exactly matching the live bot's wall-clock check.
+- Confirmation TFs are sliced to bars **fully closed** by decision time (no look-ahead).
+
+**Performance:** `precompute_indicators` builds every indicator series once per TF (CuPy on GPU or NumPy — check the log line `Backend: CuPy GPU` vs `NumPy CPU fallback`; CuPy needs the `nvidia-*-cu12` pip wheels or it silently falls back). `GPUTechnicalEngine` subclasses the live engine and overrides only the `_compute_*` methods with cache lookups — outputs are byte-identical to the live engine (regression-tested). Bar replay: pure per-bar evaluation fans out over forked worker processes (`--workers`, 0 = auto: cores−2 capped 12; <5000 bars forces 1); the entry **cooldown (5 bars) is the only sequential rule**, applied in an ordered pass afterwards — parallel and sequential books are identical. ~300 bars/s/core, ~1700 bars/s at 10 workers.
+
+**Strategy tags** (from reasons[0] markers): `XAU_PB`, `XAU_CP`, `GENERIC` (plus legacy `XAU` recognized). `--strategies` filters the book by tag. `--cp` force-enables the chart-pattern entry for candidate validation; `--cp-patterns` narrows the detector set.
+
+**CLI:**
+```bash
+python backtest/gpu_backtest.py --pair XAUUSD --days 365
+python backtest/gpu_backtest.py --pair XAUUSD --start 2025-01-01 --end 2025-06-01 --primary-tf 15min --confirm-tfs 1h,4h
+# key flags: --spread --quality A+,A,B,C --strategies --workers --max-hold-bars 300
+#            --warmup-bars 250 --cooldown-bars 5 --no-cache --export-csv out.csv --detail
+```
+Report: summary metrics (total R, PF, Sharpe on R, max DD, MFE/MAE), breakdowns by quality/strategy/regime, trade log, and per-trade reasoning dumps (indicator-by-indicator agreement at entry — required by `instructions_for_backtest.txt`). `--export-csv` flattens every indicator's signal/value/agreement into columns.
+
+**Comparability note (2026-07-16):** the parity/realism fixes of that date (EMA-ribbon port to the GPU engine, exact-pandas Bollinger precompute, raw-path entry re-anchoring, per-TF D1 warmup, live confirm-map CLI default, outlier snap-back retention) all shift books slightly vs earlier runs. Books produced before these fixes are not directly comparable — re-run the baseline before any before/after tuning comparison.
+
+Data: paginated TD fetches (5000-bar chunks) cached as parquet in `backtest/cache/` keyed by symbol/TF/window; warmup pad ≥60 days before the window for EMA200, extended per-TF so slow confirmation TFs get ≥210 bars at window start (D1 gets ~299 days — 62 days gave only ~61 bars: no EMA200, unlike live's 300-bar outputsize floor). `--confirm-tfs` defaults to the pair factory's live confirmation map (pass it explicitly to override); a hardcoded default here previously made the standard invocation backtest a different MTF setup than the bot scans.
+
+---
+
+## test_all.py — QA suite
+
+`python test_all.py` — **162 tests, 10 modules** (config, data_provider, engine, news_engine, signal_aggregator, main, telegram_bot, integration, pair_signals registry + engine-bug regressions, 2026-07-16 audit-fix regressions). No network needed (mocked). Uses a custom `@test("name")` decorator that runs immediately (NOT pytest — don't grep for `def test_`). Exit code non-zero on failure. Includes: the xauusd-factory ≡ inline-config equivalence test, engine bug-fix regression tests (Module 9b), and a live-vs-GPU engine parity sweep incl. the EMA-ribbon path (Module 11). **Any signal-logic change must keep this green and should add tests.**
+
+---
+
+## Environment variables
+
+| Variable | Purpose | Default |
+|---|---|---|
+| `TWELVE_DATA_API_KEY` | market data key | hardcoded fallback in `config.py` |
+| `TELEGRAM_TOKEN` | bot token | hardcoded fallback in `telegram_bot.py` |
+| `TELEGRAM_CHAT_ID` | owner chat id (boot auto-scan + owner lock) | hardcoded fallback |
+| `TELEGRAM_CHANNEL_ID` | subscriber channel (`-100…`); `""` disables the feed | hardcoded fallback |
+| `FB2_EMA_LO` | EMA200 "near" penalty threshold in % | `0.5` |
+
+Dependencies (`requirements.txt`): numpy, pandas, requests, twelvedata, python-telegram-bot ≥20, websockets. Python 3.12. A `venv/` exists in-repo.
+
+---
+
+## Code conventions
+
+- Type hints on public methods; dataclasses for all config and outputs.
+- `SignalStrength` (−2..+2) is the universal signal currency.
+- Key levels dict keys: `swing_high_*`, `swing_low_*`, `ema_*`, `bb_*`, `kc_*`, `atr`, `atr_pct`.
+- Private per-indicator series ride on `TechnicalAnalysis` as `_underscore` attributes; anything `_`-prefixed or weight-0 is skipped by scorers/formatters.
+- UTC everywhere; DataFrames indexed by tz-aware `Datetime`, oldest-first, capitalized OHLCV columns.
+- Logging via module loggers; the bot adds a rotating file handler.
+
+---
+
+## Rules for making changes (READ BEFORE EDITING SIGNAL LOGIC)
+
+1. **Never change a tuned value without re-validation.** The acceptance rule used throughout: positive total R in BOTH the train and validation halves of a ≥12-month window, PF ≥ 1.3, at realistic cost (XAU $0.30 round trip), with trailing exits and next-open fills. For any marginal edge additionally re-test on the 2024-06→2025-03 out-of-sample window — within-year splits alone have approved configs that failed OOS.
+2. **Per-timeframe overrides live in the pair factory** (`pair_signals/*.py`), never in global defaults — the same knob has opposite effects on different TFs (proven twice: `tp_floor_rr`, hour-9 block). Document the validation evidence in the factory docstring.
+3. **Pair isolation is a hard invariant.** XAU gates in the aggregator are display-gated (`report.display_name == "XAU/USD"`) so a future pair port cannot inherit gold-calibrated logic by accident.
+4. **Live == backtest parity**: session/weekend/hour filters must evaluate bar-CLOSE time in both paths; `xauusd.make_config` must stay equivalent to the inline config the backtester/bot would build (test-enforced). If you add a live gate, add it to the replay path (or `analyze_from_dataframes`) in the same commit.
+5. **ROC stays weighted** (see engine table) unless you retune composite thresholds and re-validate — removing it lost 13R.
+6. **Backtester realism knobs are results-critical**: raw-path resolution, spread charging, next-open entry, SL-first ties, close-only trail ratchet. Any "optimization" that touches these invalidates comparability with the pinned books (`backtest/*.csv`, memory notes).
+7. **Don't zero/disable indicators via config in production paths** — `disabled_indicators` exists for experiments; composite thresholds assume the full set.
+8. **The forming candle must stay dropped** and analysis must only ever see closed bars, or live signals repaint.
+9. Keep `test_all.py` green (`python test_all.py`); extend it for every new gate/knob.
+10. Legacy/no-op config fields exist (flagged above) — do not build new logic on them; delete only with test updates.

backtest/diagnose_filters.py

@@ -0,0 +1,609 @@
+"""
+ForexBot2 — Filter Rejection Diagnostic
+========================================
+Replays the same backtest loop as gpu_backtest.py but instead of
+collecting trades, counts how many M15 bars are rejected by each
+gate in the signal pipeline.
+
+Produces a summary table showing where potential trades are being
+lost, helping tune filter thresholds.
+
+Usage:
+    python backtest/diagnose_filters.py --pair XAUUSD --days 365
+    python backtest/diagnose_filters.py --pair XAUUSD --days 365 --sample-every 1
+"""
+
+from __future__ import annotations
+
+import argparse
+import logging
+import sys
+import time
+from collections import Counter, defaultdict
+from datetime import datetime, timedelta, timezone
+from pathlib import Path
+from typing import Dict, List, Optional, Tuple
+
+import numpy as np
+import pandas as pd
+
+# Ensure project root on sys.path.
+PROJECT_ROOT = Path(__file__).resolve().parent.parent
+if str(PROJECT_ROOT) not in sys.path:
+    sys.path.insert(0, str(PROJECT_ROOT))
+
+from config import (  # noqa: E402
+    PAIR_INFO,
+    PlatformConfig,
+    SignalStrength,
+    Timeframe,
+)
+from signal_aggregator import SignalAggregator, SignalReport  # noqa: E402
+
+# Reuse GPU backtest infrastructure (data loading, indicator pre-compute, etc.)
+from gpu_backtest import (  # noqa: E402
+    BAR_SECONDS,
+    GPUTechnicalEngine,
+    PAIR_ALIASES,
+    BacktestConfig,
+    IndicatorCache,
+    fetch_history,
+    precompute_indicators,
+    _is_xau_blocked_by_calendar,
+)
+from data_provider import ForexDataProvider  # noqa: E402
+
+logger = logging.getLogger("diagnose_filters")
+
+
+# ═════════════════════════════════════════════════════════════════════
+#  OUTCOME CLASSIFICATION
+# ═════════════════════════════════════════════════════════════════════
+
+def classify_bar(
+    report: SignalReport,
+    is_xau: bool,
+) -> Tuple[str, List[str]]:
+    """Classify a bar's outcome and list which filters killed it.
+
+    Returns:
+        (outcome, [filter_names])
+
+    Outcomes:
+        "TRADE"           — quality C+ and trade_setup exists
+        "QUALITY_OK_NO_SETUP" — quality C+ but trade_setup is None
+        "LOW_QUALITY"     — signal reached BUY/SELL but quality D or F
+        "NEUTRAL"         — signal stayed NEUTRAL (score didn't reach +/-30)
+
+    Filter names are extracted from report.warnings.
+    """
+    filters_hit: List[str] = []
+
+    # Scan warnings for known filter tags
+    filter_tags = {
+        "H4 RSI GATE":       "H4_RSI_GATE",
+        "H1 MACD GATE":      "H1_MACD_GATE",
+        "RSI ZONE GATE":     "RSI_ZONE_GATE",
+        "XAU BUY FILTER":    "XAU_BUY_FILTER",
+        "SESSION FILTER":    "SESSION_FILTER",
+        "DAY FILTER":        "DAY_FILTER",
+        "LOW ATR FILTER":    "LOW_ATR_FILTER",
+        "SELL SUPPRESSED":   "SELL_SUPPRESSED",
+        "EXHAUSTION":        "EXHAUSTION_PENALTY",
+        "TREND EXHAUSTION":  "TREND_EXHAUSTION",
+        "NO PULLBACK":       "NO_PULLBACK_PENALTY",
+        "Weak pullback":     "WEAK_PULLBACK_PENALTY",
+        "Extended entry":    "EXTENDED_ENTRY_PENALTY",
+        "Counter-trend":     "COUNTER_TREND_PENALTY",
+        "EXTREME volatility":"EXTREME_VOL_PENALTY",
+        "NEAR EMA200":       "NEAR_EMA200_PENALTY",
+        "FAR FROM EMA200":   "FAR_EMA200_PENALTY",
+        "EVENT RISK":        "EVENT_RISK",
+        "REGIME FILTER":     "REGIME_FILTER",
+        "Timeframes CONFLICT": "MTF_CONFLICT",
+    }
+
+    for w in report.warnings:
+        for tag, name in filter_tags.items():
+            if tag in w:
+                filters_hit.append(name)
+
+    # Determine outcome
+    non_neutral = report.final_signal not in (SignalStrength.NEUTRAL,)
+    quality_ok = report.signal_quality in ("A+", "A", "B", "C")
+
+    if quality_ok and report.trade_setup is not None:
+        return "TRADE", filters_hit
+    elif quality_ok and report.trade_setup is None:
+        return "QUALITY_OK_NO_SETUP", filters_hit
+    elif non_neutral and not quality_ok:
+        return "LOW_QUALITY", filters_hit
+    else:
+        return "NEUTRAL", filters_hit
+
+
+# ═════════════════════════════════════════════════════════════════════
+#  SCORE DISTRIBUTION TRACKER
+# ═════════════════════════════════════════════════════════════════════
+
+class ScoreTracker:
+    """Tracks score/confidence distributions for analysis."""
+
+    def __init__(self):
+        self.scores: List[float] = []
+        self.confidences: List[float] = []
+        self.abs_scores: List[float] = []
+        # How many bars had score in various ranges
+        self.score_bins = Counter()
+        self.conf_bins = Counter()
+        # Track "near miss" bars — score close to thresholds
+        self.near_miss_score: int = 0  # score 20-29 (almost BUY/SELL)
+        self.near_miss_conf: int = 0   # confidence 45-54 (almost passing)
+
+    def record(self, score: float, confidence: float):
+        self.scores.append(score)
+        self.confidences.append(confidence)
+        self.abs_scores.append(abs(score))
+
+        # Score bins
+        a = abs(score)
+        if a < 10:
+            self.score_bins["0-9"] += 1
+        elif a < 20:
+            self.score_bins["10-19"] += 1
+        elif a < 30:
+            self.score_bins["20-29"] += 1
+            self.near_miss_score += 1
+        elif a < 40:
+            self.score_bins["30-39"] += 1
+        elif a < 50:
+            self.score_bins["40-49"] += 1
+        else:
+            self.score_bins["50+"] += 1
+
+        # Confidence bins
+        if confidence < 30:
+            self.conf_bins["0-29"] += 1
+        elif confidence < 45:
+            self.conf_bins["30-44"] += 1
+        elif confidence < 55:
+            self.conf_bins["45-54"] += 1
+            self.near_miss_conf += 1
+        elif confidence < 65:
+            self.conf_bins["55-64"] += 1
+        elif confidence < 75:
+            self.conf_bins["65-74"] += 1
+        else:
+            self.conf_bins["75+"] += 1
+
+
+# ═════════════════════════════════════════════════════════════════════
+#  MAIN DIAGNOSTIC LOOP
+# ═════════════════════════════════════════════════════════════════════
+
+def run_diagnosis(
+    pair: str,
+    start: datetime,
+    end: datetime,
+    sample_every: int = 4,
+    use_cache: bool = True,
+    primary_tf: Timeframe = Timeframe.M15,
+    confirm_tfs: Optional[List[Timeframe]] = None,
+    warmup_bars: int = 250,
+) -> None:
+    if confirm_tfs is None:
+        confirm_tfs = [Timeframe.H1, Timeframe.H4]
+
+    pair_info = PAIR_INFO[pair]
+    display = pair_info["display"]
+    is_xau = display == "XAU/USD"
+
+    platform_cfg = PlatformConfig(
+        symbol=pair,
+        display_name=display,
+        pip_value=pair_info["pip_value"],
+        primary_timeframe=primary_tf,
+        confirmation_timeframes=list(confirm_tfs),
+    )
+
+    provider = ForexDataProvider(platform_cfg)
+
+    # ── Load data ──
+    timeframes = [primary_tf] + list(confirm_tfs)
+    pad_days = max(60, warmup_bars // 4)
+    fetch_start = start - timedelta(days=pad_days)
+
+    dfs_full: Dict[Timeframe, pd.DataFrame] = {}
+    for tf in timeframes:
+        df = fetch_history(
+            provider, pair, tf,
+            start=fetch_start, end=end,
+            use_cache=use_cache,
+        )
+        if df.empty:
+            raise RuntimeError(f"No data for {pair} {tf.value}")
+        dfs_full[tf] = df
+
+    # ── Pre-compute indicators ──
+    t0 = time.perf_counter()
+    caches: Dict[str, IndicatorCache] = {
+        tf.value: precompute_indicators(dfs_full[tf], platform_cfg.technical, gpu_log=True)
+        for tf in timeframes
+    }
+    logger.info(f"Indicator pre-compute: {time.perf_counter() - t0:.2f}s")
+
+    # ── Build aggregator with GPU engine ──
+    aggregator = SignalAggregator(platform_cfg)
+    aggregator.tech_engine = GPUTechnicalEngine(platform_cfg.technical, caches)
+
+    # ── Walk primary TF ──
+    primary_df = dfs_full[primary_tf]
+    bt_start_ts = pd.Timestamp(start)
+    bt_end_ts = pd.Timestamp(end)
+
+    first_eligible = max(
+        warmup_bars,
+        int(primary_df.index.searchsorted(bt_start_ts)),
+    )
+    last_pos = min(
+        int(primary_df.index.searchsorted(bt_end_ts, side="right")),
+        len(primary_df),
+    )
+
+    primary_dur = pd.Timedelta(seconds=BAR_SECONDS[primary_tf])
+
+    # Counters
+    total_bars_in_window = 0
+    total_bars_sampled = 0
+    outcome_counts = Counter()
+    filter_counts = Counter()
+    # Filters that killed bars that were otherwise directional (non-NEUTRAL score)
+    directional_filter_counts = Counter()
+    # Pre-pipeline rejections (XAU calendar filters applied before analyze_from_dataframes)
+    pre_pipeline = Counter()
+
+    # Track signal directions
+    signal_dir_counts = Counter()
+    quality_counts = Counter()
+
+    # Score tracker
+    tracker = ScoreTracker()
+
+    # Track "would-have-been" trades: bars where the primary score was
+    # directional but got killed by a specific filter
+    filter_kill_breakdown = defaultdict(Counter)
+
+    logger.info(
+        f"Diagnosing {display} {primary_tf.value}: "
+        f"bars {first_eligible}-{last_pos} (every {sample_every}th bar)"
+    )
+
+    t_loop = time.perf_counter()
+    for pos in range(first_eligible, last_pos, sample_every):
+        total_bars_in_window += 1
+        ts = primary_df.index[pos]
+        decision_time = ts + primary_dur
+
+        # ── XAU calendar pre-filter (same as gpu_backtest.py) ──
+        if is_xau:
+            block = _is_xau_blocked_by_calendar(ts, platform_cfg)
+            if block:
+                pre_pipeline[block] += 1
+                continue
+
+        total_bars_sampled += 1
+
+        # ── Build sliced dfs (same logic as gpu_backtest.py) ──
+        slice_dfs: Dict[Timeframe, pd.DataFrame] = {}
+        for tf in timeframes:
+            full = dfs_full[tf]
+            bar_dur = pd.Timedelta(seconds=BAR_SECONDS[tf])
+            cutoff = decision_time - bar_dur
+            cut = int(full.index.searchsorted(cutoff, side="right"))
+            if cut == 0:
+                continue
+            slice_dfs[tf] = full.iloc[:cut]
+
+        if primary_tf not in slice_dfs:
+            continue
+
+        # ── Run the full signal pipeline ──
+        report: SignalReport = aggregator.analyze_from_dataframes(
+            symbol=pair,
+            dfs=slice_dfs,
+            context=None,
+            use_news=False,
+        )
+
+        # Record score/confidence
+        tracker.record(report.final_score, report.final_confidence)
+
+        # Signal direction
+        signal_dir_counts[report.final_signal.name] += 1
+        quality_counts[report.signal_quality] += 1
+
+        # Classify
+        outcome, filters_hit = classify_bar(report, is_xau)
+        outcome_counts[outcome] += 1
+
+        for f in filters_hit:
+            filter_counts[f] += 1
+
+        # If the primary score was directional (>= 15 or <= -15), the bar
+        # had some potential — track what killed it
+        if abs(report.primary_score) >= 15:
+            for f in filters_hit:
+                directional_filter_counts[f] += 1
+
+        # For bars that got quality D/F but had a directional signal,
+        # track which filter warnings appeared
+        if outcome in ("LOW_QUALITY", "QUALITY_OK_NO_SETUP"):
+            for f in filters_hit:
+                filter_kill_breakdown[outcome][f] += 1
+
+    elapsed = time.perf_counter() - t_loop
+
+    # ═════════════════════════════════════════════════════════════════
+    #  PRINT RESULTS
+    # ═════════════════════════════════════════════════════════════════
+
+    bar = "=" * 80
+    print(f"\n{bar}")
+    print(f"  FILTER REJECTION DIAGNOSTIC — {display}")
+    print(f"  {start.date()} to {end.date()} | {primary_tf.value} "
+          f"(sampled every {sample_every}th bar)")
+    print(f"{bar}")
+
+    # Scale counts if sampling
+    scale_note = ""
+    if sample_every > 1:
+        scale_note = f"  (sampled {total_bars_sampled} of ~{total_bars_in_window * sample_every} eligible bars)"
+    print(f"\n  Total bars in backtest window: {total_bars_in_window}")
+    if pre_pipeline:
+        for reason, count in sorted(pre_pipeline.items()):
+            print(f"  Pre-pipeline rejection ({reason}): {count}")
+    print(f"  Bars analyzed: {total_bars_sampled}{scale_note}")
+    print(f"  Analysis took: {elapsed:.1f}s "
+          f"({total_bars_sampled / max(elapsed, 0.001):.0f} bars/s)")
+
+    # ── Outcome breakdown ──
+    print(f"\n{'-' * 80}")
+    print("  OUTCOME BREAKDOWN")
+    print(f"{'-' * 80}")
+    outcome_order = ["TRADE", "QUALITY_OK_NO_SETUP", "LOW_QUALITY", "NEUTRAL"]
+    outcome_labels = {
+        "TRADE":               "Actual trades (quality C+ & setup exists)",
+        "QUALITY_OK_NO_SETUP": "Quality C+ but NO trade setup generated",
+        "LOW_QUALITY":         "Directional signal but quality D/F",
+        "NEUTRAL":             "NEUTRAL (score didn't reach +/-30 or conf < 55)",
+    }
+    for o in outcome_order:
+        count = outcome_counts.get(o, 0)
+        pct = count / max(total_bars_sampled, 1) * 100
+        label = outcome_labels.get(o, o)
+        print(f"  {count:6d} ({pct:5.1f}%)  {label}")
+
+    # ── Signal direction breakdown ──
+    print(f"\n{'-' * 80}")
+    print("  SIGNAL DIRECTION BREAKDOWN")
+    print(f"{'-' * 80}")
+    for name in ["STRONG_BUY", "BUY", "NEUTRAL", "SELL", "STRONG_SELL"]:
+        count = signal_dir_counts.get(name, 0)
+        pct = count / max(total_bars_sampled, 1) * 100
+        print(f"  {count:6d} ({pct:5.1f}%)  {name}")
+
+    # ── Quality grade breakdown ──
+    print(f"\n{'-' * 80}")
+    print("  QUALITY GRADE BREAKDOWN")
+    print(f"{'-' * 80}")
+    for grade in ["A+", "A", "B", "C", "D", "F"]:
+        count = quality_counts.get(grade, 0)
+        pct = count / max(total_bars_sampled, 1) * 100
+        print(f"  {count:6d} ({pct:5.1f}%)  Grade {grade}")
+
+    # ── Filter hit counts (all bars) ──
+    print(f"\n{'-' * 80}")
+    print("  FILTER HIT COUNTS (all bars, a bar can hit multiple filters)")
+    print(f"{'-' * 80}")
+    if filter_counts:
+        for name, count in filter_counts.most_common():
+            pct = count / max(total_bars_sampled, 1) * 100
+            print(f"  {count:6d} ({pct:5.1f}%)  {name}")
+    else:
+        print("  (no filters triggered)")
+
+    # ── Filters that killed directional bars (primary_score >= 15) ──
+    print(f"\n{'-' * 80}")
+    print("  FILTERS KILLING DIRECTIONAL BARS (|primary_score| >= 15)")
+    print(f"{'-' * 80}")
+    if directional_filter_counts:
+        for name, count in directional_filter_counts.most_common():
+            print(f"  {count:6d}  {name}")
+    else:
+        print("  (none)")
+
+    # ── Filter breakdown for LOW_QUALITY bars ──
+    if filter_kill_breakdown.get("LOW_QUALITY"):
+        print(f"\n{'-' * 80}")
+        print("  FILTERS ON LOW_QUALITY BARS (had BUY/SELL but quality D/F)")
+        print(f"{'-' * 80}")
+        for name, count in filter_kill_breakdown["LOW_QUALITY"].most_common():
+            print(f"  {count:6d}  {name}")
+
+    # ── Filter breakdown for QUALITY_OK_NO_SETUP bars ──
+    if filter_kill_breakdown.get("QUALITY_OK_NO_SETUP"):
+        print(f"\n{'-' * 80}")
+        print("  FILTERS ON QUALITY_OK_NO_SETUP BARS (quality C+ but no trade setup)")
+        print(f"{'-' * 80}")
+        for name, count in filter_kill_breakdown["QUALITY_OK_NO_SETUP"].most_common():
+            print(f"  {count:6d}  {name}")
+
+    # ── Score distribution ──
+    print(f"\n{'-' * 80}")
+    print("  SCORE DISTRIBUTION (|score|)")
+    print(f"{'-' * 80}")
+    for bin_name in ["0-9", "10-19", "20-29", "30-39", "40-49", "50+"]:
+        count = tracker.score_bins.get(bin_name, 0)
+        pct = count / max(total_bars_sampled, 1) * 100
+        marker = " <-- near miss (almost BUY/SELL)" if bin_name == "20-29" else ""
+        print(f"  {count:6d} ({pct:5.1f}%)  |score| {bin_name}{marker}")
+
+    # ── Confidence distribution ──
+    print(f"\n{'-' * 80}")
+    print("  CONFIDENCE DISTRIBUTION")
+    print(f"{'-' * 80}")
+    for bin_name in ["0-29", "30-44", "45-54", "55-64", "65-74", "75+"]:
+        count = tracker.conf_bins.get(bin_name, 0)
+        pct = count / max(total_bars_sampled, 1) * 100
+        marker = " <-- near miss (almost passing)" if bin_name == "45-54" else ""
+        print(f"  {count:6d} ({pct:5.1f}%)  confidence {bin_name}{marker}")
+
+    # ── Near misses ──
+    print(f"\n{'-' * 80}")
+    print("  NEAR MISSES")
+    print(f"{'-' * 80}")
+    print(f"  {tracker.near_miss_score:6d}  Bars with |score| 20-29 (threshold is 30)")
+    print(f"  {tracker.near_miss_conf:6d}  Bars with confidence 45-54 (threshold is 55)")
+
+    # ── Summary statistics ──
+    if tracker.scores:
+        scores_arr = np.array(tracker.scores)
+        conf_arr = np.array(tracker.confidences)
+        print(f"\n{'-' * 80}")
+        print("  SCORE/CONFIDENCE STATISTICS")
+        print(f"{'-' * 80}")
+        print(f"  Score:      mean={np.mean(scores_arr):+.1f}  "
+              f"median={np.median(scores_arr):+.1f}  "
+              f"std={np.std(scores_arr):.1f}  "
+              f"min={np.min(scores_arr):+.1f}  "
+              f"max={np.max(scores_arr):+.1f}")
+        print(f"  Confidence: mean={np.mean(conf_arr):.1f}  "
+              f"median={np.median(conf_arr):.1f}  "
+              f"std={np.std(conf_arr):.1f}  "
+              f"min={np.min(conf_arr):.1f}  "
+              f"max={np.max(conf_arr):.1f}")
+
+    # ── Blocked hours analysis (for XAU) ──
+    if is_xau:
+        blocked = set(platform_cfg.xau_strategy.blocked_hours_utc)
+        all_hours = set(range(24))
+        open_hours = sorted(all_hours - blocked)
+        print(f"\n{'-' * 80}")
+        print("  XAU HOUR/DAY CONFIG")
+        print(f"{'-' * 80}")
+        print(f"  Blocked hours:  {sorted(blocked)}")
+        print(f"  Open hours:     {open_hours}")
+        print(f"  Allowed days:   {platform_cfg.xau_strategy.allowed_weekdays} "
+              f"(0=Mon..4=Fri)")
+        print(f"  Open fraction:  {len(open_hours)/24:.0%} of hours x "
+              f"{len(platform_cfg.xau_strategy.allowed_weekdays)}/5 weekdays = "
+              f"{len(open_hours)/24 * len(platform_cfg.xau_strategy.allowed_weekdays)/5:.0%} of trading time")
+
+    # ── Interpretation / key takeaways ──
+    print(f"\n{bar}")
+    print("  KEY TAKEAWAYS")
+    print(f"{bar}")
+
+    neutral_pct = outcome_counts.get("NEUTRAL", 0) / max(total_bars_sampled, 1) * 100
+    low_q_pct = outcome_counts.get("LOW_QUALITY", 0) / max(total_bars_sampled, 1) * 100
+    trade_count = outcome_counts.get("TRADE", 0)
+
+    print(f"  - {neutral_pct:.1f}% of bars never reach a directional signal (NEUTRAL)")
+    print(f"  - {low_q_pct:.1f}% of bars get a BUY/SELL but confidence/quality too low")
+    print(f"  - {trade_count} bars produce actual trades")
+
+    if filter_counts:
+        top3 = filter_counts.most_common(3)
+        print(f"  - Top rejection filters:")
+        for name, count in top3:
+            print(f"      {name}: {count} bars hit")
+
+    if tracker.near_miss_score > 50:
+        print(f"  - SIGNIFICANT: {tracker.near_miss_score} bars had |score| 20-29 "
+              f"(lowering threshold from 30 to 25 would capture many)")
+
+    if tracker.near_miss_conf > 50:
+        print(f"  - SIGNIFICANT: {tracker.near_miss_conf} bars had confidence 45-54 "
+              f"(lowering threshold from 55 to 50 would capture many)")
+
+    print(f"\n{bar}\n")
+
+
+# ═════════════════════════════════════════════════════════════════════
+#  CLI
+# ═════════════════════════════════════════════════════════════════════
+
+def _resolve_pair(s: str) -> str:
+    s_up = s.upper().strip()
+    if s_up in PAIR_ALIASES:
+        return PAIR_ALIASES[s_up]
+    if s_up in PAIR_INFO:
+        return s_up
+    raise SystemExit(f"Unknown pair '{s}'. Valid: {', '.join(sorted(PAIR_ALIASES))}")
+
+
+def _resolve_tf(s: str) -> Timeframe:
+    for tf in Timeframe:
+        if tf.value == s:
+            return tf
+    raise SystemExit(f"Unknown timeframe '{s}'")
+
+
+def main(argv: Optional[List[str]] = None) -> int:
+    p = argparse.ArgumentParser(
+        description="Diagnose which filters reject the most bars in the backtest pipeline",
+        formatter_class=argparse.ArgumentDefaultsHelpFormatter,
+    )
+    p.add_argument("--pair", required=True, help="XAUUSD | GOLD")
+    p.add_argument("--days", type=int, default=365, help="Days of history")
+    p.add_argument("--start", help="ISO date YYYY-MM-DD (overrides --days)")
+    p.add_argument("--end", help="ISO date YYYY-MM-DD (defaults to today)")
+    p.add_argument("--primary-tf", default="15min", help="Primary TF")
+    p.add_argument("--confirm-tfs", default="1h,4h", help="Comma-separated confirmation TFs")
+    p.add_argument("--warmup-bars", type=int, default=250)
+    p.add_argument("--sample-every", type=int, default=4,
+                   help="Analyze every Nth bar (1=all, 4=25%% sample for speed)")
+    p.add_argument("--no-cache", action="store_true", help="Skip parquet cache")
+    p.add_argument("--log-level", default="INFO")
+    args = p.parse_args(argv)
+
+    logging.basicConfig(
+        level=args.log_level.upper(),
+        format="%(asctime)s [%(levelname)s] %(name)s: %(message)s",
+    )
+
+    pair = _resolve_pair(args.pair)
+    primary_tf = _resolve_tf(args.primary_tf)
+    confirm_tfs = [_resolve_tf(s.strip()) for s in args.confirm_tfs.split(",") if s.strip()]
+
+    end = (
+        datetime.fromisoformat(args.end).replace(tzinfo=timezone.utc)
+        if args.end else datetime.now(timezone.utc)
+    )
+    if args.start:
+        start = datetime.fromisoformat(args.start).replace(tzinfo=timezone.utc)
+    else:
+        start = end - timedelta(days=args.days)
+
+    logger.info(
+        f"Diagnosing {PAIR_INFO[pair]['display']} | "
+        f"Window: {start.date()} -> {end.date()} | "
+        f"Sample: every {args.sample_every}th bar"
+    )
+
+    run_diagnosis(
+        pair=pair,
+        start=start,
+        end=end,
+        sample_every=args.sample_every,
+        use_cache=not args.no_cache,
+        primary_tf=primary_tf,
+        confirm_tfs=confirm_tfs,
+        warmup_bars=args.warmup_bars,
+    )
+
+    return 0
+
+
+if __name__ == "__main__":
+    sys.exit(main())

backtest/instructions_for_backtest.txt

@@ -0,0 +1,59 @@
+Note down entery and exit Why we took trade?
+What were each indicator signal on that specific time when we took trade
+How many indicators agreed at trade time
+Always backtest using engine.py and signal_aggregator.py
+Dont take trades on weekends and Fridays
+Keep TP small 1.3 Reward ratio
+At the end of test make table of results and print all values you observed.
+Trade context
+
+Time of day and trading session (Asian/London/NY/overlap)
+Day of week (track Mon/Tue/Wed/Thu performance separately)
+Market regime at entry: trending, ranging, or choppy (use ATR or ADX)
+Volatility level (ATR value or ATR percentile vs recent average)
+Pair/instrument traded
+News proximity (high-impact news within 30 min before/after entry)
+
+Trade execution details
+
+Spread at entry in pips
+Slippage on entry and exit (planned vs actual fill)
+Time in trade (minutes/hours from entry to exit)
+Maximum Adverse Excursion (MAE): how far against you it went
+Maximum Favorable Excursion (MFE): how far in profit before pullback
+
+Risk metrics
+
+Max consecutive losses (streak)
+Max consecutive wins (streak)
+Expectancy: (Win% x Avg Win) minus (Loss% x Avg Loss)
+Expectancy per hour in trade
+Recovery factor: Total R divided by Max DD
+
+Indicator/signal analysis
+
+Each indicator's standalone win rate when it agreed with the trade
+Confluence level performance (2 vs 3 vs 4+ indicators agreeing)
+Which indicator disagreed on losing trades (find the "canary")
+
+Decision quality
+
+Rate each setup A/B/C at entry, before knowing outcome
+Verify A-grade setups actually win more often
+Track system trades vs discretionary deviations separately
+
+Process notes
+
+Screenshot or chart snapshot at entry and exit
+Emotional state (calm, FOMO, revenge, tired)
+What would have happened if you held longer or exited sooner
+
+Aggregate analysis at end of backtest
+
+Equity curve, not just total R
+Performance breakdown by session, day, volatility regime
+Win rate over rolling 20-trade windows (check for edge decay)
+
+Critical extra
+"Do not trade" log: setups flagged but skipped, and what happened after
+

backtest/research/__init__.py

@@ -0,0 +1,5 @@
+"""Research tooling: calendar sourcing/store and edge-screening utilities.
+
+Nothing here is imported by the live bot; these modules support
+backtesting and edge validation only.
+"""

backtest/research/calendar_store.py

@@ -0,0 +1,94 @@
+"""Load and query the historical economic calendar (backtest side).
+
+The store is built by `ff_calendar_scraper.py`. Timestamps are unix
+epoch (UTC) — verified against known release times and price-volatility
+alignment (see scraper docstring).
+
+Key API:
+    cal = load_calendar()                          # full DataFrame, `utc` column
+    ev  = precise_events(cal, ["USD"])             # high-impact, minute-accurate
+    mask = event_window_mask(bar_index, ev.utc, before, after)
+"""
+from pathlib import Path
+from typing import Iterable, Optional
+
+import numpy as np
+import pandas as pd
+
+PARQUET = Path(__file__).resolve().parent.parent / "cache" / "calendar.parquet"
+
+# timeLabel values that mean "no intraday time" — these events cannot
+# anchor an M5/M15 window (German Prelim CPI is the notable High one).
+_UNTIMED = {"All Day", "Tentative", ""}
+
+
+def _parse_release_value(raw) -> float:
+    """'0.3%' / '-1.7%' / '228K' / '1.1M' -> float (nan if unparseable)."""
+    import re
+    if raw is None or (isinstance(raw, float) and np.isnan(raw)):
+        return np.nan
+    m = re.search(r"(-?\d+(?:\.\d+)?)\s*([kmbt])?",
+                  str(raw).replace(",", ""), re.IGNORECASE)
+    if not m:
+        return np.nan
+    mult = {"k": 1e3, "m": 1e6, "b": 1e9, "t": 1e12}.get(
+        (m.group(2) or "").lower(), 1.0)
+    return float(m.group(1)) * mult
+
+
+def load_calendar(path: Path = PARQUET) -> pd.DataFrame:
+    df = pd.read_parquet(path)
+    df["utc"] = pd.to_datetime(df.dateline, unit="s", utc=True)
+    for col in ("actual", "forecast", "previous"):
+        df[f"{col}_num"] = df[col].map(_parse_release_value)
+    df["surprise"] = df.actual_num - df.forecast_num
+    return df
+
+
+def precise_events(
+    cal: pd.DataFrame,
+    currencies: Iterable[str] = ("USD",),
+    impacts: Iterable[str] = ("high",),
+    require_values: bool = False,
+) -> pd.DataFrame:
+    """Events with a minute-accurate timestamp, filtered by currency/impact."""
+    out = cal[cal.currency.isin(list(currencies))
+              & cal.impactName.isin(list(impacts))
+              & ~cal.timeMasked.astype(bool)
+              & ~cal.timeLabel.fillna("").isin(_UNTIMED)]
+    if require_values:
+        out = out.dropna(subset=["actual_num", "forecast_num"])
+    return out.sort_values("utc").reset_index(drop=True)
+
+
+def event_window_mask(
+    bar_index: pd.DatetimeIndex,
+    event_times: pd.Series,
+    before: pd.Timedelta = pd.Timedelta(minutes=30),
+    after: pd.Timedelta = pd.Timedelta(minutes=30),
+) -> np.ndarray:
+    """Boolean mask: bar OPEN time falls inside [event-before, event+after).
+
+    Use on a backtest's entry times to identify trades entered around
+    releases (avoidance counterfactuals, event-window cost models).
+    """
+    starts = pd.DatetimeIndex(event_times) - before
+    ends = pd.DatetimeIndex(event_times) + after
+    order = np.argsort(starts.values)
+    starts, ends = starts.values[order], ends.values[order]
+    # merge overlapping windows, then interval-stab each bar time
+    merged_s, merged_e = [], []
+    for s, e in zip(starts, ends):
+        if merged_e and s <= merged_e[-1]:
+            merged_e[-1] = max(merged_e[-1], e)
+        else:
+            merged_s.append(s)
+            merged_e.append(e)
+    ms = np.array(merged_s)
+    me = np.array(merged_e)
+    ts = bar_index.values
+    pos = np.searchsorted(ms, ts, side="right") - 1
+    ok = pos >= 0
+    res = np.zeros(len(ts), dtype=bool)
+    res[ok] = ts[ok] < me[pos[ok]]
+    return res

backtest/research/ff_calendar_scraper.py

@@ -0,0 +1,144 @@
+"""Forex Factory calendar scraper -> backtest/cache/calendar.parquet.
+
+Fetches weekly calendar pages and parses the event JSON embedded in
+`window.calendarComponentStates` (the `days:` array). Event `dateline`
+is a unix epoch, so timestamps carry no timezone ambiguity — verified
+2026-07-11 against known release times (NFP 12:30/13:30 UTC across DST,
+ECB 12:15/13:15) and against EUR/USD M15 volatility: the bar containing
+a high-impact event averages 3.5x normal range, and shifting event
+times by ±1h destroys that alignment.
+
+Usage:
+    python backtest/research/ff_calendar_scraper.py --start 2024-01-01 --end 2026-07-13
+    python backtest/research/ff_calendar_scraper.py            # refresh: last stored week -> today
+
+Merges into the existing parquet on event (id, dateline), so refreshing
+is idempotent and also picks up revised `actual` values for recent weeks.
+Be polite: ~1.6s between requests (a full year is ~52 requests).
+"""
+import argparse
+import json
+import time
+from datetime import date, timedelta
+from pathlib import Path
+from typing import List, Optional
+
+import pandas as pd
+import requests
+
+CACHE_DIR = Path(__file__).resolve().parent.parent / "cache"
+PARQUET = CACHE_DIR / "calendar.parquet"
+
+UA = ("Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 "
+      "(KHTML, like Gecko) Chrome/126.0 Safari/537.36")
+
+FIELDS = ["id", "dateline", "name", "country", "currency", "impactName",
+          "timeLabel", "timeMasked", "actual", "forecast", "previous",
+          "revision", "actualBetterWorse"]
+
+
+def week_url(d: date) -> str:
+    return (f"https://www.forexfactory.com/calendar?week="
+            f"{d.strftime('%b').lower()}{d.day}.{d.year}")
+
+
+def extract_days(html: str) -> Optional[list]:
+    """Balanced-bracket extraction of the `days:` JSON array."""
+    i = html.find("days:")
+    if i < 0:
+        return None
+    j = html.find("[", i)
+    depth, in_str, esc = 0, False, False
+    for k in range(j, len(html)):
+        ch = html[k]
+        if in_str:
+            if esc:
+                esc = False
+            elif ch == "\\":
+                esc = True
+            elif ch == '"':
+                in_str = False
+        else:
+            if ch == '"':
+                in_str = True
+            elif ch == "[":
+                depth += 1
+            elif ch == "]":
+                depth -= 1
+                if depth == 0:
+                    return json.loads(html[j:k + 1])
+    return None
+
+
+def fetch_week(sess: requests.Session, d: date, retries: int = 4) -> Optional[List[dict]]:
+    for attempt in range(retries):
+        try:
+            r = sess.get(week_url(d), timeout=30)
+            if r.status_code == 200:
+                days = extract_days(r.text)
+                if days is not None:
+                    return [{k: e.get(k) for k in FIELDS}
+                            for day in days for e in day.get("events", [])]
+        except requests.RequestException:
+            pass
+        time.sleep(5 * (attempt + 1))
+    return None
+
+
+def scrape(start: date, end: date) -> pd.DataFrame:
+    sess = requests.Session()
+    sess.headers["User-Agent"] = UA
+    rows: List[dict] = []
+    failed: List[date] = []
+    d = start
+    while d <= end:
+        week_rows = fetch_week(sess, d)
+        if week_rows is None:
+            failed.append(d)
+            print(f"FAILED week {d}")
+        else:
+            rows.extend(week_rows)
+        d += timedelta(days=7)
+        time.sleep(1.6)
+    if failed:
+        print(f"WARNING: {len(failed)} weeks failed: {failed}")
+    return pd.DataFrame(rows)
+
+
+def merge_into_store(new: pd.DataFrame, path: Path = PARQUET) -> pd.DataFrame:
+    if path.exists():
+        old = pd.read_parquet(path)
+        combined = pd.concat([old, new])
+    else:
+        combined = new
+    # keep="last": a refresh overwrites stale rows (actuals fill in
+    # after release, revisions land weeks later).
+    combined = (combined.drop_duplicates(subset=["id", "dateline"], keep="last")
+                .sort_values("dateline").reset_index(drop=True))
+    combined.to_parquet(path, index=False)
+    return combined
+
+
+def main() -> None:
+    ap = argparse.ArgumentParser(description=__doc__.splitlines()[0])
+    ap.add_argument("--start", type=date.fromisoformat, default=None)
+    ap.add_argument("--end", type=date.fromisoformat, default=date.today())
+    args = ap.parse_args()
+
+    start = args.start
+    if start is None:
+        if PARQUET.exists():
+            last = pd.read_parquet(PARQUET).dateline.max()
+            start = pd.Timestamp(last, unit="s").date() - timedelta(days=7)
+        else:
+            start = date(2024, 1, 1)
+    print(f"Scraping {start} -> {args.end}")
+    new = scrape(start, args.end)
+    combined = merge_into_store(new)
+    print(f"Store: {len(combined)} events "
+          f"({pd.Timestamp(combined.dateline.min(), unit='s')} -> "
+          f"{pd.Timestamp(combined.dateline.max(), unit='s')}) -> {PARQUET}")
+
+
+if __name__ == "__main__":
+    main()

chart_patterns.py

@@ -0,0 +1,287 @@
+"""
+Classical chart-pattern detection for the XAU_CP entry archetype.
+================================================================
+
+Third entry archetype (after the composite-breakout generators and the
+XAU_PB pullback): trade the completion of a proven multi-bar price
+pattern. Pattern set is deliberately tight — the classical patterns
+with the strongest documented statistics (Bulkowski, *Encyclopedia of
+Chart Patterns*) that are unambiguous from OHLC at bar close:
+
+    double_bottom  bullish reversal   two matching swing lows, break of
+                                      the interim peak (neckline)
+    double_top     bearish reversal   mirrored
+    inv_hs         bullish reversal   inverse head & shoulders, break of
+                                      the neckline (max of the two peaks
+                                      flanking the head — conservative
+                                      horizontal line, no sloped fits)
+    hs_top         bearish reversal   head & shoulders top, mirrored
+    bull_flag      bullish continuation  strong pole, shallow ordered
+                                      retrace, close at a new high
+    bear_flag      bearish continuation  mirrored
+
+Triangles / wedges / cup-and-handle are excluded on purpose: weaker
+historical stats and apex-timing ambiguity — more parameters for less
+edge.
+
+Design contract (mirrors the rest of the signal stack):
+- Detectors see ONLY the visible closed-bar slice the aggregator was
+  given; the last row is the just-closed signal bar. No forming candle,
+  no future bars → identical behavior live and in the bar-replay
+  backtester, and signals never repaint.
+- All geometric thresholds are in ATR units so one parameter set is
+  scale-free across timeframes. ATR comes from the engine (Wilder 14).
+- Trigger discipline: a pattern fires only on the bar that COMPLETES it
+  — the first close beyond the neckline/boundary since the pattern
+  formed. One breakout → at most one signal.
+- Swing points use the same local-extremum definition as the engine's
+  structure S/R (window w bars each side), so a "swing" here is the
+  same object a chartist sees on the engine's key levels.
+
+Every detector returns None or a dict:
+    {"pattern": str, "direction": "BUY"|"SELL",
+     "anchor": float,     # SL reference (pattern invalidation price)
+     "trigger": float,    # neckline / boundary that was broken
+     "height": float}     # measured-move distance in price units
+
+Defaults below are the textbook definitions, not tuned values. The
+validation record for which patterns/timeframes are enabled lives in
+pair_signals/xauusd.py (per repo rules, per-TF adoption happens in the
+factory, never in global defaults).
+"""
+
+from typing import Dict, List, Optional
+
+import numpy as np
+import pandas as pd
+
+__all__ = ["detect_chart_pattern", "PATTERN_DETECTORS"]
+
+
+def _swing_indices(values: np.ndarray, w: int, find_high: bool) -> List[int]:
+    """Indices of local extrema with `w` bars on each side.
+
+    A bar i is a swing high if values[i] is the strict maximum of
+    values[i-w : i+w+1] (ties reject — a flat top is not a clean swing).
+    The last `w` bars can never qualify, which is exactly right: a swing
+    isn't confirmed until `w` bars close after it.
+    """
+    n = len(values)
+    out: List[int] = []
+    for i in range(w, n - w):
+        window = values[i - w : i + w + 1]
+        if np.isnan(window).any():
+            continue
+        center = values[i]
+        rest = np.delete(window, w)
+        if find_high:
+            if center > rest.max():
+                out.append(i)
+        else:
+            if center < rest.min():
+                out.append(i)
+    return out
+
+
+def _first_close_beyond(closes: np.ndarray, start: int, level: float,
+                        above: bool) -> bool:
+    """True iff the LAST close is the first one beyond `level` since
+    `start` (exclusive of the last bar)."""
+    last = closes[-1]
+    prior = closes[start:-1]
+    if above:
+        return last > level and (len(prior) == 0 or prior.max() <= level)
+    return last < level and (len(prior) == 0 or prior.min() >= level)
+
+
+# ── Double bottom / top ──────────────────────────────────────────────
+
+def _detect_double(o, h, l, c, atr, cfg, bullish: bool) -> Optional[Dict]:
+    w = cfg.cp_swing_window
+    ext = l if bullish else h
+    idx = _swing_indices(ext, w, find_high=not bullish)
+    if len(idx) < 2:
+        return None
+    i1, i2 = idx[-2], idx[-1]
+    n = len(c)
+    if i2 - i1 < cfg.cp_min_sep_bars:
+        return None
+    if (n - 1) - i2 > cfg.cp_recency_bars:
+        return None
+    e1, e2 = ext[i1], ext[i2]
+    if abs(e1 - e2) > cfg.cp_tolerance_atr * atr:
+        return None
+
+    if bullish:
+        neckline = float(h[i1 : i2 + 1].max())
+        floor = min(e1, e2)
+        height = neckline - floor
+    else:
+        neckline = float(l[i1 : i2 + 1].min())
+        ceil_ = max(e1, e2)
+        height = ceil_ - neckline
+
+    if not (cfg.cp_min_height_atr * atr <= height <= cfg.cp_max_height_atr * atr):
+        return None
+
+    # Pattern must not have been invalidated after the second extreme:
+    # a lower low (bullish) / higher high (bearish) breaks the pattern.
+    if bullish:
+        if i2 + 1 < n and l[i2 + 1 :].min() < min(e1, e2) - 0.1 * atr:
+            return None
+        if not _first_close_beyond(c, i2, neckline, above=True):
+            return None
+        return {"pattern": "double_bottom", "direction": "BUY",
+                "anchor": float(min(e1, e2)), "trigger": neckline,
+                "height": float(height)}
+    else:
+        if i2 + 1 < n and h[i2 + 1 :].max() > max(e1, e2) + 0.1 * atr:
+            return None
+        if not _first_close_beyond(c, i2, neckline, above=False):
+            return None
+        return {"pattern": "double_top", "direction": "SELL",
+                "anchor": float(max(e1, e2)), "trigger": neckline,
+                "height": float(height)}
+
+
+# ── Head & shoulders (top and inverse) ───────────────────────────────
+
+def _detect_hs(o, h, l, c, atr, cfg, bullish: bool) -> Optional[Dict]:
+    """bullish=True → inverse H&S (three swing LOWS, head deepest)."""
+    w = cfg.cp_swing_window
+    ext = l if bullish else h
+    idx = _swing_indices(ext, w, find_high=not bullish)
+    if len(idx) < 3:
+        return None
+    i1, i2, i3 = idx[-3], idx[-2], idx[-1]
+    n = len(c)
+    if (n - 1) - i3 > cfg.cp_recency_bars:
+        return None
+    s1, head, s2 = ext[i1], ext[i2], ext[i3]
+
+    if bullish:
+        # Head strictly the deepest low; shoulders roughly level.
+        if not (head < s1 - cfg.cp_hs_head_min_atr * atr
+                and head < s2 - cfg.cp_hs_head_min_atr * atr):
+            return None
+        if abs(s1 - s2) > cfg.cp_hs_shoulder_tol_atr * atr:
+            return None
+        # Neckline: the LOWER of the two interim peaks — the price must
+        # clear both, so this is the conservative horizontal trigger.
+        p1 = float(h[i1 : i2 + 1].max())
+        p2 = float(h[i2 : i3 + 1].max())
+        neckline = min(p1, p2)
+        height = neckline - head
+    else:
+        if not (head > s1 + cfg.cp_hs_head_min_atr * atr
+                and head > s2 + cfg.cp_hs_head_min_atr * atr):
+            return None
+        if abs(s1 - s2) > cfg.cp_hs_shoulder_tol_atr * atr:
+            return None
+        t1 = float(l[i1 : i2 + 1].min())
+        t2 = float(l[i2 : i3 + 1].min())
+        neckline = max(t1, t2)
+        height = head - neckline
+
+    if not (cfg.cp_min_height_atr * atr <= height <= cfg.cp_max_height_atr * atr):
+        return None
+
+    if bullish:
+        # Invalidated if price undercut the head after the right shoulder.
+        if i3 + 1 < n and l[i3 + 1 :].min() < head - 0.1 * atr:
+            return None
+        if not _first_close_beyond(c, i3, neckline, above=True):
+            return None
+        return {"pattern": "inv_hs", "direction": "BUY",
+                "anchor": float(s2), "trigger": neckline,
+                "height": float(height)}
+    else:
+        if i3 + 1 < n and h[i3 + 1 :].max() > head + 0.1 * atr:
+            return None
+        if not _first_close_beyond(c, i3, neckline, above=False):
+            return None
+        return {"pattern": "hs_top", "direction": "SELL",
+                "anchor": float(s2), "trigger": neckline,
+                "height": float(height)}
+
+
+# ── Flags (bull and bear) ────────────────────────────────────────────
+
+def _detect_flag(o, h, l, c, atr, cfg, bullish: bool) -> Optional[Dict]:
+    n = len(c)
+    lo_r, hi_r = cfg.cp_flag_retrace
+    if bullish:
+        # Pole top = highest high BEFORE the trigger bar.
+        ip = int(np.argmax(h[:-1]))
+        flag_len = (n - 1) - ip
+        if not (cfg.cp_flag_min_bars <= flag_len <= cfg.cp_flag_max_bars):
+            return None
+        pole_start = max(0, ip - cfg.cp_flag_pole_bars + 1)
+        pole_low = float(l[pole_start : ip + 1].min())
+        pole = float(h[ip]) - pole_low
+        if pole < cfg.cp_flag_pole_atr * atr:
+            return None
+        flag_low = float(l[ip + 1 : n].min())
+        retrace = (float(h[ip]) - flag_low) / pole
+        if not (lo_r <= retrace <= hi_r):
+            return None
+        if not _first_close_beyond(c, ip, float(h[ip]), above=True):
+            return None
+        return {"pattern": "bull_flag", "direction": "BUY",
+                "anchor": flag_low, "trigger": float(h[ip]),
+                "height": float(pole)}
+    else:
+        ip = int(np.argmin(l[:-1]))
+        flag_len = (n - 1) - ip
+        if not (cfg.cp_flag_min_bars <= flag_len <= cfg.cp_flag_max_bars):
+            return None
+        pole_start = max(0, ip - cfg.cp_flag_pole_bars + 1)
+        pole_high = float(h[pole_start : ip + 1].max())
+        pole = pole_high - float(l[ip])
+        if pole < cfg.cp_flag_pole_atr * atr:
+            return None
+        flag_high = float(h[ip + 1 : n].max())
+        retrace = (flag_high - float(l[ip])) / pole
+        if not (lo_r <= retrace <= hi_r):
+            return None
+        if not _first_close_beyond(c, ip, float(l[ip]), above=False):
+            return None
+        return {"pattern": "bear_flag", "direction": "SELL",
+                "anchor": flag_high, "trigger": float(l[ip]),
+                "height": float(pole)}
+
+
+# Priority order: most structured / rarest first — if two patterns
+# complete on the same bar, the more specific one names the trade.
+PATTERN_DETECTORS = {
+    "inv_hs":        lambda o, h, l, c, atr, cfg: _detect_hs(o, h, l, c, atr, cfg, True),
+    "hs_top":        lambda o, h, l, c, atr, cfg: _detect_hs(o, h, l, c, atr, cfg, False),
+    "double_bottom": lambda o, h, l, c, atr, cfg: _detect_double(o, h, l, c, atr, cfg, True),
+    "double_top":    lambda o, h, l, c, atr, cfg: _detect_double(o, h, l, c, atr, cfg, False),
+    "bull_flag":     lambda o, h, l, c, atr, cfg: _detect_flag(o, h, l, c, atr, cfg, True),
+    "bear_flag":     lambda o, h, l, c, atr, cfg: _detect_flag(o, h, l, c, atr, cfg, False),
+}
+
+
+def detect_chart_pattern(df: pd.DataFrame, atr: float, cfg) -> Optional[Dict]:
+    """Run enabled detectors over the last cp_lookback(+1) bars.
+
+    `df` is the visible history at decision time (last row = signal
+    bar). Returns the first match in PATTERN_DETECTORS priority order,
+    or None.
+    """
+    if atr <= 0 or len(df) < cfg.cp_lookback + 1:
+        return None
+    tail = df.iloc[-(cfg.cp_lookback + 1):]
+    o = tail["Open"].to_numpy(dtype=np.float64)
+    h = tail["High"].to_numpy(dtype=np.float64)
+    l = tail["Low"].to_numpy(dtype=np.float64)
+    c = tail["Close"].to_numpy(dtype=np.float64)
+    enabled = set(cfg.cp_patterns)
+    for name, det in PATTERN_DETECTORS.items():
+        if name not in enabled:
+            continue
+        hit = det(o, h, l, c, atr, cfg)
+        if hit is not None:
+            return hit
+    return None

config.py

@@ -1,8 +1,6 @@
 """
-Forex Signal Intelligence Platform - Configuration
-===================================================
-Central configuration for all signal analysis parameters.
-Tuned by market regime and validated against historical performance.
+ForexBot2 — Configuration (XAU/USD focused)
+==========================================
 """
 
 from dataclasses import dataclass, field
@@ -37,160 +35,320 @@ class MarketSession(Enum):
     OVERLAP_LONDON_NY = "london_ny_overlap"
 
 
-# ── Major Forex Pairs ──────────────────────────────────────────────
-MAJOR_PAIRS = [
-    "EURUSD=X", "GBPUSD=X", "USDJPY=X", "USDCHF=X",
-    "AUDUSD=X", "USDCAD=X", "NZDUSD=X",
-]
+# ── Supported pairs ──────────────────────────────────────────────
+XAUUSD = "XAU/USD"
 
-CROSS_PAIRS = [
-    "EURGBP=X", "EURJPY=X", "GBPJPY=X", "AUDJPY=X",
-    "EURCHF=X", "CADJPY=X",
-]
-
-# Yahoo Finance symbol mapping for display
-SYMBOL_DISPLAY = {
-    "EURUSD=X": "EUR/USD", "GBPUSD=X": "GBP/USD",
-    "USDJPY=X": "USD/JPY", "USDCHF=X": "USD/CHF",
-    "AUDUSD=X": "AUD/USD", "USDCAD=X": "USD/CAD",
-    "NZDUSD=X": "NZD/USD", "EURGBP=X": "EUR/GBP",
-    "EURJPY=X": "EUR/JPY", "GBPJPY=X": "GBP/JPY",
-    "AUDJPY=X": "AUD/JPY", "EURCHF=X": "EUR/CHF",
-    "CADJPY=X": "CAD/JPY",
+# price_precision: decimals for SL/TP rounding and display.
+# default_spread: round-trip cost per trade in price units, charged by
+# the backtester when --spread is not given.
+PAIR_INFO = {
+    XAUUSD: {"display": "XAU/USD", "pip_value": 0.10, "is_jpy_pair": False,
+             "price_precision": 2, "default_spread": 0.30},
 }
 
 
 @dataclass
 class TechnicalConfig:
-    """Technical indicator parameters - carefully tuned for Forex."""
-    # EMA periods (fast/medium/slow for trend detection)
+    """Technical indicator parameters."""
     ema_fast: int = 9
     ema_medium: int = 21
     ema_slow: int = 50
     ema_trend: int = 200
 
-    # RSI
     rsi_period: int = 14
     rsi_overbought: float = 70.0
     rsi_oversold: float = 30.0
     rsi_neutral_upper: float = 60.0
     rsi_neutral_lower: float = 40.0
 
-    # MACD (12, 26, 9 is standard)
     macd_fast: int = 12
     macd_slow: int = 26
     macd_signal: int = 9
 
-    # Bollinger Bands
     bb_period: int = 20
     bb_std: float = 2.0
 
-    # ATR for volatility measurement
     atr_period: int = 14
 
-    # Stochastic
-    stoch_k: int = 14
-    stoch_d: int = 3
-    stoch_overbought: float = 80.0
-    stoch_oversold: float = 20.0
-
-    # ADX for trend strength
     adx_period: int = 14
     adx_strong_trend: float = 25.0
     adx_very_strong: float = 50.0
 
-    # Ichimoku
-    ichimoku_tenkan: int = 9
-    ichimoku_kijun: int = 26
-    ichimoku_senkou_b: int = 52
+    divergence_lookback: int = 40
+    swing_sr_lookback: int = 50
 
-    # Volume profile
-    volume_ma_period: int = 20
+    disabled_indicators: List[str] = field(default_factory=list)
 
-    # Pivot points lookback
-    pivot_lookback: int = 1
+    ema_mode: str = "4ema"          # "4ema" (9/21/50/200) or "2ema" (50/200)
+    rsi_mode: str = "regime_aware"  # "classic" or "regime_aware"
+    macd_as_pillar: bool = True     # MACD histogram is a scored signal
 
-    # Divergence detection — how many bars back to scan for swing points
-    divergence_lookback: int = 14
+    # ROC (Rate of Change)
+    roc_period: int = 10
+    roc_threshold: float = 1.0     # % threshold for strong signal
+
+    # Keltner Channels
+    kc_period: int = 20
+    kc_atr_period: int = 10
+    kc_multiplier: float = 1.5
+
+    # ADX slope (trend acceleration detection)
+    adx_slope_lookback: int = 5
+    adx_slope_emerging_threshold: float = 3.0
+
+    regime_weights_override: Dict[str, Dict[str, float]] = field(default_factory=dict)
 
 
 @dataclass
 class SignalConfig:
-    """Signal scoring and filtering parameters."""
-    # Minimum confidence to emit a signal (0-100)
+    """Signal scoring & filtering."""
     min_confidence: float = 55.0
 
-    # Weight allocation for composite score
     weights: Dict[str, float] = field(default_factory=lambda: {
-        "trend": 0.25,          # EMA alignment, ADX
-        "momentum": 0.20,       # RSI, Stochastic, MACD
-        "volatility": 0.15,     # BB position, ATR regime
-        "price_action": 0.15,   # Candlestick patterns, S/R
-        "multi_timeframe": 0.15,# Higher TF confirmation
-        "sentiment": 0.10,      # News/calendar events
+        "ema_trend": 1.0,
+        "rsi": 1.5,
+        "bollinger": 1.5,
+        "macd": 1.5,
+        "keltner": 1.5,
+        "roc": 1.5,
+        "candlestick": 1.0,
+        "divergence": 1.5,
     })
 
-    # Multi-timeframe confirmation requirements
-    mtf_agreement_threshold: int = 2  # out of 3 timeframes must agree
-
-    # Signal cooldown (prevent over-signalling same direction)
+    mtf_agreement_threshold: int = 2  # of 3 timeframes
     cooldown_bars: int = 5
 
-    # Risk management defaults
-    default_risk_pct: float = 1.0  # risk 1% per trade
-    default_rr_ratio: float = 2.0  # 1:2 risk-reward minimum
+    default_risk_pct: float = 1.0
+    default_rr_ratio: float = 2.0
+    atr_sl_multiplier: float = 1.5
+    min_rr_ratio: float = 0.5
+    # TP tightening (May 2026) was tuned on the pre-spread backtester.
+    # Re-tested 2026-07-08 with $0.30 spread + gap fills: 0.8 nearly
+    # halves total R (+36.1R vs +66.8R, worse in both train/val halves)
+    # — smaller wins make the fixed spread a larger share of every TP.
+    tp_tightening_factor: float = 1.0
+
+    # ── Trade geometry ────────────────────────────────────────────
+    # Round-trip transaction cost estimate in price units (XAU ~$0.30).
+    round_trip_cost: float = 0.30
+    # Spread gate: reject setups whose TP1 distance is below
+    # round_trip_cost × this multiple (caps cost at ~3% of target).
+    # Validated 2026-07: small-target trades lose in both data halves
+    # on M5 while TP≥$10 trades earn ~+0.08R/trade.
+    min_tp_cost_multiple: float = 30.0
+    # Fixed SL floor in pips (80 pips = $8 on XAU). 0 disables — SL then
+    # floors at ATR × v2_sl_atr_multiplier only.
+    min_sl_pips: float = 80.0
+    # TP1 floor as a fraction of SL distance (0.5 = historical RR 0.5).
+    tp_floor_rr: float = 0.5
+
+    # ── Trailing-stop exit (2026-07) ──────────────────────────────
+    # Once favorable excursion reaches trail_arm_r × risk, the stop
+    # trails trail_dist_r × risk behind the best price (updated at bar
+    # close). TP1 becomes advisory — the trail is the exit. Validated
+    # on 12mo M5: +105R vs +48R with the fixed RR-0.5 target, balanced
+    # train/val, positive every quarter, PF 2.27. The RR-0.5 target was
+    # the binding constraint — winners kept running after TP.
+    # Set trail_arm_r = 0 to disable and exit at TP1 as before.
+    trail_arm_r: float = 0.6
+    trail_dist_r: float = 0.75
+    # Pullback (XAU_PB) trades run tighter stops (~$8 vs ~$25), so the
+    # global 0.6R arm fires on noise. Arm later; whole 0.8-arm row of
+    # the grid is +35-38R vs +32R at global params (balanced halves).
+    pb_trail_arm_r: float = 0.8
+    pb_trail_dist_r: float = 0.75
 
 
 @dataclass
 class NewsConfig:
-    """Economic calendar and sentiment configuration."""
-    # High-impact event currencies
-    tracked_currencies: List[str] = field(default_factory=lambda: [
-        "USD", "EUR", "GBP", "JPY", "CHF", "AUD", "CAD", "NZD"
-    ])
-
-    # Event impact levels to monitor
-    min_impact: str = "medium"  # "low", "medium", "high"
-
-    # Pre-event buffer (minutes before high-impact event = no new signals)
+    """Economic calendar & sentiment."""
+    tracked_currencies: List[str] = field(default_factory=lambda: ["USD", "XAU"])
     pre_event_buffer_minutes: int = 30
-
-    # Post-event volatility window (minutes after event = cautious signals)
     post_event_buffer_minutes: int = 15
+    # Whether event risk gates the SIGNAL (danger: confidence hard-capped
+    # at 30; caution: 15-point penalty). Warnings are always emitted
+    # regardless. Default False = XAU-tuned: replaying the live gates
+    # against the validated 12mo XAU M5 book (547 trades, +139.9R)
+    # showed they suppress 25-35 trades worth +4.6..+7.2R/yr — trades
+    # entered near USD releases OUTPERFORM (+0.475R vs +0.256R avg).
+    # Audit 2026-07-11 via backtest/research/calendar_store.
+    event_risk_caps_signals: bool = False
+    # Wider stay-out window after a high-impact release that surprised
+    # vs forecast (inline releases use post_event_buffer_minutes).
+    post_event_surprise_buffer_minutes: int = 45
+    # How long fetched calendar/headline data is reused across scans.
+    # Must stay below pre_event_buffer_minutes so an event can't cross
+    # into the danger window while a stale cache says it's clear.
+    context_cache_ttl_seconds: int = 600
 
-    # News sentiment sources
     rss_feeds: List[str] = field(default_factory=lambda: [
-        "https://www.forexlive.com/feed/",
+        "https://investinglive.com/feed/",
         "https://www.fxstreet.com/rss",
+        # Google News query targeted at gold drivers — also carries Kitco,
+        # Reuters etc. whose direct feeds are unavailable.
+        "https://news.google.com/rss/search?q=gold+price+OR+fed+OR+inflation&hl=en-US&gl=US&ceid=US:en",
+        # CNBC "Economy" section
+        "https://search.cnbc.com/rs/search/combinedcms/view.xml?partnerId=wrss01&id=20910258",
     ])
 
 
+@dataclass
+class XauStrategyConfig:
+    """XAU/USD strategy config.
+    Legacy gates (H4 RSI, H1 MACD, M15 score) removed in favor of
+    new momentum indicators (ROC, Keltner squeeze, ADX slope).
+    """
+    enabled: bool = True
+    rsi_entry_threshold: float = 45.0
+    tp_atr_multiplier: float = 1.0
+    sl_atr_multiplier: float = 2.0
+    min_ema200_tfs: int = 2
+    allow_generic_trades: bool = True
+    blocked_hours_utc: List[int] = field(default_factory=lambda: [0, 1, 2, 10, 11, 13, 16, 21])
+    weekend_filter: bool = True
+    min_atr_threshold: float = 1.0
+    v2_sl_atr_multiplier: float = 2.0
+    buy_threshold_boost: int = 0
+    m5_confirmation_bars: int = 0
+    h4_rsi_gate: bool = False
+    h1_macd_gate: bool = False
+    rsi_zone: tuple = (40.0, 65.0)
+    allowed_weekdays: List[int] = field(default_factory=lambda: [0, 1, 2, 3, 4, 6])  # Mon-Fri + Sun (Sydney open)
+    sell_min_quality: List[str] = field(default_factory=lambda: ["A+", "A", "B"])
+    # Volatility regimes in which SELL setups are allowed (low/normal-vol
+    # shorts lose consistently against gold's macro uptrend).
+    sell_vol_regimes: List[str] = field(default_factory=lambda: ["high", "extreme"])
+    # Market regimes in which no XAU setup may fire (any direction).
+    # Empty by default; per-timeframe factories opt in — e.g. the mixed
+    # "trending" regime (strong ADX, EMAs disagree) on M15.
+    blocked_regimes: List[str] = field(default_factory=list)
+    # Market regimes in which XAU SELL setups may not fire. Empty by
+    # default; per-timeframe factories opt in.
+    sell_blocked_regimes: List[str] = field(default_factory=list)
+    # Bar-CLOSE (decision-time) UTC hours in which XAU SELL setups may
+    # not fire. Empty by default; per-timeframe factories opt in.
+    # BUYs are unaffected (unlike blocked_hours_utc, which is both-way).
+    sell_blocked_hours_utc: List[int] = field(default_factory=list)
+
+    # ── M5-specific filters ──────────────────────────────────────
+    # M5 signals are noisy; require stronger confirmation before
+    # generating trade setups. Without these, M5 is breakeven
+    # (+0.01R/trade) with 21% drawdown.
+    m5_require_mtf: bool = True
+    m5_min_abs_score: float = 30.0
+    m5_min_confirmation_score: float = 10.0
+    # A-grade M5 SELLs enter at the extreme of temporary dips and
+    # lose (-0.017R/trade on 163 trades) — maximum conviction at
+    # the worst entry. Cap SELL quality so A-grade can't trigger
+    # the ATR-based setup generator; B-grade swing setups survive
+    # because they catch moderate drops, not extremes.
+    m5_sell_max_quality: List[str] = field(default_factory=lambda: ["B", "C"])
+    # 'high' vol regime on M5 is chop transition — elevated noise
+    # without directional expansion. 61% WR / negative total across
+    # 12mo while low/normal/extreme are all positive (validated 2026-07).
+    m5_blocked_vol_regimes: List[str] = field(default_factory=lambda: ["high"])
+    # Block M5 SELLs when RSI is already < this floor: the dip is
+    # extended and mean-reverts before the target (negative in both
+    # halves of 6mo and 12mo backtests).
+    m5_sell_min_rsi: float = 25.0
+
+    # ── M5 pullback entry (XAU_PB) ──
+    # Second entry archetype: enter on the retrace to EMA21/KC-mid in
+    # an established trend instead of at Keltner-breakout extension.
+    # Validated 2026-07: +38.3R/254 trades on 12mo, positive both
+    # halves and all four full quarters, core trades undisturbed.
+    m5_pullback_enabled: bool = True
+    # Composite score is weaker mid-pullback BY DESIGN (price sits
+    # below the momentum bands) — separate, lower floor than Gate 1.
+    m5_pullback_min_score: float = 15.0
+    m5_pullback_adx_min: float = 20.0
+    # RSI must be in the pullback zone, NOT the extended zone the
+    # breakout entries fire in (those run 65-80).
+    m5_pullback_rsi_buy: tuple = (35.0, 62.0)
+    m5_pullback_rsi_sell: tuple = (38.0, 65.0)
+    # Retrace depth from the pre-pullback swing extreme, in ATRs.
+    # Shallower = still extended; deeper = trend likely broken.
+    # M5 gold trend legs measure 3-4 ATR swing-to-trough (median 3.9),
+    # so the ceiling sits well above that of higher TFs.
+    m5_pullback_retrace_atr: tuple = (0.8, 6.0)
+    # Price must have touched EMA21/KC-mid within this many bars.
+    m5_pullback_touch_bars: int = 3
+
+    # ── Chart-pattern entry (XAU_CP) ──────────────────────────────
+    # Third entry archetype: completion of a classical chart pattern
+    # (see chart_patterns.py for the pattern set and design contract).
+    # Defaults are textbook geometry, NOT tuned values. Disabled by
+    # default — per-TF adoption happens in pair_signals/xauusd.py only
+    # after the pattern book passes the standard validation rule
+    # (positive R both halves of ≥12mo, PF ≥ 1.3, $0.30 spread,
+    # trailing exits, next-open fills).
+    cp_enabled: bool = False
+    cp_patterns: List[str] = field(default_factory=lambda: [
+        "inv_hs", "hs_top", "double_bottom", "double_top",
+        "bull_flag", "bear_flag",
+    ])
+    cp_lookback: int = 72          # bars scanned for pattern structure
+    cp_swing_window: int = 3       # bars each side to confirm a swing
+    cp_recency_bars: int = 20      # last extreme must be this fresh
+    cp_min_sep_bars: int = 8       # min bars between double extremes
+    cp_tolerance_atr: float = 0.5  # double top/bottom level match tol
+    cp_min_height_atr: float = 1.5   # pattern height floor (noise gate)
+    cp_max_height_atr: float = 8.0   # pattern height ceiling
+    cp_hs_head_min_atr: float = 0.5  # head must exceed shoulders by this
+    cp_hs_shoulder_tol_atr: float = 1.0  # shoulder level match tol
+    cp_flag_pole_atr: float = 3.0    # min pole size
+    cp_flag_pole_bars: int = 12      # pole measured over ≤ this many bars
+    cp_flag_min_bars: int = 3        # flag consolidation length bounds
+    cp_flag_max_bars: int = 12
+    cp_flag_retrace: tuple = (0.1, 0.5)  # flag depth as fraction of pole
+
+
+@dataclass
+class FilterConfig:
+    """Regime and session filters for live signals."""
+    regime_filter: bool = False
+    regime_filter_timeframe: Timeframe = Timeframe.H4
+    regime_filter_min_adx: float = 25.0
+
+    session_filter: bool = True
+    session_start_hour: int = 3
+    session_end_hour: int = 23
+    blocked_hours_utc: List[int] = field(default_factory=lambda: [0, 1, 2, 10, 11, 13, 16, 21])
+
+    allowed_weekdays: List[int] = field(default_factory=lambda: [0, 1, 2, 3, 4, 6])  # Mon-Fri + Sun (Sydney open)
+
+
 @dataclass
 class PlatformConfig:
-    """Overall platform configuration."""
+    """Top-level configuration."""
     technical: TechnicalConfig = field(default_factory=TechnicalConfig)
     signals: SignalConfig = field(default_factory=SignalConfig)
     news: NewsConfig = field(default_factory=NewsConfig)
+    backtest: FilterConfig = field(default_factory=FilterConfig)
+    xau_strategy: XauStrategyConfig = field(default_factory=XauStrategyConfig)
 
-    # Default pairs to monitor
-    watch_pairs: List[str] = field(default_factory=lambda: MAJOR_PAIRS)
+    symbol: str = XAUUSD
+    display_name: str = "XAU/USD"
+    pip_value: float = 0.10
 
-    # Analysis timeframes (primary + confirmation)
     primary_timeframe: Timeframe = Timeframe.H1
     confirmation_timeframes: List[Timeframe] = field(
         default_factory=lambda: [Timeframe.M15, Timeframe.H4]
     )
 
-    # Data source
-    data_lookback_days: int = 60  # enough for 200 EMA on H1
+    data_lookback_days: int = 60
 
-    # Twelve Data API key (overridden by TWELVE_DATA_API_KEY env var)
     twelve_data_api_key: str = "f563b22ffdfd4924955decbe1a8c8701"
 
-    # Dashboard refresh interval (seconds)
-    refresh_interval: int = 300  # 5 minutes
+    refresh_interval: int = 300
+
+    scan_pairs: List[str] = field(
+        default_factory=lambda: [XAUUSD]
+    )
+    scan_timeframes: List[Timeframe] = field(
+        default_factory=lambda: [Timeframe.M5, Timeframe.M15, Timeframe.H1, Timeframe.H4]
+    )
+    candle_buffer_seconds: int = 10
+    ws_cooldown_seconds: int = 300
 
-    # Logging
     log_level: str = "INFO"
-    log_signals_to_file: bool = True
-    signal_log_path: str = "signals.log"

data_provider.py

@@ -1,41 +1,46 @@
 """
-Forex Signal Intelligence Platform - Data Provider
-====================================================
-Retrieves and validates market data from Twelve Data API.
-Falls back to synthetic GBM data if the API is unavailable.
+ForexBot2 — Data Provider (Twelve Data SDK)
+==========================================================
+Fetches and validates OHLCV via the official `twelvedata` Python SDK.
 """
 
-import hashlib
 import logging
 import os
+import threading
 import time
 from collections import deque
-from datetime import datetime, timedelta, timezone
-from typing import Dict, List, Optional
+from typing import Dict, Optional
 
-import numpy as np
 import pandas as pd
-import requests
+from twelvedata import TDClient
 
-from config import PlatformConfig, Timeframe, SYMBOL_DISPLAY
+from config import PlatformConfig, Timeframe, XAUUSD, PAIR_INFO
 
 logger = logging.getLogger(__name__)
 
 
+def _is_forex_weekend(now=None) -> bool:
+    """Forex market closed: Friday 22:00 UTC → Sunday 22:00 UTC."""
+    if now is None:
+        now = pd.Timestamp.now(tz="UTC")
+    wd = now.weekday()
+    hour = now.hour
+    if wd == 4 and hour >= 22:
+        return True
+    if wd == 5:
+        return True
+    if wd == 6 and hour < 22:
+        return True
+    return False
+
+
 class DataIntegrityError(Exception):
     """Raised when data fails validation checks."""
     pass
 
 
 class ForexDataProvider:
-    """
-    Forex data provider backed by Twelve Data API.
-
-    Twelve Data has native 4h candles, eliminating the H1→H4 resampling
-    hack needed for Yahoo Finance. Free tier: 800 credits/day, 8 req/min.
-    """
-
-    TD_BASE_URL = "https://api.twelvedata.com"
+    """Twelve Data SDK client + on-disk parquet cache."""
 
     TD_INTERVALS = {
         Timeframe.M1:  "1min",
@@ -48,20 +53,19 @@ class ForexDataProvider:
         Timeframe.W1:  "1week",
     }
 
-    # Approximate bars per day for each timeframe (used to size outputsize)
     BARS_PER_DAY = {
-        Timeframe.M1:  1440,
-        Timeframe.M5:  288,
-        Timeframe.M15: 96,
-        Timeframe.M30: 48,
-        Timeframe.H1:  24,
-        Timeframe.H4:  6,
-        Timeframe.D1:  1,
-        Timeframe.W1:  0.143,
+        Timeframe.M1: 1440, Timeframe.M5: 288, Timeframe.M15: 96,
+        Timeframe.M30: 48, Timeframe.H1: 24, Timeframe.H4: 6,
+        Timeframe.D1: 1, Timeframe.W1: 0.143,
     }
 
-    # Free plan: 8 API credits per minute (each symbol = 1 credit)
-    RATE_LIMIT = 8
+    CANDLE_SECONDS = {
+        Timeframe.M1: 60, Timeframe.M5: 300, Timeframe.M15: 900,
+        Timeframe.M30: 1800, Timeframe.H1: 3600, Timeframe.H4: 14400,
+        Timeframe.D1: 86400, Timeframe.W1: 604800,
+    }
+
+    RATE_LIMIT = 55
     RATE_WINDOW = 60.0
 
     def __init__(self, config: PlatformConfig):
@@ -72,94 +76,138 @@ class ForexDataProvider:
         )
         self._cache: Dict[str, pd.DataFrame] = {}
         self._cache_timestamps: Dict[str, float] = {}
-        self._request_times: deque = deque()  # timestamps of recent API credits used
-        self._session = requests.Session()
-        self._session.headers.update({
-            "User-Agent": "Mozilla/5.0 (compatible; ForexIntel/1.0)"
-        })
+        # The bot shares one provider across a thread pool; cache eviction
+        # and the throttle deque are check-then-act and need locking.
+        self._cache_lock = threading.Lock()
+        self._throttle_lock = threading.Lock()
+        self._request_times: deque = deque()
+        self._td: Optional[TDClient] = (
+            TDClient(apikey=self._api_key) if self._api_key else None
+        )
 
     @staticmethod
     def _to_td_symbol(symbol: str) -> str:
-        """Convert internal symbol (EURUSD=X) to Twelve Data format (EUR/USD)."""
+        if "/" in symbol:
+            return symbol
         clean = symbol.replace("=X", "")
         return f"{clean[:3]}/{clean[3:]}"
 
+    # ── Public API ─────────────────────────────────────────────────
+
     def fetch_ohlcv(
         self,
-        symbol: str,
-        timeframe: Timeframe,
+        symbol: str = XAUUSD,
+        timeframe: Timeframe = Timeframe.H1,
         lookback_days: Optional[int] = None,
     ) -> pd.DataFrame:
-        """
-        Fetch OHLCV data for a Forex pair with full validation.
-
-        Returns a DataFrame with columns: Open, High, Low, Close, Volume
-        Index is a timezone-aware DatetimeIndex in UTC.
-        """
+        """Fetch OHLCV with in-memory caching for live mode."""
         if lookback_days is None:
             lookback_days = self.config.data_lookback_days
 
         cache_key = f"{symbol}_{timeframe.value}"
         now = time.time()
 
-        if (
-            cache_key in self._cache_timestamps
-            and (now - self._cache_timestamps[cache_key]) < self.config.refresh_interval
-            and cache_key in self._cache
-        ):
+        cache_ttl = self.CANDLE_SECONDS.get(timeframe, self.config.refresh_interval)
+        with self._cache_lock:
             cached = self._cache.get(cache_key)
-            if cached is not None and not cached.empty:
+            cached_at = self._cache_timestamps.get(cache_key)
+        if (
+            cached is not None
+            and cached_at is not None
+            and (now - cached_at) < cache_ttl
+            and not cached.empty
+        ):
+            if (self._is_data_fresh(cached, symbol, timeframe)
+                    and self._has_latest_closed_bar(cached, timeframe)):
                 return cached
+            logger.warning(f"{symbol} {timeframe.value}: cached data is stale, refetching")
+            with self._cache_lock:
+                self._cache.pop(cache_key, None)
+                self._cache_timestamps.pop(cache_key, None)
 
-        df = self._fetch_twelve_data(symbol, timeframe, lookback_days)
+        outputsize = max(
+            300,
+            min(int(lookback_days * self.BARS_PER_DAY[timeframe]) + 20, 5000),
+        )
+        df = self._fetch_twelve_data(symbol, timeframe, outputsize)
+        df = self._drop_forming_candle(df, timeframe)
         df = self._validate_and_clean(df, symbol, timeframe)
 
-        self._cache[cache_key] = df
-        self._cache_timestamps[cache_key] = now
+        if not self._is_data_fresh(df, symbol, timeframe):
+            logger.warning(
+                f"STALE DATA from API: {symbol} {timeframe.value} — "
+                f"last bar {df.index[-1]} is too old. Signal quality may be degraded."
+            )
+
+        with self._cache_lock:
+            self._cache[cache_key] = df
+            self._cache_timestamps[cache_key] = now
         return df
 
-    def prefetch_batch(
-        self,
-        symbols: List[str],
-        timeframes: List[Timeframe],
-        lookback_days: Optional[int] = None,
-    ) -> None:
+    def _has_latest_closed_bar(self, df: pd.DataFrame, timeframe: Timeframe) -> bool:
+        """Cached data must include the most recent fully closed bar.
+
+        The TTL alone is wall-clock based: a snapshot fetched mid-candle is
+        still within TTL at the next candle boundary but is missing the bar
+        that just closed, so every boundary scan would analyze — and run the
+        bar-close session check against — the previous bar.
         """
-        Batch-fetch all symbols for each timeframe in a single API call per TF.
+        now = pd.Timestamp.now(tz="UTC")
+        if _is_forex_weekend(now):
+            return True
+        candle_seconds = self.CANDLE_SECONDS.get(timeframe, 3600)
+        dur = pd.Timedelta(seconds=candle_seconds)
+        expected_label = now.floor(dur) - dur
+        return df.index[-1] >= expected_label
 
-        Twelve Data accepts comma-separated symbols, so 7 pairs × 3 timeframes
-        = 3 requests instead of 21 — well within the free-tier 8 req/min limit.
-        Results are stored in the local cache; subsequent fetch_ohlcv calls
-        for the same symbol/TF will be served from cache.
+    def _drop_forming_candle(self, df: pd.DataFrame, timeframe: Timeframe) -> pd.DataFrame:
+        """Drop the still-forming last candle so analysis only sees closed bars.
+
+        Twelve Data's time_series includes the current partial candle; the
+        scanner fires seconds after the boundary, so without this guard every
+        indicator reads a seconds-old stub bar and signals repaint.
         """
-        if lookback_days is None:
-            lookback_days = self.config.data_lookback_days
+        if df is None or df.empty:
+            return df
+        candle_seconds = self.CANDLE_SECONDS.get(timeframe, 3600)
+        now = pd.Timestamp.now(tz="UTC")
+        if df.index[-1] + pd.Timedelta(seconds=candle_seconds) > now:
+            df = df.iloc[:-1]
+        return df
 
-        for tf in timeframes:
-            try:
-                self._batch_fetch_timeframe(symbols, tf, lookback_days)
-                time.sleep(1.0)  # brief pause between TF requests
-            except Exception as e:
-                logger.warning(f"Batch prefetch failed for {tf.value}: {e}")
+    def _is_data_fresh(self, df: pd.DataFrame, symbol: str, timeframe: Timeframe) -> bool:
+        """Check if the last candle is within acceptable age for this timeframe."""
+        if df is None or df.empty:
+            return False
 
-    def fetch_multiple(
-        self,
-        symbols: List[str],
-        timeframe: Timeframe,
-        lookback_days: Optional[int] = None,
-    ) -> Dict[str, pd.DataFrame]:
-        """Fetch data for multiple symbols, returning dict of DataFrames."""
-        results = {}
-        for symbol in symbols:
-            try:
-                df = self.fetch_ohlcv(symbol, timeframe, lookback_days)
-                results[symbol] = df
-            except Exception as e:
-                logger.warning(f"Failed to fetch {symbol}: {e}")
-        return results
+        now = pd.Timestamp.now(tz="UTC")
 
-    def get_latest_price(self, symbol: str) -> Optional[float]:
-        """Return the most recent close price for a symbol."""
+        if _is_forex_weekend(now):
+            return True
+
+        last_bar = df.index[-1]
+        candle_seconds = self.CANDLE_SECONDS.get(timeframe, 3600)
+        max_age_seconds = candle_seconds * 3
+        age_seconds = (now - last_bar).total_seconds()
+
+        if age_seconds > max_age_seconds:
+            logger.warning(
+                f"{symbol} {timeframe.value}: last bar {last_bar.strftime('%Y-%m-%d %H:%M')} "
+                f"is {age_seconds / 60:.0f}m old (limit: {max_age_seconds / 60:.0f}m)"
+            )
+            return False
+
+        return True
+
+    def clear_cache(self) -> int:
+        """Wipe all cached data. Returns number of entries cleared."""
+        count = len(self._cache)
+        self._cache.clear()
+        self._cache_timestamps.clear()
+        logger.info(f"Cache cleared: {count} entries wiped")
+        return count
+
+    def get_latest_price(self, symbol: str = XAUUSD) -> Optional[float]:
         try:
             df = self.fetch_ohlcv(symbol, Timeframe.M5, lookback_days=1)
             if df is not None and not df.empty:
@@ -168,326 +216,107 @@ class ForexDataProvider:
             pass
         return None
 
-    def get_spread_estimate(self, symbol: str) -> float:
-        """Estimate typical spread from recent M5 high-low ranges."""
+    def get_spread_estimate(self, symbol: str = XAUUSD) -> float:
         try:
             df = self.fetch_ohlcv(symbol, Timeframe.M5, lookback_days=1)
             if df is not None and not df.empty:
                 return float((df["High"] - df["Low"]).median())
         except Exception:
             pass
+        info = PAIR_INFO.get(symbol, PAIR_INFO[XAUUSD])
+        return info["pip_value"]
 
-        fallback_spreads = {
-            "EURUSD=X": 0.00010, "GBPUSD=X": 0.00015,
-            "USDJPY=X": 0.010,   "USDCHF=X": 0.00015,
-            "AUDUSD=X": 0.00015, "USDCAD=X": 0.00018,
-            "NZDUSD=X": 0.00020,
-        }
-        return fallback_spreads.get(symbol, 0.00020)
+    def get_pair_info(self, symbol: str = XAUUSD) -> Dict:
+        info = PAIR_INFO.get(symbol, PAIR_INFO[XAUUSD])
+        return {"symbol": symbol, **info}
 
-    def get_pair_info(self, symbol: str) -> Dict:
-        """Return display name and pip value for a pair."""
-        display = SYMBOL_DISPLAY.get(symbol, symbol)
-        is_jpy = "JPY" in symbol
-        return {
-            "symbol": symbol,
-            "display": display,
-            "pip_value": 0.01 if is_jpy else 0.0001,
-            "is_jpy_pair": is_jpy,
-        }
-
-    # ── Private Methods ────────────────────────────────────────────
+    # ── Private ────────────────────────────────────────────────────
 
     def _throttle(self, credits: int = 1) -> None:
-        """
-        Block until `credits` API credits are available.
-
-        Twelve Data free tier: 8 credits/minute. Each symbol in any request
-        (single or batch) consumes 1 credit. This method enforces the limit
-        regardless of whether requests are batched or individual.
-        """
+        """Pro tier: 55 credits / minute. Block until budget allows."""
         while True:
-            now = time.time()
-            # Evict credits older than the rate window
-            while self._request_times and now - self._request_times[0] >= self.RATE_WINDOW:
-                self._request_times.popleft()
+            with self._throttle_lock:
+                now = time.time()
+                while self._request_times and now - self._request_times[0] >= self.RATE_WINDOW:
+                    self._request_times.popleft()
 
-            available = self.RATE_LIMIT - len(self._request_times)
-            if available >= credits:
-                for _ in range(credits):
-                    self._request_times.append(time.time())
-                return
+                available = self.RATE_LIMIT - len(self._request_times)
+                if available >= credits:
+                    for _ in range(credits):
+                        self._request_times.append(time.time())
+                    return
 
-            # Wait until the oldest credit expires
-            wait = (self._request_times[0] + self.RATE_WINDOW) - now + 0.2
+                used = len(self._request_times)
+                wait = (self._request_times[0] + self.RATE_WINDOW) - now + 0.2
             logger.info(
-                f"Rate limit: {len(self._request_times)}/{self.RATE_LIMIT} credits used "
-                f"— waiting {wait:.1f}s for next window..."
+                f"Rate limit: {used}/{self.RATE_LIMIT} credits used "
+                f"— waiting {wait:.1f}s..."
             )
             time.sleep(wait)
 
-    def _batch_fetch_timeframe(
-        self, symbols: List[str], timeframe: Timeframe, lookback_days: int
-    ) -> None:
-        """
-        Single API call fetching all symbols for one timeframe.
-        Twelve Data returns a dict keyed by symbol when multiple are requested.
-        Parsed DataFrames are validated and stored in the cache.
-        """
-        if not self._api_key:
-            return
-
-        td_symbols = [self._to_td_symbol(s) for s in symbols]
-        interval = self.TD_INTERVALS[timeframe]
-        outputsize = min(
-            int(lookback_days * self.BARS_PER_DAY[timeframe]) + 20, 5000
-        )
-        outputsize = max(outputsize, 300)
-
-        # Each symbol in a batch request costs 1 credit
-        self._throttle(credits=len(symbols))
-
-        resp = self._session.get(
-            f"{self.TD_BASE_URL}/time_series",
-            params={
-                "symbol": ",".join(td_symbols),
-                "interval": interval,
-                "outputsize": outputsize,
-                "apikey": self._api_key,
-                "timezone": "UTC",
-                "format": "json",
-            },
-            timeout=30,
-        )
-        resp.raise_for_status()
-        data = resp.json()
-
-        # If the whole batch request returned a top-level error, raise it
-        if isinstance(data, dict) and data.get("status") == "error":
-            raise DataIntegrityError(
-                f"Batch API error ({timeframe.value}): {data.get('message', 'unknown')}"
-            )
-
-        # Batch response: {"EUR/USD": {...}, "GBP/USD": {...}, ...}
-        now = time.time()
-        for symbol, td_sym in zip(symbols, td_symbols):
-            try:
-                sym_data = data.get(td_sym, {})
-                if sym_data.get("status") == "error":
-                    logger.warning(
-                        f"Batch fetch error for {td_sym}: {sym_data.get('message')}"
-                    )
-                    continue
-
-                values = sym_data.get("values")
-                if not values:
-                    logger.warning(f"No values in batch response for {td_sym}")
-                    continue
-
-                records = [
-                    {
-                        "Datetime": pd.Timestamp(v["datetime"], tz="UTC"),
-                        "Open":   float(v["open"]),
-                        "High":   float(v["high"]),
-                        "Low":    float(v["low"]),
-                        "Close":  float(v["close"]),
-                        "Volume": float(v.get("volume") or 0),
-                    }
-                    for v in reversed(values)
-                ]
-                df = pd.DataFrame(records).set_index("Datetime")
-                df.index.name = "Datetime"
-                df = self._validate_and_clean(df, symbol, timeframe)
-
-                cache_key = f"{symbol}_{timeframe.value}"
-                self._cache[cache_key] = df
-                self._cache_timestamps[cache_key] = now
-                logger.debug(f"Batch cached {len(df)} bars for {td_sym} @ {interval}")
-
-            except Exception as e:
-                logger.warning(f"Failed to parse batch data for {td_sym}: {e}")
-
     def _fetch_twelve_data(
-        self, symbol: str, timeframe: Timeframe, lookback_days: int
+        self,
+        symbol: str,
+        timeframe: Timeframe,
+        outputsize: int,
+        start_date: Optional[str] = None,
+        end_date: Optional[str] = None,
     ) -> pd.DataFrame:
-        """
-        Fetch OHLCV from Twelve Data REST API.
-        Falls back to synthetic GBM data if API is unavailable or key is missing.
-        """
-        if not self._api_key:
-            logger.warning("No Twelve Data API key — using synthetic data. "
-                           "Set TWELVE_DATA_API_KEY env var or config.twelve_data_api_key.")
-            return self._generate_synthetic_data(symbol, timeframe, lookback_days)
+        if self._td is None:
+            raise DataIntegrityError(
+                "No Twelve Data API key. Set TWELVE_DATA_API_KEY env var "
+                "or config.twelve_data_api_key."
+            )
 
         td_symbol = self._to_td_symbol(symbol)
         interval = self.TD_INTERVALS[timeframe]
-        outputsize = min(
-            int(lookback_days * self.BARS_PER_DAY[timeframe]) + 20,
-            5000,
-        )
-        outputsize = max(outputsize, 300)
 
         self._throttle(credits=1)
 
+        kwargs = {
+            "symbol": td_symbol,
+            "interval": interval,
+            "outputsize": min(outputsize, 5000),
+            "timezone": "UTC",
+        }
+        if start_date:
+            kwargs["start_date"] = start_date
+        if end_date:
+            kwargs["end_date"] = end_date
+
         try:
-            resp = self._session.get(
-                f"{self.TD_BASE_URL}/time_series",
-                params={
-                    "symbol": td_symbol,
-                    "interval": interval,
-                    "outputsize": outputsize,
-                    "apikey": self._api_key,
-                    "timezone": "UTC",
-                    "format": "json",
-                },
-                timeout=15,
-            )
-            resp.raise_for_status()
-            data = resp.json()
-
-            if data.get("status") == "error":
-                raise DataIntegrityError(
-                    f"Twelve Data API error for {td_symbol}: {data.get('message', 'unknown')}"
-                )
-
-            values = data.get("values")
-            if not values:
-                raise DataIntegrityError(f"Empty response from Twelve Data for {td_symbol}")
-
-            # Values arrive newest-first; reverse to chronological order
-            records = [
-                {
-                    "Datetime": pd.Timestamp(v["datetime"], tz="UTC"),
-                    "Open":   float(v["open"]),
-                    "High":   float(v["high"]),
-                    "Low":    float(v["low"]),
-                    "Close":  float(v["close"]),
-                    "Volume": float(v.get("volume") or 0),
-                }
-                for v in reversed(values)
-            ]
-
-            df = pd.DataFrame(records).set_index("Datetime")
-            df.index.name = "Datetime"
-            logger.debug(f"Fetched {len(df)} bars for {td_symbol} @ {interval}")
-            return df
-
-        except DataIntegrityError:
-            raise
+            df = self._td.time_series(**kwargs).as_pandas()
         except Exception as e:
-            logger.info(f"Twelve Data fetch failed for {symbol} ({e}), using synthetic data.")
-            return self._generate_synthetic_data(symbol, timeframe, lookback_days)
+            raise DataIntegrityError(
+                f"Twelve Data fetch failed for {td_symbol}: {e}"
+            ) from e
 
-    def _generate_synthetic_data(
-        self, symbol: str, timeframe: Timeframe, lookback_days: int
-    ) -> pd.DataFrame:
-        """
-        Generate realistic synthetic Forex OHLCV using GBM with mean-reversion.
-        Used as a fallback when live data is unavailable.
-        """
-        pair_params = {
-            "EURUSD=X": {"price": 1.1350, "vol": 0.07, "drift": -0.001},
-            "GBPUSD=X": {"price": 1.3200, "vol": 0.08, "drift":  0.002},
-            "USDJPY=X": {"price": 144.50, "vol": 0.09, "drift":  0.003},
-            "USDCHF=X": {"price": 0.8850, "vol": 0.07, "drift": -0.001},
-            "AUDUSD=X": {"price": 0.6550, "vol": 0.10, "drift": -0.002},
-            "USDCAD=X": {"price": 1.3650, "vol": 0.07, "drift":  0.001},
-            "NZDUSD=X": {"price": 0.6100, "vol": 0.10, "drift": -0.001},
-            "EURGBP=X": {"price": 0.8590, "vol": 0.06, "drift":  0.000},
-            "EURJPY=X": {"price": 163.90, "vol": 0.09, "drift":  0.002},
-            "GBPJPY=X": {"price": 190.70, "vol": 0.11, "drift":  0.003},
-            "AUDJPY=X": {"price": 94.60,  "vol": 0.11, "drift":  0.001},
-            "EURCHF=X": {"price": 0.9380, "vol": 0.05, "drift":  0.000},
-            "CADJPY=X": {"price": 105.80, "vol": 0.09, "drift":  0.001},
-        }
+        if df is None or df.empty:
+            raise DataIntegrityError(f"Empty response from Twelve Data for {td_symbol}")
 
-        params = pair_params.get(symbol, {"price": 1.0000, "vol": 0.08, "drift": 0.000})
-        base_price = params["price"]
-        annual_vol = params["vol"]
-        annual_drift = params["drift"]
+        # SDK returns lowercase columns; engine expects capitalised.
+        df = df.rename(columns={
+            "open": "Open", "high": "High", "low": "Low",
+            "close": "Close", "volume": "Volume",
+        })
+        for col in ("Open", "High", "Low", "Close", "Volume"):
+            if col in df.columns:
+                df[col] = pd.to_numeric(df[col], errors="coerce")
+        if "Volume" not in df.columns:
+            df["Volume"] = 0.0
 
-        tf_minutes = {
-            Timeframe.M1: 1, Timeframe.M5: 5, Timeframe.M15: 15,
-            Timeframe.M30: 30, Timeframe.H1: 60, Timeframe.H4: 240,
-            Timeframe.D1: 1440, Timeframe.W1: 10080,
-        }
-        bar_minutes = tf_minutes.get(timeframe, 60)
-        bars_per_year = (365.25 * 24 * 60) / bar_minutes
-        dt = 1.0 / bars_per_year
-
-        n_bars = min(int(lookback_days * 24 * 60 / bar_minutes), 2000)
-        n_bars = max(n_bars, 300)
-
-        seed_base = int(hashlib.md5(symbol.encode()).hexdigest()[:8], 16)
-        time_component = int(time.time()) // 3600
-        np.random.seed((seed_base + time_component) % (2**31))
-
-        bar_vol = annual_vol * np.sqrt(dt)
-        bar_drift = annual_drift * dt
-
-        regime_len = np.random.randint(20, 80)
-        trend_strength = np.random.normal(0, 0.3)
-
-        price = base_price
-        prices = [price]
-
-        for i in range(1, n_bars):
-            if i % regime_len == 0:
-                regime_len = np.random.randint(20, 80)
-                trend_strength = np.random.normal(0, 0.3)
-
-            mr_force = -0.001 * (price - base_price) / base_price
-            ret = bar_drift + trend_strength * dt + mr_force + bar_vol * np.random.randn()
-
-            if np.random.random() < 0.02:
-                ret *= np.random.uniform(2.0, 4.0)
-
-            price *= (1 + ret)
-            prices.append(price)
-
-        prices = np.array(prices)
-        opens = np.roll(prices, 1)
-        opens[0] = prices[0]
-
-        typical_range = bar_vol * base_price * 1.5
-        highs = np.maximum(prices, opens) + np.abs(np.random.normal(0, typical_range * 0.6, n_bars))
-        lows  = np.minimum(prices, opens) - np.abs(np.random.normal(0, typical_range * 0.6, n_bars))
-        highs = np.maximum(highs, np.maximum(opens, prices))
-        lows  = np.minimum(lows,  np.minimum(opens, prices))
-
-        volume = np.random.lognormal(mean=10, sigma=0.5, size=n_bars).astype(int)
-
-        freq_map = {
-            Timeframe.M1: "1min",  Timeframe.M5: "5min",
-            Timeframe.M15: "15min", Timeframe.M30: "30min",
-            Timeframe.H1: "1h",    Timeframe.H4: "4h",
-            Timeframe.D1: "D",     Timeframe.W1: "W",
-        }
-        freq = freq_map.get(timeframe, "1h")
-        end_time = pd.Timestamp.now(tz="UTC").floor(f"{bar_minutes}min")
-        index = pd.date_range(end=end_time, periods=n_bars, freq=freq, tz="UTC")
-
-        df = pd.DataFrame({
-            "Open": opens, "High": highs, "Low": lows,
-            "Close": prices, "Volume": volume,
-        }, index=index)
+        # SDK returns newest-first; engine + indicators want oldest-first.
+        df = df.sort_index()
+        if df.index.tz is None:
+            df.index = df.index.tz_localize("UTC")
         df.index.name = "Datetime"
 
-        logger.info(
-            f"Generated {n_bars} synthetic {timeframe.value} bars for "
-            f"{SYMBOL_DISPLAY.get(symbol, symbol)} @ {prices[-1]:.5f}"
-        )
+        logger.debug(f"Fetched {len(df)} bars for {td_symbol} @ {interval}")
         return df
 
     def _validate_and_clean(
         self, df: pd.DataFrame, symbol: str, timeframe: Timeframe
     ) -> pd.DataFrame:
-        """
-        Validate and clean OHLCV data.
-
-        Steps: drop all-NaN rows → ffill small gaps → remove future timestamps
-        → fix OHLC inconsistencies → remove >10-sigma outliers → sort & deduplicate.
-        """
         if df is None or df.empty:
             raise DataIntegrityError(f"Empty dataset for {symbol}")
 
@@ -497,6 +326,9 @@ class ForexDataProvider:
         df = df.ffill(limit=3)
         df = df.dropna(subset=["Close"])
 
+        df = df.sort_index()
+        df = df[~df.index.duplicated(keep="last")]
+
         now = pd.Timestamp.utcnow()
         future_mask = df.index > now + pd.Timedelta(minutes=5)
         if future_mask.any():
@@ -520,14 +352,24 @@ class ForexDataProvider:
         returns = df["Close"].pct_change()
         std_ret = returns.std()
         if std_ret > 0:
-            outlier_mask = (returns - returns.mean()).abs() > 10 * std_ret
+            dev = returns - returns.mean()
+            outlier_mask = dev.abs() > 10 * std_ret
+            # A spike bar makes the NEXT bar's return look extreme too (it is
+            # measured against the spike). If the previous bar is flagged and
+            # this bar reverses direction, it is the legitimate snap-back —
+            # keep it, or every spike would punch a two-bar hole in the data.
+            recovery = (
+                outlier_mask
+                & outlier_mask.shift(1, fill_value=False)
+                & (dev * dev.shift(1) < 0)
+            )
+            outlier_mask = outlier_mask & ~recovery
             if outlier_mask.any():
                 logger.warning(f"{symbol}: Removing {outlier_mask.sum()} extreme outlier bars")
                 df = df[~outlier_mask]
 
         min_required = max(
             self.config.technical.ema_trend,
-            self.config.technical.ichimoku_senkou_b,
             200,
         ) + 10
 
@@ -537,9 +379,6 @@ class ForexDataProvider:
                 f"(need {min_required}). Some indicators will be partial."
             )
 
-        df = df.sort_index()
-        df = df[~df.index.duplicated(keep="last")]
-
         cleaned = original_len - len(df)
         if cleaned > 0:
             logger.info(f"{symbol}: Cleaned {cleaned}/{original_len} bars")

docs/beginner_guide.html

@@ -0,0 +1,165 @@
+<!DOCTYPE html>
+<html lang="en">
+<head>
+<meta charset="utf-8">
+<title>ForexBot2 — Beginner's Guide</title>
+<style>
+  @page {
+    size: A4;
+    margin: 22mm 18mm;
+    @bottom-center { content: "ForexBot2 Beginner's Guide — page " counter(page) " of " counter(pages); font-size: 8pt; color: #888; }
+  }
+  * { box-sizing: border-box; }
+  body { font-family: "DejaVu Sans", Helvetica, Arial, sans-serif; font-size: 10.5pt; line-height: 1.65; color: #1c2733; margin: 0; }
+  h1 { font-size: 21pt; margin: 0 0 2mm; color: #0d1b2a; }
+  .subtitle { color: #5a6b7c; font-size: 10.5pt; margin-bottom: 8mm; }
+  h2 { font-size: 13.5pt; color: #0d1b2a; border-bottom: 2px solid #d4a017; padding-bottom: 1.2mm; margin: 8mm 0 3mm; page-break-after: avoid; }
+  h3 { font-size: 11pt; margin: 5mm 0 1.5mm; color: #23374d; page-break-after: avoid; }
+  p { margin: 0 0 2.8mm; }
+  table { border-collapse: collapse; width: 100%; margin: 3mm 0 4mm; font-size: 9.3pt; page-break-inside: avoid; }
+  th { background: #0d1b2a; color: #fff; text-align: left; padding: 1.8mm 2.5mm; font-weight: 600; }
+  td { border-bottom: 1px solid #dde4ea; padding: 1.8mm 2.5mm; vertical-align: top; }
+  tr:nth-child(even) td { background: #f4f7fa; }
+  table.breakable { page-break-inside: auto; }
+  table.breakable tr { page-break-inside: avoid; }
+  .signal-box { background: #f4f7fa; border: 1px solid #c9d4de; border-left: 4px solid #d4a017; border-radius: 2mm; padding: 3.5mm 4.5mm; font-family: "DejaVu Sans Mono", monospace; font-size: 8.6pt; line-height: 1.55; white-space: pre-wrap; margin: 3mm 0 4mm; page-break-inside: avoid; }
+  .callout { background: #fdf6e3; border: 1px solid #e8d49a; border-left: 4px solid #d4a017; border-radius: 2mm; padding: 3mm 4.5mm; margin: 3mm 0 4mm; page-break-inside: avoid; }
+  .callout.red { background: #fdf0ee; border-color: #eab4ac; border-left-color: #c0392b; }
+  .callout.green { background: #eef7ee; border-color: #b3d9b3; border-left-color: #2e7d32; }
+  .callout b:first-child { display: block; margin-bottom: 1mm; }
+  .example { background: #eef2f6; border-radius: 2mm; padding: 3mm 4.5mm; margin: 2mm 0 4mm; page-break-inside: avoid; }
+  .example b:first-child { color: #23374d; }
+  ul, ol { margin: 0 0 3mm; padding-left: 6mm; }
+  li { margin-bottom: 1.6mm; }
+  code { font-family: "DejaVu Sans Mono", monospace; font-size: 8.8pt; background: #eef2f6; padding: 0.3mm 1.2mm; border-radius: 1mm; }
+  .small { font-size: 8.5pt; color: #5a6b7c; }
+  .disclaimer { border: 1.5px solid #c0392b; border-radius: 2mm; padding: 3.5mm 4.5mm; margin-top: 6mm; font-size: 9pt; page-break-inside: avoid; }
+  .disclaimer b { color: #c0392b; }
+</style>
+</head>
+<body>
+
+<h1>ForexBot2 — Beginner's Guide</h1>
+<div class="subtitle">Everything explained in plain words. No experience needed. &nbsp;·&nbsp; July 2026</div>
+
+<h2>1 · What is this bot?</h2>
+<p>ForexBot2 is a computer program that watches the <b>price of gold</b> all day and all night. It studies the price charts using nine different mathematical checks. When the checks agree that the price is likely to move in one direction, it sends you a message on <b>Telegram</b> telling you:</p>
+<ul>
+<li>whether to <b>buy</b> or <b>sell</b>,</li>
+<li>at what price to get in,</li>
+<li>at what price to get out if the trade goes wrong (so your loss stays small),</li>
+<li>and how to lock in profit if the trade goes well.</li>
+</ul>
+<p>Most days the bot stays quiet for hours. <b>That is normal.</b> Quiet means "nothing worth trading right now." Never trade just because you are bored.</p>
+
+<h2>2 · Ten words you need to know</h2>
+<p>Every trading word used in this guide (and in the bot's messages) is explained here. Come back to this table whenever you see a word you don't remember.</p>
+<table class="breakable">
+<tr><th style="width:26%">Word</th><th>Plain meaning</th></tr>
+<tr><td><b>BUY (going "long")</b></td><td>You make money if the gold price goes <b>up</b>.</td></tr>
+<tr><td><b>SELL (going "short")</b></td><td>You make money if the gold price goes <b>down</b>. Yes, that's possible — your broker lets you sell gold you don't own and buy it back cheaper later.</td></tr>
+<tr><td><b>Entry</b></td><td>The price at which you open the trade.</td></tr>
+<tr><td><b>Stop loss (SL)</b></td><td>A safety brake. You tell your broker: "if the price reaches this level, close my trade automatically." It puts a hard limit on how much one trade can lose. <b>Every trade must have one.</b></td></tr>
+<tr><td><b>Take profit (TP)</b></td><td>The opposite of a stop loss: a level where the trade closes automatically <b>with a profit</b>. In this system it is only a rough guide — see section 6.</td></tr>
+<tr><td><b>Risk, or "R"</b></td><td>The amount of money you have chosen to lose if the trade fails. If you decide to risk $20, then <b>1R = $20</b>. A trade that wins "+2R" made you $40. This is how traders measure results.</td></tr>
+<tr><td><b>Trailing stop ("trail")</b></td><td>A stop loss that <b>follows the price</b> as it moves in your favor, but never moves backwards. Explained with numbers in section 6.</td></tr>
+<tr><td><b>Timeframe (TF)</b></td><td>The size of each candle on the chart. <b>M5</b> = each candle is 5 minutes, <b>M15</b> = 15 minutes, <b>H1</b> = 1 hour, <b>H4</b> = 4 hours. Bigger timeframe = slower, calmer trading.</td></tr>
+<tr><td><b>XAU/USD</b></td><td>Just the official symbol for "gold priced in US dollars."</td></tr>
+<tr><td><b>Spread</b></td><td>The tiny difference between the buying price and the selling price. It's your broker's fee, paid on every trade.</td></tr>
+</table>
+
+<h2>3 · Reading a bot message</h2>
+<p>Here is a typical signal, with plain-English notes below it:</p>
+<div class="signal-box">🔍 XAU/USD Signal Analysis  (H1)          ← gold, on the 1-hour chart
+🟢 BUY   Grade: A  |  64% confidence      ← what to do + how good the setup looks
+✅ MTF aligned                            ← bigger and smaller charts agree
+📋 Indicators …                           ← the bot's 9 checks (you can skip this part)
+━━━━━━━━━━━━━━━━━━━━
+🚀 TRADE SETUP                            ← ↓ THIS is the part you act on ↓
+📈 Direction: BUY
+▶ Entry:  4,132.50                        ← open the trade near this price
+🛑 SL:     4,088.20                        ← your safety brake (stop loss)
+🎯 TP1:    4,176.80                        ← rough profit target (a guide, not a rule)
+🔄 Trail:  once price hits 4,159.08,
+   trail SL $33.23 behind best price      ← how to lock in profit (see section 6)</div>
+<ul>
+<li><b>Grade (A+, A, B, C):</b> the bot's own mark for how clean the setup is. A+ is best. It never sends anything worse than C.</li>
+<li><b>Confidence and the indicator list:</b> extra detail for curious people. You don't need to understand it to trade the signal.</li>
+<li><b>⚠️ Warning lines:</b> things the bot noticed, like high volatility. They are "be aware," not "don't trade."</li>
+<li><b>The trade setup at the bottom is the only part you must act on.</b></li>
+</ul>
+
+<h2>4 · How much money to put in (the most important section)</h2>
+<p>You never put your whole account into one trade. You decide how much you are willing to <b>lose</b>, and work backwards from there. The rule:</p>
+<div class="callout"><b>Risk 1% of your account per trade. Never more.</b> If you have $2,000, that means one trade may lose at most $20. Losing trades are a normal part of the system — 1% means a bad week stings but never hurts you badly.</div>
+<div class="example"><b>Worked example.</b><br>
+Your account: <b>$2,000</b>. Your risk: 1% = <b>$20</b>. So for this trade, 1R = $20.<br>
+The signal says: Entry 4,132.50, Stop loss 4,088.20. The gap between them is <b>$44.30 per ounce</b> of gold.<br>
+How much gold to buy: $20 ÷ $44.30 = <b>0.45 ounces</b>.<br>
+Now, whatever happens: if the price falls to the stop loss, you lose about $20 — and no more. If the trade wins 2R, you make about $40.</div>
+<p class="small">Most broker apps have a built-in calculator: enter the stop-loss price and the amount you want to risk ($20), and it works out the position size for you. Use it — it avoids mistakes. (On many platforms gold is traded in "lots," where 0.01 lot = 1 ounce.)</p>
+
+<h2>5 · What to do when a signal arrives — step by step</h2>
+<ol>
+<li><b>Check the timeframe</b> in the first line. Only act on the one timeframe you have chosen to follow (see section 7). Ignore the rest.</li>
+<li><b>Work out your position size</b> using the 1% rule above. Do this before anything else.</li>
+<li><b>Open the trade</b> in your broker app, in the direction shown (BUY or SELL), at the current market price. Try to act soon after the message; if the price has already run far past the entry price, skip the trade — another signal will come.</li>
+<li><b>Set the stop loss immediately</b>, at the exact SL price in the message. Not "in your head" — in the app. Gold can move fast, and the stop loss is what protects you while you sleep.</li>
+<li><b>Then manage the trade with the trailing stop</b>, explained next.</li>
+</ol>
+
+<h2>6 · The trailing stop — how winners get paid</h2>
+<p>The take-profit price (TP1) in the message is only a guide. The system makes its money by <b>letting winning trades run</b> and following them with a moving safety brake. Every signal message tells you exactly when to start and how far behind to follow. Here is the idea with numbers:</p>
+<div class="example"><b>Example.</b> You bought gold at <b>4,132</b>, stop loss at <b>4,088</b> (risking $44 per ounce).<br>
+The message says: <i>"once price hits 4,159, trail SL $33 behind best price."</i><br><br>
+<b>Step 1 — wait.</b> Until gold reaches 4,159, do nothing. If it falls to 4,088 first, the stop loss closes the trade. That loss is normal and already budgeted.<br>
+<b>Step 2 — the price reaches 4,159.</b> Now move your stop loss up to 4,159 − 33 = <b>4,126</b>. From here on, the worst case is roughly break-even.<br>
+<b>Step 3 — keep following.</b> Each time the price makes a new high, move the stop up so it stays $33 below that high. Price climbs to 4,180? Stop goes to 4,147 — profit is now locked in. Price climbs to 4,210? Stop goes to 4,177.<br>
+<b>Step 4 — the price turns.</b> Eventually gold falls back $33 from its best point and hits your stop. The trade closes by itself, and you keep everything above your entry. <b>You never decide when to take profit — the trail decides.</b></div>
+<ul>
+<li>Only move the stop <b>up</b> (for a BUY). Never move it back down. For a SELL, everything is mirrored: the stop only moves down.</li>
+<li>Move it when a candle finishes, not every second. Checking once per candle (every hour, on the 1-hour timeframe) is enough.</li>
+<li>Many broker apps can do this automatically — look for a "trailing stop" order and set the distance from the message.</li>
+</ul>
+
+<h2>7 · Which timeframe should you follow?</h2>
+<p>The bot sends signals from several chart speeds. As a beginner, <b>follow only the 1-hour (H1) signals</b> and ignore the others. Here is why:</p>
+<table>
+<tr><th></th><th>M5 (5-minute)</th><th>M15 (15-minute)</th><th>H1 (1-hour) ★ start here</th></tr>
+<tr><td>Signals per week</td><td>about 10</td><td>about 10</td><td>about 3</td></tr>
+<tr><td>A trade usually lasts</td><td>a few hours</td><td>about half a day</td><td>half a day to a few days</td></tr>
+<tr><td>Time you have to react</td><td>1–2 minutes</td><td>up to 15 minutes</td><td>up to an hour</td></tr>
+<tr><td>Suits</td><td>full-time / automated</td><td>experienced, quick</td><td>anyone with a job and a life</td></tr>
+</table>
+<ul>
+<li><b>H1 is forgiving.</b> You have up to an hour to act on a message, and small delays barely matter. On the 5-minute signals, being two minutes late already costs real money — they are not for manual beginners.</li>
+<li><b>H1 trades are held longer</b> — often overnight, sometimes over days. That is normal. Your stop loss is working for you while you sleep.</li>
+<li><b>Never take the same signal on two timeframes.</b> They are usually the same gold move seen at different zoom levels. Taking both doubles your risk, not your profit.</li>
+</ul>
+
+<h2>8 · What to expect — read this before your first trade</h2>
+<ul>
+<li><b>You will lose about 4 trades out of every 10.</b> That is how the system is designed. Losses are small (1R each) and wins are bigger on average. The profit comes from many trades added together, not from any single one.</li>
+<li><b>Losing streaks happen.</b> Five or six losses in a row will happen at some point every year. It does not mean the system is broken. This is exactly why each loss is capped at 1% of your account.</li>
+<li><b>Expect a rough patch.</b> At some point in a normal year, your account will likely be around 10% below its best point before recovering. If a 10% dip would make you panic or bet bigger to "win it back," trade smaller (0.5% risk) or don't trade.</li>
+<li><b>Judge the system over months, not days.</b> On the 1-hour timeframe there are only ~3 trades a week. One bad week means nothing; look at results per quarter.</li>
+<li><b>Real results will be a little worse than tests.</b> The bot's numbers come from careful historical testing, but tests can't include your reaction speed or your broker's exact prices. Expect a bit less, not a bit more.</li>
+</ul>
+
+<h2>9 · The five rules</h2>
+<div class="callout red">
+<b>Break these and no signal system can save you.</b>
+<ol style="margin-bottom:0">
+<li><b>Risk 1% per trade, always.</b> Never increase the size to win back a loss.</li>
+<li><b>Every trade gets a stop loss in the app, at entry time.</b> No exceptions, no "mental" stops.</li>
+<li><b>One timeframe, one open trade at a time.</b></li>
+<li><b>Missed the entry? Let it go.</b> Chasing a price that already moved turns a good trade into a gamble.</li>
+<li><b>Only trade money you can afford to lose completely.</b> Never rent, savings you need, or borrowed money.</li>
+</ol>
+</div>
+<div class="callout green"><b>A good first month:</b> follow the H1 signals with the smallest position size your broker allows (or a demo account), write down every trade, and practice the trailing stop until it feels routine. Only then trade at 1% risk.</div>
+
+<div class="disclaimer"><b>⚠️ Risk warning.</b> This bot and guide are for education and research only. They are <b>not financial advice</b>. Trading gold and currencies involves a high risk of losing money, and most beginners lose. Past results — including tested results — do not guarantee future profits. Never trade money you cannot afford to lose. If you are unsure, speak to a licensed financial adviser.</div>
+
+</body>
+</html>

docs/user_guide.html

@@ -0,0 +1,140 @@
+<!DOCTYPE html>
+<html lang="en">
+<head>
+<meta charset="utf-8">
+<title>ForexBot2 — User Guide</title>
+<style>
+  @page {
+    size: A4;
+    margin: 22mm 18mm;
+    @bottom-center { content: "ForexBot2 User Guide — page " counter(page) " of " counter(pages); font-size: 8pt; color: #888; }
+  }
+  * { box-sizing: border-box; }
+  body { font-family: "DejaVu Sans", Helvetica, Arial, sans-serif; font-size: 10pt; line-height: 1.55; color: #1c2733; margin: 0; }
+  h1 { font-size: 21pt; margin: 0 0 2mm; color: #0d1b2a; }
+  .subtitle { color: #5a6b7c; font-size: 10.5pt; margin-bottom: 8mm; }
+  h2 { font-size: 13.5pt; color: #0d1b2a; border-bottom: 2px solid #d4a017; padding-bottom: 1.2mm; margin: 8mm 0 3mm; page-break-after: avoid; }
+  h3 { font-size: 11pt; margin: 5mm 0 1.5mm; color: #23374d; page-break-after: avoid; }
+  p { margin: 0 0 2.5mm; }
+  table { border-collapse: collapse; width: 100%; margin: 3mm 0 4mm; font-size: 9pt; page-break-inside: avoid; }
+  th { background: #0d1b2a; color: #fff; text-align: left; padding: 1.8mm 2.5mm; font-weight: 600; }
+  td { border-bottom: 1px solid #dde4ea; padding: 1.6mm 2.5mm; }
+  tr:nth-child(even) td { background: #f4f7fa; }
+  .signal-box { background: #f4f7fa; border: 1px solid #c9d4de; border-left: 4px solid #d4a017; border-radius: 2mm; padding: 3.5mm 4.5mm; font-family: "DejaVu Sans Mono", monospace; font-size: 8.4pt; line-height: 1.5; white-space: pre-wrap; margin: 3mm 0 4mm; page-break-inside: avoid; }
+  .callout { background: #fdf6e3; border: 1px solid #e8d49a; border-left: 4px solid #d4a017; border-radius: 2mm; padding: 3mm 4.5mm; margin: 3mm 0 4mm; page-break-inside: avoid; }
+  .callout.red { background: #fdf0ee; border-color: #eab4ac; border-left-color: #c0392b; }
+  .callout b:first-child { display: block; margin-bottom: 1mm; }
+  ul, ol { margin: 0 0 3mm; padding-left: 6mm; }
+  li { margin-bottom: 1.4mm; }
+  code { font-family: "DejaVu Sans Mono", monospace; font-size: 8.6pt; background: #eef2f6; padding: 0.3mm 1.2mm; border-radius: 1mm; }
+  .small { font-size: 8.5pt; color: #5a6b7c; }
+  .disclaimer { border: 1.5px solid #c0392b; border-radius: 2mm; padding: 3.5mm 4.5mm; margin-top: 6mm; font-size: 9pt; page-break-inside: avoid; }
+  .disclaimer b { color: #c0392b; }
+</style>
+</head>
+<body>
+
+<h1>ForexBot2 — User Guide</h1>
+<div class="subtitle">How to read, act on, and survive the signals &nbsp;·&nbsp; July 2026</div>
+
+<h2>1 · What the bot does</h2>
+<p>ForexBot2 analyzes <b>gold (XAU/USD)</b> around the clock on several chart timeframes at once. On every candle close it scores nine technical indicators (trend, momentum, volatility, divergences, candlestick patterns), adapts the weighting to the current market regime (trending / ranging / transitioning), checks that higher timeframes agree, adjusts for news and economic-calendar events, and — only when everything lines up — sends a <b>signal with a complete trade plan</b> to Telegram: direction, entry, stop loss, target, and a trailing-stop management plan.</p>
+<p>Most of the time it sends nothing. Silence means no edge was found; that is the bot working correctly, not a malfunction.</p>
+
+<h2>2 · Anatomy of a signal</h2>
+<div class="signal-box">🔍 XAU/USD Signal Analysis  (M15)
+🕐 2026-07-10 14:00 UTC
+💰 Close: 4,132.50
+📊 Regime: Trending Up | Vol: normal
+━━━━━━━━━━━━━━━━━━━━
+🟢 BUY   🅰 A  |  Score: +38.2  |  64% conf
+✅ MTF aligned
+📈 MTF: M5 ✅  H1 ✅
+📋 Indicators … (9 rows)
+⚠️ EXTREME volatility — wider stops needed
+━━━━━━━━━━━━━━━━━━━━
+🚀 TRADE SETUP
+📈 Direction: BUY
+▶ Entry:  4,132.50
+🛑 SL:     4,103.10   ($29.40 / 294 pips)
+🎯 TP1:    4,161.90   ($29.40 / 294 pips)
+⚖️ RR:     1:1.0
+🔄 Trail: once price hits 4,150.14 (+0.6R),
+   trail SL $22.05 behind best price —
+   TP1 is a guide, let it run</div>
+
+<table>
+<tr><th style="width:28%">Field</th><th>What it means</th></tr>
+<tr><td><b>Signal + grade</b></td><td>Direction (BUY / SELL / STRONG_*) plus a quality grade <b>A+ → F</b>. A trade setup is only attached at grade C or better. Higher grades = more indicator agreement and cleaner regime.</td></tr>
+<tr><td><b>Score / confidence</b></td><td>The regime-weighted composite of all nine indicators. Bigger absolute score = stronger consensus.</td></tr>
+<tr><td><b>MTF aligned</b></td><td>The two confirmation timeframes agree with the trade direction. Signals without alignment are filtered out on M5.</td></tr>
+<tr><td><b>Warnings ⚠️</b></td><td>Context, not veto. Backtesting shows warned trades perform about the same — strong momentum often triggers warnings.</td></tr>
+<tr><td><b>SL / TP1 / RR</b></td><td>Stop loss sits behind the nearest swing level with an ATR buffer. TP1 is the nearest opposing level. <b>1R</b> = the entry-to-SL distance = the amount you risk.</td></tr>
+<tr><td><b>Trail plan 🔄</b></td><td>The real exit. Once price moves +0.6R in your favor (+0.8R for pullback setups), move the stop to trail 0.75R behind the best price and update it at each candle close. <b>TP1 is advisory — let winners run.</b></td></tr>
+</table>
+
+<h2>3 · Acting on a signal — step by step</h2>
+<ol>
+<li><b>Size the position first.</b> Decide your risk per trade — <b>0.5–1% of account, never more</b>. Position size = (account × risk%) ÷ stop distance in $. The stop distance varies per signal, so the size changes every trade; the dollar risk must not.</li>
+<li><b>Enter promptly at market.</b> All backtested results assume entry at the open of the candle after the signal. On M5, minutes matter (see §4). If you are very late — price already moved a large part of the way to the arm level — skip the trade. There is always another signal.</li>
+<li><b>Place the stop loss immediately</b> at the given SL. No mental stops — gold gaps and spikes.</li>
+<li><b>Manage with the trail, not TP1.</b> When price reaches the arm level printed in the message, start trailing the stop the stated $ distance behind the best price, tightening only at candle close. This is what produced the backtested results; taking fixed TP1 profits gives most of the edge away.</li>
+<li><b>One signal, one trade.</b> The M5, M15 and H1 books confirm off each other and are usually long or short gold at the same time. Taking the same move on multiple timeframes multiplies risk, not diversification. Pick your timeframe (§4) and ignore the others.</li>
+</ol>
+
+<h2>4 · Choosing your timeframe</h2>
+<p>Twelve-month backtest (Jul 2025 – Jul 2026, real spreads, entry at next candle open):</p>
+<table>
+<tr><th></th><th>M5</th><th>M15</th><th>H1</th></tr>
+<tr><td>Trades / week</td><td>~10</td><td>~9.5</td><td>~3</td></tr>
+<tr><td>Total result (12 mo)</td><td>+135.6R</td><td>+67.2R</td><td>+49.9R</td></tr>
+<tr><td>Win rate / profit factor</td><td>59.7% / 1.81</td><td>55.8% / 1.40</td><td>59.4% / 1.94</td></tr>
+<tr><td>Typical (median) hold</td><td>2.4 h</td><td>6 h</td><td>12.5 h</td></tr>
+<tr><td>90% of trades close within</td><td>16 h</td><td>40 h</td><td>3.7 days</td></tr>
+<tr><td>Typical stop distance</td><td>~$19</td><td>~$29</td><td>~$44</td></tr>
+<tr><td>Worst drawdown</td><td>11.2R</td><td>9.4R</td><td>9.5R</td></tr>
+</table>
+<ul>
+<li><b>M5 — intraday, execution-critical.</b> Highest total edge, but stops are tight: a couple of dollars of slippage or a late entry eats a meaningful share of every trade. Suitable only if you can act within a minute or two of the alert (or automate). Trades open and close within a session; you sleep flat.</li>
+<li><b>M15 — the middle path.</b> Same frequency as M5, wider stops, up to 15 minutes to act. About half your trades will still be open overnight.</li>
+<li><b>H1 — swing trading around a normal life.</b> Best quality per trade and very forgiving of slow manual execution (you have up to an hour). The costs: positions held for days, through news and overnight sessions, and a <b>lumpy</b> equity curve — one quarter of the test year was negative and most of the profit came in a single quarter. Judge it on quarters, not weeks.</li>
+</ul>
+<div class="callout"><b>If you trade manually from the Telegram alerts, H1 (or M15) is the honest choice.</b> M5's backtested edge assumes near-instant fills that a human copying signals rarely achieves.</div>
+
+<h2>5 · What to expect (and what not to)</h2>
+<ul>
+<li><b>You will lose ~40% of trades.</b> A 56–60% win rate means losing streaks of 5+ are routine over a year. The system's edge is the average across hundreds of trades, not any single one.</li>
+<li><b>Stops hit fast, winners take time.</b> Losing trades die in roughly half the time winners take. An early stop-out is the system doing its job.</li>
+<li><b>Drawdowns of ~10R are normal.</b> At 1% risk per trade, expect to be ~10% down from a peak at some point in a normal year. This is why risking more than 1% per trade is how accounts die.</li>
+<li><b>Backtest ≠ live.</b> Results above charge real spreads and realistic fills, but cannot model your personal latency, broker slippage, or requotes. Assume live results will be somewhat worse — especially on M5.</li>
+<li><b>Quiet periods are normal.</b> Filters block bad hours, bad regimes and misaligned timeframes. Days can pass without a signal.</li>
+</ul>
+
+<h2>6 · Rules that keep you alive</h2>
+<div class="callout red">
+<b>The five rules</b>
+<ol style="margin-bottom:0">
+<li>Risk a fixed 0.5–1% per trade. Never "make it back" by sizing up after losses.</li>
+<li>Hard stop loss in the market on every trade, placed at entry time.</li>
+<li>One timeframe, one trade at a time. No stacking, no pyramiding onto bot signals.</li>
+<li>Missed the entry? Skip it. Chasing converts a positive-edge trade into a coin flip.</li>
+<li>Evaluate performance monthly at the earliest, quarterly for H1. Never after a bad day.</li>
+</ol>
+</div>
+
+<h2>7 · Bot commands (owner)</h2>
+<table>
+<tr><th style="width:30%">Command</th><th>What it does</th></tr>
+<tr><td><code>/start</code></td><td>Show the button menu (manual scans per timeframe, Scan All TFs).</td></tr>
+<tr><td><code>/scan</code></td><td>Run an immediate analysis of the primary timeframe.</td></tr>
+<tr><td><code>/autoscan</code></td><td>Toggle automatic scanning (M5 / M15 / H1 / H4 at each candle close).</td></tr>
+<tr><td><code>/status</code></td><td>Uptime, auto-scan state, WebSocket price-monitor state.</td></tr>
+<tr><td><code>/cache</code> · <code>/clearcache</code></td><td>Inspect / reset the market-data cache.</td></tr>
+<tr><td><code>/help</code></td><td>Command reference.</td></tr>
+</table>
+<p class="small">The bot also streams live prices over WebSocket and fires an immediate scan when price breaches a key level (swing S/R, Bollinger, Keltner, EMA-200). Commands respond only to the configured owner chat; subscribers receive the signal feed via the channel.</p>
+
+<div class="disclaimer"><b>⚠️ Risk disclaimer.</b> This bot and guide are for educational and research purposes only and are <b>not financial advice</b>. Trading forex, gold and CFDs on margin carries a high risk of loss; past (and backtested) performance does not guarantee future results. Do your own analysis and never risk money you cannot afford to lose.</div>
+
+</body>
+</html>

logs/telegram.log.2026-07-14

@@ -0,0 +1,21 @@
+01:51:29 [WARNING] data_provider: BTC/USD 1h: last bar 2026-07-11 05:51 is 4320m old (limit: 180m)
+01:52:02 [WARNING] data_provider: BTC/USD 1h: last bar 2026-07-11 05:52 is 4320m old (limit: 180m)
+02:42:59 [WARNING] data_provider: BTC/USD 1h: last bar 2026-07-11 06:42 is 4320m old (limit: 180m)
+03:30:24 [WARNING] data_provider: BTC/USD 1h: last bar 2026-07-11 07:30 is 4320m old (limit: 180m)
+03:31:32 [WARNING] data_provider: BTC/USD 1h: last bar 2026-07-11 07:31 is 4320m old (limit: 180m)
+11:30:20 [WARNING] data_provider: BTC/USD 1h: last bar 2026-07-11 15:30 is 4320m old (limit: 180m)
+11:41:34 [WARNING] data_provider: BTC/USD 1h: last bar 2026-07-11 15:41 is 4320m old (limit: 180m)
+11:45:46 [WARNING] data_provider: BTC/USD 1h: last bar 2026-07-11 15:45 is 4320m old (limit: 180m)
+11:56:38 [WARNING] data_provider: BTC/USD 1h: last bar 2026-07-11 15:56 is 4320m old (limit: 180m)
+11:57:11 [WARNING] data_provider: BTC/USD 1h: last bar 2026-07-11 15:57 is 4320m old (limit: 180m)
+11:59:52 [WARNING] data_provider: BTC/USD 1h: last bar 2026-07-11 15:59 is 4320m old (limit: 180m)
+12:10:05 [WARNING] data_provider: BTC/USD 1h: last bar 2026-07-11 16:10 is 4320m old (limit: 180m)
+13:05:37 [WARNING] data_provider: BTC/USD 1h: last bar 2026-07-11 17:05 is 4320m old (limit: 180m)
+13:14:25 [WARNING] data_provider: BTC/USD 1h: last bar 2026-07-11 17:14 is 4320m old (limit: 180m)
+13:38:50 [WARNING] data_provider: BTC/USD 1h: last bar 2026-07-11 17:38 is 4320m old (limit: 180m)
+13:45:14 [WARNING] data_provider: BTC/USD 1h: last bar 2026-07-11 17:45 is 4320m old (limit: 180m)
+13:51:33 [WARNING] data_provider: BTC/USD 1h: last bar 2026-07-11 17:51 is 4320m old (limit: 180m)
+15:04:41 [WARNING] data_provider: BTC/USD 1h: last bar 2026-07-11 19:04 is 4320m old (limit: 180m)
+15:04:50 [WARNING] data_provider: BTC/USD 1h: last bar 2026-07-11 19:04 is 4320m old (limit: 180m)
+15:05:17 [WARNING] data_provider: BTC/USD 1h: last bar 2026-07-11 19:05 is 4320m old (limit: 180m)
+15:06:10 [WARNING] data_provider: BTC/USD 1h: last bar 2026-07-11 19:06 is 4320m old (limit: 180m)

main.py

@@ -1,509 +1,347 @@
-#!/usr/bin/env python3
-"""
-Forex Signal Intelligence Platform
-====================================
-Main entry point. Runs full analysis and outputs formatted signal reports.
-
-Usage:
-    python main.py                          # Analyze all major pairs
-    python main.py --pairs EURUSD GBPUSD    # Specific pairs
-    python main.py --timeframe H4           # Different primary TF
-    python main.py --json                   # JSON output
-    python main.py --watch                  # Continuous monitoring mode
-"""
-
-import argparse
-import json
-import logging
-import sys
-import time
-from datetime import datetime, timezone
-from typing import List
-
-# Force UTF-8 output on Windows so box-drawing / Unicode chars render correctly
-if hasattr(sys.stdout, "reconfigure"):
-    sys.stdout.reconfigure(encoding="utf-8")
-if hasattr(sys.stderr, "reconfigure"):
-    sys.stderr.reconfigure(encoding="utf-8")
-
-from config import (
-    PlatformConfig, Timeframe, SignalStrength,
-    MAJOR_PAIRS, CROSS_PAIRS,
-)
-from signal_aggregator import SignalAggregator, SignalReport
-
-
-# ══════════════════════════════════════════════════════════════════
-#  TERMINAL FORMATTING
-# ══════════════════════════════════════════════════════════════════
-
-class Colors:
-    """ANSI color codes for terminal output."""
-    RESET = "\033[0m"
-    BOLD = "\033[1m"
-    DIM = "\033[2m"
-    UNDERLINE = "\033[4m"
-
-    RED = "\033[91m"
-    GREEN = "\033[92m"
-    YELLOW = "\033[93m"
-    BLUE = "\033[94m"
-    MAGENTA = "\033[95m"
-    CYAN = "\033[96m"
-    WHITE = "\033[97m"
-    GRAY = "\033[90m"
-
-    BG_RED = "\033[41m"
-    BG_GREEN = "\033[42m"
-    BG_YELLOW = "\033[43m"
-    BG_BLUE = "\033[44m"
-    BG_MAGENTA = "\033[45m"
-    BG_CYAN = "\033[46m"
-
-
-def signal_color(signal: SignalStrength) -> str:
-    """Get color for a signal strength."""
-    return {
-        SignalStrength.STRONG_BUY: Colors.GREEN + Colors.BOLD,
-        SignalStrength.BUY: Colors.GREEN,
-        SignalStrength.NEUTRAL: Colors.GRAY,
-        SignalStrength.SELL: Colors.RED,
-        SignalStrength.STRONG_SELL: Colors.RED + Colors.BOLD,
-    }.get(signal, Colors.RESET)
-
-
-def quality_color(quality: str) -> str:
-    """Get color for signal quality grade."""
-    return {
-        "A+": Colors.GREEN + Colors.BOLD,
-        "A": Colors.GREEN,
-        "B": Colors.CYAN,
-        "C": Colors.YELLOW,
-        "D": Colors.GRAY,
-        "F": Colors.DIM,
-    }.get(quality, Colors.RESET)
-
-
-def score_bar(score: float, width: int = 30) -> str:
-    """Create a visual score bar from -100 to +100."""
-    normalized = (score + 100) / 200  # 0 to 1
-    filled = int(normalized * width)
-    center = width // 2
-
-    bar = ""
-    for i in range(width):
-        if i == center:
-            bar += "│"
-        elif (score >= 0 and center < i <= filled) or \
-             (score < 0 and filled <= i < center):
-            bar += "█"
-        else:
-            bar += "░"
-
-    if score >= 0:
-        color = Colors.GREEN if score > 20 else Colors.GRAY
-    else:
-        color = Colors.RED if score < -20 else Colors.GRAY
-
-    return f"{color}{bar}{Colors.RESET}"
-
-
-def print_header():
-    """Print the platform header."""
-    now = datetime.now(timezone.utc).strftime("%Y-%m-%d %H:%M:%S UTC")
-    width = 80
-
-    print(f"\n{Colors.CYAN}{'═' * width}{Colors.RESET}")
-    print(f"{Colors.BOLD}{Colors.CYAN}  ╔═══════════════════════════════════════════════════════╗{Colors.RESET}")
-    print(f"{Colors.BOLD}{Colors.CYAN}  ║   FOREX SIGNAL INTELLIGENCE PLATFORM                 ║{Colors.RESET}")
-    print(f"{Colors.BOLD}{Colors.CYAN}  ║   Multi-Layer Signal Analysis Engine                  ║{Colors.RESET}")
-    print(f"{Colors.BOLD}{Colors.CYAN}  ╚═══════════════════════════════════════════════════════╝{Colors.RESET}")
-    print(f"{Colors.GRAY}  Scan Time: {now}{Colors.RESET}")
-    print(f"{Colors.CYAN}{'═' * width}{Colors.RESET}\n")
-
-
-def print_summary_table(reports: List[SignalReport]):
-    """Print a summary table of all pairs."""
-    print(f"{Colors.BOLD}{Colors.WHITE}  ┌─────────────┬───────────┬──────────────────────────────────┬────────┬────────────┐{Colors.RESET}")
-    print(f"{Colors.BOLD}{Colors.WHITE}  │ Pair        │ Signal    │ Score                            │ Grade  │ Confidence │{Colors.RESET}")
-    print(f"{Colors.BOLD}{Colors.WHITE}  ├─────────────┼───────────┼──────────────────────────────────┼────────┼────────────┤{Colors.RESET}")
-
-    for r in reports:
-        sig = r.final_signal.name.replace("_", " ")
-        sc = signal_color(r.final_signal)
-        qc = quality_color(r.signal_quality)
-        bar = score_bar(r.final_score, 30)
-
-        pair = f"{r.display_name:<11}"
-        signal_str = f"{sc}{sig:<9}{Colors.RESET}"
-        grade_str = f"{qc}{r.signal_quality:<6}{Colors.RESET}"
-        conf_str = f"{r.final_confidence:5.1f}%"
-
-        # Add warning indicator
-        warn = ""
-        if r.event_risk == "danger":
-            warn = f" {Colors.RED}⚠{Colors.RESET}"
-        elif r.event_risk == "caution":
-            warn = f" {Colors.YELLOW}△{Colors.RESET}"
-
-        print(f"  │ {pair} │ {signal_str} │ {bar} │ {grade_str} │ {conf_str:>8}{warn}  │")
-
-    print(f"{Colors.BOLD}{Colors.WHITE}  └─────────────┴───────────┴──────────────────────────────────┴────────┴────────────┘{Colors.RESET}")
-
-    # Legend
-    print(f"\n{Colors.GRAY}  Legend: "
-          f"{Colors.GREEN}██ Bullish{Colors.GRAY} | "
-          f"{Colors.RED}██ Bearish{Colors.GRAY} | "
-          f"│ Center (0) | "
-          f"{Colors.RED}⚠ Event Risk{Colors.GRAY} | "
-          f"{Colors.YELLOW}△ Caution{Colors.RESET}")
-
-
-def print_detailed_report(report: SignalReport):
-    """Print detailed analysis for a single pair."""
-    sc = signal_color(report.final_signal)
-    qc = quality_color(report.signal_quality)
-    sig_name = report.final_signal.name.replace("_", " ")
-
-    print(f"\n{Colors.CYAN}{'─' * 80}{Colors.RESET}")
-    print(f"  {Colors.BOLD}{Colors.WHITE}{report.display_name}{Colors.RESET}"
-          f"  │  Price: {Colors.BOLD}{report.current_price:.5f}{Colors.RESET}"
-          f"  │  Regime: {Colors.CYAN}{report.market_regime}{Colors.RESET}"
-          f"  │  Vol: {report.volatility_regime}")
-
-    print(f"\n  {Colors.BOLD}SIGNAL:{Colors.RESET} {sc}{sig_name}{Colors.RESET}"
-          f"  │  Score: {sc}{report.final_score:+.1f}{Colors.RESET}"
-          f"  │  Grade: {qc}{report.signal_quality}{Colors.RESET}"
-          f"  │  Confidence: {report.final_confidence:.0f}%")
-
-    print(f"  {Colors.BOLD}Action:{Colors.RESET} {report.action_recommendation}")
-
-    # Multi-timeframe breakdown
-    if report.mtf_signals:
-        print(f"\n  {Colors.BOLD}Multi-Timeframe:{Colors.RESET}", end="")
-        mtf_agree = f" {Colors.GREEN}✓ ALIGNED{Colors.RESET}" if report.mtf_agreement else f" {Colors.YELLOW}✗ DIVERGENT{Colors.RESET}"
-        print(mtf_agree)
-        for tf, sig in report.mtf_signals.items():
-            tf_sc = signal_color(sig)
-            tf_score = report.mtf_scores.get(tf, 0)
-            print(f"    {tf:>5}: {tf_sc}{sig.name.replace('_', ' '):<12}{Colors.RESET} ({tf_score:+.1f})")
-
-    # Indicators detail
-    if report.primary_analysis:
-        print(f"\n  {Colors.BOLD}Indicators ({report.primary_analysis.timeframe}):{Colors.RESET}")
-        for name, ind in report.primary_analysis.indicators.items():
-            if name.startswith("_") or ind.weight == 0:
-                continue
-            isc = signal_color(ind.signal)
-            ind_name = f"{ind.name:<18}"
-            print(f"    {ind_name} {isc}{ind.signal.name.replace('_', ' '):<12}{Colors.RESET}"
-                  f" ({ind.confidence:.0f}%) │ {Colors.GRAY}{ind.description}{Colors.RESET}")
-
-    # Bullish reasons
-    if report.bullish_reasons:
-        print(f"\n  {Colors.GREEN}▲ Bullish Factors:{Colors.RESET}")
-        for reason in report.bullish_reasons[:5]:
-            print(f"    {Colors.GREEN}+{Colors.RESET} {reason}")
-
-    # Bearish reasons
-    if report.bearish_reasons:
-        print(f"\n  {Colors.RED}▼ Bearish Factors:{Colors.RESET}")
-        for reason in report.bearish_reasons[:5]:
-            print(f"    {Colors.RED}-{Colors.RESET} {reason}")
-
-    # Warnings
-    if report.warnings:
-        print(f"\n  {Colors.YELLOW}⚠ Warnings:{Colors.RESET}")
-        for warning in report.warnings:
-            print(f"    {Colors.YELLOW}!{Colors.RESET} {warning}")
-
-    # News context
-    if report.news_description and report.news_description != "Neutral news sentiment":
-        print(f"\n  {Colors.MAGENTA}📰 News:{Colors.RESET} {report.news_description}")
-    if report.event_description and report.event_risk != "clear":
-        print(f"  {Colors.YELLOW}📅 Event:{Colors.RESET} {report.event_description}")
-
-    # Trade setup
-    if report.trade_setup:
-        ts = report.trade_setup
-        dir_color = Colors.GREEN if ts.direction == "BUY" else Colors.RED
-        print(f"\n  {Colors.BOLD}Trade Setup:{Colors.RESET}")
-        print(f"    Direction:     {dir_color}{ts.direction}{Colors.RESET}")
-        print(f"    Entry:         {ts.entry_price:.5f}")
-        print(f"    Stop Loss:     {Colors.RED}{ts.stop_loss:.5f}{Colors.RESET} ({ts.risk_pips:.1f} pips)")
-        print(f"    Take Profit 1: {Colors.GREEN}{ts.take_profit_1:.5f}{Colors.RESET} ({ts.reward_pips:.1f} pips)")
-        print(f"    Take Profit 2: {Colors.GREEN}{ts.take_profit_2:.5f}{Colors.RESET}")
-        print(f"    Risk:Reward:   1:{ts.risk_reward:.1f}")
-
-    # Key levels
-    if report.primary_analysis and report.primary_analysis.key_levels:
-        levels = report.primary_analysis.key_levels
-        print(f"\n  {Colors.BOLD}Key Levels:{Colors.RESET}")
-        level_names = ["r3", "r2", "r1", "pivot", "s1", "s2", "s3"]
-        for lvl in level_names:
-            if lvl in levels:
-                val = levels[lvl]
-                if val > report.current_price:
-                    color = Colors.RED  # resistance
-                else:
-                    color = Colors.GREEN  # support
-                label = lvl.upper()
-                print(f"    {label:>6}: {color}{val:.5f}{Colors.RESET}")
-
-
-def print_event_calendar(reports: List[SignalReport]):
-    """Print upcoming economic events affecting watched pairs."""
-    events_shown = set()
-    has_events = False
-
-    for report in reports:
-        if report.event_risk != "clear" and report.event_description not in events_shown:
-            if not has_events:
-                print(f"\n{Colors.YELLOW}{Colors.BOLD}  📅 ECONOMIC CALENDAR ALERTS{Colors.RESET}")
-                has_events = True
-            events_shown.add(report.event_description)
-            risk_color = Colors.RED if report.event_risk == "danger" else Colors.YELLOW
-            print(f"    {risk_color}● {report.event_description}{Colors.RESET} → affects {report.display_name}")
-
-    if not has_events:
-        print(f"\n{Colors.GRAY}  📅 No high-impact events in the near term{Colors.RESET}")
-
-
-def report_to_dict(report: SignalReport) -> dict:
-    """Convert a SignalReport to a JSON-serializable dict."""
-    d = {
-        "symbol": report.symbol,
-        "display_name": report.display_name,
-        "timestamp": report.timestamp,
-        "current_price": report.current_price,
-        "signal": report.final_signal.name,
-        "score": round(report.final_score, 2),
-        "confidence": round(report.final_confidence, 2),
-        "quality": report.signal_quality,
-        "action": report.action_recommendation,
-        "market_regime": report.market_regime,
-        "volatility_regime": report.volatility_regime,
-        "mtf_agreement": report.mtf_agreement,
-        "mtf_signals": {k: v.name for k, v in report.mtf_signals.items()},
-        "event_risk": report.event_risk,
-        "event_description": report.event_description,
-        "news_sentiment": round(report.news_sentiment, 3),
-        "bullish_reasons": report.bullish_reasons,
-        "bearish_reasons": report.bearish_reasons,
-        "warnings": report.warnings,
-    }
-    if report.trade_setup:
-        ts = report.trade_setup
-        d["trade_setup"] = {
-            "direction": ts.direction,
-            "entry": ts.entry_price,
-            "stop_loss": ts.stop_loss,
-            "take_profit_1": ts.take_profit_1,
-            "take_profit_2": ts.take_profit_2,
-            "risk_reward": ts.risk_reward,
-            "risk_pips": round(ts.risk_pips, 1),
-            "reward_pips": round(ts.reward_pips, 1),
-        }
-    if report.primary_analysis:
-        d["indicators"] = {}
-        for name, ind in report.primary_analysis.indicators.items():
-            if not name.startswith("_"):
-                d["indicators"][name] = {
-                    "signal": ind.signal.name,
-                    "confidence": round(ind.confidence, 1),
-                    "description": ind.description,
-                }
-        d["key_levels"] = {
-            k: round(v, 5) for k, v in report.primary_analysis.key_levels.items()
-        }
-    return d
-
-
-# ══════════════════════════════════════════════════════════════════
-#  MAIN
-# ══════════════════════════════════════════════════════════════════
-
-def parse_args():
-    parser = argparse.ArgumentParser(
-        description="Forex Signal Intelligence Platform",
-        formatter_class=argparse.RawDescriptionHelpFormatter,
-        epilog="""
-Examples:
-  python main.py                        Analyze all major pairs (H1)
-  python main.py --pairs EURUSD GBPUSD  Analyze specific pairs
-  python main.py --timeframe H4         Use 4-hour as primary TF
-  python main.py --all                  Include cross pairs
-  python main.py --json                 Output as JSON
-  python main.py --detail               Show detailed analysis per pair
-  python main.py --watch                Continuous monitoring (5min refresh)
-        """,
-    )
-    parser.add_argument(
-        "--pairs", nargs="+", default=None,
-        help="Specific pairs to analyze (e.g., EURUSD GBPUSD)",
-    )
-    parser.add_argument(
-        "--timeframe", "-tf", default="H1",
-        choices=["M5", "M15", "M30", "H1", "H4", "D1"],
-        help="Primary analysis timeframe (default: H1)",
-    )
-    parser.add_argument(
-        "--all", action="store_true",
-        help="Include cross pairs in analysis",
-    )
-    parser.add_argument(
-        "--json", action="store_true",
-        help="Output results as JSON",
-    )
-    parser.add_argument(
-        "--detail", action="store_true",
-        help="Show detailed analysis for each pair",
-    )
-    parser.add_argument(
-        "--top", type=int, default=None,
-        help="Show only top N signals by strength",
-    )
-    parser.add_argument(
-        "--watch", action="store_true",
-        help="Continuous monitoring mode (refreshes every 5 min)",
-    )
-    parser.add_argument(
-        "--min-grade", default="F",
-        choices=["A+", "A", "B", "C", "D", "F"],
-        help="Minimum signal quality to display",
-    )
-    parser.add_argument(
-        "--verbose", "-v", action="store_true",
-        help="Enable debug logging",
-    )
-    return parser.parse_args()
-
-
-def symbol_from_name(name: str) -> str:
-    """Convert pair name like 'EURUSD' to Yahoo symbol 'EURUSD=X'."""
-    name = name.upper().replace("/", "").replace("_", "")
-    if not name.endswith("=X"):
-        name += "=X"
-    return name
-
-
-def run_analysis(args) -> List[SignalReport]:
-    """Run a single analysis cycle."""
-    config = PlatformConfig()
-
-    # Configure timeframe
-    tf_map = {
-        "M5": Timeframe.M5, "M15": Timeframe.M15, "M30": Timeframe.M30,
-        "H1": Timeframe.H1, "H4": Timeframe.H4, "D1": Timeframe.D1,
-    }
-    config.primary_timeframe = tf_map[args.timeframe]
-
-    # Configure confirmation timeframes based on primary
-    tf_confirmations = {
-        "M5": [Timeframe.M1, Timeframe.M15],
-        "M15": [Timeframe.M5, Timeframe.H1],
-        "M30": [Timeframe.M15, Timeframe.H1],
-        "H1": [Timeframe.M15, Timeframe.H4],
-        "H4": [Timeframe.H1, Timeframe.D1],
-        "D1": [Timeframe.H4, Timeframe.W1],
-    }
-    config.confirmation_timeframes = tf_confirmations.get(
-        args.timeframe, [Timeframe.M15, Timeframe.H4]
-    )
-
-    # Configure pairs
-    if args.pairs:
-        config.watch_pairs = [symbol_from_name(p) for p in args.pairs]
-    elif args.all:
-        config.watch_pairs = MAJOR_PAIRS + CROSS_PAIRS
-    else:
-        config.watch_pairs = MAJOR_PAIRS
-
-    # Run analysis
-    aggregator = SignalAggregator(config)
-    reports = aggregator.analyze_all_pairs()
-
-    # Filter by grade
-    grade_order = ["A+", "A", "B", "C", "D", "F"]
-    min_idx = grade_order.index(args.min_grade) if args.min_grade in grade_order else len(grade_order) - 1
-    reports = [r for r in reports if grade_order.index(r.signal_quality) <= min_idx]
-
-    # Top N
-    if args.top:
-        reports = reports[:args.top]
-
-    return reports
-
-
-def main():
-    args = parse_args()
-
-    # Setup logging
-    log_level = logging.DEBUG if args.verbose else logging.WARNING
-    logging.basicConfig(
-        level=log_level,
-        format="%(asctime)s [%(levelname)s] %(name)s: %(message)s",
-        datefmt="%H:%M:%S",
-    )
-
-    if args.watch:
-        # Continuous monitoring mode
-        print(f"{Colors.BOLD}Starting continuous monitoring mode (Ctrl+C to stop)...{Colors.RESET}")
-        while True:
-            try:
-                reports = run_analysis(args)
-                if args.json:
-                    output = [report_to_dict(r) for r in reports]
-                    print(json.dumps(output, indent=2))
-                else:
-                    print("\033[2J\033[H")  # clear screen
-                    print_header()
-                    print_summary_table(reports)
-                    print_event_calendar(reports)
-                    if args.detail:
-                        for r in reports:
-                            print_detailed_report(r)
-                    print(f"\n{Colors.GRAY}  Next refresh in 5 minutes...{Colors.RESET}")
-
-                time.sleep(300)
-            except KeyboardInterrupt:
-                print(f"\n{Colors.YELLOW}Monitoring stopped.{Colors.RESET}")
-                break
-    else:
-        # Single run
-        print(f"{Colors.GRAY}  Fetching market data and running analysis...{Colors.RESET}\n")
-        reports = run_analysis(args)
-
-        if args.json:
-            output = [report_to_dict(r) for r in reports]
-            print(json.dumps(output, indent=2))
-        else:
-            print_header()
-            if reports:
-                print_summary_table(reports)
-                print_event_calendar(reports)
-
-                if args.detail:
-                    for r in reports:
-                        print_detailed_report(r)
-                else:
-                    # Show detail for top signals only
-                    actionable = [r for r in reports if r.signal_quality in ("A+", "A", "B")]
-                    if actionable:
-                        print(f"\n{Colors.BOLD}{Colors.WHITE}  ═══ ACTIONABLE SIGNALS (Grade B or above) ═══{Colors.RESET}")
-                        for r in actionable:
-                            print_detailed_report(r)
-                    else:
-                        print(f"\n{Colors.GRAY}  No high-grade signals detected. "
-                              f"Use --detail for full breakdown or --min-grade C to see more.{Colors.RESET}")
-            else:
-                print(f"{Colors.YELLOW}  No data available. Check your internet connection.{Colors.RESET}")
-
-            print(f"\n{Colors.CYAN}{'═' * 80}{Colors.RESET}")
-            print(f"{Colors.GRAY}  Disclaimer: This tool is for educational/research purposes only.{Colors.RESET}")
-            print(f"{Colors.GRAY}  Past performance ≠ future results. Always use proper risk management.{Colors.RESET}")
-            print(f"{Colors.CYAN}{'═' * 80}{Colors.RESET}\n")
-
-
-if __name__ == "__main__":
-    main()
+#!/usr/bin/env python3
+"""
+ForexBot2 — XAU/USD Signal Intelligence
+========================================
+Usage:
+    python main.py                              # Live signal on H1
+    python main.py --timeframe H4 --detail      # Detailed H4 signal
+    python main.py --json                       # JSON output
+"""
+
+import argparse
+import json
+import logging
+import sys
+from datetime import datetime, timezone
+from typing import Dict, List
+
+# Force UTF-8 output on Windows so box-drawing chars render
+if hasattr(sys.stdout, "reconfigure"):
+    sys.stdout.reconfigure(encoding="utf-8")
+if hasattr(sys.stderr, "reconfigure"):
+    sys.stderr.reconfigure(encoding="utf-8")
+
+from config import PlatformConfig, Timeframe, SignalStrength, XAUUSD
+from pair_signals import PAIR_PRODUCERS
+from signal_aggregator import SignalAggregator, SignalReport
+
+
+# ══════════════════════════════════════════════════════════════════
+#  TERMINAL FORMATTING
+# ══════════════════════════════════════════════════════════════════
+
+class Colors:
+    RESET = "\033[0m"
+    BOLD = "\033[1m"
+    DIM = "\033[2m"
+    UNDERLINE = "\033[4m"
+    RED = "\033[91m"
+    GREEN = "\033[92m"
+    YELLOW = "\033[93m"
+    BLUE = "\033[94m"
+    MAGENTA = "\033[95m"
+    CYAN = "\033[96m"
+    WHITE = "\033[97m"
+    GRAY = "\033[90m"
+
+
+def signal_color(signal: SignalStrength) -> str:
+    return {
+        SignalStrength.STRONG_BUY:  Colors.GREEN + Colors.BOLD,
+        SignalStrength.BUY:         Colors.GREEN,
+        SignalStrength.NEUTRAL:     Colors.GRAY,
+        SignalStrength.SELL:        Colors.RED,
+        SignalStrength.STRONG_SELL: Colors.RED + Colors.BOLD,
+    }.get(signal, Colors.RESET)
+
+
+def quality_color(q: str) -> str:
+    return {
+        "A+": Colors.GREEN + Colors.BOLD,
+        "A":  Colors.GREEN,
+        "B":  Colors.CYAN,
+        "C":  Colors.YELLOW,
+        "D":  Colors.GRAY,
+        "F":  Colors.DIM,
+    }.get(q, Colors.RESET)
+
+
+def print_header(title: str = "XAU/USD SIGNAL INTELLIGENCE"):
+    now = datetime.now(timezone.utc).strftime("%Y-%m-%d %H:%M:%S UTC")
+    print(f"\n{Colors.CYAN}{'═' * 80}{Colors.RESET}")
+    print(f"{Colors.BOLD}{Colors.CYAN}  {title}{Colors.RESET}")
+    print(f"{Colors.GRAY}  Time: {now}{Colors.RESET}")
+    print(f"{Colors.CYAN}{'═' * 80}{Colors.RESET}\n")
+
+
+# ── Live signal renderer ───────────────────────────────────────────
+
+_INDICATOR_PREFIXES = (
+    "EMA Trend", "RSI", "Bollinger Bands", "Divergence",
+    "Candlestick", "ADX", "ATR", "News", "Regime",
+)
+
+
+def _compact_reason(reason: str) -> str:
+    """Shorten a verbose reason string for the compact view."""
+    s = reason
+    if s.startswith("STRONG: "):
+        s = s[8:]
+    # Strip the leading "<Indicator>: " prefix — it's always an indicator name
+    for prefix in _INDICATOR_PREFIXES:
+        if s.startswith(prefix + ": "):
+            s = s[len(prefix) + 2:]
+            break
+    # Drop trailing detail after first ' | '
+    if " | " in s:
+        s = s.split(" | ", 1)[0]
+    return s.strip()
+
+
+def _fmt_price(price: float) -> str:
+    """2dp for gold-scale prices, 5dp for sub-10 FX-scale prices."""
+    return f"{price:.2f}" if price >= 10 else f"{price:.5f}"
+
+
+def print_signal_report(report: SignalReport, detail: bool = False):
+    sc = signal_color(report.final_signal)
+    qc = quality_color(report.signal_quality)
+    sig_name = report.final_signal.name.replace("_", " ")
+
+    # ── 1. DECISION ────────────────────────────────────────────────
+    mtf_tag = (f"{Colors.GREEN}MTF✓{Colors.RESET}" if report.mtf_agreement
+               else f"{Colors.YELLOW}MTF✗{Colors.RESET}")
+    print(f"  {sc}▶ {sig_name}{Colors.RESET}  "
+          f"{qc}{report.signal_quality}{Colors.RESET} · "
+          f"{sc}{report.final_score:+.1f}{Colors.RESET} · "
+          f"{report.final_confidence:.0f}% · {mtf_tag}")
+    print(f"  {Colors.GRAY}{report.action_recommendation}{Colors.RESET}")
+
+    # ── 2. TRADE SETUP (if signal is actionable) ───────────────────
+    if report.trade_setup:
+        ts = report.trade_setup
+        dc = Colors.GREEN if ts.direction == "BUY" else Colors.RED
+        print(f"\n  {Colors.BOLD}Setup{Colors.RESET}  "
+              f"{dc}{ts.direction}{Colors.RESET} @ {Colors.BOLD}{_fmt_price(ts.entry_price)}{Colors.RESET}  "
+              f"→ SL {Colors.RED}{_fmt_price(ts.stop_loss)}{Colors.RESET} "
+              f"({ts.risk_pips:.1f}p)  "
+              f"→ TP {Colors.GREEN}{_fmt_price(ts.take_profit_1)}{Colors.RESET} "
+              f"({ts.reward_pips:.1f}p)  "
+              f"RR 1:{ts.risk_reward:.1f}")
+
+    # ── 3. CONTEXT (price · regime · vol · MTF breakdown) ──────────
+    print(f"\n  {Colors.WHITE}{report.display_name}{Colors.RESET} "
+          f"{Colors.BOLD}{_fmt_price(report.current_price)}{Colors.RESET} · "
+          f"{Colors.CYAN}{report.market_regime}{Colors.RESET} · "
+          f"vol {report.volatility_regime}")
+
+    if report.mtf_signals:
+        parts = []
+        for tf, sig in report.mtf_signals.items():
+            tf_sc = signal_color(sig)
+            sc_val = report.mtf_scores.get(tf, 0)
+            parts.append(f"{tf} {tf_sc}{sig.name.replace('_',' ')}{Colors.RESET} {sc_val:+.1f}")
+        print(f"  {Colors.GRAY}MTF{Colors.RESET}    " + "  ·  ".join(parts))
+
+    # ── 4. WHY — top 3 each side, compact ──────────────────────────
+    bulls = [_compact_reason(r) for r in report.bullish_reasons[:3]]
+    bears = [_compact_reason(r) for r in report.bearish_reasons[:3]]
+    if bulls or bears:
+        print()
+        rows = max(len(bulls), len(bears))
+        for i in range(rows):
+            left  = f"{Colors.GREEN}+{Colors.RESET} {bulls[i]}" if i < len(bulls) else ""
+            right = f"{Colors.RED}-{Colors.RESET} {bears[i]}" if i < len(bears) else ""
+            print(f"  {left:<60}  {right}")
+
+    # ── 5. WARNINGS / NEWS / EVENTS ────────────────────────────────
+    if report.warnings:
+        for w in report.warnings:
+            print(f"  {Colors.YELLOW}⚠{Colors.RESET}  {w}")
+    if report.event_description and report.event_risk != "clear":
+        print(f"  {Colors.YELLOW}📅{Colors.RESET} {report.event_description}")
+    if report.news_description and report.news_description != "Neutral news sentiment":
+        print(f"  {Colors.MAGENTA}📰{Colors.RESET} {report.news_description}")
+
+    # ── 6. DETAIL (only with --detail) ─────────────────────────────
+    if detail and report.primary_analysis:
+        print(f"\n  {Colors.BOLD}Indicators ({report.primary_analysis.timeframe}){Colors.RESET}")
+        for name, ind in report.primary_analysis.indicators.items():
+            if name.startswith("_") or ind.weight == 0:
+                continue
+            isc = signal_color(ind.signal)
+            print(f"    {ind.name:<16} {isc}{ind.signal.name.replace('_',' '):<11}{Colors.RESET} "
+                  f"{ind.confidence:>3.0f}% {Colors.GRAY}│ {ind.description}{Colors.RESET}")
+
+        levels = report.primary_analysis.key_levels
+        if levels:
+            swing_keys = sorted(
+                [k for k in levels if k.startswith("swing_")],
+                key=lambda k: levels[k], reverse=True,
+            )
+            if swing_keys:
+                print(f"\n  {Colors.BOLD}Structure Levels{Colors.RESET}")
+                for k in swing_keys:
+                    val = levels[k]
+                    color = Colors.RED if val > report.current_price else Colors.GREEN
+                    label = k.replace("_", " ").title()
+                    print(f"    {label:>20}  {color}{val:.3f}{Colors.RESET}")
+
+
+def report_to_dict(report: SignalReport) -> dict:
+    d = {
+        "symbol": report.symbol,
+        "display_name": report.display_name,
+        "timestamp": report.timestamp,
+        "current_price": report.current_price,
+        "signal": report.final_signal.name,
+        "score": round(report.final_score, 2),
+        "confidence": round(report.final_confidence, 2),
+        "quality": report.signal_quality,
+        "action": report.action_recommendation,
+        "market_regime": report.market_regime,
+        "volatility_regime": report.volatility_regime,
+        "mtf_agreement": report.mtf_agreement,
+        "mtf_signals": {k: v.name for k, v in report.mtf_signals.items()},
+        "event_risk": report.event_risk,
+        "event_description": report.event_description,
+        "news_sentiment": round(report.news_sentiment, 3),
+        "news_description": report.news_description,
+        "bullish_reasons": report.bullish_reasons,
+        "bearish_reasons": report.bearish_reasons,
+        "warnings": report.warnings,
+    }
+    if report.trade_setup:
+        ts = report.trade_setup
+        d["trade_setup"] = {
+            "direction": ts.direction,
+            "entry": ts.entry_price,
+            "stop_loss": ts.stop_loss,
+            "take_profit_1": ts.take_profit_1,
+            "take_profit_2": ts.take_profit_2,
+            "risk_reward": ts.risk_reward,
+            "risk_pips": round(ts.risk_pips, 1),
+            "reward_pips": round(ts.reward_pips, 1),
+        }
+    if report.primary_analysis:
+        d["indicators"] = {
+            name: {
+                "signal": ind.signal.name,
+                "confidence": round(ind.confidence, 1),
+                "description": ind.description,
+            }
+            for name, ind in report.primary_analysis.indicators.items()
+            if not name.startswith("_")
+        }
+        d["key_levels"] = {
+            k: round(v, 5) for k, v in report.primary_analysis.key_levels.items()
+        }
+    return d
+
+
+
+# ══════════════════════════════════════════════════════════════════
+#  CLI
+# ══════════════════════════════════════════════════════════════════
+
+def parse_args():
+    parser = argparse.ArgumentParser(
+        description="ForexBot2 — XAU/USD Signal Intelligence",
+        formatter_class=argparse.RawDescriptionHelpFormatter,
+        epilog="""
+Examples:
+  python main.py                          Live signal (H1, summary)
+  python main.py --detail                 Live signal with full breakdown
+  python main.py --timeframe H4 --detail  H4 timeframe
+  python main.py --json                   JSON output
+        """,
+    )
+    parser.add_argument("--pair", "-p", default="XAUUSD",
+                        help="Pair to analyze: XAUUSD (default: XAUUSD)")
+    parser.add_argument("--timeframe", "-tf", default="H1",
+                        choices=["M5", "M15", "M30", "H1", "H4", "D1"],
+                        help="Primary analysis timeframe (default: H1)")
+    parser.add_argument("--detail", action="store_true",
+                        help="Show full indicator breakdown")
+    parser.add_argument("--json", action="store_true",
+                        help="Output as JSON")
+    parser.add_argument("--verbose", "-v", action="store_true",
+                        help="Debug logging")
+
+    return parser.parse_args()
+
+
+def _resolve_pair(s: str) -> str:
+    aliases = {
+        "XAUUSD": XAUUSD, "XAU/USD": XAUUSD, "GOLD": XAUUSD,
+    }
+    key = s.upper().strip()
+    if key in aliases:
+        return aliases[key]
+    raise SystemExit(f"Unknown pair '{s}'. Try one of: {', '.join(sorted(aliases))}")
+
+
+def build_config(args) -> PlatformConfig:
+    pair = _resolve_pair(args.pair)
+
+    tf_map = {
+        "M5": Timeframe.M5, "M15": Timeframe.M15, "M30": Timeframe.M30,
+        "H1": Timeframe.H1, "H4": Timeframe.H4, "D1": Timeframe.D1,
+    }
+    primary_tf = tf_map[args.timeframe]
+
+    tf_confirmations = {
+        "M5":  [Timeframe.M1,  Timeframe.M15],
+        "M15": [Timeframe.M5,  Timeframe.H1],
+        "M30": [Timeframe.M15, Timeframe.H1],
+        "H1":  [Timeframe.M15, Timeframe.H4],
+        "H4":  [Timeframe.H1,  Timeframe.D1],
+        "D1":  [Timeframe.H4,  Timeframe.W1],
+    }
+    confirmation_tfs = tf_confirmations.get(
+        args.timeframe, [Timeframe.M15, Timeframe.H4]
+    )
+
+    # Per-pair tuned configs from the pair_signals factories.
+    factory = PAIR_PRODUCERS.get(pair)
+    if factory is not None:
+        return factory(primary_tf, confirmation_tfs)
+
+    config = PlatformConfig()
+    config.primary_timeframe = primary_tf
+    config.confirmation_timeframes = confirmation_tfs
+    return config
+
+
+def run_live(args, config: PlatformConfig):
+    print(f"{Colors.GRAY}  Fetching market data and running analysis...{Colors.RESET}\n")
+    aggregator = SignalAggregator(config)
+    report = aggregator.analyze_pair(symbol=config.symbol)
+
+    if args.json:
+        print(json.dumps(report_to_dict(report), indent=2))
+        return
+
+    print_header(f"{config.display_name} SIGNAL INTELLIGENCE")
+    print_signal_report(report, detail=args.detail)
+    print(f"\n{Colors.CYAN}{'═' * 80}{Colors.RESET}")
+    print(f"{Colors.GRAY}  Disclaimer: For educational/research purposes only.{Colors.RESET}")
+    print(f"{Colors.CYAN}{'═' * 80}{Colors.RESET}\n")
+
+
+def main():
+    args = parse_args()
+
+    log_level = logging.DEBUG if args.verbose else logging.WARNING
+    logging.basicConfig(
+        level=log_level,
+        format="%(asctime)s [%(levelname)s] %(name)s: %(message)s",
+        datefmt="%H:%M:%S",
+    )
+
+    config = build_config(args)
+    run_live(args, config)
+
+
+if __name__ == "__main__":
+    main()

news_engine.py

@@ -9,6 +9,9 @@ A perfect technical setup can be destroyed by an unexpected NFP print.
 """
 
 import logging
+import re
+import threading
+import time
 import xml.etree.ElementTree as ET
 from dataclasses import dataclass, field
 from datetime import datetime, timedelta, timezone
@@ -21,6 +24,21 @@ from config import NewsConfig
 logger = logging.getLogger(__name__)
 
 
+_RELEASE_VALUE_RE = re.compile(r"(-?\d+(?:\.\d+)?)\s*([kmbt])?", re.IGNORECASE)
+_RELEASE_MULTIPLIERS = {"k": 1e3, "m": 1e6, "b": 1e9, "t": 1e12}
+
+
+def _parse_release_value(raw: Optional[str]) -> Optional[float]:
+    """Parse a calendar release value like '0.3%', '-1.7%', '228K', '1.1M'."""
+    if not raw:
+        return None
+    match = _RELEASE_VALUE_RE.search(raw.replace(",", ""))
+    if not match:
+        return None
+    multiplier = _RELEASE_MULTIPLIERS.get((match.group(2) or "").lower(), 1.0)
+    return float(match.group(1)) * multiplier
+
+
 @dataclass
 class EconomicEvent:
     """Single economic calendar event."""
@@ -31,7 +49,21 @@ class EconomicEvent:
     forecast: Optional[str] = None
     previous: Optional[str] = None
     actual: Optional[str] = None
-    surprise: Optional[str] = None  # "positive", "negative", "inline", None
+
+    @property
+    def surprise(self) -> Optional[str]:
+        """Actual vs forecast: "positive"/"negative"/"inline", or None when
+        either value is missing or non-numeric. Direction is numeric only —
+        it does not know whether higher is market-bullish for this event."""
+        actual = _parse_release_value(self.actual)
+        forecast = _parse_release_value(self.forecast)
+        if actual is None or forecast is None:
+            return None
+        if actual > forecast:
+            return "positive"
+        if actual < forecast:
+            return "negative"
+        return "inline"
 
     @property
     def is_upcoming(self) -> bool:
@@ -56,6 +88,9 @@ class NewsSentiment:
     currencies_mentioned: List[str]
     sentiment_score: float  # -1.0 (bearish) to +1.0 (bullish)
     keywords: List[str]
+    # Per-currency score: monetary-policy keywords flip sign for XAU
+    # (hawkish is dollar-bullish but gold-bearish).
+    sentiment_by_currency: Dict[str, float] = field(default_factory=dict)
 
 
 @dataclass
@@ -81,39 +116,74 @@ class NewsEngine:
     # Keywords that indicate directional sentiment
     BULLISH_KEYWORDS = {
         "rate hike", "hawkish", "strong growth", "beat expectations",
-        "above forecast", "surplus", "expansion", "employment gains",
+        "above forecast", "above estimate", "above expectation",
+        "surplus", "expansion", "employment gains",
         "strong jobs", "gdp growth", "inflation rising", "tightening",
         "positive surprise", "record high", "bullish", "rally",
         "stronger than expected", "upbeat",
+        "beats", "surges", "soars", "jumps", "climbs", "rises",
+        "higher than expected", "better than expected",
+        "strong data", "robust", "resilient", "accelerates",
     }
 
     BEARISH_KEYWORDS = {
         "rate cut", "dovish", "recession", "slowdown", "contraction",
-        "below forecast", "deficit", "miss expectations", "weak data",
+        "below forecast", "below estimate", "below expectation",
+        "deficit", "miss expectations", "weak data",
         "unemployment rise", "negative surprise", "bearish", "selloff",
         "weaker than expected", "downbeat", "easing", "stimulus",
+        "misses", "slides", "drops", "falls", "plunges", "tumbles",
+        "weakens", "slumps", "declines", "retreats",
+        "lower than expected", "worse than expected",
+        "disappointing", "soft data",
     }
 
-    # Currency-specific keywords for headline parsing
+    # Monetary-policy keywords whose direction inverts for gold: tighter
+    # policy lifts the dollar and yields, weighing on non-yielding gold;
+    # easing does the opposite. Only unambiguous policy words — economic
+    # data words (beats/misses) keep their normal direction.
+    GOLD_INVERSE_KEYWORDS = {
+        "rate hike", "hawkish", "tightening",
+        "rate cut", "dovish", "easing", "stimulus",
+    }
+
+    # Currency-specific keywords for headline parsing. Matched with word
+    # boundaries — bare "us"/"eu"/"uk" are deliberately excluded because as
+    # substrings they false-positive constantly ("versus" → USD).
     CURRENCY_KEYWORDS = {
-        "USD": ["dollar", "fed", "fomc", "treasury", "us ", "american", "nonfarm", "nfp"],
-        "EUR": ["euro", "ecb", "eurozone", "eu ", "german", "french"],
-        "GBP": ["pound", "sterling", "boe", "bank of england", "uk ", "british"],
+        "USD": ["dollar", "fed", "federal reserve", "fomc", "treasury", "american", "nonfarm", "nfp"],
+        "EUR": ["euro", "ecb", "eurozone", "german", "french"],
+        "GBP": ["pound", "sterling", "boe", "bank of england", "british"],
         "JPY": ["yen", "boj", "bank of japan", "japanese"],
         "CHF": ["franc", "snb", "swiss"],
         "AUD": ["aussie", "rba", "australian", "australia"],
         "CAD": ["loonie", "boc", "canadian", "canada"],
         "NZD": ["kiwi", "rbnz", "new zealand"],
+        "XAU": ["gold", "precious metal", "safe haven", "bullion"],
     }
 
-    # High-impact event types (these move markets significantly)
-    HIGH_IMPACT_EVENTS = {
-        "interest rate decision", "nonfarm payrolls", "non-farm",
-        "gdp", "cpi", "inflation", "employment change",
-        "retail sales", "pmi", "fomc", "ecb press conference",
-        "boe rate", "rba rate", "snb rate", "boc rate",
+    # One compiled alternation per currency (code + keywords). (?<![a-z]) /
+    # (?![a-z]) instead of \b so "0.3%fed" still matches but "versus",
+    # "audit", "cascade" don't.
+    _CURRENCY_PATTERNS = {
+        cur: re.compile(
+            r"(?<![a-z])(?:"
+            + "|".join(re.escape(kw) for kw in [cur.lower()] + kws)
+            + r")(?![a-z])"
+        )
+        for cur, kws in CURRENCY_KEYWORDS.items()
     }
 
+    _calendar_backoff_until: float = 0.0
+
+    # Fetched-data cache, class-level: a fresh NewsEngine is built per scan,
+    # so instance state would never be reused. Guarded by _cache_lock since
+    # scans run concurrently in a thread pool.
+    _cache_lock = threading.Lock()
+    _calendar_cache: Tuple[float, List[EconomicEvent]] = (0.0, [])
+    _headlines_cache: Tuple[float, List[NewsSentiment]] = (0.0, [])
+    _EMPTY_RESULT_TTL = 60.0  # retry sooner after a failed/empty fetch
+
     def __init__(self, config: NewsConfig):
         self.config = config
         self._session = requests.Session()
@@ -139,7 +209,13 @@ class NewsEngine:
             events = self._fetch_economic_calendar(currencies)
             now = datetime.now(timezone.utc)
 
-            context.events_today = [e for e in events if e.time.date() == now.date()]
+            # Compare dates in UTC — feed timestamps carry their own offset
+            # (FF sends US-Eastern), so a late-evening ET event would
+            # otherwise land on the wrong day.
+            context.events_today = [
+                e for e in events
+                if e.time.astimezone(timezone.utc).date() == now.date()
+            ]
             context.events_upcoming_1h = [
                 e for e in events
                 if 0 < e.minutes_until <= 60
@@ -173,10 +249,12 @@ class NewsEngine:
             headlines = self._fetch_news_sentiment()
             context.news_headlines = headlines
 
-            # Aggregate sentiment per currency
+            # Aggregate sentiment per currency (per-currency score where
+            # available, so gold-inverted policy keywords are respected)
             for currency in currencies:
                 scores = [
-                    h.sentiment_score for h in headlines
+                    h.sentiment_by_currency.get(currency, h.sentiment_score)
+                    for h in headlines
                     if currency in h.currencies_mentioned
                 ]
                 if scores:
@@ -187,15 +265,21 @@ class NewsEngine:
 
         return context
 
+    @staticmethod
+    def _parse_pair(pair_symbol: str) -> Tuple[str, str]:
+        if "/" in pair_symbol:
+            parts = pair_symbol.split("/")
+            return parts[0], parts[1]
+        clean = pair_symbol.replace("=X", "")
+        return clean[:3], clean[3:]
+
     def get_event_risk_level(self, pair_symbol: str, context: FundamentalContext) -> Tuple[str, str]:
         """
         Get event risk level for a specific pair.
         Returns (risk_level, description).
         risk_level: "clear", "caution", "danger"
         """
-        clean = pair_symbol.replace("=X", "")
-        base = clean[:3]
-        quote = clean[3:]
+        base, quote = self._parse_pair(pair_symbol)
 
         # Check upcoming high-impact events
         for event in context.events_upcoming_1h:
@@ -206,10 +290,24 @@ class NewsEngine:
                 else:
                     return "caution", f"{event.currency} {event.event} in {mins:.0f}min"
 
-        # Check for recent high-impact releases (within post-event buffer)
+        # Check for recent high-impact releases. A surprise print (actual
+        # away from forecast) keeps the market unstable longer than an
+        # inline release, so it gets a wider post-event window.
         for event in context.events_today:
-            if event.is_high_impact and event.currency in (base, quote):
-                if not event.is_upcoming and event.minutes_until > -self.config.post_event_buffer_minutes:
+            if event.is_high_impact and event.currency in (base, quote) and not event.is_upcoming:
+                surprise = event.surprise
+                surprised = surprise in ("positive", "negative")
+                window = (
+                    self.config.post_event_surprise_buffer_minutes
+                    if surprised
+                    else self.config.post_event_buffer_minutes
+                )
+                if event.minutes_until > -window:
+                    if surprised:
+                        return "caution", (
+                            f"{event.currency} {event.event} released — {surprise} surprise "
+                            f"(actual {event.actual} vs forecast {event.forecast})"
+                        )
                     return "caution", f"{event.currency} {event.event} just released"
 
         return "clear", "No significant events"
@@ -222,9 +320,7 @@ class NewsEngine:
         Returns (score, description) where score is -1 to +1.
         Positive = bullish for the pair, negative = bearish.
         """
-        clean = pair_symbol.replace("=X", "")
-        base = clean[:3]
-        quote = clean[3:]
+        base, quote = self._parse_pair(pair_symbol)
 
         base_sent = context.news_sentiment.get(base, 0.0)
         quote_sent = context.news_sentiment.get(quote, 0.0)
@@ -247,18 +343,52 @@ class NewsEngine:
 
     # ── Private Methods ────────────────────────────────────────────
 
+    # Maps country names in MyFXBook titles to currency codes
+    _COUNTRY_TO_CURRENCY = {
+        "united states": "USD", "euro area": "EUR", "united kingdom": "GBP",
+        "japan": "JPY", "switzerland": "CHF", "australia": "AUD",
+        "canada": "CAD", "new zealand": "NZD",
+    }
+
+    def _cached_fetch(self, cache_attr: str, fetch_fn) -> list:
+        """Return cached data for `cache_attr` or refetch when the TTL
+        expired. Successful results live for context_cache_ttl_seconds;
+        empty results only _EMPTY_RESULT_TTL so outages retry quickly.
+        The lock is held across the fetch so concurrent scans share one
+        request instead of stampeding the source."""
+        ts, data = getattr(NewsEngine, cache_attr)
+        ttl = self.config.context_cache_ttl_seconds if data else self._EMPTY_RESULT_TTL
+        if time.time() - ts < ttl:
+            return data
+
+        with NewsEngine._cache_lock:
+            ts, data = getattr(NewsEngine, cache_attr)  # re-check under lock
+            ttl = self.config.context_cache_ttl_seconds if data else self._EMPTY_RESULT_TTL
+            if time.time() - ts < ttl:
+                return data
+            data = fetch_fn()
+            setattr(NewsEngine, cache_attr, (time.time(), data))
+            return data
+
     def _fetch_economic_calendar(self, currencies: List[str]) -> List[EconomicEvent]:
+        """Cached calendar fetch, filtered to the tracked currencies."""
+        events = self._cached_fetch("_calendar_cache", self._fetch_calendar_raw)
+        return [e for e in events if e.currency in currencies]
+
+    def _fetch_calendar_raw(self) -> List[EconomicEvent]:
         """
         Fetch economic calendar events from a free API.
+        Tries Forex Factory first, falls back to MyFXBook RSS.
         Returns an empty list (with a log warning) if unavailable —
         we never generate fake events since that causes constant false alarms.
         """
-        try:
-            events = self._try_fetch_calendar_api()
-            if events:
-                return [e for e in events if e.currency in currencies]
-        except Exception as e:
-            logger.debug(f"Calendar API fetch exception: {e}")
+        for fetcher in (self._try_fetch_calendar_api, self._try_fetch_myfxbook_rss):
+            try:
+                events = fetcher()
+                if events:
+                    return events
+            except Exception as e:
+                logger.debug(f"Calendar fetch exception ({fetcher.__name__}): {e}")
 
         logger.info(
             "Economic calendar data unavailable. "
@@ -268,41 +398,144 @@ class NewsEngine:
 
     def _try_fetch_calendar_api(self) -> List[EconomicEvent]:
         """Attempt to fetch from the Forex Factory free JSON calendar."""
-        events = []
+        now = time.time()
+        if now < NewsEngine._calendar_backoff_until:
+            wait = NewsEngine._calendar_backoff_until - now
+            logger.debug(f"Calendar API in backoff for {wait:.0f}s more")
+            return []
+
         url = "https://nfs.faireconomy.media/ff_calendar_thisweek.json"
-        resp = self._session.get(url, timeout=10)
+
+        for attempt in range(3):
+            try:
+                resp = self._session.get(url, timeout=10)
+            except requests.RequestException as e:
+                logger.debug(f"Calendar request failed (attempt {attempt + 1}): {e}")
+                if attempt < 2:
+                    time.sleep(2 ** attempt)
+                continue
+
+            if resp.status_code == 429:
+                retry_after = int(resp.headers.get("Retry-After", 300))
+                retry_after = min(retry_after, 600)
+                NewsEngine._calendar_backoff_until = time.time() + retry_after
+                logger.warning(
+                    f"Calendar API rate-limited (429), backing off {retry_after}s"
+                )
+                return []
+
+            if resp.status_code != 200:
+                logger.debug(f"Calendar API returned {resp.status_code}")
+                return []
+
+            data = resp.json()
+            events = []
+            impact_map = {"High": "high", "Medium": "medium", "Low": "low"}
+
+            for item in data:
+                try:
+                    impact = impact_map.get(item.get("impact", ""), "low")
+
+                    date_str = item.get("date", "")
+                    if not date_str:
+                        continue
+                    event_time = datetime.fromisoformat(date_str.replace("Z", "+00:00"))
+
+                    events.append(EconomicEvent(
+                        time=event_time,
+                        currency=item.get("country", ""),
+                        impact=impact,
+                        event=item.get("title", ""),
+                        forecast=item.get("forecast") or None,
+                        previous=item.get("previous") or None,
+                        actual=item.get("actual") or None,
+                    ))
+                except Exception:
+                    continue
+
+            return events
+
+        return []
+
+    def _try_fetch_myfxbook_rss(self) -> List[EconomicEvent]:
+        """Fallback calendar source: MyFXBook economic calendar RSS."""
+        url = "https://www.myfxbook.com/rss/forex-economic-calendar-events"
+        try:
+            resp = self._session.get(url, timeout=10)
+        except requests.RequestException as e:
+            logger.debug(f"MyFXBook calendar request failed: {e}")
+            return []
+
         if resp.status_code != 200:
             return []
 
-        data = resp.json()
-        impact_map = {"High": "high", "Medium": "medium", "Low": "low"}
+        try:
+            root = ET.fromstring(resp.content)
+        except ET.ParseError:
+            return []
 
-        for item in data:
+        events = []
+        for item in root.iter("item"):
             try:
-                impact = impact_map.get(item.get("impact", ""), "low")
-
-                date_str = item.get("date", "")
-                if not date_str:
+                title_el = item.find("title")
+                pub_el = item.find("pubDate")
+                desc_el = item.find("description")
+                if title_el is None or title_el.text is None:
                     continue
-                event_time = datetime.fromisoformat(date_str.replace("Z", "+00:00"))
+
+                title = title_el.text.strip()
+
+                currency = ""
+                title_lower = title.lower()
+                for country, cur in self._COUNTRY_TO_CURRENCY.items():
+                    if title_lower.startswith(country):
+                        currency = cur
+                        title = title[len(country):].strip()
+                        break
+                if not currency:
+                    continue
+
+                event_time = self._parse_rss_date(
+                    pub_el.text if pub_el is not None else None
+                )
+                if event_time is None:
+                    continue
+
+                impact = "low"
+                desc_text = desc_el.text if desc_el is not None and desc_el.text else ""
+                impact_match = re.search(r"sprite-(high|medium|low)-impact", desc_text)
+                if impact_match:
+                    impact = impact_match.group(1)
+
+                cells = re.findall(r"<td[^>]*>(.*?)</td>", desc_text, re.DOTALL)
+                cells = [re.sub(r"<[^>]+>", "", c).strip() for c in cells]
+                previous = cells[2] if len(cells) > 2 and cells[2] else None
+                forecast = cells[3] if len(cells) > 3 and cells[3] else None
+                actual = cells[4] if len(cells) > 4 and cells[4] else None
 
                 events.append(EconomicEvent(
                     time=event_time,
-                    currency=item.get("country", ""),
+                    currency=currency,
                     impact=impact,
-                    event=item.get("title", ""),
-                    forecast=item.get("forecast") or None,
-                    previous=item.get("previous") or None,
-                    actual=item.get("actual") or None,
+                    event=title,
+                    forecast=forecast,
+                    previous=previous,
+                    actual=actual,
                 ))
             except Exception:
                 continue
 
+        logger.debug(f"MyFXBook calendar: parsed {len(events)} events")
         return events
 
     def _fetch_news_sentiment(self) -> List[NewsSentiment]:
+        """Cached headline fetch + sentiment analysis."""
+        return self._cached_fetch("_headlines_cache", self._fetch_news_sentiment_raw)
+
+    def _fetch_news_sentiment_raw(self) -> List[NewsSentiment]:
         """Fetch and analyze news headlines for sentiment."""
         headlines = []
+        seen_titles = set()
 
         for feed_url in self.config.rss_feeds:
             try:
@@ -328,6 +561,14 @@ class NewsEngine:
                     if pub_date and (datetime.now(timezone.utc) - pub_date).total_seconds() > 86400:
                         continue
 
+                    # Dedup within and across feeds so a syndicated story
+                    # doesn't count twice in the sentiment average. Google
+                    # News appends " - Publisher", so compare without it.
+                    title_key = title.lower().rsplit(" - ", 1)[0].strip()
+                    if title_key in seen_titles:
+                        continue
+                    seen_titles.add(title_key)
+
                     sentiment = self._analyze_headline_sentiment(title)
                     currencies = self._extract_currencies(title)
 
@@ -339,6 +580,10 @@ class NewsEngine:
                             currencies_mentioned=currencies,
                             sentiment_score=sentiment,
                             keywords=self._extract_keywords(title),
+                            sentiment_by_currency={
+                                cur: self._analyze_headline_sentiment(title, cur)
+                                for cur in currencies
+                            },
                         ))
 
             except Exception as e:
@@ -346,19 +591,38 @@ class NewsEngine:
 
         return headlines
 
-    def _analyze_headline_sentiment(self, headline: str) -> float:
+    def _analyze_headline_sentiment(
+        self, headline: str, for_currency: Optional[str] = None
+    ) -> float:
         """
         Simple keyword-based sentiment analysis.
 
         Returns -1.0 (very bearish) to +1.0 (very bullish).
 
+        When for_currency is "XAU", monetary-policy keywords contribute
+        with inverted sign (see GOLD_INVERSE_KEYWORDS).
+
         Note: This is intentionally simple. NLP models like FinBERT
         would be more accurate but require large dependencies. This catches
         the most impactful sentiment signals from headlines.
         """
         headline_lower = headline.lower()
-        bullish_score = sum(1 for kw in self.BULLISH_KEYWORDS if kw in headline_lower)
-        bearish_score = sum(1 for kw in self.BEARISH_KEYWORDS if kw in headline_lower)
+        invert_policy = for_currency == "XAU"
+
+        bullish_score = 0
+        bearish_score = 0
+        for kw in self.BULLISH_KEYWORDS:
+            if kw in headline_lower:
+                if invert_policy and kw in self.GOLD_INVERSE_KEYWORDS:
+                    bearish_score += 1
+                else:
+                    bullish_score += 1
+        for kw in self.BEARISH_KEYWORDS:
+            if kw in headline_lower:
+                if invert_policy and kw in self.GOLD_INVERSE_KEYWORDS:
+                    bullish_score += 1
+                else:
+                    bearish_score += 1
 
         total = bullish_score + bearish_score
         if total == 0:
@@ -369,18 +633,11 @@ class NewsEngine:
     def _extract_currencies(self, headline: str) -> List[str]:
         """Extract mentioned currencies from a headline."""
         headline_lower = headline.lower()
-        found = []
-
-        for currency, keywords in self.CURRENCY_KEYWORDS.items():
-            if currency.lower() in headline_lower:
-                found.append(currency)
-                continue
-            for kw in keywords:
-                if kw in headline_lower:
-                    found.append(currency)
-                    break
-
-        return list(set(found))
+        return [
+            currency
+            for currency, pattern in self._CURRENCY_PATTERNS.items()
+            if pattern.search(headline_lower)
+        ]
 
     def _extract_keywords(self, headline: str) -> List[str]:
         """Extract relevant trading keywords from headline."""
@@ -396,6 +653,8 @@ class NewsEngine:
         formats = [
             "%a, %d %b %Y %H:%M:%S %z",   # RFC 822 with numeric tz offset
             "%a, %d %b %Y %H:%M:%S %Z",   # RFC 822 with tz abbreviation (GMT/UTC)
+            "%a, %d %b %Y %H:%M %Z",      # RFC 822 without seconds (MyFXBook)
+            "%a, %d %b %Y %H:%M %z",      # RFC 822 without seconds, numeric offset
             "%Y-%m-%dT%H:%M:%S%z",         # ISO 8601 with offset
             "%Y-%m-%d %H:%M:%S",           # ISO 8601 naive (assume UTC)
         ]
@@ -413,15 +672,8 @@ class NewsEngine:
 
     @staticmethod
     def _get_affected_pairs(currency: str) -> List[str]:
-        """Get all major pairs affected by a currency event."""
-        pairs = {
-            "USD": ["EURUSD=X", "GBPUSD=X", "USDJPY=X", "USDCHF=X", "AUDUSD=X", "USDCAD=X", "NZDUSD=X"],
-            "EUR": ["EURUSD=X", "EURGBP=X", "EURJPY=X", "EURCHF=X"],
-            "GBP": ["GBPUSD=X", "EURGBP=X", "GBPJPY=X"],
-            "JPY": ["USDJPY=X", "EURJPY=X", "GBPJPY=X", "AUDJPY=X", "CADJPY=X"],
-            "CHF": ["USDCHF=X", "EURCHF=X"],
-            "AUD": ["AUDUSD=X", "AUDJPY=X"],
-            "CAD": ["USDCAD=X", "CADJPY=X"],
-            "NZD": ["NZDUSD=X"],
+        _MAP = {
+            "USD": ["XAU/USD"],
+            "XAU": ["XAU/USD"],
         }
-        return pairs.get(currency, [])
+        return _MAP.get(currency, [])

pair_signals/__init__.py

@@ -0,0 +1,50 @@
+"""
+Per-pair signal producers.
+
+Each pair module exposes:
+    make_config(primary_tf, confirmation_tfs) -> PlatformConfig
+
+To add a new pair: create pair_signals/<pair>.py with make_config(),
+then register it in this file.
+"""
+
+from typing import Optional, List, Dict, Callable
+
+from config import PlatformConfig, Timeframe
+from signal_aggregator import SignalAggregator, SignalReport
+from data_provider import ForexDataProvider
+
+PAIR_PRODUCERS: Dict[str, Callable] = {}
+
+
+def register_pair(symbol: str, config_factory: Callable):
+    PAIR_PRODUCERS[symbol] = config_factory
+
+
+def get_supported_pairs() -> List[str]:
+    return list(PAIR_PRODUCERS.keys())
+
+
+def produce_signal(
+    symbol: str,
+    primary_tf: Timeframe,
+    data_provider: ForexDataProvider,
+    confirmation_tfs: Optional[List[Timeframe]] = None,
+) -> SignalReport:
+    config_factory = PAIR_PRODUCERS.get(symbol)
+    if config_factory is None:
+        raise ValueError(
+            f"No signal producer registered for {symbol}. "
+            f"Available: {list(PAIR_PRODUCERS.keys())}"
+        )
+
+    config = config_factory(primary_tf, confirmation_tfs)
+    aggregator = SignalAggregator(config)
+    aggregator.data_provider = data_provider
+    return aggregator.analyze_pair(symbol=symbol)
+
+
+from pair_signals import xauusd as _xau_mod
+from config import XAUUSD
+
+register_pair(XAUUSD, _xau_mod.make_config)

pair_signals/xauusd.py

@@ -0,0 +1,105 @@
+"""XAU/USD signal producer config.
+
+PlatformConfig defaults ARE the tuned XAU/USD configuration, so this
+factory sets the timeframes plus documented per-timeframe overrides.
+It must stay equivalent to the inline PlatformConfig(symbol=XAUUSD,
+...) construction (plus those overrides) used by the backtester and
+Telegram bot (enforced by a test in test_all.py).
+
+M15 override (validated 2026-07-11, window 2025-03-30 -> 2026-05-31,
+$0.30 spread, trailing exits): tp_floor_rr 1.0 scores +116.6R/460
+trades PF 1.83 vs +105.4R/424 PF 1.82 at the 0.5 default — better in
+both train/val halves and on the 2024-06 -> 2025-03 out-of-sample
+window (+30.3R vs +10.6R; the 1R floor lifts targets past the min-TP
+spread gate, roughly doubling trades there). The same knob HURTS M5
+(pinned 12mo book: +131.3R/699 vs +139.9R/547, PF 1.55 vs 1.78), so it
+is per-timeframe, exactly as PENDING_STEPS' decision rule anticipated.
+
+M15 override 2 (validated 2026-07-11, 12mo window 2025-07-11 ->
+2026-07-11 + OOS 2024-06-01 -> 2025-03-30, $0.30 spread): block the
+mixed "trending" regime (ADX strong but EMAs unaligned — momentum
+without directional structure). Those trades lost -6.8R/24 on 12mo,
+negative in BOTH halves; blocking scores +65.7R/499 PF 1.38 DD 10.4R
+vs +59.0R/521 PF 1.32 DD 12.6R baseline, better in both halves. OOS
+is neutral (-1.4R = the 4 such trades that existed there; no power to
+confirm or refute). M5 never trades this regime (its gates already
+exclude it), so the block is M15-only. A second candidate — blocking
+transitioning-regime SELLs (-4.0R/47 on 12mo) — was REJECTED: those
+sells made +5.7R/32 in the 2024 OOS window (bull-market artifact).
+
+M15 override 3 (validated 2026-07-16, 5-year replay 2021-07-17 ->
+2026-07-16 under the audit-fixed backtester, $0.30 spread, trailing
+exits): block SELL entries whose bar CLOSES at 15:00 or 17:00 UTC
+(sell_blocked_hours_utc=[15, 17], gate 6g3 — BUYs unaffected; those
+hours flank the already-blocked hour 16). The pocket lost -33.5R/165
+trades, negative in 6/6 calendar years and in BOTH halves of the last
+12 months (H1 -6.8R, H2 -4.2R); each hour independently is 5/6 years
+negative with both halves negative (hours 14/18 fail that screen and
+stay open). Cross-TF the M5 book CONFIRMS (same pocket -14.1R/59,
+5/6 years negative) rather than contradicts — unlike the rejected
+trending_down block. Full replay with the gate: +176.7R/1627 PF 1.31
+DD 29.1R vs +144.9R/1781 PF 1.22 DD 34.4R control; better in ALL SIX
+years; last-12mo halves both improve (H1 +36.3 vs +29.5, H2 +46.9 vs
++43.2; 12mo PF 1.55 vs 1.43); OOS 2024-06-01 -> 2025-03-30 improves
+(+45.4R PF 1.47 DD 17.8 vs +41.9R PF 1.39 DD 21.7). SELL side goes
+-1.2R -> +30.6R over 5 years. BUYs at those hours made +34.3R, hence
+SELL-only rather than a blocked_hours_utc entry. Mechanism: NY-session
+shorts against gold's macro uptrend (15:00 = post-US-data reversal
+window, 17:00 = London close unwind).
+
+M5 override 2 (2026-07-16, chart patterns): enable BUY-only CP on M5
+(inv_hs, double_bottom, bull_flag — drop all bearish patterns). SELL CP
+is consistently negative (OOS -3.21R, 5yr +3.76R PF 1.07) while BUY CP
+is positive across windows: 12mo +41.8R PF 1.60 (train +9.66 PF 1.30,
+val +32.17 PF 1.86), 5yr +65.2R PF 1.48, OOS +1.20R (aggregate
+positive, val half -1.89R — marginal but sign-correct in aggregate).
+Gold's structural uptrend makes bullish patterns fundamentally more
+reliable. M15/H1 fail on all windows; M5-only enablement.
+
+M5 override (validated 2026-07-11, full-history replay 2020-03-17 ->
+2026-07-11 + 12mo book, $0.30 spread): block hour 9 UTC (post-London-
+open whipsaw). Hour-9 entries lost -28.9R/143 over 6 years, negative
+in 5/7 calendar years including winning years (2022 -9.2R). Blocking:
++207.1R vs +178.3R baseline on the 6-year replay, max DD 28.7R vs
+35.5R, improves 5/7 years (2023 goes -13.0R -> breakeven); the tuned
+12mo book is unchanged (+139.0R vs +139.3R). M15-and-up keep hour 9:
+it is POSITIVE there (M15 6yr +1.7R, M15 OOS +8.6R), so the block is
+M5-only, same per-timeframe pattern as the M15 overrides.
+"""
+
+from typing import Optional, List
+from config import PlatformConfig, Timeframe
+
+_CONFIRMATIONS = {
+    Timeframe.M5:  [Timeframe.M15, Timeframe.H1],
+    Timeframe.M15: [Timeframe.M5, Timeframe.H1],
+    Timeframe.H1:  [Timeframe.M15, Timeframe.H4],
+    Timeframe.H4:  [Timeframe.H1, Timeframe.D1],
+}
+
+
+def make_config(
+    primary_tf: Timeframe,
+    confirmation_tfs: Optional[List[Timeframe]] = None,
+) -> PlatformConfig:
+    config = PlatformConfig()
+    config.primary_timeframe = primary_tf
+    config.confirmation_timeframes = (
+        confirmation_tfs or _CONFIRMATIONS.get(primary_tf, [Timeframe.M15, Timeframe.H4])
+    )
+    if primary_tf == Timeframe.M5:
+        # Block hour 9 UTC on M5 only — see module docstring.
+        config.xau_strategy.blocked_hours_utc = (
+            config.xau_strategy.blocked_hours_utc + [9]
+        )
+        # BUY-only chart patterns on M5 — see module docstring.
+        config.xau_strategy.cp_enabled = True
+        config.xau_strategy.cp_patterns = ["inv_hs", "double_bottom", "bull_flag"]
+    if primary_tf == Timeframe.M15:
+        # 1R minimum target on M15 only — see module docstring.
+        config.signals.tp_floor_rr = 1.0
+        # Block the mixed "trending" regime on M15 only — see docstring.
+        config.xau_strategy.blocked_regimes = ["trending"]
+        # Block SELL entries at 15:00/17:00 UTC on M15 only — see docstring.
+        config.xau_strategy.sell_blocked_hours_utc = [15, 17]
+    return config

requirements.txt

@@ -0,0 +1,6 @@
+numpy>=1.24
+pandas>=2.0
+requests>=2.28
+twelvedata>=1.2.0
+python-telegram-bot>=20.0
+websockets>=12.0

script.txt

@@ -1 +0,0 @@
-pip install pandas numpy requests python-dateutil -q

test_ws_dual.py

@@ -0,0 +1,108 @@
+#!/usr/bin/env python3
+"""Test whether Twelve Data allows 2 simultaneous WebSocket connections for XAU/USD.
+
+Uses the twelvedata SDK which handles the WS protocol correctly.
+Runs connection 1, then adds connection 2 after 10s, monitors for 30s total.
+"""
+
+import threading
+import time
+
+from twelvedata import TDClient
+
+API_KEY = "f563b22ffdfd4924955decbe1a8c8701"
+SYMBOL = "XAU/USD"
+DURATION = 30
+
+results = {
+    "WS-1": {"ticks": 0, "errors": [], "alive": False},
+    "WS-2": {"ticks": 0, "errors": [], "alive": False},
+}
+
+
+def make_handlers(label):
+    def on_event(event):
+        if event.get("event") == "price":
+            results[label]["ticks"] += 1
+            price = event.get("price", "?")
+            print(f"[{label}] tick #{results[label]['ticks']:3d}  price={price}")
+
+    def on_error(error):
+        results[label]["errors"].append(str(error))
+        print(f"[{label}] ERROR: {error}")
+
+    def on_close(status):
+        results[label]["alive"] = False
+        print(f"[{label}] CLOSED (status={status})")
+
+    def on_connect():
+        results[label]["alive"] = True
+        print(f"[{label}] CONNECTED — subscribed to {SYMBOL}")
+
+    return on_event, on_error, on_close, on_connect
+
+
+def run_ws(label):
+    td = TDClient(apikey=API_KEY)
+    on_event, on_error, on_close, on_connect = make_handlers(label)
+    ws = td.websocket(
+        symbols=SYMBOL,
+        on_event=on_event,
+        on_error=on_error,
+        on_close=on_close,
+        on_connect=on_connect,
+    )
+    ws.connect()
+    return ws
+
+
+def main():
+    print(f"Testing dual WebSocket for {SYMBOL} ({DURATION}s)\n")
+
+    print("Starting WS-1...")
+    ws1 = run_ws("WS-1")
+
+    time.sleep(10)
+    print(f"\nWS-1 ticks so far: {results['WS-1']['ticks']}")
+    print("Starting WS-2...\n")
+    ws2 = run_ws("WS-2")
+
+    remaining = DURATION - 10
+    time.sleep(remaining)
+
+    print("\nStopping connections...")
+    try:
+        ws1.disconnect()
+    except Exception:
+        pass
+    try:
+        ws2.disconnect()
+    except Exception:
+        pass
+
+    time.sleep(1)
+
+    print("\n" + "=" * 50)
+    print("RESULTS")
+    print("=" * 50)
+    for label, info in sorted(results.items()):
+        status = "alive" if info["alive"] else "dead"
+        errs = info["errors"] or ["none"]
+        print(f"  {label}: {status}  ticks={info['ticks']}  errors={errs[0]}")
+
+    ws1_ok = results["WS-1"]["ticks"] > 0 and not results["WS-1"]["errors"]
+    ws2_ok = results["WS-2"]["ticks"] > 0 and not results["WS-2"]["errors"]
+    ws1_died = results["WS-1"]["alive"] is False and results["WS-1"]["ticks"] > 0
+
+    if ws1_ok and ws2_ok:
+        print("\n✓ Both connections worked — dual WS is supported")
+    elif ws2_ok and ws1_died:
+        print("\n✗ WS-2 replaced WS-1 — only 1 connection allowed at a time")
+    elif ws1_ok and not ws2_ok:
+        print("\n✗ WS-2 was rejected — only 1 connection allowed")
+    else:
+        print("\n? Inconclusive — check errors above")
+
+
+if __name__ == "__main__":
+    main()