Telegram: subscriber channel feed, owner lock, full risk disclaimer

- TELEGRAM_CHANNEL_ID mirrors auto-scan signal + breach alerts to a
  subscriber channel (Stars-subscription monetization; Telegram manages
  membership). Feed keeps running even when the owner's DM auto-scan
  is toggled off.
- _owner_only decorator locks all commands/buttons to TELEGRAM_CHAT_ID;
  others get a refusal pointing at the channel.
- Every signal-bearing message (detailed signal, scan summary, all-TF
  summary, breach alerts) now ends with a not-financial-advice /
  educational-use disclaimer via a single _DISCLAIMER constant.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

.gitignore

@@ -0,0 +1,27 @@
+# Secrets
+.env
+.env.local
+
+# Editor / tooling
+.claude/
+.vscode/
+.idea/
+
+# Python
+__pycache__/
+*.pyc
+*.egg-info/
+.venv/
+venv/
+
+# OS
+.DS_Store
+Thumbs.db
+
+# Logs
+*.log
+
+# Backtesting data (keep scripts, ignore generated files)
+backtest/cache/
+backtest/*.csv
+backtest/__pycache__/

CLAUDE.md

@@ -0,0 +1,130 @@
+# ForexBot2
+
+Multi-pair signal intelligence platform (XAU/USD + EUR/USD). Analyzes each pair across multiple timeframes using 9 technical indicators with regime-adaptive weights, news sentiment, and economic calendar data. Delivers signals via Telegram with full trade setups (entry, SL, TP).
+
+## Architecture
+
+```
+Twelve Data REST API + WebSocket
+        |
+  data_provider.py          (fetch, cache, validate OHLCV; rate-limited 55 calls/min)
+        |
+    engine.py                (9 indicators per timeframe → composite score)
+        |
+  signal_aggregator.py       (MTF agreement + news + final signal + trade setup)
+        |
+  pair_signals/              (per-pair config factories: xauusd.py, eurusd.py)
+        |
+  +-----+--------+
+  |              |
+telegram_bot.py  main.py
+(Telegram alerts, (CLI per-pair
+ auto-scanner,    analysis)
+ WS monitor)
+```
+
+**Per-pair modularity:** each pair registers a `make_config(primary_tf, confirmation_tfs) -> PlatformConfig` factory in `pair_signals/`. `PlatformConfig.scan_pairs` is the single switch for which pairs the bot scans (single- or multi-pair). Market constants (pip value, price precision, backtest default spread) live in `PAIR_INFO`; strategy knobs live in per-pair dataclasses (`XauStrategyConfig`, `EurUsdStrategyConfig`). All XAU-specific gates in the aggregator are display-gated — nothing gold-calibrated touches the EUR path, and vice versa (enforced by pair-isolation tests + a pinned-window XAU regression CSV).
+
+## Core Files
+
+| File | Purpose |
+|------|---------|
+| `config.py` | All configuration dataclasses, `PAIR_INFO` registry, per-pair strategy configs |
+| `data_provider.py` | Twelve Data SDK wrapper with in-memory cache and rate limiting |
+| `engine.py` | `TechnicalEngine` — computes EMA, RSI, BB, ADX, ATR, MACD, ROC, Keltner, divergences, candlestick patterns, swing S/R |
+| `signal_aggregator.py` | `SignalAggregator` — MTF analysis, news integration, diminishing penalty cap, trade setup generation |
+| `news_engine.py` | Economic calendar (Forex Factory JSON) + RSS sentiment (keyword-based) |
+| `telegram_bot.py` | Async Telegram bot with auto-scanner (M5/M15/H1/H4), WebSocket price monitor |
+| `pair_signals/` | Per-pair producer registry (`PAIR_PRODUCERS`) + config factories |
+| `main.py` | CLI entry point (`--pair XAUUSD\|EURUSD`) |
+| `test_all.py` | Comprehensive QA suite (~176 tests, no live API needed) |
+
+## Key Concepts
+
+**Signal Pipeline:** OHLCV → indicators → regime classification → regime-weighted composite score → MTF agreement check → news/event adjustment → diminishing penalty cap → final signal (STRONG_BUY/BUY/NEUTRAL/SELL/STRONG_SELL) → quality grade (A+/A/B/C/D/F) → trade setup if grade C+.
+
+**Regime-Adaptive Weights:** Market classified as `trending_up`, `trending_down`, `ranging`, or `transitioning` via ADX + EMA + ADX slope. Gold-specific weights per regime:
+- **Trending:** MACD (2.0), EMA (2.0), Keltner (1.5), ROC (1.5), Divergence (1.0)
+- **Ranging:** Divergence (2.5), Keltner (2.0), RSI (1.5), BB (1.5), MACD (1.5)
+- **Transitioning:** Keltner (2.5), MACD (2.5), ROC (2.0), Divergence (2.0), EMA (1.5)
+
+ROC is directionally redundant with MACD/Keltner/EMA (0.8–0.94 correlation) but its weight is load-bearing: composite thresholds are calibrated with it — removing it backtested 13R worse. Don't zero it without retuning thresholds.
+
+**RSI (regime-aware, default):** In trends RSI buys pullback zones instead of fading strength (classic mode backtested worse for gold). In `transitioning` only extremes (≥70/≤30) signal.
+
+**Divergence:** 40-bar lookback; price and indicator swings must align within ±2 bars to count as one divergence observation.
+
+**Transitioning Regime:** ADX < 25 but ADX slope > 3.0 = trend emerging. Catches trend birth 5-10 bars before ADX crosses 25. Boosts quality grade for early entries.
+
+**Diminishing Penalty Cap:** Penalties sorted by magnitude, applied with exponential decay (1st at 100%, 2nd at 50%, 3rd at 25%). Hard cap at 30% total. Replaces old additive stacking that could gut 30-50% confidence.
+
+**Indicators (9 scored):** EMA Trend, RSI, Bollinger Bands, MACD, ADX, Keltner Channels (TTM Squeeze), ROC (Rate of Change), Divergence, Candlestick Patterns.
+
+**Trade Setup:** SL = nearest swing low/high ± 1×ATR buffer (min 1.5×ATR). TP = nearest swing level if RR≥1.2, else calculated from ATR. Minimum RR 0.5:1.
+
+## How to Run
+
+```bash
+# CLI
+python main.py                    # XAU/USD H1 default
+python main.py --pair EURUSD --timeframe M5 --detail
+
+# Telegram bot (auto-scanner)
+TELEGRAM_TOKEN=<token> TELEGRAM_CHAT_ID=<id> python telegram_bot.py
+
+# Tests
+python test_all.py
+
+# Backtest (spread defaults per pair: XAU $0.30, EUR 0.0001 — see PAIR_INFO)
+python backtest/gpu_backtest.py --pair XAUUSD --days 365
+python backtest/gpu_backtest.py --pair EURUSD --primary-tf 5min --confirm-tfs 15min,1h --days 365
+```
+
+## Environment Variables
+
+| Variable | Purpose | Default |
+|----------|---------|---------|
+| `TWELVE_DATA_API_KEY` | Market data API key | Hardcoded fallback in config |
+| `TELEGRAM_TOKEN` | Telegram bot token | Hardcoded fallback |
+| `TELEGRAM_CHAT_ID` | Owner chat ID (enables boot auto-scan; only this user can drive the bot) | Hardcoded fallback |
+| `TELEGRAM_CHANNEL_ID` | Subscriber channel ID (`-100…`); signal + breach alerts mirror there; `""` disables | Hardcoded fallback |
+
+## Dependencies
+
+`numpy`, `pandas`, `requests`, `twelvedata`, `python-telegram-bot`, `websockets` — see `requirements.txt`.
+
+## Data Provider
+
+- **Source:** Twelve Data REST API via `twelvedata` SDK
+- **Rate limit:** 55 credits/min (Grow plan), enforced by sliding-window throttle
+- **Cache:** In-memory dict, TTL = candle duration of the timeframe
+- **Validation:** Drops NaN rows, forward-fills gaps (limit 3), fixes OHLC inconsistencies, removes outliers (>10σ), strips future timestamps, deduplicates
+- **WebSocket:** subscribes every pair in `scan_pairs`
+
+## Telegram Bot
+
+- **Auto-scan:** M5, M15, H1, H4 at candle boundaries + 10s buffer (all timeframes alert)
+- **Manual scan:** Buttons for M15/H1/H4 individual scans + "Scan All TFs" multi-timeframe summary
+- **WebSocket:** Streams live ticks for XAU/USD; triggers immediate M15 scan on key level breach (swing S/R, BB, KC, EMA200) with 5-min cooldown
+- **Commands:** `/start`, `/scan`, `/autoscan`, `/status`, `/cache`, `/clearcache`, `/help`
+- **Owner lock:** all commands/buttons respond only to `TELEGRAM_CHAT_ID`; others get a refusal message
+- **Subscriber channel:** with `TELEGRAM_CHANNEL_ID` set, auto-scan + breach alerts also post to the channel (bot must be channel admin). Monetized via Telegram Stars subscription invite links — Telegram handles membership; the bot has no payment code. Channel feed keeps running even if the owner's DM auto-scan is off
+
+## Backtesting
+
+The `backtest/` directory contains `gpu_backtest.py` — a GPU-accelerated backtester (CuPy with NumPy fallback) that replays historical OHLCV through the real SignalAggregator. Pre-computes all indicator series including ROC, Keltner, ADX slope for 100x+ speedup. Uses parquet caching for data persistence.
+
+Fill realism: charges a round-trip spread per trade (`--spread`; defaults to the pair's `default_spread` in PAIR_INFO), fills gap-through SL/TP at the bar open, and resolves trades against the RAW price path (the parquet cache stores raw bars; validation/cleaning runs on load — signals see cleaned data, trade resolution sees real spikes). Entry is at the next bar's open after the signal bar. Weekend/session filters evaluate bar-close (decision) time to match the live bot's wall clock.
+
+GPU note: CuPy needs the NVIDIA runtime pip wheels (`nvidia-cuda-runtime-cu12`, `nvidia-cuda-nvrtc-cu12`, `nvidia-curand-cu12`, `nvidia-nvjitlink-cu12`) or it silently falls back to NumPy — check the "Backend:" line in the run log. The bar-replay loop itself is single-threaded Python either way (~340 bars/s); GPU only accelerates indicator precompute and SL/TP scanning.
+
+## Code Conventions
+
+- Python 3.12, type hints on public methods
+- Dataclasses for all config and output structures
+- `SignalStrength` enum (-2 to +2) is the universal signal currency
+- Indicators stored as `Dict[str, IndicatorSignal]` in `TechnicalAnalysis`
+- Key levels stored as `Dict[str, float]` — prefixed `swing_high_`, `swing_low_`, `ema_`, `bb_`, `kc_`, `atr`
+- Private indicator data attached as `result._rsi_series`, `result._macd_histogram`, `result._adx_slope`, `result._kc_squeeze` (underscore prefix, skipped by composite scorer)
+- Blocking signal computation runs in `ThreadPoolExecutor(max_workers=4)` to avoid blocking the async Telegram event loop
+- Shared `ForexDataProvider` instance across all scans for cache reuse

PULLBACK_ENTRY_PLAN.txt

@@ -0,0 +1,112 @@
+PULLBACK ENTRY PATTERN — IMPLEMENTATION PLAN
+=============================================
+Drafted 2026-07-09. Status: IMPLEMENTED & VALIDATED (same day).
+
+OUTCOME: shipped as XAU_PB. 12mo M5 validation: 254 PB trades,
++38.3R, avgR +0.151, WR 58%, halves +20.1/+18.1, all four full
+quarters positive. Core breakout trades undisturbed (-3.1R
+displacement). Combined book: 549 trades (~10.6/wk), +140.1R,
+PF 1.78, DD 11.3R. Calibration lesson: M5 gold trend legs measure
+3-4 ATR swing-to-trough — the drafted 0.5-2.5 ATR retrace window
+rejected 81% of valid pullbacks (fixed: 0.8-6.0). PB stops are
+~$8 vs ~$25 core, so PB got its own trail arm (0.8R vs 0.6R).
+Detector lives in signal_aggregator (NOT engine.py) so the GPU
+backtest engine exercises the same code path as live.
+
+WHY (evidence from 12mo M5 backtests, $0.30 spread, trailing exits)
+-------------------------------------------------------------------
+All 311 current trades are one archetype: trend-continuation breakout
+(EMA 4/4 aligned + ADX>25 + MACD accelerating + price through Keltner
+band). We systematically enter at maximum extension:
+
+- Trades carrying the "NO PULLBACK — price >2.5 ATR from EMA21" warning
+  average +0.063R vs +0.112R without it (145 vs 103 trades, 6mo set).
+- 28% of stopped-out trades never moved toward the target — instant
+  reversals from buying the local top / selling the local bottom.
+- "Extended — RSI in trend" warning trades: +0.057R vs +0.183R without.
+
+A pullback entry catches the SAME trends at better prices: enter on the
+retrace toward EMA21 / Keltner midline, stop below the retrace swing
+instead of 2×ATR behind an extended price. Tighter risk, same trend.
+This ADDS trades (new signal source) — it does not filter existing ones.
+
+DETECTOR SPEC (engine.py — new module-level check in M5 path)
+-------------------------------------------------------------
+Trend context (all required):
+  1. EMA alignment >= 3/4 in trend direction (allow 9<21 wobble mid-pullback)
+  2. ADX >= 20 on M5 (relaxed vs breakout's 25: ADX decays during retrace)
+  3. H1 confirmation score agrees with trend direction (>= +10 / <= -10),
+     reuse Gate-3 machinery.
+
+Pullback trigger (all required):
+  4. Price touched/undercut EMA21 or KC midline within last 3 bars
+     (low <= band for BUY; high >= band for SELL).
+  5. Retrace depth 0.5–2.5 ATR from the pre-pullback swing extreme
+     (deeper = trend likely broken; shallower = still extended).
+  6. Resumption bar: close back in trend direction (close > open for BUY)
+     AND close above EMA9 (below for SELL) — do not catch the falling knife.
+  7. RSI in the pullback zone: BUY 35–55, SELL 45–65 (NOT the extended
+     zones the breakout entries fire in — that is the whole point).
+
+Suppression:
+  8. Skip if a breakout signal fired within cooldown (no double entry
+     on the same leg — cd2/cd3 tests proved same-direction pyramiding
+     is train-half-only noise).
+  9. All existing M5 gates 1–6 still apply where meaningful; Gate 1
+     (|score|>=30) will need a variant — pullback bars have weaker
+     composite scores BY DESIGN (price below the momentum bands), so
+     validate a separate m5_pullback_min_score knob (start ~20).
+
+GEOMETRY
+--------
+- SL: below the pullback swing low (BUY) − 0.5×ATR buffer, floored at
+  min_sl_pips ($8). Expect typical SL 40–60% tighter than breakout SLs.
+- Exit: same trailing stop (arm +0.6R, trail 0.75R, max_hold 300).
+  NOTE: tighter SL means R units are smaller in dollars — re-validate
+  trail params on pullback trades separately; the arm/dist plateau was
+  measured on breakout-sized risk. If needed give pullback its own
+  trail knobs.
+- Tag trades strategy="XAU_PB" in the backtester so every report
+  splits breakout vs pullback cleanly.
+
+VALIDATION PROTOCOL (same discipline as 2026-07-09 sessions)
+------------------------------------------------------------
+1. Implement detector behind config flag m5_pullback_enabled=False.
+2. 365d backtest, flag on; export CSV; analyze XAU_PB trades alone:
+   - REQUIRE: positive totalR in BOTH data halves
+   - REQUIRE: positive in >= 3 of 4 full quarters
+   - REQUIRE: does not degrade breakout-trade totals (displacement
+     check vs baseline entry_time set, like the cooldown analysis)
+   - TARGET: >= +0.15R/trade average, >= 1 trade/week added
+3. Threshold sweeps (retrace depth, RSI zone, ADX floor) via the
+   offline exit-sim harness pattern where possible; full reruns where
+   entries change.
+4. Reject-and-document if it fails — same as blocked-hours/cooldown
+   experiments (see memory: entry-side experiments rejected).
+
+BASELINE TO BEAT (do not regress)
+---------------------------------
+12mo M5: 311 trades, +105.0R, PF 2.27, DD 7.9R, WR 61.4%,
+train/val +54.6/+50.4, every quarter positive, ~6-8 trades/week.
+(commit 987ec5b, CSV: m5_365d_trail300 in session scratchpad)
+
+IMPLEMENTATION ORDER
+--------------------
+1. config.py: m5_pullback_enabled, m5_pullback_min_score,
+   pullback RSI zones, retrace depth bounds.
+2. engine.py: pullback detector producing a candidate flag + swing
+   levels (attach as result._pullback_setup, underscore-private).
+3. signal_aggregator.py: consume flag, build XAU_PB trade setup with
+   pullback geometry; route through gates with pullback score variant.
+4. gpu_backtest.py: ensure strategy tag flows to CSV (already does via
+   strategy_tag) and pre-filter loop does not drop pullback candidates.
+5. Backtest + validate per protocol; tune; commit only on pass.
+
+RISKS
+-----
+- Pullback entries in a trend that IS reversing = catching the knife;
+  the resumption-bar condition (6) is the main guard — do not weaken it.
+- Score-gate variant (9) is the most likely source of overfitting;
+  sweep it last, keep the grid coarse.
+- M5 pullbacks during news spikes: calendar buffer already applies via
+  existing pipeline; verify it fires for XAU_PB too.

USAGE.md

@@ -0,0 +1,198 @@
+# ForexBot2 — USD/JPY Signal Intelligence
+
+A focused USD/JPY signal generator + walk-forward backtester. Live signals
+and backtests run through the **same** technical engine, so what you measure
+in a backtest is exactly what you'd execute live.
+
+## Setup
+
+```bash
+pip install -r requirements.txt
+```
+
+API key (optional — a default key is in `config.py`):
+
+```bash
+export TWELVE_DATA_API_KEY=your_key_here          # Linux/Mac
+$env:TWELVE_DATA_API_KEY="your_key_here"          # PowerShell
+```
+
+---
+
+## Live Signal
+
+```bash
+python main.py                          # H1, summary view
+python main.py --detail                 # Full indicator breakdown
+python main.py --timeframe H4 --detail  # H4 instead of H1
+python main.py --json                   # JSON output (for piping)
+```
+
+| Option        | Default | Description                                |
+|---------------|---------|--------------------------------------------|
+| `--timeframe` | `H1`    | `M5 M15 M30 H1 H4 D1`                      |
+| `--detail`    | off     | Indicator breakdown, key levels, reasons   |
+| `--json`      | off     | Machine-readable output                    |
+| `--verbose`   | off     | Debug logging                              |
+
+### Output
+
+```
+USD/JPY  │  Price: 149.230  │  Regime: trending_up  │  Vol: normal
+
+SIGNAL: BUY  │  Score: +28.4  │  Grade: B  │  Confidence: 62%
+Action: BUY — Decent setup, wait for confirmation
+
+Multi-Timeframe:  ✓ ALIGNED
+   M15: BUY          (+18.2)
+    H1: BUY          (+28.4)
+    H4: STRONG BUY   (+45.1)
+
+Trade Setup:
+  Direction:     BUY
+  Entry:         149.230
+  Stop Loss:     148.960  (27.0 pips)
+  Take Profit 1: 149.770  (54.0 pips)
+  Take Profit 2: 150.040
+  Risk:Reward:   1:2.0
+```
+
+---
+
+## Backtest (Walk-Forward)
+
+The backtester replays USD/JPY history bar-by-bar through the live signal
+pipeline. No lookahead, no parallel implementation — what passes in
+backtest passes live.
+
+```bash
+python main.py --backtest                          # Full available history
+python main.py --backtest --bars 2000              # Last 2000 H1 bars
+python main.py --backtest --start 2024-01-01 --end 2024-12-31
+python main.py --backtest --timeframe H4           # Backtest on H4
+python main.py --backtest --min-grade A            # Only A/A+ entries
+python main.py --backtest --journal                # Last 10 trades printed
+python main.py --backtest --journal-csv trades.csv # Full journal to CSV
+python main.py --backtest --regime-filter          # Skip choppy/ranging bars
+python main.py --backtest --regime-filter --regime-min-adx 35
+python main.py --backtest --session-filter         # Trade only 07-16 UTC
+python main.py --backtest --session-filter --session-start 8 --session-end 17
+python main.py --backtest --auto-session-filter    # Reactive session toggle
+python main.py --backtest --auto-session-filter --auto-session-lookback 15 \
+                          --auto-session-threshold-pct -0.5
+```
+
+### Regime filter
+
+The engine is a confluence-based trend-follower — it makes money in trending
+regimes and loses in chop. `--regime-filter` gates entries on H4 ADX:
+signals only fire when ADX ≥ `--regime-min-adx` (default 30). Sweet spot is
+ADX 30–35; tighter cuts trades faster than it improves quality.
+
+### Session filter
+
+`--session-filter` restricts entries to a UTC hour window (default 07-16,
+covering London + NY). Diagnostic on 2025-11 → 2026-05 showed 47% WR /
++127 pips inside that window vs 0% WR / -571 pips outside it. Override
+bounds with `--session-start` (inclusive) and `--session-end` (exclusive).
+
+Tradeoff: it's regime-specific medicine. In trending regimes (e.g. 2024)
+it cuts winning off-hours trades and gives back ~33% of the edge. Ships
+**default off** because the pure regime filter still wins cumulative
+across the 2024 / 2023 / 2026 windows.
+
+### Auto session toggle
+
+`--auto-session-filter` is a reactive variant: it tracks the rolling mean
+`pnl_pct` over the last N closed trades (`--auto-session-lookback`,
+default 10) and activates the 07-16 UTC window only when that mean drops
+below `--auto-session-threshold-pct` (default `-0.3`). Releases
+automatically once expectancy recovers.
+
+Less aggressive than the manual flag — preserves more trend edge but
+reacts slowly because the deque must fill with losses before kicking in.
+Use this when you don't want to manually toggle the regime medicine.
+
+### Execution model
+
+| Aspect             | Behavior                                                          |
+|--------------------|-------------------------------------------------------------------|
+| Entry              | Next bar's open (no current-bar lookahead)                        |
+| Stop Loss          | 1.5 × ATR from entry (configurable)                               |
+| Take Profit        | RR × stop distance (default 1:2)                                  |
+| Same-bar SL & TP   | Conservative: SL fills first                                      |
+| Spread cost        | Half-spread on entry + half-spread on exit (full round-trip)      |
+| Slippage           | 0.5 pips per fill (entry + exit)                                  |
+| Position size      | 1% of current equity risked per trade                             |
+| Concurrency        | One open position at a time                                       |
+| News in backtest   | Disabled — RSS/calendar can't be replayed without lookahead       |
+
+### Output
+
+```
+Backtest Summary — USD/JPY
+─────────────────────────────────────────────────
+Trades:           87  (52W / 35L)
+Win Rate:         59.8%
+Profit Factor:    1.84
+Expectancy:       +12.4 pips/trade  (+12.42 per trade)
+Initial Balance:  $10,000.00
+Final Balance:    $11,084.00
+Total Return:     +10.84%
+CAGR:             +18.92%
+Max Drawdown:     -4.21%
+Sharpe Ratio:     1.42
+Sortino Ratio:    2.18
+Avg Duration:     14.3 bars
+Win Streak:       7 | Loss Streak: 4
+─────────────────────────────────────────────────
+```
+
+---
+
+## Parameter Optimization (parallel)
+
+```bash
+python main.py --backtest --optimize
+```
+
+Runs a 3×3×3 grid (27 combos) over `atr_sl_multiplier`, `default_rr_ratio`,
+and `min_score` (signal-score threshold). Combos run in parallel across all
+CPU cores via `multiprocessing.Pool`. Twelve Data is hit once — workers
+share an on-disk parquet cache (`.cache/usdjpy_<tf>.parquet`).
+
+Output ranks combos by total return:
+
+```
+Top 10 Parameter Combinations (by total return):
+
+#   Return%     PF   WinR  Trades  MaxDD%  Sharpe  Params
+──────────────────────────────────────────────────────────────────────
+1   +14.21%   2.10  62.5%     64   -3.84   1.78    atr_sl_multiplier=1.5, default_rr_ratio=2.5, min_score=25.0
+2   +11.93%   1.92  58.7%     71   -4.11   1.55    atr_sl_multiplier=1.0, default_rr_ratio=2.0, min_score=15.0
+...
+```
+
+Customize the grid in `config.py → BacktestConfig.param_grid`.
+
+---
+
+## Files
+
+| File                  | Role                                              |
+|-----------------------|---------------------------------------------------|
+| `main.py`             | CLI entry point (live + backtest + optimize)      |
+| `config.py`           | All parameters in one place                       |
+| `data_provider.py`    | Twelve Data fetcher + parquet cache               |
+| `engine.py`           | Technical indicators (8 + divergence + patterns)  |
+| `signal_aggregator.py`| Multi-TF aggregation, grading, trade setup        |
+| `news_engine.py`      | RSS sentiment + economic calendar (live only)     |
+| `backtest.py`         | Walk-forward replay + parallel optimizer          |
+| `metrics.py`          | Win rate, PF, Sharpe, max DD, trade journal       |
+
+---
+
+## Disclaimer
+
+For educational and research use. Past performance does not guarantee
+future results. Always use proper risk management.

backtest/diagnose_filters.py

@@ -0,0 +1,609 @@
+"""
+ForexBot2 — Filter Rejection Diagnostic
+========================================
+Replays the same backtest loop as gpu_backtest.py but instead of
+collecting trades, counts how many M15 bars are rejected by each
+gate in the signal pipeline.
+
+Produces a summary table showing where potential trades are being
+lost, helping tune filter thresholds.
+
+Usage:
+    python backtest/diagnose_filters.py --pair XAUUSD --days 365
+    python backtest/diagnose_filters.py --pair XAUUSD --days 365 --sample-every 1
+"""
+
+from __future__ import annotations
+
+import argparse
+import logging
+import sys
+import time
+from collections import Counter, defaultdict
+from datetime import datetime, timedelta, timezone
+from pathlib import Path
+from typing import Dict, List, Optional, Tuple
+
+import numpy as np
+import pandas as pd
+
+# Ensure project root on sys.path.
+PROJECT_ROOT = Path(__file__).resolve().parent.parent
+if str(PROJECT_ROOT) not in sys.path:
+    sys.path.insert(0, str(PROJECT_ROOT))
+
+from config import (  # noqa: E402
+    PAIR_INFO,
+    PlatformConfig,
+    SignalStrength,
+    Timeframe,
+)
+from signal_aggregator import SignalAggregator, SignalReport  # noqa: E402
+
+# Reuse GPU backtest infrastructure (data loading, indicator pre-compute, etc.)
+from gpu_backtest import (  # noqa: E402
+    BAR_SECONDS,
+    GPUTechnicalEngine,
+    PAIR_ALIASES,
+    BacktestConfig,
+    IndicatorCache,
+    fetch_history,
+    precompute_indicators,
+    _is_xau_blocked_by_calendar,
+)
+from data_provider import ForexDataProvider  # noqa: E402
+
+logger = logging.getLogger("diagnose_filters")
+
+
+# ═════════════════════════════════════════════════════════════════════
+#  OUTCOME CLASSIFICATION
+# ═════════════════════════════════════════════════════════════════════
+
+def classify_bar(
+    report: SignalReport,
+    is_xau: bool,
+) -> Tuple[str, List[str]]:
+    """Classify a bar's outcome and list which filters killed it.
+
+    Returns:
+        (outcome, [filter_names])
+
+    Outcomes:
+        "TRADE"           — quality C+ and trade_setup exists
+        "QUALITY_OK_NO_SETUP" — quality C+ but trade_setup is None
+        "LOW_QUALITY"     — signal reached BUY/SELL but quality D or F
+        "NEUTRAL"         — signal stayed NEUTRAL (score didn't reach +/-30)
+
+    Filter names are extracted from report.warnings.
+    """
+    filters_hit: List[str] = []
+
+    # Scan warnings for known filter tags
+    filter_tags = {
+        "H4 RSI GATE":       "H4_RSI_GATE",
+        "H1 MACD GATE":      "H1_MACD_GATE",
+        "RSI ZONE GATE":     "RSI_ZONE_GATE",
+        "XAU BUY FILTER":    "XAU_BUY_FILTER",
+        "SESSION FILTER":    "SESSION_FILTER",
+        "DAY FILTER":        "DAY_FILTER",
+        "LOW ATR FILTER":    "LOW_ATR_FILTER",
+        "SELL SUPPRESSED":   "SELL_SUPPRESSED",
+        "EXHAUSTION":        "EXHAUSTION_PENALTY",
+        "TREND EXHAUSTION":  "TREND_EXHAUSTION",
+        "NO PULLBACK":       "NO_PULLBACK_PENALTY",
+        "Weak pullback":     "WEAK_PULLBACK_PENALTY",
+        "Extended entry":    "EXTENDED_ENTRY_PENALTY",
+        "Counter-trend":     "COUNTER_TREND_PENALTY",
+        "EXTREME volatility":"EXTREME_VOL_PENALTY",
+        "NEAR EMA200":       "NEAR_EMA200_PENALTY",
+        "FAR FROM EMA200":   "FAR_EMA200_PENALTY",
+        "EVENT RISK":        "EVENT_RISK",
+        "REGIME FILTER":     "REGIME_FILTER",
+        "Timeframes CONFLICT": "MTF_CONFLICT",
+    }
+
+    for w in report.warnings:
+        for tag, name in filter_tags.items():
+            if tag in w:
+                filters_hit.append(name)
+
+    # Determine outcome
+    non_neutral = report.final_signal not in (SignalStrength.NEUTRAL,)
+    quality_ok = report.signal_quality in ("A+", "A", "B", "C")
+
+    if quality_ok and report.trade_setup is not None:
+        return "TRADE", filters_hit
+    elif quality_ok and report.trade_setup is None:
+        return "QUALITY_OK_NO_SETUP", filters_hit
+    elif non_neutral and not quality_ok:
+        return "LOW_QUALITY", filters_hit
+    else:
+        return "NEUTRAL", filters_hit
+
+
+# ═════════════════════════════════════════════════════════════════════
+#  SCORE DISTRIBUTION TRACKER
+# ═════════════════════════════════════════════════════════════════════
+
+class ScoreTracker:
+    """Tracks score/confidence distributions for analysis."""
+
+    def __init__(self):
+        self.scores: List[float] = []
+        self.confidences: List[float] = []
+        self.abs_scores: List[float] = []
+        # How many bars had score in various ranges
+        self.score_bins = Counter()
+        self.conf_bins = Counter()
+        # Track "near miss" bars — score close to thresholds
+        self.near_miss_score: int = 0  # score 20-29 (almost BUY/SELL)
+        self.near_miss_conf: int = 0   # confidence 45-54 (almost passing)
+
+    def record(self, score: float, confidence: float):
+        self.scores.append(score)
+        self.confidences.append(confidence)
+        self.abs_scores.append(abs(score))
+
+        # Score bins
+        a = abs(score)
+        if a < 10:
+            self.score_bins["0-9"] += 1
+        elif a < 20:
+            self.score_bins["10-19"] += 1
+        elif a < 30:
+            self.score_bins["20-29"] += 1
+            self.near_miss_score += 1
+        elif a < 40:
+            self.score_bins["30-39"] += 1
+        elif a < 50:
+            self.score_bins["40-49"] += 1
+        else:
+            self.score_bins["50+"] += 1
+
+        # Confidence bins
+        if confidence < 30:
+            self.conf_bins["0-29"] += 1
+        elif confidence < 45:
+            self.conf_bins["30-44"] += 1
+        elif confidence < 55:
+            self.conf_bins["45-54"] += 1
+            self.near_miss_conf += 1
+        elif confidence < 65:
+            self.conf_bins["55-64"] += 1
+        elif confidence < 75:
+            self.conf_bins["65-74"] += 1
+        else:
+            self.conf_bins["75+"] += 1
+
+
+# ═════════════════════════════════════════════════════════════════════
+#  MAIN DIAGNOSTIC LOOP
+# ═════════════════════════════════════════════════════════════════════
+
+def run_diagnosis(
+    pair: str,
+    start: datetime,
+    end: datetime,
+    sample_every: int = 4,
+    use_cache: bool = True,
+    primary_tf: Timeframe = Timeframe.M15,
+    confirm_tfs: Optional[List[Timeframe]] = None,
+    warmup_bars: int = 250,
+) -> None:
+    if confirm_tfs is None:
+        confirm_tfs = [Timeframe.H1, Timeframe.H4]
+
+    pair_info = PAIR_INFO[pair]
+    display = pair_info["display"]
+    is_xau = display == "XAU/USD"
+
+    platform_cfg = PlatformConfig(
+        symbol=pair,
+        display_name=display,
+        pip_value=pair_info["pip_value"],
+        primary_timeframe=primary_tf,
+        confirmation_timeframes=list(confirm_tfs),
+    )
+
+    provider = ForexDataProvider(platform_cfg)
+
+    # ── Load data ──
+    timeframes = [primary_tf] + list(confirm_tfs)
+    pad_days = max(60, warmup_bars // 4)
+    fetch_start = start - timedelta(days=pad_days)
+
+    dfs_full: Dict[Timeframe, pd.DataFrame] = {}
+    for tf in timeframes:
+        df = fetch_history(
+            provider, pair, tf,
+            start=fetch_start, end=end,
+            use_cache=use_cache,
+        )
+        if df.empty:
+            raise RuntimeError(f"No data for {pair} {tf.value}")
+        dfs_full[tf] = df
+
+    # ── Pre-compute indicators ──
+    t0 = time.perf_counter()
+    caches: Dict[str, IndicatorCache] = {
+        tf.value: precompute_indicators(dfs_full[tf], platform_cfg.technical, gpu_log=True)
+        for tf in timeframes
+    }
+    logger.info(f"Indicator pre-compute: {time.perf_counter() - t0:.2f}s")
+
+    # ── Build aggregator with GPU engine ──
+    aggregator = SignalAggregator(platform_cfg)
+    aggregator.tech_engine = GPUTechnicalEngine(platform_cfg.technical, caches)
+
+    # ── Walk primary TF ──
+    primary_df = dfs_full[primary_tf]
+    bt_start_ts = pd.Timestamp(start)
+    bt_end_ts = pd.Timestamp(end)
+
+    first_eligible = max(
+        warmup_bars,
+        int(primary_df.index.searchsorted(bt_start_ts)),
+    )
+    last_pos = min(
+        int(primary_df.index.searchsorted(bt_end_ts, side="right")),
+        len(primary_df),
+    )
+
+    primary_dur = pd.Timedelta(seconds=BAR_SECONDS[primary_tf])
+
+    # Counters
+    total_bars_in_window = 0
+    total_bars_sampled = 0
+    outcome_counts = Counter()
+    filter_counts = Counter()
+    # Filters that killed bars that were otherwise directional (non-NEUTRAL score)
+    directional_filter_counts = Counter()
+    # Pre-pipeline rejections (XAU calendar filters applied before analyze_from_dataframes)
+    pre_pipeline = Counter()
+
+    # Track signal directions
+    signal_dir_counts = Counter()
+    quality_counts = Counter()
+
+    # Score tracker
+    tracker = ScoreTracker()
+
+    # Track "would-have-been" trades: bars where the primary score was
+    # directional but got killed by a specific filter
+    filter_kill_breakdown = defaultdict(Counter)
+
+    logger.info(
+        f"Diagnosing {display} {primary_tf.value}: "
+        f"bars {first_eligible}-{last_pos} (every {sample_every}th bar)"
+    )
+
+    t_loop = time.perf_counter()
+    for pos in range(first_eligible, last_pos, sample_every):
+        total_bars_in_window += 1
+        ts = primary_df.index[pos]
+        decision_time = ts + primary_dur
+
+        # ── XAU calendar pre-filter (same as gpu_backtest.py) ──
+        if is_xau:
+            block = _is_xau_blocked_by_calendar(ts, platform_cfg)
+            if block:
+                pre_pipeline[block] += 1
+                continue
+
+        total_bars_sampled += 1
+
+        # ── Build sliced dfs (same logic as gpu_backtest.py) ──
+        slice_dfs: Dict[Timeframe, pd.DataFrame] = {}
+        for tf in timeframes:
+            full = dfs_full[tf]
+            bar_dur = pd.Timedelta(seconds=BAR_SECONDS[tf])
+            cutoff = decision_time - bar_dur
+            cut = int(full.index.searchsorted(cutoff, side="right"))
+            if cut == 0:
+                continue
+            slice_dfs[tf] = full.iloc[:cut]
+
+        if primary_tf not in slice_dfs:
+            continue
+
+        # ── Run the full signal pipeline ──
+        report: SignalReport = aggregator.analyze_from_dataframes(
+            symbol=pair,
+            dfs=slice_dfs,
+            context=None,
+            use_news=False,
+        )
+
+        # Record score/confidence
+        tracker.record(report.final_score, report.final_confidence)
+
+        # Signal direction
+        signal_dir_counts[report.final_signal.name] += 1
+        quality_counts[report.signal_quality] += 1
+
+        # Classify
+        outcome, filters_hit = classify_bar(report, is_xau)
+        outcome_counts[outcome] += 1
+
+        for f in filters_hit:
+            filter_counts[f] += 1
+
+        # If the primary score was directional (>= 15 or <= -15), the bar
+        # had some potential — track what killed it
+        if abs(report.primary_score) >= 15:
+            for f in filters_hit:
+                directional_filter_counts[f] += 1
+
+        # For bars that got quality D/F but had a directional signal,
+        # track which filter warnings appeared
+        if outcome in ("LOW_QUALITY", "QUALITY_OK_NO_SETUP"):
+            for f in filters_hit:
+                filter_kill_breakdown[outcome][f] += 1
+
+    elapsed = time.perf_counter() - t_loop
+
+    # ═════════════════════════════════════════════════════════════════
+    #  PRINT RESULTS
+    # ═════════════════════════════════════════════════════════════════
+
+    bar = "=" * 80
+    print(f"\n{bar}")
+    print(f"  FILTER REJECTION DIAGNOSTIC — {display}")
+    print(f"  {start.date()} to {end.date()} | {primary_tf.value} "
+          f"(sampled every {sample_every}th bar)")
+    print(f"{bar}")
+
+    # Scale counts if sampling
+    scale_note = ""
+    if sample_every > 1:
+        scale_note = f"  (sampled {total_bars_sampled} of ~{total_bars_in_window * sample_every} eligible bars)"
+    print(f"\n  Total bars in backtest window: {total_bars_in_window}")
+    if pre_pipeline:
+        for reason, count in sorted(pre_pipeline.items()):
+            print(f"  Pre-pipeline rejection ({reason}): {count}")
+    print(f"  Bars analyzed: {total_bars_sampled}{scale_note}")
+    print(f"  Analysis took: {elapsed:.1f}s "
+          f"({total_bars_sampled / max(elapsed, 0.001):.0f} bars/s)")
+
+    # ── Outcome breakdown ──
+    print(f"\n{'-' * 80}")
+    print("  OUTCOME BREAKDOWN")
+    print(f"{'-' * 80}")
+    outcome_order = ["TRADE", "QUALITY_OK_NO_SETUP", "LOW_QUALITY", "NEUTRAL"]
+    outcome_labels = {
+        "TRADE":               "Actual trades (quality C+ & setup exists)",
+        "QUALITY_OK_NO_SETUP": "Quality C+ but NO trade setup generated",
+        "LOW_QUALITY":         "Directional signal but quality D/F",
+        "NEUTRAL":             "NEUTRAL (score didn't reach +/-30 or conf < 55)",
+    }
+    for o in outcome_order:
+        count = outcome_counts.get(o, 0)
+        pct = count / max(total_bars_sampled, 1) * 100
+        label = outcome_labels.get(o, o)
+        print(f"  {count:6d} ({pct:5.1f}%)  {label}")
+
+    # ── Signal direction breakdown ──
+    print(f"\n{'-' * 80}")
+    print("  SIGNAL DIRECTION BREAKDOWN")
+    print(f"{'-' * 80}")
+    for name in ["STRONG_BUY", "BUY", "NEUTRAL", "SELL", "STRONG_SELL"]:
+        count = signal_dir_counts.get(name, 0)
+        pct = count / max(total_bars_sampled, 1) * 100
+        print(f"  {count:6d} ({pct:5.1f}%)  {name}")
+
+    # ── Quality grade breakdown ──
+    print(f"\n{'-' * 80}")
+    print("  QUALITY GRADE BREAKDOWN")
+    print(f"{'-' * 80}")
+    for grade in ["A+", "A", "B", "C", "D", "F"]:
+        count = quality_counts.get(grade, 0)
+        pct = count / max(total_bars_sampled, 1) * 100
+        print(f"  {count:6d} ({pct:5.1f}%)  Grade {grade}")
+
+    # ── Filter hit counts (all bars) ──
+    print(f"\n{'-' * 80}")
+    print("  FILTER HIT COUNTS (all bars, a bar can hit multiple filters)")
+    print(f"{'-' * 80}")
+    if filter_counts:
+        for name, count in filter_counts.most_common():
+            pct = count / max(total_bars_sampled, 1) * 100
+            print(f"  {count:6d} ({pct:5.1f}%)  {name}")
+    else:
+        print("  (no filters triggered)")
+
+    # ── Filters that killed directional bars (primary_score >= 15) ──
+    print(f"\n{'-' * 80}")
+    print("  FILTERS KILLING DIRECTIONAL BARS (|primary_score| >= 15)")
+    print(f"{'-' * 80}")
+    if directional_filter_counts:
+        for name, count in directional_filter_counts.most_common():
+            print(f"  {count:6d}  {name}")
+    else:
+        print("  (none)")
+
+    # ── Filter breakdown for LOW_QUALITY bars ──
+    if filter_kill_breakdown.get("LOW_QUALITY"):
+        print(f"\n{'-' * 80}")
+        print("  FILTERS ON LOW_QUALITY BARS (had BUY/SELL but quality D/F)")
+        print(f"{'-' * 80}")
+        for name, count in filter_kill_breakdown["LOW_QUALITY"].most_common():
+            print(f"  {count:6d}  {name}")
+
+    # ── Filter breakdown for QUALITY_OK_NO_SETUP bars ──
+    if filter_kill_breakdown.get("QUALITY_OK_NO_SETUP"):
+        print(f"\n{'-' * 80}")
+        print("  FILTERS ON QUALITY_OK_NO_SETUP BARS (quality C+ but no trade setup)")
+        print(f"{'-' * 80}")
+        for name, count in filter_kill_breakdown["QUALITY_OK_NO_SETUP"].most_common():
+            print(f"  {count:6d}  {name}")
+
+    # ── Score distribution ──
+    print(f"\n{'-' * 80}")
+    print("  SCORE DISTRIBUTION (|score|)")
+    print(f"{'-' * 80}")
+    for bin_name in ["0-9", "10-19", "20-29", "30-39", "40-49", "50+"]:
+        count = tracker.score_bins.get(bin_name, 0)
+        pct = count / max(total_bars_sampled, 1) * 100
+        marker = " <-- near miss (almost BUY/SELL)" if bin_name == "20-29" else ""
+        print(f"  {count:6d} ({pct:5.1f}%)  |score| {bin_name}{marker}")
+
+    # ── Confidence distribution ──
+    print(f"\n{'-' * 80}")
+    print("  CONFIDENCE DISTRIBUTION")
+    print(f"{'-' * 80}")
+    for bin_name in ["0-29", "30-44", "45-54", "55-64", "65-74", "75+"]:
+        count = tracker.conf_bins.get(bin_name, 0)
+        pct = count / max(total_bars_sampled, 1) * 100
+        marker = " <-- near miss (almost passing)" if bin_name == "45-54" else ""
+        print(f"  {count:6d} ({pct:5.1f}%)  confidence {bin_name}{marker}")
+
+    # ── Near misses ──
+    print(f"\n{'-' * 80}")
+    print("  NEAR MISSES")
+    print(f"{'-' * 80}")
+    print(f"  {tracker.near_miss_score:6d}  Bars with |score| 20-29 (threshold is 30)")
+    print(f"  {tracker.near_miss_conf:6d}  Bars with confidence 45-54 (threshold is 55)")
+
+    # ── Summary statistics ──
+    if tracker.scores:
+        scores_arr = np.array(tracker.scores)
+        conf_arr = np.array(tracker.confidences)
+        print(f"\n{'-' * 80}")
+        print("  SCORE/CONFIDENCE STATISTICS")
+        print(f"{'-' * 80}")
+        print(f"  Score:      mean={np.mean(scores_arr):+.1f}  "
+              f"median={np.median(scores_arr):+.1f}  "
+              f"std={np.std(scores_arr):.1f}  "
+              f"min={np.min(scores_arr):+.1f}  "
+              f"max={np.max(scores_arr):+.1f}")
+        print(f"  Confidence: mean={np.mean(conf_arr):.1f}  "
+              f"median={np.median(conf_arr):.1f}  "
+              f"std={np.std(conf_arr):.1f}  "
+              f"min={np.min(conf_arr):.1f}  "
+              f"max={np.max(conf_arr):.1f}")
+
+    # ── Blocked hours analysis (for XAU) ──
+    if is_xau:
+        blocked = set(platform_cfg.xau_strategy.blocked_hours_utc)
+        all_hours = set(range(24))
+        open_hours = sorted(all_hours - blocked)
+        print(f"\n{'-' * 80}")
+        print("  XAU HOUR/DAY CONFIG")
+        print(f"{'-' * 80}")
+        print(f"  Blocked hours:  {sorted(blocked)}")
+        print(f"  Open hours:     {open_hours}")
+        print(f"  Allowed days:   {platform_cfg.xau_strategy.allowed_weekdays} "
+              f"(0=Mon..4=Fri)")
+        print(f"  Open fraction:  {len(open_hours)/24:.0%} of hours x "
+              f"{len(platform_cfg.xau_strategy.allowed_weekdays)}/5 weekdays = "
+              f"{len(open_hours)/24 * len(platform_cfg.xau_strategy.allowed_weekdays)/5:.0%} of trading time")
+
+    # ── Interpretation / key takeaways ──
+    print(f"\n{bar}")
+    print("  KEY TAKEAWAYS")
+    print(f"{bar}")
+
+    neutral_pct = outcome_counts.get("NEUTRAL", 0) / max(total_bars_sampled, 1) * 100
+    low_q_pct = outcome_counts.get("LOW_QUALITY", 0) / max(total_bars_sampled, 1) * 100
+    trade_count = outcome_counts.get("TRADE", 0)
+
+    print(f"  - {neutral_pct:.1f}% of bars never reach a directional signal (NEUTRAL)")
+    print(f"  - {low_q_pct:.1f}% of bars get a BUY/SELL but confidence/quality too low")
+    print(f"  - {trade_count} bars produce actual trades")
+
+    if filter_counts:
+        top3 = filter_counts.most_common(3)
+        print(f"  - Top rejection filters:")
+        for name, count in top3:
+            print(f"      {name}: {count} bars hit")
+
+    if tracker.near_miss_score > 50:
+        print(f"  - SIGNIFICANT: {tracker.near_miss_score} bars had |score| 20-29 "
+              f"(lowering threshold from 30 to 25 would capture many)")
+
+    if tracker.near_miss_conf > 50:
+        print(f"  - SIGNIFICANT: {tracker.near_miss_conf} bars had confidence 45-54 "
+              f"(lowering threshold from 55 to 50 would capture many)")
+
+    print(f"\n{bar}\n")
+
+
+# ═════════════════════════════════════════════════════════════════════
+#  CLI
+# ═════════════════════════════════════════════════════════════════════
+
+def _resolve_pair(s: str) -> str:
+    s_up = s.upper().strip()
+    if s_up in PAIR_ALIASES:
+        return PAIR_ALIASES[s_up]
+    if s_up in PAIR_INFO:
+        return s_up
+    raise SystemExit(f"Unknown pair '{s}'. Valid: {', '.join(sorted(PAIR_ALIASES))}")
+
+
+def _resolve_tf(s: str) -> Timeframe:
+    for tf in Timeframe:
+        if tf.value == s:
+            return tf
+    raise SystemExit(f"Unknown timeframe '{s}'")
+
+
+def main(argv: Optional[List[str]] = None) -> int:
+    p = argparse.ArgumentParser(
+        description="Diagnose which filters reject the most bars in the backtest pipeline",
+        formatter_class=argparse.ArgumentDefaultsHelpFormatter,
+    )
+    p.add_argument("--pair", required=True, help="EURUSD | GBPUSD | USDJPY | XAUUSD")
+    p.add_argument("--days", type=int, default=365, help="Days of history")
+    p.add_argument("--start", help="ISO date YYYY-MM-DD (overrides --days)")
+    p.add_argument("--end", help="ISO date YYYY-MM-DD (defaults to today)")
+    p.add_argument("--primary-tf", default="15min", help="Primary TF")
+    p.add_argument("--confirm-tfs", default="1h,4h", help="Comma-separated confirmation TFs")
+    p.add_argument("--warmup-bars", type=int, default=250)
+    p.add_argument("--sample-every", type=int, default=4,
+                   help="Analyze every Nth bar (1=all, 4=25%% sample for speed)")
+    p.add_argument("--no-cache", action="store_true", help="Skip parquet cache")
+    p.add_argument("--log-level", default="INFO")
+    args = p.parse_args(argv)
+
+    logging.basicConfig(
+        level=args.log_level.upper(),
+        format="%(asctime)s [%(levelname)s] %(name)s: %(message)s",
+    )
+
+    pair = _resolve_pair(args.pair)
+    primary_tf = _resolve_tf(args.primary_tf)
+    confirm_tfs = [_resolve_tf(s.strip()) for s in args.confirm_tfs.split(",") if s.strip()]
+
+    end = (
+        datetime.fromisoformat(args.end).replace(tzinfo=timezone.utc)
+        if args.end else datetime.now(timezone.utc)
+    )
+    if args.start:
+        start = datetime.fromisoformat(args.start).replace(tzinfo=timezone.utc)
+    else:
+        start = end - timedelta(days=args.days)
+
+    logger.info(
+        f"Diagnosing {PAIR_INFO[pair]['display']} | "
+        f"Window: {start.date()} -> {end.date()} | "
+        f"Sample: every {args.sample_every}th bar"
+    )
+
+    run_diagnosis(
+        pair=pair,
+        start=start,
+        end=end,
+        sample_every=args.sample_every,
+        use_cache=not args.no_cache,
+        primary_tf=primary_tf,
+        confirm_tfs=confirm_tfs,
+        warmup_bars=args.warmup_bars,
+    )
+
+    return 0
+
+
+if __name__ == "__main__":
+    sys.exit(main())

backtest/instructions_for_backtest.txt

@@ -0,0 +1,59 @@
+Note down entery and exit Why we took trade?
+What were each indicator signal on that specific time when we took trade
+How many indicators agreed at trade time
+Always backtest using engine.py and signal_aggregator.py
+Dont take trades on weekends and Fridays
+Keep TP small 1.3 Reward ratio
+At the end of test make table of results and print all values you observed.
+Trade context
+
+Time of day and trading session (Asian/London/NY/overlap)
+Day of week (track Mon/Tue/Wed/Thu performance separately)
+Market regime at entry: trending, ranging, or choppy (use ATR or ADX)
+Volatility level (ATR value or ATR percentile vs recent average)
+Pair/instrument traded
+News proximity (high-impact news within 30 min before/after entry)
+
+Trade execution details
+
+Spread at entry in pips
+Slippage on entry and exit (planned vs actual fill)
+Time in trade (minutes/hours from entry to exit)
+Maximum Adverse Excursion (MAE): how far against you it went
+Maximum Favorable Excursion (MFE): how far in profit before pullback
+
+Risk metrics
+
+Max consecutive losses (streak)
+Max consecutive wins (streak)
+Expectancy: (Win% x Avg Win) minus (Loss% x Avg Loss)
+Expectancy per hour in trade
+Recovery factor: Total R divided by Max DD
+
+Indicator/signal analysis
+
+Each indicator's standalone win rate when it agreed with the trade
+Confluence level performance (2 vs 3 vs 4+ indicators agreeing)
+Which indicator disagreed on losing trades (find the "canary")
+
+Decision quality
+
+Rate each setup A/B/C at entry, before knowing outcome
+Verify A-grade setups actually win more often
+Track system trades vs discretionary deviations separately
+
+Process notes
+
+Screenshot or chart snapshot at entry and exit
+Emotional state (calm, FOMO, revenge, tired)
+What would have happened if you held longer or exited sooner
+
+Aggregate analysis at end of backtest
+
+Equity curve, not just total R
+Performance breakdown by session, day, volatility regime
+Win rate over rolling 20-trade windows (check for edge decay)
+
+Critical extra
+"Do not trade" log: setups flagged but skipped, and what happened after
+

backtest/research/__init__.py

@@ -0,0 +1,5 @@
+"""Research tooling: calendar sourcing/store and edge-screening utilities.
+
+Nothing here is imported by the live bot; these modules support
+backtesting and edge validation only.
+"""

backtest/research/calendar_store.py

@@ -0,0 +1,94 @@
+"""Load and query the historical economic calendar (backtest side).
+
+The store is built by `ff_calendar_scraper.py`. Timestamps are unix
+epoch (UTC) — verified against known release times and price-volatility
+alignment (see scraper docstring).
+
+Key API:
+    cal = load_calendar()                          # full DataFrame, `utc` column
+    ev  = precise_events(cal, ["USD", "EUR"])      # high-impact, minute-accurate
+    mask = event_window_mask(bar_index, ev.utc, before, after)
+"""
+from pathlib import Path
+from typing import Iterable, Optional
+
+import numpy as np
+import pandas as pd
+
+PARQUET = Path(__file__).resolve().parent.parent / "cache" / "calendar.parquet"
+
+# timeLabel values that mean "no intraday time" — these events cannot
+# anchor an M5/M15 window (German Prelim CPI is the notable High one).
+_UNTIMED = {"All Day", "Tentative", ""}
+
+
+def _parse_release_value(raw) -> float:
+    """'0.3%' / '-1.7%' / '228K' / '1.1M' -> float (nan if unparseable)."""
+    import re
+    if raw is None or (isinstance(raw, float) and np.isnan(raw)):
+        return np.nan
+    m = re.search(r"(-?\d+(?:\.\d+)?)\s*([kmbt])?",
+                  str(raw).replace(",", ""), re.IGNORECASE)
+    if not m:
+        return np.nan
+    mult = {"k": 1e3, "m": 1e6, "b": 1e9, "t": 1e12}.get(
+        (m.group(2) or "").lower(), 1.0)
+    return float(m.group(1)) * mult
+
+
+def load_calendar(path: Path = PARQUET) -> pd.DataFrame:
+    df = pd.read_parquet(path)
+    df["utc"] = pd.to_datetime(df.dateline, unit="s", utc=True)
+    for col in ("actual", "forecast", "previous"):
+        df[f"{col}_num"] = df[col].map(_parse_release_value)
+    df["surprise"] = df.actual_num - df.forecast_num
+    return df
+
+
+def precise_events(
+    cal: pd.DataFrame,
+    currencies: Iterable[str] = ("USD", "EUR"),
+    impacts: Iterable[str] = ("high",),
+    require_values: bool = False,
+) -> pd.DataFrame:
+    """Events with a minute-accurate timestamp, filtered by currency/impact."""
+    out = cal[cal.currency.isin(list(currencies))
+              & cal.impactName.isin(list(impacts))
+              & ~cal.timeMasked.astype(bool)
+              & ~cal.timeLabel.fillna("").isin(_UNTIMED)]
+    if require_values:
+        out = out.dropna(subset=["actual_num", "forecast_num"])
+    return out.sort_values("utc").reset_index(drop=True)
+
+
+def event_window_mask(
+    bar_index: pd.DatetimeIndex,
+    event_times: pd.Series,
+    before: pd.Timedelta = pd.Timedelta(minutes=30),
+    after: pd.Timedelta = pd.Timedelta(minutes=30),
+) -> np.ndarray:
+    """Boolean mask: bar OPEN time falls inside [event-before, event+after).
+
+    Use on a backtest's entry times to identify trades entered around
+    releases (avoidance counterfactuals, event-window cost models).
+    """
+    starts = pd.DatetimeIndex(event_times) - before
+    ends = pd.DatetimeIndex(event_times) + after
+    order = np.argsort(starts.values)
+    starts, ends = starts.values[order], ends.values[order]
+    # merge overlapping windows, then interval-stab each bar time
+    merged_s, merged_e = [], []
+    for s, e in zip(starts, ends):
+        if merged_e and s <= merged_e[-1]:
+            merged_e[-1] = max(merged_e[-1], e)
+        else:
+            merged_s.append(s)
+            merged_e.append(e)
+    ms = np.array(merged_s)
+    me = np.array(merged_e)
+    ts = bar_index.values
+    pos = np.searchsorted(ms, ts, side="right") - 1
+    ok = pos >= 0
+    res = np.zeros(len(ts), dtype=bool)
+    res[ok] = ts[ok] < me[pos[ok]]
+    return res

backtest/research/ff_calendar_scraper.py

@@ -0,0 +1,144 @@
+"""Forex Factory calendar scraper -> backtest/cache/calendar.parquet.
+
+Fetches weekly calendar pages and parses the event JSON embedded in
+`window.calendarComponentStates` (the `days:` array). Event `dateline`
+is a unix epoch, so timestamps carry no timezone ambiguity — verified
+2026-07-11 against known release times (NFP 12:30/13:30 UTC across DST,
+ECB 12:15/13:15) and against EUR/USD M15 volatility: the bar containing
+a high-impact event averages 3.5x normal range, and shifting event
+times by ±1h destroys that alignment.
+
+Usage:
+    python backtest/research/ff_calendar_scraper.py --start 2024-01-01 --end 2026-07-13
+    python backtest/research/ff_calendar_scraper.py            # refresh: last stored week -> today
+
+Merges into the existing parquet on event (id, dateline), so refreshing
+is idempotent and also picks up revised `actual` values for recent weeks.
+Be polite: ~1.6s between requests (a full year is ~52 requests).
+"""
+import argparse
+import json
+import time
+from datetime import date, timedelta
+from pathlib import Path
+from typing import List, Optional
+
+import pandas as pd
+import requests
+
+CACHE_DIR = Path(__file__).resolve().parent.parent / "cache"
+PARQUET = CACHE_DIR / "calendar.parquet"
+
+UA = ("Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 "
+      "(KHTML, like Gecko) Chrome/126.0 Safari/537.36")
+
+FIELDS = ["id", "dateline", "name", "country", "currency", "impactName",
+          "timeLabel", "timeMasked", "actual", "forecast", "previous",
+          "revision", "actualBetterWorse"]
+
+
+def week_url(d: date) -> str:
+    return (f"https://www.forexfactory.com/calendar?week="
+            f"{d.strftime('%b').lower()}{d.day}.{d.year}")
+
+
+def extract_days(html: str) -> Optional[list]:
+    """Balanced-bracket extraction of the `days:` JSON array."""
+    i = html.find("days:")
+    if i < 0:
+        return None
+    j = html.find("[", i)
+    depth, in_str, esc = 0, False, False
+    for k in range(j, len(html)):
+        ch = html[k]
+        if in_str:
+            if esc:
+                esc = False
+            elif ch == "\\":
+                esc = True
+            elif ch == '"':
+                in_str = False
+        else:
+            if ch == '"':
+                in_str = True
+            elif ch == "[":
+                depth += 1
+            elif ch == "]":
+                depth -= 1
+                if depth == 0:
+                    return json.loads(html[j:k + 1])
+    return None
+
+
+def fetch_week(sess: requests.Session, d: date, retries: int = 4) -> Optional[List[dict]]:
+    for attempt in range(retries):
+        try:
+            r = sess.get(week_url(d), timeout=30)
+            if r.status_code == 200:
+                days = extract_days(r.text)
+                if days is not None:
+                    return [{k: e.get(k) for k in FIELDS}
+                            for day in days for e in day.get("events", [])]
+        except requests.RequestException:
+            pass
+        time.sleep(5 * (attempt + 1))
+    return None
+
+
+def scrape(start: date, end: date) -> pd.DataFrame:
+    sess = requests.Session()
+    sess.headers["User-Agent"] = UA
+    rows: List[dict] = []
+    failed: List[date] = []
+    d = start
+    while d <= end:
+        week_rows = fetch_week(sess, d)
+        if week_rows is None:
+            failed.append(d)
+            print(f"FAILED week {d}")
+        else:
+            rows.extend(week_rows)
+        d += timedelta(days=7)
+        time.sleep(1.6)
+    if failed:
+        print(f"WARNING: {len(failed)} weeks failed: {failed}")
+    return pd.DataFrame(rows)
+
+
+def merge_into_store(new: pd.DataFrame, path: Path = PARQUET) -> pd.DataFrame:
+    if path.exists():
+        old = pd.read_parquet(path)
+        combined = pd.concat([old, new])
+    else:
+        combined = new
+    # keep="last": a refresh overwrites stale rows (actuals fill in
+    # after release, revisions land weeks later).
+    combined = (combined.drop_duplicates(subset=["id", "dateline"], keep="last")
+                .sort_values("dateline").reset_index(drop=True))
+    combined.to_parquet(path, index=False)
+    return combined
+
+
+def main() -> None:
+    ap = argparse.ArgumentParser(description=__doc__.splitlines()[0])
+    ap.add_argument("--start", type=date.fromisoformat, default=None)
+    ap.add_argument("--end", type=date.fromisoformat, default=date.today())
+    args = ap.parse_args()
+
+    start = args.start
+    if start is None:
+        if PARQUET.exists():
+            last = pd.read_parquet(PARQUET).dateline.max()
+            start = pd.Timestamp(last, unit="s").date() - timedelta(days=7)
+        else:
+            start = date(2024, 1, 1)
+    print(f"Scraping {start} -> {args.end}")
+    new = scrape(start, args.end)
+    combined = merge_into_store(new)
+    print(f"Store: {len(combined)} events "
+          f"({pd.Timestamp(combined.dateline.min(), unit='s')} -> "
+          f"{pd.Timestamp(combined.dateline.max(), unit='s')}) -> {PARQUET}")
+
+
+if __name__ == "__main__":
+    main()

config.py

@@ -1,8 +1,6 @@
 """
-Forex Signal Intelligence Platform - Configuration
+ForexBot2 — Configuration (XAU/USD focused)
-===================================================
+==========================================
-Central configuration for all signal analysis parameters.
-Tuned by market regime and validated against historical performance.
 """
 
 from dataclasses import dataclass, field
@@ -37,160 +35,333 @@ class MarketSession(Enum):
     OVERLAP_LONDON_NY = "london_ny_overlap"
 
 
-# ── Major Forex Pairs ──────────────────────────────────────────────
+# ── Supported pairs ──────────────────────────────────────────────
-MAJOR_PAIRS = [
+XAUUSD = "XAU/USD"
-    "EURUSD=X", "GBPUSD=X", "USDJPY=X", "USDCHF=X",
+EURUSD = "EUR/USD"
-    "AUDUSD=X", "USDCAD=X", "NZDUSD=X",
-]
 
-CROSS_PAIRS = [
+# price_precision: decimals for SL/TP rounding and display.
-    "EURGBP=X", "EURJPY=X", "GBPJPY=X", "AUDJPY=X",
+# default_spread: round-trip cost per trade in price units, charged by
-    "EURCHF=X", "CADJPY=X",
+# the backtester when --spread is not given.
-]
+PAIR_INFO = {
-
+    XAUUSD: {"display": "XAU/USD", "pip_value": 0.10, "is_jpy_pair": False,
-# Yahoo Finance symbol mapping for display
+             "price_precision": 2, "default_spread": 0.30},
-SYMBOL_DISPLAY = {
+    EURUSD: {"display": "EUR/USD", "pip_value": 0.0001, "is_jpy_pair": False,
-    "EURUSD=X": "EUR/USD", "GBPUSD=X": "GBP/USD",
+             "price_precision": 5, "default_spread": 0.0001},
-    "USDJPY=X": "USD/JPY", "USDCHF=X": "USD/CHF",
-    "AUDUSD=X": "AUD/USD", "USDCAD=X": "USD/CAD",
-    "NZDUSD=X": "NZD/USD", "EURGBP=X": "EUR/GBP",
-    "EURJPY=X": "EUR/JPY", "GBPJPY=X": "GBP/JPY",
-    "AUDJPY=X": "AUD/JPY", "EURCHF=X": "EUR/CHF",
-    "CADJPY=X": "CAD/JPY",
 }
 
 
 @dataclass
 class TechnicalConfig:
-    """Technical indicator parameters - carefully tuned for Forex."""
+    """Technical indicator parameters."""
-    # EMA periods (fast/medium/slow for trend detection)
     ema_fast: int = 9
     ema_medium: int = 21
     ema_slow: int = 50
     ema_trend: int = 200
 
-    # RSI
     rsi_period: int = 14
     rsi_overbought: float = 70.0
     rsi_oversold: float = 30.0
     rsi_neutral_upper: float = 60.0
     rsi_neutral_lower: float = 40.0
 
-    # MACD (12, 26, 9 is standard)
     macd_fast: int = 12
     macd_slow: int = 26
     macd_signal: int = 9
 
-    # Bollinger Bands
     bb_period: int = 20
     bb_std: float = 2.0
 
-    # ATR for volatility measurement
     atr_period: int = 14
 
-    # Stochastic
-    stoch_k: int = 14
-    stoch_d: int = 3
-    stoch_overbought: float = 80.0
-    stoch_oversold: float = 20.0
-
-    # ADX for trend strength
     adx_period: int = 14
     adx_strong_trend: float = 25.0
     adx_very_strong: float = 50.0
 
-    # Ichimoku
+    divergence_lookback: int = 40
-    ichimoku_tenkan: int = 9
+    swing_sr_lookback: int = 50
-    ichimoku_kijun: int = 26
-    ichimoku_senkou_b: int = 52
 
-    # Volume profile
+    disabled_indicators: List[str] = field(default_factory=list)
-    volume_ma_period: int = 20
 
-    # Pivot points lookback
+    ema_mode: str = "4ema"          # "4ema" (9/21/50/200) or "2ema" (50/200)
-    pivot_lookback: int = 1
+    rsi_mode: str = "regime_aware"  # "classic" or "regime_aware"
+    macd_as_pillar: bool = True     # MACD histogram is a scored signal
 
-    # Divergence detection — how many bars back to scan for swing points
+    # ROC (Rate of Change)
-    divergence_lookback: int = 14
+    roc_period: int = 10
+    roc_threshold: float = 1.0     # % threshold for strong signal
+
+    # Keltner Channels
+    kc_period: int = 20
+    kc_atr_period: int = 10
+    kc_multiplier: float = 1.5
+
+    # ADX slope (trend acceleration detection)
+    adx_slope_lookback: int = 5
+    adx_slope_emerging_threshold: float = 3.0
+
+    regime_weights_override: Dict[str, Dict[str, float]] = field(default_factory=dict)
 
 
 @dataclass
 class SignalConfig:
-    """Signal scoring and filtering parameters."""
+    """Signal scoring & filtering."""
-    # Minimum confidence to emit a signal (0-100)
     min_confidence: float = 55.0
 
-    # Weight allocation for composite score
     weights: Dict[str, float] = field(default_factory=lambda: {
-        "trend": 0.25,          # EMA alignment, ADX
+        "ema_trend": 1.0,
-        "momentum": 0.20,       # RSI, Stochastic, MACD
+        "rsi": 1.5,
-        "volatility": 0.15,     # BB position, ATR regime
+        "bollinger": 1.5,
-        "price_action": 0.15,   # Candlestick patterns, S/R
+        "macd": 1.5,
-        "multi_timeframe": 0.15,# Higher TF confirmation
+        "keltner": 1.5,
-        "sentiment": 0.10,      # News/calendar events
+        "roc": 1.5,
+        "candlestick": 1.0,
+        "divergence": 1.5,
     })
 
-    # Multi-timeframe confirmation requirements
+    mtf_agreement_threshold: int = 2  # of 3 timeframes
-    mtf_agreement_threshold: int = 2  # out of 3 timeframes must agree
-
-    # Signal cooldown (prevent over-signalling same direction)
     cooldown_bars: int = 5
 
-    # Risk management defaults
+    default_risk_pct: float = 1.0
-    default_risk_pct: float = 1.0  # risk 1% per trade
+    default_rr_ratio: float = 2.0
-    default_rr_ratio: float = 2.0  # 1:2 risk-reward minimum
+    atr_sl_multiplier: float = 1.5
+    min_rr_ratio: float = 0.5
+    # TP tightening (May 2026) was tuned on the pre-spread backtester.
+    # Re-tested 2026-07-08 with $0.30 spread + gap fills: 0.8 nearly
+    # halves total R (+36.1R vs +66.8R, worse in both train/val halves)
+    # — smaller wins make the fixed spread a larger share of every TP.
+    tp_tightening_factor: float = 1.0
+
+    # ── Trade geometry ────────────────────────────────────────────
+    # Round-trip transaction cost estimate in price units (XAU ~$0.30).
+    round_trip_cost: float = 0.30
+    # Spread gate: reject setups whose TP1 distance is below
+    # round_trip_cost × this multiple (caps cost at ~3% of target).
+    # Validated 2026-07: small-target trades lose in both data halves
+    # on M5 while TP≥$10 trades earn ~+0.08R/trade.
+    min_tp_cost_multiple: float = 30.0
+    # Fixed SL floor in pips (80 pips = $8 on XAU). 0 disables — SL then
+    # floors at ATR × v2_sl_atr_multiplier only.
+    min_sl_pips: float = 80.0
+    # TP1 floor as a fraction of SL distance (0.5 = historical RR 0.5).
+    tp_floor_rr: float = 0.5
+
+    # ── Trailing-stop exit (2026-07) ──────────────────────────────
+    # Once favorable excursion reaches trail_arm_r × risk, the stop
+    # trails trail_dist_r × risk behind the best price (updated at bar
+    # close). TP1 becomes advisory — the trail is the exit. Validated
+    # on 12mo M5: +105R vs +48R with the fixed RR-0.5 target, balanced
+    # train/val, positive every quarter, PF 2.27. The RR-0.5 target was
+    # the binding constraint — winners kept running after TP.
+    # Set trail_arm_r = 0 to disable and exit at TP1 as before.
+    trail_arm_r: float = 0.6
+    trail_dist_r: float = 0.75
+    # Pullback (XAU_PB) trades run tighter stops (~$8 vs ~$25), so the
+    # global 0.6R arm fires on noise. Arm later; whole 0.8-arm row of
+    # the grid is +35-38R vs +32R at global params (balanced halves).
+    pb_trail_arm_r: float = 0.8
+    pb_trail_dist_r: float = 0.75
 
 
 @dataclass
 class NewsConfig:
-    """Economic calendar and sentiment configuration."""
+    """Economic calendar & sentiment."""
-    # High-impact event currencies
+    tracked_currencies: List[str] = field(default_factory=lambda: ["USD", "XAU", "EUR"])
-    tracked_currencies: List[str] = field(default_factory=lambda: [
-        "USD", "EUR", "GBP", "JPY", "CHF", "AUD", "CAD", "NZD"
-    ])
-
-    # Event impact levels to monitor
-    min_impact: str = "medium"  # "low", "medium", "high"
-
-    # Pre-event buffer (minutes before high-impact event = no new signals)
     pre_event_buffer_minutes: int = 30
-
-    # Post-event volatility window (minutes after event = cautious signals)
     post_event_buffer_minutes: int = 15
+    # Whether event risk gates the SIGNAL (danger: confidence hard-capped
+    # at 30; caution: 15-point penalty). Warnings are always emitted
+    # regardless. Default False = XAU-tuned: replaying the live gates
+    # against the validated 12mo XAU M5 book (547 trades, +139.9R)
+    # showed they suppress 25-35 trades worth +4.6..+7.2R/yr — trades
+    # entered near USD releases OUTPERFORM (+0.475R vs +0.256R avg).
+    # Audit 2026-07-11 via backtest/research/calendar_store.
+    event_risk_caps_signals: bool = False
+    # Wider stay-out window after a high-impact release that surprised
+    # vs forecast (inline releases use post_event_buffer_minutes).
+    post_event_surprise_buffer_minutes: int = 45
+    # How long fetched calendar/headline data is reused across scans.
+    # Must stay below pre_event_buffer_minutes so an event can't cross
+    # into the danger window while a stale cache says it's clear.
+    context_cache_ttl_seconds: int = 600
 
-    # News sentiment sources
     rss_feeds: List[str] = field(default_factory=lambda: [
-        "https://www.forexlive.com/feed/",
+        "https://investinglive.com/feed/",
         "https://www.fxstreet.com/rss",
+        # Google News query targeted at gold drivers — also carries Kitco,
+        # Reuters etc. whose direct feeds are unavailable.
+        "https://news.google.com/rss/search?q=gold+price+OR+fed+OR+inflation&hl=en-US&gl=US&ceid=US:en",
+        # CNBC "Economy" section
+        "https://search.cnbc.com/rs/search/combinedcms/view.xml?partnerId=wrss01&id=20910258",
     ])
 
 
+@dataclass
+class XauStrategyConfig:
+    """XAU/USD strategy config.
+    Legacy gates (H4 RSI, H1 MACD, M15 score) removed in favor of
+    new momentum indicators (ROC, Keltner squeeze, ADX slope).
+    """
+    enabled: bool = True
+    rsi_entry_threshold: float = 45.0
+    tp_atr_multiplier: float = 1.0
+    sl_atr_multiplier: float = 2.0
+    min_ema200_tfs: int = 2
+    allow_generic_trades: bool = True
+    blocked_hours_utc: List[int] = field(default_factory=lambda: [0, 1, 2, 10, 11, 13, 16, 21])
+    weekend_filter: bool = True
+    min_atr_threshold: float = 1.0
+    v2_sl_atr_multiplier: float = 2.0
+    buy_threshold_boost: int = 0
+    m5_confirmation_bars: int = 0
+    h4_rsi_gate: bool = False
+    h1_macd_gate: bool = False
+    rsi_zone: tuple = (40.0, 65.0)
+    allowed_weekdays: List[int] = field(default_factory=lambda: [0, 1, 2, 3, 4, 6])  # Mon-Fri + Sun (Sydney open)
+    sell_min_quality: List[str] = field(default_factory=lambda: ["A+", "A", "B"])
+    # Volatility regimes in which SELL setups are allowed (low/normal-vol
+    # shorts lose consistently against gold's macro uptrend).
+    sell_vol_regimes: List[str] = field(default_factory=lambda: ["high", "extreme"])
+
+    # ── M5-specific filters ──────────────────────────────────────
+    # M5 signals are noisy; require stronger confirmation before
+    # generating trade setups. Without these, M5 is breakeven
+    # (+0.01R/trade) with 21% drawdown.
+    m5_require_mtf: bool = True
+    m5_min_abs_score: float = 30.0
+    m5_min_confirmation_score: float = 10.0
+    # A-grade M5 SELLs enter at the extreme of temporary dips and
+    # lose (-0.017R/trade on 163 trades) — maximum conviction at
+    # the worst entry. Cap SELL quality so A-grade can't trigger
+    # the ATR-based setup generator; B-grade swing setups survive
+    # because they catch moderate drops, not extremes.
+    m5_sell_max_quality: List[str] = field(default_factory=lambda: ["B", "C"])
+    # 'high' vol regime on M5 is chop transition — elevated noise
+    # without directional expansion. 61% WR / negative total across
+    # 12mo while low/normal/extreme are all positive (validated 2026-07).
+    m5_blocked_vol_regimes: List[str] = field(default_factory=lambda: ["high"])
+    # Block M5 SELLs when RSI is already < this floor: the dip is
+    # extended and mean-reverts before the target (negative in both
+    # halves of 6mo and 12mo backtests).
+    m5_sell_min_rsi: float = 25.0
+
+    # ── M5 pullback entry (XAU_PB — see PULLBACK_ENTRY_PLAN.txt) ──
+    # Second entry archetype: enter on the retrace to EMA21/KC-mid in
+    # an established trend instead of at Keltner-breakout extension.
+    # Validated 2026-07: +38.3R/254 trades on 12mo, positive both
+    # halves and all four full quarters, core trades undisturbed.
+    m5_pullback_enabled: bool = True
+    # Composite score is weaker mid-pullback BY DESIGN (price sits
+    # below the momentum bands) — separate, lower floor than Gate 1.
+    m5_pullback_min_score: float = 15.0
+    m5_pullback_adx_min: float = 20.0
+    # RSI must be in the pullback zone, NOT the extended zone the
+    # breakout entries fire in (those run 65-80).
+    m5_pullback_rsi_buy: tuple = (35.0, 62.0)
+    m5_pullback_rsi_sell: tuple = (38.0, 65.0)
+    # Retrace depth from the pre-pullback swing extreme, in ATRs.
+    # Shallower = still extended; deeper = trend likely broken.
+    # M5 gold trend legs measure 3-4 ATR swing-to-trough (median 3.9),
+    # so the ceiling sits well above that of higher TFs.
+    m5_pullback_retrace_atr: tuple = (0.8, 6.0)
+    # Price must have touched EMA21/KC-mid within this many bars.
+    m5_pullback_touch_bars: int = 3
+
+
+@dataclass
+class EurUsdStrategyConfig:
+    """EUR/USD strategy config — separate from XAU for independent tuning.
+
+    Values ported from feature/eurusd-support are a pre-realism baseline
+    (tuned before spread charging and next-open fills) and are re-tuned
+    against the realistic backtester. Cost knobs (round_trip_cost,
+    min_tp_cost_multiple, min_sl_pips) are EUR-local so nothing
+    gold-calibrated from SignalConfig leaks into the EUR path.
+    """
+    enabled: bool = True
+    blocked_hours_utc: List[int] = field(default_factory=lambda: [22, 23, 0, 1, 2, 3])
+    weekend_filter: bool = True
+    allowed_weekdays: List[int] = field(default_factory=lambda: [0, 1, 2, 3, 4])
+    min_atr_threshold: float = 0.0003
+    sl_atr_multiplier: float = 1.5     # swing-buffer ATR multiple for SL placement
+    tp_atr_multiplier: float = 1.2
+
+    sl_atr_floor: float = 1.2
+    min_rr_ratio: float = 1.0
+    mature_move_threshold: float = 35.0
+    tp_quality_scale: Dict[str, float] = field(default_factory=lambda: {
+        "A+": 1.0, "A": 1.0, "B": 1.0, "C": 1.0,
+    })
+    blocked_regimes: List[str] = field(default_factory=lambda: ["ranging", "transitioning"])
+    min_quality_grades: List[str] = field(default_factory=lambda: ["A+", "A", "B"])
+    max_exhaustion_tags: int = 99
+    block_near_ema200: bool = False
+    session_hours: List[int] = field(default_factory=list)
+    min_entry_score: float = 38.0
+    min_entry_confidence: float = 60.0
+    block_rsi_extended: bool = True
+    blocked_vol_regimes: List[str] = field(default_factory=list)
+    min_indicator_agreement: int = 0
+    be_trigger_r: float = 0.3
+    be_buffer_r: float = 0.15
+
+    # ── EUR-local trade-cost knobs (replace SignalConfig's gold-dollar
+    # values in the EUR path) ─────────────────────────────────────────
+    # Round-trip cost per trade in price units (1.0 pip).
+    round_trip_cost: float = 0.0001
+    # Reject setups whose TP1 distance < round_trip_cost × this multiple
+    # (10 → 10-pip minimum target; EUR M5 ATR runs ~5-8 pips).
+    min_tp_cost_multiple: float = 10.0
+    # Minimum SL distance in pips.
+    min_sl_pips: float = 8.0
+
+
+@dataclass
+class FilterConfig:
+    """Regime and session filters for live signals."""
+    regime_filter: bool = False
+    regime_filter_timeframe: Timeframe = Timeframe.H4
+    regime_filter_min_adx: float = 25.0
+
+    session_filter: bool = True
+    session_start_hour: int = 3
+    session_end_hour: int = 23
+    blocked_hours_utc: List[int] = field(default_factory=lambda: [0, 1, 2, 10, 11, 13, 16, 21])
+
+    allowed_weekdays: List[int] = field(default_factory=lambda: [0, 1, 2, 3, 4, 6])  # Mon-Fri + Sun (Sydney open)
+
+
 @dataclass
 class PlatformConfig:
-    """Overall platform configuration."""
+    """Top-level configuration."""
     technical: TechnicalConfig = field(default_factory=TechnicalConfig)
     signals: SignalConfig = field(default_factory=SignalConfig)
     news: NewsConfig = field(default_factory=NewsConfig)
+    backtest: FilterConfig = field(default_factory=FilterConfig)
+    xau_strategy: XauStrategyConfig = field(default_factory=XauStrategyConfig)
+    eur_strategy: EurUsdStrategyConfig = field(default_factory=EurUsdStrategyConfig)
 
-    # Default pairs to monitor
+    symbol: str = XAUUSD
-    watch_pairs: List[str] = field(default_factory=lambda: MAJOR_PAIRS)
+    display_name: str = "XAU/USD"
+    pip_value: float = 0.10
 
-    # Analysis timeframes (primary + confirmation)
     primary_timeframe: Timeframe = Timeframe.H1
     confirmation_timeframes: List[Timeframe] = field(
         default_factory=lambda: [Timeframe.M15, Timeframe.H4]
     )
 
-    # Data source
+    data_lookback_days: int = 60
-    data_lookback_days: int = 60  # enough for 200 EMA on H1
 
-    # Twelve Data API key (overridden by TWELVE_DATA_API_KEY env var)
     twelve_data_api_key: str = "f563b22ffdfd4924955decbe1a8c8701"
 
-    # Dashboard refresh interval (seconds)
+    refresh_interval: int = 300
-    refresh_interval: int = 300  # 5 minutes
+
+    scan_pairs: List[str] = field(
+        default_factory=lambda: [XAUUSD]
+    )
+    scan_timeframes: List[Timeframe] = field(
+        default_factory=lambda: [Timeframe.M5, Timeframe.M15, Timeframe.H1, Timeframe.H4]
+    )
+    candle_buffer_seconds: int = 10
+    ws_cooldown_seconds: int = 300
 
-    # Logging
     log_level: str = "INFO"
-    log_signals_to_file: bool = True
-    signal_log_path: str = "signals.log"

data_provider.py

@@ -1,41 +1,45 @@
 """
-Forex Signal Intelligence Platform - Data Provider
+ForexBot2 — Data Provider (Twelve Data SDK)
-====================================================
+==========================================================
-Retrieves and validates market data from Twelve Data API.
+Fetches and validates OHLCV via the official `twelvedata` Python SDK.
-Falls back to synthetic GBM data if the API is unavailable.
 """
 
-import hashlib
 import logging
 import os
 import time
 from collections import deque
-from datetime import datetime, timedelta, timezone
+from typing import Dict, Optional
-from typing import Dict, List, Optional
 
-import numpy as np
 import pandas as pd
-import requests
+from twelvedata import TDClient
 
-from config import PlatformConfig, Timeframe, SYMBOL_DISPLAY
+from config import PlatformConfig, Timeframe, XAUUSD, PAIR_INFO
 
 logger = logging.getLogger(__name__)
 
 
+def _is_forex_weekend(now=None) -> bool:
+    """Forex market closed: Friday 22:00 UTC → Sunday 22:00 UTC."""
+    if now is None:
+        now = pd.Timestamp.now(tz="UTC")
+    wd = now.weekday()
+    hour = now.hour
+    if wd == 4 and hour >= 22:
+        return True
+    if wd == 5:
+        return True
+    if wd == 6 and hour < 22:
+        return True
+    return False
+
+
 class DataIntegrityError(Exception):
     """Raised when data fails validation checks."""
     pass
 
 
 class ForexDataProvider:
-    """
+    """Twelve Data SDK client + on-disk parquet cache."""
-    Forex data provider backed by Twelve Data API.
-
-    Twelve Data has native 4h candles, eliminating the H1→H4 resampling
-    hack needed for Yahoo Finance. Free tier: 800 credits/day, 8 req/min.
-    """
-
-    TD_BASE_URL = "https://api.twelvedata.com"
 
     TD_INTERVALS = {
         Timeframe.M1:  "1min",
@@ -48,20 +52,19 @@ class ForexDataProvider:
         Timeframe.W1:  "1week",
     }
 
-    # Approximate bars per day for each timeframe (used to size outputsize)
     BARS_PER_DAY = {
-        Timeframe.M1:  1440,
+        Timeframe.M1: 1440, Timeframe.M5: 288, Timeframe.M15: 96,
-        Timeframe.M5:  288,
+        Timeframe.M30: 48, Timeframe.H1: 24, Timeframe.H4: 6,
-        Timeframe.M15: 96,
+        Timeframe.D1: 1, Timeframe.W1: 0.143,
-        Timeframe.M30: 48,
-        Timeframe.H1:  24,
-        Timeframe.H4:  6,
-        Timeframe.D1:  1,
-        Timeframe.W1:  0.143,
     }
 
-    # Free plan: 8 API credits per minute (each symbol = 1 credit)
+    CANDLE_SECONDS = {
-    RATE_LIMIT = 8
+        Timeframe.M1: 60, Timeframe.M5: 300, Timeframe.M15: 900,
+        Timeframe.M30: 1800, Timeframe.H1: 3600, Timeframe.H4: 14400,
+        Timeframe.D1: 86400, Timeframe.W1: 604800,
+    }
+
+    RATE_LIMIT = 55
     RATE_WINDOW = 60.0
 
     def __init__(self, config: PlatformConfig):
@@ -72,94 +75,113 @@ class ForexDataProvider:
         )
         self._cache: Dict[str, pd.DataFrame] = {}
         self._cache_timestamps: Dict[str, float] = {}
-        self._request_times: deque = deque()  # timestamps of recent API credits used
+        self._request_times: deque = deque()
-        self._session = requests.Session()
+        self._td: Optional[TDClient] = (
-        self._session.headers.update({
+            TDClient(apikey=self._api_key) if self._api_key else None
-            "User-Agent": "Mozilla/5.0 (compatible; ForexIntel/1.0)"
+        )
-        })
 
     @staticmethod
     def _to_td_symbol(symbol: str) -> str:
-        """Convert internal symbol (EURUSD=X) to Twelve Data format (EUR/USD)."""
+        if "/" in symbol:
+            return symbol
         clean = symbol.replace("=X", "")
         return f"{clean[:3]}/{clean[3:]}"
 
+    # ── Public API ─────────────────────────────────────────────────
+
     def fetch_ohlcv(
         self,
-        symbol: str,
+        symbol: str = XAUUSD,
-        timeframe: Timeframe,
+        timeframe: Timeframe = Timeframe.H1,
         lookback_days: Optional[int] = None,
     ) -> pd.DataFrame:
-        """
+        """Fetch OHLCV with in-memory caching for live mode."""
-        Fetch OHLCV data for a Forex pair with full validation.
-
-        Returns a DataFrame with columns: Open, High, Low, Close, Volume
-        Index is a timezone-aware DatetimeIndex in UTC.
-        """
         if lookback_days is None:
             lookback_days = self.config.data_lookback_days
 
         cache_key = f"{symbol}_{timeframe.value}"
         now = time.time()
 
+        cache_ttl = self.CANDLE_SECONDS.get(timeframe, self.config.refresh_interval)
         if (
             cache_key in self._cache_timestamps
-            and (now - self._cache_timestamps[cache_key]) < self.config.refresh_interval
+            and (now - self._cache_timestamps[cache_key]) < cache_ttl
             and cache_key in self._cache
         ):
             cached = self._cache.get(cache_key)
             if cached is not None and not cached.empty:
-                return cached
+                if self._is_data_fresh(cached, symbol, timeframe):
+                    return cached
+                logger.warning(f"{symbol} {timeframe.value}: cached data is stale, refetching")
+                del self._cache[cache_key]
+                del self._cache_timestamps[cache_key]
 
-        df = self._fetch_twelve_data(symbol, timeframe, lookback_days)
+        outputsize = max(
+            300,
+            min(int(lookback_days * self.BARS_PER_DAY[timeframe]) + 20, 5000),
+        )
+        df = self._fetch_twelve_data(symbol, timeframe, outputsize)
+        df = self._drop_forming_candle(df, timeframe)
         df = self._validate_and_clean(df, symbol, timeframe)
 
+        if not self._is_data_fresh(df, symbol, timeframe):
+            logger.warning(
+                f"STALE DATA from API: {symbol} {timeframe.value} — "
+                f"last bar {df.index[-1]} is too old. Signal quality may be degraded."
+            )
+
         self._cache[cache_key] = df
         self._cache_timestamps[cache_key] = now
         return df
 
-    def prefetch_batch(
+    def _drop_forming_candle(self, df: pd.DataFrame, timeframe: Timeframe) -> pd.DataFrame:
-        self,
+        """Drop the still-forming last candle so analysis only sees closed bars.
-        symbols: List[str],
+
-        timeframes: List[Timeframe],
+        Twelve Data's time_series includes the current partial candle; the
-        lookback_days: Optional[int] = None,
+        scanner fires seconds after the boundary, so without this guard every
-    ) -> None:
+        indicator reads a seconds-old stub bar and signals repaint.
         """
-        Batch-fetch all symbols for each timeframe in a single API call per TF.
+        if df is None or df.empty:
+            return df
+        candle_seconds = self.CANDLE_SECONDS.get(timeframe, 3600)
+        now = pd.Timestamp.now(tz="UTC")
+        if df.index[-1] + pd.Timedelta(seconds=candle_seconds) > now:
+            df = df.iloc[:-1]
+        return df
 
-        Twelve Data accepts comma-separated symbols, so 7 pairs × 3 timeframes
+    def _is_data_fresh(self, df: pd.DataFrame, symbol: str, timeframe: Timeframe) -> bool:
-        = 3 requests instead of 21 — well within the free-tier 8 req/min limit.
+        """Check if the last candle is within acceptable age for this timeframe."""
-        Results are stored in the local cache; subsequent fetch_ohlcv calls
+        if df is None or df.empty:
-        for the same symbol/TF will be served from cache.
+            return False
-        """
-        if lookback_days is None:
-            lookback_days = self.config.data_lookback_days
 
-        for tf in timeframes:
+        now = pd.Timestamp.now(tz="UTC")
-            try:
-                self._batch_fetch_timeframe(symbols, tf, lookback_days)
-                time.sleep(1.0)  # brief pause between TF requests
-            except Exception as e:
-                logger.warning(f"Batch prefetch failed for {tf.value}: {e}")
 
-    def fetch_multiple(
+        if _is_forex_weekend(now):
-        self,
+            return True
-        symbols: List[str],
-        timeframe: Timeframe,
-        lookback_days: Optional[int] = None,
-    ) -> Dict[str, pd.DataFrame]:
-        """Fetch data for multiple symbols, returning dict of DataFrames."""
-        results = {}
-        for symbol in symbols:
-            try:
-                df = self.fetch_ohlcv(symbol, timeframe, lookback_days)
-                results[symbol] = df
-            except Exception as e:
-                logger.warning(f"Failed to fetch {symbol}: {e}")
-        return results
 
-    def get_latest_price(self, symbol: str) -> Optional[float]:
+        last_bar = df.index[-1]
-        """Return the most recent close price for a symbol."""
+        candle_seconds = self.CANDLE_SECONDS.get(timeframe, 3600)
+        max_age_seconds = candle_seconds * 3
+        age_seconds = (now - last_bar).total_seconds()
+
+        if age_seconds > max_age_seconds:
+            logger.warning(
+                f"{symbol} {timeframe.value}: last bar {last_bar.strftime('%Y-%m-%d %H:%M')} "
+                f"is {age_seconds / 60:.0f}m old (limit: {max_age_seconds / 60:.0f}m)"
+            )
+            return False
+
+        return True
+
+    def clear_cache(self) -> int:
+        """Wipe all cached data. Returns number of entries cleared."""
+        count = len(self._cache)
+        self._cache.clear()
+        self._cache_timestamps.clear()
+        logger.info(f"Cache cleared: {count} entries wiped")
+        return count
+
+    def get_latest_price(self, symbol: str = XAUUSD) -> Optional[float]:
         try:
             df = self.fetch_ohlcv(symbol, Timeframe.M5, lookback_days=1)
             if df is not None and not df.empty:
@@ -168,47 +190,26 @@ class ForexDataProvider:
             pass
         return None
 
-    def get_spread_estimate(self, symbol: str) -> float:
+    def get_spread_estimate(self, symbol: str = XAUUSD) -> float:
-        """Estimate typical spread from recent M5 high-low ranges."""
         try:
             df = self.fetch_ohlcv(symbol, Timeframe.M5, lookback_days=1)
             if df is not None and not df.empty:
                 return float((df["High"] - df["Low"]).median())
         except Exception:
             pass
+        info = PAIR_INFO.get(symbol, PAIR_INFO[XAUUSD])
+        return info["pip_value"]
 
-        fallback_spreads = {
+    def get_pair_info(self, symbol: str = XAUUSD) -> Dict:
-            "EURUSD=X": 0.00010, "GBPUSD=X": 0.00015,
+        info = PAIR_INFO.get(symbol, PAIR_INFO[XAUUSD])
-            "USDJPY=X": 0.010,   "USDCHF=X": 0.00015,
+        return {"symbol": symbol, **info}
-            "AUDUSD=X": 0.00015, "USDCAD=X": 0.00018,
-            "NZDUSD=X": 0.00020,
-        }
-        return fallback_spreads.get(symbol, 0.00020)
 
-    def get_pair_info(self, symbol: str) -> Dict:
+    # ── Private ────────────────────────────────────────────────────
-        """Return display name and pip value for a pair."""
-        display = SYMBOL_DISPLAY.get(symbol, symbol)
-        is_jpy = "JPY" in symbol
-        return {
-            "symbol": symbol,
-            "display": display,
-            "pip_value": 0.01 if is_jpy else 0.0001,
-            "is_jpy_pair": is_jpy,
-        }
-
-    # ── Private Methods ────────────────────────────────────────────
 
     def _throttle(self, credits: int = 1) -> None:
-        """
+        """Pro tier: 55 credits / minute. Block until budget allows."""
-        Block until `credits` API credits are available.
-
-        Twelve Data free tier: 8 credits/minute. Each symbol in any request
-        (single or batch) consumes 1 credit. This method enforces the limit
-        regardless of whether requests are batched or individual.
-        """
         while True:
             now = time.time()
-            # Evict credits older than the rate window
             while self._request_times and now - self._request_times[0] >= self.RATE_WINDOW:
                 self._request_times.popleft()
 
@@ -218,276 +219,76 @@ class ForexDataProvider:
                     self._request_times.append(time.time())
                 return
 
-            # Wait until the oldest credit expires
             wait = (self._request_times[0] + self.RATE_WINDOW) - now + 0.2
             logger.info(
                 f"Rate limit: {len(self._request_times)}/{self.RATE_LIMIT} credits used "
-                f"— waiting {wait:.1f}s for next window..."
+                f"— waiting {wait:.1f}s..."
             )
             time.sleep(wait)
 
-    def _batch_fetch_timeframe(
-        self, symbols: List[str], timeframe: Timeframe, lookback_days: int
-    ) -> None:
-        """
-        Single API call fetching all symbols for one timeframe.
-        Twelve Data returns a dict keyed by symbol when multiple are requested.
-        Parsed DataFrames are validated and stored in the cache.
-        """
-        if not self._api_key:
-            return
-
-        td_symbols = [self._to_td_symbol(s) for s in symbols]
-        interval = self.TD_INTERVALS[timeframe]
-        outputsize = min(
-            int(lookback_days * self.BARS_PER_DAY[timeframe]) + 20, 5000
-        )
-        outputsize = max(outputsize, 300)
-
-        # Each symbol in a batch request costs 1 credit
-        self._throttle(credits=len(symbols))
-
-        resp = self._session.get(
-            f"{self.TD_BASE_URL}/time_series",
-            params={
-                "symbol": ",".join(td_symbols),
-                "interval": interval,
-                "outputsize": outputsize,
-                "apikey": self._api_key,
-                "timezone": "UTC",
-                "format": "json",
-            },
-            timeout=30,
-        )
-        resp.raise_for_status()
-        data = resp.json()
-
-        # If the whole batch request returned a top-level error, raise it
-        if isinstance(data, dict) and data.get("status") == "error":
-            raise DataIntegrityError(
-                f"Batch API error ({timeframe.value}): {data.get('message', 'unknown')}"
-            )
-
-        # Batch response: {"EUR/USD": {...}, "GBP/USD": {...}, ...}
-        now = time.time()
-        for symbol, td_sym in zip(symbols, td_symbols):
-            try:
-                sym_data = data.get(td_sym, {})
-                if sym_data.get("status") == "error":
-                    logger.warning(
-                        f"Batch fetch error for {td_sym}: {sym_data.get('message')}"
-                    )
-                    continue
-
-                values = sym_data.get("values")
-                if not values:
-                    logger.warning(f"No values in batch response for {td_sym}")
-                    continue
-
-                records = [
-                    {
-                        "Datetime": pd.Timestamp(v["datetime"], tz="UTC"),
-                        "Open":   float(v["open"]),
-                        "High":   float(v["high"]),
-                        "Low":    float(v["low"]),
-                        "Close":  float(v["close"]),
-                        "Volume": float(v.get("volume") or 0),
-                    }
-                    for v in reversed(values)
-                ]
-                df = pd.DataFrame(records).set_index("Datetime")
-                df.index.name = "Datetime"
-                df = self._validate_and_clean(df, symbol, timeframe)
-
-                cache_key = f"{symbol}_{timeframe.value}"
-                self._cache[cache_key] = df
-                self._cache_timestamps[cache_key] = now
-                logger.debug(f"Batch cached {len(df)} bars for {td_sym} @ {interval}")
-
-            except Exception as e:
-                logger.warning(f"Failed to parse batch data for {td_sym}: {e}")
-
     def _fetch_twelve_data(
-        self, symbol: str, timeframe: Timeframe, lookback_days: int
+        self,
+        symbol: str,
+        timeframe: Timeframe,
+        outputsize: int,
+        start_date: Optional[str] = None,
+        end_date: Optional[str] = None,
     ) -> pd.DataFrame:
-        """
+        if self._td is None:
-        Fetch OHLCV from Twelve Data REST API.
+            raise DataIntegrityError(
-        Falls back to synthetic GBM data if API is unavailable or key is missing.
+                "No Twelve Data API key. Set TWELVE_DATA_API_KEY env var "
-        """
+                "or config.twelve_data_api_key."
-        if not self._api_key:
+            )
-            logger.warning("No Twelve Data API key — using synthetic data. "
-                           "Set TWELVE_DATA_API_KEY env var or config.twelve_data_api_key.")
-            return self._generate_synthetic_data(symbol, timeframe, lookback_days)
 
         td_symbol = self._to_td_symbol(symbol)
         interval = self.TD_INTERVALS[timeframe]
-        outputsize = min(
-            int(lookback_days * self.BARS_PER_DAY[timeframe]) + 20,
-            5000,
-        )
-        outputsize = max(outputsize, 300)
 
         self._throttle(credits=1)
 
+        kwargs = {
+            "symbol": td_symbol,
+            "interval": interval,
+            "outputsize": min(outputsize, 5000),
+            "timezone": "UTC",
+        }
+        if start_date:
+            kwargs["start_date"] = start_date
+        if end_date:
+            kwargs["end_date"] = end_date
+
         try:
-            resp = self._session.get(
+            df = self._td.time_series(**kwargs).as_pandas()
-                f"{self.TD_BASE_URL}/time_series",
-                params={
-                    "symbol": td_symbol,
-                    "interval": interval,
-                    "outputsize": outputsize,
-                    "apikey": self._api_key,
-                    "timezone": "UTC",
-                    "format": "json",
-                },
-                timeout=15,
-            )
-            resp.raise_for_status()
-            data = resp.json()
-
-            if data.get("status") == "error":
-                raise DataIntegrityError(
-                    f"Twelve Data API error for {td_symbol}: {data.get('message', 'unknown')}"
-                )
-
-            values = data.get("values")
-            if not values:
-                raise DataIntegrityError(f"Empty response from Twelve Data for {td_symbol}")
-
-            # Values arrive newest-first; reverse to chronological order
-            records = [
-                {
-                    "Datetime": pd.Timestamp(v["datetime"], tz="UTC"),
-                    "Open":   float(v["open"]),
-                    "High":   float(v["high"]),
-                    "Low":    float(v["low"]),
-                    "Close":  float(v["close"]),
-                    "Volume": float(v.get("volume") or 0),
-                }
-                for v in reversed(values)
-            ]
-
-            df = pd.DataFrame(records).set_index("Datetime")
-            df.index.name = "Datetime"
-            logger.debug(f"Fetched {len(df)} bars for {td_symbol} @ {interval}")
-            return df
-
-        except DataIntegrityError:
-            raise
         except Exception as e:
-            logger.info(f"Twelve Data fetch failed for {symbol} ({e}), using synthetic data.")
+            raise DataIntegrityError(
-            return self._generate_synthetic_data(symbol, timeframe, lookback_days)
+                f"Twelve Data fetch failed for {td_symbol}: {e}"
+            ) from e
 
-    def _generate_synthetic_data(
+        if df is None or df.empty:
-        self, symbol: str, timeframe: Timeframe, lookback_days: int
+            raise DataIntegrityError(f"Empty response from Twelve Data for {td_symbol}")
-    ) -> pd.DataFrame:
-        """
-        Generate realistic synthetic Forex OHLCV using GBM with mean-reversion.
-        Used as a fallback when live data is unavailable.
-        """
-        pair_params = {
-            "EURUSD=X": {"price": 1.1350, "vol": 0.07, "drift": -0.001},
-            "GBPUSD=X": {"price": 1.3200, "vol": 0.08, "drift":  0.002},
-            "USDJPY=X": {"price": 144.50, "vol": 0.09, "drift":  0.003},
-            "USDCHF=X": {"price": 0.8850, "vol": 0.07, "drift": -0.001},
-            "AUDUSD=X": {"price": 0.6550, "vol": 0.10, "drift": -0.002},
-            "USDCAD=X": {"price": 1.3650, "vol": 0.07, "drift":  0.001},
-            "NZDUSD=X": {"price": 0.6100, "vol": 0.10, "drift": -0.001},
-            "EURGBP=X": {"price": 0.8590, "vol": 0.06, "drift":  0.000},
-            "EURJPY=X": {"price": 163.90, "vol": 0.09, "drift":  0.002},
-            "GBPJPY=X": {"price": 190.70, "vol": 0.11, "drift":  0.003},
-            "AUDJPY=X": {"price": 94.60,  "vol": 0.11, "drift":  0.001},
-            "EURCHF=X": {"price": 0.9380, "vol": 0.05, "drift":  0.000},
-            "CADJPY=X": {"price": 105.80, "vol": 0.09, "drift":  0.001},
-        }
 
-        params = pair_params.get(symbol, {"price": 1.0000, "vol": 0.08, "drift": 0.000})
+        # SDK returns lowercase columns; engine expects capitalised.
-        base_price = params["price"]
+        df = df.rename(columns={
-        annual_vol = params["vol"]
+            "open": "Open", "high": "High", "low": "Low",
-        annual_drift = params["drift"]
+            "close": "Close", "volume": "Volume",
+        })
+        for col in ("Open", "High", "Low", "Close", "Volume"):
+            if col in df.columns:
+                df[col] = pd.to_numeric(df[col], errors="coerce")
+        if "Volume" not in df.columns:
+            df["Volume"] = 0.0
 
-        tf_minutes = {
+        # SDK returns newest-first; engine + indicators want oldest-first.
-            Timeframe.M1: 1, Timeframe.M5: 5, Timeframe.M15: 15,
+        df = df.sort_index()
-            Timeframe.M30: 30, Timeframe.H1: 60, Timeframe.H4: 240,
+        if df.index.tz is None:
-            Timeframe.D1: 1440, Timeframe.W1: 10080,
+            df.index = df.index.tz_localize("UTC")
-        }
-        bar_minutes = tf_minutes.get(timeframe, 60)
-        bars_per_year = (365.25 * 24 * 60) / bar_minutes
-        dt = 1.0 / bars_per_year
-
-        n_bars = min(int(lookback_days * 24 * 60 / bar_minutes), 2000)
-        n_bars = max(n_bars, 300)
-
-        seed_base = int(hashlib.md5(symbol.encode()).hexdigest()[:8], 16)
-        time_component = int(time.time()) // 3600
-        np.random.seed((seed_base + time_component) % (2**31))
-
-        bar_vol = annual_vol * np.sqrt(dt)
-        bar_drift = annual_drift * dt
-
-        regime_len = np.random.randint(20, 80)
-        trend_strength = np.random.normal(0, 0.3)
-
-        price = base_price
-        prices = [price]
-
-        for i in range(1, n_bars):
-            if i % regime_len == 0:
-                regime_len = np.random.randint(20, 80)
-                trend_strength = np.random.normal(0, 0.3)
-
-            mr_force = -0.001 * (price - base_price) / base_price
-            ret = bar_drift + trend_strength * dt + mr_force + bar_vol * np.random.randn()
-
-            if np.random.random() < 0.02:
-                ret *= np.random.uniform(2.0, 4.0)
-
-            price *= (1 + ret)
-            prices.append(price)
-
-        prices = np.array(prices)
-        opens = np.roll(prices, 1)
-        opens[0] = prices[0]
-
-        typical_range = bar_vol * base_price * 1.5
-        highs = np.maximum(prices, opens) + np.abs(np.random.normal(0, typical_range * 0.6, n_bars))
-        lows  = np.minimum(prices, opens) - np.abs(np.random.normal(0, typical_range * 0.6, n_bars))
-        highs = np.maximum(highs, np.maximum(opens, prices))
-        lows  = np.minimum(lows,  np.minimum(opens, prices))
-
-        volume = np.random.lognormal(mean=10, sigma=0.5, size=n_bars).astype(int)
-
-        freq_map = {
-            Timeframe.M1: "1min",  Timeframe.M5: "5min",
-            Timeframe.M15: "15min", Timeframe.M30: "30min",
-            Timeframe.H1: "1h",    Timeframe.H4: "4h",
-            Timeframe.D1: "D",     Timeframe.W1: "W",
-        }
-        freq = freq_map.get(timeframe, "1h")
-        end_time = pd.Timestamp.now(tz="UTC").floor(f"{bar_minutes}min")
-        index = pd.date_range(end=end_time, periods=n_bars, freq=freq, tz="UTC")
-
-        df = pd.DataFrame({
-            "Open": opens, "High": highs, "Low": lows,
-            "Close": prices, "Volume": volume,
-        }, index=index)
         df.index.name = "Datetime"
 
-        logger.info(
+        logger.debug(f"Fetched {len(df)} bars for {td_symbol} @ {interval}")
-            f"Generated {n_bars} synthetic {timeframe.value} bars for "
-            f"{SYMBOL_DISPLAY.get(symbol, symbol)} @ {prices[-1]:.5f}"
-        )
         return df
 
     def _validate_and_clean(
         self, df: pd.DataFrame, symbol: str, timeframe: Timeframe
     ) -> pd.DataFrame:
-        """
-        Validate and clean OHLCV data.
-
-        Steps: drop all-NaN rows → ffill small gaps → remove future timestamps
-        → fix OHLC inconsistencies → remove >10-sigma outliers → sort & deduplicate.
-        """
         if df is None or df.empty:
             raise DataIntegrityError(f"Empty dataset for {symbol}")
 
@@ -497,6 +298,9 @@ class ForexDataProvider:
         df = df.ffill(limit=3)
         df = df.dropna(subset=["Close"])
 
+        df = df.sort_index()
+        df = df[~df.index.duplicated(keep="last")]
+
         now = pd.Timestamp.utcnow()
         future_mask = df.index > now + pd.Timedelta(minutes=5)
         if future_mask.any():
@@ -527,7 +331,6 @@ class ForexDataProvider:
 
         min_required = max(
             self.config.technical.ema_trend,
-            self.config.technical.ichimoku_senkou_b,
             200,
         ) + 10
 
@@ -537,9 +340,6 @@ class ForexDataProvider:
                 f"(need {min_required}). Some indicators will be partial."
             )
 
-        df = df.sort_index()
-        df = df[~df.index.duplicated(keep="last")]
-
         cleaned = original_len - len(df)
         if cleaned > 0:
             logger.info(f"{symbol}: Cleaned {cleaned}/{original_len} bars")

main.py

@@ -1,509 +1,348 @@
-#!/usr/bin/env python3
+#!/usr/bin/env python3
-"""
+"""
-Forex Signal Intelligence Platform
+ForexBot2 — XAU/USD Signal Intelligence
-====================================
+========================================
-Main entry point. Runs full analysis and outputs formatted signal reports.
+Usage:
-
+    python main.py                              # Live signal on H1
-Usage:
+    python main.py --timeframe H4 --detail      # Detailed H4 signal
-    python main.py                          # Analyze all major pairs
+    python main.py --json                       # JSON output
-    python main.py --pairs EURUSD GBPUSD    # Specific pairs
+"""
-    python main.py --timeframe H4           # Different primary TF
+
-    python main.py --json                   # JSON output
+import argparse
-    python main.py --watch                  # Continuous monitoring mode
+import json
-"""
+import logging
-
+import sys
-import argparse
+from datetime import datetime, timezone
-import json
+from typing import Dict, List
-import logging
+
-import sys
+# Force UTF-8 output on Windows so box-drawing chars render
-import time
+if hasattr(sys.stdout, "reconfigure"):
-from datetime import datetime, timezone
+    sys.stdout.reconfigure(encoding="utf-8")
-from typing import List
+if hasattr(sys.stderr, "reconfigure"):
-
+    sys.stderr.reconfigure(encoding="utf-8")
-# Force UTF-8 output on Windows so box-drawing / Unicode chars render correctly
+
-if hasattr(sys.stdout, "reconfigure"):
+from config import PlatformConfig, Timeframe, SignalStrength, XAUUSD, EURUSD
-    sys.stdout.reconfigure(encoding="utf-8")
+from pair_signals import PAIR_PRODUCERS
-if hasattr(sys.stderr, "reconfigure"):
+from signal_aggregator import SignalAggregator, SignalReport
-    sys.stderr.reconfigure(encoding="utf-8")
+
-
+
-from config import (
+# ══════════════════════════════════════════════════════════════════
-    PlatformConfig, Timeframe, SignalStrength,
+#  TERMINAL FORMATTING
-    MAJOR_PAIRS, CROSS_PAIRS,
+# ══════════════════════════════════════════════════════════════════
-)
+
-from signal_aggregator import SignalAggregator, SignalReport
+class Colors:
-
+    RESET = "\033[0m"
-
+    BOLD = "\033[1m"
-# ══════════════════════════════════════════════════════════════════
+    DIM = "\033[2m"
-#  TERMINAL FORMATTING
+    UNDERLINE = "\033[4m"
-# ══════════════════════════════════════════════════════════════════
+    RED = "\033[91m"
-
+    GREEN = "\033[92m"
-class Colors:
+    YELLOW = "\033[93m"
-    """ANSI color codes for terminal output."""
+    BLUE = "\033[94m"
-    RESET = "\033[0m"
+    MAGENTA = "\033[95m"
-    BOLD = "\033[1m"
+    CYAN = "\033[96m"
-    DIM = "\033[2m"
+    WHITE = "\033[97m"
-    UNDERLINE = "\033[4m"
+    GRAY = "\033[90m"
-
+
-    RED = "\033[91m"
+
-    GREEN = "\033[92m"
+def signal_color(signal: SignalStrength) -> str:
-    YELLOW = "\033[93m"
+    return {
-    BLUE = "\033[94m"
+        SignalStrength.STRONG_BUY:  Colors.GREEN + Colors.BOLD,
-    MAGENTA = "\033[95m"
+        SignalStrength.BUY:         Colors.GREEN,
-    CYAN = "\033[96m"
+        SignalStrength.NEUTRAL:     Colors.GRAY,
-    WHITE = "\033[97m"
+        SignalStrength.SELL:        Colors.RED,
-    GRAY = "\033[90m"
+        SignalStrength.STRONG_SELL: Colors.RED + Colors.BOLD,
-
+    }.get(signal, Colors.RESET)
-    BG_RED = "\033[41m"
+
-    BG_GREEN = "\033[42m"
+
-    BG_YELLOW = "\033[43m"
+def quality_color(q: str) -> str:
-    BG_BLUE = "\033[44m"
+    return {
-    BG_MAGENTA = "\033[45m"
+        "A+": Colors.GREEN + Colors.BOLD,
-    BG_CYAN = "\033[46m"
+        "A":  Colors.GREEN,
-
+        "B":  Colors.CYAN,
-
+        "C":  Colors.YELLOW,
-def signal_color(signal: SignalStrength) -> str:
+        "D":  Colors.GRAY,
-    """Get color for a signal strength."""
+        "F":  Colors.DIM,
-    return {
+    }.get(q, Colors.RESET)
-        SignalStrength.STRONG_BUY: Colors.GREEN + Colors.BOLD,
+
-        SignalStrength.BUY: Colors.GREEN,
+
-        SignalStrength.NEUTRAL: Colors.GRAY,
+def print_header(title: str = "XAU/USD SIGNAL INTELLIGENCE"):
-        SignalStrength.SELL: Colors.RED,
+    now = datetime.now(timezone.utc).strftime("%Y-%m-%d %H:%M:%S UTC")
-        SignalStrength.STRONG_SELL: Colors.RED + Colors.BOLD,
+    print(f"\n{Colors.CYAN}{'═' * 80}{Colors.RESET}")
-    }.get(signal, Colors.RESET)
+    print(f"{Colors.BOLD}{Colors.CYAN}  {title}{Colors.RESET}")
-
+    print(f"{Colors.GRAY}  Time: {now}{Colors.RESET}")
-
+    print(f"{Colors.CYAN}{'═' * 80}{Colors.RESET}\n")
-def quality_color(quality: str) -> str:
+
-    """Get color for signal quality grade."""
+
-    return {
+# ── Live signal renderer ───────────────────────────────────────────
-        "A+": Colors.GREEN + Colors.BOLD,
+
-        "A": Colors.GREEN,
+_INDICATOR_PREFIXES = (
-        "B": Colors.CYAN,
+    "EMA Trend", "RSI", "Bollinger Bands", "Divergence",
-        "C": Colors.YELLOW,
+    "Candlestick", "ADX", "ATR", "News", "Regime",
-        "D": Colors.GRAY,
+)
-        "F": Colors.DIM,
+
-    }.get(quality, Colors.RESET)
+
-
+def _compact_reason(reason: str) -> str:
-
+    """Shorten a verbose reason string for the compact view."""
-def score_bar(score: float, width: int = 30) -> str:
+    s = reason
-    """Create a visual score bar from -100 to +100."""
+    if s.startswith("STRONG: "):
-    normalized = (score + 100) / 200  # 0 to 1
+        s = s[8:]
-    filled = int(normalized * width)
+    # Strip the leading "<Indicator>: " prefix — it's always an indicator name
-    center = width // 2
+    for prefix in _INDICATOR_PREFIXES:
-
+        if s.startswith(prefix + ": "):
-    bar = ""
+            s = s[len(prefix) + 2:]
-    for i in range(width):
+            break
-        if i == center:
+    # Drop trailing detail after first ' | '
-            bar += "│"
+    if " | " in s:
-        elif (score >= 0 and center < i <= filled) or \
+        s = s.split(" | ", 1)[0]
-             (score < 0 and filled <= i < center):
+    return s.strip()
-            bar += "█"
+
-        else:
+
-            bar += "░"
+def _fmt_price(price: float) -> str:
-
+    """2dp for gold-scale prices, 5dp for FX-scale (EUR/USD etc.)."""
-    if score >= 0:
+    return f"{price:.2f}" if price >= 10 else f"{price:.5f}"
-        color = Colors.GREEN if score > 20 else Colors.GRAY
+
-    else:
+
-        color = Colors.RED if score < -20 else Colors.GRAY
+def print_signal_report(report: SignalReport, detail: bool = False):
-
+    sc = signal_color(report.final_signal)
-    return f"{color}{bar}{Colors.RESET}"
+    qc = quality_color(report.signal_quality)
-
+    sig_name = report.final_signal.name.replace("_", " ")
-
+
-def print_header():
+    # ── 1. DECISION ────────────────────────────────────────────────
-    """Print the platform header."""
+    mtf_tag = (f"{Colors.GREEN}MTF✓{Colors.RESET}" if report.mtf_agreement
-    now = datetime.now(timezone.utc).strftime("%Y-%m-%d %H:%M:%S UTC")
+               else f"{Colors.YELLOW}MTF✗{Colors.RESET}")
-    width = 80
+    print(f"  {sc}▶ {sig_name}{Colors.RESET}  "
-
+          f"{qc}{report.signal_quality}{Colors.RESET} · "
-    print(f"\n{Colors.CYAN}{'═' * width}{Colors.RESET}")
+          f"{sc}{report.final_score:+.1f}{Colors.RESET} · "
-    print(f"{Colors.BOLD}{Colors.CYAN}  ╔═══════════════════════════════════════════════════════╗{Colors.RESET}")
+          f"{report.final_confidence:.0f}% · {mtf_tag}")
-    print(f"{Colors.BOLD}{Colors.CYAN}  ║   FOREX SIGNAL INTELLIGENCE PLATFORM                 ║{Colors.RESET}")
+    print(f"  {Colors.GRAY}{report.action_recommendation}{Colors.RESET}")
-    print(f"{Colors.BOLD}{Colors.CYAN}  ║   Multi-Layer Signal Analysis Engine                  ║{Colors.RESET}")
+
-    print(f"{Colors.BOLD}{Colors.CYAN}  ╚═══════════════════════════════════════════════════════╝{Colors.RESET}")
+    # ── 2. TRADE SETUP (if signal is actionable) ───────────────────
-    print(f"{Colors.GRAY}  Scan Time: {now}{Colors.RESET}")
+    if report.trade_setup:
-    print(f"{Colors.CYAN}{'═' * width}{Colors.RESET}\n")
+        ts = report.trade_setup
-
+        dc = Colors.GREEN if ts.direction == "BUY" else Colors.RED
-
+        print(f"\n  {Colors.BOLD}Setup{Colors.RESET}  "
-def print_summary_table(reports: List[SignalReport]):
+              f"{dc}{ts.direction}{Colors.RESET} @ {Colors.BOLD}{_fmt_price(ts.entry_price)}{Colors.RESET}  "
-    """Print a summary table of all pairs."""
+              f"→ SL {Colors.RED}{_fmt_price(ts.stop_loss)}{Colors.RESET} "
-    print(f"{Colors.BOLD}{Colors.WHITE}  ┌─────────────┬───────────┬──────────────────────────────────┬────────┬────────────┐{Colors.RESET}")
+              f"({ts.risk_pips:.1f}p)  "
-    print(f"{Colors.BOLD}{Colors.WHITE}  │ Pair        │ Signal    │ Score                            │ Grade  │ Confidence │{Colors.RESET}")
+              f"→ TP {Colors.GREEN}{_fmt_price(ts.take_profit_1)}{Colors.RESET} "
-    print(f"{Colors.BOLD}{Colors.WHITE}  ├─────────────┼───────────┼──────────────────────────────────┼────────┼────────────┤{Colors.RESET}")
+              f"({ts.reward_pips:.1f}p)  "
-
+              f"RR 1:{ts.risk_reward:.1f}")
-    for r in reports:
+
-        sig = r.final_signal.name.replace("_", " ")
+    # ── 3. CONTEXT (price · regime · vol · MTF breakdown) ──────────
-        sc = signal_color(r.final_signal)
+    print(f"\n  {Colors.WHITE}{report.display_name}{Colors.RESET} "
-        qc = quality_color(r.signal_quality)
+          f"{Colors.BOLD}{_fmt_price(report.current_price)}{Colors.RESET} · "
-        bar = score_bar(r.final_score, 30)
+          f"{Colors.CYAN}{report.market_regime}{Colors.RESET} · "
-
+          f"vol {report.volatility_regime}")
-        pair = f"{r.display_name:<11}"
+
-        signal_str = f"{sc}{sig:<9}{Colors.RESET}"
+    if report.mtf_signals:
-        grade_str = f"{qc}{r.signal_quality:<6}{Colors.RESET}"
+        parts = []
-        conf_str = f"{r.final_confidence:5.1f}%"
+        for tf, sig in report.mtf_signals.items():
-
+            tf_sc = signal_color(sig)
-        # Add warning indicator
+            sc_val = report.mtf_scores.get(tf, 0)
-        warn = ""
+            parts.append(f"{tf} {tf_sc}{sig.name.replace('_',' ')}{Colors.RESET} {sc_val:+.1f}")
-        if r.event_risk == "danger":
+        print(f"  {Colors.GRAY}MTF{Colors.RESET}    " + "  ·  ".join(parts))
-            warn = f" {Colors.RED}⚠{Colors.RESET}"
+
-        elif r.event_risk == "caution":
+    # ── 4. WHY — top 3 each side, compact ──────────────────────────
-            warn = f" {Colors.YELLOW}△{Colors.RESET}"
+    bulls = [_compact_reason(r) for r in report.bullish_reasons[:3]]
-
+    bears = [_compact_reason(r) for r in report.bearish_reasons[:3]]
-        print(f"  │ {pair} │ {signal_str} │ {bar} │ {grade_str} │ {conf_str:>8}{warn}  │")
+    if bulls or bears:
-
+        print()
-    print(f"{Colors.BOLD}{Colors.WHITE}  └─────────────┴───────────┴──────────────────────────────────┴────────┴────────────┘{Colors.RESET}")
+        rows = max(len(bulls), len(bears))
-
+        for i in range(rows):
-    # Legend
+            left  = f"{Colors.GREEN}+{Colors.RESET} {bulls[i]}" if i < len(bulls) else ""
-    print(f"\n{Colors.GRAY}  Legend: "
+            right = f"{Colors.RED}-{Colors.RESET} {bears[i]}" if i < len(bears) else ""
-          f"{Colors.GREEN}██ Bullish{Colors.GRAY} | "
+            print(f"  {left:<60}  {right}")
-          f"{Colors.RED}██ Bearish{Colors.GRAY} | "
+
-          f"│ Center (0) | "
+    # ── 5. WARNINGS / NEWS / EVENTS ────────────────────────────────
-          f"{Colors.RED}⚠ Event Risk{Colors.GRAY} | "
+    if report.warnings:
-          f"{Colors.YELLOW}△ Caution{Colors.RESET}")
+        for w in report.warnings:
-
+            print(f"  {Colors.YELLOW}⚠{Colors.RESET}  {w}")
-
+    if report.event_description and report.event_risk != "clear":
-def print_detailed_report(report: SignalReport):
+        print(f"  {Colors.YELLOW}📅{Colors.RESET} {report.event_description}")
-    """Print detailed analysis for a single pair."""
+    if report.news_description and report.news_description != "Neutral news sentiment":
-    sc = signal_color(report.final_signal)
+        print(f"  {Colors.MAGENTA}📰{Colors.RESET} {report.news_description}")
-    qc = quality_color(report.signal_quality)
+
-    sig_name = report.final_signal.name.replace("_", " ")
+    # ── 6. DETAIL (only with --detail) ─────────────────────────────
-
+    if detail and report.primary_analysis:
-    print(f"\n{Colors.CYAN}{'─' * 80}{Colors.RESET}")
+        print(f"\n  {Colors.BOLD}Indicators ({report.primary_analysis.timeframe}){Colors.RESET}")
-    print(f"  {Colors.BOLD}{Colors.WHITE}{report.display_name}{Colors.RESET}"
+        for name, ind in report.primary_analysis.indicators.items():
-          f"  │  Price: {Colors.BOLD}{report.current_price:.5f}{Colors.RESET}"
+            if name.startswith("_") or ind.weight == 0:
-          f"  │  Regime: {Colors.CYAN}{report.market_regime}{Colors.RESET}"
+                continue
-          f"  │  Vol: {report.volatility_regime}")
+            isc = signal_color(ind.signal)
-
+            print(f"    {ind.name:<16} {isc}{ind.signal.name.replace('_',' '):<11}{Colors.RESET} "
-    print(f"\n  {Colors.BOLD}SIGNAL:{Colors.RESET} {sc}{sig_name}{Colors.RESET}"
+                  f"{ind.confidence:>3.0f}% {Colors.GRAY}│ {ind.description}{Colors.RESET}")
-          f"  │  Score: {sc}{report.final_score:+.1f}{Colors.RESET}"
+
-          f"  │  Grade: {qc}{report.signal_quality}{Colors.RESET}"
+        levels = report.primary_analysis.key_levels
-          f"  │  Confidence: {report.final_confidence:.0f}%")
+        if levels:
-
+            swing_keys = sorted(
-    print(f"  {Colors.BOLD}Action:{Colors.RESET} {report.action_recommendation}")
+                [k for k in levels if k.startswith("swing_")],
-
+                key=lambda k: levels[k], reverse=True,
-    # Multi-timeframe breakdown
+            )
-    if report.mtf_signals:
+            if swing_keys:
-        print(f"\n  {Colors.BOLD}Multi-Timeframe:{Colors.RESET}", end="")
+                print(f"\n  {Colors.BOLD}Structure Levels{Colors.RESET}")
-        mtf_agree = f" {Colors.GREEN}✓ ALIGNED{Colors.RESET}" if report.mtf_agreement else f" {Colors.YELLOW}✗ DIVERGENT{Colors.RESET}"
+                for k in swing_keys:
-        print(mtf_agree)
+                    val = levels[k]
-        for tf, sig in report.mtf_signals.items():
+                    color = Colors.RED if val > report.current_price else Colors.GREEN
-            tf_sc = signal_color(sig)
+                    label = k.replace("_", " ").title()
-            tf_score = report.mtf_scores.get(tf, 0)
+                    print(f"    {label:>20}  {color}{val:.3f}{Colors.RESET}")
-            print(f"    {tf:>5}: {tf_sc}{sig.name.replace('_', ' '):<12}{Colors.RESET} ({tf_score:+.1f})")
+
-
+
-    # Indicators detail
+def report_to_dict(report: SignalReport) -> dict:
-    if report.primary_analysis:
+    d = {
-        print(f"\n  {Colors.BOLD}Indicators ({report.primary_analysis.timeframe}):{Colors.RESET}")
+        "symbol": report.symbol,
-        for name, ind in report.primary_analysis.indicators.items():
+        "display_name": report.display_name,
-            if name.startswith("_") or ind.weight == 0:
+        "timestamp": report.timestamp,
-                continue
+        "current_price": report.current_price,
-            isc = signal_color(ind.signal)
+        "signal": report.final_signal.name,
-            ind_name = f"{ind.name:<18}"
+        "score": round(report.final_score, 2),
-            print(f"    {ind_name} {isc}{ind.signal.name.replace('_', ' '):<12}{Colors.RESET}"
+        "confidence": round(report.final_confidence, 2),
-                  f" ({ind.confidence:.0f}%) │ {Colors.GRAY}{ind.description}{Colors.RESET}")
+        "quality": report.signal_quality,
-
+        "action": report.action_recommendation,
-    # Bullish reasons
+        "market_regime": report.market_regime,
-    if report.bullish_reasons:
+        "volatility_regime": report.volatility_regime,
-        print(f"\n  {Colors.GREEN}▲ Bullish Factors:{Colors.RESET}")
+        "mtf_agreement": report.mtf_agreement,
-        for reason in report.bullish_reasons[:5]:
+        "mtf_signals": {k: v.name for k, v in report.mtf_signals.items()},
-            print(f"    {Colors.GREEN}+{Colors.RESET} {reason}")
+        "event_risk": report.event_risk,
-
+        "event_description": report.event_description,
-    # Bearish reasons
+        "news_sentiment": round(report.news_sentiment, 3),
-    if report.bearish_reasons:
+        "news_description": report.news_description,
-        print(f"\n  {Colors.RED}▼ Bearish Factors:{Colors.RESET}")
+        "bullish_reasons": report.bullish_reasons,
-        for reason in report.bearish_reasons[:5]:
+        "bearish_reasons": report.bearish_reasons,
-            print(f"    {Colors.RED}-{Colors.RESET} {reason}")
+        "warnings": report.warnings,
-
+    }
-    # Warnings
+    if report.trade_setup:
-    if report.warnings:
+        ts = report.trade_setup
-        print(f"\n  {Colors.YELLOW}⚠ Warnings:{Colors.RESET}")
+        d["trade_setup"] = {
-        for warning in report.warnings:
+            "direction": ts.direction,
-            print(f"    {Colors.YELLOW}!{Colors.RESET} {warning}")
+            "entry": ts.entry_price,
-
+            "stop_loss": ts.stop_loss,
-    # News context
+            "take_profit_1": ts.take_profit_1,
-    if report.news_description and report.news_description != "Neutral news sentiment":
+            "take_profit_2": ts.take_profit_2,
-        print(f"\n  {Colors.MAGENTA}📰 News:{Colors.RESET} {report.news_description}")
+            "risk_reward": ts.risk_reward,
-    if report.event_description and report.event_risk != "clear":
+            "risk_pips": round(ts.risk_pips, 1),
-        print(f"  {Colors.YELLOW}📅 Event:{Colors.RESET} {report.event_description}")
+            "reward_pips": round(ts.reward_pips, 1),
-
+        }
-    # Trade setup
+    if report.primary_analysis:
-    if report.trade_setup:
+        d["indicators"] = {
-        ts = report.trade_setup
+            name: {
-        dir_color = Colors.GREEN if ts.direction == "BUY" else Colors.RED
+                "signal": ind.signal.name,
-        print(f"\n  {Colors.BOLD}Trade Setup:{Colors.RESET}")
+                "confidence": round(ind.confidence, 1),
-        print(f"    Direction:     {dir_color}{ts.direction}{Colors.RESET}")
+                "description": ind.description,
-        print(f"    Entry:         {ts.entry_price:.5f}")
+            }
-        print(f"    Stop Loss:     {Colors.RED}{ts.stop_loss:.5f}{Colors.RESET} ({ts.risk_pips:.1f} pips)")
+            for name, ind in report.primary_analysis.indicators.items()
-        print(f"    Take Profit 1: {Colors.GREEN}{ts.take_profit_1:.5f}{Colors.RESET} ({ts.reward_pips:.1f} pips)")
+            if not name.startswith("_")
-        print(f"    Take Profit 2: {Colors.GREEN}{ts.take_profit_2:.5f}{Colors.RESET}")
+        }
-        print(f"    Risk:Reward:   1:{ts.risk_reward:.1f}")
+        d["key_levels"] = {
-
+            k: round(v, 5) for k, v in report.primary_analysis.key_levels.items()
-    # Key levels
+        }
-    if report.primary_analysis and report.primary_analysis.key_levels:
+    return d
-        levels = report.primary_analysis.key_levels
+
-        print(f"\n  {Colors.BOLD}Key Levels:{Colors.RESET}")
+
-        level_names = ["r3", "r2", "r1", "pivot", "s1", "s2", "s3"]
+
-        for lvl in level_names:
+# ══════════════════════════════════════════════════════════════════
-            if lvl in levels:
+#  CLI
-                val = levels[lvl]
+# ══════════════════════════════════════════════════════════════════
-                if val > report.current_price:
+
-                    color = Colors.RED  # resistance
+def parse_args():
-                else:
+    parser = argparse.ArgumentParser(
-                    color = Colors.GREEN  # support
+        description="ForexBot2 — XAU/USD Signal Intelligence",
-                label = lvl.upper()
+        formatter_class=argparse.RawDescriptionHelpFormatter,
-                print(f"    {label:>6}: {color}{val:.5f}{Colors.RESET}")
+        epilog="""
-
+Examples:
-
+  python main.py                          Live signal (H1, summary)
-def print_event_calendar(reports: List[SignalReport]):
+  python main.py --detail                 Live signal with full breakdown
-    """Print upcoming economic events affecting watched pairs."""
+  python main.py --timeframe H4 --detail  H4 timeframe
-    events_shown = set()
+  python main.py --json                   JSON output
-    has_events = False
+        """,
-
+    )
-    for report in reports:
+    parser.add_argument("--pair", "-p", default="XAUUSD",
-        if report.event_risk != "clear" and report.event_description not in events_shown:
+                        help="Pair to analyze: XAUUSD | EURUSD (default: XAUUSD)")
-            if not has_events:
+    parser.add_argument("--timeframe", "-tf", default="H1",
-                print(f"\n{Colors.YELLOW}{Colors.BOLD}  📅 ECONOMIC CALENDAR ALERTS{Colors.RESET}")
+                        choices=["M5", "M15", "M30", "H1", "H4", "D1"],
-                has_events = True
+                        help="Primary analysis timeframe (default: H1)")
-            events_shown.add(report.event_description)
+    parser.add_argument("--detail", action="store_true",
-            risk_color = Colors.RED if report.event_risk == "danger" else Colors.YELLOW
+                        help="Show full indicator breakdown")
-            print(f"    {risk_color}● {report.event_description}{Colors.RESET} → affects {report.display_name}")
+    parser.add_argument("--json", action="store_true",
-
+                        help="Output as JSON")
-    if not has_events:
+    parser.add_argument("--verbose", "-v", action="store_true",
-        print(f"\n{Colors.GRAY}  📅 No high-impact events in the near term{Colors.RESET}")
+                        help="Debug logging")
-
+
-
+    return parser.parse_args()
-def report_to_dict(report: SignalReport) -> dict:
+
-    """Convert a SignalReport to a JSON-serializable dict."""
+
-    d = {
+def _resolve_pair(s: str) -> str:
-        "symbol": report.symbol,
+    aliases = {
-        "display_name": report.display_name,
+        "XAUUSD": XAUUSD, "XAU/USD": XAUUSD, "GOLD": XAUUSD,
-        "timestamp": report.timestamp,
+        "EURUSD": EURUSD, "EUR/USD": EURUSD, "EUR": EURUSD,
-        "current_price": report.current_price,
+    }
-        "signal": report.final_signal.name,
+    key = s.upper().strip()
-        "score": round(report.final_score, 2),
+    if key in aliases:
-        "confidence": round(report.final_confidence, 2),
+        return aliases[key]
-        "quality": report.signal_quality,
+    raise SystemExit(f"Unknown pair '{s}'. Try one of: {', '.join(sorted(aliases))}")
-        "action": report.action_recommendation,
+
-        "market_regime": report.market_regime,
+
-        "volatility_regime": report.volatility_regime,
+def build_config(args) -> PlatformConfig:
-        "mtf_agreement": report.mtf_agreement,
+    pair = _resolve_pair(args.pair)
-        "mtf_signals": {k: v.name for k, v in report.mtf_signals.items()},
+
-        "event_risk": report.event_risk,
+    tf_map = {
-        "event_description": report.event_description,
+        "M5": Timeframe.M5, "M15": Timeframe.M15, "M30": Timeframe.M30,
-        "news_sentiment": round(report.news_sentiment, 3),
+        "H1": Timeframe.H1, "H4": Timeframe.H4, "D1": Timeframe.D1,
-        "bullish_reasons": report.bullish_reasons,
+    }
-        "bearish_reasons": report.bearish_reasons,
+    primary_tf = tf_map[args.timeframe]
-        "warnings": report.warnings,
+
-    }
+    tf_confirmations = {
-    if report.trade_setup:
+        "M5":  [Timeframe.M1,  Timeframe.M15],
-        ts = report.trade_setup
+        "M15": [Timeframe.M5,  Timeframe.H1],
-        d["trade_setup"] = {
+        "M30": [Timeframe.M15, Timeframe.H1],
-            "direction": ts.direction,
+        "H1":  [Timeframe.M15, Timeframe.H4],
-            "entry": ts.entry_price,
+        "H4":  [Timeframe.H1,  Timeframe.D1],
-            "stop_loss": ts.stop_loss,
+        "D1":  [Timeframe.H4,  Timeframe.W1],
-            "take_profit_1": ts.take_profit_1,
+    }
-            "take_profit_2": ts.take_profit_2,
+    confirmation_tfs = tf_confirmations.get(
-            "risk_reward": ts.risk_reward,
+        args.timeframe, [Timeframe.M15, Timeframe.H4]
-            "risk_pips": round(ts.risk_pips, 1),
+    )
-            "reward_pips": round(ts.reward_pips, 1),
+
-        }
+    # Per-pair tuned configs from the pair_signals factories.
-    if report.primary_analysis:
+    factory = PAIR_PRODUCERS.get(pair)
-        d["indicators"] = {}
+    if factory is not None:
-        for name, ind in report.primary_analysis.indicators.items():
+        return factory(primary_tf, confirmation_tfs)
-            if not name.startswith("_"):
+
-                d["indicators"][name] = {
+    config = PlatformConfig()
-                    "signal": ind.signal.name,
+    config.primary_timeframe = primary_tf
-                    "confidence": round(ind.confidence, 1),
+    config.confirmation_timeframes = confirmation_tfs
-                    "description": ind.description,
+    return config
-                }
+
-        d["key_levels"] = {
+
-            k: round(v, 5) for k, v in report.primary_analysis.key_levels.items()
+def run_live(args, config: PlatformConfig):
-        }
+    print(f"{Colors.GRAY}  Fetching market data and running analysis...{Colors.RESET}\n")
-    return d
+    aggregator = SignalAggregator(config)
-
+    report = aggregator.analyze_pair(symbol=config.symbol)
-
+
-# ══════════════════════════════════════════════════════════════════
+    if args.json:
-#  MAIN
+        print(json.dumps(report_to_dict(report), indent=2))
-# ══════════════════════════════════════════════════════════════════
+        return
-
+
-def parse_args():
+    print_header(f"{config.display_name} SIGNAL INTELLIGENCE")
-    parser = argparse.ArgumentParser(
+    print_signal_report(report, detail=args.detail)
-        description="Forex Signal Intelligence Platform",
+    print(f"\n{Colors.CYAN}{'═' * 80}{Colors.RESET}")
-        formatter_class=argparse.RawDescriptionHelpFormatter,
+    print(f"{Colors.GRAY}  Disclaimer: For educational/research purposes only.{Colors.RESET}")
-        epilog="""
+    print(f"{Colors.CYAN}{'═' * 80}{Colors.RESET}\n")
-Examples:
+
-  python main.py                        Analyze all major pairs (H1)
+
-  python main.py --pairs EURUSD GBPUSD  Analyze specific pairs
+def main():
-  python main.py --timeframe H4         Use 4-hour as primary TF
+    args = parse_args()
-  python main.py --all                  Include cross pairs
+
-  python main.py --json                 Output as JSON
+    log_level = logging.DEBUG if args.verbose else logging.WARNING
-  python main.py --detail               Show detailed analysis per pair
+    logging.basicConfig(
-  python main.py --watch                Continuous monitoring (5min refresh)
+        level=log_level,
-        """,
+        format="%(asctime)s [%(levelname)s] %(name)s: %(message)s",
-    )
+        datefmt="%H:%M:%S",
-    parser.add_argument(
+    )
-        "--pairs", nargs="+", default=None,
+
-        help="Specific pairs to analyze (e.g., EURUSD GBPUSD)",
+    config = build_config(args)
-    )
+    run_live(args, config)
-    parser.add_argument(
+
-        "--timeframe", "-tf", default="H1",
+
-        choices=["M5", "M15", "M30", "H1", "H4", "D1"],
+if __name__ == "__main__":
-        help="Primary analysis timeframe (default: H1)",
+    main()
-    )
-    parser.add_argument(
-        "--all", action="store_true",
-        help="Include cross pairs in analysis",
-    )
-    parser.add_argument(
-        "--json", action="store_true",
-        help="Output results as JSON",
-    )
-    parser.add_argument(
-        "--detail", action="store_true",
-        help="Show detailed analysis for each pair",
-    )
-    parser.add_argument(
-        "--top", type=int, default=None,
-        help="Show only top N signals by strength",
-    )
-    parser.add_argument(
-        "--watch", action="store_true",
-        help="Continuous monitoring mode (refreshes every 5 min)",
-    )
-    parser.add_argument(
-        "--min-grade", default="F",
-        choices=["A+", "A", "B", "C", "D", "F"],
-        help="Minimum signal quality to display",
-    )
-    parser.add_argument(
-        "--verbose", "-v", action="store_true",
-        help="Enable debug logging",
-    )
-    return parser.parse_args()
-
-
-def symbol_from_name(name: str) -> str:
-    """Convert pair name like 'EURUSD' to Yahoo symbol 'EURUSD=X'."""
-    name = name.upper().replace("/", "").replace("_", "")
-    if not name.endswith("=X"):
-        name += "=X"
-    return name
-
-
-def run_analysis(args) -> List[SignalReport]:
-    """Run a single analysis cycle."""
-    config = PlatformConfig()
-
-    # Configure timeframe
-    tf_map = {
-        "M5": Timeframe.M5, "M15": Timeframe.M15, "M30": Timeframe.M30,
-        "H1": Timeframe.H1, "H4": Timeframe.H4, "D1": Timeframe.D1,
-    }
-    config.primary_timeframe = tf_map[args.timeframe]
-
-    # Configure confirmation timeframes based on primary
-    tf_confirmations = {
-        "M5": [Timeframe.M1, Timeframe.M15],
-        "M15": [Timeframe.M5, Timeframe.H1],
-        "M30": [Timeframe.M15, Timeframe.H1],
-        "H1": [Timeframe.M15, Timeframe.H4],
-        "H4": [Timeframe.H1, Timeframe.D1],
-        "D1": [Timeframe.H4, Timeframe.W1],
-    }
-    config.confirmation_timeframes = tf_confirmations.get(
-        args.timeframe, [Timeframe.M15, Timeframe.H4]
-    )
-
-    # Configure pairs
-    if args.pairs:
-        config.watch_pairs = [symbol_from_name(p) for p in args.pairs]
-    elif args.all:
-        config.watch_pairs = MAJOR_PAIRS + CROSS_PAIRS
-    else:
-        config.watch_pairs = MAJOR_PAIRS
-
-    # Run analysis
-    aggregator = SignalAggregator(config)
-    reports = aggregator.analyze_all_pairs()
-
-    # Filter by grade
-    grade_order = ["A+", "A", "B", "C", "D", "F"]
-    min_idx = grade_order.index(args.min_grade) if args.min_grade in grade_order else len(grade_order) - 1
-    reports = [r for r in reports if grade_order.index(r.signal_quality) <= min_idx]
-
-    # Top N
-    if args.top:
-        reports = reports[:args.top]
-
-    return reports
-
-
-def main():
-    args = parse_args()
-
-    # Setup logging
-    log_level = logging.DEBUG if args.verbose else logging.WARNING
-    logging.basicConfig(
-        level=log_level,
-        format="%(asctime)s [%(levelname)s] %(name)s: %(message)s",
-        datefmt="%H:%M:%S",
-    )
-
-    if args.watch:
-        # Continuous monitoring mode
-        print(f"{Colors.BOLD}Starting continuous monitoring mode (Ctrl+C to stop)...{Colors.RESET}")
-        while True:
-            try:
-                reports = run_analysis(args)
-                if args.json:
-                    output = [report_to_dict(r) for r in reports]
-                    print(json.dumps(output, indent=2))
-                else:
-                    print("\033[2J\033[H")  # clear screen
-                    print_header()
-                    print_summary_table(reports)
-                    print_event_calendar(reports)
-                    if args.detail:
-                        for r in reports:
-                            print_detailed_report(r)
-                    print(f"\n{Colors.GRAY}  Next refresh in 5 minutes...{Colors.RESET}")
-
-                time.sleep(300)
-            except KeyboardInterrupt:
-                print(f"\n{Colors.YELLOW}Monitoring stopped.{Colors.RESET}")
-                break
-    else:
-        # Single run
-        print(f"{Colors.GRAY}  Fetching market data and running analysis...{Colors.RESET}\n")
-        reports = run_analysis(args)
-
-        if args.json:
-            output = [report_to_dict(r) for r in reports]
-            print(json.dumps(output, indent=2))
-        else:
-            print_header()
-            if reports:
-                print_summary_table(reports)
-                print_event_calendar(reports)
-
-                if args.detail:
-                    for r in reports:
-                        print_detailed_report(r)
-                else:
-                    # Show detail for top signals only
-                    actionable = [r for r in reports if r.signal_quality in ("A+", "A", "B")]
-                    if actionable:
-                        print(f"\n{Colors.BOLD}{Colors.WHITE}  ═══ ACTIONABLE SIGNALS (Grade B or above) ═══{Colors.RESET}")
-                        for r in actionable:
-                            print_detailed_report(r)
-                    else:
-                        print(f"\n{Colors.GRAY}  No high-grade signals detected. "
-                              f"Use --detail for full breakdown or --min-grade C to see more.{Colors.RESET}")
-            else:
-                print(f"{Colors.YELLOW}  No data available. Check your internet connection.{Colors.RESET}")
-
-            print(f"\n{Colors.CYAN}{'═' * 80}{Colors.RESET}")
-            print(f"{Colors.GRAY}  Disclaimer: This tool is for educational/research purposes only.{Colors.RESET}")
-            print(f"{Colors.GRAY}  Past performance ≠ future results. Always use proper risk management.{Colors.RESET}")
-            print(f"{Colors.CYAN}{'═' * 80}{Colors.RESET}\n")
-
-
-if __name__ == "__main__":
-    main()

news_engine.py

@@ -9,6 +9,9 @@ A perfect technical setup can be destroyed by an unexpected NFP print.
 """
 
 import logging
+import re
+import threading
+import time
 import xml.etree.ElementTree as ET
 from dataclasses import dataclass, field
 from datetime import datetime, timedelta, timezone
@@ -21,6 +24,21 @@ from config import NewsConfig
 logger = logging.getLogger(__name__)
 
 
+_RELEASE_VALUE_RE = re.compile(r"(-?\d+(?:\.\d+)?)\s*([kmbt])?", re.IGNORECASE)
+_RELEASE_MULTIPLIERS = {"k": 1e3, "m": 1e6, "b": 1e9, "t": 1e12}
+
+
+def _parse_release_value(raw: Optional[str]) -> Optional[float]:
+    """Parse a calendar release value like '0.3%', '-1.7%', '228K', '1.1M'."""
+    if not raw:
+        return None
+    match = _RELEASE_VALUE_RE.search(raw.replace(",", ""))
+    if not match:
+        return None
+    multiplier = _RELEASE_MULTIPLIERS.get((match.group(2) or "").lower(), 1.0)
+    return float(match.group(1)) * multiplier
+
+
 @dataclass
 class EconomicEvent:
     """Single economic calendar event."""
@@ -31,7 +49,21 @@ class EconomicEvent:
     forecast: Optional[str] = None
     previous: Optional[str] = None
     actual: Optional[str] = None
-    surprise: Optional[str] = None  # "positive", "negative", "inline", None
+
+    @property
+    def surprise(self) -> Optional[str]:
+        """Actual vs forecast: "positive"/"negative"/"inline", or None when
+        either value is missing or non-numeric. Direction is numeric only —
+        it does not know whether higher is market-bullish for this event."""
+        actual = _parse_release_value(self.actual)
+        forecast = _parse_release_value(self.forecast)
+        if actual is None or forecast is None:
+            return None
+        if actual > forecast:
+            return "positive"
+        if actual < forecast:
+            return "negative"
+        return "inline"
 
     @property
     def is_upcoming(self) -> bool:
@@ -56,6 +88,9 @@ class NewsSentiment:
     currencies_mentioned: List[str]
     sentiment_score: float  # -1.0 (bearish) to +1.0 (bullish)
     keywords: List[str]
+    # Per-currency score: monetary-policy keywords flip sign for XAU
+    # (hawkish is dollar-bullish but gold-bearish).
+    sentiment_by_currency: Dict[str, float] = field(default_factory=dict)
 
 
 @dataclass
@@ -81,39 +116,74 @@ class NewsEngine:
     # Keywords that indicate directional sentiment
     BULLISH_KEYWORDS = {
         "rate hike", "hawkish", "strong growth", "beat expectations",
-        "above forecast", "surplus", "expansion", "employment gains",
+        "above forecast", "above estimate", "above expectation",
+        "surplus", "expansion", "employment gains",
         "strong jobs", "gdp growth", "inflation rising", "tightening",
         "positive surprise", "record high", "bullish", "rally",
         "stronger than expected", "upbeat",
+        "beats", "surges", "soars", "jumps", "climbs", "rises",
+        "higher than expected", "better than expected",
+        "strong data", "robust", "resilient", "accelerates",
     }
 
     BEARISH_KEYWORDS = {
         "rate cut", "dovish", "recession", "slowdown", "contraction",
-        "below forecast", "deficit", "miss expectations", "weak data",
+        "below forecast", "below estimate", "below expectation",
+        "deficit", "miss expectations", "weak data",
         "unemployment rise", "negative surprise", "bearish", "selloff",
         "weaker than expected", "downbeat", "easing", "stimulus",
+        "misses", "slides", "drops", "falls", "plunges", "tumbles",
+        "weakens", "slumps", "declines", "retreats",
+        "lower than expected", "worse than expected",
+        "disappointing", "soft data",
     }
 
-    # Currency-specific keywords for headline parsing
+    # Monetary-policy keywords whose direction inverts for gold: tighter
+    # policy lifts the dollar and yields, weighing on non-yielding gold;
+    # easing does the opposite. Only unambiguous policy words — economic
+    # data words (beats/misses) keep their normal direction.
+    GOLD_INVERSE_KEYWORDS = {
+        "rate hike", "hawkish", "tightening",
+        "rate cut", "dovish", "easing", "stimulus",
+    }
+
+    # Currency-specific keywords for headline parsing. Matched with word
+    # boundaries — bare "us"/"eu"/"uk" are deliberately excluded because as
+    # substrings they false-positive constantly ("versus" → USD).
     CURRENCY_KEYWORDS = {
-        "USD": ["dollar", "fed", "fomc", "treasury", "us ", "american", "nonfarm", "nfp"],
+        "USD": ["dollar", "fed", "federal reserve", "fomc", "treasury", "american", "nonfarm", "nfp"],
-        "EUR": ["euro", "ecb", "eurozone", "eu ", "german", "french"],
+        "EUR": ["euro", "ecb", "eurozone", "german", "french"],
-        "GBP": ["pound", "sterling", "boe", "bank of england", "uk ", "british"],
+        "GBP": ["pound", "sterling", "boe", "bank of england", "british"],
         "JPY": ["yen", "boj", "bank of japan", "japanese"],
         "CHF": ["franc", "snb", "swiss"],
         "AUD": ["aussie", "rba", "australian", "australia"],
         "CAD": ["loonie", "boc", "canadian", "canada"],
         "NZD": ["kiwi", "rbnz", "new zealand"],
+        "XAU": ["gold", "precious metal", "safe haven", "bullion"],
     }
 
-    # High-impact event types (these move markets significantly)
+    # One compiled alternation per currency (code + keywords). (?<![a-z]) /
-    HIGH_IMPACT_EVENTS = {
+    # (?![a-z]) instead of \b so "0.3%fed" still matches but "versus",
-        "interest rate decision", "nonfarm payrolls", "non-farm",
+    # "audit", "cascade" don't.
-        "gdp", "cpi", "inflation", "employment change",
+    _CURRENCY_PATTERNS = {
-        "retail sales", "pmi", "fomc", "ecb press conference",
+        cur: re.compile(
-        "boe rate", "rba rate", "snb rate", "boc rate",
+            r"(?<![a-z])(?:"
+            + "|".join(re.escape(kw) for kw in [cur.lower()] + kws)
+            + r")(?![a-z])"
+        )
+        for cur, kws in CURRENCY_KEYWORDS.items()
     }
 
+    _calendar_backoff_until: float = 0.0
+
+    # Fetched-data cache, class-level: a fresh NewsEngine is built per scan,
+    # so instance state would never be reused. Guarded by _cache_lock since
+    # scans run concurrently in a thread pool.
+    _cache_lock = threading.Lock()
+    _calendar_cache: Tuple[float, List[EconomicEvent]] = (0.0, [])
+    _headlines_cache: Tuple[float, List[NewsSentiment]] = (0.0, [])
+    _EMPTY_RESULT_TTL = 60.0  # retry sooner after a failed/empty fetch
+
     def __init__(self, config: NewsConfig):
         self.config = config
         self._session = requests.Session()
@@ -139,7 +209,13 @@ class NewsEngine:
             events = self._fetch_economic_calendar(currencies)
             now = datetime.now(timezone.utc)
 
-            context.events_today = [e for e in events if e.time.date() == now.date()]
+            # Compare dates in UTC — feed timestamps carry their own offset
+            # (FF sends US-Eastern), so a late-evening ET event would
+            # otherwise land on the wrong day.
+            context.events_today = [
+                e for e in events
+                if e.time.astimezone(timezone.utc).date() == now.date()
+            ]
             context.events_upcoming_1h = [
                 e for e in events
                 if 0 < e.minutes_until <= 60
@@ -173,10 +249,12 @@ class NewsEngine:
             headlines = self._fetch_news_sentiment()
             context.news_headlines = headlines
 
-            # Aggregate sentiment per currency
+            # Aggregate sentiment per currency (per-currency score where
+            # available, so gold-inverted policy keywords are respected)
             for currency in currencies:
                 scores = [
-                    h.sentiment_score for h in headlines
+                    h.sentiment_by_currency.get(currency, h.sentiment_score)
+                    for h in headlines
                     if currency in h.currencies_mentioned
                 ]
                 if scores:
@@ -187,15 +265,21 @@ class NewsEngine:
 
         return context
 
+    @staticmethod
+    def _parse_pair(pair_symbol: str) -> Tuple[str, str]:
+        if "/" in pair_symbol:
+            parts = pair_symbol.split("/")
+            return parts[0], parts[1]
+        clean = pair_symbol.replace("=X", "")
+        return clean[:3], clean[3:]
+
     def get_event_risk_level(self, pair_symbol: str, context: FundamentalContext) -> Tuple[str, str]:
         """
         Get event risk level for a specific pair.
         Returns (risk_level, description).
         risk_level: "clear", "caution", "danger"
         """
-        clean = pair_symbol.replace("=X", "")
+        base, quote = self._parse_pair(pair_symbol)
-        base = clean[:3]
-        quote = clean[3:]
 
         # Check upcoming high-impact events
         for event in context.events_upcoming_1h:
@@ -206,10 +290,24 @@ class NewsEngine:
                 else:
                     return "caution", f"{event.currency} {event.event} in {mins:.0f}min"
 
-        # Check for recent high-impact releases (within post-event buffer)
+        # Check for recent high-impact releases. A surprise print (actual
+        # away from forecast) keeps the market unstable longer than an
+        # inline release, so it gets a wider post-event window.
         for event in context.events_today:
-            if event.is_high_impact and event.currency in (base, quote):
+            if event.is_high_impact and event.currency in (base, quote) and not event.is_upcoming:
-                if not event.is_upcoming and event.minutes_until > -self.config.post_event_buffer_minutes:
+                surprise = event.surprise
+                surprised = surprise in ("positive", "negative")
+                window = (
+                    self.config.post_event_surprise_buffer_minutes
+                    if surprised
+                    else self.config.post_event_buffer_minutes
+                )
+                if event.minutes_until > -window:
+                    if surprised:
+                        return "caution", (
+                            f"{event.currency} {event.event} released — {surprise} surprise "
+                            f"(actual {event.actual} vs forecast {event.forecast})"
+                        )
                     return "caution", f"{event.currency} {event.event} just released"
 
         return "clear", "No significant events"
@@ -222,9 +320,7 @@ class NewsEngine:
         Returns (score, description) where score is -1 to +1.
         Positive = bullish for the pair, negative = bearish.
         """
-        clean = pair_symbol.replace("=X", "")
+        base, quote = self._parse_pair(pair_symbol)
-        base = clean[:3]
-        quote = clean[3:]
 
         base_sent = context.news_sentiment.get(base, 0.0)
         quote_sent = context.news_sentiment.get(quote, 0.0)
@@ -247,18 +343,52 @@ class NewsEngine:
 
     # ── Private Methods ────────────────────────────────────────────
 
+    # Maps country names in MyFXBook titles to currency codes
+    _COUNTRY_TO_CURRENCY = {
+        "united states": "USD", "euro area": "EUR", "united kingdom": "GBP",
+        "japan": "JPY", "switzerland": "CHF", "australia": "AUD",
+        "canada": "CAD", "new zealand": "NZD",
+    }
+
+    def _cached_fetch(self, cache_attr: str, fetch_fn) -> list:
+        """Return cached data for `cache_attr` or refetch when the TTL
+        expired. Successful results live for context_cache_ttl_seconds;
+        empty results only _EMPTY_RESULT_TTL so outages retry quickly.
+        The lock is held across the fetch so concurrent scans share one
+        request instead of stampeding the source."""
+        ts, data = getattr(NewsEngine, cache_attr)
+        ttl = self.config.context_cache_ttl_seconds if data else self._EMPTY_RESULT_TTL
+        if time.time() - ts < ttl:
+            return data
+
+        with NewsEngine._cache_lock:
+            ts, data = getattr(NewsEngine, cache_attr)  # re-check under lock
+            ttl = self.config.context_cache_ttl_seconds if data else self._EMPTY_RESULT_TTL
+            if time.time() - ts < ttl:
+                return data
+            data = fetch_fn()
+            setattr(NewsEngine, cache_attr, (time.time(), data))
+            return data
+
     def _fetch_economic_calendar(self, currencies: List[str]) -> List[EconomicEvent]:
+        """Cached calendar fetch, filtered to the tracked currencies."""
+        events = self._cached_fetch("_calendar_cache", self._fetch_calendar_raw)
+        return [e for e in events if e.currency in currencies]
+
+    def _fetch_calendar_raw(self) -> List[EconomicEvent]:
         """
         Fetch economic calendar events from a free API.
+        Tries Forex Factory first, falls back to MyFXBook RSS.
         Returns an empty list (with a log warning) if unavailable —
         we never generate fake events since that causes constant false alarms.
         """
-        try:
+        for fetcher in (self._try_fetch_calendar_api, self._try_fetch_myfxbook_rss):
-            events = self._try_fetch_calendar_api()
+            try:
-            if events:
+                events = fetcher()
-                return [e for e in events if e.currency in currencies]
+                if events:
-        except Exception as e:
+                    return events
-            logger.debug(f"Calendar API fetch exception: {e}")
+            except Exception as e:
+                logger.debug(f"Calendar fetch exception ({fetcher.__name__}): {e}")
 
         logger.info(
             "Economic calendar data unavailable. "
@@ -268,41 +398,144 @@ class NewsEngine:
 
     def _try_fetch_calendar_api(self) -> List[EconomicEvent]:
         """Attempt to fetch from the Forex Factory free JSON calendar."""
-        events = []
+        now = time.time()
+        if now < NewsEngine._calendar_backoff_until:
+            wait = NewsEngine._calendar_backoff_until - now
+            logger.debug(f"Calendar API in backoff for {wait:.0f}s more")
+            return []
+
         url = "https://nfs.faireconomy.media/ff_calendar_thisweek.json"
-        resp = self._session.get(url, timeout=10)
+
+        for attempt in range(3):
+            try:
+                resp = self._session.get(url, timeout=10)
+            except requests.RequestException as e:
+                logger.debug(f"Calendar request failed (attempt {attempt + 1}): {e}")
+                if attempt < 2:
+                    time.sleep(2 ** attempt)
+                continue
+
+            if resp.status_code == 429:
+                retry_after = int(resp.headers.get("Retry-After", 300))
+                retry_after = min(retry_after, 600)
+                NewsEngine._calendar_backoff_until = time.time() + retry_after
+                logger.warning(
+                    f"Calendar API rate-limited (429), backing off {retry_after}s"
+                )
+                return []
+
+            if resp.status_code != 200:
+                logger.debug(f"Calendar API returned {resp.status_code}")
+                return []
+
+            data = resp.json()
+            events = []
+            impact_map = {"High": "high", "Medium": "medium", "Low": "low"}
+
+            for item in data:
+                try:
+                    impact = impact_map.get(item.get("impact", ""), "low")
+
+                    date_str = item.get("date", "")
+                    if not date_str:
+                        continue
+                    event_time = datetime.fromisoformat(date_str.replace("Z", "+00:00"))
+
+                    events.append(EconomicEvent(
+                        time=event_time,
+                        currency=item.get("country", ""),
+                        impact=impact,
+                        event=item.get("title", ""),
+                        forecast=item.get("forecast") or None,
+                        previous=item.get("previous") or None,
+                        actual=item.get("actual") or None,
+                    ))
+                except Exception:
+                    continue
+
+            return events
+
+        return []
+
+    def _try_fetch_myfxbook_rss(self) -> List[EconomicEvent]:
+        """Fallback calendar source: MyFXBook economic calendar RSS."""
+        url = "https://www.myfxbook.com/rss/forex-economic-calendar-events"
+        try:
+            resp = self._session.get(url, timeout=10)
+        except requests.RequestException as e:
+            logger.debug(f"MyFXBook calendar request failed: {e}")
+            return []
+
         if resp.status_code != 200:
             return []
 
-        data = resp.json()
+        try:
-        impact_map = {"High": "high", "Medium": "medium", "Low": "low"}
+            root = ET.fromstring(resp.content)
+        except ET.ParseError:
+            return []
 
-        for item in data:
+        events = []
+        for item in root.iter("item"):
             try:
-                impact = impact_map.get(item.get("impact", ""), "low")
+                title_el = item.find("title")
-
+                pub_el = item.find("pubDate")
-                date_str = item.get("date", "")
+                desc_el = item.find("description")
-                if not date_str:
+                if title_el is None or title_el.text is None:
                     continue
-                event_time = datetime.fromisoformat(date_str.replace("Z", "+00:00"))
+
+                title = title_el.text.strip()
+
+                currency = ""
+                title_lower = title.lower()
+                for country, cur in self._COUNTRY_TO_CURRENCY.items():
+                    if title_lower.startswith(country):
+                        currency = cur
+                        title = title[len(country):].strip()
+                        break
+                if not currency:
+                    continue
+
+                event_time = self._parse_rss_date(
+                    pub_el.text if pub_el is not None else None
+                )
+                if event_time is None:
+                    continue
+
+                impact = "low"
+                desc_text = desc_el.text if desc_el is not None and desc_el.text else ""
+                impact_match = re.search(r"sprite-(high|medium|low)-impact", desc_text)
+                if impact_match:
+                    impact = impact_match.group(1)
+
+                cells = re.findall(r"<td[^>]*>(.*?)</td>", desc_text, re.DOTALL)
+                cells = [re.sub(r"<[^>]+>", "", c).strip() for c in cells]
+                previous = cells[2] if len(cells) > 2 and cells[2] else None
+                forecast = cells[3] if len(cells) > 3 and cells[3] else None
+                actual = cells[4] if len(cells) > 4 and cells[4] else None
 
                 events.append(EconomicEvent(
                     time=event_time,
-                    currency=item.get("country", ""),
+                    currency=currency,
                     impact=impact,
-                    event=item.get("title", ""),
+                    event=title,
-                    forecast=item.get("forecast") or None,
+                    forecast=forecast,
-                    previous=item.get("previous") or None,
+                    previous=previous,
-                    actual=item.get("actual") or None,
+                    actual=actual,
                 ))
             except Exception:
                 continue
 
+        logger.debug(f"MyFXBook calendar: parsed {len(events)} events")
         return events
 
     def _fetch_news_sentiment(self) -> List[NewsSentiment]:
+        """Cached headline fetch + sentiment analysis."""
+        return self._cached_fetch("_headlines_cache", self._fetch_news_sentiment_raw)
+
+    def _fetch_news_sentiment_raw(self) -> List[NewsSentiment]:
         """Fetch and analyze news headlines for sentiment."""
         headlines = []
+        seen_titles = set()
 
         for feed_url in self.config.rss_feeds:
             try:
@@ -328,6 +561,14 @@ class NewsEngine:
                     if pub_date and (datetime.now(timezone.utc) - pub_date).total_seconds() > 86400:
                         continue
 
+                    # Dedup within and across feeds so a syndicated story
+                    # doesn't count twice in the sentiment average. Google
+                    # News appends " - Publisher", so compare without it.
+                    title_key = title.lower().rsplit(" - ", 1)[0].strip()
+                    if title_key in seen_titles:
+                        continue
+                    seen_titles.add(title_key)
+
                     sentiment = self._analyze_headline_sentiment(title)
                     currencies = self._extract_currencies(title)
 
@@ -339,6 +580,10 @@ class NewsEngine:
                             currencies_mentioned=currencies,
                             sentiment_score=sentiment,
                             keywords=self._extract_keywords(title),
+                            sentiment_by_currency={
+                                cur: self._analyze_headline_sentiment(title, cur)
+                                for cur in currencies
+                            },
                         ))
 
             except Exception as e:
@@ -346,19 +591,38 @@ class NewsEngine:
 
         return headlines
 
-    def _analyze_headline_sentiment(self, headline: str) -> float:
+    def _analyze_headline_sentiment(
+        self, headline: str, for_currency: Optional[str] = None
+    ) -> float:
         """
         Simple keyword-based sentiment analysis.
 
         Returns -1.0 (very bearish) to +1.0 (very bullish).
 
+        When for_currency is "XAU", monetary-policy keywords contribute
+        with inverted sign (see GOLD_INVERSE_KEYWORDS).
+
         Note: This is intentionally simple. NLP models like FinBERT
         would be more accurate but require large dependencies. This catches
         the most impactful sentiment signals from headlines.
         """
         headline_lower = headline.lower()
-        bullish_score = sum(1 for kw in self.BULLISH_KEYWORDS if kw in headline_lower)
+        invert_policy = for_currency == "XAU"
-        bearish_score = sum(1 for kw in self.BEARISH_KEYWORDS if kw in headline_lower)
+
+        bullish_score = 0
+        bearish_score = 0
+        for kw in self.BULLISH_KEYWORDS:
+            if kw in headline_lower:
+                if invert_policy and kw in self.GOLD_INVERSE_KEYWORDS:
+                    bearish_score += 1
+                else:
+                    bullish_score += 1
+        for kw in self.BEARISH_KEYWORDS:
+            if kw in headline_lower:
+                if invert_policy and kw in self.GOLD_INVERSE_KEYWORDS:
+                    bullish_score += 1
+                else:
+                    bearish_score += 1
 
         total = bullish_score + bearish_score
         if total == 0:
@@ -369,18 +633,11 @@ class NewsEngine:
     def _extract_currencies(self, headline: str) -> List[str]:
         """Extract mentioned currencies from a headline."""
         headline_lower = headline.lower()
-        found = []
+        return [
-
+            currency
-        for currency, keywords in self.CURRENCY_KEYWORDS.items():
+            for currency, pattern in self._CURRENCY_PATTERNS.items()
-            if currency.lower() in headline_lower:
+            if pattern.search(headline_lower)
-                found.append(currency)
+        ]
-                continue
-            for kw in keywords:
-                if kw in headline_lower:
-                    found.append(currency)
-                    break
-
-        return list(set(found))
 
     def _extract_keywords(self, headline: str) -> List[str]:
         """Extract relevant trading keywords from headline."""
@@ -396,6 +653,8 @@ class NewsEngine:
         formats = [
             "%a, %d %b %Y %H:%M:%S %z",   # RFC 822 with numeric tz offset
             "%a, %d %b %Y %H:%M:%S %Z",   # RFC 822 with tz abbreviation (GMT/UTC)
+            "%a, %d %b %Y %H:%M %Z",      # RFC 822 without seconds (MyFXBook)
+            "%a, %d %b %Y %H:%M %z",      # RFC 822 without seconds, numeric offset
             "%Y-%m-%dT%H:%M:%S%z",         # ISO 8601 with offset
             "%Y-%m-%d %H:%M:%S",           # ISO 8601 naive (assume UTC)
         ]
@@ -413,15 +672,9 @@ class NewsEngine:
 
     @staticmethod
     def _get_affected_pairs(currency: str) -> List[str]:
-        """Get all major pairs affected by a currency event."""
+        _MAP = {
-        pairs = {
+            "USD": ["XAU/USD", "EUR/USD"],
-            "USD": ["EURUSD=X", "GBPUSD=X", "USDJPY=X", "USDCHF=X", "AUDUSD=X", "USDCAD=X", "NZDUSD=X"],
+            "XAU": ["XAU/USD"],
-            "EUR": ["EURUSD=X", "EURGBP=X", "EURJPY=X", "EURCHF=X"],
+            "EUR": ["EUR/USD"],
-            "GBP": ["GBPUSD=X", "EURGBP=X", "GBPJPY=X"],
-            "JPY": ["USDJPY=X", "EURJPY=X", "GBPJPY=X", "AUDJPY=X", "CADJPY=X"],
-            "CHF": ["USDCHF=X", "EURCHF=X"],
-            "AUD": ["AUDUSD=X", "AUDJPY=X"],
-            "CAD": ["USDCAD=X", "CADJPY=X"],
-            "NZD": ["NZDUSD=X"],
         }
-        return pairs.get(currency, [])
+        return _MAP.get(currency, [])

pair_signals/__init__.py

@@ -0,0 +1,52 @@
+"""
+Per-pair signal producers.
+
+Each pair module exposes:
+    make_config(primary_tf, confirmation_tfs) -> PlatformConfig
+
+To add a new pair: create pair_signals/<pair>.py with make_config(),
+then register it in this file.
+"""
+
+from typing import Optional, List, Dict, Callable
+
+from config import PlatformConfig, Timeframe
+from signal_aggregator import SignalAggregator, SignalReport
+from data_provider import ForexDataProvider
+
+PAIR_PRODUCERS: Dict[str, Callable] = {}
+
+
+def register_pair(symbol: str, config_factory: Callable):
+    PAIR_PRODUCERS[symbol] = config_factory
+
+
+def get_supported_pairs() -> List[str]:
+    return list(PAIR_PRODUCERS.keys())
+
+
+def produce_signal(
+    symbol: str,
+    primary_tf: Timeframe,
+    data_provider: ForexDataProvider,
+    confirmation_tfs: Optional[List[Timeframe]] = None,
+) -> SignalReport:
+    config_factory = PAIR_PRODUCERS.get(symbol)
+    if config_factory is None:
+        raise ValueError(
+            f"No signal producer registered for {symbol}. "
+            f"Available: {list(PAIR_PRODUCERS.keys())}"
+        )
+
+    config = config_factory(primary_tf, confirmation_tfs)
+    aggregator = SignalAggregator(config)
+    aggregator.data_provider = data_provider
+    return aggregator.analyze_pair(symbol=symbol)
+
+
+from pair_signals import xauusd as _xau_mod
+from pair_signals import eurusd as _eur_mod
+from config import XAUUSD, EURUSD
+
+register_pair(XAUUSD, _xau_mod.make_config)
+register_pair(EURUSD, _eur_mod.make_config)

pair_signals/eurusd.py

@@ -0,0 +1,158 @@
+"""EUR/USD signal producer config.
+
+Re-tuned 2026-07-10 against the realistic backtester (1-pip round-trip
+cost, next-open fills, raw-path resolution) on 2025-07-08 → 2026-07-08
+M5 with M15/H1 confirmations. RESULT: no configuration passed the
+acceptance rule (positive in BOTH train and val halves, PF >= 1.3).
+
+The M5 defaults below are the best found — roughly breakeven
+(-1.1R / 53 trades / PF 0.93 / 62% WR on 12mo) versus the donor
+branch's -6.7R. Findings that shaped them:
+  - transitioning regime bled -27R/88 trades wide-open → blocked
+  - A-grade entries LOSE (-26R) while B-grade wins — max conviction
+    fires at move extremes (same pathology as gold M5)
+  - Fridays (-17R) and hours 8, 12, 13, 18, 19 UTC negative
+  - score 30-35 band is where the train-half losses live → floor 35
+  - trailing exits beat static TP1 (arm 0.45R / dist 0.5R best)
+
+M15 + H1 investigated 2026-07-11, verdict NO VALIDATED EDGE either
+(all at 1-pip round-trip cost, real fills):
+  - M15 wide-open (all gates off): -2.5R/270 trades 2025-26;
+    -54.8R/308 trades on 2024-25 out-of-sample. PF 0.61-0.98.
+  - Best in-sample slice (transitioning-regime-only: +15.4R, PF 1.59,
+    positive both halves, month-spread) FAILED 2024-25 OOS at PF 0.55.
+    Within-year train/val splits are not sufficient for EUR — always
+    re-test on the 2024 window before believing a config.
+  - Raw-parquet forward-return screens (M15 and M5, ~14 entry
+    conditions x 4-5 horizons: momentum, mean-reversion, breakout,
+    streak, session families): nothing clears +1 pip in both years.
+  - H1 wide-open: +9.8R PF 1.32 in 2025-26 (both halves positive) but
+    -5.7R PF 0.82 on 2024-25. Positive from Jan 2025 onward; H2-2024
+    regime negative. Closest to workable, still unvalidated.
+
+Root cause is cost-to-volatility: EUR M15 ATR ~= 10 pips, so the 1-pip
+spread taxes every trade ~10% of ATR (M5 worse). The XAU signal engine
+survives because gold's ATR/spread ratio is several times larger.
+
+EUR/USD therefore ships FUNCTIONAL BUT OFF by default: it is not in
+PlatformConfig.scan_pairs. Scan it on demand (main.py --pair EURUSD,
+or add to scan_pairs) but treat all EUR M5/M15 signals as informational
+only.
+
+Calendar + hybrid avenues CLOSED 2026-07-12 (see backtest/research/):
+  - Surprise-conditioned entries (real Forex Factory archive, verified
+    timestamps): follow-surprise loses 2-5 pips at every horizon in
+    both years under every conditioning. The market prices the
+    surprise within the release bar.
+  - Event-window avoidance: EUR effect marginal/inconsistent (and on
+    XAU it would REMOVE profit — event-window trades outperform).
+  - H1-anchored direction + M15 pullback execution (pre-registered
+    kill criteria): 2025-26 +2.8R PF 1.11, 2024-25 OOS -1.5R PF 0.92
+    -> killed.
+Remaining avenues need data we don't have: tick-level fills for
+limit-entry cost reduction, or H4/D1 swing horizons.
+"""
+
+from typing import Optional, List
+from config import PlatformConfig, Timeframe, EURUSD
+
+_CONFIRMATIONS = {
+    Timeframe.M5:  [Timeframe.M15, Timeframe.H1],
+    Timeframe.M15: [Timeframe.M5, Timeframe.H1],
+    Timeframe.H1:  [Timeframe.M15, Timeframe.H4],
+    Timeframe.H4:  [Timeframe.H1, Timeframe.D1],
+}
+
+_TRENDING = {
+    "ema_trend": 2.0, "rsi": 1.0, "macd": 2.0,
+    "bollinger": 0.5, "keltner": 1.0, "roc": 1.5,
+    "candlestick": 0.5, "divergence": 2.0, "adx": 1.0,
+}
+
+EUR_REGIME_WEIGHTS = {
+    "trending_up": _TRENDING,
+    "trending_down": _TRENDING,
+    "trending": _TRENDING,
+    "ranging": {
+        "ema_trend": 0.3, "rsi": 2.0, "macd": 1.0,
+        "bollinger": 2.0, "keltner": 1.5, "roc": 0.5,
+        "candlestick": 0.8, "divergence": 2.5, "adx": 0.5,
+    },
+    "transitioning": {
+        "ema_trend": 1.5, "rsi": 1.5, "macd": 2.0,
+        "bollinger": 1.0, "keltner": 1.5, "roc": 2.0,
+        "candlestick": 0.8, "divergence": 2.0, "adx": 1.0,
+    },
+}
+
+_M5_TRENDING = {
+    "ema_trend": 1.5, "rsi": 1.0, "macd": 2.5,
+    "bollinger": 0.5, "keltner": 2.0, "roc": 2.0,
+    "candlestick": 0.3, "divergence": 1.0, "adx": 1.0,
+}
+
+M5_REGIME_WEIGHTS = {
+    "trending_up": _M5_TRENDING,
+    "trending_down": _M5_TRENDING,
+    "trending": _M5_TRENDING,
+    "ranging": {
+        "ema_trend": 0.3, "rsi": 1.5, "macd": 1.5,
+        "bollinger": 1.5, "keltner": 2.0, "roc": 1.0,
+        "candlestick": 0.3, "divergence": 1.5, "adx": 0.5,
+    },
+    "transitioning": {
+        "ema_trend": 1.0, "rsi": 1.0, "macd": 2.5,
+        "bollinger": 0.5, "keltner": 2.5, "roc": 2.0,
+        "candlestick": 0.3, "divergence": 1.0, "adx": 1.0,
+    },
+}
+
+
+def make_config(
+    primary_tf: Timeframe,
+    confirmation_tfs: Optional[List[Timeframe]] = None,
+) -> PlatformConfig:
+    config = PlatformConfig()
+    config.symbol = EURUSD
+    config.display_name = "EUR/USD"
+    config.pip_value = 0.0001
+    config.primary_timeframe = primary_tf
+    config.confirmation_timeframes = (
+        confirmation_tfs or _CONFIRMATIONS.get(primary_tf, [Timeframe.M15, Timeframe.H4])
+    )
+
+    # XAU gates are display-gated in the aggregator; disabling here is
+    # belt-and-braces so no gold logic can ever fire on an EUR config.
+    config.xau_strategy.enabled = False
+
+    # Keep event-risk signal gating ON for EUR (the XAU-tuned default
+    # turned it off after the 2026-07-11 audit — that evidence is
+    # gold-specific; EUR event-window trades were flat-to-negative).
+    config.news.event_risk_caps_signals = True
+
+    config.backtest.session_start_hour = 7
+    config.backtest.session_end_hour = 20
+    config.backtest.blocked_hours_utc = [22, 23, 0, 1, 2, 3]
+    config.backtest.allowed_weekdays = [0, 1, 2, 3, 4]
+
+    if primary_tf == Timeframe.M5:
+        config.technical.regime_weights_override = M5_REGIME_WEIGHTS
+        config.technical.swing_sr_lookback = 20
+        config.eur_strategy.sl_atr_floor = 0.7
+        config.eur_strategy.sl_atr_multiplier = 0.3
+        config.eur_strategy.min_quality_grades = ["A+", "A", "B"]
+        # Best-found 2026-07-10 (breakeven, see module docstring):
+        config.eur_strategy.min_entry_score = 35.0
+        config.eur_strategy.min_entry_confidence = 45.0
+        config.eur_strategy.blocked_vol_regimes = []
+        config.eur_strategy.min_indicator_agreement = 0
+        config.eur_strategy.allowed_weekdays = [0, 1, 2, 3]  # Fridays bled -17R
+        config.backtest.allowed_weekdays = [0, 1, 2, 3]
+        config.backtest.blocked_hours_utc = [22, 23, 0, 1, 2, 3, 8, 12, 13, 18, 19]
+        # Trailing exit beat static TP1; earlier/tighter than gold's 0.6/0.75.
+        config.signals.trail_arm_r = 0.45
+        config.signals.trail_dist_r = 0.5
+    else:
+        config.technical.regime_weights_override = EUR_REGIME_WEIGHTS
+
+    return config

pair_signals/xauusd.py

@@ -0,0 +1,42 @@
+"""XAU/USD signal producer config.
+
+PlatformConfig defaults ARE the tuned XAU/USD configuration, so this
+factory sets the timeframes plus documented per-timeframe overrides.
+It must stay equivalent to the inline PlatformConfig(symbol=XAUUSD,
+...) construction (plus those overrides) used by the backtester and
+Telegram bot (enforced by a test in test_all.py).
+
+M15 override (validated 2026-07-11, window 2025-03-30 -> 2026-05-31,
+$0.30 spread, trailing exits): tp_floor_rr 1.0 scores +116.6R/460
+trades PF 1.83 vs +105.4R/424 PF 1.82 at the 0.5 default — better in
+both train/val halves and on the 2024-06 -> 2025-03 out-of-sample
+window (+30.3R vs +10.6R; the 1R floor lifts targets past the min-TP
+spread gate, roughly doubling trades there). The same knob HURTS M5
+(pinned 12mo book: +131.3R/699 vs +139.9R/547, PF 1.55 vs 1.78), so it
+is per-timeframe, exactly as PENDING_STEPS' decision rule anticipated.
+"""
+
+from typing import Optional, List
+from config import PlatformConfig, Timeframe
+
+_CONFIRMATIONS = {
+    Timeframe.M5:  [Timeframe.M15, Timeframe.H1],
+    Timeframe.M15: [Timeframe.M5, Timeframe.H1],
+    Timeframe.H1:  [Timeframe.M15, Timeframe.H4],
+    Timeframe.H4:  [Timeframe.H1, Timeframe.D1],
+}
+
+
+def make_config(
+    primary_tf: Timeframe,
+    confirmation_tfs: Optional[List[Timeframe]] = None,
+) -> PlatformConfig:
+    config = PlatformConfig()
+    config.primary_timeframe = primary_tf
+    config.confirmation_timeframes = (
+        confirmation_tfs or _CONFIRMATIONS.get(primary_tf, [Timeframe.M15, Timeframe.H4])
+    )
+    if primary_tf == Timeframe.M15:
+        # 1R minimum target on M15 only — see module docstring.
+        config.signals.tp_floor_rr = 1.0
+    return config

requirements.txt

@@ -0,0 +1,6 @@
+numpy>=1.24
+pandas>=2.0
+requests>=2.28
+twelvedata>=1.2.0
+python-telegram-bot>=20.0
+websockets>=12.0

script.txt

@@ -1 +0,0 @@
-pip install pandas numpy requests python-dateutil -q

test_ws_dual.py

@@ -0,0 +1,108 @@
+#!/usr/bin/env python3
+"""Test whether Twelve Data allows 2 simultaneous WebSocket connections for EUR/USD.
+
+Uses the twelvedata SDK which handles the WS protocol correctly.
+Runs connection 1, then adds connection 2 after 10s, monitors for 30s total.
+"""
+
+import threading
+import time
+
+from twelvedata import TDClient
+
+API_KEY = "f563b22ffdfd4924955decbe1a8c8701"
+SYMBOL = "EUR/USD"
+DURATION = 30
+
+results = {
+    "WS-1": {"ticks": 0, "errors": [], "alive": False},
+    "WS-2": {"ticks": 0, "errors": [], "alive": False},
+}
+
+
+def make_handlers(label):
+    def on_event(event):
+        if event.get("event") == "price":
+            results[label]["ticks"] += 1
+            price = event.get("price", "?")
+            print(f"[{label}] tick #{results[label]['ticks']:3d}  price={price}")
+
+    def on_error(error):
+        results[label]["errors"].append(str(error))
+        print(f"[{label}] ERROR: {error}")
+
+    def on_close(status):
+        results[label]["alive"] = False
+        print(f"[{label}] CLOSED (status={status})")
+
+    def on_connect():
+        results[label]["alive"] = True
+        print(f"[{label}] CONNECTED — subscribed to {SYMBOL}")
+
+    return on_event, on_error, on_close, on_connect
+
+
+def run_ws(label):
+    td = TDClient(apikey=API_KEY)
+    on_event, on_error, on_close, on_connect = make_handlers(label)
+    ws = td.websocket(
+        symbols=SYMBOL,
+        on_event=on_event,
+        on_error=on_error,
+        on_close=on_close,
+        on_connect=on_connect,
+    )
+    ws.connect()
+    return ws
+
+
+def main():
+    print(f"Testing dual WebSocket for {SYMBOL} ({DURATION}s)\n")
+
+    print("Starting WS-1...")
+    ws1 = run_ws("WS-1")
+
+    time.sleep(10)
+    print(f"\nWS-1 ticks so far: {results['WS-1']['ticks']}")
+    print("Starting WS-2...\n")
+    ws2 = run_ws("WS-2")
+
+    remaining = DURATION - 10
+    time.sleep(remaining)
+
+    print("\nStopping connections...")
+    try:
+        ws1.disconnect()
+    except Exception:
+        pass
+    try:
+        ws2.disconnect()
+    except Exception:
+        pass
+
+    time.sleep(1)
+
+    print("\n" + "=" * 50)
+    print("RESULTS")
+    print("=" * 50)
+    for label, info in sorted(results.items()):
+        status = "alive" if info["alive"] else "dead"
+        errs = info["errors"] or ["none"]
+        print(f"  {label}: {status}  ticks={info['ticks']}  errors={errs[0]}")
+
+    ws1_ok = results["WS-1"]["ticks"] > 0 and not results["WS-1"]["errors"]
+    ws2_ok = results["WS-2"]["ticks"] > 0 and not results["WS-2"]["errors"]
+    ws1_died = results["WS-1"]["alive"] is False and results["WS-1"]["ticks"] > 0
+
+    if ws1_ok and ws2_ok:
+        print("\n✓ Both connections worked — dual WS is supported")
+    elif ws2_ok and ws1_died:
+        print("\n✗ WS-2 replaced WS-1 — only 1 connection allowed at a time")
+    elif ws1_ok and not ws2_ok:
+        print("\n✗ WS-2 was rejected — only 1 connection allowed")
+    else:
+        print("\n? Inconclusive — check errors above")
+
+
+if __name__ == "__main__":
+    main()