Add WebSocket dual-connection test script

Probes whether Twelve Data allows two simultaneous WS connections
for one symbol (SDK-based, 30s run, prints per-connection verdict).

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

.gitignore

@@ -0,0 +1,27 @@
+# Secrets
+.env
+.env.local
+
+# Editor / tooling
+.claude/
+.vscode/
+.idea/
+
+# Python
+__pycache__/
+*.pyc
+*.egg-info/
+.venv/
+venv/
+
+# OS
+.DS_Store
+Thumbs.db
+
+# Logs
+*.log
+
+# Backtesting data (keep scripts, ignore generated files)
+backtest/cache/
+backtest/*.csv
+backtest/__pycache__/

CLAUDE.md

@@ -0,0 +1,121 @@
+# ForexBot2
+
+XAU/USD signal intelligence platform. Analyzes gold across multiple timeframes using 9 technical indicators with regime-adaptive weights, news sentiment, and economic calendar data. Delivers signals via Telegram with full trade setups (entry, SL, TP).
+
+## Architecture
+
+```
+Twelve Data REST API + WebSocket
+        |
+  data_provider.py          (fetch, cache, validate OHLCV; rate-limited 55 calls/min)
+        |
+    engine.py                (9 indicators per timeframe → composite score)
+        |
+  signal_aggregator.py       (MTF agreement + news + final signal + trade setup)
+        |
+  +-----+--------+
+  |              |
+telegram_bot.py  main.py
+(Telegram alerts, (CLI single-pair
+ auto-scanner,    analysis)
+ WS monitor)
+```
+
+## Core Files
+
+| File | Purpose |
+|------|---------|
+| `config.py` | All configuration dataclasses, XAU/USD pair definition, strategy config |
+| `data_provider.py` | Twelve Data SDK wrapper with in-memory cache and rate limiting |
+| `engine.py` | `TechnicalEngine` — computes EMA, RSI, BB, ADX, ATR, MACD, ROC, Keltner, divergences, candlestick patterns, swing S/R |
+| `signal_aggregator.py` | `SignalAggregator` — MTF analysis, news integration, diminishing penalty cap, trade setup generation |
+| `news_engine.py` | Economic calendar (Forex Factory JSON) + RSS sentiment (keyword-based) |
+| `telegram_bot.py` | Async Telegram bot with auto-scanner (M5/M15/H1/H4), WebSocket price monitor |
+| `main.py` | CLI entry point for XAU/USD analysis |
+| `test_all.py` | Comprehensive QA suite (~137 tests, no live API needed) |
+
+## Key Concepts
+
+**Signal Pipeline:** OHLCV → indicators → regime classification → regime-weighted composite score → MTF agreement check → news/event adjustment → diminishing penalty cap → final signal (STRONG_BUY/BUY/NEUTRAL/SELL/STRONG_SELL) → quality grade (A+/A/B/C/D/F) → trade setup if grade C+.
+
+**Regime-Adaptive Weights:** Market classified as `trending_up`, `trending_down`, `ranging`, or `transitioning` via ADX + EMA + ADX slope. Gold-specific weights per regime:
+- **Trending:** MACD (2.0), EMA (2.0), Keltner (1.5), ROC (1.5), Divergence (1.0)
+- **Ranging:** Divergence (2.5), Keltner (2.0), RSI (1.5), BB (1.5), MACD (1.5)
+- **Transitioning:** Keltner (2.5), MACD (2.5), ROC (2.0), Divergence (2.0), EMA (1.5)
+
+ROC is directionally redundant with MACD/Keltner/EMA (0.8–0.94 correlation) but its weight is load-bearing: composite thresholds are calibrated with it — removing it backtested 13R worse. Don't zero it without retuning thresholds.
+
+**RSI (regime-aware, default):** In trends RSI buys pullback zones instead of fading strength (classic mode backtested worse for gold). In `transitioning` only extremes (≥70/≤30) signal.
+
+**Divergence:** 40-bar lookback; price and indicator swings must align within ±2 bars to count as one divergence observation.
+
+**Transitioning Regime:** ADX < 25 but ADX slope > 3.0 = trend emerging. Catches trend birth 5-10 bars before ADX crosses 25. Boosts quality grade for early entries.
+
+**Diminishing Penalty Cap:** Penalties sorted by magnitude, applied with exponential decay (1st at 100%, 2nd at 50%, 3rd at 25%). Hard cap at 30% total. Replaces old additive stacking that could gut 30-50% confidence.
+
+**Indicators (9 scored):** EMA Trend, RSI, Bollinger Bands, MACD, ADX, Keltner Channels (TTM Squeeze), ROC (Rate of Change), Divergence, Candlestick Patterns.
+
+**Trade Setup:** SL = nearest swing low/high ± 1×ATR buffer (min 1.5×ATR). TP = nearest swing level if RR≥1.2, else calculated from ATR. Minimum RR 0.5:1.
+
+## How to Run
+
+```bash
+# CLI (XAU/USD analysis)
+python main.py                    # H1 default
+python main.py --timeframe M15 --detail --json
+
+# Telegram bot (auto-scanner)
+TELEGRAM_TOKEN=<token> TELEGRAM_CHAT_ID=<id> python telegram_bot.py
+
+# Tests
+python test_all.py
+
+# Backtest
+python backtest/gpu_backtest.py --pair XAUUSD --days 365
+```
+
+## Environment Variables
+
+| Variable | Purpose | Default |
+|----------|---------|---------|
+| `TWELVE_DATA_API_KEY` | Market data API key | Hardcoded fallback in config |
+| `TELEGRAM_TOKEN` | Telegram bot token | Hardcoded fallback |
+| `TELEGRAM_CHAT_ID` | Auto-scan chat ID (enables boot auto-scan) | Hardcoded fallback |
+
+## Dependencies
+
+`numpy`, `pandas`, `requests`, `twelvedata`, `python-telegram-bot`, `websockets` — see `requirements.txt`.
+
+## Data Provider
+
+- **Source:** Twelve Data REST API via `twelvedata` SDK
+- **Rate limit:** 55 credits/min (Grow plan), enforced by sliding-window throttle
+- **Cache:** In-memory dict, TTL = candle duration of the timeframe
+- **Validation:** Drops NaN rows, forward-fills gaps (limit 3), fixes OHLC inconsistencies, removes outliers (>10σ), strips future timestamps, deduplicates
+- **WebSocket:** XAU/USD only
+
+## Telegram Bot
+
+- **Auto-scan:** M5, M15, H1, H4 at candle boundaries + 10s buffer (all timeframes alert)
+- **Manual scan:** Buttons for M15/H1/H4 individual scans + "Scan All TFs" multi-timeframe summary
+- **WebSocket:** Streams live ticks for XAU/USD; triggers immediate M15 scan on key level breach (swing S/R, BB, KC, EMA200) with 5-min cooldown
+- **Commands:** `/start`, `/scan`, `/autoscan`, `/status`, `/cache`, `/clearcache`, `/help`
+
+## Backtesting
+
+The `backtest/` directory contains `gpu_backtest.py` — a GPU-accelerated backtester (CuPy with NumPy fallback) that replays historical OHLCV through the real SignalAggregator. Pre-computes all indicator series including ROC, Keltner, ADX slope for 100x+ speedup. Uses parquet caching for data persistence.
+
+Fill realism: charges a round-trip spread per trade (`--spread`, default $0.30 for XAU), fills gap-through SL/TP at the bar open, and resolves trades against the RAW price path (the parquet cache stores raw bars; validation/cleaning runs on load — signals see cleaned data, trade resolution sees real spikes). Entry is at the next bar's open after the signal bar. Weekend/session filters evaluate bar-close (decision) time to match the live bot's wall clock.
+
+GPU note: CuPy needs the NVIDIA runtime pip wheels (`nvidia-cuda-runtime-cu12`, `nvidia-cuda-nvrtc-cu12`, `nvidia-curand-cu12`, `nvidia-nvjitlink-cu12`) or it silently falls back to NumPy — check the "Backend:" line in the run log. The bar-replay loop itself is single-threaded Python either way (~340 bars/s); GPU only accelerates indicator precompute and SL/TP scanning.
+
+## Code Conventions
+
+- Python 3.12, type hints on public methods
+- Dataclasses for all config and output structures
+- `SignalStrength` enum (-2 to +2) is the universal signal currency
+- Indicators stored as `Dict[str, IndicatorSignal]` in `TechnicalAnalysis`
+- Key levels stored as `Dict[str, float]` — prefixed `swing_high_`, `swing_low_`, `ema_`, `bb_`, `kc_`, `atr`
+- Private indicator data attached as `result._rsi_series`, `result._macd_histogram`, `result._adx_slope`, `result._kc_squeeze` (underscore prefix, skipped by composite scorer)
+- Blocking signal computation runs in `ThreadPoolExecutor(max_workers=4)` to avoid blocking the async Telegram event loop
+- Shared `ForexDataProvider` instance across all scans for cache reuse

USAGE.md

@@ -0,0 +1,198 @@
+# ForexBot2 — USD/JPY Signal Intelligence
+
+A focused USD/JPY signal generator + walk-forward backtester. Live signals
+and backtests run through the **same** technical engine, so what you measure
+in a backtest is exactly what you'd execute live.
+
+## Setup
+
+```bash
+pip install -r requirements.txt
+```
+
+API key (optional — a default key is in `config.py`):
+
+```bash
+export TWELVE_DATA_API_KEY=your_key_here          # Linux/Mac
+$env:TWELVE_DATA_API_KEY="your_key_here"          # PowerShell
+```
+
+---
+
+## Live Signal
+
+```bash
+python main.py                          # H1, summary view
+python main.py --detail                 # Full indicator breakdown
+python main.py --timeframe H4 --detail  # H4 instead of H1
+python main.py --json                   # JSON output (for piping)
+```
+
+| Option        | Default | Description                                |
+|---------------|---------|--------------------------------------------|
+| `--timeframe` | `H1`    | `M5 M15 M30 H1 H4 D1`                      |
+| `--detail`    | off     | Indicator breakdown, key levels, reasons   |
+| `--json`      | off     | Machine-readable output                    |
+| `--verbose`   | off     | Debug logging                              |
+
+### Output
+
+```
+USD/JPY  │  Price: 149.230  │  Regime: trending_up  │  Vol: normal
+
+SIGNAL: BUY  │  Score: +28.4  │  Grade: B  │  Confidence: 62%
+Action: BUY — Decent setup, wait for confirmation
+
+Multi-Timeframe:  ✓ ALIGNED
+   M15: BUY          (+18.2)
+    H1: BUY          (+28.4)
+    H4: STRONG BUY   (+45.1)
+
+Trade Setup:
+  Direction:     BUY
+  Entry:         149.230
+  Stop Loss:     148.960  (27.0 pips)
+  Take Profit 1: 149.770  (54.0 pips)
+  Take Profit 2: 150.040
+  Risk:Reward:   1:2.0
+```
+
+---
+
+## Backtest (Walk-Forward)
+
+The backtester replays USD/JPY history bar-by-bar through the live signal
+pipeline. No lookahead, no parallel implementation — what passes in
+backtest passes live.
+
+```bash
+python main.py --backtest                          # Full available history
+python main.py --backtest --bars 2000              # Last 2000 H1 bars
+python main.py --backtest --start 2024-01-01 --end 2024-12-31
+python main.py --backtest --timeframe H4           # Backtest on H4
+python main.py --backtest --min-grade A            # Only A/A+ entries
+python main.py --backtest --journal                # Last 10 trades printed
+python main.py --backtest --journal-csv trades.csv # Full journal to CSV
+python main.py --backtest --regime-filter          # Skip choppy/ranging bars
+python main.py --backtest --regime-filter --regime-min-adx 35
+python main.py --backtest --session-filter         # Trade only 07-16 UTC
+python main.py --backtest --session-filter --session-start 8 --session-end 17
+python main.py --backtest --auto-session-filter    # Reactive session toggle
+python main.py --backtest --auto-session-filter --auto-session-lookback 15 \
+                          --auto-session-threshold-pct -0.5
+```
+
+### Regime filter
+
+The engine is a confluence-based trend-follower — it makes money in trending
+regimes and loses in chop. `--regime-filter` gates entries on H4 ADX:
+signals only fire when ADX ≥ `--regime-min-adx` (default 30). Sweet spot is
+ADX 30–35; tighter cuts trades faster than it improves quality.
+
+### Session filter
+
+`--session-filter` restricts entries to a UTC hour window (default 07-16,
+covering London + NY). Diagnostic on 2025-11 → 2026-05 showed 47% WR /
++127 pips inside that window vs 0% WR / -571 pips outside it. Override
+bounds with `--session-start` (inclusive) and `--session-end` (exclusive).
+
+Tradeoff: it's regime-specific medicine. In trending regimes (e.g. 2024)
+it cuts winning off-hours trades and gives back ~33% of the edge. Ships
+**default off** because the pure regime filter still wins cumulative
+across the 2024 / 2023 / 2026 windows.
+
+### Auto session toggle
+
+`--auto-session-filter` is a reactive variant: it tracks the rolling mean
+`pnl_pct` over the last N closed trades (`--auto-session-lookback`,
+default 10) and activates the 07-16 UTC window only when that mean drops
+below `--auto-session-threshold-pct` (default `-0.3`). Releases
+automatically once expectancy recovers.
+
+Less aggressive than the manual flag — preserves more trend edge but
+reacts slowly because the deque must fill with losses before kicking in.
+Use this when you don't want to manually toggle the regime medicine.
+
+### Execution model
+
+| Aspect             | Behavior                                                          |
+|--------------------|-------------------------------------------------------------------|
+| Entry              | Next bar's open (no current-bar lookahead)                        |
+| Stop Loss          | 1.5 × ATR from entry (configurable)                               |
+| Take Profit        | RR × stop distance (default 1:2)                                  |
+| Same-bar SL & TP   | Conservative: SL fills first                                      |
+| Spread cost        | Half-spread on entry + half-spread on exit (full round-trip)      |
+| Slippage           | 0.5 pips per fill (entry + exit)                                  |
+| Position size      | 1% of current equity risked per trade                             |
+| Concurrency        | One open position at a time                                       |
+| News in backtest   | Disabled — RSS/calendar can't be replayed without lookahead       |
+
+### Output
+
+```
+Backtest Summary — USD/JPY
+─────────────────────────────────────────────────
+Trades:           87  (52W / 35L)
+Win Rate:         59.8%
+Profit Factor:    1.84
+Expectancy:       +12.4 pips/trade  (+12.42 per trade)
+Initial Balance:  $10,000.00
+Final Balance:    $11,084.00
+Total Return:     +10.84%
+CAGR:             +18.92%
+Max Drawdown:     -4.21%
+Sharpe Ratio:     1.42
+Sortino Ratio:    2.18
+Avg Duration:     14.3 bars
+Win Streak:       7 | Loss Streak: 4
+─────────────────────────────────────────────────
+```
+
+---
+
+## Parameter Optimization (parallel)
+
+```bash
+python main.py --backtest --optimize
+```
+
+Runs a 3×3×3 grid (27 combos) over `atr_sl_multiplier`, `default_rr_ratio`,
+and `min_score` (signal-score threshold). Combos run in parallel across all
+CPU cores via `multiprocessing.Pool`. Twelve Data is hit once — workers
+share an on-disk parquet cache (`.cache/usdjpy_<tf>.parquet`).
+
+Output ranks combos by total return:
+
+```
+Top 10 Parameter Combinations (by total return):
+
+#   Return%     PF   WinR  Trades  MaxDD%  Sharpe  Params
+──────────────────────────────────────────────────────────────────────
+1   +14.21%   2.10  62.5%     64   -3.84   1.78    atr_sl_multiplier=1.5, default_rr_ratio=2.5, min_score=25.0
+2   +11.93%   1.92  58.7%     71   -4.11   1.55    atr_sl_multiplier=1.0, default_rr_ratio=2.0, min_score=15.0
+...
+```
+
+Customize the grid in `config.py → BacktestConfig.param_grid`.
+
+---
+
+## Files
+
+| File                  | Role                                              |
+|-----------------------|---------------------------------------------------|
+| `main.py`             | CLI entry point (live + backtest + optimize)      |
+| `config.py`           | All parameters in one place                       |
+| `data_provider.py`    | Twelve Data fetcher + parquet cache               |
+| `engine.py`           | Technical indicators (8 + divergence + patterns)  |
+| `signal_aggregator.py`| Multi-TF aggregation, grading, trade setup        |
+| `news_engine.py`      | RSS sentiment + economic calendar (live only)     |
+| `backtest.py`         | Walk-forward replay + parallel optimizer          |
+| `metrics.py`          | Win rate, PF, Sharpe, max DD, trade journal       |
+
+---
+
+## Disclaimer
+
+For educational and research use. Past performance does not guarantee
+future results. Always use proper risk management.

backtest/diagnose_filters.py

@@ -0,0 +1,609 @@
+"""
+ForexBot2 — Filter Rejection Diagnostic
+========================================
+Replays the same backtest loop as gpu_backtest.py but instead of
+collecting trades, counts how many M15 bars are rejected by each
+gate in the signal pipeline.
+
+Produces a summary table showing where potential trades are being
+lost, helping tune filter thresholds.
+
+Usage:
+    python backtest/diagnose_filters.py --pair XAUUSD --days 365
+    python backtest/diagnose_filters.py --pair XAUUSD --days 365 --sample-every 1
+"""
+
+from __future__ import annotations
+
+import argparse
+import logging
+import sys
+import time
+from collections import Counter, defaultdict
+from datetime import datetime, timedelta, timezone
+from pathlib import Path
+from typing import Dict, List, Optional, Tuple
+
+import numpy as np
+import pandas as pd
+
+# Ensure project root on sys.path.
+PROJECT_ROOT = Path(__file__).resolve().parent.parent
+if str(PROJECT_ROOT) not in sys.path:
+    sys.path.insert(0, str(PROJECT_ROOT))
+
+from config import (  # noqa: E402
+    PAIR_INFO,
+    PlatformConfig,
+    SignalStrength,
+    Timeframe,
+)
+from signal_aggregator import SignalAggregator, SignalReport  # noqa: E402
+
+# Reuse GPU backtest infrastructure (data loading, indicator pre-compute, etc.)
+from gpu_backtest import (  # noqa: E402
+    BAR_SECONDS,
+    GPUTechnicalEngine,
+    PAIR_ALIASES,
+    BacktestConfig,
+    IndicatorCache,
+    fetch_history,
+    precompute_indicators,
+    _is_xau_blocked_by_calendar,
+)
+from data_provider import ForexDataProvider  # noqa: E402
+
+logger = logging.getLogger("diagnose_filters")
+
+
+# ═════════════════════════════════════════════════════════════════════
+#  OUTCOME CLASSIFICATION
+# ═════════════════════════════════════════════════════════════════════
+
+def classify_bar(
+    report: SignalReport,
+    is_xau: bool,
+) -> Tuple[str, List[str]]:
+    """Classify a bar's outcome and list which filters killed it.
+
+    Returns:
+        (outcome, [filter_names])
+
+    Outcomes:
+        "TRADE"           — quality C+ and trade_setup exists
+        "QUALITY_OK_NO_SETUP" — quality C+ but trade_setup is None
+        "LOW_QUALITY"     — signal reached BUY/SELL but quality D or F
+        "NEUTRAL"         — signal stayed NEUTRAL (score didn't reach +/-30)
+
+    Filter names are extracted from report.warnings.
+    """
+    filters_hit: List[str] = []
+
+    # Scan warnings for known filter tags
+    filter_tags = {
+        "H4 RSI GATE":       "H4_RSI_GATE",
+        "H1 MACD GATE":      "H1_MACD_GATE",
+        "RSI ZONE GATE":     "RSI_ZONE_GATE",
+        "XAU BUY FILTER":    "XAU_BUY_FILTER",
+        "SESSION FILTER":    "SESSION_FILTER",
+        "DAY FILTER":        "DAY_FILTER",
+        "LOW ATR FILTER":    "LOW_ATR_FILTER",
+        "SELL SUPPRESSED":   "SELL_SUPPRESSED",
+        "EXHAUSTION":        "EXHAUSTION_PENALTY",
+        "TREND EXHAUSTION":  "TREND_EXHAUSTION",
+        "NO PULLBACK":       "NO_PULLBACK_PENALTY",
+        "Weak pullback":     "WEAK_PULLBACK_PENALTY",
+        "Extended entry":    "EXTENDED_ENTRY_PENALTY",
+        "Counter-trend":     "COUNTER_TREND_PENALTY",
+        "EXTREME volatility":"EXTREME_VOL_PENALTY",
+        "NEAR EMA200":       "NEAR_EMA200_PENALTY",
+        "FAR FROM EMA200":   "FAR_EMA200_PENALTY",
+        "EVENT RISK":        "EVENT_RISK",
+        "REGIME FILTER":     "REGIME_FILTER",
+        "Timeframes CONFLICT": "MTF_CONFLICT",
+    }
+
+    for w in report.warnings:
+        for tag, name in filter_tags.items():
+            if tag in w:
+                filters_hit.append(name)
+
+    # Determine outcome
+    non_neutral = report.final_signal not in (SignalStrength.NEUTRAL,)
+    quality_ok = report.signal_quality in ("A+", "A", "B", "C")
+
+    if quality_ok and report.trade_setup is not None:
+        return "TRADE", filters_hit
+    elif quality_ok and report.trade_setup is None:
+        return "QUALITY_OK_NO_SETUP", filters_hit
+    elif non_neutral and not quality_ok:
+        return "LOW_QUALITY", filters_hit
+    else:
+        return "NEUTRAL", filters_hit
+
+
+# ═════════════════════════════════════════════════════════════════════
+#  SCORE DISTRIBUTION TRACKER
+# ═════════════════════════════════════════════════════════════════════
+
+class ScoreTracker:
+    """Tracks score/confidence distributions for analysis."""
+
+    def __init__(self):
+        self.scores: List[float] = []
+        self.confidences: List[float] = []
+        self.abs_scores: List[float] = []
+        # How many bars had score in various ranges
+        self.score_bins = Counter()
+        self.conf_bins = Counter()
+        # Track "near miss" bars — score close to thresholds
+        self.near_miss_score: int = 0  # score 20-29 (almost BUY/SELL)
+        self.near_miss_conf: int = 0   # confidence 45-54 (almost passing)
+
+    def record(self, score: float, confidence: float):
+        self.scores.append(score)
+        self.confidences.append(confidence)
+        self.abs_scores.append(abs(score))
+
+        # Score bins
+        a = abs(score)
+        if a < 10:
+            self.score_bins["0-9"] += 1
+        elif a < 20:
+            self.score_bins["10-19"] += 1
+        elif a < 30:
+            self.score_bins["20-29"] += 1
+            self.near_miss_score += 1
+        elif a < 40:
+            self.score_bins["30-39"] += 1
+        elif a < 50:
+            self.score_bins["40-49"] += 1
+        else:
+            self.score_bins["50+"] += 1
+
+        # Confidence bins
+        if confidence < 30:
+            self.conf_bins["0-29"] += 1
+        elif confidence < 45:
+            self.conf_bins["30-44"] += 1
+        elif confidence < 55:
+            self.conf_bins["45-54"] += 1
+            self.near_miss_conf += 1
+        elif confidence < 65:
+            self.conf_bins["55-64"] += 1
+        elif confidence < 75:
+            self.conf_bins["65-74"] += 1
+        else:
+            self.conf_bins["75+"] += 1
+
+
+# ═════════════════════════════════════════════════════════════════════
+#  MAIN DIAGNOSTIC LOOP
+# ═════════════════════════════════════════════════════════════════════
+
+def run_diagnosis(
+    pair: str,
+    start: datetime,
+    end: datetime,
+    sample_every: int = 4,
+    use_cache: bool = True,
+    primary_tf: Timeframe = Timeframe.M15,
+    confirm_tfs: Optional[List[Timeframe]] = None,
+    warmup_bars: int = 250,
+) -> None:
+    if confirm_tfs is None:
+        confirm_tfs = [Timeframe.H1, Timeframe.H4]
+
+    pair_info = PAIR_INFO[pair]
+    display = pair_info["display"]
+    is_xau = display == "XAU/USD"
+
+    platform_cfg = PlatformConfig(
+        symbol=pair,
+        display_name=display,
+        pip_value=pair_info["pip_value"],
+        primary_timeframe=primary_tf,
+        confirmation_timeframes=list(confirm_tfs),
+    )
+
+    provider = ForexDataProvider(platform_cfg)
+
+    # ── Load data ──
+    timeframes = [primary_tf] + list(confirm_tfs)
+    pad_days = max(60, warmup_bars // 4)
+    fetch_start = start - timedelta(days=pad_days)
+
+    dfs_full: Dict[Timeframe, pd.DataFrame] = {}
+    for tf in timeframes:
+        df = fetch_history(
+            provider, pair, tf,
+            start=fetch_start, end=end,
+            use_cache=use_cache,
+        )
+        if df.empty:
+            raise RuntimeError(f"No data for {pair} {tf.value}")
+        dfs_full[tf] = df
+
+    # ── Pre-compute indicators ──
+    t0 = time.perf_counter()
+    caches: Dict[str, IndicatorCache] = {
+        tf.value: precompute_indicators(dfs_full[tf], platform_cfg.technical, gpu_log=True)
+        for tf in timeframes
+    }
+    logger.info(f"Indicator pre-compute: {time.perf_counter() - t0:.2f}s")
+
+    # ── Build aggregator with GPU engine ──
+    aggregator = SignalAggregator(platform_cfg)
+    aggregator.tech_engine = GPUTechnicalEngine(platform_cfg.technical, caches)
+
+    # ── Walk primary TF ──
+    primary_df = dfs_full[primary_tf]
+    bt_start_ts = pd.Timestamp(start)
+    bt_end_ts = pd.Timestamp(end)
+
+    first_eligible = max(
+        warmup_bars,
+        int(primary_df.index.searchsorted(bt_start_ts)),
+    )
+    last_pos = min(
+        int(primary_df.index.searchsorted(bt_end_ts, side="right")),
+        len(primary_df),
+    )
+
+    primary_dur = pd.Timedelta(seconds=BAR_SECONDS[primary_tf])
+
+    # Counters
+    total_bars_in_window = 0
+    total_bars_sampled = 0
+    outcome_counts = Counter()
+    filter_counts = Counter()
+    # Filters that killed bars that were otherwise directional (non-NEUTRAL score)
+    directional_filter_counts = Counter()
+    # Pre-pipeline rejections (XAU calendar filters applied before analyze_from_dataframes)
+    pre_pipeline = Counter()
+
+    # Track signal directions
+    signal_dir_counts = Counter()
+    quality_counts = Counter()
+
+    # Score tracker
+    tracker = ScoreTracker()
+
+    # Track "would-have-been" trades: bars where the primary score was
+    # directional but got killed by a specific filter
+    filter_kill_breakdown = defaultdict(Counter)
+
+    logger.info(
+        f"Diagnosing {display} {primary_tf.value}: "
+        f"bars {first_eligible}-{last_pos} (every {sample_every}th bar)"
+    )
+
+    t_loop = time.perf_counter()
+    for pos in range(first_eligible, last_pos, sample_every):
+        total_bars_in_window += 1
+        ts = primary_df.index[pos]
+        decision_time = ts + primary_dur
+
+        # ── XAU calendar pre-filter (same as gpu_backtest.py) ──
+        if is_xau:
+            block = _is_xau_blocked_by_calendar(ts, platform_cfg)
+            if block:
+                pre_pipeline[block] += 1
+                continue
+
+        total_bars_sampled += 1
+
+        # ── Build sliced dfs (same logic as gpu_backtest.py) ──
+        slice_dfs: Dict[Timeframe, pd.DataFrame] = {}
+        for tf in timeframes:
+            full = dfs_full[tf]
+            bar_dur = pd.Timedelta(seconds=BAR_SECONDS[tf])
+            cutoff = decision_time - bar_dur
+            cut = int(full.index.searchsorted(cutoff, side="right"))
+            if cut == 0:
+                continue
+            slice_dfs[tf] = full.iloc[:cut]
+
+        if primary_tf not in slice_dfs:
+            continue
+
+        # ── Run the full signal pipeline ──
+        report: SignalReport = aggregator.analyze_from_dataframes(
+            symbol=pair,
+            dfs=slice_dfs,
+            context=None,
+            use_news=False,
+        )
+
+        # Record score/confidence
+        tracker.record(report.final_score, report.final_confidence)
+
+        # Signal direction
+        signal_dir_counts[report.final_signal.name] += 1
+        quality_counts[report.signal_quality] += 1
+
+        # Classify
+        outcome, filters_hit = classify_bar(report, is_xau)
+        outcome_counts[outcome] += 1
+
+        for f in filters_hit:
+            filter_counts[f] += 1
+
+        # If the primary score was directional (>= 15 or <= -15), the bar
+        # had some potential — track what killed it
+        if abs(report.primary_score) >= 15:
+            for f in filters_hit:
+                directional_filter_counts[f] += 1
+
+        # For bars that got quality D/F but had a directional signal,
+        # track which filter warnings appeared
+        if outcome in ("LOW_QUALITY", "QUALITY_OK_NO_SETUP"):
+            for f in filters_hit:
+                filter_kill_breakdown[outcome][f] += 1
+
+    elapsed = time.perf_counter() - t_loop
+
+    # ═════════════════════════════════════════════════════════════════
+    #  PRINT RESULTS
+    # ═════════════════════════════════════════════════════════════════
+
+    bar = "=" * 80
+    print(f"\n{bar}")
+    print(f"  FILTER REJECTION DIAGNOSTIC — {display}")
+    print(f"  {start.date()} to {end.date()} | {primary_tf.value} "
+          f"(sampled every {sample_every}th bar)")
+    print(f"{bar}")
+
+    # Scale counts if sampling
+    scale_note = ""
+    if sample_every > 1:
+        scale_note = f"  (sampled {total_bars_sampled} of ~{total_bars_in_window * sample_every} eligible bars)"
+    print(f"\n  Total bars in backtest window: {total_bars_in_window}")
+    if pre_pipeline:
+        for reason, count in sorted(pre_pipeline.items()):
+            print(f"  Pre-pipeline rejection ({reason}): {count}")
+    print(f"  Bars analyzed: {total_bars_sampled}{scale_note}")
+    print(f"  Analysis took: {elapsed:.1f}s "
+          f"({total_bars_sampled / max(elapsed, 0.001):.0f} bars/s)")
+
+    # ── Outcome breakdown ──
+    print(f"\n{'-' * 80}")
+    print("  OUTCOME BREAKDOWN")
+    print(f"{'-' * 80}")
+    outcome_order = ["TRADE", "QUALITY_OK_NO_SETUP", "LOW_QUALITY", "NEUTRAL"]
+    outcome_labels = {
+        "TRADE":               "Actual trades (quality C+ & setup exists)",
+        "QUALITY_OK_NO_SETUP": "Quality C+ but NO trade setup generated",
+        "LOW_QUALITY":         "Directional signal but quality D/F",
+        "NEUTRAL":             "NEUTRAL (score didn't reach +/-30 or conf < 55)",
+    }
+    for o in outcome_order:
+        count = outcome_counts.get(o, 0)
+        pct = count / max(total_bars_sampled, 1) * 100
+        label = outcome_labels.get(o, o)
+        print(f"  {count:6d} ({pct:5.1f}%)  {label}")
+
+    # ── Signal direction breakdown ──
+    print(f"\n{'-' * 80}")
+    print("  SIGNAL DIRECTION BREAKDOWN")
+    print(f"{'-' * 80}")
+    for name in ["STRONG_BUY", "BUY", "NEUTRAL", "SELL", "STRONG_SELL"]:
+        count = signal_dir_counts.get(name, 0)
+        pct = count / max(total_bars_sampled, 1) * 100
+        print(f"  {count:6d} ({pct:5.1f}%)  {name}")
+
+    # ── Quality grade breakdown ──
+    print(f"\n{'-' * 80}")
+    print("  QUALITY GRADE BREAKDOWN")
+    print(f"{'-' * 80}")
+    for grade in ["A+", "A", "B", "C", "D", "F"]:
+        count = quality_counts.get(grade, 0)
+        pct = count / max(total_bars_sampled, 1) * 100
+        print(f"  {count:6d} ({pct:5.1f}%)  Grade {grade}")
+
+    # ── Filter hit counts (all bars) ──
+    print(f"\n{'-' * 80}")
+    print("  FILTER HIT COUNTS (all bars, a bar can hit multiple filters)")
+    print(f"{'-' * 80}")
+    if filter_counts:
+        for name, count in filter_counts.most_common():
+            pct = count / max(total_bars_sampled, 1) * 100
+            print(f"  {count:6d} ({pct:5.1f}%)  {name}")
+    else:
+        print("  (no filters triggered)")
+
+    # ── Filters that killed directional bars (primary_score >= 15) ──
+    print(f"\n{'-' * 80}")
+    print("  FILTERS KILLING DIRECTIONAL BARS (|primary_score| >= 15)")
+    print(f"{'-' * 80}")
+    if directional_filter_counts:
+        for name, count in directional_filter_counts.most_common():
+            print(f"  {count:6d}  {name}")
+    else:
+        print("  (none)")
+
+    # ── Filter breakdown for LOW_QUALITY bars ──
+    if filter_kill_breakdown.get("LOW_QUALITY"):
+        print(f"\n{'-' * 80}")
+        print("  FILTERS ON LOW_QUALITY BARS (had BUY/SELL but quality D/F)")
+        print(f"{'-' * 80}")
+        for name, count in filter_kill_breakdown["LOW_QUALITY"].most_common():
+            print(f"  {count:6d}  {name}")
+
+    # ── Filter breakdown for QUALITY_OK_NO_SETUP bars ──
+    if filter_kill_breakdown.get("QUALITY_OK_NO_SETUP"):
+        print(f"\n{'-' * 80}")
+        print("  FILTERS ON QUALITY_OK_NO_SETUP BARS (quality C+ but no trade setup)")
+        print(f"{'-' * 80}")
+        for name, count in filter_kill_breakdown["QUALITY_OK_NO_SETUP"].most_common():
+            print(f"  {count:6d}  {name}")
+
+    # ── Score distribution ──
+    print(f"\n{'-' * 80}")
+    print("  SCORE DISTRIBUTION (|score|)")
+    print(f"{'-' * 80}")
+    for bin_name in ["0-9", "10-19", "20-29", "30-39", "40-49", "50+"]:
+        count = tracker.score_bins.get(bin_name, 0)
+        pct = count / max(total_bars_sampled, 1) * 100
+        marker = " <-- near miss (almost BUY/SELL)" if bin_name == "20-29" else ""
+        print(f"  {count:6d} ({pct:5.1f}%)  |score| {bin_name}{marker}")
+
+    # ── Confidence distribution ──
+    print(f"\n{'-' * 80}")
+    print("  CONFIDENCE DISTRIBUTION")
+    print(f"{'-' * 80}")
+    for bin_name in ["0-29", "30-44", "45-54", "55-64", "65-74", "75+"]:
+        count = tracker.conf_bins.get(bin_name, 0)
+        pct = count / max(total_bars_sampled, 1) * 100
+        marker = " <-- near miss (almost passing)" if bin_name == "45-54" else ""
+        print(f"  {count:6d} ({pct:5.1f}%)  confidence {bin_name}{marker}")
+
+    # ── Near misses ──
+    print(f"\n{'-' * 80}")
+    print("  NEAR MISSES")
+    print(f"{'-' * 80}")
+    print(f"  {tracker.near_miss_score:6d}  Bars with |score| 20-29 (threshold is 30)")
+    print(f"  {tracker.near_miss_conf:6d}  Bars with confidence 45-54 (threshold is 55)")
+
+    # ── Summary statistics ──
+    if tracker.scores:
+        scores_arr = np.array(tracker.scores)
+        conf_arr = np.array(tracker.confidences)
+        print(f"\n{'-' * 80}")
+        print("  SCORE/CONFIDENCE STATISTICS")
+        print(f"{'-' * 80}")
+        print(f"  Score:      mean={np.mean(scores_arr):+.1f}  "
+              f"median={np.median(scores_arr):+.1f}  "
+              f"std={np.std(scores_arr):.1f}  "
+              f"min={np.min(scores_arr):+.1f}  "
+              f"max={np.max(scores_arr):+.1f}")
+        print(f"  Confidence: mean={np.mean(conf_arr):.1f}  "
+              f"median={np.median(conf_arr):.1f}  "
+              f"std={np.std(conf_arr):.1f}  "
+              f"min={np.min(conf_arr):.1f}  "
+              f"max={np.max(conf_arr):.1f}")
+
+    # ── Blocked hours analysis (for XAU) ──
+    if is_xau:
+        blocked = set(platform_cfg.xau_strategy.blocked_hours_utc)
+        all_hours = set(range(24))
+        open_hours = sorted(all_hours - blocked)
+        print(f"\n{'-' * 80}")
+        print("  XAU HOUR/DAY CONFIG")
+        print(f"{'-' * 80}")
+        print(f"  Blocked hours:  {sorted(blocked)}")
+        print(f"  Open hours:     {open_hours}")
+        print(f"  Allowed days:   {platform_cfg.xau_strategy.allowed_weekdays} "
+              f"(0=Mon..4=Fri)")
+        print(f"  Open fraction:  {len(open_hours)/24:.0%} of hours x "
+              f"{len(platform_cfg.xau_strategy.allowed_weekdays)}/5 weekdays = "
+              f"{len(open_hours)/24 * len(platform_cfg.xau_strategy.allowed_weekdays)/5:.0%} of trading time")
+
+    # ── Interpretation / key takeaways ──
+    print(f"\n{bar}")
+    print("  KEY TAKEAWAYS")
+    print(f"{bar}")
+
+    neutral_pct = outcome_counts.get("NEUTRAL", 0) / max(total_bars_sampled, 1) * 100
+    low_q_pct = outcome_counts.get("LOW_QUALITY", 0) / max(total_bars_sampled, 1) * 100
+    trade_count = outcome_counts.get("TRADE", 0)
+
+    print(f"  - {neutral_pct:.1f}% of bars never reach a directional signal (NEUTRAL)")
+    print(f"  - {low_q_pct:.1f}% of bars get a BUY/SELL but confidence/quality too low")
+    print(f"  - {trade_count} bars produce actual trades")
+
+    if filter_counts:
+        top3 = filter_counts.most_common(3)
+        print(f"  - Top rejection filters:")
+        for name, count in top3:
+            print(f"      {name}: {count} bars hit")
+
+    if tracker.near_miss_score > 50:
+        print(f"  - SIGNIFICANT: {tracker.near_miss_score} bars had |score| 20-29 "
+              f"(lowering threshold from 30 to 25 would capture many)")
+
+    if tracker.near_miss_conf > 50:
+        print(f"  - SIGNIFICANT: {tracker.near_miss_conf} bars had confidence 45-54 "
+              f"(lowering threshold from 55 to 50 would capture many)")
+
+    print(f"\n{bar}\n")
+
+
+# ═════════════════════════════════════════════════════════════════════
+#  CLI
+# ═════════════════════════════════════════════════════════════════════
+
+def _resolve_pair(s: str) -> str:
+    s_up = s.upper().strip()
+    if s_up in PAIR_ALIASES:
+        return PAIR_ALIASES[s_up]
+    if s_up in PAIR_INFO:
+        return s_up
+    raise SystemExit(f"Unknown pair '{s}'. Valid: {', '.join(sorted(PAIR_ALIASES))}")
+
+
+def _resolve_tf(s: str) -> Timeframe:
+    for tf in Timeframe:
+        if tf.value == s:
+            return tf
+    raise SystemExit(f"Unknown timeframe '{s}'")
+
+
+def main(argv: Optional[List[str]] = None) -> int:
+    p = argparse.ArgumentParser(
+        description="Diagnose which filters reject the most bars in the backtest pipeline",
+        formatter_class=argparse.ArgumentDefaultsHelpFormatter,
+    )
+    p.add_argument("--pair", required=True, help="EURUSD | GBPUSD | USDJPY | XAUUSD")
+    p.add_argument("--days", type=int, default=365, help="Days of history")
+    p.add_argument("--start", help="ISO date YYYY-MM-DD (overrides --days)")
+    p.add_argument("--end", help="ISO date YYYY-MM-DD (defaults to today)")
+    p.add_argument("--primary-tf", default="15min", help="Primary TF")
+    p.add_argument("--confirm-tfs", default="1h,4h", help="Comma-separated confirmation TFs")
+    p.add_argument("--warmup-bars", type=int, default=250)
+    p.add_argument("--sample-every", type=int, default=4,
+                   help="Analyze every Nth bar (1=all, 4=25%% sample for speed)")
+    p.add_argument("--no-cache", action="store_true", help="Skip parquet cache")
+    p.add_argument("--log-level", default="INFO")
+    args = p.parse_args(argv)
+
+    logging.basicConfig(
+        level=args.log_level.upper(),
+        format="%(asctime)s [%(levelname)s] %(name)s: %(message)s",
+    )
+
+    pair = _resolve_pair(args.pair)
+    primary_tf = _resolve_tf(args.primary_tf)
+    confirm_tfs = [_resolve_tf(s.strip()) for s in args.confirm_tfs.split(",") if s.strip()]
+
+    end = (
+        datetime.fromisoformat(args.end).replace(tzinfo=timezone.utc)
+        if args.end else datetime.now(timezone.utc)
+    )
+    if args.start:
+        start = datetime.fromisoformat(args.start).replace(tzinfo=timezone.utc)
+    else:
+        start = end - timedelta(days=args.days)
+
+    logger.info(
+        f"Diagnosing {PAIR_INFO[pair]['display']} | "
+        f"Window: {start.date()} -> {end.date()} | "
+        f"Sample: every {args.sample_every}th bar"
+    )
+
+    run_diagnosis(
+        pair=pair,
+        start=start,
+        end=end,
+        sample_every=args.sample_every,
+        use_cache=not args.no_cache,
+        primary_tf=primary_tf,
+        confirm_tfs=confirm_tfs,
+        warmup_bars=args.warmup_bars,
+    )
+
+    return 0
+
+
+if __name__ == "__main__":
+    sys.exit(main())

backtest/instructions_for_backtest.txt

@@ -0,0 +1,59 @@
+Note down entery and exit Why we took trade?
+What were each indicator signal on that specific time when we took trade
+How many indicators agreed at trade time
+Always backtest using engine.py and signal_aggregator.py
+Dont take trades on weekends and Fridays
+Keep TP small 1.3 Reward ratio
+At the end of test make table of results and print all values you observed.
+Trade context
+
+Time of day and trading session (Asian/London/NY/overlap)
+Day of week (track Mon/Tue/Wed/Thu performance separately)
+Market regime at entry: trending, ranging, or choppy (use ATR or ADX)
+Volatility level (ATR value or ATR percentile vs recent average)
+Pair/instrument traded
+News proximity (high-impact news within 30 min before/after entry)
+
+Trade execution details
+
+Spread at entry in pips
+Slippage on entry and exit (planned vs actual fill)
+Time in trade (minutes/hours from entry to exit)
+Maximum Adverse Excursion (MAE): how far against you it went
+Maximum Favorable Excursion (MFE): how far in profit before pullback
+
+Risk metrics
+
+Max consecutive losses (streak)
+Max consecutive wins (streak)
+Expectancy: (Win% x Avg Win) minus (Loss% x Avg Loss)
+Expectancy per hour in trade
+Recovery factor: Total R divided by Max DD
+
+Indicator/signal analysis
+
+Each indicator's standalone win rate when it agreed with the trade
+Confluence level performance (2 vs 3 vs 4+ indicators agreeing)
+Which indicator disagreed on losing trades (find the "canary")
+
+Decision quality
+
+Rate each setup A/B/C at entry, before knowing outcome
+Verify A-grade setups actually win more often
+Track system trades vs discretionary deviations separately
+
+Process notes
+
+Screenshot or chart snapshot at entry and exit
+Emotional state (calm, FOMO, revenge, tired)
+What would have happened if you held longer or exited sooner
+
+Aggregate analysis at end of backtest
+
+Equity curve, not just total R
+Performance breakdown by session, day, volatility regime
+Win rate over rolling 20-trade windows (check for edge decay)
+
+Critical extra
+"Do not trade" log: setups flagged but skipped, and what happened after
+

config.py

@@ -1,8 +1,6 @@
 """
-Forex Signal Intelligence Platform - Configuration
+ForexBot2 — Configuration (XAU/USD focused)
-===================================================
+==========================================
-Central configuration for all signal analysis parameters.
-Tuned by market regime and validated against historical performance.
 """
 
 from dataclasses import dataclass, field
@@ -37,160 +35,180 @@ class MarketSession(Enum):
     OVERLAP_LONDON_NY = "london_ny_overlap"
 
 
-# ── Major Forex Pairs ──────────────────────────────────────────────
+# ── Supported pairs ──────────────────────────────────────────────
-MAJOR_PAIRS = [
+XAUUSD = "XAU/USD"
-    "EURUSD=X", "GBPUSD=X", "USDJPY=X", "USDCHF=X",
-    "AUDUSD=X", "USDCAD=X", "NZDUSD=X",
-]
 
-CROSS_PAIRS = [
+PAIR_INFO = {
-    "EURGBP=X", "EURJPY=X", "GBPJPY=X", "AUDJPY=X",
+    XAUUSD: {"display": "XAU/USD", "pip_value": 0.10, "is_jpy_pair": False},
-    "EURCHF=X", "CADJPY=X",
-]
-
-# Yahoo Finance symbol mapping for display
-SYMBOL_DISPLAY = {
-    "EURUSD=X": "EUR/USD", "GBPUSD=X": "GBP/USD",
-    "USDJPY=X": "USD/JPY", "USDCHF=X": "USD/CHF",
-    "AUDUSD=X": "AUD/USD", "USDCAD=X": "USD/CAD",
-    "NZDUSD=X": "NZD/USD", "EURGBP=X": "EUR/GBP",
-    "EURJPY=X": "EUR/JPY", "GBPJPY=X": "GBP/JPY",
-    "AUDJPY=X": "AUD/JPY", "EURCHF=X": "EUR/CHF",
-    "CADJPY=X": "CAD/JPY",
 }
 
 
 @dataclass
 class TechnicalConfig:
-    """Technical indicator parameters - carefully tuned for Forex."""
+    """Technical indicator parameters."""
-    # EMA periods (fast/medium/slow for trend detection)
     ema_fast: int = 9
     ema_medium: int = 21
     ema_slow: int = 50
     ema_trend: int = 200
 
-    # RSI
     rsi_period: int = 14
     rsi_overbought: float = 70.0
     rsi_oversold: float = 30.0
     rsi_neutral_upper: float = 60.0
     rsi_neutral_lower: float = 40.0
 
-    # MACD (12, 26, 9 is standard)
     macd_fast: int = 12
     macd_slow: int = 26
     macd_signal: int = 9
 
-    # Bollinger Bands
     bb_period: int = 20
     bb_std: float = 2.0
 
-    # ATR for volatility measurement
     atr_period: int = 14
 
-    # Stochastic
-    stoch_k: int = 14
-    stoch_d: int = 3
-    stoch_overbought: float = 80.0
-    stoch_oversold: float = 20.0
-
-    # ADX for trend strength
     adx_period: int = 14
     adx_strong_trend: float = 25.0
     adx_very_strong: float = 50.0
 
-    # Ichimoku
+    divergence_lookback: int = 40
-    ichimoku_tenkan: int = 9
+    swing_sr_lookback: int = 50
-    ichimoku_kijun: int = 26
-    ichimoku_senkou_b: int = 52
 
-    # Volume profile
+    disabled_indicators: List[str] = field(default_factory=list)
-    volume_ma_period: int = 20
 
-    # Pivot points lookback
+    ema_mode: str = "4ema"          # "4ema" (9/21/50/200) or "2ema" (50/200)
-    pivot_lookback: int = 1
+    rsi_mode: str = "regime_aware"  # "classic" or "regime_aware"
+    macd_as_pillar: bool = True     # MACD histogram is a scored signal
 
-    # Divergence detection — how many bars back to scan for swing points
+    # ROC (Rate of Change)
-    divergence_lookback: int = 14
+    roc_period: int = 10
+    roc_threshold: float = 1.0     # % threshold for strong signal
+
+    # Keltner Channels
+    kc_period: int = 20
+    kc_atr_period: int = 10
+    kc_multiplier: float = 1.5
+
+    # ADX slope (trend acceleration detection)
+    adx_slope_lookback: int = 5
+    adx_slope_emerging_threshold: float = 3.0
+
+    regime_weights_override: Dict[str, Dict[str, float]] = field(default_factory=dict)
 
 
 @dataclass
 class SignalConfig:
-    """Signal scoring and filtering parameters."""
+    """Signal scoring & filtering."""
-    # Minimum confidence to emit a signal (0-100)
     min_confidence: float = 55.0
 
-    # Weight allocation for composite score
     weights: Dict[str, float] = field(default_factory=lambda: {
-        "trend": 0.25,          # EMA alignment, ADX
+        "ema_trend": 1.0,
-        "momentum": 0.20,       # RSI, Stochastic, MACD
+        "rsi": 1.5,
-        "volatility": 0.15,     # BB position, ATR regime
+        "bollinger": 1.5,
-        "price_action": 0.15,   # Candlestick patterns, S/R
+        "macd": 1.5,
-        "multi_timeframe": 0.15,# Higher TF confirmation
+        "keltner": 1.5,
-        "sentiment": 0.10,      # News/calendar events
+        "roc": 1.5,
+        "candlestick": 1.0,
+        "divergence": 1.5,
     })
 
-    # Multi-timeframe confirmation requirements
+    mtf_agreement_threshold: int = 2  # of 3 timeframes
-    mtf_agreement_threshold: int = 2  # out of 3 timeframes must agree
-
-    # Signal cooldown (prevent over-signalling same direction)
     cooldown_bars: int = 5
 
-    # Risk management defaults
+    default_risk_pct: float = 1.0
-    default_risk_pct: float = 1.0  # risk 1% per trade
+    default_rr_ratio: float = 2.0
-    default_rr_ratio: float = 2.0  # 1:2 risk-reward minimum
+    atr_sl_multiplier: float = 1.5
+    min_rr_ratio: float = 0.4
+    tp_tightening_factor: float = 0.8
 
 
 @dataclass
 class NewsConfig:
-    """Economic calendar and sentiment configuration."""
+    """Economic calendar & sentiment."""
-    # High-impact event currencies
+    tracked_currencies: List[str] = field(default_factory=lambda: ["USD", "XAU"])
-    tracked_currencies: List[str] = field(default_factory=lambda: [
+    min_impact: str = "medium"
-        "USD", "EUR", "GBP", "JPY", "CHF", "AUD", "CAD", "NZD"
-    ])
-
-    # Event impact levels to monitor
-    min_impact: str = "medium"  # "low", "medium", "high"
-
-    # Pre-event buffer (minutes before high-impact event = no new signals)
     pre_event_buffer_minutes: int = 30
-
-    # Post-event volatility window (minutes after event = cautious signals)
     post_event_buffer_minutes: int = 15
 
-    # News sentiment sources
     rss_feeds: List[str] = field(default_factory=lambda: [
-        "https://www.forexlive.com/feed/",
+        "https://investinglive.com/feed/",
         "https://www.fxstreet.com/rss",
     ])
 
 
+@dataclass
+class XauStrategyConfig:
+    """XAU/USD strategy config.
+    Legacy gates (H4 RSI, H1 MACD, M15 score) removed in favor of
+    new momentum indicators (ROC, Keltner squeeze, ADX slope).
+    """
+    enabled: bool = True
+    rsi_entry_threshold: float = 45.0
+    tp_atr_multiplier: float = 1.0
+    sl_atr_multiplier: float = 2.0
+    min_ema200_tfs: int = 2
+    allow_generic_trades: bool = True
+    blocked_hours_utc: List[int] = field(default_factory=lambda: [0, 1, 2, 10, 11, 13, 16, 21])
+    weekend_filter: bool = True
+    min_atr_threshold: float = 1.0
+    v2_sl_atr_multiplier: float = 2.0
+    buy_threshold_boost: int = 0
+    m5_confirmation_bars: int = 0
+    h4_rsi_gate: bool = False
+    h1_macd_gate: bool = False
+    rsi_zone: tuple = (40.0, 65.0)
+    allowed_weekdays: List[int] = field(default_factory=lambda: [0, 1, 2, 3, 4, 6])  # Mon-Fri + Sun (Sydney open)
+    sell_min_quality: List[str] = field(default_factory=lambda: ["A+", "A", "B"])
+    # Volatility regimes in which SELL setups are allowed (low/normal-vol
+    # shorts lose consistently against gold's macro uptrend).
+    sell_vol_regimes: List[str] = field(default_factory=lambda: ["high", "extreme"])
+
+
+@dataclass
+class FilterConfig:
+    """Regime and session filters for live signals."""
+    regime_filter: bool = False
+    regime_filter_timeframe: Timeframe = Timeframe.H4
+    regime_filter_min_adx: float = 25.0
+
+    session_filter: bool = True
+    session_start_hour: int = 3
+    session_end_hour: int = 23
+    blocked_hours_utc: List[int] = field(default_factory=lambda: [0, 1, 2, 10, 11, 13, 16, 21])
+
+    allowed_weekdays: List[int] = field(default_factory=lambda: [0, 1, 2, 3, 4, 6])  # Mon-Fri + Sun (Sydney open)
+
+
 @dataclass
 class PlatformConfig:
-    """Overall platform configuration."""
+    """Top-level configuration."""
     technical: TechnicalConfig = field(default_factory=TechnicalConfig)
     signals: SignalConfig = field(default_factory=SignalConfig)
     news: NewsConfig = field(default_factory=NewsConfig)
+    backtest: FilterConfig = field(default_factory=FilterConfig)
+    xau_strategy: XauStrategyConfig = field(default_factory=XauStrategyConfig)
 
-    # Default pairs to monitor
+    symbol: str = XAUUSD
-    watch_pairs: List[str] = field(default_factory=lambda: MAJOR_PAIRS)
+    display_name: str = "XAU/USD"
+    pip_value: float = 0.10
 
-    # Analysis timeframes (primary + confirmation)
     primary_timeframe: Timeframe = Timeframe.H1
     confirmation_timeframes: List[Timeframe] = field(
         default_factory=lambda: [Timeframe.M15, Timeframe.H4]
     )
 
-    # Data source
+    data_lookback_days: int = 60
-    data_lookback_days: int = 60  # enough for 200 EMA on H1
 
-    # Twelve Data API key (overridden by TWELVE_DATA_API_KEY env var)
     twelve_data_api_key: str = "f563b22ffdfd4924955decbe1a8c8701"
 
-    # Dashboard refresh interval (seconds)
+    refresh_interval: int = 300
-    refresh_interval: int = 300  # 5 minutes
+
+    scan_pairs: List[str] = field(
+        default_factory=lambda: [XAUUSD]
+    )
+    scan_timeframes: List[Timeframe] = field(
+        default_factory=lambda: [Timeframe.M5, Timeframe.M15, Timeframe.H1, Timeframe.H4]
+    )
+    candle_buffer_seconds: int = 10
+    ws_cooldown_seconds: int = 300
 
-    # Logging
     log_level: str = "INFO"
-    log_signals_to_file: bool = True
-    signal_log_path: str = "signals.log"

data_provider.py

@@ -1,41 +1,45 @@
 """
-Forex Signal Intelligence Platform - Data Provider
+ForexBot2 — Data Provider (Twelve Data SDK)
-====================================================
+==========================================================
-Retrieves and validates market data from Twelve Data API.
+Fetches and validates OHLCV via the official `twelvedata` Python SDK.
-Falls back to synthetic GBM data if the API is unavailable.
 """
 
-import hashlib
 import logging
 import os
 import time
 from collections import deque
-from datetime import datetime, timedelta, timezone
+from typing import Dict, Optional
-from typing import Dict, List, Optional
 
-import numpy as np
 import pandas as pd
-import requests
+from twelvedata import TDClient
 
-from config import PlatformConfig, Timeframe, SYMBOL_DISPLAY
+from config import PlatformConfig, Timeframe, XAUUSD, PAIR_INFO
 
 logger = logging.getLogger(__name__)
 
 
+def _is_forex_weekend(now=None) -> bool:
+    """Forex market closed: Friday 22:00 UTC → Sunday 22:00 UTC."""
+    if now is None:
+        now = pd.Timestamp.now(tz="UTC")
+    wd = now.weekday()
+    hour = now.hour
+    if wd == 4 and hour >= 22:
+        return True
+    if wd == 5:
+        return True
+    if wd == 6 and hour < 22:
+        return True
+    return False
+
+
 class DataIntegrityError(Exception):
     """Raised when data fails validation checks."""
     pass
 
 
 class ForexDataProvider:
-    """
+    """Twelve Data SDK client + on-disk parquet cache."""
-    Forex data provider backed by Twelve Data API.
-
-    Twelve Data has native 4h candles, eliminating the H1→H4 resampling
-    hack needed for Yahoo Finance. Free tier: 800 credits/day, 8 req/min.
-    """
-
-    TD_BASE_URL = "https://api.twelvedata.com"
 
     TD_INTERVALS = {
         Timeframe.M1:  "1min",
@@ -48,20 +52,19 @@ class ForexDataProvider:
         Timeframe.W1:  "1week",
     }
 
-    # Approximate bars per day for each timeframe (used to size outputsize)
     BARS_PER_DAY = {
-        Timeframe.M1:  1440,
+        Timeframe.M1: 1440, Timeframe.M5: 288, Timeframe.M15: 96,
-        Timeframe.M5:  288,
+        Timeframe.M30: 48, Timeframe.H1: 24, Timeframe.H4: 6,
-        Timeframe.M15: 96,
+        Timeframe.D1: 1, Timeframe.W1: 0.143,
-        Timeframe.M30: 48,
-        Timeframe.H1:  24,
-        Timeframe.H4:  6,
-        Timeframe.D1:  1,
-        Timeframe.W1:  0.143,
     }
 
-    # Free plan: 8 API credits per minute (each symbol = 1 credit)
+    CANDLE_SECONDS = {
-    RATE_LIMIT = 8
+        Timeframe.M1: 60, Timeframe.M5: 300, Timeframe.M15: 900,
+        Timeframe.M30: 1800, Timeframe.H1: 3600, Timeframe.H4: 14400,
+        Timeframe.D1: 86400, Timeframe.W1: 604800,
+    }
+
+    RATE_LIMIT = 55
     RATE_WINDOW = 60.0
 
     def __init__(self, config: PlatformConfig):
@@ -72,94 +75,113 @@ class ForexDataProvider:
         )
         self._cache: Dict[str, pd.DataFrame] = {}
         self._cache_timestamps: Dict[str, float] = {}
-        self._request_times: deque = deque()  # timestamps of recent API credits used
+        self._request_times: deque = deque()
-        self._session = requests.Session()
+        self._td: Optional[TDClient] = (
-        self._session.headers.update({
+            TDClient(apikey=self._api_key) if self._api_key else None
-            "User-Agent": "Mozilla/5.0 (compatible; ForexIntel/1.0)"
+        )
-        })
 
     @staticmethod
     def _to_td_symbol(symbol: str) -> str:
-        """Convert internal symbol (EURUSD=X) to Twelve Data format (EUR/USD)."""
+        if "/" in symbol:
+            return symbol
         clean = symbol.replace("=X", "")
         return f"{clean[:3]}/{clean[3:]}"
 
+    # ── Public API ─────────────────────────────────────────────────
+
     def fetch_ohlcv(
         self,
-        symbol: str,
+        symbol: str = XAUUSD,
-        timeframe: Timeframe,
+        timeframe: Timeframe = Timeframe.H1,
         lookback_days: Optional[int] = None,
     ) -> pd.DataFrame:
-        """
+        """Fetch OHLCV with in-memory caching for live mode."""
-        Fetch OHLCV data for a Forex pair with full validation.
-
-        Returns a DataFrame with columns: Open, High, Low, Close, Volume
-        Index is a timezone-aware DatetimeIndex in UTC.
-        """
         if lookback_days is None:
             lookback_days = self.config.data_lookback_days
 
         cache_key = f"{symbol}_{timeframe.value}"
         now = time.time()
 
+        cache_ttl = self.CANDLE_SECONDS.get(timeframe, self.config.refresh_interval)
         if (
             cache_key in self._cache_timestamps
-            and (now - self._cache_timestamps[cache_key]) < self.config.refresh_interval
+            and (now - self._cache_timestamps[cache_key]) < cache_ttl
             and cache_key in self._cache
         ):
             cached = self._cache.get(cache_key)
             if cached is not None and not cached.empty:
+                if self._is_data_fresh(cached, symbol, timeframe):
                     return cached
+                logger.warning(f"{symbol} {timeframe.value}: cached data is stale, refetching")
+                del self._cache[cache_key]
+                del self._cache_timestamps[cache_key]
 
-        df = self._fetch_twelve_data(symbol, timeframe, lookback_days)
+        outputsize = max(
+            300,
+            min(int(lookback_days * self.BARS_PER_DAY[timeframe]) + 20, 5000),
+        )
+        df = self._fetch_twelve_data(symbol, timeframe, outputsize)
+        df = self._drop_forming_candle(df, timeframe)
         df = self._validate_and_clean(df, symbol, timeframe)
 
+        if not self._is_data_fresh(df, symbol, timeframe):
+            logger.warning(
+                f"STALE DATA from API: {symbol} {timeframe.value} — "
+                f"last bar {df.index[-1]} is too old. Signal quality may be degraded."
+            )
+
         self._cache[cache_key] = df
         self._cache_timestamps[cache_key] = now
         return df
 
-    def prefetch_batch(
+    def _drop_forming_candle(self, df: pd.DataFrame, timeframe: Timeframe) -> pd.DataFrame:
-        self,
+        """Drop the still-forming last candle so analysis only sees closed bars.
-        symbols: List[str],
+
-        timeframes: List[Timeframe],
+        Twelve Data's time_series includes the current partial candle; the
-        lookback_days: Optional[int] = None,
+        scanner fires seconds after the boundary, so without this guard every
-    ) -> None:
+        indicator reads a seconds-old stub bar and signals repaint.
         """
-        Batch-fetch all symbols for each timeframe in a single API call per TF.
+        if df is None or df.empty:
+            return df
+        candle_seconds = self.CANDLE_SECONDS.get(timeframe, 3600)
+        now = pd.Timestamp.now(tz="UTC")
+        if df.index[-1] + pd.Timedelta(seconds=candle_seconds) > now:
+            df = df.iloc[:-1]
+        return df
 
-        Twelve Data accepts comma-separated symbols, so 7 pairs × 3 timeframes
+    def _is_data_fresh(self, df: pd.DataFrame, symbol: str, timeframe: Timeframe) -> bool:
-        = 3 requests instead of 21 — well within the free-tier 8 req/min limit.
+        """Check if the last candle is within acceptable age for this timeframe."""
-        Results are stored in the local cache; subsequent fetch_ohlcv calls
+        if df is None or df.empty:
-        for the same symbol/TF will be served from cache.
+            return False
-        """
-        if lookback_days is None:
-            lookback_days = self.config.data_lookback_days
 
-        for tf in timeframes:
+        now = pd.Timestamp.now(tz="UTC")
-            try:
-                self._batch_fetch_timeframe(symbols, tf, lookback_days)
-                time.sleep(1.0)  # brief pause between TF requests
-            except Exception as e:
-                logger.warning(f"Batch prefetch failed for {tf.value}: {e}")
 
-    def fetch_multiple(
+        if _is_forex_weekend(now):
-        self,
+            return True
-        symbols: List[str],
-        timeframe: Timeframe,
-        lookback_days: Optional[int] = None,
-    ) -> Dict[str, pd.DataFrame]:
-        """Fetch data for multiple symbols, returning dict of DataFrames."""
-        results = {}
-        for symbol in symbols:
-            try:
-                df = self.fetch_ohlcv(symbol, timeframe, lookback_days)
-                results[symbol] = df
-            except Exception as e:
-                logger.warning(f"Failed to fetch {symbol}: {e}")
-        return results
 
-    def get_latest_price(self, symbol: str) -> Optional[float]:
+        last_bar = df.index[-1]
-        """Return the most recent close price for a symbol."""
+        candle_seconds = self.CANDLE_SECONDS.get(timeframe, 3600)
+        max_age_seconds = candle_seconds * 3
+        age_seconds = (now - last_bar).total_seconds()
+
+        if age_seconds > max_age_seconds:
+            logger.warning(
+                f"{symbol} {timeframe.value}: last bar {last_bar.strftime('%Y-%m-%d %H:%M')} "
+                f"is {age_seconds / 60:.0f}m old (limit: {max_age_seconds / 60:.0f}m)"
+            )
+            return False
+
+        return True
+
+    def clear_cache(self) -> int:
+        """Wipe all cached data. Returns number of entries cleared."""
+        count = len(self._cache)
+        self._cache.clear()
+        self._cache_timestamps.clear()
+        logger.info(f"Cache cleared: {count} entries wiped")
+        return count
+
+    def get_latest_price(self, symbol: str = XAUUSD) -> Optional[float]:
         try:
             df = self.fetch_ohlcv(symbol, Timeframe.M5, lookback_days=1)
             if df is not None and not df.empty:
@@ -168,47 +190,26 @@ class ForexDataProvider:
             pass
         return None
 
-    def get_spread_estimate(self, symbol: str) -> float:
+    def get_spread_estimate(self, symbol: str = XAUUSD) -> float:
-        """Estimate typical spread from recent M5 high-low ranges."""
         try:
             df = self.fetch_ohlcv(symbol, Timeframe.M5, lookback_days=1)
             if df is not None and not df.empty:
                 return float((df["High"] - df["Low"]).median())
         except Exception:
             pass
+        info = PAIR_INFO.get(symbol, PAIR_INFO[XAUUSD])
+        return info["pip_value"]
 
-        fallback_spreads = {
+    def get_pair_info(self, symbol: str = XAUUSD) -> Dict:
-            "EURUSD=X": 0.00010, "GBPUSD=X": 0.00015,
+        info = PAIR_INFO.get(symbol, PAIR_INFO[XAUUSD])
-            "USDJPY=X": 0.010,   "USDCHF=X": 0.00015,
+        return {"symbol": symbol, **info}
-            "AUDUSD=X": 0.00015, "USDCAD=X": 0.00018,
-            "NZDUSD=X": 0.00020,
-        }
-        return fallback_spreads.get(symbol, 0.00020)
 
-    def get_pair_info(self, symbol: str) -> Dict:
+    # ── Private ────────────────────────────────────────────────────
-        """Return display name and pip value for a pair."""
-        display = SYMBOL_DISPLAY.get(symbol, symbol)
-        is_jpy = "JPY" in symbol
-        return {
-            "symbol": symbol,
-            "display": display,
-            "pip_value": 0.01 if is_jpy else 0.0001,
-            "is_jpy_pair": is_jpy,
-        }
-
-    # ── Private Methods ────────────────────────────────────────────
 
     def _throttle(self, credits: int = 1) -> None:
-        """
+        """Pro tier: 55 credits / minute. Block until budget allows."""
-        Block until `credits` API credits are available.
-
-        Twelve Data free tier: 8 credits/minute. Each symbol in any request
-        (single or batch) consumes 1 credit. This method enforces the limit
-        regardless of whether requests are batched or individual.
-        """
         while True:
             now = time.time()
-            # Evict credits older than the rate window
             while self._request_times and now - self._request_times[0] >= self.RATE_WINDOW:
                 self._request_times.popleft()
 
@@ -218,276 +219,76 @@ class ForexDataProvider:
                     self._request_times.append(time.time())
                 return
 
-            # Wait until the oldest credit expires
             wait = (self._request_times[0] + self.RATE_WINDOW) - now + 0.2
             logger.info(
                 f"Rate limit: {len(self._request_times)}/{self.RATE_LIMIT} credits used "
-                f"— waiting {wait:.1f}s for next window..."
+                f"— waiting {wait:.1f}s..."
             )
             time.sleep(wait)
 
-    def _batch_fetch_timeframe(
-        self, symbols: List[str], timeframe: Timeframe, lookback_days: int
-    ) -> None:
-        """
-        Single API call fetching all symbols for one timeframe.
-        Twelve Data returns a dict keyed by symbol when multiple are requested.
-        Parsed DataFrames are validated and stored in the cache.
-        """
-        if not self._api_key:
-            return
-
-        td_symbols = [self._to_td_symbol(s) for s in symbols]
-        interval = self.TD_INTERVALS[timeframe]
-        outputsize = min(
-            int(lookback_days * self.BARS_PER_DAY[timeframe]) + 20, 5000
-        )
-        outputsize = max(outputsize, 300)
-
-        # Each symbol in a batch request costs 1 credit
-        self._throttle(credits=len(symbols))
-
-        resp = self._session.get(
-            f"{self.TD_BASE_URL}/time_series",
-            params={
-                "symbol": ",".join(td_symbols),
-                "interval": interval,
-                "outputsize": outputsize,
-                "apikey": self._api_key,
-                "timezone": "UTC",
-                "format": "json",
-            },
-            timeout=30,
-        )
-        resp.raise_for_status()
-        data = resp.json()
-
-        # If the whole batch request returned a top-level error, raise it
-        if isinstance(data, dict) and data.get("status") == "error":
-            raise DataIntegrityError(
-                f"Batch API error ({timeframe.value}): {data.get('message', 'unknown')}"
-            )
-
-        # Batch response: {"EUR/USD": {...}, "GBP/USD": {...}, ...}
-        now = time.time()
-        for symbol, td_sym in zip(symbols, td_symbols):
-            try:
-                sym_data = data.get(td_sym, {})
-                if sym_data.get("status") == "error":
-                    logger.warning(
-                        f"Batch fetch error for {td_sym}: {sym_data.get('message')}"
-                    )
-                    continue
-
-                values = sym_data.get("values")
-                if not values:
-                    logger.warning(f"No values in batch response for {td_sym}")
-                    continue
-
-                records = [
-                    {
-                        "Datetime": pd.Timestamp(v["datetime"], tz="UTC"),
-                        "Open":   float(v["open"]),
-                        "High":   float(v["high"]),
-                        "Low":    float(v["low"]),
-                        "Close":  float(v["close"]),
-                        "Volume": float(v.get("volume") or 0),
-                    }
-                    for v in reversed(values)
-                ]
-                df = pd.DataFrame(records).set_index("Datetime")
-                df.index.name = "Datetime"
-                df = self._validate_and_clean(df, symbol, timeframe)
-
-                cache_key = f"{symbol}_{timeframe.value}"
-                self._cache[cache_key] = df
-                self._cache_timestamps[cache_key] = now
-                logger.debug(f"Batch cached {len(df)} bars for {td_sym} @ {interval}")
-
-            except Exception as e:
-                logger.warning(f"Failed to parse batch data for {td_sym}: {e}")
-
     def _fetch_twelve_data(
-        self, symbol: str, timeframe: Timeframe, lookback_days: int
+        self,
+        symbol: str,
+        timeframe: Timeframe,
+        outputsize: int,
+        start_date: Optional[str] = None,
+        end_date: Optional[str] = None,
     ) -> pd.DataFrame:
-        """
+        if self._td is None:
-        Fetch OHLCV from Twelve Data REST API.
+            raise DataIntegrityError(
-        Falls back to synthetic GBM data if API is unavailable or key is missing.
+                "No Twelve Data API key. Set TWELVE_DATA_API_KEY env var "
-        """
+                "or config.twelve_data_api_key."
-        if not self._api_key:
+            )
-            logger.warning("No Twelve Data API key — using synthetic data. "
-                           "Set TWELVE_DATA_API_KEY env var or config.twelve_data_api_key.")
-            return self._generate_synthetic_data(symbol, timeframe, lookback_days)
 
         td_symbol = self._to_td_symbol(symbol)
         interval = self.TD_INTERVALS[timeframe]
-        outputsize = min(
-            int(lookback_days * self.BARS_PER_DAY[timeframe]) + 20,
-            5000,
-        )
-        outputsize = max(outputsize, 300)
 
         self._throttle(credits=1)
 
-        try:
+        kwargs = {
-            resp = self._session.get(
-                f"{self.TD_BASE_URL}/time_series",
-                params={
             "symbol": td_symbol,
             "interval": interval,
-                    "outputsize": outputsize,
+            "outputsize": min(outputsize, 5000),
-                    "apikey": self._api_key,
             "timezone": "UTC",
-                    "format": "json",
+        }
-                },
+        if start_date:
-                timeout=15,
+            kwargs["start_date"] = start_date
-            )
+        if end_date:
-            resp.raise_for_status()
+            kwargs["end_date"] = end_date
-            data = resp.json()
 
-            if data.get("status") == "error":
+        try:
+            df = self._td.time_series(**kwargs).as_pandas()
+        except Exception as e:
             raise DataIntegrityError(
-                    f"Twelve Data API error for {td_symbol}: {data.get('message', 'unknown')}"
+                f"Twelve Data fetch failed for {td_symbol}: {e}"
-                )
+            ) from e
 
-            values = data.get("values")
+        if df is None or df.empty:
-            if not values:
             raise DataIntegrityError(f"Empty response from Twelve Data for {td_symbol}")
 
-            # Values arrive newest-first; reverse to chronological order
+        # SDK returns lowercase columns; engine expects capitalised.
-            records = [
+        df = df.rename(columns={
-                {
+            "open": "Open", "high": "High", "low": "Low",
-                    "Datetime": pd.Timestamp(v["datetime"], tz="UTC"),
+            "close": "Close", "volume": "Volume",
-                    "Open":   float(v["open"]),
+        })
-                    "High":   float(v["high"]),
+        for col in ("Open", "High", "Low", "Close", "Volume"):
-                    "Low":    float(v["low"]),
+            if col in df.columns:
-                    "Close":  float(v["close"]),
+                df[col] = pd.to_numeric(df[col], errors="coerce")
-                    "Volume": float(v.get("volume") or 0),
+        if "Volume" not in df.columns:
-                }
+            df["Volume"] = 0.0
-                for v in reversed(values)
-            ]
 
-            df = pd.DataFrame(records).set_index("Datetime")
+        # SDK returns newest-first; engine + indicators want oldest-first.
+        df = df.sort_index()
+        if df.index.tz is None:
+            df.index = df.index.tz_localize("UTC")
         df.index.name = "Datetime"
+
         logger.debug(f"Fetched {len(df)} bars for {td_symbol} @ {interval}")
         return df
 
-        except DataIntegrityError:
-            raise
-        except Exception as e:
-            logger.info(f"Twelve Data fetch failed for {symbol} ({e}), using synthetic data.")
-            return self._generate_synthetic_data(symbol, timeframe, lookback_days)
-
-    def _generate_synthetic_data(
-        self, symbol: str, timeframe: Timeframe, lookback_days: int
-    ) -> pd.DataFrame:
-        """
-        Generate realistic synthetic Forex OHLCV using GBM with mean-reversion.
-        Used as a fallback when live data is unavailable.
-        """
-        pair_params = {
-            "EURUSD=X": {"price": 1.1350, "vol": 0.07, "drift": -0.001},
-            "GBPUSD=X": {"price": 1.3200, "vol": 0.08, "drift":  0.002},
-            "USDJPY=X": {"price": 144.50, "vol": 0.09, "drift":  0.003},
-            "USDCHF=X": {"price": 0.8850, "vol": 0.07, "drift": -0.001},
-            "AUDUSD=X": {"price": 0.6550, "vol": 0.10, "drift": -0.002},
-            "USDCAD=X": {"price": 1.3650, "vol": 0.07, "drift":  0.001},
-            "NZDUSD=X": {"price": 0.6100, "vol": 0.10, "drift": -0.001},
-            "EURGBP=X": {"price": 0.8590, "vol": 0.06, "drift":  0.000},
-            "EURJPY=X": {"price": 163.90, "vol": 0.09, "drift":  0.002},
-            "GBPJPY=X": {"price": 190.70, "vol": 0.11, "drift":  0.003},
-            "AUDJPY=X": {"price": 94.60,  "vol": 0.11, "drift":  0.001},
-            "EURCHF=X": {"price": 0.9380, "vol": 0.05, "drift":  0.000},
-            "CADJPY=X": {"price": 105.80, "vol": 0.09, "drift":  0.001},
-        }
-
-        params = pair_params.get(symbol, {"price": 1.0000, "vol": 0.08, "drift": 0.000})
-        base_price = params["price"]
-        annual_vol = params["vol"]
-        annual_drift = params["drift"]
-
-        tf_minutes = {
-            Timeframe.M1: 1, Timeframe.M5: 5, Timeframe.M15: 15,
-            Timeframe.M30: 30, Timeframe.H1: 60, Timeframe.H4: 240,
-            Timeframe.D1: 1440, Timeframe.W1: 10080,
-        }
-        bar_minutes = tf_minutes.get(timeframe, 60)
-        bars_per_year = (365.25 * 24 * 60) / bar_minutes
-        dt = 1.0 / bars_per_year
-
-        n_bars = min(int(lookback_days * 24 * 60 / bar_minutes), 2000)
-        n_bars = max(n_bars, 300)
-
-        seed_base = int(hashlib.md5(symbol.encode()).hexdigest()[:8], 16)
-        time_component = int(time.time()) // 3600
-        np.random.seed((seed_base + time_component) % (2**31))
-
-        bar_vol = annual_vol * np.sqrt(dt)
-        bar_drift = annual_drift * dt
-
-        regime_len = np.random.randint(20, 80)
-        trend_strength = np.random.normal(0, 0.3)
-
-        price = base_price
-        prices = [price]
-
-        for i in range(1, n_bars):
-            if i % regime_len == 0:
-                regime_len = np.random.randint(20, 80)
-                trend_strength = np.random.normal(0, 0.3)
-
-            mr_force = -0.001 * (price - base_price) / base_price
-            ret = bar_drift + trend_strength * dt + mr_force + bar_vol * np.random.randn()
-
-            if np.random.random() < 0.02:
-                ret *= np.random.uniform(2.0, 4.0)
-
-            price *= (1 + ret)
-            prices.append(price)
-
-        prices = np.array(prices)
-        opens = np.roll(prices, 1)
-        opens[0] = prices[0]
-
-        typical_range = bar_vol * base_price * 1.5
-        highs = np.maximum(prices, opens) + np.abs(np.random.normal(0, typical_range * 0.6, n_bars))
-        lows  = np.minimum(prices, opens) - np.abs(np.random.normal(0, typical_range * 0.6, n_bars))
-        highs = np.maximum(highs, np.maximum(opens, prices))
-        lows  = np.minimum(lows,  np.minimum(opens, prices))
-
-        volume = np.random.lognormal(mean=10, sigma=0.5, size=n_bars).astype(int)
-
-        freq_map = {
-            Timeframe.M1: "1min",  Timeframe.M5: "5min",
-            Timeframe.M15: "15min", Timeframe.M30: "30min",
-            Timeframe.H1: "1h",    Timeframe.H4: "4h",
-            Timeframe.D1: "D",     Timeframe.W1: "W",
-        }
-        freq = freq_map.get(timeframe, "1h")
-        end_time = pd.Timestamp.now(tz="UTC").floor(f"{bar_minutes}min")
-        index = pd.date_range(end=end_time, periods=n_bars, freq=freq, tz="UTC")
-
-        df = pd.DataFrame({
-            "Open": opens, "High": highs, "Low": lows,
-            "Close": prices, "Volume": volume,
-        }, index=index)
-        df.index.name = "Datetime"
-
-        logger.info(
-            f"Generated {n_bars} synthetic {timeframe.value} bars for "
-            f"{SYMBOL_DISPLAY.get(symbol, symbol)} @ {prices[-1]:.5f}"
-        )
-        return df
-
     def _validate_and_clean(
         self, df: pd.DataFrame, symbol: str, timeframe: Timeframe
     ) -> pd.DataFrame:
-        """
-        Validate and clean OHLCV data.
-
-        Steps: drop all-NaN rows → ffill small gaps → remove future timestamps
-        → fix OHLC inconsistencies → remove >10-sigma outliers → sort & deduplicate.
-        """
         if df is None or df.empty:
             raise DataIntegrityError(f"Empty dataset for {symbol}")
 
@@ -497,6 +298,9 @@ class ForexDataProvider:
         df = df.ffill(limit=3)
         df = df.dropna(subset=["Close"])
 
+        df = df.sort_index()
+        df = df[~df.index.duplicated(keep="last")]
+
         now = pd.Timestamp.utcnow()
         future_mask = df.index > now + pd.Timedelta(minutes=5)
         if future_mask.any():
@@ -527,7 +331,6 @@ class ForexDataProvider:
 
         min_required = max(
             self.config.technical.ema_trend,
-            self.config.technical.ichimoku_senkou_b,
             200,
         ) + 10
 
@@ -537,9 +340,6 @@ class ForexDataProvider:
                 f"(need {min_required}). Some indicators will be partial."
             )
 
-        df = df.sort_index()
-        df = df[~df.index.duplicated(keep="last")]
-
         cleaned = original_len - len(df)
         if cleaned > 0:
             logger.info(f"{symbol}: Cleaned {cleaned}/{original_len} bars")

main.py

@@ -1,35 +1,27 @@
 #!/usr/bin/env python3
 """
-Forex Signal Intelligence Platform
+ForexBot2 — XAU/USD Signal Intelligence
-====================================
+========================================
-Main entry point. Runs full analysis and outputs formatted signal reports.
-
 Usage:
-    python main.py                          # Analyze all major pairs
+    python main.py                              # Live signal on H1
-    python main.py --pairs EURUSD GBPUSD    # Specific pairs
+    python main.py --timeframe H4 --detail      # Detailed H4 signal
-    python main.py --timeframe H4           # Different primary TF
     python main.py --json                       # JSON output
-    python main.py --watch                  # Continuous monitoring mode
 """
 
 import argparse
 import json
 import logging
 import sys
-import time
 from datetime import datetime, timezone
-from typing import List
+from typing import Dict, List
 
-# Force UTF-8 output on Windows so box-drawing / Unicode chars render correctly
+# Force UTF-8 output on Windows so box-drawing chars render
 if hasattr(sys.stdout, "reconfigure"):
     sys.stdout.reconfigure(encoding="utf-8")
 if hasattr(sys.stderr, "reconfigure"):
     sys.stderr.reconfigure(encoding="utf-8")
 
-from config import (
+from config import PlatformConfig, Timeframe, SignalStrength, XAUUSD
-    PlatformConfig, Timeframe, SignalStrength,
-    MAJOR_PAIRS, CROSS_PAIRS,
-)
 from signal_aggregator import SignalAggregator, SignalReport
 
 
@@ -38,12 +30,10 @@ from signal_aggregator import SignalAggregator, SignalReport
 # ══════════════════════════════════════════════════════════════════
 
 class Colors:
-    """ANSI color codes for terminal output."""
     RESET = "\033[0m"
     BOLD = "\033[1m"
     DIM = "\033[2m"
     UNDERLINE = "\033[4m"
-
     RED = "\033[91m"
     GREEN = "\033[92m"
     YELLOW = "\033[93m"
@@ -53,16 +43,8 @@ class Colors:
     WHITE = "\033[97m"
     GRAY = "\033[90m"
 
-    BG_RED = "\033[41m"
-    BG_GREEN = "\033[42m"
-    BG_YELLOW = "\033[43m"
-    BG_BLUE = "\033[44m"
-    BG_MAGENTA = "\033[45m"
-    BG_CYAN = "\033[46m"
-
 
 def signal_color(signal: SignalStrength) -> str:
-    """Get color for a signal strength."""
     return {
         SignalStrength.STRONG_BUY:  Colors.GREEN + Colors.BOLD,
         SignalStrength.BUY:         Colors.GREEN,
@@ -72,8 +54,7 @@ def signal_color(signal: SignalStrength) -> str:
     }.get(signal, Colors.RESET)
 
 
-def quality_color(quality: str) -> str:
+def quality_color(q: str) -> str:
-    """Get color for signal quality grade."""
     return {
         "A+": Colors.GREEN + Colors.BOLD,
         "A":  Colors.GREEN,
@@ -81,196 +62,127 @@ def quality_color(quality: str) -> str:
         "C":  Colors.YELLOW,
         "D":  Colors.GRAY,
         "F":  Colors.DIM,
-    }.get(quality, Colors.RESET)
+    }.get(q, Colors.RESET)
 
 
-def score_bar(score: float, width: int = 30) -> str:
+def print_header(title: str = "XAU/USD SIGNAL INTELLIGENCE"):
-    """Create a visual score bar from -100 to +100."""
-    normalized = (score + 100) / 200  # 0 to 1
-    filled = int(normalized * width)
-    center = width // 2
-
-    bar = ""
-    for i in range(width):
-        if i == center:
-            bar += "│"
-        elif (score >= 0 and center < i <= filled) or \
-             (score < 0 and filled <= i < center):
-            bar += "█"
-        else:
-            bar += "░"
-
-    if score >= 0:
-        color = Colors.GREEN if score > 20 else Colors.GRAY
-    else:
-        color = Colors.RED if score < -20 else Colors.GRAY
-
-    return f"{color}{bar}{Colors.RESET}"
-
-
-def print_header():
-    """Print the platform header."""
     now = datetime.now(timezone.utc).strftime("%Y-%m-%d %H:%M:%S UTC")
-    width = 80
+    print(f"\n{Colors.CYAN}{'═' * 80}{Colors.RESET}")
-
+    print(f"{Colors.BOLD}{Colors.CYAN}  {title}{Colors.RESET}")
-    print(f"\n{Colors.CYAN}{'═' * width}{Colors.RESET}")
+    print(f"{Colors.GRAY}  Time: {now}{Colors.RESET}")
-    print(f"{Colors.BOLD}{Colors.CYAN}  ╔═══════════════════════════════════════════════════════╗{Colors.RESET}")
+    print(f"{Colors.CYAN}{'═' * 80}{Colors.RESET}\n")
-    print(f"{Colors.BOLD}{Colors.CYAN}  ║   FOREX SIGNAL INTELLIGENCE PLATFORM                 ║{Colors.RESET}")
-    print(f"{Colors.BOLD}{Colors.CYAN}  ║   Multi-Layer Signal Analysis Engine                  ║{Colors.RESET}")
-    print(f"{Colors.BOLD}{Colors.CYAN}  ╚═══════════════════════════════════════════════════════╝{Colors.RESET}")
-    print(f"{Colors.GRAY}  Scan Time: {now}{Colors.RESET}")
-    print(f"{Colors.CYAN}{'═' * width}{Colors.RESET}\n")
 
 
-def print_summary_table(reports: List[SignalReport]):
+# ── Live signal renderer ───────────────────────────────────────────
-    """Print a summary table of all pairs."""
-    print(f"{Colors.BOLD}{Colors.WHITE}  ┌─────────────┬───────────┬──────────────────────────────────┬────────┬────────────┐{Colors.RESET}")
-    print(f"{Colors.BOLD}{Colors.WHITE}  │ Pair        │ Signal    │ Score                            │ Grade  │ Confidence │{Colors.RESET}")
-    print(f"{Colors.BOLD}{Colors.WHITE}  ├─────────────┼───────────┼──────────────────────────────────┼────────┼────────────┤{Colors.RESET}")
 
-    for r in reports:
+_INDICATOR_PREFIXES = (
-        sig = r.final_signal.name.replace("_", " ")
+    "EMA Trend", "RSI", "Bollinger Bands", "Divergence",
-        sc = signal_color(r.final_signal)
+    "Candlestick", "ADX", "ATR", "News", "Regime",
-        qc = quality_color(r.signal_quality)
+)
-        bar = score_bar(r.final_score, 30)
-
-        pair = f"{r.display_name:<11}"
-        signal_str = f"{sc}{sig:<9}{Colors.RESET}"
-        grade_str = f"{qc}{r.signal_quality:<6}{Colors.RESET}"
-        conf_str = f"{r.final_confidence:5.1f}%"
-
-        # Add warning indicator
-        warn = ""
-        if r.event_risk == "danger":
-            warn = f" {Colors.RED}⚠{Colors.RESET}"
-        elif r.event_risk == "caution":
-            warn = f" {Colors.YELLOW}△{Colors.RESET}"
-
-        print(f"  │ {pair} │ {signal_str} │ {bar} │ {grade_str} │ {conf_str:>8}{warn}  │")
-
-    print(f"{Colors.BOLD}{Colors.WHITE}  └─────────────┴───────────┴──────────────────────────────────┴────────┴────────────┘{Colors.RESET}")
-
-    # Legend
-    print(f"\n{Colors.GRAY}  Legend: "
-          f"{Colors.GREEN}██ Bullish{Colors.GRAY} | "
-          f"{Colors.RED}██ Bearish{Colors.GRAY} | "
-          f"│ Center (0) | "
-          f"{Colors.RED}⚠ Event Risk{Colors.GRAY} | "
-          f"{Colors.YELLOW}△ Caution{Colors.RESET}")
 
 
-def print_detailed_report(report: SignalReport):
+def _compact_reason(reason: str) -> str:
-    """Print detailed analysis for a single pair."""
+    """Shorten a verbose reason string for the compact view."""
+    s = reason
+    if s.startswith("STRONG: "):
+        s = s[8:]
+    # Strip the leading "<Indicator>: " prefix — it's always an indicator name
+    for prefix in _INDICATOR_PREFIXES:
+        if s.startswith(prefix + ": "):
+            s = s[len(prefix) + 2:]
+            break
+    # Drop trailing detail after first ' | '
+    if " | " in s:
+        s = s.split(" | ", 1)[0]
+    return s.strip()
+
+
+def print_signal_report(report: SignalReport, detail: bool = False):
     sc = signal_color(report.final_signal)
     qc = quality_color(report.signal_quality)
     sig_name = report.final_signal.name.replace("_", " ")
 
-    print(f"\n{Colors.CYAN}{'─' * 80}{Colors.RESET}")
+    # ── 1. DECISION ────────────────────────────────────────────────
-    print(f"  {Colors.BOLD}{Colors.WHITE}{report.display_name}{Colors.RESET}"
+    mtf_tag = (f"{Colors.GREEN}MTF✓{Colors.RESET}" if report.mtf_agreement
-          f"  │  Price: {Colors.BOLD}{report.current_price:.5f}{Colors.RESET}"
+               else f"{Colors.YELLOW}MTF✗{Colors.RESET}")
-          f"  │  Regime: {Colors.CYAN}{report.market_regime}{Colors.RESET}"
+    print(f"  {sc}▶ {sig_name}{Colors.RESET}  "
-          f"  │  Vol: {report.volatility_regime}")
+          f"{qc}{report.signal_quality}{Colors.RESET} · "
+          f"{sc}{report.final_score:+.1f}{Colors.RESET} · "
+          f"{report.final_confidence:.0f}% · {mtf_tag}")
+    print(f"  {Colors.GRAY}{report.action_recommendation}{Colors.RESET}")
 
-    print(f"\n  {Colors.BOLD}SIGNAL:{Colors.RESET} {sc}{sig_name}{Colors.RESET}"
+    # ── 2. TRADE SETUP (if signal is actionable) ───────────────────
-          f"  │  Score: {sc}{report.final_score:+.1f}{Colors.RESET}"
+    if report.trade_setup:
-          f"  │  Grade: {qc}{report.signal_quality}{Colors.RESET}"
+        ts = report.trade_setup
-          f"  │  Confidence: {report.final_confidence:.0f}%")
+        dc = Colors.GREEN if ts.direction == "BUY" else Colors.RED
+        print(f"\n  {Colors.BOLD}Setup{Colors.RESET}  "
+              f"{dc}{ts.direction}{Colors.RESET} @ {Colors.BOLD}{ts.entry_price:.3f}{Colors.RESET}  "
+              f"→ SL {Colors.RED}{ts.stop_loss:.3f}{Colors.RESET} "
+              f"({ts.risk_pips:.1f}p)  "
+              f"→ TP {Colors.GREEN}{ts.take_profit_1:.3f}{Colors.RESET} "
+              f"({ts.reward_pips:.1f}p)  "
+              f"RR 1:{ts.risk_reward:.1f}")
 
-    print(f"  {Colors.BOLD}Action:{Colors.RESET} {report.action_recommendation}")
+    # ── 3. CONTEXT (price · regime · vol · MTF breakdown) ──────────
+    print(f"\n  {Colors.WHITE}{report.display_name}{Colors.RESET} "
+          f"{Colors.BOLD}{report.current_price:.3f}{Colors.RESET} · "
+          f"{Colors.CYAN}{report.market_regime}{Colors.RESET} · "
+          f"vol {report.volatility_regime}")
 
-    # Multi-timeframe breakdown
     if report.mtf_signals:
-        print(f"\n  {Colors.BOLD}Multi-Timeframe:{Colors.RESET}", end="")
+        parts = []
-        mtf_agree = f" {Colors.GREEN}✓ ALIGNED{Colors.RESET}" if report.mtf_agreement else f" {Colors.YELLOW}✗ DIVERGENT{Colors.RESET}"
-        print(mtf_agree)
         for tf, sig in report.mtf_signals.items():
             tf_sc = signal_color(sig)
-            tf_score = report.mtf_scores.get(tf, 0)
+            sc_val = report.mtf_scores.get(tf, 0)
-            print(f"    {tf:>5}: {tf_sc}{sig.name.replace('_', ' '):<12}{Colors.RESET} ({tf_score:+.1f})")
+            parts.append(f"{tf} {tf_sc}{sig.name.replace('_',' ')}{Colors.RESET} {sc_val:+.1f}")
+        print(f"  {Colors.GRAY}MTF{Colors.RESET}    " + "  ·  ".join(parts))
 
-    # Indicators detail
+    # ── 4. WHY — top 3 each side, compact ──────────────────────────
-    if report.primary_analysis:
+    bulls = [_compact_reason(r) for r in report.bullish_reasons[:3]]
-        print(f"\n  {Colors.BOLD}Indicators ({report.primary_analysis.timeframe}):{Colors.RESET}")
+    bears = [_compact_reason(r) for r in report.bearish_reasons[:3]]
+    if bulls or bears:
+        print()
+        rows = max(len(bulls), len(bears))
+        for i in range(rows):
+            left  = f"{Colors.GREEN}+{Colors.RESET} {bulls[i]}" if i < len(bulls) else ""
+            right = f"{Colors.RED}-{Colors.RESET} {bears[i]}" if i < len(bears) else ""
+            print(f"  {left:<60}  {right}")
+
+    # ── 5. WARNINGS / NEWS / EVENTS ────────────────────────────────
+    if report.warnings:
+        for w in report.warnings:
+            print(f"  {Colors.YELLOW}⚠{Colors.RESET}  {w}")
+    if report.event_description and report.event_risk != "clear":
+        print(f"  {Colors.YELLOW}📅{Colors.RESET} {report.event_description}")
+    if report.news_description and report.news_description != "Neutral news sentiment":
+        print(f"  {Colors.MAGENTA}📰{Colors.RESET} {report.news_description}")
+
+    # ── 6. DETAIL (only with --detail) ─────────────────────────────
+    if detail and report.primary_analysis:
+        print(f"\n  {Colors.BOLD}Indicators ({report.primary_analysis.timeframe}){Colors.RESET}")
         for name, ind in report.primary_analysis.indicators.items():
             if name.startswith("_") or ind.weight == 0:
                 continue
             isc = signal_color(ind.signal)
-            ind_name = f"{ind.name:<18}"
+            print(f"    {ind.name:<16} {isc}{ind.signal.name.replace('_',' '):<11}{Colors.RESET} "
-            print(f"    {ind_name} {isc}{ind.signal.name.replace('_', ' '):<12}{Colors.RESET}"
+                  f"{ind.confidence:>3.0f}% {Colors.GRAY}│ {ind.description}{Colors.RESET}")
-                  f" ({ind.confidence:.0f}%) │ {Colors.GRAY}{ind.description}{Colors.RESET}")
 
-    # Bullish reasons
-    if report.bullish_reasons:
-        print(f"\n  {Colors.GREEN}▲ Bullish Factors:{Colors.RESET}")
-        for reason in report.bullish_reasons[:5]:
-            print(f"    {Colors.GREEN}+{Colors.RESET} {reason}")
-
-    # Bearish reasons
-    if report.bearish_reasons:
-        print(f"\n  {Colors.RED}▼ Bearish Factors:{Colors.RESET}")
-        for reason in report.bearish_reasons[:5]:
-            print(f"    {Colors.RED}-{Colors.RESET} {reason}")
-
-    # Warnings
-    if report.warnings:
-        print(f"\n  {Colors.YELLOW}⚠ Warnings:{Colors.RESET}")
-        for warning in report.warnings:
-            print(f"    {Colors.YELLOW}!{Colors.RESET} {warning}")
-
-    # News context
-    if report.news_description and report.news_description != "Neutral news sentiment":
-        print(f"\n  {Colors.MAGENTA}📰 News:{Colors.RESET} {report.news_description}")
-    if report.event_description and report.event_risk != "clear":
-        print(f"  {Colors.YELLOW}📅 Event:{Colors.RESET} {report.event_description}")
-
-    # Trade setup
-    if report.trade_setup:
-        ts = report.trade_setup
-        dir_color = Colors.GREEN if ts.direction == "BUY" else Colors.RED
-        print(f"\n  {Colors.BOLD}Trade Setup:{Colors.RESET}")
-        print(f"    Direction:     {dir_color}{ts.direction}{Colors.RESET}")
-        print(f"    Entry:         {ts.entry_price:.5f}")
-        print(f"    Stop Loss:     {Colors.RED}{ts.stop_loss:.5f}{Colors.RESET} ({ts.risk_pips:.1f} pips)")
-        print(f"    Take Profit 1: {Colors.GREEN}{ts.take_profit_1:.5f}{Colors.RESET} ({ts.reward_pips:.1f} pips)")
-        print(f"    Take Profit 2: {Colors.GREEN}{ts.take_profit_2:.5f}{Colors.RESET}")
-        print(f"    Risk:Reward:   1:{ts.risk_reward:.1f}")
-
-    # Key levels
-    if report.primary_analysis and report.primary_analysis.key_levels:
         levels = report.primary_analysis.key_levels
-        print(f"\n  {Colors.BOLD}Key Levels:{Colors.RESET}")
+        if levels:
-        level_names = ["r3", "r2", "r1", "pivot", "s1", "s2", "s3"]
+            swing_keys = sorted(
-        for lvl in level_names:
+                [k for k in levels if k.startswith("swing_")],
-            if lvl in levels:
+                key=lambda k: levels[k], reverse=True,
-                val = levels[lvl]
+            )
-                if val > report.current_price:
+            if swing_keys:
-                    color = Colors.RED  # resistance
+                print(f"\n  {Colors.BOLD}Structure Levels{Colors.RESET}")
-                else:
+                for k in swing_keys:
-                    color = Colors.GREEN  # support
+                    val = levels[k]
-                label = lvl.upper()
+                    color = Colors.RED if val > report.current_price else Colors.GREEN
-                print(f"    {label:>6}: {color}{val:.5f}{Colors.RESET}")
+                    label = k.replace("_", " ").title()
-
+                    print(f"    {label:>20}  {color}{val:.3f}{Colors.RESET}")
-
-def print_event_calendar(reports: List[SignalReport]):
-    """Print upcoming economic events affecting watched pairs."""
-    events_shown = set()
-    has_events = False
-
-    for report in reports:
-        if report.event_risk != "clear" and report.event_description not in events_shown:
-            if not has_events:
-                print(f"\n{Colors.YELLOW}{Colors.BOLD}  📅 ECONOMIC CALENDAR ALERTS{Colors.RESET}")
-                has_events = True
-            events_shown.add(report.event_description)
-            risk_color = Colors.RED if report.event_risk == "danger" else Colors.YELLOW
-            print(f"    {risk_color}● {report.event_description}{Colors.RESET} → affects {report.display_name}")
-
-    if not has_events:
-        print(f"\n{Colors.GRAY}  📅 No high-impact events in the near term{Colors.RESET}")
 
 
 def report_to_dict(report: SignalReport) -> dict:
-    """Convert a SignalReport to a JSON-serializable dict."""
     d = {
         "symbol": report.symbol,
         "display_name": report.display_name,
@@ -305,100 +217,60 @@ def report_to_dict(report: SignalReport) -> dict:
             "reward_pips": round(ts.reward_pips, 1),
         }
     if report.primary_analysis:
-        d["indicators"] = {}
+        d["indicators"] = {
-        for name, ind in report.primary_analysis.indicators.items():
+            name: {
-            if not name.startswith("_"):
-                d["indicators"][name] = {
                 "signal": ind.signal.name,
                 "confidence": round(ind.confidence, 1),
                 "description": ind.description,
             }
+            for name, ind in report.primary_analysis.indicators.items()
+            if not name.startswith("_")
+        }
         d["key_levels"] = {
             k: round(v, 5) for k, v in report.primary_analysis.key_levels.items()
         }
     return d
 
 
+
 # ══════════════════════════════════════════════════════════════════
-#  MAIN
+#  CLI
 # ══════════════════════════════════════════════════════════════════
 
 def parse_args():
     parser = argparse.ArgumentParser(
-        description="Forex Signal Intelligence Platform",
+        description="ForexBot2 — XAU/USD Signal Intelligence",
         formatter_class=argparse.RawDescriptionHelpFormatter,
         epilog="""
 Examples:
-  python main.py                        Analyze all major pairs (H1)
+  python main.py                          Live signal (H1, summary)
-  python main.py --pairs EURUSD GBPUSD  Analyze specific pairs
+  python main.py --detail                 Live signal with full breakdown
-  python main.py --timeframe H4         Use 4-hour as primary TF
+  python main.py --timeframe H4 --detail  H4 timeframe
-  python main.py --all                  Include cross pairs
+  python main.py --json                   JSON output
-  python main.py --json                 Output as JSON
-  python main.py --detail               Show detailed analysis per pair
-  python main.py --watch                Continuous monitoring (5min refresh)
         """,
     )
-    parser.add_argument(
+    parser.add_argument("--timeframe", "-tf", default="H1",
-        "--pairs", nargs="+", default=None,
-        help="Specific pairs to analyze (e.g., EURUSD GBPUSD)",
-    )
-    parser.add_argument(
-        "--timeframe", "-tf", default="H1",
                         choices=["M5", "M15", "M30", "H1", "H4", "D1"],
-        help="Primary analysis timeframe (default: H1)",
+                        help="Primary analysis timeframe (default: H1)")
-    )
+    parser.add_argument("--detail", action="store_true",
-    parser.add_argument(
+                        help="Show full indicator breakdown")
-        "--all", action="store_true",
+    parser.add_argument("--json", action="store_true",
-        help="Include cross pairs in analysis",
+                        help="Output as JSON")
-    )
+    parser.add_argument("--verbose", "-v", action="store_true",
-    parser.add_argument(
+                        help="Debug logging")
-        "--json", action="store_true",
+
-        help="Output results as JSON",
-    )
-    parser.add_argument(
-        "--detail", action="store_true",
-        help="Show detailed analysis for each pair",
-    )
-    parser.add_argument(
-        "--top", type=int, default=None,
-        help="Show only top N signals by strength",
-    )
-    parser.add_argument(
-        "--watch", action="store_true",
-        help="Continuous monitoring mode (refreshes every 5 min)",
-    )
-    parser.add_argument(
-        "--min-grade", default="F",
-        choices=["A+", "A", "B", "C", "D", "F"],
-        help="Minimum signal quality to display",
-    )
-    parser.add_argument(
-        "--verbose", "-v", action="store_true",
-        help="Enable debug logging",
-    )
     return parser.parse_args()
 
 
-def symbol_from_name(name: str) -> str:
+def build_config(args) -> PlatformConfig:
-    """Convert pair name like 'EURUSD' to Yahoo symbol 'EURUSD=X'."""
-    name = name.upper().replace("/", "").replace("_", "")
-    if not name.endswith("=X"):
-        name += "=X"
-    return name
-
-
-def run_analysis(args) -> List[SignalReport]:
-    """Run a single analysis cycle."""
     config = PlatformConfig()
 
-    # Configure timeframe
     tf_map = {
         "M5": Timeframe.M5, "M15": Timeframe.M15, "M30": Timeframe.M30,
         "H1": Timeframe.H1, "H4": Timeframe.H4, "D1": Timeframe.D1,
     }
     config.primary_timeframe = tf_map[args.timeframe]
 
-    # Configure confirmation timeframes based on primary
     tf_confirmations = {
         "M5":  [Timeframe.M1,  Timeframe.M15],
         "M15": [Timeframe.M5,  Timeframe.H1],
@@ -411,34 +283,28 @@ def run_analysis(args) -> List[SignalReport]:
         args.timeframe, [Timeframe.M15, Timeframe.H4]
     )
 
-    # Configure pairs
+    return config
-    if args.pairs:
-        config.watch_pairs = [symbol_from_name(p) for p in args.pairs]
-    elif args.all:
-        config.watch_pairs = MAJOR_PAIRS + CROSS_PAIRS
-    else:
-        config.watch_pairs = MAJOR_PAIRS
 
-    # Run analysis
+
+def run_live(args, config: PlatformConfig):
+    print(f"{Colors.GRAY}  Fetching market data and running analysis...{Colors.RESET}\n")
     aggregator = SignalAggregator(config)
-    reports = aggregator.analyze_all_pairs()
+    report = aggregator.analyze_pair()
 
-    # Filter by grade
+    if args.json:
-    grade_order = ["A+", "A", "B", "C", "D", "F"]
+        print(json.dumps(report_to_dict(report), indent=2))
-    min_idx = grade_order.index(args.min_grade) if args.min_grade in grade_order else len(grade_order) - 1
+        return
-    reports = [r for r in reports if grade_order.index(r.signal_quality) <= min_idx]
 
-    # Top N
+    print_header("XAU/USD SIGNAL INTELLIGENCE")
-    if args.top:
+    print_signal_report(report, detail=args.detail)
-        reports = reports[:args.top]
+    print(f"\n{Colors.CYAN}{'═' * 80}{Colors.RESET}")
-
+    print(f"{Colors.GRAY}  Disclaimer: For educational/research purposes only.{Colors.RESET}")
-    return reports
+    print(f"{Colors.CYAN}{'═' * 80}{Colors.RESET}\n")
 
 
 def main():
     args = parse_args()
 
-    # Setup logging
     log_level = logging.DEBUG if args.verbose else logging.WARNING
     logging.basicConfig(
         level=log_level,
@@ -446,63 +312,8 @@ def main():
         datefmt="%H:%M:%S",
     )
 
-    if args.watch:
+    config = build_config(args)
-        # Continuous monitoring mode
+    run_live(args, config)
-        print(f"{Colors.BOLD}Starting continuous monitoring mode (Ctrl+C to stop)...{Colors.RESET}")
-        while True:
-            try:
-                reports = run_analysis(args)
-                if args.json:
-                    output = [report_to_dict(r) for r in reports]
-                    print(json.dumps(output, indent=2))
-                else:
-                    print("\033[2J\033[H")  # clear screen
-                    print_header()
-                    print_summary_table(reports)
-                    print_event_calendar(reports)
-                    if args.detail:
-                        for r in reports:
-                            print_detailed_report(r)
-                    print(f"\n{Colors.GRAY}  Next refresh in 5 minutes...{Colors.RESET}")
-
-                time.sleep(300)
-            except KeyboardInterrupt:
-                print(f"\n{Colors.YELLOW}Monitoring stopped.{Colors.RESET}")
-                break
-    else:
-        # Single run
-        print(f"{Colors.GRAY}  Fetching market data and running analysis...{Colors.RESET}\n")
-        reports = run_analysis(args)
-
-        if args.json:
-            output = [report_to_dict(r) for r in reports]
-            print(json.dumps(output, indent=2))
-        else:
-            print_header()
-            if reports:
-                print_summary_table(reports)
-                print_event_calendar(reports)
-
-                if args.detail:
-                    for r in reports:
-                        print_detailed_report(r)
-                else:
-                    # Show detail for top signals only
-                    actionable = [r for r in reports if r.signal_quality in ("A+", "A", "B")]
-                    if actionable:
-                        print(f"\n{Colors.BOLD}{Colors.WHITE}  ═══ ACTIONABLE SIGNALS (Grade B or above) ═══{Colors.RESET}")
-                        for r in actionable:
-                            print_detailed_report(r)
-                    else:
-                        print(f"\n{Colors.GRAY}  No high-grade signals detected. "
-                              f"Use --detail for full breakdown or --min-grade C to see more.{Colors.RESET}")
-            else:
-                print(f"{Colors.YELLOW}  No data available. Check your internet connection.{Colors.RESET}")
-
-            print(f"\n{Colors.CYAN}{'═' * 80}{Colors.RESET}")
-            print(f"{Colors.GRAY}  Disclaimer: This tool is for educational/research purposes only.{Colors.RESET}")
-            print(f"{Colors.GRAY}  Past performance ≠ future results. Always use proper risk management.{Colors.RESET}")
-            print(f"{Colors.CYAN}{'═' * 80}{Colors.RESET}\n")
 
 
 if __name__ == "__main__":

news_engine.py

@@ -9,6 +9,8 @@ A perfect technical setup can be destroyed by an unexpected NFP print.
 """
 
 import logging
+import re
+import time
 import xml.etree.ElementTree as ET
 from dataclasses import dataclass, field
 from datetime import datetime, timedelta, timezone
@@ -81,17 +83,26 @@ class NewsEngine:
     # Keywords that indicate directional sentiment
     BULLISH_KEYWORDS = {
         "rate hike", "hawkish", "strong growth", "beat expectations",
-        "above forecast", "surplus", "expansion", "employment gains",
+        "above forecast", "above estimate", "above expectation",
+        "surplus", "expansion", "employment gains",
         "strong jobs", "gdp growth", "inflation rising", "tightening",
         "positive surprise", "record high", "bullish", "rally",
         "stronger than expected", "upbeat",
+        "beats", "surges", "soars", "jumps", "climbs", "rises",
+        "higher than expected", "better than expected",
+        "strong data", "robust", "resilient", "accelerates",
     }
 
     BEARISH_KEYWORDS = {
         "rate cut", "dovish", "recession", "slowdown", "contraction",
-        "below forecast", "deficit", "miss expectations", "weak data",
+        "below forecast", "below estimate", "below expectation",
+        "deficit", "miss expectations", "weak data",
         "unemployment rise", "negative surprise", "bearish", "selloff",
         "weaker than expected", "downbeat", "easing", "stimulus",
+        "misses", "slides", "drops", "falls", "plunges", "tumbles",
+        "weakens", "slumps", "declines", "retreats",
+        "lower than expected", "worse than expected",
+        "disappointing", "soft data",
     }
 
     # Currency-specific keywords for headline parsing
@@ -104,6 +115,7 @@ class NewsEngine:
         "AUD": ["aussie", "rba", "australian", "australia"],
         "CAD": ["loonie", "boc", "canadian", "canada"],
         "NZD": ["kiwi", "rbnz", "new zealand"],
+        "XAU": ["gold", "xau", "precious metal", "safe haven", "bullion"],
     }
 
     # High-impact event types (these move markets significantly)
@@ -114,6 +126,8 @@ class NewsEngine:
         "boe rate", "rba rate", "snb rate", "boc rate",
     }
 
+    _calendar_backoff_until: float = 0.0
+
     def __init__(self, config: NewsConfig):
         self.config = config
         self._session = requests.Session()
@@ -187,15 +201,21 @@ class NewsEngine:
 
         return context
 
+    @staticmethod
+    def _parse_pair(pair_symbol: str) -> Tuple[str, str]:
+        if "/" in pair_symbol:
+            parts = pair_symbol.split("/")
+            return parts[0], parts[1]
+        clean = pair_symbol.replace("=X", "")
+        return clean[:3], clean[3:]
+
     def get_event_risk_level(self, pair_symbol: str, context: FundamentalContext) -> Tuple[str, str]:
         """
         Get event risk level for a specific pair.
         Returns (risk_level, description).
         risk_level: "clear", "caution", "danger"
         """
-        clean = pair_symbol.replace("=X", "")
+        base, quote = self._parse_pair(pair_symbol)
-        base = clean[:3]
-        quote = clean[3:]
 
         # Check upcoming high-impact events
         for event in context.events_upcoming_1h:
@@ -222,9 +242,7 @@ class NewsEngine:
         Returns (score, description) where score is -1 to +1.
         Positive = bullish for the pair, negative = bearish.
         """
-        clean = pair_symbol.replace("=X", "")
+        base, quote = self._parse_pair(pair_symbol)
-        base = clean[:3]
-        quote = clean[3:]
 
         base_sent = context.news_sentiment.get(base, 0.0)
         quote_sent = context.news_sentiment.get(quote, 0.0)
@@ -247,18 +265,27 @@ class NewsEngine:
 
     # ── Private Methods ────────────────────────────────────────────
 
+    # Maps country names in MyFXBook titles to currency codes
+    _COUNTRY_TO_CURRENCY = {
+        "united states": "USD", "euro area": "EUR", "united kingdom": "GBP",
+        "japan": "JPY", "switzerland": "CHF", "australia": "AUD",
+        "canada": "CAD", "new zealand": "NZD",
+    }
+
     def _fetch_economic_calendar(self, currencies: List[str]) -> List[EconomicEvent]:
         """
         Fetch economic calendar events from a free API.
+        Tries Forex Factory first, falls back to MyFXBook RSS.
         Returns an empty list (with a log warning) if unavailable —
         we never generate fake events since that causes constant false alarms.
         """
+        for fetcher in (self._try_fetch_calendar_api, self._try_fetch_myfxbook_rss):
             try:
-            events = self._try_fetch_calendar_api()
+                events = fetcher()
                 if events:
                     return [e for e in events if e.currency in currencies]
             except Exception as e:
-            logger.debug(f"Calendar API fetch exception: {e}")
+                logger.debug(f"Calendar fetch exception ({fetcher.__name__}): {e}")
 
         logger.info(
             "Economic calendar data unavailable. "
@@ -268,13 +295,38 @@ class NewsEngine:
 
     def _try_fetch_calendar_api(self) -> List[EconomicEvent]:
         """Attempt to fetch from the Forex Factory free JSON calendar."""
-        events = []
+        now = time.time()
+        if now < NewsEngine._calendar_backoff_until:
+            wait = NewsEngine._calendar_backoff_until - now
+            logger.debug(f"Calendar API in backoff for {wait:.0f}s more")
+            return []
+
         url = "https://nfs.faireconomy.media/ff_calendar_thisweek.json"
+
+        for attempt in range(3):
+            try:
                 resp = self._session.get(url, timeout=10)
+            except requests.RequestException as e:
+                logger.debug(f"Calendar request failed (attempt {attempt + 1}): {e}")
+                if attempt < 2:
+                    time.sleep(2 ** attempt)
+                continue
+
+            if resp.status_code == 429:
+                retry_after = int(resp.headers.get("Retry-After", 300))
+                retry_after = min(retry_after, 600)
+                NewsEngine._calendar_backoff_until = time.time() + retry_after
+                logger.warning(
+                    f"Calendar API rate-limited (429), backing off {retry_after}s"
+                )
+                return []
+
             if resp.status_code != 200:
+                logger.debug(f"Calendar API returned {resp.status_code}")
                 return []
 
             data = resp.json()
+            events = []
             impact_map = {"High": "high", "Medium": "medium", "Low": "low"}
 
             for item in data:
@@ -300,6 +352,79 @@ class NewsEngine:
 
             return events
 
+        return []
+
+    def _try_fetch_myfxbook_rss(self) -> List[EconomicEvent]:
+        """Fallback calendar source: MyFXBook economic calendar RSS."""
+        url = "https://www.myfxbook.com/rss/forex-economic-calendar-events"
+        try:
+            resp = self._session.get(url, timeout=10)
+        except requests.RequestException as e:
+            logger.debug(f"MyFXBook calendar request failed: {e}")
+            return []
+
+        if resp.status_code != 200:
+            return []
+
+        try:
+            root = ET.fromstring(resp.content)
+        except ET.ParseError:
+            return []
+
+        events = []
+        for item in root.iter("item"):
+            try:
+                title_el = item.find("title")
+                pub_el = item.find("pubDate")
+                desc_el = item.find("description")
+                if title_el is None or title_el.text is None:
+                    continue
+
+                title = title_el.text.strip()
+
+                currency = ""
+                title_lower = title.lower()
+                for country, cur in self._COUNTRY_TO_CURRENCY.items():
+                    if title_lower.startswith(country):
+                        currency = cur
+                        title = title[len(country):].strip()
+                        break
+                if not currency:
+                    continue
+
+                event_time = self._parse_rss_date(
+                    pub_el.text if pub_el is not None else None
+                )
+                if event_time is None:
+                    continue
+
+                impact = "low"
+                desc_text = desc_el.text if desc_el is not None and desc_el.text else ""
+                impact_match = re.search(r"sprite-(high|medium|low)-impact", desc_text)
+                if impact_match:
+                    impact = impact_match.group(1)
+
+                cells = re.findall(r"<td[^>]*>(.*?)</td>", desc_text, re.DOTALL)
+                cells = [re.sub(r"<[^>]+>", "", c).strip() for c in cells]
+                previous = cells[2] if len(cells) > 2 and cells[2] else None
+                forecast = cells[3] if len(cells) > 3 and cells[3] else None
+                actual = cells[4] if len(cells) > 4 and cells[4] else None
+
+                events.append(EconomicEvent(
+                    time=event_time,
+                    currency=currency,
+                    impact=impact,
+                    event=title,
+                    forecast=forecast,
+                    previous=previous,
+                    actual=actual,
+                ))
+            except Exception:
+                continue
+
+        logger.debug(f"MyFXBook calendar: parsed {len(events)} events")
+        return events
+
     def _fetch_news_sentiment(self) -> List[NewsSentiment]:
         """Fetch and analyze news headlines for sentiment."""
         headlines = []
@@ -396,6 +521,8 @@ class NewsEngine:
         formats = [
             "%a, %d %b %Y %H:%M:%S %z",   # RFC 822 with numeric tz offset
             "%a, %d %b %Y %H:%M:%S %Z",   # RFC 822 with tz abbreviation (GMT/UTC)
+            "%a, %d %b %Y %H:%M %Z",      # RFC 822 without seconds (MyFXBook)
+            "%a, %d %b %Y %H:%M %z",      # RFC 822 without seconds, numeric offset
             "%Y-%m-%dT%H:%M:%S%z",         # ISO 8601 with offset
             "%Y-%m-%d %H:%M:%S",           # ISO 8601 naive (assume UTC)
         ]
@@ -413,15 +540,8 @@ class NewsEngine:
 
     @staticmethod
     def _get_affected_pairs(currency: str) -> List[str]:
-        """Get all major pairs affected by a currency event."""
+        _MAP = {
-        pairs = {
+            "USD": ["XAU/USD"],
-            "USD": ["EURUSD=X", "GBPUSD=X", "USDJPY=X", "USDCHF=X", "AUDUSD=X", "USDCAD=X", "NZDUSD=X"],
+            "XAU": ["XAU/USD"],
-            "EUR": ["EURUSD=X", "EURGBP=X", "EURJPY=X", "EURCHF=X"],
-            "GBP": ["GBPUSD=X", "EURGBP=X", "GBPJPY=X"],
-            "JPY": ["USDJPY=X", "EURJPY=X", "GBPJPY=X", "AUDJPY=X", "CADJPY=X"],
-            "CHF": ["USDCHF=X", "EURCHF=X"],
-            "AUD": ["AUDUSD=X", "AUDJPY=X"],
-            "CAD": ["USDCAD=X", "CADJPY=X"],
-            "NZD": ["NZDUSD=X"],
         }
-        return pairs.get(currency, [])
+        return _MAP.get(currency, [])

requirements.txt

@@ -0,0 +1,6 @@
+numpy>=1.24
+pandas>=2.0
+requests>=2.28
+twelvedata>=1.2.0
+python-telegram-bot>=20.0
+websockets>=12.0

script.txt

@@ -1 +0,0 @@
-pip install pandas numpy requests python-dateutil -q

test_ws_dual.py

@@ -0,0 +1,108 @@
+#!/usr/bin/env python3
+"""Test whether Twelve Data allows 2 simultaneous WebSocket connections for EUR/USD.
+
+Uses the twelvedata SDK which handles the WS protocol correctly.
+Runs connection 1, then adds connection 2 after 10s, monitors for 30s total.
+"""
+
+import threading
+import time
+
+from twelvedata import TDClient
+
+API_KEY = "f563b22ffdfd4924955decbe1a8c8701"
+SYMBOL = "EUR/USD"
+DURATION = 30
+
+results = {
+    "WS-1": {"ticks": 0, "errors": [], "alive": False},
+    "WS-2": {"ticks": 0, "errors": [], "alive": False},
+}
+
+
+def make_handlers(label):
+    def on_event(event):
+        if event.get("event") == "price":
+            results[label]["ticks"] += 1
+            price = event.get("price", "?")
+            print(f"[{label}] tick #{results[label]['ticks']:3d}  price={price}")
+
+    def on_error(error):
+        results[label]["errors"].append(str(error))
+        print(f"[{label}] ERROR: {error}")
+
+    def on_close(status):
+        results[label]["alive"] = False
+        print(f"[{label}] CLOSED (status={status})")
+
+    def on_connect():
+        results[label]["alive"] = True
+        print(f"[{label}] CONNECTED — subscribed to {SYMBOL}")
+
+    return on_event, on_error, on_close, on_connect
+
+
+def run_ws(label):
+    td = TDClient(apikey=API_KEY)
+    on_event, on_error, on_close, on_connect = make_handlers(label)
+    ws = td.websocket(
+        symbols=SYMBOL,
+        on_event=on_event,
+        on_error=on_error,
+        on_close=on_close,
+        on_connect=on_connect,
+    )
+    ws.connect()
+    return ws
+
+
+def main():
+    print(f"Testing dual WebSocket for {SYMBOL} ({DURATION}s)\n")
+
+    print("Starting WS-1...")
+    ws1 = run_ws("WS-1")
+
+    time.sleep(10)
+    print(f"\nWS-1 ticks so far: {results['WS-1']['ticks']}")
+    print("Starting WS-2...\n")
+    ws2 = run_ws("WS-2")
+
+    remaining = DURATION - 10
+    time.sleep(remaining)
+
+    print("\nStopping connections...")
+    try:
+        ws1.disconnect()
+    except Exception:
+        pass
+    try:
+        ws2.disconnect()
+    except Exception:
+        pass
+
+    time.sleep(1)
+
+    print("\n" + "=" * 50)
+    print("RESULTS")
+    print("=" * 50)
+    for label, info in sorted(results.items()):
+        status = "alive" if info["alive"] else "dead"
+        errs = info["errors"] or ["none"]
+        print(f"  {label}: {status}  ticks={info['ticks']}  errors={errs[0]}")
+
+    ws1_ok = results["WS-1"]["ticks"] > 0 and not results["WS-1"]["errors"]
+    ws2_ok = results["WS-2"]["ticks"] > 0 and not results["WS-2"]["errors"]
+    ws1_died = results["WS-1"]["alive"] is False and results["WS-1"]["ticks"] > 0
+
+    if ws1_ok and ws2_ok:
+        print("\n✓ Both connections worked — dual WS is supported")
+    elif ws2_ok and ws1_died:
+        print("\n✗ WS-2 replaced WS-1 — only 1 connection allowed at a time")
+    elif ws1_ok and not ws2_ok:
+        print("\n✗ WS-2 was rejected — only 1 connection allowed")
+    else:
+        print("\n? Inconclusive — check errors above")
+
+
+if __name__ == "__main__":
+    main()