dashboard: skip DOM rebuild on unchanged poll, publish-time staleness, fix boundary countdowns

poll() now compares raw payload text and re-renders only what a changed file can affect, preserving selection, disclosure, and scroll state. Hero freshness reflects publish time (STALE warning past 15 min while market open), and countdowns resolve their wrap interval from data-interval instead of reverse-looking shared TF boundaries in snapshot.timeframes.
news: reduce headline max age from 24h to 12h, make configurable
2026-08-09 18:15:31 -04:00 · 2026-08-09 16:49:48 -04:00 · 2026-08-08 10:48:10 -04:00 · 2026-08-07 23:02:08 -04:00 · 2026-08-07 22:50:10 -04:00 · 2026-08-07 22:35:15 -04:00
57 changed files with 19533 additions and 4563 deletions
@@ -1,6 +1,10 @@
 # Secrets
 .env
 .env.local
 config.json
 # Oanda daemon state
 oanda_state.json
 # Editor / tooling
 .claude/
@@ -24,4 +28,12 @@ Thumbs.db
 # Backtesting data (keep scripts, ignore generated files)
 backtest/cache/
 backtest/*.csv
 backtest/research/*.parquet
 backtest/research/duka_m5_boundary_spreads.csv
 backtest/trailgrid/
 backtest/trailgrid5m/
 backtest/__pycache__/
 .env.*
 logs/
 backtest/research/commit_ladder_2026-08/
 backtest/research/llm_news_eval_2026-08/
@@ -1,113 +1,91 @@
-# ForexBot2
+# CLAUDE.md
-XAU/USD signal intelligence platform. Analyzes gold across multiple timeframes using 9 technical indicators with regime-adaptive weights, news sentiment, and economic calendar data. Delivers signals via Telegram with full trade setups (entry, SL, TP).
+This file provides guidance to Claude Code (claude.ai/code) when working with code in this repository.
 ## What this is
 Signal-intelligence platform for **XAU/USD (gold — the only supported pair)**. It fetches OHLCV from the active data provider (**Oanda v20 since 2026-07-28**; Twelve Data and Tiingo remain registered fallbacks), scores 9 technical indicators per timeframe with regime-adaptive weights, checks multi-timeframe agreement and news risk, grades the signal (A+…F), and generates a full trade setup (entry, SL, TP1/TP2, trailing plan). Delivery: Telegram bot, CLI, and an Oanda practice-account auto-execution daemon. A realism-focused backtester replays history through the *exact same* signal code.
 Every numeric threshold in `config.py` and `pair_signals/xauusd.py` was backtested or is load-bearing for a backtested result. Validation evidence lives in the factory/module docstrings and in `~/.claude/projects/.../memory/`.
 ## Commands
 ```bash
 python test_all.py                      # QA suite: 374 tests, 23 modules, no network (mocked)
 python main.py --pair XAUUSD --timeframe H1 --detail   # CLI signal report (--json for full dump,
                                        #  --panel to run the advisory LLM agent panel on-demand)
 python telegram_bot.py                  # the bot (refuses to boot without credentials)
 python oanda_daemon.py                  # auto-execution daemon on Oanda practice account
 python dashboard_publisher.py --once    # web dashboard: one scan+publish cycle to $DASHBOARD_OUT_DIR
                                        # (omit --once for the candle-close daemon; deploy/README-dashboard.md)
 # Backtests
 python backtest/gpu_backtest.py --pair XAUUSD --days 365
 python backtest/gpu_backtest.py --pair XAUUSD --start 2025-01-01 --end 2025-06-01 \
    --primary-tf 15min --confirm-tfs 1h,4h
 # key flags: --spread --quality A+,A,B,C --strategies --workers --max-hold-bars
 #            --warmup-bars --cooldown-bars --no-cache --export-csv out.csv --detail
 #            --fixed-rr-tp N (research-only: hard TP at N×risk, trail off; verdict
 #            2026-08-05: NOT shipped — backtest/research/fixedtp_2026-08-05.md)
 ```
 **`test_all.py` is NOT pytest** — it uses a custom `@test("name")` decorator that runs each test immediately at import; don't grep for `def test_` or try `pytest test_all.py`. Run the whole file; it prints per-module results and exits non-zero on failure. There is no way to run a single test short of commenting modules out. Tests must stay green; every new gate/knob needs tests. When patching weekend logic in tests, patch `providers.base._is_forex_weekend` — but know there are THREE independent weekend implementations (`providers/base.py`, `signal_aggregator.py`, and an inline copy in `backtest/gpu_backtest.py:_is_xau_blocked_by_calendar`); patching the provider one does not reach the other two.
 Python 3.12; a `venv/` exists in-repo. Dependencies in `requirements.txt` (numpy, pandas, requests, twelvedata, python-telegram-bot ≥20, websockets).
 ## Configuration & credentials
 Precedence for every credential/deployment setting: **env var → config.json → refuse to run** (no hardcoded fallbacks). A gitignored repo-root `.env` is loaded at `app_config` import via `os.environ.setdefault`. `config.json` (gitignored; schema in `config.json.example`) holds secrets plus deployment settings: `data_providers.active` (the live-feed switch), `data_providers.streaming` (optional tick-stream provider for the level-breach monitor), `scanning.pairs`/`timeframes`, per-pair `blocked_hours_utc`, `log_level`. Key env vars: `TELEGRAM_TOKEN`, `TELEGRAM_CHAT_ID`, `TELEGRAM_CHANNEL_ID`, `TWELVE_DATA_API_KEY`, `TIINGO_API_KEY`, `OANDA_ACCOUNT_ID`/`OANDA_API_TOKEN`/`OANDA_BASE_URL`, `DEEPSEEK_API_KEY` (news LLM; the `deepseek` config.json block holds model/enabled/timeout/daily cap). The `agent_panel` config.json block (all-defaults = feature dark) controls the advisory LLM agent panel: `enabled`, per-surface `telegram`/`dashboard` toggles, `min_quality`, `max_panels_per_day`, `reuse_ttl_seconds`.
 ## Architecture
 ### End-to-end pipeline (every scan)
 ```
-Twelve Data REST API + WebSocket
+providers/<active>.fetch_ohlcv()       per TF: single-flight fetch, drop forming candle, validate/clean, cache
-        |
+                                       (only Twelve Data rate-throttles; Oanda has no client throttle)
-  data_provider.py          (fetch, cache, validate OHLCV; rate-limited 55 calls/min)
+  → engine.TechnicalEngine.analyze()   per TF: 9 scored indicators + ATR context → regime → composite score
-        |
+  → signal_aggregator.analyze_pair()   live entry point (weekend/day filters, fetch, staleness)
-    engine.py                (9 indicators per timeframe → composite score)
+      └─ analyze_from_dataframes()     replayable core: MTF agreement → news → bonuses/penalties →
-        |
+                                       final signal + quality grade → gates → trade setup → reasons
-  signal_aggregator.py       (MTF agreement + news + final signal + trade setup)
+  → SignalReport                       consumed by telegram_bot.py, main.py, oanda_daemon.py, backtester
        |
  +-----+--------+
  |              |
 telegram_bot.py  main.py
 (Telegram alerts, (CLI single-pair
 auto-scanner,    analysis)
 WS monitor,
 live dashboard)
 ```
-## Core Files
+Signal currency everywhere is `SignalStrength` (−2..+2). DataFrames are UTC tz-aware, oldest-first, capitalized OHLCV columns. Everything runs on **closed bars only** — the forming candle is dropped, and session/hour filters evaluate **bar-CLOSE time** (bar label + candle duration) identically in live and backtest.
-| File | Purpose |
+### Layers
 |------|---------|
 | `config.py` | All configuration dataclasses, XAU/USD pair definition, strategy config |
 | `data_provider.py` | Twelve Data SDK wrapper with in-memory cache and rate limiting |
 | `engine.py` | `TechnicalEngine` — computes EMA, RSI, BB, ADX, ATR, MACD, ROC, Keltner, divergences, candlestick patterns, swing S/R |
 | `signal_aggregator.py` | `SignalAggregator` — MTF analysis, news integration, diminishing penalty cap, trade setup generation |
 | `news_engine.py` | Economic calendar (Forex Factory JSON) + RSS sentiment (keyword-based) |
 | `telegram_bot.py` | Async Telegram bot with auto-scanner (M5/M15/H1/H4), WebSocket price monitor, live dashboard |
 | `main.py` | CLI entry point for XAU/USD analysis |
 | `test_all.py` | Comprehensive QA suite (~137 tests, no live API needed) |
-## Key Concepts
+- **`providers/`** — one class per provider subclassing `BaseDataProvider` (`providers/base.py`), which owns everything the pipeline depends on: TTL+boundary cache, forming-candle drop, freshness check, validate/clean, paginated `fetch_range` (feeds the backtester from any provider), optional tick streaming. Subclasses implement only `_fetch_raw` + symbol/interval maps. Registry + `make_data_provider(config, name=None)` in `providers/__init__.py`; config.json `data_providers.active` selects the live candle feed and the optional `data_providers.streaming` scalar selects a different provider for the tick stream. A test enforces that subclasses don't override shared base methods (`supports_streaming`/`iter_ticks` are the sanctioned streaming override points). Tiingo streams mid-price ticks (`wss://api.tiingo.com/fx`) — do not re-add spread display from its free FX feed (two-tier indicative quotes, removed 2026-07-23); the no-spread rule extends to the WS stream.
 - **`engine.py`** — `TechnicalEngine`: 9 scored indicators (ema_trend, rsi, macd, bollinger, adx, roc, keltner, divergence, candlestick) + ATR as context. Regime classification (trending_up/down, `trending` = mixed/direction-UNKNOWN, transitioning, ranging) drives per-indicator weight multipliers and regime-aware RSI reinterpretation. Compute order matters (ATR first; MACD always computed even when not a pillar — divergence needs the histogram). Private series ride on the result as `_underscore` attrs; scorers/formatters skip anything `_`-prefixed or weight-0.
 - **`signal_aggregator.py`** — MTF agreement, diminishing-decay penalty system, exhaustion hard block, quality grading, then an ordered chain of post-grade gates (session, XAU SELL quality/vol/hour, regime blocks, M5 noise gates), then three setup generators: XAU ATR (A+/A), generic swing-based, plus the M5 pullback (`XAU_PB`) and chart-pattern (`XAU_CP`, `chart_patterns.py`) entry paths. The PB/CP generators run AFTER the gate chain and force quality to "B"; the CP path deliberately accepts grade-F composites and re-checks only session/day/big-candle (via warning prefixes) plus its own SELL/vol/RSI gates — it does NOT re-check exhaustion or the M5 min-score/MTF gates. **This is validated behavior, not a bug** (2026-08-04 audit: 226/586 trades in the shipped M5 12mo book are XAU_CP, +38.9R, 104 of them exhaustion-flagged and net-positive) — changing it is a rule-1 revalidation event. `TradeSetup.strategy` selects trailing params; trail params are stamped on the setup from the *producing* config so per-TF overrides reach formatters.
 - **`pair_signals/`** — per-pair config factories. **Always use the factory (`produce_signal` / `make_config`), never bare `PlatformConfig()`** — the factory applies the config.json overlay first, then the validated per-TF deltas (M5 hour-9 block, M15 trail arm 0.70, M15 SELL-hour block, etc.; evidence in the docstring). A test enforces factory ≡ inline-config equivalence. Confirmation map: M5→[M15,H1], M15→[M5,H1], H1→[M15,H4], H4→[H1,D1].
 - **`news_engine.py`** — Forex Factory calendar (MyFXBook RSS fallback) + RSS keyword sentiment with gold-inverse keywords. Class-level caches (fresh engine per scan). Optional DeepSeek enrichment via `llm_client.py` (requests-only, no SDK): batched headline sentiment overriding keyword scores, gold-direction on release surprises, and a market-summary blurb (`SignalReport.news_summary`) — **display/warning-only, keyword fallback on any failure, inert without a key**; prompts are static module constants (DeepSeek prefix-cache pricing) and requests send `thinking: disabled` (V4-Flash reasoning otherwise blows the timeout). By XAU-tuned default, event risk and sentiment **warn but do not gate signals** (`event_risk_caps_signals=False`, `sentiment_affects_signals=False`) — the LLM does not change that.
 - **`agent_panel.py`** — ai-hedge-fund-style advisory panel: 4 DeepSeek persona calls (macro/technical/risk/contrarian, concurrent on a panel-private pool) + 1 synthesizer producing an **ADVISORY-ONLY** consensus (`AGREE/NEUTRAL/OPPOSE` + confidence + rationale) written onto `SignalReport.panel_*` fields. Attached **post-hoc** by consumers (`telegram_bot` alert path, `main --panel`, `dashboard_publisher`) — never runs inside `analyze_from_dataframes` (backtest parity, test-locked) and no gate reads its fields. Static prompts (prefix caching); spend bounded by an exact per-bar cache, a reuse TTL keyed (symbol, TF, direction) so persisting signals don't re-spend every bar, and a `max_panels_per_day` UTC budget (default 20 ≈ ≤100 requests) on top of the shared 500/day client cap. `<2` valid persona takes skips the synthesizer; any failure degrades to no panel, never a changed signal.
 - **`telegram_bot.py`** — owner-locked bot + subscriber channel; candle-boundary auto-scan loops (M5/M15/H1/H4, boundary+10 s), alert quality threshold + fingerprint dedup (committed only after a successful send), 📰 LLM market-context line on alerts when available, WebSocket level-breach monitor fed by the streaming provider (`data_providers.streaming`, falling back to the active provider; auto-disabled when the resolved provider can't stream or has no API key — Oanda does not stream, Tiingo does). Rotating log: `logs/telegram.log`, 3-h rotation × 240 backups.
 - **`oanda_daemon.py` / `oanda_executor.py`** — practice-account auto-execution. Book-parity rules (see daemon docstring): no broker-side TP (trailing model — TP1/TP2 advisory), 5-bar per-TF cooldown (wall-clock approximation, so it expires over weekends where the backtester would still count bars), trail ratchets at bar close once per *observed* bar (a missed/failed cycle skips those bars entirely — the intra-gap high/low never enters the peak). Accepted divergence: one position total; orders send `positionFill=OPEN_ONLY` so a state/broker divergence cancels the order instead of silently netting. State/broker re-sync runs at boot and after any suspicious fill (`_reconcile_positions`).
 - **`dashboard_publisher.py` / `static/dashboard.html`** — web dashboard as a **static publisher** (live LAMP server): standalone daemon scans at candle closes via `produce_signal` and atomically writes `snapshot.json`/`news.json`/`dashboard.html` to `$DASHBOARD_OUT_DIR` (Apache serves them; page is public by owner decision). **Hard independence rule: never imports, and is never imported by, `telegram_bot`/`oanda_daemon`** — confirmation map and boundary math are deliberate local copies, so a web-side failure can only mean a stale page. Due-TF classification uses the RAW wake time (flooring wake−buffer skips every H1/H4 scan — 2026-08-02 bug). News panel reads `NewsEngine.get_fundamental_context()` plus the additive `get_week_events()` accessor (display-only; no signal path calls it); as a separate process it has its own news caches and DeepSeek daily cap. News/calendar rows are clickable (detail dialog); headlines carry `id` = sha1(headline) in news.json, and exactly one on-demand endpoint exists: `static/news_summary.php` (installed by `install_static`), which serves a DeepSeek gold-impact summary per headline — whitelist-gated against news.json ids, cache-forever in `ai/`, own fail-closed daily cap (150/day, file-based, separate from the publisher's), key at `/etc/gold-dashboard/deepseek.key`, never reads repo config. Deployment checklist: `deploy/README-dashboard.md`.
 - **`backtest/gpu_backtest.py`** — bar-by-bar replay through the real `analyze_from_dataframes` (news off). Realism model (do not regress): next-bar-open entries re-anchored to the raw price path, round-trip `--spread` charged per trade, trade resolution on RAW bars (cleaning applies only to signal-side data), same-bar SL+TP tie → SL first, gap fills at bar open, trail ratchet at bar close only. Indicators are precomputed once per TF (CuPy GPU or NumPy fallback — check the `Backend:` log line); replay fans out over worker processes and the 5-bar cooldown is applied in an ordered pass afterwards, so parallel and sequential books are identical. Parquet cache in `backtest/cache/` keyed by provider/symbol/TF/window. `backtest/*.csv` are gitignored pinned trade books from past campaigns — used for manual before/after comparison. `backtest/instructions_for_backtest.txt` lists what every backtest report must record.
-**Signal Pipeline:** OHLCV → indicators → regime classification → regime-weighted composite score → MTF agreement check → news/event adjustment → diminishing penalty cap → final signal (STRONG_BUY/BUY/NEUTRAL/SELL/STRONG_SELL) → quality grade (A+/A/B/C/D/F) → trade setup if grade C+.
+## Rules for making changes (read before editing signal logic)
-**Regime-Adaptive Weights:** Market classified as `trending_up`, `trending_down`, `ranging`, or `transitioning` via ADX + EMA + ADX slope. Gold-specific weights per regime:
+1. **Never change a tuned value without re-validation.** Acceptance rule: positive total R in BOTH train and validation halves of a ≥12-month window, PF ≥ 1.3, at realistic cost (XAU $0.30 round trip), trailing exits, next-open fills. Marginal edges must additionally pass the 2024-06→2025-03 out-of-sample window — within-year splits alone have approved configs that later failed OOS.
- **Trending:** MACD (2.0), EMA (2.0), Divergence (2.0), Keltner (1.5), ROC (1.5)
+2. **Per-timeframe overrides live in the pair factory**, never in global defaults — the same knob has opposite effects on different TFs (proven repeatedly: `tp_floor_rr`, hour-9 block, trail params). Document validation evidence in the factory docstring.
- **Ranging:** Divergence (2.5), Keltner (2.0), RSI (1.5), BB (1.5), MACD (1.5)
+3. **Pair isolation is a hard invariant.** XAU gates in the aggregator are display-name-gated so a future pair port cannot inherit gold-calibrated logic by accident.
- **Transitioning:** Keltner (2.5), MACD (2.5), ROC (2.0), Divergence (2.0), EMA (1.5)
+4. **Live == backtest parity.** If you add a live gate, add it to the replay path in the same commit; all time filters evaluate bar-close time in both paths.
 5. **ROC stays weighted** even though it's correlated with MACD/Keltner/EMA — zeroing it backtested 13R worse; composite thresholds are calibrated with it.
 6. **Backtester realism knobs are results-critical** (raw-path resolution, spread charging, next-open entry, SL-first ties, close-only trail). Touching them invalidates comparability with all pinned books.
 7. **The forming candle must stay dropped** — live signals repaint otherwise.
 8. Don't add config fields nothing reads; don't disable indicators in production paths (`disabled_indicators` is for experiments only).
 9. EUR/USD was removed 2026-07-16 after a no-edge verdict (cost-to-volatility; last commit with EUR support: `7d3fd95`). Do not re-add without new data. BTC/ETH work lives on unmerged branches only.
-**Transitioning Regime:** ADX < 25 but ADX slope > 3.0 = trend emerging. Catches trend birth 5-10 bars before ADX crosses 25. Boosts quality grade for early entries.
+## 2026-08-04 QA audit — owner decisions ledger
-**Diminishing Penalty Cap:** Penalties sorted by magnitude, applied with exponential decay (1st at 100%, 2nd at 50%, 3rd at 25%). Hard cap at 30% total. Replaces old additive stacking that could gut 30-50% confidence.
+- **Weekend calendar: FIXED (owner-approved 2026-08-04).** All three `_is_forex_weekend` copies are NY-anchored (Fri 17:00 → Sun 18:00 ET; during US DST that is Fri 21:00 UTC — measured on 5yr of Oanda bars). Kills the weekly false-"STALE DATA/feed lagging" hours. Book impact, fully accounted: M15 byte-identical; M5 12mo dropped exactly ONE trade — a phantom whose signal fired on the final summer-Friday bar (close 21:00 UTC) and "filled" at Sunday's reopen 49 h later (−1.08R, live-impossible). M5 12mo reference at HEAD: 555 trades, +98.64R PF≈1.50 @ $0.65 (was 556/+97.56/1.48). A Module 21 test sweeps all three copies hourly across both DST transitions.
 - **Spread in exit-trigger detection (rule 6): owner DECLINED 2026-08-04.** Backtest stop triggers stay on raw mid High/Low; the measured optimism (~4–6% of total R: −3.55R @ $0.30, −4.24R @ $0.65 on the M15 12mo book) is accepted and books stay comparable. Re-open only as a conscious re-pin campaign.
 - **`FB2_EMA_LO`: HARDENED 2026-08-04 (delegated).** Research override for the EMA200-proximity band, default = tuned 0.5; an unparseable value now degrades to the default with a warning instead of crashing every scan. Documented here — it is a knob that exists.
 - **H4 seeding divergence: parked.** Live H4 fetches only 380 bars vs the backtest's unbounded history — ~2% of sampled H4 bars flip an indicator signal (composite Δ up to 12.3). M5/M15/H1 clean at their real fetch depths. Fold into any future H4 campaign; H4 remains "unproven for auto-execution".
-**Indicators (9 scored):** EMA Trend, RSI, Bollinger Bands, MACD, ADX, Keltner Channels (TTM Squeeze), ROC (Rate of Change), Divergence, Candlestick Patterns.
+## Current state notes (2026-07-30)
-**Trade Setup:** SL = nearest swing low/high ± 1×ATR buffer (min 1.5×ATR). TP = nearest swing level if RR≥1.2, else calculated from ATR. Minimum RR 0.5:1.
+- Oanda v20 is the live feed (`data_providers.active: "oanda"`), but the pinned backtest books are still Twelve-Data-pinned and the backtest cost assumption is $0.30 while measured real spreads are ~$0.65–0.72 — re-pin books on Oanda bars before treating feed-switch comparisons as valid (see Oanda/TD divergence study in memory).
-
+- Human-facing docs in `docs/` (`user_guide.html`, `beginner_guide.html` + PDF builds) must be kept in sync when signal behavior changes.
 ## How to Run
 ```bash
 # CLI (XAU/USD analysis)
 python main.py                    # H1 default
 python main.py --timeframe M15 --detail --json
 # Telegram bot (auto-scanner)
 TELEGRAM_TOKEN=<token> TELEGRAM_CHAT_ID=<id> python telegram_bot.py
 # Tests
 python test_all.py
 # Backtest
 python backtest/gpu_backtest.py --pair XAUUSD --days 365
 ```
 ## Environment Variables
 | Variable | Purpose | Default |
 |----------|---------|---------|
 | `TWELVE_DATA_API_KEY` | Market data API key | Hardcoded fallback in config |
 | `TELEGRAM_TOKEN` | Telegram bot token | Hardcoded fallback |
 | `TELEGRAM_CHAT_ID` | Auto-scan chat ID (enables boot auto-scan) | Hardcoded fallback |
 ## Dependencies
 `numpy`, `pandas`, `requests`, `twelvedata`, `python-telegram-bot`, `websockets` — see `requirements.txt`.
 ## Data Provider
 - **Source:** Twelve Data REST API via `twelvedata` SDK
 - **Rate limit:** 55 credits/min (Grow plan), enforced by sliding-window throttle
 - **Cache:** In-memory dict, TTL = candle duration of the timeframe
 - **Validation:** Drops NaN rows, forward-fills gaps (limit 3), fixes OHLC inconsistencies, removes outliers (>10σ), strips future timestamps, deduplicates
 - **WebSocket:** XAU/USD only
 ## Telegram Bot
 - **Auto-scan:** M5 (data-only, no alerts), M15, H1, H4 at candle boundaries + 10s buffer
 - **Manual scan:** Buttons for M15/H1/H4 individual scans + "Scan All TFs" multi-timeframe summary
 - **WebSocket:** Streams live ticks for XAU/USD; triggers immediate M15 scan on key level breach (swing S/R, BB, KC, EMA200) with 5-min cooldown
 - **Live dashboard:** Edit-in-place message, 30s refresh, shows XAU/USD across timeframes
 - **Commands:** `/start`, `/scan`, `/autoscan`, `/status`, `/cache`, `/clearcache`, `/help`
 ## Backtesting
 The `backtest/` directory contains `gpu_backtest.py` — a GPU-accelerated backtester (CuPy with NumPy fallback) that replays historical OHLCV through the real SignalAggregator. Pre-computes all indicator series including ROC, Keltner, ADX slope for 100x+ speedup. Uses parquet caching for data persistence.
 ## Code Conventions
 - Python 3.12, type hints on public methods
 - Dataclasses for all config and output structures
 - `SignalStrength` enum (-2 to +2) is the universal signal currency
 - Indicators stored as `Dict[str, IndicatorSignal]` in `TechnicalAnalysis`
 - Key levels stored as `Dict[str, float]` — prefixed `swing_high_`, `swing_low_`, `ema_`, `bb_`, `kc_`, `atr`
 - Private indicator data attached as `result._rsi_series`, `result._macd_histogram`, `result._adx_slope`, `result._kc_squeeze` (underscore prefix, skipped by composite scorer)
 - Blocking signal computation runs in `ThreadPoolExecutor(max_workers=4)` to avoid blocking the async Telegram event loop
 - Shared `ForexDataProvider` instance across all scans for cache reuse
@@ -1,198 +0,0 @@
 # ForexBot2 — USD/JPY Signal Intelligence
 A focused USD/JPY signal generator + walk-forward backtester. Live signals
 and backtests run through the **same** technical engine, so what you measure
 in a backtest is exactly what you'd execute live.
 ## Setup
 ```bash
 pip install -r requirements.txt
 ```
 API key (optional — a default key is in `config.py`):
 ```bash
 export TWELVE_DATA_API_KEY=your_key_here          # Linux/Mac
 $env:TWELVE_DATA_API_KEY="your_key_here"          # PowerShell
 ```
 ---
 ## Live Signal
 ```bash
 python main.py                          # H1, summary view
 python main.py --detail                 # Full indicator breakdown
 python main.py --timeframe H4 --detail  # H4 instead of H1
 python main.py --json                   # JSON output (for piping)
 ```
 | Option        | Default | Description                                |
 |---------------|---------|--------------------------------------------|
 | `--timeframe` | `H1`    | `M5 M15 M30 H1 H4 D1`                      |
 | `--detail`    | off     | Indicator breakdown, key levels, reasons   |
 | `--json`      | off     | Machine-readable output                    |
 | `--verbose`   | off     | Debug logging                              |
 ### Output
 ```
 USD/JPY  │  Price: 149.230  │  Regime: trending_up  │  Vol: normal
 SIGNAL: BUY  │  Score: +28.4  │  Grade: B  │  Confidence: 62%
 Action: BUY — Decent setup, wait for confirmation
 Multi-Timeframe:  ✓ ALIGNED
   M15: BUY          (+18.2)
    H1: BUY          (+28.4)
    H4: STRONG BUY   (+45.1)
 Trade Setup:
  Direction:     BUY
  Entry:         149.230
  Stop Loss:     148.960  (27.0 pips)
  Take Profit 1: 149.770  (54.0 pips)
  Take Profit 2: 150.040
  Risk:Reward:   1:2.0
 ```
 ---
 ## Backtest (Walk-Forward)
 The backtester replays USD/JPY history bar-by-bar through the live signal
 pipeline. No lookahead, no parallel implementation — what passes in
 backtest passes live.
 ```bash
 python main.py --backtest                          # Full available history
 python main.py --backtest --bars 2000              # Last 2000 H1 bars
 python main.py --backtest --start 2024-01-01 --end 2024-12-31
 python main.py --backtest --timeframe H4           # Backtest on H4
 python main.py --backtest --min-grade A            # Only A/A+ entries
 python main.py --backtest --journal                # Last 10 trades printed
 python main.py --backtest --journal-csv trades.csv # Full journal to CSV
 python main.py --backtest --regime-filter          # Skip choppy/ranging bars
 python main.py --backtest --regime-filter --regime-min-adx 35
 python main.py --backtest --session-filter         # Trade only 07-16 UTC
 python main.py --backtest --session-filter --session-start 8 --session-end 17
 python main.py --backtest --auto-session-filter    # Reactive session toggle
 python main.py --backtest --auto-session-filter --auto-session-lookback 15 \
                          --auto-session-threshold-pct -0.5
 ```
 ### Regime filter
 The engine is a confluence-based trend-follower — it makes money in trending
 regimes and loses in chop. `--regime-filter` gates entries on H4 ADX:
 signals only fire when ADX ≥ `--regime-min-adx` (default 30). Sweet spot is
 ADX 30–35; tighter cuts trades faster than it improves quality.
 ### Session filter
 `--session-filter` restricts entries to a UTC hour window (default 07-16,
 covering London + NY). Diagnostic on 2025-11 → 2026-05 showed 47% WR /
 +127 pips inside that window vs 0% WR / -571 pips outside it. Override
 bounds with `--session-start` (inclusive) and `--session-end` (exclusive).
 Tradeoff: it's regime-specific medicine. In trending regimes (e.g. 2024)
 it cuts winning off-hours trades and gives back ~33% of the edge. Ships
 **default off** because the pure regime filter still wins cumulative
 across the 2024 / 2023 / 2026 windows.
 ### Auto session toggle
 `--auto-session-filter` is a reactive variant: it tracks the rolling mean
 `pnl_pct` over the last N closed trades (`--auto-session-lookback`,
 default 10) and activates the 07-16 UTC window only when that mean drops
 below `--auto-session-threshold-pct` (default `-0.3`). Releases
 automatically once expectancy recovers.
 Less aggressive than the manual flag — preserves more trend edge but
 reacts slowly because the deque must fill with losses before kicking in.
 Use this when you don't want to manually toggle the regime medicine.
 ### Execution model
 | Aspect             | Behavior                                                          |
 |--------------------|-------------------------------------------------------------------|
 | Entry              | Next bar's open (no current-bar lookahead)                        |
 | Stop Loss          | 1.5 × ATR from entry (configurable)                               |
 | Take Profit        | RR × stop distance (default 1:2)                                  |
 | Same-bar SL & TP   | Conservative: SL fills first                                      |
 | Spread cost        | Half-spread on entry + half-spread on exit (full round-trip)      |
 | Slippage           | 0.5 pips per fill (entry + exit)                                  |
 | Position size      | 1% of current equity risked per trade                             |
 | Concurrency        | One open position at a time                                       |
 | News in backtest   | Disabled — RSS/calendar can't be replayed without lookahead       |
 ### Output
 ```
 Backtest Summary — USD/JPY
 ─────────────────────────────────────────────────
 Trades:           87  (52W / 35L)
 Win Rate:         59.8%
 Profit Factor:    1.84
 Expectancy:       +12.4 pips/trade  (+12.42 per trade)
 Initial Balance:  $10,000.00
 Final Balance:    $11,084.00
 Total Return:     +10.84%
 CAGR:             +18.92%
 Max Drawdown:     -4.21%
 Sharpe Ratio:     1.42
 Sortino Ratio:    2.18
 Avg Duration:     14.3 bars
 Win Streak:       7 | Loss Streak: 4
 ─────────────────────────────────────────────────
 ```
 ---
 ## Parameter Optimization (parallel)
 ```bash
 python main.py --backtest --optimize
 ```
 Runs a 3×3×3 grid (27 combos) over `atr_sl_multiplier`, `default_rr_ratio`,
 and `min_score` (signal-score threshold). Combos run in parallel across all
 CPU cores via `multiprocessing.Pool`. Twelve Data is hit once — workers
 share an on-disk parquet cache (`.cache/usdjpy_<tf>.parquet`).
 Output ranks combos by total return:
 ```
 Top 10 Parameter Combinations (by total return):
 #   Return%     PF   WinR  Trades  MaxDD%  Sharpe  Params
 ──────────────────────────────────────────────────────────────────────
 1   +14.21%   2.10  62.5%     64   -3.84   1.78    atr_sl_multiplier=1.5, default_rr_ratio=2.5, min_score=25.0
 2   +11.93%   1.92  58.7%     71   -4.11   1.55    atr_sl_multiplier=1.0, default_rr_ratio=2.0, min_score=15.0
 ...
 ```
 Customize the grid in `config.py → BacktestConfig.param_grid`.
 ---
 ## Files
 | File                  | Role                                              |
 |-----------------------|---------------------------------------------------|
 | `main.py`             | CLI entry point (live + backtest + optimize)      |
 | `config.py`           | All parameters in one place                       |
 | `data_provider.py`    | Twelve Data fetcher + parquet cache               |
 | `engine.py`           | Technical indicators (8 + divergence + patterns)  |
 | `signal_aggregator.py`| Multi-TF aggregation, grading, trade setup        |
 | `news_engine.py`      | RSS sentiment + economic calendar (live only)     |
 | `backtest.py`         | Walk-forward replay + parallel optimizer          |
 | `metrics.py`          | Win rate, PF, Sharpe, max DD, trade journal       |
 ---
 ## Disclaimer
 For educational and research use. Past performance does not guarantee
 future results. Always use proper risk management.
@@ -0,0 +1,582 @@
 """
 ForexBot2 — LLM agent panel (advisory consensus)
 =================================================
 ai-hedge-fund-style persona panel for XAU/USD signals: four DeepSeek
 personas (macro, technical, risk, contrarian) each review a finished
 SignalReport from their own angle, then a synthesizer chair weighs the
 takes and issues a consensus verdict (AGREE / NEUTRAL / OPPOSE with
 confidence and a short rationale).
 Two invariants, both load-bearing (CLAUDE.md rules 1 and 4):
 - ADVISORY / DISPLAY-ONLY. The verdict is written onto SignalReport
  fields that no gate and no scoring path ever reads. Any failure —
  LLM down, cap exhausted, garbage reply — degrades to "no panel",
  never to a changed or blocked signal.
 - NEVER runs inside ``analyze_from_dataframes``. The backtester replays
  that function; consumers (telegram_bot, main --panel,
  dashboard_publisher) attach the panel POST-HOC via ``attach_panel``,
  so replayed books can never see a panel field. Backtest parity holds
  by construction.
 Cost model: every full panel spends 5 requests (4 personas + 1
 synthesizer) against the shared DeepSeek daily cap (llm_client). Two
 layers bound the spend on top of that cap:
 - ``max_panels_per_day`` (config, default 20) — a UTC-day budget of
  panel *attempts*, mirroring the client's request cap.
 - A reuse TTL (default 3600 s) keyed on (symbol, timeframe, direction):
  a persisting signal re-serves its cached verdict across bars instead
  of re-spending. Without this, one sticky M5 dashboard signal would
  burn 5 calls per 5-minute bar — 1,440/day against the 500 cap.
 Prompts are STATIC module constants (DeepSeek prefix-cache pricing —
 see news_engine.py); all variable content travels in the user payload.
 """
 import datetime as _dt
 import logging
 import threading
 import time
 from collections import OrderedDict
 from concurrent.futures import ThreadPoolExecutor, wait as _fwait
 from dataclasses import dataclass, field
 from typing import TYPE_CHECKING, Any, Dict, List, Optional, Tuple
 from app_config import get_app_config
 from llm_client import DEFAULT_TIMEOUT, get_llm_client
 if TYPE_CHECKING:  # type hints only — keep the coupling one-way
    from news_engine import FundamentalContext
    from signal_aggregator import SignalReport
 logger = logging.getLogger(__name__)
 DEFAULT_MIN_QUALITY = "B"
 DEFAULT_MAX_PANELS_PER_DAY = 20
 DEFAULT_REUSE_TTL_SECONDS = 3600.0
 # Quality ranks for the min_quality gate. Deliberately a local copy —
 # telegram_bot's ranking is private and dashboard_publisher must not
 # import telegram_bot (hard independence rule).
 _QUALITY_RANK = {"A+": 4, "A": 3, "B": 2, "C": 1}
 # ── DeepSeek prompts ─────────────────────────────────────────────────
 # STATIC constants on purpose: DeepSeek prices a repeated byte-identical
 # prompt prefix at the cache-hit rate (~50x cheaper), so nothing here may
 # be interpolated or timestamped. All variable content travels in the
 # user payload. Panel output is display-only — it never gates signals.
 _GOLD_DOCTRINE = """\
 Gold doctrine: hawkish Fed / rate hikes / rising real yields / strong
 dollar are gold-bearish; easing / rate cuts / safe-haven flows /
 geopolitical stress / weak dollar are gold-bullish.
 """
 _TAKE_CONTRACT = """\
 Your review is advisory research on an already-generated signal; you are
 one voice on a panel, not the decision maker.
 Hard rules: use ONLY the data in the JSON you receive — do not invent
 numbers, events, or price levels. If the evidence your role needs is
 thin or missing, answer "neutral" with confidence below 30, unless your
 role's instructions say otherwise.
 Confidence scale (0-100): 80-100 strong one-sided evidence; 50-79 solid
 case with some doubt; 30-49 mixed evidence; below 30 little basis to
 judge.
 Write the note in plain everyday English for someone who has never
 traded: short simple sentences, no trading jargon, no acronyms, no
 abbreviations. Say "profit versus loss", not "RR"; "the 4-hour chart",
 not "H4"; describe what an indicator MEANS ("the price has climbed
 unusually fast"), never its name or number.
 Reply with ONLY JSON:
 {"stance": "support"|"oppose"|"neutral", "confidence": <0-100>,
 "note": "<max 25 words>"}
 "stance" says whether YOUR analysis supports the proposed trade
 direction, opposes it, or is genuinely mixed; "confidence" is how sure
 you are of your stance; "note" is your single most important point.
 """
 _MACRO_SYSTEM = """\
 You are the macro analyst on a gold (XAU/USD) trade-review panel. You
 receive JSON: a proposed trade brief ("brief": direction, timeframe,
 quality, price, regime) plus the current fundamental picture — recent
 "headlines", today's high-impact "events" (released: true means the
 number is out), a news "sentiment" score for gold, and an optional
 "news_summary" blurb.
 Judge ONE thing: does the macro backdrop support the proposed direction?
 Weigh released surprises over stale headlines; an imminent unreleased
 high-impact event is a reason for lower confidence, not a direction.
 Stance rules:
 - support: the released data and news flow clearly push gold in the
  proposed direction.
 - oppose: they clearly push gold the other way.
 - neutral: the picture is mixed or stale — or there are no meaningful
  headlines or events to judge (score that low confidence).
 """ + _GOLD_DOCTRINE + """\
 Example: brief direction BUY, headlines dominated by hawkish Fed
 repricing and a hot CPI print -> {"stance": "oppose", "confidence": 70,
 "note": "Inflation is running hot, so interest rates should stay high — that usually pushes gold down."}
 Example: brief direction BUY, headlines led by an escalating conflict
 and a soft jobs report -> {"stance": "support", "confidence": 75,
 "note": "Worries about war and a cooling economy usually send money into gold."}
 """ + _TAKE_CONTRACT
 _TECH_SYSTEM = """\
 You are the technical analyst on a gold (XAU/USD) trade-review panel.
 You receive JSON: a proposed trade brief ("brief") plus the technical
 evidence — per-"indicators" signal/confidence, multi-timeframe
 "mtf" signals and scores, "key_levels" versus the current price, and an
 "exhaustion" count of consecutive same-direction bars flagged by the
 engine.
 Judge ONE thing: is the technical picture internally coherent for the
 proposed direction? Look for indicators fighting each other, the signal
 fighting the regime (e.g. a BUY in trending_down), higher timeframes
 disagreeing, price pressed into a key level, or an exhausted move being
 chased.
 Stance rules:
 - support: the indicators, higher timeframes, and regime mostly agree
  with the proposed direction and none of the conflicts above appear.
 - oppose: one or more of those conflicts is present and serious.
 - neutral: the evidence genuinely cuts both ways, or too much of it is
  missing to judge (score that low confidence).
 Example: brief direction SELL in regime trending_up with H1 and H4 both
 BUY -> {"stance": "oppose", "confidence": 80,
 "note": "This bets on gold falling while the bigger picture still points up."}
 Example: brief direction BUY in regime trending_up with M15 and H1 both
 BUY and exhaustion 0 -> {"stance": "support", "confidence": 70,
 "note": "Every way of looking at the chart points up, so buying goes with the flow."}
 """ + _TAKE_CONTRACT
 _RISK_SYSTEM = """\
 You are the risk manager on a gold (XAU/USD) trade-review panel. You
 receive JSON: a proposed trade brief ("brief") plus the risk facts —
 the trade "setup" (entry, stop_loss, take_profit_1/2, risk_reward,
 risk_dollars), the "volatility" regime, engine "warnings", the calendar
 "event_risk" level, and the assumed "spread".
 Judge ONE thing: is this setup's risk acceptable as constructed? Check
 reward-to-risk adequacy, stop distance versus the volatility regime
 (a tight stop in extreme vol dies to noise; a huge stop in low vol is
 oversized), spread as a fraction of the stop, and event landmines about
 to hit.
 Stance rules:
 - support: the likely reward clearly justifies the risk, the stop fits
  the current volatility, and no event landmine is imminent.
 - oppose: reward too thin for the risk, a stop mismatched to the
  volatility, spread eating a big share of the stop, an imminent
  high-impact event — or no setup at all (a missing setup means the
  signal graded too low for one; oppose unless the rest is
  exceptional).
 - neutral: acceptable but unremarkable — nothing dangerous, nothing
  attractive.
 Example: RR 1.4 with volatility extreme and a high-impact event due ->
 {"stance": "oppose", "confidence": 65,
 "note": "The possible profit is small for such wild price swings, and big economic news is due soon."}
 Example: RR 2.4 with normal volatility and no events due ->
 {"stance": "support", "confidence": 70,
 "note": "The possible profit is more than double the possible loss, and conditions look calm."}
 """ + _TAKE_CONTRACT
 _CONTRARIAN_SYSTEM = """\
 You are the contrarian on a gold (XAU/USD) trade-review panel — the
 designated skeptic. You receive JSON: a proposed trade brief ("brief"),
 the engine's own "bullish_reasons" and "bearish_reasons", its
 "warnings", and an optional "news_summary".
 Your job: construct the strongest honest case AGAINST the proposed
 trade, then score how damaging that case is. Use "oppose" only when the
 counter-case is genuinely strong, "neutral" when it is real but modest,
 and "support" when even the skeptic finds nothing that bites.
 """ + _GOLD_DOCTRINE + """\
 Example: a BUY whose own engine lists three bearish reasons and a
 counter-trend warning -> {"stance": "oppose", "confidence": 60,
 "note": "The system's own list of negatives looks stronger than its reason to buy."}
 """ + _TAKE_CONTRACT
 _SYNTH_SYSTEM = """\
 You are the chair of a gold (XAU/USD) trade-review panel. You receive
 JSON: the proposed trade "direction" and the panel's "takes" — one per
 analyst ("persona", "stance", "confidence", "note").
 Weigh the takes into ONE consensus on whether to act on the signal:
 "AGREE" (the panel backs the trade), "OPPOSE" (the panel is against it),
 or "NEUTRAL" (genuinely split or low conviction). Weigh a high-
 confidence oppose from any single analyst seriously — one strong veto
 can outweigh two lukewarm supports. Do not average blindly.
 Base the verdict and rationale ONLY on the takes given — do not add
 outside analysis.
 Confidence scale (0-100): 80-100 the panel clearly leans one way; 50-79
 a lean with real dissent; below 50 split or low conviction.
 Write the rationale in plain everyday English for someone who has never
 traded: short simple sentences, no trading jargon, no acronyms, no
 abbreviations.
 Example rationale: "Three of four analysts are against this: the
 possible profit is too small and the price has already climbed
 unusually fast. Only the news backdrop supports buying."
 Reply with ONLY JSON:
 {"verdict": "AGREE"|"NEUTRAL"|"OPPOSE", "confidence": <0-100>,
 "rationale": "<max 40 words>"}
 """
 # ── Result types ─────────────────────────────────────────────────────
@dataclass
 class PersonaTake:
    persona: str        # "macro" | "technical" | "risk" | "contrarian"
    stance: str         # "support" | "oppose" | "neutral"
    confidence: float   # 0-100
    note: str
@dataclass
 class PanelVerdict:
    verdict: str        # "AGREE" | "NEUTRAL" | "OPPOSE"
    confidence: float   # 0-100
    rationale: str
    takes: List[PersonaTake] = field(default_factory=list)
    created_ts: float = 0.0
 # ── Payload builders (pure functions; llm_client json-dumps sorted) ──
 def _direction(report: "SignalReport") -> Optional[str]:
    """BUY/SELL the panel should review; None = nothing to review."""
    if report.trade_setup is not None:
        return report.trade_setup.direction
    if report.final_signal.value > 0:
        return "BUY"
    if report.final_signal.value < 0:
        return "SELL"
    return None
 def _brief(report: "SignalReport") -> Dict[str, Any]:
    analysis = report.primary_analysis
    return {
        "pair": report.display_name,
        "timeframe": analysis.timeframe if analysis else "",
        "direction": _direction(report),
        "quality": report.signal_quality,
        "score": round(report.final_score, 1),
        "confidence": round(report.final_confidence),
        "price": round(report.current_price, 2),
        "market_regime": report.market_regime,
        "volatility_regime": report.volatility_regime,
        "mtf_agreement": report.mtf_agreement,
    }
 def _macro_payload(report: "SignalReport",
                   context: Optional["FundamentalContext"]) -> Dict[str, Any]:
    headlines: List[str] = []
    events: List[Dict[str, Any]] = []
    if context is not None:
        headlines = [h.headline for h in context.news_headlines[:10]]
        events = [
            {"event": e.event, "currency": e.currency,
             "actual": e.actual, "forecast": e.forecast,
             "released": not e.is_upcoming}
            for e in context.events_today if e.is_high_impact
        ][:8]
    return {
        "brief": _brief(report),
        "headlines": headlines,
        "events": events,
        "sentiment": round(report.news_sentiment, 3),
        "news_summary": report.news_summary,
    }
 def _technical_payload(report: "SignalReport") -> Dict[str, Any]:
    indicators: Dict[str, Dict[str, Any]] = {}
    key_levels: Dict[str, float] = {}
    analysis = report.primary_analysis
    if analysis is not None:
        # Same visibility filter as main.report_to_dict: private series
        # ride as _-prefixed pseudo-indicators, weight-0 = context-only.
        indicators = {
            name: {"signal": ind.signal.name,
                   "confidence": round(ind.confidence)}
            for name, ind in analysis.indicators.items()
            if not name.startswith("_") and ind.weight != 0
        }
        key_levels = {k: round(v, 2) for k, v in analysis.key_levels.items()}
    return {
        "brief": _brief(report),
        "indicators": indicators,
        "mtf": {tf: {"signal": sig.name,
                     "score": round(report.mtf_scores.get(tf, 0.0), 1)}
                for tf, sig in report.mtf_signals.items()},
        "key_levels": key_levels,
        "exhaustion": report.exhaustion_count,
    }
 def _risk_payload(report: "SignalReport") -> Dict[str, Any]:
    payload: Dict[str, Any] = {
        "brief": _brief(report),
        "volatility": report.volatility_regime,
        "warnings": list(report.warnings),
        "event_risk": report.event_risk,
        "event_description": report.event_description,
        "spread": round(report.spread_estimate, 2),
    }
    ts = report.trade_setup
    if ts is not None:
        payload["setup"] = {
            "direction": ts.direction,
            "entry": round(ts.entry_price, 2),
            "stop_loss": round(ts.stop_loss, 2),
            "take_profit_1": round(ts.take_profit_1, 2),
            "take_profit_2": round(ts.take_profit_2, 2),
            "risk_reward": round(ts.risk_reward, 2),
            "risk_dollars": round(ts.risk_pips * ts.pip_value, 2),
        }
    return payload
 def _contrarian_payload(report: "SignalReport") -> Dict[str, Any]:
    return {
        "brief": _brief(report),
        "bullish_reasons": report.bullish_reasons[:3],
        "bearish_reasons": report.bearish_reasons[:3],
        "warnings": list(report.warnings),
        "news_summary": report.news_summary,
    }
 # ── Orchestration ────────────────────────────────────────────────────
 # Panel-private pool: the 4 persona calls run concurrently WITHOUT
 # occupying telegram_bot's shared scan executor (which has 4 workers —
 # borrowing them would stall every other scan for the panel's duration).
 _pool = ThreadPoolExecutor(max_workers=4, thread_name_prefix="agent-panel")
 _VALID_STANCES = ("support", "oppose", "neutral")
 _VALID_VERDICTS = ("AGREE", "NEUTRAL", "OPPOSE")
 def _clamp_confidence(raw: Any) -> Optional[float]:
    try:
        return max(0.0, min(100.0, float(raw)))
    except (TypeError, ValueError):
        return None
 def _validate_take(persona: str, reply: Any) -> Optional[PersonaTake]:
    if not isinstance(reply, dict):
        return None
    stance = reply.get("stance")
    if stance not in _VALID_STANCES:
        return None
    confidence = _clamp_confidence(reply.get("confidence"))
    if confidence is None:
        return None
    note = str(reply.get("note", "")).strip()[:160]
    return PersonaTake(persona=persona, stance=stance,
                       confidence=confidence, note=note)
 def _ask_persona(persona: str, system: str,
                 payload: Dict[str, Any]) -> Optional[PersonaTake]:
    # Client resolved at call time: tests install stubs via lc._client.
    reply = get_llm_client().chat_json(system, payload)
    return _validate_take(persona, reply)
 def run_panel(report: "SignalReport",
              context: Optional["FundamentalContext"] = None,
              ) -> Optional[PanelVerdict]:
    """Run 4 personas + synthesizer; None = no verdict (fail-safe).
    Requires ≥2 valid persona takes before spending the synthesizer
    request; the synthesizer's reply is validated strictly and there is
    deliberately NO programmatic fallback verdict — a panel that could
    not really convene reports nothing rather than something invented.
    """
    direction = _direction(report)
    if direction is None:
        return None
    jobs = (
        ("macro", _MACRO_SYSTEM, _macro_payload(report, context)),
        ("technical", _TECH_SYSTEM, _technical_payload(report)),
        ("risk", _RISK_SYSTEM, _risk_payload(report)),
        ("contrarian", _CONTRARIAN_SYSTEM, _contrarian_payload(report)),
    )
    futures = [_pool.submit(_ask_persona, name, system, payload)
               for name, system, payload in jobs]
    # The HTTP timeout inside the client bounds each call; the margin
    # covers pool queueing. Stragglers past the wait are abandoned (their
    # threads finish harmlessly) and simply don't vote.
    client_timeout = getattr(get_llm_client(), "_timeout", DEFAULT_TIMEOUT)
    done, _ = _fwait(futures, timeout=client_timeout + 8.0)
    takes = [f.result() for f in done if not f.exception()]
    takes = [t for t in takes if t is not None]
    if len(takes) < 2:
        logger.debug("agent panel: %d/4 persona takes — skipping synthesizer",
                     len(takes))
        return None
    order = {"macro": 0, "technical": 1, "risk": 2, "contrarian": 3}
    takes.sort(key=lambda t: order.get(t.persona, 9))
    reply = get_llm_client().chat_json(_SYNTH_SYSTEM, {
        "direction": direction,
        "takes": [{"persona": t.persona, "stance": t.stance,
                   "confidence": t.confidence, "note": t.note}
                  for t in takes],
    })
    if not isinstance(reply, dict):
        return None
    verdict = reply.get("verdict")
    confidence = _clamp_confidence(reply.get("confidence"))
    if verdict not in _VALID_VERDICTS or confidence is None:
        return None
    rationale = str(reply.get("rationale", "")).strip()[:240]
    return PanelVerdict(verdict=verdict, confidence=confidence,
                        rationale=rationale, takes=takes,
                        created_ts=time.time())
 # ── Cache, budget, attach ────────────────────────────────────────────
 _state_lock = threading.Lock()
 # Exact result per reviewed bar: repeated attaches on the same signal
 # (re-scan, multi-consumer) are free.
 _cache: "OrderedDict[Tuple[str, str, str, str], PanelVerdict]" = OrderedDict()
 _CACHE_MAX = 32
 # Reuse layer: a persisting signal (same symbol/tf/direction, newer bar)
 # re-serves its last verdict for reuse_ttl_seconds instead of re-spending
 # 5 requests per bar (see module docstring for the M5 dashboard math).
 _last_by_dir: Dict[Tuple[str, str, str], PanelVerdict] = {}
 # UTC-day budget of panel attempts (mirrors llm_client._under_cap).
 _panel_day: Optional[_dt.date] = None
 _panel_count: int = 0
 _warned_budget = False
 def reset_panel_state() -> None:
    """Drop caches and counters (tests / config reloads)."""
    global _panel_day, _panel_count, _warned_budget
    with _state_lock:
        _cache.clear()
        _last_by_dir.clear()
        _panel_day = None
        _panel_count = 0
        _warned_budget = False
 def panel_health() -> Dict[str, Any]:
    """Read-only counters for status displays."""
    settings = get_app_config().agent_panel_settings
    cap = int(settings.get("max_panels_per_day", DEFAULT_MAX_PANELS_PER_DAY))
    today = _dt.datetime.now(_dt.timezone.utc).date()
    with _state_lock:
        used = _panel_count if _panel_day == today else 0
    return {"used_today": used, "cap": cap}
 def _under_panel_budget(cap: int) -> bool:
    """Count one panel attempt against the UTC-day budget."""
    global _panel_day, _panel_count, _warned_budget
    today = _dt.datetime.now(_dt.timezone.utc).date()
    with _state_lock:
        if _panel_day != today:
            _panel_day = today
            _panel_count = 0
            _warned_budget = False
        if _panel_count >= cap:
            if not _warned_budget:
                _warned_budget = True
                logger.warning(
                    "agent panel: daily budget (%d) exhausted — no more "
                    "panels until UTC midnight", cap)
            return False
        _panel_count += 1
        return True
 def _write_fields(report: "SignalReport", v: PanelVerdict) -> None:
    report.panel_verdict = v.verdict
    report.panel_confidence = float(v.confidence)
    report.panel_rationale = v.rationale
    report.panel_takes = [
        {"persona": t.persona, "stance": t.stance,
         "confidence": t.confidence, "note": t.note} for t in v.takes]
 def attach_panel(report: "SignalReport",
                 settings: Optional[Dict[str, Any]] = None,
                 surface: str = "telegram",
                 force: bool = False,
                 context: Optional["FundamentalContext"] = None) -> bool:
    """Best-effort panel attach; True iff panel fields were populated.
    Advisory feature contract: NEVER raises — an exploding panel must
    not cost an alert send or a dashboard publish. ``force`` (CLI
    --panel) bypasses the enabled/surface/quality toggles but never the
    daily budget or the shared LLM cap.
    """
    try:
        if settings is None:
            settings = get_app_config().agent_panel_settings
        if not force:
            if not settings.get("enabled", False):
                return False
            surface_on = {
                "telegram": settings.get("telegram", True),
                "dashboard": settings.get("dashboard", False),
            }.get(surface, True)
            if not surface_on:
                return False
            min_quality = settings.get("min_quality", DEFAULT_MIN_QUALITY)
            if (_QUALITY_RANK.get(report.signal_quality, -1)
                    < _QUALITY_RANK.get(min_quality, 2)):
                return False
        direction = _direction(report)
        if direction is None:
            return False
        analysis = report.primary_analysis
        tf = analysis.timeframe if analysis else ""
        exact_key = (report.symbol, tf, report.timestamp, direction)
        dir_key = (report.symbol, tf, direction)
        ttl = float(settings.get("reuse_ttl_seconds",
                                 DEFAULT_REUSE_TTL_SECONDS))
        with _state_lock:
            hit = _cache.get(exact_key)
            if hit is None:
                prior = _last_by_dir.get(dir_key)
                if prior is not None and time.time() - prior.created_ts < ttl:
                    hit = prior
        if hit is not None:
            _write_fields(report, hit)
            return True
        cap = int(settings.get("max_panels_per_day",
                               DEFAULT_MAX_PANELS_PER_DAY))
        if not _under_panel_budget(cap):
            return False
        if context is None:
            # Best-effort; the news engine's class-level 600 s caches make
            # this nearly free when warm in both bot and dashboard.
            try:
                from config import PlatformConfig
                from news_engine import NewsEngine
                context = NewsEngine(
                    PlatformConfig().news).get_fundamental_context()
            except Exception as e:
                logger.debug("agent panel: news context unavailable: %s", e)
                context = None
        verdict = run_panel(report, context)
        if verdict is None:
            return False
        _write_fields(report, verdict)
        with _state_lock:
            _cache[exact_key] = verdict
            while len(_cache) > _CACHE_MAX:
                _cache.popitem(last=False)
            _last_by_dir[dir_key] = verdict
        return True
    except Exception:
        logger.exception("agent panel: attach failed — signal unaffected")
        return False
@@ -0,0 +1,428 @@
 """
 app_config.py — deployment-settings overlay (config.json)
 =========================================================
 Loads the gitignored config.json at the repo root ONCE per process.
 Deployment surface ONLY: Telegram credentials, data-provider selection
 and API keys, scanned pairs/timeframes, per-pair base blocked hours,
 alert-quality boot default, log level. Signal logic, weights,
 thresholds and gates stay in config.py / pair_signals.
 Precedence for every setting: env var → config.json → code default.
 A repo-root .env file (gitignored) is loaded at import via setdefault,
 so it behaves exactly like real environment variables except that a
 real shell export always wins. Secrets have NO code default — with
 neither env nor config.json the bot refuses to start
 (telegram_bot.main) and data fetches fail with a named error
 (data_provider).
 Parity contract (test-enforced): with no config.json, or one holding
 the config.json.example defaults, every produced PlatformConfig is
 byte-identical to the pure-code defaults. apply_pair_overrides() runs
 in the pair factory BEFORE the validated per-timeframe deltas, so the
 bot, CLI and backtester all see the same overlay (live == backtest).
 Spec: docs/superpowers/specs/2026-07-21-centralized-config-design.md
 """
 import json
 import logging
 import os
 from dataclasses import dataclass, field
 from typing import Any, Dict, List, Optional
 from config import PAIR_INFO, PlatformConfig, Timeframe, XAUUSD
 logger = logging.getLogger(__name__)
 CONFIG_PATH = os.path.join(
    os.path.dirname(os.path.abspath(__file__)), "config.json")
 ENV_PATH = os.path.join(
    os.path.dirname(os.path.abspath(__file__)), ".env")
 def load_dotenv(path: str = ENV_PATH) -> None:
    """Minimal .env loader (no python-dotenv dependency).
    KEY=VALUE lines land in os.environ via setdefault — a variable
    already exported in the shell always wins, keeping the
    env → config.json precedence intact (.env is just a convenient
    place to put env vars). Comments, blank lines, ``export `` prefixes
    and single/double quotes are handled; anything else is skipped.
    """
    try:
        with open(path, encoding="utf-8") as fh:
            lines = fh.read().splitlines()
    except OSError:
        return
    for line in lines:
        line = line.strip()
        if not line or line.startswith("#") or "=" not in line:
            continue
        if line.startswith("export "):
            line = line[len("export "):]
        key, _, value = line.partition("=")
        key, value = key.strip(), value.strip()
        if len(value) >= 2 and value[0] == value[-1] and value[0] in "\"'":
            value = value[1:-1]
        elif value[:1] not in "\"'":
            # Unquoted value: whitespace-then-# starts an inline comment
            # (KEY=abc  # prod). Quote the value to keep a literal " #".
            for sep in (" #", "\t#"):
                cut = value.find(sep)
                if cut != -1:
                    value = value[:cut].rstrip()
        if key:
            os.environ.setdefault(key, value)
 # Module import == process boot, before any resolver can run.
 load_dotenv()
 _VALID_QUALITIES = ("A+", "A", "B", "C")
 _VALID_LOG_LEVELS = ("DEBUG", "INFO", "WARNING", "ERROR", "CRITICAL")
 # Timeframes the bot's candle-boundary scheduler supports (must have
 # AUTOSCAN_CONFIRMATIONS entries in telegram_bot.py).
 _SCANNABLE_TFS = ("M5", "M15", "H1", "H4")
 _KNOWN_TOP = {"telegram", "data_providers", "scanning", "pairs", "log_level",
              "oanda", "deepseek", "agent_panel"}
 _KNOWN_OANDA = {
    "account_id", "api_token", "base_url", "lot_size",
    "quality_gate", "max_hold_bars",
 }
 _KNOWN_DEEPSEEK = {
    "api_key", "model", "enabled", "timeout_seconds", "daily_request_cap",
 }
 _KNOWN_AGENT_PANEL = {
    "enabled", "telegram", "dashboard", "min_quality",
    "max_panels_per_day", "reuse_ttl_seconds",
 }
 _KNOWN_TELEGRAM = {"token", "chat_id", "channel_id", "alert_min_quality"}
 _KNOWN_SCANNING = {"pairs", "timeframes"}
 _KNOWN_PAIR_KEYS = {"blocked_hours_utc"}
 class AppConfigError(Exception):
    """config.json is present but invalid — fail fast at boot."""
@dataclass
 class AppConfig:
    telegram_token: str = ""
    telegram_chat_id: str = ""
    telegram_channel_id: str = ""          # "" = channel feed disabled
    alert_min_quality: str = "C"
    active_provider: str = "twelve_data"
    streaming_provider: Optional[str] = None   # None = active provider
    provider_settings: Dict[str, Dict[str, Any]] = field(default_factory=dict)
    scan_pairs: List[str] = field(default_factory=lambda: [XAUUSD])
    scan_timeframes: List[Timeframe] = field(default_factory=lambda: [
        Timeframe.M5, Timeframe.M15, Timeframe.H1, Timeframe.H4])
    pair_overrides: Dict[str, Dict[str, Any]] = field(default_factory=dict)
    oanda_settings: Dict[str, Any] = field(default_factory=dict)
    deepseek_settings: Dict[str, Any] = field(default_factory=dict)
    agent_panel_settings: Dict[str, Any] = field(default_factory=dict)
    log_level: str = "INFO"
 def _check_keys(section: Dict[str, Any], known: set, where: str) -> None:
    if not isinstance(section, dict):
        raise AppConfigError(f"config.json: {where} must be a JSON object")
    unknown = sorted(set(section) - known)
    if unknown:
        raise AppConfigError(
            f"config.json: unknown key(s) {unknown} in {where}; "
            f"known keys: {sorted(known)}")
 def load_app_config(path: str = CONFIG_PATH) -> AppConfig:
    """Parse + validate config.json. Missing file → pure defaults."""
    app = AppConfig()
    if not os.path.exists(path):
        logger.warning(
            "config.json not found at %s — running on code defaults "
            "(secrets must come from environment variables)", path)
        return app
    try:
        with open(path, encoding="utf-8") as fh:
            raw = json.load(fh)
    except json.JSONDecodeError as e:
        raise AppConfigError(f"config.json is not valid JSON: {e}") from e
    if not isinstance(raw, dict):
        raise AppConfigError("config.json top level must be a JSON object")
    _check_keys(raw, _KNOWN_TOP, "top level")
    tg = raw.get("telegram", {})
    _check_keys(tg, _KNOWN_TELEGRAM, '"telegram"')
    app.telegram_token = str(tg.get("token", app.telegram_token))
    app.telegram_chat_id = str(tg.get("chat_id", app.telegram_chat_id))
    app.telegram_channel_id = str(
        tg.get("channel_id", app.telegram_channel_id))
    quality = tg.get("alert_min_quality", app.alert_min_quality)
    if quality not in _VALID_QUALITIES:
        raise AppConfigError(
            f'config.json: telegram.alert_min_quality "{quality}" '
            f"not in {_VALID_QUALITIES}")
    app.alert_min_quality = quality
    dp = raw.get("data_providers")
    if dp is not None:
        if not isinstance(dp, dict):
            raise AppConfigError('config.json: "data_providers" must be an object')
        app.active_provider = str(dp.get("active", app.active_provider))
        app.provider_settings = {}
        for name, block in dp.items():
            # "active" and "streaming" are reserved scalars, not blocks.
            if name in ("active", "streaming"):
                continue
            if not isinstance(block, dict):
                raise AppConfigError(
                    f'config.json: data_providers.{name} must be an object')
            app.provider_settings[name] = dict(block)
        if app.active_provider not in app.provider_settings:
            raise AppConfigError(
                f'config.json: data_providers.active "{app.active_provider}" '
                "has no matching provider block")
        streaming = dp.get("streaming")
        if streaming is not None:
            if not isinstance(streaming, str):
                raise AppConfigError(
                    'config.json: data_providers.streaming must be a string '
                    "(a provider name)")
            if streaming not in app.provider_settings:
                raise AppConfigError(
                    f'config.json: data_providers.streaming "{streaming}" '
                    "has no matching provider block")
            app.streaming_provider = streaming
    scanning = raw.get("scanning", {})
    _check_keys(scanning, _KNOWN_SCANNING, '"scanning"')
    if "pairs" in scanning:
        pairs = scanning["pairs"]
        if not isinstance(pairs, list):
            raise AppConfigError('config.json: scanning.pairs must be a list')
        bad = [p for p in pairs if p not in PAIR_INFO]
        if bad:
            raise AppConfigError(
                f"config.json: unknown pair(s) {bad} in scanning.pairs; "
                f"known: {sorted(PAIR_INFO)}")
        app.scan_pairs = list(pairs)
    if "timeframes" in scanning:
        names = scanning["timeframes"]
        if not isinstance(names, list):
            raise AppConfigError(
                'config.json: scanning.timeframes must be a list')
        bad = [n for n in names if n not in _SCANNABLE_TFS]
        if bad:
            raise AppConfigError(
                f"config.json: unsupported timeframe(s) {bad} in "
                f"scanning.timeframes; supported: {list(_SCANNABLE_TFS)}")
        app.scan_timeframes = [Timeframe[n] for n in names]
    pair_section = raw.get("pairs", {})
    if not isinstance(pair_section, dict):
        raise AppConfigError('config.json: "pairs" must be a JSON object')
    for symbol, overrides in pair_section.items():
        if symbol not in PAIR_INFO:
            raise AppConfigError(
                f'config.json: unknown pair "{symbol}" in pairs; '
                f"known: {sorted(PAIR_INFO)}")
        _check_keys(overrides, _KNOWN_PAIR_KEYS, f'pairs."{symbol}"')
        if "blocked_hours_utc" in overrides:
            hours = overrides["blocked_hours_utc"]
            valid = isinstance(hours, list) and all(
                isinstance(h, int) and not isinstance(h, bool)
                and 0 <= h <= 23 for h in hours)
            if not valid:
                raise AppConfigError(
                    f'config.json: pairs."{symbol}".blocked_hours_utc must '
                    "be a list of integers in 0-23")
    app.pair_overrides = {s: dict(o) for s, o in pair_section.items()}
    oanda = raw.get("oanda", {})
    _check_keys(oanda, _KNOWN_OANDA, '"oanda"')
    if "lot_size" in oanda:
        ls = oanda["lot_size"]
        if not isinstance(ls, int) or isinstance(ls, bool) or ls < 1:
            raise AppConfigError(
                "config.json: oanda.lot_size must be a positive integer")
    if "quality_gate" in oanda and oanda["quality_gate"] not in _VALID_QUALITIES:
        raise AppConfigError(
            f'config.json: oanda.quality_gate "{oanda["quality_gate"]}" '
            f"not in {_VALID_QUALITIES}")
    if "max_hold_bars" in oanda:
        mhb = oanda["max_hold_bars"]
        if not isinstance(mhb, int) or isinstance(mhb, bool) or mhb < 1:
            raise AppConfigError(
                "config.json: oanda.max_hold_bars must be a positive integer")
    app.oanda_settings = dict(oanda)
    deepseek = raw.get("deepseek", {})
    _check_keys(deepseek, _KNOWN_DEEPSEEK, '"deepseek"')
    if "enabled" in deepseek and not isinstance(deepseek["enabled"], bool):
        raise AppConfigError(
            "config.json: deepseek.enabled must be true or false")
    if "timeout_seconds" in deepseek:
        ts = deepseek["timeout_seconds"]
        if isinstance(ts, bool) or not isinstance(ts, (int, float)) or ts <= 0:
            raise AppConfigError(
                "config.json: deepseek.timeout_seconds must be a positive "
                "number")
    if "daily_request_cap" in deepseek:
        cap = deepseek["daily_request_cap"]
        if isinstance(cap, bool) or not isinstance(cap, int) or cap < 1:
            raise AppConfigError(
                "config.json: deepseek.daily_request_cap must be a positive "
                "integer")
    if "model" in deepseek:
        if not isinstance(deepseek["model"], str) or not deepseek["model"]:
            raise AppConfigError(
                "config.json: deepseek.model must be a non-empty string")
    app.deepseek_settings = dict(deepseek)
    panel = raw.get("agent_panel", {})
    _check_keys(panel, _KNOWN_AGENT_PANEL, '"agent_panel"')
    for flag in ("enabled", "telegram", "dashboard"):
        if flag in panel and not isinstance(panel[flag], bool):
            raise AppConfigError(
                f"config.json: agent_panel.{flag} must be true or false")
    if "min_quality" in panel and panel["min_quality"] not in _VALID_QUALITIES:
        raise AppConfigError(
            f'config.json: agent_panel.min_quality "{panel["min_quality"]}" '
            f"not in {_VALID_QUALITIES}")
    if "max_panels_per_day" in panel:
        mpd = panel["max_panels_per_day"]
        if isinstance(mpd, bool) or not isinstance(mpd, int) or mpd < 1:
            raise AppConfigError(
                "config.json: agent_panel.max_panels_per_day must be a "
                "positive integer")
    if "reuse_ttl_seconds" in panel:
        ttl = panel["reuse_ttl_seconds"]
        if isinstance(ttl, bool) or not isinstance(ttl, (int, float)) \
                or ttl < 0:
            raise AppConfigError(
                "config.json: agent_panel.reuse_ttl_seconds must be a "
                "non-negative number")
    app.agent_panel_settings = dict(panel)
    level = raw.get("log_level", app.log_level)
    if level not in _VALID_LOG_LEVELS:
        raise AppConfigError(
            f'config.json: log_level "{level}" not in {_VALID_LOG_LEVELS}')
    app.log_level = level
    return app
 # ── Process-wide singleton ────────────────────────────────────────────
 _cached: Optional[AppConfig] = None
 def get_app_config() -> AppConfig:
    global _cached
    if _cached is None:
        _cached = load_app_config()
    return _cached
 def set_app_config(app: Optional[AppConfig]) -> None:
    """Override the process-wide AppConfig (tests/experiments).
    Pass None to re-enable lazy loading from CONFIG_PATH."""
    global _cached
    _cached = app
 # ── Resolution helpers: env var → config.json → default ───────────────
 def telegram_token(app: Optional[AppConfig] = None) -> str:
    app = app or get_app_config()
    return os.environ.get("TELEGRAM_TOKEN") or app.telegram_token
 def telegram_chat_id(app: Optional[AppConfig] = None) -> str:
    app = app or get_app_config()
    return os.environ.get("TELEGRAM_CHAT_ID") or app.telegram_chat_id
 def telegram_channel_id(app: Optional[AppConfig] = None) -> str:
    """Env set-but-empty ("") means "disable the feed" and beats JSON,
    so this checks None-ness, not truthiness."""
    app = app or get_app_config()
    env = os.environ.get("TELEGRAM_CHANNEL_ID")
    return env if env is not None else app.telegram_channel_id
 # Per-provider env-var overrides for API keys (env → config.json → "").
 _PROVIDER_ENV_KEYS = {
    "twelve_data": "TWELVE_DATA_API_KEY",
    "tiingo": "TIINGO_API_KEY",
 }
 def provider_api_key(
    app: Optional[AppConfig] = None, provider: Optional[str] = None
 ) -> str:
    """API key for `provider` (default: the ACTIVE provider). The env
    override is per-provider (TWELVE_DATA_API_KEY applies only to
    twelve_data, TIINGO_API_KEY only to tiingo)."""
    app = app or get_app_config()
    name = provider or app.active_provider
    env_key = _PROVIDER_ENV_KEYS.get(name)
    if env_key:
        env = os.environ.get(env_key)
        if env:
            return env
    block = app.provider_settings.get(name, {})
    return str(block.get("api_key", ""))
 # ── Oanda resolution helpers (env → config.json → default) ───────────
 def oanda_account_id(app: Optional[AppConfig] = None) -> str:
    app = app or get_app_config()
    return os.environ.get("OANDA_ACCOUNT_ID") or str(
        app.oanda_settings.get("account_id", ""))
 def oanda_api_token(app: Optional[AppConfig] = None) -> str:
    app = app or get_app_config()
    return os.environ.get("OANDA_API_TOKEN") or str(
        app.oanda_settings.get("api_token", ""))
 def oanda_base_url(app: Optional[AppConfig] = None) -> str:
    app = app or get_app_config()
    return os.environ.get("OANDA_BASE_URL") or str(
        app.oanda_settings.get("base_url", "https://api-fxpractice.oanda.com"))
 # ── DeepSeek resolution helper (env → config.json → "") ──────────────
 def deepseek_api_key(app: Optional[AppConfig] = None) -> str:
    app = app or get_app_config()
    return os.environ.get("DEEPSEEK_API_KEY") or str(
        app.deepseek_settings.get("api_key", ""))
 # ── PlatformConfig overlay ────────────────────────────────────────────
 def apply_pair_overrides(
    cfg: PlatformConfig, app: Optional[AppConfig] = None
 ) -> PlatformConfig:
    """Stamp deployment settings from config.json onto a PlatformConfig.
    Called by the pair factories BEFORE their validated per-timeframe
    deltas, so e.g. the M5 hour-9 append lands on top of the JSON base.
    The pair's base blocked_hours_utc is set on BOTH mirrored copies
    (xau_strategy + backtest) so they cannot drift.
    """
    app = app or get_app_config()
    overrides = app.pair_overrides.get(cfg.symbol, {})
    if "blocked_hours_utc" in overrides:
        hours = list(overrides["blocked_hours_utc"])
        cfg.xau_strategy.blocked_hours_utc = hours
        cfg.backtest.blocked_hours_utc = list(hours)
    cfg.scan_pairs = list(app.scan_pairs)
    cfg.scan_timeframes = list(app.scan_timeframes)
    return cfg
@@ -51,7 +51,7 @@ from gpu_backtest import (  # noqa: E402
    precompute_indicators,
    _is_xau_blocked_by_calendar,
 )
-from data_provider import ForexDataProvider  # noqa: E402
+from providers import make_data_provider  # noqa: E402
 logger = logging.getLogger("diagnose_filters")
@@ -79,34 +79,54 @@ def classify_bar(
    """
    filters_hit: List[str] = []
-    # Scan warnings for known filter tags
+    # Known filter tags, matched against the exact warning strings the
-    filter_tags = {
+    # aggregator emits (refreshed 2026-07-30 for gates 6d-6i). ORDER
-        "H4 RSI GATE":       "H4_RSI_GATE",
+    # MATTERS: most-specific prefix first, and each warning maps to at
-        "H1 MACD GATE":      "H1_MACD_GATE",
+    # most ONE tag (break on first hit) — substring families like
-        "RSI ZONE GATE":     "RSI_ZONE_GATE",
+    # "M5 FILTER: SELL quality" vs "M5 FILTER" and "TREND EXHAUSTION"
-        "XAU BUY FILTER":    "XAU_BUY_FILTER",
+    # vs "EXHAUSTION" would otherwise double-count.
-        "SESSION FILTER":    "SESSION_FILTER",
+    filter_tags = [
-        "DAY FILTER":        "DAY_FILTER",
+        # M5 noise gates (6i, _apply_m5_filters)
-        "LOW ATR FILTER":    "LOW_ATR_FILTER",
+        ("M5 FILTER: |score|",            "M5_SCORE"),
-        "SELL SUPPRESSED":   "SELL_SUPPRESSED",
+        ("M5 FILTER: no MTF agreement",   "M5_MTF"),
-        "EXHAUSTION":        "EXHAUSTION_PENALTY",
+        ("M5 FILTER: no confirmation TF", "M5_CONF"),
-        "TREND EXHAUSTION":  "TREND_EXHAUSTION",
+        ("M5 FILTER: SELL quality",       "M5_SELL_QUALITY"),
-        "NO PULLBACK":       "NO_PULLBACK_PENALTY",
+        ("M5 FILTER: SELL blocked — RSI", "M5_SELL_RSI"),
-        "Weak pullback":     "WEAK_PULLBACK_PENALTY",
+        ("M5 FILTER: '",                  "M5_VOL"),
-        "Extended entry":    "EXTENDED_ENTRY_PENALTY",
+        ("M5 FILTER",                     "M5_OTHER"),
-        "Counter-trend":     "COUNTER_TREND_PENALTY",
+        # XAU SELL gates (6e/6g/6g2/6g3)
-        "EXTREME volatility":"EXTREME_VOL_PENALTY",
+        ("SELL FILTER: quality",          "SELL_QUALITY"),
-        "NEAR EMA200":       "NEAR_EMA200_PENALTY",
+        ("SELL FILTER: volatility",       "SELL_VOL"),
-        "FAR FROM EMA200":   "FAR_EMA200_PENALTY",
+        ("SELL FILTER: regime",           "SELL_REGIME"),
-        "EVENT RISK":        "EVENT_RISK",
+        ("UTC entry blocked for shorts",  "SELL_HOUR"),
-        "REGIME FILTER":     "REGIME_FILTER",
+        # Hard gates
-        "Timeframes CONFLICT": "MTF_CONFLICT",
+        ("MTF OPPOSITION",                "MTF_OPPOSITION"),
-    }
+        ("EXHAUSTED TREND FILTER",        "EXHAUSTED_TREND"),
        ("BIG CANDLE FILTER",             "BIG_CANDLE"),
        ("WEEKEND FILTER",                "WEEKEND_FILTER"),
        ("SESSION FILTER",                "SESSION_FILTER"),
        ("DAY FILTER",                    "DAY_FILTER"),
        ("LOW ATR FILTER",                "LOW_ATR_FILTER"),
        ("REGIME FILTER",                 "REGIME_FILTER"),
        # Soft penalties / context warnings
        ("TREND EXHAUSTION",              "TREND_EXHAUSTION"),
        ("EXHAUSTION",                    "EXHAUSTION_PENALTY"),
        ("NO PULLBACK",                   "NO_PULLBACK_PENALTY"),
        ("Weak pullback",                 "WEAK_PULLBACK_PENALTY"),
        ("Extended — RSI",                "RSI_EXTENDED_PENALTY"),
        ("Counter-trend",                 "COUNTER_TREND_PENALTY"),
        ("EXTREME volatility",            "EXTREME_VOL_PENALTY"),
        ("NEAR EMA200",                   "NEAR_EMA200_PENALTY"),
        ("FAR FROM EMA200",               "FAR_EMA200_PENALTY"),
        ("EVENT RISK",                    "EVENT_RISK"),
        ("Timeframes CONFLICT",           "MTF_CONFLICT"),
    ]
    for w in report.warnings:
-        for tag, name in filter_tags.items():
+        for tag, name in filter_tags:
            if tag in w:
                filters_hit.append(name)
                break
    # Determine outcome
    non_neutral = report.final_signal not in (SignalStrength.NEUTRAL,)
@@ -206,7 +226,7 @@ def run_diagnosis(
        confirmation_timeframes=list(confirm_tfs),
    )
-    provider = ForexDataProvider(platform_cfg)
+    provider = make_data_provider(platform_cfg)
    # ── Load data ──
    timeframes = [primary_tf] + list(confirm_tfs)
@@ -554,7 +574,7 @@ def main(argv: Optional[List[str]] = None) -> int:
        description="Diagnose which filters reject the most bars in the backtest pipeline",
        formatter_class=argparse.ArgumentDefaultsHelpFormatter,
    )
-    p.add_argument("--pair", required=True, help="EURUSD | GBPUSD | USDJPY | XAUUSD")
+    p.add_argument("--pair", required=True, help="XAUUSD | GOLD")
    p.add_argument("--days", type=int, default=365, help="Days of history")
    p.add_argument("--start", help="ISO date YYYY-MM-DD (overrides --days)")
    p.add_argument("--end", help="ISO date YYYY-MM-DD (defaults to today)")
@@ -0,0 +1,5 @@
 """Research tooling: calendar sourcing/store and edge-screening utilities.
 Nothing here is imported by the live bot; these modules support
 backtesting and edge validation only.
 """
@@ -0,0 +1,86 @@
 """Trade-book diff: relaxed/gated run vs its control.
 Usage:
    python backtest/research/book_diff.py CONTROL.csv CANDIDATE.csv BAR_MINUTES
 Decision numbers are RUN-LEVEL (ΔtotalR, ΔPF, ΔmaxDD, Δtrades) — never
 the added-trade bucket alone: the 5-bar entry cooldown means new
 entries displace existing ones. The added/removed decomposition and the
 per-decision-hour histogram of added trades are diagnostics.
 Shifted trades = removed+added pairs in the same direction within
 ±5 bars (cooldown displacement, not a genuinely new/lost signal).
 """
 import sys
 import pandas as pd
 def _metrics(df):
    r = df["r_multiple"]
    gains, losses = r[r > 0].sum(), -r[r <= 0].sum()
    cum = r.cumsum()
    return dict(trades=len(df), total_r=r.sum(),
                pf=gains / max(losses, 1e-9),
                wr=(r > 0).mean() * 100 if len(df) else 0.0,
                max_dd=(cum.cummax() - cum).max() if len(cum) else 0.0)
 def diff_books(control_csv, candidate_csv, bar_minutes, split="2026-01-29"):
    ctrl, cand = pd.read_csv(control_csv), pd.read_csv(candidate_csv)
    for df in (ctrl, cand):
        df["_t"] = pd.to_datetime(df["entry_time"], utc=True, format="mixed")
        df["_key"] = df["entry_time"].astype(str) + "|" + df["direction"]
    added = cand[~cand["_key"].isin(ctrl["_key"])].copy()
    removed = ctrl[~ctrl["_key"].isin(cand["_key"])].copy()
    # cooldown displacement: same-direction removed/added within ±5 bars
    tol = pd.Timedelta(minutes=5 * bar_minutes)
    shifted = 0
    rem_pool = removed.copy()
    for _, a in added.iterrows():
        m = rem_pool[(rem_pool["direction"] == a["direction"])
                     & ((rem_pool["_t"] - a["_t"]).abs() <= tol)]
        if len(m):
            shifted += 1
            rem_pool = rem_pool.drop(m.index[0])
    mc, mk = _metrics(ctrl), _metrics(cand)
    split_ts = pd.Timestamp(split, tz="UTC")
    halves = (cand[cand["_t"] < split_ts]["r_multiple"].sum(),
              cand[cand["_t"] >= split_ts]["r_multiple"].sum())
    hour_hist = None
    if len(added):
        added["dec_hour"] = (added["_t"]
                             - pd.Timedelta(minutes=bar_minutes)).dt.hour
        hour_hist = added.groupby("dec_hour")["r_multiple"] \
                         .agg(["count", "sum"]).round(2)
    return dict(
        control=mc, candidate=mk,
        delta_r=mk["total_r"] - mc["total_r"],
        delta_pf=mk["pf"] - mc["pf"],
        delta_dd=mk["max_dd"] - mc["max_dd"],
        delta_trades=mk["trades"] - mc["trades"],
        added_n=len(added), added_r=added["r_multiple"].sum(),
        removed_n=len(removed), removed_r=removed["r_multiple"].sum(),
        shifted=shifted, halves=halves, added_by_hour=hour_hist,
    )
 if __name__ == "__main__":
    d = diff_books(sys.argv[1], sys.argv[2], int(sys.argv[3]))
    c, k = d["control"], d["candidate"]
    print(f"control  : {c['trades']:4d} trades  {c['total_r']:+7.1f}R  "
          f"PF {c['pf']:.2f}  WR {c['wr']:.1f}%  DD {c['max_dd']:.1f}")
    print(f"candidate: {k['trades']:4d} trades  {k['total_r']:+7.1f}R  "
          f"PF {k['pf']:.2f}  WR {k['wr']:.1f}%  DD {k['max_dd']:.1f}")
    print(f"ΔR {d['delta_r']:+.1f}  ΔPF {d['delta_pf']:+.2f}  "
          f"ΔDD {d['delta_dd']:+.1f}  Δtrades {d['delta_trades']:+d}")
    print(f"added {d['added_n']} ({d['added_r']:+.1f}R)  "
          f"removed {d['removed_n']} ({d['removed_r']:+.1f}R)  "
          f"shifted≈{d['shifted']}")
    print(f"candidate halves: H1 {d['halves'][0]:+.1f}R  H2 {d['halves'][1]:+.1f}R")
    if d["added_by_hour"] is not None:
        print("\nadded trades by decision hour:")
        print(d["added_by_hour"].to_string())
@@ -0,0 +1,42 @@
 """Burst-candle frequency by UTC hour — the empirical "hedge fund hours".
 Usage:
    python backtest/research/burst_profile.py PARQUET_GLOB
 Per UTC bar-close hour of the given bars: % of bars whose range is
 >= K x the PRIOR bar's Wilder ATR(14), for K in {3,4,5}; mean bar
 range; and mean signed drift over the 6 bars following a >=4x burst
 (does price continue or snap back?). Analysis tooling only.
 """
 import glob
 import sys
 import pandas as pd
 def burst_table(parquet_glob: str, bar_minutes: int = 5):
    files = sorted(glob.glob(parquet_glob))
    if not files:
        raise SystemExit(f"no parquet matches {parquet_glob}")
    df = pd.concat([pd.read_parquet(f) for f in files]).sort_index()
    df = df[~df.index.duplicated(keep="last")]
    h, l, c = df["High"], df["Low"], df["Close"]
    tr = pd.concat([h - l, (h - c.shift()).abs(), (l - c.shift()).abs()],
                   axis=1).max(axis=1)
    atr = tr.ewm(alpha=1 / 14, min_periods=14, adjust=False).mean()
    ratio = (h - l) / atr.shift(1)
    close_hour = (df.index + pd.Timedelta(minutes=bar_minutes)).hour
    fwd6 = c.shift(-6) - c  # signed move over the 6 bars after this one
    out = pd.DataFrame({
        "bars": ratio.groupby(close_hour).size(),
        **{f"pct_ge_{k}x": (ratio >= k).groupby(close_hour).mean() * 100
           for k in (3, 4, 5)},
        "mean_range": (h - l).groupby(close_hour).mean(),
        "post4x_drift6": fwd6[ratio >= 4].groupby(close_hour[ratio >= 4]).mean(),
    })
    return out.round(3)
 if __name__ == "__main__":
    print(burst_table(sys.argv[1],
                      int(sys.argv[2]) if len(sys.argv) > 2 else 5).to_string())
@@ -0,0 +1,178 @@
 """Commit-ladder campaign driver (2026-08).
 Question: of all signal-code states shipped in the last two months, which
 configuration is the most accurate? Historical in-era backtest numbers are
 not comparable (the backtester's realism model changed on 07-08/07-16 and
 the data feed switched to Oanda on 07-28), so every historical
 configuration is recreated on TODAY's harness via --set overrides and
 replayed on the same pinned Oanda bars. Bug-fix commits stay on; the
 ladder compares shipped tuning decisions only.
 Usage:
    python backtest/research/commit_ladder.py            # full matrix
    python backtest/research/commit_ladder.py --tf 5min  # one timeframe
    python backtest/research/commit_ladder.py --windows 12mo,oos
 Resumable: a run whose CSV already exists in the output dir is skipped.
 Outputs: backtest/research/commit_ladder_2026-08/<variant>_<window>.csv
         + .log per run, summary.csv, and a printed ranking table.
 """
 from __future__ import annotations
 import argparse
 import subprocess
 import sys
 from pathlib import Path
 import pandas as pd
 ROOT = Path(__file__).resolve().parents[2]
 BT = ROOT / "backtest" / "gpu_backtest.py"
 OUT = ROOT / "backtest" / "research" / "commit_ladder_2026-08"
 WINDOWS = {
    "12mo": ("2025-07-30", "2026-07-30"),
    "oos": ("2024-06-01", "2025-03-30"),
    "5yr": ("2021-07-30", "2026-07-30"),
 }
 SPLIT_12MO = pd.Timestamp("2026-01-29", tz="UTC")  # book_diff.py's split
 # (variant, "commit date what-it-represents", [--set overrides])
 VARIANTS: dict[str, list[tuple[str, str, list[str]]]] = {
    "5min": [
        ("M5-A", "dad2432 07-09 gates+trailing+pullback",
         ["xau_strategy.blocked_hours_utc=[0,1,2,10,11,13,16,21]",
          "xau_strategy.cp_enabled=false",
          "signals.trail_dist_r=0.75",
          "xau_strategy.m5_min_abs_score=30",
          "xau_strategy.m5_min_confirmation_score=10"]),
        ("M5-B", "f3ba4d5 07-11 +hour-9 block",
         ["xau_strategy.blocked_hours_utc=[0,1,2,10,11,13,16,21,9]",
          "xau_strategy.cp_enabled=false",
          "signals.trail_dist_r=0.75",
          "xau_strategy.m5_min_abs_score=30",
          "xau_strategy.m5_min_confirmation_score=10"]),
        ("M5-C", "de6424e 07-16 +chart patterns",
         ["xau_strategy.blocked_hours_utc=[0,1,2,10,11,13,16,21,9]",
          "signals.trail_dist_r=0.75",
          "xau_strategy.m5_min_abs_score=30",
          "xau_strategy.m5_min_confirmation_score=10"]),
        ("M5-D", "3ee79b5 07-21 +trail dist 0.85",
         ["xau_strategy.blocked_hours_utc=[0,1,2,10,11,13,16,21,9]",
          "xau_strategy.m5_min_abs_score=30",
          "xau_strategy.m5_min_confirmation_score=10"]),
        ("M5-E", "0396970 07-30 relaxation (HEAD)", []),
    ],
    "15min": [
        ("M15-A", "987ec5b 07-09 pre-campaigns",
         ["signals.tp_floor_rr=0.5",
          "xau_strategy.blocked_regimes=[]",
          "xau_strategy.sell_blocked_hours_utc=[]",
          "signals.trail_arm_r=0.60"]),
        ("M15-B", "f3ba4d5/f88e39f 07-11 1R floor + regime block",
         ["xau_strategy.sell_blocked_hours_utc=[]",
          "signals.trail_arm_r=0.60"]),
        ("M15-C", "7d3fd95 07-16 +SELL hour 15/17 block",
         ["signals.trail_arm_r=0.60"]),
        ("M15-D", "bf8e5fb 07-21 trail arm 0.70 (HEAD)", []),
    ],
    "1h": [
        ("H1-A", "pre-d95d31a 07-08 TP-tightening model",
         ["signals.min_rr_ratio=0.4",
          "signals.tp_tightening_factor=0.8"]),
        ("H1-B", "HEAD", []),
    ],
 }
 def run_one(tf: str, name: str, overrides: list[str], wkey: str,
            spread: str | None = None) -> Path:
    suffix = f"_s{spread}" if spread else ""
    csv_path = OUT / f"{name}_{wkey}{suffix}.csv"
    log_path = OUT / f"{name}_{wkey}{suffix}.log"
    if csv_path.exists():
        print(f"  [skip] {csv_path.name} exists")
        return csv_path
    start, end = WINDOWS[wkey]
    cmd = [sys.executable, str(BT), "--pair", "XAUUSD",
           "--primary-tf", tf, "--start", start, "--end", end,
           "--export-csv", str(csv_path)]
    if spread:
        cmd += ["--spread", spread]
    for ov in overrides:
        cmd += ["--set", ov]
    print(f"  [run ] {csv_path.name} ...", flush=True)
    with open(log_path, "w") as lf:
        rc = subprocess.run(cmd, stdout=lf, stderr=subprocess.STDOUT,
                            cwd=ROOT).returncode
    if rc != 0:
        raise SystemExit(f"backtest failed (rc={rc}) — see {log_path}")
    log = log_path.read_text()
    n_ov = log.count("--set override")
    if n_ov != len(overrides):
        raise SystemExit(
            f"{name}_{wkey}: expected {len(overrides)} --set overrides, "
            f"log shows {n_ov} — see {log_path}")
    if "Loading cached" not in log and wkey != "sanity":
        print(f"  [warn] {log_path.name}: no 'Loading cached' line — "
              f"data was fetched live for at least one timeframe")
    return csv_path
 def metrics(csv_path: Path, wkey: str) -> dict:
    df = pd.read_csv(csv_path, parse_dates=["entry_time"])
    if df.empty:
        return {"trades": 0, "total_r": 0.0, "wr": 0.0, "pf": 0.0,
                "max_dd": 0.0, "h1_r": 0.0, "h2_r": 0.0, "years_pos": ""}
    if df["entry_time"].dt.tz is None:
        df["entry_time"] = df["entry_time"].dt.tz_localize("UTC")
    r = df["r_multiple"]
    pos, neg = r[r > 0].sum(), r[r < 0].sum()
    cum = r.cumsum()
    dd = (pd.concat([pd.Series([0.0]), cum]).cummax()[1:] - cum).max()
    out = {
        "trades": len(df),
        "total_r": round(r.sum(), 1),
        "wr": round((r > 0).mean() * 100, 1),
        "pf": round(pos / abs(neg), 2) if neg else float("inf"),
        "max_dd": round(dd, 1),
        "h1_r": "", "h2_r": "", "years_pos": "",
    }
    if wkey == "12mo":
        h1 = r[df["entry_time"] < SPLIT_12MO].sum()
        out["h1_r"] = round(h1, 1)
        out["h2_r"] = round(r.sum() - h1, 1)
    if wkey == "5yr":
        by_year = r.groupby(df["entry_time"].dt.year).sum()
        out["years_pos"] = f"{(by_year > 0).sum()}/{len(by_year)}"
    return out
 def main() -> None:
    ap = argparse.ArgumentParser()
    ap.add_argument("--tf", choices=list(VARIANTS), default=None)
    ap.add_argument("--windows", default="12mo,oos,5yr")
    args = ap.parse_args()
    OUT.mkdir(parents=True, exist_ok=True)
    wkeys = [w.strip() for w in args.windows.split(",")]
    tfs = [args.tf] if args.tf else list(VARIANTS)
    rows = []
    for tf in tfs:
        for name, label, overrides in VARIANTS[tf]:
            for wkey in wkeys:
                csv_path = run_one(tf, name, overrides, wkey)
                m = metrics(csv_path, wkey)
                rows.append({"tf": tf, "variant": name, "represents": label,
                             "window": wkey, **m})
                print(f"    {name} {wkey}: {m['trades']} trades, "
                      f"{m['total_r']}R, PF {m['pf']}, WR {m['wr']}%")
    summary = pd.DataFrame(rows)
    summary.to_csv(OUT / "summary.csv", index=False)
    print("\n=== SUMMARY (also in summary.csv) ===")
    print(summary.to_string(index=False))
 if __name__ == "__main__":
    main()
@@ -0,0 +1,194 @@
 """Dukascopy XAU/USD tick download + bid/ask spread study.
 Validates ForexBot2's $0.30 round-trip cost assumption (config:
 SignalConfig.round_trip_cost / PAIR_INFO default_spread). Round-trip
 cost of a market entry+exit = one ask-bid spread width (buy at ask,
 sell at bid), so the assumption holds iff typical spread ~ $0.30.
 bi5 format: LZMA 'alone' stream of 20-byte big-endian records
 (ms_in_hour uint32, ask uint32, bid uint32, ask_vol f32, bid_vol f32);
 XAUUSD prices scaled x1000. URL months are 0-indexed.
 """
 import lzma
 import struct
 import sys
 from concurrent.futures import ThreadPoolExecutor
 from datetime import date, timedelta
 from pathlib import Path
 import numpy as np
 import pandas as pd
 import requests
 SCALE = 1000.0
 CACHE = Path(__file__).parent / "tick_cache"
 CACHE.mkdir(exist_ok=True)
 URL = "https://datafeed.dukascopy.com/datafeed/XAUUSD/{y}/{m:02d}/{d:02d}/{h:02d}h_ticks.bi5"
 # Bot's tradable bar-close hours: session 03:00-23:00 minus XAU blocked
 # hours [0,1,2,10,11,13,16,21] (M5 additionally blocks 9 — reported
 # separately below).
 BLOCKED = {0, 1, 2, 10, 11, 13, 16, 21}
 TRADABLE = sorted(set(range(3, 23)) - BLOCKED)
 # ── Day sample ────────────────────────────────────────────────────────
 recent = [date(2026, 7, 6) + timedelta(days=i) for i in range(17)]
 recent = [d for d in recent if d.weekday() < 5]              # Mon-Fri Jul 6-22
 monthly = [date(2025, 8, 13), date(2025, 9, 10), date(2025, 10, 15),
           date(2025, 11, 12), date(2025, 12, 10), date(2026, 1, 14),
           date(2026, 2, 11), date(2026, 3, 11), date(2026, 4, 15),
           date(2026, 5, 13), date(2026, 6, 10)]             # mid-month Wednesdays
 nfp = [date(2026, 6, 5), date(2026, 7, 3)]                   # NFP Fridays
 sundays = [date(2026, 7, 12), date(2026, 7, 19)]             # Sun 22-24 open
 DAYS = sorted(set(recent + monthly + nfp + sundays))
 assert all(d.weekday() < 5 for d in recent + monthly + nfp)
 _tls = __import__("threading").local()
 def _session() -> requests.Session:
    if not hasattr(_tls, "s"):
        _tls.s = requests.Session()
        _tls.s.headers["User-Agent"] = "Mozilla/5.0 (research; spread-study)"
    return _tls.s
 def fetch(day: date, hour: int) -> Path:
    p = CACHE / f"{day.isoformat()}_{hour:02d}.bi5"
    if p.exists():
        return p
    u = URL.format(y=day.year, m=day.month - 1, d=day.day, h=hour)
    last = None
    for attempt in range(4):
        try:
            r = _session().get(u, timeout=30)
            if r.status_code != 200:
                raise RuntimeError(f"{u} -> HTTP {r.status_code}")
            p.write_bytes(r.content)
            return p
        except (requests.ConnectionError, requests.Timeout, RuntimeError) as exc:
            last = exc
            __import__("time").sleep(1.5 * (attempt + 1))
            if hasattr(_tls, "s"):
                _tls.s.close()
                del _tls.s
    raise RuntimeError(f"giving up on {u}: {last}")
 def decode(p: Path, day: date, hour: int) -> pd.DataFrame:
    blob = p.read_bytes()
    if not blob:
        return pd.DataFrame()
    raw = lzma.decompress(blob)
    n = len(raw) // 20
    a = np.frombuffer(raw[: n * 20], dtype=">u4").reshape(n, 5)
    f = np.frombuffer(raw[: n * 20], dtype=">f4").reshape(n, 5)
    df = pd.DataFrame({
        "ms": a[:, 0].astype(np.int64),
        "ask": a[:, 1] / SCALE,
        "bid": a[:, 2] / SCALE,
    })
    df["spread"] = df["ask"] - df["bid"]
    df["date"], df["hour"] = day.isoformat(), hour
    df["weekday"] = day.weekday()
    return df
 def load_all() -> pd.DataFrame:
    jobs = [(d, h) for d in DAYS for h in range(24)]
    with ThreadPoolExecutor(max_workers=3) as ex:
        paths = list(ex.map(lambda j: fetch(*j), jobs))
    frames = [decode(p, d, h) for p, (d, h) in zip(paths, jobs)]
    df = pd.concat([f for f in frames if not f.empty], ignore_index=True)
    return df
 def wstats(g: pd.DataFrame) -> pd.Series:
    """Tick stats + time-weighted mean (each tick's spread held until the
    next tick; last tick held to hour end)."""
    tw_num = tw_den = 0.0
    keys = [k for k in ("date", "hour") if k in g.columns]
    for _, grp in g.groupby(keys, sort=False):
        ms = grp["ms"].to_numpy()
        sp = grp["spread"].to_numpy()
        w = np.diff(np.append(ms, 3_600_000)).clip(min=0)
        tw_num += float((sp * w).sum())
        tw_den += float(w.sum())
    s = g["spread"]
    return pd.Series({
        "ticks": len(s),
        "tw_mean": tw_num / tw_den if tw_den else np.nan,
        "mean": s.mean(), "median": s.median(),
        "p75": s.quantile(0.75), "p90": s.quantile(0.90),
        "p99": s.quantile(0.99), "max": s.max(),
        "pct_le_030": (s <= 0.30).mean() * 100,
        "pct_ge_060": (s >= 0.60).mean() * 100,
    })
 def m5_boundary_spreads(df: pd.DataFrame) -> pd.DataFrame:
    """Spread of the FIRST tick at/after each M5 boundary — what a
    next-open fill actually pays."""
    rows = []
    for (day, hour), grp in df.groupby(["date", "hour"], sort=False):
        ms = grp["ms"].to_numpy()
        sp = grp["spread"].to_numpy()
        wd = grp["weekday"].iat[0]
        for b in range(0, 3_600_000, 300_000):
            i = np.searchsorted(ms, b)
            if i < len(ms) and ms[i] - b <= 30_000:   # tick within 30 s
                rows.append((day, wd, hour, b // 60_000, sp[i]))
    return pd.DataFrame(rows, columns=["date", "weekday", "hour", "minute", "spread"])
 def main() -> None:
    df = load_all()
    print(f"days={df['date'].nunique()}  ticks={len(df):,}  "
          f"price range {df['bid'].min():.0f}-{df['ask'].max():.0f}")
    print("\n── Overall (all sampled hours) ──")
    print(wstats(df).round(3).to_string())
    trad = df[df["hour"].isin(TRADABLE) & (df["weekday"] < 5)]
    print("\n── Bot-tradable hours (session 03-23 minus blocked, Mon-Fri) ──")
    print(wstats(trad).round(3).to_string())
    roll = df[df["hour"].isin([21, 22, 23])]
    print("\n── Rollover window 21-24 UTC ──")
    print(wstats(roll).round(3).to_string())
    print("\n── Per UTC hour ──")
    per_hour = df[df["weekday"] < 5].groupby("hour").apply(wstats, include_groups=False).round(3)
    per_hour["tradable"] = ["Y" if h in TRADABLE else "-" for h in per_hour.index]
    print(per_hour.to_string())
    per_hour.to_csv(Path(__file__).parent / "duka_spread_by_hour.csv")
    print("\n── M5 next-open boundary fills ──")
    mb = m5_boundary_spreads(df)
    mb_trad = mb[mb["hour"].isin(TRADABLE) & (mb["weekday"] < 5)]
    for name, d in [("all hours", mb), ("tradable hours", mb_trad)]:
        s = d["spread"]
        print(f"{name}: n={len(s):,} mean={s.mean():.3f} median={s.median():.3f} "
              f"p90={s.quantile(0.9):.3f} p99={s.quantile(0.99):.3f} "
              f"%<=0.30={(s <= 0.30).mean()*100:.1f} %>=0.60={(s >= 0.60).mean()*100:.1f}")
    mb.to_csv(Path(__file__).parent / "duka_m5_boundary_spreads.csv", index=False)
    print("\n── By month (tradable hours, tick median / tw_mean) ──")
    trad2 = trad.copy()
    trad2["month"] = trad2["date"].str[:7]
    bym = trad2.groupby("month").apply(wstats, include_groups=False).round(3)
    print(bym[["ticks", "tw_mean", "median", "p90", "pct_le_030", "pct_ge_060"]].to_string())
    print("\n── NFP Fridays 12:00-15:00 UTC vs same hours other days ──")
    nfp_days = {"2026-06-05", "2026-07-03"}
    win = df[df["hour"].isin([12, 13, 14])]
    for name, d in [("NFP", win[win["date"].isin(nfp_days)]),
                    ("non-NFP", win[~win["date"].isin(nfp_days) & (win["weekday"] < 5)])]:
        s = d["spread"]
        print(f"{name}: n={len(s):,} tick-median={s.median():.3f} p90={s.quantile(0.9):.3f} "
              f"p99={s.quantile(0.99):.3f} max={s.max():.2f}")
 if __name__ == "__main__":
    sys.exit(main())
@@ -0,0 +1,25 @@
 hour,ticks,tw_mean,mean,median,p75,p90,p99,max,pct_le_030,pct_ge_060,tradable
 0,362893.0,0.749,0.768,0.79,0.82,0.86,1.1,7.69,0.053,94.561,-
 1,540182.0,0.735,0.742,0.74,0.81,0.87,1.0,5.59,0.073,90.449,-
 2,431589.0,0.716,0.722,0.73,0.8,0.86,0.95,2.96,0.1,85.27,-
 3,343952.0,0.708,0.711,0.71,0.79,0.84,0.95,1.67,0.081,82.939,Y
 4,246262.0,0.704,0.702,0.71,0.78,0.82,0.94,1.86,0.098,82.627,Y
 5,388365.0,0.712,0.716,0.72,0.79,0.82,0.95,2.17,0.076,86.745,Y
 6,368999.0,0.65,0.651,0.65,0.69,0.74,0.94,2.32,0.079,71.668,Y
 7,313149.0,0.601,0.598,0.59,0.65,0.69,0.994,3.08,0.239,42.904,Y
 8,347276.0,0.612,0.616,0.6,0.66,0.74,1.0,3.04,0.308,48.349,Y
 9,321619.0,0.612,0.615,0.6,0.66,0.72,1.0,5.95,0.288,50.644,Y
 10,295271.0,0.611,0.618,0.6,0.66,0.71,1.0,9.36,0.164,49.253,-
 11,336963.0,0.619,0.624,0.6,0.67,0.79,1.0,4.97,0.203,50.178,-
 12,476773.0,0.66,0.697,0.63,0.72,0.877,1.367,14.98,0.257,59.633,Y
 13,651425.0,0.641,0.648,0.63,0.69,0.787,1.487,11.407,0.296,60.297,-
 14,673391.0,0.625,0.625,0.63,0.68,0.77,0.9,11.98,0.458,59.085,Y
 15,598398.0,0.636,0.638,0.64,0.7,0.78,0.91,4.49,0.433,66.296,Y
 16,485023.0,0.648,0.644,0.64,0.7,0.757,0.917,4.09,0.118,68.496,-
 17,450119.0,0.626,0.628,0.627,0.7,0.726,0.84,4.08,0.154,63.337,Y
 18,411538.0,0.626,0.63,0.637,0.7,0.71,0.79,4.16,0.107,67.734,Y
 19,435396.0,0.638,0.643,0.65,0.7,0.72,0.896,5.804,0.11,72.007,Y
 20,212654.0,0.684,0.665,0.66,0.7,0.76,1.217,7.15,0.226,70.215,Y
 21,31319.0,0.627,0.622,0.61,0.68,0.75,1.02,2.08,0.089,53.153,-
 22,108879.0,0.829,0.834,0.797,0.84,0.98,2.05,10.11,0.056,95.434,Y
 23,161079.0,1.032,0.913,0.8,0.89,1.307,2.77,13.174,0.092,95.41,-
@@ -0,0 +1,158 @@
 """Fetch XAU/USD M15 candles from the Oanda v20 REST API.
 Mechanical data-fetching infrastructure only — no analysis here.
 Paginates GET /v3/instruments/XAU_USD/candles from a start cursor
 through "now" and writes the result to a parquet file.
 Run from the repo root:
    venv/bin/python backtest/research/fetch_oanda_m15.py
 """
 import sys
 import time
 from datetime import datetime, timedelta, timezone
 from pathlib import Path
 import pandas as pd
 import requests
 sys.path.insert(0, str(Path(__file__).resolve().parents[2]))
 from app_config import oanda_api_token, oanda_base_url  # noqa: E402
 INSTRUMENT = "XAU_USD"
 GRANULARITY = "M15"
 START = datetime(2025, 7, 1, 0, 0, 0, tzinfo=timezone.utc)
 OUT_PATH = Path(__file__).parent / "oanda_xau_m15_20250701_now.parquet"
 COUNT = 5000
 REQUEST_SLEEP_S = 0.5
 FRESHNESS_WINDOW = timedelta(minutes=30)
 def _rfc3339(dt: datetime) -> str:
    return dt.astimezone(timezone.utc).strftime("%Y-%m-%dT%H:%M:%S.%fZ")
 def fetch_all_candles():
    token = oanda_api_token()
    base_url = oanda_base_url()
    if not token:
        raise RuntimeError(
            "No Oanda API token resolved (env OANDA_API_TOKEN / "
            "config.json oanda.api_token). Aborting.")
    url = f"{base_url}/v3/instruments/{INSTRUMENT}/candles"
    headers = {
        "Authorization": f"Bearer {token}",
        "Accept-Datetime-Format": "RFC3339",
    }
    cursor = START
    include_first = True
    all_rows = []
    n_requests = 0
    while True:
        params = {
            "granularity": GRANULARITY,
            "price": "MBA",
            "count": COUNT,
            "dailyAlignment": 0,
            "alignmentTimezone": "Etc/UTC",
            "from": _rfc3339(cursor),
            "includeFirst": "true" if include_first else "false",
        }
        resp = requests.get(url, headers=headers, params=params, timeout=30)
        n_requests += 1
        if resp.status_code != 200:
            raise RuntimeError(
                f"Oanda API request failed: status={resp.status_code}\n"
                f"URL: {resp.url}\n"
                f"Response body: {resp.text}")
        payload = resp.json()
        candles = payload.get("candles", [])
        for c in candles:
            if not c.get("complete", False):
                continue
            mid = c["mid"]
            bid = c["bid"]
            ask = c["ask"]
            all_rows.append({
                "Datetime": pd.Timestamp(c["time"]).tz_convert("UTC")
                if pd.Timestamp(c["time"]).tzinfo is not None
                else pd.Timestamp(c["time"]).tz_localize("UTC"),
                "Open": float(mid["o"]),
                "High": float(mid["h"]),
                "Low": float(mid["l"]),
                "Close": float(mid["c"]),
                "Volume": int(c["volume"]),
                "bid_o": float(bid["o"]),
                "bid_h": float(bid["h"]),
                "bid_l": float(bid["l"]),
                "bid_c": float(bid["c"]),
                "ask_o": float(ask["o"]),
                "ask_h": float(ask["h"]),
                "ask_l": float(ask["l"]),
                "ask_c": float(ask["c"]),
            })
        print(f"  request {n_requests}: from={params['from']} "
              f"got {len(candles)} candles (complete kept so far: "
              f"{len(all_rows)})", file=sys.stderr)
        if not candles:
            break
        last_time = pd.Timestamp(candles[-1]["time"])
        if last_time.tzinfo is None:
            last_time = last_time.tz_localize("UTC")
        else:
            last_time = last_time.tz_convert("UTC")
        now = datetime.now(timezone.utc)
        # Oanda's "count" includes the "from" boundary bar itself, so with
        # includeFirst=false every full page comes back as count-1, not
        # count (verified empirically: count=N, includeFirst=false always
        # yields N-1 candles when a full page of data exists ahead of the
        # cursor). Compare against the count actually expected for this
        # request's includeFirst setting, not the raw requested count.
        expected_full_page = COUNT if include_first else COUNT - 1
        got_fewer_than_requested = len(candles) < expected_full_page
        near_now = (now - last_time.to_pydatetime()) < FRESHNESS_WINDOW
        if got_fewer_than_requested or near_now:
            break
        cursor = last_time.to_pydatetime()
        include_first = False
        time.sleep(REQUEST_SLEEP_S)
    return all_rows, n_requests
 def main():
    print(f"Fetching {INSTRUMENT} {GRANULARITY} candles from "
          f"{_rfc3339(START)} through now...", file=sys.stderr)
    rows, n_requests = fetch_all_candles()
    if not rows:
        raise RuntimeError("No complete candles were fetched.")
    df = pd.DataFrame(rows)
    df = df.set_index("Datetime")
    df = df.sort_index()
    df = df[~df.index.duplicated(keep="last")]
    df.to_parquet(OUT_PATH)
    print(f"\nSaved {len(df)} rows to {OUT_PATH}", file=sys.stderr)
    print(f"Requests made: {n_requests}", file=sys.stderr)
    print(f"First timestamp: {df.index[0]}", file=sys.stderr)
    print(f"Last timestamp: {df.index[-1]}", file=sys.stderr)
 if __name__ == "__main__":
    main()
@@ -0,0 +1,144 @@
 """Forex Factory calendar scraper -> backtest/cache/calendar.parquet.
 Fetches weekly calendar pages and parses the event JSON embedded in
 `window.calendarComponentStates` (the `days:` array). Event `dateline`
 is a unix epoch, so timestamps carry no timezone ambiguity — verified
 2026-07-11 against known release times (NFP 12:30/13:30 UTC across DST,
 ECB 12:15/13:15) and against EUR/USD M15 volatility: the bar containing
 a high-impact event averages 3.5x normal range, and shifting event
 times by ±1h destroys that alignment.
 Usage:
    python backtest/research/ff_calendar_scraper.py --start 2024-01-01 --end 2026-07-13
    python backtest/research/ff_calendar_scraper.py            # refresh: last stored week -> today
 Merges into the existing parquet on event (id, dateline), so refreshing
 is idempotent and also picks up revised `actual` values for recent weeks.
 Be polite: ~1.6s between requests (a full year is ~52 requests).
 """
 import argparse
 import json
 import time
 from datetime import date, timedelta
 from pathlib import Path
 from typing import List, Optional
 import pandas as pd
 import requests
 CACHE_DIR = Path(__file__).resolve().parent.parent / "cache"
 PARQUET = CACHE_DIR / "calendar.parquet"
 UA = ("Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 "
      "(KHTML, like Gecko) Chrome/126.0 Safari/537.36")
 FIELDS = ["id", "dateline", "name", "country", "currency", "impactName",
          "timeLabel", "timeMasked", "actual", "forecast", "previous",
          "revision", "actualBetterWorse"]
 def week_url(d: date) -> str:
    return (f"https://www.forexfactory.com/calendar?week="
            f"{d.strftime('%b').lower()}{d.day}.{d.year}")
 def extract_days(html: str) -> Optional[list]:
    """Balanced-bracket extraction of the `days:` JSON array."""
    i = html.find("days:")
    if i < 0:
        return None
    j = html.find("[", i)
    depth, in_str, esc = 0, False, False
    for k in range(j, len(html)):
        ch = html[k]
        if in_str:
            if esc:
                esc = False
            elif ch == "\\":
                esc = True
            elif ch == '"':
                in_str = False
        else:
            if ch == '"':
                in_str = True
            elif ch == "[":
                depth += 1
            elif ch == "]":
                depth -= 1
                if depth == 0:
                    return json.loads(html[j:k + 1])
    return None
 def fetch_week(sess: requests.Session, d: date, retries: int = 4) -> Optional[List[dict]]:
    for attempt in range(retries):
        try:
            r = sess.get(week_url(d), timeout=30)
            if r.status_code == 200:
                days = extract_days(r.text)
                if days is not None:
                    return [{k: e.get(k) for k in FIELDS}
                            for day in days for e in day.get("events", [])]
        except requests.RequestException:
            pass
        time.sleep(5 * (attempt + 1))
    return None
 def scrape(start: date, end: date) -> pd.DataFrame:
    sess = requests.Session()
    sess.headers["User-Agent"] = UA
    rows: List[dict] = []
    failed: List[date] = []
    d = start
    while d <= end:
        week_rows = fetch_week(sess, d)
        if week_rows is None:
            failed.append(d)
            print(f"FAILED week {d}")
        else:
            rows.extend(week_rows)
        d += timedelta(days=7)
        time.sleep(1.6)
    if failed:
        print(f"WARNING: {len(failed)} weeks failed: {failed}")
    return pd.DataFrame(rows)
 def merge_into_store(new: pd.DataFrame, path: Path = PARQUET) -> pd.DataFrame:
    if path.exists():
        old = pd.read_parquet(path)
        combined = pd.concat([old, new])
    else:
        combined = new
    # keep="last": a refresh overwrites stale rows (actuals fill in
    # after release, revisions land weeks later).
    combined = (combined.drop_duplicates(subset=["id", "dateline"], keep="last")
                .sort_values("dateline").reset_index(drop=True))
    combined.to_parquet(path, index=False)
    return combined
 def main() -> None:
    ap = argparse.ArgumentParser(description=__doc__.splitlines()[0])
    ap.add_argument("--start", type=date.fromisoformat, default=None)
    ap.add_argument("--end", type=date.fromisoformat, default=date.today())
    args = ap.parse_args()
    start = args.start
    if start is None:
        if PARQUET.exists():
            last = pd.read_parquet(PARQUET).dateline.max()
            start = pd.Timestamp(last, unit="s").date() - timedelta(days=7)
        else:
            start = date(2024, 1, 1)
    print(f"Scraping {start} -> {args.end}")
    new = scrape(start, args.end)
    combined = merge_into_store(new)
    print(f"Store: {len(combined)} events "
          f"({pd.Timestamp(combined.dateline.min(), unit='s')} -> "
          f"{pd.Timestamp(combined.dateline.max(), unit='s')}) -> {PARQUET}")
 if __name__ == "__main__":
    main()
@@ -0,0 +1,123 @@
 # Fixed-RR hard-TP vs shipped trailing exit — research campaign 2026-08-05
 **Verdict: DO NOT SHIP. The trailing exit stays.** The fixed 2R target beats the trail on
 total R in every 2024+ window but the edge is a trend-regime artifact: it inverts in the
 2021–2023 ranging years, doubles drawdown, cuts win rate 10–19 points, and doubles hold
 times (amplifying the accepted live one-position divergence). Details below.
 ## Setup
 - Code: HEAD `fc5ce85` + the `--fixed-rr-tp` research flag (uncommitted at run time).
  The flag replaces every trade's TP with `mult × |entry − sl|` AFTER re-anchoring and
  disables trailing (run-level and per-trade PB params) — entries, stops, gates, and the
  cooldown pass are untouched, so both arms replay **trade-identical books** (verified:
  paired join on entry_time+direction matches 1:1 in every window). Only the exit policy
  differs → per-trade paired comparison.
 - Provider: Oanda (cache-warm; keys `oanda_XAU_USD_{5min,15min,1h}_{20250603_20260804,
  20240331_20250330,20210531_20260804}`). Charged cost `--spread 0.65` (real measured
  round trip); $0.30 letter-cost books derived analytically as `r + 0.35/risk` and
  verified exact against one real `--spread 0.30` run (max per-trade error 1.8e-15 —
  spread never enters trigger detection, per the owner-declined 2026-08-04 decision).
 - Control sanity gate: fresh M5 12mo control reproduced the pinned HEAD reference
  exactly — 555 trades, +98.64R, PF 1.483, maxDD −10.73R.
 - Commands (control = same without the flag):
  `venv/bin/python backtest/gpu_backtest.py --pair XAUUSD --primary-tf {5min,15min}
   --start 2025-08-04 --end 2026-08-04 --spread 0.65 [--fixed-rr-tp 2.0]
   --export-csv backtest/fixedtp_*.csv`
  OOS `--start 2024-06-01 --end 2025-03-30`; 5yr `--start 2021-08-01 --end 2026-08-04`.
 - Books pinned as `backtest/fixedtp_{m5,m15}_{12mo,oos,5yr}_{ctrl,rr20,...}.csv`
  (per-trade indicator breakdown, agreement counts, regime, vol-regime, weekday,
  MAE/MFE, bars-held — the full instructions_for_backtest record).
 ## Headline (@ $0.65)
 | Window | Arm | n | R | PF | WR | maxDD | recov | medBars | exits |
 |---|---|---|---|---|---|---|---|---|---|
 | M5 12mo | trail | 555 | +98.64 | 1.483 | 52.4% | −10.7 | 9.2 | 28 | TRAIL 361 / SL 183 / TO 11 |
 | M5 12mo | 2R hard | 555 | +124.85 | 1.378 | 42.9% | −24.5 | 5.1 | 50 | SL 312 / TP1 210 / TO 33 |
 | M15 12mo | trail | 397 | +73.93 | 1.525 | 62.2% | −10.8 | 6.8 | 28 | TRAIL 255 / SL 135 / TO 7 |
 | M15 12mo | 2R hard | 397 | +95.95 | 1.431 | 44.3% | −11.2 | 8.6 | 39 | SL 215 / TP1 151 / TO 31 |
 | M5 OOS | trail | 86 | +9.45 | 1.332 | 53.5% | −8.2 | 1.2 | 70 | — |
 | M5 OOS | 2R hard | 86 | +18.83 | 1.407 | 45.3% | −9.7 | 1.9 | 124 | — |
 | M15 OOS | trail | 299 | +38.77 | 1.360 | 59.2% | −15.5 | 2.5 | 38 | — |
 | M15 OOS | 2R hard | 299 | +72.14 | 1.435 | 45.2% | −13.9 | 5.2 | 82 | — |
 Halves (12mo, split 2026-02-03): M5 rr20 +67.99/+56.86 (both positive; H2 PF 1.325 vs
 ctrl 1.503), M15 rr20 +73.74/+22.21 (both positive; ctrl H2 +7.23 — the known H2-flat).
 At $0.30 all deltas keep the same sign (e.g. M5 12mo +109.0 vs +135.2).
 So the **letter** of rule 1 passes for rr20 on both TFs. The 5yr decomposition is why it
 still must not ship.
 ## Exit migration (paired, M5 12mo; same pattern everywhere)
 - SL→SL 183 trades: ΔR 0 (identical by construction — stops untouched).
 - TRAIL→TP1 207 trades: +248.9R → +424.5R (**+175.6R**) — winners the trail exits early.
 - TRAIL→SL 129 trades: +19.0R → −136.8R (**−155.7R**) — retraces the trail survives
  become full stop-outs.
 - TRAIL→TIMEOUT 25: +7.4R. Net = +26.2R. The whole effect is these two big flows nearly
  cancelling; which one wins is a regime property, not an exit-policy edge.
 - Trades with ctrl MFE ≥ 2R: 95; hard cap forfeits only +10.4R vs trail on M5 (the trail
  already gives ~0.75R back from peak, so a 2R cap costs little *in a trend*).
 ## Mini-grid (12mo @ $0.65) — the tell
 | mult | M5 R / PF / DD | M15 R / PF / DD |
 |---|---|---|
 | trail | +98.6 / 1.483 / −10.7 | +73.9 / 1.525 / −10.8 |
 | 1.0R | +52.5 / 1.205 / −12.8 | +67.1 / 1.410 / −7.3 |
 | 1.5R | +80.7 / 1.263 / −21.4 | +66.9 / 1.325 / −11.2 |
 | 2.0R | +124.9 / 1.378 / −24.5 | +96.0 / 1.431 / −11.2 |
 | 2.5R | +147.7 / 1.426 / −23.6 | +110.4 / 1.469 / −15.9 |
 | 3.0R | +155.8 / 1.432 / −21.6 | +135.8 / 1.562 / −17.5 |
 Monotone in the multiple on both TFs — "2.0" is not an optimum, it's a point on a curve
 that says *never take profit* (3R's M5 TIMEOUT bucket alone is +84.8R). That is the
 signature of a one-way trend sample, not of a superior exit rule. It also reconfirms the
 original finding that SHORT fixed targets (≤1.5R) are strictly bad (cf. RR-0.5 removal,
 `config.py:126-133`: +48R → +105R).
 ## 5yr regime decomposition (2021-08-01 → 2026-08-04 @ $0.65) — the verdict
 M15 (real sample in every year): ctrl +127.97 PF 1.209 DD −35.8; rr20 +158.97 PF 1.170
 DD **−59.8**; rr30 +204.03 PF 1.203 DD **−98.3**.
 | Year | n | trail | 2R hard | 3R hard |
 |---|---|---|---|---|
 | 2021 | 88 | **+4.00** | −16.09 | −17.41 |
 | 2022 | 284 | −9.54 | −7.63 | −26.74 |
 | 2023 | 227 | **+7.79** | −1.71 | −5.81 |
 | 2024 | 345 | +40.44 | +64.98 | +89.14 |
 | 2025 | 405 | +49.33 | +69.39 | +104.42 |
 | 2026 | 229 | +35.95 | +50.04 | +60.43 |
 M5 5yr mirrors it (ctrl +128.4 PF 1.387 DD −12.8 vs rr20 +160.2 PF 1.306 DD −24.7;
 2022: −1.1 vs −5.2), though pre-2024 M5 samples are thin (12–60 trades/yr).
 The hard target's entire outperformance lives in the 2024–2026 gold trend. In ranging
 years it flips sign exactly where the trail stays positive (M15 2021, 2023), and the
 further the target, the worse the ranging-year bleed and the deeper the DD. The 12mo,
 halves, and OOS windows all sit inside the trend era, which is why the rule-1 letter
 passes — this is precisely the within-regime-split trap rule 1's OOS clause exists to
 catch, one level up.
 ## Additional non-ship reasons
 - **Live divergence grows with hold time.** Median hold doubles (M5 28→50 bars; OOS
  70→124). The daemon holds one position total; the backtester doesn't model that
  blocking. Longer holds = more live signals skipped that the paper book banks.
 - **WR 52→43% (M5) / 62→44% (M15), max loss streak 6→11 (M15)** — materially harder to
  operate and size.
 - Aggregate tables (session/day/regime/vol, streaks, quarterly equity, rolling-20 WR)
  computed and consistent — no bucket reverses the conclusion; both arms' 12mo profit is
  concentrated in Q1-2026 (+87.7 ctrl / +102.0 rr20).
 ## Status
 - `--fixed-rr-tp` stays as a documented research-only flag (default off; conflicts with
  `--trail-arm/--trail-dist/--set signals.*trail*` are hard errors). Module 22 tests
  (5) pin the override math, TP fill + spread, SL-first tie, PB-trail-disable, and CLI
  guards. Full suite: 346 green.
 - Re-open only if gold's regime turns and a *regime-conditional* exit (trail in range,
  target in trend) is proposed with a pre-registered mechanism — a plain fixed ratio is
  now a closed hypothesis at every multiple in {0.5, 1.0, 1.5, 2.0, 2.5, 3.0}.
@@ -0,0 +1,40 @@
 """Per-hour P&L breakdown of a backtest trade book.
 Usage:
    python backtest/research/hour_analysis.py BOOK.csv BAR_MINUTES
 Reports, per UTC hour (both entry hour and DECISION hour = entry_time
 minus one bar, the clock blocked_hours_utc evaluates): trade count,
 total R, mean R, win rate, profit factor. Also a per-(hour, direction)
 table. Analysis tooling only — never imported by the live code.
 """
 import sys
 import pandas as pd
 def hour_tables(book_csv: str, bar_minutes: int):
    df = pd.read_csv(book_csv)
    et = pd.to_datetime(df["entry_time"], utc=True, format="mixed")
    df["entry_hour"] = et.dt.hour
    df["decision_hour"] = (et - pd.Timedelta(minutes=bar_minutes)).dt.hour
    def table(group_cols):
        g = df.groupby(group_cols)["r_multiple"]
        out = g.agg(count="count", total_r="sum", mean_r="mean",
                    wr=lambda r: (r > 0).mean() * 100)
        out["pf"] = g.apply(
            lambda r: r[r > 0].sum() / max(-r[r <= 0].sum(), 1e-9))
        return out.round(2)
    return (table("decision_hour"), table(["decision_hour", "direction"]),
            table("entry_hour"))
 if __name__ == "__main__":
    book, mins = sys.argv[1], int(sys.argv[2])
    by_dec, by_dec_dir, by_ent = hour_tables(book, mins)
    print(f"\n── {book} · by DECISION hour (bar-close clock) ──")
    print(by_dec.to_string())
    print(f"\n── by (decision hour, direction) ──")
    print(by_dec_dir.to_string())
@@ -0,0 +1,697 @@
 """
 DeepSeek news-analysis validation harness — XAU/USD.
 ====================================================
 Research-only (no production code changes). Answers: is the DeepSeek
 layer's news analysis *correct*?
 Part A (events): replays every high-impact USD release with a genuine
 surprise (2024-01 → 2026-07, from backtest/cache/calendar.parquet)
 through the REAL `NewsEngine._llm_event_direction` — real prompt, real
 payload, real validation — and scores its "gold bullish/bearish" calls
 against gold's actual move (Oanda M15 mids) at +15m/30m/1h/4h/24h.
 Baselines: mechanical surprise-sign doctrine, FF actualBetterWorse,
 always-bullish/bearish (2024-26 is a gold bull regime), coin flip.
 Part B (headlines): a curated golden set (llm_news_golden.csv) through
 the REAL `NewsEngine._llm_headline_scores` in production-sized batches,
 vs the keyword scorer it overrides. Includes repeat-batch (determinism)
 and shuffled-order (cross-contamination) probes.
 Implementation notes that keep the replay faithful to production:
  * `_llm_event_direction`/`_llm_headline_scores` resolve the client via
    `get_llm_client()`, so we seed the module singleton with a research
    client (raised daily cap: 586 events > production cap 500).
  * The event cache key omits the forecast, so a repeat `actual` would
    silently re-serve a direction computed for a different forecast —
    the cache is cleared before every call.
  * A "neutral" reply and a hard failure both return None; they are
    disambiguated by reading the event cache back after the call
    (absent = failure, "" = neutral, "bullish:/bearish: …" = call).
  * Raw request/response bytes are captured with a requests response
    hook — the request-building code path is untouched.
 Usage (from repo root):
  venv/bin/python backtest/research/llm_news_eval.py events --limit 5
  venv/bin/python backtest/research/llm_news_eval.py events --determinism 20
  venv/bin/python backtest/research/llm_news_eval.py headlines
  venv/bin/python backtest/research/llm_news_eval.py report
 Artifacts land in backtest/research/llm_news_eval_2026-08/ (JSONL replay
 logs are resume-safe: completed events are never re-paid). `report` is a
 pure offline function of the artifacts.
 """
 import argparse
 import json
 import math
 import random
 import sys
 import time
 from datetime import datetime, timezone
 from pathlib import Path
 import numpy as np
 import pandas as pd
 REPO_ROOT = Path(__file__).resolve().parents[2]
 sys.path.insert(0, str(REPO_ROOT))
 import llm_client
 from app_config import deepseek_api_key, get_app_config
 from config import NewsConfig
 from llm_client import DEFAULT_MODEL, DeepSeekClient
 from news_engine import EconomicEvent, NewsEngine
 RESEARCH_DIR = Path(__file__).parent
 ART_DIR = RESEARCH_DIR / "llm_news_eval_2026-08"
 CALENDAR = REPO_ROOT / "backtest" / "cache" / "calendar.parquet"
 BARS = (REPO_ROOT / "backtest" / "cache" /
        "oanda_XAU_USD_15min_20210531_20260804.parquet")
 GOLDEN = RESEARCH_DIR / "llm_news_golden.csv"
 EVENTS_JSONL = ART_DIR / "events_replay.jsonl"
 HEADLINES_JSONL = ART_DIR / "headlines_replay.jsonl"
 SCORED_CSV = ART_DIR / "events_scored.csv"
 HORIZONS = [("15m", 15), ("30m", 30), ("1h", 60), ("4h", 240),
            ("24h", 1440)]
 NEUTRAL_BAND = 0.15  # |xau| below this = neutral prediction (Part B)
 BATCH_CAP = 25       # production batch cap (news_engine.py:783)
 EXPECTED_CORPUS = 586
 # ── shared plumbing ──────────────────────────────────────────────────
 class _Recorder:
    """requests response hook capturing the exact wire traffic."""
    def __init__(self):
        self.last = None
    def hook(self, resp, **kwargs):
        body = resp.request.body
        if isinstance(body, bytes):
            body = body.decode("utf-8", "replace")
        self.last = {
            "request_body": body,
            "http_status": resp.status_code,
            "response_body": resp.text,
            "elapsed_s": round(resp.elapsed.total_seconds(), 3),
        }
        return resp
    def pop(self):
        cap, self.last = self.last, None
        return cap or {}
 def install_research_client(timeout: float):
    """Build a research DeepSeekClient (raised cap) and seed the module
    singleton so the production code paths use it. Returns (client,
    recorder)."""
    app = get_app_config()
    key = deepseek_api_key(app)
    ds = app.deepseek_settings
    if not key:
        sys.exit("No DeepSeek API key (env DEEPSEEK_API_KEY or "
                 "config.json deepseek.api_key) — aborting before spend.")
    if ds.get("enabled") is False:
        sys.exit("deepseek.enabled is false in config.json — aborting.")
    client = DeepSeekClient(
        api_key=key,
        model=str(ds.get("model", DEFAULT_MODEL)),
        timeout=timeout,
        daily_cap=10 ** 6,  # research: corpus 586 > production cap 500
    )
    rec = _Recorder()
    client._session.hooks["response"].append(rec.hook)
    with llm_client._client_lock:
        llm_client._client = client
    return client, rec
 def _s(x):
    """Parquet str columns carry float NaN — EconomicEvent needs None."""
    return None if pd.isna(x) else str(x)
 def load_corpus():
    """High-impact USD releases with a genuine surprise, through the
    production `EconomicEvent.surprise` gate. Asserts run before any
    API spend."""
    df = pd.read_parquet(CALENDAR)
    df = df[(df["impactName"] == "high") & (df["currency"] == "USD")]
    rows = []
    for r in df.itertuples():
        ev = EconomicEvent(
            time=datetime.fromtimestamp(int(r.dateline), tz=timezone.utc),
            currency="USD", impact="high", event=_s(r.name) or "",
            forecast=_s(r.forecast), previous=_s(r.previous),
            actual=_s(r.actual),
        )
        if ev.surprise not in ("positive", "negative"):
            continue
        rows.append({
            "ffid": int(r.id), "dateline": int(r.dateline),
            "utc_time": ev.time.isoformat(), "name": ev.event,
            "actual": ev.actual, "forecast": ev.forecast,
            "previous": ev.previous, "surprise": ev.surprise,
            "better_worse": int(r.actualBetterWorse), "ev": ev,
        })
    ids = [d["ffid"] for d in rows]
    if len(rows) != EXPECTED_CORPUS:
        sys.exit(f"corpus size {len(rows)} != {EXPECTED_CORPUS} — "
                 "calendar.parquet changed; re-verify before spending.")
    if len(set(ids)) != len(ids):
        sys.exit("FF ids not unique — resume keying would break.")
    # Pin the actualBetterWorse mapping on two unambiguous NFP prints.
    nfp = {d["utc_time"][:10]: d["better_worse"] for d in rows
           if d["name"] == "Non-Farm Employment Change"}
    if nfp.get("2024-01-05") != 1 or nfp.get("2024-05-03") != 2:
        sys.exit("actualBetterWorse mapping pins failed (expected "
                 "1=better-for-USD on 2024-01-05 beat, 2=worse on "
                 "2024-05-03 miss).")
    rows.sort(key=lambda d: (d["dateline"], d["ffid"]))
    return rows
 def _load_jsonl(path: Path):
    recs = []
    if path.exists():
        with open(path) as f:
            for line in f:
                line = line.strip()
                if line:
                    recs.append(json.loads(line))
    return recs
 # ── Part A: event replay ─────────────────────────────────────────────
 def _extract_reason(cap):
    try:
        content = json.loads(cap["response_body"])[
            "choices"][0]["message"]["content"]
        return json.loads(content).get("reason")
    except Exception:
        return None
 def cmd_events(args):
    ART_DIR.mkdir(exist_ok=True)
    corpus = load_corpus()
    if args.limit:
        corpus = corpus[:args.limit]
    engine = NewsEngine(NewsConfig())
    existing = _load_jsonl(EVENTS_JSONL)
    done1 = {r["ffid"] for r in existing
             if r.get("run_pass") == 1 and r.get("llm_direction")}
    done2 = {r["ffid"] for r in existing
             if r.get("run_pass") == 2 and r.get("llm_direction")}
    pending = [d for d in corpus if d["ffid"] not in done1]
    print(f"corpus {len(corpus)} | already recorded {len(done1)} | "
          f"pending {len(pending)} | determinism re-calls "
          f"{args.determinism} | model calls are ~100 tokens each "
          f"(deepseek-v4-flash, static-prefix cached) → cost ≈ cents")
    if not pending and not args.determinism:
        print("nothing to do")
        return
    _, rec = install_research_client(args.timeout)
    out = open(EVENTS_JSONL, "a")
    def call_one(d, run_pass):
        # Cache key omits forecast → cleared per call so 196 colliding
        # corpus rows can't be served a stale direction.
        with NewsEngine._cache_lock:
            NewsEngine._llm_event_cache.clear()
        ev = d["ev"]
        engine._llm_event_direction(ev)
        key = (ev.event, ev.currency, str(ev.actual or ""))
        with NewsEngine._cache_lock:
            cached = NewsEngine._llm_event_cache.get(key)
        cap = rec.pop()
        if cached is None:
            direction, text = None, None       # hard failure → retry
        elif cached == "":
            direction, text = "neutral", ""
        else:
            direction, text = cached.split(":", 1)[0], cached
        recd = {k: d[k] for k in (
            "ffid", "dateline", "utc_time", "name", "actual", "forecast",
            "previous", "surprise", "better_worse")}
        recd.update({
            "llm_direction": direction, "llm_text": text,
            "reason_raw": _extract_reason(cap), "run_pass": run_pass,
            "ts": datetime.now(timezone.utc).isoformat(), **cap,
        })
        out.write(json.dumps(recd) + "\n")
        out.flush()
        return direction
    remaining = pending
    for npass in range(1, args.retry_passes + 1):
        if not remaining:
            break
        failed = []
        for i, d in enumerate(remaining):
            if call_one(d, 1) is None:
                failed.append(d)
            if args.sleep:
                time.sleep(args.sleep)
            if (i + 1) % 25 == 0:
                print(f"  pass {npass}: {i + 1}/{len(remaining)} "
                      f"({len(failed)} failed)")
        print(f"pass {npass} done: {len(remaining) - len(failed)} ok, "
              f"{len(failed)} failed")
        remaining = failed
    if remaining:
        print(f"WARNING: {len(remaining)} events still unrecorded after "
              f"{args.retry_passes} passes — re-run `events` to resume.")
    if args.determinism:
        ok_ids = {x["ffid"] for x in remaining}
        done_now = [d for d in corpus if d["ffid"] not in ok_ids]
        done_now.sort(key=lambda d: (d["name"], d["dateline"]))
        stride = max(1, len(done_now) // args.determinism)
        sample = done_now[::stride][:args.determinism]
        todo = [d for d in sample if d["ffid"] not in done2]
        print(f"determinism probe: {len(todo)} re-calls "
              f"({len(sample) - len(todo)} already recorded)")
        for d in todo:
            call_one(d, 2)
            if args.sleep:
                time.sleep(args.sleep)
    out.close()
    print("events replay complete →", EVENTS_JSONL)
 # ── Part B: golden headlines ─────────────────────────────────────────
 def _load_golden():
    g = pd.read_csv(GOLDEN)
    need = {"headline", "category", "expected_xau_sign", "rationale"}
    if not need <= set(g.columns):
        sys.exit(f"golden CSV missing columns {need - set(g.columns)}")
    if not g["expected_xau_sign"].isin([-1, 0, 1]).all():
        sys.exit("expected_xau_sign must be -1/0/+1")
    if not g["headline"].is_unique:
        sys.exit("duplicate headlines in golden CSV")
    return g
 def cmd_headlines(args):
    ART_DIR.mkdir(exist_ok=True)
    g = _load_golden()
    heads = list(g["headline"])
    _, rec = install_research_client(args.timeout)
    engine = NewsEngine(NewsConfig())
    out = open(HEADLINES_JSONL, "w")  # tiny (~9 calls): fresh each run
    def run_batch(texts, tag, batch_idx):
        scores = None
        for attempt in range(3):
            scores = engine._llm_headline_scores(texts)
            cap = rec.pop()
            if scores is not None:
                break
            print(f"  {tag} batch {batch_idx} attempt {attempt + 1} "
                  f"failed (http {cap.get('http_status')})")
            time.sleep(2.0)
        out.write(json.dumps({
            "run_tag": tag, "batch": batch_idx, "headlines": texts,
            "scores": scores,
            "ts": datetime.now(timezone.utc).isoformat(), **cap,
        }) + "\n")
        out.flush()
        return scores
    batches = [heads[i:i + BATCH_CAP] for i in range(0, len(heads),
                                                     BATCH_CAP)]
    for bi, b in enumerate(batches):
        ok = run_batch(b, "main", bi) is not None
        print(f"main batch {bi} ({len(b)} headlines): "
              f"{'ok' if ok else 'FAILED'}")
        time.sleep(args.sleep)
    run_batch(batches[0], "repeat", 0)
    print("repeat probe done")
    shuffled = heads[:]
    random.Random(20260806).shuffle(shuffled)
    for bi in range(0, len(shuffled), BATCH_CAP):
        run_batch(shuffled[bi:bi + BATCH_CAP], "shuffled",
                  bi // BATCH_CAP)
    print("shuffled probe done →", HEADLINES_JSONL)
    out.close()
 # ── report: pure offline function of the artifacts ───────────────────
 def _binom_p(k, n):
    """Exact two-sided binomial p-value vs p=0.5 (no scipy)."""
    if n == 0:
        return float("nan")
    dist = [math.comb(n, i) for i in range(n + 1)]
    tot = float(2 ** n)
    pk = dist[k]
    return min(1.0, sum(c for c in dist if c <= pk) / tot)
 def _mcnemar_p(hit_a, hit_b):
    """Exact McNemar via binomial on discordant pairs. Inputs: aligned
    boolean lists."""
    b = sum(1 for x, y in zip(hit_a, hit_b) if x and not y)
    c = sum(1 for x, y in zip(hit_a, hit_b) if not x and y)
    return b, c, _binom_p(min(b, c), b + c)
 def _latest_events(recs):
    """Last pass-1 record per ffid (retries append; last wins)."""
    out = {}
    for r in recs:
        if r.get("run_pass") == 1:
            out[r["ffid"]] = r
    return list(out.values())
 def _fmt_rate(hits, n, label=""):
    if n == 0:
        return f"{label:<24} n=0"
    p = _binom_p(hits, n)
    return (f"{label:<24} {hits:>4}/{n:<4} = {hits / n * 100:5.1f}%  "
            f"(binom p={p:.3f})")
 def report_events():
    recs = _latest_events(_load_jsonl(EVENTS_JSONL))
    if not recs:
        print("no event replay records — run `events` first")
        return
    bars = pd.read_parquet(BARS)
    closes = bars["Close"].to_numpy()
    close_ns = (bars.index + pd.Timedelta(minutes=15)).asi8
    def anchor(t_ns, h_min):
        i = np.searchsorted(close_ns, t_ns, side="right") - 1
        if i < 0:
            return None
        stale_ns = max(45.0, h_min / 4.0) * 60e9
        if t_ns - close_ns[i] > stale_ns:
            return None
        return closes[i]
    rows, skip = [], {h: 0 for h, _ in HORIZONS}
    base_skipped = 0
    for r in recs:
        t_ns = int(r["dateline"]) * 10 ** 9
        base = anchor(t_ns, 0)
        if base is None:
            base_skipped += 1
            continue
        row = dict(r)
        row.pop("request_body", None)
        row.pop("response_body", None)
        row["year"] = r["utc_time"][:4]
        row["base"] = base
        for hname, hmin in HORIZONS:
            p = anchor(t_ns + hmin * 60 * 10 ** 9, hmin)
            if p is None:
                skip[hname] += 1
                row[f"d_{hname}"] = np.nan
            else:
                row[f"d_{hname}"] = p - base
        rows.append(row)
    df = pd.DataFrame(rows)
    df.to_csv(SCORED_CSV, index=False)
    n_fail = int((df["llm_direction"].isna()).sum())
    print("═" * 70)
    print("PART A — event gold-direction vs actual XAU move")
    print("═" * 70)
    print(f"events replayed: {len(df)} (baseline-price skipped: "
          f"{base_skipped}) | LLM hard failures: {n_fail}")
    print("horizon skip counts (stale/closed):",
          {k: v for k, v in skip.items() if v})
    dist = df["llm_direction"].value_counts(dropna=False).to_dict()
    print("LLM call distribution:", dist)
    # Unconditional per-horizon noise scale (null): median |Δ| over all
    # 2024+ bar times, position-shifted.
    bars24 = bars[bars.index >= "2024-01-01"]
    null_med = {}
    for hname, hmin in HORIZONS:
        k = hmin // 15
        d = (bars24["Close"].shift(-k) - bars24["Close"]).abs().dropna()
        null_med[hname] = float(d.median())
    print("unconditional null median |Δ| ($):",
          {k: round(v, 2) for k, v in null_med.items()})
    # Strategy call columns: +1 gold up, −1 gold down, 0/NaN excluded.
    df["c_llm"] = df["llm_direction"].map(
        {"bullish": 1, "bearish": -1}).fillna(0)
    df["c_doctrine"] = df["surprise"].map({"positive": -1, "negative": 1})
    df["c_ffbw"] = df["better_worse"].map({1: -1, 2: 1}).fillna(0)
    df["c_bull"] = 1
    df["c_bear"] = -1
    strategies = [("LLM", "c_llm"), ("doctrine(surprise)", "c_doctrine"),
                  ("FF betterWorse", "c_ffbw"), ("always-bullish", "c_bull"),
                  ("always-bearish", "c_bear")]
    for hname, _ in HORIZONS:
        dcol = f"d_{hname}"
        sub = df[df[dcol].notna()]
        print(f"\n── horizon {hname} (n scored bars={len(sub)}) ──")
        zeros = int((sub[dcol] == 0).sum())
        if zeros:
            print(f"  Δ==0 count (scored as miss): {zeros}")
        for label, col in strategies:
            s = sub[sub[col] != 0]
            hits = int((np.sign(s[dcol]) == s[col]).sum())
            print("  " + _fmt_rate(hits, len(s), label))
        # noise-filtered LLM hit rate
        s = sub[(sub["c_llm"] != 0) &
                (sub[dcol].abs() >= null_med[hname])]
        hits = int((np.sign(s[dcol]) == s["c_llm"]).sum())
        print("  " + _fmt_rate(hits, len(s),
                               f"LLM |Δ|≥null-med"))
        # event-window magnitude vs null
        mag = sub[dcol].abs().median()
        print(f"  event-window median |Δ| = ${mag:.2f} "
              f"(null ${null_med[hname]:.2f})")
        # paired McNemar: LLM vs each baseline on common directional set
        for label, col in [("doctrine", "c_doctrine"),
                           ("FF betterWorse", "c_ffbw"),
                           ("always-bullish", "c_bull")]:
            s = sub[(sub["c_llm"] != 0) & (sub[col] != 0)]
            ha = list(np.sign(s[dcol]) == s["c_llm"])
            hb = list(np.sign(s[dcol]) == s[col])
            b, c, p = _mcnemar_p(ha, hb)
            print(f"  McNemar LLM vs {label:<15} b={b:>3} c={c:>3} "
                  f"p={p:.3f}")
    # neutral-call flatness (15m)
    neu = df[(df["llm_direction"] == "neutral") & df["d_15m"].notna()]
    dire = df[(df["c_llm"] != 0) & df["d_15m"].notna()]
    if len(neu):
        print(f"\nneutral calls (n={len(neu)}): median |Δ_15m| "
              f"${neu['d_15m'].abs().median():.2f} vs directional "
              f"(n={len(dire)}) ${dire['d_15m'].abs().median():.2f} "
              f"(null ${null_med['15m']:.2f})")
    # breakdowns at 1h
    dcol = "d_1h"
    sub = df[df[dcol].notna() & (df["c_llm"] != 0)]
    print("\n── LLM hit rate by event name (1h, top 10 by count) ──")
    for name, grp in sorted(sub.groupby("name"), key=lambda kv: -len(kv[1]))[:10]:
        hits = int((np.sign(grp[dcol]) == grp["c_llm"]).sum())
        print("  " + _fmt_rate(hits, len(grp), name[:24]))
    print("── LLM hit rate by year (1h) ──")
    for yr, grp in sub.groupby("year"):
        hits = int((np.sign(grp[dcol]) == grp["c_llm"]).sum())
        print("  " + _fmt_rate(hits, len(grp), yr))
    # cluster view: shared datelines
    print("\n── same-timestamp clusters ──")
    clus = df.groupby("dateline").filter(lambda g: len(g) > 1)
    groups = list(clus.groupby("dateline"))
    consistent = 0
    net_hits, net_n = 0, 0
    for _, g in groups:
        signs = set(g.loc[g["c_llm"] != 0, "c_llm"])
        if len(signs) <= 1:
            consistent += 1
        net = int(g["c_llm"].sum())
        d = g["d_1h"].iloc[0]
        if net != 0 and pd.notna(d) and d != 0:
            net_n += 1
            if np.sign(d) == np.sign(net):
                net_hits += 1
    print(f"clusters: {len(groups)} datelines "
          f"({len(clus)} events, {len(clus) / max(len(df), 1) * 100:.0f}% of "
          f"corpus) | internally consistent: {consistent}/{len(groups)}")
    print("  " + _fmt_rate(net_hits, net_n, "cluster net-call (1h)"))
    # determinism probe
    p2 = {r["ffid"]: r for r in _load_jsonl(EVENTS_JSONL)
          if r.get("run_pass") == 2 and r.get("llm_direction")}
    if p2:
        p1 = {r["ffid"]: r for r in recs}
        agree = same_reason = n = 0
        for fid, r2 in p2.items():
            r1 = p1.get(fid)
            if not r1 or not r1.get("llm_direction"):
                continue
            n += 1
            if r1["llm_direction"] == r2["llm_direction"]:
                agree += 1
            if r1.get("reason_raw") == r2.get("reason_raw"):
                same_reason += 1
        print(f"\ndeterminism probe: direction agreement {agree}/{n}, "
              f"identical reason text {same_reason}/{n}")
    # stratified reason examples
    print("\n── reason samples (1h hit/miss) ──")
    sub = df[df[dcol].notna() & (df["c_llm"] != 0)].copy()
    sub["hit"] = np.sign(sub[dcol]) == sub["c_llm"]
    for verdict in (True, False):
        picked, seen = [], set()
        for _, r in sub[sub["hit"] == verdict].iterrows():
            if r["name"] in seen:
                continue
            seen.add(r["name"])
            picked.append(r)
            if len(picked) == 10:
                break
        tag = "HIT " if verdict else "MISS"
        for r in picked:
            print(f"  [{tag}] {r['utc_time'][:16]} {r['name'][:30]:<30} "
                  f"a={r['actual']} f={r['forecast']} → "
                  f"{r['llm_direction']:<7} Δ1h={r[dcol]:+7.2f}  "
                  f"“{(r['reason_raw'] or '')[:60]}”")
    print(f"\nscored table → {SCORED_CSV}")
 def report_headlines():
    recs = _load_jsonl(HEADLINES_JSONL)
    if not recs:
        print("\n(no headline replay records — run `headlines` first)")
        return
    g = _load_golden()
    engine = NewsEngine(NewsConfig())
    def collect(tag):
        m = {}
        for r in recs:
            if r["run_tag"] == tag and r["scores"]:
                for h, sc in zip(r["headlines"], r["scores"]):
                    m[h] = tuple(sc)
        return m
    main, shuf = collect("main"), collect("shuffled")
    print("\n" + "═" * 70)
    print("PART B — golden headline set: LLM vs keyword baseline")
    print("═" * 70)
    missing = [h for h in g["headline"] if h not in main]
    if missing:
        print(f"WARNING: {len(missing)} headlines missing from main run")
    def cls(x):
        if x >= NEUTRAL_BAND:
            return 1
        if x <= -NEUTRAL_BAND:
            return -1
        return 0
    rows = []
    for _, r in g.iterrows():
        h = r["headline"]
        if h not in main:
            continue
        llm_xau = main[h][1]
        kw = engine._analyze_headline_sentiment(h, for_currency="XAU")
        rows.append({
            "headline": h, "category": r["category"],
            "exp": int(r["expected_xau_sign"]),
            "llm_xau": llm_xau, "llm_cls": cls(llm_xau),
            "kw": kw, "kw_cls": cls(kw),
        })
    t = pd.DataFrame(rows)
    for who in ("llm", "kw"):
        acc = (t[f"{who}_cls"] == t["exp"]).mean()
        print(f"{who.upper():<4} 3-class accuracy (band ±{NEUTRAL_BAND}): "
              f"{(t[f'{who}_cls'] == t['exp']).sum()}/{len(t)} "
              f"= {acc * 100:.1f}%")
    ha = list(t["llm_cls"] == t["exp"])
    hb = list(t["kw_cls"] == t["exp"])
    b, c, p = _mcnemar_p(ha, hb)
    print(f"paired McNemar LLM vs keyword: b={b} c={c} p={p:.3f}")
    print("\nper-category accuracy (LLM | keyword):")
    for cat, grp in t.groupby("category"):
        a1 = (grp["llm_cls"] == grp["exp"]).sum()
        a2 = (grp["kw_cls"] == grp["exp"]).sum()
        print(f"  {cat:<18} {a1:>2}/{len(grp):<2} | {a2:>2}/{len(grp)}")
    print("\nLLM confusion (rows=expected, cols=predicted -1/0/+1):")
    cm = pd.crosstab(t["exp"], t["llm_cls"]).reindex(
        index=[-1, 0, 1], columns=[-1, 0, 1], fill_value=0)
    print(cm.to_string())
    print("\nLLM misclassified headlines:")
    for _, r in t[t["llm_cls"] != t["exp"]].iterrows():
        print(f"  exp {r['exp']:+d} got {r['llm_xau']:+.2f} "
              f"[{r['category']}] {r['headline'][:70]}")
    # repeat-batch determinism
    m_b0 = [r for r in recs if r["run_tag"] == "main" and r["batch"] == 0]
    r_b0 = [r for r in recs if r["run_tag"] == "repeat"]
    if m_b0 and r_b0 and m_b0[0]["scores"] and r_b0[0]["scores"]:
        a = np.array(m_b0[0]["scores"], dtype=float)
        b_ = np.array(r_b0[0]["scores"], dtype=float)
        exact = int((a == b_).all(axis=1).sum())
        print(f"\nrepeat-batch: {exact}/{len(a)} score pairs identical, "
              f"max |diff| = {np.abs(a - b_).max():.3f}")
    # shuffled-order drift
    if shuf:
        both = [h for h in t["headline"] if h in shuf]
        drift = [abs(main[h][1] - shuf[h][1]) for h in both]
        if drift:
            print(f"shuffled-order xau drift over {len(both)} headlines: "
                  f"mean {np.mean(drift):.3f}, max {np.max(drift):.3f}")
 def cmd_report(args):
    report_events()
    report_headlines()
 # ── CLI ──────────────────────────────────────────────────────────────
 def main():
    ap = argparse.ArgumentParser(description=__doc__.split("\n")[1])
    sub = ap.add_subparsers(dest="cmd", required=True)
    e = sub.add_parser("events", help="replay release surprises")
    e.add_argument("--limit", type=int, default=0,
                   help="smoke-test on first N events")
    e.add_argument("--determinism", type=int, default=0,
                   help="re-call N stratified events a second time")
    e.add_argument("--sleep", type=float, default=0.3)
    e.add_argument("--retry-passes", type=int, default=3)
    e.add_argument("--timeout", type=float, default=30.0)
    e.set_defaults(fn=cmd_events)
    h = sub.add_parser("headlines", help="golden-set probes")
    h.add_argument("--sleep", type=float, default=0.3)
    h.add_argument("--timeout", type=float, default=30.0)
    h.set_defaults(fn=cmd_headlines)
    r = sub.add_parser("report", help="offline scoring report")
    r.set_defaults(fn=cmd_report)
    args = ap.parse_args()
    args.fn(args)
 if __name__ == "__main__":
    main()
@@ -0,0 +1,86 @@
 # DeepSeek news-analysis validation — 2026-08-06
 **Verdict: headline sentiment scoring is excellent; event-direction calls are
 directionally sound but ~11% of them misread the numeric comparison, and the
 failure concentrates on Unemployment Claims / close K-M-suffixed values.**
 Method (owner-directed, token-lean): independent expert re-analysis of a
 stratified sample of DeepSeek's recorded answers, not a full statistical
 backtest. Harness: `backtest/research/llm_news_eval.py` (replays releases
 through the REAL `_llm_event_direction` / `_llm_headline_scores` — real
 prompts, payloads, validation; raw wire traffic logged). 135 of the 586-event
 corpus were replayed before the full run was stopped by owner request
 (artifacts retained in `llm_news_eval_2026-08/`, resume-safe).
 ## Part A — event gold-direction (28-event stratified sample, Jan–Jun 2024)
 Reviewer agreement with DeepSeek: **24/28 (86%)**. All 4 disagreements are
 DeepSeek errors, in exactly two failure modes:
 1. **Numeric comparison misread (3/28 ≈ 11%)** — the doctrine logic is right
   but the premise is backwards:
   - JOLTS 8.79M vs f 8.84M (a miss) → DS: "Job openings **above** forecast" → bearish (should be bullish)
   - Claims 202K vs f 217K (fewer claims) → DS: "Claims **higher** than forecast" → bullish (should be bearish)
   - Claims 202K vs f 209K — same backwards read, same wrong call
   Both Unemployment Claims events in the sample were misread the same way —
   looks systematic on that event type, possibly an anchoring prior that
   "claims rose". All misreads involve close values with K/M suffixes.
 2. **Direction field contradicts its own reason (1/28)** — Empire State
   −43.7 vs f −4.9: reason text says "gold may rise", `gold_direction` says
   `bearish`.
 Where the numbers were read correctly, hawkish/dovish→gold mapping was
 consistently correct (CPI/PPI/NFP/ADP/GDP/PMI/wages/unemployment-rate all
 handled properly, including inverted metrics when read right).
 Price cross-check (1h horizon, cluster-confounded, indicative only):
 reviewer calls 22/28 with the subsequent move, DeepSeek 20/28.
 ## Part B — headline sentiment (78-headline golden set)
 DeepSeek matched the reviewer's expected gold-direction label on
 **78/78 (100%)** — including all 8 negation traps ("signals no rate hike",
 "no rate cuts until…") and all 6 inverted-metric phrasings ("claims hit
 two-year high"). The keyword scorer it overrides got **24/78 (31%)**.
 Repeat-batch and shuffled-order probes recorded (headlines_replay.jsonl).
 ## Implications
 - The headline-sentiment override (the main live LLM surface) is validated —
  a large, measured upgrade over keywords.
 - The event-direction enrichment ("— gold bullish/bearish" appended to
  surprise descriptions) is right ~85–90% of the time but can state a
  factually backwards comparison in the displayed reason. It is display-only
  (never gates signals), so exposure is a possibly-misleading Telegram/CLI
  line, not trades.
 - **Hardening SHIPPED same day (owner-approved):** the payload now carries a
  code-computed `vs_forecast` field ("above/below/in line with forecast",
  from `EconomicEvent.surprise`) and the prompt tells the model to trust it.
  First iteration fixed the arithmetic but exposed a masked doctrine error
  (model read "claims below forecast" correctly, then called it a *weaker*
  labor market — and regressed the unemployment-rate case the same way), so
  the prompt's inverted-metrics line was expanded to an explicit rule plus a
  worked claims example. Verification: **18/18** — all 4 original misreads
  fixed, 11/11 no-regression cases unchanged, 3/3 fresh probes on the side
  the prompt example does NOT cover (claims 245K vs f 220K → bullish,
  unemployment 4.1% vs 3.9% → bullish, JOLTS 9.10M vs 8.80M → bearish).
  Suite green at 346 tests, incl. a new payload assertion in Module 19.
 - Production observations found while building the harness (not fixed):
  `_llm_event_cache` key omits forecast → a repeated `actual` re-serves a
  direction computed for a different forecast; a "neutral" reply is
  indistinguishable from a failure in the return value.
 ## Reproduce
 ```
 venv/bin/python backtest/research/llm_news_eval.py events --limit 5   # smoke
 venv/bin/python backtest/research/llm_news_eval.py events --determinism 20
 venv/bin/python backtest/research/llm_news_eval.py headlines
 venv/bin/python backtest/research/llm_news_eval.py report             # offline
 ```
 Corpus: 586 high-impact USD surprises 2024-01→2026-07 (calendar.parquet,
 NFP-verified UTC); prices: Oanda M15 mids. Full-run cost ≈ cents. The
 `report` subcommand (hit rates vs always-bullish/doctrine/FF-betterWorse
 baselines, McNemar, cluster view) is built and untested at scale — run it
 after completing the replay if a statistical answer is ever wanted.
@@ -0,0 +1,79 @@
 headline,category,expected_xau_sign,rationale
 "Powell says restrictive policy needed for longer as inflation proves sticky",hawkish_fed,-1,"higher-for-longer rates lift real yields, gold-bearish"
 "Fed minutes show officials leaning toward two more rate hikes this year",hawkish_fed,-1,"more hikes = stronger dollar/yields"
 "Hot CPI print fuels bets on additional Fed tightening",hawkish_fed,-1,"tightening expectations weigh on gold"
 "Fed's Waller: higher-for-longer rates warranted by resilient economy",hawkish_fed,-1,"hawkish guidance"
 "FOMC raises rates 50bp, flags further increases ahead",hawkish_fed,-1,"outsized hike plus hawkish guidance"
 "Treasury yields spike as Fed pushes back on rate cut hopes",hawkish_fed,-1,"cut hopes dashed, yields up"
 "Core inflation accelerates, cementing case for hawkish Fed stance",hawkish_fed,-1,"hawkish repricing"
 "Fed dot plot shifts higher; markets price out easing for the year",hawkish_fed,-1,"easing priced out"
 "Fed cuts rates 25bp and signals more easing to come",dovish_fed,1,"easing cycle is gold-bullish"
 "Powell hints at rate cuts as inflation cools toward target",dovish_fed,1,"cut guidance"
 "FOMC minutes reveal growing support for policy easing",dovish_fed,1,"dovish minutes"
 "Fed's Williams says rate cuts appropriate later this year",dovish_fed,1,"cut guidance"
 "Soft inflation data boosts odds of a dovish Fed pivot",dovish_fed,1,"pivot odds up"
 "Fed pauses hikes, markets price aggressive easing cycle",dovish_fed,1,"easing priced in"
 "Central banks turn dovish as global growth falters",dovish_fed,1,"global easing backdrop"
 "Fed to restart bond purchases to support flagging economy",dovish_fed,1,"QE is gold-bullish"
 "Dollar index hits 12-month high on robust US economic data",strong_usd,-1,"strong dollar pressures gold"
 "US GDP grows 4.1% annualized, blowing past expectations",strong_usd,-1,"strong economy, hawkish implication"
 "Blockbuster jobs report sends dollar surging across the board",strong_usd,-1,"strong jobs lift dollar and yields"
 "US retail sales smash forecasts, underscoring consumer strength",strong_usd,-1,"strong data"
 "Greenback rallies as US data outshines Europe and Asia",strong_usd,-1,"dollar strength"
 "ISM services index jumps to two-year high on strong demand",strong_usd,-1,"strong data"
 "Consumer confidence surges to post-pandemic high in the US",strong_usd,-1,"strong data"
 "Dollar tumbles to eight-month low after dismal manufacturing data",weak_usd,1,"weak dollar lifts gold"
 "US economy adds far fewer jobs than expected; dollar slides",weak_usd,1,"weak jobs, dovish implication"
 "Greenback weakens as US retail sales unexpectedly contract",weak_usd,1,"weak data"
 "US GDP growth revised down sharply, stoking slowdown fears",weak_usd,1,"slowdown is dovish"
 "Dollar slumps as trade deficit balloons to record",weak_usd,1,"dollar weakness"
 "Soft services PMI adds to evidence of cooling US economy",weak_usd,1,"cooling economy"
 "US consumer sentiment plunges to two-year low",weak_usd,1,"weak data"
 "Real yields climb to multi-year highs as TIPS sell off",real_yields,-1,"rising real yields raise gold's opportunity cost"
 "10-year Treasury yield surges past 5% on supply glut",real_yields,-1,"yield spike"
 "Rising real interest rates dim appeal of non-yielding assets",real_yields,-1,"explicit opportunity-cost story"
 "Treasury yields sink as investors flock to bonds",real_yields,1,"falling yields support gold"
 "Real yields turn negative as inflation outpaces rates",real_yields,1,"negative real yields are gold's best backdrop"
 "Bond rally drives 10-year yield to lowest level in a year",real_yields,1,"falling yields"
 "Geopolitical tensions escalate in Middle East, investors seek safety",safe_haven,1,"safe-haven bid"
 "Stocks crater worldwide as banking crisis fears spread",safe_haven,1,"crisis flight to safety"
 "Escalating conflict drives flight to safe-haven assets",safe_haven,1,"explicit haven flow"
 "Investors dump risk assets after surprise sovereign default",safe_haven,1,"default shock"
 "Military escalation on NATO border rattles global markets",safe_haven,1,"geopolitical stress"
 "Wall Street plunges 4% in worst session since pandemic",safe_haven,1,"risk-off"
 "Currency crisis in emerging markets sparks contagion fears",safe_haven,1,"contagion, haven bid"
 "Nuclear rhetoric intensifies, sending investors to havens",safe_haven,1,"extreme geopolitical stress"
 "Ceasefire agreement lifts global markets, haven demand fades",risk_on,-1,"haven unwind"
 "Stocks roar to record highs as trade deal signed",risk_on,-1,"risk-on drains haven demand"
 "Risk appetite returns as banking fears subside",risk_on,-1,"risk-on"
 "Global equities rally on strong earnings and easing tensions",risk_on,-1,"risk-on"
 "Gold hits record high as central banks keep buying bullion",gold_direct,1,"direct demand"
 "Gold ETFs post biggest weekly inflows in three years",gold_direct,1,"investment inflows"
 "China's central bank extends gold buying spree to 18th month",gold_direct,1,"official-sector demand"
 "Miners struggle to meet gold demand as supply deficit widens",gold_direct,1,"supply deficit"
 "Gold slides 2% as investors book profits after record run",gold_direct,-1,"profit-taking pressure"
 "Gold tumbles below key support as ETF outflows accelerate",gold_direct,-1,"outflows"
 "Bullion demand weakens as Indian jewellery buying stalls",gold_direct,-1,"physical demand soft"
 "Speculators slash bullish gold bets as dollar strengthens",gold_direct,-1,"positioning unwind"
 "Fed holds rates steady, signals no rate hike on the horizon",negation_traps,1,"dovish hold; keyword layer sees the phrase rate hike and flips it gold-bearish"
 "Powell says no rate cuts until inflation shows sustained cooling",negation_traps,-1,"hawkish hold; keyword layer sees rate cut and calls it gold-bullish"
 "Recession fears fade as US data streak beats estimates",negation_traps,-1,"fading fear is risk-on; keywords see recession plus beats and cancel out"
 "Inflation fails to accelerate, easing pressure on the Fed",negation_traps,1,"disinflation opens door to cuts"
 "Fed's Bowman: not yet confident inflation is beaten, cuts premature",negation_traps,-1,"hawkish pushback phrased around negations"
 "Central bank stops short of hiking, but warns tightening may resume",negation_traps,-1,"hawkish hold with tightening bias"
 "No safe-haven bid for gold despite equity wobble, analysts say",negation_traps,-1,"explicitly negated haven demand"
 "Jobs report not as weak as feared, dollar recovers",negation_traps,-1,"less-bad data is dollar-positive"
 "US manufacturing output plunges as new orders dry up",inverted_metrics,1,"weak economy is dovish/gold-bullish; keyword layer reads plunges as bearish"
 "US factory activity slumps to three-year low",inverted_metrics,1,"weak data; keyword reads slumps as bearish"
 "Consumer spending falls for third straight month",inverted_metrics,1,"weak data; keyword reads falls as bearish"
 "Private payrolls growth slides to weakest pace since 2020",inverted_metrics,1,"weak jobs; keyword reads slides as bearish"
 "Jobless claims hit two-year high as layoffs spread",inverted_metrics,1,"higher claims = weaker economy = gold-bullish"
 "US unemployment rate hits 4.5%, highest since 2021",inverted_metrics,1,"higher unemployment = dovish = gold-bullish"
 "Lakers clinch playoff berth with overtime win over Suns",distractors,0,"irrelevant"
 "New Marvel film smashes box office records on opening weekend",distractors,0,"irrelevant"
 "Apple unveils thinner MacBook with faster chip lineup",distractors,0,"irrelevant"
 "Scientists discover new deep-sea species off Pacific coast",distractors,0,"irrelevant"
 "Heavy rain forecast for the Midwest through the weekend",distractors,0,"irrelevant"
 "Streaming service raises subscription prices in Europe",distractors,0,"irrelevant"
 "Electric vehicle startup recalls 12,000 cars over software glitch",distractors,0,"irrelevant"
 "Local elections see record turnout in several US states",distractors,0,"irrelevant"
@@ -0,0 +1,199 @@
 """
 Oanda vs Twelve Data bar-divergence study — XAU/USD M15, 12 months.
 ====================================================================
 Compares the mid-price OHLC bars the signal engine would see from each
 provider over the same window. Purpose: quantify whether migrating the
 signal feed from Twelve Data to Oanda is a formality (bars ~identical)
 or a recalibration (bars differ at a scale that moves indicators).
 Inputs:
  --td      Twelve Data parquet (raw bars from backtest/cache/)
  --oanda   Oanda parquet (from fetch_oanda_m5.py / fetch_oanda_m15.py)
  --start / --end  analysis window (UTC dates)
 Outputs:
  printed report
  backtest/research/oanda_td_divergence_by_hour.csv
  backtest/research/oanda_td_bar_diffs.parquet  (per-bar diffs, common index)
 """
 import argparse
 from pathlib import Path
 import numpy as np
 import pandas as pd
 RESEARCH_DIR = Path(__file__).parent
 PIP = 0.10  # XAU pip in $
 def wilder_atr(df: pd.DataFrame, period: int = 14) -> pd.Series:
    hl = df["High"] - df["Low"]
    hc = (df["High"] - df["Close"].shift(1)).abs()
    lc = (df["Low"] - df["Close"].shift(1)).abs()
    tr = pd.concat([hl, hc, lc], axis=1).max(axis=1)
    return tr.ewm(alpha=1.0 / period, adjust=False).mean()
 def pct_stats(s: pd.Series) -> dict:
    a = s.abs()
    return {
        "mean": a.mean(), "median": a.median(),
        "p90": a.quantile(0.90), "p95": a.quantile(0.95),
        "p99": a.quantile(0.99), "max": a.max(),
        "bias(mean signed)": s.mean(),
    }
 def fmt_stats(name: str, s: pd.Series) -> str:
    st = pct_stats(s)
    return (f"  {name:<6} mean ${st['mean']:.3f}  med ${st['median']:.3f}  "
            f"p95 ${st['p95']:.3f}  p99 ${st['p99']:.3f}  max ${st['max']:.2f}  "
            f"bias {st['bias(mean signed)']:+.4f}")
 def main():
    ap = argparse.ArgumentParser()
    ap.add_argument("--td", default=str(RESEARCH_DIR.parent / "cache" /
                                        "XAU_USD_15min_20240522_20260723.parquet"))
    ap.add_argument("--oanda", default=str(RESEARCH_DIR /
                                           "oanda_xau_m15_20250701_now.parquet"))
    ap.add_argument("--start", default="2025-07-24")
    ap.add_argument("--end", default="2026-07-23")
    args = ap.parse_args()
    td = pd.read_parquet(args.td)
    oa = pd.read_parquet(args.oanda)
    start = pd.Timestamp(args.start, tz="UTC")
    end = pd.Timestamp(args.end, tz="UTC") + pd.Timedelta(days=1)
    start = max(start, td.index[0], oa.index[0])
    end = min(end, td.index[-1], oa.index[-1])
    td = td.loc[(td.index >= start) & (td.index <= end)]
    oa = oa.loc[(oa.index >= start) & (oa.index <= end)]
    print("=" * 78)
    print("OANDA vs TWELVE DATA — XAU/USD M15 bar divergence")
    print(f"Window: {start} -> {end}  ({(end - start).days} days)")
    print("=" * 78)
    # ── Coverage ─────────────────────────────────────────────────────
    common = td.index.intersection(oa.index)
    td_only = td.index.difference(oa.index)
    oa_only = oa.index.difference(td.index)
    print(f"\nCOVERAGE")
    print(f"  Twelve Data bars : {len(td):>7,}")
    print(f"  Oanda bars       : {len(oa):>7,}")
    print(f"  common           : {len(common):>7,}")
    print(f"  TD-only          : {len(td_only):>7,}  "
          f"({100 * len(td_only) / max(len(td), 1):.2f}% of TD)")
    print(f"  Oanda-only       : {len(oa_only):>7,}  "
          f"({100 * len(oa_only) / max(len(oa), 1):.2f}% of Oanda)")
    for label, idx in (("TD-only", td_only), ("Oanda-only", oa_only)):
        if len(idx):
            by_date = pd.Series(1, index=idx).groupby(idx.date).sum()
            top = by_date.sort_values(ascending=False).head(8)
            print(f"  {label} bars cluster on: "
                  + ", ".join(f"{d} ({n})" for d, n in top.items()))
            by_hour = pd.Series(1, index=idx).groupby(idx.hour).sum()
            top_h = by_hour.sort_values(ascending=False).head(6)
            print(f"  {label} by UTC hour: "
                  + ", ".join(f"{h:02d}h ({n})" for h, n in top_h.items()))
    t = td.loc[common]
    o = oa.loc[common]
    # ── Scale references ─────────────────────────────────────────────
    atr = wilder_atr(t).dropna()
    mean_price = t["Close"].mean()
    print(f"\nSCALE REFERENCES (Twelve Data feed)")
    print(f"  mean close      : ${mean_price:,.2f}")
    print(f"  mean bar range  : ${(t['High'] - t['Low']).mean():.3f}")
    print(f"  mean ATR14      : ${atr.mean():.3f}   median ${atr.median():.3f}")
    # ── Per-field diffs (TD − Oanda mid) ─────────────────────────────
    print(f"\nPER-BAR DIFFS  (TD − Oanda, $; pip = $0.10)")
    diffs = {}
    for col in ("Open", "High", "Low", "Close"):
        d = t[col] - o[col]
        diffs[col] = d
        print(fmt_stats(col, d))
    dclose = diffs["Close"].abs()
    print(f"\n  |ΔClose| as % of mean ATR14 : "
          f"mean {100 * dclose.mean() / atr.mean():.1f}%   "
          f"p95 {100 * dclose.quantile(0.95) / atr.mean():.1f}%")
    print(f"\nTHRESHOLD EXCEEDANCE  (|ΔClose|)")
    for thr in (0.10, 0.30, 0.65, 1.00, 2.00, 5.00):
        pct = 100 * (dclose > thr).mean()
        print(f"  > ${thr:<5.2f}: {pct:6.2f}% of bars  ({(dclose > thr).sum():,})")
    # ── Direction / return agreement ─────────────────────────────────
    t_ret = t["Close"].pct_change()
    o_ret = o["Close"].pct_change()
    # only consecutive common bars (gap = one M15 step)
    step_ok = pd.Series(common, index=common).diff().eq(pd.Timedelta(minutes=15))
    tr, orr = t_ret[step_ok], o_ret[step_ok]
    corr = tr.corr(orr)
    t_dir = np.sign(t["Close"] - t["Open"])
    o_dir = np.sign(o["Close"] - o["Open"])
    both_nonflat = (t_dir != 0) & (o_dir != 0)
    dir_agree = 100 * (t_dir[both_nonflat] == o_dir[both_nonflat]).mean()
    print(f"\nAGREEMENT")
    print(f"  close-return correlation (consecutive bars): {corr:.5f}  "
          f"(n={step_ok.sum():,})")
    print(f"  bar direction (sign of close−open) agree   : {dir_agree:.2f}%")
    # ── By-hour breakdown ────────────────────────────────────────────
    hours = common.hour
    rows = []
    for h in range(24):
        m = hours == h
        if not m.any():
            continue
        row = {"hour_utc": h, "bars": int(m.sum()),
               "mean_abs_dclose": dclose[m].mean(),
               "p95_abs_dclose": dclose[m].quantile(0.95),
               "mean_abs_dhigh": diffs["High"][m].abs().mean(),
               "mean_abs_dlow": diffs["Low"][m].abs().mean()}
        if "bid_c" in o.columns:
            row["oanda_median_spread"] = (
                (o["ask_c"] - o["bid_c"])[m].median())
        rows.append(row)
    by_hour = pd.DataFrame(rows)
    out_hour = RESEARCH_DIR / "oanda_td_divergence_by_hour.csv"
    by_hour.to_csv(out_hour, index=False, float_format="%.4f")
    print(f"\nBY UTC HOUR  (written to {out_hour.name})")
    print(by_hour.to_string(index=False,
                            float_format=lambda x: f"{x:.3f}"))
    # ── By-month drift ───────────────────────────────────────────────
    print(f"\nBY MONTH  (mean |ΔClose|, $)")
    by_month = dclose.groupby(common.to_period("M")).mean()
    for p, v in by_month.items():
        print(f"  {p}: {v:.3f}")
    # ── Oanda real spread (bonus, ties to Dukascopy study) ───────────
    if "bid_c" in o.columns:
        spread = o["ask_c"] - o["bid_c"]
        liquid = ~o.index.hour.isin([21, 22, 23, 0])
        print(f"\nOANDA REAL SPREAD (close bid/ask, $)")
        print(f"  all bars      : median {spread.median():.3f}  "
              f"p25 {spread.quantile(0.25):.3f}  p75 {spread.quantile(0.75):.3f}")
        print(f"  liquid hours  : median {spread[liquid].median():.3f}")
        print(f"  21-00 UTC     : median {spread[~liquid].median():.3f}")
    # ── Persist per-bar diffs ────────────────────────────────────────
    out = pd.DataFrame({
        "td_close": t["Close"], "oa_close": o["Close"],
        "d_open": diffs["Open"], "d_high": diffs["High"],
        "d_low": diffs["Low"], "d_close": diffs["Close"],
    })
    out_path = RESEARCH_DIR / "oanda_td_bar_diffs.parquet"
    out.to_parquet(out_path)
    print(f"\nPer-bar diffs written to {out_path}")
 if __name__ == "__main__":
    main()
@@ -0,0 +1,25 @@
 hour_utc,bars,mean_abs_dclose,p95_abs_dclose,mean_abs_dhigh,mean_abs_dlow,oanda_median_spread
 0,1029,3.3483,13.9460,3.2906,3.4310,0.7900
 1,1028,3.3483,12.8292,3.3477,3.6398,0.7700
 2,1028,3.0288,10.8960,2.9469,3.1317,0.7700
 3,1028,2.8783,11.1300,2.8419,2.9992,0.7600
 4,1028,2.8232,10.4710,2.8165,2.8455,0.7400
 5,1028,2.9268,10.6027,2.9234,3.0038,0.7400
 6,1028,3.0466,10.9265,2.9487,3.0796,0.7200
 7,1028,3.0286,11.3512,3.0301,3.0797,0.6700
 8,1028,3.0223,11.1483,2.8989,3.0801,0.6600
 9,1028,3.0460,11.6345,2.9307,3.0609,0.6600
 10,1028,2.9280,10.9015,2.8647,3.0029,0.6600
 11,1028,3.0340,11.4712,2.9591,2.9472,0.6600
 12,1028,3.0986,11.3282,2.9412,3.1407,0.7000
 13,1028,3.4014,12.3343,3.1077,3.4228,0.7300
 14,1028,3.5074,11.3628,3.3960,3.6569,0.6900
 15,1028,3.4514,11.8982,3.4124,3.6733,0.6700
 16,1028,3.1275,12.5155,3.2038,3.3061,0.6700
 17,1020,2.9826,12.6435,3.0064,3.1008,0.6900
 18,1015,2.8721,12.5480,2.9101,2.9996,0.6900
 19,1004,2.8637,12.1335,2.8824,2.9454,0.6900
 20,996,2.8006,11.9325,2.8486,2.8577,0.7600
 21,335,1.1085,4.8600,1.1212,1.2034,0.9400
 22,676,4.0827,13.6787,4.8422,4.6761,0.8300
 23,1027,3.2722,12.5725,3.5617,3.7747,0.8700
@@ -0,0 +1,290 @@
 """
 Classical chart-pattern detection for the XAU_CP entry archetype.
 ================================================================
 Third entry archetype (after the composite-breakout generators and the
 XAU_PB pullback): trade the completion of a proven multi-bar price
 pattern. Pattern set is deliberately tight — the classical patterns
 with the strongest documented statistics (Bulkowski, *Encyclopedia of
 Chart Patterns*) that are unambiguous from OHLC at bar close:
    double_bottom  bullish reversal   two matching swing lows, break of
                                      the interim peak (neckline)
    double_top     bearish reversal   mirrored
    inv_hs         bullish reversal   inverse head & shoulders, break of
                                      the neckline (the LOWER of the two
                                      peaks flanking the head — the earlier
                                      horizontal trigger, no sloped fits;
                                      validated as-is, see detector note)
    hs_top         bearish reversal   head & shoulders top, mirrored
    bull_flag      bullish continuation  strong pole, shallow ordered
                                      retrace, close at a new high
    bear_flag      bearish continuation  mirrored
 Triangles / wedges / cup-and-handle are excluded on purpose: weaker
 historical stats and apex-timing ambiguity — more parameters for less
 edge.
 Design contract (mirrors the rest of the signal stack):
 - Detectors see ONLY the visible closed-bar slice the aggregator was
  given; the last row is the just-closed signal bar. No forming candle,
  no future bars → identical behavior live and in the bar-replay
  backtester, and signals never repaint.
 - All geometric thresholds are in ATR units so one parameter set is
  scale-free across timeframes. ATR comes from the engine (Wilder 14).
 - Trigger discipline: a pattern fires only on the bar that COMPLETES it
  — the first close beyond the neckline/boundary since the pattern
  formed. One breakout → at most one signal.
 - Swing points use the same local-extremum definition as the engine's
  structure S/R (window w bars each side), so a "swing" here is the
  same object a chartist sees on the engine's key levels.
 Every detector returns None or a dict:
    {"pattern": str, "direction": "BUY"|"SELL",
     "anchor": float,     # SL reference (pattern invalidation price)
     "trigger": float,    # neckline / boundary that was broken
     "height": float}     # measured-move distance in price units
 Defaults below are the textbook definitions, not tuned values. The
 validation record for which patterns/timeframes are enabled lives in
 pair_signals/xauusd.py (per repo rules, per-TF adoption happens in the
 factory, never in global defaults).
 """
 from typing import Dict, List, Optional
 import numpy as np
 import pandas as pd
 __all__ = ["detect_chart_pattern", "PATTERN_DETECTORS"]
 def _swing_indices(values: np.ndarray, w: int, find_high: bool) -> List[int]:
    """Indices of local extrema with `w` bars on each side.
    A bar i is a swing high if values[i] is the strict maximum of
    values[i-w : i+w+1] (ties reject — a flat top is not a clean swing).
    The last `w` bars can never qualify, which is exactly right: a swing
    isn't confirmed until `w` bars close after it.
    """
    n = len(values)
    out: List[int] = []
    for i in range(w, n - w):
        window = values[i - w : i + w + 1]
        if np.isnan(window).any():
            continue
        center = values[i]
        rest = np.delete(window, w)
        if find_high:
            if center > rest.max():
                out.append(i)
        else:
            if center < rest.min():
                out.append(i)
    return out
 def _first_close_beyond(closes: np.ndarray, start: int, level: float,
                        above: bool) -> bool:
    """True iff the LAST close is the first one beyond `level` since
    `start` (exclusive of the last bar)."""
    last = closes[-1]
    prior = closes[start:-1]
    if above:
        return last > level and (len(prior) == 0 or prior.max() <= level)
    return last < level and (len(prior) == 0 or prior.min() >= level)
 # ── Double bottom / top ──────────────────────────────────────────────
 def _detect_double(o, h, l, c, atr, cfg, bullish: bool) -> Optional[Dict]:
    w = cfg.cp_swing_window
    ext = l if bullish else h
    idx = _swing_indices(ext, w, find_high=not bullish)
    if len(idx) < 2:
        return None
    i1, i2 = idx[-2], idx[-1]
    n = len(c)
    if i2 - i1 < cfg.cp_min_sep_bars:
        return None
    if (n - 1) - i2 > cfg.cp_recency_bars:
        return None
    e1, e2 = ext[i1], ext[i2]
    if abs(e1 - e2) > cfg.cp_tolerance_atr * atr:
        return None
    if bullish:
        neckline = float(h[i1 : i2 + 1].max())
        floor = min(e1, e2)
        height = neckline - floor
    else:
        neckline = float(l[i1 : i2 + 1].min())
        ceil_ = max(e1, e2)
        height = ceil_ - neckline
    if not (cfg.cp_min_height_atr * atr <= height <= cfg.cp_max_height_atr * atr):
        return None
    # Pattern must not have been invalidated after the second extreme:
    # a lower low (bullish) / higher high (bearish) breaks the pattern.
    if bullish:
        if i2 + 1 < n and l[i2 + 1 :].min() < min(e1, e2) - 0.1 * atr:
            return None
        if not _first_close_beyond(c, i2, neckline, above=True):
            return None
        return {"pattern": "double_bottom", "direction": "BUY",
                "anchor": float(min(e1, e2)), "trigger": neckline,
                "height": float(height)}
    else:
        if i2 + 1 < n and h[i2 + 1 :].max() > max(e1, e2) + 0.1 * atr:
            return None
        if not _first_close_beyond(c, i2, neckline, above=False):
            return None
        return {"pattern": "double_top", "direction": "SELL",
                "anchor": float(max(e1, e2)), "trigger": neckline,
                "height": float(height)}
 # ── Head & shoulders (top and inverse) ───────────────────────────────
 def _detect_hs(o, h, l, c, atr, cfg, bullish: bool) -> Optional[Dict]:
    """bullish=True → inverse H&S (three swing LOWS, head deepest)."""
    w = cfg.cp_swing_window
    ext = l if bullish else h
    idx = _swing_indices(ext, w, find_high=not bullish)
    if len(idx) < 3:
        return None
    i1, i2, i3 = idx[-3], idx[-2], idx[-1]
    n = len(c)
    if (n - 1) - i3 > cfg.cp_recency_bars:
        return None
    s1, head, s2 = ext[i1], ext[i2], ext[i3]
    if bullish:
        # Head strictly the deepest low; shoulders roughly level.
        if not (head < s1 - cfg.cp_hs_head_min_atr * atr
                and head < s2 - cfg.cp_hs_head_min_atr * atr):
            return None
        if abs(s1 - s2) > cfg.cp_hs_shoulder_tol_atr * atr:
            return None
        # Neckline: the LOWER of the two interim peaks — the EARLIER of
        # the two horizontal triggers (max() would demand clearing both
        # peaks). The whole validated CP book (M5 BUY-only) ran on this
        # trigger — do not flip to max() without full re-validation.
        p1 = float(h[i1 : i2 + 1].max())
        p2 = float(h[i2 : i3 + 1].max())
        neckline = min(p1, p2)
        height = neckline - head
    else:
        if not (head > s1 + cfg.cp_hs_head_min_atr * atr
                and head > s2 + cfg.cp_hs_head_min_atr * atr):
            return None
        if abs(s1 - s2) > cfg.cp_hs_shoulder_tol_atr * atr:
            return None
        t1 = float(l[i1 : i2 + 1].min())
        t2 = float(l[i2 : i3 + 1].min())
        neckline = max(t1, t2)
        height = head - neckline
    if not (cfg.cp_min_height_atr * atr <= height <= cfg.cp_max_height_atr * atr):
        return None
    if bullish:
        # Invalidated if price undercut the head after the right shoulder.
        if i3 + 1 < n and l[i3 + 1 :].min() < head - 0.1 * atr:
            return None
        if not _first_close_beyond(c, i3, neckline, above=True):
            return None
        return {"pattern": "inv_hs", "direction": "BUY",
                "anchor": float(s2), "trigger": neckline,
                "height": float(height)}
    else:
        if i3 + 1 < n and h[i3 + 1 :].max() > head + 0.1 * atr:
            return None
        if not _first_close_beyond(c, i3, neckline, above=False):
            return None
        return {"pattern": "hs_top", "direction": "SELL",
                "anchor": float(s2), "trigger": neckline,
                "height": float(height)}
 # ── Flags (bull and bear) ────────────────────────────────────────────
 def _detect_flag(o, h, l, c, atr, cfg, bullish: bool) -> Optional[Dict]:
    n = len(c)
    lo_r, hi_r = cfg.cp_flag_retrace
    if bullish:
        # Pole top = highest high BEFORE the trigger bar.
        ip = int(np.argmax(h[:-1]))
        flag_len = (n - 1) - ip
        if not (cfg.cp_flag_min_bars <= flag_len <= cfg.cp_flag_max_bars):
            return None
        pole_start = max(0, ip - cfg.cp_flag_pole_bars + 1)
        pole_low = float(l[pole_start : ip + 1].min())
        pole = float(h[ip]) - pole_low
        if pole < cfg.cp_flag_pole_atr * atr:
            return None
        flag_low = float(l[ip + 1 : n].min())
        retrace = (float(h[ip]) - flag_low) / pole
        if not (lo_r <= retrace <= hi_r):
            return None
        if not _first_close_beyond(c, ip, float(h[ip]), above=True):
            return None
        return {"pattern": "bull_flag", "direction": "BUY",
                "anchor": flag_low, "trigger": float(h[ip]),
                "height": float(pole)}
    else:
        ip = int(np.argmin(l[:-1]))
        flag_len = (n - 1) - ip
        if not (cfg.cp_flag_min_bars <= flag_len <= cfg.cp_flag_max_bars):
            return None
        pole_start = max(0, ip - cfg.cp_flag_pole_bars + 1)
        pole_high = float(h[pole_start : ip + 1].max())
        pole = pole_high - float(l[ip])
        if pole < cfg.cp_flag_pole_atr * atr:
            return None
        flag_high = float(h[ip + 1 : n].max())
        retrace = (flag_high - float(l[ip])) / pole
        if not (lo_r <= retrace <= hi_r):
            return None
        if not _first_close_beyond(c, ip, float(l[ip]), above=False):
            return None
        return {"pattern": "bear_flag", "direction": "SELL",
                "anchor": flag_high, "trigger": float(l[ip]),
                "height": float(pole)}
 # Priority order: most structured / rarest first — if two patterns
 # complete on the same bar, the more specific one names the trade.
 PATTERN_DETECTORS = {
    "inv_hs":        lambda o, h, l, c, atr, cfg: _detect_hs(o, h, l, c, atr, cfg, True),
    "hs_top":        lambda o, h, l, c, atr, cfg: _detect_hs(o, h, l, c, atr, cfg, False),
    "double_bottom": lambda o, h, l, c, atr, cfg: _detect_double(o, h, l, c, atr, cfg, True),
    "double_top":    lambda o, h, l, c, atr, cfg: _detect_double(o, h, l, c, atr, cfg, False),
    "bull_flag":     lambda o, h, l, c, atr, cfg: _detect_flag(o, h, l, c, atr, cfg, True),
    "bear_flag":     lambda o, h, l, c, atr, cfg: _detect_flag(o, h, l, c, atr, cfg, False),
 }
 def detect_chart_pattern(df: pd.DataFrame, atr: float, cfg) -> Optional[Dict]:
    """Run enabled detectors over the last cp_lookback(+1) bars.
    `df` is the visible history at decision time (last row = signal
    bar). Returns the first match in PATTERN_DETECTORS priority order,
    or None.
    """
    if atr <= 0 or len(df) < cfg.cp_lookback + 1:
        return None
    tail = df.iloc[-(cfg.cp_lookback + 1):]
    o = tail["Open"].to_numpy(dtype=np.float64)
    h = tail["High"].to_numpy(dtype=np.float64)
    l = tail["Low"].to_numpy(dtype=np.float64)
    c = tail["Close"].to_numpy(dtype=np.float64)
    enabled = set(cfg.cp_patterns)
    for name, det in PATTERN_DETECTORS.items():
        if name not in enabled:
            continue
        hit = det(o, h, l, c, atr, cfg)
        if hit is not None:
            return hit
    return None
@@ -0,0 +1,52 @@
 {
  "telegram": {
    "token": "123456789:REPLACE_WITH_YOUR_BOT_TOKEN",
    "chat_id": "REPLACE_WITH_OWNER_CHAT_ID",
    "channel_id": "-100REPLACE_WITH_CHANNEL_ID_OR_EMPTY_STRING",
    "alert_min_quality": "C"
  },
  "data_providers": {
    "active": "oanda",
    "streaming": "tiingo",
    "twelve_data": {
      "api_key": "REPLACE_WITH_YOUR_TWELVE_DATA_API_KEY"
    },
    "oanda": {},
    "tiingo": {
      "api_key": "REPLACE_WITH_YOUR_TIINGO_API_KEY_OR_EMPTY"
    }
  },
  "scanning": {
    "pairs": ["XAU/USD"],
    "timeframes": ["M5", "M15", "H1", "H4"]
  },
  "pairs": {
    "XAU/USD": {
      "blocked_hours_utc": [0, 1, 2, 10, 11, 13, 16, 21]
    }
  },
  "oanda": {
    "account_id": "REPLACE_WITH_ACCOUNT_ID",
    "api_token": "REPLACE_WITH_API_TOKEN",
    "base_url": "https://api-fxpractice.oanda.com",
    "lot_size": 1,
    "quality_gate": "C",
    "max_hold_bars": 300
  },
  "deepseek": {
    "api_key": "REPLACE_WITH_YOUR_DEEPSEEK_API_KEY_OR_EMPTY",
    "model": "deepseek-v4-flash",
    "enabled": true,
    "timeout_seconds": 12,
    "daily_request_cap": 500
  },
  "agent_panel": {
    "enabled": false,
    "telegram": true,
    "dashboard": false,
    "min_quality": "B",
    "max_panels_per_day": 20,
    "reuse_ttl_seconds": 3600
  },
  "log_level": "INFO"
 }
@@ -27,21 +27,15 @@ class SignalStrength(Enum):
    STRONG_SELL = -2
 class MarketSession(Enum):
    SYDNEY = "sydney"
    TOKYO = "tokyo"
    LONDON = "london"
    NEW_YORK = "new_york"
    OVERLAP_LONDON_NY = "london_ny_overlap"
 # ── Supported pairs ──────────────────────────────────────────────
 XAUUSD = "XAU/USD"
 EURUSD = "EUR/USD"
 # price_precision: decimals for SL/TP rounding and display.
 # default_spread: round-trip cost per trade in price units, charged by
 # the backtester when --spread is not given.
 PAIR_INFO = {
-    XAUUSD: {"display": "XAU/USD", "pip_value": 0.10, "is_jpy_pair": False},
+    XAUUSD: {"display": "XAU/USD", "pip_value": 0.10,
-    EURUSD: {"display": "EUR/USD", "pip_value": 0.0001, "is_jpy_pair": False},
+             "price_precision": 2, "default_spread": 0.30},
 }
@@ -52,6 +46,12 @@ class TechnicalConfig:
    ema_medium: int = 21
    ema_slow: int = 50
    ema_trend: int = 200
    # 4-EMA mode only: score partial ribbon alignment (2-3 of 4 checks)
    # as NEUTRAL instead of a weak BUY/SELL. Research knob (2026-08-04
    # indicator audit: partial-alignment trades averaged -0.008R vs
    # +0.230R fully aligned on the M5 12mo Oanda book); default False =
    # legacy behavior until a replay campaign validates flipping it.
    ema_partial_neutral: bool = False
    rsi_period: int = 14
    rsi_overbought: float = 70.0
@@ -72,13 +72,13 @@ class TechnicalConfig:
    adx_strong_trend: float = 25.0
    adx_very_strong: float = 50.0
-    divergence_lookback: int = 14
+    divergence_lookback: int = 40
    swing_sr_lookback: int = 50
    disabled_indicators: List[str] = field(default_factory=list)
    ema_mode: str = "4ema"          # "4ema" (9/21/50/200) or "2ema" (50/200)
-    rsi_mode: str = "classic"       # "classic" or "regime_aware"
+    rsi_mode: str = "regime_aware"  # "classic" or "regime_aware"
    macd_as_pillar: bool = True     # MACD histogram is a scored signal
    # ROC (Rate of Change)
@@ -100,94 +100,201 @@ class TechnicalConfig:
@dataclass
 class SignalConfig:
    """Signal scoring & filtering."""
    min_confidence: float = 55.0
    weights: Dict[str, float] = field(default_factory=lambda: {
        "ema_trend": 1.0,
        "rsi": 1.5,
        "bollinger": 1.5,
        "macd": 1.5,
        "keltner": 1.5,
        "roc": 1.5,
        "candlestick": 1.0,
        "divergence": 1.5,
    })
    mtf_agreement_threshold: int = 2  # of 3 timeframes
    cooldown_bars: int = 5
    default_risk_pct: float = 1.0
    default_rr_ratio: float = 2.0
    atr_sl_multiplier: float = 1.5
    min_rr_ratio: float = 0.5
    # TP tightening (May 2026) was tuned on the pre-spread backtester.
    # Re-tested 2026-07-08 with $0.30 spread + gap fills: 0.8 nearly
    # halves total R (+36.1R vs +66.8R, worse in both train/val halves)
    # — smaller wins make the fixed spread a larger share of every TP.
    tp_tightening_factor: float = 1.0
    # ── Trade geometry ────────────────────────────────────────────
    # Round-trip transaction cost estimate in price units (XAU ~$0.30).
    round_trip_cost: float = 0.30
    # Spread gate: reject setups whose TP1 distance is below
    # round_trip_cost × this multiple (caps cost at ~3% of target).
    # Validated 2026-07: small-target trades lose in both data halves
    # on M5 while TP≥$10 trades earn ~+0.08R/trade.
    min_tp_cost_multiple: float = 30.0
    # Fixed SL floor in pips (80 pips = $8 on XAU). 0 disables — SL then
    # floors at ATR × v2_sl_atr_multiplier only.
    min_sl_pips: float = 80.0
    # TP1 floor as a fraction of SL distance (0.5 = historical RR 0.5).
    tp_floor_rr: float = 0.5
    # ── Trailing-stop exit (2026-07) ──────────────────────────────
    # Once favorable excursion reaches trail_arm_r × risk, the stop
    # trails trail_dist_r × risk behind the best price (updated at bar
    # close). TP1 becomes advisory — the trail is the exit. Validated
    # on 12mo M5: +105R vs +48R with the fixed RR-0.5 target, balanced
    # train/val, positive every quarter, PF 2.27. The RR-0.5 target was
    # the binding constraint — winners kept running after TP.
    # Set trail_arm_r = 0 to disable and exit at TP1 as before.
    trail_arm_r: float = 0.6
    trail_dist_r: float = 0.75
    # Pullback (XAU_PB) trades run tighter stops (~$8 vs ~$25), so the
    # global 0.6R arm fires on noise. Arm later; whole 0.8-arm row of
    # the grid is +35-38R vs +32R at global params (balanced halves).
    pb_trail_arm_r: float = 0.8
    pb_trail_dist_r: float = 0.75
@dataclass
 class NewsConfig:
    """Economic calendar & sentiment."""
-    tracked_currencies: List[str] = field(default_factory=lambda: ["USD", "XAU", "EUR"])
+    tracked_currencies: List[str] = field(default_factory=lambda: ["USD", "XAU"])
    min_impact: str = "medium"
    pre_event_buffer_minutes: int = 30
    post_event_buffer_minutes: int = 15
    # Whether event risk gates the SIGNAL (danger: confidence hard-capped
    # at 30; caution: 15-point penalty). Warnings are always emitted
    # regardless. Default False = XAU-tuned: replaying the live gates
    # against the validated 12mo XAU M5 book (547 trades, +139.9R)
    # showed they suppress 25-35 trades worth +4.6..+7.2R/yr — trades
    # entered near USD releases OUTPERFORM (+0.475R vs +0.256R avg).
    # Audit 2026-07-11 (calendar-replay study; tooling since removed).
    event_risk_caps_signals: bool = False
    # Whether headline sentiment moves the SIGNAL (aligned: +5 confidence;
    # conflicting: 10-point penalty). Warnings are always emitted
    # regardless. Default False = news is informational only; decisions
    # are pure technical, matching the backtester (use_news=False).
    sentiment_affects_signals: bool = False
    # Wider stay-out window after a high-impact release that surprised
    # vs forecast (inline releases use post_event_buffer_minutes).
    post_event_surprise_buffer_minutes: int = 45
    # How long fetched calendar/headline data is reused across scans.
    # Must stay below pre_event_buffer_minutes so an event can't cross
    # into the danger window while a stale cache says it's clear.
    context_cache_ttl_seconds: int = 600
    # Maximum age of RSS headlines accepted for scoring. Older headlines
    # are discarded before keyword analysis and LLM enrichment.
    headline_max_age_seconds: int = 43200  # 12 hours
    rss_feeds: List[str] = field(default_factory=lambda: [
        "https://investinglive.com/feed/",
        "https://www.fxstreet.com/rss",
        # Google News query targeted at gold drivers — also carries Kitco,
        # Reuters etc. whose direct feeds are unavailable.
        "https://news.google.com/rss/search?q=gold+price+OR+fed+OR+inflation&hl=en-US&gl=US&ceid=US:en",
        # CNBC "Economy" section
        "https://search.cnbc.com/rs/search/combinedcms/view.xml?partnerId=wrss01&id=20910258",
    ])
@dataclass
 class XauStrategyConfig:
-    """XAU/USD strategy config.
+    """XAU/USD strategy config."""
    Legacy gates (H4 RSI, H1 MACD, M15 score) removed in favor of
    new momentum indicators (ROC, Keltner squeeze, ADX slope).
    """
    enabled: bool = True
    rsi_entry_threshold: float = 45.0
    tp_atr_multiplier: float = 1.0
    sl_atr_multiplier: float = 2.0
-    min_ema200_tfs: int = 2
+    blocked_hours_utc: List[int] = field(default_factory=lambda: [0, 1, 2, 10, 11, 13, 16, 21])
    allow_generic_trades: bool = True
    blocked_hours_utc: List[int] = field(default_factory=lambda: [0, 1, 2])
    weekend_filter: bool = True
    min_atr_threshold: float = 1.0
-    v2_sl_atr_multiplier: float = 2.5
+    v2_sl_atr_multiplier: float = 2.0
    buy_threshold_boost: int = 0
    m5_confirmation_bars: int = 0
    h4_rsi_gate: bool = False
    h1_macd_gate: bool = False
    rsi_zone: tuple = (40.0, 65.0)
    allowed_weekdays: List[int] = field(default_factory=lambda: [0, 1, 2, 3, 4, 6])  # Mon-Fri + Sun (Sydney open)
    sell_min_quality: List[str] = field(default_factory=lambda: ["A+", "A", "B"])
-
+    # Volatility regimes in which SELL setups are allowed (low/normal-vol
-
+    # shorts lose consistently against gold's macro uptrend).
-@dataclass
+    sell_vol_regimes: List[str] = field(default_factory=lambda: ["high", "extreme"])
-class EurUsdStrategyConfig:
+    # Market regimes in which no XAU setup may fire (any direction).
-    """EUR/USD strategy config — separate from XAU for independent tuning."""
+    # Empty by default; per-timeframe factories opt in — e.g. the mixed
-    enabled: bool = True
+    # "trending" regime (strong ADX, EMAs disagree) on M15.
    blocked_hours_utc: List[int] = field(default_factory=lambda: [22, 23, 0, 1, 2, 3])
    weekend_filter: bool = True
    allowed_weekdays: List[int] = field(default_factory=lambda: [0, 1, 2, 3, 4])
    min_atr_threshold: float = 0.0003
    sl_atr_multiplier: float = 0.5
    tp_atr_multiplier: float = 1.5
    sl_atr_floor: float = 1.2
    min_rr_ratio: float = 1.0
    mature_move_threshold: float = 35.0
    tp_quality_scale: Dict[str, float] = field(default_factory=lambda: {
        "A+": 1.0, "A": 1.0, "B": 1.0, "C": 1.0,
    })
    blocked_regimes: List[str] = field(default_factory=list)
-    min_quality_grades: List[str] = field(default_factory=lambda: ["A+", "A"])
+    # Market regimes in which XAU SELL setups may not fire. Empty by
-    max_exhaustion_tags: int = 99
+    # default; per-timeframe factories opt in.
-    block_near_ema200: bool = False
+    sell_blocked_regimes: List[str] = field(default_factory=list)
-    session_hours: List[int] = field(default_factory=list)
+    # Bar-CLOSE (decision-time) UTC hours in which XAU SELL setups may
-    min_entry_score: float = 30.0
+    # not fire. Empty by default; per-timeframe factories opt in.
-    min_entry_confidence: float = 50.0
+    # BUYs are unaffected (unlike blocked_hours_utc, which is both-way).
-    block_rsi_extended: bool = True
+    sell_blocked_hours_utc: List[int] = field(default_factory=list)
-    be_trigger_r: float = 0.3
+
-    be_buffer_r: float = 0.05
+    # ── M5-specific filters ──────────────────────────────────────
    # M5 signals are noisy; require stronger confirmation before
    # generating trade setups. Without these, M5 is breakeven
    # (+0.01R/trade) with 21% drawdown.
    # Relaxed 2026-07-30 (30→25, 10→5) in the relaxation campaign:
    # the combined config (with the M5 hour-21 unblock) beat control
    # in ALL 6 calendar years of the 5yr Oanda replay (+230.9R vs
    # +168.0R, PF 1.26 vs 1.21, max DD 21.3 vs 27.0), 12mo +107.4R
    # vs +84.6R PF 1.36 both halves positive, OOS 2024-06→2025-03
    # +26.1R vs +17.2R, and at $0.70 spread +86.9R vs +66.4R.
    # Score floors below 25 are vacuous (grade C requires |score|≥25).
    m5_require_mtf: bool = True
    m5_min_abs_score: float = 25.0
    m5_min_confirmation_score: float = 5.0
    # A-grade M5 SELLs enter at the extreme of temporary dips and
    # lose (-0.017R/trade on 163 trades) — maximum conviction at
    # the worst entry. Cap SELL quality so A-grade can't trigger
    # the ATR-based setup generator; B-grade swing setups survive
    # because they catch moderate drops, not extremes.
    m5_sell_max_quality: List[str] = field(default_factory=lambda: ["B", "C"])
    # 'high' vol regime on M5 is chop transition — elevated noise
    # without directional expansion. 61% WR / negative total across
    # 12mo while low/normal/extreme are all positive (validated 2026-07).
    m5_blocked_vol_regimes: List[str] = field(default_factory=lambda: ["high"])
    # Block M5 SELLs when RSI is already < this floor: the dip is
    # extended and mean-reverts before the target (negative in both
    # halves of 6mo and 12mo backtests).
    m5_sell_min_rsi: float = 25.0
    # ── M5 pullback entry (XAU_PB) ──
    # Second entry archetype: enter on the retrace to EMA21/KC-mid in
    # an established trend instead of at Keltner-breakout extension.
    # Validated 2026-07: +38.3R/254 trades on 12mo, positive both
    # halves and all four full quarters, core trades undisturbed.
    m5_pullback_enabled: bool = True
    # Composite score is weaker mid-pullback BY DESIGN (price sits
    # below the momentum bands) — separate, lower floor than Gate 1.
    m5_pullback_min_score: float = 15.0
    m5_pullback_adx_min: float = 20.0
    # RSI must be in the pullback zone, NOT the extended zone the
    # breakout entries fire in (those run 65-80).
    m5_pullback_rsi_buy: tuple = (35.0, 62.0)
    m5_pullback_rsi_sell: tuple = (38.0, 65.0)
    # Retrace depth from the pre-pullback swing extreme, in ATRs.
    # Shallower = still extended; deeper = trend likely broken.
    # M5 gold trend legs measure 3-4 ATR swing-to-trough (median 3.9),
    # so the ceiling sits well above that of higher TFs.
    m5_pullback_retrace_atr: tuple = (0.8, 6.0)
    # Price must have touched EMA21/KC-mid within this many bars.
    m5_pullback_touch_bars: int = 3
    # ── Chart-pattern entry (XAU_CP) ──────────────────────────────
    # Third entry archetype: completion of a classical chart pattern
    # (see chart_patterns.py for the pattern set and design contract).
    # Defaults are textbook geometry, NOT tuned values. Disabled by
    # default — per-TF adoption happens in pair_signals/xauusd.py only
    # after the pattern book passes the standard validation rule
    # (positive R both halves of ≥12mo, PF ≥ 1.3, $0.30 spread,
    # trailing exits, next-open fills).
    cp_enabled: bool = False
    cp_patterns: List[str] = field(default_factory=lambda: [
        "inv_hs", "hs_top", "double_bottom", "double_top",
        "bull_flag", "bear_flag",
    ])
    cp_lookback: int = 72          # bars scanned for pattern structure
    cp_swing_window: int = 3       # bars each side to confirm a swing
    cp_recency_bars: int = 20      # last extreme must be this fresh
    cp_min_sep_bars: int = 8       # min bars between double extremes
    cp_tolerance_atr: float = 0.5  # double top/bottom level match tol
    cp_min_height_atr: float = 1.5   # pattern height floor (noise gate)
    cp_max_height_atr: float = 8.0   # pattern height ceiling
    cp_hs_head_min_atr: float = 0.5  # head must exceed shoulders by this
    cp_hs_shoulder_tol_atr: float = 1.0  # shoulder level match tol
    cp_flag_pole_atr: float = 3.0    # min pole size
    cp_flag_pole_bars: int = 12      # pole measured over ≤ this many bars
    cp_flag_min_bars: int = 3        # flag consolidation length bounds
    cp_flag_max_bars: int = 12
    cp_flag_retrace: tuple = (0.1, 0.5)  # flag depth as fraction of pole
    # ── Big-candle entry gate ──────────────────────────────────────
    # When the recent window on the scan's own primary TF contains an
    # abnormally large candle (bar range >= K x the PRIOR bar's ATR —
    # institutional burst; price may reverse violently), no XAU entry
    # of ANY kind may fire (core, XAU_PB, XAU_CP). 0.0 = disabled.
    # Per-TF factories opt in only after the standard validation rule.
    big_candle_atr_multiple: float = 0.0
    big_candle_lookback_hours: float = 24.0
@dataclass
@@ -200,7 +307,7 @@ class FilterConfig:
    session_filter: bool = True
    session_start_hour: int = 3
    session_end_hour: int = 23
-    blocked_hours_utc: List[int] = field(default_factory=lambda: [0, 1, 2])
+    blocked_hours_utc: List[int] = field(default_factory=lambda: [0, 1, 2, 10, 11, 13, 16, 21])
    allowed_weekdays: List[int] = field(default_factory=lambda: [0, 1, 2, 3, 4, 6])  # Mon-Fri + Sun (Sydney open)
@@ -213,7 +320,6 @@ class PlatformConfig:
    news: NewsConfig = field(default_factory=NewsConfig)
    backtest: FilterConfig = field(default_factory=FilterConfig)
    xau_strategy: XauStrategyConfig = field(default_factory=XauStrategyConfig)
    eur_strategy: EurUsdStrategyConfig = field(default_factory=EurUsdStrategyConfig)
    symbol: str = XAUUSD
    display_name: str = "XAU/USD"
@@ -226,8 +332,6 @@ class PlatformConfig:
    data_lookback_days: int = 60
    twelve_data_api_key: str = "f563b22ffdfd4924955decbe1a8c8701"
    refresh_interval: int = 300
    scan_pairs: List[str] = field(
@@ -236,6 +340,6 @@ class PlatformConfig:
    scan_timeframes: List[Timeframe] = field(
        default_factory=lambda: [Timeframe.M5, Timeframe.M15, Timeframe.H1, Timeframe.H4]
    )
-    candle_buffer_seconds: int = 10
+    ws_cooldown_seconds: int = 300
    log_level: str = "INFO"
@@ -0,0 +1,459 @@
 #!/usr/bin/env python3
 """
 XAU/USD static dashboard publisher.
 Standalone daemon that scans XAU/USD at candle closes and atomically
 writes `snapshot.json` + `news.json` (plus a copy of
 `static/dashboard.html`) into an Apache-served directory. Apache does
 all web serving — this process never opens a port, so a failure here
 can only ever mean a stale page.
 Independence rules (owner requirement — the telegram bot must be
 immune to anything on the web side):
  * Runs as its own process; never imported by, and never imports,
    `telegram_bot` / `oanda_daemon` (telegram_bot resolves credentials
    at import). The confirmation map and boundary math are therefore
    duplicated here as local constants.
  * The only shared-code dependency beyond the standard signal path is
    the additive `NewsEngine.get_week_events` accessor, which no
    production signal path calls.
 Scheduling: every M15/H1/H4 boundary is also an M5 boundary (all
 intervals are multiples of 5, anchored to UTC midnight), so one
 sequential loop sleeps to the next M5 boundary (+10 s buffer) and scans
 whichever timeframes just closed. Due-classification uses the RAW wake
 time — flooring 12:00:10 to the 5-minute grid yields the 12:00 boundary
 that just closed. Do NOT subtract the buffer first: 12:00:10 − 15 s
 floors to 11:55 and silently skips every H1/H4 scan (v1 bug, 2026-08-02).
 Scans run sequentially M5→M15→H1→H4 through one shared data provider so
 later timeframes hit the warm boundary cache — that ordering IS the
 duplicate-fetch dedup (providers have no in-flight dedup; Oanda has no
 throttle). Keep it sequential.
 Run:  python dashboard_publisher.py [--once] [--out DIR]
 Out dir precedence: --out → DASHBOARD_OUT_DIR env → /var/www/trade
 """
 import argparse
 import json
 import hashlib
 import logging
 import math
 import numbers
 import os
 import shutil
 import sys
 import time
 from datetime import datetime, timezone
 from logging.handlers import TimedRotatingFileHandler
 from typing import Dict, List, Optional
 from agent_panel import attach_panel
 from config import PlatformConfig, Timeframe, XAUUSD
 from main import report_to_dict
 from news_engine import NewsEngine
 from pair_signals import produce_signal
 from providers import make_data_provider
 import providers.base as _providers_base
 logger = logging.getLogger("dashboard")
 DASHBOARD_TFS = [Timeframe.M5, Timeframe.M15, Timeframe.H1, Timeframe.H4]
 # Local copy of the bot's confirmation map (see independence rules above).
 CONFIRMATIONS = {
    Timeframe.M5:  [Timeframe.M15, Timeframe.H1],
    Timeframe.M15: [Timeframe.M5, Timeframe.H1],
    Timeframe.H1:  [Timeframe.M15, Timeframe.H4],
    Timeframe.H4:  [Timeframe.H1, Timeframe.D1],
 }
 TF_INTERVAL_MINUTES = {
    Timeframe.M5: 5, Timeframe.M15: 15, Timeframe.H1: 60, Timeframe.H4: 240,
 }
 DEFAULT_OUT_DIR = "/var/www/trade"
 SCAN_BUFFER_SECONDS = 10
 SNAPSHOT_FILE = "snapshot.json"
 NEWS_FILE = "news.json"
 PAGE_FILE = "dashboard.html"
 SUMMARY_PHP_FILE = "news_summary.php"
 # Repo files copied into the out dir at daemon start (git pull + restart
 # updates them). The PHP endpoint reads news.json in place, so it lives
 # next to the page.
 STATIC_FILES = [PAGE_FILE, SUMMARY_PHP_FILE]
 # ══════════════════════════════════════════════════════════════════
 #  Pure layer (no network, fully covered by test_all.py)
 # ══════════════════════════════════════════════════════════════════
 def _safe_num(v):
    """Coerce to a JSON-safe float; NaN/±inf become None."""
    try:
        f = float(v)
    except (TypeError, ValueError):
        return v
    if math.isnan(f) or math.isinf(f):
        return None
    return f
 def _sanitize(obj):
    """Recursively make a payload safe for json.dumps(allow_nan=False):
    NaN/inf → None, numpy scalars → native ints/floats."""
    if isinstance(obj, bool):
        return obj
    if isinstance(obj, dict):
        return {k: _sanitize(v) for k, v in obj.items()}
    if isinstance(obj, (list, tuple)):
        return [_sanitize(v) for v in obj]
    if isinstance(obj, numbers.Integral):
        return int(obj)
    if isinstance(obj, numbers.Real):
        return _safe_num(obj)
    return obj
 def analysis_to_dict(analysis) -> Optional[dict]:
    """Serialize a TechnicalAnalysis. Iterates analysis.indicators only —
    never vars()/asdict(), which would drag dynamic pandas attrs
    (_rsi_series, _macd_histogram, …) into the payload. ATR (weight 0)
    is surfaced separately as context, not in the directional grid."""
    if analysis is None:
        return None
    indicators = {}
    atr_context = None
    for name, ind in analysis.indicators.items():
        if name.startswith("_"):
            continue
        entry = {
            "signal": ind.signal.name,
            "value": _safe_num(ind.value),
            "confidence": _safe_num(ind.confidence),
            "weight": _safe_num(ind.weight),
            "description": ind.description,
        }
        if name == "atr":
            atr_context = entry
        elif ind.weight != 0:
            indicators[name] = entry
    return {
        "timeframe": analysis.timeframe,
        "timestamp": str(analysis.timestamp),
        "composite_signal": analysis.composite_signal.name,
        "composite_score": _safe_num(analysis.composite_score),
        "confidence": _safe_num(analysis.confidence),
        "market_regime": analysis.market_regime,
        "volatility_regime": analysis.volatility_regime,
        "pattern_signals": list(analysis.pattern_signals),
        "key_levels": {k: _safe_num(v) for k, v in analysis.key_levels.items()},
        "indicators": indicators,
        "atr_context": atr_context,
    }
 def report_to_webapp_dict(report) -> dict:
    """main.report_to_dict + the per-TF detail the dashboard needs."""
    d = report_to_dict(report)
    d["primary_signal"] = report.primary_signal.name
    d["primary_score"] = _safe_num(report.primary_score)
    d["mtf_scores"] = {k: _safe_num(v) for k, v in report.mtf_scores.items()}
    d["exhaustion_count"] = report.exhaustion_count
    d["primary_analysis"] = analysis_to_dict(report.primary_analysis)
    d["confirmation_analyses"] = [
        analysis_to_dict(a) for a in report.confirmation_analyses
    ]
    return _sanitize(d)
 def event_to_dict(ev) -> dict:
    """EconomicEvent → dict. surprise/minutes_until/is_high_impact are
    properties, so asdict() would skip them — added explicitly.
    actual/forecast/previous stay strings as published ('228K', '0.3%')."""
    return {
        "time": ev.time.astimezone(timezone.utc).isoformat(),
        "currency": ev.currency,
        "impact": ev.impact,
        "event": ev.event,
        "forecast": ev.forecast,
        "previous": ev.previous,
        "actual": ev.actual,
        "surprise": ev.surprise,
        "minutes_until": _safe_num(ev.minutes_until),
        "is_high_impact": ev.is_high_impact,
    }
 def headline_to_dict(h) -> dict:
    # id uses the same sha1(headline) scheme as NewsEngine._llm_sent_cache;
    # it is the join key the PHP summary endpoint whitelists against.
    return {
        "id": hashlib.sha1(h.headline.encode("utf-8")).hexdigest(),
        "headline": h.headline,
        "source": h.source,
        "link": h.link,
        "description": h.description,
        "publisher": h.publisher,
        "timestamp": h.timestamp.astimezone(timezone.utc).isoformat()
                     if h.timestamp else None,
        "currencies": list(h.currencies_mentioned),
        "score": _safe_num(h.sentiment_score),
        "keywords": list(h.keywords),
        "score_by_currency": {
            k: _safe_num(v) for k, v in h.sentiment_by_currency.items()
        },
    }
 def build_news_payload(context, week_events, now: datetime) -> dict:
    """news.json: full calendar week + scored headlines + LLM summary."""
    headlines = sorted(
        context.news_headlines,
        key=lambda h: h.timestamp or now, reverse=True,
    )
    return _sanitize({
        "generated_at": now.isoformat(),
        "llm_summary": context.llm_summary,
        "sentiment": dict(context.news_sentiment),
        "high_impact_warning": context.high_impact_warning,
        "warning_message": context.warning_message,
        "calendar_available": context.calendar_available,
        "events": [event_to_dict(e) for e in week_events],
        "headlines": [headline_to_dict(h) for h in headlines],
    })
 def due_timeframes(wake_dt: datetime) -> List[Timeframe]:
    """Timeframes whose candle just closed at the boundary this wake
    belongs to. Takes the RAW wake time (boundary + buffer): flooring to
    the 5-minute grid yields the boundary that just passed. Never
    subtract the buffer first — see module docstring (v1 bug)."""
    minutes = wake_dt.hour * 60 + wake_dt.minute
    floored = minutes - (minutes % 5)
    return [tf for tf in DASHBOARD_TFS if floored % TF_INTERVAL_MINUTES[tf] == 0]
 def seconds_until_boundary(interval_minutes: int, buffer: int = 10) -> float:
    # Same math as telegram_bot.py / oanda_daemon.py — kept local so the
    # dashboard never imports bot modules (independence rules above).
    now = datetime.now(timezone.utc)
    total_minutes = now.hour * 60 + now.minute
    elapsed_minutes = total_minutes % interval_minutes
    elapsed_seconds = elapsed_minutes * 60 + now.second + now.microsecond / 1e6
    interval_seconds = interval_minutes * 60
    remaining = interval_seconds - elapsed_seconds + buffer
    # NOTE: remaining > buffer always (elapsed < interval by construction);
    # a call at the exact boundary instant targets the NEXT boundary —
    # steady-state loops wake at boundary+buffer, so only a process
    # started in the boundary microsecond skips one scan (2026-08-04
    # audit I42; the old `remaining <= buffer` guard was unreachable).
    return remaining
 def _empty_tf_entry() -> dict:
    return {"status": "warming", "scanned_at": None, "scan_seconds": None,
            "error": None, "error_at": None, "report": None}
 def build_snapshot_payload(snapshots: Dict[str, dict], now: datetime) -> dict:
    """snapshot.json: one entry per dashboard TF. Timeframes not yet
    scanned appear as status="warming" so the page can render a
    placeholder instead of a hole. server_time_epoch lets the client
    compute its clock offset once so countdowns don't drift."""
    timeframes = {}
    for tf in DASHBOARD_TFS:
        interval = TF_INTERVAL_MINUTES[tf]
        minutes = now.hour * 60 + now.minute
        secs_into = (minutes % interval) * 60 + now.second + now.microsecond / 1e6
        next_close = now.timestamp() + (interval * 60 - secs_into)
        entry = _empty_tf_entry()
        entry.update(snapshots.get(tf.value) or {})
        entry["next_close_epoch"] = next_close
        entry["next_scan_epoch"] = next_close + SCAN_BUFFER_SECONDS
        timeframes[tf.value] = entry
    return _sanitize({
        "symbol": XAUUSD,
        "server_time_epoch": now.timestamp(),
        "market_closed": _providers_base._is_forex_weekend(now),
        "timeframes": timeframes,
    })
 def publish(obj: dict, path: str) -> None:
    """Atomic JSON write: temp file + os.replace, so Apache never serves
    a torn file. allow_nan=False — a NaN slipping through would render
    the page blank (JSON.parse rejects bare NaN)."""
    data = json.dumps(obj, allow_nan=False, separators=(",", ":"))
    tmp = path + ".tmp"
    with open(tmp, "w", encoding="utf-8") as fh:
        fh.write(data)
    os.replace(tmp, path)
 # ══════════════════════════════════════════════════════════════════
 #  Runtime layer (constructed only from main(); never at import)
 # ══════════════════════════════════════════════════════════════════
 def _setup_logging(level: str = "INFO") -> None:
    log_dir = os.path.join(os.path.dirname(os.path.abspath(__file__)), "logs")
    os.makedirs(log_dir, exist_ok=True)
    log_file = os.path.join(log_dir, "dashboard.log")
    fmt = logging.Formatter(
        "%(asctime)s [%(levelname)s] %(name)s: %(message)s",
        datefmt="%Y-%m-%d %H:%M:%S")
    file_handler = TimedRotatingFileHandler(
        log_file, when="h", interval=3, backupCount=240, utc=True)
    file_handler.setFormatter(fmt)
    console_handler = logging.StreamHandler(sys.stdout)
    console_handler.setFormatter(fmt)
    logging.basicConfig(
        level=getattr(logging, level.upper(), logging.INFO),
        handlers=[file_handler, console_handler],
        force=True)
 class DashboardPublisher:
    def __init__(self, out_dir: str):
        self.out_dir = out_dir
        self.snapshots: Dict[str, dict] = {}
        self.news_payload: Optional[dict] = None
        self._dp = None
        self._news_engine: Optional[NewsEngine] = None
    def _provider(self):
        if self._dp is None:
            self._dp = make_data_provider(PlatformConfig())
        return self._dp
    def _news(self) -> NewsEngine:
        if self._news_engine is None:
            self._news_engine = NewsEngine(PlatformConfig().news)
        return self._news_engine
    def install_static(self) -> None:
        """Copy the repo's static files (page + PHP summary endpoint)
        into the out dir (atomic), so git pull + restart also updates
        them. The ai/ cache dir is NOT created here — the daemon runs as
        a different user than Apache; that dir is a one-time deploy step
        (deploy/README-dashboard.md)."""
        static_dir = os.path.join(
            os.path.dirname(os.path.abspath(__file__)), "static")
        for name in STATIC_FILES:
            src = os.path.join(static_dir, name)
            if not os.path.exists(src):
                logger.warning(f"{src} missing — not installed")
                continue
            dst = os.path.join(self.out_dir, name)
            shutil.copyfile(src, dst + ".tmp")
            os.replace(dst + ".tmp", dst)
    def scan_tf(self, tf: Timeframe) -> None:
        started = time.time()
        try:
            report = produce_signal(XAUUSD, tf, self._provider(),
                                    CONFIRMATIONS[tf])
            # Advisory AI panel, post-hoc and display-only. Off unless
            # config.json agent_panel.dashboard is true. The reuse-TTL
            # cache + max_panels_per_day budget bound the spend against
            # this process's own DeepSeek cap (a persisting M5 signal
            # re-serves its cached verdict instead of re-running every
            # bar); a fresh panel adds ≤~25 s before this TF publishes.
            attach_panel(report, surface="dashboard")
            self.snapshots[tf.value] = {
                "status": "ok",
                "scanned_at": datetime.now(timezone.utc).isoformat(),
                "scan_seconds": round(time.time() - started, 2),
                "error": None,
                "error_at": None,
                "report": report_to_webapp_dict(report),
            }
            logger.info(f"{tf.value}: {report.final_signal.name} "
                        f"q={report.signal_quality or '-'} "
                        f"({time.time() - started:.1f}s)")
        except Exception as e:
            # Keep the last good report; the page shows it with an
            # error badge instead of going blank.
            logger.exception(f"{tf.value} scan failed")
            entry = self.snapshots.get(tf.value) or _empty_tf_entry()
            entry["status"] = "error"
            entry["error"] = f"{type(e).__name__}: {e}"
            entry["error_at"] = datetime.now(timezone.utc).isoformat()
            self.snapshots[tf.value] = entry
    def refresh_news(self) -> None:
        try:
            ne = self._news()
            context = ne.get_fundamental_context()
            week = ne.get_week_events()
            self.news_payload = build_news_payload(
                context, week, datetime.now(timezone.utc))
        except Exception:
            logger.exception("news refresh failed — keeping last payload")
    def publish_snapshot(self) -> None:
        publish(build_snapshot_payload(
            self.snapshots, datetime.now(timezone.utc)),
            os.path.join(self.out_dir, SNAPSHOT_FILE))
    def publish_news(self) -> None:
        if self.news_payload is not None:
            publish(self.news_payload, os.path.join(self.out_dir, NEWS_FILE))
    def publish_all(self) -> None:
        self.publish_snapshot()
        self.publish_news()
    def run_cycle(self, tfs: List[Timeframe]) -> None:
        for tf in tfs:  # sequential — see module docstring (cache dedup)
            self.scan_tf(tf)
        # Snapshot lands BEFORE the news refresh: when the news TTL has
        # expired, refresh_news does FF + 4 RSS fetches (and possibly
        # DeepSeek calls) — 20-30 s that must not keep fresh signals off
        # the page.
        self.publish_snapshot()
        self.refresh_news()
        self.publish_news()
    def run_forever(self) -> None:
        self.run_cycle(DASHBOARD_TFS)  # populate the page at boot
        while True:
            delay = seconds_until_boundary(5, buffer=SCAN_BUFFER_SECONDS)
            logger.info(f"next M5 boundary in {delay:.0f}s")
            time.sleep(delay)
            try:
                due = due_timeframes(datetime.now(timezone.utc))
                logger.info(f"boundary wake — scanning {[t.value for t in due]}")
                self.run_cycle(due)
            except Exception:
                logger.exception("cycle failed")
 def main(argv=None) -> None:
    ap = argparse.ArgumentParser(description="XAU/USD static dashboard publisher")
    ap.add_argument("--once", action="store_true",
                    help="one scan+publish cycle, then exit")
    ap.add_argument("--out",
                    default=os.environ.get("DASHBOARD_OUT_DIR", DEFAULT_OUT_DIR),
                    help="output directory served by Apache "
                         f"(default: $DASHBOARD_OUT_DIR or {DEFAULT_OUT_DIR})")
    args = ap.parse_args(argv)
    _setup_logging()
    os.makedirs(args.out, exist_ok=True)
    pub = DashboardPublisher(args.out)
    pub.install_static()
    if args.once:
        pub.run_cycle(DASHBOARD_TFS)
        print(f"Published to {args.out}")
        return
    pub.run_forever()
 if __name__ == "__main__":
    main()
@@ -1,331 +0,0 @@
 """
 ForexBot2 — Data Provider (Twelve Data SDK)
 ==========================================================
 Fetches and validates OHLCV via the official `twelvedata` Python SDK.
 """
 import logging
 import os
 import time
 from collections import deque
 from typing import Dict, Optional
 import pandas as pd
 from twelvedata import TDClient
 from config import PlatformConfig, Timeframe, XAUUSD, PAIR_INFO
 logger = logging.getLogger(__name__)
 def _is_forex_weekend(now=None) -> bool:
    """Forex market closed: Friday 22:00 UTC → Sunday 22:00 UTC."""
    if now is None:
        now = pd.Timestamp.now(tz="UTC")
    wd = now.weekday()
    hour = now.hour
    if wd == 4 and hour >= 22:
        return True
    if wd == 5:
        return True
    if wd == 6 and hour < 22:
        return True
    return False
 class DataIntegrityError(Exception):
    """Raised when data fails validation checks."""
    pass
 class ForexDataProvider:
    """Twelve Data SDK client + on-disk parquet cache."""
    TD_INTERVALS = {
        Timeframe.M1:  "1min",
        Timeframe.M5:  "5min",
        Timeframe.M15: "15min",
        Timeframe.M30: "30min",
        Timeframe.H1:  "1h",
        Timeframe.H4:  "4h",
        Timeframe.D1:  "1day",
        Timeframe.W1:  "1week",
    }
    BARS_PER_DAY = {
        Timeframe.M1: 1440, Timeframe.M5: 288, Timeframe.M15: 96,
        Timeframe.M30: 48, Timeframe.H1: 24, Timeframe.H4: 6,
        Timeframe.D1: 1, Timeframe.W1: 0.143,
    }
    CANDLE_SECONDS = {
        Timeframe.M1: 60, Timeframe.M5: 300, Timeframe.M15: 900,
        Timeframe.M30: 1800, Timeframe.H1: 3600, Timeframe.H4: 14400,
        Timeframe.D1: 86400, Timeframe.W1: 604800,
    }
    RATE_LIMIT = 55
    RATE_WINDOW = 60.0
    def __init__(self, config: PlatformConfig):
        self.config = config
        self._api_key = (
            os.environ.get("TWELVE_DATA_API_KEY")
            or config.twelve_data_api_key
        )
        self._cache: Dict[str, pd.DataFrame] = {}
        self._cache_timestamps: Dict[str, float] = {}
        self._request_times: deque = deque()
        self._td: Optional[TDClient] = (
            TDClient(apikey=self._api_key) if self._api_key else None
        )
    @staticmethod
    def _to_td_symbol(symbol: str) -> str:
        if "/" in symbol:
            return symbol
        clean = symbol.replace("=X", "")
        return f"{clean[:3]}/{clean[3:]}"
    # ── Public API ─────────────────────────────────────────────────
    def fetch_ohlcv(
        self,
        symbol: str = XAUUSD,
        timeframe: Timeframe = Timeframe.H1,
        lookback_days: Optional[int] = None,
    ) -> pd.DataFrame:
        """Fetch OHLCV with in-memory caching for live mode."""
        if lookback_days is None:
            lookback_days = self.config.data_lookback_days
        cache_key = f"{symbol}_{timeframe.value}"
        now = time.time()
        cache_ttl = self.CANDLE_SECONDS.get(timeframe, self.config.refresh_interval)
        if (
            cache_key in self._cache_timestamps
            and (now - self._cache_timestamps[cache_key]) < cache_ttl
            and cache_key in self._cache
        ):
            cached = self._cache.get(cache_key)
            if cached is not None and not cached.empty:
                if self._is_data_fresh(cached, symbol, timeframe):
                    return cached
                logger.warning(f"{symbol} {timeframe.value}: cached data is stale, refetching")
                del self._cache[cache_key]
                del self._cache_timestamps[cache_key]
        outputsize = max(
            300,
            min(int(lookback_days * self.BARS_PER_DAY[timeframe]) + 20, 5000),
        )
        df = self._fetch_twelve_data(symbol, timeframe, outputsize)
        df = self._validate_and_clean(df, symbol, timeframe)
        if not self._is_data_fresh(df, symbol, timeframe):
            logger.warning(
                f"STALE DATA from API: {symbol} {timeframe.value} — "
                f"last bar {df.index[-1]} is too old. Signal quality may be degraded."
            )
        self._cache[cache_key] = df
        self._cache_timestamps[cache_key] = now
        return df
    def _is_data_fresh(self, df: pd.DataFrame, symbol: str, timeframe: Timeframe) -> bool:
        """Check if the last candle is within acceptable age for this timeframe."""
        if df is None or df.empty:
            return False
        now = pd.Timestamp.now(tz="UTC")
        if _is_forex_weekend(now):
            return True
        last_bar = df.index[-1]
        candle_seconds = self.CANDLE_SECONDS.get(timeframe, 3600)
        max_age_seconds = candle_seconds * 3
        age_seconds = (now - last_bar).total_seconds()
        if age_seconds > max_age_seconds:
            logger.warning(
                f"{symbol} {timeframe.value}: last bar {last_bar.strftime('%Y-%m-%d %H:%M')} "
                f"is {age_seconds / 60:.0f}m old (limit: {max_age_seconds / 60:.0f}m)"
            )
            return False
        return True
    def clear_cache(self) -> int:
        """Wipe all cached data. Returns number of entries cleared."""
        count = len(self._cache)
        self._cache.clear()
        self._cache_timestamps.clear()
        logger.info(f"Cache cleared: {count} entries wiped")
        return count
    def get_latest_price(self, symbol: str = XAUUSD) -> Optional[float]:
        try:
            df = self.fetch_ohlcv(symbol, Timeframe.M5, lookback_days=1)
            if df is not None and not df.empty:
                return float(df["Close"].iloc[-1])
        except Exception:
            pass
        return None
    def get_spread_estimate(self, symbol: str = XAUUSD) -> float:
        try:
            df = self.fetch_ohlcv(symbol, Timeframe.M5, lookback_days=1)
            if df is not None and not df.empty:
                return float((df["High"] - df["Low"]).median())
        except Exception:
            pass
        info = PAIR_INFO.get(symbol, PAIR_INFO[XAUUSD])
        return info["pip_value"]
    def get_pair_info(self, symbol: str = XAUUSD) -> Dict:
        info = PAIR_INFO.get(symbol, PAIR_INFO[XAUUSD])
        return {"symbol": symbol, **info}
    # ── Private ────────────────────────────────────────────────────
    def _throttle(self, credits: int = 1) -> None:
        """Pro tier: 55 credits / minute. Block until budget allows."""
        while True:
            now = time.time()
            while self._request_times and now - self._request_times[0] >= self.RATE_WINDOW:
                self._request_times.popleft()
            available = self.RATE_LIMIT - len(self._request_times)
            if available >= credits:
                for _ in range(credits):
                    self._request_times.append(time.time())
                return
            wait = (self._request_times[0] + self.RATE_WINDOW) - now + 0.2
            logger.info(
                f"Rate limit: {len(self._request_times)}/{self.RATE_LIMIT} credits used "
                f"— waiting {wait:.1f}s..."
            )
            time.sleep(wait)
    def _fetch_twelve_data(
        self,
        symbol: str,
        timeframe: Timeframe,
        outputsize: int,
        start_date: Optional[str] = None,
        end_date: Optional[str] = None,
    ) -> pd.DataFrame:
        if self._td is None:
            raise DataIntegrityError(
                "No Twelve Data API key. Set TWELVE_DATA_API_KEY env var "
                "or config.twelve_data_api_key."
            )
        td_symbol = self._to_td_symbol(symbol)
        interval = self.TD_INTERVALS[timeframe]
        self._throttle(credits=1)
        kwargs = {
            "symbol": td_symbol,
            "interval": interval,
            "outputsize": min(outputsize, 5000),
            "timezone": "UTC",
        }
        if start_date:
            kwargs["start_date"] = start_date
        if end_date:
            kwargs["end_date"] = end_date
        try:
            df = self._td.time_series(**kwargs).as_pandas()
        except Exception as e:
            raise DataIntegrityError(
                f"Twelve Data fetch failed for {td_symbol}: {e}"
            ) from e
        if df is None or df.empty:
            raise DataIntegrityError(f"Empty response from Twelve Data for {td_symbol}")
        # SDK returns lowercase columns; engine expects capitalised.
        df = df.rename(columns={
            "open": "Open", "high": "High", "low": "Low",
            "close": "Close", "volume": "Volume",
        })
        for col in ("Open", "High", "Low", "Close", "Volume"):
            if col in df.columns:
                df[col] = pd.to_numeric(df[col], errors="coerce")
        if "Volume" not in df.columns:
            df["Volume"] = 0.0
        # SDK returns newest-first; engine + indicators want oldest-first.
        df = df.sort_index()
        if df.index.tz is None:
            df.index = df.index.tz_localize("UTC")
        df.index.name = "Datetime"
        logger.debug(f"Fetched {len(df)} bars for {td_symbol} @ {interval}")
        return df
    def _validate_and_clean(
        self, df: pd.DataFrame, symbol: str, timeframe: Timeframe
    ) -> pd.DataFrame:
        if df is None or df.empty:
            raise DataIntegrityError(f"Empty dataset for {symbol}")
        original_len = len(df)
        df = df.dropna(subset=["Open", "High", "Low", "Close"], how="all")
        df = df.ffill(limit=3)
        df = df.dropna(subset=["Close"])
        df = df.sort_index()
        df = df[~df.index.duplicated(keep="last")]
        now = pd.Timestamp.utcnow()
        future_mask = df.index > now + pd.Timedelta(minutes=5)
        if future_mask.any():
            logger.warning(f"{symbol}: Dropping {future_mask.sum()} future timestamps")
            df = df[~future_mask]
        invalid_ohlc = (
            (df["High"] < df["Low"]) |
            (df["High"] < df["Open"]) |
            (df["High"] < df["Close"]) |
            (df["Low"] > df["Open"]) |
            (df["Low"] > df["Close"])
        )
        if invalid_ohlc.any():
            logger.warning(f"{symbol}: Fixing {invalid_ohlc.sum()} OHLC-inconsistent bars")
            for idx in df[invalid_ohlc].index:
                vals = [df.loc[idx, "Open"], df.loc[idx, "Close"]]
                df.loc[idx, "High"] = max(vals + [df.loc[idx, "High"]])
                df.loc[idx, "Low"]  = min(vals + [df.loc[idx, "Low"]])
        returns = df["Close"].pct_change()
        std_ret = returns.std()
        if std_ret > 0:
            outlier_mask = (returns - returns.mean()).abs() > 10 * std_ret
            if outlier_mask.any():
                logger.warning(f"{symbol}: Removing {outlier_mask.sum()} extreme outlier bars")
                df = df[~outlier_mask]
        min_required = max(
            self.config.technical.ema_trend,
            200,
        ) + 10
        if len(df) < min_required:
            logger.warning(
                f"{symbol} {timeframe.value}: Only {len(df)} bars "
                f"(need {min_required}). Some indicators will be partial."
            )
        cleaned = original_len - len(df)
        if cleaned > 0:
            logger.info(f"{symbol}: Cleaned {cleaned}/{original_len} bars")
        return df
@@ -0,0 +1,157 @@
 # Deploying the web dashboard (live LAMP server)
 The dashboard is a **static publisher**: `dashboard_publisher.py` runs as its
 own daemon, scans XAU/USD at candle closes, and atomically writes
 `dashboard.html` + `news_summary.php` + `snapshot.json` + `news.json` into a
 folder Apache already serves. Apache does all web serving — no new port, no
 Python web server, and the telegram bot process is completely unaffected by
 anything that happens here (see the module docstring for the independence
 rules). The one dynamic piece is `news_summary.php` (§6): an on-demand
 DeepSeek summary for a clicked headline, whitelist-gated against news.json.
 All steps below run **on the live server** (`/home/Forex/ForexBot2`).
 ## 1. Get the code
 ```bash
 cd /home/Forex/ForexBot2
 git pull
 venv/bin/python test_all.py        # optional sanity: suite must be green
 ```
 No new Python dependencies are required.
 ## 2. Make the web folder writable
 The dashboard publishes straight into the portal docroot `/var/www/trade/`
 (owner decision 2026-08-03) — it adds exactly four files there:
 `dashboard.html`, `news_summary.php`, `snapshot.json`, `news.json` (plus the
 apache-owned `ai/` cache dir from §6). The daemon user must be able to write
 them:
 ```bash
 # if the portal files can be owned by the daemon user:
 sudo chown Forex: /var/www/trade
 # otherwise grant write without changing ownership:
 sudo setfacl -m u:Forex:rwx /var/www/trade
 ```
 (A different path is one edit away: the unit's `Environment=DASHBOARD_OUT_DIR=`
 line, or `--out` for one-off runs.)
 SELinux (RHEL/Alma/CentOS enforcing): directories under `/var/www` normally
 already carry `httpd_sys_content_t`, and new files inherit the parent label, so
 no action is usually needed. If Apache 403s despite correct permissions:
 ```bash
 sudo restorecon -R /var/www/trade
 ```
 ## 3. Smoke-test once
 ```bash
 venv/bin/python dashboard_publisher.py --once
 ls /var/www/trade    # dashboard.html snapshot.json news.json added next to the portal
 ```
 Then open `http://<server>/dashboard.html` (path relative to however the
 vhost maps `/var/www/trade`) — all four timeframe cards, the calendar, and
 the newswire should populate.
 ## 4. Install the daemon
 ```bash
 sudo cp deploy/gold-dashboard.service /etc/systemd/system/
 # edit User= if the repo owner is not "Forex"
 sudo systemctl daemon-reload
 sudo systemctl enable --now gold-dashboard
 systemctl status gold-dashboard
 tail -f logs/dashboard.log
 ```
 The daemon scans all four timeframes at boot, then wakes at every M5
 boundary (+10 s) and rescans whichever timeframes just closed. Logs rotate
 every 3 h (240 backups), same scheme as the bot.
 ## 5. Optional Apache polish
 The page polls its JSON with conditional GETs (`cache: "no-cache"` → Apache
 answers unchanged files with 304), so no Apache config is strictly needed. To
 be explicit anyway:
 ```apache
 <Directory "/var/www/trade">
    <FilesMatch "\.json$">
        Header set Cache-Control "no-cache"
    </FilesMatch>
 </Directory>
 ```
 The page is deliberately public (owner decision 2026-08-03). To restrict it
 later, add standard Apache basic auth (`htpasswd` + `Require valid-user`) on
 that directory — no code changes needed.
 ## 6. AI summary endpoint (PHP) — one-time setup
 Clicking a newswire headline opens a detail dialog that fetches an on-demand
 DeepSeek gold-impact summary from `news_summary.php` (installed next to the
 page by the daemon). Owner decision 2026-08-07: a real on-demand endpoint on
 the public page, accepted with these guards baked into the PHP — the id must
 match a headline in the published news.json (arbitrary text can never reach
 the LLM), each headline's summary is cached forever in `ai/` (≤1 API call per
 headline, ever), and a file-based daily cap fails **closed** on I/O errors.
 Requirements: `php` + the curl extension (`php -m | grep curl`).
 ```bash
 # 1. SELinux: allow httpd outbound HTTPS — WITHOUT this every summary call
 #    dies as a curl timeout even with perfect perms (RHEL-family default).
 sudo setsebool -P httpd_can_network_connect 1
 # 2. API key OUTSIDE the docroot, readable by Apache only:
 sudo install -d -o root -g root -m 755 /etc/gold-dashboard
 sudo sh -c 'umask 137; echo "sk-..." > /etc/gold-dashboard/deepseek.key'
 sudo chown root:apache /etc/gold-dashboard/deepseek.key   # mode 640
 # 3. apache-owned cache dir (the daemon runs as Forex and can't create it):
 sudo install -d -o apache -g apache -m 755 /var/www/trade/ai
 sudo semanage fcontext -a -t httpd_sys_rw_content_t '/var/www/trade/ai(/.*)?'
 sudo restorecon -Rv /var/www/trade/ai
 ```
 Knobs (Apache `SetEnv` in the vhost, both optional):
 `AI_SUMMARY_DAILY_CAP` (default 150), `DEEPSEEK_KEY_FILE` (default
 `/etc/gold-dashboard/deepseek.key`).
 Verify from the shell — expect 400 / 404 / 200 / 200-from-cache:
 ```bash
 curl -s "http://<server>/news_summary.php?id=zzz"                 # {"error":"bad_id"}
 curl -s "http://<server>/news_summary.php?id=$(printf 'a%.0s' {1..40})"  # unknown_id
 ID=$(python3 -c "import json;print(json.load(open('/var/www/trade/news.json'))['headlines'][0]['id'])")
 curl -s "http://<server>/news_summary.php?id=$ID"   # real summary (2-12 s)
 curl -s "http://<server>/news_summary.php?id=$ID"   # instant, from ai/<id>.json
 ```
 ## Operational notes
 - **Updating**: `git pull` + `systemctl restart gold-dashboard`. The daemon
  re-copies `static/dashboard.html` and `static/news_summary.php` into the
  web folder at startup.
 - **If the daemon dies**, systemd restarts it in 30 s; worst case the page
  shows stale data with its own "updated N min ago" evidence. Apache and the
  telegram bot are unaffected.
 - **DeepSeek**: the publisher process has its own daily request cap
  (`deepseek.daily_request_cap`, default 500) separate from the bot's —
  expect roughly up to ~430 extra calls/day worst case, V4-Flash pricing.
 - **Two independent DeepSeek meters** hit the same API key: the publisher's
  in-process cap above, and the PHP endpoint's file cap (`ai/.ht_cap.json`,
  default 150/day). They don't communicate — worst-case spend is their sum;
  DeepSeek-side spend alerting is the true backstop. Kept separate on
  purpose (independence rule: the PHP never imports/reads bot code or
  config.json).
 - **Restart window**: for the first minutes after a daemon restart the new
  page can meet a pre-upgrade news.json without headline ids — those rows
  simply render non-clickable until the first news publish.
 - **Credentials**: uses the same env/.env/config.json resolution as the rest
  of the platform (Oanda candles feed; news needs no key; DeepSeek optional).
@@ -0,0 +1,23 @@
 # systemd unit for the XAU/USD static dashboard publisher.
 # Install (as root, on the live server):
 #   cp deploy/gold-dashboard.service /etc/systemd/system/
 #   systemctl daemon-reload && systemctl enable --now gold-dashboard
 # See deploy/README-dashboard.md for the full checklist.
 [Unit]
 Description=XAU/USD dashboard publisher (writes static JSON/HTML for Apache)
 After=network-online.target
 Wants=network-online.target
 [Service]
 Type=simple
 # Adjust to the account that owns /home/Forex/ForexBot2:
 User=Forex
 WorkingDirectory=/home/Forex/ForexBot2
 Environment=DASHBOARD_OUT_DIR=/var/www/trade
 ExecStart=/home/Forex/ForexBot2/venv/bin/python dashboard_publisher.py
 Restart=always
 RestartSec=30
 [Install]
 WantedBy=multi-user.target
@@ -0,0 +1,199 @@
 """ForexBot2 architecture diagram → docs/architecture.png.
 Regenerate after architecture changes:  venv/bin/python docs/architecture_diagram.py
 (matplotlib only — no graphviz on the dev box)."""
 import os
 import matplotlib
 matplotlib.use("Agg")
 import matplotlib.pyplot as plt
 from matplotlib.patches import FancyBboxPatch, FancyArrowPatch
 W, H = 100.0, 100.0
 fig, ax = plt.subplots(figsize=(16, 11.5))
 ax.set_xlim(0, W); ax.set_ylim(0, H); ax.axis("off")
 INK = "#263238"; MUT = "#607D8B"; ARR = "#455A64"; HAUL = "#7B8FA1"
 def box(x, y, w, h, title, lines, fc, ec, title_fs=8.2, body_fs=6.6, lw=1.3):
    ax.add_patch(FancyBboxPatch((x, y), w, h,
                 boxstyle="round,pad=0.35,rounding_size=0.9",
                 facecolor=fc, edgecolor=ec, linewidth=lw, zorder=2))
    cy = y + h - 1.9
    ax.text(x + w / 2, cy, title, ha="center", va="center", fontsize=title_fs,
            fontweight="bold", color=INK, zorder=3)
    cy -= 2.3
    for ln in lines:
        ax.text(x + w / 2, cy, ln, ha="center", va="center", fontsize=body_fs,
                color=INK, zorder=3)
        cy -= 1.85
 def band(x, y, w, h, label, color):
    ax.add_patch(FancyBboxPatch((x, y), w, h,
                 boxstyle="round,pad=0.3,rounding_size=1.2",
                 facecolor="none", edgecolor=color, linewidth=1.0,
                 linestyle=(0, (4, 3)), zorder=1))
    ax.text(x + 0.9, y + h - 1.4, label, ha="left", va="center",
            fontsize=7.2, fontweight="bold", color=color, zorder=3)
 def arrow(p1, p2, color=ARR, lw=1.5, style="-|>", conn="arc3,rad=0.0",
          label=None, lx=0, ly=0, ls="solid", fs=6.4, rot=0):
    ax.add_patch(FancyArrowPatch(p1, p2, arrowstyle=style, mutation_scale=11,
                 color=color, linewidth=lw, linestyle=ls,
                 connectionstyle=conn, zorder=4))
    if label:
        mx, my = (p1[0] + p2[0]) / 2 + lx, (p1[1] + p2[1]) / 2 + ly
        ax.text(mx, my, label, fontsize=fs, color=MUT, ha="center",
                va="center", zorder=5, rotation=rot,
                bbox=dict(boxstyle="round,pad=0.15", fc="white", ec="none", alpha=0.9))
 # ── title ────────────────────────────────────────────────────────────
 ax.text(2, 98.4, "ForexBot2 — XAU/USD Signal-Intelligence Platform",
        fontsize=15, fontweight="bold", color=INK, ha="left")
 ax.text(2, 96.0, "one pipeline, four delivery surfaces · closed bars only · live == backtest parity",
        fontsize=8.5, color=MUT, ha="left")
 EXT_FC, EXT_EC = "#ECEFF1", "#90A4AE"
 DAT_FC, DAT_EC = "#E3F2FD", "#5C9BD1"
 NWS_FC, NWS_EC = "#E0F2F1", "#3D9C8F"
 COR_FC, COR_EC = "#FFFDF5", "#C9A227"
 REP_FC, REP_EC = "#FFE9A8", "#B8860B"
 DEL_FC, DEL_EC = "#E8F5E9", "#5FAF63"
 BTS_FC, BTS_EC = "#EDE7F6", "#8E6BC6"
 # ── external services (top band) ─────────────────────────────────────
 band(1, 85.0, 98, 9.4, "EXTERNAL SERVICES", "#78909C")
 box(2.5, 86.2, 15.5, 6.4, "Oanda v20 API",
    ["candles (live feed)", "practice orders"], EXT_FC, EXT_EC)
 box(19.5, 86.2, 15.5, 6.4, "Twelve Data · Tiingo",
    ["registered fallbacks", "(Tiingo: tick stream)"], EXT_FC, EXT_EC)
 box(36.5, 86.2, 15.5, 6.4, "Forex Factory",
    ["economic calendar", "MyFXBook RSS fallback"], EXT_FC, EXT_EC)
 box(53.5, 86.2, 12.5, 6.4, "News RSS ×4",
    ["gold / macro", "headlines"], EXT_FC, EXT_EC)
 box(67.5, 86.2, 13.5, 6.4, "DeepSeek API",
    ["V4-Flash LLM", "thinking: disabled"], EXT_FC, EXT_EC)
 box(82.5, 86.2, 15, 6.4, "Telegram API",
    ["owner bot +", "subscriber channel"], EXT_FC, EXT_EC)
 # ── data & context layer ─────────────────────────────────────────────
 box(2.5, 69.8, 33, 11.0, "providers/  (BaseDataProvider)",
    ["single-flight fetch · TTL+boundary cache",
     "forming-candle DROP · validate/clean",
     "paginated fetch_range · optional tick stream",
     "active feed knob: data_providers.active = oanda"],
    DAT_FC, DAT_EC)
 box(38.5, 69.8, 33, 11.0, "news_engine.py  +  llm_client.py",
    ["calendar events + keyword sentiment (gold-inverse)",
     "DeepSeek enrichment: batch headline scores,",
     "event gold-direction, market blurb — display-only,",
     "keyword fallback, daily request cap"],
    NWS_FC, NWS_EC)
 box(74.5, 69.8, 23, 11.0, "app_config.py",
    ["credentials & deployment:",
     "env → config.json → refuse",
     "(no hardcoded fallbacks)",
     ".env loaded at import"],
    "#FAFAFA", "#9E9E9E")
 # ── signal core container ────────────────────────────────────────────
 ax.add_patch(FancyBboxPatch((2.5, 40.5), 95, 26.5,
             boxstyle="round,pad=0.35,rounding_size=1.2",
             facecolor=COR_FC, edgecolor=COR_EC, linewidth=1.6, zorder=1))
 ax.text(50, 65.2, "SIGNAL CORE — runs identically in live scans and backtests (closed bars only)",
        ha="center", fontsize=8.4, fontweight="bold", color="#8a6d1a", zorder=3)
 box(4.5, 51.8, 28, 12.0, "engine.py — TechnicalEngine",
    ["9 scored indicators + ATR context:",
     "EMA · RSI · MACD · BB · ADX · ROC ·",
     "Keltner · divergence · candlestick",
     "regime → adaptive weights → composite"],
    "white", COR_EC)
 box(4.5, 42.0, 28, 8.4, "pair_signals/xauusd.py — factory",
    ["ALWAYS used: config.json overlay +",
     "validated per-TF deltas (hour blocks, trail…)"],
    "white", COR_EC)
 box(35.5, 42.0, 60, 20.7, "signal_aggregator.py — analyze_pair → analyze_from_dataframes",
    ["multi-TF agreement  (M5→[M15,H1] · M15→[M5,H1] · H1→[M15,H4] · H4→[H1,D1])",
     "→ news context (warn-only by default) → diminishing-decay penalties",
     "→ exhaustion hard block → quality grade A+…F",
     "→ ordered post-grade gate chain (session · XAU SELL vol/hour · regime · M5 noise)",
     "→ setup generators: XAU ATR (A+/A) · generic swing · XAU_PB · XAU_CP",
     "→ entry · SL · TP1/TP2 · trailing plan"],
    "white", COR_EC, body_fs=6.9)
 # ── SignalReport hub ─────────────────────────────────────────────────
 box(31, 33.2, 38, 5.2, "SignalReport",
    ["signal (−2…+2) · grade · trade setup · reasons · news summary"],
    REP_FC, REP_EC, title_fs=8.8)
 # ── delivery surfaces ────────────────────────────────────────────────
 band(1, 13.2, 98, 17.2, "DELIVERY SURFACES", "#4C8F50")
 box(2.5, 15.0, 12, 12.6, "main.py",
    ["CLI report", "--detail", "--json"], DEL_FC, DEL_EC)
 box(16, 15.0, 24, 12.6, "oanda_daemon.py + executor",
    ["practice auto-execution",
     "book-parity rules: no broker TP,",
     "5-bar cooldown, once-per-bar",
     "trail, OPEN_ONLY fills,",
     "boot reconcile · orders → Oanda"],
    DEL_FC, DEL_EC, title_fs=7.8)
 box(42.5, 15.0, 31, 12.6, "dashboard_publisher.py  →  Apache",
    ["independent daemon · scans at candle closes",
     "atomic publish to /var/www/trade:",
     "dashboard.html · snapshot.json · news.json",
     "+ news_summary.php — on-demand AI",
     "gold-impact summary (whitelist · cache · cap)"],
    DEL_FC, DEL_EC)
 box(76, 15.0, 21.5, 12.6, "telegram_bot.py",
    ["candle-boundary auto-scans", "quality threshold + dedup",
     "alerts + LLM context line", "WS level-breach monitor"],
    DEL_FC, DEL_EC)
 # ── backtest / QA ────────────────────────────────────────────────────
 box(2.5, 3.2, 71.8, 8.0, "backtest/gpu_backtest.py — bar-by-bar replay through the SAME analyze_from_dataframes",
    ["realism (results-critical): next-open entries · raw-path resolution · spread charged · SL-first ties · close-only trail",
     "CuPy GPU / NumPy fallback · parquet cache · fed by providers.fetch_range (news off)"],
    BTS_FC, BTS_EC)
 box(77.5, 3.2, 20, 8.0, "test_all.py",
    ["359 tests · 22 modules", "no network (mocked)"], "#FAFAFA", "#9E9E9E")
 # ── arrows: short structural hops (solid dark) ───────────────────────
 arrow((10, 86.2), (14, 81.2), label="OHLCV", lx=-3.4, ly=0.2)          # oanda → providers
 arrow((27, 86.2), (24, 81.2), ls=(0, (3, 2)), lw=1.1)                  # fallbacks → providers
 arrow((44, 86.2), (48, 81.2))                                          # FF → news
 arrow((59.5, 86.2), (56, 81.2))                                        # RSS → news
 arrow((72, 86.2), (62, 81.2), label="LLM", lx=3.4, ly=0.6)             # deepseek → news
 arrow((19, 69.8), (19, 64.2), label="DataFrames (UTC, closed bars)", lx=0.2, ly=0.9)
 arrow((33.5, 57.5), (35.5, 55.5))                                      # engine → aggregator
 arrow((32.5, 46.0), (35.5, 48.0))                                      # factory → aggregator
 arrow((55, 69.8), (55, 63.1), label="warn-only context", lx=0, ly=1.0)
 arrow((86, 69.8), (86, 67.3), ls=(0, (3, 2)), lw=1.1)                  # config → core
 arrow((50, 40.5), (50, 38.8))                                          # core → report
 arrow((36, 34.5), (8.5, 27.95), conn="arc3,rad=0.12")                  # report → cli
 arrow((40, 33.2), (28, 27.95), conn="arc3,rad=0.06")                   # report → daemon
 arrow((60, 33.2), (58, 27.95))                                         # report → dashboard
 arrow((66, 34.5), (86.5, 27.95), conn="arc3,rad=-0.12")                # report → telegram
 # ── long-haul edges (thin dashed, routed through gutters) ────────────
 # telegram bot ↔ Telegram API — right-margin arc, clear of app_config
 arrow((95, 86.2), (95, 27.95), conn="arc3,rad=0.15", color=HAUL,
      lw=1.2, ls=(0, (4, 2)))
 ax.text(99.2, 57, "alerts / commands", fontsize=6.4, color=MUT,
        rotation=90, ha="center", va="center")
 # browser click → news_summary.php → DeepSeek
 # (oanda_daemon's order flow is noted inside its box — no long arc)
 arrow((66, 27.95), (73, 86.2), conn="arc3,rad=-0.18", color=HAUL,
      lw=1.2, ls=(0, (4, 2)),
      label="on-demand AI summary\n(browser click → PHP)", lx=2.5, ly=11.6)
 # providers.fetch_range → backtester — far-left margin channel
 arrow((1.5, 69.8), (1.5, 11.0), color=BTS_EC, lw=1.1, ls=(0, (3, 2)),
      label="fetch_range\n(history)", lx=3.6, ly=-1.2)
 # backtester replays the signal core — right gutter between boxes
 arrow((74.0, 11.6), (74.9, 40.3), color=HAUL, lw=1.3)
 ax.text(81.8, 38.9, "replays the exact signal code", fontsize=6.4,
        color=MUT, ha="center", va="center", zorder=5,
        bbox=dict(boxstyle="round,pad=0.15", fc="white", ec="none", alpha=0.9))
 out = os.path.join(os.path.dirname(os.path.abspath(__file__)), "architecture.png")
 fig.savefig(out, dpi=200, bbox_inches="tight", facecolor="white")
 print(f"saved {out}")
@@ -0,0 +1,165 @@
 <!DOCTYPE html>
 <html lang="en">
 <head>
 <meta charset="utf-8">
 <title>ForexBot2 — Beginner's Guide</title>
 <style>
  @page {
    size: A4;
    margin: 22mm 18mm;
    @bottom-center { content: "ForexBot2 Beginner's Guide — page " counter(page) " of " counter(pages); font-size: 8pt; color: #888; }
  }
  * { box-sizing: border-box; }
  body { font-family: "DejaVu Sans", Helvetica, Arial, sans-serif; font-size: 10.5pt; line-height: 1.65; color: #1c2733; margin: 0; }
  h1 { font-size: 21pt; margin: 0 0 2mm; color: #0d1b2a; }
  .subtitle { color: #5a6b7c; font-size: 10.5pt; margin-bottom: 8mm; }
  h2 { font-size: 13.5pt; color: #0d1b2a; border-bottom: 2px solid #d4a017; padding-bottom: 1.2mm; margin: 8mm 0 3mm; page-break-after: avoid; }
  h3 { font-size: 11pt; margin: 5mm 0 1.5mm; color: #23374d; page-break-after: avoid; }
  p { margin: 0 0 2.8mm; }
  table { border-collapse: collapse; width: 100%; margin: 3mm 0 4mm; font-size: 9.3pt; page-break-inside: avoid; }
  th { background: #0d1b2a; color: #fff; text-align: left; padding: 1.8mm 2.5mm; font-weight: 600; }
  td { border-bottom: 1px solid #dde4ea; padding: 1.8mm 2.5mm; vertical-align: top; }
  tr:nth-child(even) td { background: #f4f7fa; }
  table.breakable { page-break-inside: auto; }
  table.breakable tr { page-break-inside: avoid; }
  .signal-box { background: #f4f7fa; border: 1px solid #c9d4de; border-left: 4px solid #d4a017; border-radius: 2mm; padding: 3.5mm 4.5mm; font-family: "DejaVu Sans Mono", monospace; font-size: 8.6pt; line-height: 1.55; white-space: pre-wrap; margin: 3mm 0 4mm; page-break-inside: avoid; }
  .callout { background: #fdf6e3; border: 1px solid #e8d49a; border-left: 4px solid #d4a017; border-radius: 2mm; padding: 3mm 4.5mm; margin: 3mm 0 4mm; page-break-inside: avoid; }
  .callout.red { background: #fdf0ee; border-color: #eab4ac; border-left-color: #c0392b; }
  .callout.green { background: #eef7ee; border-color: #b3d9b3; border-left-color: #2e7d32; }
  .callout b:first-child { display: block; margin-bottom: 1mm; }
  .example { background: #eef2f6; border-radius: 2mm; padding: 3mm 4.5mm; margin: 2mm 0 4mm; page-break-inside: avoid; }
  .example b:first-child { color: #23374d; }
  ul, ol { margin: 0 0 3mm; padding-left: 6mm; }
  li { margin-bottom: 1.6mm; }
  code { font-family: "DejaVu Sans Mono", monospace; font-size: 8.8pt; background: #eef2f6; padding: 0.3mm 1.2mm; border-radius: 1mm; }
  .small { font-size: 8.5pt; color: #5a6b7c; }
  .disclaimer { border: 1.5px solid #c0392b; border-radius: 2mm; padding: 3.5mm 4.5mm; margin-top: 6mm; font-size: 9pt; page-break-inside: avoid; }
  .disclaimer b { color: #c0392b; }
 </style>
 </head>
 <body>
 <h1>ForexBot2 — Beginner's Guide</h1>
 <div class="subtitle">Everything explained in plain words. No experience needed. &nbsp;·&nbsp; July 2026</div>
 <h2>1 · What is this bot?</h2>
 <p>ForexBot2 is a computer program that watches the <b>price of gold</b> all day and all night. It studies the price charts using nine different mathematical checks. When the checks agree that the price is likely to move in one direction, it sends you a message on <b>Telegram</b> telling you:</p>
 <ul>
 <li>whether to <b>buy</b> or <b>sell</b>,</li>
 <li>at what price to get in,</li>
 <li>at what price to get out if the trade goes wrong (so your loss stays small),</li>
 <li>and how to lock in profit if the trade goes well.</li>
 </ul>
 <p>Most days the bot stays quiet for hours. <b>That is normal.</b> Quiet means "nothing worth trading right now." Never trade just because you are bored.</p>
 <h2>2 · Ten words you need to know</h2>
 <p>Every trading word used in this guide (and in the bot's messages) is explained here. Come back to this table whenever you see a word you don't remember.</p>
 <table class="breakable">
 <tr><th style="width:26%">Word</th><th>Plain meaning</th></tr>
 <tr><td><b>BUY (going "long")</b></td><td>You make money if the gold price goes <b>up</b>.</td></tr>
 <tr><td><b>SELL (going "short")</b></td><td>You make money if the gold price goes <b>down</b>. Yes, that's possible — your broker lets you sell gold you don't own and buy it back cheaper later.</td></tr>
 <tr><td><b>Entry</b></td><td>The price at which you open the trade.</td></tr>
 <tr><td><b>Stop loss (SL)</b></td><td>A safety brake. You tell your broker: "if the price reaches this level, close my trade automatically." It puts a hard limit on how much one trade can lose. <b>Every trade must have one.</b></td></tr>
 <tr><td><b>Take profit (TP)</b></td><td>The opposite of a stop loss: a level where the trade closes automatically <b>with a profit</b>. In this system it is only a rough guide — see section 6.</td></tr>
 <tr><td><b>Risk, or "R"</b></td><td>The amount of money you have chosen to lose if the trade fails. If you decide to risk $20, then <b>1R = $20</b>. A trade that wins "+2R" made you $40. This is how traders measure results.</td></tr>
 <tr><td><b>Trailing stop ("trail")</b></td><td>A stop loss that <b>follows the price</b> as it moves in your favor, but never moves backwards. Explained with numbers in section 6.</td></tr>
 <tr><td><b>Timeframe (TF)</b></td><td>The size of each candle on the chart. <b>M5</b> = each candle is 5 minutes, <b>M15</b> = 15 minutes, <b>H1</b> = 1 hour, <b>H4</b> = 4 hours. Bigger timeframe = slower, calmer trading.</td></tr>
 <tr><td><b>XAU/USD</b></td><td>Just the official symbol for "gold priced in US dollars."</td></tr>
 <tr><td><b>Spread</b></td><td>The tiny difference between the buying price and the selling price. It's your broker's fee, paid on every trade.</td></tr>
 </table>
 <h2>3 · Reading a bot message</h2>
 <p>Here is a typical signal, with plain-English notes below it:</p>
 <div class="signal-box">🔍 XAU/USD Signal Analysis  (H1)          ← gold, on the 1-hour chart
 🟢 BUY   Grade: A  |  64% confidence      ← what to do + how good the setup looks
 ✅ MTF aligned                            ← bigger and smaller charts agree
 📋 Indicators …                           ← the bot's 9 checks (you can skip this part)
 ━━━━━━━━━━━━━━━━━━━━
 🚀 TRADE SETUP                            ← ↓ THIS is the part you act on ↓
 📈 Direction: BUY
 ▶ Entry:  4,132.50                        ← open the trade near this price
 🛑 SL:     4,088.20                        ← your safety brake (stop loss)
 🎯 TP1:    4,176.80                        ← rough profit target (a guide, not a rule)
 🔄 Trail:  once price hits 4,159.08,
   trail SL $33.23 behind best price      ← how to lock in profit (see section 6)</div>
 <ul>
 <li><b>Grade (A+, A, B, C):</b> the bot's own mark for how clean the setup is. A+ is best. It never sends anything worse than C.</li>
 <li><b>Confidence and the indicator list:</b> extra detail for curious people. You don't need to understand it to trade the signal.</li>
 <li><b>⚠️ Warning lines:</b> things the bot noticed, like high volatility. They are "be aware," not "don't trade."</li>
 <li><b>The trade setup at the bottom is the only part you must act on.</b></li>
 </ul>
 <h2>4 · How much money to put in (the most important section)</h2>
 <p>You never put your whole account into one trade. You decide how much you are willing to <b>lose</b>, and work backwards from there. The rule:</p>
 <div class="callout"><b>Risk 1% of your account per trade. Never more.</b> If you have $2,000, that means one trade may lose at most $20. Losing trades are a normal part of the system — 1% means a bad week stings but never hurts you badly.</div>
 <div class="example"><b>Worked example.</b><br>
 Your account: <b>$2,000</b>. Your risk: 1% = <b>$20</b>. So for this trade, 1R = $20.<br>
 The signal says: Entry 4,132.50, Stop loss 4,088.20. The gap between them is <b>$44.30 per ounce</b> of gold.<br>
 How much gold to buy: $20 ÷ $44.30 = <b>0.45 ounces</b>.<br>
 Now, whatever happens: if the price falls to the stop loss, you lose about $20 — and no more. If the trade wins 2R, you make about $40.</div>
 <p class="small">Most broker apps have a built-in calculator: enter the stop-loss price and the amount you want to risk ($20), and it works out the position size for you. Use it — it avoids mistakes. (On many platforms gold is traded in "lots," where 0.01 lot = 1 ounce.)</p>
 <h2>5 · What to do when a signal arrives — step by step</h2>
 <ol>
 <li><b>Check the timeframe</b> in the first line. Only act on the one timeframe you have chosen to follow (see section 7). Ignore the rest.</li>
 <li><b>Work out your position size</b> using the 1% rule above. Do this before anything else.</li>
 <li><b>Open the trade</b> in your broker app, in the direction shown (BUY or SELL), at the current market price. Try to act soon after the message; if the price has already run far past the entry price, skip the trade — another signal will come.</li>
 <li><b>Set the stop loss immediately</b>, at the exact SL price in the message. Not "in your head" — in the app. Gold can move fast, and the stop loss is what protects you while you sleep.</li>
 <li><b>Then manage the trade with the trailing stop</b>, explained next.</li>
 </ol>
 <h2>6 · The trailing stop — how winners get paid</h2>
 <p>The take-profit price (TP1) in the message is only a guide. The system makes its money by <b>letting winning trades run</b> and following them with a moving safety brake. Every signal message tells you exactly when to start and how far behind to follow. Here is the idea with numbers:</p>
 <div class="example"><b>Example.</b> You bought gold at <b>4,132</b>, stop loss at <b>4,088</b> (risking $44 per ounce).<br>
 The message says: <i>"once price hits 4,159, trail SL $33 behind best price."</i><br><br>
 <b>Step 1 — wait.</b> Until gold reaches 4,159, do nothing. If it falls to 4,088 first, the stop loss closes the trade. That loss is normal and already budgeted.<br>
 <b>Step 2 — the price reaches 4,159.</b> Now move your stop loss up to 4,159 − 33 = <b>4,126</b>. From here on, the worst case is roughly break-even.<br>
 <b>Step 3 — keep following.</b> Each time the price makes a new high, move the stop up so it stays $33 below that high. Price climbs to 4,180? Stop goes to 4,147 — profit is now locked in. Price climbs to 4,210? Stop goes to 4,177.<br>
 <b>Step 4 — the price turns.</b> Eventually gold falls back $33 from its best point and hits your stop. The trade closes by itself, and you keep everything above your entry. <b>You never decide when to take profit — the trail decides.</b></div>
 <ul>
 <li>Only move the stop <b>up</b> (for a BUY). Never move it back down. For a SELL, everything is mirrored: the stop only moves down.</li>
 <li>Move it when a candle finishes, not every second. Checking once per candle (every hour, on the 1-hour timeframe) is enough.</li>
 <li>Many broker apps can do this automatically — look for a "trailing stop" order and set the distance from the message.</li>
 </ul>
 <h2>7 · Which timeframe should you follow?</h2>
 <p>The bot sends signals from several chart speeds. As a beginner, <b>follow only the 1-hour (H1) signals</b> and ignore the others. Here is why:</p>
 <table>
 <tr><th></th><th>M5 (5-minute)</th><th>M15 (15-minute)</th><th>H1 (1-hour) ★ start here</th></tr>
 <tr><td>Signals per week</td><td>about 10</td><td>about 10</td><td>about 3</td></tr>
 <tr><td>A trade usually lasts</td><td>a few hours</td><td>about half a day</td><td>half a day to a few days</td></tr>
 <tr><td>Time you have to react</td><td>1–2 minutes</td><td>up to 15 minutes</td><td>up to an hour</td></tr>
 <tr><td>Suits</td><td>full-time / automated</td><td>experienced, quick</td><td>anyone with a job and a life</td></tr>
 </table>
 <ul>
 <li><b>H1 is forgiving.</b> You have up to an hour to act on a message, and small delays barely matter. On the 5-minute signals, being two minutes late already costs real money — they are not for manual beginners.</li>
 <li><b>H1 trades are held longer</b> — often overnight, sometimes over days. That is normal. Your stop loss is working for you while you sleep.</li>
 <li><b>Never take the same signal on two timeframes.</b> They are usually the same gold move seen at different zoom levels. Taking both doubles your risk, not your profit.</li>
 </ul>
 <h2>8 · What to expect — read this before your first trade</h2>
 <ul>
 <li><b>You will lose about 4 trades out of every 10.</b> That is how the system is designed. Losses are small (1R each) and wins are bigger on average. The profit comes from many trades added together, not from any single one.</li>
 <li><b>Losing streaks happen.</b> Five or six losses in a row will happen at some point every year. It does not mean the system is broken. This is exactly why each loss is capped at 1% of your account.</li>
 <li><b>Expect a rough patch.</b> At some point in a normal year, your account will likely be around 10% below its best point before recovering. If a 10% dip would make you panic or bet bigger to "win it back," trade smaller (0.5% risk) or don't trade.</li>
 <li><b>Judge the system over months, not days.</b> On the 1-hour timeframe there are only ~3 trades a week. One bad week means nothing; look at results per quarter.</li>
 <li><b>Real results will be a little worse than tests.</b> The bot's numbers come from careful historical testing, but tests can't include your reaction speed or your broker's exact prices. Expect a bit less, not a bit more.</li>
 </ul>
 <h2>9 · The five rules</h2>
 <div class="callout red">
 <b>Break these and no signal system can save you.</b>
 <ol style="margin-bottom:0">
 <li><b>Risk 1% per trade, always.</b> Never increase the size to win back a loss.</li>
 <li><b>Every trade gets a stop loss in the app, at entry time.</b> No exceptions, no "mental" stops.</li>
 <li><b>One timeframe, one open trade at a time.</b></li>
 <li><b>Missed the entry? Let it go.</b> Chasing a price that already moved turns a good trade into a gamble.</li>
 <li><b>Only trade money you can afford to lose completely.</b> Never rent, savings you need, or borrowed money.</li>
 </ol>
 </div>
 <div class="callout green"><b>A good first month:</b> follow the H1 signals with the smallest position size your broker allows (or a demo account), write down every trade, and practice the trailing stop until it feels routine. Only then trade at 1% risk.</div>
 <div class="disclaimer"><b>⚠️ Risk warning.</b> This bot and guide are for education and research only. They are <b>not financial advice</b>. Trading gold and currencies involves a high risk of losing money, and most beginners lose. Past results — including tested results — do not guarantee future profits. Never trade money you cannot afford to lose. If you are unsure, speak to a licensed financial adviser.</div>
 </body>
 </html>
@@ -0,0 +1,154 @@
 # Centralized Configuration (config.json) — Design
 **Date:** 2026-07-21
 **Status:** Approved (user confirmed all four scoping decisions)
 ## Goal
 Move deployment settings — Telegram credentials, data-provider API keys and
 selection, scanned pairs, scan timeframes, per-pair base blocked hours, alert
 quality boot default, log level — into a single gitignored `config.json` at the
 repo root, loaded once at startup.
 **Hard constraint:** zero change to signal logic, indicator weights, thresholds,
 or gates. With an unedited `config.json` (or none at all), every produced
 `PlatformConfig` must be byte-identical to today's factory output. This is
 test-enforced (see Testing).
 ## Scoping decisions (user-confirmed 2026-07-21)
 1. **Secrets live in config.json** (gitignored). A committed
   `config.json.example` carries placeholders. Env vars still override.
   The hardcoded credential fallbacks in `config.py` and `telegram_bot.py`
   are **deleted** — on a machine with neither config.json nor env vars, the
   bot refuses to start with an explicit error instead of silently using
   committed credentials. (Recommend rotating the Telegram token and Twelve
   Data key after merge; both are exposed in git history.)
 2. **Multi-provider is structure only**: a `data_providers` section with named
   provider blocks and an `active` selector. Only Twelve Data is implemented.
   No failover logic (nothing to fail over to yet).
 3. **Blocked hours: base list per pair only.** The validated per-timeframe
   deltas (M5 appends hour 9; M15 SELL-block 15/17 UTC) stay in
   `pair_signals/xauusd.py` next to their validation evidence. The overlay is
   applied **before** those deltas and updates **both** mirrored copies
   (`xau_strategy.blocked_hours_utc` and `backtest.blocked_hours_utc`) so they
   cannot drift.
 4. **Extra keys included:** scan timeframes, alert min quality (boot default),
   log level. Everything else (indicator params, RR geometry, M5 gates,
   CP knobs, news config) deliberately stays code-only.
 ## File schema
 `config.json` (gitignored; `config.json.example` committed with placeholders
 for secrets and today's tuned values for everything else):
 ```json
 {
  "telegram": {
    "token": "<bot token>",
    "chat_id": "<owner chat id>",
    "channel_id": "<-100… or empty string to disable the channel feed>",
    "alert_min_quality": "C"
  },
  "data_providers": {
    "active": "twelve_data",
    "twelve_data": { "api_key": "<key>" }
  },
  "scanning": {
    "pairs": ["XAU/USD"],
    "timeframes": ["M5", "M15", "H1", "H4"]
  },
  "pairs": {
    "XAU/USD": { "blocked_hours_utc": [0, 1, 2, 10, 11, 13, 16, 21] }
  },
  "log_level": "INFO"
 }
 ```
 Timeframe strings use the `Timeframe` enum names (`M5`, `M15`, `H1`, `H4`, …).
 Pair keys use canonical symbols (`XAU/USD`).
 ## Architecture
 ### New module: `app_config.py`
 Single responsibility: read + validate `config.json`, expose typed accessors,
 apply the pair-level overlay. No signal knowledge.
 - `AppConfig` dataclass: `telegram` (token, chat_id, channel_id,
  alert_min_quality), `data_providers` (active name + per-provider dicts),
  `scan_pairs`, `scan_timeframes`, `pair_overrides`
  (symbol → {blocked_hours_utc}), `log_level`.
 - `load_app_config(path=REPO_ROOT/"config.json") -> AppConfig` — module-level
  cached (loaded once per process). Missing file → `AppConfig` of pure
  defaults + one warning log. Malformed JSON, unknown top-level/section keys,
  unknown pair symbols, hours outside 0–23, unknown timeframe names, unknown
  `active` provider → raise `AppConfigError` with a named message (fail fast
  at boot).
 - Precedence, applied inside the accessors:
  **env var → config.json → code default**. Secrets (`TELEGRAM_TOKEN`,
  `TELEGRAM_CHAT_ID`, `TELEGRAM_CHANNEL_ID`, `TWELVE_DATA_API_KEY`) have **no
  code default** after this change.
 - `apply_pair_overrides(cfg: PlatformConfig, app: AppConfig) -> PlatformConfig`
  — sets the pair's base `blocked_hours_utc` on **both**
  `cfg.xau_strategy.blocked_hours_utc` and `cfg.backtest.blocked_hours_utc`,
  plus `cfg.scan_pairs` / `cfg.scan_timeframes`. Called by the pair factory
  (see below). A no-op when the pair has no override block.
 ### Wiring points (all consumers)
 | Consumer | Change |
 |---|---|
 | `pair_signals/xauusd.py` | `make_config` calls `apply_pair_overrides` **before** the per-TF deltas (so M5 still appends hour 9 to whatever base the JSON sets, and mirrors stay in sync). |
 | `data_provider.py` | API key resolution becomes `app_config` accessor for the **active** provider (env `TWELVE_DATA_API_KEY` still wins). `config.twelve_data_api_key` field and its hardcoded default are deleted. Missing key keeps today's behavior: provider constructed with `_td=None`, errors on first fetch. |
 | `telegram_bot.py` | `TOKEN`, `CHAT_ID`, `CHANNEL_ID` come from `app_config` (hardcoded fallbacks deleted; missing token/chat_id → explicit startup error). `_alert_min_quality` boot default and `logging` level come from `app_config`. `SCAN_PAIRS` (today built from a bare `PlatformConfig()`) reads `app_config.scan_pairs` directly; `apply_pair_overrides` also stamps `cfg.scan_pairs`/`cfg.scan_timeframes` so the dataclass field stays truthful for any other reader. |
 | `main.py` | No structural change — it uses the factory, which now applies the overlay. |
 | `backtest/gpu_backtest.py` | No structural change — it uses the factory too, so an edited base blocked-hours list flows into replays identically to live (parity by construction). Data fetches pick up the provider key via `data_provider.py`. |
 | `config.py` | `twelve_data_api_key` field removed. Everything else untouched. |
 ### Multi-provider structure
 `data_provider.py` gains a minimal registry:
 `PROVIDER_REGISTRY = {"twelve_data": ForexDataProvider}`. `active` selects
 from it; an unknown name fails at load. Adding a provider later = one config
 block + one adapter class registered here. No shared base class extracted now
 (YAGNI — with one provider there is nothing to abstract over yet).
 ## Error handling summary
 | Condition | Behavior |
 |---|---|
 | No config.json | Defaults + warning. Secrets must come from env or startup fails (bot) / fetch fails (provider). |
 | Malformed JSON / bad values / unknown keys | `AppConfigError` at load, process exits with named message. |
 | Missing Telegram token or chat_id at bot boot | Explicit startup error naming the missing key and both places it can be set. |
 | Missing data-provider key | Same as today: provider builds, first fetch errors. |
 ## Migration
 - Create the local `config.json` populated with the **current live values**
  (the ones being deleted from source) so the running bot keeps working
  unchanged. `.gitignore` gains `config.json`.
 - Commit `config.json.example` with placeholders.
 - CLAUDE.md: update the env-var table, repository map, and config section.
 - Post-merge recommendation (user action): rotate the Telegram token and
  Twelve Data key, since both are in git history.
 ## Testing (`test_all.py`, new module)
 1. **Parity pin (the critical one):** a config.json containing the example
   defaults, and no config.json at all, both produce factory output identical
   to today's (full dataclass equality across all four TFs).
 2. Overlay mapping: JSON blocked hours land on **both** mirrored fields; M5
   factory still appends hour 9 on top of a custom base.
 3. Precedence: env var beats config.json beats default, per key.
 4. Validation: malformed JSON, bad hour, unknown pair/timeframe/provider each
   raise `AppConfigError`.
 5. Missing-file behavior: defaults + no crash.
 6. Existing factory-equivalence test stays green (equivalence is now defined
   over factory-with-overlay vs inline-config-with-overlay).
 ## Out of scope
 Hot reload (config is read once at boot), a second concrete provider,
 failover, moving any tuned strategy value beyond the base blocked-hours list,
 news/indicator/geometry settings in JSON.
@@ -0,0 +1,142 @@
 <!DOCTYPE html>
 <html lang="en">
 <head>
 <meta charset="utf-8">
 <title>ForexBot2 — User Guide</title>
 <style>
  @page {
    size: A4;
    margin: 22mm 18mm;
    @bottom-center { content: "ForexBot2 User Guide — page " counter(page) " of " counter(pages); font-size: 8pt; color: #888; }
  }
  * { box-sizing: border-box; }
  body { font-family: "DejaVu Sans", Helvetica, Arial, sans-serif; font-size: 10pt; line-height: 1.55; color: #1c2733; margin: 0; }
  h1 { font-size: 21pt; margin: 0 0 2mm; color: #0d1b2a; }
  .subtitle { color: #5a6b7c; font-size: 10.5pt; margin-bottom: 8mm; }
  h2 { font-size: 13.5pt; color: #0d1b2a; border-bottom: 2px solid #d4a017; padding-bottom: 1.2mm; margin: 8mm 0 3mm; page-break-after: avoid; }
  h3 { font-size: 11pt; margin: 5mm 0 1.5mm; color: #23374d; page-break-after: avoid; }
  p { margin: 0 0 2.5mm; }
  table { border-collapse: collapse; width: 100%; margin: 3mm 0 4mm; font-size: 9pt; page-break-inside: avoid; }
  th { background: #0d1b2a; color: #fff; text-align: left; padding: 1.8mm 2.5mm; font-weight: 600; }
  td { border-bottom: 1px solid #dde4ea; padding: 1.6mm 2.5mm; }
  tr:nth-child(even) td { background: #f4f7fa; }
  .signal-box { background: #f4f7fa; border: 1px solid #c9d4de; border-left: 4px solid #d4a017; border-radius: 2mm; padding: 3.5mm 4.5mm; font-family: "DejaVu Sans Mono", monospace; font-size: 8.4pt; line-height: 1.5; white-space: pre-wrap; margin: 3mm 0 4mm; page-break-inside: avoid; }
  .callout { background: #fdf6e3; border: 1px solid #e8d49a; border-left: 4px solid #d4a017; border-radius: 2mm; padding: 3mm 4.5mm; margin: 3mm 0 4mm; page-break-inside: avoid; }
  .callout.red { background: #fdf0ee; border-color: #eab4ac; border-left-color: #c0392b; }
  .callout b:first-child { display: block; margin-bottom: 1mm; }
  ul, ol { margin: 0 0 3mm; padding-left: 6mm; }
  li { margin-bottom: 1.4mm; }
  code { font-family: "DejaVu Sans Mono", monospace; font-size: 8.6pt; background: #eef2f6; padding: 0.3mm 1.2mm; border-radius: 1mm; }
  .small { font-size: 8.5pt; color: #5a6b7c; }
  .disclaimer { border: 1.5px solid #c0392b; border-radius: 2mm; padding: 3.5mm 4.5mm; margin-top: 6mm; font-size: 9pt; page-break-inside: avoid; }
  .disclaimer b { color: #c0392b; }
 </style>
 </head>
 <body>
 <h1>ForexBot2 — User Guide</h1>
 <div class="subtitle">How to read, act on, and survive the signals &nbsp;·&nbsp; July 2026</div>
 <h2>1 · What the bot does</h2>
 <p>ForexBot2 analyzes <b>gold (XAU/USD)</b> around the clock on several chart timeframes at once. On every candle close it scores nine technical indicators (trend, momentum, volatility, divergences, candlestick patterns), adapts the weighting to the current market regime (trending / ranging / transitioning), checks that higher timeframes agree, adjusts for news and economic-calendar events, and — only when everything lines up — sends a <b>signal with a complete trade plan</b> to Telegram: direction, entry, stop loss, target, and a trailing-stop management plan.</p>
 <p>Most of the time it sends nothing. Silence means no edge was found; that is the bot working correctly, not a malfunction.</p>
 <h2>2 · Anatomy of a signal</h2>
 <div class="signal-box">🔍 XAU/USD Signal Analysis  (M15)
 🕐 2026-07-10 14:00 UTC
 💰 Close: 4,132.50
 📊 Regime: Trending Up | Vol: normal
 ━━━━━━━━━━━━━━━━━━━━
 🟢 BUY   🅰 A  |  Score: +38.2  |  64% conf
 ✅ MTF aligned
 📈 MTF: M5 ✅  H1 ✅
 📋 Indicators … (9 rows)
 ⚠️ EXTREME volatility — wider stops needed
 ━━━━━━━━━━━━━━━━━━━━
 🚀 TRADE SETUP
 📈 Direction: BUY
 ▶ Entry:  4,132.50
 🛑 SL:     4,103.10   ($29.40 / 294 pips)
 🎯 TP1:    4,161.90   ($29.40 / 294 pips)
 ⚖️ RR:     1:1.0
 🔄 Trail: once price hits 4,150.14 (+0.6R),
   trail SL $22.05 behind best price —
   TP1 is a guide, let it run</div>
 <table>
 <tr><th style="width:28%">Field</th><th>What it means</th></tr>
 <tr><td><b>Signal + grade</b></td><td>Direction (BUY / SELL / STRONG_*) plus a quality grade <b>A+ → F</b>. A trade setup is only attached at grade C or better. Higher grades = more indicator agreement and cleaner regime.</td></tr>
 <tr><td><b>Score / confidence</b></td><td>The regime-weighted composite of all nine indicators. Bigger absolute score = stronger consensus.</td></tr>
 <tr><td><b>MTF aligned</b></td><td>The two confirmation timeframes agree with the trade direction. Signals without alignment are filtered out on M5.</td></tr>
 <tr><td><b>Warnings ⚠️</b></td><td>Context, not veto. Backtesting shows warned trades perform about the same — strong momentum often triggers warnings.</td></tr>
 <tr><td><b>Market context 📰</b></td><td>An AI-written one-liner summarizing today's macro backdrop (headlines + high-impact releases) and its likely pull on gold. Informational only — it never blocks or grades a signal.</td></tr>
 <tr><td><b>SL / TP1 / RR</b></td><td>Stop loss sits behind the nearest swing level with an ATR buffer. TP1 is the nearest opposing level. <b>1R</b> = the entry-to-SL distance = the amount you risk.</td></tr>
 <tr><td><b>Trail plan 🔄</b></td><td>The real exit. Once price moves +0.6R in your favor, move the stop to trail 0.75R behind the best price and update it at each candle close. <b>TP1 is advisory — let winners run.</b></td></tr>
 </table>
 <h2>3 · Acting on a signal — step by step</h2>
 <ol>
 <li><b>Size the position first.</b> Decide your risk per trade — <b>0.5–1% of account, never more</b>. Position size = (account × risk%) ÷ stop distance in $. The stop distance varies per signal, so the size changes every trade; the dollar risk must not.</li>
 <li><b>Enter promptly at market.</b> All backtested results assume entry at the open of the candle after the signal. On M5, minutes matter (see §4). If you are very late — price already moved a large part of the way to the arm level — skip the trade. There is always another signal.</li>
 <li><b>Place the stop loss immediately</b> at the given SL. No mental stops — gold gaps and spikes.</li>
 <li><b>Manage with the trail, not TP1.</b> When price reaches the arm level printed in the message, start trailing the stop the stated $ distance behind the best price, tightening only at candle close. This is what produced the backtested results; taking fixed TP1 profits gives most of the edge away.</li>
 <li><b>One signal, one trade.</b> The M5, M15 and H1 books confirm off each other and are usually long or short gold at the same time. Taking the same move on multiple timeframes multiplies risk, not diversification. Pick your timeframe (§4) and ignore the others.</li>
 </ol>
 <h2>4 · Choosing your timeframe</h2>
 <p>Twelve-month backtest (Jul 2025 – Jul 2026, real spreads, entry at next candle open):</p>
 <table>
 <tr><th></th><th>M5</th><th>M15</th><th>H1</th></tr>
 <tr><td>Trades / week</td><td>~10</td><td>~9.5</td><td>~3</td></tr>
 <tr><td>Total result (12 mo)</td><td>+135.6R</td><td>+67.2R</td><td>+49.9R</td></tr>
 <tr><td>Win rate / profit factor</td><td>59.7% / 1.81</td><td>55.8% / 1.40</td><td>59.4% / 1.94</td></tr>
 <tr><td>Typical (median) hold</td><td>2.4 h</td><td>6 h</td><td>12.5 h</td></tr>
 <tr><td>90% of trades close within</td><td>16 h</td><td>40 h</td><td>3.7 days</td></tr>
 <tr><td>Typical stop distance</td><td>~$19</td><td>~$29</td><td>~$44</td></tr>
 <tr><td>Worst drawdown</td><td>11.2R</td><td>9.4R</td><td>9.5R</td></tr>
 </table>
 <ul>
 <li><b>M5 — intraday, execution-critical.</b> Highest total edge, but stops are tight: a couple of dollars of slippage or a late entry eats a meaningful share of every trade. Suitable only if you can act within a minute or two of the alert (or automate). Trades open and close within a session; you sleep flat.</li>
 <li><b>M15 — the middle path.</b> Same frequency as M5, wider stops, up to 15 minutes to act. About half your trades will still be open overnight.</li>
 <li><b>H1 — swing trading around a normal life.</b> Best quality per trade and very forgiving of slow manual execution (you have up to an hour). The costs: positions held for days, through news and overnight sessions, and a <b>lumpy</b> equity curve — one quarter of the test year was negative and most of the profit came in a single quarter. Judge it on quarters, not weeks.</li>
 </ul>
 <div class="callout"><b>If you trade manually from the Telegram alerts, H1 (or M15) is the honest choice.</b> M5's backtested edge assumes near-instant fills that a human copying signals rarely achieves.</div>
 <h2>5 · What to expect (and what not to)</h2>
 <ul>
 <li><b>You will lose ~40% of trades.</b> A 56–60% win rate means losing streaks of 5+ are routine over a year. The system's edge is the average across hundreds of trades, not any single one.</li>
 <li><b>Stops hit fast, winners take time.</b> Losing trades die in roughly half the time winners take. An early stop-out is the system doing its job.</li>
 <li><b>Drawdowns of ~10R are normal.</b> At 1% risk per trade, expect to be ~10% down from a peak at some point in a normal year. This is why risking more than 1% per trade is how accounts die.</li>
 <li><b>Backtest ≠ live.</b> Results above charge real spreads and realistic fills, but cannot model your personal latency, broker slippage, or requotes. Assume live results will be somewhat worse — especially on M5.</li>
 <li><b>Quiet periods are normal.</b> Filters block bad hours, bad regimes and misaligned timeframes. Days can pass without a signal.</li>
 </ul>
 <h2>6 · Rules that keep you alive</h2>
 <div class="callout red">
 <b>The five rules</b>
 <ol style="margin-bottom:0">
 <li>Risk a fixed 0.5–1% per trade. Never "make it back" by sizing up after losses.</li>
 <li>Hard stop loss in the market on every trade, placed at entry time.</li>
 <li>One timeframe, one trade at a time. No stacking, no pyramiding onto bot signals.</li>
 <li>Missed the entry? Skip it. Chasing converts a positive-edge trade into a coin flip.</li>
 <li>Evaluate performance monthly at the earliest, quarterly for H1. Never after a bad day.</li>
 </ol>
 </div>
 <h2>7 · Bot commands (owner)</h2>
 <table>
 <tr><th style="width:30%">Command</th><th>What it does</th></tr>
 <tr><td><code>/start</code></td><td>Show the button menu (manual scans per timeframe, Scan All TFs).</td></tr>
 <tr><td><code>/scan</code></td><td>Run an immediate analysis of the primary timeframe.</td></tr>
 <tr><td><code>/autoscan</code></td><td>Toggle automatic scanning (M5 / M15 / H1 / H4 at each candle close).</td></tr>
 <tr><td><code>/status</code></td><td>Uptime, auto-scan state, WebSocket price-monitor state.</td></tr>
 <tr><td><code>/news</code></td><td>Net XAU/USD news sentiment score (informational only — never gates signals).</td></tr>
 <tr><td><code>/cache</code> · <code>/clearcache</code></td><td>Inspect / reset the market-data cache.</td></tr>
 <tr><td><code>/help</code></td><td>Command reference.</td></tr>
 </table>
 <p class="small">The bot also streams live prices over WebSocket and fires an immediate scan when price breaches a key level (swing S/R, Bollinger, Keltner, EMA-200). Commands respond only to the configured owner chat; subscribers receive the signal feed via the channel.</p>
 <div class="disclaimer"><b>⚠️ Risk disclaimer.</b> This bot and guide are for educational and research purposes only and are <b>not financial advice</b>. Trading forex, gold and CFDs on margin carries a high risk of loss; past (and backtested) performance does not guarantee future results. Do your own analysis and never risk money you cannot afford to lose.</div>
 </body>
 </html>
@@ -13,7 +13,7 @@ Design philosophy:
 import logging
 from dataclasses import dataclass, field
-from typing import Dict, List, Optional, Tuple
+from typing import Dict, List, Tuple
 import numpy as np
 import pandas as pd
@@ -85,13 +85,18 @@ class TechnicalEngine:
        # ── Compute all indicators ─────────────────────────────────
        disabled = set(self.config.disabled_indicators or [])
        # ATR first: other indicators (EMA ribbon, candlestick) use it
        if "atr"         not in disabled: self._compute_atr(df, result)
        if "ema_trend"   not in disabled: self._compute_ema_signals(df, result)
        if "rsi"         not in disabled: self._compute_rsi(df, result)
        if "bollinger"   not in disabled: self._compute_bollinger(df, result)
        if "adx"         not in disabled: self._compute_adx(df, result)
        if "atr"         not in disabled: self._compute_atr(df, result)
        # MACD always runs — divergence needs the histogram — but the
        # scored pillar must still honour disabled_indicators.
        self._compute_macd(df, result)
        if "macd" in disabled:
            result.indicators.pop("macd", None)
        if "roc"         not in disabled: self._compute_roc(df, result)
        if "keltner"     not in disabled: self._compute_keltner(df, result)
@@ -240,6 +245,12 @@ class TechnicalEngine:
            signal = SignalStrength.NEUTRAL
            conf = 40
        # Partial alignment (plain BUY/SELL, 2-3 of 4 checks) carries no
        # edge on M5 — squash to NEUTRAL when the research knob is on.
        if c.ema_partial_neutral and signal in (SignalStrength.BUY, SignalStrength.SELL):
            signal = SignalStrength.NEUTRAL
            conf = 40
        crossover_desc = ""
        if len(ema_fast) >= 3:
            prev_f, prev_m = ema_fast.iloc[-2], ema_medium.iloc[-2]
@@ -478,11 +489,14 @@ class TechnicalEngine:
        tr = pd.concat([tr1, tr2, tr3], axis=1).max(axis=1)
        # Directional Movement
-        plus_dm = high.diff()
+        plus_dm_raw = high.diff()
-        minus_dm = -low.diff()
+        minus_dm_raw = -low.diff()
-        plus_dm = plus_dm.where((plus_dm > minus_dm) & (plus_dm > 0), 0.0)
+        # Compare raw vs raw: sequential filtering would zero +DM on a tie
-        minus_dm = minus_dm.where((minus_dm > plus_dm) & (minus_dm > 0), 0.0)
+        # but keep -DM (compared against the already-zeroed +DM) — a
        # systematic bearish nudge. Wilder zeroes both on ties.
        plus_dm = plus_dm_raw.where((plus_dm_raw > minus_dm_raw) & (plus_dm_raw > 0), 0.0)
        minus_dm = minus_dm_raw.where((minus_dm_raw > plus_dm_raw) & (minus_dm_raw > 0), 0.0)
        atr = tr.ewm(alpha=1/c.adx_period, min_periods=c.adx_period, adjust=False).mean()
        plus_di = 100 * (plus_dm.ewm(alpha=1/c.adx_period, min_periods=c.adx_period, adjust=False).mean() / atr)
@@ -630,15 +644,10 @@ class TechnicalEngine:
                signal = SignalStrength.SELL
                conf = min(70, 50 + roc_abs * 10)
                desc = f"ROC {roc_val:.2f}% — Falling momentum"
        elif roc_prev <= 0 < roc_val:
            signal = SignalStrength.BUY
            conf = 65
            desc = f"ROC crossed above zero (+{roc_val:.2f}%)"
        elif roc_prev >= 0 > roc_val:
            signal = SignalStrength.SELL
            conf = 65
            desc = f"ROC crossed below zero ({roc_val:.2f}%)"
        else:
            # Zero-crosses land in the branches above by definition (a
            # cross up is positive and rising, a cross down negative and
            # falling) — only flat/fading momentum reaches here.
            signal = SignalStrength.NEUTRAL
            conf = 30
            desc = f"ROC {roc_val:+.2f}% — No clear momentum"
@@ -766,20 +775,34 @@ class TechnicalEngine:
            price_window = close.iloc[-lookback:]
            ind_window = indicator_series.iloc[-lookback:]
            # A price swing and an indicator swing only form one divergence
            # observation if they mark the SAME market turn — require the
            # swing bars to line up within a small tolerance, otherwise
            # unrelated wiggles get compared and the signal is noise.
            def _aligned(price_pts, ind_pts, tol=2):
                pairs = []
                for p_idx, p_val in price_pts:
                    best = min(ind_pts, key=lambda ip: abs(ip[0] - p_idx), default=None)
                    if best is not None and abs(best[0] - p_idx) <= tol:
                        pairs.append(((p_idx, p_val), best))
                return pairs
            # Bullish divergence: lower price low but higher indicator low
            price_lows = self._find_swing_points(price_window, mode="low")
            ind_lows = self._find_swing_points(ind_window, mode="low")
-
+            low_pairs = _aligned(price_lows, ind_lows)
-            if len(price_lows) >= 2 and len(ind_lows) >= 2:
+            if len(low_pairs) >= 2:
-                if price_lows[-1][1] < price_lows[-2][1] and ind_lows[-1][1] > ind_lows[-2][1]:
+                (p1, i1), (p2, i2) = low_pairs[-2], low_pairs[-1]
                if p2[1] < p1[1] and i2[1] > i1[1]:
                    divergences.append(f"Bullish {name} divergence (reversal warning)")
            # Bearish divergence: higher price high but lower indicator high
            price_highs = self._find_swing_points(price_window, mode="high")
            ind_highs = self._find_swing_points(ind_window, mode="high")
-
+            high_pairs = _aligned(price_highs, ind_highs)
-            if len(price_highs) >= 2 and len(ind_highs) >= 2:
+            if len(high_pairs) >= 2:
-                if price_highs[-1][1] > price_highs[-2][1] and ind_highs[-1][1] < ind_highs[-2][1]:
+                (p1, i1), (p2, i2) = high_pairs[-2], high_pairs[-1]
                if p2[1] > p1[1] and i2[1] < i1[1]:
                    divergences.append(f"Bearish {name} divergence (reversal warning)")
        if divergences:
@@ -804,7 +827,10 @@ class TechnicalEngine:
    def _compute_structure_sr(self, df: pd.DataFrame, result: TechnicalAnalysis):
        """Detect swing high/low structure levels for S/R and trade setup."""
-        lookback = min(self.config.swing_sr_lookback, len(df) - 1)
+        base_lookback = self.config.swing_sr_lookback
        if result.timeframe in ("1min", "5min"):
            base_lookback = max(base_lookback, 120)
        lookback = min(base_lookback, len(df) - 1)
        if lookback < 10:
            return
@@ -934,7 +960,7 @@ class TechnicalEngine:
        rsi_val = rsi_ind.value
        regime = result.market_regime
-        if regime in ("trending_up", "trending"):
+        if regime == "trending_up":
            if 38 <= rsi_val <= 55:
                signal = SignalStrength.STRONG_BUY
                conf = 75
@@ -980,6 +1006,24 @@ class TechnicalEngine:
                desc = f"RSI {rsi_val:.1f} — Overbought in downtrend, possible reversal"
            else:
                return
        elif regime in ("transitioning", "trending"):
            # Trend being born (transitioning) or strong ADX with EMAs
            # unaligned (mixed "trending") — in both, RSI can't tell
            # direction. Treating mixed-"trending" as an uptrend here was
            # a systematic bullish bias (23 BUY / 1 SELL, -6.8R on the
            # 6yr M15 book). Only extremes carry information.
            if rsi_val >= 70:
                signal = SignalStrength.SELL
                conf = 50
                desc = f"RSI {rsi_val:.1f} — Overbought while trend forming"
            elif rsi_val <= 30:
                signal = SignalStrength.BUY
                conf = 50
                desc = f"RSI {rsi_val:.1f} — Oversold while trend forming"
            else:
                signal = SignalStrength.NEUTRAL
                conf = 30
                desc = f"RSI {rsi_val:.1f} — No RSI edge while trend forms"
        else:  # ranging
            if rsi_val >= 70:
                signal = SignalStrength.SELL
@@ -1041,10 +1085,15 @@ class TechnicalEngine:
        if self.config.regime_weights_override:
            regime_weights = self.config.regime_weights_override
        else:
            # NOTE: ROC is directionally redundant with MACD/Keltner/EMA
            # (0.8-0.94 correlation), but zeroing its weight was backtested
            # 2026-07 and LOST 13R — the redundant vote effectively raises
            # the composite bar and the thresholds are calibrated with it.
            # Don't remove without retuning score thresholds.
            _trending = {
                "ema_trend": 2.0, "rsi": 1.0, "macd": 2.0,
                "bollinger": 0.5, "keltner": 1.5, "roc": 1.5,
-                "candlestick": 0.8, "divergence": 2.0, "adx": 0.8,
+                "candlestick": 0.8, "divergence": 1.0, "adx": 0.8,
            }
            regime_weights = {
                "trending_up": _trending,
@@ -0,0 +1,195 @@
 """
 ForexBot2 — DeepSeek LLM client
 ================================
 Thin requests-based client for DeepSeek's OpenAI-compatible chat API,
 used by `news_engine` for display/warning-only enrichment (headline
 sentiment, event surprise direction, market summary). Signals are NEVER
 gated on LLM output — every caller has a non-LLM fallback and treats a
 `None` return as "use the fallback".
 Design constraints:
 - One attempt per call, no retries: news runs inside scan cycles; the
  news engine's 600 s context cache is the natural retry cadence.
 - A UTC-day request cap bounds spend even if a bug causes call storms.
 - Cost: DeepSeek caches identical prompt PREFIXES server-side (cache-hit
  input is ~50x cheaper), so callers pass a STATIC system prompt (no
  timestamps or interpolation) and put all variable content in the user
  payload.
 This is a module-level singleton (`get_llm_client`) because the daily
 cap counter must survive the per-scan reconstruction of `NewsEngine`.
 """
 import datetime as _dt
 import json
 import logging
 import threading
 import time
 from typing import Any, Dict, Optional, Tuple
 import requests
 from app_config import deepseek_api_key, get_app_config
 logger = logging.getLogger(__name__)
 DEEPSEEK_URL = "https://api.deepseek.com/chat/completions"
 DEFAULT_MODEL = "deepseek-v4-flash"
 # 8s timed out on a measured cold-prefix-cache headline batch (2026-08-03);
 # warm-cache calls come back well under that. 12s buys cold-start headroom
 # while still bounding worst-case scan latency (≤3 calls per cache window).
 DEFAULT_TIMEOUT = 12.0
 DEFAULT_DAILY_CAP = 500
 class DeepSeekClient:
    """JSON-in/JSON-out chat wrapper. All failures return None."""
    def __init__(
        self,
        api_key: str,
        model: str = DEFAULT_MODEL,
        timeout: float = DEFAULT_TIMEOUT,
        daily_cap: int = DEFAULT_DAILY_CAP,
        enabled: bool = True,
    ):
        self._api_key = api_key or ""
        self._model = model
        self._timeout = timeout
        self._daily_cap = daily_cap
        self._enabled = enabled
        self._session = requests.Session()
        self._session.headers.update({
            "Content-Type": "application/json",
            "Authorization": f"Bearer {self._api_key}",
        })
        self._lock = threading.Lock()
        self._count_day: Optional[_dt.date] = None
        self._count = 0
        self._warned: set = set()
        # Health stamps for status displays: last successful call and
        # last failure (ts, kind). Read-only for consumers.
        self.last_ok: float = 0.0
        self.last_error: Optional[Tuple[float, str]] = None
    @property
    def available(self) -> bool:
        return self._enabled and bool(self._api_key)
    def _warn_once(self, kind: str, msg: str, *args: Any) -> None:
        if kind in self._warned:
            logger.debug(msg, *args)
        else:
            self._warned.add(kind)
            logger.warning(msg, *args)
    def _under_cap(self) -> bool:
        """Count a request against the UTC-day cap; False = don't call."""
        today = _dt.datetime.now(_dt.timezone.utc).date()
        with self._lock:
            if self._count_day != today:
                self._count_day = today
                self._count = 0
                self._warned.discard("cap")
            if self._count >= self._daily_cap:
                self._warn_once(
                    "cap", "DeepSeek daily request cap (%d) exhausted — "
                    "falling back to keyword analysis until UTC midnight",
                    self._daily_cap)
                return False
            self._count += 1
            return True
    def chat_json(self, system_prompt: str,
                  user_payload: Dict[str, Any]) -> Optional[Dict[str, Any]]:
        """One chat call; returns the model's JSON object or None."""
        if not self.available or not self._under_cap():
            return None
        body = {
            "model": self._model,
            "messages": [
                {"role": "system", "content": system_prompt},
                {"role": "user",
                 "content": json.dumps(user_payload, sort_keys=True)},
            ],
            "response_format": {"type": "json_object"},
            "temperature": 0,
            # V4-Flash is a hybrid thinking model; without this flag it
            # burns hundreds of reasoning tokens per call — measured
            # 2026-08-03: a 25-headline batch blew straight through the
            # timeout AND could exhaust max_tokens before the answer,
            # returning empty content. The doctrine + few-shot examples
            # in the system prompts carry the quality instead.
            "thinking": {"type": "disabled"},
            # Must fit the largest reply: a full sentiment batch is
            # ~10 tokens per headline; 1000 truncated mid-JSON on a real
            # RSS cycle (2026-08-03).
            "max_tokens": 2000,
        }
        try:
            resp = self._session.post(
                DEEPSEEK_URL, json=body, timeout=self._timeout)
        except requests.RequestException as e:
            self.last_error = (time.time(), "network")
            self._warn_once("network", "DeepSeek call failed: %s", e)
            return None
        if resp.status_code != 200:
            self.last_error = (time.time(), f"http_{resp.status_code}")
            self._warn_once(
                f"http_{resp.status_code}",
                "DeepSeek HTTP %s: %s", resp.status_code, resp.text[:200])
            return None
        try:
            parsed = self._parse_response(resp.json())
        except ValueError:
            parsed = None
        if parsed is None:
            self.last_error = (time.time(), "parse")
            self._warn_once("parse", "DeepSeek response was not the "
                            "expected JSON object — ignoring")
        else:
            self.last_ok = time.time()
        return parsed
    @staticmethod
    def _parse_response(body: Any) -> Optional[Dict[str, Any]]:
        """Extract the assistant's JSON object from a chat completion."""
        try:
            content = body["choices"][0]["message"]["content"]
        except (KeyError, IndexError, TypeError):
            return None
        try:
            obj = json.loads(content)
        except (TypeError, ValueError):
            return None
        return obj if isinstance(obj, dict) else None
 # ── Process-wide singleton ────────────────────────────────────────────
 _client: Optional[DeepSeekClient] = None
 _client_lock = threading.Lock()
 def get_llm_client() -> DeepSeekClient:
    global _client
    with _client_lock:
        if _client is None:
            settings = get_app_config().deepseek_settings
            _client = DeepSeekClient(
                api_key=deepseek_api_key(),
                model=str(settings.get("model", DEFAULT_MODEL)),
                timeout=float(settings.get("timeout_seconds",
                                           DEFAULT_TIMEOUT)),
                daily_cap=int(settings.get("daily_request_cap",
                                           DEFAULT_DAILY_CAP)),
                enabled=settings.get("enabled", True) is not False,
            )
        return _client
 def reset_llm_client() -> None:
    """Drop the singleton (tests / config reloads)."""
    global _client
    with _client_lock:
        _client = None
@@ -0,0 +1,252 @@
 ================================================================================
 MULTI-PROVIDER LLM LAYER - PLAN
 (DeepSeek for news summary, Anthropic Claude for agent panel)
 Written 2026-08-08. Plan only - no code has been changed.
 ================================================================================
 WHY
 ---
 Today every LLM feature (news enrichment + agent panel) goes through one
 hardwired client: llm_client.py -> DeepSeekClient, a module singleton reached
 via get_llm_client(). This plan describes how to route different features to
 different providers - e.g. keep DeepSeek for cheap high-volume news scoring,
 put the agent panel on Anthropic Claude for better reasoning.
 The repo already solved this problem once for market data: the providers/
 package (BaseDataProvider owns shared machinery, subclasses implement only
 _fetch_raw, a registry + one config knob selects the provider). This plan
 mirrors that proven pattern at the LLM layer, scaled down to a single module.
 BLAST RADIUS (verified by grep)
 -------------------------------
 get_llm_client() call sites:
  - news_engine.py ....... 4 sites
  - agent_panel.py ....... 3 sites (+1 timeout getattr)
  - telegram_bot.py ...... 1 site (/status health; reads private _count,
                           _count_day, _daily_cap fields)
  - test_all.py .......... 41 locations stub via  lc._client = _StubLLM(...)
 Out of scope: static/news_summary.php (own PHP DeepSeek call + key file,
 separate deployment).
 DESIGN INVARIANTS (carried over unchanged)
 ------------------------------------------
 1. Fail-safe contract: every call returns Optional[Dict]; any failure -> None;
   callers keep their non-LLM fallbacks; LLM output never gates signals.
 2. Single attempt, no retries (news cache cadence is the retry); per-provider
   UTC daily cap; warn-once logging; health stamps (last_ok, last_error).
 3. Static system prompts stay byte-identical constants - provider-neutral,
   and they map to prompt caching on BOTH providers (DeepSeek prefix cache;
   Anthropic cache_control ephemeral).
 4. Test compatibility: all 41 existing lc._client stub sites keep working
   with zero edits.
 ================================================================================
 1. llm_client.py REFACTOR  (~195 -> ~430 lines, no new files)
 ================================================================================
 1a. BaseLLMClient - shared machinery (extracted from today's DeepSeekClient)
    - __init__(api_key, model, timeout, daily_cap, enabled)
    - public: name ("deepseek"/"anthropic"), timeout property,
      usage() -> (used_today, cap)   # so telegram_bot stops reading privates
    - owns: available property, _warn_once, _under_cap() (UTC-day counter,
      lock), health stamps, and the template method:
        def chat_json(self, system_prompt, user_payload):
            if not self.available or not self._under_cap():
                return None
            try:
                return self._request(system_prompt, user_payload)  # hook
            except <provider-agnostic error mapping>:
                # stamps last_error, returns None
    - subclasses implement ONLY _request(system_prompt, payload) -> dict|None.
 1b. DeepSeekClient(BaseLLMClient) - behavior-identical move
    Current requests-based logic relocates into _request: same URL,
    thinking:disabled, response_format:json_object, temperature 0,
    max_tokens 2000, sort_keys=True payload, _parse_response.
    Review criterion: Module 19 tests pass UNMODIFIED (they patch
    client._session, which stays).
 1c. AnthropicClient(BaseLLMClient) - new, official SDK
    - Official `anthropic` Python SDK (add to requirements.txt), NOT raw
      requests. LAZY IMPORT inside __init__: package missing -> warn-once +
      client unavailable, so DeepSeek-only deployments never install it.
    - Constructor: anthropic.Anthropic(api_key=..., timeout=<configured>,
      max_retries=0)   # house rule: single attempt
    - _request:
        resp = self._client.messages.create(
            model=self._model,                  # default "claude-opus-5"
            max_tokens=4000,                    # covers thinking + JSON reply
            output_config={"effort": "low"},    # cheap structured takes
            system=[{"type": "text", "text": system_prompt,
                     "cache_control": {"type": "ephemeral"}}],
            messages=[{"role": "user",
                       "content": json.dumps(user_payload, sort_keys=True)}],
        )
        if resp.stop_reason == "refusal":       # Opus 5: HTTP 200, check first
            return None
        text = first text block in resp.content
        return self._parse_json(text)   # tolerant: strip ```json fences,
                                        # json.loads, dict check
    - DELIBERATELY ABSENT (would 400 on Claude Opus 5): temperature (removed
      on Opus 5 - do NOT copy from the DeepSeek client), budget_tokens,
      prefills, response_format.
    - Error mapping: APIConnectionError -> "network",
      APIStatusError -> "http_<code>", parse fail -> "parse"
      (same last_error kinds the /status display already understands).
    - Defaults: model claude-opus-5 (config-overridable, e.g. claude-haiku-4-5
      for ~5x cheaper), timeout 30s (Claude slower than V4-Flash), own
      daily_request_cap 500.
 1d. Registry + per-consumer routing
        _PROVIDERS = {"deepseek": DeepSeekClient, "anthropic": AnthropicClient}
        LLM_CONSUMERS = ("news", "agent_panel")
        _clients = {}     # keyed by PROVIDER name -> two consumers on the
                          # same provider share one instance and one cap
        _client = None    # kept: GLOBAL TEST OVERRIDE
        def get_llm_client(consumer="news"):
            if _client is not None:          # tests: lc._client = stub
                return _client               #   -> shadows ALL consumers
            name = get_app_config().llm_routing.get(consumer, "deepseek")
            create-once per provider, return _clients[name]
        def reset_llm_client():              # clears _client AND _clients
    Keeping _client as a documented test-override global is what makes the
    41 existing stub sites work untouched.
 ================================================================================
 2. CONFIG  (app_config.py + config.json.example)
 ================================================================================
 - New top-level "anthropic" block, mirroring "deepseek":
    {api_key, model, enabled, timeout_seconds, daily_request_cap}
  plus anthropic_api_key() resolver: ANTHROPIC_API_KEY env -> config.json -> "".
 - New top-level "llm" ROUTING block:
    {"news": "deepseek", "agent_panel": "deepseek"}
  Validation: keys within {news, agent_panel}; values within registered
  provider names. Absent block -> {} -> everything on DeepSeek (today's
  behavior; zero-migration default). AppConfig.llm_routing: Dict[str, str].
 - The target state is then PURE CONFIG - no code changes on switch day:
    "llm":       { "news": "deepseek", "agent_panel": "anthropic" },
    "anthropic": { "api_key": "", "model": "claude-opus-5", "enabled": true,
                   "timeout_seconds": 30, "daily_request_cap": 500 }
 ================================================================================
 3. CONSUMER EDITS  (mechanical, ~10 lines total)
 ================================================================================
 - news_engine.py:  4 call sites -> get_llm_client("news")
 - agent_panel.py:  3 call sites -> get_llm_client("agent_panel");
                   getattr(client, "_timeout", ...) -> public client.timeout
 - telegram_bot.py _news_health_lines: iterate the DISTINCT providers actually
  routed (news + panel); per provider print "<name>: used/cap calls today"
  via the new usage() accessor instead of private fields. DeepSeek keeps its
  "headlines scored" extra; cap/error warnings work off base-class stamps.
 - NO changes to prompts, payload builders, caches, or the panel's own
  max_panels_per_day budget - all provider-agnostic already.
 ================================================================================
 4. TESTS  (new "Module 24: LLM provider routing", ~12 tests, ~200 lines)
 ================================================================================
 Reuse _StubLLM; new _FakeAnthropic (fake messages.create returning a stub
 response object) injected as client._client.
 1. app_config: anthropic block parses/validates/defaults empty.
 2. app_config: llm routing block - valid map round-trips; unknown consumer
    key, unknown provider name, non-dict -> AppConfigError; absent -> {}.
 3. anthropic_api_key resolves env -> config.json -> empty.
 4. Base contract on BOTH subclasses (parametrized): keyless/disabled -> no
    call; daily cap blocks + UTC-roll reset; failure stamps last_error,
    success stamps last_ok.
 5. AnthropicClient parse: clean JSON -> dict; ```json-fenced -> dict;
    non-JSON -> None; stop_reason "refusal" -> None; no text block -> None.
 6. AnthropicClient request shape: NO temperature key, max_retries=0,
    cache_control present on system block (assert on captured kwargs).
 7. AnthropicClient with `anthropic` package missing (simulated) ->
    available False, chat_json None, no crash at import.
 8. Routing: no llm block -> both consumers get the SAME DeepSeek singleton
    (shared cap); routed config -> news gets DeepSeekClient while
    agent_panel gets AnthropicClient; same-provider consumers share one
    instance.
 9. Global lc._client override shadows BOTH consumers (locks the 41 legacy
    stub sites' semantics).
 10. reset_llm_client() clears both the override and the per-provider cache.
 11. End-to-end: routing agent_panel->anthropic, attach_panel reaches the
    anthropic fake (5 calls) while a news call reaches the deepseek stub.
 12. telegram _news_health_lines renders per-provider lines from usage()
    without touching private fields.
 GATE: Modules 19 and 23 must pass UNMODIFIED - the no-regression proof for
 the refactor.
 ================================================================================
 5. DOCS
 ================================================================================
 - CLAUDE.md: llm_client bullet -> provider-abstracted (BaseLLMClient;
  deepseek + anthropic registered); config.json "llm" block routes consumers;
  absent = all-DeepSeek. Add ANTHROPIC_API_KEY to env-var list. Test counts.
 - config.json.example: both new blocks.
 ================================================================================
 COMMIT SEQUENCE  (each commit green)
 ================================================================================
 1. llm_client: extract BaseLLMClient; DeepSeekClient unchanged behavior
   (+ base-contract tests)  - pure refactor, Modules 19/23 green untouched.
 2. config: anthropic block + llm consumer-routing block (+ config tests).
 3. llm_client: AnthropicClient (official SDK, lazy import) + requirements.txt
   (+ parse/shape tests).
 4. routing: per-consumer get_llm_client + health lines - consumer call-site
   edits, telegram usage() lines, routing/e2e tests.
 5. docs: CLAUDE.md + example config.
 ================================================================================
 VERIFICATION
 ================================================================================
 - python test_all.py green at every commit (~374 -> ~386).
 - Live smoke A (no behavior change): default config, main.py --panel ->
  still DeepSeek, /status unchanged.
 - Live smoke B (the goal): llm.agent_panel="anthropic" + real
  ANTHROPIC_API_KEY, run --panel -> one real Claude panel (5 requests,
  validates JSON contract + latency + refusal-free path) while a news
  refresh still hits DeepSeek; /status shows two provider lines.
 - Backtest untouched by construction (no LLM in replay path); one 30-day
  run as a formality.
 ================================================================================
 EFFORT ESTIMATE
 ================================================================================
 0 new files, 8 modified, ~520 lines, est. 3-5 hours.
  llm_client.py        base class + AnthropicClient + routing   +235
  test_all.py          Module 24                                +200
  app_config.py        anthropic + llm blocks, resolver          +55
  telegram_bot.py      per-provider health lines                 +15
  news_engine.py       consumer tags on call sites                +4
  agent_panel.py       consumer tags on call sites                +4
  config.json.example  blocks                                    +14
  requirements.txt     anthropic                                  +1
  CLAUDE.md            docs                                      +10
 ================================================================================
 KEY DECISIONS EMBEDDED IN THIS PLAN
 ================================================================================
 - Official Anthropic SDK, not raw requests (current API guidance for Python;
  lazy import keeps it optional for DeepSeek-only installs).
 - Default Claude model: claude-opus-5 ($5/$25 per MTok; a full 5-call panel
  ~ 8k in / 500 out ~ $0.05). Config-overridable to claude-haiku-4-5.
 - Two Claude gotchas handled: temperature is REMOVED on Opus 5 (sending it
  400s - must not be copied from the DeepSeek request body), and refusals
  arrive as HTTP 200 with stop_reason "refusal" (checked before parsing,
  mapped to the normal None/fallback path).
 - Caps are per-provider, not per-consumer: two consumers on one provider
  share its cap (today's semantics); splitting across two providers doubles
  total headroom naturally.
 - The panel's own max_panels_per_day budget is untouched - it sits above
  whichever provider serves it.
 - Not in scope: static/news_summary.php (own PHP client/key), streaming,
  tool use, per-consumer model overrides (add a "model" key per consumer
  later if ever needed).
 ================================================================================
@@ -13,7 +13,6 @@ import json
 import logging
 import sys
 from datetime import datetime, timezone
 from typing import Dict, List
 # Force UTF-8 output on Windows so box-drawing chars render
 if hasattr(sys.stdout, "reconfigure"):
@@ -21,7 +20,9 @@ if hasattr(sys.stdout, "reconfigure"):
 if hasattr(sys.stderr, "reconfigure"):
    sys.stderr.reconfigure(encoding="utf-8")
 from agent_panel import attach_panel
 from config import PlatformConfig, Timeframe, SignalStrength, XAUUSD
 from pair_signals import PAIR_PRODUCERS
 from signal_aggregator import SignalAggregator, SignalReport
@@ -97,6 +98,11 @@ def _compact_reason(reason: str) -> str:
    return s.strip()
 def _fmt_price(price: float) -> str:
    """2dp for gold-scale prices, 5dp for sub-10 FX-scale prices."""
    return f"{price:.2f}" if price >= 10 else f"{price:.5f}"
 def print_signal_report(report: SignalReport, detail: bool = False):
    sc = signal_color(report.final_signal)
    qc = quality_color(report.signal_quality)
@@ -116,16 +122,16 @@ def print_signal_report(report: SignalReport, detail: bool = False):
        ts = report.trade_setup
        dc = Colors.GREEN if ts.direction == "BUY" else Colors.RED
        print(f"\n  {Colors.BOLD}Setup{Colors.RESET}  "
-              f"{dc}{ts.direction}{Colors.RESET} @ {Colors.BOLD}{ts.entry_price:.3f}{Colors.RESET}  "
+              f"{dc}{ts.direction}{Colors.RESET} @ {Colors.BOLD}{_fmt_price(ts.entry_price)}{Colors.RESET}  "
-              f"→ SL {Colors.RED}{ts.stop_loss:.3f}{Colors.RESET} "
+              f"→ SL {Colors.RED}{_fmt_price(ts.stop_loss)}{Colors.RESET} "
              f"({ts.risk_pips:.1f}p)  "
-              f"→ TP {Colors.GREEN}{ts.take_profit_1:.3f}{Colors.RESET} "
+              f"→ TP {Colors.GREEN}{_fmt_price(ts.take_profit_1)}{Colors.RESET} "
              f"({ts.reward_pips:.1f}p)  "
              f"RR 1:{ts.risk_reward:.1f}")
    # ── 3. CONTEXT (price · regime · vol · MTF breakdown) ──────────
    print(f"\n  {Colors.WHITE}{report.display_name}{Colors.RESET} "
-          f"{Colors.BOLD}{report.current_price:.3f}{Colors.RESET} · "
+          f"{Colors.BOLD}{_fmt_price(report.current_price)}{Colors.RESET} · "
          f"{Colors.CYAN}{report.market_regime}{Colors.RESET} · "
          f"vol {report.volatility_regime}")
@@ -156,6 +162,15 @@ def print_signal_report(report: SignalReport, detail: bool = False):
        print(f"  {Colors.YELLOW}📅{Colors.RESET} {report.event_description}")
    if report.news_description and report.news_description != "Neutral news sentiment":
        print(f"  {Colors.MAGENTA}📰{Colors.RESET} {report.news_description}")
    if report.news_summary:
        print(f"  {Colors.MAGENTA}🧠{Colors.RESET} {report.news_summary}")
    if report.panel_verdict:
        print(f"  {Colors.CYAN}🤖{Colors.RESET} AI Panel: "
              f"{Colors.BOLD}{report.panel_verdict}{Colors.RESET} "
              f"({report.panel_confidence:.0f}%) — {report.panel_rationale}")
        for t in report.panel_takes:
            print(f"     {Colors.GRAY}{t['persona']:<11} {t['stance']:<8} "
                  f"{t['note']}{Colors.RESET}")
    # ── 6. DETAIL (only with --detail) ─────────────────────────────
    if detail and report.primary_analysis:
@@ -200,6 +215,12 @@ def report_to_dict(report: SignalReport) -> dict:
        "event_risk": report.event_risk,
        "event_description": report.event_description,
        "news_sentiment": round(report.news_sentiment, 3),
        "news_description": report.news_description,
        "news_summary": report.news_summary,
        "panel_verdict": report.panel_verdict,
        "panel_confidence": round(report.panel_confidence, 1),
        "panel_rationale": report.panel_rationale,
        "panel_takes": report.panel_takes,
        "bullish_reasons": report.bullish_reasons,
        "bearish_reasons": report.bearish_reasons,
        "warnings": report.warnings,
@@ -215,6 +236,9 @@ def report_to_dict(report: SignalReport) -> dict:
            "risk_reward": ts.risk_reward,
            "risk_pips": round(ts.risk_pips, 1),
            "reward_pips": round(ts.reward_pips, 1),
            "strategy": ts.strategy,
            "trail_arm_r": ts.trail_arm_r,
            "trail_dist_r": ts.trail_dist_r,
        }
    if report.primary_analysis:
        d["indicators"] = {
@@ -224,7 +248,7 @@ def report_to_dict(report: SignalReport) -> dict:
                "description": ind.description,
            }
            for name, ind in report.primary_analysis.indicators.items()
-            if not name.startswith("_")
+            if not name.startswith("_") and ind.weight != 0
        }
        d["key_levels"] = {
            k: round(v, 5) for k, v in report.primary_analysis.key_levels.items()
@@ -249,11 +273,16 @@ Examples:
  python main.py --json                   JSON output
        """,
    )
    parser.add_argument("--pair", "-p", default="XAUUSD",
                        help="Pair to analyze: XAUUSD (default: XAUUSD)")
    parser.add_argument("--timeframe", "-tf", default="H1",
                        choices=["M5", "M15", "M30", "H1", "H4", "D1"],
                        help="Primary analysis timeframe (default: H1)")
    parser.add_argument("--detail", action="store_true",
                        help="Show full indicator breakdown")
    parser.add_argument("--panel", action="store_true",
                        help="Run the LLM agent panel on the result "
                             "(advisory only; needs a DeepSeek key)")
    parser.add_argument("--json", action="store_true",
                        help="Output as JSON")
    parser.add_argument("--verbose", "-v", action="store_true",
@@ -262,14 +291,24 @@ Examples:
    return parser.parse_args()
 def _resolve_pair(s: str) -> str:
    aliases = {
        "XAUUSD": XAUUSD, "XAU/USD": XAUUSD, "GOLD": XAUUSD,
    }
    key = s.upper().strip()
    if key in aliases:
        return aliases[key]
    raise SystemExit(f"Unknown pair '{s}'. Try one of: {', '.join(sorted(aliases))}")
 def build_config(args) -> PlatformConfig:
-    config = PlatformConfig()
+    pair = _resolve_pair(args.pair)
    tf_map = {
        "M5": Timeframe.M5, "M15": Timeframe.M15, "M30": Timeframe.M30,
        "H1": Timeframe.H1, "H4": Timeframe.H4, "D1": Timeframe.D1,
    }
-    config.primary_timeframe = tf_map[args.timeframe]
+    primary_tf = tf_map[args.timeframe]
    tf_confirmations = {
        "M5":  [Timeframe.M1,  Timeframe.M15],
@@ -279,23 +318,36 @@ def build_config(args) -> PlatformConfig:
        "H4":  [Timeframe.H1,  Timeframe.D1],
        "D1":  [Timeframe.H4,  Timeframe.W1],
    }
-    config.confirmation_timeframes = tf_confirmations.get(
+    confirmation_tfs = tf_confirmations.get(
        args.timeframe, [Timeframe.M15, Timeframe.H4]
    )
    # Per-pair tuned configs from the pair_signals factories.
    factory = PAIR_PRODUCERS.get(pair)
    if factory is not None:
        return factory(primary_tf, confirmation_tfs)
    config = PlatformConfig()
    config.primary_timeframe = primary_tf
    config.confirmation_timeframes = confirmation_tfs
    return config
 def run_live(args, config: PlatformConfig):
    print(f"{Colors.GRAY}  Fetching market data and running analysis...{Colors.RESET}\n")
    aggregator = SignalAggregator(config)
-    report = aggregator.analyze_pair()
+    report = aggregator.analyze_pair(symbol=config.symbol)
    # Explicit user intent: force bypasses the enabled/surface/quality
    # toggles, but the daily panel budget and LLM cap still apply.
    if getattr(args, "panel", False):
        attach_panel(report, surface="cli", force=True)
    if args.json:
        print(json.dumps(report_to_dict(report), indent=2))
        return
-    print_header("XAU/USD SIGNAL INTELLIGENCE")
+    print_header(f"{config.display_name} SIGNAL INTELLIGENCE")
    print_signal_report(report, detail=args.detail)
    print(f"\n{Colors.CYAN}{'═' * 80}{Colors.RESET}")
    print(f"{Colors.GRAY}  Disclaimer: For educational/research purposes only.{Colors.RESET}")
@@ -8,21 +8,140 @@ Key principle: High-impact news events override technical signals.
 A perfect technical setup can be destroyed by an unexpected NFP print.
 """
 import hashlib
 import html
 import logging
 import re
 import threading
 import time
 import xml.etree.ElementTree as ET
 from dataclasses import dataclass, field
-from datetime import datetime, timedelta, timezone
+from datetime import datetime, timezone
 from email.utils import parsedate_to_datetime
 from typing import Dict, List, Optional, Tuple
 import requests
 from config import NewsConfig
 from llm_client import get_llm_client
 logger = logging.getLogger(__name__)
 # ── DeepSeek prompts ─────────────────────────────────────────────────
 # STATIC constants on purpose: DeepSeek prices a repeated byte-identical
 # prompt prefix at the cache-hit rate (~50x cheaper), so nothing here may
 # be interpolated or timestamped. All variable content travels in the
 # user payload. LLM output is display/warning-only — it never gates
 # signals (see NewsConfig.sentiment_affects_signals).
 _LLM_SENTIMENT_SYSTEM = """\
 You are a forex/gold news analyst. You receive JSON:
 {"headlines": ["...", ...]}
 For EACH headline, in the same order, score:
 - "s": overall market sentiment of the headline, -1.0 (very bearish)
  to 1.0 (very bullish), 0.0 if irrelevant or neutral.
 - "xau": the effect on the GOLD (XAU/USD) price, -1.0 (gold down)
  to 1.0 (gold up), 0.0 if irrelevant.
 Gold doctrine: hawkish Fed / rate hikes / rising real yields / strong
 dollar are gold-bearish; easing / rate cuts / safe-haven flows /
 geopolitical stress / weak dollar are gold-bullish.
 Handle negation correctly ("no rate hike" is not hawkish).
 Reply with ONLY JSON: {"scores": [[s, xau], ...]} — exactly one
 [s, xau] pair per input headline, in the same order, nothing else.
 Examples:
 "Fed signals faster rate hikes as inflation stays hot" -> [0.4, -0.7]
 "Gold surges to record high on safe-haven demand" -> [0.8, 0.9]
 "Dollar slips after soft jobs report; Treasury yields fall" -> [-0.4, 0.6]
 """
 _LLM_EVENT_SYSTEM = """\
 You are a forex/gold analyst. You receive one released economic event as
 JSON: {"event", "currency", "actual", "forecast", "previous",
 "vs_forecast"}. "vs_forecast" states whether the actual print came in
 above or below the forecast; it is computed programmatically — trust it
 over your own comparison of the numbers. If it is "unknown", compare
 actual and forecast yourself.
 Decide the likely effect of this RELEASE on the GOLD (XAU/USD) price.
 A stronger-than-forecast USD economy or a hawkish implication is usually
 gold-bearish; weaker data or a dovish implication is usually
 gold-bullish.
 Inverted metrics — for unemployment rate and unemployment/jobless
 claims a LOWER number means a STRONGER labor market: below forecast =
 strong economy = usually gold-bearish; above forecast = weak economy =
 usually gold-bullish.
 Example: {"event": "Unemployment Claims", "actual": "202K", "forecast":
 "217K", "vs_forecast": "below forecast"} ->
 {"gold_direction": "bearish", "reason": "fewer claims show strong labor market"}
 Reply with ONLY JSON:
 {"gold_direction": "bullish"|"bearish"|"neutral", "reason": "<max 10 words>"}
 """
 _LLM_SUMMARY_SYSTEM = """\
 You are a forex/gold market-context writer. You receive JSON with recent
 "headlines" and today's high-impact "events" (released: true means the
 number is out). Write a 1-2 sentence plain-English summary of the
 current macro backdrop for a GOLD (XAU/USD) trader: the dominant theme
 and its likely pull on gold. No advice, no hedging boilerplate, max 45
 words.
 Reply with ONLY JSON: {"summary": "..."}
 """
 _RELEASE_VALUE_RE = re.compile(r"(-?\d+(?:\.\d+)?)\s*([kmbt])?", re.IGNORECASE)
 _RELEASE_MULTIPLIERS = {"k": 1e3, "m": 1e6, "b": 1e9, "t": 1e12}
 def _parse_release_value(raw: Optional[str]) -> Optional[float]:
    """Parse a calendar release value like '0.3%', '-1.7%', '228K', '1.1M'."""
    if not raw:
        return None
    match = _RELEASE_VALUE_RE.search(raw.replace(",", ""))
    if not match:
        return None
    multiplier = _RELEASE_MULTIPLIERS.get((match.group(2) or "").lower(), 1.0)
    return float(match.group(1)) * multiplier
 _DESC_MAX_CHARS = 500
 _MAX_LINK_CHARS = 2048
 def _clean_html_text(raw: Optional[str],
                     limit: int = _DESC_MAX_CHARS) -> Optional[str]:
    """RSS <description> → plain text: unescape entities, strip tags,
    collapse whitespace, truncate. Unescape and strip both run twice
    because feeds routinely double-encode (&amp;lt;p&amp;gt; wrapping
    real markup) — the second unescape can reveal fresh tags."""
    if not raw:
        return None
    text = html.unescape(raw)
    text = re.sub(r"<[^>]+>", " ", text)
    text = html.unescape(text)
    text = re.sub(r"<[^>]+>", " ", text)
    text = re.sub(r"\s+", " ", text).strip()
    if not text:
        return None
    if len(text) > limit:
        text = text[:limit].rstrip() + "…"
    return text
 def _safe_link(raw: Optional[str]) -> Optional[str]:
    """Article URL for display: http(s) only (an RSS <link> is untrusted
    and lands in an href), bounded length (Google News redirect URLs run
    1-2 KB; anything past that is bloat, not a link)."""
    if not raw:
        return None
    link = raw.strip()
    if not link.lower().startswith(("http://", "https://")):
        return None
    if len(link) > _MAX_LINK_CHARS:
        return None
    return link
@dataclass
 class EconomicEvent:
    """Single economic calendar event."""
@@ -33,7 +152,21 @@ class EconomicEvent:
    forecast: Optional[str] = None
    previous: Optional[str] = None
    actual: Optional[str] = None
-    surprise: Optional[str] = None  # "positive", "negative", "inline", None
+
    @property
    def surprise(self) -> Optional[str]:
        """Actual vs forecast: "positive"/"negative"/"inline", or None when
        either value is missing or non-numeric. Direction is numeric only —
        it does not know whether higher is market-bullish for this event."""
        actual = _parse_release_value(self.actual)
        forecast = _parse_release_value(self.forecast)
        if actual is None or forecast is None:
            return None
        if actual > forecast:
            return "positive"
        if actual < forecast:
            return "negative"
        return "inline"
    @property
    def is_upcoming(self) -> bool:
@@ -58,6 +191,14 @@ class NewsSentiment:
    currencies_mentioned: List[str]
    sentiment_score: float  # -1.0 (bearish) to +1.0 (bullish)
    keywords: List[str]
    # Per-currency score: monetary-policy keywords flip sign for XAU
    # (hawkish is dollar-bullish but gold-bearish).
    sentiment_by_currency: Dict[str, float] = field(default_factory=dict)
    # Display-only enrichment (dashboard detail dialog). Optional with
    # defaults so existing constructor calls keep working.
    link: Optional[str] = None         # article URL (Google News = redirect)
    description: Optional[str] = None  # plain text, tags stripped, ≤500 chars
    publisher: Optional[str] = None    # Google News <source> text, else None
@dataclass
@@ -71,6 +212,7 @@ class FundamentalContext:
    news_headlines: List[NewsSentiment] = field(default_factory=list)
    event_impact_on_pairs: Dict[str, str] = field(default_factory=dict)
    calendar_available: bool = False  # True if live calendar data was fetched
    llm_summary: str = ""  # DeepSeek market-context blurb (display-only)
 class NewsEngine:
@@ -105,29 +247,71 @@ class NewsEngine:
        "disappointing", "soft data",
    }
-    # Currency-specific keywords for headline parsing
+    # Monetary-policy keywords whose direction inverts for gold: tighter
    # policy lifts the dollar and yields, weighing on non-yielding gold;
    # easing does the opposite. Only unambiguous policy words — economic
    # data words (beats/misses) keep their normal direction.
    GOLD_INVERSE_KEYWORDS = {
        "rate hike", "hawkish", "tightening",
        "rate cut", "dovish", "easing", "stimulus",
    }
    # Currency-specific keywords for headline parsing. Matched with word
    # boundaries — bare "us"/"eu"/"uk" are deliberately excluded because as
    # substrings they false-positive constantly ("versus" → USD).
    CURRENCY_KEYWORDS = {
-        "USD": ["dollar", "fed", "fomc", "treasury", "us ", "american", "nonfarm", "nfp"],
+        "USD": ["dollar", "fed", "federal reserve", "fomc", "treasury", "american", "nonfarm", "nfp"],
-        "EUR": ["euro", "ecb", "eurozone", "eu ", "german", "french"],
+        "EUR": ["euro", "ecb", "eurozone", "german", "french"],
-        "GBP": ["pound", "sterling", "boe", "bank of england", "uk ", "british"],
+        "GBP": ["pound", "sterling", "boe", "bank of england", "british"],
        "JPY": ["yen", "boj", "bank of japan", "japanese"],
        "CHF": ["franc", "snb", "swiss"],
        "AUD": ["aussie", "rba", "australian", "australia"],
        "CAD": ["loonie", "boc", "canadian", "canada"],
        "NZD": ["kiwi", "rbnz", "new zealand"],
-        "XAU": ["gold", "xau", "precious metal", "safe haven", "bullion"],
+        "XAU": ["gold", "precious metal", "safe haven", "bullion"],
    }
-    # High-impact event types (these move markets significantly)
+    # One compiled alternation per currency (code + keywords). (?<![a-z]) /
-    HIGH_IMPACT_EVENTS = {
+    # (?![a-z]) instead of \b so "0.3%fed" still matches but "versus",
-        "interest rate decision", "nonfarm payrolls", "non-farm",
+    # "audit", "cascade" don't.
-        "gdp", "cpi", "inflation", "employment change",
+    _CURRENCY_PATTERNS = {
-        "retail sales", "pmi", "fomc", "ecb press conference",
+        cur: re.compile(
-        "boe rate", "rba rate", "snb rate", "boc rate",
+            r"(?<![a-z])(?:"
            + "|".join(re.escape(kw) for kw in [cur.lower()] + kws)
            + r")(?![a-z])"
        )
        for cur, kws in CURRENCY_KEYWORDS.items()
    }
    _calendar_backoff_until: float = 0.0
    # Fetched-data cache, class-level: a fresh NewsEngine is built per scan,
    # so instance state would never be reused. Guarded by _cache_lock since
    # scans run concurrently in a thread pool.
    _cache_lock = threading.Lock()
    _calendar_cache: Tuple[float, List[EconomicEvent]] = (0.0, [])
    _headlines_cache: Tuple[float, List[NewsSentiment]] = (0.0, [])
    _EMPTY_RESULT_TTL = 60.0  # retry sooner after a failed/empty fetch
    # DeepSeek result caches. Headlines and releases are immutable, so a
    # scored one never needs re-scoring; bounded by dropping the oldest
    # insertion. _llm_sent_cache is only touched under _cache_lock (its
    # writer runs inside _cached_fetch); the other two take the lock
    # around cache access but keep HTTP calls outside it.
    _llm_sent_cache: Dict[str, Tuple[float, float]] = {}   # sha1 → (s, xau)
    _llm_event_cache: Dict[Tuple[str, str, str], str] = {}
    _llm_summary_cache: Tuple[float, str] = (0.0, "")
    _LLM_SENT_CACHE_MAX = 400
    _LLM_EVENT_CACHE_MAX = 100
    # Per-source fetch health for status displays (telegram /status).
    # name → {"ok": ts|None, "err": ts|None, "detail": str, "extra": str}.
    # All writers run under _cache_lock (fetches happen inside
    # _cached_fetch); readers render without the lock — worst case a
    # display line is one cycle stale. A source is "failing" when its
    # last error is newer than its last success.
    _source_health: Dict[str, dict] = {}
    def __init__(self, config: NewsConfig):
        self.config = config
        self._session = requests.Session()
@@ -153,7 +337,13 @@ class NewsEngine:
            events = self._fetch_economic_calendar(currencies)
            now = datetime.now(timezone.utc)
-            context.events_today = [e for e in events if e.time.date() == now.date()]
+            # Compare dates in UTC — feed timestamps carry their own offset
            # (FF sends US-Eastern), so a late-evening ET event would
            # otherwise land on the wrong day.
            context.events_today = [
                e for e in events
                if e.time.astimezone(timezone.utc).date() == now.date()
            ]
            context.events_upcoming_1h = [
                e for e in events
                if 0 < e.minutes_until <= 60
@@ -187,10 +377,12 @@ class NewsEngine:
            headlines = self._fetch_news_sentiment()
            context.news_headlines = headlines
-            # Aggregate sentiment per currency
+            # Aggregate sentiment per currency (per-currency score where
            # available, so gold-inverted policy keywords are respected)
            for currency in currencies:
                scores = [
-                    h.sentiment_score for h in headlines
+                    h.sentiment_by_currency.get(currency, h.sentiment_score)
                    for h in headlines
                    if currency in h.currencies_mentioned
                ]
                if scores:
@@ -199,6 +391,12 @@ class NewsEngine:
        except Exception as e:
            logger.warning(f"News sentiment fetch failed: {e}")
        # 3. LLM market-context summary (display-only)
        try:
            context.llm_summary = self._llm_market_summary(context)
        except Exception as e:
            logger.debug(f"LLM market summary failed: {e}")
        return context
    @staticmethod
@@ -217,19 +415,46 @@ class NewsEngine:
        """
        base, quote = self._parse_pair(pair_symbol)
-        # Check upcoming high-impact events
+        # Check upcoming high-impact events. Grade from the SOONEST
        # matching event, not the first listed — an unsorted feed could
        # otherwise report "caution" for a far event while a nearer one
        # warrants "danger".
        soonest = None
        for event in context.events_upcoming_1h:
            if event.is_high_impact and event.currency in (base, quote):
-                mins = event.minutes_until
+                if soonest is None or event.minutes_until < soonest.minutes_until:
-                if mins <= self.config.pre_event_buffer_minutes:
+                    soonest = event
-                    return "danger", f"{event.currency} {event.event} in {mins:.0f}min"
+        if soonest is not None:
-                else:
+            mins = soonest.minutes_until
-                    return "caution", f"{event.currency} {event.event} in {mins:.0f}min"
+            if mins <= self.config.pre_event_buffer_minutes:
                return "danger", f"{soonest.currency} {soonest.event} in {mins:.0f}min"
            return "caution", f"{soonest.currency} {soonest.event} in {mins:.0f}min"
-        # Check for recent high-impact releases (within post-event buffer)
+        # Check for recent high-impact releases. A surprise print (actual
        # away from forecast) keeps the market unstable longer than an
        # inline release, so it gets a wider post-event window.
        for event in context.events_today:
-            if event.is_high_impact and event.currency in (base, quote):
+            if event.is_high_impact and event.currency in (base, quote) and not event.is_upcoming:
-                if not event.is_upcoming and event.minutes_until > -self.config.post_event_buffer_minutes:
+                surprise = event.surprise
                surprised = surprise in ("positive", "negative")
                window = (
                    self.config.post_event_surprise_buffer_minutes
                    if surprised
                    else self.config.post_event_buffer_minutes
                )
                if event.minutes_until > -window:
                    if surprised:
                        desc = (
                            f"{event.currency} {event.event} released — {surprise} surprise "
                            f"(actual {event.actual} vs forecast {event.forecast})"
                        )
                        # LLM enrichment touches the DESCRIPTION only —
                        # `surprise` is numeric-direction and does not know
                        # whether a beat is gold-bullish; the LLM does.
                        gold_dir = self._llm_event_direction(event)
                        if gold_dir:
                            desc += f" — gold {gold_dir}"
                        return "caution", desc
                    return "caution", f"{event.currency} {event.event} just released"
        return "clear", "No significant events"
@@ -263,6 +488,24 @@ class NewsEngine:
        return net, desc
    def get_week_events(
        self, currencies: Optional[List[str]] = None
    ) -> List[EconomicEvent]:
        """Full cached calendar week (past and upcoming), filtered to
        `currencies` (default: tracked_currencies), sorted by time.
        Read-only accessor for display surfaces (the web dashboard) —
        rides the same class-level cache as signal scans, so it adds no
        network I/O. No production signal path calls this.
        """
        if currencies is None:
            currencies = self.config.tracked_currencies
        events = self._cached_fetch("_calendar_cache", self._fetch_calendar_raw)
        return sorted(
            (e for e in events if e.currency in currencies),
            key=lambda e: e.time,
        )
    # ── Private Methods ────────────────────────────────────────────
    # Maps country names in MyFXBook titles to currency codes
@@ -272,21 +515,89 @@ class NewsEngine:
        "canada": "CAD", "new zealand": "NZD",
    }
    @classmethod
    def _record_source(cls, name: str, ok: bool, detail: str = "",
                       extra: str = "") -> None:
        """Track one dependency's fetch outcome for status displays.
        Logs state TRANSITIONS so the log file carries the story without
        a line every 10-min cycle: first failure WARNING (with detail),
        repeat failures DEBUG, recovery INFO."""
        h = cls._source_health.setdefault(
            name, {"ok": None, "err": None, "detail": "", "extra": ""})
        was_failing = h["err"] is not None and (
            h["ok"] is None or h["err"] > h["ok"])
        if ok:
            h["ok"] = time.time()
            h["extra"] = extra
            if was_failing:
                logger.info("Source recovered: %s", name)
        else:
            h["err"] = time.time()
            h["detail"] = detail
            if was_failing:
                logger.debug("Source still failing: %s (%s)", name, detail)
            else:
                logger.warning("Source failing: %s (%s)", name, detail)
    @staticmethod
    def source_is_failing(entry: Optional[dict]) -> bool:
        """True when the entry's last error is newer than its last ok."""
        return bool(entry and entry["err"] is not None and
                    (entry["ok"] is None or entry["err"] > entry["ok"]))
    def _cached_fetch(self, cache_attr: str, fetch_fn) -> list:
        """Return cached data for `cache_attr` or refetch when the TTL
        expired. Successful results live for context_cache_ttl_seconds;
        empty results only _EMPTY_RESULT_TTL so outages retry quickly.
        The lock is held across the fetch so concurrent scans share one
        request instead of stampeding the source."""
        ts, data = getattr(NewsEngine, cache_attr)
        ttl = self.config.context_cache_ttl_seconds if data else self._EMPTY_RESULT_TTL
        if time.time() - ts < ttl:
            return data
        with NewsEngine._cache_lock:
            ts, data = getattr(NewsEngine, cache_attr)  # re-check under lock
            ttl = self.config.context_cache_ttl_seconds if data else self._EMPTY_RESULT_TTL
            if time.time() - ts < ttl:
                return data
            data = fetch_fn()
            setattr(NewsEngine, cache_attr, (time.time(), data))
            return data
    def _fetch_economic_calendar(self, currencies: List[str]) -> List[EconomicEvent]:
        """Cached calendar fetch, filtered to the tracked currencies."""
        events = self._cached_fetch("_calendar_cache", self._fetch_calendar_raw)
        return [e for e in events if e.currency in currencies]
    def _fetch_calendar_raw(self) -> List[EconomicEvent]:
        """
        Fetch economic calendar events from a free API.
        Tries Forex Factory first, falls back to MyFXBook RSS.
        Returns an empty list (with a log warning) if unavailable —
        we never generate fake events since that causes constant false alarms.
        """
-        for fetcher in (self._try_fetch_calendar_api, self._try_fetch_myfxbook_rss):
+        sources = (
            (self._try_fetch_calendar_api, "calendar:forexfactory",
             "ForexFactory"),
            (self._try_fetch_myfxbook_rss, "calendar:myfxbook", "MyFXBook"),
        )
        for fetcher, src, label in sources:
            try:
                events = fetcher()
                if events:
-                    return [e for e in events if e.currency in currencies]
+                    self._record_source(src, True)
                    self._record_source(
                        "calendar", True,
                        extra=f"{len(events)} events via {label}")
                    return events
                self._record_source(src, False, "no events returned")
            except Exception as e:
                self._record_source(
                    src, False, f"{type(e).__name__}: {e}")
                logger.debug(f"Calendar fetch exception ({fetcher.__name__}): {e}")
        self._record_source("calendar", False, "all sources failed")
        logger.info(
            "Economic calendar data unavailable. "
            "Event risk checks will be skipped for this cycle."
@@ -426,13 +737,23 @@ class NewsEngine:
        return events
    def _fetch_news_sentiment(self) -> List[NewsSentiment]:
        """Cached headline fetch + sentiment analysis."""
        return self._cached_fetch("_headlines_cache", self._fetch_news_sentiment_raw)
    def _fetch_news_sentiment_raw(self) -> List[NewsSentiment]:
        """Fetch and analyze news headlines for sentiment."""
        headlines = []
        seen_titles = set()
        ok_feeds = 0
        for feed_url in self.config.rss_feeds:
            host = feed_url.split("/")[2] if "//" in feed_url else feed_url
            kept_before = len(headlines)
            try:
                resp = self._session.get(feed_url, timeout=10)
                if resp.status_code != 200:
                    self._record_source(
                        f"rss:{host}", False, f"HTTP {resp.status_code}")
                    continue
                root = ET.fromstring(resp.content)
@@ -449,14 +770,37 @@ class NewsEngine:
                        pub_date_el.text if pub_date_el is not None else None
                    )
-                    # Only recent headlines (last 24h)
+                    # Only recent headlines (configurable: headline_max_age_seconds)
-                    if pub_date and (datetime.now(timezone.utc) - pub_date).total_seconds() > 86400:
+                    if pub_date and (datetime.now(timezone.utc) - pub_date).total_seconds() > self.config.headline_max_age_seconds:
                        continue
                    # Dedup within and across feeds so a syndicated story
                    # doesn't count twice in the sentiment average. Google
                    # News appends " - Publisher", so compare without it.
                    title_key = title.lower().rsplit(" - ", 1)[0].strip()
                    if title_key in seen_titles:
                        continue
                    seen_titles.add(title_key)
                    sentiment = self._analyze_headline_sentiment(title)
                    currencies = self._extract_currencies(title)
                    if currencies:
                        link_el = item.find("link")
                        desc_el = item.find("description")
                        src_el = item.find("source")
                        # itertext() survives CDATA and stray child
                        # elements where .text stops at the first child.
                        desc = _clean_html_text(
                            "".join(desc_el.itertext())
                            if desc_el is not None else None)
                        # Google News descriptions are often just the
                        # headline again inside an anchor — worthless.
                        if desc and desc.strip() == title.strip():
                            desc = None
                        publisher = None
                        if src_el is not None:
                            publisher = (src_el.text or "").strip() or None
                        headlines.append(NewsSentiment(
                            headline=title,
                            source=feed_url,
@@ -464,26 +808,70 @@ class NewsEngine:
                            currencies_mentioned=currencies,
                            sentiment_score=sentiment,
                            keywords=self._extract_keywords(title),
                            sentiment_by_currency={
                                cur: self._analyze_headline_sentiment(title, cur)
                                for cur in currencies
                            },
                            link=_safe_link(
                                link_el.text if link_el is not None else None),
                            description=desc,
                            publisher=publisher,
                        ))
                ok_feeds += 1
                self._record_source(
                    f"rss:{host}", True,
                    extra=f"{len(headlines) - kept_before} kept")
            except Exception as e:
                self._record_source(
                    f"rss:{host}", False, f"{type(e).__name__}: {e}")
                logger.debug(f"Failed to fetch RSS from {feed_url}: {e}")
        self._record_source(
            "headlines", ok_feeds > 0,
            detail="all feeds failed" if ok_feeds == 0 else "",
            extra=(f"{len(headlines)} headlines from "
                   f"{ok_feeds}/{len(self.config.rss_feeds)} feeds"))
        try:
            self._apply_llm_sentiment(headlines)
        except Exception as e:
            logger.debug(f"LLM sentiment failed, keyword scores stand: {e}")
        return headlines
-    def _analyze_headline_sentiment(self, headline: str) -> float:
+    def _analyze_headline_sentiment(
        self, headline: str, for_currency: Optional[str] = None
    ) -> float:
        """
        Simple keyword-based sentiment analysis.
        Returns -1.0 (very bearish) to +1.0 (very bullish).
        When for_currency is "XAU", monetary-policy keywords contribute
        with inverted sign (see GOLD_INVERSE_KEYWORDS).
        Note: This is intentionally simple. NLP models like FinBERT
        would be more accurate but require large dependencies. This catches
        the most impactful sentiment signals from headlines.
        """
        headline_lower = headline.lower()
-        bullish_score = sum(1 for kw in self.BULLISH_KEYWORDS if kw in headline_lower)
+        invert_policy = for_currency == "XAU"
-        bearish_score = sum(1 for kw in self.BEARISH_KEYWORDS if kw in headline_lower)
+
        bullish_score = 0
        bearish_score = 0
        for kw in self.BULLISH_KEYWORDS:
            if kw in headline_lower:
                if invert_policy and kw in self.GOLD_INVERSE_KEYWORDS:
                    bearish_score += 1
                else:
                    bullish_score += 1
        for kw in self.BEARISH_KEYWORDS:
            if kw in headline_lower:
                if invert_policy and kw in self.GOLD_INVERSE_KEYWORDS:
                    bullish_score += 1
                else:
                    bearish_score += 1
        total = bullish_score + bearish_score
        if total == 0:
@@ -491,22 +879,150 @@ class NewsEngine:
        return (bullish_score - bearish_score) / total
    # ── DeepSeek enrichment (display/warning-only, keyword fallback) ──
    def _llm_headline_scores(
        self, texts: List[str]
    ) -> Optional[List[Tuple[float, float]]]:
        """One batched DeepSeek call → [(overall, xau), ...] aligned with
        `texts`, or None on any failure/mismatch (caller keeps keyword
        scores). Tests stub this method directly."""
        result = get_llm_client().chat_json(
            _LLM_SENTIMENT_SYSTEM, {"headlines": texts})
        if not result:
            return None
        scores = result.get("scores")
        if not isinstance(scores, list) or len(scores) != len(texts):
            return None
        out = []
        for item in scores:
            try:
                s = max(-1.0, min(1.0, float(item[0])))
                x = max(-1.0, min(1.0, float(item[1])))
            except (TypeError, KeyError, ValueError, IndexError):
                return None
            out.append((s, x))
        return out
    def _apply_llm_sentiment(self, headlines: List[NewsSentiment]) -> None:
        """Overwrite keyword sentiment on `headlines` with LLM scores.
        Runs inside _cached_fetch, i.e. UNDER _cache_lock — cache access
        is direct and the (rare) HTTP call blocks sibling scans at most
        once per cache window, matching the existing shared-fetch design.
        Only headlines missing from the cache are sent; a failed batch
        changes nothing.
        """
        if not headlines or not get_llm_client().available:
            return
        cache = NewsEngine._llm_sent_cache
        keyed = [(hashlib.sha1(h.headline.encode()).hexdigest(), h)
                 for h in headlines]
        # Cap the batch so the reply fits the client's max_tokens — the
        # overflow stays keyword-scored this cycle and, still uncached,
        # goes out with the next window's batch.
        new = [(k, h) for k, h in keyed if k not in cache][:25]
        if new:
            scores = self._llm_headline_scores([h.headline for _, h in new])
            if scores is not None:
                for (k, _), sc in zip(new, scores):
                    cache[k] = sc
                while len(cache) > self._LLM_SENT_CACHE_MAX:
                    cache.pop(next(iter(cache)))
        for k, h in keyed:
            sc = cache.get(k)
            if sc is None:
                continue
            overall, xau = sc
            h.sentiment_score = overall
            h.sentiment_by_currency = {
                cur: (xau if cur == "XAU" else overall)
                for cur in h.currencies_mentioned
            }
    def _llm_event_direction(self, event: EconomicEvent) -> Optional[str]:
        """Gold direction for a released surprise, e.g. "bearish: strong
        jobs lift yields". None → caller keeps the numeric description.
        Cached per release; failures are NOT cached (retry next scan).
        The actual-vs-forecast comparison is precomputed and sent as
        "vs_forecast": the 2026-08-06 validation found the model reads
        close K/M-suffixed pairs backwards ~11% of the time (e.g. claims
        202K vs 217K called "higher than forecast") when left to compare
        the raw numbers itself."""
        key = (event.event, event.currency, str(event.actual or ""))
        with NewsEngine._cache_lock:
            if key in NewsEngine._llm_event_cache:
                return NewsEngine._llm_event_cache[key] or None
        client = get_llm_client()
        if not client.available:
            return None
        vs_forecast = {
            "positive": "above forecast",
            "negative": "below forecast",
            "inline": "in line with forecast",
        }.get(event.surprise or "", "unknown")
        result = client.chat_json(_LLM_EVENT_SYSTEM, {
            "event": event.event,
            "currency": event.currency,
            "actual": event.actual,
            "forecast": event.forecast,
            "previous": event.previous,
            "vs_forecast": vs_forecast,
        })
        if not result:
            return None
        direction = result.get("gold_direction")
        if direction not in ("bullish", "bearish", "neutral"):
            return None
        reason = str(result.get("reason", "")).strip()[:80]
        text = "" if direction == "neutral" else (
            f"{direction}: {reason}" if reason else direction)
        with NewsEngine._cache_lock:
            NewsEngine._llm_event_cache[key] = text
            while len(NewsEngine._llm_event_cache) > self._LLM_EVENT_CACHE_MAX:
                NewsEngine._llm_event_cache.pop(
                    next(iter(NewsEngine._llm_event_cache)))
        return text or None
    def _llm_market_summary(self, context: FundamentalContext) -> str:
        """1-2 sentence macro blurb for alerts; "" when unavailable.
        Cached (success OR failure) for the context-cache TTL so at most
        one call is made per window."""
        with NewsEngine._cache_lock:
            ts, cached = NewsEngine._llm_summary_cache
            if time.time() - ts < self.config.context_cache_ttl_seconds:
                return cached
        high_impact = [e for e in context.events_today if e.is_high_impact]
        if not context.news_headlines and not high_impact:
            return ""
        client = get_llm_client()
        if not client.available:
            return ""
        result = client.chat_json(_LLM_SUMMARY_SYSTEM, {
            "headlines": [h.headline for h in context.news_headlines[:15]],
            "events": [
                {"event": e.event, "currency": e.currency,
                 "actual": e.actual, "forecast": e.forecast,
                 "released": not e.is_upcoming}
                for e in high_impact[:10]
            ],
        })
        summary = ""
        if result:
            summary = str(result.get("summary", "")).strip()[:300]
        with NewsEngine._cache_lock:
            NewsEngine._llm_summary_cache = (time.time(), summary)
        return summary
    def _extract_currencies(self, headline: str) -> List[str]:
        """Extract mentioned currencies from a headline."""
        headline_lower = headline.lower()
-        found = []
+        return [
-
+            currency
-        for currency, keywords in self.CURRENCY_KEYWORDS.items():
+            for currency, pattern in self._CURRENCY_PATTERNS.items()
-            if re.search(r'\b' + re.escape(currency.lower()) + r'\b', headline_lower):
+            if pattern.search(headline_lower)
-                found.append(currency)
+        ]
                continue
            for kw in keywords:
                kw_stripped = kw.strip()
                if kw_stripped and re.search(r'\b' + re.escape(kw_stripped) + r'\b', headline_lower):
                    found.append(currency)
                    break
        return list(set(found))
    def _extract_keywords(self, headline: str) -> List[str]:
        """Extract relevant trading keywords from headline."""
@@ -519,11 +1035,20 @@ class NewsEngine:
        if not date_str:
            return None
        # RFC 822/2822 forms, with or without seconds, numeric offsets AND
        # named zones (GMT, EST/EDT, PST…). The old strptime "%Z" path
        # accepted ANY abbreviation but attached no offset, so e.g. an EST
        # pubDate was silently stamped UTC — a 5-hour error.
        try:
            dt = parsedate_to_datetime(date_str.strip())
            if dt is not None:
                if dt.tzinfo is None:   # unknown zone name — assume UTC
                    return dt.replace(tzinfo=timezone.utc)
                return dt.astimezone(timezone.utc)
        except (TypeError, ValueError):
            pass
        formats = [
            "%a, %d %b %Y %H:%M:%S %z",   # RFC 822 with numeric tz offset
            "%a, %d %b %Y %H:%M:%S %Z",   # RFC 822 with tz abbreviation (GMT/UTC)
            "%a, %d %b %Y %H:%M %Z",      # RFC 822 without seconds (MyFXBook)
            "%a, %d %b %Y %H:%M %z",      # RFC 822 without seconds, numeric offset
            "%Y-%m-%dT%H:%M:%S%z",         # ISO 8601 with offset
            "%Y-%m-%d %H:%M:%S",           # ISO 8601 naive (assume UTC)
        ]
@@ -542,8 +1067,7 @@ class NewsEngine:
    @staticmethod
    def _get_affected_pairs(currency: str) -> List[str]:
        _MAP = {
-            "USD": ["XAU/USD", "EUR/USD"],
+            "USD": ["XAU/USD"],
            "XAU": ["XAU/USD"],
            "EUR": ["EUR/USD"],
        }
        return _MAP.get(currency, [])
@@ -0,0 +1,854 @@
 """
 ForexBot2 — Oanda Automated Trading Daemon
 ============================================
 Autonomous trading process for XAU/USD on an Oanda practice account.
 Runs candle-boundary scan loops (M5/M15/H1/H4), executes trades via
 the Oanda v20 API, and manages trailing stops at bar close — matching
 the backtested bar-close ratchet model.
 Book-parity notes (matching backtest/gpu_backtest.py):
 - No broker-side take-profit: in the trailing model TP1/TP2 are
  advisory; the only exits are SL (broker), TRAIL (daemon ratchet),
  and TIMEOUT (max_hold_bars). A hard TP would cap winners the trail
  lets run and diverge the live book from the backtest.
 - Entry cooldown is 5 bars per timeframe (the backtester's only
  sequential rule) — not a fingerprint block.
 - The trail ratchets at bar CLOSE only, and each closed bar is
  processed exactly once (last_bar_ts guard), so weekends and feed
  lag can't double-count bars or inflate the hold counter.
 Divergence from the backtest book (accepted for demo safety): one
 position total. While a trade is open, other signals are logged and
 skipped, so skipped entries can be compared against the book later.
 Usage:  python oanda_daemon.py
 Config: env vars OANDA_ACCOUNT_ID, OANDA_API_TOKEN, OANDA_BASE_URL
        or config.json "oanda" section (env wins).
 """
 import asyncio
 import json
 import logging
 import os
 import signal
 import sys
 import threading
 from concurrent.futures import ThreadPoolExecutor
 from datetime import datetime, timezone
 from logging.handlers import TimedRotatingFileHandler
 from typing import Dict, Optional
 import pandas as pd
 from app_config import (
    get_app_config, oanda_account_id, oanda_api_token, oanda_base_url,
 )
 from config import PAIR_INFO, PlatformConfig, Timeframe, XAUUSD
 from signal_aggregator import SignalAggregator, SignalReport, SignalStrength
 from pair_signals import PAIR_PRODUCERS
 from providers.oanda import OandaDataProvider
 from oanda_executor import OandaExecutor, OandaAPIError
 logger = logging.getLogger(__name__)
 # ── Constants ──────────────────────────────────────────────────────────
 SYMBOL = XAUUSD
 INSTRUMENT = "XAU_USD"
 SCAN_CONFIRMATIONS = {
    Timeframe.M5:  [Timeframe.M15, Timeframe.H1],
    Timeframe.M15: [Timeframe.M5, Timeframe.H1],
    Timeframe.H1:  [Timeframe.M15, Timeframe.H4],
    Timeframe.H4:  [Timeframe.H1, Timeframe.D1],
 }
 TF_INTERVAL_MINUTES = {
    Timeframe.M5: 5, Timeframe.M15: 15, Timeframe.M30: 30,
    Timeframe.H1: 60, Timeframe.H4: 240, Timeframe.D1: 1440,
 }
 SCAN_TIMEFRAMES = [Timeframe.M5, Timeframe.M15, Timeframe.H1, Timeframe.H4]
 QUALITY_RANK = {"A+": 4, "A": 3, "B": 2, "C": 1, "D": 0, "F": -1}
 # Matches the backtester's --cooldown-bars default (the only sequential
 # rule in the replay): no new entry within 5 bars of the previous entry
 # on the same timeframe.
 COOLDOWN_BARS = 5
 # How often the heartbeat confirms the process is alive and the account
 # is still reachable — independent of the candle-boundary loops, so a
 # wedged scan or mid-run credential expiry still surfaces in the log.
 HEARTBEAT_SECONDS = 3600
 STATE_FILE = os.path.join(
    os.path.dirname(os.path.abspath(__file__)), "oanda_state.json")
 # ── Config resolution (env → config.json → default) ──────────────────
 def _oanda_setting_int(env_key: str, json_key: str, default: int) -> int:
    env = os.environ.get(env_key)
    if env:
        try:
            return int(env)
        except ValueError:
            raise SystemExit(
                f"{env_key}={env!r} is not an integer — refusing to start")
    try:
        return int(get_app_config().oanda_settings.get(json_key, default))
    except (TypeError, ValueError):
        raise SystemExit(
            f"config.json oanda.{json_key} is not an integer — "
            f"refusing to start")
 def _oanda_setting_str(env_key: str, json_key: str, default: str) -> str:
    return (os.environ.get(env_key)
            or str(get_app_config().oanda_settings.get(json_key, default)))
 # ── Logging setup ──────────────────────────────────────────────────────
 def _setup_logging(level: str = "INFO") -> None:
    log_dir = os.path.join(os.path.dirname(os.path.abspath(__file__)), "logs")
    os.makedirs(log_dir, exist_ok=True)
    log_file = os.path.join(log_dir, "daemon.log")
    fmt = logging.Formatter(
        "%(asctime)s [%(levelname)s] %(name)s: %(message)s",
        datefmt="%Y-%m-%d %H:%M:%S")
    file_handler = TimedRotatingFileHandler(
        log_file, when="h", interval=3, backupCount=240, utc=True)
    file_handler.setFormatter(fmt)
    console_handler = logging.StreamHandler(sys.stdout)
    console_handler.setFormatter(fmt)
    logging.basicConfig(
        level=getattr(logging, level.upper(), logging.INFO),
        handlers=[file_handler, console_handler],
        force=True)
 # ── Boundary timing (same as telegram_bot.py) ─────────────────────────
 def _seconds_until_boundary(interval_minutes: int, buffer: int = 10) -> float:
    now = datetime.now(timezone.utc)
    total_minutes = now.hour * 60 + now.minute
    elapsed_minutes = total_minutes % interval_minutes
    elapsed_seconds = elapsed_minutes * 60 + now.second + now.microsecond / 1e6
    interval_seconds = interval_minutes * 60
    remaining = interval_seconds - elapsed_seconds + buffer
    # NOTE: remaining > buffer always (elapsed < interval by construction);
    # a call at the exact boundary instant targets the NEXT boundary —
    # steady-state loops wake at boundary+buffer, so only a process
    # started in the boundary microsecond skips one scan (2026-08-04
    # audit I42; the old `remaining <= buffer` guard was unreachable).
    return remaining
 # ── Trade state persistence ───────────────────────────────────────────
 def _load_state() -> Dict:
    if os.path.exists(STATE_FILE):
        try:
            with open(STATE_FILE, encoding="utf-8") as fh:
                return json.load(fh)
        except (OSError, json.JSONDecodeError):
            logger.warning("Could not load state file, starting fresh")
    return {"trades": {}, "last_entries": {}}
 def _save_state(state: Dict) -> None:
    try:
        tmp = STATE_FILE + ".tmp"
        with open(tmp, "w", encoding="utf-8") as fh:
            json.dump(state, fh, indent=2)
        os.replace(tmp, STATE_FILE)
    except OSError:
        logger.exception("Failed to save state file")
 # ── Trading Daemon ─────────────────────────────────────────────────────
 class TradingDaemon:
    def __init__(self):
        account_id = oanda_account_id()
        api_token = oanda_api_token()
        base_url = oanda_base_url()
        lot_size = _oanda_setting_int("OANDA_LOT_SIZE", "lot_size", 1)
        self._quality_gate = _oanda_setting_str(
            "OANDA_QUALITY_GATE", "quality_gate", "C").strip()
        if self._quality_gate not in QUALITY_RANK:
            # Fail CLOSED: an unrecognized gate must never silently fall
            # back to the most permissive rank (QUALITY_RANK.get default).
            raise SystemExit(
                f"OANDA_QUALITY_GATE/quality_gate {self._quality_gate!r} "
                f"is not one of {sorted(QUALITY_RANK)} — refusing to start")
        self._max_hold_bars = _oanda_setting_int(
            "OANDA_MAX_HOLD_BARS", "max_hold_bars", 300)
        self._price_precision = PAIR_INFO[SYMBOL]["price_precision"]
        if not account_id or not api_token:
            msg = (
                "Missing Oanda credentials. Set OANDA_ACCOUNT_ID and "
                "OANDA_API_TOKEN env vars or config.json oanda section.")
            logger.critical(msg)
            raise SystemExit(msg)
        self._provider = OandaDataProvider(
            PlatformConfig(), api_token, base_url)
        self._executor = OandaExecutor(
            account_id, api_token, base_url, lot_size,
            price_precision=self._price_precision)
        self._thread_pool = ThreadPoolExecutor(max_workers=4)
        self._state = _load_state()
        self._state.setdefault("trades", {})
        self._state.setdefault("last_entries", {})
        self._state_lock = threading.Lock()
        # Serializes the position-check + order-placement critical section
        # across concurrent TF loops (all four fire at aligned boundaries).
        self._entry_lock = threading.Lock()
        self._running = True
    # ── Main entry point ───────────────────────────────────────────
    async def run(self) -> None:
        loop = asyncio.get_running_loop()
        loop.set_exception_handler(self._asyncio_exception_handler)
        for sig in (signal.SIGINT, signal.SIGTERM):
            loop.add_signal_handler(sig, self._request_shutdown)
        logger.info("=== Oanda Trading Daemon starting ===")
        logger.info(
            "Quality gate: %s | Max hold: %d bars | Cooldown: %d bars | "
            "Timeframes: %s",
            self._quality_gate, self._max_hold_bars, COOLDOWN_BARS,
            "/".join(tf.name for tf in SCAN_TIMEFRAMES))
        # Credentials first: with a bad token every later call fails, so
        # refuse to start before touching positions or spawning loops.
        self._verify_account_access()
        self._discover_existing_positions()
        tasks = []
        for tf in SCAN_TIMEFRAMES:
            interval = TF_INTERVAL_MINUTES[tf]
            tasks.append(asyncio.create_task(
                self._candle_loop(tf, interval)))
        tasks.append(asyncio.create_task(self._heartbeat_loop()))
        logger.info("Candle-boundary loops started")
        try:
            await asyncio.gather(*tasks)
        except asyncio.CancelledError:
            pass
        finally:
            self._shutdown()
    def _verify_account_access(self) -> None:
        """Boot-time credential check. A 4xx here means the token/account
        id is wrong — that never fixes itself, so refuse to start rather
        than run a daemon that can't place or manage trades. A connection
        failure or 5xx is a transient outage: log it and continue, since
        broker-side stops already protect any open position and the
        hourly heartbeat will keep reporting account health."""
        try:
            self._executor.get_account_summary()
        except OandaAPIError as e:
            if 400 <= e.status_code < 500:
                logger.critical(
                    "Oanda rejected the account credentials (HTTP %d): "
                    "%s — refusing to start", e.status_code, e)
                raise SystemExit(
                    f"Oanda rejected the account credentials "
                    f"(HTTP {e.status_code}): {e}")
            logger.error(
                "Failed to fetch account summary at boot (HTTP %d): %s",
                e.status_code, e)
    def _asyncio_exception_handler(self, loop, context) -> None:
        exc = context.get("exception")
        if exc is not None:
            logger.critical(
                "Uncaught asyncio exception: %s",
                context.get("message", ""), exc_info=exc)
        else:
            logger.critical(
                "Uncaught asyncio error: %s", context.get("message", ""))
    def _request_shutdown(self) -> None:
        logger.info("Shutdown requested")
        self._running = False
        for task in asyncio.all_tasks():
            task.cancel()
    def _shutdown(self) -> None:
        with self._state_lock:
            _save_state(self._state)
        self._thread_pool.shutdown(wait=False)
        logger.info("=== Oanda Trading Daemon stopped ===")
    # ── Position discovery (restart recovery) ──────────────────────
    def _discover_existing_positions(self) -> None:
        self._reconcile_positions("boot discovery")
    def _reconcile_positions(self, reason: str) -> None:
        """Re-sync tracked state with the broker's open trades. Safe to
        call mid-run (lock-guarded); used at boot and whenever a fill or
        placement error suggests state and broker may have diverged."""
        try:
            open_trades = self._executor.get_open_trades(INSTRUMENT)
        except OandaAPIError:
            logger.exception(
                "Failed to query open trades (%s) — reconcile skipped",
                reason)
            return
        live_ids = {t["id"] for t in open_trades}
        with self._state_lock:
            saved = self._state.get("trades", {})
            stale = [tid for tid in saved if tid not in live_ids]
            for tid in stale:
                logger.info(
                    "Stale trade %s in state file (closed on Oanda), "
                    "removing", tid)
                del saved[tid]
            for trade in open_trades:
                tid = trade["id"]
                if tid in saved:
                    logger.info(
                        "Recovered trade %s from state: %s %s @ %.2f, SL %.2f",
                        tid, saved[tid].get("direction"), INSTRUMENT,
                        saved[tid].get("entry"), saved[tid].get("current_sl"))
                    continue
                direction = ("BUY" if float(trade.get("currentUnits", 0)) > 0
                             else "SELL")
                entry = float(trade.get("price", 0))
                sl_order = trade.get("stopLossOrder") or {}
                sl_price = sl_order.get("price")
                if sl_price is None:
                    # No broker-side stop at all: risk is undefined, so the
                    # trailing model cannot manage this trade. Scream — a
                    # naked position needs a human (or a policy) to set a
                    # stop; fabricating one here would invent trading
                    # behavior the books never validated.
                    logger.critical(
                        "Adopted trade %s (%s @ %.2f) has NO STOP LOSS on "
                        "the broker — it will NOT be trailed. Set a stop "
                        "manually or close it.", tid, direction, entry)
                    sl = entry
                    risk = 0.0
                else:
                    sl = float(sl_price)
                    risk = abs(entry - sl)
                saved[tid] = {
                    "tf": "H1",
                    "direction": direction,
                    "entry": entry,
                    "sl_initial": sl,
                    "current_sl": sl,
                    "risk": risk,
                    "peak": entry,
                    "trail_armed": False,
                    "trail_arm_r": 0.6,
                    "trail_dist_r": 0.75,
                    "bars_held": 0,
                    "last_bar_ts": None,
                    "signal_bar_ts": None,
                }
                logger.info(
                    "Adopted unknown trade %s: %s @ %.2f, SL %.2f, risk %.2f "
                    "(conservative defaults)",
                    tid, direction, entry, sl, risk)
            self._state["trades"] = saved
            _save_state(self._state)
        logger.info(
            "Position reconcile (%s): %d active trade(s)", reason, len(saved))
    # ── Candle-boundary loop ───────────────────────────────────────
    async def _candle_loop(self, tf: Timeframe, interval_minutes: int) -> None:
        label = tf.value
        try:
            while self._running:
                wait = _seconds_until_boundary(interval_minutes, buffer=10)
                logger.info("%s loop: next scan in %.0fs", label, wait)
                await asyncio.sleep(wait)
                if not self._running:
                    break
                logger.info("%s loop: firing scan", label)
                try:
                    await self._scan_and_manage(tf)
                except Exception:
                    logger.exception("%s loop: cycle failed", label)
        except asyncio.CancelledError:
            pass
        finally:
            logger.info("%s loop stopped", label)
    async def _scan_and_manage(self, tf: Timeframe) -> None:
        loop = asyncio.get_running_loop()
        # Trade management FIRST: the SL ratchet must not wait on (or be
        # starved by) the signal scan — news fetches can take seconds and
        # a failed scan must never cost an open trade its trail update.
        try:
            await loop.run_in_executor(
                self._thread_pool, self._manage_open_trades, tf)
        except Exception:
            logger.exception("%s: trade management failed", tf.value)
        report = await loop.run_in_executor(
            self._thread_pool, self._run_signal, tf)
        if not self._should_execute(report, tf):
            return
        await loop.run_in_executor(
            self._thread_pool, self._execute_trade, report, tf)
    def _manage_open_trades(self, tf: Timeframe) -> None:
        self._check_closed_trades()
        self._manage_trailing(tf)
    # ── Heartbeat ─────────────────────────────────────────────────
    def _heartbeat_once(self) -> None:
        with self._state_lock:
            n = len(self._state.get("trades", {}))
        try:
            self._executor.get_account_summary()
        except OandaAPIError as e:
            logger.error("HEARTBEAT: account summary failed: %s", e)
        # Logged unconditionally — even on a failed summary call the
        # heartbeat line itself must reach the log, so a wedged process
        # doesn't look identical to a healthy but silent one.
        logger.info("HEARTBEAT: alive | tracked trades: %d", n)
    async def _heartbeat_loop(self) -> None:
        loop = asyncio.get_running_loop()
        try:
            while self._running:
                await asyncio.sleep(HEARTBEAT_SECONDS)
                if not self._running:
                    break
                try:
                    await loop.run_in_executor(
                        self._thread_pool, self._heartbeat_once)
                except Exception:
                    logger.exception("heartbeat loop: cycle failed")
        except asyncio.CancelledError:
            pass
        finally:
            logger.info("heartbeat loop stopped")
    # ── Signal pipeline ────────────────────────────────────────────
    def _run_signal(self, tf: Timeframe) -> SignalReport:
        conf_tfs = SCAN_CONFIRMATIONS.get(
            tf, [Timeframe.M15, Timeframe.H4])
        factory = PAIR_PRODUCERS.get(SYMBOL)
        if factory is not None:
            config = factory(tf, conf_tfs)
        else:
            config = PlatformConfig()
            config.primary_timeframe = tf
            config.confirmation_timeframes = conf_tfs
        aggregator = SignalAggregator(config)
        aggregator.data_provider = self._provider
        report = aggregator.analyze_pair(symbol=SYMBOL)
        logger.info(
            "%s scan: signal=%s score=%.1f conf=%.0f quality=%s "
            "regime=%s vol=%s price=%.2f spread=%.4f",
            tf.value,
            report.final_signal.name, report.final_score,
            report.final_confidence, report.signal_quality,
            report.market_regime, report.volatility_regime,
            report.current_price,
            self._provider.get_spread(SYMBOL))
        # Stale-data/suppression warnings must be visible at WARNING;
        # routine session-filter notices go to DEBUG — the M5 loop fires
        # 288x/day and would drown the log at WARNING otherwise.
        for w in report.warnings:
            if "STALE" in w.upper() or "FAIL" in w.upper():
                logger.warning("%s scan: %s", tf.value, w)
            else:
                logger.debug("%s scan: %s", tf.value, w)
        return report
    # ── Execution decision ─────────────────────────────────────────
    def _should_execute(self, report: SignalReport, tf: Timeframe) -> bool:
        if report.final_signal == SignalStrength.NEUTRAL:
            return False
        if report.trade_setup is None:
            return False
        quality_rank = QUALITY_RANK.get(report.signal_quality, -1)
        gate_rank = QUALITY_RANK.get(self._quality_gate, 1)
        if quality_rank < gate_rank:
            return False
        if self._in_cooldown(tf):
            logger.info(
                "%s: actionable signal (%s %s) but inside %d-bar entry "
                "cooldown, skipping",
                tf.value, report.trade_setup.direction,
                report.signal_quality, COOLDOWN_BARS)
            return False
        with self._state_lock:
            if self._state.get("trades"):
                logger.info(
                    "%s: actionable signal (%s %s) but position already "
                    "open, skipping",
                    tf.value, report.trade_setup.direction,
                    report.signal_quality)
                return False
        return True
    def _in_cooldown(self, tf: Timeframe) -> bool:
        """Backtest parity: no entry within COOLDOWN_BARS bars of the
        previous entry on the same timeframe (wall-clock approximation)."""
        with self._state_lock:
            last = self._state.get("last_entries", {}).get(tf.value)
        if not last:
            return False
        try:
            last_ts = datetime.fromisoformat(last)
        except ValueError:
            return False
        interval_s = TF_INTERVAL_MINUTES[tf] * 60
        elapsed = (datetime.now(timezone.utc) - last_ts).total_seconds()
        return elapsed < COOLDOWN_BARS * interval_s
    # ── Trade execution ────────────────────────────────────────────
    def _execute_trade(self, report: SignalReport, tf: Timeframe) -> None:
        ts = report.trade_setup
        if ts is None:
            return
        # All four TF loops can fire at an aligned boundary (e.g. :00) and
        # pass _should_execute before any of them has recorded a fill —
        # re-check under the entry lock so only one order goes out.
        with self._entry_lock:
            with self._state_lock:
                if self._state.get("trades"):
                    logger.info(
                        "%s: position opened by a concurrent scan, skipping",
                        tf.value)
                    return
            # NO take-profit on the order: the backtested trailing model
            # exits via SL/TRAIL/TIMEOUT only — a hard TP would cap
            # winners and diverge the live book from the backtest.
            try:
                result = self._executor.place_market_order(
                    instrument=INSTRUMENT,
                    direction=ts.direction,
                    stop_loss=ts.stop_loss,
                )
            except OandaAPIError:
                # The order is never retried (a timeout after a server-side
                # fill would double-up on retry), but the server MAY have
                # filled before the error reached us — reconcile so a live
                # position can't stay untracked until the next restart.
                logger.exception(
                    "Failed to place %s order for %s %s — reconciling "
                    "broker positions",
                    ts.direction, SYMBOL, tf.value)
                self._reconcile_positions("order placement error")
                return
            fill_tx = result.get("orderFillTransaction")
            if not fill_tx:
                logger.warning(
                    "Order not filled for %s %s", SYMBOL, tf.value)
                return
            trade_opened = fill_tx.get("tradeOpened", {})
            trade_id = trade_opened.get("tradeID")
            if not trade_id:
                # A fill with no tradeOpened means the order interacted
                # with an existing broker position (tradesClosed /
                # tradeReduced on a netting account) — state and broker
                # have diverged. Surface it and re-sync instead of
                # silently forgetting a real fill.
                logger.critical(
                    "Order FILLED but opened no trade (netting close/"
                    "reduce?) — fill: closed=%s reduced=%s. Reconciling.",
                    fill_tx.get("tradesClosed"), fill_tx.get("tradeReduced"))
                self._reconcile_positions("fill without tradeOpened")
                return
            fill_price = float(fill_tx.get("price", ts.entry_price))
            risk = abs(fill_price - ts.stop_loss)
            trail_arm = ts.trail_arm_r if ts.trail_arm_r is not None else 0.6
            trail_dist = ts.trail_dist_r if ts.trail_dist_r is not None else 0.75
            now_iso = datetime.now(timezone.utc).isoformat()
            trade_state = {
                "tf": tf.value,
                "direction": ts.direction,
                "entry": fill_price,
                "sl_initial": ts.stop_loss,
                "current_sl": ts.stop_loss,
                "tp1": ts.take_profit_1,
                "tp2": ts.take_profit_2,
                "risk": risk,
                "peak": fill_price,
                "trail_armed": False,
                "trail_arm_r": trail_arm,
                "trail_dist_r": trail_dist,
                "bars_held": 0,
                # The signal bar is already closed; the first bar this
                # trade lives through is the one that just started. Guard
                # so a lagging feed can't trail on the signal bar itself.
                "signal_bar_ts": report.timestamp,
                "last_bar_ts": None,
                "strategy": ts.strategy,
                "quality": report.signal_quality,
                "rr": ts.risk_reward,
                "opened_at": now_iso,
            }
            with self._state_lock:
                self._state["trades"][trade_id] = trade_state
                self._state["last_entries"][tf.value] = now_iso
                _save_state(self._state)
        logger.info(
            "TRADE OPENED: %s %s @ %.2f | SL %.2f | TP1 %.2f (advisory) | "
            "risk $%.2f | RR %.2f | quality %s | strategy %s | TF %s | "
            "trail arm %.2fR dist %.2fR",
            ts.direction, SYMBOL, fill_price, ts.stop_loss,
            ts.take_profit_1,
            risk, ts.risk_reward, report.signal_quality,
            ts.strategy, tf.value, trail_arm, trail_dist)
    # ── Trailing stop management ───────────────────────────────────
    def _manage_trailing(self, tf: Timeframe) -> None:
        # Copy the per-trade dicts too: the loop mutates info (peak,
        # bars_held, last_bar_ts) and must not touch live state outside
        # the lock — another thread may be serializing it in _save_state.
        with self._state_lock:
            trades = {tid: dict(info)
                      for tid, info in self._state.get("trades", {}).items()}
        for trade_id, info in trades.items():
            if info.get("tf") != tf.value:
                continue
            try:
                df = self._provider.fetch_ohlcv(SYMBOL, tf)
            except Exception:
                logger.exception(
                    "Failed to fetch candle data for trail update")
                continue
            if df.empty:
                logger.warning(
                    "%s: no candle data for trail update of trade %s",
                    tf.value, trade_id)
                continue
            bar_ts = df.index[-1].isoformat()
            # Process each closed bar exactly once. Weekends and feed lag
            # re-serve the same last bar; without this guard bars_held
            # inflates (an M5 trade would "time out" over a weekend) and
            # the peak gets double-counted.
            if bar_ts == info.get("last_bar_ts"):
                continue
            sig_ts = info.get("signal_bar_ts")
            if sig_ts:
                try:
                    if df.index[-1] <= pd.Timestamp(sig_ts):
                        continue
                except (ValueError, TypeError):
                    pass
            info["last_bar_ts"] = bar_ts
            info["bars_held"] = info.get("bars_held", 0) + 1
            if info["bars_held"] >= self._max_hold_bars:
                logger.info(
                    "TIMEOUT: trade %s held %d bars, closing",
                    trade_id, info["bars_held"])
                try:
                    self._executor.close_trade(trade_id)
                except OandaAPIError:
                    # Keep tracking — the trade is still live on the
                    # broker (incl. a 200 whose close order was cancelled,
                    # e.g. MARKET_HALTED — close_trade raises on no-fill);
                    # we retry the close at the next bar.
                    logger.exception(
                        "Failed to close timed-out trade %s — will retry",
                        trade_id)
                    with self._state_lock:
                        if trade_id in self._state["trades"]:
                            self._state["trades"][trade_id] = info
                            _save_state(self._state)
                    continue
                with self._state_lock:
                    self._state["trades"].pop(trade_id, None)
                    _save_state(self._state)
                continue
            last_bar = df.iloc[-1]
            bar_high = float(last_bar["High"])
            bar_low = float(last_bar["Low"])
            if info.get("risk", 0) <= 0:
                # Adopted naked position (no broker SL → risk undefined):
                # never manageable by the trailing model — keep it loud
                # instead of silently skipping forever.
                logger.warning(
                    "Trade %s has risk<=0 (no stop loss) — trailing "
                    "skipped; set a stop manually", trade_id)
            new_sl = self._compute_trail(info, bar_high, bar_low)
            if new_sl is not None:
                # Round to instrument precision and track exactly what the
                # broker holds, so tracked vs broker SL can never drift.
                new_sl = round(new_sl, self._price_precision)
                if new_sl != info["current_sl"]:
                    try:
                        self._executor.modify_trade_sl(trade_id, new_sl)
                        info["current_sl"] = new_sl
                    except OandaAPIError:
                        logger.exception(
                            "Failed to update trailing SL for trade %s",
                            trade_id)
            with self._state_lock:
                # Write back ONLY if the trade is still tracked — another
                # loop's _check_closed_trades may have popped it while we
                # iterated (broker-side close). Re-inserting would create
                # a ghost that blocks new entries.
                if trade_id in self._state["trades"]:
                    self._state["trades"][trade_id] = info
                    _save_state(self._state)
    def _compute_trail(
        self, info: Dict, bar_high: float, bar_low: float
    ) -> Optional[float]:
        buy = info["direction"] == "BUY"
        risk = info["risk"]
        if risk <= 0:
            return None
        arm = info.get("trail_arm_r", 0.6)
        dist = info.get("trail_dist_r", 0.75)
        if arm <= 0:
            return None
        fav = bar_high if buy else bar_low
        if buy:
            info["peak"] = max(info.get("peak", info["entry"]), fav)
        else:
            info["peak"] = min(info.get("peak", info["entry"]), fav)
        exc = ((info["peak"] - info["entry"]) if buy
               else (info["entry"] - info["peak"])) / risk
        if exc >= arm:
            if buy:
                lvl = info["peak"] - dist * risk
                cur_sl = info.get("current_sl", info["sl_initial"])
                if lvl > cur_sl:
                    info["trail_armed"] = True
                    return lvl
            else:
                lvl = info["peak"] + dist * risk
                cur_sl = info.get("current_sl", info["sl_initial"])
                if lvl < cur_sl:
                    info["trail_armed"] = True
                    return lvl
        return None
    # ── Closed-trade detection ─────────────────────────────────────
    def _check_closed_trades(self) -> None:
        with self._state_lock:
            tracked = set(self._state.get("trades", {}).keys())
        if not tracked:
            return
        try:
            open_trades = self._executor.get_open_trades(INSTRUMENT)
        except OandaAPIError:
            logger.exception("Failed to query open trades")
            return
        live_ids = {t["id"] for t in open_trades}
        closed = tracked - live_ids
        for tid in closed:
            with self._state_lock:
                info = self._state["trades"].pop(tid, {})
            entry = info.get("entry", 0)
            direction = info.get("direction", "?")
            risk = info.get("risk", 1)
            sl = info.get("current_sl", 0)
            was_trailed = info.get("trail_armed", False)
            logger.info(
                "TRADE CLOSED BY BROKER: %s %s %s | entry %.2f | "
                "SL was %.2f | risk $%.2f | trailed=%s | bars_held=%d",
                tid, direction, SYMBOL, entry, sl, risk,
                was_trailed, info.get("bars_held", 0))
        if closed:
            with self._state_lock:
                _save_state(self._state)
 # ── Uncaught exception handlers ───────────────────────────────────────
 def _excepthook(exc_type, exc_value, exc_tb):
    if issubclass(exc_type, KeyboardInterrupt):
        sys.__excepthook__(exc_type, exc_value, exc_tb)
        return
    logger.critical(
        "Uncaught exception", exc_info=(exc_type, exc_value, exc_tb))
 def _threading_excepthook(args):
    logger.critical(
        "Uncaught exception in thread %s",
        args.thread.name if args.thread else "?",
        exc_info=(args.exc_type, args.exc_value, args.exc_traceback))
 # ── Entry point ────────────────────────────────────────────────────────
 def main() -> None:
    _setup_logging(
        os.environ.get("OANDA_LOG_LEVEL") or get_app_config().log_level)
    sys.excepthook = _excepthook
    threading.excepthook = _threading_excepthook
    daemon = TradingDaemon()
    asyncio.run(daemon.run())
 if __name__ == "__main__":
    main()
@@ -0,0 +1,230 @@
 """
 ForexBot2 — Oanda v20 Order Executor
 ======================================
 Handles all Oanda account operations: market orders, trade queries,
 SL/TP modification, and trade closure.  Uses raw ``requests`` against
 the Oanda v20 REST API — no extra dependencies.
 """
 import logging
 import time
 from typing import Dict, List, Optional
 import requests
 logger = logging.getLogger(__name__)
 class OandaAPIError(Exception):
    """Raised when an Oanda API call fails."""
    def __init__(self, message: str, status_code: int = 0, body: str = ""):
        super().__init__(message)
        self.status_code = status_code
        self.body = body
 class OandaExecutor:
    """Oanda v20 order execution and trade management."""
    MAX_RETRIES = 1
    RETRY_DELAY = 2.0
    def __init__(
        self,
        account_id: str,
        api_token: str,
        base_url: str = "https://api-fxpractice.oanda.com",
        lot_size: int = 1,
        price_precision: int = 2,
    ):
        self._account_id = account_id
        self._base_url = base_url.rstrip("/")
        self._lot_size = lot_size
        self._price_precision = price_precision
        self._session = requests.Session()
        self._session.headers.update({
            "Authorization": f"Bearer {api_token}",
            "Accept-Datetime-Format": "RFC3339",
            "Content-Type": "application/json",
        })
    # ── Order Placement ────────────────────────────────────────────
    def place_market_order(
        self,
        instrument: str,
        direction: str,
        stop_loss: float,
        take_profit: Optional[float] = None,
        units: Optional[int] = None,
    ) -> Dict:
        if units is None:
            units = self._lot_size
        signed_units = str(units) if direction.upper() == "BUY" else str(-units)
        order_body = {
            "order": {
                "type": "MARKET",
                "instrument": instrument,
                "units": signed_units,
                "timeInForce": "FOK",
                # The book model is one position total; on a netting
                # account an opposite-side order would silently CLOSE an
                # existing (possibly untracked) position instead of opening
                # the signal trade. OPEN_ONLY makes that case fail loudly
                # (order cancelled) so the divergence surfaces instead of
                # executing an unintended close.
                "positionFill": "OPEN_ONLY",
                "stopLossOnFill": {
                    "price": self._fmt_price(stop_loss),
                },
            }
        }
        if take_profit is not None:
            order_body["order"]["takeProfitOnFill"] = {
                "price": self._fmt_price(take_profit),
            }
        url = f"{self._base_url}/v3/accounts/{self._account_id}/orders"
        # Never retry order placement: a timeout after the server already
        # filled would open a SECOND position on retry. Fail loudly instead;
        # the next candle boundary re-evaluates.
        resp = self._request_with_retry("POST", url, json=order_body, retries=0)
        data = self._parse_json(resp)
        fill_tx = data.get("orderFillTransaction")
        if fill_tx:
            trade_id = fill_tx.get("tradeOpened", {}).get("tradeID", "unknown")
            fill_price = float(fill_tx.get("price", 0))
            logger.info(
                "ORDER FILLED: %s %s %s units @ %.2f — trade %s, SL %.2f%s",
                direction.upper(), instrument, signed_units, fill_price,
                trade_id, stop_loss,
                f", TP {take_profit:.2f}" if take_profit else "")
        else:
            cancel_tx = data.get("orderCancelTransaction", {})
            reason = cancel_tx.get("reason", "unknown")
            logger.warning(
                "ORDER NOT FILLED: %s %s — reason: %s",
                direction.upper(), instrument, reason)
        return data
    # ── Trade Queries ──────────────────────────────────────────────
    def get_open_trades(self, instrument: Optional[str] = None) -> List[Dict]:
        url = f"{self._base_url}/v3/accounts/{self._account_id}/openTrades"
        resp = self._request_with_retry("GET", url)
        trades = self._parse_json(resp).get("trades", [])
        if instrument:
            trades = [t for t in trades if t.get("instrument") == instrument]
        return trades
    def get_trade(self, trade_id: str) -> Dict:
        url = (f"{self._base_url}/v3/accounts/{self._account_id}"
               f"/trades/{trade_id}")
        resp = self._request_with_retry("GET", url)
        return self._parse_json(resp).get("trade", {})
    def get_account_summary(self) -> Dict:
        url = f"{self._base_url}/v3/accounts/{self._account_id}/summary"
        resp = self._request_with_retry("GET", url)
        account = self._parse_json(resp).get("account", {})
        logger.info(
            "Account: balance=%.2f, NAV=%.2f, unrealizedPL=%.2f, openTrades=%s",
            float(account.get("balance", 0)),
            float(account.get("NAV", 0)),
            float(account.get("unrealizedPL", 0)),
            account.get("openTradeCount", 0))
        return account
    # ── Trade Modification ─────────────────────────────────────────
    def modify_trade_sl(self, trade_id: str, new_sl: float) -> Dict:
        url = (f"{self._base_url}/v3/accounts/{self._account_id}"
               f"/trades/{trade_id}/orders")
        body = {"stopLoss": {"price": self._fmt_price(new_sl)}}
        resp = self._request_with_retry("PUT", url, json=body)
        logger.info(
            "TRAIL UPDATE: trade %s — new SL %.2f", trade_id, new_sl)
        return self._parse_json(resp)
    def modify_trade_tp(self, trade_id: str, new_tp: float) -> Dict:
        url = (f"{self._base_url}/v3/accounts/{self._account_id}"
               f"/trades/{trade_id}/orders")
        body = {"takeProfit": {"price": self._fmt_price(new_tp)}}
        resp = self._request_with_retry("PUT", url, json=body)
        logger.info("TP UPDATE: trade %s — new TP %.2f", trade_id, new_tp)
        return self._parse_json(resp)
    def close_trade(self, trade_id: str) -> Dict:
        url = (f"{self._base_url}/v3/accounts/{self._account_id}"
               f"/trades/{trade_id}/close")
        resp = self._request_with_retry("PUT", url, json={"units": "ALL"})
        data = self._parse_json(resp)
        close_tx = data.get("orderFillTransaction")
        if not close_tx:
            # A 200 whose close order was created but NOT filled (e.g.
            # cancelled with MARKET_HALTED during the settlement pause)
            # means the position is STILL OPEN. Raising keeps the caller
            # tracking it and retrying — silently returning here would
            # orphan a live position.
            reason = data.get("orderCancelTransaction", {}).get(
                "reason", "no orderFillTransaction in response")
            raise OandaAPIError(
                f"Close of trade {trade_id} did not fill: {reason}",
                resp.status_code, str(data)[:300])
        close_price = float(close_tx.get("price", 0))
        pl = float(close_tx.get("pl", 0))
        logger.info(
            "TRADE CLOSED: trade %s @ %.2f — PL %.2f",
            trade_id, close_price, pl)
        return data
    # ── Internals ──────────────────────────────────────────────────
    @staticmethod
    def _parse_json(resp: requests.Response) -> Dict:
        try:
            return resp.json()
        except ValueError as e:
            raise OandaAPIError(
                f"Oanda returned a non-JSON response (HTTP {resp.status_code})",
                resp.status_code, resp.text[:300]) from e
    def _fmt_price(self, price: float) -> str:
        # Oanda rejects prices with more decimals than the instrument's
        # displayPrecision (XAU_USD: sub-cent precision is invalid).
        return f"{price:.{self._price_precision}f}"
    def _request_with_retry(
        self, method: str, url: str, retries: Optional[int] = None, **kwargs
    ) -> requests.Response:
        if retries is None:
            retries = self.MAX_RETRIES
        last_err = None
        for attempt in range(1 + retries):
            try:
                resp = self._session.request(method, url, timeout=15, **kwargs)
                if resp.status_code >= 500:
                    raise OandaAPIError(
                        f"Oanda server error {resp.status_code}",
                        resp.status_code, resp.text)
                if resp.status_code >= 400:
                    raise OandaAPIError(
                        f"Oanda API error {resp.status_code}: {resp.text}",
                        resp.status_code, resp.text)
                return resp
            except (requests.ConnectionError, requests.Timeout,
                    OandaAPIError) as e:
                last_err = e
                if isinstance(e, OandaAPIError) and 400 <= e.status_code < 500:
                    raise
                if attempt < retries:
                    logger.warning(
                        "Oanda request failed (attempt %d/%d): %s — retrying in %.0fs",
                        attempt + 1, 1 + retries, e,
                        self.RETRY_DELAY)
                    time.sleep(self.RETRY_DELAY)
        raise OandaAPIError(f"Oanda request failed after retries: {last_err}")
@@ -12,7 +12,7 @@ from typing import Optional, List, Dict, Callable
 from config import PlatformConfig, Timeframe
 from signal_aggregator import SignalAggregator, SignalReport
-from data_provider import ForexDataProvider
+from providers import BaseDataProvider
 PAIR_PRODUCERS: Dict[str, Callable] = {}
@@ -28,7 +28,7 @@ def get_supported_pairs() -> List[str]:
 def produce_signal(
    symbol: str,
    primary_tf: Timeframe,
-    data_provider: ForexDataProvider,
+    data_provider: BaseDataProvider,
    confirmation_tfs: Optional[List[Timeframe]] = None,
 ) -> SignalReport:
    config_factory = PAIR_PRODUCERS.get(symbol)
@@ -45,8 +45,6 @@ def produce_signal(
 from pair_signals import xauusd as _xau_mod
-from pair_signals import eurusd as _eur_mod
+from config import XAUUSD
 from config import XAUUSD, EURUSD
 register_pair(XAUUSD, _xau_mod.make_config)
 register_pair(EURUSD, _eur_mod.make_config)
@@ -1,88 +0,0 @@
 """EUR/USD signal producer config."""
 from typing import Optional, List
 from config import PlatformConfig, Timeframe, EURUSD
 _CONFIRMATIONS = {
    Timeframe.M5:  [Timeframe.M15, Timeframe.H1],
    Timeframe.M15: [Timeframe.M5, Timeframe.H1],
    Timeframe.H1:  [Timeframe.M15, Timeframe.H4],
    Timeframe.H4:  [Timeframe.H1, Timeframe.D1],
 }
 _TRENDING = {
    "ema_trend": 2.0, "rsi": 1.0, "macd": 2.0,
    "bollinger": 0.5, "keltner": 1.0, "roc": 1.5,
    "candlestick": 0.5, "divergence": 2.0, "adx": 1.0,
 }
 EUR_REGIME_WEIGHTS = {
    "trending_up": _TRENDING,
    "trending_down": _TRENDING,
    "trending": _TRENDING,
    "ranging": {
        "ema_trend": 0.3, "rsi": 2.0, "macd": 1.0,
        "bollinger": 2.0, "keltner": 1.5, "roc": 0.5,
        "candlestick": 0.8, "divergence": 2.5, "adx": 0.5,
    },
    "transitioning": {
        "ema_trend": 1.5, "rsi": 1.5, "macd": 2.0,
        "bollinger": 1.0, "keltner": 1.5, "roc": 2.0,
        "candlestick": 0.8, "divergence": 2.0, "adx": 1.0,
    },
 }
 _M5_TRENDING = {
    "ema_trend": 1.5, "rsi": 1.0, "macd": 2.5,
    "bollinger": 0.5, "keltner": 2.0, "roc": 2.0,
    "candlestick": 0.3, "divergence": 1.0, "adx": 1.0,
 }
 M5_REGIME_WEIGHTS = {
    "trending_up": _M5_TRENDING,
    "trending_down": _M5_TRENDING,
    "trending": _M5_TRENDING,
    "ranging": {
        "ema_trend": 0.3, "rsi": 1.5, "macd": 1.5,
        "bollinger": 1.5, "keltner": 2.0, "roc": 1.0,
        "candlestick": 0.3, "divergence": 1.5, "adx": 0.5,
    },
    "transitioning": {
        "ema_trend": 1.0, "rsi": 1.0, "macd": 2.5,
        "bollinger": 0.5, "keltner": 2.5, "roc": 2.0,
        "candlestick": 0.3, "divergence": 1.0, "adx": 1.0,
    },
 }
 def make_config(
    primary_tf: Timeframe,
    confirmation_tfs: Optional[List[Timeframe]] = None,
 ) -> PlatformConfig:
    config = PlatformConfig()
    config.symbol = EURUSD
    config.display_name = "EUR/USD"
    config.pip_value = 0.0001
    config.primary_timeframe = primary_tf
    config.confirmation_timeframes = (
        confirmation_tfs or _CONFIRMATIONS.get(primary_tf, [Timeframe.M15, Timeframe.H4])
    )
    config.xau_strategy.enabled = False
    config.backtest.session_start_hour = 7
    config.backtest.session_end_hour = 20
    config.backtest.blocked_hours_utc = [22, 23, 0, 1, 2, 3]
    config.backtest.allowed_weekdays = [0, 1, 2, 3, 4]
    if primary_tf == Timeframe.M5:
        config.technical.regime_weights_override = M5_REGIME_WEIGHTS
        config.eur_strategy.sl_atr_floor = 0.8
        config.eur_strategy.sl_atr_multiplier = 0.3
        config.eur_strategy.min_quality_grades = ["A+", "A", "B"]
        config.eur_strategy.min_entry_score = 28.0
        config.eur_strategy.min_entry_confidence = 50.0
    else:
        config.technical.regime_weights_override = EUR_REGIME_WEIGHTS
    return config
@@ -1,7 +1,175 @@
-"""XAU/USD signal producer config."""
+"""XAU/USD signal producer config.
 PlatformConfig defaults ARE the tuned XAU/USD configuration, so this
 factory sets the timeframes plus documented per-timeframe overrides.
 It must stay equivalent to the inline PlatformConfig(symbol=XAUUSD,
 ...) construction (plus those overrides) used by the backtester and
 Telegram bot (enforced by a test in test_all.py).
 M15 override (validated 2026-07-11, window 2025-03-30 -> 2026-05-31,
 $0.30 spread, trailing exits): tp_floor_rr 1.0 scores +116.6R/460
 trades PF 1.83 vs +105.4R/424 PF 1.82 at the 0.5 default — better in
 both train/val halves and on the 2024-06 -> 2025-03 out-of-sample
 window (+30.3R vs +10.6R; the 1R floor lifts targets past the min-TP
 spread gate, roughly doubling trades there). The same knob HURTS M5
 (pinned 12mo book: +131.3R/699 vs +139.9R/547, PF 1.55 vs 1.78), so it
 is per-timeframe, exactly as PENDING_STEPS' decision rule anticipated.
 M15 override 2 (validated 2026-07-11, 12mo window 2025-07-11 ->
 2026-07-11 + OOS 2024-06-01 -> 2025-03-30, $0.30 spread): block the
 mixed "trending" regime (ADX strong but EMAs unaligned — momentum
 without directional structure). Those trades lost -6.8R/24 on 12mo,
 negative in BOTH halves; blocking scores +65.7R/499 PF 1.38 DD 10.4R
 vs +59.0R/521 PF 1.32 DD 12.6R baseline, better in both halves. OOS
 is neutral (-1.4R = the 4 such trades that existed there; no power to
 confirm or refute). M5 never trades this regime (its gates already
 exclude it), so the block is M15-only. A second candidate — blocking
 transitioning-regime SELLs (-4.0R/47 on 12mo) — was REJECTED: those
 sells made +5.7R/32 in the 2024 OOS window (bull-market artifact).
 M15 override 3 (validated 2026-07-16, 5-year replay 2021-07-17 ->
 2026-07-16 under the audit-fixed backtester, $0.30 spread, trailing
 exits): block SELL entries whose bar CLOSES at 15:00 or 17:00 UTC
 (sell_blocked_hours_utc=[15, 17], gate 6g3 — BUYs unaffected; those
 hours flank the already-blocked hour 16). The pocket lost -33.5R/165
 trades, negative in 6/6 calendar years and in BOTH halves of the last
 12 months (H1 -6.8R, H2 -4.2R); each hour independently is 5/6 years
 negative with both halves negative (hours 14/18 fail that screen and
 stay open). Cross-TF the M5 book CONFIRMS (same pocket -14.1R/59,
 5/6 years negative) rather than contradicts — unlike the rejected
 trending_down block. Full replay with the gate: +176.7R/1627 PF 1.31
 DD 29.1R vs +144.9R/1781 PF 1.22 DD 34.4R control; better in ALL SIX
 years; last-12mo halves both improve (H1 +36.3 vs +29.5, H2 +46.9 vs
 +43.2; 12mo PF 1.55 vs 1.43); OOS 2024-06-01 -> 2025-03-30 improves
 (+45.4R PF 1.47 DD 17.8 vs +41.9R PF 1.39 DD 21.7). SELL side goes
 -1.2R -> +30.6R over 5 years. BUYs at those hours made +34.3R, hence
 SELL-only rather than a blocked_hours_utc entry. Mechanism: NY-session
 shorts against gold's macro uptrend (15:00 = post-US-data reversal
 window, 17:00 = London close unwind).
 M5 override 2 (2026-07-16, chart patterns): enable BUY-only CP on M5
 (inv_hs, double_bottom, bull_flag — drop all bearish patterns). SELL CP
 is consistently negative (OOS -3.21R, 5yr +3.76R PF 1.07) while BUY CP
 is positive across windows: 12mo +41.8R PF 1.60 (train +9.66 PF 1.30,
 val +32.17 PF 1.86), 5yr +65.2R PF 1.48, OOS +1.20R (aggregate
 positive, val half -1.89R — marginal but sign-correct in aggregate).
 Gold's structural uptrend makes bullish patterns fundamentally more
 reliable. M15/H1 fail on all windows; M5-only enablement.
 M5 override 3 (validated 2026-07-21, 5-year replay 2021-07-21 ->
 2026-07-21, $0.30 spread, next-open fills): trailing distance widened
 to 0.85R (trail_dist_r 0.85, arm unchanged 0.60). Same 30-cell grid
 methodology as the M15 campaign, opposite finding — on M5 the arm
 point barely matters; the winners need a looser leash. Plateau arm
 0.5-0.7 x dist 0.85-1.0 (+227..+234R vs +218.0R baseline). Chosen
 0.60/0.85 (OOS-preferred end of the plateau): +231.2R/2028 PF 1.30 WR
 57.1% vs +218.0R PF 1.29 WR 58.3%; both last-12mo halves better
 (+56.8/+115.1 vs +54.0/+104.6); 5yr max DD 19.2R vs 19.1R (noise);
 OOS 2024-06-01 -> 2025-03-30 +14.5R DD 15.1 vs +11.1R DD 15.4.
 dist 1.0 was REJECTED at the OOS gate (+10.9R < baseline +11.1R)
 despite the best in-sample total (+233.3R) — the OOS ordering picks
 the conservative plateau edge. XAU_PB trades are unaffected (they
 carry their own validated pb_trail 0.8/0.75 per-trade overrides);
 XAU_CP trades share the global params and were validated inside this
 book. M15 arms at 0.70 (its own campaign, same day); the divergence
 is the per-TF pattern yet again.
 M15 override 4 (validated 2026-07-21, 5-year replay 2021-07-21 ->
 2026-07-21, $0.30 spread, next-open fills): trailing-stop arm raised to
 0.70R (trail_arm_r 0.70, dist unchanged 0.75). Full grid arm 0.3-0.9 x
 dist 0.5-1.0 (30 cells): every earlier-arm cell (0.3/0.45) is WORSE —
 the "arm earlier to save fading trades" hypothesis is refuted; the
 plateau is arm 0.70-0.80 x dist 0.70-0.85 (all >= +190R). Chosen
 plateau center 0.70/0.75: +210.7R/1629 PF 1.35 WR 57.8% vs baseline
 0.60/0.75 +175.7R PF 1.30 WR 53.9%; both last-12mo halves improve
 (+49.7/+50.0 vs +36.4/+46.0); per-year better 4/6 (2022 -0.9R, 2023
 -1.4R, both within tolerance); OOS 2024-06-01 -> 2025-03-30 +57.4R
 PF 1.57 DD 17.0R vs +45.4R PF 1.47 DD 17.8R — better on every OOS
 metric. One clause waived BY OWNER DECISION 2026-07-21: 5yr max DD is
 33.9R vs 29.1R baseline (+4.8R deeper worst episode, 2022-23 chop),
 accepted in exchange for +20% total R. M5 keeps the global 0.6 arm
 (same per-TF divergence pattern as tp_floor_rr and the hour-9 block).
 M5 override (validated 2026-07-11, full-history replay 2020-03-17 ->
 2026-07-11 + 12mo book, $0.30 spread): block hour 9 UTC (post-London-
 open whipsaw). Hour-9 entries lost -28.9R/143 over 6 years, negative
 in 5/7 calendar years including winning years (2022 -9.2R). Blocking:
 +207.1R vs +178.3R baseline on the 6-year replay, max DD 28.7R vs
 35.5R, improves 5/7 years (2023 goes -13.0R -> breakeven); the tuned
 12mo book is unchanged (+139.0R vs +139.3R). M15-and-up keep hour 9:
 it is POSITIVE there (M15 6yr +1.7R, M15 OOS +8.6R), so the block is
 M5-only, same per-timeframe pattern as the M15 overrides.
 M5 override 2 — relaxation campaign (validated 2026-07-30, Oanda
 bars, $0.30 spread, trailing exits): UNBLOCK hour 21 UTC on M5 only,
 combined with the config.py M5 gate relaxations (m5_min_abs_score
 30->25, m5_min_confirmation_score 10->5). The combined config beat
 the pre-relaxation control in ALL 6 calendar years of the 5yr replay
 2021-07-30 -> 2026-07-30 (+230.9R/2319 PF 1.26 DD 21.3 vs
 +168.0R/2048 PF 1.21 DD 27.0), on 12mo (+107.4R PF 1.36, halves
 +68.6/+38.8 vs +84.6R PF 1.33), on OOS 2024-06-01 -> 2025-03-30
 (+26.1R vs +17.2R), and at $0.70 spread (+86.9R vs +66.4R). Hour-21
 trades alone: +28.0R/22, positive every year — overturning the old
 block per the full-evidence rule. Hour 21 stays BLOCKED on M15+
 (12mo M15 unblock tested -0.3R). Declined-but-open from the same
 campaign: an M5 hour-22 block (+4.8R on 12mo — the burst-heavy
 "institutional" hour — but -3.2R on 5yr and -1.1R on OOS: regime-
 specific); M15 hour-16 unblock (+0.8R/+57 trades 12mo but -18.3R
 DD +34% on 5yr); every SELL-gate/vol/TP-size relaxation retested
 negative (see memory notes for the full grid).
 M5 override 4 — XAU_PB disabled (validated 2026-08-04, Oanda bars,
 $0.65 REAL spread — first campaign validated at measured cost on the
 live feed, per the 2026-08-04 re-pin baseline): m5_pullback_enabled
 False on M5. Loss forensics on the Oanda books identified XAU_PB as
 the largest wrong-direction loss source (12mo: -43.5R of the -156.6R
 wrong-direction bucket; 5yr: -274.5R). Disabling: 12mo +96.6R/588
 PF 1.44 max DD 13.9R halves +51.2/+45.4 vs +82.3R/788 PF 1.26 DD
 23.9R shipped (which FAILS the PF>=1.3 clause at real cost); OOS
 2024-06-01 -> 2025-03-30 +9.5R PF 1.34; 5yr +123.9R/956 PF 1.35 vs
 PF 1.16 with PB. PB's 5yr net contribution (+23.6R) was almost
 entirely the 2022 chop year (+36.1R vs -2.2R without) — a regime
 earner that has been a drag on trend-regime bars since; its tight
 stops are also the most cost-punished at $0.65. PB code and the
 pb_trail_* params STAY for future regime-gated experiments (same
 pattern as cp_enabled). Rejected in the same campaign (4 candidates
 total, N logged): hour-19 block (+3.7R 12mo but -3.3R OOS, -1.5R
 5yr — OOS gate, the classic trap); hours-14/15 block (-6.9R 12mo
 once PB is removed — their raw negativity WAS the PB losses);
 trail_dist_r 0.75 (-2.6R 12mo; the 0.88R average give-back is the
 validated price of riding trends, exits confirmed NOT the loss
 driver: exit-management accounts for 1-2% of lost R vs 50%
 wrong-direction / 49% entry-timing).
 M5 override 5 — EMA partial-alignment squash (validated 2026-08-04,
 Oanda bars, $0.65 real spread): ema_partial_neutral True on M5 —
 partial 4-EMA ribbon alignment (2-3 of 4 checks) scores NEUTRAL
 instead of a weak BUY/SELL. Motivated by the indicator audit: partial-
 alignment trades averaged -0.008R/84 vs +0.230R/341 fully aligned.
 Replay (knob mirrored in the backtester's precomputed EMA scorer in
 the same commit — live==backtest parity): 12mo +97.6R/556 PF 1.48
 DD 10.7R halves +47.8/+49.8 vs control +96.6R/588 PF 1.44 DD 13.9R;
 5yr +127.4R/912 PF 1.38 DD 12.8R vs +123.9R/956 PF 1.35 DD 13.2R,
 2022 improves -2.2 -> -1.1; OOS 2024-06-01 -> 2025-03-30 +9.4R
 PF 1.33 vs +9.5R PF 1.34 — a one-trade tie, documented as such (the
 OOS window holds up; it neither prefers nor punishes). Net effect:
 32 fewer marginal trades, every risk-adjusted measure better.
 Rejected in the same round (trial-logged): confirmation-TF swap
 M5->[M15,H4] (12mo +104.0R PF 1.51 DD 10.3 but OOS +7.8R PF 1.27
 vs control +9.5R PF 1.34 and 2022 3x worse — the H4 confirm keeps
 endorsing stale direction in chop; OOS-ordering rejection, same
 precedent as trail dist 1.0); confirm [H1,H4] (-31.8R 12mo);
 triple-confirm [M15,H1,H4] (-4.9R); divergence weight -1/3
 (-3.8R 12mo); candlestick weight +50% (-3.0R 12mo). The weight
 vector keeps proving calibrated as a system: divergence/candlestick/
 bollinger all look mis-weighted in-book and all replay worse when
 "fixed" (bollinger weight-0: +80.9R PF 1.29 DD 23.8 — its constant
 oppose vote is a load-bearing quality bar, like ROC).
 """
 from typing import Optional, List
 from config import PlatformConfig, Timeframe
 from app_config import apply_pair_overrides
 _CONFIRMATIONS = {
    Timeframe.M5:  [Timeframe.M15, Timeframe.H1],
@@ -16,8 +184,46 @@ def make_config(
    confirmation_tfs: Optional[List[Timeframe]] = None,
 ) -> PlatformConfig:
    config = PlatformConfig()
    # Deployment overlay (config.json) BEFORE the validated per-TF
    # deltas below, so e.g. the M5 hour-9 append lands on the JSON base.
    apply_pair_overrides(config)
    config.primary_timeframe = primary_tf
    config.confirmation_timeframes = (
        confirmation_tfs or _CONFIRMATIONS.get(primary_tf, [Timeframe.M15, Timeframe.H4])
    )
    if primary_tf == Timeframe.M5:
        # Block hour 9 UTC on M5 only; UNBLOCK hour 21 on M5 only
        # (positive every year of the 5yr replay) — see docstring.
        config.xau_strategy.blocked_hours_utc = [
            h for h in config.xau_strategy.blocked_hours_utc if h != 21
        ] + [9]
        # Deliberately NOT mirrored into config.backtest.blocked_hours_utc:
        # that copy is the generic non-XAU fallback carrying the JSON base
        # only. Propagating a gold-validated per-TF delta there would leak
        # XAU-calibrated hours to a future pair (pair-isolation rule).
        # Every XAU consumer branches to the xau_strategy copy. Enforced
        # by the Module 13 test "M5 factory appends hour 9 on top of a
        # JSON base".
        # BUY-only chart patterns on M5 — see module docstring.
        config.xau_strategy.cp_enabled = True
        config.xau_strategy.cp_patterns = ["inv_hs", "double_bottom", "bull_flag"]
        # XAU_PB pullback entries OFF — largest wrong-direction loss
        # source at real cost on Oanda bars — see module docstring.
        config.xau_strategy.m5_pullback_enabled = False
        # Partial EMA-ribbon alignment scores NEUTRAL on M5 — its
        # trades carried no edge — see module docstring.
        config.technical.ema_partial_neutral = True
        # Wider trailing distance on M5 only — see module docstring.
        config.signals.trail_arm_r = 0.60
        config.signals.trail_dist_r = 0.85
    if primary_tf == Timeframe.M15:
        # 1R minimum target on M15 only — see module docstring.
        config.signals.tp_floor_rr = 1.0
        # Block the mixed "trending" regime on M15 only — see docstring.
        config.xau_strategy.blocked_regimes = ["trending"]
        # Block SELL entries at 15:00/17:00 UTC on M15 only — see docstring.
        config.xau_strategy.sell_blocked_hours_utc = [15, 17]
        # Later trailing-stop arm on M15 only — see module docstring.
        config.signals.trail_arm_r = 0.70
        config.signals.trail_dist_r = 0.75
    return config
@@ -0,0 +1,57 @@
 """
 ForexBot2 — Data-provider package
 ==================================
 One class per provider, all subclassing `BaseDataProvider` (which owns
 caching, validation, boundary guarantees, and range pagination — the
 behavior the signal pipeline depends on).
 Adding a provider = one module implementing `_fetch_raw` + one registry
 entry + one config.json `data_providers` block. The live candle feed is
 selected by config.json `data_providers.active`; the optional
 `data_providers.streaming` scalar selects a different provider for the
 tick stream (level-breach monitor). Registration alone changes nothing.
 """
 from typing import Dict, Optional
 from app_config import AppConfigError, get_app_config
 from config import PlatformConfig
 from providers.base import BaseDataProvider, DataIntegrityError, _is_forex_weekend
 from providers.twelve_data import TwelveDataProvider
 from providers.oanda import OandaDataError, OandaDataProvider
 from providers.tiingo import TiingoDataError, TiingoDataProvider
 #: config.json data_providers.active selects from here.
 PROVIDER_REGISTRY: Dict[str, type] = {
    "twelve_data": TwelveDataProvider,
    "oanda": OandaDataProvider,
    "tiingo": TiingoDataProvider,
 }
 def make_data_provider(config: PlatformConfig,
                       name: Optional[str] = None) -> BaseDataProvider:
    """Build a provider by name (default: config.json data_providers.active)."""
    if name is None:
        name = get_app_config().active_provider
    cls = PROVIDER_REGISTRY.get(name)
    if cls is None:
        raise AppConfigError(
            f'Unknown data provider "{name}"; '
            f"registered: {sorted(PROVIDER_REGISTRY)}")
    return cls(config)
 __all__ = [
    "BaseDataProvider",
    "DataIntegrityError",
    "OandaDataError",
    "OandaDataProvider",
    "TiingoDataError",
    "TiingoDataProvider",
    "TwelveDataProvider",
    "PROVIDER_REGISTRY",
    "make_data_provider",
    "_is_forex_weekend",
 ]
@@ -0,0 +1,532 @@
 """
 ForexBot2 — Provider base class
 ================================
 `BaseDataProvider` owns everything that must behave identically no matter
 where the bars come from: in-memory caching with candle-duration TTL, the
 closed-bar boundary guarantee, freshness checks, validation/cleaning, the
 forming-candle drop, and paginated date-range fetching for the backtester.
 A concrete provider implements only:
    _fetch_raw(symbol, timeframe, outputsize, start, end) -> pd.DataFrame
 returning a normalized frame (capitalized OHLCV columns, UTC tz-aware
 DatetimeIndex named "Datetime", oldest-first), plus optionally
 `get_spread` and the streaming-quote capability (`supports_streaming` +
 `iter_ticks`).
 The signal pipeline (engine + aggregator + backtester) consumes ONLY the
 public contract defined here, so swapping providers never touches signal
 code.
 """
 import logging
 import threading
 import time
 from abc import ABC, abstractmethod
 from datetime import datetime, timedelta
 from typing import AsyncIterator, Callable, Dict, Optional, Sequence, Tuple
 from zoneinfo import ZoneInfo
 import pandas as pd
 from config import PAIR_INFO, PlatformConfig, Timeframe, XAUUSD
 logger = logging.getLogger(__name__)
 # The gold/FX market week is defined in New York time (see
 # _is_forex_weekend); a fixed UTC hour is wrong for half the year.
 _NY_TZ = ZoneInfo("America/New_York")
 def _is_forex_weekend(now=None) -> bool:
    """Gold/FX market closed for the weekend.
    The market week is anchored to NEW YORK time — Friday 17:00 ET →
    Sunday 18:00 ET — not a fixed UTC hour. During US DST the feed
    actually stops at 21:00 UTC (measured on 5 years of Oanda bars:
    159/162 summer Fridays; winter close 22:00, winter reopen Sunday
    23:00 UTC). The old hardcoded Fri 22:00→Sun 22:00 UTC rule produced
    two false-"STALE DATA/feed lagging" hours every week (2026-08-04
    audit I43, fixed per owner decision).
    Kept as three deliberate copies (here, signal_aggregator.py,
    gpu_backtest._is_xau_blocked_by_calendar) — a Module 21 test asserts
    they agree hour-by-hour across DST transitions.
    """
    if now is None:
        now = pd.Timestamp.now(tz="UTC")
    ny = now.astimezone(_NY_TZ)
    wd = ny.weekday()
    if wd == 4:                      # Friday
        return ny.hour >= 17
    if wd == 5:                      # Saturday
        return True
    if wd == 6:                      # Sunday
        return ny.hour < 18
    return False
 class DataIntegrityError(Exception):
    """Raised when data fails validation checks."""
    pass
 class BaseDataProvider(ABC):
    """Shared OHLCV provider behavior; subclasses supply the raw fetch."""
    #: Registry name; overridden per provider.
    name: str = "base"
    #: Exception type this provider raises for fetch/validation failures.
    #: Provider-specific errors subclass DataIntegrityError so consumers
    #: can catch one type regardless of the active provider.
    DATA_ERROR = DataIntegrityError
    BARS_PER_DAY = {
        Timeframe.M1: 1440, Timeframe.M5: 288, Timeframe.M15: 96,
        Timeframe.M30: 48, Timeframe.H1: 24, Timeframe.H4: 6,
        Timeframe.D1: 1, Timeframe.W1: 0.143,
    }
    CANDLE_SECONDS = {
        Timeframe.M1: 60, Timeframe.M5: 300, Timeframe.M15: 900,
        Timeframe.M30: 1800, Timeframe.H1: 3600, Timeframe.H4: 14400,
        Timeframe.D1: 86400, Timeframe.W1: 604800,
    }
    #: Hard per-request bar cap shared by Twelve Data and Oanda.
    MAX_BARS_PER_REQUEST = 5000
    # Grace wait before the one retry when a fresh response at a candle
    # boundary is missing the just-closed bar (feed latency).
    FRESH_FETCH_RETRY_SECONDS = 3.0
    def __init__(self, config: PlatformConfig):
        self.config = config
        self._cache: Dict[str, pd.DataFrame] = {}
        self._cache_timestamps: Dict[str, float] = {}
        # Requested fetch depth per key: a cached short frame must not
        # satisfy a caller asking for a longer lookback.
        self._cache_outputsize: Dict[str, int] = {}
        # The bot shares one provider across a thread pool; cache eviction
        # is check-then-act and needs locking.
        self._cache_lock = threading.Lock()
        # Single-flight: one API fetch per (symbol, TF) at a time; other
        # threads wait and then read the cache.
        self._inflight: Dict[str, threading.Event] = {}
    # ── Abstract: the only provider-specific piece ─────────────────
    @abstractmethod
    def _fetch_raw(
        self,
        symbol: str,
        timeframe: Timeframe,
        outputsize: int,
        start: Optional[datetime] = None,
        end: Optional[datetime] = None,
    ) -> pd.DataFrame:
        """Fetch up to `outputsize` bars, optionally bounded by start/end
        (tz-aware UTC datetimes). Must return a normalized frame:
        capitalized OHLCV columns, UTC DatetimeIndex named "Datetime",
        oldest-first. Raise self.DATA_ERROR on failure."""
    # ── Public API ─────────────────────────────────────────────────
    def fetch_ohlcv(
        self,
        symbol: str = XAUUSD,
        timeframe: Timeframe = Timeframe.H1,
        lookback_days: Optional[int] = None,
    ) -> pd.DataFrame:
        """Fetch OHLCV with in-memory caching for live mode."""
        if lookback_days is None:
            lookback_days = self.config.data_lookback_days
        cache_key = f"{symbol}_{timeframe.value}"
        outputsize = max(
            300,
            min(int(lookback_days * self.BARS_PER_DAY[timeframe]) + 20,
                self.MAX_BARS_PER_REQUEST),
        )
        while True:
            now = time.time()
            cache_ttl = self.CANDLE_SECONDS.get(
                timeframe, self.config.refresh_interval)
            with self._cache_lock:
                cached = self._cache.get(cache_key)
                cached_at = self._cache_timestamps.get(cache_key)
                cached_size = self._cache_outputsize.get(cache_key, 0)
            if (
                cached is not None
                and cached_at is not None
                and (now - cached_at) < cache_ttl
                and not cached.empty
                # A caller asking for MORE history than the cached fetch
                # requested must not be served the short frame silently.
                and cached_size >= outputsize
            ):
                if (self._is_data_fresh(cached, symbol, timeframe)
                        and self._has_latest_closed_bar(cached, timeframe)):
                    return cached
                logger.warning(f"{symbol} {timeframe.value}: cached data is stale, refetching")
                with self._cache_lock:
                    self._cache.pop(cache_key, None)
                    self._cache_timestamps.pop(cache_key, None)
                    self._cache_outputsize.pop(cache_key, None)
            # Single-flight: concurrent scans for the same (symbol, TF)
            # must not each hit the API (the four TF loops fire together
            # at aligned boundaries and Oanda has no client throttle).
            # First caller claims the key and fetches; the rest wait,
            # then re-check the cache.
            with self._cache_lock:
                inflight = self._inflight.get(cache_key)
                if inflight is None:
                    self._inflight[cache_key] = threading.Event()
                    break
            inflight.wait(timeout=90)
        try:
            df = self._fetch_raw(symbol, timeframe, outputsize)
            df = self._drop_forming_candle(df, timeframe)
            df = self._validate_and_clean(df, symbol, timeframe)
            # The closed-bar guarantee must hold for FRESH responses too,
            # not just cache hits: the scanner fires seconds after the
            # boundary, and a lagging feed can answer without the
            # just-closed bar — the scan would silently analyze (and
            # session-check) the previous bar. Give the feed one short
            # grace retry, then proceed loudly.
            if not df.empty and not self._has_latest_closed_bar(df, timeframe):
                time.sleep(self.FRESH_FETCH_RETRY_SECONDS)
                try:
                    retry = self._fetch_raw(symbol, timeframe, outputsize)
                    retry = self._drop_forming_candle(retry, timeframe)
                    retry = self._validate_and_clean(retry, symbol, timeframe)
                    if retry is not None and not retry.empty:
                        df = retry
                except DataIntegrityError as e:
                    logger.warning(
                        f"{symbol} {timeframe.value}: boundary retry refetch "
                        f"failed ({e}) — keeping original data")
                if not self._has_latest_closed_bar(df, timeframe):
                    logger.warning(
                        f"{symbol} {timeframe.value}: feed lagging — just-closed bar "
                        f"still missing after retry; this scan may run one bar behind"
                    )
            if not self._is_data_fresh(df, symbol, timeframe):
                logger.warning(
                    f"STALE DATA from API: {symbol} {timeframe.value} — "
                    f"last bar {df.index[-1]} is too old. Signal quality may be degraded."
                )
            with self._cache_lock:
                self._cache[cache_key] = df
                self._cache_timestamps[cache_key] = now
                self._cache_outputsize[cache_key] = outputsize
            return df
        finally:
            # Release waiters even when the fetch raised — they re-check
            # the cache and (on a miss) claim the key themselves.
            with self._cache_lock:
                ev = self._inflight.pop(cache_key, None)
            if ev is not None:
                ev.set()
    def fetch_range(
        self,
        symbol: str,
        timeframe: Timeframe,
        start: datetime,
        end: datetime,
    ) -> pd.DataFrame:
        """Fetch the RAW closed-bar series between start/end, paginating
        under the per-request bar cap. Returns uncleaned bars (the
        backtester resolves trades on the raw price path); callers wanting
        the signal-side series run `validate_and_clean` on the result."""
        bars_per_day = self.BARS_PER_DAY[timeframe]
        days_per_chunk = max(1, int(4500 / bars_per_day))
        chunks = []
        cursor_end = end
        empty_streak = 0
        error_streak = 0
        while cursor_end > start:
            cursor_start = max(start, cursor_end - timedelta(days=days_per_chunk))
            try:
                chunk = self._fetch_raw(
                    symbol, timeframe, self.MAX_BARS_PER_REQUEST,
                    start=cursor_start, end=cursor_end,
                )
            except Exception as e:
                # A raising chunk is a PROVIDER ERROR (auth, 429, 5xx,
                # malformed payload) — not "no data". Three in a row means
                # the rest of the history is unreachable: raise instead of
                # returning a silently truncated frame (the backtester
                # would treat it as complete).
                logger.warning(
                    f"Chunk {cursor_start.date()}–{cursor_end.date()} failed: {e}"
                )
                error_streak += 1
                if error_streak >= 3:
                    raise self.DATA_ERROR(
                        f"{symbol} {timeframe.value}: {error_streak} "
                        f"consecutive chunk failures walking back from "
                        f"{cursor_end.date()} — refusing to return "
                        f"truncated history (last error: {e})") from e
                cursor_end = cursor_start - timedelta(seconds=1)
                continue
            if chunk is None or chunk.empty:
                # Genuinely empty windows (weekends/holidays/asset gaps)
                # are normal; several in a row = start of history.
                empty_streak += 1
                if empty_streak >= 3:
                    break
                cursor_end = cursor_start - timedelta(seconds=1)
                continue
            empty_streak = 0
            error_streak = 0
            chunks.append(chunk)
            # Step back; dedup on concat.
            new_cursor = chunk.index[0] - timedelta(seconds=1)
            if new_cursor >= cursor_end:
                # A provider ignoring start/end would pin the cursor and
                # loop forever — refuse to spin.
                logger.warning(
                    f"{symbol} {timeframe.value}: pagination made no "
                    f"progress at {cursor_end} — stopping")
                break
            cursor_end = new_cursor
            if len(chunk) < 2:
                break
        if not chunks:
            raise RuntimeError(f"No data fetched for {symbol} {timeframe.value}")
        df = pd.concat(chunks).sort_index()
        df = df[~df.index.duplicated(keep="last")]
        # Fetches ending "now" include the still-forming candle — drop it
        # so callers only ever see closed bars.
        df = self._drop_forming_candle(df, timeframe)
        start_ts = pd.Timestamp(start)
        end_ts = pd.Timestamp(end)
        df = df[(df.index >= start_ts) & (df.index <= end_ts)]
        # Coverage sanity: a head materially later than the requested
        # start means history was truncated (empty-streak bail or provider
        # depth limit). Loud, not fatal — some assets genuinely start late.
        if not df.empty and (df.index[0] - start_ts) > pd.Timedelta(days=7):
            logger.error(
                f"{symbol} {timeframe.value}: fetched history starts "
                f"{df.index[0]} but {start_ts} was requested — "
                f"~{(df.index[0] - start_ts).days} days missing from the "
                f"head; treat downstream results accordingly")
        return df
    def is_data_fresh(
        self, df: pd.DataFrame, symbol: str, timeframe: Timeframe
    ) -> bool:
        """Public freshness check (last bar within 3 candle durations)."""
        return self._is_data_fresh(df, symbol, timeframe)
    def validate_and_clean(
        self, df: pd.DataFrame, symbol: str, timeframe: Timeframe
    ) -> pd.DataFrame:
        """Public entry to the shared cleaning pipeline."""
        return self._validate_and_clean(df, symbol, timeframe)
    def clear_cache(self) -> int:
        """Wipe all cached data. Returns number of entries cleared."""
        with self._cache_lock:
            count = len(self._cache)
            self._cache.clear()
            self._cache_timestamps.clear()
        logger.info(f"Cache cleared: {count} entries wiped")
        return count
    def get_pair_info(self, symbol: str = XAUUSD) -> Dict:
        info = PAIR_INFO.get(symbol, PAIR_INFO[XAUUSD])
        return {"symbol": symbol, **info}
    def get_spread(self, symbol: str = XAUUSD) -> float:
        """Latest observed round-trip spread; providers without live
        bid/ask fall back to the pair's assumed default."""
        return PAIR_INFO.get(symbol, PAIR_INFO[XAUUSD])["default_spread"]
    # ── Streaming-quote capability (optional) ──────────────────────
    #: True when the provider can stream live ticks via `iter_ticks`.
    supports_streaming: bool = False
    def iter_ticks(
        self,
        symbols: Sequence[str],
        on_connect: Optional[Callable[[], None]] = None,
    ) -> AsyncIterator[Tuple[str, float]]:
        """Async generator yielding (canonical_symbol, price) ticks for one
        connection's lifetime; the caller owns reconnect/backoff. Providers
        that support it set `supports_streaming = True` and override."""
        raise NotImplementedError(
            f"{self.name} provider does not support streaming quotes")
    # ── Shared internals ───────────────────────────────────────────
    def _has_latest_closed_bar(self, df: pd.DataFrame, timeframe: Timeframe) -> bool:
        """Cached data must include the most recent fully closed bar.
        The TTL alone is wall-clock based: a snapshot fetched mid-candle is
        still within TTL at the next candle boundary but is missing the bar
        that just closed, so every boundary scan would analyze — and run the
        bar-close session check against — the previous bar.
        """
        if df is None or df.empty:
            return False
        now = pd.Timestamp.now(tz="UTC")
        if _is_forex_weekend(now):
            return True
        candle_seconds = self.CANDLE_SECONDS.get(timeframe, 3600)
        dur = pd.Timedelta(seconds=candle_seconds)
        expected_label = now.floor(dur) - dur
        return df.index[-1] >= expected_label
    def _drop_forming_candle(self, df: pd.DataFrame, timeframe: Timeframe) -> pd.DataFrame:
        """Drop the still-forming last candle so analysis only sees closed bars.
        Some feeds include the current partial candle; the scanner fires
        seconds after the boundary, so without this guard every indicator
        reads a seconds-old stub bar and signals repaint. Feeds that only
        return complete candles pass through unchanged.
        """
        if df is None or df.empty:
            return df
        candle_seconds = self.CANDLE_SECONDS.get(timeframe, 3600)
        now = pd.Timestamp.now(tz="UTC")
        # Loop, not a single slice: a misbehaving feed can return MORE
        # than one not-yet-closed bar, and a single iloc[:-1] would let
        # the residual forming bar through cleaning (future-tolerance is
        # +5 min) straight into the signal engine.
        dropped = 0
        while (not df.empty
               and df.index[-1] + pd.Timedelta(seconds=candle_seconds) > now):
            df = df.iloc[:-1]
            dropped += 1
        if dropped > 1:
            logger.warning(
                "%s: feed returned %d forming/future bars — all dropped",
                timeframe.value, dropped)
        return df
    def _is_data_fresh(self, df: pd.DataFrame, symbol: str, timeframe: Timeframe) -> bool:
        """Check if the last candle is within acceptable age for this timeframe."""
        if df is None or df.empty:
            return False
        now = pd.Timestamp.now(tz="UTC")
        if _is_forex_weekend(now):
            return True
        last_bar = df.index[-1]
        candle_seconds = self.CANDLE_SECONDS.get(timeframe, 3600)
        max_age_seconds = candle_seconds * 3
        age_seconds = (now - last_bar).total_seconds()
        if age_seconds > max_age_seconds:
            logger.warning(
                f"{symbol} {timeframe.value}: last bar {last_bar.strftime('%Y-%m-%d %H:%M')} "
                f"is {age_seconds / 60:.0f}m old (limit: {max_age_seconds / 60:.0f}m)"
            )
            return False
        return True
    def _validate_and_clean(
        self, df: pd.DataFrame, symbol: str, timeframe: Timeframe
    ) -> pd.DataFrame:
        if df is None or df.empty:
            raise self.DATA_ERROR(f"Empty dataset for {symbol}")
        original_len = len(df)
        df = df.dropna(subset=["Open", "High", "Low", "Close"], how="all")
        df = df.ffill(limit=3)
        df = df.dropna(subset=["Close"])
        df = df.sort_index()
        df = df[~df.index.duplicated(keep="last")]
        now = pd.Timestamp.utcnow()
        future_mask = df.index > now + pd.Timedelta(minutes=5)
        if future_mask.any():
            logger.warning(f"{symbol}: Dropping {future_mask.sum()} future timestamps")
            df = df[~future_mask]
        # Non-positive prices are corrupt, never real gold quotes. Reject
        # the row BEFORE the consistency repair below — repairing High/Low
        # from a corrupt 0.0 Open/Close would propagate the zero into
        # previously-valid fields (2026-08-04 audit, I45).
        nonpositive = (df[["Open", "High", "Low", "Close"]] <= 0).any(axis=1)
        if nonpositive.any():
            logger.warning(
                f"{symbol}: Dropping {nonpositive.sum()} bars with "
                f"non-positive prices (corrupt feed data)")
            df = df[~nonpositive]
            if df.empty:
                raise self.DATA_ERROR(
                    f"{symbol}: all bars had non-positive prices")
        invalid_ohlc = (
            (df["High"] < df["Low"]) |
            (df["High"] < df["Open"]) |
            (df["High"] < df["Close"]) |
            (df["Low"] > df["Open"]) |
            (df["Low"] > df["Close"])
        )
        if invalid_ohlc.any():
            logger.warning(f"{symbol}: Fixing {invalid_ohlc.sum()} OHLC-inconsistent bars")
            for idx in df[invalid_ohlc].index:
                vals = [df.loc[idx, "Open"], df.loc[idx, "Close"]]
                df.loc[idx, "High"] = max(vals + [df.loc[idx, "High"]])
                df.loc[idx, "Low"]  = min(vals + [df.loc[idx, "Low"]])
        # inf returns (division by a zero/na Close) would NaN the std and
        # silently disarm the whole 10σ pass — neutralize them first.
        returns = df["Close"].pct_change()
        returns = returns.replace([float("inf"), float("-inf")], float("nan"))
        std_ret = returns.std()
        if std_ret > 0:
            dev = returns - returns.mean()
            outlier_mask = dev.abs() > 10 * std_ret
            # A spike bar makes the NEXT bar's return look extreme too (it is
            # measured against the spike). If the previous bar is flagged and
            # this bar reverses direction, it is the legitimate snap-back —
            # keep it, or every spike would punch a two-bar hole in the data.
            recovery = (
                outlier_mask
                & outlier_mask.shift(1, fill_value=False)
                & (dev * dev.shift(1) < 0)
            )
            outlier_mask = outlier_mask & ~recovery
            if outlier_mask.any():
                logger.warning(f"{symbol}: Removing {outlier_mask.sum()} extreme outlier bars")
                df = df[~outlier_mask]
        min_required = max(
            self.config.technical.ema_trend,
            200,
        ) + 10
        if len(df) < min_required:
            logger.warning(
                f"{symbol} {timeframe.value}: Only {len(df)} bars "
                f"(need {min_required}). Some indicators will be partial."
            )
        cleaned = original_len - len(df)
        if cleaned > 0:
            logger.info(f"{symbol}: Cleaned {cleaned}/{original_len} bars")
        return df
@@ -0,0 +1,232 @@
 """
 ForexBot2 — Oanda v20 data provider
 ====================================
 OHLCV candles from the Oanda v20 REST API. All shared behavior (cache,
 validation, boundary guarantees, range pagination) lives in
 `BaseDataProvider`; this module only maps symbols/granularities, parses
 Oanda's candle JSON, and exposes the real bid/ask spread from MBA candles.
 """
 import logging
 import time
 from datetime import datetime
 from typing import Dict, Optional
 import pandas as pd
 import requests
 from app_config import oanda_api_token, oanda_base_url
 from config import PAIR_INFO, PlatformConfig, Timeframe, XAUUSD
 from providers.base import BaseDataProvider, DataIntegrityError
 logger = logging.getLogger(__name__)
 class OandaDataError(DataIntegrityError):
    """Raised when an Oanda data fetch fails or returns invalid data."""
 class OandaDataProvider(BaseDataProvider):
    """Oanda v20 REST candle provider."""
    name = "oanda"
    DATA_ERROR = OandaDataError
    RETRY_DELAY_S = 2.0
    OANDA_GRANULARITY = {
        Timeframe.M1: "M1",
        Timeframe.M5: "M5",
        Timeframe.M15: "M15",
        Timeframe.M30: "M30",
        Timeframe.H1: "H1",
        Timeframe.H4: "H4",
        Timeframe.D1: "D",
        Timeframe.W1: "W",
    }
    OANDA_INSTRUMENT = {
        "XAU/USD": "XAU_USD",
    }
    def __init__(
        self,
        config: PlatformConfig,
        api_token: Optional[str] = None,
        base_url: Optional[str] = None,
    ):
        super().__init__(config)
        # Credentials default to the config.json `oanda` block / env vars
        # so the provider registry can build this class with config only.
        if api_token is None:
            api_token = oanda_api_token()
        if base_url is None:
            base_url = oanda_base_url()
        self._base_url = base_url.rstrip("/")
        self._session = requests.Session()
        self._session.headers.update({
            "Authorization": f"Bearer {api_token}",
            "Accept-Datetime-Format": "RFC3339",
            "Content-Type": "application/json",
        })
        self._spread_cache: Dict[str, float] = {}
    # ── Public extras ──────────────────────────────────────────────
    def get_spread(self, symbol: str = XAUUSD) -> float:
        return self._spread_cache.get(
            symbol, PAIR_INFO.get(symbol, PAIR_INFO[XAUUSD])["default_spread"])
    # ── Raw fetch ──────────────────────────────────────────────────
    def _fetch_raw(
        self,
        symbol: str,
        timeframe: Timeframe,
        outputsize: int,
        start: Optional[datetime] = None,
        end: Optional[datetime] = None,
    ) -> pd.DataFrame:
        if start is not None or end is not None:
            return self._fetch_oanda_candles(
                symbol, timeframe, outputsize, start, end)
        return self._fetch_oanda_candles(symbol, timeframe, outputsize)
    def _to_oanda_instrument(self, symbol: str) -> str:
        if symbol in self.OANDA_INSTRUMENT:
            return self.OANDA_INSTRUMENT[symbol]
        return symbol.replace("/", "_")
    def _fetch_oanda_candles(
        self,
        symbol: str,
        timeframe: Timeframe,
        count: int,
        start: Optional[datetime] = None,
        end: Optional[datetime] = None,
    ) -> pd.DataFrame:
        instrument = self._to_oanda_instrument(symbol)
        granularity = self.OANDA_GRANULARITY.get(timeframe)
        if granularity is None:
            raise OandaDataError(
                f"Unsupported timeframe {timeframe} for Oanda")
        url = f"{self._base_url}/v3/instruments/{instrument}/candles"
        params = {
            "granularity": granularity,
            "price": "MBA",
            "dailyAlignment": 0,
            "alignmentTimezone": "Etc/UTC",
        }
        # v20 rejects `count` combined with both `from` and `to`; a date
        # window replaces it (used by fetch_range pagination).
        if start is not None and end is not None:
            params["from"] = pd.Timestamp(start).isoformat()
            params["to"] = pd.Timestamp(end).isoformat()
        else:
            params["count"] = min(count, self.MAX_BARS_PER_REQUEST)
        # One retry absorbs Oanda's sporadic spurious 401s (~15-20/day on
        # the practice API — the same token succeeds on adjacent requests)
        # plus transient 5xx/network faults. A real credential problem
        # fails both attempts and raises exactly as before. Other 4xx
        # (bad request, 429) are never retried.
        last_err: Optional[OandaDataError] = None
        for attempt in range(2):
            if attempt:
                time.sleep(self.RETRY_DELAY_S)
            try:
                resp = self._session.get(url, params=params, timeout=30)
            except requests.RequestException as e:
                last_err = OandaDataError(
                    f"Oanda candle fetch failed for {instrument}: {e}")
                last_err.__cause__ = e
                continue
            if resp.status_code == 401 or resp.status_code >= 500:
                last_err = OandaDataError(
                    f"Oanda candle fetch failed for {instrument}: "
                    f"HTTP {resp.status_code}: {resp.text[:300]}")
                if attempt == 0:
                    logger.warning(
                        f"Oanda fetch for {instrument} got HTTP "
                        f"{resp.status_code}; retrying in "
                        f"{self.RETRY_DELAY_S}s")
                continue
            break
        else:
            raise last_err
        if resp.status_code != 200:
            raise OandaDataError(
                f"Oanda candle fetch failed for {instrument}: "
                f"HTTP {resp.status_code}: {resp.text[:300]}")
        try:
            data = resp.json()
        except ValueError as e:
            raise OandaDataError(
                f"Oanda candle response for {instrument} is not valid "
                f"JSON: {e}") from e
        candles = data.get("candles", [])
        if not candles:
            raise OandaDataError(
                f"Empty candle response from Oanda for {instrument}")
        rows = []
        last_bid_close = None
        last_ask_close = None
        skipped_malformed = 0
        for c in candles:
            if not c.get("complete", False):
                continue
            mid = c.get("mid", {})
            bid = c.get("bid", {})
            ask = c.get("ask", {})
            # A candle missing any OHLC field must be SKIPPED, never
            # defaulted to 0.0: a zero price flows through cleaning as a
            # "legit" bar and poisons ATR/Keltner/SL math — and a
            # mid-frame zero Close makes pct_change() emit inf, which
            # NaNs the std and disarms the 10σ outlier filter for the
            # whole frame (2026-08-04 audit, I45).
            if any(mid.get(k) is None for k in ("o", "h", "l", "c")):
                skipped_malformed += 1
                continue
            rows.append({
                "Datetime": pd.Timestamp(c["time"]).tz_convert("UTC"),
                "Open": float(mid["o"]),
                "High": float(mid["h"]),
                "Low": float(mid["l"]),
                "Close": float(mid["c"]),
                "Volume": int(c.get("volume", 0)),
            })
            if bid and ask:
                last_bid_close = float(bid.get("c", 0))
                last_ask_close = float(ask.get("c", 0))
        if skipped_malformed:
            logger.warning(
                "Oanda returned %d malformed candle(s) for %s (missing "
                "OHLC fields) — skipped", skipped_malformed, instrument)
        if not rows:
            raise OandaDataError(
                f"No complete candles from Oanda for {instrument}")
        if last_bid_close and last_ask_close:
            self._spread_cache[symbol] = round(
                last_ask_close - last_bid_close, 4)
        df = pd.DataFrame(rows)
        df = df.set_index("Datetime")
        df = df.sort_index()
        if df.index.tz is None:
            df.index = df.index.tz_localize("UTC")
        df.index.name = "Datetime"
        for col in ("Open", "High", "Low", "Close"):
            df[col] = pd.to_numeric(df[col], errors="coerce")
        df["Volume"] = pd.to_numeric(df["Volume"], errors="coerce").fillna(0).astype(float)
        logger.debug(
            "Fetched %d bars for %s @ %s from Oanda",
            len(df), instrument, granularity)
        return df
@@ -0,0 +1,306 @@
 """
 ForexBot2 — Tiingo data provider
 =================================
 OHLCV candles from Tiingo's FX endpoint
 (`/tiingo/fx/{ticker}/prices`, resampled mid-price bars, no volume).
 All shared behavior (cache, validation, boundary guarantees, range
 pagination) lives in `BaseDataProvider`; this module only maps
 symbols/resample frequencies, sizes the request window, and parses
 Tiingo's JSON into the normalized frame.
 Every failure mode raises `TiingoDataError` (a `DataIntegrityError`)
 with a specific message — the scan loops log it and carry on; nothing
 fails silently. NOTE: candles and mid-price ticks only. Tiingo's free
 `/tiingo/fx/top` quote endpoint publishes indicative two-tier spreads
 and must not be used for spread display (removed 2026-07-23) —
 `get_spread` deliberately stays on the base-class default, and the
 WebSocket stream (`iter_ticks`) exposes the mid price only, never
 bid/ask or a derived spread.
 """
 import json
 import logging
 from datetime import datetime
 from typing import AsyncIterator, Callable, Optional, Sequence, Tuple
 import pandas as pd
 import requests
 from app_config import provider_api_key
 from config import PlatformConfig, Timeframe
 from providers.base import BaseDataProvider, DataIntegrityError
 logger = logging.getLogger(__name__)
 class TiingoDataError(DataIntegrityError):
    """Raised when a Tiingo data fetch fails or returns invalid data."""
 class TiingoDataProvider(BaseDataProvider):
    """Tiingo FX REST candle provider."""
    name = "tiingo"
    DATA_ERROR = TiingoDataError
    BASE_URL = "https://api.tiingo.com"
    TIINGO_RESAMPLE = {
        Timeframe.M1: "1min",
        Timeframe.M5: "5min",
        Timeframe.M15: "15min",
        Timeframe.M30: "30min",
        Timeframe.H1: "1hour",
        Timeframe.H4: "4hour",
        Timeframe.D1: "1day",
        # W1 intentionally unmapped: the FX endpoint has no weekly
        # resample — requesting it raises loudly instead of guessing.
    }
    TIINGO_TICKER = {
        "XAU/USD": "xauusd",
    }
    # Window padding over the theoretical bar span: FX is closed ~2/7 of
    # the week (Fri 22:00 → Sun 22:00 UTC), so a calendar window must be
    # wider than `bars × candle_seconds` to yield the requested count.
    WINDOW_PAD_FACTOR = 1.6
    def __init__(self, config: PlatformConfig, api_key: Optional[str] = None):
        super().__init__(config)
        # Credentials default to env TIINGO_API_KEY → config.json
        # data_providers.tiingo.api_key, so the provider registry can
        # build this class with config only.
        if api_key is None:
            api_key = provider_api_key(provider="tiingo")
        self._api_key = api_key or None
        if self._api_key is None:
            logger.warning(
                "Tiingo provider built WITHOUT an API key — every fetch "
                "will fail with a named error. Set the TIINGO_API_KEY env "
                "var or data_providers.tiingo.api_key in config.json.")
        self._session = requests.Session()
        self._session.headers.update({
            "Content-Type": "application/json",
            # Header auth keeps the key out of URLs (and out of logs).
            "Authorization": f"Token {api_key}" if api_key else "",
        })
    def _to_tiingo_ticker(self, symbol: str) -> str:
        if symbol in self.TIINGO_TICKER:
            return self.TIINGO_TICKER[symbol]
        return symbol.replace("/", "").replace("=X", "").lower()
    # ── Raw fetch ──────────────────────────────────────────────────
    def _fetch_raw(
        self,
        symbol: str,
        timeframe: Timeframe,
        outputsize: int,
        start: Optional[datetime] = None,
        end: Optional[datetime] = None,
    ) -> pd.DataFrame:
        if self._api_key is None:
            raise TiingoDataError(
                "No Tiingo API key. Set the TIINGO_API_KEY env var or "
                "data_providers.tiingo.api_key in config.json.")
        resample = self.TIINGO_RESAMPLE.get(timeframe)
        if resample is None:
            raise TiingoDataError(
                f"Unsupported timeframe {timeframe} for Tiingo "
                f"(no FX resample frequency)")
        ticker = self._to_tiingo_ticker(symbol)
        # Tiingo takes date windows, not bar counts: derive the start
        # from the requested count for live fetches.
        if start is None:
            span = self.CANDLE_SECONDS[timeframe] * (outputsize + 10)
            start_ts = (pd.Timestamp.now(tz="UTC")
                        - pd.Timedelta(seconds=span * self.WINDOW_PAD_FACTOR))
        else:
            start_ts = pd.Timestamp(start)
        params = {
            "resampleFreq": resample,
            "startDate": start_ts.strftime("%Y-%m-%dT%H:%M:%S"),
        }
        if end is not None:
            params["endDate"] = pd.Timestamp(end).strftime("%Y-%m-%dT%H:%M:%S")
        url = f"{self.BASE_URL}/tiingo/fx/{ticker}/prices"
        try:
            resp = self._session.get(url, params=params, timeout=30)
        except requests.RequestException as e:
            raise TiingoDataError(
                f"Tiingo candle fetch failed for {ticker}: {e}") from e
        if resp.status_code == 429:
            raise TiingoDataError(
                f"Tiingo rate limit hit for {ticker} (HTTP 429) — "
                f"plan quota exhausted; scan will retry next boundary")
        if resp.status_code in (401, 403):
            raise TiingoDataError(
                f"Tiingo auth failed for {ticker} (HTTP {resp.status_code}) "
                f"— check TIINGO_API_KEY / config.json api_key")
        if resp.status_code != 200:
            raise TiingoDataError(
                f"Tiingo candle fetch failed for {ticker}: "
                f"HTTP {resp.status_code}: {resp.text[:300]}")
        try:
            data = resp.json()
        except ValueError as e:
            raise TiingoDataError(
                f"Tiingo candle response for {ticker} is not valid "
                f"JSON: {e}") from e
        if not isinstance(data, list) or not data:
            raise TiingoDataError(
                f"Empty candle response from Tiingo for {ticker} "
                f"(window {params['startDate']} → "
                f"{params.get('endDate', 'now')}; check ticker and plan "
                f"history limits)")
        rows = []
        for item in data:
            if "date" not in item:
                raise TiingoDataError(
                    f"Malformed Tiingo candle for {ticker}: missing "
                    f"'date' in {str(item)[:120]}")
            rows.append({
                "Datetime": pd.Timestamp(item["date"]),
                "Open": item.get("open"),
                "High": item.get("high"),
                "Low": item.get("low"),
                "Close": item.get("close"),
                # FX endpoint has no volume; 0.0 keeps the engine contract.
                "Volume": 0.0,
            })
        df = pd.DataFrame(rows)
        df = df.set_index("Datetime")
        df = df.sort_index()
        if df.index.tz is None:
            df.index = df.index.tz_localize("UTC")
        else:
            df.index = df.index.tz_convert("UTC")
        df.index.name = "Datetime"
        for col in ("Open", "High", "Low", "Close"):
            df[col] = pd.to_numeric(df[col], errors="coerce")
        if df["Close"].isna().all():
            raise TiingoDataError(
                f"Tiingo response for {ticker} has no numeric closes — "
                f"feed format changed?")
        logger.debug(
            "Fetched %d bars for %s @ %s from Tiingo",
            len(df), ticker, resample)
        return df
    # ── Streaming quotes ───────────────────────────────────────────
    supports_streaming = True
    WS_URL = "wss://api.tiingo.com/fx"
    @staticmethod
    def _parse_ws_message(raw: str) -> Optional[Tuple[str, float]]:
        """Parse one WS message; return (tiingo_ticker, mid_price) for
        quote updates, None for heartbeats ("H"), subscription acks
        ("I"), malformed JSON, or quotes without a usable price.
        Server error messages ("E", e.g. bad token) RAISE TiingoDataError:
        Tiingo authenticates inside the socket via the subscribe message,
        so an auth failure arrives as a data message followed by a clean
        close — swallowing it would make the caller's reconnect loop
        hammer the endpoint with no backoff (backoff only grows on
        exceptions).
        """
        try:
            msg = json.loads(raw)
        except json.JSONDecodeError:
            logger.warning("Tiingo WS: malformed JSON, skipping")
            return None
        mtype = msg.get("messageType")
        if mtype == "H":
            return None
        if mtype == "I":
            logger.info("Tiingo WS: subscription acknowledged")
            return None
        if mtype == "E":
            raise TiingoDataError(
                f"Tiingo WS server error: {str(msg.get('response'))[:200]}")
        if mtype != "A":
            return None
        data = msg.get("data")
        if not isinstance(data, list) or len(data) < 8 or data[0] != "Q":
            return None
        # Live-verified field order (2026-08-03): ["Q", ticker, timestamp,
        # bidSize, bid, mid, askSize, ask] — ask is index 7, NOT 6 as the
        # docs' table suggests (index 6 is askSize).
        ticker = data[1]
        bid, mid, ask = data[4], data[5], data[7]
        if mid is None and bid is not None and ask is not None:
            mid = (bid + ask) / 2
        try:
            return ticker, float(mid)
        except (TypeError, ValueError):
            logger.warning("Tiingo WS: quote for %s has no usable price",
                           ticker)
            return None
    async def iter_ticks(
        self,
        symbols: Sequence[str],
        on_connect: Optional[Callable[[], None]] = None,
    ) -> AsyncIterator[Tuple[str, float]]:
        """One connection's lifetime of (canonical_symbol, mid) ticks.
        The caller owns the reconnect/backoff loop around this generator.
        Divergences from the Twelve Data reference implementation, both
        deliberate: `on_connect` fires on the FIRST RECEIVED message (the
        "I" ack in practice) rather than after the subscribe send, so a
        rejected token cannot reset the caller's backoff; and there is no
        client heartbeat task — Tiingo sends "H" frames itself and the
        websockets library's built-in ping keepalive detects dead sockets.
        """
        import websockets
        if self._api_key is None:
            raise TiingoDataError(
                "No Tiingo API key. Set the TIINGO_API_KEY env var or "
                "data_providers.tiingo.api_key in config.json.")
        ws_to_pair = {self._to_tiingo_ticker(s): s for s in symbols}
        async with websockets.connect(self.WS_URL) as ws:
            await ws.send(json.dumps({
                "eventName": "subscribe",
                "authorization": self._api_key,
                "eventData": {
                    "thresholdLevel": 5,
                    "tickers": sorted(ws_to_pair),
                },
            }))
            logger.info(
                "Tiingo WS connected, subscribing to %d tickers",
                len(ws_to_pair))
            connected = False
            async for raw in ws:
                parsed = self._parse_ws_message(raw)
                if not connected:
                    connected = True
                    if on_connect is not None:
                        on_connect()
                if parsed is None:
                    continue
                pair_symbol = ws_to_pair.get(parsed[0])
                if pair_symbol is None:
                    continue
                yield pair_symbol, parsed[1]
@@ -0,0 +1,247 @@
 """
 ForexBot2 — Twelve Data provider
 =================================
 OHLCV via the official `twelvedata` SDK plus streaming quotes over the
 Twelve Data WebSocket. All shared behavior (cache, validation, boundary
 guarantees, range pagination) lives in `BaseDataProvider`.
 """
 import asyncio
 import json
 import logging
 import threading
 import time
 from collections import deque
 from datetime import datetime
 from typing import AsyncIterator, Callable, Optional, Sequence, Tuple
 import pandas as pd
 from twelvedata import TDClient
 from app_config import provider_api_key
 from config import PAIR_INFO, PlatformConfig, Timeframe
 from providers.base import BaseDataProvider, DataIntegrityError
 logger = logging.getLogger(__name__)
 class TwelveDataProvider(BaseDataProvider):
    """Twelve Data SDK client with credit throttling and WS quotes."""
    name = "twelve_data"
    TD_INTERVALS = {
        Timeframe.M1:  "1min",
        Timeframe.M5:  "5min",
        Timeframe.M15: "15min",
        Timeframe.M30: "30min",
        Timeframe.H1:  "1h",
        Timeframe.H4:  "4h",
        Timeframe.D1:  "1day",
        Timeframe.W1:  "1week",
    }
    RATE_LIMIT = 55
    RATE_WINDOW = 60.0
    WS_URL = "wss://ws.twelvedata.com/v1/quotes/price?apikey={api_key}"
    def __init__(self, config: PlatformConfig):
        super().__init__(config)
        self._api_key = provider_api_key(provider="twelve_data") or None
        self._throttle_lock = threading.Lock()
        self._request_times: deque = deque()
        self._td: Optional[TDClient] = (
            TDClient(apikey=self._api_key) if self._api_key else None
        )
    @staticmethod
    def _to_td_symbol(symbol: str) -> str:
        if "/" in symbol:
            return symbol
        clean = symbol.replace("=X", "")
        return f"{clean[:3]}/{clean[3:]}"
    # ── Raw fetch ──────────────────────────────────────────────────
    def _fetch_raw(
        self,
        symbol: str,
        timeframe: Timeframe,
        outputsize: int,
        start: Optional[datetime] = None,
        end: Optional[datetime] = None,
    ) -> pd.DataFrame:
        return self._fetch_twelve_data(
            symbol, timeframe, outputsize,
            start_date=start.strftime("%Y-%m-%d %H:%M:%S") if start else None,
            end_date=end.strftime("%Y-%m-%d %H:%M:%S") if end else None,
        )
    def _throttle(self, credits: int = 1) -> None:
        """Pro tier: 55 credits / minute. Block until budget allows."""
        while True:
            with self._throttle_lock:
                now = time.time()
                while self._request_times and now - self._request_times[0] >= self.RATE_WINDOW:
                    self._request_times.popleft()
                available = self.RATE_LIMIT - len(self._request_times)
                if available >= credits:
                    for _ in range(credits):
                        self._request_times.append(time.time())
                    return
                used = len(self._request_times)
                wait = (self._request_times[0] + self.RATE_WINDOW) - now + 0.2
            logger.info(
                f"Rate limit: {used}/{self.RATE_LIMIT} credits used "
                f"— waiting {wait:.1f}s..."
            )
            time.sleep(wait)
    def _fetch_twelve_data(
        self,
        symbol: str,
        timeframe: Timeframe,
        outputsize: int,
        start_date: Optional[str] = None,
        end_date: Optional[str] = None,
    ) -> pd.DataFrame:
        if self._td is None:
            raise DataIntegrityError(
                "No data-provider API key. Set the TWELVE_DATA_API_KEY env "
                "var or data_providers.twelve_data.api_key in config.json."
            )
        td_symbol = self._to_td_symbol(symbol)
        interval = self.TD_INTERVALS[timeframe]
        self._throttle(credits=1)
        kwargs = {
            "symbol": td_symbol,
            "interval": interval,
            "outputsize": min(outputsize, self.MAX_BARS_PER_REQUEST),
            "timezone": "UTC",
        }
        if start_date:
            kwargs["start_date"] = start_date
        if end_date:
            kwargs["end_date"] = end_date
        try:
            df = self._td.time_series(**kwargs).as_pandas()
        except Exception as e:
            raise DataIntegrityError(
                f"Twelve Data fetch failed for {td_symbol}: {e}"
            ) from e
        if df is None or df.empty:
            raise DataIntegrityError(f"Empty response from Twelve Data for {td_symbol}")
        # SDK returns lowercase columns; engine expects capitalised.
        df = df.rename(columns={
            "open": "Open", "high": "High", "low": "Low",
            "close": "Close", "volume": "Volume",
        })
        for col in ("Open", "High", "Low", "Close", "Volume"):
            if col in df.columns:
                df[col] = pd.to_numeric(df[col], errors="coerce")
        if "Volume" not in df.columns:
            df["Volume"] = 0.0
        # SDK returns newest-first; engine + indicators want oldest-first.
        df = df.sort_index()
        if df.index.tz is None:
            df.index = df.index.tz_localize("UTC")
        df.index.name = "Datetime"
        logger.debug(f"Fetched {len(df)} bars for {td_symbol} @ {interval}")
        return df
    # ── Streaming quotes ───────────────────────────────────────────
    supports_streaming = True
    @staticmethod
    def _parse_ws_price(raw: str) -> Optional[Tuple[str, float]]:
        """Parse one WS message; return (ws_symbol, price) for price
        events, None for everything else (logged where noteworthy)."""
        try:
            msg = json.loads(raw)
        except json.JSONDecodeError:
            logger.warning("WebSocket: malformed JSON, skipping")
            return None
        event = msg.get("event")
        if event == "subscribe-status":
            fails = msg.get("fails") or []
            if fails:
                logger.warning(f"WebSocket: failed to subscribe: {fails}")
            return None
        if event != "price":
            return None
        ws_symbol = msg.get("symbol")
        try:
            price = float(msg["price"])
        except (KeyError, ValueError, TypeError) as e:
            logger.warning(f"WebSocket: bad price data for {ws_symbol}: {e}")
            return None
        return ws_symbol, price
    @staticmethod
    async def _ws_heartbeat(ws) -> None:
        """Twelve Data drops idle WS connections; heartbeat every 10s keeps
        it alive through quiet periods (weekends, low-liquidity hours)."""
        while True:
            await asyncio.sleep(10)
            await ws.send(json.dumps({"action": "heartbeat"}))
    async def iter_ticks(
        self,
        symbols: Sequence[str],
        on_connect: Optional[Callable[[], None]] = None,
    ) -> AsyncIterator[Tuple[str, float]]:
        """One connection's lifetime of (canonical_symbol, price) ticks.
        The TD WS speaks display symbols ("XAU/USD"); ticks are mapped back
        to the canonical pair symbols passed in. The caller owns the
        reconnect/backoff loop around this generator.
        `on_connect` fires on the FIRST RECEIVED frame, not after the
        subscribe send (2026-08-04 audit H38b, mirroring the Tiingo
        provider) — firing pre-receive let a rejected API key reset the
        caller's reconnect backoff on every attempt, pinning it at the
        minimum forever.
        """
        import websockets
        ws_to_pair = {PAIR_INFO[s]["display"]: s for s in symbols}
        url = self.WS_URL.format(api_key=self._api_key or "")
        async with websockets.connect(url) as ws:
            subscribe = {
                "action": "subscribe",
                "params": {"symbols": ",".join(ws_to_pair)},
            }
            await ws.send(json.dumps(subscribe))
            logger.info(f"WebSocket connected, subscribed to {len(ws_to_pair)} symbols")
            hb_task = asyncio.create_task(self._ws_heartbeat(ws))
            connected = False
            try:
                async for raw in ws:
                    if not connected:
                        connected = True
                        if on_connect is not None:
                            on_connect()
                    parsed = self._parse_ws_price(raw)
                    if parsed is None:
                        continue
                    pair_symbol = ws_to_pair.get(parsed[0])
                    if pair_symbol is None:
                        continue
                    yield pair_symbol, parsed[1]
            finally:
                hb_task.cancel()
@@ -3,3 +3,4 @@ pandas>=2.0
 requests>=2.28
 twelvedata>=1.2.0
 python-telegram-bot>=20.0
 websockets>=12.0
@@ -0,0 +1,245 @@
 <?php
 /**
 * On-demand gold-impact AI summary for one published headline.
 *
 * The ONLY dynamic piece of the otherwise-static dashboard. Serves
 * GET ?id=<sha1-of-headline> and returns {"summary": ..., "generated_at": ...}.
 *
 * Abuse model (the page is public):
 *   - WHITELIST: the id must match a headline in the local news.json —
 *     arbitrary text can never reach the LLM through this endpoint.
 *   - CACHE FOREVER: headlines are immutable, so each id costs at most
 *     one DeepSeek call, ever (ai/<id>.json). Cache is checked BEFORE
 *     the whitelist on purpose: summaries stay reachable after the
 *     headline ages out of the 24 h feed window.
 *   - DAILY CAP: file-based UTC-day counter, FAIL CLOSED on counter I/O
 *     errors (a perms mistake must not mean unmetered spend).
 *
 * DeepSeek doctrine (mirrors llm_client.py — deliberate local copy, the
 * dashboard never imports bot code and this file never reads the repo's
 * config.json):
 *   - STATIC byte-identical system prompt (prefix-cache pricing).
 *   - "thinking": disabled is MANDATORY for V4-Flash (else it burns the
 *     timeout reasoning), temperature 0, json response_format.
 *   - One attempt, no retries; every failure returns a JSON error the
 *     page renders as a graceful fallback.
 *
 * Key file lives OUTSIDE the docroot: /etc/gold-dashboard/deepseek.key
 * (root:apache 640). Override paths/caps via Apache SetEnv:
 * DEEPSEEK_KEY_FILE, AI_SUMMARY_DAILY_CAP.
 */
 const NEWS_FILE = __DIR__ . '/news.json';
 const CACHE_DIR = __DIR__ . '/ai';   // apache-owned; created at deploy time
 const API_URL   = 'https://api.deepseek.com/chat/completions';
 const MODEL     = 'deepseek-v4-flash';
 const TIMEOUT   = 12;
 const MAX_TOKENS = 200;
 const SUMMARY_MAX_CHARS = 400;
 // .ht_ prefix: Apache's default <Files ".ht*"> rule denies these even
 // though they sit in the public docroot.
 const CAP_FILE = CACHE_DIR . '/.ht_cap.json';
 const SYSTEM_PROMPT = <<<'PROMPT'
 You are a forex/gold news analyst. You receive one news item as JSON:
 {"headline", "description", "score", "xau_score", "keywords", "currencies"}.
 "xau_score" is the platform's automated gold-effect score for this item,
 -1.0 (gold down) to 1.0 (gold up); "score" is overall market sentiment.
 Both are automated and can be wrong — judge the news yourself.
 In 1-2 plain-English sentences (max 45 words) explain what this news
 means for the GOLD (XAU/USD) price and WHY it is bullish, bearish, or
 neutral. If your reading clearly contradicts xau_score, add "(the
 automated score disagrees)". Never write the words "xau_score" or
 "score" — a general reader sees this text.
 Gold doctrine: hawkish Fed / rate hikes / rising real yields / strong
 dollar are gold-bearish; easing / rate cuts / safe-haven flows /
 geopolitical stress / weak dollar are gold-bullish.
 Reply with ONLY JSON: {"summary": "..."}
 PROMPT;
 function respond(int $status, array $obj, string $cache = 'no-store'): void
 {
    http_response_code($status);
    header('Content-Type: application/json; charset=utf-8');
    header('Cache-Control: ' . $cache);
    echo json_encode($obj, JSON_UNESCAPED_SLASHES | JSON_UNESCAPED_UNICODE);
    exit;
 }
 function atomic_write(string $path, string $data): bool
 {
    $tmp = $path . '.tmp.' . getmypid();
    if (@file_put_contents($tmp, $data) === false) {
        return false;
    }
    if (!@rename($tmp, $path)) {
        @unlink($tmp);
        return false;
    }
    return true;
 }
 // ── 1. id validation ────────────────────────────────────────────────
 $id = $_GET['id'] ?? '';
 if (!preg_match('/^[a-f0-9]{40}$/', $id)) {
    respond(400, ['error' => 'bad_id']);
 }
 // ── 2. cache first (immutable headlines → summaries never change) ───
 $cachePath = CACHE_DIR . '/' . $id . '.json';
 $cached = @file_get_contents($cachePath);
 if ($cached !== false && $cached !== '') {
    http_response_code(200);
    header('Content-Type: application/json; charset=utf-8');
    header('Cache-Control: max-age=3600');
    echo $cached;
    exit;
 }
 // ── 3. whitelist against the published feed ─────────────────────────
 $news = json_decode((string)@file_get_contents(NEWS_FILE), true);
 if (!is_array($news)) {
    respond(503, ['error' => 'no_feed']);
 }
 $hit = null;
 foreach ($news['headlines'] ?? [] as $h) {
    // Fallback sha1(): a publisher that predates the id field still
    // serves a valid whitelist during the restart window.
    $hid = $h['id'] ?? sha1($h['headline'] ?? '');
    if ($hid === $id) {
        $hit = $h;
        break;
    }
 }
 if ($hit === null) {
    respond(404, ['error'   => 'unknown_id',
                  'message' => 'Headline is no longer in the feed.']);
 }
 // ── 4. daily cap (fail closed) ──────────────────────────────────────
 $dailyCap = (int)(getenv('AI_SUMMARY_DAILY_CAP') ?: 150);
 $capFh = @fopen(CAP_FILE, 'c+');
 if ($capFh === false || !flock($capFh, LOCK_EX)) {
    respond(503, ['error' => 'cap_io']);
 }
 $today = gmdate('Y-m-d');
 $cap = json_decode((string)stream_get_contents($capFh), true);
 if (!is_array($cap) || ($cap['day'] ?? '') !== $today) {
    $cap = ['day' => $today, 'count' => 0];
 }
 if ($cap['count'] >= $dailyCap) {
    flock($capFh, LOCK_UN);
    fclose($capFh);
    respond(429, ['error' => 'cap_exceeded']);
 }
 // Increment BEFORE the LLM call — attempts count against the cap
 // (same semantics as llm_client._under_cap).
 $cap['count']++;
 rewind($capFh);
 if (ftruncate($capFh, 0) === false
        || fwrite($capFh, json_encode($cap)) === false) {
    flock($capFh, LOCK_UN);
    fclose($capFh);
    respond(503, ['error' => 'cap_io']);
 }
 fflush($capFh);
 flock($capFh, LOCK_UN);
 fclose($capFh);
 // ── 5. API key ──────────────────────────────────────────────────────
 $keyFile = getenv('DEEPSEEK_KEY_FILE') ?: '/etc/gold-dashboard/deepseek.key';
 $apiKey = trim((string)@file_get_contents($keyFile));
 if ($apiKey === '') {
    respond(503, ['error' => 'no_key']);
 }
 // ── 6. per-id stampede lock, then re-check the cache ────────────────
 $lockFh = @fopen(CACHE_DIR . '/.ht_lock_' . $id, 'c');
 if ($lockFh !== false) {
    flock($lockFh, LOCK_EX);   // blocks while a concurrent winner generates
    $cached = @file_get_contents($cachePath);
    if ($cached !== false && $cached !== '') {
        flock($lockFh, LOCK_UN);
        fclose($lockFh);
        http_response_code(200);
        header('Content-Type: application/json; charset=utf-8');
        header('Cache-Control: max-age=3600');
        echo $cached;
        exit;
    }
 }
 // ── 7. DeepSeek call (exact chat_json body shape) ───────────────────
 $userPayload = json_encode([
    'headline'    => $hit['headline'] ?? '',
    'description' => $hit['description'] ?? null,
    'score'       => $hit['score'] ?? null,
    'xau_score'   => $hit['score_by_currency']['XAU'] ?? null,
    'keywords'    => $hit['keywords'] ?? [],
    'currencies'  => $hit['currencies'] ?? [],
 ], JSON_UNESCAPED_UNICODE);
 $body = json_encode([
    'model'    => MODEL,
    'messages' => [
        ['role' => 'system', 'content' => SYSTEM_PROMPT],
        ['role' => 'user',   'content' => $userPayload],
    ],
    'response_format' => ['type' => 'json_object'],
    'temperature'     => 0,
    'thinking'        => ['type' => 'disabled'],
    'max_tokens'      => MAX_TOKENS,
 ]);
 $ch = curl_init(API_URL);
 curl_setopt_array($ch, [
    CURLOPT_POST           => true,
    CURLOPT_POSTFIELDS     => $body,
    CURLOPT_HTTPHEADER     => ['Content-Type: application/json',
                               'Authorization: Bearer ' . $apiKey],
    CURLOPT_RETURNTRANSFER => true,
    CURLOPT_TIMEOUT        => TIMEOUT,
 ]);
 $raw = curl_exec($ch);
 $httpCode = curl_getinfo($ch, CURLINFO_RESPONSE_CODE);
 curl_close($ch);
 // ── 8. parse (any failure → 502, never cached) ──────────────────────
 $summary = null;
 if ($raw !== false && $httpCode === 200) {
    $resp = json_decode($raw, true);
    $content = $resp['choices'][0]['message']['content'] ?? '';
    $parsed = json_decode((string)$content, true);
    if (is_array($parsed) && is_string($parsed['summary'] ?? null)
            && trim($parsed['summary']) !== '') {
        $summary = trim($parsed['summary']);
        if (mb_strlen($summary) > SUMMARY_MAX_CHARS) {
            $summary = mb_substr($summary, 0, SUMMARY_MAX_CHARS) . '…';
        }
    }
 }
 if ($lockFh !== false && $summary === null) {
    flock($lockFh, LOCK_UN);
    fclose($lockFh);
 }
 if ($summary === null) {
    respond(502, ['error' => 'llm_failed']);
 }
 // ── 9. cache + respond ──────────────────────────────────────────────
 $out = json_encode(['summary' => $summary, 'generated_at' => gmdate('c')],
                   JSON_UNESCAPED_SLASHES | JSON_UNESCAPED_UNICODE);
 if (!atomic_write($cachePath, $out)) {
    // Degrade to per-click generation rather than a broken feature —
    // the daily cap still bounds spend.
    error_log('news_summary.php: cache write failed for ' . $cachePath);
 }
 if ($lockFh !== false) {
    flock($lockFh, LOCK_UN);
    fclose($lockFh);
 }
 http_response_code(200);
 header('Content-Type: application/json; charset=utf-8');
 header('Cache-Control: max-age=3600');
 echo $out;
@@ -1,929 +0,0 @@
 #!/usr/bin/env python3
 """
 ForexBot2 — Telegram Bot
 ========================
 Async Telegram interface with candle-boundary auto-scanner.
 Manual scan: XAU/USD + EUR/USD via inline buttons.
 Auto-scan:   Starts automatically on boot if TELEGRAM_CHAT_ID is set.
             /autoscan toggles for additional chats or to stop.
             M5  fires at :00/:05/:10/… + 10s buffer (data only, no signals).
             M15 fires at :00/:15/:30/:45 + 10s buffer.
             H1  fires at :00 + 10s buffer.
             H4  fires at :00 every 4 hours + 10s buffer.
 Usage:
    python testing_telegram_bot.py
    TELEGRAM_TOKEN=<token> TELEGRAM_CHAT_ID=<id> python testing_telegram_bot.py
 """
 import asyncio
 import logging
 import logging.handlers
 import os
 import signal
 import sys
 import time
 from concurrent.futures import ThreadPoolExecutor
 from datetime import datetime, timezone
 from zoneinfo import ZoneInfo
 from typing import Dict, List, Optional, Set
 from telegram import InlineKeyboardButton, InlineKeyboardMarkup, Update
 from telegram.constants import ParseMode
 from telegram.ext import (
    Application,
    CallbackQueryHandler,
    CommandHandler,
    ContextTypes,
 )
 from config import (
    PlatformConfig, SignalStrength, Timeframe,
    XAUUSD, EURUSD, PAIR_INFO,
 )
 from data_provider import ForexDataProvider
 from signal_aggregator import SignalAggregator, SignalReport
 from pair_signals import produce_signal
 # ── Force UTF-8 on Windows ────────────────────────────────────────────
 if hasattr(sys.stdout, "reconfigure"):
    sys.stdout.reconfigure(encoding="utf-8")
 _LOG_DIR = os.path.join(os.path.dirname(os.path.abspath(__file__)), "logs")
 os.makedirs(_LOG_DIR, exist_ok=True)
 _log_file = os.path.join(_LOG_DIR, f"telegram_{datetime.now().strftime('%Y%m%d')}.log")
 _fmt = logging.Formatter("%(asctime)s [%(levelname)s] %(name)s: %(message)s", datefmt="%H:%M:%S")
 _console = logging.StreamHandler()
 _console.setFormatter(_fmt)
 _file_handler = logging.handlers.TimedRotatingFileHandler(
    _log_file, when="midnight", backupCount=30, encoding="utf-8",
 )
 _file_handler.setFormatter(_fmt)
 logging.basicConfig(level=logging.INFO, handlers=[_console, _file_handler])
 logging.getLogger("httpx").setLevel(logging.WARNING)
 logging.getLogger("httpcore").setLevel(logging.WARNING)
 logger = logging.getLogger("telegram_bot")
 TOKEN = os.environ.get("TELEGRAM_TOKEN", "8882954582:AAEOi2MzoMbJjozDMiRtFBHc2muix_MtaQQ")
 CHAT_ID = os.environ.get("TELEGRAM_CHAT_ID", "6025899022")
 # ── Pairs & buttons ───────────────────────────────────────────────────
 SCAN_PAIRS = [XAUUSD, EURUSD]
 _LEVEL_KEYS = (
    "swing_high_1", "swing_high_2", "swing_high_3",
    "swing_low_1", "swing_low_2", "swing_low_3",
    "ema_trend", "bb_upper", "bb_lower",
    "kc_upper", "kc_lower",
 )
 # ── Live signal stats cache ──────────────────────────────────────────
 # symbol → { tf_label → SignalReport }
 _signal_stats: Dict[str, Dict[str, SignalReport]] = {}
 _stats_updated: Optional[float] = None
 SCAN_BUTTONS = [
    {"cb": "scan_xauusd_m5",  "symbol": XAUUSD, "tf": Timeframe.M5,  "label": "XAU 5m"},
    {"cb": "scan_xauusd_m15", "symbol": XAUUSD, "tf": Timeframe.M15, "label": "XAU 15m"},
    {"cb": "scan_xauusd_h1",  "symbol": XAUUSD, "tf": Timeframe.H1,  "label": "XAU 1h"},
    {"cb": "scan_xauusd_h4",  "symbol": XAUUSD, "tf": Timeframe.H4,  "label": "XAU 4h"},
    {"cb": "scan_xauusd_all", "symbol": XAUUSD, "tf": None,          "label": "XAU All"},
    {"cb": "scan_eurusd_m5",  "symbol": EURUSD, "tf": Timeframe.M5,  "label": "EUR 5m"},
    {"cb": "scan_eurusd_m15", "symbol": EURUSD, "tf": Timeframe.M15, "label": "EUR 15m"},
    {"cb": "scan_eurusd_h1",  "symbol": EURUSD, "tf": Timeframe.H1,  "label": "EUR 1h"},
    {"cb": "scan_eurusd_h4",  "symbol": EURUSD, "tf": Timeframe.H4,  "label": "EUR 4h"},
    {"cb": "scan_eurusd_all", "symbol": EURUSD, "tf": None,          "label": "EUR All"},
 ]
 SCAN_LOOKUP = {b["cb"]: b for b in SCAN_BUTTONS}
 TF_CONFIRMATIONS = {
    Timeframe.M5:  [Timeframe.M15, Timeframe.H1],
    Timeframe.M15: [Timeframe.M5, Timeframe.H1],
    Timeframe.H1:  [Timeframe.M15, Timeframe.H4],
    Timeframe.H4:  [Timeframe.H1, Timeframe.D1],
 }
 AUTOSCAN_CONFIRMATIONS = {
    Timeframe.M5:  [Timeframe.M15, Timeframe.H1],
    Timeframe.M15: [Timeframe.M5, Timeframe.H1],
    Timeframe.H1:  [Timeframe.M15, Timeframe.H4],
    Timeframe.H4:  [Timeframe.H1, Timeframe.D1],
 }
 _executor = ThreadPoolExecutor(max_workers=4)
 # ── Markdown escaping ─────────────────────────────────────────────────
 def _escape_md(text: str) -> str:
    for ch in ("*", "_", "`", "[", "]"):
        text = text.replace(ch, "\\" + ch)
    return text
 def _fmt_price(price: float) -> str:
    return f"{price:.2f}" if price >= 10 else f"{price:.5f}"
 # ── Shared data provider (single cache across all scans) ─────────────
 _shared_dp: Optional[ForexDataProvider] = None
 def _get_shared_dp() -> ForexDataProvider:
    global _shared_dp
    if _shared_dp is None:
        _shared_dp = ForexDataProvider(PlatformConfig())
    return _shared_dp
 # ── Signal formatting ─────────────────────────────────────────────────
 _SIGNAL_EMOJI = {
    SignalStrength.STRONG_BUY:  "🟢",
    SignalStrength.BUY:         "🟩",
    SignalStrength.NEUTRAL:     "⬜",
    SignalStrength.SELL:        "🟥",
    SignalStrength.STRONG_SELL: "🔴",
 }
 _QUALITY_EMOJI = {
    "A+": "🏆", "A": "🥇", "B": "🥈", "C": "🥉", "D": "⚠️", "F": "❌",
 }
 _SIGNAL_TAG = {
    SignalStrength.STRONG_BUY:  "Strong Buy",
    SignalStrength.BUY:         "Bullish",
    SignalStrength.NEUTRAL:     "Neutral",
    SignalStrength.SELL:        "Bearish",
    SignalStrength.STRONG_SELL: "Strong Sell",
 }
 def _signal_label(sig: SignalStrength) -> str:
    return sig.name.replace("_", " ")
 def _mtf_line(report: SignalReport) -> str:
    if not report.mtf_signals:
        return ""
    parts = []
    for tf, sig in report.mtf_signals.items():
        score = report.mtf_scores.get(tf, 0)
        parts.append(f"{tf}: {_signal_label(sig)} {score:+.1f}")
    return " | ".join(parts)
 def _indicator_line(name: str, ind) -> str:
    tag = _SIGNAL_TAG.get(ind.signal, "Neutral")
    short = ind.description.split(" | ")[0] if " | " in ind.description else ind.description
    return f"[{tag}] *{_escape_md(ind.name)}*: {_escape_md(short)}"
 def format_signal_message(report: SignalReport) -> str:
    sig = report.final_signal
    emoji = _SIGNAL_EMOJI.get(sig, "⬜")
    qe = _QUALITY_EMOJI.get(report.signal_quality, "")
    label = _signal_label(sig)
    mtf_tag = "✅ MTF aligned" if report.mtf_agreement else "❌ MTF diverged"
    lines: list[str] = []
    now = datetime.now(timezone.utc).strftime("%Y-%m-%d %H:%M UTC")
    regime_str = _escape_md(report.market_regime.replace("_", " ").title())
    vol_str = _escape_md(str(report.volatility_regime))
    lines += [
        f"🔍 *{report.display_name} Signal Analysis*  ({report.primary_analysis.timeframe if report.primary_analysis else ''})",
        f"🕐 {now}",
        "",
        f"💰 Price: *{_fmt_price(report.current_price)}*",
        f"📊 Regime: {regime_str} | Vol: {vol_str}",
        "",
        "━━━━━━━━━━━━━━━━━━━━",
        f"{emoji} *{label}*  {qe}{report.signal_quality}  |  Score: {report.final_score:+.1f}  |  {report.final_confidence:.0f}% conf",
        f"_{_escape_md(report.action_recommendation)}_",
        f"{mtf_tag}",
    ]
    mtf = _mtf_line(report)
    if mtf:
        lines += ["", f"📈 *MTF*: {mtf}"]
    if report.primary_analysis:
        lines += ["", "━━━━━━━━━━━━━━━━━━━━", "📋 *Indicators*"]
        for name, ind in report.primary_analysis.indicators.items():
            if name.startswith("_") or ind.weight == 0:
                continue
            lines.append(_indicator_line(name, ind))
    if report.warnings:
        lines.append("")
        for w in report.warnings:
            lines.append(f"⚠️ {_escape_md(w)}")
    if report.trade_setup and sig not in (SignalStrength.NEUTRAL,):
        ts = report.trade_setup
        dir_emoji = "📈" if ts.direction == "BUY" else "📉"
        lines += [
            "",
            "━━━━━━━━━━━━━━━━━━━━",
            f"🚀 *TRADE SETUP*",
            f"{dir_emoji} Direction: *{ts.direction}*",
            f"▶ Entry:  `{_fmt_price(ts.entry_price)}`",
            f"🛑 SL:     `{_fmt_price(ts.stop_loss)}`  ({ts.risk_pips:.1f} pips)",
            f"🎯 TP1:    `{_fmt_price(ts.take_profit_1)}`  ({ts.reward_pips:.1f} pips)",
        ]
        if ts.take_profit_2:
            lines.append(f"🎯 TP2:    `{_fmt_price(ts.take_profit_2)}`")
        lines.append(f"⚖️  RR:     1:{ts.risk_reward:.1f}")
    lines += ["", "━━━━━━━━━━━━━━━━━━━━",
              "_Educational / research use only._"]
    return "\n".join(lines)
 # ── Auto-scan summary format ─────────────────────────────────────────
 def _format_scan_summary(reports: List[SignalReport], tf: Timeframe) -> str:
    now = datetime.now(timezone.utc).strftime("%H:%M UTC")
    lines = [
        f"📊 *{tf.value} Auto-Scan* | {now}",
        "━━━━━━━━━━━━━━━━━━━━",
    ]
    for r in reports:
        emoji = _SIGNAL_EMOJI.get(r.final_signal, "⬜")
        sig = _signal_label(r.final_signal)
        mtf = "✅" if r.mtf_agreement else "❌"
        lines.append(
            f"{emoji} *{r.display_name}*  {sig}  "
            f"{_QUALITY_EMOJI.get(r.signal_quality, '')}{r.signal_quality}  "
            f"{r.final_score:+.1f}  {r.final_confidence:.0f}%  {mtf}MTF"
        )
    lines.append("━━━━━━━━━━━━━━━━━━━━")
    actionable = [r for r in reports if r.signal_quality in ("A+", "A", "B", "C")]
    if actionable:
        names = ", ".join(r.display_name for r in actionable)
        lines.append(f"🔔 Actionable: {names}")
    else:
        lines.append("No actionable signals this cycle.")
    return "\n".join(lines)
 def _update_signal_stats(reports: List[SignalReport], tf: Timeframe) -> None:
    global _stats_updated
    for r in reports:
        if r.symbol not in _signal_stats:
            _signal_stats[r.symbol] = {}
        _signal_stats[r.symbol][tf.value] = r
    _stats_updated = time.time()
 _SHORT_SIGNAL = {
    SignalStrength.STRONG_BUY:  "Buy",
    SignalStrength.BUY:         "Buy",
    SignalStrength.NEUTRAL:     "Neutral",
    SignalStrength.SELL:        "Sell",
    SignalStrength.STRONG_SELL: "Sell",
 }
 _TF_DISPLAY = {"5min": "M5", "15min": "M15", "1h": "H1", "4h": "H4", "1d": "D1"}
 _DASH_TFS = [Timeframe.M15, Timeframe.H1, Timeframe.H4]
 def _format_stats_dashboard() -> str:
    if not _signal_stats:
        return "📊 *Live Dashboard*\n\n_Waiting for first scan…_"
    _ny_tz = ZoneInfo("America/New_York")
    now = datetime.now(_ny_tz).strftime("%H:%M:%S %Z")
    lines = [f"📊 *Live Dashboard* — {now}", ""]
    for symbol in SCAN_PAIRS:
        display = PAIR_INFO.get(symbol, {}).get("display", symbol)
        pair_data = _signal_stats.get(symbol, {})
        latest = None
        for tf in reversed(_DASH_TFS):
            if tf.value in pair_data:
                latest = pair_data[tf.value]
                break
        price = latest.current_price if latest else 0
        price_str = f"`{_fmt_price(price)}`"
        regime = latest.market_regime.replace("_", " ").title() if latest else ""
        tf_parts = []
        for tf in _DASH_TFS:
            label = _TF_DISPLAY.get(tf.value, tf.value)
            r = pair_data.get(tf.value)
            if not r:
                tf_parts.append(f"{label}: ⬜—")
                continue
            emoji = _SIGNAL_EMOJI.get(r.final_signal, "⬜")
            sig = _SHORT_SIGNAL.get(r.final_signal, "Neutral")
            tf_parts.append(f"{label}: {emoji}{sig}")
        lines.append(f"*{display}*  {price_str}  _{_escape_md(regime)}_")
        lines.append(" | ".join(tf_parts))
        if latest and latest.trade_setup and latest.final_signal not in (SignalStrength.NEUTRAL,):
            ts = latest.trade_setup
            d = "📈" if ts.direction == "BUY" else "📉"
            lines.append(
                f"{d} {ts.direction} `{_fmt_price(ts.entry_price)}` "
                f"SL `{_fmt_price(ts.stop_loss)}` TP `{_fmt_price(ts.take_profit_1)}` "
                f"RR 1:{ts.risk_reward:.1f}"
            )
        lines.append("")
    lines.append("_🔄 live refresh_")
    return "\n".join(lines)
 # ── Live dashboard (edit-in-place) ───────────────────────────────────
 _dashboard_msg_ids: Dict[int, int] = {}
 _dashboard_task: Optional[asyncio.Task] = None
 _last_dashboard_refresh: float = 0
 _DASHBOARD_REFRESH_INTERVAL = 30
 _dashboard_lock = asyncio.Lock()
 async def _send_dashboard(bot, chat_id: int) -> None:
    text = _format_stats_dashboard()
    try:
        msg = await bot.send_message(chat_id, text, parse_mode=ParseMode.MARKDOWN)
        _dashboard_msg_ids[chat_id] = msg.message_id
    except Exception as exc:
        logger.warning(f"Dashboard send failed for chat {chat_id}: {exc}")
 async def _refresh_dashboard(bot, force: bool = False) -> None:
    global _last_dashboard_refresh
    async with _dashboard_lock:
        now = time.time()
        if not force and (now - _last_dashboard_refresh) < _DASHBOARD_REFRESH_INTERVAL:
            return
        _last_dashboard_refresh = now
        text = _format_stats_dashboard()
        for chat_id in list(_autoscan_chats):
            msg_id = _dashboard_msg_ids.get(chat_id)
            if not msg_id:
                await _send_dashboard(bot, chat_id)
                continue
            try:
                await bot.edit_message_text(
                    text, chat_id=chat_id, message_id=msg_id,
                    parse_mode=ParseMode.MARKDOWN,
                )
            except Exception as exc:
                if "message is not modified" not in str(exc).lower():
                    logger.warning(f"Dashboard edit failed for chat {chat_id}: {exc}")
 async def _dashboard_loop(bot) -> None:
    try:
        while _autoscan_chats:
            await asyncio.sleep(_DASHBOARD_REFRESH_INTERVAL)
            if not _autoscan_chats:
                break
            await _refresh_dashboard(bot)
    except asyncio.CancelledError:
        pass
    finally:
        logger.info("Dashboard refresh loop stopped")
 # ── Keyboard helper ───────────────────────────────────────────────────
 def scan_keyboard() -> InlineKeyboardMarkup:
    rows = []
    for pair in SCAN_PAIRS:
        pair_btns = [b for b in SCAN_BUTTONS if b["symbol"] == pair]
        row = [
            InlineKeyboardButton(b["label"], callback_data=b["cb"])
            for b in pair_btns
        ]
        rows.append(row)
    return InlineKeyboardMarkup(rows)
 # ── Signal runner (shared data provider for cache reuse) ──────────────
 def _run_signal_sync(
    symbol: str,
    timeframe: Timeframe,
    confirmation_tfs: Optional[List[Timeframe]] = None,
 ) -> SignalReport:
    if confirmation_tfs is None:
        confirmation_tfs = TF_CONFIRMATIONS.get(
            timeframe, [Timeframe.M15, Timeframe.H4]
        )
    return produce_signal(
        symbol=symbol,
        primary_tf=timeframe,
        data_provider=_get_shared_dp(),
        confirmation_tfs=confirmation_tfs,
    )
 async def run_signal(
    symbol: str,
    timeframe: Timeframe,
    confirmation_tfs: Optional[List[Timeframe]] = None,
 ) -> SignalReport:
    loop = asyncio.get_running_loop()
    return await loop.run_in_executor(
        _executor, _run_signal_sync, symbol, timeframe, confirmation_tfs
    )
 # ── Candle-boundary scheduler ────────────────────────────────────────
 _autoscan_chats: Set[int] = set()
 _autoscan_tasks: list[asyncio.Task] = []
 def _seconds_until_boundary(interval_minutes: int, buffer: int = 10) -> float:
    """Seconds until the next candle close + buffer."""
    now = datetime.now(timezone.utc)
    total_minutes = now.hour * 60 + now.minute
    elapsed_minutes = total_minutes % interval_minutes
    elapsed_seconds = elapsed_minutes * 60 + now.second + now.microsecond / 1e6
    interval_seconds = interval_minutes * 60
    remaining = interval_seconds - elapsed_seconds + buffer
    if remaining <= buffer:
        remaining += interval_seconds
    return remaining
 async def _scan_timeframe(
    bot,
    tf: Timeframe,
    confirmation_tfs: List[Timeframe],
 ) -> None:
    """Scan all pairs for one timeframe and broadcast results."""
    tasks = [run_signal(symbol, tf, confirmation_tfs) for symbol in SCAN_PAIRS]
    results = await asyncio.gather(*tasks, return_exceptions=True)
    reports: List[SignalReport] = []
    for i, result in enumerate(results):
        if isinstance(result, Exception):
            logger.exception(f"Auto-scan failed for {SCAN_PAIRS[i]} {tf.value}: {result}")
        else:
            reports.append(result)
    if not reports:
        return
    _update_level_cache(reports)
    _update_signal_stats(reports, tf)
    await _refresh_dashboard(bot)
    actionable = [r for r in reports if r.signal_quality in ("A+", "A", "B", "C")]
    for chat_id in list(_autoscan_chats):
        for report in actionable:
            detail = format_signal_message(report)
            try:
                await bot.send_message(chat_id, detail, parse_mode=ParseMode.MARKDOWN)
            except Exception as exc:
                logger.warning(f"Failed to send signal for chat {chat_id}: {exc}")
 # ── WebSocket level-breach monitor ──────────────────────────────────
 _level_cache: Dict[str, Dict[str, float]] = {}
 _level_cache_ts: Dict[str, float] = {}
 _LEVEL_CACHE_TTL = 3600
 def _update_level_cache(reports: List[SignalReport]) -> None:
    now = time.time()
    for r in reports:
        if r.primary_analysis and r.primary_analysis.key_levels:
            levels = {
                k: v for k, v in r.primary_analysis.key_levels.items()
                if k in _LEVEL_KEYS
            }
            if levels:
                _level_cache[r.symbol] = levels
                _level_cache_ts[r.symbol] = now
 def _check_level_breach(symbol: str, price: float) -> Optional[str]:
    levels = _level_cache.get(symbol)
    if not levels:
        return None
    cache_age = time.time() - _level_cache_ts.get(symbol, 0)
    if cache_age > _LEVEL_CACHE_TTL:
        return None
    for name, level in levels.items():
        diff_pct = abs(price - level) / level if level != 0 else 0
        if diff_pct < 0.001:
            return name
    return None
 async def _candle_loop(bot, tf: Timeframe, interval_minutes: int) -> None:
    label = tf.value
    try:
        while _autoscan_chats:
            wait = _seconds_until_boundary(interval_minutes, buffer=10)
            logger.info(f"{label} auto-scan: next in {wait:.0f}s")
            await asyncio.sleep(wait)
            if not _autoscan_chats:
                break
            logger.info(f"{label} auto-scan: firing")
            await _scan_timeframe(
                bot, tf, AUTOSCAN_CONFIRMATIONS[tf]
            )
    except asyncio.CancelledError:
        pass
    finally:
        logger.info(f"{label} auto-scan loop stopped")
 _AUTOSCAN_LOOPS = [
    (Timeframe.M5,  5),
    (Timeframe.M15, 15),
    (Timeframe.H1,  60),
    (Timeframe.H4,  240),
 ]
 def _start_scheduler(bot) -> None:
    global _dashboard_task
    alive = [t for t in _autoscan_tasks if not t.done()]
    if not alive:
        _autoscan_tasks.clear()
        for tf, interval in _AUTOSCAN_LOOPS:
            _autoscan_tasks.append(
                asyncio.create_task(_candle_loop(bot, tf, interval))
            )
        logger.info("Candle-boundary scheduler started (M5/M15/H1/H4)")
    if _dashboard_task is None or _dashboard_task.done():
        _dashboard_task = asyncio.create_task(_dashboard_loop(bot))
        logger.info(f"Dashboard {_DASHBOARD_REFRESH_INTERVAL}s refresh loop started")
 def _stop_scheduler() -> None:
    global _dashboard_task
    for t in _autoscan_tasks:
        t.cancel()
    _autoscan_tasks.clear()
    if _dashboard_task and not _dashboard_task.done():
        _dashboard_task.cancel()
    _dashboard_task = None
    _dashboard_msg_ids.clear()
    logger.info("Scheduler and dashboard loop stopped")
 # ── Handlers ──────────────────────────────────────────────────────────
 async def cmd_start(update: Update, context: ContextTypes.DEFAULT_TYPE) -> None:
    scanning = "running" if _autoscan_chats else "off"
    text = (
        "👋 *ForexBot2 — Signal Intelligence*\n\n"
        "Pick a pair and timeframe to scan.\n"
        "Signals include a full trade setup with entry, SL and TP.\n\n"
        f"Auto-scan: *{scanning}* — /autoscan to toggle\n"
        "/scan — Quick scan buttons\n"
        "/status — Bot status\n"
        "/help — All commands"
    )
    await update.message.reply_text(
        text,
        parse_mode=ParseMode.MARKDOWN,
        reply_markup=scan_keyboard(),
    )
 async def cmd_help(update: Update, context: ContextTypes.DEFAULT_TYPE) -> None:
    text = (
        "📖 *ForexBot2 Commands*\n\n"
        "/start — Welcome & scan buttons\n"
        "/scan — Quick scan buttons (XAU/USD + EUR/USD)\n"
        "/autoscan — Toggle auto-scan on/off\n"
        "/status — Bot status\n"
        "/cache — Show data cache age per pair/TF\n"
        "/clearcache — Wipe data cache, force fresh fetch\n"
        "/help — This message\n\n"
        "*Auto-scan* fires at candle close: M5/M15/H1/H4.\n"
        f"Live dashboard auto-refreshes every {_DASHBOARD_REFRESH_INTERVAL}s."
    )
    await update.message.reply_text(text, parse_mode=ParseMode.MARKDOWN)
 async def cmd_status(update: Update, context: ContextTypes.DEFAULT_TYPE) -> None:
    lines = ["📊 *ForexBot2 Status*\n"]
    if _autoscan_chats:
        lines.append(f"✅ Auto-scan: *running* ({len(_autoscan_chats)} chat)")
        tasks_alive = sum(1 for t in _autoscan_tasks if not t.done())
        lines.append(f"   Candle loops: {tasks_alive}/{len(_AUTOSCAN_LOOPS)}")
    else:
        lines.append("⏹ Auto-scan: *stopped*")
    cached_pairs = len(_level_cache)
    cached_levels = sum(len(v) for v in _level_cache.values())
    if cached_levels:
        oldest_age = time.time() - min(_level_cache_ts.values()) if _level_cache_ts else 0
        lines.append(f"\n*Level Cache:* {cached_levels} levels across {cached_pairs} pairs (oldest {oldest_age:.0f}s)")
    await update.message.reply_text("\n".join(lines), parse_mode=ParseMode.MARKDOWN)
 async def cmd_cache(update: Update, context: ContextTypes.DEFAULT_TYPE) -> None:
    dp = _get_shared_dp()
    if not dp._cache_timestamps:
        await update.message.reply_text("📭 Cache is empty — no data fetched yet.")
        return
    now = time.time()
    lines = ["📦 *Data Cache Status*\n"]
    for pair in SCAN_PAIRS:
        display = PAIR_INFO.get(pair, {}).get("display", pair)
        pair_lines = []
        for tf in [Timeframe.M5, Timeframe.M15, Timeframe.H1, Timeframe.H4, Timeframe.D1]:
            cache_key = f"{pair}_{tf.value}"
            ts = dp._cache_timestamps.get(cache_key)
            if ts is None:
                continue
            age_min = (now - ts) / 60
            df = dp._cache.get(cache_key)
            last_bar = df.index[-1].strftime("%H:%M") if df is not None and not df.empty else "?"
            fresh = "✅" if age_min < dp.CANDLE_SECONDS.get(tf, 3600) / 60 else "⚠️"
            pair_lines.append(f"  {tf.value:<5} {fresh} fetched `{age_min:.0f}m` ago | last bar `{last_bar}`")
        if pair_lines:
            lines.append(f"*{display}*")
            lines.extend(pair_lines)
    await update.message.reply_text("\n".join(lines), parse_mode=ParseMode.MARKDOWN)
 async def cmd_clearcache(update: Update, context: ContextTypes.DEFAULT_TYPE) -> None:
    dp = _get_shared_dp()
    count = dp.clear_cache()
    await update.message.reply_text(
        f"🗑 Cache cleared ({count} entries wiped).\nNext scan will fetch fresh data from API.",
        parse_mode=ParseMode.MARKDOWN,
    )
 async def cmd_autoscan(update: Update, context: ContextTypes.DEFAULT_TYPE) -> None:
    chat_id = update.effective_chat.id
    if chat_id in _autoscan_chats:
        _autoscan_chats.discard(chat_id)
        if not _autoscan_chats:
            _stop_scheduler()
        await update.message.reply_text(
            "⏹ Auto-scan stopped.",
            parse_mode=ParseMode.MARKDOWN,
        )
        return
    _autoscan_chats.add(chat_id)
    _start_scheduler(context.bot)
    next_m5 = _seconds_until_boundary(5, buffer=10)
    next_m15 = _seconds_until_boundary(15, buffer=10)
    next_h1 = _seconds_until_boundary(60, buffer=10)
    next_h4 = _seconds_until_boundary(240, buffer=10)
    await update.message.reply_text(
        "✅ *Auto-scan enabled*\n\n"
        f"Scanning XAU/USD + EUR/USD on M5/M15/H1/H4 at candle close.\n"
        f"📊 Live dashboard refreshes every {_DASHBOARD_REFRESH_INTERVAL}s.\n"
        f"📡 Live price monitoring active.\n"
        f"Next: M5 {next_m5 / 60:.0f}m | M15 {next_m15 / 60:.0f}m | "
        f"H1 {next_h1 / 60:.0f}m | H4 {next_h4 / 60:.0f}m\n\n"
        "Use /autoscan again to stop.",
        parse_mode=ParseMode.MARKDOWN,
    )
 async def cmd_scan(update: Update, context: ContextTypes.DEFAULT_TYPE) -> None:
    await update.message.reply_text(
        "🔍 *Pick a pair & timeframe*",
        parse_mode=ParseMode.MARKDOWN,
        reply_markup=scan_keyboard(),
    )
 async def cb_scan(update: Update, context: ContextTypes.DEFAULT_TYPE) -> None:
    query = update.callback_query
    await query.answer()
    btn = SCAN_LOOKUP.get(query.data)
    if not btn:
        return
    if btn["tf"] is None:
        await _cb_scan_all(query, symbol=btn["symbol"])
        return
    display = PAIR_INFO.get(btn["symbol"], {}).get("display", btn["symbol"])
    await query.edit_message_text(
        f"⏳ Scanning {display} {btn['label']}…",
        parse_mode=ParseMode.MARKDOWN,
    )
    try:
        report = await run_signal(btn["symbol"], btn["tf"])
        text = format_signal_message(report)
    except Exception as exc:
        logger.exception("Signal scan failed")
        text = f"❌ Scan failed: {exc}"
    await query.edit_message_text(
        text,
        parse_mode=ParseMode.MARKDOWN,
        reply_markup=scan_keyboard(),
    )
 async def _cb_scan_all(query, symbol: str = XAUUSD) -> None:
    display = PAIR_INFO.get(symbol, {}).get("display", symbol)
    await query.edit_message_text(
        f"⏳ Scanning {display} across M5 / M15 / H1 / H4…",
        parse_mode=ParseMode.MARKDOWN,
    )
    tfs = [Timeframe.M5, Timeframe.M15, Timeframe.H1, Timeframe.H4]
    tasks = [run_signal(symbol, tf) for tf in tfs]
    results = await asyncio.gather(*tasks, return_exceptions=True)
    reports = []
    errors = []
    for tf, result in zip(tfs, results):
        if isinstance(result, Exception):
            errors.append(f"❌ {tf.value}: {result}")
        else:
            reports.append(result)
            _update_signal_stats([result], tf)
    if reports:
        _update_level_cache(reports)
    lines = [
        f"📊 *{display} — All Timeframes*",
        f"🕐 {datetime.now(timezone.utc).strftime('%Y-%m-%d %H:%M UTC')}",
        "",
    ]
    for r in reports:
        emoji = _SIGNAL_EMOJI.get(r.final_signal, "⬜")
        qe = _QUALITY_EMOJI.get(r.signal_quality, "")
        tf_label = r.primary_analysis.timeframe if r.primary_analysis else "?"
        sig = _signal_label(r.final_signal)
        regime = r.market_regime.replace("_", " ").title()
        lines.append(
            f"{emoji} *{tf_label}*  {sig}  {qe}{r.signal_quality}  "
            f"Score: {r.final_score:+.1f}  {r.final_confidence:.0f}%"
        )
        lines.append(f"   _{_escape_md(regime)}_ | {'✅ MTF' if r.mtf_agreement else '❌ MTF'}")
        if r.trade_setup and r.final_signal != SignalStrength.NEUTRAL:
            ts = r.trade_setup
            d = "📈" if ts.direction == "BUY" else "📉"
            lines.append(
                f"   {d} {ts.direction} `{_fmt_price(ts.entry_price)}` "
                f"SL `{_fmt_price(ts.stop_loss)}` TP `{_fmt_price(ts.take_profit_1)}` "
                f"RR 1:{ts.risk_reward:.1f}"
            )
        lines.append("")
    for err in errors:
        lines.append(err)
    lines.append("_Use buttons below for detailed single-TF view._")
    await query.edit_message_text(
        "\n".join(lines),
        parse_mode=ParseMode.MARKDOWN,
        reply_markup=scan_keyboard(),
    )
 # ── Boot-time auto-start ──────────────────────────────────────────────
 async def _post_init(app: Application) -> None:
    """Register command menu and start scheduler on boot."""
    await app.bot.set_my_commands([
        ("start", "Welcome & scan buttons"),
        ("scan", "Quick scan buttons (XAU/USD + EUR/USD)"),
        ("autoscan", "Toggle auto-scan on/off"),
        ("status", "Bot status, WS connection, live prices"),
        ("cache", "Show data cache age per pair/TF"),
        ("clearcache", "Wipe data cache, force fresh fetch"),
        ("help", "All commands"),
    ])
    if not CHAT_ID:
        logger.info("TELEGRAM_CHAT_ID not set — auto-scan disabled on boot. "
                     "Use /autoscan in chat to enable.")
        return
    chat_id = int(CHAT_ID)
    _autoscan_chats.add(chat_id)
    logger.info("Running initial scan to populate level cache and signal stats...")
    try:
        tasks = [
            asyncio.wait_for(run_signal(sym, Timeframe.M15), timeout=30)
            for sym in SCAN_PAIRS
        ]
        results = await asyncio.gather(*tasks, return_exceptions=True)
        init_reports = [r for r in results if isinstance(r, SignalReport)]
        for i, r in enumerate(results):
            if isinstance(r, Exception):
                logger.error(f"Initial scan failed for {SCAN_PAIRS[i]}: {r}")
        if init_reports:
            _update_level_cache(init_reports)
            _update_signal_stats(init_reports, Timeframe.M15)
            logger.info(f"Level cache populated: {sum(len(v) for v in _level_cache.values())} levels")
    except Exception:
        logger.exception("Initial scan failed — WS will wait for first candle scan")
    _start_scheduler(app.bot)
    next_m5 = _seconds_until_boundary(5, buffer=10)
    next_m15 = _seconds_until_boundary(15, buffer=10)
    next_h1 = _seconds_until_boundary(60, buffer=10)
    next_h4 = _seconds_until_boundary(240, buffer=10)
    logger.info(
        f"Auto-scan started for chat {chat_id} — "
        f"M5 {next_m5:.0f}s, M15 {next_m15:.0f}s, H1 {next_h1:.0f}s, H4 {next_h4:.0f}s"
    )
    try:
        await app.bot.send_message(
            chat_id,
            f"🤖 *ForexBot2 started*\n\n"
            f"Auto-scanning XAU/USD + EUR/USD on M5/M15/H1/H4.\n"
            f"📊 Live dashboard refreshes every {_DASHBOARD_REFRESH_INTERVAL}s.\n"
            f"📡 Live price monitoring active.\n"
            f"Next: M5 {next_m5 / 60:.0f}m | M15 {next_m15 / 60:.0f}m | "
            f"H1 {next_h1 / 60:.0f}m | H4 {next_h4 / 60:.0f}m\n\n"
            f"/autoscan to stop",
            parse_mode=ParseMode.MARKDOWN,
        )
        await _send_dashboard(app.bot, chat_id)
    except Exception:
        logger.exception("Failed to send boot message")
 # ── Graceful shutdown ─────────────────────────────────────────────────
 async def _post_shutdown(app: Application) -> None:
    logger.info("Shutting down ForexBot2...")
    _stop_scheduler()
    _executor.shutdown(wait=False)
    logger.info("Shutdown complete")
 # ── Entry point ───────────────────────────────────────────────────────
 def main() -> None:
    if not TOKEN:
        sys.exit("TELEGRAM_TOKEN not set")
    logger.info("Starting ForexBot2 Telegram bot…")
    app = (
        Application.builder()
        .token(TOKEN)
        .post_init(_post_init)
        .post_shutdown(_post_shutdown)
        .build()
    )
    app.add_handler(CommandHandler("start", cmd_start))
    app.add_handler(CommandHandler("scan", cmd_scan))
    app.add_handler(CommandHandler("help", cmd_help))
    app.add_handler(CommandHandler("status", cmd_status))
    app.add_handler(CommandHandler("autoscan", cmd_autoscan))
    app.add_handler(CommandHandler("cache", cmd_cache))
    app.add_handler(CommandHandler("clearcache", cmd_clearcache))
    app.add_handler(CallbackQueryHandler(cb_scan, pattern="^scan_"))
    logger.info("Bot running — press Ctrl+C to stop")
    app.run_polling(allowed_updates=Update.ALL_TYPES)
 if __name__ == "__main__":
    main()