Switch active data source to Oanda v20 (practice feed)

config.json.example: data_providers.active = "oanda" (credentials stay in env/.env or the top-level oanda block; TD and Tiingo remain registered fallbacks). Backtest parquet cache is now keyed per provider — Oanda and TD bars diverge (2026-07-24 study), so a feed switch must never serve another feed's cached bars; twelve_data keeps the legacy un-prefixed filename so pinned-book caches stay valid. 254 tests green; live smoke-tested against the practice API (66 contract checks: all scanned TFs, boundary/freshness guarantees, real spread $0.71, fetch_range pagination, typed error paths, CLI + backtest end-to-end). Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Mc7XKYyPySVXiUpcN82QEK
Provider abstraction: providers/ package with BaseDataProvider template
2026-07-28 20:12:59 -04:00 · 2026-07-28 20:09:00 -04:00 · 2026-07-28 20:08:53 -04:00 · 2026-07-24 15:12:37 -04:00 · 2026-07-24 14:10:45 -04:00 · 2026-07-24 10:58:22 -04:00
45 changed files with 1948 additions and 8621 deletions
@@ -35,5 +35,3 @@ backtest/trailgrid5m/
 backtest/__pycache__/
 .env.*
 logs/
 backtest/research/commit_ladder_2026-08/
 backtest/research/llm_news_eval_2026-08/
@@ -1,89 +1,381 @@
-# CLAUDE.md
+# ForexBot2 — Codebase Reference
-This file provides guidance to Claude Code (claude.ai/code) when working with code in this repository.
+Signal-intelligence platform for **XAU/USD (gold — live, validated)**. It fetches OHLCV from the active data provider (**Oanda v20 since 2026-07-28**; Twelve Data and Tiingo stay registered as fallbacks — config.json `data_providers.active` switches), scores 9 technical indicators per timeframe with regime-adaptive weights, checks multi-timeframe agreement and news/calendar risk, grades the signal (A+…F), and generates a full trade setup (entry, SL, TP1/TP2, trailing plan). Delivery: Telegram bot (auto-scanner + subscriber channel) and a CLI. A realism-focused backtester replays history through the *exact same* signal code.
-## What this is
+This file is the authoritative developer/agent reference. The human-facing docs live in `docs/` (`user_guide.html` + `beginner_guide.html`, and their PDF builds) — keep them in sync when signal behavior changes. `backtest/instructions_for_backtest.txt` lists what every backtest report must record.
-Signal-intelligence platform for **XAU/USD (gold — the only supported pair)**. It fetches OHLCV from the active data provider (**Oanda v20 since 2026-07-28**; Twelve Data and Tiingo remain registered fallbacks), scores 9 technical indicators per timeframe with regime-adaptive weights, checks multi-timeframe agreement and news risk, grades the signal (A+…F), and generates a full trade setup (entry, SL, TP1/TP2, trailing plan). Delivery: Telegram bot, CLI, and an Oanda practice-account auto-execution daemon. A realism-focused backtester replays history through the *exact same* signal code.
+**Read this whole file before changing signal logic.** Every numeric threshold below was either backtested or is load-bearing for a backtested result. Rules for changes are at the bottom ("Rules for making changes").
-Every numeric threshold in `config.py` and `pair_signals/xauusd.py` was backtested or is load-bearing for a backtested result. Validation evidence lives in the factory/module docstrings and in `~/.claude/projects/.../memory/`.
+---
-## Commands
+## Repository map
-```bash
+| Path | Lines | Purpose |
-python test_all.py                      # QA suite: 359 tests, 22 modules, no network (mocked)
+|------|-------|---------|
-python main.py --pair XAUUSD --timeframe H1 --detail   # CLI signal report (--json for full dump)
+| `config.py` | ~374 | All config dataclasses, `PAIR_INFO` registry, per-pair strategy configs. **`PlatformConfig` defaults ARE the tuned XAU config.** |
-python telegram_bot.py                  # the bot (refuses to boot without credentials)
+| `app_config.py` | ~290 | config.json loader/validation, env-precedence resolvers, repo-root `.env` loader (setdefault at import), `apply_pair_overrides` deployment overlay |
-python oanda_daemon.py                  # auto-execution daemon on Oanda practice account
+| `config.json.example` | 24 | committed schema template; real gitignored config.json holds secrets + deployment settings |
-python dashboard_publisher.py --once    # web dashboard: one scan+publish cycle to $DASHBOARD_OUT_DIR
+| `providers/base.py` | ~400 | `BaseDataProvider`: shared cache, validation/cleaning, freshness + boundary guarantees, `fetch_range` pagination, streaming stubs. Subclasses implement only `_fetch_raw` |
-                                        # (omit --once for the candle-close daemon; deploy/README-dashboard.md)
+| `providers/twelve_data.py` | ~240 | `TwelveDataProvider` (alias `ForexDataProvider`): TD SDK fetch, rate-limit (55 credits/min), WS quote streaming |
 | `providers/oanda.py` | ~190 | `OandaDataProvider`: v20 REST MBA candles, real bid/ask spread, date-window fetch |
 | `providers/tiingo.py` | ~200 | `TiingoDataProvider`: FX endpoint mid-price candles (no volume, no spread — see Tiingo note), named errors for every failure shape |
 | `providers/__init__.py` | ~55 | `PROVIDER_REGISTRY` + `make_data_provider` (config.json `data_providers.active` selects) |
 | `data_provider.py` / `oanda_provider.py` | ~25 | back-compat shims re-exporting from `providers/` |
 | `engine.py` | ~1173 | `TechnicalEngine`: computes all indicators on one DataFrame → `TechnicalAnalysis` (composite score, regime, key levels) |
 | `chart_patterns.py` | ~280 | Chart-pattern detectors for the XAU_CP archetype (pure functions, ATR-unit geometry, first-close-beyond triggers) |
 | `signal_aggregator.py` | ~1447 | `SignalAggregator`: MTF analysis, news integration, gates/penalties, quality grade, trade-setup generation → `SignalReport` |
 | `news_engine.py` | ~680 | Economic calendar (Forex Factory JSON + MyFXBook RSS fallback) + keyword RSS sentiment |
 | `telegram_bot.py` | ~1259 | Async Telegram bot: candle-boundary auto-scanner, WebSocket level-breach monitor, owner lock, subscriber channel |
 | `pair_signals/__init__.py` | 52 | `PAIR_PRODUCERS` registry + `produce_signal()` |
 | `pair_signals/xauusd.py` | 71 | XAU config factory + per-timeframe overrides (docstring = validation record) |
 | `main.py` | ~348 | CLI: `python main.py --pair XAUUSD --timeframe H1 --detail/--json` |
 | `test_all.py` | ~4100 | QA suite: **253 tests in 16 modules**, no live API needed. Must stay green. |
 | `backtest/gpu_backtest.py` | ~2231 | Bar-replay backtester (CuPy→NumPy fallback) using the real `SignalAggregator` |
 | `backtest/diagnose_filters.py` | 609 | Filter-diagnosis tooling |
 | `backtest/research/` | — | Calendar research infra (`ff_calendar_scraper.py`, `calendar_store.py`) + result CSVs |
 | `backtest/*.csv` | — | Verification trade books from past tuning campaigns (`xau_*.csv`, …). **Gitignored, local-only** — used for manual before/after comparison, not by tests |
 | `backtest/cache/` | — | Parquet OHLCV cache (raw bars) |
 | `logs/telegram.log` | — | Bot log (3-hour rotation × 240 backups = 30-day retention) |
-# Backtests
+---
 python backtest/gpu_backtest.py --pair XAUUSD --days 365
 python backtest/gpu_backtest.py --pair XAUUSD --start 2025-01-01 --end 2025-06-01 \
    --primary-tf 15min --confirm-tfs 1h,4h
 # key flags: --spread --quality A+,A,B,C --strategies --workers --max-hold-bars
 #            --warmup-bars --cooldown-bars --no-cache --export-csv out.csv --detail
 #            --fixed-rr-tp N (research-only: hard TP at N×risk, trail off; verdict
 #            2026-08-05: NOT shipped — backtest/research/fixedtp_2026-08-05.md)
 ```
-**`test_all.py` is NOT pytest** — it uses a custom `@test("name")` decorator that runs each test immediately at import; don't grep for `def test_` or try `pytest test_all.py`. Run the whole file; it prints per-module results and exits non-zero on failure. There is no way to run a single test short of commenting modules out. Tests must stay green; every new gate/knob needs tests. When patching weekend logic in tests, patch `providers.base._is_forex_weekend` — but know there are THREE independent weekend implementations (`providers/base.py`, `signal_aggregator.py`, and an inline copy in `backtest/gpu_backtest.py:_is_xau_blocked_by_calendar`); patching the provider one does not reach the other two.
+## Pair status
-Python 3.12; a `venv/` exists in-repo. Dependencies in `requirements.txt` (numpy, pandas, requests, twelvedata, python-telegram-bot ≥20, websockets).
+- **XAU/USD**: the tuned, validated product — and the ONLY supported pair. All defaults in `PlatformConfig`/`XauStrategyConfig` are its tuned values.
 - **EUR/USD**: REMOVED from the codebase (2026-07-16, branch `xau-only-chart-patterns`). It never had a validated edge: the 2026-07-11/12 verdict found no configuration passing validation on M5, M15, or H1 at a 1-pip round-trip cost; root cause was cost-to-volatility (EUR M15 ATR ≈ 10 pips → 1-pip spread taxes ~10% of ATR; gold's ATR/spread ratio is several times larger). Calendar-surprise and H1-hybrid avenues were also tested and closed. The full EUR implementation and verdict history live in git history (last commit with EUR support: `7d3fd95`). Do not re-add EUR without new data (tick fills or H4/D1 horizons).
-## Configuration & credentials
+---
-Precedence for every credential/deployment setting: **env var → config.json → refuse to run** (no hardcoded fallbacks). A gitignored repo-root `.env` is loaded at `app_config` import via `os.environ.setdefault`. `config.json` (gitignored; schema in `config.json.example`) holds secrets plus deployment settings: `data_providers.active` (the live-feed switch), `data_providers.streaming` (optional tick-stream provider for the level-breach monitor), `scanning.pairs`/`timeframes`, per-pair `blocked_hours_utc`, `log_level`. Key env vars: `TELEGRAM_TOKEN`, `TELEGRAM_CHAT_ID`, `TELEGRAM_CHANNEL_ID`, `TWELVE_DATA_API_KEY`, `TIINGO_API_KEY`, `OANDA_ACCOUNT_ID`/`OANDA_API_TOKEN`/`OANDA_BASE_URL`, `DEEPSEEK_API_KEY` (news LLM; the `deepseek` config.json block holds model/enabled/timeout/daily cap).
+## End-to-end pipeline (what happens on every scan)
 ## Architecture
 ### End-to-end pipeline (every scan)
 ```
-providers/<active>.fetch_ohlcv()       per TF: single-flight fetch, drop forming candle, validate/clean, cache
+Twelve Data REST (+ WebSocket ticks in the bot)
-                                       (only Twelve Data rate-throttles; Oanda has no client throttle)
+  → data_provider.fetch_ohlcv()        per timeframe: throttle, fetch, drop forming candle, validate/clean, cache
-  → engine.TechnicalEngine.analyze()   per TF: 9 scored indicators + ATR context → regime → composite score
+  → engine.TechnicalEngine.analyze()   per timeframe: 9 scored indicators + ATR context → regime → composite score
  → signal_aggregator.analyze_pair()   live entry point (weekend/day filters, fetch, staleness)
      └─ analyze_from_dataframes()     replayable core: MTF agreement → news → bonuses/penalties →
-                                       final signal + quality grade → gates → trade setup → reasons
+                                       final signal + quality grade → gates (session, SELL, M5, regime) →
-  → SignalReport                       consumed by telegram_bot.py, main.py, oanda_daemon.py, backtester
+                                       trade setup (3 generators) → reasons
  → SignalReport                       consumed by telegram_bot.py, main.py, backtest/gpu_backtest.py
 ```
-Signal currency everywhere is `SignalStrength` (−2..+2). DataFrames are UTC tz-aware, oldest-first, capitalized OHLCV columns. Everything runs on **closed bars only** — the forming candle is dropped, and session/hour filters evaluate **bar-CLOSE time** (bar label + candle duration) identically in live and backtest.
+Signal currency everywhere is the `SignalStrength` enum: `STRONG_BUY=2, BUY=1, NEUTRAL=0, SELL=-1, STRONG_SELL=-2`.
-### Layers
+---
- **`providers/`** — one class per provider subclassing `BaseDataProvider` (`providers/base.py`), which owns everything the pipeline depends on: TTL+boundary cache, forming-candle drop, freshness check, validate/clean, paginated `fetch_range` (feeds the backtester from any provider), optional tick streaming. Subclasses implement only `_fetch_raw` + symbol/interval maps. Registry + `make_data_provider(config, name=None)` in `providers/__init__.py`; config.json `data_providers.active` selects the live candle feed and the optional `data_providers.streaming` scalar selects a different provider for the tick stream. A test enforces that subclasses don't override shared base methods (`supports_streaming`/`iter_ticks` are the sanctioned streaming override points). Tiingo streams mid-price ticks (`wss://api.tiingo.com/fx`) — do not re-add spread display from its free FX feed (two-tier indicative quotes, removed 2026-07-23); the no-spread rule extends to the WS stream.
+## config.py — every knob
 - **`engine.py`** — `TechnicalEngine`: 9 scored indicators (ema_trend, rsi, macd, bollinger, adx, roc, keltner, divergence, candlestick) + ATR as context. Regime classification (trending_up/down, `trending` = mixed/direction-UNKNOWN, transitioning, ranging) drives per-indicator weight multipliers and regime-aware RSI reinterpretation. Compute order matters (ATR first; MACD always computed even when not a pillar — divergence needs the histogram). Private series ride on the result as `_underscore` attrs; scorers/formatters skip anything `_`-prefixed or weight-0.
 - **`signal_aggregator.py`** — MTF agreement, diminishing-decay penalty system, exhaustion hard block, quality grading, then an ordered chain of post-grade gates (session, XAU SELL quality/vol/hour, regime blocks, M5 noise gates), then three setup generators: XAU ATR (A+/A), generic swing-based, plus the M5 pullback (`XAU_PB`) and chart-pattern (`XAU_CP`, `chart_patterns.py`) entry paths. The PB/CP generators run AFTER the gate chain and force quality to "B"; the CP path deliberately accepts grade-F composites and re-checks only session/day/big-candle (via warning prefixes) plus its own SELL/vol/RSI gates — it does NOT re-check exhaustion or the M5 min-score/MTF gates. **This is validated behavior, not a bug** (2026-08-04 audit: 226/586 trades in the shipped M5 12mo book are XAU_CP, +38.9R, 104 of them exhaustion-flagged and net-positive) — changing it is a rule-1 revalidation event. `TradeSetup.strategy` selects trailing params; trail params are stamped on the setup from the *producing* config so per-TF overrides reach formatters.
 - **`pair_signals/`** — per-pair config factories. **Always use the factory (`produce_signal` / `make_config`), never bare `PlatformConfig()`** — the factory applies the config.json overlay first, then the validated per-TF deltas (M5 hour-9 block, M15 trail arm 0.70, M15 SELL-hour block, etc.; evidence in the docstring). A test enforces factory ≡ inline-config equivalence. Confirmation map: M5→[M15,H1], M15→[M5,H1], H1→[M15,H4], H4→[H1,D1].
 - **`news_engine.py`** — Forex Factory calendar (MyFXBook RSS fallback) + RSS keyword sentiment with gold-inverse keywords. Class-level caches (fresh engine per scan). Optional DeepSeek enrichment via `llm_client.py` (requests-only, no SDK): batched headline sentiment overriding keyword scores, gold-direction on release surprises, and a market-summary blurb (`SignalReport.news_summary`) — **display/warning-only, keyword fallback on any failure, inert without a key**; prompts are static module constants (DeepSeek prefix-cache pricing) and requests send `thinking: disabled` (V4-Flash reasoning otherwise blows the timeout). By XAU-tuned default, event risk and sentiment **warn but do not gate signals** (`event_risk_caps_signals=False`, `sentiment_affects_signals=False`) — the LLM does not change that.
 - **`telegram_bot.py`** — owner-locked bot + subscriber channel; candle-boundary auto-scan loops (M5/M15/H1/H4, boundary+10 s), alert quality threshold + fingerprint dedup (committed only after a successful send), 📰 LLM market-context line on alerts when available, WebSocket level-breach monitor fed by the streaming provider (`data_providers.streaming`, falling back to the active provider; auto-disabled when the resolved provider can't stream or has no API key — Oanda does not stream, Tiingo does). Rotating log: `logs/telegram.log`, 3-h rotation × 240 backups.
 - **`oanda_daemon.py` / `oanda_executor.py`** — practice-account auto-execution. Book-parity rules (see daemon docstring): no broker-side TP (trailing model — TP1/TP2 advisory), 5-bar per-TF cooldown (wall-clock approximation, so it expires over weekends where the backtester would still count bars), trail ratchets at bar close once per *observed* bar (a missed/failed cycle skips those bars entirely — the intra-gap high/low never enters the peak). Accepted divergence: one position total; orders send `positionFill=OPEN_ONLY` so a state/broker divergence cancels the order instead of silently netting. State/broker re-sync runs at boot and after any suspicious fill (`_reconcile_positions`).
 - **`dashboard_publisher.py` / `static/dashboard.html`** — web dashboard as a **static publisher** (live LAMP server): standalone daemon scans at candle closes via `produce_signal` and atomically writes `snapshot.json`/`news.json`/`dashboard.html` to `$DASHBOARD_OUT_DIR` (Apache serves them; page is public by owner decision). **Hard independence rule: never imports, and is never imported by, `telegram_bot`/`oanda_daemon`** — confirmation map and boundary math are deliberate local copies, so a web-side failure can only mean a stale page. Due-TF classification uses the RAW wake time (flooring wake−buffer skips every H1/H4 scan — 2026-08-02 bug). News panel reads `NewsEngine.get_fundamental_context()` plus the additive `get_week_events()` accessor (display-only; no signal path calls it); as a separate process it has its own news caches and DeepSeek daily cap. News/calendar rows are clickable (detail dialog); headlines carry `id` = sha1(headline) in news.json, and exactly one on-demand endpoint exists: `static/news_summary.php` (installed by `install_static`), which serves a DeepSeek gold-impact summary per headline — whitelist-gated against news.json ids, cache-forever in `ai/`, own fail-closed daily cap (150/day, file-based, separate from the publisher's), key at `/etc/gold-dashboard/deepseek.key`, never reads repo config. Deployment checklist: `deploy/README-dashboard.md`.
 - **`backtest/gpu_backtest.py`** — bar-by-bar replay through the real `analyze_from_dataframes` (news off). Realism model (do not regress): next-bar-open entries re-anchored to the raw price path, round-trip `--spread` charged per trade, trade resolution on RAW bars (cleaning applies only to signal-side data), same-bar SL+TP tie → SL first, gap fills at bar open, trail ratchet at bar close only. Indicators are precomputed once per TF (CuPy GPU or NumPy fallback — check the `Backend:` log line); replay fans out over worker processes and the 5-bar cooldown is applied in an ordered pass afterwards, so parallel and sequential books are identical. Parquet cache in `backtest/cache/` keyed by provider/symbol/TF/window. `backtest/*.csv` are gitignored pinned trade books from past campaigns — used for manual before/after comparison. `backtest/instructions_for_backtest.txt` lists what every backtest report must record.
-## Rules for making changes (read before editing signal logic)
+### PAIR_INFO (market constants; key is the canonical symbol `"XAU/USD"`)
 | Field | XAU/USD | Meaning |
 |---|---|---|
 | `pip_value` | 0.10 | price units per pip |
 | `price_precision` | 2 | decimals for SL/TP rounding & display |
 | `default_spread` | 0.30 | round-trip cost charged by the backtester when `--spread` not given |
-1. **Never change a tuned value without re-validation.** Acceptance rule: positive total R in BOTH train and validation halves of a ≥12-month window, PF ≥ 1.3, at realistic cost (XAU $0.30 round trip), trailing exits, next-open fills. Marginal edges must additionally pass the 2024-06→2025-03 out-of-sample window — within-year splits alone have approved configs that later failed OOS.
+### TechnicalConfig (indicator parameters)
-2. **Per-timeframe overrides live in the pair factory**, never in global defaults — the same knob has opposite effects on different TFs (proven repeatedly: `tp_floor_rr`, hour-9 block, trail params). Document validation evidence in the factory docstring.
+EMA 9/21/50/200 (`ema_mode`: `"4ema"` default or `"2ema"` = 50/200 only); RSI 14 (OB 70 / OS 30 / neutral zone 40–60; `rsi_mode`: `"regime_aware"` default or `"classic"`); MACD 12/26/9 (`macd_as_pillar=True` — MACD is a scored indicator, not just divergence input); BB 20/2.0σ; ATR 14; ADX 14 (strong ≥25, very strong ≥50); divergence lookback 40; swing-S/R lookback 50; ROC period 10, threshold 1.0%; Keltner: EMA 20 mid ± 1.5 × ATR(10); ADX slope lookback 5, "emerging" threshold 3.0; `disabled_indicators` list (removes the pillar from scoring; for MACD the histogram is still computed — divergence needs it); `regime_weights_override` dict (replaces the gold regime-multiplier table for experiments).
 3. **Pair isolation is a hard invariant.** XAU gates in the aggregator are display-name-gated so a future pair port cannot inherit gold-calibrated logic by accident.
 4. **Live == backtest parity.** If you add a live gate, add it to the replay path in the same commit; all time filters evaluate bar-close time in both paths.
 5. **ROC stays weighted** even though it's correlated with MACD/Keltner/EMA — zeroing it backtested 13R worse; composite thresholds are calibrated with it.
 6. **Backtester realism knobs are results-critical** (raw-path resolution, spread charging, next-open entry, SL-first ties, close-only trail). Touching them invalidates comparability with all pinned books.
 7. **The forming candle must stay dropped** — live signals repaint otherwise.
 8. Don't add config fields nothing reads; don't disable indicators in production paths (`disabled_indicators` is for experiments only).
 9. EUR/USD was removed 2026-07-16 after a no-edge verdict (cost-to-volatility; last commit with EUR support: `7d3fd95`). Do not re-add without new data. BTC/ETH work lives on unmerged branches only.
-## 2026-08-04 QA audit — owner decisions ledger
+### SignalConfig (scoring & trade geometry — XAU-calibrated)
 | Field | Default | Meaning |
 |---|---|---|
 | `mtf_agreement_threshold` | 2 | timeframes (of primary + confirmations) that must agree |
 | `min_rr_ratio` | 0.5 | reject setups below this RR |
 | `tp_tightening_factor` | 1.0 | TP scale factor. **Leave at 1.0**: 0.8 was re-tested 2026-07-08 with spread+gap fills and nearly halved total R |
 | `round_trip_cost` | 0.30 | XAU $ round trip |
 | `min_tp_cost_multiple` | 30.0 | spread gate: TP1 distance must be ≥ cost × this (= $9 on XAU). Validated: sub-$10 targets lose |
 | `min_sl_pips` | 80.0 | SL floor in pips ($8 on XAU) |
 | `tp_floor_rr` | 0.5 | TP1 floor as fraction of SL distance (M15 factory overrides to 1.0) |
 | `trail_arm_r` / `trail_dist_r` | 0.6 / 0.75 | trailing exit: once favorable excursion ≥ arm×risk, stop trails dist×risk behind best price (bar-close updates). TP1 becomes advisory. Validated on 12mo M5: +105R vs +48R static. 0 disables. **M15 factory overrides arm to 0.70** (2026-07-21 grid: +210.7R vs +175.7R/5yr, OOS +57.4 vs +45.4; DD clause waived by owner); **M5 factory overrides dist to 0.85** (same-day grid: +231.2R vs +218.0R/5yr, OOS +14.5 vs +11.1, DD flat; dist 1.0 rejected at the OOS gate — see factory docstring) |
 | `pb_trail_arm_r` / `pb_trail_dist_r` | 0.8 / 0.75 | trailing params for XAU_PB pullback trades (tighter stops → arm later) |
- **Weekend calendar: FIXED (owner-approved 2026-08-04).** All three `_is_forex_weekend` copies are NY-anchored (Fri 17:00 → Sun 18:00 ET; during US DST that is Fri 21:00 UTC — measured on 5yr of Oanda bars). Kills the weekly false-"STALE DATA/feed lagging" hours. Book impact, fully accounted: M15 byte-identical; M5 12mo dropped exactly ONE trade — a phantom whose signal fired on the final summer-Friday bar (close 21:00 UTC) and "filled" at Sunday's reopen 49 h later (−1.08R, live-impossible). M5 12mo reference at HEAD: 555 trades, +98.64R PF≈1.50 @ $0.65 (was 556/+97.56/1.48). A Module 21 test sweeps all three copies hourly across both DST transitions.
+### NewsConfig
- **Spread in exit-trigger detection (rule 6): owner DECLINED 2026-08-04.** Backtest stop triggers stay on raw mid High/Low; the measured optimism (~4–6% of total R: −3.55R @ $0.30, −4.24R @ $0.65 on the M15 12mo book) is accepted and books stay comparable. Re-open only as a conscious re-pin campaign.
+Tracked currencies `["USD","XAU"]`; pre-event buffer 30 min, post-event 15 min (45 min after a *surprise* print); `event_risk_caps_signals=False` — **XAU default: event risk warns but does NOT cap confidence** (2026-07-11 audit: the caps suppressed trades worth +4.6..+7.2R/yr; XAU trades near USD releases outperform); `sentiment_affects_signals=False` — **headline sentiment is informational only** (warnings still emitted; the +5 alignment bonus and 10-pt `news_conflict` penalty are gated off, so live decisions are pure technical, matching backtests). Context cache TTL 600 s (must stay < pre-event buffer). 4 RSS feeds (investinglive, FXStreet, Google-News gold query, CNBC economy).
 - **`FB2_EMA_LO`: HARDENED 2026-08-04 (delegated).** Research override for the EMA200-proximity band, default = tuned 0.5; an unparseable value now degrades to the default with a warning instead of crashing every scan. Documented here — it is a knob that exists.
 - **H4 seeding divergence: parked.** Live H4 fetches only 380 bars vs the backtest's unbounded history — ~2% of sampled H4 bars flip an indicator signal (composite Δ up to 12.3). M5/M15/H1 clean at their real fetch depths. Fold into any future H4 campaign; H4 remains "unproven for auto-execution".
-## Current state notes (2026-07-30)
+### XauStrategyConfig (all XAU gates; `enabled=True`)
 | Field | Default | Notes |
 |---|---|---|
 | `blocked_hours_utc` | [0,1,2,10,11,13,16,21] | bar-close hours never traded. M5 factory appends 9 (post-London-open whipsaw, −28.9R/143 over 6 yr) |
 | `allowed_weekdays` | [0,1,2,3,4,6] | Mon–Fri + Sun (Sydney open). `weekend_filter=True`: Fri 22:00 → Sun 22:00 UTC closed |
 | `min_atr_threshold` | 1.0 | $1 ATR floor for swing-based setups |
 | `v2_sl_atr_multiplier` | 2.0 | SL ATR floor in the generic setup generator |
 | `sl_atr_multiplier` / `tp_atr_multiplier` | 2.0 / 1.0 | used by the XAU ATR setup generator (A+/A only) |
 | `sell_min_quality` | ["A+","A","B"] | SELLs below B are blocked |
 | `sell_vol_regimes` | ["high","extreme"] | SELLs only in these volatility regimes (quiet shorts fade vs gold's macro uptrend) |
 | `blocked_regimes` / `sell_blocked_regimes` | [] / [] | per-TF factory opt-ins. M15 factory sets `blocked_regimes=["trending"]` (mixed regime: strong ADX, EMAs unaligned — lost −6.8R/24) |
 | `sell_blocked_hours_utc` | [] | bar-CLOSE UTC hours where SELLs are blocked (BUYs unaffected). Per-TF factory opt-in: M15 sets [15, 17] (NY-session shorts, −33.5R/165 over 5 yr, negative 6/6 years both halves; BUYs there +34.3R) |
 | M5 gates | | `m5_require_mtf=True`; `m5_min_abs_score=30`; `m5_min_confirmation_score=10`; `m5_sell_max_quality=["B","C"]` (A-grade M5 SELLs enter at dip extremes and lose); `m5_blocked_vol_regimes=["high"]` (chop transition — 61% WR, negative total); `m5_sell_min_rsi=25` |
 | M5 pullback (XAU_PB) | | `m5_pullback_enabled=True`; min score 15 (lower than gate 1 BY DESIGN — mid-pullback scores weaker); ADX ≥ 20; RSI zones BUY (35,62) / SELL (38,65); retrace depth 0.8–6.0 ATR from the 40-bar swing extreme; band touch within 3 bars. Validated +38.3R/254 trades on 12mo |
 | Chart patterns (XAU_CP) | `cp_enabled=False` | `cp_*` fields: pattern list, lookback 72, swing window 3, recency 20, double tolerance 0.5 ATR, height 1.5–8 ATR, H&S head ≥0.5 ATR / shoulder tol 1.0 ATR, flag pole ≥3 ATR in ≤12 bars, flag 3–12 bars retracing 10–50%. Textbook defaults, NOT tuned. **Default OFF** — M5 factory enables BUY-only (`inv_hs`, `double_bottom`, `bull_flag`); M15/H1/H4 remain disabled (all failed OOS validation 2026-07-16). Bearish patterns (`hs_top`, `double_top`, `bear_flag`) dropped: SELL CP lost −3.21R on OOS, −4.52R on M15 5yr |
- Oanda v20 is the live feed (`data_providers.active: "oanda"`), but the pinned backtest books are still Twelve-Data-pinned and the backtest cost assumption is $0.30 while measured real spreads are ~$0.65–0.72 — re-pin books on Oanda bars before treating feed-switch comparisons as valid (see Oanda/TD divergence study in memory).
+### FilterConfig (`config.backtest` — despite the name it applies LIVE too)
- Human-facing docs in `docs/` (`user_guide.html`, `beginner_guide.html` + PDF builds) must be kept in sync when signal behavior changes.
+`session_filter=True`, window 03:00–23:00 UTC, `blocked_hours_utc` mirrors XAU's list (non-XAU pairs read this copy), `allowed_weekdays` Mon–Fri+Sun. `regime_filter=False` (optional H4-ADX≥25 gate, off).
 ### PlatformConfig (top level)
 `symbol="XAU/USD"`, primary TF H1, confirmations [M15, H4], `data_lookback_days=60`, `scan_pairs=[XAUUSD]` (mirrors config.json `scanning.pairs` — see below), `scan_timeframes=[M5,M15,H1,H4]`, `ws_cooldown_seconds=300`.
 Per-pair base blocked hours plus `scan_pairs`/`scan_timeframes` overlay from config.json via `app_config.apply_pair_overrides`, applied in the pair factory BEFORE per-TF deltas. Credentials, alert quality, and log level never touch `PlatformConfig` — `telegram_bot.py` and `data_provider.py` read them directly from `app_config.get_app_config()` and its resolvers. `scanning.pairs` in config.json is the deployment switch for what the bot scans — `PlatformConfig.scan_pairs` just carries the resolved value.
 ---
 ## engine.py — TechnicalEngine
 **Input contract:** pandas DataFrame, UTC tz-aware DatetimeIndex, oldest-first, columns `Open High Low Close Volume` (capitalized). `< 30` bars → empty NEUTRAL result. EMA200 needs ≥ 200 bars (otherwise EMA-trend indicator degrades gracefully).
 **Compute order matters:** ATR first (EMA-ribbon and candlestick proximity use it) → EMA → RSI → BB → ADX → MACD (always computed, even if not a pillar — divergence needs the histogram) → ROC → Keltner (needs BB levels + MACD histogram) → divergence → structure S/R → candlesticks → regime classification → regime-aware RSI reinterpretation → composite.
 Results land in `TechnicalAnalysis.indicators: Dict[str, IndicatorSignal]` (fields: name, value, signal, confidence 0-100, description, weight). Private series ride on the result object as underscore attrs (`_rsi_series`, `_macd_histogram`, `_adx_slope`, `_kc_squeeze`, `_kc_squeeze_released`) and are skipped by the composite scorer and all formatters (skip rule: name starts with `_` or `weight == 0`).
 ### The 9 scored indicators (+ ATR as context)
 | Key | Base weight | Signal logic (exact) |
 |---|---|---|
 | `ema_trend` | 1.0 | 4-EMA mode: alignment score over checks (price>EMA9, 9>21, 21>50, 50>200), normalized to [-1,1]. >0.6 STRONG_BUY (conf ≤90), >0.2 BUY, mirrored sells, else NEUTRAL (40). Fast/medium cross ±10 conf. Ribbon (9/21/50 spread < 1 ATR) = "COMPRESSED"; compressed→expanded = +10 conf. Emits `ema_fast/medium/slow/trend` key levels |
 | `rsi` | 1.5 | Wilder RSI (ewm α=1/14). Classic pass first: ≥70 SELL, ≤30 BUY, ≥60 BUY-momentum (45), ≤40 SELL-momentum (45), else NEUTRAL (30). **Then overwritten by regime reinterpretation** (below) when `rsi_mode="regime_aware"` |
 | `macd` | 1.5 | Histogram = MACD line − signal line. Cross above 0 → BUY (conf 55+); accelerating (hist>prev>0) → STRONG_BUY; mirrored down; positive-but-fading / negative-but-fading → NEUTRAL (35); flat → NEUTRAL (30). Confidence scaled by |hist| vs its 20-bar mean |
 | `bollinger` | 1.5 | Position = (close−lower)/(upper−lower). >1.0 SELL (65), <0.0 BUY (65), >0.8 SELL (45), <0.2 BUY (45), else NEUTRAL (30). Squeeze = width < 0.75 × 20-bar avg width → +10 conf. Emits `bb_upper/lower/mid` |
 | `adx` | 0.5 | Wilder ADX/DI (ties zero BOTH DMs — fixes a systematic bearish nudge). ADX ≥ 25 → BUY/SELL by DI dominance, else NEUTRAL. Conf 40/65/80 by ADX band. ADX slope (5-bar diff): <25 with slope > 3.0 → "TREND EMERGING" +15 conf; also strengthening/weakening adjustments. Sets `_adx_slope` |
 | `roc` | 1.5 | 10-period %change. Positive & rising → BUY (STRONG if >2× threshold=2%); negative & falling → SELL (zero-crosses land in these branches by definition — a separate cross branch was provably dead and removed 2026-07-23); else NEUTRAL. **Directionally redundant with MACD/Keltner/EMA (0.8–0.94 corr) but its weight is load-bearing: zeroing it backtested 13R WORSE — composite thresholds are calibrated with it. Do not remove without retuning thresholds** |
 | `keltner` | 1.5 | EMA20 mid ± 1.5×ATR(10). TTM squeeze = BB fully inside KC. Squeeze released (was→not) + MACD hist sign → STRONG_BUY/STRONG_SELL (80); squeeze active → NEUTRAL (60); close above/below channel → BUY/SELL continuation (65); inside → NEUTRAL (30). Emits `kc_upper/lower/mid` |
 | `divergence` | 1.5 | RSI + MACD-hist vs price over 40 bars. Swing points (window 3) must align within ±2 bars to pair up; last two pairs: price LL + indicator HL = bullish, price HH + indicator LH = bearish. Any found → BUY/SELL/NEUTRAL-if-both, conf 70, value = count |
 | `candlestick` | 1.0 | Last bar: Doji (body<10% range), Hammer/Hanging Man (lower wick>2×body), Shooting Star/Inverted Hammer, Engulfing (vs prev bar), Marubozu (body>85%). Majority vote → signal, conf 50; +20 if pattern sits within 0.5 ATR of a swing S/R level |
 | `atr` | **0.0** (context) | Wilder ATR. Volatility regime from percentile vs last 50 ATR values: >80 extreme, >60 high, >30 normal, else low. Emits `atr`, `atr_pct` |
 Structure S/R (not scored): swing highs/lows (window 3) over `swing_sr_lookback` (min 120 bars on 1min/5min TFs); up to 3 above and 3 below stored as `swing_high_1..3` / `swing_low_1..3` key levels.
 ### Market regime classification (exact rules)
 - ADX ≥ 25 **and** EMA-trend signal bullish → `trending_up`; bearish → `trending_down`; EMA mixed → **`trending`** (the "mixed" regime — momentum without directional structure; treated as direction-UNKNOWN everywhere; a past bullish-bias bug here cost −6.8R and is fixed)
 - else ADX slope > 3.0 → `transitioning` (trend birth, ~5–10 bars before ADX crosses 25)
 - else → `ranging`
 ### Regime-aware RSI reinterpretation (`rsi_mode="regime_aware"`, the default — classic mode backtested worse for gold)
 | Regime | RSI zones |
 |---|---|
 | trending_up | 38–55 STRONG_BUY 75 ("pullback zone, ideal entry"); 30–38 BUY 60; 55–65 BUY 50; >65 NEUTRAL ("extended, wait"); <30 NEUTRAL ("possible reversal") |
 | trending_down | mirrored: 45–62 STRONG_SELL 75; 62–70 SELL 60; 35–45 SELL 50; <35 / >70 NEUTRAL |
 | transitioning / trending (mixed) | ONLY extremes: ≥70 SELL 50, ≤30 BUY 50, else NEUTRAL 30 |
 | ranging | ≥70 SELL 70, ≤30 BUY 70, ≥60 SELL 45, ≤40 BUY 45, else NEUTRAL |
 ### Composite score
 For each scored indicator: `effective_weight = base_weight × regime_multiplier`; contribution = `signal.value × effective_weight × confidence/100`. Composite = `(Σ contributions / Σ effective_weights) × 50`, clamped to ±100. Confidence = weight-averaged indicator confidence. Composite signal: ≥40 STRONG_BUY, ≥15 BUY, ≤−40 STRONG_SELL, ≤−15 SELL, else NEUTRAL.
 **Gold regime multipliers** (defaults in `_compute_composite`; `technical.regime_weights_override` replaces the whole dict):
 | Indicator | trending_up/down/trending | ranging | transitioning |
 |---|---|---|---|
 | ema_trend | 2.0 | 0.3 | 1.5 |
 | rsi | 1.0 | 1.5 | 1.0 |
 | macd | 2.0 | 1.5 | 2.5 |
 | bollinger | 0.5 | 1.5 | 1.0 |
 | keltner | 1.5 | 2.0 | 2.5 |
 | roc | 1.5 | 0.5 | 2.0 |
 | candlestick | 0.8 | 1.0 | 1.0 |
 | divergence | 1.0 | 2.5 | 2.0 |
 | adx | 0.8 | 0.5 | 1.0 |
 ---
 ## signal_aggregator.py — SignalAggregator
 Two entry points:
 - **`analyze_pair(symbol)`** — LIVE: applies XAU weekend + allowed-weekday filters on wall clock, fetches all TFs (primary + confirmations), flags stale data (primary stale → signal suppressed to NEUTRAL/F), delegates to `analyze_from_dataframes`, then applies the live session check on the **bar-CLOSE time** (bar timestamp + candle duration): outside 03:00–23:00 or in blocked hours → NO TRADE. This is the exact predicate the backtester replays, so live == backtest.
 - **`analyze_from_dataframes(symbol, dfs, context, use_news)`** — replayable core; treats each df as the visible history at decision time (caller must slice; no weekend filter here — the backtester applies it).
 ### Pipeline inside `analyze_from_dataframes` (numbered as in the source)
 1. Primary TF analysis → seeds report (score, confidence, regime, vol regime, price, timestamp = last bar's index label).
 2. Confirmation TF analyses → `mtf_signals` / `mtf_scores` keyed by TF string ("5min", "1h", …).
 3. **MTF agreement**: count BUY-ish vs SELL-ish across ALL TFs (primary included); agreement = count ≥ 2 **in the primary's direction** (two bullish confirms cannot "confirm" a bearish primary).
 4. News (if `use_news`): event risk (`clear`/`caution`/`danger`) + pair sentiment (−1..+1). Backtests pass `use_news=False`.
 5. Spread estimate = median(High−Low) of primary df.
 6. **`_compute_final_signal`** (bonuses → event cap → penalties → thresholds → grade):
   - Bonuses: MTF agreement → +10 conf and `score = 0.6×primary + 0.4×mean(all TF scores)`; news sentiment aligned (|sent|>0.2) → +5 (gated by `news.sentiment_affects_signals`, False by default); in a *directional* trend, RSI in pullback zone → +5, BB position favorable → +5 (mixed `trending` regime gets NO directional bonus).
   - Event risk `danger`: warning always; confidence hard-capped at 30 ONLY if `news.event_risk_caps_signals` (False by default — XAU-tuned).
   - **Penalty system (diminishing, replaces old additive stacking):** collect, sort by size desc, apply with decay ×1.0, ×0.5, ×0.25…, total capped at 30. Penalties: `mtf_disagreement` 15; `news_conflict` 10 (gated by `sentiment_affects_signals`, off by default — warning still emitted); `event_caution` 15 (config-gated); `counter_trend` 15; `extreme_vol` 10; `ema200_near` 5 (price within 0.5% of EMA200 — env `FB2_EMA_LO` overrides); `ema200_far` 5 (>2.5%); `exhaustion` 8 (price >3.5 ATR beyond the FARTHEST of the 3 tracked swing levels — fires earlier than a nearest-swing rule would; calibrated as-is in every validated book); `no_pullback` 8 (>2.5 ATR from EMA21 in trend) / `weak_pullback` 5 (>2.0); `rsi_extended` 5 (RSI>65 with BUY / <35 with SELL in any strong-ADX regime, both directions in mixed `trending`); `bb_overextended` 8 (BB pos >0.85 BUY / <0.15 SELL).
   - **Exhaustion hard block**: ≥2 exhaustion-tagged penalties ({exhaustion, no_pullback, rsi_extended, bb_overextended} → `report.exhaustion_count`) AND momentum fading (ROC sign against score, or MACD signal against score) → confidence = 0.
   - Confidence clamped [0, 95]. **Final signal**: score ≥35 & conf ≥55 STRONG_BUY; ≥25 & ≥50 BUY; mirrored sells; else NEUTRAL.
   - **Quality grade**: |score|≥40 & conf≥65 & MTF → **A+**; ≥33 & ≥55 & MTF → **A**; ≥28 & ≥50 → **B**; ≥25 & ≥45 → **C**; ≥15 → **D**; else **F**.
 7. Post-grade gates, in order:
   - **6b session filter** (bar-label hour + weekday; XAU reads `xau_strategy.blocked_hours_utc`/`allowed_weekdays`, others read `config.backtest.*`) → NEUTRAL/F.
   - **6c transitioning boost**: regime `transitioning` and grade B/C/D → A if |score|≥30 & conf≥50 & MTF; else B if |score|≥25 & conf≥45 (early trend entries).
   - **6d exhausted-trend cap**: trending regime + exhaustion_count ≥2 + grade C/D → D, setup killed.
   - **6e XAU SELL quality gate**: sell-direction & grade actionable (A+..C) & not in `sell_min_quality` → D. The actionable guard is load-bearing: without it a session-blocked F report was *upgraded* to D, re-opening the M5 pullback path (its entry check is `!= "F"`).
   - **6f MTF opposition**: ≥2 TFs pointing against the trade direction → D (never trade against consensus).
   - **6g XAU SELL volatility gate**: sell-direction & vol regime not in {high, extreme} → D.
   - **6g2 XAU regime gates**: regime in `blocked_regimes` (M15: mixed `trending`) → D; sell-direction & regime in `sell_blocked_regimes` → D.
   - **6g3 XAU SELL hour gate**: sell-direction & bar-CLOSE hour (label + candle duration — the entry hour under next-open fills) in `sell_blocked_hours_utc` → D. Per-TF opt-in; M15 factory blocks 15/17 UTC.
   - **6i XAU M5 noise gates** (`_apply_m5_filters`, primary TF = M5 only): (1) |score| ≥ 30; (2) MTF agreement required; (3) ≥1 confirmation TF with score ≥ ±10 in trade direction (NEUTRAL confirms don't count); (4) SELL grade must be B/C (A-grade M5 shorts lose −0.017R/trade); (5) vol regime `high` blocked; (6) SELL blocked when RSI < 25. Each failure → D, setup killed.
 8. **Trade setup**: XAU + grade A+/A → `_generate_xau_atr_trade_setup` first; any pair + grade A+/A/B/C and no setup yet → `_generate_trade_setup` (swing-based).
 9. Reasons: every non-neutral indicator description → `bullish_reasons`/`bearish_reasons` (the backtester tags strategy from reasons[0] markers).
 10. **XAU_PB pullback** (M5 XAU only, `m5_pullback_enabled`, no setup yet, grade ≠ F): `_try_m5_pullback` — see below.
 11. **XAU_CP chart pattern** (any TF, XAU only, `cp_enabled`, no setup yet, NOT session/day-blocked — weak-composite grade F is allowed BY DESIGN, reversals complete on weak composites): `_try_chart_pattern` — see below.
 ### Trade setup generators (exact geometry)
 **A. `_generate_xau_atr_trade_setup`** (XAU, grade A+/A, ATR>0): TP quality scale A+ 2.0 / A 1.5 (A with exhaustion≥2 → 1.0). TP1 = entry ± `tp_atr_multiplier`(1.0)×scale×ATR; TP2 = 1.5×that; SL = entry ∓ 2.0×ATR. Gates: RR ≥ 0.5; TP1 distance ≥ $9 (spread gate). Prices rounded to 2 dp; pip 0.10 hardcoded — **this generator is XAU-only by construction**.
 **B. `_generate_trade_setup`** (swing-based, any pair, grade A+..C): XAU requires ATR ≥ $1 (LOW ATR FILTER). Per-side:
 - SL anchor = nearest swing low below price (BUY) / swing high above (SELL) ∓/± `sl_swing_buffer`×ATR (1.0); fallback (no swing) = (1+buffer)×ATR. SL distance floored at max(ATR × sl_atr_floor, min_sl_pips×pip) — XAU: max(2.0×ATR, $8).
 - TP1 = nearest swing beyond entry ± 0.3 ATR if it's within `tp_swing_cap` ATR (2.5) AND ≥ SL×RR-floor; else entry ± max(`tp_mult`×ATR, SL×RR-floor). tp_mult 1.5, RR-floor = max(min_rr 0.5, tp_floor_rr — 1.0 on M15).
 - TP2 = second swing if within 5 ATR, else entry ± max((tp_mult+1.5)×ATR, SL×floor×1.5); forced beyond TP1 by ≥1 ATR.
 - Gates: RR ≥ min_rr; spread gate TP1 distance ≥ cost×multiple ($9 on XAU).
 **C. `_try_m5_pullback` (XAU_PB)**: detection = EMA alignment ≥3 of 4 checks (9>21, 21>50, 50>200, close>200 — mirrored for SELL), ADX ≥ 20, price touched max(EMA21, KC-mid) within last 3 bars, resumption bar closed back in trend direction beyond EMA9, retrace depth 0.8–6.0 ATR from the 40-bar extreme, RSI in pullback zone (BUY 35–62 / SELL 38–65). Then gates: directional score ≥ 15; ≥1 confirmation TF ≥ ±10; not against MTF consensus; vol regime gates (same as breakout path). Geometry: SL = pullback 8-bar extreme ∓ 0.5 ATR (min $8); TP1 = 1.5×risk, TP2 = 2.5×risk; spread gate $9. Sets `strategy="XAU_PB"`, grade min B, marker `"XAU PULLBACK STRATEGY: …"` at reasons[0]. Rejection funnel counters in class attr `SignalAggregator.pb_funnel`.
 **D. `_try_chart_pattern` (XAU_CP)**: detection in `chart_patterns.py` — 6 classical patterns (double_bottom/double_top, inv_hs/hs_top, bull_flag/bear_flag), swing-point based (strict local extrema, window `cp_swing_window`=3), all thresholds in ATR units, scanned over the last `cp_lookback`=72 closed bars. Trigger = FIRST close beyond the neckline/boundary since the pattern formed (one breakout → one signal, no repaint); reversal patterns are invalidated by a new extreme beyond the pattern. No composite-score gate BY DESIGN. Gates: MTF non-opposition (6f mirror); SELL patterns respect `sell_vol_regimes`, `sell_blocked_hours_utc` (bar-close hour), M5 RSI floor; M5 blocks `m5_blocked_vol_regimes` both ways; ATR ≥ $1. Geometry: SL = pattern anchor ∓ 0.5 ATR (min $8); TP1 = measured move (pattern height), TP2 = 1.5×height; RR ≥ 0.5; spread gate $9. Sets `strategy="XAU_CP"` (global trailing params), grade min B, marker `"XAU CHART PATTERN: …"` at reasons[0]. Funnel in `SignalAggregator.cp_funnel`.
 `TradeSetup` fields: pair, direction, entry, SL, TP1, TP2, RR, risk/reward pips, pip_value, `strategy` ("GENERIC", "XAU_PB", or "XAU_CP" — selects trailing params; XAU_CP uses the global pair), `trail_arm_r`/`trail_dist_r` (stamped from the PRODUCING config at generation so per-TF factory overrides reach formatters; `None` = legacy unstamped → display falls back to platform defaults).
 ---
 ## news_engine.py — NewsEngine
 - **Calendar**: Forex Factory free JSON (`nfs.faireconomy.media/ff_calendar_thisweek.json`), 3 retries, honors 429 Retry-After with class-level backoff (≤600 s); fallback MyFXBook RSS. Never fabricates events — unavailable calendar = risk checks skipped that cycle.
 - **Caches are class-level** (a fresh NewsEngine is built per scan): calendar + headlines, TTL = 600 s on success, 60 s on empty result; lock held across fetch so concurrent scans share one request.
 - **Event risk** (`get_event_risk_level`): high-impact event for base/quote within 30 min → `danger`; within 60 min → `caution`; already-released high-impact → `caution` for 15 min after (45 min if actual ≠ forecast — "surprise"). `EconomicEvent.surprise` compares parsed numeric actual vs forecast (handles "%", "K/M/B" suffixes); direction is numeric-only (doesn't know if higher is bullish).
 - **Sentiment**: keyword sets over headlines from 4 RSS feeds, deduped by title (Google News " - Publisher" suffix stripped), last 24 h only. Per-currency scores; **`GOLD_INVERSE_KEYWORDS`** (rate hike/hawkish/tightening/rate cut/dovish/easing/stimulus) flip sign for XAU (hawkish = dollar-bullish = gold-bearish). Currency detection via compiled word-boundary-ish regexes (bare "us"/"eu"/"uk" deliberately excluded). Pair sentiment = base − quote.
 ---
 ## providers/ — data-provider abstraction (refactored 2026-07-25)
 One class per provider, all subclassing **`providers/base.py BaseDataProvider`**, which owns every behavior the signal pipeline depends on — defined ONCE (a test enforces that neither subclass overrides shared methods):
 - **In-memory cache**: key `{symbol}_{tf}`, TTL = candle duration (M5 300 s … H4 14400 s); a cached frame is evicted and refetched if it fails the freshness check **or the boundary check** (`_has_latest_closed_bar`: the frame must contain the most recent fully closed bar, weekend-exempt — a wall-clock TTL alone would serve mid-candle snapshots at the next boundary and scans would run one bar behind). Fresh fetches enforce the same boundary guarantee: a response missing the just-closed bar (feed latency at the boundary) gets one grace retry after `FRESH_FETCH_RETRY_SECONDS` (3 s), then proceeds with a loud warning. Cache is lock-protected (one provider is shared across the bot's thread pool).
 - **`_drop_forming_candle`**: some feeds include the current partial candle; the scanner fires seconds after boundary, so the last bar is dropped if its close time is in the future. Without this every indicator would read a seconds-old stub and signals would repaint. (No-op on feeds that only return complete candles, e.g. Oanda.)
 - **Freshness** (`is_data_fresh`, public): last bar older than 3× candle duration → stale (warns; live path suppresses signal if the PRIMARY TF is stale). Weekends exempt (Fri 22:00→Sun 22:00 UTC).
 - **`_validate_and_clean`** (public wrapper `validate_and_clean`): drop all-NaN rows → ffill (limit 3) → drop NaN closes → sort + dedupe (keep last) → drop future timestamps (>now+5 min) → repair OHLC inconsistencies (High=max, Low=min of O/C/existing) → remove >10σ return outliers (spike bar only — the opposite-direction snap-back bar right after it is kept, else every spike punched a two-bar hole) → warn if < ~210 bars.
 - **`fetch_range(symbol, tf, start, end)`**: paginated RAW date-range fetch under `MAX_BARS_PER_REQUEST` (5000) — the backtester's `fetch_history` calls this, so ANY registered provider can feed the backtester.
 - `outputsize = max(300, min(days×bars_per_day+20, 5000))`; `CANDLE_SECONDS`/`BARS_PER_DAY` tables live on the base class (the aggregator's bar-close clock reads `BaseDataProvider.CANDLE_SECONDS`).
 - **Streaming capability (optional)**: `supports_streaming` flag + async generator `iter_ticks(symbols, on_connect)` yielding `(canonical_symbol, price)` for one connection's lifetime; the caller owns reconnect/backoff. Errors: each provider raises its own `DATA_ERROR`, always a subclass of `DataIntegrityError` — catch the base type provider-agnostically.
 **A subclass implements only `_fetch_raw(symbol, tf, outputsize, start=None, end=None)`** (normalized frame: capitalized OHLCV, UTC tz-aware index named "Datetime", oldest-first) plus symbol/interval maps and optional capabilities. To add a provider: one module in `providers/`, one `PROVIDER_REGISTRY` entry in `providers/__init__.py`, one config.json `data_providers` block. The live feed is selected by config.json `data_providers.active` (registration alone changes nothing).
 - **`providers/twelve_data.py` `TwelveDataProvider`** (alias `ForexDataProvider`): TD SDK (`TDClient.time_series(...).as_pandas()`), symbols normalized to `"XAU/USD"` form; lowercase→capitalized columns, newest-first→oldest-first; **rate limit** 55 credits/min sliding window (`_throttle`, lock-protected, blocks with sleep). Implements streaming (`ws.twelvedata.com` quotes WS, 10 s heartbeat, `_parse_ws_price`).
 - **`providers/oanda.py` `OandaDataProvider`**: v20 REST MBA candles (complete-only), real bid/ask spread via `get_spread` override; credentials default to the config.json `oanda` block so the registry can build it config-only (explicit `api_token`/`base_url` args still accepted — the daemon passes them). Date-window fetches send `from`/`to` and omit `count` (v20 rejects the combination).
 - **`providers/tiingo.py` `TiingoDataProvider`**: `/tiingo/fx/{ticker}/prices` resampled mid-price candles; no volume (filled 0.0), W1 unmapped (raises). Tiingo takes date windows, not bar counts — live fetches derive `startDate` from the requested count × candle duration × 1.6 weekend pad. Auth in the header (key never in URLs/logs). Every failure shape raises `TiingoDataError` with a named cause (429 quota, 401/403 auth, empty response, malformed candle, missing key). **Candles only** — `get_spread` stays on the base default; the free quote endpoint's indicative spread must not be re-added (see Tiingo spread-monitor removal note).
 - **`data_provider.py` / `oanda_provider.py` are back-compat shims** re-exporting from `providers/`; `ForexDataProvider` is the historical name of the TD provider.
 - **No silent failures**: provider errors all subclass `DataIntegrityError` and carry a named cause; the aggregator logs per-TF fetch failures and degrades the report (missing primary → NEUTRAL/F "data unavailable"); `make_data_provider` raises on an unknown `active` name; keyless providers warn at construction and raise at fetch.
 ---
 ## pair_signals/ — per-pair factories (ALWAYS use these, never bare PlatformConfig for a pair)
 `PAIR_PRODUCERS: Dict[symbol, factory]`; factory signature `make_config(primary_tf, confirmation_tfs=None) -> PlatformConfig`. `produce_signal(symbol, tf, data_provider)` builds config → aggregator → `analyze_pair`. Registered: XAU/USD.
 Inside `make_config`, call order is: `PlatformConfig()` defaults → `app_config.apply_pair_overrides(config)` (stamps config.json's non-credential deployment settings onto the config: base `xau_strategy`/`backtest` `blocked_hours_utc`, `scan_pairs`, `scan_timeframes`) → the validated per-TF deltas below. The overlay runs first so e.g. the M5 hour-9 append lands on top of the JSON base, not the other way around.
 **A bare `PlatformConfig()` silently runs any pair with XAU defaults — always go through the factory** (the bot, CLI, and backtester all do).
 Default confirmation map (same in the factory, the bot, and backtest usage): M5→[M15,H1], M15→[M5,H1], H1→[M15,H4], H4→[H1,D1]. (Exception: `main.py`'s CLI map uses M5→[M1,M15] — a CLI-only quirk.)
 **xauusd.py**: `PlatformConfig()` defaults + per-TF overrides (each validated; full evidence in the module docstring):
 - M5: append hour 9 UTC to blocked hours (6-yr replay: +207.1R vs +178.3R; M15-and-up keep hour 9 — it's positive there). Enable BUY-only chart patterns (`cp_enabled=True`, `cp_patterns=["inv_hs","double_bottom","bull_flag"]`): 12mo +41.8R PF 1.60 (train/val both positive), 5yr +65.2R PF 1.48; bearish CP dropped (OOS −3.21R). M15/H1 CP remain disabled (all failed OOS).
 - M5: `signals.trail_dist_r = 0.85` (wider trail leash, arm stays 0.60; validated 2026-07-21 5-yr grid: +231.2R PF 1.30 vs +218.0R PF 1.29, halves and OOS better, DD flat; dist 1.0 had the best in-sample total but FAILED the OOS gate. XAU_PB keeps pb_trail 0.8/0.75; opposite finding vs M15's later-arm — per-TF divergence again).
 - M15: `signals.tp_floor_rr = 1.0` (1R minimum targets: +116.6R vs +105.4R, better in train/val and 2024 OOS; **hurts M5** — per-TF only) and `xau_strategy.blocked_regimes = ["trending"]` (mixed regime block: +65.7R vs +59.0R). A transitioning-SELL block was tested and REJECTED (bull-market artifact — failed 2024 OOS).
 - M15: `xau_strategy.sell_blocked_hours_utc = [15, 17]` (SELL-only hour block, validated 2026-07-16 on the 5-yr replay: +176.7R PF 1.31 vs +144.9R PF 1.22, better in all 6 years, both 12-mo halves, and 2024-06→2025-03 OOS; M5 cross-TF pocket confirms; BUYs at those hours are +34.3R so `blocked_hours_utc` would be wrong).
 - M15: `signals.trail_arm_r = 0.70` (trail arms later; validated 2026-07-21, 5-yr grid arm 0.3-0.9 × dist 0.5-1.0: +210.7R PF 1.35 WR 57.8% vs +175.7R PF 1.30 WR 53.9% at the 0.6 default; both halves and OOS better; 5yr max-DD clause 33.9R vs 29.1R waived by owner. Earlier-arm cells uniformly worse. M5 keeps 0.6).
 - The factory must stay equivalent to inline `PlatformConfig(symbol=XAUUSD, …)` + overrides — **enforced by a test in test_all.py**.
 ---
 ## telegram_bot.py — the bot
 Run: `python telegram_bot.py` (credentials resolve env `TELEGRAM_TOKEN`/`TELEGRAM_CHAT_ID`/`TELEGRAM_CHANNEL_ID` → config.json `telegram.*` → refuses to boot; see Environment variables).
 - **Owner lock**: every command/button is wrapped in `_owner_only` — only `TELEGRAM_CHAT_ID` gets responses; others get a refusal. The bot is the owner's control panel; the product is the channel.
 - **Subscriber channel** (`TELEGRAM_CHANNEL_ID`, `-100…`; `""` disables): auto-scan signal alerts + breach alerts also post there (bot needs channel-admin). Monetization = Telegram Stars subscription invite links; Telegram handles membership, the bot has zero payment code. The channel feed keeps running even when the owner toggles DM auto-scan off (`_feed_active()` = DM chats OR channel).
 - **Auto-scan**: boot-starts if `TELEGRAM_CHAT_ID` set (initial M15 scan populates the level cache first). Candle-boundary loops for M5/M15/H1/H4, firing at boundary + 10 s. Each scan runs all `scan_pairs` concurrently, updates the level cache, then broadcasts.
 - **Alert gating**: quality threshold (`_alert_min_quality`, default "C", menu cycles C→B→A) + fingerprint dedup per (symbol, TF) on (signal, quality, direction) with 1-hour re-arm — a persisting identical signal doesn't re-send every candle, but a setup whose entry moved ≥ half the last-ALERTED risk distance (new pullback/swing anchor; compared against the stored last-alert entry/risk, a stable reference immune to per-candle ATR drift) re-alerts. The fingerprint is committed only **after** a message reaches ≥1 target (`_commit_alert`), so a failed Telegram send doesn't consume the alert.
 - **WebSocket monitor**: the provider owns the connection/protocol (`BaseDataProvider.iter_ticks` streaming capability — TD implements it; heartbeat + message parsing live in `providers/twelve_data.py`); the bot owns exponential reconnect (≤60 s) + breach dispatch, and disables the monitor with a log line if the active provider has `supports_streaming=False`. On tick crossing any cached key level (`swing_high/low_1..3`, `ema_trend`, `bb_upper/lower`, `kc_upper/lower`; cache TTL 1 h) → immediate M15 rescan → "LEVEL BREAK" alert if actionable+changed (same gates as auto-scan), 5-min per-pair cooldown.
 - **Commands**: `/start` (menu), `/scan` (buttons: per pair — M5/M15/H1/H4 + Scan All TFs), `/autoscan` (toggle DM copy), `/status` (loops, WS, live prices, level cache, cooldowns), `/cache` (age per pair/TF), `/clearcache`, `/help`.
 - **Tiingo spread monitor: REMOVED 2026-07-23** (shipped 2026-07-22, removed next day). Tiingo's free `/tiingo/fx/top` publishes a two-tier *indicative* spread for gold — exactly $0.10 in liquid hours and exactly $0.70 in the 21:00–00:00 UTC rollover window / news moments (verified: 273-sample production log + live probes; only those two values ever occur). Quotes are mid ± a posted increment, not executable depth, so the feature could never validate the $0.30 backtest cost assumption. Do not re-add a spread display from Tiingo's free FX feed.
 - Signal messages (redesigned 2026-07-24, `format_signal_message`): verdict header first, then trade setup, then a condensed one-line indicator summary (`EMA↑ RSI→ …`, `_indicator_summary`) + warnings. The setup block includes an **informational 100-pip TP line** (`_INFO_TP_PIPS`, display-only — entry ± 100 pips, ≈R vs stop; NOT part of the backtested exit model) and the **trailing-stop management plan** (arm price + trail distance, read from the `trail_arm_r`/`trail_dist_r` stamped on the setup by the producing per-TF config). Risk disclaimer on every outbound message.
 - Blocking scan work runs in `ThreadPoolExecutor(max_workers=4)`; one shared `ForexDataProvider` across all scans (cache reuse).
 - **Logging (hardened 2026-07-22)**: root-logger handlers (`force=True`) → console + `logs/telegram.log` (3-hour rotation, 240 backups = 30 days), full-date timestamps; every module logger (data_provider, news_engine, websockets, telegram.ext) lands in the same file. Failure surfaces all covered: candle loops catch-and-continue per cycle (one bad scan can't kill a loop), breach tasks log exceptions via done-callback, PTB `add_error_handler` registered, and `sys.excepthook`/`threading.excepthook` (installed in `main()`) route uncaught crashes to the log. WS monitor keeps its exponential-backoff reconnect + `logger.exception`.
 ---
 ## main.py — CLI
 `python main.py [--pair XAUUSD] [--timeframe M5|M15|M30|H1|H4|D1] [--detail] [--json] [--verbose]`. Uses the pair factory, renders a compact terminal report (decision, setup, context, top-3 reasons per side, warnings) or full JSON via `report_to_dict`.
 ---
 ## backtest/gpu_backtest.py — the backtester
 Replays history bar-by-bar through the **real `SignalAggregator.analyze_from_dataframes`** — same gates, same setup math. News is disabled (`use_news=False`): backtests are pure technical.
 **Realism model (all validated fixes — do not regress):**
 - Entry at the **next bar's open** after the signal bar; SL/TP distances preserved relative to actual entry. Entries are **re-anchored to the raw price path** (`_reanchor_entries_to_raw`): if cleaning dropped a >10σ spike bar right after the signal bar, the trade enters at the spike bar's open, not the bar after it — skipping it was a favorable bias.
 - Round-trip `--spread` subtracted from every trade's PnL before computing R (default = pair's `default_spread`: XAU $0.30). Results before 2026-07-08 (+59.5R baseline era) charged no spread — not comparable.
 - Trade resolution runs against the **RAW price path** (`fetch_history(clean=False)`): the parquet cache stores raw bars; cleaning (which drops >10σ news spikes that hit stops in reality) is applied only to the signal-side data. Caches written before this fix contain cleaned data — refetch to restore spike bars.
 - Same-bar SL+TP tie → SL first (conservative). Gap through SL/TP → filled at the bar open.
 - **Trailing exits** (default on, from `signals.trail_arm_r/trail_dist_r`; XAU_PB trades carry per-trade 0.8/0.75 overrides): stop ratchets at bar close only, so a new stop can only fill from the next bar (conservative). Exit reasons: SL / TRAIL / TP1 (static mode) / TIMEOUT / NO_DATA.
 - Session + weekend + weekday filters evaluate the **decision time = bar close** (`ts + primary_duration`) — exactly matching the live bot's wall-clock check.
 - The optional H4-ADX regime filter (`config.backtest.regime_filter`, default off) is applied in the replay by calling the live `_apply_regime_filter_live` method itself under the same guard, and the replay loads/slices the regime-filter TF whenever the gate is on (like live `analyze_pair`) — previously M5/M15 primaries silently no-opped the filter because H4 wasn't in their confirm maps. Live and backtest cannot diverge if it's enabled (2026-07-21 parity fix; the unregistered-pair config fallback also applies the config.json overlay now).
 - Confirmation TFs are sliced to bars **fully closed** by decision time (no look-ahead).
 **Performance:** `precompute_indicators` builds every indicator series once per TF (CuPy on GPU or NumPy — check the log line `Backend: CuPy GPU` vs `NumPy CPU fallback`; CuPy needs the `nvidia-*-cu12` pip wheels or it silently falls back). `GPUTechnicalEngine` subclasses the live engine and overrides only the `_compute_*` methods with cache lookups — outputs are byte-identical to the live engine (regression-tested). Bar replay: pure per-bar evaluation fans out over forked worker processes (`--workers`, 0 = auto: cores−2 capped 12; <5000 bars forces 1); the entry **cooldown (5 bars) is the only sequential rule**, applied in an ordered pass afterwards — parallel and sequential books are identical. ~300 bars/s/core, ~1700 bars/s at 10 workers.
 **Strategy tags** (from reasons[0] markers): `XAU_PB`, `XAU_CP`, `GENERIC` (plus legacy `XAU` recognized). `--strategies` filters the book by tag. `--cp` force-enables the chart-pattern entry for candidate validation; `--cp-patterns` narrows the detector set.
 **CLI:**
 ```bash
 python backtest/gpu_backtest.py --pair XAUUSD --days 365
 python backtest/gpu_backtest.py --pair XAUUSD --start 2025-01-01 --end 2025-06-01 --primary-tf 15min --confirm-tfs 1h,4h
 # key flags: --spread --quality A+,A,B,C --strategies --workers --max-hold-bars 300
 #            --warmup-bars 250 --cooldown-bars 5 --no-cache --export-csv out.csv --detail
 ```
 Report: summary metrics (total R, PF, Sharpe on R, max DD, MFE/MAE), breakdowns by quality/strategy/regime, trade log, and per-trade reasoning dumps (indicator-by-indicator agreement at entry — required by `instructions_for_backtest.txt`). `--export-csv` flattens every indicator's signal/value/agreement into columns.
 **Comparability note (2026-07-16):** the parity/realism fixes of that date (EMA-ribbon port to the GPU engine, exact-pandas Bollinger precompute, raw-path entry re-anchoring, per-TF D1 warmup, live confirm-map CLI default, outlier snap-back retention) all shift books slightly vs earlier runs. Books produced before these fixes are not directly comparable — re-run the baseline before any before/after tuning comparison.
 Data: paginated fetches via `provider.fetch_range` (5000-bar chunks; works with any registered provider) cached as parquet in `backtest/cache/` keyed by **provider**/symbol/TF/window (bars differ between feeds — Oanda/TD divergence study; `twelve_data` keeps the legacy un-prefixed filename so pinned-book caches stay valid); warmup pad ≥60 days before the window for EMA200, extended per-TF so slow confirmation TFs get ≥210 bars at window start (D1 gets ~299 days — 62 days gave only ~61 bars: no EMA200, unlike live's 300-bar outputsize floor). `--confirm-tfs` defaults to the pair factory's live confirmation map (pass it explicitly to override); a hardcoded default here previously made the standard invocation backtest a different MTF setup than the bot scans.
 ---
 ## test_all.py — QA suite
 `python test_all.py` — **253 tests, 16 modules** (config, data_provider, engine, news_engine, signal_aggregator, main, telegram_bot, integration, pair_signals registry + engine-bug regressions, 2026-07-16 audit-fix regressions, chart patterns, app_config, .env loader + bot resilience, Oanda provider/executor/daemon, provider abstraction). No network needed (mocked). Uses a custom `@test("name")` decorator that runs immediately (NOT pytest — don't grep for `def test_`). Exit code non-zero on failure. Includes: the xauusd-factory ≡ inline-config equivalence test, engine bug-fix regression tests (Module 9b), a live-vs-GPU engine parity sweep incl. the EMA-ribbon path (Module 11), config.json loading/precedence/validation coverage (Module 13), and `.env` loader + bot-resilience coverage (Module 14). **Any signal-logic change must keep this green and should add tests.**
 ---
 ## Environment variables
 Precedence for every credential/deployment setting: **env var → config.json → none**. A repo-root `.env` file (gitignored) is loaded at `app_config` import via `os.environ.setdefault` — it behaves like real env vars except a real shell export always wins. Hardcoded fallbacks (previously in `config.py` / `telegram_bot.py`) were **REMOVED 2026-07-21**; the bot now refuses to boot (`telegram_bot.main`) and data fetches fail with a named error (`data_provider`) if a secret resolves to neither source. See `app_config.py` and `config.json.example`.
 | Variable | Purpose | Default |
 |---|---|---|
 | `TWELVE_DATA_API_KEY` | market data key (twelve_data only) | none — falls through to config.json `data_providers.twelve_data.api_key`, then fails fast |
 | `TIINGO_API_KEY` | market data key (tiingo only) | none — falls through to config.json `data_providers.tiingo.api_key`, then fails fast |
 | `TELEGRAM_TOKEN` | bot token | none — falls through to config.json `telegram.token`, then refuses to boot |
 | `TELEGRAM_CHAT_ID` | owner chat id (boot auto-scan + owner lock) | none — falls through to config.json `telegram.chat_id`, then refuses to boot |
 | `TELEGRAM_CHANNEL_ID` | subscriber channel (`-100…`); `""` disables the feed | none — falls through to config.json `telegram.channel_id` |
 | `FB2_EMA_LO` | EMA200 "near" penalty threshold in % | `0.5` |
 Dependencies (`requirements.txt`): numpy, pandas, requests, twelvedata, python-telegram-bot ≥20, websockets. Python 3.12. A `venv/` exists in-repo.
 ---
 ## Code conventions
 - Type hints on public methods; dataclasses for all config and outputs.
 - `SignalStrength` (−2..+2) is the universal signal currency.
 - Key levels dict keys: `swing_high_*`, `swing_low_*`, `ema_*`, `bb_*`, `kc_*`, `atr`, `atr_pct`.
 - Private per-indicator series ride on `TechnicalAnalysis` as `_underscore` attributes; anything `_`-prefixed or weight-0 is skipped by scorers/formatters.
 - UTC everywhere; DataFrames indexed by tz-aware `Datetime`, oldest-first, capitalized OHLCV columns.
 - Logging via module loggers; the bot adds a rotating file handler.
 ---
 ## Rules for making changes (READ BEFORE EDITING SIGNAL LOGIC)
 1. **Never change a tuned value without re-validation.** The acceptance rule used throughout: positive total R in BOTH the train and validation halves of a ≥12-month window, PF ≥ 1.3, at realistic cost (XAU $0.30 round trip), with trailing exits and next-open fills. For any marginal edge additionally re-test on the 2024-06→2025-03 out-of-sample window — within-year splits alone have approved configs that failed OOS.
 2. **Per-timeframe overrides live in the pair factory** (`pair_signals/*.py`), never in global defaults — the same knob has opposite effects on different TFs (proven twice: `tp_floor_rr`, hour-9 block). Document the validation evidence in the factory docstring.
 3. **Pair isolation is a hard invariant.** XAU gates in the aggregator are display-gated (`report.display_name == "XAU/USD"`) so a future pair port cannot inherit gold-calibrated logic by accident.
 4. **Live == backtest parity**: session/weekend/hour filters must evaluate bar-CLOSE time in both paths; `xauusd.make_config` must stay equivalent to the inline config the backtester/bot would build (test-enforced). If you add a live gate, add it to the replay path (or `analyze_from_dataframes`) in the same commit.
 5. **ROC stays weighted** (see engine table) unless you retune composite thresholds and re-validate — removing it lost 13R.
 6. **Backtester realism knobs are results-critical**: raw-path resolution, spread charging, next-open entry, SL-first ties, close-only trail ratchet. Any "optimization" that touches these invalidates comparability with the pinned books (`backtest/*.csv`, memory notes).
 7. **Don't zero/disable indicators via config in production paths** — `disabled_indicators` exists for experiments; composite thresholds assume the full set.
 8. **The forming candle must stay dropped** and analysis must only ever see closed bars, or live signals repaint.
 9. Keep `test_all.py` green (`python test_all.py`); extend it for every new gate/knob.
 10. Dead/legacy config fields were removed 2026-07-21 (`MarketSession`, `is_jpy_pair`, `candle_buffer_seconds`, the unused `SignalConfig`/`XauStrategyConfig` legacy knobs). Do not add config fields that nothing reads; delete dead fields only with test updates.
@@ -0,0 +1,85 @@
 # Full Migration to Oanda v20 as Market-Data Source — Roadmap
 **Status: reference roadmap, not scheduled for implementation.** Owner decision 2026-07-24: document only; when executed, Twelve Data stays as a **permanent fallback** (no removal phase).
 ## Context
 The `oanda_demo` branch already executes trades on an Oanda practice account, but the signal feed (Telegram bot, CLI, backtester) still runs on Twelve Data. Oanda's API is free with no metering (120 req/s vs TD's 55 credits/min), serves real bid/ask spreads, and matches the actual trading calendar. The 2026-07-24 divergence study (`backtest/research/oanda_td_divergence.py`) proved the switch is **not plumbing-only**: steady-state bars differ by median $0.69 (11% of M15 ATR), bar direction agrees only 89.9%, TD prints ~4,000 phantom Saturday bars/yr + 21:00 UTC rollover bars that contaminate Monday indicator windows, and Oanda's real spread (~$0.72 liquid / $0.84 rollover) is ~2.4× the $0.30 the validated books assume. So the migration = plumbing (Phases 1–3, inert) + shadow evidence (4) + book re-validation with owner gates (5) + reversible cutover (6). CLAUDE.md rules 1 (no tuned-value change without re-validation) and 4 (live == backtest parity) govern throughout.
 **Master sequencing rule:** Phases 1–3 land inert behind `data_providers.active == "twelve_data"`. Nothing changes live behavior until the Phase 6 config flip, which stays reversible forever (TD is a permanent fallback).
 ## Phase 1 — Provider plumbing (inert)
 Files: `oanda_provider.py`, `data_provider.py`, `config.py`, `config.json.example`.
 1. **Constructor self-resolution**: `OandaDataProvider.__init__(config, api_token=None, base_url=None)` — defaults resolve via existing `app_config.oanda_api_token()` / `oanda_base_url()`; explicit args keep `oanda_daemon.py:180` working. Fail lazily at first fetch (mirror ForexDataProvider's `_td is None` pattern) so credential-less construction doesn't break the hermetic test header.
 2. **Registry**: add `"oanda": OandaDataProvider` to `PROVIDER_REGISTRY` (`data_provider.py:392`); import at bottom of `data_provider.py` (one-direction edge, no cycle — `oanda_provider` must never import from `data_provider`). Add `PROVIDER_NAME` class attr to both providers. Cutover config needs an `"oanda": {}` block under `data_providers` (creds stay in the existing top-level `oanda` section / env).
 3. **Dedupe shared constants**: move `CANDLE_SECONDS` / `BARS_PER_DAY` to `config.py`, alias as class attrs in both providers — all existing call sites (`signal_aggregator.py:18,217,640,942`, `telegram_bot.py:1062`, `gpu_backtest.py:975,1706`, tests) keep working unchanged. Also move `_is_forex_weekend` to `config.py` eventually (currently duplicated + sync-tested).
 4. **`_fetch_window(symbol, tf, start, end)`** on both providers — the backtester's common interface. Contract: raw closed bars labeled in [start, end], oldest-first, capitalized OHLCV, UTC index; empty DataFrame (not exception) for empty windows; caller keeps windows ≤ ~4500 bars. TD: thin wrapper over existing `_fetch_twelve_data(start_date=, end_date=)` (keep its exception behavior byte-identical). Oanda: new `from`/`to` RFC3339 request (`price=MBA, dailyAlignment=0, alignmentTimezone=Etc/UTC`); refactor `_fetch_oanda_candles`'s parsing body (complete-filter, mid extraction, spread cache) into a shared `_parse_candles()`. The from+count quirk (Oanda's `count` includes the boundary bar; see `backtest/research/fetch_oanda_m15.py`) does not apply to from/to windows. Add a no-op-by-construction `_drop_forming_candle` to OandaDataProvider (`gpu_backtest.py:1025` calls it unconditionally). No throttle needed for Oanda.
 5. **Tests (extend Module 15)**: registry selection; constructor self-resolution + explicit-args-win; `_fetch_window` both providers (mock `_session.get` / `_fetch_twelve_data`); CANDLE_SECONDS/BARS_PER_DAY cross-provider sync; provider-contract hasattr test covering everything callers reach for (`fetch_ohlcv, get_pair_info, clear_cache, _is_data_fresh, _has_latest_closed_bar, _validate_and_clean, _drop_forming_candle, _fetch_window, CANDLE_SECONDS, BARS_PER_DAY, _cache, _cache_timestamps, PROVIDER_NAME`); port Module 11's two cache/boundary tests (`test_all.py:2278,2313`) to Oanda variants patching `_fetch_oanda_candles`. Tests that swap `set_app_config` must restore `_TEST_APP` in `finally`.
 ## Phase 2 — Backtester provider-agnosticism
 File: `backtest/gpu_backtest.py` (`fetch_history`, lines 943–1036).
 1. Line 984: `provider._fetch_twelve_data(...)` → `provider._fetch_window(...)`. Keep the backward-stepping chunk loop, `empty_streak` bail, cursor stepping byte-identical (rule 6 — this loop produced every pinned cache).
 2. Lines 975/1706: `ForexDataProvider.BARS_PER_DAY` → `provider.BARS_PER_DAY`.
 3. **Provider-namespaced cache keys** in `_cache_path` (line 938): TD keeps legacy names (pinned parquets stay loadable, needed for A/B), Oanda gets `oanda_` prefix (`oanda_XAU_USD_15min_....parquet`) — no collisions. Comment the asymmetry.
 4. `clean=True/False` raw-path semantics untouched.
 5. **Verification before merge**: one M15 12-month TD replay pre/post commit — trade books must be identical. **Alignment audit**: fetch one month of Oanda H4 + D1 via `_fetch_window`, assert label sets match TD's cached labels (H4 at 00/04/08 UTC, D1 at 00:00 UTC) — a mismatch would silently shift every MTF confirmation.
 6. Tests: `fetch_history` dispatch with a stub provider (concat/dedupe/trim, clean=False skips cleaning); `_cache_path` namespacing regression.
 ## Phase 3 — Telegram bot: TD WebSocket → Oanda pricing stream
 File: `telegram_bot.py`; new resolver in `app_config.py`.
 1. New `oanda_stream_url()` resolver: env `OANDA_STREAM_URL` → config.json `oanda.stream_url` (add to `_KNOWN_OANDA`) → derived `oanda_base_url().replace("api-", "stream-", 1)`.
 2. Pure parser `_parse_stream_line(line) -> Optional[(pair_symbol, mid_price)]`: type==PRICE → mid from closeoutBid/Ask (fallback bids[0]/asks[0]); HEARTBEAT/garbage → None. `_STREAM_TO_PAIR` from `OandaDataProvider.OANDA_INSTRUMENT` (replaces TD-display-keyed `_WS_TO_PAIR`).
 3. Reader thread (no new deps): `requests.get(.../v3/accounts/{id}/pricing/stream?instruments=..., stream=True, timeout=(10,30))`, iterate `iter_lines(chunk_size=1)` — **chunk_size=1 is load-bearing** (urllib3 buffering vs 5s server heartbeats); push lines to asyncio queue via `loop.call_soon_threadsafe`; interrupt a blocked read with `resp.close()` from the async side.
 4. Async monitor `_oanda_stream_monitor`: dedicated thread (NOT the 4-worker scan executor), `await asyncio.wait_for(queue.get(), timeout=15)` as staleness watchdog (3× server heartbeat) → reconnect with existing exponential-backoff pattern. PRICE lines feed the identical existing body: `_check_level_breach` → `_handle_breach` → M15 rescan, 5-min cooldown. No client heartbeat.
 5. **Provider-gated selection** in `_start_scheduler` (line 889): `active_provider == "oanda"` → stream monitor, else existing `_ws_monitor`. This is the rollback hinge and stays permanently (TD fallback). `_stop_scheduler` must stop the reader thread. Generalize `/status` labels (`_status_text` iterates `SCAN_PAIRS`, not `_WS_TO_PAIR`).
 6. Tests: `_parse_stream_line` variants (PRICE/closeout, PRICE/arrays, HEARTBEAT, malformed, unknown instrument); `_STREAM_TO_PAIR` round-trip; monitor selection under both `active` values; watchdog-reconnect (structure one connection lifecycle as a testable inner coroutine).
 ## Phase 4 — Shadow / parity evidence (3–4 weeks, no hot-path code)
 The daemon already runs the full signal stack on Oanda bars. Formalize evidence before any cutover:
 1. **Code parity (exact)**: replay the daemon's shadow window through the backtester on Oanda data (Phase 2 enables this); diff against daemon-logged decisions per (tf, bar_close_ts) — signal, quality, direction, entry/SL/TP. Acceptance: 100% modulo documented daemon divergences (single-position rule, quality gate). Any mismatch is a bug, not feed noise.
 2. **Feed divergence (measured, not zero-gated)**: `backtest/research/shadow_compare.py` parses `logs/telegram.log` + daemon log into per-scan rows: signal agreement rate, direction flips, grade drift, clustering by hour/weekday. Expect high-single-digit % divergence concentrated near grade thresholds and Monday sessions (TD Saturday/rollover bars). Deliverable: written note attributing every divergence class.
 3. **Operational telemetry**: boundary grace-retry frequency (is Oanda's just-closed bar present at boundary+10s?), `get_spread()` time series (live input to the Phase 5 cost decision), stream disconnect counts, Monday-open indicator snapshots TD-vs-Oanda.
 4. **Owner note**: at cutover the data feed host family should match the execution host (practice vs live pricing can micro-differ).
 ## Phase 5 — Re-validation campaign (rule 1, owner gates)
 Fetch Oanda history (M5/M15/H1/H4/D1, 2020-07 → present; Oanda XAU reaches ~2005). Run matrix per primary TF (M5/M15/H1, current factory configs): 12-mo train/val halves, 5-yr, OOS 2024-06→2025-03 — each at `--spread 0.30` (pure feed-effect A/B vs pinned TD books), `0.70` (honest Oanda cost), `0.85` (rollover stress). Caution: Jul–Nov 2025 is provider-sensitive (weekly offsets −$6..+$18.5) — lean on 5-yr + OOS for decisions. Acceptance unchanged: positive both halves, PF ≥ 1.3, trailing exits, next-open fills, OOS positive.
 **Owner decision gates (never silent):**
 1. **Canonical cost** — (a) `round_trip_cost=0.70` everywhere (moves the live TP1 spread gate $9→$21 — a strategy re-tune); (b) `default_spread=0.70` for backtest PnL but retune `min_tp_cost_multiple` 30→~13 to preserve the validated $9 gate; (c) keep $0.30 with documented optimism (only if $0.70 books pass anyway). Full acceptance matrix at chosen values.
 2. **Degradation handling** per failing TF: re-run tuning grids on Oanda data / drop TF from `scanning.timeframes` / keep Oanda execution-only (status quo).
 3. **Re-pin**: passing books become the new pinned CSVs + factory-docstring evidence; CLAUDE.md tuned-value annotations update in the cutover commit, not before.
 Files (per decisions): `pair_signals/xauusd.py`, `config.py` (`PAIR_INFO.default_spread`, `SignalConfig.round_trip_cost`/`min_tp_cost_multiple`), `test_all.py` tuned-constant assertions, CLAUDE.md.
 ## Phase 6 — Cutover (reversible forever; no TD removal)
 1. config.json: `data_providers.active: "oanda"` + `"oanda": {}` block; restart bot. Everything flips through two seams: `make_data_provider` (bot shared dp, CLI, backtester) and the Phase 3 monitor selector. Any Phase 5 tuned-value changes land in this commit with their validation records.
 2. **Bake ≥1 week** (full weekly cycle incl. Monday open): grace-retry rate, stream stability, alert plausibility vs daemon, `/cache` freshness.
 3. **Rollback = flip `active` back** — TD code, key, caches, and WS monitor all stay maintained indefinitely (owner decision). Dual-provider posture: cross-provider sync tests (Phase 1) keep both honest; TD subscription/key must stay valid to remain a real fallback.
 4. Docs: CLAUDE.md (pipeline header, data_provider section, env-var table + `OANDA_STREAM_URL`, backtester cache note, comparability note: "pre-Oanda books are TD-feed books"), `docs/` guides if they name the data source.
 ## Key risks
 1. **Cost-gate coupling**: naively setting `round_trip_cost=0.70` silently moves the live $9 TP1 gate to $21 — owner gate 5.1 exists precisely for this.
 2. **Cutover before re-pinning** — feed divergence is material (11% of ATR, 10% direction flips); sequencing rule prevents it.
 3. **Hermetic test header** — `_TEST_APP` pins twelve_data; provider-swapping tests must restore it or later modules fail mysteriously.
 4. **Stream liveness triad** — `chunk_size=1`, `resp.close()` interruption, 15s watchdog.
 5. **H4/D1 alignment** — audit before trusting any Oanda backtest.
 6. **Monday/rollover divergence class** — attribute in Phase 4 evidence; do not "fix".
 ## Verification (per phase)
 - Phases 1–3: `python test_all.py` green (235 + new); live behavior provably unchanged (TD active).
 - Phase 2: pre/post TD book identity diff; Oanda H4/D1 alignment audit.
 - Phase 3: soak a second bot instance with `active=oanda` scratch config; observe stream reconnects in log.
 - Phase 5: full acceptance matrix, owner sign-offs recorded in factory docstrings.
 - Phase 6: 1-week bake checklist above; rollback drill (flip back, confirm TD scan cycle) once during bake.
@@ -85,15 +85,11 @@ _VALID_LOG_LEVELS = ("DEBUG", "INFO", "WARNING", "ERROR", "CRITICAL")
 # AUTOSCAN_CONFIRMATIONS entries in telegram_bot.py).
 _SCANNABLE_TFS = ("M5", "M15", "H1", "H4")
-_KNOWN_TOP = {"telegram", "data_providers", "scanning", "pairs", "log_level",
+_KNOWN_TOP = {"telegram", "data_providers", "scanning", "pairs", "log_level", "oanda"}
              "oanda", "deepseek"}
 _KNOWN_OANDA = {
    "account_id", "api_token", "base_url", "lot_size",
    "quality_gate", "max_hold_bars",
 }
 _KNOWN_DEEPSEEK = {
    "api_key", "model", "enabled", "timeout_seconds", "daily_request_cap",
 }
 _KNOWN_TELEGRAM = {"token", "chat_id", "channel_id", "alert_min_quality"}
 _KNOWN_SCANNING = {"pairs", "timeframes"}
 _KNOWN_PAIR_KEYS = {"blocked_hours_utc"}
@@ -110,14 +106,12 @@ class AppConfig:
    telegram_channel_id: str = ""          # "" = channel feed disabled
    alert_min_quality: str = "C"
    active_provider: str = "twelve_data"
    streaming_provider: Optional[str] = None   # None = active provider
    provider_settings: Dict[str, Dict[str, Any]] = field(default_factory=dict)
    scan_pairs: List[str] = field(default_factory=lambda: [XAUUSD])
    scan_timeframes: List[Timeframe] = field(default_factory=lambda: [
        Timeframe.M5, Timeframe.M15, Timeframe.H1, Timeframe.H4])
    pair_overrides: Dict[str, Dict[str, Any]] = field(default_factory=dict)
    oanda_settings: Dict[str, Any] = field(default_factory=dict)
    deepseek_settings: Dict[str, Any] = field(default_factory=dict)
    log_level: str = "INFO"
@@ -168,8 +162,7 @@ def load_app_config(path: str = CONFIG_PATH) -> AppConfig:
        app.active_provider = str(dp.get("active", app.active_provider))
        app.provider_settings = {}
        for name, block in dp.items():
-            # "active" and "streaming" are reserved scalars, not blocks.
+            if name == "active":
            if name in ("active", "streaming"):
                continue
            if not isinstance(block, dict):
                raise AppConfigError(
@@ -179,17 +172,6 @@ def load_app_config(path: str = CONFIG_PATH) -> AppConfig:
            raise AppConfigError(
                f'config.json: data_providers.active "{app.active_provider}" '
                "has no matching provider block")
        streaming = dp.get("streaming")
        if streaming is not None:
            if not isinstance(streaming, str):
                raise AppConfigError(
                    'config.json: data_providers.streaming must be a string '
                    "(a provider name)")
            if streaming not in app.provider_settings:
                raise AppConfigError(
                    f'config.json: data_providers.streaming "{streaming}" '
                    "has no matching provider block")
            app.streaming_provider = streaming
    scanning = raw.get("scanning", {})
    _check_keys(scanning, _KNOWN_SCANNING, '"scanning"')
@@ -253,29 +235,6 @@ def load_app_config(path: str = CONFIG_PATH) -> AppConfig:
                "config.json: oanda.max_hold_bars must be a positive integer")
    app.oanda_settings = dict(oanda)
    deepseek = raw.get("deepseek", {})
    _check_keys(deepseek, _KNOWN_DEEPSEEK, '"deepseek"')
    if "enabled" in deepseek and not isinstance(deepseek["enabled"], bool):
        raise AppConfigError(
            "config.json: deepseek.enabled must be true or false")
    if "timeout_seconds" in deepseek:
        ts = deepseek["timeout_seconds"]
        if isinstance(ts, bool) or not isinstance(ts, (int, float)) or ts <= 0:
            raise AppConfigError(
                "config.json: deepseek.timeout_seconds must be a positive "
                "number")
    if "daily_request_cap" in deepseek:
        cap = deepseek["daily_request_cap"]
        if isinstance(cap, bool) or not isinstance(cap, int) or cap < 1:
            raise AppConfigError(
                "config.json: deepseek.daily_request_cap must be a positive "
                "integer")
    if "model" in deepseek:
        if not isinstance(deepseek["model"], str) or not deepseek["model"]:
            raise AppConfigError(
                "config.json: deepseek.model must be a non-empty string")
    app.deepseek_settings = dict(deepseek)
    level = raw.get("log_level", app.log_level)
    if level not in _VALID_LOG_LEVELS:
        raise AppConfigError(
@@ -367,14 +326,6 @@ def oanda_base_url(app: Optional[AppConfig] = None) -> str:
        app.oanda_settings.get("base_url", "https://api-fxpractice.oanda.com"))
 # ── DeepSeek resolution helper (env → config.json → "") ──────────────
 def deepseek_api_key(app: Optional[AppConfig] = None) -> str:
    app = app or get_app_config()
    return os.environ.get("DEEPSEEK_API_KEY") or str(
        app.deepseek_settings.get("api_key", ""))
 # ── PlatformConfig overlay ────────────────────────────────────────────
 def apply_pair_overrides(
@@ -79,54 +79,34 @@ def classify_bar(
    """
    filters_hit: List[str] = []
-    # Known filter tags, matched against the exact warning strings the
+    # Scan warnings for known filter tags
-    # aggregator emits (refreshed 2026-07-30 for gates 6d-6i). ORDER
+    filter_tags = {
-    # MATTERS: most-specific prefix first, and each warning maps to at
+        "H4 RSI GATE":       "H4_RSI_GATE",
-    # most ONE tag (break on first hit) — substring families like
+        "H1 MACD GATE":      "H1_MACD_GATE",
-    # "M5 FILTER: SELL quality" vs "M5 FILTER" and "TREND EXHAUSTION"
+        "RSI ZONE GATE":     "RSI_ZONE_GATE",
-    # vs "EXHAUSTION" would otherwise double-count.
+        "XAU BUY FILTER":    "XAU_BUY_FILTER",
-    filter_tags = [
+        "SESSION FILTER":    "SESSION_FILTER",
-        # M5 noise gates (6i, _apply_m5_filters)
+        "DAY FILTER":        "DAY_FILTER",
-        ("M5 FILTER: |score|",            "M5_SCORE"),
+        "LOW ATR FILTER":    "LOW_ATR_FILTER",
-        ("M5 FILTER: no MTF agreement",   "M5_MTF"),
+        "SELL SUPPRESSED":   "SELL_SUPPRESSED",
-        ("M5 FILTER: no confirmation TF", "M5_CONF"),
+        "EXHAUSTION":        "EXHAUSTION_PENALTY",
-        ("M5 FILTER: SELL quality",       "M5_SELL_QUALITY"),
+        "TREND EXHAUSTION":  "TREND_EXHAUSTION",
-        ("M5 FILTER: SELL blocked — RSI", "M5_SELL_RSI"),
+        "NO PULLBACK":       "NO_PULLBACK_PENALTY",
-        ("M5 FILTER: '",                  "M5_VOL"),
+        "Weak pullback":     "WEAK_PULLBACK_PENALTY",
-        ("M5 FILTER",                     "M5_OTHER"),
+        "Extended entry":    "EXTENDED_ENTRY_PENALTY",
-        # XAU SELL gates (6e/6g/6g2/6g3)
+        "Counter-trend":     "COUNTER_TREND_PENALTY",
-        ("SELL FILTER: quality",          "SELL_QUALITY"),
+        "EXTREME volatility":"EXTREME_VOL_PENALTY",
-        ("SELL FILTER: volatility",       "SELL_VOL"),
+        "NEAR EMA200":       "NEAR_EMA200_PENALTY",
-        ("SELL FILTER: regime",           "SELL_REGIME"),
+        "FAR FROM EMA200":   "FAR_EMA200_PENALTY",
-        ("UTC entry blocked for shorts",  "SELL_HOUR"),
+        "EVENT RISK":        "EVENT_RISK",
-        # Hard gates
+        "REGIME FILTER":     "REGIME_FILTER",
-        ("MTF OPPOSITION",                "MTF_OPPOSITION"),
+        "Timeframes CONFLICT": "MTF_CONFLICT",
-        ("EXHAUSTED TREND FILTER",        "EXHAUSTED_TREND"),
+    }
        ("BIG CANDLE FILTER",             "BIG_CANDLE"),
        ("WEEKEND FILTER",                "WEEKEND_FILTER"),
        ("SESSION FILTER",                "SESSION_FILTER"),
        ("DAY FILTER",                    "DAY_FILTER"),
        ("LOW ATR FILTER",                "LOW_ATR_FILTER"),
        ("REGIME FILTER",                 "REGIME_FILTER"),
        # Soft penalties / context warnings
        ("TREND EXHAUSTION",              "TREND_EXHAUSTION"),
        ("EXHAUSTION",                    "EXHAUSTION_PENALTY"),
        ("NO PULLBACK",                   "NO_PULLBACK_PENALTY"),
        ("Weak pullback",                 "WEAK_PULLBACK_PENALTY"),
        ("Extended — RSI",                "RSI_EXTENDED_PENALTY"),
        ("Counter-trend",                 "COUNTER_TREND_PENALTY"),
        ("EXTREME volatility",            "EXTREME_VOL_PENALTY"),
        ("NEAR EMA200",                   "NEAR_EMA200_PENALTY"),
        ("FAR FROM EMA200",               "FAR_EMA200_PENALTY"),
        ("EVENT RISK",                    "EVENT_RISK"),
        ("Timeframes CONFLICT",           "MTF_CONFLICT"),
    ]
    for w in report.warnings:
-        for tag, name in filter_tags:
+        for tag, name in filter_tags.items():
            if tag in w:
                filters_hit.append(name)
                break
    # Determine outcome
    non_neutral = report.final_signal not in (SignalStrength.NEUTRAL,)
@@ -30,7 +30,6 @@ Usage:
 from __future__ import annotations
 import argparse
 import json
 import logging
 import multiprocessing as mp
 import os
@@ -40,13 +39,10 @@ from dataclasses import dataclass, field
 from datetime import datetime, timedelta, timezone
 from pathlib import Path
 from typing import Dict, List, Optional, Tuple
 from zoneinfo import ZoneInfo
 import numpy as np
 import pandas as pd
 _NY_TZ = ZoneInfo("America/New_York")
 # Ensure project root on sys.path so this script can be invoked from anywhere.
 PROJECT_ROOT = Path(__file__).resolve().parent.parent
 if str(PROJECT_ROOT) not in sys.path:
@@ -405,11 +401,6 @@ class GPUTechnicalEngine(TechnicalEngine):
        # cache key = timeframe.value (e.g. "15min", "1h", "4h")
        self.caches = caches
        self._current_tf: Optional[str] = None
        # Cache-miss fallbacks recompute with the live engine's pandas
        # math (different EWM seeding than the precomputed cache). That
        # is a parity hazard, so it must never happen silently.
        self.fallback_count = 0
        self._fallback_warned = False
    def analyze(self, df: pd.DataFrame, symbol: str, timeframe: str) -> TechnicalAnalysis:
        # Latch the timeframe so per-indicator overrides know which cache to use.
@@ -427,23 +418,9 @@ class GPUTechnicalEngine(TechnicalEngine):
            return None, -1
        cache = self.caches.get(self._current_tf)
        if cache is None:
            self._note_fallback(f"no cache for TF {self._current_tf!r}")
            return None, -1
        pos = cache.index_pos.get(df.index[-1])
-        if pos is None:
+        return (cache, pos) if pos is not None else (None, -1)
            self._note_fallback(
                f"{self._current_tf}: slice end {df.index[-1]} not in cache")
            return None, -1
        return cache, pos
    def _note_fallback(self, why: str) -> None:
        self.fallback_count += 1
        if not self._fallback_warned:
            self._fallback_warned = True
            logger.warning(
                "GPU cache miss → falling back to live-engine recompute "
                "(%s). Further misses counted silently; check "
                "fallback_count at end of run.", why)
    # ── EMA overrides ──────────────────────────────────────────────
@@ -551,11 +528,6 @@ class GPUTechnicalEngine(TechnicalEngine):
            signal = SignalStrength.NEUTRAL
            conf = 40
        # Mirrors the live engine's ema_partial_neutral squash (parity).
        if c.ema_partial_neutral and signal in (SignalStrength.BUY, SignalStrength.SELL):
            signal = SignalStrength.NEUTRAL
            conf = 40
        crossover_desc = ""
        # Live: `if len(ema_fast) >= 3:` then iloc[-2] — needs len ≥ 2 functionally,
        # but live's guard requires len ≥ 3 so we match that to stay byte-identical.
@@ -848,12 +820,7 @@ class GPUTechnicalEngine(TechnicalEngine):
        c = self.config
        n = c.roc_period
-        # Mirror the live guard (engine._compute_roc: len < n + 5 → skip,
+        if pos < n + 1:
        # i.e. emit only from pos >= n + 4). Unreachable in practice —
        # analyze() floors at 30 bars and n+4 < 30 for shipped periods —
        # but keep the engines textually aligned so a future period/floor
        # change can't open a silent divergence window.
        if pos < n + 4:
            return
        roc_val = float(cache.roc[pos])
@@ -903,11 +870,6 @@ class GPUTechnicalEngine(TechnicalEngine):
            return super()._compute_keltner(df, result)
        c = self.config
        # Mirror the live guard (engine._compute_keltner: len < max(kc_period,
        # kc_atr_period) + 5 → skip). Unreachable below the 30-bar analyze()
        # floor with shipped periods; kept aligned for the same reason as ROC.
        if pos < max(c.kc_period, c.kc_atr_period) + 4:
            return
        latest = float(df["Close"].iloc[-1])
        kc_upper_val = float(cache.kc_upper[pos])
        kc_lower_val = float(cache.kc_lower[pos])
@@ -1011,16 +973,6 @@ def fetch_history(
    path = _cache_path(symbol, timeframe, start, end, provider.name)
    if use_cache and path.exists():
        logger.info(f"Loading cached {symbol} {timeframe.value}: {path.name}")
        # Files written before the raw-cache change (2026-07-08, commit
        # 6a96ed6) contain CLEANED bars; loading one with clean=False
        # would silently miss the spike bars the raw path exists to model.
        # The key format predates versioning, so detect by mtime.
        _RAW_CACHE_EPOCH = 1783468800.0   # 2026-07-08 00:00 UTC
        if path.stat().st_mtime < _RAW_CACHE_EPOCH:
            logger.error(
                f"{path.name} predates the raw-cache format (2026-07-08) "
                f"and likely contains CLEANED bars — delete it and refetch "
                f"before trusting trade resolution on this window")
        df = pd.read_parquet(path)
        if df.index.tz is None:
            df.index = df.index.tz_localize("UTC")
@@ -1172,15 +1124,6 @@ def _simulate_trailing(
        reason = "TIMEOUT"
        for j in range(n):
            if np.isnan(h[j]):
                # Data ends here: close at the LAST VALID bar, not at
                # c[n-1] (which may itself be NaN — a NaN exit price used
                # to poison r_multiple and every downstream metric).
                if j == 0:
                    exit_idx = 0
                    reason = "NO_DATA"
                else:
                    exit_idx = j - 1
                    exit_price = float(c[exit_idx])
                break
            hit = (l[j] <= cur_sl) if buy else (h[j] >= cur_sl)
            if hit:
@@ -1202,15 +1145,8 @@ def _simulate_trailing(
                       else peak + dist * risk)
                cur_sl = max(cur_sl, lvl) if buy else min(cur_sl, lvl)
        if reason == "NO_DATA":
            t.exit_reason = reason
            continue
        if exit_price is None:
            exit_price = float(c[exit_idx])
        if not np.isfinite(exit_price):
            # Last-resort guard: never emit a NaN exit/R into the book.
            t.exit_reason = "NO_DATA"
            continue
        t.exit_reason = reason
        t.bars_held = exit_idx + 1
        t.exit_price = float(exit_price)
@@ -1383,12 +1319,6 @@ def simulate_trades(
            last_idx = int(n_valid) - 1
            t.exit_price = float(closes_arr[i, last_idx])
            t.exit_time = exit_times[i][last_idx]
            if not np.isfinite(t.exit_price):
                # NaN close on the final window bar (corrupt raw data)
                # must not become a NaN R.
                t.exit_reason = "NO_DATA"
                t.exit_price = None
                continue
        # Realized R from actual fill prices, net of round-trip spread.
        risk = abs(t.entry_price - t.sl)
@@ -1441,12 +1371,6 @@ class BacktestConfig:
    # Round-trip transaction cost in price units, charged on every trade
    # (XAU/USD retail spread ≈ $0.30). 0 disables.
    spread: float = 0.30
    # RESEARCH (2026-08): >0 replaces the exit model with a hard TP at
    # this multiple of risk from entry (2.0 = fixed 1:2 RR); trailing is
    # disabled for ALL trades, including per-trade XAU_PB overrides.
    # Entries/stops/gates are untouched, so the signal book is identical
    # to the baseline — only the exit policy changes. Never shipped.
    fixed_rr_tp: float = 0.0
    # Replay worker processes for the bar loop. 0 = auto (cpu_count - 2,
    # capped at 12), 1 = sequential in-process. Signal evaluation per bar
    # is pure (past data only), so it fans out across forked workers; the
@@ -1459,22 +1383,18 @@ class BacktestConfig:
 def _is_xau_blocked_by_calendar(ts: pd.Timestamp, cfg: PlatformConfig) -> Optional[str]:
    """Apply XAU weekend + allowed-weekday filter matching live `analyze_pair`.
-    Market closed Friday 17:00 -> Sunday 18:00 NEW YORK time (= Fri
+    Forex market closed: Friday 22:00 UTC -> Sunday 22:00 UTC (Sydney open).
-    21:00 UTC in US summer, 22:00 in winter — matches measured Oanda
+    Must match `_is_forex_weekend` in signal_aggregator.py exactly.
    feed behavior; 2026-08-04 audit I43). Must match `_is_forex_weekend`
    in signal_aggregator.py exactly (rule 4; Module 21 agreement test).
    Note the weekday gates below stay on UTC weekday — they mirror the
    live gates, which evaluate UTC weekdays.
    """
    xau = cfg.xau_strategy
    if xau.weekend_filter:
-        ny = ts.astimezone(_NY_TZ)
+        wd = ts.weekday()
-        wd = ny.weekday()
+        hour = ts.hour
-        if wd == 4 and ny.hour >= 17:
+        if wd == 4 and hour >= 22:
            return "WEEKEND_FILTER"
        if wd == 5:
            return "WEEKEND_FILTER"
-        if wd == 6 and ny.hour < 18:
+        if wd == 6 and hour < 22:
            return "WEEKEND_FILTER"
    if xau.allowed_weekdays and ts.weekday() not in xau.allowed_weekdays:
        return "DAY_FILTER"
@@ -1691,43 +1611,6 @@ def _reanchor_entries_to_raw(
        )
 def _apply_cooldown(
    candidates: List[Tuple[int, Trade]], cooldown_bars: int
 ) -> List[Tuple[int, Trade]]:
    """Ordered cooldown pass over (bar_pos, Trade) candidates: drop any
    candidate within `cooldown_bars` bars of the previously ACCEPTED
    entry — exactly what a sequential replay loop would have skipped.
    Candidates must already be sorted by position (stable merge from the
    worker spans)."""
    accepted: List[Tuple[int, Trade]] = []
    last_entry_pos = -10_000
    for pos, trade in candidates:
        if pos - last_entry_pos < cooldown_bars:
            continue
        accepted.append((pos, trade))
        last_entry_pos = pos
    return accepted
 def _apply_fixed_rr_tp(trades: List[Trade], mult: float) -> None:
    """RESEARCH: replace each trade's TP with a hard target at
    `mult` × |entry − sl| and zero the per-trade trail params so the
    XAU_PB overrides cannot resurrect trailing. Risk is measured off the
    final (re-anchored) entry/SL, so call this after
    `_reanchor_entries_to_raw`. The caller must also pass
    trail_arm_r=0 to `simulate_trades` — dispatch is on the run-level arg."""
    for t in trades:
        risk = abs(t.entry_price - t.sl)
        t.tp1 = (t.entry_price + mult * risk if t.direction == "BUY"
                 else t.entry_price - mult * risk)
        t.tp2 = t.tp1              # static branch never reads tp2; CSV hygiene
        t.trail_arm_r = 0.0
        t.trail_dist_r = 0.0
        t.rr_planned = mult
        if t.pip_value:
            t.reward_pips = mult * risk / t.pip_value
 def run_backtest(
    bt_cfg: BacktestConfig,
    platform_cfg: Optional[PlatformConfig] = None,
@@ -1855,10 +1738,18 @@ def run_backtest(
        candidates.extend(cands)
    candidates.sort(key=lambda pc: pc[0])
-    # ── Phase 2: entry cooldown — the only sequential rule.
+    # ── Phase 2: entry cooldown — the only sequential rule. A candidate
-    accepted = _apply_cooldown(candidates, bt_cfg.cooldown_bars)
+    # inside the cooldown window of the previously ACCEPTED entry is
-    trades: List[Trade] = [t for _, t in accepted]
+    # dropped, exactly as the sequential loop skipped those bars.
-    signal_ts_list: List[pd.Timestamp] = [primary_df.index[p] for p, _ in accepted]
+    trades: List[Trade] = []
    signal_ts_list: List[pd.Timestamp] = []
    last_entry_pos = -10_000
    for pos, trade in candidates:
        if pos - last_entry_pos < bt_cfg.cooldown_bars:
            continue
        trades.append(trade)
        signal_ts_list.append(primary_df.index[pos])
        last_entry_pos = pos
    logger.info(
        f"Replay complete: {len(trades)} entries in "
@@ -1878,17 +1769,10 @@ def run_backtest(
        use_cache=bt_cfg.use_cache, clean=False,
    )
    _reanchor_entries_to_raw(trades, signal_ts_list, sim_df)
    if bt_cfg.fixed_rr_tp > 0:
        _apply_fixed_rr_tp(trades, bt_cfg.fixed_rr_tp)
        logger.info(
            f"RESEARCH override: fixed-RR TP = {bt_cfg.fixed_rr_tp} x risk; "
            f"trailing exit DISABLED for all {len(trades)} trades (incl. XAU_PB)"
        )
    run_arm = 0.0 if bt_cfg.fixed_rr_tp > 0 else platform_cfg.signals.trail_arm_r
    t_sim = time.perf_counter()
    simulate_trades(trades, sim_df, max_bars=bt_cfg.max_hold_bars,
                    spread=bt_cfg.spread,
-                    trail_arm_r=run_arm,
+                    trail_arm_r=platform_cfg.signals.trail_arm_r,
                    trail_dist_r=platform_cfg.signals.trail_dist_r)
    logger.info(
        f"Trade simulation took {time.perf_counter() - t_sim:.2f}s "
@@ -1938,14 +1822,6 @@ def _summarize(rs: List[float]) -> Dict:
 def compute_metrics(trades: List[Trade]) -> Metrics:
    resolved = [t for t in trades if t.exit_reason and t.exit_reason != "NO_DATA"]
    # Last line of defense: one non-finite R (NaN exit on corrupt data)
    # would silently NaN every aggregate below. Exclude loudly.
    bad = [t for t in resolved if not np.isfinite(t.r_multiple)]
    if bad:
        logger.error(
            "compute_metrics: excluding %d trade(s) with non-finite R "
            "(corrupt exit data) — first at %s", len(bad), bad[0].entry_time)
        resolved = [t for t in resolved if np.isfinite(t.r_multiple)]
    if not resolved:
        return Metrics(0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, {}, {}, {})
@@ -2232,10 +2108,6 @@ def _parse_args(argv: Optional[List[str]] = None) -> argparse.Namespace:
                   help="Override signals.trail_arm_r for this run (research knob)")
    p.add_argument("--trail-dist", type=float, default=None,
                   help="Override signals.trail_dist_r for this run (research knob)")
    p.add_argument("--fixed-rr-tp", type=float, default=None,
                   help="RESEARCH: exit at a hard TP of MULT x risk (e.g. 2.0 = "
                        "fixed 1:2 RR) instead of the trailing stop; "
                        "entries/stops/gates unchanged.")
    p.add_argument("--quality", default="A+,A,B,C", help="Comma-separated allowed grades")
    p.add_argument("--strategies", default="",
                   help="Comma-separated strategy tags to include (XAU,XAU_PB,XAU_CP,GENERIC). Empty = all.")
@@ -2245,19 +2117,6 @@ def _parse_args(argv: Optional[List[str]] = None) -> argparse.Namespace:
    p.add_argument("--cp-patterns", default=None,
                   help="Comma-separated pattern subset for --cp (e.g. "
                        "double_bottom,bull_flag). Default: all detectors.")
    p.add_argument("--big-candle", type=float, default=None,
                   help="Force xau_strategy.big_candle_atr_multiple=K for this "
                        "run (no entries while the lookback window contains a "
                        "bar with range >= K x prior ATR) — for validating the "
                        "candidate.")
    p.add_argument("--big-candle-hours", type=float, default=None,
                   help="Override big_candle_lookback_hours for --big-candle "
                        "runs (default 24).")
    p.add_argument("--set", action="append", default=[], metavar="PATH=JSON",
                   help="Override any PlatformConfig leaf by dotted path; the "
                        "value is JSON-parsed for typing (research knob; "
                        "repeatable). e.g. --set xau_strategy.m5_min_abs_score=25 "
                        "--set 'xau_strategy.sell_min_quality=[\"A+\",\"A\",\"B\",\"C\"]'")
    p.add_argument("--no-cache", action="store_true", help="Skip parquet cache (force API fetch)")
    p.add_argument("--spread", type=float, default=None,
                   help="Round-trip cost in price units per trade (default: pair's "
@@ -2289,39 +2148,6 @@ def _resolve_tf(s: str) -> Timeframe:
    raise SystemExit(f"Unknown timeframe '{s}'. Valid: {[t.value for t in Timeframe]}")
 def _apply_config_overrides(cfg: PlatformConfig, overrides: List[str]) -> None:
    """Apply repeatable --set PATH=JSON research overrides onto the config.
    Unknown paths are a hard error: dataclass instances silently accept
    setattr of any name, so a typo'd path would otherwise produce a
    no-op run that looks like 'the relaxation had no effect'. Every
    applied override is logged — the run log is the config audit trail
    required by instructions_for_backtest.txt.
    """
    for item in overrides:
        path, sep, raw = item.partition("=")
        if not sep:
            raise SystemExit(f"--set expects PATH=VALUE, got '{item}'")
        try:
            value = json.loads(raw)
        except json.JSONDecodeError:
            value = raw  # bare-string convenience ("oanda" etc.)
        obj = cfg
        parts = path.strip().split(".")
        for attr in parts[:-1]:
            if not hasattr(obj, attr):
                raise SystemExit(f"--set: unknown config path '{path}'")
            obj = getattr(obj, attr)
        leaf = parts[-1]
        if not hasattr(obj, leaf):
            raise SystemExit(f"--set: unknown config path '{path}'")
        old = getattr(obj, leaf)
        if hasattr(old, "__dataclass_fields__"):
            raise SystemExit(f"--set: '{path}' is a config section, not a leaf")
        setattr(obj, leaf, value)
        logger.info(f"--set override: {path} = {value!r} (was {old!r})")
 def main(argv: Optional[List[str]] = None) -> int:
    args = _parse_args(argv)
@@ -2359,16 +2185,6 @@ def main(argv: Optional[List[str]] = None) -> int:
        # make_config; the fallback must match (live == backtest).
        from app_config import apply_pair_overrides  # noqa: E402
        apply_pair_overrides(platform_cfg)
    if args.fixed_rr_tp is not None:
        if args.fixed_rr_tp <= 0:
            raise SystemExit("--fixed-rr-tp must be > 0")
        if (args.trail_arm is not None or args.trail_dist is not None or
                any(s.split("=", 1)[0].strip().startswith(("signals.trail",
                                                           "signals.pb_trail"))
                    for s in (args.set or []))):
            raise SystemExit(
                "--fixed-rr-tp conflicts with --trail-arm/--trail-dist/"
                "--set signals.*trail* (the flag disables trailing entirely)")
    if args.swing_sr_lookback > 0:
        platform_cfg.technical.swing_sr_lookback = args.swing_sr_lookback
    if args.trail_arm is not None:
@@ -2381,12 +2197,6 @@ def main(argv: Optional[List[str]] = None) -> int:
            platform_cfg.xau_strategy.cp_patterns = [
                s.strip() for s in args.cp_patterns.split(",") if s.strip()
            ]
    if args.big_candle is not None:
        platform_cfg.xau_strategy.big_candle_atr_multiple = args.big_candle
    if args.big_candle_hours is not None:
        platform_cfg.xau_strategy.big_candle_lookback_hours = args.big_candle_hours
    if args.set:
        _apply_config_overrides(platform_cfg, args.set)
    confirm_tfs = list(platform_cfg.confirmation_timeframes)
    end = (
@@ -2415,7 +2225,6 @@ def main(argv: Optional[List[str]] = None) -> int:
        strategies_only=[s.strip() for s in args.strategies.split(",") if s.strip()],
        spread=spread,
        workers=args.workers,
        fixed_rr_tp=args.fixed_rr_tp or 0.0,
    )
    logger.info(
@@ -1,86 +0,0 @@
 """Trade-book diff: relaxed/gated run vs its control.
 Usage:
    python backtest/research/book_diff.py CONTROL.csv CANDIDATE.csv BAR_MINUTES
 Decision numbers are RUN-LEVEL (ΔtotalR, ΔPF, ΔmaxDD, Δtrades) — never
 the added-trade bucket alone: the 5-bar entry cooldown means new
 entries displace existing ones. The added/removed decomposition and the
 per-decision-hour histogram of added trades are diagnostics.
 Shifted trades = removed+added pairs in the same direction within
 ±5 bars (cooldown displacement, not a genuinely new/lost signal).
 """
 import sys
 import pandas as pd
 def _metrics(df):
    r = df["r_multiple"]
    gains, losses = r[r > 0].sum(), -r[r <= 0].sum()
    cum = r.cumsum()
    return dict(trades=len(df), total_r=r.sum(),
                pf=gains / max(losses, 1e-9),
                wr=(r > 0).mean() * 100 if len(df) else 0.0,
                max_dd=(cum.cummax() - cum).max() if len(cum) else 0.0)
 def diff_books(control_csv, candidate_csv, bar_minutes, split="2026-01-29"):
    ctrl, cand = pd.read_csv(control_csv), pd.read_csv(candidate_csv)
    for df in (ctrl, cand):
        df["_t"] = pd.to_datetime(df["entry_time"], utc=True, format="mixed")
        df["_key"] = df["entry_time"].astype(str) + "|" + df["direction"]
    added = cand[~cand["_key"].isin(ctrl["_key"])].copy()
    removed = ctrl[~ctrl["_key"].isin(cand["_key"])].copy()
    # cooldown displacement: same-direction removed/added within ±5 bars
    tol = pd.Timedelta(minutes=5 * bar_minutes)
    shifted = 0
    rem_pool = removed.copy()
    for _, a in added.iterrows():
        m = rem_pool[(rem_pool["direction"] == a["direction"])
                     & ((rem_pool["_t"] - a["_t"]).abs() <= tol)]
        if len(m):
            shifted += 1
            rem_pool = rem_pool.drop(m.index[0])
    mc, mk = _metrics(ctrl), _metrics(cand)
    split_ts = pd.Timestamp(split, tz="UTC")
    halves = (cand[cand["_t"] < split_ts]["r_multiple"].sum(),
              cand[cand["_t"] >= split_ts]["r_multiple"].sum())
    hour_hist = None
    if len(added):
        added["dec_hour"] = (added["_t"]
                             - pd.Timedelta(minutes=bar_minutes)).dt.hour
        hour_hist = added.groupby("dec_hour")["r_multiple"] \
                         .agg(["count", "sum"]).round(2)
    return dict(
        control=mc, candidate=mk,
        delta_r=mk["total_r"] - mc["total_r"],
        delta_pf=mk["pf"] - mc["pf"],
        delta_dd=mk["max_dd"] - mc["max_dd"],
        delta_trades=mk["trades"] - mc["trades"],
        added_n=len(added), added_r=added["r_multiple"].sum(),
        removed_n=len(removed), removed_r=removed["r_multiple"].sum(),
        shifted=shifted, halves=halves, added_by_hour=hour_hist,
    )
 if __name__ == "__main__":
    d = diff_books(sys.argv[1], sys.argv[2], int(sys.argv[3]))
    c, k = d["control"], d["candidate"]
    print(f"control  : {c['trades']:4d} trades  {c['total_r']:+7.1f}R  "
          f"PF {c['pf']:.2f}  WR {c['wr']:.1f}%  DD {c['max_dd']:.1f}")
    print(f"candidate: {k['trades']:4d} trades  {k['total_r']:+7.1f}R  "
          f"PF {k['pf']:.2f}  WR {k['wr']:.1f}%  DD {k['max_dd']:.1f}")
    print(f"ΔR {d['delta_r']:+.1f}  ΔPF {d['delta_pf']:+.2f}  "
          f"ΔDD {d['delta_dd']:+.1f}  Δtrades {d['delta_trades']:+d}")
    print(f"added {d['added_n']} ({d['added_r']:+.1f}R)  "
          f"removed {d['removed_n']} ({d['removed_r']:+.1f}R)  "
          f"shifted≈{d['shifted']}")
    print(f"candidate halves: H1 {d['halves'][0]:+.1f}R  H2 {d['halves'][1]:+.1f}R")
    if d["added_by_hour"] is not None:
        print("\nadded trades by decision hour:")
        print(d["added_by_hour"].to_string())
@@ -1,42 +0,0 @@
 """Burst-candle frequency by UTC hour — the empirical "hedge fund hours".
 Usage:
    python backtest/research/burst_profile.py PARQUET_GLOB
 Per UTC bar-close hour of the given bars: % of bars whose range is
 >= K x the PRIOR bar's Wilder ATR(14), for K in {3,4,5}; mean bar
 range; and mean signed drift over the 6 bars following a >=4x burst
 (does price continue or snap back?). Analysis tooling only.
 """
 import glob
 import sys
 import pandas as pd
 def burst_table(parquet_glob: str, bar_minutes: int = 5):
    files = sorted(glob.glob(parquet_glob))
    if not files:
        raise SystemExit(f"no parquet matches {parquet_glob}")
    df = pd.concat([pd.read_parquet(f) for f in files]).sort_index()
    df = df[~df.index.duplicated(keep="last")]
    h, l, c = df["High"], df["Low"], df["Close"]
    tr = pd.concat([h - l, (h - c.shift()).abs(), (l - c.shift()).abs()],
                   axis=1).max(axis=1)
    atr = tr.ewm(alpha=1 / 14, min_periods=14, adjust=False).mean()
    ratio = (h - l) / atr.shift(1)
    close_hour = (df.index + pd.Timedelta(minutes=bar_minutes)).hour
    fwd6 = c.shift(-6) - c  # signed move over the 6 bars after this one
    out = pd.DataFrame({
        "bars": ratio.groupby(close_hour).size(),
        **{f"pct_ge_{k}x": (ratio >= k).groupby(close_hour).mean() * 100
           for k in (3, 4, 5)},
        "mean_range": (h - l).groupby(close_hour).mean(),
        "post4x_drift6": fwd6[ratio >= 4].groupby(close_hour[ratio >= 4]).mean(),
    })
    return out.round(3)
 if __name__ == "__main__":
    print(burst_table(sys.argv[1],
                      int(sys.argv[2]) if len(sys.argv) > 2 else 5).to_string())
@@ -0,0 +1,94 @@
 """Load and query the historical economic calendar (backtest side).
 The store is built by `ff_calendar_scraper.py`. Timestamps are unix
 epoch (UTC) — verified against known release times and price-volatility
 alignment (see scraper docstring).
 Key API:
    cal = load_calendar()                          # full DataFrame, `utc` column
    ev  = precise_events(cal, ["USD"])             # high-impact, minute-accurate
    mask = event_window_mask(bar_index, ev.utc, before, after)
 """
 from pathlib import Path
 from typing import Iterable, Optional
 import numpy as np
 import pandas as pd
 PARQUET = Path(__file__).resolve().parent.parent / "cache" / "calendar.parquet"
 # timeLabel values that mean "no intraday time" — these events cannot
 # anchor an M5/M15 window (German Prelim CPI is the notable High one).
 _UNTIMED = {"All Day", "Tentative", ""}
 def _parse_release_value(raw) -> float:
    """'0.3%' / '-1.7%' / '228K' / '1.1M' -> float (nan if unparseable)."""
    import re
    if raw is None or (isinstance(raw, float) and np.isnan(raw)):
        return np.nan
    m = re.search(r"(-?\d+(?:\.\d+)?)\s*([kmbt])?",
                  str(raw).replace(",", ""), re.IGNORECASE)
    if not m:
        return np.nan
    mult = {"k": 1e3, "m": 1e6, "b": 1e9, "t": 1e12}.get(
        (m.group(2) or "").lower(), 1.0)
    return float(m.group(1)) * mult
 def load_calendar(path: Path = PARQUET) -> pd.DataFrame:
    df = pd.read_parquet(path)
    df["utc"] = pd.to_datetime(df.dateline, unit="s", utc=True)
    for col in ("actual", "forecast", "previous"):
        df[f"{col}_num"] = df[col].map(_parse_release_value)
    df["surprise"] = df.actual_num - df.forecast_num
    return df
 def precise_events(
    cal: pd.DataFrame,
    currencies: Iterable[str] = ("USD",),
    impacts: Iterable[str] = ("high",),
    require_values: bool = False,
 ) -> pd.DataFrame:
    """Events with a minute-accurate timestamp, filtered by currency/impact."""
    out = cal[cal.currency.isin(list(currencies))
              & cal.impactName.isin(list(impacts))
              & ~cal.timeMasked.astype(bool)
              & ~cal.timeLabel.fillna("").isin(_UNTIMED)]
    if require_values:
        out = out.dropna(subset=["actual_num", "forecast_num"])
    return out.sort_values("utc").reset_index(drop=True)
 def event_window_mask(
    bar_index: pd.DatetimeIndex,
    event_times: pd.Series,
    before: pd.Timedelta = pd.Timedelta(minutes=30),
    after: pd.Timedelta = pd.Timedelta(minutes=30),
 ) -> np.ndarray:
    """Boolean mask: bar OPEN time falls inside [event-before, event+after).
    Use on a backtest's entry times to identify trades entered around
    releases (avoidance counterfactuals, event-window cost models).
    """
    starts = pd.DatetimeIndex(event_times) - before
    ends = pd.DatetimeIndex(event_times) + after
    order = np.argsort(starts.values)
    starts, ends = starts.values[order], ends.values[order]
    # merge overlapping windows, then interval-stab each bar time
    merged_s, merged_e = [], []
    for s, e in zip(starts, ends):
        if merged_e and s <= merged_e[-1]:
            merged_e[-1] = max(merged_e[-1], e)
        else:
            merged_s.append(s)
            merged_e.append(e)
    ms = np.array(merged_s)
    me = np.array(merged_e)
    ts = bar_index.values
    pos = np.searchsorted(ms, ts, side="right") - 1
    ok = pos >= 0
    res = np.zeros(len(ts), dtype=bool)
    res[ok] = ts[ok] < me[pos[ok]]
    return res
@@ -1,178 +0,0 @@
 """Commit-ladder campaign driver (2026-08).
 Question: of all signal-code states shipped in the last two months, which
 configuration is the most accurate? Historical in-era backtest numbers are
 not comparable (the backtester's realism model changed on 07-08/07-16 and
 the data feed switched to Oanda on 07-28), so every historical
 configuration is recreated on TODAY's harness via --set overrides and
 replayed on the same pinned Oanda bars. Bug-fix commits stay on; the
 ladder compares shipped tuning decisions only.
 Usage:
    python backtest/research/commit_ladder.py            # full matrix
    python backtest/research/commit_ladder.py --tf 5min  # one timeframe
    python backtest/research/commit_ladder.py --windows 12mo,oos
 Resumable: a run whose CSV already exists in the output dir is skipped.
 Outputs: backtest/research/commit_ladder_2026-08/<variant>_<window>.csv
         + .log per run, summary.csv, and a printed ranking table.
 """
 from __future__ import annotations
 import argparse
 import subprocess
 import sys
 from pathlib import Path
 import pandas as pd
 ROOT = Path(__file__).resolve().parents[2]
 BT = ROOT / "backtest" / "gpu_backtest.py"
 OUT = ROOT / "backtest" / "research" / "commit_ladder_2026-08"
 WINDOWS = {
    "12mo": ("2025-07-30", "2026-07-30"),
    "oos": ("2024-06-01", "2025-03-30"),
    "5yr": ("2021-07-30", "2026-07-30"),
 }
 SPLIT_12MO = pd.Timestamp("2026-01-29", tz="UTC")  # book_diff.py's split
 # (variant, "commit date what-it-represents", [--set overrides])
 VARIANTS: dict[str, list[tuple[str, str, list[str]]]] = {
    "5min": [
        ("M5-A", "dad2432 07-09 gates+trailing+pullback",
         ["xau_strategy.blocked_hours_utc=[0,1,2,10,11,13,16,21]",
          "xau_strategy.cp_enabled=false",
          "signals.trail_dist_r=0.75",
          "xau_strategy.m5_min_abs_score=30",
          "xau_strategy.m5_min_confirmation_score=10"]),
        ("M5-B", "f3ba4d5 07-11 +hour-9 block",
         ["xau_strategy.blocked_hours_utc=[0,1,2,10,11,13,16,21,9]",
          "xau_strategy.cp_enabled=false",
          "signals.trail_dist_r=0.75",
          "xau_strategy.m5_min_abs_score=30",
          "xau_strategy.m5_min_confirmation_score=10"]),
        ("M5-C", "de6424e 07-16 +chart patterns",
         ["xau_strategy.blocked_hours_utc=[0,1,2,10,11,13,16,21,9]",
          "signals.trail_dist_r=0.75",
          "xau_strategy.m5_min_abs_score=30",
          "xau_strategy.m5_min_confirmation_score=10"]),
        ("M5-D", "3ee79b5 07-21 +trail dist 0.85",
         ["xau_strategy.blocked_hours_utc=[0,1,2,10,11,13,16,21,9]",
          "xau_strategy.m5_min_abs_score=30",
          "xau_strategy.m5_min_confirmation_score=10"]),
        ("M5-E", "0396970 07-30 relaxation (HEAD)", []),
    ],
    "15min": [
        ("M15-A", "987ec5b 07-09 pre-campaigns",
         ["signals.tp_floor_rr=0.5",
          "xau_strategy.blocked_regimes=[]",
          "xau_strategy.sell_blocked_hours_utc=[]",
          "signals.trail_arm_r=0.60"]),
        ("M15-B", "f3ba4d5/f88e39f 07-11 1R floor + regime block",
         ["xau_strategy.sell_blocked_hours_utc=[]",
          "signals.trail_arm_r=0.60"]),
        ("M15-C", "7d3fd95 07-16 +SELL hour 15/17 block",
         ["signals.trail_arm_r=0.60"]),
        ("M15-D", "bf8e5fb 07-21 trail arm 0.70 (HEAD)", []),
    ],
    "1h": [
        ("H1-A", "pre-d95d31a 07-08 TP-tightening model",
         ["signals.min_rr_ratio=0.4",
          "signals.tp_tightening_factor=0.8"]),
        ("H1-B", "HEAD", []),
    ],
 }
 def run_one(tf: str, name: str, overrides: list[str], wkey: str,
            spread: str | None = None) -> Path:
    suffix = f"_s{spread}" if spread else ""
    csv_path = OUT / f"{name}_{wkey}{suffix}.csv"
    log_path = OUT / f"{name}_{wkey}{suffix}.log"
    if csv_path.exists():
        print(f"  [skip] {csv_path.name} exists")
        return csv_path
    start, end = WINDOWS[wkey]
    cmd = [sys.executable, str(BT), "--pair", "XAUUSD",
           "--primary-tf", tf, "--start", start, "--end", end,
           "--export-csv", str(csv_path)]
    if spread:
        cmd += ["--spread", spread]
    for ov in overrides:
        cmd += ["--set", ov]
    print(f"  [run ] {csv_path.name} ...", flush=True)
    with open(log_path, "w") as lf:
        rc = subprocess.run(cmd, stdout=lf, stderr=subprocess.STDOUT,
                            cwd=ROOT).returncode
    if rc != 0:
        raise SystemExit(f"backtest failed (rc={rc}) — see {log_path}")
    log = log_path.read_text()
    n_ov = log.count("--set override")
    if n_ov != len(overrides):
        raise SystemExit(
            f"{name}_{wkey}: expected {len(overrides)} --set overrides, "
            f"log shows {n_ov} — see {log_path}")
    if "Loading cached" not in log and wkey != "sanity":
        print(f"  [warn] {log_path.name}: no 'Loading cached' line — "
              f"data was fetched live for at least one timeframe")
    return csv_path
 def metrics(csv_path: Path, wkey: str) -> dict:
    df = pd.read_csv(csv_path, parse_dates=["entry_time"])
    if df.empty:
        return {"trades": 0, "total_r": 0.0, "wr": 0.0, "pf": 0.0,
                "max_dd": 0.0, "h1_r": 0.0, "h2_r": 0.0, "years_pos": ""}
    if df["entry_time"].dt.tz is None:
        df["entry_time"] = df["entry_time"].dt.tz_localize("UTC")
    r = df["r_multiple"]
    pos, neg = r[r > 0].sum(), r[r < 0].sum()
    cum = r.cumsum()
    dd = (pd.concat([pd.Series([0.0]), cum]).cummax()[1:] - cum).max()
    out = {
        "trades": len(df),
        "total_r": round(r.sum(), 1),
        "wr": round((r > 0).mean() * 100, 1),
        "pf": round(pos / abs(neg), 2) if neg else float("inf"),
        "max_dd": round(dd, 1),
        "h1_r": "", "h2_r": "", "years_pos": "",
    }
    if wkey == "12mo":
        h1 = r[df["entry_time"] < SPLIT_12MO].sum()
        out["h1_r"] = round(h1, 1)
        out["h2_r"] = round(r.sum() - h1, 1)
    if wkey == "5yr":
        by_year = r.groupby(df["entry_time"].dt.year).sum()
        out["years_pos"] = f"{(by_year > 0).sum()}/{len(by_year)}"
    return out
 def main() -> None:
    ap = argparse.ArgumentParser()
    ap.add_argument("--tf", choices=list(VARIANTS), default=None)
    ap.add_argument("--windows", default="12mo,oos,5yr")
    args = ap.parse_args()
    OUT.mkdir(parents=True, exist_ok=True)
    wkeys = [w.strip() for w in args.windows.split(",")]
    tfs = [args.tf] if args.tf else list(VARIANTS)
    rows = []
    for tf in tfs:
        for name, label, overrides in VARIANTS[tf]:
            for wkey in wkeys:
                csv_path = run_one(tf, name, overrides, wkey)
                m = metrics(csv_path, wkey)
                rows.append({"tf": tf, "variant": name, "represents": label,
                             "window": wkey, **m})
                print(f"    {name} {wkey}: {m['trades']} trades, "
                      f"{m['total_r']}R, PF {m['pf']}, WR {m['wr']}%")
    summary = pd.DataFrame(rows)
    summary.to_csv(OUT / "summary.csv", index=False)
    print("\n=== SUMMARY (also in summary.csv) ===")
    print(summary.to_string(index=False))
 if __name__ == "__main__":
    main()
@@ -1,123 +0,0 @@
 # Fixed-RR hard-TP vs shipped trailing exit — research campaign 2026-08-05
 **Verdict: DO NOT SHIP. The trailing exit stays.** The fixed 2R target beats the trail on
 total R in every 2024+ window but the edge is a trend-regime artifact: it inverts in the
 2021–2023 ranging years, doubles drawdown, cuts win rate 10–19 points, and doubles hold
 times (amplifying the accepted live one-position divergence). Details below.
 ## Setup
 - Code: HEAD `fc5ce85` + the `--fixed-rr-tp` research flag (uncommitted at run time).
  The flag replaces every trade's TP with `mult × |entry − sl|` AFTER re-anchoring and
  disables trailing (run-level and per-trade PB params) — entries, stops, gates, and the
  cooldown pass are untouched, so both arms replay **trade-identical books** (verified:
  paired join on entry_time+direction matches 1:1 in every window). Only the exit policy
  differs → per-trade paired comparison.
 - Provider: Oanda (cache-warm; keys `oanda_XAU_USD_{5min,15min,1h}_{20250603_20260804,
  20240331_20250330,20210531_20260804}`). Charged cost `--spread 0.65` (real measured
  round trip); $0.30 letter-cost books derived analytically as `r + 0.35/risk` and
  verified exact against one real `--spread 0.30` run (max per-trade error 1.8e-15 —
  spread never enters trigger detection, per the owner-declined 2026-08-04 decision).
 - Control sanity gate: fresh M5 12mo control reproduced the pinned HEAD reference
  exactly — 555 trades, +98.64R, PF 1.483, maxDD −10.73R.
 - Commands (control = same without the flag):
  `venv/bin/python backtest/gpu_backtest.py --pair XAUUSD --primary-tf {5min,15min}
   --start 2025-08-04 --end 2026-08-04 --spread 0.65 [--fixed-rr-tp 2.0]
   --export-csv backtest/fixedtp_*.csv`
  OOS `--start 2024-06-01 --end 2025-03-30`; 5yr `--start 2021-08-01 --end 2026-08-04`.
 - Books pinned as `backtest/fixedtp_{m5,m15}_{12mo,oos,5yr}_{ctrl,rr20,...}.csv`
  (per-trade indicator breakdown, agreement counts, regime, vol-regime, weekday,
  MAE/MFE, bars-held — the full instructions_for_backtest record).
 ## Headline (@ $0.65)
 | Window | Arm | n | R | PF | WR | maxDD | recov | medBars | exits |
 |---|---|---|---|---|---|---|---|---|---|
 | M5 12mo | trail | 555 | +98.64 | 1.483 | 52.4% | −10.7 | 9.2 | 28 | TRAIL 361 / SL 183 / TO 11 |
 | M5 12mo | 2R hard | 555 | +124.85 | 1.378 | 42.9% | −24.5 | 5.1 | 50 | SL 312 / TP1 210 / TO 33 |
 | M15 12mo | trail | 397 | +73.93 | 1.525 | 62.2% | −10.8 | 6.8 | 28 | TRAIL 255 / SL 135 / TO 7 |
 | M15 12mo | 2R hard | 397 | +95.95 | 1.431 | 44.3% | −11.2 | 8.6 | 39 | SL 215 / TP1 151 / TO 31 |
 | M5 OOS | trail | 86 | +9.45 | 1.332 | 53.5% | −8.2 | 1.2 | 70 | — |
 | M5 OOS | 2R hard | 86 | +18.83 | 1.407 | 45.3% | −9.7 | 1.9 | 124 | — |
 | M15 OOS | trail | 299 | +38.77 | 1.360 | 59.2% | −15.5 | 2.5 | 38 | — |
 | M15 OOS | 2R hard | 299 | +72.14 | 1.435 | 45.2% | −13.9 | 5.2 | 82 | — |
 Halves (12mo, split 2026-02-03): M5 rr20 +67.99/+56.86 (both positive; H2 PF 1.325 vs
 ctrl 1.503), M15 rr20 +73.74/+22.21 (both positive; ctrl H2 +7.23 — the known H2-flat).
 At $0.30 all deltas keep the same sign (e.g. M5 12mo +109.0 vs +135.2).
 So the **letter** of rule 1 passes for rr20 on both TFs. The 5yr decomposition is why it
 still must not ship.
 ## Exit migration (paired, M5 12mo; same pattern everywhere)
 - SL→SL 183 trades: ΔR 0 (identical by construction — stops untouched).
 - TRAIL→TP1 207 trades: +248.9R → +424.5R (**+175.6R**) — winners the trail exits early.
 - TRAIL→SL 129 trades: +19.0R → −136.8R (**−155.7R**) — retraces the trail survives
  become full stop-outs.
 - TRAIL→TIMEOUT 25: +7.4R. Net = +26.2R. The whole effect is these two big flows nearly
  cancelling; which one wins is a regime property, not an exit-policy edge.
 - Trades with ctrl MFE ≥ 2R: 95; hard cap forfeits only +10.4R vs trail on M5 (the trail
  already gives ~0.75R back from peak, so a 2R cap costs little *in a trend*).
 ## Mini-grid (12mo @ $0.65) — the tell
 | mult | M5 R / PF / DD | M15 R / PF / DD |
 |---|---|---|
 | trail | +98.6 / 1.483 / −10.7 | +73.9 / 1.525 / −10.8 |
 | 1.0R | +52.5 / 1.205 / −12.8 | +67.1 / 1.410 / −7.3 |
 | 1.5R | +80.7 / 1.263 / −21.4 | +66.9 / 1.325 / −11.2 |
 | 2.0R | +124.9 / 1.378 / −24.5 | +96.0 / 1.431 / −11.2 |
 | 2.5R | +147.7 / 1.426 / −23.6 | +110.4 / 1.469 / −15.9 |
 | 3.0R | +155.8 / 1.432 / −21.6 | +135.8 / 1.562 / −17.5 |
 Monotone in the multiple on both TFs — "2.0" is not an optimum, it's a point on a curve
 that says *never take profit* (3R's M5 TIMEOUT bucket alone is +84.8R). That is the
 signature of a one-way trend sample, not of a superior exit rule. It also reconfirms the
 original finding that SHORT fixed targets (≤1.5R) are strictly bad (cf. RR-0.5 removal,
 `config.py:126-133`: +48R → +105R).
 ## 5yr regime decomposition (2021-08-01 → 2026-08-04 @ $0.65) — the verdict
 M15 (real sample in every year): ctrl +127.97 PF 1.209 DD −35.8; rr20 +158.97 PF 1.170
 DD **−59.8**; rr30 +204.03 PF 1.203 DD **−98.3**.
 | Year | n | trail | 2R hard | 3R hard |
 |---|---|---|---|---|
 | 2021 | 88 | **+4.00** | −16.09 | −17.41 |
 | 2022 | 284 | −9.54 | −7.63 | −26.74 |
 | 2023 | 227 | **+7.79** | −1.71 | −5.81 |
 | 2024 | 345 | +40.44 | +64.98 | +89.14 |
 | 2025 | 405 | +49.33 | +69.39 | +104.42 |
 | 2026 | 229 | +35.95 | +50.04 | +60.43 |
 M5 5yr mirrors it (ctrl +128.4 PF 1.387 DD −12.8 vs rr20 +160.2 PF 1.306 DD −24.7;
 2022: −1.1 vs −5.2), though pre-2024 M5 samples are thin (12–60 trades/yr).
 The hard target's entire outperformance lives in the 2024–2026 gold trend. In ranging
 years it flips sign exactly where the trail stays positive (M15 2021, 2023), and the
 further the target, the worse the ranging-year bleed and the deeper the DD. The 12mo,
 halves, and OOS windows all sit inside the trend era, which is why the rule-1 letter
 passes — this is precisely the within-regime-split trap rule 1's OOS clause exists to
 catch, one level up.
 ## Additional non-ship reasons
 - **Live divergence grows with hold time.** Median hold doubles (M5 28→50 bars; OOS
  70→124). The daemon holds one position total; the backtester doesn't model that
  blocking. Longer holds = more live signals skipped that the paper book banks.
 - **WR 52→43% (M5) / 62→44% (M15), max loss streak 6→11 (M15)** — materially harder to
  operate and size.
 - Aggregate tables (session/day/regime/vol, streaks, quarterly equity, rolling-20 WR)
  computed and consistent — no bucket reverses the conclusion; both arms' 12mo profit is
  concentrated in Q1-2026 (+87.7 ctrl / +102.0 rr20).
 ## Status
 - `--fixed-rr-tp` stays as a documented research-only flag (default off; conflicts with
  `--trail-arm/--trail-dist/--set signals.*trail*` are hard errors). Module 22 tests
  (5) pin the override math, TP fill + spread, SL-first tie, PB-trail-disable, and CLI
  guards. Full suite: 346 green.
 - Re-open only if gold's regime turns and a *regime-conditional* exit (trail in range,
  target in trend) is proposed with a pre-registered mechanism — a plain fixed ratio is
  now a closed hypothesis at every multiple in {0.5, 1.0, 1.5, 2.0, 2.5, 3.0}.
@@ -1,40 +0,0 @@
 """Per-hour P&L breakdown of a backtest trade book.
 Usage:
    python backtest/research/hour_analysis.py BOOK.csv BAR_MINUTES
 Reports, per UTC hour (both entry hour and DECISION hour = entry_time
 minus one bar, the clock blocked_hours_utc evaluates): trade count,
 total R, mean R, win rate, profit factor. Also a per-(hour, direction)
 table. Analysis tooling only — never imported by the live code.
 """
 import sys
 import pandas as pd
 def hour_tables(book_csv: str, bar_minutes: int):
    df = pd.read_csv(book_csv)
    et = pd.to_datetime(df["entry_time"], utc=True, format="mixed")
    df["entry_hour"] = et.dt.hour
    df["decision_hour"] = (et - pd.Timedelta(minutes=bar_minutes)).dt.hour
    def table(group_cols):
        g = df.groupby(group_cols)["r_multiple"]
        out = g.agg(count="count", total_r="sum", mean_r="mean",
                    wr=lambda r: (r > 0).mean() * 100)
        out["pf"] = g.apply(
            lambda r: r[r > 0].sum() / max(-r[r <= 0].sum(), 1e-9))
        return out.round(2)
    return (table("decision_hour"), table(["decision_hour", "direction"]),
            table("entry_hour"))
 if __name__ == "__main__":
    book, mins = sys.argv[1], int(sys.argv[2])
    by_dec, by_dec_dir, by_ent = hour_tables(book, mins)
    print(f"\n── {book} · by DECISION hour (bar-close clock) ──")
    print(by_dec.to_string())
    print(f"\n── by (decision hour, direction) ──")
    print(by_dec_dir.to_string())
@@ -1,697 +0,0 @@
 """
 DeepSeek news-analysis validation harness — XAU/USD.
 ====================================================
 Research-only (no production code changes). Answers: is the DeepSeek
 layer's news analysis *correct*?
 Part A (events): replays every high-impact USD release with a genuine
 surprise (2024-01 → 2026-07, from backtest/cache/calendar.parquet)
 through the REAL `NewsEngine._llm_event_direction` — real prompt, real
 payload, real validation — and scores its "gold bullish/bearish" calls
 against gold's actual move (Oanda M15 mids) at +15m/30m/1h/4h/24h.
 Baselines: mechanical surprise-sign doctrine, FF actualBetterWorse,
 always-bullish/bearish (2024-26 is a gold bull regime), coin flip.
 Part B (headlines): a curated golden set (llm_news_golden.csv) through
 the REAL `NewsEngine._llm_headline_scores` in production-sized batches,
 vs the keyword scorer it overrides. Includes repeat-batch (determinism)
 and shuffled-order (cross-contamination) probes.
 Implementation notes that keep the replay faithful to production:
  * `_llm_event_direction`/`_llm_headline_scores` resolve the client via
    `get_llm_client()`, so we seed the module singleton with a research
    client (raised daily cap: 586 events > production cap 500).
  * The event cache key omits the forecast, so a repeat `actual` would
    silently re-serve a direction computed for a different forecast —
    the cache is cleared before every call.
  * A "neutral" reply and a hard failure both return None; they are
    disambiguated by reading the event cache back after the call
    (absent = failure, "" = neutral, "bullish:/bearish: …" = call).
  * Raw request/response bytes are captured with a requests response
    hook — the request-building code path is untouched.
 Usage (from repo root):
  venv/bin/python backtest/research/llm_news_eval.py events --limit 5
  venv/bin/python backtest/research/llm_news_eval.py events --determinism 20
  venv/bin/python backtest/research/llm_news_eval.py headlines
  venv/bin/python backtest/research/llm_news_eval.py report
 Artifacts land in backtest/research/llm_news_eval_2026-08/ (JSONL replay
 logs are resume-safe: completed events are never re-paid). `report` is a
 pure offline function of the artifacts.
 """
 import argparse
 import json
 import math
 import random
 import sys
 import time
 from datetime import datetime, timezone
 from pathlib import Path
 import numpy as np
 import pandas as pd
 REPO_ROOT = Path(__file__).resolve().parents[2]
 sys.path.insert(0, str(REPO_ROOT))
 import llm_client
 from app_config import deepseek_api_key, get_app_config
 from config import NewsConfig
 from llm_client import DEFAULT_MODEL, DeepSeekClient
 from news_engine import EconomicEvent, NewsEngine
 RESEARCH_DIR = Path(__file__).parent
 ART_DIR = RESEARCH_DIR / "llm_news_eval_2026-08"
 CALENDAR = REPO_ROOT / "backtest" / "cache" / "calendar.parquet"
 BARS = (REPO_ROOT / "backtest" / "cache" /
        "oanda_XAU_USD_15min_20210531_20260804.parquet")
 GOLDEN = RESEARCH_DIR / "llm_news_golden.csv"
 EVENTS_JSONL = ART_DIR / "events_replay.jsonl"
 HEADLINES_JSONL = ART_DIR / "headlines_replay.jsonl"
 SCORED_CSV = ART_DIR / "events_scored.csv"
 HORIZONS = [("15m", 15), ("30m", 30), ("1h", 60), ("4h", 240),
            ("24h", 1440)]
 NEUTRAL_BAND = 0.15  # |xau| below this = neutral prediction (Part B)
 BATCH_CAP = 25       # production batch cap (news_engine.py:783)
 EXPECTED_CORPUS = 586
 # ── shared plumbing ──────────────────────────────────────────────────
 class _Recorder:
    """requests response hook capturing the exact wire traffic."""
    def __init__(self):
        self.last = None
    def hook(self, resp, **kwargs):
        body = resp.request.body
        if isinstance(body, bytes):
            body = body.decode("utf-8", "replace")
        self.last = {
            "request_body": body,
            "http_status": resp.status_code,
            "response_body": resp.text,
            "elapsed_s": round(resp.elapsed.total_seconds(), 3),
        }
        return resp
    def pop(self):
        cap, self.last = self.last, None
        return cap or {}
 def install_research_client(timeout: float):
    """Build a research DeepSeekClient (raised cap) and seed the module
    singleton so the production code paths use it. Returns (client,
    recorder)."""
    app = get_app_config()
    key = deepseek_api_key(app)
    ds = app.deepseek_settings
    if not key:
        sys.exit("No DeepSeek API key (env DEEPSEEK_API_KEY or "
                 "config.json deepseek.api_key) — aborting before spend.")
    if ds.get("enabled") is False:
        sys.exit("deepseek.enabled is false in config.json — aborting.")
    client = DeepSeekClient(
        api_key=key,
        model=str(ds.get("model", DEFAULT_MODEL)),
        timeout=timeout,
        daily_cap=10 ** 6,  # research: corpus 586 > production cap 500
    )
    rec = _Recorder()
    client._session.hooks["response"].append(rec.hook)
    with llm_client._client_lock:
        llm_client._client = client
    return client, rec
 def _s(x):
    """Parquet str columns carry float NaN — EconomicEvent needs None."""
    return None if pd.isna(x) else str(x)
 def load_corpus():
    """High-impact USD releases with a genuine surprise, through the
    production `EconomicEvent.surprise` gate. Asserts run before any
    API spend."""
    df = pd.read_parquet(CALENDAR)
    df = df[(df["impactName"] == "high") & (df["currency"] == "USD")]
    rows = []
    for r in df.itertuples():
        ev = EconomicEvent(
            time=datetime.fromtimestamp(int(r.dateline), tz=timezone.utc),
            currency="USD", impact="high", event=_s(r.name) or "",
            forecast=_s(r.forecast), previous=_s(r.previous),
            actual=_s(r.actual),
        )
        if ev.surprise not in ("positive", "negative"):
            continue
        rows.append({
            "ffid": int(r.id), "dateline": int(r.dateline),
            "utc_time": ev.time.isoformat(), "name": ev.event,
            "actual": ev.actual, "forecast": ev.forecast,
            "previous": ev.previous, "surprise": ev.surprise,
            "better_worse": int(r.actualBetterWorse), "ev": ev,
        })
    ids = [d["ffid"] for d in rows]
    if len(rows) != EXPECTED_CORPUS:
        sys.exit(f"corpus size {len(rows)} != {EXPECTED_CORPUS} — "
                 "calendar.parquet changed; re-verify before spending.")
    if len(set(ids)) != len(ids):
        sys.exit("FF ids not unique — resume keying would break.")
    # Pin the actualBetterWorse mapping on two unambiguous NFP prints.
    nfp = {d["utc_time"][:10]: d["better_worse"] for d in rows
           if d["name"] == "Non-Farm Employment Change"}
    if nfp.get("2024-01-05") != 1 or nfp.get("2024-05-03") != 2:
        sys.exit("actualBetterWorse mapping pins failed (expected "
                 "1=better-for-USD on 2024-01-05 beat, 2=worse on "
                 "2024-05-03 miss).")
    rows.sort(key=lambda d: (d["dateline"], d["ffid"]))
    return rows
 def _load_jsonl(path: Path):
    recs = []
    if path.exists():
        with open(path) as f:
            for line in f:
                line = line.strip()
                if line:
                    recs.append(json.loads(line))
    return recs
 # ── Part A: event replay ─────────────────────────────────────────────
 def _extract_reason(cap):
    try:
        content = json.loads(cap["response_body"])[
            "choices"][0]["message"]["content"]
        return json.loads(content).get("reason")
    except Exception:
        return None
 def cmd_events(args):
    ART_DIR.mkdir(exist_ok=True)
    corpus = load_corpus()
    if args.limit:
        corpus = corpus[:args.limit]
    engine = NewsEngine(NewsConfig())
    existing = _load_jsonl(EVENTS_JSONL)
    done1 = {r["ffid"] for r in existing
             if r.get("run_pass") == 1 and r.get("llm_direction")}
    done2 = {r["ffid"] for r in existing
             if r.get("run_pass") == 2 and r.get("llm_direction")}
    pending = [d for d in corpus if d["ffid"] not in done1]
    print(f"corpus {len(corpus)} | already recorded {len(done1)} | "
          f"pending {len(pending)} | determinism re-calls "
          f"{args.determinism} | model calls are ~100 tokens each "
          f"(deepseek-v4-flash, static-prefix cached) → cost ≈ cents")
    if not pending and not args.determinism:
        print("nothing to do")
        return
    _, rec = install_research_client(args.timeout)
    out = open(EVENTS_JSONL, "a")
    def call_one(d, run_pass):
        # Cache key omits forecast → cleared per call so 196 colliding
        # corpus rows can't be served a stale direction.
        with NewsEngine._cache_lock:
            NewsEngine._llm_event_cache.clear()
        ev = d["ev"]
        engine._llm_event_direction(ev)
        key = (ev.event, ev.currency, str(ev.actual or ""))
        with NewsEngine._cache_lock:
            cached = NewsEngine._llm_event_cache.get(key)
        cap = rec.pop()
        if cached is None:
            direction, text = None, None       # hard failure → retry
        elif cached == "":
            direction, text = "neutral", ""
        else:
            direction, text = cached.split(":", 1)[0], cached
        recd = {k: d[k] for k in (
            "ffid", "dateline", "utc_time", "name", "actual", "forecast",
            "previous", "surprise", "better_worse")}
        recd.update({
            "llm_direction": direction, "llm_text": text,
            "reason_raw": _extract_reason(cap), "run_pass": run_pass,
            "ts": datetime.now(timezone.utc).isoformat(), **cap,
        })
        out.write(json.dumps(recd) + "\n")
        out.flush()
        return direction
    remaining = pending
    for npass in range(1, args.retry_passes + 1):
        if not remaining:
            break
        failed = []
        for i, d in enumerate(remaining):
            if call_one(d, 1) is None:
                failed.append(d)
            if args.sleep:
                time.sleep(args.sleep)
            if (i + 1) % 25 == 0:
                print(f"  pass {npass}: {i + 1}/{len(remaining)} "
                      f"({len(failed)} failed)")
        print(f"pass {npass} done: {len(remaining) - len(failed)} ok, "
              f"{len(failed)} failed")
        remaining = failed
    if remaining:
        print(f"WARNING: {len(remaining)} events still unrecorded after "
              f"{args.retry_passes} passes — re-run `events` to resume.")
    if args.determinism:
        ok_ids = {x["ffid"] for x in remaining}
        done_now = [d for d in corpus if d["ffid"] not in ok_ids]
        done_now.sort(key=lambda d: (d["name"], d["dateline"]))
        stride = max(1, len(done_now) // args.determinism)
        sample = done_now[::stride][:args.determinism]
        todo = [d for d in sample if d["ffid"] not in done2]
        print(f"determinism probe: {len(todo)} re-calls "
              f"({len(sample) - len(todo)} already recorded)")
        for d in todo:
            call_one(d, 2)
            if args.sleep:
                time.sleep(args.sleep)
    out.close()
    print("events replay complete →", EVENTS_JSONL)
 # ── Part B: golden headlines ─────────────────────────────────────────
 def _load_golden():
    g = pd.read_csv(GOLDEN)
    need = {"headline", "category", "expected_xau_sign", "rationale"}
    if not need <= set(g.columns):
        sys.exit(f"golden CSV missing columns {need - set(g.columns)}")
    if not g["expected_xau_sign"].isin([-1, 0, 1]).all():
        sys.exit("expected_xau_sign must be -1/0/+1")
    if not g["headline"].is_unique:
        sys.exit("duplicate headlines in golden CSV")
    return g
 def cmd_headlines(args):
    ART_DIR.mkdir(exist_ok=True)
    g = _load_golden()
    heads = list(g["headline"])
    _, rec = install_research_client(args.timeout)
    engine = NewsEngine(NewsConfig())
    out = open(HEADLINES_JSONL, "w")  # tiny (~9 calls): fresh each run
    def run_batch(texts, tag, batch_idx):
        scores = None
        for attempt in range(3):
            scores = engine._llm_headline_scores(texts)
            cap = rec.pop()
            if scores is not None:
                break
            print(f"  {tag} batch {batch_idx} attempt {attempt + 1} "
                  f"failed (http {cap.get('http_status')})")
            time.sleep(2.0)
        out.write(json.dumps({
            "run_tag": tag, "batch": batch_idx, "headlines": texts,
            "scores": scores,
            "ts": datetime.now(timezone.utc).isoformat(), **cap,
        }) + "\n")
        out.flush()
        return scores
    batches = [heads[i:i + BATCH_CAP] for i in range(0, len(heads),
                                                     BATCH_CAP)]
    for bi, b in enumerate(batches):
        ok = run_batch(b, "main", bi) is not None
        print(f"main batch {bi} ({len(b)} headlines): "
              f"{'ok' if ok else 'FAILED'}")
        time.sleep(args.sleep)
    run_batch(batches[0], "repeat", 0)
    print("repeat probe done")
    shuffled = heads[:]
    random.Random(20260806).shuffle(shuffled)
    for bi in range(0, len(shuffled), BATCH_CAP):
        run_batch(shuffled[bi:bi + BATCH_CAP], "shuffled",
                  bi // BATCH_CAP)
    print("shuffled probe done →", HEADLINES_JSONL)
    out.close()
 # ── report: pure offline function of the artifacts ───────────────────
 def _binom_p(k, n):
    """Exact two-sided binomial p-value vs p=0.5 (no scipy)."""
    if n == 0:
        return float("nan")
    dist = [math.comb(n, i) for i in range(n + 1)]
    tot = float(2 ** n)
    pk = dist[k]
    return min(1.0, sum(c for c in dist if c <= pk) / tot)
 def _mcnemar_p(hit_a, hit_b):
    """Exact McNemar via binomial on discordant pairs. Inputs: aligned
    boolean lists."""
    b = sum(1 for x, y in zip(hit_a, hit_b) if x and not y)
    c = sum(1 for x, y in zip(hit_a, hit_b) if not x and y)
    return b, c, _binom_p(min(b, c), b + c)
 def _latest_events(recs):
    """Last pass-1 record per ffid (retries append; last wins)."""
    out = {}
    for r in recs:
        if r.get("run_pass") == 1:
            out[r["ffid"]] = r
    return list(out.values())
 def _fmt_rate(hits, n, label=""):
    if n == 0:
        return f"{label:<24} n=0"
    p = _binom_p(hits, n)
    return (f"{label:<24} {hits:>4}/{n:<4} = {hits / n * 100:5.1f}%  "
            f"(binom p={p:.3f})")
 def report_events():
    recs = _latest_events(_load_jsonl(EVENTS_JSONL))
    if not recs:
        print("no event replay records — run `events` first")
        return
    bars = pd.read_parquet(BARS)
    closes = bars["Close"].to_numpy()
    close_ns = (bars.index + pd.Timedelta(minutes=15)).asi8
    def anchor(t_ns, h_min):
        i = np.searchsorted(close_ns, t_ns, side="right") - 1
        if i < 0:
            return None
        stale_ns = max(45.0, h_min / 4.0) * 60e9
        if t_ns - close_ns[i] > stale_ns:
            return None
        return closes[i]
    rows, skip = [], {h: 0 for h, _ in HORIZONS}
    base_skipped = 0
    for r in recs:
        t_ns = int(r["dateline"]) * 10 ** 9
        base = anchor(t_ns, 0)
        if base is None:
            base_skipped += 1
            continue
        row = dict(r)
        row.pop("request_body", None)
        row.pop("response_body", None)
        row["year"] = r["utc_time"][:4]
        row["base"] = base
        for hname, hmin in HORIZONS:
            p = anchor(t_ns + hmin * 60 * 10 ** 9, hmin)
            if p is None:
                skip[hname] += 1
                row[f"d_{hname}"] = np.nan
            else:
                row[f"d_{hname}"] = p - base
        rows.append(row)
    df = pd.DataFrame(rows)
    df.to_csv(SCORED_CSV, index=False)
    n_fail = int((df["llm_direction"].isna()).sum())
    print("═" * 70)
    print("PART A — event gold-direction vs actual XAU move")
    print("═" * 70)
    print(f"events replayed: {len(df)} (baseline-price skipped: "
          f"{base_skipped}) | LLM hard failures: {n_fail}")
    print("horizon skip counts (stale/closed):",
          {k: v for k, v in skip.items() if v})
    dist = df["llm_direction"].value_counts(dropna=False).to_dict()
    print("LLM call distribution:", dist)
    # Unconditional per-horizon noise scale (null): median |Δ| over all
    # 2024+ bar times, position-shifted.
    bars24 = bars[bars.index >= "2024-01-01"]
    null_med = {}
    for hname, hmin in HORIZONS:
        k = hmin // 15
        d = (bars24["Close"].shift(-k) - bars24["Close"]).abs().dropna()
        null_med[hname] = float(d.median())
    print("unconditional null median |Δ| ($):",
          {k: round(v, 2) for k, v in null_med.items()})
    # Strategy call columns: +1 gold up, −1 gold down, 0/NaN excluded.
    df["c_llm"] = df["llm_direction"].map(
        {"bullish": 1, "bearish": -1}).fillna(0)
    df["c_doctrine"] = df["surprise"].map({"positive": -1, "negative": 1})
    df["c_ffbw"] = df["better_worse"].map({1: -1, 2: 1}).fillna(0)
    df["c_bull"] = 1
    df["c_bear"] = -1
    strategies = [("LLM", "c_llm"), ("doctrine(surprise)", "c_doctrine"),
                  ("FF betterWorse", "c_ffbw"), ("always-bullish", "c_bull"),
                  ("always-bearish", "c_bear")]
    for hname, _ in HORIZONS:
        dcol = f"d_{hname}"
        sub = df[df[dcol].notna()]
        print(f"\n── horizon {hname} (n scored bars={len(sub)}) ──")
        zeros = int((sub[dcol] == 0).sum())
        if zeros:
            print(f"  Δ==0 count (scored as miss): {zeros}")
        for label, col in strategies:
            s = sub[sub[col] != 0]
            hits = int((np.sign(s[dcol]) == s[col]).sum())
            print("  " + _fmt_rate(hits, len(s), label))
        # noise-filtered LLM hit rate
        s = sub[(sub["c_llm"] != 0) &
                (sub[dcol].abs() >= null_med[hname])]
        hits = int((np.sign(s[dcol]) == s["c_llm"]).sum())
        print("  " + _fmt_rate(hits, len(s),
                               f"LLM |Δ|≥null-med"))
        # event-window magnitude vs null
        mag = sub[dcol].abs().median()
        print(f"  event-window median |Δ| = ${mag:.2f} "
              f"(null ${null_med[hname]:.2f})")
        # paired McNemar: LLM vs each baseline on common directional set
        for label, col in [("doctrine", "c_doctrine"),
                           ("FF betterWorse", "c_ffbw"),
                           ("always-bullish", "c_bull")]:
            s = sub[(sub["c_llm"] != 0) & (sub[col] != 0)]
            ha = list(np.sign(s[dcol]) == s["c_llm"])
            hb = list(np.sign(s[dcol]) == s[col])
            b, c, p = _mcnemar_p(ha, hb)
            print(f"  McNemar LLM vs {label:<15} b={b:>3} c={c:>3} "
                  f"p={p:.3f}")
    # neutral-call flatness (15m)
    neu = df[(df["llm_direction"] == "neutral") & df["d_15m"].notna()]
    dire = df[(df["c_llm"] != 0) & df["d_15m"].notna()]
    if len(neu):
        print(f"\nneutral calls (n={len(neu)}): median |Δ_15m| "
              f"${neu['d_15m'].abs().median():.2f} vs directional "
              f"(n={len(dire)}) ${dire['d_15m'].abs().median():.2f} "
              f"(null ${null_med['15m']:.2f})")
    # breakdowns at 1h
    dcol = "d_1h"
    sub = df[df[dcol].notna() & (df["c_llm"] != 0)]
    print("\n── LLM hit rate by event name (1h, top 10 by count) ──")
    for name, grp in sorted(sub.groupby("name"), key=lambda kv: -len(kv[1]))[:10]:
        hits = int((np.sign(grp[dcol]) == grp["c_llm"]).sum())
        print("  " + _fmt_rate(hits, len(grp), name[:24]))
    print("── LLM hit rate by year (1h) ──")
    for yr, grp in sub.groupby("year"):
        hits = int((np.sign(grp[dcol]) == grp["c_llm"]).sum())
        print("  " + _fmt_rate(hits, len(grp), yr))
    # cluster view: shared datelines
    print("\n── same-timestamp clusters ──")
    clus = df.groupby("dateline").filter(lambda g: len(g) > 1)
    groups = list(clus.groupby("dateline"))
    consistent = 0
    net_hits, net_n = 0, 0
    for _, g in groups:
        signs = set(g.loc[g["c_llm"] != 0, "c_llm"])
        if len(signs) <= 1:
            consistent += 1
        net = int(g["c_llm"].sum())
        d = g["d_1h"].iloc[0]
        if net != 0 and pd.notna(d) and d != 0:
            net_n += 1
            if np.sign(d) == np.sign(net):
                net_hits += 1
    print(f"clusters: {len(groups)} datelines "
          f"({len(clus)} events, {len(clus) / max(len(df), 1) * 100:.0f}% of "
          f"corpus) | internally consistent: {consistent}/{len(groups)}")
    print("  " + _fmt_rate(net_hits, net_n, "cluster net-call (1h)"))
    # determinism probe
    p2 = {r["ffid"]: r for r in _load_jsonl(EVENTS_JSONL)
          if r.get("run_pass") == 2 and r.get("llm_direction")}
    if p2:
        p1 = {r["ffid"]: r for r in recs}
        agree = same_reason = n = 0
        for fid, r2 in p2.items():
            r1 = p1.get(fid)
            if not r1 or not r1.get("llm_direction"):
                continue
            n += 1
            if r1["llm_direction"] == r2["llm_direction"]:
                agree += 1
            if r1.get("reason_raw") == r2.get("reason_raw"):
                same_reason += 1
        print(f"\ndeterminism probe: direction agreement {agree}/{n}, "
              f"identical reason text {same_reason}/{n}")
    # stratified reason examples
    print("\n── reason samples (1h hit/miss) ──")
    sub = df[df[dcol].notna() & (df["c_llm"] != 0)].copy()
    sub["hit"] = np.sign(sub[dcol]) == sub["c_llm"]
    for verdict in (True, False):
        picked, seen = [], set()
        for _, r in sub[sub["hit"] == verdict].iterrows():
            if r["name"] in seen:
                continue
            seen.add(r["name"])
            picked.append(r)
            if len(picked) == 10:
                break
        tag = "HIT " if verdict else "MISS"
        for r in picked:
            print(f"  [{tag}] {r['utc_time'][:16]} {r['name'][:30]:<30} "
                  f"a={r['actual']} f={r['forecast']} → "
                  f"{r['llm_direction']:<7} Δ1h={r[dcol]:+7.2f}  "
                  f"“{(r['reason_raw'] or '')[:60]}”")
    print(f"\nscored table → {SCORED_CSV}")
 def report_headlines():
    recs = _load_jsonl(HEADLINES_JSONL)
    if not recs:
        print("\n(no headline replay records — run `headlines` first)")
        return
    g = _load_golden()
    engine = NewsEngine(NewsConfig())
    def collect(tag):
        m = {}
        for r in recs:
            if r["run_tag"] == tag and r["scores"]:
                for h, sc in zip(r["headlines"], r["scores"]):
                    m[h] = tuple(sc)
        return m
    main, shuf = collect("main"), collect("shuffled")
    print("\n" + "═" * 70)
    print("PART B — golden headline set: LLM vs keyword baseline")
    print("═" * 70)
    missing = [h for h in g["headline"] if h not in main]
    if missing:
        print(f"WARNING: {len(missing)} headlines missing from main run")
    def cls(x):
        if x >= NEUTRAL_BAND:
            return 1
        if x <= -NEUTRAL_BAND:
            return -1
        return 0
    rows = []
    for _, r in g.iterrows():
        h = r["headline"]
        if h not in main:
            continue
        llm_xau = main[h][1]
        kw = engine._analyze_headline_sentiment(h, for_currency="XAU")
        rows.append({
            "headline": h, "category": r["category"],
            "exp": int(r["expected_xau_sign"]),
            "llm_xau": llm_xau, "llm_cls": cls(llm_xau),
            "kw": kw, "kw_cls": cls(kw),
        })
    t = pd.DataFrame(rows)
    for who in ("llm", "kw"):
        acc = (t[f"{who}_cls"] == t["exp"]).mean()
        print(f"{who.upper():<4} 3-class accuracy (band ±{NEUTRAL_BAND}): "
              f"{(t[f'{who}_cls'] == t['exp']).sum()}/{len(t)} "
              f"= {acc * 100:.1f}%")
    ha = list(t["llm_cls"] == t["exp"])
    hb = list(t["kw_cls"] == t["exp"])
    b, c, p = _mcnemar_p(ha, hb)
    print(f"paired McNemar LLM vs keyword: b={b} c={c} p={p:.3f}")
    print("\nper-category accuracy (LLM | keyword):")
    for cat, grp in t.groupby("category"):
        a1 = (grp["llm_cls"] == grp["exp"]).sum()
        a2 = (grp["kw_cls"] == grp["exp"]).sum()
        print(f"  {cat:<18} {a1:>2}/{len(grp):<2} | {a2:>2}/{len(grp)}")
    print("\nLLM confusion (rows=expected, cols=predicted -1/0/+1):")
    cm = pd.crosstab(t["exp"], t["llm_cls"]).reindex(
        index=[-1, 0, 1], columns=[-1, 0, 1], fill_value=0)
    print(cm.to_string())
    print("\nLLM misclassified headlines:")
    for _, r in t[t["llm_cls"] != t["exp"]].iterrows():
        print(f"  exp {r['exp']:+d} got {r['llm_xau']:+.2f} "
              f"[{r['category']}] {r['headline'][:70]}")
    # repeat-batch determinism
    m_b0 = [r for r in recs if r["run_tag"] == "main" and r["batch"] == 0]
    r_b0 = [r for r in recs if r["run_tag"] == "repeat"]
    if m_b0 and r_b0 and m_b0[0]["scores"] and r_b0[0]["scores"]:
        a = np.array(m_b0[0]["scores"], dtype=float)
        b_ = np.array(r_b0[0]["scores"], dtype=float)
        exact = int((a == b_).all(axis=1).sum())
        print(f"\nrepeat-batch: {exact}/{len(a)} score pairs identical, "
              f"max |diff| = {np.abs(a - b_).max():.3f}")
    # shuffled-order drift
    if shuf:
        both = [h for h in t["headline"] if h in shuf]
        drift = [abs(main[h][1] - shuf[h][1]) for h in both]
        if drift:
            print(f"shuffled-order xau drift over {len(both)} headlines: "
                  f"mean {np.mean(drift):.3f}, max {np.max(drift):.3f}")
 def cmd_report(args):
    report_events()
    report_headlines()
 # ── CLI ──────────────────────────────────────────────────────────────
 def main():
    ap = argparse.ArgumentParser(description=__doc__.split("\n")[1])
    sub = ap.add_subparsers(dest="cmd", required=True)
    e = sub.add_parser("events", help="replay release surprises")
    e.add_argument("--limit", type=int, default=0,
                   help="smoke-test on first N events")
    e.add_argument("--determinism", type=int, default=0,
                   help="re-call N stratified events a second time")
    e.add_argument("--sleep", type=float, default=0.3)
    e.add_argument("--retry-passes", type=int, default=3)
    e.add_argument("--timeout", type=float, default=30.0)
    e.set_defaults(fn=cmd_events)
    h = sub.add_parser("headlines", help="golden-set probes")
    h.add_argument("--sleep", type=float, default=0.3)
    h.add_argument("--timeout", type=float, default=30.0)
    h.set_defaults(fn=cmd_headlines)
    r = sub.add_parser("report", help="offline scoring report")
    r.set_defaults(fn=cmd_report)
    args = ap.parse_args()
    args.fn(args)
 if __name__ == "__main__":
    main()
@@ -1,86 +0,0 @@
 # DeepSeek news-analysis validation — 2026-08-06
 **Verdict: headline sentiment scoring is excellent; event-direction calls are
 directionally sound but ~11% of them misread the numeric comparison, and the
 failure concentrates on Unemployment Claims / close K-M-suffixed values.**
 Method (owner-directed, token-lean): independent expert re-analysis of a
 stratified sample of DeepSeek's recorded answers, not a full statistical
 backtest. Harness: `backtest/research/llm_news_eval.py` (replays releases
 through the REAL `_llm_event_direction` / `_llm_headline_scores` — real
 prompts, payloads, validation; raw wire traffic logged). 135 of the 586-event
 corpus were replayed before the full run was stopped by owner request
 (artifacts retained in `llm_news_eval_2026-08/`, resume-safe).
 ## Part A — event gold-direction (28-event stratified sample, Jan–Jun 2024)
 Reviewer agreement with DeepSeek: **24/28 (86%)**. All 4 disagreements are
 DeepSeek errors, in exactly two failure modes:
 1. **Numeric comparison misread (3/28 ≈ 11%)** — the doctrine logic is right
   but the premise is backwards:
   - JOLTS 8.79M vs f 8.84M (a miss) → DS: "Job openings **above** forecast" → bearish (should be bullish)
   - Claims 202K vs f 217K (fewer claims) → DS: "Claims **higher** than forecast" → bullish (should be bearish)
   - Claims 202K vs f 209K — same backwards read, same wrong call
   Both Unemployment Claims events in the sample were misread the same way —
   looks systematic on that event type, possibly an anchoring prior that
   "claims rose". All misreads involve close values with K/M suffixes.
 2. **Direction field contradicts its own reason (1/28)** — Empire State
   −43.7 vs f −4.9: reason text says "gold may rise", `gold_direction` says
   `bearish`.
 Where the numbers were read correctly, hawkish/dovish→gold mapping was
 consistently correct (CPI/PPI/NFP/ADP/GDP/PMI/wages/unemployment-rate all
 handled properly, including inverted metrics when read right).
 Price cross-check (1h horizon, cluster-confounded, indicative only):
 reviewer calls 22/28 with the subsequent move, DeepSeek 20/28.
 ## Part B — headline sentiment (78-headline golden set)
 DeepSeek matched the reviewer's expected gold-direction label on
 **78/78 (100%)** — including all 8 negation traps ("signals no rate hike",
 "no rate cuts until…") and all 6 inverted-metric phrasings ("claims hit
 two-year high"). The keyword scorer it overrides got **24/78 (31%)**.
 Repeat-batch and shuffled-order probes recorded (headlines_replay.jsonl).
 ## Implications
 - The headline-sentiment override (the main live LLM surface) is validated —
  a large, measured upgrade over keywords.
 - The event-direction enrichment ("— gold bullish/bearish" appended to
  surprise descriptions) is right ~85–90% of the time but can state a
  factually backwards comparison in the displayed reason. It is display-only
  (never gates signals), so exposure is a possibly-misleading Telegram/CLI
  line, not trades.
 - **Hardening SHIPPED same day (owner-approved):** the payload now carries a
  code-computed `vs_forecast` field ("above/below/in line with forecast",
  from `EconomicEvent.surprise`) and the prompt tells the model to trust it.
  First iteration fixed the arithmetic but exposed a masked doctrine error
  (model read "claims below forecast" correctly, then called it a *weaker*
  labor market — and regressed the unemployment-rate case the same way), so
  the prompt's inverted-metrics line was expanded to an explicit rule plus a
  worked claims example. Verification: **18/18** — all 4 original misreads
  fixed, 11/11 no-regression cases unchanged, 3/3 fresh probes on the side
  the prompt example does NOT cover (claims 245K vs f 220K → bullish,
  unemployment 4.1% vs 3.9% → bullish, JOLTS 9.10M vs 8.80M → bearish).
  Suite green at 346 tests, incl. a new payload assertion in Module 19.
 - Production observations found while building the harness (not fixed):
  `_llm_event_cache` key omits forecast → a repeated `actual` re-serves a
  direction computed for a different forecast; a "neutral" reply is
  indistinguishable from a failure in the return value.
 ## Reproduce
 ```
 venv/bin/python backtest/research/llm_news_eval.py events --limit 5   # smoke
 venv/bin/python backtest/research/llm_news_eval.py events --determinism 20
 venv/bin/python backtest/research/llm_news_eval.py headlines
 venv/bin/python backtest/research/llm_news_eval.py report             # offline
 ```
 Corpus: 586 high-impact USD surprises 2024-01→2026-07 (calendar.parquet,
 NFP-verified UTC); prices: Oanda M15 mids. Full-run cost ≈ cents. The
 `report` subcommand (hit rates vs always-bullish/doctrine/FF-betterWorse
 baselines, McNemar, cluster view) is built and untested at scale — run it
 after completing the replay if a statistical answer is ever wanted.
@@ -1,79 +0,0 @@
 headline,category,expected_xau_sign,rationale
 "Powell says restrictive policy needed for longer as inflation proves sticky",hawkish_fed,-1,"higher-for-longer rates lift real yields, gold-bearish"
 "Fed minutes show officials leaning toward two more rate hikes this year",hawkish_fed,-1,"more hikes = stronger dollar/yields"
 "Hot CPI print fuels bets on additional Fed tightening",hawkish_fed,-1,"tightening expectations weigh on gold"
 "Fed's Waller: higher-for-longer rates warranted by resilient economy",hawkish_fed,-1,"hawkish guidance"
 "FOMC raises rates 50bp, flags further increases ahead",hawkish_fed,-1,"outsized hike plus hawkish guidance"
 "Treasury yields spike as Fed pushes back on rate cut hopes",hawkish_fed,-1,"cut hopes dashed, yields up"
 "Core inflation accelerates, cementing case for hawkish Fed stance",hawkish_fed,-1,"hawkish repricing"
 "Fed dot plot shifts higher; markets price out easing for the year",hawkish_fed,-1,"easing priced out"
 "Fed cuts rates 25bp and signals more easing to come",dovish_fed,1,"easing cycle is gold-bullish"
 "Powell hints at rate cuts as inflation cools toward target",dovish_fed,1,"cut guidance"
 "FOMC minutes reveal growing support for policy easing",dovish_fed,1,"dovish minutes"
 "Fed's Williams says rate cuts appropriate later this year",dovish_fed,1,"cut guidance"
 "Soft inflation data boosts odds of a dovish Fed pivot",dovish_fed,1,"pivot odds up"
 "Fed pauses hikes, markets price aggressive easing cycle",dovish_fed,1,"easing priced in"
 "Central banks turn dovish as global growth falters",dovish_fed,1,"global easing backdrop"
 "Fed to restart bond purchases to support flagging economy",dovish_fed,1,"QE is gold-bullish"
 "Dollar index hits 12-month high on robust US economic data",strong_usd,-1,"strong dollar pressures gold"
 "US GDP grows 4.1% annualized, blowing past expectations",strong_usd,-1,"strong economy, hawkish implication"
 "Blockbuster jobs report sends dollar surging across the board",strong_usd,-1,"strong jobs lift dollar and yields"
 "US retail sales smash forecasts, underscoring consumer strength",strong_usd,-1,"strong data"
 "Greenback rallies as US data outshines Europe and Asia",strong_usd,-1,"dollar strength"
 "ISM services index jumps to two-year high on strong demand",strong_usd,-1,"strong data"
 "Consumer confidence surges to post-pandemic high in the US",strong_usd,-1,"strong data"
 "Dollar tumbles to eight-month low after dismal manufacturing data",weak_usd,1,"weak dollar lifts gold"
 "US economy adds far fewer jobs than expected; dollar slides",weak_usd,1,"weak jobs, dovish implication"
 "Greenback weakens as US retail sales unexpectedly contract",weak_usd,1,"weak data"
 "US GDP growth revised down sharply, stoking slowdown fears",weak_usd,1,"slowdown is dovish"
 "Dollar slumps as trade deficit balloons to record",weak_usd,1,"dollar weakness"
 "Soft services PMI adds to evidence of cooling US economy",weak_usd,1,"cooling economy"
 "US consumer sentiment plunges to two-year low",weak_usd,1,"weak data"
 "Real yields climb to multi-year highs as TIPS sell off",real_yields,-1,"rising real yields raise gold's opportunity cost"
 "10-year Treasury yield surges past 5% on supply glut",real_yields,-1,"yield spike"
 "Rising real interest rates dim appeal of non-yielding assets",real_yields,-1,"explicit opportunity-cost story"
 "Treasury yields sink as investors flock to bonds",real_yields,1,"falling yields support gold"
 "Real yields turn negative as inflation outpaces rates",real_yields,1,"negative real yields are gold's best backdrop"
 "Bond rally drives 10-year yield to lowest level in a year",real_yields,1,"falling yields"
 "Geopolitical tensions escalate in Middle East, investors seek safety",safe_haven,1,"safe-haven bid"
 "Stocks crater worldwide as banking crisis fears spread",safe_haven,1,"crisis flight to safety"
 "Escalating conflict drives flight to safe-haven assets",safe_haven,1,"explicit haven flow"
 "Investors dump risk assets after surprise sovereign default",safe_haven,1,"default shock"
 "Military escalation on NATO border rattles global markets",safe_haven,1,"geopolitical stress"
 "Wall Street plunges 4% in worst session since pandemic",safe_haven,1,"risk-off"
 "Currency crisis in emerging markets sparks contagion fears",safe_haven,1,"contagion, haven bid"
 "Nuclear rhetoric intensifies, sending investors to havens",safe_haven,1,"extreme geopolitical stress"
 "Ceasefire agreement lifts global markets, haven demand fades",risk_on,-1,"haven unwind"
 "Stocks roar to record highs as trade deal signed",risk_on,-1,"risk-on drains haven demand"
 "Risk appetite returns as banking fears subside",risk_on,-1,"risk-on"
 "Global equities rally on strong earnings and easing tensions",risk_on,-1,"risk-on"
 "Gold hits record high as central banks keep buying bullion",gold_direct,1,"direct demand"
 "Gold ETFs post biggest weekly inflows in three years",gold_direct,1,"investment inflows"
 "China's central bank extends gold buying spree to 18th month",gold_direct,1,"official-sector demand"
 "Miners struggle to meet gold demand as supply deficit widens",gold_direct,1,"supply deficit"
 "Gold slides 2% as investors book profits after record run",gold_direct,-1,"profit-taking pressure"
 "Gold tumbles below key support as ETF outflows accelerate",gold_direct,-1,"outflows"
 "Bullion demand weakens as Indian jewellery buying stalls",gold_direct,-1,"physical demand soft"
 "Speculators slash bullish gold bets as dollar strengthens",gold_direct,-1,"positioning unwind"
 "Fed holds rates steady, signals no rate hike on the horizon",negation_traps,1,"dovish hold; keyword layer sees the phrase rate hike and flips it gold-bearish"
 "Powell says no rate cuts until inflation shows sustained cooling",negation_traps,-1,"hawkish hold; keyword layer sees rate cut and calls it gold-bullish"
 "Recession fears fade as US data streak beats estimates",negation_traps,-1,"fading fear is risk-on; keywords see recession plus beats and cancel out"
 "Inflation fails to accelerate, easing pressure on the Fed",negation_traps,1,"disinflation opens door to cuts"
 "Fed's Bowman: not yet confident inflation is beaten, cuts premature",negation_traps,-1,"hawkish pushback phrased around negations"
 "Central bank stops short of hiking, but warns tightening may resume",negation_traps,-1,"hawkish hold with tightening bias"
 "No safe-haven bid for gold despite equity wobble, analysts say",negation_traps,-1,"explicitly negated haven demand"
 "Jobs report not as weak as feared, dollar recovers",negation_traps,-1,"less-bad data is dollar-positive"
 "US manufacturing output plunges as new orders dry up",inverted_metrics,1,"weak economy is dovish/gold-bullish; keyword layer reads plunges as bearish"
 "US factory activity slumps to three-year low",inverted_metrics,1,"weak data; keyword reads slumps as bearish"
 "Consumer spending falls for third straight month",inverted_metrics,1,"weak data; keyword reads falls as bearish"
 "Private payrolls growth slides to weakest pace since 2020",inverted_metrics,1,"weak jobs; keyword reads slides as bearish"
 "Jobless claims hit two-year high as layoffs spread",inverted_metrics,1,"higher claims = weaker economy = gold-bullish"
 "US unemployment rate hits 4.5%, highest since 2021",inverted_metrics,1,"higher unemployment = dovish = gold-bullish"
 "Lakers clinch playoff berth with overtime win over Suns",distractors,0,"irrelevant"
 "New Marvel film smashes box office records on opening weekend",distractors,0,"irrelevant"
 "Apple unveils thinner MacBook with faster chip lineup",distractors,0,"irrelevant"
 "Scientists discover new deep-sea species off Pacific coast",distractors,0,"irrelevant"
 "Heavy rain forecast for the Midwest through the weekend",distractors,0,"irrelevant"
 "Streaming service raises subscription prices in Europe",distractors,0,"irrelevant"
 "Electric vehicle startup recalls 12,000 cars over software glitch",distractors,0,"irrelevant"
 "Local elections see record turnout in several US states",distractors,0,"irrelevant"
@@ -7,7 +7,6 @@
  },
  "data_providers": {
    "active": "oanda",
    "streaming": "tiingo",
    "twelve_data": {
      "api_key": "REPLACE_WITH_YOUR_TWELVE_DATA_API_KEY"
    },
@@ -33,12 +32,5 @@
    "quality_gate": "C",
    "max_hold_bars": 300
  },
  "deepseek": {
    "api_key": "REPLACE_WITH_YOUR_DEEPSEEK_API_KEY_OR_EMPTY",
    "model": "deepseek-v4-flash",
    "enabled": true,
    "timeout_seconds": 12,
    "daily_request_cap": 500
  },
  "log_level": "INFO"
 }
@@ -46,12 +46,6 @@ class TechnicalConfig:
    ema_medium: int = 21
    ema_slow: int = 50
    ema_trend: int = 200
    # 4-EMA mode only: score partial ribbon alignment (2-3 of 4 checks)
    # as NEUTRAL instead of a weak BUY/SELL. Research knob (2026-08-04
    # indicator audit: partial-alignment trades averaged -0.008R vs
    # +0.230R fully aligned on the M5 12mo Oanda book); default False =
    # legacy behavior until a replay campaign validates flipping it.
    ema_partial_neutral: bool = False
    rsi_period: int = 14
    rsi_overbought: float = 70.0
@@ -152,7 +146,7 @@ class NewsConfig:
    # against the validated 12mo XAU M5 book (547 trades, +139.9R)
    # showed they suppress 25-35 trades worth +4.6..+7.2R/yr — trades
    # entered near USD releases OUTPERFORM (+0.475R vs +0.256R avg).
-    # Audit 2026-07-11 (calendar-replay study; tooling since removed).
+    # Audit 2026-07-11 via backtest/research/calendar_store.
    event_risk_caps_signals: bool = False
    # Whether headline sentiment moves the SIGNAL (aligned: +5 confidence;
    # conflicting: 10-point penalty). Warnings are always emitted
@@ -209,16 +203,9 @@ class XauStrategyConfig:
    # M5 signals are noisy; require stronger confirmation before
    # generating trade setups. Without these, M5 is breakeven
    # (+0.01R/trade) with 21% drawdown.
    # Relaxed 2026-07-30 (30→25, 10→5) in the relaxation campaign:
    # the combined config (with the M5 hour-21 unblock) beat control
    # in ALL 6 calendar years of the 5yr Oanda replay (+230.9R vs
    # +168.0R, PF 1.26 vs 1.21, max DD 21.3 vs 27.0), 12mo +107.4R
    # vs +84.6R PF 1.36 both halves positive, OOS 2024-06→2025-03
    # +26.1R vs +17.2R, and at $0.70 spread +86.9R vs +66.4R.
    # Score floors below 25 are vacuous (grade C requires |score|≥25).
    m5_require_mtf: bool = True
-    m5_min_abs_score: float = 25.0
+    m5_min_abs_score: float = 30.0
-    m5_min_confirmation_score: float = 5.0
+    m5_min_confirmation_score: float = 10.0
    # A-grade M5 SELLs enter at the extreme of temporary dips and
    # lose (-0.017R/trade on 163 trades) — maximum conviction at
    # the worst entry. Cap SELL quality so A-grade can't trigger
@@ -284,15 +271,6 @@ class XauStrategyConfig:
    cp_flag_max_bars: int = 12
    cp_flag_retrace: tuple = (0.1, 0.5)  # flag depth as fraction of pole
    # ── Big-candle entry gate ──────────────────────────────────────
    # When the recent window on the scan's own primary TF contains an
    # abnormally large candle (bar range >= K x the PRIOR bar's ATR —
    # institutional burst; price may reverse violently), no XAU entry
    # of ANY kind may fire (core, XAU_PB, XAU_CP). 0.0 = disabled.
    # Per-TF factories opt in only after the standard validation rule.
    big_candle_atr_multiple: float = 0.0
    big_candle_lookback_hours: float = 24.0
@dataclass
 class FilterConfig:
@@ -1,451 +0,0 @@
 #!/usr/bin/env python3
 """
 XAU/USD static dashboard publisher.
 Standalone daemon that scans XAU/USD at candle closes and atomically
 writes `snapshot.json` + `news.json` (plus a copy of
 `static/dashboard.html`) into an Apache-served directory. Apache does
 all web serving — this process never opens a port, so a failure here
 can only ever mean a stale page.
 Independence rules (owner requirement — the telegram bot must be
 immune to anything on the web side):
  * Runs as its own process; never imported by, and never imports,
    `telegram_bot` / `oanda_daemon` (telegram_bot resolves credentials
    at import). The confirmation map and boundary math are therefore
    duplicated here as local constants.
  * The only shared-code dependency beyond the standard signal path is
    the additive `NewsEngine.get_week_events` accessor, which no
    production signal path calls.
 Scheduling: every M15/H1/H4 boundary is also an M5 boundary (all
 intervals are multiples of 5, anchored to UTC midnight), so one
 sequential loop sleeps to the next M5 boundary (+10 s buffer) and scans
 whichever timeframes just closed. Due-classification uses the RAW wake
 time — flooring 12:00:10 to the 5-minute grid yields the 12:00 boundary
 that just closed. Do NOT subtract the buffer first: 12:00:10 − 15 s
 floors to 11:55 and silently skips every H1/H4 scan (v1 bug, 2026-08-02).
 Scans run sequentially M5→M15→H1→H4 through one shared data provider so
 later timeframes hit the warm boundary cache — that ordering IS the
 duplicate-fetch dedup (providers have no in-flight dedup; Oanda has no
 throttle). Keep it sequential.
 Run:  python dashboard_publisher.py [--once] [--out DIR]
 Out dir precedence: --out → DASHBOARD_OUT_DIR env → /var/www/trade
 """
 import argparse
 import json
 import hashlib
 import logging
 import math
 import numbers
 import os
 import shutil
 import sys
 import time
 from datetime import datetime, timezone
 from logging.handlers import TimedRotatingFileHandler
 from typing import Dict, List, Optional
 from config import PlatformConfig, Timeframe, XAUUSD
 from main import report_to_dict
 from news_engine import NewsEngine
 from pair_signals import produce_signal
 from providers import make_data_provider
 import providers.base as _providers_base
 logger = logging.getLogger("dashboard")
 DASHBOARD_TFS = [Timeframe.M5, Timeframe.M15, Timeframe.H1, Timeframe.H4]
 # Local copy of the bot's confirmation map (see independence rules above).
 CONFIRMATIONS = {
    Timeframe.M5:  [Timeframe.M15, Timeframe.H1],
    Timeframe.M15: [Timeframe.M5, Timeframe.H1],
    Timeframe.H1:  [Timeframe.M15, Timeframe.H4],
    Timeframe.H4:  [Timeframe.H1, Timeframe.D1],
 }
 TF_INTERVAL_MINUTES = {
    Timeframe.M5: 5, Timeframe.M15: 15, Timeframe.H1: 60, Timeframe.H4: 240,
 }
 DEFAULT_OUT_DIR = "/var/www/trade"
 SCAN_BUFFER_SECONDS = 10
 SNAPSHOT_FILE = "snapshot.json"
 NEWS_FILE = "news.json"
 PAGE_FILE = "dashboard.html"
 SUMMARY_PHP_FILE = "news_summary.php"
 # Repo files copied into the out dir at daemon start (git pull + restart
 # updates them). The PHP endpoint reads news.json in place, so it lives
 # next to the page.
 STATIC_FILES = [PAGE_FILE, SUMMARY_PHP_FILE]
 # ══════════════════════════════════════════════════════════════════
 #  Pure layer (no network, fully covered by test_all.py)
 # ══════════════════════════════════════════════════════════════════
 def _safe_num(v):
    """Coerce to a JSON-safe float; NaN/±inf become None."""
    try:
        f = float(v)
    except (TypeError, ValueError):
        return v
    if math.isnan(f) or math.isinf(f):
        return None
    return f
 def _sanitize(obj):
    """Recursively make a payload safe for json.dumps(allow_nan=False):
    NaN/inf → None, numpy scalars → native ints/floats."""
    if isinstance(obj, bool):
        return obj
    if isinstance(obj, dict):
        return {k: _sanitize(v) for k, v in obj.items()}
    if isinstance(obj, (list, tuple)):
        return [_sanitize(v) for v in obj]
    if isinstance(obj, numbers.Integral):
        return int(obj)
    if isinstance(obj, numbers.Real):
        return _safe_num(obj)
    return obj
 def analysis_to_dict(analysis) -> Optional[dict]:
    """Serialize a TechnicalAnalysis. Iterates analysis.indicators only —
    never vars()/asdict(), which would drag dynamic pandas attrs
    (_rsi_series, _macd_histogram, …) into the payload. ATR (weight 0)
    is surfaced separately as context, not in the directional grid."""
    if analysis is None:
        return None
    indicators = {}
    atr_context = None
    for name, ind in analysis.indicators.items():
        if name.startswith("_"):
            continue
        entry = {
            "signal": ind.signal.name,
            "value": _safe_num(ind.value),
            "confidence": _safe_num(ind.confidence),
            "weight": _safe_num(ind.weight),
            "description": ind.description,
        }
        if name == "atr":
            atr_context = entry
        elif ind.weight != 0:
            indicators[name] = entry
    return {
        "timeframe": analysis.timeframe,
        "timestamp": str(analysis.timestamp),
        "composite_signal": analysis.composite_signal.name,
        "composite_score": _safe_num(analysis.composite_score),
        "confidence": _safe_num(analysis.confidence),
        "market_regime": analysis.market_regime,
        "volatility_regime": analysis.volatility_regime,
        "pattern_signals": list(analysis.pattern_signals),
        "key_levels": {k: _safe_num(v) for k, v in analysis.key_levels.items()},
        "indicators": indicators,
        "atr_context": atr_context,
    }
 def report_to_webapp_dict(report) -> dict:
    """main.report_to_dict + the per-TF detail the dashboard needs."""
    d = report_to_dict(report)
    d["primary_signal"] = report.primary_signal.name
    d["primary_score"] = _safe_num(report.primary_score)
    d["mtf_scores"] = {k: _safe_num(v) for k, v in report.mtf_scores.items()}
    d["exhaustion_count"] = report.exhaustion_count
    d["primary_analysis"] = analysis_to_dict(report.primary_analysis)
    d["confirmation_analyses"] = [
        analysis_to_dict(a) for a in report.confirmation_analyses
    ]
    return _sanitize(d)
 def event_to_dict(ev) -> dict:
    """EconomicEvent → dict. surprise/minutes_until/is_high_impact are
    properties, so asdict() would skip them — added explicitly.
    actual/forecast/previous stay strings as published ('228K', '0.3%')."""
    return {
        "time": ev.time.astimezone(timezone.utc).isoformat(),
        "currency": ev.currency,
        "impact": ev.impact,
        "event": ev.event,
        "forecast": ev.forecast,
        "previous": ev.previous,
        "actual": ev.actual,
        "surprise": ev.surprise,
        "minutes_until": _safe_num(ev.minutes_until),
        "is_high_impact": ev.is_high_impact,
    }
 def headline_to_dict(h) -> dict:
    # id uses the same sha1(headline) scheme as NewsEngine._llm_sent_cache;
    # it is the join key the PHP summary endpoint whitelists against.
    return {
        "id": hashlib.sha1(h.headline.encode("utf-8")).hexdigest(),
        "headline": h.headline,
        "source": h.source,
        "link": h.link,
        "description": h.description,
        "publisher": h.publisher,
        "timestamp": h.timestamp.astimezone(timezone.utc).isoformat()
                     if h.timestamp else None,
        "currencies": list(h.currencies_mentioned),
        "score": _safe_num(h.sentiment_score),
        "keywords": list(h.keywords),
        "score_by_currency": {
            k: _safe_num(v) for k, v in h.sentiment_by_currency.items()
        },
    }
 def build_news_payload(context, week_events, now: datetime) -> dict:
    """news.json: full calendar week + scored headlines + LLM summary."""
    headlines = sorted(
        context.news_headlines,
        key=lambda h: h.timestamp or now, reverse=True,
    )
    return _sanitize({
        "generated_at": now.isoformat(),
        "llm_summary": context.llm_summary,
        "sentiment": dict(context.news_sentiment),
        "high_impact_warning": context.high_impact_warning,
        "warning_message": context.warning_message,
        "calendar_available": context.calendar_available,
        "events": [event_to_dict(e) for e in week_events],
        "headlines": [headline_to_dict(h) for h in headlines],
    })
 def due_timeframes(wake_dt: datetime) -> List[Timeframe]:
    """Timeframes whose candle just closed at the boundary this wake
    belongs to. Takes the RAW wake time (boundary + buffer): flooring to
    the 5-minute grid yields the boundary that just passed. Never
    subtract the buffer first — see module docstring (v1 bug)."""
    minutes = wake_dt.hour * 60 + wake_dt.minute
    floored = minutes - (minutes % 5)
    return [tf for tf in DASHBOARD_TFS if floored % TF_INTERVAL_MINUTES[tf] == 0]
 def seconds_until_boundary(interval_minutes: int, buffer: int = 10) -> float:
    # Same math as telegram_bot.py / oanda_daemon.py — kept local so the
    # dashboard never imports bot modules (independence rules above).
    now = datetime.now(timezone.utc)
    total_minutes = now.hour * 60 + now.minute
    elapsed_minutes = total_minutes % interval_minutes
    elapsed_seconds = elapsed_minutes * 60 + now.second + now.microsecond / 1e6
    interval_seconds = interval_minutes * 60
    remaining = interval_seconds - elapsed_seconds + buffer
    # NOTE: remaining > buffer always (elapsed < interval by construction);
    # a call at the exact boundary instant targets the NEXT boundary —
    # steady-state loops wake at boundary+buffer, so only a process
    # started in the boundary microsecond skips one scan (2026-08-04
    # audit I42; the old `remaining <= buffer` guard was unreachable).
    return remaining
 def _empty_tf_entry() -> dict:
    return {"status": "warming", "scanned_at": None, "scan_seconds": None,
            "error": None, "error_at": None, "report": None}
 def build_snapshot_payload(snapshots: Dict[str, dict], now: datetime) -> dict:
    """snapshot.json: one entry per dashboard TF. Timeframes not yet
    scanned appear as status="warming" so the page can render a
    placeholder instead of a hole. server_time_epoch lets the client
    compute its clock offset once so countdowns don't drift."""
    timeframes = {}
    for tf in DASHBOARD_TFS:
        interval = TF_INTERVAL_MINUTES[tf]
        minutes = now.hour * 60 + now.minute
        secs_into = (minutes % interval) * 60 + now.second + now.microsecond / 1e6
        next_close = now.timestamp() + (interval * 60 - secs_into)
        entry = _empty_tf_entry()
        entry.update(snapshots.get(tf.value) or {})
        entry["next_close_epoch"] = next_close
        entry["next_scan_epoch"] = next_close + SCAN_BUFFER_SECONDS
        timeframes[tf.value] = entry
    return _sanitize({
        "symbol": XAUUSD,
        "server_time_epoch": now.timestamp(),
        "market_closed": _providers_base._is_forex_weekend(now),
        "timeframes": timeframes,
    })
 def publish(obj: dict, path: str) -> None:
    """Atomic JSON write: temp file + os.replace, so Apache never serves
    a torn file. allow_nan=False — a NaN slipping through would render
    the page blank (JSON.parse rejects bare NaN)."""
    data = json.dumps(obj, allow_nan=False, separators=(",", ":"))
    tmp = path + ".tmp"
    with open(tmp, "w", encoding="utf-8") as fh:
        fh.write(data)
    os.replace(tmp, path)
 # ══════════════════════════════════════════════════════════════════
 #  Runtime layer (constructed only from main(); never at import)
 # ══════════════════════════════════════════════════════════════════
 def _setup_logging(level: str = "INFO") -> None:
    log_dir = os.path.join(os.path.dirname(os.path.abspath(__file__)), "logs")
    os.makedirs(log_dir, exist_ok=True)
    log_file = os.path.join(log_dir, "dashboard.log")
    fmt = logging.Formatter(
        "%(asctime)s [%(levelname)s] %(name)s: %(message)s",
        datefmt="%Y-%m-%d %H:%M:%S")
    file_handler = TimedRotatingFileHandler(
        log_file, when="h", interval=3, backupCount=240, utc=True)
    file_handler.setFormatter(fmt)
    console_handler = logging.StreamHandler(sys.stdout)
    console_handler.setFormatter(fmt)
    logging.basicConfig(
        level=getattr(logging, level.upper(), logging.INFO),
        handlers=[file_handler, console_handler],
        force=True)
 class DashboardPublisher:
    def __init__(self, out_dir: str):
        self.out_dir = out_dir
        self.snapshots: Dict[str, dict] = {}
        self.news_payload: Optional[dict] = None
        self._dp = None
        self._news_engine: Optional[NewsEngine] = None
    def _provider(self):
        if self._dp is None:
            self._dp = make_data_provider(PlatformConfig())
        return self._dp
    def _news(self) -> NewsEngine:
        if self._news_engine is None:
            self._news_engine = NewsEngine(PlatformConfig().news)
        return self._news_engine
    def install_static(self) -> None:
        """Copy the repo's static files (page + PHP summary endpoint)
        into the out dir (atomic), so git pull + restart also updates
        them. The ai/ cache dir is NOT created here — the daemon runs as
        a different user than Apache; that dir is a one-time deploy step
        (deploy/README-dashboard.md)."""
        static_dir = os.path.join(
            os.path.dirname(os.path.abspath(__file__)), "static")
        for name in STATIC_FILES:
            src = os.path.join(static_dir, name)
            if not os.path.exists(src):
                logger.warning(f"{src} missing — not installed")
                continue
            dst = os.path.join(self.out_dir, name)
            shutil.copyfile(src, dst + ".tmp")
            os.replace(dst + ".tmp", dst)
    def scan_tf(self, tf: Timeframe) -> None:
        started = time.time()
        try:
            report = produce_signal(XAUUSD, tf, self._provider(),
                                    CONFIRMATIONS[tf])
            self.snapshots[tf.value] = {
                "status": "ok",
                "scanned_at": datetime.now(timezone.utc).isoformat(),
                "scan_seconds": round(time.time() - started, 2),
                "error": None,
                "error_at": None,
                "report": report_to_webapp_dict(report),
            }
            logger.info(f"{tf.value}: {report.final_signal.name} "
                        f"q={report.signal_quality or '-'} "
                        f"({time.time() - started:.1f}s)")
        except Exception as e:
            # Keep the last good report; the page shows it with an
            # error badge instead of going blank.
            logger.exception(f"{tf.value} scan failed")
            entry = self.snapshots.get(tf.value) or _empty_tf_entry()
            entry["status"] = "error"
            entry["error"] = f"{type(e).__name__}: {e}"
            entry["error_at"] = datetime.now(timezone.utc).isoformat()
            self.snapshots[tf.value] = entry
    def refresh_news(self) -> None:
        try:
            ne = self._news()
            context = ne.get_fundamental_context()
            week = ne.get_week_events()
            self.news_payload = build_news_payload(
                context, week, datetime.now(timezone.utc))
        except Exception:
            logger.exception("news refresh failed — keeping last payload")
    def publish_snapshot(self) -> None:
        publish(build_snapshot_payload(
            self.snapshots, datetime.now(timezone.utc)),
            os.path.join(self.out_dir, SNAPSHOT_FILE))
    def publish_news(self) -> None:
        if self.news_payload is not None:
            publish(self.news_payload, os.path.join(self.out_dir, NEWS_FILE))
    def publish_all(self) -> None:
        self.publish_snapshot()
        self.publish_news()
    def run_cycle(self, tfs: List[Timeframe]) -> None:
        for tf in tfs:  # sequential — see module docstring (cache dedup)
            self.scan_tf(tf)
        # Snapshot lands BEFORE the news refresh: when the news TTL has
        # expired, refresh_news does FF + 4 RSS fetches (and possibly
        # DeepSeek calls) — 20-30 s that must not keep fresh signals off
        # the page.
        self.publish_snapshot()
        self.refresh_news()
        self.publish_news()
    def run_forever(self) -> None:
        self.run_cycle(DASHBOARD_TFS)  # populate the page at boot
        while True:
            delay = seconds_until_boundary(5, buffer=SCAN_BUFFER_SECONDS)
            logger.info(f"next M5 boundary in {delay:.0f}s")
            time.sleep(delay)
            try:
                due = due_timeframes(datetime.now(timezone.utc))
                logger.info(f"boundary wake — scanning {[t.value for t in due]}")
                self.run_cycle(due)
            except Exception:
                logger.exception("cycle failed")
 def main(argv=None) -> None:
    ap = argparse.ArgumentParser(description="XAU/USD static dashboard publisher")
    ap.add_argument("--once", action="store_true",
                    help="one scan+publish cycle, then exit")
    ap.add_argument("--out",
                    default=os.environ.get("DASHBOARD_OUT_DIR", DEFAULT_OUT_DIR),
                    help="output directory served by Apache "
                         f"(default: $DASHBOARD_OUT_DIR or {DEFAULT_OUT_DIR})")
    args = ap.parse_args(argv)
    _setup_logging()
    os.makedirs(args.out, exist_ok=True)
    pub = DashboardPublisher(args.out)
    pub.install_static()
    if args.once:
        pub.run_cycle(DASHBOARD_TFS)
        print(f"Published to {args.out}")
        return
    pub.run_forever()
 if __name__ == "__main__":
    main()
@@ -0,0 +1,24 @@
 """
 ForexBot2 — Data Provider (back-compat shim)
 =============================================
 The provider implementations live in the `providers/` package:
    providers/base.py         BaseDataProvider (shared contract + behavior)
    providers/twelve_data.py  TwelveDataProvider (SDK fetch, throttle, WS)
    providers/oanda.py        OandaDataProvider (v20 REST, real spread)
 `ForexDataProvider` is the historical name of the Twelve Data provider and
 is kept as an alias so existing imports keep working.
 """
 from providers import (  # noqa: F401
    BaseDataProvider,
    DataIntegrityError,
    PROVIDER_REGISTRY,
    TwelveDataProvider,
    make_data_provider,
    _is_forex_weekend,
 )
 # Historical name for the Twelve Data provider.
 ForexDataProvider = TwelveDataProvider
@@ -1,157 +0,0 @@
 # Deploying the web dashboard (live LAMP server)
 The dashboard is a **static publisher**: `dashboard_publisher.py` runs as its
 own daemon, scans XAU/USD at candle closes, and atomically writes
 `dashboard.html` + `news_summary.php` + `snapshot.json` + `news.json` into a
 folder Apache already serves. Apache does all web serving — no new port, no
 Python web server, and the telegram bot process is completely unaffected by
 anything that happens here (see the module docstring for the independence
 rules). The one dynamic piece is `news_summary.php` (§6): an on-demand
 DeepSeek summary for a clicked headline, whitelist-gated against news.json.
 All steps below run **on the live server** (`/home/Forex/ForexBot2`).
 ## 1. Get the code
 ```bash
 cd /home/Forex/ForexBot2
 git pull
 venv/bin/python test_all.py        # optional sanity: suite must be green
 ```
 No new Python dependencies are required.
 ## 2. Make the web folder writable
 The dashboard publishes straight into the portal docroot `/var/www/trade/`
 (owner decision 2026-08-03) — it adds exactly four files there:
 `dashboard.html`, `news_summary.php`, `snapshot.json`, `news.json` (plus the
 apache-owned `ai/` cache dir from §6). The daemon user must be able to write
 them:
 ```bash
 # if the portal files can be owned by the daemon user:
 sudo chown Forex: /var/www/trade
 # otherwise grant write without changing ownership:
 sudo setfacl -m u:Forex:rwx /var/www/trade
 ```
 (A different path is one edit away: the unit's `Environment=DASHBOARD_OUT_DIR=`
 line, or `--out` for one-off runs.)
 SELinux (RHEL/Alma/CentOS enforcing): directories under `/var/www` normally
 already carry `httpd_sys_content_t`, and new files inherit the parent label, so
 no action is usually needed. If Apache 403s despite correct permissions:
 ```bash
 sudo restorecon -R /var/www/trade
 ```
 ## 3. Smoke-test once
 ```bash
 venv/bin/python dashboard_publisher.py --once
 ls /var/www/trade    # dashboard.html snapshot.json news.json added next to the portal
 ```
 Then open `http://<server>/dashboard.html` (path relative to however the
 vhost maps `/var/www/trade`) — all four timeframe cards, the calendar, and
 the newswire should populate.
 ## 4. Install the daemon
 ```bash
 sudo cp deploy/gold-dashboard.service /etc/systemd/system/
 # edit User= if the repo owner is not "Forex"
 sudo systemctl daemon-reload
 sudo systemctl enable --now gold-dashboard
 systemctl status gold-dashboard
 tail -f logs/dashboard.log
 ```
 The daemon scans all four timeframes at boot, then wakes at every M5
 boundary (+10 s) and rescans whichever timeframes just closed. Logs rotate
 every 3 h (240 backups), same scheme as the bot.
 ## 5. Optional Apache polish
 The page polls its JSON with conditional GETs (`cache: "no-cache"` → Apache
 answers unchanged files with 304), so no Apache config is strictly needed. To
 be explicit anyway:
 ```apache
 <Directory "/var/www/trade">
    <FilesMatch "\.json$">
        Header set Cache-Control "no-cache"
    </FilesMatch>
 </Directory>
 ```
 The page is deliberately public (owner decision 2026-08-03). To restrict it
 later, add standard Apache basic auth (`htpasswd` + `Require valid-user`) on
 that directory — no code changes needed.
 ## 6. AI summary endpoint (PHP) — one-time setup
 Clicking a newswire headline opens a detail dialog that fetches an on-demand
 DeepSeek gold-impact summary from `news_summary.php` (installed next to the
 page by the daemon). Owner decision 2026-08-07: a real on-demand endpoint on
 the public page, accepted with these guards baked into the PHP — the id must
 match a headline in the published news.json (arbitrary text can never reach
 the LLM), each headline's summary is cached forever in `ai/` (≤1 API call per
 headline, ever), and a file-based daily cap fails **closed** on I/O errors.
 Requirements: `php` + the curl extension (`php -m | grep curl`).
 ```bash
 # 1. SELinux: allow httpd outbound HTTPS — WITHOUT this every summary call
 #    dies as a curl timeout even with perfect perms (RHEL-family default).
 sudo setsebool -P httpd_can_network_connect 1
 # 2. API key OUTSIDE the docroot, readable by Apache only:
 sudo install -d -o root -g root -m 755 /etc/gold-dashboard
 sudo sh -c 'umask 137; echo "sk-..." > /etc/gold-dashboard/deepseek.key'
 sudo chown root:apache /etc/gold-dashboard/deepseek.key   # mode 640
 # 3. apache-owned cache dir (the daemon runs as Forex and can't create it):
 sudo install -d -o apache -g apache -m 755 /var/www/trade/ai
 sudo semanage fcontext -a -t httpd_sys_rw_content_t '/var/www/trade/ai(/.*)?'
 sudo restorecon -Rv /var/www/trade/ai
 ```
 Knobs (Apache `SetEnv` in the vhost, both optional):
 `AI_SUMMARY_DAILY_CAP` (default 150), `DEEPSEEK_KEY_FILE` (default
 `/etc/gold-dashboard/deepseek.key`).
 Verify from the shell — expect 400 / 404 / 200 / 200-from-cache:
 ```bash
 curl -s "http://<server>/news_summary.php?id=zzz"                 # {"error":"bad_id"}
 curl -s "http://<server>/news_summary.php?id=$(printf 'a%.0s' {1..40})"  # unknown_id
 ID=$(python3 -c "import json;print(json.load(open('/var/www/trade/news.json'))['headlines'][0]['id'])")
 curl -s "http://<server>/news_summary.php?id=$ID"   # real summary (2-12 s)
 curl -s "http://<server>/news_summary.php?id=$ID"   # instant, from ai/<id>.json
 ```
 ## Operational notes
 - **Updating**: `git pull` + `systemctl restart gold-dashboard`. The daemon
  re-copies `static/dashboard.html` and `static/news_summary.php` into the
  web folder at startup.
 - **If the daemon dies**, systemd restarts it in 30 s; worst case the page
  shows stale data with its own "updated N min ago" evidence. Apache and the
  telegram bot are unaffected.
 - **DeepSeek**: the publisher process has its own daily request cap
  (`deepseek.daily_request_cap`, default 500) separate from the bot's —
  expect roughly up to ~430 extra calls/day worst case, V4-Flash pricing.
 - **Two independent DeepSeek meters** hit the same API key: the publisher's
  in-process cap above, and the PHP endpoint's file cap (`ai/.ht_cap.json`,
  default 150/day). They don't communicate — worst-case spend is their sum;
  DeepSeek-side spend alerting is the true backstop. Kept separate on
  purpose (independence rule: the PHP never imports/reads bot code or
  config.json).
 - **Restart window**: for the first minutes after a daemon restart the new
  page can meet a pre-upgrade news.json without headline ids — those rows
  simply render non-clickable until the first news publish.
 - **Credentials**: uses the same env/.env/config.json resolution as the rest
  of the platform (Oanda candles feed; news needs no key; DeepSeek optional).
@@ -1,23 +0,0 @@
 # systemd unit for the XAU/USD static dashboard publisher.
 # Install (as root, on the live server):
 #   cp deploy/gold-dashboard.service /etc/systemd/system/
 #   systemctl daemon-reload && systemctl enable --now gold-dashboard
 # See deploy/README-dashboard.md for the full checklist.
 [Unit]
 Description=XAU/USD dashboard publisher (writes static JSON/HTML for Apache)
 After=network-online.target
 Wants=network-online.target
 [Service]
 Type=simple
 # Adjust to the account that owns /home/Forex/ForexBot2:
 User=Forex
 WorkingDirectory=/home/Forex/ForexBot2
 Environment=DASHBOARD_OUT_DIR=/var/www/trade
 ExecStart=/home/Forex/ForexBot2/venv/bin/python dashboard_publisher.py
 Restart=always
 RestartSec=30
 [Install]
 WantedBy=multi-user.target
@@ -1,199 +0,0 @@
 """ForexBot2 architecture diagram → docs/architecture.png.
 Regenerate after architecture changes:  venv/bin/python docs/architecture_diagram.py
 (matplotlib only — no graphviz on the dev box)."""
 import os
 import matplotlib
 matplotlib.use("Agg")
 import matplotlib.pyplot as plt
 from matplotlib.patches import FancyBboxPatch, FancyArrowPatch
 W, H = 100.0, 100.0
 fig, ax = plt.subplots(figsize=(16, 11.5))
 ax.set_xlim(0, W); ax.set_ylim(0, H); ax.axis("off")
 INK = "#263238"; MUT = "#607D8B"; ARR = "#455A64"; HAUL = "#7B8FA1"
 def box(x, y, w, h, title, lines, fc, ec, title_fs=8.2, body_fs=6.6, lw=1.3):
    ax.add_patch(FancyBboxPatch((x, y), w, h,
                 boxstyle="round,pad=0.35,rounding_size=0.9",
                 facecolor=fc, edgecolor=ec, linewidth=lw, zorder=2))
    cy = y + h - 1.9
    ax.text(x + w / 2, cy, title, ha="center", va="center", fontsize=title_fs,
            fontweight="bold", color=INK, zorder=3)
    cy -= 2.3
    for ln in lines:
        ax.text(x + w / 2, cy, ln, ha="center", va="center", fontsize=body_fs,
                color=INK, zorder=3)
        cy -= 1.85
 def band(x, y, w, h, label, color):
    ax.add_patch(FancyBboxPatch((x, y), w, h,
                 boxstyle="round,pad=0.3,rounding_size=1.2",
                 facecolor="none", edgecolor=color, linewidth=1.0,
                 linestyle=(0, (4, 3)), zorder=1))
    ax.text(x + 0.9, y + h - 1.4, label, ha="left", va="center",
            fontsize=7.2, fontweight="bold", color=color, zorder=3)
 def arrow(p1, p2, color=ARR, lw=1.5, style="-|>", conn="arc3,rad=0.0",
          label=None, lx=0, ly=0, ls="solid", fs=6.4, rot=0):
    ax.add_patch(FancyArrowPatch(p1, p2, arrowstyle=style, mutation_scale=11,
                 color=color, linewidth=lw, linestyle=ls,
                 connectionstyle=conn, zorder=4))
    if label:
        mx, my = (p1[0] + p2[0]) / 2 + lx, (p1[1] + p2[1]) / 2 + ly
        ax.text(mx, my, label, fontsize=fs, color=MUT, ha="center",
                va="center", zorder=5, rotation=rot,
                bbox=dict(boxstyle="round,pad=0.15", fc="white", ec="none", alpha=0.9))
 # ── title ────────────────────────────────────────────────────────────
 ax.text(2, 98.4, "ForexBot2 — XAU/USD Signal-Intelligence Platform",
        fontsize=15, fontweight="bold", color=INK, ha="left")
 ax.text(2, 96.0, "one pipeline, four delivery surfaces · closed bars only · live == backtest parity",
        fontsize=8.5, color=MUT, ha="left")
 EXT_FC, EXT_EC = "#ECEFF1", "#90A4AE"
 DAT_FC, DAT_EC = "#E3F2FD", "#5C9BD1"
 NWS_FC, NWS_EC = "#E0F2F1", "#3D9C8F"
 COR_FC, COR_EC = "#FFFDF5", "#C9A227"
 REP_FC, REP_EC = "#FFE9A8", "#B8860B"
 DEL_FC, DEL_EC = "#E8F5E9", "#5FAF63"
 BTS_FC, BTS_EC = "#EDE7F6", "#8E6BC6"
 # ── external services (top band) ─────────────────────────────────────
 band(1, 85.0, 98, 9.4, "EXTERNAL SERVICES", "#78909C")
 box(2.5, 86.2, 15.5, 6.4, "Oanda v20 API",
    ["candles (live feed)", "practice orders"], EXT_FC, EXT_EC)
 box(19.5, 86.2, 15.5, 6.4, "Twelve Data · Tiingo",
    ["registered fallbacks", "(Tiingo: tick stream)"], EXT_FC, EXT_EC)
 box(36.5, 86.2, 15.5, 6.4, "Forex Factory",
    ["economic calendar", "MyFXBook RSS fallback"], EXT_FC, EXT_EC)
 box(53.5, 86.2, 12.5, 6.4, "News RSS ×4",
    ["gold / macro", "headlines"], EXT_FC, EXT_EC)
 box(67.5, 86.2, 13.5, 6.4, "DeepSeek API",
    ["V4-Flash LLM", "thinking: disabled"], EXT_FC, EXT_EC)
 box(82.5, 86.2, 15, 6.4, "Telegram API",
    ["owner bot +", "subscriber channel"], EXT_FC, EXT_EC)
 # ── data & context layer ─────────────────────────────────────────────
 box(2.5, 69.8, 33, 11.0, "providers/  (BaseDataProvider)",
    ["single-flight fetch · TTL+boundary cache",
     "forming-candle DROP · validate/clean",
     "paginated fetch_range · optional tick stream",
     "active feed knob: data_providers.active = oanda"],
    DAT_FC, DAT_EC)
 box(38.5, 69.8, 33, 11.0, "news_engine.py  +  llm_client.py",
    ["calendar events + keyword sentiment (gold-inverse)",
     "DeepSeek enrichment: batch headline scores,",
     "event gold-direction, market blurb — display-only,",
     "keyword fallback, daily request cap"],
    NWS_FC, NWS_EC)
 box(74.5, 69.8, 23, 11.0, "app_config.py",
    ["credentials & deployment:",
     "env → config.json → refuse",
     "(no hardcoded fallbacks)",
     ".env loaded at import"],
    "#FAFAFA", "#9E9E9E")
 # ── signal core container ────────────────────────────────────────────
 ax.add_patch(FancyBboxPatch((2.5, 40.5), 95, 26.5,
             boxstyle="round,pad=0.35,rounding_size=1.2",
             facecolor=COR_FC, edgecolor=COR_EC, linewidth=1.6, zorder=1))
 ax.text(50, 65.2, "SIGNAL CORE — runs identically in live scans and backtests (closed bars only)",
        ha="center", fontsize=8.4, fontweight="bold", color="#8a6d1a", zorder=3)
 box(4.5, 51.8, 28, 12.0, "engine.py — TechnicalEngine",
    ["9 scored indicators + ATR context:",
     "EMA · RSI · MACD · BB · ADX · ROC ·",
     "Keltner · divergence · candlestick",
     "regime → adaptive weights → composite"],
    "white", COR_EC)
 box(4.5, 42.0, 28, 8.4, "pair_signals/xauusd.py — factory",
    ["ALWAYS used: config.json overlay +",
     "validated per-TF deltas (hour blocks, trail…)"],
    "white", COR_EC)
 box(35.5, 42.0, 60, 20.7, "signal_aggregator.py — analyze_pair → analyze_from_dataframes",
    ["multi-TF agreement  (M5→[M15,H1] · M15→[M5,H1] · H1→[M15,H4] · H4→[H1,D1])",
     "→ news context (warn-only by default) → diminishing-decay penalties",
     "→ exhaustion hard block → quality grade A+…F",
     "→ ordered post-grade gate chain (session · XAU SELL vol/hour · regime · M5 noise)",
     "→ setup generators: XAU ATR (A+/A) · generic swing · XAU_PB · XAU_CP",
     "→ entry · SL · TP1/TP2 · trailing plan"],
    "white", COR_EC, body_fs=6.9)
 # ── SignalReport hub ─────────────────────────────────────────────────
 box(31, 33.2, 38, 5.2, "SignalReport",
    ["signal (−2…+2) · grade · trade setup · reasons · news summary"],
    REP_FC, REP_EC, title_fs=8.8)
 # ── delivery surfaces ────────────────────────────────────────────────
 band(1, 13.2, 98, 17.2, "DELIVERY SURFACES", "#4C8F50")
 box(2.5, 15.0, 12, 12.6, "main.py",
    ["CLI report", "--detail", "--json"], DEL_FC, DEL_EC)
 box(16, 15.0, 24, 12.6, "oanda_daemon.py + executor",
    ["practice auto-execution",
     "book-parity rules: no broker TP,",
     "5-bar cooldown, once-per-bar",
     "trail, OPEN_ONLY fills,",
     "boot reconcile · orders → Oanda"],
    DEL_FC, DEL_EC, title_fs=7.8)
 box(42.5, 15.0, 31, 12.6, "dashboard_publisher.py  →  Apache",
    ["independent daemon · scans at candle closes",
     "atomic publish to /var/www/trade:",
     "dashboard.html · snapshot.json · news.json",
     "+ news_summary.php — on-demand AI",
     "gold-impact summary (whitelist · cache · cap)"],
    DEL_FC, DEL_EC)
 box(76, 15.0, 21.5, 12.6, "telegram_bot.py",
    ["candle-boundary auto-scans", "quality threshold + dedup",
     "alerts + LLM context line", "WS level-breach monitor"],
    DEL_FC, DEL_EC)
 # ── backtest / QA ────────────────────────────────────────────────────
 box(2.5, 3.2, 71.8, 8.0, "backtest/gpu_backtest.py — bar-by-bar replay through the SAME analyze_from_dataframes",
    ["realism (results-critical): next-open entries · raw-path resolution · spread charged · SL-first ties · close-only trail",
     "CuPy GPU / NumPy fallback · parquet cache · fed by providers.fetch_range (news off)"],
    BTS_FC, BTS_EC)
 box(77.5, 3.2, 20, 8.0, "test_all.py",
    ["359 tests · 22 modules", "no network (mocked)"], "#FAFAFA", "#9E9E9E")
 # ── arrows: short structural hops (solid dark) ───────────────────────
 arrow((10, 86.2), (14, 81.2), label="OHLCV", lx=-3.4, ly=0.2)          # oanda → providers
 arrow((27, 86.2), (24, 81.2), ls=(0, (3, 2)), lw=1.1)                  # fallbacks → providers
 arrow((44, 86.2), (48, 81.2))                                          # FF → news
 arrow((59.5, 86.2), (56, 81.2))                                        # RSS → news
 arrow((72, 86.2), (62, 81.2), label="LLM", lx=3.4, ly=0.6)             # deepseek → news
 arrow((19, 69.8), (19, 64.2), label="DataFrames (UTC, closed bars)", lx=0.2, ly=0.9)
 arrow((33.5, 57.5), (35.5, 55.5))                                      # engine → aggregator
 arrow((32.5, 46.0), (35.5, 48.0))                                      # factory → aggregator
 arrow((55, 69.8), (55, 63.1), label="warn-only context", lx=0, ly=1.0)
 arrow((86, 69.8), (86, 67.3), ls=(0, (3, 2)), lw=1.1)                  # config → core
 arrow((50, 40.5), (50, 38.8))                                          # core → report
 arrow((36, 34.5), (8.5, 27.95), conn="arc3,rad=0.12")                  # report → cli
 arrow((40, 33.2), (28, 27.95), conn="arc3,rad=0.06")                   # report → daemon
 arrow((60, 33.2), (58, 27.95))                                         # report → dashboard
 arrow((66, 34.5), (86.5, 27.95), conn="arc3,rad=-0.12")                # report → telegram
 # ── long-haul edges (thin dashed, routed through gutters) ────────────
 # telegram bot ↔ Telegram API — right-margin arc, clear of app_config
 arrow((95, 86.2), (95, 27.95), conn="arc3,rad=0.15", color=HAUL,
      lw=1.2, ls=(0, (4, 2)))
 ax.text(99.2, 57, "alerts / commands", fontsize=6.4, color=MUT,
        rotation=90, ha="center", va="center")
 # browser click → news_summary.php → DeepSeek
 # (oanda_daemon's order flow is noted inside its box — no long arc)
 arrow((66, 27.95), (73, 86.2), conn="arc3,rad=-0.18", color=HAUL,
      lw=1.2, ls=(0, (4, 2)),
      label="on-demand AI summary\n(browser click → PHP)", lx=2.5, ly=11.6)
 # providers.fetch_range → backtester — far-left margin channel
 arrow((1.5, 69.8), (1.5, 11.0), color=BTS_EC, lw=1.1, ls=(0, (3, 2)),
      label="fetch_range\n(history)", lx=3.6, ly=-1.2)
 # backtester replays the signal core — right gutter between boxes
 arrow((74.0, 11.6), (74.9, 40.3), color=HAUL, lw=1.3)
 ax.text(81.8, 38.9, "replays the exact signal code", fontsize=6.4,
        color=MUT, ha="center", va="center", zorder=5,
        bbox=dict(boxstyle="round,pad=0.15", fc="white", ec="none", alpha=0.9))
 out = os.path.join(os.path.dirname(os.path.abspath(__file__)), "architecture.png")
 fig.savefig(out, dpi=200, bbox_inches="tight", facecolor="white")
 print(f"saved {out}")
@@ -0,0 +1,980 @@
 # Centralized config.json Implementation Plan
 > **For agentic workers:** REQUIRED SUB-SKILL: Use superpowers:subagent-driven-development (recommended) or superpowers:executing-plans to implement this plan task-by-task. Steps use checkbox (`- [ ]`) syntax for tracking.
 **Goal:** Move deployment settings (Telegram credentials, data-provider API keys/selection, scanned pairs, scan timeframes, per-pair base blocked hours, alert quality boot default, log level) into a gitignored `config.json` loaded once at startup, and delete the hardcoded credentials from source.
 **Architecture:** A new `app_config.py` module loads and validates `config.json` into an `AppConfig` dataclass (cached singleton), with resolver functions applying the precedence **env var → config.json → code default**. The pair factory calls `apply_pair_overrides()` on every fresh `PlatformConfig` *before* its validated per-timeframe deltas, so the bot, CLI, and backtester all pick up the overlay identically (live == backtest parity by construction). Spec: `docs/superpowers/specs/2026-07-21-centralized-config-design.md`.
 **Tech Stack:** Python 3.12 stdlib only (`json`, `os`, `dataclasses`). No new dependencies.
 ## Global Constraints
 - **Zero signal-logic change.** No indicator weight, threshold, gate, or geometry value may change. With no `config.json`, or one holding the example defaults, every produced `PlatformConfig` must be byte-identical (`dataclasses.asdict` equality) to today's factory output — pinned by a test.
 - The test suite is `python test_all.py` — a custom harness (`@test("name")` decorator, tests execute at import time, exit code non-zero on failure). It is NOT pytest; there is no single-test selection. "Run the test" always means `python test_all.py` and checking the named line in the output. Baseline before this work: **184 passed, 0 failed**.
 - The suite must be hermetic: it must pass identically whether or not the developer's machine has a `config.json`, and regardless of `TELEGRAM_*`/`TWELVE_DATA_API_KEY` env vars.
 - Secrets get **no code default**. After Task 4 the only places credentials exist are `config.json` (gitignored) and env vars.
 - `config.json.example` (already committed at repo root) is the schema reference. `config.json` is already in `.gitignore` — never commit it.
 - Repo root: `/home/harwinder/Development/ForexBot2`. Use `venv/bin/python` if bare `python` lacks pandas/numpy.
 - Every commit message ends with:
  ```
  Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
  Claude-Session: https://claude.ai/code/session_01VPNF37MP9Kuv62B1VBrZ1b
  ```
 - Work on branch `centralized-config` (create from `main` at start of Task 1: `git checkout -b centralized-config`).
 ---
 ### Task 1: `app_config.py` — loader, validation, resolvers
 **Files:**
 - Create: `app_config.py`
 - Modify: `test_all.py` (new Module 13 before the SUMMARY block, ~line 2495; hermetic header after line 19)
 - Test: `test_all.py` (same file — this repo keeps all tests in one suite)
 **Interfaces:**
 - Consumes: `config.PAIR_INFO`, `config.Timeframe`, `config.XAUUSD`, `config.PlatformConfig` (all existing).
 - Produces (later tasks rely on these exact names):
  - `class AppConfigError(Exception)`
  - `@dataclass class AppConfig` — fields: `telegram_token: str = ""`, `telegram_chat_id: str = ""`, `telegram_channel_id: str = ""`, `alert_min_quality: str = "C"`, `active_provider: str = "twelve_data"`, `provider_settings: Dict[str, Dict[str, Any]] = {}`, `scan_pairs: List[str] = [XAUUSD]`, `scan_timeframes: List[Timeframe] = [M5, M15, H1, H4]`, `pair_overrides: Dict[str, Dict[str, Any]] = {}`, `log_level: str = "INFO"`
  - `load_app_config(path: str = CONFIG_PATH) -> AppConfig`
  - `get_app_config() -> AppConfig` (cached), `set_app_config(app: Optional[AppConfig]) -> None` (test hook; `None` re-enables lazy load)
  - `telegram_token(app=None) -> str`, `telegram_chat_id(app=None) -> str`, `telegram_channel_id(app=None) -> str`, `provider_api_key(app=None) -> str`
  - `apply_pair_overrides(cfg: PlatformConfig, app: Optional[AppConfig] = None) -> PlatformConfig` (implemented here, wired in Task 2)
 - [ ] **Step 1: Create branch**
 ```bash
 cd /home/harwinder/Development/ForexBot2 && git checkout -b centralized-config
 ```
 - [ ] **Step 2: Add the hermetic test header and Module 13 failing tests to `test_all.py`**
 In `test_all.py`, directly after `sys.path.insert(0, os.path.dirname(__file__))` (line 19), add:
 ```python
 # ── Hermetic app-config: the suite must not depend on the developer's
 #    local config.json or TELEGRAM_*/TWELVE_DATA_API_KEY env vars. The
 #    dummy TD key mirrors the pre-config.json behavior where a key was
 #    always present (TDClient built, no network at construction).
 import json
 import app_config
 for _k in ("TELEGRAM_TOKEN", "TELEGRAM_CHAT_ID", "TELEGRAM_CHANNEL_ID",
           "TWELVE_DATA_API_KEY"):
    os.environ.pop(_k, None)
 _TEST_APP = app_config.AppConfig(
    provider_settings={"twelve_data": {"api_key": "test-key"}}
 )
 app_config.set_app_config(_TEST_APP)
 ```
 Then insert Module 13 immediately BEFORE the `#  SUMMARY` comment block near the end of the file (search for `SUMMARY`):
 ```python
 # ══════════════════════════════════════════════════════════════════════
 #  MODULE 13: app_config.py (config.json overlay)
 # ══════════════════════════════════════════════════════════════════════
 print("\n═══ Module 13: app_config.py ═══")
 import tempfile
 from app_config import (
    AppConfig, AppConfigError, load_app_config, set_app_config,
    telegram_token, telegram_chat_id, telegram_channel_id,
    provider_api_key, apply_pair_overrides,
 )
 def _write_json(payload) -> str:
    fh = tempfile.NamedTemporaryFile(
        "w", suffix=".json", delete=False, encoding="utf-8")
    if isinstance(payload, str):
        fh.write(payload)
    else:
        json.dump(payload, fh)
    fh.close()
    return fh.name
 _FULL_CFG = {
    "telegram": {"token": "t0k", "chat_id": "111", "channel_id": "-100222",
                 "alert_min_quality": "B"},
    "data_providers": {"active": "twelve_data",
                       "twelve_data": {"api_key": "json-key"}},
    "scanning": {"pairs": ["XAU/USD"], "timeframes": ["M15", "H1"]},
    "pairs": {"XAU/USD": {"blocked_hours_utc": [3, 4]}},
    "log_level": "WARNING",
 }
@test("app_config: valid config.json maps every field")
 def _():
    app = load_app_config(_write_json(_FULL_CFG))
    assert app.telegram_token == "t0k"
    assert app.telegram_chat_id == "111"
    assert app.telegram_channel_id == "-100222"
    assert app.alert_min_quality == "B"
    assert app.active_provider == "twelve_data"
    assert app.provider_settings["twelve_data"]["api_key"] == "json-key"
    assert app.scan_pairs == [XAUUSD]
    assert app.scan_timeframes == [Timeframe.M15, Timeframe.H1]
    assert app.pair_overrides[XAUUSD]["blocked_hours_utc"] == [3, 4]
    assert app.log_level == "WARNING"
@test("app_config: missing file returns pure defaults")
 def _():
    app = load_app_config("/nonexistent/config.json")
    assert app == AppConfig(), "absent config.json must yield pure defaults"
@test("app_config: malformed JSON raises AppConfigError")
 def _():
    path = _write_json("{not json")
    try:
        load_app_config(path)
        assert False, "should have raised"
    except AppConfigError as e:
        assert "not valid JSON" in str(e)
@test("app_config: unknown keys raise AppConfigError")
 def _():
    for payload in (
        {"telgram": {}},                                   # top-level typo
        {"telegram": {"tokn": "x"}},                       # section typo
        {"pairs": {"XAU/USD": {"blocked_hours": [1]}}},    # pair-block typo
        {"scanning": {"pair": ["XAU/USD"]}},               # scanning typo
    ):
        try:
            load_app_config(_write_json(payload))
            assert False, f"should have raised for {payload}"
        except AppConfigError as e:
            assert "unknown key" in str(e).lower()
@test("app_config: invalid values raise AppConfigError")
 def _():
    bad_payloads = [
        {"pairs": {"XAU/USD": {"blocked_hours_utc": [24]}}},      # hour range
        {"pairs": {"XAU/USD": {"blocked_hours_utc": "3"}}},       # not a list
        {"pairs": {"EUR/USD": {"blocked_hours_utc": [3]}}},       # unknown pair
        {"scanning": {"pairs": ["EUR/USD"]}},                     # unknown pair
        {"scanning": {"timeframes": ["M7"]}},                     # bad TF name
        {"scanning": {"timeframes": ["W1"]}},                     # unscannable TF
        {"data_providers": {"active": "polygon",
                            "twelve_data": {"api_key": "k"}}},    # no block
        {"telegram": {"alert_min_quality": "Z"}},                 # bad quality
        {"log_level": "CHATTY"},                                  # bad level
    ]
    for payload in bad_payloads:
        try:
            load_app_config(_write_json(payload))
            assert False, f"should have raised for {payload}"
        except AppConfigError:
            pass
@test("app_config: env precedence env → config.json → default")
 def _():
    app = load_app_config(_write_json(_FULL_CFG))
    with patch.dict(os.environ, {}, clear=True):
        assert telegram_token(app) == "t0k"
        assert telegram_chat_id(app) == "111"
        assert telegram_channel_id(app) == "-100222"
        assert provider_api_key(app) == "json-key"
    with patch.dict(os.environ, {"TELEGRAM_TOKEN": "env-tok",
                                 "TELEGRAM_CHAT_ID": "999",
                                 "TWELVE_DATA_API_KEY": "env-key"}):
        assert telegram_token(app) == "env-tok"
        assert telegram_chat_id(app) == "999"
        assert provider_api_key(app) == "env-key"
    # channel: env set-but-empty means "disable feed" and must beat JSON
    with patch.dict(os.environ, {"TELEGRAM_CHANNEL_ID": ""}):
        assert telegram_channel_id(app) == ""
 ```
 - [ ] **Step 3: Run to verify the new tests fail (import error)**
 ```bash
 python test_all.py 2>&1 | head -5
 ```
 Expected: `ModuleNotFoundError: No module named 'app_config'` (the hermetic header imports it before any test runs).
 - [ ] **Step 4: Write `app_config.py`**
 Create `app_config.py` with exactly this content:
 ```python
 """
 app_config.py — deployment-settings overlay (config.json)
 =========================================================
 Loads the gitignored config.json at the repo root ONCE per process.
 Deployment surface ONLY: Telegram credentials, data-provider selection
 and API keys, scanned pairs/timeframes, per-pair base blocked hours,
 alert-quality boot default, log level. Signal logic, weights,
 thresholds and gates stay in config.py / pair_signals.
 Precedence for every setting: env var → config.json → code default.
 Secrets have NO code default — with neither env nor config.json the
 bot refuses to start (telegram_bot.main) and data fetches fail with a
 named error (data_provider).
 Parity contract (test-enforced): with no config.json, or one holding
 the config.json.example defaults, every produced PlatformConfig is
 byte-identical to the pure-code defaults. apply_pair_overrides() runs
 in the pair factory BEFORE the validated per-timeframe deltas, so the
 bot, CLI and backtester all see the same overlay (live == backtest).
 Spec: docs/superpowers/specs/2026-07-21-centralized-config-design.md
 """
 import json
 import logging
 import os
 from dataclasses import dataclass, field
 from typing import Any, Dict, List, Optional
 from config import PAIR_INFO, PlatformConfig, Timeframe, XAUUSD
 logger = logging.getLogger(__name__)
 CONFIG_PATH = os.path.join(
    os.path.dirname(os.path.abspath(__file__)), "config.json")
 _VALID_QUALITIES = ("A+", "A", "B", "C")
 _VALID_LOG_LEVELS = ("DEBUG", "INFO", "WARNING", "ERROR", "CRITICAL")
 # Timeframes the bot's candle-boundary scheduler can run (W1 has no loop).
 _SCANNABLE_TFS = ("M5", "M15", "M30", "H1", "H4", "D1")
 _KNOWN_TOP = {"telegram", "data_providers", "scanning", "pairs", "log_level"}
 _KNOWN_TELEGRAM = {"token", "chat_id", "channel_id", "alert_min_quality"}
 _KNOWN_SCANNING = {"pairs", "timeframes"}
 _KNOWN_PAIR_KEYS = {"blocked_hours_utc"}
 class AppConfigError(Exception):
    """config.json is present but invalid — fail fast at boot."""
@dataclass
 class AppConfig:
    telegram_token: str = ""
    telegram_chat_id: str = ""
    telegram_channel_id: str = ""          # "" = channel feed disabled
    alert_min_quality: str = "C"
    active_provider: str = "twelve_data"
    provider_settings: Dict[str, Dict[str, Any]] = field(default_factory=dict)
    scan_pairs: List[str] = field(default_factory=lambda: [XAUUSD])
    scan_timeframes: List[Timeframe] = field(default_factory=lambda: [
        Timeframe.M5, Timeframe.M15, Timeframe.H1, Timeframe.H4])
    pair_overrides: Dict[str, Dict[str, Any]] = field(default_factory=dict)
    log_level: str = "INFO"
 def _check_keys(section: Dict[str, Any], known: set, where: str) -> None:
    unknown = sorted(set(section) - known)
    if unknown:
        raise AppConfigError(
            f"config.json: unknown key(s) {unknown} in {where}; "
            f"known keys: {sorted(known)}")
 def load_app_config(path: str = CONFIG_PATH) -> AppConfig:
    """Parse + validate config.json. Missing file → pure defaults."""
    app = AppConfig()
    if not os.path.exists(path):
        logger.warning(
            "config.json not found at %s — running on code defaults "
            "(secrets must come from environment variables)", path)
        return app
    try:
        with open(path, encoding="utf-8") as fh:
            raw = json.load(fh)
    except json.JSONDecodeError as e:
        raise AppConfigError(f"config.json is not valid JSON: {e}") from e
    if not isinstance(raw, dict):
        raise AppConfigError("config.json top level must be a JSON object")
    _check_keys(raw, _KNOWN_TOP, "top level")
    tg = raw.get("telegram", {})
    _check_keys(tg, _KNOWN_TELEGRAM, '"telegram"')
    app.telegram_token = str(tg.get("token", app.telegram_token))
    app.telegram_chat_id = str(tg.get("chat_id", app.telegram_chat_id))
    app.telegram_channel_id = str(
        tg.get("channel_id", app.telegram_channel_id))
    quality = tg.get("alert_min_quality", app.alert_min_quality)
    if quality not in _VALID_QUALITIES:
        raise AppConfigError(
            f'config.json: telegram.alert_min_quality "{quality}" '
            f"not in {_VALID_QUALITIES}")
    app.alert_min_quality = quality
    dp = raw.get("data_providers")
    if dp is not None:
        if not isinstance(dp, dict):
            raise AppConfigError('config.json: "data_providers" must be an object')
        app.active_provider = str(dp.get("active", app.active_provider))
        app.provider_settings = {}
        for name, block in dp.items():
            if name == "active":
                continue
            if not isinstance(block, dict):
                raise AppConfigError(
                    f'config.json: data_providers.{name} must be an object')
            app.provider_settings[name] = dict(block)
        if app.active_provider not in app.provider_settings:
            raise AppConfigError(
                f'config.json: data_providers.active "{app.active_provider}" '
                "has no matching provider block")
    scanning = raw.get("scanning", {})
    _check_keys(scanning, _KNOWN_SCANNING, '"scanning"')
    if "pairs" in scanning:
        pairs = scanning["pairs"]
        bad = [p for p in pairs if p not in PAIR_INFO]
        if bad:
            raise AppConfigError(
                f"config.json: unknown pair(s) {bad} in scanning.pairs; "
                f"known: {sorted(PAIR_INFO)}")
        app.scan_pairs = list(pairs)
    if "timeframes" in scanning:
        names = scanning["timeframes"]
        bad = [n for n in names if n not in _SCANNABLE_TFS]
        if bad:
            raise AppConfigError(
                f"config.json: unsupported timeframe(s) {bad} in "
                f"scanning.timeframes; supported: {list(_SCANNABLE_TFS)}")
        app.scan_timeframes = [Timeframe[n] for n in names]
    pair_section = raw.get("pairs", {})
    for symbol, overrides in pair_section.items():
        if symbol not in PAIR_INFO:
            raise AppConfigError(
                f'config.json: unknown pair "{symbol}" in pairs; '
                f"known: {sorted(PAIR_INFO)}")
        _check_keys(overrides, _KNOWN_PAIR_KEYS, f'pairs."{symbol}"')
        if "blocked_hours_utc" in overrides:
            hours = overrides["blocked_hours_utc"]
            valid = isinstance(hours, list) and all(
                isinstance(h, int) and not isinstance(h, bool)
                and 0 <= h <= 23 for h in hours)
            if not valid:
                raise AppConfigError(
                    f'config.json: pairs."{symbol}".blocked_hours_utc must '
                    "be a list of integers in 0-23")
    app.pair_overrides = {s: dict(o) for s, o in pair_section.items()}
    level = raw.get("log_level", app.log_level)
    if level not in _VALID_LOG_LEVELS:
        raise AppConfigError(
            f'config.json: log_level "{level}" not in {_VALID_LOG_LEVELS}')
    app.log_level = level
    return app
 # ── Process-wide singleton ────────────────────────────────────────────
 _cached: Optional[AppConfig] = None
 def get_app_config() -> AppConfig:
    global _cached
    if _cached is None:
        _cached = load_app_config()
    return _cached
 def set_app_config(app: Optional[AppConfig]) -> None:
    """Override the process-wide AppConfig (tests/experiments).
    Pass None to re-enable lazy loading from CONFIG_PATH."""
    global _cached
    _cached = app
 # ── Resolution helpers: env var → config.json → default ───────────────
 def telegram_token(app: Optional[AppConfig] = None) -> str:
    app = app or get_app_config()
    return os.environ.get("TELEGRAM_TOKEN") or app.telegram_token
 def telegram_chat_id(app: Optional[AppConfig] = None) -> str:
    app = app or get_app_config()
    return os.environ.get("TELEGRAM_CHAT_ID") or app.telegram_chat_id
 def telegram_channel_id(app: Optional[AppConfig] = None) -> str:
    """Env set-but-empty ("") means "disable the feed" and beats JSON,
    so this checks None-ness, not truthiness."""
    app = app or get_app_config()
    env = os.environ.get("TELEGRAM_CHANNEL_ID")
    return env if env is not None else app.telegram_channel_id
 def provider_api_key(app: Optional[AppConfig] = None) -> str:
    """API key for the ACTIVE provider. The env override is per-provider
    (TWELVE_DATA_API_KEY applies only when twelve_data is active)."""
    app = app or get_app_config()
    if app.active_provider == "twelve_data":
        env = os.environ.get("TWELVE_DATA_API_KEY")
        if env:
            return env
    block = app.provider_settings.get(app.active_provider, {})
    return str(block.get("api_key", ""))
 # ── PlatformConfig overlay ────────────────────────────────────────────
 def apply_pair_overrides(
    cfg: PlatformConfig, app: Optional[AppConfig] = None
 ) -> PlatformConfig:
    """Stamp deployment settings from config.json onto a PlatformConfig.
    Called by the pair factories BEFORE their validated per-timeframe
    deltas, so e.g. the M5 hour-9 append lands on top of the JSON base.
    The pair's base blocked_hours_utc is set on BOTH mirrored copies
    (xau_strategy + backtest) so they cannot drift.
    """
    app = app or get_app_config()
    overrides = app.pair_overrides.get(cfg.symbol, {})
    if "blocked_hours_utc" in overrides:
        hours = list(overrides["blocked_hours_utc"])
        cfg.xau_strategy.blocked_hours_utc = hours
        cfg.backtest.blocked_hours_utc = list(hours)
    cfg.scan_pairs = list(app.scan_pairs)
    cfg.scan_timeframes = list(app.scan_timeframes)
    return cfg
 ```
 - [ ] **Step 5: Run tests, verify Module 13 passes and total is 190**
 ```bash
 python test_all.py 2>&1 | tail -25
 ```
 Expected: `RESULTS: 190 passed, 0 failed, 190 total` (184 baseline + 6 new). All six `app_config:` lines show `✓`.
 - [ ] **Step 6: Commit**
 ```bash
 git add app_config.py test_all.py
 git commit -m "Add app_config.py: config.json loader, validation, env-precedence resolvers"
 ```
 ---
 ### Task 2: Factory overlay wiring + parity pins
 **Files:**
 - Modify: `pair_signals/xauusd.py:81-90` (make_config), `test_all.py` (Module 9 equivalence test ~line 1851; Module 13 additions)
 **Interfaces:**
 - Consumes: `app_config.apply_pair_overrides`, `app_config.set_app_config`, `app_config.AppConfig`, `app_config.load_app_config` (Task 1).
 - Produces: `make_config` now applies the overlay — every consumer (bot, CLI, backtester) inherits it with no further change. Test symbol `_TEST_APP` (Task 1) is the restore target for any test that swaps the singleton.
 - [ ] **Step 1: Add failing tests (Module 13 additions)**
 Append to Module 13 in `test_all.py`:
 ```python
@test("app_config: apply_pair_overrides stamps both blocked-hours mirrors")
 def _():
    app = AppConfig(pair_overrides={XAUUSD: {"blocked_hours_utc": [3, 4]}},
                    scan_pairs=[XAUUSD],
                    scan_timeframes=[Timeframe.M15])
    cfg = apply_pair_overrides(PlatformConfig(), app)
    assert cfg.xau_strategy.blocked_hours_utc == [3, 4]
    assert cfg.backtest.blocked_hours_utc == [3, 4]
    assert cfg.scan_timeframes == [Timeframe.M15]
    # the two mirrors must be independent lists, not aliases
    cfg.xau_strategy.blocked_hours_utc.append(5)
    assert cfg.backtest.blocked_hours_utc == [3, 4]
@test("app_config: M5 factory appends hour 9 on top of a JSON base")
 def _():
    from pair_signals import PAIR_PRODUCERS
    app = AppConfig(pair_overrides={XAUUSD: {"blocked_hours_utc": [3, 4]}},
                    provider_settings={"twelve_data": {"api_key": "test-key"}})
    set_app_config(app)
    try:
        cfg = PAIR_PRODUCERS[XAUUSD](Timeframe.M5, None)
    finally:
        set_app_config(_TEST_APP)
    assert cfg.xau_strategy.blocked_hours_utc == [3, 4, 9], \
        "validated M5 hour-9 delta must land on top of the JSON base"
    assert cfg.backtest.blocked_hours_utc == [3, 4], \
        "mirror carries the base only (per-TF delta is XAU-strategy-side)"
@test("app_config: config.json.example is valid and behavior-neutral")
 def _():
    import dataclasses
    from pair_signals import PAIR_PRODUCERS
    example = load_app_config(
        os.path.join(os.path.dirname(__file__), "config.json.example"))
    for tf in (Timeframe.M5, Timeframe.M15, Timeframe.H1, Timeframe.H4):
        baseline = PAIR_PRODUCERS[XAUUSD](tf, None)   # under _TEST_APP
        set_app_config(example)
        try:
            with_example = PAIR_PRODUCERS[XAUUSD](tf, None)
        finally:
            set_app_config(_TEST_APP)
        assert dataclasses.asdict(baseline) == dataclasses.asdict(with_example), \
            f"example config.json must be byte-identical to defaults on {tf}"
 ```
 - [ ] **Step 2: Run to verify the new tests fail**
 ```bash
 python test_all.py 2>&1 | grep "app_config: M5 factory"
 ```
 Expected: `✗ app_config: M5 factory appends hour 9 on top of a JSON base` (the factory doesn't apply the overlay yet, so blocked hours are the defaults `[0,1,2,10,11,13,16,21,9]`, not `[3,4,9]`).
 - [ ] **Step 3: Wire the overlay into the factory**
 In `pair_signals/xauusd.py`, change the import line and the top of `make_config`:
 ```python
 from typing import Optional, List
 from config import PlatformConfig, Timeframe
 from app_config import apply_pair_overrides
 ```
 and in `make_config`, immediately after `config = PlatformConfig()`:
 ```python
    config = PlatformConfig()
    # Deployment overlay (config.json) BEFORE the validated per-TF
    # deltas below, so e.g. the M5 hour-9 append lands on the JSON base.
    apply_pair_overrides(config)
    config.primary_timeframe = primary_tf
 ```
 - [ ] **Step 4: Update the Module 9 equivalence test**
 In `test_all.py` (~line 1851, `"xauusd.make_config equals inline PlatformConfig construction"`), the inline construction must apply the same overlay in the same order. After the `inline_cfg = PlatformConfig(...)` construction and BEFORE the per-TF override lines, add:
 ```python
        app_config.apply_pair_overrides(inline_cfg)
 ```
 (`app_config` is already imported at the top of `test_all.py` from Task 1. Under `_TEST_APP` this is a no-op on blocked hours, but it keeps the equivalence definition exact: factory ≡ inline + overlay + per-TF deltas, whatever the active AppConfig.)
 - [ ] **Step 5: Run tests, verify 193 passing**
 ```bash
 python test_all.py 2>&1 | tail -25
 ```
 Expected: `RESULTS: 193 passed, 0 failed, 193 total`. Watch specifically for `✓ xauusd.make_config equals inline PlatformConfig construction` (Module 9) and the three new Module 13 lines.
 - [ ] **Step 6: Commit**
 ```bash
 git add pair_signals/xauusd.py test_all.py
 git commit -m "Wire config.json overlay into the pair factory (before per-TF deltas)"
 ```
 ---
 ### Task 3: Data provider — key from app_config, registry, delete hardcoded key
 **Files:**
 - Modify: `config.py:346` (delete field), `data_provider.py:72-76` (key resolution), `data_provider.py:265-269` (error message), `data_provider.py` end (registry), `signal_aggregator.py:107`, `backtest/gpu_backtest.py:1682`, `test_all.py` (Module 13 additions)
 **Interfaces:**
 - Consumes: `app_config.provider_api_key`, `app_config.get_app_config`, `app_config.AppConfigError` (Task 1).
 - Produces: `data_provider.PROVIDER_REGISTRY: Dict[str, type]` and `data_provider.make_data_provider(config: PlatformConfig) -> ForexDataProvider` — Task 4 uses `make_data_provider` in `telegram_bot.py`.
 - [ ] **Step 1: Add failing tests (Module 13 additions)**
 ```python
@test("data_provider: api key resolves env → config.json (no code default)")
 def _():
    from data_provider import ForexDataProvider
    app = AppConfig(provider_settings={"twelve_data": {"api_key": "json-key"}})
    set_app_config(app)
    try:
        with patch.dict(os.environ, {}, clear=True):
            dp = ForexDataProvider(PlatformConfig())
            assert dp._api_key == "json-key"
        with patch.dict(os.environ, {"TWELVE_DATA_API_KEY": "env-key"}):
            dp = ForexDataProvider(PlatformConfig())
            assert dp._api_key == "env-key"
        set_app_config(AppConfig())        # no key anywhere
        with patch.dict(os.environ, {}, clear=True):
            dp = ForexDataProvider(PlatformConfig())
            assert dp._api_key is None and dp._td is None
    finally:
        set_app_config(_TEST_APP)
@test("data_provider: make_data_provider dispatches on active provider")
 def _():
    from data_provider import ForexDataProvider, make_data_provider
    assert isinstance(make_data_provider(PlatformConfig()), ForexDataProvider)
    set_app_config(AppConfig(active_provider="polygon",
                             provider_settings={"polygon": {"api_key": "k"}}))
    try:
        make_data_provider(PlatformConfig())
        assert False, "unregistered provider must raise"
    except AppConfigError as e:
        assert "polygon" in str(e)
    finally:
        set_app_config(_TEST_APP)
@test("config.py: hardcoded twelve_data_api_key field is gone")
 def _():
    assert not hasattr(PlatformConfig(), "twelve_data_api_key"), \
        "credentials must not live in committed source"
 ```
 - [ ] **Step 2: Run to verify they fail**
 ```bash
 python test_all.py 2>&1 | grep -A1 "hardcoded twelve_data"
 ```
 Expected: `✗ config.py: hardcoded twelve_data_api_key field is gone` (plus the make_data_provider import failure).
 - [ ] **Step 3: Implement**
 `config.py`: delete line 346 (`twelve_data_api_key: str = "<TWELVE_DATA_API_KEY>"`).
 `data_provider.py` — add to the local imports:
 ```python
 from app_config import AppConfigError, get_app_config, provider_api_key
 ```
 In `ForexDataProvider.__init__` replace the key resolution:
 ```python
        self.config = config
        self._api_key = provider_api_key() or None
 ```
 In `_fetch_twelve_data`, update the no-key error:
 ```python
        if self._td is None:
            raise DataIntegrityError(
                "No data-provider API key. Set the TWELVE_DATA_API_KEY env "
                "var or data_providers.twelve_data.api_key in config.json."
            )
 ```
 At the bottom of `data_provider.py` (after the class):
 ```python
 # ── Provider registry ─────────────────────────────────────────────────
 # config.json data_providers.active selects from here. Adding a provider
 # later = one adapter class + one entry + one config block.
 PROVIDER_REGISTRY: Dict[str, type] = {"twelve_data": ForexDataProvider}
 def make_data_provider(config: PlatformConfig) -> ForexDataProvider:
    """Build the ACTIVE data provider (config.json: data_providers.active)."""
    name = get_app_config().active_provider
    cls = PROVIDER_REGISTRY.get(name)
    if cls is None:
        raise AppConfigError(
            f'Unknown data provider "{name}"; '
            f"registered: {sorted(PROVIDER_REGISTRY)}")
    return cls(config)
 ```
 `signal_aggregator.py` line 18 + 107 — import and use the registry:
 ```python
 from data_provider import ForexDataProvider, make_data_provider
 ```
 ```python
        self.data_provider = make_data_provider(config)
 ```
 `backtest/gpu_backtest.py` line 1682 (and its import of `ForexDataProvider`):
 ```python
    provider = make_data_provider(platform_cfg)
 ```
 - [ ] **Step 4: Run tests, verify 196 passing**
 ```bash
 python test_all.py 2>&1 | tail -25
 ```
 Expected: `RESULTS: 196 passed, 0 failed, 196 total`. If any Module 2 test fails on a missing key, it means it constructed `ForexDataProvider` outside `_TEST_APP` scope — fix the test to run under `_TEST_APP` (which supplies the dummy key), never by re-adding a code default.
 - [ ] **Step 5: Commit**
 ```bash
 git add config.py data_provider.py signal_aggregator.py backtest/gpu_backtest.py test_all.py
 git commit -m "Data provider: key from config.json/env, provider registry, drop hardcoded key"
 ```
 ---
 ### Task 4: telegram_bot wiring + startup guard
 **Files:**
 - Modify: `telegram_bot.py` (credentials block ~93-105, logging ~88, SCAN_PAIRS ~113, alert quality ~478, WS key ~611-614, `_AUTOSCAN_LOOPS` ~797-802, scheduler log ~817, `_get_shared_dp` ~195, `main()` ~1261), `test_all.py` (Module 13 addition)
 **Interfaces:**
 - Consumes: `app_config.get_app_config`, resolvers, `data_provider.make_data_provider` (Tasks 1+3).
 - Produces: nothing consumed later; this is the final wiring task.
 - [ ] **Step 1: Add failing test (Module 13 addition)**
 ```python
@test("telegram_bot: credentials, quality, pairs, loops from app_config")
 def _():
    import importlib
    import telegram_bot
    app = AppConfig(telegram_token="json-tok", telegram_chat_id="42",
                    telegram_channel_id="", alert_min_quality="B",
                    scan_timeframes=[Timeframe.M15, Timeframe.H1],
                    provider_settings={"twelve_data": {"api_key": "k"}})
    set_app_config(app)
    try:
        with patch.dict(os.environ, {}, clear=True):
            tb = importlib.reload(telegram_bot)
            assert tb.TOKEN == "json-tok"
            assert tb.OWNER_ID == 42
            assert tb.CHANNEL_ID is None, '"" channel id must disable the feed'
            assert tb._alert_min_quality == "B"
            assert [tf for tf, _ in tb._AUTOSCAN_LOOPS] == \
                [Timeframe.M15, Timeframe.H1]
            assert tb._AUTOSCAN_LOOPS == [(Timeframe.M15, 15),
                                          (Timeframe.H1, 60)]
    finally:
        set_app_config(_TEST_APP)
        importlib.reload(telegram_bot)
@test("telegram_bot: main() refuses to start without credentials")
 def _():
    import telegram_bot
    if telegram_bot.TOKEN:      # only meaningful in the hermetic no-cred env
        return
    try:
        telegram_bot.main()
        assert False, "main() must refuse to start without token/chat_id"
    except SystemExit as e:
        assert "credentials" in str(e)
 ```
 - [ ] **Step 2: Run to verify failure**
 ```bash
 python test_all.py 2>&1 | grep "telegram_bot: credentials"
 ```
 Expected: `✗ telegram_bot: credentials, quality, pairs, loops from app_config` (TOKEN still comes from the hardcoded fallback, not "json-tok").
 - [ ] **Step 3: Implement the wiring**
 `telegram_bot.py` — add to the local imports (after the `config` import block):
 ```python
 from app_config import (
    get_app_config, telegram_token, telegram_chat_id, telegram_channel_id,
 )
 from data_provider import ForexDataProvider, make_data_provider
 ```
 Immediately after the imports (before the logging block), add:
 ```python
 _APP = get_app_config()
 ```
 Change the `logging.basicConfig` line (~88) to honor the configured level:
 ```python
 logging.basicConfig(level=getattr(logging, _APP.log_level, logging.INFO),
                    handlers=[_console_handler, _file_handler])
 ```
 Replace the credentials block (lines ~93-105) — keep the existing comments about the owner lock and subscriber channel, change only the assignments:
 ```python
 TOKEN = telegram_token()
 CHAT_ID = telegram_chat_id()
 # ... (owner-lock comment unchanged)
 OWNER_ID: Optional[int] = int(CHAT_ID) if CHAT_ID else None
 # ... (subscriber-channel comment unchanged)
 _CHANNEL_ID_RAW = telegram_channel_id()
 CHANNEL_ID: Optional[int] = int(_CHANNEL_ID_RAW) if _CHANNEL_ID_RAW else None
 ```
 `SCAN_PAIRS` (~113):
 ```python
 SCAN_PAIRS = list(_APP.scan_pairs)
 ```
 `_get_shared_dp` (~195):
 ```python
        _shared_dp = make_data_provider(PlatformConfig())
 ```
 Alert quality boot default (~478):
 ```python
 _alert_min_quality = _APP.alert_min_quality
 ```
 WS key (~611-614) — `provider_api_key` needs adding to the `app_config` import list:
 ```python
 _cfg = PlatformConfig()
 _WS_API_KEY = provider_api_key()
 _WS_URL = f"wss://ws.twelvedata.com/v1/quotes/price?apikey={_WS_API_KEY}"
 _WS_COOLDOWN = _cfg.ws_cooldown_seconds
 ```
 `_AUTOSCAN_LOOPS` (~797-802) — derive from configured timeframes:
 ```python
 _TF_INTERVAL_MINUTES = {
    Timeframe.M5: 5, Timeframe.M15: 15, Timeframe.M30: 30,
    Timeframe.H1: 60, Timeframe.H4: 240, Timeframe.D1: 1440,
 }
 _AUTOSCAN_LOOPS = [
    (tf, _TF_INTERVAL_MINUTES[tf]) for tf in _APP.scan_timeframes
 ]
 ```
 Scheduler log line (~817):
 ```python
        logger.info("Candle-boundary scheduler started (%s)",
                    "/".join(tf.name for tf, _ in _AUTOSCAN_LOOPS))
 ```
 (The manual `/scan` buttons — `_TF_BUTTON_LABELS` — deliberately stay hardcoded M5/M15/H1/H4: they are the owner's on-demand UI, not the automated feed. Documented decision, not an omission.)
 `main()` (~1261) — first lines of the function body:
 ```python
 def main() -> None:
    if not TOKEN or not CHAT_ID:
        raise SystemExit(
            "telegram_bot: missing Telegram credentials. Set telegram.token "
            "and telegram.chat_id in config.json, or the TELEGRAM_TOKEN and "
            "TELEGRAM_CHAT_ID environment variables.")
 ```
 - [ ] **Step 4: Run tests, verify 198 passing**
 ```bash
 python test_all.py 2>&1 | tail -25
 ```
 Expected: `RESULTS: 198 passed, 0 failed, 198 total`. If Module 7 tests break on `OWNER_ID` being `None`, they were relying on the hardcoded chat id — fix by having those tests set `telegram_bot.OWNER_ID`/`CHAT_ID` explicitly in their setup, never by re-adding fallbacks.
 - [ ] **Step 5: Commit**
 ```bash
 git add telegram_bot.py test_all.py
 git commit -m "telegram_bot: credentials/quality/scan-loops from config.json, startup guard"
 ```
 ---
 ### Task 5: Migration, docs, final verification
 **Files:**
 - Create: `config.json` (LOCAL ONLY — gitignored, never committed)
 - Modify: `CLAUDE.md` (repo map, env-var table, config section, test count)
 **Interfaces:**
 - Consumes: everything above.
 - Produces: the deployed machine's real `config.json`; updated authoritative docs.
 - [ ] **Step 1: Create the local config.json with the current live values**
 These are the exact values being deleted from source (from pre-change `telegram_bot.py:93-105` and `config.py:346`), so the running bot behaves identically:
 ```bash
 cat > /home/harwinder/Development/ForexBot2/config.json <<'EOF'
 {
  "telegram": {
    "token": "<TELEGRAM_BOT_TOKEN>",
    "chat_id": "<OWNER_CHAT_ID>",
    "channel_id": "<CHANNEL_ID>",
    "alert_min_quality": "C"
  },
  "data_providers": {
    "active": "twelve_data",
    "twelve_data": { "api_key": "<TWELVE_DATA_API_KEY>" }
  },
  "scanning": {
    "pairs": ["XAU/USD"],
    "timeframes": ["M5", "M15", "H1", "H4"]
  },
  "pairs": {
    "XAU/USD": { "blocked_hours_utc": [0, 1, 2, 10, 11, 13, 16, 21] }
  },
  "log_level": "INFO"
 }
 EOF
 git status --short   # config.json must NOT appear (gitignored)
 ```
 (Real values were scrubbed from this doc 2026-07-21 — take them from the previously deployed telegram_bot.py/config.py in git history, and rotate both credentials after merge.)
 - [ ] **Step 2: Boot smoke test (no network needed)**
 ```bash
 python - <<'EOF'
 from app_config import get_app_config, telegram_token, provider_api_key, apply_pair_overrides
 from pair_signals import PAIR_PRODUCERS
 from config import Timeframe, XAUUSD
 app = get_app_config()
 assert telegram_token(app), "token missing"
 assert provider_api_key(app), "api key missing"
 cfg = PAIR_PRODUCERS[XAUUSD](Timeframe.M5, None)
 assert cfg.xau_strategy.blocked_hours_utc == [0, 1, 2, 10, 11, 13, 16, 21, 9]
 print("boot smoke OK — live config.json loads and factory output unchanged")
 EOF
 ```
 Expected: `boot smoke OK — live config.json loads and factory output unchanged`.
 - [ ] **Step 3: Update CLAUDE.md**
 - Repository-map table: add rows for `app_config.py` ("config.json loader/validation, env-precedence resolvers, `apply_pair_overrides` deployment overlay") and `config.json.example` ("committed schema template; real gitignored config.json holds secrets + deployment settings").
 - Environment-variables table: note precedence "env var → config.json → none"; state that hardcoded fallbacks were REMOVED 2026-07-21 and the bot refuses to boot without credentials.
 - `PlatformConfig` section: delete the `twelve_data_api_key` mention ("hardcoded fallback"); add one line: "Deployment settings (credentials, scan pairs/TFs, per-pair base blocked hours, alert quality, log level) overlay from config.json via `app_config.apply_pair_overrides`, applied in the pair factory BEFORE per-TF deltas."
 - `pair_signals` section: mention the overlay call order in `make_config`.
 - Test-suite line: update "184 tests in 12 modules" → "198 tests in 13 modules" (verify against the actual final count from Step 4 before writing).
 - [ ] **Step 4: Full-suite final verification**
 ```bash
 python test_all.py 2>&1 | tail -15
 ```
 Expected: `RESULTS: 198 passed, 0 failed, 198 total` (use the real number if it differs, and make CLAUDE.md match).
 - [ ] **Step 5: Commit docs**
 ```bash
 git add CLAUDE.md
 git commit -m "Docs: config.json deployment overlay (CLAUDE.md repo map, env table, test count)"
 ```
 - [ ] **Step 6: Merge/PR decision**
 Implementation complete → follow superpowers:finishing-a-development-branch (options: merge `centralized-config` to main, or push the branch to `origin`/`gitea` for review). After merge, remind the user: **rotate the Telegram token and Twelve Data key** — the old values remain in git history on both remotes.
@@ -69,9 +69,8 @@
 <tr><td><b>Score / confidence</b></td><td>The regime-weighted composite of all nine indicators. Bigger absolute score = stronger consensus.</td></tr>
 <tr><td><b>MTF aligned</b></td><td>The two confirmation timeframes agree with the trade direction. Signals without alignment are filtered out on M5.</td></tr>
 <tr><td><b>Warnings ⚠️</b></td><td>Context, not veto. Backtesting shows warned trades perform about the same — strong momentum often triggers warnings.</td></tr>
 <tr><td><b>Market context 📰</b></td><td>An AI-written one-liner summarizing today's macro backdrop (headlines + high-impact releases) and its likely pull on gold. Informational only — it never blocks or grades a signal.</td></tr>
 <tr><td><b>SL / TP1 / RR</b></td><td>Stop loss sits behind the nearest swing level with an ATR buffer. TP1 is the nearest opposing level. <b>1R</b> = the entry-to-SL distance = the amount you risk.</td></tr>
-<tr><td><b>Trail plan 🔄</b></td><td>The real exit. Once price moves +0.6R in your favor, move the stop to trail 0.75R behind the best price and update it at each candle close. <b>TP1 is advisory — let winners run.</b></td></tr>
+<tr><td><b>Trail plan 🔄</b></td><td>The real exit. Once price moves +0.6R in your favor (+0.8R for pullback setups), move the stop to trail 0.75R behind the best price and update it at each candle close. <b>TP1 is advisory — let winners run.</b></td></tr>
 </table>
 <h2>3 · Acting on a signal — step by step</h2>
@@ -130,7 +129,6 @@
 <tr><td><code>/scan</code></td><td>Run an immediate analysis of the primary timeframe.</td></tr>
 <tr><td><code>/autoscan</code></td><td>Toggle automatic scanning (M5 / M15 / H1 / H4 at each candle close).</td></tr>
 <tr><td><code>/status</code></td><td>Uptime, auto-scan state, WebSocket price-monitor state.</td></tr>
 <tr><td><code>/news</code></td><td>Net XAU/USD news sentiment score (informational only — never gates signals).</td></tr>
 <tr><td><code>/cache</code> · <code>/clearcache</code></td><td>Inspect / reset the market-data cache.</td></tr>
 <tr><td><code>/help</code></td><td>Command reference.</td></tr>
 </table>
@@ -245,12 +245,6 @@ class TechnicalEngine:
            signal = SignalStrength.NEUTRAL
            conf = 40
        # Partial alignment (plain BUY/SELL, 2-3 of 4 checks) carries no
        # edge on M5 — squash to NEUTRAL when the research knob is on.
        if c.ema_partial_neutral and signal in (SignalStrength.BUY, SignalStrength.SELL):
            signal = SignalStrength.NEUTRAL
            conf = 40
        crossover_desc = ""
        if len(ema_fast) >= 3:
            prev_f, prev_m = ema_fast.iloc[-2], ema_medium.iloc[-2]
@@ -1,195 +0,0 @@
 """
 ForexBot2 — DeepSeek LLM client
 ================================
 Thin requests-based client for DeepSeek's OpenAI-compatible chat API,
 used by `news_engine` for display/warning-only enrichment (headline
 sentiment, event surprise direction, market summary). Signals are NEVER
 gated on LLM output — every caller has a non-LLM fallback and treats a
 `None` return as "use the fallback".
 Design constraints:
 - One attempt per call, no retries: news runs inside scan cycles; the
  news engine's 600 s context cache is the natural retry cadence.
 - A UTC-day request cap bounds spend even if a bug causes call storms.
 - Cost: DeepSeek caches identical prompt PREFIXES server-side (cache-hit
  input is ~50x cheaper), so callers pass a STATIC system prompt (no
  timestamps or interpolation) and put all variable content in the user
  payload.
 This is a module-level singleton (`get_llm_client`) because the daily
 cap counter must survive the per-scan reconstruction of `NewsEngine`.
 """
 import datetime as _dt
 import json
 import logging
 import threading
 import time
 from typing import Any, Dict, Optional, Tuple
 import requests
 from app_config import deepseek_api_key, get_app_config
 logger = logging.getLogger(__name__)
 DEEPSEEK_URL = "https://api.deepseek.com/chat/completions"
 DEFAULT_MODEL = "deepseek-v4-flash"
 # 8s timed out on a measured cold-prefix-cache headline batch (2026-08-03);
 # warm-cache calls come back well under that. 12s buys cold-start headroom
 # while still bounding worst-case scan latency (≤3 calls per cache window).
 DEFAULT_TIMEOUT = 12.0
 DEFAULT_DAILY_CAP = 500
 class DeepSeekClient:
    """JSON-in/JSON-out chat wrapper. All failures return None."""
    def __init__(
        self,
        api_key: str,
        model: str = DEFAULT_MODEL,
        timeout: float = DEFAULT_TIMEOUT,
        daily_cap: int = DEFAULT_DAILY_CAP,
        enabled: bool = True,
    ):
        self._api_key = api_key or ""
        self._model = model
        self._timeout = timeout
        self._daily_cap = daily_cap
        self._enabled = enabled
        self._session = requests.Session()
        self._session.headers.update({
            "Content-Type": "application/json",
            "Authorization": f"Bearer {self._api_key}",
        })
        self._lock = threading.Lock()
        self._count_day: Optional[_dt.date] = None
        self._count = 0
        self._warned: set = set()
        # Health stamps for status displays: last successful call and
        # last failure (ts, kind). Read-only for consumers.
        self.last_ok: float = 0.0
        self.last_error: Optional[Tuple[float, str]] = None
    @property
    def available(self) -> bool:
        return self._enabled and bool(self._api_key)
    def _warn_once(self, kind: str, msg: str, *args: Any) -> None:
        if kind in self._warned:
            logger.debug(msg, *args)
        else:
            self._warned.add(kind)
            logger.warning(msg, *args)
    def _under_cap(self) -> bool:
        """Count a request against the UTC-day cap; False = don't call."""
        today = _dt.datetime.now(_dt.timezone.utc).date()
        with self._lock:
            if self._count_day != today:
                self._count_day = today
                self._count = 0
                self._warned.discard("cap")
            if self._count >= self._daily_cap:
                self._warn_once(
                    "cap", "DeepSeek daily request cap (%d) exhausted — "
                    "falling back to keyword analysis until UTC midnight",
                    self._daily_cap)
                return False
            self._count += 1
            return True
    def chat_json(self, system_prompt: str,
                  user_payload: Dict[str, Any]) -> Optional[Dict[str, Any]]:
        """One chat call; returns the model's JSON object or None."""
        if not self.available or not self._under_cap():
            return None
        body = {
            "model": self._model,
            "messages": [
                {"role": "system", "content": system_prompt},
                {"role": "user",
                 "content": json.dumps(user_payload, sort_keys=True)},
            ],
            "response_format": {"type": "json_object"},
            "temperature": 0,
            # V4-Flash is a hybrid thinking model; without this flag it
            # burns hundreds of reasoning tokens per call — measured
            # 2026-08-03: a 25-headline batch blew straight through the
            # timeout AND could exhaust max_tokens before the answer,
            # returning empty content. The doctrine + few-shot examples
            # in the system prompts carry the quality instead.
            "thinking": {"type": "disabled"},
            # Must fit the largest reply: a full sentiment batch is
            # ~10 tokens per headline; 1000 truncated mid-JSON on a real
            # RSS cycle (2026-08-03).
            "max_tokens": 2000,
        }
        try:
            resp = self._session.post(
                DEEPSEEK_URL, json=body, timeout=self._timeout)
        except requests.RequestException as e:
            self.last_error = (time.time(), "network")
            self._warn_once("network", "DeepSeek call failed: %s", e)
            return None
        if resp.status_code != 200:
            self.last_error = (time.time(), f"http_{resp.status_code}")
            self._warn_once(
                f"http_{resp.status_code}",
                "DeepSeek HTTP %s: %s", resp.status_code, resp.text[:200])
            return None
        try:
            parsed = self._parse_response(resp.json())
        except ValueError:
            parsed = None
        if parsed is None:
            self.last_error = (time.time(), "parse")
            self._warn_once("parse", "DeepSeek response was not the "
                            "expected JSON object — ignoring")
        else:
            self.last_ok = time.time()
        return parsed
    @staticmethod
    def _parse_response(body: Any) -> Optional[Dict[str, Any]]:
        """Extract the assistant's JSON object from a chat completion."""
        try:
            content = body["choices"][0]["message"]["content"]
        except (KeyError, IndexError, TypeError):
            return None
        try:
            obj = json.loads(content)
        except (TypeError, ValueError):
            return None
        return obj if isinstance(obj, dict) else None
 # ── Process-wide singleton ────────────────────────────────────────────
 _client: Optional[DeepSeekClient] = None
 _client_lock = threading.Lock()
 def get_llm_client() -> DeepSeekClient:
    global _client
    with _client_lock:
        if _client is None:
            settings = get_app_config().deepseek_settings
            _client = DeepSeekClient(
                api_key=deepseek_api_key(),
                model=str(settings.get("model", DEFAULT_MODEL)),
                timeout=float(settings.get("timeout_seconds",
                                           DEFAULT_TIMEOUT)),
                daily_cap=int(settings.get("daily_request_cap",
                                           DEFAULT_DAILY_CAP)),
                enabled=settings.get("enabled", True) is not False,
            )
        return _client
 def reset_llm_client() -> None:
    """Drop the singleton (tests / config reloads)."""
    global _client
    with _client_lock:
        _client = None
@@ -161,8 +161,6 @@ def print_signal_report(report: SignalReport, detail: bool = False):
        print(f"  {Colors.YELLOW}📅{Colors.RESET} {report.event_description}")
    if report.news_description and report.news_description != "Neutral news sentiment":
        print(f"  {Colors.MAGENTA}📰{Colors.RESET} {report.news_description}")
    if report.news_summary:
        print(f"  {Colors.MAGENTA}🧠{Colors.RESET} {report.news_summary}")
    # ── 6. DETAIL (only with --detail) ─────────────────────────────
    if detail and report.primary_analysis:
@@ -208,7 +206,6 @@ def report_to_dict(report: SignalReport) -> dict:
        "event_description": report.event_description,
        "news_sentiment": round(report.news_sentiment, 3),
        "news_description": report.news_description,
        "news_summary": report.news_summary,
        "bullish_reasons": report.bullish_reasons,
        "bearish_reasons": report.bearish_reasons,
        "warnings": report.warnings,
@@ -8,8 +8,6 @@ Key principle: High-impact news events override technical signals.
 A perfect technical setup can be destroyed by an unexpected NFP print.
 """
 import hashlib
 import html
 import logging
 import re
 import threading
@@ -23,72 +21,10 @@ from typing import Dict, List, Optional, Tuple
 import requests
 from config import NewsConfig
 from llm_client import get_llm_client
 logger = logging.getLogger(__name__)
 # ── DeepSeek prompts ─────────────────────────────────────────────────
 # STATIC constants on purpose: DeepSeek prices a repeated byte-identical
 # prompt prefix at the cache-hit rate (~50x cheaper), so nothing here may
 # be interpolated or timestamped. All variable content travels in the
 # user payload. LLM output is display/warning-only — it never gates
 # signals (see NewsConfig.sentiment_affects_signals).
 _LLM_SENTIMENT_SYSTEM = """\
 You are a forex/gold news analyst. You receive JSON:
 {"headlines": ["...", ...]}
 For EACH headline, in the same order, score:
 - "s": overall market sentiment of the headline, -1.0 (very bearish)
  to 1.0 (very bullish), 0.0 if irrelevant or neutral.
 - "xau": the effect on the GOLD (XAU/USD) price, -1.0 (gold down)
  to 1.0 (gold up), 0.0 if irrelevant.
 Gold doctrine: hawkish Fed / rate hikes / rising real yields / strong
 dollar are gold-bearish; easing / rate cuts / safe-haven flows /
 geopolitical stress / weak dollar are gold-bullish.
 Handle negation correctly ("no rate hike" is not hawkish).
 Reply with ONLY JSON: {"scores": [[s, xau], ...]} — exactly one
 [s, xau] pair per input headline, in the same order, nothing else.
 Examples:
 "Fed signals faster rate hikes as inflation stays hot" -> [0.4, -0.7]
 "Gold surges to record high on safe-haven demand" -> [0.8, 0.9]
 "Dollar slips after soft jobs report; Treasury yields fall" -> [-0.4, 0.6]
 """
 _LLM_EVENT_SYSTEM = """\
 You are a forex/gold analyst. You receive one released economic event as
 JSON: {"event", "currency", "actual", "forecast", "previous",
 "vs_forecast"}. "vs_forecast" states whether the actual print came in
 above or below the forecast; it is computed programmatically — trust it
 over your own comparison of the numbers. If it is "unknown", compare
 actual and forecast yourself.
 Decide the likely effect of this RELEASE on the GOLD (XAU/USD) price.
 A stronger-than-forecast USD economy or a hawkish implication is usually
 gold-bearish; weaker data or a dovish implication is usually
 gold-bullish.
 Inverted metrics — for unemployment rate and unemployment/jobless
 claims a LOWER number means a STRONGER labor market: below forecast =
 strong economy = usually gold-bearish; above forecast = weak economy =
 usually gold-bullish.
 Example: {"event": "Unemployment Claims", "actual": "202K", "forecast":
 "217K", "vs_forecast": "below forecast"} ->
 {"gold_direction": "bearish", "reason": "fewer claims show strong labor market"}
 Reply with ONLY JSON:
 {"gold_direction": "bullish"|"bearish"|"neutral", "reason": "<max 10 words>"}
 """
 _LLM_SUMMARY_SYSTEM = """\
 You are a forex/gold market-context writer. You receive JSON with recent
 "headlines" and today's high-impact "events" (released: true means the
 number is out). Write a 1-2 sentence plain-English summary of the
 current macro backdrop for a GOLD (XAU/USD) trader: the dominant theme
 and its likely pull on gold. No advice, no hedging boilerplate, max 45
 words.
 Reply with ONLY JSON: {"summary": "..."}
 """
 _RELEASE_VALUE_RE = re.compile(r"(-?\d+(?:\.\d+)?)\s*([kmbt])?", re.IGNORECASE)
 _RELEASE_MULTIPLIERS = {"k": 1e3, "m": 1e6, "b": 1e9, "t": 1e12}
@@ -104,44 +40,6 @@ def _parse_release_value(raw: Optional[str]) -> Optional[float]:
    return float(match.group(1)) * multiplier
 _DESC_MAX_CHARS = 500
 _MAX_LINK_CHARS = 2048
 def _clean_html_text(raw: Optional[str],
                     limit: int = _DESC_MAX_CHARS) -> Optional[str]:
    """RSS <description> → plain text: unescape entities, strip tags,
    collapse whitespace, truncate. Unescape and strip both run twice
    because feeds routinely double-encode (&amp;lt;p&amp;gt; wrapping
    real markup) — the second unescape can reveal fresh tags."""
    if not raw:
        return None
    text = html.unescape(raw)
    text = re.sub(r"<[^>]+>", " ", text)
    text = html.unescape(text)
    text = re.sub(r"<[^>]+>", " ", text)
    text = re.sub(r"\s+", " ", text).strip()
    if not text:
        return None
    if len(text) > limit:
        text = text[:limit].rstrip() + "…"
    return text
 def _safe_link(raw: Optional[str]) -> Optional[str]:
    """Article URL for display: http(s) only (an RSS <link> is untrusted
    and lands in an href), bounded length (Google News redirect URLs run
    1-2 KB; anything past that is bloat, not a link)."""
    if not raw:
        return None
    link = raw.strip()
    if not link.lower().startswith(("http://", "https://")):
        return None
    if len(link) > _MAX_LINK_CHARS:
        return None
    return link
@dataclass
 class EconomicEvent:
    """Single economic calendar event."""
@@ -194,11 +92,6 @@ class NewsSentiment:
    # Per-currency score: monetary-policy keywords flip sign for XAU
    # (hawkish is dollar-bullish but gold-bearish).
    sentiment_by_currency: Dict[str, float] = field(default_factory=dict)
    # Display-only enrichment (dashboard detail dialog). Optional with
    # defaults so existing constructor calls keep working.
    link: Optional[str] = None         # article URL (Google News = redirect)
    description: Optional[str] = None  # plain text, tags stripped, ≤500 chars
    publisher: Optional[str] = None    # Google News <source> text, else None
@dataclass
@@ -212,7 +105,6 @@ class FundamentalContext:
    news_headlines: List[NewsSentiment] = field(default_factory=list)
    event_impact_on_pairs: Dict[str, str] = field(default_factory=dict)
    calendar_available: bool = False  # True if live calendar data was fetched
    llm_summary: str = ""  # DeepSeek market-context blurb (display-only)
 class NewsEngine:
@@ -293,25 +185,6 @@ class NewsEngine:
    _headlines_cache: Tuple[float, List[NewsSentiment]] = (0.0, [])
    _EMPTY_RESULT_TTL = 60.0  # retry sooner after a failed/empty fetch
    # DeepSeek result caches. Headlines and releases are immutable, so a
    # scored one never needs re-scoring; bounded by dropping the oldest
    # insertion. _llm_sent_cache is only touched under _cache_lock (its
    # writer runs inside _cached_fetch); the other two take the lock
    # around cache access but keep HTTP calls outside it.
    _llm_sent_cache: Dict[str, Tuple[float, float]] = {}   # sha1 → (s, xau)
    _llm_event_cache: Dict[Tuple[str, str, str], str] = {}
    _llm_summary_cache: Tuple[float, str] = (0.0, "")
    _LLM_SENT_CACHE_MAX = 400
    _LLM_EVENT_CACHE_MAX = 100
    # Per-source fetch health for status displays (telegram /status).
    # name → {"ok": ts|None, "err": ts|None, "detail": str, "extra": str}.
    # All writers run under _cache_lock (fetches happen inside
    # _cached_fetch); readers render without the lock — worst case a
    # display line is one cycle stale. A source is "failing" when its
    # last error is newer than its last success.
    _source_health: Dict[str, dict] = {}
    def __init__(self, config: NewsConfig):
        self.config = config
        self._session = requests.Session()
@@ -391,12 +264,6 @@ class NewsEngine:
        except Exception as e:
            logger.warning(f"News sentiment fetch failed: {e}")
        # 3. LLM market-context summary (display-only)
        try:
            context.llm_summary = self._llm_market_summary(context)
        except Exception as e:
            logger.debug(f"LLM market summary failed: {e}")
        return context
    @staticmethod
@@ -444,17 +311,10 @@ class NewsEngine:
                )
                if event.minutes_until > -window:
                    if surprised:
-                        desc = (
+                        return "caution", (
                            f"{event.currency} {event.event} released — {surprise} surprise "
                            f"(actual {event.actual} vs forecast {event.forecast})"
                        )
                        # LLM enrichment touches the DESCRIPTION only —
                        # `surprise` is numeric-direction and does not know
                        # whether a beat is gold-bullish; the LLM does.
                        gold_dir = self._llm_event_direction(event)
                        if gold_dir:
                            desc += f" — gold {gold_dir}"
                        return "caution", desc
                    return "caution", f"{event.currency} {event.event} just released"
        return "clear", "No significant events"
@@ -488,24 +348,6 @@ class NewsEngine:
        return net, desc
    def get_week_events(
        self, currencies: Optional[List[str]] = None
    ) -> List[EconomicEvent]:
        """Full cached calendar week (past and upcoming), filtered to
        `currencies` (default: tracked_currencies), sorted by time.
        Read-only accessor for display surfaces (the web dashboard) —
        rides the same class-level cache as signal scans, so it adds no
        network I/O. No production signal path calls this.
        """
        if currencies is None:
            currencies = self.config.tracked_currencies
        events = self._cached_fetch("_calendar_cache", self._fetch_calendar_raw)
        return sorted(
            (e for e in events if e.currency in currencies),
            key=lambda e: e.time,
        )
    # ── Private Methods ────────────────────────────────────────────
    # Maps country names in MyFXBook titles to currency codes
@@ -515,36 +357,6 @@ class NewsEngine:
        "canada": "CAD", "new zealand": "NZD",
    }
    @classmethod
    def _record_source(cls, name: str, ok: bool, detail: str = "",
                       extra: str = "") -> None:
        """Track one dependency's fetch outcome for status displays.
        Logs state TRANSITIONS so the log file carries the story without
        a line every 10-min cycle: first failure WARNING (with detail),
        repeat failures DEBUG, recovery INFO."""
        h = cls._source_health.setdefault(
            name, {"ok": None, "err": None, "detail": "", "extra": ""})
        was_failing = h["err"] is not None and (
            h["ok"] is None or h["err"] > h["ok"])
        if ok:
            h["ok"] = time.time()
            h["extra"] = extra
            if was_failing:
                logger.info("Source recovered: %s", name)
        else:
            h["err"] = time.time()
            h["detail"] = detail
            if was_failing:
                logger.debug("Source still failing: %s (%s)", name, detail)
            else:
                logger.warning("Source failing: %s (%s)", name, detail)
    @staticmethod
    def source_is_failing(entry: Optional[dict]) -> bool:
        """True when the entry's last error is newer than its last ok."""
        return bool(entry and entry["err"] is not None and
                    (entry["ok"] is None or entry["err"] > entry["ok"]))
    def _cached_fetch(self, cache_attr: str, fetch_fn) -> list:
        """Return cached data for `cache_attr` or refetch when the TTL
        expired. Successful results live for context_cache_ttl_seconds;
@@ -577,27 +389,14 @@ class NewsEngine:
        Returns an empty list (with a log warning) if unavailable —
        we never generate fake events since that causes constant false alarms.
        """
-        sources = (
+        for fetcher in (self._try_fetch_calendar_api, self._try_fetch_myfxbook_rss):
            (self._try_fetch_calendar_api, "calendar:forexfactory",
             "ForexFactory"),
            (self._try_fetch_myfxbook_rss, "calendar:myfxbook", "MyFXBook"),
        )
        for fetcher, src, label in sources:
            try:
                events = fetcher()
                if events:
                    self._record_source(src, True)
                    self._record_source(
                        "calendar", True,
                        extra=f"{len(events)} events via {label}")
                    return events
                self._record_source(src, False, "no events returned")
            except Exception as e:
                self._record_source(
                    src, False, f"{type(e).__name__}: {e}")
                logger.debug(f"Calendar fetch exception ({fetcher.__name__}): {e}")
        self._record_source("calendar", False, "all sources failed")
        logger.info(
            "Economic calendar data unavailable. "
            "Event risk checks will be skipped for this cycle."
@@ -744,16 +543,11 @@ class NewsEngine:
        """Fetch and analyze news headlines for sentiment."""
        headlines = []
        seen_titles = set()
        ok_feeds = 0
        for feed_url in self.config.rss_feeds:
            host = feed_url.split("/")[2] if "//" in feed_url else feed_url
            kept_before = len(headlines)
            try:
                resp = self._session.get(feed_url, timeout=10)
                if resp.status_code != 200:
                    self._record_source(
                        f"rss:{host}", False, f"HTTP {resp.status_code}")
                    continue
                root = ET.fromstring(resp.content)
@@ -786,21 +580,6 @@ class NewsEngine:
                    currencies = self._extract_currencies(title)
                    if currencies:
                        link_el = item.find("link")
                        desc_el = item.find("description")
                        src_el = item.find("source")
                        # itertext() survives CDATA and stray child
                        # elements where .text stops at the first child.
                        desc = _clean_html_text(
                            "".join(desc_el.itertext())
                            if desc_el is not None else None)
                        # Google News descriptions are often just the
                        # headline again inside an anchor — worthless.
                        if desc and desc.strip() == title.strip():
                            desc = None
                        publisher = None
                        if src_el is not None:
                            publisher = (src_el.text or "").strip() or None
                        headlines.append(NewsSentiment(
                            headline=title,
                            source=feed_url,
@@ -812,32 +591,11 @@ class NewsEngine:
                                cur: self._analyze_headline_sentiment(title, cur)
                                for cur in currencies
                            },
                            link=_safe_link(
                                link_el.text if link_el is not None else None),
                            description=desc,
                            publisher=publisher,
                        ))
                ok_feeds += 1
                self._record_source(
                    f"rss:{host}", True,
                    extra=f"{len(headlines) - kept_before} kept")
            except Exception as e:
                self._record_source(
                    f"rss:{host}", False, f"{type(e).__name__}: {e}")
                logger.debug(f"Failed to fetch RSS from {feed_url}: {e}")
        self._record_source(
            "headlines", ok_feeds > 0,
            detail="all feeds failed" if ok_feeds == 0 else "",
            extra=(f"{len(headlines)} headlines from "
                   f"{ok_feeds}/{len(self.config.rss_feeds)} feeds"))
        try:
            self._apply_llm_sentiment(headlines)
        except Exception as e:
            logger.debug(f"LLM sentiment failed, keyword scores stand: {e}")
        return headlines
    def _analyze_headline_sentiment(
@@ -879,142 +637,6 @@ class NewsEngine:
        return (bullish_score - bearish_score) / total
    # ── DeepSeek enrichment (display/warning-only, keyword fallback) ──
    def _llm_headline_scores(
        self, texts: List[str]
    ) -> Optional[List[Tuple[float, float]]]:
        """One batched DeepSeek call → [(overall, xau), ...] aligned with
        `texts`, or None on any failure/mismatch (caller keeps keyword
        scores). Tests stub this method directly."""
        result = get_llm_client().chat_json(
            _LLM_SENTIMENT_SYSTEM, {"headlines": texts})
        if not result:
            return None
        scores = result.get("scores")
        if not isinstance(scores, list) or len(scores) != len(texts):
            return None
        out = []
        for item in scores:
            try:
                s = max(-1.0, min(1.0, float(item[0])))
                x = max(-1.0, min(1.0, float(item[1])))
            except (TypeError, KeyError, ValueError, IndexError):
                return None
            out.append((s, x))
        return out
    def _apply_llm_sentiment(self, headlines: List[NewsSentiment]) -> None:
        """Overwrite keyword sentiment on `headlines` with LLM scores.
        Runs inside _cached_fetch, i.e. UNDER _cache_lock — cache access
        is direct and the (rare) HTTP call blocks sibling scans at most
        once per cache window, matching the existing shared-fetch design.
        Only headlines missing from the cache are sent; a failed batch
        changes nothing.
        """
        if not headlines or not get_llm_client().available:
            return
        cache = NewsEngine._llm_sent_cache
        keyed = [(hashlib.sha1(h.headline.encode()).hexdigest(), h)
                 for h in headlines]
        # Cap the batch so the reply fits the client's max_tokens — the
        # overflow stays keyword-scored this cycle and, still uncached,
        # goes out with the next window's batch.
        new = [(k, h) for k, h in keyed if k not in cache][:25]
        if new:
            scores = self._llm_headline_scores([h.headline for _, h in new])
            if scores is not None:
                for (k, _), sc in zip(new, scores):
                    cache[k] = sc
                while len(cache) > self._LLM_SENT_CACHE_MAX:
                    cache.pop(next(iter(cache)))
        for k, h in keyed:
            sc = cache.get(k)
            if sc is None:
                continue
            overall, xau = sc
            h.sentiment_score = overall
            h.sentiment_by_currency = {
                cur: (xau if cur == "XAU" else overall)
                for cur in h.currencies_mentioned
            }
    def _llm_event_direction(self, event: EconomicEvent) -> Optional[str]:
        """Gold direction for a released surprise, e.g. "bearish: strong
        jobs lift yields". None → caller keeps the numeric description.
        Cached per release; failures are NOT cached (retry next scan).
        The actual-vs-forecast comparison is precomputed and sent as
        "vs_forecast": the 2026-08-06 validation found the model reads
        close K/M-suffixed pairs backwards ~11% of the time (e.g. claims
        202K vs 217K called "higher than forecast") when left to compare
        the raw numbers itself."""
        key = (event.event, event.currency, str(event.actual or ""))
        with NewsEngine._cache_lock:
            if key in NewsEngine._llm_event_cache:
                return NewsEngine._llm_event_cache[key] or None
        client = get_llm_client()
        if not client.available:
            return None
        vs_forecast = {
            "positive": "above forecast",
            "negative": "below forecast",
            "inline": "in line with forecast",
        }.get(event.surprise or "", "unknown")
        result = client.chat_json(_LLM_EVENT_SYSTEM, {
            "event": event.event,
            "currency": event.currency,
            "actual": event.actual,
            "forecast": event.forecast,
            "previous": event.previous,
            "vs_forecast": vs_forecast,
        })
        if not result:
            return None
        direction = result.get("gold_direction")
        if direction not in ("bullish", "bearish", "neutral"):
            return None
        reason = str(result.get("reason", "")).strip()[:80]
        text = "" if direction == "neutral" else (
            f"{direction}: {reason}" if reason else direction)
        with NewsEngine._cache_lock:
            NewsEngine._llm_event_cache[key] = text
            while len(NewsEngine._llm_event_cache) > self._LLM_EVENT_CACHE_MAX:
                NewsEngine._llm_event_cache.pop(
                    next(iter(NewsEngine._llm_event_cache)))
        return text or None
    def _llm_market_summary(self, context: FundamentalContext) -> str:
        """1-2 sentence macro blurb for alerts; "" when unavailable.
        Cached (success OR failure) for the context-cache TTL so at most
        one call is made per window."""
        with NewsEngine._cache_lock:
            ts, cached = NewsEngine._llm_summary_cache
            if time.time() - ts < self.config.context_cache_ttl_seconds:
                return cached
        high_impact = [e for e in context.events_today if e.is_high_impact]
        if not context.news_headlines and not high_impact:
            return ""
        client = get_llm_client()
        if not client.available:
            return ""
        result = client.chat_json(_LLM_SUMMARY_SYSTEM, {
            "headlines": [h.headline for h in context.news_headlines[:15]],
            "events": [
                {"event": e.event, "currency": e.currency,
                 "actual": e.actual, "forecast": e.forecast,
                 "released": not e.is_upcoming}
                for e in high_impact[:10]
            ],
        })
        summary = ""
        if result:
            summary = str(result.get("summary", "")).strip()[:300]
        with NewsEngine._cache_lock:
            NewsEngine._llm_summary_cache = (time.time(), summary)
        return summary
    def _extract_currencies(self, headline: str) -> List[str]:
        """Extract mentioned currencies from a headline."""
        headline_lower = headline.lower()
@@ -90,17 +90,8 @@ STATE_FILE = os.path.join(
 def _oanda_setting_int(env_key: str, json_key: str, default: int) -> int:
    env = os.environ.get(env_key)
    if env:
-        try:
+        return int(env)
-            return int(env)
+    return int(get_app_config().oanda_settings.get(json_key, default))
        except ValueError:
            raise SystemExit(
                f"{env_key}={env!r} is not an integer — refusing to start")
    try:
        return int(get_app_config().oanda_settings.get(json_key, default))
    except (TypeError, ValueError):
        raise SystemExit(
            f"config.json oanda.{json_key} is not an integer — "
            f"refusing to start")
 def _oanda_setting_str(env_key: str, json_key: str, default: str) -> str:
@@ -139,11 +130,8 @@ def _seconds_until_boundary(interval_minutes: int, buffer: int = 10) -> float:
    elapsed_seconds = elapsed_minutes * 60 + now.second + now.microsecond / 1e6
    interval_seconds = interval_minutes * 60
    remaining = interval_seconds - elapsed_seconds + buffer
-    # NOTE: remaining > buffer always (elapsed < interval by construction);
+    if remaining <= buffer:
-    # a call at the exact boundary instant targets the NEXT boundary —
+        remaining += interval_seconds
    # steady-state loops wake at boundary+buffer, so only a process
    # started in the boundary microsecond skips one scan (2026-08-04
    # audit I42; the old `remaining <= buffer` guard was unreachable).
    return remaining
 # ── Trade state persistence ───────────────────────────────────────────
@@ -177,13 +165,7 @@ class TradingDaemon:
        base_url = oanda_base_url()
        lot_size = _oanda_setting_int("OANDA_LOT_SIZE", "lot_size", 1)
        self._quality_gate = _oanda_setting_str(
-            "OANDA_QUALITY_GATE", "quality_gate", "C").strip()
+            "OANDA_QUALITY_GATE", "quality_gate", "C")
        if self._quality_gate not in QUALITY_RANK:
            # Fail CLOSED: an unrecognized gate must never silently fall
            # back to the most permissive rank (QUALITY_RANK.get default).
            raise SystemExit(
                f"OANDA_QUALITY_GATE/quality_gate {self._quality_gate!r} "
                f"is not one of {sorted(QUALITY_RANK)} — refusing to start")
        self._max_hold_bars = _oanda_setting_int(
            "OANDA_MAX_HOLD_BARS", "max_hold_bars", 300)
        self._price_precision = PAIR_INFO[SYMBOL]["price_precision"]
@@ -293,86 +275,60 @@ class TradingDaemon:
    # ── Position discovery (restart recovery) ──────────────────────
    def _discover_existing_positions(self) -> None:
        self._reconcile_positions("boot discovery")
    def _reconcile_positions(self, reason: str) -> None:
        """Re-sync tracked state with the broker's open trades. Safe to
        call mid-run (lock-guarded); used at boot and whenever a fill or
        placement error suggests state and broker may have diverged."""
        try:
            open_trades = self._executor.get_open_trades(INSTRUMENT)
        except OandaAPIError:
-            logger.exception(
+            logger.exception("Failed to query open trades at boot")
                "Failed to query open trades (%s) — reconcile skipped",
                reason)
            return
        saved = self._state.get("trades", {})
        live_ids = {t["id"] for t in open_trades}
-        with self._state_lock:
+        stale = [tid for tid in saved if tid not in live_ids]
-            saved = self._state.get("trades", {})
+        for tid in stale:
            logger.info(
                "Stale trade %s in state file (closed on Oanda), removing", tid)
            del saved[tid]
-            stale = [tid for tid in saved if tid not in live_ids]
+        for trade in open_trades:
-            for tid in stale:
+            tid = trade["id"]
            if tid in saved:
                logger.info(
-                    "Stale trade %s in state file (closed on Oanda), "
+                    "Recovered trade %s from state: %s %s @ %.2f, SL %.2f",
-                    "removing", tid)
+                    tid, saved[tid].get("direction"), INSTRUMENT,
-                del saved[tid]
+                    saved[tid].get("entry"), saved[tid].get("current_sl"))
                continue
-            for trade in open_trades:
+            direction = "BUY" if float(trade.get("currentUnits", 0)) > 0 else "SELL"
-                tid = trade["id"]
+            entry = float(trade.get("price", 0))
-                if tid in saved:
+            sl_order = trade.get("stopLossOrder", {})
-                    logger.info(
+            sl = float(sl_order.get("price", entry))
-                        "Recovered trade %s from state: %s %s @ %.2f, SL %.2f",
+            risk = abs(entry - sl) if sl else 1.0
                        tid, saved[tid].get("direction"), INSTRUMENT,
                        saved[tid].get("entry"), saved[tid].get("current_sl"))
                    continue
-                direction = ("BUY" if float(trade.get("currentUnits", 0)) > 0
+            saved[tid] = {
-                             else "SELL")
+                "tf": "H1",
-                entry = float(trade.get("price", 0))
+                "direction": direction,
-                sl_order = trade.get("stopLossOrder") or {}
+                "entry": entry,
-                sl_price = sl_order.get("price")
+                "sl_initial": sl,
-                if sl_price is None:
+                "current_sl": sl,
-                    # No broker-side stop at all: risk is undefined, so the
+                "risk": risk,
-                    # trailing model cannot manage this trade. Scream — a
+                "peak": entry,
-                    # naked position needs a human (or a policy) to set a
+                "trail_armed": False,
-                    # stop; fabricating one here would invent trading
+                "trail_arm_r": 0.6,
-                    # behavior the books never validated.
+                "trail_dist_r": 0.75,
-                    logger.critical(
+                "bars_held": 0,
-                        "Adopted trade %s (%s @ %.2f) has NO STOP LOSS on "
+                "last_bar_ts": None,
-                        "the broker — it will NOT be trailed. Set a stop "
+                "signal_bar_ts": None,
-                        "manually or close it.", tid, direction, entry)
+            }
-                    sl = entry
+            logger.info(
-                    risk = 0.0
+                "Adopted unknown trade %s: %s @ %.2f, SL %.2f, risk %.2f "
-                else:
+                "(conservative defaults)",
-                    sl = float(sl_price)
+                tid, direction, entry, sl, risk)
                    risk = abs(entry - sl)
-                saved[tid] = {
+        self._state["trades"] = saved
-                    "tf": "H1",
+        _save_state(self._state)
                    "direction": direction,
                    "entry": entry,
                    "sl_initial": sl,
                    "current_sl": sl,
                    "risk": risk,
                    "peak": entry,
                    "trail_armed": False,
                    "trail_arm_r": 0.6,
                    "trail_dist_r": 0.75,
                    "bars_held": 0,
                    "last_bar_ts": None,
                    "signal_bar_ts": None,
                }
                logger.info(
                    "Adopted unknown trade %s: %s @ %.2f, SL %.2f, risk %.2f "
                    "(conservative defaults)",
                    tid, direction, entry, sl, risk)
            self._state["trades"] = saved
            _save_state(self._state)
        logger.info(
-            "Position reconcile (%s): %d active trade(s)", reason, len(saved))
+            "Position discovery: %d active trade(s)", len(saved))
    # ── Candle-boundary loop ───────────────────────────────────────
@@ -565,15 +521,9 @@ class TradingDaemon:
                    stop_loss=ts.stop_loss,
                )
            except OandaAPIError:
                # The order is never retried (a timeout after a server-side
                # fill would double-up on retry), but the server MAY have
                # filled before the error reached us — reconcile so a live
                # position can't stay untracked until the next restart.
                logger.exception(
-                    "Failed to place %s order for %s %s — reconciling "
+                    "Failed to place %s order for %s %s",
                    "broker positions",
                    ts.direction, SYMBOL, tf.value)
                self._reconcile_positions("order placement error")
                return
            fill_tx = result.get("orderFillTransaction")
@@ -585,16 +535,7 @@ class TradingDaemon:
            trade_opened = fill_tx.get("tradeOpened", {})
            trade_id = trade_opened.get("tradeID")
            if not trade_id:
-                # A fill with no tradeOpened means the order interacted
+                logger.warning("No trade ID in fill response")
                # with an existing broker position (tradesClosed /
                # tradeReduced on a netting account) — state and broker
                # have diverged. Surface it and re-sync instead of
                # silently forgetting a real fill.
                logger.critical(
                    "Order FILLED but opened no trade (netting close/"
                    "reduce?) — fill: closed=%s reduced=%s. Reconciling.",
                    fill_tx.get("tradesClosed"), fill_tx.get("tradeReduced"))
                self._reconcile_positions("fill without tradeOpened")
                return
            fill_price = float(fill_tx.get("price", ts.entry_price))
@@ -646,12 +587,8 @@ class TradingDaemon:
    # ── Trailing stop management ───────────────────────────────────
    def _manage_trailing(self, tf: Timeframe) -> None:
        # Copy the per-trade dicts too: the loop mutates info (peak,
        # bars_held, last_bar_ts) and must not touch live state outside
        # the lock — another thread may be serializing it in _save_state.
        with self._state_lock:
-            trades = {tid: dict(info)
+            trades = dict(self._state.get("trades", {}))
                      for tid, info in self._state.get("trades", {}).items()}
        for trade_id, info in trades.items():
            if info.get("tf") != tf.value:
@@ -697,16 +634,13 @@ class TradingDaemon:
                    self._executor.close_trade(trade_id)
                except OandaAPIError:
                    # Keep tracking — the trade is still live on the
-                    # broker (incl. a 200 whose close order was cancelled,
+                    # broker; we retry the close at the next bar.
                    # e.g. MARKET_HALTED — close_trade raises on no-fill);
                    # we retry the close at the next bar.
                    logger.exception(
                        "Failed to close timed-out trade %s — will retry",
                        trade_id)
                    with self._state_lock:
-                        if trade_id in self._state["trades"]:
+                        self._state["trades"][trade_id] = info
-                            self._state["trades"][trade_id] = info
+                        _save_state(self._state)
                            _save_state(self._state)
                    continue
                with self._state_lock:
                    self._state["trades"].pop(trade_id, None)
@@ -717,14 +651,6 @@ class TradingDaemon:
            bar_high = float(last_bar["High"])
            bar_low = float(last_bar["Low"])
            if info.get("risk", 0) <= 0:
                # Adopted naked position (no broker SL → risk undefined):
                # never manageable by the trailing model — keep it loud
                # instead of silently skipping forever.
                logger.warning(
                    "Trade %s has risk<=0 (no stop loss) — trailing "
                    "skipped; set a stop manually", trade_id)
            new_sl = self._compute_trail(info, bar_high, bar_low)
            if new_sl is not None:
@@ -741,13 +667,8 @@ class TradingDaemon:
                            trade_id)
            with self._state_lock:
-                # Write back ONLY if the trade is still tracked — another
+                self._state["trades"][trade_id] = info
-                # loop's _check_closed_trades may have popped it while we
+                _save_state(self._state)
                # iterated (broker-side close). Re-inserting would create
                # a ghost that blocks new entries.
                if trade_id in self._state["trades"]:
                    self._state["trades"][trade_id] = info
                    _save_state(self._state)
    def _compute_trail(
        self, info: Dict, bar_high: float, bar_low: float
@@ -69,13 +69,6 @@ class OandaExecutor:
                "instrument": instrument,
                "units": signed_units,
                "timeInForce": "FOK",
                # The book model is one position total; on a netting
                # account an opposite-side order would silently CLOSE an
                # existing (possibly untracked) position instead of opening
                # the signal trade. OPEN_ONLY makes that case fail loudly
                # (order cancelled) so the divergence surfaces instead of
                # executing an unintended close.
                "positionFill": "OPEN_ONLY",
                "stopLossOnFill": {
                    "price": self._fmt_price(stop_loss),
                },
@@ -163,18 +156,7 @@ class OandaExecutor:
               f"/trades/{trade_id}/close")
        resp = self._request_with_retry("PUT", url, json={"units": "ALL"})
        data = self._parse_json(resp)
-        close_tx = data.get("orderFillTransaction")
+        close_tx = data.get("orderFillTransaction", {})
        if not close_tx:
            # A 200 whose close order was created but NOT filled (e.g.
            # cancelled with MARKET_HALTED during the settlement pause)
            # means the position is STILL OPEN. Raising keeps the caller
            # tracking it and retrying — silently returning here would
            # orphan a live position.
            reason = data.get("orderCancelTransaction", {}).get(
                "reason", "no orderFillTransaction in response")
            raise OandaAPIError(
                f"Close of trade {trade_id} did not fill: {reason}",
                resp.status_code, str(data)[:300])
        close_price = float(close_tx.get("price", 0))
        pl = float(close_tx.get("pl", 0))
        logger.info(
@@ -0,0 +1,9 @@
 """
 ForexBot2 — Oanda provider (back-compat shim)
 ==============================================
 The implementation lives in `providers/oanda.py`; shared behavior is in
 `providers/base.py`. This module keeps the historical import path working.
 """
 from providers.base import _is_forex_weekend  # noqa: F401
 from providers.oanda import OandaDataError, OandaDataProvider  # noqa: F401
@@ -99,72 +99,6 @@ in 5/7 calendar years including winning years (2022 -9.2R). Blocking:
 12mo book is unchanged (+139.0R vs +139.3R). M15-and-up keep hour 9:
 it is POSITIVE there (M15 6yr +1.7R, M15 OOS +8.6R), so the block is
 M5-only, same per-timeframe pattern as the M15 overrides.
 M5 override 2 — relaxation campaign (validated 2026-07-30, Oanda
 bars, $0.30 spread, trailing exits): UNBLOCK hour 21 UTC on M5 only,
 combined with the config.py M5 gate relaxations (m5_min_abs_score
 30->25, m5_min_confirmation_score 10->5). The combined config beat
 the pre-relaxation control in ALL 6 calendar years of the 5yr replay
 2021-07-30 -> 2026-07-30 (+230.9R/2319 PF 1.26 DD 21.3 vs
 +168.0R/2048 PF 1.21 DD 27.0), on 12mo (+107.4R PF 1.36, halves
 +68.6/+38.8 vs +84.6R PF 1.33), on OOS 2024-06-01 -> 2025-03-30
 (+26.1R vs +17.2R), and at $0.70 spread (+86.9R vs +66.4R). Hour-21
 trades alone: +28.0R/22, positive every year — overturning the old
 block per the full-evidence rule. Hour 21 stays BLOCKED on M15+
 (12mo M15 unblock tested -0.3R). Declined-but-open from the same
 campaign: an M5 hour-22 block (+4.8R on 12mo — the burst-heavy
 "institutional" hour — but -3.2R on 5yr and -1.1R on OOS: regime-
 specific); M15 hour-16 unblock (+0.8R/+57 trades 12mo but -18.3R
 DD +34% on 5yr); every SELL-gate/vol/TP-size relaxation retested
 negative (see memory notes for the full grid).
 M5 override 4 — XAU_PB disabled (validated 2026-08-04, Oanda bars,
 $0.65 REAL spread — first campaign validated at measured cost on the
 live feed, per the 2026-08-04 re-pin baseline): m5_pullback_enabled
 False on M5. Loss forensics on the Oanda books identified XAU_PB as
 the largest wrong-direction loss source (12mo: -43.5R of the -156.6R
 wrong-direction bucket; 5yr: -274.5R). Disabling: 12mo +96.6R/588
 PF 1.44 max DD 13.9R halves +51.2/+45.4 vs +82.3R/788 PF 1.26 DD
 23.9R shipped (which FAILS the PF>=1.3 clause at real cost); OOS
 2024-06-01 -> 2025-03-30 +9.5R PF 1.34; 5yr +123.9R/956 PF 1.35 vs
 PF 1.16 with PB. PB's 5yr net contribution (+23.6R) was almost
 entirely the 2022 chop year (+36.1R vs -2.2R without) — a regime
 earner that has been a drag on trend-regime bars since; its tight
 stops are also the most cost-punished at $0.65. PB code and the
 pb_trail_* params STAY for future regime-gated experiments (same
 pattern as cp_enabled). Rejected in the same campaign (4 candidates
 total, N logged): hour-19 block (+3.7R 12mo but -3.3R OOS, -1.5R
 5yr — OOS gate, the classic trap); hours-14/15 block (-6.9R 12mo
 once PB is removed — their raw negativity WAS the PB losses);
 trail_dist_r 0.75 (-2.6R 12mo; the 0.88R average give-back is the
 validated price of riding trends, exits confirmed NOT the loss
 driver: exit-management accounts for 1-2% of lost R vs 50%
 wrong-direction / 49% entry-timing).
 M5 override 5 — EMA partial-alignment squash (validated 2026-08-04,
 Oanda bars, $0.65 real spread): ema_partial_neutral True on M5 —
 partial 4-EMA ribbon alignment (2-3 of 4 checks) scores NEUTRAL
 instead of a weak BUY/SELL. Motivated by the indicator audit: partial-
 alignment trades averaged -0.008R/84 vs +0.230R/341 fully aligned.
 Replay (knob mirrored in the backtester's precomputed EMA scorer in
 the same commit — live==backtest parity): 12mo +97.6R/556 PF 1.48
 DD 10.7R halves +47.8/+49.8 vs control +96.6R/588 PF 1.44 DD 13.9R;
 5yr +127.4R/912 PF 1.38 DD 12.8R vs +123.9R/956 PF 1.35 DD 13.2R,
 2022 improves -2.2 -> -1.1; OOS 2024-06-01 -> 2025-03-30 +9.4R
 PF 1.33 vs +9.5R PF 1.34 — a one-trade tie, documented as such (the
 OOS window holds up; it neither prefers nor punishes). Net effect:
 32 fewer marginal trades, every risk-adjusted measure better.
 Rejected in the same round (trial-logged): confirmation-TF swap
 M5->[M15,H4] (12mo +104.0R PF 1.51 DD 10.3 but OOS +7.8R PF 1.27
 vs control +9.5R PF 1.34 and 2022 3x worse — the H4 confirm keeps
 endorsing stale direction in chop; OOS-ordering rejection, same
 precedent as trail dist 1.0); confirm [H1,H4] (-31.8R 12mo);
 triple-confirm [M15,H1,H4] (-4.9R); divergence weight -1/3
 (-3.8R 12mo); candlestick weight +50% (-3.0R 12mo). The weight
 vector keeps proving calibrated as a system: divergence/candlestick/
 bollinger all look mis-weighted in-book and all replay worse when
 "fixed" (bollinger weight-0: +80.9R PF 1.29 DD 23.8 — its constant
 oppose vote is a load-bearing quality bar, like ROC).
 """
 from typing import Optional, List
@@ -192,27 +126,13 @@ def make_config(
        confirmation_tfs or _CONFIRMATIONS.get(primary_tf, [Timeframe.M15, Timeframe.H4])
    )
    if primary_tf == Timeframe.M5:
-        # Block hour 9 UTC on M5 only; UNBLOCK hour 21 on M5 only
+        # Block hour 9 UTC on M5 only — see module docstring.
-        # (positive every year of the 5yr replay) — see docstring.
+        config.xau_strategy.blocked_hours_utc = (
-        config.xau_strategy.blocked_hours_utc = [
+            config.xau_strategy.blocked_hours_utc + [9]
-            h for h in config.xau_strategy.blocked_hours_utc if h != 21
+        )
        ] + [9]
        # Deliberately NOT mirrored into config.backtest.blocked_hours_utc:
        # that copy is the generic non-XAU fallback carrying the JSON base
        # only. Propagating a gold-validated per-TF delta there would leak
        # XAU-calibrated hours to a future pair (pair-isolation rule).
        # Every XAU consumer branches to the xau_strategy copy. Enforced
        # by the Module 13 test "M5 factory appends hour 9 on top of a
        # JSON base".
        # BUY-only chart patterns on M5 — see module docstring.
        config.xau_strategy.cp_enabled = True
        config.xau_strategy.cp_patterns = ["inv_hs", "double_bottom", "bull_flag"]
        # XAU_PB pullback entries OFF — largest wrong-direction loss
        # source at real cost on Oanda bars — see module docstring.
        config.xau_strategy.m5_pullback_enabled = False
        # Partial EMA-ribbon alignment scores NEUTRAL on M5 — its
        # trades carried no edge — see module docstring.
        config.technical.ema_partial_neutral = True
        # Wider trailing distance on M5 only — see module docstring.
        config.signals.trail_arm_r = 0.60
        config.signals.trail_dist_r = 0.85
@@ -6,13 +6,12 @@ caching, validation, boundary guarantees, and range pagination — the
 behavior the signal pipeline depends on).
 Adding a provider = one module implementing `_fetch_raw` + one registry
-entry + one config.json `data_providers` block. The live candle feed is
+entry + one config.json `data_providers` block. The live feed is selected
-selected by config.json `data_providers.active`; the optional
+by config.json `data_providers.active`; registration alone changes
-`data_providers.streaming` scalar selects a different provider for the
+nothing.
 tick stream (level-breach monitor). Registration alone changes nothing.
 """
-from typing import Dict, Optional
+from typing import Dict
 from app_config import AppConfigError, get_app_config
 from config import PlatformConfig
@@ -30,11 +29,9 @@ PROVIDER_REGISTRY: Dict[str, type] = {
 }
-def make_data_provider(config: PlatformConfig,
+def make_data_provider(config: PlatformConfig) -> BaseDataProvider:
-                       name: Optional[str] = None) -> BaseDataProvider:
+    """Build the ACTIVE data provider (config.json: data_providers.active)."""
-    """Build a provider by name (default: config.json data_providers.active)."""
+    name = get_app_config().active_provider
    if name is None:
        name = get_app_config().active_provider
    cls = PROVIDER_REGISTRY.get(name)
    if cls is None:
        raise AppConfigError(
@@ -24,7 +24,6 @@ import time
 from abc import ABC, abstractmethod
 from datetime import datetime, timedelta
 from typing import AsyncIterator, Callable, Dict, Optional, Sequence, Tuple
 from zoneinfo import ZoneInfo
 import pandas as pd
@@ -32,36 +31,19 @@ from config import PAIR_INFO, PlatformConfig, Timeframe, XAUUSD
 logger = logging.getLogger(__name__)
 # The gold/FX market week is defined in New York time (see
 # _is_forex_weekend); a fixed UTC hour is wrong for half the year.
 _NY_TZ = ZoneInfo("America/New_York")
 def _is_forex_weekend(now=None) -> bool:
-    """Gold/FX market closed for the weekend.
+    """Forex market closed: Friday 22:00 UTC → Sunday 22:00 UTC."""
    The market week is anchored to NEW YORK time — Friday 17:00 ET →
    Sunday 18:00 ET — not a fixed UTC hour. During US DST the feed
    actually stops at 21:00 UTC (measured on 5 years of Oanda bars:
    159/162 summer Fridays; winter close 22:00, winter reopen Sunday
    23:00 UTC). The old hardcoded Fri 22:00→Sun 22:00 UTC rule produced
    two false-"STALE DATA/feed lagging" hours every week (2026-08-04
    audit I43, fixed per owner decision).
    Kept as three deliberate copies (here, signal_aggregator.py,
    gpu_backtest._is_xau_blocked_by_calendar) — a Module 21 test asserts
    they agree hour-by-hour across DST transitions.
    """
    if now is None:
        now = pd.Timestamp.now(tz="UTC")
-    ny = now.astimezone(_NY_TZ)
+    wd = now.weekday()
-    wd = ny.weekday()
+    hour = now.hour
-    if wd == 4:                      # Friday
+    if wd == 4 and hour >= 22:
-        return ny.hour >= 17
+        return True
-    if wd == 5:                      # Saturday
+    if wd == 5:
        return True
    if wd == 6 and hour < 22:
        return True
    if wd == 6:                      # Sunday
        return ny.hour < 18
    return False
@@ -104,15 +86,9 @@ class BaseDataProvider(ABC):
        self.config = config
        self._cache: Dict[str, pd.DataFrame] = {}
        self._cache_timestamps: Dict[str, float] = {}
        # Requested fetch depth per key: a cached short frame must not
        # satisfy a caller asking for a longer lookback.
        self._cache_outputsize: Dict[str, int] = {}
        # The bot shares one provider across a thread pool; cache eviction
        # is check-then-act and needs locking.
        self._cache_lock = threading.Lock()
        # Single-flight: one API fetch per (symbol, TF) at a time; other
        # threads wait and then read the cache.
        self._inflight: Dict[str, threading.Event] = {}
    # ── Abstract: the only provider-specific piece ─────────────────
@@ -143,97 +119,68 @@ class BaseDataProvider(ABC):
            lookback_days = self.config.data_lookback_days
        cache_key = f"{symbol}_{timeframe.value}"
        now = time.time()
        cache_ttl = self.CANDLE_SECONDS.get(timeframe, self.config.refresh_interval)
        with self._cache_lock:
            cached = self._cache.get(cache_key)
            cached_at = self._cache_timestamps.get(cache_key)
        if (
            cached is not None
            and cached_at is not None
            and (now - cached_at) < cache_ttl
            and not cached.empty
        ):
            if (self._is_data_fresh(cached, symbol, timeframe)
                    and self._has_latest_closed_bar(cached, timeframe)):
                return cached
            logger.warning(f"{symbol} {timeframe.value}: cached data is stale, refetching")
            with self._cache_lock:
                self._cache.pop(cache_key, None)
                self._cache_timestamps.pop(cache_key, None)
        outputsize = max(
            300,
            min(int(lookback_days * self.BARS_PER_DAY[timeframe]) + 20,
                self.MAX_BARS_PER_REQUEST),
        )
        df = self._fetch_raw(symbol, timeframe, outputsize)
        df = self._drop_forming_candle(df, timeframe)
        df = self._validate_and_clean(df, symbol, timeframe)
-        while True:
+        # The closed-bar guarantee must hold for FRESH responses too, not
-            now = time.time()
+        # just cache hits: the scanner fires seconds after the boundary,
-            cache_ttl = self.CANDLE_SECONDS.get(
+        # and a lagging feed can answer without the just-closed bar — the
-                timeframe, self.config.refresh_interval)
+        # scan would silently analyze (and session-check) the previous
-            with self._cache_lock:
+        # bar. Give the feed one short grace retry, then proceed loudly.
-                cached = self._cache.get(cache_key)
+        if not df.empty and not self._has_latest_closed_bar(df, timeframe):
-                cached_at = self._cache_timestamps.get(cache_key)
+            time.sleep(self.FRESH_FETCH_RETRY_SECONDS)
-                cached_size = self._cache_outputsize.get(cache_key, 0)
+            try:
-            if (
+                retry = self._fetch_raw(symbol, timeframe, outputsize)
-                cached is not None
+                retry = self._drop_forming_candle(retry, timeframe)
-                and cached_at is not None
+                retry = self._validate_and_clean(retry, symbol, timeframe)
-                and (now - cached_at) < cache_ttl
+                if retry is not None and not retry.empty:
-                and not cached.empty
+                    df = retry
-                # A caller asking for MORE history than the cached fetch
+            except DataIntegrityError as e:
                # requested must not be served the short frame silently.
                and cached_size >= outputsize
            ):
                if (self._is_data_fresh(cached, symbol, timeframe)
                        and self._has_latest_closed_bar(cached, timeframe)):
                    return cached
                logger.warning(f"{symbol} {timeframe.value}: cached data is stale, refetching")
                with self._cache_lock:
                    self._cache.pop(cache_key, None)
                    self._cache_timestamps.pop(cache_key, None)
                    self._cache_outputsize.pop(cache_key, None)
            # Single-flight: concurrent scans for the same (symbol, TF)
            # must not each hit the API (the four TF loops fire together
            # at aligned boundaries and Oanda has no client throttle).
            # First caller claims the key and fetches; the rest wait,
            # then re-check the cache.
            with self._cache_lock:
                inflight = self._inflight.get(cache_key)
                if inflight is None:
                    self._inflight[cache_key] = threading.Event()
                    break
            inflight.wait(timeout=90)
        try:
            df = self._fetch_raw(symbol, timeframe, outputsize)
            df = self._drop_forming_candle(df, timeframe)
            df = self._validate_and_clean(df, symbol, timeframe)
            # The closed-bar guarantee must hold for FRESH responses too,
            # not just cache hits: the scanner fires seconds after the
            # boundary, and a lagging feed can answer without the
            # just-closed bar — the scan would silently analyze (and
            # session-check) the previous bar. Give the feed one short
            # grace retry, then proceed loudly.
            if not df.empty and not self._has_latest_closed_bar(df, timeframe):
                time.sleep(self.FRESH_FETCH_RETRY_SECONDS)
                try:
                    retry = self._fetch_raw(symbol, timeframe, outputsize)
                    retry = self._drop_forming_candle(retry, timeframe)
                    retry = self._validate_and_clean(retry, symbol, timeframe)
                    if retry is not None and not retry.empty:
                        df = retry
                except DataIntegrityError as e:
                    logger.warning(
                        f"{symbol} {timeframe.value}: boundary retry refetch "
                        f"failed ({e}) — keeping original data")
                if not self._has_latest_closed_bar(df, timeframe):
                    logger.warning(
                        f"{symbol} {timeframe.value}: feed lagging — just-closed bar "
                        f"still missing after retry; this scan may run one bar behind"
                    )
            if not self._is_data_fresh(df, symbol, timeframe):
                logger.warning(
-                    f"STALE DATA from API: {symbol} {timeframe.value} — "
+                    f"{symbol} {timeframe.value}: boundary retry refetch "
-                    f"last bar {df.index[-1]} is too old. Signal quality may be degraded."
+                    f"failed ({e}) — keeping original data")
            if not self._has_latest_closed_bar(df, timeframe):
                logger.warning(
                    f"{symbol} {timeframe.value}: feed lagging — just-closed bar "
                    f"still missing after retry; this scan may run one bar behind"
                )
-            with self._cache_lock:
+        if not self._is_data_fresh(df, symbol, timeframe):
-                self._cache[cache_key] = df
+            logger.warning(
-                self._cache_timestamps[cache_key] = now
+                f"STALE DATA from API: {symbol} {timeframe.value} — "
-                self._cache_outputsize[cache_key] = outputsize
+                f"last bar {df.index[-1]} is too old. Signal quality may be degraded."
-            return df
+            )
-        finally:
+
-            # Release waiters even when the fetch raised — they re-check
+        with self._cache_lock:
-            # the cache and (on a miss) claim the key themselves.
+            self._cache[cache_key] = df
-            with self._cache_lock:
+            self._cache_timestamps[cache_key] = now
-                ev = self._inflight.pop(cache_key, None)
+        return df
            if ev is not None:
                ev.set()
    def fetch_range(
        self,
@@ -252,7 +199,6 @@ class BaseDataProvider(ABC):
        chunks = []
        cursor_end = end
        empty_streak = 0
        error_streak = 0
        while cursor_end > start:
            cursor_start = max(start, cursor_end - timedelta(days=days_per_chunk))
            try:
@@ -261,27 +207,19 @@ class BaseDataProvider(ABC):
                    start=cursor_start, end=cursor_end,
                )
            except Exception as e:
-                # A raising chunk is a PROVIDER ERROR (auth, 429, 5xx,
+                # Providers raise for windows with no data (weekends,
-                # malformed payload) — not "no data". Three in a row means
+                # holidays, asset gaps). Skip the window, step back, retry
-                # the rest of the history is unreachable: raise instead of
+                # — but bail after a few consecutive empties.
                # returning a silently truncated frame (the backtester
                # would treat it as complete).
                logger.warning(
                    f"Chunk {cursor_start.date()}–{cursor_end.date()} failed: {e}"
                )
-                error_streak += 1
+                empty_streak += 1
-                if error_streak >= 3:
+                if empty_streak >= 3:
-                    raise self.DATA_ERROR(
+                    break
                        f"{symbol} {timeframe.value}: {error_streak} "
                        f"consecutive chunk failures walking back from "
                        f"{cursor_end.date()} — refusing to return "
                        f"truncated history (last error: {e})") from e
                cursor_end = cursor_start - timedelta(seconds=1)
                continue
            if chunk is None or chunk.empty:
                # Genuinely empty windows (weekends/holidays/asset gaps)
                # are normal; several in a row = start of history.
                empty_streak += 1
                if empty_streak >= 3:
                    break
@@ -289,18 +227,9 @@ class BaseDataProvider(ABC):
                continue
            empty_streak = 0
            error_streak = 0
            chunks.append(chunk)
            # Step back; dedup on concat.
-            new_cursor = chunk.index[0] - timedelta(seconds=1)
+            cursor_end = chunk.index[0] - timedelta(seconds=1)
            if new_cursor >= cursor_end:
                # A provider ignoring start/end would pin the cursor and
                # loop forever — refuse to spin.
                logger.warning(
                    f"{symbol} {timeframe.value}: pagination made no "
                    f"progress at {cursor_end} — stopping")
                break
            cursor_end = new_cursor
            if len(chunk) < 2:
                break
@@ -314,17 +243,7 @@ class BaseDataProvider(ABC):
        df = self._drop_forming_candle(df, timeframe)
        start_ts = pd.Timestamp(start)
        end_ts = pd.Timestamp(end)
-        df = df[(df.index >= start_ts) & (df.index <= end_ts)]
+        return df[(df.index >= start_ts) & (df.index <= end_ts)]
        # Coverage sanity: a head materially later than the requested
        # start means history was truncated (empty-streak bail or provider
        # depth limit). Loud, not fatal — some assets genuinely start late.
        if not df.empty and (df.index[0] - start_ts) > pd.Timedelta(days=7):
            logger.error(
                f"{symbol} {timeframe.value}: fetched history starts "
                f"{df.index[0]} but {start_ts} was requested — "
                f"~{(df.index[0] - start_ts).days} days missing from the "
                f"head; treat downstream results accordingly")
        return df
    def is_data_fresh(
        self, df: pd.DataFrame, symbol: str, timeframe: Timeframe
@@ -404,19 +323,8 @@ class BaseDataProvider(ABC):
            return df
        candle_seconds = self.CANDLE_SECONDS.get(timeframe, 3600)
        now = pd.Timestamp.now(tz="UTC")
-        # Loop, not a single slice: a misbehaving feed can return MORE
+        if df.index[-1] + pd.Timedelta(seconds=candle_seconds) > now:
        # than one not-yet-closed bar, and a single iloc[:-1] would let
        # the residual forming bar through cleaning (future-tolerance is
        # +5 min) straight into the signal engine.
        dropped = 0
        while (not df.empty
               and df.index[-1] + pd.Timedelta(seconds=candle_seconds) > now):
            df = df.iloc[:-1]
            dropped += 1
        if dropped > 1:
            logger.warning(
                "%s: feed returned %d forming/future bars — all dropped",
                timeframe.value, dropped)
        return df
    def _is_data_fresh(self, df: pd.DataFrame, symbol: str, timeframe: Timeframe) -> bool:
@@ -464,20 +372,6 @@ class BaseDataProvider(ABC):
            logger.warning(f"{symbol}: Dropping {future_mask.sum()} future timestamps")
            df = df[~future_mask]
        # Non-positive prices are corrupt, never real gold quotes. Reject
        # the row BEFORE the consistency repair below — repairing High/Low
        # from a corrupt 0.0 Open/Close would propagate the zero into
        # previously-valid fields (2026-08-04 audit, I45).
        nonpositive = (df[["Open", "High", "Low", "Close"]] <= 0).any(axis=1)
        if nonpositive.any():
            logger.warning(
                f"{symbol}: Dropping {nonpositive.sum()} bars with "
                f"non-positive prices (corrupt feed data)")
            df = df[~nonpositive]
            if df.empty:
                raise self.DATA_ERROR(
                    f"{symbol}: all bars had non-positive prices")
        invalid_ohlc = (
            (df["High"] < df["Low"]) |
            (df["High"] < df["Open"]) |
@@ -492,10 +386,7 @@ class BaseDataProvider(ABC):
                df.loc[idx, "High"] = max(vals + [df.loc[idx, "High"]])
                df.loc[idx, "Low"]  = min(vals + [df.loc[idx, "Low"]])
        # inf returns (division by a zero/na Close) would NaN the std and
        # silently disarm the whole 10σ pass — neutralize them first.
        returns = df["Close"].pct_change()
        returns = returns.replace([float("inf"), float("-inf")], float("nan"))
        std_ret = returns.std()
        if std_ret > 0:
            dev = returns - returns.mean()
@@ -8,7 +8,6 @@ Oanda's candle JSON, and exposes the real bid/ask spread from MBA candles.
 """
 import logging
 import time
 from datetime import datetime
 from typing import Dict, Optional
@@ -32,7 +31,6 @@ class OandaDataProvider(BaseDataProvider):
    name = "oanda"
    DATA_ERROR = OandaDataError
    RETRY_DELAY_S = 2.0
    OANDA_GRANULARITY = {
        Timeframe.M1: "M1",
@@ -126,35 +124,11 @@ class OandaDataProvider(BaseDataProvider):
        else:
            params["count"] = min(count, self.MAX_BARS_PER_REQUEST)
-        # One retry absorbs Oanda's sporadic spurious 401s (~15-20/day on
+        try:
-        # the practice API — the same token succeeds on adjacent requests)
+            resp = self._session.get(url, params=params, timeout=30)
-        # plus transient 5xx/network faults. A real credential problem
+        except requests.RequestException as e:
-        # fails both attempts and raises exactly as before. Other 4xx
+            raise OandaDataError(
-        # (bad request, 429) are never retried.
+                f"Oanda candle fetch failed for {instrument}: {e}") from e
        last_err: Optional[OandaDataError] = None
        for attempt in range(2):
            if attempt:
                time.sleep(self.RETRY_DELAY_S)
            try:
                resp = self._session.get(url, params=params, timeout=30)
            except requests.RequestException as e:
                last_err = OandaDataError(
                    f"Oanda candle fetch failed for {instrument}: {e}")
                last_err.__cause__ = e
                continue
            if resp.status_code == 401 or resp.status_code >= 500:
                last_err = OandaDataError(
                    f"Oanda candle fetch failed for {instrument}: "
                    f"HTTP {resp.status_code}: {resp.text[:300]}")
                if attempt == 0:
                    logger.warning(
                        f"Oanda fetch for {instrument} got HTTP "
                        f"{resp.status_code}; retrying in "
                        f"{self.RETRY_DELAY_S}s")
                continue
            break
        else:
            raise last_err
        if resp.status_code != 200:
            raise OandaDataError(
@@ -175,37 +149,23 @@ class OandaDataProvider(BaseDataProvider):
        rows = []
        last_bid_close = None
        last_ask_close = None
        skipped_malformed = 0
        for c in candles:
            if not c.get("complete", False):
                continue
            mid = c.get("mid", {})
            bid = c.get("bid", {})
            ask = c.get("ask", {})
            # A candle missing any OHLC field must be SKIPPED, never
            # defaulted to 0.0: a zero price flows through cleaning as a
            # "legit" bar and poisons ATR/Keltner/SL math — and a
            # mid-frame zero Close makes pct_change() emit inf, which
            # NaNs the std and disarms the 10σ outlier filter for the
            # whole frame (2026-08-04 audit, I45).
            if any(mid.get(k) is None for k in ("o", "h", "l", "c")):
                skipped_malformed += 1
                continue
            rows.append({
                "Datetime": pd.Timestamp(c["time"]).tz_convert("UTC"),
-                "Open": float(mid["o"]),
+                "Open": float(mid.get("o", 0)),
-                "High": float(mid["h"]),
+                "High": float(mid.get("h", 0)),
-                "Low": float(mid["l"]),
+                "Low": float(mid.get("l", 0)),
-                "Close": float(mid["c"]),
+                "Close": float(mid.get("c", 0)),
                "Volume": int(c.get("volume", 0)),
            })
            if bid and ask:
                last_bid_close = float(bid.get("c", 0))
                last_ask_close = float(ask.get("c", 0))
        if skipped_malformed:
            logger.warning(
                "Oanda returned %d malformed candle(s) for %s (missing "
                "OHLC fields) — skipped", skipped_malformed, instrument)
        if not rows:
            raise OandaDataError(
@@ -10,18 +10,15 @@ Tiingo's JSON into the normalized frame.
 Every failure mode raises `TiingoDataError` (a `DataIntegrityError`)
 with a specific message — the scan loops log it and carry on; nothing
-fails silently. NOTE: candles and mid-price ticks only. Tiingo's free
+fails silently. NOTE: this is the CANDLE feed only. Tiingo's free
 `/tiingo/fx/top` quote endpoint publishes indicative two-tier spreads
 and must not be used for spread display (removed 2026-07-23) —
-`get_spread` deliberately stays on the base-class default, and the
+`get_spread` deliberately stays on the base-class default.
 WebSocket stream (`iter_ticks`) exposes the mid price only, never
 bid/ask or a derived spread.
 """
 import json
 import logging
 from datetime import datetime
-from typing import AsyncIterator, Callable, Optional, Sequence, Tuple
+from typing import Optional
 import pandas as pd
 import requests
@@ -201,106 +198,3 @@ class TiingoDataProvider(BaseDataProvider):
            "Fetched %d bars for %s @ %s from Tiingo",
            len(df), ticker, resample)
        return df
    # ── Streaming quotes ───────────────────────────────────────────
    supports_streaming = True
    WS_URL = "wss://api.tiingo.com/fx"
    @staticmethod
    def _parse_ws_message(raw: str) -> Optional[Tuple[str, float]]:
        """Parse one WS message; return (tiingo_ticker, mid_price) for
        quote updates, None for heartbeats ("H"), subscription acks
        ("I"), malformed JSON, or quotes without a usable price.
        Server error messages ("E", e.g. bad token) RAISE TiingoDataError:
        Tiingo authenticates inside the socket via the subscribe message,
        so an auth failure arrives as a data message followed by a clean
        close — swallowing it would make the caller's reconnect loop
        hammer the endpoint with no backoff (backoff only grows on
        exceptions).
        """
        try:
            msg = json.loads(raw)
        except json.JSONDecodeError:
            logger.warning("Tiingo WS: malformed JSON, skipping")
            return None
        mtype = msg.get("messageType")
        if mtype == "H":
            return None
        if mtype == "I":
            logger.info("Tiingo WS: subscription acknowledged")
            return None
        if mtype == "E":
            raise TiingoDataError(
                f"Tiingo WS server error: {str(msg.get('response'))[:200]}")
        if mtype != "A":
            return None
        data = msg.get("data")
        if not isinstance(data, list) or len(data) < 8 or data[0] != "Q":
            return None
        # Live-verified field order (2026-08-03): ["Q", ticker, timestamp,
        # bidSize, bid, mid, askSize, ask] — ask is index 7, NOT 6 as the
        # docs' table suggests (index 6 is askSize).
        ticker = data[1]
        bid, mid, ask = data[4], data[5], data[7]
        if mid is None and bid is not None and ask is not None:
            mid = (bid + ask) / 2
        try:
            return ticker, float(mid)
        except (TypeError, ValueError):
            logger.warning("Tiingo WS: quote for %s has no usable price",
                           ticker)
            return None
    async def iter_ticks(
        self,
        symbols: Sequence[str],
        on_connect: Optional[Callable[[], None]] = None,
    ) -> AsyncIterator[Tuple[str, float]]:
        """One connection's lifetime of (canonical_symbol, mid) ticks.
        The caller owns the reconnect/backoff loop around this generator.
        Divergences from the Twelve Data reference implementation, both
        deliberate: `on_connect` fires on the FIRST RECEIVED message (the
        "I" ack in practice) rather than after the subscribe send, so a
        rejected token cannot reset the caller's backoff; and there is no
        client heartbeat task — Tiingo sends "H" frames itself and the
        websockets library's built-in ping keepalive detects dead sockets.
        """
        import websockets
        if self._api_key is None:
            raise TiingoDataError(
                "No Tiingo API key. Set the TIINGO_API_KEY env var or "
                "data_providers.tiingo.api_key in config.json.")
        ws_to_pair = {self._to_tiingo_ticker(s): s for s in symbols}
        async with websockets.connect(self.WS_URL) as ws:
            await ws.send(json.dumps({
                "eventName": "subscribe",
                "authorization": self._api_key,
                "eventData": {
                    "thresholdLevel": 5,
                    "tickers": sorted(ws_to_pair),
                },
            }))
            logger.info(
                "Tiingo WS connected, subscribing to %d tickers",
                len(ws_to_pair))
            connected = False
            async for raw in ws:
                parsed = self._parse_ws_message(raw)
                if not connected:
                    connected = True
                    if on_connect is not None:
                        on_connect()
                if parsed is None:
                    continue
                pair_symbol = ws_to_pair.get(parsed[0])
                if pair_symbol is None:
                    continue
                yield pair_symbol, parsed[1]
@@ -210,12 +210,6 @@ class TwelveDataProvider(BaseDataProvider):
        The TD WS speaks display symbols ("XAU/USD"); ticks are mapped back
        to the canonical pair symbols passed in. The caller owns the
        reconnect/backoff loop around this generator.
        `on_connect` fires on the FIRST RECEIVED frame, not after the
        subscribe send (2026-08-04 audit H38b, mirroring the Tiingo
        provider) — firing pre-receive let a rejected API key reset the
        caller's reconnect backoff on every attempt, pinning it at the
        minimum forever.
        """
        import websockets
@@ -228,14 +222,11 @@ class TwelveDataProvider(BaseDataProvider):
            }
            await ws.send(json.dumps(subscribe))
            logger.info(f"WebSocket connected, subscribed to {len(ws_to_pair)} symbols")
            if on_connect is not None:
                on_connect()
            hb_task = asyncio.create_task(self._ws_heartbeat(ws))
            connected = False
            try:
                async for raw in ws:
                    if not connected:
                        connected = True
                        if on_connect is not None:
                            on_connect()
                    parsed = self._parse_ws_price(raw)
                    if parsed is None:
                        continue
@@ -10,12 +10,9 @@ import os
 from dataclasses import dataclass, field
 from datetime import datetime, timezone
 from typing import Dict, List, Optional
 from zoneinfo import ZoneInfo
 import pandas as pd
 _NY_TZ = ZoneInfo("America/New_York")
 from chart_patterns import detect_chart_pattern
 from config import PlatformConfig, SignalStrength, Timeframe
 from providers import BaseDataProvider, make_data_provider
@@ -26,19 +23,15 @@ logger = logging.getLogger(__name__)
 def _is_forex_weekend(now_utc: datetime) -> bool:
-    """Gold/FX market closed for the weekend: Friday 17:00 → Sunday 18:00
+    """Forex market closed: Friday 22:00 UTC → Sunday 22:00 UTC (Sydney open)."""
-    NEW YORK time (= Fri 21:00 UTC in US summer, 22:00 in winter —
+    wd = now_utc.weekday()
-    matches measured Oanda feed behavior; 2026-08-04 audit I43).
+    hour = now_utc.hour
-    Deliberate copy of providers.base._is_forex_weekend — a Module 21
+    if wd == 4 and hour >= 22:  # Friday after 22:00
-    test asserts the copies agree across DST transitions."""
+        return True
-    ny = now_utc.astimezone(_NY_TZ)
+    if wd == 5:  # All Saturday
-    wd = ny.weekday()
+        return True
-    if wd == 4:                      # Friday
+    if wd == 6 and hour < 22:  # Sunday before 22:00
        return ny.hour >= 17
    if wd == 5:                      # Saturday
        return True
    if wd == 6:                      # Sunday
        return ny.hour < 18
    return False
@@ -86,7 +79,6 @@ class SignalReport:
    event_description: str = ""
    news_sentiment: float = 0.0
    news_description: str = ""
    news_summary: str = ""  # LLM market-context blurb (display-only)
    final_signal: SignalStrength = SignalStrength.NEUTRAL
    final_score: float = 0.0
@@ -314,44 +306,6 @@ class SignalAggregator:
            return None
        return float(adx.iloc[-1])
    def _detect_big_candle(self, primary_df: pd.DataFrame) -> Optional[tuple]:
        """Largest abnormal candle in the recent primary-TF window.
        Returns (timestamp, ratio) of the biggest bar whose range is
        >= big_candle_atr_multiple x the PRIOR bar's ATR within the last
        big_candle_lookback_hours of primary-TF bars, else None. The
        prior-bar ATR (shift 1) matters: a huge candle inflates the
        Wilder ATR immediately, so comparing against the concurrent ATR
        under-detects the very bars this gate exists to catch.
        Pure pandas on the caller-sliced df, so the live path and the
        backtest replay (which calls analyze_from_dataframes on prefix
        slices) compute bit-identical answers. The tail slice keeps the
        per-call cost bounded (~n + 42 rows); min_periods NaNs and short
        histories degrade to "no trip".
        """
        cfg = self.config.xau_strategy
        bar_secs = BaseDataProvider.CANDLE_SECONDS[self.config.primary_timeframe]
        n = max(1, int(cfg.big_candle_lookback_hours * 3600 // bar_secs))
        period = self.config.technical.atr_period
        tail = primary_df.iloc[-(n + 3 * period):]
        high, low, close = tail["High"], tail["Low"], tail["Close"]
        tr = pd.concat(
            [high - low,
             (high - close.shift()).abs(),
             (low - close.shift()).abs()],
            axis=1,
        ).max(axis=1)
        atr = tr.ewm(alpha=1 / period, min_periods=period, adjust=False).mean()
        ratio = (high - low) / atr.shift(1)
        window = ratio.iloc[-n:]
        if window.empty:
            return None
        max_ratio = window.max()
        if pd.isna(max_ratio) or max_ratio < cfg.big_candle_atr_multiple:
            return None
        return window.idxmax(), float(max_ratio)
    def _apply_m5_filters(self, report: SignalReport) -> None:
        """Extra selectivity for M5 — noisy TF needs stronger confirmation."""
        if report.signal_quality not in ("A+", "A", "B", "C"):
@@ -840,13 +794,11 @@ class SignalAggregator:
            sentiment, sent_desc = news.get_pair_sentiment(symbol, context)
            report.news_sentiment = sentiment
            report.news_description = sent_desc
            report.news_summary = context.llm_summary
        else:
            report.event_risk = "clear"
            report.event_description = ""
            report.news_sentiment = 0.0
            report.news_description = ""
            report.news_summary = ""
        # ── 5. Spread estimate from primary data (avoids extra API call) ─
        try:
@@ -1005,30 +957,6 @@ class SignalAggregator:
                    report.signal_quality = "D"
                    report.trade_setup = None
        # ── 6h. Big-candle entry gate (opt-in via per-TF factories) ──
        # An abnormally large candle in the recent window = institutional
        # burst; price may reverse violently, so NO entry of any kind may
        # fire. Detection runs for ANY grade: the warning is the
        # suppression channel for the XAU_PB path (which accepts D) and
        # the XAU_CP path (which accepts even F). The D-downgrade itself
        # touches actionable grades only — F stays F (same convention as
        # 6e).
        if (self.config.xau_strategy.enabled
                and report.display_name == "XAU/USD"
                and self.config.xau_strategy.big_candle_atr_multiple > 0):
            big_candle = self._detect_big_candle(primary_df)
            if big_candle is not None:
                bc_ts, bc_ratio = big_candle
                report.warnings.append(
                    f"BIG CANDLE FILTER: {bc_ratio:.1f}x ATR bar at {bc_ts} "
                    f"within "
                    f"{self.config.xau_strategy.big_candle_lookback_hours:g}h "
                    f"— no entries"
                )
                if report.signal_quality in ("A+", "A", "B", "C"):
                    report.signal_quality = "D"
                    report.trade_setup = None
        # ── 6i. M5 noise gate (gold-tuned thresholds — XAU only) ──
        if (self.config.primary_timeframe == Timeframe.M5
                and report.display_name == "XAU/USD"):
@@ -1057,9 +985,7 @@ class SignalAggregator:
                and report.display_name == "XAU/USD"
                and self.config.xau_strategy.m5_pullback_enabled
                and report.trade_setup is None
-                and report.signal_quality != "F"
+                and report.signal_quality != "F"):
                and not any(w.startswith("BIG CANDLE FILTER")
                            for w in report.warnings)):
            self._try_m5_pullback(report, primary_df, pip_value)
        # ── 10. Chart-pattern entry (XAU_CP) ────────────────────────
@@ -1076,8 +1002,7 @@ class SignalAggregator:
                and self.config.xau_strategy.cp_enabled
                and report.trade_setup is None
                and not any(w.startswith(("SESSION FILTER", "DAY FILTER",
-                                          "WEEKEND FILTER",
+                                          "WEEKEND FILTER"))
                                          "BIG CANDLE FILTER"))
                            for w in report.warnings)):
            self._try_chart_pattern(report, primary_df, pip_value)
@@ -1260,17 +1185,7 @@ class SignalAggregator:
            ema200 = primary.key_levels.get("ema_trend")
            if ema200 and ema200 > 0 and report.current_price > 0:
                ema_dist_pct = abs(report.current_price - ema200) / ema200 * 100.0
-                # FB2_EMA_LO: research override for the EMA200-proximity
+                _lo = float(os.environ.get("FB2_EMA_LO", "0.5"))
                # penalty band; the tuned default is 0.5. An unparseable
                # value must degrade to the default, never crash the scan
                # (2026-08-04 audit; hardened per owner delegation).
                try:
                    _lo = float(os.environ.get("FB2_EMA_LO", "0.5"))
                except (TypeError, ValueError):
                    logger.warning(
                        "FB2_EMA_LO=%r is not a number — using tuned "
                        "default 0.5", os.environ.get("FB2_EMA_LO"))
                    _lo = 0.5
                if ema_dist_pct < _lo:
                    report.warnings.append(f"NEAR EMA200 ({ema_dist_pct:.2f}%) — chop risk")
                    penalties.append(("ema200_near", 5))
@@ -1,245 +0,0 @@
 <?php
 /**
 * On-demand gold-impact AI summary for one published headline.
 *
 * The ONLY dynamic piece of the otherwise-static dashboard. Serves
 * GET ?id=<sha1-of-headline> and returns {"summary": ..., "generated_at": ...}.
 *
 * Abuse model (the page is public):
 *   - WHITELIST: the id must match a headline in the local news.json —
 *     arbitrary text can never reach the LLM through this endpoint.
 *   - CACHE FOREVER: headlines are immutable, so each id costs at most
 *     one DeepSeek call, ever (ai/<id>.json). Cache is checked BEFORE
 *     the whitelist on purpose: summaries stay reachable after the
 *     headline ages out of the 24 h feed window.
 *   - DAILY CAP: file-based UTC-day counter, FAIL CLOSED on counter I/O
 *     errors (a perms mistake must not mean unmetered spend).
 *
 * DeepSeek doctrine (mirrors llm_client.py — deliberate local copy, the
 * dashboard never imports bot code and this file never reads the repo's
 * config.json):
 *   - STATIC byte-identical system prompt (prefix-cache pricing).
 *   - "thinking": disabled is MANDATORY for V4-Flash (else it burns the
 *     timeout reasoning), temperature 0, json response_format.
 *   - One attempt, no retries; every failure returns a JSON error the
 *     page renders as a graceful fallback.
 *
 * Key file lives OUTSIDE the docroot: /etc/gold-dashboard/deepseek.key
 * (root:apache 640). Override paths/caps via Apache SetEnv:
 * DEEPSEEK_KEY_FILE, AI_SUMMARY_DAILY_CAP.
 */
 const NEWS_FILE = __DIR__ . '/news.json';
 const CACHE_DIR = __DIR__ . '/ai';   // apache-owned; created at deploy time
 const API_URL   = 'https://api.deepseek.com/chat/completions';
 const MODEL     = 'deepseek-v4-flash';
 const TIMEOUT   = 12;
 const MAX_TOKENS = 200;
 const SUMMARY_MAX_CHARS = 400;
 // .ht_ prefix: Apache's default <Files ".ht*"> rule denies these even
 // though they sit in the public docroot.
 const CAP_FILE = CACHE_DIR . '/.ht_cap.json';
 const SYSTEM_PROMPT = <<<'PROMPT'
 You are a forex/gold news analyst. You receive one news item as JSON:
 {"headline", "description", "score", "xau_score", "keywords", "currencies"}.
 "xau_score" is the platform's automated gold-effect score for this item,
 -1.0 (gold down) to 1.0 (gold up); "score" is overall market sentiment.
 Both are automated and can be wrong — judge the news yourself.
 In 1-2 plain-English sentences (max 45 words) explain what this news
 means for the GOLD (XAU/USD) price and WHY it is bullish, bearish, or
 neutral. If your reading clearly contradicts xau_score, add "(the
 automated score disagrees)". Never write the words "xau_score" or
 "score" — a general reader sees this text.
 Gold doctrine: hawkish Fed / rate hikes / rising real yields / strong
 dollar are gold-bearish; easing / rate cuts / safe-haven flows /
 geopolitical stress / weak dollar are gold-bullish.
 Reply with ONLY JSON: {"summary": "..."}
 PROMPT;
 function respond(int $status, array $obj, string $cache = 'no-store'): void
 {
    http_response_code($status);
    header('Content-Type: application/json; charset=utf-8');
    header('Cache-Control: ' . $cache);
    echo json_encode($obj, JSON_UNESCAPED_SLASHES | JSON_UNESCAPED_UNICODE);
    exit;
 }
 function atomic_write(string $path, string $data): bool
 {
    $tmp = $path . '.tmp.' . getmypid();
    if (@file_put_contents($tmp, $data) === false) {
        return false;
    }
    if (!@rename($tmp, $path)) {
        @unlink($tmp);
        return false;
    }
    return true;
 }
 // ── 1. id validation ────────────────────────────────────────────────
 $id = $_GET['id'] ?? '';
 if (!preg_match('/^[a-f0-9]{40}$/', $id)) {
    respond(400, ['error' => 'bad_id']);
 }
 // ── 2. cache first (immutable headlines → summaries never change) ───
 $cachePath = CACHE_DIR . '/' . $id . '.json';
 $cached = @file_get_contents($cachePath);
 if ($cached !== false && $cached !== '') {
    http_response_code(200);
    header('Content-Type: application/json; charset=utf-8');
    header('Cache-Control: max-age=3600');
    echo $cached;
    exit;
 }
 // ── 3. whitelist against the published feed ─────────────────────────
 $news = json_decode((string)@file_get_contents(NEWS_FILE), true);
 if (!is_array($news)) {
    respond(503, ['error' => 'no_feed']);
 }
 $hit = null;
 foreach ($news['headlines'] ?? [] as $h) {
    // Fallback sha1(): a publisher that predates the id field still
    // serves a valid whitelist during the restart window.
    $hid = $h['id'] ?? sha1($h['headline'] ?? '');
    if ($hid === $id) {
        $hit = $h;
        break;
    }
 }
 if ($hit === null) {
    respond(404, ['error'   => 'unknown_id',
                  'message' => 'Headline is no longer in the feed.']);
 }
 // ── 4. daily cap (fail closed) ──────────────────────────────────────
 $dailyCap = (int)(getenv('AI_SUMMARY_DAILY_CAP') ?: 150);
 $capFh = @fopen(CAP_FILE, 'c+');
 if ($capFh === false || !flock($capFh, LOCK_EX)) {
    respond(503, ['error' => 'cap_io']);
 }
 $today = gmdate('Y-m-d');
 $cap = json_decode((string)stream_get_contents($capFh), true);
 if (!is_array($cap) || ($cap['day'] ?? '') !== $today) {
    $cap = ['day' => $today, 'count' => 0];
 }
 if ($cap['count'] >= $dailyCap) {
    flock($capFh, LOCK_UN);
    fclose($capFh);
    respond(429, ['error' => 'cap_exceeded']);
 }
 // Increment BEFORE the LLM call — attempts count against the cap
 // (same semantics as llm_client._under_cap).
 $cap['count']++;
 rewind($capFh);
 if (ftruncate($capFh, 0) === false
        || fwrite($capFh, json_encode($cap)) === false) {
    flock($capFh, LOCK_UN);
    fclose($capFh);
    respond(503, ['error' => 'cap_io']);
 }
 fflush($capFh);
 flock($capFh, LOCK_UN);
 fclose($capFh);
 // ── 5. API key ──────────────────────────────────────────────────────
 $keyFile = getenv('DEEPSEEK_KEY_FILE') ?: '/etc/gold-dashboard/deepseek.key';
 $apiKey = trim((string)@file_get_contents($keyFile));
 if ($apiKey === '') {
    respond(503, ['error' => 'no_key']);
 }
 // ── 6. per-id stampede lock, then re-check the cache ────────────────
 $lockFh = @fopen(CACHE_DIR . '/.ht_lock_' . $id, 'c');
 if ($lockFh !== false) {
    flock($lockFh, LOCK_EX);   // blocks while a concurrent winner generates
    $cached = @file_get_contents($cachePath);
    if ($cached !== false && $cached !== '') {
        flock($lockFh, LOCK_UN);
        fclose($lockFh);
        http_response_code(200);
        header('Content-Type: application/json; charset=utf-8');
        header('Cache-Control: max-age=3600');
        echo $cached;
        exit;
    }
 }
 // ── 7. DeepSeek call (exact chat_json body shape) ───────────────────
 $userPayload = json_encode([
    'headline'    => $hit['headline'] ?? '',
    'description' => $hit['description'] ?? null,
    'score'       => $hit['score'] ?? null,
    'xau_score'   => $hit['score_by_currency']['XAU'] ?? null,
    'keywords'    => $hit['keywords'] ?? [],
    'currencies'  => $hit['currencies'] ?? [],
 ], JSON_UNESCAPED_UNICODE);
 $body = json_encode([
    'model'    => MODEL,
    'messages' => [
        ['role' => 'system', 'content' => SYSTEM_PROMPT],
        ['role' => 'user',   'content' => $userPayload],
    ],
    'response_format' => ['type' => 'json_object'],
    'temperature'     => 0,
    'thinking'        => ['type' => 'disabled'],
    'max_tokens'      => MAX_TOKENS,
 ]);
 $ch = curl_init(API_URL);
 curl_setopt_array($ch, [
    CURLOPT_POST           => true,
    CURLOPT_POSTFIELDS     => $body,
    CURLOPT_HTTPHEADER     => ['Content-Type: application/json',
                               'Authorization: Bearer ' . $apiKey],
    CURLOPT_RETURNTRANSFER => true,
    CURLOPT_TIMEOUT        => TIMEOUT,
 ]);
 $raw = curl_exec($ch);
 $httpCode = curl_getinfo($ch, CURLINFO_RESPONSE_CODE);
 curl_close($ch);
 // ── 8. parse (any failure → 502, never cached) ──────────────────────
 $summary = null;
 if ($raw !== false && $httpCode === 200) {
    $resp = json_decode($raw, true);
    $content = $resp['choices'][0]['message']['content'] ?? '';
    $parsed = json_decode((string)$content, true);
    if (is_array($parsed) && is_string($parsed['summary'] ?? null)
            && trim($parsed['summary']) !== '') {
        $summary = trim($parsed['summary']);
        if (mb_strlen($summary) > SUMMARY_MAX_CHARS) {
            $summary = mb_substr($summary, 0, SUMMARY_MAX_CHARS) . '…';
        }
    }
 }
 if ($lockFh !== false && $summary === null) {
    flock($lockFh, LOCK_UN);
    fclose($lockFh);
 }
 if ($summary === null) {
    respond(502, ['error' => 'llm_failed']);
 }
 // ── 9. cache + respond ──────────────────────────────────────────────
 $out = json_encode(['summary' => $summary, 'generated_at' => gmdate('c')],
                   JSON_UNESCAPED_SLASHES | JSON_UNESCAPED_UNICODE);
 if (!atomic_write($cachePath, $out)) {
    // Degrade to per-click generation rather than a broken feature —
    // the daily cap still bounds spend.
    error_log('news_summary.php: cache write failed for ' . $cachePath);
 }
 if ($lockFh !== false) {
    flock($lockFh, LOCK_UN);
    fclose($lockFh);
 }
 http_response_code(200);
 header('Content-Type: application/json; charset=utf-8');
 header('Cache-Control: max-age=3600');
 echo $out;
@@ -12,15 +12,11 @@ Auto-scan:   Starts automatically on boot if TELEGRAM_CHAT_ID is set.
             M15 fires at :00/:15/:30/:45 + 10s buffer.
             H1  fires at :00 + 10s buffer.
             H4  fires at :00 every 4 hours + 10s buffer.
-WebSocket:   Streams live ticks from the streaming provider — config.json
+WebSocket:   Streams live ticks from TwelveData for XAU/USD.
             `data_providers.streaming` when set (e.g. tiingo), else the
             active provider if it streams; auto-disabled otherwise.
             When price crosses a key structure level (swing S/R, BB,
             EMA200) cached from the last boundary scan, triggers an
             immediate M15 scan and sends a Level Break Alert.
             5-minute cooldown prevents spam.
 News:        /news replies with the net XAU/USD news sentiment score
             (display-only — news never gates signals).
 Owner lock:  All commands and buttons only respond to TELEGRAM_CHAT_ID
             (the owner). Everyone else gets a polite refusal — the bot
@@ -71,8 +67,6 @@ from app_config import (
    telegram_channel_id,
 )
 from providers import BaseDataProvider, make_data_provider
 from llm_client import get_llm_client
 from news_engine import FundamentalContext, NewsEngine
 from pair_signals import PAIR_PRODUCERS
 from signal_aggregator import SignalAggregator, SignalReport
@@ -104,12 +98,6 @@ logging.basicConfig(level=getattr(logging, _APP.log_level, logging.INFO),
                    force=True)
 logging.getLogger("httpx").setLevel(logging.WARNING)
 logging.getLogger("httpcore").setLevel(logging.WARNING)
 # Never let DEBUG runs write credentials into the 30-day rotating log:
 # the websockets client logger emits the connect URI (Twelve Data puts
 # the API key in the query string) and frame contents (Tiingo's auth
 # frame carries the key) — 2026-08-04 audit, finding K47.
 logging.getLogger("websockets").setLevel(logging.INFO)
 logging.getLogger("websockets.client").setLevel(logging.INFO)
 logger = logging.getLogger("telegram_bot")
 TOKEN = telegram_token()
@@ -215,27 +203,6 @@ def _get_shared_dp() -> BaseDataProvider:
    return _shared_dp
 _streaming_dp: Optional[BaseDataProvider] = None
 def _get_streaming_dp() -> BaseDataProvider:
    """Provider for the tick stream: config.json data_providers.streaming
    when set (a SEPARATE instance — the candle cache stays on the active
    provider), else the shared active provider. Reads app config at call
    time, not the import-time _APP snapshot, so tests can swap it via
    set_app_config()."""
    global _streaming_dp
    if _streaming_dp is None:
        app = get_app_config()
        if app.streaming_provider is None or \
                app.streaming_provider == app.active_provider:
            _streaming_dp = _get_shared_dp()
        else:
            _streaming_dp = make_data_provider(
                PlatformConfig(), app.streaming_provider)
    return _streaming_dp
 # ── Signal formatting ─────────────────────────────────────────────────
 _SIGNAL_EMOJI = {
@@ -405,8 +372,6 @@ def format_signal_message(report: SignalReport) -> str:
        extra.append(f"📋 {ind_summary}")
    for w in report.warnings:
        extra.append(f"⚠️ {_escape_md(w)}")
    if report.news_summary:
        extra.append(f"📰 {_escape_md(report.news_summary)}")
    if extra:
        lines += [""] + extra
@@ -607,45 +572,6 @@ def _commit_alert(key: str, report: SignalReport) -> None:
        _last_alert_setup.pop(key, None)
 # Per-key locks serializing the _should_alert → send → _commit_alert
 # window. The auto-scan loop and a concurrent breach handler share the
 # "XAU/USD_15min" dedup key; without the lock both pass _should_alert
 # while the other's send is still awaiting Telegram, and the same signal
 # goes out twice (2026-08-04 audit H36). Per-key rather than one global
 # lock so a slow send on one timeframe cannot head-of-line-block another
 # timeframe's alert. Keys are symbol × TF, so the dict stays tiny.
 _alert_locks: Dict[str, asyncio.Lock] = {}
 def _alert_lock(key: str) -> asyncio.Lock:
    lock = _alert_locks.get(key)
    if lock is None:
        lock = _alert_locks[key] = asyncio.Lock()
    return lock
 async def _send_alert(bot, chat_id: int, text: str) -> bool:
    """Markdown alert send with a plain-text fallback on parse failures,
    mirroring _safe_edit (2026-08-04 audit H35 — alert paths previously
    dropped the message on a BadRequest with only a warning logged).
    Returns True when a message reached the chat by either attempt."""
    try:
        await bot.send_message(chat_id, text, parse_mode=ParseMode.MARKDOWN)
        return True
    except BadRequest as exc:
        logger.warning(f"Markdown alert failed for chat {chat_id} ({exc}); "
                       f"retrying as plain text")
        try:
            await bot.send_message(chat_id, text)
            return True
        except Exception as exc2:
            logger.warning(f"Failed to send alert to chat {chat_id}: {exc2}")
            return False
    except Exception as exc:
        logger.warning(f"Failed to send alert to chat {chat_id}: {exc}")
        return False
 # ── Candle-boundary scheduler ────────────────────────────────────────
 _autoscan_chats: Set[int] = set()
@@ -674,11 +600,8 @@ def _seconds_until_boundary(interval_minutes: int, buffer: int = 10) -> float:
    elapsed_seconds = elapsed_minutes * 60 + now.second + now.microsecond / 1e6
    interval_seconds = interval_minutes * 60
    remaining = interval_seconds - elapsed_seconds + buffer
-    # NOTE: remaining > buffer always (elapsed < interval by construction);
+    if remaining <= buffer:
-    # a call at the exact boundary instant targets the NEXT boundary —
+        remaining += interval_seconds
    # steady-state loops wake at boundary+buffer, so only a process
    # started in the boundary microsecond skips one scan (2026-08-04
    # audit I42; the old `remaining <= buffer` guard was unreachable).
    return remaining
@@ -693,13 +616,7 @@ async def _scan_timeframe(
    reports: List[SignalReport] = []
    for i, result in enumerate(results):
        if isinstance(result, Exception):
-            # gather(return_exceptions=True) means there is no active
+            logger.exception(f"Auto-scan failed for {SCAN_PAIRS[i]} {tf.value}: {result}")
            # exception here — logger.exception would log "NoneType: None"
            # instead of the real stack; pass the instance explicitly.
            logger.error(
                f"Auto-scan failed for {SCAN_PAIRS[i]} {tf.value}: {result}",
                exc_info=result,
            )
        else:
            reports.append(result)
@@ -708,51 +625,36 @@ async def _scan_timeframe(
    _update_level_cache(reports)
-    candidates = [
+    actionable = [
        r for r in reports
        if r.final_signal != SignalStrength.NEUTRAL   # same gate as breach path
        and _passes_quality(r.signal_quality)
        and _should_alert(f"{r.symbol}_{tf.value}", r)
    ]
-    for report in candidates:
+    sent_ok: Set[int] = set()
-        key = f"{report.symbol}_{tf.value}"
+    for chat_id in _signal_targets():
-        # The lock spans check → send → commit so a concurrent breach
+        for i, report in enumerate(actionable):
        # handler on the same key cannot double-alert (H36).
        async with _alert_lock(key):
            if not _should_alert(key, report):
                continue
            detail = format_signal_message(report)
-            delivered = False
+            try:
-            for chat_id in _signal_targets():
+                await bot.send_message(chat_id, detail, parse_mode=ParseMode.MARKDOWN)
-                if await _send_alert(bot, chat_id, detail):
+                sent_ok.add(i)
-                    delivered = True
+            except Exception as exc:
-            # Commit only when the alert reached at least one target;
+                logger.warning(f"Failed to send signal for chat {chat_id}: {exc}")
-            # a fully failed send stays eligible for the next scan.
+    # Commit fingerprints only for alerts that reached at least one target;
-            if delivered:
+    # a fully failed send stays eligible for the next scan.
-                _commit_alert(key, report)
+    for i, report in enumerate(actionable):
        if i in sent_ok:
            _commit_alert(f"{report.symbol}_{tf.value}", report)
 # ── WebSocket level-breach monitor ──────────────────────────────────
-# Level cache keyed symbol → timeframe → {level_name: price}, with a
+_level_cache: Dict[str, Dict[str, float]] = {}
-# matching per-TF write timestamp. Keyed per TF (2026-08-04 audit H34)
+_level_cache_ts: Dict[str, float] = {}
 # because every scan path writes here — the M5/M15/H1/H4 boundary loops,
 # the breach rescan, the manual Scan-All button — and a single flat dict
 # meant the last writer wiped the other timeframes' swing/EMA levels
 # within minutes. Breach checks scan the union of all fresh TF entries.
 _level_cache: Dict[str, Dict[str, Dict[str, float]]] = {}
 _level_cache_ts: Dict[str, Dict[str, float]] = {}
 _LEVEL_CACHE_TTL = 3600
 _last_prices: Dict[str, float] = {}
 _last_price_ts: Dict[str, float] = {}
 _scan_cooldowns: Dict[str, float] = {}
 # Breaches are edge-triggered (only the crossing tick fires), so one that
 # lands while the symbol is inside the rescan cooldown would otherwise be
 # dropped forever (H33b). Park it here; the tick loop fires it once the
 # cooldown expires. symbol → (price, level_name, recorded_at).
 _pending_breach: Dict[str, tuple] = {}
 # A failed breach rescan shrinks the just-set cooldown to this retry
 # window instead of silencing the symbol for the full cooldown (H33a).
 _BREACH_RETRY_SECONDS = 30
 _cfg = PlatformConfig()
 _WS_COOLDOWN = _cfg.ws_cooldown_seconds
@@ -761,8 +663,6 @@ _breach_tasks: Set[asyncio.Task] = set()
 def _update_level_cache(reports: List[SignalReport]) -> None:
    """Each report updates only its own timeframe's slot — an M5 scan can
    no longer overwrite the H1/H4 swing levels (H34)."""
    now = time.time()
    for r in reports:
        if r.primary_analysis and r.primary_analysis.key_levels:
@@ -771,31 +671,26 @@ def _update_level_cache(reports: List[SignalReport]) -> None:
                if k in _LEVEL_KEYS
            }
            if levels:
-                tf = r.primary_analysis.timeframe
+                _level_cache[r.symbol] = levels
-                _level_cache.setdefault(r.symbol, {})[tf] = levels
+                _level_cache_ts[r.symbol] = now
                _level_cache_ts.setdefault(r.symbol, {})[tf] = now
 def _check_level_breach(symbol: str, price: float) -> Optional[str]:
-    """Scan the union of every fresh timeframe's levels; returns
+    levels = _level_cache.get(symbol)
-    '<tf> <level_name>' (e.g. '15min swing_high_1') on a crossing so the
+    if not levels:
-    alert label says which timeframe's level broke."""
+        return None
-    per_tf = _level_cache.get(symbol)
+
-    if not per_tf:
+    cache_age = time.time() - _level_cache_ts.get(symbol, 0)
    if cache_age > _LEVEL_CACHE_TTL:
        return None
    last = _last_prices.get(symbol)
    if last is None:
        return None
-    now = time.time()
+    for name, level in levels.items():
-    ts_map = _level_cache_ts.get(symbol, {})
+        if (last < level <= price) or (last > level >= price):
-    for tf, levels in per_tf.items():
+            return name
        if now - ts_map.get(tf, 0) > _LEVEL_CACHE_TTL:
            continue   # stale TF entry — skip, fresher TFs still count
        for name, level in levels.items():
            if (last < level <= price) or (last > level >= price):
                return f"{tf} {name}"
    return None
@@ -810,17 +705,9 @@ def _cleanup_cooldowns() -> None:
 async def _handle_breach(bot, pair_symbol: str, price: float, level_name: str) -> None:
    now = time.time()
    if now - _scan_cooldowns.get(pair_symbol, 0) < _WS_COOLDOWN:
        # Park the crossing instead of dropping it — price holding beyond
        # the level never re-fires the edge trigger, so without this the
        # breach would be lost forever (H33b). The tick loop fires it
        # once the cooldown expires; a newer crossing overwrites it.
        _pending_breach[pair_symbol] = (price, level_name, now)
        return
    # Early-set on purpose: this is what stops concurrent breach tasks
    # from stacking rescans. The failure path below rolls it back.
    _scan_cooldowns[pair_symbol] = now
    _pending_breach.pop(pair_symbol, None)   # this crossing is being handled
    display = PAIR_INFO.get(pair_symbol, {}).get("display", pair_symbol)
    label = level_name.replace("_", " ").title()
    logger.info(f"Level breach: {display} crossed {label} at {price} — rescanning M15")
@@ -830,42 +717,29 @@ async def _handle_breach(bot, pair_symbol: str, price: float, level_name: str) -
        _update_level_cache([report])
    except Exception:
        logger.exception(f"Breach scan failed for {display}")
        # Shrink the just-set cooldown to a short retry window so a
        # transient fetch error doesn't silence the symbol for the full
        # cooldown (H33a); the floor bounds the retry rate while the
        # provider stays down. Re-park the breach so the tick loop
        # retries it (setdefault: a newer crossing recorded meanwhile
        # wins over this stale one).
        retry = min(_BREACH_RETRY_SECONDS, _WS_COOLDOWN)
        _scan_cooldowns[pair_symbol] = now - _WS_COOLDOWN + retry
        _pending_breach.setdefault(pair_symbol, (price, level_name, now))
        return
    # Alert only when the rescan produced an actionable, changed signal —
    # same gates as the auto-scanner, so a breach can't spam the channel.
    if (report.final_signal == SignalStrength.NEUTRAL
-            or not _passes_quality(report.signal_quality)):
+            or not _passes_quality(report.signal_quality)
            or not _should_alert(f"{pair_symbol}_{Timeframe.M15.value}", report)):
        return
-    key = f"{pair_symbol}_{Timeframe.M15.value}"
+    header = (
-    # Lock spans check → send → commit; without it this handler and an
+        f"⚡ *LEVEL BREAK* — {_escape_md(display)} crossed "
-    # in-flight M15 auto-scan both pass _should_alert and double-alert
+        f"*{_escape_md(label)}* @ `{_fmt_price(price)}`\n\n"
-    # the same signal (H36).
+    )
-    async with _alert_lock(key):
+    sent = False
-        if not _should_alert(key, report):
+    for chat_id in _signal_targets():
            return
        header = (
            f"⚡ *LEVEL BREAK* — {_escape_md(display)} crossed "
            f"*{_escape_md(label)}* @ `{_fmt_price(price)}`\n\n"
        )
        detail = header + format_signal_message(report)
-        sent = False
+        try:
-        for chat_id in _signal_targets():
+            await bot.send_message(chat_id, detail, parse_mode=ParseMode.MARKDOWN)
-            if await _send_alert(bot, chat_id, detail):
+            sent = True
-                sent = True
+        except Exception as exc:
-        if sent:
+            logger.warning(f"Failed to send breach alert to chat {chat_id}: {exc}")
-            _commit_alert(key, report)
+    if sent:
        _commit_alert(f"{pair_symbol}_{Timeframe.M15.value}", report)
 def _breach_task_done(task: asyncio.Task) -> None:
@@ -881,32 +755,22 @@ async def _ws_monitor(bot) -> None:
    """Level-breach tick monitor. The provider owns the connection and
    protocol (`iter_ticks`); this loop owns reconnect/backoff and breach
    dispatch. Providers without streaming quotes simply disable it."""
-    dp = _get_streaming_dp()
+    dp = _get_shared_dp()
    if not dp.supports_streaming:
        logger.info(
            "Data provider %r has no streaming quotes — "
            "level-breach monitor disabled", dp.name)
        return
    if getattr(dp, "_api_key", "unused") is None:
        # Guard, not a traceback loop: without a key every connect
        # attempt would raise and re-log a full traceback every ≤60s.
        logger.warning(
            "Streaming provider %r has no API key — "
            "level-breach monitor disabled", dp.name)
        return
    backoff = 1
    while _feed_active():
        got_tick = False
        try:
-            async for pair_symbol, price in dp.iter_ticks(SCAN_PAIRS):
+            def _on_connect() -> None:
-                if not got_tick:
+                nonlocal backoff
-                    # Reset backoff only once the connection actually
+                backoff = 1
-                    # produced data — resetting on connect let a
+
-                    # rejected key or accept-then-close server pin the
+            async for pair_symbol, price in dp.iter_ticks(
-                    # backoff at minimum forever (H38a/H38b).
+                    SCAN_PAIRS, on_connect=_on_connect):
                    got_tick = True
                    backoff = 1
                if not _feed_active():
                    break
@@ -915,32 +779,11 @@ async def _ws_monitor(bot) -> None:
                _last_price_ts[pair_symbol] = time.time()
                if breached:
                    # A fresh crossing supersedes any parked one.
                    _pending_breach.pop(pair_symbol, None)
                    task = asyncio.create_task(
                        _handle_breach(bot, pair_symbol, price, breached)
                    )
                    _breach_tasks.add(task)
                    task.add_done_callback(_breach_task_done)
                elif pair_symbol in _pending_breach:
                    pend = _pending_breach[pair_symbol]
                    if time.time() - pend[2] > _LEVEL_CACHE_TTL:
                        # Ancient parked breach (feed gap/weekend) — an
                        # alert about an hours-old crossing would only
                        # mislead; the level cache it came from is stale
                        # anyway.
                        _pending_breach.pop(pair_symbol)
                    elif (time.time() - _scan_cooldowns.get(pair_symbol, 0)
                            >= _WS_COOLDOWN):
                        # Crossing parked during the cooldown: fire it now
                        # that the window expired — the edge trigger alone
                        # would have dropped it forever (H33b).
                        _pending_breach.pop(pair_symbol)
                        task = asyncio.create_task(
                            _handle_breach(bot, pair_symbol, pend[0], pend[1])
                        )
                        _breach_tasks.add(task)
                        task.add_done_callback(_breach_task_done)
                _cleanup_cooldowns()
@@ -950,16 +793,6 @@ async def _ws_monitor(bot) -> None:
            logger.exception(f"WebSocket error, reconnecting in {backoff}s")
            await asyncio.sleep(backoff)
            backoff = min(backoff * 2, 60)
        else:
            # Clean server close: the async-for ended without raising.
            # This must back off and log like the error path — silently
            # reconnecting with zero delay hot-loops against a server
            # that accepts and immediately closes (H38a).
            if _feed_active():
                logger.warning(
                    f"WebSocket closed cleanly, reconnecting in {backoff}s")
                await asyncio.sleep(backoff)
                backoff = min(backoff * 2, 60)
    logger.info("WebSocket monitor stopped")
@@ -1002,25 +835,15 @@ _ws_task: Optional[asyncio.Task] = None
 def _start_scheduler(bot) -> None:
    global _ws_task
-    # Per-TF respawn (2026-08-04 audit H37): drop finished loop tasks and
+    alive = [t for t in _autoscan_tasks if not t.done()]
-    # recreate only the missing ones. The old all-or-nothing alive check
+    if not alive:
-    # left a single dead TF loop unmonitored forever while its siblings
+        _autoscan_tasks.clear()
-    # lived — and with the channel feed active, /autoscan never reaches
+        for tf, interval in _AUTOSCAN_LOOPS:
-    # _stop_scheduler, so nothing short of a process restart recovered it.
+            _autoscan_tasks.append(
-    _autoscan_tasks[:] = [t for t in _autoscan_tasks if not t.done()]
+                asyncio.create_task(_candle_loop(bot, tf, interval))
-    alive_names = {t.get_name() for t in _autoscan_tasks}
+            )
    started = []
    for tf, interval in _AUTOSCAN_LOOPS:
        name = f"candle_loop_{tf.name}"
        if name in alive_names:
            continue
        _autoscan_tasks.append(
            asyncio.create_task(_candle_loop(bot, tf, interval), name=name)
        )
        started.append(tf.name)
    if started:
        logger.info("Candle-boundary scheduler started (%s)",
-                    "/".join(started))
+                    "/".join(tf.name for tf, _ in _AUTOSCAN_LOOPS))
    if _ws_task is None or _ws_task.done():
        _ws_task = asyncio.create_task(_ws_monitor(bot))
@@ -1096,7 +919,6 @@ _HELP_TEXT = (
    "/scan — Quick scan menu\n"
    "/autoscan — Toggle auto-scan on/off\n"
    "/status — Bot status, WS connection, live prices\n"
    "/news — Net news sentiment score for XAU/USD\n"
    "/cache — Show data cache age per pair/TF\n"
    "/clearcache — Wipe data cache, force fresh fetch\n"
    "/help — This message\n\n"
@@ -1115,87 +937,6 @@ async def cmd_help(update: Update, context: ContextTypes.DEFAULT_TYPE) -> None:
    await update.message.reply_text(_HELP_TEXT, parse_mode=ParseMode.MARKDOWN)
 def _fmt_age(ts: Optional[float]) -> str:
    """Compact age like '45s', '12m', '3.2h' for status lines."""
    if not ts:
        return "?"
    s = time.time() - ts
    if s < 90:
        return f"{s:.0f}s"
    if s < 5400:
        return f"{s / 60:.0f}m"
    return f"{s / 3600:.1f}h"
 def _news_health_lines() -> List[str]:
    """News / calendar / DeepSeek source health for /status.
    Reads the class-level `NewsEngine._source_health` written by the
    scan-path fetches plus the LLM client's health stamps — display
    only, never triggers a fetch. ⚠️ marks a degraded dependency;
    details live in the rotating log (transition-logged WARNINGs)."""
    lines = ["\n*News & AI:*"]
    # Snapshot: a scan thread may insert a source's FIRST entry while we
    # iterate (dict-changed-size RuntimeError); entry mutation is safe.
    for _ in range(3):
        try:
            sh = dict(NewsEngine._source_health)
            break
        except RuntimeError:
            continue
    else:
        sh = {}
    if not sh:
        lines.append("  _no news cycle yet (fills after first scan)_")
    else:
        cal = sh.get("calendar")
        ff = sh.get("calendar:forexfactory")
        if cal:
            if NewsEngine.source_is_failing(cal):
                lines.append(f"  ⚠️ Calendar: unavailable — {cal['detail']}"
                             f" ({_fmt_age(cal['err'])})")
            elif NewsEngine.source_is_failing(ff):
                lines.append(f"  ⚠️ Calendar: {cal['extra']} — ForexFactory"
                             f" down: {ff['detail']} ({_fmt_age(ff['err'])})")
            else:
                lines.append(f"  ✅ Calendar: {cal['extra']}"
                             f" ({_fmt_age(cal['ok'])} ago)")
        heads = sh.get("headlines")
        if heads:
            failing = NewsEngine.source_is_failing(heads)
            mark = "⚠️" if failing else "✅"
            detail = f" — {heads['detail']}" if failing else ""
            ts = heads["err"] if failing else heads["ok"]
            lines.append(f"  {mark} Headlines: {heads['extra']}{detail}"
                         f" ({_fmt_age(ts)} ago)")
        for name, entry in sorted(sh.items()):
            if name.startswith("rss:") and NewsEngine.source_is_failing(entry):
                lines.append(f"  ⚠️ {name[4:]}: {entry['detail']}"
                             f" ({_fmt_age(entry['err'])})")
    client = get_llm_client()
    if not client.available:
        lines.append("  DeepSeek: off — keyword scoring only")
    else:
        # The client's counter only resets on the first call of a new
        # UTC day — show 0 (not yesterday's total, or a phantom
        # cap-exhausted warning) until that reset happens.
        today = datetime.now(timezone.utc).date()
        used = client._count if client._count_day == today else 0
        cap = client._daily_cap
        scored = len(NewsEngine._llm_sent_cache)
        err = client.last_error
        if used >= cap:
            lines.append(f"  ⚠️ DeepSeek: daily cap {cap} exhausted —"
                         " keyword fallback until UTC midnight")
        elif err and err[0] > client.last_ok:
            lines.append(f"  ⚠️ DeepSeek: last call failed ({err[1]},"
                         f" {_fmt_age(err[0])}) — keyword fallback")
        else:
            lines.append(f"  ✅ DeepSeek: {used}/{cap} calls today,"
                         f" {scored} headlines scored")
    return lines
 def _status_text() -> str:
    lines = ["📊 *ForexBot2 Status*\n"]
@@ -1210,8 +951,7 @@ def _status_text() -> str:
        lines.append(f"📢 Channel feed: *on* (`{CHANNEL_ID}`)")
    if _ws_task and not _ws_task.done():
-        provider = _streaming_dp.name if _streaming_dp else "?"
+        lines.append("📡 WebSocket: *connected*")
        lines.append(f"📡 WebSocket: *connected* ({provider})")
    else:
        lines.append("❌ WebSocket: *disconnected*")
@@ -1229,16 +969,9 @@ def _status_text() -> str:
        lines.append("\n_No live prices yet (WS not streaming)_")
    cached_pairs = len(_level_cache)
-    cached_levels = sum(
+    cached_levels = sum(len(v) for v in _level_cache.values())
        len(levels) for per_tf in _level_cache.values()
        for levels in per_tf.values()
    )
    if cached_levels:
-        all_ts = [
+        oldest_age = time.time() - min(_level_cache_ts.values()) if _level_cache_ts else 0
            ts for per_tf in _level_cache_ts.values()
            for ts in per_tf.values()
        ]
        oldest_age = time.time() - min(all_ts) if all_ts else 0
        lines.append(f"\n*Level Cache:* {cached_levels} levels across {cached_pairs} pairs (oldest {oldest_age:.0f}s)")
    else:
        lines.append("\n*Level Cache:* empty — breach alerts start after first candle scan")
@@ -1254,8 +987,6 @@ def _status_text() -> str:
    if active_breach:
        lines.append(f"*Breach tasks:* {active_breach} running")
    lines.extend(_news_health_lines())
    lines.append(f"\n*Alert quality:* {_alert_min_quality}+ "
                 f"| re-arm {_ALERT_REARM_SECONDS // 60}m")
    return "\n".join(lines)
@@ -1266,38 +997,6 @@ async def cmd_status(update: Update, context: ContextTypes.DEFAULT_TYPE) -> None
    await update.message.reply_text(_status_text(), parse_mode=ParseMode.MARKDOWN)
 def _news_text(context: FundamentalContext, engine: NewsEngine) -> str:
    net, desc = engine.get_pair_sentiment(XAUUSD, context)
    if round(net, 2) == 0:
        net = 0.0  # a tiny negative mean would render as "-0.00"
    lines = [
        "📰 *XAU/USD News Score*\n",
        f"*Net score:* `{net:+.2f}` — {desc}",
    ]
    # A cold/failed feed yields 0.00 "Neutral" — flag it so a data outage
    # can't be mistaken for a genuinely neutral read.
    if not context.news_sentiment:
        lines.append("_no scored headlines yet — score defaults to 0_")
    return "\n".join(lines)
 def _news_report_sync() -> str:
    engine = NewsEngine(PlatformConfig().news)
    return _news_text(engine.get_fundamental_context(), engine)
@_owner_only
 async def cmd_news(update: Update, context: ContextTypes.DEFAULT_TYPE) -> None:
    msg = await update.message.reply_text("⏳ Fetching news…")
    loop = asyncio.get_running_loop()
    try:
        text = await loop.run_in_executor(_executor, _news_report_sync)
    except Exception as exc:
        logger.exception("News fetch failed")
        text = f"❌ News fetch failed: {_escape_md(str(exc))}"
    await msg.edit_text(text, parse_mode=ParseMode.MARKDOWN)
@_owner_only
 async def cmd_cache(update: Update, context: ContextTypes.DEFAULT_TYPE) -> None:
    dp = _get_shared_dp()
@@ -1513,7 +1212,6 @@ async def _post_init(app: Application) -> None:
        ("scan", "Quick scan buttons"),
        ("autoscan", "Toggle auto-scan on/off"),
        ("status", "Bot status, WS connection, live prices"),
        ("news", "XAU/USD net news sentiment score"),
        ("cache", "Show data cache age per pair/TF"),
        ("clearcache", "Wipe data cache, force fresh fetch"),
        ("help", "All commands"),
@@ -1543,11 +1241,7 @@ async def _post_init(app: Application) -> None:
                logger.error(f"Initial scan failed for {SCAN_PAIRS[i]}: {r}")
        if init_reports:
            _update_level_cache(init_reports)
-            n_levels = sum(
+            logger.info(f"Level cache populated: {sum(len(v) for v in _level_cache.values())} levels")
                len(levels) for per_tf in _level_cache.values()
                for levels in per_tf.values()
            )
            logger.info(f"Level cache populated: {n_levels} levels")
    except Exception:
        logger.exception("Initial scan failed — WS will wait for first candle scan")
@@ -1647,7 +1341,6 @@ def main() -> None:
    app.add_handler(CommandHandler("scan", cmd_scan))
    app.add_handler(CommandHandler("help", cmd_help))
    app.add_handler(CommandHandler("status", cmd_status))
    app.add_handler(CommandHandler("news", cmd_news))
    app.add_handler(CommandHandler("autoscan", cmd_autoscan))
    app.add_handler(CommandHandler("cache", cmd_cache))
    app.add_handler(CommandHandler("clearcache", cmd_clearcache))