Enable BUY-only chart patterns on M5 — bearish patterns dropped (OOS negative)

M5 factory: cp_enabled=True, patterns limited to inv_hs, double_bottom,
bull_flag. SELL CP lost -3.21R on OOS; BUY CP: +41.8R PF 1.60 on 12mo,
+65.2R PF 1.48 on 5yr. Adds +22R to the M5 book (708 vs 513 trades).
M15/H1 remain disabled — all failed OOS validation.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_015kby7M9JwpYi8NQT2yoxAi

.gitignore

@@ -1,10 +1,6 @@
 # Secrets
 .env
 .env.local
-config.json
-
-# Oanda daemon state
-oanda_state.json
 
 # Editor / tooling
 .claude/
@@ -28,12 +24,4 @@ Thumbs.db
 # Backtesting data (keep scripts, ignore generated files)
 backtest/cache/
 backtest/*.csv
-backtest/research/*.parquet
-backtest/research/duka_m5_boundary_spreads.csv
-backtest/trailgrid/
-backtest/trailgrid5m/
 backtest/__pycache__/
-.env.*
-logs/
-backtest/research/commit_ladder_2026-08/
-backtest/research/llm_news_eval_2026-08/

CLAUDE.md

@@ -1,89 +1,357 @@
-# CLAUDE.md
+# ForexBot2 — Codebase Reference
 
-This file provides guidance to Claude Code (claude.ai/code) when working with code in this repository.
+Signal-intelligence platform for **XAU/USD (gold — live, validated)**. It fetches OHLCV from Twelve Data, scores 9 technical indicators per timeframe with regime-adaptive weights, checks multi-timeframe agreement and news/calendar risk, grades the signal (A+…F), and generates a full trade setup (entry, SL, TP1/TP2, trailing plan). Delivery: Telegram bot (auto-scanner + subscriber channel) and a CLI. A realism-focused backtester replays history through the *exact same* signal code.
 
-## What this is
+This file is the authoritative developer/agent reference. The human-facing docs live in `docs/` (`user_guide.html` + `beginner_guide.html`, and their PDF builds) — keep them in sync when signal behavior changes. `backtest/instructions_for_backtest.txt` lists what every backtest report must record.
 
-Signal-intelligence platform for **XAU/USD (gold — the only supported pair)**. It fetches OHLCV from the active data provider (**Oanda v20 since 2026-07-28**; Twelve Data and Tiingo remain registered fallbacks), scores 9 technical indicators per timeframe with regime-adaptive weights, checks multi-timeframe agreement and news risk, grades the signal (A+…F), and generates a full trade setup (entry, SL, TP1/TP2, trailing plan). Delivery: Telegram bot, CLI, and an Oanda practice-account auto-execution daemon. A realism-focused backtester replays history through the *exact same* signal code.
+**Read this whole file before changing signal logic.** Every numeric threshold below was either backtested or is load-bearing for a backtested result. Rules for changes are at the bottom ("Rules for making changes").
 
-Every numeric threshold in `config.py` and `pair_signals/xauusd.py` was backtested or is load-bearing for a backtested result. Validation evidence lives in the factory/module docstrings and in `~/.claude/projects/.../memory/`.
+---
 
-## Commands
+## Repository map
 
-```bash
-python test_all.py                      # QA suite: 359 tests, 22 modules, no network (mocked)
-python main.py --pair XAUUSD --timeframe H1 --detail   # CLI signal report (--json for full dump)
-python telegram_bot.py                  # the bot (refuses to boot without credentials)
-python oanda_daemon.py                  # auto-execution daemon on Oanda practice account
-python dashboard_publisher.py --once    # web dashboard: one scan+publish cycle to $DASHBOARD_OUT_DIR
-                                        # (omit --once for the candle-close daemon; deploy/README-dashboard.md)
+| Path | Lines | Purpose |
+|------|-------|---------|
+| `config.py` | ~374 | All config dataclasses, `PAIR_INFO` registry, per-pair strategy configs. **`PlatformConfig` defaults ARE the tuned XAU config.** |
+| `data_provider.py` | ~347 | Twelve Data SDK wrapper: fetch, rate-limit (55 credits/min), in-memory cache, validation/cleaning, freshness checks |
+| `engine.py` | ~1173 | `TechnicalEngine`: computes all indicators on one DataFrame → `TechnicalAnalysis` (composite score, regime, key levels) |
+| `chart_patterns.py` | ~280 | Chart-pattern detectors for the XAU_CP archetype (pure functions, ATR-unit geometry, first-close-beyond triggers) |
+| `signal_aggregator.py` | ~1447 | `SignalAggregator`: MTF analysis, news integration, gates/penalties, quality grade, trade-setup generation → `SignalReport` |
+| `news_engine.py` | ~680 | Economic calendar (Forex Factory JSON + MyFXBook RSS fallback) + keyword RSS sentiment |
+| `telegram_bot.py` | ~1259 | Async Telegram bot: candle-boundary auto-scanner, WebSocket level-breach monitor, owner lock, subscriber channel |
+| `pair_signals/__init__.py` | 52 | `PAIR_PRODUCERS` registry + `produce_signal()` |
+| `pair_signals/xauusd.py` | 71 | XAU config factory + per-timeframe overrides (docstring = validation record) |
+| `main.py` | ~348 | CLI: `python main.py --pair XAUUSD --timeframe H1 --detail/--json` |
+| `test_all.py` | ~2100 | QA suite: **162 tests in 10 modules**, no live API needed. Must stay green. |
+| `backtest/gpu_backtest.py` | ~2231 | Bar-replay backtester (CuPy→NumPy fallback) using the real `SignalAggregator` |
+| `backtest/diagnose_filters.py` | 609 | Filter-diagnosis tooling |
+| `backtest/research/` | — | Calendar research infra (`ff_calendar_scraper.py`, `calendar_store.py`) + result CSVs |
+| `backtest/*.csv` | — | Verification trade books from past tuning campaigns (`xau_*.csv`, …). **Gitignored, local-only** — used for manual before/after comparison, not by tests |
+| `backtest/cache/` | — | Parquet OHLCV cache (raw bars) |
+| `test_ws_dual.py` | 108 | Standalone WebSocket dual-connection probe |
+| `logs/telegram.log` | — | Bot log (midnight rotation, 30 backups) |
 
-# Backtests
-python backtest/gpu_backtest.py --pair XAUUSD --days 365
-python backtest/gpu_backtest.py --pair XAUUSD --start 2025-01-01 --end 2025-06-01 \
-    --primary-tf 15min --confirm-tfs 1h,4h
-# key flags: --spread --quality A+,A,B,C --strategies --workers --max-hold-bars
-#            --warmup-bars --cooldown-bars --no-cache --export-csv out.csv --detail
-#            --fixed-rr-tp N (research-only: hard TP at N×risk, trail off; verdict
-#            2026-08-05: NOT shipped — backtest/research/fixedtp_2026-08-05.md)
-```
+---
 
-**`test_all.py` is NOT pytest** — it uses a custom `@test("name")` decorator that runs each test immediately at import; don't grep for `def test_` or try `pytest test_all.py`. Run the whole file; it prints per-module results and exits non-zero on failure. There is no way to run a single test short of commenting modules out. Tests must stay green; every new gate/knob needs tests. When patching weekend logic in tests, patch `providers.base._is_forex_weekend` — but know there are THREE independent weekend implementations (`providers/base.py`, `signal_aggregator.py`, and an inline copy in `backtest/gpu_backtest.py:_is_xau_blocked_by_calendar`); patching the provider one does not reach the other two.
+## Pair status
 
-Python 3.12; a `venv/` exists in-repo. Dependencies in `requirements.txt` (numpy, pandas, requests, twelvedata, python-telegram-bot ≥20, websockets).
+- **XAU/USD**: the tuned, validated product — and the ONLY supported pair. All defaults in `PlatformConfig`/`XauStrategyConfig` are its tuned values.
+- **EUR/USD**: REMOVED from the codebase (2026-07-16, branch `xau-only-chart-patterns`). It never had a validated edge: the 2026-07-11/12 verdict found no configuration passing validation on M5, M15, or H1 at a 1-pip round-trip cost; root cause was cost-to-volatility (EUR M15 ATR ≈ 10 pips → 1-pip spread taxes ~10% of ATR; gold's ATR/spread ratio is several times larger). Calendar-surprise and H1-hybrid avenues were also tested and closed. The full EUR implementation and verdict history live in git history (last commit with EUR support: `7d3fd95`). Do not re-add EUR without new data (tick fills or H4/D1 horizons).
 
-## Configuration & credentials
+---
 
-Precedence for every credential/deployment setting: **env var → config.json → refuse to run** (no hardcoded fallbacks). A gitignored repo-root `.env` is loaded at `app_config` import via `os.environ.setdefault`. `config.json` (gitignored; schema in `config.json.example`) holds secrets plus deployment settings: `data_providers.active` (the live-feed switch), `data_providers.streaming` (optional tick-stream provider for the level-breach monitor), `scanning.pairs`/`timeframes`, per-pair `blocked_hours_utc`, `log_level`. Key env vars: `TELEGRAM_TOKEN`, `TELEGRAM_CHAT_ID`, `TELEGRAM_CHANNEL_ID`, `TWELVE_DATA_API_KEY`, `TIINGO_API_KEY`, `OANDA_ACCOUNT_ID`/`OANDA_API_TOKEN`/`OANDA_BASE_URL`, `DEEPSEEK_API_KEY` (news LLM; the `deepseek` config.json block holds model/enabled/timeout/daily cap).
-
-## Architecture
-
-### End-to-end pipeline (every scan)
+## End-to-end pipeline (what happens on every scan)
 
 ```
-providers/<active>.fetch_ohlcv()       per TF: single-flight fetch, drop forming candle, validate/clean, cache
-                                       (only Twelve Data rate-throttles; Oanda has no client throttle)
-  → engine.TechnicalEngine.analyze()   per TF: 9 scored indicators + ATR context → regime → composite score
+Twelve Data REST (+ WebSocket ticks in the bot)
+  → data_provider.fetch_ohlcv()        per timeframe: throttle, fetch, drop forming candle, validate/clean, cache
+  → engine.TechnicalEngine.analyze()   per timeframe: 9 scored indicators + ATR context → regime → composite score
   → signal_aggregator.analyze_pair()   live entry point (weekend/day filters, fetch, staleness)
       └─ analyze_from_dataframes()     replayable core: MTF agreement → news → bonuses/penalties →
-                                       final signal + quality grade → gates → trade setup → reasons
-  → SignalReport                       consumed by telegram_bot.py, main.py, oanda_daemon.py, backtester
+                                       final signal + quality grade → gates (session, SELL, M5, regime) →
+                                       trade setup (3 generators) → reasons
+  → SignalReport                       consumed by telegram_bot.py, main.py, backtest/gpu_backtest.py
 ```
 
-Signal currency everywhere is `SignalStrength` (−2..+2). DataFrames are UTC tz-aware, oldest-first, capitalized OHLCV columns. Everything runs on **closed bars only** — the forming candle is dropped, and session/hour filters evaluate **bar-CLOSE time** (bar label + candle duration) identically in live and backtest.
+Signal currency everywhere is the `SignalStrength` enum: `STRONG_BUY=2, BUY=1, NEUTRAL=0, SELL=-1, STRONG_SELL=-2`.
 
-### Layers
+---
 
-- **`providers/`** — one class per provider subclassing `BaseDataProvider` (`providers/base.py`), which owns everything the pipeline depends on: TTL+boundary cache, forming-candle drop, freshness check, validate/clean, paginated `fetch_range` (feeds the backtester from any provider), optional tick streaming. Subclasses implement only `_fetch_raw` + symbol/interval maps. Registry + `make_data_provider(config, name=None)` in `providers/__init__.py`; config.json `data_providers.active` selects the live candle feed and the optional `data_providers.streaming` scalar selects a different provider for the tick stream. A test enforces that subclasses don't override shared base methods (`supports_streaming`/`iter_ticks` are the sanctioned streaming override points). Tiingo streams mid-price ticks (`wss://api.tiingo.com/fx`) — do not re-add spread display from its free FX feed (two-tier indicative quotes, removed 2026-07-23); the no-spread rule extends to the WS stream.
-- **`engine.py`** — `TechnicalEngine`: 9 scored indicators (ema_trend, rsi, macd, bollinger, adx, roc, keltner, divergence, candlestick) + ATR as context. Regime classification (trending_up/down, `trending` = mixed/direction-UNKNOWN, transitioning, ranging) drives per-indicator weight multipliers and regime-aware RSI reinterpretation. Compute order matters (ATR first; MACD always computed even when not a pillar — divergence needs the histogram). Private series ride on the result as `_underscore` attrs; scorers/formatters skip anything `_`-prefixed or weight-0.
-- **`signal_aggregator.py`** — MTF agreement, diminishing-decay penalty system, exhaustion hard block, quality grading, then an ordered chain of post-grade gates (session, XAU SELL quality/vol/hour, regime blocks, M5 noise gates), then three setup generators: XAU ATR (A+/A), generic swing-based, plus the M5 pullback (`XAU_PB`) and chart-pattern (`XAU_CP`, `chart_patterns.py`) entry paths. The PB/CP generators run AFTER the gate chain and force quality to "B"; the CP path deliberately accepts grade-F composites and re-checks only session/day/big-candle (via warning prefixes) plus its own SELL/vol/RSI gates — it does NOT re-check exhaustion or the M5 min-score/MTF gates. **This is validated behavior, not a bug** (2026-08-04 audit: 226/586 trades in the shipped M5 12mo book are XAU_CP, +38.9R, 104 of them exhaustion-flagged and net-positive) — changing it is a rule-1 revalidation event. `TradeSetup.strategy` selects trailing params; trail params are stamped on the setup from the *producing* config so per-TF overrides reach formatters.
-- **`pair_signals/`** — per-pair config factories. **Always use the factory (`produce_signal` / `make_config`), never bare `PlatformConfig()`** — the factory applies the config.json overlay first, then the validated per-TF deltas (M5 hour-9 block, M15 trail arm 0.70, M15 SELL-hour block, etc.; evidence in the docstring). A test enforces factory ≡ inline-config equivalence. Confirmation map: M5→[M15,H1], M15→[M5,H1], H1→[M15,H4], H4→[H1,D1].
-- **`news_engine.py`** — Forex Factory calendar (MyFXBook RSS fallback) + RSS keyword sentiment with gold-inverse keywords. Class-level caches (fresh engine per scan). Optional DeepSeek enrichment via `llm_client.py` (requests-only, no SDK): batched headline sentiment overriding keyword scores, gold-direction on release surprises, and a market-summary blurb (`SignalReport.news_summary`) — **display/warning-only, keyword fallback on any failure, inert without a key**; prompts are static module constants (DeepSeek prefix-cache pricing) and requests send `thinking: disabled` (V4-Flash reasoning otherwise blows the timeout). By XAU-tuned default, event risk and sentiment **warn but do not gate signals** (`event_risk_caps_signals=False`, `sentiment_affects_signals=False`) — the LLM does not change that.
-- **`telegram_bot.py`** — owner-locked bot + subscriber channel; candle-boundary auto-scan loops (M5/M15/H1/H4, boundary+10 s), alert quality threshold + fingerprint dedup (committed only after a successful send), 📰 LLM market-context line on alerts when available, WebSocket level-breach monitor fed by the streaming provider (`data_providers.streaming`, falling back to the active provider; auto-disabled when the resolved provider can't stream or has no API key — Oanda does not stream, Tiingo does). Rotating log: `logs/telegram.log`, 3-h rotation × 240 backups.
-- **`oanda_daemon.py` / `oanda_executor.py`** — practice-account auto-execution. Book-parity rules (see daemon docstring): no broker-side TP (trailing model — TP1/TP2 advisory), 5-bar per-TF cooldown (wall-clock approximation, so it expires over weekends where the backtester would still count bars), trail ratchets at bar close once per *observed* bar (a missed/failed cycle skips those bars entirely — the intra-gap high/low never enters the peak). Accepted divergence: one position total; orders send `positionFill=OPEN_ONLY` so a state/broker divergence cancels the order instead of silently netting. State/broker re-sync runs at boot and after any suspicious fill (`_reconcile_positions`).
-- **`dashboard_publisher.py` / `static/dashboard.html`** — web dashboard as a **static publisher** (live LAMP server): standalone daemon scans at candle closes via `produce_signal` and atomically writes `snapshot.json`/`news.json`/`dashboard.html` to `$DASHBOARD_OUT_DIR` (Apache serves them; page is public by owner decision). **Hard independence rule: never imports, and is never imported by, `telegram_bot`/`oanda_daemon`** — confirmation map and boundary math are deliberate local copies, so a web-side failure can only mean a stale page. Due-TF classification uses the RAW wake time (flooring wake−buffer skips every H1/H4 scan — 2026-08-02 bug). News panel reads `NewsEngine.get_fundamental_context()` plus the additive `get_week_events()` accessor (display-only; no signal path calls it); as a separate process it has its own news caches and DeepSeek daily cap. News/calendar rows are clickable (detail dialog); headlines carry `id` = sha1(headline) in news.json, and exactly one on-demand endpoint exists: `static/news_summary.php` (installed by `install_static`), which serves a DeepSeek gold-impact summary per headline — whitelist-gated against news.json ids, cache-forever in `ai/`, own fail-closed daily cap (150/day, file-based, separate from the publisher's), key at `/etc/gold-dashboard/deepseek.key`, never reads repo config. Deployment checklist: `deploy/README-dashboard.md`.
-- **`backtest/gpu_backtest.py`** — bar-by-bar replay through the real `analyze_from_dataframes` (news off). Realism model (do not regress): next-bar-open entries re-anchored to the raw price path, round-trip `--spread` charged per trade, trade resolution on RAW bars (cleaning applies only to signal-side data), same-bar SL+TP tie → SL first, gap fills at bar open, trail ratchet at bar close only. Indicators are precomputed once per TF (CuPy GPU or NumPy fallback — check the `Backend:` log line); replay fans out over worker processes and the 5-bar cooldown is applied in an ordered pass afterwards, so parallel and sequential books are identical. Parquet cache in `backtest/cache/` keyed by provider/symbol/TF/window. `backtest/*.csv` are gitignored pinned trade books from past campaigns — used for manual before/after comparison. `backtest/instructions_for_backtest.txt` lists what every backtest report must record.
+## config.py — every knob
 
-## Rules for making changes (read before editing signal logic)
+### PAIR_INFO (market constants; key is the canonical symbol `"XAU/USD"`)
+| Field | XAU/USD | Meaning |
+|---|---|---|
+| `pip_value` | 0.10 | price units per pip |
+| `price_precision` | 2 | decimals for SL/TP rounding & display |
+| `default_spread` | 0.30 | round-trip cost charged by the backtester when `--spread` not given |
 
-1. **Never change a tuned value without re-validation.** Acceptance rule: positive total R in BOTH train and validation halves of a ≥12-month window, PF ≥ 1.3, at realistic cost (XAU $0.30 round trip), trailing exits, next-open fills. Marginal edges must additionally pass the 2024-06→2025-03 out-of-sample window — within-year splits alone have approved configs that later failed OOS.
-2. **Per-timeframe overrides live in the pair factory**, never in global defaults — the same knob has opposite effects on different TFs (proven repeatedly: `tp_floor_rr`, hour-9 block, trail params). Document validation evidence in the factory docstring.
-3. **Pair isolation is a hard invariant.** XAU gates in the aggregator are display-name-gated so a future pair port cannot inherit gold-calibrated logic by accident.
-4. **Live == backtest parity.** If you add a live gate, add it to the replay path in the same commit; all time filters evaluate bar-close time in both paths.
-5. **ROC stays weighted** even though it's correlated with MACD/Keltner/EMA — zeroing it backtested 13R worse; composite thresholds are calibrated with it.
-6. **Backtester realism knobs are results-critical** (raw-path resolution, spread charging, next-open entry, SL-first ties, close-only trail). Touching them invalidates comparability with all pinned books.
-7. **The forming candle must stay dropped** — live signals repaint otherwise.
-8. Don't add config fields nothing reads; don't disable indicators in production paths (`disabled_indicators` is for experiments only).
-9. EUR/USD was removed 2026-07-16 after a no-edge verdict (cost-to-volatility; last commit with EUR support: `7d3fd95`). Do not re-add without new data. BTC/ETH work lives on unmerged branches only.
+### TechnicalConfig (indicator parameters)
+EMA 9/21/50/200 (`ema_mode`: `"4ema"` default or `"2ema"` = 50/200 only); RSI 14 (OB 70 / OS 30 / neutral zone 40–60; `rsi_mode`: `"regime_aware"` default or `"classic"`); MACD 12/26/9 (`macd_as_pillar=True` — MACD is a scored indicator, not just divergence input); BB 20/2.0σ; ATR 14; ADX 14 (strong ≥25, very strong ≥50); divergence lookback 40; swing-S/R lookback 50; ROC period 10, threshold 1.0%; Keltner: EMA 20 mid ± 1.5 × ATR(10); ADX slope lookback 5, "emerging" threshold 3.0; `disabled_indicators` list (removes the pillar from scoring; for MACD the histogram is still computed — divergence needs it); `regime_weights_override` dict (replaces the gold regime-multiplier table for experiments).
 
-## 2026-08-04 QA audit — owner decisions ledger
+### SignalConfig (scoring & trade geometry — XAU-calibrated)
+| Field | Default | Meaning |
+|---|---|---|
+| `mtf_agreement_threshold` | 2 | timeframes (of primary + confirmations) that must agree |
+| `min_rr_ratio` | 0.5 | reject setups below this RR |
+| `tp_tightening_factor` | 1.0 | TP scale factor. **Leave at 1.0**: 0.8 was re-tested 2026-07-08 with spread+gap fills and nearly halved total R |
+| `round_trip_cost` | 0.30 | XAU $ round trip |
+| `min_tp_cost_multiple` | 30.0 | spread gate: TP1 distance must be ≥ cost × this (= $9 on XAU). Validated: sub-$10 targets lose |
+| `min_sl_pips` | 80.0 | SL floor in pips ($8 on XAU) |
+| `tp_floor_rr` | 0.5 | TP1 floor as fraction of SL distance (M15 factory overrides to 1.0) |
+| `trail_arm_r` / `trail_dist_r` | 0.6 / 0.75 | trailing exit: once favorable excursion ≥ arm×risk, stop trails dist×risk behind best price (bar-close updates). TP1 becomes advisory. Validated on 12mo M5: +105R vs +48R static. 0 disables |
+| `pb_trail_arm_r` / `pb_trail_dist_r` | 0.8 / 0.75 | trailing params for XAU_PB pullback trades (tighter stops → arm later) |
+| `weights`, `min_confidence`, `cooldown_bars`, `default_risk_pct`, `default_rr_ratio`, `atr_sl_multiplier` | — | **LEGACY / UNUSED** outside tests. The engine has its own weights; backtest cooldown lives in `BacktestConfig` |
 
-- **Weekend calendar: FIXED (owner-approved 2026-08-04).** All three `_is_forex_weekend` copies are NY-anchored (Fri 17:00 → Sun 18:00 ET; during US DST that is Fri 21:00 UTC — measured on 5yr of Oanda bars). Kills the weekly false-"STALE DATA/feed lagging" hours. Book impact, fully accounted: M15 byte-identical; M5 12mo dropped exactly ONE trade — a phantom whose signal fired on the final summer-Friday bar (close 21:00 UTC) and "filled" at Sunday's reopen 49 h later (−1.08R, live-impossible). M5 12mo reference at HEAD: 555 trades, +98.64R PF≈1.50 @ $0.65 (was 556/+97.56/1.48). A Module 21 test sweeps all three copies hourly across both DST transitions.
-- **Spread in exit-trigger detection (rule 6): owner DECLINED 2026-08-04.** Backtest stop triggers stay on raw mid High/Low; the measured optimism (~4–6% of total R: −3.55R @ $0.30, −4.24R @ $0.65 on the M15 12mo book) is accepted and books stay comparable. Re-open only as a conscious re-pin campaign.
-- **`FB2_EMA_LO`: HARDENED 2026-08-04 (delegated).** Research override for the EMA200-proximity band, default = tuned 0.5; an unparseable value now degrades to the default with a warning instead of crashing every scan. Documented here — it is a knob that exists.
-- **H4 seeding divergence: parked.** Live H4 fetches only 380 bars vs the backtest's unbounded history — ~2% of sampled H4 bars flip an indicator signal (composite Δ up to 12.3). M5/M15/H1 clean at their real fetch depths. Fold into any future H4 campaign; H4 remains "unproven for auto-execution".
+### NewsConfig
+Tracked currencies `["USD","XAU"]`; pre-event buffer 30 min, post-event 15 min (45 min after a *surprise* print); `event_risk_caps_signals=False` — **XAU default: event risk warns but does NOT cap confidence** (2026-07-11 audit: the caps suppressed trades worth +4.6..+7.2R/yr; XAU trades near USD releases outperform). Context cache TTL 600 s (must stay < pre-event buffer). 4 RSS feeds (investinglive, FXStreet, Google-News gold query, CNBC economy).
 
-## Current state notes (2026-07-30)
+### XauStrategyConfig (all XAU gates; `enabled=True`)
+| Field | Default | Notes |
+|---|---|---|
+| `blocked_hours_utc` | [0,1,2,10,11,13,16,21] | bar-close hours never traded. M5 factory appends 9 (post-London-open whipsaw, −28.9R/143 over 6 yr) |
+| `allowed_weekdays` | [0,1,2,3,4,6] | Mon–Fri + Sun (Sydney open). `weekend_filter=True`: Fri 22:00 → Sun 22:00 UTC closed |
+| `min_atr_threshold` | 1.0 | $1 ATR floor for swing-based setups |
+| `v2_sl_atr_multiplier` | 2.0 | SL ATR floor in the generic setup generator |
+| `sl_atr_multiplier` / `tp_atr_multiplier` | 2.0 / 1.0 | used by the XAU ATR setup generator (A+/A only) |
+| `sell_min_quality` | ["A+","A","B"] | SELLs below B are blocked |
+| `sell_vol_regimes` | ["high","extreme"] | SELLs only in these volatility regimes (quiet shorts fade vs gold's macro uptrend) |
+| `blocked_regimes` / `sell_blocked_regimes` | [] / [] | per-TF factory opt-ins. M15 factory sets `blocked_regimes=["trending"]` (mixed regime: strong ADX, EMAs unaligned — lost −6.8R/24) |
+| `sell_blocked_hours_utc` | [] | bar-CLOSE UTC hours where SELLs are blocked (BUYs unaffected). Per-TF factory opt-in: M15 sets [15, 17] (NY-session shorts, −33.5R/165 over 5 yr, negative 6/6 years both halves; BUYs there +34.3R) |
+| M5 gates | | `m5_require_mtf=True`; `m5_min_abs_score=30`; `m5_min_confirmation_score=10`; `m5_sell_max_quality=["B","C"]` (A-grade M5 SELLs enter at dip extremes and lose); `m5_blocked_vol_regimes=["high"]` (chop transition — 61% WR, negative total); `m5_sell_min_rsi=25` |
+| M5 pullback (XAU_PB) | | `m5_pullback_enabled=True`; min score 15 (lower than gate 1 BY DESIGN — mid-pullback scores weaker); ADX ≥ 20; RSI zones BUY (35,62) / SELL (38,65); retrace depth 0.8–6.0 ATR from the 40-bar swing extreme; band touch within 3 bars. Validated +38.3R/254 trades on 12mo |
+| Chart patterns (XAU_CP) | `cp_enabled=False` | `cp_*` fields: pattern list, lookback 72, swing window 3, recency 20, double tolerance 0.5 ATR, height 1.5–8 ATR, H&S head ≥0.5 ATR / shoulder tol 1.0 ATR, flag pole ≥3 ATR in ≤12 bars, flag 3–12 bars retracing 10–50%. Textbook defaults, NOT tuned. **Default OFF** — M5 factory enables BUY-only (`inv_hs`, `double_bottom`, `bull_flag`); M15/H1/H4 remain disabled (all failed OOS validation 2026-07-16). Bearish patterns (`hs_top`, `double_top`, `bear_flag`) dropped: SELL CP lost −3.21R on OOS, −4.52R on M15 5yr |
 
-- Oanda v20 is the live feed (`data_providers.active: "oanda"`), but the pinned backtest books are still Twelve-Data-pinned and the backtest cost assumption is $0.30 while measured real spreads are ~$0.65–0.72 — re-pin books on Oanda bars before treating feed-switch comparisons as valid (see Oanda/TD divergence study in memory).
-- Human-facing docs in `docs/` (`user_guide.html`, `beginner_guide.html` + PDF builds) must be kept in sync when signal behavior changes.
+Legacy fields kept for config-shape compatibility: `rsi_entry_threshold`, `min_ema200_tfs`, `allow_generic_trades`, `buy_threshold_boost`, `m5_confirmation_bars`, `h4_rsi_gate=False`, `h1_macd_gate=False`, `rsi_zone` — not wired to current gates.
+
+### FilterConfig (`config.backtest` — despite the name it applies LIVE too)
+`session_filter=True`, window 03:00–23:00 UTC, `blocked_hours_utc` mirrors XAU's list (non-XAU pairs read this copy), `allowed_weekdays` Mon–Fri+Sun. `regime_filter=False` (optional H4-ADX≥25 gate, off).
+
+### PlatformConfig (top level)
+`symbol="XAU/USD"`, primary TF H1, confirmations [M15, H4], `data_lookback_days=60`, `scan_pairs=[XAUUSD]` (**the single switch for what the bot scans**), `scan_timeframes=[M5,M15,H1,H4]`, `candle_buffer_seconds=10`, `ws_cooldown_seconds=300`, hardcoded `twelve_data_api_key` fallback.
+
+---
+
+## engine.py — TechnicalEngine
+
+**Input contract:** pandas DataFrame, UTC tz-aware DatetimeIndex, oldest-first, columns `Open High Low Close Volume` (capitalized). `< 30` bars → empty NEUTRAL result. EMA200 needs ≥ 200 bars (otherwise EMA-trend indicator degrades gracefully).
+
+**Compute order matters:** ATR first (EMA-ribbon and candlestick proximity use it) → EMA → RSI → BB → ADX → MACD (always computed, even if not a pillar — divergence needs the histogram) → ROC → Keltner (needs BB levels + MACD histogram) → divergence → structure S/R → candlesticks → regime classification → regime-aware RSI reinterpretation → composite.
+
+Results land in `TechnicalAnalysis.indicators: Dict[str, IndicatorSignal]` (fields: name, value, signal, confidence 0-100, description, weight). Private series ride on the result object as underscore attrs (`_rsi_series`, `_macd_histogram`, `_adx_slope`, `_kc_squeeze`, `_kc_squeeze_released`) and are skipped by the composite scorer and all formatters (skip rule: name starts with `_` or `weight == 0`).
+
+### The 9 scored indicators (+ ATR as context)
+
+| Key | Base weight | Signal logic (exact) |
+|---|---|---|
+| `ema_trend` | 1.0 | 4-EMA mode: alignment score over checks (price>EMA9, 9>21, 21>50, 50>200), normalized to [-1,1]. >0.6 STRONG_BUY (conf ≤90), >0.2 BUY, mirrored sells, else NEUTRAL (40). Fast/medium cross ±10 conf. Ribbon (9/21/50 spread < 1 ATR) = "COMPRESSED"; compressed→expanded = +10 conf. Emits `ema_fast/medium/slow/trend` key levels |
+| `rsi` | 1.5 | Wilder RSI (ewm α=1/14). Classic pass first: ≥70 SELL, ≤30 BUY, ≥60 BUY-momentum (45), ≤40 SELL-momentum (45), else NEUTRAL (30). **Then overwritten by regime reinterpretation** (below) when `rsi_mode="regime_aware"` |
+| `macd` | 1.5 | Histogram = MACD line − signal line. Cross above 0 → BUY (conf 55+); accelerating (hist>prev>0) → STRONG_BUY; mirrored down; positive-but-fading / negative-but-fading → NEUTRAL (35); flat → NEUTRAL (30). Confidence scaled by |hist| vs its 20-bar mean |
+| `bollinger` | 1.5 | Position = (close−lower)/(upper−lower). >1.0 SELL (65), <0.0 BUY (65), >0.8 SELL (45), <0.2 BUY (45), else NEUTRAL (30). Squeeze = width < 0.75 × 20-bar avg width → +10 conf. Emits `bb_upper/lower/mid` |
+| `adx` | 0.5 | Wilder ADX/DI (ties zero BOTH DMs — fixes a systematic bearish nudge). ADX ≥ 25 → BUY/SELL by DI dominance, else NEUTRAL. Conf 40/65/80 by ADX band. ADX slope (5-bar diff): <25 with slope > 3.0 → "TREND EMERGING" +15 conf; also strengthening/weakening adjustments. Sets `_adx_slope` |
+| `roc` | 1.5 | 10-period %change. Positive & rising → BUY (STRONG if >2× threshold=2%); negative & falling → SELL; zero-cross → BUY/SELL (65); else NEUTRAL. **Directionally redundant with MACD/Keltner/EMA (0.8–0.94 corr) but its weight is load-bearing: zeroing it backtested 13R WORSE — composite thresholds are calibrated with it. Do not remove without retuning thresholds** |
+| `keltner` | 1.5 | EMA20 mid ± 1.5×ATR(10). TTM squeeze = BB fully inside KC. Squeeze released (was→not) + MACD hist sign → STRONG_BUY/STRONG_SELL (80); squeeze active → NEUTRAL (60); close above/below channel → BUY/SELL continuation (65); inside → NEUTRAL (30). Emits `kc_upper/lower/mid` |
+| `divergence` | 1.5 | RSI + MACD-hist vs price over 40 bars. Swing points (window 3) must align within ±2 bars to pair up; last two pairs: price LL + indicator HL = bullish, price HH + indicator LH = bearish. Any found → BUY/SELL/NEUTRAL-if-both, conf 70, value = count |
+| `candlestick` | 1.0 | Last bar: Doji (body<10% range), Hammer/Hanging Man (lower wick>2×body), Shooting Star/Inverted Hammer, Engulfing (vs prev bar), Marubozu (body>85%). Majority vote → signal, conf 50; +20 if pattern sits within 0.5 ATR of a swing S/R level |
+| `atr` | **0.0** (context) | Wilder ATR. Volatility regime from percentile vs last 50 ATR values: >80 extreme, >60 high, >30 normal, else low. Emits `atr`, `atr_pct` |
+
+Structure S/R (not scored): swing highs/lows (window 3) over `swing_sr_lookback` (min 120 bars on 1min/5min TFs); up to 3 above and 3 below stored as `swing_high_1..3` / `swing_low_1..3` key levels.
+
+### Market regime classification (exact rules)
+- ADX ≥ 25 **and** EMA-trend signal bullish → `trending_up`; bearish → `trending_down`; EMA mixed → **`trending`** (the "mixed" regime — momentum without directional structure; treated as direction-UNKNOWN everywhere; a past bullish-bias bug here cost −6.8R and is fixed)
+- else ADX slope > 3.0 → `transitioning` (trend birth, ~5–10 bars before ADX crosses 25)
+- else → `ranging`
+
+### Regime-aware RSI reinterpretation (`rsi_mode="regime_aware"`, the default — classic mode backtested worse for gold)
+| Regime | RSI zones |
+|---|---|
+| trending_up | 38–55 STRONG_BUY 75 ("pullback zone, ideal entry"); 30–38 BUY 60; 55–65 BUY 50; >65 NEUTRAL ("extended, wait"); <30 NEUTRAL ("possible reversal") |
+| trending_down | mirrored: 45–62 STRONG_SELL 75; 62–70 SELL 60; 35–45 SELL 50; <35 / >70 NEUTRAL |
+| transitioning / trending (mixed) | ONLY extremes: ≥70 SELL 50, ≤30 BUY 50, else NEUTRAL 30 |
+| ranging | ≥70 SELL 70, ≤30 BUY 70, ≥60 SELL 45, ≤40 BUY 45, else NEUTRAL |
+
+### Composite score
+For each scored indicator: `effective_weight = base_weight × regime_multiplier`; contribution = `signal.value × effective_weight × confidence/100`. Composite = `(Σ contributions / Σ effective_weights) × 50`, clamped to ±100. Confidence = weight-averaged indicator confidence. Composite signal: ≥40 STRONG_BUY, ≥15 BUY, ≤−40 STRONG_SELL, ≤−15 SELL, else NEUTRAL.
+
+**Gold regime multipliers** (defaults in `_compute_composite`; `technical.regime_weights_override` replaces the whole dict):
+
+| Indicator | trending_up/down/trending | ranging | transitioning |
+|---|---|---|---|
+| ema_trend | 2.0 | 0.3 | 1.5 |
+| rsi | 1.0 | 1.5 | 1.0 |
+| macd | 2.0 | 1.5 | 2.5 |
+| bollinger | 0.5 | 1.5 | 1.0 |
+| keltner | 1.5 | 2.0 | 2.5 |
+| roc | 1.5 | 0.5 | 2.0 |
+| candlestick | 0.8 | 1.0 | 1.0 |
+| divergence | 1.0 | 2.5 | 2.0 |
+| adx | 0.8 | 0.5 | 1.0 |
+
+---
+
+## signal_aggregator.py — SignalAggregator
+
+Two entry points:
+- **`analyze_pair(symbol)`** — LIVE: applies XAU weekend + allowed-weekday filters on wall clock, fetches all TFs (primary + confirmations), flags stale data (primary stale → signal suppressed to NEUTRAL/F), delegates to `analyze_from_dataframes`, then applies the live session check on the **bar-CLOSE time** (bar timestamp + candle duration): outside 03:00–23:00 or in blocked hours → NO TRADE. This is the exact predicate the backtester replays, so live == backtest.
+- **`analyze_from_dataframes(symbol, dfs, context, use_news)`** — replayable core; treats each df as the visible history at decision time (caller must slice; no weekend filter here — the backtester applies it).
+
+### Pipeline inside `analyze_from_dataframes` (numbered as in the source)
+1. Primary TF analysis → seeds report (score, confidence, regime, vol regime, price, timestamp = last bar's index label).
+2. Confirmation TF analyses → `mtf_signals` / `mtf_scores` keyed by TF string ("5min", "1h", …).
+3. **MTF agreement**: count BUY-ish vs SELL-ish across ALL TFs (primary included); agreement = count ≥ 2 **in the primary's direction** (two bullish confirms cannot "confirm" a bearish primary).
+4. News (if `use_news`): event risk (`clear`/`caution`/`danger`) + pair sentiment (−1..+1). Backtests pass `use_news=False`.
+5. Spread estimate = median(High−Low) of primary df.
+6. **`_compute_final_signal`** (bonuses → event cap → penalties → thresholds → grade):
+   - Bonuses: MTF agreement → +10 conf and `score = 0.6×primary + 0.4×mean(all TF scores)`; news sentiment aligned (|sent|>0.2) → +5; in a *directional* trend, RSI in pullback zone → +5, BB position favorable → +5 (mixed `trending` regime gets NO directional bonus).
+   - Event risk `danger`: warning always; confidence hard-capped at 30 ONLY if `news.event_risk_caps_signals` (False by default — XAU-tuned).
+   - **Penalty system (diminishing, replaces old additive stacking):** collect, sort by size desc, apply with decay ×1.0, ×0.5, ×0.25…, total capped at 30. Penalties: `mtf_disagreement` 15; `news_conflict` 10; `event_caution` 15 (config-gated); `counter_trend` 15; `extreme_vol` 10; `ema200_near` 5 (price within 0.5% of EMA200 — env `FB2_EMA_LO` overrides); `ema200_far` 5 (>2.5%); `exhaustion` 8 (price >3.5 ATR beyond nearest swing); `no_pullback` 8 (>2.5 ATR from EMA21 in trend) / `weak_pullback` 5 (>2.0); `rsi_extended` 5 (RSI>65 with BUY / <35 with SELL in any strong-ADX regime, both directions in mixed `trending`); `bb_overextended` 8 (BB pos >0.85 BUY / <0.15 SELL).
+   - **Exhaustion hard block**: ≥2 exhaustion-tagged penalties ({exhaustion, no_pullback, rsi_extended, bb_overextended} → `report.exhaustion_count`) AND momentum fading (ROC sign against score, or MACD signal against score) → confidence = 0.
+   - Confidence clamped [0, 95]. **Final signal**: score ≥35 & conf ≥55 STRONG_BUY; ≥25 & ≥50 BUY; mirrored sells; else NEUTRAL.
+   - **Quality grade**: |score|≥40 & conf≥65 & MTF → **A+**; ≥33 & ≥55 & MTF → **A**; ≥28 & ≥50 → **B**; ≥25 & ≥45 → **C**; ≥15 → **D**; else **F**.
+7. Post-grade gates, in order:
+   - **6b session filter** (bar-label hour + weekday; XAU reads `xau_strategy.blocked_hours_utc`/`allowed_weekdays`, others read `config.backtest.*`) → NEUTRAL/F.
+   - **6c transitioning boost**: regime `transitioning` and grade B/C/D → A if |score|≥30 & conf≥50 & MTF; else B if |score|≥25 & conf≥45 (early trend entries).
+   - **6d exhausted-trend cap**: trending regime + exhaustion_count ≥2 + grade C/D → D, setup killed.
+   - **6e XAU SELL quality gate**: sell-direction & grade actionable (A+..C) & not in `sell_min_quality` → D. The actionable guard is load-bearing: without it a session-blocked F report was *upgraded* to D, re-opening the M5 pullback path (its entry check is `!= "F"`).
+   - **6f MTF opposition**: ≥2 TFs pointing against the trade direction → D (never trade against consensus).
+   - **6g XAU SELL volatility gate**: sell-direction & vol regime not in {high, extreme} → D.
+   - **6g2 XAU regime gates**: regime in `blocked_regimes` (M15: mixed `trending`) → D; sell-direction & regime in `sell_blocked_regimes` → D.
+   - **6g3 XAU SELL hour gate**: sell-direction & bar-CLOSE hour (label + candle duration — the entry hour under next-open fills) in `sell_blocked_hours_utc` → D. Per-TF opt-in; M15 factory blocks 15/17 UTC.
+   - **6i XAU M5 noise gates** (`_apply_m5_filters`, primary TF = M5 only): (1) |score| ≥ 30; (2) MTF agreement required; (3) ≥1 confirmation TF with score ≥ ±10 in trade direction (NEUTRAL confirms don't count); (4) SELL grade must be B/C (A-grade M5 shorts lose −0.017R/trade); (5) vol regime `high` blocked; (6) SELL blocked when RSI < 25. Each failure → D, setup killed.
+8. **Trade setup**: XAU + grade A+/A → `_generate_xau_atr_trade_setup` first; any pair + grade A+/A/B/C and no setup yet → `_generate_trade_setup` (swing-based).
+9. Reasons: every non-neutral indicator description → `bullish_reasons`/`bearish_reasons` (the backtester tags strategy from reasons[0] markers).
+10. **XAU_PB pullback** (M5 XAU only, `m5_pullback_enabled`, no setup yet, grade ≠ F): `_try_m5_pullback` — see below.
+11. **XAU_CP chart pattern** (any TF, XAU only, `cp_enabled`, no setup yet, NOT session/day-blocked — weak-composite grade F is allowed BY DESIGN, reversals complete on weak composites): `_try_chart_pattern` — see below.
+
+### Trade setup generators (exact geometry)
+
+**A. `_generate_xau_atr_trade_setup`** (XAU, grade A+/A, ATR>0): TP quality scale A+ 2.0 / A 1.5 (A with exhaustion≥2 → 1.0). TP1 = entry ± `tp_atr_multiplier`(1.0)×scale×ATR; TP2 = 1.5×that; SL = entry ∓ 2.0×ATR. Gates: RR ≥ 0.5; TP1 distance ≥ $9 (spread gate). Prices rounded to 2 dp; pip 0.10 hardcoded — **this generator is XAU-only by construction**.
+
+**B. `_generate_trade_setup`** (swing-based, any pair, grade A+..C): XAU requires ATR ≥ $1 (LOW ATR FILTER). Per-side:
+- SL anchor = nearest swing low below price (BUY) / swing high above (SELL) ∓/± `sl_swing_buffer`×ATR (1.0); fallback (no swing) = (1+buffer)×ATR. SL distance floored at max(ATR × sl_atr_floor, min_sl_pips×pip) — XAU: max(2.0×ATR, $8).
+- TP1 = nearest swing beyond entry ± 0.3 ATR if it's within `tp_swing_cap` ATR (2.5) AND ≥ SL×RR-floor; else entry ± max(`tp_mult`×ATR, SL×RR-floor). tp_mult 1.5, RR-floor = max(min_rr 0.5, tp_floor_rr — 1.0 on M15).
+- TP2 = second swing if within 5 ATR, else entry ± max((tp_mult+1.5)×ATR, SL×floor×1.5); forced beyond TP1 by ≥1 ATR.
+- Gates: RR ≥ min_rr; spread gate TP1 distance ≥ cost×multiple ($9 on XAU).
+
+**C. `_try_m5_pullback` (XAU_PB)**: detection = EMA alignment ≥3 of 4 checks (9>21, 21>50, 50>200, close>200 — mirrored for SELL), ADX ≥ 20, price touched max(EMA21, KC-mid) within last 3 bars, resumption bar closed back in trend direction beyond EMA9, retrace depth 0.8–6.0 ATR from the 40-bar extreme, RSI in pullback zone (BUY 35–62 / SELL 38–65). Then gates: directional score ≥ 15; ≥1 confirmation TF ≥ ±10; not against MTF consensus; vol regime gates (same as breakout path). Geometry: SL = pullback 8-bar extreme ∓ 0.5 ATR (min $8); TP1 = 1.5×risk, TP2 = 2.5×risk; spread gate $9. Sets `strategy="XAU_PB"`, grade min B, marker `"XAU PULLBACK STRATEGY: …"` at reasons[0]. Rejection funnel counters in class attr `SignalAggregator.pb_funnel`.
+
+**D. `_try_chart_pattern` (XAU_CP)**: detection in `chart_patterns.py` — 6 classical patterns (double_bottom/double_top, inv_hs/hs_top, bull_flag/bear_flag), swing-point based (strict local extrema, window `cp_swing_window`=3), all thresholds in ATR units, scanned over the last `cp_lookback`=72 closed bars. Trigger = FIRST close beyond the neckline/boundary since the pattern formed (one breakout → one signal, no repaint); reversal patterns are invalidated by a new extreme beyond the pattern. No composite-score gate BY DESIGN. Gates: MTF non-opposition (6f mirror); SELL patterns respect `sell_vol_regimes`, `sell_blocked_hours_utc` (bar-close hour), M5 RSI floor; M5 blocks `m5_blocked_vol_regimes` both ways; ATR ≥ $1. Geometry: SL = pattern anchor ∓ 0.5 ATR (min $8); TP1 = measured move (pattern height), TP2 = 1.5×height; RR ≥ 0.5; spread gate $9. Sets `strategy="XAU_CP"` (global trailing params), grade min B, marker `"XAU CHART PATTERN: …"` at reasons[0]. Funnel in `SignalAggregator.cp_funnel`.
+
+`TradeSetup` fields: pair, direction, entry, SL, TP1, TP2, RR, risk/reward pips, pip_value, `strategy` ("GENERIC", "XAU_PB", or "XAU_CP" — selects trailing params; XAU_CP uses the global pair).
+
+---
+
+## news_engine.py — NewsEngine
+
+- **Calendar**: Forex Factory free JSON (`nfs.faireconomy.media/ff_calendar_thisweek.json`), 3 retries, honors 429 Retry-After with class-level backoff (≤600 s); fallback MyFXBook RSS. Never fabricates events — unavailable calendar = risk checks skipped that cycle.
+- **Caches are class-level** (a fresh NewsEngine is built per scan): calendar + headlines, TTL = 600 s on success, 60 s on empty result; lock held across fetch so concurrent scans share one request.
+- **Event risk** (`get_event_risk_level`): high-impact event for base/quote within 30 min → `danger`; within 60 min → `caution`; already-released high-impact → `caution` for 15 min after (45 min if actual ≠ forecast — "surprise"). `EconomicEvent.surprise` compares parsed numeric actual vs forecast (handles "%", "K/M/B" suffixes); direction is numeric-only (doesn't know if higher is bullish).
+- **Sentiment**: keyword sets over headlines from 4 RSS feeds, deduped by title (Google News " - Publisher" suffix stripped), last 24 h only. Per-currency scores; **`GOLD_INVERSE_KEYWORDS`** (rate hike/hawkish/tightening/rate cut/dovish/easing/stimulus) flip sign for XAU (hawkish = dollar-bullish = gold-bearish). Currency detection via compiled word-boundary-ish regexes (bare "us"/"eu"/"uk" deliberately excluded). Pair sentiment = base − quote.
+
+---
+
+## data_provider.py — ForexDataProvider
+
+- Twelve Data SDK (`TDClient.time_series(...).as_pandas()`), symbols normalized to `"XAU/USD"` form; renames lowercase columns → capitalized; sorts oldest-first; localizes UTC.
+- **Rate limit**: 55 credits/min sliding window (`_throttle`), blocks with sleep when exhausted.
+- **In-memory cache**: key `{symbol}_{tf}`, TTL = candle duration (M5 300 s … H4 14400 s); a cached frame is evicted and refetched if it fails the freshness check **or the boundary check** (`_has_latest_closed_bar`: the frame must contain the most recent fully closed bar, weekend-exempt — a wall-clock TTL alone would serve mid-candle snapshots at the next boundary and scans would run one bar behind). Cache and throttle are lock-protected (one provider is shared across the bot's thread pool).
+- **`_drop_forming_candle`**: Twelve Data includes the current partial candle; the scanner fires seconds after boundary, so the last bar is dropped if its close time is in the future. Without this every indicator would read a seconds-old stub and signals would repaint.
+- **Freshness** (`_is_data_fresh`): last bar older than 3× candle duration → stale (warns; live path suppresses signal if the PRIMARY TF is stale). Weekends exempt (Fri 22:00→Sun 22:00 UTC).
+- **`_validate_and_clean`**: drop all-NaN rows → ffill (limit 3) → drop NaN closes → sort + dedupe (keep last) → drop future timestamps (>now+5 min) → repair OHLC inconsistencies (High=max, Low=min of O/C/existing) → remove >10σ return outliers (spike bar only — the opposite-direction snap-back bar right after it is kept, else every spike punched a two-bar hole) → warn if < ~210 bars.
+- `outputsize = max(300, min(days×bars_per_day+20, 5000))`; TD caps at 5000/call (the backtester paginates by date).
+- `get_latest_price` / `get_spread_estimate` (median M5 High−Low) helpers.
+
+---
+
+## pair_signals/ — per-pair factories (ALWAYS use these, never bare PlatformConfig for a pair)
+
+`PAIR_PRODUCERS: Dict[symbol, factory]`; factory signature `make_config(primary_tf, confirmation_tfs=None) -> PlatformConfig`. `produce_signal(symbol, tf, data_provider)` builds config → aggregator → `analyze_pair`. Registered: XAU/USD.
+
+**A bare `PlatformConfig()` silently runs any pair with XAU defaults — always go through the factory** (the bot, CLI, and backtester all do).
+
+Default confirmation map (same in the factory, the bot, and backtest usage): M5→[M15,H1], M15→[M5,H1], H1→[M15,H4], H4→[H1,D1]. (Exception: `main.py`'s CLI map uses M5→[M1,M15] — a CLI-only quirk.)
+
+**xauusd.py**: `PlatformConfig()` defaults + per-TF overrides (each validated; full evidence in the module docstring):
+- M5: append hour 9 UTC to blocked hours (6-yr replay: +207.1R vs +178.3R; M15-and-up keep hour 9 — it's positive there). Enable BUY-only chart patterns (`cp_enabled=True`, `cp_patterns=["inv_hs","double_bottom","bull_flag"]`): 12mo +41.8R PF 1.60 (train/val both positive), 5yr +65.2R PF 1.48; bearish CP dropped (OOS −3.21R). M15/H1 CP remain disabled (all failed OOS).
+- M15: `signals.tp_floor_rr = 1.0` (1R minimum targets: +116.6R vs +105.4R, better in train/val and 2024 OOS; **hurts M5** — per-TF only) and `xau_strategy.blocked_regimes = ["trending"]` (mixed regime block: +65.7R vs +59.0R). A transitioning-SELL block was tested and REJECTED (bull-market artifact — failed 2024 OOS).
+- M15: `xau_strategy.sell_blocked_hours_utc = [15, 17]` (SELL-only hour block, validated 2026-07-16 on the 5-yr replay: +176.7R PF 1.31 vs +144.9R PF 1.22, better in all 6 years, both 12-mo halves, and 2024-06→2025-03 OOS; M5 cross-TF pocket confirms; BUYs at those hours are +34.3R so `blocked_hours_utc` would be wrong).
+- The factory must stay equivalent to inline `PlatformConfig(symbol=XAUUSD, …)` + overrides — **enforced by a test in test_all.py**.
+
+---
+
+## telegram_bot.py — the bot
+
+Run: `python telegram_bot.py` (env `TELEGRAM_TOKEN`, `TELEGRAM_CHAT_ID`, `TELEGRAM_CHANNEL_ID`; hardcoded fallbacks exist in the file).
+
+- **Owner lock**: every command/button is wrapped in `_owner_only` — only `TELEGRAM_CHAT_ID` gets responses; others get a refusal. The bot is the owner's control panel; the product is the channel.
+- **Subscriber channel** (`TELEGRAM_CHANNEL_ID`, `-100…`; `""` disables): auto-scan signal alerts + breach alerts also post there (bot needs channel-admin). Monetization = Telegram Stars subscription invite links; Telegram handles membership, the bot has zero payment code. The channel feed keeps running even when the owner toggles DM auto-scan off (`_feed_active()` = DM chats OR channel).
+- **Auto-scan**: boot-starts if `TELEGRAM_CHAT_ID` set (initial M15 scan populates the level cache first). Candle-boundary loops for M5/M15/H1/H4, firing at boundary + 10 s. Each scan runs all `scan_pairs` concurrently, updates the level cache, then broadcasts.
+- **Alert gating**: quality threshold (`_alert_min_quality`, default "C", menu cycles C→B→A) + fingerprint dedup per (symbol, TF) on (signal, quality, direction, entry bucket) with 1-hour re-arm — a persisting identical signal doesn't re-send every candle, but a setup that moved ≥ half its risk distance (new pullback/swing anchor) re-alerts. The fingerprint is committed only **after** a message reaches ≥1 target (`_commit_alert`), so a failed Telegram send doesn't consume the alert.
+- **WebSocket monitor**: subscribes every `scan_pairs` symbol to TD quotes WS; 10 s heartbeat; exponential reconnect (≤60 s). On tick crossing any cached key level (`swing_high/low_1..3`, `ema_trend`, `bb_upper/lower`, `kc_upper/lower`; cache TTL 1 h) → immediate M15 rescan → "LEVEL BREAK" alert if actionable+changed (same gates as auto-scan), 5-min per-pair cooldown.
+- **Commands**: `/start` (menu), `/scan` (buttons: per pair — M5/M15/H1/H4 + Scan All TFs), `/autoscan` (toggle DM copy), `/status` (loops, WS, live prices, level cache, cooldowns), `/cache` (age per pair/TF), `/clearcache`, `/help`.
+- Signal messages include the full indicator breakdown, warnings, trade setup, **trailing-stop management plan** (arm price + trail distance, from Signal/pb params by `strategy`), and a risk disclaimer on every outbound message.
+- Blocking scan work runs in `ThreadPoolExecutor(max_workers=4)`; one shared `ForexDataProvider` across all scans (cache reuse). Logs to `logs/telegram.log`.
+
+---
+
+## main.py — CLI
+
+`python main.py [--pair XAUUSD] [--timeframe M5|M15|M30|H1|H4|D1] [--detail] [--json] [--verbose]`. Uses the pair factory, renders a compact terminal report (decision, setup, context, top-3 reasons per side, warnings) or full JSON via `report_to_dict`.
+
+---
+
+## backtest/gpu_backtest.py — the backtester
+
+Replays history bar-by-bar through the **real `SignalAggregator.analyze_from_dataframes`** — same gates, same setup math. News is disabled (`use_news=False`): backtests are pure technical.
+
+**Realism model (all validated fixes — do not regress):**
+- Entry at the **next bar's open** after the signal bar; SL/TP distances preserved relative to actual entry. Entries are **re-anchored to the raw price path** (`_reanchor_entries_to_raw`): if cleaning dropped a >10σ spike bar right after the signal bar, the trade enters at the spike bar's open, not the bar after it — skipping it was a favorable bias.
+- Round-trip `--spread` subtracted from every trade's PnL before computing R (default = pair's `default_spread`: XAU $0.30). Results before 2026-07-08 (+59.5R baseline era) charged no spread — not comparable.
+- Trade resolution runs against the **RAW price path** (`fetch_history(clean=False)`): the parquet cache stores raw bars; cleaning (which drops >10σ news spikes that hit stops in reality) is applied only to the signal-side data. Caches written before this fix contain cleaned data — refetch to restore spike bars.
+- Same-bar SL+TP tie → SL first (conservative). Gap through SL/TP → filled at the bar open.
+- **Trailing exits** (default on, from `signals.trail_arm_r/trail_dist_r`; XAU_PB trades carry per-trade 0.8/0.75 overrides): stop ratchets at bar close only, so a new stop can only fill from the next bar (conservative). Exit reasons: SL / TRAIL / TP1 (static mode) / TIMEOUT / NO_DATA.
+- Session + weekend + weekday filters evaluate the **decision time = bar close** (`ts + primary_duration`) — exactly matching the live bot's wall-clock check.
+- Confirmation TFs are sliced to bars **fully closed** by decision time (no look-ahead).
+
+**Performance:** `precompute_indicators` builds every indicator series once per TF (CuPy on GPU or NumPy — check the log line `Backend: CuPy GPU` vs `NumPy CPU fallback`; CuPy needs the `nvidia-*-cu12` pip wheels or it silently falls back). `GPUTechnicalEngine` subclasses the live engine and overrides only the `_compute_*` methods with cache lookups — outputs are byte-identical to the live engine (regression-tested). Bar replay: pure per-bar evaluation fans out over forked worker processes (`--workers`, 0 = auto: cores−2 capped 12; <5000 bars forces 1); the entry **cooldown (5 bars) is the only sequential rule**, applied in an ordered pass afterwards — parallel and sequential books are identical. ~300 bars/s/core, ~1700 bars/s at 10 workers.
+
+**Strategy tags** (from reasons[0] markers): `XAU_PB`, `XAU_CP`, `GENERIC` (plus legacy `XAU` recognized). `--strategies` filters the book by tag. `--cp` force-enables the chart-pattern entry for candidate validation; `--cp-patterns` narrows the detector set.
+
+**CLI:**
+```bash
+python backtest/gpu_backtest.py --pair XAUUSD --days 365
+python backtest/gpu_backtest.py --pair XAUUSD --start 2025-01-01 --end 2025-06-01 --primary-tf 15min --confirm-tfs 1h,4h
+# key flags: --spread --quality A+,A,B,C --strategies --workers --max-hold-bars 300
+#            --warmup-bars 250 --cooldown-bars 5 --no-cache --export-csv out.csv --detail
+```
+Report: summary metrics (total R, PF, Sharpe on R, max DD, MFE/MAE), breakdowns by quality/strategy/regime, trade log, and per-trade reasoning dumps (indicator-by-indicator agreement at entry — required by `instructions_for_backtest.txt`). `--export-csv` flattens every indicator's signal/value/agreement into columns.
+
+**Comparability note (2026-07-16):** the parity/realism fixes of that date (EMA-ribbon port to the GPU engine, exact-pandas Bollinger precompute, raw-path entry re-anchoring, per-TF D1 warmup, live confirm-map CLI default, outlier snap-back retention) all shift books slightly vs earlier runs. Books produced before these fixes are not directly comparable — re-run the baseline before any before/after tuning comparison.
+
+Data: paginated TD fetches (5000-bar chunks) cached as parquet in `backtest/cache/` keyed by symbol/TF/window; warmup pad ≥60 days before the window for EMA200, extended per-TF so slow confirmation TFs get ≥210 bars at window start (D1 gets ~299 days — 62 days gave only ~61 bars: no EMA200, unlike live's 300-bar outputsize floor). `--confirm-tfs` defaults to the pair factory's live confirmation map (pass it explicitly to override); a hardcoded default here previously made the standard invocation backtest a different MTF setup than the bot scans.
+
+---
+
+## test_all.py — QA suite
+
+`python test_all.py` — **162 tests, 10 modules** (config, data_provider, engine, news_engine, signal_aggregator, main, telegram_bot, integration, pair_signals registry + engine-bug regressions, 2026-07-16 audit-fix regressions). No network needed (mocked). Uses a custom `@test("name")` decorator that runs immediately (NOT pytest — don't grep for `def test_`). Exit code non-zero on failure. Includes: the xauusd-factory ≡ inline-config equivalence test, engine bug-fix regression tests (Module 9b), and a live-vs-GPU engine parity sweep incl. the EMA-ribbon path (Module 11). **Any signal-logic change must keep this green and should add tests.**
+
+---
+
+## Environment variables
+
+| Variable | Purpose | Default |
+|---|---|---|
+| `TWELVE_DATA_API_KEY` | market data key | hardcoded fallback in `config.py` |
+| `TELEGRAM_TOKEN` | bot token | hardcoded fallback in `telegram_bot.py` |
+| `TELEGRAM_CHAT_ID` | owner chat id (boot auto-scan + owner lock) | hardcoded fallback |
+| `TELEGRAM_CHANNEL_ID` | subscriber channel (`-100…`); `""` disables the feed | hardcoded fallback |
+| `FB2_EMA_LO` | EMA200 "near" penalty threshold in % | `0.5` |
+
+Dependencies (`requirements.txt`): numpy, pandas, requests, twelvedata, python-telegram-bot ≥20, websockets. Python 3.12. A `venv/` exists in-repo.
+
+---
+
+## Code conventions
+
+- Type hints on public methods; dataclasses for all config and outputs.
+- `SignalStrength` (−2..+2) is the universal signal currency.
+- Key levels dict keys: `swing_high_*`, `swing_low_*`, `ema_*`, `bb_*`, `kc_*`, `atr`, `atr_pct`.
+- Private per-indicator series ride on `TechnicalAnalysis` as `_underscore` attributes; anything `_`-prefixed or weight-0 is skipped by scorers/formatters.
+- UTC everywhere; DataFrames indexed by tz-aware `Datetime`, oldest-first, capitalized OHLCV columns.
+- Logging via module loggers; the bot adds a rotating file handler.
+
+---
+
+## Rules for making changes (READ BEFORE EDITING SIGNAL LOGIC)
+
+1. **Never change a tuned value without re-validation.** The acceptance rule used throughout: positive total R in BOTH the train and validation halves of a ≥12-month window, PF ≥ 1.3, at realistic cost (XAU $0.30 round trip), with trailing exits and next-open fills. For any marginal edge additionally re-test on the 2024-06→2025-03 out-of-sample window — within-year splits alone have approved configs that failed OOS.
+2. **Per-timeframe overrides live in the pair factory** (`pair_signals/*.py`), never in global defaults — the same knob has opposite effects on different TFs (proven twice: `tp_floor_rr`, hour-9 block). Document the validation evidence in the factory docstring.
+3. **Pair isolation is a hard invariant.** XAU gates in the aggregator are display-gated (`report.display_name == "XAU/USD"`) so a future pair port cannot inherit gold-calibrated logic by accident.
+4. **Live == backtest parity**: session/weekend/hour filters must evaluate bar-CLOSE time in both paths; `xauusd.make_config` must stay equivalent to the inline config the backtester/bot would build (test-enforced). If you add a live gate, add it to the replay path (or `analyze_from_dataframes`) in the same commit.
+5. **ROC stays weighted** (see engine table) unless you retune composite thresholds and re-validate — removing it lost 13R.
+6. **Backtester realism knobs are results-critical**: raw-path resolution, spread charging, next-open entry, SL-first ties, close-only trail ratchet. Any "optimization" that touches these invalidates comparability with the pinned books (`backtest/*.csv`, memory notes).
+7. **Don't zero/disable indicators via config in production paths** — `disabled_indicators` exists for experiments; composite thresholds assume the full set.
+8. **The forming candle must stay dropped** and analysis must only ever see closed bars, or live signals repaint.
+9. Keep `test_all.py` green (`python test_all.py`); extend it for every new gate/knob.
+10. Legacy/no-op config fields exist (flagged above) — do not build new logic on them; delete only with test updates.

app_config.py

@@ -1,398 +0,0 @@
-"""
-app_config.py — deployment-settings overlay (config.json)
-=========================================================
-
-Loads the gitignored config.json at the repo root ONCE per process.
-Deployment surface ONLY: Telegram credentials, data-provider selection
-and API keys, scanned pairs/timeframes, per-pair base blocked hours,
-alert-quality boot default, log level. Signal logic, weights,
-thresholds and gates stay in config.py / pair_signals.
-
-Precedence for every setting: env var → config.json → code default.
-A repo-root .env file (gitignored) is loaded at import via setdefault,
-so it behaves exactly like real environment variables except that a
-real shell export always wins. Secrets have NO code default — with
-neither env nor config.json the bot refuses to start
-(telegram_bot.main) and data fetches fail with a named error
-(data_provider).
-
-Parity contract (test-enforced): with no config.json, or one holding
-the config.json.example defaults, every produced PlatformConfig is
-byte-identical to the pure-code defaults. apply_pair_overrides() runs
-in the pair factory BEFORE the validated per-timeframe deltas, so the
-bot, CLI and backtester all see the same overlay (live == backtest).
-
-Spec: docs/superpowers/specs/2026-07-21-centralized-config-design.md
-"""
-
-import json
-import logging
-import os
-from dataclasses import dataclass, field
-from typing import Any, Dict, List, Optional
-
-from config import PAIR_INFO, PlatformConfig, Timeframe, XAUUSD
-
-logger = logging.getLogger(__name__)
-
-CONFIG_PATH = os.path.join(
-    os.path.dirname(os.path.abspath(__file__)), "config.json")
-ENV_PATH = os.path.join(
-    os.path.dirname(os.path.abspath(__file__)), ".env")
-
-
-def load_dotenv(path: str = ENV_PATH) -> None:
-    """Minimal .env loader (no python-dotenv dependency).
-
-    KEY=VALUE lines land in os.environ via setdefault — a variable
-    already exported in the shell always wins, keeping the
-    env → config.json precedence intact (.env is just a convenient
-    place to put env vars). Comments, blank lines, ``export `` prefixes
-    and single/double quotes are handled; anything else is skipped.
-    """
-    try:
-        with open(path, encoding="utf-8") as fh:
-            lines = fh.read().splitlines()
-    except OSError:
-        return
-    for line in lines:
-        line = line.strip()
-        if not line or line.startswith("#") or "=" not in line:
-            continue
-        if line.startswith("export "):
-            line = line[len("export "):]
-        key, _, value = line.partition("=")
-        key, value = key.strip(), value.strip()
-        if len(value) >= 2 and value[0] == value[-1] and value[0] in "\"'":
-            value = value[1:-1]
-        elif value[:1] not in "\"'":
-            # Unquoted value: whitespace-then-# starts an inline comment
-            # (KEY=abc  # prod). Quote the value to keep a literal " #".
-            for sep in (" #", "\t#"):
-                cut = value.find(sep)
-                if cut != -1:
-                    value = value[:cut].rstrip()
-        if key:
-            os.environ.setdefault(key, value)
-
-
-# Module import == process boot, before any resolver can run.
-load_dotenv()
-
-_VALID_QUALITIES = ("A+", "A", "B", "C")
-_VALID_LOG_LEVELS = ("DEBUG", "INFO", "WARNING", "ERROR", "CRITICAL")
-# Timeframes the bot's candle-boundary scheduler supports (must have
-# AUTOSCAN_CONFIRMATIONS entries in telegram_bot.py).
-_SCANNABLE_TFS = ("M5", "M15", "H1", "H4")
-
-_KNOWN_TOP = {"telegram", "data_providers", "scanning", "pairs", "log_level",
-              "oanda", "deepseek"}
-_KNOWN_OANDA = {
-    "account_id", "api_token", "base_url", "lot_size",
-    "quality_gate", "max_hold_bars",
-}
-_KNOWN_DEEPSEEK = {
-    "api_key", "model", "enabled", "timeout_seconds", "daily_request_cap",
-}
-_KNOWN_TELEGRAM = {"token", "chat_id", "channel_id", "alert_min_quality"}
-_KNOWN_SCANNING = {"pairs", "timeframes"}
-_KNOWN_PAIR_KEYS = {"blocked_hours_utc"}
-
-
-class AppConfigError(Exception):
-    """config.json is present but invalid — fail fast at boot."""
-
-
-@dataclass
-class AppConfig:
-    telegram_token: str = ""
-    telegram_chat_id: str = ""
-    telegram_channel_id: str = ""          # "" = channel feed disabled
-    alert_min_quality: str = "C"
-    active_provider: str = "twelve_data"
-    streaming_provider: Optional[str] = None   # None = active provider
-    provider_settings: Dict[str, Dict[str, Any]] = field(default_factory=dict)
-    scan_pairs: List[str] = field(default_factory=lambda: [XAUUSD])
-    scan_timeframes: List[Timeframe] = field(default_factory=lambda: [
-        Timeframe.M5, Timeframe.M15, Timeframe.H1, Timeframe.H4])
-    pair_overrides: Dict[str, Dict[str, Any]] = field(default_factory=dict)
-    oanda_settings: Dict[str, Any] = field(default_factory=dict)
-    deepseek_settings: Dict[str, Any] = field(default_factory=dict)
-    log_level: str = "INFO"
-
-
-def _check_keys(section: Dict[str, Any], known: set, where: str) -> None:
-    if not isinstance(section, dict):
-        raise AppConfigError(f"config.json: {where} must be a JSON object")
-    unknown = sorted(set(section) - known)
-    if unknown:
-        raise AppConfigError(
-            f"config.json: unknown key(s) {unknown} in {where}; "
-            f"known keys: {sorted(known)}")
-
-
-def load_app_config(path: str = CONFIG_PATH) -> AppConfig:
-    """Parse + validate config.json. Missing file → pure defaults."""
-    app = AppConfig()
-    if not os.path.exists(path):
-        logger.warning(
-            "config.json not found at %s — running on code defaults "
-            "(secrets must come from environment variables)", path)
-        return app
-    try:
-        with open(path, encoding="utf-8") as fh:
-            raw = json.load(fh)
-    except json.JSONDecodeError as e:
-        raise AppConfigError(f"config.json is not valid JSON: {e}") from e
-    if not isinstance(raw, dict):
-        raise AppConfigError("config.json top level must be a JSON object")
-    _check_keys(raw, _KNOWN_TOP, "top level")
-
-    tg = raw.get("telegram", {})
-    _check_keys(tg, _KNOWN_TELEGRAM, '"telegram"')
-    app.telegram_token = str(tg.get("token", app.telegram_token))
-    app.telegram_chat_id = str(tg.get("chat_id", app.telegram_chat_id))
-    app.telegram_channel_id = str(
-        tg.get("channel_id", app.telegram_channel_id))
-    quality = tg.get("alert_min_quality", app.alert_min_quality)
-    if quality not in _VALID_QUALITIES:
-        raise AppConfigError(
-            f'config.json: telegram.alert_min_quality "{quality}" '
-            f"not in {_VALID_QUALITIES}")
-    app.alert_min_quality = quality
-
-    dp = raw.get("data_providers")
-    if dp is not None:
-        if not isinstance(dp, dict):
-            raise AppConfigError('config.json: "data_providers" must be an object')
-        app.active_provider = str(dp.get("active", app.active_provider))
-        app.provider_settings = {}
-        for name, block in dp.items():
-            # "active" and "streaming" are reserved scalars, not blocks.
-            if name in ("active", "streaming"):
-                continue
-            if not isinstance(block, dict):
-                raise AppConfigError(
-                    f'config.json: data_providers.{name} must be an object')
-            app.provider_settings[name] = dict(block)
-        if app.active_provider not in app.provider_settings:
-            raise AppConfigError(
-                f'config.json: data_providers.active "{app.active_provider}" '
-                "has no matching provider block")
-        streaming = dp.get("streaming")
-        if streaming is not None:
-            if not isinstance(streaming, str):
-                raise AppConfigError(
-                    'config.json: data_providers.streaming must be a string '
-                    "(a provider name)")
-            if streaming not in app.provider_settings:
-                raise AppConfigError(
-                    f'config.json: data_providers.streaming "{streaming}" '
-                    "has no matching provider block")
-            app.streaming_provider = streaming
-
-    scanning = raw.get("scanning", {})
-    _check_keys(scanning, _KNOWN_SCANNING, '"scanning"')
-    if "pairs" in scanning:
-        pairs = scanning["pairs"]
-        if not isinstance(pairs, list):
-            raise AppConfigError('config.json: scanning.pairs must be a list')
-        bad = [p for p in pairs if p not in PAIR_INFO]
-        if bad:
-            raise AppConfigError(
-                f"config.json: unknown pair(s) {bad} in scanning.pairs; "
-                f"known: {sorted(PAIR_INFO)}")
-        app.scan_pairs = list(pairs)
-    if "timeframes" in scanning:
-        names = scanning["timeframes"]
-        if not isinstance(names, list):
-            raise AppConfigError(
-                'config.json: scanning.timeframes must be a list')
-        bad = [n for n in names if n not in _SCANNABLE_TFS]
-        if bad:
-            raise AppConfigError(
-                f"config.json: unsupported timeframe(s) {bad} in "
-                f"scanning.timeframes; supported: {list(_SCANNABLE_TFS)}")
-        app.scan_timeframes = [Timeframe[n] for n in names]
-
-    pair_section = raw.get("pairs", {})
-    if not isinstance(pair_section, dict):
-        raise AppConfigError('config.json: "pairs" must be a JSON object')
-    for symbol, overrides in pair_section.items():
-        if symbol not in PAIR_INFO:
-            raise AppConfigError(
-                f'config.json: unknown pair "{symbol}" in pairs; '
-                f"known: {sorted(PAIR_INFO)}")
-        _check_keys(overrides, _KNOWN_PAIR_KEYS, f'pairs."{symbol}"')
-        if "blocked_hours_utc" in overrides:
-            hours = overrides["blocked_hours_utc"]
-            valid = isinstance(hours, list) and all(
-                isinstance(h, int) and not isinstance(h, bool)
-                and 0 <= h <= 23 for h in hours)
-            if not valid:
-                raise AppConfigError(
-                    f'config.json: pairs."{symbol}".blocked_hours_utc must '
-                    "be a list of integers in 0-23")
-    app.pair_overrides = {s: dict(o) for s, o in pair_section.items()}
-
-    oanda = raw.get("oanda", {})
-    _check_keys(oanda, _KNOWN_OANDA, '"oanda"')
-    if "lot_size" in oanda:
-        ls = oanda["lot_size"]
-        if not isinstance(ls, int) or isinstance(ls, bool) or ls < 1:
-            raise AppConfigError(
-                "config.json: oanda.lot_size must be a positive integer")
-    if "quality_gate" in oanda and oanda["quality_gate"] not in _VALID_QUALITIES:
-        raise AppConfigError(
-            f'config.json: oanda.quality_gate "{oanda["quality_gate"]}" '
-            f"not in {_VALID_QUALITIES}")
-    if "max_hold_bars" in oanda:
-        mhb = oanda["max_hold_bars"]
-        if not isinstance(mhb, int) or isinstance(mhb, bool) or mhb < 1:
-            raise AppConfigError(
-                "config.json: oanda.max_hold_bars must be a positive integer")
-    app.oanda_settings = dict(oanda)
-
-    deepseek = raw.get("deepseek", {})
-    _check_keys(deepseek, _KNOWN_DEEPSEEK, '"deepseek"')
-    if "enabled" in deepseek and not isinstance(deepseek["enabled"], bool):
-        raise AppConfigError(
-            "config.json: deepseek.enabled must be true or false")
-    if "timeout_seconds" in deepseek:
-        ts = deepseek["timeout_seconds"]
-        if isinstance(ts, bool) or not isinstance(ts, (int, float)) or ts <= 0:
-            raise AppConfigError(
-                "config.json: deepseek.timeout_seconds must be a positive "
-                "number")
-    if "daily_request_cap" in deepseek:
-        cap = deepseek["daily_request_cap"]
-        if isinstance(cap, bool) or not isinstance(cap, int) or cap < 1:
-            raise AppConfigError(
-                "config.json: deepseek.daily_request_cap must be a positive "
-                "integer")
-    if "model" in deepseek:
-        if not isinstance(deepseek["model"], str) or not deepseek["model"]:
-            raise AppConfigError(
-                "config.json: deepseek.model must be a non-empty string")
-    app.deepseek_settings = dict(deepseek)
-
-    level = raw.get("log_level", app.log_level)
-    if level not in _VALID_LOG_LEVELS:
-        raise AppConfigError(
-            f'config.json: log_level "{level}" not in {_VALID_LOG_LEVELS}')
-    app.log_level = level
-    return app
-
-
-# ── Process-wide singleton ────────────────────────────────────────────
-
-_cached: Optional[AppConfig] = None
-
-
-def get_app_config() -> AppConfig:
-    global _cached
-    if _cached is None:
-        _cached = load_app_config()
-    return _cached
-
-
-def set_app_config(app: Optional[AppConfig]) -> None:
-    """Override the process-wide AppConfig (tests/experiments).
-    Pass None to re-enable lazy loading from CONFIG_PATH."""
-    global _cached
-    _cached = app
-
-
-# ── Resolution helpers: env var → config.json → default ───────────────
-
-def telegram_token(app: Optional[AppConfig] = None) -> str:
-    app = app or get_app_config()
-    return os.environ.get("TELEGRAM_TOKEN") or app.telegram_token
-
-
-def telegram_chat_id(app: Optional[AppConfig] = None) -> str:
-    app = app or get_app_config()
-    return os.environ.get("TELEGRAM_CHAT_ID") or app.telegram_chat_id
-
-
-def telegram_channel_id(app: Optional[AppConfig] = None) -> str:
-    """Env set-but-empty ("") means "disable the feed" and beats JSON,
-    so this checks None-ness, not truthiness."""
-    app = app or get_app_config()
-    env = os.environ.get("TELEGRAM_CHANNEL_ID")
-    return env if env is not None else app.telegram_channel_id
-
-
-# Per-provider env-var overrides for API keys (env → config.json → "").
-_PROVIDER_ENV_KEYS = {
-    "twelve_data": "TWELVE_DATA_API_KEY",
-    "tiingo": "TIINGO_API_KEY",
-}
-
-
-def provider_api_key(
-    app: Optional[AppConfig] = None, provider: Optional[str] = None
-) -> str:
-    """API key for `provider` (default: the ACTIVE provider). The env
-    override is per-provider (TWELVE_DATA_API_KEY applies only to
-    twelve_data, TIINGO_API_KEY only to tiingo)."""
-    app = app or get_app_config()
-    name = provider or app.active_provider
-    env_key = _PROVIDER_ENV_KEYS.get(name)
-    if env_key:
-        env = os.environ.get(env_key)
-        if env:
-            return env
-    block = app.provider_settings.get(name, {})
-    return str(block.get("api_key", ""))
-
-
-# ── Oanda resolution helpers (env → config.json → default) ───────────
-
-def oanda_account_id(app: Optional[AppConfig] = None) -> str:
-    app = app or get_app_config()
-    return os.environ.get("OANDA_ACCOUNT_ID") or str(
-        app.oanda_settings.get("account_id", ""))
-
-
-def oanda_api_token(app: Optional[AppConfig] = None) -> str:
-    app = app or get_app_config()
-    return os.environ.get("OANDA_API_TOKEN") or str(
-        app.oanda_settings.get("api_token", ""))
-
-
-def oanda_base_url(app: Optional[AppConfig] = None) -> str:
-    app = app or get_app_config()
-    return os.environ.get("OANDA_BASE_URL") or str(
-        app.oanda_settings.get("base_url", "https://api-fxpractice.oanda.com"))
-
-
-# ── DeepSeek resolution helper (env → config.json → "") ──────────────
-
-def deepseek_api_key(app: Optional[AppConfig] = None) -> str:
-    app = app or get_app_config()
-    return os.environ.get("DEEPSEEK_API_KEY") or str(
-        app.deepseek_settings.get("api_key", ""))
-
-
-# ── PlatformConfig overlay ────────────────────────────────────────────
-
-def apply_pair_overrides(
-    cfg: PlatformConfig, app: Optional[AppConfig] = None
-) -> PlatformConfig:
-    """Stamp deployment settings from config.json onto a PlatformConfig.
-
-    Called by the pair factories BEFORE their validated per-timeframe
-    deltas, so e.g. the M5 hour-9 append lands on top of the JSON base.
-    The pair's base blocked_hours_utc is set on BOTH mirrored copies
-    (xau_strategy + backtest) so they cannot drift.
-    """
-    app = app or get_app_config()
-    overrides = app.pair_overrides.get(cfg.symbol, {})
-    if "blocked_hours_utc" in overrides:
-        hours = list(overrides["blocked_hours_utc"])
-        cfg.xau_strategy.blocked_hours_utc = hours
-        cfg.backtest.blocked_hours_utc = list(hours)
-    cfg.scan_pairs = list(app.scan_pairs)
-    cfg.scan_timeframes = list(app.scan_timeframes)
-    return cfg

backtest/diagnose_filters.py

@@ -51,7 +51,7 @@ from gpu_backtest import (  # noqa: E402
     precompute_indicators,
     _is_xau_blocked_by_calendar,
 )
-from providers import make_data_provider  # noqa: E402
+from data_provider import ForexDataProvider  # noqa: E402
 
 logger = logging.getLogger("diagnose_filters")
 
@@ -79,54 +79,34 @@ def classify_bar(
     """
     filters_hit: List[str] = []
 
-    # Known filter tags, matched against the exact warning strings the
-    # aggregator emits (refreshed 2026-07-30 for gates 6d-6i). ORDER
-    # MATTERS: most-specific prefix first, and each warning maps to at
-    # most ONE tag (break on first hit) — substring families like
-    # "M5 FILTER: SELL quality" vs "M5 FILTER" and "TREND EXHAUSTION"
-    # vs "EXHAUSTION" would otherwise double-count.
-    filter_tags = [
-        # M5 noise gates (6i, _apply_m5_filters)
-        ("M5 FILTER: |score|",            "M5_SCORE"),
-        ("M5 FILTER: no MTF agreement",   "M5_MTF"),
-        ("M5 FILTER: no confirmation TF", "M5_CONF"),
-        ("M5 FILTER: SELL quality",       "M5_SELL_QUALITY"),
-        ("M5 FILTER: SELL blocked — RSI", "M5_SELL_RSI"),
-        ("M5 FILTER: '",                  "M5_VOL"),
-        ("M5 FILTER",                     "M5_OTHER"),
-        # XAU SELL gates (6e/6g/6g2/6g3)
-        ("SELL FILTER: quality",          "SELL_QUALITY"),
-        ("SELL FILTER: volatility",       "SELL_VOL"),
-        ("SELL FILTER: regime",           "SELL_REGIME"),
-        ("UTC entry blocked for shorts",  "SELL_HOUR"),
-        # Hard gates
-        ("MTF OPPOSITION",                "MTF_OPPOSITION"),
-        ("EXHAUSTED TREND FILTER",        "EXHAUSTED_TREND"),
-        ("BIG CANDLE FILTER",             "BIG_CANDLE"),
-        ("WEEKEND FILTER",                "WEEKEND_FILTER"),
-        ("SESSION FILTER",                "SESSION_FILTER"),
-        ("DAY FILTER",                    "DAY_FILTER"),
-        ("LOW ATR FILTER",                "LOW_ATR_FILTER"),
-        ("REGIME FILTER",                 "REGIME_FILTER"),
-        # Soft penalties / context warnings
-        ("TREND EXHAUSTION",              "TREND_EXHAUSTION"),
-        ("EXHAUSTION",                    "EXHAUSTION_PENALTY"),
-        ("NO PULLBACK",                   "NO_PULLBACK_PENALTY"),
-        ("Weak pullback",                 "WEAK_PULLBACK_PENALTY"),
-        ("Extended — RSI",                "RSI_EXTENDED_PENALTY"),
-        ("Counter-trend",                 "COUNTER_TREND_PENALTY"),
-        ("EXTREME volatility",            "EXTREME_VOL_PENALTY"),
-        ("NEAR EMA200",                   "NEAR_EMA200_PENALTY"),
-        ("FAR FROM EMA200",               "FAR_EMA200_PENALTY"),
-        ("EVENT RISK",                    "EVENT_RISK"),
-        ("Timeframes CONFLICT",           "MTF_CONFLICT"),
-    ]
+    # Scan warnings for known filter tags
+    filter_tags = {
+        "H4 RSI GATE":       "H4_RSI_GATE",
+        "H1 MACD GATE":      "H1_MACD_GATE",
+        "RSI ZONE GATE":     "RSI_ZONE_GATE",
+        "XAU BUY FILTER":    "XAU_BUY_FILTER",
+        "SESSION FILTER":    "SESSION_FILTER",
+        "DAY FILTER":        "DAY_FILTER",
+        "LOW ATR FILTER":    "LOW_ATR_FILTER",
+        "SELL SUPPRESSED":   "SELL_SUPPRESSED",
+        "EXHAUSTION":        "EXHAUSTION_PENALTY",
+        "TREND EXHAUSTION":  "TREND_EXHAUSTION",
+        "NO PULLBACK":       "NO_PULLBACK_PENALTY",
+        "Weak pullback":     "WEAK_PULLBACK_PENALTY",
+        "Extended entry":    "EXTENDED_ENTRY_PENALTY",
+        "Counter-trend":     "COUNTER_TREND_PENALTY",
+        "EXTREME volatility":"EXTREME_VOL_PENALTY",
+        "NEAR EMA200":       "NEAR_EMA200_PENALTY",
+        "FAR FROM EMA200":   "FAR_EMA200_PENALTY",
+        "EVENT RISK":        "EVENT_RISK",
+        "REGIME FILTER":     "REGIME_FILTER",
+        "Timeframes CONFLICT": "MTF_CONFLICT",
+    }
 
     for w in report.warnings:
-        for tag, name in filter_tags:
+        for tag, name in filter_tags.items():
             if tag in w:
                 filters_hit.append(name)
-                break
 
     # Determine outcome
     non_neutral = report.final_signal not in (SignalStrength.NEUTRAL,)
@@ -226,7 +206,7 @@ def run_diagnosis(
         confirmation_timeframes=list(confirm_tfs),
     )
 
-    provider = make_data_provider(platform_cfg)
+    provider = ForexDataProvider(platform_cfg)
 
     # ── Load data ──
     timeframes = [primary_tf] + list(confirm_tfs)

backtest/gpu_backtest.py

@@ -30,7 +30,6 @@ Usage:
 from __future__ import annotations
 
 import argparse
-import json
 import logging
 import multiprocessing as mp
 import os
@@ -40,13 +39,10 @@ from dataclasses import dataclass, field
 from datetime import datetime, timedelta, timezone
 from pathlib import Path
 from typing import Dict, List, Optional, Tuple
-from zoneinfo import ZoneInfo
 
 import numpy as np
 import pandas as pd
 
-_NY_TZ = ZoneInfo("America/New_York")
-
 # Ensure project root on sys.path so this script can be invoked from anywhere.
 PROJECT_ROOT = Path(__file__).resolve().parent.parent
 if str(PROJECT_ROOT) not in sys.path:
@@ -59,7 +55,7 @@ from config import (  # noqa: E402
     Timeframe,
     XAUUSD,
 )
-from providers import BaseDataProvider, make_data_provider  # noqa: E402
+from data_provider import ForexDataProvider  # noqa: E402
 from engine import IndicatorSignal, TechnicalAnalysis, TechnicalEngine  # noqa: E402
 from signal_aggregator import SignalAggregator, SignalReport, TradeSetup  # noqa: E402
 
@@ -405,11 +401,6 @@ class GPUTechnicalEngine(TechnicalEngine):
         # cache key = timeframe.value (e.g. "15min", "1h", "4h")
         self.caches = caches
         self._current_tf: Optional[str] = None
-        # Cache-miss fallbacks recompute with the live engine's pandas
-        # math (different EWM seeding than the precomputed cache). That
-        # is a parity hazard, so it must never happen silently.
-        self.fallback_count = 0
-        self._fallback_warned = False
 
     def analyze(self, df: pd.DataFrame, symbol: str, timeframe: str) -> TechnicalAnalysis:
         # Latch the timeframe so per-indicator overrides know which cache to use.
@@ -427,23 +418,9 @@ class GPUTechnicalEngine(TechnicalEngine):
             return None, -1
         cache = self.caches.get(self._current_tf)
         if cache is None:
-            self._note_fallback(f"no cache for TF {self._current_tf!r}")
             return None, -1
         pos = cache.index_pos.get(df.index[-1])
-        if pos is None:
-            self._note_fallback(
-                f"{self._current_tf}: slice end {df.index[-1]} not in cache")
-            return None, -1
-        return cache, pos
-
-    def _note_fallback(self, why: str) -> None:
-        self.fallback_count += 1
-        if not self._fallback_warned:
-            self._fallback_warned = True
-            logger.warning(
-                "GPU cache miss → falling back to live-engine recompute "
-                "(%s). Further misses counted silently; check "
-                "fallback_count at end of run.", why)
+        return (cache, pos) if pos is not None else (None, -1)
 
     # ── EMA overrides ──────────────────────────────────────────────
 
@@ -551,11 +528,6 @@ class GPUTechnicalEngine(TechnicalEngine):
             signal = SignalStrength.NEUTRAL
             conf = 40
 
-        # Mirrors the live engine's ema_partial_neutral squash (parity).
-        if c.ema_partial_neutral and signal in (SignalStrength.BUY, SignalStrength.SELL):
-            signal = SignalStrength.NEUTRAL
-            conf = 40
-
         crossover_desc = ""
         # Live: `if len(ema_fast) >= 3:` then iloc[-2] — needs len ≥ 2 functionally,
         # but live's guard requires len ≥ 3 so we match that to stay byte-identical.
@@ -848,12 +820,7 @@ class GPUTechnicalEngine(TechnicalEngine):
         c = self.config
         n = c.roc_period
 
-        # Mirror the live guard (engine._compute_roc: len < n + 5 → skip,
-        # i.e. emit only from pos >= n + 4). Unreachable in practice —
-        # analyze() floors at 30 bars and n+4 < 30 for shipped periods —
-        # but keep the engines textually aligned so a future period/floor
-        # change can't open a silent divergence window.
-        if pos < n + 4:
+        if pos < n + 1:
             return
 
         roc_val = float(cache.roc[pos])
@@ -884,10 +851,15 @@ class GPUTechnicalEngine(TechnicalEngine):
                 signal = SignalStrength.SELL
                 conf = min(70, 50 + roc_abs * 10)
                 desc = f"ROC {roc_val:.2f}% — Falling momentum"
+        elif roc_prev <= 0 < roc_val:
+            signal = SignalStrength.BUY
+            conf = 65
+            desc = f"ROC crossed above zero (+{roc_val:.2f}%)"
+        elif roc_prev >= 0 > roc_val:
+            signal = SignalStrength.SELL
+            conf = 65
+            desc = f"ROC crossed below zero ({roc_val:.2f}%)"
         else:
-            # Zero-crosses land in the branches above by definition (a
-            # cross up is positive and rising, a cross down negative and
-            # falling) — only flat/fading momentum reaches here.
             signal = SignalStrength.NEUTRAL
             conf = 30
             desc = f"ROC {roc_val:+.2f}% — No clear momentum"
@@ -903,11 +875,6 @@ class GPUTechnicalEngine(TechnicalEngine):
             return super()._compute_keltner(df, result)
 
         c = self.config
-        # Mirror the live guard (engine._compute_keltner: len < max(kc_period,
-        # kc_atr_period) + 5 → skip). Unreachable below the 30-bar analyze()
-        # floor with shipped periods; kept aligned for the same reason as ROC.
-        if pos < max(c.kc_period, c.kc_atr_period) + 4:
-            return
         latest = float(df["Close"].iloc[-1])
         kc_upper_val = float(cache.kc_upper[pos])
         kc_lower_val = float(cache.kc_lower[pos])
@@ -973,21 +940,13 @@ class GPUTechnicalEngine(TechnicalEngine):
 CACHE_DIR = Path(__file__).resolve().parent / "cache"
 
 
-def _cache_path(
-    symbol: str, tf: Timeframe, start: datetime, end: datetime,
-    provider_name: str = "twelve_data",
-) -> Path:
+def _cache_path(symbol: str, tf: Timeframe, start: datetime, end: datetime) -> Path:
     safe_sym = symbol.replace("/", "_").replace("=X", "")
-    # Bars differ between feeds (Oanda vs TD divergence study 2026-07-24),
-    # so the cache is keyed per provider. twelve_data keeps the legacy
-    # un-prefixed filename so pinned-book caches stay valid.
-    prefix = "" if provider_name == "twelve_data" else f"{provider_name}_"
-    return CACHE_DIR / (
-        f"{prefix}{safe_sym}_{tf.value}_{start:%Y%m%d}_{end:%Y%m%d}.parquet")
+    return CACHE_DIR / f"{safe_sym}_{tf.value}_{start:%Y%m%d}_{end:%Y%m%d}.parquet"
 
 
 def fetch_history(
-    provider: BaseDataProvider,
+    provider: ForexDataProvider,
     symbol: str,
     timeframe: Timeframe,
     start: datetime,
@@ -995,45 +954,91 @@ def fetch_history(
     use_cache: bool = True,
     clean: bool = True,
 ) -> pd.DataFrame:
-    """Fetch OHLCV between start/end with parquet caching.
+    """Fetch OHLCV between start/end with parquet caching and pagination.
 
-    Pagination under the provider's per-request bar cap lives in
-    `BaseDataProvider.fetch_range`, so any registered provider can feed
-    the backtester.
+    Twelve Data caps responses at 5000 bars; we paginate by date range so
+    multi-month M5/M15 backtests work.
 
-    The parquet cache stores the RAW series; `validate_and_clean` runs on
+    The parquet cache stores the RAW series; `_validate_and_clean` runs on
     load. Pass clean=False to get the raw price path — trade resolution must
     use it, because cleaning drops >10σ spike bars (news candles) that hit
     stops in reality. (Caches written before this change already contain
     cleaned data; a refetch is needed to restore dropped spike bars.)
     """
     CACHE_DIR.mkdir(parents=True, exist_ok=True)
-    path = _cache_path(symbol, timeframe, start, end, provider.name)
+    path = _cache_path(symbol, timeframe, start, end)
     if use_cache and path.exists():
         logger.info(f"Loading cached {symbol} {timeframe.value}: {path.name}")
-        # Files written before the raw-cache change (2026-07-08, commit
-        # 6a96ed6) contain CLEANED bars; loading one with clean=False
-        # would silently miss the spike bars the raw path exists to model.
-        # The key format predates versioning, so detect by mtime.
-        _RAW_CACHE_EPOCH = 1783468800.0   # 2026-07-08 00:00 UTC
-        if path.stat().st_mtime < _RAW_CACHE_EPOCH:
-            logger.error(
-                f"{path.name} predates the raw-cache format (2026-07-08) "
-                f"and likely contains CLEANED bars — delete it and refetch "
-                f"before trusting trade resolution on this window")
         df = pd.read_parquet(path)
         if df.index.tz is None:
             df.index = df.index.tz_localize("UTC")
-        return provider.validate_and_clean(df, symbol, timeframe) if clean else df
+        return provider._validate_and_clean(df, symbol, timeframe) if clean else df
 
     logger.info(f"Fetching {symbol} {timeframe.value} from {start.date()} to {end.date()}")
 
-    df = provider.fetch_range(symbol, timeframe, start, end)
+    # Bars-per-call estimate to size each window safely under the 5000 cap.
+    bars_per_day = ForexDataProvider.BARS_PER_DAY[timeframe]
+    days_per_chunk = max(1, int(4500 / bars_per_day))
+
+    chunks: List[pd.DataFrame] = []
+    cursor_end = end
+    empty_streak = 0
+    while cursor_end > start:
+        cursor_start = max(start, cursor_end - timedelta(days=days_per_chunk))
+        try:
+            chunk = provider._fetch_twelve_data(
+                symbol=symbol,
+                timeframe=timeframe,
+                outputsize=5000,
+                start_date=cursor_start.strftime("%Y-%m-%d %H:%M:%S"),
+                end_date=cursor_end.strftime("%Y-%m-%d %H:%M:%S"),
+            )
+        except Exception as e:
+            # Twelve Data raises BadRequest for windows with no data (weekends,
+            # holidays, asset gaps). Skip the window, step back, retry — but
+            # bail out after a few consecutive empties to avoid spinning.
+            logger.warning(
+                f"Chunk {cursor_start.date()}–{cursor_end.date()} failed: {e}"
+            )
+            empty_streak += 1
+            if empty_streak >= 3:
+                break
+            cursor_end = cursor_start - timedelta(seconds=1)
+            continue
+
+        if chunk is None or chunk.empty:
+            empty_streak += 1
+            if empty_streak >= 3:
+                break
+            cursor_end = cursor_start - timedelta(seconds=1)
+            continue
+
+        empty_streak = 0
+        chunks.append(chunk)
+        # Step back; dedup on concat.
+        cursor_end = chunk.index[0] - timedelta(seconds=1)
+        if len(chunk) < 2:
+            break
+
+    if not chunks:
+        raise RuntimeError(f"No data fetched for {symbol} {timeframe.value}")
+
+    df = pd.concat(chunks).sort_index()
+    df = df[~df.index.duplicated(keep="last")]
+    # Fetches ending "now" include the still-forming candle — drop it so the
+    # cache only ever contains closed bars.
+    df = provider._drop_forming_candle(df, timeframe)
+    # Trim to requested window. `start`/`end` are already tz-aware UTC
+    # datetimes — wrapping with pd.Timestamp(..., tz=...) would raise on
+    # already-localized inputs.
+    end_ts = pd.Timestamp(end)
+    start_ts = pd.Timestamp(start)
+    df = df[(df.index >= start_ts) & (df.index <= end_ts)]
 
     if use_cache:
         df.to_parquet(path)
         logger.info(f"Cached {len(df)} raw bars to {path.name}")
-    return provider.validate_and_clean(df, symbol, timeframe) if clean else df
+    return provider._validate_and_clean(df, symbol, timeframe) if clean else df
 
 
 # ═════════════════════════════════════════════════════════════════════
@@ -1172,15 +1177,6 @@ def _simulate_trailing(
         reason = "TIMEOUT"
         for j in range(n):
             if np.isnan(h[j]):
-                # Data ends here: close at the LAST VALID bar, not at
-                # c[n-1] (which may itself be NaN — a NaN exit price used
-                # to poison r_multiple and every downstream metric).
-                if j == 0:
-                    exit_idx = 0
-                    reason = "NO_DATA"
-                else:
-                    exit_idx = j - 1
-                    exit_price = float(c[exit_idx])
                 break
             hit = (l[j] <= cur_sl) if buy else (h[j] >= cur_sl)
             if hit:
@@ -1202,15 +1198,8 @@ def _simulate_trailing(
                        else peak + dist * risk)
                 cur_sl = max(cur_sl, lvl) if buy else min(cur_sl, lvl)
 
-        if reason == "NO_DATA":
-            t.exit_reason = reason
-            continue
         if exit_price is None:
             exit_price = float(c[exit_idx])
-        if not np.isfinite(exit_price):
-            # Last-resort guard: never emit a NaN exit/R into the book.
-            t.exit_reason = "NO_DATA"
-            continue
         t.exit_reason = reason
         t.bars_held = exit_idx + 1
         t.exit_price = float(exit_price)
@@ -1383,12 +1372,6 @@ def simulate_trades(
             last_idx = int(n_valid) - 1
             t.exit_price = float(closes_arr[i, last_idx])
             t.exit_time = exit_times[i][last_idx]
-            if not np.isfinite(t.exit_price):
-                # NaN close on the final window bar (corrupt raw data)
-                # must not become a NaN R.
-                t.exit_reason = "NO_DATA"
-                t.exit_price = None
-                continue
 
         # Realized R from actual fill prices, net of round-trip spread.
         risk = abs(t.entry_price - t.sl)
@@ -1441,12 +1424,6 @@ class BacktestConfig:
     # Round-trip transaction cost in price units, charged on every trade
     # (XAU/USD retail spread ≈ $0.30). 0 disables.
     spread: float = 0.30
-    # RESEARCH (2026-08): >0 replaces the exit model with a hard TP at
-    # this multiple of risk from entry (2.0 = fixed 1:2 RR); trailing is
-    # disabled for ALL trades, including per-trade XAU_PB overrides.
-    # Entries/stops/gates are untouched, so the signal book is identical
-    # to the baseline — only the exit policy changes. Never shipped.
-    fixed_rr_tp: float = 0.0
     # Replay worker processes for the bar loop. 0 = auto (cpu_count - 2,
     # capped at 12), 1 = sequential in-process. Signal evaluation per bar
     # is pure (past data only), so it fans out across forked workers; the
@@ -1459,22 +1436,18 @@ class BacktestConfig:
 def _is_xau_blocked_by_calendar(ts: pd.Timestamp, cfg: PlatformConfig) -> Optional[str]:
     """Apply XAU weekend + allowed-weekday filter matching live `analyze_pair`.
 
-    Market closed Friday 17:00 -> Sunday 18:00 NEW YORK time (= Fri
-    21:00 UTC in US summer, 22:00 in winter — matches measured Oanda
-    feed behavior; 2026-08-04 audit I43). Must match `_is_forex_weekend`
-    in signal_aggregator.py exactly (rule 4; Module 21 agreement test).
-    Note the weekday gates below stay on UTC weekday — they mirror the
-    live gates, which evaluate UTC weekdays.
+    Forex market closed: Friday 22:00 UTC -> Sunday 22:00 UTC (Sydney open).
+    Must match `_is_forex_weekend` in signal_aggregator.py exactly.
     """
     xau = cfg.xau_strategy
     if xau.weekend_filter:
-        ny = ts.astimezone(_NY_TZ)
-        wd = ny.weekday()
-        if wd == 4 and ny.hour >= 17:
+        wd = ts.weekday()
+        hour = ts.hour
+        if wd == 4 and hour >= 22:
             return "WEEKEND_FILTER"
         if wd == 5:
             return "WEEKEND_FILTER"
-        if wd == 6 and ny.hour < 18:
+        if wd == 6 and hour < 22:
             return "WEEKEND_FILTER"
     if xau.allowed_weekdays and ts.weekday() not in xau.allowed_weekdays:
         return "DAY_FILTER"
@@ -1557,13 +1530,6 @@ def _replay_span(span: Tuple[int, int]):
             use_news=False,
         )
 
-        # Live parity: analyze_pair applies the optional higher-TF ADX
-        # regime filter after delegation, under this exact guard. Calling
-        # the SAME method on the SAME sliced dfs keeps the two paths
-        # identical if the (default-off) filter is ever enabled.
-        if platform_cfg.backtest.regime_filter:
-            aggregator._apply_regime_filter_live(report, slice_dfs)
-
         if report.trade_setup is None:
             continue
         if report.signal_quality not in bt_cfg.quality_filter:
@@ -1691,43 +1657,6 @@ def _reanchor_entries_to_raw(
         )
 
 
-def _apply_cooldown(
-    candidates: List[Tuple[int, Trade]], cooldown_bars: int
-) -> List[Tuple[int, Trade]]:
-    """Ordered cooldown pass over (bar_pos, Trade) candidates: drop any
-    candidate within `cooldown_bars` bars of the previously ACCEPTED
-    entry — exactly what a sequential replay loop would have skipped.
-    Candidates must already be sorted by position (stable merge from the
-    worker spans)."""
-    accepted: List[Tuple[int, Trade]] = []
-    last_entry_pos = -10_000
-    for pos, trade in candidates:
-        if pos - last_entry_pos < cooldown_bars:
-            continue
-        accepted.append((pos, trade))
-        last_entry_pos = pos
-    return accepted
-
-
-def _apply_fixed_rr_tp(trades: List[Trade], mult: float) -> None:
-    """RESEARCH: replace each trade's TP with a hard target at
-    `mult` × |entry − sl| and zero the per-trade trail params so the
-    XAU_PB overrides cannot resurrect trailing. Risk is measured off the
-    final (re-anchored) entry/SL, so call this after
-    `_reanchor_entries_to_raw`. The caller must also pass
-    trail_arm_r=0 to `simulate_trades` — dispatch is on the run-level arg."""
-    for t in trades:
-        risk = abs(t.entry_price - t.sl)
-        t.tp1 = (t.entry_price + mult * risk if t.direction == "BUY"
-                 else t.entry_price - mult * risk)
-        t.tp2 = t.tp1              # static branch never reads tp2; CSV hygiene
-        t.trail_arm_r = 0.0
-        t.trail_dist_r = 0.0
-        t.rr_planned = mult
-        if t.pip_value:
-            t.reward_pips = mult * risk / t.pip_value
-
-
 def run_backtest(
     bt_cfg: BacktestConfig,
     platform_cfg: Optional[PlatformConfig] = None,
@@ -1750,17 +1679,10 @@ def run_backtest(
             f"vs PlatformConfig={platform_cfg.primary_timeframe.value}"
         )
 
-    provider = make_data_provider(platform_cfg)
+    provider = ForexDataProvider(platform_cfg)
 
     # ── Load all required timeframes ──
     timeframes = [bt_cfg.primary_tf] + list(bt_cfg.confirmation_tfs)
-    # Live parity: analyze_pair also fetches the regime-filter TF when the
-    # optional H4-ADX gate is enabled. Without it here, M5/M15 primaries
-    # (whose confirm maps lack H4) would make _apply_regime_filter_live
-    # silently no-op in the replay while live applies the gate.
-    flt = platform_cfg.backtest
-    if flt.regime_filter and flt.regime_filter_timeframe not in timeframes:
-        timeframes.append(flt.regime_filter_timeframe)
     # Pad lookback so earliest TF has enough warmup for EMA200, ADX, etc.
     pad_days = max(60, bt_cfg.warmup_bars // 4)
     fetch_start = bt_cfg.start - timedelta(days=pad_days)
@@ -1772,7 +1694,7 @@ def run_backtest(
         # ≥210 bars (outputsize floor). Target data_provider's min_required
         # (210 bars); ×7/5 converts trading days to calendar days. Intraday
         # TFs stay on the base pad, so their books are unchanged.
-        bars_per_day = BaseDataProvider.BARS_PER_DAY[tf]
+        bars_per_day = ForexDataProvider.BARS_PER_DAY[tf]
         min_pad_days = int(np.ceil(210 / bars_per_day * 7 / 5)) + 5
         tf_start = bt_cfg.start - timedelta(days=max(pad_days, min_pad_days))
         df = fetch_history(
@@ -1855,10 +1777,18 @@ def run_backtest(
         candidates.extend(cands)
     candidates.sort(key=lambda pc: pc[0])
 
-    # ── Phase 2: entry cooldown — the only sequential rule.
-    accepted = _apply_cooldown(candidates, bt_cfg.cooldown_bars)
-    trades: List[Trade] = [t for _, t in accepted]
-    signal_ts_list: List[pd.Timestamp] = [primary_df.index[p] for p, _ in accepted]
+    # ── Phase 2: entry cooldown — the only sequential rule. A candidate
+    # inside the cooldown window of the previously ACCEPTED entry is
+    # dropped, exactly as the sequential loop skipped those bars.
+    trades: List[Trade] = []
+    signal_ts_list: List[pd.Timestamp] = []
+    last_entry_pos = -10_000
+    for pos, trade in candidates:
+        if pos - last_entry_pos < bt_cfg.cooldown_bars:
+            continue
+        trades.append(trade)
+        signal_ts_list.append(primary_df.index[pos])
+        last_entry_pos = pos
 
     logger.info(
         f"Replay complete: {len(trades)} entries in "
@@ -1878,17 +1808,10 @@ def run_backtest(
         use_cache=bt_cfg.use_cache, clean=False,
     )
     _reanchor_entries_to_raw(trades, signal_ts_list, sim_df)
-    if bt_cfg.fixed_rr_tp > 0:
-        _apply_fixed_rr_tp(trades, bt_cfg.fixed_rr_tp)
-        logger.info(
-            f"RESEARCH override: fixed-RR TP = {bt_cfg.fixed_rr_tp} x risk; "
-            f"trailing exit DISABLED for all {len(trades)} trades (incl. XAU_PB)"
-        )
-    run_arm = 0.0 if bt_cfg.fixed_rr_tp > 0 else platform_cfg.signals.trail_arm_r
     t_sim = time.perf_counter()
     simulate_trades(trades, sim_df, max_bars=bt_cfg.max_hold_bars,
                     spread=bt_cfg.spread,
-                    trail_arm_r=run_arm,
+                    trail_arm_r=platform_cfg.signals.trail_arm_r,
                     trail_dist_r=platform_cfg.signals.trail_dist_r)
     logger.info(
         f"Trade simulation took {time.perf_counter() - t_sim:.2f}s "
@@ -1938,14 +1861,6 @@ def _summarize(rs: List[float]) -> Dict:
 
 def compute_metrics(trades: List[Trade]) -> Metrics:
     resolved = [t for t in trades if t.exit_reason and t.exit_reason != "NO_DATA"]
-    # Last line of defense: one non-finite R (NaN exit on corrupt data)
-    # would silently NaN every aggregate below. Exclude loudly.
-    bad = [t for t in resolved if not np.isfinite(t.r_multiple)]
-    if bad:
-        logger.error(
-            "compute_metrics: excluding %d trade(s) with non-finite R "
-            "(corrupt exit data) — first at %s", len(bad), bad[0].entry_time)
-        resolved = [t for t in resolved if np.isfinite(t.r_multiple)]
     if not resolved:
         return Metrics(0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, {}, {}, {})
 
@@ -1957,9 +1872,7 @@ def compute_metrics(trades: List[Trade]) -> Metrics:
     gross_loss = float(-rs[losses_mask].sum()) if losses_mask.any() else 0.0
     pf = gross_profit / gross_loss if gross_loss > 0 else float("inf")
 
-    # Prepend the flat starting equity so an opening losing streak counts
-    # as drawdown from 0, not from the first trade's already-negative cum.
-    cum = np.concatenate(([0.0], np.cumsum(rs)))
+    cum = np.cumsum(rs)
     high_water = np.maximum.accumulate(cum)
     drawdown = high_water - cum
     max_dd = float(drawdown.max()) if len(drawdown) else 0.0
@@ -2228,14 +2141,6 @@ def _parse_args(argv: Optional[List[str]] = None) -> argparse.Namespace:
     p.add_argument("--max-hold-bars", type=int, default=300)
     p.add_argument("--swing-sr-lookback", type=int, default=0,
                    help="Override TechnicalConfig.swing_sr_lookback (0 = use config default)")
-    p.add_argument("--trail-arm", type=float, default=None,
-                   help="Override signals.trail_arm_r for this run (research knob)")
-    p.add_argument("--trail-dist", type=float, default=None,
-                   help="Override signals.trail_dist_r for this run (research knob)")
-    p.add_argument("--fixed-rr-tp", type=float, default=None,
-                   help="RESEARCH: exit at a hard TP of MULT x risk (e.g. 2.0 = "
-                        "fixed 1:2 RR) instead of the trailing stop; "
-                        "entries/stops/gates unchanged.")
     p.add_argument("--quality", default="A+,A,B,C", help="Comma-separated allowed grades")
     p.add_argument("--strategies", default="",
                    help="Comma-separated strategy tags to include (XAU,XAU_PB,XAU_CP,GENERIC). Empty = all.")
@@ -2245,19 +2150,6 @@ def _parse_args(argv: Optional[List[str]] = None) -> argparse.Namespace:
     p.add_argument("--cp-patterns", default=None,
                    help="Comma-separated pattern subset for --cp (e.g. "
                         "double_bottom,bull_flag). Default: all detectors.")
-    p.add_argument("--big-candle", type=float, default=None,
-                   help="Force xau_strategy.big_candle_atr_multiple=K for this "
-                        "run (no entries while the lookback window contains a "
-                        "bar with range >= K x prior ATR) — for validating the "
-                        "candidate.")
-    p.add_argument("--big-candle-hours", type=float, default=None,
-                   help="Override big_candle_lookback_hours for --big-candle "
-                        "runs (default 24).")
-    p.add_argument("--set", action="append", default=[], metavar="PATH=JSON",
-                   help="Override any PlatformConfig leaf by dotted path; the "
-                        "value is JSON-parsed for typing (research knob; "
-                        "repeatable). e.g. --set xau_strategy.m5_min_abs_score=25 "
-                        "--set 'xau_strategy.sell_min_quality=[\"A+\",\"A\",\"B\",\"C\"]'")
     p.add_argument("--no-cache", action="store_true", help="Skip parquet cache (force API fetch)")
     p.add_argument("--spread", type=float, default=None,
                    help="Round-trip cost in price units per trade (default: pair's "
@@ -2289,39 +2181,6 @@ def _resolve_tf(s: str) -> Timeframe:
     raise SystemExit(f"Unknown timeframe '{s}'. Valid: {[t.value for t in Timeframe]}")
 
 
-def _apply_config_overrides(cfg: PlatformConfig, overrides: List[str]) -> None:
-    """Apply repeatable --set PATH=JSON research overrides onto the config.
-
-    Unknown paths are a hard error: dataclass instances silently accept
-    setattr of any name, so a typo'd path would otherwise produce a
-    no-op run that looks like 'the relaxation had no effect'. Every
-    applied override is logged — the run log is the config audit trail
-    required by instructions_for_backtest.txt.
-    """
-    for item in overrides:
-        path, sep, raw = item.partition("=")
-        if not sep:
-            raise SystemExit(f"--set expects PATH=VALUE, got '{item}'")
-        try:
-            value = json.loads(raw)
-        except json.JSONDecodeError:
-            value = raw  # bare-string convenience ("oanda" etc.)
-        obj = cfg
-        parts = path.strip().split(".")
-        for attr in parts[:-1]:
-            if not hasattr(obj, attr):
-                raise SystemExit(f"--set: unknown config path '{path}'")
-            obj = getattr(obj, attr)
-        leaf = parts[-1]
-        if not hasattr(obj, leaf):
-            raise SystemExit(f"--set: unknown config path '{path}'")
-        old = getattr(obj, leaf)
-        if hasattr(old, "__dataclass_fields__"):
-            raise SystemExit(f"--set: '{path}' is a config section, not a leaf")
-        setattr(obj, leaf, value)
-        logger.info(f"--set override: {path} = {value!r} (was {old!r})")
-
-
 def main(argv: Optional[List[str]] = None) -> int:
     args = _parse_args(argv)
 
@@ -2355,38 +2214,14 @@ def main(argv: Optional[List[str]] = None) -> int:
             primary_timeframe=primary_tf,
             confirmation_timeframes=confirm_tfs or [Timeframe.H1, Timeframe.H4],
         )
-        # The factory path applies the config.json overlay inside
-        # make_config; the fallback must match (live == backtest).
-        from app_config import apply_pair_overrides  # noqa: E402
-        apply_pair_overrides(platform_cfg)
-    if args.fixed_rr_tp is not None:
-        if args.fixed_rr_tp <= 0:
-            raise SystemExit("--fixed-rr-tp must be > 0")
-        if (args.trail_arm is not None or args.trail_dist is not None or
-                any(s.split("=", 1)[0].strip().startswith(("signals.trail",
-                                                           "signals.pb_trail"))
-                    for s in (args.set or []))):
-            raise SystemExit(
-                "--fixed-rr-tp conflicts with --trail-arm/--trail-dist/"
-                "--set signals.*trail* (the flag disables trailing entirely)")
     if args.swing_sr_lookback > 0:
         platform_cfg.technical.swing_sr_lookback = args.swing_sr_lookback
-    if args.trail_arm is not None:
-        platform_cfg.signals.trail_arm_r = args.trail_arm
-    if args.trail_dist is not None:
-        platform_cfg.signals.trail_dist_r = args.trail_dist
     if args.cp:
         platform_cfg.xau_strategy.cp_enabled = True
         if args.cp_patterns:
             platform_cfg.xau_strategy.cp_patterns = [
                 s.strip() for s in args.cp_patterns.split(",") if s.strip()
             ]
-    if args.big_candle is not None:
-        platform_cfg.xau_strategy.big_candle_atr_multiple = args.big_candle
-    if args.big_candle_hours is not None:
-        platform_cfg.xau_strategy.big_candle_lookback_hours = args.big_candle_hours
-    if args.set:
-        _apply_config_overrides(platform_cfg, args.set)
     confirm_tfs = list(platform_cfg.confirmation_timeframes)
 
     end = (
@@ -2415,7 +2250,6 @@ def main(argv: Optional[List[str]] = None) -> int:
         strategies_only=[s.strip() for s in args.strategies.split(",") if s.strip()],
         spread=spread,
         workers=args.workers,
-        fixed_rr_tp=args.fixed_rr_tp or 0.0,
     )
 
     logger.info(

backtest/research/book_diff.py

@@ -1,86 +0,0 @@
-"""Trade-book diff: relaxed/gated run vs its control.
-
-Usage:
-    python backtest/research/book_diff.py CONTROL.csv CANDIDATE.csv BAR_MINUTES
-
-Decision numbers are RUN-LEVEL (ΔtotalR, ΔPF, ΔmaxDD, Δtrades) — never
-the added-trade bucket alone: the 5-bar entry cooldown means new
-entries displace existing ones. The added/removed decomposition and the
-per-decision-hour histogram of added trades are diagnostics.
-
-Shifted trades = removed+added pairs in the same direction within
-±5 bars (cooldown displacement, not a genuinely new/lost signal).
-"""
-import sys
-
-import pandas as pd
-
-
-def _metrics(df):
-    r = df["r_multiple"]
-    gains, losses = r[r > 0].sum(), -r[r <= 0].sum()
-    cum = r.cumsum()
-    return dict(trades=len(df), total_r=r.sum(),
-                pf=gains / max(losses, 1e-9),
-                wr=(r > 0).mean() * 100 if len(df) else 0.0,
-                max_dd=(cum.cummax() - cum).max() if len(cum) else 0.0)
-
-
-def diff_books(control_csv, candidate_csv, bar_minutes, split="2026-01-29"):
-    ctrl, cand = pd.read_csv(control_csv), pd.read_csv(candidate_csv)
-    for df in (ctrl, cand):
-        df["_t"] = pd.to_datetime(df["entry_time"], utc=True, format="mixed")
-        df["_key"] = df["entry_time"].astype(str) + "|" + df["direction"]
-
-    added = cand[~cand["_key"].isin(ctrl["_key"])].copy()
-    removed = ctrl[~ctrl["_key"].isin(cand["_key"])].copy()
-
-    # cooldown displacement: same-direction removed/added within ±5 bars
-    tol = pd.Timedelta(minutes=5 * bar_minutes)
-    shifted = 0
-    rem_pool = removed.copy()
-    for _, a in added.iterrows():
-        m = rem_pool[(rem_pool["direction"] == a["direction"])
-                     & ((rem_pool["_t"] - a["_t"]).abs() <= tol)]
-        if len(m):
-            shifted += 1
-            rem_pool = rem_pool.drop(m.index[0])
-
-    mc, mk = _metrics(ctrl), _metrics(cand)
-    split_ts = pd.Timestamp(split, tz="UTC")
-    halves = (cand[cand["_t"] < split_ts]["r_multiple"].sum(),
-              cand[cand["_t"] >= split_ts]["r_multiple"].sum())
-    hour_hist = None
-    if len(added):
-        added["dec_hour"] = (added["_t"]
-                             - pd.Timedelta(minutes=bar_minutes)).dt.hour
-        hour_hist = added.groupby("dec_hour")["r_multiple"] \
-                         .agg(["count", "sum"]).round(2)
-    return dict(
-        control=mc, candidate=mk,
-        delta_r=mk["total_r"] - mc["total_r"],
-        delta_pf=mk["pf"] - mc["pf"],
-        delta_dd=mk["max_dd"] - mc["max_dd"],
-        delta_trades=mk["trades"] - mc["trades"],
-        added_n=len(added), added_r=added["r_multiple"].sum(),
-        removed_n=len(removed), removed_r=removed["r_multiple"].sum(),
-        shifted=shifted, halves=halves, added_by_hour=hour_hist,
-    )
-
-
-if __name__ == "__main__":
-    d = diff_books(sys.argv[1], sys.argv[2], int(sys.argv[3]))
-    c, k = d["control"], d["candidate"]
-    print(f"control  : {c['trades']:4d} trades  {c['total_r']:+7.1f}R  "
-          f"PF {c['pf']:.2f}  WR {c['wr']:.1f}%  DD {c['max_dd']:.1f}")
-    print(f"candidate: {k['trades']:4d} trades  {k['total_r']:+7.1f}R  "
-          f"PF {k['pf']:.2f}  WR {k['wr']:.1f}%  DD {k['max_dd']:.1f}")
-    print(f"ΔR {d['delta_r']:+.1f}  ΔPF {d['delta_pf']:+.2f}  "
-          f"ΔDD {d['delta_dd']:+.1f}  Δtrades {d['delta_trades']:+d}")
-    print(f"added {d['added_n']} ({d['added_r']:+.1f}R)  "
-          f"removed {d['removed_n']} ({d['removed_r']:+.1f}R)  "
-          f"shifted≈{d['shifted']}")
-    print(f"candidate halves: H1 {d['halves'][0]:+.1f}R  H2 {d['halves'][1]:+.1f}R")
-    if d["added_by_hour"] is not None:
-        print("\nadded trades by decision hour:")
-        print(d["added_by_hour"].to_string())

backtest/research/burst_profile.py

@@ -1,42 +0,0 @@
-"""Burst-candle frequency by UTC hour — the empirical "hedge fund hours".
-
-Usage:
-    python backtest/research/burst_profile.py PARQUET_GLOB
-
-Per UTC bar-close hour of the given bars: % of bars whose range is
->= K x the PRIOR bar's Wilder ATR(14), for K in {3,4,5}; mean bar
-range; and mean signed drift over the 6 bars following a >=4x burst
-(does price continue or snap back?). Analysis tooling only.
-"""
-import glob
-import sys
-
-import pandas as pd
-
-
-def burst_table(parquet_glob: str, bar_minutes: int = 5):
-    files = sorted(glob.glob(parquet_glob))
-    if not files:
-        raise SystemExit(f"no parquet matches {parquet_glob}")
-    df = pd.concat([pd.read_parquet(f) for f in files]).sort_index()
-    df = df[~df.index.duplicated(keep="last")]
-    h, l, c = df["High"], df["Low"], df["Close"]
-    tr = pd.concat([h - l, (h - c.shift()).abs(), (l - c.shift()).abs()],
-                   axis=1).max(axis=1)
-    atr = tr.ewm(alpha=1 / 14, min_periods=14, adjust=False).mean()
-    ratio = (h - l) / atr.shift(1)
-    close_hour = (df.index + pd.Timedelta(minutes=bar_minutes)).hour
-    fwd6 = c.shift(-6) - c  # signed move over the 6 bars after this one
-    out = pd.DataFrame({
-        "bars": ratio.groupby(close_hour).size(),
-        **{f"pct_ge_{k}x": (ratio >= k).groupby(close_hour).mean() * 100
-           for k in (3, 4, 5)},
-        "mean_range": (h - l).groupby(close_hour).mean(),
-        "post4x_drift6": fwd6[ratio >= 4].groupby(close_hour[ratio >= 4]).mean(),
-    })
-    return out.round(3)
-
-
-if __name__ == "__main__":
-    print(burst_table(sys.argv[1],
-                      int(sys.argv[2]) if len(sys.argv) > 2 else 5).to_string())

backtest/research/calendar_store.py

@@ -0,0 +1,94 @@
+"""Load and query the historical economic calendar (backtest side).
+
+The store is built by `ff_calendar_scraper.py`. Timestamps are unix
+epoch (UTC) — verified against known release times and price-volatility
+alignment (see scraper docstring).
+
+Key API:
+    cal = load_calendar()                          # full DataFrame, `utc` column
+    ev  = precise_events(cal, ["USD"])             # high-impact, minute-accurate
+    mask = event_window_mask(bar_index, ev.utc, before, after)
+"""
+from pathlib import Path
+from typing import Iterable, Optional
+
+import numpy as np
+import pandas as pd
+
+PARQUET = Path(__file__).resolve().parent.parent / "cache" / "calendar.parquet"
+
+# timeLabel values that mean "no intraday time" — these events cannot
+# anchor an M5/M15 window (German Prelim CPI is the notable High one).
+_UNTIMED = {"All Day", "Tentative", ""}
+
+
+def _parse_release_value(raw) -> float:
+    """'0.3%' / '-1.7%' / '228K' / '1.1M' -> float (nan if unparseable)."""
+    import re
+    if raw is None or (isinstance(raw, float) and np.isnan(raw)):
+        return np.nan
+    m = re.search(r"(-?\d+(?:\.\d+)?)\s*([kmbt])?",
+                  str(raw).replace(",", ""), re.IGNORECASE)
+    if not m:
+        return np.nan
+    mult = {"k": 1e3, "m": 1e6, "b": 1e9, "t": 1e12}.get(
+        (m.group(2) or "").lower(), 1.0)
+    return float(m.group(1)) * mult
+
+
+def load_calendar(path: Path = PARQUET) -> pd.DataFrame:
+    df = pd.read_parquet(path)
+    df["utc"] = pd.to_datetime(df.dateline, unit="s", utc=True)
+    for col in ("actual", "forecast", "previous"):
+        df[f"{col}_num"] = df[col].map(_parse_release_value)
+    df["surprise"] = df.actual_num - df.forecast_num
+    return df
+
+
+def precise_events(
+    cal: pd.DataFrame,
+    currencies: Iterable[str] = ("USD",),
+    impacts: Iterable[str] = ("high",),
+    require_values: bool = False,
+) -> pd.DataFrame:
+    """Events with a minute-accurate timestamp, filtered by currency/impact."""
+    out = cal[cal.currency.isin(list(currencies))
+              & cal.impactName.isin(list(impacts))
+              & ~cal.timeMasked.astype(bool)
+              & ~cal.timeLabel.fillna("").isin(_UNTIMED)]
+    if require_values:
+        out = out.dropna(subset=["actual_num", "forecast_num"])
+    return out.sort_values("utc").reset_index(drop=True)
+
+
+def event_window_mask(
+    bar_index: pd.DatetimeIndex,
+    event_times: pd.Series,
+    before: pd.Timedelta = pd.Timedelta(minutes=30),
+    after: pd.Timedelta = pd.Timedelta(minutes=30),
+) -> np.ndarray:
+    """Boolean mask: bar OPEN time falls inside [event-before, event+after).
+
+    Use on a backtest's entry times to identify trades entered around
+    releases (avoidance counterfactuals, event-window cost models).
+    """
+    starts = pd.DatetimeIndex(event_times) - before
+    ends = pd.DatetimeIndex(event_times) + after
+    order = np.argsort(starts.values)
+    starts, ends = starts.values[order], ends.values[order]
+    # merge overlapping windows, then interval-stab each bar time
+    merged_s, merged_e = [], []
+    for s, e in zip(starts, ends):
+        if merged_e and s <= merged_e[-1]:
+            merged_e[-1] = max(merged_e[-1], e)
+        else:
+            merged_s.append(s)
+            merged_e.append(e)
+    ms = np.array(merged_s)
+    me = np.array(merged_e)
+    ts = bar_index.values
+    pos = np.searchsorted(ms, ts, side="right") - 1
+    ok = pos >= 0
+    res = np.zeros(len(ts), dtype=bool)
+    res[ok] = ts[ok] < me[pos[ok]]
+    return res

backtest/research/commit_ladder.py

@@ -1,178 +0,0 @@
-"""Commit-ladder campaign driver (2026-08).
-
-Question: of all signal-code states shipped in the last two months, which
-configuration is the most accurate? Historical in-era backtest numbers are
-not comparable (the backtester's realism model changed on 07-08/07-16 and
-the data feed switched to Oanda on 07-28), so every historical
-configuration is recreated on TODAY's harness via --set overrides and
-replayed on the same pinned Oanda bars. Bug-fix commits stay on; the
-ladder compares shipped tuning decisions only.
-
-Usage:
-    python backtest/research/commit_ladder.py            # full matrix
-    python backtest/research/commit_ladder.py --tf 5min  # one timeframe
-    python backtest/research/commit_ladder.py --windows 12mo,oos
-
-Resumable: a run whose CSV already exists in the output dir is skipped.
-Outputs: backtest/research/commit_ladder_2026-08/<variant>_<window>.csv
-         + .log per run, summary.csv, and a printed ranking table.
-"""
-from __future__ import annotations
-
-import argparse
-import subprocess
-import sys
-from pathlib import Path
-
-import pandas as pd
-
-ROOT = Path(__file__).resolve().parents[2]
-BT = ROOT / "backtest" / "gpu_backtest.py"
-OUT = ROOT / "backtest" / "research" / "commit_ladder_2026-08"
-
-WINDOWS = {
-    "12mo": ("2025-07-30", "2026-07-30"),
-    "oos": ("2024-06-01", "2025-03-30"),
-    "5yr": ("2021-07-30", "2026-07-30"),
-}
-SPLIT_12MO = pd.Timestamp("2026-01-29", tz="UTC")  # book_diff.py's split
-
-# (variant, "commit date what-it-represents", [--set overrides])
-VARIANTS: dict[str, list[tuple[str, str, list[str]]]] = {
-    "5min": [
-        ("M5-A", "dad2432 07-09 gates+trailing+pullback",
-         ["xau_strategy.blocked_hours_utc=[0,1,2,10,11,13,16,21]",
-          "xau_strategy.cp_enabled=false",
-          "signals.trail_dist_r=0.75",
-          "xau_strategy.m5_min_abs_score=30",
-          "xau_strategy.m5_min_confirmation_score=10"]),
-        ("M5-B", "f3ba4d5 07-11 +hour-9 block",
-         ["xau_strategy.blocked_hours_utc=[0,1,2,10,11,13,16,21,9]",
-          "xau_strategy.cp_enabled=false",
-          "signals.trail_dist_r=0.75",
-          "xau_strategy.m5_min_abs_score=30",
-          "xau_strategy.m5_min_confirmation_score=10"]),
-        ("M5-C", "de6424e 07-16 +chart patterns",
-         ["xau_strategy.blocked_hours_utc=[0,1,2,10,11,13,16,21,9]",
-          "signals.trail_dist_r=0.75",
-          "xau_strategy.m5_min_abs_score=30",
-          "xau_strategy.m5_min_confirmation_score=10"]),
-        ("M5-D", "3ee79b5 07-21 +trail dist 0.85",
-         ["xau_strategy.blocked_hours_utc=[0,1,2,10,11,13,16,21,9]",
-          "xau_strategy.m5_min_abs_score=30",
-          "xau_strategy.m5_min_confirmation_score=10"]),
-        ("M5-E", "0396970 07-30 relaxation (HEAD)", []),
-    ],
-    "15min": [
-        ("M15-A", "987ec5b 07-09 pre-campaigns",
-         ["signals.tp_floor_rr=0.5",
-          "xau_strategy.blocked_regimes=[]",
-          "xau_strategy.sell_blocked_hours_utc=[]",
-          "signals.trail_arm_r=0.60"]),
-        ("M15-B", "f3ba4d5/f88e39f 07-11 1R floor + regime block",
-         ["xau_strategy.sell_blocked_hours_utc=[]",
-          "signals.trail_arm_r=0.60"]),
-        ("M15-C", "7d3fd95 07-16 +SELL hour 15/17 block",
-         ["signals.trail_arm_r=0.60"]),
-        ("M15-D", "bf8e5fb 07-21 trail arm 0.70 (HEAD)", []),
-    ],
-    "1h": [
-        ("H1-A", "pre-d95d31a 07-08 TP-tightening model",
-         ["signals.min_rr_ratio=0.4",
-          "signals.tp_tightening_factor=0.8"]),
-        ("H1-B", "HEAD", []),
-    ],
-}
-
-
-def run_one(tf: str, name: str, overrides: list[str], wkey: str,
-            spread: str | None = None) -> Path:
-    suffix = f"_s{spread}" if spread else ""
-    csv_path = OUT / f"{name}_{wkey}{suffix}.csv"
-    log_path = OUT / f"{name}_{wkey}{suffix}.log"
-    if csv_path.exists():
-        print(f"  [skip] {csv_path.name} exists")
-        return csv_path
-    start, end = WINDOWS[wkey]
-    cmd = [sys.executable, str(BT), "--pair", "XAUUSD",
-           "--primary-tf", tf, "--start", start, "--end", end,
-           "--export-csv", str(csv_path)]
-    if spread:
-        cmd += ["--spread", spread]
-    for ov in overrides:
-        cmd += ["--set", ov]
-    print(f"  [run ] {csv_path.name} ...", flush=True)
-    with open(log_path, "w") as lf:
-        rc = subprocess.run(cmd, stdout=lf, stderr=subprocess.STDOUT,
-                            cwd=ROOT).returncode
-    if rc != 0:
-        raise SystemExit(f"backtest failed (rc={rc}) — see {log_path}")
-    log = log_path.read_text()
-    n_ov = log.count("--set override")
-    if n_ov != len(overrides):
-        raise SystemExit(
-            f"{name}_{wkey}: expected {len(overrides)} --set overrides, "
-            f"log shows {n_ov} — see {log_path}")
-    if "Loading cached" not in log and wkey != "sanity":
-        print(f"  [warn] {log_path.name}: no 'Loading cached' line — "
-              f"data was fetched live for at least one timeframe")
-    return csv_path
-
-
-def metrics(csv_path: Path, wkey: str) -> dict:
-    df = pd.read_csv(csv_path, parse_dates=["entry_time"])
-    if df.empty:
-        return {"trades": 0, "total_r": 0.0, "wr": 0.0, "pf": 0.0,
-                "max_dd": 0.0, "h1_r": 0.0, "h2_r": 0.0, "years_pos": ""}
-    if df["entry_time"].dt.tz is None:
-        df["entry_time"] = df["entry_time"].dt.tz_localize("UTC")
-    r = df["r_multiple"]
-    pos, neg = r[r > 0].sum(), r[r < 0].sum()
-    cum = r.cumsum()
-    dd = (pd.concat([pd.Series([0.0]), cum]).cummax()[1:] - cum).max()
-    out = {
-        "trades": len(df),
-        "total_r": round(r.sum(), 1),
-        "wr": round((r > 0).mean() * 100, 1),
-        "pf": round(pos / abs(neg), 2) if neg else float("inf"),
-        "max_dd": round(dd, 1),
-        "h1_r": "", "h2_r": "", "years_pos": "",
-    }
-    if wkey == "12mo":
-        h1 = r[df["entry_time"] < SPLIT_12MO].sum()
-        out["h1_r"] = round(h1, 1)
-        out["h2_r"] = round(r.sum() - h1, 1)
-    if wkey == "5yr":
-        by_year = r.groupby(df["entry_time"].dt.year).sum()
-        out["years_pos"] = f"{(by_year > 0).sum()}/{len(by_year)}"
-    return out
-
-
-def main() -> None:
-    ap = argparse.ArgumentParser()
-    ap.add_argument("--tf", choices=list(VARIANTS), default=None)
-    ap.add_argument("--windows", default="12mo,oos,5yr")
-    args = ap.parse_args()
-    OUT.mkdir(parents=True, exist_ok=True)
-    wkeys = [w.strip() for w in args.windows.split(",")]
-    tfs = [args.tf] if args.tf else list(VARIANTS)
-
-    rows = []
-    for tf in tfs:
-        for name, label, overrides in VARIANTS[tf]:
-            for wkey in wkeys:
-                csv_path = run_one(tf, name, overrides, wkey)
-                m = metrics(csv_path, wkey)
-                rows.append({"tf": tf, "variant": name, "represents": label,
-                             "window": wkey, **m})
-                print(f"    {name} {wkey}: {m['trades']} trades, "
-                      f"{m['total_r']}R, PF {m['pf']}, WR {m['wr']}%")
-
-    summary = pd.DataFrame(rows)
-    summary.to_csv(OUT / "summary.csv", index=False)
-    print("\n=== SUMMARY (also in summary.csv) ===")
-    print(summary.to_string(index=False))
-
-
-if __name__ == "__main__":
-    main()

backtest/research/duka_spread.py

@@ -1,194 +0,0 @@
-"""Dukascopy XAU/USD tick download + bid/ask spread study.
-
-Validates ForexBot2's $0.30 round-trip cost assumption (config:
-SignalConfig.round_trip_cost / PAIR_INFO default_spread). Round-trip
-cost of a market entry+exit = one ask-bid spread width (buy at ask,
-sell at bid), so the assumption holds iff typical spread ~ $0.30.
-
-bi5 format: LZMA 'alone' stream of 20-byte big-endian records
-(ms_in_hour uint32, ask uint32, bid uint32, ask_vol f32, bid_vol f32);
-XAUUSD prices scaled x1000. URL months are 0-indexed.
-"""
-import lzma
-import struct
-import sys
-from concurrent.futures import ThreadPoolExecutor
-from datetime import date, timedelta
-from pathlib import Path
-
-import numpy as np
-import pandas as pd
-import requests
-
-SCALE = 1000.0
-CACHE = Path(__file__).parent / "tick_cache"
-CACHE.mkdir(exist_ok=True)
-URL = "https://datafeed.dukascopy.com/datafeed/XAUUSD/{y}/{m:02d}/{d:02d}/{h:02d}h_ticks.bi5"
-
-# Bot's tradable bar-close hours: session 03:00-23:00 minus XAU blocked
-# hours [0,1,2,10,11,13,16,21] (M5 additionally blocks 9 — reported
-# separately below).
-BLOCKED = {0, 1, 2, 10, 11, 13, 16, 21}
-TRADABLE = sorted(set(range(3, 23)) - BLOCKED)
-
-# ── Day sample ────────────────────────────────────────────────────────
-recent = [date(2026, 7, 6) + timedelta(days=i) for i in range(17)]
-recent = [d for d in recent if d.weekday() < 5]              # Mon-Fri Jul 6-22
-monthly = [date(2025, 8, 13), date(2025, 9, 10), date(2025, 10, 15),
-           date(2025, 11, 12), date(2025, 12, 10), date(2026, 1, 14),
-           date(2026, 2, 11), date(2026, 3, 11), date(2026, 4, 15),
-           date(2026, 5, 13), date(2026, 6, 10)]             # mid-month Wednesdays
-nfp = [date(2026, 6, 5), date(2026, 7, 3)]                   # NFP Fridays
-sundays = [date(2026, 7, 12), date(2026, 7, 19)]             # Sun 22-24 open
-DAYS = sorted(set(recent + monthly + nfp + sundays))
-assert all(d.weekday() < 5 for d in recent + monthly + nfp)
-
-
-_tls = __import__("threading").local()
-
-
-def _session() -> requests.Session:
-    if not hasattr(_tls, "s"):
-        _tls.s = requests.Session()
-        _tls.s.headers["User-Agent"] = "Mozilla/5.0 (research; spread-study)"
-    return _tls.s
-
-
-def fetch(day: date, hour: int) -> Path:
-    p = CACHE / f"{day.isoformat()}_{hour:02d}.bi5"
-    if p.exists():
-        return p
-    u = URL.format(y=day.year, m=day.month - 1, d=day.day, h=hour)
-    last = None
-    for attempt in range(4):
-        try:
-            r = _session().get(u, timeout=30)
-            if r.status_code != 200:
-                raise RuntimeError(f"{u} -> HTTP {r.status_code}")
-            p.write_bytes(r.content)
-            return p
-        except (requests.ConnectionError, requests.Timeout, RuntimeError) as exc:
-            last = exc
-            __import__("time").sleep(1.5 * (attempt + 1))
-            if hasattr(_tls, "s"):
-                _tls.s.close()
-                del _tls.s
-    raise RuntimeError(f"giving up on {u}: {last}")
-
-
-def decode(p: Path, day: date, hour: int) -> pd.DataFrame:
-    blob = p.read_bytes()
-    if not blob:
-        return pd.DataFrame()
-    raw = lzma.decompress(blob)
-    n = len(raw) // 20
-    a = np.frombuffer(raw[: n * 20], dtype=">u4").reshape(n, 5)
-    f = np.frombuffer(raw[: n * 20], dtype=">f4").reshape(n, 5)
-    df = pd.DataFrame({
-        "ms": a[:, 0].astype(np.int64),
-        "ask": a[:, 1] / SCALE,
-        "bid": a[:, 2] / SCALE,
-    })
-    df["spread"] = df["ask"] - df["bid"]
-    df["date"], df["hour"] = day.isoformat(), hour
-    df["weekday"] = day.weekday()
-    return df
-
-
-def load_all() -> pd.DataFrame:
-    jobs = [(d, h) for d in DAYS for h in range(24)]
-    with ThreadPoolExecutor(max_workers=3) as ex:
-        paths = list(ex.map(lambda j: fetch(*j), jobs))
-    frames = [decode(p, d, h) for p, (d, h) in zip(paths, jobs)]
-    df = pd.concat([f for f in frames if not f.empty], ignore_index=True)
-    return df
-
-
-def wstats(g: pd.DataFrame) -> pd.Series:
-    """Tick stats + time-weighted mean (each tick's spread held until the
-    next tick; last tick held to hour end)."""
-    tw_num = tw_den = 0.0
-    keys = [k for k in ("date", "hour") if k in g.columns]
-    for _, grp in g.groupby(keys, sort=False):
-        ms = grp["ms"].to_numpy()
-        sp = grp["spread"].to_numpy()
-        w = np.diff(np.append(ms, 3_600_000)).clip(min=0)
-        tw_num += float((sp * w).sum())
-        tw_den += float(w.sum())
-    s = g["spread"]
-    return pd.Series({
-        "ticks": len(s),
-        "tw_mean": tw_num / tw_den if tw_den else np.nan,
-        "mean": s.mean(), "median": s.median(),
-        "p75": s.quantile(0.75), "p90": s.quantile(0.90),
-        "p99": s.quantile(0.99), "max": s.max(),
-        "pct_le_030": (s <= 0.30).mean() * 100,
-        "pct_ge_060": (s >= 0.60).mean() * 100,
-    })
-
-
-def m5_boundary_spreads(df: pd.DataFrame) -> pd.DataFrame:
-    """Spread of the FIRST tick at/after each M5 boundary — what a
-    next-open fill actually pays."""
-    rows = []
-    for (day, hour), grp in df.groupby(["date", "hour"], sort=False):
-        ms = grp["ms"].to_numpy()
-        sp = grp["spread"].to_numpy()
-        wd = grp["weekday"].iat[0]
-        for b in range(0, 3_600_000, 300_000):
-            i = np.searchsorted(ms, b)
-            if i < len(ms) and ms[i] - b <= 30_000:   # tick within 30 s
-                rows.append((day, wd, hour, b // 60_000, sp[i]))
-    return pd.DataFrame(rows, columns=["date", "weekday", "hour", "minute", "spread"])
-
-
-def main() -> None:
-    df = load_all()
-    print(f"days={df['date'].nunique()}  ticks={len(df):,}  "
-          f"price range {df['bid'].min():.0f}-{df['ask'].max():.0f}")
-
-    print("\n── Overall (all sampled hours) ──")
-    print(wstats(df).round(3).to_string())
-
-    trad = df[df["hour"].isin(TRADABLE) & (df["weekday"] < 5)]
-    print("\n── Bot-tradable hours (session 03-23 minus blocked, Mon-Fri) ──")
-    print(wstats(trad).round(3).to_string())
-
-    roll = df[df["hour"].isin([21, 22, 23])]
-    print("\n── Rollover window 21-24 UTC ──")
-    print(wstats(roll).round(3).to_string())
-
-    print("\n── Per UTC hour ──")
-    per_hour = df[df["weekday"] < 5].groupby("hour").apply(wstats, include_groups=False).round(3)
-    per_hour["tradable"] = ["Y" if h in TRADABLE else "-" for h in per_hour.index]
-    print(per_hour.to_string())
-    per_hour.to_csv(Path(__file__).parent / "duka_spread_by_hour.csv")
-
-    print("\n── M5 next-open boundary fills ──")
-    mb = m5_boundary_spreads(df)
-    mb_trad = mb[mb["hour"].isin(TRADABLE) & (mb["weekday"] < 5)]
-    for name, d in [("all hours", mb), ("tradable hours", mb_trad)]:
-        s = d["spread"]
-        print(f"{name}: n={len(s):,} mean={s.mean():.3f} median={s.median():.3f} "
-              f"p90={s.quantile(0.9):.3f} p99={s.quantile(0.99):.3f} "
-              f"%<=0.30={(s <= 0.30).mean()*100:.1f} %>=0.60={(s >= 0.60).mean()*100:.1f}")
-    mb.to_csv(Path(__file__).parent / "duka_m5_boundary_spreads.csv", index=False)
-
-    print("\n── By month (tradable hours, tick median / tw_mean) ──")
-    trad2 = trad.copy()
-    trad2["month"] = trad2["date"].str[:7]
-    bym = trad2.groupby("month").apply(wstats, include_groups=False).round(3)
-    print(bym[["ticks", "tw_mean", "median", "p90", "pct_le_030", "pct_ge_060"]].to_string())
-
-    print("\n── NFP Fridays 12:00-15:00 UTC vs same hours other days ──")
-    nfp_days = {"2026-06-05", "2026-07-03"}
-    win = df[df["hour"].isin([12, 13, 14])]
-    for name, d in [("NFP", win[win["date"].isin(nfp_days)]),
-                    ("non-NFP", win[~win["date"].isin(nfp_days) & (win["weekday"] < 5)])]:
-        s = d["spread"]
-        print(f"{name}: n={len(s):,} tick-median={s.median():.3f} p90={s.quantile(0.9):.3f} "
-              f"p99={s.quantile(0.99):.3f} max={s.max():.2f}")
-
-
-if __name__ == "__main__":
-    sys.exit(main())

backtest/research/duka_spread_by_hour.csv

@@ -1,25 +0,0 @@
-hour,ticks,tw_mean,mean,median,p75,p90,p99,max,pct_le_030,pct_ge_060,tradable
-0,362893.0,0.749,0.768,0.79,0.82,0.86,1.1,7.69,0.053,94.561,-
-1,540182.0,0.735,0.742,0.74,0.81,0.87,1.0,5.59,0.073,90.449,-
-2,431589.0,0.716,0.722,0.73,0.8,0.86,0.95,2.96,0.1,85.27,-
-3,343952.0,0.708,0.711,0.71,0.79,0.84,0.95,1.67,0.081,82.939,Y
-4,246262.0,0.704,0.702,0.71,0.78,0.82,0.94,1.86,0.098,82.627,Y
-5,388365.0,0.712,0.716,0.72,0.79,0.82,0.95,2.17,0.076,86.745,Y
-6,368999.0,0.65,0.651,0.65,0.69,0.74,0.94,2.32,0.079,71.668,Y
-7,313149.0,0.601,0.598,0.59,0.65,0.69,0.994,3.08,0.239,42.904,Y
-8,347276.0,0.612,0.616,0.6,0.66,0.74,1.0,3.04,0.308,48.349,Y
-9,321619.0,0.612,0.615,0.6,0.66,0.72,1.0,5.95,0.288,50.644,Y
-10,295271.0,0.611,0.618,0.6,0.66,0.71,1.0,9.36,0.164,49.253,-
-11,336963.0,0.619,0.624,0.6,0.67,0.79,1.0,4.97,0.203,50.178,-
-12,476773.0,0.66,0.697,0.63,0.72,0.877,1.367,14.98,0.257,59.633,Y
-13,651425.0,0.641,0.648,0.63,0.69,0.787,1.487,11.407,0.296,60.297,-
-14,673391.0,0.625,0.625,0.63,0.68,0.77,0.9,11.98,0.458,59.085,Y
-15,598398.0,0.636,0.638,0.64,0.7,0.78,0.91,4.49,0.433,66.296,Y
-16,485023.0,0.648,0.644,0.64,0.7,0.757,0.917,4.09,0.118,68.496,-
-17,450119.0,0.626,0.628,0.627,0.7,0.726,0.84,4.08,0.154,63.337,Y
-18,411538.0,0.626,0.63,0.637,0.7,0.71,0.79,4.16,0.107,67.734,Y
-19,435396.0,0.638,0.643,0.65,0.7,0.72,0.896,5.804,0.11,72.007,Y
-20,212654.0,0.684,0.665,0.66,0.7,0.76,1.217,7.15,0.226,70.215,Y
-21,31319.0,0.627,0.622,0.61,0.68,0.75,1.02,2.08,0.089,53.153,-
-22,108879.0,0.829,0.834,0.797,0.84,0.98,2.05,10.11,0.056,95.434,Y
-23,161079.0,1.032,0.913,0.8,0.89,1.307,2.77,13.174,0.092,95.41,-

backtest/research/fetch_oanda_m15.py

@@ -1,158 +0,0 @@
-"""Fetch XAU/USD M15 candles from the Oanda v20 REST API.
-
-Mechanical data-fetching infrastructure only — no analysis here.
-Paginates GET /v3/instruments/XAU_USD/candles from a start cursor
-through "now" and writes the result to a parquet file.
-
-Run from the repo root:
-    venv/bin/python backtest/research/fetch_oanda_m15.py
-"""
-import sys
-import time
-from datetime import datetime, timedelta, timezone
-from pathlib import Path
-
-import pandas as pd
-import requests
-
-sys.path.insert(0, str(Path(__file__).resolve().parents[2]))
-
-from app_config import oanda_api_token, oanda_base_url  # noqa: E402
-
-INSTRUMENT = "XAU_USD"
-GRANULARITY = "M15"
-START = datetime(2025, 7, 1, 0, 0, 0, tzinfo=timezone.utc)
-OUT_PATH = Path(__file__).parent / "oanda_xau_m15_20250701_now.parquet"
-COUNT = 5000
-REQUEST_SLEEP_S = 0.5
-FRESHNESS_WINDOW = timedelta(minutes=30)
-
-
-def _rfc3339(dt: datetime) -> str:
-    return dt.astimezone(timezone.utc).strftime("%Y-%m-%dT%H:%M:%S.%fZ")
-
-
-def fetch_all_candles():
-    token = oanda_api_token()
-    base_url = oanda_base_url()
-    if not token:
-        raise RuntimeError(
-            "No Oanda API token resolved (env OANDA_API_TOKEN / "
-            "config.json oanda.api_token). Aborting.")
-
-    url = f"{base_url}/v3/instruments/{INSTRUMENT}/candles"
-    headers = {
-        "Authorization": f"Bearer {token}",
-        "Accept-Datetime-Format": "RFC3339",
-    }
-
-    cursor = START
-    include_first = True
-    all_rows = []
-    n_requests = 0
-
-    while True:
-        params = {
-            "granularity": GRANULARITY,
-            "price": "MBA",
-            "count": COUNT,
-            "dailyAlignment": 0,
-            "alignmentTimezone": "Etc/UTC",
-            "from": _rfc3339(cursor),
-            "includeFirst": "true" if include_first else "false",
-        }
-
-        resp = requests.get(url, headers=headers, params=params, timeout=30)
-        n_requests += 1
-        if resp.status_code != 200:
-            raise RuntimeError(
-                f"Oanda API request failed: status={resp.status_code}\n"
-                f"URL: {resp.url}\n"
-                f"Response body: {resp.text}")
-
-        payload = resp.json()
-        candles = payload.get("candles", [])
-
-        for c in candles:
-            if not c.get("complete", False):
-                continue
-            mid = c["mid"]
-            bid = c["bid"]
-            ask = c["ask"]
-            all_rows.append({
-                "Datetime": pd.Timestamp(c["time"]).tz_convert("UTC")
-                if pd.Timestamp(c["time"]).tzinfo is not None
-                else pd.Timestamp(c["time"]).tz_localize("UTC"),
-                "Open": float(mid["o"]),
-                "High": float(mid["h"]),
-                "Low": float(mid["l"]),
-                "Close": float(mid["c"]),
-                "Volume": int(c["volume"]),
-                "bid_o": float(bid["o"]),
-                "bid_h": float(bid["h"]),
-                "bid_l": float(bid["l"]),
-                "bid_c": float(bid["c"]),
-                "ask_o": float(ask["o"]),
-                "ask_h": float(ask["h"]),
-                "ask_l": float(ask["l"]),
-                "ask_c": float(ask["c"]),
-            })
-
-        print(f"  request {n_requests}: from={params['from']} "
-              f"got {len(candles)} candles (complete kept so far: "
-              f"{len(all_rows)})", file=sys.stderr)
-
-        if not candles:
-            break
-
-        last_time = pd.Timestamp(candles[-1]["time"])
-        if last_time.tzinfo is None:
-            last_time = last_time.tz_localize("UTC")
-        else:
-            last_time = last_time.tz_convert("UTC")
-
-        now = datetime.now(timezone.utc)
-        # Oanda's "count" includes the "from" boundary bar itself, so with
-        # includeFirst=false every full page comes back as count-1, not
-        # count (verified empirically: count=N, includeFirst=false always
-        # yields N-1 candles when a full page of data exists ahead of the
-        # cursor). Compare against the count actually expected for this
-        # request's includeFirst setting, not the raw requested count.
-        expected_full_page = COUNT if include_first else COUNT - 1
-        got_fewer_than_requested = len(candles) < expected_full_page
-        near_now = (now - last_time.to_pydatetime()) < FRESHNESS_WINDOW
-
-        if got_fewer_than_requested or near_now:
-            break
-
-        cursor = last_time.to_pydatetime()
-        include_first = False
-        time.sleep(REQUEST_SLEEP_S)
-
-    return all_rows, n_requests
-
-
-def main():
-    print(f"Fetching {INSTRUMENT} {GRANULARITY} candles from "
-          f"{_rfc3339(START)} through now...", file=sys.stderr)
-
-    rows, n_requests = fetch_all_candles()
-
-    if not rows:
-        raise RuntimeError("No complete candles were fetched.")
-
-    df = pd.DataFrame(rows)
-    df = df.set_index("Datetime")
-    df = df.sort_index()
-    df = df[~df.index.duplicated(keep="last")]
-
-    df.to_parquet(OUT_PATH)
-
-    print(f"\nSaved {len(df)} rows to {OUT_PATH}", file=sys.stderr)
-    print(f"Requests made: {n_requests}", file=sys.stderr)
-    print(f"First timestamp: {df.index[0]}", file=sys.stderr)
-    print(f"Last timestamp: {df.index[-1]}", file=sys.stderr)
-
-
-if __name__ == "__main__":
-    main()

backtest/research/fixedtp_2026-08-05.md

@@ -1,123 +0,0 @@
-# Fixed-RR hard-TP vs shipped trailing exit — research campaign 2026-08-05
-
-**Verdict: DO NOT SHIP. The trailing exit stays.** The fixed 2R target beats the trail on
-total R in every 2024+ window but the edge is a trend-regime artifact: it inverts in the
-2021–2023 ranging years, doubles drawdown, cuts win rate 10–19 points, and doubles hold
-times (amplifying the accepted live one-position divergence). Details below.
-
-## Setup
-
-- Code: HEAD `fc5ce85` + the `--fixed-rr-tp` research flag (uncommitted at run time).
-  The flag replaces every trade's TP with `mult × |entry − sl|` AFTER re-anchoring and
-  disables trailing (run-level and per-trade PB params) — entries, stops, gates, and the
-  cooldown pass are untouched, so both arms replay **trade-identical books** (verified:
-  paired join on entry_time+direction matches 1:1 in every window). Only the exit policy
-  differs → per-trade paired comparison.
-- Provider: Oanda (cache-warm; keys `oanda_XAU_USD_{5min,15min,1h}_{20250603_20260804,
-  20240331_20250330,20210531_20260804}`). Charged cost `--spread 0.65` (real measured
-  round trip); $0.30 letter-cost books derived analytically as `r + 0.35/risk` and
-  verified exact against one real `--spread 0.30` run (max per-trade error 1.8e-15 —
-  spread never enters trigger detection, per the owner-declined 2026-08-04 decision).
-- Control sanity gate: fresh M5 12mo control reproduced the pinned HEAD reference
-  exactly — 555 trades, +98.64R, PF 1.483, maxDD −10.73R.
-- Commands (control = same without the flag):
-  `venv/bin/python backtest/gpu_backtest.py --pair XAUUSD --primary-tf {5min,15min}
-   --start 2025-08-04 --end 2026-08-04 --spread 0.65 [--fixed-rr-tp 2.0]
-   --export-csv backtest/fixedtp_*.csv`
-  OOS `--start 2024-06-01 --end 2025-03-30`; 5yr `--start 2021-08-01 --end 2026-08-04`.
-- Books pinned as `backtest/fixedtp_{m5,m15}_{12mo,oos,5yr}_{ctrl,rr20,...}.csv`
-  (per-trade indicator breakdown, agreement counts, regime, vol-regime, weekday,
-  MAE/MFE, bars-held — the full instructions_for_backtest record).
-
-## Headline (@ $0.65)
-
-| Window | Arm | n | R | PF | WR | maxDD | recov | medBars | exits |
-|---|---|---|---|---|---|---|---|---|---|
-| M5 12mo | trail | 555 | +98.64 | 1.483 | 52.4% | −10.7 | 9.2 | 28 | TRAIL 361 / SL 183 / TO 11 |
-| M5 12mo | 2R hard | 555 | +124.85 | 1.378 | 42.9% | −24.5 | 5.1 | 50 | SL 312 / TP1 210 / TO 33 |
-| M15 12mo | trail | 397 | +73.93 | 1.525 | 62.2% | −10.8 | 6.8 | 28 | TRAIL 255 / SL 135 / TO 7 |
-| M15 12mo | 2R hard | 397 | +95.95 | 1.431 | 44.3% | −11.2 | 8.6 | 39 | SL 215 / TP1 151 / TO 31 |
-| M5 OOS | trail | 86 | +9.45 | 1.332 | 53.5% | −8.2 | 1.2 | 70 | — |
-| M5 OOS | 2R hard | 86 | +18.83 | 1.407 | 45.3% | −9.7 | 1.9 | 124 | — |
-| M15 OOS | trail | 299 | +38.77 | 1.360 | 59.2% | −15.5 | 2.5 | 38 | — |
-| M15 OOS | 2R hard | 299 | +72.14 | 1.435 | 45.2% | −13.9 | 5.2 | 82 | — |
-
-Halves (12mo, split 2026-02-03): M5 rr20 +67.99/+56.86 (both positive; H2 PF 1.325 vs
-ctrl 1.503), M15 rr20 +73.74/+22.21 (both positive; ctrl H2 +7.23 — the known H2-flat).
-At $0.30 all deltas keep the same sign (e.g. M5 12mo +109.0 vs +135.2).
-
-So the **letter** of rule 1 passes for rr20 on both TFs. The 5yr decomposition is why it
-still must not ship.
-
-## Exit migration (paired, M5 12mo; same pattern everywhere)
-
-- SL→SL 183 trades: ΔR 0 (identical by construction — stops untouched).
-- TRAIL→TP1 207 trades: +248.9R → +424.5R (**+175.6R**) — winners the trail exits early.
-- TRAIL→SL 129 trades: +19.0R → −136.8R (**−155.7R**) — retraces the trail survives
-  become full stop-outs.
-- TRAIL→TIMEOUT 25: +7.4R. Net = +26.2R. The whole effect is these two big flows nearly
-  cancelling; which one wins is a regime property, not an exit-policy edge.
-- Trades with ctrl MFE ≥ 2R: 95; hard cap forfeits only +10.4R vs trail on M5 (the trail
-  already gives ~0.75R back from peak, so a 2R cap costs little *in a trend*).
-
-## Mini-grid (12mo @ $0.65) — the tell
-
-| mult | M5 R / PF / DD | M15 R / PF / DD |
-|---|---|---|
-| trail | +98.6 / 1.483 / −10.7 | +73.9 / 1.525 / −10.8 |
-| 1.0R | +52.5 / 1.205 / −12.8 | +67.1 / 1.410 / −7.3 |
-| 1.5R | +80.7 / 1.263 / −21.4 | +66.9 / 1.325 / −11.2 |
-| 2.0R | +124.9 / 1.378 / −24.5 | +96.0 / 1.431 / −11.2 |
-| 2.5R | +147.7 / 1.426 / −23.6 | +110.4 / 1.469 / −15.9 |
-| 3.0R | +155.8 / 1.432 / −21.6 | +135.8 / 1.562 / −17.5 |
-
-Monotone in the multiple on both TFs — "2.0" is not an optimum, it's a point on a curve
-that says *never take profit* (3R's M5 TIMEOUT bucket alone is +84.8R). That is the
-signature of a one-way trend sample, not of a superior exit rule. It also reconfirms the
-original finding that SHORT fixed targets (≤1.5R) are strictly bad (cf. RR-0.5 removal,
-`config.py:126-133`: +48R → +105R).
-
-## 5yr regime decomposition (2021-08-01 → 2026-08-04 @ $0.65) — the verdict
-
-M15 (real sample in every year): ctrl +127.97 PF 1.209 DD −35.8; rr20 +158.97 PF 1.170
-DD **−59.8**; rr30 +204.03 PF 1.203 DD **−98.3**.
-
-| Year | n | trail | 2R hard | 3R hard |
-|---|---|---|---|---|
-| 2021 | 88 | **+4.00** | −16.09 | −17.41 |
-| 2022 | 284 | −9.54 | −7.63 | −26.74 |
-| 2023 | 227 | **+7.79** | −1.71 | −5.81 |
-| 2024 | 345 | +40.44 | +64.98 | +89.14 |
-| 2025 | 405 | +49.33 | +69.39 | +104.42 |
-| 2026 | 229 | +35.95 | +50.04 | +60.43 |
-
-M5 5yr mirrors it (ctrl +128.4 PF 1.387 DD −12.8 vs rr20 +160.2 PF 1.306 DD −24.7;
-2022: −1.1 vs −5.2), though pre-2024 M5 samples are thin (12–60 trades/yr).
-
-The hard target's entire outperformance lives in the 2024–2026 gold trend. In ranging
-years it flips sign exactly where the trail stays positive (M15 2021, 2023), and the
-further the target, the worse the ranging-year bleed and the deeper the DD. The 12mo,
-halves, and OOS windows all sit inside the trend era, which is why the rule-1 letter
-passes — this is precisely the within-regime-split trap rule 1's OOS clause exists to
-catch, one level up.
-
-## Additional non-ship reasons
-
-- **Live divergence grows with hold time.** Median hold doubles (M5 28→50 bars; OOS
-  70→124). The daemon holds one position total; the backtester doesn't model that
-  blocking. Longer holds = more live signals skipped that the paper book banks.
-- **WR 52→43% (M5) / 62→44% (M15), max loss streak 6→11 (M15)** — materially harder to
-  operate and size.
-- Aggregate tables (session/day/regime/vol, streaks, quarterly equity, rolling-20 WR)
-  computed and consistent — no bucket reverses the conclusion; both arms' 12mo profit is
-  concentrated in Q1-2026 (+87.7 ctrl / +102.0 rr20).
-
-## Status
-
-- `--fixed-rr-tp` stays as a documented research-only flag (default off; conflicts with
-  `--trail-arm/--trail-dist/--set signals.*trail*` are hard errors). Module 22 tests
-  (5) pin the override math, TP fill + spread, SL-first tie, PB-trail-disable, and CLI
-  guards. Full suite: 346 green.
-- Re-open only if gold's regime turns and a *regime-conditional* exit (trail in range,
-  target in trend) is proposed with a pre-registered mechanism — a plain fixed ratio is
-  now a closed hypothesis at every multiple in {0.5, 1.0, 1.5, 2.0, 2.5, 3.0}.

backtest/research/hour_analysis.py

@@ -1,40 +0,0 @@
-"""Per-hour P&L breakdown of a backtest trade book.
-
-Usage:
-    python backtest/research/hour_analysis.py BOOK.csv BAR_MINUTES
-
-Reports, per UTC hour (both entry hour and DECISION hour = entry_time
-minus one bar, the clock blocked_hours_utc evaluates): trade count,
-total R, mean R, win rate, profit factor. Also a per-(hour, direction)
-table. Analysis tooling only — never imported by the live code.
-"""
-import sys
-
-import pandas as pd
-
-
-def hour_tables(book_csv: str, bar_minutes: int):
-    df = pd.read_csv(book_csv)
-    et = pd.to_datetime(df["entry_time"], utc=True, format="mixed")
-    df["entry_hour"] = et.dt.hour
-    df["decision_hour"] = (et - pd.Timedelta(minutes=bar_minutes)).dt.hour
-
-    def table(group_cols):
-        g = df.groupby(group_cols)["r_multiple"]
-        out = g.agg(count="count", total_r="sum", mean_r="mean",
-                    wr=lambda r: (r > 0).mean() * 100)
-        out["pf"] = g.apply(
-            lambda r: r[r > 0].sum() / max(-r[r <= 0].sum(), 1e-9))
-        return out.round(2)
-
-    return (table("decision_hour"), table(["decision_hour", "direction"]),
-            table("entry_hour"))
-
-
-if __name__ == "__main__":
-    book, mins = sys.argv[1], int(sys.argv[2])
-    by_dec, by_dec_dir, by_ent = hour_tables(book, mins)
-    print(f"\n── {book} · by DECISION hour (bar-close clock) ──")
-    print(by_dec.to_string())
-    print(f"\n── by (decision hour, direction) ──")
-    print(by_dec_dir.to_string())

backtest/research/llm_news_eval.py

@@ -1,697 +0,0 @@
-"""
-DeepSeek news-analysis validation harness — XAU/USD.
-====================================================
-Research-only (no production code changes). Answers: is the DeepSeek
-layer's news analysis *correct*?
-
-Part A (events): replays every high-impact USD release with a genuine
-surprise (2024-01 → 2026-07, from backtest/cache/calendar.parquet)
-through the REAL `NewsEngine._llm_event_direction` — real prompt, real
-payload, real validation — and scores its "gold bullish/bearish" calls
-against gold's actual move (Oanda M15 mids) at +15m/30m/1h/4h/24h.
-Baselines: mechanical surprise-sign doctrine, FF actualBetterWorse,
-always-bullish/bearish (2024-26 is a gold bull regime), coin flip.
-
-Part B (headlines): a curated golden set (llm_news_golden.csv) through
-the REAL `NewsEngine._llm_headline_scores` in production-sized batches,
-vs the keyword scorer it overrides. Includes repeat-batch (determinism)
-and shuffled-order (cross-contamination) probes.
-
-Implementation notes that keep the replay faithful to production:
-  * `_llm_event_direction`/`_llm_headline_scores` resolve the client via
-    `get_llm_client()`, so we seed the module singleton with a research
-    client (raised daily cap: 586 events > production cap 500).
-  * The event cache key omits the forecast, so a repeat `actual` would
-    silently re-serve a direction computed for a different forecast —
-    the cache is cleared before every call.
-  * A "neutral" reply and a hard failure both return None; they are
-    disambiguated by reading the event cache back after the call
-    (absent = failure, "" = neutral, "bullish:/bearish: …" = call).
-  * Raw request/response bytes are captured with a requests response
-    hook — the request-building code path is untouched.
-
-Usage (from repo root):
-  venv/bin/python backtest/research/llm_news_eval.py events --limit 5
-  venv/bin/python backtest/research/llm_news_eval.py events --determinism 20
-  venv/bin/python backtest/research/llm_news_eval.py headlines
-  venv/bin/python backtest/research/llm_news_eval.py report
-
-Artifacts land in backtest/research/llm_news_eval_2026-08/ (JSONL replay
-logs are resume-safe: completed events are never re-paid). `report` is a
-pure offline function of the artifacts.
-"""
-
-import argparse
-import json
-import math
-import random
-import sys
-import time
-from datetime import datetime, timezone
-from pathlib import Path
-
-import numpy as np
-import pandas as pd
-
-REPO_ROOT = Path(__file__).resolve().parents[2]
-sys.path.insert(0, str(REPO_ROOT))
-
-import llm_client
-from app_config import deepseek_api_key, get_app_config
-from config import NewsConfig
-from llm_client import DEFAULT_MODEL, DeepSeekClient
-from news_engine import EconomicEvent, NewsEngine
-
-RESEARCH_DIR = Path(__file__).parent
-ART_DIR = RESEARCH_DIR / "llm_news_eval_2026-08"
-CALENDAR = REPO_ROOT / "backtest" / "cache" / "calendar.parquet"
-BARS = (REPO_ROOT / "backtest" / "cache" /
-        "oanda_XAU_USD_15min_20210531_20260804.parquet")
-GOLDEN = RESEARCH_DIR / "llm_news_golden.csv"
-EVENTS_JSONL = ART_DIR / "events_replay.jsonl"
-HEADLINES_JSONL = ART_DIR / "headlines_replay.jsonl"
-SCORED_CSV = ART_DIR / "events_scored.csv"
-
-HORIZONS = [("15m", 15), ("30m", 30), ("1h", 60), ("4h", 240),
-            ("24h", 1440)]
-NEUTRAL_BAND = 0.15  # |xau| below this = neutral prediction (Part B)
-BATCH_CAP = 25       # production batch cap (news_engine.py:783)
-EXPECTED_CORPUS = 586
-
-
-# ── shared plumbing ──────────────────────────────────────────────────
-
-class _Recorder:
-    """requests response hook capturing the exact wire traffic."""
-
-    def __init__(self):
-        self.last = None
-
-    def hook(self, resp, **kwargs):
-        body = resp.request.body
-        if isinstance(body, bytes):
-            body = body.decode("utf-8", "replace")
-        self.last = {
-            "request_body": body,
-            "http_status": resp.status_code,
-            "response_body": resp.text,
-            "elapsed_s": round(resp.elapsed.total_seconds(), 3),
-        }
-        return resp
-
-    def pop(self):
-        cap, self.last = self.last, None
-        return cap or {}
-
-
-def install_research_client(timeout: float):
-    """Build a research DeepSeekClient (raised cap) and seed the module
-    singleton so the production code paths use it. Returns (client,
-    recorder)."""
-    app = get_app_config()
-    key = deepseek_api_key(app)
-    ds = app.deepseek_settings
-    if not key:
-        sys.exit("No DeepSeek API key (env DEEPSEEK_API_KEY or "
-                 "config.json deepseek.api_key) — aborting before spend.")
-    if ds.get("enabled") is False:
-        sys.exit("deepseek.enabled is false in config.json — aborting.")
-    client = DeepSeekClient(
-        api_key=key,
-        model=str(ds.get("model", DEFAULT_MODEL)),
-        timeout=timeout,
-        daily_cap=10 ** 6,  # research: corpus 586 > production cap 500
-    )
-    rec = _Recorder()
-    client._session.hooks["response"].append(rec.hook)
-    with llm_client._client_lock:
-        llm_client._client = client
-    return client, rec
-
-
-def _s(x):
-    """Parquet str columns carry float NaN — EconomicEvent needs None."""
-    return None if pd.isna(x) else str(x)
-
-
-def load_corpus():
-    """High-impact USD releases with a genuine surprise, through the
-    production `EconomicEvent.surprise` gate. Asserts run before any
-    API spend."""
-    df = pd.read_parquet(CALENDAR)
-    df = df[(df["impactName"] == "high") & (df["currency"] == "USD")]
-    rows = []
-    for r in df.itertuples():
-        ev = EconomicEvent(
-            time=datetime.fromtimestamp(int(r.dateline), tz=timezone.utc),
-            currency="USD", impact="high", event=_s(r.name) or "",
-            forecast=_s(r.forecast), previous=_s(r.previous),
-            actual=_s(r.actual),
-        )
-        if ev.surprise not in ("positive", "negative"):
-            continue
-        rows.append({
-            "ffid": int(r.id), "dateline": int(r.dateline),
-            "utc_time": ev.time.isoformat(), "name": ev.event,
-            "actual": ev.actual, "forecast": ev.forecast,
-            "previous": ev.previous, "surprise": ev.surprise,
-            "better_worse": int(r.actualBetterWorse), "ev": ev,
-        })
-    ids = [d["ffid"] for d in rows]
-    if len(rows) != EXPECTED_CORPUS:
-        sys.exit(f"corpus size {len(rows)} != {EXPECTED_CORPUS} — "
-                 "calendar.parquet changed; re-verify before spending.")
-    if len(set(ids)) != len(ids):
-        sys.exit("FF ids not unique — resume keying would break.")
-    # Pin the actualBetterWorse mapping on two unambiguous NFP prints.
-    nfp = {d["utc_time"][:10]: d["better_worse"] for d in rows
-           if d["name"] == "Non-Farm Employment Change"}
-    if nfp.get("2024-01-05") != 1 or nfp.get("2024-05-03") != 2:
-        sys.exit("actualBetterWorse mapping pins failed (expected "
-                 "1=better-for-USD on 2024-01-05 beat, 2=worse on "
-                 "2024-05-03 miss).")
-    rows.sort(key=lambda d: (d["dateline"], d["ffid"]))
-    return rows
-
-
-def _load_jsonl(path: Path):
-    recs = []
-    if path.exists():
-        with open(path) as f:
-            for line in f:
-                line = line.strip()
-                if line:
-                    recs.append(json.loads(line))
-    return recs
-
-
-# ── Part A: event replay ─────────────────────────────────────────────
-
-def _extract_reason(cap):
-    try:
-        content = json.loads(cap["response_body"])[
-            "choices"][0]["message"]["content"]
-        return json.loads(content).get("reason")
-    except Exception:
-        return None
-
-
-def cmd_events(args):
-    ART_DIR.mkdir(exist_ok=True)
-    corpus = load_corpus()
-    if args.limit:
-        corpus = corpus[:args.limit]
-    engine = NewsEngine(NewsConfig())
-    existing = _load_jsonl(EVENTS_JSONL)
-    done1 = {r["ffid"] for r in existing
-             if r.get("run_pass") == 1 and r.get("llm_direction")}
-    done2 = {r["ffid"] for r in existing
-             if r.get("run_pass") == 2 and r.get("llm_direction")}
-    pending = [d for d in corpus if d["ffid"] not in done1]
-    print(f"corpus {len(corpus)} | already recorded {len(done1)} | "
-          f"pending {len(pending)} | determinism re-calls "
-          f"{args.determinism} | model calls are ~100 tokens each "
-          f"(deepseek-v4-flash, static-prefix cached) → cost ≈ cents")
-    if not pending and not args.determinism:
-        print("nothing to do")
-        return
-    _, rec = install_research_client(args.timeout)
-
-    out = open(EVENTS_JSONL, "a")
-
-    def call_one(d, run_pass):
-        # Cache key omits forecast → cleared per call so 196 colliding
-        # corpus rows can't be served a stale direction.
-        with NewsEngine._cache_lock:
-            NewsEngine._llm_event_cache.clear()
-        ev = d["ev"]
-        engine._llm_event_direction(ev)
-        key = (ev.event, ev.currency, str(ev.actual or ""))
-        with NewsEngine._cache_lock:
-            cached = NewsEngine._llm_event_cache.get(key)
-        cap = rec.pop()
-        if cached is None:
-            direction, text = None, None       # hard failure → retry
-        elif cached == "":
-            direction, text = "neutral", ""
-        else:
-            direction, text = cached.split(":", 1)[0], cached
-        recd = {k: d[k] for k in (
-            "ffid", "dateline", "utc_time", "name", "actual", "forecast",
-            "previous", "surprise", "better_worse")}
-        recd.update({
-            "llm_direction": direction, "llm_text": text,
-            "reason_raw": _extract_reason(cap), "run_pass": run_pass,
-            "ts": datetime.now(timezone.utc).isoformat(), **cap,
-        })
-        out.write(json.dumps(recd) + "\n")
-        out.flush()
-        return direction
-
-    remaining = pending
-    for npass in range(1, args.retry_passes + 1):
-        if not remaining:
-            break
-        failed = []
-        for i, d in enumerate(remaining):
-            if call_one(d, 1) is None:
-                failed.append(d)
-            if args.sleep:
-                time.sleep(args.sleep)
-            if (i + 1) % 25 == 0:
-                print(f"  pass {npass}: {i + 1}/{len(remaining)} "
-                      f"({len(failed)} failed)")
-        print(f"pass {npass} done: {len(remaining) - len(failed)} ok, "
-              f"{len(failed)} failed")
-        remaining = failed
-    if remaining:
-        print(f"WARNING: {len(remaining)} events still unrecorded after "
-              f"{args.retry_passes} passes — re-run `events` to resume.")
-
-    if args.determinism:
-        ok_ids = {x["ffid"] for x in remaining}
-        done_now = [d for d in corpus if d["ffid"] not in ok_ids]
-        done_now.sort(key=lambda d: (d["name"], d["dateline"]))
-        stride = max(1, len(done_now) // args.determinism)
-        sample = done_now[::stride][:args.determinism]
-        todo = [d for d in sample if d["ffid"] not in done2]
-        print(f"determinism probe: {len(todo)} re-calls "
-              f"({len(sample) - len(todo)} already recorded)")
-        for d in todo:
-            call_one(d, 2)
-            if args.sleep:
-                time.sleep(args.sleep)
-    out.close()
-    print("events replay complete →", EVENTS_JSONL)
-
-
-# ── Part B: golden headlines ─────────────────────────────────────────
-
-def _load_golden():
-    g = pd.read_csv(GOLDEN)
-    need = {"headline", "category", "expected_xau_sign", "rationale"}
-    if not need <= set(g.columns):
-        sys.exit(f"golden CSV missing columns {need - set(g.columns)}")
-    if not g["expected_xau_sign"].isin([-1, 0, 1]).all():
-        sys.exit("expected_xau_sign must be -1/0/+1")
-    if not g["headline"].is_unique:
-        sys.exit("duplicate headlines in golden CSV")
-    return g
-
-
-def cmd_headlines(args):
-    ART_DIR.mkdir(exist_ok=True)
-    g = _load_golden()
-    heads = list(g["headline"])
-    _, rec = install_research_client(args.timeout)
-    engine = NewsEngine(NewsConfig())
-    out = open(HEADLINES_JSONL, "w")  # tiny (~9 calls): fresh each run
-
-    def run_batch(texts, tag, batch_idx):
-        scores = None
-        for attempt in range(3):
-            scores = engine._llm_headline_scores(texts)
-            cap = rec.pop()
-            if scores is not None:
-                break
-            print(f"  {tag} batch {batch_idx} attempt {attempt + 1} "
-                  f"failed (http {cap.get('http_status')})")
-            time.sleep(2.0)
-        out.write(json.dumps({
-            "run_tag": tag, "batch": batch_idx, "headlines": texts,
-            "scores": scores,
-            "ts": datetime.now(timezone.utc).isoformat(), **cap,
-        }) + "\n")
-        out.flush()
-        return scores
-
-    batches = [heads[i:i + BATCH_CAP] for i in range(0, len(heads),
-                                                     BATCH_CAP)]
-    for bi, b in enumerate(batches):
-        ok = run_batch(b, "main", bi) is not None
-        print(f"main batch {bi} ({len(b)} headlines): "
-              f"{'ok' if ok else 'FAILED'}")
-        time.sleep(args.sleep)
-    run_batch(batches[0], "repeat", 0)
-    print("repeat probe done")
-    shuffled = heads[:]
-    random.Random(20260806).shuffle(shuffled)
-    for bi in range(0, len(shuffled), BATCH_CAP):
-        run_batch(shuffled[bi:bi + BATCH_CAP], "shuffled",
-                  bi // BATCH_CAP)
-    print("shuffled probe done →", HEADLINES_JSONL)
-    out.close()
-
-
-# ── report: pure offline function of the artifacts ───────────────────
-
-def _binom_p(k, n):
-    """Exact two-sided binomial p-value vs p=0.5 (no scipy)."""
-    if n == 0:
-        return float("nan")
-    dist = [math.comb(n, i) for i in range(n + 1)]
-    tot = float(2 ** n)
-    pk = dist[k]
-    return min(1.0, sum(c for c in dist if c <= pk) / tot)
-
-
-def _mcnemar_p(hit_a, hit_b):
-    """Exact McNemar via binomial on discordant pairs. Inputs: aligned
-    boolean lists."""
-    b = sum(1 for x, y in zip(hit_a, hit_b) if x and not y)
-    c = sum(1 for x, y in zip(hit_a, hit_b) if not x and y)
-    return b, c, _binom_p(min(b, c), b + c)
-
-
-def _latest_events(recs):
-    """Last pass-1 record per ffid (retries append; last wins)."""
-    out = {}
-    for r in recs:
-        if r.get("run_pass") == 1:
-            out[r["ffid"]] = r
-    return list(out.values())
-
-
-def _fmt_rate(hits, n, label=""):
-    if n == 0:
-        return f"{label:<24} n=0"
-    p = _binom_p(hits, n)
-    return (f"{label:<24} {hits:>4}/{n:<4} = {hits / n * 100:5.1f}%  "
-            f"(binom p={p:.3f})")
-
-
-def report_events():
-    recs = _latest_events(_load_jsonl(EVENTS_JSONL))
-    if not recs:
-        print("no event replay records — run `events` first")
-        return
-    bars = pd.read_parquet(BARS)
-    closes = bars["Close"].to_numpy()
-    close_ns = (bars.index + pd.Timedelta(minutes=15)).asi8
-
-    def anchor(t_ns, h_min):
-        i = np.searchsorted(close_ns, t_ns, side="right") - 1
-        if i < 0:
-            return None
-        stale_ns = max(45.0, h_min / 4.0) * 60e9
-        if t_ns - close_ns[i] > stale_ns:
-            return None
-        return closes[i]
-
-    rows, skip = [], {h: 0 for h, _ in HORIZONS}
-    base_skipped = 0
-    for r in recs:
-        t_ns = int(r["dateline"]) * 10 ** 9
-        base = anchor(t_ns, 0)
-        if base is None:
-            base_skipped += 1
-            continue
-        row = dict(r)
-        row.pop("request_body", None)
-        row.pop("response_body", None)
-        row["year"] = r["utc_time"][:4]
-        row["base"] = base
-        for hname, hmin in HORIZONS:
-            p = anchor(t_ns + hmin * 60 * 10 ** 9, hmin)
-            if p is None:
-                skip[hname] += 1
-                row[f"d_{hname}"] = np.nan
-            else:
-                row[f"d_{hname}"] = p - base
-        rows.append(row)
-    df = pd.DataFrame(rows)
-    df.to_csv(SCORED_CSV, index=False)
-
-    n_fail = int((df["llm_direction"].isna()).sum())
-    print("═" * 70)
-    print("PART A — event gold-direction vs actual XAU move")
-    print("═" * 70)
-    print(f"events replayed: {len(df)} (baseline-price skipped: "
-          f"{base_skipped}) | LLM hard failures: {n_fail}")
-    print("horizon skip counts (stale/closed):",
-          {k: v for k, v in skip.items() if v})
-    dist = df["llm_direction"].value_counts(dropna=False).to_dict()
-    print("LLM call distribution:", dist)
-
-    # Unconditional per-horizon noise scale (null): median |Δ| over all
-    # 2024+ bar times, position-shifted.
-    bars24 = bars[bars.index >= "2024-01-01"]
-    null_med = {}
-    for hname, hmin in HORIZONS:
-        k = hmin // 15
-        d = (bars24["Close"].shift(-k) - bars24["Close"]).abs().dropna()
-        null_med[hname] = float(d.median())
-    print("unconditional null median |Δ| ($):",
-          {k: round(v, 2) for k, v in null_med.items()})
-
-    # Strategy call columns: +1 gold up, −1 gold down, 0/NaN excluded.
-    df["c_llm"] = df["llm_direction"].map(
-        {"bullish": 1, "bearish": -1}).fillna(0)
-    df["c_doctrine"] = df["surprise"].map({"positive": -1, "negative": 1})
-    df["c_ffbw"] = df["better_worse"].map({1: -1, 2: 1}).fillna(0)
-    df["c_bull"] = 1
-    df["c_bear"] = -1
-
-    strategies = [("LLM", "c_llm"), ("doctrine(surprise)", "c_doctrine"),
-                  ("FF betterWorse", "c_ffbw"), ("always-bullish", "c_bull"),
-                  ("always-bearish", "c_bear")]
-
-    for hname, _ in HORIZONS:
-        dcol = f"d_{hname}"
-        sub = df[df[dcol].notna()]
-        print(f"\n── horizon {hname} (n scored bars={len(sub)}) ──")
-        zeros = int((sub[dcol] == 0).sum())
-        if zeros:
-            print(f"  Δ==0 count (scored as miss): {zeros}")
-        for label, col in strategies:
-            s = sub[sub[col] != 0]
-            hits = int((np.sign(s[dcol]) == s[col]).sum())
-            print("  " + _fmt_rate(hits, len(s), label))
-        # noise-filtered LLM hit rate
-        s = sub[(sub["c_llm"] != 0) &
-                (sub[dcol].abs() >= null_med[hname])]
-        hits = int((np.sign(s[dcol]) == s["c_llm"]).sum())
-        print("  " + _fmt_rate(hits, len(s),
-                               f"LLM |Δ|≥null-med"))
-        # event-window magnitude vs null
-        mag = sub[dcol].abs().median()
-        print(f"  event-window median |Δ| = ${mag:.2f} "
-              f"(null ${null_med[hname]:.2f})")
-        # paired McNemar: LLM vs each baseline on common directional set
-        for label, col in [("doctrine", "c_doctrine"),
-                           ("FF betterWorse", "c_ffbw"),
-                           ("always-bullish", "c_bull")]:
-            s = sub[(sub["c_llm"] != 0) & (sub[col] != 0)]
-            ha = list(np.sign(s[dcol]) == s["c_llm"])
-            hb = list(np.sign(s[dcol]) == s[col])
-            b, c, p = _mcnemar_p(ha, hb)
-            print(f"  McNemar LLM vs {label:<15} b={b:>3} c={c:>3} "
-                  f"p={p:.3f}")
-
-    # neutral-call flatness (15m)
-    neu = df[(df["llm_direction"] == "neutral") & df["d_15m"].notna()]
-    dire = df[(df["c_llm"] != 0) & df["d_15m"].notna()]
-    if len(neu):
-        print(f"\nneutral calls (n={len(neu)}): median |Δ_15m| "
-              f"${neu['d_15m'].abs().median():.2f} vs directional "
-              f"(n={len(dire)}) ${dire['d_15m'].abs().median():.2f} "
-              f"(null ${null_med['15m']:.2f})")
-
-    # breakdowns at 1h
-    dcol = "d_1h"
-    sub = df[df[dcol].notna() & (df["c_llm"] != 0)]
-    print("\n── LLM hit rate by event name (1h, top 10 by count) ──")
-    for name, grp in sorted(sub.groupby("name"), key=lambda kv: -len(kv[1]))[:10]:
-        hits = int((np.sign(grp[dcol]) == grp["c_llm"]).sum())
-        print("  " + _fmt_rate(hits, len(grp), name[:24]))
-    print("── LLM hit rate by year (1h) ──")
-    for yr, grp in sub.groupby("year"):
-        hits = int((np.sign(grp[dcol]) == grp["c_llm"]).sum())
-        print("  " + _fmt_rate(hits, len(grp), yr))
-
-    # cluster view: shared datelines
-    print("\n── same-timestamp clusters ──")
-    clus = df.groupby("dateline").filter(lambda g: len(g) > 1)
-    groups = list(clus.groupby("dateline"))
-    consistent = 0
-    net_hits, net_n = 0, 0
-    for _, g in groups:
-        signs = set(g.loc[g["c_llm"] != 0, "c_llm"])
-        if len(signs) <= 1:
-            consistent += 1
-        net = int(g["c_llm"].sum())
-        d = g["d_1h"].iloc[0]
-        if net != 0 and pd.notna(d) and d != 0:
-            net_n += 1
-            if np.sign(d) == np.sign(net):
-                net_hits += 1
-    print(f"clusters: {len(groups)} datelines "
-          f"({len(clus)} events, {len(clus) / max(len(df), 1) * 100:.0f}% of "
-          f"corpus) | internally consistent: {consistent}/{len(groups)}")
-    print("  " + _fmt_rate(net_hits, net_n, "cluster net-call (1h)"))
-
-    # determinism probe
-    p2 = {r["ffid"]: r for r in _load_jsonl(EVENTS_JSONL)
-          if r.get("run_pass") == 2 and r.get("llm_direction")}
-    if p2:
-        p1 = {r["ffid"]: r for r in recs}
-        agree = same_reason = n = 0
-        for fid, r2 in p2.items():
-            r1 = p1.get(fid)
-            if not r1 or not r1.get("llm_direction"):
-                continue
-            n += 1
-            if r1["llm_direction"] == r2["llm_direction"]:
-                agree += 1
-            if r1.get("reason_raw") == r2.get("reason_raw"):
-                same_reason += 1
-        print(f"\ndeterminism probe: direction agreement {agree}/{n}, "
-              f"identical reason text {same_reason}/{n}")
-
-    # stratified reason examples
-    print("\n── reason samples (1h hit/miss) ──")
-    sub = df[df[dcol].notna() & (df["c_llm"] != 0)].copy()
-    sub["hit"] = np.sign(sub[dcol]) == sub["c_llm"]
-    for verdict in (True, False):
-        picked, seen = [], set()
-        for _, r in sub[sub["hit"] == verdict].iterrows():
-            if r["name"] in seen:
-                continue
-            seen.add(r["name"])
-            picked.append(r)
-            if len(picked) == 10:
-                break
-        tag = "HIT " if verdict else "MISS"
-        for r in picked:
-            print(f"  [{tag}] {r['utc_time'][:16]} {r['name'][:30]:<30} "
-                  f"a={r['actual']} f={r['forecast']} → "
-                  f"{r['llm_direction']:<7} Δ1h={r[dcol]:+7.2f}  "
-                  f"“{(r['reason_raw'] or '')[:60]}”")
-    print(f"\nscored table → {SCORED_CSV}")
-
-
-def report_headlines():
-    recs = _load_jsonl(HEADLINES_JSONL)
-    if not recs:
-        print("\n(no headline replay records — run `headlines` first)")
-        return
-    g = _load_golden()
-    engine = NewsEngine(NewsConfig())
-
-    def collect(tag):
-        m = {}
-        for r in recs:
-            if r["run_tag"] == tag and r["scores"]:
-                for h, sc in zip(r["headlines"], r["scores"]):
-                    m[h] = tuple(sc)
-        return m
-
-    main, shuf = collect("main"), collect("shuffled")
-    print("\n" + "═" * 70)
-    print("PART B — golden headline set: LLM vs keyword baseline")
-    print("═" * 70)
-    missing = [h for h in g["headline"] if h not in main]
-    if missing:
-        print(f"WARNING: {len(missing)} headlines missing from main run")
-
-    def cls(x):
-        if x >= NEUTRAL_BAND:
-            return 1
-        if x <= -NEUTRAL_BAND:
-            return -1
-        return 0
-
-    rows = []
-    for _, r in g.iterrows():
-        h = r["headline"]
-        if h not in main:
-            continue
-        llm_xau = main[h][1]
-        kw = engine._analyze_headline_sentiment(h, for_currency="XAU")
-        rows.append({
-            "headline": h, "category": r["category"],
-            "exp": int(r["expected_xau_sign"]),
-            "llm_xau": llm_xau, "llm_cls": cls(llm_xau),
-            "kw": kw, "kw_cls": cls(kw),
-        })
-    t = pd.DataFrame(rows)
-    for who in ("llm", "kw"):
-        acc = (t[f"{who}_cls"] == t["exp"]).mean()
-        print(f"{who.upper():<4} 3-class accuracy (band ±{NEUTRAL_BAND}): "
-              f"{(t[f'{who}_cls'] == t['exp']).sum()}/{len(t)} "
-              f"= {acc * 100:.1f}%")
-    ha = list(t["llm_cls"] == t["exp"])
-    hb = list(t["kw_cls"] == t["exp"])
-    b, c, p = _mcnemar_p(ha, hb)
-    print(f"paired McNemar LLM vs keyword: b={b} c={c} p={p:.3f}")
-
-    print("\nper-category accuracy (LLM | keyword):")
-    for cat, grp in t.groupby("category"):
-        a1 = (grp["llm_cls"] == grp["exp"]).sum()
-        a2 = (grp["kw_cls"] == grp["exp"]).sum()
-        print(f"  {cat:<18} {a1:>2}/{len(grp):<2} | {a2:>2}/{len(grp)}")
-
-    print("\nLLM confusion (rows=expected, cols=predicted -1/0/+1):")
-    cm = pd.crosstab(t["exp"], t["llm_cls"]).reindex(
-        index=[-1, 0, 1], columns=[-1, 0, 1], fill_value=0)
-    print(cm.to_string())
-
-    print("\nLLM misclassified headlines:")
-    for _, r in t[t["llm_cls"] != t["exp"]].iterrows():
-        print(f"  exp {r['exp']:+d} got {r['llm_xau']:+.2f} "
-              f"[{r['category']}] {r['headline'][:70]}")
-
-    # repeat-batch determinism
-    m_b0 = [r for r in recs if r["run_tag"] == "main" and r["batch"] == 0]
-    r_b0 = [r for r in recs if r["run_tag"] == "repeat"]
-    if m_b0 and r_b0 and m_b0[0]["scores"] and r_b0[0]["scores"]:
-        a = np.array(m_b0[0]["scores"], dtype=float)
-        b_ = np.array(r_b0[0]["scores"], dtype=float)
-        exact = int((a == b_).all(axis=1).sum())
-        print(f"\nrepeat-batch: {exact}/{len(a)} score pairs identical, "
-              f"max |diff| = {np.abs(a - b_).max():.3f}")
-
-    # shuffled-order drift
-    if shuf:
-        both = [h for h in t["headline"] if h in shuf]
-        drift = [abs(main[h][1] - shuf[h][1]) for h in both]
-        if drift:
-            print(f"shuffled-order xau drift over {len(both)} headlines: "
-                  f"mean {np.mean(drift):.3f}, max {np.max(drift):.3f}")
-
-
-def cmd_report(args):
-    report_events()
-    report_headlines()
-
-
-# ── CLI ──────────────────────────────────────────────────────────────
-
-def main():
-    ap = argparse.ArgumentParser(description=__doc__.split("\n")[1])
-    sub = ap.add_subparsers(dest="cmd", required=True)
-
-    e = sub.add_parser("events", help="replay release surprises")
-    e.add_argument("--limit", type=int, default=0,
-                   help="smoke-test on first N events")
-    e.add_argument("--determinism", type=int, default=0,
-                   help="re-call N stratified events a second time")
-    e.add_argument("--sleep", type=float, default=0.3)
-    e.add_argument("--retry-passes", type=int, default=3)
-    e.add_argument("--timeout", type=float, default=30.0)
-    e.set_defaults(fn=cmd_events)
-
-    h = sub.add_parser("headlines", help="golden-set probes")
-    h.add_argument("--sleep", type=float, default=0.3)
-    h.add_argument("--timeout", type=float, default=30.0)
-    h.set_defaults(fn=cmd_headlines)
-
-    r = sub.add_parser("report", help="offline scoring report")
-    r.set_defaults(fn=cmd_report)
-
-    args = ap.parse_args()
-    args.fn(args)
-
-
-if __name__ == "__main__":
-    main()

backtest/research/llm_news_eval_2026-08-06.md

@@ -1,86 +0,0 @@
-# DeepSeek news-analysis validation — 2026-08-06
-
-**Verdict: headline sentiment scoring is excellent; event-direction calls are
-directionally sound but ~11% of them misread the numeric comparison, and the
-failure concentrates on Unemployment Claims / close K-M-suffixed values.**
-
-Method (owner-directed, token-lean): independent expert re-analysis of a
-stratified sample of DeepSeek's recorded answers, not a full statistical
-backtest. Harness: `backtest/research/llm_news_eval.py` (replays releases
-through the REAL `_llm_event_direction` / `_llm_headline_scores` — real
-prompts, payloads, validation; raw wire traffic logged). 135 of the 586-event
-corpus were replayed before the full run was stopped by owner request
-(artifacts retained in `llm_news_eval_2026-08/`, resume-safe).
-
-## Part A — event gold-direction (28-event stratified sample, Jan–Jun 2024)
-
-Reviewer agreement with DeepSeek: **24/28 (86%)**. All 4 disagreements are
-DeepSeek errors, in exactly two failure modes:
-
-1. **Numeric comparison misread (3/28 ≈ 11%)** — the doctrine logic is right
-   but the premise is backwards:
-   - JOLTS 8.79M vs f 8.84M (a miss) → DS: "Job openings **above** forecast" → bearish (should be bullish)
-   - Claims 202K vs f 217K (fewer claims) → DS: "Claims **higher** than forecast" → bullish (should be bearish)
-   - Claims 202K vs f 209K — same backwards read, same wrong call
-   Both Unemployment Claims events in the sample were misread the same way —
-   looks systematic on that event type, possibly an anchoring prior that
-   "claims rose". All misreads involve close values with K/M suffixes.
-2. **Direction field contradicts its own reason (1/28)** — Empire State
-   −43.7 vs f −4.9: reason text says "gold may rise", `gold_direction` says
-   `bearish`.
-
-Where the numbers were read correctly, hawkish/dovish→gold mapping was
-consistently correct (CPI/PPI/NFP/ADP/GDP/PMI/wages/unemployment-rate all
-handled properly, including inverted metrics when read right).
-
-Price cross-check (1h horizon, cluster-confounded, indicative only):
-reviewer calls 22/28 with the subsequent move, DeepSeek 20/28.
-
-## Part B — headline sentiment (78-headline golden set)
-
-DeepSeek matched the reviewer's expected gold-direction label on
-**78/78 (100%)** — including all 8 negation traps ("signals no rate hike",
-"no rate cuts until…") and all 6 inverted-metric phrasings ("claims hit
-two-year high"). The keyword scorer it overrides got **24/78 (31%)**.
-Repeat-batch and shuffled-order probes recorded (headlines_replay.jsonl).
-
-## Implications
-
-- The headline-sentiment override (the main live LLM surface) is validated —
-  a large, measured upgrade over keywords.
-- The event-direction enrichment ("— gold bullish/bearish" appended to
-  surprise descriptions) is right ~85–90% of the time but can state a
-  factually backwards comparison in the displayed reason. It is display-only
-  (never gates signals), so exposure is a possibly-misleading Telegram/CLI
-  line, not trades.
-- **Hardening SHIPPED same day (owner-approved):** the payload now carries a
-  code-computed `vs_forecast` field ("above/below/in line with forecast",
-  from `EconomicEvent.surprise`) and the prompt tells the model to trust it.
-  First iteration fixed the arithmetic but exposed a masked doctrine error
-  (model read "claims below forecast" correctly, then called it a *weaker*
-  labor market — and regressed the unemployment-rate case the same way), so
-  the prompt's inverted-metrics line was expanded to an explicit rule plus a
-  worked claims example. Verification: **18/18** — all 4 original misreads
-  fixed, 11/11 no-regression cases unchanged, 3/3 fresh probes on the side
-  the prompt example does NOT cover (claims 245K vs f 220K → bullish,
-  unemployment 4.1% vs 3.9% → bullish, JOLTS 9.10M vs 8.80M → bearish).
-  Suite green at 346 tests, incl. a new payload assertion in Module 19.
-- Production observations found while building the harness (not fixed):
-  `_llm_event_cache` key omits forecast → a repeated `actual` re-serves a
-  direction computed for a different forecast; a "neutral" reply is
-  indistinguishable from a failure in the return value.
-
-## Reproduce
-
-```
-venv/bin/python backtest/research/llm_news_eval.py events --limit 5   # smoke
-venv/bin/python backtest/research/llm_news_eval.py events --determinism 20
-venv/bin/python backtest/research/llm_news_eval.py headlines
-venv/bin/python backtest/research/llm_news_eval.py report             # offline
-```
-
-Corpus: 586 high-impact USD surprises 2024-01→2026-07 (calendar.parquet,
-NFP-verified UTC); prices: Oanda M15 mids. Full-run cost ≈ cents. The
-`report` subcommand (hit rates vs always-bullish/doctrine/FF-betterWorse
-baselines, McNemar, cluster view) is built and untested at scale — run it
-after completing the replay if a statistical answer is ever wanted.

backtest/research/llm_news_golden.csv

@@ -1,79 +0,0 @@
-headline,category,expected_xau_sign,rationale
-"Powell says restrictive policy needed for longer as inflation proves sticky",hawkish_fed,-1,"higher-for-longer rates lift real yields, gold-bearish"
-"Fed minutes show officials leaning toward two more rate hikes this year",hawkish_fed,-1,"more hikes = stronger dollar/yields"
-"Hot CPI print fuels bets on additional Fed tightening",hawkish_fed,-1,"tightening expectations weigh on gold"
-"Fed's Waller: higher-for-longer rates warranted by resilient economy",hawkish_fed,-1,"hawkish guidance"
-"FOMC raises rates 50bp, flags further increases ahead",hawkish_fed,-1,"outsized hike plus hawkish guidance"
-"Treasury yields spike as Fed pushes back on rate cut hopes",hawkish_fed,-1,"cut hopes dashed, yields up"
-"Core inflation accelerates, cementing case for hawkish Fed stance",hawkish_fed,-1,"hawkish repricing"
-"Fed dot plot shifts higher; markets price out easing for the year",hawkish_fed,-1,"easing priced out"
-"Fed cuts rates 25bp and signals more easing to come",dovish_fed,1,"easing cycle is gold-bullish"
-"Powell hints at rate cuts as inflation cools toward target",dovish_fed,1,"cut guidance"
-"FOMC minutes reveal growing support for policy easing",dovish_fed,1,"dovish minutes"
-"Fed's Williams says rate cuts appropriate later this year",dovish_fed,1,"cut guidance"
-"Soft inflation data boosts odds of a dovish Fed pivot",dovish_fed,1,"pivot odds up"
-"Fed pauses hikes, markets price aggressive easing cycle",dovish_fed,1,"easing priced in"
-"Central banks turn dovish as global growth falters",dovish_fed,1,"global easing backdrop"
-"Fed to restart bond purchases to support flagging economy",dovish_fed,1,"QE is gold-bullish"
-"Dollar index hits 12-month high on robust US economic data",strong_usd,-1,"strong dollar pressures gold"
-"US GDP grows 4.1% annualized, blowing past expectations",strong_usd,-1,"strong economy, hawkish implication"
-"Blockbuster jobs report sends dollar surging across the board",strong_usd,-1,"strong jobs lift dollar and yields"
-"US retail sales smash forecasts, underscoring consumer strength",strong_usd,-1,"strong data"
-"Greenback rallies as US data outshines Europe and Asia",strong_usd,-1,"dollar strength"
-"ISM services index jumps to two-year high on strong demand",strong_usd,-1,"strong data"
-"Consumer confidence surges to post-pandemic high in the US",strong_usd,-1,"strong data"
-"Dollar tumbles to eight-month low after dismal manufacturing data",weak_usd,1,"weak dollar lifts gold"
-"US economy adds far fewer jobs than expected; dollar slides",weak_usd,1,"weak jobs, dovish implication"
-"Greenback weakens as US retail sales unexpectedly contract",weak_usd,1,"weak data"
-"US GDP growth revised down sharply, stoking slowdown fears",weak_usd,1,"slowdown is dovish"
-"Dollar slumps as trade deficit balloons to record",weak_usd,1,"dollar weakness"
-"Soft services PMI adds to evidence of cooling US economy",weak_usd,1,"cooling economy"
-"US consumer sentiment plunges to two-year low",weak_usd,1,"weak data"
-"Real yields climb to multi-year highs as TIPS sell off",real_yields,-1,"rising real yields raise gold's opportunity cost"
-"10-year Treasury yield surges past 5% on supply glut",real_yields,-1,"yield spike"
-"Rising real interest rates dim appeal of non-yielding assets",real_yields,-1,"explicit opportunity-cost story"
-"Treasury yields sink as investors flock to bonds",real_yields,1,"falling yields support gold"
-"Real yields turn negative as inflation outpaces rates",real_yields,1,"negative real yields are gold's best backdrop"
-"Bond rally drives 10-year yield to lowest level in a year",real_yields,1,"falling yields"
-"Geopolitical tensions escalate in Middle East, investors seek safety",safe_haven,1,"safe-haven bid"
-"Stocks crater worldwide as banking crisis fears spread",safe_haven,1,"crisis flight to safety"
-"Escalating conflict drives flight to safe-haven assets",safe_haven,1,"explicit haven flow"
-"Investors dump risk assets after surprise sovereign default",safe_haven,1,"default shock"
-"Military escalation on NATO border rattles global markets",safe_haven,1,"geopolitical stress"
-"Wall Street plunges 4% in worst session since pandemic",safe_haven,1,"risk-off"
-"Currency crisis in emerging markets sparks contagion fears",safe_haven,1,"contagion, haven bid"
-"Nuclear rhetoric intensifies, sending investors to havens",safe_haven,1,"extreme geopolitical stress"
-"Ceasefire agreement lifts global markets, haven demand fades",risk_on,-1,"haven unwind"
-"Stocks roar to record highs as trade deal signed",risk_on,-1,"risk-on drains haven demand"
-"Risk appetite returns as banking fears subside",risk_on,-1,"risk-on"
-"Global equities rally on strong earnings and easing tensions",risk_on,-1,"risk-on"
-"Gold hits record high as central banks keep buying bullion",gold_direct,1,"direct demand"
-"Gold ETFs post biggest weekly inflows in three years",gold_direct,1,"investment inflows"
-"China's central bank extends gold buying spree to 18th month",gold_direct,1,"official-sector demand"
-"Miners struggle to meet gold demand as supply deficit widens",gold_direct,1,"supply deficit"
-"Gold slides 2% as investors book profits after record run",gold_direct,-1,"profit-taking pressure"
-"Gold tumbles below key support as ETF outflows accelerate",gold_direct,-1,"outflows"
-"Bullion demand weakens as Indian jewellery buying stalls",gold_direct,-1,"physical demand soft"
-"Speculators slash bullish gold bets as dollar strengthens",gold_direct,-1,"positioning unwind"
-"Fed holds rates steady, signals no rate hike on the horizon",negation_traps,1,"dovish hold; keyword layer sees the phrase rate hike and flips it gold-bearish"
-"Powell says no rate cuts until inflation shows sustained cooling",negation_traps,-1,"hawkish hold; keyword layer sees rate cut and calls it gold-bullish"
-"Recession fears fade as US data streak beats estimates",negation_traps,-1,"fading fear is risk-on; keywords see recession plus beats and cancel out"
-"Inflation fails to accelerate, easing pressure on the Fed",negation_traps,1,"disinflation opens door to cuts"
-"Fed's Bowman: not yet confident inflation is beaten, cuts premature",negation_traps,-1,"hawkish pushback phrased around negations"
-"Central bank stops short of hiking, but warns tightening may resume",negation_traps,-1,"hawkish hold with tightening bias"
-"No safe-haven bid for gold despite equity wobble, analysts say",negation_traps,-1,"explicitly negated haven demand"
-"Jobs report not as weak as feared, dollar recovers",negation_traps,-1,"less-bad data is dollar-positive"
-"US manufacturing output plunges as new orders dry up",inverted_metrics,1,"weak economy is dovish/gold-bullish; keyword layer reads plunges as bearish"
-"US factory activity slumps to three-year low",inverted_metrics,1,"weak data; keyword reads slumps as bearish"
-"Consumer spending falls for third straight month",inverted_metrics,1,"weak data; keyword reads falls as bearish"
-"Private payrolls growth slides to weakest pace since 2020",inverted_metrics,1,"weak jobs; keyword reads slides as bearish"
-"Jobless claims hit two-year high as layoffs spread",inverted_metrics,1,"higher claims = weaker economy = gold-bullish"
-"US unemployment rate hits 4.5%, highest since 2021",inverted_metrics,1,"higher unemployment = dovish = gold-bullish"
-"Lakers clinch playoff berth with overtime win over Suns",distractors,0,"irrelevant"
-"New Marvel film smashes box office records on opening weekend",distractors,0,"irrelevant"
-"Apple unveils thinner MacBook with faster chip lineup",distractors,0,"irrelevant"
-"Scientists discover new deep-sea species off Pacific coast",distractors,0,"irrelevant"
-"Heavy rain forecast for the Midwest through the weekend",distractors,0,"irrelevant"
-"Streaming service raises subscription prices in Europe",distractors,0,"irrelevant"
-"Electric vehicle startup recalls 12,000 cars over software glitch",distractors,0,"irrelevant"
-"Local elections see record turnout in several US states",distractors,0,"irrelevant"

backtest/research/oanda_td_divergence.py

@@ -1,199 +0,0 @@
-"""
-Oanda vs Twelve Data bar-divergence study — XAU/USD M15, 12 months.
-====================================================================
-Compares the mid-price OHLC bars the signal engine would see from each
-provider over the same window. Purpose: quantify whether migrating the
-signal feed from Twelve Data to Oanda is a formality (bars ~identical)
-or a recalibration (bars differ at a scale that moves indicators).
-
-Inputs:
-  --td      Twelve Data parquet (raw bars from backtest/cache/)
-  --oanda   Oanda parquet (from fetch_oanda_m5.py / fetch_oanda_m15.py)
-  --start / --end  analysis window (UTC dates)
-
-Outputs:
-  printed report
-  backtest/research/oanda_td_divergence_by_hour.csv
-  backtest/research/oanda_td_bar_diffs.parquet  (per-bar diffs, common index)
-"""
-
-import argparse
-from pathlib import Path
-
-import numpy as np
-import pandas as pd
-
-RESEARCH_DIR = Path(__file__).parent
-PIP = 0.10  # XAU pip in $
-
-
-def wilder_atr(df: pd.DataFrame, period: int = 14) -> pd.Series:
-    hl = df["High"] - df["Low"]
-    hc = (df["High"] - df["Close"].shift(1)).abs()
-    lc = (df["Low"] - df["Close"].shift(1)).abs()
-    tr = pd.concat([hl, hc, lc], axis=1).max(axis=1)
-    return tr.ewm(alpha=1.0 / period, adjust=False).mean()
-
-
-def pct_stats(s: pd.Series) -> dict:
-    a = s.abs()
-    return {
-        "mean": a.mean(), "median": a.median(),
-        "p90": a.quantile(0.90), "p95": a.quantile(0.95),
-        "p99": a.quantile(0.99), "max": a.max(),
-        "bias(mean signed)": s.mean(),
-    }
-
-
-def fmt_stats(name: str, s: pd.Series) -> str:
-    st = pct_stats(s)
-    return (f"  {name:<6} mean ${st['mean']:.3f}  med ${st['median']:.3f}  "
-            f"p95 ${st['p95']:.3f}  p99 ${st['p99']:.3f}  max ${st['max']:.2f}  "
-            f"bias {st['bias(mean signed)']:+.4f}")
-
-
-def main():
-    ap = argparse.ArgumentParser()
-    ap.add_argument("--td", default=str(RESEARCH_DIR.parent / "cache" /
-                                        "XAU_USD_15min_20240522_20260723.parquet"))
-    ap.add_argument("--oanda", default=str(RESEARCH_DIR /
-                                           "oanda_xau_m15_20250701_now.parquet"))
-    ap.add_argument("--start", default="2025-07-24")
-    ap.add_argument("--end", default="2026-07-23")
-    args = ap.parse_args()
-
-    td = pd.read_parquet(args.td)
-    oa = pd.read_parquet(args.oanda)
-
-    start = pd.Timestamp(args.start, tz="UTC")
-    end = pd.Timestamp(args.end, tz="UTC") + pd.Timedelta(days=1)
-    start = max(start, td.index[0], oa.index[0])
-    end = min(end, td.index[-1], oa.index[-1])
-    td = td.loc[(td.index >= start) & (td.index <= end)]
-    oa = oa.loc[(oa.index >= start) & (oa.index <= end)]
-
-    print("=" * 78)
-    print("OANDA vs TWELVE DATA — XAU/USD M15 bar divergence")
-    print(f"Window: {start} -> {end}  ({(end - start).days} days)")
-    print("=" * 78)
-
-    # ── Coverage ─────────────────────────────────────────────────────
-    common = td.index.intersection(oa.index)
-    td_only = td.index.difference(oa.index)
-    oa_only = oa.index.difference(td.index)
-    print(f"\nCOVERAGE")
-    print(f"  Twelve Data bars : {len(td):>7,}")
-    print(f"  Oanda bars       : {len(oa):>7,}")
-    print(f"  common           : {len(common):>7,}")
-    print(f"  TD-only          : {len(td_only):>7,}  "
-          f"({100 * len(td_only) / max(len(td), 1):.2f}% of TD)")
-    print(f"  Oanda-only       : {len(oa_only):>7,}  "
-          f"({100 * len(oa_only) / max(len(oa), 1):.2f}% of Oanda)")
-    for label, idx in (("TD-only", td_only), ("Oanda-only", oa_only)):
-        if len(idx):
-            by_date = pd.Series(1, index=idx).groupby(idx.date).sum()
-            top = by_date.sort_values(ascending=False).head(8)
-            print(f"  {label} bars cluster on: "
-                  + ", ".join(f"{d} ({n})" for d, n in top.items()))
-            by_hour = pd.Series(1, index=idx).groupby(idx.hour).sum()
-            top_h = by_hour.sort_values(ascending=False).head(6)
-            print(f"  {label} by UTC hour: "
-                  + ", ".join(f"{h:02d}h ({n})" for h, n in top_h.items()))
-
-    t = td.loc[common]
-    o = oa.loc[common]
-
-    # ── Scale references ─────────────────────────────────────────────
-    atr = wilder_atr(t).dropna()
-    mean_price = t["Close"].mean()
-    print(f"\nSCALE REFERENCES (Twelve Data feed)")
-    print(f"  mean close      : ${mean_price:,.2f}")
-    print(f"  mean bar range  : ${(t['High'] - t['Low']).mean():.3f}")
-    print(f"  mean ATR14      : ${atr.mean():.3f}   median ${atr.median():.3f}")
-
-    # ── Per-field diffs (TD − Oanda mid) ─────────────────────────────
-    print(f"\nPER-BAR DIFFS  (TD − Oanda, $; pip = $0.10)")
-    diffs = {}
-    for col in ("Open", "High", "Low", "Close"):
-        d = t[col] - o[col]
-        diffs[col] = d
-        print(fmt_stats(col, d))
-
-    dclose = diffs["Close"].abs()
-    print(f"\n  |ΔClose| as % of mean ATR14 : "
-          f"mean {100 * dclose.mean() / atr.mean():.1f}%   "
-          f"p95 {100 * dclose.quantile(0.95) / atr.mean():.1f}%")
-
-    print(f"\nTHRESHOLD EXCEEDANCE  (|ΔClose|)")
-    for thr in (0.10, 0.30, 0.65, 1.00, 2.00, 5.00):
-        pct = 100 * (dclose > thr).mean()
-        print(f"  > ${thr:<5.2f}: {pct:6.2f}% of bars  ({(dclose > thr).sum():,})")
-
-    # ── Direction / return agreement ─────────────────────────────────
-    t_ret = t["Close"].pct_change()
-    o_ret = o["Close"].pct_change()
-    # only consecutive common bars (gap = one M15 step)
-    step_ok = pd.Series(common, index=common).diff().eq(pd.Timedelta(minutes=15))
-    tr, orr = t_ret[step_ok], o_ret[step_ok]
-    corr = tr.corr(orr)
-    t_dir = np.sign(t["Close"] - t["Open"])
-    o_dir = np.sign(o["Close"] - o["Open"])
-    both_nonflat = (t_dir != 0) & (o_dir != 0)
-    dir_agree = 100 * (t_dir[both_nonflat] == o_dir[both_nonflat]).mean()
-    print(f"\nAGREEMENT")
-    print(f"  close-return correlation (consecutive bars): {corr:.5f}  "
-          f"(n={step_ok.sum():,})")
-    print(f"  bar direction (sign of close−open) agree   : {dir_agree:.2f}%")
-
-    # ── By-hour breakdown ────────────────────────────────────────────
-    hours = common.hour
-    rows = []
-    for h in range(24):
-        m = hours == h
-        if not m.any():
-            continue
-        row = {"hour_utc": h, "bars": int(m.sum()),
-               "mean_abs_dclose": dclose[m].mean(),
-               "p95_abs_dclose": dclose[m].quantile(0.95),
-               "mean_abs_dhigh": diffs["High"][m].abs().mean(),
-               "mean_abs_dlow": diffs["Low"][m].abs().mean()}
-        if "bid_c" in o.columns:
-            row["oanda_median_spread"] = (
-                (o["ask_c"] - o["bid_c"])[m].median())
-        rows.append(row)
-    by_hour = pd.DataFrame(rows)
-    out_hour = RESEARCH_DIR / "oanda_td_divergence_by_hour.csv"
-    by_hour.to_csv(out_hour, index=False, float_format="%.4f")
-    print(f"\nBY UTC HOUR  (written to {out_hour.name})")
-    print(by_hour.to_string(index=False,
-                            float_format=lambda x: f"{x:.3f}"))
-
-    # ── By-month drift ───────────────────────────────────────────────
-    print(f"\nBY MONTH  (mean |ΔClose|, $)")
-    by_month = dclose.groupby(common.to_period("M")).mean()
-    for p, v in by_month.items():
-        print(f"  {p}: {v:.3f}")
-
-    # ── Oanda real spread (bonus, ties to Dukascopy study) ───────────
-    if "bid_c" in o.columns:
-        spread = o["ask_c"] - o["bid_c"]
-        liquid = ~o.index.hour.isin([21, 22, 23, 0])
-        print(f"\nOANDA REAL SPREAD (close bid/ask, $)")
-        print(f"  all bars      : median {spread.median():.3f}  "
-              f"p25 {spread.quantile(0.25):.3f}  p75 {spread.quantile(0.75):.3f}")
-        print(f"  liquid hours  : median {spread[liquid].median():.3f}")
-        print(f"  21-00 UTC     : median {spread[~liquid].median():.3f}")
-
-    # ── Persist per-bar diffs ────────────────────────────────────────
-    out = pd.DataFrame({
-        "td_close": t["Close"], "oa_close": o["Close"],
-        "d_open": diffs["Open"], "d_high": diffs["High"],
-        "d_low": diffs["Low"], "d_close": diffs["Close"],
-    })
-    out_path = RESEARCH_DIR / "oanda_td_bar_diffs.parquet"
-    out.to_parquet(out_path)
-    print(f"\nPer-bar diffs written to {out_path}")
-
-
-if __name__ == "__main__":
-    main()

backtest/research/oanda_td_divergence_by_hour.csv

@@ -1,25 +0,0 @@
-hour_utc,bars,mean_abs_dclose,p95_abs_dclose,mean_abs_dhigh,mean_abs_dlow,oanda_median_spread
-0,1029,3.3483,13.9460,3.2906,3.4310,0.7900
-1,1028,3.3483,12.8292,3.3477,3.6398,0.7700
-2,1028,3.0288,10.8960,2.9469,3.1317,0.7700
-3,1028,2.8783,11.1300,2.8419,2.9992,0.7600
-4,1028,2.8232,10.4710,2.8165,2.8455,0.7400
-5,1028,2.9268,10.6027,2.9234,3.0038,0.7400
-6,1028,3.0466,10.9265,2.9487,3.0796,0.7200
-7,1028,3.0286,11.3512,3.0301,3.0797,0.6700
-8,1028,3.0223,11.1483,2.8989,3.0801,0.6600
-9,1028,3.0460,11.6345,2.9307,3.0609,0.6600
-10,1028,2.9280,10.9015,2.8647,3.0029,0.6600
-11,1028,3.0340,11.4712,2.9591,2.9472,0.6600
-12,1028,3.0986,11.3282,2.9412,3.1407,0.7000
-13,1028,3.4014,12.3343,3.1077,3.4228,0.7300
-14,1028,3.5074,11.3628,3.3960,3.6569,0.6900
-15,1028,3.4514,11.8982,3.4124,3.6733,0.6700
-16,1028,3.1275,12.5155,3.2038,3.3061,0.6700
-17,1020,2.9826,12.6435,3.0064,3.1008,0.6900
-18,1015,2.8721,12.5480,2.9101,2.9996,0.6900
-19,1004,2.8637,12.1335,2.8824,2.9454,0.6900
-20,996,2.8006,11.9325,2.8486,2.8577,0.7600
-21,335,1.1085,4.8600,1.1212,1.2034,0.9400
-22,676,4.0827,13.6787,4.8422,4.6761,0.8300
-23,1027,3.2722,12.5725,3.5617,3.7747,0.8700

chart_patterns.py

@@ -12,10 +12,9 @@ Chart Patterns*) that are unambiguous from OHLC at bar close:
                                       the interim peak (neckline)
     double_top     bearish reversal   mirrored
     inv_hs         bullish reversal   inverse head & shoulders, break of
-                                      the neckline (the LOWER of the two
-                                      peaks flanking the head — the earlier
-                                      horizontal trigger, no sloped fits;
-                                      validated as-is, see detector note)
+                                      the neckline (max of the two peaks
+                                      flanking the head — conservative
+                                      horizontal line, no sloped fits)
     hs_top         bearish reversal   head & shoulders top, mirrored
     bull_flag      bullish continuation  strong pole, shallow ordered
                                       retrace, close at a new high
@@ -167,10 +166,8 @@ def _detect_hs(o, h, l, c, atr, cfg, bullish: bool) -> Optional[Dict]:
             return None
         if abs(s1 - s2) > cfg.cp_hs_shoulder_tol_atr * atr:
             return None
-        # Neckline: the LOWER of the two interim peaks — the EARLIER of
-        # the two horizontal triggers (max() would demand clearing both
-        # peaks). The whole validated CP book (M5 BUY-only) ran on this
-        # trigger — do not flip to max() without full re-validation.
+        # Neckline: the LOWER of the two interim peaks — the price must
+        # clear both, so this is the conservative horizontal trigger.
         p1 = float(h[i1 : i2 + 1].max())
         p2 = float(h[i2 : i3 + 1].max())
         neckline = min(p1, p2)

config.json.example

@@ -1,44 +0,0 @@
-{
-  "telegram": {
-    "token": "123456789:REPLACE_WITH_YOUR_BOT_TOKEN",
-    "chat_id": "REPLACE_WITH_OWNER_CHAT_ID",
-    "channel_id": "-100REPLACE_WITH_CHANNEL_ID_OR_EMPTY_STRING",
-    "alert_min_quality": "C"
-  },
-  "data_providers": {
-    "active": "oanda",
-    "streaming": "tiingo",
-    "twelve_data": {
-      "api_key": "REPLACE_WITH_YOUR_TWELVE_DATA_API_KEY"
-    },
-    "oanda": {},
-    "tiingo": {
-      "api_key": "REPLACE_WITH_YOUR_TIINGO_API_KEY_OR_EMPTY"
-    }
-  },
-  "scanning": {
-    "pairs": ["XAU/USD"],
-    "timeframes": ["M5", "M15", "H1", "H4"]
-  },
-  "pairs": {
-    "XAU/USD": {
-      "blocked_hours_utc": [0, 1, 2, 10, 11, 13, 16, 21]
-    }
-  },
-  "oanda": {
-    "account_id": "REPLACE_WITH_ACCOUNT_ID",
-    "api_token": "REPLACE_WITH_API_TOKEN",
-    "base_url": "https://api-fxpractice.oanda.com",
-    "lot_size": 1,
-    "quality_gate": "C",
-    "max_hold_bars": 300
-  },
-  "deepseek": {
-    "api_key": "REPLACE_WITH_YOUR_DEEPSEEK_API_KEY_OR_EMPTY",
-    "model": "deepseek-v4-flash",
-    "enabled": true,
-    "timeout_seconds": 12,
-    "daily_request_cap": 500
-  },
-  "log_level": "INFO"
-}

config.py

@@ -27,6 +27,14 @@ class SignalStrength(Enum):
     STRONG_SELL = -2
 
 
+class MarketSession(Enum):
+    SYDNEY = "sydney"
+    TOKYO = "tokyo"
+    LONDON = "london"
+    NEW_YORK = "new_york"
+    OVERLAP_LONDON_NY = "london_ny_overlap"
+
+
 # ── Supported pairs ──────────────────────────────────────────────
 XAUUSD = "XAU/USD"
 
@@ -34,7 +42,7 @@ XAUUSD = "XAU/USD"
 # default_spread: round-trip cost per trade in price units, charged by
 # the backtester when --spread is not given.
 PAIR_INFO = {
-    XAUUSD: {"display": "XAU/USD", "pip_value": 0.10,
+    XAUUSD: {"display": "XAU/USD", "pip_value": 0.10, "is_jpy_pair": False,
              "price_precision": 2, "default_spread": 0.30},
 }
 
@@ -46,12 +54,6 @@ class TechnicalConfig:
     ema_medium: int = 21
     ema_slow: int = 50
     ema_trend: int = 200
-    # 4-EMA mode only: score partial ribbon alignment (2-3 of 4 checks)
-    # as NEUTRAL instead of a weak BUY/SELL. Research knob (2026-08-04
-    # indicator audit: partial-alignment trades averaged -0.008R vs
-    # +0.230R fully aligned on the M5 12mo Oanda book); default False =
-    # legacy behavior until a replay campaign validates flipping it.
-    ema_partial_neutral: bool = False
 
     rsi_period: int = 14
     rsi_overbought: float = 70.0
@@ -100,8 +102,25 @@ class TechnicalConfig:
 @dataclass
 class SignalConfig:
     """Signal scoring & filtering."""
-    mtf_agreement_threshold: int = 2  # of 3 timeframes
+    min_confidence: float = 55.0
 
+    weights: Dict[str, float] = field(default_factory=lambda: {
+        "ema_trend": 1.0,
+        "rsi": 1.5,
+        "bollinger": 1.5,
+        "macd": 1.5,
+        "keltner": 1.5,
+        "roc": 1.5,
+        "candlestick": 1.0,
+        "divergence": 1.5,
+    })
+
+    mtf_agreement_threshold: int = 2  # of 3 timeframes
+    cooldown_bars: int = 5
+
+    default_risk_pct: float = 1.0
+    default_rr_ratio: float = 2.0
+    atr_sl_multiplier: float = 1.5
     min_rr_ratio: float = 0.5
     # TP tightening (May 2026) was tuned on the pre-spread backtester.
     # Re-tested 2026-07-08 with $0.30 spread + gap fills: 0.8 nearly
@@ -152,13 +171,8 @@ class NewsConfig:
     # against the validated 12mo XAU M5 book (547 trades, +139.9R)
     # showed they suppress 25-35 trades worth +4.6..+7.2R/yr — trades
     # entered near USD releases OUTPERFORM (+0.475R vs +0.256R avg).
-    # Audit 2026-07-11 (calendar-replay study; tooling since removed).
+    # Audit 2026-07-11 via backtest/research/calendar_store.
     event_risk_caps_signals: bool = False
-    # Whether headline sentiment moves the SIGNAL (aligned: +5 confidence;
-    # conflicting: 10-point penalty). Warnings are always emitted
-    # regardless. Default False = news is informational only; decisions
-    # are pure technical, matching the backtester (use_news=False).
-    sentiment_affects_signals: bool = False
     # Wider stay-out window after a high-impact release that surprised
     # vs forecast (inline releases use post_event_buffer_minutes).
     post_event_surprise_buffer_minutes: int = 45
@@ -180,14 +194,25 @@ class NewsConfig:
 
 @dataclass
 class XauStrategyConfig:
-    """XAU/USD strategy config."""
+    """XAU/USD strategy config.
+    Legacy gates (H4 RSI, H1 MACD, M15 score) removed in favor of
+    new momentum indicators (ROC, Keltner squeeze, ADX slope).
+    """
     enabled: bool = True
+    rsi_entry_threshold: float = 45.0
     tp_atr_multiplier: float = 1.0
     sl_atr_multiplier: float = 2.0
+    min_ema200_tfs: int = 2
+    allow_generic_trades: bool = True
     blocked_hours_utc: List[int] = field(default_factory=lambda: [0, 1, 2, 10, 11, 13, 16, 21])
     weekend_filter: bool = True
     min_atr_threshold: float = 1.0
     v2_sl_atr_multiplier: float = 2.0
+    buy_threshold_boost: int = 0
+    m5_confirmation_bars: int = 0
+    h4_rsi_gate: bool = False
+    h1_macd_gate: bool = False
+    rsi_zone: tuple = (40.0, 65.0)
     allowed_weekdays: List[int] = field(default_factory=lambda: [0, 1, 2, 3, 4, 6])  # Mon-Fri + Sun (Sydney open)
     sell_min_quality: List[str] = field(default_factory=lambda: ["A+", "A", "B"])
     # Volatility regimes in which SELL setups are allowed (low/normal-vol
@@ -209,16 +234,9 @@ class XauStrategyConfig:
     # M5 signals are noisy; require stronger confirmation before
     # generating trade setups. Without these, M5 is breakeven
     # (+0.01R/trade) with 21% drawdown.
-    # Relaxed 2026-07-30 (30→25, 10→5) in the relaxation campaign:
-    # the combined config (with the M5 hour-21 unblock) beat control
-    # in ALL 6 calendar years of the 5yr Oanda replay (+230.9R vs
-    # +168.0R, PF 1.26 vs 1.21, max DD 21.3 vs 27.0), 12mo +107.4R
-    # vs +84.6R PF 1.36 both halves positive, OOS 2024-06→2025-03
-    # +26.1R vs +17.2R, and at $0.70 spread +86.9R vs +66.4R.
-    # Score floors below 25 are vacuous (grade C requires |score|≥25).
     m5_require_mtf: bool = True
-    m5_min_abs_score: float = 25.0
-    m5_min_confirmation_score: float = 5.0
+    m5_min_abs_score: float = 30.0
+    m5_min_confirmation_score: float = 10.0
     # A-grade M5 SELLs enter at the extreme of temporary dips and
     # lose (-0.017R/trade on 163 trades) — maximum conviction at
     # the worst entry. Cap SELL quality so A-grade can't trigger
@@ -284,15 +302,6 @@ class XauStrategyConfig:
     cp_flag_max_bars: int = 12
     cp_flag_retrace: tuple = (0.1, 0.5)  # flag depth as fraction of pole
 
-    # ── Big-candle entry gate ──────────────────────────────────────
-    # When the recent window on the scan's own primary TF contains an
-    # abnormally large candle (bar range >= K x the PRIOR bar's ATR —
-    # institutional burst; price may reverse violently), no XAU entry
-    # of ANY kind may fire (core, XAU_PB, XAU_CP). 0.0 = disabled.
-    # Per-TF factories opt in only after the standard validation rule.
-    big_candle_atr_multiple: float = 0.0
-    big_candle_lookback_hours: float = 24.0
-
 
 @dataclass
 class FilterConfig:
@@ -329,6 +338,8 @@ class PlatformConfig:
 
     data_lookback_days: int = 60
 
+    twelve_data_api_key: str = "f563b22ffdfd4924955decbe1a8c8701"
+
     refresh_interval: int = 300
 
     scan_pairs: List[str] = field(
@@ -337,6 +348,7 @@ class PlatformConfig:
     scan_timeframes: List[Timeframe] = field(
         default_factory=lambda: [Timeframe.M5, Timeframe.M15, Timeframe.H1, Timeframe.H4]
     )
+    candle_buffer_seconds: int = 10
     ws_cooldown_seconds: int = 300
 
     log_level: str = "INFO"

dashboard_publisher.py

@@ -1,451 +0,0 @@
-#!/usr/bin/env python3
-"""
-XAU/USD static dashboard publisher.
-
-Standalone daemon that scans XAU/USD at candle closes and atomically
-writes `snapshot.json` + `news.json` (plus a copy of
-`static/dashboard.html`) into an Apache-served directory. Apache does
-all web serving — this process never opens a port, so a failure here
-can only ever mean a stale page.
-
-Independence rules (owner requirement — the telegram bot must be
-immune to anything on the web side):
-  * Runs as its own process; never imported by, and never imports,
-    `telegram_bot` / `oanda_daemon` (telegram_bot resolves credentials
-    at import). The confirmation map and boundary math are therefore
-    duplicated here as local constants.
-  * The only shared-code dependency beyond the standard signal path is
-    the additive `NewsEngine.get_week_events` accessor, which no
-    production signal path calls.
-
-Scheduling: every M15/H1/H4 boundary is also an M5 boundary (all
-intervals are multiples of 5, anchored to UTC midnight), so one
-sequential loop sleeps to the next M5 boundary (+10 s buffer) and scans
-whichever timeframes just closed. Due-classification uses the RAW wake
-time — flooring 12:00:10 to the 5-minute grid yields the 12:00 boundary
-that just closed. Do NOT subtract the buffer first: 12:00:10 − 15 s
-floors to 11:55 and silently skips every H1/H4 scan (v1 bug, 2026-08-02).
-
-Scans run sequentially M5→M15→H1→H4 through one shared data provider so
-later timeframes hit the warm boundary cache — that ordering IS the
-duplicate-fetch dedup (providers have no in-flight dedup; Oanda has no
-throttle). Keep it sequential.
-
-Run:  python dashboard_publisher.py [--once] [--out DIR]
-Out dir precedence: --out → DASHBOARD_OUT_DIR env → /var/www/trade
-"""
-
-import argparse
-import json
-import hashlib
-import logging
-import math
-import numbers
-import os
-import shutil
-import sys
-import time
-from datetime import datetime, timezone
-from logging.handlers import TimedRotatingFileHandler
-from typing import Dict, List, Optional
-
-from config import PlatformConfig, Timeframe, XAUUSD
-from main import report_to_dict
-from news_engine import NewsEngine
-from pair_signals import produce_signal
-from providers import make_data_provider
-import providers.base as _providers_base
-
-logger = logging.getLogger("dashboard")
-
-DASHBOARD_TFS = [Timeframe.M5, Timeframe.M15, Timeframe.H1, Timeframe.H4]
-
-# Local copy of the bot's confirmation map (see independence rules above).
-CONFIRMATIONS = {
-    Timeframe.M5:  [Timeframe.M15, Timeframe.H1],
-    Timeframe.M15: [Timeframe.M5, Timeframe.H1],
-    Timeframe.H1:  [Timeframe.M15, Timeframe.H4],
-    Timeframe.H4:  [Timeframe.H1, Timeframe.D1],
-}
-
-TF_INTERVAL_MINUTES = {
-    Timeframe.M5: 5, Timeframe.M15: 15, Timeframe.H1: 60, Timeframe.H4: 240,
-}
-
-DEFAULT_OUT_DIR = "/var/www/trade"
-SCAN_BUFFER_SECONDS = 10
-SNAPSHOT_FILE = "snapshot.json"
-NEWS_FILE = "news.json"
-PAGE_FILE = "dashboard.html"
-SUMMARY_PHP_FILE = "news_summary.php"
-# Repo files copied into the out dir at daemon start (git pull + restart
-# updates them). The PHP endpoint reads news.json in place, so it lives
-# next to the page.
-STATIC_FILES = [PAGE_FILE, SUMMARY_PHP_FILE]
-
-
-# ══════════════════════════════════════════════════════════════════
-#  Pure layer (no network, fully covered by test_all.py)
-# ══════════════════════════════════════════════════════════════════
-
-def _safe_num(v):
-    """Coerce to a JSON-safe float; NaN/±inf become None."""
-    try:
-        f = float(v)
-    except (TypeError, ValueError):
-        return v
-    if math.isnan(f) or math.isinf(f):
-        return None
-    return f
-
-
-def _sanitize(obj):
-    """Recursively make a payload safe for json.dumps(allow_nan=False):
-    NaN/inf → None, numpy scalars → native ints/floats."""
-    if isinstance(obj, bool):
-        return obj
-    if isinstance(obj, dict):
-        return {k: _sanitize(v) for k, v in obj.items()}
-    if isinstance(obj, (list, tuple)):
-        return [_sanitize(v) for v in obj]
-    if isinstance(obj, numbers.Integral):
-        return int(obj)
-    if isinstance(obj, numbers.Real):
-        return _safe_num(obj)
-    return obj
-
-
-def analysis_to_dict(analysis) -> Optional[dict]:
-    """Serialize a TechnicalAnalysis. Iterates analysis.indicators only —
-    never vars()/asdict(), which would drag dynamic pandas attrs
-    (_rsi_series, _macd_histogram, …) into the payload. ATR (weight 0)
-    is surfaced separately as context, not in the directional grid."""
-    if analysis is None:
-        return None
-    indicators = {}
-    atr_context = None
-    for name, ind in analysis.indicators.items():
-        if name.startswith("_"):
-            continue
-        entry = {
-            "signal": ind.signal.name,
-            "value": _safe_num(ind.value),
-            "confidence": _safe_num(ind.confidence),
-            "weight": _safe_num(ind.weight),
-            "description": ind.description,
-        }
-        if name == "atr":
-            atr_context = entry
-        elif ind.weight != 0:
-            indicators[name] = entry
-    return {
-        "timeframe": analysis.timeframe,
-        "timestamp": str(analysis.timestamp),
-        "composite_signal": analysis.composite_signal.name,
-        "composite_score": _safe_num(analysis.composite_score),
-        "confidence": _safe_num(analysis.confidence),
-        "market_regime": analysis.market_regime,
-        "volatility_regime": analysis.volatility_regime,
-        "pattern_signals": list(analysis.pattern_signals),
-        "key_levels": {k: _safe_num(v) for k, v in analysis.key_levels.items()},
-        "indicators": indicators,
-        "atr_context": atr_context,
-    }
-
-
-def report_to_webapp_dict(report) -> dict:
-    """main.report_to_dict + the per-TF detail the dashboard needs."""
-    d = report_to_dict(report)
-    d["primary_signal"] = report.primary_signal.name
-    d["primary_score"] = _safe_num(report.primary_score)
-    d["mtf_scores"] = {k: _safe_num(v) for k, v in report.mtf_scores.items()}
-    d["exhaustion_count"] = report.exhaustion_count
-    d["primary_analysis"] = analysis_to_dict(report.primary_analysis)
-    d["confirmation_analyses"] = [
-        analysis_to_dict(a) for a in report.confirmation_analyses
-    ]
-    return _sanitize(d)
-
-
-def event_to_dict(ev) -> dict:
-    """EconomicEvent → dict. surprise/minutes_until/is_high_impact are
-    properties, so asdict() would skip them — added explicitly.
-    actual/forecast/previous stay strings as published ('228K', '0.3%')."""
-    return {
-        "time": ev.time.astimezone(timezone.utc).isoformat(),
-        "currency": ev.currency,
-        "impact": ev.impact,
-        "event": ev.event,
-        "forecast": ev.forecast,
-        "previous": ev.previous,
-        "actual": ev.actual,
-        "surprise": ev.surprise,
-        "minutes_until": _safe_num(ev.minutes_until),
-        "is_high_impact": ev.is_high_impact,
-    }
-
-
-def headline_to_dict(h) -> dict:
-    # id uses the same sha1(headline) scheme as NewsEngine._llm_sent_cache;
-    # it is the join key the PHP summary endpoint whitelists against.
-    return {
-        "id": hashlib.sha1(h.headline.encode("utf-8")).hexdigest(),
-        "headline": h.headline,
-        "source": h.source,
-        "link": h.link,
-        "description": h.description,
-        "publisher": h.publisher,
-        "timestamp": h.timestamp.astimezone(timezone.utc).isoformat()
-                     if h.timestamp else None,
-        "currencies": list(h.currencies_mentioned),
-        "score": _safe_num(h.sentiment_score),
-        "keywords": list(h.keywords),
-        "score_by_currency": {
-            k: _safe_num(v) for k, v in h.sentiment_by_currency.items()
-        },
-    }
-
-
-def build_news_payload(context, week_events, now: datetime) -> dict:
-    """news.json: full calendar week + scored headlines + LLM summary."""
-    headlines = sorted(
-        context.news_headlines,
-        key=lambda h: h.timestamp or now, reverse=True,
-    )
-    return _sanitize({
-        "generated_at": now.isoformat(),
-        "llm_summary": context.llm_summary,
-        "sentiment": dict(context.news_sentiment),
-        "high_impact_warning": context.high_impact_warning,
-        "warning_message": context.warning_message,
-        "calendar_available": context.calendar_available,
-        "events": [event_to_dict(e) for e in week_events],
-        "headlines": [headline_to_dict(h) for h in headlines],
-    })
-
-
-def due_timeframes(wake_dt: datetime) -> List[Timeframe]:
-    """Timeframes whose candle just closed at the boundary this wake
-    belongs to. Takes the RAW wake time (boundary + buffer): flooring to
-    the 5-minute grid yields the boundary that just passed. Never
-    subtract the buffer first — see module docstring (v1 bug)."""
-    minutes = wake_dt.hour * 60 + wake_dt.minute
-    floored = minutes - (minutes % 5)
-    return [tf for tf in DASHBOARD_TFS if floored % TF_INTERVAL_MINUTES[tf] == 0]
-
-
-def seconds_until_boundary(interval_minutes: int, buffer: int = 10) -> float:
-    # Same math as telegram_bot.py / oanda_daemon.py — kept local so the
-    # dashboard never imports bot modules (independence rules above).
-    now = datetime.now(timezone.utc)
-    total_minutes = now.hour * 60 + now.minute
-    elapsed_minutes = total_minutes % interval_minutes
-    elapsed_seconds = elapsed_minutes * 60 + now.second + now.microsecond / 1e6
-    interval_seconds = interval_minutes * 60
-    remaining = interval_seconds - elapsed_seconds + buffer
-    # NOTE: remaining > buffer always (elapsed < interval by construction);
-    # a call at the exact boundary instant targets the NEXT boundary —
-    # steady-state loops wake at boundary+buffer, so only a process
-    # started in the boundary microsecond skips one scan (2026-08-04
-    # audit I42; the old `remaining <= buffer` guard was unreachable).
-    return remaining
-
-
-def _empty_tf_entry() -> dict:
-    return {"status": "warming", "scanned_at": None, "scan_seconds": None,
-            "error": None, "error_at": None, "report": None}
-
-
-def build_snapshot_payload(snapshots: Dict[str, dict], now: datetime) -> dict:
-    """snapshot.json: one entry per dashboard TF. Timeframes not yet
-    scanned appear as status="warming" so the page can render a
-    placeholder instead of a hole. server_time_epoch lets the client
-    compute its clock offset once so countdowns don't drift."""
-    timeframes = {}
-    for tf in DASHBOARD_TFS:
-        interval = TF_INTERVAL_MINUTES[tf]
-        minutes = now.hour * 60 + now.minute
-        secs_into = (minutes % interval) * 60 + now.second + now.microsecond / 1e6
-        next_close = now.timestamp() + (interval * 60 - secs_into)
-        entry = _empty_tf_entry()
-        entry.update(snapshots.get(tf.value) or {})
-        entry["next_close_epoch"] = next_close
-        entry["next_scan_epoch"] = next_close + SCAN_BUFFER_SECONDS
-        timeframes[tf.value] = entry
-    return _sanitize({
-        "symbol": XAUUSD,
-        "server_time_epoch": now.timestamp(),
-        "market_closed": _providers_base._is_forex_weekend(now),
-        "timeframes": timeframes,
-    })
-
-
-def publish(obj: dict, path: str) -> None:
-    """Atomic JSON write: temp file + os.replace, so Apache never serves
-    a torn file. allow_nan=False — a NaN slipping through would render
-    the page blank (JSON.parse rejects bare NaN)."""
-    data = json.dumps(obj, allow_nan=False, separators=(",", ":"))
-    tmp = path + ".tmp"
-    with open(tmp, "w", encoding="utf-8") as fh:
-        fh.write(data)
-    os.replace(tmp, path)
-
-
-# ══════════════════════════════════════════════════════════════════
-#  Runtime layer (constructed only from main(); never at import)
-# ══════════════════════════════════════════════════════════════════
-
-def _setup_logging(level: str = "INFO") -> None:
-    log_dir = os.path.join(os.path.dirname(os.path.abspath(__file__)), "logs")
-    os.makedirs(log_dir, exist_ok=True)
-    log_file = os.path.join(log_dir, "dashboard.log")
-
-    fmt = logging.Formatter(
-        "%(asctime)s [%(levelname)s] %(name)s: %(message)s",
-        datefmt="%Y-%m-%d %H:%M:%S")
-
-    file_handler = TimedRotatingFileHandler(
-        log_file, when="h", interval=3, backupCount=240, utc=True)
-    file_handler.setFormatter(fmt)
-
-    console_handler = logging.StreamHandler(sys.stdout)
-    console_handler.setFormatter(fmt)
-
-    logging.basicConfig(
-        level=getattr(logging, level.upper(), logging.INFO),
-        handlers=[file_handler, console_handler],
-        force=True)
-
-
-class DashboardPublisher:
-    def __init__(self, out_dir: str):
-        self.out_dir = out_dir
-        self.snapshots: Dict[str, dict] = {}
-        self.news_payload: Optional[dict] = None
-        self._dp = None
-        self._news_engine: Optional[NewsEngine] = None
-
-    def _provider(self):
-        if self._dp is None:
-            self._dp = make_data_provider(PlatformConfig())
-        return self._dp
-
-    def _news(self) -> NewsEngine:
-        if self._news_engine is None:
-            self._news_engine = NewsEngine(PlatformConfig().news)
-        return self._news_engine
-
-    def install_static(self) -> None:
-        """Copy the repo's static files (page + PHP summary endpoint)
-        into the out dir (atomic), so git pull + restart also updates
-        them. The ai/ cache dir is NOT created here — the daemon runs as
-        a different user than Apache; that dir is a one-time deploy step
-        (deploy/README-dashboard.md)."""
-        static_dir = os.path.join(
-            os.path.dirname(os.path.abspath(__file__)), "static")
-        for name in STATIC_FILES:
-            src = os.path.join(static_dir, name)
-            if not os.path.exists(src):
-                logger.warning(f"{src} missing — not installed")
-                continue
-            dst = os.path.join(self.out_dir, name)
-            shutil.copyfile(src, dst + ".tmp")
-            os.replace(dst + ".tmp", dst)
-
-    def scan_tf(self, tf: Timeframe) -> None:
-        started = time.time()
-        try:
-            report = produce_signal(XAUUSD, tf, self._provider(),
-                                    CONFIRMATIONS[tf])
-            self.snapshots[tf.value] = {
-                "status": "ok",
-                "scanned_at": datetime.now(timezone.utc).isoformat(),
-                "scan_seconds": round(time.time() - started, 2),
-                "error": None,
-                "error_at": None,
-                "report": report_to_webapp_dict(report),
-            }
-            logger.info(f"{tf.value}: {report.final_signal.name} "
-                        f"q={report.signal_quality or '-'} "
-                        f"({time.time() - started:.1f}s)")
-        except Exception as e:
-            # Keep the last good report; the page shows it with an
-            # error badge instead of going blank.
-            logger.exception(f"{tf.value} scan failed")
-            entry = self.snapshots.get(tf.value) or _empty_tf_entry()
-            entry["status"] = "error"
-            entry["error"] = f"{type(e).__name__}: {e}"
-            entry["error_at"] = datetime.now(timezone.utc).isoformat()
-            self.snapshots[tf.value] = entry
-
-    def refresh_news(self) -> None:
-        try:
-            ne = self._news()
-            context = ne.get_fundamental_context()
-            week = ne.get_week_events()
-            self.news_payload = build_news_payload(
-                context, week, datetime.now(timezone.utc))
-        except Exception:
-            logger.exception("news refresh failed — keeping last payload")
-
-    def publish_snapshot(self) -> None:
-        publish(build_snapshot_payload(
-            self.snapshots, datetime.now(timezone.utc)),
-            os.path.join(self.out_dir, SNAPSHOT_FILE))
-
-    def publish_news(self) -> None:
-        if self.news_payload is not None:
-            publish(self.news_payload, os.path.join(self.out_dir, NEWS_FILE))
-
-    def publish_all(self) -> None:
-        self.publish_snapshot()
-        self.publish_news()
-
-    def run_cycle(self, tfs: List[Timeframe]) -> None:
-        for tf in tfs:  # sequential — see module docstring (cache dedup)
-            self.scan_tf(tf)
-        # Snapshot lands BEFORE the news refresh: when the news TTL has
-        # expired, refresh_news does FF + 4 RSS fetches (and possibly
-        # DeepSeek calls) — 20-30 s that must not keep fresh signals off
-        # the page.
-        self.publish_snapshot()
-        self.refresh_news()
-        self.publish_news()
-
-    def run_forever(self) -> None:
-        self.run_cycle(DASHBOARD_TFS)  # populate the page at boot
-        while True:
-            delay = seconds_until_boundary(5, buffer=SCAN_BUFFER_SECONDS)
-            logger.info(f"next M5 boundary in {delay:.0f}s")
-            time.sleep(delay)
-            try:
-                due = due_timeframes(datetime.now(timezone.utc))
-                logger.info(f"boundary wake — scanning {[t.value for t in due]}")
-                self.run_cycle(due)
-            except Exception:
-                logger.exception("cycle failed")
-
-
-def main(argv=None) -> None:
-    ap = argparse.ArgumentParser(description="XAU/USD static dashboard publisher")
-    ap.add_argument("--once", action="store_true",
-                    help="one scan+publish cycle, then exit")
-    ap.add_argument("--out",
-                    default=os.environ.get("DASHBOARD_OUT_DIR", DEFAULT_OUT_DIR),
-                    help="output directory served by Apache "
-                         f"(default: $DASHBOARD_OUT_DIR or {DEFAULT_OUT_DIR})")
-    args = ap.parse_args(argv)
-
-    _setup_logging()
-    os.makedirs(args.out, exist_ok=True)
-    pub = DashboardPublisher(args.out)
-    pub.install_static()
-    if args.once:
-        pub.run_cycle(DASHBOARD_TFS)
-        print(f"Published to {args.out}")
-        return
-    pub.run_forever()
-
-
-if __name__ == "__main__":
-    main()

data_provider.py

@@ -0,0 +1,386 @@
+"""
+ForexBot2 — Data Provider (Twelve Data SDK)
+==========================================================
+Fetches and validates OHLCV via the official `twelvedata` Python SDK.
+"""
+
+import logging
+import os
+import threading
+import time
+from collections import deque
+from typing import Dict, Optional
+
+import pandas as pd
+from twelvedata import TDClient
+
+from config import PlatformConfig, Timeframe, XAUUSD, PAIR_INFO
+
+logger = logging.getLogger(__name__)
+
+
+def _is_forex_weekend(now=None) -> bool:
+    """Forex market closed: Friday 22:00 UTC → Sunday 22:00 UTC."""
+    if now is None:
+        now = pd.Timestamp.now(tz="UTC")
+    wd = now.weekday()
+    hour = now.hour
+    if wd == 4 and hour >= 22:
+        return True
+    if wd == 5:
+        return True
+    if wd == 6 and hour < 22:
+        return True
+    return False
+
+
+class DataIntegrityError(Exception):
+    """Raised when data fails validation checks."""
+    pass
+
+
+class ForexDataProvider:
+    """Twelve Data SDK client + on-disk parquet cache."""
+
+    TD_INTERVALS = {
+        Timeframe.M1:  "1min",
+        Timeframe.M5:  "5min",
+        Timeframe.M15: "15min",
+        Timeframe.M30: "30min",
+        Timeframe.H1:  "1h",
+        Timeframe.H4:  "4h",
+        Timeframe.D1:  "1day",
+        Timeframe.W1:  "1week",
+    }
+
+    BARS_PER_DAY = {
+        Timeframe.M1: 1440, Timeframe.M5: 288, Timeframe.M15: 96,
+        Timeframe.M30: 48, Timeframe.H1: 24, Timeframe.H4: 6,
+        Timeframe.D1: 1, Timeframe.W1: 0.143,
+    }
+
+    CANDLE_SECONDS = {
+        Timeframe.M1: 60, Timeframe.M5: 300, Timeframe.M15: 900,
+        Timeframe.M30: 1800, Timeframe.H1: 3600, Timeframe.H4: 14400,
+        Timeframe.D1: 86400, Timeframe.W1: 604800,
+    }
+
+    RATE_LIMIT = 55
+    RATE_WINDOW = 60.0
+
+    def __init__(self, config: PlatformConfig):
+        self.config = config
+        self._api_key = (
+            os.environ.get("TWELVE_DATA_API_KEY")
+            or config.twelve_data_api_key
+        )
+        self._cache: Dict[str, pd.DataFrame] = {}
+        self._cache_timestamps: Dict[str, float] = {}
+        # The bot shares one provider across a thread pool; cache eviction
+        # and the throttle deque are check-then-act and need locking.
+        self._cache_lock = threading.Lock()
+        self._throttle_lock = threading.Lock()
+        self._request_times: deque = deque()
+        self._td: Optional[TDClient] = (
+            TDClient(apikey=self._api_key) if self._api_key else None
+        )
+
+    @staticmethod
+    def _to_td_symbol(symbol: str) -> str:
+        if "/" in symbol:
+            return symbol
+        clean = symbol.replace("=X", "")
+        return f"{clean[:3]}/{clean[3:]}"
+
+    # ── Public API ─────────────────────────────────────────────────
+
+    def fetch_ohlcv(
+        self,
+        symbol: str = XAUUSD,
+        timeframe: Timeframe = Timeframe.H1,
+        lookback_days: Optional[int] = None,
+    ) -> pd.DataFrame:
+        """Fetch OHLCV with in-memory caching for live mode."""
+        if lookback_days is None:
+            lookback_days = self.config.data_lookback_days
+
+        cache_key = f"{symbol}_{timeframe.value}"
+        now = time.time()
+
+        cache_ttl = self.CANDLE_SECONDS.get(timeframe, self.config.refresh_interval)
+        with self._cache_lock:
+            cached = self._cache.get(cache_key)
+            cached_at = self._cache_timestamps.get(cache_key)
+        if (
+            cached is not None
+            and cached_at is not None
+            and (now - cached_at) < cache_ttl
+            and not cached.empty
+        ):
+            if (self._is_data_fresh(cached, symbol, timeframe)
+                    and self._has_latest_closed_bar(cached, timeframe)):
+                return cached
+            logger.warning(f"{symbol} {timeframe.value}: cached data is stale, refetching")
+            with self._cache_lock:
+                self._cache.pop(cache_key, None)
+                self._cache_timestamps.pop(cache_key, None)
+
+        outputsize = max(
+            300,
+            min(int(lookback_days * self.BARS_PER_DAY[timeframe]) + 20, 5000),
+        )
+        df = self._fetch_twelve_data(symbol, timeframe, outputsize)
+        df = self._drop_forming_candle(df, timeframe)
+        df = self._validate_and_clean(df, symbol, timeframe)
+
+        if not self._is_data_fresh(df, symbol, timeframe):
+            logger.warning(
+                f"STALE DATA from API: {symbol} {timeframe.value} — "
+                f"last bar {df.index[-1]} is too old. Signal quality may be degraded."
+            )
+
+        with self._cache_lock:
+            self._cache[cache_key] = df
+            self._cache_timestamps[cache_key] = now
+        return df
+
+    def _has_latest_closed_bar(self, df: pd.DataFrame, timeframe: Timeframe) -> bool:
+        """Cached data must include the most recent fully closed bar.
+
+        The TTL alone is wall-clock based: a snapshot fetched mid-candle is
+        still within TTL at the next candle boundary but is missing the bar
+        that just closed, so every boundary scan would analyze — and run the
+        bar-close session check against — the previous bar.
+        """
+        now = pd.Timestamp.now(tz="UTC")
+        if _is_forex_weekend(now):
+            return True
+        candle_seconds = self.CANDLE_SECONDS.get(timeframe, 3600)
+        dur = pd.Timedelta(seconds=candle_seconds)
+        expected_label = now.floor(dur) - dur
+        return df.index[-1] >= expected_label
+
+    def _drop_forming_candle(self, df: pd.DataFrame, timeframe: Timeframe) -> pd.DataFrame:
+        """Drop the still-forming last candle so analysis only sees closed bars.
+
+        Twelve Data's time_series includes the current partial candle; the
+        scanner fires seconds after the boundary, so without this guard every
+        indicator reads a seconds-old stub bar and signals repaint.
+        """
+        if df is None or df.empty:
+            return df
+        candle_seconds = self.CANDLE_SECONDS.get(timeframe, 3600)
+        now = pd.Timestamp.now(tz="UTC")
+        if df.index[-1] + pd.Timedelta(seconds=candle_seconds) > now:
+            df = df.iloc[:-1]
+        return df
+
+    def _is_data_fresh(self, df: pd.DataFrame, symbol: str, timeframe: Timeframe) -> bool:
+        """Check if the last candle is within acceptable age for this timeframe."""
+        if df is None or df.empty:
+            return False
+
+        now = pd.Timestamp.now(tz="UTC")
+
+        if _is_forex_weekend(now):
+            return True
+
+        last_bar = df.index[-1]
+        candle_seconds = self.CANDLE_SECONDS.get(timeframe, 3600)
+        max_age_seconds = candle_seconds * 3
+        age_seconds = (now - last_bar).total_seconds()
+
+        if age_seconds > max_age_seconds:
+            logger.warning(
+                f"{symbol} {timeframe.value}: last bar {last_bar.strftime('%Y-%m-%d %H:%M')} "
+                f"is {age_seconds / 60:.0f}m old (limit: {max_age_seconds / 60:.0f}m)"
+            )
+            return False
+
+        return True
+
+    def clear_cache(self) -> int:
+        """Wipe all cached data. Returns number of entries cleared."""
+        count = len(self._cache)
+        self._cache.clear()
+        self._cache_timestamps.clear()
+        logger.info(f"Cache cleared: {count} entries wiped")
+        return count
+
+    def get_latest_price(self, symbol: str = XAUUSD) -> Optional[float]:
+        try:
+            df = self.fetch_ohlcv(symbol, Timeframe.M5, lookback_days=1)
+            if df is not None and not df.empty:
+                return float(df["Close"].iloc[-1])
+        except Exception:
+            pass
+        return None
+
+    def get_spread_estimate(self, symbol: str = XAUUSD) -> float:
+        try:
+            df = self.fetch_ohlcv(symbol, Timeframe.M5, lookback_days=1)
+            if df is not None and not df.empty:
+                return float((df["High"] - df["Low"]).median())
+        except Exception:
+            pass
+        info = PAIR_INFO.get(symbol, PAIR_INFO[XAUUSD])
+        return info["pip_value"]
+
+    def get_pair_info(self, symbol: str = XAUUSD) -> Dict:
+        info = PAIR_INFO.get(symbol, PAIR_INFO[XAUUSD])
+        return {"symbol": symbol, **info}
+
+    # ── Private ────────────────────────────────────────────────────
+
+    def _throttle(self, credits: int = 1) -> None:
+        """Pro tier: 55 credits / minute. Block until budget allows."""
+        while True:
+            with self._throttle_lock:
+                now = time.time()
+                while self._request_times and now - self._request_times[0] >= self.RATE_WINDOW:
+                    self._request_times.popleft()
+
+                available = self.RATE_LIMIT - len(self._request_times)
+                if available >= credits:
+                    for _ in range(credits):
+                        self._request_times.append(time.time())
+                    return
+
+                used = len(self._request_times)
+                wait = (self._request_times[0] + self.RATE_WINDOW) - now + 0.2
+            logger.info(
+                f"Rate limit: {used}/{self.RATE_LIMIT} credits used "
+                f"— waiting {wait:.1f}s..."
+            )
+            time.sleep(wait)
+
+    def _fetch_twelve_data(
+        self,
+        symbol: str,
+        timeframe: Timeframe,
+        outputsize: int,
+        start_date: Optional[str] = None,
+        end_date: Optional[str] = None,
+    ) -> pd.DataFrame:
+        if self._td is None:
+            raise DataIntegrityError(
+                "No Twelve Data API key. Set TWELVE_DATA_API_KEY env var "
+                "or config.twelve_data_api_key."
+            )
+
+        td_symbol = self._to_td_symbol(symbol)
+        interval = self.TD_INTERVALS[timeframe]
+
+        self._throttle(credits=1)
+
+        kwargs = {
+            "symbol": td_symbol,
+            "interval": interval,
+            "outputsize": min(outputsize, 5000),
+            "timezone": "UTC",
+        }
+        if start_date:
+            kwargs["start_date"] = start_date
+        if end_date:
+            kwargs["end_date"] = end_date
+
+        try:
+            df = self._td.time_series(**kwargs).as_pandas()
+        except Exception as e:
+            raise DataIntegrityError(
+                f"Twelve Data fetch failed for {td_symbol}: {e}"
+            ) from e
+
+        if df is None or df.empty:
+            raise DataIntegrityError(f"Empty response from Twelve Data for {td_symbol}")
+
+        # SDK returns lowercase columns; engine expects capitalised.
+        df = df.rename(columns={
+            "open": "Open", "high": "High", "low": "Low",
+            "close": "Close", "volume": "Volume",
+        })
+        for col in ("Open", "High", "Low", "Close", "Volume"):
+            if col in df.columns:
+                df[col] = pd.to_numeric(df[col], errors="coerce")
+        if "Volume" not in df.columns:
+            df["Volume"] = 0.0
+
+        # SDK returns newest-first; engine + indicators want oldest-first.
+        df = df.sort_index()
+        if df.index.tz is None:
+            df.index = df.index.tz_localize("UTC")
+        df.index.name = "Datetime"
+
+        logger.debug(f"Fetched {len(df)} bars for {td_symbol} @ {interval}")
+        return df
+
+    def _validate_and_clean(
+        self, df: pd.DataFrame, symbol: str, timeframe: Timeframe
+    ) -> pd.DataFrame:
+        if df is None or df.empty:
+            raise DataIntegrityError(f"Empty dataset for {symbol}")
+
+        original_len = len(df)
+
+        df = df.dropna(subset=["Open", "High", "Low", "Close"], how="all")
+        df = df.ffill(limit=3)
+        df = df.dropna(subset=["Close"])
+
+        df = df.sort_index()
+        df = df[~df.index.duplicated(keep="last")]
+
+        now = pd.Timestamp.utcnow()
+        future_mask = df.index > now + pd.Timedelta(minutes=5)
+        if future_mask.any():
+            logger.warning(f"{symbol}: Dropping {future_mask.sum()} future timestamps")
+            df = df[~future_mask]
+
+        invalid_ohlc = (
+            (df["High"] < df["Low"]) |
+            (df["High"] < df["Open"]) |
+            (df["High"] < df["Close"]) |
+            (df["Low"] > df["Open"]) |
+            (df["Low"] > df["Close"])
+        )
+        if invalid_ohlc.any():
+            logger.warning(f"{symbol}: Fixing {invalid_ohlc.sum()} OHLC-inconsistent bars")
+            for idx in df[invalid_ohlc].index:
+                vals = [df.loc[idx, "Open"], df.loc[idx, "Close"]]
+                df.loc[idx, "High"] = max(vals + [df.loc[idx, "High"]])
+                df.loc[idx, "Low"]  = min(vals + [df.loc[idx, "Low"]])
+
+        returns = df["Close"].pct_change()
+        std_ret = returns.std()
+        if std_ret > 0:
+            dev = returns - returns.mean()
+            outlier_mask = dev.abs() > 10 * std_ret
+            # A spike bar makes the NEXT bar's return look extreme too (it is
+            # measured against the spike). If the previous bar is flagged and
+            # this bar reverses direction, it is the legitimate snap-back —
+            # keep it, or every spike would punch a two-bar hole in the data.
+            recovery = (
+                outlier_mask
+                & outlier_mask.shift(1, fill_value=False)
+                & (dev * dev.shift(1) < 0)
+            )
+            outlier_mask = outlier_mask & ~recovery
+            if outlier_mask.any():
+                logger.warning(f"{symbol}: Removing {outlier_mask.sum()} extreme outlier bars")
+                df = df[~outlier_mask]
+
+        min_required = max(
+            self.config.technical.ema_trend,
+            200,
+        ) + 10
+
+        if len(df) < min_required:
+            logger.warning(
+                f"{symbol} {timeframe.value}: Only {len(df)} bars "
+                f"(need {min_required}). Some indicators will be partial."
+            )
+
+        cleaned = original_len - len(df)
+        if cleaned > 0:
+            logger.info(f"{symbol}: Cleaned {cleaned}/{original_len} bars")
+
+        return df

deploy/README-dashboard.md

@@ -1,157 +0,0 @@
-# Deploying the web dashboard (live LAMP server)
-
-The dashboard is a **static publisher**: `dashboard_publisher.py` runs as its
-own daemon, scans XAU/USD at candle closes, and atomically writes
-`dashboard.html` + `news_summary.php` + `snapshot.json` + `news.json` into a
-folder Apache already serves. Apache does all web serving — no new port, no
-Python web server, and the telegram bot process is completely unaffected by
-anything that happens here (see the module docstring for the independence
-rules). The one dynamic piece is `news_summary.php` (§6): an on-demand
-DeepSeek summary for a clicked headline, whitelist-gated against news.json.
-
-All steps below run **on the live server** (`/home/Forex/ForexBot2`).
-
-## 1. Get the code
-
-```bash
-cd /home/Forex/ForexBot2
-git pull
-venv/bin/python test_all.py        # optional sanity: suite must be green
-```
-
-No new Python dependencies are required.
-
-## 2. Make the web folder writable
-
-The dashboard publishes straight into the portal docroot `/var/www/trade/`
-(owner decision 2026-08-03) — it adds exactly four files there:
-`dashboard.html`, `news_summary.php`, `snapshot.json`, `news.json` (plus the
-apache-owned `ai/` cache dir from §6). The daemon user must be able to write
-them:
-
-```bash
-# if the portal files can be owned by the daemon user:
-sudo chown Forex: /var/www/trade
-# otherwise grant write without changing ownership:
-sudo setfacl -m u:Forex:rwx /var/www/trade
-```
-
-(A different path is one edit away: the unit's `Environment=DASHBOARD_OUT_DIR=`
-line, or `--out` for one-off runs.)
-
-SELinux (RHEL/Alma/CentOS enforcing): directories under `/var/www` normally
-already carry `httpd_sys_content_t`, and new files inherit the parent label, so
-no action is usually needed. If Apache 403s despite correct permissions:
-
-```bash
-sudo restorecon -R /var/www/trade
-```
-
-## 3. Smoke-test once
-
-```bash
-venv/bin/python dashboard_publisher.py --once
-ls /var/www/trade    # dashboard.html snapshot.json news.json added next to the portal
-```
-
-Then open `http://<server>/dashboard.html` (path relative to however the
-vhost maps `/var/www/trade`) — all four timeframe cards, the calendar, and
-the newswire should populate.
-
-## 4. Install the daemon
-
-```bash
-sudo cp deploy/gold-dashboard.service /etc/systemd/system/
-# edit User= if the repo owner is not "Forex"
-sudo systemctl daemon-reload
-sudo systemctl enable --now gold-dashboard
-systemctl status gold-dashboard
-tail -f logs/dashboard.log
-```
-
-The daemon scans all four timeframes at boot, then wakes at every M5
-boundary (+10 s) and rescans whichever timeframes just closed. Logs rotate
-every 3 h (240 backups), same scheme as the bot.
-
-## 5. Optional Apache polish
-
-The page polls its JSON with conditional GETs (`cache: "no-cache"` → Apache
-answers unchanged files with 304), so no Apache config is strictly needed. To
-be explicit anyway:
-
-```apache
-<Directory "/var/www/trade">
-    <FilesMatch "\.json$">
-        Header set Cache-Control "no-cache"
-    </FilesMatch>
-</Directory>
-```
-
-The page is deliberately public (owner decision 2026-08-03). To restrict it
-later, add standard Apache basic auth (`htpasswd` + `Require valid-user`) on
-that directory — no code changes needed.
-
-## 6. AI summary endpoint (PHP) — one-time setup
-
-Clicking a newswire headline opens a detail dialog that fetches an on-demand
-DeepSeek gold-impact summary from `news_summary.php` (installed next to the
-page by the daemon). Owner decision 2026-08-07: a real on-demand endpoint on
-the public page, accepted with these guards baked into the PHP — the id must
-match a headline in the published news.json (arbitrary text can never reach
-the LLM), each headline's summary is cached forever in `ai/` (≤1 API call per
-headline, ever), and a file-based daily cap fails **closed** on I/O errors.
-
-Requirements: `php` + the curl extension (`php -m | grep curl`).
-
-```bash
-# 1. SELinux: allow httpd outbound HTTPS — WITHOUT this every summary call
-#    dies as a curl timeout even with perfect perms (RHEL-family default).
-sudo setsebool -P httpd_can_network_connect 1
-
-# 2. API key OUTSIDE the docroot, readable by Apache only:
-sudo install -d -o root -g root -m 755 /etc/gold-dashboard
-sudo sh -c 'umask 137; echo "sk-..." > /etc/gold-dashboard/deepseek.key'
-sudo chown root:apache /etc/gold-dashboard/deepseek.key   # mode 640
-
-# 3. apache-owned cache dir (the daemon runs as Forex and can't create it):
-sudo install -d -o apache -g apache -m 755 /var/www/trade/ai
-sudo semanage fcontext -a -t httpd_sys_rw_content_t '/var/www/trade/ai(/.*)?'
-sudo restorecon -Rv /var/www/trade/ai
-```
-
-Knobs (Apache `SetEnv` in the vhost, both optional):
-`AI_SUMMARY_DAILY_CAP` (default 150), `DEEPSEEK_KEY_FILE` (default
-`/etc/gold-dashboard/deepseek.key`).
-
-Verify from the shell — expect 400 / 404 / 200 / 200-from-cache:
-
-```bash
-curl -s "http://<server>/news_summary.php?id=zzz"                 # {"error":"bad_id"}
-curl -s "http://<server>/news_summary.php?id=$(printf 'a%.0s' {1..40})"  # unknown_id
-ID=$(python3 -c "import json;print(json.load(open('/var/www/trade/news.json'))['headlines'][0]['id'])")
-curl -s "http://<server>/news_summary.php?id=$ID"   # real summary (2-12 s)
-curl -s "http://<server>/news_summary.php?id=$ID"   # instant, from ai/<id>.json
-```
-
-## Operational notes
-
-- **Updating**: `git pull` + `systemctl restart gold-dashboard`. The daemon
-  re-copies `static/dashboard.html` and `static/news_summary.php` into the
-  web folder at startup.
-- **If the daemon dies**, systemd restarts it in 30 s; worst case the page
-  shows stale data with its own "updated N min ago" evidence. Apache and the
-  telegram bot are unaffected.
-- **DeepSeek**: the publisher process has its own daily request cap
-  (`deepseek.daily_request_cap`, default 500) separate from the bot's —
-  expect roughly up to ~430 extra calls/day worst case, V4-Flash pricing.
-- **Two independent DeepSeek meters** hit the same API key: the publisher's
-  in-process cap above, and the PHP endpoint's file cap (`ai/.ht_cap.json`,
-  default 150/day). They don't communicate — worst-case spend is their sum;
-  DeepSeek-side spend alerting is the true backstop. Kept separate on
-  purpose (independence rule: the PHP never imports/reads bot code or
-  config.json).
-- **Restart window**: for the first minutes after a daemon restart the new
-  page can meet a pre-upgrade news.json without headline ids — those rows
-  simply render non-clickable until the first news publish.
-- **Credentials**: uses the same env/.env/config.json resolution as the rest
-  of the platform (Oanda candles feed; news needs no key; DeepSeek optional).

deploy/gold-dashboard.service

@@ -1,23 +0,0 @@
-# systemd unit for the XAU/USD static dashboard publisher.
-# Install (as root, on the live server):
-#   cp deploy/gold-dashboard.service /etc/systemd/system/
-#   systemctl daemon-reload && systemctl enable --now gold-dashboard
-# See deploy/README-dashboard.md for the full checklist.
-
-[Unit]
-Description=XAU/USD dashboard publisher (writes static JSON/HTML for Apache)
-After=network-online.target
-Wants=network-online.target
-
-[Service]
-Type=simple
-# Adjust to the account that owns /home/Forex/ForexBot2:
-User=Forex
-WorkingDirectory=/home/Forex/ForexBot2
-Environment=DASHBOARD_OUT_DIR=/var/www/trade
-ExecStart=/home/Forex/ForexBot2/venv/bin/python dashboard_publisher.py
-Restart=always
-RestartSec=30
-
-[Install]
-WantedBy=multi-user.target

docs/architecture_diagram.py

@@ -1,199 +0,0 @@
-"""ForexBot2 architecture diagram → docs/architecture.png.
-
-Regenerate after architecture changes:  venv/bin/python docs/architecture_diagram.py
-(matplotlib only — no graphviz on the dev box)."""
-import os
-import matplotlib
-matplotlib.use("Agg")
-import matplotlib.pyplot as plt
-from matplotlib.patches import FancyBboxPatch, FancyArrowPatch
-
-W, H = 100.0, 100.0
-fig, ax = plt.subplots(figsize=(16, 11.5))
-ax.set_xlim(0, W); ax.set_ylim(0, H); ax.axis("off")
-
-INK = "#263238"; MUT = "#607D8B"; ARR = "#455A64"; HAUL = "#7B8FA1"
-
-def box(x, y, w, h, title, lines, fc, ec, title_fs=8.2, body_fs=6.6, lw=1.3):
-    ax.add_patch(FancyBboxPatch((x, y), w, h,
-                 boxstyle="round,pad=0.35,rounding_size=0.9",
-                 facecolor=fc, edgecolor=ec, linewidth=lw, zorder=2))
-    cy = y + h - 1.9
-    ax.text(x + w / 2, cy, title, ha="center", va="center", fontsize=title_fs,
-            fontweight="bold", color=INK, zorder=3)
-    cy -= 2.3
-    for ln in lines:
-        ax.text(x + w / 2, cy, ln, ha="center", va="center", fontsize=body_fs,
-                color=INK, zorder=3)
-        cy -= 1.85
-
-def band(x, y, w, h, label, color):
-    ax.add_patch(FancyBboxPatch((x, y), w, h,
-                 boxstyle="round,pad=0.3,rounding_size=1.2",
-                 facecolor="none", edgecolor=color, linewidth=1.0,
-                 linestyle=(0, (4, 3)), zorder=1))
-    ax.text(x + 0.9, y + h - 1.4, label, ha="left", va="center",
-            fontsize=7.2, fontweight="bold", color=color, zorder=3)
-
-def arrow(p1, p2, color=ARR, lw=1.5, style="-|>", conn="arc3,rad=0.0",
-          label=None, lx=0, ly=0, ls="solid", fs=6.4, rot=0):
-    ax.add_patch(FancyArrowPatch(p1, p2, arrowstyle=style, mutation_scale=11,
-                 color=color, linewidth=lw, linestyle=ls,
-                 connectionstyle=conn, zorder=4))
-    if label:
-        mx, my = (p1[0] + p2[0]) / 2 + lx, (p1[1] + p2[1]) / 2 + ly
-        ax.text(mx, my, label, fontsize=fs, color=MUT, ha="center",
-                va="center", zorder=5, rotation=rot,
-                bbox=dict(boxstyle="round,pad=0.15", fc="white", ec="none", alpha=0.9))
-
-# ── title ────────────────────────────────────────────────────────────
-ax.text(2, 98.4, "ForexBot2 — XAU/USD Signal-Intelligence Platform",
-        fontsize=15, fontweight="bold", color=INK, ha="left")
-ax.text(2, 96.0, "one pipeline, four delivery surfaces · closed bars only · live == backtest parity",
-        fontsize=8.5, color=MUT, ha="left")
-
-EXT_FC, EXT_EC = "#ECEFF1", "#90A4AE"
-DAT_FC, DAT_EC = "#E3F2FD", "#5C9BD1"
-NWS_FC, NWS_EC = "#E0F2F1", "#3D9C8F"
-COR_FC, COR_EC = "#FFFDF5", "#C9A227"
-REP_FC, REP_EC = "#FFE9A8", "#B8860B"
-DEL_FC, DEL_EC = "#E8F5E9", "#5FAF63"
-BTS_FC, BTS_EC = "#EDE7F6", "#8E6BC6"
-
-# ── external services (top band) ─────────────────────────────────────
-band(1, 85.0, 98, 9.4, "EXTERNAL SERVICES", "#78909C")
-box(2.5, 86.2, 15.5, 6.4, "Oanda v20 API",
-    ["candles (live feed)", "practice orders"], EXT_FC, EXT_EC)
-box(19.5, 86.2, 15.5, 6.4, "Twelve Data · Tiingo",
-    ["registered fallbacks", "(Tiingo: tick stream)"], EXT_FC, EXT_EC)
-box(36.5, 86.2, 15.5, 6.4, "Forex Factory",
-    ["economic calendar", "MyFXBook RSS fallback"], EXT_FC, EXT_EC)
-box(53.5, 86.2, 12.5, 6.4, "News RSS ×4",
-    ["gold / macro", "headlines"], EXT_FC, EXT_EC)
-box(67.5, 86.2, 13.5, 6.4, "DeepSeek API",
-    ["V4-Flash LLM", "thinking: disabled"], EXT_FC, EXT_EC)
-box(82.5, 86.2, 15, 6.4, "Telegram API",
-    ["owner bot +", "subscriber channel"], EXT_FC, EXT_EC)
-
-# ── data & context layer ─────────────────────────────────────────────
-box(2.5, 69.8, 33, 11.0, "providers/  (BaseDataProvider)",
-    ["single-flight fetch · TTL+boundary cache",
-     "forming-candle DROP · validate/clean",
-     "paginated fetch_range · optional tick stream",
-     "active feed knob: data_providers.active = oanda"],
-    DAT_FC, DAT_EC)
-box(38.5, 69.8, 33, 11.0, "news_engine.py  +  llm_client.py",
-    ["calendar events + keyword sentiment (gold-inverse)",
-     "DeepSeek enrichment: batch headline scores,",
-     "event gold-direction, market blurb — display-only,",
-     "keyword fallback, daily request cap"],
-    NWS_FC, NWS_EC)
-box(74.5, 69.8, 23, 11.0, "app_config.py",
-    ["credentials & deployment:",
-     "env → config.json → refuse",
-     "(no hardcoded fallbacks)",
-     ".env loaded at import"],
-    "#FAFAFA", "#9E9E9E")
-
-# ── signal core container ────────────────────────────────────────────
-ax.add_patch(FancyBboxPatch((2.5, 40.5), 95, 26.5,
-             boxstyle="round,pad=0.35,rounding_size=1.2",
-             facecolor=COR_FC, edgecolor=COR_EC, linewidth=1.6, zorder=1))
-ax.text(50, 65.2, "SIGNAL CORE — runs identically in live scans and backtests (closed bars only)",
-        ha="center", fontsize=8.4, fontweight="bold", color="#8a6d1a", zorder=3)
-
-box(4.5, 51.8, 28, 12.0, "engine.py — TechnicalEngine",
-    ["9 scored indicators + ATR context:",
-     "EMA · RSI · MACD · BB · ADX · ROC ·",
-     "Keltner · divergence · candlestick",
-     "regime → adaptive weights → composite"],
-    "white", COR_EC)
-box(4.5, 42.0, 28, 8.4, "pair_signals/xauusd.py — factory",
-    ["ALWAYS used: config.json overlay +",
-     "validated per-TF deltas (hour blocks, trail…)"],
-    "white", COR_EC)
-box(35.5, 42.0, 60, 20.7, "signal_aggregator.py — analyze_pair → analyze_from_dataframes",
-    ["multi-TF agreement  (M5→[M15,H1] · M15→[M5,H1] · H1→[M15,H4] · H4→[H1,D1])",
-     "→ news context (warn-only by default) → diminishing-decay penalties",
-     "→ exhaustion hard block → quality grade A+…F",
-     "→ ordered post-grade gate chain (session · XAU SELL vol/hour · regime · M5 noise)",
-     "→ setup generators: XAU ATR (A+/A) · generic swing · XAU_PB · XAU_CP",
-     "→ entry · SL · TP1/TP2 · trailing plan"],
-    "white", COR_EC, body_fs=6.9)
-
-# ── SignalReport hub ─────────────────────────────────────────────────
-box(31, 33.2, 38, 5.2, "SignalReport",
-    ["signal (−2…+2) · grade · trade setup · reasons · news summary"],
-    REP_FC, REP_EC, title_fs=8.8)
-
-# ── delivery surfaces ────────────────────────────────────────────────
-band(1, 13.2, 98, 17.2, "DELIVERY SURFACES", "#4C8F50")
-box(2.5, 15.0, 12, 12.6, "main.py",
-    ["CLI report", "--detail", "--json"], DEL_FC, DEL_EC)
-box(16, 15.0, 24, 12.6, "oanda_daemon.py + executor",
-    ["practice auto-execution",
-     "book-parity rules: no broker TP,",
-     "5-bar cooldown, once-per-bar",
-     "trail, OPEN_ONLY fills,",
-     "boot reconcile · orders → Oanda"],
-    DEL_FC, DEL_EC, title_fs=7.8)
-box(42.5, 15.0, 31, 12.6, "dashboard_publisher.py  →  Apache",
-    ["independent daemon · scans at candle closes",
-     "atomic publish to /var/www/trade:",
-     "dashboard.html · snapshot.json · news.json",
-     "+ news_summary.php — on-demand AI",
-     "gold-impact summary (whitelist · cache · cap)"],
-    DEL_FC, DEL_EC)
-box(76, 15.0, 21.5, 12.6, "telegram_bot.py",
-    ["candle-boundary auto-scans", "quality threshold + dedup",
-     "alerts + LLM context line", "WS level-breach monitor"],
-    DEL_FC, DEL_EC)
-
-# ── backtest / QA ────────────────────────────────────────────────────
-box(2.5, 3.2, 71.8, 8.0, "backtest/gpu_backtest.py — bar-by-bar replay through the SAME analyze_from_dataframes",
-    ["realism (results-critical): next-open entries · raw-path resolution · spread charged · SL-first ties · close-only trail",
-     "CuPy GPU / NumPy fallback · parquet cache · fed by providers.fetch_range (news off)"],
-    BTS_FC, BTS_EC)
-box(77.5, 3.2, 20, 8.0, "test_all.py",
-    ["359 tests · 22 modules", "no network (mocked)"], "#FAFAFA", "#9E9E9E")
-
-# ── arrows: short structural hops (solid dark) ───────────────────────
-arrow((10, 86.2), (14, 81.2), label="OHLCV", lx=-3.4, ly=0.2)          # oanda → providers
-arrow((27, 86.2), (24, 81.2), ls=(0, (3, 2)), lw=1.1)                  # fallbacks → providers
-arrow((44, 86.2), (48, 81.2))                                          # FF → news
-arrow((59.5, 86.2), (56, 81.2))                                        # RSS → news
-arrow((72, 86.2), (62, 81.2), label="LLM", lx=3.4, ly=0.6)             # deepseek → news
-arrow((19, 69.8), (19, 64.2), label="DataFrames (UTC, closed bars)", lx=0.2, ly=0.9)
-arrow((33.5, 57.5), (35.5, 55.5))                                      # engine → aggregator
-arrow((32.5, 46.0), (35.5, 48.0))                                      # factory → aggregator
-arrow((55, 69.8), (55, 63.1), label="warn-only context", lx=0, ly=1.0)
-arrow((86, 69.8), (86, 67.3), ls=(0, (3, 2)), lw=1.1)                  # config → core
-arrow((50, 40.5), (50, 38.8))                                          # core → report
-arrow((36, 34.5), (8.5, 27.95), conn="arc3,rad=0.12")                  # report → cli
-arrow((40, 33.2), (28, 27.95), conn="arc3,rad=0.06")                   # report → daemon
-arrow((60, 33.2), (58, 27.95))                                         # report → dashboard
-arrow((66, 34.5), (86.5, 27.95), conn="arc3,rad=-0.12")                # report → telegram
-
-# ── long-haul edges (thin dashed, routed through gutters) ────────────
-# telegram bot ↔ Telegram API — right-margin arc, clear of app_config
-arrow((95, 86.2), (95, 27.95), conn="arc3,rad=0.15", color=HAUL,
-      lw=1.2, ls=(0, (4, 2)))
-ax.text(99.2, 57, "alerts / commands", fontsize=6.4, color=MUT,
-        rotation=90, ha="center", va="center")
-# browser click → news_summary.php → DeepSeek
-# (oanda_daemon's order flow is noted inside its box — no long arc)
-arrow((66, 27.95), (73, 86.2), conn="arc3,rad=-0.18", color=HAUL,
-      lw=1.2, ls=(0, (4, 2)),
-      label="on-demand AI summary\n(browser click → PHP)", lx=2.5, ly=11.6)
-# providers.fetch_range → backtester — far-left margin channel
-arrow((1.5, 69.8), (1.5, 11.0), color=BTS_EC, lw=1.1, ls=(0, (3, 2)),
-      label="fetch_range\n(history)", lx=3.6, ly=-1.2)
-# backtester replays the signal core — right gutter between boxes
-arrow((74.0, 11.6), (74.9, 40.3), color=HAUL, lw=1.3)
-ax.text(81.8, 38.9, "replays the exact signal code", fontsize=6.4,
-        color=MUT, ha="center", va="center", zorder=5,
-        bbox=dict(boxstyle="round,pad=0.15", fc="white", ec="none", alpha=0.9))
-
-out = os.path.join(os.path.dirname(os.path.abspath(__file__)), "architecture.png")
-fig.savefig(out, dpi=200, bbox_inches="tight", facecolor="white")
-print(f"saved {out}")

docs/superpowers/specs/2026-07-21-centralized-config-design.md

@@ -1,154 +0,0 @@
-# Centralized Configuration (config.json) — Design
-
-**Date:** 2026-07-21
-**Status:** Approved (user confirmed all four scoping decisions)
-
-## Goal
-
-Move deployment settings — Telegram credentials, data-provider API keys and
-selection, scanned pairs, scan timeframes, per-pair base blocked hours, alert
-quality boot default, log level — into a single gitignored `config.json` at the
-repo root, loaded once at startup.
-
-**Hard constraint:** zero change to signal logic, indicator weights, thresholds,
-or gates. With an unedited `config.json` (or none at all), every produced
-`PlatformConfig` must be byte-identical to today's factory output. This is
-test-enforced (see Testing).
-
-## Scoping decisions (user-confirmed 2026-07-21)
-
-1. **Secrets live in config.json** (gitignored). A committed
-   `config.json.example` carries placeholders. Env vars still override.
-   The hardcoded credential fallbacks in `config.py` and `telegram_bot.py`
-   are **deleted** — on a machine with neither config.json nor env vars, the
-   bot refuses to start with an explicit error instead of silently using
-   committed credentials. (Recommend rotating the Telegram token and Twelve
-   Data key after merge; both are exposed in git history.)
-2. **Multi-provider is structure only**: a `data_providers` section with named
-   provider blocks and an `active` selector. Only Twelve Data is implemented.
-   No failover logic (nothing to fail over to yet).
-3. **Blocked hours: base list per pair only.** The validated per-timeframe
-   deltas (M5 appends hour 9; M15 SELL-block 15/17 UTC) stay in
-   `pair_signals/xauusd.py` next to their validation evidence. The overlay is
-   applied **before** those deltas and updates **both** mirrored copies
-   (`xau_strategy.blocked_hours_utc` and `backtest.blocked_hours_utc`) so they
-   cannot drift.
-4. **Extra keys included:** scan timeframes, alert min quality (boot default),
-   log level. Everything else (indicator params, RR geometry, M5 gates,
-   CP knobs, news config) deliberately stays code-only.
-
-## File schema
-
-`config.json` (gitignored; `config.json.example` committed with placeholders
-for secrets and today's tuned values for everything else):
-
-```json
-{
-  "telegram": {
-    "token": "<bot token>",
-    "chat_id": "<owner chat id>",
-    "channel_id": "<-100… or empty string to disable the channel feed>",
-    "alert_min_quality": "C"
-  },
-  "data_providers": {
-    "active": "twelve_data",
-    "twelve_data": { "api_key": "<key>" }
-  },
-  "scanning": {
-    "pairs": ["XAU/USD"],
-    "timeframes": ["M5", "M15", "H1", "H4"]
-  },
-  "pairs": {
-    "XAU/USD": { "blocked_hours_utc": [0, 1, 2, 10, 11, 13, 16, 21] }
-  },
-  "log_level": "INFO"
-}
-```
-
-Timeframe strings use the `Timeframe` enum names (`M5`, `M15`, `H1`, `H4`, …).
-Pair keys use canonical symbols (`XAU/USD`).
-
-## Architecture
-
-### New module: `app_config.py`
-
-Single responsibility: read + validate `config.json`, expose typed accessors,
-apply the pair-level overlay. No signal knowledge.
-
-- `AppConfig` dataclass: `telegram` (token, chat_id, channel_id,
-  alert_min_quality), `data_providers` (active name + per-provider dicts),
-  `scan_pairs`, `scan_timeframes`, `pair_overrides`
-  (symbol → {blocked_hours_utc}), `log_level`.
-- `load_app_config(path=REPO_ROOT/"config.json") -> AppConfig` — module-level
-  cached (loaded once per process). Missing file → `AppConfig` of pure
-  defaults + one warning log. Malformed JSON, unknown top-level/section keys,
-  unknown pair symbols, hours outside 0–23, unknown timeframe names, unknown
-  `active` provider → raise `AppConfigError` with a named message (fail fast
-  at boot).
-- Precedence, applied inside the accessors:
-  **env var → config.json → code default**. Secrets (`TELEGRAM_TOKEN`,
-  `TELEGRAM_CHAT_ID`, `TELEGRAM_CHANNEL_ID`, `TWELVE_DATA_API_KEY`) have **no
-  code default** after this change.
-- `apply_pair_overrides(cfg: PlatformConfig, app: AppConfig) -> PlatformConfig`
-  — sets the pair's base `blocked_hours_utc` on **both**
-  `cfg.xau_strategy.blocked_hours_utc` and `cfg.backtest.blocked_hours_utc`,
-  plus `cfg.scan_pairs` / `cfg.scan_timeframes`. Called by the pair factory
-  (see below). A no-op when the pair has no override block.
-
-### Wiring points (all consumers)
-
-| Consumer | Change |
-|---|---|
-| `pair_signals/xauusd.py` | `make_config` calls `apply_pair_overrides` **before** the per-TF deltas (so M5 still appends hour 9 to whatever base the JSON sets, and mirrors stay in sync). |
-| `data_provider.py` | API key resolution becomes `app_config` accessor for the **active** provider (env `TWELVE_DATA_API_KEY` still wins). `config.twelve_data_api_key` field and its hardcoded default are deleted. Missing key keeps today's behavior: provider constructed with `_td=None`, errors on first fetch. |
-| `telegram_bot.py` | `TOKEN`, `CHAT_ID`, `CHANNEL_ID` come from `app_config` (hardcoded fallbacks deleted; missing token/chat_id → explicit startup error). `_alert_min_quality` boot default and `logging` level come from `app_config`. `SCAN_PAIRS` (today built from a bare `PlatformConfig()`) reads `app_config.scan_pairs` directly; `apply_pair_overrides` also stamps `cfg.scan_pairs`/`cfg.scan_timeframes` so the dataclass field stays truthful for any other reader. |
-| `main.py` | No structural change — it uses the factory, which now applies the overlay. |
-| `backtest/gpu_backtest.py` | No structural change — it uses the factory too, so an edited base blocked-hours list flows into replays identically to live (parity by construction). Data fetches pick up the provider key via `data_provider.py`. |
-| `config.py` | `twelve_data_api_key` field removed. Everything else untouched. |
-
-### Multi-provider structure
-
-`data_provider.py` gains a minimal registry:
-`PROVIDER_REGISTRY = {"twelve_data": ForexDataProvider}`. `active` selects
-from it; an unknown name fails at load. Adding a provider later = one config
-block + one adapter class registered here. No shared base class extracted now
-(YAGNI — with one provider there is nothing to abstract over yet).
-
-## Error handling summary
-
-| Condition | Behavior |
-|---|---|
-| No config.json | Defaults + warning. Secrets must come from env or startup fails (bot) / fetch fails (provider). |
-| Malformed JSON / bad values / unknown keys | `AppConfigError` at load, process exits with named message. |
-| Missing Telegram token or chat_id at bot boot | Explicit startup error naming the missing key and both places it can be set. |
-| Missing data-provider key | Same as today: provider builds, first fetch errors. |
-
-## Migration
-
-- Create the local `config.json` populated with the **current live values**
-  (the ones being deleted from source) so the running bot keeps working
-  unchanged. `.gitignore` gains `config.json`.
-- Commit `config.json.example` with placeholders.
-- CLAUDE.md: update the env-var table, repository map, and config section.
-- Post-merge recommendation (user action): rotate the Telegram token and
-  Twelve Data key, since both are in git history.
-
-## Testing (`test_all.py`, new module)
-
-1. **Parity pin (the critical one):** a config.json containing the example
-   defaults, and no config.json at all, both produce factory output identical
-   to today's (full dataclass equality across all four TFs).
-2. Overlay mapping: JSON blocked hours land on **both** mirrored fields; M5
-   factory still appends hour 9 on top of a custom base.
-3. Precedence: env var beats config.json beats default, per key.
-4. Validation: malformed JSON, bad hour, unknown pair/timeframe/provider each
-   raise `AppConfigError`.
-5. Missing-file behavior: defaults + no crash.
-6. Existing factory-equivalence test stays green (equivalence is now defined
-   over factory-with-overlay vs inline-config-with-overlay).
-
-## Out of scope
-
-Hot reload (config is read once at boot), a second concrete provider,
-failover, moving any tuned strategy value beyond the base blocked-hours list,
-news/indicator/geometry settings in JSON.

docs/user_guide.html

@@ -69,9 +69,8 @@
 <tr><td><b>Score / confidence</b></td><td>The regime-weighted composite of all nine indicators. Bigger absolute score = stronger consensus.</td></tr>
 <tr><td><b>MTF aligned</b></td><td>The two confirmation timeframes agree with the trade direction. Signals without alignment are filtered out on M5.</td></tr>
 <tr><td><b>Warnings ⚠️</b></td><td>Context, not veto. Backtesting shows warned trades perform about the same — strong momentum often triggers warnings.</td></tr>
-<tr><td><b>Market context 📰</b></td><td>An AI-written one-liner summarizing today's macro backdrop (headlines + high-impact releases) and its likely pull on gold. Informational only — it never blocks or grades a signal.</td></tr>
 <tr><td><b>SL / TP1 / RR</b></td><td>Stop loss sits behind the nearest swing level with an ATR buffer. TP1 is the nearest opposing level. <b>1R</b> = the entry-to-SL distance = the amount you risk.</td></tr>
-<tr><td><b>Trail plan 🔄</b></td><td>The real exit. Once price moves +0.6R in your favor, move the stop to trail 0.75R behind the best price and update it at each candle close. <b>TP1 is advisory — let winners run.</b></td></tr>
+<tr><td><b>Trail plan 🔄</b></td><td>The real exit. Once price moves +0.6R in your favor (+0.8R for pullback setups), move the stop to trail 0.75R behind the best price and update it at each candle close. <b>TP1 is advisory — let winners run.</b></td></tr>
 </table>
 
 <h2>3 · Acting on a signal — step by step</h2>
@@ -130,7 +129,6 @@
 <tr><td><code>/scan</code></td><td>Run an immediate analysis of the primary timeframe.</td></tr>
 <tr><td><code>/autoscan</code></td><td>Toggle automatic scanning (M5 / M15 / H1 / H4 at each candle close).</td></tr>
 <tr><td><code>/status</code></td><td>Uptime, auto-scan state, WebSocket price-monitor state.</td></tr>
-<tr><td><code>/news</code></td><td>Net XAU/USD news sentiment score (informational only — never gates signals).</td></tr>
 <tr><td><code>/cache</code> · <code>/clearcache</code></td><td>Inspect / reset the market-data cache.</td></tr>
 <tr><td><code>/help</code></td><td>Command reference.</td></tr>
 </table>

engine.py

@@ -13,7 +13,7 @@ Design philosophy:
 
 import logging
 from dataclasses import dataclass, field
-from typing import Dict, List, Tuple
+from typing import Dict, List, Optional, Tuple
 
 import numpy as np
 import pandas as pd
@@ -245,12 +245,6 @@ class TechnicalEngine:
             signal = SignalStrength.NEUTRAL
             conf = 40
 
-        # Partial alignment (plain BUY/SELL, 2-3 of 4 checks) carries no
-        # edge on M5 — squash to NEUTRAL when the research knob is on.
-        if c.ema_partial_neutral and signal in (SignalStrength.BUY, SignalStrength.SELL):
-            signal = SignalStrength.NEUTRAL
-            conf = 40
-
         crossover_desc = ""
         if len(ema_fast) >= 3:
             prev_f, prev_m = ema_fast.iloc[-2], ema_medium.iloc[-2]
@@ -644,10 +638,15 @@ class TechnicalEngine:
                 signal = SignalStrength.SELL
                 conf = min(70, 50 + roc_abs * 10)
                 desc = f"ROC {roc_val:.2f}% — Falling momentum"
+        elif roc_prev <= 0 < roc_val:
+            signal = SignalStrength.BUY
+            conf = 65
+            desc = f"ROC crossed above zero (+{roc_val:.2f}%)"
+        elif roc_prev >= 0 > roc_val:
+            signal = SignalStrength.SELL
+            conf = 65
+            desc = f"ROC crossed below zero ({roc_val:.2f}%)"
         else:
-            # Zero-crosses land in the branches above by definition (a
-            # cross up is positive and rising, a cross down negative and
-            # falling) — only flat/fading momentum reaches here.
             signal = SignalStrength.NEUTRAL
             conf = 30
             desc = f"ROC {roc_val:+.2f}% — No clear momentum"

llm_client.py

@@ -1,195 +0,0 @@
-"""
-ForexBot2 — DeepSeek LLM client
-================================
-Thin requests-based client for DeepSeek's OpenAI-compatible chat API,
-used by `news_engine` for display/warning-only enrichment (headline
-sentiment, event surprise direction, market summary). Signals are NEVER
-gated on LLM output — every caller has a non-LLM fallback and treats a
-`None` return as "use the fallback".
-
-Design constraints:
-- One attempt per call, no retries: news runs inside scan cycles; the
-  news engine's 600 s context cache is the natural retry cadence.
-- A UTC-day request cap bounds spend even if a bug causes call storms.
-- Cost: DeepSeek caches identical prompt PREFIXES server-side (cache-hit
-  input is ~50x cheaper), so callers pass a STATIC system prompt (no
-  timestamps or interpolation) and put all variable content in the user
-  payload.
-
-This is a module-level singleton (`get_llm_client`) because the daily
-cap counter must survive the per-scan reconstruction of `NewsEngine`.
-"""
-
-import datetime as _dt
-import json
-import logging
-import threading
-import time
-from typing import Any, Dict, Optional, Tuple
-
-import requests
-
-from app_config import deepseek_api_key, get_app_config
-
-logger = logging.getLogger(__name__)
-
-DEEPSEEK_URL = "https://api.deepseek.com/chat/completions"
-DEFAULT_MODEL = "deepseek-v4-flash"
-# 8s timed out on a measured cold-prefix-cache headline batch (2026-08-03);
-# warm-cache calls come back well under that. 12s buys cold-start headroom
-# while still bounding worst-case scan latency (≤3 calls per cache window).
-DEFAULT_TIMEOUT = 12.0
-DEFAULT_DAILY_CAP = 500
-
-
-class DeepSeekClient:
-    """JSON-in/JSON-out chat wrapper. All failures return None."""
-
-    def __init__(
-        self,
-        api_key: str,
-        model: str = DEFAULT_MODEL,
-        timeout: float = DEFAULT_TIMEOUT,
-        daily_cap: int = DEFAULT_DAILY_CAP,
-        enabled: bool = True,
-    ):
-        self._api_key = api_key or ""
-        self._model = model
-        self._timeout = timeout
-        self._daily_cap = daily_cap
-        self._enabled = enabled
-        self._session = requests.Session()
-        self._session.headers.update({
-            "Content-Type": "application/json",
-            "Authorization": f"Bearer {self._api_key}",
-        })
-        self._lock = threading.Lock()
-        self._count_day: Optional[_dt.date] = None
-        self._count = 0
-        self._warned: set = set()
-        # Health stamps for status displays: last successful call and
-        # last failure (ts, kind). Read-only for consumers.
-        self.last_ok: float = 0.0
-        self.last_error: Optional[Tuple[float, str]] = None
-
-    @property
-    def available(self) -> bool:
-        return self._enabled and bool(self._api_key)
-
-    def _warn_once(self, kind: str, msg: str, *args: Any) -> None:
-        if kind in self._warned:
-            logger.debug(msg, *args)
-        else:
-            self._warned.add(kind)
-            logger.warning(msg, *args)
-
-    def _under_cap(self) -> bool:
-        """Count a request against the UTC-day cap; False = don't call."""
-        today = _dt.datetime.now(_dt.timezone.utc).date()
-        with self._lock:
-            if self._count_day != today:
-                self._count_day = today
-                self._count = 0
-                self._warned.discard("cap")
-            if self._count >= self._daily_cap:
-                self._warn_once(
-                    "cap", "DeepSeek daily request cap (%d) exhausted — "
-                    "falling back to keyword analysis until UTC midnight",
-                    self._daily_cap)
-                return False
-            self._count += 1
-            return True
-
-    def chat_json(self, system_prompt: str,
-                  user_payload: Dict[str, Any]) -> Optional[Dict[str, Any]]:
-        """One chat call; returns the model's JSON object or None."""
-        if not self.available or not self._under_cap():
-            return None
-        body = {
-            "model": self._model,
-            "messages": [
-                {"role": "system", "content": system_prompt},
-                {"role": "user",
-                 "content": json.dumps(user_payload, sort_keys=True)},
-            ],
-            "response_format": {"type": "json_object"},
-            "temperature": 0,
-            # V4-Flash is a hybrid thinking model; without this flag it
-            # burns hundreds of reasoning tokens per call — measured
-            # 2026-08-03: a 25-headline batch blew straight through the
-            # timeout AND could exhaust max_tokens before the answer,
-            # returning empty content. The doctrine + few-shot examples
-            # in the system prompts carry the quality instead.
-            "thinking": {"type": "disabled"},
-            # Must fit the largest reply: a full sentiment batch is
-            # ~10 tokens per headline; 1000 truncated mid-JSON on a real
-            # RSS cycle (2026-08-03).
-            "max_tokens": 2000,
-        }
-        try:
-            resp = self._session.post(
-                DEEPSEEK_URL, json=body, timeout=self._timeout)
-        except requests.RequestException as e:
-            self.last_error = (time.time(), "network")
-            self._warn_once("network", "DeepSeek call failed: %s", e)
-            return None
-        if resp.status_code != 200:
-            self.last_error = (time.time(), f"http_{resp.status_code}")
-            self._warn_once(
-                f"http_{resp.status_code}",
-                "DeepSeek HTTP %s: %s", resp.status_code, resp.text[:200])
-            return None
-        try:
-            parsed = self._parse_response(resp.json())
-        except ValueError:
-            parsed = None
-        if parsed is None:
-            self.last_error = (time.time(), "parse")
-            self._warn_once("parse", "DeepSeek response was not the "
-                            "expected JSON object — ignoring")
-        else:
-            self.last_ok = time.time()
-        return parsed
-
-    @staticmethod
-    def _parse_response(body: Any) -> Optional[Dict[str, Any]]:
-        """Extract the assistant's JSON object from a chat completion."""
-        try:
-            content = body["choices"][0]["message"]["content"]
-        except (KeyError, IndexError, TypeError):
-            return None
-        try:
-            obj = json.loads(content)
-        except (TypeError, ValueError):
-            return None
-        return obj if isinstance(obj, dict) else None
-
-
-# ── Process-wide singleton ────────────────────────────────────────────
-
-_client: Optional[DeepSeekClient] = None
-_client_lock = threading.Lock()
-
-
-def get_llm_client() -> DeepSeekClient:
-    global _client
-    with _client_lock:
-        if _client is None:
-            settings = get_app_config().deepseek_settings
-            _client = DeepSeekClient(
-                api_key=deepseek_api_key(),
-                model=str(settings.get("model", DEFAULT_MODEL)),
-                timeout=float(settings.get("timeout_seconds",
-                                           DEFAULT_TIMEOUT)),
-                daily_cap=int(settings.get("daily_request_cap",
-                                           DEFAULT_DAILY_CAP)),
-                enabled=settings.get("enabled", True) is not False,
-            )
-        return _client
-
-
-def reset_llm_client() -> None:
-    """Drop the singleton (tests / config reloads)."""
-    global _client
-    with _client_lock:
-        _client = None

logs/telegram.log.2026-07-14

@@ -0,0 +1,21 @@
+01:51:29 [WARNING] data_provider: BTC/USD 1h: last bar 2026-07-11 05:51 is 4320m old (limit: 180m)
+01:52:02 [WARNING] data_provider: BTC/USD 1h: last bar 2026-07-11 05:52 is 4320m old (limit: 180m)
+02:42:59 [WARNING] data_provider: BTC/USD 1h: last bar 2026-07-11 06:42 is 4320m old (limit: 180m)
+03:30:24 [WARNING] data_provider: BTC/USD 1h: last bar 2026-07-11 07:30 is 4320m old (limit: 180m)
+03:31:32 [WARNING] data_provider: BTC/USD 1h: last bar 2026-07-11 07:31 is 4320m old (limit: 180m)
+11:30:20 [WARNING] data_provider: BTC/USD 1h: last bar 2026-07-11 15:30 is 4320m old (limit: 180m)
+11:41:34 [WARNING] data_provider: BTC/USD 1h: last bar 2026-07-11 15:41 is 4320m old (limit: 180m)
+11:45:46 [WARNING] data_provider: BTC/USD 1h: last bar 2026-07-11 15:45 is 4320m old (limit: 180m)
+11:56:38 [WARNING] data_provider: BTC/USD 1h: last bar 2026-07-11 15:56 is 4320m old (limit: 180m)
+11:57:11 [WARNING] data_provider: BTC/USD 1h: last bar 2026-07-11 15:57 is 4320m old (limit: 180m)
+11:59:52 [WARNING] data_provider: BTC/USD 1h: last bar 2026-07-11 15:59 is 4320m old (limit: 180m)
+12:10:05 [WARNING] data_provider: BTC/USD 1h: last bar 2026-07-11 16:10 is 4320m old (limit: 180m)
+13:05:37 [WARNING] data_provider: BTC/USD 1h: last bar 2026-07-11 17:05 is 4320m old (limit: 180m)
+13:14:25 [WARNING] data_provider: BTC/USD 1h: last bar 2026-07-11 17:14 is 4320m old (limit: 180m)
+13:38:50 [WARNING] data_provider: BTC/USD 1h: last bar 2026-07-11 17:38 is 4320m old (limit: 180m)
+13:45:14 [WARNING] data_provider: BTC/USD 1h: last bar 2026-07-11 17:45 is 4320m old (limit: 180m)
+13:51:33 [WARNING] data_provider: BTC/USD 1h: last bar 2026-07-11 17:51 is 4320m old (limit: 180m)
+15:04:41 [WARNING] data_provider: BTC/USD 1h: last bar 2026-07-11 19:04 is 4320m old (limit: 180m)
+15:04:50 [WARNING] data_provider: BTC/USD 1h: last bar 2026-07-11 19:04 is 4320m old (limit: 180m)
+15:05:17 [WARNING] data_provider: BTC/USD 1h: last bar 2026-07-11 19:05 is 4320m old (limit: 180m)
+15:06:10 [WARNING] data_provider: BTC/USD 1h: last bar 2026-07-11 19:06 is 4320m old (limit: 180m)

main.py

@@ -13,6 +13,7 @@ import json
 import logging
 import sys
 from datetime import datetime, timezone
+from typing import Dict, List
 
 # Force UTF-8 output on Windows so box-drawing chars render
 if hasattr(sys.stdout, "reconfigure"):
@@ -161,8 +162,6 @@ def print_signal_report(report: SignalReport, detail: bool = False):
         print(f"  {Colors.YELLOW}📅{Colors.RESET} {report.event_description}")
     if report.news_description and report.news_description != "Neutral news sentiment":
         print(f"  {Colors.MAGENTA}📰{Colors.RESET} {report.news_description}")
-    if report.news_summary:
-        print(f"  {Colors.MAGENTA}🧠{Colors.RESET} {report.news_summary}")
 
     # ── 6. DETAIL (only with --detail) ─────────────────────────────
     if detail and report.primary_analysis:
@@ -208,7 +207,6 @@ def report_to_dict(report: SignalReport) -> dict:
         "event_description": report.event_description,
         "news_sentiment": round(report.news_sentiment, 3),
         "news_description": report.news_description,
-        "news_summary": report.news_summary,
         "bullish_reasons": report.bullish_reasons,
         "bearish_reasons": report.bearish_reasons,
         "warnings": report.warnings,
@@ -224,9 +222,6 @@ def report_to_dict(report: SignalReport) -> dict:
             "risk_reward": ts.risk_reward,
             "risk_pips": round(ts.risk_pips, 1),
             "reward_pips": round(ts.reward_pips, 1),
-            "strategy": ts.strategy,
-            "trail_arm_r": ts.trail_arm_r,
-            "trail_dist_r": ts.trail_dist_r,
         }
     if report.primary_analysis:
         d["indicators"] = {
@@ -236,7 +231,7 @@ def report_to_dict(report: SignalReport) -> dict:
                 "description": ind.description,
             }
             for name, ind in report.primary_analysis.indicators.items()
-            if not name.startswith("_") and ind.weight != 0
+            if not name.startswith("_")
         }
         d["key_levels"] = {
             k: round(v, 5) for k, v in report.primary_analysis.key_levels.items()

news_engine.py

@@ -8,87 +8,22 @@ Key principle: High-impact news events override technical signals.
 A perfect technical setup can be destroyed by an unexpected NFP print.
 """
 
-import hashlib
-import html
 import logging
 import re
 import threading
 import time
 import xml.etree.ElementTree as ET
 from dataclasses import dataclass, field
-from datetime import datetime, timezone
-from email.utils import parsedate_to_datetime
+from datetime import datetime, timedelta, timezone
 from typing import Dict, List, Optional, Tuple
 
 import requests
 
 from config import NewsConfig
-from llm_client import get_llm_client
 
 logger = logging.getLogger(__name__)
 
 
-# ── DeepSeek prompts ─────────────────────────────────────────────────
-# STATIC constants on purpose: DeepSeek prices a repeated byte-identical
-# prompt prefix at the cache-hit rate (~50x cheaper), so nothing here may
-# be interpolated or timestamped. All variable content travels in the
-# user payload. LLM output is display/warning-only — it never gates
-# signals (see NewsConfig.sentiment_affects_signals).
-
-_LLM_SENTIMENT_SYSTEM = """\
-You are a forex/gold news analyst. You receive JSON:
-{"headlines": ["...", ...]}
-For EACH headline, in the same order, score:
-- "s": overall market sentiment of the headline, -1.0 (very bearish)
-  to 1.0 (very bullish), 0.0 if irrelevant or neutral.
-- "xau": the effect on the GOLD (XAU/USD) price, -1.0 (gold down)
-  to 1.0 (gold up), 0.0 if irrelevant.
-Gold doctrine: hawkish Fed / rate hikes / rising real yields / strong
-dollar are gold-bearish; easing / rate cuts / safe-haven flows /
-geopolitical stress / weak dollar are gold-bullish.
-Handle negation correctly ("no rate hike" is not hawkish).
-Reply with ONLY JSON: {"scores": [[s, xau], ...]} — exactly one
-[s, xau] pair per input headline, in the same order, nothing else.
-
-Examples:
-"Fed signals faster rate hikes as inflation stays hot" -> [0.4, -0.7]
-"Gold surges to record high on safe-haven demand" -> [0.8, 0.9]
-"Dollar slips after soft jobs report; Treasury yields fall" -> [-0.4, 0.6]
-"""
-
-_LLM_EVENT_SYSTEM = """\
-You are a forex/gold analyst. You receive one released economic event as
-JSON: {"event", "currency", "actual", "forecast", "previous",
-"vs_forecast"}. "vs_forecast" states whether the actual print came in
-above or below the forecast; it is computed programmatically — trust it
-over your own comparison of the numbers. If it is "unknown", compare
-actual and forecast yourself.
-Decide the likely effect of this RELEASE on the GOLD (XAU/USD) price.
-A stronger-than-forecast USD economy or a hawkish implication is usually
-gold-bearish; weaker data or a dovish implication is usually
-gold-bullish.
-Inverted metrics — for unemployment rate and unemployment/jobless
-claims a LOWER number means a STRONGER labor market: below forecast =
-strong economy = usually gold-bearish; above forecast = weak economy =
-usually gold-bullish.
-Example: {"event": "Unemployment Claims", "actual": "202K", "forecast":
-"217K", "vs_forecast": "below forecast"} ->
-{"gold_direction": "bearish", "reason": "fewer claims show strong labor market"}
-Reply with ONLY JSON:
-{"gold_direction": "bullish"|"bearish"|"neutral", "reason": "<max 10 words>"}
-"""
-
-_LLM_SUMMARY_SYSTEM = """\
-You are a forex/gold market-context writer. You receive JSON with recent
-"headlines" and today's high-impact "events" (released: true means the
-number is out). Write a 1-2 sentence plain-English summary of the
-current macro backdrop for a GOLD (XAU/USD) trader: the dominant theme
-and its likely pull on gold. No advice, no hedging boilerplate, max 45
-words.
-Reply with ONLY JSON: {"summary": "..."}
-"""
-
-
 _RELEASE_VALUE_RE = re.compile(r"(-?\d+(?:\.\d+)?)\s*([kmbt])?", re.IGNORECASE)
 _RELEASE_MULTIPLIERS = {"k": 1e3, "m": 1e6, "b": 1e9, "t": 1e12}
 
@@ -104,44 +39,6 @@ def _parse_release_value(raw: Optional[str]) -> Optional[float]:
     return float(match.group(1)) * multiplier
 
 
-_DESC_MAX_CHARS = 500
-_MAX_LINK_CHARS = 2048
-
-
-def _clean_html_text(raw: Optional[str],
-                     limit: int = _DESC_MAX_CHARS) -> Optional[str]:
-    """RSS <description> → plain text: unescape entities, strip tags,
-    collapse whitespace, truncate. Unescape and strip both run twice
-    because feeds routinely double-encode (&amp;lt;p&amp;gt; wrapping
-    real markup) — the second unescape can reveal fresh tags."""
-    if not raw:
-        return None
-    text = html.unescape(raw)
-    text = re.sub(r"<[^>]+>", " ", text)
-    text = html.unescape(text)
-    text = re.sub(r"<[^>]+>", " ", text)
-    text = re.sub(r"\s+", " ", text).strip()
-    if not text:
-        return None
-    if len(text) > limit:
-        text = text[:limit].rstrip() + "…"
-    return text
-
-
-def _safe_link(raw: Optional[str]) -> Optional[str]:
-    """Article URL for display: http(s) only (an RSS <link> is untrusted
-    and lands in an href), bounded length (Google News redirect URLs run
-    1-2 KB; anything past that is bloat, not a link)."""
-    if not raw:
-        return None
-    link = raw.strip()
-    if not link.lower().startswith(("http://", "https://")):
-        return None
-    if len(link) > _MAX_LINK_CHARS:
-        return None
-    return link
-
-
 @dataclass
 class EconomicEvent:
     """Single economic calendar event."""
@@ -194,11 +91,6 @@ class NewsSentiment:
     # Per-currency score: monetary-policy keywords flip sign for XAU
     # (hawkish is dollar-bullish but gold-bearish).
     sentiment_by_currency: Dict[str, float] = field(default_factory=dict)
-    # Display-only enrichment (dashboard detail dialog). Optional with
-    # defaults so existing constructor calls keep working.
-    link: Optional[str] = None         # article URL (Google News = redirect)
-    description: Optional[str] = None  # plain text, tags stripped, ≤500 chars
-    publisher: Optional[str] = None    # Google News <source> text, else None
 
 
 @dataclass
@@ -212,7 +104,6 @@ class FundamentalContext:
     news_headlines: List[NewsSentiment] = field(default_factory=list)
     event_impact_on_pairs: Dict[str, str] = field(default_factory=dict)
     calendar_available: bool = False  # True if live calendar data was fetched
-    llm_summary: str = ""  # DeepSeek market-context blurb (display-only)
 
 
 class NewsEngine:
@@ -293,25 +184,6 @@ class NewsEngine:
     _headlines_cache: Tuple[float, List[NewsSentiment]] = (0.0, [])
     _EMPTY_RESULT_TTL = 60.0  # retry sooner after a failed/empty fetch
 
-    # DeepSeek result caches. Headlines and releases are immutable, so a
-    # scored one never needs re-scoring; bounded by dropping the oldest
-    # insertion. _llm_sent_cache is only touched under _cache_lock (its
-    # writer runs inside _cached_fetch); the other two take the lock
-    # around cache access but keep HTTP calls outside it.
-    _llm_sent_cache: Dict[str, Tuple[float, float]] = {}   # sha1 → (s, xau)
-    _llm_event_cache: Dict[Tuple[str, str, str], str] = {}
-    _llm_summary_cache: Tuple[float, str] = (0.0, "")
-    _LLM_SENT_CACHE_MAX = 400
-    _LLM_EVENT_CACHE_MAX = 100
-
-    # Per-source fetch health for status displays (telegram /status).
-    # name → {"ok": ts|None, "err": ts|None, "detail": str, "extra": str}.
-    # All writers run under _cache_lock (fetches happen inside
-    # _cached_fetch); readers render without the lock — worst case a
-    # display line is one cycle stale. A source is "failing" when its
-    # last error is newer than its last success.
-    _source_health: Dict[str, dict] = {}
-
     def __init__(self, config: NewsConfig):
         self.config = config
         self._session = requests.Session()
@@ -391,12 +263,6 @@ class NewsEngine:
         except Exception as e:
             logger.warning(f"News sentiment fetch failed: {e}")
 
-        # 3. LLM market-context summary (display-only)
-        try:
-            context.llm_summary = self._llm_market_summary(context)
-        except Exception as e:
-            logger.debug(f"LLM market summary failed: {e}")
-
         return context
 
     @staticmethod
@@ -415,20 +281,14 @@ class NewsEngine:
         """
         base, quote = self._parse_pair(pair_symbol)
 
-        # Check upcoming high-impact events. Grade from the SOONEST
-        # matching event, not the first listed — an unsorted feed could
-        # otherwise report "caution" for a far event while a nearer one
-        # warrants "danger".
-        soonest = None
+        # Check upcoming high-impact events
         for event in context.events_upcoming_1h:
             if event.is_high_impact and event.currency in (base, quote):
-                if soonest is None or event.minutes_until < soonest.minutes_until:
-                    soonest = event
-        if soonest is not None:
-            mins = soonest.minutes_until
-            if mins <= self.config.pre_event_buffer_minutes:
-                return "danger", f"{soonest.currency} {soonest.event} in {mins:.0f}min"
-            return "caution", f"{soonest.currency} {soonest.event} in {mins:.0f}min"
+                mins = event.minutes_until
+                if mins <= self.config.pre_event_buffer_minutes:
+                    return "danger", f"{event.currency} {event.event} in {mins:.0f}min"
+                else:
+                    return "caution", f"{event.currency} {event.event} in {mins:.0f}min"
 
         # Check for recent high-impact releases. A surprise print (actual
         # away from forecast) keeps the market unstable longer than an
@@ -444,17 +304,10 @@ class NewsEngine:
                 )
                 if event.minutes_until > -window:
                     if surprised:
-                        desc = (
+                        return "caution", (
                             f"{event.currency} {event.event} released — {surprise} surprise "
                             f"(actual {event.actual} vs forecast {event.forecast})"
                         )
-                        # LLM enrichment touches the DESCRIPTION only —
-                        # `surprise` is numeric-direction and does not know
-                        # whether a beat is gold-bullish; the LLM does.
-                        gold_dir = self._llm_event_direction(event)
-                        if gold_dir:
-                            desc += f" — gold {gold_dir}"
-                        return "caution", desc
                     return "caution", f"{event.currency} {event.event} just released"
 
         return "clear", "No significant events"
@@ -488,24 +341,6 @@ class NewsEngine:
 
         return net, desc
 
-    def get_week_events(
-        self, currencies: Optional[List[str]] = None
-    ) -> List[EconomicEvent]:
-        """Full cached calendar week (past and upcoming), filtered to
-        `currencies` (default: tracked_currencies), sorted by time.
-
-        Read-only accessor for display surfaces (the web dashboard) —
-        rides the same class-level cache as signal scans, so it adds no
-        network I/O. No production signal path calls this.
-        """
-        if currencies is None:
-            currencies = self.config.tracked_currencies
-        events = self._cached_fetch("_calendar_cache", self._fetch_calendar_raw)
-        return sorted(
-            (e for e in events if e.currency in currencies),
-            key=lambda e: e.time,
-        )
-
     # ── Private Methods ────────────────────────────────────────────
 
     # Maps country names in MyFXBook titles to currency codes
@@ -515,36 +350,6 @@ class NewsEngine:
         "canada": "CAD", "new zealand": "NZD",
     }
 
-    @classmethod
-    def _record_source(cls, name: str, ok: bool, detail: str = "",
-                       extra: str = "") -> None:
-        """Track one dependency's fetch outcome for status displays.
-        Logs state TRANSITIONS so the log file carries the story without
-        a line every 10-min cycle: first failure WARNING (with detail),
-        repeat failures DEBUG, recovery INFO."""
-        h = cls._source_health.setdefault(
-            name, {"ok": None, "err": None, "detail": "", "extra": ""})
-        was_failing = h["err"] is not None and (
-            h["ok"] is None or h["err"] > h["ok"])
-        if ok:
-            h["ok"] = time.time()
-            h["extra"] = extra
-            if was_failing:
-                logger.info("Source recovered: %s", name)
-        else:
-            h["err"] = time.time()
-            h["detail"] = detail
-            if was_failing:
-                logger.debug("Source still failing: %s (%s)", name, detail)
-            else:
-                logger.warning("Source failing: %s (%s)", name, detail)
-
-    @staticmethod
-    def source_is_failing(entry: Optional[dict]) -> bool:
-        """True when the entry's last error is newer than its last ok."""
-        return bool(entry and entry["err"] is not None and
-                    (entry["ok"] is None or entry["err"] > entry["ok"]))
-
     def _cached_fetch(self, cache_attr: str, fetch_fn) -> list:
         """Return cached data for `cache_attr` or refetch when the TTL
         expired. Successful results live for context_cache_ttl_seconds;
@@ -577,27 +382,14 @@ class NewsEngine:
         Returns an empty list (with a log warning) if unavailable —
         we never generate fake events since that causes constant false alarms.
         """
-        sources = (
-            (self._try_fetch_calendar_api, "calendar:forexfactory",
-             "ForexFactory"),
-            (self._try_fetch_myfxbook_rss, "calendar:myfxbook", "MyFXBook"),
-        )
-        for fetcher, src, label in sources:
+        for fetcher in (self._try_fetch_calendar_api, self._try_fetch_myfxbook_rss):
             try:
                 events = fetcher()
                 if events:
-                    self._record_source(src, True)
-                    self._record_source(
-                        "calendar", True,
-                        extra=f"{len(events)} events via {label}")
                     return events
-                self._record_source(src, False, "no events returned")
             except Exception as e:
-                self._record_source(
-                    src, False, f"{type(e).__name__}: {e}")
                 logger.debug(f"Calendar fetch exception ({fetcher.__name__}): {e}")
 
-        self._record_source("calendar", False, "all sources failed")
         logger.info(
             "Economic calendar data unavailable. "
             "Event risk checks will be skipped for this cycle."
@@ -744,16 +536,11 @@ class NewsEngine:
         """Fetch and analyze news headlines for sentiment."""
         headlines = []
         seen_titles = set()
-        ok_feeds = 0
 
         for feed_url in self.config.rss_feeds:
-            host = feed_url.split("/")[2] if "//" in feed_url else feed_url
-            kept_before = len(headlines)
             try:
                 resp = self._session.get(feed_url, timeout=10)
                 if resp.status_code != 200:
-                    self._record_source(
-                        f"rss:{host}", False, f"HTTP {resp.status_code}")
                     continue
 
                 root = ET.fromstring(resp.content)
@@ -786,21 +573,6 @@ class NewsEngine:
                     currencies = self._extract_currencies(title)
 
                     if currencies:
-                        link_el = item.find("link")
-                        desc_el = item.find("description")
-                        src_el = item.find("source")
-                        # itertext() survives CDATA and stray child
-                        # elements where .text stops at the first child.
-                        desc = _clean_html_text(
-                            "".join(desc_el.itertext())
-                            if desc_el is not None else None)
-                        # Google News descriptions are often just the
-                        # headline again inside an anchor — worthless.
-                        if desc and desc.strip() == title.strip():
-                            desc = None
-                        publisher = None
-                        if src_el is not None:
-                            publisher = (src_el.text or "").strip() or None
                         headlines.append(NewsSentiment(
                             headline=title,
                             source=feed_url,
@@ -812,32 +584,11 @@ class NewsEngine:
                                 cur: self._analyze_headline_sentiment(title, cur)
                                 for cur in currencies
                             },
-                            link=_safe_link(
-                                link_el.text if link_el is not None else None),
-                            description=desc,
-                            publisher=publisher,
                         ))
 
-                ok_feeds += 1
-                self._record_source(
-                    f"rss:{host}", True,
-                    extra=f"{len(headlines) - kept_before} kept")
             except Exception as e:
-                self._record_source(
-                    f"rss:{host}", False, f"{type(e).__name__}: {e}")
                 logger.debug(f"Failed to fetch RSS from {feed_url}: {e}")
 
-        self._record_source(
-            "headlines", ok_feeds > 0,
-            detail="all feeds failed" if ok_feeds == 0 else "",
-            extra=(f"{len(headlines)} headlines from "
-                   f"{ok_feeds}/{len(self.config.rss_feeds)} feeds"))
-
-        try:
-            self._apply_llm_sentiment(headlines)
-        except Exception as e:
-            logger.debug(f"LLM sentiment failed, keyword scores stand: {e}")
-
         return headlines
 
     def _analyze_headline_sentiment(
@@ -879,142 +630,6 @@ class NewsEngine:
 
         return (bullish_score - bearish_score) / total
 
-    # ── DeepSeek enrichment (display/warning-only, keyword fallback) ──
-
-    def _llm_headline_scores(
-        self, texts: List[str]
-    ) -> Optional[List[Tuple[float, float]]]:
-        """One batched DeepSeek call → [(overall, xau), ...] aligned with
-        `texts`, or None on any failure/mismatch (caller keeps keyword
-        scores). Tests stub this method directly."""
-        result = get_llm_client().chat_json(
-            _LLM_SENTIMENT_SYSTEM, {"headlines": texts})
-        if not result:
-            return None
-        scores = result.get("scores")
-        if not isinstance(scores, list) or len(scores) != len(texts):
-            return None
-        out = []
-        for item in scores:
-            try:
-                s = max(-1.0, min(1.0, float(item[0])))
-                x = max(-1.0, min(1.0, float(item[1])))
-            except (TypeError, KeyError, ValueError, IndexError):
-                return None
-            out.append((s, x))
-        return out
-
-    def _apply_llm_sentiment(self, headlines: List[NewsSentiment]) -> None:
-        """Overwrite keyword sentiment on `headlines` with LLM scores.
-
-        Runs inside _cached_fetch, i.e. UNDER _cache_lock — cache access
-        is direct and the (rare) HTTP call blocks sibling scans at most
-        once per cache window, matching the existing shared-fetch design.
-        Only headlines missing from the cache are sent; a failed batch
-        changes nothing.
-        """
-        if not headlines or not get_llm_client().available:
-            return
-        cache = NewsEngine._llm_sent_cache
-        keyed = [(hashlib.sha1(h.headline.encode()).hexdigest(), h)
-                 for h in headlines]
-        # Cap the batch so the reply fits the client's max_tokens — the
-        # overflow stays keyword-scored this cycle and, still uncached,
-        # goes out with the next window's batch.
-        new = [(k, h) for k, h in keyed if k not in cache][:25]
-        if new:
-            scores = self._llm_headline_scores([h.headline for _, h in new])
-            if scores is not None:
-                for (k, _), sc in zip(new, scores):
-                    cache[k] = sc
-                while len(cache) > self._LLM_SENT_CACHE_MAX:
-                    cache.pop(next(iter(cache)))
-        for k, h in keyed:
-            sc = cache.get(k)
-            if sc is None:
-                continue
-            overall, xau = sc
-            h.sentiment_score = overall
-            h.sentiment_by_currency = {
-                cur: (xau if cur == "XAU" else overall)
-                for cur in h.currencies_mentioned
-            }
-
-    def _llm_event_direction(self, event: EconomicEvent) -> Optional[str]:
-        """Gold direction for a released surprise, e.g. "bearish: strong
-        jobs lift yields". None → caller keeps the numeric description.
-        Cached per release; failures are NOT cached (retry next scan).
-
-        The actual-vs-forecast comparison is precomputed and sent as
-        "vs_forecast": the 2026-08-06 validation found the model reads
-        close K/M-suffixed pairs backwards ~11% of the time (e.g. claims
-        202K vs 217K called "higher than forecast") when left to compare
-        the raw numbers itself."""
-        key = (event.event, event.currency, str(event.actual or ""))
-        with NewsEngine._cache_lock:
-            if key in NewsEngine._llm_event_cache:
-                return NewsEngine._llm_event_cache[key] or None
-        client = get_llm_client()
-        if not client.available:
-            return None
-        vs_forecast = {
-            "positive": "above forecast",
-            "negative": "below forecast",
-            "inline": "in line with forecast",
-        }.get(event.surprise or "", "unknown")
-        result = client.chat_json(_LLM_EVENT_SYSTEM, {
-            "event": event.event,
-            "currency": event.currency,
-            "actual": event.actual,
-            "forecast": event.forecast,
-            "previous": event.previous,
-            "vs_forecast": vs_forecast,
-        })
-        if not result:
-            return None
-        direction = result.get("gold_direction")
-        if direction not in ("bullish", "bearish", "neutral"):
-            return None
-        reason = str(result.get("reason", "")).strip()[:80]
-        text = "" if direction == "neutral" else (
-            f"{direction}: {reason}" if reason else direction)
-        with NewsEngine._cache_lock:
-            NewsEngine._llm_event_cache[key] = text
-            while len(NewsEngine._llm_event_cache) > self._LLM_EVENT_CACHE_MAX:
-                NewsEngine._llm_event_cache.pop(
-                    next(iter(NewsEngine._llm_event_cache)))
-        return text or None
-
-    def _llm_market_summary(self, context: FundamentalContext) -> str:
-        """1-2 sentence macro blurb for alerts; "" when unavailable.
-        Cached (success OR failure) for the context-cache TTL so at most
-        one call is made per window."""
-        with NewsEngine._cache_lock:
-            ts, cached = NewsEngine._llm_summary_cache
-            if time.time() - ts < self.config.context_cache_ttl_seconds:
-                return cached
-        high_impact = [e for e in context.events_today if e.is_high_impact]
-        if not context.news_headlines and not high_impact:
-            return ""
-        client = get_llm_client()
-        if not client.available:
-            return ""
-        result = client.chat_json(_LLM_SUMMARY_SYSTEM, {
-            "headlines": [h.headline for h in context.news_headlines[:15]],
-            "events": [
-                {"event": e.event, "currency": e.currency,
-                 "actual": e.actual, "forecast": e.forecast,
-                 "released": not e.is_upcoming}
-                for e in high_impact[:10]
-            ],
-        })
-        summary = ""
-        if result:
-            summary = str(result.get("summary", "")).strip()[:300]
-        with NewsEngine._cache_lock:
-            NewsEngine._llm_summary_cache = (time.time(), summary)
-        return summary
-
     def _extract_currencies(self, headline: str) -> List[str]:
         """Extract mentioned currencies from a headline."""
         headline_lower = headline.lower()
@@ -1035,20 +650,11 @@ class NewsEngine:
         if not date_str:
             return None
 
-        # RFC 822/2822 forms, with or without seconds, numeric offsets AND
-        # named zones (GMT, EST/EDT, PST…). The old strptime "%Z" path
-        # accepted ANY abbreviation but attached no offset, so e.g. an EST
-        # pubDate was silently stamped UTC — a 5-hour error.
-        try:
-            dt = parsedate_to_datetime(date_str.strip())
-            if dt is not None:
-                if dt.tzinfo is None:   # unknown zone name — assume UTC
-                    return dt.replace(tzinfo=timezone.utc)
-                return dt.astimezone(timezone.utc)
-        except (TypeError, ValueError):
-            pass
-
         formats = [
+            "%a, %d %b %Y %H:%M:%S %z",   # RFC 822 with numeric tz offset
+            "%a, %d %b %Y %H:%M:%S %Z",   # RFC 822 with tz abbreviation (GMT/UTC)
+            "%a, %d %b %Y %H:%M %Z",      # RFC 822 without seconds (MyFXBook)
+            "%a, %d %b %Y %H:%M %z",      # RFC 822 without seconds, numeric offset
             "%Y-%m-%dT%H:%M:%S%z",         # ISO 8601 with offset
             "%Y-%m-%d %H:%M:%S",           # ISO 8601 naive (assume UTC)
         ]

oanda_daemon.py

@@ -1,854 +0,0 @@
-"""
-ForexBot2 — Oanda Automated Trading Daemon
-============================================
-Autonomous trading process for XAU/USD on an Oanda practice account.
-Runs candle-boundary scan loops (M5/M15/H1/H4), executes trades via
-the Oanda v20 API, and manages trailing stops at bar close — matching
-the backtested bar-close ratchet model.
-
-Book-parity notes (matching backtest/gpu_backtest.py):
-- No broker-side take-profit: in the trailing model TP1/TP2 are
-  advisory; the only exits are SL (broker), TRAIL (daemon ratchet),
-  and TIMEOUT (max_hold_bars). A hard TP would cap winners the trail
-  lets run and diverge the live book from the backtest.
-- Entry cooldown is 5 bars per timeframe (the backtester's only
-  sequential rule) — not a fingerprint block.
-- The trail ratchets at bar CLOSE only, and each closed bar is
-  processed exactly once (last_bar_ts guard), so weekends and feed
-  lag can't double-count bars or inflate the hold counter.
-
-Divergence from the backtest book (accepted for demo safety): one
-position total. While a trade is open, other signals are logged and
-skipped, so skipped entries can be compared against the book later.
-
-Usage:  python oanda_daemon.py
-Config: env vars OANDA_ACCOUNT_ID, OANDA_API_TOKEN, OANDA_BASE_URL
-        or config.json "oanda" section (env wins).
-"""
-
-import asyncio
-import json
-import logging
-import os
-import signal
-import sys
-import threading
-from concurrent.futures import ThreadPoolExecutor
-from datetime import datetime, timezone
-from logging.handlers import TimedRotatingFileHandler
-from typing import Dict, Optional
-
-import pandas as pd
-
-from app_config import (
-    get_app_config, oanda_account_id, oanda_api_token, oanda_base_url,
-)
-from config import PAIR_INFO, PlatformConfig, Timeframe, XAUUSD
-from signal_aggregator import SignalAggregator, SignalReport, SignalStrength
-from pair_signals import PAIR_PRODUCERS
-from providers.oanda import OandaDataProvider
-from oanda_executor import OandaExecutor, OandaAPIError
-
-logger = logging.getLogger(__name__)
-
-# ── Constants ──────────────────────────────────────────────────────────
-
-SYMBOL = XAUUSD
-INSTRUMENT = "XAU_USD"
-
-SCAN_CONFIRMATIONS = {
-    Timeframe.M5:  [Timeframe.M15, Timeframe.H1],
-    Timeframe.M15: [Timeframe.M5, Timeframe.H1],
-    Timeframe.H1:  [Timeframe.M15, Timeframe.H4],
-    Timeframe.H4:  [Timeframe.H1, Timeframe.D1],
-}
-
-TF_INTERVAL_MINUTES = {
-    Timeframe.M5: 5, Timeframe.M15: 15, Timeframe.M30: 30,
-    Timeframe.H1: 60, Timeframe.H4: 240, Timeframe.D1: 1440,
-}
-
-SCAN_TIMEFRAMES = [Timeframe.M5, Timeframe.M15, Timeframe.H1, Timeframe.H4]
-
-QUALITY_RANK = {"A+": 4, "A": 3, "B": 2, "C": 1, "D": 0, "F": -1}
-
-# Matches the backtester's --cooldown-bars default (the only sequential
-# rule in the replay): no new entry within 5 bars of the previous entry
-# on the same timeframe.
-COOLDOWN_BARS = 5
-
-# How often the heartbeat confirms the process is alive and the account
-# is still reachable — independent of the candle-boundary loops, so a
-# wedged scan or mid-run credential expiry still surfaces in the log.
-HEARTBEAT_SECONDS = 3600
-
-STATE_FILE = os.path.join(
-    os.path.dirname(os.path.abspath(__file__)), "oanda_state.json")
-
-# ── Config resolution (env → config.json → default) ──────────────────
-
-def _oanda_setting_int(env_key: str, json_key: str, default: int) -> int:
-    env = os.environ.get(env_key)
-    if env:
-        try:
-            return int(env)
-        except ValueError:
-            raise SystemExit(
-                f"{env_key}={env!r} is not an integer — refusing to start")
-    try:
-        return int(get_app_config().oanda_settings.get(json_key, default))
-    except (TypeError, ValueError):
-        raise SystemExit(
-            f"config.json oanda.{json_key} is not an integer — "
-            f"refusing to start")
-
-
-def _oanda_setting_str(env_key: str, json_key: str, default: str) -> str:
-    return (os.environ.get(env_key)
-            or str(get_app_config().oanda_settings.get(json_key, default)))
-
-# ── Logging setup ──────────────────────────────────────────────────────
-
-def _setup_logging(level: str = "INFO") -> None:
-    log_dir = os.path.join(os.path.dirname(os.path.abspath(__file__)), "logs")
-    os.makedirs(log_dir, exist_ok=True)
-    log_file = os.path.join(log_dir, "daemon.log")
-
-    fmt = logging.Formatter(
-        "%(asctime)s [%(levelname)s] %(name)s: %(message)s",
-        datefmt="%Y-%m-%d %H:%M:%S")
-
-    file_handler = TimedRotatingFileHandler(
-        log_file, when="h", interval=3, backupCount=240, utc=True)
-    file_handler.setFormatter(fmt)
-
-    console_handler = logging.StreamHandler(sys.stdout)
-    console_handler.setFormatter(fmt)
-
-    logging.basicConfig(
-        level=getattr(logging, level.upper(), logging.INFO),
-        handlers=[file_handler, console_handler],
-        force=True)
-
-# ── Boundary timing (same as telegram_bot.py) ─────────────────────────
-
-def _seconds_until_boundary(interval_minutes: int, buffer: int = 10) -> float:
-    now = datetime.now(timezone.utc)
-    total_minutes = now.hour * 60 + now.minute
-    elapsed_minutes = total_minutes % interval_minutes
-    elapsed_seconds = elapsed_minutes * 60 + now.second + now.microsecond / 1e6
-    interval_seconds = interval_minutes * 60
-    remaining = interval_seconds - elapsed_seconds + buffer
-    # NOTE: remaining > buffer always (elapsed < interval by construction);
-    # a call at the exact boundary instant targets the NEXT boundary —
-    # steady-state loops wake at boundary+buffer, so only a process
-    # started in the boundary microsecond skips one scan (2026-08-04
-    # audit I42; the old `remaining <= buffer` guard was unreachable).
-    return remaining
-
-# ── Trade state persistence ───────────────────────────────────────────
-
-def _load_state() -> Dict:
-    if os.path.exists(STATE_FILE):
-        try:
-            with open(STATE_FILE, encoding="utf-8") as fh:
-                return json.load(fh)
-        except (OSError, json.JSONDecodeError):
-            logger.warning("Could not load state file, starting fresh")
-    return {"trades": {}, "last_entries": {}}
-
-
-def _save_state(state: Dict) -> None:
-    try:
-        tmp = STATE_FILE + ".tmp"
-        with open(tmp, "w", encoding="utf-8") as fh:
-            json.dump(state, fh, indent=2)
-        os.replace(tmp, STATE_FILE)
-    except OSError:
-        logger.exception("Failed to save state file")
-
-# ── Trading Daemon ─────────────────────────────────────────────────────
-
-class TradingDaemon:
-
-    def __init__(self):
-        account_id = oanda_account_id()
-        api_token = oanda_api_token()
-        base_url = oanda_base_url()
-        lot_size = _oanda_setting_int("OANDA_LOT_SIZE", "lot_size", 1)
-        self._quality_gate = _oanda_setting_str(
-            "OANDA_QUALITY_GATE", "quality_gate", "C").strip()
-        if self._quality_gate not in QUALITY_RANK:
-            # Fail CLOSED: an unrecognized gate must never silently fall
-            # back to the most permissive rank (QUALITY_RANK.get default).
-            raise SystemExit(
-                f"OANDA_QUALITY_GATE/quality_gate {self._quality_gate!r} "
-                f"is not one of {sorted(QUALITY_RANK)} — refusing to start")
-        self._max_hold_bars = _oanda_setting_int(
-            "OANDA_MAX_HOLD_BARS", "max_hold_bars", 300)
-        self._price_precision = PAIR_INFO[SYMBOL]["price_precision"]
-
-        if not account_id or not api_token:
-            msg = (
-                "Missing Oanda credentials. Set OANDA_ACCOUNT_ID and "
-                "OANDA_API_TOKEN env vars or config.json oanda section.")
-            logger.critical(msg)
-            raise SystemExit(msg)
-
-        self._provider = OandaDataProvider(
-            PlatformConfig(), api_token, base_url)
-        self._executor = OandaExecutor(
-            account_id, api_token, base_url, lot_size,
-            price_precision=self._price_precision)
-        self._thread_pool = ThreadPoolExecutor(max_workers=4)
-
-        self._state = _load_state()
-        self._state.setdefault("trades", {})
-        self._state.setdefault("last_entries", {})
-        self._state_lock = threading.Lock()
-        # Serializes the position-check + order-placement critical section
-        # across concurrent TF loops (all four fire at aligned boundaries).
-        self._entry_lock = threading.Lock()
-        self._running = True
-
-    # ── Main entry point ───────────────────────────────────────────
-
-    async def run(self) -> None:
-        loop = asyncio.get_running_loop()
-        loop.set_exception_handler(self._asyncio_exception_handler)
-        for sig in (signal.SIGINT, signal.SIGTERM):
-            loop.add_signal_handler(sig, self._request_shutdown)
-
-        logger.info("=== Oanda Trading Daemon starting ===")
-        logger.info(
-            "Quality gate: %s | Max hold: %d bars | Cooldown: %d bars | "
-            "Timeframes: %s",
-            self._quality_gate, self._max_hold_bars, COOLDOWN_BARS,
-            "/".join(tf.name for tf in SCAN_TIMEFRAMES))
-
-        # Credentials first: with a bad token every later call fails, so
-        # refuse to start before touching positions or spawning loops.
-        self._verify_account_access()
-
-        self._discover_existing_positions()
-
-        tasks = []
-        for tf in SCAN_TIMEFRAMES:
-            interval = TF_INTERVAL_MINUTES[tf]
-            tasks.append(asyncio.create_task(
-                self._candle_loop(tf, interval)))
-        tasks.append(asyncio.create_task(self._heartbeat_loop()))
-
-        logger.info("Candle-boundary loops started")
-        try:
-            await asyncio.gather(*tasks)
-        except asyncio.CancelledError:
-            pass
-        finally:
-            self._shutdown()
-
-    def _verify_account_access(self) -> None:
-        """Boot-time credential check. A 4xx here means the token/account
-        id is wrong — that never fixes itself, so refuse to start rather
-        than run a daemon that can't place or manage trades. A connection
-        failure or 5xx is a transient outage: log it and continue, since
-        broker-side stops already protect any open position and the
-        hourly heartbeat will keep reporting account health."""
-        try:
-            self._executor.get_account_summary()
-        except OandaAPIError as e:
-            if 400 <= e.status_code < 500:
-                logger.critical(
-                    "Oanda rejected the account credentials (HTTP %d): "
-                    "%s — refusing to start", e.status_code, e)
-                raise SystemExit(
-                    f"Oanda rejected the account credentials "
-                    f"(HTTP {e.status_code}): {e}")
-            logger.error(
-                "Failed to fetch account summary at boot (HTTP %d): %s",
-                e.status_code, e)
-
-    def _asyncio_exception_handler(self, loop, context) -> None:
-        exc = context.get("exception")
-        if exc is not None:
-            logger.critical(
-                "Uncaught asyncio exception: %s",
-                context.get("message", ""), exc_info=exc)
-        else:
-            logger.critical(
-                "Uncaught asyncio error: %s", context.get("message", ""))
-
-    def _request_shutdown(self) -> None:
-        logger.info("Shutdown requested")
-        self._running = False
-        for task in asyncio.all_tasks():
-            task.cancel()
-
-    def _shutdown(self) -> None:
-        with self._state_lock:
-            _save_state(self._state)
-        self._thread_pool.shutdown(wait=False)
-        logger.info("=== Oanda Trading Daemon stopped ===")
-
-    # ── Position discovery (restart recovery) ──────────────────────
-
-    def _discover_existing_positions(self) -> None:
-        self._reconcile_positions("boot discovery")
-
-    def _reconcile_positions(self, reason: str) -> None:
-        """Re-sync tracked state with the broker's open trades. Safe to
-        call mid-run (lock-guarded); used at boot and whenever a fill or
-        placement error suggests state and broker may have diverged."""
-        try:
-            open_trades = self._executor.get_open_trades(INSTRUMENT)
-        except OandaAPIError:
-            logger.exception(
-                "Failed to query open trades (%s) — reconcile skipped",
-                reason)
-            return
-
-        live_ids = {t["id"] for t in open_trades}
-
-        with self._state_lock:
-            saved = self._state.get("trades", {})
-
-            stale = [tid for tid in saved if tid not in live_ids]
-            for tid in stale:
-                logger.info(
-                    "Stale trade %s in state file (closed on Oanda), "
-                    "removing", tid)
-                del saved[tid]
-
-            for trade in open_trades:
-                tid = trade["id"]
-                if tid in saved:
-                    logger.info(
-                        "Recovered trade %s from state: %s %s @ %.2f, SL %.2f",
-                        tid, saved[tid].get("direction"), INSTRUMENT,
-                        saved[tid].get("entry"), saved[tid].get("current_sl"))
-                    continue
-
-                direction = ("BUY" if float(trade.get("currentUnits", 0)) > 0
-                             else "SELL")
-                entry = float(trade.get("price", 0))
-                sl_order = trade.get("stopLossOrder") or {}
-                sl_price = sl_order.get("price")
-                if sl_price is None:
-                    # No broker-side stop at all: risk is undefined, so the
-                    # trailing model cannot manage this trade. Scream — a
-                    # naked position needs a human (or a policy) to set a
-                    # stop; fabricating one here would invent trading
-                    # behavior the books never validated.
-                    logger.critical(
-                        "Adopted trade %s (%s @ %.2f) has NO STOP LOSS on "
-                        "the broker — it will NOT be trailed. Set a stop "
-                        "manually or close it.", tid, direction, entry)
-                    sl = entry
-                    risk = 0.0
-                else:
-                    sl = float(sl_price)
-                    risk = abs(entry - sl)
-
-                saved[tid] = {
-                    "tf": "H1",
-                    "direction": direction,
-                    "entry": entry,
-                    "sl_initial": sl,
-                    "current_sl": sl,
-                    "risk": risk,
-                    "peak": entry,
-                    "trail_armed": False,
-                    "trail_arm_r": 0.6,
-                    "trail_dist_r": 0.75,
-                    "bars_held": 0,
-                    "last_bar_ts": None,
-                    "signal_bar_ts": None,
-                }
-                logger.info(
-                    "Adopted unknown trade %s: %s @ %.2f, SL %.2f, risk %.2f "
-                    "(conservative defaults)",
-                    tid, direction, entry, sl, risk)
-
-            self._state["trades"] = saved
-            _save_state(self._state)
-        logger.info(
-            "Position reconcile (%s): %d active trade(s)", reason, len(saved))
-
-    # ── Candle-boundary loop ───────────────────────────────────────
-
-    async def _candle_loop(self, tf: Timeframe, interval_minutes: int) -> None:
-        label = tf.value
-        try:
-            while self._running:
-                wait = _seconds_until_boundary(interval_minutes, buffer=10)
-                logger.info("%s loop: next scan in %.0fs", label, wait)
-                await asyncio.sleep(wait)
-                if not self._running:
-                    break
-                logger.info("%s loop: firing scan", label)
-                try:
-                    await self._scan_and_manage(tf)
-                except Exception:
-                    logger.exception("%s loop: cycle failed", label)
-        except asyncio.CancelledError:
-            pass
-        finally:
-            logger.info("%s loop stopped", label)
-
-    async def _scan_and_manage(self, tf: Timeframe) -> None:
-        loop = asyncio.get_running_loop()
-
-        # Trade management FIRST: the SL ratchet must not wait on (or be
-        # starved by) the signal scan — news fetches can take seconds and
-        # a failed scan must never cost an open trade its trail update.
-        try:
-            await loop.run_in_executor(
-                self._thread_pool, self._manage_open_trades, tf)
-        except Exception:
-            logger.exception("%s: trade management failed", tf.value)
-
-        report = await loop.run_in_executor(
-            self._thread_pool, self._run_signal, tf)
-
-        if not self._should_execute(report, tf):
-            return
-
-        await loop.run_in_executor(
-            self._thread_pool, self._execute_trade, report, tf)
-
-    def _manage_open_trades(self, tf: Timeframe) -> None:
-        self._check_closed_trades()
-        self._manage_trailing(tf)
-
-    # ── Heartbeat ─────────────────────────────────────────────────
-
-    def _heartbeat_once(self) -> None:
-        with self._state_lock:
-            n = len(self._state.get("trades", {}))
-        try:
-            self._executor.get_account_summary()
-        except OandaAPIError as e:
-            logger.error("HEARTBEAT: account summary failed: %s", e)
-        # Logged unconditionally — even on a failed summary call the
-        # heartbeat line itself must reach the log, so a wedged process
-        # doesn't look identical to a healthy but silent one.
-        logger.info("HEARTBEAT: alive | tracked trades: %d", n)
-
-    async def _heartbeat_loop(self) -> None:
-        loop = asyncio.get_running_loop()
-        try:
-            while self._running:
-                await asyncio.sleep(HEARTBEAT_SECONDS)
-                if not self._running:
-                    break
-                try:
-                    await loop.run_in_executor(
-                        self._thread_pool, self._heartbeat_once)
-                except Exception:
-                    logger.exception("heartbeat loop: cycle failed")
-        except asyncio.CancelledError:
-            pass
-        finally:
-            logger.info("heartbeat loop stopped")
-
-    # ── Signal pipeline ────────────────────────────────────────────
-
-    def _run_signal(self, tf: Timeframe) -> SignalReport:
-        conf_tfs = SCAN_CONFIRMATIONS.get(
-            tf, [Timeframe.M15, Timeframe.H4])
-        factory = PAIR_PRODUCERS.get(SYMBOL)
-        if factory is not None:
-            config = factory(tf, conf_tfs)
-        else:
-            config = PlatformConfig()
-            config.primary_timeframe = tf
-            config.confirmation_timeframes = conf_tfs
-
-        aggregator = SignalAggregator(config)
-        aggregator.data_provider = self._provider
-        report = aggregator.analyze_pair(symbol=SYMBOL)
-
-        logger.info(
-            "%s scan: signal=%s score=%.1f conf=%.0f quality=%s "
-            "regime=%s vol=%s price=%.2f spread=%.4f",
-            tf.value,
-            report.final_signal.name, report.final_score,
-            report.final_confidence, report.signal_quality,
-            report.market_regime, report.volatility_regime,
-            report.current_price,
-            self._provider.get_spread(SYMBOL))
-
-        # Stale-data/suppression warnings must be visible at WARNING;
-        # routine session-filter notices go to DEBUG — the M5 loop fires
-        # 288x/day and would drown the log at WARNING otherwise.
-        for w in report.warnings:
-            if "STALE" in w.upper() or "FAIL" in w.upper():
-                logger.warning("%s scan: %s", tf.value, w)
-            else:
-                logger.debug("%s scan: %s", tf.value, w)
-
-        return report
-
-    # ── Execution decision ─────────────────────────────────────────
-
-    def _should_execute(self, report: SignalReport, tf: Timeframe) -> bool:
-        if report.final_signal == SignalStrength.NEUTRAL:
-            return False
-
-        if report.trade_setup is None:
-            return False
-
-        quality_rank = QUALITY_RANK.get(report.signal_quality, -1)
-        gate_rank = QUALITY_RANK.get(self._quality_gate, 1)
-        if quality_rank < gate_rank:
-            return False
-
-        if self._in_cooldown(tf):
-            logger.info(
-                "%s: actionable signal (%s %s) but inside %d-bar entry "
-                "cooldown, skipping",
-                tf.value, report.trade_setup.direction,
-                report.signal_quality, COOLDOWN_BARS)
-            return False
-
-        with self._state_lock:
-            if self._state.get("trades"):
-                logger.info(
-                    "%s: actionable signal (%s %s) but position already "
-                    "open, skipping",
-                    tf.value, report.trade_setup.direction,
-                    report.signal_quality)
-                return False
-
-        return True
-
-    def _in_cooldown(self, tf: Timeframe) -> bool:
-        """Backtest parity: no entry within COOLDOWN_BARS bars of the
-        previous entry on the same timeframe (wall-clock approximation)."""
-        with self._state_lock:
-            last = self._state.get("last_entries", {}).get(tf.value)
-        if not last:
-            return False
-        try:
-            last_ts = datetime.fromisoformat(last)
-        except ValueError:
-            return False
-        interval_s = TF_INTERVAL_MINUTES[tf] * 60
-        elapsed = (datetime.now(timezone.utc) - last_ts).total_seconds()
-        return elapsed < COOLDOWN_BARS * interval_s
-
-    # ── Trade execution ────────────────────────────────────────────
-
-    def _execute_trade(self, report: SignalReport, tf: Timeframe) -> None:
-        ts = report.trade_setup
-        if ts is None:
-            return
-
-        # All four TF loops can fire at an aligned boundary (e.g. :00) and
-        # pass _should_execute before any of them has recorded a fill —
-        # re-check under the entry lock so only one order goes out.
-        with self._entry_lock:
-            with self._state_lock:
-                if self._state.get("trades"):
-                    logger.info(
-                        "%s: position opened by a concurrent scan, skipping",
-                        tf.value)
-                    return
-
-            # NO take-profit on the order: the backtested trailing model
-            # exits via SL/TRAIL/TIMEOUT only — a hard TP would cap
-            # winners and diverge the live book from the backtest.
-            try:
-                result = self._executor.place_market_order(
-                    instrument=INSTRUMENT,
-                    direction=ts.direction,
-                    stop_loss=ts.stop_loss,
-                )
-            except OandaAPIError:
-                # The order is never retried (a timeout after a server-side
-                # fill would double-up on retry), but the server MAY have
-                # filled before the error reached us — reconcile so a live
-                # position can't stay untracked until the next restart.
-                logger.exception(
-                    "Failed to place %s order for %s %s — reconciling "
-                    "broker positions",
-                    ts.direction, SYMBOL, tf.value)
-                self._reconcile_positions("order placement error")
-                return
-
-            fill_tx = result.get("orderFillTransaction")
-            if not fill_tx:
-                logger.warning(
-                    "Order not filled for %s %s", SYMBOL, tf.value)
-                return
-
-            trade_opened = fill_tx.get("tradeOpened", {})
-            trade_id = trade_opened.get("tradeID")
-            if not trade_id:
-                # A fill with no tradeOpened means the order interacted
-                # with an existing broker position (tradesClosed /
-                # tradeReduced on a netting account) — state and broker
-                # have diverged. Surface it and re-sync instead of
-                # silently forgetting a real fill.
-                logger.critical(
-                    "Order FILLED but opened no trade (netting close/"
-                    "reduce?) — fill: closed=%s reduced=%s. Reconciling.",
-                    fill_tx.get("tradesClosed"), fill_tx.get("tradeReduced"))
-                self._reconcile_positions("fill without tradeOpened")
-                return
-
-            fill_price = float(fill_tx.get("price", ts.entry_price))
-            risk = abs(fill_price - ts.stop_loss)
-
-            trail_arm = ts.trail_arm_r if ts.trail_arm_r is not None else 0.6
-            trail_dist = ts.trail_dist_r if ts.trail_dist_r is not None else 0.75
-
-            now_iso = datetime.now(timezone.utc).isoformat()
-            trade_state = {
-                "tf": tf.value,
-                "direction": ts.direction,
-                "entry": fill_price,
-                "sl_initial": ts.stop_loss,
-                "current_sl": ts.stop_loss,
-                "tp1": ts.take_profit_1,
-                "tp2": ts.take_profit_2,
-                "risk": risk,
-                "peak": fill_price,
-                "trail_armed": False,
-                "trail_arm_r": trail_arm,
-                "trail_dist_r": trail_dist,
-                "bars_held": 0,
-                # The signal bar is already closed; the first bar this
-                # trade lives through is the one that just started. Guard
-                # so a lagging feed can't trail on the signal bar itself.
-                "signal_bar_ts": report.timestamp,
-                "last_bar_ts": None,
-                "strategy": ts.strategy,
-                "quality": report.signal_quality,
-                "rr": ts.risk_reward,
-                "opened_at": now_iso,
-            }
-
-            with self._state_lock:
-                self._state["trades"][trade_id] = trade_state
-                self._state["last_entries"][tf.value] = now_iso
-                _save_state(self._state)
-
-        logger.info(
-            "TRADE OPENED: %s %s @ %.2f | SL %.2f | TP1 %.2f (advisory) | "
-            "risk $%.2f | RR %.2f | quality %s | strategy %s | TF %s | "
-            "trail arm %.2fR dist %.2fR",
-            ts.direction, SYMBOL, fill_price, ts.stop_loss,
-            ts.take_profit_1,
-            risk, ts.risk_reward, report.signal_quality,
-            ts.strategy, tf.value, trail_arm, trail_dist)
-
-    # ── Trailing stop management ───────────────────────────────────
-
-    def _manage_trailing(self, tf: Timeframe) -> None:
-        # Copy the per-trade dicts too: the loop mutates info (peak,
-        # bars_held, last_bar_ts) and must not touch live state outside
-        # the lock — another thread may be serializing it in _save_state.
-        with self._state_lock:
-            trades = {tid: dict(info)
-                      for tid, info in self._state.get("trades", {}).items()}
-
-        for trade_id, info in trades.items():
-            if info.get("tf") != tf.value:
-                continue
-
-            try:
-                df = self._provider.fetch_ohlcv(SYMBOL, tf)
-            except Exception:
-                logger.exception(
-                    "Failed to fetch candle data for trail update")
-                continue
-
-            if df.empty:
-                logger.warning(
-                    "%s: no candle data for trail update of trade %s",
-                    tf.value, trade_id)
-                continue
-
-            bar_ts = df.index[-1].isoformat()
-
-            # Process each closed bar exactly once. Weekends and feed lag
-            # re-serve the same last bar; without this guard bars_held
-            # inflates (an M5 trade would "time out" over a weekend) and
-            # the peak gets double-counted.
-            if bar_ts == info.get("last_bar_ts"):
-                continue
-            sig_ts = info.get("signal_bar_ts")
-            if sig_ts:
-                try:
-                    if df.index[-1] <= pd.Timestamp(sig_ts):
-                        continue
-                except (ValueError, TypeError):
-                    pass
-            info["last_bar_ts"] = bar_ts
-
-            info["bars_held"] = info.get("bars_held", 0) + 1
-
-            if info["bars_held"] >= self._max_hold_bars:
-                logger.info(
-                    "TIMEOUT: trade %s held %d bars, closing",
-                    trade_id, info["bars_held"])
-                try:
-                    self._executor.close_trade(trade_id)
-                except OandaAPIError:
-                    # Keep tracking — the trade is still live on the
-                    # broker (incl. a 200 whose close order was cancelled,
-                    # e.g. MARKET_HALTED — close_trade raises on no-fill);
-                    # we retry the close at the next bar.
-                    logger.exception(
-                        "Failed to close timed-out trade %s — will retry",
-                        trade_id)
-                    with self._state_lock:
-                        if trade_id in self._state["trades"]:
-                            self._state["trades"][trade_id] = info
-                            _save_state(self._state)
-                    continue
-                with self._state_lock:
-                    self._state["trades"].pop(trade_id, None)
-                    _save_state(self._state)
-                continue
-
-            last_bar = df.iloc[-1]
-            bar_high = float(last_bar["High"])
-            bar_low = float(last_bar["Low"])
-
-            if info.get("risk", 0) <= 0:
-                # Adopted naked position (no broker SL → risk undefined):
-                # never manageable by the trailing model — keep it loud
-                # instead of silently skipping forever.
-                logger.warning(
-                    "Trade %s has risk<=0 (no stop loss) — trailing "
-                    "skipped; set a stop manually", trade_id)
-
-            new_sl = self._compute_trail(info, bar_high, bar_low)
-
-            if new_sl is not None:
-                # Round to instrument precision and track exactly what the
-                # broker holds, so tracked vs broker SL can never drift.
-                new_sl = round(new_sl, self._price_precision)
-                if new_sl != info["current_sl"]:
-                    try:
-                        self._executor.modify_trade_sl(trade_id, new_sl)
-                        info["current_sl"] = new_sl
-                    except OandaAPIError:
-                        logger.exception(
-                            "Failed to update trailing SL for trade %s",
-                            trade_id)
-
-            with self._state_lock:
-                # Write back ONLY if the trade is still tracked — another
-                # loop's _check_closed_trades may have popped it while we
-                # iterated (broker-side close). Re-inserting would create
-                # a ghost that blocks new entries.
-                if trade_id in self._state["trades"]:
-                    self._state["trades"][trade_id] = info
-                    _save_state(self._state)
-
-    def _compute_trail(
-        self, info: Dict, bar_high: float, bar_low: float
-    ) -> Optional[float]:
-        buy = info["direction"] == "BUY"
-        risk = info["risk"]
-        if risk <= 0:
-            return None
-        arm = info.get("trail_arm_r", 0.6)
-        dist = info.get("trail_dist_r", 0.75)
-        if arm <= 0:
-            return None
-
-        fav = bar_high if buy else bar_low
-        if buy:
-            info["peak"] = max(info.get("peak", info["entry"]), fav)
-        else:
-            info["peak"] = min(info.get("peak", info["entry"]), fav)
-
-        exc = ((info["peak"] - info["entry"]) if buy
-               else (info["entry"] - info["peak"])) / risk
-
-        if exc >= arm:
-            if buy:
-                lvl = info["peak"] - dist * risk
-                cur_sl = info.get("current_sl", info["sl_initial"])
-                if lvl > cur_sl:
-                    info["trail_armed"] = True
-                    return lvl
-            else:
-                lvl = info["peak"] + dist * risk
-                cur_sl = info.get("current_sl", info["sl_initial"])
-                if lvl < cur_sl:
-                    info["trail_armed"] = True
-                    return lvl
-
-        return None
-
-    # ── Closed-trade detection ─────────────────────────────────────
-
-    def _check_closed_trades(self) -> None:
-        with self._state_lock:
-            tracked = set(self._state.get("trades", {}).keys())
-
-        if not tracked:
-            return
-
-        try:
-            open_trades = self._executor.get_open_trades(INSTRUMENT)
-        except OandaAPIError:
-            logger.exception("Failed to query open trades")
-            return
-
-        live_ids = {t["id"] for t in open_trades}
-        closed = tracked - live_ids
-
-        for tid in closed:
-            with self._state_lock:
-                info = self._state["trades"].pop(tid, {})
-            entry = info.get("entry", 0)
-            direction = info.get("direction", "?")
-            risk = info.get("risk", 1)
-            sl = info.get("current_sl", 0)
-            was_trailed = info.get("trail_armed", False)
-
-            logger.info(
-                "TRADE CLOSED BY BROKER: %s %s %s | entry %.2f | "
-                "SL was %.2f | risk $%.2f | trailed=%s | bars_held=%d",
-                tid, direction, SYMBOL, entry, sl, risk,
-                was_trailed, info.get("bars_held", 0))
-
-        if closed:
-            with self._state_lock:
-                _save_state(self._state)
-
-# ── Uncaught exception handlers ───────────────────────────────────────
-
-def _excepthook(exc_type, exc_value, exc_tb):
-    if issubclass(exc_type, KeyboardInterrupt):
-        sys.__excepthook__(exc_type, exc_value, exc_tb)
-        return
-    logger.critical(
-        "Uncaught exception", exc_info=(exc_type, exc_value, exc_tb))
-
-def _threading_excepthook(args):
-    logger.critical(
-        "Uncaught exception in thread %s",
-        args.thread.name if args.thread else "?",
-        exc_info=(args.exc_type, args.exc_value, args.exc_traceback))
-
-# ── Entry point ────────────────────────────────────────────────────────
-
-def main() -> None:
-    _setup_logging(
-        os.environ.get("OANDA_LOG_LEVEL") or get_app_config().log_level)
-    sys.excepthook = _excepthook
-    threading.excepthook = _threading_excepthook
-
-    daemon = TradingDaemon()
-    asyncio.run(daemon.run())
-
-
-if __name__ == "__main__":
-    main()

oanda_executor.py

@@ -1,230 +0,0 @@
-"""
-ForexBot2 — Oanda v20 Order Executor
-======================================
-Handles all Oanda account operations: market orders, trade queries,
-SL/TP modification, and trade closure.  Uses raw ``requests`` against
-the Oanda v20 REST API — no extra dependencies.
-"""
-
-import logging
-import time
-from typing import Dict, List, Optional
-
-import requests
-
-logger = logging.getLogger(__name__)
-
-
-class OandaAPIError(Exception):
-    """Raised when an Oanda API call fails."""
-
-    def __init__(self, message: str, status_code: int = 0, body: str = ""):
-        super().__init__(message)
-        self.status_code = status_code
-        self.body = body
-
-
-class OandaExecutor:
-    """Oanda v20 order execution and trade management."""
-
-    MAX_RETRIES = 1
-    RETRY_DELAY = 2.0
-
-    def __init__(
-        self,
-        account_id: str,
-        api_token: str,
-        base_url: str = "https://api-fxpractice.oanda.com",
-        lot_size: int = 1,
-        price_precision: int = 2,
-    ):
-        self._account_id = account_id
-        self._base_url = base_url.rstrip("/")
-        self._lot_size = lot_size
-        self._price_precision = price_precision
-        self._session = requests.Session()
-        self._session.headers.update({
-            "Authorization": f"Bearer {api_token}",
-            "Accept-Datetime-Format": "RFC3339",
-            "Content-Type": "application/json",
-        })
-
-    # ── Order Placement ────────────────────────────────────────────
-
-    def place_market_order(
-        self,
-        instrument: str,
-        direction: str,
-        stop_loss: float,
-        take_profit: Optional[float] = None,
-        units: Optional[int] = None,
-    ) -> Dict:
-        if units is None:
-            units = self._lot_size
-        signed_units = str(units) if direction.upper() == "BUY" else str(-units)
-
-        order_body = {
-            "order": {
-                "type": "MARKET",
-                "instrument": instrument,
-                "units": signed_units,
-                "timeInForce": "FOK",
-                # The book model is one position total; on a netting
-                # account an opposite-side order would silently CLOSE an
-                # existing (possibly untracked) position instead of opening
-                # the signal trade. OPEN_ONLY makes that case fail loudly
-                # (order cancelled) so the divergence surfaces instead of
-                # executing an unintended close.
-                "positionFill": "OPEN_ONLY",
-                "stopLossOnFill": {
-                    "price": self._fmt_price(stop_loss),
-                },
-            }
-        }
-        if take_profit is not None:
-            order_body["order"]["takeProfitOnFill"] = {
-                "price": self._fmt_price(take_profit),
-            }
-
-        url = f"{self._base_url}/v3/accounts/{self._account_id}/orders"
-        # Never retry order placement: a timeout after the server already
-        # filled would open a SECOND position on retry. Fail loudly instead;
-        # the next candle boundary re-evaluates.
-        resp = self._request_with_retry("POST", url, json=order_body, retries=0)
-        data = self._parse_json(resp)
-
-        fill_tx = data.get("orderFillTransaction")
-        if fill_tx:
-            trade_id = fill_tx.get("tradeOpened", {}).get("tradeID", "unknown")
-            fill_price = float(fill_tx.get("price", 0))
-            logger.info(
-                "ORDER FILLED: %s %s %s units @ %.2f — trade %s, SL %.2f%s",
-                direction.upper(), instrument, signed_units, fill_price,
-                trade_id, stop_loss,
-                f", TP {take_profit:.2f}" if take_profit else "")
-        else:
-            cancel_tx = data.get("orderCancelTransaction", {})
-            reason = cancel_tx.get("reason", "unknown")
-            logger.warning(
-                "ORDER NOT FILLED: %s %s — reason: %s",
-                direction.upper(), instrument, reason)
-
-        return data
-
-    # ── Trade Queries ──────────────────────────────────────────────
-
-    def get_open_trades(self, instrument: Optional[str] = None) -> List[Dict]:
-        url = f"{self._base_url}/v3/accounts/{self._account_id}/openTrades"
-        resp = self._request_with_retry("GET", url)
-        trades = self._parse_json(resp).get("trades", [])
-        if instrument:
-            trades = [t for t in trades if t.get("instrument") == instrument]
-        return trades
-
-    def get_trade(self, trade_id: str) -> Dict:
-        url = (f"{self._base_url}/v3/accounts/{self._account_id}"
-               f"/trades/{trade_id}")
-        resp = self._request_with_retry("GET", url)
-        return self._parse_json(resp).get("trade", {})
-
-    def get_account_summary(self) -> Dict:
-        url = f"{self._base_url}/v3/accounts/{self._account_id}/summary"
-        resp = self._request_with_retry("GET", url)
-        account = self._parse_json(resp).get("account", {})
-        logger.info(
-            "Account: balance=%.2f, NAV=%.2f, unrealizedPL=%.2f, openTrades=%s",
-            float(account.get("balance", 0)),
-            float(account.get("NAV", 0)),
-            float(account.get("unrealizedPL", 0)),
-            account.get("openTradeCount", 0))
-        return account
-
-    # ── Trade Modification ─────────────────────────────────────────
-
-    def modify_trade_sl(self, trade_id: str, new_sl: float) -> Dict:
-        url = (f"{self._base_url}/v3/accounts/{self._account_id}"
-               f"/trades/{trade_id}/orders")
-        body = {"stopLoss": {"price": self._fmt_price(new_sl)}}
-        resp = self._request_with_retry("PUT", url, json=body)
-        logger.info(
-            "TRAIL UPDATE: trade %s — new SL %.2f", trade_id, new_sl)
-        return self._parse_json(resp)
-
-    def modify_trade_tp(self, trade_id: str, new_tp: float) -> Dict:
-        url = (f"{self._base_url}/v3/accounts/{self._account_id}"
-               f"/trades/{trade_id}/orders")
-        body = {"takeProfit": {"price": self._fmt_price(new_tp)}}
-        resp = self._request_with_retry("PUT", url, json=body)
-        logger.info("TP UPDATE: trade %s — new TP %.2f", trade_id, new_tp)
-        return self._parse_json(resp)
-
-    def close_trade(self, trade_id: str) -> Dict:
-        url = (f"{self._base_url}/v3/accounts/{self._account_id}"
-               f"/trades/{trade_id}/close")
-        resp = self._request_with_retry("PUT", url, json={"units": "ALL"})
-        data = self._parse_json(resp)
-        close_tx = data.get("orderFillTransaction")
-        if not close_tx:
-            # A 200 whose close order was created but NOT filled (e.g.
-            # cancelled with MARKET_HALTED during the settlement pause)
-            # means the position is STILL OPEN. Raising keeps the caller
-            # tracking it and retrying — silently returning here would
-            # orphan a live position.
-            reason = data.get("orderCancelTransaction", {}).get(
-                "reason", "no orderFillTransaction in response")
-            raise OandaAPIError(
-                f"Close of trade {trade_id} did not fill: {reason}",
-                resp.status_code, str(data)[:300])
-        close_price = float(close_tx.get("price", 0))
-        pl = float(close_tx.get("pl", 0))
-        logger.info(
-            "TRADE CLOSED: trade %s @ %.2f — PL %.2f",
-            trade_id, close_price, pl)
-        return data
-
-    # ── Internals ──────────────────────────────────────────────────
-
-    @staticmethod
-    def _parse_json(resp: requests.Response) -> Dict:
-        try:
-            return resp.json()
-        except ValueError as e:
-            raise OandaAPIError(
-                f"Oanda returned a non-JSON response (HTTP {resp.status_code})",
-                resp.status_code, resp.text[:300]) from e
-
-    def _fmt_price(self, price: float) -> str:
-        # Oanda rejects prices with more decimals than the instrument's
-        # displayPrecision (XAU_USD: sub-cent precision is invalid).
-        return f"{price:.{self._price_precision}f}"
-
-    def _request_with_retry(
-        self, method: str, url: str, retries: Optional[int] = None, **kwargs
-    ) -> requests.Response:
-        if retries is None:
-            retries = self.MAX_RETRIES
-        last_err = None
-        for attempt in range(1 + retries):
-            try:
-                resp = self._session.request(method, url, timeout=15, **kwargs)
-                if resp.status_code >= 500:
-                    raise OandaAPIError(
-                        f"Oanda server error {resp.status_code}",
-                        resp.status_code, resp.text)
-                if resp.status_code >= 400:
-                    raise OandaAPIError(
-                        f"Oanda API error {resp.status_code}: {resp.text}",
-                        resp.status_code, resp.text)
-                return resp
-            except (requests.ConnectionError, requests.Timeout,
-                    OandaAPIError) as e:
-                last_err = e
-                if isinstance(e, OandaAPIError) and 400 <= e.status_code < 500:
-                    raise
-                if attempt < retries:
-                    logger.warning(
-                        "Oanda request failed (attempt %d/%d): %s — retrying in %.0fs",
-                        attempt + 1, 1 + retries, e,
-                        self.RETRY_DELAY)
-                    time.sleep(self.RETRY_DELAY)
-        raise OandaAPIError(f"Oanda request failed after retries: {last_err}")

pair_signals/__init__.py

@@ -12,7 +12,7 @@ from typing import Optional, List, Dict, Callable
 
 from config import PlatformConfig, Timeframe
 from signal_aggregator import SignalAggregator, SignalReport
-from providers import BaseDataProvider
+from data_provider import ForexDataProvider
 
 PAIR_PRODUCERS: Dict[str, Callable] = {}
 
@@ -28,7 +28,7 @@ def get_supported_pairs() -> List[str]:
 def produce_signal(
     symbol: str,
     primary_tf: Timeframe,
-    data_provider: BaseDataProvider,
+    data_provider: ForexDataProvider,
     confirmation_tfs: Optional[List[Timeframe]] = None,
 ) -> SignalReport:
     config_factory = PAIR_PRODUCERS.get(symbol)

pair_signals/xauusd.py

@@ -56,40 +56,6 @@ positive, val half -1.89R — marginal but sign-correct in aggregate).
 Gold's structural uptrend makes bullish patterns fundamentally more
 reliable. M15/H1 fail on all windows; M5-only enablement.
 
-M5 override 3 (validated 2026-07-21, 5-year replay 2021-07-21 ->
-2026-07-21, $0.30 spread, next-open fills): trailing distance widened
-to 0.85R (trail_dist_r 0.85, arm unchanged 0.60). Same 30-cell grid
-methodology as the M15 campaign, opposite finding — on M5 the arm
-point barely matters; the winners need a looser leash. Plateau arm
-0.5-0.7 x dist 0.85-1.0 (+227..+234R vs +218.0R baseline). Chosen
-0.60/0.85 (OOS-preferred end of the plateau): +231.2R/2028 PF 1.30 WR
-57.1% vs +218.0R PF 1.29 WR 58.3%; both last-12mo halves better
-(+56.8/+115.1 vs +54.0/+104.6); 5yr max DD 19.2R vs 19.1R (noise);
-OOS 2024-06-01 -> 2025-03-30 +14.5R DD 15.1 vs +11.1R DD 15.4.
-dist 1.0 was REJECTED at the OOS gate (+10.9R < baseline +11.1R)
-despite the best in-sample total (+233.3R) — the OOS ordering picks
-the conservative plateau edge. XAU_PB trades are unaffected (they
-carry their own validated pb_trail 0.8/0.75 per-trade overrides);
-XAU_CP trades share the global params and were validated inside this
-book. M15 arms at 0.70 (its own campaign, same day); the divergence
-is the per-TF pattern yet again.
-
-M15 override 4 (validated 2026-07-21, 5-year replay 2021-07-21 ->
-2026-07-21, $0.30 spread, next-open fills): trailing-stop arm raised to
-0.70R (trail_arm_r 0.70, dist unchanged 0.75). Full grid arm 0.3-0.9 x
-dist 0.5-1.0 (30 cells): every earlier-arm cell (0.3/0.45) is WORSE —
-the "arm earlier to save fading trades" hypothesis is refuted; the
-plateau is arm 0.70-0.80 x dist 0.70-0.85 (all >= +190R). Chosen
-plateau center 0.70/0.75: +210.7R/1629 PF 1.35 WR 57.8% vs baseline
-0.60/0.75 +175.7R PF 1.30 WR 53.9%; both last-12mo halves improve
-(+49.7/+50.0 vs +36.4/+46.0); per-year better 4/6 (2022 -0.9R, 2023
--1.4R, both within tolerance); OOS 2024-06-01 -> 2025-03-30 +57.4R
-PF 1.57 DD 17.0R vs +45.4R PF 1.47 DD 17.8R — better on every OOS
-metric. One clause waived BY OWNER DECISION 2026-07-21: 5yr max DD is
-33.9R vs 29.1R baseline (+4.8R deeper worst episode, 2022-23 chop),
-accepted in exchange for +20% total R. M5 keeps the global 0.6 arm
-(same per-TF divergence pattern as tp_floor_rr and the hour-9 block).
-
 M5 override (validated 2026-07-11, full-history replay 2020-03-17 ->
 2026-07-11 + 12mo book, $0.30 spread): block hour 9 UTC (post-London-
 open whipsaw). Hour-9 entries lost -28.9R/143 over 6 years, negative
@@ -99,77 +65,10 @@ in 5/7 calendar years including winning years (2022 -9.2R). Blocking:
 12mo book is unchanged (+139.0R vs +139.3R). M15-and-up keep hour 9:
 it is POSITIVE there (M15 6yr +1.7R, M15 OOS +8.6R), so the block is
 M5-only, same per-timeframe pattern as the M15 overrides.
-
-M5 override 2 — relaxation campaign (validated 2026-07-30, Oanda
-bars, $0.30 spread, trailing exits): UNBLOCK hour 21 UTC on M5 only,
-combined with the config.py M5 gate relaxations (m5_min_abs_score
-30->25, m5_min_confirmation_score 10->5). The combined config beat
-the pre-relaxation control in ALL 6 calendar years of the 5yr replay
-2021-07-30 -> 2026-07-30 (+230.9R/2319 PF 1.26 DD 21.3 vs
-+168.0R/2048 PF 1.21 DD 27.0), on 12mo (+107.4R PF 1.36, halves
-+68.6/+38.8 vs +84.6R PF 1.33), on OOS 2024-06-01 -> 2025-03-30
-(+26.1R vs +17.2R), and at $0.70 spread (+86.9R vs +66.4R). Hour-21
-trades alone: +28.0R/22, positive every year — overturning the old
-block per the full-evidence rule. Hour 21 stays BLOCKED on M15+
-(12mo M15 unblock tested -0.3R). Declined-but-open from the same
-campaign: an M5 hour-22 block (+4.8R on 12mo — the burst-heavy
-"institutional" hour — but -3.2R on 5yr and -1.1R on OOS: regime-
-specific); M15 hour-16 unblock (+0.8R/+57 trades 12mo but -18.3R
-DD +34% on 5yr); every SELL-gate/vol/TP-size relaxation retested
-negative (see memory notes for the full grid).
-
-M5 override 4 — XAU_PB disabled (validated 2026-08-04, Oanda bars,
-$0.65 REAL spread — first campaign validated at measured cost on the
-live feed, per the 2026-08-04 re-pin baseline): m5_pullback_enabled
-False on M5. Loss forensics on the Oanda books identified XAU_PB as
-the largest wrong-direction loss source (12mo: -43.5R of the -156.6R
-wrong-direction bucket; 5yr: -274.5R). Disabling: 12mo +96.6R/588
-PF 1.44 max DD 13.9R halves +51.2/+45.4 vs +82.3R/788 PF 1.26 DD
-23.9R shipped (which FAILS the PF>=1.3 clause at real cost); OOS
-2024-06-01 -> 2025-03-30 +9.5R PF 1.34; 5yr +123.9R/956 PF 1.35 vs
-PF 1.16 with PB. PB's 5yr net contribution (+23.6R) was almost
-entirely the 2022 chop year (+36.1R vs -2.2R without) — a regime
-earner that has been a drag on trend-regime bars since; its tight
-stops are also the most cost-punished at $0.65. PB code and the
-pb_trail_* params STAY for future regime-gated experiments (same
-pattern as cp_enabled). Rejected in the same campaign (4 candidates
-total, N logged): hour-19 block (+3.7R 12mo but -3.3R OOS, -1.5R
-5yr — OOS gate, the classic trap); hours-14/15 block (-6.9R 12mo
-once PB is removed — their raw negativity WAS the PB losses);
-trail_dist_r 0.75 (-2.6R 12mo; the 0.88R average give-back is the
-validated price of riding trends, exits confirmed NOT the loss
-driver: exit-management accounts for 1-2% of lost R vs 50%
-wrong-direction / 49% entry-timing).
-
-M5 override 5 — EMA partial-alignment squash (validated 2026-08-04,
-Oanda bars, $0.65 real spread): ema_partial_neutral True on M5 —
-partial 4-EMA ribbon alignment (2-3 of 4 checks) scores NEUTRAL
-instead of a weak BUY/SELL. Motivated by the indicator audit: partial-
-alignment trades averaged -0.008R/84 vs +0.230R/341 fully aligned.
-Replay (knob mirrored in the backtester's precomputed EMA scorer in
-the same commit — live==backtest parity): 12mo +97.6R/556 PF 1.48
-DD 10.7R halves +47.8/+49.8 vs control +96.6R/588 PF 1.44 DD 13.9R;
-5yr +127.4R/912 PF 1.38 DD 12.8R vs +123.9R/956 PF 1.35 DD 13.2R,
-2022 improves -2.2 -> -1.1; OOS 2024-06-01 -> 2025-03-30 +9.4R
-PF 1.33 vs +9.5R PF 1.34 — a one-trade tie, documented as such (the
-OOS window holds up; it neither prefers nor punishes). Net effect:
-32 fewer marginal trades, every risk-adjusted measure better.
-Rejected in the same round (trial-logged): confirmation-TF swap
-M5->[M15,H4] (12mo +104.0R PF 1.51 DD 10.3 but OOS +7.8R PF 1.27
-vs control +9.5R PF 1.34 and 2022 3x worse — the H4 confirm keeps
-endorsing stale direction in chop; OOS-ordering rejection, same
-precedent as trail dist 1.0); confirm [H1,H4] (-31.8R 12mo);
-triple-confirm [M15,H1,H4] (-4.9R); divergence weight -1/3
-(-3.8R 12mo); candlestick weight +50% (-3.0R 12mo). The weight
-vector keeps proving calibrated as a system: divergence/candlestick/
-bollinger all look mis-weighted in-book and all replay worse when
-"fixed" (bollinger weight-0: +80.9R PF 1.29 DD 23.8 — its constant
-oppose vote is a load-bearing quality bar, like ROC).
 """
 
 from typing import Optional, List
 from config import PlatformConfig, Timeframe
-from app_config import apply_pair_overrides
 
 _CONFIRMATIONS = {
     Timeframe.M5:  [Timeframe.M15, Timeframe.H1],
@@ -184,38 +83,18 @@ def make_config(
     confirmation_tfs: Optional[List[Timeframe]] = None,
 ) -> PlatformConfig:
     config = PlatformConfig()
-    # Deployment overlay (config.json) BEFORE the validated per-TF
-    # deltas below, so e.g. the M5 hour-9 append lands on the JSON base.
-    apply_pair_overrides(config)
     config.primary_timeframe = primary_tf
     config.confirmation_timeframes = (
         confirmation_tfs or _CONFIRMATIONS.get(primary_tf, [Timeframe.M15, Timeframe.H4])
     )
     if primary_tf == Timeframe.M5:
-        # Block hour 9 UTC on M5 only; UNBLOCK hour 21 on M5 only
-        # (positive every year of the 5yr replay) — see docstring.
-        config.xau_strategy.blocked_hours_utc = [
-            h for h in config.xau_strategy.blocked_hours_utc if h != 21
-        ] + [9]
-        # Deliberately NOT mirrored into config.backtest.blocked_hours_utc:
-        # that copy is the generic non-XAU fallback carrying the JSON base
-        # only. Propagating a gold-validated per-TF delta there would leak
-        # XAU-calibrated hours to a future pair (pair-isolation rule).
-        # Every XAU consumer branches to the xau_strategy copy. Enforced
-        # by the Module 13 test "M5 factory appends hour 9 on top of a
-        # JSON base".
+        # Block hour 9 UTC on M5 only — see module docstring.
+        config.xau_strategy.blocked_hours_utc = (
+            config.xau_strategy.blocked_hours_utc + [9]
+        )
         # BUY-only chart patterns on M5 — see module docstring.
         config.xau_strategy.cp_enabled = True
         config.xau_strategy.cp_patterns = ["inv_hs", "double_bottom", "bull_flag"]
-        # XAU_PB pullback entries OFF — largest wrong-direction loss
-        # source at real cost on Oanda bars — see module docstring.
-        config.xau_strategy.m5_pullback_enabled = False
-        # Partial EMA-ribbon alignment scores NEUTRAL on M5 — its
-        # trades carried no edge — see module docstring.
-        config.technical.ema_partial_neutral = True
-        # Wider trailing distance on M5 only — see module docstring.
-        config.signals.trail_arm_r = 0.60
-        config.signals.trail_dist_r = 0.85
     if primary_tf == Timeframe.M15:
         # 1R minimum target on M15 only — see module docstring.
         config.signals.tp_floor_rr = 1.0
@@ -223,7 +102,4 @@ def make_config(
         config.xau_strategy.blocked_regimes = ["trending"]
         # Block SELL entries at 15:00/17:00 UTC on M15 only — see docstring.
         config.xau_strategy.sell_blocked_hours_utc = [15, 17]
-        # Later trailing-stop arm on M15 only — see module docstring.
-        config.signals.trail_arm_r = 0.70
-        config.signals.trail_dist_r = 0.75
     return config

providers/__init__.py

@@ -1,57 +0,0 @@
-"""
-ForexBot2 — Data-provider package
-==================================
-One class per provider, all subclassing `BaseDataProvider` (which owns
-caching, validation, boundary guarantees, and range pagination — the
-behavior the signal pipeline depends on).
-
-Adding a provider = one module implementing `_fetch_raw` + one registry
-entry + one config.json `data_providers` block. The live candle feed is
-selected by config.json `data_providers.active`; the optional
-`data_providers.streaming` scalar selects a different provider for the
-tick stream (level-breach monitor). Registration alone changes nothing.
-"""
-
-from typing import Dict, Optional
-
-from app_config import AppConfigError, get_app_config
-from config import PlatformConfig
-
-from providers.base import BaseDataProvider, DataIntegrityError, _is_forex_weekend
-from providers.twelve_data import TwelveDataProvider
-from providers.oanda import OandaDataError, OandaDataProvider
-from providers.tiingo import TiingoDataError, TiingoDataProvider
-
-#: config.json data_providers.active selects from here.
-PROVIDER_REGISTRY: Dict[str, type] = {
-    "twelve_data": TwelveDataProvider,
-    "oanda": OandaDataProvider,
-    "tiingo": TiingoDataProvider,
-}
-
-
-def make_data_provider(config: PlatformConfig,
-                       name: Optional[str] = None) -> BaseDataProvider:
-    """Build a provider by name (default: config.json data_providers.active)."""
-    if name is None:
-        name = get_app_config().active_provider
-    cls = PROVIDER_REGISTRY.get(name)
-    if cls is None:
-        raise AppConfigError(
-            f'Unknown data provider "{name}"; '
-            f"registered: {sorted(PROVIDER_REGISTRY)}")
-    return cls(config)
-
-
-__all__ = [
-    "BaseDataProvider",
-    "DataIntegrityError",
-    "OandaDataError",
-    "OandaDataProvider",
-    "TiingoDataError",
-    "TiingoDataProvider",
-    "TwelveDataProvider",
-    "PROVIDER_REGISTRY",
-    "make_data_provider",
-    "_is_forex_weekend",
-]

providers/base.py

@@ -1,532 +0,0 @@
-"""
-ForexBot2 — Provider base class
-================================
-`BaseDataProvider` owns everything that must behave identically no matter
-where the bars come from: in-memory caching with candle-duration TTL, the
-closed-bar boundary guarantee, freshness checks, validation/cleaning, the
-forming-candle drop, and paginated date-range fetching for the backtester.
-
-A concrete provider implements only:
-    _fetch_raw(symbol, timeframe, outputsize, start, end) -> pd.DataFrame
-returning a normalized frame (capitalized OHLCV columns, UTC tz-aware
-DatetimeIndex named "Datetime", oldest-first), plus optionally
-`get_spread` and the streaming-quote capability (`supports_streaming` +
-`iter_ticks`).
-
-The signal pipeline (engine + aggregator + backtester) consumes ONLY the
-public contract defined here, so swapping providers never touches signal
-code.
-"""
-
-import logging
-import threading
-import time
-from abc import ABC, abstractmethod
-from datetime import datetime, timedelta
-from typing import AsyncIterator, Callable, Dict, Optional, Sequence, Tuple
-from zoneinfo import ZoneInfo
-
-import pandas as pd
-
-from config import PAIR_INFO, PlatformConfig, Timeframe, XAUUSD
-
-logger = logging.getLogger(__name__)
-
-# The gold/FX market week is defined in New York time (see
-# _is_forex_weekend); a fixed UTC hour is wrong for half the year.
-_NY_TZ = ZoneInfo("America/New_York")
-
-
-def _is_forex_weekend(now=None) -> bool:
-    """Gold/FX market closed for the weekend.
-
-    The market week is anchored to NEW YORK time — Friday 17:00 ET →
-    Sunday 18:00 ET — not a fixed UTC hour. During US DST the feed
-    actually stops at 21:00 UTC (measured on 5 years of Oanda bars:
-    159/162 summer Fridays; winter close 22:00, winter reopen Sunday
-    23:00 UTC). The old hardcoded Fri 22:00→Sun 22:00 UTC rule produced
-    two false-"STALE DATA/feed lagging" hours every week (2026-08-04
-    audit I43, fixed per owner decision).
-
-    Kept as three deliberate copies (here, signal_aggregator.py,
-    gpu_backtest._is_xau_blocked_by_calendar) — a Module 21 test asserts
-    they agree hour-by-hour across DST transitions.
-    """
-    if now is None:
-        now = pd.Timestamp.now(tz="UTC")
-    ny = now.astimezone(_NY_TZ)
-    wd = ny.weekday()
-    if wd == 4:                      # Friday
-        return ny.hour >= 17
-    if wd == 5:                      # Saturday
-        return True
-    if wd == 6:                      # Sunday
-        return ny.hour < 18
-    return False
-
-
-class DataIntegrityError(Exception):
-    """Raised when data fails validation checks."""
-    pass
-
-
-class BaseDataProvider(ABC):
-    """Shared OHLCV provider behavior; subclasses supply the raw fetch."""
-
-    #: Registry name; overridden per provider.
-    name: str = "base"
-
-    #: Exception type this provider raises for fetch/validation failures.
-    #: Provider-specific errors subclass DataIntegrityError so consumers
-    #: can catch one type regardless of the active provider.
-    DATA_ERROR = DataIntegrityError
-
-    BARS_PER_DAY = {
-        Timeframe.M1: 1440, Timeframe.M5: 288, Timeframe.M15: 96,
-        Timeframe.M30: 48, Timeframe.H1: 24, Timeframe.H4: 6,
-        Timeframe.D1: 1, Timeframe.W1: 0.143,
-    }
-
-    CANDLE_SECONDS = {
-        Timeframe.M1: 60, Timeframe.M5: 300, Timeframe.M15: 900,
-        Timeframe.M30: 1800, Timeframe.H1: 3600, Timeframe.H4: 14400,
-        Timeframe.D1: 86400, Timeframe.W1: 604800,
-    }
-
-    #: Hard per-request bar cap shared by Twelve Data and Oanda.
-    MAX_BARS_PER_REQUEST = 5000
-
-    # Grace wait before the one retry when a fresh response at a candle
-    # boundary is missing the just-closed bar (feed latency).
-    FRESH_FETCH_RETRY_SECONDS = 3.0
-
-    def __init__(self, config: PlatformConfig):
-        self.config = config
-        self._cache: Dict[str, pd.DataFrame] = {}
-        self._cache_timestamps: Dict[str, float] = {}
-        # Requested fetch depth per key: a cached short frame must not
-        # satisfy a caller asking for a longer lookback.
-        self._cache_outputsize: Dict[str, int] = {}
-        # The bot shares one provider across a thread pool; cache eviction
-        # is check-then-act and needs locking.
-        self._cache_lock = threading.Lock()
-        # Single-flight: one API fetch per (symbol, TF) at a time; other
-        # threads wait and then read the cache.
-        self._inflight: Dict[str, threading.Event] = {}
-
-    # ── Abstract: the only provider-specific piece ─────────────────
-
-    @abstractmethod
-    def _fetch_raw(
-        self,
-        symbol: str,
-        timeframe: Timeframe,
-        outputsize: int,
-        start: Optional[datetime] = None,
-        end: Optional[datetime] = None,
-    ) -> pd.DataFrame:
-        """Fetch up to `outputsize` bars, optionally bounded by start/end
-        (tz-aware UTC datetimes). Must return a normalized frame:
-        capitalized OHLCV columns, UTC DatetimeIndex named "Datetime",
-        oldest-first. Raise self.DATA_ERROR on failure."""
-
-    # ── Public API ─────────────────────────────────────────────────
-
-    def fetch_ohlcv(
-        self,
-        symbol: str = XAUUSD,
-        timeframe: Timeframe = Timeframe.H1,
-        lookback_days: Optional[int] = None,
-    ) -> pd.DataFrame:
-        """Fetch OHLCV with in-memory caching for live mode."""
-        if lookback_days is None:
-            lookback_days = self.config.data_lookback_days
-
-        cache_key = f"{symbol}_{timeframe.value}"
-        outputsize = max(
-            300,
-            min(int(lookback_days * self.BARS_PER_DAY[timeframe]) + 20,
-                self.MAX_BARS_PER_REQUEST),
-        )
-
-        while True:
-            now = time.time()
-            cache_ttl = self.CANDLE_SECONDS.get(
-                timeframe, self.config.refresh_interval)
-            with self._cache_lock:
-                cached = self._cache.get(cache_key)
-                cached_at = self._cache_timestamps.get(cache_key)
-                cached_size = self._cache_outputsize.get(cache_key, 0)
-            if (
-                cached is not None
-                and cached_at is not None
-                and (now - cached_at) < cache_ttl
-                and not cached.empty
-                # A caller asking for MORE history than the cached fetch
-                # requested must not be served the short frame silently.
-                and cached_size >= outputsize
-            ):
-                if (self._is_data_fresh(cached, symbol, timeframe)
-                        and self._has_latest_closed_bar(cached, timeframe)):
-                    return cached
-                logger.warning(f"{symbol} {timeframe.value}: cached data is stale, refetching")
-                with self._cache_lock:
-                    self._cache.pop(cache_key, None)
-                    self._cache_timestamps.pop(cache_key, None)
-                    self._cache_outputsize.pop(cache_key, None)
-
-            # Single-flight: concurrent scans for the same (symbol, TF)
-            # must not each hit the API (the four TF loops fire together
-            # at aligned boundaries and Oanda has no client throttle).
-            # First caller claims the key and fetches; the rest wait,
-            # then re-check the cache.
-            with self._cache_lock:
-                inflight = self._inflight.get(cache_key)
-                if inflight is None:
-                    self._inflight[cache_key] = threading.Event()
-                    break
-            inflight.wait(timeout=90)
-
-        try:
-            df = self._fetch_raw(symbol, timeframe, outputsize)
-            df = self._drop_forming_candle(df, timeframe)
-            df = self._validate_and_clean(df, symbol, timeframe)
-
-            # The closed-bar guarantee must hold for FRESH responses too,
-            # not just cache hits: the scanner fires seconds after the
-            # boundary, and a lagging feed can answer without the
-            # just-closed bar — the scan would silently analyze (and
-            # session-check) the previous bar. Give the feed one short
-            # grace retry, then proceed loudly.
-            if not df.empty and not self._has_latest_closed_bar(df, timeframe):
-                time.sleep(self.FRESH_FETCH_RETRY_SECONDS)
-                try:
-                    retry = self._fetch_raw(symbol, timeframe, outputsize)
-                    retry = self._drop_forming_candle(retry, timeframe)
-                    retry = self._validate_and_clean(retry, symbol, timeframe)
-                    if retry is not None and not retry.empty:
-                        df = retry
-                except DataIntegrityError as e:
-                    logger.warning(
-                        f"{symbol} {timeframe.value}: boundary retry refetch "
-                        f"failed ({e}) — keeping original data")
-                if not self._has_latest_closed_bar(df, timeframe):
-                    logger.warning(
-                        f"{symbol} {timeframe.value}: feed lagging — just-closed bar "
-                        f"still missing after retry; this scan may run one bar behind"
-                    )
-
-            if not self._is_data_fresh(df, symbol, timeframe):
-                logger.warning(
-                    f"STALE DATA from API: {symbol} {timeframe.value} — "
-                    f"last bar {df.index[-1]} is too old. Signal quality may be degraded."
-                )
-
-            with self._cache_lock:
-                self._cache[cache_key] = df
-                self._cache_timestamps[cache_key] = now
-                self._cache_outputsize[cache_key] = outputsize
-            return df
-        finally:
-            # Release waiters even when the fetch raised — they re-check
-            # the cache and (on a miss) claim the key themselves.
-            with self._cache_lock:
-                ev = self._inflight.pop(cache_key, None)
-            if ev is not None:
-                ev.set()
-
-    def fetch_range(
-        self,
-        symbol: str,
-        timeframe: Timeframe,
-        start: datetime,
-        end: datetime,
-    ) -> pd.DataFrame:
-        """Fetch the RAW closed-bar series between start/end, paginating
-        under the per-request bar cap. Returns uncleaned bars (the
-        backtester resolves trades on the raw price path); callers wanting
-        the signal-side series run `validate_and_clean` on the result."""
-        bars_per_day = self.BARS_PER_DAY[timeframe]
-        days_per_chunk = max(1, int(4500 / bars_per_day))
-
-        chunks = []
-        cursor_end = end
-        empty_streak = 0
-        error_streak = 0
-        while cursor_end > start:
-            cursor_start = max(start, cursor_end - timedelta(days=days_per_chunk))
-            try:
-                chunk = self._fetch_raw(
-                    symbol, timeframe, self.MAX_BARS_PER_REQUEST,
-                    start=cursor_start, end=cursor_end,
-                )
-            except Exception as e:
-                # A raising chunk is a PROVIDER ERROR (auth, 429, 5xx,
-                # malformed payload) — not "no data". Three in a row means
-                # the rest of the history is unreachable: raise instead of
-                # returning a silently truncated frame (the backtester
-                # would treat it as complete).
-                logger.warning(
-                    f"Chunk {cursor_start.date()}–{cursor_end.date()} failed: {e}"
-                )
-                error_streak += 1
-                if error_streak >= 3:
-                    raise self.DATA_ERROR(
-                        f"{symbol} {timeframe.value}: {error_streak} "
-                        f"consecutive chunk failures walking back from "
-                        f"{cursor_end.date()} — refusing to return "
-                        f"truncated history (last error: {e})") from e
-                cursor_end = cursor_start - timedelta(seconds=1)
-                continue
-
-            if chunk is None or chunk.empty:
-                # Genuinely empty windows (weekends/holidays/asset gaps)
-                # are normal; several in a row = start of history.
-                empty_streak += 1
-                if empty_streak >= 3:
-                    break
-                cursor_end = cursor_start - timedelta(seconds=1)
-                continue
-
-            empty_streak = 0
-            error_streak = 0
-            chunks.append(chunk)
-            # Step back; dedup on concat.
-            new_cursor = chunk.index[0] - timedelta(seconds=1)
-            if new_cursor >= cursor_end:
-                # A provider ignoring start/end would pin the cursor and
-                # loop forever — refuse to spin.
-                logger.warning(
-                    f"{symbol} {timeframe.value}: pagination made no "
-                    f"progress at {cursor_end} — stopping")
-                break
-            cursor_end = new_cursor
-            if len(chunk) < 2:
-                break
-
-        if not chunks:
-            raise RuntimeError(f"No data fetched for {symbol} {timeframe.value}")
-
-        df = pd.concat(chunks).sort_index()
-        df = df[~df.index.duplicated(keep="last")]
-        # Fetches ending "now" include the still-forming candle — drop it
-        # so callers only ever see closed bars.
-        df = self._drop_forming_candle(df, timeframe)
-        start_ts = pd.Timestamp(start)
-        end_ts = pd.Timestamp(end)
-        df = df[(df.index >= start_ts) & (df.index <= end_ts)]
-        # Coverage sanity: a head materially later than the requested
-        # start means history was truncated (empty-streak bail or provider
-        # depth limit). Loud, not fatal — some assets genuinely start late.
-        if not df.empty and (df.index[0] - start_ts) > pd.Timedelta(days=7):
-            logger.error(
-                f"{symbol} {timeframe.value}: fetched history starts "
-                f"{df.index[0]} but {start_ts} was requested — "
-                f"~{(df.index[0] - start_ts).days} days missing from the "
-                f"head; treat downstream results accordingly")
-        return df
-
-    def is_data_fresh(
-        self, df: pd.DataFrame, symbol: str, timeframe: Timeframe
-    ) -> bool:
-        """Public freshness check (last bar within 3 candle durations)."""
-        return self._is_data_fresh(df, symbol, timeframe)
-
-    def validate_and_clean(
-        self, df: pd.DataFrame, symbol: str, timeframe: Timeframe
-    ) -> pd.DataFrame:
-        """Public entry to the shared cleaning pipeline."""
-        return self._validate_and_clean(df, symbol, timeframe)
-
-    def clear_cache(self) -> int:
-        """Wipe all cached data. Returns number of entries cleared."""
-        with self._cache_lock:
-            count = len(self._cache)
-            self._cache.clear()
-            self._cache_timestamps.clear()
-        logger.info(f"Cache cleared: {count} entries wiped")
-        return count
-
-    def get_pair_info(self, symbol: str = XAUUSD) -> Dict:
-        info = PAIR_INFO.get(symbol, PAIR_INFO[XAUUSD])
-        return {"symbol": symbol, **info}
-
-    def get_spread(self, symbol: str = XAUUSD) -> float:
-        """Latest observed round-trip spread; providers without live
-        bid/ask fall back to the pair's assumed default."""
-        return PAIR_INFO.get(symbol, PAIR_INFO[XAUUSD])["default_spread"]
-
-    # ── Streaming-quote capability (optional) ──────────────────────
-
-    #: True when the provider can stream live ticks via `iter_ticks`.
-    supports_streaming: bool = False
-
-    def iter_ticks(
-        self,
-        symbols: Sequence[str],
-        on_connect: Optional[Callable[[], None]] = None,
-    ) -> AsyncIterator[Tuple[str, float]]:
-        """Async generator yielding (canonical_symbol, price) ticks for one
-        connection's lifetime; the caller owns reconnect/backoff. Providers
-        that support it set `supports_streaming = True` and override."""
-        raise NotImplementedError(
-            f"{self.name} provider does not support streaming quotes")
-
-    # ── Shared internals ───────────────────────────────────────────
-
-    def _has_latest_closed_bar(self, df: pd.DataFrame, timeframe: Timeframe) -> bool:
-        """Cached data must include the most recent fully closed bar.
-
-        The TTL alone is wall-clock based: a snapshot fetched mid-candle is
-        still within TTL at the next candle boundary but is missing the bar
-        that just closed, so every boundary scan would analyze — and run the
-        bar-close session check against — the previous bar.
-        """
-        if df is None or df.empty:
-            return False
-        now = pd.Timestamp.now(tz="UTC")
-        if _is_forex_weekend(now):
-            return True
-        candle_seconds = self.CANDLE_SECONDS.get(timeframe, 3600)
-        dur = pd.Timedelta(seconds=candle_seconds)
-        expected_label = now.floor(dur) - dur
-        return df.index[-1] >= expected_label
-
-    def _drop_forming_candle(self, df: pd.DataFrame, timeframe: Timeframe) -> pd.DataFrame:
-        """Drop the still-forming last candle so analysis only sees closed bars.
-
-        Some feeds include the current partial candle; the scanner fires
-        seconds after the boundary, so without this guard every indicator
-        reads a seconds-old stub bar and signals repaint. Feeds that only
-        return complete candles pass through unchanged.
-        """
-        if df is None or df.empty:
-            return df
-        candle_seconds = self.CANDLE_SECONDS.get(timeframe, 3600)
-        now = pd.Timestamp.now(tz="UTC")
-        # Loop, not a single slice: a misbehaving feed can return MORE
-        # than one not-yet-closed bar, and a single iloc[:-1] would let
-        # the residual forming bar through cleaning (future-tolerance is
-        # +5 min) straight into the signal engine.
-        dropped = 0
-        while (not df.empty
-               and df.index[-1] + pd.Timedelta(seconds=candle_seconds) > now):
-            df = df.iloc[:-1]
-            dropped += 1
-        if dropped > 1:
-            logger.warning(
-                "%s: feed returned %d forming/future bars — all dropped",
-                timeframe.value, dropped)
-        return df
-
-    def _is_data_fresh(self, df: pd.DataFrame, symbol: str, timeframe: Timeframe) -> bool:
-        """Check if the last candle is within acceptable age for this timeframe."""
-        if df is None or df.empty:
-            return False
-
-        now = pd.Timestamp.now(tz="UTC")
-
-        if _is_forex_weekend(now):
-            return True
-
-        last_bar = df.index[-1]
-        candle_seconds = self.CANDLE_SECONDS.get(timeframe, 3600)
-        max_age_seconds = candle_seconds * 3
-        age_seconds = (now - last_bar).total_seconds()
-
-        if age_seconds > max_age_seconds:
-            logger.warning(
-                f"{symbol} {timeframe.value}: last bar {last_bar.strftime('%Y-%m-%d %H:%M')} "
-                f"is {age_seconds / 60:.0f}m old (limit: {max_age_seconds / 60:.0f}m)"
-            )
-            return False
-
-        return True
-
-    def _validate_and_clean(
-        self, df: pd.DataFrame, symbol: str, timeframe: Timeframe
-    ) -> pd.DataFrame:
-        if df is None or df.empty:
-            raise self.DATA_ERROR(f"Empty dataset for {symbol}")
-
-        original_len = len(df)
-
-        df = df.dropna(subset=["Open", "High", "Low", "Close"], how="all")
-        df = df.ffill(limit=3)
-        df = df.dropna(subset=["Close"])
-
-        df = df.sort_index()
-        df = df[~df.index.duplicated(keep="last")]
-
-        now = pd.Timestamp.utcnow()
-        future_mask = df.index > now + pd.Timedelta(minutes=5)
-        if future_mask.any():
-            logger.warning(f"{symbol}: Dropping {future_mask.sum()} future timestamps")
-            df = df[~future_mask]
-
-        # Non-positive prices are corrupt, never real gold quotes. Reject
-        # the row BEFORE the consistency repair below — repairing High/Low
-        # from a corrupt 0.0 Open/Close would propagate the zero into
-        # previously-valid fields (2026-08-04 audit, I45).
-        nonpositive = (df[["Open", "High", "Low", "Close"]] <= 0).any(axis=1)
-        if nonpositive.any():
-            logger.warning(
-                f"{symbol}: Dropping {nonpositive.sum()} bars with "
-                f"non-positive prices (corrupt feed data)")
-            df = df[~nonpositive]
-            if df.empty:
-                raise self.DATA_ERROR(
-                    f"{symbol}: all bars had non-positive prices")
-
-        invalid_ohlc = (
-            (df["High"] < df["Low"]) |
-            (df["High"] < df["Open"]) |
-            (df["High"] < df["Close"]) |
-            (df["Low"] > df["Open"]) |
-            (df["Low"] > df["Close"])
-        )
-        if invalid_ohlc.any():
-            logger.warning(f"{symbol}: Fixing {invalid_ohlc.sum()} OHLC-inconsistent bars")
-            for idx in df[invalid_ohlc].index:
-                vals = [df.loc[idx, "Open"], df.loc[idx, "Close"]]
-                df.loc[idx, "High"] = max(vals + [df.loc[idx, "High"]])
-                df.loc[idx, "Low"]  = min(vals + [df.loc[idx, "Low"]])
-
-        # inf returns (division by a zero/na Close) would NaN the std and
-        # silently disarm the whole 10σ pass — neutralize them first.
-        returns = df["Close"].pct_change()
-        returns = returns.replace([float("inf"), float("-inf")], float("nan"))
-        std_ret = returns.std()
-        if std_ret > 0:
-            dev = returns - returns.mean()
-            outlier_mask = dev.abs() > 10 * std_ret
-            # A spike bar makes the NEXT bar's return look extreme too (it is
-            # measured against the spike). If the previous bar is flagged and
-            # this bar reverses direction, it is the legitimate snap-back —
-            # keep it, or every spike would punch a two-bar hole in the data.
-            recovery = (
-                outlier_mask
-                & outlier_mask.shift(1, fill_value=False)
-                & (dev * dev.shift(1) < 0)
-            )
-            outlier_mask = outlier_mask & ~recovery
-            if outlier_mask.any():
-                logger.warning(f"{symbol}: Removing {outlier_mask.sum()} extreme outlier bars")
-                df = df[~outlier_mask]
-
-        min_required = max(
-            self.config.technical.ema_trend,
-            200,
-        ) + 10
-
-        if len(df) < min_required:
-            logger.warning(
-                f"{symbol} {timeframe.value}: Only {len(df)} bars "
-                f"(need {min_required}). Some indicators will be partial."
-            )
-
-        cleaned = original_len - len(df)
-        if cleaned > 0:
-            logger.info(f"{symbol}: Cleaned {cleaned}/{original_len} bars")
-
-        return df

providers/oanda.py

@@ -1,232 +0,0 @@
-"""
-ForexBot2 — Oanda v20 data provider
-====================================
-OHLCV candles from the Oanda v20 REST API. All shared behavior (cache,
-validation, boundary guarantees, range pagination) lives in
-`BaseDataProvider`; this module only maps symbols/granularities, parses
-Oanda's candle JSON, and exposes the real bid/ask spread from MBA candles.
-"""
-
-import logging
-import time
-from datetime import datetime
-from typing import Dict, Optional
-
-import pandas as pd
-import requests
-
-from app_config import oanda_api_token, oanda_base_url
-from config import PAIR_INFO, PlatformConfig, Timeframe, XAUUSD
-
-from providers.base import BaseDataProvider, DataIntegrityError
-
-logger = logging.getLogger(__name__)
-
-
-class OandaDataError(DataIntegrityError):
-    """Raised when an Oanda data fetch fails or returns invalid data."""
-
-
-class OandaDataProvider(BaseDataProvider):
-    """Oanda v20 REST candle provider."""
-
-    name = "oanda"
-    DATA_ERROR = OandaDataError
-    RETRY_DELAY_S = 2.0
-
-    OANDA_GRANULARITY = {
-        Timeframe.M1: "M1",
-        Timeframe.M5: "M5",
-        Timeframe.M15: "M15",
-        Timeframe.M30: "M30",
-        Timeframe.H1: "H1",
-        Timeframe.H4: "H4",
-        Timeframe.D1: "D",
-        Timeframe.W1: "W",
-    }
-
-    OANDA_INSTRUMENT = {
-        "XAU/USD": "XAU_USD",
-    }
-
-    def __init__(
-        self,
-        config: PlatformConfig,
-        api_token: Optional[str] = None,
-        base_url: Optional[str] = None,
-    ):
-        super().__init__(config)
-        # Credentials default to the config.json `oanda` block / env vars
-        # so the provider registry can build this class with config only.
-        if api_token is None:
-            api_token = oanda_api_token()
-        if base_url is None:
-            base_url = oanda_base_url()
-        self._base_url = base_url.rstrip("/")
-        self._session = requests.Session()
-        self._session.headers.update({
-            "Authorization": f"Bearer {api_token}",
-            "Accept-Datetime-Format": "RFC3339",
-            "Content-Type": "application/json",
-        })
-        self._spread_cache: Dict[str, float] = {}
-
-    # ── Public extras ──────────────────────────────────────────────
-
-    def get_spread(self, symbol: str = XAUUSD) -> float:
-        return self._spread_cache.get(
-            symbol, PAIR_INFO.get(symbol, PAIR_INFO[XAUUSD])["default_spread"])
-
-    # ── Raw fetch ──────────────────────────────────────────────────
-
-    def _fetch_raw(
-        self,
-        symbol: str,
-        timeframe: Timeframe,
-        outputsize: int,
-        start: Optional[datetime] = None,
-        end: Optional[datetime] = None,
-    ) -> pd.DataFrame:
-        if start is not None or end is not None:
-            return self._fetch_oanda_candles(
-                symbol, timeframe, outputsize, start, end)
-        return self._fetch_oanda_candles(symbol, timeframe, outputsize)
-
-    def _to_oanda_instrument(self, symbol: str) -> str:
-        if symbol in self.OANDA_INSTRUMENT:
-            return self.OANDA_INSTRUMENT[symbol]
-        return symbol.replace("/", "_")
-
-    def _fetch_oanda_candles(
-        self,
-        symbol: str,
-        timeframe: Timeframe,
-        count: int,
-        start: Optional[datetime] = None,
-        end: Optional[datetime] = None,
-    ) -> pd.DataFrame:
-        instrument = self._to_oanda_instrument(symbol)
-        granularity = self.OANDA_GRANULARITY.get(timeframe)
-        if granularity is None:
-            raise OandaDataError(
-                f"Unsupported timeframe {timeframe} for Oanda")
-
-        url = f"{self._base_url}/v3/instruments/{instrument}/candles"
-        params = {
-            "granularity": granularity,
-            "price": "MBA",
-            "dailyAlignment": 0,
-            "alignmentTimezone": "Etc/UTC",
-        }
-        # v20 rejects `count` combined with both `from` and `to`; a date
-        # window replaces it (used by fetch_range pagination).
-        if start is not None and end is not None:
-            params["from"] = pd.Timestamp(start).isoformat()
-            params["to"] = pd.Timestamp(end).isoformat()
-        else:
-            params["count"] = min(count, self.MAX_BARS_PER_REQUEST)
-
-        # One retry absorbs Oanda's sporadic spurious 401s (~15-20/day on
-        # the practice API — the same token succeeds on adjacent requests)
-        # plus transient 5xx/network faults. A real credential problem
-        # fails both attempts and raises exactly as before. Other 4xx
-        # (bad request, 429) are never retried.
-        last_err: Optional[OandaDataError] = None
-        for attempt in range(2):
-            if attempt:
-                time.sleep(self.RETRY_DELAY_S)
-            try:
-                resp = self._session.get(url, params=params, timeout=30)
-            except requests.RequestException as e:
-                last_err = OandaDataError(
-                    f"Oanda candle fetch failed for {instrument}: {e}")
-                last_err.__cause__ = e
-                continue
-            if resp.status_code == 401 or resp.status_code >= 500:
-                last_err = OandaDataError(
-                    f"Oanda candle fetch failed for {instrument}: "
-                    f"HTTP {resp.status_code}: {resp.text[:300]}")
-                if attempt == 0:
-                    logger.warning(
-                        f"Oanda fetch for {instrument} got HTTP "
-                        f"{resp.status_code}; retrying in "
-                        f"{self.RETRY_DELAY_S}s")
-                continue
-            break
-        else:
-            raise last_err
-
-        if resp.status_code != 200:
-            raise OandaDataError(
-                f"Oanda candle fetch failed for {instrument}: "
-                f"HTTP {resp.status_code}: {resp.text[:300]}")
-
-        try:
-            data = resp.json()
-        except ValueError as e:
-            raise OandaDataError(
-                f"Oanda candle response for {instrument} is not valid "
-                f"JSON: {e}") from e
-        candles = data.get("candles", [])
-        if not candles:
-            raise OandaDataError(
-                f"Empty candle response from Oanda for {instrument}")
-
-        rows = []
-        last_bid_close = None
-        last_ask_close = None
-        skipped_malformed = 0
-        for c in candles:
-            if not c.get("complete", False):
-                continue
-            mid = c.get("mid", {})
-            bid = c.get("bid", {})
-            ask = c.get("ask", {})
-            # A candle missing any OHLC field must be SKIPPED, never
-            # defaulted to 0.0: a zero price flows through cleaning as a
-            # "legit" bar and poisons ATR/Keltner/SL math — and a
-            # mid-frame zero Close makes pct_change() emit inf, which
-            # NaNs the std and disarms the 10σ outlier filter for the
-            # whole frame (2026-08-04 audit, I45).
-            if any(mid.get(k) is None for k in ("o", "h", "l", "c")):
-                skipped_malformed += 1
-                continue
-            rows.append({
-                "Datetime": pd.Timestamp(c["time"]).tz_convert("UTC"),
-                "Open": float(mid["o"]),
-                "High": float(mid["h"]),
-                "Low": float(mid["l"]),
-                "Close": float(mid["c"]),
-                "Volume": int(c.get("volume", 0)),
-            })
-            if bid and ask:
-                last_bid_close = float(bid.get("c", 0))
-                last_ask_close = float(ask.get("c", 0))
-        if skipped_malformed:
-            logger.warning(
-                "Oanda returned %d malformed candle(s) for %s (missing "
-                "OHLC fields) — skipped", skipped_malformed, instrument)
-
-        if not rows:
-            raise OandaDataError(
-                f"No complete candles from Oanda for {instrument}")
-
-        if last_bid_close and last_ask_close:
-            self._spread_cache[symbol] = round(
-                last_ask_close - last_bid_close, 4)
-
-        df = pd.DataFrame(rows)
-        df = df.set_index("Datetime")
-        df = df.sort_index()
-        if df.index.tz is None:
-            df.index = df.index.tz_localize("UTC")
-        df.index.name = "Datetime"
-
-        for col in ("Open", "High", "Low", "Close"):
-            df[col] = pd.to_numeric(df[col], errors="coerce")
-        df["Volume"] = pd.to_numeric(df["Volume"], errors="coerce").fillna(0).astype(float)
-
-        logger.debug(
-            "Fetched %d bars for %s @ %s from Oanda",
-            len(df), instrument, granularity)
-        return df

providers/tiingo.py

@@ -1,306 +0,0 @@
-"""
-ForexBot2 — Tiingo data provider
-=================================
-OHLCV candles from Tiingo's FX endpoint
-(`/tiingo/fx/{ticker}/prices`, resampled mid-price bars, no volume).
-All shared behavior (cache, validation, boundary guarantees, range
-pagination) lives in `BaseDataProvider`; this module only maps
-symbols/resample frequencies, sizes the request window, and parses
-Tiingo's JSON into the normalized frame.
-
-Every failure mode raises `TiingoDataError` (a `DataIntegrityError`)
-with a specific message — the scan loops log it and carry on; nothing
-fails silently. NOTE: candles and mid-price ticks only. Tiingo's free
-`/tiingo/fx/top` quote endpoint publishes indicative two-tier spreads
-and must not be used for spread display (removed 2026-07-23) —
-`get_spread` deliberately stays on the base-class default, and the
-WebSocket stream (`iter_ticks`) exposes the mid price only, never
-bid/ask or a derived spread.
-"""
-
-import json
-import logging
-from datetime import datetime
-from typing import AsyncIterator, Callable, Optional, Sequence, Tuple
-
-import pandas as pd
-import requests
-
-from app_config import provider_api_key
-from config import PlatformConfig, Timeframe
-
-from providers.base import BaseDataProvider, DataIntegrityError
-
-logger = logging.getLogger(__name__)
-
-
-class TiingoDataError(DataIntegrityError):
-    """Raised when a Tiingo data fetch fails or returns invalid data."""
-
-
-class TiingoDataProvider(BaseDataProvider):
-    """Tiingo FX REST candle provider."""
-
-    name = "tiingo"
-    DATA_ERROR = TiingoDataError
-
-    BASE_URL = "https://api.tiingo.com"
-
-    TIINGO_RESAMPLE = {
-        Timeframe.M1: "1min",
-        Timeframe.M5: "5min",
-        Timeframe.M15: "15min",
-        Timeframe.M30: "30min",
-        Timeframe.H1: "1hour",
-        Timeframe.H4: "4hour",
-        Timeframe.D1: "1day",
-        # W1 intentionally unmapped: the FX endpoint has no weekly
-        # resample — requesting it raises loudly instead of guessing.
-    }
-
-    TIINGO_TICKER = {
-        "XAU/USD": "xauusd",
-    }
-
-    # Window padding over the theoretical bar span: FX is closed ~2/7 of
-    # the week (Fri 22:00 → Sun 22:00 UTC), so a calendar window must be
-    # wider than `bars × candle_seconds` to yield the requested count.
-    WINDOW_PAD_FACTOR = 1.6
-
-    def __init__(self, config: PlatformConfig, api_key: Optional[str] = None):
-        super().__init__(config)
-        # Credentials default to env TIINGO_API_KEY → config.json
-        # data_providers.tiingo.api_key, so the provider registry can
-        # build this class with config only.
-        if api_key is None:
-            api_key = provider_api_key(provider="tiingo")
-        self._api_key = api_key or None
-        if self._api_key is None:
-            logger.warning(
-                "Tiingo provider built WITHOUT an API key — every fetch "
-                "will fail with a named error. Set the TIINGO_API_KEY env "
-                "var or data_providers.tiingo.api_key in config.json.")
-        self._session = requests.Session()
-        self._session.headers.update({
-            "Content-Type": "application/json",
-            # Header auth keeps the key out of URLs (and out of logs).
-            "Authorization": f"Token {api_key}" if api_key else "",
-        })
-
-    def _to_tiingo_ticker(self, symbol: str) -> str:
-        if symbol in self.TIINGO_TICKER:
-            return self.TIINGO_TICKER[symbol]
-        return symbol.replace("/", "").replace("=X", "").lower()
-
-    # ── Raw fetch ──────────────────────────────────────────────────
-
-    def _fetch_raw(
-        self,
-        symbol: str,
-        timeframe: Timeframe,
-        outputsize: int,
-        start: Optional[datetime] = None,
-        end: Optional[datetime] = None,
-    ) -> pd.DataFrame:
-        if self._api_key is None:
-            raise TiingoDataError(
-                "No Tiingo API key. Set the TIINGO_API_KEY env var or "
-                "data_providers.tiingo.api_key in config.json.")
-
-        resample = self.TIINGO_RESAMPLE.get(timeframe)
-        if resample is None:
-            raise TiingoDataError(
-                f"Unsupported timeframe {timeframe} for Tiingo "
-                f"(no FX resample frequency)")
-
-        ticker = self._to_tiingo_ticker(symbol)
-
-        # Tiingo takes date windows, not bar counts: derive the start
-        # from the requested count for live fetches.
-        if start is None:
-            span = self.CANDLE_SECONDS[timeframe] * (outputsize + 10)
-            start_ts = (pd.Timestamp.now(tz="UTC")
-                        - pd.Timedelta(seconds=span * self.WINDOW_PAD_FACTOR))
-        else:
-            start_ts = pd.Timestamp(start)
-
-        params = {
-            "resampleFreq": resample,
-            "startDate": start_ts.strftime("%Y-%m-%dT%H:%M:%S"),
-        }
-        if end is not None:
-            params["endDate"] = pd.Timestamp(end).strftime("%Y-%m-%dT%H:%M:%S")
-
-        url = f"{self.BASE_URL}/tiingo/fx/{ticker}/prices"
-        try:
-            resp = self._session.get(url, params=params, timeout=30)
-        except requests.RequestException as e:
-            raise TiingoDataError(
-                f"Tiingo candle fetch failed for {ticker}: {e}") from e
-
-        if resp.status_code == 429:
-            raise TiingoDataError(
-                f"Tiingo rate limit hit for {ticker} (HTTP 429) — "
-                f"plan quota exhausted; scan will retry next boundary")
-        if resp.status_code in (401, 403):
-            raise TiingoDataError(
-                f"Tiingo auth failed for {ticker} (HTTP {resp.status_code}) "
-                f"— check TIINGO_API_KEY / config.json api_key")
-        if resp.status_code != 200:
-            raise TiingoDataError(
-                f"Tiingo candle fetch failed for {ticker}: "
-                f"HTTP {resp.status_code}: {resp.text[:300]}")
-
-        try:
-            data = resp.json()
-        except ValueError as e:
-            raise TiingoDataError(
-                f"Tiingo candle response for {ticker} is not valid "
-                f"JSON: {e}") from e
-
-        if not isinstance(data, list) or not data:
-            raise TiingoDataError(
-                f"Empty candle response from Tiingo for {ticker} "
-                f"(window {params['startDate']} → "
-                f"{params.get('endDate', 'now')}; check ticker and plan "
-                f"history limits)")
-
-        rows = []
-        for item in data:
-            if "date" not in item:
-                raise TiingoDataError(
-                    f"Malformed Tiingo candle for {ticker}: missing "
-                    f"'date' in {str(item)[:120]}")
-            rows.append({
-                "Datetime": pd.Timestamp(item["date"]),
-                "Open": item.get("open"),
-                "High": item.get("high"),
-                "Low": item.get("low"),
-                "Close": item.get("close"),
-                # FX endpoint has no volume; 0.0 keeps the engine contract.
-                "Volume": 0.0,
-            })
-
-        df = pd.DataFrame(rows)
-        df = df.set_index("Datetime")
-        df = df.sort_index()
-        if df.index.tz is None:
-            df.index = df.index.tz_localize("UTC")
-        else:
-            df.index = df.index.tz_convert("UTC")
-        df.index.name = "Datetime"
-
-        for col in ("Open", "High", "Low", "Close"):
-            df[col] = pd.to_numeric(df[col], errors="coerce")
-        if df["Close"].isna().all():
-            raise TiingoDataError(
-                f"Tiingo response for {ticker} has no numeric closes — "
-                f"feed format changed?")
-
-        logger.debug(
-            "Fetched %d bars for %s @ %s from Tiingo",
-            len(df), ticker, resample)
-        return df
-
-    # ── Streaming quotes ───────────────────────────────────────────
-
-    supports_streaming = True
-
-    WS_URL = "wss://api.tiingo.com/fx"
-
-    @staticmethod
-    def _parse_ws_message(raw: str) -> Optional[Tuple[str, float]]:
-        """Parse one WS message; return (tiingo_ticker, mid_price) for
-        quote updates, None for heartbeats ("H"), subscription acks
-        ("I"), malformed JSON, or quotes without a usable price.
-
-        Server error messages ("E", e.g. bad token) RAISE TiingoDataError:
-        Tiingo authenticates inside the socket via the subscribe message,
-        so an auth failure arrives as a data message followed by a clean
-        close — swallowing it would make the caller's reconnect loop
-        hammer the endpoint with no backoff (backoff only grows on
-        exceptions).
-        """
-        try:
-            msg = json.loads(raw)
-        except json.JSONDecodeError:
-            logger.warning("Tiingo WS: malformed JSON, skipping")
-            return None
-
-        mtype = msg.get("messageType")
-        if mtype == "H":
-            return None
-        if mtype == "I":
-            logger.info("Tiingo WS: subscription acknowledged")
-            return None
-        if mtype == "E":
-            raise TiingoDataError(
-                f"Tiingo WS server error: {str(msg.get('response'))[:200]}")
-        if mtype != "A":
-            return None
-
-        data = msg.get("data")
-        if not isinstance(data, list) or len(data) < 8 or data[0] != "Q":
-            return None
-        # Live-verified field order (2026-08-03): ["Q", ticker, timestamp,
-        # bidSize, bid, mid, askSize, ask] — ask is index 7, NOT 6 as the
-        # docs' table suggests (index 6 is askSize).
-        ticker = data[1]
-        bid, mid, ask = data[4], data[5], data[7]
-        if mid is None and bid is not None and ask is not None:
-            mid = (bid + ask) / 2
-        try:
-            return ticker, float(mid)
-        except (TypeError, ValueError):
-            logger.warning("Tiingo WS: quote for %s has no usable price",
-                           ticker)
-            return None
-
-    async def iter_ticks(
-        self,
-        symbols: Sequence[str],
-        on_connect: Optional[Callable[[], None]] = None,
-    ) -> AsyncIterator[Tuple[str, float]]:
-        """One connection's lifetime of (canonical_symbol, mid) ticks.
-
-        The caller owns the reconnect/backoff loop around this generator.
-        Divergences from the Twelve Data reference implementation, both
-        deliberate: `on_connect` fires on the FIRST RECEIVED message (the
-        "I" ack in practice) rather than after the subscribe send, so a
-        rejected token cannot reset the caller's backoff; and there is no
-        client heartbeat task — Tiingo sends "H" frames itself and the
-        websockets library's built-in ping keepalive detects dead sockets.
-        """
-        import websockets
-
-        if self._api_key is None:
-            raise TiingoDataError(
-                "No Tiingo API key. Set the TIINGO_API_KEY env var or "
-                "data_providers.tiingo.api_key in config.json.")
-
-        ws_to_pair = {self._to_tiingo_ticker(s): s for s in symbols}
-        async with websockets.connect(self.WS_URL) as ws:
-            await ws.send(json.dumps({
-                "eventName": "subscribe",
-                "authorization": self._api_key,
-                "eventData": {
-                    "thresholdLevel": 5,
-                    "tickers": sorted(ws_to_pair),
-                },
-            }))
-            logger.info(
-                "Tiingo WS connected, subscribing to %d tickers",
-                len(ws_to_pair))
-            connected = False
-            async for raw in ws:
-                parsed = self._parse_ws_message(raw)
-                if not connected:
-                    connected = True
-                    if on_connect is not None:
-                        on_connect()
-                if parsed is None:
-                    continue
-                pair_symbol = ws_to_pair.get(parsed[0])
-                if pair_symbol is None:
-                    continue
-                yield pair_symbol, parsed[1]

providers/twelve_data.py

@@ -1,247 +0,0 @@
-"""
-ForexBot2 — Twelve Data provider
-=================================
-OHLCV via the official `twelvedata` SDK plus streaming quotes over the
-Twelve Data WebSocket. All shared behavior (cache, validation, boundary
-guarantees, range pagination) lives in `BaseDataProvider`.
-"""
-
-import asyncio
-import json
-import logging
-import threading
-import time
-from collections import deque
-from datetime import datetime
-from typing import AsyncIterator, Callable, Optional, Sequence, Tuple
-
-import pandas as pd
-from twelvedata import TDClient
-
-from app_config import provider_api_key
-from config import PAIR_INFO, PlatformConfig, Timeframe
-
-from providers.base import BaseDataProvider, DataIntegrityError
-
-logger = logging.getLogger(__name__)
-
-
-class TwelveDataProvider(BaseDataProvider):
-    """Twelve Data SDK client with credit throttling and WS quotes."""
-
-    name = "twelve_data"
-
-    TD_INTERVALS = {
-        Timeframe.M1:  "1min",
-        Timeframe.M5:  "5min",
-        Timeframe.M15: "15min",
-        Timeframe.M30: "30min",
-        Timeframe.H1:  "1h",
-        Timeframe.H4:  "4h",
-        Timeframe.D1:  "1day",
-        Timeframe.W1:  "1week",
-    }
-
-    RATE_LIMIT = 55
-    RATE_WINDOW = 60.0
-
-    WS_URL = "wss://ws.twelvedata.com/v1/quotes/price?apikey={api_key}"
-
-    def __init__(self, config: PlatformConfig):
-        super().__init__(config)
-        self._api_key = provider_api_key(provider="twelve_data") or None
-        self._throttle_lock = threading.Lock()
-        self._request_times: deque = deque()
-        self._td: Optional[TDClient] = (
-            TDClient(apikey=self._api_key) if self._api_key else None
-        )
-
-    @staticmethod
-    def _to_td_symbol(symbol: str) -> str:
-        if "/" in symbol:
-            return symbol
-        clean = symbol.replace("=X", "")
-        return f"{clean[:3]}/{clean[3:]}"
-
-    # ── Raw fetch ──────────────────────────────────────────────────
-
-    def _fetch_raw(
-        self,
-        symbol: str,
-        timeframe: Timeframe,
-        outputsize: int,
-        start: Optional[datetime] = None,
-        end: Optional[datetime] = None,
-    ) -> pd.DataFrame:
-        return self._fetch_twelve_data(
-            symbol, timeframe, outputsize,
-            start_date=start.strftime("%Y-%m-%d %H:%M:%S") if start else None,
-            end_date=end.strftime("%Y-%m-%d %H:%M:%S") if end else None,
-        )
-
-    def _throttle(self, credits: int = 1) -> None:
-        """Pro tier: 55 credits / minute. Block until budget allows."""
-        while True:
-            with self._throttle_lock:
-                now = time.time()
-                while self._request_times and now - self._request_times[0] >= self.RATE_WINDOW:
-                    self._request_times.popleft()
-
-                available = self.RATE_LIMIT - len(self._request_times)
-                if available >= credits:
-                    for _ in range(credits):
-                        self._request_times.append(time.time())
-                    return
-
-                used = len(self._request_times)
-                wait = (self._request_times[0] + self.RATE_WINDOW) - now + 0.2
-            logger.info(
-                f"Rate limit: {used}/{self.RATE_LIMIT} credits used "
-                f"— waiting {wait:.1f}s..."
-            )
-            time.sleep(wait)
-
-    def _fetch_twelve_data(
-        self,
-        symbol: str,
-        timeframe: Timeframe,
-        outputsize: int,
-        start_date: Optional[str] = None,
-        end_date: Optional[str] = None,
-    ) -> pd.DataFrame:
-        if self._td is None:
-            raise DataIntegrityError(
-                "No data-provider API key. Set the TWELVE_DATA_API_KEY env "
-                "var or data_providers.twelve_data.api_key in config.json."
-            )
-
-        td_symbol = self._to_td_symbol(symbol)
-        interval = self.TD_INTERVALS[timeframe]
-
-        self._throttle(credits=1)
-
-        kwargs = {
-            "symbol": td_symbol,
-            "interval": interval,
-            "outputsize": min(outputsize, self.MAX_BARS_PER_REQUEST),
-            "timezone": "UTC",
-        }
-        if start_date:
-            kwargs["start_date"] = start_date
-        if end_date:
-            kwargs["end_date"] = end_date
-
-        try:
-            df = self._td.time_series(**kwargs).as_pandas()
-        except Exception as e:
-            raise DataIntegrityError(
-                f"Twelve Data fetch failed for {td_symbol}: {e}"
-            ) from e
-
-        if df is None or df.empty:
-            raise DataIntegrityError(f"Empty response from Twelve Data for {td_symbol}")
-
-        # SDK returns lowercase columns; engine expects capitalised.
-        df = df.rename(columns={
-            "open": "Open", "high": "High", "low": "Low",
-            "close": "Close", "volume": "Volume",
-        })
-        for col in ("Open", "High", "Low", "Close", "Volume"):
-            if col in df.columns:
-                df[col] = pd.to_numeric(df[col], errors="coerce")
-        if "Volume" not in df.columns:
-            df["Volume"] = 0.0
-
-        # SDK returns newest-first; engine + indicators want oldest-first.
-        df = df.sort_index()
-        if df.index.tz is None:
-            df.index = df.index.tz_localize("UTC")
-        df.index.name = "Datetime"
-
-        logger.debug(f"Fetched {len(df)} bars for {td_symbol} @ {interval}")
-        return df
-
-    # ── Streaming quotes ───────────────────────────────────────────
-
-    supports_streaming = True
-
-    @staticmethod
-    def _parse_ws_price(raw: str) -> Optional[Tuple[str, float]]:
-        """Parse one WS message; return (ws_symbol, price) for price
-        events, None for everything else (logged where noteworthy)."""
-        try:
-            msg = json.loads(raw)
-        except json.JSONDecodeError:
-            logger.warning("WebSocket: malformed JSON, skipping")
-            return None
-
-        event = msg.get("event")
-        if event == "subscribe-status":
-            fails = msg.get("fails") or []
-            if fails:
-                logger.warning(f"WebSocket: failed to subscribe: {fails}")
-            return None
-        if event != "price":
-            return None
-
-        ws_symbol = msg.get("symbol")
-        try:
-            price = float(msg["price"])
-        except (KeyError, ValueError, TypeError) as e:
-            logger.warning(f"WebSocket: bad price data for {ws_symbol}: {e}")
-            return None
-        return ws_symbol, price
-
-    @staticmethod
-    async def _ws_heartbeat(ws) -> None:
-        """Twelve Data drops idle WS connections; heartbeat every 10s keeps
-        it alive through quiet periods (weekends, low-liquidity hours)."""
-        while True:
-            await asyncio.sleep(10)
-            await ws.send(json.dumps({"action": "heartbeat"}))
-
-    async def iter_ticks(
-        self,
-        symbols: Sequence[str],
-        on_connect: Optional[Callable[[], None]] = None,
-    ) -> AsyncIterator[Tuple[str, float]]:
-        """One connection's lifetime of (canonical_symbol, price) ticks.
-
-        The TD WS speaks display symbols ("XAU/USD"); ticks are mapped back
-        to the canonical pair symbols passed in. The caller owns the
-        reconnect/backoff loop around this generator.
-
-        `on_connect` fires on the FIRST RECEIVED frame, not after the
-        subscribe send (2026-08-04 audit H38b, mirroring the Tiingo
-        provider) — firing pre-receive let a rejected API key reset the
-        caller's reconnect backoff on every attempt, pinning it at the
-        minimum forever.
-        """
-        import websockets
-
-        ws_to_pair = {PAIR_INFO[s]["display"]: s for s in symbols}
-        url = self.WS_URL.format(api_key=self._api_key or "")
-        async with websockets.connect(url) as ws:
-            subscribe = {
-                "action": "subscribe",
-                "params": {"symbols": ",".join(ws_to_pair)},
-            }
-            await ws.send(json.dumps(subscribe))
-            logger.info(f"WebSocket connected, subscribed to {len(ws_to_pair)} symbols")
-            hb_task = asyncio.create_task(self._ws_heartbeat(ws))
-            connected = False
-            try:
-                async for raw in ws:
-                    if not connected:
-                        connected = True
-                        if on_connect is not None:
-                            on_connect()
-                    parsed = self._parse_ws_price(raw)
-                    if parsed is None:
-                        continue
-                    pair_symbol = ws_to_pair.get(parsed[0])
-                    if pair_symbol is None:
-                        continue
-                    yield pair_symbol, parsed[1]
-            finally:
-                hb_task.cancel()

signal_aggregator.py

@@ -10,15 +10,12 @@ import os
 from dataclasses import dataclass, field
 from datetime import datetime, timezone
 from typing import Dict, List, Optional
-from zoneinfo import ZoneInfo
 
 import pandas as pd
 
-_NY_TZ = ZoneInfo("America/New_York")
-
 from chart_patterns import detect_chart_pattern
 from config import PlatformConfig, SignalStrength, Timeframe
-from providers import BaseDataProvider, make_data_provider
+from data_provider import ForexDataProvider
 from engine import TechnicalEngine, TechnicalAnalysis
 from news_engine import NewsEngine, FundamentalContext
 
@@ -26,19 +23,15 @@ logger = logging.getLogger(__name__)
 
 
 def _is_forex_weekend(now_utc: datetime) -> bool:
-    """Gold/FX market closed for the weekend: Friday 17:00 → Sunday 18:00
-    NEW YORK time (= Fri 21:00 UTC in US summer, 22:00 in winter —
-    matches measured Oanda feed behavior; 2026-08-04 audit I43).
-    Deliberate copy of providers.base._is_forex_weekend — a Module 21
-    test asserts the copies agree across DST transitions."""
-    ny = now_utc.astimezone(_NY_TZ)
-    wd = ny.weekday()
-    if wd == 4:                      # Friday
-        return ny.hour >= 17
-    if wd == 5:                      # Saturday
+    """Forex market closed: Friday 22:00 UTC → Sunday 22:00 UTC (Sydney open)."""
+    wd = now_utc.weekday()
+    hour = now_utc.hour
+    if wd == 4 and hour >= 22:  # Friday after 22:00
+        return True
+    if wd == 5:  # All Saturday
+        return True
+    if wd == 6 and hour < 22:  # Sunday before 22:00
         return True
-    if wd == 6:                      # Sunday
-        return ny.hour < 18
     return False
 
 
@@ -57,12 +50,6 @@ class TradeSetup:
     pip_value: float
     # Entry archetype ("GENERIC", "XAU_PB") — selects trail params.
     strategy: str = "GENERIC"
-    # Trailing exit params stamped from the PRODUCING config, so per-TF
-    # factory overrides (e.g. M15 arm 0.70, M5 dist 0.85) reach every
-    # consumer. None = legacy/unstamped setup — formatters fall back to
-    # platform defaults; 0.0 means trailing is genuinely disabled.
-    trail_arm_r: Optional[float] = None
-    trail_dist_r: Optional[float] = None
 
 
 @dataclass
@@ -86,7 +73,6 @@ class SignalReport:
     event_description: str = ""
     news_sentiment: float = 0.0
     news_description: str = ""
-    news_summary: str = ""  # LLM market-context blurb (display-only)
 
     final_signal: SignalStrength = SignalStrength.NEUTRAL
     final_score: float = 0.0
@@ -118,7 +104,7 @@ class SignalAggregator:
 
     def __init__(self, config: PlatformConfig):
         self.config = config
-        self.data_provider = make_data_provider(config)
+        self.data_provider = ForexDataProvider(config)
         self.tech_engine = TechnicalEngine(config.technical)
         self._news_engine: Optional[NewsEngine] = None  # lazy
 
@@ -188,7 +174,7 @@ class SignalAggregator:
             try:
                 df = self.data_provider.fetch_ohlcv(symbol, tf)
                 dfs[tf] = df
-                if not self.data_provider.is_data_fresh(df, symbol, tf):
+                if not self.data_provider._is_data_fresh(df, symbol, tf):
                     stale_tfs.append(tf.value)
             except Exception as e:
                 logger.warning(f"Fetch failed for {symbol} {tf.value}: {e}")
@@ -222,7 +208,7 @@ class SignalAggregator:
                 bar_close = (
                     pd.Timestamp(report.timestamp)
                     + pd.Timedelta(
-                        seconds=BaseDataProvider.CANDLE_SECONDS[
+                        seconds=ForexDataProvider.CANDLE_SECONDS[
                             self.config.primary_timeframe
                         ]
                     )
@@ -242,12 +228,7 @@ class SignalAggregator:
                     report.action_recommendation = "NO TRADE — Outside tradeable session"
                     report.trade_setup = None
             except Exception:
-                # Fail-open by design (an unparseable timestamp shouldn't
-                # kill the scan) but never silently: this filter guards
-                # real money hours.
-                logger.exception(
-                    f"{symbol}: live session filter errored — filter SKIPPED this scan"
-                )
+                pass
 
         return report
 
@@ -314,44 +295,6 @@ class SignalAggregator:
             return None
         return float(adx.iloc[-1])
 
-    def _detect_big_candle(self, primary_df: pd.DataFrame) -> Optional[tuple]:
-        """Largest abnormal candle in the recent primary-TF window.
-
-        Returns (timestamp, ratio) of the biggest bar whose range is
-        >= big_candle_atr_multiple x the PRIOR bar's ATR within the last
-        big_candle_lookback_hours of primary-TF bars, else None. The
-        prior-bar ATR (shift 1) matters: a huge candle inflates the
-        Wilder ATR immediately, so comparing against the concurrent ATR
-        under-detects the very bars this gate exists to catch.
-
-        Pure pandas on the caller-sliced df, so the live path and the
-        backtest replay (which calls analyze_from_dataframes on prefix
-        slices) compute bit-identical answers. The tail slice keeps the
-        per-call cost bounded (~n + 42 rows); min_periods NaNs and short
-        histories degrade to "no trip".
-        """
-        cfg = self.config.xau_strategy
-        bar_secs = BaseDataProvider.CANDLE_SECONDS[self.config.primary_timeframe]
-        n = max(1, int(cfg.big_candle_lookback_hours * 3600 // bar_secs))
-        period = self.config.technical.atr_period
-        tail = primary_df.iloc[-(n + 3 * period):]
-        high, low, close = tail["High"], tail["Low"], tail["Close"]
-        tr = pd.concat(
-            [high - low,
-             (high - close.shift()).abs(),
-             (low - close.shift()).abs()],
-            axis=1,
-        ).max(axis=1)
-        atr = tr.ewm(alpha=1 / period, min_periods=period, adjust=False).mean()
-        ratio = (high - low) / atr.shift(1)
-        window = ratio.iloc[-n:]
-        if window.empty:
-            return None
-        max_ratio = window.max()
-        if pd.isna(max_ratio) or max_ratio < cfg.big_candle_atr_multiple:
-            return None
-        return window.idxmax(), float(max_ratio)
-
     def _apply_m5_filters(self, report: SignalReport) -> None:
         """Extra selectivity for M5 — noisy TF needs stronger confirmation."""
         if report.signal_quality not in ("A+", "A", "B", "C"):
@@ -595,7 +538,6 @@ class SignalAggregator:
         min_tp = (self.config.signals.round_trip_cost
                   * self.config.signals.min_tp_cost_multiple)
         if 1.5 * sl_dist < min_tp:
-            _reject("spread_gate")
             return
 
         report.trade_setup = TradeSetup(
@@ -611,10 +553,7 @@ class SignalAggregator:
             reward_pips=1.5 * sl_dist / pip_value,
             pip_value=pip_value,
             strategy="XAU_PB",
-            trail_arm_r=self.config.signals.pb_trail_arm_r,
-            trail_dist_r=self.config.signals.pb_trail_dist_r,
         )
-        _reject("accepted")   # funnel tick, mirrors the CP path
         if report.signal_quality not in ("A+", "A", "B"):
             report.signal_quality = "B"
         report.final_signal = (SignalStrength.BUY if is_buy
@@ -683,17 +622,13 @@ class SignalAggregator:
                     close_h = (
                         pd.Timestamp(report.timestamp)
                         + pd.Timedelta(
-                            seconds=BaseDataProvider.CANDLE_SECONDS[
+                            seconds=ForexDataProvider.CANDLE_SECONDS[
                                 self.config.primary_timeframe
                             ]
                         )
                     ).hour
                 except Exception:
                     close_h = None
-                    logger.warning(
-                        f"{report.symbol}: bar-close hour underivable — "
-                        f"SELL hour gate skipped"
-                    )
                 if close_h in cfg.sell_blocked_hours_utc:
                     _reject("sell_hour")
                     return
@@ -748,8 +683,6 @@ class SignalAggregator:
             reward_pips=height / pip_value,
             pip_value=pip_value,
             strategy="XAU_CP",
-            trail_arm_r=self.config.signals.trail_arm_r,
-            trail_dist_r=self.config.signals.trail_dist_r,
         )
         if report.signal_quality not in ("A+", "A", "B"):
             report.signal_quality = "B"
@@ -840,13 +773,11 @@ class SignalAggregator:
             sentiment, sent_desc = news.get_pair_sentiment(symbol, context)
             report.news_sentiment = sentiment
             report.news_description = sent_desc
-            report.news_summary = context.llm_summary
         else:
             report.event_risk = "clear"
             report.event_description = ""
             report.news_sentiment = 0.0
             report.news_description = ""
-            report.news_summary = ""
 
         # ── 5. Spread estimate from primary data (avoids extra API call) ─
         try:
@@ -883,10 +814,7 @@ class SignalAggregator:
                         f"DAY FILTER: {day_names.get(bar_wd, '?')} not in allowed days"
                     )
             except Exception:
-                # Fail-open by design, but never silently (see live wrapper).
-                logger.exception(
-                    f"{symbol}: session/day filter errored — filter SKIPPED"
-                )
+                pass
 
         # ── 6c. Transitioning regime quality boost ──────────────────
         if report.market_regime == "transitioning" and report.signal_quality in ("B", "C", "D"):
@@ -987,17 +915,13 @@ class SignalAggregator:
                     close_h = (
                         pd.Timestamp(report.timestamp)
                         + pd.Timedelta(
-                            seconds=BaseDataProvider.CANDLE_SECONDS[
+                            seconds=ForexDataProvider.CANDLE_SECONDS[
                                 self.config.primary_timeframe
                             ]
                         )
                     ).hour
                 except Exception:
                     close_h = None
-                    logger.warning(
-                        f"{report.symbol}: bar-close hour underivable — "
-                        f"SELL hour gate skipped"
-                    )
                 if close_h in self.config.xau_strategy.sell_blocked_hours_utc:
                     report.warnings.append(
                         f"SELL FILTER: {close_h:02}:00 UTC entry blocked for shorts"
@@ -1005,30 +929,6 @@ class SignalAggregator:
                     report.signal_quality = "D"
                     report.trade_setup = None
 
-        # ── 6h. Big-candle entry gate (opt-in via per-TF factories) ──
-        # An abnormally large candle in the recent window = institutional
-        # burst; price may reverse violently, so NO entry of any kind may
-        # fire. Detection runs for ANY grade: the warning is the
-        # suppression channel for the XAU_PB path (which accepts D) and
-        # the XAU_CP path (which accepts even F). The D-downgrade itself
-        # touches actionable grades only — F stays F (same convention as
-        # 6e).
-        if (self.config.xau_strategy.enabled
-                and report.display_name == "XAU/USD"
-                and self.config.xau_strategy.big_candle_atr_multiple > 0):
-            big_candle = self._detect_big_candle(primary_df)
-            if big_candle is not None:
-                bc_ts, bc_ratio = big_candle
-                report.warnings.append(
-                    f"BIG CANDLE FILTER: {bc_ratio:.1f}x ATR bar at {bc_ts} "
-                    f"within "
-                    f"{self.config.xau_strategy.big_candle_lookback_hours:g}h "
-                    f"— no entries"
-                )
-                if report.signal_quality in ("A+", "A", "B", "C"):
-                    report.signal_quality = "D"
-                    report.trade_setup = None
-
         # ── 6i. M5 noise gate (gold-tuned thresholds — XAU only) ──
         if (self.config.primary_timeframe == Timeframe.M5
                 and report.display_name == "XAU/USD"):
@@ -1057,9 +957,7 @@ class SignalAggregator:
                 and report.display_name == "XAU/USD"
                 and self.config.xau_strategy.m5_pullback_enabled
                 and report.trade_setup is None
-                and report.signal_quality != "F"
-                and not any(w.startswith("BIG CANDLE FILTER")
-                            for w in report.warnings)):
+                and report.signal_quality != "F"):
             self._try_m5_pullback(report, primary_df, pip_value)
 
         # ── 10. Chart-pattern entry (XAU_CP) ────────────────────────
@@ -1076,8 +974,7 @@ class SignalAggregator:
                 and self.config.xau_strategy.cp_enabled
                 and report.trade_setup is None
                 and not any(w.startswith(("SESSION FILTER", "DAY FILTER",
-                                          "WEEKEND FILTER",
-                                          "BIG CANDLE FILTER"))
+                                          "WEEKEND FILTER"))
                             for w in report.warnings)):
             self._try_chart_pattern(report, primary_df, pip_value)
 
@@ -1151,8 +1048,6 @@ class SignalAggregator:
             risk_pips=round(sl_distance / pip_value, 1),
             reward_pips=round(tp_distance / pip_value, 1),
             pip_value=pip_value,
-            trail_arm_r=self.config.signals.trail_arm_r,
-            trail_dist_r=self.config.signals.trail_dist_r,
         )
 
     def _check_mtf_agreement(self, report: SignalReport) -> bool:
@@ -1184,7 +1079,7 @@ class SignalAggregator:
                 mtf_avg = sum(all_scores) / len(all_scores)
                 score = score * 0.6 + mtf_avg * 0.4
 
-        if self.config.news.sentiment_affects_signals and abs(report.news_sentiment) > 0.2:
+        if abs(report.news_sentiment) > 0.2:
             if (report.news_sentiment > 0 and score > 0) or \
                (report.news_sentiment < 0 and score < 0):
                 confidence += 5
@@ -1234,8 +1129,7 @@ class SignalAggregator:
             if (report.news_sentiment > 0 and score < -10) or \
                (report.news_sentiment < 0 and score > 10):
                 report.warnings.append("News sentiment CONFLICTS with technical signal")
-                if self.config.news.sentiment_affects_signals:
-                    penalties.append(("news_conflict", 10))
+                penalties.append(("news_conflict", 10))
 
         if report.event_risk == "caution":
             report.warnings.append(f"Upcoming event: {report.event_description}")
@@ -1260,17 +1154,7 @@ class SignalAggregator:
             ema200 = primary.key_levels.get("ema_trend")
             if ema200 and ema200 > 0 and report.current_price > 0:
                 ema_dist_pct = abs(report.current_price - ema200) / ema200 * 100.0
-                # FB2_EMA_LO: research override for the EMA200-proximity
-                # penalty band; the tuned default is 0.5. An unparseable
-                # value must degrade to the default, never crash the scan
-                # (2026-08-04 audit; hardened per owner delegation).
-                try:
-                    _lo = float(os.environ.get("FB2_EMA_LO", "0.5"))
-                except (TypeError, ValueError):
-                    logger.warning(
-                        "FB2_EMA_LO=%r is not a number — using tuned "
-                        "default 0.5", os.environ.get("FB2_EMA_LO"))
-                    _lo = 0.5
+                _lo = float(os.environ.get("FB2_EMA_LO", "0.5"))
                 if ema_dist_pct < _lo:
                     report.warnings.append(f"NEAR EMA200 ({ema_dist_pct:.2f}%) — chop risk")
                     penalties.append(("ema200_near", 5))
@@ -1280,11 +1164,6 @@ class SignalAggregator:
 
             atr = primary.key_levels.get("atr", 0)
             if atr > 0 and score != 0:
-                # Distance is measured from the FARTHEST of the 3 tracked
-                # swings (sorted()[0] = deepest low / highest high), so the
-                # penalty fires earlier than a nearest-swing rule would.
-                # Calibrated as-is in every validated book — changing the
-                # anchor is a signal-logic change requiring re-validation.
                 if score > 0:
                     swing_lows = sorted(
                         v for k, v in primary.key_levels.items() if k.startswith("swing_low_")
@@ -1565,8 +1444,6 @@ class SignalAggregator:
             risk_pips=risk_pips,
             reward_pips=reward_pips,
             pip_value=pip_value,
-            trail_arm_r=self.config.signals.trail_arm_r,
-            trail_dist_r=self.config.signals.trail_dist_r,
         )
 
     def _collect_reasons(self, report: SignalReport):

static/news_summary.php

@@ -1,245 +0,0 @@
-<?php
-/**
- * On-demand gold-impact AI summary for one published headline.
- *
- * The ONLY dynamic piece of the otherwise-static dashboard. Serves
- * GET ?id=<sha1-of-headline> and returns {"summary": ..., "generated_at": ...}.
- *
- * Abuse model (the page is public):
- *   - WHITELIST: the id must match a headline in the local news.json —
- *     arbitrary text can never reach the LLM through this endpoint.
- *   - CACHE FOREVER: headlines are immutable, so each id costs at most
- *     one DeepSeek call, ever (ai/<id>.json). Cache is checked BEFORE
- *     the whitelist on purpose: summaries stay reachable after the
- *     headline ages out of the 24 h feed window.
- *   - DAILY CAP: file-based UTC-day counter, FAIL CLOSED on counter I/O
- *     errors (a perms mistake must not mean unmetered spend).
- *
- * DeepSeek doctrine (mirrors llm_client.py — deliberate local copy, the
- * dashboard never imports bot code and this file never reads the repo's
- * config.json):
- *   - STATIC byte-identical system prompt (prefix-cache pricing).
- *   - "thinking": disabled is MANDATORY for V4-Flash (else it burns the
- *     timeout reasoning), temperature 0, json response_format.
- *   - One attempt, no retries; every failure returns a JSON error the
- *     page renders as a graceful fallback.
- *
- * Key file lives OUTSIDE the docroot: /etc/gold-dashboard/deepseek.key
- * (root:apache 640). Override paths/caps via Apache SetEnv:
- * DEEPSEEK_KEY_FILE, AI_SUMMARY_DAILY_CAP.
- */
-
-const NEWS_FILE = __DIR__ . '/news.json';
-const CACHE_DIR = __DIR__ . '/ai';   // apache-owned; created at deploy time
-const API_URL   = 'https://api.deepseek.com/chat/completions';
-const MODEL     = 'deepseek-v4-flash';
-const TIMEOUT   = 12;
-const MAX_TOKENS = 200;
-const SUMMARY_MAX_CHARS = 400;
-
-// .ht_ prefix: Apache's default <Files ".ht*"> rule denies these even
-// though they sit in the public docroot.
-const CAP_FILE = CACHE_DIR . '/.ht_cap.json';
-
-const SYSTEM_PROMPT = <<<'PROMPT'
-You are a forex/gold news analyst. You receive one news item as JSON:
-{"headline", "description", "score", "xau_score", "keywords", "currencies"}.
-"xau_score" is the platform's automated gold-effect score for this item,
--1.0 (gold down) to 1.0 (gold up); "score" is overall market sentiment.
-Both are automated and can be wrong — judge the news yourself.
-In 1-2 plain-English sentences (max 45 words) explain what this news
-means for the GOLD (XAU/USD) price and WHY it is bullish, bearish, or
-neutral. If your reading clearly contradicts xau_score, add "(the
-automated score disagrees)". Never write the words "xau_score" or
-"score" — a general reader sees this text.
-Gold doctrine: hawkish Fed / rate hikes / rising real yields / strong
-dollar are gold-bearish; easing / rate cuts / safe-haven flows /
-geopolitical stress / weak dollar are gold-bullish.
-Reply with ONLY JSON: {"summary": "..."}
-PROMPT;
-
-function respond(int $status, array $obj, string $cache = 'no-store'): void
-{
-    http_response_code($status);
-    header('Content-Type: application/json; charset=utf-8');
-    header('Cache-Control: ' . $cache);
-    echo json_encode($obj, JSON_UNESCAPED_SLASHES | JSON_UNESCAPED_UNICODE);
-    exit;
-}
-
-function atomic_write(string $path, string $data): bool
-{
-    $tmp = $path . '.tmp.' . getmypid();
-    if (@file_put_contents($tmp, $data) === false) {
-        return false;
-    }
-    if (!@rename($tmp, $path)) {
-        @unlink($tmp);
-        return false;
-    }
-    return true;
-}
-
-// ── 1. id validation ────────────────────────────────────────────────
-$id = $_GET['id'] ?? '';
-if (!preg_match('/^[a-f0-9]{40}$/', $id)) {
-    respond(400, ['error' => 'bad_id']);
-}
-
-// ── 2. cache first (immutable headlines → summaries never change) ───
-$cachePath = CACHE_DIR . '/' . $id . '.json';
-$cached = @file_get_contents($cachePath);
-if ($cached !== false && $cached !== '') {
-    http_response_code(200);
-    header('Content-Type: application/json; charset=utf-8');
-    header('Cache-Control: max-age=3600');
-    echo $cached;
-    exit;
-}
-
-// ── 3. whitelist against the published feed ─────────────────────────
-$news = json_decode((string)@file_get_contents(NEWS_FILE), true);
-if (!is_array($news)) {
-    respond(503, ['error' => 'no_feed']);
-}
-$hit = null;
-foreach ($news['headlines'] ?? [] as $h) {
-    // Fallback sha1(): a publisher that predates the id field still
-    // serves a valid whitelist during the restart window.
-    $hid = $h['id'] ?? sha1($h['headline'] ?? '');
-    if ($hid === $id) {
-        $hit = $h;
-        break;
-    }
-}
-if ($hit === null) {
-    respond(404, ['error'   => 'unknown_id',
-                  'message' => 'Headline is no longer in the feed.']);
-}
-
-// ── 4. daily cap (fail closed) ──────────────────────────────────────
-$dailyCap = (int)(getenv('AI_SUMMARY_DAILY_CAP') ?: 150);
-$capFh = @fopen(CAP_FILE, 'c+');
-if ($capFh === false || !flock($capFh, LOCK_EX)) {
-    respond(503, ['error' => 'cap_io']);
-}
-$today = gmdate('Y-m-d');
-$cap = json_decode((string)stream_get_contents($capFh), true);
-if (!is_array($cap) || ($cap['day'] ?? '') !== $today) {
-    $cap = ['day' => $today, 'count' => 0];
-}
-if ($cap['count'] >= $dailyCap) {
-    flock($capFh, LOCK_UN);
-    fclose($capFh);
-    respond(429, ['error' => 'cap_exceeded']);
-}
-// Increment BEFORE the LLM call — attempts count against the cap
-// (same semantics as llm_client._under_cap).
-$cap['count']++;
-rewind($capFh);
-if (ftruncate($capFh, 0) === false
-        || fwrite($capFh, json_encode($cap)) === false) {
-    flock($capFh, LOCK_UN);
-    fclose($capFh);
-    respond(503, ['error' => 'cap_io']);
-}
-fflush($capFh);
-flock($capFh, LOCK_UN);
-fclose($capFh);
-
-// ── 5. API key ──────────────────────────────────────────────────────
-$keyFile = getenv('DEEPSEEK_KEY_FILE') ?: '/etc/gold-dashboard/deepseek.key';
-$apiKey = trim((string)@file_get_contents($keyFile));
-if ($apiKey === '') {
-    respond(503, ['error' => 'no_key']);
-}
-
-// ── 6. per-id stampede lock, then re-check the cache ────────────────
-$lockFh = @fopen(CACHE_DIR . '/.ht_lock_' . $id, 'c');
-if ($lockFh !== false) {
-    flock($lockFh, LOCK_EX);   // blocks while a concurrent winner generates
-    $cached = @file_get_contents($cachePath);
-    if ($cached !== false && $cached !== '') {
-        flock($lockFh, LOCK_UN);
-        fclose($lockFh);
-        http_response_code(200);
-        header('Content-Type: application/json; charset=utf-8');
-        header('Cache-Control: max-age=3600');
-        echo $cached;
-        exit;
-    }
-}
-
-// ── 7. DeepSeek call (exact chat_json body shape) ───────────────────
-$userPayload = json_encode([
-    'headline'    => $hit['headline'] ?? '',
-    'description' => $hit['description'] ?? null,
-    'score'       => $hit['score'] ?? null,
-    'xau_score'   => $hit['score_by_currency']['XAU'] ?? null,
-    'keywords'    => $hit['keywords'] ?? [],
-    'currencies'  => $hit['currencies'] ?? [],
-], JSON_UNESCAPED_UNICODE);
-
-$body = json_encode([
-    'model'    => MODEL,
-    'messages' => [
-        ['role' => 'system', 'content' => SYSTEM_PROMPT],
-        ['role' => 'user',   'content' => $userPayload],
-    ],
-    'response_format' => ['type' => 'json_object'],
-    'temperature'     => 0,
-    'thinking'        => ['type' => 'disabled'],
-    'max_tokens'      => MAX_TOKENS,
-]);
-
-$ch = curl_init(API_URL);
-curl_setopt_array($ch, [
-    CURLOPT_POST           => true,
-    CURLOPT_POSTFIELDS     => $body,
-    CURLOPT_HTTPHEADER     => ['Content-Type: application/json',
-                               'Authorization: Bearer ' . $apiKey],
-    CURLOPT_RETURNTRANSFER => true,
-    CURLOPT_TIMEOUT        => TIMEOUT,
-]);
-$raw = curl_exec($ch);
-$httpCode = curl_getinfo($ch, CURLINFO_RESPONSE_CODE);
-curl_close($ch);
-
-// ── 8. parse (any failure → 502, never cached) ──────────────────────
-$summary = null;
-if ($raw !== false && $httpCode === 200) {
-    $resp = json_decode($raw, true);
-    $content = $resp['choices'][0]['message']['content'] ?? '';
-    $parsed = json_decode((string)$content, true);
-    if (is_array($parsed) && is_string($parsed['summary'] ?? null)
-            && trim($parsed['summary']) !== '') {
-        $summary = trim($parsed['summary']);
-        if (mb_strlen($summary) > SUMMARY_MAX_CHARS) {
-            $summary = mb_substr($summary, 0, SUMMARY_MAX_CHARS) . '…';
-        }
-    }
-}
-if ($lockFh !== false && $summary === null) {
-    flock($lockFh, LOCK_UN);
-    fclose($lockFh);
-}
-if ($summary === null) {
-    respond(502, ['error' => 'llm_failed']);
-}
-
-// ── 9. cache + respond ──────────────────────────────────────────────
-$out = json_encode(['summary' => $summary, 'generated_at' => gmdate('c')],
-                   JSON_UNESCAPED_SLASHES | JSON_UNESCAPED_UNICODE);
-if (!atomic_write($cachePath, $out)) {
-    // Degrade to per-click generation rather than a broken feature —
-    // the daily cap still bounds spend.
-    error_log('news_summary.php: cache write failed for ' . $cachePath);
-}
-if ($lockFh !== false) {
-    flock($lockFh, LOCK_UN);
-    fclose($lockFh);
-}
-http_response_code(200);
-header('Content-Type: application/json; charset=utf-8');
-header('Cache-Control: max-age=3600');
-echo $out;

test_ws_dual.py

@@ -0,0 +1,108 @@
+#!/usr/bin/env python3
+"""Test whether Twelve Data allows 2 simultaneous WebSocket connections for XAU/USD.
+
+Uses the twelvedata SDK which handles the WS protocol correctly.
+Runs connection 1, then adds connection 2 after 10s, monitors for 30s total.
+"""
+
+import threading
+import time
+
+from twelvedata import TDClient
+
+API_KEY = "f563b22ffdfd4924955decbe1a8c8701"
+SYMBOL = "XAU/USD"
+DURATION = 30
+
+results = {
+    "WS-1": {"ticks": 0, "errors": [], "alive": False},
+    "WS-2": {"ticks": 0, "errors": [], "alive": False},
+}
+
+
+def make_handlers(label):
+    def on_event(event):
+        if event.get("event") == "price":
+            results[label]["ticks"] += 1
+            price = event.get("price", "?")
+            print(f"[{label}] tick #{results[label]['ticks']:3d}  price={price}")
+
+    def on_error(error):
+        results[label]["errors"].append(str(error))
+        print(f"[{label}] ERROR: {error}")
+
+    def on_close(status):
+        results[label]["alive"] = False
+        print(f"[{label}] CLOSED (status={status})")
+
+    def on_connect():
+        results[label]["alive"] = True
+        print(f"[{label}] CONNECTED — subscribed to {SYMBOL}")
+
+    return on_event, on_error, on_close, on_connect
+
+
+def run_ws(label):
+    td = TDClient(apikey=API_KEY)
+    on_event, on_error, on_close, on_connect = make_handlers(label)
+    ws = td.websocket(
+        symbols=SYMBOL,
+        on_event=on_event,
+        on_error=on_error,
+        on_close=on_close,
+        on_connect=on_connect,
+    )
+    ws.connect()
+    return ws
+
+
+def main():
+    print(f"Testing dual WebSocket for {SYMBOL} ({DURATION}s)\n")
+
+    print("Starting WS-1...")
+    ws1 = run_ws("WS-1")
+
+    time.sleep(10)
+    print(f"\nWS-1 ticks so far: {results['WS-1']['ticks']}")
+    print("Starting WS-2...\n")
+    ws2 = run_ws("WS-2")
+
+    remaining = DURATION - 10
+    time.sleep(remaining)
+
+    print("\nStopping connections...")
+    try:
+        ws1.disconnect()
+    except Exception:
+        pass
+    try:
+        ws2.disconnect()
+    except Exception:
+        pass
+
+    time.sleep(1)
+
+    print("\n" + "=" * 50)
+    print("RESULTS")
+    print("=" * 50)
+    for label, info in sorted(results.items()):
+        status = "alive" if info["alive"] else "dead"
+        errs = info["errors"] or ["none"]
+        print(f"  {label}: {status}  ticks={info['ticks']}  errors={errs[0]}")
+
+    ws1_ok = results["WS-1"]["ticks"] > 0 and not results["WS-1"]["errors"]
+    ws2_ok = results["WS-2"]["ticks"] > 0 and not results["WS-2"]["errors"]
+    ws1_died = results["WS-1"]["alive"] is False and results["WS-1"]["ticks"] > 0
+
+    if ws1_ok and ws2_ok:
+        print("\n✓ Both connections worked — dual WS is supported")
+    elif ws2_ok and ws1_died:
+        print("\n✗ WS-2 replaced WS-1 — only 1 connection allowed at a time")
+    elif ws1_ok and not ws2_ok:
+        print("\n✗ WS-2 was rejected — only 1 connection allowed")
+    else:
+        print("\n? Inconclusive — check errors above")
+
+
+if __name__ == "__main__":
+    main()