audit: fix 9 defects from the 2026-09-04 hidden-bug sweep

None of these touch the signal path. The 12-month M15 book
(2024-03-31..2025-03-30, $0.65, cached Oanda bars) is byte-identical
before and after (md5 c59d3e7e685c3c880b614640adfea74b, 375 trades),
and signal/indicator logging to logs/signals.db is unchanged — verified
end to end: per-TF indicators for all 3 TFs, 14 candles each, MTF and
reasons all still persisted.

agent_panel: the >=2-take quorum guards the CHAIR's synthesizer spend,
  but ran BEFORE the macro branch — so the shipped default
  (verdict_source="macro", which never calls the synthesizer) silently
  dropped a good macro verdict whenever the other three personas were
  slow. Quorum moved below the macro path.
analyze_signal_log: _forward_return clamped to the last available bar
  when the horizon had not elapsed, reporting a near-zero "forward move"
  for every row logged within `hours` of the end of the data — on every
  run, blended into the hit-rate/mean-move aggregate. Now returns None.
analyze_signal_log: persona_scorecard iterated raw rows through an index
  that maps EVERY member row of an event, so a re-logged scan appended
  the same event R twice. One vote per event, matching
  news_outcome_buckets. Measured inflation on the live DB: 1.03x.
signal_log: dedup required a non-negative logged_at gap, but logged_at is
  stamped before the row wins the write lock (busy_timeout 5 s), so two
  processes logging one setup at a boundary can invert timestamp order
  against id order and seed a second event. Now abs(gap); same in
  backfill_seeds. Zero occurrences in the current 366 setup rows.
advisory_outcome: the catch-all turned real evaluator faults into the
  same UNRESOLVABLE used for legitimate weekend gaps, with nothing in the
  log. Still fail-safe, now logged.
llm_client: only the "cap" warn-once flag was ever reset, so in a daemon
  the second and every later network/http/parse outage logged at DEBUG
  only. Cleared on UTC-day roll and re-armed by a recovered call.
dashboard.js: renderCards() redraws the selector for every timeframe but
  was gated on the SELECTED one changing, leaving the other buttons stale
  (up to 4 h on H4) while renderHero's rail already showed the new value.
news_summary.php: atomic_write treated a short write (disk full) as
  success, publishing a truncated body the cache-first branch then served
  forever; and the daily cap was charged before the no-key check and
  before the stampede lock re-checked the cache, so a click burst could
  exhaust the 150/day budget with zero LLM calls.
migrate_signal_log_news: logged_at is ISO with 'T' while SQLite's
  datetime() uses a space, so string-comparing them put the whole cutoff
  calendar day on the wrong side and it never pruned. NOTE: this now
  deletes one day more than before — it only runs when invoked by hand.

Tests 414 -> 422 (new Module 28, one per fix). The existing
"<2 valid takes" test was retargeted at the chair — the case it actually
protects — rather than deleted, with new coverage for macro-alone.

Not fixed, recorded for the owner:
- Gate 6c's boost condition is textually identical to the C-grade
  definition, so every C in a "transitioning" regime is promoted to B
  before gate 6e reads it, making sell_min_quality's exclusion of "C" a
  no-op on M15/H4 (H1 and M5 are immune by other means). The backtester
  runs the same code, so every pinned book already includes this —
  changing it is a rule-1 revalidation event.
- Safe-haven/geopolitical headlines score exactly 0.00 in the keyword
  fallback and are now dropped entirely by the ±0.1 neutral filter,
  contradicting the platform's own gold doctrine. Bites only when
  DeepSeek is capped/disabled/failing.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Ah5EB1goi5inQmSd37Ne6a

.gitignore

@@ -0,0 +1,39 @@
+# Secrets
+.env
+.env.local
+config.json
+
+# Oanda daemon state
+oanda_state.json
+
+# Editor / tooling
+.claude/
+.vscode/
+.idea/
+
+# Python
+__pycache__/
+*.pyc
+*.egg-info/
+.venv/
+venv/
+
+# OS
+.DS_Store
+Thumbs.db
+
+# Logs
+*.log
+
+# Backtesting data (keep scripts, ignore generated files)
+backtest/cache/
+backtest/*.csv
+backtest/research/*.parquet
+backtest/research/duka_m5_boundary_spreads.csv
+backtest/trailgrid/
+backtest/trailgrid5m/
+backtest/__pycache__/
+.env.*
+logs/
+backtest/research/commit_ladder_2026-08/
+backtest/research/llm_news_eval_2026-08/

CLAUDE.md

@@ -0,0 +1,96 @@
+# CLAUDE.md
+
+This file provides guidance to Claude Code (claude.ai/code) when working with code in this repository.
+
+## What this is
+
+Signal-intelligence platform for **XAU/USD (gold — the only supported pair)**. It fetches OHLCV from the active data provider (**Oanda v20 since 2026-07-28**; Twelve Data and Tiingo remain registered fallbacks), scores 9 technical indicators per timeframe with regime-adaptive weights, checks multi-timeframe agreement and news risk, grades the signal (A+…F), and generates a full trade setup (entry, SL, TP1/TP2, trailing plan). Delivery: Telegram bot and CLI. A realism-focused backtester replays history through the *exact same* signal code.
+
+Every numeric threshold in `config.py` and `pair_signals/xauusd.py` was backtested or is load-bearing for a backtested result. Validation evidence lives in the factory/module docstrings and in `~/.claude/projects/.../memory/`.
+
+## Commands
+
+```bash
+python test_all.py                      # QA suite: 414 tests, 27 modules, no network (mocked)
+python main.py --pair XAUUSD --timeframe H1 --detail   # CLI signal report (--json for full dump,
+                                        #  --panel to run the advisory LLM agent panel on-demand)
+python telegram_bot.py                  # the bot (refuses to boot without credentials)
+python dashboard_publisher.py --once    # web dashboard: one scan+publish cycle to $DASHBOARD_OUT_DIR
+                                        # (omit --once for the candle-close daemon; deploy/README-dashboard.md)
+
+# Backtests
+python backtest/gpu_backtest.py --pair XAUUSD --days 365
+python backtest/gpu_backtest.py --pair XAUUSD --start 2025-01-01 --end 2025-06-01 \
+    --primary-tf 15min --confirm-tfs 1h,4h
+# key flags: --spread --quality A+,A,B,C --strategies --workers --max-hold-bars
+#            --warmup-bars --cooldown-bars --no-cache --export-csv out.csv --detail
+#            --fixed-rr-tp N (research-only: hard TP at N×risk, trail off; verdict
+#            2026-08-05: NOT shipped — backtest/research/fixedtp_2026-08-05.md)
+```
+
+**`test_all.py` is NOT pytest** — it uses a custom `@test("name")` decorator that runs each test immediately at import; don't grep for `def test_` or try `pytest test_all.py`. Run the whole file; it prints per-module results and exits non-zero on failure. There is no way to run a single test short of commenting modules out. Tests must stay green; every new gate/knob needs tests. Signal-log fixtures: use a WEEKDAY signal bar when a test exercises candle continuity — the outcome model treats weekend and 21:00 UTC gaps as scheduled, so Saturday-dated bars never produce a gap. `_candles_json` stores `df.tail(14)`, so a test snapshot longer than 14 bars silently drops its head. When patching weekend logic in tests, patch `providers.base._is_forex_weekend` — but know there are THREE independent signal-path weekend implementations (`providers/base.py`, `signal_aggregator.py`, and an inline copy in `backtest/gpu_backtest.py:_is_xau_blocked_by_calendar`) plus the advisory-only continuity copy in `advisory_outcome.py`; patching one does not reach the others.
+
+Python 3.12; a `venv/` exists in-repo. Dependencies in `requirements.txt` (numpy, pandas, requests, twelvedata, python-telegram-bot ≥20, websockets).
+
+## Configuration & credentials
+
+Precedence for every credential/deployment setting: **env var → config.json → refuse to run** (no hardcoded fallbacks). A gitignored repo-root `.env` is loaded at `app_config` import via `os.environ.setdefault`. `config.json` (gitignored; schema in `config.json.example`) holds secrets plus deployment settings: `data_providers.active` (the live-feed switch), `data_providers.streaming` (optional tick-stream provider for the level-breach monitor), `scanning.pairs`/`timeframes`, per-pair `blocked_hours_utc`, `log_level`. Key env vars: `TELEGRAM_TOKEN`, `TELEGRAM_CHAT_ID`, `TELEGRAM_CHANNEL_ID`, `TWELVE_DATA_API_KEY`, `TIINGO_API_KEY`, `OANDA_ACCOUNT_ID`/`OANDA_API_TOKEN`/`OANDA_BASE_URL`, `DEEPSEEK_API_KEY` (news LLM; the `deepseek` config.json block holds model/enabled/timeout/daily cap). The `agent_panel` config.json block (all-defaults = feature dark) controls the advisory LLM agent panel: `enabled`, per-surface `telegram`/`dashboard` toggles, `min_quality`, `max_panels_per_day`, `reuse_ttl_seconds`, plus the technical-payload enrichment knobs `candles_enabled`/`candle_bars_primary`/`candle_bars_confirm`. The dark-by-default `market_outlook` block controls the hourly advisory market brief: `enabled`, `telegram`, `max_per_day`, `candles_bars`; `MARKET_OUTLOOK_ENABLED` overrides its enabled setting.
+
+## Architecture
+
+### End-to-end pipeline (every scan)
+
+```
+providers/<active>.fetch_ohlcv()       per TF: single-flight fetch, drop forming candle, validate/clean, cache
+                                       (only Twelve Data rate-throttles; Oanda has no client throttle)
+  → engine.TechnicalEngine.analyze()   per TF: 9 scored indicators + ATR context → regime → composite score
+  → signal_aggregator.analyze_pair()   live entry point (weekend/day filters, fetch, staleness)
+      └─ analyze_from_dataframes()     replayable core: MTF agreement → news → bonuses/penalties →
+                                       final signal + quality grade → gates → trade setup → reasons
+  → SignalReport                       consumed by telegram_bot.py, main.py, backtester
+```
+
+Signal currency everywhere is `SignalStrength` (−2..+2). DataFrames are UTC tz-aware, oldest-first, capitalized OHLCV columns. Everything runs on **closed bars only** — the forming candle is dropped, and session/hour filters evaluate **bar-CLOSE time** (bar label + candle duration) identically in live and backtest.
+
+### Layers
+
+- **`providers/`** — one class per provider subclassing `BaseDataProvider` (`providers/base.py`), which owns everything the pipeline depends on: TTL+boundary cache, forming-candle drop, freshness check, validate/clean, paginated `fetch_range` (feeds the backtester from any provider), optional tick streaming. Subclasses implement only `_fetch_raw` + symbol/interval maps. Registry + `make_data_provider(config, name=None)` in `providers/__init__.py`; config.json `data_providers.active` selects the live candle feed and the optional `data_providers.streaming` scalar selects a different provider for the tick stream. A test enforces that subclasses don't override shared base methods (`supports_streaming`/`iter_ticks` are the sanctioned streaming override points). Tiingo streams mid-price ticks (`wss://api.tiingo.com/fx`) — do not re-add spread display from its free FX feed (two-tier indicative quotes, removed 2026-07-23); the no-spread rule extends to the WS stream.
+- **`engine.py`** — `TechnicalEngine`: 9 scored indicators (ema_trend, rsi, macd, bollinger, adx, roc, keltner, divergence, candlestick) + ATR as context. Regime classification (trending_up/down, `trending` = mixed/direction-UNKNOWN, transitioning, ranging) drives per-indicator weight multipliers and regime-aware RSI reinterpretation. Compute order matters (ATR first; MACD always computed even when not a pillar — divergence needs the histogram). Private series ride on the result as `_underscore` attrs; scorers/formatters skip anything `_`-prefixed or weight-0.
+- **`signal_aggregator.py`** — MTF agreement, diminishing-decay penalty system, exhaustion hard block, quality grading, then an ordered chain of post-grade gates (session, XAU SELL quality/vol/hour, regime blocks, M5 noise gates), then three setup generators: XAU ATR (A+/A), generic swing-based, plus the M5 pullback (`XAU_PB`) and chart-pattern (`XAU_CP`, `chart_patterns.py`) entry paths. The PB/CP generators run AFTER the gate chain and force quality to "B"; the CP path deliberately accepts grade-F composites and re-checks only session/day/big-candle (via warning prefixes) plus its own SELL/vol/RSI gates — it does NOT re-check exhaustion or the M5 min-score/MTF gates. **This is validated behavior, not a bug** (2026-08-04 audit: 226/586 trades in the shipped M5 12mo book are XAU_CP, +38.9R, 104 of them exhaustion-flagged and net-positive) — changing it is a rule-1 revalidation event. `TradeSetup.strategy` selects trailing params; trail params are stamped on the setup from the *producing* config so per-TF overrides reach formatters.
+- **`pair_signals/`** — per-pair config factories. **Always use the factory (`produce_signal` / `make_config`), never bare `PlatformConfig()`** — the factory applies the config.json overlay first, then the validated per-TF deltas (M5 hour-9 block, M15 trail arm 0.70, M15 SELL-hour block, M5 SELL-in-transitioning block, H1 all-shorts block (2026-09-03), etc.; evidence in the docstring). A test enforces factory ≡ inline-config equivalence. Confirmation map: M5→[M15,H1], M15→[M5,H1], H1→[M15,H4], H4→[H1,D1].
+- **`signal_log.py` / `advisory_outcome.py`** — SQLite signal-history logger (stdlib only). Hooked into `SignalAggregator.analyze_pair` only (the live entry point — the backtester never writes here, preserving parity), so every live scan from every surface (Telegram `telegram`, CLI `cli`, dashboard `dashboard`) lands one row in `logs/signals.db`: signal/grade/price, MTF + trade-setup + reasons JSON, full per-TF indicator snapshots, the last 14 closed OHLCV candles per fetched TF, and the news at scan time (event risk + calendar events + scored headlines, re-fetched via the NewsEngine class-level caches). Config.json `signal_log` block (`enabled`, `path`, `log_blocked_scans`; env `SIGNAL_LOG_ENABLED`/`SIGNAL_LOG_PATH`/`SIGNAL_LOG_BLOCKED_SCANS`). **Fail-safe by contract** — a logging failure can never break a scan. Panel fields arrive post-hoc: `log_signal` stashes the rowid and `agent_panel.attach_panel` writes verdict + creation/cache provenance + payload hash onto the same row. **Advisory outcomes (2026-09):** EVERY row with a trade setup is seeded at log time (not only panel rows); same-setup re-logs within 60 min from any surface (`setup_key` = tf|direction|entry|stop|strategy) are `DUPLICATE` rows pointing at the canonical row via `advisory_event_id`, so sum `advisory_r_multiple` over `RESOLVED` rows only. The versioned model advances from later stored closed-candle snapshots (next-open reanchoring, intrabar peak with close-only ratchet, SL-first, scheduled weekend/21:00-UTC closures allowed, 300-bar timeout, $0.30/$0.65 views); when a snapshot no longer overlaps the outcome's last bar it rebuilds the stream from every stored snapshot of that TF before giving up. It is a cleaned-mid evidence model, not execution P&L. **Post-gate truth columns:** `gate_reason` (the warning that removed the setup), `actionable` (grade ≥ C AND setup — the `action` text is written BEFORE the gate chain and can say "STRONG SELL — High-quality setup" on a D row), `cleaned_warnings_json` (what the XAU_CP acceptance stripped), `bar_range_median` (= the misnamed `spread` column: median High-Low, NOT a broker spread), `scan_kind` (`blocked` weekend/day early returns carry no news/candles), `config_fingerprint` (gate config hash — surfaces config drift between processes). **News is content-addressed:** `headlines_json`/`events_json` are interned into `news_snapshots` (they were 73% of a 667 MB DB); read through the `signals_with_news` view or `signal_log.read_news`. Readers must open with `mode=ro`, never `immutable=1` (blind to WAL rows). Tools: `analyze_signal_log.py` (`--stored-candles` DB-only replay with event/opportunity/one-position books + persona scorecard, `--news` news-sentiment lead/lag + extremes + outcome buckets, `--parity` telegram-vs-dashboard divergence report), `scripts/backfill_signal_outcomes.py` (seed/dedup/advance legacy rows), `scripts/migrate_signal_log_news.py` (intern legacy blobs; `--vacuum` with both writers stopped).
+- **`news_engine.py`** — Forex Factory calendar (MyFXBook RSS fallback) + RSS keyword sentiment with gold-inverse keywords. Class-level caches (fresh engine per scan). Optional DeepSeek enrichment via `llm_client.py` (requests-only, no SDK): batched headline sentiment overriding keyword scores, gold-direction on release surprises, and a market-summary blurb (`SignalReport.news_summary`) — **display/warning-only, keyword fallback on any failure, inert without a key**; prompts are static module constants (DeepSeek prefix-cache pricing) and requests send `thinking: disabled` (V4-Flash reasoning otherwise blows the timeout). By XAU-tuned default, event risk and sentiment **warn but do not gate signals** (`event_risk_caps_signals=False`, `sentiment_affects_signals=False`) — the LLM does not change that. **Headline pool (2026-09-04):** only headlines ≤ 4 h old with a parsable publish time (`headline_max_age_seconds=14400`, undated items dropped), and `get_fundamental_context` drops headlines whose every relevant score is inside ±0.1 (`_is_neutral_headline`, `NEUTRAL_HEADLINE_THRESHOLD`) before the per-currency mean — so `news_sentiment` from that date is a mean over directional headlines only (bolder values; log comparisons must split at the deploy date). The market summary prompt writes for someone who has never traded (no acronyms), sends headlines newest-first with the platform's own gold direction, and caps at 400 chars. **Verdict 2026-09-04** (`backtest/research/news_sentiment_value_2026-09-04.md`): `news_sentiment` is a trailing-24h echo of price (r≈0.67 with the past day, ≈0.03 with the next; flips 44 h after tops / 16 h after bottoms) — both gating flags stay False; re-check with `analyze_signal_log.py --stored-candles --news`.
+- **`agent_panel.py`** — ai-hedge-fund-style advisory panel: 4 DeepSeek persona calls (macro/technical/risk/contrarian, concurrent on a panel-private pool) + 1 synthesizer producing an **ADVISORY-ONLY** consensus (`AGREE/NEUTRAL/OPPOSE` + confidence + rationale) written onto `SignalReport.panel_*` fields. Attached **post-hoc** by consumers (`telegram_bot` alert path, `main --panel`, `dashboard_publisher`) — never runs inside `analyze_from_dataframes` (backtest parity, test-locked) and no gate reads its fields. Persona payloads are role-filtered: macro gets high-impact events plus only code-scored directional news; technical gets strategy/pattern, primary indicators, MTF, levels, and ATR-normalized candles; risk gets setup strategy, strategy cost assumption, trailing semantics, volatility/events/warnings (never the median candle range as spread); contrarian gets directional news, setup, timestamped latest-closed price, indicators, reasons, and warnings. Spend is bounded by exact cache, full-evidence-identity TTL reuse, and `max_panels_per_day`; changed evidence never inherits a direction-only cached verdict (test-locked — which also means TTL reuse effectively never fires live because price/candles change every bar; the daily cap is the real spend bound). Verdict provenance logs creation time, `MISS`/`EXACT_HIT`/`TTL_REUSE`, and a SHA-256 payload hash. Prompts forbid outside knowledge and outputs remain short. **OPPOSE guard (2026-09):** the 3-week signal-log scorecard showed the contrarian/risk/technical personas oppose almost every trend entry (chair said OPPOSE 73% of the time; opposed trades made +30R, PF 2.6) while the macro persona was the only informative voice (macro=support → 88% win). `_apply_oppose_guard` therefore downgrades a chair OPPOSE to NEUTRAL unless macro opposes, or technical opposes while macro is not supportive; the synthesizer prompt states the same rule. Re-score with `analyze_signal_log.py --stored-candles` (persona scorecard) before changing it again. **Verdict source (2026-09-03, owner decision):** all four personas still run and are logged (`panel_takes_json`), but `agent_panel.verdict_source` defaults to `"macro"` — the alert verdict is the macro persona's stance (support/neutral/oppose → AGREE/NEUTRAL/OPPOSE) and the synthesizer call is skipped (4 LLM calls per panel instead of 5); `"chair"` restores the four-voice synthesizer + OPPOSE guard. Re-score personas monthly with `analyze_signal_log.py --stored-candles`; weights are not to be tuned before ~300 graded outcomes with a held-out half.
+- **`market_outlook.py`** — hourly DeepSeek market brief for XAU/USD, using ATR-normalized M15/H1/H4 candles, visible indicator states, levels, news, and recent signal history (read `mode=ro`). It returns a written **ADVISORY-ONLY** bias/setup-forming assessment and persists it independently in `outlooks` within `logs/signals.db` together with the news, indicator/regime/level, and candle evidence sent to the model. It may be sent post-scan by Telegram or requested with `main.py --outlook`; it never imports or runs inside the signal path. Hourly dedup is by UTC hour BUCKET (the old `now − last < 3600` rule dropped every other hour because the H1 loop fires at boundary+10 s+scan time) plus its own UTC-day cap (default 12); every silent early return logs `market outlook: skipped (...)` and an empty LLM reply logs a warning — a dark feature must be visible in `logs/telegram.log`. Provider/news failures leave a safe partial brief while LLM/database failures degrade to no outlook.
+- **`telegram_bot.py`** — owner-locked bot + subscriber channel; candle-boundary auto-scan loops (M5/M15/H1/H4, boundary+10 s), alert quality threshold + fingerprint dedup (committed only after a successful send; the fingerprint is signal+direction — grade is NOT part of it, a grade UPGRADE re-alerts, an A↔B flicker does not), 📰 LLM market-context line on alerts when available, WebSocket level-breach monitor fed by the streaming provider (`data_providers.streaming`, falling back to the active provider; auto-disabled when the resolved provider can't stream or has no API key — Oanda does not stream, Tiingo does). The breach handler reuses this bar's M15 report (`_cached_or_scan`, keyed by closed-bar epoch) instead of rescanning — it used to log 3.5 M15 rows per candle. Rotating log: `logs/telegram.log`, 3-h rotation × 240 backups.
+- **`dashboard_publisher.py` / `static/dashboard.html`** — web dashboard as a **static publisher** (live LAMP server): standalone daemon scans at candle closes via `produce_signal` and atomically writes `snapshot.json`/`news.json`/`dashboard.html` to `$DASHBOARD_OUT_DIR` (Apache serves them; page is public by owner decision). **Hard independence rule: never imports, and is never imported by, `telegram_bot`** — confirmation map and boundary math are deliberate local copies, so a web-side failure can only mean a stale page. Due-TF classification uses the RAW wake time (flooring wake−buffer skips every H1/H4 scan — 2026-08-02 bug). News panel reads `NewsEngine.get_fundamental_context()` plus the additive `get_week_events()` accessor (display-only; no signal path calls it); as a separate process it has its own news caches and DeepSeek daily cap. News/calendar rows are clickable (detail dialog); headlines carry `id` = sha1(headline) in news.json, and exactly one on-demand endpoint exists: `static/news_summary.php` (installed by `install_static`), which serves a DeepSeek gold-impact summary per headline — whitelist-gated against news.json ids, cache-forever in `ai/`, own fail-closed daily cap (150/day, file-based, separate from the publisher's), key at `/etc/gold-dashboard/deepseek.key`, never reads repo config. Deployment checklist: `deploy/README-dashboard.md`.
+- **`backtest/gpu_backtest.py`** — bar-by-bar replay through the real `analyze_from_dataframes` (news off). Realism model (do not regress): next-bar-open entries re-anchored to the raw price path, round-trip `--spread` charged per trade, trade resolution on RAW bars (cleaning applies only to signal-side data), same-bar SL+TP tie → SL first, gap fills at bar open, trail ratchet at bar close only. Indicators are precomputed once per TF (CuPy GPU or NumPy fallback — check the `Backend:` log line); replay fans out over worker processes and the 5-bar cooldown is applied in an ordered pass afterwards, so parallel and sequential books are identical. Parquet cache in `backtest/cache/` keyed by provider/symbol/TF/window. `backtest/*.csv` are gitignored pinned trade books from past campaigns — used for manual before/after comparison. `backtest/instructions_for_backtest.txt` lists what every backtest report must record.
+
+## Rules for making changes (read before editing signal logic)
+
+1. **Never change a tuned value without re-validation.** Acceptance rule: positive total R in BOTH train and validation halves of a ≥12-month window, PF ≥ 1.3, at realistic cost (XAU $0.65 round trip headline; retain $0.30 for TD-era cross-book comparability), trailing exits, next-open fills. `round_trip_cost` stays $0.30 because it also drives the live TP spread gate; changing it is a separate rule-1 event. Marginal edges must additionally pass the 2024-06→2025-03 out-of-sample window — within-year splits alone have approved configs that later failed OOS.
+2. **Per-timeframe overrides live in the pair factory**, never in global defaults — the same knob has opposite effects on different TFs (proven repeatedly: `tp_floor_rr`, hour-9 block, trail params). Document validation evidence in the factory docstring.
+3. **Pair isolation is a hard invariant.** XAU gates in the aggregator are display-name-gated so a future pair port cannot inherit gold-calibrated logic by accident.
+4. **Live == backtest parity.** If you add a live gate, add it to the replay path in the same commit; all time filters evaluate bar-close time in both paths.
+5. **ROC stays weighted** even though it's correlated with MACD/Keltner/EMA — zeroing it backtested 13R worse; composite thresholds are calibrated with it.
+6. **Backtester realism knobs are results-critical** (raw-path resolution, spread charging, next-open entry, SL-first ties, close-only trail). Touching them invalidates comparability with all pinned books.
+7. **The forming candle must stay dropped** — live signals repaint otherwise.
+8. Don't add config fields nothing reads; don't disable indicators in production paths (`disabled_indicators` is for experiments only).
+9. EUR/USD was removed 2026-07-16 after a no-edge verdict (cost-to-volatility; last commit with EUR support: `7d3fd95`). Do not re-add without new data. BTC/ETH work lives on unmerged branches only.
+
+## 2026-08-04 QA audit — owner decisions ledger
+
+- **Weekend calendar: FIXED (owner-approved 2026-08-04).** All three signal-path `_is_forex_weekend` copies are NY-anchored (Fri 17:00 → Sun 18:00 ET; during US DST that is Fri 21:00 UTC — measured on 5yr of Oanda bars). Kills the weekly false-"STALE DATA/feed lagging" hours. Book impact at the audit HEAD: M15 byte-identical; M5 12mo dropped exactly ONE trade — a phantom whose signal fired on the final summer-Friday bar (close 21:00 UTC) and "filled" at Sunday's reopen 49 h later (−1.08R, live-impossible), leaving 555 trades / +98.64R / PF≈1.50 @ $0.65 (was 556/+97.56/1.48). The current-HEAD canonical reference is recorded below. A Module 21 test sweeps those copies plus the advisory outcome's continuity copy hourly across both DST transitions.
+- **Spread in exit-trigger detection (rule 6): owner DECLINED 2026-08-04.** Backtest stop triggers stay on raw mid High/Low; the measured optimism (~4–6% of total R: −3.55R @ $0.30, −4.24R @ $0.65 on the M15 12mo book) is accepted and books stay comparable. Re-open only as a conscious re-pin campaign.
+- **`FB2_EMA_LO`: HARDENED 2026-08-04 (delegated).** Research override for the EMA200-proximity band, default = tuned 0.5; an unparseable value now degrades to the default with a warning instead of crashing every scan. Documented here — it is a knob that exists.
+- **H4 seeding divergence: parked.** Live H4 fetches only 380 bars vs the backtest's unbounded history — ~2% of sampled H4 bars flip an indicator signal (composite Δ up to 12.3). M5/M15/H1 clean at their real fetch depths. Fold into any future H4 campaign; H4 remains "unproven".
+
+## Current state notes (2026-08-14)
+
+- Oanda v20 is the live feed (`data_providers.active: "oanda"`). The 2026-08-04 re-pin made the 27 `backtest/oanda_*.csv` books the baseline; all TD-era books are superseded. Current-HEAD M5 canonical book: `backtest/oanda_m5_12mo_065_nopb_20260814.csv`, +106.82R PF 1.51 at $0.65 (both halves positive; PF 1.46 / 1.55). M15 reference: +73.93R PF 1.53. See `backtest/research/repin_verification_2026-08-14.md`.
+- Cost reporting is dual: $0.65 is the measured-Oanda realistic headline ($0.65-$0.72 round trip), while $0.30 remains for cross-book comparability and as the live TP spread-gate input. Raising `round_trip_cost` from $0.30 would move the TP floor from about $9 to $19.50 and requires full rule-1 revalidation.
+- Twelve Data is signal-incompatible with Oanda (Phase 1: about 19% M5 bar-direction disagreement and a systematic ~$1 high bias; see `backtest/research/feed_divergence_2026-08-14.md`). On an Oanda candle outage, stop scanning rather than switching to TD candles.
+- Tiingo is streaming-only (`data_providers.streaming: "tiingo"`), not a designated candle fallback. Oanda-Tiingo signal parity is documented in `backtest/research/signal_parity_2026-08-14.md`; Phase 4's two-week live shadow scan remains a manual operational step.
+- Human-facing docs in `docs/` (`user_guide.html`, `beginner_guide.html` + PDF builds) must be kept in sync when signal behavior changes.
+- 2026-09-03 signal-log campaign shipped two new per-TF gates (`backtest/research/signal_log_campaign_2026-09.md`): M5 `sell_blocked_regimes=["transitioning"]` and H1 `sell_min_quality=[]` (all H1 shorts blocked). Current-HEAD M5 canonical book is now `backtest/oanda_m5_12mo_065_nopb_20260903.csv` (557 / +112.21R / PF 1.56, halves +55.9/+56.4); H1 reference `backtest/oanda_h1_12mo_065_20260903.csv` (91 / +22.2R / PF 1.79; 5yr +49.1R PF 1.46; OOS +16.7R PF 1.74).
+- Auto-execution (oanda_daemon.py / oanda_executor.py) was removed 2026-09-03 — the owner is not auto-trading; last commit containing it is 31002cb. Book-parity rules that lived in its docstring (no broker TP, 5-bar cooldown, once-per-bar trail) remain the backtester's model.

advisory_outcome.py

@@ -0,0 +1,354 @@
+"""Deterministic advisory trade outcomes from stored closed candles.
+
+This module measures the hypothetical setup reviewed by the LLM panel. It
+never changes a signal, panel verdict, order, or daemon state. The model uses
+the same next-open reanchoring and close-only trailing rules as the backtester,
+but its evidence is the cleaned mid-candle snapshots stored in signals.db.
+"""
+
+from __future__ import annotations
+
+import logging
+import math
+from datetime import datetime, timedelta
+from typing import Any, Dict, Iterable, List, Optional, Tuple
+from zoneinfo import ZoneInfo
+
+logger = logging.getLogger(__name__)
+
+MODEL = "stored-closed-mid-next-open-trail-v1"
+MAX_HOLD_BARS = 300
+REFERENCE_COST = 0.30
+REALISTIC_COST = 0.65
+
+_TF_ALIASES = {
+    "M1": "1min", "1MIN": "1min", "1min": "1min",
+    "M5": "5min", "5MIN": "5min", "5min": "5min",
+    "M15": "15min", "15MIN": "15min", "15min": "15min",
+    "M30": "30min", "30MIN": "30min", "30min": "30min",
+    "H1": "1h", "1H": "1h", "1h": "1h",
+    "H4": "4h", "4H": "4h", "4h": "4h",
+    "D1": "1day", "1D": "1day", "1d": "1day",
+    "1DAY": "1day", "1day": "1day",
+}
+
+_TF_SECONDS = {
+    "1min": 60,
+    "5min": 5 * 60,
+    "15min": 15 * 60,
+    "30min": 30 * 60,
+    "1h": 60 * 60,
+    "4h": 4 * 60 * 60,
+    "1day": 24 * 60 * 60,
+}
+
+_NY_TZ = ZoneInfo("America/New_York")
+
+
+def _is_forex_weekend(now: datetime) -> bool:
+    """NY-anchored XAU weekend closure; test-locked to the signal copies."""
+    ny = now.astimezone(_NY_TZ)
+    weekday = ny.weekday()
+    if weekday == 4:
+        return ny.hour >= 17
+    if weekday == 5:
+        return True
+    if weekday == 6:
+        return ny.hour < 18
+    return False
+
+
+def normalize_timeframe(value: Any) -> str:
+    text = str(getattr(value, "value", value) or "")
+    return _TF_ALIASES.get(text, _TF_ALIASES.get(text.upper(), text))
+
+
+def timeframe_value(mapping: Any, timeframe: Any) -> Any:
+    """Read a timeframe-keyed snapshot regardless of enum/storage alias."""
+    if not isinstance(mapping, dict):
+        return None
+    wanted = normalize_timeframe(timeframe)
+    for key, value in mapping.items():
+        if normalize_timeframe(key) == wanted:
+            return value
+    return None
+
+
+def _finite(value: Any) -> Optional[float]:
+    try:
+        number = float(value)
+    except (TypeError, ValueError):
+        return None
+    return number if math.isfinite(number) else None
+
+
+def _timestamp(value: Any) -> Optional[datetime]:
+    try:
+        return datetime.fromisoformat(str(value).replace(" ", "T"))
+    except (TypeError, ValueError):
+        return None
+
+
+def initial_outcome(
+    setup: Optional[Dict[str, Any]],
+    indicators: Optional[Dict[str, Any]],
+    timeframe: Any,
+) -> Tuple[str, Dict[str, Any]]:
+    """Freeze the setup and deterministic evaluation policy."""
+    if not setup:
+        return "NOT_APPLICABLE", {
+            "model": MODEL,
+            "status": "NOT_APPLICABLE",
+            "reason": "panel row has no trade setup",
+        }
+
+    tf = normalize_timeframe(timeframe)
+    analysis = timeframe_value(indicators, tf)
+    signal_bar = analysis.get("timestamp") if isinstance(analysis, dict) else None
+    direction = str(setup.get("direction", ""))
+    entry = _finite(setup.get("entry"))
+    stop = _finite(setup.get("stop_loss"))
+    tp1 = _finite(setup.get("take_profit_1"))
+    arm = _finite(setup.get("trail_arm_r"))
+    distance = _finite(setup.get("trail_dist_r"))
+    risk = abs(entry - stop) if entry is not None and stop is not None else 0.0
+    if (direction not in ("BUY", "SELL") or not signal_bar
+            or _timestamp(signal_bar) is None or risk <= 0 or tp1 is None
+            or arm is None or distance is None or arm < 0 or distance < 0):
+        return "UNRESOLVABLE", {
+            "model": MODEL,
+            "status": "UNRESOLVABLE",
+            "reason": "invalid or incomplete setup/outcome seed",
+        }
+
+    state = {
+        "model": MODEL,
+        "status": "PENDING",
+        "timeframe": tf,
+        "signal_bar_at": str(signal_bar),
+        "setup": {
+            "direction": direction,
+            "reported_entry": entry,
+            "stop_distance": risk,
+            "tp1_distance": abs(tp1 - entry),
+            "strategy": setup.get("strategy"),
+            "setup_pattern": setup.get("setup_pattern"),
+        },
+        "policy": {
+            "entry": "next_primary_bar_open_reanchored",
+            "max_hold_bars": MAX_HOLD_BARS,
+            "trail_arm_r": arm,
+            "trail_dist_r": distance,
+            "tp1_is_advisory": arm > 0,
+            "reference_cost": REFERENCE_COST,
+            "realistic_cost": REALISTIC_COST,
+        },
+        "progress": {
+            "last_processed_bar": str(signal_bar),
+            "entry_at": None,
+            "entry_price": None,
+            "initial_stop": None,
+            "current_stop": None,
+            "peak": None,
+            "bars_held": 0,
+            "mfe_r": 0.0,
+            "mae_r": 0.0,
+        },
+        "result": None,
+    }
+    return "PENDING", state
+
+
+def _valid_bars(bars: Iterable[Dict[str, Any]]) -> List[Dict[str, Any]]:
+    valid = []
+    for bar in bars:
+        stamp = _timestamp(bar.get("timestamp")) if isinstance(bar, dict) else None
+        values = [_finite(bar.get(key)) for key in ("open", "high", "low", "close")]
+        if stamp is None or any(value is None for value in values):
+            continue
+        valid.append({
+            "timestamp": str(bar["timestamp"]),
+            "_timestamp": stamp,
+            "open": values[0], "high": values[1],
+            "low": values[2], "close": values[3],
+        })
+    valid.sort(key=lambda item: item["_timestamp"])
+    return valid
+
+
+def _has_continuity(
+    bars: List[Dict[str, Any]], last: datetime, timeframe: str
+) -> bool:
+    """Reject missing tradable bars while allowing scheduled XAU closures."""
+    seconds = _TF_SECONDS.get(normalize_timeframe(timeframe))
+    if seconds is None:
+        return False
+    step = timedelta(seconds=seconds)
+    previous = last
+    seen = {last}
+    for bar in bars:
+        stamp = bar["_timestamp"]
+        if stamp <= previous or stamp in seen:
+            continue
+        expected = previous + step
+        while expected < stamp:
+            ny = expected.astimezone(_NY_TZ)
+            daily_maintenance = ny.weekday() < 4 and ny.hour == 17
+            if not (_is_forex_weekend(expected) or daily_maintenance):
+                return False
+            expected += step
+        if expected != stamp:
+            return False
+        previous = stamp
+        seen.add(stamp)
+    return True
+
+
+def _finish(
+    state: Dict[str, Any], reason: str, exit_at: str, exit_price: float
+) -> None:
+    setup = state["setup"]
+    progress = state["progress"]
+    entry = float(progress["entry_price"])
+    risk = float(setup["stop_distance"])
+    pnl = ((exit_price - entry) if setup["direction"] == "BUY"
+           else (entry - exit_price))
+    gross_r = pnl / risk
+    state["status"] = "RESOLVED"
+    state["result"] = {
+        "exit_at": exit_at,
+        "exit_price": exit_price,
+        "exit_reason": reason,
+        "bars_held": progress["bars_held"],
+        "gross_r": gross_r,
+        "net_r_030": (pnl - REFERENCE_COST) / risk,
+        "net_r_065": (pnl - REALISTIC_COST) / risk,
+        "mfe_r": progress["mfe_r"],
+        "mae_r": progress["mae_r"],
+    }
+
+
+def advance_outcome(
+    state: Dict[str, Any], bars: Iterable[Dict[str, Any]]
+) -> Dict[str, Any]:
+    """Advance one pending outcome with overlapping closed-bar snapshots."""
+    if state.get("status") != "PENDING":
+        return state
+    try:
+        progress = state["progress"]
+        setup = state["setup"]
+        policy = state["policy"]
+        last = _timestamp(progress["last_processed_bar"])
+        signal_bar = _timestamp(state["signal_bar_at"])
+        if last is None or signal_bar is None:
+            raise ValueError("invalid progress timestamp")
+
+        buy = setup["direction"] == "BUY"
+        risk = float(setup["stop_distance"])
+        trail_arm = float(policy["trail_arm_r"])
+        trail_distance = float(policy["trail_dist_r"])
+
+        valid_bars = _valid_bars(bars)
+        has_future = any(bar["_timestamp"] > last for bar in valid_bars)
+        has_overlap = any(bar["_timestamp"] == last for bar in valid_bars)
+        if has_future and not has_overlap:
+            state["status"] = "UNRESOLVABLE"
+            state["result"] = None
+            state["reason"] = "stored candle snapshots lost continuity"
+            return state
+        if has_future and not _has_continuity(valid_bars, last,
+                                              state["timeframe"]):
+            state["status"] = "UNRESOLVABLE"
+            state["result"] = None
+            state["reason"] = "stored candle snapshots contain an internal gap"
+            return state
+
+        for bar in valid_bars:
+            stamp = bar["_timestamp"]
+            if stamp <= signal_bar or stamp <= last:
+                continue
+
+            if progress["entry_price"] is None:
+                entry = bar["open"]
+                initial_stop = entry - risk if buy else entry + risk
+                progress.update({
+                    "entry_at": bar["timestamp"],
+                    "entry_price": entry,
+                    "initial_stop": initial_stop,
+                    "current_stop": initial_stop,
+                    "peak": entry,
+                })
+
+            entry = float(progress["entry_price"])
+            current_stop = float(progress["current_stop"])
+            initial_stop = float(progress["initial_stop"])
+            progress["bars_held"] += 1
+            progress["last_processed_bar"] = bar["timestamp"]
+            last = stamp
+
+            if buy:
+                mfe = (bar["high"] - entry) / risk
+                mae = (entry - bar["low"]) / risk
+                stop_hit = bar["low"] <= current_stop
+            else:
+                mfe = (entry - bar["low"]) / risk
+                mae = (bar["high"] - entry) / risk
+                stop_hit = bar["high"] >= current_stop
+            progress["mfe_r"] = max(float(progress["mfe_r"]), max(0.0, mfe))
+            progress["mae_r"] = max(float(progress["mae_r"]), max(0.0, mae))
+
+            if stop_hit:
+                exit_price = current_stop
+                if (buy and bar["open"] < current_stop) or \
+                        (not buy and bar["open"] > current_stop):
+                    exit_price = bar["open"]
+                reason = "SL" if current_stop == initial_stop else "TRAIL"
+                _finish(state, reason, bar["timestamp"], exit_price)
+                return state
+
+            if trail_arm <= 0:
+                target = entry + setup["tp1_distance"] if buy \
+                    else entry - setup["tp1_distance"]
+                target_hit = bar["high"] >= target if buy else bar["low"] <= target
+                if target_hit:
+                    exit_price = target
+                    if (buy and bar["open"] > target) or \
+                            (not buy and bar["open"] < target):
+                        exit_price = bar["open"]
+                    _finish(state, "TP1", bar["timestamp"], exit_price)
+                    return state
+            else:
+                favorable = bar["high"] if buy else bar["low"]
+                peak = max(float(progress["peak"]), favorable) if buy \
+                    else min(float(progress["peak"]), favorable)
+                progress["peak"] = peak
+                excursion = ((peak - entry) if buy else (entry - peak)) / risk
+                if excursion >= trail_arm:
+                    level = peak - trail_distance * risk if buy \
+                        else peak + trail_distance * risk
+                    progress["current_stop"] = max(current_stop, level) if buy \
+                        else min(current_stop, level)
+
+            if progress["bars_held"] >= int(policy["max_hold_bars"]):
+                _finish(state, "TIMEOUT", bar["timestamp"], bar["close"])
+                return state
+        return state
+    except Exception as exc:
+        # Still fail-safe (never raises into a scan), but no longer
+        # SILENT: a real defect here used to be indistinguishable from a
+        # legitimate weekend/downtime gap, discoverable only by grepping
+        # advisory_outcome_json. Callers persist the state either way.
+        logger.exception(
+            "advisory outcome evaluator failed for the %s outcome seeded "
+            "at %s — recorded UNRESOLVABLE",
+            state.get("timeframe"), state.get("signal_bar_at"))
+        state["status"] = "UNRESOLVABLE"
+        state["result"] = None
+        state["reason"] = f"outcome evaluator error: {type(exc).__name__}"
+        return state
+
+
+def canonical_r(state: Dict[str, Any]) -> Optional[float]:
+    result = state.get("result")
+    value = result.get("net_r_065") if isinstance(result, dict) else None
+    return _finite(value)

app_config.py

@@ -0,0 +1,521 @@
+"""
+app_config.py — deployment-settings overlay (config.json)
+=========================================================
+
+Loads the gitignored config.json at the repo root ONCE per process.
+Deployment surface ONLY: Telegram credentials, data-provider selection
+and API keys, scanned pairs/timeframes, per-pair base blocked hours,
+alert-quality boot default, log level. Signal logic, weights,
+thresholds and gates stay in config.py / pair_signals.
+
+Precedence for every setting: env var → config.json → code default.
+A repo-root .env file (gitignored) is loaded at import via setdefault,
+so it behaves exactly like real environment variables except that a
+real shell export always wins. Secrets have NO code default — with
+neither env nor config.json the bot refuses to start
+(telegram_bot.main) and data fetches fail with a named error
+(data_provider).
+
+Parity contract (test-enforced): with no config.json, or one holding
+the config.json.example defaults, every produced PlatformConfig is
+byte-identical to the pure-code defaults. apply_pair_overrides() runs
+in the pair factory BEFORE the validated per-timeframe deltas, so the
+bot, CLI and backtester all see the same overlay (live == backtest).
+
+Spec: docs/superpowers/specs/2026-07-21-centralized-config-design.md
+"""
+
+import json
+import logging
+import os
+from dataclasses import dataclass, field
+from typing import Any, Dict, List, Optional
+
+from config import PAIR_INFO, PlatformConfig, Timeframe, XAUUSD
+
+logger = logging.getLogger(__name__)
+
+CONFIG_PATH = os.path.join(
+    os.path.dirname(os.path.abspath(__file__)), "config.json")
+ENV_PATH = os.path.join(
+    os.path.dirname(os.path.abspath(__file__)), ".env")
+
+
+def load_dotenv(path: str = ENV_PATH) -> None:
+    """Minimal .env loader (no python-dotenv dependency).
+
+    KEY=VALUE lines land in os.environ via setdefault — a variable
+    already exported in the shell always wins, keeping the
+    env → config.json precedence intact (.env is just a convenient
+    place to put env vars). Comments, blank lines, ``export `` prefixes
+    and single/double quotes are handled; anything else is skipped.
+    """
+    try:
+        with open(path, encoding="utf-8") as fh:
+            lines = fh.read().splitlines()
+    except OSError:
+        return
+    for line in lines:
+        line = line.strip()
+        if not line or line.startswith("#") or "=" not in line:
+            continue
+        if line.startswith("export "):
+            line = line[len("export "):]
+        key, _, value = line.partition("=")
+        key, value = key.strip(), value.strip()
+        if len(value) >= 2 and value[0] == value[-1] and value[0] in "\"'":
+            value = value[1:-1]
+        elif value[:1] not in "\"'":
+            # Unquoted value: whitespace-then-# starts an inline comment
+            # (KEY=abc  # prod). Quote the value to keep a literal " #".
+            for sep in (" #", "\t#"):
+                cut = value.find(sep)
+                if cut != -1:
+                    value = value[:cut].rstrip()
+        if key:
+            os.environ.setdefault(key, value)
+
+
+# Module import == process boot, before any resolver can run.
+load_dotenv()
+
+_VALID_QUALITIES = ("A+", "A", "B", "C")
+_VALID_LOG_LEVELS = ("DEBUG", "INFO", "WARNING", "ERROR", "CRITICAL")
+# Timeframes the bot's candle-boundary scheduler supports (must have
+# AUTOSCAN_CONFIRMATIONS entries in telegram_bot.py).
+_SCANNABLE_TFS = ("M5", "M15", "H1", "H4")
+
+_KNOWN_TOP = {"telegram", "data_providers", "scanning", "pairs", "log_level",
+               "oanda", "deepseek", "agent_panel", "market_outlook",
+               "signal_log"}
+_KNOWN_OANDA = {
+    "account_id", "api_token", "base_url",
+}
+_KNOWN_DEEPSEEK = {
+    "api_key", "model", "enabled", "timeout_seconds", "daily_request_cap",
+}
+_KNOWN_AGENT_PANEL = {
+    "enabled", "telegram", "dashboard", "min_quality",
+    "max_panels_per_day", "reuse_ttl_seconds",
+    "candles_enabled", "candle_bars_primary", "candle_bars_confirm",
+    "verdict_source",
+}
+_KNOWN_MARKET_OUTLOOK = {"enabled", "telegram", "max_per_day", "candles_bars"}
+_KNOWN_SIGNAL_LOG = {"enabled", "path", "log_blocked_scans"}
+_KNOWN_TELEGRAM = {"token", "chat_id", "channel_id", "alert_min_quality"}
+_KNOWN_SCANNING = {"pairs", "timeframes"}
+_KNOWN_PAIR_KEYS = {"blocked_hours_utc"}
+
+_MARKET_OUTLOOK_DEFAULTS = {
+    "enabled": False,
+    "telegram": False,
+    "max_per_day": 12,
+    "candles_bars": 20,
+}
+
+
+class AppConfigError(Exception):
+    """config.json is present but invalid — fail fast at boot."""
+
+
+@dataclass
+class AppConfig:
+    telegram_token: str = ""
+    telegram_chat_id: str = ""
+    telegram_channel_id: str = ""          # "" = channel feed disabled
+    alert_min_quality: str = "C"
+    active_provider: str = "twelve_data"
+    streaming_provider: Optional[str] = None   # None = active provider
+    provider_settings: Dict[str, Dict[str, Any]] = field(default_factory=dict)
+    scan_pairs: List[str] = field(default_factory=lambda: [XAUUSD])
+    scan_timeframes: List[Timeframe] = field(default_factory=lambda: [
+        Timeframe.M5, Timeframe.M15, Timeframe.H1, Timeframe.H4])
+    pair_overrides: Dict[str, Dict[str, Any]] = field(default_factory=dict)
+    oanda_settings: Dict[str, Any] = field(default_factory=dict)
+    deepseek_settings: Dict[str, Any] = field(default_factory=dict)
+    agent_panel_settings: Dict[str, Any] = field(default_factory=dict)
+    market_outlook_settings: Dict[str, Any] = field(
+        default_factory=lambda: dict(_MARKET_OUTLOOK_DEFAULTS))
+    signal_log_settings: Dict[str, Any] = field(default_factory=dict)
+    log_level: str = "INFO"
+
+
+def _check_keys(section: Dict[str, Any], known: set, where: str) -> None:
+    if not isinstance(section, dict):
+        raise AppConfigError(f"config.json: {where} must be a JSON object")
+    unknown = sorted(set(section) - known)
+    if unknown:
+        raise AppConfigError(
+            f"config.json: unknown key(s) {unknown} in {where}; "
+            f"known keys: {sorted(known)}")
+
+
+def load_app_config(path: str = CONFIG_PATH) -> AppConfig:
+    """Parse + validate config.json. Missing file → pure defaults."""
+    app = AppConfig()
+    if not os.path.exists(path):
+        logger.warning(
+            "config.json not found at %s — running on code defaults "
+            "(secrets must come from environment variables)", path)
+        return app
+    try:
+        with open(path, encoding="utf-8") as fh:
+            raw = json.load(fh)
+    except json.JSONDecodeError as e:
+        raise AppConfigError(f"config.json is not valid JSON: {e}") from e
+    if not isinstance(raw, dict):
+        raise AppConfigError("config.json top level must be a JSON object")
+    _check_keys(raw, _KNOWN_TOP, "top level")
+
+    tg = raw.get("telegram", {})
+    _check_keys(tg, _KNOWN_TELEGRAM, '"telegram"')
+    app.telegram_token = str(tg.get("token", app.telegram_token))
+    app.telegram_chat_id = str(tg.get("chat_id", app.telegram_chat_id))
+    app.telegram_channel_id = str(
+        tg.get("channel_id", app.telegram_channel_id))
+    quality = tg.get("alert_min_quality", app.alert_min_quality)
+    if quality not in _VALID_QUALITIES:
+        raise AppConfigError(
+            f'config.json: telegram.alert_min_quality "{quality}" '
+            f"not in {_VALID_QUALITIES}")
+    app.alert_min_quality = quality
+
+    dp = raw.get("data_providers")
+    if dp is not None:
+        if not isinstance(dp, dict):
+            raise AppConfigError('config.json: "data_providers" must be an object')
+        app.active_provider = str(dp.get("active", app.active_provider))
+        app.provider_settings = {}
+        for name, block in dp.items():
+            # "active" and "streaming" are reserved scalars, not blocks.
+            if name in ("active", "streaming"):
+                continue
+            if not isinstance(block, dict):
+                raise AppConfigError(
+                    f'config.json: data_providers.{name} must be an object')
+            app.provider_settings[name] = dict(block)
+        if app.active_provider not in app.provider_settings:
+            raise AppConfigError(
+                f'config.json: data_providers.active "{app.active_provider}" '
+                "has no matching provider block")
+        streaming = dp.get("streaming")
+        if streaming is not None:
+            if not isinstance(streaming, str):
+                raise AppConfigError(
+                    'config.json: data_providers.streaming must be a string '
+                    "(a provider name)")
+            if streaming not in app.provider_settings:
+                raise AppConfigError(
+                    f'config.json: data_providers.streaming "{streaming}" '
+                    "has no matching provider block")
+            app.streaming_provider = streaming
+
+    scanning = raw.get("scanning", {})
+    _check_keys(scanning, _KNOWN_SCANNING, '"scanning"')
+    if "pairs" in scanning:
+        pairs = scanning["pairs"]
+        if not isinstance(pairs, list):
+            raise AppConfigError('config.json: scanning.pairs must be a list')
+        bad = [p for p in pairs if p not in PAIR_INFO]
+        if bad:
+            raise AppConfigError(
+                f"config.json: unknown pair(s) {bad} in scanning.pairs; "
+                f"known: {sorted(PAIR_INFO)}")
+        app.scan_pairs = list(pairs)
+    if "timeframes" in scanning:
+        names = scanning["timeframes"]
+        if not isinstance(names, list):
+            raise AppConfigError(
+                'config.json: scanning.timeframes must be a list')
+        bad = [n for n in names if n not in _SCANNABLE_TFS]
+        if bad:
+            raise AppConfigError(
+                f"config.json: unsupported timeframe(s) {bad} in "
+                f"scanning.timeframes; supported: {list(_SCANNABLE_TFS)}")
+        app.scan_timeframes = [Timeframe[n] for n in names]
+
+    pair_section = raw.get("pairs", {})
+    if not isinstance(pair_section, dict):
+        raise AppConfigError('config.json: "pairs" must be a JSON object')
+    for symbol, overrides in pair_section.items():
+        if symbol not in PAIR_INFO:
+            raise AppConfigError(
+                f'config.json: unknown pair "{symbol}" in pairs; '
+                f"known: {sorted(PAIR_INFO)}")
+        _check_keys(overrides, _KNOWN_PAIR_KEYS, f'pairs."{symbol}"')
+        if "blocked_hours_utc" in overrides:
+            hours = overrides["blocked_hours_utc"]
+            valid = isinstance(hours, list) and all(
+                isinstance(h, int) and not isinstance(h, bool)
+                and 0 <= h <= 23 for h in hours)
+            if not valid:
+                raise AppConfigError(
+                    f'config.json: pairs."{symbol}".blocked_hours_utc must '
+                    "be a list of integers in 0-23")
+    app.pair_overrides = {s: dict(o) for s, o in pair_section.items()}
+
+    oanda = raw.get("oanda", {})
+    _check_keys(oanda, _KNOWN_OANDA, '"oanda"')
+    app.oanda_settings = dict(oanda)
+
+    deepseek = raw.get("deepseek", {})
+    _check_keys(deepseek, _KNOWN_DEEPSEEK, '"deepseek"')
+    if "enabled" in deepseek and not isinstance(deepseek["enabled"], bool):
+        raise AppConfigError(
+            "config.json: deepseek.enabled must be true or false")
+    if "timeout_seconds" in deepseek:
+        ts = deepseek["timeout_seconds"]
+        if isinstance(ts, bool) or not isinstance(ts, (int, float)) or ts <= 0:
+            raise AppConfigError(
+                "config.json: deepseek.timeout_seconds must be a positive "
+                "number")
+    if "daily_request_cap" in deepseek:
+        cap = deepseek["daily_request_cap"]
+        if isinstance(cap, bool) or not isinstance(cap, int) or cap < 1:
+            raise AppConfigError(
+                "config.json: deepseek.daily_request_cap must be a positive "
+                "integer")
+    if "model" in deepseek:
+        if not isinstance(deepseek["model"], str) or not deepseek["model"]:
+            raise AppConfigError(
+                "config.json: deepseek.model must be a non-empty string")
+    app.deepseek_settings = dict(deepseek)
+
+    panel = raw.get("agent_panel", {})
+    _check_keys(panel, _KNOWN_AGENT_PANEL, '"agent_panel"')
+    for flag in ("enabled", "telegram", "dashboard"):
+        if flag in panel and not isinstance(panel[flag], bool):
+            raise AppConfigError(
+                f"config.json: agent_panel.{flag} must be true or false")
+    if "min_quality" in panel and panel["min_quality"] not in _VALID_QUALITIES:
+        raise AppConfigError(
+            f'config.json: agent_panel.min_quality "{panel["min_quality"]}" '
+            f"not in {_VALID_QUALITIES}")
+    if "max_panels_per_day" in panel:
+        mpd = panel["max_panels_per_day"]
+        if isinstance(mpd, bool) or not isinstance(mpd, int) or mpd < 1:
+            raise AppConfigError(
+                "config.json: agent_panel.max_panels_per_day must be a "
+                "positive integer")
+    if "reuse_ttl_seconds" in panel:
+        ttl = panel["reuse_ttl_seconds"]
+        if isinstance(ttl, bool) or not isinstance(ttl, (int, float)) \
+                or ttl < 0:
+            raise AppConfigError(
+                "config.json: agent_panel.reuse_ttl_seconds must be a "
+                "non-negative number")
+    if "candles_enabled" in panel \
+            and not isinstance(panel["candles_enabled"], bool):
+        raise AppConfigError(
+            "config.json: agent_panel.candles_enabled must be true or false")
+    for key in ("candle_bars_primary", "candle_bars_confirm"):
+        if key in panel:
+            n = panel[key]
+            if isinstance(n, bool) or not isinstance(n, int) \
+                    or not 1 <= n <= 100:
+                raise AppConfigError(
+                    f"config.json: agent_panel.{key} must be an integer "
+                    "between 1 and 100")
+    if "verdict_source" in panel and panel["verdict_source"] not in \
+            ("macro", "chair"):
+        raise AppConfigError(
+            'config.json: agent_panel.verdict_source must be "macro" or '
+            '"chair"')
+    app.agent_panel_settings = dict(panel)
+
+    outlook = raw.get("market_outlook", {})
+    _check_keys(outlook, _KNOWN_MARKET_OUTLOOK, '"market_outlook"')
+    for flag in ("enabled", "telegram"):
+        if flag in outlook and not isinstance(outlook[flag], bool):
+            raise AppConfigError(
+                f"config.json: market_outlook.{flag} must be true or false")
+    for key in ("max_per_day", "candles_bars"):
+        if key in outlook:
+            value = outlook[key]
+            if isinstance(value, bool) or not isinstance(value, int) or value < 1:
+                raise AppConfigError(
+                    f"config.json: market_outlook.{key} must be a positive integer")
+    app.market_outlook_settings.update(outlook)
+
+    slog = raw.get("signal_log", {})
+    _check_keys(slog, _KNOWN_SIGNAL_LOG, '"signal_log"')
+    if "enabled" in slog and not isinstance(slog["enabled"], bool):
+        raise AppConfigError(
+            "config.json: signal_log.enabled must be true or false")
+    if "path" in slog and (not isinstance(slog["path"], str)
+                           or not slog["path"]):
+        raise AppConfigError(
+            "config.json: signal_log.path must be a non-empty string")
+    if ("log_blocked_scans" in slog
+            and not isinstance(slog["log_blocked_scans"], bool)):
+        raise AppConfigError(
+            "config.json: signal_log.log_blocked_scans must be true or false")
+    app.signal_log_settings = dict(slog)
+
+    level = raw.get("log_level", app.log_level)
+    if level not in _VALID_LOG_LEVELS:
+        raise AppConfigError(
+            f'config.json: log_level "{level}" not in {_VALID_LOG_LEVELS}')
+    app.log_level = level
+    return app
+
+
+# ── Process-wide singleton ────────────────────────────────────────────
+
+_cached: Optional[AppConfig] = None
+
+
+def get_app_config() -> AppConfig:
+    global _cached
+    if _cached is None:
+        _cached = load_app_config()
+    return _cached
+
+
+def set_app_config(app: Optional[AppConfig]) -> None:
+    """Override the process-wide AppConfig (tests/experiments).
+    Pass None to re-enable lazy loading from CONFIG_PATH."""
+    global _cached
+    _cached = app
+
+
+# ── Resolution helpers: env var → config.json → default ───────────────
+
+def telegram_token(app: Optional[AppConfig] = None) -> str:
+    app = app or get_app_config()
+    return os.environ.get("TELEGRAM_TOKEN") or app.telegram_token
+
+
+def telegram_chat_id(app: Optional[AppConfig] = None) -> str:
+    app = app or get_app_config()
+    return os.environ.get("TELEGRAM_CHAT_ID") or app.telegram_chat_id
+
+
+def telegram_channel_id(app: Optional[AppConfig] = None) -> str:
+    """Env set-but-empty ("") means "disable the feed" and beats JSON,
+    so this checks None-ness, not truthiness."""
+    app = app or get_app_config()
+    env = os.environ.get("TELEGRAM_CHANNEL_ID")
+    return env if env is not None else app.telegram_channel_id
+
+
+# Per-provider env-var overrides for API keys (env → config.json → "").
+_PROVIDER_ENV_KEYS = {
+    "twelve_data": "TWELVE_DATA_API_KEY",
+    "tiingo": "TIINGO_API_KEY",
+}
+
+
+def provider_api_key(
+    app: Optional[AppConfig] = None, provider: Optional[str] = None
+) -> str:
+    """API key for `provider` (default: the ACTIVE provider). The env
+    override is per-provider (TWELVE_DATA_API_KEY applies only to
+    twelve_data, TIINGO_API_KEY only to tiingo)."""
+    app = app or get_app_config()
+    name = provider or app.active_provider
+    env_key = _PROVIDER_ENV_KEYS.get(name)
+    if env_key:
+        env = os.environ.get(env_key)
+        if env:
+            return env
+    block = app.provider_settings.get(name, {})
+    return str(block.get("api_key", ""))
+
+
+# ── Oanda resolution helpers (env → config.json → default) ───────────
+
+def oanda_account_id(app: Optional[AppConfig] = None) -> str:
+    app = app or get_app_config()
+    return os.environ.get("OANDA_ACCOUNT_ID") or str(
+        app.oanda_settings.get("account_id", ""))
+
+
+def oanda_api_token(app: Optional[AppConfig] = None) -> str:
+    app = app or get_app_config()
+    return os.environ.get("OANDA_API_TOKEN") or str(
+        app.oanda_settings.get("api_token", ""))
+
+
+def oanda_base_url(app: Optional[AppConfig] = None) -> str:
+    app = app or get_app_config()
+    return os.environ.get("OANDA_BASE_URL") or str(
+        app.oanda_settings.get("base_url", "https://api-fxpractice.oanda.com"))
+
+
+# ── DeepSeek resolution helper (env → config.json → "") ──────────────
+
+def deepseek_api_key(app: Optional[AppConfig] = None) -> str:
+    app = app or get_app_config()
+    return os.environ.get("DEEPSEEK_API_KEY") or str(
+        app.deepseek_settings.get("api_key", ""))
+
+
+# ── AI market-outlook resolution helper ──────────────────────────────
+
+def market_outlook_enabled(app: Optional[AppConfig] = None) -> bool:
+    """Whether the advisory outlook feature is enabled (env wins)."""
+    env = os.environ.get("MARKET_OUTLOOK_ENABLED")
+    if env:
+        return env.strip().lower() in ("1", "true", "yes", "on")
+    app = app or get_app_config()
+    return bool(app.market_outlook_settings.get("enabled", False))
+
+
+# ── Signal-log resolution helpers (env → config.json → default) ──────
+
+def signal_log_enabled(app: Optional[AppConfig] = None) -> bool:
+    """Default ON — signal history is the platform's audit trail."""
+    env = os.environ.get("SIGNAL_LOG_ENABLED")
+    if env:
+        return env.strip().lower() in ("1", "true", "yes", "on")
+    app = app or get_app_config()
+    return bool(app.signal_log_settings.get("enabled", True))
+
+
+def signal_log_blocked_scans(app: Optional[AppConfig] = None) -> bool:
+    """Whether weekend/day-filter early returns are logged (as minimal
+    rows without news/candles). Default ON — they are liveness evidence;
+    turn off to keep the history table to real scans only."""
+    env = os.environ.get("SIGNAL_LOG_BLOCKED_SCANS")
+    if env:
+        return env.strip().lower() in ("1", "true", "yes", "on")
+    app = app or get_app_config()
+    return bool(app.signal_log_settings.get("log_blocked_scans", True))
+
+
+def signal_log_path(app: Optional[AppConfig] = None) -> str:
+    """Absolute path to the SQLite signal-history DB. A relative config
+    value resolves against the repo root so CWD never matters."""
+    env = os.environ.get("SIGNAL_LOG_PATH")
+    if env:
+        path = env
+    else:
+        app = app or get_app_config()
+        path = str(app.signal_log_settings.get("path", "logs/signals.db"))
+    if not os.path.isabs(path):
+        path = os.path.join(os.path.dirname(os.path.abspath(__file__)), path)
+    return path
+
+
+# ── PlatformConfig overlay ────────────────────────────────────────────
+
+def apply_pair_overrides(
+    cfg: PlatformConfig, app: Optional[AppConfig] = None
+) -> PlatformConfig:
+    """Stamp deployment settings from config.json onto a PlatformConfig.
+
+    Called by the pair factories BEFORE their validated per-timeframe
+    deltas, so e.g. the M5 hour-9 append lands on top of the JSON base.
+    The pair's base blocked_hours_utc is set on BOTH mirrored copies
+    (xau_strategy + backtest) so they cannot drift.
+    """
+    app = app or get_app_config()
+    overrides = app.pair_overrides.get(cfg.symbol, {})
+    if "blocked_hours_utc" in overrides:
+        hours = list(overrides["blocked_hours_utc"])
+        cfg.xau_strategy.blocked_hours_utc = hours
+        cfg.backtest.blocked_hours_utc = list(hours)
+    cfg.scan_pairs = list(app.scan_pairs)
+    cfg.scan_timeframes = list(app.scan_timeframes)
+    return cfg

backtest/diagnose_filters.py

@@ -0,0 +1,629 @@
+"""
+ForexBot2 — Filter Rejection Diagnostic
+========================================
+Replays the same backtest loop as gpu_backtest.py but instead of
+collecting trades, counts how many M15 bars are rejected by each
+gate in the signal pipeline.
+
+Produces a summary table showing where potential trades are being
+lost, helping tune filter thresholds.
+
+Usage:
+    python backtest/diagnose_filters.py --pair XAUUSD --days 365
+    python backtest/diagnose_filters.py --pair XAUUSD --days 365 --sample-every 1
+"""
+
+from __future__ import annotations
+
+import argparse
+import logging
+import sys
+import time
+from collections import Counter, defaultdict
+from datetime import datetime, timedelta, timezone
+from pathlib import Path
+from typing import Dict, List, Optional, Tuple
+
+import numpy as np
+import pandas as pd
+
+# Ensure project root on sys.path.
+PROJECT_ROOT = Path(__file__).resolve().parent.parent
+if str(PROJECT_ROOT) not in sys.path:
+    sys.path.insert(0, str(PROJECT_ROOT))
+
+from config import (  # noqa: E402
+    PAIR_INFO,
+    PlatformConfig,
+    SignalStrength,
+    Timeframe,
+)
+from signal_aggregator import SignalAggregator, SignalReport  # noqa: E402
+
+# Reuse GPU backtest infrastructure (data loading, indicator pre-compute, etc.)
+from gpu_backtest import (  # noqa: E402
+    BAR_SECONDS,
+    GPUTechnicalEngine,
+    PAIR_ALIASES,
+    BacktestConfig,
+    IndicatorCache,
+    fetch_history,
+    precompute_indicators,
+    _is_xau_blocked_by_calendar,
+)
+from providers import make_data_provider  # noqa: E402
+
+logger = logging.getLogger("diagnose_filters")
+
+
+# ═════════════════════════════════════════════════════════════════════
+#  OUTCOME CLASSIFICATION
+# ═════════════════════════════════════════════════════════════════════
+
+def classify_bar(
+    report: SignalReport,
+    is_xau: bool,
+) -> Tuple[str, List[str]]:
+    """Classify a bar's outcome and list which filters killed it.
+
+    Returns:
+        (outcome, [filter_names])
+
+    Outcomes:
+        "TRADE"           — quality C+ and trade_setup exists
+        "QUALITY_OK_NO_SETUP" — quality C+ but trade_setup is None
+        "LOW_QUALITY"     — signal reached BUY/SELL but quality D or F
+        "NEUTRAL"         — signal stayed NEUTRAL (score didn't reach +/-30)
+
+    Filter names are extracted from report.warnings.
+    """
+    filters_hit: List[str] = []
+
+    # Known filter tags, matched against the exact warning strings the
+    # aggregator emits (refreshed 2026-07-30 for gates 6d-6i). ORDER
+    # MATTERS: most-specific prefix first, and each warning maps to at
+    # most ONE tag (break on first hit) — substring families like
+    # "M5 FILTER: SELL quality" vs "M5 FILTER" and "TREND EXHAUSTION"
+    # vs "EXHAUSTION" would otherwise double-count.
+    filter_tags = [
+        # M5 noise gates (6i, _apply_m5_filters)
+        ("M5 FILTER: |score|",            "M5_SCORE"),
+        ("M5 FILTER: no MTF agreement",   "M5_MTF"),
+        ("M5 FILTER: no confirmation TF", "M5_CONF"),
+        ("M5 FILTER: SELL quality",       "M5_SELL_QUALITY"),
+        ("M5 FILTER: SELL blocked — RSI", "M5_SELL_RSI"),
+        ("M5 FILTER: '",                  "M5_VOL"),
+        ("M5 FILTER",                     "M5_OTHER"),
+        # XAU SELL gates (6e/6g/6g2/6g3)
+        ("SELL FILTER: quality",          "SELL_QUALITY"),
+        ("SELL FILTER: volatility",       "SELL_VOL"),
+        ("SELL FILTER: regime",           "SELL_REGIME"),
+        ("UTC entry blocked for shorts",  "SELL_HOUR"),
+        # Hard gates
+        ("MTF OPPOSITION",                "MTF_OPPOSITION"),
+        ("EXHAUSTED TREND FILTER",        "EXHAUSTED_TREND"),
+        ("BIG CANDLE FILTER",             "BIG_CANDLE"),
+        ("WEEKEND FILTER",                "WEEKEND_FILTER"),
+        ("SESSION FILTER",                "SESSION_FILTER"),
+        ("DAY FILTER",                    "DAY_FILTER"),
+        ("LOW ATR FILTER",                "LOW_ATR_FILTER"),
+        ("REGIME FILTER",                 "REGIME_FILTER"),
+        # Soft penalties / context warnings
+        ("TREND EXHAUSTION",              "TREND_EXHAUSTION"),
+        ("EXHAUSTION",                    "EXHAUSTION_PENALTY"),
+        ("NO PULLBACK",                   "NO_PULLBACK_PENALTY"),
+        ("Weak pullback",                 "WEAK_PULLBACK_PENALTY"),
+        ("Extended — RSI",                "RSI_EXTENDED_PENALTY"),
+        ("Counter-trend",                 "COUNTER_TREND_PENALTY"),
+        ("EXTREME volatility",            "EXTREME_VOL_PENALTY"),
+        ("NEAR EMA200",                   "NEAR_EMA200_PENALTY"),
+        ("FAR FROM EMA200",               "FAR_EMA200_PENALTY"),
+        ("EVENT RISK",                    "EVENT_RISK"),
+        ("Timeframes CONFLICT",           "MTF_CONFLICT"),
+    ]
+
+    for w in report.warnings:
+        for tag, name in filter_tags:
+            if tag in w:
+                filters_hit.append(name)
+                break
+
+    # Determine outcome
+    non_neutral = report.final_signal not in (SignalStrength.NEUTRAL,)
+    quality_ok = report.signal_quality in ("A+", "A", "B", "C")
+
+    if quality_ok and report.trade_setup is not None:
+        return "TRADE", filters_hit
+    elif quality_ok and report.trade_setup is None:
+        return "QUALITY_OK_NO_SETUP", filters_hit
+    elif non_neutral and not quality_ok:
+        return "LOW_QUALITY", filters_hit
+    else:
+        return "NEUTRAL", filters_hit
+
+
+# ═════════════════════════════════════════════════════════════════════
+#  SCORE DISTRIBUTION TRACKER
+# ═════════════════════════════════════════════════════════════════════
+
+class ScoreTracker:
+    """Tracks score/confidence distributions for analysis."""
+
+    def __init__(self):
+        self.scores: List[float] = []
+        self.confidences: List[float] = []
+        self.abs_scores: List[float] = []
+        # How many bars had score in various ranges
+        self.score_bins = Counter()
+        self.conf_bins = Counter()
+        # Track "near miss" bars — score close to thresholds
+        self.near_miss_score: int = 0  # score 20-29 (almost BUY/SELL)
+        self.near_miss_conf: int = 0   # confidence 45-54 (almost passing)
+
+    def record(self, score: float, confidence: float):
+        self.scores.append(score)
+        self.confidences.append(confidence)
+        self.abs_scores.append(abs(score))
+
+        # Score bins
+        a = abs(score)
+        if a < 10:
+            self.score_bins["0-9"] += 1
+        elif a < 20:
+            self.score_bins["10-19"] += 1
+        elif a < 30:
+            self.score_bins["20-29"] += 1
+            self.near_miss_score += 1
+        elif a < 40:
+            self.score_bins["30-39"] += 1
+        elif a < 50:
+            self.score_bins["40-49"] += 1
+        else:
+            self.score_bins["50+"] += 1
+
+        # Confidence bins
+        if confidence < 30:
+            self.conf_bins["0-29"] += 1
+        elif confidence < 45:
+            self.conf_bins["30-44"] += 1
+        elif confidence < 55:
+            self.conf_bins["45-54"] += 1
+            self.near_miss_conf += 1
+        elif confidence < 65:
+            self.conf_bins["55-64"] += 1
+        elif confidence < 75:
+            self.conf_bins["65-74"] += 1
+        else:
+            self.conf_bins["75+"] += 1
+
+
+# ═════════════════════════════════════════════════════════════════════
+#  MAIN DIAGNOSTIC LOOP
+# ═════════════════════════════════════════════════════════════════════
+
+def run_diagnosis(
+    pair: str,
+    start: datetime,
+    end: datetime,
+    sample_every: int = 4,
+    use_cache: bool = True,
+    primary_tf: Timeframe = Timeframe.M15,
+    confirm_tfs: Optional[List[Timeframe]] = None,
+    warmup_bars: int = 250,
+) -> None:
+    if confirm_tfs is None:
+        confirm_tfs = [Timeframe.H1, Timeframe.H4]
+
+    pair_info = PAIR_INFO[pair]
+    display = pair_info["display"]
+    is_xau = display == "XAU/USD"
+
+    platform_cfg = PlatformConfig(
+        symbol=pair,
+        display_name=display,
+        pip_value=pair_info["pip_value"],
+        primary_timeframe=primary_tf,
+        confirmation_timeframes=list(confirm_tfs),
+    )
+
+    provider = make_data_provider(platform_cfg)
+
+    # ── Load data ──
+    timeframes = [primary_tf] + list(confirm_tfs)
+    pad_days = max(60, warmup_bars // 4)
+    fetch_start = start - timedelta(days=pad_days)
+
+    dfs_full: Dict[Timeframe, pd.DataFrame] = {}
+    for tf in timeframes:
+        df = fetch_history(
+            provider, pair, tf,
+            start=fetch_start, end=end,
+            use_cache=use_cache,
+        )
+        if df.empty:
+            raise RuntimeError(f"No data for {pair} {tf.value}")
+        dfs_full[tf] = df
+
+    # ── Pre-compute indicators ──
+    t0 = time.perf_counter()
+    caches: Dict[str, IndicatorCache] = {
+        tf.value: precompute_indicators(dfs_full[tf], platform_cfg.technical, gpu_log=True)
+        for tf in timeframes
+    }
+    logger.info(f"Indicator pre-compute: {time.perf_counter() - t0:.2f}s")
+
+    # ── Build aggregator with GPU engine ──
+    aggregator = SignalAggregator(platform_cfg)
+    aggregator.tech_engine = GPUTechnicalEngine(platform_cfg.technical, caches)
+
+    # ── Walk primary TF ──
+    primary_df = dfs_full[primary_tf]
+    bt_start_ts = pd.Timestamp(start)
+    bt_end_ts = pd.Timestamp(end)
+
+    first_eligible = max(
+        warmup_bars,
+        int(primary_df.index.searchsorted(bt_start_ts)),
+    )
+    last_pos = min(
+        int(primary_df.index.searchsorted(bt_end_ts, side="right")),
+        len(primary_df),
+    )
+
+    primary_dur = pd.Timedelta(seconds=BAR_SECONDS[primary_tf])
+
+    # Counters
+    total_bars_in_window = 0
+    total_bars_sampled = 0
+    outcome_counts = Counter()
+    filter_counts = Counter()
+    # Filters that killed bars that were otherwise directional (non-NEUTRAL score)
+    directional_filter_counts = Counter()
+    # Pre-pipeline rejections (XAU calendar filters applied before analyze_from_dataframes)
+    pre_pipeline = Counter()
+
+    # Track signal directions
+    signal_dir_counts = Counter()
+    quality_counts = Counter()
+
+    # Score tracker
+    tracker = ScoreTracker()
+
+    # Track "would-have-been" trades: bars where the primary score was
+    # directional but got killed by a specific filter
+    filter_kill_breakdown = defaultdict(Counter)
+
+    logger.info(
+        f"Diagnosing {display} {primary_tf.value}: "
+        f"bars {first_eligible}-{last_pos} (every {sample_every}th bar)"
+    )
+
+    t_loop = time.perf_counter()
+    for pos in range(first_eligible, last_pos, sample_every):
+        total_bars_in_window += 1
+        ts = primary_df.index[pos]
+        decision_time = ts + primary_dur
+
+        # ── XAU calendar pre-filter (same as gpu_backtest.py) ──
+        if is_xau:
+            block = _is_xau_blocked_by_calendar(ts, platform_cfg)
+            if block:
+                pre_pipeline[block] += 1
+                continue
+
+        total_bars_sampled += 1
+
+        # ── Build sliced dfs (same logic as gpu_backtest.py) ──
+        slice_dfs: Dict[Timeframe, pd.DataFrame] = {}
+        for tf in timeframes:
+            full = dfs_full[tf]
+            bar_dur = pd.Timedelta(seconds=BAR_SECONDS[tf])
+            cutoff = decision_time - bar_dur
+            cut = int(full.index.searchsorted(cutoff, side="right"))
+            if cut == 0:
+                continue
+            slice_dfs[tf] = full.iloc[:cut]
+
+        if primary_tf not in slice_dfs:
+            continue
+
+        # ── Run the full signal pipeline ──
+        report: SignalReport = aggregator.analyze_from_dataframes(
+            symbol=pair,
+            dfs=slice_dfs,
+            context=None,
+            use_news=False,
+        )
+
+        # Record score/confidence
+        tracker.record(report.final_score, report.final_confidence)
+
+        # Signal direction
+        signal_dir_counts[report.final_signal.name] += 1
+        quality_counts[report.signal_quality] += 1
+
+        # Classify
+        outcome, filters_hit = classify_bar(report, is_xau)
+        outcome_counts[outcome] += 1
+
+        for f in filters_hit:
+            filter_counts[f] += 1
+
+        # If the primary score was directional (>= 15 or <= -15), the bar
+        # had some potential — track what killed it
+        if abs(report.primary_score) >= 15:
+            for f in filters_hit:
+                directional_filter_counts[f] += 1
+
+        # For bars that got quality D/F but had a directional signal,
+        # track which filter warnings appeared
+        if outcome in ("LOW_QUALITY", "QUALITY_OK_NO_SETUP"):
+            for f in filters_hit:
+                filter_kill_breakdown[outcome][f] += 1
+
+    elapsed = time.perf_counter() - t_loop
+
+    # ═════════════════════════════════════════════════════════════════
+    #  PRINT RESULTS
+    # ═════════════════════════════════════════════════════════════════
+
+    bar = "=" * 80
+    print(f"\n{bar}")
+    print(f"  FILTER REJECTION DIAGNOSTIC — {display}")
+    print(f"  {start.date()} to {end.date()} | {primary_tf.value} "
+          f"(sampled every {sample_every}th bar)")
+    print(f"{bar}")
+
+    # Scale counts if sampling
+    scale_note = ""
+    if sample_every > 1:
+        scale_note = f"  (sampled {total_bars_sampled} of ~{total_bars_in_window * sample_every} eligible bars)"
+    print(f"\n  Total bars in backtest window: {total_bars_in_window}")
+    if pre_pipeline:
+        for reason, count in sorted(pre_pipeline.items()):
+            print(f"  Pre-pipeline rejection ({reason}): {count}")
+    print(f"  Bars analyzed: {total_bars_sampled}{scale_note}")
+    print(f"  Analysis took: {elapsed:.1f}s "
+          f"({total_bars_sampled / max(elapsed, 0.001):.0f} bars/s)")
+
+    # ── Outcome breakdown ──
+    print(f"\n{'-' * 80}")
+    print("  OUTCOME BREAKDOWN")
+    print(f"{'-' * 80}")
+    outcome_order = ["TRADE", "QUALITY_OK_NO_SETUP", "LOW_QUALITY", "NEUTRAL"]
+    outcome_labels = {
+        "TRADE":               "Actual trades (quality C+ & setup exists)",
+        "QUALITY_OK_NO_SETUP": "Quality C+ but NO trade setup generated",
+        "LOW_QUALITY":         "Directional signal but quality D/F",
+        "NEUTRAL":             "NEUTRAL (score didn't reach +/-30 or conf < 55)",
+    }
+    for o in outcome_order:
+        count = outcome_counts.get(o, 0)
+        pct = count / max(total_bars_sampled, 1) * 100
+        label = outcome_labels.get(o, o)
+        print(f"  {count:6d} ({pct:5.1f}%)  {label}")
+
+    # ── Signal direction breakdown ──
+    print(f"\n{'-' * 80}")
+    print("  SIGNAL DIRECTION BREAKDOWN")
+    print(f"{'-' * 80}")
+    for name in ["STRONG_BUY", "BUY", "NEUTRAL", "SELL", "STRONG_SELL"]:
+        count = signal_dir_counts.get(name, 0)
+        pct = count / max(total_bars_sampled, 1) * 100
+        print(f"  {count:6d} ({pct:5.1f}%)  {name}")
+
+    # ── Quality grade breakdown ──
+    print(f"\n{'-' * 80}")
+    print("  QUALITY GRADE BREAKDOWN")
+    print(f"{'-' * 80}")
+    for grade in ["A+", "A", "B", "C", "D", "F"]:
+        count = quality_counts.get(grade, 0)
+        pct = count / max(total_bars_sampled, 1) * 100
+        print(f"  {count:6d} ({pct:5.1f}%)  Grade {grade}")
+
+    # ── Filter hit counts (all bars) ──
+    print(f"\n{'-' * 80}")
+    print("  FILTER HIT COUNTS (all bars, a bar can hit multiple filters)")
+    print(f"{'-' * 80}")
+    if filter_counts:
+        for name, count in filter_counts.most_common():
+            pct = count / max(total_bars_sampled, 1) * 100
+            print(f"  {count:6d} ({pct:5.1f}%)  {name}")
+    else:
+        print("  (no filters triggered)")
+
+    # ── Filters that killed directional bars (primary_score >= 15) ──
+    print(f"\n{'-' * 80}")
+    print("  FILTERS KILLING DIRECTIONAL BARS (|primary_score| >= 15)")
+    print(f"{'-' * 80}")
+    if directional_filter_counts:
+        for name, count in directional_filter_counts.most_common():
+            print(f"  {count:6d}  {name}")
+    else:
+        print("  (none)")
+
+    # ── Filter breakdown for LOW_QUALITY bars ──
+    if filter_kill_breakdown.get("LOW_QUALITY"):
+        print(f"\n{'-' * 80}")
+        print("  FILTERS ON LOW_QUALITY BARS (had BUY/SELL but quality D/F)")
+        print(f"{'-' * 80}")
+        for name, count in filter_kill_breakdown["LOW_QUALITY"].most_common():
+            print(f"  {count:6d}  {name}")
+
+    # ── Filter breakdown for QUALITY_OK_NO_SETUP bars ──
+    if filter_kill_breakdown.get("QUALITY_OK_NO_SETUP"):
+        print(f"\n{'-' * 80}")
+        print("  FILTERS ON QUALITY_OK_NO_SETUP BARS (quality C+ but no trade setup)")
+        print(f"{'-' * 80}")
+        for name, count in filter_kill_breakdown["QUALITY_OK_NO_SETUP"].most_common():
+            print(f"  {count:6d}  {name}")
+
+    # ── Score distribution ──
+    print(f"\n{'-' * 80}")
+    print("  SCORE DISTRIBUTION (|score|)")
+    print(f"{'-' * 80}")
+    for bin_name in ["0-9", "10-19", "20-29", "30-39", "40-49", "50+"]:
+        count = tracker.score_bins.get(bin_name, 0)
+        pct = count / max(total_bars_sampled, 1) * 100
+        marker = " <-- near miss (almost BUY/SELL)" if bin_name == "20-29" else ""
+        print(f"  {count:6d} ({pct:5.1f}%)  |score| {bin_name}{marker}")
+
+    # ── Confidence distribution ──
+    print(f"\n{'-' * 80}")
+    print("  CONFIDENCE DISTRIBUTION")
+    print(f"{'-' * 80}")
+    for bin_name in ["0-29", "30-44", "45-54", "55-64", "65-74", "75+"]:
+        count = tracker.conf_bins.get(bin_name, 0)
+        pct = count / max(total_bars_sampled, 1) * 100
+        marker = " <-- near miss (almost passing)" if bin_name == "45-54" else ""
+        print(f"  {count:6d} ({pct:5.1f}%)  confidence {bin_name}{marker}")
+
+    # ── Near misses ──
+    print(f"\n{'-' * 80}")
+    print("  NEAR MISSES")
+    print(f"{'-' * 80}")
+    print(f"  {tracker.near_miss_score:6d}  Bars with |score| 20-29 (threshold is 30)")
+    print(f"  {tracker.near_miss_conf:6d}  Bars with confidence 45-54 (threshold is 55)")
+
+    # ── Summary statistics ──
+    if tracker.scores:
+        scores_arr = np.array(tracker.scores)
+        conf_arr = np.array(tracker.confidences)
+        print(f"\n{'-' * 80}")
+        print("  SCORE/CONFIDENCE STATISTICS")
+        print(f"{'-' * 80}")
+        print(f"  Score:      mean={np.mean(scores_arr):+.1f}  "
+              f"median={np.median(scores_arr):+.1f}  "
+              f"std={np.std(scores_arr):.1f}  "
+              f"min={np.min(scores_arr):+.1f}  "
+              f"max={np.max(scores_arr):+.1f}")
+        print(f"  Confidence: mean={np.mean(conf_arr):.1f}  "
+              f"median={np.median(conf_arr):.1f}  "
+              f"std={np.std(conf_arr):.1f}  "
+              f"min={np.min(conf_arr):.1f}  "
+              f"max={np.max(conf_arr):.1f}")
+
+    # ── Blocked hours analysis (for XAU) ──
+    if is_xau:
+        blocked = set(platform_cfg.xau_strategy.blocked_hours_utc)
+        all_hours = set(range(24))
+        open_hours = sorted(all_hours - blocked)
+        print(f"\n{'-' * 80}")
+        print("  XAU HOUR/DAY CONFIG")
+        print(f"{'-' * 80}")
+        print(f"  Blocked hours:  {sorted(blocked)}")
+        print(f"  Open hours:     {open_hours}")
+        print(f"  Allowed days:   {platform_cfg.xau_strategy.allowed_weekdays} "
+              f"(0=Mon..4=Fri)")
+        print(f"  Open fraction:  {len(open_hours)/24:.0%} of hours x "
+              f"{len(platform_cfg.xau_strategy.allowed_weekdays)}/5 weekdays = "
+              f"{len(open_hours)/24 * len(platform_cfg.xau_strategy.allowed_weekdays)/5:.0%} of trading time")
+
+    # ── Interpretation / key takeaways ──
+    print(f"\n{bar}")
+    print("  KEY TAKEAWAYS")
+    print(f"{bar}")
+
+    neutral_pct = outcome_counts.get("NEUTRAL", 0) / max(total_bars_sampled, 1) * 100
+    low_q_pct = outcome_counts.get("LOW_QUALITY", 0) / max(total_bars_sampled, 1) * 100
+    trade_count = outcome_counts.get("TRADE", 0)
+
+    print(f"  - {neutral_pct:.1f}% of bars never reach a directional signal (NEUTRAL)")
+    print(f"  - {low_q_pct:.1f}% of bars get a BUY/SELL but confidence/quality too low")
+    print(f"  - {trade_count} bars produce actual trades")
+
+    if filter_counts:
+        top3 = filter_counts.most_common(3)
+        print(f"  - Top rejection filters:")
+        for name, count in top3:
+            print(f"      {name}: {count} bars hit")
+
+    if tracker.near_miss_score > 50:
+        print(f"  - SIGNIFICANT: {tracker.near_miss_score} bars had |score| 20-29 "
+              f"(lowering threshold from 30 to 25 would capture many)")
+
+    if tracker.near_miss_conf > 50:
+        print(f"  - SIGNIFICANT: {tracker.near_miss_conf} bars had confidence 45-54 "
+              f"(lowering threshold from 55 to 50 would capture many)")
+
+    print(f"\n{bar}\n")
+
+
+# ═════════════════════════════════════════════════════════════════════
+#  CLI
+# ═════════════════════════════════════════════════════════════════════
+
+def _resolve_pair(s: str) -> str:
+    s_up = s.upper().strip()
+    if s_up in PAIR_ALIASES:
+        return PAIR_ALIASES[s_up]
+    if s_up in PAIR_INFO:
+        return s_up
+    raise SystemExit(f"Unknown pair '{s}'. Valid: {', '.join(sorted(PAIR_ALIASES))}")
+
+
+def _resolve_tf(s: str) -> Timeframe:
+    for tf in Timeframe:
+        if tf.value == s:
+            return tf
+    raise SystemExit(f"Unknown timeframe '{s}'")
+
+
+def main(argv: Optional[List[str]] = None) -> int:
+    p = argparse.ArgumentParser(
+        description="Diagnose which filters reject the most bars in the backtest pipeline",
+        formatter_class=argparse.ArgumentDefaultsHelpFormatter,
+    )
+    p.add_argument("--pair", required=True, help="XAUUSD | GOLD")
+    p.add_argument("--days", type=int, default=365, help="Days of history")
+    p.add_argument("--start", help="ISO date YYYY-MM-DD (overrides --days)")
+    p.add_argument("--end", help="ISO date YYYY-MM-DD (defaults to today)")
+    p.add_argument("--primary-tf", default="15min", help="Primary TF")
+    p.add_argument("--confirm-tfs", default="1h,4h", help="Comma-separated confirmation TFs")
+    p.add_argument("--warmup-bars", type=int, default=250)
+    p.add_argument("--sample-every", type=int, default=4,
+                   help="Analyze every Nth bar (1=all, 4=25%% sample for speed)")
+    p.add_argument("--no-cache", action="store_true", help="Skip parquet cache")
+    p.add_argument("--log-level", default="INFO")
+    args = p.parse_args(argv)
+
+    logging.basicConfig(
+        level=args.log_level.upper(),
+        format="%(asctime)s [%(levelname)s] %(name)s: %(message)s",
+    )
+
+    pair = _resolve_pair(args.pair)
+    primary_tf = _resolve_tf(args.primary_tf)
+    confirm_tfs = [_resolve_tf(s.strip()) for s in args.confirm_tfs.split(",") if s.strip()]
+
+    end = (
+        datetime.fromisoformat(args.end).replace(tzinfo=timezone.utc)
+        if args.end else datetime.now(timezone.utc)
+    )
+    if args.start:
+        start = datetime.fromisoformat(args.start).replace(tzinfo=timezone.utc)
+    else:
+        start = end - timedelta(days=args.days)
+
+    logger.info(
+        f"Diagnosing {PAIR_INFO[pair]['display']} | "
+        f"Window: {start.date()} -> {end.date()} | "
+        f"Sample: every {args.sample_every}th bar"
+    )
+
+    run_diagnosis(
+        pair=pair,
+        start=start,
+        end=end,
+        sample_every=args.sample_every,
+        use_cache=not args.no_cache,
+        primary_tf=primary_tf,
+        confirm_tfs=confirm_tfs,
+        warmup_bars=args.warmup_bars,
+    )
+
+    return 0
+
+
+if __name__ == "__main__":
+    sys.exit(main())

backtest/instructions_for_backtest.txt

@@ -0,0 +1,59 @@
+Note down entery and exit Why we took trade?
+What were each indicator signal on that specific time when we took trade
+How many indicators agreed at trade time
+Always backtest using engine.py and signal_aggregator.py
+Dont take trades on weekends and Fridays
+Keep TP small 1.3 Reward ratio
+At the end of test make table of results and print all values you observed.
+Trade context
+
+Time of day and trading session (Asian/London/NY/overlap)
+Day of week (track Mon/Tue/Wed/Thu performance separately)
+Market regime at entry: trending, ranging, or choppy (use ATR or ADX)
+Volatility level (ATR value or ATR percentile vs recent average)
+Pair/instrument traded
+News proximity (high-impact news within 30 min before/after entry)
+
+Trade execution details
+
+Spread at entry in pips
+Slippage on entry and exit (planned vs actual fill)
+Time in trade (minutes/hours from entry to exit)
+Maximum Adverse Excursion (MAE): how far against you it went
+Maximum Favorable Excursion (MFE): how far in profit before pullback
+
+Risk metrics
+
+Max consecutive losses (streak)
+Max consecutive wins (streak)
+Expectancy: (Win% x Avg Win) minus (Loss% x Avg Loss)
+Expectancy per hour in trade
+Recovery factor: Total R divided by Max DD
+
+Indicator/signal analysis
+
+Each indicator's standalone win rate when it agreed with the trade
+Confluence level performance (2 vs 3 vs 4+ indicators agreeing)
+Which indicator disagreed on losing trades (find the "canary")
+
+Decision quality
+
+Rate each setup A/B/C at entry, before knowing outcome
+Verify A-grade setups actually win more often
+Track system trades vs discretionary deviations separately
+
+Process notes
+
+Screenshot or chart snapshot at entry and exit
+Emotional state (calm, FOMO, revenge, tired)
+What would have happened if you held longer or exited sooner
+
+Aggregate analysis at end of backtest
+
+Equity curve, not just total R
+Performance breakdown by session, day, volatility regime
+Win rate over rolling 20-trade windows (check for edge decay)
+
+Critical extra
+"Do not trade" log: setups flagged but skipped, and what happened after
+

backtest/research/__init__.py

@@ -0,0 +1,5 @@
+"""Research tooling: calendar sourcing/store and edge-screening utilities.
+
+Nothing here is imported by the live bot; these modules support
+backtesting and edge validation only.
+"""

backtest/research/ai_signal_enrichment_2026-08-15.md

@@ -0,0 +1,116 @@
+# AI signal enrichment — analysis + build record (2026-08-15)
+
+Owner questions:
+1. If the AI panel also received indicator values and (limited) candle data, would the system be more accurate?
+2. If AI reviewed every candle alongside the local score system, would prediction accuracy justify the token burn — or is there a middle ground?
+
+## What the panel already received (pre-change baseline)
+
+The four personas (`agent_panel.py`) already got a *distilled* picture:
+per-indicator signal+confidence, MTF signals/scores, key levels,
+exhaustion count, full trade-setup math, warnings, headlines/events/
+sentiment. Missing: raw indicator **values** and any **candle** data.
+
+Cost guardrails already in place: 5 requests/panel, `max_panels_per_day`
+20 (≤100 requests/day), 3600 s reuse-TTL, per-bar exact cache, shared
+500 req/day DeepSeek cap, static prompts on the ~50× prefix cache.
+
+## Evidence used
+
+- **LLM is strong at text, weak at raw numerics** (measured in-repo,
+  `llm_news_eval_2026-08-06.md`): headline sentiment 100% (78/78) vs
+  keyword scorer 31%; but ~11% of event-direction calls misread close
+  numeric comparisons until code pre-computed the comparison
+  (`vs_forecast`) and the model only mapped it to doctrine.
+- **The book any filter would act on** (M5 canonical,
+  `oanda_m5_12mo_065_nopb_20260814.csv`): 577 trades, +106.82R,
+  PF 1.51, win rate 52.9%; A-grade 59.5%, B-grade 51.4% (+67.8R of the
+  total), C-grade 43.8% (−0.23R, 16 trades). Average edge ≈ 0.185R/trade.
+
+## Answer 1 — enrich the existing panel? YES (cheap, safe)
+
+- A compact candle is ~15–25 tokens; 20 primary-TF + 10 confirm-TF bars
+  ≈ 500–1,500 tokens per persona call → cents/day at the 20-panel cap.
+  **Tokens are not the constraint; the request cap is.**
+- Measured after the build (dry-test wire log): enriched technical
+  payload with 20 candles ≈ **~380 tokens**; replies ≈ 25–35 tokens
+  under the new 15-word note cap.
+- Caveats: (a) the panel is advisory-only — richer payloads improve the
+  human's context, not the book's P&L; (b) raw prices are the model's
+  weakest input, so candles are sent **ATR-normalized** (code does the
+  arithmetic, LLM reads the shape) — the same pattern that fixed the
+  event-direction misreads.
+
+## Answer 2 — AI on every candle? NO
+
+- **Feasibility**: M5+M15+H1+H4 ≈ 414 bars/day → one call/bar nearly
+  exhausts the shared 500/day cap (news needs it too); a panel per bar
+  ≈ 2,070/day = 4× over. Most of that spend reviews HOLD bars.
+- **Parity (rule 4)**: an AI gate cannot be replayed by the backtester
+  (news off; no recorded verdicts) — every pinned book becomes
+  non-comparable, and model-version drift silently mutates the strategy.
+- **Edge**: the engine already compresses 380 bars into 9 tuned
+  indicators + regime + MTF validated over 5 years; an LLM's generic
+  prior on raw candles won't beat fitted thresholds, and the margin it
+  would filter is ~0.185R/trade. B-grade holds +67.8R at 51.4% win —
+  a blunt veto is net-negative.
+
+## The middle ground (adopted)
+
+1. **Enrich payloads** (built today, see below) — near-zero cost, no
+   rule-1 exposure.
+2. **Log panel verdicts to signals.db** (built today) — creates the
+   "does OPPOSE predict losers?" dataset with zero behavior change.
+3. **After 4–8 weeks**: join logged verdicts to outcomes; simulate an
+   OPPOSE-veto against the pinned book offline. Only then consider any
+   gating, as a full rule-1 revalidation.
+
+## Build record (2026-08-15, owner-approved)
+
+- `agent_panel.py`
+  - Technical persona payload now carries indicator **values** (units
+    legend added to the prompt) and **ATR-normalized candles** per TF:
+    `{"ref_close", "atr", "bars": [[o,h,l,c]…]}` oldest-first, each price
+    as `(price − ref_close) / atr`, 1 decimal. Window: 20 bars primary /
+    10 per confirmation TF.
+  - New config keys (validated in `app_config.py`, in
+    `config.json.example`): `candles_enabled` (default true),
+    `candle_bars_primary` (20), `candle_bars_confirm` (10).
+  - Panel-private provider singleton (`_dp`, patchable in tests);
+    TTL cache keeps fetches near-free; any failure → candles silently
+    omitted, panel unaffected.
+  - Prompt hardening: training data declared stale and OFF-LIMITS;
+    "if a fact is not in the JSON, treat it as unknown"; thin evidence
+    → neutral < 30 (unchanged); note cap 25→**15 words**, rationale
+    40→**30 words** (output-token savings).
+  - `attach_panel` now calls `signal_log.log_panel_update(report)` after
+    every successful attach (fresh, exact-cache, and reuse-TTL paths).
+- `signal_log.py`
+  - `log()` returns the rowid; `log_signal` stashes it on the report as
+    `_signal_log_rowid`; new `log_panel_update()` fills the 4 panel
+    columns on that row post-hoc. No rowid (backtest replay, logging
+    disabled, other process) → silent no-op. Fail-safe contract kept.
+- Tests: 381 → **388 passed** (7 new: candle normalization math, payload
+  values/candles, context windows + ATR gating + disable flag, attach
+  injection (technical persona only), prompt-contract assertions,
+  panel row update, rowid hand-off + no-op paths). Existing attach tests
+  stub `ap._dp` to stay off the network.
+- **Live dry test** (`panel_payload_drytest_2026-08-15.py`, fabricated
+  numbers only, 13 real DeepSeek requests total): **7/7 probes pass** —
+  coherent-bullish → support@70; bearish-vs-BUY → oppose@85; genuinely
+  mixed → neutral@30; empty payload → neutral@20 (contract); no invented
+  numbers in any note; all notes ≤ 15 words; synthesizer AGREE with a
+  21-word rationale. Raw wire log:
+  `backtest/research/panel_payload_drytest_2026-08-15.jsonl`.
+  - Notable: an early P3 variant with H1 SELL against a BUY brief got
+    oppose@60 — *correct* per the prompt (higher-TF disagreement is a
+    listed conflict), so the probe was tightened to a genuinely mixed
+    scenario. Documented so nobody "fixes" the prompt for it later.
+
+## What this deliberately did NOT change
+
+- No gate reads panel fields; the panel remains advisory-only.
+- No backtest-path change (`analyze_from_dataframes` untouched; parity
+  test still locks panel fields empty in replay).
+- No verdict-based filtering — that decision waits for the signals.db
+  dataset (4–8 weeks) and an offline veto replay against the pinned book.

backtest/research/book_diff.py

@@ -0,0 +1,86 @@
+"""Trade-book diff: relaxed/gated run vs its control.
+
+Usage:
+    python backtest/research/book_diff.py CONTROL.csv CANDIDATE.csv BAR_MINUTES
+
+Decision numbers are RUN-LEVEL (ΔtotalR, ΔPF, ΔmaxDD, Δtrades) — never
+the added-trade bucket alone: the 5-bar entry cooldown means new
+entries displace existing ones. The added/removed decomposition and the
+per-decision-hour histogram of added trades are diagnostics.
+
+Shifted trades = removed+added pairs in the same direction within
+±5 bars (cooldown displacement, not a genuinely new/lost signal).
+"""
+import sys
+
+import pandas as pd
+
+
+def _metrics(df):
+    r = df["r_multiple"]
+    gains, losses = r[r > 0].sum(), -r[r <= 0].sum()
+    cum = r.cumsum()
+    return dict(trades=len(df), total_r=r.sum(),
+                pf=gains / max(losses, 1e-9),
+                wr=(r > 0).mean() * 100 if len(df) else 0.0,
+                max_dd=(cum.cummax() - cum).max() if len(cum) else 0.0)
+
+
+def diff_books(control_csv, candidate_csv, bar_minutes, split="2026-01-29"):
+    ctrl, cand = pd.read_csv(control_csv), pd.read_csv(candidate_csv)
+    for df in (ctrl, cand):
+        df["_t"] = pd.to_datetime(df["entry_time"], utc=True, format="mixed")
+        df["_key"] = df["entry_time"].astype(str) + "|" + df["direction"]
+
+    added = cand[~cand["_key"].isin(ctrl["_key"])].copy()
+    removed = ctrl[~ctrl["_key"].isin(cand["_key"])].copy()
+
+    # cooldown displacement: same-direction removed/added within ±5 bars
+    tol = pd.Timedelta(minutes=5 * bar_minutes)
+    shifted = 0
+    rem_pool = removed.copy()
+    for _, a in added.iterrows():
+        m = rem_pool[(rem_pool["direction"] == a["direction"])
+                     & ((rem_pool["_t"] - a["_t"]).abs() <= tol)]
+        if len(m):
+            shifted += 1
+            rem_pool = rem_pool.drop(m.index[0])
+
+    mc, mk = _metrics(ctrl), _metrics(cand)
+    split_ts = pd.Timestamp(split, tz="UTC")
+    halves = (cand[cand["_t"] < split_ts]["r_multiple"].sum(),
+              cand[cand["_t"] >= split_ts]["r_multiple"].sum())
+    hour_hist = None
+    if len(added):
+        added["dec_hour"] = (added["_t"]
+                             - pd.Timedelta(minutes=bar_minutes)).dt.hour
+        hour_hist = added.groupby("dec_hour")["r_multiple"] \
+                         .agg(["count", "sum"]).round(2)
+    return dict(
+        control=mc, candidate=mk,
+        delta_r=mk["total_r"] - mc["total_r"],
+        delta_pf=mk["pf"] - mc["pf"],
+        delta_dd=mk["max_dd"] - mc["max_dd"],
+        delta_trades=mk["trades"] - mc["trades"],
+        added_n=len(added), added_r=added["r_multiple"].sum(),
+        removed_n=len(removed), removed_r=removed["r_multiple"].sum(),
+        shifted=shifted, halves=halves, added_by_hour=hour_hist,
+    )
+
+
+if __name__ == "__main__":
+    d = diff_books(sys.argv[1], sys.argv[2], int(sys.argv[3]))
+    c, k = d["control"], d["candidate"]
+    print(f"control  : {c['trades']:4d} trades  {c['total_r']:+7.1f}R  "
+          f"PF {c['pf']:.2f}  WR {c['wr']:.1f}%  DD {c['max_dd']:.1f}")
+    print(f"candidate: {k['trades']:4d} trades  {k['total_r']:+7.1f}R  "
+          f"PF {k['pf']:.2f}  WR {k['wr']:.1f}%  DD {k['max_dd']:.1f}")
+    print(f"ΔR {d['delta_r']:+.1f}  ΔPF {d['delta_pf']:+.2f}  "
+          f"ΔDD {d['delta_dd']:+.1f}  Δtrades {d['delta_trades']:+d}")
+    print(f"added {d['added_n']} ({d['added_r']:+.1f}R)  "
+          f"removed {d['removed_n']} ({d['removed_r']:+.1f}R)  "
+          f"shifted≈{d['shifted']}")
+    print(f"candidate halves: H1 {d['halves'][0]:+.1f}R  H2 {d['halves'][1]:+.1f}R")
+    if d["added_by_hour"] is not None:
+        print("\nadded trades by decision hour:")
+        print(d["added_by_hour"].to_string())

backtest/research/burst_profile.py

@@ -0,0 +1,42 @@
+"""Burst-candle frequency by UTC hour — the empirical "hedge fund hours".
+
+Usage:
+    python backtest/research/burst_profile.py PARQUET_GLOB
+
+Per UTC bar-close hour of the given bars: % of bars whose range is
+>= K x the PRIOR bar's Wilder ATR(14), for K in {3,4,5}; mean bar
+range; and mean signed drift over the 6 bars following a >=4x burst
+(does price continue or snap back?). Analysis tooling only.
+"""
+import glob
+import sys
+
+import pandas as pd
+
+
+def burst_table(parquet_glob: str, bar_minutes: int = 5):
+    files = sorted(glob.glob(parquet_glob))
+    if not files:
+        raise SystemExit(f"no parquet matches {parquet_glob}")
+    df = pd.concat([pd.read_parquet(f) for f in files]).sort_index()
+    df = df[~df.index.duplicated(keep="last")]
+    h, l, c = df["High"], df["Low"], df["Close"]
+    tr = pd.concat([h - l, (h - c.shift()).abs(), (l - c.shift()).abs()],
+                   axis=1).max(axis=1)
+    atr = tr.ewm(alpha=1 / 14, min_periods=14, adjust=False).mean()
+    ratio = (h - l) / atr.shift(1)
+    close_hour = (df.index + pd.Timedelta(minutes=bar_minutes)).hour
+    fwd6 = c.shift(-6) - c  # signed move over the 6 bars after this one
+    out = pd.DataFrame({
+        "bars": ratio.groupby(close_hour).size(),
+        **{f"pct_ge_{k}x": (ratio >= k).groupby(close_hour).mean() * 100
+           for k in (3, 4, 5)},
+        "mean_range": (h - l).groupby(close_hour).mean(),
+        "post4x_drift6": fwd6[ratio >= 4].groupby(close_hour[ratio >= 4]).mean(),
+    })
+    return out.round(3)
+
+
+if __name__ == "__main__":
+    print(burst_table(sys.argv[1],
+                      int(sys.argv[2]) if len(sys.argv) > 2 else 5).to_string())

backtest/research/commit_ladder.py

@@ -0,0 +1,178 @@
+"""Commit-ladder campaign driver (2026-08).
+
+Question: of all signal-code states shipped in the last two months, which
+configuration is the most accurate? Historical in-era backtest numbers are
+not comparable (the backtester's realism model changed on 07-08/07-16 and
+the data feed switched to Oanda on 07-28), so every historical
+configuration is recreated on TODAY's harness via --set overrides and
+replayed on the same pinned Oanda bars. Bug-fix commits stay on; the
+ladder compares shipped tuning decisions only.
+
+Usage:
+    python backtest/research/commit_ladder.py            # full matrix
+    python backtest/research/commit_ladder.py --tf 5min  # one timeframe
+    python backtest/research/commit_ladder.py --windows 12mo,oos
+
+Resumable: a run whose CSV already exists in the output dir is skipped.
+Outputs: backtest/research/commit_ladder_2026-08/<variant>_<window>.csv
+         + .log per run, summary.csv, and a printed ranking table.
+"""
+from __future__ import annotations
+
+import argparse
+import subprocess
+import sys
+from pathlib import Path
+
+import pandas as pd
+
+ROOT = Path(__file__).resolve().parents[2]
+BT = ROOT / "backtest" / "gpu_backtest.py"
+OUT = ROOT / "backtest" / "research" / "commit_ladder_2026-08"
+
+WINDOWS = {
+    "12mo": ("2025-07-30", "2026-07-30"),
+    "oos": ("2024-06-01", "2025-03-30"),
+    "5yr": ("2021-07-30", "2026-07-30"),
+}
+SPLIT_12MO = pd.Timestamp("2026-01-29", tz="UTC")  # book_diff.py's split
+
+# (variant, "commit date what-it-represents", [--set overrides])
+VARIANTS: dict[str, list[tuple[str, str, list[str]]]] = {
+    "5min": [
+        ("M5-A", "dad2432 07-09 gates+trailing+pullback",
+         ["xau_strategy.blocked_hours_utc=[0,1,2,10,11,13,16,21]",
+          "xau_strategy.cp_enabled=false",
+          "signals.trail_dist_r=0.75",
+          "xau_strategy.m5_min_abs_score=30",
+          "xau_strategy.m5_min_confirmation_score=10"]),
+        ("M5-B", "f3ba4d5 07-11 +hour-9 block",
+         ["xau_strategy.blocked_hours_utc=[0,1,2,10,11,13,16,21,9]",
+          "xau_strategy.cp_enabled=false",
+          "signals.trail_dist_r=0.75",
+          "xau_strategy.m5_min_abs_score=30",
+          "xau_strategy.m5_min_confirmation_score=10"]),
+        ("M5-C", "de6424e 07-16 +chart patterns",
+         ["xau_strategy.blocked_hours_utc=[0,1,2,10,11,13,16,21,9]",
+          "signals.trail_dist_r=0.75",
+          "xau_strategy.m5_min_abs_score=30",
+          "xau_strategy.m5_min_confirmation_score=10"]),
+        ("M5-D", "3ee79b5 07-21 +trail dist 0.85",
+         ["xau_strategy.blocked_hours_utc=[0,1,2,10,11,13,16,21,9]",
+          "xau_strategy.m5_min_abs_score=30",
+          "xau_strategy.m5_min_confirmation_score=10"]),
+        ("M5-E", "0396970 07-30 relaxation (HEAD)", []),
+    ],
+    "15min": [
+        ("M15-A", "987ec5b 07-09 pre-campaigns",
+         ["signals.tp_floor_rr=0.5",
+          "xau_strategy.blocked_regimes=[]",
+          "xau_strategy.sell_blocked_hours_utc=[]",
+          "signals.trail_arm_r=0.60"]),
+        ("M15-B", "f3ba4d5/f88e39f 07-11 1R floor + regime block",
+         ["xau_strategy.sell_blocked_hours_utc=[]",
+          "signals.trail_arm_r=0.60"]),
+        ("M15-C", "7d3fd95 07-16 +SELL hour 15/17 block",
+         ["signals.trail_arm_r=0.60"]),
+        ("M15-D", "bf8e5fb 07-21 trail arm 0.70 (HEAD)", []),
+    ],
+    "1h": [
+        ("H1-A", "pre-d95d31a 07-08 TP-tightening model",
+         ["signals.min_rr_ratio=0.4",
+          "signals.tp_tightening_factor=0.8"]),
+        ("H1-B", "HEAD", []),
+    ],
+}
+
+
+def run_one(tf: str, name: str, overrides: list[str], wkey: str,
+            spread: str | None = None) -> Path:
+    suffix = f"_s{spread}" if spread else ""
+    csv_path = OUT / f"{name}_{wkey}{suffix}.csv"
+    log_path = OUT / f"{name}_{wkey}{suffix}.log"
+    if csv_path.exists():
+        print(f"  [skip] {csv_path.name} exists")
+        return csv_path
+    start, end = WINDOWS[wkey]
+    cmd = [sys.executable, str(BT), "--pair", "XAUUSD",
+           "--primary-tf", tf, "--start", start, "--end", end,
+           "--export-csv", str(csv_path)]
+    if spread:
+        cmd += ["--spread", spread]
+    for ov in overrides:
+        cmd += ["--set", ov]
+    print(f"  [run ] {csv_path.name} ...", flush=True)
+    with open(log_path, "w") as lf:
+        rc = subprocess.run(cmd, stdout=lf, stderr=subprocess.STDOUT,
+                            cwd=ROOT).returncode
+    if rc != 0:
+        raise SystemExit(f"backtest failed (rc={rc}) — see {log_path}")
+    log = log_path.read_text()
+    n_ov = log.count("--set override")
+    if n_ov != len(overrides):
+        raise SystemExit(
+            f"{name}_{wkey}: expected {len(overrides)} --set overrides, "
+            f"log shows {n_ov} — see {log_path}")
+    if "Loading cached" not in log and wkey != "sanity":
+        print(f"  [warn] {log_path.name}: no 'Loading cached' line — "
+              f"data was fetched live for at least one timeframe")
+    return csv_path
+
+
+def metrics(csv_path: Path, wkey: str) -> dict:
+    df = pd.read_csv(csv_path, parse_dates=["entry_time"])
+    if df.empty:
+        return {"trades": 0, "total_r": 0.0, "wr": 0.0, "pf": 0.0,
+                "max_dd": 0.0, "h1_r": 0.0, "h2_r": 0.0, "years_pos": ""}
+    if df["entry_time"].dt.tz is None:
+        df["entry_time"] = df["entry_time"].dt.tz_localize("UTC")
+    r = df["r_multiple"]
+    pos, neg = r[r > 0].sum(), r[r < 0].sum()
+    cum = r.cumsum()
+    dd = (pd.concat([pd.Series([0.0]), cum]).cummax()[1:] - cum).max()
+    out = {
+        "trades": len(df),
+        "total_r": round(r.sum(), 1),
+        "wr": round((r > 0).mean() * 100, 1),
+        "pf": round(pos / abs(neg), 2) if neg else float("inf"),
+        "max_dd": round(dd, 1),
+        "h1_r": "", "h2_r": "", "years_pos": "",
+    }
+    if wkey == "12mo":
+        h1 = r[df["entry_time"] < SPLIT_12MO].sum()
+        out["h1_r"] = round(h1, 1)
+        out["h2_r"] = round(r.sum() - h1, 1)
+    if wkey == "5yr":
+        by_year = r.groupby(df["entry_time"].dt.year).sum()
+        out["years_pos"] = f"{(by_year > 0).sum()}/{len(by_year)}"
+    return out
+
+
+def main() -> None:
+    ap = argparse.ArgumentParser()
+    ap.add_argument("--tf", choices=list(VARIANTS), default=None)
+    ap.add_argument("--windows", default="12mo,oos,5yr")
+    args = ap.parse_args()
+    OUT.mkdir(parents=True, exist_ok=True)
+    wkeys = [w.strip() for w in args.windows.split(",")]
+    tfs = [args.tf] if args.tf else list(VARIANTS)
+
+    rows = []
+    for tf in tfs:
+        for name, label, overrides in VARIANTS[tf]:
+            for wkey in wkeys:
+                csv_path = run_one(tf, name, overrides, wkey)
+                m = metrics(csv_path, wkey)
+                rows.append({"tf": tf, "variant": name, "represents": label,
+                             "window": wkey, **m})
+                print(f"    {name} {wkey}: {m['trades']} trades, "
+                      f"{m['total_r']}R, PF {m['pf']}, WR {m['wr']}%")
+
+    summary = pd.DataFrame(rows)
+    summary.to_csv(OUT / "summary.csv", index=False)
+    print("\n=== SUMMARY (also in summary.csv) ===")
+    print(summary.to_string(index=False))
+
+
+if __name__ == "__main__":
+    main()

backtest/research/duka_spread.py

@@ -0,0 +1,194 @@
+"""Dukascopy XAU/USD tick download + bid/ask spread study.
+
+Validates ForexBot2's $0.30 round-trip cost assumption (config:
+SignalConfig.round_trip_cost / PAIR_INFO default_spread). Round-trip
+cost of a market entry+exit = one ask-bid spread width (buy at ask,
+sell at bid), so the assumption holds iff typical spread ~ $0.30.
+
+bi5 format: LZMA 'alone' stream of 20-byte big-endian records
+(ms_in_hour uint32, ask uint32, bid uint32, ask_vol f32, bid_vol f32);
+XAUUSD prices scaled x1000. URL months are 0-indexed.
+"""
+import lzma
+import struct
+import sys
+from concurrent.futures import ThreadPoolExecutor
+from datetime import date, timedelta
+from pathlib import Path
+
+import numpy as np
+import pandas as pd
+import requests
+
+SCALE = 1000.0
+CACHE = Path(__file__).parent / "tick_cache"
+CACHE.mkdir(exist_ok=True)
+URL = "https://datafeed.dukascopy.com/datafeed/XAUUSD/{y}/{m:02d}/{d:02d}/{h:02d}h_ticks.bi5"
+
+# Bot's tradable bar-close hours: session 03:00-23:00 minus XAU blocked
+# hours [0,1,2,10,11,13,16,21] (M5 additionally blocks 9 — reported
+# separately below).
+BLOCKED = {0, 1, 2, 10, 11, 13, 16, 21}
+TRADABLE = sorted(set(range(3, 23)) - BLOCKED)
+
+# ── Day sample ────────────────────────────────────────────────────────
+recent = [date(2026, 7, 6) + timedelta(days=i) for i in range(17)]
+recent = [d for d in recent if d.weekday() < 5]              # Mon-Fri Jul 6-22
+monthly = [date(2025, 8, 13), date(2025, 9, 10), date(2025, 10, 15),
+           date(2025, 11, 12), date(2025, 12, 10), date(2026, 1, 14),
+           date(2026, 2, 11), date(2026, 3, 11), date(2026, 4, 15),
+           date(2026, 5, 13), date(2026, 6, 10)]             # mid-month Wednesdays
+nfp = [date(2026, 6, 5), date(2026, 7, 3)]                   # NFP Fridays
+sundays = [date(2026, 7, 12), date(2026, 7, 19)]             # Sun 22-24 open
+DAYS = sorted(set(recent + monthly + nfp + sundays))
+assert all(d.weekday() < 5 for d in recent + monthly + nfp)
+
+
+_tls = __import__("threading").local()
+
+
+def _session() -> requests.Session:
+    if not hasattr(_tls, "s"):
+        _tls.s = requests.Session()
+        _tls.s.headers["User-Agent"] = "Mozilla/5.0 (research; spread-study)"
+    return _tls.s
+
+
+def fetch(day: date, hour: int) -> Path:
+    p = CACHE / f"{day.isoformat()}_{hour:02d}.bi5"
+    if p.exists():
+        return p
+    u = URL.format(y=day.year, m=day.month - 1, d=day.day, h=hour)
+    last = None
+    for attempt in range(4):
+        try:
+            r = _session().get(u, timeout=30)
+            if r.status_code != 200:
+                raise RuntimeError(f"{u} -> HTTP {r.status_code}")
+            p.write_bytes(r.content)
+            return p
+        except (requests.ConnectionError, requests.Timeout, RuntimeError) as exc:
+            last = exc
+            __import__("time").sleep(1.5 * (attempt + 1))
+            if hasattr(_tls, "s"):
+                _tls.s.close()
+                del _tls.s
+    raise RuntimeError(f"giving up on {u}: {last}")
+
+
+def decode(p: Path, day: date, hour: int) -> pd.DataFrame:
+    blob = p.read_bytes()
+    if not blob:
+        return pd.DataFrame()
+    raw = lzma.decompress(blob)
+    n = len(raw) // 20
+    a = np.frombuffer(raw[: n * 20], dtype=">u4").reshape(n, 5)
+    f = np.frombuffer(raw[: n * 20], dtype=">f4").reshape(n, 5)
+    df = pd.DataFrame({
+        "ms": a[:, 0].astype(np.int64),
+        "ask": a[:, 1] / SCALE,
+        "bid": a[:, 2] / SCALE,
+    })
+    df["spread"] = df["ask"] - df["bid"]
+    df["date"], df["hour"] = day.isoformat(), hour
+    df["weekday"] = day.weekday()
+    return df
+
+
+def load_all() -> pd.DataFrame:
+    jobs = [(d, h) for d in DAYS for h in range(24)]
+    with ThreadPoolExecutor(max_workers=3) as ex:
+        paths = list(ex.map(lambda j: fetch(*j), jobs))
+    frames = [decode(p, d, h) for p, (d, h) in zip(paths, jobs)]
+    df = pd.concat([f for f in frames if not f.empty], ignore_index=True)
+    return df
+
+
+def wstats(g: pd.DataFrame) -> pd.Series:
+    """Tick stats + time-weighted mean (each tick's spread held until the
+    next tick; last tick held to hour end)."""
+    tw_num = tw_den = 0.0
+    keys = [k for k in ("date", "hour") if k in g.columns]
+    for _, grp in g.groupby(keys, sort=False):
+        ms = grp["ms"].to_numpy()
+        sp = grp["spread"].to_numpy()
+        w = np.diff(np.append(ms, 3_600_000)).clip(min=0)
+        tw_num += float((sp * w).sum())
+        tw_den += float(w.sum())
+    s = g["spread"]
+    return pd.Series({
+        "ticks": len(s),
+        "tw_mean": tw_num / tw_den if tw_den else np.nan,
+        "mean": s.mean(), "median": s.median(),
+        "p75": s.quantile(0.75), "p90": s.quantile(0.90),
+        "p99": s.quantile(0.99), "max": s.max(),
+        "pct_le_030": (s <= 0.30).mean() * 100,
+        "pct_ge_060": (s >= 0.60).mean() * 100,
+    })
+
+
+def m5_boundary_spreads(df: pd.DataFrame) -> pd.DataFrame:
+    """Spread of the FIRST tick at/after each M5 boundary — what a
+    next-open fill actually pays."""
+    rows = []
+    for (day, hour), grp in df.groupby(["date", "hour"], sort=False):
+        ms = grp["ms"].to_numpy()
+        sp = grp["spread"].to_numpy()
+        wd = grp["weekday"].iat[0]
+        for b in range(0, 3_600_000, 300_000):
+            i = np.searchsorted(ms, b)
+            if i < len(ms) and ms[i] - b <= 30_000:   # tick within 30 s
+                rows.append((day, wd, hour, b // 60_000, sp[i]))
+    return pd.DataFrame(rows, columns=["date", "weekday", "hour", "minute", "spread"])
+
+
+def main() -> None:
+    df = load_all()
+    print(f"days={df['date'].nunique()}  ticks={len(df):,}  "
+          f"price range {df['bid'].min():.0f}-{df['ask'].max():.0f}")
+
+    print("\n── Overall (all sampled hours) ──")
+    print(wstats(df).round(3).to_string())
+
+    trad = df[df["hour"].isin(TRADABLE) & (df["weekday"] < 5)]
+    print("\n── Bot-tradable hours (session 03-23 minus blocked, Mon-Fri) ──")
+    print(wstats(trad).round(3).to_string())
+
+    roll = df[df["hour"].isin([21, 22, 23])]
+    print("\n── Rollover window 21-24 UTC ──")
+    print(wstats(roll).round(3).to_string())
+
+    print("\n── Per UTC hour ──")
+    per_hour = df[df["weekday"] < 5].groupby("hour").apply(wstats, include_groups=False).round(3)
+    per_hour["tradable"] = ["Y" if h in TRADABLE else "-" for h in per_hour.index]
+    print(per_hour.to_string())
+    per_hour.to_csv(Path(__file__).parent / "duka_spread_by_hour.csv")
+
+    print("\n── M5 next-open boundary fills ──")
+    mb = m5_boundary_spreads(df)
+    mb_trad = mb[mb["hour"].isin(TRADABLE) & (mb["weekday"] < 5)]
+    for name, d in [("all hours", mb), ("tradable hours", mb_trad)]:
+        s = d["spread"]
+        print(f"{name}: n={len(s):,} mean={s.mean():.3f} median={s.median():.3f} "
+              f"p90={s.quantile(0.9):.3f} p99={s.quantile(0.99):.3f} "
+              f"%<=0.30={(s <= 0.30).mean()*100:.1f} %>=0.60={(s >= 0.60).mean()*100:.1f}")
+    mb.to_csv(Path(__file__).parent / "duka_m5_boundary_spreads.csv", index=False)
+
+    print("\n── By month (tradable hours, tick median / tw_mean) ──")
+    trad2 = trad.copy()
+    trad2["month"] = trad2["date"].str[:7]
+    bym = trad2.groupby("month").apply(wstats, include_groups=False).round(3)
+    print(bym[["ticks", "tw_mean", "median", "p90", "pct_le_030", "pct_ge_060"]].to_string())
+
+    print("\n── NFP Fridays 12:00-15:00 UTC vs same hours other days ──")
+    nfp_days = {"2026-06-05", "2026-07-03"}
+    win = df[df["hour"].isin([12, 13, 14])]
+    for name, d in [("NFP", win[win["date"].isin(nfp_days)]),
+                    ("non-NFP", win[~win["date"].isin(nfp_days) & (win["weekday"] < 5)])]:
+        s = d["spread"]
+        print(f"{name}: n={len(s):,} tick-median={s.median():.3f} p90={s.quantile(0.9):.3f} "
+              f"p99={s.quantile(0.99):.3f} max={s.max():.2f}")
+
+
+if __name__ == "__main__":
+    sys.exit(main())

backtest/research/duka_spread_by_hour.csv

@@ -0,0 +1,25 @@
+hour,ticks,tw_mean,mean,median,p75,p90,p99,max,pct_le_030,pct_ge_060,tradable
+0,362893.0,0.749,0.768,0.79,0.82,0.86,1.1,7.69,0.053,94.561,-
+1,540182.0,0.735,0.742,0.74,0.81,0.87,1.0,5.59,0.073,90.449,-
+2,431589.0,0.716,0.722,0.73,0.8,0.86,0.95,2.96,0.1,85.27,-
+3,343952.0,0.708,0.711,0.71,0.79,0.84,0.95,1.67,0.081,82.939,Y
+4,246262.0,0.704,0.702,0.71,0.78,0.82,0.94,1.86,0.098,82.627,Y
+5,388365.0,0.712,0.716,0.72,0.79,0.82,0.95,2.17,0.076,86.745,Y
+6,368999.0,0.65,0.651,0.65,0.69,0.74,0.94,2.32,0.079,71.668,Y
+7,313149.0,0.601,0.598,0.59,0.65,0.69,0.994,3.08,0.239,42.904,Y
+8,347276.0,0.612,0.616,0.6,0.66,0.74,1.0,3.04,0.308,48.349,Y
+9,321619.0,0.612,0.615,0.6,0.66,0.72,1.0,5.95,0.288,50.644,Y
+10,295271.0,0.611,0.618,0.6,0.66,0.71,1.0,9.36,0.164,49.253,-
+11,336963.0,0.619,0.624,0.6,0.67,0.79,1.0,4.97,0.203,50.178,-
+12,476773.0,0.66,0.697,0.63,0.72,0.877,1.367,14.98,0.257,59.633,Y
+13,651425.0,0.641,0.648,0.63,0.69,0.787,1.487,11.407,0.296,60.297,-
+14,673391.0,0.625,0.625,0.63,0.68,0.77,0.9,11.98,0.458,59.085,Y
+15,598398.0,0.636,0.638,0.64,0.7,0.78,0.91,4.49,0.433,66.296,Y
+16,485023.0,0.648,0.644,0.64,0.7,0.757,0.917,4.09,0.118,68.496,-
+17,450119.0,0.626,0.628,0.627,0.7,0.726,0.84,4.08,0.154,63.337,Y
+18,411538.0,0.626,0.63,0.637,0.7,0.71,0.79,4.16,0.107,67.734,Y
+19,435396.0,0.638,0.643,0.65,0.7,0.72,0.896,5.804,0.11,72.007,Y
+20,212654.0,0.684,0.665,0.66,0.7,0.76,1.217,7.15,0.226,70.215,Y
+21,31319.0,0.627,0.622,0.61,0.68,0.75,1.02,2.08,0.089,53.153,-
+22,108879.0,0.829,0.834,0.797,0.84,0.98,2.05,10.11,0.056,95.434,Y
+23,161079.0,1.032,0.913,0.8,0.89,1.307,2.77,13.174,0.092,95.41,-

backtest/research/fetch_oanda_m15.py

@@ -0,0 +1,158 @@
+"""Fetch XAU/USD M15 candles from the Oanda v20 REST API.
+
+Mechanical data-fetching infrastructure only — no analysis here.
+Paginates GET /v3/instruments/XAU_USD/candles from a start cursor
+through "now" and writes the result to a parquet file.
+
+Run from the repo root:
+    venv/bin/python backtest/research/fetch_oanda_m15.py
+"""
+import sys
+import time
+from datetime import datetime, timedelta, timezone
+from pathlib import Path
+
+import pandas as pd
+import requests
+
+sys.path.insert(0, str(Path(__file__).resolve().parents[2]))
+
+from app_config import oanda_api_token, oanda_base_url  # noqa: E402
+
+INSTRUMENT = "XAU_USD"
+GRANULARITY = "M15"
+START = datetime(2025, 7, 1, 0, 0, 0, tzinfo=timezone.utc)
+OUT_PATH = Path(__file__).parent / "oanda_xau_m15_20250701_now.parquet"
+COUNT = 5000
+REQUEST_SLEEP_S = 0.5
+FRESHNESS_WINDOW = timedelta(minutes=30)
+
+
+def _rfc3339(dt: datetime) -> str:
+    return dt.astimezone(timezone.utc).strftime("%Y-%m-%dT%H:%M:%S.%fZ")
+
+
+def fetch_all_candles():
+    token = oanda_api_token()
+    base_url = oanda_base_url()
+    if not token:
+        raise RuntimeError(
+            "No Oanda API token resolved (env OANDA_API_TOKEN / "
+            "config.json oanda.api_token). Aborting.")
+
+    url = f"{base_url}/v3/instruments/{INSTRUMENT}/candles"
+    headers = {
+        "Authorization": f"Bearer {token}",
+        "Accept-Datetime-Format": "RFC3339",
+    }
+
+    cursor = START
+    include_first = True
+    all_rows = []
+    n_requests = 0
+
+    while True:
+        params = {
+            "granularity": GRANULARITY,
+            "price": "MBA",
+            "count": COUNT,
+            "dailyAlignment": 0,
+            "alignmentTimezone": "Etc/UTC",
+            "from": _rfc3339(cursor),
+            "includeFirst": "true" if include_first else "false",
+        }
+
+        resp = requests.get(url, headers=headers, params=params, timeout=30)
+        n_requests += 1
+        if resp.status_code != 200:
+            raise RuntimeError(
+                f"Oanda API request failed: status={resp.status_code}\n"
+                f"URL: {resp.url}\n"
+                f"Response body: {resp.text}")
+
+        payload = resp.json()
+        candles = payload.get("candles", [])
+
+        for c in candles:
+            if not c.get("complete", False):
+                continue
+            mid = c["mid"]
+            bid = c["bid"]
+            ask = c["ask"]
+            all_rows.append({
+                "Datetime": pd.Timestamp(c["time"]).tz_convert("UTC")
+                if pd.Timestamp(c["time"]).tzinfo is not None
+                else pd.Timestamp(c["time"]).tz_localize("UTC"),
+                "Open": float(mid["o"]),
+                "High": float(mid["h"]),
+                "Low": float(mid["l"]),
+                "Close": float(mid["c"]),
+                "Volume": int(c["volume"]),
+                "bid_o": float(bid["o"]),
+                "bid_h": float(bid["h"]),
+                "bid_l": float(bid["l"]),
+                "bid_c": float(bid["c"]),
+                "ask_o": float(ask["o"]),
+                "ask_h": float(ask["h"]),
+                "ask_l": float(ask["l"]),
+                "ask_c": float(ask["c"]),
+            })
+
+        print(f"  request {n_requests}: from={params['from']} "
+              f"got {len(candles)} candles (complete kept so far: "
+              f"{len(all_rows)})", file=sys.stderr)
+
+        if not candles:
+            break
+
+        last_time = pd.Timestamp(candles[-1]["time"])
+        if last_time.tzinfo is None:
+            last_time = last_time.tz_localize("UTC")
+        else:
+            last_time = last_time.tz_convert("UTC")
+
+        now = datetime.now(timezone.utc)
+        # Oanda's "count" includes the "from" boundary bar itself, so with
+        # includeFirst=false every full page comes back as count-1, not
+        # count (verified empirically: count=N, includeFirst=false always
+        # yields N-1 candles when a full page of data exists ahead of the
+        # cursor). Compare against the count actually expected for this
+        # request's includeFirst setting, not the raw requested count.
+        expected_full_page = COUNT if include_first else COUNT - 1
+        got_fewer_than_requested = len(candles) < expected_full_page
+        near_now = (now - last_time.to_pydatetime()) < FRESHNESS_WINDOW
+
+        if got_fewer_than_requested or near_now:
+            break
+
+        cursor = last_time.to_pydatetime()
+        include_first = False
+        time.sleep(REQUEST_SLEEP_S)
+
+    return all_rows, n_requests
+
+
+def main():
+    print(f"Fetching {INSTRUMENT} {GRANULARITY} candles from "
+          f"{_rfc3339(START)} through now...", file=sys.stderr)
+
+    rows, n_requests = fetch_all_candles()
+
+    if not rows:
+        raise RuntimeError("No complete candles were fetched.")
+
+    df = pd.DataFrame(rows)
+    df = df.set_index("Datetime")
+    df = df.sort_index()
+    df = df[~df.index.duplicated(keep="last")]
+
+    df.to_parquet(OUT_PATH)
+
+    print(f"\nSaved {len(df)} rows to {OUT_PATH}", file=sys.stderr)
+    print(f"Requests made: {n_requests}", file=sys.stderr)
+    print(f"First timestamp: {df.index[0]}", file=sys.stderr)
+    print(f"Last timestamp: {df.index[-1]}", file=sys.stderr)
+
+
+if __name__ == "__main__":
+    main()

backtest/research/ff_calendar_scraper.py

@@ -0,0 +1,144 @@
+"""Forex Factory calendar scraper -> backtest/cache/calendar.parquet.
+
+Fetches weekly calendar pages and parses the event JSON embedded in
+`window.calendarComponentStates` (the `days:` array). Event `dateline`
+is a unix epoch, so timestamps carry no timezone ambiguity — verified
+2026-07-11 against known release times (NFP 12:30/13:30 UTC across DST,
+ECB 12:15/13:15) and against EUR/USD M15 volatility: the bar containing
+a high-impact event averages 3.5x normal range, and shifting event
+times by ±1h destroys that alignment.
+
+Usage:
+    python backtest/research/ff_calendar_scraper.py --start 2024-01-01 --end 2026-07-13
+    python backtest/research/ff_calendar_scraper.py            # refresh: last stored week -> today
+
+Merges into the existing parquet on event (id, dateline), so refreshing
+is idempotent and also picks up revised `actual` values for recent weeks.
+Be polite: ~1.6s between requests (a full year is ~52 requests).
+"""
+import argparse
+import json
+import time
+from datetime import date, timedelta
+from pathlib import Path
+from typing import List, Optional
+
+import pandas as pd
+import requests
+
+CACHE_DIR = Path(__file__).resolve().parent.parent / "cache"
+PARQUET = CACHE_DIR / "calendar.parquet"
+
+UA = ("Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 "
+      "(KHTML, like Gecko) Chrome/126.0 Safari/537.36")
+
+FIELDS = ["id", "dateline", "name", "country", "currency", "impactName",
+          "timeLabel", "timeMasked", "actual", "forecast", "previous",
+          "revision", "actualBetterWorse"]
+
+
+def week_url(d: date) -> str:
+    return (f"https://www.forexfactory.com/calendar?week="
+            f"{d.strftime('%b').lower()}{d.day}.{d.year}")
+
+
+def extract_days(html: str) -> Optional[list]:
+    """Balanced-bracket extraction of the `days:` JSON array."""
+    i = html.find("days:")
+    if i < 0:
+        return None
+    j = html.find("[", i)
+    depth, in_str, esc = 0, False, False
+    for k in range(j, len(html)):
+        ch = html[k]
+        if in_str:
+            if esc:
+                esc = False
+            elif ch == "\\":
+                esc = True
+            elif ch == '"':
+                in_str = False
+        else:
+            if ch == '"':
+                in_str = True
+            elif ch == "[":
+                depth += 1
+            elif ch == "]":
+                depth -= 1
+                if depth == 0:
+                    return json.loads(html[j:k + 1])
+    return None
+
+
+def fetch_week(sess: requests.Session, d: date, retries: int = 4) -> Optional[List[dict]]:
+    for attempt in range(retries):
+        try:
+            r = sess.get(week_url(d), timeout=30)
+            if r.status_code == 200:
+                days = extract_days(r.text)
+                if days is not None:
+                    return [{k: e.get(k) for k in FIELDS}
+                            for day in days for e in day.get("events", [])]
+        except requests.RequestException:
+            pass
+        time.sleep(5 * (attempt + 1))
+    return None
+
+
+def scrape(start: date, end: date) -> pd.DataFrame:
+    sess = requests.Session()
+    sess.headers["User-Agent"] = UA
+    rows: List[dict] = []
+    failed: List[date] = []
+    d = start
+    while d <= end:
+        week_rows = fetch_week(sess, d)
+        if week_rows is None:
+            failed.append(d)
+            print(f"FAILED week {d}")
+        else:
+            rows.extend(week_rows)
+        d += timedelta(days=7)
+        time.sleep(1.6)
+    if failed:
+        print(f"WARNING: {len(failed)} weeks failed: {failed}")
+    return pd.DataFrame(rows)
+
+
+def merge_into_store(new: pd.DataFrame, path: Path = PARQUET) -> pd.DataFrame:
+    if path.exists():
+        old = pd.read_parquet(path)
+        combined = pd.concat([old, new])
+    else:
+        combined = new
+    # keep="last": a refresh overwrites stale rows (actuals fill in
+    # after release, revisions land weeks later).
+    combined = (combined.drop_duplicates(subset=["id", "dateline"], keep="last")
+                .sort_values("dateline").reset_index(drop=True))
+    combined.to_parquet(path, index=False)
+    return combined
+
+
+def main() -> None:
+    ap = argparse.ArgumentParser(description=__doc__.splitlines()[0])
+    ap.add_argument("--start", type=date.fromisoformat, default=None)
+    ap.add_argument("--end", type=date.fromisoformat, default=date.today())
+    args = ap.parse_args()
+
+    start = args.start
+    if start is None:
+        if PARQUET.exists():
+            last = pd.read_parquet(PARQUET).dateline.max()
+            start = pd.Timestamp(last, unit="s").date() - timedelta(days=7)
+        else:
+            start = date(2024, 1, 1)
+    print(f"Scraping {start} -> {args.end}")
+    new = scrape(start, args.end)
+    combined = merge_into_store(new)
+    print(f"Store: {len(combined)} events "
+          f"({pd.Timestamp(combined.dateline.min(), unit='s')} -> "
+          f"{pd.Timestamp(combined.dateline.max(), unit='s')}) -> {PARQUET}")
+
+
+if __name__ == "__main__":
+    main()

backtest/research/fixedtp_2026-08-05.md

@@ -0,0 +1,123 @@
+# Fixed-RR hard-TP vs shipped trailing exit — research campaign 2026-08-05
+
+**Verdict: DO NOT SHIP. The trailing exit stays.** The fixed 2R target beats the trail on
+total R in every 2024+ window but the edge is a trend-regime artifact: it inverts in the
+2021–2023 ranging years, doubles drawdown, cuts win rate 10–19 points, and doubles hold
+times (amplifying the accepted live one-position divergence). Details below.
+
+## Setup
+
+- Code: HEAD `fc5ce85` + the `--fixed-rr-tp` research flag (uncommitted at run time).
+  The flag replaces every trade's TP with `mult × |entry − sl|` AFTER re-anchoring and
+  disables trailing (run-level and per-trade PB params) — entries, stops, gates, and the
+  cooldown pass are untouched, so both arms replay **trade-identical books** (verified:
+  paired join on entry_time+direction matches 1:1 in every window). Only the exit policy
+  differs → per-trade paired comparison.
+- Provider: Oanda (cache-warm; keys `oanda_XAU_USD_{5min,15min,1h}_{20250603_20260804,
+  20240331_20250330,20210531_20260804}`). Charged cost `--spread 0.65` (real measured
+  round trip); $0.30 letter-cost books derived analytically as `r + 0.35/risk` and
+  verified exact against one real `--spread 0.30` run (max per-trade error 1.8e-15 —
+  spread never enters trigger detection, per the owner-declined 2026-08-04 decision).
+- Control sanity gate: fresh M5 12mo control reproduced the pinned HEAD reference
+  exactly — 555 trades, +98.64R, PF 1.483, maxDD −10.73R.
+- Commands (control = same without the flag):
+  `venv/bin/python backtest/gpu_backtest.py --pair XAUUSD --primary-tf {5min,15min}
+   --start 2025-08-04 --end 2026-08-04 --spread 0.65 [--fixed-rr-tp 2.0]
+   --export-csv backtest/fixedtp_*.csv`
+  OOS `--start 2024-06-01 --end 2025-03-30`; 5yr `--start 2021-08-01 --end 2026-08-04`.
+- Books pinned as `backtest/fixedtp_{m5,m15}_{12mo,oos,5yr}_{ctrl,rr20,...}.csv`
+  (per-trade indicator breakdown, agreement counts, regime, vol-regime, weekday,
+  MAE/MFE, bars-held — the full instructions_for_backtest record).
+
+## Headline (@ $0.65)
+
+| Window | Arm | n | R | PF | WR | maxDD | recov | medBars | exits |
+|---|---|---|---|---|---|---|---|---|---|
+| M5 12mo | trail | 555 | +98.64 | 1.483 | 52.4% | −10.7 | 9.2 | 28 | TRAIL 361 / SL 183 / TO 11 |
+| M5 12mo | 2R hard | 555 | +124.85 | 1.378 | 42.9% | −24.5 | 5.1 | 50 | SL 312 / TP1 210 / TO 33 |
+| M15 12mo | trail | 397 | +73.93 | 1.525 | 62.2% | −10.8 | 6.8 | 28 | TRAIL 255 / SL 135 / TO 7 |
+| M15 12mo | 2R hard | 397 | +95.95 | 1.431 | 44.3% | −11.2 | 8.6 | 39 | SL 215 / TP1 151 / TO 31 |
+| M5 OOS | trail | 86 | +9.45 | 1.332 | 53.5% | −8.2 | 1.2 | 70 | — |
+| M5 OOS | 2R hard | 86 | +18.83 | 1.407 | 45.3% | −9.7 | 1.9 | 124 | — |
+| M15 OOS | trail | 299 | +38.77 | 1.360 | 59.2% | −15.5 | 2.5 | 38 | — |
+| M15 OOS | 2R hard | 299 | +72.14 | 1.435 | 45.2% | −13.9 | 5.2 | 82 | — |
+
+Halves (12mo, split 2026-02-03): M5 rr20 +67.99/+56.86 (both positive; H2 PF 1.325 vs
+ctrl 1.503), M15 rr20 +73.74/+22.21 (both positive; ctrl H2 +7.23 — the known H2-flat).
+At $0.30 all deltas keep the same sign (e.g. M5 12mo +109.0 vs +135.2).
+
+So the **letter** of rule 1 passes for rr20 on both TFs. The 5yr decomposition is why it
+still must not ship.
+
+## Exit migration (paired, M5 12mo; same pattern everywhere)
+
+- SL→SL 183 trades: ΔR 0 (identical by construction — stops untouched).
+- TRAIL→TP1 207 trades: +248.9R → +424.5R (**+175.6R**) — winners the trail exits early.
+- TRAIL→SL 129 trades: +19.0R → −136.8R (**−155.7R**) — retraces the trail survives
+  become full stop-outs.
+- TRAIL→TIMEOUT 25: +7.4R. Net = +26.2R. The whole effect is these two big flows nearly
+  cancelling; which one wins is a regime property, not an exit-policy edge.
+- Trades with ctrl MFE ≥ 2R: 95; hard cap forfeits only +10.4R vs trail on M5 (the trail
+  already gives ~0.75R back from peak, so a 2R cap costs little *in a trend*).
+
+## Mini-grid (12mo @ $0.65) — the tell
+
+| mult | M5 R / PF / DD | M15 R / PF / DD |
+|---|---|---|
+| trail | +98.6 / 1.483 / −10.7 | +73.9 / 1.525 / −10.8 |
+| 1.0R | +52.5 / 1.205 / −12.8 | +67.1 / 1.410 / −7.3 |
+| 1.5R | +80.7 / 1.263 / −21.4 | +66.9 / 1.325 / −11.2 |
+| 2.0R | +124.9 / 1.378 / −24.5 | +96.0 / 1.431 / −11.2 |
+| 2.5R | +147.7 / 1.426 / −23.6 | +110.4 / 1.469 / −15.9 |
+| 3.0R | +155.8 / 1.432 / −21.6 | +135.8 / 1.562 / −17.5 |
+
+Monotone in the multiple on both TFs — "2.0" is not an optimum, it's a point on a curve
+that says *never take profit* (3R's M5 TIMEOUT bucket alone is +84.8R). That is the
+signature of a one-way trend sample, not of a superior exit rule. It also reconfirms the
+original finding that SHORT fixed targets (≤1.5R) are strictly bad (cf. RR-0.5 removal,
+`config.py:126-133`: +48R → +105R).
+
+## 5yr regime decomposition (2021-08-01 → 2026-08-04 @ $0.65) — the verdict
+
+M15 (real sample in every year): ctrl +127.97 PF 1.209 DD −35.8; rr20 +158.97 PF 1.170
+DD **−59.8**; rr30 +204.03 PF 1.203 DD **−98.3**.
+
+| Year | n | trail | 2R hard | 3R hard |
+|---|---|---|---|---|
+| 2021 | 88 | **+4.00** | −16.09 | −17.41 |
+| 2022 | 284 | −9.54 | −7.63 | −26.74 |
+| 2023 | 227 | **+7.79** | −1.71 | −5.81 |
+| 2024 | 345 | +40.44 | +64.98 | +89.14 |
+| 2025 | 405 | +49.33 | +69.39 | +104.42 |
+| 2026 | 229 | +35.95 | +50.04 | +60.43 |
+
+M5 5yr mirrors it (ctrl +128.4 PF 1.387 DD −12.8 vs rr20 +160.2 PF 1.306 DD −24.7;
+2022: −1.1 vs −5.2), though pre-2024 M5 samples are thin (12–60 trades/yr).
+
+The hard target's entire outperformance lives in the 2024–2026 gold trend. In ranging
+years it flips sign exactly where the trail stays positive (M15 2021, 2023), and the
+further the target, the worse the ranging-year bleed and the deeper the DD. The 12mo,
+halves, and OOS windows all sit inside the trend era, which is why the rule-1 letter
+passes — this is precisely the within-regime-split trap rule 1's OOS clause exists to
+catch, one level up.
+
+## Additional non-ship reasons
+
+- **Live divergence grows with hold time.** Median hold doubles (M5 28→50 bars; OOS
+  70→124). The daemon holds one position total; the backtester doesn't model that
+  blocking. Longer holds = more live signals skipped that the paper book banks.
+- **WR 52→43% (M5) / 62→44% (M15), max loss streak 6→11 (M15)** — materially harder to
+  operate and size.
+- Aggregate tables (session/day/regime/vol, streaks, quarterly equity, rolling-20 WR)
+  computed and consistent — no bucket reverses the conclusion; both arms' 12mo profit is
+  concentrated in Q1-2026 (+87.7 ctrl / +102.0 rr20).
+
+## Status
+
+- `--fixed-rr-tp` stays as a documented research-only flag (default off; conflicts with
+  `--trail-arm/--trail-dist/--set signals.*trail*` are hard errors). Module 22 tests
+  (5) pin the override math, TP fill + spread, SL-first tie, PB-trail-disable, and CLI
+  guards. Full suite: 346 green.
+- Re-open only if gold's regime turns and a *regime-conditional* exit (trail in range,
+  target in trend) is proposed with a pre-registered mechanism — a plain fixed ratio is
+  now a closed hypothesis at every multiple in {0.5, 1.0, 1.5, 2.0, 2.5, 3.0}.

backtest/research/hour_analysis.py

@@ -0,0 +1,40 @@
+"""Per-hour P&L breakdown of a backtest trade book.
+
+Usage:
+    python backtest/research/hour_analysis.py BOOK.csv BAR_MINUTES
+
+Reports, per UTC hour (both entry hour and DECISION hour = entry_time
+minus one bar, the clock blocked_hours_utc evaluates): trade count,
+total R, mean R, win rate, profit factor. Also a per-(hour, direction)
+table. Analysis tooling only — never imported by the live code.
+"""
+import sys
+
+import pandas as pd
+
+
+def hour_tables(book_csv: str, bar_minutes: int):
+    df = pd.read_csv(book_csv)
+    et = pd.to_datetime(df["entry_time"], utc=True, format="mixed")
+    df["entry_hour"] = et.dt.hour
+    df["decision_hour"] = (et - pd.Timedelta(minutes=bar_minutes)).dt.hour
+
+    def table(group_cols):
+        g = df.groupby(group_cols)["r_multiple"]
+        out = g.agg(count="count", total_r="sum", mean_r="mean",
+                    wr=lambda r: (r > 0).mean() * 100)
+        out["pf"] = g.apply(
+            lambda r: r[r > 0].sum() / max(-r[r <= 0].sum(), 1e-9))
+        return out.round(2)
+
+    return (table("decision_hour"), table(["decision_hour", "direction"]),
+            table("entry_hour"))
+
+
+if __name__ == "__main__":
+    book, mins = sys.argv[1], int(sys.argv[2])
+    by_dec, by_dec_dir, by_ent = hour_tables(book, mins)
+    print(f"\n── {book} · by DECISION hour (bar-close clock) ──")
+    print(by_dec.to_string())
+    print(f"\n── by (decision hour, direction) ──")
+    print(by_dec_dir.to_string())

backtest/research/llm_news_eval.py

@@ -0,0 +1,697 @@
+"""
+DeepSeek news-analysis validation harness — XAU/USD.
+====================================================
+Research-only (no production code changes). Answers: is the DeepSeek
+layer's news analysis *correct*?
+
+Part A (events): replays every high-impact USD release with a genuine
+surprise (2024-01 → 2026-07, from backtest/cache/calendar.parquet)
+through the REAL `NewsEngine._llm_event_direction` — real prompt, real
+payload, real validation — and scores its "gold bullish/bearish" calls
+against gold's actual move (Oanda M15 mids) at +15m/30m/1h/4h/24h.
+Baselines: mechanical surprise-sign doctrine, FF actualBetterWorse,
+always-bullish/bearish (2024-26 is a gold bull regime), coin flip.
+
+Part B (headlines): a curated golden set (llm_news_golden.csv) through
+the REAL `NewsEngine._llm_headline_scores` in production-sized batches,
+vs the keyword scorer it overrides. Includes repeat-batch (determinism)
+and shuffled-order (cross-contamination) probes.
+
+Implementation notes that keep the replay faithful to production:
+  * `_llm_event_direction`/`_llm_headline_scores` resolve the client via
+    `get_llm_client()`, so we seed the module singleton with a research
+    client (raised daily cap: 586 events > production cap 500).
+  * The event cache key omits the forecast, so a repeat `actual` would
+    silently re-serve a direction computed for a different forecast —
+    the cache is cleared before every call.
+  * A "neutral" reply and a hard failure both return None; they are
+    disambiguated by reading the event cache back after the call
+    (absent = failure, "" = neutral, "bullish:/bearish: …" = call).
+  * Raw request/response bytes are captured with a requests response
+    hook — the request-building code path is untouched.
+
+Usage (from repo root):
+  venv/bin/python backtest/research/llm_news_eval.py events --limit 5
+  venv/bin/python backtest/research/llm_news_eval.py events --determinism 20
+  venv/bin/python backtest/research/llm_news_eval.py headlines
+  venv/bin/python backtest/research/llm_news_eval.py report
+
+Artifacts land in backtest/research/llm_news_eval_2026-08/ (JSONL replay
+logs are resume-safe: completed events are never re-paid). `report` is a
+pure offline function of the artifacts.
+"""
+
+import argparse
+import json
+import math
+import random
+import sys
+import time
+from datetime import datetime, timezone
+from pathlib import Path
+
+import numpy as np
+import pandas as pd
+
+REPO_ROOT = Path(__file__).resolve().parents[2]
+sys.path.insert(0, str(REPO_ROOT))
+
+import llm_client
+from app_config import deepseek_api_key, get_app_config
+from config import NewsConfig
+from llm_client import DEFAULT_MODEL, DeepSeekClient
+from news_engine import EconomicEvent, NewsEngine
+
+RESEARCH_DIR = Path(__file__).parent
+ART_DIR = RESEARCH_DIR / "llm_news_eval_2026-08"
+CALENDAR = REPO_ROOT / "backtest" / "cache" / "calendar.parquet"
+BARS = (REPO_ROOT / "backtest" / "cache" /
+        "oanda_XAU_USD_15min_20210531_20260804.parquet")
+GOLDEN = RESEARCH_DIR / "llm_news_golden.csv"
+EVENTS_JSONL = ART_DIR / "events_replay.jsonl"
+HEADLINES_JSONL = ART_DIR / "headlines_replay.jsonl"
+SCORED_CSV = ART_DIR / "events_scored.csv"
+
+HORIZONS = [("15m", 15), ("30m", 30), ("1h", 60), ("4h", 240),
+            ("24h", 1440)]
+NEUTRAL_BAND = 0.15  # |xau| below this = neutral prediction (Part B)
+BATCH_CAP = 25       # production batch cap (news_engine.py:783)
+EXPECTED_CORPUS = 586
+
+
+# ── shared plumbing ──────────────────────────────────────────────────
+
+class _Recorder:
+    """requests response hook capturing the exact wire traffic."""
+
+    def __init__(self):
+        self.last = None
+
+    def hook(self, resp, **kwargs):
+        body = resp.request.body
+        if isinstance(body, bytes):
+            body = body.decode("utf-8", "replace")
+        self.last = {
+            "request_body": body,
+            "http_status": resp.status_code,
+            "response_body": resp.text,
+            "elapsed_s": round(resp.elapsed.total_seconds(), 3),
+        }
+        return resp
+
+    def pop(self):
+        cap, self.last = self.last, None
+        return cap or {}
+
+
+def install_research_client(timeout: float):
+    """Build a research DeepSeekClient (raised cap) and seed the module
+    singleton so the production code paths use it. Returns (client,
+    recorder)."""
+    app = get_app_config()
+    key = deepseek_api_key(app)
+    ds = app.deepseek_settings
+    if not key:
+        sys.exit("No DeepSeek API key (env DEEPSEEK_API_KEY or "
+                 "config.json deepseek.api_key) — aborting before spend.")
+    if ds.get("enabled") is False:
+        sys.exit("deepseek.enabled is false in config.json — aborting.")
+    client = DeepSeekClient(
+        api_key=key,
+        model=str(ds.get("model", DEFAULT_MODEL)),
+        timeout=timeout,
+        daily_cap=10 ** 6,  # research: corpus 586 > production cap 500
+    )
+    rec = _Recorder()
+    client._session.hooks["response"].append(rec.hook)
+    with llm_client._client_lock:
+        llm_client._client = client
+    return client, rec
+
+
+def _s(x):
+    """Parquet str columns carry float NaN — EconomicEvent needs None."""
+    return None if pd.isna(x) else str(x)
+
+
+def load_corpus():
+    """High-impact USD releases with a genuine surprise, through the
+    production `EconomicEvent.surprise` gate. Asserts run before any
+    API spend."""
+    df = pd.read_parquet(CALENDAR)
+    df = df[(df["impactName"] == "high") & (df["currency"] == "USD")]
+    rows = []
+    for r in df.itertuples():
+        ev = EconomicEvent(
+            time=datetime.fromtimestamp(int(r.dateline), tz=timezone.utc),
+            currency="USD", impact="high", event=_s(r.name) or "",
+            forecast=_s(r.forecast), previous=_s(r.previous),
+            actual=_s(r.actual),
+        )
+        if ev.surprise not in ("positive", "negative"):
+            continue
+        rows.append({
+            "ffid": int(r.id), "dateline": int(r.dateline),
+            "utc_time": ev.time.isoformat(), "name": ev.event,
+            "actual": ev.actual, "forecast": ev.forecast,
+            "previous": ev.previous, "surprise": ev.surprise,
+            "better_worse": int(r.actualBetterWorse), "ev": ev,
+        })
+    ids = [d["ffid"] for d in rows]
+    if len(rows) != EXPECTED_CORPUS:
+        sys.exit(f"corpus size {len(rows)} != {EXPECTED_CORPUS} — "
+                 "calendar.parquet changed; re-verify before spending.")
+    if len(set(ids)) != len(ids):
+        sys.exit("FF ids not unique — resume keying would break.")
+    # Pin the actualBetterWorse mapping on two unambiguous NFP prints.
+    nfp = {d["utc_time"][:10]: d["better_worse"] for d in rows
+           if d["name"] == "Non-Farm Employment Change"}
+    if nfp.get("2024-01-05") != 1 or nfp.get("2024-05-03") != 2:
+        sys.exit("actualBetterWorse mapping pins failed (expected "
+                 "1=better-for-USD on 2024-01-05 beat, 2=worse on "
+                 "2024-05-03 miss).")
+    rows.sort(key=lambda d: (d["dateline"], d["ffid"]))
+    return rows
+
+
+def _load_jsonl(path: Path):
+    recs = []
+    if path.exists():
+        with open(path) as f:
+            for line in f:
+                line = line.strip()
+                if line:
+                    recs.append(json.loads(line))
+    return recs
+
+
+# ── Part A: event replay ─────────────────────────────────────────────
+
+def _extract_reason(cap):
+    try:
+        content = json.loads(cap["response_body"])[
+            "choices"][0]["message"]["content"]
+        return json.loads(content).get("reason")
+    except Exception:
+        return None
+
+
+def cmd_events(args):
+    ART_DIR.mkdir(exist_ok=True)
+    corpus = load_corpus()
+    if args.limit:
+        corpus = corpus[:args.limit]
+    engine = NewsEngine(NewsConfig())
+    existing = _load_jsonl(EVENTS_JSONL)
+    done1 = {r["ffid"] for r in existing
+             if r.get("run_pass") == 1 and r.get("llm_direction")}
+    done2 = {r["ffid"] for r in existing
+             if r.get("run_pass") == 2 and r.get("llm_direction")}
+    pending = [d for d in corpus if d["ffid"] not in done1]
+    print(f"corpus {len(corpus)} | already recorded {len(done1)} | "
+          f"pending {len(pending)} | determinism re-calls "
+          f"{args.determinism} | model calls are ~100 tokens each "
+          f"(deepseek-v4-flash, static-prefix cached) → cost ≈ cents")
+    if not pending and not args.determinism:
+        print("nothing to do")
+        return
+    _, rec = install_research_client(args.timeout)
+
+    out = open(EVENTS_JSONL, "a")
+
+    def call_one(d, run_pass):
+        # Cache key omits forecast → cleared per call so 196 colliding
+        # corpus rows can't be served a stale direction.
+        with NewsEngine._cache_lock:
+            NewsEngine._llm_event_cache.clear()
+        ev = d["ev"]
+        engine._llm_event_direction(ev)
+        key = (ev.event, ev.currency, str(ev.actual or ""))
+        with NewsEngine._cache_lock:
+            cached = NewsEngine._llm_event_cache.get(key)
+        cap = rec.pop()
+        if cached is None:
+            direction, text = None, None       # hard failure → retry
+        elif cached == "":
+            direction, text = "neutral", ""
+        else:
+            direction, text = cached.split(":", 1)[0], cached
+        recd = {k: d[k] for k in (
+            "ffid", "dateline", "utc_time", "name", "actual", "forecast",
+            "previous", "surprise", "better_worse")}
+        recd.update({
+            "llm_direction": direction, "llm_text": text,
+            "reason_raw": _extract_reason(cap), "run_pass": run_pass,
+            "ts": datetime.now(timezone.utc).isoformat(), **cap,
+        })
+        out.write(json.dumps(recd) + "\n")
+        out.flush()
+        return direction
+
+    remaining = pending
+    for npass in range(1, args.retry_passes + 1):
+        if not remaining:
+            break
+        failed = []
+        for i, d in enumerate(remaining):
+            if call_one(d, 1) is None:
+                failed.append(d)
+            if args.sleep:
+                time.sleep(args.sleep)
+            if (i + 1) % 25 == 0:
+                print(f"  pass {npass}: {i + 1}/{len(remaining)} "
+                      f"({len(failed)} failed)")
+        print(f"pass {npass} done: {len(remaining) - len(failed)} ok, "
+              f"{len(failed)} failed")
+        remaining = failed
+    if remaining:
+        print(f"WARNING: {len(remaining)} events still unrecorded after "
+              f"{args.retry_passes} passes — re-run `events` to resume.")
+
+    if args.determinism:
+        ok_ids = {x["ffid"] for x in remaining}
+        done_now = [d for d in corpus if d["ffid"] not in ok_ids]
+        done_now.sort(key=lambda d: (d["name"], d["dateline"]))
+        stride = max(1, len(done_now) // args.determinism)
+        sample = done_now[::stride][:args.determinism]
+        todo = [d for d in sample if d["ffid"] not in done2]
+        print(f"determinism probe: {len(todo)} re-calls "
+              f"({len(sample) - len(todo)} already recorded)")
+        for d in todo:
+            call_one(d, 2)
+            if args.sleep:
+                time.sleep(args.sleep)
+    out.close()
+    print("events replay complete →", EVENTS_JSONL)
+
+
+# ── Part B: golden headlines ─────────────────────────────────────────
+
+def _load_golden():
+    g = pd.read_csv(GOLDEN)
+    need = {"headline", "category", "expected_xau_sign", "rationale"}
+    if not need <= set(g.columns):
+        sys.exit(f"golden CSV missing columns {need - set(g.columns)}")
+    if not g["expected_xau_sign"].isin([-1, 0, 1]).all():
+        sys.exit("expected_xau_sign must be -1/0/+1")
+    if not g["headline"].is_unique:
+        sys.exit("duplicate headlines in golden CSV")
+    return g
+
+
+def cmd_headlines(args):
+    ART_DIR.mkdir(exist_ok=True)
+    g = _load_golden()
+    heads = list(g["headline"])
+    _, rec = install_research_client(args.timeout)
+    engine = NewsEngine(NewsConfig())
+    out = open(HEADLINES_JSONL, "w")  # tiny (~9 calls): fresh each run
+
+    def run_batch(texts, tag, batch_idx):
+        scores = None
+        for attempt in range(3):
+            scores = engine._llm_headline_scores(texts)
+            cap = rec.pop()
+            if scores is not None:
+                break
+            print(f"  {tag} batch {batch_idx} attempt {attempt + 1} "
+                  f"failed (http {cap.get('http_status')})")
+            time.sleep(2.0)
+        out.write(json.dumps({
+            "run_tag": tag, "batch": batch_idx, "headlines": texts,
+            "scores": scores,
+            "ts": datetime.now(timezone.utc).isoformat(), **cap,
+        }) + "\n")
+        out.flush()
+        return scores
+
+    batches = [heads[i:i + BATCH_CAP] for i in range(0, len(heads),
+                                                     BATCH_CAP)]
+    for bi, b in enumerate(batches):
+        ok = run_batch(b, "main", bi) is not None
+        print(f"main batch {bi} ({len(b)} headlines): "
+              f"{'ok' if ok else 'FAILED'}")
+        time.sleep(args.sleep)
+    run_batch(batches[0], "repeat", 0)
+    print("repeat probe done")
+    shuffled = heads[:]
+    random.Random(20260806).shuffle(shuffled)
+    for bi in range(0, len(shuffled), BATCH_CAP):
+        run_batch(shuffled[bi:bi + BATCH_CAP], "shuffled",
+                  bi // BATCH_CAP)
+    print("shuffled probe done →", HEADLINES_JSONL)
+    out.close()
+
+
+# ── report: pure offline function of the artifacts ───────────────────
+
+def _binom_p(k, n):
+    """Exact two-sided binomial p-value vs p=0.5 (no scipy)."""
+    if n == 0:
+        return float("nan")
+    dist = [math.comb(n, i) for i in range(n + 1)]
+    tot = float(2 ** n)
+    pk = dist[k]
+    return min(1.0, sum(c for c in dist if c <= pk) / tot)
+
+
+def _mcnemar_p(hit_a, hit_b):
+    """Exact McNemar via binomial on discordant pairs. Inputs: aligned
+    boolean lists."""
+    b = sum(1 for x, y in zip(hit_a, hit_b) if x and not y)
+    c = sum(1 for x, y in zip(hit_a, hit_b) if not x and y)
+    return b, c, _binom_p(min(b, c), b + c)
+
+
+def _latest_events(recs):
+    """Last pass-1 record per ffid (retries append; last wins)."""
+    out = {}
+    for r in recs:
+        if r.get("run_pass") == 1:
+            out[r["ffid"]] = r
+    return list(out.values())
+
+
+def _fmt_rate(hits, n, label=""):
+    if n == 0:
+        return f"{label:<24} n=0"
+    p = _binom_p(hits, n)
+    return (f"{label:<24} {hits:>4}/{n:<4} = {hits / n * 100:5.1f}%  "
+            f"(binom p={p:.3f})")
+
+
+def report_events():
+    recs = _latest_events(_load_jsonl(EVENTS_JSONL))
+    if not recs:
+        print("no event replay records — run `events` first")
+        return
+    bars = pd.read_parquet(BARS)
+    closes = bars["Close"].to_numpy()
+    close_ns = (bars.index + pd.Timedelta(minutes=15)).asi8
+
+    def anchor(t_ns, h_min):
+        i = np.searchsorted(close_ns, t_ns, side="right") - 1
+        if i < 0:
+            return None
+        stale_ns = max(45.0, h_min / 4.0) * 60e9
+        if t_ns - close_ns[i] > stale_ns:
+            return None
+        return closes[i]
+
+    rows, skip = [], {h: 0 for h, _ in HORIZONS}
+    base_skipped = 0
+    for r in recs:
+        t_ns = int(r["dateline"]) * 10 ** 9
+        base = anchor(t_ns, 0)
+        if base is None:
+            base_skipped += 1
+            continue
+        row = dict(r)
+        row.pop("request_body", None)
+        row.pop("response_body", None)
+        row["year"] = r["utc_time"][:4]
+        row["base"] = base
+        for hname, hmin in HORIZONS:
+            p = anchor(t_ns + hmin * 60 * 10 ** 9, hmin)
+            if p is None:
+                skip[hname] += 1
+                row[f"d_{hname}"] = np.nan
+            else:
+                row[f"d_{hname}"] = p - base
+        rows.append(row)
+    df = pd.DataFrame(rows)
+    df.to_csv(SCORED_CSV, index=False)
+
+    n_fail = int((df["llm_direction"].isna()).sum())
+    print("═" * 70)
+    print("PART A — event gold-direction vs actual XAU move")
+    print("═" * 70)
+    print(f"events replayed: {len(df)} (baseline-price skipped: "
+          f"{base_skipped}) | LLM hard failures: {n_fail}")
+    print("horizon skip counts (stale/closed):",
+          {k: v for k, v in skip.items() if v})
+    dist = df["llm_direction"].value_counts(dropna=False).to_dict()
+    print("LLM call distribution:", dist)
+
+    # Unconditional per-horizon noise scale (null): median |Δ| over all
+    # 2024+ bar times, position-shifted.
+    bars24 = bars[bars.index >= "2024-01-01"]
+    null_med = {}
+    for hname, hmin in HORIZONS:
+        k = hmin // 15
+        d = (bars24["Close"].shift(-k) - bars24["Close"]).abs().dropna()
+        null_med[hname] = float(d.median())
+    print("unconditional null median |Δ| ($):",
+          {k: round(v, 2) for k, v in null_med.items()})
+
+    # Strategy call columns: +1 gold up, −1 gold down, 0/NaN excluded.
+    df["c_llm"] = df["llm_direction"].map(
+        {"bullish": 1, "bearish": -1}).fillna(0)
+    df["c_doctrine"] = df["surprise"].map({"positive": -1, "negative": 1})
+    df["c_ffbw"] = df["better_worse"].map({1: -1, 2: 1}).fillna(0)
+    df["c_bull"] = 1
+    df["c_bear"] = -1
+
+    strategies = [("LLM", "c_llm"), ("doctrine(surprise)", "c_doctrine"),
+                  ("FF betterWorse", "c_ffbw"), ("always-bullish", "c_bull"),
+                  ("always-bearish", "c_bear")]
+
+    for hname, _ in HORIZONS:
+        dcol = f"d_{hname}"
+        sub = df[df[dcol].notna()]
+        print(f"\n── horizon {hname} (n scored bars={len(sub)}) ──")
+        zeros = int((sub[dcol] == 0).sum())
+        if zeros:
+            print(f"  Δ==0 count (scored as miss): {zeros}")
+        for label, col in strategies:
+            s = sub[sub[col] != 0]
+            hits = int((np.sign(s[dcol]) == s[col]).sum())
+            print("  " + _fmt_rate(hits, len(s), label))
+        # noise-filtered LLM hit rate
+        s = sub[(sub["c_llm"] != 0) &
+                (sub[dcol].abs() >= null_med[hname])]
+        hits = int((np.sign(s[dcol]) == s["c_llm"]).sum())
+        print("  " + _fmt_rate(hits, len(s),
+                               f"LLM |Δ|≥null-med"))
+        # event-window magnitude vs null
+        mag = sub[dcol].abs().median()
+        print(f"  event-window median |Δ| = ${mag:.2f} "
+              f"(null ${null_med[hname]:.2f})")
+        # paired McNemar: LLM vs each baseline on common directional set
+        for label, col in [("doctrine", "c_doctrine"),
+                           ("FF betterWorse", "c_ffbw"),
+                           ("always-bullish", "c_bull")]:
+            s = sub[(sub["c_llm"] != 0) & (sub[col] != 0)]
+            ha = list(np.sign(s[dcol]) == s["c_llm"])
+            hb = list(np.sign(s[dcol]) == s[col])
+            b, c, p = _mcnemar_p(ha, hb)
+            print(f"  McNemar LLM vs {label:<15} b={b:>3} c={c:>3} "
+                  f"p={p:.3f}")
+
+    # neutral-call flatness (15m)
+    neu = df[(df["llm_direction"] == "neutral") & df["d_15m"].notna()]
+    dire = df[(df["c_llm"] != 0) & df["d_15m"].notna()]
+    if len(neu):
+        print(f"\nneutral calls (n={len(neu)}): median |Δ_15m| "
+              f"${neu['d_15m'].abs().median():.2f} vs directional "
+              f"(n={len(dire)}) ${dire['d_15m'].abs().median():.2f} "
+              f"(null ${null_med['15m']:.2f})")
+
+    # breakdowns at 1h
+    dcol = "d_1h"
+    sub = df[df[dcol].notna() & (df["c_llm"] != 0)]
+    print("\n── LLM hit rate by event name (1h, top 10 by count) ──")
+    for name, grp in sorted(sub.groupby("name"), key=lambda kv: -len(kv[1]))[:10]:
+        hits = int((np.sign(grp[dcol]) == grp["c_llm"]).sum())
+        print("  " + _fmt_rate(hits, len(grp), name[:24]))
+    print("── LLM hit rate by year (1h) ──")
+    for yr, grp in sub.groupby("year"):
+        hits = int((np.sign(grp[dcol]) == grp["c_llm"]).sum())
+        print("  " + _fmt_rate(hits, len(grp), yr))
+
+    # cluster view: shared datelines
+    print("\n── same-timestamp clusters ──")
+    clus = df.groupby("dateline").filter(lambda g: len(g) > 1)
+    groups = list(clus.groupby("dateline"))
+    consistent = 0
+    net_hits, net_n = 0, 0
+    for _, g in groups:
+        signs = set(g.loc[g["c_llm"] != 0, "c_llm"])
+        if len(signs) <= 1:
+            consistent += 1
+        net = int(g["c_llm"].sum())
+        d = g["d_1h"].iloc[0]
+        if net != 0 and pd.notna(d) and d != 0:
+            net_n += 1
+            if np.sign(d) == np.sign(net):
+                net_hits += 1
+    print(f"clusters: {len(groups)} datelines "
+          f"({len(clus)} events, {len(clus) / max(len(df), 1) * 100:.0f}% of "
+          f"corpus) | internally consistent: {consistent}/{len(groups)}")
+    print("  " + _fmt_rate(net_hits, net_n, "cluster net-call (1h)"))
+
+    # determinism probe
+    p2 = {r["ffid"]: r for r in _load_jsonl(EVENTS_JSONL)
+          if r.get("run_pass") == 2 and r.get("llm_direction")}
+    if p2:
+        p1 = {r["ffid"]: r for r in recs}
+        agree = same_reason = n = 0
+        for fid, r2 in p2.items():
+            r1 = p1.get(fid)
+            if not r1 or not r1.get("llm_direction"):
+                continue
+            n += 1
+            if r1["llm_direction"] == r2["llm_direction"]:
+                agree += 1
+            if r1.get("reason_raw") == r2.get("reason_raw"):
+                same_reason += 1
+        print(f"\ndeterminism probe: direction agreement {agree}/{n}, "
+              f"identical reason text {same_reason}/{n}")
+
+    # stratified reason examples
+    print("\n── reason samples (1h hit/miss) ──")
+    sub = df[df[dcol].notna() & (df["c_llm"] != 0)].copy()
+    sub["hit"] = np.sign(sub[dcol]) == sub["c_llm"]
+    for verdict in (True, False):
+        picked, seen = [], set()
+        for _, r in sub[sub["hit"] == verdict].iterrows():
+            if r["name"] in seen:
+                continue
+            seen.add(r["name"])
+            picked.append(r)
+            if len(picked) == 10:
+                break
+        tag = "HIT " if verdict else "MISS"
+        for r in picked:
+            print(f"  [{tag}] {r['utc_time'][:16]} {r['name'][:30]:<30} "
+                  f"a={r['actual']} f={r['forecast']} → "
+                  f"{r['llm_direction']:<7} Δ1h={r[dcol]:+7.2f}  "
+                  f"“{(r['reason_raw'] or '')[:60]}”")
+    print(f"\nscored table → {SCORED_CSV}")
+
+
+def report_headlines():
+    recs = _load_jsonl(HEADLINES_JSONL)
+    if not recs:
+        print("\n(no headline replay records — run `headlines` first)")
+        return
+    g = _load_golden()
+    engine = NewsEngine(NewsConfig())
+
+    def collect(tag):
+        m = {}
+        for r in recs:
+            if r["run_tag"] == tag and r["scores"]:
+                for h, sc in zip(r["headlines"], r["scores"]):
+                    m[h] = tuple(sc)
+        return m
+
+    main, shuf = collect("main"), collect("shuffled")
+    print("\n" + "═" * 70)
+    print("PART B — golden headline set: LLM vs keyword baseline")
+    print("═" * 70)
+    missing = [h for h in g["headline"] if h not in main]
+    if missing:
+        print(f"WARNING: {len(missing)} headlines missing from main run")
+
+    def cls(x):
+        if x >= NEUTRAL_BAND:
+            return 1
+        if x <= -NEUTRAL_BAND:
+            return -1
+        return 0
+
+    rows = []
+    for _, r in g.iterrows():
+        h = r["headline"]
+        if h not in main:
+            continue
+        llm_xau = main[h][1]
+        kw = engine._analyze_headline_sentiment(h, for_currency="XAU")
+        rows.append({
+            "headline": h, "category": r["category"],
+            "exp": int(r["expected_xau_sign"]),
+            "llm_xau": llm_xau, "llm_cls": cls(llm_xau),
+            "kw": kw, "kw_cls": cls(kw),
+        })
+    t = pd.DataFrame(rows)
+    for who in ("llm", "kw"):
+        acc = (t[f"{who}_cls"] == t["exp"]).mean()
+        print(f"{who.upper():<4} 3-class accuracy (band ±{NEUTRAL_BAND}): "
+              f"{(t[f'{who}_cls'] == t['exp']).sum()}/{len(t)} "
+              f"= {acc * 100:.1f}%")
+    ha = list(t["llm_cls"] == t["exp"])
+    hb = list(t["kw_cls"] == t["exp"])
+    b, c, p = _mcnemar_p(ha, hb)
+    print(f"paired McNemar LLM vs keyword: b={b} c={c} p={p:.3f}")
+
+    print("\nper-category accuracy (LLM | keyword):")
+    for cat, grp in t.groupby("category"):
+        a1 = (grp["llm_cls"] == grp["exp"]).sum()
+        a2 = (grp["kw_cls"] == grp["exp"]).sum()
+        print(f"  {cat:<18} {a1:>2}/{len(grp):<2} | {a2:>2}/{len(grp)}")
+
+    print("\nLLM confusion (rows=expected, cols=predicted -1/0/+1):")
+    cm = pd.crosstab(t["exp"], t["llm_cls"]).reindex(
+        index=[-1, 0, 1], columns=[-1, 0, 1], fill_value=0)
+    print(cm.to_string())
+
+    print("\nLLM misclassified headlines:")
+    for _, r in t[t["llm_cls"] != t["exp"]].iterrows():
+        print(f"  exp {r['exp']:+d} got {r['llm_xau']:+.2f} "
+              f"[{r['category']}] {r['headline'][:70]}")
+
+    # repeat-batch determinism
+    m_b0 = [r for r in recs if r["run_tag"] == "main" and r["batch"] == 0]
+    r_b0 = [r for r in recs if r["run_tag"] == "repeat"]
+    if m_b0 and r_b0 and m_b0[0]["scores"] and r_b0[0]["scores"]:
+        a = np.array(m_b0[0]["scores"], dtype=float)
+        b_ = np.array(r_b0[0]["scores"], dtype=float)
+        exact = int((a == b_).all(axis=1).sum())
+        print(f"\nrepeat-batch: {exact}/{len(a)} score pairs identical, "
+              f"max |diff| = {np.abs(a - b_).max():.3f}")
+
+    # shuffled-order drift
+    if shuf:
+        both = [h for h in t["headline"] if h in shuf]
+        drift = [abs(main[h][1] - shuf[h][1]) for h in both]
+        if drift:
+            print(f"shuffled-order xau drift over {len(both)} headlines: "
+                  f"mean {np.mean(drift):.3f}, max {np.max(drift):.3f}")
+
+
+def cmd_report(args):
+    report_events()
+    report_headlines()
+
+
+# ── CLI ──────────────────────────────────────────────────────────────
+
+def main():
+    ap = argparse.ArgumentParser(description=__doc__.split("\n")[1])
+    sub = ap.add_subparsers(dest="cmd", required=True)
+
+    e = sub.add_parser("events", help="replay release surprises")
+    e.add_argument("--limit", type=int, default=0,
+                   help="smoke-test on first N events")
+    e.add_argument("--determinism", type=int, default=0,
+                   help="re-call N stratified events a second time")
+    e.add_argument("--sleep", type=float, default=0.3)
+    e.add_argument("--retry-passes", type=int, default=3)
+    e.add_argument("--timeout", type=float, default=30.0)
+    e.set_defaults(fn=cmd_events)
+
+    h = sub.add_parser("headlines", help="golden-set probes")
+    h.add_argument("--sleep", type=float, default=0.3)
+    h.add_argument("--timeout", type=float, default=30.0)
+    h.set_defaults(fn=cmd_headlines)
+
+    r = sub.add_parser("report", help="offline scoring report")
+    r.set_defaults(fn=cmd_report)
+
+    args = ap.parse_args()
+    args.fn(args)
+
+
+if __name__ == "__main__":
+    main()

backtest/research/llm_news_eval_2026-08-06.md

@@ -0,0 +1,86 @@
+# DeepSeek news-analysis validation — 2026-08-06
+
+**Verdict: headline sentiment scoring is excellent; event-direction calls are
+directionally sound but ~11% of them misread the numeric comparison, and the
+failure concentrates on Unemployment Claims / close K-M-suffixed values.**
+
+Method (owner-directed, token-lean): independent expert re-analysis of a
+stratified sample of DeepSeek's recorded answers, not a full statistical
+backtest. Harness: `backtest/research/llm_news_eval.py` (replays releases
+through the REAL `_llm_event_direction` / `_llm_headline_scores` — real
+prompts, payloads, validation; raw wire traffic logged). 135 of the 586-event
+corpus were replayed before the full run was stopped by owner request
+(artifacts retained in `llm_news_eval_2026-08/`, resume-safe).
+
+## Part A — event gold-direction (28-event stratified sample, Jan–Jun 2024)
+
+Reviewer agreement with DeepSeek: **24/28 (86%)**. All 4 disagreements are
+DeepSeek errors, in exactly two failure modes:
+
+1. **Numeric comparison misread (3/28 ≈ 11%)** — the doctrine logic is right
+   but the premise is backwards:
+   - JOLTS 8.79M vs f 8.84M (a miss) → DS: "Job openings **above** forecast" → bearish (should be bullish)
+   - Claims 202K vs f 217K (fewer claims) → DS: "Claims **higher** than forecast" → bullish (should be bearish)
+   - Claims 202K vs f 209K — same backwards read, same wrong call
+   Both Unemployment Claims events in the sample were misread the same way —
+   looks systematic on that event type, possibly an anchoring prior that
+   "claims rose". All misreads involve close values with K/M suffixes.
+2. **Direction field contradicts its own reason (1/28)** — Empire State
+   −43.7 vs f −4.9: reason text says "gold may rise", `gold_direction` says
+   `bearish`.
+
+Where the numbers were read correctly, hawkish/dovish→gold mapping was
+consistently correct (CPI/PPI/NFP/ADP/GDP/PMI/wages/unemployment-rate all
+handled properly, including inverted metrics when read right).
+
+Price cross-check (1h horizon, cluster-confounded, indicative only):
+reviewer calls 22/28 with the subsequent move, DeepSeek 20/28.
+
+## Part B — headline sentiment (78-headline golden set)
+
+DeepSeek matched the reviewer's expected gold-direction label on
+**78/78 (100%)** — including all 8 negation traps ("signals no rate hike",
+"no rate cuts until…") and all 6 inverted-metric phrasings ("claims hit
+two-year high"). The keyword scorer it overrides got **24/78 (31%)**.
+Repeat-batch and shuffled-order probes recorded (headlines_replay.jsonl).
+
+## Implications
+
+- The headline-sentiment override (the main live LLM surface) is validated —
+  a large, measured upgrade over keywords.
+- The event-direction enrichment ("— gold bullish/bearish" appended to
+  surprise descriptions) is right ~85–90% of the time but can state a
+  factually backwards comparison in the displayed reason. It is display-only
+  (never gates signals), so exposure is a possibly-misleading Telegram/CLI
+  line, not trades.
+- **Hardening SHIPPED same day (owner-approved):** the payload now carries a
+  code-computed `vs_forecast` field ("above/below/in line with forecast",
+  from `EconomicEvent.surprise`) and the prompt tells the model to trust it.
+  First iteration fixed the arithmetic but exposed a masked doctrine error
+  (model read "claims below forecast" correctly, then called it a *weaker*
+  labor market — and regressed the unemployment-rate case the same way), so
+  the prompt's inverted-metrics line was expanded to an explicit rule plus a
+  worked claims example. Verification: **18/18** — all 4 original misreads
+  fixed, 11/11 no-regression cases unchanged, 3/3 fresh probes on the side
+  the prompt example does NOT cover (claims 245K vs f 220K → bullish,
+  unemployment 4.1% vs 3.9% → bullish, JOLTS 9.10M vs 8.80M → bearish).
+  Suite green at 346 tests, incl. a new payload assertion in Module 19.
+- Production observations found while building the harness (not fixed):
+  `_llm_event_cache` key omits forecast → a repeated `actual` re-serves a
+  direction computed for a different forecast; a "neutral" reply is
+  indistinguishable from a failure in the return value.
+
+## Reproduce
+
+```
+venv/bin/python backtest/research/llm_news_eval.py events --limit 5   # smoke
+venv/bin/python backtest/research/llm_news_eval.py events --determinism 20
+venv/bin/python backtest/research/llm_news_eval.py headlines
+venv/bin/python backtest/research/llm_news_eval.py report             # offline
+```
+
+Corpus: 586 high-impact USD surprises 2024-01→2026-07 (calendar.parquet,
+NFP-verified UTC); prices: Oanda M15 mids. Full-run cost ≈ cents. The
+`report` subcommand (hit rates vs always-bullish/doctrine/FF-betterWorse
+baselines, McNemar, cluster view) is built and untested at scale — run it
+after completing the replay if a statistical answer is ever wanted.

backtest/research/llm_news_golden.csv

@@ -0,0 +1,79 @@
+headline,category,expected_xau_sign,rationale
+"Powell says restrictive policy needed for longer as inflation proves sticky",hawkish_fed,-1,"higher-for-longer rates lift real yields, gold-bearish"
+"Fed minutes show officials leaning toward two more rate hikes this year",hawkish_fed,-1,"more hikes = stronger dollar/yields"
+"Hot CPI print fuels bets on additional Fed tightening",hawkish_fed,-1,"tightening expectations weigh on gold"
+"Fed's Waller: higher-for-longer rates warranted by resilient economy",hawkish_fed,-1,"hawkish guidance"
+"FOMC raises rates 50bp, flags further increases ahead",hawkish_fed,-1,"outsized hike plus hawkish guidance"
+"Treasury yields spike as Fed pushes back on rate cut hopes",hawkish_fed,-1,"cut hopes dashed, yields up"
+"Core inflation accelerates, cementing case for hawkish Fed stance",hawkish_fed,-1,"hawkish repricing"
+"Fed dot plot shifts higher; markets price out easing for the year",hawkish_fed,-1,"easing priced out"
+"Fed cuts rates 25bp and signals more easing to come",dovish_fed,1,"easing cycle is gold-bullish"
+"Powell hints at rate cuts as inflation cools toward target",dovish_fed,1,"cut guidance"
+"FOMC minutes reveal growing support for policy easing",dovish_fed,1,"dovish minutes"
+"Fed's Williams says rate cuts appropriate later this year",dovish_fed,1,"cut guidance"
+"Soft inflation data boosts odds of a dovish Fed pivot",dovish_fed,1,"pivot odds up"
+"Fed pauses hikes, markets price aggressive easing cycle",dovish_fed,1,"easing priced in"
+"Central banks turn dovish as global growth falters",dovish_fed,1,"global easing backdrop"
+"Fed to restart bond purchases to support flagging economy",dovish_fed,1,"QE is gold-bullish"
+"Dollar index hits 12-month high on robust US economic data",strong_usd,-1,"strong dollar pressures gold"
+"US GDP grows 4.1% annualized, blowing past expectations",strong_usd,-1,"strong economy, hawkish implication"
+"Blockbuster jobs report sends dollar surging across the board",strong_usd,-1,"strong jobs lift dollar and yields"
+"US retail sales smash forecasts, underscoring consumer strength",strong_usd,-1,"strong data"
+"Greenback rallies as US data outshines Europe and Asia",strong_usd,-1,"dollar strength"
+"ISM services index jumps to two-year high on strong demand",strong_usd,-1,"strong data"
+"Consumer confidence surges to post-pandemic high in the US",strong_usd,-1,"strong data"
+"Dollar tumbles to eight-month low after dismal manufacturing data",weak_usd,1,"weak dollar lifts gold"
+"US economy adds far fewer jobs than expected; dollar slides",weak_usd,1,"weak jobs, dovish implication"
+"Greenback weakens as US retail sales unexpectedly contract",weak_usd,1,"weak data"
+"US GDP growth revised down sharply, stoking slowdown fears",weak_usd,1,"slowdown is dovish"
+"Dollar slumps as trade deficit balloons to record",weak_usd,1,"dollar weakness"
+"Soft services PMI adds to evidence of cooling US economy",weak_usd,1,"cooling economy"
+"US consumer sentiment plunges to two-year low",weak_usd,1,"weak data"
+"Real yields climb to multi-year highs as TIPS sell off",real_yields,-1,"rising real yields raise gold's opportunity cost"
+"10-year Treasury yield surges past 5% on supply glut",real_yields,-1,"yield spike"
+"Rising real interest rates dim appeal of non-yielding assets",real_yields,-1,"explicit opportunity-cost story"
+"Treasury yields sink as investors flock to bonds",real_yields,1,"falling yields support gold"
+"Real yields turn negative as inflation outpaces rates",real_yields,1,"negative real yields are gold's best backdrop"
+"Bond rally drives 10-year yield to lowest level in a year",real_yields,1,"falling yields"
+"Geopolitical tensions escalate in Middle East, investors seek safety",safe_haven,1,"safe-haven bid"
+"Stocks crater worldwide as banking crisis fears spread",safe_haven,1,"crisis flight to safety"
+"Escalating conflict drives flight to safe-haven assets",safe_haven,1,"explicit haven flow"
+"Investors dump risk assets after surprise sovereign default",safe_haven,1,"default shock"
+"Military escalation on NATO border rattles global markets",safe_haven,1,"geopolitical stress"
+"Wall Street plunges 4% in worst session since pandemic",safe_haven,1,"risk-off"
+"Currency crisis in emerging markets sparks contagion fears",safe_haven,1,"contagion, haven bid"
+"Nuclear rhetoric intensifies, sending investors to havens",safe_haven,1,"extreme geopolitical stress"
+"Ceasefire agreement lifts global markets, haven demand fades",risk_on,-1,"haven unwind"
+"Stocks roar to record highs as trade deal signed",risk_on,-1,"risk-on drains haven demand"
+"Risk appetite returns as banking fears subside",risk_on,-1,"risk-on"
+"Global equities rally on strong earnings and easing tensions",risk_on,-1,"risk-on"
+"Gold hits record high as central banks keep buying bullion",gold_direct,1,"direct demand"
+"Gold ETFs post biggest weekly inflows in three years",gold_direct,1,"investment inflows"
+"China's central bank extends gold buying spree to 18th month",gold_direct,1,"official-sector demand"
+"Miners struggle to meet gold demand as supply deficit widens",gold_direct,1,"supply deficit"
+"Gold slides 2% as investors book profits after record run",gold_direct,-1,"profit-taking pressure"
+"Gold tumbles below key support as ETF outflows accelerate",gold_direct,-1,"outflows"
+"Bullion demand weakens as Indian jewellery buying stalls",gold_direct,-1,"physical demand soft"
+"Speculators slash bullish gold bets as dollar strengthens",gold_direct,-1,"positioning unwind"
+"Fed holds rates steady, signals no rate hike on the horizon",negation_traps,1,"dovish hold; keyword layer sees the phrase rate hike and flips it gold-bearish"
+"Powell says no rate cuts until inflation shows sustained cooling",negation_traps,-1,"hawkish hold; keyword layer sees rate cut and calls it gold-bullish"
+"Recession fears fade as US data streak beats estimates",negation_traps,-1,"fading fear is risk-on; keywords see recession plus beats and cancel out"
+"Inflation fails to accelerate, easing pressure on the Fed",negation_traps,1,"disinflation opens door to cuts"
+"Fed's Bowman: not yet confident inflation is beaten, cuts premature",negation_traps,-1,"hawkish pushback phrased around negations"
+"Central bank stops short of hiking, but warns tightening may resume",negation_traps,-1,"hawkish hold with tightening bias"
+"No safe-haven bid for gold despite equity wobble, analysts say",negation_traps,-1,"explicitly negated haven demand"
+"Jobs report not as weak as feared, dollar recovers",negation_traps,-1,"less-bad data is dollar-positive"
+"US manufacturing output plunges as new orders dry up",inverted_metrics,1,"weak economy is dovish/gold-bullish; keyword layer reads plunges as bearish"
+"US factory activity slumps to three-year low",inverted_metrics,1,"weak data; keyword reads slumps as bearish"
+"Consumer spending falls for third straight month",inverted_metrics,1,"weak data; keyword reads falls as bearish"
+"Private payrolls growth slides to weakest pace since 2020",inverted_metrics,1,"weak jobs; keyword reads slides as bearish"
+"Jobless claims hit two-year high as layoffs spread",inverted_metrics,1,"higher claims = weaker economy = gold-bullish"
+"US unemployment rate hits 4.5%, highest since 2021",inverted_metrics,1,"higher unemployment = dovish = gold-bullish"
+"Lakers clinch playoff berth with overtime win over Suns",distractors,0,"irrelevant"
+"New Marvel film smashes box office records on opening weekend",distractors,0,"irrelevant"
+"Apple unveils thinner MacBook with faster chip lineup",distractors,0,"irrelevant"
+"Scientists discover new deep-sea species off Pacific coast",distractors,0,"irrelevant"
+"Heavy rain forecast for the Midwest through the weekend",distractors,0,"irrelevant"
+"Streaming service raises subscription prices in Europe",distractors,0,"irrelevant"
+"Electric vehicle startup recalls 12,000 cars over software glitch",distractors,0,"irrelevant"
+"Local elections see record turnout in several US states",distractors,0,"irrelevant"

backtest/research/news_sentiment_value_2026-09-04.md

@@ -0,0 +1,129 @@
+# News sentiment vs. price — does the news score carry information? — 2026-09-04
+
+**Source evidence:** `logs/signals.db` 2026-08-13 → 2026-09-03 (14,683 rows, 13,947 with
+a price; candle snapshots from 08-18, full-shape headline blobs from 08-19), read with
+`mode=ro`. A continuous H1/M15/M5 candle stream was rebuilt from the stored 14-bar
+`candles_json` snapshots (H1: 293 bars, 08-18 05:00 → 09-03 22:00). Headlines were
+resolved through `news_snapshots` (4,054 distinct headlines, 2,024 gold-tagged).
+Outcomes are the repo's own advisory model (`advisory_r_multiple` on the 144 `RESOLVED`
+rows, R at $0.65). One-year context from the Oanda H1 parquet
+(`backtest/cache/oanda_XAU_USD_1h_20250814_20260814.parquet`). No backtest: the
+backtester runs with news off, and no headline history exists beyond these 3 weeks, so a
+replay cannot test news. The same check reruns with
+`python analyze_signal_log.py --stored-candles --news`.
+
+Question asked by the owner: the log reads bullish most of the time — what did the
+indicators and the news say when gold fell, and is news scoring a waste?
+
+## Findings
+
+**The numeric news score is a rear-view mirror.** `news_sentiment` (= mean gold-headline
+score − mean dollar-headline score over the last 12 h) tracks what gold did over the
+*previous* day and says nothing stable about the next one.
+
+| hourly mean sentiment vs. gold return (n≈293 h) | Pearson |
+|---|---|
+| past 24 h | **+0.67** |
+| past 8 h / 4 h / 1 h | +0.32 / +0.20 / +0.07 |
+| next 1 h / 4 h / 8 h / 24 h | +0.05 / +0.10 / +0.21 / +0.14 |
+| next 8 h, first half (08-18→25) / second half (08-26→09-03) | **−0.37 / +0.12** |
+| next 24 h, first half / second half | −0.12 / −0.09 |
+| daily mean vs same-day / previous-day / next-day move | +0.39 / +0.37 / **+0.03** |
+| *change* in sentiment vs next 4 h / 8 h / 24 h | −0.09 / −0.03 / −0.01 |
+
+Partial correlation with the next 8 h after removing the past-24 h return is +0.26 pooled
+but flips sign between halves. Alternative aggregates (gold-specific score only, dropping
+price-report headlines, only headlines < 2 h old) show the same shape: past-24 h
+correlation 0.59–0.82, no stable forward correlation. A naive "hold in the direction of
+sentiment" rule made +$272 over 279 hours vs +$120 buy-and-hold, but trading the *change*
+in sentiment lost −$200: the gain is trend persistence (sentiment ≈ trailing return), not
+news.
+
+**Why the log reads bullish.** 62% of rows have sentiment ≥ +0.1, 21% ≤ −0.1. Component
+means: gold headlines +0.158, dollar headlines +0.048. The window was a 16% August rally,
+so the average headline was bullish because price had been rising. Of the gold-tagged
+headlines, 42% merely restate the chart (price-up reports 489, mean +0.37; price-down 366,
+mean −0.18) and 18% are long-horizon forecast pieces ("$5,400 target", "$50,000/oz"; 60%
+positive, 21% of all positive score mass). The per-headline LLM labels are correct — every
+regex-flagged "mis-score" was a correctly scored "yields fall → gold up" headline — which
+matches the 78/78 golden set of 2026-08-06. The averaging, the 12 h window and the
+headline universe are what destroy the information.
+
+**What news and indicators said on the way down** (top $4,697 on 08-25 00:00 UTC, low
+$4,283 on 09-02 03:00; an 8.8% drop in a week):
+
+| moment | price | news sentiment | engine (M15) |
+|---|---|---|---|
+| top, 08-25 00:00 | 4,697 | +0.30, rising to **+0.44** two hours after the top (window high) | ranging, composite +5.9; 9 BUY setups that day: −7.4R, 11% win |
+| 08-26/27 | −$60 | still +0.15…+0.37; first bearish hour 08-26 20:00, **44 h after the top**, price already −$102 | trending_down 134/138 rows, composite −4.6/−7.8; SELL setups 0/3 on 08-27 |
+| 08-28 Warsh speech 13:00–16:00 | −$122 in 3 h, −$140 day | +0.24…+0.30 overnight, **0.00 at 13:00**, −0.17 only at 16:00; summary all morning: "a hawkish tone could pressure gold, dovish hints may lift it" | composite −6.2; SELL setups 4/4, +3.3R |
+| 09-01 | −$128 day | −0.04…−0.20, "gold under pressure" all day (correct, describing the move) | trending_down 275 rows, composite −12.2, 51 SELL rows; SELL setups 7/7, +3.5R |
+| low, 09-02 03:00 | 4,283 | −0.16 at the low, −0.19 six hours later; first bullish hour 09-02 19:00, **16 h after the low**, price already +$108 | SELL setups at the bottom 0/2, −2.05R |
+
+Sentiment peaked at the top and troughed at the bottom. The engine also lagged (it is
+trend-following) but flipped regime and composite within hours; its SELL setups won on the
+trend days and lost at the turns, the same as its BUYs.
+
+**Outcomes** (144 RESOLVED, +37.8R): BUY 121 / +43.2R / 70% win; SELL 23 / −5.4R / 48%.
+
+| direction × sentiment bucket (≥+0.15 bull, ≤−0.05 bear) | n | R | mean | win |
+|---|---|---|---|---|
+| BUY, bullish news | 81 | +32.0 | +0.40 | 73% |
+| BUY, neutral | 34 | +10.6 | +0.31 | 68% |
+| BUY, bearish news | 6 | +0.65 | +0.11 | 50% |
+| SELL, bearish news | 14 | +0.13 | +0.01 | 71% |
+| SELL, neutral | 5 | −1.46 | −0.29 | 20% |
+| SELL, bullish news | 4 | −4.08 | −1.02 | 0% |
+
+The splits are the trend confound again ("BUY with bullish news" = "BUY in an uptrend").
+Cleanest counter-example: 08-25, the most bullish news day of the drop period (+0.25),
+9 BUYs at the top lost −7.4R. corr(sentiment, R): BUY +0.10, SELL −0.45 (n=23).
+
+**Macro persona** (reads the same headlines plus high-impact events): macro=support 51
+BUYs → +32.6R, 90% win; macro=neutral 16 → −3.2R, 38%. Within bullish-sentiment BUYs:
+support 42 → +25.2R (88%) vs neutral 7 → −2.9R (29%). On 08-25 all 7 panel rows were
+macro=neutral ("awaiting PCE, mixed") while the numeric aggregate said +0.25 bullish.
+Confounded with regime (64 of 84 supports were trending_up) and n≈16 on the informative
+side. Not enough to promote or demote; the standing rule (re-score at ~300 outcomes)
+applies.
+
+**Event risk** cannot be evaluated here: only 4 event windows in the sample
+(08-19 17:00, 08-26 12:00, 08-28 13:00, 09-01 13:00), none produced a setup. The standing
+evidence remains the 2026-07-11 audit (event gates would cost +4.6…+7.2R/yr).
+
+**One-year context:** daily moves ≤ −$100 happened 17 times in the year to 2026-08-14;
+15 of 47 rolling 3-week windows contained a ≥ 8% drawdown. This sample is one ordinary
+rally-then-drop regime.
+
+**Simulated 4 h window + neutral drop** (the change shipped with this note, on 11,136
+full-shape scans): never leaves a scan empty (median 19 directional headlines, 8
+gold-specific; worst hour 02:00 UTC has < 3 gold headlines 28% of the time), the aggregate
+correlates 0.78 with the old one and becomes bolder (share |score| ≥ 0.3 rises 17% → 41%),
+with the same lack of forward information (next-24 h corr +0.20; next-8 h halves
+−0.22 / −0.10).
+
+## Method
+
+- Hourly join convention: a scan in hour H (boundary + 10 s) sees the bar closed at H,
+  i.e. bar H's open; forward returns run from bar H's open, past returns end there.
+- Correlations are Pearson on hourly means (no scipy); halves split at 2026-08-26 00:00.
+- Headline classes by regex on the title (price-up / price-down / forecast-style); the
+  gold-specific score is `score_by_currency.XAU` where present, else `score`.
+- Outcome buckets use the representative row's `news_sentiment`; macro stance from
+  `panel_takes_json`.
+- Everything above reproduces from the DB alone; nothing depends on live feeds.
+
+## Decisions
+
+- `sentiment_affects_signals` and `event_risk_caps_signals` stay `False`. No gate is
+  built in either direction (follow or fade): 3 weeks, one regime.
+- The headline labelling is not the problem; the pool and the averaging are. Shipped the
+  same day: `headline_max_age_seconds` 12 h → 4 h with undated items dropped, and
+  `get_fundamental_context` drops headlines whose every relevant score is inside ±0.1
+  (`_is_neutral_headline`). The market-summary prompt now writes for someone who has
+  never traded, sends headlines newest-first with the platform's own gold direction, and
+  caps at 400 chars. **`news_sentiment` rows from the deploy date onward are a mean over
+  directional ≤ 4 h headlines — log comparisons must split at that date.**
+- Keep the macro persona running and logging; re-check with `--news` and the persona
+  scorecard at ~300 outcomes.
+- Refuted hypothesis, do not re-propose without new data: "news sentiment leads price".

backtest/research/oanda_td_divergence.py

@@ -0,0 +1,199 @@
+"""
+Oanda vs Twelve Data bar-divergence study — XAU/USD M15, 12 months.
+====================================================================
+Compares the mid-price OHLC bars the signal engine would see from each
+provider over the same window. Purpose: quantify whether migrating the
+signal feed from Twelve Data to Oanda is a formality (bars ~identical)
+or a recalibration (bars differ at a scale that moves indicators).
+
+Inputs:
+  --td      Twelve Data parquet (raw bars from backtest/cache/)
+  --oanda   Oanda parquet (from fetch_oanda_m5.py / fetch_oanda_m15.py)
+  --start / --end  analysis window (UTC dates)
+
+Outputs:
+  printed report
+  backtest/research/oanda_td_divergence_by_hour.csv
+  backtest/research/oanda_td_bar_diffs.parquet  (per-bar diffs, common index)
+"""
+
+import argparse
+from pathlib import Path
+
+import numpy as np
+import pandas as pd
+
+RESEARCH_DIR = Path(__file__).parent
+PIP = 0.10  # XAU pip in $
+
+
+def wilder_atr(df: pd.DataFrame, period: int = 14) -> pd.Series:
+    hl = df["High"] - df["Low"]
+    hc = (df["High"] - df["Close"].shift(1)).abs()
+    lc = (df["Low"] - df["Close"].shift(1)).abs()
+    tr = pd.concat([hl, hc, lc], axis=1).max(axis=1)
+    return tr.ewm(alpha=1.0 / period, adjust=False).mean()
+
+
+def pct_stats(s: pd.Series) -> dict:
+    a = s.abs()
+    return {
+        "mean": a.mean(), "median": a.median(),
+        "p90": a.quantile(0.90), "p95": a.quantile(0.95),
+        "p99": a.quantile(0.99), "max": a.max(),
+        "bias(mean signed)": s.mean(),
+    }
+
+
+def fmt_stats(name: str, s: pd.Series) -> str:
+    st = pct_stats(s)
+    return (f"  {name:<6} mean ${st['mean']:.3f}  med ${st['median']:.3f}  "
+            f"p95 ${st['p95']:.3f}  p99 ${st['p99']:.3f}  max ${st['max']:.2f}  "
+            f"bias {st['bias(mean signed)']:+.4f}")
+
+
+def main():
+    ap = argparse.ArgumentParser()
+    ap.add_argument("--td", default=str(RESEARCH_DIR.parent / "cache" /
+                                        "XAU_USD_15min_20240522_20260723.parquet"))
+    ap.add_argument("--oanda", default=str(RESEARCH_DIR /
+                                           "oanda_xau_m15_20250701_now.parquet"))
+    ap.add_argument("--start", default="2025-07-24")
+    ap.add_argument("--end", default="2026-07-23")
+    args = ap.parse_args()
+
+    td = pd.read_parquet(args.td)
+    oa = pd.read_parquet(args.oanda)
+
+    start = pd.Timestamp(args.start, tz="UTC")
+    end = pd.Timestamp(args.end, tz="UTC") + pd.Timedelta(days=1)
+    start = max(start, td.index[0], oa.index[0])
+    end = min(end, td.index[-1], oa.index[-1])
+    td = td.loc[(td.index >= start) & (td.index <= end)]
+    oa = oa.loc[(oa.index >= start) & (oa.index <= end)]
+
+    print("=" * 78)
+    print("OANDA vs TWELVE DATA — XAU/USD M15 bar divergence")
+    print(f"Window: {start} -> {end}  ({(end - start).days} days)")
+    print("=" * 78)
+
+    # ── Coverage ─────────────────────────────────────────────────────
+    common = td.index.intersection(oa.index)
+    td_only = td.index.difference(oa.index)
+    oa_only = oa.index.difference(td.index)
+    print(f"\nCOVERAGE")
+    print(f"  Twelve Data bars : {len(td):>7,}")
+    print(f"  Oanda bars       : {len(oa):>7,}")
+    print(f"  common           : {len(common):>7,}")
+    print(f"  TD-only          : {len(td_only):>7,}  "
+          f"({100 * len(td_only) / max(len(td), 1):.2f}% of TD)")
+    print(f"  Oanda-only       : {len(oa_only):>7,}  "
+          f"({100 * len(oa_only) / max(len(oa), 1):.2f}% of Oanda)")
+    for label, idx in (("TD-only", td_only), ("Oanda-only", oa_only)):
+        if len(idx):
+            by_date = pd.Series(1, index=idx).groupby(idx.date).sum()
+            top = by_date.sort_values(ascending=False).head(8)
+            print(f"  {label} bars cluster on: "
+                  + ", ".join(f"{d} ({n})" for d, n in top.items()))
+            by_hour = pd.Series(1, index=idx).groupby(idx.hour).sum()
+            top_h = by_hour.sort_values(ascending=False).head(6)
+            print(f"  {label} by UTC hour: "
+                  + ", ".join(f"{h:02d}h ({n})" for h, n in top_h.items()))
+
+    t = td.loc[common]
+    o = oa.loc[common]
+
+    # ── Scale references ─────────────────────────────────────────────
+    atr = wilder_atr(t).dropna()
+    mean_price = t["Close"].mean()
+    print(f"\nSCALE REFERENCES (Twelve Data feed)")
+    print(f"  mean close      : ${mean_price:,.2f}")
+    print(f"  mean bar range  : ${(t['High'] - t['Low']).mean():.3f}")
+    print(f"  mean ATR14      : ${atr.mean():.3f}   median ${atr.median():.3f}")
+
+    # ── Per-field diffs (TD − Oanda mid) ─────────────────────────────
+    print(f"\nPER-BAR DIFFS  (TD − Oanda, $; pip = $0.10)")
+    diffs = {}
+    for col in ("Open", "High", "Low", "Close"):
+        d = t[col] - o[col]
+        diffs[col] = d
+        print(fmt_stats(col, d))
+
+    dclose = diffs["Close"].abs()
+    print(f"\n  |ΔClose| as % of mean ATR14 : "
+          f"mean {100 * dclose.mean() / atr.mean():.1f}%   "
+          f"p95 {100 * dclose.quantile(0.95) / atr.mean():.1f}%")
+
+    print(f"\nTHRESHOLD EXCEEDANCE  (|ΔClose|)")
+    for thr in (0.10, 0.30, 0.65, 1.00, 2.00, 5.00):
+        pct = 100 * (dclose > thr).mean()
+        print(f"  > ${thr:<5.2f}: {pct:6.2f}% of bars  ({(dclose > thr).sum():,})")
+
+    # ── Direction / return agreement ─────────────────────────────────
+    t_ret = t["Close"].pct_change()
+    o_ret = o["Close"].pct_change()
+    # only consecutive common bars (gap = one M15 step)
+    step_ok = pd.Series(common, index=common).diff().eq(pd.Timedelta(minutes=15))
+    tr, orr = t_ret[step_ok], o_ret[step_ok]
+    corr = tr.corr(orr)
+    t_dir = np.sign(t["Close"] - t["Open"])
+    o_dir = np.sign(o["Close"] - o["Open"])
+    both_nonflat = (t_dir != 0) & (o_dir != 0)
+    dir_agree = 100 * (t_dir[both_nonflat] == o_dir[both_nonflat]).mean()
+    print(f"\nAGREEMENT")
+    print(f"  close-return correlation (consecutive bars): {corr:.5f}  "
+          f"(n={step_ok.sum():,})")
+    print(f"  bar direction (sign of close−open) agree   : {dir_agree:.2f}%")
+
+    # ── By-hour breakdown ────────────────────────────────────────────
+    hours = common.hour
+    rows = []
+    for h in range(24):
+        m = hours == h
+        if not m.any():
+            continue
+        row = {"hour_utc": h, "bars": int(m.sum()),
+               "mean_abs_dclose": dclose[m].mean(),
+               "p95_abs_dclose": dclose[m].quantile(0.95),
+               "mean_abs_dhigh": diffs["High"][m].abs().mean(),
+               "mean_abs_dlow": diffs["Low"][m].abs().mean()}
+        if "bid_c" in o.columns:
+            row["oanda_median_spread"] = (
+                (o["ask_c"] - o["bid_c"])[m].median())
+        rows.append(row)
+    by_hour = pd.DataFrame(rows)
+    out_hour = RESEARCH_DIR / "oanda_td_divergence_by_hour.csv"
+    by_hour.to_csv(out_hour, index=False, float_format="%.4f")
+    print(f"\nBY UTC HOUR  (written to {out_hour.name})")
+    print(by_hour.to_string(index=False,
+                            float_format=lambda x: f"{x:.3f}"))
+
+    # ── By-month drift ───────────────────────────────────────────────
+    print(f"\nBY MONTH  (mean |ΔClose|, $)")
+    by_month = dclose.groupby(common.to_period("M")).mean()
+    for p, v in by_month.items():
+        print(f"  {p}: {v:.3f}")
+
+    # ── Oanda real spread (bonus, ties to Dukascopy study) ───────────
+    if "bid_c" in o.columns:
+        spread = o["ask_c"] - o["bid_c"]
+        liquid = ~o.index.hour.isin([21, 22, 23, 0])
+        print(f"\nOANDA REAL SPREAD (close bid/ask, $)")
+        print(f"  all bars      : median {spread.median():.3f}  "
+              f"p25 {spread.quantile(0.25):.3f}  p75 {spread.quantile(0.75):.3f}")
+        print(f"  liquid hours  : median {spread[liquid].median():.3f}")
+        print(f"  21-00 UTC     : median {spread[~liquid].median():.3f}")
+
+    # ── Persist per-bar diffs ────────────────────────────────────────
+    out = pd.DataFrame({
+        "td_close": t["Close"], "oa_close": o["Close"],
+        "d_open": diffs["Open"], "d_high": diffs["High"],
+        "d_low": diffs["Low"], "d_close": diffs["Close"],
+    })
+    out_path = RESEARCH_DIR / "oanda_td_bar_diffs.parquet"
+    out.to_parquet(out_path)
+    print(f"\nPer-bar diffs written to {out_path}")
+
+
+if __name__ == "__main__":
+    main()

backtest/research/oanda_td_divergence_by_hour.csv

@@ -0,0 +1,25 @@
+hour_utc,bars,mean_abs_dclose,p95_abs_dclose,mean_abs_dhigh,mean_abs_dlow,oanda_median_spread
+0,1029,3.3483,13.9460,3.2906,3.4310,0.7900
+1,1028,3.3483,12.8292,3.3477,3.6398,0.7700
+2,1028,3.0288,10.8960,2.9469,3.1317,0.7700
+3,1028,2.8783,11.1300,2.8419,2.9992,0.7600
+4,1028,2.8232,10.4710,2.8165,2.8455,0.7400
+5,1028,2.9268,10.6027,2.9234,3.0038,0.7400
+6,1028,3.0466,10.9265,2.9487,3.0796,0.7200
+7,1028,3.0286,11.3512,3.0301,3.0797,0.6700
+8,1028,3.0223,11.1483,2.8989,3.0801,0.6600
+9,1028,3.0460,11.6345,2.9307,3.0609,0.6600
+10,1028,2.9280,10.9015,2.8647,3.0029,0.6600
+11,1028,3.0340,11.4712,2.9591,2.9472,0.6600
+12,1028,3.0986,11.3282,2.9412,3.1407,0.7000
+13,1028,3.4014,12.3343,3.1077,3.4228,0.7300
+14,1028,3.5074,11.3628,3.3960,3.6569,0.6900
+15,1028,3.4514,11.8982,3.4124,3.6733,0.6700
+16,1028,3.1275,12.5155,3.2038,3.3061,0.6700
+17,1020,2.9826,12.6435,3.0064,3.1008,0.6900
+18,1015,2.8721,12.5480,2.9101,2.9996,0.6900
+19,1004,2.8637,12.1335,2.8824,2.9454,0.6900
+20,996,2.8006,11.9325,2.8486,2.8577,0.7600
+21,335,1.1085,4.8600,1.1212,1.2034,0.9400
+22,676,4.0827,13.6787,4.8422,4.6761,0.8300
+23,1027,3.2722,12.5725,3.5617,3.7747,0.8700

backtest/research/panel_payload_drytest_2026-08-15.jsonl

@@ -0,0 +1,7 @@
+{"probe": "P1 coherent-bullish", "payload": {"brief": {"pair": "XAU/USD", "timeframe": "15min", "direction": "BUY", "quality": "A", "score": 38.5, "confidence": 62, "price": 2000.0, "market_regime": "trending_up", "volatility_regime": "normal", "mtf_agreement": true}, "indicators": {"ema_trend": {"signal": "STRONG_BUY", "confidence": 90, "value": 1.0}, "rsi": {"signal": "BUY", "confidence": 65, "value": 58.0}, "macd": {"signal": "STRONG_BUY", "confidence": 80, "value": 1.25}, "bollinger": {"signal": "BUY", "confidence": 60, "value": 0.7}, "adx": {"signal": "BUY", "confidence": 70, "value": 32.0}, "roc": {"signal": "BUY", "confidence": 55, "value": 0.35}, "keltner": {"signal": "BUY", "confidence": 65, "value": 3.1}, "candlestick": {"signal": "BUY", "confidence": 60, "value": 2.0}}, "mtf": {"5min": {"signal": "BUY", "score": 30.0}, "1h": {"signal": "STRONG_BUY", "score": 45.0}}, "key_levels": {"resistance": 2012.0, "support": 1985.0}, "exhaustion": 0, "candles": {"15min": {"ref_close": 2000.0, "atr": 5.0, "bars": [[-2.4, -2.2, -2.4, -2.3], [-2.3, -2.1, -2.3, -2.2], [-2.2, -2.0, -2.2, -2.0], [-2.0, -1.9, -2.1, -1.9], [-1.9, -1.8, -2.0, -1.8], [-1.8, -1.6, -1.8, -1.7], [-1.7, -1.5, -1.7, -1.6], [-1.6, -1.4, -1.6, -1.4], [-1.4, -1.3, -1.5, -1.3], [-1.3, -1.2, -1.4, -1.2], [-1.2, -1.0, -1.2, -1.1], [-1.1, -0.9, -1.1, -1.0], [-1.0, -0.8, -1.0, -0.8], [-0.8, -0.7, -0.9, -0.7], [-0.7, -0.6, -0.8, -0.6], [-0.6, -0.4, -0.6, -0.5], [-0.5, -0.3, -0.5, -0.4], [-0.4, -0.2, -0.4, -0.2], [-0.2, -0.1, -0.3, -0.1], [-0.1, 0.0, -0.2, 0.0]]}}}, "raw_reply": {"stance": "support", "confidence": 70, "note": "The trend is up and all signs agree, so buying fits the flow."}}
+{"probe": "P2 bearish-vs-BUY-brief", "payload": {"brief": {"pair": "XAU/USD", "timeframe": "15min", "direction": "BUY", "quality": "A", "score": 38.5, "confidence": 62, "price": 2000.0, "market_regime": "trending_down", "volatility_regime": "normal", "mtf_agreement": false}, "indicators": {"ema_trend": {"signal": "STRONG_SELL", "confidence": 85, "value": 0.1}, "rsi": {"signal": "SELL", "confidence": 70, "value": 74.0}, "macd": {"signal": "STRONG_SELL", "confidence": 80, "value": -1.4}, "bollinger": {"signal": "SELL", "confidence": 60, "value": 0.95}, "adx": {"signal": "SELL", "confidence": 70, "value": 35.0}, "roc": {"signal": "SELL", "confidence": 60, "value": -0.5}, "keltner": {"signal": "SELL", "confidence": 65, "value": -2.8}, "candlestick": {"signal": "SELL", "confidence": 55, "value": -2.0}}, "mtf": {"5min": {"signal": "SELL", "score": -28.0}, "1h": {"signal": "STRONG_SELL", "score": -41.0}}, "key_levels": {"resistance": 2012.0, "support": 1985.0}, "exhaustion": 3, "candles": {"15min": {"ref_close": 2000.0, "atr": 5.0, "bars": [[2.4, 2.4, 2.2, 2.3], [2.3, 2.3, 2.1, 2.2], [2.2, 2.2, 2.0, 2.0], [2.0, 2.1, 1.9, 1.9], [1.9, 2.0, 1.8, 1.8], [1.8, 1.8, 1.6, 1.7], [1.7, 1.7, 1.5, 1.6], [1.6, 1.6, 1.4, 1.4], [1.4, 1.5, 1.3, 1.3], [1.3, 1.4, 1.2, 1.2], [1.2, 1.2, 1.0, 1.1], [1.1, 1.1, 0.9, 1.0], [1.0, 1.0, 0.8, 0.8], [0.8, 0.9, 0.7, 0.7], [0.7, 0.8, 0.6, 0.6], [0.6, 0.6, 0.4, 0.5], [0.5, 0.5, 0.3, 0.4], [0.4, 0.4, 0.2, 0.2], [0.2, 0.3, 0.1, 0.1], [0.1, 0.2, -0.0, 0.0]]}}}, "raw_reply": {"stance": "oppose", "confidence": 85, "note": "The price is falling and every indicator says sell, so buying fights the trend."}}
+{"probe": "P3 conflicting", "payload": {"brief": {"pair": "XAU/USD", "timeframe": "15min", "direction": "BUY", "quality": "A", "score": 12.0, "confidence": 35, "price": 2000.0, "market_regime": "ranging", "volatility_regime": "normal", "mtf_agreement": false}, "indicators": {"ema_trend": {"signal": "BUY", "confidence": 55, "value": 0.6}, "rsi": {"signal": "NEUTRAL", "confidence": 40, "value": 51.0}, "macd": {"signal": "SELL", "confidence": 55, "value": -0.3}, "bollinger": {"signal": "NEUTRAL", "confidence": 35, "value": 0.5}, "adx": {"signal": "NEUTRAL", "confidence": 40, "value": 18.0}, "roc": {"signal": "SELL", "confidence": 50, "value": -0.1}, "keltner": {"signal": "BUY", "confidence": 50, "value": 0.8}, "candlestick": {"signal": "NEUTRAL", "confidence": 30, "value": 0.0}}, "mtf": {"5min": {"signal": "NEUTRAL", "score": 4.0}, "1h": {"signal": "NEUTRAL", "score": -3.0}}, "key_levels": {}, "exhaustion": 0, "candles": {"15min": {"ref_close": 2000.0, "atr": 5.0, "bars": [[0.0, 0.4, -0.1, 0.3], [0.3, 0.4, -0.1, 0.0], [0.0, 0.4, -0.1, 0.3], [0.3, 0.4, -0.1, 0.0], [0.0, 0.4, -0.1, 0.3], [0.3, 0.4, -0.1, 0.0], [0.0, 0.4, -0.1, 0.3], [0.3, 0.4, -0.1, 0.0], [0.0, 0.4, -0.1, 0.3], [0.3, 0.4, -0.1, 0.0], [0.0, 0.4, -0.1, 0.3], [0.3, 0.4, -0.1, 0.0], [0.0, 0.4, -0.1, 0.3], [0.3, 0.4, -0.1, 0.0], [0.0, 0.4, -0.1, 0.3], [0.3, 0.4, -0.1, 0.0], [0.0, 0.4, -0.1, 0.3], [0.3, 0.4, -0.1, 0.0], [0.0, 0.4, -0.1, 0.3], [0.3, 0.4, -0.1, 0.0]]}}}, "raw_reply": {"stance": "neutral", "confidence": 30, "note": "Mixed signals with no clear direction, so cannot strongly support or oppose."}}
+{"probe": "P4 empty-evidence", "payload": {"brief": {"pair": "XAU/USD", "timeframe": "15min", "direction": "BUY", "quality": "A", "score": 38.5, "confidence": 62, "price": 2000.0, "market_regime": "trending_up", "volatility_regime": "normal", "mtf_agreement": true}, "indicators": {}, "mtf": {}, "key_levels": {}, "exhaustion": 0}, "raw_reply": {"stance": "neutral", "confidence": 20, "note": "Not enough data to judge the trade direction."}}
+{"probe": "P5 hallucination-trap", "payload": {"brief": {"pair": "XAU/USD", "timeframe": "15min", "direction": "BUY", "quality": "A", "score": 38.5, "confidence": 62, "price": 2000.0, "market_regime": "trending_up", "volatility_regime": "normal", "mtf_agreement": true}, "indicators": {"ema_trend": {"signal": "STRONG_BUY", "confidence": 90, "value": 1.0}, "rsi": {"signal": "BUY", "confidence": 65, "value": 58.0}, "macd": {"signal": "STRONG_BUY", "confidence": 80, "value": 1.25}, "bollinger": {"signal": "BUY", "confidence": 60, "value": 0.7}, "adx": {"signal": "BUY", "confidence": 70, "value": 32.0}, "roc": {"signal": "BUY", "confidence": 55, "value": 0.35}, "keltner": {"signal": "BUY", "confidence": 65, "value": 3.1}, "candlestick": {"signal": "BUY", "confidence": 60, "value": 2.0}}, "mtf": {"5min": {"signal": "BUY", "score": 30.0}, "1h": {"signal": "STRONG_BUY", "score": 45.0}}, "key_levels": {}, "exhaustion": 0}, "raw_reply": {"stance": "support", "confidence": 70, "note": "The trend is up and all signs agree, so buying fits the flow."}}
+{"probe": "P6 output-discipline", "payload": {}, "raw_reply": null}
+{"probe": "P7 synthesizer", "payload": {"direction": "BUY", "takes": [{"persona": "macro", "stance": "support", "confidence": 70, "note": "evidence aligned"}, {"persona": "technical", "stance": "support", "confidence": 80, "note": "evidence aligned"}, {"persona": "risk", "stance": "support", "confidence": 65, "note": "evidence aligned"}, {"persona": "contrarian", "stance": "support", "confidence": 60, "note": "evidence aligned"}]}, "raw_reply": {"verdict": "AGREE", "confidence": 85, "rationale": "All four analysts support buying, with high confidence and consistent evidence. No one opposes, so the panel clearly backs the trade."}}

backtest/research/panel_payload_drytest_2026-08-15.py

@@ -0,0 +1,271 @@
+#!/usr/bin/env python3
+"""
+panel_payload_drytest_2026-08-15.py — live DeepSeek validation of the
+enriched agent-panel payloads + hardened prompts (2026-08-15).
+
+Spends ~6 REAL DeepSeek requests (against the shared 500/day cap) with
+FABRICATED payloads — made-up prices/indicators, no real market data —
+to verify the new technical-persona contract end to end:
+
+  P1 coherent bullish evidence       -> technical persona SUPPORTS
+  P2 bearish evidence vs BUY brief   -> technical persona OPPOSES
+  P3 conflicting evidence            -> NEUTRAL or low confidence
+  P4 empty evidence                  -> NEUTRAL with confidence < 30
+                                        (the anti-hallucination contract)
+  P5 hallucination trap              -> note invents no price levels
+  P6 output discipline               -> <=20-word notes, exact JSON keys
+  P7 synthesizer over 4 supports     -> AGREE verdict
+
+Usage:  venv/bin/python backtest/research/panel_payload_drytest_2026-08-15.py
+Exit code 0 = all probes passed. Raw replies land in
+backtest/research/panel_drytest_2026-08-15.jsonl
+"""
+
+import json
+import re
+import sys
+from pathlib import Path
+
+import pandas as pd
+
+REPO_ROOT = Path(__file__).resolve().parents[2]
+sys.path.insert(0, str(REPO_ROOT))
+
+import agent_panel as ap                      # noqa: E402
+from app_config import deepseek_api_key, get_app_config  # noqa: E402
+from llm_client import get_llm_client         # noqa: E402
+
+ARTIFACT = Path(__file__).with_suffix(".jsonl")
+
+
+# ── Fabrication helpers (all numbers are invented) ───────────────────
+
+def _staircase(start: float, step: float, n: int) -> pd.DataFrame:
+    """n synthetic bars drifting `step` per bar; last close = start+n*step."""
+    rows, price = [], start
+    for _ in range(n):
+        o, c = price, price + step
+        rows.append({"Open": o, "Close": c,
+                     "High": max(o, c) + abs(step) * 0.4,
+                     "Low": min(o, c) - abs(step) * 0.4})
+        price = c
+    return pd.DataFrame(rows)
+
+
+def _choppy(center: float, n: int) -> pd.DataFrame:
+    rows, price, sign = [], center, 1
+    for _ in range(n):
+        step = sign * 1.5
+        o, c = price, price + step
+        rows.append({"Open": o, "Close": c,
+                     "High": max(o, c) + 0.6, "Low": min(o, c) - 0.6})
+        price, sign = c, -sign
+    return pd.DataFrame(rows)
+
+
+_FAKE_PRICE = 2000.00          # deliberately made-up
+_FAKE_ATR = 5.0
+
+_BRIEF_BUY = {"pair": "XAU/USD", "timeframe": "15min", "direction": "BUY",
+              "quality": "A", "score": 38.5, "confidence": 62,
+              "price": _FAKE_PRICE, "market_regime": "trending_up",
+              "volatility_regime": "normal", "mtf_agreement": True}
+
+_BULL_INDICATORS = {
+    "ema_trend": {"signal": "STRONG_BUY", "confidence": 90, "value": 1.0},
+    "rsi": {"signal": "BUY", "confidence": 65, "value": 58.0},
+    "macd": {"signal": "STRONG_BUY", "confidence": 80, "value": 1.25},
+    "bollinger": {"signal": "BUY", "confidence": 60, "value": 0.7},
+    "adx": {"signal": "BUY", "confidence": 70, "value": 32.0},
+    "roc": {"signal": "BUY", "confidence": 55, "value": 0.35},
+    "keltner": {"signal": "BUY", "confidence": 65, "value": 3.1},
+    "candlestick": {"signal": "BUY", "confidence": 60, "value": 2.0},
+}
+
+_BEAR_INDICATORS = {
+    "ema_trend": {"signal": "STRONG_SELL", "confidence": 85, "value": 0.1},
+    "rsi": {"signal": "SELL", "confidence": 70, "value": 74.0},
+    "macd": {"signal": "STRONG_SELL", "confidence": 80, "value": -1.4},
+    "bollinger": {"signal": "SELL", "confidence": 60, "value": 0.95},
+    "adx": {"signal": "SELL", "confidence": 70, "value": 35.0},
+    "roc": {"signal": "SELL", "confidence": 60, "value": -0.5},
+    "keltner": {"signal": "SELL", "confidence": 65, "value": -2.8},
+    "candlestick": {"signal": "SELL", "confidence": 55, "value": -2.0},
+}
+
+_MIXED_INDICATORS = {
+    "ema_trend": {"signal": "BUY", "confidence": 55, "value": 0.6},
+    "rsi": {"signal": "NEUTRAL", "confidence": 40, "value": 51.0},
+    "macd": {"signal": "SELL", "confidence": 55, "value": -0.3},
+    "bollinger": {"signal": "NEUTRAL", "confidence": 35, "value": 0.5},
+    "adx": {"signal": "NEUTRAL", "confidence": 40, "value": 18.0},
+    "roc": {"signal": "SELL", "confidence": 50, "value": -0.1},
+    "keltner": {"signal": "BUY", "confidence": 50, "value": 0.8},
+    "candlestick": {"signal": "NEUTRAL", "confidence": 30, "value": 0.0},
+}
+
+
+def _tech_payload(indicators, mtf, candles_df=None, key_levels=None,
+                  brief=None, exhaustion=0):
+    payload = {"brief": brief or dict(_BRIEF_BUY),
+               "indicators": indicators, "mtf": mtf,
+               "key_levels": key_levels or {}, "exhaustion": exhaustion}
+    if candles_df is not None:
+        # the REAL production normalizer, not a hand-rolled copy
+        payload["candles"] = {
+            "15min": ap._normalize_candles(candles_df, _FAKE_ATR,
+                                           len(candles_df))}
+    return payload
+
+
+# ── Probe runner ─────────────────────────────────────────────────────
+
+_RESULTS = []
+
+
+def _record(probe, payload, reply, take, passed, detail):
+    _RESULTS.append({"probe": probe, "pass": bool(passed),
+                     "detail": detail,
+                     "stance": getattr(take, "stance", None),
+                     "confidence": getattr(take, "confidence", None),
+                     "note": getattr(take, "note", None)})
+    with ARTIFACT.open("a") as fh:
+        fh.write(json.dumps({"probe": probe, "payload": payload,
+                             "raw_reply": reply}, default=str) + "\n")
+    mark = "PASS" if passed else "FAIL"
+    print(f"[{mark}] {probe}: {detail}")
+    if take is not None:
+        print(f"       -> {take.stance} @ {take.confidence:.0f} "
+              f"| {take.note}")
+
+
+def _ask(payload):
+    """Real API call through the production client + production
+    validation (the same path run_panel uses)."""
+    reply = get_llm_client().chat_json(ap._TECH_SYSTEM, payload)
+    return reply, ap._validate_take("technical", reply)
+
+
+def _want(probe: str) -> bool:
+    """--only P3 runs just that probe (saves requests on re-runs)."""
+    if "--only" not in sys.argv:
+        return True
+    return sys.argv[sys.argv.index("--only") + 1] == probe
+
+
+def main() -> int:
+    app = get_app_config()
+    if not deepseek_api_key(app):
+        print("ABORT: no DeepSeek API key (env DEEPSEEK_API_KEY / "
+              "config.json deepseek.api_key).")
+        return 2
+    if app.deepseek_settings.get("enabled", True) is False:
+        print("ABORT: deepseek.enabled is false in config.json.")
+        return 2
+    if ARTIFACT.exists() and "--only" not in sys.argv:
+        ARTIFACT.unlink()  # full run = fresh wire log; --only appends
+
+    mtf_agree = {"5min": {"signal": "BUY", "score": 30.0},
+                 "1h": {"signal": "STRONG_BUY", "score": 45.0}}
+    levels = {"resistance": 2012.0, "support": 1985.0}
+
+    # P1 — coherent bullish, with rising ATR staircase (last close 2000)
+    if _want("P1"):
+        p1 = _tech_payload(_BULL_INDICATORS, mtf_agree,
+                           candles_df=_staircase(_FAKE_PRICE - 20 * 0.6,
+                                                 0.6, 20),
+                           key_levels=levels)
+        reply, take = _ask(p1)
+        _record("P1 coherent-bullish", p1, reply, take,
+                take is not None and take.stance == "support"
+                and take.confidence >= 50,
+                "expect support @ conf>=50")
+
+    # P2 — bearish everything against a BUY brief, falling staircase
+    if _want("P2"):
+        brief2 = dict(_BRIEF_BUY, market_regime="trending_down",
+                      mtf_agreement=False)
+        mtf_sell = {"5min": {"signal": "SELL", "score": -28.0},
+                    "1h": {"signal": "STRONG_SELL", "score": -41.0}}
+        p2 = _tech_payload(_BEAR_INDICATORS, mtf_sell, brief=brief2,
+                           candles_df=_staircase(_FAKE_PRICE + 20 * 0.6,
+                                                 -0.6, 20),
+                           key_levels=levels, exhaustion=3)
+        reply, take = _ask(p2)
+        _record("P2 bearish-vs-BUY-brief", p2, reply, take,
+                take is not None and take.stance == "oppose",
+                "expect oppose (BUY brief fights all evidence)")
+
+    # P3 — genuinely mixed evidence: MTF neutral both ways, ranging
+    # regime, choppy candles. (An H1 SELL against a BUY is a *listed
+    # conflict* in the prompt — oppose is correct there, not mixed.)
+    if _want("P3"):
+        brief3 = dict(_BRIEF_BUY, market_regime="ranging", score=12.0,
+                      confidence=35, mtf_agreement=False)
+        mtf_mixed = {"5min": {"signal": "NEUTRAL", "score": 4.0},
+                     "1h": {"signal": "NEUTRAL", "score": -3.0}}
+        p3 = _tech_payload(_MIXED_INDICATORS, mtf_mixed, brief=brief3,
+                           candles_df=_choppy(_FAKE_PRICE, 20))
+        reply, take = _ask(p3)
+        _record("P3 conflicting", p3, reply, take,
+                take is not None and (take.stance == "neutral"
+                                      or take.confidence < 60),
+                "expect neutral or conf<60 on genuinely mixed evidence")
+
+    # P4 — empty evidence: contract demands neutral @ conf<30
+    if _want("P4"):
+        p4 = _tech_payload({}, {}, key_levels=None)
+        reply, take = _ask(p4)
+        _record("P4 empty-evidence", p4, reply, take,
+                take is not None and take.stance == "neutral"
+                and take.confidence < 30,
+                "contract: thin/missing evidence -> neutral, conf<30")
+
+    # P5 — hallucination trap: no key_levels, no candles; the note must
+    # not cite any 3+-digit number absent from the payload
+    if _want("P5"):
+        p5 = _tech_payload(_BULL_INDICATORS, mtf_agree, key_levels=None)
+        reply, take = _ask(p5)
+        invented = []
+        if take is not None:
+            payload_blob = json.dumps(p5)
+            for num in re.findall(r"\d{3,}", take.note):
+                if num not in payload_blob:
+                    invented.append(num)
+        _record("P5 hallucination-trap", p5, reply, take,
+                take is not None and not invented,
+                f"invented numbers in note: {invented or 'none'}")
+
+    # P6 — output discipline across the persona replies gathered above
+    notes = [r for r in _RESULTS if r["note"]]
+    wordy = [r["probe"] for r in notes if len(r["note"].split()) > 20]
+    _record("P6 output-discipline", {}, None, None,
+            notes and not wordy,
+            f"notes >20 words: {wordy or 'none'} "
+            f"(cap is 15; 20 = fail threshold)")
+
+    # P7 — synthesizer over four fabricated SUPPORT takes
+    if _want("P7"):
+        takes = [{"persona": p, "stance": "support", "confidence": c,
+                  "note": "evidence aligned"}
+                 for p, c in (("macro", 70), ("technical", 80),
+                              ("risk", 65), ("contrarian", 60))]
+        synth_payload = {"direction": "BUY", "takes": takes}
+        reply = get_llm_client().chat_json(ap._SYNTH_SYSTEM, synth_payload)
+        verdict = reply.get("verdict") if isinstance(reply, dict) else None
+        rationale = (reply or {}).get("rationale", "")
+        _record("P7 synthesizer", synth_payload, reply, None,
+                verdict == "AGREE" and len(str(rationale).split()) <= 40,
+                f"expect AGREE + <=40-word rationale; got {verdict!r} "
+                f"({len(str(rationale).split())} words)")
+
+    n_pass = sum(1 for r in _RESULTS if r["pass"])
+    n_api = sum(1 for r in _RESULTS if r["probe"] != "P6 output-discipline")
+    print(f"\n{n_pass}/{len(_RESULTS)} probes passed "
+          f"({n_api} API requests spent; "
+          f"raw wire log: {ARTIFACT.name})")
+    return 0 if n_pass == len(_RESULTS) else 1
+
+
+if __name__ == "__main__":
+    sys.exit(main())

backtest/research/signal_log_campaign_2026-09.md

@@ -0,0 +1,180 @@
+# Signal-log audit → prediction-quality campaign — 2026-09-03
+
+**Source evidence:** `logs/signals.db` 2026-08-13 → 2026-09-03 (14,683 rows, candle
+snapshots from 08-18), replayed with the repo's own advisory outcome model
+(`analyze_signal_log.py --stored-candles`; the replay reproduces
+`advisory_outcome.py` to 5e-17 on the 49 panel-seeded rows), cross-checked
+against the pinned Oanda books (`backtest/oanda_*_065*.csv`).
+
+## Live-log findings that motivated the campaign
+
+| View (3 weeks, $0.65 rt) | n | WR | R | PF |
+|---|---|---|---|---|
+| every unique setup event | 144 | 67% | +37.8 | 1.93 |
+| opportunities (first per tf+direction, 60 min) | 66 | 67% | +18.3 | 1.92 |
+| one-position-per-TF (5-bar cooldown) | 32 | 62% | +6.6 | 1.53 |
+| direction BUY / SELL | 121 / 23 | 70% / 48% | +43.2 / −5.4 | 2.53 / 0.56 |
+
+- Two rally days (08-19, 08-21) contribute essentially all of the R.
+- SELL is negative in every view; in the books SELL is the weakest bucket on
+  every TF (H1 SELL −16.7R/5yr PF 0.77, −11.5R OOS PF 0.30; M15 SELL −11.7R OOS;
+  M5 SELL in `transitioning` −7.9R/12mo PF 0.21, −5.2R/5yr PF 0.49).
+- Bollinger is anti-predictive on all four TFs (forward-return hit 0.26–0.50 at
+  12 bars) and "opposes" 93% of taken book trades; divergence is inverted on
+  M15/H1 live and trades where it opposed outperform in the books.
+- Refuted live hypotheses (books disagree): ADX<20 gate, exhaustion gate,
+  NEAR-EMA200 gate, EXTREME-volatility gate, capping SL/ATR.
+
+## Method (rule 1)
+
+Harness: `backtest/gpu_backtest.py --pair XAUUSD --spread 0.65` on the CPU
+backend (CuPy present but the GPU was not reachable; the M5 12mo control
+reproduces the pinned `oanda_m5_12mo_065_nopb_20260814.csv` trade-for-trade
+except the two trades on the end date, which `--end` excludes). Every candidate
+is compared with a CONTROL produced on the same harness, never with the pinned
+book directly. Windows (cache start = start − 62 days):
+
+| window | `--start` | `--end` |
+|---|---|---|
+| 12mo (M15/H1) | 2025-08-04 | 2026-08-04 |
+| 12mo (M5 canonical) | 2025-08-14 | 2026-08-14 |
+| 5yr | 2021-08-01 | 2026-08-04 |
+| OOS | 2024-06-01 | 2025-03-30 |
+
+Acceptance: positive total R in both 12mo halves, PF ≥ 1.3, OOS not worse
+than control, and a candidate that removes a net-positive bucket for a PF-only
+gain is rejected (big-candle precedent). Runner:
+`<scratchpad>/campaign/run_campaign.sh`; summary: `summarize.py`.
+
+Candidates (all `--set`, no code):
+
+- **B1 SELL gating** — H1 `xau_strategy.sell_min_quality=[]` (block all H1
+  shorts); M5 `xau_strategy.sell_blocked_regimes=["transitioning"]`; M15
+  `xau_strategy.sell_min_quality=["B"]`.
+- **B2 divergence off** — `technical.disabled_indicators=["divergence"]` on
+  M15 and H1 (M5 divergence-down was rejected 2026-08-04; not re-run).
+- **B3 Bollinger weight 0 in trending regimes** — full
+  `technical.regime_weights_override` with bollinger 0.0 in
+  trending_up/trending_down/trending, ranging/transitioning unchanged; M5, M15, H1.
+
+## Results
+
+Every control reproduced its pinned book: M15 12mo (`ctl_m15_12mo`) is exact
+at 397/+73.93R/PF 1.53; all three H1 controls are exact; the M5 12mo control
+(`base_m5_12mo`) is exact except for the two trades that fall on the end
+date, which `--end` excludes on this harness (see Method above).
+
+| book | n | R | PF | DD | H1 | H2 | sellN | sellR | ΔR vs control | ΔPF |
+|---|---|---|---|---|---|---|---|---|---|---|
+| b1_h1_nosell_12mo | 91 | +22.2 | 1.79 | 4.1 | +20.6 | +1.6 | 0 | +0.0 | -1.8 | +0.25 |
+| b1_h1_nosell_5yr | 335 | +49.1 | 1.46 | 13.2 | -5.0 | +54.1 | 0 | +0.0 | +16.7 | +0.28 |
+| b1_h1_nosell_oos | 78 | +16.7 | 1.74 | 3.6 | +7.3 | +9.4 | 0 | +0.0 | +11.5 | +0.61 |
+| b1_m15_sellB_12mo | 341 | +74.6 | 1.62 | 10.8 | +68.4 | +6.2 | 82 | +9.1 | +0.7 | +0.09 |
+| b1_m15_sellB_5yr | 1378 | +119.1 | 1.22 | 41.4 | -21.3 | +140.3 | 335 | +3.7 | -8.9 | +0.02 |
+| b1_m15_sellB_oos | 272 | +47.1 | 1.51 | 13.8 | +12.3 | +34.8 | 59 | -3.4 | +8.3 | +0.15 |
+| b1_m5_selltrans_12mo | 557 | +112.2 | 1.56 | 9.9 | +55.9 | +56.4 | 75 | +14.8 | +7.9 | +0.06 |
+| b1_m5_selltrans_5yr | 889 | +133.7 | 1.41 | 12.8 | -8.3 | +142.0 | 88 | +9.0 | +5.3 | +0.03 |
+| b1_m5_selltrans_oos | 86 | +9.4 | 1.33 | 8.2 | -1.4 | +10.9 | 4 | +1.5 | +0.0 | +0.00 |
+| b2_h1_nodiv_12mo | 149 | +28.6 | 1.62 | 6.5 | +23.6 | +5.0 | 49 | +5.5 | +4.6 | +0.07 |
+| b2_h1_nodiv_5yr | 567 | +40.5 | 1.20 | 14.1 | -5.2 | +45.6 | 203 | -18.2 | +8.1 | +0.02 |
+| b2_h1_nodiv_oos | 119 | +7.6 | 1.17 | 7.7 | -1.9 | +9.5 | 31 | -14.3 | +2.4 | +0.04 |
+| b2_m15_nodiv_12mo | 439 | +81.3 | 1.51 | 13.1 | +72.3 | +8.9 | 147 | +7.0 | +7.4 | -0.02 |
+| b2_m15_nodiv_5yr | 1771 | +139.9 | 1.20 | 47.4 | -16.2 | +156.1 | 578 | +11.7 | +11.9 | -0.01 |
+| b2_m15_nodiv_oos | 341 | +52.3 | 1.43 | 17.1 | +15.6 | +36.7 | 93 | -10.0 | +13.5 | +0.07 |
+| b3_h1_bb0_12mo | 154 | +31.2 | 1.64 | 9.4 | +22.8 | +8.4 | 52 | +6.7 | +7.3 | +0.09 |
+| b3_h1_bb0_5yr | 583 | +37.3 | 1.18 | 18.3 | -7.4 | +44.7 | 206 | -20.4 | +4.9 | +0.00 |
+| b3_h1_bb0_oos | 122 | +5.6 | 1.12 | 8.7 | -2.0 | +7.6 | 32 | -13.8 | +0.4 | -0.01 |
+| b3_m15_bb0_12mo | 439 | +49.9 | 1.29 | 19.4 | +61.6 | -11.8 | 151 | +0.3 | -24.0 | -0.23 |
+| b3_m15_bb0_5yr | 1790 | +126.6 | 1.18 | 42.3 | -1.8 | +128.5 | 592 | +17.4 | -1.3 | -0.03 |
+| b3_m15_bb0_oos | 345 | +55.5 | 1.47 | 12.4 | +18.2 | +37.3 | 94 | -8.4 | +16.7 | +0.11 |
+| b3_m5_bb0_12mo | 621 | +84.0 | 1.35 | 16.6 | +49.3 | +34.7 | 104 | +9.7 | -20.4 | -0.14 |
+| b3_m5_bb0_5yr | 987 | +113.0 | 1.31 | 15.3 | -7.7 | +120.6 | 134 | +12.0 | -15.5 | -0.08 |
+| b3_m5_bb0_oos | 93 | +10.6 | 1.35 | 5.6 | -2.9 | +13.5 | 9 | +3.7 | +1.2 | +0.01 |
+| b12_h1_nosell_nodiv_12mo | 100 | +23.1 | 1.75 | 4.6 | +21.8 | +1.3 | 0 | +0.0 | -0.8 | +0.20 |
+| b12_h1_nosell_nodiv_5yr | 364 | +58.7 | 1.52 | 12.9 | -4.1 | +62.8 | 0 | +0.0 | +26.3 | +0.34 |
+| b12_h1_nosell_nodiv_oos | 88 | +21.9 | 1.92 | 4.8 | +9.4 | +12.5 | 0 | +0.0 | +16.8 | +0.79 |
+| base_m5_12mo | 575 | +104.3 | 1.50 | 10.7 | +50.8 | +53.5 | 93 | +6.9 | — | — |
+| ctl_h1_12mo | 138 | +24.0 | 1.55 | 8.0 | +19.4 | +4.6 | 47 | +1.8 | — | — |
+| ctl_h1_5yr | 515 | +32.4 | 1.18 | 15.2 | -5.9 | +38.3 | 180 | -16.7 | — | — |
+| ctl_h1_oos | 104 | +5.1 | 1.13 | 7.5 | -0.9 | +6.1 | 26 | -11.5 | — | — |
+| ctl_m15_12mo | 397 | +73.9 | 1.53 | 10.8 | +66.7 | +7.2 | 138 | +8.5 | — | — |
+| ctl_m15_5yr | 1578 | +128.0 | 1.21 | 35.8 | -3.4 | +131.4 | 535 | +12.6 | — | — |
+| ctl_m15_oos | 299 | +38.8 | 1.36 | 15.5 | +5.9 | +32.9 | 86 | -11.7 | — | — |
+| ctl_m5_5yr | 911 | +128.4 | 1.39 | 12.8 | -8.3 | +136.7 | 110 | +3.8 | — | — |
+| ctl_m5_oos | 86 | +9.4 | 1.33 | 8.2 | -1.4 | +10.9 | 4 | +1.5 | — | — |
+
+Raw table (n / R / PF / WR% / DD / H1 / H2 / PF-half-1 / PF-half-2 /
+sellN / sellR / ΔR / ΔPF / Δn vs control; control and `base_m5_12mo`
+rows carry no delta):
+
+```
+book                           n        R    PF   WR%    DD      H1      H2   PF1   PF2 sellN   sellR |      ΔR    ΔPF    Δn
+b1_h1_nosell_12mo             91    +22.2  1.79  62.6   4.1   +20.6    +1.6  2.10  1.18     0    +0.0 |    -1.8  +0.25  -47
+b1_h1_nosell_5yr             335    +49.1  1.46  58.2  13.2    -5.0   +54.1  0.87  1.78     0    +0.0 |   +16.7  +0.28 -180
+b1_h1_nosell_oos              78    +16.7  1.74  62.8   3.6    +7.3    +9.4  1.67  1.81     0    +0.0 |   +11.5  +0.61  -26
+b1_m15_sellB_12mo            341    +74.6  1.62  62.8  10.8   +68.4    +6.2  2.15  1.10    82    +9.1 |    +0.7  +0.09  -56
+b1_m15_sellB_5yr            1378   +119.1  1.22  57.0  41.4   -21.3  +140.3  0.91  1.46   335    +3.7 |    -8.9  +0.02 -200
+b1_m15_sellB_oos             272    +47.1  1.51  60.7  13.8   +12.3   +34.8  1.24  1.85    59    -3.4 |    +8.3  +0.15  -27
+b1_m5_selltrans_12mo         557   +112.2  1.56  53.7   9.9   +55.9   +56.4  1.54  1.58    75   +14.8 |    +7.9  +0.06  -18
+b1_m5_selltrans_5yr          889   +133.7  1.41  51.5  12.8    -8.3  +142.0  0.84  1.52    88    +9.0 |    +5.3  +0.03  -22
+b1_m5_selltrans_oos           86     +9.4  1.33  53.5   8.2    -1.4   +10.9  0.91  1.95     4    +1.5 |    +0.0  +0.00   +0
+b2_h1_nodiv_12mo             149    +28.6  1.62  62.4   6.5   +23.6    +5.0  1.93  1.24    49    +5.5 |    +4.6  +0.07  +11
+b2_h1_nodiv_5yr              567    +40.5  1.20  56.3  14.1    -5.2   +45.6  0.93  1.37   203   -18.2 |    +8.1  +0.02  +52
+b2_h1_nodiv_oos              119     +7.6  1.17  56.3   7.7    -1.9    +9.5  0.91  1.44    31   -14.3 |    +2.4  +0.04  +15
+b2_m15_nodiv_12mo            439    +81.3  1.51  60.8  13.1   +72.3    +8.9  1.90  1.11   147    +7.0 |    +7.4  -0.02  +42
+b2_m15_nodiv_5yr            1771   +139.9  1.20  55.8  47.4   -16.2  +156.1  0.94  1.39   578   +11.7 |   +11.9  -0.01 +193
+b2_m15_nodiv_oos             341    +52.3  1.43  59.2  17.1   +15.6   +36.7  1.23  1.68    93   -10.0 |   +13.5  +0.07  +42
+b3_h1_bb0_12mo               154    +31.2  1.64  60.4   9.4   +22.8    +8.4  1.89  1.36    52    +6.7 |    +7.3  +0.09  +16
+b3_h1_bb0_5yr                583    +37.3  1.18  54.5  18.3    -7.4   +44.7  0.90  1.35   206   -20.4 |    +4.9  +0.00  +68
+b3_h1_bb0_oos                122     +5.6  1.12  55.7   8.7    -2.0    +7.6  0.91  1.33    32   -13.8 |    +0.4  -0.01  +18
+b3_m15_bb0_12mo              439    +49.9  1.29  59.0  19.4   +61.6   -11.8  1.77  0.87   151    +0.3 |   -24.0  -0.23  +42
+b3_m15_bb0_5yr              1790   +126.6  1.18  56.1  42.3    -1.8  +128.5  0.99  1.31   592   +17.4 |    -1.3  -0.03 +212
+b3_m15_bb0_oos               345    +55.5  1.47  61.2  12.4   +18.2   +37.3  1.29  1.66    94    -8.4 |   +16.7  +0.11  +46
+b3_m5_bb0_12mo               621    +84.0  1.35  51.4  16.6   +49.3   +34.7  1.43  1.28   104    +9.7 |   -20.4  -0.14  +46
+b3_m5_bb0_5yr                987   +113.0  1.31  50.7  15.3    -7.7  +120.6  0.85  1.38   134   +12.0 |   -15.5  -0.08  +76
+b3_m5_bb0_oos                 93    +10.6  1.35  54.8   5.6    -2.9   +13.5  0.85  2.17     9    +3.7 |    +1.2  +0.01   +7
+b12_h1_nosell_nodiv_12mo     100    +23.1  1.75  63.0   4.6   +21.8    +1.3  2.00  1.15     0    +0.0 |    -0.8  +0.20  -38
+b12_h1_nosell_nodiv_5yr      364    +58.7  1.52  59.6  12.9    -4.1   +62.8  0.90  1.87     0    +0.0 |   +26.3  +0.34 -151
+b12_h1_nosell_nodiv_oos       88    +21.9  1.92  63.6   4.8    +9.4   +12.5  1.93  1.92     0    +0.0 |   +16.8  +0.79  -16
+base_m5_12mo                 575   +104.3  1.50  52.9  10.7   +50.8   +53.5  1.46  1.53    93    +6.9
+ctl_h1_12mo                  138    +24.0  1.55  60.9   8.0   +19.4    +4.6  1.84  1.22    47    +1.8
+ctl_h1_5yr                   515    +32.4  1.18  55.7  15.2    -5.9   +38.3  0.91  1.34   180   -16.7
+ctl_h1_oos                   104     +5.1  1.13  56.7   7.5    -0.9    +6.1  0.95  1.32    26   -11.5
+ctl_m15_12mo                 397    +73.9  1.53  62.2  10.8   +66.7    +7.2  1.95  1.10   138    +8.5
+ctl_m15_5yr                 1578   +128.0  1.21  56.5  35.8    -3.4  +131.4  0.99  1.37   535   +12.6
+ctl_m15_oos                  299    +38.8  1.36  59.2  15.5    +5.9   +32.9  1.10  1.66    86   -11.7
+ctl_m5_5yr                   911   +128.4  1.39  51.3  12.8    -8.3  +136.7  0.84  1.48   110    +3.8
+ctl_m5_oos                    86     +9.4  1.33  53.5   8.2    -1.4   +10.9  0.91  1.95     4    +1.5
+```
+
+## Decisions
+
+- **SHIPPED** — M5 `xau_strategy.sell_blocked_regimes=["transitioning"]`
+  (b1_m5_selltrans): 12mo +7.9R, 5yr +5.3R, OOS unchanged (no such trades
+  in that window); positive in both 12mo halves on every window. H1
+  `xau_strategy.sell_min_quality=[]` (b1_h1_nosell, all H1 shorts blocked):
+  5yr +16.7R and OOS +11.5R more than pay for the 12mo cost of -1.8R; H1
+  shorts were -16.7R/5yr and -11.5R/OOS in the control. Both land in
+  `pair_signals/xauusd.py`.
+- **REJECTED** — M15 `sell_min_quality=["B"]` (b1_m15_sellB): 5yr -8.9R
+  with max DD widening 35.8 -> 41.4R for a marginal +0.7R on 12mo; fails
+  the drawdown clause. All three B3 Bollinger-weight-0-in-trending-regimes
+  runs (b3_h1_bb0, b3_m15_bb0, b3_m5_bb0): negative on M5 (-20.4R/12mo,
+  -15.5R/5yr) and M15 (-24.0R/12mo, PF 1.29 -> still below the 12mo H2
+  going negative at -11.8R); H1's B3 5yr gain (+4.9R) doesn't clear the
+  bar once M5/M15 fail outright — Bollinger's constant-oppose vote stays
+  a load-bearing quality bar, consistent with the 2026-08-04 audit.
+- **DECLINED** — B2-M15 (`technical.disabled_indicators=["divergence"]`
+  on M15, b2_m15_nodiv): R improves on every window (+7.4/+11.9/+13.5R)
+  but 5yr max DD widens 35.8 -> 47.4R while PF stays flat (1.21 -> 1.20) —
+  more risk for no risk-adjusted gain; owner call.
+- **PARKED** — B2-H1 (`technical.disabled_indicators=["divergence"]` on
+  H1, b2_h1_nodiv, standalone: +4.6/+8.1/+2.4R, PF 1.62/1.20/1.17). The
+  combined run with the H1 no-short override (b12_h1_nosell_nodiv) passes
+  rule 1 and beats the no-shorts override alone on both 5yr and OOS: 12mo
+  100/+23.1R PF 1.75 (halves +21.8/+1.3), 5yr 364/+58.7R PF 1.52, OOS
+  88/+21.9R PF 1.92. Not shipped anyway: it would route through
+  `technical.disabled_indicators`, which CLAUDE.md rule 8 reserves for
+  experiments, not production paths. Owner decision; recommended vehicle
+  is an H1-only `regime_weights_override` with divergence weight 0 plus a
+  fresh validation run under that mechanism.

backtest/research/signal_log_eval_2026-08-18.md

@@ -0,0 +1,113 @@
+# Signal-log live performance evaluation — 2026-08-18
+
+**Tool:** `analyze_signal_log.py` (new, repo root). Read-only on
+`logs/signals.db` (sqlite `immutable=1`); replays every logged trade
+setup through the backtester's own `simulate_trades` on the RAW Oanda
+price path (next-open fill, SL/TP distances preserved, SL-first ties,
+gap-at-open fills, close-only trail ratchet with the per-setup stamped
+trail params). Reproduce:
+
+```bash
+python analyze_signal_log.py --csv backtest/signal_log_eval_20260818.csv
+```
+
+## Window & sample
+
+- Log window: **2026-08-13 14:31 → 2026-08-18 14:13 UTC (5 days)** —
+  the entire signal_log history to date.
+- 2,654 rows (telegram 2,180 / dashboard 474); M5 1,308, M15 1,189,
+  H1 119, H4 38.
+- 39 rows carry a `trade_setup` → **17 unique signal events** after
+  dedup (cross-source double-logs + persisting-signal re-fires collapse
+  on (tf, direction, entry, SL, strategy) with a 60-min gap).
+- **No real executions exist** (`oanda_state.json` empty, no
+  `oanda_daemon` rows) — every outcome below is a hypothetical replayed
+  fill, not a broker fill.
+- Cost: dual convention — **$0.65 rt headline**, $0.30 comparative
+  (derived analytically, R is linear in cost).
+
+## Trade replay results
+
+| First logged | TF | Signal | Grade | Strategy | Fill | Exit | Bars | R@0.65 |
+|---|---|---|---|---|---|---|---|---|
+| 08-13 15:30 | 5min | SELL | B | GENERIC | 4365.39 | TRAIL | 19 | −0.20 |
+| 08-13 18:00 | 15min | SELL | B | GENERIC | 4356.05 | TRAIL | 44 | +0.91 |
+| 08-14 07:45 | 5min | BUY | B | XAU_CP | 4339.72 | TRAIL | 60 | +2.56 |
+| 08-14 12:15 | 15min | BUY | B | GENERIC | 4371.91 | TRAIL | 27 | +0.06 |
+| 08-14 12:25 | 5min | STRONG_BUY | A | GENERIC | 4379.12 | TRAIL | 277 | +0.20 |
+| 08-14 12:30 | 5min | STRONG_BUY | A | GENERIC | 4379.87 | TRAIL | 276 | +0.16 |
+| 08-14 12:30 | 15min | BUY | B | GENERIC | 4379.87 | TRAIL | 63 | +0.18 |
+| 08-17 08:00 | 1h | BUY | C | GENERIC | 4405.19 | TRAIL | 18 | −0.13 |
+| 08-17 14:15 | 5min | STRONG_BUY | A | GENERIC | 4412.35 | TRAIL | 46 | −0.10 |
+| 08-17 14:20 | 5min | STRONG_BUY | A | GENERIC | 4418.55 | TRAIL | 125 | −0.23 |
+| 08-17 14:25 | 5min | STRONG_BUY | A | GENERIC | 4424.10 | SL | 151 | −1.02 |
+| 08-17 14:30 | 15min | BUY | A | GENERIC | 4418.23 | SL | 17 | −1.03 |
+| 08-17 14:45 | 15min | STRONG_BUY | A | GENERIC | 4417.49 | SL | 16 | −1.03 |
+| 08-17 15:00 | 15min | STRONG_BUY | A | GENERIC | 4417.48 | SL | 15 | −1.03 |
+| 08-17 15:00 | 5min | NEUTRAL | C | GENERIC | 4417.48 | TRAIL | 117 | −0.16 |
+| 08-17 15:15 | 15min | STRONG_BUY | A | GENERIC | 4421.30 | SL | 14 | −1.04 |
+| 08-17 15:30 | 15min | BUY | B | GENERIC | 4425.70 | SL | 85 | −1.02 |
+
+**Aggregate: 17 closed, 6 wins (35%), total −2.91R @0.65 / −2.67R
+@0.30, PF 0.58.** By grade: A n=9 → −5.12R; B n=6 → +2.50R; C n=2 →
+−0.30R.
+
+Reading: the book is two regimes. Everything through the 08-14 rally
+made money (+4.7R across 8 events, incl. the XAU_CP +2.56R); the 08-17
+afternoon BUY/STRONG_BUY cluster chased the spike to ~4425 and got
+stopped as price faded — 9 events, −7.6R combined. One bad afternoon
+dominates the 5-day sample. Notable: a NEUTRAL-graded row carried a
+setup (post-gate CP/PB-style path) and lost small.
+
+## Advisory panel scorecard
+
+Only **7 verdicts** logged so far (2 NEUTRAL, 5 OPPOSE, 0 AGREE) — the
+panel's min-quality gate + daily budget keep it dark on most scans.
+
+| Row | When | Signal | Verdict | Outcome basis | Impact |
+|---|---|---|---|---|---|
+| #1358 | 08-17 07:40 M5 | BUY B | NEUTRAL @55% | +4h move | −$8.19 in dir (pass was right) |
+| #1361 | 08-17 07:45 M5 | BUY A | NEUTRAL @55% | +4h move | −$12.43 in dir (pass was right) |
+| #1550 | 08-17 14:15 M5 | STRONG_BUY A | OPPOSE @80% | replay | saved 0.10R |
+| #1553 | 08-17 14:25 M5 | STRONG_BUY A | OPPOSE @80% | replay | saved 1.02R |
+| #1555 | 08-17 14:30 M15 | BUY A | OPPOSE @75% | replay | saved 1.03R |
+| #1561 | 08-17 14:45 M15 | STRONG_BUY A | OPPOSE @75% | replay | saved 1.03R |
+| #1588 | 08-17 15:30 M15 | BUY B | OPPOSE @75% | replay | saved 1.02R |
+
+**Every OPPOSE was correct: ≈ +4.2R would have been lost taking those
+five trades.** The panel's rationale ("price climbed too fast, profit
+too small") correctly identified the 08-17 exhaustion spike that the
+composite scored A. Both NEUTRAL passes also preceded adverse moves.
+Caveat: n=7, one market episode, zero AGREE verdicts — the panel has
+only proven itself as a veto so far, never as a confirmation.
+
+## Directional hit-rate (non-setup signals)
+
+27 deduped non-setup BUY/SELL rows, forward close-to-close on M5:
+
+| Horizon | Hit rate | Mean move in direction |
+|---|---|---|
+| +1h | 48% | −$0.72 |
+| +4h | 44% | −$2.30 |
+| +24h | 7% | −$27.45 |
+
+The 24h figure is dominated by the 08-17 buy-cluster reversal; not
+independently meaningful at n=27.
+
+## Method notes / caveats
+
+- **Small sample**: 5 days, 17 events, one trending episode + one
+  reversal. Directional read only; do not tune anything on this.
+- Entry model = next bar open after the scan boundary (live scans run
+  at boundary+10s), SL/TP distances preserved from the logged setup —
+  identical to the backtester replay loop.
+- Cross-source dedup verified: telegram+dashboard double-logs and
+  5-min re-fires of a persisting signal collapse to one event each.
+- The stray repo-root `signals.db` (828 rows, Aug 13–14, pre-panel
+  schema) is an early duplicate subset of `logs/signals.db` — excluded.
+- Tests: Module 25 in `test_all.py` was only added on 2026-09-03 (the
+  2026-08-18 claim of 7 tests was never true at HEAD). It now covers
+  boundary flooring, clustering (shared `signal_log.setup_key`), trade
+  building, the stored-candle replay books, surface parity, WAL-visible
+  loading (`mode=ro`; `immutable=1` missed the newest rows) and hit-rate
+  signing.

chart_patterns.py

@@ -0,0 +1,290 @@
+"""
+Classical chart-pattern detection for the XAU_CP entry archetype.
+================================================================
+
+Third entry archetype (after the composite-breakout generators and the
+XAU_PB pullback): trade the completion of a proven multi-bar price
+pattern. Pattern set is deliberately tight — the classical patterns
+with the strongest documented statistics (Bulkowski, *Encyclopedia of
+Chart Patterns*) that are unambiguous from OHLC at bar close:
+
+    double_bottom  bullish reversal   two matching swing lows, break of
+                                      the interim peak (neckline)
+    double_top     bearish reversal   mirrored
+    inv_hs         bullish reversal   inverse head & shoulders, break of
+                                      the neckline (the LOWER of the two
+                                      peaks flanking the head — the earlier
+                                      horizontal trigger, no sloped fits;
+                                      validated as-is, see detector note)
+    hs_top         bearish reversal   head & shoulders top, mirrored
+    bull_flag      bullish continuation  strong pole, shallow ordered
+                                      retrace, close at a new high
+    bear_flag      bearish continuation  mirrored
+
+Triangles / wedges / cup-and-handle are excluded on purpose: weaker
+historical stats and apex-timing ambiguity — more parameters for less
+edge.
+
+Design contract (mirrors the rest of the signal stack):
+- Detectors see ONLY the visible closed-bar slice the aggregator was
+  given; the last row is the just-closed signal bar. No forming candle,
+  no future bars → identical behavior live and in the bar-replay
+  backtester, and signals never repaint.
+- All geometric thresholds are in ATR units so one parameter set is
+  scale-free across timeframes. ATR comes from the engine (Wilder 14).
+- Trigger discipline: a pattern fires only on the bar that COMPLETES it
+  — the first close beyond the neckline/boundary since the pattern
+  formed. One breakout → at most one signal.
+- Swing points use the same local-extremum definition as the engine's
+  structure S/R (window w bars each side), so a "swing" here is the
+  same object a chartist sees on the engine's key levels.
+
+Every detector returns None or a dict:
+    {"pattern": str, "direction": "BUY"|"SELL",
+     "anchor": float,     # SL reference (pattern invalidation price)
+     "trigger": float,    # neckline / boundary that was broken
+     "height": float}     # measured-move distance in price units
+
+Defaults below are the textbook definitions, not tuned values. The
+validation record for which patterns/timeframes are enabled lives in
+pair_signals/xauusd.py (per repo rules, per-TF adoption happens in the
+factory, never in global defaults).
+"""
+
+from typing import Dict, List, Optional
+
+import numpy as np
+import pandas as pd
+
+__all__ = ["detect_chart_pattern", "PATTERN_DETECTORS"]
+
+
+def _swing_indices(values: np.ndarray, w: int, find_high: bool) -> List[int]:
+    """Indices of local extrema with `w` bars on each side.
+
+    A bar i is a swing high if values[i] is the strict maximum of
+    values[i-w : i+w+1] (ties reject — a flat top is not a clean swing).
+    The last `w` bars can never qualify, which is exactly right: a swing
+    isn't confirmed until `w` bars close after it.
+    """
+    n = len(values)
+    out: List[int] = []
+    for i in range(w, n - w):
+        window = values[i - w : i + w + 1]
+        if np.isnan(window).any():
+            continue
+        center = values[i]
+        rest = np.delete(window, w)
+        if find_high:
+            if center > rest.max():
+                out.append(i)
+        else:
+            if center < rest.min():
+                out.append(i)
+    return out
+
+
+def _first_close_beyond(closes: np.ndarray, start: int, level: float,
+                        above: bool) -> bool:
+    """True iff the LAST close is the first one beyond `level` since
+    `start` (exclusive of the last bar)."""
+    last = closes[-1]
+    prior = closes[start:-1]
+    if above:
+        return last > level and (len(prior) == 0 or prior.max() <= level)
+    return last < level and (len(prior) == 0 or prior.min() >= level)
+
+
+# ── Double bottom / top ──────────────────────────────────────────────
+
+def _detect_double(o, h, l, c, atr, cfg, bullish: bool) -> Optional[Dict]:
+    w = cfg.cp_swing_window
+    ext = l if bullish else h
+    idx = _swing_indices(ext, w, find_high=not bullish)
+    if len(idx) < 2:
+        return None
+    i1, i2 = idx[-2], idx[-1]
+    n = len(c)
+    if i2 - i1 < cfg.cp_min_sep_bars:
+        return None
+    if (n - 1) - i2 > cfg.cp_recency_bars:
+        return None
+    e1, e2 = ext[i1], ext[i2]
+    if abs(e1 - e2) > cfg.cp_tolerance_atr * atr:
+        return None
+
+    if bullish:
+        neckline = float(h[i1 : i2 + 1].max())
+        floor = min(e1, e2)
+        height = neckline - floor
+    else:
+        neckline = float(l[i1 : i2 + 1].min())
+        ceil_ = max(e1, e2)
+        height = ceil_ - neckline
+
+    if not (cfg.cp_min_height_atr * atr <= height <= cfg.cp_max_height_atr * atr):
+        return None
+
+    # Pattern must not have been invalidated after the second extreme:
+    # a lower low (bullish) / higher high (bearish) breaks the pattern.
+    if bullish:
+        if i2 + 1 < n and l[i2 + 1 :].min() < min(e1, e2) - 0.1 * atr:
+            return None
+        if not _first_close_beyond(c, i2, neckline, above=True):
+            return None
+        return {"pattern": "double_bottom", "direction": "BUY",
+                "anchor": float(min(e1, e2)), "trigger": neckline,
+                "height": float(height)}
+    else:
+        if i2 + 1 < n and h[i2 + 1 :].max() > max(e1, e2) + 0.1 * atr:
+            return None
+        if not _first_close_beyond(c, i2, neckline, above=False):
+            return None
+        return {"pattern": "double_top", "direction": "SELL",
+                "anchor": float(max(e1, e2)), "trigger": neckline,
+                "height": float(height)}
+
+
+# ── Head & shoulders (top and inverse) ───────────────────────────────
+
+def _detect_hs(o, h, l, c, atr, cfg, bullish: bool) -> Optional[Dict]:
+    """bullish=True → inverse H&S (three swing LOWS, head deepest)."""
+    w = cfg.cp_swing_window
+    ext = l if bullish else h
+    idx = _swing_indices(ext, w, find_high=not bullish)
+    if len(idx) < 3:
+        return None
+    i1, i2, i3 = idx[-3], idx[-2], idx[-1]
+    n = len(c)
+    if (n - 1) - i3 > cfg.cp_recency_bars:
+        return None
+    s1, head, s2 = ext[i1], ext[i2], ext[i3]
+
+    if bullish:
+        # Head strictly the deepest low; shoulders roughly level.
+        if not (head < s1 - cfg.cp_hs_head_min_atr * atr
+                and head < s2 - cfg.cp_hs_head_min_atr * atr):
+            return None
+        if abs(s1 - s2) > cfg.cp_hs_shoulder_tol_atr * atr:
+            return None
+        # Neckline: the LOWER of the two interim peaks — the EARLIER of
+        # the two horizontal triggers (max() would demand clearing both
+        # peaks). The whole validated CP book (M5 BUY-only) ran on this
+        # trigger — do not flip to max() without full re-validation.
+        p1 = float(h[i1 : i2 + 1].max())
+        p2 = float(h[i2 : i3 + 1].max())
+        neckline = min(p1, p2)
+        height = neckline - head
+    else:
+        if not (head > s1 + cfg.cp_hs_head_min_atr * atr
+                and head > s2 + cfg.cp_hs_head_min_atr * atr):
+            return None
+        if abs(s1 - s2) > cfg.cp_hs_shoulder_tol_atr * atr:
+            return None
+        t1 = float(l[i1 : i2 + 1].min())
+        t2 = float(l[i2 : i3 + 1].min())
+        neckline = max(t1, t2)
+        height = head - neckline
+
+    if not (cfg.cp_min_height_atr * atr <= height <= cfg.cp_max_height_atr * atr):
+        return None
+
+    if bullish:
+        # Invalidated if price undercut the head after the right shoulder.
+        if i3 + 1 < n and l[i3 + 1 :].min() < head - 0.1 * atr:
+            return None
+        if not _first_close_beyond(c, i3, neckline, above=True):
+            return None
+        return {"pattern": "inv_hs", "direction": "BUY",
+                "anchor": float(s2), "trigger": neckline,
+                "height": float(height)}
+    else:
+        if i3 + 1 < n and h[i3 + 1 :].max() > head + 0.1 * atr:
+            return None
+        if not _first_close_beyond(c, i3, neckline, above=False):
+            return None
+        return {"pattern": "hs_top", "direction": "SELL",
+                "anchor": float(s2), "trigger": neckline,
+                "height": float(height)}
+
+
+# ── Flags (bull and bear) ────────────────────────────────────────────
+
+def _detect_flag(o, h, l, c, atr, cfg, bullish: bool) -> Optional[Dict]:
+    n = len(c)
+    lo_r, hi_r = cfg.cp_flag_retrace
+    if bullish:
+        # Pole top = highest high BEFORE the trigger bar.
+        ip = int(np.argmax(h[:-1]))
+        flag_len = (n - 1) - ip
+        if not (cfg.cp_flag_min_bars <= flag_len <= cfg.cp_flag_max_bars):
+            return None
+        pole_start = max(0, ip - cfg.cp_flag_pole_bars + 1)
+        pole_low = float(l[pole_start : ip + 1].min())
+        pole = float(h[ip]) - pole_low
+        if pole < cfg.cp_flag_pole_atr * atr:
+            return None
+        flag_low = float(l[ip + 1 : n].min())
+        retrace = (float(h[ip]) - flag_low) / pole
+        if not (lo_r <= retrace <= hi_r):
+            return None
+        if not _first_close_beyond(c, ip, float(h[ip]), above=True):
+            return None
+        return {"pattern": "bull_flag", "direction": "BUY",
+                "anchor": flag_low, "trigger": float(h[ip]),
+                "height": float(pole)}
+    else:
+        ip = int(np.argmin(l[:-1]))
+        flag_len = (n - 1) - ip
+        if not (cfg.cp_flag_min_bars <= flag_len <= cfg.cp_flag_max_bars):
+            return None
+        pole_start = max(0, ip - cfg.cp_flag_pole_bars + 1)
+        pole_high = float(h[pole_start : ip + 1].max())
+        pole = pole_high - float(l[ip])
+        if pole < cfg.cp_flag_pole_atr * atr:
+            return None
+        flag_high = float(h[ip + 1 : n].max())
+        retrace = (flag_high - float(l[ip])) / pole
+        if not (lo_r <= retrace <= hi_r):
+            return None
+        if not _first_close_beyond(c, ip, float(l[ip]), above=False):
+            return None
+        return {"pattern": "bear_flag", "direction": "SELL",
+                "anchor": flag_high, "trigger": float(l[ip]),
+                "height": float(pole)}
+
+
+# Priority order: most structured / rarest first — if two patterns
+# complete on the same bar, the more specific one names the trade.
+PATTERN_DETECTORS = {
+    "inv_hs":        lambda o, h, l, c, atr, cfg: _detect_hs(o, h, l, c, atr, cfg, True),
+    "hs_top":        lambda o, h, l, c, atr, cfg: _detect_hs(o, h, l, c, atr, cfg, False),
+    "double_bottom": lambda o, h, l, c, atr, cfg: _detect_double(o, h, l, c, atr, cfg, True),
+    "double_top":    lambda o, h, l, c, atr, cfg: _detect_double(o, h, l, c, atr, cfg, False),
+    "bull_flag":     lambda o, h, l, c, atr, cfg: _detect_flag(o, h, l, c, atr, cfg, True),
+    "bear_flag":     lambda o, h, l, c, atr, cfg: _detect_flag(o, h, l, c, atr, cfg, False),
+}
+
+
+def detect_chart_pattern(df: pd.DataFrame, atr: float, cfg) -> Optional[Dict]:
+    """Run enabled detectors over the last cp_lookback(+1) bars.
+
+    `df` is the visible history at decision time (last row = signal
+    bar). Returns the first match in PATTERN_DETECTORS priority order,
+    or None.
+    """
+    if atr <= 0 or len(df) < cfg.cp_lookback + 1:
+        return None
+    tail = df.iloc[-(cfg.cp_lookback + 1):]
+    o = tail["Open"].to_numpy(dtype=np.float64)
+    h = tail["High"].to_numpy(dtype=np.float64)
+    l = tail["Low"].to_numpy(dtype=np.float64)
+    c = tail["Close"].to_numpy(dtype=np.float64)
+    enabled = set(cfg.cp_patterns)
+    for name, det in PATTERN_DETECTORS.items():
+        if name not in enabled:
+            continue
+        hit = det(o, h, l, c, atr, cfg)
+        if hit is not None:
+            return hit
+    return None

config.json.example

@@ -0,0 +1,64 @@
+{
+  "telegram": {
+    "token": "123456789:REPLACE_WITH_YOUR_BOT_TOKEN",
+    "chat_id": "REPLACE_WITH_OWNER_CHAT_ID",
+    "channel_id": "-100REPLACE_WITH_CHANNEL_ID_OR_EMPTY_STRING",
+    "alert_min_quality": "C"
+  },
+  "data_providers": {
+    "active": "oanda",
+    "streaming": "tiingo",
+    "twelve_data": {
+      "api_key": "REPLACE_WITH_YOUR_TWELVE_DATA_API_KEY"
+    },
+    "oanda": {},
+    "tiingo": {
+      "api_key": "REPLACE_WITH_YOUR_TIINGO_API_KEY_OR_EMPTY"
+    }
+  },
+  "scanning": {
+    "pairs": ["XAU/USD"],
+    "timeframes": ["M5", "M15", "H1", "H4"]
+  },
+  "pairs": {
+    "XAU/USD": {
+      "blocked_hours_utc": [0, 1, 2, 10, 11, 13, 16, 21]
+    }
+  },
+  "oanda": {
+    "account_id": "REPLACE_WITH_ACCOUNT_ID",
+    "api_token": "REPLACE_WITH_API_TOKEN",
+    "base_url": "https://api-fxpractice.oanda.com"
+  },
+  "deepseek": {
+    "api_key": "REPLACE_WITH_YOUR_DEEPSEEK_API_KEY_OR_EMPTY",
+    "model": "deepseek-v4-flash",
+    "enabled": true,
+    "timeout_seconds": 12,
+    "daily_request_cap": 500
+  },
+  "agent_panel": {
+    "enabled": false,
+    "telegram": true,
+    "dashboard": false,
+    "min_quality": "B",
+    "max_panels_per_day": 20,
+    "reuse_ttl_seconds": 3600,
+    "candles_enabled": true,
+    "candle_bars_primary": 20,
+    "candle_bars_confirm": 10,
+    "verdict_source": "macro"
+  },
+  "market_outlook": {
+    "enabled": false,
+    "telegram": false,
+    "max_per_day": 12,
+    "candles_bars": 20
+  },
+  "signal_log": {
+    "enabled": true,
+    "path": "logs/signals.db",
+    "log_blocked_scans": true
+  },
+  "log_level": "INFO"
+}

config.py

@@ -1,8 +1,6 @@
 """
-Forex Signal Intelligence Platform - Configuration
-===================================================
-Central configuration for all signal analysis parameters.
-Tuned by market regime and validated against historical performance.
+ForexBot2 — Configuration (XAU/USD focused)
+==========================================
 """
 
 from dataclasses import dataclass, field
@@ -29,168 +27,328 @@ class SignalStrength(Enum):
     STRONG_SELL = -2
 
 
-class MarketSession(Enum):
-    SYDNEY = "sydney"
-    TOKYO = "tokyo"
-    LONDON = "london"
-    NEW_YORK = "new_york"
-    OVERLAP_LONDON_NY = "london_ny_overlap"
+# ── Supported pairs ──────────────────────────────────────────────
+XAUUSD = "XAU/USD"
 
-
-# ── Major Forex Pairs ──────────────────────────────────────────────
-MAJOR_PAIRS = [
-    "EURUSD=X", "GBPUSD=X", "USDJPY=X", "USDCHF=X",
-    "AUDUSD=X", "USDCAD=X", "NZDUSD=X",
-]
-
-CROSS_PAIRS = [
-    "EURGBP=X", "EURJPY=X", "GBPJPY=X", "AUDJPY=X",
-    "EURCHF=X", "CADJPY=X",
-]
-
-# Yahoo Finance symbol mapping for display
-SYMBOL_DISPLAY = {
-    "EURUSD=X": "EUR/USD", "GBPUSD=X": "GBP/USD",
-    "USDJPY=X": "USD/JPY", "USDCHF=X": "USD/CHF",
-    "AUDUSD=X": "AUD/USD", "USDCAD=X": "USD/CAD",
-    "NZDUSD=X": "NZD/USD", "EURGBP=X": "EUR/GBP",
-    "EURJPY=X": "EUR/JPY", "GBPJPY=X": "GBP/JPY",
-    "AUDJPY=X": "AUD/JPY", "EURCHF=X": "EUR/CHF",
-    "CADJPY=X": "CAD/JPY",
+# price_precision: decimals for SL/TP rounding and display.
+# default_spread: $0.30 round-trip cost retained for TD-era cross-book
+# comparability and used when the backtester receives no --spread override.
+# Oanda's measured round trip is $0.65-$0.72; research reports headline it
+# separately with --spread 0.65.
+PAIR_INFO = {
+    XAUUSD: {"display": "XAU/USD", "pip_value": 0.10,
+             "price_precision": 2, "default_spread": 0.30},
 }
 
 
 @dataclass
 class TechnicalConfig:
-    """Technical indicator parameters - carefully tuned for Forex."""
-    # EMA periods (fast/medium/slow for trend detection)
+    """Technical indicator parameters."""
     ema_fast: int = 9
     ema_medium: int = 21
     ema_slow: int = 50
     ema_trend: int = 200
+    # 4-EMA mode only: score partial ribbon alignment (2-3 of 4 checks)
+    # as NEUTRAL instead of a weak BUY/SELL. Research knob (2026-08-04
+    # indicator audit: partial-alignment trades averaged -0.008R vs
+    # +0.230R fully aligned on the M5 12mo Oanda book); default False =
+    # legacy behavior until a replay campaign validates flipping it.
+    ema_partial_neutral: bool = False
 
-    # RSI
     rsi_period: int = 14
     rsi_overbought: float = 70.0
     rsi_oversold: float = 30.0
     rsi_neutral_upper: float = 60.0
     rsi_neutral_lower: float = 40.0
 
-    # MACD (12, 26, 9 is standard)
     macd_fast: int = 12
     macd_slow: int = 26
     macd_signal: int = 9
 
-    # Bollinger Bands
     bb_period: int = 20
     bb_std: float = 2.0
 
-    # ATR for volatility measurement
     atr_period: int = 14
 
-    # Stochastic
-    stoch_k: int = 14
-    stoch_d: int = 3
-    stoch_overbought: float = 80.0
-    stoch_oversold: float = 20.0
-
-    # ADX for trend strength
     adx_period: int = 14
     adx_strong_trend: float = 25.0
     adx_very_strong: float = 50.0
 
-    # Ichimoku
-    ichimoku_tenkan: int = 9
-    ichimoku_kijun: int = 26
-    ichimoku_senkou_b: int = 52
+    divergence_lookback: int = 40
+    swing_sr_lookback: int = 50
 
-    # Volume profile
-    volume_ma_period: int = 20
+    disabled_indicators: List[str] = field(default_factory=list)
 
-    # Pivot points lookback
-    pivot_lookback: int = 1
+    ema_mode: str = "4ema"          # "4ema" (9/21/50/200) or "2ema" (50/200)
+    rsi_mode: str = "regime_aware"  # "classic" or "regime_aware"
+    macd_as_pillar: bool = True     # MACD histogram is a scored signal
 
-    # Divergence detection — how many bars back to scan for swing points
-    divergence_lookback: int = 14
+    # ROC (Rate of Change)
+    roc_period: int = 10
+    roc_threshold: float = 1.0     # % threshold for strong signal
+
+    # Keltner Channels
+    kc_period: int = 20
+    kc_atr_period: int = 10
+    kc_multiplier: float = 1.5
+
+    # ADX slope (trend acceleration detection)
+    adx_slope_lookback: int = 5
+    adx_slope_emerging_threshold: float = 3.0
+
+    regime_weights_override: Dict[str, Dict[str, float]] = field(default_factory=dict)
 
 
 @dataclass
 class SignalConfig:
-    """Signal scoring and filtering parameters."""
-    # Minimum confidence to emit a signal (0-100)
-    min_confidence: float = 55.0
+    """Signal scoring & filtering."""
+    mtf_agreement_threshold: int = 2  # of 3 timeframes
 
-    # Weight allocation for composite score
-    weights: Dict[str, float] = field(default_factory=lambda: {
-        "trend": 0.25,          # EMA alignment, ADX
-        "momentum": 0.20,       # RSI, Stochastic, MACD
-        "volatility": 0.15,     # BB position, ATR regime
-        "price_action": 0.15,   # Candlestick patterns, S/R
-        "multi_timeframe": 0.15,# Higher TF confirmation
-        "sentiment": 0.10,      # News/calendar events
-    })
+    min_rr_ratio: float = 0.5
+    # TP tightening (May 2026) was tuned on the pre-spread backtester.
+    # Re-tested 2026-07-08 with $0.30 spread + gap fills: 0.8 nearly
+    # halves total R (+36.1R vs +66.8R, worse in both train/val halves)
+    # — smaller wins make the fixed spread a larger share of every TP.
+    tp_tightening_factor: float = 1.0
 
-    # Multi-timeframe confirmation requirements
-    mtf_agreement_threshold: int = 2  # out of 3 timeframes must agree
+    # ── Trade geometry ────────────────────────────────────────────
+    # Deliberately $0.30 for TD-era cross-book comparability. Oanda's measured
+    # round trip is $0.65-$0.72, reported separately in re-pin campaigns.
+    # This is also a LIVE gate input: increasing it to $0.65 raises the TP1
+    # floor from about $9 to $19.50, so it requires rule-1 revalidation.
+    round_trip_cost: float = 0.30
+    # Spread gate: reject setups whose TP1 distance is below
+    # round_trip_cost × this multiple (caps cost at ~3% of target).
+    # Validated 2026-07: small-target trades lose in both data halves
+    # on M5 while TP≥$10 trades earn ~+0.08R/trade.
+    min_tp_cost_multiple: float = 30.0
+    # Fixed SL floor in pips (80 pips = $8 on XAU). 0 disables — SL then
+    # floors at ATR × v2_sl_atr_multiplier only.
+    min_sl_pips: float = 80.0
+    # TP1 floor as a fraction of SL distance (0.5 = historical RR 0.5).
+    tp_floor_rr: float = 0.5
 
-    # Signal cooldown (prevent over-signalling same direction)
-    cooldown_bars: int = 5
-
-    # Risk management defaults
-    default_risk_pct: float = 1.0  # risk 1% per trade
-    default_rr_ratio: float = 2.0  # 1:2 risk-reward minimum
+    # ── Trailing-stop exit (2026-07) ──────────────────────────────
+    # Once favorable excursion reaches trail_arm_r × risk, the stop
+    # trails trail_dist_r × risk behind the best price (updated at bar
+    # close). TP1 becomes advisory — the trail is the exit. Validated
+    # on 12mo M5: +105R vs +48R with the fixed RR-0.5 target, balanced
+    # train/val, positive every quarter, PF 2.27. The RR-0.5 target was
+    # the binding constraint — winners kept running after TP.
+    # Set trail_arm_r = 0 to disable and exit at TP1 as before.
+    trail_arm_r: float = 0.6
+    trail_dist_r: float = 0.75
+    # Pullback (XAU_PB) trades run tighter stops (~$8 vs ~$25), so the
+    # global 0.6R arm fires on noise. Arm later; whole 0.8-arm row of
+    # the grid is +35-38R vs +32R at global params (balanced halves).
+    pb_trail_arm_r: float = 0.8
+    pb_trail_dist_r: float = 0.75
 
 
 @dataclass
 class NewsConfig:
-    """Economic calendar and sentiment configuration."""
-    # High-impact event currencies
-    tracked_currencies: List[str] = field(default_factory=lambda: [
-        "USD", "EUR", "GBP", "JPY", "CHF", "AUD", "CAD", "NZD"
-    ])
-
-    # Event impact levels to monitor
-    min_impact: str = "medium"  # "low", "medium", "high"
-
-    # Pre-event buffer (minutes before high-impact event = no new signals)
+    """Economic calendar & sentiment."""
+    tracked_currencies: List[str] = field(default_factory=lambda: ["USD", "XAU"])
     pre_event_buffer_minutes: int = 30
-
-    # Post-event volatility window (minutes after event = cautious signals)
     post_event_buffer_minutes: int = 15
+    # Whether event risk gates the SIGNAL (danger: confidence hard-capped
+    # at 30; caution: 15-point penalty). Warnings are always emitted
+    # regardless. Default False = XAU-tuned: replaying the live gates
+    # against the validated 12mo XAU M5 book (547 trades, +139.9R)
+    # showed they suppress 25-35 trades worth +4.6..+7.2R/yr — trades
+    # entered near USD releases OUTPERFORM (+0.475R vs +0.256R avg).
+    # Audit 2026-07-11 (calendar-replay study; tooling since removed).
+    event_risk_caps_signals: bool = False
+    # Whether headline sentiment moves the SIGNAL (aligned: +5 confidence;
+    # conflicting: 10-point penalty). Warnings are always emitted
+    # regardless. Default False = news is informational only; decisions
+    # are pure technical, matching the backtester (use_news=False).
+    sentiment_affects_signals: bool = False
+    # Wider stay-out window after a high-impact release that surprised
+    # vs forecast (inline releases use post_event_buffer_minutes).
+    post_event_surprise_buffer_minutes: int = 45
+    # How long fetched calendar/headline data is reused across scans.
+    # Must stay below pre_event_buffer_minutes so an event can't cross
+    # into the danger window while a stale cache says it's clear.
+    context_cache_ttl_seconds: int = 600
+    # Maximum age of RSS headlines accepted for scoring. Older headlines
+    # (and headlines with no parsable publish time) are discarded before
+    # keyword analysis and LLM enrichment. 12 h → 4 h on 2026-09-04: the
+    # 3-week signal log showed the 12 h mean was a trailing echo of the
+    # previous day's price move (see backtest/research/
+    # news_sentiment_value_2026-09-04.md). Display/warning-only either way.
+    headline_max_age_seconds: int = 14400  # 4 hours
 
-    # News sentiment sources
     rss_feeds: List[str] = field(default_factory=lambda: [
-        "https://www.forexlive.com/feed/",
+        "https://investinglive.com/feed/",
         "https://www.fxstreet.com/rss",
+        # Google News query targeted at gold drivers — also carries Kitco,
+        # Reuters etc. whose direct feeds are unavailable.
+        "https://news.google.com/rss/search?q=gold+price+OR+fed+OR+inflation&hl=en-US&gl=US&ceid=US:en",
+        # CNBC "Economy" section
+        "https://search.cnbc.com/rs/search/combinedcms/view.xml?partnerId=wrss01&id=20910258",
     ])
 
 
+@dataclass
+class XauStrategyConfig:
+    """XAU/USD strategy config."""
+    enabled: bool = True
+    tp_atr_multiplier: float = 1.0
+    sl_atr_multiplier: float = 2.0
+    blocked_hours_utc: List[int] = field(default_factory=lambda: [0, 1, 2, 10, 11, 13, 16, 21])
+    weekend_filter: bool = True
+    min_atr_threshold: float = 1.0
+    v2_sl_atr_multiplier: float = 2.0
+    allowed_weekdays: List[int] = field(default_factory=lambda: [0, 1, 2, 3, 4, 6])  # Mon-Fri + Sun (Sydney open)
+    sell_min_quality: List[str] = field(default_factory=lambda: ["A+", "A", "B"])
+    # Volatility regimes in which SELL setups are allowed (low/normal-vol
+    # shorts lose consistently against gold's macro uptrend).
+    sell_vol_regimes: List[str] = field(default_factory=lambda: ["high", "extreme"])
+    # Market regimes in which no XAU setup may fire (any direction).
+    # Empty by default; per-timeframe factories opt in — e.g. the mixed
+    # "trending" regime (strong ADX, EMAs disagree) on M15.
+    blocked_regimes: List[str] = field(default_factory=list)
+    # Market regimes in which XAU SELL setups may not fire. Empty by
+    # default; per-timeframe factories opt in.
+    sell_blocked_regimes: List[str] = field(default_factory=list)
+    # Bar-CLOSE (decision-time) UTC hours in which XAU SELL setups may
+    # not fire. Empty by default; per-timeframe factories opt in.
+    # BUYs are unaffected (unlike blocked_hours_utc, which is both-way).
+    sell_blocked_hours_utc: List[int] = field(default_factory=list)
+
+    # ── M5-specific filters ──────────────────────────────────────
+    # M5 signals are noisy; require stronger confirmation before
+    # generating trade setups. Without these, M5 is breakeven
+    # (+0.01R/trade) with 21% drawdown.
+    # Relaxed 2026-07-30 (30→25, 10→5) in the relaxation campaign:
+    # the combined config (with the M5 hour-21 unblock) beat control
+    # in ALL 6 calendar years of the 5yr Oanda replay (+230.9R vs
+    # +168.0R, PF 1.26 vs 1.21, max DD 21.3 vs 27.0), 12mo +107.4R
+    # vs +84.6R PF 1.36 both halves positive, OOS 2024-06→2025-03
+    # +26.1R vs +17.2R, and at $0.70 spread +86.9R vs +66.4R.
+    # Score floors below 25 are vacuous (grade C requires |score|≥25).
+    m5_require_mtf: bool = True
+    m5_min_abs_score: float = 25.0
+    m5_min_confirmation_score: float = 5.0
+    # A-grade M5 SELLs enter at the extreme of temporary dips and
+    # lose (-0.017R/trade on 163 trades) — maximum conviction at
+    # the worst entry. Cap SELL quality so A-grade can't trigger
+    # the ATR-based setup generator; B-grade swing setups survive
+    # because they catch moderate drops, not extremes.
+    m5_sell_max_quality: List[str] = field(default_factory=lambda: ["B", "C"])
+    # 'high' vol regime on M5 is chop transition — elevated noise
+    # without directional expansion. 61% WR / negative total across
+    # 12mo while low/normal/extreme are all positive (validated 2026-07).
+    m5_blocked_vol_regimes: List[str] = field(default_factory=lambda: ["high"])
+    # Block M5 SELLs when RSI is already < this floor: the dip is
+    # extended and mean-reverts before the target (negative in both
+    # halves of 6mo and 12mo backtests).
+    m5_sell_min_rsi: float = 25.0
+
+    # ── M5 pullback entry (XAU_PB) ──
+    # Second entry archetype: enter on the retrace to EMA21/KC-mid in
+    # an established trend instead of at Keltner-breakout extension.
+    # Validated 2026-07: +38.3R/254 trades on 12mo, positive both
+    # halves and all four full quarters, core trades undisturbed.
+    m5_pullback_enabled: bool = True
+    # Composite score is weaker mid-pullback BY DESIGN (price sits
+    # below the momentum bands) — separate, lower floor than Gate 1.
+    m5_pullback_min_score: float = 15.0
+    m5_pullback_adx_min: float = 20.0
+    # RSI must be in the pullback zone, NOT the extended zone the
+    # breakout entries fire in (those run 65-80).
+    m5_pullback_rsi_buy: tuple = (35.0, 62.0)
+    m5_pullback_rsi_sell: tuple = (38.0, 65.0)
+    # Retrace depth from the pre-pullback swing extreme, in ATRs.
+    # Shallower = still extended; deeper = trend likely broken.
+    # M5 gold trend legs measure 3-4 ATR swing-to-trough (median 3.9),
+    # so the ceiling sits well above that of higher TFs.
+    m5_pullback_retrace_atr: tuple = (0.8, 6.0)
+    # Price must have touched EMA21/KC-mid within this many bars.
+    m5_pullback_touch_bars: int = 3
+
+    # ── Chart-pattern entry (XAU_CP) ──────────────────────────────
+    # Third entry archetype: completion of a classical chart pattern
+    # (see chart_patterns.py for the pattern set and design contract).
+    # Defaults are textbook geometry, NOT tuned values. Disabled by
+    # default — per-TF adoption happens in pair_signals/xauusd.py only
+    # after the pattern book passes the standard validation rule
+    # (positive R both halves of ≥12mo, PF ≥ 1.3, $0.30 spread,
+    # trailing exits, next-open fills).
+    cp_enabled: bool = False
+    cp_patterns: List[str] = field(default_factory=lambda: [
+        "inv_hs", "hs_top", "double_bottom", "double_top",
+        "bull_flag", "bear_flag",
+    ])
+    cp_lookback: int = 72          # bars scanned for pattern structure
+    cp_swing_window: int = 3       # bars each side to confirm a swing
+    cp_recency_bars: int = 20      # last extreme must be this fresh
+    cp_min_sep_bars: int = 8       # min bars between double extremes
+    cp_tolerance_atr: float = 0.5  # double top/bottom level match tol
+    cp_min_height_atr: float = 1.5   # pattern height floor (noise gate)
+    cp_max_height_atr: float = 8.0   # pattern height ceiling
+    cp_hs_head_min_atr: float = 0.5  # head must exceed shoulders by this
+    cp_hs_shoulder_tol_atr: float = 1.0  # shoulder level match tol
+    cp_flag_pole_atr: float = 3.0    # min pole size
+    cp_flag_pole_bars: int = 12      # pole measured over ≤ this many bars
+    cp_flag_min_bars: int = 3        # flag consolidation length bounds
+    cp_flag_max_bars: int = 12
+    cp_flag_retrace: tuple = (0.1, 0.5)  # flag depth as fraction of pole
+
+    # ── Big-candle entry gate ──────────────────────────────────────
+    # When the recent window on the scan's own primary TF contains an
+    # abnormally large candle (bar range >= K x the PRIOR bar's ATR —
+    # institutional burst; price may reverse violently), no XAU entry
+    # of ANY kind may fire (core, XAU_PB, XAU_CP). 0.0 = disabled.
+    # Per-TF factories opt in only after the standard validation rule.
+    big_candle_atr_multiple: float = 0.0
+    big_candle_lookback_hours: float = 24.0
+
+
+@dataclass
+class FilterConfig:
+    """Regime and session filters for live signals."""
+    regime_filter: bool = False
+    regime_filter_timeframe: Timeframe = Timeframe.H4
+    regime_filter_min_adx: float = 25.0
+
+    session_filter: bool = True
+    session_start_hour: int = 3
+    session_end_hour: int = 23
+    blocked_hours_utc: List[int] = field(default_factory=lambda: [0, 1, 2, 10, 11, 13, 16, 21])
+
+    allowed_weekdays: List[int] = field(default_factory=lambda: [0, 1, 2, 3, 4, 6])  # Mon-Fri + Sun (Sydney open)
+
+
 @dataclass
 class PlatformConfig:
-    """Overall platform configuration."""
+    """Top-level configuration."""
     technical: TechnicalConfig = field(default_factory=TechnicalConfig)
     signals: SignalConfig = field(default_factory=SignalConfig)
     news: NewsConfig = field(default_factory=NewsConfig)
+    backtest: FilterConfig = field(default_factory=FilterConfig)
+    xau_strategy: XauStrategyConfig = field(default_factory=XauStrategyConfig)
 
-    # Default pairs to monitor
-    watch_pairs: List[str] = field(default_factory=lambda: MAJOR_PAIRS)
+    symbol: str = XAUUSD
+    display_name: str = "XAU/USD"
+    pip_value: float = 0.10
 
-    # Analysis timeframes (primary + confirmation)
     primary_timeframe: Timeframe = Timeframe.H1
     confirmation_timeframes: List[Timeframe] = field(
         default_factory=lambda: [Timeframe.M15, Timeframe.H4]
     )
 
-    # Data source
-    data_lookback_days: int = 60  # enough for 200 EMA on H1
+    data_lookback_days: int = 60
 
-    # Twelve Data API key (overridden by TWELVE_DATA_API_KEY env var)
-    twelve_data_api_key: str = "f563b22ffdfd4924955decbe1a8c8701"
+    refresh_interval: int = 300
 
-    # Dashboard refresh interval (seconds)
-    refresh_interval: int = 300  # 5 minutes
+    scan_pairs: List[str] = field(
+        default_factory=lambda: [XAUUSD]
+    )
+    scan_timeframes: List[Timeframe] = field(
+        default_factory=lambda: [Timeframe.M5, Timeframe.M15, Timeframe.H1, Timeframe.H4]
+    )
+    ws_cooldown_seconds: int = 300
 
-    # Logging
     log_level: str = "INFO"
-    log_signals_to_file: bool = True
-    signal_log_path: str = "signals.log"

dashboard_publisher.py

@@ -0,0 +1,479 @@
+#!/usr/bin/env python3
+"""
+XAU/USD static dashboard publisher.
+
+Standalone daemon that scans XAU/USD at candle closes and atomically
+writes `snapshot.json` + `news.json` (plus a copy of
+`static/dashboard.html`) into an Apache-served directory. Apache does
+all web serving — this process never opens a port, so a failure here
+can only ever mean a stale page.
+
+Independence rules (owner requirement — the telegram bot must be
+immune to anything on the web side):
+  * Runs as its own process; never imported by, and never imports,
+    `telegram_bot` (telegram_bot resolves credentials at import). The
+    confirmation map and boundary math are therefore duplicated here
+    as local constants.
+  * The only shared-code dependency beyond the standard signal path is
+    the additive `NewsEngine.get_week_events` accessor, which no
+    production signal path calls.
+
+Scheduling: every M15/H1/H4 boundary is also an M5 boundary (all
+intervals are multiples of 5, anchored to UTC midnight), so one
+sequential loop sleeps to the next M5 boundary (+10 s buffer) and scans
+whichever timeframes just closed. Due-classification uses the RAW wake
+time — flooring 12:00:10 to the 5-minute grid yields the 12:00 boundary
+that just closed. Do NOT subtract the buffer first: 12:00:10 − 15 s
+floors to 11:55 and silently skips every H1/H4 scan (v1 bug, 2026-08-02).
+
+Scans run sequentially M5→M15→H1→H4 through one shared data provider so
+later timeframes hit the warm boundary cache — that ordering IS the
+duplicate-fetch dedup (providers have no in-flight dedup; Oanda has no
+throttle). Keep it sequential.
+
+Run:  python dashboard_publisher.py [--once] [--out DIR]
+Out dir precedence: --out → DASHBOARD_OUT_DIR env → /var/www/trade
+"""
+
+import argparse
+import json
+import hashlib
+import logging
+import math
+import numbers
+import os
+import shutil
+import sys
+import time
+from datetime import datetime, timezone
+from logging.handlers import TimedRotatingFileHandler
+from typing import Dict, List, Optional
+
+from agent_panel import attach_panel
+from config import PlatformConfig, Timeframe, XAUUSD
+from main import report_to_dict
+from news_engine import NewsEngine
+from pair_signals import produce_signal
+from providers import make_data_provider
+import providers.base as _providers_base
+
+logger = logging.getLogger("dashboard")
+
+DASHBOARD_TFS = [Timeframe.M5, Timeframe.M15, Timeframe.H1, Timeframe.H4]
+
+# Local copy of the bot's confirmation map (see independence rules above).
+CONFIRMATIONS = {
+    Timeframe.M5:  [Timeframe.M15, Timeframe.H1],
+    Timeframe.M15: [Timeframe.M5, Timeframe.H1],
+    Timeframe.H1:  [Timeframe.M15, Timeframe.H4],
+    Timeframe.H4:  [Timeframe.H1, Timeframe.D1],
+}
+
+TF_INTERVAL_MINUTES = {
+    Timeframe.M5: 5, Timeframe.M15: 15, Timeframe.H1: 60, Timeframe.H4: 240,
+}
+
+DEFAULT_OUT_DIR = "/var/www/trade"
+SCAN_BUFFER_SECONDS = 10
+SNAPSHOT_FILE = "snapshot.json"
+NEWS_FILE = "news.json"
+PAGE_FILE = "dashboard.html"
+SUMMARY_PHP_FILE = "news_summary.php"
+# Repo files copied into the out dir at daemon start (git pull + restart
+# updates them). The PHP endpoint reads news.json in place, so it lives
+# next to the page.
+STATIC_FILES = [PAGE_FILE, "dashboard.css", "dashboard.js", SUMMARY_PHP_FILE]
+
+
+# ══════════════════════════════════════════════════════════════════
+#  Pure layer (no network, fully covered by test_all.py)
+# ══════════════════════════════════════════════════════════════════
+
+def _safe_num(v):
+    """Coerce to a JSON-safe float; NaN/±inf become None."""
+    try:
+        f = float(v)
+    except (TypeError, ValueError):
+        return v
+    if math.isnan(f) or math.isinf(f):
+        return None
+    return f
+
+
+def _sanitize(obj):
+    """Recursively make a payload safe for json.dumps(allow_nan=False):
+    NaN/inf → None, numpy scalars → native ints/floats."""
+    if isinstance(obj, bool):
+        return obj
+    if isinstance(obj, dict):
+        return {k: _sanitize(v) for k, v in obj.items()}
+    if isinstance(obj, (list, tuple)):
+        return [_sanitize(v) for v in obj]
+    if isinstance(obj, numbers.Integral):
+        return int(obj)
+    if isinstance(obj, numbers.Real):
+        return _safe_num(obj)
+    return obj
+
+
+def analysis_to_dict(analysis) -> Optional[dict]:
+    """Serialize a TechnicalAnalysis. Iterates analysis.indicators only —
+    never vars()/asdict(), which would drag dynamic pandas attrs
+    (_rsi_series, _macd_histogram, …) into the payload. ATR (weight 0)
+    is surfaced separately as context, not in the directional grid."""
+    if analysis is None:
+        return None
+    indicators = {}
+    atr_context = None
+    for name, ind in analysis.indicators.items():
+        if name.startswith("_"):
+            continue
+        entry = {
+            "signal": ind.signal.name,
+            "value": _safe_num(ind.value),
+            "confidence": _safe_num(ind.confidence),
+            "weight": _safe_num(ind.weight),
+            "description": ind.description,
+        }
+        if name == "atr":
+            atr_context = entry
+        elif ind.weight != 0:
+            indicators[name] = entry
+    return {
+        "timeframe": analysis.timeframe,
+        "timestamp": str(analysis.timestamp),
+        "composite_signal": analysis.composite_signal.name,
+        "composite_score": _safe_num(analysis.composite_score),
+        "confidence": _safe_num(analysis.confidence),
+        "market_regime": analysis.market_regime,
+        "volatility_regime": analysis.volatility_regime,
+        "pattern_signals": list(analysis.pattern_signals),
+        "key_levels": {k: _safe_num(v) for k, v in analysis.key_levels.items()},
+        "indicators": indicators,
+        "atr_context": atr_context,
+    }
+
+
+def report_to_webapp_dict(report) -> dict:
+    """main.report_to_dict + the per-TF detail the dashboard needs."""
+    d = report_to_dict(report)
+    d["primary_signal"] = report.primary_signal.name
+    d["primary_score"] = _safe_num(report.primary_score)
+    d["mtf_scores"] = {k: _safe_num(v) for k, v in report.mtf_scores.items()}
+    d["exhaustion_count"] = report.exhaustion_count
+    d["primary_analysis"] = analysis_to_dict(report.primary_analysis)
+    d["confirmation_analyses"] = [
+        analysis_to_dict(a) for a in report.confirmation_analyses
+    ]
+    return _sanitize(d)
+
+
+def event_to_dict(ev) -> dict:
+    """EconomicEvent → dict. surprise/minutes_until/is_high_impact are
+    properties, so asdict() would skip them — added explicitly.
+    actual/forecast/previous stay strings as published ('228K', '0.3%')."""
+    return {
+        "time": ev.time.astimezone(timezone.utc).isoformat(),
+        "currency": ev.currency,
+        "impact": ev.impact,
+        "event": ev.event,
+        "forecast": ev.forecast,
+        "previous": ev.previous,
+        "actual": ev.actual,
+        "surprise": ev.surprise,
+        "minutes_until": _safe_num(ev.minutes_until),
+        "is_high_impact": ev.is_high_impact,
+    }
+
+
+def headline_to_dict(h) -> dict:
+    # id uses the same sha1(headline) scheme as NewsEngine._llm_sent_cache;
+    # it is the join key the PHP summary endpoint whitelists against.
+    return {
+        "id": hashlib.sha1(h.headline.encode("utf-8")).hexdigest(),
+        "headline": h.headline,
+        "source": h.source,
+        "link": h.link,
+        "description": h.description,
+        "publisher": h.publisher,
+        "timestamp": h.timestamp.astimezone(timezone.utc).isoformat()
+                     if h.timestamp else None,
+        "currencies": list(h.currencies_mentioned),
+        "score": _safe_num(h.sentiment_score),
+        "keywords": list(h.keywords),
+        "score_by_currency": {
+            k: _safe_num(v) for k, v in h.sentiment_by_currency.items()
+        },
+    }
+
+
+def build_news_payload(context, week_events, now: datetime) -> dict:
+    """news.json: full calendar week + scored headlines + LLM summary."""
+    headlines = sorted(
+        context.news_headlines,
+        key=lambda h: h.timestamp or now, reverse=True,
+    )
+    return _sanitize({
+        "generated_at": now.isoformat(),
+        "llm_summary": context.llm_summary,
+        "sentiment": dict(context.news_sentiment),
+        "high_impact_warning": context.high_impact_warning,
+        "warning_message": context.warning_message,
+        "calendar_available": context.calendar_available,
+        "events": [event_to_dict(e) for e in week_events],
+        "headlines": [headline_to_dict(h) for h in headlines],
+    })
+
+
+def due_timeframes(wake_dt: datetime) -> List[Timeframe]:
+    """Timeframes whose candle just closed at the boundary this wake
+    belongs to. Takes the RAW wake time (boundary + buffer): flooring to
+    the 5-minute grid yields the boundary that just passed. Never
+    subtract the buffer first — see module docstring (v1 bug)."""
+    minutes = wake_dt.hour * 60 + wake_dt.minute
+    floored = minutes - (minutes % 5)
+    return [tf for tf in DASHBOARD_TFS if floored % TF_INTERVAL_MINUTES[tf] == 0]
+
+
+def seconds_until_boundary(interval_minutes: int, buffer: int = 10) -> float:
+    # Same math as telegram_bot.py — kept local so the dashboard never
+    # imports bot modules (independence rules above).
+    now = datetime.now(timezone.utc)
+    total_minutes = now.hour * 60 + now.minute
+    elapsed_minutes = total_minutes % interval_minutes
+    elapsed_seconds = elapsed_minutes * 60 + now.second + now.microsecond / 1e6
+    interval_seconds = interval_minutes * 60
+    remaining = interval_seconds - elapsed_seconds + buffer
+    # NOTE: remaining > buffer always (elapsed < interval by construction);
+    # a call at the exact boundary instant targets the NEXT boundary —
+    # steady-state loops wake at boundary+buffer, so only a process
+    # started in the boundary microsecond skips one scan (2026-08-04
+    # audit I42; the old `remaining <= buffer` guard was unreachable).
+    return remaining
+
+
+def _empty_tf_entry() -> dict:
+    return {"status": "warming", "scanned_at": None, "scan_seconds": None,
+            "error": None, "error_at": None, "report": None}
+
+
+def build_snapshot_payload(snapshots: Dict[str, dict], now: datetime) -> dict:
+    """snapshot.json: one entry per dashboard TF. Timeframes not yet
+    scanned appear as status="warming" so the page can render a
+    placeholder instead of a hole. server_time_epoch lets the client
+    compute its clock offset once so countdowns don't drift."""
+    timeframes = {}
+    for tf in DASHBOARD_TFS:
+        interval = TF_INTERVAL_MINUTES[tf]
+        minutes = now.hour * 60 + now.minute
+        secs_into = (minutes % interval) * 60 + now.second + now.microsecond / 1e6
+        next_close = now.timestamp() + (interval * 60 - secs_into)
+        entry = _empty_tf_entry()
+        entry.update(snapshots.get(tf.value) or {})
+        entry["next_close_epoch"] = next_close
+        entry["next_scan_epoch"] = next_close + SCAN_BUFFER_SECONDS
+        timeframes[tf.value] = entry
+    return _sanitize({
+        "symbol": XAUUSD,
+        "server_time_epoch": now.timestamp(),
+        "market_closed": _providers_base._is_forex_weekend(now),
+        "timeframes": timeframes,
+    })
+
+
+def publish(obj: dict, path: str) -> None:
+    """Atomic JSON write: temp file + os.replace, so Apache never serves
+    a torn file. allow_nan=False — a NaN slipping through would render
+    the page blank (JSON.parse rejects bare NaN)."""
+    data = json.dumps(obj, allow_nan=False, separators=(",", ":"))
+    tmp = path + ".tmp"
+    with open(tmp, "w", encoding="utf-8") as fh:
+        fh.write(data)
+    os.replace(tmp, path)
+
+
+# ══════════════════════════════════════════════════════════════════
+#  Runtime layer (constructed only from main(); never at import)
+# ══════════════════════════════════════════════════════════════════
+
+def _setup_logging(level: str = "INFO") -> None:
+    log_dir = os.path.join(os.path.dirname(os.path.abspath(__file__)), "logs")
+    os.makedirs(log_dir, exist_ok=True)
+    log_file = os.path.join(log_dir, "dashboard.log")
+
+    fmt = logging.Formatter(
+        "%(asctime)s [%(levelname)s] %(name)s: %(message)s",
+        datefmt="%Y-%m-%d %H:%M:%S")
+
+    file_handler = TimedRotatingFileHandler(
+        log_file, when="h", interval=3, backupCount=240, utc=True)
+    file_handler.setFormatter(fmt)
+
+    console_handler = logging.StreamHandler(sys.stdout)
+    console_handler.setFormatter(fmt)
+
+    logging.basicConfig(
+        level=getattr(logging, level.upper(), logging.INFO),
+        handlers=[file_handler, console_handler],
+        force=True)
+
+
+class DashboardPublisher:
+    def __init__(self, out_dir: str):
+        self.out_dir = out_dir
+        self.snapshots: Dict[str, dict] = {}
+        self._reports = {}
+        self.news_payload: Optional[dict] = None
+        self._dp = None
+        self._news_engine: Optional[NewsEngine] = None
+
+    def _provider(self):
+        if self._dp is None:
+            self._dp = make_data_provider(PlatformConfig())
+        return self._dp
+
+    def _news(self) -> NewsEngine:
+        if self._news_engine is None:
+            self._news_engine = NewsEngine(PlatformConfig().news)
+        return self._news_engine
+
+    def install_static(self) -> None:
+        """Copy the repo's static files (page + PHP summary endpoint)
+        into the out dir (atomic), so git pull + restart also updates
+        them. The ai/ cache dir is NOT created here — the daemon runs as
+        a different user than Apache; that dir is a one-time deploy step
+        (deploy/README-dashboard.md)."""
+        static_dir = os.path.join(
+            os.path.dirname(os.path.abspath(__file__)), "static")
+        for name in STATIC_FILES:
+            src = os.path.join(static_dir, name)
+            if not os.path.exists(src):
+                logger.warning(f"{src} missing — not installed")
+                continue
+            dst = os.path.join(self.out_dir, name)
+            shutil.copyfile(src, dst + ".tmp")
+            os.replace(dst + ".tmp", dst)
+
+    def scan_tf(self, tf: Timeframe) -> None:
+        started = time.time()
+        try:
+            report = produce_signal(XAUUSD, tf, self._provider(),
+                                    CONFIRMATIONS[tf])
+            self._reports[tf.value] = report
+            self.snapshots[tf.value] = {
+                "status": "ok",
+                "scanned_at": datetime.now(timezone.utc).isoformat(),
+                "scan_seconds": round(time.time() - started, 2),
+                "error": None,
+                "error_at": None,
+                "report": report_to_webapp_dict(report),
+            }
+            logger.info(f"{tf.value}: {report.final_signal.name} "
+                        f"q={report.signal_quality or '-'} "
+                        f"({time.time() - started:.1f}s)")
+        except Exception:
+            # Keep the last good report; the page shows it with an
+            # error badge instead of going blank.
+            logger.exception(f"{tf.value} scan failed")
+            self._reports.pop(tf.value, None)
+            entry = self.snapshots.get(tf.value) or _empty_tf_entry()
+            entry["status"] = "error"
+            # The public snapshot carries a stable message; exception
+            # detail remains in dashboard.log for the operator.
+            entry["error"] = "Scan unavailable"
+            entry["error_at"] = datetime.now(timezone.utc).isoformat()
+            self.snapshots[tf.value] = entry
+
+    def attach_panels(self, tfs: List[Timeframe]) -> bool:
+        """Add advisory panels after publishing the technical snapshot.
+
+        A panel is display-only and can take tens of seconds on a cold LLM
+        request. Keeping its report object locally lets the page receive the
+        candle-close signal first, then a second atomic snapshot only when a
+        panel produced a result.
+        """
+        updated = False
+        for tf in tfs:
+            report = self._reports.get(tf.value)
+            if report is None:
+                continue
+            if not attach_panel(report, surface="dashboard"):
+                continue
+            entry = self.snapshots.get(tf.value)
+            if entry is None or entry.get("status") != "ok":
+                continue
+            entry["report"] = report_to_webapp_dict(report)
+            updated = True
+        return updated
+
+    def refresh_news(self) -> None:
+        try:
+            ne = self._news()
+            context = ne.get_fundamental_context()
+            week = ne.get_week_events()
+            self.news_payload = build_news_payload(
+                context, week, datetime.now(timezone.utc))
+        except Exception:
+            logger.exception("news refresh failed — keeping last payload")
+
+    def publish_snapshot(self) -> None:
+        publish(build_snapshot_payload(
+            self.snapshots, datetime.now(timezone.utc)),
+            os.path.join(self.out_dir, SNAPSHOT_FILE))
+
+    def publish_news(self) -> None:
+        if self.news_payload is not None:
+            publish(self.news_payload, os.path.join(self.out_dir, NEWS_FILE))
+
+    def publish_all(self) -> None:
+        self.publish_snapshot()
+        self.publish_news()
+
+    def run_cycle(self, tfs: List[Timeframe]) -> None:
+        for tf in tfs:  # sequential — see module docstring (cache dedup)
+            self.scan_tf(tf)
+        # Technical signals land before any optional slow work. News and the
+        # advisory panel are display-only and may take 20-30 seconds cold.
+        self.publish_snapshot()
+        if self.attach_panels(tfs):
+            self.publish_snapshot()
+        self.refresh_news()
+        self.publish_news()
+
+    def run_forever(self) -> None:
+        self.run_cycle(DASHBOARD_TFS)  # populate the page at boot
+        while True:
+            delay = seconds_until_boundary(5, buffer=SCAN_BUFFER_SECONDS)
+            logger.info(f"next M5 boundary in {delay:.0f}s")
+            time.sleep(delay)
+            try:
+                due = due_timeframes(datetime.now(timezone.utc))
+                logger.info(f"boundary wake — scanning {[t.value for t in due]}")
+                self.run_cycle(due)
+            except Exception:
+                logger.exception("cycle failed")
+
+
+def main(argv=None) -> None:
+    ap = argparse.ArgumentParser(description="XAU/USD static dashboard publisher")
+    ap.add_argument("--once", action="store_true",
+                    help="one scan+publish cycle, then exit")
+    ap.add_argument("--out",
+                    default=os.environ.get("DASHBOARD_OUT_DIR", DEFAULT_OUT_DIR),
+                    help="output directory served by Apache "
+                         f"(default: $DASHBOARD_OUT_DIR or {DEFAULT_OUT_DIR})")
+    args = ap.parse_args(argv)
+
+    _setup_logging()
+    os.makedirs(args.out, exist_ok=True)
+    pub = DashboardPublisher(args.out)
+    pub.install_static()
+    if args.once:
+        pub.run_cycle(DASHBOARD_TFS)
+        print(f"Published to {args.out}")
+        return
+    pub.run_forever()
+
+
+if __name__ == "__main__":
+    main()

data_provider.py

@@ -1,547 +0,0 @@
-"""
-Forex Signal Intelligence Platform - Data Provider
-====================================================
-Retrieves and validates market data from Twelve Data API.
-Falls back to synthetic GBM data if the API is unavailable.
-"""
-
-import hashlib
-import logging
-import os
-import time
-from collections import deque
-from datetime import datetime, timedelta, timezone
-from typing import Dict, List, Optional
-
-import numpy as np
-import pandas as pd
-import requests
-
-from config import PlatformConfig, Timeframe, SYMBOL_DISPLAY
-
-logger = logging.getLogger(__name__)
-
-
-class DataIntegrityError(Exception):
-    """Raised when data fails validation checks."""
-    pass
-
-
-class ForexDataProvider:
-    """
-    Forex data provider backed by Twelve Data API.
-
-    Twelve Data has native 4h candles, eliminating the H1→H4 resampling
-    hack needed for Yahoo Finance. Free tier: 800 credits/day, 8 req/min.
-    """
-
-    TD_BASE_URL = "https://api.twelvedata.com"
-
-    TD_INTERVALS = {
-        Timeframe.M1:  "1min",
-        Timeframe.M5:  "5min",
-        Timeframe.M15: "15min",
-        Timeframe.M30: "30min",
-        Timeframe.H1:  "1h",
-        Timeframe.H4:  "4h",
-        Timeframe.D1:  "1day",
-        Timeframe.W1:  "1week",
-    }
-
-    # Approximate bars per day for each timeframe (used to size outputsize)
-    BARS_PER_DAY = {
-        Timeframe.M1:  1440,
-        Timeframe.M5:  288,
-        Timeframe.M15: 96,
-        Timeframe.M30: 48,
-        Timeframe.H1:  24,
-        Timeframe.H4:  6,
-        Timeframe.D1:  1,
-        Timeframe.W1:  0.143,
-    }
-
-    # Free plan: 8 API credits per minute (each symbol = 1 credit)
-    RATE_LIMIT = 8
-    RATE_WINDOW = 60.0
-
-    def __init__(self, config: PlatformConfig):
-        self.config = config
-        self._api_key = (
-            os.environ.get("TWELVE_DATA_API_KEY")
-            or config.twelve_data_api_key
-        )
-        self._cache: Dict[str, pd.DataFrame] = {}
-        self._cache_timestamps: Dict[str, float] = {}
-        self._request_times: deque = deque()  # timestamps of recent API credits used
-        self._session = requests.Session()
-        self._session.headers.update({
-            "User-Agent": "Mozilla/5.0 (compatible; ForexIntel/1.0)"
-        })
-
-    @staticmethod
-    def _to_td_symbol(symbol: str) -> str:
-        """Convert internal symbol (EURUSD=X) to Twelve Data format (EUR/USD)."""
-        clean = symbol.replace("=X", "")
-        return f"{clean[:3]}/{clean[3:]}"
-
-    def fetch_ohlcv(
-        self,
-        symbol: str,
-        timeframe: Timeframe,
-        lookback_days: Optional[int] = None,
-    ) -> pd.DataFrame:
-        """
-        Fetch OHLCV data for a Forex pair with full validation.
-
-        Returns a DataFrame with columns: Open, High, Low, Close, Volume
-        Index is a timezone-aware DatetimeIndex in UTC.
-        """
-        if lookback_days is None:
-            lookback_days = self.config.data_lookback_days
-
-        cache_key = f"{symbol}_{timeframe.value}"
-        now = time.time()
-
-        if (
-            cache_key in self._cache_timestamps
-            and (now - self._cache_timestamps[cache_key]) < self.config.refresh_interval
-            and cache_key in self._cache
-        ):
-            cached = self._cache.get(cache_key)
-            if cached is not None and not cached.empty:
-                return cached
-
-        df = self._fetch_twelve_data(symbol, timeframe, lookback_days)
-        df = self._validate_and_clean(df, symbol, timeframe)
-
-        self._cache[cache_key] = df
-        self._cache_timestamps[cache_key] = now
-        return df
-
-    def prefetch_batch(
-        self,
-        symbols: List[str],
-        timeframes: List[Timeframe],
-        lookback_days: Optional[int] = None,
-    ) -> None:
-        """
-        Batch-fetch all symbols for each timeframe in a single API call per TF.
-
-        Twelve Data accepts comma-separated symbols, so 7 pairs × 3 timeframes
-        = 3 requests instead of 21 — well within the free-tier 8 req/min limit.
-        Results are stored in the local cache; subsequent fetch_ohlcv calls
-        for the same symbol/TF will be served from cache.
-        """
-        if lookback_days is None:
-            lookback_days = self.config.data_lookback_days
-
-        for tf in timeframes:
-            try:
-                self._batch_fetch_timeframe(symbols, tf, lookback_days)
-                time.sleep(1.0)  # brief pause between TF requests
-            except Exception as e:
-                logger.warning(f"Batch prefetch failed for {tf.value}: {e}")
-
-    def fetch_multiple(
-        self,
-        symbols: List[str],
-        timeframe: Timeframe,
-        lookback_days: Optional[int] = None,
-    ) -> Dict[str, pd.DataFrame]:
-        """Fetch data for multiple symbols, returning dict of DataFrames."""
-        results = {}
-        for symbol in symbols:
-            try:
-                df = self.fetch_ohlcv(symbol, timeframe, lookback_days)
-                results[symbol] = df
-            except Exception as e:
-                logger.warning(f"Failed to fetch {symbol}: {e}")
-        return results
-
-    def get_latest_price(self, symbol: str) -> Optional[float]:
-        """Return the most recent close price for a symbol."""
-        try:
-            df = self.fetch_ohlcv(symbol, Timeframe.M5, lookback_days=1)
-            if df is not None and not df.empty:
-                return float(df["Close"].iloc[-1])
-        except Exception:
-            pass
-        return None
-
-    def get_spread_estimate(self, symbol: str) -> float:
-        """Estimate typical spread from recent M5 high-low ranges."""
-        try:
-            df = self.fetch_ohlcv(symbol, Timeframe.M5, lookback_days=1)
-            if df is not None and not df.empty:
-                return float((df["High"] - df["Low"]).median())
-        except Exception:
-            pass
-
-        fallback_spreads = {
-            "EURUSD=X": 0.00010, "GBPUSD=X": 0.00015,
-            "USDJPY=X": 0.010,   "USDCHF=X": 0.00015,
-            "AUDUSD=X": 0.00015, "USDCAD=X": 0.00018,
-            "NZDUSD=X": 0.00020,
-        }
-        return fallback_spreads.get(symbol, 0.00020)
-
-    def get_pair_info(self, symbol: str) -> Dict:
-        """Return display name and pip value for a pair."""
-        display = SYMBOL_DISPLAY.get(symbol, symbol)
-        is_jpy = "JPY" in symbol
-        return {
-            "symbol": symbol,
-            "display": display,
-            "pip_value": 0.01 if is_jpy else 0.0001,
-            "is_jpy_pair": is_jpy,
-        }
-
-    # ── Private Methods ────────────────────────────────────────────
-
-    def _throttle(self, credits: int = 1) -> None:
-        """
-        Block until `credits` API credits are available.
-
-        Twelve Data free tier: 8 credits/minute. Each symbol in any request
-        (single or batch) consumes 1 credit. This method enforces the limit
-        regardless of whether requests are batched or individual.
-        """
-        while True:
-            now = time.time()
-            # Evict credits older than the rate window
-            while self._request_times and now - self._request_times[0] >= self.RATE_WINDOW:
-                self._request_times.popleft()
-
-            available = self.RATE_LIMIT - len(self._request_times)
-            if available >= credits:
-                for _ in range(credits):
-                    self._request_times.append(time.time())
-                return
-
-            # Wait until the oldest credit expires
-            wait = (self._request_times[0] + self.RATE_WINDOW) - now + 0.2
-            logger.info(
-                f"Rate limit: {len(self._request_times)}/{self.RATE_LIMIT} credits used "
-                f"— waiting {wait:.1f}s for next window..."
-            )
-            time.sleep(wait)
-
-    def _batch_fetch_timeframe(
-        self, symbols: List[str], timeframe: Timeframe, lookback_days: int
-    ) -> None:
-        """
-        Single API call fetching all symbols for one timeframe.
-        Twelve Data returns a dict keyed by symbol when multiple are requested.
-        Parsed DataFrames are validated and stored in the cache.
-        """
-        if not self._api_key:
-            return
-
-        td_symbols = [self._to_td_symbol(s) for s in symbols]
-        interval = self.TD_INTERVALS[timeframe]
-        outputsize = min(
-            int(lookback_days * self.BARS_PER_DAY[timeframe]) + 20, 5000
-        )
-        outputsize = max(outputsize, 300)
-
-        # Each symbol in a batch request costs 1 credit
-        self._throttle(credits=len(symbols))
-
-        resp = self._session.get(
-            f"{self.TD_BASE_URL}/time_series",
-            params={
-                "symbol": ",".join(td_symbols),
-                "interval": interval,
-                "outputsize": outputsize,
-                "apikey": self._api_key,
-                "timezone": "UTC",
-                "format": "json",
-            },
-            timeout=30,
-        )
-        resp.raise_for_status()
-        data = resp.json()
-
-        # If the whole batch request returned a top-level error, raise it
-        if isinstance(data, dict) and data.get("status") == "error":
-            raise DataIntegrityError(
-                f"Batch API error ({timeframe.value}): {data.get('message', 'unknown')}"
-            )
-
-        # Batch response: {"EUR/USD": {...}, "GBP/USD": {...}, ...}
-        now = time.time()
-        for symbol, td_sym in zip(symbols, td_symbols):
-            try:
-                sym_data = data.get(td_sym, {})
-                if sym_data.get("status") == "error":
-                    logger.warning(
-                        f"Batch fetch error for {td_sym}: {sym_data.get('message')}"
-                    )
-                    continue
-
-                values = sym_data.get("values")
-                if not values:
-                    logger.warning(f"No values in batch response for {td_sym}")
-                    continue
-
-                records = [
-                    {
-                        "Datetime": pd.Timestamp(v["datetime"], tz="UTC"),
-                        "Open":   float(v["open"]),
-                        "High":   float(v["high"]),
-                        "Low":    float(v["low"]),
-                        "Close":  float(v["close"]),
-                        "Volume": float(v.get("volume") or 0),
-                    }
-                    for v in reversed(values)
-                ]
-                df = pd.DataFrame(records).set_index("Datetime")
-                df.index.name = "Datetime"
-                df = self._validate_and_clean(df, symbol, timeframe)
-
-                cache_key = f"{symbol}_{timeframe.value}"
-                self._cache[cache_key] = df
-                self._cache_timestamps[cache_key] = now
-                logger.debug(f"Batch cached {len(df)} bars for {td_sym} @ {interval}")
-
-            except Exception as e:
-                logger.warning(f"Failed to parse batch data for {td_sym}: {e}")
-
-    def _fetch_twelve_data(
-        self, symbol: str, timeframe: Timeframe, lookback_days: int
-    ) -> pd.DataFrame:
-        """
-        Fetch OHLCV from Twelve Data REST API.
-        Falls back to synthetic GBM data if API is unavailable or key is missing.
-        """
-        if not self._api_key:
-            logger.warning("No Twelve Data API key — using synthetic data. "
-                           "Set TWELVE_DATA_API_KEY env var or config.twelve_data_api_key.")
-            return self._generate_synthetic_data(symbol, timeframe, lookback_days)
-
-        td_symbol = self._to_td_symbol(symbol)
-        interval = self.TD_INTERVALS[timeframe]
-        outputsize = min(
-            int(lookback_days * self.BARS_PER_DAY[timeframe]) + 20,
-            5000,
-        )
-        outputsize = max(outputsize, 300)
-
-        self._throttle(credits=1)
-
-        try:
-            resp = self._session.get(
-                f"{self.TD_BASE_URL}/time_series",
-                params={
-                    "symbol": td_symbol,
-                    "interval": interval,
-                    "outputsize": outputsize,
-                    "apikey": self._api_key,
-                    "timezone": "UTC",
-                    "format": "json",
-                },
-                timeout=15,
-            )
-            resp.raise_for_status()
-            data = resp.json()
-
-            if data.get("status") == "error":
-                raise DataIntegrityError(
-                    f"Twelve Data API error for {td_symbol}: {data.get('message', 'unknown')}"
-                )
-
-            values = data.get("values")
-            if not values:
-                raise DataIntegrityError(f"Empty response from Twelve Data for {td_symbol}")
-
-            # Values arrive newest-first; reverse to chronological order
-            records = [
-                {
-                    "Datetime": pd.Timestamp(v["datetime"], tz="UTC"),
-                    "Open":   float(v["open"]),
-                    "High":   float(v["high"]),
-                    "Low":    float(v["low"]),
-                    "Close":  float(v["close"]),
-                    "Volume": float(v.get("volume") or 0),
-                }
-                for v in reversed(values)
-            ]
-
-            df = pd.DataFrame(records).set_index("Datetime")
-            df.index.name = "Datetime"
-            logger.debug(f"Fetched {len(df)} bars for {td_symbol} @ {interval}")
-            return df
-
-        except DataIntegrityError:
-            raise
-        except Exception as e:
-            logger.info(f"Twelve Data fetch failed for {symbol} ({e}), using synthetic data.")
-            return self._generate_synthetic_data(symbol, timeframe, lookback_days)
-
-    def _generate_synthetic_data(
-        self, symbol: str, timeframe: Timeframe, lookback_days: int
-    ) -> pd.DataFrame:
-        """
-        Generate realistic synthetic Forex OHLCV using GBM with mean-reversion.
-        Used as a fallback when live data is unavailable.
-        """
-        pair_params = {
-            "EURUSD=X": {"price": 1.1350, "vol": 0.07, "drift": -0.001},
-            "GBPUSD=X": {"price": 1.3200, "vol": 0.08, "drift":  0.002},
-            "USDJPY=X": {"price": 144.50, "vol": 0.09, "drift":  0.003},
-            "USDCHF=X": {"price": 0.8850, "vol": 0.07, "drift": -0.001},
-            "AUDUSD=X": {"price": 0.6550, "vol": 0.10, "drift": -0.002},
-            "USDCAD=X": {"price": 1.3650, "vol": 0.07, "drift":  0.001},
-            "NZDUSD=X": {"price": 0.6100, "vol": 0.10, "drift": -0.001},
-            "EURGBP=X": {"price": 0.8590, "vol": 0.06, "drift":  0.000},
-            "EURJPY=X": {"price": 163.90, "vol": 0.09, "drift":  0.002},
-            "GBPJPY=X": {"price": 190.70, "vol": 0.11, "drift":  0.003},
-            "AUDJPY=X": {"price": 94.60,  "vol": 0.11, "drift":  0.001},
-            "EURCHF=X": {"price": 0.9380, "vol": 0.05, "drift":  0.000},
-            "CADJPY=X": {"price": 105.80, "vol": 0.09, "drift":  0.001},
-        }
-
-        params = pair_params.get(symbol, {"price": 1.0000, "vol": 0.08, "drift": 0.000})
-        base_price = params["price"]
-        annual_vol = params["vol"]
-        annual_drift = params["drift"]
-
-        tf_minutes = {
-            Timeframe.M1: 1, Timeframe.M5: 5, Timeframe.M15: 15,
-            Timeframe.M30: 30, Timeframe.H1: 60, Timeframe.H4: 240,
-            Timeframe.D1: 1440, Timeframe.W1: 10080,
-        }
-        bar_minutes = tf_minutes.get(timeframe, 60)
-        bars_per_year = (365.25 * 24 * 60) / bar_minutes
-        dt = 1.0 / bars_per_year
-
-        n_bars = min(int(lookback_days * 24 * 60 / bar_minutes), 2000)
-        n_bars = max(n_bars, 300)
-
-        seed_base = int(hashlib.md5(symbol.encode()).hexdigest()[:8], 16)
-        time_component = int(time.time()) // 3600
-        np.random.seed((seed_base + time_component) % (2**31))
-
-        bar_vol = annual_vol * np.sqrt(dt)
-        bar_drift = annual_drift * dt
-
-        regime_len = np.random.randint(20, 80)
-        trend_strength = np.random.normal(0, 0.3)
-
-        price = base_price
-        prices = [price]
-
-        for i in range(1, n_bars):
-            if i % regime_len == 0:
-                regime_len = np.random.randint(20, 80)
-                trend_strength = np.random.normal(0, 0.3)
-
-            mr_force = -0.001 * (price - base_price) / base_price
-            ret = bar_drift + trend_strength * dt + mr_force + bar_vol * np.random.randn()
-
-            if np.random.random() < 0.02:
-                ret *= np.random.uniform(2.0, 4.0)
-
-            price *= (1 + ret)
-            prices.append(price)
-
-        prices = np.array(prices)
-        opens = np.roll(prices, 1)
-        opens[0] = prices[0]
-
-        typical_range = bar_vol * base_price * 1.5
-        highs = np.maximum(prices, opens) + np.abs(np.random.normal(0, typical_range * 0.6, n_bars))
-        lows  = np.minimum(prices, opens) - np.abs(np.random.normal(0, typical_range * 0.6, n_bars))
-        highs = np.maximum(highs, np.maximum(opens, prices))
-        lows  = np.minimum(lows,  np.minimum(opens, prices))
-
-        volume = np.random.lognormal(mean=10, sigma=0.5, size=n_bars).astype(int)
-
-        freq_map = {
-            Timeframe.M1: "1min",  Timeframe.M5: "5min",
-            Timeframe.M15: "15min", Timeframe.M30: "30min",
-            Timeframe.H1: "1h",    Timeframe.H4: "4h",
-            Timeframe.D1: "D",     Timeframe.W1: "W",
-        }
-        freq = freq_map.get(timeframe, "1h")
-        end_time = pd.Timestamp.now(tz="UTC").floor(f"{bar_minutes}min")
-        index = pd.date_range(end=end_time, periods=n_bars, freq=freq, tz="UTC")
-
-        df = pd.DataFrame({
-            "Open": opens, "High": highs, "Low": lows,
-            "Close": prices, "Volume": volume,
-        }, index=index)
-        df.index.name = "Datetime"
-
-        logger.info(
-            f"Generated {n_bars} synthetic {timeframe.value} bars for "
-            f"{SYMBOL_DISPLAY.get(symbol, symbol)} @ {prices[-1]:.5f}"
-        )
-        return df
-
-    def _validate_and_clean(
-        self, df: pd.DataFrame, symbol: str, timeframe: Timeframe
-    ) -> pd.DataFrame:
-        """
-        Validate and clean OHLCV data.
-
-        Steps: drop all-NaN rows → ffill small gaps → remove future timestamps
-        → fix OHLC inconsistencies → remove >10-sigma outliers → sort & deduplicate.
-        """
-        if df is None or df.empty:
-            raise DataIntegrityError(f"Empty dataset for {symbol}")
-
-        original_len = len(df)
-
-        df = df.dropna(subset=["Open", "High", "Low", "Close"], how="all")
-        df = df.ffill(limit=3)
-        df = df.dropna(subset=["Close"])
-
-        now = pd.Timestamp.utcnow()
-        future_mask = df.index > now + pd.Timedelta(minutes=5)
-        if future_mask.any():
-            logger.warning(f"{symbol}: Dropping {future_mask.sum()} future timestamps")
-            df = df[~future_mask]
-
-        invalid_ohlc = (
-            (df["High"] < df["Low"]) |
-            (df["High"] < df["Open"]) |
-            (df["High"] < df["Close"]) |
-            (df["Low"] > df["Open"]) |
-            (df["Low"] > df["Close"])
-        )
-        if invalid_ohlc.any():
-            logger.warning(f"{symbol}: Fixing {invalid_ohlc.sum()} OHLC-inconsistent bars")
-            for idx in df[invalid_ohlc].index:
-                vals = [df.loc[idx, "Open"], df.loc[idx, "Close"]]
-                df.loc[idx, "High"] = max(vals + [df.loc[idx, "High"]])
-                df.loc[idx, "Low"]  = min(vals + [df.loc[idx, "Low"]])
-
-        returns = df["Close"].pct_change()
-        std_ret = returns.std()
-        if std_ret > 0:
-            outlier_mask = (returns - returns.mean()).abs() > 10 * std_ret
-            if outlier_mask.any():
-                logger.warning(f"{symbol}: Removing {outlier_mask.sum()} extreme outlier bars")
-                df = df[~outlier_mask]
-
-        min_required = max(
-            self.config.technical.ema_trend,
-            self.config.technical.ichimoku_senkou_b,
-            200,
-        ) + 10
-
-        if len(df) < min_required:
-            logger.warning(
-                f"{symbol} {timeframe.value}: Only {len(df)} bars "
-                f"(need {min_required}). Some indicators will be partial."
-            )
-
-        df = df.sort_index()
-        df = df[~df.index.duplicated(keep="last")]
-
-        cleaned = original_len - len(df)
-        if cleaned > 0:
-            logger.info(f"{symbol}: Cleaned {cleaned}/{original_len} bars")
-
-        return df

deploy/README-dashboard.md

@@ -0,0 +1,162 @@
+# Deploying the web dashboard (live LAMP server)
+
+The dashboard is a **static publisher**: `dashboard_publisher.py` runs as its
+own daemon, scans XAU/USD at candle closes, and atomically writes
+`dashboard.html` + `news_summary.php` + `snapshot.json` + `news.json` into a
+folder Apache already serves. Apache does all web serving — no new port, no
+Python web server, and the telegram bot process is completely unaffected by
+anything that happens here (see the module docstring for the independence
+rules). The one dynamic piece is `news_summary.php` (§6): an on-demand
+DeepSeek summary for a clicked headline, whitelist-gated against news.json.
+
+All steps below run **on the live server** (`/home/Forex/ForexBot2`).
+
+## 1. Get the code
+
+```bash
+cd /home/Forex/ForexBot2
+git pull
+venv/bin/python test_all.py        # optional sanity: suite must be green
+```
+
+No new Python dependencies are required.
+
+## 2. Make the web folder writable
+
+The dashboard publishes straight into the portal docroot `/var/www/trade/`
+(owner decision 2026-08-03) — it adds exactly four files there:
+`dashboard.html`, `news_summary.php`, `snapshot.json`, `news.json` (plus the
+apache-owned `ai/` cache dir from §6). The daemon user must be able to write
+them:
+
+```bash
+# if the portal files can be owned by the daemon user:
+sudo chown Forex: /var/www/trade
+# otherwise grant write without changing ownership:
+sudo setfacl -m u:Forex:rwx /var/www/trade
+```
+
+(A different path is one edit away: the unit's `Environment=DASHBOARD_OUT_DIR=`
+line, or `--out` for one-off runs.)
+
+SELinux (RHEL/Alma/CentOS enforcing): directories under `/var/www` normally
+already carry `httpd_sys_content_t`, and new files inherit the parent label, so
+no action is usually needed. If Apache 403s despite correct permissions:
+
+```bash
+sudo restorecon -R /var/www/trade
+```
+
+## 3. Smoke-test once
+
+```bash
+venv/bin/python dashboard_publisher.py --once
+ls /var/www/trade    # dashboard.html snapshot.json news.json added next to the portal
+```
+
+Then open `http://<server>/dashboard.html` (path relative to however the
+vhost maps `/var/www/trade`) — all four timeframe cards, the calendar, and
+the newswire should populate.
+
+## 4. Install the daemon
+
+```bash
+sudo cp deploy/gold-dashboard.service /etc/systemd/system/
+# edit User= if the repo owner is not "Forex"
+sudo systemctl daemon-reload
+sudo systemctl enable --now gold-dashboard
+systemctl status gold-dashboard
+tail -f logs/dashboard.log
+```
+
+The daemon scans all four timeframes at boot, then wakes at every M5
+boundary (+10 s) and rescans whichever timeframes just closed. Logs rotate
+every 3 h (240 backups), same scheme as the bot.
+
+## 5. Optional Apache polish
+
+The page polls its JSON with conditional GETs (`cache: "no-cache"` → Apache
+answers unchanged files with 304), so no Apache config is strictly needed. To
+be explicit anyway:
+
+```apache
+<Directory "/var/www/trade">
+    <FilesMatch "\.json$">
+        Header set Cache-Control "no-cache"
+    </FilesMatch>
+</Directory>
+```
+
+The page is deliberately public (owner decision 2026-08-03). To restrict it
+later, add standard Apache basic auth (`htpasswd` + `Require valid-user`) on
+that directory — no code changes needed.
+
+## 6. AI summary endpoint (PHP) — one-time setup
+
+Clicking a newswire headline opens a detail dialog that fetches an on-demand
+DeepSeek gold-impact summary from `news_summary.php` (installed next to the
+page by the daemon). Owner decision 2026-08-07: a real on-demand endpoint on
+the public page, accepted with these guards baked into the PHP — the id must
+match a headline in the published news.json (arbitrary text can never reach
+the LLM), each headline's summary is cached forever in `ai/` (≤1 API call per
+headline, ever), and a file-based daily cap fails **closed** on I/O errors.
+Since 2026-09-04 the engine publishes only headlines up to 4 h old whose
+score leans one way (neutral items are dropped), so a headline leaves the
+whitelist sooner: a dialog left open past that point returns `unknown_id`
+("This headline has left the live feed"); summaries already cached in `ai/`
+stay reachable.
+
+Requirements: `php` + the curl extension (`php -m | grep curl`).
+
+```bash
+# 1. SELinux: allow httpd outbound HTTPS — WITHOUT this every summary call
+#    dies as a curl timeout even with perfect perms (RHEL-family default).
+sudo setsebool -P httpd_can_network_connect 1
+
+# 2. API key OUTSIDE the docroot, readable by Apache only:
+sudo install -d -o root -g root -m 755 /etc/gold-dashboard
+sudo sh -c 'umask 137; echo "sk-..." > /etc/gold-dashboard/deepseek.key'
+sudo chown root:apache /etc/gold-dashboard/deepseek.key   # mode 640
+
+# 3. apache-owned cache dir (the daemon runs as Forex and can't create it):
+sudo install -d -o apache -g apache -m 755 /var/www/trade/ai
+sudo semanage fcontext -a -t httpd_sys_rw_content_t '/var/www/trade/ai(/.*)?'
+sudo restorecon -Rv /var/www/trade/ai
+```
+
+Knobs (Apache `SetEnv` in the vhost, both optional):
+`AI_SUMMARY_DAILY_CAP` (default 150), `DEEPSEEK_KEY_FILE` (default
+`/etc/gold-dashboard/deepseek.key`).
+
+Verify from the shell — expect 400 / 404 / 200 / 200-from-cache:
+
+```bash
+curl -s "http://<server>/news_summary.php?id=zzz"                 # {"error":"bad_id"}
+curl -s "http://<server>/news_summary.php?id=$(printf 'a%.0s' {1..40})"  # unknown_id
+ID=$(python3 -c "import json;print(json.load(open('/var/www/trade/news.json'))['headlines'][0]['id'])")
+curl -s "http://<server>/news_summary.php?id=$ID"   # real summary (2-12 s)
+curl -s "http://<server>/news_summary.php?id=$ID"   # instant, from ai/<id>.json
+```
+
+## Operational notes
+
+- **Updating**: `git pull` + `systemctl restart gold-dashboard`. The daemon
+  re-copies `static/dashboard.html` and `static/news_summary.php` into the
+  web folder at startup.
+- **If the daemon dies**, systemd restarts it in 30 s; worst case the page
+  shows stale data with its own "updated N min ago" evidence. Apache and the
+  telegram bot are unaffected.
+- **DeepSeek**: the publisher process has its own daily request cap
+  (`deepseek.daily_request_cap`, default 500) separate from the bot's —
+  expect roughly up to ~430 extra calls/day worst case, V4-Flash pricing.
+- **Two independent DeepSeek meters** hit the same API key: the publisher's
+  in-process cap above, and the PHP endpoint's file cap (`ai/.ht_cap.json`,
+  default 150/day). They don't communicate — worst-case spend is their sum;
+  DeepSeek-side spend alerting is the true backstop. Kept separate on
+  purpose (independence rule: the PHP never imports/reads bot code or
+  config.json).
+- **Restart window**: for the first minutes after a daemon restart the new
+  page can meet a pre-upgrade news.json without headline ids — those rows
+  simply render non-clickable until the first news publish.
+- **Credentials**: uses the same env/.env/config.json resolution as the rest
+  of the platform (Oanda candles feed; news needs no key; DeepSeek optional).

deploy/gold-dashboard.service

@@ -0,0 +1,23 @@
+# systemd unit for the XAU/USD static dashboard publisher.
+# Install (as root, on the live server):
+#   cp deploy/gold-dashboard.service /etc/systemd/system/
+#   systemctl daemon-reload && systemctl enable --now gold-dashboard
+# See deploy/README-dashboard.md for the full checklist.
+
+[Unit]
+Description=XAU/USD dashboard publisher (writes static JSON/HTML for Apache)
+After=network-online.target
+Wants=network-online.target
+
+[Service]
+Type=simple
+# Adjust to the account that owns /home/Forex/ForexBot2:
+User=Forex
+WorkingDirectory=/home/Forex/ForexBot2
+Environment=DASHBOARD_OUT_DIR=/var/www/trade
+ExecStart=/home/Forex/ForexBot2/venv/bin/python dashboard_publisher.py
+Restart=always
+RestartSec=30
+
+[Install]
+WantedBy=multi-user.target

docs/architecture_diagram.py

@@ -0,0 +1,190 @@
+"""ForexBot2 architecture diagram → docs/architecture.png.
+
+Regenerate after architecture changes:  venv/bin/python docs/architecture_diagram.py
+(matplotlib only — no graphviz on the dev box)."""
+import os
+import matplotlib
+matplotlib.use("Agg")
+import matplotlib.pyplot as plt
+from matplotlib.patches import FancyBboxPatch, FancyArrowPatch
+
+W, H = 100.0, 100.0
+fig, ax = plt.subplots(figsize=(16, 11.5))
+ax.set_xlim(0, W); ax.set_ylim(0, H); ax.axis("off")
+
+INK = "#263238"; MUT = "#607D8B"; ARR = "#455A64"; HAUL = "#7B8FA1"
+
+def box(x, y, w, h, title, lines, fc, ec, title_fs=8.2, body_fs=6.6, lw=1.3):
+    ax.add_patch(FancyBboxPatch((x, y), w, h,
+                 boxstyle="round,pad=0.35,rounding_size=0.9",
+                 facecolor=fc, edgecolor=ec, linewidth=lw, zorder=2))
+    cy = y + h - 1.9
+    ax.text(x + w / 2, cy, title, ha="center", va="center", fontsize=title_fs,
+            fontweight="bold", color=INK, zorder=3)
+    cy -= 2.3
+    for ln in lines:
+        ax.text(x + w / 2, cy, ln, ha="center", va="center", fontsize=body_fs,
+                color=INK, zorder=3)
+        cy -= 1.85
+
+def band(x, y, w, h, label, color):
+    ax.add_patch(FancyBboxPatch((x, y), w, h,
+                 boxstyle="round,pad=0.3,rounding_size=1.2",
+                 facecolor="none", edgecolor=color, linewidth=1.0,
+                 linestyle=(0, (4, 3)), zorder=1))
+    ax.text(x + 0.9, y + h - 1.4, label, ha="left", va="center",
+            fontsize=7.2, fontweight="bold", color=color, zorder=3)
+
+def arrow(p1, p2, color=ARR, lw=1.5, style="-|>", conn="arc3,rad=0.0",
+          label=None, lx=0, ly=0, ls="solid", fs=6.4, rot=0):
+    ax.add_patch(FancyArrowPatch(p1, p2, arrowstyle=style, mutation_scale=11,
+                 color=color, linewidth=lw, linestyle=ls,
+                 connectionstyle=conn, zorder=4))
+    if label:
+        mx, my = (p1[0] + p2[0]) / 2 + lx, (p1[1] + p2[1]) / 2 + ly
+        ax.text(mx, my, label, fontsize=fs, color=MUT, ha="center",
+                va="center", zorder=5, rotation=rot,
+                bbox=dict(boxstyle="round,pad=0.15", fc="white", ec="none", alpha=0.9))
+
+# ── title ────────────────────────────────────────────────────────────
+ax.text(2, 98.4, "ForexBot2 — XAU/USD Signal-Intelligence Platform",
+        fontsize=15, fontweight="bold", color=INK, ha="left")
+ax.text(2, 96.0, "one pipeline, four delivery surfaces · closed bars only · live == backtest parity",
+        fontsize=8.5, color=MUT, ha="left")
+
+EXT_FC, EXT_EC = "#ECEFF1", "#90A4AE"
+DAT_FC, DAT_EC = "#E3F2FD", "#5C9BD1"
+NWS_FC, NWS_EC = "#E0F2F1", "#3D9C8F"
+COR_FC, COR_EC = "#FFFDF5", "#C9A227"
+REP_FC, REP_EC = "#FFE9A8", "#B8860B"
+DEL_FC, DEL_EC = "#E8F5E9", "#5FAF63"
+BTS_FC, BTS_EC = "#EDE7F6", "#8E6BC6"
+
+# ── external services (top band) ─────────────────────────────────────
+band(1, 85.0, 98, 9.4, "EXTERNAL SERVICES", "#78909C")
+box(2.5, 86.2, 15.5, 6.4, "Oanda v20 API",
+    ["candles (live feed)", "practice orders"], EXT_FC, EXT_EC)
+box(19.5, 86.2, 15.5, 6.4, "Twelve Data · Tiingo",
+    ["registered fallbacks", "(Tiingo: tick stream)"], EXT_FC, EXT_EC)
+box(36.5, 86.2, 15.5, 6.4, "Forex Factory",
+    ["economic calendar", "MyFXBook RSS fallback"], EXT_FC, EXT_EC)
+box(53.5, 86.2, 12.5, 6.4, "News RSS ×4",
+    ["gold / macro", "headlines"], EXT_FC, EXT_EC)
+box(67.5, 86.2, 13.5, 6.4, "DeepSeek API",
+    ["V4-Flash LLM", "thinking: disabled"], EXT_FC, EXT_EC)
+box(82.5, 86.2, 15, 6.4, "Telegram API",
+    ["owner bot +", "subscriber channel"], EXT_FC, EXT_EC)
+
+# ── data & context layer ─────────────────────────────────────────────
+box(2.5, 69.8, 33, 11.0, "providers/  (BaseDataProvider)",
+    ["single-flight fetch · TTL+boundary cache",
+     "forming-candle DROP · validate/clean",
+     "paginated fetch_range · optional tick stream",
+     "active feed knob: data_providers.active = oanda"],
+    DAT_FC, DAT_EC)
+box(38.5, 69.8, 33, 11.0, "news_engine.py  +  llm_client.py",
+    ["calendar events + keyword sentiment (gold-inverse)",
+     "DeepSeek enrichment: batch headline scores,",
+     "event gold-direction, market blurb — display-only,",
+     "keyword fallback, daily request cap"],
+    NWS_FC, NWS_EC)
+box(74.5, 69.8, 23, 11.0, "app_config.py",
+    ["credentials & deployment:",
+     "env → config.json → refuse",
+     "(no hardcoded fallbacks)",
+     ".env loaded at import"],
+    "#FAFAFA", "#9E9E9E")
+
+# ── signal core container ────────────────────────────────────────────
+ax.add_patch(FancyBboxPatch((2.5, 40.5), 95, 26.5,
+             boxstyle="round,pad=0.35,rounding_size=1.2",
+             facecolor=COR_FC, edgecolor=COR_EC, linewidth=1.6, zorder=1))
+ax.text(50, 65.2, "SIGNAL CORE — runs identically in live scans and backtests (closed bars only)",
+        ha="center", fontsize=8.4, fontweight="bold", color="#8a6d1a", zorder=3)
+
+box(4.5, 51.8, 28, 12.0, "engine.py — TechnicalEngine",
+    ["9 scored indicators + ATR context:",
+     "EMA · RSI · MACD · BB · ADX · ROC ·",
+     "Keltner · divergence · candlestick",
+     "regime → adaptive weights → composite"],
+    "white", COR_EC)
+box(4.5, 42.0, 28, 8.4, "pair_signals/xauusd.py — factory",
+    ["ALWAYS used: config.json overlay +",
+     "validated per-TF deltas (hour blocks, trail…)"],
+    "white", COR_EC)
+box(35.5, 42.0, 60, 20.7, "signal_aggregator.py — analyze_pair → analyze_from_dataframes",
+    ["multi-TF agreement  (M5→[M15,H1] · M15→[M5,H1] · H1→[M15,H4] · H4→[H1,D1])",
+     "→ news context (warn-only by default) → diminishing-decay penalties",
+     "→ exhaustion hard block → quality grade A+…F",
+     "→ ordered post-grade gate chain (session · XAU SELL vol/hour · regime · M5 noise)",
+     "→ setup generators: XAU ATR (A+/A) · generic swing · XAU_PB · XAU_CP",
+     "→ entry · SL · TP1/TP2 · trailing plan"],
+    "white", COR_EC, body_fs=6.9)
+
+# ── SignalReport hub ─────────────────────────────────────────────────
+box(31, 33.2, 38, 5.2, "SignalReport",
+    ["signal (−2…+2) · grade · trade setup · reasons · news summary"],
+    REP_FC, REP_EC, title_fs=8.8)
+
+# ── delivery surfaces ────────────────────────────────────────────────
+band(1, 13.2, 98, 17.2, "DELIVERY SURFACES", "#4C8F50")
+box(2.5, 15.0, 12, 12.6, "main.py",
+    ["CLI report", "--detail", "--json"], DEL_FC, DEL_EC)
+box(42.5, 15.0, 31, 12.6, "dashboard_publisher.py  →  Apache",
+    ["independent daemon · scans at candle closes",
+     "atomic publish to /var/www/trade:",
+     "dashboard.html · snapshot.json · news.json",
+     "+ news_summary.php — on-demand AI",
+     "gold-impact summary (whitelist · cache · cap)"],
+    DEL_FC, DEL_EC)
+box(76, 15.0, 21.5, 12.6, "telegram_bot.py",
+    ["candle-boundary auto-scans", "quality threshold + dedup",
+     "alerts + LLM context line", "WS level-breach monitor"],
+    DEL_FC, DEL_EC)
+
+# ── backtest / QA ────────────────────────────────────────────────────
+box(2.5, 3.2, 71.8, 8.0, "backtest/gpu_backtest.py — bar-by-bar replay through the SAME analyze_from_dataframes",
+    ["realism (results-critical): next-open entries · raw-path resolution · spread charged · SL-first ties · close-only trail",
+     "CuPy GPU / NumPy fallback · parquet cache · fed by providers.fetch_range (news off)"],
+    BTS_FC, BTS_EC)
+box(77.5, 3.2, 20, 8.0, "test_all.py",
+    ["359 tests · 22 modules", "no network (mocked)"], "#FAFAFA", "#9E9E9E")
+
+# ── arrows: short structural hops (solid dark) ───────────────────────
+arrow((10, 86.2), (14, 81.2), label="OHLCV", lx=-3.4, ly=0.2)          # oanda → providers
+arrow((27, 86.2), (24, 81.2), ls=(0, (3, 2)), lw=1.1)                  # fallbacks → providers
+arrow((44, 86.2), (48, 81.2))                                          # FF → news
+arrow((59.5, 86.2), (56, 81.2))                                        # RSS → news
+arrow((72, 86.2), (62, 81.2), label="LLM", lx=3.4, ly=0.6)             # deepseek → news
+arrow((19, 69.8), (19, 64.2), label="DataFrames (UTC, closed bars)", lx=0.2, ly=0.9)
+arrow((33.5, 57.5), (35.5, 55.5))                                      # engine → aggregator
+arrow((32.5, 46.0), (35.5, 48.0))                                      # factory → aggregator
+arrow((55, 69.8), (55, 63.1), label="warn-only context", lx=0, ly=1.0)
+arrow((86, 69.8), (86, 67.3), ls=(0, (3, 2)), lw=1.1)                  # config → core
+arrow((50, 40.5), (50, 38.8))                                          # core → report
+arrow((36, 34.5), (8.5, 27.95), conn="arc3,rad=0.12")                  # report → cli
+arrow((60, 33.2), (58, 27.95))                                         # report → dashboard
+arrow((66, 34.5), (86.5, 27.95), conn="arc3,rad=-0.12")                # report → telegram
+
+# ── long-haul edges (thin dashed, routed through gutters) ────────────
+# telegram bot ↔ Telegram API — right-margin arc, clear of app_config
+arrow((95, 86.2), (95, 27.95), conn="arc3,rad=0.15", color=HAUL,
+      lw=1.2, ls=(0, (4, 2)))
+ax.text(99.2, 57, "alerts / commands", fontsize=6.4, color=MUT,
+        rotation=90, ha="center", va="center")
+# browser click → news_summary.php → DeepSeek
+arrow((66, 27.95), (73, 86.2), conn="arc3,rad=-0.18", color=HAUL,
+      lw=1.2, ls=(0, (4, 2)),
+      label="on-demand AI summary\n(browser click → PHP)", lx=2.5, ly=11.6)
+# providers.fetch_range → backtester — far-left margin channel
+arrow((1.5, 69.8), (1.5, 11.0), color=BTS_EC, lw=1.1, ls=(0, (3, 2)),
+      label="fetch_range\n(history)", lx=3.6, ly=-1.2)
+# backtester replays the signal core — right gutter between boxes
+arrow((74.0, 11.6), (74.9, 40.3), color=HAUL, lw=1.3)
+ax.text(81.8, 38.9, "replays the exact signal code", fontsize=6.4,
+        color=MUT, ha="center", va="center", zorder=5,
+        bbox=dict(boxstyle="round,pad=0.15", fc="white", ec="none", alpha=0.9))
+
+out = os.path.join(os.path.dirname(os.path.abspath(__file__)), "architecture.png")
+fig.savefig(out, dpi=200, bbox_inches="tight", facecolor="white")
+print(f"saved {out}")

docs/beginner_guide.html

@@ -0,0 +1,168 @@
+<!DOCTYPE html>
+<html lang="en">
+<head>
+<meta charset="utf-8">
+<title>ForexBot2 — Beginner's Guide</title>
+<style>
+  @page {
+    size: A4;
+    margin: 22mm 18mm;
+    @bottom-center { content: "ForexBot2 Beginner's Guide — page " counter(page) " of " counter(pages); font-size: 8pt; color: #888; }
+  }
+  * { box-sizing: border-box; }
+  body { font-family: "DejaVu Sans", Helvetica, Arial, sans-serif; font-size: 10.5pt; line-height: 1.65; color: #1c2733; margin: 0; }
+  h1 { font-size: 21pt; margin: 0 0 2mm; color: #0d1b2a; }
+  .subtitle { color: #5a6b7c; font-size: 10.5pt; margin-bottom: 8mm; }
+  h2 { font-size: 13.5pt; color: #0d1b2a; border-bottom: 2px solid #d4a017; padding-bottom: 1.2mm; margin: 8mm 0 3mm; page-break-after: avoid; }
+  h3 { font-size: 11pt; margin: 5mm 0 1.5mm; color: #23374d; page-break-after: avoid; }
+  p { margin: 0 0 2.8mm; }
+  table { border-collapse: collapse; width: 100%; margin: 3mm 0 4mm; font-size: 9.3pt; page-break-inside: avoid; }
+  th { background: #0d1b2a; color: #fff; text-align: left; padding: 1.8mm 2.5mm; font-weight: 600; }
+  td { border-bottom: 1px solid #dde4ea; padding: 1.8mm 2.5mm; vertical-align: top; }
+  tr:nth-child(even) td { background: #f4f7fa; }
+  table.breakable { page-break-inside: auto; }
+  table.breakable tr { page-break-inside: avoid; }
+  .signal-box { background: #f4f7fa; border: 1px solid #c9d4de; border-left: 4px solid #d4a017; border-radius: 2mm; padding: 3.5mm 4.5mm; font-family: "DejaVu Sans Mono", monospace; font-size: 8.6pt; line-height: 1.55; white-space: pre-wrap; margin: 3mm 0 4mm; page-break-inside: avoid; }
+  .callout { background: #fdf6e3; border: 1px solid #e8d49a; border-left: 4px solid #d4a017; border-radius: 2mm; padding: 3mm 4.5mm; margin: 3mm 0 4mm; page-break-inside: avoid; }
+  .callout.red { background: #fdf0ee; border-color: #eab4ac; border-left-color: #c0392b; }
+  .callout.green { background: #eef7ee; border-color: #b3d9b3; border-left-color: #2e7d32; }
+  .callout b:first-child { display: block; margin-bottom: 1mm; }
+  .example { background: #eef2f6; border-radius: 2mm; padding: 3mm 4.5mm; margin: 2mm 0 4mm; page-break-inside: avoid; }
+  .example b:first-child { color: #23374d; }
+  ul, ol { margin: 0 0 3mm; padding-left: 6mm; }
+  li { margin-bottom: 1.6mm; }
+  code { font-family: "DejaVu Sans Mono", monospace; font-size: 8.8pt; background: #eef2f6; padding: 0.3mm 1.2mm; border-radius: 1mm; }
+  .small { font-size: 8.5pt; color: #5a6b7c; }
+  .disclaimer { border: 1.5px solid #c0392b; border-radius: 2mm; padding: 3.5mm 4.5mm; margin-top: 6mm; font-size: 9pt; page-break-inside: avoid; }
+  .disclaimer b { color: #c0392b; }
+</style>
+</head>
+<body>
+
+<h1>ForexBot2 — Beginner's Guide</h1>
+<div class="subtitle">Everything explained in plain words. No experience needed. &nbsp;·&nbsp; July 2026</div>
+
+<h2>1 · What is this bot?</h2>
+<p>ForexBot2 is a computer program that watches the <b>price of gold</b> all day and all night. It studies the price charts using nine different mathematical checks. When the checks agree that the price is likely to move in one direction, it sends you a message on <b>Telegram</b> telling you:</p>
+<ul>
+<li>whether to <b>buy</b> or <b>sell</b>,</li>
+<li>at what price to get in,</li>
+<li>at what price to get out if the trade goes wrong (so your loss stays small),</li>
+<li>and how to lock in profit if the trade goes well.</li>
+</ul>
+<p>Most days the bot stays quiet for hours. <b>That is normal.</b> Quiet means "nothing worth trading right now." Never trade just because you are bored.</p>
+
+<h2>2 · Ten words you need to know</h2>
+<p>Every trading word used in this guide (and in the bot's messages) is explained here. Come back to this table whenever you see a word you don't remember.</p>
+<table class="breakable">
+<tr><th style="width:26%">Word</th><th>Plain meaning</th></tr>
+<tr><td><b>BUY (going "long")</b></td><td>You make money if the gold price goes <b>up</b>.</td></tr>
+<tr><td><b>SELL (going "short")</b></td><td>You make money if the gold price goes <b>down</b>. Yes, that's possible — your broker lets you sell gold you don't own and buy it back cheaper later.</td></tr>
+<tr><td><b>Entry</b></td><td>The price at which you open the trade.</td></tr>
+<tr><td><b>Stop loss (SL)</b></td><td>A safety brake. You tell your broker: "if the price reaches this level, close my trade automatically." It puts a hard limit on how much one trade can lose. <b>Every trade must have one.</b></td></tr>
+<tr><td><b>Take profit (TP)</b></td><td>The opposite of a stop loss: a level where the trade closes automatically <b>with a profit</b>. In this system it is only a rough guide — see section 6.</td></tr>
+<tr><td><b>Risk, or "R"</b></td><td>The amount of money you have chosen to lose if the trade fails. If you decide to risk $20, then <b>1R = $20</b>. A trade that wins "+2R" made you $40. This is how traders measure results.</td></tr>
+<tr><td><b>Trailing stop ("trail")</b></td><td>A stop loss that <b>follows the price</b> as it moves in your favor, but never moves backwards. Explained with numbers in section 6.</td></tr>
+<tr><td><b>Timeframe (TF)</b></td><td>The size of each candle on the chart. <b>M5</b> = each candle is 5 minutes, <b>M15</b> = 15 minutes, <b>H1</b> = 1 hour, <b>H4</b> = 4 hours. Bigger timeframe = slower, calmer trading.</td></tr>
+<tr><td><b>XAU/USD</b></td><td>Just the official symbol for "gold priced in US dollars."</td></tr>
+<tr><td><b>Spread</b></td><td>The tiny difference between the buying price and the selling price. It's your broker's fee, paid on every trade.</td></tr>
+</table>
+
+<h2>3 · Reading a bot message</h2>
+<p>Here is a typical signal, with plain-English notes below it:</p>
+<div class="signal-box">🔍 XAU/USD Signal Analysis  (H1)          ← gold, on the 1-hour chart
+🟢 BUY   Grade: A  |  64% confidence      ← what to do + how good the setup looks
+✅ MTF aligned                            ← bigger and smaller charts agree
+📋 Indicators …                           ← the bot's 9 checks (you can skip this part)
+━━━━━━━━━━━━━━━━━━━━
+🚀 TRADE SETUP                            ← ↓ THIS is the part you act on ↓
+📈 Direction: BUY
+▶ Entry:  4,132.50                        ← open the trade near this price
+🛑 SL:     4,088.20                        ← your safety brake (stop loss)
+🎯 TP1:    4,176.80                        ← rough profit target (a guide, not a rule)
+🔄 Trail:  once price hits 4,159.08,
+   trail SL $33.23 behind best price      ← how to lock in profit (see section 6)</div>
+<ul>
+<li><b>Grade (A+, A, B, C):</b> the bot's own mark for how clean the setup is. A+ is best. It never sends anything worse than C.</li>
+<li><b>Confidence and the indicator list:</b> extra detail for curious people. You don't need to understand it to trade the signal.</li>
+<li><b>⚠️ Warning lines:</b> things the bot noticed, like high volatility. They are "be aware," not "don't trade."</li>
+<li><b>📰 News line:</b> a short plain-English note on what is moving the gold price right now and why, written from the last few hours of headlines. It is background information, not a reason to trade.</li>
+<li><b>The trade setup at the bottom is the only part you must act on.</b></li>
+</ul>
+
+<h2>4 · How much money to put in (the most important section)</h2>
+<p>You never put your whole account into one trade. You decide how much you are willing to <b>lose</b>, and work backwards from there. The rule:</p>
+<div class="callout"><b>Risk 1% of your account per trade. Never more.</b> If you have $2,000, that means one trade may lose at most $20. Losing trades are a normal part of the system — 1% means a bad week stings but never hurts you badly.</div>
+<div class="example"><b>Worked example.</b><br>
+Your account: <b>$2,000</b>. Your risk: 1% = <b>$20</b>. So for this trade, 1R = $20.<br>
+The signal says: Entry 4,132.50, Stop loss 4,088.20. The gap between them is <b>$44.30 per ounce</b> of gold.<br>
+How much gold to buy: $20 ÷ $44.30 = <b>0.45 ounces</b>.<br>
+Now, whatever happens: if the price falls to the stop loss, you lose about $20 — and no more. If the trade wins 2R, you make about $40.</div>
+<p class="small">Most broker apps have a built-in calculator: enter the stop-loss price and the amount you want to risk ($20), and it works out the position size for you. Use it — it avoids mistakes. (On many platforms gold is traded in "lots," where 0.01 lot = 1 ounce.)</p>
+
+<h2>5 · What to do when a signal arrives — step by step</h2>
+<ol>
+<li><b>Check the timeframe</b> in the first line. Only act on the one timeframe you have chosen to follow (see section 7). Ignore the rest.</li>
+<li><b>Work out your position size</b> using the 1% rule above. Do this before anything else.</li>
+<li><b>Open the trade</b> in your broker app, in the direction shown (BUY or SELL), at the current market price. Try to act soon after the message; if the price has already run far past the entry price, skip the trade — another signal will come.</li>
+<li><b>Set the stop loss immediately</b>, at the exact SL price in the message. Not "in your head" — in the app. Gold can move fast, and the stop loss is what protects you while you sleep.</li>
+<li><b>Then manage the trade with the trailing stop</b>, explained next.</li>
+</ol>
+
+<h2>6 · The trailing stop — how winners get paid</h2>
+<p>The take-profit price (TP1) in the message is only a guide. The system makes its money by <b>letting winning trades run</b> and following them with a moving safety brake. Every signal message tells you exactly when to start and how far behind to follow. Here is the idea with numbers:</p>
+<div class="example"><b>Example.</b> You bought gold at <b>4,132</b>, stop loss at <b>4,088</b> (risking $44 per ounce).<br>
+The message says: <i>"once price hits 4,159, trail SL $33 behind best price."</i><br><br>
+<b>Step 1 — wait.</b> Until gold reaches 4,159, do nothing. If it falls to 4,088 first, the stop loss closes the trade. That loss is normal and already budgeted.<br>
+<b>Step 2 — the price reaches 4,159.</b> Now move your stop loss up to 4,159 − 33 = <b>4,126</b>. From here on, the worst case is roughly break-even.<br>
+<b>Step 3 — keep following.</b> Each time the price makes a new high, move the stop up so it stays $33 below that high. Price climbs to 4,180? Stop goes to 4,147 — profit is now locked in. Price climbs to 4,210? Stop goes to 4,177.<br>
+<b>Step 4 — the price turns.</b> Eventually gold falls back $33 from its best point and hits your stop. The trade closes by itself, and you keep everything above your entry. <b>You never decide when to take profit — the trail decides.</b></div>
+<ul>
+<li>Only move the stop <b>up</b> (for a BUY). Never move it back down. For a SELL, everything is mirrored: the stop only moves down.</li>
+<li>Move it when a candle finishes, not every second. Checking once per candle (every hour, on the 1-hour timeframe) is enough.</li>
+<li>Many broker apps can do this automatically — look for a "trailing stop" order and set the distance from the message.</li>
+</ul>
+
+<h2>7 · Which timeframe should you follow?</h2>
+<p>The bot sends signals from several chart speeds. As a beginner, <b>follow only the 1-hour (H1) signals</b> and ignore the others. Here is why:</p>
+<table>
+<tr><th></th><th>M5 (5-minute)</th><th>M15 (15-minute)</th><th>H1 (1-hour) ★ start here</th></tr>
+<tr><td>Signals per week</td><td>about 10</td><td>about 10</td><td>about 3</td></tr>
+<tr><td>A trade usually lasts</td><td>a few hours</td><td>about half a day</td><td>half a day to a few days</td></tr>
+<tr><td>Time you have to react</td><td>1–2 minutes</td><td>up to 15 minutes</td><td>up to an hour</td></tr>
+<tr><td>Suits</td><td>full-time / automated</td><td>experienced, quick</td><td>anyone with a job and a life</td></tr>
+</table>
+<ul>
+<li><b>H1 is forgiving.</b> You have up to an hour to act on a message, and small delays barely matter. On the 5-minute signals, being two minutes late already costs real money — they are not for manual beginners.</li>
+<li><b>H1 trades are held longer</b> — often overnight, sometimes over days. That is normal. Your stop loss is working for you while you sleep.</li>
+<li><b>H1 is BUY-only.</b> All H1 SELL signals are blocked (since 2026-09-03) because they were a losing side on this timeframe — expect to see BUY signals here, never SELL.</li>
+<li><b>M5 also holds back some SELL signals</b> — specifically while the market is in a choppy "transitioning" phase, since those shorts tested poorly.</li>
+<li><b>Never take the same signal on two timeframes.</b> They are usually the same gold move seen at different zoom levels. Taking both doubles your risk, not your profit.</li>
+</ul>
+
+<h2>8 · What to expect — read this before your first trade</h2>
+<ul>
+<li><b>You will lose about 4 trades out of every 10.</b> That is how the system is designed. Losses are small (1R each) and wins are bigger on average. The profit comes from many trades added together, not from any single one.</li>
+<li><b>Losing streaks happen.</b> Five or six losses in a row will happen at some point every year. It does not mean the system is broken. This is exactly why each loss is capped at 1% of your account.</li>
+<li><b>Expect a rough patch.</b> At some point in a normal year, your account will likely be around 10% below its best point before recovering. If a 10% dip would make you panic or bet bigger to "win it back," trade smaller (0.5% risk) or don't trade.</li>
+<li><b>Judge the system over months, not days.</b> On the 1-hour timeframe there are only ~3 trades a week. One bad week means nothing; look at results per quarter.</li>
+<li><b>Real results will be a little worse than tests.</b> The bot's numbers come from careful historical testing, but tests can't include your reaction speed or your broker's exact prices. Expect a bit less, not a bit more.</li>
+</ul>
+
+<h2>9 · The five rules</h2>
+<div class="callout red">
+<b>Break these and no signal system can save you.</b>
+<ol style="margin-bottom:0">
+<li><b>Risk 1% per trade, always.</b> Never increase the size to win back a loss.</li>
+<li><b>Every trade gets a stop loss in the app, at entry time.</b> No exceptions, no "mental" stops.</li>
+<li><b>One timeframe, one open trade at a time.</b></li>
+<li><b>Missed the entry? Let it go.</b> Chasing a price that already moved turns a good trade into a gamble.</li>
+<li><b>Only trade money you can afford to lose completely.</b> Never rent, savings you need, or borrowed money.</li>
+</ol>
+</div>
+<div class="callout green"><b>A good first month:</b> follow the H1 signals with the smallest position size your broker allows (or a demo account), write down every trade, and practice the trailing stop until it feels routine. Only then trade at 1% risk.</div>
+
+<div class="disclaimer"><b>⚠️ Risk warning.</b> This bot and guide are for education and research only. They are <b>not financial advice</b>. Trading gold and currencies involves a high risk of losing money, and most beginners lose. Past results — including tested results — do not guarantee future profits. Never trade money you cannot afford to lose. If you are unsure, speak to a licensed financial adviser.</div>
+
+</body>
+</html>

docs/superpowers/specs/2026-07-21-centralized-config-design.md

@@ -0,0 +1,154 @@
+# Centralized Configuration (config.json) — Design
+
+**Date:** 2026-07-21
+**Status:** Approved (user confirmed all four scoping decisions)
+
+## Goal
+
+Move deployment settings — Telegram credentials, data-provider API keys and
+selection, scanned pairs, scan timeframes, per-pair base blocked hours, alert
+quality boot default, log level — into a single gitignored `config.json` at the
+repo root, loaded once at startup.
+
+**Hard constraint:** zero change to signal logic, indicator weights, thresholds,
+or gates. With an unedited `config.json` (or none at all), every produced
+`PlatformConfig` must be byte-identical to today's factory output. This is
+test-enforced (see Testing).
+
+## Scoping decisions (user-confirmed 2026-07-21)
+
+1. **Secrets live in config.json** (gitignored). A committed
+   `config.json.example` carries placeholders. Env vars still override.
+   The hardcoded credential fallbacks in `config.py` and `telegram_bot.py`
+   are **deleted** — on a machine with neither config.json nor env vars, the
+   bot refuses to start with an explicit error instead of silently using
+   committed credentials. (Recommend rotating the Telegram token and Twelve
+   Data key after merge; both are exposed in git history.)
+2. **Multi-provider is structure only**: a `data_providers` section with named
+   provider blocks and an `active` selector. Only Twelve Data is implemented.
+   No failover logic (nothing to fail over to yet).
+3. **Blocked hours: base list per pair only.** The validated per-timeframe
+   deltas (M5 appends hour 9; M15 SELL-block 15/17 UTC) stay in
+   `pair_signals/xauusd.py` next to their validation evidence. The overlay is
+   applied **before** those deltas and updates **both** mirrored copies
+   (`xau_strategy.blocked_hours_utc` and `backtest.blocked_hours_utc`) so they
+   cannot drift.
+4. **Extra keys included:** scan timeframes, alert min quality (boot default),
+   log level. Everything else (indicator params, RR geometry, M5 gates,
+   CP knobs, news config) deliberately stays code-only.
+
+## File schema
+
+`config.json` (gitignored; `config.json.example` committed with placeholders
+for secrets and today's tuned values for everything else):
+
+```json
+{
+  "telegram": {
+    "token": "<bot token>",
+    "chat_id": "<owner chat id>",
+    "channel_id": "<-100… or empty string to disable the channel feed>",
+    "alert_min_quality": "C"
+  },
+  "data_providers": {
+    "active": "twelve_data",
+    "twelve_data": { "api_key": "<key>" }
+  },
+  "scanning": {
+    "pairs": ["XAU/USD"],
+    "timeframes": ["M5", "M15", "H1", "H4"]
+  },
+  "pairs": {
+    "XAU/USD": { "blocked_hours_utc": [0, 1, 2, 10, 11, 13, 16, 21] }
+  },
+  "log_level": "INFO"
+}
+```
+
+Timeframe strings use the `Timeframe` enum names (`M5`, `M15`, `H1`, `H4`, …).
+Pair keys use canonical symbols (`XAU/USD`).
+
+## Architecture
+
+### New module: `app_config.py`
+
+Single responsibility: read + validate `config.json`, expose typed accessors,
+apply the pair-level overlay. No signal knowledge.
+
+- `AppConfig` dataclass: `telegram` (token, chat_id, channel_id,
+  alert_min_quality), `data_providers` (active name + per-provider dicts),
+  `scan_pairs`, `scan_timeframes`, `pair_overrides`
+  (symbol → {blocked_hours_utc}), `log_level`.
+- `load_app_config(path=REPO_ROOT/"config.json") -> AppConfig` — module-level
+  cached (loaded once per process). Missing file → `AppConfig` of pure
+  defaults + one warning log. Malformed JSON, unknown top-level/section keys,
+  unknown pair symbols, hours outside 0–23, unknown timeframe names, unknown
+  `active` provider → raise `AppConfigError` with a named message (fail fast
+  at boot).
+- Precedence, applied inside the accessors:
+  **env var → config.json → code default**. Secrets (`TELEGRAM_TOKEN`,
+  `TELEGRAM_CHAT_ID`, `TELEGRAM_CHANNEL_ID`, `TWELVE_DATA_API_KEY`) have **no
+  code default** after this change.
+- `apply_pair_overrides(cfg: PlatformConfig, app: AppConfig) -> PlatformConfig`
+  — sets the pair's base `blocked_hours_utc` on **both**
+  `cfg.xau_strategy.blocked_hours_utc` and `cfg.backtest.blocked_hours_utc`,
+  plus `cfg.scan_pairs` / `cfg.scan_timeframes`. Called by the pair factory
+  (see below). A no-op when the pair has no override block.
+
+### Wiring points (all consumers)
+
+| Consumer | Change |
+|---|---|
+| `pair_signals/xauusd.py` | `make_config` calls `apply_pair_overrides` **before** the per-TF deltas (so M5 still appends hour 9 to whatever base the JSON sets, and mirrors stay in sync). |
+| `data_provider.py` | API key resolution becomes `app_config` accessor for the **active** provider (env `TWELVE_DATA_API_KEY` still wins). `config.twelve_data_api_key` field and its hardcoded default are deleted. Missing key keeps today's behavior: provider constructed with `_td=None`, errors on first fetch. |
+| `telegram_bot.py` | `TOKEN`, `CHAT_ID`, `CHANNEL_ID` come from `app_config` (hardcoded fallbacks deleted; missing token/chat_id → explicit startup error). `_alert_min_quality` boot default and `logging` level come from `app_config`. `SCAN_PAIRS` (today built from a bare `PlatformConfig()`) reads `app_config.scan_pairs` directly; `apply_pair_overrides` also stamps `cfg.scan_pairs`/`cfg.scan_timeframes` so the dataclass field stays truthful for any other reader. |
+| `main.py` | No structural change — it uses the factory, which now applies the overlay. |
+| `backtest/gpu_backtest.py` | No structural change — it uses the factory too, so an edited base blocked-hours list flows into replays identically to live (parity by construction). Data fetches pick up the provider key via `data_provider.py`. |
+| `config.py` | `twelve_data_api_key` field removed. Everything else untouched. |
+
+### Multi-provider structure
+
+`data_provider.py` gains a minimal registry:
+`PROVIDER_REGISTRY = {"twelve_data": ForexDataProvider}`. `active` selects
+from it; an unknown name fails at load. Adding a provider later = one config
+block + one adapter class registered here. No shared base class extracted now
+(YAGNI — with one provider there is nothing to abstract over yet).
+
+## Error handling summary
+
+| Condition | Behavior |
+|---|---|
+| No config.json | Defaults + warning. Secrets must come from env or startup fails (bot) / fetch fails (provider). |
+| Malformed JSON / bad values / unknown keys | `AppConfigError` at load, process exits with named message. |
+| Missing Telegram token or chat_id at bot boot | Explicit startup error naming the missing key and both places it can be set. |
+| Missing data-provider key | Same as today: provider builds, first fetch errors. |
+
+## Migration
+
+- Create the local `config.json` populated with the **current live values**
+  (the ones being deleted from source) so the running bot keeps working
+  unchanged. `.gitignore` gains `config.json`.
+- Commit `config.json.example` with placeholders.
+- CLAUDE.md: update the env-var table, repository map, and config section.
+- Post-merge recommendation (user action): rotate the Telegram token and
+  Twelve Data key, since both are in git history.
+
+## Testing (`test_all.py`, new module)
+
+1. **Parity pin (the critical one):** a config.json containing the example
+   defaults, and no config.json at all, both produce factory output identical
+   to today's (full dataclass equality across all four TFs).
+2. Overlay mapping: JSON blocked hours land on **both** mirrored fields; M5
+   factory still appends hour 9 on top of a custom base.
+3. Precedence: env var beats config.json beats default, per key.
+4. Validation: malformed JSON, bad hour, unknown pair/timeframe/provider each
+   raise `AppConfigError`.
+5. Missing-file behavior: defaults + no crash.
+6. Existing factory-equivalence test stays green (equivalence is now defined
+   over factory-with-overlay vs inline-config-with-overlay).
+
+## Out of scope
+
+Hot reload (config is read once at boot), a second concrete provider,
+failover, moving any tuned strategy value beyond the base blocked-hours list,
+news/indicator/geometry settings in JSON.

docs/user_guide.html

@@ -0,0 +1,142 @@
+<!DOCTYPE html>
+<html lang="en">
+<head>
+<meta charset="utf-8">
+<title>ForexBot2 — User Guide</title>
+<style>
+  @page {
+    size: A4;
+    margin: 22mm 18mm;
+    @bottom-center { content: "ForexBot2 User Guide — page " counter(page) " of " counter(pages); font-size: 8pt; color: #888; }
+  }
+  * { box-sizing: border-box; }
+  body { font-family: "DejaVu Sans", Helvetica, Arial, sans-serif; font-size: 10pt; line-height: 1.55; color: #1c2733; margin: 0; }
+  h1 { font-size: 21pt; margin: 0 0 2mm; color: #0d1b2a; }
+  .subtitle { color: #5a6b7c; font-size: 10.5pt; margin-bottom: 8mm; }
+  h2 { font-size: 13.5pt; color: #0d1b2a; border-bottom: 2px solid #d4a017; padding-bottom: 1.2mm; margin: 8mm 0 3mm; page-break-after: avoid; }
+  h3 { font-size: 11pt; margin: 5mm 0 1.5mm; color: #23374d; page-break-after: avoid; }
+  p { margin: 0 0 2.5mm; }
+  table { border-collapse: collapse; width: 100%; margin: 3mm 0 4mm; font-size: 9pt; page-break-inside: avoid; }
+  th { background: #0d1b2a; color: #fff; text-align: left; padding: 1.8mm 2.5mm; font-weight: 600; }
+  td { border-bottom: 1px solid #dde4ea; padding: 1.6mm 2.5mm; }
+  tr:nth-child(even) td { background: #f4f7fa; }
+  .signal-box { background: #f4f7fa; border: 1px solid #c9d4de; border-left: 4px solid #d4a017; border-radius: 2mm; padding: 3.5mm 4.5mm; font-family: "DejaVu Sans Mono", monospace; font-size: 8.4pt; line-height: 1.5; white-space: pre-wrap; margin: 3mm 0 4mm; page-break-inside: avoid; }
+  .callout { background: #fdf6e3; border: 1px solid #e8d49a; border-left: 4px solid #d4a017; border-radius: 2mm; padding: 3mm 4.5mm; margin: 3mm 0 4mm; page-break-inside: avoid; }
+  .callout.red { background: #fdf0ee; border-color: #eab4ac; border-left-color: #c0392b; }
+  .callout b:first-child { display: block; margin-bottom: 1mm; }
+  ul, ol { margin: 0 0 3mm; padding-left: 6mm; }
+  li { margin-bottom: 1.4mm; }
+  code { font-family: "DejaVu Sans Mono", monospace; font-size: 8.6pt; background: #eef2f6; padding: 0.3mm 1.2mm; border-radius: 1mm; }
+  .small { font-size: 8.5pt; color: #5a6b7c; }
+  .disclaimer { border: 1.5px solid #c0392b; border-radius: 2mm; padding: 3.5mm 4.5mm; margin-top: 6mm; font-size: 9pt; page-break-inside: avoid; }
+  .disclaimer b { color: #c0392b; }
+</style>
+</head>
+<body>
+
+<h1>ForexBot2 — User Guide</h1>
+<div class="subtitle">How to read, act on, and survive the signals &nbsp;·&nbsp; July 2026</div>
+
+<h2>1 · What the bot does</h2>
+<p>ForexBot2 analyzes <b>gold (XAU/USD)</b> around the clock on several chart timeframes at once. On every candle close it scores nine technical indicators (trend, momentum, volatility, divergences, candlestick patterns), adapts the weighting to the current market regime (trending / ranging / transitioning), checks that higher timeframes agree, adjusts for news and economic-calendar events, and — only when everything lines up — sends a <b>signal with a complete trade plan</b> to Telegram: direction, entry, stop loss, target, and a trailing-stop management plan.</p>
+<p>Most of the time it sends nothing. Silence means no edge was found; that is the bot working correctly, not a malfunction.</p>
+
+<h2>2 · Anatomy of a signal</h2>
+<div class="signal-box">🔍 XAU/USD Signal Analysis  (M15)
+🕐 2026-07-10 14:00 UTC
+💰 Close: 4,132.50
+📊 Regime: Trending Up | Vol: normal
+━━━━━━━━━━━━━━━━━━━━
+🟢 BUY   🅰 A  |  Score: +38.2  |  64% conf
+✅ MTF aligned
+📈 MTF: M5 ✅  H1 ✅
+📋 Indicators … (9 rows)
+⚠️ EXTREME volatility — wider stops needed
+━━━━━━━━━━━━━━━━━━━━
+🚀 TRADE SETUP
+📈 Direction: BUY
+▶ Entry:  4,132.50
+🛑 SL:     4,103.10   ($29.40 / 294 pips)
+🎯 TP1:    4,161.90   ($29.40 / 294 pips)
+⚖️ RR:     1:1.0
+🔄 Trail: once price hits 4,150.14 (+0.6R),
+   trail SL $22.05 behind best price —
+   TP1 is a guide, let it run</div>
+
+<table>
+<tr><th style="width:28%">Field</th><th>What it means</th></tr>
+<tr><td><b>Signal + grade</b></td><td>Direction (BUY / SELL / STRONG_*) plus a quality grade <b>A+ → F</b>. A trade setup is only attached at grade C or better. Higher grades = more indicator agreement and cleaner regime.</td></tr>
+<tr><td><b>Score / confidence</b></td><td>The regime-weighted composite of all nine indicators. Bigger absolute score = stronger consensus.</td></tr>
+<tr><td><b>MTF aligned</b></td><td>The two confirmation timeframes agree with the trade direction. Signals without alignment are filtered out on M5.</td></tr>
+<tr><td><b>Warnings ⚠️</b></td><td>Context, not veto. Backtesting shows warned trades perform about the same — strong momentum often triggers warnings.</td></tr>
+<tr><td><b>Market context 📰</b></td><td>An AI-written note in plain English on what is moving the gold price right now, why it matters, and the next report to watch. It is built only from headlines published in the last 4 hours that lean one way (neutral items are dropped) plus today's high-impact releases. Informational only — it never blocks or grades a signal.</td></tr>
+<tr><td><b>SL / TP1 / RR</b></td><td>Stop loss sits behind the nearest swing level with an ATR buffer. TP1 is the nearest opposing level. <b>1R</b> = the entry-to-SL distance = the amount you risk.</td></tr>
+<tr><td><b>Trail plan 🔄</b></td><td>The real exit. Once price moves +0.6R in your favor, move the stop to trail 0.75R behind the best price and update it at each candle close. <b>TP1 is advisory — let winners run.</b></td></tr>
+</table>
+
+<h2>3 · Acting on a signal — step by step</h2>
+<ol>
+<li><b>Size the position first.</b> Decide your risk per trade — <b>0.5–1% of account, never more</b>. Position size = (account × risk%) ÷ stop distance in $. The stop distance varies per signal, so the size changes every trade; the dollar risk must not.</li>
+<li><b>Enter promptly at market.</b> All backtested results assume entry at the open of the candle after the signal. On M5, minutes matter (see §4). If you are very late — price already moved a large part of the way to the arm level — skip the trade. There is always another signal.</li>
+<li><b>Place the stop loss immediately</b> at the given SL. No mental stops — gold gaps and spikes.</li>
+<li><b>Manage with the trail, not TP1.</b> When price reaches the arm level printed in the message, start trailing the stop the stated $ distance behind the best price, tightening only at candle close. This is what produced the backtested results; taking fixed TP1 profits gives most of the edge away.</li>
+<li><b>One signal, one trade.</b> The M5, M15 and H1 books confirm off each other and are usually long or short gold at the same time. Taking the same move on multiple timeframes multiplies risk, not diversification. Pick your timeframe (§4) and ignore the others.</li>
+</ol>
+
+<h2>4 · Choosing your timeframe</h2>
+<p>Twelve-month Oanda-candle backtest (Aug 2025 – Aug 2026, $0.65 round-trip cost, entry at next candle open):</p>
+<table>
+<tr><th></th><th>M5</th><th>M15</th><th>H1</th></tr>
+<tr><td>Trades / week</td><td>~11.3</td><td>~7.6</td><td>~2.7</td></tr>
+<tr><td>Total result (12 mo)</td><td>+106.8R</td><td>+73.9R</td><td>+24.0R</td></tr>
+<tr><td>Win rate / profit factor</td><td>52.9% / 1.51</td><td>62.2% / 1.53</td><td>60.9% / 1.55</td></tr>
+<tr><td>Typical (median) hold</td><td>2.4 h</td><td>7 h</td><td>15.5 h</td></tr>
+<tr><td>90% of trades close within</td><td>10.5 h</td><td>29.8 h</td><td>3.2 days</td></tr>
+<tr><td>Typical stop distance</td><td>~$21</td><td>~$32</td><td>~$52</td></tr>
+<tr><td>Worst drawdown</td><td>10.7R</td><td>10.8R</td><td>8.0R</td></tr>
+</table>
+<ul>
+<li><b>M5 — intraday, execution-critical.</b> Highest total edge, but stops are tight: a couple of dollars of slippage or a late entry eats a meaningful share of every trade. Suitable only if you can act within a minute or two of the alert (or automate). Trades open and close within a session; you sleep flat. SELL signals are held back while the market regime reads "transitioning" — a validated filter, not a bug if you notice fewer shorts there.</li>
+<li><b>M15 — the middle path.</b> Same frequency as M5, wider stops, up to 15 minutes to act. About half your trades will still be open overnight.</li>
+<li><b>H1 — swing trading around a normal life.</b> Best quality per trade and very forgiving of slow manual execution (you have up to an hour). The costs: positions held for days, through news and overnight sessions, and a <b>lumpy</b> equity curve — one quarter of the test year was negative and most of the profit came in a single quarter. Judge it on quarters, not weeks. H1 no longer issues SELL signals at all (blocked since 2026-09-03) — it is BUY-only on this timeframe.</li>
+</ul>
+<div class="callout"><b>If you trade manually from the Telegram alerts, H1 (or M15) is the honest choice.</b> M5's backtested edge assumes near-instant fills that a human copying signals rarely achieves.</div>
+
+<h2>5 · What to expect (and what not to)</h2>
+<ul>
+<li><b>You will lose roughly 38–47% of trades.</b> A 53–62% win rate means losing streaks of 5+ are routine over a year. The system's edge is the average across hundreds of trades, not any single one.</li>
+<li><b>Stops hit fast, winners take time.</b> Losing trades die in roughly half the time winners take. An early stop-out is the system doing its job.</li>
+<li><b>Drawdowns of ~10R are normal.</b> At 1% risk per trade, expect to be ~10% down from a peak at some point in a normal year. This is why risking more than 1% per trade is how accounts die.</li>
+<li><b>Backtest ≠ live.</b> Results above charge real spreads and realistic fills, but cannot model your personal latency, broker slippage, or requotes. Assume live results will be somewhat worse — especially on M5.</li>
+<li><b>Quiet periods are normal.</b> Filters block bad hours, bad regimes and misaligned timeframes. Days can pass without a signal.</li>
+</ul>
+
+<h2>6 · Rules that keep you alive</h2>
+<div class="callout red">
+<b>The five rules</b>
+<ol style="margin-bottom:0">
+<li>Risk a fixed 0.5–1% per trade. Never "make it back" by sizing up after losses.</li>
+<li>Hard stop loss in the market on every trade, placed at entry time.</li>
+<li>One timeframe, one trade at a time. No stacking, no pyramiding onto bot signals.</li>
+<li>Missed the entry? Skip it. Chasing converts a positive-edge trade into a coin flip.</li>
+<li>Evaluate performance monthly at the earliest, quarterly for H1. Never after a bad day.</li>
+</ol>
+</div>
+
+<h2>7 · Bot commands (owner)</h2>
+<table>
+<tr><th style="width:30%">Command</th><th>What it does</th></tr>
+<tr><td><code>/start</code></td><td>Show the button menu (manual scans per timeframe, Scan All TFs).</td></tr>
+<tr><td><code>/scan</code></td><td>Run an immediate analysis of the primary timeframe.</td></tr>
+<tr><td><code>/autoscan</code></td><td>Toggle automatic scanning (M5 / M15 / H1 / H4 at each candle close).</td></tr>
+<tr><td><code>/status</code></td><td>Uptime, auto-scan state, WebSocket price-monitor state.</td></tr>
+<tr><td><code>/news</code></td><td>Net XAU/USD news score from the last 4 hours of headlines (neutral headlines are removed before averaging; informational only — never gates signals).</td></tr>
+<tr><td><code>/cache</code> · <code>/clearcache</code></td><td>Inspect / reset the market-data cache.</td></tr>
+<tr><td><code>/help</code></td><td>Command reference.</td></tr>
+</table>
+<p class="small">The bot also streams live prices over WebSocket and fires an immediate scan when price breaches a key level (swing S/R, Bollinger, Keltner, EMA-200). Commands respond only to the configured owner chat; subscribers receive the signal feed via the channel.</p>
+
+<div class="disclaimer"><b>⚠️ Risk disclaimer.</b> This bot and guide are for educational and research purposes only and are <b>not financial advice</b>. Trading forex, gold and CFDs on margin carries a high risk of loss; past (and backtested) performance does not guarantee future results. Do your own analysis and never risk money you cannot afford to lose.</div>
+
+</body>
+</html>

llm_client.py

@@ -0,0 +1,207 @@
+"""
+ForexBot2 — DeepSeek LLM client
+================================
+Thin requests-based client for DeepSeek's OpenAI-compatible chat API,
+used by `news_engine` for display/warning-only enrichment (headline
+sentiment, event surprise direction, market summary). Signals are NEVER
+gated on LLM output — every caller has a non-LLM fallback and treats a
+`None` return as "use the fallback".
+
+Design constraints:
+- One attempt per call, no retries: news runs inside scan cycles; the
+  news engine's 600 s context cache is the natural retry cadence.
+- A UTC-day request cap bounds spend even if a bug causes call storms.
+- Cost: DeepSeek caches identical prompt PREFIXES server-side (cache-hit
+  input is ~50x cheaper), so callers pass a STATIC system prompt (no
+  timestamps or interpolation) and put all variable content in the user
+  payload.
+
+This is a module-level singleton (`get_llm_client`) because the daily
+cap counter must survive the per-scan reconstruction of `NewsEngine`.
+"""
+
+import datetime as _dt
+import json
+import logging
+import threading
+import time
+from typing import Any, Dict, Optional, Tuple
+
+import requests
+
+from app_config import deepseek_api_key, get_app_config
+
+logger = logging.getLogger(__name__)
+
+DEEPSEEK_URL = "https://api.deepseek.com/chat/completions"
+DEFAULT_MODEL = "deepseek-v4-flash"
+# 8s timed out on a measured cold-prefix-cache headline batch (2026-08-03);
+# warm-cache calls come back well under that. 12s buys cold-start headroom
+# while still bounding worst-case scan latency (≤3 calls per cache window).
+DEFAULT_TIMEOUT = 12.0
+DEFAULT_DAILY_CAP = 500
+
+
+class DeepSeekClient:
+    """JSON-in/JSON-out chat wrapper. All failures return None."""
+
+    def __init__(
+        self,
+        api_key: str,
+        model: str = DEFAULT_MODEL,
+        timeout: float = DEFAULT_TIMEOUT,
+        daily_cap: int = DEFAULT_DAILY_CAP,
+        enabled: bool = True,
+    ):
+        self._api_key = api_key or ""
+        self._model = model
+        self._timeout = timeout
+        self._daily_cap = daily_cap
+        self._enabled = enabled
+        self._session = requests.Session()
+        self._session.headers.update({
+            "Content-Type": "application/json",
+            "Authorization": f"Bearer {self._api_key}",
+        })
+        self._lock = threading.Lock()
+        self._count_day: Optional[_dt.date] = None
+        self._count = 0
+        self._warned: set = set()
+        # Health stamps for status displays: last successful call and
+        # last failure (ts, kind). Read-only for consumers.
+        self.last_ok: float = 0.0
+        self.last_error: Optional[Tuple[float, str]] = None
+
+    @property
+    def available(self) -> bool:
+        return self._enabled and bool(self._api_key)
+
+    def _warn_once(self, kind: str, msg: str, *args: Any) -> None:
+        if kind in self._warned:
+            logger.debug(msg, *args)
+        else:
+            self._warned.add(kind)
+            logger.warning(msg, *args)
+
+    def _under_cap(self) -> bool:
+        """Count a request against the UTC-day cap; False = don't call."""
+        today = _dt.datetime.now(_dt.timezone.utc).date()
+        with self._lock:
+            if self._count_day != today:
+                self._count_day = today
+                self._count = 0
+                # Clear ALL warn-once kinds, not just "cap": the network/
+                # http_*/parse flags used to be set once and never reset,
+                # so in a daemon that runs for weeks the second and every
+                # later outage logged at DEBUG only.
+                self._warned.clear()
+            if self._count >= self._daily_cap:
+                self._warn_once(
+                    "cap", "DeepSeek daily request cap (%d) exhausted — "
+                    "falling back to keyword analysis until UTC midnight",
+                    self._daily_cap)
+                return False
+            self._count += 1
+            return True
+
+    def chat_json(self, system_prompt: str,
+                  user_payload: Dict[str, Any]) -> Optional[Dict[str, Any]]:
+        """One chat call; returns the model's JSON object or None."""
+        if not self.available or not self._under_cap():
+            return None
+        body = {
+            "model": self._model,
+            "messages": [
+                {"role": "system", "content": system_prompt},
+                {"role": "user",
+                 "content": json.dumps(user_payload, sort_keys=True)},
+            ],
+            "response_format": {"type": "json_object"},
+            "temperature": 0,
+            # V4-Flash is a hybrid thinking model; without this flag it
+            # burns hundreds of reasoning tokens per call — measured
+            # 2026-08-03: a 25-headline batch blew straight through the
+            # timeout AND could exhaust max_tokens before the answer,
+            # returning empty content. The doctrine + few-shot examples
+            # in the system prompts carry the quality instead.
+            "thinking": {"type": "disabled"},
+            # Must fit the largest reply: a full sentiment batch is
+            # ~10 tokens per headline; 1000 truncated mid-JSON on a real
+            # RSS cycle (2026-08-03).
+            "max_tokens": 2000,
+        }
+        try:
+            resp = self._session.post(
+                DEEPSEEK_URL, json=body, timeout=self._timeout)
+        except requests.RequestException as e:
+            self.last_error = (time.time(), "network")
+            self._warn_once("network", "DeepSeek call failed: %s", e)
+            return None
+        if resp.status_code != 200:
+            self.last_error = (time.time(), f"http_{resp.status_code}")
+            self._warn_once(
+                f"http_{resp.status_code}",
+                "DeepSeek HTTP %s: %s", resp.status_code, resp.text[:200])
+            return None
+        try:
+            parsed = self._parse_response(resp.json())
+        except ValueError:
+            parsed = None
+        if parsed is None:
+            self.last_error = (time.time(), "parse")
+            self._warn_once("parse", "DeepSeek response was not the "
+                            "expected JSON object — ignoring")
+        else:
+            self.last_ok = time.time()
+            # A recovered call re-arms the failure warnings so the NEXT
+            # outage is visible at WARNING again. "cap" keeps its own
+            # UTC-day reset in _under_cap. (Taken under the lock; safe
+            # here because chat_json holds no lock — _under_cap has
+            # already returned.)
+            if self._warned - {"cap"}:
+                with self._lock:
+                    self._warned = {k for k in self._warned if k == "cap"}
+        return parsed
+
+    @staticmethod
+    def _parse_response(body: Any) -> Optional[Dict[str, Any]]:
+        """Extract the assistant's JSON object from a chat completion."""
+        try:
+            content = body["choices"][0]["message"]["content"]
+        except (KeyError, IndexError, TypeError):
+            return None
+        try:
+            obj = json.loads(content)
+        except (TypeError, ValueError):
+            return None
+        return obj if isinstance(obj, dict) else None
+
+
+# ── Process-wide singleton ────────────────────────────────────────────
+
+_client: Optional[DeepSeekClient] = None
+_client_lock = threading.Lock()
+
+
+def get_llm_client() -> DeepSeekClient:
+    global _client
+    with _client_lock:
+        if _client is None:
+            settings = get_app_config().deepseek_settings
+            _client = DeepSeekClient(
+                api_key=deepseek_api_key(),
+                model=str(settings.get("model", DEFAULT_MODEL)),
+                timeout=float(settings.get("timeout_seconds",
+                                           DEFAULT_TIMEOUT)),
+                daily_cap=int(settings.get("daily_request_cap",
+                                           DEFAULT_DAILY_CAP)),
+                enabled=settings.get("enabled", True) is not False,
+            )
+        return _client
+
+
+def reset_llm_client() -> None:
+    """Drop the singleton (tests / config reloads)."""
+    global _client
+    with _client_lock:
+        _client = None

llm_provider_switch_plan.txt

@@ -0,0 +1,252 @@
+================================================================================
+MULTI-PROVIDER LLM LAYER - PLAN
+(DeepSeek for news summary, Anthropic Claude for agent panel)
+Written 2026-08-08. Plan only - no code has been changed.
+================================================================================
+
+WHY
+---
+Today every LLM feature (news enrichment + agent panel) goes through one
+hardwired client: llm_client.py -> DeepSeekClient, a module singleton reached
+via get_llm_client(). This plan describes how to route different features to
+different providers - e.g. keep DeepSeek for cheap high-volume news scoring,
+put the agent panel on Anthropic Claude for better reasoning.
+
+The repo already solved this problem once for market data: the providers/
+package (BaseDataProvider owns shared machinery, subclasses implement only
+_fetch_raw, a registry + one config knob selects the provider). This plan
+mirrors that proven pattern at the LLM layer, scaled down to a single module.
+
+BLAST RADIUS (verified by grep)
+-------------------------------
+get_llm_client() call sites:
+  - news_engine.py ....... 4 sites
+  - agent_panel.py ....... 3 sites (+1 timeout getattr)
+  - telegram_bot.py ...... 1 site (/status health; reads private _count,
+                           _count_day, _daily_cap fields)
+  - test_all.py .......... 41 locations stub via  lc._client = _StubLLM(...)
+Out of scope: static/news_summary.php (own PHP DeepSeek call + key file,
+separate deployment).
+
+DESIGN INVARIANTS (carried over unchanged)
+------------------------------------------
+1. Fail-safe contract: every call returns Optional[Dict]; any failure -> None;
+   callers keep their non-LLM fallbacks; LLM output never gates signals.
+2. Single attempt, no retries (news cache cadence is the retry); per-provider
+   UTC daily cap; warn-once logging; health stamps (last_ok, last_error).
+3. Static system prompts stay byte-identical constants - provider-neutral,
+   and they map to prompt caching on BOTH providers (DeepSeek prefix cache;
+   Anthropic cache_control ephemeral).
+4. Test compatibility: all 41 existing lc._client stub sites keep working
+   with zero edits.
+
+================================================================================
+1. llm_client.py REFACTOR  (~195 -> ~430 lines, no new files)
+================================================================================
+
+1a. BaseLLMClient - shared machinery (extracted from today's DeepSeekClient)
+    - __init__(api_key, model, timeout, daily_cap, enabled)
+    - public: name ("deepseek"/"anthropic"), timeout property,
+      usage() -> (used_today, cap)   # so telegram_bot stops reading privates
+    - owns: available property, _warn_once, _under_cap() (UTC-day counter,
+      lock), health stamps, and the template method:
+
+        def chat_json(self, system_prompt, user_payload):
+            if not self.available or not self._under_cap():
+                return None
+            try:
+                return self._request(system_prompt, user_payload)  # hook
+            except <provider-agnostic error mapping>:
+                # stamps last_error, returns None
+
+    - subclasses implement ONLY _request(system_prompt, payload) -> dict|None.
+
+1b. DeepSeekClient(BaseLLMClient) - behavior-identical move
+    Current requests-based logic relocates into _request: same URL,
+    thinking:disabled, response_format:json_object, temperature 0,
+    max_tokens 2000, sort_keys=True payload, _parse_response.
+    Review criterion: Module 19 tests pass UNMODIFIED (they patch
+    client._session, which stays).
+
+1c. AnthropicClient(BaseLLMClient) - new, official SDK
+    - Official `anthropic` Python SDK (add to requirements.txt), NOT raw
+      requests. LAZY IMPORT inside __init__: package missing -> warn-once +
+      client unavailable, so DeepSeek-only deployments never install it.
+    - Constructor: anthropic.Anthropic(api_key=..., timeout=<configured>,
+      max_retries=0)   # house rule: single attempt
+    - _request:
+        resp = self._client.messages.create(
+            model=self._model,                  # default "claude-opus-5"
+            max_tokens=4000,                    # covers thinking + JSON reply
+            output_config={"effort": "low"},    # cheap structured takes
+            system=[{"type": "text", "text": system_prompt,
+                     "cache_control": {"type": "ephemeral"}}],
+            messages=[{"role": "user",
+                       "content": json.dumps(user_payload, sort_keys=True)}],
+        )
+        if resp.stop_reason == "refusal":       # Opus 5: HTTP 200, check first
+            return None
+        text = first text block in resp.content
+        return self._parse_json(text)   # tolerant: strip ```json fences,
+                                        # json.loads, dict check
+    - DELIBERATELY ABSENT (would 400 on Claude Opus 5): temperature (removed
+      on Opus 5 - do NOT copy from the DeepSeek client), budget_tokens,
+      prefills, response_format.
+    - Error mapping: APIConnectionError -> "network",
+      APIStatusError -> "http_<code>", parse fail -> "parse"
+      (same last_error kinds the /status display already understands).
+    - Defaults: model claude-opus-5 (config-overridable, e.g. claude-haiku-4-5
+      for ~5x cheaper), timeout 30s (Claude slower than V4-Flash), own
+      daily_request_cap 500.
+
+1d. Registry + per-consumer routing
+        _PROVIDERS = {"deepseek": DeepSeekClient, "anthropic": AnthropicClient}
+        LLM_CONSUMERS = ("news", "agent_panel")
+        _clients = {}     # keyed by PROVIDER name -> two consumers on the
+                          # same provider share one instance and one cap
+        _client = None    # kept: GLOBAL TEST OVERRIDE
+
+        def get_llm_client(consumer="news"):
+            if _client is not None:          # tests: lc._client = stub
+                return _client               #   -> shadows ALL consumers
+            name = get_app_config().llm_routing.get(consumer, "deepseek")
+            create-once per provider, return _clients[name]
+
+        def reset_llm_client():              # clears _client AND _clients
+
+    Keeping _client as a documented test-override global is what makes the
+    41 existing stub sites work untouched.
+
+================================================================================
+2. CONFIG  (app_config.py + config.json.example)
+================================================================================
+- New top-level "anthropic" block, mirroring "deepseek":
+    {api_key, model, enabled, timeout_seconds, daily_request_cap}
+  plus anthropic_api_key() resolver: ANTHROPIC_API_KEY env -> config.json -> "".
+- New top-level "llm" ROUTING block:
+    {"news": "deepseek", "agent_panel": "deepseek"}
+  Validation: keys within {news, agent_panel}; values within registered
+  provider names. Absent block -> {} -> everything on DeepSeek (today's
+  behavior; zero-migration default). AppConfig.llm_routing: Dict[str, str].
+
+- The target state is then PURE CONFIG - no code changes on switch day:
+
+    "llm":       { "news": "deepseek", "agent_panel": "anthropic" },
+    "anthropic": { "api_key": "", "model": "claude-opus-5", "enabled": true,
+                   "timeout_seconds": 30, "daily_request_cap": 500 }
+
+================================================================================
+3. CONSUMER EDITS  (mechanical, ~10 lines total)
+================================================================================
+- news_engine.py:  4 call sites -> get_llm_client("news")
+- agent_panel.py:  3 call sites -> get_llm_client("agent_panel");
+                   getattr(client, "_timeout", ...) -> public client.timeout
+- telegram_bot.py _news_health_lines: iterate the DISTINCT providers actually
+  routed (news + panel); per provider print "<name>: used/cap calls today"
+  via the new usage() accessor instead of private fields. DeepSeek keeps its
+  "headlines scored" extra; cap/error warnings work off base-class stamps.
+- NO changes to prompts, payload builders, caches, or the panel's own
+  max_panels_per_day budget - all provider-agnostic already.
+
+================================================================================
+4. TESTS  (new "Module 24: LLM provider routing", ~12 tests, ~200 lines)
+================================================================================
+Reuse _StubLLM; new _FakeAnthropic (fake messages.create returning a stub
+response object) injected as client._client.
+
+ 1. app_config: anthropic block parses/validates/defaults empty.
+ 2. app_config: llm routing block - valid map round-trips; unknown consumer
+    key, unknown provider name, non-dict -> AppConfigError; absent -> {}.
+ 3. anthropic_api_key resolves env -> config.json -> empty.
+ 4. Base contract on BOTH subclasses (parametrized): keyless/disabled -> no
+    call; daily cap blocks + UTC-roll reset; failure stamps last_error,
+    success stamps last_ok.
+ 5. AnthropicClient parse: clean JSON -> dict; ```json-fenced -> dict;
+    non-JSON -> None; stop_reason "refusal" -> None; no text block -> None.
+ 6. AnthropicClient request shape: NO temperature key, max_retries=0,
+    cache_control present on system block (assert on captured kwargs).
+ 7. AnthropicClient with `anthropic` package missing (simulated) ->
+    available False, chat_json None, no crash at import.
+ 8. Routing: no llm block -> both consumers get the SAME DeepSeek singleton
+    (shared cap); routed config -> news gets DeepSeekClient while
+    agent_panel gets AnthropicClient; same-provider consumers share one
+    instance.
+ 9. Global lc._client override shadows BOTH consumers (locks the 41 legacy
+    stub sites' semantics).
+10. reset_llm_client() clears both the override and the per-provider cache.
+11. End-to-end: routing agent_panel->anthropic, attach_panel reaches the
+    anthropic fake (5 calls) while a news call reaches the deepseek stub.
+12. telegram _news_health_lines renders per-provider lines from usage()
+    without touching private fields.
+
+GATE: Modules 19 and 23 must pass UNMODIFIED - the no-regression proof for
+the refactor.
+
+================================================================================
+5. DOCS
+================================================================================
+- CLAUDE.md: llm_client bullet -> provider-abstracted (BaseLLMClient;
+  deepseek + anthropic registered); config.json "llm" block routes consumers;
+  absent = all-DeepSeek. Add ANTHROPIC_API_KEY to env-var list. Test counts.
+- config.json.example: both new blocks.
+
+================================================================================
+COMMIT SEQUENCE  (each commit green)
+================================================================================
+1. llm_client: extract BaseLLMClient; DeepSeekClient unchanged behavior
+   (+ base-contract tests)  - pure refactor, Modules 19/23 green untouched.
+2. config: anthropic block + llm consumer-routing block (+ config tests).
+3. llm_client: AnthropicClient (official SDK, lazy import) + requirements.txt
+   (+ parse/shape tests).
+4. routing: per-consumer get_llm_client + health lines - consumer call-site
+   edits, telegram usage() lines, routing/e2e tests.
+5. docs: CLAUDE.md + example config.
+
+================================================================================
+VERIFICATION
+================================================================================
+- python test_all.py green at every commit (~374 -> ~386).
+- Live smoke A (no behavior change): default config, main.py --panel ->
+  still DeepSeek, /status unchanged.
+- Live smoke B (the goal): llm.agent_panel="anthropic" + real
+  ANTHROPIC_API_KEY, run --panel -> one real Claude panel (5 requests,
+  validates JSON contract + latency + refusal-free path) while a news
+  refresh still hits DeepSeek; /status shows two provider lines.
+- Backtest untouched by construction (no LLM in replay path); one 30-day
+  run as a formality.
+
+================================================================================
+EFFORT ESTIMATE
+================================================================================
+0 new files, 8 modified, ~520 lines, est. 3-5 hours.
+
+  llm_client.py        base class + AnthropicClient + routing   +235
+  test_all.py          Module 24                                +200
+  app_config.py        anthropic + llm blocks, resolver          +55
+  telegram_bot.py      per-provider health lines                 +15
+  news_engine.py       consumer tags on call sites                +4
+  agent_panel.py       consumer tags on call sites                +4
+  config.json.example  blocks                                    +14
+  requirements.txt     anthropic                                  +1
+  CLAUDE.md            docs                                      +10
+
+================================================================================
+KEY DECISIONS EMBEDDED IN THIS PLAN
+================================================================================
+- Official Anthropic SDK, not raw requests (current API guidance for Python;
+  lazy import keeps it optional for DeepSeek-only installs).
+- Default Claude model: claude-opus-5 ($5/$25 per MTok; a full 5-call panel
+  ~ 8k in / 500 out ~ $0.05). Config-overridable to claude-haiku-4-5.
+- Two Claude gotchas handled: temperature is REMOVED on Opus 5 (sending it
+  400s - must not be copied from the DeepSeek request body), and refusals
+  arrive as HTTP 200 with stop_reason "refusal" (checked before parsing,
+  mapped to the normal None/fallback path).
+- Caps are per-provider, not per-consumer: two consumers on one provider
+  share its cap (today's semantics); splitting across two providers doubles
+  total headroom naturally.
+- The panel's own max_panels_per_day budget is untouched - it sits above
+  whichever provider serves it.
+- Not in scope: static/news_summary.php (own PHP client/key), streaming,
+  tool use, per-consumer model overrides (add a "model" key per consumer
+  later if ever needed).
+================================================================================

main.py

@@ -1,35 +1,30 @@
 #!/usr/bin/env python3
 """
-Forex Signal Intelligence Platform
-====================================
-Main entry point. Runs full analysis and outputs formatted signal reports.
-
+ForexBot2 — XAU/USD Signal Intelligence
+========================================
 Usage:
-    python main.py                          # Analyze all major pairs
-    python main.py --pairs EURUSD GBPUSD    # Specific pairs
-    python main.py --timeframe H4           # Different primary TF
-    python main.py --json                   # JSON output
-    python main.py --watch                  # Continuous monitoring mode
+    python main.py                              # Live signal on H1
+    python main.py --timeframe H4 --detail      # Detailed H4 signal
+    python main.py --json                       # JSON output
 """
 
 import argparse
 import json
 import logging
 import sys
-import time
 from datetime import datetime, timezone
-from typing import List
+from typing import Optional
 
-# Force UTF-8 output on Windows so box-drawing / Unicode chars render correctly
+# Force UTF-8 output on Windows so box-drawing chars render
 if hasattr(sys.stdout, "reconfigure"):
     sys.stdout.reconfigure(encoding="utf-8")
 if hasattr(sys.stderr, "reconfigure"):
     sys.stderr.reconfigure(encoding="utf-8")
 
-from config import (
-    PlatformConfig, Timeframe, SignalStrength,
-    MAJOR_PAIRS, CROSS_PAIRS,
-)
+from agent_panel import attach_panel
+from config import PlatformConfig, Timeframe, SignalStrength, XAUUSD
+from market_outlook import OutlookResult, maybe_run_outlook
+from pair_signals import PAIR_PRODUCERS
 from signal_aggregator import SignalAggregator, SignalReport
 
 
@@ -38,12 +33,10 @@ from signal_aggregator import SignalAggregator, SignalReport
 # ══════════════════════════════════════════════════════════════════
 
 class Colors:
-    """ANSI color codes for terminal output."""
     RESET = "\033[0m"
     BOLD = "\033[1m"
     DIM = "\033[2m"
     UNDERLINE = "\033[4m"
-
     RED = "\033[91m"
     GREEN = "\033[92m"
     YELLOW = "\033[93m"
@@ -53,224 +46,180 @@ class Colors:
     WHITE = "\033[97m"
     GRAY = "\033[90m"
 
-    BG_RED = "\033[41m"
-    BG_GREEN = "\033[42m"
-    BG_YELLOW = "\033[43m"
-    BG_BLUE = "\033[44m"
-    BG_MAGENTA = "\033[45m"
-    BG_CYAN = "\033[46m"
-
 
 def signal_color(signal: SignalStrength) -> str:
-    """Get color for a signal strength."""
     return {
-        SignalStrength.STRONG_BUY: Colors.GREEN + Colors.BOLD,
-        SignalStrength.BUY: Colors.GREEN,
-        SignalStrength.NEUTRAL: Colors.GRAY,
-        SignalStrength.SELL: Colors.RED,
+        SignalStrength.STRONG_BUY:  Colors.GREEN + Colors.BOLD,
+        SignalStrength.BUY:         Colors.GREEN,
+        SignalStrength.NEUTRAL:     Colors.GRAY,
+        SignalStrength.SELL:        Colors.RED,
         SignalStrength.STRONG_SELL: Colors.RED + Colors.BOLD,
     }.get(signal, Colors.RESET)
 
 
-def quality_color(quality: str) -> str:
-    """Get color for signal quality grade."""
+def quality_color(q: str) -> str:
     return {
         "A+": Colors.GREEN + Colors.BOLD,
-        "A": Colors.GREEN,
-        "B": Colors.CYAN,
-        "C": Colors.YELLOW,
-        "D": Colors.GRAY,
-        "F": Colors.DIM,
-    }.get(quality, Colors.RESET)
+        "A":  Colors.GREEN,
+        "B":  Colors.CYAN,
+        "C":  Colors.YELLOW,
+        "D":  Colors.GRAY,
+        "F":  Colors.DIM,
+    }.get(q, Colors.RESET)
 
 
-def score_bar(score: float, width: int = 30) -> str:
-    """Create a visual score bar from -100 to +100."""
-    normalized = (score + 100) / 200  # 0 to 1
-    filled = int(normalized * width)
-    center = width // 2
-
-    bar = ""
-    for i in range(width):
-        if i == center:
-            bar += "│"
-        elif (score >= 0 and center < i <= filled) or \
-             (score < 0 and filled <= i < center):
-            bar += "█"
-        else:
-            bar += "░"
-
-    if score >= 0:
-        color = Colors.GREEN if score > 20 else Colors.GRAY
-    else:
-        color = Colors.RED if score < -20 else Colors.GRAY
-
-    return f"{color}{bar}{Colors.RESET}"
-
-
-def print_header():
-    """Print the platform header."""
+def print_header(title: str = "XAU/USD SIGNAL INTELLIGENCE"):
     now = datetime.now(timezone.utc).strftime("%Y-%m-%d %H:%M:%S UTC")
-    width = 80
-
-    print(f"\n{Colors.CYAN}{'═' * width}{Colors.RESET}")
-    print(f"{Colors.BOLD}{Colors.CYAN}  ╔═══════════════════════════════════════════════════════╗{Colors.RESET}")
-    print(f"{Colors.BOLD}{Colors.CYAN}  ║   FOREX SIGNAL INTELLIGENCE PLATFORM                 ║{Colors.RESET}")
-    print(f"{Colors.BOLD}{Colors.CYAN}  ║   Multi-Layer Signal Analysis Engine                  ║{Colors.RESET}")
-    print(f"{Colors.BOLD}{Colors.CYAN}  ╚═══════════════════════════════════════════════════════╝{Colors.RESET}")
-    print(f"{Colors.GRAY}  Scan Time: {now}{Colors.RESET}")
-    print(f"{Colors.CYAN}{'═' * width}{Colors.RESET}\n")
+    print(f"\n{Colors.CYAN}{'═' * 80}{Colors.RESET}")
+    print(f"{Colors.BOLD}{Colors.CYAN}  {title}{Colors.RESET}")
+    print(f"{Colors.GRAY}  Time: {now}{Colors.RESET}")
+    print(f"{Colors.CYAN}{'═' * 80}{Colors.RESET}\n")
 
 
-def print_summary_table(reports: List[SignalReport]):
-    """Print a summary table of all pairs."""
-    print(f"{Colors.BOLD}{Colors.WHITE}  ┌─────────────┬───────────┬──────────────────────────────────┬────────┬────────────┐{Colors.RESET}")
-    print(f"{Colors.BOLD}{Colors.WHITE}  │ Pair        │ Signal    │ Score                            │ Grade  │ Confidence │{Colors.RESET}")
-    print(f"{Colors.BOLD}{Colors.WHITE}  ├─────────────┼───────────┼──────────────────────────────────┼────────┼────────────┤{Colors.RESET}")
+# ── Live signal renderer ───────────────────────────────────────────
 
-    for r in reports:
-        sig = r.final_signal.name.replace("_", " ")
-        sc = signal_color(r.final_signal)
-        qc = quality_color(r.signal_quality)
-        bar = score_bar(r.final_score, 30)
-
-        pair = f"{r.display_name:<11}"
-        signal_str = f"{sc}{sig:<9}{Colors.RESET}"
-        grade_str = f"{qc}{r.signal_quality:<6}{Colors.RESET}"
-        conf_str = f"{r.final_confidence:5.1f}%"
-
-        # Add warning indicator
-        warn = ""
-        if r.event_risk == "danger":
-            warn = f" {Colors.RED}⚠{Colors.RESET}"
-        elif r.event_risk == "caution":
-            warn = f" {Colors.YELLOW}△{Colors.RESET}"
-
-        print(f"  │ {pair} │ {signal_str} │ {bar} │ {grade_str} │ {conf_str:>8}{warn}  │")
-
-    print(f"{Colors.BOLD}{Colors.WHITE}  └─────────────┴───────────┴──────────────────────────────────┴────────┴────────────┘{Colors.RESET}")
-
-    # Legend
-    print(f"\n{Colors.GRAY}  Legend: "
-          f"{Colors.GREEN}██ Bullish{Colors.GRAY} | "
-          f"{Colors.RED}██ Bearish{Colors.GRAY} | "
-          f"│ Center (0) | "
-          f"{Colors.RED}⚠ Event Risk{Colors.GRAY} | "
-          f"{Colors.YELLOW}△ Caution{Colors.RESET}")
+_INDICATOR_PREFIXES = (
+    "EMA Trend", "RSI", "Bollinger Bands", "Divergence",
+    "Candlestick", "ADX", "ATR", "News", "Regime",
+)
 
 
-def print_detailed_report(report: SignalReport):
-    """Print detailed analysis for a single pair."""
+def _compact_reason(reason: str) -> str:
+    """Shorten a verbose reason string for the compact view."""
+    s = reason
+    if s.startswith("STRONG: "):
+        s = s[8:]
+    # Strip the leading "<Indicator>: " prefix — it's always an indicator name
+    for prefix in _INDICATOR_PREFIXES:
+        if s.startswith(prefix + ": "):
+            s = s[len(prefix) + 2:]
+            break
+    # Drop trailing detail after first ' | '
+    if " | " in s:
+        s = s.split(" | ", 1)[0]
+    return s.strip()
+
+
+def _fmt_price(price: float) -> str:
+    """2dp for gold-scale prices, 5dp for sub-10 FX-scale prices."""
+    return f"{price:.2f}" if price >= 10 else f"{price:.5f}"
+
+
+def print_signal_report(report: SignalReport, detail: bool = False):
     sc = signal_color(report.final_signal)
     qc = quality_color(report.signal_quality)
     sig_name = report.final_signal.name.replace("_", " ")
 
-    print(f"\n{Colors.CYAN}{'─' * 80}{Colors.RESET}")
-    print(f"  {Colors.BOLD}{Colors.WHITE}{report.display_name}{Colors.RESET}"
-          f"  │  Price: {Colors.BOLD}{report.current_price:.5f}{Colors.RESET}"
-          f"  │  Regime: {Colors.CYAN}{report.market_regime}{Colors.RESET}"
-          f"  │  Vol: {report.volatility_regime}")
+    # ── 1. DECISION ────────────────────────────────────────────────
+    mtf_tag = (f"{Colors.GREEN}MTF✓{Colors.RESET}" if report.mtf_agreement
+               else f"{Colors.YELLOW}MTF✗{Colors.RESET}")
+    print(f"  {sc}▶ {sig_name}{Colors.RESET}  "
+          f"{qc}{report.signal_quality}{Colors.RESET} · "
+          f"{sc}{report.final_score:+.1f}{Colors.RESET} · "
+          f"{report.final_confidence:.0f}% · {mtf_tag}")
+    print(f"  {Colors.GRAY}{report.action_recommendation}{Colors.RESET}")
 
-    print(f"\n  {Colors.BOLD}SIGNAL:{Colors.RESET} {sc}{sig_name}{Colors.RESET}"
-          f"  │  Score: {sc}{report.final_score:+.1f}{Colors.RESET}"
-          f"  │  Grade: {qc}{report.signal_quality}{Colors.RESET}"
-          f"  │  Confidence: {report.final_confidence:.0f}%")
+    # ── 2. TRADE SETUP (if signal is actionable) ───────────────────
+    if report.trade_setup:
+        ts = report.trade_setup
+        dc = Colors.GREEN if ts.direction == "BUY" else Colors.RED
+        print(f"\n  {Colors.BOLD}Setup{Colors.RESET}  "
+              f"{dc}{ts.direction}{Colors.RESET} @ {Colors.BOLD}{_fmt_price(ts.entry_price)}{Colors.RESET}  "
+              f"→ SL {Colors.RED}{_fmt_price(ts.stop_loss)}{Colors.RESET} "
+              f"({ts.risk_pips:.1f}p)  "
+              f"→ TP {Colors.GREEN}{_fmt_price(ts.take_profit_1)}{Colors.RESET} "
+              f"({ts.reward_pips:.1f}p)  "
+              f"RR 1:{ts.risk_reward:.1f}")
 
-    print(f"  {Colors.BOLD}Action:{Colors.RESET} {report.action_recommendation}")
+    # ── 3. CONTEXT (price · regime · vol · MTF breakdown) ──────────
+    print(f"\n  {Colors.WHITE}{report.display_name}{Colors.RESET} "
+          f"{Colors.BOLD}{_fmt_price(report.current_price)}{Colors.RESET} · "
+          f"{Colors.CYAN}{report.market_regime}{Colors.RESET} · "
+          f"vol {report.volatility_regime}")
 
-    # Multi-timeframe breakdown
     if report.mtf_signals:
-        print(f"\n  {Colors.BOLD}Multi-Timeframe:{Colors.RESET}", end="")
-        mtf_agree = f" {Colors.GREEN}✓ ALIGNED{Colors.RESET}" if report.mtf_agreement else f" {Colors.YELLOW}✗ DIVERGENT{Colors.RESET}"
-        print(mtf_agree)
+        parts = []
         for tf, sig in report.mtf_signals.items():
             tf_sc = signal_color(sig)
-            tf_score = report.mtf_scores.get(tf, 0)
-            print(f"    {tf:>5}: {tf_sc}{sig.name.replace('_', ' '):<12}{Colors.RESET} ({tf_score:+.1f})")
+            sc_val = report.mtf_scores.get(tf, 0)
+            parts.append(f"{tf} {tf_sc}{sig.name.replace('_',' ')}{Colors.RESET} {sc_val:+.1f}")
+        print(f"  {Colors.GRAY}MTF{Colors.RESET}    " + "  ·  ".join(parts))
 
-    # Indicators detail
-    if report.primary_analysis:
-        print(f"\n  {Colors.BOLD}Indicators ({report.primary_analysis.timeframe}):{Colors.RESET}")
+    # ── 4. WHY — top 3 each side, compact ──────────────────────────
+    bulls = [_compact_reason(r) for r in report.bullish_reasons[:3]]
+    bears = [_compact_reason(r) for r in report.bearish_reasons[:3]]
+    if bulls or bears:
+        print()
+        rows = max(len(bulls), len(bears))
+        for i in range(rows):
+            left  = f"{Colors.GREEN}+{Colors.RESET} {bulls[i]}" if i < len(bulls) else ""
+            right = f"{Colors.RED}-{Colors.RESET} {bears[i]}" if i < len(bears) else ""
+            print(f"  {left:<60}  {right}")
+
+    # ── 5. WARNINGS / NEWS / EVENTS ────────────────────────────────
+    if report.warnings:
+        for w in report.warnings:
+            print(f"  {Colors.YELLOW}⚠{Colors.RESET}  {w}")
+    if report.event_description and report.event_risk != "clear":
+        print(f"  {Colors.YELLOW}📅{Colors.RESET} {report.event_description}")
+    if report.news_description and report.news_description != "Neutral news sentiment":
+        print(f"  {Colors.MAGENTA}📰{Colors.RESET} {report.news_description}")
+    if report.news_summary:
+        print(f"  {Colors.MAGENTA}🧠{Colors.RESET} {report.news_summary}")
+    if report.panel_verdict:
+        panel_label = ("AI macro view"
+                      if report.panel_verdict_source == "macro" else "AI Panel")
+        print(f"  {Colors.CYAN}🤖{Colors.RESET} {panel_label}: "
+              f"{Colors.BOLD}{report.panel_verdict}{Colors.RESET} "
+              f"({report.panel_confidence:.0f}%) — {report.panel_rationale}")
+        for t in report.panel_takes:
+            print(f"     {Colors.GRAY}{t['persona']:<11} {t['stance']:<8} "
+                  f"{t['note']}{Colors.RESET}")
+
+    # ── 6. DETAIL (only with --detail) ─────────────────────────────
+    if detail and report.primary_analysis:
+        print(f"\n  {Colors.BOLD}Indicators ({report.primary_analysis.timeframe}){Colors.RESET}")
         for name, ind in report.primary_analysis.indicators.items():
             if name.startswith("_") or ind.weight == 0:
                 continue
             isc = signal_color(ind.signal)
-            ind_name = f"{ind.name:<18}"
-            print(f"    {ind_name} {isc}{ind.signal.name.replace('_', ' '):<12}{Colors.RESET}"
-                  f" ({ind.confidence:.0f}%) │ {Colors.GRAY}{ind.description}{Colors.RESET}")
+            print(f"    {ind.name:<16} {isc}{ind.signal.name.replace('_',' '):<11}{Colors.RESET} "
+                  f"{ind.confidence:>3.0f}% {Colors.GRAY}│ {ind.description}{Colors.RESET}")
 
-    # Bullish reasons
-    if report.bullish_reasons:
-        print(f"\n  {Colors.GREEN}▲ Bullish Factors:{Colors.RESET}")
-        for reason in report.bullish_reasons[:5]:
-            print(f"    {Colors.GREEN}+{Colors.RESET} {reason}")
-
-    # Bearish reasons
-    if report.bearish_reasons:
-        print(f"\n  {Colors.RED}▼ Bearish Factors:{Colors.RESET}")
-        for reason in report.bearish_reasons[:5]:
-            print(f"    {Colors.RED}-{Colors.RESET} {reason}")
-
-    # Warnings
-    if report.warnings:
-        print(f"\n  {Colors.YELLOW}⚠ Warnings:{Colors.RESET}")
-        for warning in report.warnings:
-            print(f"    {Colors.YELLOW}!{Colors.RESET} {warning}")
-
-    # News context
-    if report.news_description and report.news_description != "Neutral news sentiment":
-        print(f"\n  {Colors.MAGENTA}📰 News:{Colors.RESET} {report.news_description}")
-    if report.event_description and report.event_risk != "clear":
-        print(f"  {Colors.YELLOW}📅 Event:{Colors.RESET} {report.event_description}")
-
-    # Trade setup
-    if report.trade_setup:
-        ts = report.trade_setup
-        dir_color = Colors.GREEN if ts.direction == "BUY" else Colors.RED
-        print(f"\n  {Colors.BOLD}Trade Setup:{Colors.RESET}")
-        print(f"    Direction:     {dir_color}{ts.direction}{Colors.RESET}")
-        print(f"    Entry:         {ts.entry_price:.5f}")
-        print(f"    Stop Loss:     {Colors.RED}{ts.stop_loss:.5f}{Colors.RESET} ({ts.risk_pips:.1f} pips)")
-        print(f"    Take Profit 1: {Colors.GREEN}{ts.take_profit_1:.5f}{Colors.RESET} ({ts.reward_pips:.1f} pips)")
-        print(f"    Take Profit 2: {Colors.GREEN}{ts.take_profit_2:.5f}{Colors.RESET}")
-        print(f"    Risk:Reward:   1:{ts.risk_reward:.1f}")
-
-    # Key levels
-    if report.primary_analysis and report.primary_analysis.key_levels:
         levels = report.primary_analysis.key_levels
-        print(f"\n  {Colors.BOLD}Key Levels:{Colors.RESET}")
-        level_names = ["r3", "r2", "r1", "pivot", "s1", "s2", "s3"]
-        for lvl in level_names:
-            if lvl in levels:
-                val = levels[lvl]
-                if val > report.current_price:
-                    color = Colors.RED  # resistance
-                else:
-                    color = Colors.GREEN  # support
-                label = lvl.upper()
-                print(f"    {label:>6}: {color}{val:.5f}{Colors.RESET}")
+        if levels:
+            swing_keys = sorted(
+                [k for k in levels if k.startswith("swing_")],
+                key=lambda k: levels[k], reverse=True,
+            )
+            if swing_keys:
+                print(f"\n  {Colors.BOLD}Structure Levels{Colors.RESET}")
+                for k in swing_keys:
+                    val = levels[k]
+                    color = Colors.RED if val > report.current_price else Colors.GREEN
+                    label = k.replace("_", " ").title()
+                    print(f"    {label:>20}  {color}{val:.3f}{Colors.RESET}")
 
 
-def print_event_calendar(reports: List[SignalReport]):
-    """Print upcoming economic events affecting watched pairs."""
-    events_shown = set()
-    has_events = False
-
-    for report in reports:
-        if report.event_risk != "clear" and report.event_description not in events_shown:
-            if not has_events:
-                print(f"\n{Colors.YELLOW}{Colors.BOLD}  📅 ECONOMIC CALENDAR ALERTS{Colors.RESET}")
-                has_events = True
-            events_shown.add(report.event_description)
-            risk_color = Colors.RED if report.event_risk == "danger" else Colors.YELLOW
-            print(f"    {risk_color}● {report.event_description}{Colors.RESET} → affects {report.display_name}")
-
-    if not has_events:
-        print(f"\n{Colors.GRAY}  📅 No high-impact events in the near term{Colors.RESET}")
+def print_market_outlook(outlook: Optional[OutlookResult]) -> None:
+    """Render the independent advisory outlook for explicit CLI use."""
+    print(f"\n  {Colors.BOLD}AI Market Outlook{Colors.RESET}")
+    if outlook is None:
+        print(f"  {Colors.GRAY}Unavailable (disabled, market closed, or no LLM response).{Colors.RESET}")
+        return
+    forming = {
+        "buy": "possible buy forming",
+        "sell": "possible sell forming",
+        "none": "no clear setup forming",
+    }[outlook.setup_forming]
+    print(f"  {outlook.bias.title()} · {forming} · {outlook.confidence:.0f}%")
+    if outlook.watch:
+        print(f"  Watch: {outlook.watch}")
+    if outlook.note:
+        print(f"  {Colors.GRAY}{outlook.note}{Colors.RESET}")
 
 
 def report_to_dict(report: SignalReport) -> dict:
-    """Convert a SignalReport to a JSON-serializable dict."""
     d = {
         "symbol": report.symbol,
         "display_name": report.display_name,
@@ -288,6 +237,15 @@ def report_to_dict(report: SignalReport) -> dict:
         "event_risk": report.event_risk,
         "event_description": report.event_description,
         "news_sentiment": round(report.news_sentiment, 3),
+        "news_description": report.news_description,
+        "news_summary": report.news_summary,
+        "panel_verdict": report.panel_verdict,
+        "panel_confidence": round(report.panel_confidence, 1),
+        "panel_rationale": report.panel_rationale,
+        "panel_takes": report.panel_takes,
+        "panel_created_at": report.panel_created_at,
+        "panel_cache_status": report.panel_cache_status,
+        "panel_payload_hash": report.panel_payload_hash,
         "bullish_reasons": report.bullish_reasons,
         "bearish_reasons": report.bearish_reasons,
         "warnings": report.warnings,
@@ -303,142 +261,150 @@ def report_to_dict(report: SignalReport) -> dict:
             "risk_reward": ts.risk_reward,
             "risk_pips": round(ts.risk_pips, 1),
             "reward_pips": round(ts.reward_pips, 1),
+            "strategy": ts.strategy,
+            "setup_pattern": ts.setup_pattern,
+            "round_trip_cost": ts.round_trip_cost,
+            "trail_arm_r": ts.trail_arm_r,
+            "trail_dist_r": ts.trail_dist_r,
         }
     if report.primary_analysis:
-        d["indicators"] = {}
-        for name, ind in report.primary_analysis.indicators.items():
-            if not name.startswith("_"):
-                d["indicators"][name] = {
-                    "signal": ind.signal.name,
-                    "confidence": round(ind.confidence, 1),
-                    "description": ind.description,
-                }
+        d["indicators"] = {
+            name: {
+                "signal": ind.signal.name,
+                "confidence": round(ind.confidence, 1),
+                "description": ind.description,
+            }
+            for name, ind in report.primary_analysis.indicators.items()
+            if not name.startswith("_") and ind.weight != 0
+        }
         d["key_levels"] = {
             k: round(v, 5) for k, v in report.primary_analysis.key_levels.items()
         }
     return d
 
 
+
 # ══════════════════════════════════════════════════════════════════
-#  MAIN
+#  CLI
 # ══════════════════════════════════════════════════════════════════
 
 def parse_args():
     parser = argparse.ArgumentParser(
-        description="Forex Signal Intelligence Platform",
+        description="ForexBot2 — XAU/USD Signal Intelligence",
         formatter_class=argparse.RawDescriptionHelpFormatter,
         epilog="""
 Examples:
-  python main.py                        Analyze all major pairs (H1)
-  python main.py --pairs EURUSD GBPUSD  Analyze specific pairs
-  python main.py --timeframe H4         Use 4-hour as primary TF
-  python main.py --all                  Include cross pairs
-  python main.py --json                 Output as JSON
-  python main.py --detail               Show detailed analysis per pair
-  python main.py --watch                Continuous monitoring (5min refresh)
+  python main.py                          Live signal (H1, summary)
+  python main.py --detail                 Live signal with full breakdown
+  python main.py --timeframe H4 --detail  H4 timeframe
+  python main.py --json                   JSON output
         """,
     )
-    parser.add_argument(
-        "--pairs", nargs="+", default=None,
-        help="Specific pairs to analyze (e.g., EURUSD GBPUSD)",
-    )
-    parser.add_argument(
-        "--timeframe", "-tf", default="H1",
-        choices=["M5", "M15", "M30", "H1", "H4", "D1"],
-        help="Primary analysis timeframe (default: H1)",
-    )
-    parser.add_argument(
-        "--all", action="store_true",
-        help="Include cross pairs in analysis",
-    )
-    parser.add_argument(
-        "--json", action="store_true",
-        help="Output results as JSON",
-    )
-    parser.add_argument(
-        "--detail", action="store_true",
-        help="Show detailed analysis for each pair",
-    )
-    parser.add_argument(
-        "--top", type=int, default=None,
-        help="Show only top N signals by strength",
-    )
-    parser.add_argument(
-        "--watch", action="store_true",
-        help="Continuous monitoring mode (refreshes every 5 min)",
-    )
-    parser.add_argument(
-        "--min-grade", default="F",
-        choices=["A+", "A", "B", "C", "D", "F"],
-        help="Minimum signal quality to display",
-    )
-    parser.add_argument(
-        "--verbose", "-v", action="store_true",
-        help="Enable debug logging",
-    )
+    parser.add_argument("--pair", "-p", default="XAUUSD",
+                        help="Pair to analyze: XAUUSD (default: XAUUSD)")
+    parser.add_argument("--timeframe", "-tf", default="H1",
+                        choices=["M5", "M15", "M30", "H1", "H4", "D1"],
+                        help="Primary analysis timeframe (default: H1)")
+    parser.add_argument("--detail", action="store_true",
+                        help="Show full indicator breakdown")
+    parser.add_argument("--panel", action="store_true",
+                        help="Run the LLM agent panel on the result "
+                             "(advisory only; needs a DeepSeek key)")
+    parser.add_argument("--outlook", action="store_true",
+                        help="Run the hourly AI market outlook now "
+                             "(advisory only; needs a DeepSeek key)")
+    parser.add_argument("--json", action="store_true",
+                        help="Output as JSON")
+    parser.add_argument("--verbose", "-v", action="store_true",
+                        help="Debug logging")
+
     return parser.parse_args()
 
 
-def symbol_from_name(name: str) -> str:
-    """Convert pair name like 'EURUSD' to Yahoo symbol 'EURUSD=X'."""
-    name = name.upper().replace("/", "").replace("_", "")
-    if not name.endswith("=X"):
-        name += "=X"
-    return name
+def _resolve_pair(s: str) -> str:
+    aliases = {
+        "XAUUSD": XAUUSD, "XAU/USD": XAUUSD, "GOLD": XAUUSD,
+    }
+    key = s.upper().strip()
+    if key in aliases:
+        return aliases[key]
+    raise SystemExit(f"Unknown pair '{s}'. Try one of: {', '.join(sorted(aliases))}")
 
 
-def run_analysis(args) -> List[SignalReport]:
-    """Run a single analysis cycle."""
-    config = PlatformConfig()
+def build_config(args) -> PlatformConfig:
+    pair = _resolve_pair(args.pair)
 
-    # Configure timeframe
     tf_map = {
         "M5": Timeframe.M5, "M15": Timeframe.M15, "M30": Timeframe.M30,
         "H1": Timeframe.H1, "H4": Timeframe.H4, "D1": Timeframe.D1,
     }
-    config.primary_timeframe = tf_map[args.timeframe]
+    primary_tf = tf_map[args.timeframe]
 
-    # Configure confirmation timeframes based on primary
     tf_confirmations = {
-        "M5": [Timeframe.M1, Timeframe.M15],
-        "M15": [Timeframe.M5, Timeframe.H1],
+        "M5":  [Timeframe.M1,  Timeframe.M15],
+        "M15": [Timeframe.M5,  Timeframe.H1],
         "M30": [Timeframe.M15, Timeframe.H1],
-        "H1": [Timeframe.M15, Timeframe.H4],
-        "H4": [Timeframe.H1, Timeframe.D1],
-        "D1": [Timeframe.H4, Timeframe.W1],
+        "H1":  [Timeframe.M15, Timeframe.H4],
+        "H4":  [Timeframe.H1,  Timeframe.D1],
+        "D1":  [Timeframe.H4,  Timeframe.W1],
     }
-    config.confirmation_timeframes = tf_confirmations.get(
+    confirmation_tfs = tf_confirmations.get(
         args.timeframe, [Timeframe.M15, Timeframe.H4]
     )
 
-    # Configure pairs
-    if args.pairs:
-        config.watch_pairs = [symbol_from_name(p) for p in args.pairs]
-    elif args.all:
-        config.watch_pairs = MAJOR_PAIRS + CROSS_PAIRS
-    else:
-        config.watch_pairs = MAJOR_PAIRS
+    # Per-pair tuned configs from the pair_signals factories.
+    factory = PAIR_PRODUCERS.get(pair)
+    if factory is not None:
+        return factory(primary_tf, confirmation_tfs)
 
-    # Run analysis
+    config = PlatformConfig()
+    config.primary_timeframe = primary_tf
+    config.confirmation_timeframes = confirmation_tfs
+    return config
+
+
+def run_live(args, config: PlatformConfig):
+    print(f"{Colors.GRAY}  Fetching market data and running analysis...{Colors.RESET}\n")
     aggregator = SignalAggregator(config)
-    reports = aggregator.analyze_all_pairs()
+    report = aggregator.analyze_pair(symbol=config.symbol, log_source="cli")
 
-    # Filter by grade
-    grade_order = ["A+", "A", "B", "C", "D", "F"]
-    min_idx = grade_order.index(args.min_grade) if args.min_grade in grade_order else len(grade_order) - 1
-    reports = [r for r in reports if grade_order.index(r.signal_quality) <= min_idx]
+    # Explicit user intent: force bypasses the enabled/surface/quality
+    # toggles, but the daily panel budget and LLM cap still apply.
+    if getattr(args, "panel", False):
+        attach_panel(report, surface="cli", force=True)
 
-    # Top N
-    if args.top:
-        reports = reports[:args.top]
+    outlook = None
+    if getattr(args, "outlook", False):
+        outlook = maybe_run_outlook(surface="cli", force=True)
 
-    return reports
+    if args.json:
+        payload = report_to_dict(report)
+        if getattr(args, "outlook", False):
+            payload["market_outlook"] = (
+                None if outlook is None else {
+                    "bias": outlook.bias,
+                    "setup_forming": outlook.setup_forming,
+                    "confidence": outlook.confidence,
+                    "watch": outlook.watch,
+                    "note": outlook.note,
+                    "created_ts": outlook.created_ts,
+                }
+            )
+        print(json.dumps(payload, indent=2))
+        return
+
+    print_header(f"{config.display_name} SIGNAL INTELLIGENCE")
+    print_signal_report(report, detail=args.detail)
+    if getattr(args, "outlook", False):
+        print_market_outlook(outlook)
+    print(f"\n{Colors.CYAN}{'═' * 80}{Colors.RESET}")
+    print(f"{Colors.GRAY}  Disclaimer: For educational/research purposes only.{Colors.RESET}")
+    print(f"{Colors.CYAN}{'═' * 80}{Colors.RESET}\n")
 
 
 def main():
     args = parse_args()
 
-    # Setup logging
     log_level = logging.DEBUG if args.verbose else logging.WARNING
     logging.basicConfig(
         level=log_level,
@@ -446,63 +412,8 @@ def main():
         datefmt="%H:%M:%S",
     )
 
-    if args.watch:
-        # Continuous monitoring mode
-        print(f"{Colors.BOLD}Starting continuous monitoring mode (Ctrl+C to stop)...{Colors.RESET}")
-        while True:
-            try:
-                reports = run_analysis(args)
-                if args.json:
-                    output = [report_to_dict(r) for r in reports]
-                    print(json.dumps(output, indent=2))
-                else:
-                    print("\033[2J\033[H")  # clear screen
-                    print_header()
-                    print_summary_table(reports)
-                    print_event_calendar(reports)
-                    if args.detail:
-                        for r in reports:
-                            print_detailed_report(r)
-                    print(f"\n{Colors.GRAY}  Next refresh in 5 minutes...{Colors.RESET}")
-
-                time.sleep(300)
-            except KeyboardInterrupt:
-                print(f"\n{Colors.YELLOW}Monitoring stopped.{Colors.RESET}")
-                break
-    else:
-        # Single run
-        print(f"{Colors.GRAY}  Fetching market data and running analysis...{Colors.RESET}\n")
-        reports = run_analysis(args)
-
-        if args.json:
-            output = [report_to_dict(r) for r in reports]
-            print(json.dumps(output, indent=2))
-        else:
-            print_header()
-            if reports:
-                print_summary_table(reports)
-                print_event_calendar(reports)
-
-                if args.detail:
-                    for r in reports:
-                        print_detailed_report(r)
-                else:
-                    # Show detail for top signals only
-                    actionable = [r for r in reports if r.signal_quality in ("A+", "A", "B")]
-                    if actionable:
-                        print(f"\n{Colors.BOLD}{Colors.WHITE}  ═══ ACTIONABLE SIGNALS (Grade B or above) ═══{Colors.RESET}")
-                        for r in actionable:
-                            print_detailed_report(r)
-                    else:
-                        print(f"\n{Colors.GRAY}  No high-grade signals detected. "
-                              f"Use --detail for full breakdown or --min-grade C to see more.{Colors.RESET}")
-            else:
-                print(f"{Colors.YELLOW}  No data available. Check your internet connection.{Colors.RESET}")
-
-            print(f"\n{Colors.CYAN}{'═' * 80}{Colors.RESET}")
-            print(f"{Colors.GRAY}  Disclaimer: This tool is for educational/research purposes only.{Colors.RESET}")
-            print(f"{Colors.GRAY}  Past performance ≠ future results. Always use proper risk management.{Colors.RESET}")
-            print(f"{Colors.CYAN}{'═' * 80}{Colors.RESET}\n")
+    config = build_config(args)
+    run_live(args, config)
 
 
 if __name__ == "__main__":

market_outlook.py

@@ -0,0 +1,442 @@
+"""
+ForexBot2 — AI market outlook (advisory hourly assessment)
+===========================================================
+Builds a broad XAU/USD market brief and asks DeepSeek whether a good setup
+appears to be forming. The result is written to the signal-history database
+and may be displayed by consumers, but it is ADVISORY-ONLY: it never changes
+signal scoring, gates, grades, or trade setup generation.
+
+The outlook runs outside the replayable signal path. Individual candle/news
+failures produce a partial brief; an unavailable LLM, malformed reply, or
+locked database never reaches a scan caller as an exception.
+
+The static prompt is intentional: DeepSeek prices byte-identical prompt
+prefixes at its cache-hit rate, so timestamps and all other changing evidence
+live only in the JSON user payload.
+"""
+
+import datetime as _dt
+import logging
+import math
+import os
+import sqlite3
+import threading
+import time
+from dataclasses import dataclass
+from typing import Any, Dict, List, Optional
+
+from agent_panel import _normalize_candles
+from app_config import (get_app_config, market_outlook_enabled,
+                        signal_log_path)
+from config import PlatformConfig, Timeframe, XAUUSD
+from engine import TechnicalEngine
+from llm_client import get_llm_client
+from providers import base as _provider_base
+
+logger = logging.getLogger(__name__)
+
+DEFAULT_MAX_PER_DAY = 12
+DEFAULT_CANDLE_BARS = 20
+_OUTLOOK_TFS = (Timeframe.M15, Timeframe.H1, Timeframe.H4)
+_VALID_BIASES = ("bullish", "bearish", "neutral")
+_VALID_SETUPS = ("buy", "sell", "none")
+
+
+# STATIC prompt: repeated prompt prefixes are much cheaper with DeepSeek's
+# server-side cache. Variable market evidence belongs exclusively in payload.
+SYSTEM_PROMPT = """\
+You are writing an advisory market outlook for gold (XAU/USD). You receive a
+JSON market brief covering recent candle shape, indicator states, price
+levels, news, and recent technical signals. Assess what may be forming soon;
+you are not a signal generator and must not provide an entry price, stop, or
+target.
+
+Gold doctrine: hawkish Fed / rate hikes / rising real yields / strong
+dollar are gold-bearish; easing / rate cuts / safe-haven flows /
+geopolitical stress / weak dollar are gold-bullish.
+
+Indicator "value" units: rsi 0-100 (above 70 overbought, below 30
+oversold); adx 0-100 (above 25 = strong trend); macd = histogram in
+dollars (momentum); roc = percent; bollinger = position 0-1 (0 = lower
+band, 1 = upper band); keltner = dollar distance from the mid-line;
+ema_trend = 0-1 alignment score; divergence and candlestick = pattern
+counts. "candles" is per timeframe: {"ref_close", "atr", "bars"}; bars
+are oldest-first [open, high, low, close] in ATR units relative to the
+latest close. Read their shape, never quote them as prices.
+
+Hard rules: use ONLY the JSON provided. Do not invent numbers, events, or
+price levels. Your training data about gold, markets, and current events is
+stale and OFF-LIMITS: never use outside knowledge; if a fact is not in the
+JSON, treat it as unknown. If evidence is thin or missing, answer "neutral"
+with confidence below 30. Use plain everyday English for someone unfamiliar
+with markets: short simple sentences, no acronyms or jargon. Keep "watch"
+and "note" to at most 25 words each.
+
+Reply with ONLY JSON:
+{"bias": "bullish"|"bearish"|"neutral",
+ "setup_forming": "buy"|"sell"|"none",
+ "confidence": 0-100,
+ "watch": "<what to watch for, max 25 words>",
+ "note": "<plain-English summary, max 25 words>"}
+"""
+
+
+@dataclass
+class OutlookResult:
+    bias: str
+    setup_forming: str
+    confidence: float
+    watch: str
+    note: str
+    created_ts: float
+
+
+def _finite(x: Any) -> Optional[float]:
+    """float(x) or None for non-numeric / non-finite JSON values."""
+    try:
+        value = float(x)
+    except (TypeError, ValueError):
+        return None
+    return value if math.isfinite(value) else None
+
+
+def _round_finite(x: Any, digits: int) -> Optional[float]:
+    value = _finite(x)
+    return round(value, digits) if value is not None else None
+
+
+def _json_safe(value: Any) -> Any:
+    """Replace any accidental NaN/inf nested in provider data with null."""
+    if isinstance(value, dict):
+        return {str(key): _json_safe(item) for key, item in value.items()}
+    if isinstance(value, (list, tuple)):
+        return [_json_safe(item) for item in value]
+    if isinstance(value, bool) or value is None or isinstance(value, str):
+        return value
+    if isinstance(value, (int, float)):
+        return _finite(value)
+    return value
+
+
+def _clamp_confidence(raw: Any) -> Optional[float]:
+    value = _finite(raw)
+    return max(0.0, min(100.0, value)) if value is not None else None
+
+
+def _word_limit(text: Any, words: int = 25) -> str:
+    """Bound a response field by words as well as a practical char limit."""
+    clean = str(text or "").strip()
+    return " ".join(clean.split()[:words])[:200]
+
+
+def _validate_outlook(reply: Any) -> Optional[OutlookResult]:
+    """Validate only model replies that fully match the requested contract."""
+    if not isinstance(reply, dict):
+        return None
+    bias = reply.get("bias")
+    setup_forming = reply.get("setup_forming")
+    confidence = _clamp_confidence(reply.get("confidence"))
+    if (bias not in _VALID_BIASES or setup_forming not in _VALID_SETUPS
+            or confidence is None):
+        return None
+    return OutlookResult(
+        bias=bias,
+        setup_forming=setup_forming,
+        confidence=confidence,
+        watch=_word_limit(reply.get("watch")),
+        note=_word_limit(reply.get("note")),
+        created_ts=time.time(),
+    )
+
+
+# ── Data collection ──────────────────────────────────────────────────
+
+_dp: Any = None  # Patchable singleton, mirroring agent_panel._dp.
+
+
+def _outlook_provider() -> Any:
+    """Lazy provider singleton so its closed-bar cache stays warm hourly."""
+    global _dp
+    if _dp is None:
+        from providers import make_data_provider
+        _dp = make_data_provider(PlatformConfig())
+    return _dp
+
+
+def _indicator_payload(analysis: Any) -> Dict[str, Dict[str, Any]]:
+    """Visible scored indicators with finite, compact values only."""
+    out: Dict[str, Dict[str, Any]] = {}
+    for name, indicator in (getattr(analysis, "indicators", None) or {}).items():
+        if name.startswith("_") or getattr(indicator, "weight", 0) == 0:
+            continue
+        entry: Dict[str, Any] = {
+            "signal": getattr(getattr(indicator, "signal", None), "name", ""),
+            "confidence": _round_finite(
+                getattr(indicator, "confidence", None), 1),
+            "value": _round_finite(getattr(indicator, "value", None), 2),
+        }
+        out[name] = entry
+    return out
+
+
+def _analysis_payload(analyses: Dict[str, Any]) -> Dict[str, Any]:
+    """Pure formatter for the technical portion of the market brief."""
+    indicators: Dict[str, Any] = {}
+    market_regime: Dict[str, Optional[str]] = {}
+    volatility_regime: Dict[str, Optional[str]] = {}
+    key_levels: Dict[str, Dict[str, Optional[float]]] = {}
+    for tf in _OUTLOOK_TFS:
+        name = tf.value
+        analysis = analyses.get(name)
+        indicators[name] = _indicator_payload(analysis)
+        market_regime[name] = getattr(analysis, "market_regime", None)
+        volatility_regime[name] = getattr(analysis, "volatility_regime", None)
+        key_levels[name] = {
+            level: _round_finite(value, 2)
+            for level, value in (getattr(analysis, "key_levels", None) or {}).items()
+        }
+    return {
+        "indicators": indicators,
+        "market_regime": market_regime,
+        "volatility_regime": volatility_regime,
+        "key_levels": key_levels,
+    }
+
+
+def _empty_news_payload() -> Dict[str, Any]:
+    return {
+        "headlines": None,
+        "events": None,
+        "gold_sentiment": None,
+        "remaining_high_impact_events": None,
+    }
+
+
+def _news_payload() -> Dict[str, Any]:
+    """Collect current news without allowing any news failure to block us."""
+    try:
+        from news_engine import NewsEngine
+
+        news = NewsEngine(PlatformConfig().news)
+        context = news.get_fundamental_context()
+        headlines = [str(getattr(h, "headline", ""))
+                     for h in (context.news_headlines or [])[:10]
+                     if getattr(h, "headline", None)]
+        events = [
+            {
+                "event": getattr(event, "event", None),
+                "currency": getattr(event, "currency", None),
+                "actual": getattr(event, "actual", None),
+                "forecast": getattr(event, "forecast", None),
+                "released": not event.is_upcoming,
+            }
+            for event in (context.events_today or [])
+            if event.is_high_impact
+        ][:8]
+        gold_sentiment, _ = news.get_pair_sentiment(XAUUSD, context)
+        remaining = sum(
+            1 for event in news.get_week_events()
+            if event.is_high_impact and event.is_upcoming
+        )
+        return {
+            "headlines": headlines,
+            "events": events,
+            "gold_sentiment": _round_finite(gold_sentiment, 2),
+            "remaining_high_impact_events": remaining,
+        }
+    except Exception as exc:
+        logger.debug("market outlook: news context unavailable: %s", exc)
+        return _empty_news_payload()
+
+
+def _recent_signals() -> List[Dict[str, Any]]:
+    """Read the latest signal history through SQLite's immutable URI."""
+    try:
+        path = os.path.abspath(signal_log_path())
+        # mode=ro (not immutable=1): immutable ignores the WAL, so the
+        # newest rows written by the live loggers were invisible here.
+        uri = f"file:{path}?mode=ro"
+        con = sqlite3.connect(uri, uri=True)
+        try:
+            rows = con.execute(
+                "SELECT logged_at, timeframe, signal, quality, panel_verdict "
+                "FROM signals ORDER BY id DESC LIMIT 10").fetchall()
+        finally:
+            con.close()
+        return [
+            {
+                "logged_at": row[0],
+                "timeframe": row[1],
+                "signal": row[2],
+                "quality": row[3],
+                "panel_verdict": row[4],
+            }
+            for row in reversed(rows)
+        ]
+    except Exception as exc:
+        logger.debug("market outlook: signal history unavailable: %s", exc)
+        return []
+
+
+def _build_payload(
+    analyses: Dict[str, Any],
+    candles: Dict[str, Any],
+    news: Dict[str, Any],
+    recent_signals: List[Dict[str, Any]],
+    as_of: _dt.datetime,
+) -> Dict[str, Any]:
+    """Pure market-brief formatter. Every numeric field is finite or null."""
+    technical = _analysis_payload(analyses)
+    return _json_safe({
+        "as_of": as_of.astimezone(_dt.timezone.utc).isoformat(),
+        "candles": candles,
+        **technical,
+        "news": news,
+        "recent_signals": recent_signals,
+    })
+
+
+def _collect_payload(settings: Dict[str, Any]) -> Dict[str, Any]:
+    """Fetch closed bars and turn them into a structured market brief."""
+    analyses: Dict[str, Any] = {}
+    candles: Dict[str, Any] = {tf.value: None for tf in _OUTLOOK_TFS}
+    bars = int(settings.get("candles_bars", DEFAULT_CANDLE_BARS))
+    provider = _outlook_provider()
+    engine = TechnicalEngine(PlatformConfig().technical)
+
+    for tf in _OUTLOOK_TFS:
+        try:
+            df = provider.fetch_ohlcv(XAUUSD, tf)
+            analysis = engine.analyze(df, XAUUSD, tf.value)
+            analyses[tf.value] = analysis
+            atr = _finite(getattr(
+                (getattr(analysis, "indicators", None) or {}).get("atr"),
+                "value", None))
+            if atr is not None and atr > 0 and df is not None and len(df) >= 2:
+                candles[tf.value] = _normalize_candles(df, atr, bars)
+        except Exception as exc:
+            logger.debug("market outlook: %s data unavailable: %s", tf.value, exc)
+
+    return _build_payload(
+        analyses=analyses,
+        candles=candles,
+        news=_news_payload(),
+        recent_signals=_recent_signals(),
+        as_of=_dt.datetime.now(_dt.timezone.utc),
+    )
+
+
+# ── Orchestration, hourly dedup, and request budget ──────────────────
+
+_state_lock = threading.Lock()
+_last_run_ts: Optional[float] = None
+_last_run_hour: Optional[int] = None   # UTC hour bucket of the last attempt
+_outlook_day: Optional[_dt.date] = None
+_outlook_count = 0
+_warned_budget = False
+
+
+def _skip(reason: str) -> None:
+    """Heartbeat for every silent early return — a dark outlook feature
+    (zero rows for 15 days, 2026-08/09) must be visible in the log."""
+    logger.info("market outlook: skipped (%s)", reason)
+
+
+def reset_outlook_state() -> None:
+    """Drop hourly and daily state (tests / deployment config reloads)."""
+    global _last_run_ts, _last_run_hour, _outlook_day, _outlook_count
+    global _warned_budget
+    with _state_lock:
+        _last_run_ts = None
+        _last_run_hour = None
+        _outlook_day = None
+        _outlook_count = 0
+        _warned_budget = False
+
+
+def _begin_attempt(cap: int, force: bool) -> bool:
+    """Atomically apply hourly dedup, then reserve a daily attempt.
+
+    Hourly dedup is by UTC hour BUCKET, not ``now - last < 3600``: the
+    H1 loop calls this at boundary+10 s+scan time, so consecutive hours
+    were routinely 3,590 s apart and every other hour was dropped."""
+    global _last_run_ts, _last_run_hour, _outlook_day, _outlook_count
+    global _warned_budget
+    now = time.time()
+    hour_bucket = int(now // 3600)
+    today = _dt.datetime.now(_dt.timezone.utc).date()
+    with _state_lock:
+        if not force and _last_run_hour == hour_bucket:
+            _skip(f"hour bucket {hour_bucket} already attempted")
+            return False
+        if _outlook_day != today:
+            _outlook_day = today
+            _outlook_count = 0
+            _warned_budget = False
+        if _outlook_count >= cap:
+            if not _warned_budget:
+                _warned_budget = True
+                logger.warning(
+                    "market outlook: daily budget (%d) exhausted — no more "
+                    "outlooks until UTC midnight", cap)
+            else:
+                _skip("daily budget exhausted")
+            return False
+        _outlook_count += 1
+        _last_run_ts = now
+        _last_run_hour = hour_bucket
+        return True
+
+
+def run_outlook(settings: Dict[str, Any],
+                surface: str = "live") -> Optional[OutlookResult]:
+    """Build, ask, validate, and persist one advisory outlook."""
+    try:
+        payload = _collect_payload(settings)
+        reply = get_llm_client().chat_json(SYSTEM_PROMPT, payload)
+        result = _validate_outlook(reply)
+        if result is None:
+            logger.warning(
+                "market outlook: no outlook produced (%s)",
+                "LLM returned nothing — cap/timeout/disabled" if reply is None
+                else "reply failed validation")
+            return None
+        try:
+            from signal_log import log_outlook
+            log_outlook(result, surface=surface, payload=payload)
+        except Exception:
+            logger.debug("market outlook: persistence unavailable", exc_info=True)
+        return result
+    except Exception:
+        logger.warning("market outlook: run failed", exc_info=True)
+        return None
+
+
+def maybe_run_outlook(surface: str, force: bool = False) -> Optional[OutlookResult]:
+    """Best-effort public entry point. It NEVER raises to a scan caller.
+
+    ``force`` is for explicit CLI use: it bypasses enabled, surface, and
+    hourly gates, but not the independent outlook budget or shared LLM cap.
+    """
+    try:
+        settings = get_app_config().market_outlook_settings
+        if not force:
+            if not market_outlook_enabled():
+                _skip("disabled")
+                return None
+            if surface == "telegram" and not settings.get("telegram", False):
+                _skip("telegram surface off")
+                return None
+        if _provider_base._is_forex_weekend():
+            _skip("weekend")
+            return None
+        cap = int(settings.get("max_per_day", DEFAULT_MAX_PER_DAY))
+        if cap < 1:
+            _skip("max_per_day < 1")
+            return None
+        if not _begin_attempt(cap, force):
+            return None
+        return run_outlook(settings, surface=surface)
+    except Exception:
+        logger.warning("market outlook: unavailable", exc_info=True)
+        return None

pair_signals/__init__.py

@@ -0,0 +1,51 @@
+"""
+Per-pair signal producers.
+
+Each pair module exposes:
+    make_config(primary_tf, confirmation_tfs) -> PlatformConfig
+
+To add a new pair: create pair_signals/<pair>.py with make_config(),
+then register it in this file.
+"""
+
+from typing import Optional, List, Dict, Callable
+
+from config import PlatformConfig, Timeframe
+from signal_aggregator import SignalAggregator, SignalReport
+from providers import BaseDataProvider
+
+PAIR_PRODUCERS: Dict[str, Callable] = {}
+
+
+def register_pair(symbol: str, config_factory: Callable):
+    PAIR_PRODUCERS[symbol] = config_factory
+
+
+def get_supported_pairs() -> List[str]:
+    return list(PAIR_PRODUCERS.keys())
+
+
+def produce_signal(
+    symbol: str,
+    primary_tf: Timeframe,
+    data_provider: BaseDataProvider,
+    confirmation_tfs: Optional[List[Timeframe]] = None,
+    log_source: str = "dashboard",
+) -> SignalReport:
+    config_factory = PAIR_PRODUCERS.get(symbol)
+    if config_factory is None:
+        raise ValueError(
+            f"No signal producer registered for {symbol}. "
+            f"Available: {list(PAIR_PRODUCERS.keys())}"
+        )
+
+    config = config_factory(primary_tf, confirmation_tfs)
+    aggregator = SignalAggregator(config)
+    aggregator.data_provider = data_provider
+    return aggregator.analyze_pair(symbol=symbol, log_source=log_source)
+
+
+from pair_signals import xauusd as _xau_mod
+from config import XAUUSD
+
+register_pair(XAUUSD, _xau_mod.make_config)

pair_signals/xauusd.py

@@ -0,0 +1,281 @@
+"""XAU/USD signal producer config.
+
+PlatformConfig defaults ARE the tuned XAU/USD configuration, so this
+factory sets the timeframes plus documented per-timeframe overrides.
+It must stay equivalent to the inline PlatformConfig(symbol=XAUUSD,
+...) construction (plus those overrides) used by the backtester and
+Telegram bot (enforced by a test in test_all.py).
+
+M15 override (validated 2026-07-11, window 2025-03-30 -> 2026-05-31,
+$0.30 spread, trailing exits): tp_floor_rr 1.0 scores +116.6R/460
+trades PF 1.83 vs +105.4R/424 PF 1.82 at the 0.5 default — better in
+both train/val halves and on the 2024-06 -> 2025-03 out-of-sample
+window (+30.3R vs +10.6R; the 1R floor lifts targets past the min-TP
+spread gate, roughly doubling trades there). The same knob HURTS M5
+(pinned 12mo book: +131.3R/699 vs +139.9R/547, PF 1.55 vs 1.78), so it
+is per-timeframe, exactly as PENDING_STEPS' decision rule anticipated.
+
+M15 override 2 (validated 2026-07-11, 12mo window 2025-07-11 ->
+2026-07-11 + OOS 2024-06-01 -> 2025-03-30, $0.30 spread): block the
+mixed "trending" regime (ADX strong but EMAs unaligned — momentum
+without directional structure). Those trades lost -6.8R/24 on 12mo,
+negative in BOTH halves; blocking scores +65.7R/499 PF 1.38 DD 10.4R
+vs +59.0R/521 PF 1.32 DD 12.6R baseline, better in both halves. OOS
+is neutral (-1.4R = the 4 such trades that existed there; no power to
+confirm or refute). M5 never trades this regime (its gates already
+exclude it), so the block is M15-only. A second candidate — blocking
+transitioning-regime SELLs (-4.0R/47 on 12mo) — was REJECTED: those
+sells made +5.7R/32 in the 2024 OOS window (bull-market artifact).
+
+M15 override 3 (validated 2026-07-16, 5-year replay 2021-07-17 ->
+2026-07-16 under the audit-fixed backtester, $0.30 spread, trailing
+exits): block SELL entries whose bar CLOSES at 15:00 or 17:00 UTC
+(sell_blocked_hours_utc=[15, 17], gate 6g3 — BUYs unaffected; those
+hours flank the already-blocked hour 16). The pocket lost -33.5R/165
+trades, negative in 6/6 calendar years and in BOTH halves of the last
+12 months (H1 -6.8R, H2 -4.2R); each hour independently is 5/6 years
+negative with both halves negative (hours 14/18 fail that screen and
+stay open). Cross-TF the M5 book CONFIRMS (same pocket -14.1R/59,
+5/6 years negative) rather than contradicts — unlike the rejected
+trending_down block. Full replay with the gate: +176.7R/1627 PF 1.31
+DD 29.1R vs +144.9R/1781 PF 1.22 DD 34.4R control; better in ALL SIX
+years; last-12mo halves both improve (H1 +36.3 vs +29.5, H2 +46.9 vs
++43.2; 12mo PF 1.55 vs 1.43); OOS 2024-06-01 -> 2025-03-30 improves
+(+45.4R PF 1.47 DD 17.8 vs +41.9R PF 1.39 DD 21.7). SELL side goes
+-1.2R -> +30.6R over 5 years. BUYs at those hours made +34.3R, hence
+SELL-only rather than a blocked_hours_utc entry. Mechanism: NY-session
+shorts against gold's macro uptrend (15:00 = post-US-data reversal
+window, 17:00 = London close unwind).
+
+M5 override 2 (2026-07-16, chart patterns): enable BUY-only CP on M5
+(inv_hs, double_bottom, bull_flag — drop all bearish patterns). SELL CP
+is consistently negative (OOS -3.21R, 5yr +3.76R PF 1.07) while BUY CP
+is positive across windows: 12mo +41.8R PF 1.60 (train +9.66 PF 1.30,
+val +32.17 PF 1.86), 5yr +65.2R PF 1.48, OOS +1.20R (aggregate
+positive, val half -1.89R — marginal but sign-correct in aggregate).
+Gold's structural uptrend makes bullish patterns fundamentally more
+reliable. M15/H1 fail on all windows; M5-only enablement.
+
+M5 override 3 (validated 2026-07-21, 5-year replay 2021-07-21 ->
+2026-07-21, $0.30 spread, next-open fills): trailing distance widened
+to 0.85R (trail_dist_r 0.85, arm unchanged 0.60). Same 30-cell grid
+methodology as the M15 campaign, opposite finding — on M5 the arm
+point barely matters; the winners need a looser leash. Plateau arm
+0.5-0.7 x dist 0.85-1.0 (+227..+234R vs +218.0R baseline). Chosen
+0.60/0.85 (OOS-preferred end of the plateau): +231.2R/2028 PF 1.30 WR
+57.1% vs +218.0R PF 1.29 WR 58.3%; both last-12mo halves better
+(+56.8/+115.1 vs +54.0/+104.6); 5yr max DD 19.2R vs 19.1R (noise);
+OOS 2024-06-01 -> 2025-03-30 +14.5R DD 15.1 vs +11.1R DD 15.4.
+dist 1.0 was REJECTED at the OOS gate (+10.9R < baseline +11.1R)
+despite the best in-sample total (+233.3R) — the OOS ordering picks
+the conservative plateau edge. XAU_PB trades are unaffected (they
+carry their own validated pb_trail 0.8/0.75 per-trade overrides);
+XAU_CP trades share the global params and were validated inside this
+book. M15 arms at 0.70 (its own campaign, same day); the divergence
+is the per-TF pattern yet again.
+
+M15 override 4 (validated 2026-07-21, 5-year replay 2021-07-21 ->
+2026-07-21, $0.30 spread, next-open fills): trailing-stop arm raised to
+0.70R (trail_arm_r 0.70, dist unchanged 0.75). Full grid arm 0.3-0.9 x
+dist 0.5-1.0 (30 cells): every earlier-arm cell (0.3/0.45) is WORSE —
+the "arm earlier to save fading trades" hypothesis is refuted; the
+plateau is arm 0.70-0.80 x dist 0.70-0.85 (all >= +190R). Chosen
+plateau center 0.70/0.75: +210.7R/1629 PF 1.35 WR 57.8% vs baseline
+0.60/0.75 +175.7R PF 1.30 WR 53.9%; both last-12mo halves improve
+(+49.7/+50.0 vs +36.4/+46.0); per-year better 4/6 (2022 -0.9R, 2023
+-1.4R, both within tolerance); OOS 2024-06-01 -> 2025-03-30 +57.4R
+PF 1.57 DD 17.0R vs +45.4R PF 1.47 DD 17.8R — better on every OOS
+metric. One clause waived BY OWNER DECISION 2026-07-21: 5yr max DD is
+33.9R vs 29.1R baseline (+4.8R deeper worst episode, 2022-23 chop),
+accepted in exchange for +20% total R. M5 keeps the global 0.6 arm
+(same per-TF divergence pattern as tp_floor_rr and the hour-9 block).
+
+M5 override (validated 2026-07-11, full-history replay 2020-03-17 ->
+2026-07-11 + 12mo book, $0.30 spread): block hour 9 UTC (post-London-
+open whipsaw). Hour-9 entries lost -28.9R/143 over 6 years, negative
+in 5/7 calendar years including winning years (2022 -9.2R). Blocking:
++207.1R vs +178.3R baseline on the 6-year replay, max DD 28.7R vs
+35.5R, improves 5/7 years (2023 goes -13.0R -> breakeven); the tuned
+12mo book is unchanged (+139.0R vs +139.3R). M15-and-up keep hour 9:
+it is POSITIVE there (M15 6yr +1.7R, M15 OOS +8.6R), so the block is
+M5-only, same per-timeframe pattern as the M15 overrides.
+
+M5 override 2 — relaxation campaign (validated 2026-07-30, Oanda
+bars, $0.30 spread, trailing exits): UNBLOCK hour 21 UTC on M5 only,
+combined with the config.py M5 gate relaxations (m5_min_abs_score
+30->25, m5_min_confirmation_score 10->5). The combined config beat
+the pre-relaxation control in ALL 6 calendar years of the 5yr replay
+2021-07-30 -> 2026-07-30 (+230.9R/2319 PF 1.26 DD 21.3 vs
++168.0R/2048 PF 1.21 DD 27.0), on 12mo (+107.4R PF 1.36, halves
++68.6/+38.8 vs +84.6R PF 1.33), on OOS 2024-06-01 -> 2025-03-30
+(+26.1R vs +17.2R), and at $0.70 spread (+86.9R vs +66.4R). Hour-21
+trades alone: +28.0R/22, positive every year — overturning the old
+block per the full-evidence rule. Hour 21 stays BLOCKED on M15+
+(12mo M15 unblock tested -0.3R). Declined-but-open from the same
+campaign: an M5 hour-22 block (+4.8R on 12mo — the burst-heavy
+"institutional" hour — but -3.2R on 5yr and -1.1R on OOS: regime-
+specific); M15 hour-16 unblock (+0.8R/+57 trades 12mo but -18.3R
+DD +34% on 5yr); every SELL-gate/vol/TP-size relaxation retested
+negative (see memory notes for the full grid).
+
+M5 override 4 — XAU_PB disabled (validated 2026-08-04, Oanda bars,
+$0.65 REAL spread — first campaign validated at measured cost on the
+live feed, per the 2026-08-04 re-pin baseline): m5_pullback_enabled
+False on M5. Loss forensics on the Oanda books identified XAU_PB as
+the largest wrong-direction loss source (12mo: -43.5R of the -156.6R
+wrong-direction bucket; 5yr: -274.5R). Disabling: 12mo +96.6R/588
+PF 1.44 max DD 13.9R halves +51.2/+45.4 vs +82.3R/788 PF 1.26 DD
+23.9R shipped (which FAILS the PF>=1.3 clause at real cost); OOS
+2024-06-01 -> 2025-03-30 +9.5R PF 1.34; 5yr +123.9R/956 PF 1.35 vs
+PF 1.16 with PB. PB's 5yr net contribution (+23.6R) was almost
+entirely the 2022 chop year (+36.1R vs -2.2R without) — a regime
+earner that has been a drag on trend-regime bars since; its tight
+stops are also the most cost-punished at $0.65. PB code and the
+pb_trail_* params STAY for future regime-gated experiments (same
+pattern as cp_enabled). Rejected in the same campaign (4 candidates
+total, N logged): hour-19 block (+3.7R 12mo but -3.3R OOS, -1.5R
+5yr — OOS gate, the classic trap); hours-14/15 block (-6.9R 12mo
+once PB is removed — their raw negativity WAS the PB losses);
+trail_dist_r 0.75 (-2.6R 12mo; the 0.88R average give-back is the
+validated price of riding trends, exits confirmed NOT the loss
+driver: exit-management accounts for 1-2% of lost R vs 50%
+wrong-direction / 49% entry-timing).
+
+M5 override 6 — SELL block in transitioning regime (validated
+2026-09-03, signal-log campaign, `backtest/research/
+signal_log_campaign_2026-09.md`, $0.65 round trip, same CPU harness
+as its control): sell_blocked_regimes=["transitioning"]. Live
+evidence: M5 SELLs in transitioning were -7.9R/12mo PF 0.21 and
+-5.2R/5yr PF 0.49 in the pinned books; the live signal log showed the
+same pocket -7.9R/PF 0.56 over 3 weeks. Blocking: 12mo (2025-08-14 ->
+2026-08-14) 557 trades +112.21R PF 1.56 (halves +55.9/+56.4, PF
+1.54/1.58) vs control 575/+104.34R PF 1.50; 5yr (2021-08-01 ->
+2026-08-04) 889/+133.7R PF 1.41 vs 911/+128.4R PF 1.39; OOS
+2024-06-01 -> 2025-03-30 unchanged (86/+9.45R PF 1.33 — no such
+trades in that window). Removed trades: 18 in 12mo (-7.9R), 22 in
+5yr (-5.3R).
+
+H1 override — all SELL entries blocked (validated 2026-09-03, same
+campaign): sell_min_quality=[] (gate 6e in signal_aggregator.py
+downgrades any actionable-grade SELL whose quality is not in this
+list to D, so an empty list downgrades every H1 short). H1 shorts
+were a structural loser: -16.7R over the 5yr control window and
+-11.5R OOS. Blocking: 12mo (2025-08-04 -> 2026-08-04) 91/+22.2R
+PF 1.79 (halves +20.6/+1.6) vs control 138/+24.0R PF 1.55 (12mo cost
+-1.8R); 5yr 335/+49.1R PF 1.46 vs 515/+32.4R PF 1.18; OOS 78/+16.7R
+PF 1.74 vs 104/+5.1R PF 1.13. New reference books:
+backtest/oanda_h1_{12mo,5yr,oos}_065_20260903.csv,
+backtest/oanda_m5_12mo_065_nopb_20260903.csv,
+backtest/oanda_m5_5yr_065_nopb_20260903.csv.
+
+Rejected/parked in the same campaign (see the memo for the full
+table): M15 sell_min_quality=["B"] (5yr -8.9R, max DD 35.8 -> 41.4R
+— REJECTED); Bollinger weight 0 in trending regimes on M5, M15, H1
+(M5 -20.4R/12mo, -15.5R/5yr; M15 -24.0R/12mo — REJECTED across the
+board); divergence disabled on M15 (R improves +7.4/+11.9/+13.5R
+across windows but 5yr max DD 35.8 -> 47.4R with PF flat — DECLINED,
+owner call); divergence disabled on H1 (+4.6/+8.1/+2.4R, PF
+1.62/1.20/1.17) — PARKED: it passes rule 1 in combination with the
+H1 no-short override (b12_h1_nosell_nodiv: 12mo 100/+23.1R PF 1.75,
+halves +21.8/+1.3; 5yr 364/+58.7R PF 1.52; OOS 88/+21.9R PF 1.92 —
+better than the no-shorts override alone on both 5yr and OOS), but
+shipping it would route through technical.disabled_indicators, which
+CLAUDE.md rule 8 reserves for experiments; owner decision, recommended
+vehicle is an H1-only regime_weights_override with divergence weight
+0 plus a fresh validation run.
+
+M5 override 5 — EMA partial-alignment squash (validated 2026-08-04,
+Oanda bars, $0.65 real spread): ema_partial_neutral True on M5 —
+partial 4-EMA ribbon alignment (2-3 of 4 checks) scores NEUTRAL
+instead of a weak BUY/SELL. Motivated by the indicator audit: partial-
+alignment trades averaged -0.008R/84 vs +0.230R/341 fully aligned.
+Replay (knob mirrored in the backtester's precomputed EMA scorer in
+the same commit — live==backtest parity): 12mo +97.6R/556 PF 1.48
+DD 10.7R halves +47.8/+49.8 vs control +96.6R/588 PF 1.44 DD 13.9R;
+5yr +127.4R/912 PF 1.38 DD 12.8R vs +123.9R/956 PF 1.35 DD 13.2R,
+2022 improves -2.2 -> -1.1; OOS 2024-06-01 -> 2025-03-30 +9.4R
+PF 1.33 vs +9.5R PF 1.34 — a one-trade tie, documented as such (the
+OOS window holds up; it neither prefers nor punishes). Net effect:
+32 fewer marginal trades, every risk-adjusted measure better.
+Rejected in the same round (trial-logged): confirmation-TF swap
+M5->[M15,H4] (12mo +104.0R PF 1.51 DD 10.3 but OOS +7.8R PF 1.27
+vs control +9.5R PF 1.34 and 2022 3x worse — the H4 confirm keeps
+endorsing stale direction in chop; OOS-ordering rejection, same
+precedent as trail dist 1.0); confirm [H1,H4] (-31.8R 12mo);
+triple-confirm [M15,H1,H4] (-4.9R); divergence weight -1/3
+(-3.8R 12mo); candlestick weight +50% (-3.0R 12mo). The weight
+vector keeps proving calibrated as a system: divergence/candlestick/
+bollinger all look mis-weighted in-book and all replay worse when
+"fixed" (bollinger weight-0: +80.9R PF 1.29 DD 23.8 — its constant
+oppose vote is a load-bearing quality bar, like ROC).
+"""
+
+from typing import Optional, List
+from config import PlatformConfig, Timeframe
+from app_config import apply_pair_overrides
+
+_CONFIRMATIONS = {
+    Timeframe.M5:  [Timeframe.M15, Timeframe.H1],
+    Timeframe.M15: [Timeframe.M5, Timeframe.H1],
+    Timeframe.H1:  [Timeframe.M15, Timeframe.H4],
+    Timeframe.H4:  [Timeframe.H1, Timeframe.D1],
+}
+
+
+def make_config(
+    primary_tf: Timeframe,
+    confirmation_tfs: Optional[List[Timeframe]] = None,
+) -> PlatformConfig:
+    config = PlatformConfig()
+    # Deployment overlay (config.json) BEFORE the validated per-TF
+    # deltas below, so e.g. the M5 hour-9 append lands on the JSON base.
+    apply_pair_overrides(config)
+    config.primary_timeframe = primary_tf
+    config.confirmation_timeframes = (
+        confirmation_tfs or _CONFIRMATIONS.get(primary_tf, [Timeframe.M15, Timeframe.H4])
+    )
+    if primary_tf == Timeframe.M5:
+        # Block hour 9 UTC on M5 only; UNBLOCK hour 21 on M5 only
+        # (positive every year of the 5yr replay) — see docstring.
+        config.xau_strategy.blocked_hours_utc = [
+            h for h in config.xau_strategy.blocked_hours_utc if h != 21
+        ] + [9]
+        # Deliberately NOT mirrored into config.backtest.blocked_hours_utc:
+        # that copy is the generic non-XAU fallback carrying the JSON base
+        # only. Propagating a gold-validated per-TF delta there would leak
+        # XAU-calibrated hours to a future pair (pair-isolation rule).
+        # Every XAU consumer branches to the xau_strategy copy. Enforced
+        # by the Module 13 test "M5 factory appends hour 9 on top of a
+        # JSON base".
+        # BUY-only chart patterns on M5 — see module docstring.
+        config.xau_strategy.cp_enabled = True
+        config.xau_strategy.cp_patterns = ["inv_hs", "double_bottom", "bull_flag"]
+        # XAU_PB pullback entries OFF — largest wrong-direction loss
+        # source at real cost on Oanda bars — see module docstring.
+        config.xau_strategy.m5_pullback_enabled = False
+        # Partial EMA-ribbon alignment scores NEUTRAL on M5 — its
+        # trades carried no edge — see module docstring.
+        config.technical.ema_partial_neutral = True
+        # Wider trailing distance on M5 only — see module docstring.
+        config.signals.trail_arm_r = 0.60
+        config.signals.trail_dist_r = 0.85
+        # Block SELL entries while the regime is "transitioning" on M5
+        # only — see module docstring (M5 override 6).
+        config.xau_strategy.sell_blocked_regimes = ["transitioning"]
+    if primary_tf == Timeframe.M15:
+        # 1R minimum target on M15 only — see module docstring.
+        config.signals.tp_floor_rr = 1.0
+        # Block the mixed "trending" regime on M15 only — see docstring.
+        config.xau_strategy.blocked_regimes = ["trending"]
+        # Block SELL entries at 15:00/17:00 UTC on M15 only — see docstring.
+        config.xau_strategy.sell_blocked_hours_utc = [15, 17]
+        # Later trailing-stop arm on M15 only — see module docstring.
+        config.signals.trail_arm_r = 0.70
+        config.signals.trail_dist_r = 0.75
+    if primary_tf == Timeframe.H1:
+        # Block every H1 SELL — the SELL quality gate (6e in
+        # signal_aggregator.py) downgrades any actionable-grade short
+        # to D when its grade is not in this list; an empty list means
+        # no grade qualifies. H1-only — see module docstring.
+        config.xau_strategy.sell_min_quality = []
+    return config

providers/__init__.py

@@ -0,0 +1,57 @@
+"""
+ForexBot2 — Data-provider package
+==================================
+One class per provider, all subclassing `BaseDataProvider` (which owns
+caching, validation, boundary guarantees, and range pagination — the
+behavior the signal pipeline depends on).
+
+Adding a provider = one module implementing `_fetch_raw` + one registry
+entry + one config.json `data_providers` block. The live candle feed is
+selected by config.json `data_providers.active`; the optional
+`data_providers.streaming` scalar selects a different provider for the
+tick stream (level-breach monitor). Registration alone changes nothing.
+"""
+
+from typing import Dict, Optional
+
+from app_config import AppConfigError, get_app_config
+from config import PlatformConfig
+
+from providers.base import BaseDataProvider, DataIntegrityError, _is_forex_weekend
+from providers.twelve_data import TwelveDataProvider
+from providers.oanda import OandaDataError, OandaDataProvider
+from providers.tiingo import TiingoDataError, TiingoDataProvider
+
+#: config.json data_providers.active selects from here.
+PROVIDER_REGISTRY: Dict[str, type] = {
+    "twelve_data": TwelveDataProvider,
+    "oanda": OandaDataProvider,
+    "tiingo": TiingoDataProvider,
+}
+
+
+def make_data_provider(config: PlatformConfig,
+                       name: Optional[str] = None) -> BaseDataProvider:
+    """Build a provider by name (default: config.json data_providers.active)."""
+    if name is None:
+        name = get_app_config().active_provider
+    cls = PROVIDER_REGISTRY.get(name)
+    if cls is None:
+        raise AppConfigError(
+            f'Unknown data provider "{name}"; '
+            f"registered: {sorted(PROVIDER_REGISTRY)}")
+    return cls(config)
+
+
+__all__ = [
+    "BaseDataProvider",
+    "DataIntegrityError",
+    "OandaDataError",
+    "OandaDataProvider",
+    "TiingoDataError",
+    "TiingoDataProvider",
+    "TwelveDataProvider",
+    "PROVIDER_REGISTRY",
+    "make_data_provider",
+    "_is_forex_weekend",
+]

providers/base.py

@@ -0,0 +1,532 @@
+"""
+ForexBot2 — Provider base class
+================================
+`BaseDataProvider` owns everything that must behave identically no matter
+where the bars come from: in-memory caching with candle-duration TTL, the
+closed-bar boundary guarantee, freshness checks, validation/cleaning, the
+forming-candle drop, and paginated date-range fetching for the backtester.
+
+A concrete provider implements only:
+    _fetch_raw(symbol, timeframe, outputsize, start, end) -> pd.DataFrame
+returning a normalized frame (capitalized OHLCV columns, UTC tz-aware
+DatetimeIndex named "Datetime", oldest-first), plus optionally
+`get_spread` and the streaming-quote capability (`supports_streaming` +
+`iter_ticks`).
+
+The signal pipeline (engine + aggregator + backtester) consumes ONLY the
+public contract defined here, so swapping providers never touches signal
+code.
+"""
+
+import logging
+import threading
+import time
+from abc import ABC, abstractmethod
+from datetime import datetime, timedelta
+from typing import AsyncIterator, Callable, Dict, Optional, Sequence, Tuple
+from zoneinfo import ZoneInfo
+
+import pandas as pd
+
+from config import PAIR_INFO, PlatformConfig, Timeframe, XAUUSD
+
+logger = logging.getLogger(__name__)
+
+# The gold/FX market week is defined in New York time (see
+# _is_forex_weekend); a fixed UTC hour is wrong for half the year.
+_NY_TZ = ZoneInfo("America/New_York")
+
+
+def _is_forex_weekend(now=None) -> bool:
+    """Gold/FX market closed for the weekend.
+
+    The market week is anchored to NEW YORK time — Friday 17:00 ET →
+    Sunday 18:00 ET — not a fixed UTC hour. During US DST the feed
+    actually stops at 21:00 UTC (measured on 5 years of Oanda bars:
+    159/162 summer Fridays; winter close 22:00, winter reopen Sunday
+    23:00 UTC). The old hardcoded Fri 22:00→Sun 22:00 UTC rule produced
+    two false-"STALE DATA/feed lagging" hours every week (2026-08-04
+    audit I43, fixed per owner decision).
+
+    Kept as three deliberate copies (here, signal_aggregator.py,
+    gpu_backtest._is_xau_blocked_by_calendar) — a Module 21 test asserts
+    they agree hour-by-hour across DST transitions.
+    """
+    if now is None:
+        now = pd.Timestamp.now(tz="UTC")
+    ny = now.astimezone(_NY_TZ)
+    wd = ny.weekday()
+    if wd == 4:                      # Friday
+        return ny.hour >= 17
+    if wd == 5:                      # Saturday
+        return True
+    if wd == 6:                      # Sunday
+        return ny.hour < 18
+    return False
+
+
+class DataIntegrityError(Exception):
+    """Raised when data fails validation checks."""
+    pass
+
+
+class BaseDataProvider(ABC):
+    """Shared OHLCV provider behavior; subclasses supply the raw fetch."""
+
+    #: Registry name; overridden per provider.
+    name: str = "base"
+
+    #: Exception type this provider raises for fetch/validation failures.
+    #: Provider-specific errors subclass DataIntegrityError so consumers
+    #: can catch one type regardless of the active provider.
+    DATA_ERROR = DataIntegrityError
+
+    BARS_PER_DAY = {
+        Timeframe.M1: 1440, Timeframe.M5: 288, Timeframe.M15: 96,
+        Timeframe.M30: 48, Timeframe.H1: 24, Timeframe.H4: 6,
+        Timeframe.D1: 1, Timeframe.W1: 0.143,
+    }
+
+    CANDLE_SECONDS = {
+        Timeframe.M1: 60, Timeframe.M5: 300, Timeframe.M15: 900,
+        Timeframe.M30: 1800, Timeframe.H1: 3600, Timeframe.H4: 14400,
+        Timeframe.D1: 86400, Timeframe.W1: 604800,
+    }
+
+    #: Hard per-request bar cap shared by Twelve Data and Oanda.
+    MAX_BARS_PER_REQUEST = 5000
+
+    # Grace wait before the one retry when a fresh response at a candle
+    # boundary is missing the just-closed bar (feed latency).
+    FRESH_FETCH_RETRY_SECONDS = 3.0
+
+    def __init__(self, config: PlatformConfig):
+        self.config = config
+        self._cache: Dict[str, pd.DataFrame] = {}
+        self._cache_timestamps: Dict[str, float] = {}
+        # Requested fetch depth per key: a cached short frame must not
+        # satisfy a caller asking for a longer lookback.
+        self._cache_outputsize: Dict[str, int] = {}
+        # The bot shares one provider across a thread pool; cache eviction
+        # is check-then-act and needs locking.
+        self._cache_lock = threading.Lock()
+        # Single-flight: one API fetch per (symbol, TF) at a time; other
+        # threads wait and then read the cache.
+        self._inflight: Dict[str, threading.Event] = {}
+
+    # ── Abstract: the only provider-specific piece ─────────────────
+
+    @abstractmethod
+    def _fetch_raw(
+        self,
+        symbol: str,
+        timeframe: Timeframe,
+        outputsize: int,
+        start: Optional[datetime] = None,
+        end: Optional[datetime] = None,
+    ) -> pd.DataFrame:
+        """Fetch up to `outputsize` bars, optionally bounded by start/end
+        (tz-aware UTC datetimes). Must return a normalized frame:
+        capitalized OHLCV columns, UTC DatetimeIndex named "Datetime",
+        oldest-first. Raise self.DATA_ERROR on failure."""
+
+    # ── Public API ─────────────────────────────────────────────────
+
+    def fetch_ohlcv(
+        self,
+        symbol: str = XAUUSD,
+        timeframe: Timeframe = Timeframe.H1,
+        lookback_days: Optional[int] = None,
+    ) -> pd.DataFrame:
+        """Fetch OHLCV with in-memory caching for live mode."""
+        if lookback_days is None:
+            lookback_days = self.config.data_lookback_days
+
+        cache_key = f"{symbol}_{timeframe.value}"
+        outputsize = max(
+            300,
+            min(int(lookback_days * self.BARS_PER_DAY[timeframe]) + 20,
+                self.MAX_BARS_PER_REQUEST),
+        )
+
+        while True:
+            now = time.time()
+            cache_ttl = self.CANDLE_SECONDS.get(
+                timeframe, self.config.refresh_interval)
+            with self._cache_lock:
+                cached = self._cache.get(cache_key)
+                cached_at = self._cache_timestamps.get(cache_key)
+                cached_size = self._cache_outputsize.get(cache_key, 0)
+            if (
+                cached is not None
+                and cached_at is not None
+                and (now - cached_at) < cache_ttl
+                and not cached.empty
+                # A caller asking for MORE history than the cached fetch
+                # requested must not be served the short frame silently.
+                and cached_size >= outputsize
+            ):
+                if (self._is_data_fresh(cached, symbol, timeframe)
+                        and self._has_latest_closed_bar(cached, timeframe)):
+                    return cached
+                logger.warning(f"{symbol} {timeframe.value}: cached data is stale, refetching")
+                with self._cache_lock:
+                    self._cache.pop(cache_key, None)
+                    self._cache_timestamps.pop(cache_key, None)
+                    self._cache_outputsize.pop(cache_key, None)
+
+            # Single-flight: concurrent scans for the same (symbol, TF)
+            # must not each hit the API (the four TF loops fire together
+            # at aligned boundaries and Oanda has no client throttle).
+            # First caller claims the key and fetches; the rest wait,
+            # then re-check the cache.
+            with self._cache_lock:
+                inflight = self._inflight.get(cache_key)
+                if inflight is None:
+                    self._inflight[cache_key] = threading.Event()
+                    break
+            inflight.wait(timeout=90)
+
+        try:
+            df = self._fetch_raw(symbol, timeframe, outputsize)
+            df = self._drop_forming_candle(df, timeframe)
+            df = self._validate_and_clean(df, symbol, timeframe)
+
+            # The closed-bar guarantee must hold for FRESH responses too,
+            # not just cache hits: the scanner fires seconds after the
+            # boundary, and a lagging feed can answer without the
+            # just-closed bar — the scan would silently analyze (and
+            # session-check) the previous bar. Give the feed one short
+            # grace retry, then proceed loudly.
+            if not df.empty and not self._has_latest_closed_bar(df, timeframe):
+                time.sleep(self.FRESH_FETCH_RETRY_SECONDS)
+                try:
+                    retry = self._fetch_raw(symbol, timeframe, outputsize)
+                    retry = self._drop_forming_candle(retry, timeframe)
+                    retry = self._validate_and_clean(retry, symbol, timeframe)
+                    if retry is not None and not retry.empty:
+                        df = retry
+                except DataIntegrityError as e:
+                    logger.warning(
+                        f"{symbol} {timeframe.value}: boundary retry refetch "
+                        f"failed ({e}) — keeping original data")
+                if not self._has_latest_closed_bar(df, timeframe):
+                    logger.warning(
+                        f"{symbol} {timeframe.value}: feed lagging — just-closed bar "
+                        f"still missing after retry; this scan may run one bar behind"
+                    )
+
+            if not self._is_data_fresh(df, symbol, timeframe):
+                logger.warning(
+                    f"STALE DATA from API: {symbol} {timeframe.value} — "
+                    f"last bar {df.index[-1]} is too old. Signal quality may be degraded."
+                )
+
+            with self._cache_lock:
+                self._cache[cache_key] = df
+                self._cache_timestamps[cache_key] = now
+                self._cache_outputsize[cache_key] = outputsize
+            return df
+        finally:
+            # Release waiters even when the fetch raised — they re-check
+            # the cache and (on a miss) claim the key themselves.
+            with self._cache_lock:
+                ev = self._inflight.pop(cache_key, None)
+            if ev is not None:
+                ev.set()
+
+    def fetch_range(
+        self,
+        symbol: str,
+        timeframe: Timeframe,
+        start: datetime,
+        end: datetime,
+    ) -> pd.DataFrame:
+        """Fetch the RAW closed-bar series between start/end, paginating
+        under the per-request bar cap. Returns uncleaned bars (the
+        backtester resolves trades on the raw price path); callers wanting
+        the signal-side series run `validate_and_clean` on the result."""
+        bars_per_day = self.BARS_PER_DAY[timeframe]
+        days_per_chunk = max(1, int(4500 / bars_per_day))
+
+        chunks = []
+        cursor_end = end
+        empty_streak = 0
+        error_streak = 0
+        while cursor_end > start:
+            cursor_start = max(start, cursor_end - timedelta(days=days_per_chunk))
+            try:
+                chunk = self._fetch_raw(
+                    symbol, timeframe, self.MAX_BARS_PER_REQUEST,
+                    start=cursor_start, end=cursor_end,
+                )
+            except Exception as e:
+                # A raising chunk is a PROVIDER ERROR (auth, 429, 5xx,
+                # malformed payload) — not "no data". Three in a row means
+                # the rest of the history is unreachable: raise instead of
+                # returning a silently truncated frame (the backtester
+                # would treat it as complete).
+                logger.warning(
+                    f"Chunk {cursor_start.date()}–{cursor_end.date()} failed: {e}"
+                )
+                error_streak += 1
+                if error_streak >= 3:
+                    raise self.DATA_ERROR(
+                        f"{symbol} {timeframe.value}: {error_streak} "
+                        f"consecutive chunk failures walking back from "
+                        f"{cursor_end.date()} — refusing to return "
+                        f"truncated history (last error: {e})") from e
+                cursor_end = cursor_start - timedelta(seconds=1)
+                continue
+
+            if chunk is None or chunk.empty:
+                # Genuinely empty windows (weekends/holidays/asset gaps)
+                # are normal; several in a row = start of history.
+                empty_streak += 1
+                if empty_streak >= 3:
+                    break
+                cursor_end = cursor_start - timedelta(seconds=1)
+                continue
+
+            empty_streak = 0
+            error_streak = 0
+            chunks.append(chunk)
+            # Step back; dedup on concat.
+            new_cursor = chunk.index[0] - timedelta(seconds=1)
+            if new_cursor >= cursor_end:
+                # A provider ignoring start/end would pin the cursor and
+                # loop forever — refuse to spin.
+                logger.warning(
+                    f"{symbol} {timeframe.value}: pagination made no "
+                    f"progress at {cursor_end} — stopping")
+                break
+            cursor_end = new_cursor
+            if len(chunk) < 2:
+                break
+
+        if not chunks:
+            raise RuntimeError(f"No data fetched for {symbol} {timeframe.value}")
+
+        df = pd.concat(chunks).sort_index()
+        df = df[~df.index.duplicated(keep="last")]
+        # Fetches ending "now" include the still-forming candle — drop it
+        # so callers only ever see closed bars.
+        df = self._drop_forming_candle(df, timeframe)
+        start_ts = pd.Timestamp(start)
+        end_ts = pd.Timestamp(end)
+        df = df[(df.index >= start_ts) & (df.index <= end_ts)]
+        # Coverage sanity: a head materially later than the requested
+        # start means history was truncated (empty-streak bail or provider
+        # depth limit). Loud, not fatal — some assets genuinely start late.
+        if not df.empty and (df.index[0] - start_ts) > pd.Timedelta(days=7):
+            logger.error(
+                f"{symbol} {timeframe.value}: fetched history starts "
+                f"{df.index[0]} but {start_ts} was requested — "
+                f"~{(df.index[0] - start_ts).days} days missing from the "
+                f"head; treat downstream results accordingly")
+        return df
+
+    def is_data_fresh(
+        self, df: pd.DataFrame, symbol: str, timeframe: Timeframe
+    ) -> bool:
+        """Public freshness check (last bar within 3 candle durations)."""
+        return self._is_data_fresh(df, symbol, timeframe)
+
+    def validate_and_clean(
+        self, df: pd.DataFrame, symbol: str, timeframe: Timeframe
+    ) -> pd.DataFrame:
+        """Public entry to the shared cleaning pipeline."""
+        return self._validate_and_clean(df, symbol, timeframe)
+
+    def clear_cache(self) -> int:
+        """Wipe all cached data. Returns number of entries cleared."""
+        with self._cache_lock:
+            count = len(self._cache)
+            self._cache.clear()
+            self._cache_timestamps.clear()
+        logger.info(f"Cache cleared: {count} entries wiped")
+        return count
+
+    def get_pair_info(self, symbol: str = XAUUSD) -> Dict:
+        info = PAIR_INFO.get(symbol, PAIR_INFO[XAUUSD])
+        return {"symbol": symbol, **info}
+
+    def get_spread(self, symbol: str = XAUUSD) -> float:
+        """Latest observed round-trip spread; providers without live
+        bid/ask fall back to the pair's assumed default."""
+        return PAIR_INFO.get(symbol, PAIR_INFO[XAUUSD])["default_spread"]
+
+    # ── Streaming-quote capability (optional) ──────────────────────
+
+    #: True when the provider can stream live ticks via `iter_ticks`.
+    supports_streaming: bool = False
+
+    def iter_ticks(
+        self,
+        symbols: Sequence[str],
+        on_connect: Optional[Callable[[], None]] = None,
+    ) -> AsyncIterator[Tuple[str, float]]:
+        """Async generator yielding (canonical_symbol, price) ticks for one
+        connection's lifetime; the caller owns reconnect/backoff. Providers
+        that support it set `supports_streaming = True` and override."""
+        raise NotImplementedError(
+            f"{self.name} provider does not support streaming quotes")
+
+    # ── Shared internals ───────────────────────────────────────────
+
+    def _has_latest_closed_bar(self, df: pd.DataFrame, timeframe: Timeframe) -> bool:
+        """Cached data must include the most recent fully closed bar.
+
+        The TTL alone is wall-clock based: a snapshot fetched mid-candle is
+        still within TTL at the next candle boundary but is missing the bar
+        that just closed, so every boundary scan would analyze — and run the
+        bar-close session check against — the previous bar.
+        """
+        if df is None or df.empty:
+            return False
+        now = pd.Timestamp.now(tz="UTC")
+        if _is_forex_weekend(now):
+            return True
+        candle_seconds = self.CANDLE_SECONDS.get(timeframe, 3600)
+        dur = pd.Timedelta(seconds=candle_seconds)
+        expected_label = now.floor(dur) - dur
+        return df.index[-1] >= expected_label
+
+    def _drop_forming_candle(self, df: pd.DataFrame, timeframe: Timeframe) -> pd.DataFrame:
+        """Drop the still-forming last candle so analysis only sees closed bars.
+
+        Some feeds include the current partial candle; the scanner fires
+        seconds after the boundary, so without this guard every indicator
+        reads a seconds-old stub bar and signals repaint. Feeds that only
+        return complete candles pass through unchanged.
+        """
+        if df is None or df.empty:
+            return df
+        candle_seconds = self.CANDLE_SECONDS.get(timeframe, 3600)
+        now = pd.Timestamp.now(tz="UTC")
+        # Loop, not a single slice: a misbehaving feed can return MORE
+        # than one not-yet-closed bar, and a single iloc[:-1] would let
+        # the residual forming bar through cleaning (future-tolerance is
+        # +5 min) straight into the signal engine.
+        dropped = 0
+        while (not df.empty
+               and df.index[-1] + pd.Timedelta(seconds=candle_seconds) > now):
+            df = df.iloc[:-1]
+            dropped += 1
+        if dropped > 1:
+            logger.warning(
+                "%s: feed returned %d forming/future bars — all dropped",
+                timeframe.value, dropped)
+        return df
+
+    def _is_data_fresh(self, df: pd.DataFrame, symbol: str, timeframe: Timeframe) -> bool:
+        """Check if the last candle is within acceptable age for this timeframe."""
+        if df is None or df.empty:
+            return False
+
+        now = pd.Timestamp.now(tz="UTC")
+
+        if _is_forex_weekend(now):
+            return True
+
+        last_bar = df.index[-1]
+        candle_seconds = self.CANDLE_SECONDS.get(timeframe, 3600)
+        max_age_seconds = candle_seconds * 3
+        age_seconds = (now - last_bar).total_seconds()
+
+        if age_seconds > max_age_seconds:
+            logger.warning(
+                f"{symbol} {timeframe.value}: last bar {last_bar.strftime('%Y-%m-%d %H:%M')} "
+                f"is {age_seconds / 60:.0f}m old (limit: {max_age_seconds / 60:.0f}m)"
+            )
+            return False
+
+        return True
+
+    def _validate_and_clean(
+        self, df: pd.DataFrame, symbol: str, timeframe: Timeframe
+    ) -> pd.DataFrame:
+        if df is None or df.empty:
+            raise self.DATA_ERROR(f"Empty dataset for {symbol}")
+
+        original_len = len(df)
+
+        df = df.dropna(subset=["Open", "High", "Low", "Close"], how="all")
+        df = df.ffill(limit=3)
+        df = df.dropna(subset=["Close"])
+
+        df = df.sort_index()
+        df = df[~df.index.duplicated(keep="last")]
+
+        now = pd.Timestamp.utcnow()
+        future_mask = df.index > now + pd.Timedelta(minutes=5)
+        if future_mask.any():
+            logger.warning(f"{symbol}: Dropping {future_mask.sum()} future timestamps")
+            df = df[~future_mask]
+
+        # Non-positive prices are corrupt, never real gold quotes. Reject
+        # the row BEFORE the consistency repair below — repairing High/Low
+        # from a corrupt 0.0 Open/Close would propagate the zero into
+        # previously-valid fields (2026-08-04 audit, I45).
+        nonpositive = (df[["Open", "High", "Low", "Close"]] <= 0).any(axis=1)
+        if nonpositive.any():
+            logger.warning(
+                f"{symbol}: Dropping {nonpositive.sum()} bars with "
+                f"non-positive prices (corrupt feed data)")
+            df = df[~nonpositive]
+            if df.empty:
+                raise self.DATA_ERROR(
+                    f"{symbol}: all bars had non-positive prices")
+
+        invalid_ohlc = (
+            (df["High"] < df["Low"]) |
+            (df["High"] < df["Open"]) |
+            (df["High"] < df["Close"]) |
+            (df["Low"] > df["Open"]) |
+            (df["Low"] > df["Close"])
+        )
+        if invalid_ohlc.any():
+            logger.warning(f"{symbol}: Fixing {invalid_ohlc.sum()} OHLC-inconsistent bars")
+            for idx in df[invalid_ohlc].index:
+                vals = [df.loc[idx, "Open"], df.loc[idx, "Close"]]
+                df.loc[idx, "High"] = max(vals + [df.loc[idx, "High"]])
+                df.loc[idx, "Low"]  = min(vals + [df.loc[idx, "Low"]])
+
+        # inf returns (division by a zero/na Close) would NaN the std and
+        # silently disarm the whole 10σ pass — neutralize them first.
+        returns = df["Close"].pct_change()
+        returns = returns.replace([float("inf"), float("-inf")], float("nan"))
+        std_ret = returns.std()
+        if std_ret > 0:
+            dev = returns - returns.mean()
+            outlier_mask = dev.abs() > 10 * std_ret
+            # A spike bar makes the NEXT bar's return look extreme too (it is
+            # measured against the spike). If the previous bar is flagged and
+            # this bar reverses direction, it is the legitimate snap-back —
+            # keep it, or every spike would punch a two-bar hole in the data.
+            recovery = (
+                outlier_mask
+                & outlier_mask.shift(1, fill_value=False)
+                & (dev * dev.shift(1) < 0)
+            )
+            outlier_mask = outlier_mask & ~recovery
+            if outlier_mask.any():
+                logger.warning(f"{symbol}: Removing {outlier_mask.sum()} extreme outlier bars")
+                df = df[~outlier_mask]
+
+        min_required = max(
+            self.config.technical.ema_trend,
+            200,
+        ) + 10
+
+        if len(df) < min_required:
+            logger.warning(
+                f"{symbol} {timeframe.value}: Only {len(df)} bars "
+                f"(need {min_required}). Some indicators will be partial."
+            )
+
+        cleaned = original_len - len(df)
+        if cleaned > 0:
+            logger.info(f"{symbol}: Cleaned {cleaned}/{original_len} bars")
+
+        return df

providers/oanda.py

@@ -0,0 +1,232 @@
+"""
+ForexBot2 — Oanda v20 data provider
+====================================
+OHLCV candles from the Oanda v20 REST API. All shared behavior (cache,
+validation, boundary guarantees, range pagination) lives in
+`BaseDataProvider`; this module only maps symbols/granularities, parses
+Oanda's candle JSON, and exposes the real bid/ask spread from MBA candles.
+"""
+
+import logging
+import time
+from datetime import datetime
+from typing import Dict, Optional
+
+import pandas as pd
+import requests
+
+from app_config import oanda_api_token, oanda_base_url
+from config import PAIR_INFO, PlatformConfig, Timeframe, XAUUSD
+
+from providers.base import BaseDataProvider, DataIntegrityError
+
+logger = logging.getLogger(__name__)
+
+
+class OandaDataError(DataIntegrityError):
+    """Raised when an Oanda data fetch fails or returns invalid data."""
+
+
+class OandaDataProvider(BaseDataProvider):
+    """Oanda v20 REST candle provider."""
+
+    name = "oanda"
+    DATA_ERROR = OandaDataError
+    RETRY_DELAY_S = 2.0
+
+    OANDA_GRANULARITY = {
+        Timeframe.M1: "M1",
+        Timeframe.M5: "M5",
+        Timeframe.M15: "M15",
+        Timeframe.M30: "M30",
+        Timeframe.H1: "H1",
+        Timeframe.H4: "H4",
+        Timeframe.D1: "D",
+        Timeframe.W1: "W",
+    }
+
+    OANDA_INSTRUMENT = {
+        "XAU/USD": "XAU_USD",
+    }
+
+    def __init__(
+        self,
+        config: PlatformConfig,
+        api_token: Optional[str] = None,
+        base_url: Optional[str] = None,
+    ):
+        super().__init__(config)
+        # Credentials default to the config.json `oanda` block / env vars
+        # so the provider registry can build this class with config only.
+        if api_token is None:
+            api_token = oanda_api_token()
+        if base_url is None:
+            base_url = oanda_base_url()
+        self._base_url = base_url.rstrip("/")
+        self._session = requests.Session()
+        self._session.headers.update({
+            "Authorization": f"Bearer {api_token}",
+            "Accept-Datetime-Format": "RFC3339",
+            "Content-Type": "application/json",
+        })
+        self._spread_cache: Dict[str, float] = {}
+
+    # ── Public extras ──────────────────────────────────────────────
+
+    def get_spread(self, symbol: str = XAUUSD) -> float:
+        return self._spread_cache.get(
+            symbol, PAIR_INFO.get(symbol, PAIR_INFO[XAUUSD])["default_spread"])
+
+    # ── Raw fetch ──────────────────────────────────────────────────
+
+    def _fetch_raw(
+        self,
+        symbol: str,
+        timeframe: Timeframe,
+        outputsize: int,
+        start: Optional[datetime] = None,
+        end: Optional[datetime] = None,
+    ) -> pd.DataFrame:
+        if start is not None or end is not None:
+            return self._fetch_oanda_candles(
+                symbol, timeframe, outputsize, start, end)
+        return self._fetch_oanda_candles(symbol, timeframe, outputsize)
+
+    def _to_oanda_instrument(self, symbol: str) -> str:
+        if symbol in self.OANDA_INSTRUMENT:
+            return self.OANDA_INSTRUMENT[symbol]
+        return symbol.replace("/", "_")
+
+    def _fetch_oanda_candles(
+        self,
+        symbol: str,
+        timeframe: Timeframe,
+        count: int,
+        start: Optional[datetime] = None,
+        end: Optional[datetime] = None,
+    ) -> pd.DataFrame:
+        instrument = self._to_oanda_instrument(symbol)
+        granularity = self.OANDA_GRANULARITY.get(timeframe)
+        if granularity is None:
+            raise OandaDataError(
+                f"Unsupported timeframe {timeframe} for Oanda")
+
+        url = f"{self._base_url}/v3/instruments/{instrument}/candles"
+        params = {
+            "granularity": granularity,
+            "price": "MBA",
+            "dailyAlignment": 0,
+            "alignmentTimezone": "Etc/UTC",
+        }
+        # v20 rejects `count` combined with both `from` and `to`; a date
+        # window replaces it (used by fetch_range pagination).
+        if start is not None and end is not None:
+            params["from"] = pd.Timestamp(start).isoformat()
+            params["to"] = pd.Timestamp(end).isoformat()
+        else:
+            params["count"] = min(count, self.MAX_BARS_PER_REQUEST)
+
+        # One retry absorbs Oanda's sporadic spurious 401s (~15-20/day on
+        # the practice API — the same token succeeds on adjacent requests)
+        # plus transient 5xx/network faults. A real credential problem
+        # fails both attempts and raises exactly as before. Other 4xx
+        # (bad request, 429) are never retried.
+        last_err: Optional[OandaDataError] = None
+        for attempt in range(2):
+            if attempt:
+                time.sleep(self.RETRY_DELAY_S)
+            try:
+                resp = self._session.get(url, params=params, timeout=30)
+            except requests.RequestException as e:
+                last_err = OandaDataError(
+                    f"Oanda candle fetch failed for {instrument}: {e}")
+                last_err.__cause__ = e
+                continue
+            if resp.status_code == 401 or resp.status_code >= 500:
+                last_err = OandaDataError(
+                    f"Oanda candle fetch failed for {instrument}: "
+                    f"HTTP {resp.status_code}: {resp.text[:300]}")
+                if attempt == 0:
+                    logger.warning(
+                        f"Oanda fetch for {instrument} got HTTP "
+                        f"{resp.status_code}; retrying in "
+                        f"{self.RETRY_DELAY_S}s")
+                continue
+            break
+        else:
+            raise last_err
+
+        if resp.status_code != 200:
+            raise OandaDataError(
+                f"Oanda candle fetch failed for {instrument}: "
+                f"HTTP {resp.status_code}: {resp.text[:300]}")
+
+        try:
+            data = resp.json()
+        except ValueError as e:
+            raise OandaDataError(
+                f"Oanda candle response for {instrument} is not valid "
+                f"JSON: {e}") from e
+        candles = data.get("candles", [])
+        if not candles:
+            raise OandaDataError(
+                f"Empty candle response from Oanda for {instrument}")
+
+        rows = []
+        last_bid_close = None
+        last_ask_close = None
+        skipped_malformed = 0
+        for c in candles:
+            if not c.get("complete", False):
+                continue
+            mid = c.get("mid", {})
+            bid = c.get("bid", {})
+            ask = c.get("ask", {})
+            # A candle missing any OHLC field must be SKIPPED, never
+            # defaulted to 0.0: a zero price flows through cleaning as a
+            # "legit" bar and poisons ATR/Keltner/SL math — and a
+            # mid-frame zero Close makes pct_change() emit inf, which
+            # NaNs the std and disarms the 10σ outlier filter for the
+            # whole frame (2026-08-04 audit, I45).
+            if any(mid.get(k) is None for k in ("o", "h", "l", "c")):
+                skipped_malformed += 1
+                continue
+            rows.append({
+                "Datetime": pd.Timestamp(c["time"]).tz_convert("UTC"),
+                "Open": float(mid["o"]),
+                "High": float(mid["h"]),
+                "Low": float(mid["l"]),
+                "Close": float(mid["c"]),
+                "Volume": int(c.get("volume", 0)),
+            })
+            if bid and ask:
+                last_bid_close = float(bid.get("c", 0))
+                last_ask_close = float(ask.get("c", 0))
+        if skipped_malformed:
+            logger.warning(
+                "Oanda returned %d malformed candle(s) for %s (missing "
+                "OHLC fields) — skipped", skipped_malformed, instrument)
+
+        if not rows:
+            raise OandaDataError(
+                f"No complete candles from Oanda for {instrument}")
+
+        if last_bid_close and last_ask_close:
+            self._spread_cache[symbol] = round(
+                last_ask_close - last_bid_close, 4)
+
+        df = pd.DataFrame(rows)
+        df = df.set_index("Datetime")
+        df = df.sort_index()
+        if df.index.tz is None:
+            df.index = df.index.tz_localize("UTC")
+        df.index.name = "Datetime"
+
+        for col in ("Open", "High", "Low", "Close"):
+            df[col] = pd.to_numeric(df[col], errors="coerce")
+        df["Volume"] = pd.to_numeric(df["Volume"], errors="coerce").fillna(0).astype(float)
+
+        logger.debug(
+            "Fetched %d bars for %s @ %s from Oanda",
+            len(df), instrument, granularity)
+        return df

providers/tiingo.py

@@ -0,0 +1,308 @@
+"""
+ForexBot2 — Tiingo data provider
+=================================
+OHLCV candles from Tiingo's FX endpoint
+(`/tiingo/fx/{ticker}/prices`, resampled mid-price bars, no volume).
+Tiingo is streaming-only in production: its REST candles remain available
+for comparison research, but it is not a designated candle fallback.
+All shared behavior (cache, validation, boundary guarantees, range
+pagination) lives in `BaseDataProvider`; this module only maps
+symbols/resample frequencies, sizes the request window, and parses
+Tiingo's JSON into the normalized frame.
+
+Every failure mode raises `TiingoDataError` (a `DataIntegrityError`)
+with a specific message — the scan loops log it and carry on; nothing
+fails silently. NOTE: candles and mid-price ticks only. Tiingo's free
+`/tiingo/fx/top` quote endpoint publishes indicative two-tier spreads
+and must not be used for spread display (removed 2026-07-23) —
+`get_spread` deliberately stays on the base-class default, and the
+WebSocket stream (`iter_ticks`) exposes the mid price only, never
+bid/ask or a derived spread.
+"""
+
+import json
+import logging
+from datetime import datetime
+from typing import AsyncIterator, Callable, Optional, Sequence, Tuple
+
+import pandas as pd
+import requests
+
+from app_config import provider_api_key
+from config import PlatformConfig, Timeframe
+
+from providers.base import BaseDataProvider, DataIntegrityError
+
+logger = logging.getLogger(__name__)
+
+
+class TiingoDataError(DataIntegrityError):
+    """Raised when a Tiingo data fetch fails or returns invalid data."""
+
+
+class TiingoDataProvider(BaseDataProvider):
+    """Tiingo FX REST candle provider."""
+
+    name = "tiingo"
+    DATA_ERROR = TiingoDataError
+
+    BASE_URL = "https://api.tiingo.com"
+
+    TIINGO_RESAMPLE = {
+        Timeframe.M1: "1min",
+        Timeframe.M5: "5min",
+        Timeframe.M15: "15min",
+        Timeframe.M30: "30min",
+        Timeframe.H1: "1hour",
+        Timeframe.H4: "4hour",
+        Timeframe.D1: "1day",
+        # W1 intentionally unmapped: the FX endpoint has no weekly
+        # resample — requesting it raises loudly instead of guessing.
+    }
+
+    TIINGO_TICKER = {
+        "XAU/USD": "xauusd",
+    }
+
+    # Window padding over the theoretical bar span: FX is closed ~2/7 of
+    # the week (Fri 22:00 → Sun 22:00 UTC), so a calendar window must be
+    # wider than `bars × candle_seconds` to yield the requested count.
+    WINDOW_PAD_FACTOR = 1.6
+
+    def __init__(self, config: PlatformConfig, api_key: Optional[str] = None):
+        super().__init__(config)
+        # Credentials default to env TIINGO_API_KEY → config.json
+        # data_providers.tiingo.api_key, so the provider registry can
+        # build this class with config only.
+        if api_key is None:
+            api_key = provider_api_key(provider="tiingo")
+        self._api_key = api_key or None
+        if self._api_key is None:
+            logger.warning(
+                "Tiingo provider built WITHOUT an API key — every fetch "
+                "will fail with a named error. Set the TIINGO_API_KEY env "
+                "var or data_providers.tiingo.api_key in config.json.")
+        self._session = requests.Session()
+        self._session.headers.update({
+            "Content-Type": "application/json",
+            # Header auth keeps the key out of URLs (and out of logs).
+            "Authorization": f"Token {api_key}" if api_key else "",
+        })
+
+    def _to_tiingo_ticker(self, symbol: str) -> str:
+        if symbol in self.TIINGO_TICKER:
+            return self.TIINGO_TICKER[symbol]
+        return symbol.replace("/", "").replace("=X", "").lower()
+
+    # ── Raw fetch ──────────────────────────────────────────────────
+
+    def _fetch_raw(
+        self,
+        symbol: str,
+        timeframe: Timeframe,
+        outputsize: int,
+        start: Optional[datetime] = None,
+        end: Optional[datetime] = None,
+    ) -> pd.DataFrame:
+        if self._api_key is None:
+            raise TiingoDataError(
+                "No Tiingo API key. Set the TIINGO_API_KEY env var or "
+                "data_providers.tiingo.api_key in config.json.")
+
+        resample = self.TIINGO_RESAMPLE.get(timeframe)
+        if resample is None:
+            raise TiingoDataError(
+                f"Unsupported timeframe {timeframe} for Tiingo "
+                f"(no FX resample frequency)")
+
+        ticker = self._to_tiingo_ticker(symbol)
+
+        # Tiingo takes date windows, not bar counts: derive the start
+        # from the requested count for live fetches.
+        if start is None:
+            span = self.CANDLE_SECONDS[timeframe] * (outputsize + 10)
+            start_ts = (pd.Timestamp.now(tz="UTC")
+                        - pd.Timedelta(seconds=span * self.WINDOW_PAD_FACTOR))
+        else:
+            start_ts = pd.Timestamp(start)
+
+        params = {
+            "resampleFreq": resample,
+            "startDate": start_ts.strftime("%Y-%m-%dT%H:%M:%S"),
+        }
+        if end is not None:
+            params["endDate"] = pd.Timestamp(end).strftime("%Y-%m-%dT%H:%M:%S")
+
+        url = f"{self.BASE_URL}/tiingo/fx/{ticker}/prices"
+        try:
+            resp = self._session.get(url, params=params, timeout=30)
+        except requests.RequestException as e:
+            raise TiingoDataError(
+                f"Tiingo candle fetch failed for {ticker}: {e}") from e
+
+        if resp.status_code == 429:
+            raise TiingoDataError(
+                f"Tiingo rate limit hit for {ticker} (HTTP 429) — "
+                f"plan quota exhausted; scan will retry next boundary")
+        if resp.status_code in (401, 403):
+            raise TiingoDataError(
+                f"Tiingo auth failed for {ticker} (HTTP {resp.status_code}) "
+                f"— check TIINGO_API_KEY / config.json api_key")
+        if resp.status_code != 200:
+            raise TiingoDataError(
+                f"Tiingo candle fetch failed for {ticker}: "
+                f"HTTP {resp.status_code}: {resp.text[:300]}")
+
+        try:
+            data = resp.json()
+        except ValueError as e:
+            raise TiingoDataError(
+                f"Tiingo candle response for {ticker} is not valid "
+                f"JSON: {e}") from e
+
+        if not isinstance(data, list) or not data:
+            raise TiingoDataError(
+                f"Empty candle response from Tiingo for {ticker} "
+                f"(window {params['startDate']} → "
+                f"{params.get('endDate', 'now')}; check ticker and plan "
+                f"history limits)")
+
+        rows = []
+        for item in data:
+            if "date" not in item:
+                raise TiingoDataError(
+                    f"Malformed Tiingo candle for {ticker}: missing "
+                    f"'date' in {str(item)[:120]}")
+            rows.append({
+                "Datetime": pd.Timestamp(item["date"]),
+                "Open": item.get("open"),
+                "High": item.get("high"),
+                "Low": item.get("low"),
+                "Close": item.get("close"),
+                # FX endpoint has no volume; 0.0 keeps the engine contract.
+                "Volume": 0.0,
+            })
+
+        df = pd.DataFrame(rows)
+        df = df.set_index("Datetime")
+        df = df.sort_index()
+        if df.index.tz is None:
+            df.index = df.index.tz_localize("UTC")
+        else:
+            df.index = df.index.tz_convert("UTC")
+        df.index.name = "Datetime"
+
+        for col in ("Open", "High", "Low", "Close"):
+            df[col] = pd.to_numeric(df[col], errors="coerce")
+        if df["Close"].isna().all():
+            raise TiingoDataError(
+                f"Tiingo response for {ticker} has no numeric closes — "
+                f"feed format changed?")
+
+        logger.debug(
+            "Fetched %d bars for %s @ %s from Tiingo",
+            len(df), ticker, resample)
+        return df
+
+    # ── Streaming quotes ───────────────────────────────────────────
+
+    supports_streaming = True
+
+    WS_URL = "wss://api.tiingo.com/fx"
+
+    @staticmethod
+    def _parse_ws_message(raw: str) -> Optional[Tuple[str, float]]:
+        """Parse one WS message; return (tiingo_ticker, mid_price) for
+        quote updates, None for heartbeats ("H"), subscription acks
+        ("I"), malformed JSON, or quotes without a usable price.
+
+        Server error messages ("E", e.g. bad token) RAISE TiingoDataError:
+        Tiingo authenticates inside the socket via the subscribe message,
+        so an auth failure arrives as a data message followed by a clean
+        close — swallowing it would make the caller's reconnect loop
+        hammer the endpoint with no backoff (backoff only grows on
+        exceptions).
+        """
+        try:
+            msg = json.loads(raw)
+        except json.JSONDecodeError:
+            logger.warning("Tiingo WS: malformed JSON, skipping")
+            return None
+
+        mtype = msg.get("messageType")
+        if mtype == "H":
+            return None
+        if mtype == "I":
+            logger.info("Tiingo WS: subscription acknowledged")
+            return None
+        if mtype == "E":
+            raise TiingoDataError(
+                f"Tiingo WS server error: {str(msg.get('response'))[:200]}")
+        if mtype != "A":
+            return None
+
+        data = msg.get("data")
+        if not isinstance(data, list) or len(data) < 8 or data[0] != "Q":
+            return None
+        # Live-verified field order (2026-08-03): ["Q", ticker, timestamp,
+        # bidSize, bid, mid, askSize, ask] — ask is index 7, NOT 6 as the
+        # docs' table suggests (index 6 is askSize).
+        ticker = data[1]
+        bid, mid, ask = data[4], data[5], data[7]
+        if mid is None and bid is not None and ask is not None:
+            mid = (bid + ask) / 2
+        try:
+            return ticker, float(mid)
+        except (TypeError, ValueError):
+            logger.warning("Tiingo WS: quote for %s has no usable price",
+                           ticker)
+            return None
+
+    async def iter_ticks(
+        self,
+        symbols: Sequence[str],
+        on_connect: Optional[Callable[[], None]] = None,
+    ) -> AsyncIterator[Tuple[str, float]]:
+        """One connection's lifetime of (canonical_symbol, mid) ticks.
+
+        The caller owns the reconnect/backoff loop around this generator.
+        Divergences from the Twelve Data reference implementation, both
+        deliberate: `on_connect` fires on the FIRST RECEIVED message (the
+        "I" ack in practice) rather than after the subscribe send, so a
+        rejected token cannot reset the caller's backoff; and there is no
+        client heartbeat task — Tiingo sends "H" frames itself and the
+        websockets library's built-in ping keepalive detects dead sockets.
+        """
+        import websockets
+
+        if self._api_key is None:
+            raise TiingoDataError(
+                "No Tiingo API key. Set the TIINGO_API_KEY env var or "
+                "data_providers.tiingo.api_key in config.json.")
+
+        ws_to_pair = {self._to_tiingo_ticker(s): s for s in symbols}
+        async with websockets.connect(self.WS_URL) as ws:
+            await ws.send(json.dumps({
+                "eventName": "subscribe",
+                "authorization": self._api_key,
+                "eventData": {
+                    "thresholdLevel": 5,
+                    "tickers": sorted(ws_to_pair),
+                },
+            }))
+            logger.info(
+                "Tiingo WS connected, subscribing to %d tickers",
+                len(ws_to_pair))
+            connected = False
+            async for raw in ws:
+                parsed = self._parse_ws_message(raw)
+                if not connected:
+                    connected = True
+                    if on_connect is not None:
+                        on_connect()
+                if parsed is None:
+                    continue
+                pair_symbol = ws_to_pair.get(parsed[0])
+                if pair_symbol is None:
+                    continue
+                yield pair_symbol, parsed[1]

providers/twelve_data.py

@@ -0,0 +1,247 @@
+"""
+ForexBot2 — Twelve Data provider
+=================================
+OHLCV via the official `twelvedata` SDK plus streaming quotes over the
+Twelve Data WebSocket. All shared behavior (cache, validation, boundary
+guarantees, range pagination) lives in `BaseDataProvider`.
+"""
+
+import asyncio
+import json
+import logging
+import threading
+import time
+from collections import deque
+from datetime import datetime
+from typing import AsyncIterator, Callable, Optional, Sequence, Tuple
+
+import pandas as pd
+from twelvedata import TDClient
+
+from app_config import provider_api_key
+from config import PAIR_INFO, PlatformConfig, Timeframe
+
+from providers.base import BaseDataProvider, DataIntegrityError
+
+logger = logging.getLogger(__name__)
+
+
+class TwelveDataProvider(BaseDataProvider):
+    """Twelve Data SDK client with credit throttling and WS quotes."""
+
+    name = "twelve_data"
+
+    TD_INTERVALS = {
+        Timeframe.M1:  "1min",
+        Timeframe.M5:  "5min",
+        Timeframe.M15: "15min",
+        Timeframe.M30: "30min",
+        Timeframe.H1:  "1h",
+        Timeframe.H4:  "4h",
+        Timeframe.D1:  "1day",
+        Timeframe.W1:  "1week",
+    }
+
+    RATE_LIMIT = 55
+    RATE_WINDOW = 60.0
+
+    WS_URL = "wss://ws.twelvedata.com/v1/quotes/price?apikey={api_key}"
+
+    def __init__(self, config: PlatformConfig):
+        super().__init__(config)
+        self._api_key = provider_api_key(provider="twelve_data") or None
+        self._throttle_lock = threading.Lock()
+        self._request_times: deque = deque()
+        self._td: Optional[TDClient] = (
+            TDClient(apikey=self._api_key) if self._api_key else None
+        )
+
+    @staticmethod
+    def _to_td_symbol(symbol: str) -> str:
+        if "/" in symbol:
+            return symbol
+        clean = symbol.replace("=X", "")
+        return f"{clean[:3]}/{clean[3:]}"
+
+    # ── Raw fetch ──────────────────────────────────────────────────
+
+    def _fetch_raw(
+        self,
+        symbol: str,
+        timeframe: Timeframe,
+        outputsize: int,
+        start: Optional[datetime] = None,
+        end: Optional[datetime] = None,
+    ) -> pd.DataFrame:
+        return self._fetch_twelve_data(
+            symbol, timeframe, outputsize,
+            start_date=start.strftime("%Y-%m-%d %H:%M:%S") if start else None,
+            end_date=end.strftime("%Y-%m-%d %H:%M:%S") if end else None,
+        )
+
+    def _throttle(self, credits: int = 1) -> None:
+        """Pro tier: 55 credits / minute. Block until budget allows."""
+        while True:
+            with self._throttle_lock:
+                now = time.time()
+                while self._request_times and now - self._request_times[0] >= self.RATE_WINDOW:
+                    self._request_times.popleft()
+
+                available = self.RATE_LIMIT - len(self._request_times)
+                if available >= credits:
+                    for _ in range(credits):
+                        self._request_times.append(time.time())
+                    return
+
+                used = len(self._request_times)
+                wait = (self._request_times[0] + self.RATE_WINDOW) - now + 0.2
+            logger.info(
+                f"Rate limit: {used}/{self.RATE_LIMIT} credits used "
+                f"— waiting {wait:.1f}s..."
+            )
+            time.sleep(wait)
+
+    def _fetch_twelve_data(
+        self,
+        symbol: str,
+        timeframe: Timeframe,
+        outputsize: int,
+        start_date: Optional[str] = None,
+        end_date: Optional[str] = None,
+    ) -> pd.DataFrame:
+        if self._td is None:
+            raise DataIntegrityError(
+                "No data-provider API key. Set the TWELVE_DATA_API_KEY env "
+                "var or data_providers.twelve_data.api_key in config.json."
+            )
+
+        td_symbol = self._to_td_symbol(symbol)
+        interval = self.TD_INTERVALS[timeframe]
+
+        self._throttle(credits=1)
+
+        kwargs = {
+            "symbol": td_symbol,
+            "interval": interval,
+            "outputsize": min(outputsize, self.MAX_BARS_PER_REQUEST),
+            "timezone": "UTC",
+        }
+        if start_date:
+            kwargs["start_date"] = start_date
+        if end_date:
+            kwargs["end_date"] = end_date
+
+        try:
+            df = self._td.time_series(**kwargs).as_pandas()
+        except Exception as e:
+            raise DataIntegrityError(
+                f"Twelve Data fetch failed for {td_symbol}: {e}"
+            ) from e
+
+        if df is None or df.empty:
+            raise DataIntegrityError(f"Empty response from Twelve Data for {td_symbol}")
+
+        # SDK returns lowercase columns; engine expects capitalised.
+        df = df.rename(columns={
+            "open": "Open", "high": "High", "low": "Low",
+            "close": "Close", "volume": "Volume",
+        })
+        for col in ("Open", "High", "Low", "Close", "Volume"):
+            if col in df.columns:
+                df[col] = pd.to_numeric(df[col], errors="coerce")
+        if "Volume" not in df.columns:
+            df["Volume"] = 0.0
+
+        # SDK returns newest-first; engine + indicators want oldest-first.
+        df = df.sort_index()
+        if df.index.tz is None:
+            df.index = df.index.tz_localize("UTC")
+        df.index.name = "Datetime"
+
+        logger.debug(f"Fetched {len(df)} bars for {td_symbol} @ {interval}")
+        return df
+
+    # ── Streaming quotes ───────────────────────────────────────────
+
+    supports_streaming = True
+
+    @staticmethod
+    def _parse_ws_price(raw: str) -> Optional[Tuple[str, float]]:
+        """Parse one WS message; return (ws_symbol, price) for price
+        events, None for everything else (logged where noteworthy)."""
+        try:
+            msg = json.loads(raw)
+        except json.JSONDecodeError:
+            logger.warning("WebSocket: malformed JSON, skipping")
+            return None
+
+        event = msg.get("event")
+        if event == "subscribe-status":
+            fails = msg.get("fails") or []
+            if fails:
+                logger.warning(f"WebSocket: failed to subscribe: {fails}")
+            return None
+        if event != "price":
+            return None
+
+        ws_symbol = msg.get("symbol")
+        try:
+            price = float(msg["price"])
+        except (KeyError, ValueError, TypeError) as e:
+            logger.warning(f"WebSocket: bad price data for {ws_symbol}: {e}")
+            return None
+        return ws_symbol, price
+
+    @staticmethod
+    async def _ws_heartbeat(ws) -> None:
+        """Twelve Data drops idle WS connections; heartbeat every 10s keeps
+        it alive through quiet periods (weekends, low-liquidity hours)."""
+        while True:
+            await asyncio.sleep(10)
+            await ws.send(json.dumps({"action": "heartbeat"}))
+
+    async def iter_ticks(
+        self,
+        symbols: Sequence[str],
+        on_connect: Optional[Callable[[], None]] = None,
+    ) -> AsyncIterator[Tuple[str, float]]:
+        """One connection's lifetime of (canonical_symbol, price) ticks.
+
+        The TD WS speaks display symbols ("XAU/USD"); ticks are mapped back
+        to the canonical pair symbols passed in. The caller owns the
+        reconnect/backoff loop around this generator.
+
+        `on_connect` fires on the FIRST RECEIVED frame, not after the
+        subscribe send (2026-08-04 audit H38b, mirroring the Tiingo
+        provider) — firing pre-receive let a rejected API key reset the
+        caller's reconnect backoff on every attempt, pinning it at the
+        minimum forever.
+        """
+        import websockets
+
+        ws_to_pair = {PAIR_INFO[s]["display"]: s for s in symbols}
+        url = self.WS_URL.format(api_key=self._api_key or "")
+        async with websockets.connect(url) as ws:
+            subscribe = {
+                "action": "subscribe",
+                "params": {"symbols": ",".join(ws_to_pair)},
+            }
+            await ws.send(json.dumps(subscribe))
+            logger.info(f"WebSocket connected, subscribed to {len(ws_to_pair)} symbols")
+            hb_task = asyncio.create_task(self._ws_heartbeat(ws))
+            connected = False
+            try:
+                async for raw in ws:
+                    if not connected:
+                        connected = True
+                        if on_connect is not None:
+                            on_connect()
+                    parsed = self._parse_ws_price(raw)
+                    if parsed is None:
+                        continue
+                    pair_symbol = ws_to_pair.get(parsed[0])
+                    if pair_symbol is None:
+                        continue
+                    yield pair_symbol, parsed[1]
+            finally:
+                hb_task.cancel()

requirements.txt

@@ -0,0 +1,6 @@
+numpy>=1.24
+pandas>=2.0
+requests>=2.28
+twelvedata>=1.2.0
+python-telegram-bot>=20.0
+websockets>=12.0

script.txt

@@ -1 +0,0 @@
-pip install pandas numpy requests python-dateutil -q

scripts/backfill_signal_outcomes.py

@@ -0,0 +1,91 @@
+#!/usr/bin/env python3
+"""
+backfill_signal_outcomes.py — seed and advance advisory outcomes for every
+logged trade setup (not only panel rows), one per advisory event.
+
+Pass A (offline, no network): assign setup_key / advisory_event_id to all
+legacy setup rows, mark same-setup re-logs within 60 min as DUPLICATE, and
+seed a PENDING outcome on each canonical row that has none. Existing
+(panel-seeded) outcomes are never overwritten.
+
+Pass B: advance PENDING rows from the closed-candle streams already stored
+in the DB (union across every snapshot of the timeframe — no provider).
+
+Pass C (--reseed-unresolvable): bounded re-seed of continuity-broken rows,
+then Pass B again.
+
+Safe beside the live writers: short per-row transactions with a busy
+timeout. Read-only preview with --dry-run.
+
+    python scripts/backfill_signal_outcomes.py --dry-run
+    python scripts/backfill_signal_outcomes.py
+    python scripts/backfill_signal_outcomes.py --reseed-unresolvable
+"""
+import argparse
+import os
+import sqlite3
+import sys
+
+sys.path.insert(0, os.path.dirname(os.path.dirname(os.path.abspath(__file__))))
+
+import app_config  # noqa: E402
+from signal_log import SignalLogger  # noqa: E402
+
+
+def _status_counts(con: sqlite3.Connection) -> dict:
+    return dict(con.execute(
+        "SELECT COALESCE(advisory_outcome_status, 'NULL'), COUNT(*)"
+        " FROM signals WHERE setup_json IS NOT NULL AND setup_json != 'null'"
+        " GROUP BY 1").fetchall())
+
+
+def main(argv=None) -> int:
+    ap = argparse.ArgumentParser(description=__doc__.splitlines()[1])
+    ap.add_argument("--db", default=None, help="defaults to signal_log.path")
+    ap.add_argument("--dry-run", action="store_true")
+    ap.add_argument("--reseed-unresolvable", action="store_true")
+    ap.add_argument("--symbol", default="XAU/USD")
+    args = ap.parse_args(argv)
+    path = args.db or app_config.signal_log_path()
+    if not os.path.exists(path):
+        print(f"no database at {path}")
+        return 1
+    if args.dry_run:
+        con = sqlite3.connect(f"file:{os.path.abspath(path)}?mode=ro", uri=True)
+        print("setup rows by outcome status:", _status_counts(con))
+        have = {r[1] for r in con.execute("PRAGMA table_info(signals)")}
+        if "setup_key" in have:
+            n_keyless = con.execute(
+                "SELECT COUNT(*) FROM signals WHERE setup_json IS NOT NULL"
+                " AND setup_json != 'null' AND setup_key IS NULL").fetchone()[0]
+            print(f"setup rows without setup_key: {n_keyless}")
+        else:
+            print("setup_key column not present yet (schema migrates on"
+                  " the first non-dry run)")
+        con.close()
+        return 0
+    lg = SignalLogger(path)
+    try:
+        print("before:", _status_counts(lg._conn))
+        print("pass A (seed/dedup):", lg.backfill_seeds())
+        total = 0
+        for _ in range(50):
+            n = lg.resolve_from_store(args.symbol)
+            total += n
+            if n == 0:
+                break
+        print(f"pass B (advance from stored bars): {total} rows changed")
+        if args.reseed_unresolvable:
+            reseeded = lg.reseed_unresolvable()
+            print(f"pass C (re-seed): {len(reseeded)} rows re-seeded")
+            for _ in range(50):
+                if lg.resolve_from_store(args.symbol) == 0:
+                    break
+        print("after:", _status_counts(lg._conn))
+    finally:
+        lg.close()
+    return 0
+
+
+if __name__ == "__main__":
+    sys.exit(main())

scripts/compare_oanda_tiingo.py

@@ -0,0 +1,116 @@
+#!/usr/bin/env python3
+"""
+Scratchpad — Oanda vs Tiingo candle parity check
+=================================================
+Fetches the last N closed XAU/USD candles from Oanda and Tiingo for
+M5/M15/H1 via the normal provider stack (make_data_provider →
+fetch_ohlcv: forming-candle drop + validate/clean included, i.e. exactly
+what the signal pipeline consumes) and reports how well they line up:
+
+  * last-bar timestamp per provider (feed-lag check)
+  * timestamp mismatches (bars present in one feed only)
+  * per-bar OHLC diff table for the common tail
+  * max/mean |diff| per column, with the Close diff also in % of price
+
+Both feeds are mid-price (Oanda mid of MBA candles; Tiingo resampled
+mid), so small diffs are expected — this measures how small.
+
+Usage:
+    venv/bin/python scripts/compare_oanda_tiingo.py [--n 10]
+"""
+
+import argparse
+import sys
+from pathlib import Path
+
+import pandas as pd
+
+sys.path.insert(0, str(Path(__file__).resolve().parent.parent))
+
+from config import PlatformConfig, Timeframe, XAUUSD
+from providers import make_data_provider
+
+TIMEFRAMES = [Timeframe.M5, Timeframe.M15, Timeframe.H1]
+OHLC = ["Open", "High", "Low", "Close"]
+
+
+def fetch_tail(provider, tf, n):
+    """Last n closed bars from one provider, or None on failure."""
+    try:
+        df = provider.fetch_ohlcv(symbol=XAUUSD, timeframe=tf)
+    except Exception as e:  # scratchpad: report, don't crash
+        print(f"  !! {provider.name} {tf.value} fetch failed: {e}")
+        return None
+    return df.tail(n)
+
+
+def compare_tf(tf, oanda_df, tiingo_df, n):
+    print(f"\n{'=' * 78}\n{tf.value} — last {n} closed bars\n{'=' * 78}")
+    if oanda_df is None or tiingo_df is None:
+        print("  skipped (a provider failed)")
+        return
+
+    print(f"  Oanda last bar: {oanda_df.index[-1]}   "
+          f"Tiingo last bar: {tiingo_df.index[-1]}")
+
+    common = oanda_df.index.intersection(tiingo_df.index)
+    only_o = oanda_df.index.difference(tiingo_df.index)
+    only_t = tiingo_df.index.difference(oanda_df.index)
+    if len(only_o):
+        print(f"  bars only in Oanda : {[str(t) for t in only_o]}")
+    if len(only_t):
+        print(f"  bars only in Tiingo: {[str(t) for t in only_t]}")
+    if not len(common):
+        print("  NO common timestamps — feeds are on different bar grids!")
+        return
+
+    o = oanda_df.loc[common, OHLC]
+    t = tiingo_df.loc[common, OHLC]
+    diff = o - t
+
+    print(f"\n  {len(common)}/{n} bars share timestamps "
+          f"(window {common[0]} → {common[-1]})\n")
+
+    # Per-bar Close table — the number that matters most.
+    hdr = f"  {'bar close (UTC)':<20}{'Oanda C':>12}{'Tiingo C':>12}{'ΔC ($)':>10}"
+    print(hdr + f"{'ΔO':>8}{'ΔH':>8}{'ΔL':>8}")
+    for ts in common:
+        row = (f"  {str(ts):<20}{o.loc[ts, 'Close']:>12.2f}"
+               f"{t.loc[ts, 'Close']:>12.2f}{diff.loc[ts, 'Close']:>10.3f}")
+        row += (f"{diff.loc[ts, 'Open']:>8.3f}{diff.loc[ts, 'High']:>8.3f}"
+                f"{diff.loc[ts, 'Low']:>8.3f}")
+        print(row)
+
+    print("\n  summary over common bars:")
+    for col in OHLC:
+        d = diff[col].abs()
+        print(f"    {col:<6} max |Δ| = {d.max():8.4f}   mean |Δ| = {d.mean():8.4f}")
+    px = o["Close"].iloc[-1]
+    dc = diff["Close"].abs()
+    print(f"    Close Δ as % of price: max {dc.max() / px * 100:.4f}%   "
+          f"mean {dc.mean() / px * 100:.4f}%")
+    exact = (diff[OHLC].abs() < 1e-9).all(axis=1).sum()
+    print(f"    bars with identical OHLC: {exact}/{len(common)}")
+
+
+def main():
+    ap = argparse.ArgumentParser()
+    ap.add_argument("--n", type=int, default=10,
+                    help="bars per timeframe to compare (default 10)")
+    args = ap.parse_args()
+
+    cfg = PlatformConfig()
+    oanda = make_data_provider(cfg, "oanda")
+    tiingo = make_data_provider(cfg, "tiingo")
+
+    for tf in TIMEFRAMES:
+        o_df = fetch_tail(oanda, tf, args.n)
+        t_df = fetch_tail(tiingo, tf, args.n)
+        compare_tf(tf, o_df, t_df, args.n)
+
+    print("\nDone. Note: Oanda=mid of MBA candles, Tiingo=resampled mid — "
+          "sub-$1 diffs on gold are normal feed divergence, not a bug.")
+
+
+if __name__ == "__main__":
+    sys.exit(main())

scripts/feed_divergence_study.py

@@ -0,0 +1,259 @@
+#!/usr/bin/env python3
+"""
+Phase 1 — historical candle divergence study (data source change test)
+=======================================================================
+Fetches the RAW closed-bar series for XAU/USD from two providers over a
+multi-month window via the normal provider stack (`fetch_range`, which
+paginates under the per-request bar cap and drops the forming candle),
+then quantifies how the feeds diverge:
+
+  * coverage: % shared timestamps, feed-only bars, where they cluster
+  * OHLC |Δ| distribution (mean/median/p90/p95/p99/max, $ and bias)
+  * |ΔClose| vs mean ATR14 (the scale the signal engine cares about)
+  * threshold exceedance (how many bars differ by more than X $)
+  * bar-direction agreement (sign of close−open)
+  * close-return correlation on consecutive bars
+  * per-hour and per-month drift
+
+Both feeds are mid-price (Oanda mid of MBA candles; Tiingo resampled mid;
+Twelve Data its own mid). Frames are cached to parquet so re-runs are
+instant; pass --no-cache to force a fresh fetch.
+
+Usage:
+    venv/bin/python scripts/feed_divergence_study.py --a oanda --b tiingo \
+        --tfs M15,H1,H4 --days 365
+"""
+
+import argparse
+import sys
+import time
+from pathlib import Path
+
+import numpy as np
+import pandas as pd
+
+sys.path.insert(0, str(Path(__file__).resolve().parent.parent))
+
+from config import PlatformConfig, Timeframe, XAUUSD
+from providers import make_data_provider
+
+REPO = Path(__file__).resolve().parent.parent
+CACHE_DIR = REPO / "backtest" / "cache"
+RESEARCH_DIR = REPO / "backtest" / "research"
+
+_TF_LOOKUP = {t.value: t for t in Timeframe}
+_TF_LOOKUP.update({
+    "M1": Timeframe.M1, "M5": Timeframe.M5, "M15": Timeframe.M15,
+    "M30": Timeframe.M30, "H1": Timeframe.H1, "H4": Timeframe.H4,
+    "D1": Timeframe.D1, "W1": Timeframe.W1,
+})
+
+
+def wilder_atr(df: pd.DataFrame, period: int = 14) -> pd.Series:
+    hl = df["High"] - df["Low"]
+    hc = (df["High"] - df["Close"].shift(1)).abs()
+    lc = (df["Low"] - df["Close"].shift(1)).abs()
+    tr = pd.concat([hl, hc, lc], axis=1).max(axis=1)
+    return tr.ewm(alpha=1.0 / period, adjust=False).mean()
+
+
+def fmt_stats(name: str, s: pd.Series) -> str:
+    a = s.abs()
+    return (f"  {name:<6} mean ${a.mean():.3f}  med ${a.median():.3f}  "
+            f"p90 ${a.quantile(0.90):.3f}  p95 ${a.quantile(0.95):.3f}  "
+            f"p99 ${a.quantile(0.99):.3f}  max ${a.max():.2f}  "
+            f"bias {s.mean():+.4f}")
+
+
+def cache_path(name: str, tf: Timeframe, start: pd.Timestamp,
+               end: pd.Timestamp) -> Path:
+    return (CACHE_DIR /
+            f"{name}_XAU_USD_{tf.value}_{start:%Y%m%d}_{end:%Y%m%d}.parquet")
+
+
+def load_or_fetch(provider, name: str, tf: Timeframe, start, end,
+                  no_cache: bool):
+    cp = cache_path(name, tf, start, end)
+    if not no_cache and cp.exists():
+        df = pd.read_parquet(cp)
+        print(f"  [{name}] loaded {len(df):,} bars from cache "
+              f"{cp.name}")
+        return df
+    t0 = time.time()
+    print(f"  [{name}] fetching {tf.value} {start.date()} → {end.date()} ...")
+    df = provider.fetch_range(XAUUSD, tf, start, end)
+    print(f"  [{name}] fetched {len(df):,} bars in {time.time() - t0:.1f}s")
+    df.to_parquet(cp)
+    print(f"  [{name}] cached → {cp.name}")
+    return df
+
+
+def compare(tf: Timeframe, a_name: str, b_name: str, a: pd.DataFrame,
+            b: pd.DataFrame):
+    print("\n" + "=" * 80)
+    print(f"{tf.value} — {a_name.upper()} vs {b_name.upper()}, XAU/USD")
+    print("=" * 80)
+
+    common = a.index.intersection(b.index)
+    a_only = a.index.difference(b.index)
+    b_only = b.index.difference(a.index)
+
+    print(f"\nCOVERAGE")
+    print(f"  {a_name} bars : {len(a):>8,}   head {a.index[0]}  tail {a.index[-1]}")
+    print(f"  {b_name} bars : {len(b):>8,}   head {b.index[0]}  tail {b.index[-1]}")
+    print(f"  shared        : {len(common):>8,}  "
+          f"({100 * len(common) / max(len(a.index.union(b.index)), 1):.2f}% of union)")
+    print(f"  {a_name}-only : {len(a_only):>8,}  "
+          f"({100 * len(a_only) / max(len(a), 1):.2f}% of {a_name})")
+    print(f"  {b_name}-only : {len(b_only):>8,}  "
+          f"({100 * len(b_only) / max(len(b), 1):.2f}% of {b_name})")
+
+    if len(common) == 0:
+        print("  !! no shared timestamps — feeds on different bar grids")
+        return
+
+    for label, idx in ((f"{a_name}-only", a_only),
+                       (f"{b_name}-only", b_only)):
+        if not len(idx):
+            continue
+        by_date = pd.Series(1, index=idx).groupby(idx.date).sum()
+        top = by_date.sort_values(ascending=False).head(6)
+        print(f"  {label} clusters on: "
+              + ", ".join(f"{d} ({n})" for d, n in top.items()))
+
+    x = a.loc[common]
+    y = b.loc[common]
+
+    atr = wilder_atr(x).dropna()
+    mean_price = x["Close"].mean()
+    print(f"\nSCALE REFERENCES ({a_name} feed)")
+    print(f"  mean close    : ${mean_price:,.2f}")
+    print(f"  mean bar range: ${(x['High'] - x['Low']).mean():.3f}")
+    print(f"  mean ATR14    : ${atr.mean():.3f}   median ${atr.median():.3f}")
+
+    print(f"\nPER-BAR DIFFS  ({a_name} − {b_name}, $)")
+    diffs = {}
+    for col in ("Open", "High", "Low", "Close"):
+        d = x[col] - y[col]
+        diffs[col] = d
+        print(fmt_stats(col, d))
+
+    dclose = diffs["Close"].abs()
+    print(f"\n  |ΔClose| as % of mean ATR14 : "
+          f"mean {100 * dclose.mean() / atr.mean():.1f}%   "
+          f"p95 {100 * dclose.quantile(0.95) / atr.mean():.1f}%")
+
+    print(f"\nTHRESHOLD EXCEEDANCE  (|ΔClose|)")
+    for thr in (0.10, 0.30, 0.65, 1.00, 2.00, 5.00):
+        pct = 100 * (dclose > thr).mean()
+        print(f"  > ${thr:<5.2f}: {pct:6.2f}% of bars  ({(dclose > thr).sum():,})")
+
+    # direction agreement
+    a_dir = np.sign(x["Close"] - x["Open"])
+    b_dir = np.sign(y["Close"] - y["Open"])
+    both_nonflat = (a_dir != 0) & (b_dir != 0)
+    dir_agree = 100 * (a_dir[both_nonflat] == b_dir[both_nonflat]).mean()
+    print(f"\nAGREEMENT")
+    print(f"  bar direction (sign of close−open) agree: {dir_agree:.2f}%"
+          f"  (n={both_nonflat.sum():,})")
+
+    step = pd.Series(common, index=common).diff().median()
+    a_ret = x["Close"].pct_change()
+    b_ret = y["Close"].pct_change()
+    step_ok = pd.Series(common, index=common).diff().le(step + pd.Timedelta(seconds=1))
+    corr = a_ret[step_ok].corr(b_ret[step_ok])
+    print(f"  close-return correlation (consecutive bars): {corr:.5f}")
+
+    # by-hour
+    hours = common.hour
+    rows = []
+    for h in range(24):
+        m = hours == h
+        if not m.any():
+            continue
+        rows.append({"hour_utc": h, "bars": int(m.sum()),
+                     "mean_abs_dclose": dclose[m].mean(),
+                     "p95_abs_dclose": dclose[m].quantile(0.95)})
+    by_hour = pd.DataFrame(rows)
+    print(f"\nBY UTC HOUR  (mean |ΔClose|, $)")
+    print(by_hour.to_string(index=False,
+                            float_format=lambda v: f"{v:.3f}"))
+
+    print(f"\nBY MONTH  (mean |ΔClose|, $)")
+    months = dclose.groupby(common.tz_localize(None).to_period("M")).mean()
+    for p, v in months.items():
+        print(f"  {p}: {v:.3f}")
+
+    return {
+        "tf": tf.value, "a": a_name, "b": b_name,
+        "a_bars": len(a), "b_bars": len(b), "common": len(common),
+        "a_only": len(a_only), "b_only": len(b_only),
+        "dir_agree_pct": round(dir_agree, 3),
+        "ret_corr": round(corr, 5),
+        "mean_abs_dclose": dclose.mean(),
+        "p95_abs_dclose": dclose.quantile(0.95),
+        "max_abs_dclose": dclose.max(),
+        "dclose_pct_atr_mean": 100 * dclose.mean() / atr.mean(),
+    }
+
+
+def main():
+    ap = argparse.ArgumentParser()
+    ap.add_argument("--a", required=True)
+    ap.add_argument("--b", required=True)
+    ap.add_argument("--tfs", default="M15,H1,H4",
+                    help="comma-separated timeframe values (default M15,H1,H4)")
+    ap.add_argument("--days", type=int, default=365)
+    ap.add_argument("--end", default=None, help="UTC end date (default now)")
+    ap.add_argument("--no-cache", action="store_true")
+    args = ap.parse_args()
+
+    tfs = []
+    for s in args.tfs.split(","):
+        s = s.strip()
+        tf = _TF_LOOKUP.get(s)
+        if tf is None:
+            sys.exit(f"unknown timeframe '{s}'; use one of "
+                     f"{sorted(_TF_LOOKUP)}")
+        tfs.append(tf)
+
+    end = pd.Timestamp(args.end, tz="UTC") if args.end \
+        else pd.Timestamp.now(tz="UTC")
+    start = end - pd.Timedelta(days=args.days)
+
+    cfg = PlatformConfig()
+    pa = make_data_provider(cfg, args.a)
+    pb = make_data_provider(cfg, args.b)
+
+    summary = []
+    for tf in tfs:
+        print(f"\n{'─' * 80}\nWINDOW {start.date()} → {end.date()}  "
+              f"({args.days} days), {tf.value}")
+        try:
+            a = load_or_fetch(pa, args.a, tf, start, end, args.no_cache)
+        except Exception as e:
+            print(f"  !! {args.a} {tf.value} fetch failed: {e}")
+            continue
+        try:
+            b = load_or_fetch(pb, args.b, tf, start, end, args.no_cache)
+        except Exception as e:
+            print(f"  !! {args.b} {tf.value} fetch failed: {e}")
+            continue
+        r = compare(tf, args.a, args.b, a, b)
+        if r:
+            summary.append(r)
+
+    print("\n" + "=" * 80)
+    print("SUMMARY TABLE")
+    print("=" * 80)
+    sdf = pd.DataFrame(summary)
+    print(sdf.to_string(index=False,
+                        float_format=lambda v: f"{v:.4f}"))
+    out = RESEARCH_DIR / f"feed_divergence_{args.a}_vs_{args.b}.csv"
+    if len(sdf):
+        sdf.to_csv(out, index=False, float_format="%.6f")
+        print(f"\nSummary written to {out}")
+
+
+if __name__ == "__main__":
+    main()

scripts/migrate_signal_log_news.py

@@ -0,0 +1,177 @@
+#!/usr/bin/env python3
+"""
+migrate_signal_log_news.py — move inline news blobs into news_snapshots.
+
+Legacy rows store the full RSS headline list (~33 KB) and calendar events
+(~1.2 KB) inline on EVERY scan row; the same blob repeats ~5× because the
+news caches outlive a candle. This script interns each distinct blob once
+(content hash), stamps `headlines_hash` / `events_hash` on the row and
+NULLs the inline columns. Readers use the `signals_with_news` view or
+`signal_log.read_news`.
+
+Idempotent and resumable (batched transactions). Run --vacuum with BOTH
+live writers (telegram bot, dashboard publisher) stopped — VACUUM needs
+exclusive access and ~1× the final size of free disk.
+
+    python scripts/migrate_signal_log_news.py --dry-run
+    python scripts/migrate_signal_log_news.py
+    python scripts/migrate_signal_log_news.py --vacuum      # bots stopped
+"""
+import argparse
+import hashlib
+import os
+import sqlite3
+import sys
+from datetime import datetime, timezone
+
+sys.path.insert(0, os.path.dirname(os.path.dirname(os.path.abspath(__file__))))
+
+import app_config  # noqa: E402
+from signal_log import SignalLogger  # noqa: E402
+
+
+def _hash(text: str) -> str:
+    return hashlib.sha256(text.encode("utf-8")).hexdigest()
+
+
+def migrate(con: sqlite3.Connection, batch: int = 500) -> dict:
+    """Intern legacy inline blobs; returns counters. Caller owns the
+    connection (schema must already exist — open a SignalLogger once)."""
+    stats = {"rows": 0, "headlines_interned": 0, "events_interned": 0}
+    now = datetime.now(timezone.utc).isoformat()
+    while True:
+        rows = con.execute(
+            "SELECT id, headlines_json, events_json FROM signals"
+            " WHERE (headlines_json IS NOT NULL AND headlines_hash IS NULL)"
+            "    OR (events_json IS NOT NULL AND events_hash IS NULL)"
+            " ORDER BY id LIMIT ?", (batch,)).fetchall()
+        if not rows:
+            break
+        con.execute("BEGIN IMMEDIATE")
+        try:
+            for rowid, headlines, events in rows:
+                sets, params = [], []
+                if headlines is not None:
+                    h = _hash(headlines)
+                    cur = con.execute(
+                        "INSERT OR IGNORE INTO news_snapshots"
+                        " (hash, kind, body, first_seen, last_seen)"
+                        " VALUES (?, 'headlines', ?, ?, ?)",
+                        (h, headlines, now, now))
+                    stats["headlines_interned"] += cur.rowcount
+                    sets += ["headlines_hash=?", "headlines_json=NULL"]
+                    params.append(h)
+                if events is not None:
+                    h = _hash(events)
+                    cur = con.execute(
+                        "INSERT OR IGNORE INTO news_snapshots"
+                        " (hash, kind, body, first_seen, last_seen)"
+                        " VALUES (?, 'events', ?, ?, ?)",
+                        (h, events, now, now))
+                    stats["events_interned"] += cur.rowcount
+                    sets += ["events_hash=?", "events_json=NULL"]
+                    params.append(h)
+                params.append(rowid)
+                con.execute(
+                    f"UPDATE signals SET {', '.join(sets)} WHERE id=?",
+                    params)
+                stats["rows"] += 1
+            con.commit()
+        except Exception:
+            con.rollback()
+            raise
+    return stats
+
+
+def prune_orphans(con: sqlite3.Connection) -> int:
+    cur = con.execute(
+        "DELETE FROM news_snapshots WHERE hash NOT IN"
+        " (SELECT headlines_hash FROM signals WHERE headlines_hash IS NOT NULL"
+        "  UNION SELECT events_hash FROM signals WHERE events_hash IS NOT NULL)")
+    con.commit()
+    return cur.rowcount
+
+
+def prune_news_older_than(con: sqlite3.Connection, days: int) -> int:
+    """Drop the news reference (not the row) from rows older than N days,
+    then delete snapshots nothing references any more."""
+    # logged_at is ISO8601 with a 'T' and a +00:00 offset; SQLite's
+    # datetime() returns 'YYYY-MM-DD HH:MM:SS'. Comparing them as raw
+    # strings put every row from the cutoff CALENDAR DAY on the wrong
+    # side ('T' > ' '), so a full day never pruned. Normalise to the
+    # same shape first — all logged_at values are UTC (see _iso()).
+    cur = con.execute(
+        "UPDATE signals SET headlines_hash=NULL, events_hash=NULL"
+        " WHERE substr(replace(logged_at, 'T', ' '), 1, 19)"
+        " < datetime('now', ?)", (f"-{int(days)} days",))
+    con.commit()
+    prune_orphans(con)
+    return cur.rowcount
+
+
+def main(argv=None) -> int:
+    ap = argparse.ArgumentParser(description=__doc__.splitlines()[1])
+    ap.add_argument("--db", default=None)
+    ap.add_argument("--dry-run", action="store_true")
+    ap.add_argument("--batch", type=int, default=500)
+    ap.add_argument("--prune-news-older-than", type=int, default=None,
+                    metavar="DAYS")
+    ap.add_argument("--prune-orphans", action="store_true")
+    ap.add_argument("--vacuum", action="store_true",
+                    help="checkpoint + VACUUM (all writers must be stopped)")
+    ap.add_argument("--no-backup", action="store_true")
+    args = ap.parse_args(argv)
+    path = args.db or app_config.signal_log_path()
+    if not os.path.exists(path):
+        print(f"no database at {path}")
+        return 1
+    size0 = os.path.getsize(path)
+    if args.dry_run:
+        con = sqlite3.connect(f"file:{os.path.abspath(path)}?mode=ro", uri=True)
+        n = con.execute(
+            "SELECT COUNT(*) FROM signals WHERE headlines_json IS NOT NULL"
+            " OR events_json IS NOT NULL").fetchone()[0]
+        d = con.execute(
+            "SELECT COUNT(DISTINCT headlines_json) FROM signals"
+            " WHERE headlines_json IS NOT NULL").fetchone()[0]
+        print(f"{n} rows carry inline news blobs ({d} distinct headline "
+              f"blobs); file size {size0 / 1e6:.1f} MB")
+        con.close()
+        return 0
+    if not args.no_backup:
+        backup = f"{path}.pre-news-{datetime.now(timezone.utc):%Y%m%d}"
+        if not os.path.exists(backup):
+            src = sqlite3.connect(path)
+            dst = sqlite3.connect(backup)
+            src.backup(dst)
+            dst.close()
+            src.close()
+            print(f"backup written: {backup}")
+    lg = SignalLogger(path)          # applies the additive schema
+    lg.close()
+    con = sqlite3.connect(path)
+    con.execute("PRAGMA busy_timeout=10000")
+    try:
+        print("migrate:", migrate(con, batch=args.batch))
+        if args.prune_news_older_than is not None:
+            print("rows de-referenced:",
+                  prune_news_older_than(con, args.prune_news_older_than))
+        if args.prune_orphans:
+            print("orphan snapshots removed:", prune_orphans(con))
+        if args.vacuum:
+            try:
+                con.execute("PRAGMA wal_checkpoint(TRUNCATE)")
+                con.execute("VACUUM")
+            except sqlite3.OperationalError as exc:
+                print(f"VACUUM failed ({exc}) — stop the bot and dashboard "
+                      "publisher first, then re-run with --vacuum")
+                return 2
+    finally:
+        con.close()
+    print(f"size {size0 / 1e6:.1f} MB -> {os.path.getsize(path) / 1e6:.1f} MB"
+          + ("" if args.vacuum else " (run --vacuum to reclaim space)"))
+    return 0
+
+
+if __name__ == "__main__":
+    sys.exit(main())

scripts/shadow_scan.py

@@ -0,0 +1,194 @@
+#!/usr/bin/env python3
+"""
+Phase 4 — live shadow scan (data source change test)
+=====================================================
+Runs TWO data providers side-by-side at every candle close (M5/M15/H1)
+and logs how their signals diverge, so the feed switch can be validated
+under real fetch depths / caching / boundary timing for ≥2 weeks.
+
+Unlike the history studies, this uses the LIVE fetch path (fetch_ohlcv,
+forming-candle drop, freshness checks) for both providers, but calls
+`analyze_from_dataframes(use_news=False)` directly so the (provider-
+independent) news/DeepSeek budget is not burned on shadow scans. The
+weekend filter is applied here (skip the whole cycle), and the session/
+day/hour gates run inside analyze_from_dataframes as in the backtest.
+
+No orders, no Telegram, no signal-log writes — output is a side-by-side
+CSV plus a rolling agreement tally.
+
+Usage:
+    venv/bin/python scripts/shadow_scan.py --a oanda --b tiingo \
+        --tfs M5,M15,H1 --out logs/shadow_oanda_tiingo.csv --once
+    # production (2+ weeks):
+    nohup venv/bin/python scripts/shadow_scan.py --a oanda --b tiingo \
+        --out logs/shadow_oanda_tiingo.csv >> logs/shadow_scan.log 2>&1 &
+
+CSV columns (one row per TF per boundary):
+    ts, tf, a_<field>, b_<field> for direction/grade/setup/tag/entry/sl/tp1,
+    a_price, b_price, a_spread, b_spread, a_warnings, b_warnings,
+    dir_agree, grade_agree, setup_agree
+"""
+
+import argparse
+import csv
+import sys
+import time
+from datetime import datetime, timezone
+from pathlib import Path
+
+import pandas as pd
+
+sys.path.insert(0, str(Path(__file__).resolve().parent.parent))
+
+from config import PlatformConfig, Timeframe, XAUUSD
+from pair_signals.xauusd import make_config
+from providers import make_data_provider
+from providers.base import _is_forex_weekend
+from signal_aggregator import SignalAggregator, SignalReport
+
+_TF_LOOKUP = {"M5": Timeframe.M5, "M15": Timeframe.M15, "H1": Timeframe.H1,
+              "H4": Timeframe.H4}
+_TF_MINUTES = {Timeframe.M5: 5, Timeframe.M15: 15, Timeframe.H1: 60,
+               Timeframe.H4: 240}
+_CONFIRMS = {Timeframe.M5: [Timeframe.M15, Timeframe.H1],
+             Timeframe.M15: [Timeframe.M5, Timeframe.H1],
+             Timeframe.H1: [Timeframe.M15, Timeframe.H4],
+             Timeframe.H4: [Timeframe.H1, Timeframe.D1]}
+
+FIELDS = ["direction", "grade", "setup", "tag", "entry", "sl", "tp1"]
+
+
+def seconds_until_boundary(interval_minutes: int, buffer: int = 10) -> float:
+    now = datetime.now(timezone.utc)
+    elapsed = (now.minute % interval_minutes) * 60 + now.second \
+        + now.microsecond / 1e6
+    return interval_minutes * 60 - elapsed + buffer
+
+
+def due_timeframes(wake: datetime, tfs) -> list:
+    minutes = wake.hour * 60 + wake.minute
+    floored = minutes - (minutes % 5)
+    return [tf for tf in tfs if floored % _TF_MINUTES[tf] == 0]
+
+
+def tag_of(report) -> str:
+    fb = report.bullish_reasons[0] if report.bullish_reasons else ""
+    fbe = report.bearish_reasons[0] if report.bearish_reasons else ""
+    if "XAU PULLBACK" in fb or "XAU PULLBACK" in fbe:
+        return "XAU_PB"
+    if "XAU CHART PATTERN" in fb or "XAU CHART PATTERN" in fbe:
+        return "XAU_CP"
+    if "XAU STRATEGY" in fb:
+        return "XAU"
+    return "GENERIC"
+
+
+def scan(provider, symbol, tf) -> SignalReport:
+    confirms = _CONFIRMS[tf]
+    cfg = make_config(tf, confirms)
+    agg = SignalAggregator(cfg)
+    agg.data_provider = provider
+    dfs = {}
+    for t in [tf] + confirms:
+        dfs[t] = provider.fetch_ohlcv(symbol, t)
+    return agg.analyze_from_dataframes(symbol, dfs, use_news=False)
+
+
+def extract(provider, report) -> dict:
+    s = report.trade_setup
+    return {
+        "direction": report.final_signal.value,
+        "grade": report.signal_quality,
+        "setup": 1 if s is not None else 0,
+        "tag": tag_of(report) if s else "",
+        "entry": round(s.entry_price, 3) if s else "",
+        "sl": round(s.stop_loss, 3) if s else "",
+        "tp1": round(s.take_profit_1, 3) if s else "",
+        "price": round(report.current_price, 3),
+        "spread": round(provider.get_spread(XAUUSD), 3),
+        "warnings": " | ".join(report.warnings)[:200],
+    }
+
+
+def main():
+    ap = argparse.ArgumentParser()
+    ap.add_argument("--a", default="oanda")
+    ap.add_argument("--b", default="tiingo")
+    ap.add_argument("--tfs", default="M5,M15,H1")
+    ap.add_argument("--out", default="logs/shadow_scan.csv")
+    ap.add_argument("--once", action="store_true",
+                    help="one immediate scan cycle, then exit")
+    args = ap.parse_args()
+
+    tfs = [_TF_LOOKUP[s.strip()] for s in args.tfs.split(",")]
+    pa = make_data_provider(PlatformConfig(), args.a)
+    pb = make_data_provider(PlatformConfig(), args.b)
+
+    out_path = Path(args.out)
+    out_path.parent.mkdir(parents=True, exist_ok=True)
+    new_file = not out_path.exists()
+    tally = {"cycles": 0, "dir": 0, "grade": 0, "setup": 0}
+
+    with open(out_path, "a", newline="") as fh:
+        w = csv.writer(fh)
+        if new_file:
+            w.writerow(["ts", "tf"] + [f"a_{x}" for x in FIELDS + ["price", "spread", "warnings"]]
+                       + [f"b_{x}" for x in FIELDS + ["price", "spread", "warnings"]]
+                       + ["dir_agree", "grade_agree", "setup_agree"])
+
+        while True:
+            if args.once:
+                wake = datetime.now(timezone.utc)
+            else:
+                wait = seconds_until_boundary(5, buffer=10)
+                time.sleep(wait)
+                wake = datetime.now(timezone.utc)
+
+            if _is_forex_weekend():
+                if args.once:
+                    print("weekend — nothing to scan")
+                    return
+                continue
+
+            for tf in due_timeframes(wake, tfs):
+                try:
+                    ra = scan(pa, XAUUSD, tf)
+                except Exception as e:
+                    print(f"[{wake:%H:%M:%S}] {args.a} {tf.value} FAILED: {e}")
+                    continue
+                try:
+                    rb = scan(pb, XAUUSD, tf)
+                except Exception as e:
+                    print(f"[{wake:%H:%M:%S}] {args.b} {tf.value} FAILED: {e}")
+                    continue
+
+                ea, eb = extract(pa, ra), extract(pb, rb)
+                da = ea["direction"] == eb["direction"]
+                ga = ea["grade"] == eb["grade"]
+                sa = ea["setup"] == eb["setup"]
+                tally["cycles"] += 1
+                tally["dir"] += da; tally["grade"] += ga; tally["setup"] += sa
+
+                row = [wake.isoformat(), tf.value]
+                row += [ea[x] for x in FIELDS + ["price", "spread", "warnings"]]
+                row += [eb[x] for x in FIELDS + ["price", "spread", "warnings"]]
+                row += [int(da), int(ga), int(sa)]
+                w.writerow(row)
+                fh.flush()
+
+                print(f"[{wake:%Y-%m-%d %H:%M:%S}] {tf.value:<5} "
+                      f"{args.a}={ea['direction']:+d}/{ea['grade'] or '-':>2} "
+                      f"{args.b}={eb['direction']:+d}/{eb['grade'] or '-':>2} "
+                      f"dir={'OK' if da else 'FLIP'} "
+                      f"| Δprice {abs(ea['price']-eb['price']):.2f} "
+                      f"| agree dir {tally['dir']}/{tally['cycles']} "
+                      f"grade {tally['grade']}/{tally['cycles']} "
+                      f"setup {tally['setup']}/{tally['cycles']}")
+
+            if args.once:
+                print("shadow scan complete (--once)")
+                return
+
+
+if __name__ == "__main__":
+    main()

scripts/signal_parity_study.py

@@ -0,0 +1,246 @@
+#!/usr/bin/env python3
+"""
+Phase 2 — signal-level replay parity (data source change test)
+===============================================================
+Replays the REAL decision core (`SignalAggregator.analyze_from_dataframes`,
+news off, use_news=False) over the same window for two providers and
+compares the *decisions* bar-by-bar: direction, composite/final score,
+quality grade, gate/setup presence, strategy tag, and entry/SL/TP levels.
+
+This turns the Phase-1 price-diff stats into flip rates. Feeds are read
+from the Phase-1 parquet cache (scripts/feed_divergence_study.py writes
+them); pass --no-cache-fetch only if you want to fetch fresh via
+fetch_range first.
+
+Slicing matches gpu_backtest.py exactly: decision at close of bar `pos`,
+each TF sliced to bars fully closed by that decision time, primary at the
+shared grid, so the two feeds are compared on identical timestamps.
+
+Usage:
+    venv/bin/python scripts/signal_parity_study.py --a oanda --b tiingo \
+        --tf M15 --stride 8 --days 365
+"""
+
+import argparse
+import sys
+import time
+from pathlib import Path
+
+import numpy as np
+import pandas as pd
+
+sys.path.insert(0, str(Path(__file__).resolve().parent.parent))
+
+from config import Timeframe, XAUUSD
+from pair_signals.xauusd import make_config
+from signal_aggregator import SignalAggregator
+
+REPO = Path(__file__).resolve().parent.parent
+CACHE_DIR = REPO / "backtest" / "cache"
+RESEARCH_DIR = REPO / "backtest" / "research"
+
+BAR_SECONDS = {
+    Timeframe.M1: 60, Timeframe.M5: 300, Timeframe.M15: 900,
+    Timeframe.M30: 1800, Timeframe.H1: 3600, Timeframe.H4: 14400,
+    Timeframe.D1: 86400, Timeframe.W1: 604800,
+}
+_TF_LOOKUP = {t.value: t for t in Timeframe}
+_TF_LOOKUP.update({"M1": Timeframe.M1, "M5": Timeframe.M5,
+                   "M15": Timeframe.M15, "M30": Timeframe.M30,
+                   "H1": Timeframe.H1, "H4": Timeframe.H4,
+                   "D1": Timeframe.D1, "W1": Timeframe.W1})
+_CONFIRMS = {
+    Timeframe.M5: [Timeframe.M15, Timeframe.H1],
+    Timeframe.M15: [Timeframe.M5, Timeframe.H1],
+    Timeframe.H1: [Timeframe.M15, Timeframe.H4],
+    Timeframe.H4: [Timeframe.H1, Timeframe.D1],
+}
+
+
+def load_cached(name: str, tf: Timeframe, start: pd.Timestamp,
+                end: pd.Timestamp) -> pd.DataFrame:
+    cp = CACHE_DIR / f"{name}_XAU_USD_{tf.value}_{start:%Y%m%d}_{end:%Y%m%d}.parquet"
+    if not cp.exists():
+        sys.exit(f"missing cache {cp}; run scripts/feed_divergence_study.py "
+                 f"--a oanda --b {name} --tfs {tf.value} --days ... first")
+    return pd.read_parquet(cp)
+
+
+def tag_of(report) -> str:
+    first_bull = report.bullish_reasons[0] if report.bullish_reasons else ""
+    first_bear = report.bearish_reasons[0] if report.bearish_reasons else ""
+    if "XAU PULLBACK" in first_bull or "XAU PULLBACK" in first_bear:
+        return "XAU_PB"
+    if "XAU CHART PATTERN" in first_bull or "XAU CHART PATTERN" in first_bear:
+        return "XAU_CP"
+    if "XAU STRATEGY" in first_bull:
+        return "XAU"
+    return "GENERIC"
+
+
+def decide(agg, dfs, tss, stride, warmup) -> list:
+    """Replay decisions over the shared grid, returning one row per bar."""
+    primary = dfs[agg.config.primary_timeframe]
+    dur = pd.Timedelta(seconds=BAR_SECONDS[agg.config.primary_timeframe])
+
+    # positions of sampled bars in the primary frame
+    primary_pos = primary.index.get_indexer(tss)
+    rows = []
+    for i in range(0, len(tss), stride):
+        ts = tss[i]
+        pos = int(primary_pos[i])
+        if pos < warmup:
+            continue
+        slices = {}
+        for tf, full in dfs.items():
+            cutoff = ts  # decision_time = ts + dur; bars closed by ts
+            cut = int(full.index.searchsorted(cutoff, side="right"))
+            if cut <= 0:
+                continue
+            slices[tf] = full.iloc[:cut]
+        if agg.config.primary_timeframe not in slices:
+            continue
+        r = agg.analyze_from_dataframes(XAUUSD, slices, use_news=False)
+        ts_setup = r.trade_setup
+        rows.append({
+            "ts": ts,
+            "direction": r.final_signal.value,
+            "final_score": r.final_score,
+            "primary_score": r.primary_score,
+            "grade": r.signal_quality,
+            "has_setup": ts_setup is not None,
+            "tag": tag_of(r) if ts_setup else "",
+            "entry": ts_setup.entry_price if ts_setup else np.nan,
+            "sl": ts_setup.stop_loss if ts_setup else np.nan,
+            "tp1": ts_setup.take_profit_1 if ts_setup else np.nan,
+            "tp2": ts_setup.take_profit_2 if ts_setup else np.nan,
+        })
+    return rows
+
+
+def main():
+    ap = argparse.ArgumentParser()
+    ap.add_argument("--a", required=True)
+    ap.add_argument("--b", required=True)
+    ap.add_argument("--tf", default="M15")
+    ap.add_argument("--stride", type=int, default=1,
+                    help="sample every Nth shared bar (default 1)")
+    ap.add_argument("--days", type=int, default=365)
+    ap.add_argument("--end", default=None)
+    ap.add_argument("--warmup", type=int, default=300)
+    args = ap.parse_args()
+
+    tf = _TF_LOOKUP.get(args.tf)
+    if tf is None:
+        sys.exit(f"unknown tf {args.tf}")
+    confirms = _CONFIRMS[tf]
+    tfs = [tf] + confirms
+
+    end = pd.Timestamp(args.end, tz="UTC") if args.end \
+        else pd.Timestamp.now(tz="UTC")
+    start = end - pd.Timedelta(days=args.days)
+
+    cfg = make_config(tf, confirms)
+    agg = SignalAggregator(cfg)
+
+    a_dfs, b_dfs = {}, {}
+    for t in tfs:
+        a_dfs[t] = load_cached(args.a, t, start, end)
+        b_dfs[t] = load_cached(args.b, t, start, end)
+
+    # shared grid = primary timestamps present in both feeds
+    shared = a_dfs[tf].index.intersection(b_dfs[tf].index)
+
+    t0 = time.time()
+    print(f"replaying {len(shared)} shared {tf.value} bars "
+          f"(stride {args.stride} → ~{len(shared)//args.stride}) on {args.a}...")
+    a_rows = decide(agg, a_dfs, shared, args.stride, args.warmup)
+    print(f"  {args.a}: {len(a_rows)} decisions in {time.time()-t0:.1f}s")
+    t0 = time.time()
+    b_rows = decide(agg, b_dfs, shared, args.stride, args.warmup)
+    print(f"  {args.b}: {len(b_rows)} decisions in {time.time()-t0:.1f}s")
+
+    A = pd.DataFrame(a_rows).set_index("ts")
+    B = pd.DataFrame(b_rows).set_index("ts")
+    common = A.index.intersection(B.index)
+    A, B = A.loc[common], B.loc[common]
+    n = len(common)
+
+    print(f"\n{'=' * 78}")
+    print(f"SIGNAL-LEVEL PARITY — {tf.value} — {args.a} vs {args.b} "
+          f"({n:,} bars)")
+    print("=" * 78)
+
+    def pct(x): return f"{100 * x:.2f}%"
+
+    same_dir = (A["direction"] == B["direction"])
+    same_grade = (A["grade"] == B["grade"])
+    same_setup = (A["has_setup"] == B["has_setup"])
+    same_all = same_dir & same_grade & same_setup
+
+    print(f"\nIDENTITY")
+    print(f"  same direction        : {pct(same_dir.mean())}")
+    print(f"  same grade            : {pct(same_grade.mean())}")
+    print(f"  same setup presence   : {pct(same_setup.mean())}")
+    print(f"  same dir+grade+setup  : {pct(same_all.mean())}")
+
+    both_sig = (A["direction"] != 0) & (B["direction"] != 0)
+    flips = both_sig & (A["direction"] != B["direction"])
+    opp = flips & (A["direction"] == -B["direction"])
+    print(f"\nDIRECTION")
+    print(f"  bars both actionable    : {both_sig.sum():,} "
+          f"({pct(both_sig.mean())})")
+    print(f"  direction flips         : {flips.sum():,} "
+          f"({pct(flips.sum()/max(both_sig.sum(),1))} of actionable)")
+    print(f"  opposite-sign flips     : {opp.sum():,}")
+
+    grade_a = set(A.loc[A["has_setup"], "grade"])
+    print(f"\nGRADE (of {args.a} setups, grades seen: {sorted(grade_a)})")
+    changed = A.loc[both_sig, "grade"] != B.loc[both_sig, "grade"]
+    print(f"  grade changed on actionable bars: "
+          f"{changed.sum():,} ({pct(changed.mean())})")
+
+    # score deltas on shared bars
+    ds = (A["final_score"] - B["final_score"]).abs()
+    dp = (A["primary_score"] - B["primary_score"]).abs()
+    print(f"\nSCORE Δ  ({args.a} − {args.b})")
+    print(f"  final_score   mean {ds.mean():6.2f}  p95 {ds.quantile(.95):6.2f}  "
+          f"max {ds.max():6.2f}")
+    print(f"  primary_score mean {dp.mean():6.2f}  p95 {dp.quantile(.95):6.2f}  "
+          f"max {dp.max():6.2f}")
+
+    # setup price diffs on bars where BOTH generated a setup
+    both_setup = A["has_setup"] & B["has_setup"]
+    if both_setup.any():
+        print(f"\nSETUP LEVELS (both-setup bars: {both_setup.sum():,})")
+        for col, label in (("entry", "entry"), ("sl", "SL"), ("tp1", "TP1"),
+                           ("tp2", "TP2")):
+            d = (A.loc[both_setup, col] - B.loc[both_setup, col]).abs()
+            print(f"  {label:<6} mean ${d.mean():6.3f}  p95 ${d.quantile(.95):7.3f}  "
+                  f"max ${d.max():7.2f}")
+
+    # tag disagreement on both-setup bars
+    if both_setup.any():
+        same_tag = (A.loc[both_setup, "tag"] == B.loc[both_setup, "tag"])
+        print(f"  same strategy tag      : {pct(same_tag.mean())}")
+
+    # where do flips cluster (hour)?
+    flip_hours = A.loc[flips].index.hour.value_counts().sort_index()
+    if len(flip_hours):
+        print(f"\nFLIPS BY UTC HOUR")
+        for h, c in flip_hours.items():
+            print(f"  {h:02d}: {c}")
+
+    out = {
+        "tf": tf.value, "a": args.a, "b": args.b, "bars": n,
+        "same_dir": same_dir.mean(), "same_grade": same_grade.mean(),
+        "same_setup": same_setup.mean(), "same_all": same_all.mean(),
+        "flip_rate_actionable": flips.sum() / max(both_sig.sum(), 1),
+        "mean_abs_final_score_delta": ds.mean(),
+        "mean_abs_primary_score_delta": dp.mean(),
+    }
+    print(f"\n{out}")
+
+
+if __name__ == "__main__":
+    main()

static/dashboard.css

@@ -0,0 +1,351 @@
+/* Gold Desk — XAU/USD dashboard styles
+ *
+ * Builds on Bootstrap 5.3 color modes (data-bs-theme). Only app-specific
+ * tokens and components live here; layout, typography, tables, badges,
+ * cards, alerts, modals, etc. come from Bootstrap.
+ */
+
+:root {
+  --xau-font-sans: "Inter", system-ui, -apple-system, "Segoe UI", sans-serif;
+  --xau-font-mono: "JetBrains Mono", ui-monospace, "SF Mono", Consolas, Menlo, monospace;
+  --xau-gold: #f5b301;
+}
+
+/* Forced dark theme (owner decision 2026-08-16): dark background, pure
+ * white text (no gray), vivid red/green for data accents. Loaded after
+ * Bootstrap so these overrides win. */
+[data-bs-theme="dark"] {
+  --bs-body-bg: #0a0e14;
+  --bs-secondary-bg: #111823;
+  --bs-tertiary-bg: #0e141d;
+  --bs-body-color: #ffffff;
+  --bs-emphasis-color: #ffffff;
+  --bs-heading-color: #ffffff;
+  --bs-secondary-color: rgba(255, 255, 255, 0.92);
+  --bs-tertiary-color: rgba(255, 255, 255, 0.8);
+  --bs-border-color: rgba(255, 255, 255, 0.14);
+  --bs-card-bg: #0e141d;
+  --bs-card-border-color: rgba(255, 255, 255, 0.12);
+  --bs-modal-bg: #0e141d;
+  --bs-success: #22c55e;
+  --bs-success-rgb: 34, 197, 94;
+  --bs-success-text-emphasis: #4ade80;
+  --bs-success-bg-subtle: rgba(34, 197, 94, 0.16);
+  --bs-success-border-subtle: rgba(34, 197, 94, 0.45);
+  --bs-danger: #ef4444;
+  --bs-danger-rgb: 239, 68, 68;
+  --bs-danger-text-emphasis: #f87171;
+  --bs-danger-bg-subtle: rgba(239, 68, 68, 0.16);
+  --bs-danger-border-subtle: rgba(239, 68, 68, 0.45);
+}
+
+html, body {
+  font-family: var(--xau-font-sans);
+  letter-spacing: -0.005em;
+  -webkit-font-smoothing: antialiased;
+  background-color: var(--bs-body-bg);
+  color: var(--bs-body-color);
+}
+
+.num, .xau-price, .xau-symbol {
+  font-family: var(--xau-font-mono);
+  font-variant-numeric: tabular-nums;
+}
+
+.xau-symbol {
+  font-size: 0.95rem;
+  font-weight: 700;
+  letter-spacing: 0.08em;
+  color: var(--xau-gold);
+}
+
+.xau-price {
+  line-height: 1;
+  color: #ffffff;
+}
+
+.live-dot {
+  display: inline-block;
+  width: 8px;
+  height: 8px;
+  border-radius: 50%;
+  background: var(--bs-success);
+  margin-right: 5px;
+  animation: pulse 2.4s ease-in-out infinite;
+}
+
+.live-dot.closed {
+  background: var(--bs-warning);
+  animation: none;
+}
+
+@keyframes pulse {
+  50% { opacity: 0.35; }
+}
+
+/* signal-coloured accents */
+.sig-buy   { --xau-signal: var(--bs-success); }
+.sig-sell  { --xau-signal: var(--bs-danger); }
+.sig-neutral { --xau-signal: var(--bs-secondary); }
+
+.border-signal {
+  border-left: 3px solid var(--xau-signal, var(--bs-secondary)) !important;
+}
+
+.text-signal {
+  color: var(--xau-signal, var(--bs-secondary)) !important;
+}
+
+.tf-name {
+  font-family: var(--xau-font-mono);
+  font-weight: 700;
+  font-size: 1.1rem;
+  color: var(--bs-primary);
+  letter-spacing: 0.05em;
+}
+
+.tf-selector {
+  display: grid;
+  grid-template-columns: repeat(4, minmax(0, 1fr));
+  gap: 0.5rem;
+  margin-bottom: 0.75rem;
+}
+
+.tf-select {
+  min-width: 0;
+  padding: 0.7rem 0.8rem;
+  border: 1px solid var(--bs-border-color);
+  border-radius: var(--bs-border-radius);
+  background: var(--bs-tertiary-bg);
+  color: var(--bs-body-color);
+  text-align: left;
+  transition: border-color 0.15s ease, background-color 0.15s ease, box-shadow 0.15s ease;
+}
+
+.tf-select:hover {
+  border-color: rgba(var(--bs-primary-rgb), 0.65);
+}
+
+.tf-select[aria-pressed="true"] {
+  border-color: var(--xau-gold);
+  background: rgba(245, 179, 1, 0.09);
+  box-shadow: inset 0 0 0 1px rgba(245, 179, 1, 0.16);
+}
+
+.tf-select .tf-name {
+  font-size: 1rem;
+}
+
+.workspace-card {
+  overflow: hidden;
+}
+
+.workspace-signal {
+  font-family: var(--xau-font-mono);
+  font-size: clamp(1.55rem, 3vw, 2.35rem);
+  font-weight: 700;
+  letter-spacing: 0.03em;
+  line-height: 1;
+}
+
+.workspace-meta {
+  min-width: 0;
+}
+
+.workspace-reasons > div + div {
+  margin-top: 0.35rem;
+}
+
+/* score bar: zero-centered diverging with midpoint tick */
+.scorebar {
+  display: flex;
+  align-items: center;
+  gap: 0.75rem;
+}
+
+.scorebar .gauge-container {
+  position: relative;
+  flex: 1;
+  min-width: 60px;
+  height: 10px;
+  border-radius: 5px;
+  background: var(--bs-secondary-bg);
+  overflow: hidden;
+}
+
+.scorebar .gauge-container::after {
+  content: "";
+  position: absolute;
+  left: 50%;
+  top: 0;
+  bottom: 0;
+  width: 2px;
+  background: var(--bs-secondary);
+  transform: translateX(-1px);
+}
+
+.scorebar .fill {
+  position: absolute;
+  top: 0;
+  bottom: 0;
+  border-radius: 4px;
+  width: var(--score-width);
+  transition: width 0.25s ease;
+}
+
+.scorebar .fill.pos {
+  left: 50%;
+  background: var(--bs-success);
+}
+
+.scorebar .fill.neg {
+  right: 50%;
+  background: var(--bs-danger);
+}
+
+.scorebar .val {
+  width: 58px;
+  text-align: right;
+  font-size: 0.95rem;
+  font-weight: 500;
+}
+
+.scorebar .lbl {
+  font-size: 0.8rem;
+  color: var(--bs-secondary);
+  width: 44px;
+}
+
+/* rail chips in hero */
+.rail-chip {
+  border: 1px solid var(--bs-border-color);
+  border-radius: var(--bs-border-radius);
+  background: var(--bs-tertiary-bg);
+  color: var(--bs-body-color);
+  font-family: var(--xau-font-mono);
+  font-size: 0.9rem;
+  line-height: 1.3;
+  padding: 0.3rem 0.5rem;
+}
+
+.rail-chip:hover, .rail-chip[aria-current="true"] {
+  border-color: var(--xau-gold);
+}
+
+.rail-chip:focus-visible, .tf-select:focus-visible, .wire-item:focus-visible,
+.cal-event:focus-visible {
+  outline: 2px solid var(--xau-gold);
+  outline-offset: 2px;
+}
+
+/* detail dialog sections */
+.d-stat .k {
+  display: block;
+  font-size: 0.7rem;
+  letter-spacing: 0.08em;
+  text-transform: uppercase;
+  color: var(--bs-secondary);
+  font-weight: 600;
+}
+
+.d-stat .v {
+  font-size: 1.15rem;
+  font-weight: 600;
+}
+
+.d-table {
+  font-size: 0.9rem;
+}
+
+.detail-table {
+  overflow-x: auto;
+}
+
+.d-table .ind-k {
+  font-family: var(--xau-font-mono);
+  font-weight: 600;
+  white-space: nowrap;
+}
+
+.wbar {
+  display: block;
+  height: 4px;
+  border-radius: 2px;
+  background: var(--bs-secondary-bg);
+  margin-top: 3px;
+}
+
+.wbar > i {
+  display: block;
+  height: 100%;
+  border-radius: 2px;
+  background: var(--bs-primary);
+}
+
+/* news */
+.wire-item {
+  display: flex;
+  width: 100%;
+  border: 0;
+  border-radius: var(--bs-border-radius);
+  background: transparent;
+  color: inherit;
+  cursor: pointer;
+  font: inherit;
+  text-align: start;
+}
+
+.wire-item:hover {
+  background: rgba(var(--bs-primary-rgb), 0.08);
+}
+
+.cal-event {
+  display: block;
+  width: 100%;
+  padding: 0;
+  border: 0;
+  background: transparent;
+  color: inherit;
+  text-align: inherit;
+}
+
+.cal-event:hover {
+  color: var(--bs-primary);
+  text-decoration: underline;
+}
+
+.wire-score {
+  font-family: var(--xau-font-mono);
+  font-size: 0.85rem;
+  font-weight: 700;
+  padding: 2px 7px;
+  border-radius: 5px;
+  white-space: nowrap;
+}
+
+/* calendar highlight for next upcoming event */
+.cal-next-up td:first-child {
+  box-shadow: inset 4px 0 0 var(--bs-primary);
+  padding-left: 0.75rem !important;
+}
+
+/* reduced motion */
+@media (prefers-reduced-motion: reduce) {
+  .live-dot { animation: none; }
+  .scorebar .fill { transition: none; }
+  * { transition: none !important; }
+}
+
+@media (max-width: 575.98px) {
+  .tf-selector {
+    grid-template-columns: repeat(2, minmax(0, 1fr));
+  }
+
+  .workspace-card .card-body {
+    padding: 1rem;
+  }
+
+  .rail-chip {
+    flex: 1 1 calc(50% - 0.5rem);
+  }
+}

static/dashboard.html

@@ -0,0 +1,149 @@
+<!DOCTYPE html>
+<html lang="en" data-bs-theme="dark">
+<head>
+<meta charset="utf-8">
+<meta name="viewport" content="width=device-width, initial-scale=1">
+<title>Gold Desk — XAU/USD</title>
+<link rel="icon" href="data:image/svg+xml,%3Csvg xmlns='http://www.w3.org/2000/svg' viewBox='0 0 16 16'%3E%3Ccircle cx='8' cy='8' r='7' fill='%23f5b301'/%3E%3C/svg%3E">
+<link rel="preconnect" href="https://fonts.googleapis.com">
+<link rel="preconnect" href="https://fonts.gstatic.com" crossorigin>
+<link rel="stylesheet" href="https://cdn.jsdelivr.net/npm/bootstrap@5.3.3/dist/css/bootstrap.min.css" crossorigin="anonymous">
+<link rel="stylesheet" href="https://cdn.jsdelivr.net/npm/bootstrap-icons@1.11.3/font/bootstrap-icons.css" crossorigin="anonymous">
+<link rel="stylesheet" href="https://fonts.googleapis.com/css2?family=Inter:wght@400;500;600;700&family=JetBrains+Mono:wght@400;500;700&display=swap">
+<link rel="stylesheet" href="dashboard.css">
+</head>
+<body>
+
+<div class="container-fluid container-xl py-4">
+
+  <header class="card mb-3" id="hero">
+    <div class="card-body d-flex align-items-center flex-wrap gap-3">
+      <div>
+        <h1 class="xau-symbol mb-1"><i class="bi bi-coin me-1"></i>XAU/USD</h1>
+        <div class="display-5 fw-semibold xau-price" id="hero-price">—</div>
+      </div>
+      <div class="d-flex flex-wrap gap-2" id="hero-rail" aria-label="Select timeframe"></div>
+      <div class="ms-auto text-end">
+        <div id="hero-status" role="status"><span class="live-dot"></span>…</div>
+        <div class="small text-secondary"><i class="bi bi-hourglass-split me-1"></i>M5 close in <span class="num" id="hero-cd">--:--</span></div>
+        <div class="small text-secondary" id="hero-updated">updated —</div>
+      </div>
+    </div>
+  </header>
+
+  <div class="alert alert-warning d-none" id="banner-closed" role="alert">
+    <i class="bi bi-moon-fill me-2"></i>Market closed (weekend) — showing Friday's last state. Calendar and news stay live.
+  </div>
+  <div class="alert alert-danger d-none" id="banner-fail" role="alert"></div>
+  <div class="visually-hidden" id="dashboard-live" aria-live="polite"></div>
+
+  <main>
+    <section class="card mb-4 d-none" id="context" aria-label="Market context">
+      <div class="card-body d-flex flex-wrap gap-3 align-items-start">
+        <div class="flex-grow-1" id="ctx-blurb"><strong>Market context</strong></div>
+        <div class="d-flex flex-wrap gap-2" id="ctx-chips"></div>
+      </div>
+    </section>
+
+    <section class="mb-4" aria-labelledby="workspace-heading">
+      <h2 class="text-uppercase text-secondary small fw-bold mb-3 d-flex align-items-center gap-2" id="workspace-heading">
+        <i class="bi bi-crosshair text-primary"></i>
+        Signal workspace
+        <span class="fw-normal text-body-secondary" style="text-transform:none">— select a timeframe</span>
+      </h2>
+      <div id="tf-grid"></div>
+    </section>
+
+    <section aria-labelledby="fundamentals-heading">
+      <h2 class="text-uppercase text-secondary small fw-bold mb-3 mt-4 d-flex align-items-center gap-2" id="fundamentals-heading">
+        <i class="bi bi-globe2 text-primary"></i>Fundamentals
+      </h2>
+      <div class="row g-3">
+        <div class="col-lg-8">
+          <div class="card h-100">
+            <div class="card-body">
+              <div class="d-flex align-items-center gap-3 mb-3 flex-wrap">
+                <h3 class="h5 card-title mb-0"><i class="bi bi-calendar-event me-2 text-primary"></i>Economic calendar</h3>
+                <span class="text-secondary small" id="calendar-note">times local · USD events</span>
+                <div class="form-check form-switch ms-auto">
+                  <input class="form-check-input" type="checkbox" id="hi-only">
+                  <label class="form-check-label small" for="hi-only">high impact only</label>
+                </div>
+              </div>
+              <div class="table-responsive">
+                <table class="table table-sm" id="cal-table">
+                  <thead>
+                    <tr>
+                      <th>Time</th>
+                      <th>Cur</th>
+                      <th>Impact</th>
+                      <th>Event</th>
+                      <th class="text-end">Fcst</th>
+                      <th class="text-end">Prev</th>
+                      <th class="text-end">Actual</th>
+                    </tr>
+                  </thead>
+                  <tbody id="cal-body"><tr><td colspan="7" class="text-secondary text-center py-3">Loading calendar…</td></tr></tbody>
+                </table>
+              </div>
+            </div>
+          </div>
+        </div>
+        <div class="col-lg-4">
+          <div class="card h-100">
+            <div class="card-body">
+              <div class="d-flex align-items-center gap-3 mb-3 flex-wrap">
+                <h3 class="h5 card-title mb-0"><i class="bi bi-newspaper me-2 text-primary"></i>Newswire</h3>
+                <span class="text-secondary small" id="wire-note"></span>
+              </div>
+              <div id="wire-total" class="d-none"></div>
+              <div id="wire"></div>
+            </div>
+          </div>
+        </div>
+      </div>
+    </section>
+  </main>
+
+  <footer class="mt-4 pt-3 border-top text-secondary small">
+    <i class="bi bi-info-circle me-1"></i>
+    Signals are generated on closed candles only and refresh at candle boundaries
+    (M5/M15/H1/H4). Not financial advice — educational and research use only.
+  </footer>
+
+</div>
+
+<div class="modal fade" id="tf-detail" tabindex="-1" aria-labelledby="detail-title" aria-hidden="true">
+  <div class="modal-dialog modal-xl modal-dialog-scrollable">
+    <div class="modal-content">
+      <div class="modal-header">
+        <div>
+          <h5 class="modal-title mb-0" id="detail-title">—</h5>
+          <div class="small text-secondary">
+            <span class="num" id="detail-price">—</span>
+            <span id="detail-sub"></span>
+          </div>
+        </div>
+        <button id="detail-close" type="button" class="btn-close" aria-label="Close detail view"></button>
+      </div>
+      <div class="modal-body" id="detail-body"></div>
+    </div>
+  </div>
+</div>
+
+<div class="modal fade" id="news-detail" tabindex="-1" aria-labelledby="nd-title" aria-hidden="true">
+  <div class="modal-dialog modal-lg modal-dialog-scrollable">
+    <div class="modal-content">
+      <div class="modal-header">
+        <h5 class="modal-title" id="nd-title">News detail</h5>
+        <button id="nd-close" type="button" class="btn-close" aria-label="Close news detail"></button>
+      </div>
+      <div class="modal-body" id="nd-body"></div>
+    </div>
+  </div>
+</div>
+
+<script src="https://cdn.jsdelivr.net/npm/bootstrap@5.3.3/dist/js/bootstrap.bundle.min.js" crossorigin="anonymous"></script>
+<script src="dashboard.js"></script>
+</body>
+</html>

static/dashboard.js

@@ -0,0 +1,962 @@
+"use strict";
+/* Gold Desk — XAU/USD dashboard frontend
+ *
+ * Reads snapshot.json + news.json published by dashboard_publisher.py and
+ * renders a responsive, keyboard-accessible dashboard. Uses Bootstrap 5 for
+ * layout/components/modals and a small custom CSS file for app-specific
+ * visuals (score bar, signal colours, mono fonts).
+ *
+ * This file is intentionally self-contained: it mirrors a few constants from
+ * the Python signal pipeline so the HTML does not depend on telegram_bot.
+ */
+
+const IND_ORDER = ["ema_trend", "rsi", "macd", "bollinger", "adx", "roc", "keltner", "divergence", "candlestick"];
+const IND_SHORT = {
+  ema_trend: "EMA", rsi: "RSI", macd: "MACD", bollinger: "BB",
+  adx: "ADX", roc: "ROC", keltner: "KC", divergence: "Div", candlestick: "Cndl"
+};
+const ARROW = { STRONG_BUY: "⇈", BUY: "↑", NEUTRAL: "→", SELL: "↓", STRONG_SELL: "⇊" };
+const TF_LABEL = { "5min": "M5", "15min": "M15", "1h": "H1", "4h": "H4" };
+const TF_ORDER = ["5min", "15min", "1h", "4h"];
+const TF_SECONDS = { "5min": 300, "15min": 900, "1h": 3600, "4h": 14400 };
+const FALLBACK_POLL_MS = 60000;
+const RETRY_MS = 15000;
+const HIGH_IMPACT_PREF = "goldDeskHighImpactOnly";
+const TIMEFRAME_PREF = "goldDeskSelectedTimeframe";
+
+const LEVEL_LABEL = {
+  ema_fast: "EMA fast", ema_medium: "EMA medium", ema_slow: "EMA slow", ema_trend: "EMA trend",
+  bb_upper: "Bollinger upper", bb_mid: "Bollinger mid", bb_lower: "Bollinger lower",
+  kc_upper: "Keltner upper", kc_mid: "Keltner mid", kc_lower: "Keltner lower",
+  swing_high_1: "Swing high 1", swing_high_2: "Swing high 2", swing_high_3: "Swing high 3",
+  swing_low_1: "Swing low 1", swing_low_2: "Swing low 2", swing_low_3: "Swing low 3"
+};
+
+let snapshot = null, news = null;
+let clockOffset = 0;
+let lastPollOk = null;
+let lastSnapText = null, lastNewsText = null;
+let refetchTimer = null;
+let fallbackTimer = null;
+let pollInFlight = false;
+let pollQueued = false;
+let selectedTf = "5min";
+
+let detailTf = null;
+let ndState = null;
+const aiMemo = new Map();
+const aiErr = new Map();
+const aiPending = new Set();
+
+let tfModal = null;
+let newsModal = null;
+let tfModalOpen = false;
+let newsModalOpen = false;
+
+const $ = id => document.getElementById(id);
+
+/* ── helpers ──────────────────────────────────────────────────── */
+const esc = s => String(s ?? "").replace(/[&<>"']/g,
+  c => ({"&": "&amp;", "<": "&lt;", ">": "&gt;", '"': "&quot;", "'": "&#39;"}[c]));
+
+const fmt$ = v => v == null ? "—" : Number(v).toLocaleString("en-US",
+  { minimumFractionDigits: 2, maximumFractionDigits: 2 });
+
+const sigClass = s => s?.includes("BUY") ? "buy" : s?.includes("SELL") ? "sell" : "neutral";
+const sigLabel = s => (s || "NEUTRAL").replace("_", " ");
+const nowServer = () => Date.now() / 1000 + clockOffset;
+
+function localGet(key) {
+  try { return window.localStorage.getItem(key); } catch { return null; }
+}
+
+function localSet(key, value) {
+  try { window.localStorage.setItem(key, value); } catch { /* storage is optional */ }
+}
+
+function announce(message) {
+  const live = $("dashboard-live");
+  if (live) live.textContent = message;
+}
+
+function fmtCountdown(secs) {
+  if (secs == null || !isFinite(secs)) return "--:--";
+  secs = Math.max(0, Math.floor(secs));
+  const h = Math.floor(secs / 3600), m = Math.floor((secs % 3600) / 60), s = secs % 60;
+  const mm = String(m).padStart(2, "0"), ss = String(s).padStart(2, "0");
+  return h > 0 ? `${h}:${mm}:${ss}` : `${mm}:${ss}`;
+}
+
+function fmtAge(iso) {
+  if (!iso) return "";
+  const secs = nowServer() - Date.parse(iso) / 1000;
+  if (secs < 90) return `${Math.max(0, Math.round(secs))}s ago`;
+  if (secs < 5400) return `${Math.round(secs / 60)}m ago`;
+  if (secs < 172800) return `${Math.round(secs / 3600)}h ago`;
+  return `${Math.round(secs / 86400)}d ago`;
+}
+
+const signed = (v, dp = 2) => v == null ? "—" : (v >= 0 ? "+" : "") + Number(v).toFixed(dp);
+const badgeCls = s => s === "buy" ? "bg-success-subtle text-success-emphasis border-success-subtle"
+                 : s === "sell" ? "bg-danger-subtle text-danger-emphasis border-danger-subtle"
+                 : "bg-secondary-subtle text-secondary-emphasis border-secondary-subtle";
+
+/* ── theme ────────────────────────────────────────────────────── */
+// Dark theme is forced site-wide (owner decision 2026-08-16): dark
+// background, pure-white text, red/green data accents.
+document.documentElement.setAttribute("data-bs-theme", "dark");
+
+/* ── hero ─────────────────────────────────────────────────────── */
+function renderHero() {
+  if (!snapshot) return;
+  let price = null;
+  for (const tf of TF_ORDER) {
+    const p = snapshot.timeframes[tf]?.report?.current_price;
+    if (p) { price = p; break; }
+  }
+  $("hero-price").textContent = price ? fmt$(price) : "—";
+
+  $("hero-rail").innerHTML = TF_ORDER.map(tf => {
+    const r = snapshot.timeframes[tf]?.report;
+    const sig = r?.signal || null;
+    const cls = sigClass(sig);
+    const arrow = sig ? (ARROW[sig] || "→") : "·";
+    const score = r?.score != null ? r.score.toFixed(0) : "–";
+    const textColor = cls === "buy" ? "text-success" : cls === "sell" ? "text-danger" : "text-secondary";
+    return `<button type="button" class="rail-chip" data-select-tf="${tf}"
+      aria-current="${tf === selectedTf ? "true" : "false"}"
+      title="${esc(sigLabel(sig))} (score ${esc(score)}) — select ${TF_LABEL[tf]} workspace">
+      <span class="tf-name">${TF_LABEL[tf]}</span>
+      <span class="${textColor} fw-bold">${arrow}</span>
+      <span>${esc(score)}</span>
+    </button>`;
+  }).join("");
+
+  const closed = snapshot.market_closed;
+  $("hero-status").innerHTML =
+    `<span class="live-dot${closed ? " closed" : ""}"></span>${closed ? "CLOSED" : "OPEN"}`;
+  $("banner-closed").classList.toggle("d-none", !closed);
+}
+
+/* ── market context strip ─────────────────────────────────────── */
+function renderContext() {
+  if (!news) return;
+  $("context").classList.remove("d-none");
+  $("ctx-blurb").innerHTML = news.llm_summary
+    ? `<strong>Market context:</strong> ${esc(news.llm_summary)}`
+    : `<strong>Market context:</strong> <span class="text-secondary">no summary available</span>`;
+
+  let chips = "";
+  for (const cur of ["XAU", "USD"]) {
+    const v = news.sentiment?.[cur];
+    if (v == null) continue;
+    const cls = v > 0.05 ? "text-bg-success" : v < -0.05 ? "text-bg-danger" : "text-bg-secondary";
+    chips += `<span class="badge ${cls}" title="Mean headline sentiment for ${cur} (−1…+1)">
+      ${cur} news <span class="num">${v >= 0 ? "+" : ""}${v.toFixed(2)}</span></span>`;
+  }
+  let risk = "clear", riskDesc = "No significant events";
+  if (snapshot) {
+    for (const tf of TF_ORDER) {
+      const r = snapshot.timeframes[tf]?.report;
+      if (r?.event_risk && r.event_risk !== "clear") { risk = r.event_risk; riskDesc = r.event_description; break; }
+      if (r?.event_description) riskDesc = r.event_description;
+    }
+  }
+  if (news.high_impact_warning) { risk = "danger"; riskDesc = news.warning_message || riskDesc; }
+  const riskCls = risk === "clear" ? "text-bg-success"
+                : risk === "danger" ? "text-bg-danger"
+                : "text-bg-warning";
+  chips += `<span class="badge ${riskCls}" title="${esc(riskDesc)}">event risk: ${risk.toUpperCase()}</span>`;
+  $("ctx-chips").innerHTML = chips;
+}
+
+/* ── timeframe cards ──────────────────────────────────────────── */
+function scoreBar(label, score) {
+  const v = Math.max(-100, Math.min(100, score ?? 0));
+  const width = (Math.abs(v) / 2).toFixed(1);
+  return `<div class="scorebar">
+    <span class="lbl">${label}</span>
+    <div class="gauge-container" aria-hidden="true">
+      <i class="fill ${v >= 0 ? "pos" : "neg"}" style="--score-width:${width}%"></i>
+    </div>
+    <span class="val num ${v > 0 ? "text-success" : v < 0 ? "text-danger" : ""}">${v >= 0 ? "+" : ""}${v.toFixed(1)}</span>
+  </div>`;
+}
+
+function setupCard(ts) {
+  if (!ts) return "";
+  const dir = ts.direction === "BUY" ? "buy" : "sell";
+  const dirIcon = ts.direction === "BUY" ? "bi-arrow-up-circle-fill" : "bi-arrow-down-circle-fill";
+  const trail = (ts.trail_arm_r != null && ts.trail_dist_r != null)
+    ? `arm ${ts.trail_arm_r}R · dist ${ts.trail_dist_r}R` : "platform default";
+  return `<div class="card border-signal sig-${dir} bg-body-tertiary">
+    <div class="card-body py-2 px-3">
+      <div class="d-flex align-items-baseline gap-2 mb-2">
+        <span class="fw-bold text-signal sig-${dir}"><i class="bi ${dirIcon} me-1"></i>${ts.direction}</span>
+        <span class="small text-secondary font-monospace">${esc(ts.strategy || "")}</span>
+        <span class="small text-secondary ms-auto font-monospace">R:R ${(+ts.risk_reward).toFixed(1)}</span>
+      </div>
+      <div class="table-responsive"><table class="table table-sm table-borderless mb-0">
+        <tbody><tr><td class="text-secondary"><i class="bi bi-box-arrow-in-right me-1"></i>Entry</td><td class="num">${fmt$(ts.entry)}</td>
+            <td class="text-secondary ps-4"><i class="bi bi-flag-fill me-1 text-success"></i>TP1</td><td class="num text-success">${fmt$(ts.take_profit_1)}</td></tr>
+        <tr><td class="text-secondary"><i class="bi bi-shield-x me-1 text-danger"></i>Stop</td><td class="num text-danger">${fmt$(ts.stop_loss)}</td>
+            <td class="text-secondary ps-4"><i class="bi bi-flag-fill me-1 text-success"></i>TP2</td><td class="num text-success">${fmt$(ts.take_profit_2)}</td></tr>
+        <tr><td class="text-secondary"><i class="bi bi-signpost-split me-1"></i>Trail</td><td class="num" colspan="3">${esc(trail)}</td></tr></tbody>
+      </table></div>
+    </div>
+  </div>`;
+}
+
+function workspacePanel(tf) {
+  const entry = snapshot?.timeframes?.[tf] || {};
+  const r = entry.report;
+  if (!r) {
+    return `<article class="card workspace-card border-signal sig-neutral" id="workspace-panel" aria-labelledby="workspace-panel-heading">
+      <div class="card-body py-4 text-center text-secondary">
+        <h3 class="tf-name mb-2" id="workspace-panel-heading">${TF_LABEL[tf]}</h3>
+        ${entry.status === "error"
+          ? `<span class="badge text-bg-danger">SCAN UNAVAILABLE</span><div class="small mt-2">Showing again when the next scan succeeds.</div>`
+          : "Warming up — no completed scan for this timeframe yet."}
+      </div>
+    </article>`;
+  }
+
+  const cls = sigClass(r.signal);
+  const textCls = cls === "buy" ? "text-success" : cls === "sell" ? "text-danger" : "text-secondary";
+  const qHi = r.quality === "A+" || r.quality === "A";
+  const pa = r.primary_analysis;
+  const meta = [r.market_regime, r.volatility_regime && r.volatility_regime + " volatility",
+    r.mtf_agreement ? "MTF aligned" : "MTF mixed"].filter(Boolean).join(" · ");
+  const facts = `<div class="row row-cols-2 row-cols-md-4 g-2 mt-1">
+    ${tile("Confidence", r.confidence != null ? Math.round(r.confidence) + "%" : "—")}
+    ${tile("Quality", esc(r.quality || "—"), qHi ? "text-primary" : "")}
+    ${tile("Event risk", esc((r.event_risk || "—").toUpperCase()), r.event_risk === "clear" ? "text-success" : r.event_risk ? "text-warning" : "")}
+    ${tile("MTF", r.mtf_agreement ? "aligned" : "mixed", r.mtf_agreement ? "text-success" : "")}
+  </div>`;
+  const setup = r.trade_setup
+    ? setupCard(r.trade_setup)
+    : `<div class="text-secondary small">No setup published. The signal has not cleared its setup gate.</div>`;
+  return `<article class="card workspace-card border-signal sig-${cls}" id="workspace-panel" aria-labelledby="workspace-panel-heading">
+    <div class="card-body p-3 p-lg-4">
+      <div class="row g-4">
+        <div class="col-lg-7 workspace-meta">
+          <div class="d-flex justify-content-between align-items-start gap-3 flex-wrap mb-3">
+            <div>
+              <div class="d-flex align-items-center gap-2 flex-wrap mb-2">
+                <h3 class="tf-name mb-0" id="workspace-panel-heading">${TF_LABEL[tf]} signal</h3>
+                <span class="badge ${badgeCls(cls)}">${esc(sigLabel(r.signal))}</span>
+                ${r.quality ? `<span class="badge ${qHi ? "text-bg-primary" : "text-bg-secondary"}">quality ${esc(r.quality)}</span>` : ""}
+                ${entry.status === "error" ? `<span class="badge text-bg-danger">STALE</span>` : ""}
+              </div>
+              <div class="workspace-signal ${textCls}">${esc(r.action || sigLabel(r.signal))}</div>
+            </div>
+            <button type="button" class="btn btn-sm btn-outline-primary" data-open-detail="${tf}">
+              Full analysis <i class="bi bi-arrow-up-right ms-1"></i>
+            </button>
+          </div>
+          ${scoreBar("score", pa ? pa.composite_score : r.score)}
+          <div class="small text-secondary mt-2">${esc(meta)}</div>
+          ${facts}
+          <div class="workspace-reasons mt-3 pt-3 border-top">${listBlock(r) || "<div class=\"small text-secondary\">No supporting reasons recorded on this scan.</div>"}</div>
+          <div class="small text-secondary mt-3">${entry.scanned_at ? "scanned " + fmtAge(entry.scanned_at) : ""} · closes in <span class="num" data-close="${entry.next_close_epoch || ""}" data-interval="${TF_SECONDS[tf]}">--:--</span></div>
+        </div>
+        <div class="col-lg-5">
+          <div class="small text-uppercase text-secondary fw-bold mb-2">Trade plan</div>
+          ${setup}
+        </div>
+      </div>
+    </div>
+  </article>`;
+}
+
+function renderCards() {
+  if (!snapshot) return;
+  if (!TF_LABEL[selectedTf]) selectedTf = "5min";
+  const selector = TF_ORDER.map(tf => {
+    const entry = snapshot.timeframes[tf] || {};
+    const r = entry.report;
+    const cls = r ? sigClass(r.signal) : "neutral";
+    const arrow = r ? (ARROW[r.signal] || "→") : "·";
+    const score = r?.score != null ? signed(r.score, 0) : "—";
+    const signal = r ? sigLabel(r.signal) : entry.status === "error" ? "UNAVAILABLE" : "WARMING";
+    return `<button type="button" class="tf-select" data-select-tf="${tf}" aria-pressed="${tf === selectedTf}">
+      <span class="d-flex align-items-center justify-content-between gap-2">
+        <span class="tf-name">${TF_LABEL[tf]}</span>
+        <span class="${cls === "buy" ? "text-success" : cls === "sell" ? "text-danger" : "text-secondary"} fw-bold">${arrow}</span>
+      </span>
+      <span class="small ${cls === "buy" ? "text-success" : cls === "sell" ? "text-danger" : "text-secondary"}">${esc(signal)}</span>
+      <span class="small num float-end">${esc(score)}</span>
+    </button>`;
+  }).join("");
+  $("tf-grid").innerHTML = `<div class="tf-selector">${selector}</div>${workspacePanel(selectedTf)}`;
+  tick();
+}
+
+/* ── timeframe detail dialog ──────────────────────────────────── */
+const tile = (k, v, cls = "") =>
+  `<div class="col"><div class="card d-stat bg-body-tertiary h-100"><div class="card-body p-2">
+    <span class="k">${esc(k)}</span>
+    <span class="v ${cls}">${v}</span>
+  </div></div></div>`;
+
+const SEC_ICON = {
+  "At a glance": "bi-speedometer2",
+  "Trade setup": "bi-bullseye",
+  "Why this signal": "bi-lightbulb",
+  "Multi-timeframe confirmation": "bi-layers",
+  "News context": "bi-newspaper",
+  "AI panel (advisory)": "bi-robot",
+  "Scan detail": "bi-clock-history",
+  "Score": "bi-graph-up",
+  "From the source": "bi-quote",
+  "AI gold-impact summary": "bi-stars",
+  "Timing": "bi-alarm",
+  "Numbers": "bi-calculator"
+};
+const sec = (title, html) => {
+  if (!html) return "";
+  const icon = SEC_ICON[title];
+  return `<section class="mb-4"><h6 class="text-uppercase text-secondary small fw-bold mb-2 d-flex align-items-center gap-2">${icon ? `<i class="bi ${icon} text-primary"></i>` : ""}<span>${esc(title)}</span><span class="flex-fill border-bottom"></span></h6>${html}</section>`;
+};
+
+function indTable(src) {
+  const rows = IND_ORDER.map(name => {
+    const i = src?.[name];
+    const k = `<td class="ind-k">${IND_SHORT[name]}</td>`;
+    if (!i) return `<tr>${k}<td class="text-secondary">— absent</td>
+      <td class="num text-end">—</td><td class="num text-end">—</td><td class="num text-end">—</td>
+      <td class="text-secondary small">no signal this scan</td></tr>`;
+    const w = i.weight == null ? null : +i.weight;
+    return `<tr>${k}
+      <td class="fw-bold ${sigClass(i.signal) === "buy" ? "text-success" : sigClass(i.signal) === "sell" ? "text-danger" : "text-secondary"}">${esc(sigLabel(i.signal))}</td>
+      <td class="num text-end">${i.value != null ? (+i.value).toFixed(2) : "—"}</td>
+      <td class="num text-end">${i.confidence != null ? Math.round(i.confidence) : "—"}</td>
+      <td class="num text-end">${w != null ? w.toFixed(1) : "—"}${w != null
+        ? `<i class="wbar"><i style="width:${Math.min(100, w / 2 * 100)}%"></i></i>` : ""}</td>
+      <td class="text-secondary small">${esc(i.description || "")}</td></tr>`;
+  }).join("");
+  return `<div class="detail-table"><table class="table table-sm d-table"><thead><tr>
+    <th>Indicator</th><th>Signal</th><th class="text-end">Value</th><th class="text-end">Conf</th>
+    <th class="text-end">Weight</th><th>Reading</th></tr></thead><tbody>${rows}</tbody></table></div>`;
+}
+
+function ladder(levels, price) {
+  if (!levels) return "";
+  const atr = levels.atr;
+  const rows = Object.keys(LEVEL_LABEL)
+    .filter(k => levels[k] != null)
+    .map(k => ({ label: LEVEL_LABEL[k], v: +levels[k] }));
+  if (!rows.length) return "";
+  if (price != null) rows.push({ here: true, v: +price });
+  rows.sort((a, b) => b.v - a.v);
+  const body = rows.map(row => {
+    if (row.here) return `<tr class="table-primary fw-bold"><td><i class="bi bi-geo-alt-fill me-1"></i>current price</td>
+      <td class="num text-end">${fmt$(row.v)}</td><td class="num text-end">—</td><td class="small text-secondary">spot</td></tr>`;
+    const d = price != null ? row.v - price : null;
+    const cls = d == null ? "" : d > 0 ? "text-danger" : "text-success";
+    const icon = d == null ? "" : d > 0 ? "bi-arrow-up-short" : "bi-arrow-down-short";
+    return `<tr>
+      <td class="text-secondary">${esc(row.label)}</td>
+      <td class="num text-end ${cls}">${fmt$(row.v)}</td>
+      <td class="num text-end ${cls}">${d == null ? "" : signed(d) + (atr ? ` · ${signed(d / atr, 2)} ATR` : "")}</td>
+      <td class="small ${cls}">${d == null ? "" : `<i class="bi ${icon}"></i>${d > 0 ? "resistance" : "support"}`}</td></tr>`;
+  }).join("");
+  return `<div class="detail-table"><table class="table table-sm ladder"><thead><tr><th>Level</th><th class="text-end">Price</th>
+    <th class="text-end">vs. spot</th><th>Role</th></tr></thead><tbody>${body}</tbody></table></div>`;
+}
+
+function analysisBlock(a, price) {
+  if (!a) return "";
+  const pats = a.pattern_signals?.length
+    ? `<div class="d-flex flex-wrap gap-1 mb-2">${a.pattern_signals.map(p =>
+        `<span class="badge text-bg-primary bg-opacity-10">${esc(p)}</span>`).join("")}</div>` : "";
+  const atr = a.atr_context
+    ? `<div class="small text-secondary mb-2">${esc(a.atr_context.description ||
+        "ATR " + (+a.atr_context.value).toFixed(2))}</div>` : "";
+  const lv = ladder(a.key_levels, price);
+  return indTable(a.indicators) + atr + pats +
+    (lv ? `<div class="small text-secondary fw-semibold mt-2 mb-1">Key levels</div>${lv}` : "");
+}
+
+function listBlock(r) {
+  const group = (title, arr, cls, icon) => arr?.length
+    ? `<div class="small text-secondary fw-semibold mt-2 mb-1">${title} (${arr.length})</div>
+       <div class="small">${arr.map(t => `<div class="${cls}"><i class="bi ${icon} me-1"></i>${esc(t)}</div>`).join("")}</div>` : "";
+  return group("Warnings", r.warnings, "text-warning", "bi-exclamation-triangle-fill")
+       + group("Bullish", r.bullish_reasons, "text-success", "bi-arrow-up-circle-fill")
+       + group("Bearish", r.bearish_reasons, "text-danger", "bi-arrow-down-circle-fill");
+}
+
+function detailHTML(tf) {
+  const entry = snapshot?.timeframes?.[tf] || {};
+  const r = entry.report;
+  if (!r) return `<div class="text-center text-secondary py-4">${entry.status === "error"
+    ? `Scan error — ${esc(entry.error || "unknown")}`
+    : "Warming up — no completed scan for this timeframe yet."}</div>`;
+
+  const pa = r.primary_analysis;
+  const price = r.current_price;
+  const cls = sigClass(r.signal);
+  const textCls = cls === "buy" ? "text-success" : cls === "sell" ? "text-danger" : "text-secondary";
+  const borderCls = cls === "buy" ? "border-start border-success border-4"
+                  : cls === "sell" ? "border-start border-danger border-4"
+                  : "border-start border-secondary border-4";
+
+  let out = `<div class="card ${borderCls} mb-3"><div class="card-body">
+    <div class="fw-bold fs-5 ${textCls}">${esc(r.action || sigLabel(r.signal))}</div>
+    ${r.news_description || r.event_description
+      ? `<div class="small text-secondary">${esc([r.event_description, r.news_description].filter(Boolean).join(" · "))}</div>`
+      : ""}
+  </div></div>`;
+
+  const ns = r.news_sentiment;
+  out += sec("At a glance", `<div class="row row-cols-2 row-cols-md-3 row-cols-lg-4 g-2">
+    ${tile("Signal", esc(sigLabel(r.signal)), textCls)}
+    ${tile("Score", signed(r.score, 1), textCls)}
+    ${tile("Confidence", r.confidence != null ? Math.round(r.confidence) + "%" : "—")}
+    ${tile("Quality", esc(r.quality || "—"))}
+    ${tile("Regime", esc(r.market_regime || "—"))}
+    ${tile("Volatility", esc(r.volatility_regime || "—"))}
+    ${tile("Event risk", esc((r.event_risk || "—").toUpperCase()),
+        r.event_risk === "clear" ? "text-success" : r.event_risk ? "text-warning" : "")}
+    ${tile("News sentiment", ns != null ? signed(ns, 2) : "—",
+        ns > 0.05 ? "text-success" : ns < -0.05 ? "text-danger" : "")}
+    ${tile("MTF", r.mtf_agreement ? "aligned" : "mixed", r.mtf_agreement ? "text-success" : "")}
+    ${tile("Primary score", signed(r.primary_score, 1), sigClass(r.primary_signal) === "buy" ? "text-success" : sigClass(r.primary_signal) === "sell" ? "text-danger" : "")}
+    ${tile("Exhaustion", esc(r.exhaustion_count ?? "—"), r.exhaustion_count ? "text-warning" : "")}
+  </div>`);
+
+  out += sec("Trade setup", r.trade_setup
+    ? setupCard(r.trade_setup)
+    : `<div class="text-secondary small">No setup published for this timeframe — the engine only emits entry/stop/target once a signal clears its quality gate.</div>`);
+
+  out += sec("Why this signal", listBlock(r) || `<div class="text-secondary small">No reasons recorded on this scan.</div>`);
+
+  out += sec(`${TF_LABEL[tf]} indicators (primary)`,
+    analysisBlock(pa, price) || indTable(r.indicators) +
+      (r.key_levels ? `<div class="small text-secondary fw-semibold mt-2 mb-1">Key levels</div>${ladder(r.key_levels, price)}` : ""));
+
+  const mtf = Object.entries(r.mtf_signals || {});
+  if (mtf.length) {
+    const chips = mtf.map(([k, sig]) => {
+      const s = r.mtf_scores?.[k];
+      const sc = sigClass(sig);
+      return `<span class="badge ${badgeCls(sc)}">${esc(TF_LABEL[k] || k)}
+        <span class="${sc === "buy" ? "text-success" : sc === "sell" ? "text-danger" : "text-secondary"}">${esc(sigLabel(sig))}</span>
+        <span class="num">${s != null ? signed(s, 1) : ""}</span></span>`;
+    }).join("");
+    const confs = (r.confirmation_analyses || []).filter(Boolean).map(a => {
+      const lbl = TF_LABEL[a.timeframe] || a.timeframe;
+      const sc = sigClass(a.composite_signal);
+      return `<details data-dkey="conf-${esc(a.timeframe)}">
+        <summary><strong>${esc(lbl)}</strong>
+          <span class="${sc === "buy" ? "text-success" : sc === "sell" ? "text-danger" : "text-secondary"} fw-bold">${esc(sigLabel(a.composite_signal))}</span>
+          <span class="text-secondary small">${signed(a.composite_score, 1)} · conf ${a.confidence != null ? Math.round(a.confidence) : "—"} · ${esc(a.market_regime || "")}</span></summary>
+        ${analysisBlock(a, price)}</details>`;
+    }).join("");
+    out += sec("Multi-timeframe confirmation",
+      `<div class="d-flex flex-wrap gap-2 align-items-center mb-2">${chips}
+        <span class="fw-bold ${r.mtf_agreement ? "text-success" : "text-secondary"}">${r.mtf_agreement ? "✓ aligned" : "✗ mixed"}</span></div>${confs}`);
+  }
+
+  if (r.news_summary || r.news_description) {
+    out += sec("News context", `<div class="text-secondary small">
+      ${r.news_description ? `<strong>${esc(r.news_description)}</strong>${ns != null ? ` (${signed(ns, 2)})` : ""}<br>` : ""}
+      ${esc(r.news_summary || "")}</div>`);
+  }
+
+  if (r.panel_verdict) {
+    const vcls = r.panel_verdict === "AGREE" ? "text-success"
+      : r.panel_verdict === "OPPOSE" ? "text-danger" : "text-secondary";
+    const takes = (r.panel_takes || []).map(t =>
+      `<tr><td class="ind-k">${esc(t.persona)}</td>
+        <td>${esc(t.stance)}</td>
+        <td class="num text-end">${t.confidence != null ? Math.round(t.confidence) : "—"}</td>
+        <td class="small text-secondary">${esc(t.note || "")}</td></tr>`).join("");
+    out += sec("AI panel (advisory)", `<div class="text-secondary small mb-2">
+      <strong class="${vcls}">${esc(r.panel_verdict)}</strong> ·
+      conf ${Math.round(r.panel_confidence || 0)}%<br>
+      ${esc(r.panel_rationale || "")}</div>
+      ${takes ? `<div class="detail-table"><table class="table table-sm d-table"><tbody>${takes}</tbody></table></div>` : ""}`);
+  }
+
+  out += sec("Scan detail", `<div class="detail-table"><table class="table table-sm d-table"><tbody>
+    <tr><td class="ind-k">Instrument</td><td colspan="5">${esc(r.display_name || r.symbol || "—")}</td></tr>
+    <tr><td class="ind-k">Candle</td><td colspan="5" class="num">${esc(r.timestamp || "—")}</td></tr>
+    <tr><td class="ind-k">Scanned</td><td colspan="5">${esc(entry.scanned_at || "—")}
+      (${fmtAge(entry.scanned_at)}${entry.scan_seconds ? `, took ${entry.scan_seconds}s` : ""})</td></tr>
+    <tr><td class="ind-k">Status</td><td colspan="5">${esc(entry.status || "—")}${entry.error
+      ? ` — <span class="text-danger">${esc(entry.error)}</span>
+          ${entry.error_at ? `(${fmtAge(entry.error_at)})` : ""}` : ""}</td></tr>
+   </tbody></table></div>`);
+  return out;
+}
+
+function renderDetail() {
+  if (!detailTf || !tfModalOpen) return;
+  const entry = snapshot?.timeframes?.[detailTf] || {};
+  const r = entry.report;
+  const openKeys = new Set([...document.querySelectorAll("#detail-body details[open][data-dkey]")]
+    .map(d => d.dataset.dkey));
+  const scroll = $("detail-body").scrollTop;
+
+  $("detail-title").textContent = TF_LABEL[detailTf] || detailTf;
+  $("detail-price").textContent = r?.current_price ? fmt$(r.current_price) : "—";
+  $("detail-sub").innerHTML = `${snapshot?.symbol ? esc(snapshot.symbol) + " · " : ""}closes in
+    <span class="num" data-close="${entry.next_close_epoch || ""}" data-interval="${TF_SECONDS[detailTf]}">--:--</span>`;
+  $("detail-body").innerHTML = detailHTML(detailTf);
+
+  document.querySelectorAll("#detail-body details[data-dkey]").forEach(d => {
+    if (openKeys.has(d.dataset.dkey)) d.open = true;
+  });
+  $("detail-body").scrollTop = scroll;
+  tick();
+}
+
+function openDetail(tf) {
+  if (!TF_LABEL[tf]) return;
+  detailTf = tf;
+  tfModal.show();
+  renderDetail();
+  $("detail-body").scrollTop = 0;
+  $("detail-close").focus();
+}
+
+$("detail-close")?.addEventListener("click", () => tfModal.hide());
+
+/* ── calendar ─────────────────────────────────────────────────── */
+function renderCalendar() {
+  const body = $("cal-body");
+  if (!news) return;
+  const zone = Intl.DateTimeFormat().resolvedOptions().timeZone || "local time";
+  $("calendar-note").textContent = `times local (${zone}) · USD events${news.generated_at ? " · updated " + fmtAge(news.generated_at) : ""}`;
+  const hiOnly = $("hi-only").checked;
+  let events = (news.events || []);
+  if (hiOnly) events = events.filter(e => e.impact === "high");
+  if (!events.length) {
+    body.innerHTML = `<tr><td colspan="7" class="text-secondary text-center py-3">${
+      news.calendar_available ? "No matching events this week." : "Calendar unavailable — source could not be reached."}</td></tr>`;
+    return;
+  }
+  const now = nowServer();
+  let nextUpMarked = false, lastDay = "", rows = "";
+  for (const e of events) {
+    const t = Date.parse(e.time) / 1000;
+    const d = new Date(t * 1000);
+    const day = d.toLocaleDateString(undefined, { weekday: "short", month: "short", day: "numeric" });
+    if (day !== lastDay) {
+      rows += `<tr class="table-group-divider"><td colspan="7" class="text-primary small fw-bold text-uppercase">${esc(day)}</td></tr>`;
+      lastDay = day;
+    }
+    const past = t < now;
+    const nextUp = !past && !nextUpMarked && (nextUpMarked = true);
+    const time = d.toLocaleTimeString(undefined, { hour: "2-digit", minute: "2-digit" });
+    const surp = e.surprise
+      ? `<span class="${e.surprise === "positive" ? "text-success" : e.surprise === "negative" ? "text-danger" : "text-secondary"}" title="actual ${e.surprise === "positive" ? "above" : e.surprise === "negative" ? "below" : "matches"} forecast (numeric only — not gold direction)">${e.surprise === "positive" ? "▲" : e.surprise === "negative" ? "▼" : "="}</span>` : "";
+    rows += `<tr class="${past ? "text-secondary" : ""}${nextUp ? " cal-next-up" : ""}">
+      <td class="num text-nowrap">${esc(time)}</td>
+      <td class="text-secondary">${esc(e.currency)}</td>
+      <td><span class="badge ${e.impact === "high" ? "text-bg-danger" : e.impact === "medium" ? "text-bg-warning" : "text-bg-secondary"}">${e.impact === "medium" ? "Med" : e.impact.charAt(0).toUpperCase() + e.impact.slice(1)}</span></td>
+      <td><button type="button" class="cal-event" data-ev-key="${esc(evKey(e))}" aria-haspopup="dialog">${esc(e.event)}</button></td>
+      <td class="num text-end">${esc(e.forecast ?? "—")}</td>
+      <td class="num text-end">${esc(e.previous ?? "—")}</td>
+      <td class="num text-end">${esc(e.actual ?? "—")} ${surp}</td></tr>`;
+  }
+  body.innerHTML = rows;
+}
+
+/* ── newswire ─────────────────────────────────────────────────── */
+function renderWireTotal() {
+  const box = $("wire-total");
+  const scored = (news?.headlines || []).filter(h => typeof h.score === "number");
+  if (!scored.length) { box.classList.add("d-none"); return; }
+  box.classList.remove("d-none");
+
+  const sum = scored.reduce((a, h) => a + h.score, 0);
+  const mean = sum / scored.length;
+  const up = scored.filter(h => h.score > 0.05).length;
+  const dn = scored.filter(h => h.score < -0.05).length;
+  const cls = mean > 0.05 ? "text-success" : mean < -0.05 ? "text-danger" : "text-secondary";
+  const dir = mean > 0.05 ? "BULLISH" : mean < -0.05 ? "BEARISH" : "NEUTRAL";
+  const wrapCls = mean > 0.05 ? "bg-success-subtle border-success-subtle"
+                : mean < -0.05 ? "bg-danger-subtle border-danger-subtle"
+                : "bg-body-tertiary";
+
+  box.className = `card mb-3 ${wrapCls}`;
+  box.innerHTML = `<div class="card-body p-2">
+    <div class="d-flex align-items-baseline gap-2 flex-wrap">
+      <span class="small text-secondary text-uppercase fw-bold">Total news score</span>
+      <span class="display-6 fw-bold ${cls}">${signed(mean, 2)}</span>
+      <span class="fw-bold ${cls}">${dir}</span>
+      <span class="ms-auto text-end small text-secondary">${scored.length} headlines · sum ${signed(sum, 1)}<br>
+        ${up} bullish · ${dn} bearish</span>
+    </div>
+  </div>`;
+  box.title = `Mean sentiment of all ${scored.length} scored headlines (−1…+1). Sum ${signed(sum, 1)}. Only headlines from the last 4 hours that lean one way are published (neutral ones are removed before scoring).`;
+}
+
+function renderWire() {
+  if (!news) return;
+  const items = (news.headlines || []).slice(0, 40);
+  $("wire-note").textContent = news.generated_at ? `updated ${fmtAge(news.generated_at)}` : "";
+  renderWireTotal();
+  if (!items.length) {
+    $("wire").innerHTML = `<div class="text-secondary small py-2">No gold or dollar headlines in the last 4 hours.</div>`;
+    return;
+  }
+  const item = h => {
+    const s = h.score;
+    const cls = s == null || Math.abs(s) < 0.05 ? "zero" : s > 0 ? "pos" : "neg";
+    const scoreCls = cls === "pos" ? "bg-success-subtle text-success"
+                   : cls === "neg" ? "bg-danger-subtle text-danger"
+                   : "bg-body-tertiary text-secondary";
+    const scoreTxt = s == null ? "·" : (s > 0 ? "+" : "") + s.toFixed(1);
+    let host = "";
+    try { host = new URL(h.source).hostname.replace(/^www\./, ""); } catch {}
+    const content = `<span class="wire-score ${scoreCls}" title="Headline sentiment ${esc(String(s ?? "n/a"))} (−1…+1)${h.currencies?.length ? " · " + esc(h.currencies.join(", ")) : ""}">${scoreTxt}</span>
+       <span class="small text-secondary flex-fill">${esc(h.headline)}</span>
+       <span class="small text-secondary text-nowrap">${esc(h.publisher || host)}${h.timestamp ? " · " + fmtAge(h.timestamp) : ""}</span>`;
+    if (!h.id) return `<div class="wire-item gap-2 align-items-baseline py-1 px-1">${content}</div>`;
+    return `<button type="button" class="wire-item gap-2 align-items-baseline py-1 px-1" data-news-id="${h.id}" aria-haspopup="dialog">
+      ${content}
+    </button>`;
+  };
+  const head = items.slice(0, 12).map(item).join("");
+  const rest = items.slice(12);
+  const moreOpen = $("wire")?.querySelector("details.more")?.open ?? false;
+  $("wire").innerHTML = head + (rest.length
+    ? `<details class="more mt-1"${moreOpen ? " open" : ""}><summary class="small text-secondary">${rest.length} more headlines…</summary>
+       <div class="mt-1">${rest.map(item).join("")}</div></details>` : "");
+}
+
+/* ── news / calendar detail dialog ────────────────────────────── */
+const evKey = e => `${e.time}|${e.currency}|${e.event}`;
+const findHeadline = id => (news?.headlines || []).find(h => h.id === id);
+const findEvent = key => (news?.events || []).find(e => evKey(e) === key);
+
+const AI_ERR_MSG = {
+  unknown_id: "This headline has left the live feed — no summary available.",
+  cap_exceeded: "Daily AI budget reached — try again tomorrow.",
+  no_key: "AI summaries are not configured on this server."
+};
+
+async function fetchAiSummary(id) {
+  aiPending.add(id);
+  try {
+    const res = await fetch("news_summary.php?id=" + encodeURIComponent(id));
+    const data = await res.json().catch(() => null);
+    if (res.ok && data?.summary) aiMemo.set(id, data);
+    else aiErr.set(id, data?.error || `http_${res.status}`);
+  } catch {
+    aiErr.set(id, "network");
+  } finally {
+    aiPending.delete(id);
+    if (ndState?.kind === "wire" && ndState.id === id) renderNewsDlg();
+  }
+}
+
+function scoreTiles(h) {
+  const cls = v => v > 0.05 ? "text-success" : v < -0.05 ? "text-danger" : "text-secondary";
+  let tiles = tile("Overall sentiment", signed(h.score, 2), cls(h.score));
+  const xau = h.score_by_currency?.XAU;
+  if (xau != null) tiles += tile("Gold (XAU) effect", signed(xau, 2), cls(xau));
+  for (const [cur, v] of Object.entries(h.score_by_currency || {})) {
+    if (cur !== "XAU") tiles += tile(cur, signed(v, 2), cls(v));
+  }
+  let out = `<div class="row row-cols-2 row-cols-md-3 g-2">${tiles}</div>`;
+  if (h.keywords?.length) {
+    out += `<div class="small text-secondary mt-2">Matched keywords: ${esc(h.keywords.join(", "))}</div>`;
+  }
+  out += `<div class="small text-secondary">Scores range −1 (bearish) to +1 (bullish); the gold score flips monetary-policy keywords (hawkish = dollar-bullish, gold-bearish).</div>`;
+  return out;
+}
+
+function aiSection(id) {
+  if (aiMemo.has(id)) {
+    const m = aiMemo.get(id);
+    return `<div class="text-secondary small">${esc(m.summary)}</div>` + (m.generated_at
+      ? `<div class="small text-secondary">generated ${fmtAge(m.generated_at)}</div>` : "");
+  }
+  if (aiPending.has(id)) return `<div class="text-secondary small">Generating gold-impact summary…</div>`;
+  if (aiErr.has(id)) {
+    return `<div class="text-secondary small">${esc(AI_ERR_MSG[aiErr.get(id)] || "AI summary unavailable right now.")}</div>`;
+  }
+  return `<div class="text-secondary small">Generating gold-impact summary…</div>`;
+}
+
+function wireDlgHTML() {
+  const fresh = findHeadline(ndState.id);
+  const h = fresh || ndState.snap;
+  if (!h) return "";
+  let host = "";
+  try { host = new URL(h.source).hostname.replace(/^www\./, ""); } catch {}
+  const when = h.timestamp
+    ? `${new Date(h.timestamp).toLocaleString(undefined, { dateStyle: "medium", timeStyle: "short" })} · ${fmtAge(h.timestamp)}` : "";
+  let out = `<div class="fs-5 fw-semibold mb-1">${esc(h.headline)}</div>
+    <div class="small text-secondary mb-3">${esc(h.publisher || host)}${when ? " · " + when : ""}</div>`;
+  if (!fresh) out += `<div class="small text-secondary mb-2">No longer in the live feed — showing saved copy.</div>`;
+  out += sec("Score", scoreTiles(h));
+  if (h.description) out += sec("From the source", `<div class="text-secondary small">${esc(h.description)}</div>`);
+  out += sec("AI gold-impact summary", aiSection(ndState.id));
+  if (h.link && /^https?:\/\//.test(h.link)) {
+    out += `<a class="btn btn-outline-primary" href="${esc(h.link)}" target="_blank" rel="noopener noreferrer">Read full article ↗</a>`;
+  }
+  return out;
+}
+
+function calDlgHTML() {
+  const fresh = findEvent(ndState.key);
+  const e = fresh || ndState.snap;
+  if (!e) return "";
+  const t = Date.parse(e.time) / 1000;
+  const when = new Date(e.time).toLocaleString(undefined, { dateStyle: "full", timeStyle: "short" });
+  const impact = `<span class="badge ${e.impact === "high" ? "text-bg-danger" : e.impact === "medium" ? "text-bg-warning" : "text-bg-secondary"}">${e.impact === "medium" ? "Med" : e.impact.charAt(0).toUpperCase() + e.impact.slice(1)}</span>`;
+  let out = `<div class="fs-5 fw-semibold mb-1">${esc(e.event)}</div>
+    <div class="small text-secondary mb-3">${esc(e.currency)} · ${when}</div>`;
+  const timing = t > nowServer()
+    ? `releases in <span class="num" data-close="${t}">--:--</span>`
+    : `released ${fmtAge(e.time)}`;
+  out += sec("Timing", `<div class="text-secondary small">${impact} impact · ${timing}</div>`);
+  const surpTxt = { positive: "actual above forecast", negative: "actual below forecast", inline: "actual matches forecast" }[e.surprise];
+  out += sec("Numbers", `<div class="row row-cols-3 g-2">${
+      tile("Forecast", esc(e.forecast ?? "—")) +
+      tile("Previous", esc(e.previous ?? "—")) +
+      tile("Actual", esc(e.actual ?? "—"), e.surprise === "positive" ? "text-success" : e.surprise === "negative" ? "text-danger" : "")
+    }</div>` + (surpTxt
+      ? `<div class="small text-secondary mt-2">${surpTxt} (numeric only — not gold direction)</div>` : ""));
+  return out;
+}
+
+function renderNewsDlg() {
+  if (!ndState || !newsModalOpen) return;
+  const scroll = $("nd-body").scrollTop;
+  $("nd-title").textContent = ndState.kind === "wire" ? "Newswire" : "Economic calendar";
+  $("nd-body").innerHTML = ndState.kind === "wire" ? wireDlgHTML() : calDlgHTML();
+  $("nd-body").scrollTop = scroll;
+  tick();
+}
+
+function openNewsDetail(id) {
+  const h = findHeadline(id);
+  if (!h) return;
+  ndState = { kind: "wire", id, snap: h };
+  aiErr.delete(id);
+  newsModal.show();
+  if (!aiMemo.has(id) && !aiPending.has(id)) fetchAiSummary(id);
+  renderNewsDlg();
+  $("nd-body").scrollTop = 0;
+  $("nd-close").focus();
+}
+
+function openCalDetail(key) {
+  const e = findEvent(key);
+  if (!e) return;
+  ndState = { kind: "cal", key, snap: e };
+  newsModal.show();
+  renderNewsDlg();
+  $("nd-body").scrollTop = 0;
+  $("nd-close").focus();
+}
+
+$("nd-close")?.addEventListener("click", () => newsModal.hide());
+
+/* ── polling ──────────────────────────────────────────────────── */
+async function fetchJSON(name) {
+  const res = await fetch(name, { cache: "no-cache" });
+  if (!res.ok) throw new Error(`${name}: HTTP ${res.status}`);
+  const dateHdr = res.headers.get("Date");
+  const text = await res.text();
+  return { text, serverNow: dateHdr ? Date.parse(dateHdr) / 1000 : null };
+}
+
+function tfFingerprint(entry) {
+  if (!entry) return "";
+  return JSON.stringify({
+    status: entry.status,
+    scanned_at: entry.scanned_at,
+    error: entry.error,
+    error_at: entry.error_at,
+    report: entry.report
+  });
+}
+
+function scheduleRefresh(delay) {
+  clearTimeout(refetchTimer);
+  refetchTimer = setTimeout(() => {
+    refetchTimer = null;
+    poll();
+  }, Math.max(0, delay));
+}
+
+function scheduleFallbackRefresh() {
+  clearTimeout(fallbackTimer);
+  fallbackTimer = setTimeout(() => {
+    fallbackTimer = null;
+    poll();
+    scheduleFallbackRefresh();
+  }, FALLBACK_POLL_MS);
+}
+
+function scheduleBoundaryRefresh() {
+  if (!snapshot) return;
+  const nextScan = Math.min(...TF_ORDER.map(tf =>
+    snapshot.timeframes[tf]?.next_scan_epoch || Infinity));
+  if (!isFinite(nextScan)) return;
+  const delay = (nextScan - nowServer() + 2) * 1000;
+  scheduleRefresh(delay > 0 ? delay : RETRY_MS);
+}
+
+async function poll() {
+  if (pollInFlight) {
+    pollQueued = true;
+    return;
+  }
+  pollInFlight = true;
+  try {
+    const [snap, nw] = await Promise.all([
+      fetchJSON("snapshot.json"),
+      fetchJSON("news.json").catch(() => null)
+    ]);
+    const priorSnapshot = snapshot;
+    let snapChanged = false;
+    let selectedChanged = false;
+    let cardsChanged = false;
+    let detailChanged = false;
+    if (snap.text !== lastSnapText) {
+      const incoming = JSON.parse(snap.text);
+      if (!priorSnapshot || incoming.server_time_epoch >= priorSnapshot.server_time_epoch) {
+        snapChanged = true;
+        selectedChanged = tfFingerprint(priorSnapshot?.timeframes?.[selectedTf]) !==
+          tfFingerprint(incoming.timeframes?.[selectedTf]);
+        // renderCards() redraws the selector for EVERY timeframe, so it
+        // must react to any of them changing — not just the selected one.
+        // Gating it on selectedChanged left the other buttons showing a
+        // stale arrow/score (up to 4 h on H4) while renderHero's rail,
+        // which redraws unconditionally, already showed the new value.
+        cardsChanged = TF_ORDER.some(tf =>
+          tfFingerprint(priorSnapshot?.timeframes?.[tf]) !==
+          tfFingerprint(incoming.timeframes?.[tf]));
+        detailChanged = detailTf != null && tfFingerprint(priorSnapshot?.timeframes?.[detailTf]) !==
+          tfFingerprint(incoming.timeframes?.[detailTf]);
+        snapshot = incoming;
+        lastSnapText = snap.text;
+      }
+    }
+    const newsChanged = nw != null && nw.text !== lastNewsText;
+    if (newsChanged) { news = JSON.parse(nw.text); lastNewsText = nw.text; }
+    clockOffset = (snap.serverNow ?? snapshot.server_time_epoch) - Date.now() / 1000;
+    lastPollOk = new Date();
+    $("banner-fail").classList.add("d-none");
+
+    if (snapChanged) {
+      renderHero();
+      if (cardsChanged) renderCards();
+      if (detailChanged) renderDetail();
+      if (selectedChanged) {
+        const signal = snapshot.timeframes[selectedTf]?.report?.signal;
+        if (signal) announce(`${TF_LABEL[selectedTf]} signal updated: ${sigLabel(signal)}.`);
+      }
+    }
+    if (newsChanged) {
+      renderCalendar();
+      renderWire();
+      renderNewsDlg();
+    }
+    if (snapChanged || newsChanged) renderContext();
+    tick();
+    scheduleBoundaryRefresh();
+  } catch (err) {
+    const last = lastPollOk ? ` Last update ${lastPollOk.toLocaleTimeString()}.` : "";
+    const b = $("banner-fail");
+    b.textContent = `Dashboard data is temporarily unavailable.${last} Retrying…`;
+    b.classList.remove("d-none");
+    scheduleRefresh(RETRY_MS);
+  } finally {
+    pollInFlight = false;
+    if (pollQueued) {
+      pollQueued = false;
+      poll();
+    }
+  }
+}
+
+function tick() {
+  if (!snapshot) return;
+  const now = nowServer();
+  document.querySelectorAll("[data-close]").forEach(el => {
+    const t = parseFloat(el.dataset.close);
+    if (!t) return;
+    let secs = t - now;
+    if (secs < 0) {
+      const interval = parseFloat(el.dataset.interval);
+      secs = interval > 0 ? ((secs % interval) + interval) % interval : 0;
+    }
+    el.textContent = fmtCountdown(secs);
+  });
+  const m5 = snapshot.timeframes["5min"]?.next_close_epoch;
+  if (m5) {
+    let secs = m5 - now;
+    if (secs < 0) secs = ((secs % TF_SECONDS["5min"]) + TF_SECONDS["5min"]) % TF_SECONDS["5min"];
+    $("hero-cd").textContent = fmtCountdown(secs);
+  }
+  const pubEpoch = snapshot.server_time_epoch;
+  if (pubEpoch) {
+    const age = now - pubEpoch;
+    const el = $("hero-updated");
+    const stale = !snapshot.market_closed && age > 900;
+    el.textContent = (stale ? "STALE — published " : "published ") +
+      fmtAge(new Date(pubEpoch * 1000).toISOString());
+    el.classList.toggle("text-warning", stale);
+    el.classList.toggle("fw-semibold", stale);
+  }
+}
+
+/* ── boot ─────────────────────────────────────────────────────── */
+document.addEventListener("DOMContentLoaded", () => {
+  tfModal = new bootstrap.Modal($("tf-detail"));
+  newsModal = new bootstrap.Modal($("news-detail"));
+  const storedTf = localGet(TIMEFRAME_PREF);
+  if (TF_LABEL[storedTf]) selectedTf = storedTf;
+  $("hi-only").checked = localGet(HIGH_IMPACT_PREF) === "true";
+
+  $("tf-detail")?.addEventListener("shown.bs.modal", () => { tfModalOpen = true; renderDetail(); });
+  $("tf-detail")?.addEventListener("hidden.bs.modal", () => { tfModalOpen = false; detailTf = null; });
+  $("news-detail")?.addEventListener("shown.bs.modal", () => { newsModalOpen = true; renderNewsDlg(); });
+  $("news-detail")?.addEventListener("hidden.bs.modal", () => { newsModalOpen = false; ndState = null; });
+
+  const selectTf = tf => {
+    if (!TF_LABEL[tf]) return;
+    const changed = selectedTf !== tf;
+    selectedTf = tf;
+    localSet(TIMEFRAME_PREF, tf);
+    renderHero();
+    renderCards();
+    if (changed) announce(`${TF_LABEL[tf]} workspace selected.`);
+  };
+  $("hero-rail")?.addEventListener("click", e => {
+    const button = e.target.closest("[data-select-tf]");
+    if (button) selectTf(button.dataset.selectTf);
+  });
+  $("tf-grid")?.addEventListener("click", e => {
+    const detail = e.target.closest("[data-open-detail]");
+    if (detail) return openDetail(detail.dataset.openDetail);
+    const button = e.target.closest("[data-select-tf]");
+    if (button) selectTf(button.dataset.selectTf);
+  });
+  $("wire")?.addEventListener("click", e => {
+    const button = e.target.closest("[data-news-id]");
+    if (button) openNewsDetail(button.dataset.newsId);
+  });
+  $("cal-body")?.addEventListener("click", e => {
+    const button = e.target.closest("[data-ev-key]");
+    if (button) openCalDetail(button.dataset.evKey);
+  });
+  $("hi-only")?.addEventListener("change", () => {
+    localSet(HIGH_IMPACT_PREF, String($("hi-only").checked));
+    renderCalendar();
+  });
+
+  poll();
+  scheduleFallbackRefresh();
+  setInterval(tick, 1000);
+});

static/news_summary.php

@@ -0,0 +1,268 @@
+<?php
+/**
+ * On-demand gold-impact AI summary for one published headline.
+ *
+ * The ONLY dynamic piece of the otherwise-static dashboard. Serves
+ * GET ?id=<sha1-of-headline> and returns {"summary": ..., "generated_at": ...}.
+ *
+ * Abuse model (the page is public):
+ *   - WHITELIST: the id must match a headline in the local news.json —
+ *     arbitrary text can never reach the LLM through this endpoint.
+ *   - CACHE FOREVER: headlines are immutable, so each id costs at most
+ *     one DeepSeek call, ever (ai/<id>.json). Cache is checked BEFORE
+ *     the whitelist on purpose: summaries stay reachable after the
+ *     headline ages out of the 24 h feed window.
+ *   - DAILY CAP: file-based UTC-day counter, FAIL CLOSED on counter I/O
+ *     errors (a perms mistake must not mean unmetered spend).
+ *
+ * DeepSeek doctrine (mirrors llm_client.py — deliberate local copy, the
+ * dashboard never imports bot code and this file never reads the repo's
+ * config.json):
+ *   - STATIC byte-identical system prompt (prefix-cache pricing).
+ *   - "thinking": disabled is MANDATORY for V4-Flash (else it burns the
+ *     timeout reasoning), temperature 0, json response_format.
+ *   - One attempt, no retries; every failure returns a JSON error the
+ *     page renders as a graceful fallback.
+ *
+ * Key file lives OUTSIDE the docroot: /etc/gold-dashboard/deepseek.key
+ * (root:apache 640). Override paths/caps via Apache SetEnv:
+ * DEEPSEEK_KEY_FILE, AI_SUMMARY_DAILY_CAP.
+ */
+
+const NEWS_FILE = __DIR__ . '/news.json';
+const CACHE_DIR = __DIR__ . '/ai';   // apache-owned; created at deploy time
+const API_URL   = 'https://api.deepseek.com/chat/completions';
+const MODEL     = 'deepseek-v4-flash';
+const TIMEOUT   = 12;
+const MAX_TOKENS = 200;
+const SUMMARY_MAX_CHARS = 400;
+
+// .ht_ prefix: Apache's default <Files ".ht*"> rule denies these even
+// though they sit in the public docroot.
+const CAP_FILE = CACHE_DIR . '/.ht_cap.json';
+
+const SYSTEM_PROMPT = <<<'PROMPT'
+You are a forex/gold news analyst. You receive one news item as JSON:
+{"headline", "description", "score", "xau_score", "keywords", "currencies"}.
+"xau_score" is the platform's automated gold-effect score for this item,
+-1.0 (gold down) to 1.0 (gold up); "score" is overall market sentiment.
+Both are automated and can be wrong — judge the news yourself.
+In 1-2 plain-English sentences (max 45 words) explain what this news
+means for the GOLD (XAU/USD) price and WHY it is bullish, bearish, or
+neutral. If your reading clearly contradicts xau_score, add "(the
+automated score disagrees)". Never write the words "xau_score" or
+"score" — a general reader sees this text. Write for someone who has
+never traded: short simple sentences, no acronyms or abbreviations
+(say "the US central bank", not "the Fed"; "interest rates", not
+"yields").
+Gold doctrine: hawkish Fed / rate hikes / rising real yields / strong
+dollar are gold-bearish; easing / rate cuts / safe-haven flows /
+geopolitical stress / weak dollar are gold-bullish.
+Reply with ONLY JSON: {"summary": "..."}
+PROMPT;
+
+function respond(int $status, array $obj, string $cache = 'no-store'): void
+{
+    http_response_code($status);
+    header('Content-Type: application/json; charset=utf-8');
+    header('Cache-Control: ' . $cache);
+    echo json_encode($obj, JSON_UNESCAPED_SLASHES | JSON_UNESCAPED_UNICODE);
+    exit;
+}
+
+function atomic_write(string $path, string $data): bool
+{
+    $tmp = $path . '.tmp.' . getmypid();
+    $written = @file_put_contents($tmp, $data);
+    // A SHORT write (disk full / quota) returns a byte count, NOT false.
+    // Renaming that into place would publish a truncated JSON body which
+    // the cache-first branch above then serves forever.
+    if ($written === false || $written !== strlen($data)) {
+        @unlink($tmp);
+        return false;
+    }
+    if (!@rename($tmp, $path)) {
+        @unlink($tmp);
+        return false;
+    }
+    return true;
+}
+
+// ── 1. id validation ────────────────────────────────────────────────
+$id = $_GET['id'] ?? '';
+if (!preg_match('/^[a-f0-9]{40}$/', $id)) {
+    respond(400, ['error' => 'bad_id']);
+}
+
+// ── 2. cache first (immutable headlines → summaries never change) ───
+$cachePath = CACHE_DIR . '/' . $id . '.json';
+$cached = @file_get_contents($cachePath);
+if ($cached !== false && $cached !== '') {
+    http_response_code(200);
+    header('Content-Type: application/json; charset=utf-8');
+    header('Cache-Control: max-age=3600');
+    echo $cached;
+    exit;
+}
+
+// ── 3. whitelist against the published feed ─────────────────────────
+$news = json_decode((string)@file_get_contents(NEWS_FILE), true);
+if (!is_array($news)) {
+    respond(503, ['error' => 'no_feed']);
+}
+$hit = null;
+foreach ($news['headlines'] ?? [] as $h) {
+    // Fallback sha1(): a publisher that predates the id field still
+    // serves a valid whitelist during the restart window.
+    $hid = $h['id'] ?? sha1($h['headline'] ?? '');
+    if ($hid === $id) {
+        $hit = $h;
+        break;
+    }
+}
+if ($hit === null) {
+    respond(404, ['error'   => 'unknown_id',
+                  'message' => 'Headline is no longer in the feed.']);
+}
+
+// ── 4. API key ──────────────────────────────────────────────────────
+$keyFile = getenv('DEEPSEEK_KEY_FILE') ?: '/etc/gold-dashboard/deepseek.key';
+$apiKey = trim((string)@file_get_contents($keyFile));
+if ($apiKey === '') {
+    respond(503, ['error' => 'no_key']);
+}
+
+// ── 5. per-id stampede lock, then re-check the cache ────────────────
+$lockFh = @fopen(CACHE_DIR . '/.ht_lock_' . $id, 'c');
+if ($lockFh !== false) {
+    flock($lockFh, LOCK_EX);   // blocks while a concurrent winner generates
+    $cached = @file_get_contents($cachePath);
+    if ($cached !== false && $cached !== '') {
+        flock($lockFh, LOCK_UN);
+        fclose($lockFh);
+        http_response_code(200);
+        header('Content-Type: application/json; charset=utf-8');
+        header('Cache-Control: max-age=3600');
+        echo $cached;
+        exit;
+    }
+}
+$releaseLock = function () use (&$lockFh) {
+    if ($lockFh !== false) {
+        @flock($lockFh, LOCK_UN);
+        @fclose($lockFh);
+        $lockFh = false;
+    }
+};
+
+// ── 6. daily cap (fail closed) ──────────────────────────────────────
+// Counted HERE, not before the lock: a burst of clicks on one headline
+// makes exactly one DeepSeek call, and the losers are served from the
+// re-checked cache above — charging them against the budget (and
+// charging a request that then found no key) could exhaust the whole
+// 150/day allowance with zero LLM calls made.
+$dailyCap = (int)(getenv('AI_SUMMARY_DAILY_CAP') ?: 150);
+$capFh = @fopen(CAP_FILE, 'c+');
+if ($capFh === false || !flock($capFh, LOCK_EX)) {
+    $releaseLock();
+    respond(503, ['error' => 'cap_io']);
+}
+$today = gmdate('Y-m-d');
+$cap = json_decode((string)stream_get_contents($capFh), true);
+if (!is_array($cap) || ($cap['day'] ?? '') !== $today) {
+    $cap = ['day' => $today, 'count' => 0];
+}
+if ($cap['count'] >= $dailyCap) {
+    flock($capFh, LOCK_UN);
+    fclose($capFh);
+    $releaseLock();
+    respond(429, ['error' => 'cap_exceeded']);
+}
+// Increment BEFORE the LLM call — attempts count against the cap
+// (same semantics as llm_client._under_cap).
+$cap['count']++;
+rewind($capFh);
+if (ftruncate($capFh, 0) === false
+        || fwrite($capFh, json_encode($cap)) === false) {
+    flock($capFh, LOCK_UN);
+    fclose($capFh);
+    $releaseLock();
+    respond(503, ['error' => 'cap_io']);
+}
+fflush($capFh);
+flock($capFh, LOCK_UN);
+fclose($capFh);
+
+// ── 7. DeepSeek call (exact chat_json body shape) ───────────────────
+$userPayload = json_encode([
+    'headline'    => $hit['headline'] ?? '',
+    'description' => $hit['description'] ?? null,
+    'score'       => $hit['score'] ?? null,
+    'xau_score'   => $hit['score_by_currency']['XAU'] ?? null,
+    'keywords'    => $hit['keywords'] ?? [],
+    'currencies'  => $hit['currencies'] ?? [],
+], JSON_UNESCAPED_UNICODE);
+
+$body = json_encode([
+    'model'    => MODEL,
+    'messages' => [
+        ['role' => 'system', 'content' => SYSTEM_PROMPT],
+        ['role' => 'user',   'content' => $userPayload],
+    ],
+    'response_format' => ['type' => 'json_object'],
+    'temperature'     => 0,
+    'thinking'        => ['type' => 'disabled'],
+    'max_tokens'      => MAX_TOKENS,
+]);
+
+$ch = curl_init(API_URL);
+curl_setopt_array($ch, [
+    CURLOPT_POST           => true,
+    CURLOPT_POSTFIELDS     => $body,
+    CURLOPT_HTTPHEADER     => ['Content-Type: application/json',
+                               'Authorization: Bearer ' . $apiKey],
+    CURLOPT_RETURNTRANSFER => true,
+    CURLOPT_TIMEOUT        => TIMEOUT,
+]);
+$raw = curl_exec($ch);
+$httpCode = curl_getinfo($ch, CURLINFO_RESPONSE_CODE);
+curl_close($ch);
+
+// ── 8. parse (any failure → 502, never cached) ──────────────────────
+$summary = null;
+if ($raw !== false && $httpCode === 200) {
+    $resp = json_decode($raw, true);
+    $content = $resp['choices'][0]['message']['content'] ?? '';
+    $parsed = json_decode((string)$content, true);
+    if (is_array($parsed) && is_string($parsed['summary'] ?? null)
+            && trim($parsed['summary']) !== '') {
+        $summary = trim($parsed['summary']);
+        if (mb_strlen($summary) > SUMMARY_MAX_CHARS) {
+            $summary = mb_substr($summary, 0, SUMMARY_MAX_CHARS) . '…';
+        }
+    }
+}
+if ($lockFh !== false && $summary === null) {
+    flock($lockFh, LOCK_UN);
+    fclose($lockFh);
+}
+if ($summary === null) {
+    respond(502, ['error' => 'llm_failed']);
+}
+
+// ── 9. cache + respond ──────────────────────────────────────────────
+$out = json_encode(['summary' => $summary, 'generated_at' => gmdate('c')],
+                   JSON_UNESCAPED_SLASHES | JSON_UNESCAPED_UNICODE);
+if (!atomic_write($cachePath, $out)) {
+    // Degrade to per-click generation rather than a broken feature —
+    // the daily cap still bounds spend.
+    error_log('news_summary.php: cache write failed for ' . $cachePath);
+}
+if ($lockFh !== false) {
+    flock($lockFh, LOCK_UN);
+    fclose($lockFh);
+}
+http_response_code(200);
+header('Content-Type: application/json; charset=utf-8');
+header('Cache-Control: max-age=3600');
+echo $out;