Add commit-ladder campaign driver; ignore its 26MB results folder

Added: backtest/research/commit_ladder.py (recreates historical configs
via --set on the fixed harness; 2026-08-03 campaign confirmed HEAD is
the best config on all timeframes). Results stay local like all pinned
trade books.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Nsd8HcrV67GbCZXxZEn1Ma

.gitignore

@@ -36,4 +36,3 @@ backtest/__pycache__/
 .env.*
 logs/
 backtest/research/commit_ladder_2026-08/
-backtest/research/llm_news_eval_2026-08/

CLAUDE.md

@@ -11,12 +11,10 @@ Every numeric threshold in `config.py` and `pair_signals/xauusd.py` was backtest
 ## Commands
 
 ```bash
-python test_all.py                      # QA suite: 359 tests, 22 modules, no network (mocked)
+python test_all.py                      # QA suite: 286 tests, 19 modules, no network (mocked)
 python main.py --pair XAUUSD --timeframe H1 --detail   # CLI signal report (--json for full dump)
 python telegram_bot.py                  # the bot (refuses to boot without credentials)
 python oanda_daemon.py                  # auto-execution daemon on Oanda practice account
-python dashboard_publisher.py --once    # web dashboard: one scan+publish cycle to $DASHBOARD_OUT_DIR
-                                        # (omit --once for the candle-close daemon; deploy/README-dashboard.md)
 
 # Backtests
 python backtest/gpu_backtest.py --pair XAUUSD --days 365
@@ -24,11 +22,9 @@ python backtest/gpu_backtest.py --pair XAUUSD --start 2025-01-01 --end 2025-06-0
     --primary-tf 15min --confirm-tfs 1h,4h
 # key flags: --spread --quality A+,A,B,C --strategies --workers --max-hold-bars
 #            --warmup-bars --cooldown-bars --no-cache --export-csv out.csv --detail
-#            --fixed-rr-tp N (research-only: hard TP at N×risk, trail off; verdict
-#            2026-08-05: NOT shipped — backtest/research/fixedtp_2026-08-05.md)
 ```
 
-**`test_all.py` is NOT pytest** — it uses a custom `@test("name")` decorator that runs each test immediately at import; don't grep for `def test_` or try `pytest test_all.py`. Run the whole file; it prints per-module results and exits non-zero on failure. There is no way to run a single test short of commenting modules out. Tests must stay green; every new gate/knob needs tests. When patching weekend logic in tests, patch `providers.base._is_forex_weekend` — but know there are THREE independent weekend implementations (`providers/base.py`, `signal_aggregator.py`, and an inline copy in `backtest/gpu_backtest.py:_is_xau_blocked_by_calendar`); patching the provider one does not reach the other two.
+**`test_all.py` is NOT pytest** — it uses a custom `@test("name")` decorator that runs each test immediately at import; don't grep for `def test_` or try `pytest test_all.py`. Run the whole file; it prints per-module results and exits non-zero on failure. There is no way to run a single test short of commenting modules out. Tests must stay green; every new gate/knob needs tests. When patching weekend logic in tests, patch `providers.base._is_forex_weekend`.
 
 Python 3.12; a `venv/` exists in-repo. Dependencies in `requirements.txt` (numpy, pandas, requests, twelvedata, python-telegram-bot ≥20, websockets).
 
@@ -41,8 +37,7 @@ Precedence for every credential/deployment setting: **env var → config.json
 ### End-to-end pipeline (every scan)
 
 ```
-providers/<active>.fetch_ohlcv()       per TF: single-flight fetch, drop forming candle, validate/clean, cache
+providers/<active>.fetch_ohlcv()       per TF: throttle, fetch, drop forming candle, validate/clean, cache
-                                       (only Twelve Data rate-throttles; Oanda has no client throttle)
   → engine.TechnicalEngine.analyze()   per TF: 9 scored indicators + ATR context → regime → composite score
   → signal_aggregator.analyze_pair()   live entry point (weekend/day filters, fetch, staleness)
       └─ analyze_from_dataframes()     replayable core: MTF agreement → news → bonuses/penalties →
@@ -56,12 +51,11 @@ Signal currency everywhere is `SignalStrength` (−2..+2). DataFrames are UTC tz
 
 - **`providers/`** — one class per provider subclassing `BaseDataProvider` (`providers/base.py`), which owns everything the pipeline depends on: TTL+boundary cache, forming-candle drop, freshness check, validate/clean, paginated `fetch_range` (feeds the backtester from any provider), optional tick streaming. Subclasses implement only `_fetch_raw` + symbol/interval maps. Registry + `make_data_provider(config, name=None)` in `providers/__init__.py`; config.json `data_providers.active` selects the live candle feed and the optional `data_providers.streaming` scalar selects a different provider for the tick stream. A test enforces that subclasses don't override shared base methods (`supports_streaming`/`iter_ticks` are the sanctioned streaming override points). Tiingo streams mid-price ticks (`wss://api.tiingo.com/fx`) — do not re-add spread display from its free FX feed (two-tier indicative quotes, removed 2026-07-23); the no-spread rule extends to the WS stream.
 - **`engine.py`** — `TechnicalEngine`: 9 scored indicators (ema_trend, rsi, macd, bollinger, adx, roc, keltner, divergence, candlestick) + ATR as context. Regime classification (trending_up/down, `trending` = mixed/direction-UNKNOWN, transitioning, ranging) drives per-indicator weight multipliers and regime-aware RSI reinterpretation. Compute order matters (ATR first; MACD always computed even when not a pillar — divergence needs the histogram). Private series ride on the result as `_underscore` attrs; scorers/formatters skip anything `_`-prefixed or weight-0.
-- **`signal_aggregator.py`** — MTF agreement, diminishing-decay penalty system, exhaustion hard block, quality grading, then an ordered chain of post-grade gates (session, XAU SELL quality/vol/hour, regime blocks, M5 noise gates), then three setup generators: XAU ATR (A+/A), generic swing-based, plus the M5 pullback (`XAU_PB`) and chart-pattern (`XAU_CP`, `chart_patterns.py`) entry paths. The PB/CP generators run AFTER the gate chain and force quality to "B"; the CP path deliberately accepts grade-F composites and re-checks only session/day/big-candle (via warning prefixes) plus its own SELL/vol/RSI gates — it does NOT re-check exhaustion or the M5 min-score/MTF gates. **This is validated behavior, not a bug** (2026-08-04 audit: 226/586 trades in the shipped M5 12mo book are XAU_CP, +38.9R, 104 of them exhaustion-flagged and net-positive) — changing it is a rule-1 revalidation event. `TradeSetup.strategy` selects trailing params; trail params are stamped on the setup from the *producing* config so per-TF overrides reach formatters.
+- **`signal_aggregator.py`** — MTF agreement, diminishing-decay penalty system, exhaustion hard block, quality grading, then an ordered chain of post-grade gates (session, XAU SELL quality/vol/hour, regime blocks, M5 noise gates), then three setup generators: XAU ATR (A+/A), generic swing-based, plus the M5 pullback (`XAU_PB`) and chart-pattern (`XAU_CP`, `chart_patterns.py`) entry paths. `TradeSetup.strategy` selects trailing params; trail params are stamped on the setup from the *producing* config so per-TF overrides reach formatters.
 - **`pair_signals/`** — per-pair config factories. **Always use the factory (`produce_signal` / `make_config`), never bare `PlatformConfig()`** — the factory applies the config.json overlay first, then the validated per-TF deltas (M5 hour-9 block, M15 trail arm 0.70, M15 SELL-hour block, etc.; evidence in the docstring). A test enforces factory ≡ inline-config equivalence. Confirmation map: M5→[M15,H1], M15→[M5,H1], H1→[M15,H4], H4→[H1,D1].
 - **`news_engine.py`** — Forex Factory calendar (MyFXBook RSS fallback) + RSS keyword sentiment with gold-inverse keywords. Class-level caches (fresh engine per scan). Optional DeepSeek enrichment via `llm_client.py` (requests-only, no SDK): batched headline sentiment overriding keyword scores, gold-direction on release surprises, and a market-summary blurb (`SignalReport.news_summary`) — **display/warning-only, keyword fallback on any failure, inert without a key**; prompts are static module constants (DeepSeek prefix-cache pricing) and requests send `thinking: disabled` (V4-Flash reasoning otherwise blows the timeout). By XAU-tuned default, event risk and sentiment **warn but do not gate signals** (`event_risk_caps_signals=False`, `sentiment_affects_signals=False`) — the LLM does not change that.
 - **`telegram_bot.py`** — owner-locked bot + subscriber channel; candle-boundary auto-scan loops (M5/M15/H1/H4, boundary+10 s), alert quality threshold + fingerprint dedup (committed only after a successful send), 📰 LLM market-context line on alerts when available, WebSocket level-breach monitor fed by the streaming provider (`data_providers.streaming`, falling back to the active provider; auto-disabled when the resolved provider can't stream or has no API key — Oanda does not stream, Tiingo does). Rotating log: `logs/telegram.log`, 3-h rotation × 240 backups.
-- **`oanda_daemon.py` / `oanda_executor.py`** — practice-account auto-execution. Book-parity rules (see daemon docstring): no broker-side TP (trailing model — TP1/TP2 advisory), 5-bar per-TF cooldown (wall-clock approximation, so it expires over weekends where the backtester would still count bars), trail ratchets at bar close once per *observed* bar (a missed/failed cycle skips those bars entirely — the intra-gap high/low never enters the peak). Accepted divergence: one position total; orders send `positionFill=OPEN_ONLY` so a state/broker divergence cancels the order instead of silently netting. State/broker re-sync runs at boot and after any suspicious fill (`_reconcile_positions`).
+- **`oanda_daemon.py` / `oanda_executor.py`** — practice-account auto-execution. Book-parity rules (see daemon docstring): no broker-side TP (trailing model — TP1/TP2 advisory), 5-bar per-TF cooldown, trail ratchets at bar close exactly once per bar. Accepted divergence: one position total.
-- **`dashboard_publisher.py` / `static/dashboard.html`** — web dashboard as a **static publisher** (live LAMP server): standalone daemon scans at candle closes via `produce_signal` and atomically writes `snapshot.json`/`news.json`/`dashboard.html` to `$DASHBOARD_OUT_DIR` (Apache serves them; page is public by owner decision). **Hard independence rule: never imports, and is never imported by, `telegram_bot`/`oanda_daemon`** — confirmation map and boundary math are deliberate local copies, so a web-side failure can only mean a stale page. Due-TF classification uses the RAW wake time (flooring wake−buffer skips every H1/H4 scan — 2026-08-02 bug). News panel reads `NewsEngine.get_fundamental_context()` plus the additive `get_week_events()` accessor (display-only; no signal path calls it); as a separate process it has its own news caches and DeepSeek daily cap. News/calendar rows are clickable (detail dialog); headlines carry `id` = sha1(headline) in news.json, and exactly one on-demand endpoint exists: `static/news_summary.php` (installed by `install_static`), which serves a DeepSeek gold-impact summary per headline — whitelist-gated against news.json ids, cache-forever in `ai/`, own fail-closed daily cap (150/day, file-based, separate from the publisher's), key at `/etc/gold-dashboard/deepseek.key`, never reads repo config. Deployment checklist: `deploy/README-dashboard.md`.
 - **`backtest/gpu_backtest.py`** — bar-by-bar replay through the real `analyze_from_dataframes` (news off). Realism model (do not regress): next-bar-open entries re-anchored to the raw price path, round-trip `--spread` charged per trade, trade resolution on RAW bars (cleaning applies only to signal-side data), same-bar SL+TP tie → SL first, gap fills at bar open, trail ratchet at bar close only. Indicators are precomputed once per TF (CuPy GPU or NumPy fallback — check the `Backend:` log line); replay fans out over worker processes and the 5-bar cooldown is applied in an ordered pass afterwards, so parallel and sequential books are identical. Parquet cache in `backtest/cache/` keyed by provider/symbol/TF/window. `backtest/*.csv` are gitignored pinned trade books from past campaigns — used for manual before/after comparison. `backtest/instructions_for_backtest.txt` lists what every backtest report must record.
 
 ## Rules for making changes (read before editing signal logic)
@@ -76,13 +70,6 @@ Signal currency everywhere is `SignalStrength` (−2..+2). DataFrames are UTC tz
 8. Don't add config fields nothing reads; don't disable indicators in production paths (`disabled_indicators` is for experiments only).
 9. EUR/USD was removed 2026-07-16 after a no-edge verdict (cost-to-volatility; last commit with EUR support: `7d3fd95`). Do not re-add without new data. BTC/ETH work lives on unmerged branches only.
 
-## 2026-08-04 QA audit — owner decisions ledger
-
-- **Weekend calendar: FIXED (owner-approved 2026-08-04).** All three `_is_forex_weekend` copies are NY-anchored (Fri 17:00 → Sun 18:00 ET; during US DST that is Fri 21:00 UTC — measured on 5yr of Oanda bars). Kills the weekly false-"STALE DATA/feed lagging" hours. Book impact, fully accounted: M15 byte-identical; M5 12mo dropped exactly ONE trade — a phantom whose signal fired on the final summer-Friday bar (close 21:00 UTC) and "filled" at Sunday's reopen 49 h later (−1.08R, live-impossible). M5 12mo reference at HEAD: 555 trades, +98.64R PF≈1.50 @ $0.65 (was 556/+97.56/1.48). A Module 21 test sweeps all three copies hourly across both DST transitions.
-- **Spread in exit-trigger detection (rule 6): owner DECLINED 2026-08-04.** Backtest stop triggers stay on raw mid High/Low; the measured optimism (~4–6% of total R: −3.55R @ $0.30, −4.24R @ $0.65 on the M15 12mo book) is accepted and books stay comparable. Re-open only as a conscious re-pin campaign.
-- **`FB2_EMA_LO`: HARDENED 2026-08-04 (delegated).** Research override for the EMA200-proximity band, default = tuned 0.5; an unparseable value now degrades to the default with a warning instead of crashing every scan. Documented here — it is a knob that exists.
-- **H4 seeding divergence: parked.** Live H4 fetches only 380 bars vs the backtest's unbounded history — ~2% of sampled H4 bars flip an indicator signal (composite Δ up to 12.3). M5/M15/H1 clean at their real fetch depths. Fold into any future H4 campaign; H4 remains "unproven for auto-execution".
-
 ## Current state notes (2026-07-30)
 
 - Oanda v20 is the live feed (`data_providers.active: "oanda"`), but the pinned backtest books are still Twelve-Data-pinned and the backtest cost assumption is $0.30 while measured real spreads are ~$0.65–0.72 — re-pin books on Oanda bars before treating feed-switch comparisons as valid (see Oanda/TD divergence study in memory).

OANDA_MIGRATION_ROADMAP.md

@@ -0,0 +1,85 @@
+# Full Migration to Oanda v20 as Market-Data Source — Roadmap
+
+**Status: partially executed (updated 2026-08-03).** Phases 1–2 shipped 2026-07-28 via the `providers/` package refactor and the live feed switched to Oanda (`data_providers.active: "oanda"`). Phases 3–6 remain unbuilt: the level-breach stream monitor is simply disabled for Oanda, and the pinned backtest books are still Twelve-Data-pinned at the $0.30 cost assumption. Owner decision 2026-07-24 stands: Twelve Data is a **permanent fallback** (no removal phase).
+
+## Context
+
+The daemon already executes trades on an Oanda practice account, and since 2026-07-28 the signal feed (Telegram bot, CLI, backtester) also runs on Oanda. Oanda's API is free with no metering (120 req/s vs TD's 55 credits/min), serves real bid/ask spreads, and matches the actual trading calendar. The 2026-07-24 divergence study (`backtest/research/oanda_td_divergence.py`) proved the switch is **not plumbing-only**: steady-state bars differ by median $0.69 (11% of M15 ATR), bar direction agrees only 89.9%, TD prints ~4,000 phantom Saturday bars/yr + 21:00 UTC rollover bars that contaminate Monday indicator windows, and Oanda's real spread (~$0.72 liquid / $0.84 rollover) is ~2.4× the $0.30 the validated books assume. So the migration = plumbing (Phases 1–3, inert) + shadow evidence (4) + book re-validation with owner gates (5) + reversible cutover (6). CLAUDE.md rules 1 (no tuned-value change without re-validation) and 4 (live == backtest parity) govern throughout.
+
+**Master sequencing rule:** Phases 1–3 land inert behind `data_providers.active == "twelve_data"`. Nothing changes live behavior until the Phase 6 config flip, which stays reversible forever (TD is a permanent fallback).
+
+## Phase 1 — Provider plumbing (inert)
+
+Files: `oanda_provider.py`, `data_provider.py`, `config.py`, `config.json.example`.
+
+1. **Constructor self-resolution**: `OandaDataProvider.__init__(config, api_token=None, base_url=None)` — defaults resolve via existing `app_config.oanda_api_token()` / `oanda_base_url()`; explicit args keep `oanda_daemon.py:180` working. Fail lazily at first fetch (mirror ForexDataProvider's `_td is None` pattern) so credential-less construction doesn't break the hermetic test header.
+2. **Registry**: add `"oanda": OandaDataProvider` to `PROVIDER_REGISTRY` (`data_provider.py:392`); import at bottom of `data_provider.py` (one-direction edge, no cycle — `oanda_provider` must never import from `data_provider`). Add `PROVIDER_NAME` class attr to both providers. Cutover config needs an `"oanda": {}` block under `data_providers` (creds stay in the existing top-level `oanda` section / env).
+3. **Dedupe shared constants**: move `CANDLE_SECONDS` / `BARS_PER_DAY` to `config.py`, alias as class attrs in both providers — all existing call sites (`signal_aggregator.py:18,217,640,942`, `telegram_bot.py:1062`, `gpu_backtest.py:975,1706`, tests) keep working unchanged. Also move `_is_forex_weekend` to `config.py` eventually (currently duplicated + sync-tested).
+4. **`_fetch_window(symbol, tf, start, end)`** on both providers — the backtester's common interface. Contract: raw closed bars labeled in [start, end], oldest-first, capitalized OHLCV, UTC index; empty DataFrame (not exception) for empty windows; caller keeps windows ≤ ~4500 bars. TD: thin wrapper over existing `_fetch_twelve_data(start_date=, end_date=)` (keep its exception behavior byte-identical). Oanda: new `from`/`to` RFC3339 request (`price=MBA, dailyAlignment=0, alignmentTimezone=Etc/UTC`); refactor `_fetch_oanda_candles`'s parsing body (complete-filter, mid extraction, spread cache) into a shared `_parse_candles()`. The from+count quirk (Oanda's `count` includes the boundary bar; see `backtest/research/fetch_oanda_m15.py`) does not apply to from/to windows. Add a no-op-by-construction `_drop_forming_candle` to OandaDataProvider (`gpu_backtest.py:1025` calls it unconditionally). No throttle needed for Oanda.
+5. **Tests (extend Module 15)**: registry selection; constructor self-resolution + explicit-args-win; `_fetch_window` both providers (mock `_session.get` / `_fetch_twelve_data`); CANDLE_SECONDS/BARS_PER_DAY cross-provider sync; provider-contract hasattr test covering everything callers reach for (`fetch_ohlcv, get_pair_info, clear_cache, _is_data_fresh, _has_latest_closed_bar, _validate_and_clean, _drop_forming_candle, _fetch_window, CANDLE_SECONDS, BARS_PER_DAY, _cache, _cache_timestamps, PROVIDER_NAME`); port Module 11's two cache/boundary tests (`test_all.py:2278,2313`) to Oanda variants patching `_fetch_oanda_candles`. Tests that swap `set_app_config` must restore `_TEST_APP` in `finally`.
+
+## Phase 2 — Backtester provider-agnosticism
+
+File: `backtest/gpu_backtest.py` (`fetch_history`, lines 943–1036).
+
+1. Line 984: `provider._fetch_twelve_data(...)` → `provider._fetch_window(...)`. Keep the backward-stepping chunk loop, `empty_streak` bail, cursor stepping byte-identical (rule 6 — this loop produced every pinned cache).
+2. Lines 975/1706: `ForexDataProvider.BARS_PER_DAY` → `provider.BARS_PER_DAY`.
+3. **Provider-namespaced cache keys** in `_cache_path` (line 938): TD keeps legacy names (pinned parquets stay loadable, needed for A/B), Oanda gets `oanda_` prefix (`oanda_XAU_USD_15min_....parquet`) — no collisions. Comment the asymmetry.
+4. `clean=True/False` raw-path semantics untouched.
+5. **Verification before merge**: one M15 12-month TD replay pre/post commit — trade books must be identical. **Alignment audit**: fetch one month of Oanda H4 + D1 via `_fetch_window`, assert label sets match TD's cached labels (H4 at 00/04/08 UTC, D1 at 00:00 UTC) — a mismatch would silently shift every MTF confirmation.
+6. Tests: `fetch_history` dispatch with a stub provider (concat/dedupe/trim, clean=False skips cleaning); `_cache_path` namespacing regression.
+
+## Phase 3 — Telegram bot: TD WebSocket → Oanda pricing stream
+
+File: `telegram_bot.py`; new resolver in `app_config.py`.
+
+1. New `oanda_stream_url()` resolver: env `OANDA_STREAM_URL` → config.json `oanda.stream_url` (add to `_KNOWN_OANDA`) → derived `oanda_base_url().replace("api-", "stream-", 1)`.
+2. Pure parser `_parse_stream_line(line) -> Optional[(pair_symbol, mid_price)]`: type==PRICE → mid from closeoutBid/Ask (fallback bids[0]/asks[0]); HEARTBEAT/garbage → None. `_STREAM_TO_PAIR` from `OandaDataProvider.OANDA_INSTRUMENT` (replaces TD-display-keyed `_WS_TO_PAIR`).
+3. Reader thread (no new deps): `requests.get(.../v3/accounts/{id}/pricing/stream?instruments=..., stream=True, timeout=(10,30))`, iterate `iter_lines(chunk_size=1)` — **chunk_size=1 is load-bearing** (urllib3 buffering vs 5s server heartbeats); push lines to asyncio queue via `loop.call_soon_threadsafe`; interrupt a blocked read with `resp.close()` from the async side.
+4. Async monitor `_oanda_stream_monitor`: dedicated thread (NOT the 4-worker scan executor), `await asyncio.wait_for(queue.get(), timeout=15)` as staleness watchdog (3× server heartbeat) → reconnect with existing exponential-backoff pattern. PRICE lines feed the identical existing body: `_check_level_breach` → `_handle_breach` → M15 rescan, 5-min cooldown. No client heartbeat.
+5. **Provider-gated selection** in `_start_scheduler` (line 889): `active_provider == "oanda"` → stream monitor, else existing `_ws_monitor`. This is the rollback hinge and stays permanently (TD fallback). `_stop_scheduler` must stop the reader thread. Generalize `/status` labels (`_status_text` iterates `SCAN_PAIRS`, not `_WS_TO_PAIR`).
+6. Tests: `_parse_stream_line` variants (PRICE/closeout, PRICE/arrays, HEARTBEAT, malformed, unknown instrument); `_STREAM_TO_PAIR` round-trip; monitor selection under both `active` values; watchdog-reconnect (structure one connection lifecycle as a testable inner coroutine).
+
+## Phase 4 — Shadow / parity evidence (3–4 weeks, no hot-path code)
+
+The daemon already runs the full signal stack on Oanda bars. Formalize evidence before any cutover:
+
+1. **Code parity (exact)**: replay the daemon's shadow window through the backtester on Oanda data (Phase 2 enables this); diff against daemon-logged decisions per (tf, bar_close_ts) — signal, quality, direction, entry/SL/TP. Acceptance: 100% modulo documented daemon divergences (single-position rule, quality gate). Any mismatch is a bug, not feed noise.
+2. **Feed divergence (measured, not zero-gated)**: `backtest/research/shadow_compare.py` parses `logs/telegram.log` + daemon log into per-scan rows: signal agreement rate, direction flips, grade drift, clustering by hour/weekday. Expect high-single-digit % divergence concentrated near grade thresholds and Monday sessions (TD Saturday/rollover bars). Deliverable: written note attributing every divergence class.
+3. **Operational telemetry**: boundary grace-retry frequency (is Oanda's just-closed bar present at boundary+10s?), `get_spread()` time series (live input to the Phase 5 cost decision), stream disconnect counts, Monday-open indicator snapshots TD-vs-Oanda.
+4. **Owner note**: at cutover the data feed host family should match the execution host (practice vs live pricing can micro-differ).
+
+## Phase 5 — Re-validation campaign (rule 1, owner gates)
+
+Fetch Oanda history (M5/M15/H1/H4/D1, 2020-07 → present; Oanda XAU reaches ~2005). Run matrix per primary TF (M5/M15/H1, current factory configs): 12-mo train/val halves, 5-yr, OOS 2024-06→2025-03 — each at `--spread 0.30` (pure feed-effect A/B vs pinned TD books), `0.70` (honest Oanda cost), `0.85` (rollover stress). Caution: Jul–Nov 2025 is provider-sensitive (weekly offsets −$6..+$18.5) — lean on 5-yr + OOS for decisions. Acceptance unchanged: positive both halves, PF ≥ 1.3, trailing exits, next-open fills, OOS positive.
+
+**Owner decision gates (never silent):**
+1. **Canonical cost** — (a) `round_trip_cost=0.70` everywhere (moves the live TP1 spread gate $9→$21 — a strategy re-tune); (b) `default_spread=0.70` for backtest PnL but retune `min_tp_cost_multiple` 30→~13 to preserve the validated $9 gate; (c) keep $0.30 with documented optimism (only if $0.70 books pass anyway). Full acceptance matrix at chosen values.
+2. **Degradation handling** per failing TF: re-run tuning grids on Oanda data / drop TF from `scanning.timeframes` / keep Oanda execution-only (status quo).
+3. **Re-pin**: passing books become the new pinned CSVs + factory-docstring evidence; CLAUDE.md tuned-value annotations update in the cutover commit, not before.
+
+Files (per decisions): `pair_signals/xauusd.py`, `config.py` (`PAIR_INFO.default_spread`, `SignalConfig.round_trip_cost`/`min_tp_cost_multiple`), `test_all.py` tuned-constant assertions, CLAUDE.md.
+
+## Phase 6 — Cutover (reversible forever; no TD removal)
+
+1. config.json: `data_providers.active: "oanda"` + `"oanda": {}` block; restart bot. Everything flips through two seams: `make_data_provider` (bot shared dp, CLI, backtester) and the Phase 3 monitor selector. Any Phase 5 tuned-value changes land in this commit with their validation records.
+2. **Bake ≥1 week** (full weekly cycle incl. Monday open): grace-retry rate, stream stability, alert plausibility vs daemon, `/cache` freshness.
+3. **Rollback = flip `active` back** — TD code, key, caches, and WS monitor all stay maintained indefinitely (owner decision). Dual-provider posture: cross-provider sync tests (Phase 1) keep both honest; TD subscription/key must stay valid to remain a real fallback.
+4. Docs: CLAUDE.md (pipeline header, data_provider section, env-var table + `OANDA_STREAM_URL`, backtester cache note, comparability note: "pre-Oanda books are TD-feed books"), `docs/` guides if they name the data source.
+
+## Key risks
+
+1. **Cost-gate coupling**: naively setting `round_trip_cost=0.70` silently moves the live $9 TP1 gate to $21 — owner gate 5.1 exists precisely for this.
+2. **Cutover before re-pinning** — feed divergence is material (11% of ATR, 10% direction flips); sequencing rule prevents it.
+3. **Hermetic test header** — `_TEST_APP` pins twelve_data; provider-swapping tests must restore it or later modules fail mysteriously.
+4. **Stream liveness triad** — `chunk_size=1`, `resp.close()` interruption, 15s watchdog.
+5. **H4/D1 alignment** — audit before trusting any Oanda backtest.
+6. **Monday/rollover divergence class** — attribute in Phase 4 evidence; do not "fix".
+
+## Verification (per phase)
+
+- Phases 1–3: `python test_all.py` green (235 + new); live behavior provably unchanged (TD active).
+- Phase 2: pre/post TD book identity diff; Oanda H4/D1 alignment audit.
+- Phase 3: soak a second bot instance with `active=oanda` scratch config; observe stream reconnects in log.
+- Phase 5: full acceptance matrix, owner sign-offs recorded in factory docstrings.
+- Phase 6: 1-week bake checklist above; rollback drill (flip back, confirm TD scan cycle) once during bake.

backtest/gpu_backtest.py

@@ -40,13 +40,10 @@ from dataclasses import dataclass, field
 from datetime import datetime, timedelta, timezone
 from pathlib import Path
 from typing import Dict, List, Optional, Tuple
-from zoneinfo import ZoneInfo
 
 import numpy as np
 import pandas as pd
 
-_NY_TZ = ZoneInfo("America/New_York")
-
 # Ensure project root on sys.path so this script can be invoked from anywhere.
 PROJECT_ROOT = Path(__file__).resolve().parent.parent
 if str(PROJECT_ROOT) not in sys.path:
@@ -405,11 +402,6 @@ class GPUTechnicalEngine(TechnicalEngine):
         # cache key = timeframe.value (e.g. "15min", "1h", "4h")
         self.caches = caches
         self._current_tf: Optional[str] = None
-        # Cache-miss fallbacks recompute with the live engine's pandas
-        # math (different EWM seeding than the precomputed cache). That
-        # is a parity hazard, so it must never happen silently.
-        self.fallback_count = 0
-        self._fallback_warned = False
 
     def analyze(self, df: pd.DataFrame, symbol: str, timeframe: str) -> TechnicalAnalysis:
         # Latch the timeframe so per-indicator overrides know which cache to use.
@@ -427,23 +419,9 @@ class GPUTechnicalEngine(TechnicalEngine):
             return None, -1
         cache = self.caches.get(self._current_tf)
         if cache is None:
-            self._note_fallback(f"no cache for TF {self._current_tf!r}")
             return None, -1
         pos = cache.index_pos.get(df.index[-1])
-        if pos is None:
+        return (cache, pos) if pos is not None else (None, -1)
-            self._note_fallback(
-                f"{self._current_tf}: slice end {df.index[-1]} not in cache")
-            return None, -1
-        return cache, pos
-
-    def _note_fallback(self, why: str) -> None:
-        self.fallback_count += 1
-        if not self._fallback_warned:
-            self._fallback_warned = True
-            logger.warning(
-                "GPU cache miss → falling back to live-engine recompute "
-                "(%s). Further misses counted silently; check "
-                "fallback_count at end of run.", why)
 
     # ── EMA overrides ──────────────────────────────────────────────
 
@@ -551,11 +529,6 @@ class GPUTechnicalEngine(TechnicalEngine):
             signal = SignalStrength.NEUTRAL
             conf = 40
 
-        # Mirrors the live engine's ema_partial_neutral squash (parity).
-        if c.ema_partial_neutral and signal in (SignalStrength.BUY, SignalStrength.SELL):
-            signal = SignalStrength.NEUTRAL
-            conf = 40
-
         crossover_desc = ""
         # Live: `if len(ema_fast) >= 3:` then iloc[-2] — needs len ≥ 2 functionally,
         # but live's guard requires len ≥ 3 so we match that to stay byte-identical.
@@ -848,12 +821,7 @@ class GPUTechnicalEngine(TechnicalEngine):
         c = self.config
         n = c.roc_period
 
-        # Mirror the live guard (engine._compute_roc: len < n + 5 → skip,
+        if pos < n + 1:
-        # i.e. emit only from pos >= n + 4). Unreachable in practice —
-        # analyze() floors at 30 bars and n+4 < 30 for shipped periods —
-        # but keep the engines textually aligned so a future period/floor
-        # change can't open a silent divergence window.
-        if pos < n + 4:
             return
 
         roc_val = float(cache.roc[pos])
@@ -903,11 +871,6 @@ class GPUTechnicalEngine(TechnicalEngine):
             return super()._compute_keltner(df, result)
 
         c = self.config
-        # Mirror the live guard (engine._compute_keltner: len < max(kc_period,
-        # kc_atr_period) + 5 → skip). Unreachable below the 30-bar analyze()
-        # floor with shipped periods; kept aligned for the same reason as ROC.
-        if pos < max(c.kc_period, c.kc_atr_period) + 4:
-            return
         latest = float(df["Close"].iloc[-1])
         kc_upper_val = float(cache.kc_upper[pos])
         kc_lower_val = float(cache.kc_lower[pos])
@@ -1011,16 +974,6 @@ def fetch_history(
     path = _cache_path(symbol, timeframe, start, end, provider.name)
     if use_cache and path.exists():
         logger.info(f"Loading cached {symbol} {timeframe.value}: {path.name}")
-        # Files written before the raw-cache change (2026-07-08, commit
-        # 6a96ed6) contain CLEANED bars; loading one with clean=False
-        # would silently miss the spike bars the raw path exists to model.
-        # The key format predates versioning, so detect by mtime.
-        _RAW_CACHE_EPOCH = 1783468800.0   # 2026-07-08 00:00 UTC
-        if path.stat().st_mtime < _RAW_CACHE_EPOCH:
-            logger.error(
-                f"{path.name} predates the raw-cache format (2026-07-08) "
-                f"and likely contains CLEANED bars — delete it and refetch "
-                f"before trusting trade resolution on this window")
         df = pd.read_parquet(path)
         if df.index.tz is None:
             df.index = df.index.tz_localize("UTC")
@@ -1172,15 +1125,6 @@ def _simulate_trailing(
         reason = "TIMEOUT"
         for j in range(n):
             if np.isnan(h[j]):
-                # Data ends here: close at the LAST VALID bar, not at
-                # c[n-1] (which may itself be NaN — a NaN exit price used
-                # to poison r_multiple and every downstream metric).
-                if j == 0:
-                    exit_idx = 0
-                    reason = "NO_DATA"
-                else:
-                    exit_idx = j - 1
-                    exit_price = float(c[exit_idx])
                 break
             hit = (l[j] <= cur_sl) if buy else (h[j] >= cur_sl)
             if hit:
@@ -1202,15 +1146,8 @@ def _simulate_trailing(
                        else peak + dist * risk)
                 cur_sl = max(cur_sl, lvl) if buy else min(cur_sl, lvl)
 
-        if reason == "NO_DATA":
-            t.exit_reason = reason
-            continue
         if exit_price is None:
             exit_price = float(c[exit_idx])
-        if not np.isfinite(exit_price):
-            # Last-resort guard: never emit a NaN exit/R into the book.
-            t.exit_reason = "NO_DATA"
-            continue
         t.exit_reason = reason
         t.bars_held = exit_idx + 1
         t.exit_price = float(exit_price)
@@ -1383,12 +1320,6 @@ def simulate_trades(
             last_idx = int(n_valid) - 1
             t.exit_price = float(closes_arr[i, last_idx])
             t.exit_time = exit_times[i][last_idx]
-            if not np.isfinite(t.exit_price):
-                # NaN close on the final window bar (corrupt raw data)
-                # must not become a NaN R.
-                t.exit_reason = "NO_DATA"
-                t.exit_price = None
-                continue
 
         # Realized R from actual fill prices, net of round-trip spread.
         risk = abs(t.entry_price - t.sl)
@@ -1441,12 +1372,6 @@ class BacktestConfig:
     # Round-trip transaction cost in price units, charged on every trade
     # (XAU/USD retail spread ≈ $0.30). 0 disables.
     spread: float = 0.30
-    # RESEARCH (2026-08): >0 replaces the exit model with a hard TP at
-    # this multiple of risk from entry (2.0 = fixed 1:2 RR); trailing is
-    # disabled for ALL trades, including per-trade XAU_PB overrides.
-    # Entries/stops/gates are untouched, so the signal book is identical
-    # to the baseline — only the exit policy changes. Never shipped.
-    fixed_rr_tp: float = 0.0
     # Replay worker processes for the bar loop. 0 = auto (cpu_count - 2,
     # capped at 12), 1 = sequential in-process. Signal evaluation per bar
     # is pure (past data only), so it fans out across forked workers; the
@@ -1459,22 +1384,18 @@ class BacktestConfig:
 def _is_xau_blocked_by_calendar(ts: pd.Timestamp, cfg: PlatformConfig) -> Optional[str]:
     """Apply XAU weekend + allowed-weekday filter matching live `analyze_pair`.
 
-    Market closed Friday 17:00 -> Sunday 18:00 NEW YORK time (= Fri
+    Forex market closed: Friday 22:00 UTC -> Sunday 22:00 UTC (Sydney open).
-    21:00 UTC in US summer, 22:00 in winter — matches measured Oanda
+    Must match `_is_forex_weekend` in signal_aggregator.py exactly.
-    feed behavior; 2026-08-04 audit I43). Must match `_is_forex_weekend`
-    in signal_aggregator.py exactly (rule 4; Module 21 agreement test).
-    Note the weekday gates below stay on UTC weekday — they mirror the
-    live gates, which evaluate UTC weekdays.
     """
     xau = cfg.xau_strategy
     if xau.weekend_filter:
-        ny = ts.astimezone(_NY_TZ)
+        wd = ts.weekday()
-        wd = ny.weekday()
+        hour = ts.hour
-        if wd == 4 and ny.hour >= 17:
+        if wd == 4 and hour >= 22:
             return "WEEKEND_FILTER"
         if wd == 5:
             return "WEEKEND_FILTER"
-        if wd == 6 and ny.hour < 18:
+        if wd == 6 and hour < 22:
             return "WEEKEND_FILTER"
     if xau.allowed_weekdays and ts.weekday() not in xau.allowed_weekdays:
         return "DAY_FILTER"
@@ -1691,43 +1612,6 @@ def _reanchor_entries_to_raw(
         )
 
 
-def _apply_cooldown(
-    candidates: List[Tuple[int, Trade]], cooldown_bars: int
-) -> List[Tuple[int, Trade]]:
-    """Ordered cooldown pass over (bar_pos, Trade) candidates: drop any
-    candidate within `cooldown_bars` bars of the previously ACCEPTED
-    entry — exactly what a sequential replay loop would have skipped.
-    Candidates must already be sorted by position (stable merge from the
-    worker spans)."""
-    accepted: List[Tuple[int, Trade]] = []
-    last_entry_pos = -10_000
-    for pos, trade in candidates:
-        if pos - last_entry_pos < cooldown_bars:
-            continue
-        accepted.append((pos, trade))
-        last_entry_pos = pos
-    return accepted
-
-
-def _apply_fixed_rr_tp(trades: List[Trade], mult: float) -> None:
-    """RESEARCH: replace each trade's TP with a hard target at
-    `mult` × |entry − sl| and zero the per-trade trail params so the
-    XAU_PB overrides cannot resurrect trailing. Risk is measured off the
-    final (re-anchored) entry/SL, so call this after
-    `_reanchor_entries_to_raw`. The caller must also pass
-    trail_arm_r=0 to `simulate_trades` — dispatch is on the run-level arg."""
-    for t in trades:
-        risk = abs(t.entry_price - t.sl)
-        t.tp1 = (t.entry_price + mult * risk if t.direction == "BUY"
-                 else t.entry_price - mult * risk)
-        t.tp2 = t.tp1              # static branch never reads tp2; CSV hygiene
-        t.trail_arm_r = 0.0
-        t.trail_dist_r = 0.0
-        t.rr_planned = mult
-        if t.pip_value:
-            t.reward_pips = mult * risk / t.pip_value
-
-
 def run_backtest(
     bt_cfg: BacktestConfig,
     platform_cfg: Optional[PlatformConfig] = None,
@@ -1855,10 +1739,18 @@ def run_backtest(
         candidates.extend(cands)
     candidates.sort(key=lambda pc: pc[0])
 
-    # ── Phase 2: entry cooldown — the only sequential rule.
+    # ── Phase 2: entry cooldown — the only sequential rule. A candidate
-    accepted = _apply_cooldown(candidates, bt_cfg.cooldown_bars)
+    # inside the cooldown window of the previously ACCEPTED entry is
-    trades: List[Trade] = [t for _, t in accepted]
+    # dropped, exactly as the sequential loop skipped those bars.
-    signal_ts_list: List[pd.Timestamp] = [primary_df.index[p] for p, _ in accepted]
+    trades: List[Trade] = []
+    signal_ts_list: List[pd.Timestamp] = []
+    last_entry_pos = -10_000
+    for pos, trade in candidates:
+        if pos - last_entry_pos < bt_cfg.cooldown_bars:
+            continue
+        trades.append(trade)
+        signal_ts_list.append(primary_df.index[pos])
+        last_entry_pos = pos
 
     logger.info(
         f"Replay complete: {len(trades)} entries in "
@@ -1878,17 +1770,10 @@ def run_backtest(
         use_cache=bt_cfg.use_cache, clean=False,
     )
     _reanchor_entries_to_raw(trades, signal_ts_list, sim_df)
-    if bt_cfg.fixed_rr_tp > 0:
-        _apply_fixed_rr_tp(trades, bt_cfg.fixed_rr_tp)
-        logger.info(
-            f"RESEARCH override: fixed-RR TP = {bt_cfg.fixed_rr_tp} x risk; "
-            f"trailing exit DISABLED for all {len(trades)} trades (incl. XAU_PB)"
-        )
-    run_arm = 0.0 if bt_cfg.fixed_rr_tp > 0 else platform_cfg.signals.trail_arm_r
     t_sim = time.perf_counter()
     simulate_trades(trades, sim_df, max_bars=bt_cfg.max_hold_bars,
                     spread=bt_cfg.spread,
-                    trail_arm_r=run_arm,
+                    trail_arm_r=platform_cfg.signals.trail_arm_r,
                     trail_dist_r=platform_cfg.signals.trail_dist_r)
     logger.info(
         f"Trade simulation took {time.perf_counter() - t_sim:.2f}s "
@@ -1938,14 +1823,6 @@ def _summarize(rs: List[float]) -> Dict:
 
 def compute_metrics(trades: List[Trade]) -> Metrics:
     resolved = [t for t in trades if t.exit_reason and t.exit_reason != "NO_DATA"]
-    # Last line of defense: one non-finite R (NaN exit on corrupt data)
-    # would silently NaN every aggregate below. Exclude loudly.
-    bad = [t for t in resolved if not np.isfinite(t.r_multiple)]
-    if bad:
-        logger.error(
-            "compute_metrics: excluding %d trade(s) with non-finite R "
-            "(corrupt exit data) — first at %s", len(bad), bad[0].entry_time)
-        resolved = [t for t in resolved if np.isfinite(t.r_multiple)]
     if not resolved:
         return Metrics(0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, {}, {}, {})
 
@@ -2232,10 +2109,6 @@ def _parse_args(argv: Optional[List[str]] = None) -> argparse.Namespace:
                    help="Override signals.trail_arm_r for this run (research knob)")
     p.add_argument("--trail-dist", type=float, default=None,
                    help="Override signals.trail_dist_r for this run (research knob)")
-    p.add_argument("--fixed-rr-tp", type=float, default=None,
-                   help="RESEARCH: exit at a hard TP of MULT x risk (e.g. 2.0 = "
-                        "fixed 1:2 RR) instead of the trailing stop; "
-                        "entries/stops/gates unchanged.")
     p.add_argument("--quality", default="A+,A,B,C", help="Comma-separated allowed grades")
     p.add_argument("--strategies", default="",
                    help="Comma-separated strategy tags to include (XAU,XAU_PB,XAU_CP,GENERIC). Empty = all.")
@@ -2359,16 +2232,6 @@ def main(argv: Optional[List[str]] = None) -> int:
         # make_config; the fallback must match (live == backtest).
         from app_config import apply_pair_overrides  # noqa: E402
         apply_pair_overrides(platform_cfg)
-    if args.fixed_rr_tp is not None:
-        if args.fixed_rr_tp <= 0:
-            raise SystemExit("--fixed-rr-tp must be > 0")
-        if (args.trail_arm is not None or args.trail_dist is not None or
-                any(s.split("=", 1)[0].strip().startswith(("signals.trail",
-                                                           "signals.pb_trail"))
-                    for s in (args.set or []))):
-            raise SystemExit(
-                "--fixed-rr-tp conflicts with --trail-arm/--trail-dist/"
-                "--set signals.*trail* (the flag disables trailing entirely)")
     if args.swing_sr_lookback > 0:
         platform_cfg.technical.swing_sr_lookback = args.swing_sr_lookback
     if args.trail_arm is not None:
@@ -2415,7 +2278,6 @@ def main(argv: Optional[List[str]] = None) -> int:
         strategies_only=[s.strip() for s in args.strategies.split(",") if s.strip()],
         spread=spread,
         workers=args.workers,
-        fixed_rr_tp=args.fixed_rr_tp or 0.0,
     )
 
     logger.info(

backtest/research/fixedtp_2026-08-05.md

@@ -1,123 +0,0 @@
-# Fixed-RR hard-TP vs shipped trailing exit — research campaign 2026-08-05
-
-**Verdict: DO NOT SHIP. The trailing exit stays.** The fixed 2R target beats the trail on
-total R in every 2024+ window but the edge is a trend-regime artifact: it inverts in the
-2021–2023 ranging years, doubles drawdown, cuts win rate 10–19 points, and doubles hold
-times (amplifying the accepted live one-position divergence). Details below.
-
-## Setup
-
-- Code: HEAD `fc5ce85` + the `--fixed-rr-tp` research flag (uncommitted at run time).
-  The flag replaces every trade's TP with `mult × |entry − sl|` AFTER re-anchoring and
-  disables trailing (run-level and per-trade PB params) — entries, stops, gates, and the
-  cooldown pass are untouched, so both arms replay **trade-identical books** (verified:
-  paired join on entry_time+direction matches 1:1 in every window). Only the exit policy
-  differs → per-trade paired comparison.
-- Provider: Oanda (cache-warm; keys `oanda_XAU_USD_{5min,15min,1h}_{20250603_20260804,
-  20240331_20250330,20210531_20260804}`). Charged cost `--spread 0.65` (real measured
-  round trip); $0.30 letter-cost books derived analytically as `r + 0.35/risk` and
-  verified exact against one real `--spread 0.30` run (max per-trade error 1.8e-15 —
-  spread never enters trigger detection, per the owner-declined 2026-08-04 decision).
-- Control sanity gate: fresh M5 12mo control reproduced the pinned HEAD reference
-  exactly — 555 trades, +98.64R, PF 1.483, maxDD −10.73R.
-- Commands (control = same without the flag):
-  `venv/bin/python backtest/gpu_backtest.py --pair XAUUSD --primary-tf {5min,15min}
-   --start 2025-08-04 --end 2026-08-04 --spread 0.65 [--fixed-rr-tp 2.0]
-   --export-csv backtest/fixedtp_*.csv`
-  OOS `--start 2024-06-01 --end 2025-03-30`; 5yr `--start 2021-08-01 --end 2026-08-04`.
-- Books pinned as `backtest/fixedtp_{m5,m15}_{12mo,oos,5yr}_{ctrl,rr20,...}.csv`
-  (per-trade indicator breakdown, agreement counts, regime, vol-regime, weekday,
-  MAE/MFE, bars-held — the full instructions_for_backtest record).
-
-## Headline (@ $0.65)
-
-| Window | Arm | n | R | PF | WR | maxDD | recov | medBars | exits |
-|---|---|---|---|---|---|---|---|---|---|
-| M5 12mo | trail | 555 | +98.64 | 1.483 | 52.4% | −10.7 | 9.2 | 28 | TRAIL 361 / SL 183 / TO 11 |
-| M5 12mo | 2R hard | 555 | +124.85 | 1.378 | 42.9% | −24.5 | 5.1 | 50 | SL 312 / TP1 210 / TO 33 |
-| M15 12mo | trail | 397 | +73.93 | 1.525 | 62.2% | −10.8 | 6.8 | 28 | TRAIL 255 / SL 135 / TO 7 |
-| M15 12mo | 2R hard | 397 | +95.95 | 1.431 | 44.3% | −11.2 | 8.6 | 39 | SL 215 / TP1 151 / TO 31 |
-| M5 OOS | trail | 86 | +9.45 | 1.332 | 53.5% | −8.2 | 1.2 | 70 | — |
-| M5 OOS | 2R hard | 86 | +18.83 | 1.407 | 45.3% | −9.7 | 1.9 | 124 | — |
-| M15 OOS | trail | 299 | +38.77 | 1.360 | 59.2% | −15.5 | 2.5 | 38 | — |
-| M15 OOS | 2R hard | 299 | +72.14 | 1.435 | 45.2% | −13.9 | 5.2 | 82 | — |
-
-Halves (12mo, split 2026-02-03): M5 rr20 +67.99/+56.86 (both positive; H2 PF 1.325 vs
-ctrl 1.503), M15 rr20 +73.74/+22.21 (both positive; ctrl H2 +7.23 — the known H2-flat).
-At $0.30 all deltas keep the same sign (e.g. M5 12mo +109.0 vs +135.2).
-
-So the **letter** of rule 1 passes for rr20 on both TFs. The 5yr decomposition is why it
-still must not ship.
-
-## Exit migration (paired, M5 12mo; same pattern everywhere)
-
-- SL→SL 183 trades: ΔR 0 (identical by construction — stops untouched).
-- TRAIL→TP1 207 trades: +248.9R → +424.5R (**+175.6R**) — winners the trail exits early.
-- TRAIL→SL 129 trades: +19.0R → −136.8R (**−155.7R**) — retraces the trail survives
-  become full stop-outs.
-- TRAIL→TIMEOUT 25: +7.4R. Net = +26.2R. The whole effect is these two big flows nearly
-  cancelling; which one wins is a regime property, not an exit-policy edge.
-- Trades with ctrl MFE ≥ 2R: 95; hard cap forfeits only +10.4R vs trail on M5 (the trail
-  already gives ~0.75R back from peak, so a 2R cap costs little *in a trend*).
-
-## Mini-grid (12mo @ $0.65) — the tell
-
-| mult | M5 R / PF / DD | M15 R / PF / DD |
-|---|---|---|
-| trail | +98.6 / 1.483 / −10.7 | +73.9 / 1.525 / −10.8 |
-| 1.0R | +52.5 / 1.205 / −12.8 | +67.1 / 1.410 / −7.3 |
-| 1.5R | +80.7 / 1.263 / −21.4 | +66.9 / 1.325 / −11.2 |
-| 2.0R | +124.9 / 1.378 / −24.5 | +96.0 / 1.431 / −11.2 |
-| 2.5R | +147.7 / 1.426 / −23.6 | +110.4 / 1.469 / −15.9 |
-| 3.0R | +155.8 / 1.432 / −21.6 | +135.8 / 1.562 / −17.5 |
-
-Monotone in the multiple on both TFs — "2.0" is not an optimum, it's a point on a curve
-that says *never take profit* (3R's M5 TIMEOUT bucket alone is +84.8R). That is the
-signature of a one-way trend sample, not of a superior exit rule. It also reconfirms the
-original finding that SHORT fixed targets (≤1.5R) are strictly bad (cf. RR-0.5 removal,
-`config.py:126-133`: +48R → +105R).
-
-## 5yr regime decomposition (2021-08-01 → 2026-08-04 @ $0.65) — the verdict
-
-M15 (real sample in every year): ctrl +127.97 PF 1.209 DD −35.8; rr20 +158.97 PF 1.170
-DD **−59.8**; rr30 +204.03 PF 1.203 DD **−98.3**.
-
-| Year | n | trail | 2R hard | 3R hard |
-|---|---|---|---|---|
-| 2021 | 88 | **+4.00** | −16.09 | −17.41 |
-| 2022 | 284 | −9.54 | −7.63 | −26.74 |
-| 2023 | 227 | **+7.79** | −1.71 | −5.81 |
-| 2024 | 345 | +40.44 | +64.98 | +89.14 |
-| 2025 | 405 | +49.33 | +69.39 | +104.42 |
-| 2026 | 229 | +35.95 | +50.04 | +60.43 |
-
-M5 5yr mirrors it (ctrl +128.4 PF 1.387 DD −12.8 vs rr20 +160.2 PF 1.306 DD −24.7;
-2022: −1.1 vs −5.2), though pre-2024 M5 samples are thin (12–60 trades/yr).
-
-The hard target's entire outperformance lives in the 2024–2026 gold trend. In ranging
-years it flips sign exactly where the trail stays positive (M15 2021, 2023), and the
-further the target, the worse the ranging-year bleed and the deeper the DD. The 12mo,
-halves, and OOS windows all sit inside the trend era, which is why the rule-1 letter
-passes — this is precisely the within-regime-split trap rule 1's OOS clause exists to
-catch, one level up.
-
-## Additional non-ship reasons
-
-- **Live divergence grows with hold time.** Median hold doubles (M5 28→50 bars; OOS
-  70→124). The daemon holds one position total; the backtester doesn't model that
-  blocking. Longer holds = more live signals skipped that the paper book banks.
-- **WR 52→43% (M5) / 62→44% (M15), max loss streak 6→11 (M15)** — materially harder to
-  operate and size.
-- Aggregate tables (session/day/regime/vol, streaks, quarterly equity, rolling-20 WR)
-  computed and consistent — no bucket reverses the conclusion; both arms' 12mo profit is
-  concentrated in Q1-2026 (+87.7 ctrl / +102.0 rr20).
-
-## Status
-
-- `--fixed-rr-tp` stays as a documented research-only flag (default off; conflicts with
-  `--trail-arm/--trail-dist/--set signals.*trail*` are hard errors). Module 22 tests
-  (5) pin the override math, TP fill + spread, SL-first tie, PB-trail-disable, and CLI
-  guards. Full suite: 346 green.
-- Re-open only if gold's regime turns and a *regime-conditional* exit (trail in range,
-  target in trend) is proposed with a pre-registered mechanism — a plain fixed ratio is
-  now a closed hypothesis at every multiple in {0.5, 1.0, 1.5, 2.0, 2.5, 3.0}.

backtest/research/llm_news_eval.py

@@ -1,697 +0,0 @@
-"""
-DeepSeek news-analysis validation harness — XAU/USD.
-====================================================
-Research-only (no production code changes). Answers: is the DeepSeek
-layer's news analysis *correct*?
-
-Part A (events): replays every high-impact USD release with a genuine
-surprise (2024-01 → 2026-07, from backtest/cache/calendar.parquet)
-through the REAL `NewsEngine._llm_event_direction` — real prompt, real
-payload, real validation — and scores its "gold bullish/bearish" calls
-against gold's actual move (Oanda M15 mids) at +15m/30m/1h/4h/24h.
-Baselines: mechanical surprise-sign doctrine, FF actualBetterWorse,
-always-bullish/bearish (2024-26 is a gold bull regime), coin flip.
-
-Part B (headlines): a curated golden set (llm_news_golden.csv) through
-the REAL `NewsEngine._llm_headline_scores` in production-sized batches,
-vs the keyword scorer it overrides. Includes repeat-batch (determinism)
-and shuffled-order (cross-contamination) probes.
-
-Implementation notes that keep the replay faithful to production:
-  * `_llm_event_direction`/`_llm_headline_scores` resolve the client via
-    `get_llm_client()`, so we seed the module singleton with a research
-    client (raised daily cap: 586 events > production cap 500).
-  * The event cache key omits the forecast, so a repeat `actual` would
-    silently re-serve a direction computed for a different forecast —
-    the cache is cleared before every call.
-  * A "neutral" reply and a hard failure both return None; they are
-    disambiguated by reading the event cache back after the call
-    (absent = failure, "" = neutral, "bullish:/bearish: …" = call).
-  * Raw request/response bytes are captured with a requests response
-    hook — the request-building code path is untouched.
-
-Usage (from repo root):
-  venv/bin/python backtest/research/llm_news_eval.py events --limit 5
-  venv/bin/python backtest/research/llm_news_eval.py events --determinism 20
-  venv/bin/python backtest/research/llm_news_eval.py headlines
-  venv/bin/python backtest/research/llm_news_eval.py report
-
-Artifacts land in backtest/research/llm_news_eval_2026-08/ (JSONL replay
-logs are resume-safe: completed events are never re-paid). `report` is a
-pure offline function of the artifacts.
-"""
-
-import argparse
-import json
-import math
-import random
-import sys
-import time
-from datetime import datetime, timezone
-from pathlib import Path
-
-import numpy as np
-import pandas as pd
-
-REPO_ROOT = Path(__file__).resolve().parents[2]
-sys.path.insert(0, str(REPO_ROOT))
-
-import llm_client
-from app_config import deepseek_api_key, get_app_config
-from config import NewsConfig
-from llm_client import DEFAULT_MODEL, DeepSeekClient
-from news_engine import EconomicEvent, NewsEngine
-
-RESEARCH_DIR = Path(__file__).parent
-ART_DIR = RESEARCH_DIR / "llm_news_eval_2026-08"
-CALENDAR = REPO_ROOT / "backtest" / "cache" / "calendar.parquet"
-BARS = (REPO_ROOT / "backtest" / "cache" /
-        "oanda_XAU_USD_15min_20210531_20260804.parquet")
-GOLDEN = RESEARCH_DIR / "llm_news_golden.csv"
-EVENTS_JSONL = ART_DIR / "events_replay.jsonl"
-HEADLINES_JSONL = ART_DIR / "headlines_replay.jsonl"
-SCORED_CSV = ART_DIR / "events_scored.csv"
-
-HORIZONS = [("15m", 15), ("30m", 30), ("1h", 60), ("4h", 240),
-            ("24h", 1440)]
-NEUTRAL_BAND = 0.15  # |xau| below this = neutral prediction (Part B)
-BATCH_CAP = 25       # production batch cap (news_engine.py:783)
-EXPECTED_CORPUS = 586
-
-
-# ── shared plumbing ──────────────────────────────────────────────────
-
-class _Recorder:
-    """requests response hook capturing the exact wire traffic."""
-
-    def __init__(self):
-        self.last = None
-
-    def hook(self, resp, **kwargs):
-        body = resp.request.body
-        if isinstance(body, bytes):
-            body = body.decode("utf-8", "replace")
-        self.last = {
-            "request_body": body,
-            "http_status": resp.status_code,
-            "response_body": resp.text,
-            "elapsed_s": round(resp.elapsed.total_seconds(), 3),
-        }
-        return resp
-
-    def pop(self):
-        cap, self.last = self.last, None
-        return cap or {}
-
-
-def install_research_client(timeout: float):
-    """Build a research DeepSeekClient (raised cap) and seed the module
-    singleton so the production code paths use it. Returns (client,
-    recorder)."""
-    app = get_app_config()
-    key = deepseek_api_key(app)
-    ds = app.deepseek_settings
-    if not key:
-        sys.exit("No DeepSeek API key (env DEEPSEEK_API_KEY or "
-                 "config.json deepseek.api_key) — aborting before spend.")
-    if ds.get("enabled") is False:
-        sys.exit("deepseek.enabled is false in config.json — aborting.")
-    client = DeepSeekClient(
-        api_key=key,
-        model=str(ds.get("model", DEFAULT_MODEL)),
-        timeout=timeout,
-        daily_cap=10 ** 6,  # research: corpus 586 > production cap 500
-    )
-    rec = _Recorder()
-    client._session.hooks["response"].append(rec.hook)
-    with llm_client._client_lock:
-        llm_client._client = client
-    return client, rec
-
-
-def _s(x):
-    """Parquet str columns carry float NaN — EconomicEvent needs None."""
-    return None if pd.isna(x) else str(x)
-
-
-def load_corpus():
-    """High-impact USD releases with a genuine surprise, through the
-    production `EconomicEvent.surprise` gate. Asserts run before any
-    API spend."""
-    df = pd.read_parquet(CALENDAR)
-    df = df[(df["impactName"] == "high") & (df["currency"] == "USD")]
-    rows = []
-    for r in df.itertuples():
-        ev = EconomicEvent(
-            time=datetime.fromtimestamp(int(r.dateline), tz=timezone.utc),
-            currency="USD", impact="high", event=_s(r.name) or "",
-            forecast=_s(r.forecast), previous=_s(r.previous),
-            actual=_s(r.actual),
-        )
-        if ev.surprise not in ("positive", "negative"):
-            continue
-        rows.append({
-            "ffid": int(r.id), "dateline": int(r.dateline),
-            "utc_time": ev.time.isoformat(), "name": ev.event,
-            "actual": ev.actual, "forecast": ev.forecast,
-            "previous": ev.previous, "surprise": ev.surprise,
-            "better_worse": int(r.actualBetterWorse), "ev": ev,
-        })
-    ids = [d["ffid"] for d in rows]
-    if len(rows) != EXPECTED_CORPUS:
-        sys.exit(f"corpus size {len(rows)} != {EXPECTED_CORPUS} — "
-                 "calendar.parquet changed; re-verify before spending.")
-    if len(set(ids)) != len(ids):
-        sys.exit("FF ids not unique — resume keying would break.")
-    # Pin the actualBetterWorse mapping on two unambiguous NFP prints.
-    nfp = {d["utc_time"][:10]: d["better_worse"] for d in rows
-           if d["name"] == "Non-Farm Employment Change"}
-    if nfp.get("2024-01-05") != 1 or nfp.get("2024-05-03") != 2:
-        sys.exit("actualBetterWorse mapping pins failed (expected "
-                 "1=better-for-USD on 2024-01-05 beat, 2=worse on "
-                 "2024-05-03 miss).")
-    rows.sort(key=lambda d: (d["dateline"], d["ffid"]))
-    return rows
-
-
-def _load_jsonl(path: Path):
-    recs = []
-    if path.exists():
-        with open(path) as f:
-            for line in f:
-                line = line.strip()
-                if line:
-                    recs.append(json.loads(line))
-    return recs
-
-
-# ── Part A: event replay ─────────────────────────────────────────────
-
-def _extract_reason(cap):
-    try:
-        content = json.loads(cap["response_body"])[
-            "choices"][0]["message"]["content"]
-        return json.loads(content).get("reason")
-    except Exception:
-        return None
-
-
-def cmd_events(args):
-    ART_DIR.mkdir(exist_ok=True)
-    corpus = load_corpus()
-    if args.limit:
-        corpus = corpus[:args.limit]
-    engine = NewsEngine(NewsConfig())
-    existing = _load_jsonl(EVENTS_JSONL)
-    done1 = {r["ffid"] for r in existing
-             if r.get("run_pass") == 1 and r.get("llm_direction")}
-    done2 = {r["ffid"] for r in existing
-             if r.get("run_pass") == 2 and r.get("llm_direction")}
-    pending = [d for d in corpus if d["ffid"] not in done1]
-    print(f"corpus {len(corpus)} | already recorded {len(done1)} | "
-          f"pending {len(pending)} | determinism re-calls "
-          f"{args.determinism} | model calls are ~100 tokens each "
-          f"(deepseek-v4-flash, static-prefix cached) → cost ≈ cents")
-    if not pending and not args.determinism:
-        print("nothing to do")
-        return
-    _, rec = install_research_client(args.timeout)
-
-    out = open(EVENTS_JSONL, "a")
-
-    def call_one(d, run_pass):
-        # Cache key omits forecast → cleared per call so 196 colliding
-        # corpus rows can't be served a stale direction.
-        with NewsEngine._cache_lock:
-            NewsEngine._llm_event_cache.clear()
-        ev = d["ev"]
-        engine._llm_event_direction(ev)
-        key = (ev.event, ev.currency, str(ev.actual or ""))
-        with NewsEngine._cache_lock:
-            cached = NewsEngine._llm_event_cache.get(key)
-        cap = rec.pop()
-        if cached is None:
-            direction, text = None, None       # hard failure → retry
-        elif cached == "":
-            direction, text = "neutral", ""
-        else:
-            direction, text = cached.split(":", 1)[0], cached
-        recd = {k: d[k] for k in (
-            "ffid", "dateline", "utc_time", "name", "actual", "forecast",
-            "previous", "surprise", "better_worse")}
-        recd.update({
-            "llm_direction": direction, "llm_text": text,
-            "reason_raw": _extract_reason(cap), "run_pass": run_pass,
-            "ts": datetime.now(timezone.utc).isoformat(), **cap,
-        })
-        out.write(json.dumps(recd) + "\n")
-        out.flush()
-        return direction
-
-    remaining = pending
-    for npass in range(1, args.retry_passes + 1):
-        if not remaining:
-            break
-        failed = []
-        for i, d in enumerate(remaining):
-            if call_one(d, 1) is None:
-                failed.append(d)
-            if args.sleep:
-                time.sleep(args.sleep)
-            if (i + 1) % 25 == 0:
-                print(f"  pass {npass}: {i + 1}/{len(remaining)} "
-                      f"({len(failed)} failed)")
-        print(f"pass {npass} done: {len(remaining) - len(failed)} ok, "
-              f"{len(failed)} failed")
-        remaining = failed
-    if remaining:
-        print(f"WARNING: {len(remaining)} events still unrecorded after "
-              f"{args.retry_passes} passes — re-run `events` to resume.")
-
-    if args.determinism:
-        ok_ids = {x["ffid"] for x in remaining}
-        done_now = [d for d in corpus if d["ffid"] not in ok_ids]
-        done_now.sort(key=lambda d: (d["name"], d["dateline"]))
-        stride = max(1, len(done_now) // args.determinism)
-        sample = done_now[::stride][:args.determinism]
-        todo = [d for d in sample if d["ffid"] not in done2]
-        print(f"determinism probe: {len(todo)} re-calls "
-              f"({len(sample) - len(todo)} already recorded)")
-        for d in todo:
-            call_one(d, 2)
-            if args.sleep:
-                time.sleep(args.sleep)
-    out.close()
-    print("events replay complete →", EVENTS_JSONL)
-
-
-# ── Part B: golden headlines ─────────────────────────────────────────
-
-def _load_golden():
-    g = pd.read_csv(GOLDEN)
-    need = {"headline", "category", "expected_xau_sign", "rationale"}
-    if not need <= set(g.columns):
-        sys.exit(f"golden CSV missing columns {need - set(g.columns)}")
-    if not g["expected_xau_sign"].isin([-1, 0, 1]).all():
-        sys.exit("expected_xau_sign must be -1/0/+1")
-    if not g["headline"].is_unique:
-        sys.exit("duplicate headlines in golden CSV")
-    return g
-
-
-def cmd_headlines(args):
-    ART_DIR.mkdir(exist_ok=True)
-    g = _load_golden()
-    heads = list(g["headline"])
-    _, rec = install_research_client(args.timeout)
-    engine = NewsEngine(NewsConfig())
-    out = open(HEADLINES_JSONL, "w")  # tiny (~9 calls): fresh each run
-
-    def run_batch(texts, tag, batch_idx):
-        scores = None
-        for attempt in range(3):
-            scores = engine._llm_headline_scores(texts)
-            cap = rec.pop()
-            if scores is not None:
-                break
-            print(f"  {tag} batch {batch_idx} attempt {attempt + 1} "
-                  f"failed (http {cap.get('http_status')})")
-            time.sleep(2.0)
-        out.write(json.dumps({
-            "run_tag": tag, "batch": batch_idx, "headlines": texts,
-            "scores": scores,
-            "ts": datetime.now(timezone.utc).isoformat(), **cap,
-        }) + "\n")
-        out.flush()
-        return scores
-
-    batches = [heads[i:i + BATCH_CAP] for i in range(0, len(heads),
-                                                     BATCH_CAP)]
-    for bi, b in enumerate(batches):
-        ok = run_batch(b, "main", bi) is not None
-        print(f"main batch {bi} ({len(b)} headlines): "
-              f"{'ok' if ok else 'FAILED'}")
-        time.sleep(args.sleep)
-    run_batch(batches[0], "repeat", 0)
-    print("repeat probe done")
-    shuffled = heads[:]
-    random.Random(20260806).shuffle(shuffled)
-    for bi in range(0, len(shuffled), BATCH_CAP):
-        run_batch(shuffled[bi:bi + BATCH_CAP], "shuffled",
-                  bi // BATCH_CAP)
-    print("shuffled probe done →", HEADLINES_JSONL)
-    out.close()
-
-
-# ── report: pure offline function of the artifacts ───────────────────
-
-def _binom_p(k, n):
-    """Exact two-sided binomial p-value vs p=0.5 (no scipy)."""
-    if n == 0:
-        return float("nan")
-    dist = [math.comb(n, i) for i in range(n + 1)]
-    tot = float(2 ** n)
-    pk = dist[k]
-    return min(1.0, sum(c for c in dist if c <= pk) / tot)
-
-
-def _mcnemar_p(hit_a, hit_b):
-    """Exact McNemar via binomial on discordant pairs. Inputs: aligned
-    boolean lists."""
-    b = sum(1 for x, y in zip(hit_a, hit_b) if x and not y)
-    c = sum(1 for x, y in zip(hit_a, hit_b) if not x and y)
-    return b, c, _binom_p(min(b, c), b + c)
-
-
-def _latest_events(recs):
-    """Last pass-1 record per ffid (retries append; last wins)."""
-    out = {}
-    for r in recs:
-        if r.get("run_pass") == 1:
-            out[r["ffid"]] = r
-    return list(out.values())
-
-
-def _fmt_rate(hits, n, label=""):
-    if n == 0:
-        return f"{label:<24} n=0"
-    p = _binom_p(hits, n)
-    return (f"{label:<24} {hits:>4}/{n:<4} = {hits / n * 100:5.1f}%  "
-            f"(binom p={p:.3f})")
-
-
-def report_events():
-    recs = _latest_events(_load_jsonl(EVENTS_JSONL))
-    if not recs:
-        print("no event replay records — run `events` first")
-        return
-    bars = pd.read_parquet(BARS)
-    closes = bars["Close"].to_numpy()
-    close_ns = (bars.index + pd.Timedelta(minutes=15)).asi8
-
-    def anchor(t_ns, h_min):
-        i = np.searchsorted(close_ns, t_ns, side="right") - 1
-        if i < 0:
-            return None
-        stale_ns = max(45.0, h_min / 4.0) * 60e9
-        if t_ns - close_ns[i] > stale_ns:
-            return None
-        return closes[i]
-
-    rows, skip = [], {h: 0 for h, _ in HORIZONS}
-    base_skipped = 0
-    for r in recs:
-        t_ns = int(r["dateline"]) * 10 ** 9
-        base = anchor(t_ns, 0)
-        if base is None:
-            base_skipped += 1
-            continue
-        row = dict(r)
-        row.pop("request_body", None)
-        row.pop("response_body", None)
-        row["year"] = r["utc_time"][:4]
-        row["base"] = base
-        for hname, hmin in HORIZONS:
-            p = anchor(t_ns + hmin * 60 * 10 ** 9, hmin)
-            if p is None:
-                skip[hname] += 1
-                row[f"d_{hname}"] = np.nan
-            else:
-                row[f"d_{hname}"] = p - base
-        rows.append(row)
-    df = pd.DataFrame(rows)
-    df.to_csv(SCORED_CSV, index=False)
-
-    n_fail = int((df["llm_direction"].isna()).sum())
-    print("═" * 70)
-    print("PART A — event gold-direction vs actual XAU move")
-    print("═" * 70)
-    print(f"events replayed: {len(df)} (baseline-price skipped: "
-          f"{base_skipped}) | LLM hard failures: {n_fail}")
-    print("horizon skip counts (stale/closed):",
-          {k: v for k, v in skip.items() if v})
-    dist = df["llm_direction"].value_counts(dropna=False).to_dict()
-    print("LLM call distribution:", dist)
-
-    # Unconditional per-horizon noise scale (null): median |Δ| over all
-    # 2024+ bar times, position-shifted.
-    bars24 = bars[bars.index >= "2024-01-01"]
-    null_med = {}
-    for hname, hmin in HORIZONS:
-        k = hmin // 15
-        d = (bars24["Close"].shift(-k) - bars24["Close"]).abs().dropna()
-        null_med[hname] = float(d.median())
-    print("unconditional null median |Δ| ($):",
-          {k: round(v, 2) for k, v in null_med.items()})
-
-    # Strategy call columns: +1 gold up, −1 gold down, 0/NaN excluded.
-    df["c_llm"] = df["llm_direction"].map(
-        {"bullish": 1, "bearish": -1}).fillna(0)
-    df["c_doctrine"] = df["surprise"].map({"positive": -1, "negative": 1})
-    df["c_ffbw"] = df["better_worse"].map({1: -1, 2: 1}).fillna(0)
-    df["c_bull"] = 1
-    df["c_bear"] = -1
-
-    strategies = [("LLM", "c_llm"), ("doctrine(surprise)", "c_doctrine"),
-                  ("FF betterWorse", "c_ffbw"), ("always-bullish", "c_bull"),
-                  ("always-bearish", "c_bear")]
-
-    for hname, _ in HORIZONS:
-        dcol = f"d_{hname}"
-        sub = df[df[dcol].notna()]
-        print(f"\n── horizon {hname} (n scored bars={len(sub)}) ──")
-        zeros = int((sub[dcol] == 0).sum())
-        if zeros:
-            print(f"  Δ==0 count (scored as miss): {zeros}")
-        for label, col in strategies:
-            s = sub[sub[col] != 0]
-            hits = int((np.sign(s[dcol]) == s[col]).sum())
-            print("  " + _fmt_rate(hits, len(s), label))
-        # noise-filtered LLM hit rate
-        s = sub[(sub["c_llm"] != 0) &
-                (sub[dcol].abs() >= null_med[hname])]
-        hits = int((np.sign(s[dcol]) == s["c_llm"]).sum())
-        print("  " + _fmt_rate(hits, len(s),
-                               f"LLM |Δ|≥null-med"))
-        # event-window magnitude vs null
-        mag = sub[dcol].abs().median()
-        print(f"  event-window median |Δ| = ${mag:.2f} "
-              f"(null ${null_med[hname]:.2f})")
-        # paired McNemar: LLM vs each baseline on common directional set
-        for label, col in [("doctrine", "c_doctrine"),
-                           ("FF betterWorse", "c_ffbw"),
-                           ("always-bullish", "c_bull")]:
-            s = sub[(sub["c_llm"] != 0) & (sub[col] != 0)]
-            ha = list(np.sign(s[dcol]) == s["c_llm"])
-            hb = list(np.sign(s[dcol]) == s[col])
-            b, c, p = _mcnemar_p(ha, hb)
-            print(f"  McNemar LLM vs {label:<15} b={b:>3} c={c:>3} "
-                  f"p={p:.3f}")
-
-    # neutral-call flatness (15m)
-    neu = df[(df["llm_direction"] == "neutral") & df["d_15m"].notna()]
-    dire = df[(df["c_llm"] != 0) & df["d_15m"].notna()]
-    if len(neu):
-        print(f"\nneutral calls (n={len(neu)}): median |Δ_15m| "
-              f"${neu['d_15m'].abs().median():.2f} vs directional "
-              f"(n={len(dire)}) ${dire['d_15m'].abs().median():.2f} "
-              f"(null ${null_med['15m']:.2f})")
-
-    # breakdowns at 1h
-    dcol = "d_1h"
-    sub = df[df[dcol].notna() & (df["c_llm"] != 0)]
-    print("\n── LLM hit rate by event name (1h, top 10 by count) ──")
-    for name, grp in sorted(sub.groupby("name"), key=lambda kv: -len(kv[1]))[:10]:
-        hits = int((np.sign(grp[dcol]) == grp["c_llm"]).sum())
-        print("  " + _fmt_rate(hits, len(grp), name[:24]))
-    print("── LLM hit rate by year (1h) ──")
-    for yr, grp in sub.groupby("year"):
-        hits = int((np.sign(grp[dcol]) == grp["c_llm"]).sum())
-        print("  " + _fmt_rate(hits, len(grp), yr))
-
-    # cluster view: shared datelines
-    print("\n── same-timestamp clusters ──")
-    clus = df.groupby("dateline").filter(lambda g: len(g) > 1)
-    groups = list(clus.groupby("dateline"))
-    consistent = 0
-    net_hits, net_n = 0, 0
-    for _, g in groups:
-        signs = set(g.loc[g["c_llm"] != 0, "c_llm"])
-        if len(signs) <= 1:
-            consistent += 1
-        net = int(g["c_llm"].sum())
-        d = g["d_1h"].iloc[0]
-        if net != 0 and pd.notna(d) and d != 0:
-            net_n += 1
-            if np.sign(d) == np.sign(net):
-                net_hits += 1
-    print(f"clusters: {len(groups)} datelines "
-          f"({len(clus)} events, {len(clus) / max(len(df), 1) * 100:.0f}% of "
-          f"corpus) | internally consistent: {consistent}/{len(groups)}")
-    print("  " + _fmt_rate(net_hits, net_n, "cluster net-call (1h)"))
-
-    # determinism probe
-    p2 = {r["ffid"]: r for r in _load_jsonl(EVENTS_JSONL)
-          if r.get("run_pass") == 2 and r.get("llm_direction")}
-    if p2:
-        p1 = {r["ffid"]: r for r in recs}
-        agree = same_reason = n = 0
-        for fid, r2 in p2.items():
-            r1 = p1.get(fid)
-            if not r1 or not r1.get("llm_direction"):
-                continue
-            n += 1
-            if r1["llm_direction"] == r2["llm_direction"]:
-                agree += 1
-            if r1.get("reason_raw") == r2.get("reason_raw"):
-                same_reason += 1
-        print(f"\ndeterminism probe: direction agreement {agree}/{n}, "
-              f"identical reason text {same_reason}/{n}")
-
-    # stratified reason examples
-    print("\n── reason samples (1h hit/miss) ──")
-    sub = df[df[dcol].notna() & (df["c_llm"] != 0)].copy()
-    sub["hit"] = np.sign(sub[dcol]) == sub["c_llm"]
-    for verdict in (True, False):
-        picked, seen = [], set()
-        for _, r in sub[sub["hit"] == verdict].iterrows():
-            if r["name"] in seen:
-                continue
-            seen.add(r["name"])
-            picked.append(r)
-            if len(picked) == 10:
-                break
-        tag = "HIT " if verdict else "MISS"
-        for r in picked:
-            print(f"  [{tag}] {r['utc_time'][:16]} {r['name'][:30]:<30} "
-                  f"a={r['actual']} f={r['forecast']} → "
-                  f"{r['llm_direction']:<7} Δ1h={r[dcol]:+7.2f}  "
-                  f"“{(r['reason_raw'] or '')[:60]}”")
-    print(f"\nscored table → {SCORED_CSV}")
-
-
-def report_headlines():
-    recs = _load_jsonl(HEADLINES_JSONL)
-    if not recs:
-        print("\n(no headline replay records — run `headlines` first)")
-        return
-    g = _load_golden()
-    engine = NewsEngine(NewsConfig())
-
-    def collect(tag):
-        m = {}
-        for r in recs:
-            if r["run_tag"] == tag and r["scores"]:
-                for h, sc in zip(r["headlines"], r["scores"]):
-                    m[h] = tuple(sc)
-        return m
-
-    main, shuf = collect("main"), collect("shuffled")
-    print("\n" + "═" * 70)
-    print("PART B — golden headline set: LLM vs keyword baseline")
-    print("═" * 70)
-    missing = [h for h in g["headline"] if h not in main]
-    if missing:
-        print(f"WARNING: {len(missing)} headlines missing from main run")
-
-    def cls(x):
-        if x >= NEUTRAL_BAND:
-            return 1
-        if x <= -NEUTRAL_BAND:
-            return -1
-        return 0
-
-    rows = []
-    for _, r in g.iterrows():
-        h = r["headline"]
-        if h not in main:
-            continue
-        llm_xau = main[h][1]
-        kw = engine._analyze_headline_sentiment(h, for_currency="XAU")
-        rows.append({
-            "headline": h, "category": r["category"],
-            "exp": int(r["expected_xau_sign"]),
-            "llm_xau": llm_xau, "llm_cls": cls(llm_xau),
-            "kw": kw, "kw_cls": cls(kw),
-        })
-    t = pd.DataFrame(rows)
-    for who in ("llm", "kw"):
-        acc = (t[f"{who}_cls"] == t["exp"]).mean()
-        print(f"{who.upper():<4} 3-class accuracy (band ±{NEUTRAL_BAND}): "
-              f"{(t[f'{who}_cls'] == t['exp']).sum()}/{len(t)} "
-              f"= {acc * 100:.1f}%")
-    ha = list(t["llm_cls"] == t["exp"])
-    hb = list(t["kw_cls"] == t["exp"])
-    b, c, p = _mcnemar_p(ha, hb)
-    print(f"paired McNemar LLM vs keyword: b={b} c={c} p={p:.3f}")
-
-    print("\nper-category accuracy (LLM | keyword):")
-    for cat, grp in t.groupby("category"):
-        a1 = (grp["llm_cls"] == grp["exp"]).sum()
-        a2 = (grp["kw_cls"] == grp["exp"]).sum()
-        print(f"  {cat:<18} {a1:>2}/{len(grp):<2} | {a2:>2}/{len(grp)}")
-
-    print("\nLLM confusion (rows=expected, cols=predicted -1/0/+1):")
-    cm = pd.crosstab(t["exp"], t["llm_cls"]).reindex(
-        index=[-1, 0, 1], columns=[-1, 0, 1], fill_value=0)
-    print(cm.to_string())
-
-    print("\nLLM misclassified headlines:")
-    for _, r in t[t["llm_cls"] != t["exp"]].iterrows():
-        print(f"  exp {r['exp']:+d} got {r['llm_xau']:+.2f} "
-              f"[{r['category']}] {r['headline'][:70]}")
-
-    # repeat-batch determinism
-    m_b0 = [r for r in recs if r["run_tag"] == "main" and r["batch"] == 0]
-    r_b0 = [r for r in recs if r["run_tag"] == "repeat"]
-    if m_b0 and r_b0 and m_b0[0]["scores"] and r_b0[0]["scores"]:
-        a = np.array(m_b0[0]["scores"], dtype=float)
-        b_ = np.array(r_b0[0]["scores"], dtype=float)
-        exact = int((a == b_).all(axis=1).sum())
-        print(f"\nrepeat-batch: {exact}/{len(a)} score pairs identical, "
-              f"max |diff| = {np.abs(a - b_).max():.3f}")
-
-    # shuffled-order drift
-    if shuf:
-        both = [h for h in t["headline"] if h in shuf]
-        drift = [abs(main[h][1] - shuf[h][1]) for h in both]
-        if drift:
-            print(f"shuffled-order xau drift over {len(both)} headlines: "
-                  f"mean {np.mean(drift):.3f}, max {np.max(drift):.3f}")
-
-
-def cmd_report(args):
-    report_events()
-    report_headlines()
-
-
-# ── CLI ──────────────────────────────────────────────────────────────
-
-def main():
-    ap = argparse.ArgumentParser(description=__doc__.split("\n")[1])
-    sub = ap.add_subparsers(dest="cmd", required=True)
-
-    e = sub.add_parser("events", help="replay release surprises")
-    e.add_argument("--limit", type=int, default=0,
-                   help="smoke-test on first N events")
-    e.add_argument("--determinism", type=int, default=0,
-                   help="re-call N stratified events a second time")
-    e.add_argument("--sleep", type=float, default=0.3)
-    e.add_argument("--retry-passes", type=int, default=3)
-    e.add_argument("--timeout", type=float, default=30.0)
-    e.set_defaults(fn=cmd_events)
-
-    h = sub.add_parser("headlines", help="golden-set probes")
-    h.add_argument("--sleep", type=float, default=0.3)
-    h.add_argument("--timeout", type=float, default=30.0)
-    h.set_defaults(fn=cmd_headlines)
-
-    r = sub.add_parser("report", help="offline scoring report")
-    r.set_defaults(fn=cmd_report)
-
-    args = ap.parse_args()
-    args.fn(args)
-
-
-if __name__ == "__main__":
-    main()

backtest/research/llm_news_eval_2026-08-06.md

@@ -1,86 +0,0 @@
-# DeepSeek news-analysis validation — 2026-08-06
-
-**Verdict: headline sentiment scoring is excellent; event-direction calls are
-directionally sound but ~11% of them misread the numeric comparison, and the
-failure concentrates on Unemployment Claims / close K-M-suffixed values.**
-
-Method (owner-directed, token-lean): independent expert re-analysis of a
-stratified sample of DeepSeek's recorded answers, not a full statistical
-backtest. Harness: `backtest/research/llm_news_eval.py` (replays releases
-through the REAL `_llm_event_direction` / `_llm_headline_scores` — real
-prompts, payloads, validation; raw wire traffic logged). 135 of the 586-event
-corpus were replayed before the full run was stopped by owner request
-(artifacts retained in `llm_news_eval_2026-08/`, resume-safe).
-
-## Part A — event gold-direction (28-event stratified sample, Jan–Jun 2024)
-
-Reviewer agreement with DeepSeek: **24/28 (86%)**. All 4 disagreements are
-DeepSeek errors, in exactly two failure modes:
-
-1. **Numeric comparison misread (3/28 ≈ 11%)** — the doctrine logic is right
-   but the premise is backwards:
-   - JOLTS 8.79M vs f 8.84M (a miss) → DS: "Job openings **above** forecast" → bearish (should be bullish)
-   - Claims 202K vs f 217K (fewer claims) → DS: "Claims **higher** than forecast" → bullish (should be bearish)
-   - Claims 202K vs f 209K — same backwards read, same wrong call
-   Both Unemployment Claims events in the sample were misread the same way —
-   looks systematic on that event type, possibly an anchoring prior that
-   "claims rose". All misreads involve close values with K/M suffixes.
-2. **Direction field contradicts its own reason (1/28)** — Empire State
-   −43.7 vs f −4.9: reason text says "gold may rise", `gold_direction` says
-   `bearish`.
-
-Where the numbers were read correctly, hawkish/dovish→gold mapping was
-consistently correct (CPI/PPI/NFP/ADP/GDP/PMI/wages/unemployment-rate all
-handled properly, including inverted metrics when read right).
-
-Price cross-check (1h horizon, cluster-confounded, indicative only):
-reviewer calls 22/28 with the subsequent move, DeepSeek 20/28.
-
-## Part B — headline sentiment (78-headline golden set)
-
-DeepSeek matched the reviewer's expected gold-direction label on
-**78/78 (100%)** — including all 8 negation traps ("signals no rate hike",
-"no rate cuts until…") and all 6 inverted-metric phrasings ("claims hit
-two-year high"). The keyword scorer it overrides got **24/78 (31%)**.
-Repeat-batch and shuffled-order probes recorded (headlines_replay.jsonl).
-
-## Implications
-
-- The headline-sentiment override (the main live LLM surface) is validated —
-  a large, measured upgrade over keywords.
-- The event-direction enrichment ("— gold bullish/bearish" appended to
-  surprise descriptions) is right ~85–90% of the time but can state a
-  factually backwards comparison in the displayed reason. It is display-only
-  (never gates signals), so exposure is a possibly-misleading Telegram/CLI
-  line, not trades.
-- **Hardening SHIPPED same day (owner-approved):** the payload now carries a
-  code-computed `vs_forecast` field ("above/below/in line with forecast",
-  from `EconomicEvent.surprise`) and the prompt tells the model to trust it.
-  First iteration fixed the arithmetic but exposed a masked doctrine error
-  (model read "claims below forecast" correctly, then called it a *weaker*
-  labor market — and regressed the unemployment-rate case the same way), so
-  the prompt's inverted-metrics line was expanded to an explicit rule plus a
-  worked claims example. Verification: **18/18** — all 4 original misreads
-  fixed, 11/11 no-regression cases unchanged, 3/3 fresh probes on the side
-  the prompt example does NOT cover (claims 245K vs f 220K → bullish,
-  unemployment 4.1% vs 3.9% → bullish, JOLTS 9.10M vs 8.80M → bearish).
-  Suite green at 346 tests, incl. a new payload assertion in Module 19.
-- Production observations found while building the harness (not fixed):
-  `_llm_event_cache` key omits forecast → a repeated `actual` re-serves a
-  direction computed for a different forecast; a "neutral" reply is
-  indistinguishable from a failure in the return value.
-
-## Reproduce
-
-```
-venv/bin/python backtest/research/llm_news_eval.py events --limit 5   # smoke
-venv/bin/python backtest/research/llm_news_eval.py events --determinism 20
-venv/bin/python backtest/research/llm_news_eval.py headlines
-venv/bin/python backtest/research/llm_news_eval.py report             # offline
-```
-
-Corpus: 586 high-impact USD surprises 2024-01→2026-07 (calendar.parquet,
-NFP-verified UTC); prices: Oanda M15 mids. Full-run cost ≈ cents. The
-`report` subcommand (hit rates vs always-bullish/doctrine/FF-betterWorse
-baselines, McNemar, cluster view) is built and untested at scale — run it
-after completing the replay if a statistical answer is ever wanted.

backtest/research/llm_news_golden.csv

@@ -1,79 +0,0 @@
-headline,category,expected_xau_sign,rationale
-"Powell says restrictive policy needed for longer as inflation proves sticky",hawkish_fed,-1,"higher-for-longer rates lift real yields, gold-bearish"
-"Fed minutes show officials leaning toward two more rate hikes this year",hawkish_fed,-1,"more hikes = stronger dollar/yields"
-"Hot CPI print fuels bets on additional Fed tightening",hawkish_fed,-1,"tightening expectations weigh on gold"
-"Fed's Waller: higher-for-longer rates warranted by resilient economy",hawkish_fed,-1,"hawkish guidance"
-"FOMC raises rates 50bp, flags further increases ahead",hawkish_fed,-1,"outsized hike plus hawkish guidance"
-"Treasury yields spike as Fed pushes back on rate cut hopes",hawkish_fed,-1,"cut hopes dashed, yields up"
-"Core inflation accelerates, cementing case for hawkish Fed stance",hawkish_fed,-1,"hawkish repricing"
-"Fed dot plot shifts higher; markets price out easing for the year",hawkish_fed,-1,"easing priced out"
-"Fed cuts rates 25bp and signals more easing to come",dovish_fed,1,"easing cycle is gold-bullish"
-"Powell hints at rate cuts as inflation cools toward target",dovish_fed,1,"cut guidance"
-"FOMC minutes reveal growing support for policy easing",dovish_fed,1,"dovish minutes"
-"Fed's Williams says rate cuts appropriate later this year",dovish_fed,1,"cut guidance"
-"Soft inflation data boosts odds of a dovish Fed pivot",dovish_fed,1,"pivot odds up"
-"Fed pauses hikes, markets price aggressive easing cycle",dovish_fed,1,"easing priced in"
-"Central banks turn dovish as global growth falters",dovish_fed,1,"global easing backdrop"
-"Fed to restart bond purchases to support flagging economy",dovish_fed,1,"QE is gold-bullish"
-"Dollar index hits 12-month high on robust US economic data",strong_usd,-1,"strong dollar pressures gold"
-"US GDP grows 4.1% annualized, blowing past expectations",strong_usd,-1,"strong economy, hawkish implication"
-"Blockbuster jobs report sends dollar surging across the board",strong_usd,-1,"strong jobs lift dollar and yields"
-"US retail sales smash forecasts, underscoring consumer strength",strong_usd,-1,"strong data"
-"Greenback rallies as US data outshines Europe and Asia",strong_usd,-1,"dollar strength"
-"ISM services index jumps to two-year high on strong demand",strong_usd,-1,"strong data"
-"Consumer confidence surges to post-pandemic high in the US",strong_usd,-1,"strong data"
-"Dollar tumbles to eight-month low after dismal manufacturing data",weak_usd,1,"weak dollar lifts gold"
-"US economy adds far fewer jobs than expected; dollar slides",weak_usd,1,"weak jobs, dovish implication"
-"Greenback weakens as US retail sales unexpectedly contract",weak_usd,1,"weak data"
-"US GDP growth revised down sharply, stoking slowdown fears",weak_usd,1,"slowdown is dovish"
-"Dollar slumps as trade deficit balloons to record",weak_usd,1,"dollar weakness"
-"Soft services PMI adds to evidence of cooling US economy",weak_usd,1,"cooling economy"
-"US consumer sentiment plunges to two-year low",weak_usd,1,"weak data"
-"Real yields climb to multi-year highs as TIPS sell off",real_yields,-1,"rising real yields raise gold's opportunity cost"
-"10-year Treasury yield surges past 5% on supply glut",real_yields,-1,"yield spike"
-"Rising real interest rates dim appeal of non-yielding assets",real_yields,-1,"explicit opportunity-cost story"
-"Treasury yields sink as investors flock to bonds",real_yields,1,"falling yields support gold"
-"Real yields turn negative as inflation outpaces rates",real_yields,1,"negative real yields are gold's best backdrop"
-"Bond rally drives 10-year yield to lowest level in a year",real_yields,1,"falling yields"
-"Geopolitical tensions escalate in Middle East, investors seek safety",safe_haven,1,"safe-haven bid"
-"Stocks crater worldwide as banking crisis fears spread",safe_haven,1,"crisis flight to safety"
-"Escalating conflict drives flight to safe-haven assets",safe_haven,1,"explicit haven flow"
-"Investors dump risk assets after surprise sovereign default",safe_haven,1,"default shock"
-"Military escalation on NATO border rattles global markets",safe_haven,1,"geopolitical stress"
-"Wall Street plunges 4% in worst session since pandemic",safe_haven,1,"risk-off"
-"Currency crisis in emerging markets sparks contagion fears",safe_haven,1,"contagion, haven bid"
-"Nuclear rhetoric intensifies, sending investors to havens",safe_haven,1,"extreme geopolitical stress"
-"Ceasefire agreement lifts global markets, haven demand fades",risk_on,-1,"haven unwind"
-"Stocks roar to record highs as trade deal signed",risk_on,-1,"risk-on drains haven demand"
-"Risk appetite returns as banking fears subside",risk_on,-1,"risk-on"
-"Global equities rally on strong earnings and easing tensions",risk_on,-1,"risk-on"
-"Gold hits record high as central banks keep buying bullion",gold_direct,1,"direct demand"
-"Gold ETFs post biggest weekly inflows in three years",gold_direct,1,"investment inflows"
-"China's central bank extends gold buying spree to 18th month",gold_direct,1,"official-sector demand"
-"Miners struggle to meet gold demand as supply deficit widens",gold_direct,1,"supply deficit"
-"Gold slides 2% as investors book profits after record run",gold_direct,-1,"profit-taking pressure"
-"Gold tumbles below key support as ETF outflows accelerate",gold_direct,-1,"outflows"
-"Bullion demand weakens as Indian jewellery buying stalls",gold_direct,-1,"physical demand soft"
-"Speculators slash bullish gold bets as dollar strengthens",gold_direct,-1,"positioning unwind"
-"Fed holds rates steady, signals no rate hike on the horizon",negation_traps,1,"dovish hold; keyword layer sees the phrase rate hike and flips it gold-bearish"
-"Powell says no rate cuts until inflation shows sustained cooling",negation_traps,-1,"hawkish hold; keyword layer sees rate cut and calls it gold-bullish"
-"Recession fears fade as US data streak beats estimates",negation_traps,-1,"fading fear is risk-on; keywords see recession plus beats and cancel out"
-"Inflation fails to accelerate, easing pressure on the Fed",negation_traps,1,"disinflation opens door to cuts"
-"Fed's Bowman: not yet confident inflation is beaten, cuts premature",negation_traps,-1,"hawkish pushback phrased around negations"
-"Central bank stops short of hiking, but warns tightening may resume",negation_traps,-1,"hawkish hold with tightening bias"
-"No safe-haven bid for gold despite equity wobble, analysts say",negation_traps,-1,"explicitly negated haven demand"
-"Jobs report not as weak as feared, dollar recovers",negation_traps,-1,"less-bad data is dollar-positive"
-"US manufacturing output plunges as new orders dry up",inverted_metrics,1,"weak economy is dovish/gold-bullish; keyword layer reads plunges as bearish"
-"US factory activity slumps to three-year low",inverted_metrics,1,"weak data; keyword reads slumps as bearish"
-"Consumer spending falls for third straight month",inverted_metrics,1,"weak data; keyword reads falls as bearish"
-"Private payrolls growth slides to weakest pace since 2020",inverted_metrics,1,"weak jobs; keyword reads slides as bearish"
-"Jobless claims hit two-year high as layoffs spread",inverted_metrics,1,"higher claims = weaker economy = gold-bullish"
-"US unemployment rate hits 4.5%, highest since 2021",inverted_metrics,1,"higher unemployment = dovish = gold-bullish"
-"Lakers clinch playoff berth with overtime win over Suns",distractors,0,"irrelevant"
-"New Marvel film smashes box office records on opening weekend",distractors,0,"irrelevant"
-"Apple unveils thinner MacBook with faster chip lineup",distractors,0,"irrelevant"
-"Scientists discover new deep-sea species off Pacific coast",distractors,0,"irrelevant"
-"Heavy rain forecast for the Midwest through the weekend",distractors,0,"irrelevant"
-"Streaming service raises subscription prices in Europe",distractors,0,"irrelevant"
-"Electric vehicle startup recalls 12,000 cars over software glitch",distractors,0,"irrelevant"
-"Local elections see record turnout in several US states",distractors,0,"irrelevant"

config.py

@@ -46,12 +46,6 @@ class TechnicalConfig:
     ema_medium: int = 21
     ema_slow: int = 50
     ema_trend: int = 200
-    # 4-EMA mode only: score partial ribbon alignment (2-3 of 4 checks)
-    # as NEUTRAL instead of a weak BUY/SELL. Research knob (2026-08-04
-    # indicator audit: partial-alignment trades averaged -0.008R vs
-    # +0.230R fully aligned on the M5 12mo Oanda book); default False =
-    # legacy behavior until a replay campaign validates flipping it.
-    ema_partial_neutral: bool = False
 
     rsi_period: int = 14
     rsi_overbought: float = 70.0

dashboard_publisher.py

@@ -1,451 +0,0 @@
-#!/usr/bin/env python3
-"""
-XAU/USD static dashboard publisher.
-
-Standalone daemon that scans XAU/USD at candle closes and atomically
-writes `snapshot.json` + `news.json` (plus a copy of
-`static/dashboard.html`) into an Apache-served directory. Apache does
-all web serving — this process never opens a port, so a failure here
-can only ever mean a stale page.
-
-Independence rules (owner requirement — the telegram bot must be
-immune to anything on the web side):
-  * Runs as its own process; never imported by, and never imports,
-    `telegram_bot` / `oanda_daemon` (telegram_bot resolves credentials
-    at import). The confirmation map and boundary math are therefore
-    duplicated here as local constants.
-  * The only shared-code dependency beyond the standard signal path is
-    the additive `NewsEngine.get_week_events` accessor, which no
-    production signal path calls.
-
-Scheduling: every M15/H1/H4 boundary is also an M5 boundary (all
-intervals are multiples of 5, anchored to UTC midnight), so one
-sequential loop sleeps to the next M5 boundary (+10 s buffer) and scans
-whichever timeframes just closed. Due-classification uses the RAW wake
-time — flooring 12:00:10 to the 5-minute grid yields the 12:00 boundary
-that just closed. Do NOT subtract the buffer first: 12:00:10 − 15 s
-floors to 11:55 and silently skips every H1/H4 scan (v1 bug, 2026-08-02).
-
-Scans run sequentially M5→M15→H1→H4 through one shared data provider so
-later timeframes hit the warm boundary cache — that ordering IS the
-duplicate-fetch dedup (providers have no in-flight dedup; Oanda has no
-throttle). Keep it sequential.
-
-Run:  python dashboard_publisher.py [--once] [--out DIR]
-Out dir precedence: --out → DASHBOARD_OUT_DIR env → /var/www/trade
-"""
-
-import argparse
-import json
-import hashlib
-import logging
-import math
-import numbers
-import os
-import shutil
-import sys
-import time
-from datetime import datetime, timezone
-from logging.handlers import TimedRotatingFileHandler
-from typing import Dict, List, Optional
-
-from config import PlatformConfig, Timeframe, XAUUSD
-from main import report_to_dict
-from news_engine import NewsEngine
-from pair_signals import produce_signal
-from providers import make_data_provider
-import providers.base as _providers_base
-
-logger = logging.getLogger("dashboard")
-
-DASHBOARD_TFS = [Timeframe.M5, Timeframe.M15, Timeframe.H1, Timeframe.H4]
-
-# Local copy of the bot's confirmation map (see independence rules above).
-CONFIRMATIONS = {
-    Timeframe.M5:  [Timeframe.M15, Timeframe.H1],
-    Timeframe.M15: [Timeframe.M5, Timeframe.H1],
-    Timeframe.H1:  [Timeframe.M15, Timeframe.H4],
-    Timeframe.H4:  [Timeframe.H1, Timeframe.D1],
-}
-
-TF_INTERVAL_MINUTES = {
-    Timeframe.M5: 5, Timeframe.M15: 15, Timeframe.H1: 60, Timeframe.H4: 240,
-}
-
-DEFAULT_OUT_DIR = "/var/www/trade"
-SCAN_BUFFER_SECONDS = 10
-SNAPSHOT_FILE = "snapshot.json"
-NEWS_FILE = "news.json"
-PAGE_FILE = "dashboard.html"
-SUMMARY_PHP_FILE = "news_summary.php"
-# Repo files copied into the out dir at daemon start (git pull + restart
-# updates them). The PHP endpoint reads news.json in place, so it lives
-# next to the page.
-STATIC_FILES = [PAGE_FILE, SUMMARY_PHP_FILE]
-
-
-# ══════════════════════════════════════════════════════════════════
-#  Pure layer (no network, fully covered by test_all.py)
-# ══════════════════════════════════════════════════════════════════
-
-def _safe_num(v):
-    """Coerce to a JSON-safe float; NaN/±inf become None."""
-    try:
-        f = float(v)
-    except (TypeError, ValueError):
-        return v
-    if math.isnan(f) or math.isinf(f):
-        return None
-    return f
-
-
-def _sanitize(obj):
-    """Recursively make a payload safe for json.dumps(allow_nan=False):
-    NaN/inf → None, numpy scalars → native ints/floats."""
-    if isinstance(obj, bool):
-        return obj
-    if isinstance(obj, dict):
-        return {k: _sanitize(v) for k, v in obj.items()}
-    if isinstance(obj, (list, tuple)):
-        return [_sanitize(v) for v in obj]
-    if isinstance(obj, numbers.Integral):
-        return int(obj)
-    if isinstance(obj, numbers.Real):
-        return _safe_num(obj)
-    return obj
-
-
-def analysis_to_dict(analysis) -> Optional[dict]:
-    """Serialize a TechnicalAnalysis. Iterates analysis.indicators only —
-    never vars()/asdict(), which would drag dynamic pandas attrs
-    (_rsi_series, _macd_histogram, …) into the payload. ATR (weight 0)
-    is surfaced separately as context, not in the directional grid."""
-    if analysis is None:
-        return None
-    indicators = {}
-    atr_context = None
-    for name, ind in analysis.indicators.items():
-        if name.startswith("_"):
-            continue
-        entry = {
-            "signal": ind.signal.name,
-            "value": _safe_num(ind.value),
-            "confidence": _safe_num(ind.confidence),
-            "weight": _safe_num(ind.weight),
-            "description": ind.description,
-        }
-        if name == "atr":
-            atr_context = entry
-        elif ind.weight != 0:
-            indicators[name] = entry
-    return {
-        "timeframe": analysis.timeframe,
-        "timestamp": str(analysis.timestamp),
-        "composite_signal": analysis.composite_signal.name,
-        "composite_score": _safe_num(analysis.composite_score),
-        "confidence": _safe_num(analysis.confidence),
-        "market_regime": analysis.market_regime,
-        "volatility_regime": analysis.volatility_regime,
-        "pattern_signals": list(analysis.pattern_signals),
-        "key_levels": {k: _safe_num(v) for k, v in analysis.key_levels.items()},
-        "indicators": indicators,
-        "atr_context": atr_context,
-    }
-
-
-def report_to_webapp_dict(report) -> dict:
-    """main.report_to_dict + the per-TF detail the dashboard needs."""
-    d = report_to_dict(report)
-    d["primary_signal"] = report.primary_signal.name
-    d["primary_score"] = _safe_num(report.primary_score)
-    d["mtf_scores"] = {k: _safe_num(v) for k, v in report.mtf_scores.items()}
-    d["exhaustion_count"] = report.exhaustion_count
-    d["primary_analysis"] = analysis_to_dict(report.primary_analysis)
-    d["confirmation_analyses"] = [
-        analysis_to_dict(a) for a in report.confirmation_analyses
-    ]
-    return _sanitize(d)
-
-
-def event_to_dict(ev) -> dict:
-    """EconomicEvent → dict. surprise/minutes_until/is_high_impact are
-    properties, so asdict() would skip them — added explicitly.
-    actual/forecast/previous stay strings as published ('228K', '0.3%')."""
-    return {
-        "time": ev.time.astimezone(timezone.utc).isoformat(),
-        "currency": ev.currency,
-        "impact": ev.impact,
-        "event": ev.event,
-        "forecast": ev.forecast,
-        "previous": ev.previous,
-        "actual": ev.actual,
-        "surprise": ev.surprise,
-        "minutes_until": _safe_num(ev.minutes_until),
-        "is_high_impact": ev.is_high_impact,
-    }
-
-
-def headline_to_dict(h) -> dict:
-    # id uses the same sha1(headline) scheme as NewsEngine._llm_sent_cache;
-    # it is the join key the PHP summary endpoint whitelists against.
-    return {
-        "id": hashlib.sha1(h.headline.encode("utf-8")).hexdigest(),
-        "headline": h.headline,
-        "source": h.source,
-        "link": h.link,
-        "description": h.description,
-        "publisher": h.publisher,
-        "timestamp": h.timestamp.astimezone(timezone.utc).isoformat()
-                     if h.timestamp else None,
-        "currencies": list(h.currencies_mentioned),
-        "score": _safe_num(h.sentiment_score),
-        "keywords": list(h.keywords),
-        "score_by_currency": {
-            k: _safe_num(v) for k, v in h.sentiment_by_currency.items()
-        },
-    }
-
-
-def build_news_payload(context, week_events, now: datetime) -> dict:
-    """news.json: full calendar week + scored headlines + LLM summary."""
-    headlines = sorted(
-        context.news_headlines,
-        key=lambda h: h.timestamp or now, reverse=True,
-    )
-    return _sanitize({
-        "generated_at": now.isoformat(),
-        "llm_summary": context.llm_summary,
-        "sentiment": dict(context.news_sentiment),
-        "high_impact_warning": context.high_impact_warning,
-        "warning_message": context.warning_message,
-        "calendar_available": context.calendar_available,
-        "events": [event_to_dict(e) for e in week_events],
-        "headlines": [headline_to_dict(h) for h in headlines],
-    })
-
-
-def due_timeframes(wake_dt: datetime) -> List[Timeframe]:
-    """Timeframes whose candle just closed at the boundary this wake
-    belongs to. Takes the RAW wake time (boundary + buffer): flooring to
-    the 5-minute grid yields the boundary that just passed. Never
-    subtract the buffer first — see module docstring (v1 bug)."""
-    minutes = wake_dt.hour * 60 + wake_dt.minute
-    floored = minutes - (minutes % 5)
-    return [tf for tf in DASHBOARD_TFS if floored % TF_INTERVAL_MINUTES[tf] == 0]
-
-
-def seconds_until_boundary(interval_minutes: int, buffer: int = 10) -> float:
-    # Same math as telegram_bot.py / oanda_daemon.py — kept local so the
-    # dashboard never imports bot modules (independence rules above).
-    now = datetime.now(timezone.utc)
-    total_minutes = now.hour * 60 + now.minute
-    elapsed_minutes = total_minutes % interval_minutes
-    elapsed_seconds = elapsed_minutes * 60 + now.second + now.microsecond / 1e6
-    interval_seconds = interval_minutes * 60
-    remaining = interval_seconds - elapsed_seconds + buffer
-    # NOTE: remaining > buffer always (elapsed < interval by construction);
-    # a call at the exact boundary instant targets the NEXT boundary —
-    # steady-state loops wake at boundary+buffer, so only a process
-    # started in the boundary microsecond skips one scan (2026-08-04
-    # audit I42; the old `remaining <= buffer` guard was unreachable).
-    return remaining
-
-
-def _empty_tf_entry() -> dict:
-    return {"status": "warming", "scanned_at": None, "scan_seconds": None,
-            "error": None, "error_at": None, "report": None}
-
-
-def build_snapshot_payload(snapshots: Dict[str, dict], now: datetime) -> dict:
-    """snapshot.json: one entry per dashboard TF. Timeframes not yet
-    scanned appear as status="warming" so the page can render a
-    placeholder instead of a hole. server_time_epoch lets the client
-    compute its clock offset once so countdowns don't drift."""
-    timeframes = {}
-    for tf in DASHBOARD_TFS:
-        interval = TF_INTERVAL_MINUTES[tf]
-        minutes = now.hour * 60 + now.minute
-        secs_into = (minutes % interval) * 60 + now.second + now.microsecond / 1e6
-        next_close = now.timestamp() + (interval * 60 - secs_into)
-        entry = _empty_tf_entry()
-        entry.update(snapshots.get(tf.value) or {})
-        entry["next_close_epoch"] = next_close
-        entry["next_scan_epoch"] = next_close + SCAN_BUFFER_SECONDS
-        timeframes[tf.value] = entry
-    return _sanitize({
-        "symbol": XAUUSD,
-        "server_time_epoch": now.timestamp(),
-        "market_closed": _providers_base._is_forex_weekend(now),
-        "timeframes": timeframes,
-    })
-
-
-def publish(obj: dict, path: str) -> None:
-    """Atomic JSON write: temp file + os.replace, so Apache never serves
-    a torn file. allow_nan=False — a NaN slipping through would render
-    the page blank (JSON.parse rejects bare NaN)."""
-    data = json.dumps(obj, allow_nan=False, separators=(",", ":"))
-    tmp = path + ".tmp"
-    with open(tmp, "w", encoding="utf-8") as fh:
-        fh.write(data)
-    os.replace(tmp, path)
-
-
-# ══════════════════════════════════════════════════════════════════
-#  Runtime layer (constructed only from main(); never at import)
-# ══════════════════════════════════════════════════════════════════
-
-def _setup_logging(level: str = "INFO") -> None:
-    log_dir = os.path.join(os.path.dirname(os.path.abspath(__file__)), "logs")
-    os.makedirs(log_dir, exist_ok=True)
-    log_file = os.path.join(log_dir, "dashboard.log")
-
-    fmt = logging.Formatter(
-        "%(asctime)s [%(levelname)s] %(name)s: %(message)s",
-        datefmt="%Y-%m-%d %H:%M:%S")
-
-    file_handler = TimedRotatingFileHandler(
-        log_file, when="h", interval=3, backupCount=240, utc=True)
-    file_handler.setFormatter(fmt)
-
-    console_handler = logging.StreamHandler(sys.stdout)
-    console_handler.setFormatter(fmt)
-
-    logging.basicConfig(
-        level=getattr(logging, level.upper(), logging.INFO),
-        handlers=[file_handler, console_handler],
-        force=True)
-
-
-class DashboardPublisher:
-    def __init__(self, out_dir: str):
-        self.out_dir = out_dir
-        self.snapshots: Dict[str, dict] = {}
-        self.news_payload: Optional[dict] = None
-        self._dp = None
-        self._news_engine: Optional[NewsEngine] = None
-
-    def _provider(self):
-        if self._dp is None:
-            self._dp = make_data_provider(PlatformConfig())
-        return self._dp
-
-    def _news(self) -> NewsEngine:
-        if self._news_engine is None:
-            self._news_engine = NewsEngine(PlatformConfig().news)
-        return self._news_engine
-
-    def install_static(self) -> None:
-        """Copy the repo's static files (page + PHP summary endpoint)
-        into the out dir (atomic), so git pull + restart also updates
-        them. The ai/ cache dir is NOT created here — the daemon runs as
-        a different user than Apache; that dir is a one-time deploy step
-        (deploy/README-dashboard.md)."""
-        static_dir = os.path.join(
-            os.path.dirname(os.path.abspath(__file__)), "static")
-        for name in STATIC_FILES:
-            src = os.path.join(static_dir, name)
-            if not os.path.exists(src):
-                logger.warning(f"{src} missing — not installed")
-                continue
-            dst = os.path.join(self.out_dir, name)
-            shutil.copyfile(src, dst + ".tmp")
-            os.replace(dst + ".tmp", dst)
-
-    def scan_tf(self, tf: Timeframe) -> None:
-        started = time.time()
-        try:
-            report = produce_signal(XAUUSD, tf, self._provider(),
-                                    CONFIRMATIONS[tf])
-            self.snapshots[tf.value] = {
-                "status": "ok",
-                "scanned_at": datetime.now(timezone.utc).isoformat(),
-                "scan_seconds": round(time.time() - started, 2),
-                "error": None,
-                "error_at": None,
-                "report": report_to_webapp_dict(report),
-            }
-            logger.info(f"{tf.value}: {report.final_signal.name} "
-                        f"q={report.signal_quality or '-'} "
-                        f"({time.time() - started:.1f}s)")
-        except Exception as e:
-            # Keep the last good report; the page shows it with an
-            # error badge instead of going blank.
-            logger.exception(f"{tf.value} scan failed")
-            entry = self.snapshots.get(tf.value) or _empty_tf_entry()
-            entry["status"] = "error"
-            entry["error"] = f"{type(e).__name__}: {e}"
-            entry["error_at"] = datetime.now(timezone.utc).isoformat()
-            self.snapshots[tf.value] = entry
-
-    def refresh_news(self) -> None:
-        try:
-            ne = self._news()
-            context = ne.get_fundamental_context()
-            week = ne.get_week_events()
-            self.news_payload = build_news_payload(
-                context, week, datetime.now(timezone.utc))
-        except Exception:
-            logger.exception("news refresh failed — keeping last payload")
-
-    def publish_snapshot(self) -> None:
-        publish(build_snapshot_payload(
-            self.snapshots, datetime.now(timezone.utc)),
-            os.path.join(self.out_dir, SNAPSHOT_FILE))
-
-    def publish_news(self) -> None:
-        if self.news_payload is not None:
-            publish(self.news_payload, os.path.join(self.out_dir, NEWS_FILE))
-
-    def publish_all(self) -> None:
-        self.publish_snapshot()
-        self.publish_news()
-
-    def run_cycle(self, tfs: List[Timeframe]) -> None:
-        for tf in tfs:  # sequential — see module docstring (cache dedup)
-            self.scan_tf(tf)
-        # Snapshot lands BEFORE the news refresh: when the news TTL has
-        # expired, refresh_news does FF + 4 RSS fetches (and possibly
-        # DeepSeek calls) — 20-30 s that must not keep fresh signals off
-        # the page.
-        self.publish_snapshot()
-        self.refresh_news()
-        self.publish_news()
-
-    def run_forever(self) -> None:
-        self.run_cycle(DASHBOARD_TFS)  # populate the page at boot
-        while True:
-            delay = seconds_until_boundary(5, buffer=SCAN_BUFFER_SECONDS)
-            logger.info(f"next M5 boundary in {delay:.0f}s")
-            time.sleep(delay)
-            try:
-                due = due_timeframes(datetime.now(timezone.utc))
-                logger.info(f"boundary wake — scanning {[t.value for t in due]}")
-                self.run_cycle(due)
-            except Exception:
-                logger.exception("cycle failed")
-
-
-def main(argv=None) -> None:
-    ap = argparse.ArgumentParser(description="XAU/USD static dashboard publisher")
-    ap.add_argument("--once", action="store_true",
-                    help="one scan+publish cycle, then exit")
-    ap.add_argument("--out",
-                    default=os.environ.get("DASHBOARD_OUT_DIR", DEFAULT_OUT_DIR),
-                    help="output directory served by Apache "
-                         f"(default: $DASHBOARD_OUT_DIR or {DEFAULT_OUT_DIR})")
-    args = ap.parse_args(argv)
-
-    _setup_logging()
-    os.makedirs(args.out, exist_ok=True)
-    pub = DashboardPublisher(args.out)
-    pub.install_static()
-    if args.once:
-        pub.run_cycle(DASHBOARD_TFS)
-        print(f"Published to {args.out}")
-        return
-    pub.run_forever()
-
-
-if __name__ == "__main__":
-    main()

deploy/README-dashboard.md

@@ -1,157 +0,0 @@
-# Deploying the web dashboard (live LAMP server)
-
-The dashboard is a **static publisher**: `dashboard_publisher.py` runs as its
-own daemon, scans XAU/USD at candle closes, and atomically writes
-`dashboard.html` + `news_summary.php` + `snapshot.json` + `news.json` into a
-folder Apache already serves. Apache does all web serving — no new port, no
-Python web server, and the telegram bot process is completely unaffected by
-anything that happens here (see the module docstring for the independence
-rules). The one dynamic piece is `news_summary.php` (§6): an on-demand
-DeepSeek summary for a clicked headline, whitelist-gated against news.json.
-
-All steps below run **on the live server** (`/home/Forex/ForexBot2`).
-
-## 1. Get the code
-
-```bash
-cd /home/Forex/ForexBot2
-git pull
-venv/bin/python test_all.py        # optional sanity: suite must be green
-```
-
-No new Python dependencies are required.
-
-## 2. Make the web folder writable
-
-The dashboard publishes straight into the portal docroot `/var/www/trade/`
-(owner decision 2026-08-03) — it adds exactly four files there:
-`dashboard.html`, `news_summary.php`, `snapshot.json`, `news.json` (plus the
-apache-owned `ai/` cache dir from §6). The daemon user must be able to write
-them:
-
-```bash
-# if the portal files can be owned by the daemon user:
-sudo chown Forex: /var/www/trade
-# otherwise grant write without changing ownership:
-sudo setfacl -m u:Forex:rwx /var/www/trade
-```
-
-(A different path is one edit away: the unit's `Environment=DASHBOARD_OUT_DIR=`
-line, or `--out` for one-off runs.)
-
-SELinux (RHEL/Alma/CentOS enforcing): directories under `/var/www` normally
-already carry `httpd_sys_content_t`, and new files inherit the parent label, so
-no action is usually needed. If Apache 403s despite correct permissions:
-
-```bash
-sudo restorecon -R /var/www/trade
-```
-
-## 3. Smoke-test once
-
-```bash
-venv/bin/python dashboard_publisher.py --once
-ls /var/www/trade    # dashboard.html snapshot.json news.json added next to the portal
-```
-
-Then open `http://<server>/dashboard.html` (path relative to however the
-vhost maps `/var/www/trade`) — all four timeframe cards, the calendar, and
-the newswire should populate.
-
-## 4. Install the daemon
-
-```bash
-sudo cp deploy/gold-dashboard.service /etc/systemd/system/
-# edit User= if the repo owner is not "Forex"
-sudo systemctl daemon-reload
-sudo systemctl enable --now gold-dashboard
-systemctl status gold-dashboard
-tail -f logs/dashboard.log
-```
-
-The daemon scans all four timeframes at boot, then wakes at every M5
-boundary (+10 s) and rescans whichever timeframes just closed. Logs rotate
-every 3 h (240 backups), same scheme as the bot.
-
-## 5. Optional Apache polish
-
-The page polls its JSON with conditional GETs (`cache: "no-cache"` → Apache
-answers unchanged files with 304), so no Apache config is strictly needed. To
-be explicit anyway:
-
-```apache
-<Directory "/var/www/trade">
-    <FilesMatch "\.json$">
-        Header set Cache-Control "no-cache"
-    </FilesMatch>
-</Directory>
-```
-
-The page is deliberately public (owner decision 2026-08-03). To restrict it
-later, add standard Apache basic auth (`htpasswd` + `Require valid-user`) on
-that directory — no code changes needed.
-
-## 6. AI summary endpoint (PHP) — one-time setup
-
-Clicking a newswire headline opens a detail dialog that fetches an on-demand
-DeepSeek gold-impact summary from `news_summary.php` (installed next to the
-page by the daemon). Owner decision 2026-08-07: a real on-demand endpoint on
-the public page, accepted with these guards baked into the PHP — the id must
-match a headline in the published news.json (arbitrary text can never reach
-the LLM), each headline's summary is cached forever in `ai/` (≤1 API call per
-headline, ever), and a file-based daily cap fails **closed** on I/O errors.
-
-Requirements: `php` + the curl extension (`php -m | grep curl`).
-
-```bash
-# 1. SELinux: allow httpd outbound HTTPS — WITHOUT this every summary call
-#    dies as a curl timeout even with perfect perms (RHEL-family default).
-sudo setsebool -P httpd_can_network_connect 1
-
-# 2. API key OUTSIDE the docroot, readable by Apache only:
-sudo install -d -o root -g root -m 755 /etc/gold-dashboard
-sudo sh -c 'umask 137; echo "sk-..." > /etc/gold-dashboard/deepseek.key'
-sudo chown root:apache /etc/gold-dashboard/deepseek.key   # mode 640
-
-# 3. apache-owned cache dir (the daemon runs as Forex and can't create it):
-sudo install -d -o apache -g apache -m 755 /var/www/trade/ai
-sudo semanage fcontext -a -t httpd_sys_rw_content_t '/var/www/trade/ai(/.*)?'
-sudo restorecon -Rv /var/www/trade/ai
-```
-
-Knobs (Apache `SetEnv` in the vhost, both optional):
-`AI_SUMMARY_DAILY_CAP` (default 150), `DEEPSEEK_KEY_FILE` (default
-`/etc/gold-dashboard/deepseek.key`).
-
-Verify from the shell — expect 400 / 404 / 200 / 200-from-cache:
-
-```bash
-curl -s "http://<server>/news_summary.php?id=zzz"                 # {"error":"bad_id"}
-curl -s "http://<server>/news_summary.php?id=$(printf 'a%.0s' {1..40})"  # unknown_id
-ID=$(python3 -c "import json;print(json.load(open('/var/www/trade/news.json'))['headlines'][0]['id'])")
-curl -s "http://<server>/news_summary.php?id=$ID"   # real summary (2-12 s)
-curl -s "http://<server>/news_summary.php?id=$ID"   # instant, from ai/<id>.json
-```
-
-## Operational notes
-
-- **Updating**: `git pull` + `systemctl restart gold-dashboard`. The daemon
-  re-copies `static/dashboard.html` and `static/news_summary.php` into the
-  web folder at startup.
-- **If the daemon dies**, systemd restarts it in 30 s; worst case the page
-  shows stale data with its own "updated N min ago" evidence. Apache and the
-  telegram bot are unaffected.
-- **DeepSeek**: the publisher process has its own daily request cap
-  (`deepseek.daily_request_cap`, default 500) separate from the bot's —
-  expect roughly up to ~430 extra calls/day worst case, V4-Flash pricing.
-- **Two independent DeepSeek meters** hit the same API key: the publisher's
-  in-process cap above, and the PHP endpoint's file cap (`ai/.ht_cap.json`,
-  default 150/day). They don't communicate — worst-case spend is their sum;
-  DeepSeek-side spend alerting is the true backstop. Kept separate on
-  purpose (independence rule: the PHP never imports/reads bot code or
-  config.json).
-- **Restart window**: for the first minutes after a daemon restart the new
-  page can meet a pre-upgrade news.json without headline ids — those rows
-  simply render non-clickable until the first news publish.
-- **Credentials**: uses the same env/.env/config.json resolution as the rest
-  of the platform (Oanda candles feed; news needs no key; DeepSeek optional).

deploy/gold-dashboard.service

@@ -1,23 +0,0 @@
-# systemd unit for the XAU/USD static dashboard publisher.
-# Install (as root, on the live server):
-#   cp deploy/gold-dashboard.service /etc/systemd/system/
-#   systemctl daemon-reload && systemctl enable --now gold-dashboard
-# See deploy/README-dashboard.md for the full checklist.
-
-[Unit]
-Description=XAU/USD dashboard publisher (writes static JSON/HTML for Apache)
-After=network-online.target
-Wants=network-online.target
-
-[Service]
-Type=simple
-# Adjust to the account that owns /home/Forex/ForexBot2:
-User=Forex
-WorkingDirectory=/home/Forex/ForexBot2
-Environment=DASHBOARD_OUT_DIR=/var/www/trade
-ExecStart=/home/Forex/ForexBot2/venv/bin/python dashboard_publisher.py
-Restart=always
-RestartSec=30
-
-[Install]
-WantedBy=multi-user.target

docs/architecture_diagram.py

@@ -1,199 +0,0 @@
-"""ForexBot2 architecture diagram → docs/architecture.png.
-
-Regenerate after architecture changes:  venv/bin/python docs/architecture_diagram.py
-(matplotlib only — no graphviz on the dev box)."""
-import os
-import matplotlib
-matplotlib.use("Agg")
-import matplotlib.pyplot as plt
-from matplotlib.patches import FancyBboxPatch, FancyArrowPatch
-
-W, H = 100.0, 100.0
-fig, ax = plt.subplots(figsize=(16, 11.5))
-ax.set_xlim(0, W); ax.set_ylim(0, H); ax.axis("off")
-
-INK = "#263238"; MUT = "#607D8B"; ARR = "#455A64"; HAUL = "#7B8FA1"
-
-def box(x, y, w, h, title, lines, fc, ec, title_fs=8.2, body_fs=6.6, lw=1.3):
-    ax.add_patch(FancyBboxPatch((x, y), w, h,
-                 boxstyle="round,pad=0.35,rounding_size=0.9",
-                 facecolor=fc, edgecolor=ec, linewidth=lw, zorder=2))
-    cy = y + h - 1.9
-    ax.text(x + w / 2, cy, title, ha="center", va="center", fontsize=title_fs,
-            fontweight="bold", color=INK, zorder=3)
-    cy -= 2.3
-    for ln in lines:
-        ax.text(x + w / 2, cy, ln, ha="center", va="center", fontsize=body_fs,
-                color=INK, zorder=3)
-        cy -= 1.85
-
-def band(x, y, w, h, label, color):
-    ax.add_patch(FancyBboxPatch((x, y), w, h,
-                 boxstyle="round,pad=0.3,rounding_size=1.2",
-                 facecolor="none", edgecolor=color, linewidth=1.0,
-                 linestyle=(0, (4, 3)), zorder=1))
-    ax.text(x + 0.9, y + h - 1.4, label, ha="left", va="center",
-            fontsize=7.2, fontweight="bold", color=color, zorder=3)
-
-def arrow(p1, p2, color=ARR, lw=1.5, style="-|>", conn="arc3,rad=0.0",
-          label=None, lx=0, ly=0, ls="solid", fs=6.4, rot=0):
-    ax.add_patch(FancyArrowPatch(p1, p2, arrowstyle=style, mutation_scale=11,
-                 color=color, linewidth=lw, linestyle=ls,
-                 connectionstyle=conn, zorder=4))
-    if label:
-        mx, my = (p1[0] + p2[0]) / 2 + lx, (p1[1] + p2[1]) / 2 + ly
-        ax.text(mx, my, label, fontsize=fs, color=MUT, ha="center",
-                va="center", zorder=5, rotation=rot,
-                bbox=dict(boxstyle="round,pad=0.15", fc="white", ec="none", alpha=0.9))
-
-# ── title ────────────────────────────────────────────────────────────
-ax.text(2, 98.4, "ForexBot2 — XAU/USD Signal-Intelligence Platform",
-        fontsize=15, fontweight="bold", color=INK, ha="left")
-ax.text(2, 96.0, "one pipeline, four delivery surfaces · closed bars only · live == backtest parity",
-        fontsize=8.5, color=MUT, ha="left")
-
-EXT_FC, EXT_EC = "#ECEFF1", "#90A4AE"
-DAT_FC, DAT_EC = "#E3F2FD", "#5C9BD1"
-NWS_FC, NWS_EC = "#E0F2F1", "#3D9C8F"
-COR_FC, COR_EC = "#FFFDF5", "#C9A227"
-REP_FC, REP_EC = "#FFE9A8", "#B8860B"
-DEL_FC, DEL_EC = "#E8F5E9", "#5FAF63"
-BTS_FC, BTS_EC = "#EDE7F6", "#8E6BC6"
-
-# ── external services (top band) ─────────────────────────────────────
-band(1, 85.0, 98, 9.4, "EXTERNAL SERVICES", "#78909C")
-box(2.5, 86.2, 15.5, 6.4, "Oanda v20 API",
-    ["candles (live feed)", "practice orders"], EXT_FC, EXT_EC)
-box(19.5, 86.2, 15.5, 6.4, "Twelve Data · Tiingo",
-    ["registered fallbacks", "(Tiingo: tick stream)"], EXT_FC, EXT_EC)
-box(36.5, 86.2, 15.5, 6.4, "Forex Factory",
-    ["economic calendar", "MyFXBook RSS fallback"], EXT_FC, EXT_EC)
-box(53.5, 86.2, 12.5, 6.4, "News RSS ×4",
-    ["gold / macro", "headlines"], EXT_FC, EXT_EC)
-box(67.5, 86.2, 13.5, 6.4, "DeepSeek API",
-    ["V4-Flash LLM", "thinking: disabled"], EXT_FC, EXT_EC)
-box(82.5, 86.2, 15, 6.4, "Telegram API",
-    ["owner bot +", "subscriber channel"], EXT_FC, EXT_EC)
-
-# ── data & context layer ─────────────────────────────────────────────
-box(2.5, 69.8, 33, 11.0, "providers/  (BaseDataProvider)",
-    ["single-flight fetch · TTL+boundary cache",
-     "forming-candle DROP · validate/clean",
-     "paginated fetch_range · optional tick stream",
-     "active feed knob: data_providers.active = oanda"],
-    DAT_FC, DAT_EC)
-box(38.5, 69.8, 33, 11.0, "news_engine.py  +  llm_client.py",
-    ["calendar events + keyword sentiment (gold-inverse)",
-     "DeepSeek enrichment: batch headline scores,",
-     "event gold-direction, market blurb — display-only,",
-     "keyword fallback, daily request cap"],
-    NWS_FC, NWS_EC)
-box(74.5, 69.8, 23, 11.0, "app_config.py",
-    ["credentials & deployment:",
-     "env → config.json → refuse",
-     "(no hardcoded fallbacks)",
-     ".env loaded at import"],
-    "#FAFAFA", "#9E9E9E")
-
-# ── signal core container ────────────────────────────────────────────
-ax.add_patch(FancyBboxPatch((2.5, 40.5), 95, 26.5,
-             boxstyle="round,pad=0.35,rounding_size=1.2",
-             facecolor=COR_FC, edgecolor=COR_EC, linewidth=1.6, zorder=1))
-ax.text(50, 65.2, "SIGNAL CORE — runs identically in live scans and backtests (closed bars only)",
-        ha="center", fontsize=8.4, fontweight="bold", color="#8a6d1a", zorder=3)
-
-box(4.5, 51.8, 28, 12.0, "engine.py — TechnicalEngine",
-    ["9 scored indicators + ATR context:",
-     "EMA · RSI · MACD · BB · ADX · ROC ·",
-     "Keltner · divergence · candlestick",
-     "regime → adaptive weights → composite"],
-    "white", COR_EC)
-box(4.5, 42.0, 28, 8.4, "pair_signals/xauusd.py — factory",
-    ["ALWAYS used: config.json overlay +",
-     "validated per-TF deltas (hour blocks, trail…)"],
-    "white", COR_EC)
-box(35.5, 42.0, 60, 20.7, "signal_aggregator.py — analyze_pair → analyze_from_dataframes",
-    ["multi-TF agreement  (M5→[M15,H1] · M15→[M5,H1] · H1→[M15,H4] · H4→[H1,D1])",
-     "→ news context (warn-only by default) → diminishing-decay penalties",
-     "→ exhaustion hard block → quality grade A+…F",
-     "→ ordered post-grade gate chain (session · XAU SELL vol/hour · regime · M5 noise)",
-     "→ setup generators: XAU ATR (A+/A) · generic swing · XAU_PB · XAU_CP",
-     "→ entry · SL · TP1/TP2 · trailing plan"],
-    "white", COR_EC, body_fs=6.9)
-
-# ── SignalReport hub ─────────────────────────────────────────────────
-box(31, 33.2, 38, 5.2, "SignalReport",
-    ["signal (−2…+2) · grade · trade setup · reasons · news summary"],
-    REP_FC, REP_EC, title_fs=8.8)
-
-# ── delivery surfaces ────────────────────────────────────────────────
-band(1, 13.2, 98, 17.2, "DELIVERY SURFACES", "#4C8F50")
-box(2.5, 15.0, 12, 12.6, "main.py",
-    ["CLI report", "--detail", "--json"], DEL_FC, DEL_EC)
-box(16, 15.0, 24, 12.6, "oanda_daemon.py + executor",
-    ["practice auto-execution",
-     "book-parity rules: no broker TP,",
-     "5-bar cooldown, once-per-bar",
-     "trail, OPEN_ONLY fills,",
-     "boot reconcile · orders → Oanda"],
-    DEL_FC, DEL_EC, title_fs=7.8)
-box(42.5, 15.0, 31, 12.6, "dashboard_publisher.py  →  Apache",
-    ["independent daemon · scans at candle closes",
-     "atomic publish to /var/www/trade:",
-     "dashboard.html · snapshot.json · news.json",
-     "+ news_summary.php — on-demand AI",
-     "gold-impact summary (whitelist · cache · cap)"],
-    DEL_FC, DEL_EC)
-box(76, 15.0, 21.5, 12.6, "telegram_bot.py",
-    ["candle-boundary auto-scans", "quality threshold + dedup",
-     "alerts + LLM context line", "WS level-breach monitor"],
-    DEL_FC, DEL_EC)
-
-# ── backtest / QA ────────────────────────────────────────────────────
-box(2.5, 3.2, 71.8, 8.0, "backtest/gpu_backtest.py — bar-by-bar replay through the SAME analyze_from_dataframes",
-    ["realism (results-critical): next-open entries · raw-path resolution · spread charged · SL-first ties · close-only trail",
-     "CuPy GPU / NumPy fallback · parquet cache · fed by providers.fetch_range (news off)"],
-    BTS_FC, BTS_EC)
-box(77.5, 3.2, 20, 8.0, "test_all.py",
-    ["359 tests · 22 modules", "no network (mocked)"], "#FAFAFA", "#9E9E9E")
-
-# ── arrows: short structural hops (solid dark) ───────────────────────
-arrow((10, 86.2), (14, 81.2), label="OHLCV", lx=-3.4, ly=0.2)          # oanda → providers
-arrow((27, 86.2), (24, 81.2), ls=(0, (3, 2)), lw=1.1)                  # fallbacks → providers
-arrow((44, 86.2), (48, 81.2))                                          # FF → news
-arrow((59.5, 86.2), (56, 81.2))                                        # RSS → news
-arrow((72, 86.2), (62, 81.2), label="LLM", lx=3.4, ly=0.6)             # deepseek → news
-arrow((19, 69.8), (19, 64.2), label="DataFrames (UTC, closed bars)", lx=0.2, ly=0.9)
-arrow((33.5, 57.5), (35.5, 55.5))                                      # engine → aggregator
-arrow((32.5, 46.0), (35.5, 48.0))                                      # factory → aggregator
-arrow((55, 69.8), (55, 63.1), label="warn-only context", lx=0, ly=1.0)
-arrow((86, 69.8), (86, 67.3), ls=(0, (3, 2)), lw=1.1)                  # config → core
-arrow((50, 40.5), (50, 38.8))                                          # core → report
-arrow((36, 34.5), (8.5, 27.95), conn="arc3,rad=0.12")                  # report → cli
-arrow((40, 33.2), (28, 27.95), conn="arc3,rad=0.06")                   # report → daemon
-arrow((60, 33.2), (58, 27.95))                                         # report → dashboard
-arrow((66, 34.5), (86.5, 27.95), conn="arc3,rad=-0.12")                # report → telegram
-
-# ── long-haul edges (thin dashed, routed through gutters) ────────────
-# telegram bot ↔ Telegram API — right-margin arc, clear of app_config
-arrow((95, 86.2), (95, 27.95), conn="arc3,rad=0.15", color=HAUL,
-      lw=1.2, ls=(0, (4, 2)))
-ax.text(99.2, 57, "alerts / commands", fontsize=6.4, color=MUT,
-        rotation=90, ha="center", va="center")
-# browser click → news_summary.php → DeepSeek
-# (oanda_daemon's order flow is noted inside its box — no long arc)
-arrow((66, 27.95), (73, 86.2), conn="arc3,rad=-0.18", color=HAUL,
-      lw=1.2, ls=(0, (4, 2)),
-      label="on-demand AI summary\n(browser click → PHP)", lx=2.5, ly=11.6)
-# providers.fetch_range → backtester — far-left margin channel
-arrow((1.5, 69.8), (1.5, 11.0), color=BTS_EC, lw=1.1, ls=(0, (3, 2)),
-      label="fetch_range\n(history)", lx=3.6, ly=-1.2)
-# backtester replays the signal core — right gutter between boxes
-arrow((74.0, 11.6), (74.9, 40.3), color=HAUL, lw=1.3)
-ax.text(81.8, 38.9, "replays the exact signal code", fontsize=6.4,
-        color=MUT, ha="center", va="center", zorder=5,
-        bbox=dict(boxstyle="round,pad=0.15", fc="white", ec="none", alpha=0.9))
-
-out = os.path.join(os.path.dirname(os.path.abspath(__file__)), "architecture.png")
-fig.savefig(out, dpi=200, bbox_inches="tight", facecolor="white")
-print(f"saved {out}")

docs/user_guide.html

@@ -71,7 +71,7 @@
 <tr><td><b>Warnings ⚠️</b></td><td>Context, not veto. Backtesting shows warned trades perform about the same — strong momentum often triggers warnings.</td></tr>
 <tr><td><b>Market context 📰</b></td><td>An AI-written one-liner summarizing today's macro backdrop (headlines + high-impact releases) and its likely pull on gold. Informational only — it never blocks or grades a signal.</td></tr>
 <tr><td><b>SL / TP1 / RR</b></td><td>Stop loss sits behind the nearest swing level with an ATR buffer. TP1 is the nearest opposing level. <b>1R</b> = the entry-to-SL distance = the amount you risk.</td></tr>
-<tr><td><b>Trail plan 🔄</b></td><td>The real exit. Once price moves +0.6R in your favor, move the stop to trail 0.75R behind the best price and update it at each candle close. <b>TP1 is advisory — let winners run.</b></td></tr>
+<tr><td><b>Trail plan 🔄</b></td><td>The real exit. Once price moves +0.6R in your favor (+0.8R for pullback setups), move the stop to trail 0.75R behind the best price and update it at each candle close. <b>TP1 is advisory — let winners run.</b></td></tr>
 </table>
 
 <h2>3 · Acting on a signal — step by step</h2>
@@ -130,7 +130,6 @@
 <tr><td><code>/scan</code></td><td>Run an immediate analysis of the primary timeframe.</td></tr>
 <tr><td><code>/autoscan</code></td><td>Toggle automatic scanning (M5 / M15 / H1 / H4 at each candle close).</td></tr>
 <tr><td><code>/status</code></td><td>Uptime, auto-scan state, WebSocket price-monitor state.</td></tr>
-<tr><td><code>/news</code></td><td>Net XAU/USD news sentiment score (informational only — never gates signals).</td></tr>
 <tr><td><code>/cache</code> · <code>/clearcache</code></td><td>Inspect / reset the market-data cache.</td></tr>
 <tr><td><code>/help</code></td><td>Command reference.</td></tr>
 </table>

engine.py

@@ -245,12 +245,6 @@ class TechnicalEngine:
             signal = SignalStrength.NEUTRAL
             conf = 40
 
-        # Partial alignment (plain BUY/SELL, 2-3 of 4 checks) carries no
-        # edge on M5 — squash to NEUTRAL when the research knob is on.
-        if c.ema_partial_neutral and signal in (SignalStrength.BUY, SignalStrength.SELL):
-            signal = SignalStrength.NEUTRAL
-            conf = 40
-
         crossover_desc = ""
         if len(ema_fast) >= 3:
             prev_f, prev_m = ema_fast.iloc[-2], ema_medium.iloc[-2]

llm_client.py

@@ -24,8 +24,7 @@ import datetime as _dt
 import json
 import logging
 import threading
-import time
+from typing import Any, Dict, Optional
-from typing import Any, Dict, Optional, Tuple
 
 import requests
 
@@ -67,10 +66,6 @@ class DeepSeekClient:
         self._count_day: Optional[_dt.date] = None
         self._count = 0
         self._warned: set = set()
-        # Health stamps for status displays: last successful call and
-        # last failure (ts, kind). Read-only for consumers.
-        self.last_ok: float = 0.0
-        self.last_error: Optional[Tuple[float, str]] = None
 
     @property
     def available(self) -> bool:
@@ -130,11 +125,9 @@ class DeepSeekClient:
             resp = self._session.post(
                 DEEPSEEK_URL, json=body, timeout=self._timeout)
         except requests.RequestException as e:
-            self.last_error = (time.time(), "network")
             self._warn_once("network", "DeepSeek call failed: %s", e)
             return None
         if resp.status_code != 200:
-            self.last_error = (time.time(), f"http_{resp.status_code}")
             self._warn_once(
                 f"http_{resp.status_code}",
                 "DeepSeek HTTP %s: %s", resp.status_code, resp.text[:200])
@@ -144,11 +137,8 @@ class DeepSeekClient:
         except ValueError:
             parsed = None
         if parsed is None:
-            self.last_error = (time.time(), "parse")
             self._warn_once("parse", "DeepSeek response was not the "
                             "expected JSON object — ignoring")
-        else:
-            self.last_ok = time.time()
         return parsed
 
     @staticmethod

news_engine.py

@@ -9,7 +9,6 @@ A perfect technical setup can be destroyed by an unexpected NFP print.
 """
 
 import hashlib
-import html
 import logging
 import re
 import threading
@@ -58,22 +57,12 @@ Examples:
 
 _LLM_EVENT_SYSTEM = """\
 You are a forex/gold analyst. You receive one released economic event as
-JSON: {"event", "currency", "actual", "forecast", "previous",
+JSON: {"event", "currency", "actual", "forecast", "previous"}.
-"vs_forecast"}. "vs_forecast" states whether the actual print came in
+Decide the likely effect of this RELEASE (actual vs forecast) on the
-above or below the forecast; it is computed programmatically — trust it
+GOLD (XAU/USD) price. A stronger-than-forecast USD economy or a hawkish
-over your own comparison of the numbers. If it is "unknown", compare
+implication is usually gold-bearish; weaker data or a dovish implication
-actual and forecast yourself.
+is usually gold-bullish. Mind inverted metrics: higher unemployment or
-Decide the likely effect of this RELEASE on the GOLD (XAU/USD) price.
+jobless claims = weaker economy.
-A stronger-than-forecast USD economy or a hawkish implication is usually
-gold-bearish; weaker data or a dovish implication is usually
-gold-bullish.
-Inverted metrics — for unemployment rate and unemployment/jobless
-claims a LOWER number means a STRONGER labor market: below forecast =
-strong economy = usually gold-bearish; above forecast = weak economy =
-usually gold-bullish.
-Example: {"event": "Unemployment Claims", "actual": "202K", "forecast":
-"217K", "vs_forecast": "below forecast"} ->
-{"gold_direction": "bearish", "reason": "fewer claims show strong labor market"}
 Reply with ONLY JSON:
 {"gold_direction": "bullish"|"bearish"|"neutral", "reason": "<max 10 words>"}
 """
@@ -104,44 +93,6 @@ def _parse_release_value(raw: Optional[str]) -> Optional[float]:
     return float(match.group(1)) * multiplier
 
 
-_DESC_MAX_CHARS = 500
-_MAX_LINK_CHARS = 2048
-
-
-def _clean_html_text(raw: Optional[str],
-                     limit: int = _DESC_MAX_CHARS) -> Optional[str]:
-    """RSS <description> → plain text: unescape entities, strip tags,
-    collapse whitespace, truncate. Unescape and strip both run twice
-    because feeds routinely double-encode (&amp;lt;p&amp;gt; wrapping
-    real markup) — the second unescape can reveal fresh tags."""
-    if not raw:
-        return None
-    text = html.unescape(raw)
-    text = re.sub(r"<[^>]+>", " ", text)
-    text = html.unescape(text)
-    text = re.sub(r"<[^>]+>", " ", text)
-    text = re.sub(r"\s+", " ", text).strip()
-    if not text:
-        return None
-    if len(text) > limit:
-        text = text[:limit].rstrip() + "…"
-    return text
-
-
-def _safe_link(raw: Optional[str]) -> Optional[str]:
-    """Article URL for display: http(s) only (an RSS <link> is untrusted
-    and lands in an href), bounded length (Google News redirect URLs run
-    1-2 KB; anything past that is bloat, not a link)."""
-    if not raw:
-        return None
-    link = raw.strip()
-    if not link.lower().startswith(("http://", "https://")):
-        return None
-    if len(link) > _MAX_LINK_CHARS:
-        return None
-    return link
-
-
 @dataclass
 class EconomicEvent:
     """Single economic calendar event."""
@@ -194,11 +145,6 @@ class NewsSentiment:
     # Per-currency score: monetary-policy keywords flip sign for XAU
     # (hawkish is dollar-bullish but gold-bearish).
     sentiment_by_currency: Dict[str, float] = field(default_factory=dict)
-    # Display-only enrichment (dashboard detail dialog). Optional with
-    # defaults so existing constructor calls keep working.
-    link: Optional[str] = None         # article URL (Google News = redirect)
-    description: Optional[str] = None  # plain text, tags stripped, ≤500 chars
-    publisher: Optional[str] = None    # Google News <source> text, else None
 
 
 @dataclass
@@ -304,14 +250,6 @@ class NewsEngine:
     _LLM_SENT_CACHE_MAX = 400
     _LLM_EVENT_CACHE_MAX = 100
 
-    # Per-source fetch health for status displays (telegram /status).
-    # name → {"ok": ts|None, "err": ts|None, "detail": str, "extra": str}.
-    # All writers run under _cache_lock (fetches happen inside
-    # _cached_fetch); readers render without the lock — worst case a
-    # display line is one cycle stale. A source is "failing" when its
-    # last error is newer than its last success.
-    _source_health: Dict[str, dict] = {}
-
     def __init__(self, config: NewsConfig):
         self.config = config
         self._session = requests.Session()
@@ -488,24 +426,6 @@ class NewsEngine:
 
         return net, desc
 
-    def get_week_events(
-        self, currencies: Optional[List[str]] = None
-    ) -> List[EconomicEvent]:
-        """Full cached calendar week (past and upcoming), filtered to
-        `currencies` (default: tracked_currencies), sorted by time.
-
-        Read-only accessor for display surfaces (the web dashboard) —
-        rides the same class-level cache as signal scans, so it adds no
-        network I/O. No production signal path calls this.
-        """
-        if currencies is None:
-            currencies = self.config.tracked_currencies
-        events = self._cached_fetch("_calendar_cache", self._fetch_calendar_raw)
-        return sorted(
-            (e for e in events if e.currency in currencies),
-            key=lambda e: e.time,
-        )
-
     # ── Private Methods ────────────────────────────────────────────
 
     # Maps country names in MyFXBook titles to currency codes
@@ -515,36 +435,6 @@ class NewsEngine:
         "canada": "CAD", "new zealand": "NZD",
     }
 
-    @classmethod
-    def _record_source(cls, name: str, ok: bool, detail: str = "",
-                       extra: str = "") -> None:
-        """Track one dependency's fetch outcome for status displays.
-        Logs state TRANSITIONS so the log file carries the story without
-        a line every 10-min cycle: first failure WARNING (with detail),
-        repeat failures DEBUG, recovery INFO."""
-        h = cls._source_health.setdefault(
-            name, {"ok": None, "err": None, "detail": "", "extra": ""})
-        was_failing = h["err"] is not None and (
-            h["ok"] is None or h["err"] > h["ok"])
-        if ok:
-            h["ok"] = time.time()
-            h["extra"] = extra
-            if was_failing:
-                logger.info("Source recovered: %s", name)
-        else:
-            h["err"] = time.time()
-            h["detail"] = detail
-            if was_failing:
-                logger.debug("Source still failing: %s (%s)", name, detail)
-            else:
-                logger.warning("Source failing: %s (%s)", name, detail)
-
-    @staticmethod
-    def source_is_failing(entry: Optional[dict]) -> bool:
-        """True when the entry's last error is newer than its last ok."""
-        return bool(entry and entry["err"] is not None and
-                    (entry["ok"] is None or entry["err"] > entry["ok"]))
-
     def _cached_fetch(self, cache_attr: str, fetch_fn) -> list:
         """Return cached data for `cache_attr` or refetch when the TTL
         expired. Successful results live for context_cache_ttl_seconds;
@@ -577,27 +467,14 @@ class NewsEngine:
         Returns an empty list (with a log warning) if unavailable —
         we never generate fake events since that causes constant false alarms.
         """
-        sources = (
+        for fetcher in (self._try_fetch_calendar_api, self._try_fetch_myfxbook_rss):
-            (self._try_fetch_calendar_api, "calendar:forexfactory",
-             "ForexFactory"),
-            (self._try_fetch_myfxbook_rss, "calendar:myfxbook", "MyFXBook"),
-        )
-        for fetcher, src, label in sources:
             try:
                 events = fetcher()
                 if events:
-                    self._record_source(src, True)
-                    self._record_source(
-                        "calendar", True,
-                        extra=f"{len(events)} events via {label}")
                     return events
-                self._record_source(src, False, "no events returned")
             except Exception as e:
-                self._record_source(
-                    src, False, f"{type(e).__name__}: {e}")
                 logger.debug(f"Calendar fetch exception ({fetcher.__name__}): {e}")
 
-        self._record_source("calendar", False, "all sources failed")
         logger.info(
             "Economic calendar data unavailable. "
             "Event risk checks will be skipped for this cycle."
@@ -744,16 +621,11 @@ class NewsEngine:
         """Fetch and analyze news headlines for sentiment."""
         headlines = []
         seen_titles = set()
-        ok_feeds = 0
 
         for feed_url in self.config.rss_feeds:
-            host = feed_url.split("/")[2] if "//" in feed_url else feed_url
-            kept_before = len(headlines)
             try:
                 resp = self._session.get(feed_url, timeout=10)
                 if resp.status_code != 200:
-                    self._record_source(
-                        f"rss:{host}", False, f"HTTP {resp.status_code}")
                     continue
 
                 root = ET.fromstring(resp.content)
@@ -786,21 +658,6 @@ class NewsEngine:
                     currencies = self._extract_currencies(title)
 
                     if currencies:
-                        link_el = item.find("link")
-                        desc_el = item.find("description")
-                        src_el = item.find("source")
-                        # itertext() survives CDATA and stray child
-                        # elements where .text stops at the first child.
-                        desc = _clean_html_text(
-                            "".join(desc_el.itertext())
-                            if desc_el is not None else None)
-                        # Google News descriptions are often just the
-                        # headline again inside an anchor — worthless.
-                        if desc and desc.strip() == title.strip():
-                            desc = None
-                        publisher = None
-                        if src_el is not None:
-                            publisher = (src_el.text or "").strip() or None
                         headlines.append(NewsSentiment(
                             headline=title,
                             source=feed_url,
@@ -812,27 +669,11 @@ class NewsEngine:
                                 cur: self._analyze_headline_sentiment(title, cur)
                                 for cur in currencies
                             },
-                            link=_safe_link(
-                                link_el.text if link_el is not None else None),
-                            description=desc,
-                            publisher=publisher,
                         ))
 
-                ok_feeds += 1
-                self._record_source(
-                    f"rss:{host}", True,
-                    extra=f"{len(headlines) - kept_before} kept")
             except Exception as e:
-                self._record_source(
-                    f"rss:{host}", False, f"{type(e).__name__}: {e}")
                 logger.debug(f"Failed to fetch RSS from {feed_url}: {e}")
 
-        self._record_source(
-            "headlines", ok_feeds > 0,
-            detail="all feeds failed" if ok_feeds == 0 else "",
-            extra=(f"{len(headlines)} headlines from "
-                   f"{ok_feeds}/{len(self.config.rss_feeds)} feeds"))
-
         try:
             self._apply_llm_sentiment(headlines)
         except Exception as e:
@@ -943,13 +784,7 @@ class NewsEngine:
     def _llm_event_direction(self, event: EconomicEvent) -> Optional[str]:
         """Gold direction for a released surprise, e.g. "bearish: strong
         jobs lift yields". None → caller keeps the numeric description.
-        Cached per release; failures are NOT cached (retry next scan).
+        Cached per release; failures are NOT cached (retry next scan)."""
-
-        The actual-vs-forecast comparison is precomputed and sent as
-        "vs_forecast": the 2026-08-06 validation found the model reads
-        close K/M-suffixed pairs backwards ~11% of the time (e.g. claims
-        202K vs 217K called "higher than forecast") when left to compare
-        the raw numbers itself."""
         key = (event.event, event.currency, str(event.actual or ""))
         with NewsEngine._cache_lock:
             if key in NewsEngine._llm_event_cache:
@@ -957,18 +792,12 @@ class NewsEngine:
         client = get_llm_client()
         if not client.available:
             return None
-        vs_forecast = {
-            "positive": "above forecast",
-            "negative": "below forecast",
-            "inline": "in line with forecast",
-        }.get(event.surprise or "", "unknown")
         result = client.chat_json(_LLM_EVENT_SYSTEM, {
             "event": event.event,
             "currency": event.currency,
             "actual": event.actual,
             "forecast": event.forecast,
             "previous": event.previous,
-            "vs_forecast": vs_forecast,
         })
         if not result:
             return None

oanda_daemon.py

@@ -90,17 +90,8 @@ STATE_FILE = os.path.join(
 def _oanda_setting_int(env_key: str, json_key: str, default: int) -> int:
     env = os.environ.get(env_key)
     if env:
-        try:
+        return int(env)
-            return int(env)
+    return int(get_app_config().oanda_settings.get(json_key, default))
-        except ValueError:
-            raise SystemExit(
-                f"{env_key}={env!r} is not an integer — refusing to start")
-    try:
-        return int(get_app_config().oanda_settings.get(json_key, default))
-    except (TypeError, ValueError):
-        raise SystemExit(
-            f"config.json oanda.{json_key} is not an integer — "
-            f"refusing to start")
 
 
 def _oanda_setting_str(env_key: str, json_key: str, default: str) -> str:
@@ -139,11 +130,8 @@ def _seconds_until_boundary(interval_minutes: int, buffer: int = 10) -> float:
     elapsed_seconds = elapsed_minutes * 60 + now.second + now.microsecond / 1e6
     interval_seconds = interval_minutes * 60
     remaining = interval_seconds - elapsed_seconds + buffer
-    # NOTE: remaining > buffer always (elapsed < interval by construction);
+    if remaining <= buffer:
-    # a call at the exact boundary instant targets the NEXT boundary —
+        remaining += interval_seconds
-    # steady-state loops wake at boundary+buffer, so only a process
-    # started in the boundary microsecond skips one scan (2026-08-04
-    # audit I42; the old `remaining <= buffer` guard was unreachable).
     return remaining
 
 # ── Trade state persistence ───────────────────────────────────────────
@@ -177,13 +165,7 @@ class TradingDaemon:
         base_url = oanda_base_url()
         lot_size = _oanda_setting_int("OANDA_LOT_SIZE", "lot_size", 1)
         self._quality_gate = _oanda_setting_str(
-            "OANDA_QUALITY_GATE", "quality_gate", "C").strip()
+            "OANDA_QUALITY_GATE", "quality_gate", "C")
-        if self._quality_gate not in QUALITY_RANK:
-            # Fail CLOSED: an unrecognized gate must never silently fall
-            # back to the most permissive rank (QUALITY_RANK.get default).
-            raise SystemExit(
-                f"OANDA_QUALITY_GATE/quality_gate {self._quality_gate!r} "
-                f"is not one of {sorted(QUALITY_RANK)} — refusing to start")
         self._max_hold_bars = _oanda_setting_int(
             "OANDA_MAX_HOLD_BARS", "max_hold_bars", 300)
         self._price_precision = PAIR_INFO[SYMBOL]["price_precision"]
@@ -293,86 +275,60 @@ class TradingDaemon:
     # ── Position discovery (restart recovery) ──────────────────────
 
     def _discover_existing_positions(self) -> None:
-        self._reconcile_positions("boot discovery")
-
-    def _reconcile_positions(self, reason: str) -> None:
-        """Re-sync tracked state with the broker's open trades. Safe to
-        call mid-run (lock-guarded); used at boot and whenever a fill or
-        placement error suggests state and broker may have diverged."""
         try:
             open_trades = self._executor.get_open_trades(INSTRUMENT)
         except OandaAPIError:
-            logger.exception(
+            logger.exception("Failed to query open trades at boot")
-                "Failed to query open trades (%s) — reconcile skipped",
-                reason)
             return
 
+        saved = self._state.get("trades", {})
         live_ids = {t["id"] for t in open_trades}
 
-        with self._state_lock:
+        stale = [tid for tid in saved if tid not in live_ids]
-            saved = self._state.get("trades", {})
+        for tid in stale:
+            logger.info(
+                "Stale trade %s in state file (closed on Oanda), removing", tid)
+            del saved[tid]
 
-            stale = [tid for tid in saved if tid not in live_ids]
+        for trade in open_trades:
-            for tid in stale:
+            tid = trade["id"]
+            if tid in saved:
                 logger.info(
-                    "Stale trade %s in state file (closed on Oanda), "
+                    "Recovered trade %s from state: %s %s @ %.2f, SL %.2f",
-                    "removing", tid)
+                    tid, saved[tid].get("direction"), INSTRUMENT,
-                del saved[tid]
+                    saved[tid].get("entry"), saved[tid].get("current_sl"))
+                continue
 
-            for trade in open_trades:
+            direction = "BUY" if float(trade.get("currentUnits", 0)) > 0 else "SELL"
-                tid = trade["id"]
+            entry = float(trade.get("price", 0))
-                if tid in saved:
+            sl_order = trade.get("stopLossOrder", {})
-                    logger.info(
+            sl = float(sl_order.get("price", entry))
-                        "Recovered trade %s from state: %s %s @ %.2f, SL %.2f",
+            risk = abs(entry - sl) if sl else 1.0
-                        tid, saved[tid].get("direction"), INSTRUMENT,
-                        saved[tid].get("entry"), saved[tid].get("current_sl"))
-                    continue
 
-                direction = ("BUY" if float(trade.get("currentUnits", 0)) > 0
+            saved[tid] = {
-                             else "SELL")
+                "tf": "H1",
-                entry = float(trade.get("price", 0))
+                "direction": direction,
-                sl_order = trade.get("stopLossOrder") or {}
+                "entry": entry,
-                sl_price = sl_order.get("price")
+                "sl_initial": sl,
-                if sl_price is None:
+                "current_sl": sl,
-                    # No broker-side stop at all: risk is undefined, so the
+                "risk": risk,
-                    # trailing model cannot manage this trade. Scream — a
+                "peak": entry,
-                    # naked position needs a human (or a policy) to set a
+                "trail_armed": False,
-                    # stop; fabricating one here would invent trading
+                "trail_arm_r": 0.6,
-                    # behavior the books never validated.
+                "trail_dist_r": 0.75,
-                    logger.critical(
+                "bars_held": 0,
-                        "Adopted trade %s (%s @ %.2f) has NO STOP LOSS on "
+                "last_bar_ts": None,
-                        "the broker — it will NOT be trailed. Set a stop "
+                "signal_bar_ts": None,
-                        "manually or close it.", tid, direction, entry)
+            }
-                    sl = entry
+            logger.info(
-                    risk = 0.0
+                "Adopted unknown trade %s: %s @ %.2f, SL %.2f, risk %.2f "
-                else:
+                "(conservative defaults)",
-                    sl = float(sl_price)
+                tid, direction, entry, sl, risk)
-                    risk = abs(entry - sl)
 
-                saved[tid] = {
+        self._state["trades"] = saved
-                    "tf": "H1",
+        _save_state(self._state)
-                    "direction": direction,
-                    "entry": entry,
-                    "sl_initial": sl,
-                    "current_sl": sl,
-                    "risk": risk,
-                    "peak": entry,
-                    "trail_armed": False,
-                    "trail_arm_r": 0.6,
-                    "trail_dist_r": 0.75,
-                    "bars_held": 0,
-                    "last_bar_ts": None,
-                    "signal_bar_ts": None,
-                }
-                logger.info(
-                    "Adopted unknown trade %s: %s @ %.2f, SL %.2f, risk %.2f "
-                    "(conservative defaults)",
-                    tid, direction, entry, sl, risk)
-
-            self._state["trades"] = saved
-            _save_state(self._state)
         logger.info(
-            "Position reconcile (%s): %d active trade(s)", reason, len(saved))
+            "Position discovery: %d active trade(s)", len(saved))
 
     # ── Candle-boundary loop ───────────────────────────────────────
 
@@ -565,15 +521,9 @@ class TradingDaemon:
                     stop_loss=ts.stop_loss,
                 )
             except OandaAPIError:
-                # The order is never retried (a timeout after a server-side
-                # fill would double-up on retry), but the server MAY have
-                # filled before the error reached us — reconcile so a live
-                # position can't stay untracked until the next restart.
                 logger.exception(
-                    "Failed to place %s order for %s %s — reconciling "
+                    "Failed to place %s order for %s %s",
-                    "broker positions",
                     ts.direction, SYMBOL, tf.value)
-                self._reconcile_positions("order placement error")
                 return
 
             fill_tx = result.get("orderFillTransaction")
@@ -585,16 +535,7 @@ class TradingDaemon:
             trade_opened = fill_tx.get("tradeOpened", {})
             trade_id = trade_opened.get("tradeID")
             if not trade_id:
-                # A fill with no tradeOpened means the order interacted
+                logger.warning("No trade ID in fill response")
-                # with an existing broker position (tradesClosed /
-                # tradeReduced on a netting account) — state and broker
-                # have diverged. Surface it and re-sync instead of
-                # silently forgetting a real fill.
-                logger.critical(
-                    "Order FILLED but opened no trade (netting close/"
-                    "reduce?) — fill: closed=%s reduced=%s. Reconciling.",
-                    fill_tx.get("tradesClosed"), fill_tx.get("tradeReduced"))
-                self._reconcile_positions("fill without tradeOpened")
                 return
 
             fill_price = float(fill_tx.get("price", ts.entry_price))
@@ -646,12 +587,8 @@ class TradingDaemon:
     # ── Trailing stop management ───────────────────────────────────
 
     def _manage_trailing(self, tf: Timeframe) -> None:
-        # Copy the per-trade dicts too: the loop mutates info (peak,
-        # bars_held, last_bar_ts) and must not touch live state outside
-        # the lock — another thread may be serializing it in _save_state.
         with self._state_lock:
-            trades = {tid: dict(info)
+            trades = dict(self._state.get("trades", {}))
-                      for tid, info in self._state.get("trades", {}).items()}
 
         for trade_id, info in trades.items():
             if info.get("tf") != tf.value:
@@ -697,16 +634,13 @@ class TradingDaemon:
                     self._executor.close_trade(trade_id)
                 except OandaAPIError:
                     # Keep tracking — the trade is still live on the
-                    # broker (incl. a 200 whose close order was cancelled,
+                    # broker; we retry the close at the next bar.
-                    # e.g. MARKET_HALTED — close_trade raises on no-fill);
-                    # we retry the close at the next bar.
                     logger.exception(
                         "Failed to close timed-out trade %s — will retry",
                         trade_id)
                     with self._state_lock:
-                        if trade_id in self._state["trades"]:
+                        self._state["trades"][trade_id] = info
-                            self._state["trades"][trade_id] = info
+                        _save_state(self._state)
-                            _save_state(self._state)
                     continue
                 with self._state_lock:
                     self._state["trades"].pop(trade_id, None)
@@ -717,14 +651,6 @@ class TradingDaemon:
             bar_high = float(last_bar["High"])
             bar_low = float(last_bar["Low"])
 
-            if info.get("risk", 0) <= 0:
-                # Adopted naked position (no broker SL → risk undefined):
-                # never manageable by the trailing model — keep it loud
-                # instead of silently skipping forever.
-                logger.warning(
-                    "Trade %s has risk<=0 (no stop loss) — trailing "
-                    "skipped; set a stop manually", trade_id)
-
             new_sl = self._compute_trail(info, bar_high, bar_low)
 
             if new_sl is not None:
@@ -741,13 +667,8 @@ class TradingDaemon:
                             trade_id)
 
             with self._state_lock:
-                # Write back ONLY if the trade is still tracked — another
+                self._state["trades"][trade_id] = info
-                # loop's _check_closed_trades may have popped it while we
+                _save_state(self._state)
-                # iterated (broker-side close). Re-inserting would create
-                # a ghost that blocks new entries.
-                if trade_id in self._state["trades"]:
-                    self._state["trades"][trade_id] = info
-                    _save_state(self._state)
 
     def _compute_trail(
         self, info: Dict, bar_high: float, bar_low: float

oanda_executor.py

@@ -69,13 +69,6 @@ class OandaExecutor:
                 "instrument": instrument,
                 "units": signed_units,
                 "timeInForce": "FOK",
-                # The book model is one position total; on a netting
-                # account an opposite-side order would silently CLOSE an
-                # existing (possibly untracked) position instead of opening
-                # the signal trade. OPEN_ONLY makes that case fail loudly
-                # (order cancelled) so the divergence surfaces instead of
-                # executing an unintended close.
-                "positionFill": "OPEN_ONLY",
                 "stopLossOnFill": {
                     "price": self._fmt_price(stop_loss),
                 },
@@ -163,18 +156,7 @@ class OandaExecutor:
                f"/trades/{trade_id}/close")
         resp = self._request_with_retry("PUT", url, json={"units": "ALL"})
         data = self._parse_json(resp)
-        close_tx = data.get("orderFillTransaction")
+        close_tx = data.get("orderFillTransaction", {})
-        if not close_tx:
-            # A 200 whose close order was created but NOT filled (e.g.
-            # cancelled with MARKET_HALTED during the settlement pause)
-            # means the position is STILL OPEN. Raising keeps the caller
-            # tracking it and retrying — silently returning here would
-            # orphan a live position.
-            reason = data.get("orderCancelTransaction", {}).get(
-                "reason", "no orderFillTransaction in response")
-            raise OandaAPIError(
-                f"Close of trade {trade_id} did not fill: {reason}",
-                resp.status_code, str(data)[:300])
         close_price = float(close_tx.get("price", 0))
         pl = float(close_tx.get("pl", 0))
         logger.info(

pair_signals/xauusd.py

@@ -117,54 +117,6 @@ campaign: an M5 hour-22 block (+4.8R on 12mo — the burst-heavy
 specific); M15 hour-16 unblock (+0.8R/+57 trades 12mo but -18.3R
 DD +34% on 5yr); every SELL-gate/vol/TP-size relaxation retested
 negative (see memory notes for the full grid).
-
-M5 override 4 — XAU_PB disabled (validated 2026-08-04, Oanda bars,
-$0.65 REAL spread — first campaign validated at measured cost on the
-live feed, per the 2026-08-04 re-pin baseline): m5_pullback_enabled
-False on M5. Loss forensics on the Oanda books identified XAU_PB as
-the largest wrong-direction loss source (12mo: -43.5R of the -156.6R
-wrong-direction bucket; 5yr: -274.5R). Disabling: 12mo +96.6R/588
-PF 1.44 max DD 13.9R halves +51.2/+45.4 vs +82.3R/788 PF 1.26 DD
-23.9R shipped (which FAILS the PF>=1.3 clause at real cost); OOS
-2024-06-01 -> 2025-03-30 +9.5R PF 1.34; 5yr +123.9R/956 PF 1.35 vs
-PF 1.16 with PB. PB's 5yr net contribution (+23.6R) was almost
-entirely the 2022 chop year (+36.1R vs -2.2R without) — a regime
-earner that has been a drag on trend-regime bars since; its tight
-stops are also the most cost-punished at $0.65. PB code and the
-pb_trail_* params STAY for future regime-gated experiments (same
-pattern as cp_enabled). Rejected in the same campaign (4 candidates
-total, N logged): hour-19 block (+3.7R 12mo but -3.3R OOS, -1.5R
-5yr — OOS gate, the classic trap); hours-14/15 block (-6.9R 12mo
-once PB is removed — their raw negativity WAS the PB losses);
-trail_dist_r 0.75 (-2.6R 12mo; the 0.88R average give-back is the
-validated price of riding trends, exits confirmed NOT the loss
-driver: exit-management accounts for 1-2% of lost R vs 50%
-wrong-direction / 49% entry-timing).
-
-M5 override 5 — EMA partial-alignment squash (validated 2026-08-04,
-Oanda bars, $0.65 real spread): ema_partial_neutral True on M5 —
-partial 4-EMA ribbon alignment (2-3 of 4 checks) scores NEUTRAL
-instead of a weak BUY/SELL. Motivated by the indicator audit: partial-
-alignment trades averaged -0.008R/84 vs +0.230R/341 fully aligned.
-Replay (knob mirrored in the backtester's precomputed EMA scorer in
-the same commit — live==backtest parity): 12mo +97.6R/556 PF 1.48
-DD 10.7R halves +47.8/+49.8 vs control +96.6R/588 PF 1.44 DD 13.9R;
-5yr +127.4R/912 PF 1.38 DD 12.8R vs +123.9R/956 PF 1.35 DD 13.2R,
-2022 improves -2.2 -> -1.1; OOS 2024-06-01 -> 2025-03-30 +9.4R
-PF 1.33 vs +9.5R PF 1.34 — a one-trade tie, documented as such (the
-OOS window holds up; it neither prefers nor punishes). Net effect:
-32 fewer marginal trades, every risk-adjusted measure better.
-Rejected in the same round (trial-logged): confirmation-TF swap
-M5->[M15,H4] (12mo +104.0R PF 1.51 DD 10.3 but OOS +7.8R PF 1.27
-vs control +9.5R PF 1.34 and 2022 3x worse — the H4 confirm keeps
-endorsing stale direction in chop; OOS-ordering rejection, same
-precedent as trail dist 1.0); confirm [H1,H4] (-31.8R 12mo);
-triple-confirm [M15,H1,H4] (-4.9R); divergence weight -1/3
-(-3.8R 12mo); candlestick weight +50% (-3.0R 12mo). The weight
-vector keeps proving calibrated as a system: divergence/candlestick/
-bollinger all look mis-weighted in-book and all replay worse when
-"fixed" (bollinger weight-0: +80.9R PF 1.29 DD 23.8 — its constant
-oppose vote is a load-bearing quality bar, like ROC).
 """
 
 from typing import Optional, List
@@ -197,22 +149,9 @@ def make_config(
         config.xau_strategy.blocked_hours_utc = [
             h for h in config.xau_strategy.blocked_hours_utc if h != 21
         ] + [9]
-        # Deliberately NOT mirrored into config.backtest.blocked_hours_utc:
-        # that copy is the generic non-XAU fallback carrying the JSON base
-        # only. Propagating a gold-validated per-TF delta there would leak
-        # XAU-calibrated hours to a future pair (pair-isolation rule).
-        # Every XAU consumer branches to the xau_strategy copy. Enforced
-        # by the Module 13 test "M5 factory appends hour 9 on top of a
-        # JSON base".
         # BUY-only chart patterns on M5 — see module docstring.
         config.xau_strategy.cp_enabled = True
         config.xau_strategy.cp_patterns = ["inv_hs", "double_bottom", "bull_flag"]
-        # XAU_PB pullback entries OFF — largest wrong-direction loss
-        # source at real cost on Oanda bars — see module docstring.
-        config.xau_strategy.m5_pullback_enabled = False
-        # Partial EMA-ribbon alignment scores NEUTRAL on M5 — its
-        # trades carried no edge — see module docstring.
-        config.technical.ema_partial_neutral = True
         # Wider trailing distance on M5 only — see module docstring.
         config.signals.trail_arm_r = 0.60
         config.signals.trail_dist_r = 0.85

providers/base.py

@@ -24,7 +24,6 @@ import time
 from abc import ABC, abstractmethod
 from datetime import datetime, timedelta
 from typing import AsyncIterator, Callable, Dict, Optional, Sequence, Tuple
-from zoneinfo import ZoneInfo
 
 import pandas as pd
 
@@ -32,36 +31,19 @@ from config import PAIR_INFO, PlatformConfig, Timeframe, XAUUSD
 
 logger = logging.getLogger(__name__)
 
-# The gold/FX market week is defined in New York time (see
-# _is_forex_weekend); a fixed UTC hour is wrong for half the year.
-_NY_TZ = ZoneInfo("America/New_York")
-
 
 def _is_forex_weekend(now=None) -> bool:
-    """Gold/FX market closed for the weekend.
+    """Forex market closed: Friday 22:00 UTC → Sunday 22:00 UTC."""
-
-    The market week is anchored to NEW YORK time — Friday 17:00 ET →
-    Sunday 18:00 ET — not a fixed UTC hour. During US DST the feed
-    actually stops at 21:00 UTC (measured on 5 years of Oanda bars:
-    159/162 summer Fridays; winter close 22:00, winter reopen Sunday
-    23:00 UTC). The old hardcoded Fri 22:00→Sun 22:00 UTC rule produced
-    two false-"STALE DATA/feed lagging" hours every week (2026-08-04
-    audit I43, fixed per owner decision).
-
-    Kept as three deliberate copies (here, signal_aggregator.py,
-    gpu_backtest._is_xau_blocked_by_calendar) — a Module 21 test asserts
-    they agree hour-by-hour across DST transitions.
-    """
     if now is None:
         now = pd.Timestamp.now(tz="UTC")
-    ny = now.astimezone(_NY_TZ)
+    wd = now.weekday()
-    wd = ny.weekday()
+    hour = now.hour
-    if wd == 4:                      # Friday
+    if wd == 4 and hour >= 22:
-        return ny.hour >= 17
+        return True
-    if wd == 5:                      # Saturday
+    if wd == 5:
+        return True
+    if wd == 6 and hour < 22:
         return True
-    if wd == 6:                      # Sunday
-        return ny.hour < 18
     return False
 
 
@@ -104,15 +86,9 @@ class BaseDataProvider(ABC):
         self.config = config
         self._cache: Dict[str, pd.DataFrame] = {}
         self._cache_timestamps: Dict[str, float] = {}
-        # Requested fetch depth per key: a cached short frame must not
-        # satisfy a caller asking for a longer lookback.
-        self._cache_outputsize: Dict[str, int] = {}
         # The bot shares one provider across a thread pool; cache eviction
         # is check-then-act and needs locking.
         self._cache_lock = threading.Lock()
-        # Single-flight: one API fetch per (symbol, TF) at a time; other
-        # threads wait and then read the cache.
-        self._inflight: Dict[str, threading.Event] = {}
 
     # ── Abstract: the only provider-specific piece ─────────────────
 
@@ -143,97 +119,68 @@ class BaseDataProvider(ABC):
             lookback_days = self.config.data_lookback_days
 
         cache_key = f"{symbol}_{timeframe.value}"
+        now = time.time()
+
+        cache_ttl = self.CANDLE_SECONDS.get(timeframe, self.config.refresh_interval)
+        with self._cache_lock:
+            cached = self._cache.get(cache_key)
+            cached_at = self._cache_timestamps.get(cache_key)
+        if (
+            cached is not None
+            and cached_at is not None
+            and (now - cached_at) < cache_ttl
+            and not cached.empty
+        ):
+            if (self._is_data_fresh(cached, symbol, timeframe)
+                    and self._has_latest_closed_bar(cached, timeframe)):
+                return cached
+            logger.warning(f"{symbol} {timeframe.value}: cached data is stale, refetching")
+            with self._cache_lock:
+                self._cache.pop(cache_key, None)
+                self._cache_timestamps.pop(cache_key, None)
+
         outputsize = max(
             300,
             min(int(lookback_days * self.BARS_PER_DAY[timeframe]) + 20,
                 self.MAX_BARS_PER_REQUEST),
         )
+        df = self._fetch_raw(symbol, timeframe, outputsize)
+        df = self._drop_forming_candle(df, timeframe)
+        df = self._validate_and_clean(df, symbol, timeframe)
 
-        while True:
+        # The closed-bar guarantee must hold for FRESH responses too, not
-            now = time.time()
+        # just cache hits: the scanner fires seconds after the boundary,
-            cache_ttl = self.CANDLE_SECONDS.get(
+        # and a lagging feed can answer without the just-closed bar — the
-                timeframe, self.config.refresh_interval)
+        # scan would silently analyze (and session-check) the previous
-            with self._cache_lock:
+        # bar. Give the feed one short grace retry, then proceed loudly.
-                cached = self._cache.get(cache_key)
+        if not df.empty and not self._has_latest_closed_bar(df, timeframe):
-                cached_at = self._cache_timestamps.get(cache_key)
+            time.sleep(self.FRESH_FETCH_RETRY_SECONDS)
-                cached_size = self._cache_outputsize.get(cache_key, 0)
+            try:
-            if (
+                retry = self._fetch_raw(symbol, timeframe, outputsize)
-                cached is not None
+                retry = self._drop_forming_candle(retry, timeframe)
-                and cached_at is not None
+                retry = self._validate_and_clean(retry, symbol, timeframe)
-                and (now - cached_at) < cache_ttl
+                if retry is not None and not retry.empty:
-                and not cached.empty
+                    df = retry
-                # A caller asking for MORE history than the cached fetch
+            except DataIntegrityError as e:
-                # requested must not be served the short frame silently.
-                and cached_size >= outputsize
-            ):
-                if (self._is_data_fresh(cached, symbol, timeframe)
-                        and self._has_latest_closed_bar(cached, timeframe)):
-                    return cached
-                logger.warning(f"{symbol} {timeframe.value}: cached data is stale, refetching")
-                with self._cache_lock:
-                    self._cache.pop(cache_key, None)
-                    self._cache_timestamps.pop(cache_key, None)
-                    self._cache_outputsize.pop(cache_key, None)
-
-            # Single-flight: concurrent scans for the same (symbol, TF)
-            # must not each hit the API (the four TF loops fire together
-            # at aligned boundaries and Oanda has no client throttle).
-            # First caller claims the key and fetches; the rest wait,
-            # then re-check the cache.
-            with self._cache_lock:
-                inflight = self._inflight.get(cache_key)
-                if inflight is None:
-                    self._inflight[cache_key] = threading.Event()
-                    break
-            inflight.wait(timeout=90)
-
-        try:
-            df = self._fetch_raw(symbol, timeframe, outputsize)
-            df = self._drop_forming_candle(df, timeframe)
-            df = self._validate_and_clean(df, symbol, timeframe)
-
-            # The closed-bar guarantee must hold for FRESH responses too,
-            # not just cache hits: the scanner fires seconds after the
-            # boundary, and a lagging feed can answer without the
-            # just-closed bar — the scan would silently analyze (and
-            # session-check) the previous bar. Give the feed one short
-            # grace retry, then proceed loudly.
-            if not df.empty and not self._has_latest_closed_bar(df, timeframe):
-                time.sleep(self.FRESH_FETCH_RETRY_SECONDS)
-                try:
-                    retry = self._fetch_raw(symbol, timeframe, outputsize)
-                    retry = self._drop_forming_candle(retry, timeframe)
-                    retry = self._validate_and_clean(retry, symbol, timeframe)
-                    if retry is not None and not retry.empty:
-                        df = retry
-                except DataIntegrityError as e:
-                    logger.warning(
-                        f"{symbol} {timeframe.value}: boundary retry refetch "
-                        f"failed ({e}) — keeping original data")
-                if not self._has_latest_closed_bar(df, timeframe):
-                    logger.warning(
-                        f"{symbol} {timeframe.value}: feed lagging — just-closed bar "
-                        f"still missing after retry; this scan may run one bar behind"
-                    )
-
-            if not self._is_data_fresh(df, symbol, timeframe):
                 logger.warning(
-                    f"STALE DATA from API: {symbol} {timeframe.value} — "
+                    f"{symbol} {timeframe.value}: boundary retry refetch "
-                    f"last bar {df.index[-1]} is too old. Signal quality may be degraded."
+                    f"failed ({e}) — keeping original data")
+            if not self._has_latest_closed_bar(df, timeframe):
+                logger.warning(
+                    f"{symbol} {timeframe.value}: feed lagging — just-closed bar "
+                    f"still missing after retry; this scan may run one bar behind"
                 )
 
-            with self._cache_lock:
+        if not self._is_data_fresh(df, symbol, timeframe):
-                self._cache[cache_key] = df
+            logger.warning(
-                self._cache_timestamps[cache_key] = now
+                f"STALE DATA from API: {symbol} {timeframe.value} — "
-                self._cache_outputsize[cache_key] = outputsize
+                f"last bar {df.index[-1]} is too old. Signal quality may be degraded."
-            return df
+            )
-        finally:
+
-            # Release waiters even when the fetch raised — they re-check
+        with self._cache_lock:
-            # the cache and (on a miss) claim the key themselves.
+            self._cache[cache_key] = df
-            with self._cache_lock:
+            self._cache_timestamps[cache_key] = now
-                ev = self._inflight.pop(cache_key, None)
+        return df
-            if ev is not None:
-                ev.set()
 
     def fetch_range(
         self,
@@ -252,7 +199,6 @@ class BaseDataProvider(ABC):
         chunks = []
         cursor_end = end
         empty_streak = 0
-        error_streak = 0
         while cursor_end > start:
             cursor_start = max(start, cursor_end - timedelta(days=days_per_chunk))
             try:
@@ -261,27 +207,19 @@ class BaseDataProvider(ABC):
                     start=cursor_start, end=cursor_end,
                 )
             except Exception as e:
-                # A raising chunk is a PROVIDER ERROR (auth, 429, 5xx,
+                # Providers raise for windows with no data (weekends,
-                # malformed payload) — not "no data". Three in a row means
+                # holidays, asset gaps). Skip the window, step back, retry
-                # the rest of the history is unreachable: raise instead of
+                # — but bail after a few consecutive empties.
-                # returning a silently truncated frame (the backtester
-                # would treat it as complete).
                 logger.warning(
                     f"Chunk {cursor_start.date()}–{cursor_end.date()} failed: {e}"
                 )
-                error_streak += 1
+                empty_streak += 1
-                if error_streak >= 3:
+                if empty_streak >= 3:
-                    raise self.DATA_ERROR(
+                    break
-                        f"{symbol} {timeframe.value}: {error_streak} "
-                        f"consecutive chunk failures walking back from "
-                        f"{cursor_end.date()} — refusing to return "
-                        f"truncated history (last error: {e})") from e
                 cursor_end = cursor_start - timedelta(seconds=1)
                 continue
 
             if chunk is None or chunk.empty:
-                # Genuinely empty windows (weekends/holidays/asset gaps)
-                # are normal; several in a row = start of history.
                 empty_streak += 1
                 if empty_streak >= 3:
                     break
@@ -289,18 +227,9 @@ class BaseDataProvider(ABC):
                 continue
 
             empty_streak = 0
-            error_streak = 0
             chunks.append(chunk)
             # Step back; dedup on concat.
-            new_cursor = chunk.index[0] - timedelta(seconds=1)
+            cursor_end = chunk.index[0] - timedelta(seconds=1)
-            if new_cursor >= cursor_end:
-                # A provider ignoring start/end would pin the cursor and
-                # loop forever — refuse to spin.
-                logger.warning(
-                    f"{symbol} {timeframe.value}: pagination made no "
-                    f"progress at {cursor_end} — stopping")
-                break
-            cursor_end = new_cursor
             if len(chunk) < 2:
                 break
 
@@ -314,17 +243,7 @@ class BaseDataProvider(ABC):
         df = self._drop_forming_candle(df, timeframe)
         start_ts = pd.Timestamp(start)
         end_ts = pd.Timestamp(end)
-        df = df[(df.index >= start_ts) & (df.index <= end_ts)]
+        return df[(df.index >= start_ts) & (df.index <= end_ts)]
-        # Coverage sanity: a head materially later than the requested
-        # start means history was truncated (empty-streak bail or provider
-        # depth limit). Loud, not fatal — some assets genuinely start late.
-        if not df.empty and (df.index[0] - start_ts) > pd.Timedelta(days=7):
-            logger.error(
-                f"{symbol} {timeframe.value}: fetched history starts "
-                f"{df.index[0]} but {start_ts} was requested — "
-                f"~{(df.index[0] - start_ts).days} days missing from the "
-                f"head; treat downstream results accordingly")
-        return df
 
     def is_data_fresh(
         self, df: pd.DataFrame, symbol: str, timeframe: Timeframe
@@ -404,19 +323,8 @@ class BaseDataProvider(ABC):
             return df
         candle_seconds = self.CANDLE_SECONDS.get(timeframe, 3600)
         now = pd.Timestamp.now(tz="UTC")
-        # Loop, not a single slice: a misbehaving feed can return MORE
+        if df.index[-1] + pd.Timedelta(seconds=candle_seconds) > now:
-        # than one not-yet-closed bar, and a single iloc[:-1] would let
-        # the residual forming bar through cleaning (future-tolerance is
-        # +5 min) straight into the signal engine.
-        dropped = 0
-        while (not df.empty
-               and df.index[-1] + pd.Timedelta(seconds=candle_seconds) > now):
             df = df.iloc[:-1]
-            dropped += 1
-        if dropped > 1:
-            logger.warning(
-                "%s: feed returned %d forming/future bars — all dropped",
-                timeframe.value, dropped)
         return df
 
     def _is_data_fresh(self, df: pd.DataFrame, symbol: str, timeframe: Timeframe) -> bool:
@@ -464,20 +372,6 @@ class BaseDataProvider(ABC):
             logger.warning(f"{symbol}: Dropping {future_mask.sum()} future timestamps")
             df = df[~future_mask]
 
-        # Non-positive prices are corrupt, never real gold quotes. Reject
-        # the row BEFORE the consistency repair below — repairing High/Low
-        # from a corrupt 0.0 Open/Close would propagate the zero into
-        # previously-valid fields (2026-08-04 audit, I45).
-        nonpositive = (df[["Open", "High", "Low", "Close"]] <= 0).any(axis=1)
-        if nonpositive.any():
-            logger.warning(
-                f"{symbol}: Dropping {nonpositive.sum()} bars with "
-                f"non-positive prices (corrupt feed data)")
-            df = df[~nonpositive]
-            if df.empty:
-                raise self.DATA_ERROR(
-                    f"{symbol}: all bars had non-positive prices")
-
         invalid_ohlc = (
             (df["High"] < df["Low"]) |
             (df["High"] < df["Open"]) |
@@ -492,10 +386,7 @@ class BaseDataProvider(ABC):
                 df.loc[idx, "High"] = max(vals + [df.loc[idx, "High"]])
                 df.loc[idx, "Low"]  = min(vals + [df.loc[idx, "Low"]])
 
-        # inf returns (division by a zero/na Close) would NaN the std and
-        # silently disarm the whole 10σ pass — neutralize them first.
         returns = df["Close"].pct_change()
-        returns = returns.replace([float("inf"), float("-inf")], float("nan"))
         std_ret = returns.std()
         if std_ret > 0:
             dev = returns - returns.mean()

providers/oanda.py

@@ -8,7 +8,6 @@ Oanda's candle JSON, and exposes the real bid/ask spread from MBA candles.
 """
 
 import logging
-import time
 from datetime import datetime
 from typing import Dict, Optional
 
@@ -32,7 +31,6 @@ class OandaDataProvider(BaseDataProvider):
 
     name = "oanda"
     DATA_ERROR = OandaDataError
-    RETRY_DELAY_S = 2.0
 
     OANDA_GRANULARITY = {
         Timeframe.M1: "M1",
@@ -126,35 +124,11 @@ class OandaDataProvider(BaseDataProvider):
         else:
             params["count"] = min(count, self.MAX_BARS_PER_REQUEST)
 
-        # One retry absorbs Oanda's sporadic spurious 401s (~15-20/day on
+        try:
-        # the practice API — the same token succeeds on adjacent requests)
+            resp = self._session.get(url, params=params, timeout=30)
-        # plus transient 5xx/network faults. A real credential problem
+        except requests.RequestException as e:
-        # fails both attempts and raises exactly as before. Other 4xx
+            raise OandaDataError(
-        # (bad request, 429) are never retried.
+                f"Oanda candle fetch failed for {instrument}: {e}") from e
-        last_err: Optional[OandaDataError] = None
-        for attempt in range(2):
-            if attempt:
-                time.sleep(self.RETRY_DELAY_S)
-            try:
-                resp = self._session.get(url, params=params, timeout=30)
-            except requests.RequestException as e:
-                last_err = OandaDataError(
-                    f"Oanda candle fetch failed for {instrument}: {e}")
-                last_err.__cause__ = e
-                continue
-            if resp.status_code == 401 or resp.status_code >= 500:
-                last_err = OandaDataError(
-                    f"Oanda candle fetch failed for {instrument}: "
-                    f"HTTP {resp.status_code}: {resp.text[:300]}")
-                if attempt == 0:
-                    logger.warning(
-                        f"Oanda fetch for {instrument} got HTTP "
-                        f"{resp.status_code}; retrying in "
-                        f"{self.RETRY_DELAY_S}s")
-                continue
-            break
-        else:
-            raise last_err
 
         if resp.status_code != 200:
             raise OandaDataError(
@@ -175,37 +149,23 @@ class OandaDataProvider(BaseDataProvider):
         rows = []
         last_bid_close = None
         last_ask_close = None
-        skipped_malformed = 0
         for c in candles:
             if not c.get("complete", False):
                 continue
             mid = c.get("mid", {})
             bid = c.get("bid", {})
             ask = c.get("ask", {})
-            # A candle missing any OHLC field must be SKIPPED, never
-            # defaulted to 0.0: a zero price flows through cleaning as a
-            # "legit" bar and poisons ATR/Keltner/SL math — and a
-            # mid-frame zero Close makes pct_change() emit inf, which
-            # NaNs the std and disarms the 10σ outlier filter for the
-            # whole frame (2026-08-04 audit, I45).
-            if any(mid.get(k) is None for k in ("o", "h", "l", "c")):
-                skipped_malformed += 1
-                continue
             rows.append({
                 "Datetime": pd.Timestamp(c["time"]).tz_convert("UTC"),
-                "Open": float(mid["o"]),
+                "Open": float(mid.get("o", 0)),
-                "High": float(mid["h"]),
+                "High": float(mid.get("h", 0)),
-                "Low": float(mid["l"]),
+                "Low": float(mid.get("l", 0)),
-                "Close": float(mid["c"]),
+                "Close": float(mid.get("c", 0)),
                 "Volume": int(c.get("volume", 0)),
             })
             if bid and ask:
                 last_bid_close = float(bid.get("c", 0))
                 last_ask_close = float(ask.get("c", 0))
-        if skipped_malformed:
-            logger.warning(
-                "Oanda returned %d malformed candle(s) for %s (missing "
-                "OHLC fields) — skipped", skipped_malformed, instrument)
 
         if not rows:
             raise OandaDataError(

providers/twelve_data.py

@@ -210,12 +210,6 @@ class TwelveDataProvider(BaseDataProvider):
         The TD WS speaks display symbols ("XAU/USD"); ticks are mapped back
         to the canonical pair symbols passed in. The caller owns the
         reconnect/backoff loop around this generator.
-
-        `on_connect` fires on the FIRST RECEIVED frame, not after the
-        subscribe send (2026-08-04 audit H38b, mirroring the Tiingo
-        provider) — firing pre-receive let a rejected API key reset the
-        caller's reconnect backoff on every attempt, pinning it at the
-        minimum forever.
         """
         import websockets
 
@@ -228,14 +222,11 @@ class TwelveDataProvider(BaseDataProvider):
             }
             await ws.send(json.dumps(subscribe))
             logger.info(f"WebSocket connected, subscribed to {len(ws_to_pair)} symbols")
+            if on_connect is not None:
+                on_connect()
             hb_task = asyncio.create_task(self._ws_heartbeat(ws))
-            connected = False
             try:
                 async for raw in ws:
-                    if not connected:
-                        connected = True
-                        if on_connect is not None:
-                            on_connect()
                     parsed = self._parse_ws_price(raw)
                     if parsed is None:
                         continue

signal_aggregator.py

@@ -10,12 +10,9 @@ import os
 from dataclasses import dataclass, field
 from datetime import datetime, timezone
 from typing import Dict, List, Optional
-from zoneinfo import ZoneInfo
 
 import pandas as pd
 
-_NY_TZ = ZoneInfo("America/New_York")
-
 from chart_patterns import detect_chart_pattern
 from config import PlatformConfig, SignalStrength, Timeframe
 from providers import BaseDataProvider, make_data_provider
@@ -26,19 +23,15 @@ logger = logging.getLogger(__name__)
 
 
 def _is_forex_weekend(now_utc: datetime) -> bool:
-    """Gold/FX market closed for the weekend: Friday 17:00 → Sunday 18:00
+    """Forex market closed: Friday 22:00 UTC → Sunday 22:00 UTC (Sydney open)."""
-    NEW YORK time (= Fri 21:00 UTC in US summer, 22:00 in winter —
+    wd = now_utc.weekday()
-    matches measured Oanda feed behavior; 2026-08-04 audit I43).
+    hour = now_utc.hour
-    Deliberate copy of providers.base._is_forex_weekend — a Module 21
+    if wd == 4 and hour >= 22:  # Friday after 22:00
-    test asserts the copies agree across DST transitions."""
+        return True
-    ny = now_utc.astimezone(_NY_TZ)
+    if wd == 5:  # All Saturday
-    wd = ny.weekday()
+        return True
-    if wd == 4:                      # Friday
+    if wd == 6 and hour < 22:  # Sunday before 22:00
-        return ny.hour >= 17
-    if wd == 5:                      # Saturday
         return True
-    if wd == 6:                      # Sunday
-        return ny.hour < 18
     return False
 
 
@@ -1260,17 +1253,7 @@ class SignalAggregator:
             ema200 = primary.key_levels.get("ema_trend")
             if ema200 and ema200 > 0 and report.current_price > 0:
                 ema_dist_pct = abs(report.current_price - ema200) / ema200 * 100.0
-                # FB2_EMA_LO: research override for the EMA200-proximity
+                _lo = float(os.environ.get("FB2_EMA_LO", "0.5"))
-                # penalty band; the tuned default is 0.5. An unparseable
-                # value must degrade to the default, never crash the scan
-                # (2026-08-04 audit; hardened per owner delegation).
-                try:
-                    _lo = float(os.environ.get("FB2_EMA_LO", "0.5"))
-                except (TypeError, ValueError):
-                    logger.warning(
-                        "FB2_EMA_LO=%r is not a number — using tuned "
-                        "default 0.5", os.environ.get("FB2_EMA_LO"))
-                    _lo = 0.5
                 if ema_dist_pct < _lo:
                     report.warnings.append(f"NEAR EMA200 ({ema_dist_pct:.2f}%) — chop risk")
                     penalties.append(("ema200_near", 5))

static/news_summary.php

@@ -1,245 +0,0 @@
-<?php
-/**
- * On-demand gold-impact AI summary for one published headline.
- *
- * The ONLY dynamic piece of the otherwise-static dashboard. Serves
- * GET ?id=<sha1-of-headline> and returns {"summary": ..., "generated_at": ...}.
- *
- * Abuse model (the page is public):
- *   - WHITELIST: the id must match a headline in the local news.json —
- *     arbitrary text can never reach the LLM through this endpoint.
- *   - CACHE FOREVER: headlines are immutable, so each id costs at most
- *     one DeepSeek call, ever (ai/<id>.json). Cache is checked BEFORE
- *     the whitelist on purpose: summaries stay reachable after the
- *     headline ages out of the 24 h feed window.
- *   - DAILY CAP: file-based UTC-day counter, FAIL CLOSED on counter I/O
- *     errors (a perms mistake must not mean unmetered spend).
- *
- * DeepSeek doctrine (mirrors llm_client.py — deliberate local copy, the
- * dashboard never imports bot code and this file never reads the repo's
- * config.json):
- *   - STATIC byte-identical system prompt (prefix-cache pricing).
- *   - "thinking": disabled is MANDATORY for V4-Flash (else it burns the
- *     timeout reasoning), temperature 0, json response_format.
- *   - One attempt, no retries; every failure returns a JSON error the
- *     page renders as a graceful fallback.
- *
- * Key file lives OUTSIDE the docroot: /etc/gold-dashboard/deepseek.key
- * (root:apache 640). Override paths/caps via Apache SetEnv:
- * DEEPSEEK_KEY_FILE, AI_SUMMARY_DAILY_CAP.
- */
-
-const NEWS_FILE = __DIR__ . '/news.json';
-const CACHE_DIR = __DIR__ . '/ai';   // apache-owned; created at deploy time
-const API_URL   = 'https://api.deepseek.com/chat/completions';
-const MODEL     = 'deepseek-v4-flash';
-const TIMEOUT   = 12;
-const MAX_TOKENS = 200;
-const SUMMARY_MAX_CHARS = 400;
-
-// .ht_ prefix: Apache's default <Files ".ht*"> rule denies these even
-// though they sit in the public docroot.
-const CAP_FILE = CACHE_DIR . '/.ht_cap.json';
-
-const SYSTEM_PROMPT = <<<'PROMPT'
-You are a forex/gold news analyst. You receive one news item as JSON:
-{"headline", "description", "score", "xau_score", "keywords", "currencies"}.
-"xau_score" is the platform's automated gold-effect score for this item,
--1.0 (gold down) to 1.0 (gold up); "score" is overall market sentiment.
-Both are automated and can be wrong — judge the news yourself.
-In 1-2 plain-English sentences (max 45 words) explain what this news
-means for the GOLD (XAU/USD) price and WHY it is bullish, bearish, or
-neutral. If your reading clearly contradicts xau_score, add "(the
-automated score disagrees)". Never write the words "xau_score" or
-"score" — a general reader sees this text.
-Gold doctrine: hawkish Fed / rate hikes / rising real yields / strong
-dollar are gold-bearish; easing / rate cuts / safe-haven flows /
-geopolitical stress / weak dollar are gold-bullish.
-Reply with ONLY JSON: {"summary": "..."}
-PROMPT;
-
-function respond(int $status, array $obj, string $cache = 'no-store'): void
-{
-    http_response_code($status);
-    header('Content-Type: application/json; charset=utf-8');
-    header('Cache-Control: ' . $cache);
-    echo json_encode($obj, JSON_UNESCAPED_SLASHES | JSON_UNESCAPED_UNICODE);
-    exit;
-}
-
-function atomic_write(string $path, string $data): bool
-{
-    $tmp = $path . '.tmp.' . getmypid();
-    if (@file_put_contents($tmp, $data) === false) {
-        return false;
-    }
-    if (!@rename($tmp, $path)) {
-        @unlink($tmp);
-        return false;
-    }
-    return true;
-}
-
-// ── 1. id validation ────────────────────────────────────────────────
-$id = $_GET['id'] ?? '';
-if (!preg_match('/^[a-f0-9]{40}$/', $id)) {
-    respond(400, ['error' => 'bad_id']);
-}
-
-// ── 2. cache first (immutable headlines → summaries never change) ───
-$cachePath = CACHE_DIR . '/' . $id . '.json';
-$cached = @file_get_contents($cachePath);
-if ($cached !== false && $cached !== '') {
-    http_response_code(200);
-    header('Content-Type: application/json; charset=utf-8');
-    header('Cache-Control: max-age=3600');
-    echo $cached;
-    exit;
-}
-
-// ── 3. whitelist against the published feed ─────────────────────────
-$news = json_decode((string)@file_get_contents(NEWS_FILE), true);
-if (!is_array($news)) {
-    respond(503, ['error' => 'no_feed']);
-}
-$hit = null;
-foreach ($news['headlines'] ?? [] as $h) {
-    // Fallback sha1(): a publisher that predates the id field still
-    // serves a valid whitelist during the restart window.
-    $hid = $h['id'] ?? sha1($h['headline'] ?? '');
-    if ($hid === $id) {
-        $hit = $h;
-        break;
-    }
-}
-if ($hit === null) {
-    respond(404, ['error'   => 'unknown_id',
-                  'message' => 'Headline is no longer in the feed.']);
-}
-
-// ── 4. daily cap (fail closed) ──────────────────────────────────────
-$dailyCap = (int)(getenv('AI_SUMMARY_DAILY_CAP') ?: 150);
-$capFh = @fopen(CAP_FILE, 'c+');
-if ($capFh === false || !flock($capFh, LOCK_EX)) {
-    respond(503, ['error' => 'cap_io']);
-}
-$today = gmdate('Y-m-d');
-$cap = json_decode((string)stream_get_contents($capFh), true);
-if (!is_array($cap) || ($cap['day'] ?? '') !== $today) {
-    $cap = ['day' => $today, 'count' => 0];
-}
-if ($cap['count'] >= $dailyCap) {
-    flock($capFh, LOCK_UN);
-    fclose($capFh);
-    respond(429, ['error' => 'cap_exceeded']);
-}
-// Increment BEFORE the LLM call — attempts count against the cap
-// (same semantics as llm_client._under_cap).
-$cap['count']++;
-rewind($capFh);
-if (ftruncate($capFh, 0) === false
-        || fwrite($capFh, json_encode($cap)) === false) {
-    flock($capFh, LOCK_UN);
-    fclose($capFh);
-    respond(503, ['error' => 'cap_io']);
-}
-fflush($capFh);
-flock($capFh, LOCK_UN);
-fclose($capFh);
-
-// ── 5. API key ──────────────────────────────────────────────────────
-$keyFile = getenv('DEEPSEEK_KEY_FILE') ?: '/etc/gold-dashboard/deepseek.key';
-$apiKey = trim((string)@file_get_contents($keyFile));
-if ($apiKey === '') {
-    respond(503, ['error' => 'no_key']);
-}
-
-// ── 6. per-id stampede lock, then re-check the cache ────────────────
-$lockFh = @fopen(CACHE_DIR . '/.ht_lock_' . $id, 'c');
-if ($lockFh !== false) {
-    flock($lockFh, LOCK_EX);   // blocks while a concurrent winner generates
-    $cached = @file_get_contents($cachePath);
-    if ($cached !== false && $cached !== '') {
-        flock($lockFh, LOCK_UN);
-        fclose($lockFh);
-        http_response_code(200);
-        header('Content-Type: application/json; charset=utf-8');
-        header('Cache-Control: max-age=3600');
-        echo $cached;
-        exit;
-    }
-}
-
-// ── 7. DeepSeek call (exact chat_json body shape) ───────────────────
-$userPayload = json_encode([
-    'headline'    => $hit['headline'] ?? '',
-    'description' => $hit['description'] ?? null,
-    'score'       => $hit['score'] ?? null,
-    'xau_score'   => $hit['score_by_currency']['XAU'] ?? null,
-    'keywords'    => $hit['keywords'] ?? [],
-    'currencies'  => $hit['currencies'] ?? [],
-], JSON_UNESCAPED_UNICODE);
-
-$body = json_encode([
-    'model'    => MODEL,
-    'messages' => [
-        ['role' => 'system', 'content' => SYSTEM_PROMPT],
-        ['role' => 'user',   'content' => $userPayload],
-    ],
-    'response_format' => ['type' => 'json_object'],
-    'temperature'     => 0,
-    'thinking'        => ['type' => 'disabled'],
-    'max_tokens'      => MAX_TOKENS,
-]);
-
-$ch = curl_init(API_URL);
-curl_setopt_array($ch, [
-    CURLOPT_POST           => true,
-    CURLOPT_POSTFIELDS     => $body,
-    CURLOPT_HTTPHEADER     => ['Content-Type: application/json',
-                               'Authorization: Bearer ' . $apiKey],
-    CURLOPT_RETURNTRANSFER => true,
-    CURLOPT_TIMEOUT        => TIMEOUT,
-]);
-$raw = curl_exec($ch);
-$httpCode = curl_getinfo($ch, CURLINFO_RESPONSE_CODE);
-curl_close($ch);
-
-// ── 8. parse (any failure → 502, never cached) ──────────────────────
-$summary = null;
-if ($raw !== false && $httpCode === 200) {
-    $resp = json_decode($raw, true);
-    $content = $resp['choices'][0]['message']['content'] ?? '';
-    $parsed = json_decode((string)$content, true);
-    if (is_array($parsed) && is_string($parsed['summary'] ?? null)
-            && trim($parsed['summary']) !== '') {
-        $summary = trim($parsed['summary']);
-        if (mb_strlen($summary) > SUMMARY_MAX_CHARS) {
-            $summary = mb_substr($summary, 0, SUMMARY_MAX_CHARS) . '…';
-        }
-    }
-}
-if ($lockFh !== false && $summary === null) {
-    flock($lockFh, LOCK_UN);
-    fclose($lockFh);
-}
-if ($summary === null) {
-    respond(502, ['error' => 'llm_failed']);
-}
-
-// ── 9. cache + respond ──────────────────────────────────────────────
-$out = json_encode(['summary' => $summary, 'generated_at' => gmdate('c')],
-                   JSON_UNESCAPED_SLASHES | JSON_UNESCAPED_UNICODE);
-if (!atomic_write($cachePath, $out)) {
-    // Degrade to per-click generation rather than a broken feature —
-    // the daily cap still bounds spend.
-    error_log('news_summary.php: cache write failed for ' . $cachePath);
-}
-if ($lockFh !== false) {
-    flock($lockFh, LOCK_UN);
-    fclose($lockFh);
-}
-http_response_code(200);
-header('Content-Type: application/json; charset=utf-8');
-header('Cache-Control: max-age=3600');
-echo $out;

telegram_bot.py

@@ -19,8 +19,6 @@ WebSocket:   Streams live ticks from the streaming provider — config.json
              EMA200) cached from the last boundary scan, triggers an
              immediate M15 scan and sends a Level Break Alert.
              5-minute cooldown prevents spam.
-News:        /news replies with the net XAU/USD news sentiment score
-             (display-only — news never gates signals).
 
 Owner lock:  All commands and buttons only respond to TELEGRAM_CHAT_ID
              (the owner). Everyone else gets a polite refusal — the bot
@@ -71,8 +69,6 @@ from app_config import (
     telegram_channel_id,
 )
 from providers import BaseDataProvider, make_data_provider
-from llm_client import get_llm_client
-from news_engine import FundamentalContext, NewsEngine
 from pair_signals import PAIR_PRODUCERS
 from signal_aggregator import SignalAggregator, SignalReport
 
@@ -104,12 +100,6 @@ logging.basicConfig(level=getattr(logging, _APP.log_level, logging.INFO),
                     force=True)
 logging.getLogger("httpx").setLevel(logging.WARNING)
 logging.getLogger("httpcore").setLevel(logging.WARNING)
-# Never let DEBUG runs write credentials into the 30-day rotating log:
-# the websockets client logger emits the connect URI (Twelve Data puts
-# the API key in the query string) and frame contents (Tiingo's auth
-# frame carries the key) — 2026-08-04 audit, finding K47.
-logging.getLogger("websockets").setLevel(logging.INFO)
-logging.getLogger("websockets.client").setLevel(logging.INFO)
 logger = logging.getLogger("telegram_bot")
 
 TOKEN = telegram_token()
@@ -607,45 +597,6 @@ def _commit_alert(key: str, report: SignalReport) -> None:
         _last_alert_setup.pop(key, None)
 
 
-# Per-key locks serializing the _should_alert → send → _commit_alert
-# window. The auto-scan loop and a concurrent breach handler share the
-# "XAU/USD_15min" dedup key; without the lock both pass _should_alert
-# while the other's send is still awaiting Telegram, and the same signal
-# goes out twice (2026-08-04 audit H36). Per-key rather than one global
-# lock so a slow send on one timeframe cannot head-of-line-block another
-# timeframe's alert. Keys are symbol × TF, so the dict stays tiny.
-_alert_locks: Dict[str, asyncio.Lock] = {}
-
-
-def _alert_lock(key: str) -> asyncio.Lock:
-    lock = _alert_locks.get(key)
-    if lock is None:
-        lock = _alert_locks[key] = asyncio.Lock()
-    return lock
-
-
-async def _send_alert(bot, chat_id: int, text: str) -> bool:
-    """Markdown alert send with a plain-text fallback on parse failures,
-    mirroring _safe_edit (2026-08-04 audit H35 — alert paths previously
-    dropped the message on a BadRequest with only a warning logged).
-    Returns True when a message reached the chat by either attempt."""
-    try:
-        await bot.send_message(chat_id, text, parse_mode=ParseMode.MARKDOWN)
-        return True
-    except BadRequest as exc:
-        logger.warning(f"Markdown alert failed for chat {chat_id} ({exc}); "
-                       f"retrying as plain text")
-        try:
-            await bot.send_message(chat_id, text)
-            return True
-        except Exception as exc2:
-            logger.warning(f"Failed to send alert to chat {chat_id}: {exc2}")
-            return False
-    except Exception as exc:
-        logger.warning(f"Failed to send alert to chat {chat_id}: {exc}")
-        return False
-
-
 # ── Candle-boundary scheduler ────────────────────────────────────────
 
 _autoscan_chats: Set[int] = set()
@@ -674,11 +625,8 @@ def _seconds_until_boundary(interval_minutes: int, buffer: int = 10) -> float:
     elapsed_seconds = elapsed_minutes * 60 + now.second + now.microsecond / 1e6
     interval_seconds = interval_minutes * 60
     remaining = interval_seconds - elapsed_seconds + buffer
-    # NOTE: remaining > buffer always (elapsed < interval by construction);
+    if remaining <= buffer:
-    # a call at the exact boundary instant targets the NEXT boundary —
+        remaining += interval_seconds
-    # steady-state loops wake at boundary+buffer, so only a process
-    # started in the boundary microsecond skips one scan (2026-08-04
-    # audit I42; the old `remaining <= buffer` guard was unreachable).
     return remaining
 
 
@@ -693,13 +641,7 @@ async def _scan_timeframe(
     reports: List[SignalReport] = []
     for i, result in enumerate(results):
         if isinstance(result, Exception):
-            # gather(return_exceptions=True) means there is no active
+            logger.exception(f"Auto-scan failed for {SCAN_PAIRS[i]} {tf.value}: {result}")
-            # exception here — logger.exception would log "NoneType: None"
-            # instead of the real stack; pass the instance explicitly.
-            logger.error(
-                f"Auto-scan failed for {SCAN_PAIRS[i]} {tf.value}: {result}",
-                exc_info=result,
-            )
         else:
             reports.append(result)
 
@@ -708,51 +650,36 @@ async def _scan_timeframe(
 
     _update_level_cache(reports)
 
-    candidates = [
+    actionable = [
         r for r in reports
         if r.final_signal != SignalStrength.NEUTRAL   # same gate as breach path
         and _passes_quality(r.signal_quality)
+        and _should_alert(f"{r.symbol}_{tf.value}", r)
     ]
-    for report in candidates:
+    sent_ok: Set[int] = set()
-        key = f"{report.symbol}_{tf.value}"
+    for chat_id in _signal_targets():
-        # The lock spans check → send → commit so a concurrent breach
+        for i, report in enumerate(actionable):
-        # handler on the same key cannot double-alert (H36).
-        async with _alert_lock(key):
-            if not _should_alert(key, report):
-                continue
             detail = format_signal_message(report)
-            delivered = False
+            try:
-            for chat_id in _signal_targets():
+                await bot.send_message(chat_id, detail, parse_mode=ParseMode.MARKDOWN)
-                if await _send_alert(bot, chat_id, detail):
+                sent_ok.add(i)
-                    delivered = True
+            except Exception as exc:
-            # Commit only when the alert reached at least one target;
+                logger.warning(f"Failed to send signal for chat {chat_id}: {exc}")
-            # a fully failed send stays eligible for the next scan.
+    # Commit fingerprints only for alerts that reached at least one target;
-            if delivered:
+    # a fully failed send stays eligible for the next scan.
-                _commit_alert(key, report)
+    for i, report in enumerate(actionable):
+        if i in sent_ok:
+            _commit_alert(f"{report.symbol}_{tf.value}", report)
 
 
 # ── WebSocket level-breach monitor ──────────────────────────────────
 
-# Level cache keyed symbol → timeframe → {level_name: price}, with a
+_level_cache: Dict[str, Dict[str, float]] = {}
-# matching per-TF write timestamp. Keyed per TF (2026-08-04 audit H34)
+_level_cache_ts: Dict[str, float] = {}
-# because every scan path writes here — the M5/M15/H1/H4 boundary loops,
-# the breach rescan, the manual Scan-All button — and a single flat dict
-# meant the last writer wiped the other timeframes' swing/EMA levels
-# within minutes. Breach checks scan the union of all fresh TF entries.
-_level_cache: Dict[str, Dict[str, Dict[str, float]]] = {}
-_level_cache_ts: Dict[str, Dict[str, float]] = {}
 _LEVEL_CACHE_TTL = 3600
 _last_prices: Dict[str, float] = {}
 _last_price_ts: Dict[str, float] = {}
 _scan_cooldowns: Dict[str, float] = {}
-# Breaches are edge-triggered (only the crossing tick fires), so one that
-# lands while the symbol is inside the rescan cooldown would otherwise be
-# dropped forever (H33b). Park it here; the tick loop fires it once the
-# cooldown expires. symbol → (price, level_name, recorded_at).
-_pending_breach: Dict[str, tuple] = {}
-# A failed breach rescan shrinks the just-set cooldown to this retry
-# window instead of silencing the symbol for the full cooldown (H33a).
-_BREACH_RETRY_SECONDS = 30
 
 _cfg = PlatformConfig()
 _WS_COOLDOWN = _cfg.ws_cooldown_seconds
@@ -761,8 +688,6 @@ _breach_tasks: Set[asyncio.Task] = set()
 
 
 def _update_level_cache(reports: List[SignalReport]) -> None:
-    """Each report updates only its own timeframe's slot — an M5 scan can
-    no longer overwrite the H1/H4 swing levels (H34)."""
     now = time.time()
     for r in reports:
         if r.primary_analysis and r.primary_analysis.key_levels:
@@ -771,31 +696,26 @@ def _update_level_cache(reports: List[SignalReport]) -> None:
                 if k in _LEVEL_KEYS
             }
             if levels:
-                tf = r.primary_analysis.timeframe
+                _level_cache[r.symbol] = levels
-                _level_cache.setdefault(r.symbol, {})[tf] = levels
+                _level_cache_ts[r.symbol] = now
-                _level_cache_ts.setdefault(r.symbol, {})[tf] = now
 
 
 def _check_level_breach(symbol: str, price: float) -> Optional[str]:
-    """Scan the union of every fresh timeframe's levels; returns
+    levels = _level_cache.get(symbol)
-    '<tf> <level_name>' (e.g. '15min swing_high_1') on a crossing so the
+    if not levels:
-    alert label says which timeframe's level broke."""
+        return None
-    per_tf = _level_cache.get(symbol)
+
-    if not per_tf:
+    cache_age = time.time() - _level_cache_ts.get(symbol, 0)
+    if cache_age > _LEVEL_CACHE_TTL:
         return None
 
     last = _last_prices.get(symbol)
     if last is None:
         return None
 
-    now = time.time()
+    for name, level in levels.items():
-    ts_map = _level_cache_ts.get(symbol, {})
+        if (last < level <= price) or (last > level >= price):
-    for tf, levels in per_tf.items():
+            return name
-        if now - ts_map.get(tf, 0) > _LEVEL_CACHE_TTL:
-            continue   # stale TF entry — skip, fresher TFs still count
-        for name, level in levels.items():
-            if (last < level <= price) or (last > level >= price):
-                return f"{tf} {name}"
 
     return None
 
@@ -810,17 +730,9 @@ def _cleanup_cooldowns() -> None:
 async def _handle_breach(bot, pair_symbol: str, price: float, level_name: str) -> None:
     now = time.time()
     if now - _scan_cooldowns.get(pair_symbol, 0) < _WS_COOLDOWN:
-        # Park the crossing instead of dropping it — price holding beyond
-        # the level never re-fires the edge trigger, so without this the
-        # breach would be lost forever (H33b). The tick loop fires it
-        # once the cooldown expires; a newer crossing overwrites it.
-        _pending_breach[pair_symbol] = (price, level_name, now)
         return
 
-    # Early-set on purpose: this is what stops concurrent breach tasks
-    # from stacking rescans. The failure path below rolls it back.
     _scan_cooldowns[pair_symbol] = now
-    _pending_breach.pop(pair_symbol, None)   # this crossing is being handled
     display = PAIR_INFO.get(pair_symbol, {}).get("display", pair_symbol)
     label = level_name.replace("_", " ").title()
     logger.info(f"Level breach: {display} crossed {label} at {price} — rescanning M15")
@@ -830,42 +742,29 @@ async def _handle_breach(bot, pair_symbol: str, price: float, level_name: str) -
         _update_level_cache([report])
     except Exception:
         logger.exception(f"Breach scan failed for {display}")
-        # Shrink the just-set cooldown to a short retry window so a
-        # transient fetch error doesn't silence the symbol for the full
-        # cooldown (H33a); the floor bounds the retry rate while the
-        # provider stays down. Re-park the breach so the tick loop
-        # retries it (setdefault: a newer crossing recorded meanwhile
-        # wins over this stale one).
-        retry = min(_BREACH_RETRY_SECONDS, _WS_COOLDOWN)
-        _scan_cooldowns[pair_symbol] = now - _WS_COOLDOWN + retry
-        _pending_breach.setdefault(pair_symbol, (price, level_name, now))
         return
 
     # Alert only when the rescan produced an actionable, changed signal —
     # same gates as the auto-scanner, so a breach can't spam the channel.
     if (report.final_signal == SignalStrength.NEUTRAL
-            or not _passes_quality(report.signal_quality)):
+            or not _passes_quality(report.signal_quality)
+            or not _should_alert(f"{pair_symbol}_{Timeframe.M15.value}", report)):
         return
 
-    key = f"{pair_symbol}_{Timeframe.M15.value}"
+    header = (
-    # Lock spans check → send → commit; without it this handler and an
+        f"⚡ *LEVEL BREAK* — {_escape_md(display)} crossed "
-    # in-flight M15 auto-scan both pass _should_alert and double-alert
+        f"*{_escape_md(label)}* @ `{_fmt_price(price)}`\n\n"
-    # the same signal (H36).
+    )
-    async with _alert_lock(key):
+    sent = False
-        if not _should_alert(key, report):
+    for chat_id in _signal_targets():
-            return
-
-        header = (
-            f"⚡ *LEVEL BREAK* — {_escape_md(display)} crossed "
-            f"*{_escape_md(label)}* @ `{_fmt_price(price)}`\n\n"
-        )
         detail = header + format_signal_message(report)
-        sent = False
+        try:
-        for chat_id in _signal_targets():
+            await bot.send_message(chat_id, detail, parse_mode=ParseMode.MARKDOWN)
-            if await _send_alert(bot, chat_id, detail):
+            sent = True
-                sent = True
+        except Exception as exc:
-        if sent:
+            logger.warning(f"Failed to send breach alert to chat {chat_id}: {exc}")
-            _commit_alert(key, report)
+    if sent:
+        _commit_alert(f"{pair_symbol}_{Timeframe.M15.value}", report)
 
 
 def _breach_task_done(task: asyncio.Task) -> None:
@@ -897,16 +796,13 @@ async def _ws_monitor(bot) -> None:
 
     backoff = 1
     while _feed_active():
-        got_tick = False
         try:
-            async for pair_symbol, price in dp.iter_ticks(SCAN_PAIRS):
+            def _on_connect() -> None:
-                if not got_tick:
+                nonlocal backoff
-                    # Reset backoff only once the connection actually
+                backoff = 1
-                    # produced data — resetting on connect let a
+
-                    # rejected key or accept-then-close server pin the
+            async for pair_symbol, price in dp.iter_ticks(
-                    # backoff at minimum forever (H38a/H38b).
+                    SCAN_PAIRS, on_connect=_on_connect):
-                    got_tick = True
-                    backoff = 1
                 if not _feed_active():
                     break
 
@@ -915,32 +811,11 @@ async def _ws_monitor(bot) -> None:
                 _last_price_ts[pair_symbol] = time.time()
 
                 if breached:
-                    # A fresh crossing supersedes any parked one.
-                    _pending_breach.pop(pair_symbol, None)
                     task = asyncio.create_task(
                         _handle_breach(bot, pair_symbol, price, breached)
                     )
                     _breach_tasks.add(task)
                     task.add_done_callback(_breach_task_done)
-                elif pair_symbol in _pending_breach:
-                    pend = _pending_breach[pair_symbol]
-                    if time.time() - pend[2] > _LEVEL_CACHE_TTL:
-                        # Ancient parked breach (feed gap/weekend) — an
-                        # alert about an hours-old crossing would only
-                        # mislead; the level cache it came from is stale
-                        # anyway.
-                        _pending_breach.pop(pair_symbol)
-                    elif (time.time() - _scan_cooldowns.get(pair_symbol, 0)
-                            >= _WS_COOLDOWN):
-                        # Crossing parked during the cooldown: fire it now
-                        # that the window expired — the edge trigger alone
-                        # would have dropped it forever (H33b).
-                        _pending_breach.pop(pair_symbol)
-                        task = asyncio.create_task(
-                            _handle_breach(bot, pair_symbol, pend[0], pend[1])
-                        )
-                        _breach_tasks.add(task)
-                        task.add_done_callback(_breach_task_done)
 
                 _cleanup_cooldowns()
 
@@ -950,16 +825,6 @@ async def _ws_monitor(bot) -> None:
             logger.exception(f"WebSocket error, reconnecting in {backoff}s")
             await asyncio.sleep(backoff)
             backoff = min(backoff * 2, 60)
-        else:
-            # Clean server close: the async-for ended without raising.
-            # This must back off and log like the error path — silently
-            # reconnecting with zero delay hot-loops against a server
-            # that accepts and immediately closes (H38a).
-            if _feed_active():
-                logger.warning(
-                    f"WebSocket closed cleanly, reconnecting in {backoff}s")
-                await asyncio.sleep(backoff)
-                backoff = min(backoff * 2, 60)
 
     logger.info("WebSocket monitor stopped")
 
@@ -1002,25 +867,15 @@ _ws_task: Optional[asyncio.Task] = None
 
 def _start_scheduler(bot) -> None:
     global _ws_task
-    # Per-TF respawn (2026-08-04 audit H37): drop finished loop tasks and
+    alive = [t for t in _autoscan_tasks if not t.done()]
-    # recreate only the missing ones. The old all-or-nothing alive check
+    if not alive:
-    # left a single dead TF loop unmonitored forever while its siblings
+        _autoscan_tasks.clear()
-    # lived — and with the channel feed active, /autoscan never reaches
+        for tf, interval in _AUTOSCAN_LOOPS:
-    # _stop_scheduler, so nothing short of a process restart recovered it.
+            _autoscan_tasks.append(
-    _autoscan_tasks[:] = [t for t in _autoscan_tasks if not t.done()]
+                asyncio.create_task(_candle_loop(bot, tf, interval))
-    alive_names = {t.get_name() for t in _autoscan_tasks}
+            )
-    started = []
-    for tf, interval in _AUTOSCAN_LOOPS:
-        name = f"candle_loop_{tf.name}"
-        if name in alive_names:
-            continue
-        _autoscan_tasks.append(
-            asyncio.create_task(_candle_loop(bot, tf, interval), name=name)
-        )
-        started.append(tf.name)
-    if started:
         logger.info("Candle-boundary scheduler started (%s)",
-                    "/".join(started))
+                    "/".join(tf.name for tf, _ in _AUTOSCAN_LOOPS))
 
     if _ws_task is None or _ws_task.done():
         _ws_task = asyncio.create_task(_ws_monitor(bot))
@@ -1096,7 +951,6 @@ _HELP_TEXT = (
     "/scan — Quick scan menu\n"
     "/autoscan — Toggle auto-scan on/off\n"
     "/status — Bot status, WS connection, live prices\n"
-    "/news — Net news sentiment score for XAU/USD\n"
     "/cache — Show data cache age per pair/TF\n"
     "/clearcache — Wipe data cache, force fresh fetch\n"
     "/help — This message\n\n"
@@ -1115,87 +969,6 @@ async def cmd_help(update: Update, context: ContextTypes.DEFAULT_TYPE) -> None:
     await update.message.reply_text(_HELP_TEXT, parse_mode=ParseMode.MARKDOWN)
 
 
-def _fmt_age(ts: Optional[float]) -> str:
-    """Compact age like '45s', '12m', '3.2h' for status lines."""
-    if not ts:
-        return "?"
-    s = time.time() - ts
-    if s < 90:
-        return f"{s:.0f}s"
-    if s < 5400:
-        return f"{s / 60:.0f}m"
-    return f"{s / 3600:.1f}h"
-
-
-def _news_health_lines() -> List[str]:
-    """News / calendar / DeepSeek source health for /status.
-
-    Reads the class-level `NewsEngine._source_health` written by the
-    scan-path fetches plus the LLM client's health stamps — display
-    only, never triggers a fetch. ⚠️ marks a degraded dependency;
-    details live in the rotating log (transition-logged WARNINGs)."""
-    lines = ["\n*News & AI:*"]
-    # Snapshot: a scan thread may insert a source's FIRST entry while we
-    # iterate (dict-changed-size RuntimeError); entry mutation is safe.
-    for _ in range(3):
-        try:
-            sh = dict(NewsEngine._source_health)
-            break
-        except RuntimeError:
-            continue
-    else:
-        sh = {}
-    if not sh:
-        lines.append("  _no news cycle yet (fills after first scan)_")
-    else:
-        cal = sh.get("calendar")
-        ff = sh.get("calendar:forexfactory")
-        if cal:
-            if NewsEngine.source_is_failing(cal):
-                lines.append(f"  ⚠️ Calendar: unavailable — {cal['detail']}"
-                             f" ({_fmt_age(cal['err'])})")
-            elif NewsEngine.source_is_failing(ff):
-                lines.append(f"  ⚠️ Calendar: {cal['extra']} — ForexFactory"
-                             f" down: {ff['detail']} ({_fmt_age(ff['err'])})")
-            else:
-                lines.append(f"  ✅ Calendar: {cal['extra']}"
-                             f" ({_fmt_age(cal['ok'])} ago)")
-        heads = sh.get("headlines")
-        if heads:
-            failing = NewsEngine.source_is_failing(heads)
-            mark = "⚠️" if failing else "✅"
-            detail = f" — {heads['detail']}" if failing else ""
-            ts = heads["err"] if failing else heads["ok"]
-            lines.append(f"  {mark} Headlines: {heads['extra']}{detail}"
-                         f" ({_fmt_age(ts)} ago)")
-        for name, entry in sorted(sh.items()):
-            if name.startswith("rss:") and NewsEngine.source_is_failing(entry):
-                lines.append(f"  ⚠️ {name[4:]}: {entry['detail']}"
-                             f" ({_fmt_age(entry['err'])})")
-    client = get_llm_client()
-    if not client.available:
-        lines.append("  DeepSeek: off — keyword scoring only")
-    else:
-        # The client's counter only resets on the first call of a new
-        # UTC day — show 0 (not yesterday's total, or a phantom
-        # cap-exhausted warning) until that reset happens.
-        today = datetime.now(timezone.utc).date()
-        used = client._count if client._count_day == today else 0
-        cap = client._daily_cap
-        scored = len(NewsEngine._llm_sent_cache)
-        err = client.last_error
-        if used >= cap:
-            lines.append(f"  ⚠️ DeepSeek: daily cap {cap} exhausted —"
-                         " keyword fallback until UTC midnight")
-        elif err and err[0] > client.last_ok:
-            lines.append(f"  ⚠️ DeepSeek: last call failed ({err[1]},"
-                         f" {_fmt_age(err[0])}) — keyword fallback")
-        else:
-            lines.append(f"  ✅ DeepSeek: {used}/{cap} calls today,"
-                         f" {scored} headlines scored")
-    return lines
-
-
 def _status_text() -> str:
     lines = ["📊 *ForexBot2 Status*\n"]
 
@@ -1229,16 +1002,9 @@ def _status_text() -> str:
         lines.append("\n_No live prices yet (WS not streaming)_")
 
     cached_pairs = len(_level_cache)
-    cached_levels = sum(
+    cached_levels = sum(len(v) for v in _level_cache.values())
-        len(levels) for per_tf in _level_cache.values()
-        for levels in per_tf.values()
-    )
     if cached_levels:
-        all_ts = [
+        oldest_age = time.time() - min(_level_cache_ts.values()) if _level_cache_ts else 0
-            ts for per_tf in _level_cache_ts.values()
-            for ts in per_tf.values()
-        ]
-        oldest_age = time.time() - min(all_ts) if all_ts else 0
         lines.append(f"\n*Level Cache:* {cached_levels} levels across {cached_pairs} pairs (oldest {oldest_age:.0f}s)")
     else:
         lines.append("\n*Level Cache:* empty — breach alerts start after first candle scan")
@@ -1254,8 +1020,6 @@ def _status_text() -> str:
     if active_breach:
         lines.append(f"*Breach tasks:* {active_breach} running")
 
-    lines.extend(_news_health_lines())
-
     lines.append(f"\n*Alert quality:* {_alert_min_quality}+ "
                  f"| re-arm {_ALERT_REARM_SECONDS // 60}m")
     return "\n".join(lines)
@@ -1266,38 +1030,6 @@ async def cmd_status(update: Update, context: ContextTypes.DEFAULT_TYPE) -> None
     await update.message.reply_text(_status_text(), parse_mode=ParseMode.MARKDOWN)
 
 
-def _news_text(context: FundamentalContext, engine: NewsEngine) -> str:
-    net, desc = engine.get_pair_sentiment(XAUUSD, context)
-    if round(net, 2) == 0:
-        net = 0.0  # a tiny negative mean would render as "-0.00"
-    lines = [
-        "📰 *XAU/USD News Score*\n",
-        f"*Net score:* `{net:+.2f}` — {desc}",
-    ]
-    # A cold/failed feed yields 0.00 "Neutral" — flag it so a data outage
-    # can't be mistaken for a genuinely neutral read.
-    if not context.news_sentiment:
-        lines.append("_no scored headlines yet — score defaults to 0_")
-    return "\n".join(lines)
-
-
-def _news_report_sync() -> str:
-    engine = NewsEngine(PlatformConfig().news)
-    return _news_text(engine.get_fundamental_context(), engine)
-
-
-@_owner_only
-async def cmd_news(update: Update, context: ContextTypes.DEFAULT_TYPE) -> None:
-    msg = await update.message.reply_text("⏳ Fetching news…")
-    loop = asyncio.get_running_loop()
-    try:
-        text = await loop.run_in_executor(_executor, _news_report_sync)
-    except Exception as exc:
-        logger.exception("News fetch failed")
-        text = f"❌ News fetch failed: {_escape_md(str(exc))}"
-    await msg.edit_text(text, parse_mode=ParseMode.MARKDOWN)
-
-
 @_owner_only
 async def cmd_cache(update: Update, context: ContextTypes.DEFAULT_TYPE) -> None:
     dp = _get_shared_dp()
@@ -1513,7 +1245,6 @@ async def _post_init(app: Application) -> None:
         ("scan", "Quick scan buttons"),
         ("autoscan", "Toggle auto-scan on/off"),
         ("status", "Bot status, WS connection, live prices"),
-        ("news", "XAU/USD net news sentiment score"),
         ("cache", "Show data cache age per pair/TF"),
         ("clearcache", "Wipe data cache, force fresh fetch"),
         ("help", "All commands"),
@@ -1543,11 +1274,7 @@ async def _post_init(app: Application) -> None:
                 logger.error(f"Initial scan failed for {SCAN_PAIRS[i]}: {r}")
         if init_reports:
             _update_level_cache(init_reports)
-            n_levels = sum(
+            logger.info(f"Level cache populated: {sum(len(v) for v in _level_cache.values())} levels")
-                len(levels) for per_tf in _level_cache.values()
-                for levels in per_tf.values()
-            )
-            logger.info(f"Level cache populated: {n_levels} levels")
     except Exception:
         logger.exception("Initial scan failed — WS will wait for first candle scan")
 
@@ -1647,7 +1374,6 @@ def main() -> None:
     app.add_handler(CommandHandler("scan", cmd_scan))
     app.add_handler(CommandHandler("help", cmd_help))
     app.add_handler(CommandHandler("status", cmd_status))
-    app.add_handler(CommandHandler("news", cmd_news))
     app.add_handler(CommandHandler("autoscan", cmd_autoscan))
     app.add_handler(CommandHandler("cache", cmd_cache))
     app.add_handler(CommandHandler("clearcache", cmd_clearcache))