Cleanup: remove dead code, stale docs, and all non-main branches

- Delete back-compat shims data_provider.py / oanda_provider.py;
  test_all.py now imports providers.* directly (ForexDataProvider name
  retired; the two shim-only tests removed, 268 -> 266 tests)
- Delete backtest/research/calendar_store.py (no importer, no entry
  point); config.py audit comment reworded
- Delete executed plan doc (2026-07-21 centralized-config) and the two
  unbuildable guide PDFs; HTML guides remain the source of truth
- OANDA_MIGRATION_ROADMAP.md status header corrected: phases 1-2
  shipped 2026-07-28, feed is live on Oanda; phases 3-6 still unbuilt
- Branches deleted locally + on both remotes: oanda_demo (merged),
  web-dashboard, eurusd-modular, reversal-exit-alert (owner-approved
  permanent deletion); July stash dropped

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_019zR58p8ACW1oJ5QXVUZ5es

.gitignore

@@ -35,5 +35,3 @@ backtest/trailgrid5m/
 backtest/__pycache__/
 .env.*
 logs/
-backtest/research/commit_ladder_2026-08/
-backtest/research/llm_news_eval_2026-08/

CLAUDE.md

@@ -11,13 +11,10 @@ Every numeric threshold in `config.py` and `pair_signals/xauusd.py` was backtest
 ## Commands
 
 ```bash
-python test_all.py                      # QA suite: 374 tests, 23 modules, no network (mocked)
+python test_all.py                      # QA suite: 266 tests, 18 modules, no network (mocked)
-python main.py --pair XAUUSD --timeframe H1 --detail   # CLI signal report (--json for full dump,
+python main.py --pair XAUUSD --timeframe H1 --detail   # CLI signal report (--json for full dump)
-                                        #  --panel to run the advisory LLM agent panel on-demand)
 python telegram_bot.py                  # the bot (refuses to boot without credentials)
 python oanda_daemon.py                  # auto-execution daemon on Oanda practice account
-python dashboard_publisher.py --once    # web dashboard: one scan+publish cycle to $DASHBOARD_OUT_DIR
-                                        # (omit --once for the candle-close daemon; deploy/README-dashboard.md)
 
 # Backtests
 python backtest/gpu_backtest.py --pair XAUUSD --days 365
@@ -25,25 +22,22 @@ python backtest/gpu_backtest.py --pair XAUUSD --start 2025-01-01 --end 2025-06-0
     --primary-tf 15min --confirm-tfs 1h,4h
 # key flags: --spread --quality A+,A,B,C --strategies --workers --max-hold-bars
 #            --warmup-bars --cooldown-bars --no-cache --export-csv out.csv --detail
-#            --fixed-rr-tp N (research-only: hard TP at N×risk, trail off; verdict
-#            2026-08-05: NOT shipped — backtest/research/fixedtp_2026-08-05.md)
 ```
 
-**`test_all.py` is NOT pytest** — it uses a custom `@test("name")` decorator that runs each test immediately at import; don't grep for `def test_` or try `pytest test_all.py`. Run the whole file; it prints per-module results and exits non-zero on failure. There is no way to run a single test short of commenting modules out. Tests must stay green; every new gate/knob needs tests. When patching weekend logic in tests, patch `providers.base._is_forex_weekend` — but know there are THREE independent weekend implementations (`providers/base.py`, `signal_aggregator.py`, and an inline copy in `backtest/gpu_backtest.py:_is_xau_blocked_by_calendar`); patching the provider one does not reach the other two.
+**`test_all.py` is NOT pytest** — it uses a custom `@test("name")` decorator that runs each test immediately at import; don't grep for `def test_` or try `pytest test_all.py`. Run the whole file; it prints per-module results and exits non-zero on failure. There is no way to run a single test short of commenting modules out. Tests must stay green; every new gate/knob needs tests. When patching weekend logic in tests, patch `providers.base._is_forex_weekend`.
 
 Python 3.12; a `venv/` exists in-repo. Dependencies in `requirements.txt` (numpy, pandas, requests, twelvedata, python-telegram-bot ≥20, websockets).
 
 ## Configuration & credentials
 
-Precedence for every credential/deployment setting: **env var → config.json → refuse to run** (no hardcoded fallbacks). A gitignored repo-root `.env` is loaded at `app_config` import via `os.environ.setdefault`. `config.json` (gitignored; schema in `config.json.example`) holds secrets plus deployment settings: `data_providers.active` (the live-feed switch), `data_providers.streaming` (optional tick-stream provider for the level-breach monitor), `scanning.pairs`/`timeframes`, per-pair `blocked_hours_utc`, `log_level`. Key env vars: `TELEGRAM_TOKEN`, `TELEGRAM_CHAT_ID`, `TELEGRAM_CHANNEL_ID`, `TWELVE_DATA_API_KEY`, `TIINGO_API_KEY`, `OANDA_ACCOUNT_ID`/`OANDA_API_TOKEN`/`OANDA_BASE_URL`, `DEEPSEEK_API_KEY` (news LLM; the `deepseek` config.json block holds model/enabled/timeout/daily cap). The `agent_panel` config.json block (all-defaults = feature dark) controls the advisory LLM agent panel: `enabled`, per-surface `telegram`/`dashboard` toggles, `min_quality`, `max_panels_per_day`, `reuse_ttl_seconds`.
+Precedence for every credential/deployment setting: **env var → config.json → refuse to run** (no hardcoded fallbacks). A gitignored repo-root `.env` is loaded at `app_config` import via `os.environ.setdefault`. `config.json` (gitignored; schema in `config.json.example`) holds secrets plus deployment settings: `data_providers.active` (the live-feed switch), `scanning.pairs`/`timeframes`, per-pair `blocked_hours_utc`, `log_level`. Key env vars: `TELEGRAM_TOKEN`, `TELEGRAM_CHAT_ID`, `TELEGRAM_CHANNEL_ID`, `TWELVE_DATA_API_KEY`, `TIINGO_API_KEY`, `OANDA_ACCOUNT_ID`/`OANDA_API_TOKEN`/`OANDA_BASE_URL`.
 
 ## Architecture
 
 ### End-to-end pipeline (every scan)
 
 ```
-providers/<active>.fetch_ohlcv()       per TF: single-flight fetch, drop forming candle, validate/clean, cache
+providers/<active>.fetch_ohlcv()       per TF: throttle, fetch, drop forming candle, validate/clean, cache
-                                       (only Twelve Data rate-throttles; Oanda has no client throttle)
   → engine.TechnicalEngine.analyze()   per TF: 9 scored indicators + ATR context → regime → composite score
   → signal_aggregator.analyze_pair()   live entry point (weekend/day filters, fetch, staleness)
       └─ analyze_from_dataframes()     replayable core: MTF agreement → news → bonuses/penalties →
@@ -55,15 +49,13 @@ Signal currency everywhere is `SignalStrength` (−2..+2). DataFrames are UTC tz
 
 ### Layers
 
-- **`providers/`** — one class per provider subclassing `BaseDataProvider` (`providers/base.py`), which owns everything the pipeline depends on: TTL+boundary cache, forming-candle drop, freshness check, validate/clean, paginated `fetch_range` (feeds the backtester from any provider), optional tick streaming. Subclasses implement only `_fetch_raw` + symbol/interval maps. Registry + `make_data_provider(config, name=None)` in `providers/__init__.py`; config.json `data_providers.active` selects the live candle feed and the optional `data_providers.streaming` scalar selects a different provider for the tick stream. A test enforces that subclasses don't override shared base methods (`supports_streaming`/`iter_ticks` are the sanctioned streaming override points). Tiingo streams mid-price ticks (`wss://api.tiingo.com/fx`) — do not re-add spread display from its free FX feed (two-tier indicative quotes, removed 2026-07-23); the no-spread rule extends to the WS stream.
+- **`providers/`** — one class per provider subclassing `BaseDataProvider` (`providers/base.py`), which owns everything the pipeline depends on: TTL+boundary cache, forming-candle drop, freshness check, validate/clean, paginated `fetch_range` (feeds the backtester from any provider), optional tick streaming. Subclasses implement only `_fetch_raw` + symbol/interval maps. Registry + `make_data_provider` in `providers/__init__.py`; config.json `data_providers.active` selects the live feed. A test enforces that subclasses don't override shared base methods. Tiingo is candles-only — do not re-add spread display from its free FX feed (two-tier indicative quotes, removed 2026-07-23).
 - **`engine.py`** — `TechnicalEngine`: 9 scored indicators (ema_trend, rsi, macd, bollinger, adx, roc, keltner, divergence, candlestick) + ATR as context. Regime classification (trending_up/down, `trending` = mixed/direction-UNKNOWN, transitioning, ranging) drives per-indicator weight multipliers and regime-aware RSI reinterpretation. Compute order matters (ATR first; MACD always computed even when not a pillar — divergence needs the histogram). Private series ride on the result as `_underscore` attrs; scorers/formatters skip anything `_`-prefixed or weight-0.
-- **`signal_aggregator.py`** — MTF agreement, diminishing-decay penalty system, exhaustion hard block, quality grading, then an ordered chain of post-grade gates (session, XAU SELL quality/vol/hour, regime blocks, M5 noise gates), then three setup generators: XAU ATR (A+/A), generic swing-based, plus the M5 pullback (`XAU_PB`) and chart-pattern (`XAU_CP`, `chart_patterns.py`) entry paths. The PB/CP generators run AFTER the gate chain and force quality to "B"; the CP path deliberately accepts grade-F composites and re-checks only session/day/big-candle (via warning prefixes) plus its own SELL/vol/RSI gates — it does NOT re-check exhaustion or the M5 min-score/MTF gates. **This is validated behavior, not a bug** (2026-08-04 audit: 226/586 trades in the shipped M5 12mo book are XAU_CP, +38.9R, 104 of them exhaustion-flagged and net-positive) — changing it is a rule-1 revalidation event. `TradeSetup.strategy` selects trailing params; trail params are stamped on the setup from the *producing* config so per-TF overrides reach formatters.
+- **`signal_aggregator.py`** — MTF agreement, diminishing-decay penalty system, exhaustion hard block, quality grading, then an ordered chain of post-grade gates (session, XAU SELL quality/vol/hour, regime blocks, M5 noise gates), then three setup generators: XAU ATR (A+/A), generic swing-based, plus the M5 pullback (`XAU_PB`) and chart-pattern (`XAU_CP`, `chart_patterns.py`) entry paths. `TradeSetup.strategy` selects trailing params; trail params are stamped on the setup from the *producing* config so per-TF overrides reach formatters.
 - **`pair_signals/`** — per-pair config factories. **Always use the factory (`produce_signal` / `make_config`), never bare `PlatformConfig()`** — the factory applies the config.json overlay first, then the validated per-TF deltas (M5 hour-9 block, M15 trail arm 0.70, M15 SELL-hour block, etc.; evidence in the docstring). A test enforces factory ≡ inline-config equivalence. Confirmation map: M5→[M15,H1], M15→[M5,H1], H1→[M15,H4], H4→[H1,D1].
-- **`news_engine.py`** — Forex Factory calendar (MyFXBook RSS fallback) + RSS keyword sentiment with gold-inverse keywords. Class-level caches (fresh engine per scan). Optional DeepSeek enrichment via `llm_client.py` (requests-only, no SDK): batched headline sentiment overriding keyword scores, gold-direction on release surprises, and a market-summary blurb (`SignalReport.news_summary`) — **display/warning-only, keyword fallback on any failure, inert without a key**; prompts are static module constants (DeepSeek prefix-cache pricing) and requests send `thinking: disabled` (V4-Flash reasoning otherwise blows the timeout). By XAU-tuned default, event risk and sentiment **warn but do not gate signals** (`event_risk_caps_signals=False`, `sentiment_affects_signals=False`) — the LLM does not change that.
+- **`news_engine.py`** — Forex Factory calendar (MyFXBook RSS fallback) + RSS keyword sentiment with gold-inverse keywords. Class-level caches (fresh engine per scan). By XAU-tuned default, event risk and sentiment **warn but do not gate signals** (`event_risk_caps_signals=False`, `sentiment_affects_signals=False`).
-- **`agent_panel.py`** — ai-hedge-fund-style advisory panel: 4 DeepSeek persona calls (macro/technical/risk/contrarian, concurrent on a panel-private pool) + 1 synthesizer producing an **ADVISORY-ONLY** consensus (`AGREE/NEUTRAL/OPPOSE` + confidence + rationale) written onto `SignalReport.panel_*` fields. Attached **post-hoc** by consumers (`telegram_bot` alert path, `main --panel`, `dashboard_publisher`) — never runs inside `analyze_from_dataframes` (backtest parity, test-locked) and no gate reads its fields. Static prompts (prefix caching); spend bounded by an exact per-bar cache, a reuse TTL keyed (symbol, TF, direction) so persisting signals don't re-spend every bar, and a `max_panels_per_day` UTC budget (default 20 ≈ ≤100 requests) on top of the shared 500/day client cap. `<2` valid persona takes skips the synthesizer; any failure degrades to no panel, never a changed signal.
+- **`telegram_bot.py`** — owner-locked bot + subscriber channel; candle-boundary auto-scan loops (M5/M15/H1/H4, boundary+10 s), alert quality threshold + fingerprint dedup (committed only after a successful send), WebSocket level-breach monitor (auto-disabled when the active provider has `supports_streaming=False` — Oanda does not stream). Rotating log: `logs/telegram.log`, 3-h rotation × 240 backups.
-- **`telegram_bot.py`** — owner-locked bot + subscriber channel; candle-boundary auto-scan loops (M5/M15/H1/H4, boundary+10 s), alert quality threshold + fingerprint dedup (committed only after a successful send), 📰 LLM market-context line on alerts when available, WebSocket level-breach monitor fed by the streaming provider (`data_providers.streaming`, falling back to the active provider; auto-disabled when the resolved provider can't stream or has no API key — Oanda does not stream, Tiingo does). Rotating log: `logs/telegram.log`, 3-h rotation × 240 backups.
+- **`oanda_daemon.py` / `oanda_executor.py`** — practice-account auto-execution. Book-parity rules (see daemon docstring): no broker-side TP (trailing model — TP1/TP2 advisory), 5-bar per-TF cooldown, trail ratchets at bar close exactly once per bar. Accepted divergence: one position total.
-- **`oanda_daemon.py` / `oanda_executor.py`** — practice-account auto-execution. Book-parity rules (see daemon docstring): no broker-side TP (trailing model — TP1/TP2 advisory), 5-bar per-TF cooldown (wall-clock approximation, so it expires over weekends where the backtester would still count bars), trail ratchets at bar close once per *observed* bar (a missed/failed cycle skips those bars entirely — the intra-gap high/low never enters the peak). Accepted divergence: one position total; orders send `positionFill=OPEN_ONLY` so a state/broker divergence cancels the order instead of silently netting. State/broker re-sync runs at boot and after any suspicious fill (`_reconcile_positions`).
-- **`dashboard_publisher.py` / `static/dashboard.html`** — web dashboard as a **static publisher** (live LAMP server): standalone daemon scans at candle closes via `produce_signal` and atomically writes `snapshot.json`/`news.json`/`dashboard.html` to `$DASHBOARD_OUT_DIR` (Apache serves them; page is public by owner decision). **Hard independence rule: never imports, and is never imported by, `telegram_bot`/`oanda_daemon`** — confirmation map and boundary math are deliberate local copies, so a web-side failure can only mean a stale page. Due-TF classification uses the RAW wake time (flooring wake−buffer skips every H1/H4 scan — 2026-08-02 bug). News panel reads `NewsEngine.get_fundamental_context()` plus the additive `get_week_events()` accessor (display-only; no signal path calls it); as a separate process it has its own news caches and DeepSeek daily cap. News/calendar rows are clickable (detail dialog); headlines carry `id` = sha1(headline) in news.json, and exactly one on-demand endpoint exists: `static/news_summary.php` (installed by `install_static`), which serves a DeepSeek gold-impact summary per headline — whitelist-gated against news.json ids, cache-forever in `ai/`, own fail-closed daily cap (150/day, file-based, separate from the publisher's), key at `/etc/gold-dashboard/deepseek.key`, never reads repo config. Deployment checklist: `deploy/README-dashboard.md`.
 - **`backtest/gpu_backtest.py`** — bar-by-bar replay through the real `analyze_from_dataframes` (news off). Realism model (do not regress): next-bar-open entries re-anchored to the raw price path, round-trip `--spread` charged per trade, trade resolution on RAW bars (cleaning applies only to signal-side data), same-bar SL+TP tie → SL first, gap fills at bar open, trail ratchet at bar close only. Indicators are precomputed once per TF (CuPy GPU or NumPy fallback — check the `Backend:` log line); replay fans out over worker processes and the 5-bar cooldown is applied in an ordered pass afterwards, so parallel and sequential books are identical. Parquet cache in `backtest/cache/` keyed by provider/symbol/TF/window. `backtest/*.csv` are gitignored pinned trade books from past campaigns — used for manual before/after comparison. `backtest/instructions_for_backtest.txt` lists what every backtest report must record.
 
 ## Rules for making changes (read before editing signal logic)
@@ -78,13 +70,6 @@ Signal currency everywhere is `SignalStrength` (−2..+2). DataFrames are UTC tz
 8. Don't add config fields nothing reads; don't disable indicators in production paths (`disabled_indicators` is for experiments only).
 9. EUR/USD was removed 2026-07-16 after a no-edge verdict (cost-to-volatility; last commit with EUR support: `7d3fd95`). Do not re-add without new data. BTC/ETH work lives on unmerged branches only.
 
-## 2026-08-04 QA audit — owner decisions ledger
-
-- **Weekend calendar: FIXED (owner-approved 2026-08-04).** All three `_is_forex_weekend` copies are NY-anchored (Fri 17:00 → Sun 18:00 ET; during US DST that is Fri 21:00 UTC — measured on 5yr of Oanda bars). Kills the weekly false-"STALE DATA/feed lagging" hours. Book impact, fully accounted: M15 byte-identical; M5 12mo dropped exactly ONE trade — a phantom whose signal fired on the final summer-Friday bar (close 21:00 UTC) and "filled" at Sunday's reopen 49 h later (−1.08R, live-impossible). M5 12mo reference at HEAD: 555 trades, +98.64R PF≈1.50 @ $0.65 (was 556/+97.56/1.48). A Module 21 test sweeps all three copies hourly across both DST transitions.
-- **Spread in exit-trigger detection (rule 6): owner DECLINED 2026-08-04.** Backtest stop triggers stay on raw mid High/Low; the measured optimism (~4–6% of total R: −3.55R @ $0.30, −4.24R @ $0.65 on the M15 12mo book) is accepted and books stay comparable. Re-open only as a conscious re-pin campaign.
-- **`FB2_EMA_LO`: HARDENED 2026-08-04 (delegated).** Research override for the EMA200-proximity band, default = tuned 0.5; an unparseable value now degrades to the default with a warning instead of crashing every scan. Documented here — it is a knob that exists.
-- **H4 seeding divergence: parked.** Live H4 fetches only 380 bars vs the backtest's unbounded history — ~2% of sampled H4 bars flip an indicator signal (composite Δ up to 12.3). M5/M15/H1 clean at their real fetch depths. Fold into any future H4 campaign; H4 remains "unproven for auto-execution".
-
 ## Current state notes (2026-07-30)
 
 - Oanda v20 is the live feed (`data_providers.active: "oanda"`), but the pinned backtest books are still Twelve-Data-pinned and the backtest cost assumption is $0.30 while measured real spreads are ~$0.65–0.72 — re-pin books on Oanda bars before treating feed-switch comparisons as valid (see Oanda/TD divergence study in memory).

OANDA_MIGRATION_ROADMAP.md

@@ -0,0 +1,85 @@
+# Full Migration to Oanda v20 as Market-Data Source — Roadmap
+
+**Status: partially executed (updated 2026-08-03).** Phases 1–2 shipped 2026-07-28 via the `providers/` package refactor and the live feed switched to Oanda (`data_providers.active: "oanda"`). Phases 3–6 remain unbuilt: the level-breach stream monitor is simply disabled for Oanda, and the pinned backtest books are still Twelve-Data-pinned at the $0.30 cost assumption. Owner decision 2026-07-24 stands: Twelve Data is a **permanent fallback** (no removal phase).
+
+## Context
+
+The daemon already executes trades on an Oanda practice account, and since 2026-07-28 the signal feed (Telegram bot, CLI, backtester) also runs on Oanda. Oanda's API is free with no metering (120 req/s vs TD's 55 credits/min), serves real bid/ask spreads, and matches the actual trading calendar. The 2026-07-24 divergence study (`backtest/research/oanda_td_divergence.py`) proved the switch is **not plumbing-only**: steady-state bars differ by median $0.69 (11% of M15 ATR), bar direction agrees only 89.9%, TD prints ~4,000 phantom Saturday bars/yr + 21:00 UTC rollover bars that contaminate Monday indicator windows, and Oanda's real spread (~$0.72 liquid / $0.84 rollover) is ~2.4× the $0.30 the validated books assume. So the migration = plumbing (Phases 1–3, inert) + shadow evidence (4) + book re-validation with owner gates (5) + reversible cutover (6). CLAUDE.md rules 1 (no tuned-value change without re-validation) and 4 (live == backtest parity) govern throughout.
+
+**Master sequencing rule:** Phases 1–3 land inert behind `data_providers.active == "twelve_data"`. Nothing changes live behavior until the Phase 6 config flip, which stays reversible forever (TD is a permanent fallback).
+
+## Phase 1 — Provider plumbing (inert)
+
+Files: `oanda_provider.py`, `data_provider.py`, `config.py`, `config.json.example`.
+
+1. **Constructor self-resolution**: `OandaDataProvider.__init__(config, api_token=None, base_url=None)` — defaults resolve via existing `app_config.oanda_api_token()` / `oanda_base_url()`; explicit args keep `oanda_daemon.py:180` working. Fail lazily at first fetch (mirror ForexDataProvider's `_td is None` pattern) so credential-less construction doesn't break the hermetic test header.
+2. **Registry**: add `"oanda": OandaDataProvider` to `PROVIDER_REGISTRY` (`data_provider.py:392`); import at bottom of `data_provider.py` (one-direction edge, no cycle — `oanda_provider` must never import from `data_provider`). Add `PROVIDER_NAME` class attr to both providers. Cutover config needs an `"oanda": {}` block under `data_providers` (creds stay in the existing top-level `oanda` section / env).
+3. **Dedupe shared constants**: move `CANDLE_SECONDS` / `BARS_PER_DAY` to `config.py`, alias as class attrs in both providers — all existing call sites (`signal_aggregator.py:18,217,640,942`, `telegram_bot.py:1062`, `gpu_backtest.py:975,1706`, tests) keep working unchanged. Also move `_is_forex_weekend` to `config.py` eventually (currently duplicated + sync-tested).
+4. **`_fetch_window(symbol, tf, start, end)`** on both providers — the backtester's common interface. Contract: raw closed bars labeled in [start, end], oldest-first, capitalized OHLCV, UTC index; empty DataFrame (not exception) for empty windows; caller keeps windows ≤ ~4500 bars. TD: thin wrapper over existing `_fetch_twelve_data(start_date=, end_date=)` (keep its exception behavior byte-identical). Oanda: new `from`/`to` RFC3339 request (`price=MBA, dailyAlignment=0, alignmentTimezone=Etc/UTC`); refactor `_fetch_oanda_candles`'s parsing body (complete-filter, mid extraction, spread cache) into a shared `_parse_candles()`. The from+count quirk (Oanda's `count` includes the boundary bar; see `backtest/research/fetch_oanda_m15.py`) does not apply to from/to windows. Add a no-op-by-construction `_drop_forming_candle` to OandaDataProvider (`gpu_backtest.py:1025` calls it unconditionally). No throttle needed for Oanda.
+5. **Tests (extend Module 15)**: registry selection; constructor self-resolution + explicit-args-win; `_fetch_window` both providers (mock `_session.get` / `_fetch_twelve_data`); CANDLE_SECONDS/BARS_PER_DAY cross-provider sync; provider-contract hasattr test covering everything callers reach for (`fetch_ohlcv, get_pair_info, clear_cache, _is_data_fresh, _has_latest_closed_bar, _validate_and_clean, _drop_forming_candle, _fetch_window, CANDLE_SECONDS, BARS_PER_DAY, _cache, _cache_timestamps, PROVIDER_NAME`); port Module 11's two cache/boundary tests (`test_all.py:2278,2313`) to Oanda variants patching `_fetch_oanda_candles`. Tests that swap `set_app_config` must restore `_TEST_APP` in `finally`.
+
+## Phase 2 — Backtester provider-agnosticism
+
+File: `backtest/gpu_backtest.py` (`fetch_history`, lines 943–1036).
+
+1. Line 984: `provider._fetch_twelve_data(...)` → `provider._fetch_window(...)`. Keep the backward-stepping chunk loop, `empty_streak` bail, cursor stepping byte-identical (rule 6 — this loop produced every pinned cache).
+2. Lines 975/1706: `ForexDataProvider.BARS_PER_DAY` → `provider.BARS_PER_DAY`.
+3. **Provider-namespaced cache keys** in `_cache_path` (line 938): TD keeps legacy names (pinned parquets stay loadable, needed for A/B), Oanda gets `oanda_` prefix (`oanda_XAU_USD_15min_....parquet`) — no collisions. Comment the asymmetry.
+4. `clean=True/False` raw-path semantics untouched.
+5. **Verification before merge**: one M15 12-month TD replay pre/post commit — trade books must be identical. **Alignment audit**: fetch one month of Oanda H4 + D1 via `_fetch_window`, assert label sets match TD's cached labels (H4 at 00/04/08 UTC, D1 at 00:00 UTC) — a mismatch would silently shift every MTF confirmation.
+6. Tests: `fetch_history` dispatch with a stub provider (concat/dedupe/trim, clean=False skips cleaning); `_cache_path` namespacing regression.
+
+## Phase 3 — Telegram bot: TD WebSocket → Oanda pricing stream
+
+File: `telegram_bot.py`; new resolver in `app_config.py`.
+
+1. New `oanda_stream_url()` resolver: env `OANDA_STREAM_URL` → config.json `oanda.stream_url` (add to `_KNOWN_OANDA`) → derived `oanda_base_url().replace("api-", "stream-", 1)`.
+2. Pure parser `_parse_stream_line(line) -> Optional[(pair_symbol, mid_price)]`: type==PRICE → mid from closeoutBid/Ask (fallback bids[0]/asks[0]); HEARTBEAT/garbage → None. `_STREAM_TO_PAIR` from `OandaDataProvider.OANDA_INSTRUMENT` (replaces TD-display-keyed `_WS_TO_PAIR`).
+3. Reader thread (no new deps): `requests.get(.../v3/accounts/{id}/pricing/stream?instruments=..., stream=True, timeout=(10,30))`, iterate `iter_lines(chunk_size=1)` — **chunk_size=1 is load-bearing** (urllib3 buffering vs 5s server heartbeats); push lines to asyncio queue via `loop.call_soon_threadsafe`; interrupt a blocked read with `resp.close()` from the async side.
+4. Async monitor `_oanda_stream_monitor`: dedicated thread (NOT the 4-worker scan executor), `await asyncio.wait_for(queue.get(), timeout=15)` as staleness watchdog (3× server heartbeat) → reconnect with existing exponential-backoff pattern. PRICE lines feed the identical existing body: `_check_level_breach` → `_handle_breach` → M15 rescan, 5-min cooldown. No client heartbeat.
+5. **Provider-gated selection** in `_start_scheduler` (line 889): `active_provider == "oanda"` → stream monitor, else existing `_ws_monitor`. This is the rollback hinge and stays permanently (TD fallback). `_stop_scheduler` must stop the reader thread. Generalize `/status` labels (`_status_text` iterates `SCAN_PAIRS`, not `_WS_TO_PAIR`).
+6. Tests: `_parse_stream_line` variants (PRICE/closeout, PRICE/arrays, HEARTBEAT, malformed, unknown instrument); `_STREAM_TO_PAIR` round-trip; monitor selection under both `active` values; watchdog-reconnect (structure one connection lifecycle as a testable inner coroutine).
+
+## Phase 4 — Shadow / parity evidence (3–4 weeks, no hot-path code)
+
+The daemon already runs the full signal stack on Oanda bars. Formalize evidence before any cutover:
+
+1. **Code parity (exact)**: replay the daemon's shadow window through the backtester on Oanda data (Phase 2 enables this); diff against daemon-logged decisions per (tf, bar_close_ts) — signal, quality, direction, entry/SL/TP. Acceptance: 100% modulo documented daemon divergences (single-position rule, quality gate). Any mismatch is a bug, not feed noise.
+2. **Feed divergence (measured, not zero-gated)**: `backtest/research/shadow_compare.py` parses `logs/telegram.log` + daemon log into per-scan rows: signal agreement rate, direction flips, grade drift, clustering by hour/weekday. Expect high-single-digit % divergence concentrated near grade thresholds and Monday sessions (TD Saturday/rollover bars). Deliverable: written note attributing every divergence class.
+3. **Operational telemetry**: boundary grace-retry frequency (is Oanda's just-closed bar present at boundary+10s?), `get_spread()` time series (live input to the Phase 5 cost decision), stream disconnect counts, Monday-open indicator snapshots TD-vs-Oanda.
+4. **Owner note**: at cutover the data feed host family should match the execution host (practice vs live pricing can micro-differ).
+
+## Phase 5 — Re-validation campaign (rule 1, owner gates)
+
+Fetch Oanda history (M5/M15/H1/H4/D1, 2020-07 → present; Oanda XAU reaches ~2005). Run matrix per primary TF (M5/M15/H1, current factory configs): 12-mo train/val halves, 5-yr, OOS 2024-06→2025-03 — each at `--spread 0.30` (pure feed-effect A/B vs pinned TD books), `0.70` (honest Oanda cost), `0.85` (rollover stress). Caution: Jul–Nov 2025 is provider-sensitive (weekly offsets −$6..+$18.5) — lean on 5-yr + OOS for decisions. Acceptance unchanged: positive both halves, PF ≥ 1.3, trailing exits, next-open fills, OOS positive.
+
+**Owner decision gates (never silent):**
+1. **Canonical cost** — (a) `round_trip_cost=0.70` everywhere (moves the live TP1 spread gate $9→$21 — a strategy re-tune); (b) `default_spread=0.70` for backtest PnL but retune `min_tp_cost_multiple` 30→~13 to preserve the validated $9 gate; (c) keep $0.30 with documented optimism (only if $0.70 books pass anyway). Full acceptance matrix at chosen values.
+2. **Degradation handling** per failing TF: re-run tuning grids on Oanda data / drop TF from `scanning.timeframes` / keep Oanda execution-only (status quo).
+3. **Re-pin**: passing books become the new pinned CSVs + factory-docstring evidence; CLAUDE.md tuned-value annotations update in the cutover commit, not before.
+
+Files (per decisions): `pair_signals/xauusd.py`, `config.py` (`PAIR_INFO.default_spread`, `SignalConfig.round_trip_cost`/`min_tp_cost_multiple`), `test_all.py` tuned-constant assertions, CLAUDE.md.
+
+## Phase 6 — Cutover (reversible forever; no TD removal)
+
+1. config.json: `data_providers.active: "oanda"` + `"oanda": {}` block; restart bot. Everything flips through two seams: `make_data_provider` (bot shared dp, CLI, backtester) and the Phase 3 monitor selector. Any Phase 5 tuned-value changes land in this commit with their validation records.
+2. **Bake ≥1 week** (full weekly cycle incl. Monday open): grace-retry rate, stream stability, alert plausibility vs daemon, `/cache` freshness.
+3. **Rollback = flip `active` back** — TD code, key, caches, and WS monitor all stay maintained indefinitely (owner decision). Dual-provider posture: cross-provider sync tests (Phase 1) keep both honest; TD subscription/key must stay valid to remain a real fallback.
+4. Docs: CLAUDE.md (pipeline header, data_provider section, env-var table + `OANDA_STREAM_URL`, backtester cache note, comparability note: "pre-Oanda books are TD-feed books"), `docs/` guides if they name the data source.
+
+## Key risks
+
+1. **Cost-gate coupling**: naively setting `round_trip_cost=0.70` silently moves the live $9 TP1 gate to $21 — owner gate 5.1 exists precisely for this.
+2. **Cutover before re-pinning** — feed divergence is material (11% of ATR, 10% direction flips); sequencing rule prevents it.
+3. **Hermetic test header** — `_TEST_APP` pins twelve_data; provider-swapping tests must restore it or later modules fail mysteriously.
+4. **Stream liveness triad** — `chunk_size=1`, `resp.close()` interruption, 15s watchdog.
+5. **H4/D1 alignment** — audit before trusting any Oanda backtest.
+6. **Monday/rollover divergence class** — attribute in Phase 4 evidence; do not "fix".
+
+## Verification (per phase)
+
+- Phases 1–3: `python test_all.py` green (235 + new); live behavior provably unchanged (TD active).
+- Phase 2: pre/post TD book identity diff; Oanda H4/D1 alignment audit.
+- Phase 3: soak a second bot instance with `active=oanda` scratch config; observe stream reconnects in log.
+- Phase 5: full acceptance matrix, owner sign-offs recorded in factory docstrings.
+- Phase 6: 1-week bake checklist above; rollback drill (flip back, confirm TD scan cycle) once during bake.

agent_panel.py

@@ -1,582 +0,0 @@
-"""
-ForexBot2 — LLM agent panel (advisory consensus)
-=================================================
-ai-hedge-fund-style persona panel for XAU/USD signals: four DeepSeek
-personas (macro, technical, risk, contrarian) each review a finished
-SignalReport from their own angle, then a synthesizer chair weighs the
-takes and issues a consensus verdict (AGREE / NEUTRAL / OPPOSE with
-confidence and a short rationale).
-
-Two invariants, both load-bearing (CLAUDE.md rules 1 and 4):
-
-- ADVISORY / DISPLAY-ONLY. The verdict is written onto SignalReport
-  fields that no gate and no scoring path ever reads. Any failure —
-  LLM down, cap exhausted, garbage reply — degrades to "no panel",
-  never to a changed or blocked signal.
-- NEVER runs inside ``analyze_from_dataframes``. The backtester replays
-  that function; consumers (telegram_bot, main --panel,
-  dashboard_publisher) attach the panel POST-HOC via ``attach_panel``,
-  so replayed books can never see a panel field. Backtest parity holds
-  by construction.
-
-Cost model: every full panel spends 5 requests (4 personas + 1
-synthesizer) against the shared DeepSeek daily cap (llm_client). Two
-layers bound the spend on top of that cap:
-
-- ``max_panels_per_day`` (config, default 20) — a UTC-day budget of
-  panel *attempts*, mirroring the client's request cap.
-- A reuse TTL (default 3600 s) keyed on (symbol, timeframe, direction):
-  a persisting signal re-serves its cached verdict across bars instead
-  of re-spending. Without this, one sticky M5 dashboard signal would
-  burn 5 calls per 5-minute bar — 1,440/day against the 500 cap.
-
-Prompts are STATIC module constants (DeepSeek prefix-cache pricing —
-see news_engine.py); all variable content travels in the user payload.
-"""
-
-import datetime as _dt
-import logging
-import threading
-import time
-from collections import OrderedDict
-from concurrent.futures import ThreadPoolExecutor, wait as _fwait
-from dataclasses import dataclass, field
-from typing import TYPE_CHECKING, Any, Dict, List, Optional, Tuple
-
-from app_config import get_app_config
-from llm_client import DEFAULT_TIMEOUT, get_llm_client
-
-if TYPE_CHECKING:  # type hints only — keep the coupling one-way
-    from news_engine import FundamentalContext
-    from signal_aggregator import SignalReport
-
-logger = logging.getLogger(__name__)
-
-DEFAULT_MIN_QUALITY = "B"
-DEFAULT_MAX_PANELS_PER_DAY = 20
-DEFAULT_REUSE_TTL_SECONDS = 3600.0
-
-# Quality ranks for the min_quality gate. Deliberately a local copy —
-# telegram_bot's ranking is private and dashboard_publisher must not
-# import telegram_bot (hard independence rule).
-_QUALITY_RANK = {"A+": 4, "A": 3, "B": 2, "C": 1}
-
-
-# ── DeepSeek prompts ─────────────────────────────────────────────────
-# STATIC constants on purpose: DeepSeek prices a repeated byte-identical
-# prompt prefix at the cache-hit rate (~50x cheaper), so nothing here may
-# be interpolated or timestamped. All variable content travels in the
-# user payload. Panel output is display-only — it never gates signals.
-
-_GOLD_DOCTRINE = """\
-Gold doctrine: hawkish Fed / rate hikes / rising real yields / strong
-dollar are gold-bearish; easing / rate cuts / safe-haven flows /
-geopolitical stress / weak dollar are gold-bullish.
-"""
-
-_TAKE_CONTRACT = """\
-Your review is advisory research on an already-generated signal; you are
-one voice on a panel, not the decision maker.
-Hard rules: use ONLY the data in the JSON you receive — do not invent
-numbers, events, or price levels. If the evidence your role needs is
-thin or missing, answer "neutral" with confidence below 30, unless your
-role's instructions say otherwise.
-Confidence scale (0-100): 80-100 strong one-sided evidence; 50-79 solid
-case with some doubt; 30-49 mixed evidence; below 30 little basis to
-judge.
-Write the note in plain everyday English for someone who has never
-traded: short simple sentences, no trading jargon, no acronyms, no
-abbreviations. Say "profit versus loss", not "RR"; "the 4-hour chart",
-not "H4"; describe what an indicator MEANS ("the price has climbed
-unusually fast"), never its name or number.
-Reply with ONLY JSON:
-{"stance": "support"|"oppose"|"neutral", "confidence": <0-100>,
- "note": "<max 25 words>"}
-"stance" says whether YOUR analysis supports the proposed trade
-direction, opposes it, or is genuinely mixed; "confidence" is how sure
-you are of your stance; "note" is your single most important point.
-"""
-
-_MACRO_SYSTEM = """\
-You are the macro analyst on a gold (XAU/USD) trade-review panel. You
-receive JSON: a proposed trade brief ("brief": direction, timeframe,
-quality, price, regime) plus the current fundamental picture — recent
-"headlines", today's high-impact "events" (released: true means the
-number is out), a news "sentiment" score for gold, and an optional
-"news_summary" blurb.
-Judge ONE thing: does the macro backdrop support the proposed direction?
-Weigh released surprises over stale headlines; an imminent unreleased
-high-impact event is a reason for lower confidence, not a direction.
-Stance rules:
-- support: the released data and news flow clearly push gold in the
-  proposed direction.
-- oppose: they clearly push gold the other way.
-- neutral: the picture is mixed or stale — or there are no meaningful
-  headlines or events to judge (score that low confidence).
-""" + _GOLD_DOCTRINE + """\
-Example: brief direction BUY, headlines dominated by hawkish Fed
-repricing and a hot CPI print -> {"stance": "oppose", "confidence": 70,
-"note": "Inflation is running hot, so interest rates should stay high — that usually pushes gold down."}
-Example: brief direction BUY, headlines led by an escalating conflict
-and a soft jobs report -> {"stance": "support", "confidence": 75,
-"note": "Worries about war and a cooling economy usually send money into gold."}
-""" + _TAKE_CONTRACT
-
-_TECH_SYSTEM = """\
-You are the technical analyst on a gold (XAU/USD) trade-review panel.
-You receive JSON: a proposed trade brief ("brief") plus the technical
-evidence — per-"indicators" signal/confidence, multi-timeframe
-"mtf" signals and scores, "key_levels" versus the current price, and an
-"exhaustion" count of consecutive same-direction bars flagged by the
-engine.
-Judge ONE thing: is the technical picture internally coherent for the
-proposed direction? Look for indicators fighting each other, the signal
-fighting the regime (e.g. a BUY in trending_down), higher timeframes
-disagreeing, price pressed into a key level, or an exhausted move being
-chased.
-Stance rules:
-- support: the indicators, higher timeframes, and regime mostly agree
-  with the proposed direction and none of the conflicts above appear.
-- oppose: one or more of those conflicts is present and serious.
-- neutral: the evidence genuinely cuts both ways, or too much of it is
-  missing to judge (score that low confidence).
-Example: brief direction SELL in regime trending_up with H1 and H4 both
-BUY -> {"stance": "oppose", "confidence": 80,
-"note": "This bets on gold falling while the bigger picture still points up."}
-Example: brief direction BUY in regime trending_up with M15 and H1 both
-BUY and exhaustion 0 -> {"stance": "support", "confidence": 70,
-"note": "Every way of looking at the chart points up, so buying goes with the flow."}
-""" + _TAKE_CONTRACT
-
-_RISK_SYSTEM = """\
-You are the risk manager on a gold (XAU/USD) trade-review panel. You
-receive JSON: a proposed trade brief ("brief") plus the risk facts —
-the trade "setup" (entry, stop_loss, take_profit_1/2, risk_reward,
-risk_dollars), the "volatility" regime, engine "warnings", the calendar
-"event_risk" level, and the assumed "spread".
-Judge ONE thing: is this setup's risk acceptable as constructed? Check
-reward-to-risk adequacy, stop distance versus the volatility regime
-(a tight stop in extreme vol dies to noise; a huge stop in low vol is
-oversized), spread as a fraction of the stop, and event landmines about
-to hit.
-Stance rules:
-- support: the likely reward clearly justifies the risk, the stop fits
-  the current volatility, and no event landmine is imminent.
-- oppose: reward too thin for the risk, a stop mismatched to the
-  volatility, spread eating a big share of the stop, an imminent
-  high-impact event — or no setup at all (a missing setup means the
-  signal graded too low for one; oppose unless the rest is
-  exceptional).
-- neutral: acceptable but unremarkable — nothing dangerous, nothing
-  attractive.
-Example: RR 1.4 with volatility extreme and a high-impact event due ->
-{"stance": "oppose", "confidence": 65,
-"note": "The possible profit is small for such wild price swings, and big economic news is due soon."}
-Example: RR 2.4 with normal volatility and no events due ->
-{"stance": "support", "confidence": 70,
-"note": "The possible profit is more than double the possible loss, and conditions look calm."}
-""" + _TAKE_CONTRACT
-
-_CONTRARIAN_SYSTEM = """\
-You are the contrarian on a gold (XAU/USD) trade-review panel — the
-designated skeptic. You receive JSON: a proposed trade brief ("brief"),
-the engine's own "bullish_reasons" and "bearish_reasons", its
-"warnings", and an optional "news_summary".
-Your job: construct the strongest honest case AGAINST the proposed
-trade, then score how damaging that case is. Use "oppose" only when the
-counter-case is genuinely strong, "neutral" when it is real but modest,
-and "support" when even the skeptic finds nothing that bites.
-""" + _GOLD_DOCTRINE + """\
-Example: a BUY whose own engine lists three bearish reasons and a
-counter-trend warning -> {"stance": "oppose", "confidence": 60,
-"note": "The system's own list of negatives looks stronger than its reason to buy."}
-""" + _TAKE_CONTRACT
-
-_SYNTH_SYSTEM = """\
-You are the chair of a gold (XAU/USD) trade-review panel. You receive
-JSON: the proposed trade "direction" and the panel's "takes" — one per
-analyst ("persona", "stance", "confidence", "note").
-Weigh the takes into ONE consensus on whether to act on the signal:
-"AGREE" (the panel backs the trade), "OPPOSE" (the panel is against it),
-or "NEUTRAL" (genuinely split or low conviction). Weigh a high-
-confidence oppose from any single analyst seriously — one strong veto
-can outweigh two lukewarm supports. Do not average blindly.
-Base the verdict and rationale ONLY on the takes given — do not add
-outside analysis.
-Confidence scale (0-100): 80-100 the panel clearly leans one way; 50-79
-a lean with real dissent; below 50 split or low conviction.
-Write the rationale in plain everyday English for someone who has never
-traded: short simple sentences, no trading jargon, no acronyms, no
-abbreviations.
-Example rationale: "Three of four analysts are against this: the
-possible profit is too small and the price has already climbed
-unusually fast. Only the news backdrop supports buying."
-Reply with ONLY JSON:
-{"verdict": "AGREE"|"NEUTRAL"|"OPPOSE", "confidence": <0-100>,
- "rationale": "<max 40 words>"}
-"""
-
-
-# ── Result types ─────────────────────────────────────────────────────
-
-@dataclass
-class PersonaTake:
-    persona: str        # "macro" | "technical" | "risk" | "contrarian"
-    stance: str         # "support" | "oppose" | "neutral"
-    confidence: float   # 0-100
-    note: str
-
-
-@dataclass
-class PanelVerdict:
-    verdict: str        # "AGREE" | "NEUTRAL" | "OPPOSE"
-    confidence: float   # 0-100
-    rationale: str
-    takes: List[PersonaTake] = field(default_factory=list)
-    created_ts: float = 0.0
-
-
-# ── Payload builders (pure functions; llm_client json-dumps sorted) ──
-
-def _direction(report: "SignalReport") -> Optional[str]:
-    """BUY/SELL the panel should review; None = nothing to review."""
-    if report.trade_setup is not None:
-        return report.trade_setup.direction
-    if report.final_signal.value > 0:
-        return "BUY"
-    if report.final_signal.value < 0:
-        return "SELL"
-    return None
-
-
-def _brief(report: "SignalReport") -> Dict[str, Any]:
-    analysis = report.primary_analysis
-    return {
-        "pair": report.display_name,
-        "timeframe": analysis.timeframe if analysis else "",
-        "direction": _direction(report),
-        "quality": report.signal_quality,
-        "score": round(report.final_score, 1),
-        "confidence": round(report.final_confidence),
-        "price": round(report.current_price, 2),
-        "market_regime": report.market_regime,
-        "volatility_regime": report.volatility_regime,
-        "mtf_agreement": report.mtf_agreement,
-    }
-
-
-def _macro_payload(report: "SignalReport",
-                   context: Optional["FundamentalContext"]) -> Dict[str, Any]:
-    headlines: List[str] = []
-    events: List[Dict[str, Any]] = []
-    if context is not None:
-        headlines = [h.headline for h in context.news_headlines[:10]]
-        events = [
-            {"event": e.event, "currency": e.currency,
-             "actual": e.actual, "forecast": e.forecast,
-             "released": not e.is_upcoming}
-            for e in context.events_today if e.is_high_impact
-        ][:8]
-    return {
-        "brief": _brief(report),
-        "headlines": headlines,
-        "events": events,
-        "sentiment": round(report.news_sentiment, 3),
-        "news_summary": report.news_summary,
-    }
-
-
-def _technical_payload(report: "SignalReport") -> Dict[str, Any]:
-    indicators: Dict[str, Dict[str, Any]] = {}
-    key_levels: Dict[str, float] = {}
-    analysis = report.primary_analysis
-    if analysis is not None:
-        # Same visibility filter as main.report_to_dict: private series
-        # ride as _-prefixed pseudo-indicators, weight-0 = context-only.
-        indicators = {
-            name: {"signal": ind.signal.name,
-                   "confidence": round(ind.confidence)}
-            for name, ind in analysis.indicators.items()
-            if not name.startswith("_") and ind.weight != 0
-        }
-        key_levels = {k: round(v, 2) for k, v in analysis.key_levels.items()}
-    return {
-        "brief": _brief(report),
-        "indicators": indicators,
-        "mtf": {tf: {"signal": sig.name,
-                     "score": round(report.mtf_scores.get(tf, 0.0), 1)}
-                for tf, sig in report.mtf_signals.items()},
-        "key_levels": key_levels,
-        "exhaustion": report.exhaustion_count,
-    }
-
-
-def _risk_payload(report: "SignalReport") -> Dict[str, Any]:
-    payload: Dict[str, Any] = {
-        "brief": _brief(report),
-        "volatility": report.volatility_regime,
-        "warnings": list(report.warnings),
-        "event_risk": report.event_risk,
-        "event_description": report.event_description,
-        "spread": round(report.spread_estimate, 2),
-    }
-    ts = report.trade_setup
-    if ts is not None:
-        payload["setup"] = {
-            "direction": ts.direction,
-            "entry": round(ts.entry_price, 2),
-            "stop_loss": round(ts.stop_loss, 2),
-            "take_profit_1": round(ts.take_profit_1, 2),
-            "take_profit_2": round(ts.take_profit_2, 2),
-            "risk_reward": round(ts.risk_reward, 2),
-            "risk_dollars": round(ts.risk_pips * ts.pip_value, 2),
-        }
-    return payload
-
-
-def _contrarian_payload(report: "SignalReport") -> Dict[str, Any]:
-    return {
-        "brief": _brief(report),
-        "bullish_reasons": report.bullish_reasons[:3],
-        "bearish_reasons": report.bearish_reasons[:3],
-        "warnings": list(report.warnings),
-        "news_summary": report.news_summary,
-    }
-
-
-# ── Orchestration ────────────────────────────────────────────────────
-
-# Panel-private pool: the 4 persona calls run concurrently WITHOUT
-# occupying telegram_bot's shared scan executor (which has 4 workers —
-# borrowing them would stall every other scan for the panel's duration).
-_pool = ThreadPoolExecutor(max_workers=4, thread_name_prefix="agent-panel")
-
-_VALID_STANCES = ("support", "oppose", "neutral")
-_VALID_VERDICTS = ("AGREE", "NEUTRAL", "OPPOSE")
-
-
-def _clamp_confidence(raw: Any) -> Optional[float]:
-    try:
-        return max(0.0, min(100.0, float(raw)))
-    except (TypeError, ValueError):
-        return None
-
-
-def _validate_take(persona: str, reply: Any) -> Optional[PersonaTake]:
-    if not isinstance(reply, dict):
-        return None
-    stance = reply.get("stance")
-    if stance not in _VALID_STANCES:
-        return None
-    confidence = _clamp_confidence(reply.get("confidence"))
-    if confidence is None:
-        return None
-    note = str(reply.get("note", "")).strip()[:160]
-    return PersonaTake(persona=persona, stance=stance,
-                       confidence=confidence, note=note)
-
-
-def _ask_persona(persona: str, system: str,
-                 payload: Dict[str, Any]) -> Optional[PersonaTake]:
-    # Client resolved at call time: tests install stubs via lc._client.
-    reply = get_llm_client().chat_json(system, payload)
-    return _validate_take(persona, reply)
-
-
-def run_panel(report: "SignalReport",
-              context: Optional["FundamentalContext"] = None,
-              ) -> Optional[PanelVerdict]:
-    """Run 4 personas + synthesizer; None = no verdict (fail-safe).
-
-    Requires ≥2 valid persona takes before spending the synthesizer
-    request; the synthesizer's reply is validated strictly and there is
-    deliberately NO programmatic fallback verdict — a panel that could
-    not really convene reports nothing rather than something invented.
-    """
-    direction = _direction(report)
-    if direction is None:
-        return None
-    jobs = (
-        ("macro", _MACRO_SYSTEM, _macro_payload(report, context)),
-        ("technical", _TECH_SYSTEM, _technical_payload(report)),
-        ("risk", _RISK_SYSTEM, _risk_payload(report)),
-        ("contrarian", _CONTRARIAN_SYSTEM, _contrarian_payload(report)),
-    )
-    futures = [_pool.submit(_ask_persona, name, system, payload)
-               for name, system, payload in jobs]
-    # The HTTP timeout inside the client bounds each call; the margin
-    # covers pool queueing. Stragglers past the wait are abandoned (their
-    # threads finish harmlessly) and simply don't vote.
-    client_timeout = getattr(get_llm_client(), "_timeout", DEFAULT_TIMEOUT)
-    done, _ = _fwait(futures, timeout=client_timeout + 8.0)
-    takes = [f.result() for f in done if not f.exception()]
-    takes = [t for t in takes if t is not None]
-    if len(takes) < 2:
-        logger.debug("agent panel: %d/4 persona takes — skipping synthesizer",
-                     len(takes))
-        return None
-    order = {"macro": 0, "technical": 1, "risk": 2, "contrarian": 3}
-    takes.sort(key=lambda t: order.get(t.persona, 9))
-    reply = get_llm_client().chat_json(_SYNTH_SYSTEM, {
-        "direction": direction,
-        "takes": [{"persona": t.persona, "stance": t.stance,
-                   "confidence": t.confidence, "note": t.note}
-                  for t in takes],
-    })
-    if not isinstance(reply, dict):
-        return None
-    verdict = reply.get("verdict")
-    confidence = _clamp_confidence(reply.get("confidence"))
-    if verdict not in _VALID_VERDICTS or confidence is None:
-        return None
-    rationale = str(reply.get("rationale", "")).strip()[:240]
-    return PanelVerdict(verdict=verdict, confidence=confidence,
-                        rationale=rationale, takes=takes,
-                        created_ts=time.time())
-
-
-# ── Cache, budget, attach ────────────────────────────────────────────
-
-_state_lock = threading.Lock()
-# Exact result per reviewed bar: repeated attaches on the same signal
-# (re-scan, multi-consumer) are free.
-_cache: "OrderedDict[Tuple[str, str, str, str], PanelVerdict]" = OrderedDict()
-_CACHE_MAX = 32
-# Reuse layer: a persisting signal (same symbol/tf/direction, newer bar)
-# re-serves its last verdict for reuse_ttl_seconds instead of re-spending
-# 5 requests per bar (see module docstring for the M5 dashboard math).
-_last_by_dir: Dict[Tuple[str, str, str], PanelVerdict] = {}
-# UTC-day budget of panel attempts (mirrors llm_client._under_cap).
-_panel_day: Optional[_dt.date] = None
-_panel_count: int = 0
-_warned_budget = False
-
-
-def reset_panel_state() -> None:
-    """Drop caches and counters (tests / config reloads)."""
-    global _panel_day, _panel_count, _warned_budget
-    with _state_lock:
-        _cache.clear()
-        _last_by_dir.clear()
-        _panel_day = None
-        _panel_count = 0
-        _warned_budget = False
-
-
-def panel_health() -> Dict[str, Any]:
-    """Read-only counters for status displays."""
-    settings = get_app_config().agent_panel_settings
-    cap = int(settings.get("max_panels_per_day", DEFAULT_MAX_PANELS_PER_DAY))
-    today = _dt.datetime.now(_dt.timezone.utc).date()
-    with _state_lock:
-        used = _panel_count if _panel_day == today else 0
-    return {"used_today": used, "cap": cap}
-
-
-def _under_panel_budget(cap: int) -> bool:
-    """Count one panel attempt against the UTC-day budget."""
-    global _panel_day, _panel_count, _warned_budget
-    today = _dt.datetime.now(_dt.timezone.utc).date()
-    with _state_lock:
-        if _panel_day != today:
-            _panel_day = today
-            _panel_count = 0
-            _warned_budget = False
-        if _panel_count >= cap:
-            if not _warned_budget:
-                _warned_budget = True
-                logger.warning(
-                    "agent panel: daily budget (%d) exhausted — no more "
-                    "panels until UTC midnight", cap)
-            return False
-        _panel_count += 1
-        return True
-
-
-def _write_fields(report: "SignalReport", v: PanelVerdict) -> None:
-    report.panel_verdict = v.verdict
-    report.panel_confidence = float(v.confidence)
-    report.panel_rationale = v.rationale
-    report.panel_takes = [
-        {"persona": t.persona, "stance": t.stance,
-         "confidence": t.confidence, "note": t.note} for t in v.takes]
-
-
-def attach_panel(report: "SignalReport",
-                 settings: Optional[Dict[str, Any]] = None,
-                 surface: str = "telegram",
-                 force: bool = False,
-                 context: Optional["FundamentalContext"] = None) -> bool:
-    """Best-effort panel attach; True iff panel fields were populated.
-
-    Advisory feature contract: NEVER raises — an exploding panel must
-    not cost an alert send or a dashboard publish. ``force`` (CLI
-    --panel) bypasses the enabled/surface/quality toggles but never the
-    daily budget or the shared LLM cap.
-    """
-    try:
-        if settings is None:
-            settings = get_app_config().agent_panel_settings
-        if not force:
-            if not settings.get("enabled", False):
-                return False
-            surface_on = {
-                "telegram": settings.get("telegram", True),
-                "dashboard": settings.get("dashboard", False),
-            }.get(surface, True)
-            if not surface_on:
-                return False
-            min_quality = settings.get("min_quality", DEFAULT_MIN_QUALITY)
-            if (_QUALITY_RANK.get(report.signal_quality, -1)
-                    < _QUALITY_RANK.get(min_quality, 2)):
-                return False
-        direction = _direction(report)
-        if direction is None:
-            return False
-
-        analysis = report.primary_analysis
-        tf = analysis.timeframe if analysis else ""
-        exact_key = (report.symbol, tf, report.timestamp, direction)
-        dir_key = (report.symbol, tf, direction)
-        ttl = float(settings.get("reuse_ttl_seconds",
-                                 DEFAULT_REUSE_TTL_SECONDS))
-        with _state_lock:
-            hit = _cache.get(exact_key)
-            if hit is None:
-                prior = _last_by_dir.get(dir_key)
-                if prior is not None and time.time() - prior.created_ts < ttl:
-                    hit = prior
-        if hit is not None:
-            _write_fields(report, hit)
-            return True
-
-        cap = int(settings.get("max_panels_per_day",
-                               DEFAULT_MAX_PANELS_PER_DAY))
-        if not _under_panel_budget(cap):
-            return False
-
-        if context is None:
-            # Best-effort; the news engine's class-level 600 s caches make
-            # this nearly free when warm in both bot and dashboard.
-            try:
-                from config import PlatformConfig
-                from news_engine import NewsEngine
-                context = NewsEngine(
-                    PlatformConfig().news).get_fundamental_context()
-            except Exception as e:
-                logger.debug("agent panel: news context unavailable: %s", e)
-                context = None
-
-        verdict = run_panel(report, context)
-        if verdict is None:
-            return False
-        _write_fields(report, verdict)
-        with _state_lock:
-            _cache[exact_key] = verdict
-            while len(_cache) > _CACHE_MAX:
-                _cache.popitem(last=False)
-            _last_by_dir[dir_key] = verdict
-        return True
-    except Exception:
-        logger.exception("agent panel: attach failed — signal unaffected")
-        return False

app_config.py

@@ -85,19 +85,11 @@ _VALID_LOG_LEVELS = ("DEBUG", "INFO", "WARNING", "ERROR", "CRITICAL")
 # AUTOSCAN_CONFIRMATIONS entries in telegram_bot.py).
 _SCANNABLE_TFS = ("M5", "M15", "H1", "H4")
 
-_KNOWN_TOP = {"telegram", "data_providers", "scanning", "pairs", "log_level",
+_KNOWN_TOP = {"telegram", "data_providers", "scanning", "pairs", "log_level", "oanda"}
-              "oanda", "deepseek", "agent_panel"}
 _KNOWN_OANDA = {
     "account_id", "api_token", "base_url", "lot_size",
     "quality_gate", "max_hold_bars",
 }
-_KNOWN_DEEPSEEK = {
-    "api_key", "model", "enabled", "timeout_seconds", "daily_request_cap",
-}
-_KNOWN_AGENT_PANEL = {
-    "enabled", "telegram", "dashboard", "min_quality",
-    "max_panels_per_day", "reuse_ttl_seconds",
-}
 _KNOWN_TELEGRAM = {"token", "chat_id", "channel_id", "alert_min_quality"}
 _KNOWN_SCANNING = {"pairs", "timeframes"}
 _KNOWN_PAIR_KEYS = {"blocked_hours_utc"}
@@ -114,15 +106,12 @@ class AppConfig:
     telegram_channel_id: str = ""          # "" = channel feed disabled
     alert_min_quality: str = "C"
     active_provider: str = "twelve_data"
-    streaming_provider: Optional[str] = None   # None = active provider
     provider_settings: Dict[str, Dict[str, Any]] = field(default_factory=dict)
     scan_pairs: List[str] = field(default_factory=lambda: [XAUUSD])
     scan_timeframes: List[Timeframe] = field(default_factory=lambda: [
         Timeframe.M5, Timeframe.M15, Timeframe.H1, Timeframe.H4])
     pair_overrides: Dict[str, Dict[str, Any]] = field(default_factory=dict)
     oanda_settings: Dict[str, Any] = field(default_factory=dict)
-    deepseek_settings: Dict[str, Any] = field(default_factory=dict)
-    agent_panel_settings: Dict[str, Any] = field(default_factory=dict)
     log_level: str = "INFO"
 
 
@@ -173,8 +162,7 @@ def load_app_config(path: str = CONFIG_PATH) -> AppConfig:
         app.active_provider = str(dp.get("active", app.active_provider))
         app.provider_settings = {}
         for name, block in dp.items():
-            # "active" and "streaming" are reserved scalars, not blocks.
+            if name == "active":
-            if name in ("active", "streaming"):
                 continue
             if not isinstance(block, dict):
                 raise AppConfigError(
@@ -184,17 +172,6 @@ def load_app_config(path: str = CONFIG_PATH) -> AppConfig:
             raise AppConfigError(
                 f'config.json: data_providers.active "{app.active_provider}" '
                 "has no matching provider block")
-        streaming = dp.get("streaming")
-        if streaming is not None:
-            if not isinstance(streaming, str):
-                raise AppConfigError(
-                    'config.json: data_providers.streaming must be a string '
-                    "(a provider name)")
-            if streaming not in app.provider_settings:
-                raise AppConfigError(
-                    f'config.json: data_providers.streaming "{streaming}" '
-                    "has no matching provider block")
-            app.streaming_provider = streaming
 
     scanning = raw.get("scanning", {})
     _check_keys(scanning, _KNOWN_SCANNING, '"scanning"')
@@ -258,54 +235,6 @@ def load_app_config(path: str = CONFIG_PATH) -> AppConfig:
                 "config.json: oanda.max_hold_bars must be a positive integer")
     app.oanda_settings = dict(oanda)
 
-    deepseek = raw.get("deepseek", {})
-    _check_keys(deepseek, _KNOWN_DEEPSEEK, '"deepseek"')
-    if "enabled" in deepseek and not isinstance(deepseek["enabled"], bool):
-        raise AppConfigError(
-            "config.json: deepseek.enabled must be true or false")
-    if "timeout_seconds" in deepseek:
-        ts = deepseek["timeout_seconds"]
-        if isinstance(ts, bool) or not isinstance(ts, (int, float)) or ts <= 0:
-            raise AppConfigError(
-                "config.json: deepseek.timeout_seconds must be a positive "
-                "number")
-    if "daily_request_cap" in deepseek:
-        cap = deepseek["daily_request_cap"]
-        if isinstance(cap, bool) or not isinstance(cap, int) or cap < 1:
-            raise AppConfigError(
-                "config.json: deepseek.daily_request_cap must be a positive "
-                "integer")
-    if "model" in deepseek:
-        if not isinstance(deepseek["model"], str) or not deepseek["model"]:
-            raise AppConfigError(
-                "config.json: deepseek.model must be a non-empty string")
-    app.deepseek_settings = dict(deepseek)
-
-    panel = raw.get("agent_panel", {})
-    _check_keys(panel, _KNOWN_AGENT_PANEL, '"agent_panel"')
-    for flag in ("enabled", "telegram", "dashboard"):
-        if flag in panel and not isinstance(panel[flag], bool):
-            raise AppConfigError(
-                f"config.json: agent_panel.{flag} must be true or false")
-    if "min_quality" in panel and panel["min_quality"] not in _VALID_QUALITIES:
-        raise AppConfigError(
-            f'config.json: agent_panel.min_quality "{panel["min_quality"]}" '
-            f"not in {_VALID_QUALITIES}")
-    if "max_panels_per_day" in panel:
-        mpd = panel["max_panels_per_day"]
-        if isinstance(mpd, bool) or not isinstance(mpd, int) or mpd < 1:
-            raise AppConfigError(
-                "config.json: agent_panel.max_panels_per_day must be a "
-                "positive integer")
-    if "reuse_ttl_seconds" in panel:
-        ttl = panel["reuse_ttl_seconds"]
-        if isinstance(ttl, bool) or not isinstance(ttl, (int, float)) \
-                or ttl < 0:
-            raise AppConfigError(
-                "config.json: agent_panel.reuse_ttl_seconds must be a "
-                "non-negative number")
-    app.agent_panel_settings = dict(panel)
-
     level = raw.get("log_level", app.log_level)
     if level not in _VALID_LOG_LEVELS:
         raise AppConfigError(
@@ -397,14 +326,6 @@ def oanda_base_url(app: Optional[AppConfig] = None) -> str:
         app.oanda_settings.get("base_url", "https://api-fxpractice.oanda.com"))
 
 
-# ── DeepSeek resolution helper (env → config.json → "") ──────────────
-
-def deepseek_api_key(app: Optional[AppConfig] = None) -> str:
-    app = app or get_app_config()
-    return os.environ.get("DEEPSEEK_API_KEY") or str(
-        app.deepseek_settings.get("api_key", ""))
-
-
 # ── PlatformConfig overlay ────────────────────────────────────────────
 
 def apply_pair_overrides(

backtest/gpu_backtest.py

@@ -40,13 +40,10 @@ from dataclasses import dataclass, field
 from datetime import datetime, timedelta, timezone
 from pathlib import Path
 from typing import Dict, List, Optional, Tuple
-from zoneinfo import ZoneInfo
 
 import numpy as np
 import pandas as pd
 
-_NY_TZ = ZoneInfo("America/New_York")
-
 # Ensure project root on sys.path so this script can be invoked from anywhere.
 PROJECT_ROOT = Path(__file__).resolve().parent.parent
 if str(PROJECT_ROOT) not in sys.path:
@@ -405,11 +402,6 @@ class GPUTechnicalEngine(TechnicalEngine):
         # cache key = timeframe.value (e.g. "15min", "1h", "4h")
         self.caches = caches
         self._current_tf: Optional[str] = None
-        # Cache-miss fallbacks recompute with the live engine's pandas
-        # math (different EWM seeding than the precomputed cache). That
-        # is a parity hazard, so it must never happen silently.
-        self.fallback_count = 0
-        self._fallback_warned = False
 
     def analyze(self, df: pd.DataFrame, symbol: str, timeframe: str) -> TechnicalAnalysis:
         # Latch the timeframe so per-indicator overrides know which cache to use.
@@ -427,23 +419,9 @@ class GPUTechnicalEngine(TechnicalEngine):
             return None, -1
         cache = self.caches.get(self._current_tf)
         if cache is None:
-            self._note_fallback(f"no cache for TF {self._current_tf!r}")
             return None, -1
         pos = cache.index_pos.get(df.index[-1])
-        if pos is None:
+        return (cache, pos) if pos is not None else (None, -1)
-            self._note_fallback(
-                f"{self._current_tf}: slice end {df.index[-1]} not in cache")
-            return None, -1
-        return cache, pos
-
-    def _note_fallback(self, why: str) -> None:
-        self.fallback_count += 1
-        if not self._fallback_warned:
-            self._fallback_warned = True
-            logger.warning(
-                "GPU cache miss → falling back to live-engine recompute "
-                "(%s). Further misses counted silently; check "
-                "fallback_count at end of run.", why)
 
     # ── EMA overrides ──────────────────────────────────────────────
 
@@ -551,11 +529,6 @@ class GPUTechnicalEngine(TechnicalEngine):
             signal = SignalStrength.NEUTRAL
             conf = 40
 
-        # Mirrors the live engine's ema_partial_neutral squash (parity).
-        if c.ema_partial_neutral and signal in (SignalStrength.BUY, SignalStrength.SELL):
-            signal = SignalStrength.NEUTRAL
-            conf = 40
-
         crossover_desc = ""
         # Live: `if len(ema_fast) >= 3:` then iloc[-2] — needs len ≥ 2 functionally,
         # but live's guard requires len ≥ 3 so we match that to stay byte-identical.
@@ -848,12 +821,7 @@ class GPUTechnicalEngine(TechnicalEngine):
         c = self.config
         n = c.roc_period
 
-        # Mirror the live guard (engine._compute_roc: len < n + 5 → skip,
+        if pos < n + 1:
-        # i.e. emit only from pos >= n + 4). Unreachable in practice —
-        # analyze() floors at 30 bars and n+4 < 30 for shipped periods —
-        # but keep the engines textually aligned so a future period/floor
-        # change can't open a silent divergence window.
-        if pos < n + 4:
             return
 
         roc_val = float(cache.roc[pos])
@@ -903,11 +871,6 @@ class GPUTechnicalEngine(TechnicalEngine):
             return super()._compute_keltner(df, result)
 
         c = self.config
-        # Mirror the live guard (engine._compute_keltner: len < max(kc_period,
-        # kc_atr_period) + 5 → skip). Unreachable below the 30-bar analyze()
-        # floor with shipped periods; kept aligned for the same reason as ROC.
-        if pos < max(c.kc_period, c.kc_atr_period) + 4:
-            return
         latest = float(df["Close"].iloc[-1])
         kc_upper_val = float(cache.kc_upper[pos])
         kc_lower_val = float(cache.kc_lower[pos])
@@ -1011,16 +974,6 @@ def fetch_history(
     path = _cache_path(symbol, timeframe, start, end, provider.name)
     if use_cache and path.exists():
         logger.info(f"Loading cached {symbol} {timeframe.value}: {path.name}")
-        # Files written before the raw-cache change (2026-07-08, commit
-        # 6a96ed6) contain CLEANED bars; loading one with clean=False
-        # would silently miss the spike bars the raw path exists to model.
-        # The key format predates versioning, so detect by mtime.
-        _RAW_CACHE_EPOCH = 1783468800.0   # 2026-07-08 00:00 UTC
-        if path.stat().st_mtime < _RAW_CACHE_EPOCH:
-            logger.error(
-                f"{path.name} predates the raw-cache format (2026-07-08) "
-                f"and likely contains CLEANED bars — delete it and refetch "
-                f"before trusting trade resolution on this window")
         df = pd.read_parquet(path)
         if df.index.tz is None:
             df.index = df.index.tz_localize("UTC")
@@ -1172,15 +1125,6 @@ def _simulate_trailing(
         reason = "TIMEOUT"
         for j in range(n):
             if np.isnan(h[j]):
-                # Data ends here: close at the LAST VALID bar, not at
-                # c[n-1] (which may itself be NaN — a NaN exit price used
-                # to poison r_multiple and every downstream metric).
-                if j == 0:
-                    exit_idx = 0
-                    reason = "NO_DATA"
-                else:
-                    exit_idx = j - 1
-                    exit_price = float(c[exit_idx])
                 break
             hit = (l[j] <= cur_sl) if buy else (h[j] >= cur_sl)
             if hit:
@@ -1202,15 +1146,8 @@ def _simulate_trailing(
                        else peak + dist * risk)
                 cur_sl = max(cur_sl, lvl) if buy else min(cur_sl, lvl)
 
-        if reason == "NO_DATA":
-            t.exit_reason = reason
-            continue
         if exit_price is None:
             exit_price = float(c[exit_idx])
-        if not np.isfinite(exit_price):
-            # Last-resort guard: never emit a NaN exit/R into the book.
-            t.exit_reason = "NO_DATA"
-            continue
         t.exit_reason = reason
         t.bars_held = exit_idx + 1
         t.exit_price = float(exit_price)
@@ -1383,12 +1320,6 @@ def simulate_trades(
             last_idx = int(n_valid) - 1
             t.exit_price = float(closes_arr[i, last_idx])
             t.exit_time = exit_times[i][last_idx]
-            if not np.isfinite(t.exit_price):
-                # NaN close on the final window bar (corrupt raw data)
-                # must not become a NaN R.
-                t.exit_reason = "NO_DATA"
-                t.exit_price = None
-                continue
 
         # Realized R from actual fill prices, net of round-trip spread.
         risk = abs(t.entry_price - t.sl)
@@ -1441,12 +1372,6 @@ class BacktestConfig:
     # Round-trip transaction cost in price units, charged on every trade
     # (XAU/USD retail spread ≈ $0.30). 0 disables.
     spread: float = 0.30
-    # RESEARCH (2026-08): >0 replaces the exit model with a hard TP at
-    # this multiple of risk from entry (2.0 = fixed 1:2 RR); trailing is
-    # disabled for ALL trades, including per-trade XAU_PB overrides.
-    # Entries/stops/gates are untouched, so the signal book is identical
-    # to the baseline — only the exit policy changes. Never shipped.
-    fixed_rr_tp: float = 0.0
     # Replay worker processes for the bar loop. 0 = auto (cpu_count - 2,
     # capped at 12), 1 = sequential in-process. Signal evaluation per bar
     # is pure (past data only), so it fans out across forked workers; the
@@ -1459,22 +1384,18 @@ class BacktestConfig:
 def _is_xau_blocked_by_calendar(ts: pd.Timestamp, cfg: PlatformConfig) -> Optional[str]:
     """Apply XAU weekend + allowed-weekday filter matching live `analyze_pair`.
 
-    Market closed Friday 17:00 -> Sunday 18:00 NEW YORK time (= Fri
+    Forex market closed: Friday 22:00 UTC -> Sunday 22:00 UTC (Sydney open).
-    21:00 UTC in US summer, 22:00 in winter — matches measured Oanda
+    Must match `_is_forex_weekend` in signal_aggregator.py exactly.
-    feed behavior; 2026-08-04 audit I43). Must match `_is_forex_weekend`
-    in signal_aggregator.py exactly (rule 4; Module 21 agreement test).
-    Note the weekday gates below stay on UTC weekday — they mirror the
-    live gates, which evaluate UTC weekdays.
     """
     xau = cfg.xau_strategy
     if xau.weekend_filter:
-        ny = ts.astimezone(_NY_TZ)
+        wd = ts.weekday()
-        wd = ny.weekday()
+        hour = ts.hour
-        if wd == 4 and ny.hour >= 17:
+        if wd == 4 and hour >= 22:
             return "WEEKEND_FILTER"
         if wd == 5:
             return "WEEKEND_FILTER"
-        if wd == 6 and ny.hour < 18:
+        if wd == 6 and hour < 22:
             return "WEEKEND_FILTER"
     if xau.allowed_weekdays and ts.weekday() not in xau.allowed_weekdays:
         return "DAY_FILTER"
@@ -1691,43 +1612,6 @@ def _reanchor_entries_to_raw(
         )
 
 
-def _apply_cooldown(
-    candidates: List[Tuple[int, Trade]], cooldown_bars: int
-) -> List[Tuple[int, Trade]]:
-    """Ordered cooldown pass over (bar_pos, Trade) candidates: drop any
-    candidate within `cooldown_bars` bars of the previously ACCEPTED
-    entry — exactly what a sequential replay loop would have skipped.
-    Candidates must already be sorted by position (stable merge from the
-    worker spans)."""
-    accepted: List[Tuple[int, Trade]] = []
-    last_entry_pos = -10_000
-    for pos, trade in candidates:
-        if pos - last_entry_pos < cooldown_bars:
-            continue
-        accepted.append((pos, trade))
-        last_entry_pos = pos
-    return accepted
-
-
-def _apply_fixed_rr_tp(trades: List[Trade], mult: float) -> None:
-    """RESEARCH: replace each trade's TP with a hard target at
-    `mult` × |entry − sl| and zero the per-trade trail params so the
-    XAU_PB overrides cannot resurrect trailing. Risk is measured off the
-    final (re-anchored) entry/SL, so call this after
-    `_reanchor_entries_to_raw`. The caller must also pass
-    trail_arm_r=0 to `simulate_trades` — dispatch is on the run-level arg."""
-    for t in trades:
-        risk = abs(t.entry_price - t.sl)
-        t.tp1 = (t.entry_price + mult * risk if t.direction == "BUY"
-                 else t.entry_price - mult * risk)
-        t.tp2 = t.tp1              # static branch never reads tp2; CSV hygiene
-        t.trail_arm_r = 0.0
-        t.trail_dist_r = 0.0
-        t.rr_planned = mult
-        if t.pip_value:
-            t.reward_pips = mult * risk / t.pip_value
-
-
 def run_backtest(
     bt_cfg: BacktestConfig,
     platform_cfg: Optional[PlatformConfig] = None,
@@ -1855,10 +1739,18 @@ def run_backtest(
         candidates.extend(cands)
     candidates.sort(key=lambda pc: pc[0])
 
-    # ── Phase 2: entry cooldown — the only sequential rule.
+    # ── Phase 2: entry cooldown — the only sequential rule. A candidate
-    accepted = _apply_cooldown(candidates, bt_cfg.cooldown_bars)
+    # inside the cooldown window of the previously ACCEPTED entry is
-    trades: List[Trade] = [t for _, t in accepted]
+    # dropped, exactly as the sequential loop skipped those bars.
-    signal_ts_list: List[pd.Timestamp] = [primary_df.index[p] for p, _ in accepted]
+    trades: List[Trade] = []
+    signal_ts_list: List[pd.Timestamp] = []
+    last_entry_pos = -10_000
+    for pos, trade in candidates:
+        if pos - last_entry_pos < bt_cfg.cooldown_bars:
+            continue
+        trades.append(trade)
+        signal_ts_list.append(primary_df.index[pos])
+        last_entry_pos = pos
 
     logger.info(
         f"Replay complete: {len(trades)} entries in "
@@ -1878,17 +1770,10 @@ def run_backtest(
         use_cache=bt_cfg.use_cache, clean=False,
     )
     _reanchor_entries_to_raw(trades, signal_ts_list, sim_df)
-    if bt_cfg.fixed_rr_tp > 0:
-        _apply_fixed_rr_tp(trades, bt_cfg.fixed_rr_tp)
-        logger.info(
-            f"RESEARCH override: fixed-RR TP = {bt_cfg.fixed_rr_tp} x risk; "
-            f"trailing exit DISABLED for all {len(trades)} trades (incl. XAU_PB)"
-        )
-    run_arm = 0.0 if bt_cfg.fixed_rr_tp > 0 else platform_cfg.signals.trail_arm_r
     t_sim = time.perf_counter()
     simulate_trades(trades, sim_df, max_bars=bt_cfg.max_hold_bars,
                     spread=bt_cfg.spread,
-                    trail_arm_r=run_arm,
+                    trail_arm_r=platform_cfg.signals.trail_arm_r,
                     trail_dist_r=platform_cfg.signals.trail_dist_r)
     logger.info(
         f"Trade simulation took {time.perf_counter() - t_sim:.2f}s "
@@ -1938,14 +1823,6 @@ def _summarize(rs: List[float]) -> Dict:
 
 def compute_metrics(trades: List[Trade]) -> Metrics:
     resolved = [t for t in trades if t.exit_reason and t.exit_reason != "NO_DATA"]
-    # Last line of defense: one non-finite R (NaN exit on corrupt data)
-    # would silently NaN every aggregate below. Exclude loudly.
-    bad = [t for t in resolved if not np.isfinite(t.r_multiple)]
-    if bad:
-        logger.error(
-            "compute_metrics: excluding %d trade(s) with non-finite R "
-            "(corrupt exit data) — first at %s", len(bad), bad[0].entry_time)
-        resolved = [t for t in resolved if np.isfinite(t.r_multiple)]
     if not resolved:
         return Metrics(0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, {}, {}, {})
 
@@ -2232,10 +2109,6 @@ def _parse_args(argv: Optional[List[str]] = None) -> argparse.Namespace:
                    help="Override signals.trail_arm_r for this run (research knob)")
     p.add_argument("--trail-dist", type=float, default=None,
                    help="Override signals.trail_dist_r for this run (research knob)")
-    p.add_argument("--fixed-rr-tp", type=float, default=None,
-                   help="RESEARCH: exit at a hard TP of MULT x risk (e.g. 2.0 = "
-                        "fixed 1:2 RR) instead of the trailing stop; "
-                        "entries/stops/gates unchanged.")
     p.add_argument("--quality", default="A+,A,B,C", help="Comma-separated allowed grades")
     p.add_argument("--strategies", default="",
                    help="Comma-separated strategy tags to include (XAU,XAU_PB,XAU_CP,GENERIC). Empty = all.")
@@ -2359,16 +2232,6 @@ def main(argv: Optional[List[str]] = None) -> int:
         # make_config; the fallback must match (live == backtest).
         from app_config import apply_pair_overrides  # noqa: E402
         apply_pair_overrides(platform_cfg)
-    if args.fixed_rr_tp is not None:
-        if args.fixed_rr_tp <= 0:
-            raise SystemExit("--fixed-rr-tp must be > 0")
-        if (args.trail_arm is not None or args.trail_dist is not None or
-                any(s.split("=", 1)[0].strip().startswith(("signals.trail",
-                                                           "signals.pb_trail"))
-                    for s in (args.set or []))):
-            raise SystemExit(
-                "--fixed-rr-tp conflicts with --trail-arm/--trail-dist/"
-                "--set signals.*trail* (the flag disables trailing entirely)")
     if args.swing_sr_lookback > 0:
         platform_cfg.technical.swing_sr_lookback = args.swing_sr_lookback
     if args.trail_arm is not None:
@@ -2415,7 +2278,6 @@ def main(argv: Optional[List[str]] = None) -> int:
         strategies_only=[s.strip() for s in args.strategies.split(",") if s.strip()],
         spread=spread,
         workers=args.workers,
-        fixed_rr_tp=args.fixed_rr_tp or 0.0,
     )
 
     logger.info(

backtest/research/commit_ladder.py

@@ -1,178 +0,0 @@
-"""Commit-ladder campaign driver (2026-08).
-
-Question: of all signal-code states shipped in the last two months, which
-configuration is the most accurate? Historical in-era backtest numbers are
-not comparable (the backtester's realism model changed on 07-08/07-16 and
-the data feed switched to Oanda on 07-28), so every historical
-configuration is recreated on TODAY's harness via --set overrides and
-replayed on the same pinned Oanda bars. Bug-fix commits stay on; the
-ladder compares shipped tuning decisions only.
-
-Usage:
-    python backtest/research/commit_ladder.py            # full matrix
-    python backtest/research/commit_ladder.py --tf 5min  # one timeframe
-    python backtest/research/commit_ladder.py --windows 12mo,oos
-
-Resumable: a run whose CSV already exists in the output dir is skipped.
-Outputs: backtest/research/commit_ladder_2026-08/<variant>_<window>.csv
-         + .log per run, summary.csv, and a printed ranking table.
-"""
-from __future__ import annotations
-
-import argparse
-import subprocess
-import sys
-from pathlib import Path
-
-import pandas as pd
-
-ROOT = Path(__file__).resolve().parents[2]
-BT = ROOT / "backtest" / "gpu_backtest.py"
-OUT = ROOT / "backtest" / "research" / "commit_ladder_2026-08"
-
-WINDOWS = {
-    "12mo": ("2025-07-30", "2026-07-30"),
-    "oos": ("2024-06-01", "2025-03-30"),
-    "5yr": ("2021-07-30", "2026-07-30"),
-}
-SPLIT_12MO = pd.Timestamp("2026-01-29", tz="UTC")  # book_diff.py's split
-
-# (variant, "commit date what-it-represents", [--set overrides])
-VARIANTS: dict[str, list[tuple[str, str, list[str]]]] = {
-    "5min": [
-        ("M5-A", "dad2432 07-09 gates+trailing+pullback",
-         ["xau_strategy.blocked_hours_utc=[0,1,2,10,11,13,16,21]",
-          "xau_strategy.cp_enabled=false",
-          "signals.trail_dist_r=0.75",
-          "xau_strategy.m5_min_abs_score=30",
-          "xau_strategy.m5_min_confirmation_score=10"]),
-        ("M5-B", "f3ba4d5 07-11 +hour-9 block",
-         ["xau_strategy.blocked_hours_utc=[0,1,2,10,11,13,16,21,9]",
-          "xau_strategy.cp_enabled=false",
-          "signals.trail_dist_r=0.75",
-          "xau_strategy.m5_min_abs_score=30",
-          "xau_strategy.m5_min_confirmation_score=10"]),
-        ("M5-C", "de6424e 07-16 +chart patterns",
-         ["xau_strategy.blocked_hours_utc=[0,1,2,10,11,13,16,21,9]",
-          "signals.trail_dist_r=0.75",
-          "xau_strategy.m5_min_abs_score=30",
-          "xau_strategy.m5_min_confirmation_score=10"]),
-        ("M5-D", "3ee79b5 07-21 +trail dist 0.85",
-         ["xau_strategy.blocked_hours_utc=[0,1,2,10,11,13,16,21,9]",
-          "xau_strategy.m5_min_abs_score=30",
-          "xau_strategy.m5_min_confirmation_score=10"]),
-        ("M5-E", "0396970 07-30 relaxation (HEAD)", []),
-    ],
-    "15min": [
-        ("M15-A", "987ec5b 07-09 pre-campaigns",
-         ["signals.tp_floor_rr=0.5",
-          "xau_strategy.blocked_regimes=[]",
-          "xau_strategy.sell_blocked_hours_utc=[]",
-          "signals.trail_arm_r=0.60"]),
-        ("M15-B", "f3ba4d5/f88e39f 07-11 1R floor + regime block",
-         ["xau_strategy.sell_blocked_hours_utc=[]",
-          "signals.trail_arm_r=0.60"]),
-        ("M15-C", "7d3fd95 07-16 +SELL hour 15/17 block",
-         ["signals.trail_arm_r=0.60"]),
-        ("M15-D", "bf8e5fb 07-21 trail arm 0.70 (HEAD)", []),
-    ],
-    "1h": [
-        ("H1-A", "pre-d95d31a 07-08 TP-tightening model",
-         ["signals.min_rr_ratio=0.4",
-          "signals.tp_tightening_factor=0.8"]),
-        ("H1-B", "HEAD", []),
-    ],
-}
-
-
-def run_one(tf: str, name: str, overrides: list[str], wkey: str,
-            spread: str | None = None) -> Path:
-    suffix = f"_s{spread}" if spread else ""
-    csv_path = OUT / f"{name}_{wkey}{suffix}.csv"
-    log_path = OUT / f"{name}_{wkey}{suffix}.log"
-    if csv_path.exists():
-        print(f"  [skip] {csv_path.name} exists")
-        return csv_path
-    start, end = WINDOWS[wkey]
-    cmd = [sys.executable, str(BT), "--pair", "XAUUSD",
-           "--primary-tf", tf, "--start", start, "--end", end,
-           "--export-csv", str(csv_path)]
-    if spread:
-        cmd += ["--spread", spread]
-    for ov in overrides:
-        cmd += ["--set", ov]
-    print(f"  [run ] {csv_path.name} ...", flush=True)
-    with open(log_path, "w") as lf:
-        rc = subprocess.run(cmd, stdout=lf, stderr=subprocess.STDOUT,
-                            cwd=ROOT).returncode
-    if rc != 0:
-        raise SystemExit(f"backtest failed (rc={rc}) — see {log_path}")
-    log = log_path.read_text()
-    n_ov = log.count("--set override")
-    if n_ov != len(overrides):
-        raise SystemExit(
-            f"{name}_{wkey}: expected {len(overrides)} --set overrides, "
-            f"log shows {n_ov} — see {log_path}")
-    if "Loading cached" not in log and wkey != "sanity":
-        print(f"  [warn] {log_path.name}: no 'Loading cached' line — "
-              f"data was fetched live for at least one timeframe")
-    return csv_path
-
-
-def metrics(csv_path: Path, wkey: str) -> dict:
-    df = pd.read_csv(csv_path, parse_dates=["entry_time"])
-    if df.empty:
-        return {"trades": 0, "total_r": 0.0, "wr": 0.0, "pf": 0.0,
-                "max_dd": 0.0, "h1_r": 0.0, "h2_r": 0.0, "years_pos": ""}
-    if df["entry_time"].dt.tz is None:
-        df["entry_time"] = df["entry_time"].dt.tz_localize("UTC")
-    r = df["r_multiple"]
-    pos, neg = r[r > 0].sum(), r[r < 0].sum()
-    cum = r.cumsum()
-    dd = (pd.concat([pd.Series([0.0]), cum]).cummax()[1:] - cum).max()
-    out = {
-        "trades": len(df),
-        "total_r": round(r.sum(), 1),
-        "wr": round((r > 0).mean() * 100, 1),
-        "pf": round(pos / abs(neg), 2) if neg else float("inf"),
-        "max_dd": round(dd, 1),
-        "h1_r": "", "h2_r": "", "years_pos": "",
-    }
-    if wkey == "12mo":
-        h1 = r[df["entry_time"] < SPLIT_12MO].sum()
-        out["h1_r"] = round(h1, 1)
-        out["h2_r"] = round(r.sum() - h1, 1)
-    if wkey == "5yr":
-        by_year = r.groupby(df["entry_time"].dt.year).sum()
-        out["years_pos"] = f"{(by_year > 0).sum()}/{len(by_year)}"
-    return out
-
-
-def main() -> None:
-    ap = argparse.ArgumentParser()
-    ap.add_argument("--tf", choices=list(VARIANTS), default=None)
-    ap.add_argument("--windows", default="12mo,oos,5yr")
-    args = ap.parse_args()
-    OUT.mkdir(parents=True, exist_ok=True)
-    wkeys = [w.strip() for w in args.windows.split(",")]
-    tfs = [args.tf] if args.tf else list(VARIANTS)
-
-    rows = []
-    for tf in tfs:
-        for name, label, overrides in VARIANTS[tf]:
-            for wkey in wkeys:
-                csv_path = run_one(tf, name, overrides, wkey)
-                m = metrics(csv_path, wkey)
-                rows.append({"tf": tf, "variant": name, "represents": label,
-                             "window": wkey, **m})
-                print(f"    {name} {wkey}: {m['trades']} trades, "
-                      f"{m['total_r']}R, PF {m['pf']}, WR {m['wr']}%")
-
-    summary = pd.DataFrame(rows)
-    summary.to_csv(OUT / "summary.csv", index=False)
-    print("\n=== SUMMARY (also in summary.csv) ===")
-    print(summary.to_string(index=False))
-
-
-if __name__ == "__main__":
-    main()

backtest/research/fixedtp_2026-08-05.md

@@ -1,123 +0,0 @@
-# Fixed-RR hard-TP vs shipped trailing exit — research campaign 2026-08-05
-
-**Verdict: DO NOT SHIP. The trailing exit stays.** The fixed 2R target beats the trail on
-total R in every 2024+ window but the edge is a trend-regime artifact: it inverts in the
-2021–2023 ranging years, doubles drawdown, cuts win rate 10–19 points, and doubles hold
-times (amplifying the accepted live one-position divergence). Details below.
-
-## Setup
-
-- Code: HEAD `fc5ce85` + the `--fixed-rr-tp` research flag (uncommitted at run time).
-  The flag replaces every trade's TP with `mult × |entry − sl|` AFTER re-anchoring and
-  disables trailing (run-level and per-trade PB params) — entries, stops, gates, and the
-  cooldown pass are untouched, so both arms replay **trade-identical books** (verified:
-  paired join on entry_time+direction matches 1:1 in every window). Only the exit policy
-  differs → per-trade paired comparison.
-- Provider: Oanda (cache-warm; keys `oanda_XAU_USD_{5min,15min,1h}_{20250603_20260804,
-  20240331_20250330,20210531_20260804}`). Charged cost `--spread 0.65` (real measured
-  round trip); $0.30 letter-cost books derived analytically as `r + 0.35/risk` and
-  verified exact against one real `--spread 0.30` run (max per-trade error 1.8e-15 —
-  spread never enters trigger detection, per the owner-declined 2026-08-04 decision).
-- Control sanity gate: fresh M5 12mo control reproduced the pinned HEAD reference
-  exactly — 555 trades, +98.64R, PF 1.483, maxDD −10.73R.
-- Commands (control = same without the flag):
-  `venv/bin/python backtest/gpu_backtest.py --pair XAUUSD --primary-tf {5min,15min}
-   --start 2025-08-04 --end 2026-08-04 --spread 0.65 [--fixed-rr-tp 2.0]
-   --export-csv backtest/fixedtp_*.csv`
-  OOS `--start 2024-06-01 --end 2025-03-30`; 5yr `--start 2021-08-01 --end 2026-08-04`.
-- Books pinned as `backtest/fixedtp_{m5,m15}_{12mo,oos,5yr}_{ctrl,rr20,...}.csv`
-  (per-trade indicator breakdown, agreement counts, regime, vol-regime, weekday,
-  MAE/MFE, bars-held — the full instructions_for_backtest record).
-
-## Headline (@ $0.65)
-
-| Window | Arm | n | R | PF | WR | maxDD | recov | medBars | exits |
-|---|---|---|---|---|---|---|---|---|---|
-| M5 12mo | trail | 555 | +98.64 | 1.483 | 52.4% | −10.7 | 9.2 | 28 | TRAIL 361 / SL 183 / TO 11 |
-| M5 12mo | 2R hard | 555 | +124.85 | 1.378 | 42.9% | −24.5 | 5.1 | 50 | SL 312 / TP1 210 / TO 33 |
-| M15 12mo | trail | 397 | +73.93 | 1.525 | 62.2% | −10.8 | 6.8 | 28 | TRAIL 255 / SL 135 / TO 7 |
-| M15 12mo | 2R hard | 397 | +95.95 | 1.431 | 44.3% | −11.2 | 8.6 | 39 | SL 215 / TP1 151 / TO 31 |
-| M5 OOS | trail | 86 | +9.45 | 1.332 | 53.5% | −8.2 | 1.2 | 70 | — |
-| M5 OOS | 2R hard | 86 | +18.83 | 1.407 | 45.3% | −9.7 | 1.9 | 124 | — |
-| M15 OOS | trail | 299 | +38.77 | 1.360 | 59.2% | −15.5 | 2.5 | 38 | — |
-| M15 OOS | 2R hard | 299 | +72.14 | 1.435 | 45.2% | −13.9 | 5.2 | 82 | — |
-
-Halves (12mo, split 2026-02-03): M5 rr20 +67.99/+56.86 (both positive; H2 PF 1.325 vs
-ctrl 1.503), M15 rr20 +73.74/+22.21 (both positive; ctrl H2 +7.23 — the known H2-flat).
-At $0.30 all deltas keep the same sign (e.g. M5 12mo +109.0 vs +135.2).
-
-So the **letter** of rule 1 passes for rr20 on both TFs. The 5yr decomposition is why it
-still must not ship.
-
-## Exit migration (paired, M5 12mo; same pattern everywhere)
-
-- SL→SL 183 trades: ΔR 0 (identical by construction — stops untouched).
-- TRAIL→TP1 207 trades: +248.9R → +424.5R (**+175.6R**) — winners the trail exits early.
-- TRAIL→SL 129 trades: +19.0R → −136.8R (**−155.7R**) — retraces the trail survives
-  become full stop-outs.
-- TRAIL→TIMEOUT 25: +7.4R. Net = +26.2R. The whole effect is these two big flows nearly
-  cancelling; which one wins is a regime property, not an exit-policy edge.
-- Trades with ctrl MFE ≥ 2R: 95; hard cap forfeits only +10.4R vs trail on M5 (the trail
-  already gives ~0.75R back from peak, so a 2R cap costs little *in a trend*).
-
-## Mini-grid (12mo @ $0.65) — the tell
-
-| mult | M5 R / PF / DD | M15 R / PF / DD |
-|---|---|---|
-| trail | +98.6 / 1.483 / −10.7 | +73.9 / 1.525 / −10.8 |
-| 1.0R | +52.5 / 1.205 / −12.8 | +67.1 / 1.410 / −7.3 |
-| 1.5R | +80.7 / 1.263 / −21.4 | +66.9 / 1.325 / −11.2 |
-| 2.0R | +124.9 / 1.378 / −24.5 | +96.0 / 1.431 / −11.2 |
-| 2.5R | +147.7 / 1.426 / −23.6 | +110.4 / 1.469 / −15.9 |
-| 3.0R | +155.8 / 1.432 / −21.6 | +135.8 / 1.562 / −17.5 |
-
-Monotone in the multiple on both TFs — "2.0" is not an optimum, it's a point on a curve
-that says *never take profit* (3R's M5 TIMEOUT bucket alone is +84.8R). That is the
-signature of a one-way trend sample, not of a superior exit rule. It also reconfirms the
-original finding that SHORT fixed targets (≤1.5R) are strictly bad (cf. RR-0.5 removal,
-`config.py:126-133`: +48R → +105R).
-
-## 5yr regime decomposition (2021-08-01 → 2026-08-04 @ $0.65) — the verdict
-
-M15 (real sample in every year): ctrl +127.97 PF 1.209 DD −35.8; rr20 +158.97 PF 1.170
-DD **−59.8**; rr30 +204.03 PF 1.203 DD **−98.3**.
-
-| Year | n | trail | 2R hard | 3R hard |
-|---|---|---|---|---|
-| 2021 | 88 | **+4.00** | −16.09 | −17.41 |
-| 2022 | 284 | −9.54 | −7.63 | −26.74 |
-| 2023 | 227 | **+7.79** | −1.71 | −5.81 |
-| 2024 | 345 | +40.44 | +64.98 | +89.14 |
-| 2025 | 405 | +49.33 | +69.39 | +104.42 |
-| 2026 | 229 | +35.95 | +50.04 | +60.43 |
-
-M5 5yr mirrors it (ctrl +128.4 PF 1.387 DD −12.8 vs rr20 +160.2 PF 1.306 DD −24.7;
-2022: −1.1 vs −5.2), though pre-2024 M5 samples are thin (12–60 trades/yr).
-
-The hard target's entire outperformance lives in the 2024–2026 gold trend. In ranging
-years it flips sign exactly where the trail stays positive (M15 2021, 2023), and the
-further the target, the worse the ranging-year bleed and the deeper the DD. The 12mo,
-halves, and OOS windows all sit inside the trend era, which is why the rule-1 letter
-passes — this is precisely the within-regime-split trap rule 1's OOS clause exists to
-catch, one level up.
-
-## Additional non-ship reasons
-
-- **Live divergence grows with hold time.** Median hold doubles (M5 28→50 bars; OOS
-  70→124). The daemon holds one position total; the backtester doesn't model that
-  blocking. Longer holds = more live signals skipped that the paper book banks.
-- **WR 52→43% (M5) / 62→44% (M15), max loss streak 6→11 (M15)** — materially harder to
-  operate and size.
-- Aggregate tables (session/day/regime/vol, streaks, quarterly equity, rolling-20 WR)
-  computed and consistent — no bucket reverses the conclusion; both arms' 12mo profit is
-  concentrated in Q1-2026 (+87.7 ctrl / +102.0 rr20).
-
-## Status
-
-- `--fixed-rr-tp` stays as a documented research-only flag (default off; conflicts with
-  `--trail-arm/--trail-dist/--set signals.*trail*` are hard errors). Module 22 tests
-  (5) pin the override math, TP fill + spread, SL-first tie, PB-trail-disable, and CLI
-  guards. Full suite: 346 green.
-- Re-open only if gold's regime turns and a *regime-conditional* exit (trail in range,
-  target in trend) is proposed with a pre-registered mechanism — a plain fixed ratio is
-  now a closed hypothesis at every multiple in {0.5, 1.0, 1.5, 2.0, 2.5, 3.0}.

backtest/research/llm_news_eval.py

@@ -1,697 +0,0 @@
-"""
-DeepSeek news-analysis validation harness — XAU/USD.
-====================================================
-Research-only (no production code changes). Answers: is the DeepSeek
-layer's news analysis *correct*?
-
-Part A (events): replays every high-impact USD release with a genuine
-surprise (2024-01 → 2026-07, from backtest/cache/calendar.parquet)
-through the REAL `NewsEngine._llm_event_direction` — real prompt, real
-payload, real validation — and scores its "gold bullish/bearish" calls
-against gold's actual move (Oanda M15 mids) at +15m/30m/1h/4h/24h.
-Baselines: mechanical surprise-sign doctrine, FF actualBetterWorse,
-always-bullish/bearish (2024-26 is a gold bull regime), coin flip.
-
-Part B (headlines): a curated golden set (llm_news_golden.csv) through
-the REAL `NewsEngine._llm_headline_scores` in production-sized batches,
-vs the keyword scorer it overrides. Includes repeat-batch (determinism)
-and shuffled-order (cross-contamination) probes.
-
-Implementation notes that keep the replay faithful to production:
-  * `_llm_event_direction`/`_llm_headline_scores` resolve the client via
-    `get_llm_client()`, so we seed the module singleton with a research
-    client (raised daily cap: 586 events > production cap 500).
-  * The event cache key omits the forecast, so a repeat `actual` would
-    silently re-serve a direction computed for a different forecast —
-    the cache is cleared before every call.
-  * A "neutral" reply and a hard failure both return None; they are
-    disambiguated by reading the event cache back after the call
-    (absent = failure, "" = neutral, "bullish:/bearish: …" = call).
-  * Raw request/response bytes are captured with a requests response
-    hook — the request-building code path is untouched.
-
-Usage (from repo root):
-  venv/bin/python backtest/research/llm_news_eval.py events --limit 5
-  venv/bin/python backtest/research/llm_news_eval.py events --determinism 20
-  venv/bin/python backtest/research/llm_news_eval.py headlines
-  venv/bin/python backtest/research/llm_news_eval.py report
-
-Artifacts land in backtest/research/llm_news_eval_2026-08/ (JSONL replay
-logs are resume-safe: completed events are never re-paid). `report` is a
-pure offline function of the artifacts.
-"""
-
-import argparse
-import json
-import math
-import random
-import sys
-import time
-from datetime import datetime, timezone
-from pathlib import Path
-
-import numpy as np
-import pandas as pd
-
-REPO_ROOT = Path(__file__).resolve().parents[2]
-sys.path.insert(0, str(REPO_ROOT))
-
-import llm_client
-from app_config import deepseek_api_key, get_app_config
-from config import NewsConfig
-from llm_client import DEFAULT_MODEL, DeepSeekClient
-from news_engine import EconomicEvent, NewsEngine
-
-RESEARCH_DIR = Path(__file__).parent
-ART_DIR = RESEARCH_DIR / "llm_news_eval_2026-08"
-CALENDAR = REPO_ROOT / "backtest" / "cache" / "calendar.parquet"
-BARS = (REPO_ROOT / "backtest" / "cache" /
-        "oanda_XAU_USD_15min_20210531_20260804.parquet")
-GOLDEN = RESEARCH_DIR / "llm_news_golden.csv"
-EVENTS_JSONL = ART_DIR / "events_replay.jsonl"
-HEADLINES_JSONL = ART_DIR / "headlines_replay.jsonl"
-SCORED_CSV = ART_DIR / "events_scored.csv"
-
-HORIZONS = [("15m", 15), ("30m", 30), ("1h", 60), ("4h", 240),
-            ("24h", 1440)]
-NEUTRAL_BAND = 0.15  # |xau| below this = neutral prediction (Part B)
-BATCH_CAP = 25       # production batch cap (news_engine.py:783)
-EXPECTED_CORPUS = 586
-
-
-# ── shared plumbing ──────────────────────────────────────────────────
-
-class _Recorder:
-    """requests response hook capturing the exact wire traffic."""
-
-    def __init__(self):
-        self.last = None
-
-    def hook(self, resp, **kwargs):
-        body = resp.request.body
-        if isinstance(body, bytes):
-            body = body.decode("utf-8", "replace")
-        self.last = {
-            "request_body": body,
-            "http_status": resp.status_code,
-            "response_body": resp.text,
-            "elapsed_s": round(resp.elapsed.total_seconds(), 3),
-        }
-        return resp
-
-    def pop(self):
-        cap, self.last = self.last, None
-        return cap or {}
-
-
-def install_research_client(timeout: float):
-    """Build a research DeepSeekClient (raised cap) and seed the module
-    singleton so the production code paths use it. Returns (client,
-    recorder)."""
-    app = get_app_config()
-    key = deepseek_api_key(app)
-    ds = app.deepseek_settings
-    if not key:
-        sys.exit("No DeepSeek API key (env DEEPSEEK_API_KEY or "
-                 "config.json deepseek.api_key) — aborting before spend.")
-    if ds.get("enabled") is False:
-        sys.exit("deepseek.enabled is false in config.json — aborting.")
-    client = DeepSeekClient(
-        api_key=key,
-        model=str(ds.get("model", DEFAULT_MODEL)),
-        timeout=timeout,
-        daily_cap=10 ** 6,  # research: corpus 586 > production cap 500
-    )
-    rec = _Recorder()
-    client._session.hooks["response"].append(rec.hook)
-    with llm_client._client_lock:
-        llm_client._client = client
-    return client, rec
-
-
-def _s(x):
-    """Parquet str columns carry float NaN — EconomicEvent needs None."""
-    return None if pd.isna(x) else str(x)
-
-
-def load_corpus():
-    """High-impact USD releases with a genuine surprise, through the
-    production `EconomicEvent.surprise` gate. Asserts run before any
-    API spend."""
-    df = pd.read_parquet(CALENDAR)
-    df = df[(df["impactName"] == "high") & (df["currency"] == "USD")]
-    rows = []
-    for r in df.itertuples():
-        ev = EconomicEvent(
-            time=datetime.fromtimestamp(int(r.dateline), tz=timezone.utc),
-            currency="USD", impact="high", event=_s(r.name) or "",
-            forecast=_s(r.forecast), previous=_s(r.previous),
-            actual=_s(r.actual),
-        )
-        if ev.surprise not in ("positive", "negative"):
-            continue
-        rows.append({
-            "ffid": int(r.id), "dateline": int(r.dateline),
-            "utc_time": ev.time.isoformat(), "name": ev.event,
-            "actual": ev.actual, "forecast": ev.forecast,
-            "previous": ev.previous, "surprise": ev.surprise,
-            "better_worse": int(r.actualBetterWorse), "ev": ev,
-        })
-    ids = [d["ffid"] for d in rows]
-    if len(rows) != EXPECTED_CORPUS:
-        sys.exit(f"corpus size {len(rows)} != {EXPECTED_CORPUS} — "
-                 "calendar.parquet changed; re-verify before spending.")
-    if len(set(ids)) != len(ids):
-        sys.exit("FF ids not unique — resume keying would break.")
-    # Pin the actualBetterWorse mapping on two unambiguous NFP prints.
-    nfp = {d["utc_time"][:10]: d["better_worse"] for d in rows
-           if d["name"] == "Non-Farm Employment Change"}
-    if nfp.get("2024-01-05") != 1 or nfp.get("2024-05-03") != 2:
-        sys.exit("actualBetterWorse mapping pins failed (expected "
-                 "1=better-for-USD on 2024-01-05 beat, 2=worse on "
-                 "2024-05-03 miss).")
-    rows.sort(key=lambda d: (d["dateline"], d["ffid"]))
-    return rows
-
-
-def _load_jsonl(path: Path):
-    recs = []
-    if path.exists():
-        with open(path) as f:
-            for line in f:
-                line = line.strip()
-                if line:
-                    recs.append(json.loads(line))
-    return recs
-
-
-# ── Part A: event replay ─────────────────────────────────────────────
-
-def _extract_reason(cap):
-    try:
-        content = json.loads(cap["response_body"])[
-            "choices"][0]["message"]["content"]
-        return json.loads(content).get("reason")
-    except Exception:
-        return None
-
-
-def cmd_events(args):
-    ART_DIR.mkdir(exist_ok=True)
-    corpus = load_corpus()
-    if args.limit:
-        corpus = corpus[:args.limit]
-    engine = NewsEngine(NewsConfig())
-    existing = _load_jsonl(EVENTS_JSONL)
-    done1 = {r["ffid"] for r in existing
-             if r.get("run_pass") == 1 and r.get("llm_direction")}
-    done2 = {r["ffid"] for r in existing
-             if r.get("run_pass") == 2 and r.get("llm_direction")}
-    pending = [d for d in corpus if d["ffid"] not in done1]
-    print(f"corpus {len(corpus)} | already recorded {len(done1)} | "
-          f"pending {len(pending)} | determinism re-calls "
-          f"{args.determinism} | model calls are ~100 tokens each "
-          f"(deepseek-v4-flash, static-prefix cached) → cost ≈ cents")
-    if not pending and not args.determinism:
-        print("nothing to do")
-        return
-    _, rec = install_research_client(args.timeout)
-
-    out = open(EVENTS_JSONL, "a")
-
-    def call_one(d, run_pass):
-        # Cache key omits forecast → cleared per call so 196 colliding
-        # corpus rows can't be served a stale direction.
-        with NewsEngine._cache_lock:
-            NewsEngine._llm_event_cache.clear()
-        ev = d["ev"]
-        engine._llm_event_direction(ev)
-        key = (ev.event, ev.currency, str(ev.actual or ""))
-        with NewsEngine._cache_lock:
-            cached = NewsEngine._llm_event_cache.get(key)
-        cap = rec.pop()
-        if cached is None:
-            direction, text = None, None       # hard failure → retry
-        elif cached == "":
-            direction, text = "neutral", ""
-        else:
-            direction, text = cached.split(":", 1)[0], cached
-        recd = {k: d[k] for k in (
-            "ffid", "dateline", "utc_time", "name", "actual", "forecast",
-            "previous", "surprise", "better_worse")}
-        recd.update({
-            "llm_direction": direction, "llm_text": text,
-            "reason_raw": _extract_reason(cap), "run_pass": run_pass,
-            "ts": datetime.now(timezone.utc).isoformat(), **cap,
-        })
-        out.write(json.dumps(recd) + "\n")
-        out.flush()
-        return direction
-
-    remaining = pending
-    for npass in range(1, args.retry_passes + 1):
-        if not remaining:
-            break
-        failed = []
-        for i, d in enumerate(remaining):
-            if call_one(d, 1) is None:
-                failed.append(d)
-            if args.sleep:
-                time.sleep(args.sleep)
-            if (i + 1) % 25 == 0:
-                print(f"  pass {npass}: {i + 1}/{len(remaining)} "
-                      f"({len(failed)} failed)")
-        print(f"pass {npass} done: {len(remaining) - len(failed)} ok, "
-              f"{len(failed)} failed")
-        remaining = failed
-    if remaining:
-        print(f"WARNING: {len(remaining)} events still unrecorded after "
-              f"{args.retry_passes} passes — re-run `events` to resume.")
-
-    if args.determinism:
-        ok_ids = {x["ffid"] for x in remaining}
-        done_now = [d for d in corpus if d["ffid"] not in ok_ids]
-        done_now.sort(key=lambda d: (d["name"], d["dateline"]))
-        stride = max(1, len(done_now) // args.determinism)
-        sample = done_now[::stride][:args.determinism]
-        todo = [d for d in sample if d["ffid"] not in done2]
-        print(f"determinism probe: {len(todo)} re-calls "
-              f"({len(sample) - len(todo)} already recorded)")
-        for d in todo:
-            call_one(d, 2)
-            if args.sleep:
-                time.sleep(args.sleep)
-    out.close()
-    print("events replay complete →", EVENTS_JSONL)
-
-
-# ── Part B: golden headlines ─────────────────────────────────────────
-
-def _load_golden():
-    g = pd.read_csv(GOLDEN)
-    need = {"headline", "category", "expected_xau_sign", "rationale"}
-    if not need <= set(g.columns):
-        sys.exit(f"golden CSV missing columns {need - set(g.columns)}")
-    if not g["expected_xau_sign"].isin([-1, 0, 1]).all():
-        sys.exit("expected_xau_sign must be -1/0/+1")
-    if not g["headline"].is_unique:
-        sys.exit("duplicate headlines in golden CSV")
-    return g
-
-
-def cmd_headlines(args):
-    ART_DIR.mkdir(exist_ok=True)
-    g = _load_golden()
-    heads = list(g["headline"])
-    _, rec = install_research_client(args.timeout)
-    engine = NewsEngine(NewsConfig())
-    out = open(HEADLINES_JSONL, "w")  # tiny (~9 calls): fresh each run
-
-    def run_batch(texts, tag, batch_idx):
-        scores = None
-        for attempt in range(3):
-            scores = engine._llm_headline_scores(texts)
-            cap = rec.pop()
-            if scores is not None:
-                break
-            print(f"  {tag} batch {batch_idx} attempt {attempt + 1} "
-                  f"failed (http {cap.get('http_status')})")
-            time.sleep(2.0)
-        out.write(json.dumps({
-            "run_tag": tag, "batch": batch_idx, "headlines": texts,
-            "scores": scores,
-            "ts": datetime.now(timezone.utc).isoformat(), **cap,
-        }) + "\n")
-        out.flush()
-        return scores
-
-    batches = [heads[i:i + BATCH_CAP] for i in range(0, len(heads),
-                                                     BATCH_CAP)]
-    for bi, b in enumerate(batches):
-        ok = run_batch(b, "main", bi) is not None
-        print(f"main batch {bi} ({len(b)} headlines): "
-              f"{'ok' if ok else 'FAILED'}")
-        time.sleep(args.sleep)
-    run_batch(batches[0], "repeat", 0)
-    print("repeat probe done")
-    shuffled = heads[:]
-    random.Random(20260806).shuffle(shuffled)
-    for bi in range(0, len(shuffled), BATCH_CAP):
-        run_batch(shuffled[bi:bi + BATCH_CAP], "shuffled",
-                  bi // BATCH_CAP)
-    print("shuffled probe done →", HEADLINES_JSONL)
-    out.close()
-
-
-# ── report: pure offline function of the artifacts ───────────────────
-
-def _binom_p(k, n):
-    """Exact two-sided binomial p-value vs p=0.5 (no scipy)."""
-    if n == 0:
-        return float("nan")
-    dist = [math.comb(n, i) for i in range(n + 1)]
-    tot = float(2 ** n)
-    pk = dist[k]
-    return min(1.0, sum(c for c in dist if c <= pk) / tot)
-
-
-def _mcnemar_p(hit_a, hit_b):
-    """Exact McNemar via binomial on discordant pairs. Inputs: aligned
-    boolean lists."""
-    b = sum(1 for x, y in zip(hit_a, hit_b) if x and not y)
-    c = sum(1 for x, y in zip(hit_a, hit_b) if not x and y)
-    return b, c, _binom_p(min(b, c), b + c)
-
-
-def _latest_events(recs):
-    """Last pass-1 record per ffid (retries append; last wins)."""
-    out = {}
-    for r in recs:
-        if r.get("run_pass") == 1:
-            out[r["ffid"]] = r
-    return list(out.values())
-
-
-def _fmt_rate(hits, n, label=""):
-    if n == 0:
-        return f"{label:<24} n=0"
-    p = _binom_p(hits, n)
-    return (f"{label:<24} {hits:>4}/{n:<4} = {hits / n * 100:5.1f}%  "
-            f"(binom p={p:.3f})")
-
-
-def report_events():
-    recs = _latest_events(_load_jsonl(EVENTS_JSONL))
-    if not recs:
-        print("no event replay records — run `events` first")
-        return
-    bars = pd.read_parquet(BARS)
-    closes = bars["Close"].to_numpy()
-    close_ns = (bars.index + pd.Timedelta(minutes=15)).asi8
-
-    def anchor(t_ns, h_min):
-        i = np.searchsorted(close_ns, t_ns, side="right") - 1
-        if i < 0:
-            return None
-        stale_ns = max(45.0, h_min / 4.0) * 60e9
-        if t_ns - close_ns[i] > stale_ns:
-            return None
-        return closes[i]
-
-    rows, skip = [], {h: 0 for h, _ in HORIZONS}
-    base_skipped = 0
-    for r in recs:
-        t_ns = int(r["dateline"]) * 10 ** 9
-        base = anchor(t_ns, 0)
-        if base is None:
-            base_skipped += 1
-            continue
-        row = dict(r)
-        row.pop("request_body", None)
-        row.pop("response_body", None)
-        row["year"] = r["utc_time"][:4]
-        row["base"] = base
-        for hname, hmin in HORIZONS:
-            p = anchor(t_ns + hmin * 60 * 10 ** 9, hmin)
-            if p is None:
-                skip[hname] += 1
-                row[f"d_{hname}"] = np.nan
-            else:
-                row[f"d_{hname}"] = p - base
-        rows.append(row)
-    df = pd.DataFrame(rows)
-    df.to_csv(SCORED_CSV, index=False)
-
-    n_fail = int((df["llm_direction"].isna()).sum())
-    print("═" * 70)
-    print("PART A — event gold-direction vs actual XAU move")
-    print("═" * 70)
-    print(f"events replayed: {len(df)} (baseline-price skipped: "
-          f"{base_skipped}) | LLM hard failures: {n_fail}")
-    print("horizon skip counts (stale/closed):",
-          {k: v for k, v in skip.items() if v})
-    dist = df["llm_direction"].value_counts(dropna=False).to_dict()
-    print("LLM call distribution:", dist)
-
-    # Unconditional per-horizon noise scale (null): median |Δ| over all
-    # 2024+ bar times, position-shifted.
-    bars24 = bars[bars.index >= "2024-01-01"]
-    null_med = {}
-    for hname, hmin in HORIZONS:
-        k = hmin // 15
-        d = (bars24["Close"].shift(-k) - bars24["Close"]).abs().dropna()
-        null_med[hname] = float(d.median())
-    print("unconditional null median |Δ| ($):",
-          {k: round(v, 2) for k, v in null_med.items()})
-
-    # Strategy call columns: +1 gold up, −1 gold down, 0/NaN excluded.
-    df["c_llm"] = df["llm_direction"].map(
-        {"bullish": 1, "bearish": -1}).fillna(0)
-    df["c_doctrine"] = df["surprise"].map({"positive": -1, "negative": 1})
-    df["c_ffbw"] = df["better_worse"].map({1: -1, 2: 1}).fillna(0)
-    df["c_bull"] = 1
-    df["c_bear"] = -1
-
-    strategies = [("LLM", "c_llm"), ("doctrine(surprise)", "c_doctrine"),
-                  ("FF betterWorse", "c_ffbw"), ("always-bullish", "c_bull"),
-                  ("always-bearish", "c_bear")]
-
-    for hname, _ in HORIZONS:
-        dcol = f"d_{hname}"
-        sub = df[df[dcol].notna()]
-        print(f"\n── horizon {hname} (n scored bars={len(sub)}) ──")
-        zeros = int((sub[dcol] == 0).sum())
-        if zeros:
-            print(f"  Δ==0 count (scored as miss): {zeros}")
-        for label, col in strategies:
-            s = sub[sub[col] != 0]
-            hits = int((np.sign(s[dcol]) == s[col]).sum())
-            print("  " + _fmt_rate(hits, len(s), label))
-        # noise-filtered LLM hit rate
-        s = sub[(sub["c_llm"] != 0) &
-                (sub[dcol].abs() >= null_med[hname])]
-        hits = int((np.sign(s[dcol]) == s["c_llm"]).sum())
-        print("  " + _fmt_rate(hits, len(s),
-                               f"LLM |Δ|≥null-med"))
-        # event-window magnitude vs null
-        mag = sub[dcol].abs().median()
-        print(f"  event-window median |Δ| = ${mag:.2f} "
-              f"(null ${null_med[hname]:.2f})")
-        # paired McNemar: LLM vs each baseline on common directional set
-        for label, col in [("doctrine", "c_doctrine"),
-                           ("FF betterWorse", "c_ffbw"),
-                           ("always-bullish", "c_bull")]:
-            s = sub[(sub["c_llm"] != 0) & (sub[col] != 0)]
-            ha = list(np.sign(s[dcol]) == s["c_llm"])
-            hb = list(np.sign(s[dcol]) == s[col])
-            b, c, p = _mcnemar_p(ha, hb)
-            print(f"  McNemar LLM vs {label:<15} b={b:>3} c={c:>3} "
-                  f"p={p:.3f}")
-
-    # neutral-call flatness (15m)
-    neu = df[(df["llm_direction"] == "neutral") & df["d_15m"].notna()]
-    dire = df[(df["c_llm"] != 0) & df["d_15m"].notna()]
-    if len(neu):
-        print(f"\nneutral calls (n={len(neu)}): median |Δ_15m| "
-              f"${neu['d_15m'].abs().median():.2f} vs directional "
-              f"(n={len(dire)}) ${dire['d_15m'].abs().median():.2f} "
-              f"(null ${null_med['15m']:.2f})")
-
-    # breakdowns at 1h
-    dcol = "d_1h"
-    sub = df[df[dcol].notna() & (df["c_llm"] != 0)]
-    print("\n── LLM hit rate by event name (1h, top 10 by count) ──")
-    for name, grp in sorted(sub.groupby("name"), key=lambda kv: -len(kv[1]))[:10]:
-        hits = int((np.sign(grp[dcol]) == grp["c_llm"]).sum())
-        print("  " + _fmt_rate(hits, len(grp), name[:24]))
-    print("── LLM hit rate by year (1h) ──")
-    for yr, grp in sub.groupby("year"):
-        hits = int((np.sign(grp[dcol]) == grp["c_llm"]).sum())
-        print("  " + _fmt_rate(hits, len(grp), yr))
-
-    # cluster view: shared datelines
-    print("\n── same-timestamp clusters ──")
-    clus = df.groupby("dateline").filter(lambda g: len(g) > 1)
-    groups = list(clus.groupby("dateline"))
-    consistent = 0
-    net_hits, net_n = 0, 0
-    for _, g in groups:
-        signs = set(g.loc[g["c_llm"] != 0, "c_llm"])
-        if len(signs) <= 1:
-            consistent += 1
-        net = int(g["c_llm"].sum())
-        d = g["d_1h"].iloc[0]
-        if net != 0 and pd.notna(d) and d != 0:
-            net_n += 1
-            if np.sign(d) == np.sign(net):
-                net_hits += 1
-    print(f"clusters: {len(groups)} datelines "
-          f"({len(clus)} events, {len(clus) / max(len(df), 1) * 100:.0f}% of "
-          f"corpus) | internally consistent: {consistent}/{len(groups)}")
-    print("  " + _fmt_rate(net_hits, net_n, "cluster net-call (1h)"))
-
-    # determinism probe
-    p2 = {r["ffid"]: r for r in _load_jsonl(EVENTS_JSONL)
-          if r.get("run_pass") == 2 and r.get("llm_direction")}
-    if p2:
-        p1 = {r["ffid"]: r for r in recs}
-        agree = same_reason = n = 0
-        for fid, r2 in p2.items():
-            r1 = p1.get(fid)
-            if not r1 or not r1.get("llm_direction"):
-                continue
-            n += 1
-            if r1["llm_direction"] == r2["llm_direction"]:
-                agree += 1
-            if r1.get("reason_raw") == r2.get("reason_raw"):
-                same_reason += 1
-        print(f"\ndeterminism probe: direction agreement {agree}/{n}, "
-              f"identical reason text {same_reason}/{n}")
-
-    # stratified reason examples
-    print("\n── reason samples (1h hit/miss) ──")
-    sub = df[df[dcol].notna() & (df["c_llm"] != 0)].copy()
-    sub["hit"] = np.sign(sub[dcol]) == sub["c_llm"]
-    for verdict in (True, False):
-        picked, seen = [], set()
-        for _, r in sub[sub["hit"] == verdict].iterrows():
-            if r["name"] in seen:
-                continue
-            seen.add(r["name"])
-            picked.append(r)
-            if len(picked) == 10:
-                break
-        tag = "HIT " if verdict else "MISS"
-        for r in picked:
-            print(f"  [{tag}] {r['utc_time'][:16]} {r['name'][:30]:<30} "
-                  f"a={r['actual']} f={r['forecast']} → "
-                  f"{r['llm_direction']:<7} Δ1h={r[dcol]:+7.2f}  "
-                  f"“{(r['reason_raw'] or '')[:60]}”")
-    print(f"\nscored table → {SCORED_CSV}")
-
-
-def report_headlines():
-    recs = _load_jsonl(HEADLINES_JSONL)
-    if not recs:
-        print("\n(no headline replay records — run `headlines` first)")
-        return
-    g = _load_golden()
-    engine = NewsEngine(NewsConfig())
-
-    def collect(tag):
-        m = {}
-        for r in recs:
-            if r["run_tag"] == tag and r["scores"]:
-                for h, sc in zip(r["headlines"], r["scores"]):
-                    m[h] = tuple(sc)
-        return m
-
-    main, shuf = collect("main"), collect("shuffled")
-    print("\n" + "═" * 70)
-    print("PART B — golden headline set: LLM vs keyword baseline")
-    print("═" * 70)
-    missing = [h for h in g["headline"] if h not in main]
-    if missing:
-        print(f"WARNING: {len(missing)} headlines missing from main run")
-
-    def cls(x):
-        if x >= NEUTRAL_BAND:
-            return 1
-        if x <= -NEUTRAL_BAND:
-            return -1
-        return 0
-
-    rows = []
-    for _, r in g.iterrows():
-        h = r["headline"]
-        if h not in main:
-            continue
-        llm_xau = main[h][1]
-        kw = engine._analyze_headline_sentiment(h, for_currency="XAU")
-        rows.append({
-            "headline": h, "category": r["category"],
-            "exp": int(r["expected_xau_sign"]),
-            "llm_xau": llm_xau, "llm_cls": cls(llm_xau),
-            "kw": kw, "kw_cls": cls(kw),
-        })
-    t = pd.DataFrame(rows)
-    for who in ("llm", "kw"):
-        acc = (t[f"{who}_cls"] == t["exp"]).mean()
-        print(f"{who.upper():<4} 3-class accuracy (band ±{NEUTRAL_BAND}): "
-              f"{(t[f'{who}_cls'] == t['exp']).sum()}/{len(t)} "
-              f"= {acc * 100:.1f}%")
-    ha = list(t["llm_cls"] == t["exp"])
-    hb = list(t["kw_cls"] == t["exp"])
-    b, c, p = _mcnemar_p(ha, hb)
-    print(f"paired McNemar LLM vs keyword: b={b} c={c} p={p:.3f}")
-
-    print("\nper-category accuracy (LLM | keyword):")
-    for cat, grp in t.groupby("category"):
-        a1 = (grp["llm_cls"] == grp["exp"]).sum()
-        a2 = (grp["kw_cls"] == grp["exp"]).sum()
-        print(f"  {cat:<18} {a1:>2}/{len(grp):<2} | {a2:>2}/{len(grp)}")
-
-    print("\nLLM confusion (rows=expected, cols=predicted -1/0/+1):")
-    cm = pd.crosstab(t["exp"], t["llm_cls"]).reindex(
-        index=[-1, 0, 1], columns=[-1, 0, 1], fill_value=0)
-    print(cm.to_string())
-
-    print("\nLLM misclassified headlines:")
-    for _, r in t[t["llm_cls"] != t["exp"]].iterrows():
-        print(f"  exp {r['exp']:+d} got {r['llm_xau']:+.2f} "
-              f"[{r['category']}] {r['headline'][:70]}")
-
-    # repeat-batch determinism
-    m_b0 = [r for r in recs if r["run_tag"] == "main" and r["batch"] == 0]
-    r_b0 = [r for r in recs if r["run_tag"] == "repeat"]
-    if m_b0 and r_b0 and m_b0[0]["scores"] and r_b0[0]["scores"]:
-        a = np.array(m_b0[0]["scores"], dtype=float)
-        b_ = np.array(r_b0[0]["scores"], dtype=float)
-        exact = int((a == b_).all(axis=1).sum())
-        print(f"\nrepeat-batch: {exact}/{len(a)} score pairs identical, "
-              f"max |diff| = {np.abs(a - b_).max():.3f}")
-
-    # shuffled-order drift
-    if shuf:
-        both = [h for h in t["headline"] if h in shuf]
-        drift = [abs(main[h][1] - shuf[h][1]) for h in both]
-        if drift:
-            print(f"shuffled-order xau drift over {len(both)} headlines: "
-                  f"mean {np.mean(drift):.3f}, max {np.max(drift):.3f}")
-
-
-def cmd_report(args):
-    report_events()
-    report_headlines()
-
-
-# ── CLI ──────────────────────────────────────────────────────────────
-
-def main():
-    ap = argparse.ArgumentParser(description=__doc__.split("\n")[1])
-    sub = ap.add_subparsers(dest="cmd", required=True)
-
-    e = sub.add_parser("events", help="replay release surprises")
-    e.add_argument("--limit", type=int, default=0,
-                   help="smoke-test on first N events")
-    e.add_argument("--determinism", type=int, default=0,
-                   help="re-call N stratified events a second time")
-    e.add_argument("--sleep", type=float, default=0.3)
-    e.add_argument("--retry-passes", type=int, default=3)
-    e.add_argument("--timeout", type=float, default=30.0)
-    e.set_defaults(fn=cmd_events)
-
-    h = sub.add_parser("headlines", help="golden-set probes")
-    h.add_argument("--sleep", type=float, default=0.3)
-    h.add_argument("--timeout", type=float, default=30.0)
-    h.set_defaults(fn=cmd_headlines)
-
-    r = sub.add_parser("report", help="offline scoring report")
-    r.set_defaults(fn=cmd_report)
-
-    args = ap.parse_args()
-    args.fn(args)
-
-
-if __name__ == "__main__":
-    main()

backtest/research/llm_news_eval_2026-08-06.md

@@ -1,86 +0,0 @@
-# DeepSeek news-analysis validation — 2026-08-06
-
-**Verdict: headline sentiment scoring is excellent; event-direction calls are
-directionally sound but ~11% of them misread the numeric comparison, and the
-failure concentrates on Unemployment Claims / close K-M-suffixed values.**
-
-Method (owner-directed, token-lean): independent expert re-analysis of a
-stratified sample of DeepSeek's recorded answers, not a full statistical
-backtest. Harness: `backtest/research/llm_news_eval.py` (replays releases
-through the REAL `_llm_event_direction` / `_llm_headline_scores` — real
-prompts, payloads, validation; raw wire traffic logged). 135 of the 586-event
-corpus were replayed before the full run was stopped by owner request
-(artifacts retained in `llm_news_eval_2026-08/`, resume-safe).
-
-## Part A — event gold-direction (28-event stratified sample, Jan–Jun 2024)
-
-Reviewer agreement with DeepSeek: **24/28 (86%)**. All 4 disagreements are
-DeepSeek errors, in exactly two failure modes:
-
-1. **Numeric comparison misread (3/28 ≈ 11%)** — the doctrine logic is right
-   but the premise is backwards:
-   - JOLTS 8.79M vs f 8.84M (a miss) → DS: "Job openings **above** forecast" → bearish (should be bullish)
-   - Claims 202K vs f 217K (fewer claims) → DS: "Claims **higher** than forecast" → bullish (should be bearish)
-   - Claims 202K vs f 209K — same backwards read, same wrong call
-   Both Unemployment Claims events in the sample were misread the same way —
-   looks systematic on that event type, possibly an anchoring prior that
-   "claims rose". All misreads involve close values with K/M suffixes.
-2. **Direction field contradicts its own reason (1/28)** — Empire State
-   −43.7 vs f −4.9: reason text says "gold may rise", `gold_direction` says
-   `bearish`.
-
-Where the numbers were read correctly, hawkish/dovish→gold mapping was
-consistently correct (CPI/PPI/NFP/ADP/GDP/PMI/wages/unemployment-rate all
-handled properly, including inverted metrics when read right).
-
-Price cross-check (1h horizon, cluster-confounded, indicative only):
-reviewer calls 22/28 with the subsequent move, DeepSeek 20/28.
-
-## Part B — headline sentiment (78-headline golden set)
-
-DeepSeek matched the reviewer's expected gold-direction label on
-**78/78 (100%)** — including all 8 negation traps ("signals no rate hike",
-"no rate cuts until…") and all 6 inverted-metric phrasings ("claims hit
-two-year high"). The keyword scorer it overrides got **24/78 (31%)**.
-Repeat-batch and shuffled-order probes recorded (headlines_replay.jsonl).
-
-## Implications
-
-- The headline-sentiment override (the main live LLM surface) is validated —
-  a large, measured upgrade over keywords.
-- The event-direction enrichment ("— gold bullish/bearish" appended to
-  surprise descriptions) is right ~85–90% of the time but can state a
-  factually backwards comparison in the displayed reason. It is display-only
-  (never gates signals), so exposure is a possibly-misleading Telegram/CLI
-  line, not trades.
-- **Hardening SHIPPED same day (owner-approved):** the payload now carries a
-  code-computed `vs_forecast` field ("above/below/in line with forecast",
-  from `EconomicEvent.surprise`) and the prompt tells the model to trust it.
-  First iteration fixed the arithmetic but exposed a masked doctrine error
-  (model read "claims below forecast" correctly, then called it a *weaker*
-  labor market — and regressed the unemployment-rate case the same way), so
-  the prompt's inverted-metrics line was expanded to an explicit rule plus a
-  worked claims example. Verification: **18/18** — all 4 original misreads
-  fixed, 11/11 no-regression cases unchanged, 3/3 fresh probes on the side
-  the prompt example does NOT cover (claims 245K vs f 220K → bullish,
-  unemployment 4.1% vs 3.9% → bullish, JOLTS 9.10M vs 8.80M → bearish).
-  Suite green at 346 tests, incl. a new payload assertion in Module 19.
-- Production observations found while building the harness (not fixed):
-  `_llm_event_cache` key omits forecast → a repeated `actual` re-serves a
-  direction computed for a different forecast; a "neutral" reply is
-  indistinguishable from a failure in the return value.
-
-## Reproduce
-
-```
-venv/bin/python backtest/research/llm_news_eval.py events --limit 5   # smoke
-venv/bin/python backtest/research/llm_news_eval.py events --determinism 20
-venv/bin/python backtest/research/llm_news_eval.py headlines
-venv/bin/python backtest/research/llm_news_eval.py report             # offline
-```
-
-Corpus: 586 high-impact USD surprises 2024-01→2026-07 (calendar.parquet,
-NFP-verified UTC); prices: Oanda M15 mids. Full-run cost ≈ cents. The
-`report` subcommand (hit rates vs always-bullish/doctrine/FF-betterWorse
-baselines, McNemar, cluster view) is built and untested at scale — run it
-after completing the replay if a statistical answer is ever wanted.

backtest/research/llm_news_golden.csv

@@ -1,79 +0,0 @@
-headline,category,expected_xau_sign,rationale
-"Powell says restrictive policy needed for longer as inflation proves sticky",hawkish_fed,-1,"higher-for-longer rates lift real yields, gold-bearish"
-"Fed minutes show officials leaning toward two more rate hikes this year",hawkish_fed,-1,"more hikes = stronger dollar/yields"
-"Hot CPI print fuels bets on additional Fed tightening",hawkish_fed,-1,"tightening expectations weigh on gold"
-"Fed's Waller: higher-for-longer rates warranted by resilient economy",hawkish_fed,-1,"hawkish guidance"
-"FOMC raises rates 50bp, flags further increases ahead",hawkish_fed,-1,"outsized hike plus hawkish guidance"
-"Treasury yields spike as Fed pushes back on rate cut hopes",hawkish_fed,-1,"cut hopes dashed, yields up"
-"Core inflation accelerates, cementing case for hawkish Fed stance",hawkish_fed,-1,"hawkish repricing"
-"Fed dot plot shifts higher; markets price out easing for the year",hawkish_fed,-1,"easing priced out"
-"Fed cuts rates 25bp and signals more easing to come",dovish_fed,1,"easing cycle is gold-bullish"
-"Powell hints at rate cuts as inflation cools toward target",dovish_fed,1,"cut guidance"
-"FOMC minutes reveal growing support for policy easing",dovish_fed,1,"dovish minutes"
-"Fed's Williams says rate cuts appropriate later this year",dovish_fed,1,"cut guidance"
-"Soft inflation data boosts odds of a dovish Fed pivot",dovish_fed,1,"pivot odds up"
-"Fed pauses hikes, markets price aggressive easing cycle",dovish_fed,1,"easing priced in"
-"Central banks turn dovish as global growth falters",dovish_fed,1,"global easing backdrop"
-"Fed to restart bond purchases to support flagging economy",dovish_fed,1,"QE is gold-bullish"
-"Dollar index hits 12-month high on robust US economic data",strong_usd,-1,"strong dollar pressures gold"
-"US GDP grows 4.1% annualized, blowing past expectations",strong_usd,-1,"strong economy, hawkish implication"
-"Blockbuster jobs report sends dollar surging across the board",strong_usd,-1,"strong jobs lift dollar and yields"
-"US retail sales smash forecasts, underscoring consumer strength",strong_usd,-1,"strong data"
-"Greenback rallies as US data outshines Europe and Asia",strong_usd,-1,"dollar strength"
-"ISM services index jumps to two-year high on strong demand",strong_usd,-1,"strong data"
-"Consumer confidence surges to post-pandemic high in the US",strong_usd,-1,"strong data"
-"Dollar tumbles to eight-month low after dismal manufacturing data",weak_usd,1,"weak dollar lifts gold"
-"US economy adds far fewer jobs than expected; dollar slides",weak_usd,1,"weak jobs, dovish implication"
-"Greenback weakens as US retail sales unexpectedly contract",weak_usd,1,"weak data"
-"US GDP growth revised down sharply, stoking slowdown fears",weak_usd,1,"slowdown is dovish"
-"Dollar slumps as trade deficit balloons to record",weak_usd,1,"dollar weakness"
-"Soft services PMI adds to evidence of cooling US economy",weak_usd,1,"cooling economy"
-"US consumer sentiment plunges to two-year low",weak_usd,1,"weak data"
-"Real yields climb to multi-year highs as TIPS sell off",real_yields,-1,"rising real yields raise gold's opportunity cost"
-"10-year Treasury yield surges past 5% on supply glut",real_yields,-1,"yield spike"
-"Rising real interest rates dim appeal of non-yielding assets",real_yields,-1,"explicit opportunity-cost story"
-"Treasury yields sink as investors flock to bonds",real_yields,1,"falling yields support gold"
-"Real yields turn negative as inflation outpaces rates",real_yields,1,"negative real yields are gold's best backdrop"
-"Bond rally drives 10-year yield to lowest level in a year",real_yields,1,"falling yields"
-"Geopolitical tensions escalate in Middle East, investors seek safety",safe_haven,1,"safe-haven bid"
-"Stocks crater worldwide as banking crisis fears spread",safe_haven,1,"crisis flight to safety"
-"Escalating conflict drives flight to safe-haven assets",safe_haven,1,"explicit haven flow"
-"Investors dump risk assets after surprise sovereign default",safe_haven,1,"default shock"
-"Military escalation on NATO border rattles global markets",safe_haven,1,"geopolitical stress"
-"Wall Street plunges 4% in worst session since pandemic",safe_haven,1,"risk-off"
-"Currency crisis in emerging markets sparks contagion fears",safe_haven,1,"contagion, haven bid"
-"Nuclear rhetoric intensifies, sending investors to havens",safe_haven,1,"extreme geopolitical stress"
-"Ceasefire agreement lifts global markets, haven demand fades",risk_on,-1,"haven unwind"
-"Stocks roar to record highs as trade deal signed",risk_on,-1,"risk-on drains haven demand"
-"Risk appetite returns as banking fears subside",risk_on,-1,"risk-on"
-"Global equities rally on strong earnings and easing tensions",risk_on,-1,"risk-on"
-"Gold hits record high as central banks keep buying bullion",gold_direct,1,"direct demand"
-"Gold ETFs post biggest weekly inflows in three years",gold_direct,1,"investment inflows"
-"China's central bank extends gold buying spree to 18th month",gold_direct,1,"official-sector demand"
-"Miners struggle to meet gold demand as supply deficit widens",gold_direct,1,"supply deficit"
-"Gold slides 2% as investors book profits after record run",gold_direct,-1,"profit-taking pressure"
-"Gold tumbles below key support as ETF outflows accelerate",gold_direct,-1,"outflows"
-"Bullion demand weakens as Indian jewellery buying stalls",gold_direct,-1,"physical demand soft"
-"Speculators slash bullish gold bets as dollar strengthens",gold_direct,-1,"positioning unwind"
-"Fed holds rates steady, signals no rate hike on the horizon",negation_traps,1,"dovish hold; keyword layer sees the phrase rate hike and flips it gold-bearish"
-"Powell says no rate cuts until inflation shows sustained cooling",negation_traps,-1,"hawkish hold; keyword layer sees rate cut and calls it gold-bullish"
-"Recession fears fade as US data streak beats estimates",negation_traps,-1,"fading fear is risk-on; keywords see recession plus beats and cancel out"
-"Inflation fails to accelerate, easing pressure on the Fed",negation_traps,1,"disinflation opens door to cuts"
-"Fed's Bowman: not yet confident inflation is beaten, cuts premature",negation_traps,-1,"hawkish pushback phrased around negations"
-"Central bank stops short of hiking, but warns tightening may resume",negation_traps,-1,"hawkish hold with tightening bias"
-"No safe-haven bid for gold despite equity wobble, analysts say",negation_traps,-1,"explicitly negated haven demand"
-"Jobs report not as weak as feared, dollar recovers",negation_traps,-1,"less-bad data is dollar-positive"
-"US manufacturing output plunges as new orders dry up",inverted_metrics,1,"weak economy is dovish/gold-bullish; keyword layer reads plunges as bearish"
-"US factory activity slumps to three-year low",inverted_metrics,1,"weak data; keyword reads slumps as bearish"
-"Consumer spending falls for third straight month",inverted_metrics,1,"weak data; keyword reads falls as bearish"
-"Private payrolls growth slides to weakest pace since 2020",inverted_metrics,1,"weak jobs; keyword reads slides as bearish"
-"Jobless claims hit two-year high as layoffs spread",inverted_metrics,1,"higher claims = weaker economy = gold-bullish"
-"US unemployment rate hits 4.5%, highest since 2021",inverted_metrics,1,"higher unemployment = dovish = gold-bullish"
-"Lakers clinch playoff berth with overtime win over Suns",distractors,0,"irrelevant"
-"New Marvel film smashes box office records on opening weekend",distractors,0,"irrelevant"
-"Apple unveils thinner MacBook with faster chip lineup",distractors,0,"irrelevant"
-"Scientists discover new deep-sea species off Pacific coast",distractors,0,"irrelevant"
-"Heavy rain forecast for the Midwest through the weekend",distractors,0,"irrelevant"
-"Streaming service raises subscription prices in Europe",distractors,0,"irrelevant"
-"Electric vehicle startup recalls 12,000 cars over software glitch",distractors,0,"irrelevant"
-"Local elections see record turnout in several US states",distractors,0,"irrelevant"

config.json.example

@@ -7,7 +7,6 @@
   },
   "data_providers": {
     "active": "oanda",
-    "streaming": "tiingo",
     "twelve_data": {
       "api_key": "REPLACE_WITH_YOUR_TWELVE_DATA_API_KEY"
     },
@@ -33,20 +32,5 @@
     "quality_gate": "C",
     "max_hold_bars": 300
   },
-  "deepseek": {
-    "api_key": "REPLACE_WITH_YOUR_DEEPSEEK_API_KEY_OR_EMPTY",
-    "model": "deepseek-v4-flash",
-    "enabled": true,
-    "timeout_seconds": 12,
-    "daily_request_cap": 500
-  },
-  "agent_panel": {
-    "enabled": false,
-    "telegram": true,
-    "dashboard": false,
-    "min_quality": "B",
-    "max_panels_per_day": 20,
-    "reuse_ttl_seconds": 3600
-  },
   "log_level": "INFO"
 }

config.py

@@ -46,12 +46,6 @@ class TechnicalConfig:
     ema_medium: int = 21
     ema_slow: int = 50
     ema_trend: int = 200
-    # 4-EMA mode only: score partial ribbon alignment (2-3 of 4 checks)
-    # as NEUTRAL instead of a weak BUY/SELL. Research knob (2026-08-04
-    # indicator audit: partial-alignment trades averaged -0.008R vs
-    # +0.230R fully aligned on the M5 12mo Oanda book); default False =
-    # legacy behavior until a replay campaign validates flipping it.
-    ema_partial_neutral: bool = False
 
     rsi_period: int = 14
     rsi_overbought: float = 70.0
@@ -166,9 +160,6 @@ class NewsConfig:
     # Must stay below pre_event_buffer_minutes so an event can't cross
     # into the danger window while a stale cache says it's clear.
     context_cache_ttl_seconds: int = 600
-    # Maximum age of RSS headlines accepted for scoring. Older headlines
-    # are discarded before keyword analysis and LLM enrichment.
-    headline_max_age_seconds: int = 43200  # 12 hours
 
     rss_feeds: List[str] = field(default_factory=lambda: [
         "https://investinglive.com/feed/",

dashboard_publisher.py

@@ -1,459 +0,0 @@
-#!/usr/bin/env python3
-"""
-XAU/USD static dashboard publisher.
-
-Standalone daemon that scans XAU/USD at candle closes and atomically
-writes `snapshot.json` + `news.json` (plus a copy of
-`static/dashboard.html`) into an Apache-served directory. Apache does
-all web serving — this process never opens a port, so a failure here
-can only ever mean a stale page.
-
-Independence rules (owner requirement — the telegram bot must be
-immune to anything on the web side):
-  * Runs as its own process; never imported by, and never imports,
-    `telegram_bot` / `oanda_daemon` (telegram_bot resolves credentials
-    at import). The confirmation map and boundary math are therefore
-    duplicated here as local constants.
-  * The only shared-code dependency beyond the standard signal path is
-    the additive `NewsEngine.get_week_events` accessor, which no
-    production signal path calls.
-
-Scheduling: every M15/H1/H4 boundary is also an M5 boundary (all
-intervals are multiples of 5, anchored to UTC midnight), so one
-sequential loop sleeps to the next M5 boundary (+10 s buffer) and scans
-whichever timeframes just closed. Due-classification uses the RAW wake
-time — flooring 12:00:10 to the 5-minute grid yields the 12:00 boundary
-that just closed. Do NOT subtract the buffer first: 12:00:10 − 15 s
-floors to 11:55 and silently skips every H1/H4 scan (v1 bug, 2026-08-02).
-
-Scans run sequentially M5→M15→H1→H4 through one shared data provider so
-later timeframes hit the warm boundary cache — that ordering IS the
-duplicate-fetch dedup (providers have no in-flight dedup; Oanda has no
-throttle). Keep it sequential.
-
-Run:  python dashboard_publisher.py [--once] [--out DIR]
-Out dir precedence: --out → DASHBOARD_OUT_DIR env → /var/www/trade
-"""
-
-import argparse
-import json
-import hashlib
-import logging
-import math
-import numbers
-import os
-import shutil
-import sys
-import time
-from datetime import datetime, timezone
-from logging.handlers import TimedRotatingFileHandler
-from typing import Dict, List, Optional
-
-from agent_panel import attach_panel
-from config import PlatformConfig, Timeframe, XAUUSD
-from main import report_to_dict
-from news_engine import NewsEngine
-from pair_signals import produce_signal
-from providers import make_data_provider
-import providers.base as _providers_base
-
-logger = logging.getLogger("dashboard")
-
-DASHBOARD_TFS = [Timeframe.M5, Timeframe.M15, Timeframe.H1, Timeframe.H4]
-
-# Local copy of the bot's confirmation map (see independence rules above).
-CONFIRMATIONS = {
-    Timeframe.M5:  [Timeframe.M15, Timeframe.H1],
-    Timeframe.M15: [Timeframe.M5, Timeframe.H1],
-    Timeframe.H1:  [Timeframe.M15, Timeframe.H4],
-    Timeframe.H4:  [Timeframe.H1, Timeframe.D1],
-}
-
-TF_INTERVAL_MINUTES = {
-    Timeframe.M5: 5, Timeframe.M15: 15, Timeframe.H1: 60, Timeframe.H4: 240,
-}
-
-DEFAULT_OUT_DIR = "/var/www/trade"
-SCAN_BUFFER_SECONDS = 10
-SNAPSHOT_FILE = "snapshot.json"
-NEWS_FILE = "news.json"
-PAGE_FILE = "dashboard.html"
-SUMMARY_PHP_FILE = "news_summary.php"
-# Repo files copied into the out dir at daemon start (git pull + restart
-# updates them). The PHP endpoint reads news.json in place, so it lives
-# next to the page.
-STATIC_FILES = [PAGE_FILE, SUMMARY_PHP_FILE]
-
-
-# ══════════════════════════════════════════════════════════════════
-#  Pure layer (no network, fully covered by test_all.py)
-# ══════════════════════════════════════════════════════════════════
-
-def _safe_num(v):
-    """Coerce to a JSON-safe float; NaN/±inf become None."""
-    try:
-        f = float(v)
-    except (TypeError, ValueError):
-        return v
-    if math.isnan(f) or math.isinf(f):
-        return None
-    return f
-
-
-def _sanitize(obj):
-    """Recursively make a payload safe for json.dumps(allow_nan=False):
-    NaN/inf → None, numpy scalars → native ints/floats."""
-    if isinstance(obj, bool):
-        return obj
-    if isinstance(obj, dict):
-        return {k: _sanitize(v) for k, v in obj.items()}
-    if isinstance(obj, (list, tuple)):
-        return [_sanitize(v) for v in obj]
-    if isinstance(obj, numbers.Integral):
-        return int(obj)
-    if isinstance(obj, numbers.Real):
-        return _safe_num(obj)
-    return obj
-
-
-def analysis_to_dict(analysis) -> Optional[dict]:
-    """Serialize a TechnicalAnalysis. Iterates analysis.indicators only —
-    never vars()/asdict(), which would drag dynamic pandas attrs
-    (_rsi_series, _macd_histogram, …) into the payload. ATR (weight 0)
-    is surfaced separately as context, not in the directional grid."""
-    if analysis is None:
-        return None
-    indicators = {}
-    atr_context = None
-    for name, ind in analysis.indicators.items():
-        if name.startswith("_"):
-            continue
-        entry = {
-            "signal": ind.signal.name,
-            "value": _safe_num(ind.value),
-            "confidence": _safe_num(ind.confidence),
-            "weight": _safe_num(ind.weight),
-            "description": ind.description,
-        }
-        if name == "atr":
-            atr_context = entry
-        elif ind.weight != 0:
-            indicators[name] = entry
-    return {
-        "timeframe": analysis.timeframe,
-        "timestamp": str(analysis.timestamp),
-        "composite_signal": analysis.composite_signal.name,
-        "composite_score": _safe_num(analysis.composite_score),
-        "confidence": _safe_num(analysis.confidence),
-        "market_regime": analysis.market_regime,
-        "volatility_regime": analysis.volatility_regime,
-        "pattern_signals": list(analysis.pattern_signals),
-        "key_levels": {k: _safe_num(v) for k, v in analysis.key_levels.items()},
-        "indicators": indicators,
-        "atr_context": atr_context,
-    }
-
-
-def report_to_webapp_dict(report) -> dict:
-    """main.report_to_dict + the per-TF detail the dashboard needs."""
-    d = report_to_dict(report)
-    d["primary_signal"] = report.primary_signal.name
-    d["primary_score"] = _safe_num(report.primary_score)
-    d["mtf_scores"] = {k: _safe_num(v) for k, v in report.mtf_scores.items()}
-    d["exhaustion_count"] = report.exhaustion_count
-    d["primary_analysis"] = analysis_to_dict(report.primary_analysis)
-    d["confirmation_analyses"] = [
-        analysis_to_dict(a) for a in report.confirmation_analyses
-    ]
-    return _sanitize(d)
-
-
-def event_to_dict(ev) -> dict:
-    """EconomicEvent → dict. surprise/minutes_until/is_high_impact are
-    properties, so asdict() would skip them — added explicitly.
-    actual/forecast/previous stay strings as published ('228K', '0.3%')."""
-    return {
-        "time": ev.time.astimezone(timezone.utc).isoformat(),
-        "currency": ev.currency,
-        "impact": ev.impact,
-        "event": ev.event,
-        "forecast": ev.forecast,
-        "previous": ev.previous,
-        "actual": ev.actual,
-        "surprise": ev.surprise,
-        "minutes_until": _safe_num(ev.minutes_until),
-        "is_high_impact": ev.is_high_impact,
-    }
-
-
-def headline_to_dict(h) -> dict:
-    # id uses the same sha1(headline) scheme as NewsEngine._llm_sent_cache;
-    # it is the join key the PHP summary endpoint whitelists against.
-    return {
-        "id": hashlib.sha1(h.headline.encode("utf-8")).hexdigest(),
-        "headline": h.headline,
-        "source": h.source,
-        "link": h.link,
-        "description": h.description,
-        "publisher": h.publisher,
-        "timestamp": h.timestamp.astimezone(timezone.utc).isoformat()
-                     if h.timestamp else None,
-        "currencies": list(h.currencies_mentioned),
-        "score": _safe_num(h.sentiment_score),
-        "keywords": list(h.keywords),
-        "score_by_currency": {
-            k: _safe_num(v) for k, v in h.sentiment_by_currency.items()
-        },
-    }
-
-
-def build_news_payload(context, week_events, now: datetime) -> dict:
-    """news.json: full calendar week + scored headlines + LLM summary."""
-    headlines = sorted(
-        context.news_headlines,
-        key=lambda h: h.timestamp or now, reverse=True,
-    )
-    return _sanitize({
-        "generated_at": now.isoformat(),
-        "llm_summary": context.llm_summary,
-        "sentiment": dict(context.news_sentiment),
-        "high_impact_warning": context.high_impact_warning,
-        "warning_message": context.warning_message,
-        "calendar_available": context.calendar_available,
-        "events": [event_to_dict(e) for e in week_events],
-        "headlines": [headline_to_dict(h) for h in headlines],
-    })
-
-
-def due_timeframes(wake_dt: datetime) -> List[Timeframe]:
-    """Timeframes whose candle just closed at the boundary this wake
-    belongs to. Takes the RAW wake time (boundary + buffer): flooring to
-    the 5-minute grid yields the boundary that just passed. Never
-    subtract the buffer first — see module docstring (v1 bug)."""
-    minutes = wake_dt.hour * 60 + wake_dt.minute
-    floored = minutes - (minutes % 5)
-    return [tf for tf in DASHBOARD_TFS if floored % TF_INTERVAL_MINUTES[tf] == 0]
-
-
-def seconds_until_boundary(interval_minutes: int, buffer: int = 10) -> float:
-    # Same math as telegram_bot.py / oanda_daemon.py — kept local so the
-    # dashboard never imports bot modules (independence rules above).
-    now = datetime.now(timezone.utc)
-    total_minutes = now.hour * 60 + now.minute
-    elapsed_minutes = total_minutes % interval_minutes
-    elapsed_seconds = elapsed_minutes * 60 + now.second + now.microsecond / 1e6
-    interval_seconds = interval_minutes * 60
-    remaining = interval_seconds - elapsed_seconds + buffer
-    # NOTE: remaining > buffer always (elapsed < interval by construction);
-    # a call at the exact boundary instant targets the NEXT boundary —
-    # steady-state loops wake at boundary+buffer, so only a process
-    # started in the boundary microsecond skips one scan (2026-08-04
-    # audit I42; the old `remaining <= buffer` guard was unreachable).
-    return remaining
-
-
-def _empty_tf_entry() -> dict:
-    return {"status": "warming", "scanned_at": None, "scan_seconds": None,
-            "error": None, "error_at": None, "report": None}
-
-
-def build_snapshot_payload(snapshots: Dict[str, dict], now: datetime) -> dict:
-    """snapshot.json: one entry per dashboard TF. Timeframes not yet
-    scanned appear as status="warming" so the page can render a
-    placeholder instead of a hole. server_time_epoch lets the client
-    compute its clock offset once so countdowns don't drift."""
-    timeframes = {}
-    for tf in DASHBOARD_TFS:
-        interval = TF_INTERVAL_MINUTES[tf]
-        minutes = now.hour * 60 + now.minute
-        secs_into = (minutes % interval) * 60 + now.second + now.microsecond / 1e6
-        next_close = now.timestamp() + (interval * 60 - secs_into)
-        entry = _empty_tf_entry()
-        entry.update(snapshots.get(tf.value) or {})
-        entry["next_close_epoch"] = next_close
-        entry["next_scan_epoch"] = next_close + SCAN_BUFFER_SECONDS
-        timeframes[tf.value] = entry
-    return _sanitize({
-        "symbol": XAUUSD,
-        "server_time_epoch": now.timestamp(),
-        "market_closed": _providers_base._is_forex_weekend(now),
-        "timeframes": timeframes,
-    })
-
-
-def publish(obj: dict, path: str) -> None:
-    """Atomic JSON write: temp file + os.replace, so Apache never serves
-    a torn file. allow_nan=False — a NaN slipping through would render
-    the page blank (JSON.parse rejects bare NaN)."""
-    data = json.dumps(obj, allow_nan=False, separators=(",", ":"))
-    tmp = path + ".tmp"
-    with open(tmp, "w", encoding="utf-8") as fh:
-        fh.write(data)
-    os.replace(tmp, path)
-
-
-# ══════════════════════════════════════════════════════════════════
-#  Runtime layer (constructed only from main(); never at import)
-# ══════════════════════════════════════════════════════════════════
-
-def _setup_logging(level: str = "INFO") -> None:
-    log_dir = os.path.join(os.path.dirname(os.path.abspath(__file__)), "logs")
-    os.makedirs(log_dir, exist_ok=True)
-    log_file = os.path.join(log_dir, "dashboard.log")
-
-    fmt = logging.Formatter(
-        "%(asctime)s [%(levelname)s] %(name)s: %(message)s",
-        datefmt="%Y-%m-%d %H:%M:%S")
-
-    file_handler = TimedRotatingFileHandler(
-        log_file, when="h", interval=3, backupCount=240, utc=True)
-    file_handler.setFormatter(fmt)
-
-    console_handler = logging.StreamHandler(sys.stdout)
-    console_handler.setFormatter(fmt)
-
-    logging.basicConfig(
-        level=getattr(logging, level.upper(), logging.INFO),
-        handlers=[file_handler, console_handler],
-        force=True)
-
-
-class DashboardPublisher:
-    def __init__(self, out_dir: str):
-        self.out_dir = out_dir
-        self.snapshots: Dict[str, dict] = {}
-        self.news_payload: Optional[dict] = None
-        self._dp = None
-        self._news_engine: Optional[NewsEngine] = None
-
-    def _provider(self):
-        if self._dp is None:
-            self._dp = make_data_provider(PlatformConfig())
-        return self._dp
-
-    def _news(self) -> NewsEngine:
-        if self._news_engine is None:
-            self._news_engine = NewsEngine(PlatformConfig().news)
-        return self._news_engine
-
-    def install_static(self) -> None:
-        """Copy the repo's static files (page + PHP summary endpoint)
-        into the out dir (atomic), so git pull + restart also updates
-        them. The ai/ cache dir is NOT created here — the daemon runs as
-        a different user than Apache; that dir is a one-time deploy step
-        (deploy/README-dashboard.md)."""
-        static_dir = os.path.join(
-            os.path.dirname(os.path.abspath(__file__)), "static")
-        for name in STATIC_FILES:
-            src = os.path.join(static_dir, name)
-            if not os.path.exists(src):
-                logger.warning(f"{src} missing — not installed")
-                continue
-            dst = os.path.join(self.out_dir, name)
-            shutil.copyfile(src, dst + ".tmp")
-            os.replace(dst + ".tmp", dst)
-
-    def scan_tf(self, tf: Timeframe) -> None:
-        started = time.time()
-        try:
-            report = produce_signal(XAUUSD, tf, self._provider(),
-                                    CONFIRMATIONS[tf])
-            # Advisory AI panel, post-hoc and display-only. Off unless
-            # config.json agent_panel.dashboard is true. The reuse-TTL
-            # cache + max_panels_per_day budget bound the spend against
-            # this process's own DeepSeek cap (a persisting M5 signal
-            # re-serves its cached verdict instead of re-running every
-            # bar); a fresh panel adds ≤~25 s before this TF publishes.
-            attach_panel(report, surface="dashboard")
-            self.snapshots[tf.value] = {
-                "status": "ok",
-                "scanned_at": datetime.now(timezone.utc).isoformat(),
-                "scan_seconds": round(time.time() - started, 2),
-                "error": None,
-                "error_at": None,
-                "report": report_to_webapp_dict(report),
-            }
-            logger.info(f"{tf.value}: {report.final_signal.name} "
-                        f"q={report.signal_quality or '-'} "
-                        f"({time.time() - started:.1f}s)")
-        except Exception as e:
-            # Keep the last good report; the page shows it with an
-            # error badge instead of going blank.
-            logger.exception(f"{tf.value} scan failed")
-            entry = self.snapshots.get(tf.value) or _empty_tf_entry()
-            entry["status"] = "error"
-            entry["error"] = f"{type(e).__name__}: {e}"
-            entry["error_at"] = datetime.now(timezone.utc).isoformat()
-            self.snapshots[tf.value] = entry
-
-    def refresh_news(self) -> None:
-        try:
-            ne = self._news()
-            context = ne.get_fundamental_context()
-            week = ne.get_week_events()
-            self.news_payload = build_news_payload(
-                context, week, datetime.now(timezone.utc))
-        except Exception:
-            logger.exception("news refresh failed — keeping last payload")
-
-    def publish_snapshot(self) -> None:
-        publish(build_snapshot_payload(
-            self.snapshots, datetime.now(timezone.utc)),
-            os.path.join(self.out_dir, SNAPSHOT_FILE))
-
-    def publish_news(self) -> None:
-        if self.news_payload is not None:
-            publish(self.news_payload, os.path.join(self.out_dir, NEWS_FILE))
-
-    def publish_all(self) -> None:
-        self.publish_snapshot()
-        self.publish_news()
-
-    def run_cycle(self, tfs: List[Timeframe]) -> None:
-        for tf in tfs:  # sequential — see module docstring (cache dedup)
-            self.scan_tf(tf)
-        # Snapshot lands BEFORE the news refresh: when the news TTL has
-        # expired, refresh_news does FF + 4 RSS fetches (and possibly
-        # DeepSeek calls) — 20-30 s that must not keep fresh signals off
-        # the page.
-        self.publish_snapshot()
-        self.refresh_news()
-        self.publish_news()
-
-    def run_forever(self) -> None:
-        self.run_cycle(DASHBOARD_TFS)  # populate the page at boot
-        while True:
-            delay = seconds_until_boundary(5, buffer=SCAN_BUFFER_SECONDS)
-            logger.info(f"next M5 boundary in {delay:.0f}s")
-            time.sleep(delay)
-            try:
-                due = due_timeframes(datetime.now(timezone.utc))
-                logger.info(f"boundary wake — scanning {[t.value for t in due]}")
-                self.run_cycle(due)
-            except Exception:
-                logger.exception("cycle failed")
-
-
-def main(argv=None) -> None:
-    ap = argparse.ArgumentParser(description="XAU/USD static dashboard publisher")
-    ap.add_argument("--once", action="store_true",
-                    help="one scan+publish cycle, then exit")
-    ap.add_argument("--out",
-                    default=os.environ.get("DASHBOARD_OUT_DIR", DEFAULT_OUT_DIR),
-                    help="output directory served by Apache "
-                         f"(default: $DASHBOARD_OUT_DIR or {DEFAULT_OUT_DIR})")
-    args = ap.parse_args(argv)
-
-    _setup_logging()
-    os.makedirs(args.out, exist_ok=True)
-    pub = DashboardPublisher(args.out)
-    pub.install_static()
-    if args.once:
-        pub.run_cycle(DASHBOARD_TFS)
-        print(f"Published to {args.out}")
-        return
-    pub.run_forever()
-
-
-if __name__ == "__main__":
-    main()

deploy/README-dashboard.md

@@ -1,157 +0,0 @@
-# Deploying the web dashboard (live LAMP server)
-
-The dashboard is a **static publisher**: `dashboard_publisher.py` runs as its
-own daemon, scans XAU/USD at candle closes, and atomically writes
-`dashboard.html` + `news_summary.php` + `snapshot.json` + `news.json` into a
-folder Apache already serves. Apache does all web serving — no new port, no
-Python web server, and the telegram bot process is completely unaffected by
-anything that happens here (see the module docstring for the independence
-rules). The one dynamic piece is `news_summary.php` (§6): an on-demand
-DeepSeek summary for a clicked headline, whitelist-gated against news.json.
-
-All steps below run **on the live server** (`/home/Forex/ForexBot2`).
-
-## 1. Get the code
-
-```bash
-cd /home/Forex/ForexBot2
-git pull
-venv/bin/python test_all.py        # optional sanity: suite must be green
-```
-
-No new Python dependencies are required.
-
-## 2. Make the web folder writable
-
-The dashboard publishes straight into the portal docroot `/var/www/trade/`
-(owner decision 2026-08-03) — it adds exactly four files there:
-`dashboard.html`, `news_summary.php`, `snapshot.json`, `news.json` (plus the
-apache-owned `ai/` cache dir from §6). The daemon user must be able to write
-them:
-
-```bash
-# if the portal files can be owned by the daemon user:
-sudo chown Forex: /var/www/trade
-# otherwise grant write without changing ownership:
-sudo setfacl -m u:Forex:rwx /var/www/trade
-```
-
-(A different path is one edit away: the unit's `Environment=DASHBOARD_OUT_DIR=`
-line, or `--out` for one-off runs.)
-
-SELinux (RHEL/Alma/CentOS enforcing): directories under `/var/www` normally
-already carry `httpd_sys_content_t`, and new files inherit the parent label, so
-no action is usually needed. If Apache 403s despite correct permissions:
-
-```bash
-sudo restorecon -R /var/www/trade
-```
-
-## 3. Smoke-test once
-
-```bash
-venv/bin/python dashboard_publisher.py --once
-ls /var/www/trade    # dashboard.html snapshot.json news.json added next to the portal
-```
-
-Then open `http://<server>/dashboard.html` (path relative to however the
-vhost maps `/var/www/trade`) — all four timeframe cards, the calendar, and
-the newswire should populate.
-
-## 4. Install the daemon
-
-```bash
-sudo cp deploy/gold-dashboard.service /etc/systemd/system/
-# edit User= if the repo owner is not "Forex"
-sudo systemctl daemon-reload
-sudo systemctl enable --now gold-dashboard
-systemctl status gold-dashboard
-tail -f logs/dashboard.log
-```
-
-The daemon scans all four timeframes at boot, then wakes at every M5
-boundary (+10 s) and rescans whichever timeframes just closed. Logs rotate
-every 3 h (240 backups), same scheme as the bot.
-
-## 5. Optional Apache polish
-
-The page polls its JSON with conditional GETs (`cache: "no-cache"` → Apache
-answers unchanged files with 304), so no Apache config is strictly needed. To
-be explicit anyway:
-
-```apache
-<Directory "/var/www/trade">
-    <FilesMatch "\.json$">
-        Header set Cache-Control "no-cache"
-    </FilesMatch>
-</Directory>
-```
-
-The page is deliberately public (owner decision 2026-08-03). To restrict it
-later, add standard Apache basic auth (`htpasswd` + `Require valid-user`) on
-that directory — no code changes needed.
-
-## 6. AI summary endpoint (PHP) — one-time setup
-
-Clicking a newswire headline opens a detail dialog that fetches an on-demand
-DeepSeek gold-impact summary from `news_summary.php` (installed next to the
-page by the daemon). Owner decision 2026-08-07: a real on-demand endpoint on
-the public page, accepted with these guards baked into the PHP — the id must
-match a headline in the published news.json (arbitrary text can never reach
-the LLM), each headline's summary is cached forever in `ai/` (≤1 API call per
-headline, ever), and a file-based daily cap fails **closed** on I/O errors.
-
-Requirements: `php` + the curl extension (`php -m | grep curl`).
-
-```bash
-# 1. SELinux: allow httpd outbound HTTPS — WITHOUT this every summary call
-#    dies as a curl timeout even with perfect perms (RHEL-family default).
-sudo setsebool -P httpd_can_network_connect 1
-
-# 2. API key OUTSIDE the docroot, readable by Apache only:
-sudo install -d -o root -g root -m 755 /etc/gold-dashboard
-sudo sh -c 'umask 137; echo "sk-..." > /etc/gold-dashboard/deepseek.key'
-sudo chown root:apache /etc/gold-dashboard/deepseek.key   # mode 640
-
-# 3. apache-owned cache dir (the daemon runs as Forex and can't create it):
-sudo install -d -o apache -g apache -m 755 /var/www/trade/ai
-sudo semanage fcontext -a -t httpd_sys_rw_content_t '/var/www/trade/ai(/.*)?'
-sudo restorecon -Rv /var/www/trade/ai
-```
-
-Knobs (Apache `SetEnv` in the vhost, both optional):
-`AI_SUMMARY_DAILY_CAP` (default 150), `DEEPSEEK_KEY_FILE` (default
-`/etc/gold-dashboard/deepseek.key`).
-
-Verify from the shell — expect 400 / 404 / 200 / 200-from-cache:
-
-```bash
-curl -s "http://<server>/news_summary.php?id=zzz"                 # {"error":"bad_id"}
-curl -s "http://<server>/news_summary.php?id=$(printf 'a%.0s' {1..40})"  # unknown_id
-ID=$(python3 -c "import json;print(json.load(open('/var/www/trade/news.json'))['headlines'][0]['id'])")
-curl -s "http://<server>/news_summary.php?id=$ID"   # real summary (2-12 s)
-curl -s "http://<server>/news_summary.php?id=$ID"   # instant, from ai/<id>.json
-```
-
-## Operational notes
-
-- **Updating**: `git pull` + `systemctl restart gold-dashboard`. The daemon
-  re-copies `static/dashboard.html` and `static/news_summary.php` into the
-  web folder at startup.
-- **If the daemon dies**, systemd restarts it in 30 s; worst case the page
-  shows stale data with its own "updated N min ago" evidence. Apache and the
-  telegram bot are unaffected.
-- **DeepSeek**: the publisher process has its own daily request cap
-  (`deepseek.daily_request_cap`, default 500) separate from the bot's —
-  expect roughly up to ~430 extra calls/day worst case, V4-Flash pricing.
-- **Two independent DeepSeek meters** hit the same API key: the publisher's
-  in-process cap above, and the PHP endpoint's file cap (`ai/.ht_cap.json`,
-  default 150/day). They don't communicate — worst-case spend is their sum;
-  DeepSeek-side spend alerting is the true backstop. Kept separate on
-  purpose (independence rule: the PHP never imports/reads bot code or
-  config.json).
-- **Restart window**: for the first minutes after a daemon restart the new
-  page can meet a pre-upgrade news.json without headline ids — those rows
-  simply render non-clickable until the first news publish.
-- **Credentials**: uses the same env/.env/config.json resolution as the rest
-  of the platform (Oanda candles feed; news needs no key; DeepSeek optional).

deploy/gold-dashboard.service

@@ -1,23 +0,0 @@
-# systemd unit for the XAU/USD static dashboard publisher.
-# Install (as root, on the live server):
-#   cp deploy/gold-dashboard.service /etc/systemd/system/
-#   systemctl daemon-reload && systemctl enable --now gold-dashboard
-# See deploy/README-dashboard.md for the full checklist.
-
-[Unit]
-Description=XAU/USD dashboard publisher (writes static JSON/HTML for Apache)
-After=network-online.target
-Wants=network-online.target
-
-[Service]
-Type=simple
-# Adjust to the account that owns /home/Forex/ForexBot2:
-User=Forex
-WorkingDirectory=/home/Forex/ForexBot2
-Environment=DASHBOARD_OUT_DIR=/var/www/trade
-ExecStart=/home/Forex/ForexBot2/venv/bin/python dashboard_publisher.py
-Restart=always
-RestartSec=30
-
-[Install]
-WantedBy=multi-user.target

docs/architecture_diagram.py

@@ -1,199 +0,0 @@
-"""ForexBot2 architecture diagram → docs/architecture.png.
-
-Regenerate after architecture changes:  venv/bin/python docs/architecture_diagram.py
-(matplotlib only — no graphviz on the dev box)."""
-import os
-import matplotlib
-matplotlib.use("Agg")
-import matplotlib.pyplot as plt
-from matplotlib.patches import FancyBboxPatch, FancyArrowPatch
-
-W, H = 100.0, 100.0
-fig, ax = plt.subplots(figsize=(16, 11.5))
-ax.set_xlim(0, W); ax.set_ylim(0, H); ax.axis("off")
-
-INK = "#263238"; MUT = "#607D8B"; ARR = "#455A64"; HAUL = "#7B8FA1"
-
-def box(x, y, w, h, title, lines, fc, ec, title_fs=8.2, body_fs=6.6, lw=1.3):
-    ax.add_patch(FancyBboxPatch((x, y), w, h,
-                 boxstyle="round,pad=0.35,rounding_size=0.9",
-                 facecolor=fc, edgecolor=ec, linewidth=lw, zorder=2))
-    cy = y + h - 1.9
-    ax.text(x + w / 2, cy, title, ha="center", va="center", fontsize=title_fs,
-            fontweight="bold", color=INK, zorder=3)
-    cy -= 2.3
-    for ln in lines:
-        ax.text(x + w / 2, cy, ln, ha="center", va="center", fontsize=body_fs,
-                color=INK, zorder=3)
-        cy -= 1.85
-
-def band(x, y, w, h, label, color):
-    ax.add_patch(FancyBboxPatch((x, y), w, h,
-                 boxstyle="round,pad=0.3,rounding_size=1.2",
-                 facecolor="none", edgecolor=color, linewidth=1.0,
-                 linestyle=(0, (4, 3)), zorder=1))
-    ax.text(x + 0.9, y + h - 1.4, label, ha="left", va="center",
-            fontsize=7.2, fontweight="bold", color=color, zorder=3)
-
-def arrow(p1, p2, color=ARR, lw=1.5, style="-|>", conn="arc3,rad=0.0",
-          label=None, lx=0, ly=0, ls="solid", fs=6.4, rot=0):
-    ax.add_patch(FancyArrowPatch(p1, p2, arrowstyle=style, mutation_scale=11,
-                 color=color, linewidth=lw, linestyle=ls,
-                 connectionstyle=conn, zorder=4))
-    if label:
-        mx, my = (p1[0] + p2[0]) / 2 + lx, (p1[1] + p2[1]) / 2 + ly
-        ax.text(mx, my, label, fontsize=fs, color=MUT, ha="center",
-                va="center", zorder=5, rotation=rot,
-                bbox=dict(boxstyle="round,pad=0.15", fc="white", ec="none", alpha=0.9))
-
-# ── title ────────────────────────────────────────────────────────────
-ax.text(2, 98.4, "ForexBot2 — XAU/USD Signal-Intelligence Platform",
-        fontsize=15, fontweight="bold", color=INK, ha="left")
-ax.text(2, 96.0, "one pipeline, four delivery surfaces · closed bars only · live == backtest parity",
-        fontsize=8.5, color=MUT, ha="left")
-
-EXT_FC, EXT_EC = "#ECEFF1", "#90A4AE"
-DAT_FC, DAT_EC = "#E3F2FD", "#5C9BD1"
-NWS_FC, NWS_EC = "#E0F2F1", "#3D9C8F"
-COR_FC, COR_EC = "#FFFDF5", "#C9A227"
-REP_FC, REP_EC = "#FFE9A8", "#B8860B"
-DEL_FC, DEL_EC = "#E8F5E9", "#5FAF63"
-BTS_FC, BTS_EC = "#EDE7F6", "#8E6BC6"
-
-# ── external services (top band) ─────────────────────────────────────
-band(1, 85.0, 98, 9.4, "EXTERNAL SERVICES", "#78909C")
-box(2.5, 86.2, 15.5, 6.4, "Oanda v20 API",
-    ["candles (live feed)", "practice orders"], EXT_FC, EXT_EC)
-box(19.5, 86.2, 15.5, 6.4, "Twelve Data · Tiingo",
-    ["registered fallbacks", "(Tiingo: tick stream)"], EXT_FC, EXT_EC)
-box(36.5, 86.2, 15.5, 6.4, "Forex Factory",
-    ["economic calendar", "MyFXBook RSS fallback"], EXT_FC, EXT_EC)
-box(53.5, 86.2, 12.5, 6.4, "News RSS ×4",
-    ["gold / macro", "headlines"], EXT_FC, EXT_EC)
-box(67.5, 86.2, 13.5, 6.4, "DeepSeek API",
-    ["V4-Flash LLM", "thinking: disabled"], EXT_FC, EXT_EC)
-box(82.5, 86.2, 15, 6.4, "Telegram API",
-    ["owner bot +", "subscriber channel"], EXT_FC, EXT_EC)
-
-# ── data & context layer ─────────────────────────────────────────────
-box(2.5, 69.8, 33, 11.0, "providers/  (BaseDataProvider)",
-    ["single-flight fetch · TTL+boundary cache",
-     "forming-candle DROP · validate/clean",
-     "paginated fetch_range · optional tick stream",
-     "active feed knob: data_providers.active = oanda"],
-    DAT_FC, DAT_EC)
-box(38.5, 69.8, 33, 11.0, "news_engine.py  +  llm_client.py",
-    ["calendar events + keyword sentiment (gold-inverse)",
-     "DeepSeek enrichment: batch headline scores,",
-     "event gold-direction, market blurb — display-only,",
-     "keyword fallback, daily request cap"],
-    NWS_FC, NWS_EC)
-box(74.5, 69.8, 23, 11.0, "app_config.py",
-    ["credentials & deployment:",
-     "env → config.json → refuse",
-     "(no hardcoded fallbacks)",
-     ".env loaded at import"],
-    "#FAFAFA", "#9E9E9E")
-
-# ── signal core container ────────────────────────────────────────────
-ax.add_patch(FancyBboxPatch((2.5, 40.5), 95, 26.5,
-             boxstyle="round,pad=0.35,rounding_size=1.2",
-             facecolor=COR_FC, edgecolor=COR_EC, linewidth=1.6, zorder=1))
-ax.text(50, 65.2, "SIGNAL CORE — runs identically in live scans and backtests (closed bars only)",
-        ha="center", fontsize=8.4, fontweight="bold", color="#8a6d1a", zorder=3)
-
-box(4.5, 51.8, 28, 12.0, "engine.py — TechnicalEngine",
-    ["9 scored indicators + ATR context:",
-     "EMA · RSI · MACD · BB · ADX · ROC ·",
-     "Keltner · divergence · candlestick",
-     "regime → adaptive weights → composite"],
-    "white", COR_EC)
-box(4.5, 42.0, 28, 8.4, "pair_signals/xauusd.py — factory",
-    ["ALWAYS used: config.json overlay +",
-     "validated per-TF deltas (hour blocks, trail…)"],
-    "white", COR_EC)
-box(35.5, 42.0, 60, 20.7, "signal_aggregator.py — analyze_pair → analyze_from_dataframes",
-    ["multi-TF agreement  (M5→[M15,H1] · M15→[M5,H1] · H1→[M15,H4] · H4→[H1,D1])",
-     "→ news context (warn-only by default) → diminishing-decay penalties",
-     "→ exhaustion hard block → quality grade A+…F",
-     "→ ordered post-grade gate chain (session · XAU SELL vol/hour · regime · M5 noise)",
-     "→ setup generators: XAU ATR (A+/A) · generic swing · XAU_PB · XAU_CP",
-     "→ entry · SL · TP1/TP2 · trailing plan"],
-    "white", COR_EC, body_fs=6.9)
-
-# ── SignalReport hub ─────────────────────────────────────────────────
-box(31, 33.2, 38, 5.2, "SignalReport",
-    ["signal (−2…+2) · grade · trade setup · reasons · news summary"],
-    REP_FC, REP_EC, title_fs=8.8)
-
-# ── delivery surfaces ────────────────────────────────────────────────
-band(1, 13.2, 98, 17.2, "DELIVERY SURFACES", "#4C8F50")
-box(2.5, 15.0, 12, 12.6, "main.py",
-    ["CLI report", "--detail", "--json"], DEL_FC, DEL_EC)
-box(16, 15.0, 24, 12.6, "oanda_daemon.py + executor",
-    ["practice auto-execution",
-     "book-parity rules: no broker TP,",
-     "5-bar cooldown, once-per-bar",
-     "trail, OPEN_ONLY fills,",
-     "boot reconcile · orders → Oanda"],
-    DEL_FC, DEL_EC, title_fs=7.8)
-box(42.5, 15.0, 31, 12.6, "dashboard_publisher.py  →  Apache",
-    ["independent daemon · scans at candle closes",
-     "atomic publish to /var/www/trade:",
-     "dashboard.html · snapshot.json · news.json",
-     "+ news_summary.php — on-demand AI",
-     "gold-impact summary (whitelist · cache · cap)"],
-    DEL_FC, DEL_EC)
-box(76, 15.0, 21.5, 12.6, "telegram_bot.py",
-    ["candle-boundary auto-scans", "quality threshold + dedup",
-     "alerts + LLM context line", "WS level-breach monitor"],
-    DEL_FC, DEL_EC)
-
-# ── backtest / QA ────────────────────────────────────────────────────
-box(2.5, 3.2, 71.8, 8.0, "backtest/gpu_backtest.py — bar-by-bar replay through the SAME analyze_from_dataframes",
-    ["realism (results-critical): next-open entries · raw-path resolution · spread charged · SL-first ties · close-only trail",
-     "CuPy GPU / NumPy fallback · parquet cache · fed by providers.fetch_range (news off)"],
-    BTS_FC, BTS_EC)
-box(77.5, 3.2, 20, 8.0, "test_all.py",
-    ["359 tests · 22 modules", "no network (mocked)"], "#FAFAFA", "#9E9E9E")
-
-# ── arrows: short structural hops (solid dark) ───────────────────────
-arrow((10, 86.2), (14, 81.2), label="OHLCV", lx=-3.4, ly=0.2)          # oanda → providers
-arrow((27, 86.2), (24, 81.2), ls=(0, (3, 2)), lw=1.1)                  # fallbacks → providers
-arrow((44, 86.2), (48, 81.2))                                          # FF → news
-arrow((59.5, 86.2), (56, 81.2))                                        # RSS → news
-arrow((72, 86.2), (62, 81.2), label="LLM", lx=3.4, ly=0.6)             # deepseek → news
-arrow((19, 69.8), (19, 64.2), label="DataFrames (UTC, closed bars)", lx=0.2, ly=0.9)
-arrow((33.5, 57.5), (35.5, 55.5))                                      # engine → aggregator
-arrow((32.5, 46.0), (35.5, 48.0))                                      # factory → aggregator
-arrow((55, 69.8), (55, 63.1), label="warn-only context", lx=0, ly=1.0)
-arrow((86, 69.8), (86, 67.3), ls=(0, (3, 2)), lw=1.1)                  # config → core
-arrow((50, 40.5), (50, 38.8))                                          # core → report
-arrow((36, 34.5), (8.5, 27.95), conn="arc3,rad=0.12")                  # report → cli
-arrow((40, 33.2), (28, 27.95), conn="arc3,rad=0.06")                   # report → daemon
-arrow((60, 33.2), (58, 27.95))                                         # report → dashboard
-arrow((66, 34.5), (86.5, 27.95), conn="arc3,rad=-0.12")                # report → telegram
-
-# ── long-haul edges (thin dashed, routed through gutters) ────────────
-# telegram bot ↔ Telegram API — right-margin arc, clear of app_config
-arrow((95, 86.2), (95, 27.95), conn="arc3,rad=0.15", color=HAUL,
-      lw=1.2, ls=(0, (4, 2)))
-ax.text(99.2, 57, "alerts / commands", fontsize=6.4, color=MUT,
-        rotation=90, ha="center", va="center")
-# browser click → news_summary.php → DeepSeek
-# (oanda_daemon's order flow is noted inside its box — no long arc)
-arrow((66, 27.95), (73, 86.2), conn="arc3,rad=-0.18", color=HAUL,
-      lw=1.2, ls=(0, (4, 2)),
-      label="on-demand AI summary\n(browser click → PHP)", lx=2.5, ly=11.6)
-# providers.fetch_range → backtester — far-left margin channel
-arrow((1.5, 69.8), (1.5, 11.0), color=BTS_EC, lw=1.1, ls=(0, (3, 2)),
-      label="fetch_range\n(history)", lx=3.6, ly=-1.2)
-# backtester replays the signal core — right gutter between boxes
-arrow((74.0, 11.6), (74.9, 40.3), color=HAUL, lw=1.3)
-ax.text(81.8, 38.9, "replays the exact signal code", fontsize=6.4,
-        color=MUT, ha="center", va="center", zorder=5,
-        bbox=dict(boxstyle="round,pad=0.15", fc="white", ec="none", alpha=0.9))
-
-out = os.path.join(os.path.dirname(os.path.abspath(__file__)), "architecture.png")
-fig.savefig(out, dpi=200, bbox_inches="tight", facecolor="white")
-print(f"saved {out}")

docs/user_guide.html

@@ -69,9 +69,8 @@
 <tr><td><b>Score / confidence</b></td><td>The regime-weighted composite of all nine indicators. Bigger absolute score = stronger consensus.</td></tr>
 <tr><td><b>MTF aligned</b></td><td>The two confirmation timeframes agree with the trade direction. Signals without alignment are filtered out on M5.</td></tr>
 <tr><td><b>Warnings ⚠️</b></td><td>Context, not veto. Backtesting shows warned trades perform about the same — strong momentum often triggers warnings.</td></tr>
-<tr><td><b>Market context 📰</b></td><td>An AI-written one-liner summarizing today's macro backdrop (headlines + high-impact releases) and its likely pull on gold. Informational only — it never blocks or grades a signal.</td></tr>
 <tr><td><b>SL / TP1 / RR</b></td><td>Stop loss sits behind the nearest swing level with an ATR buffer. TP1 is the nearest opposing level. <b>1R</b> = the entry-to-SL distance = the amount you risk.</td></tr>
-<tr><td><b>Trail plan 🔄</b></td><td>The real exit. Once price moves +0.6R in your favor, move the stop to trail 0.75R behind the best price and update it at each candle close. <b>TP1 is advisory — let winners run.</b></td></tr>
+<tr><td><b>Trail plan 🔄</b></td><td>The real exit. Once price moves +0.6R in your favor (+0.8R for pullback setups), move the stop to trail 0.75R behind the best price and update it at each candle close. <b>TP1 is advisory — let winners run.</b></td></tr>
 </table>
 
 <h2>3 · Acting on a signal — step by step</h2>
@@ -130,7 +129,6 @@
 <tr><td><code>/scan</code></td><td>Run an immediate analysis of the primary timeframe.</td></tr>
 <tr><td><code>/autoscan</code></td><td>Toggle automatic scanning (M5 / M15 / H1 / H4 at each candle close).</td></tr>
 <tr><td><code>/status</code></td><td>Uptime, auto-scan state, WebSocket price-monitor state.</td></tr>
-<tr><td><code>/news</code></td><td>Net XAU/USD news sentiment score (informational only — never gates signals).</td></tr>
 <tr><td><code>/cache</code> · <code>/clearcache</code></td><td>Inspect / reset the market-data cache.</td></tr>
 <tr><td><code>/help</code></td><td>Command reference.</td></tr>
 </table>

engine.py

@@ -245,12 +245,6 @@ class TechnicalEngine:
             signal = SignalStrength.NEUTRAL
             conf = 40
 
-        # Partial alignment (plain BUY/SELL, 2-3 of 4 checks) carries no
-        # edge on M5 — squash to NEUTRAL when the research knob is on.
-        if c.ema_partial_neutral and signal in (SignalStrength.BUY, SignalStrength.SELL):
-            signal = SignalStrength.NEUTRAL
-            conf = 40
-
         crossover_desc = ""
         if len(ema_fast) >= 3:
             prev_f, prev_m = ema_fast.iloc[-2], ema_medium.iloc[-2]

llm_client.py

@@ -1,195 +0,0 @@
-"""
-ForexBot2 — DeepSeek LLM client
-================================
-Thin requests-based client for DeepSeek's OpenAI-compatible chat API,
-used by `news_engine` for display/warning-only enrichment (headline
-sentiment, event surprise direction, market summary). Signals are NEVER
-gated on LLM output — every caller has a non-LLM fallback and treats a
-`None` return as "use the fallback".
-
-Design constraints:
-- One attempt per call, no retries: news runs inside scan cycles; the
-  news engine's 600 s context cache is the natural retry cadence.
-- A UTC-day request cap bounds spend even if a bug causes call storms.
-- Cost: DeepSeek caches identical prompt PREFIXES server-side (cache-hit
-  input is ~50x cheaper), so callers pass a STATIC system prompt (no
-  timestamps or interpolation) and put all variable content in the user
-  payload.
-
-This is a module-level singleton (`get_llm_client`) because the daily
-cap counter must survive the per-scan reconstruction of `NewsEngine`.
-"""
-
-import datetime as _dt
-import json
-import logging
-import threading
-import time
-from typing import Any, Dict, Optional, Tuple
-
-import requests
-
-from app_config import deepseek_api_key, get_app_config
-
-logger = logging.getLogger(__name__)
-
-DEEPSEEK_URL = "https://api.deepseek.com/chat/completions"
-DEFAULT_MODEL = "deepseek-v4-flash"
-# 8s timed out on a measured cold-prefix-cache headline batch (2026-08-03);
-# warm-cache calls come back well under that. 12s buys cold-start headroom
-# while still bounding worst-case scan latency (≤3 calls per cache window).
-DEFAULT_TIMEOUT = 12.0
-DEFAULT_DAILY_CAP = 500
-
-
-class DeepSeekClient:
-    """JSON-in/JSON-out chat wrapper. All failures return None."""
-
-    def __init__(
-        self,
-        api_key: str,
-        model: str = DEFAULT_MODEL,
-        timeout: float = DEFAULT_TIMEOUT,
-        daily_cap: int = DEFAULT_DAILY_CAP,
-        enabled: bool = True,
-    ):
-        self._api_key = api_key or ""
-        self._model = model
-        self._timeout = timeout
-        self._daily_cap = daily_cap
-        self._enabled = enabled
-        self._session = requests.Session()
-        self._session.headers.update({
-            "Content-Type": "application/json",
-            "Authorization": f"Bearer {self._api_key}",
-        })
-        self._lock = threading.Lock()
-        self._count_day: Optional[_dt.date] = None
-        self._count = 0
-        self._warned: set = set()
-        # Health stamps for status displays: last successful call and
-        # last failure (ts, kind). Read-only for consumers.
-        self.last_ok: float = 0.0
-        self.last_error: Optional[Tuple[float, str]] = None
-
-    @property
-    def available(self) -> bool:
-        return self._enabled and bool(self._api_key)
-
-    def _warn_once(self, kind: str, msg: str, *args: Any) -> None:
-        if kind in self._warned:
-            logger.debug(msg, *args)
-        else:
-            self._warned.add(kind)
-            logger.warning(msg, *args)
-
-    def _under_cap(self) -> bool:
-        """Count a request against the UTC-day cap; False = don't call."""
-        today = _dt.datetime.now(_dt.timezone.utc).date()
-        with self._lock:
-            if self._count_day != today:
-                self._count_day = today
-                self._count = 0
-                self._warned.discard("cap")
-            if self._count >= self._daily_cap:
-                self._warn_once(
-                    "cap", "DeepSeek daily request cap (%d) exhausted — "
-                    "falling back to keyword analysis until UTC midnight",
-                    self._daily_cap)
-                return False
-            self._count += 1
-            return True
-
-    def chat_json(self, system_prompt: str,
-                  user_payload: Dict[str, Any]) -> Optional[Dict[str, Any]]:
-        """One chat call; returns the model's JSON object or None."""
-        if not self.available or not self._under_cap():
-            return None
-        body = {
-            "model": self._model,
-            "messages": [
-                {"role": "system", "content": system_prompt},
-                {"role": "user",
-                 "content": json.dumps(user_payload, sort_keys=True)},
-            ],
-            "response_format": {"type": "json_object"},
-            "temperature": 0,
-            # V4-Flash is a hybrid thinking model; without this flag it
-            # burns hundreds of reasoning tokens per call — measured
-            # 2026-08-03: a 25-headline batch blew straight through the
-            # timeout AND could exhaust max_tokens before the answer,
-            # returning empty content. The doctrine + few-shot examples
-            # in the system prompts carry the quality instead.
-            "thinking": {"type": "disabled"},
-            # Must fit the largest reply: a full sentiment batch is
-            # ~10 tokens per headline; 1000 truncated mid-JSON on a real
-            # RSS cycle (2026-08-03).
-            "max_tokens": 2000,
-        }
-        try:
-            resp = self._session.post(
-                DEEPSEEK_URL, json=body, timeout=self._timeout)
-        except requests.RequestException as e:
-            self.last_error = (time.time(), "network")
-            self._warn_once("network", "DeepSeek call failed: %s", e)
-            return None
-        if resp.status_code != 200:
-            self.last_error = (time.time(), f"http_{resp.status_code}")
-            self._warn_once(
-                f"http_{resp.status_code}",
-                "DeepSeek HTTP %s: %s", resp.status_code, resp.text[:200])
-            return None
-        try:
-            parsed = self._parse_response(resp.json())
-        except ValueError:
-            parsed = None
-        if parsed is None:
-            self.last_error = (time.time(), "parse")
-            self._warn_once("parse", "DeepSeek response was not the "
-                            "expected JSON object — ignoring")
-        else:
-            self.last_ok = time.time()
-        return parsed
-
-    @staticmethod
-    def _parse_response(body: Any) -> Optional[Dict[str, Any]]:
-        """Extract the assistant's JSON object from a chat completion."""
-        try:
-            content = body["choices"][0]["message"]["content"]
-        except (KeyError, IndexError, TypeError):
-            return None
-        try:
-            obj = json.loads(content)
-        except (TypeError, ValueError):
-            return None
-        return obj if isinstance(obj, dict) else None
-
-
-# ── Process-wide singleton ────────────────────────────────────────────
-
-_client: Optional[DeepSeekClient] = None
-_client_lock = threading.Lock()
-
-
-def get_llm_client() -> DeepSeekClient:
-    global _client
-    with _client_lock:
-        if _client is None:
-            settings = get_app_config().deepseek_settings
-            _client = DeepSeekClient(
-                api_key=deepseek_api_key(),
-                model=str(settings.get("model", DEFAULT_MODEL)),
-                timeout=float(settings.get("timeout_seconds",
-                                           DEFAULT_TIMEOUT)),
-                daily_cap=int(settings.get("daily_request_cap",
-                                           DEFAULT_DAILY_CAP)),
-                enabled=settings.get("enabled", True) is not False,
-            )
-        return _client
-
-
-def reset_llm_client() -> None:
-    """Drop the singleton (tests / config reloads)."""
-    global _client
-    with _client_lock:
-        _client = None

llm_provider_switch_plan.txt

@@ -1,252 +0,0 @@
-================================================================================
-MULTI-PROVIDER LLM LAYER - PLAN
-(DeepSeek for news summary, Anthropic Claude for agent panel)
-Written 2026-08-08. Plan only - no code has been changed.
-================================================================================
-
-WHY
----
-Today every LLM feature (news enrichment + agent panel) goes through one
-hardwired client: llm_client.py -> DeepSeekClient, a module singleton reached
-via get_llm_client(). This plan describes how to route different features to
-different providers - e.g. keep DeepSeek for cheap high-volume news scoring,
-put the agent panel on Anthropic Claude for better reasoning.
-
-The repo already solved this problem once for market data: the providers/
-package (BaseDataProvider owns shared machinery, subclasses implement only
-_fetch_raw, a registry + one config knob selects the provider). This plan
-mirrors that proven pattern at the LLM layer, scaled down to a single module.
-
-BLAST RADIUS (verified by grep)
--------------------------------
-get_llm_client() call sites:
-  - news_engine.py ....... 4 sites
-  - agent_panel.py ....... 3 sites (+1 timeout getattr)
-  - telegram_bot.py ...... 1 site (/status health; reads private _count,
-                           _count_day, _daily_cap fields)
-  - test_all.py .......... 41 locations stub via  lc._client = _StubLLM(...)
-Out of scope: static/news_summary.php (own PHP DeepSeek call + key file,
-separate deployment).
-
-DESIGN INVARIANTS (carried over unchanged)
-------------------------------------------
-1. Fail-safe contract: every call returns Optional[Dict]; any failure -> None;
-   callers keep their non-LLM fallbacks; LLM output never gates signals.
-2. Single attempt, no retries (news cache cadence is the retry); per-provider
-   UTC daily cap; warn-once logging; health stamps (last_ok, last_error).
-3. Static system prompts stay byte-identical constants - provider-neutral,
-   and they map to prompt caching on BOTH providers (DeepSeek prefix cache;
-   Anthropic cache_control ephemeral).
-4. Test compatibility: all 41 existing lc._client stub sites keep working
-   with zero edits.
-
-================================================================================
-1. llm_client.py REFACTOR  (~195 -> ~430 lines, no new files)
-================================================================================
-
-1a. BaseLLMClient - shared machinery (extracted from today's DeepSeekClient)
-    - __init__(api_key, model, timeout, daily_cap, enabled)
-    - public: name ("deepseek"/"anthropic"), timeout property,
-      usage() -> (used_today, cap)   # so telegram_bot stops reading privates
-    - owns: available property, _warn_once, _under_cap() (UTC-day counter,
-      lock), health stamps, and the template method:
-
-        def chat_json(self, system_prompt, user_payload):
-            if not self.available or not self._under_cap():
-                return None
-            try:
-                return self._request(system_prompt, user_payload)  # hook
-            except <provider-agnostic error mapping>:
-                # stamps last_error, returns None
-
-    - subclasses implement ONLY _request(system_prompt, payload) -> dict|None.
-
-1b. DeepSeekClient(BaseLLMClient) - behavior-identical move
-    Current requests-based logic relocates into _request: same URL,
-    thinking:disabled, response_format:json_object, temperature 0,
-    max_tokens 2000, sort_keys=True payload, _parse_response.
-    Review criterion: Module 19 tests pass UNMODIFIED (they patch
-    client._session, which stays).
-
-1c. AnthropicClient(BaseLLMClient) - new, official SDK
-    - Official `anthropic` Python SDK (add to requirements.txt), NOT raw
-      requests. LAZY IMPORT inside __init__: package missing -> warn-once +
-      client unavailable, so DeepSeek-only deployments never install it.
-    - Constructor: anthropic.Anthropic(api_key=..., timeout=<configured>,
-      max_retries=0)   # house rule: single attempt
-    - _request:
-        resp = self._client.messages.create(
-            model=self._model,                  # default "claude-opus-5"
-            max_tokens=4000,                    # covers thinking + JSON reply
-            output_config={"effort": "low"},    # cheap structured takes
-            system=[{"type": "text", "text": system_prompt,
-                     "cache_control": {"type": "ephemeral"}}],
-            messages=[{"role": "user",
-                       "content": json.dumps(user_payload, sort_keys=True)}],
-        )
-        if resp.stop_reason == "refusal":       # Opus 5: HTTP 200, check first
-            return None
-        text = first text block in resp.content
-        return self._parse_json(text)   # tolerant: strip ```json fences,
-                                        # json.loads, dict check
-    - DELIBERATELY ABSENT (would 400 on Claude Opus 5): temperature (removed
-      on Opus 5 - do NOT copy from the DeepSeek client), budget_tokens,
-      prefills, response_format.
-    - Error mapping: APIConnectionError -> "network",
-      APIStatusError -> "http_<code>", parse fail -> "parse"
-      (same last_error kinds the /status display already understands).
-    - Defaults: model claude-opus-5 (config-overridable, e.g. claude-haiku-4-5
-      for ~5x cheaper), timeout 30s (Claude slower than V4-Flash), own
-      daily_request_cap 500.
-
-1d. Registry + per-consumer routing
-        _PROVIDERS = {"deepseek": DeepSeekClient, "anthropic": AnthropicClient}
-        LLM_CONSUMERS = ("news", "agent_panel")
-        _clients = {}     # keyed by PROVIDER name -> two consumers on the
-                          # same provider share one instance and one cap
-        _client = None    # kept: GLOBAL TEST OVERRIDE
-
-        def get_llm_client(consumer="news"):
-            if _client is not None:          # tests: lc._client = stub
-                return _client               #   -> shadows ALL consumers
-            name = get_app_config().llm_routing.get(consumer, "deepseek")
-            create-once per provider, return _clients[name]
-
-        def reset_llm_client():              # clears _client AND _clients
-
-    Keeping _client as a documented test-override global is what makes the
-    41 existing stub sites work untouched.
-
-================================================================================
-2. CONFIG  (app_config.py + config.json.example)
-================================================================================
-- New top-level "anthropic" block, mirroring "deepseek":
-    {api_key, model, enabled, timeout_seconds, daily_request_cap}
-  plus anthropic_api_key() resolver: ANTHROPIC_API_KEY env -> config.json -> "".
-- New top-level "llm" ROUTING block:
-    {"news": "deepseek", "agent_panel": "deepseek"}
-  Validation: keys within {news, agent_panel}; values within registered
-  provider names. Absent block -> {} -> everything on DeepSeek (today's
-  behavior; zero-migration default). AppConfig.llm_routing: Dict[str, str].
-
-- The target state is then PURE CONFIG - no code changes on switch day:
-
-    "llm":       { "news": "deepseek", "agent_panel": "anthropic" },
-    "anthropic": { "api_key": "", "model": "claude-opus-5", "enabled": true,
-                   "timeout_seconds": 30, "daily_request_cap": 500 }
-
-================================================================================
-3. CONSUMER EDITS  (mechanical, ~10 lines total)
-================================================================================
-- news_engine.py:  4 call sites -> get_llm_client("news")
-- agent_panel.py:  3 call sites -> get_llm_client("agent_panel");
-                   getattr(client, "_timeout", ...) -> public client.timeout
-- telegram_bot.py _news_health_lines: iterate the DISTINCT providers actually
-  routed (news + panel); per provider print "<name>: used/cap calls today"
-  via the new usage() accessor instead of private fields. DeepSeek keeps its
-  "headlines scored" extra; cap/error warnings work off base-class stamps.
-- NO changes to prompts, payload builders, caches, or the panel's own
-  max_panels_per_day budget - all provider-agnostic already.
-
-================================================================================
-4. TESTS  (new "Module 24: LLM provider routing", ~12 tests, ~200 lines)
-================================================================================
-Reuse _StubLLM; new _FakeAnthropic (fake messages.create returning a stub
-response object) injected as client._client.
-
- 1. app_config: anthropic block parses/validates/defaults empty.
- 2. app_config: llm routing block - valid map round-trips; unknown consumer
-    key, unknown provider name, non-dict -> AppConfigError; absent -> {}.
- 3. anthropic_api_key resolves env -> config.json -> empty.
- 4. Base contract on BOTH subclasses (parametrized): keyless/disabled -> no
-    call; daily cap blocks + UTC-roll reset; failure stamps last_error,
-    success stamps last_ok.
- 5. AnthropicClient parse: clean JSON -> dict; ```json-fenced -> dict;
-    non-JSON -> None; stop_reason "refusal" -> None; no text block -> None.
- 6. AnthropicClient request shape: NO temperature key, max_retries=0,
-    cache_control present on system block (assert on captured kwargs).
- 7. AnthropicClient with `anthropic` package missing (simulated) ->
-    available False, chat_json None, no crash at import.
- 8. Routing: no llm block -> both consumers get the SAME DeepSeek singleton
-    (shared cap); routed config -> news gets DeepSeekClient while
-    agent_panel gets AnthropicClient; same-provider consumers share one
-    instance.
- 9. Global lc._client override shadows BOTH consumers (locks the 41 legacy
-    stub sites' semantics).
-10. reset_llm_client() clears both the override and the per-provider cache.
-11. End-to-end: routing agent_panel->anthropic, attach_panel reaches the
-    anthropic fake (5 calls) while a news call reaches the deepseek stub.
-12. telegram _news_health_lines renders per-provider lines from usage()
-    without touching private fields.
-
-GATE: Modules 19 and 23 must pass UNMODIFIED - the no-regression proof for
-the refactor.
-
-================================================================================
-5. DOCS
-================================================================================
-- CLAUDE.md: llm_client bullet -> provider-abstracted (BaseLLMClient;
-  deepseek + anthropic registered); config.json "llm" block routes consumers;
-  absent = all-DeepSeek. Add ANTHROPIC_API_KEY to env-var list. Test counts.
-- config.json.example: both new blocks.
-
-================================================================================
-COMMIT SEQUENCE  (each commit green)
-================================================================================
-1. llm_client: extract BaseLLMClient; DeepSeekClient unchanged behavior
-   (+ base-contract tests)  - pure refactor, Modules 19/23 green untouched.
-2. config: anthropic block + llm consumer-routing block (+ config tests).
-3. llm_client: AnthropicClient (official SDK, lazy import) + requirements.txt
-   (+ parse/shape tests).
-4. routing: per-consumer get_llm_client + health lines - consumer call-site
-   edits, telegram usage() lines, routing/e2e tests.
-5. docs: CLAUDE.md + example config.
-
-================================================================================
-VERIFICATION
-================================================================================
-- python test_all.py green at every commit (~374 -> ~386).
-- Live smoke A (no behavior change): default config, main.py --panel ->
-  still DeepSeek, /status unchanged.
-- Live smoke B (the goal): llm.agent_panel="anthropic" + real
-  ANTHROPIC_API_KEY, run --panel -> one real Claude panel (5 requests,
-  validates JSON contract + latency + refusal-free path) while a news
-  refresh still hits DeepSeek; /status shows two provider lines.
-- Backtest untouched by construction (no LLM in replay path); one 30-day
-  run as a formality.
-
-================================================================================
-EFFORT ESTIMATE
-================================================================================
-0 new files, 8 modified, ~520 lines, est. 3-5 hours.
-
-  llm_client.py        base class + AnthropicClient + routing   +235
-  test_all.py          Module 24                                +200
-  app_config.py        anthropic + llm blocks, resolver          +55
-  telegram_bot.py      per-provider health lines                 +15
-  news_engine.py       consumer tags on call sites                +4
-  agent_panel.py       consumer tags on call sites                +4
-  config.json.example  blocks                                    +14
-  requirements.txt     anthropic                                  +1
-  CLAUDE.md            docs                                      +10
-
-================================================================================
-KEY DECISIONS EMBEDDED IN THIS PLAN
-================================================================================
-- Official Anthropic SDK, not raw requests (current API guidance for Python;
-  lazy import keeps it optional for DeepSeek-only installs).
-- Default Claude model: claude-opus-5 ($5/$25 per MTok; a full 5-call panel
-  ~ 8k in / 500 out ~ $0.05). Config-overridable to claude-haiku-4-5.
-- Two Claude gotchas handled: temperature is REMOVED on Opus 5 (sending it
-  400s - must not be copied from the DeepSeek request body), and refusals
-  arrive as HTTP 200 with stop_reason "refusal" (checked before parsing,
-  mapped to the normal None/fallback path).
-- Caps are per-provider, not per-consumer: two consumers on one provider
-  share its cap (today's semantics); splitting across two providers doubles
-  total headroom naturally.
-- The panel's own max_panels_per_day budget is untouched - it sits above
-  whichever provider serves it.
-- Not in scope: static/news_summary.php (own PHP client/key), streaming,
-  tool use, per-consumer model overrides (add a "model" key per consumer
-  later if ever needed).
-================================================================================

main.py

@@ -20,7 +20,6 @@ if hasattr(sys.stdout, "reconfigure"):
 if hasattr(sys.stderr, "reconfigure"):
     sys.stderr.reconfigure(encoding="utf-8")
 
-from agent_panel import attach_panel
 from config import PlatformConfig, Timeframe, SignalStrength, XAUUSD
 from pair_signals import PAIR_PRODUCERS
 from signal_aggregator import SignalAggregator, SignalReport
@@ -162,15 +161,6 @@ def print_signal_report(report: SignalReport, detail: bool = False):
         print(f"  {Colors.YELLOW}📅{Colors.RESET} {report.event_description}")
     if report.news_description and report.news_description != "Neutral news sentiment":
         print(f"  {Colors.MAGENTA}📰{Colors.RESET} {report.news_description}")
-    if report.news_summary:
-        print(f"  {Colors.MAGENTA}🧠{Colors.RESET} {report.news_summary}")
-    if report.panel_verdict:
-        print(f"  {Colors.CYAN}🤖{Colors.RESET} AI Panel: "
-              f"{Colors.BOLD}{report.panel_verdict}{Colors.RESET} "
-              f"({report.panel_confidence:.0f}%) — {report.panel_rationale}")
-        for t in report.panel_takes:
-            print(f"     {Colors.GRAY}{t['persona']:<11} {t['stance']:<8} "
-                  f"{t['note']}{Colors.RESET}")
 
     # ── 6. DETAIL (only with --detail) ─────────────────────────────
     if detail and report.primary_analysis:
@@ -216,11 +206,6 @@ def report_to_dict(report: SignalReport) -> dict:
         "event_description": report.event_description,
         "news_sentiment": round(report.news_sentiment, 3),
         "news_description": report.news_description,
-        "news_summary": report.news_summary,
-        "panel_verdict": report.panel_verdict,
-        "panel_confidence": round(report.panel_confidence, 1),
-        "panel_rationale": report.panel_rationale,
-        "panel_takes": report.panel_takes,
         "bullish_reasons": report.bullish_reasons,
         "bearish_reasons": report.bearish_reasons,
         "warnings": report.warnings,
@@ -280,9 +265,6 @@ Examples:
                         help="Primary analysis timeframe (default: H1)")
     parser.add_argument("--detail", action="store_true",
                         help="Show full indicator breakdown")
-    parser.add_argument("--panel", action="store_true",
-                        help="Run the LLM agent panel on the result "
-                             "(advisory only; needs a DeepSeek key)")
     parser.add_argument("--json", action="store_true",
                         help="Output as JSON")
     parser.add_argument("--verbose", "-v", action="store_true",
@@ -338,11 +320,6 @@ def run_live(args, config: PlatformConfig):
     aggregator = SignalAggregator(config)
     report = aggregator.analyze_pair(symbol=config.symbol)
 
-    # Explicit user intent: force bypasses the enabled/surface/quality
-    # toggles, but the daily panel budget and LLM cap still apply.
-    if getattr(args, "panel", False):
-        attach_panel(report, surface="cli", force=True)
-
     if args.json:
         print(json.dumps(report_to_dict(report), indent=2))
         return

news_engine.py

@@ -8,8 +8,6 @@ Key principle: High-impact news events override technical signals.
 A perfect technical setup can be destroyed by an unexpected NFP print.
 """
 
-import hashlib
-import html
 import logging
 import re
 import threading
@@ -23,72 +21,10 @@ from typing import Dict, List, Optional, Tuple
 import requests
 
 from config import NewsConfig
-from llm_client import get_llm_client
 
 logger = logging.getLogger(__name__)
 
 
-# ── DeepSeek prompts ─────────────────────────────────────────────────
-# STATIC constants on purpose: DeepSeek prices a repeated byte-identical
-# prompt prefix at the cache-hit rate (~50x cheaper), so nothing here may
-# be interpolated or timestamped. All variable content travels in the
-# user payload. LLM output is display/warning-only — it never gates
-# signals (see NewsConfig.sentiment_affects_signals).
-
-_LLM_SENTIMENT_SYSTEM = """\
-You are a forex/gold news analyst. You receive JSON:
-{"headlines": ["...", ...]}
-For EACH headline, in the same order, score:
-- "s": overall market sentiment of the headline, -1.0 (very bearish)
-  to 1.0 (very bullish), 0.0 if irrelevant or neutral.
-- "xau": the effect on the GOLD (XAU/USD) price, -1.0 (gold down)
-  to 1.0 (gold up), 0.0 if irrelevant.
-Gold doctrine: hawkish Fed / rate hikes / rising real yields / strong
-dollar are gold-bearish; easing / rate cuts / safe-haven flows /
-geopolitical stress / weak dollar are gold-bullish.
-Handle negation correctly ("no rate hike" is not hawkish).
-Reply with ONLY JSON: {"scores": [[s, xau], ...]} — exactly one
-[s, xau] pair per input headline, in the same order, nothing else.
-
-Examples:
-"Fed signals faster rate hikes as inflation stays hot" -> [0.4, -0.7]
-"Gold surges to record high on safe-haven demand" -> [0.8, 0.9]
-"Dollar slips after soft jobs report; Treasury yields fall" -> [-0.4, 0.6]
-"""
-
-_LLM_EVENT_SYSTEM = """\
-You are a forex/gold analyst. You receive one released economic event as
-JSON: {"event", "currency", "actual", "forecast", "previous",
-"vs_forecast"}. "vs_forecast" states whether the actual print came in
-above or below the forecast; it is computed programmatically — trust it
-over your own comparison of the numbers. If it is "unknown", compare
-actual and forecast yourself.
-Decide the likely effect of this RELEASE on the GOLD (XAU/USD) price.
-A stronger-than-forecast USD economy or a hawkish implication is usually
-gold-bearish; weaker data or a dovish implication is usually
-gold-bullish.
-Inverted metrics — for unemployment rate and unemployment/jobless
-claims a LOWER number means a STRONGER labor market: below forecast =
-strong economy = usually gold-bearish; above forecast = weak economy =
-usually gold-bullish.
-Example: {"event": "Unemployment Claims", "actual": "202K", "forecast":
-"217K", "vs_forecast": "below forecast"} ->
-{"gold_direction": "bearish", "reason": "fewer claims show strong labor market"}
-Reply with ONLY JSON:
-{"gold_direction": "bullish"|"bearish"|"neutral", "reason": "<max 10 words>"}
-"""
-
-_LLM_SUMMARY_SYSTEM = """\
-You are a forex/gold market-context writer. You receive JSON with recent
-"headlines" and today's high-impact "events" (released: true means the
-number is out). Write a 1-2 sentence plain-English summary of the
-current macro backdrop for a GOLD (XAU/USD) trader: the dominant theme
-and its likely pull on gold. No advice, no hedging boilerplate, max 45
-words.
-Reply with ONLY JSON: {"summary": "..."}
-"""
-
-
 _RELEASE_VALUE_RE = re.compile(r"(-?\d+(?:\.\d+)?)\s*([kmbt])?", re.IGNORECASE)
 _RELEASE_MULTIPLIERS = {"k": 1e3, "m": 1e6, "b": 1e9, "t": 1e12}
 
@@ -104,44 +40,6 @@ def _parse_release_value(raw: Optional[str]) -> Optional[float]:
     return float(match.group(1)) * multiplier
 
 
-_DESC_MAX_CHARS = 500
-_MAX_LINK_CHARS = 2048
-
-
-def _clean_html_text(raw: Optional[str],
-                     limit: int = _DESC_MAX_CHARS) -> Optional[str]:
-    """RSS <description> → plain text: unescape entities, strip tags,
-    collapse whitespace, truncate. Unescape and strip both run twice
-    because feeds routinely double-encode (&amp;lt;p&amp;gt; wrapping
-    real markup) — the second unescape can reveal fresh tags."""
-    if not raw:
-        return None
-    text = html.unescape(raw)
-    text = re.sub(r"<[^>]+>", " ", text)
-    text = html.unescape(text)
-    text = re.sub(r"<[^>]+>", " ", text)
-    text = re.sub(r"\s+", " ", text).strip()
-    if not text:
-        return None
-    if len(text) > limit:
-        text = text[:limit].rstrip() + "…"
-    return text
-
-
-def _safe_link(raw: Optional[str]) -> Optional[str]:
-    """Article URL for display: http(s) only (an RSS <link> is untrusted
-    and lands in an href), bounded length (Google News redirect URLs run
-    1-2 KB; anything past that is bloat, not a link)."""
-    if not raw:
-        return None
-    link = raw.strip()
-    if not link.lower().startswith(("http://", "https://")):
-        return None
-    if len(link) > _MAX_LINK_CHARS:
-        return None
-    return link
-
-
 @dataclass
 class EconomicEvent:
     """Single economic calendar event."""
@@ -194,11 +92,6 @@ class NewsSentiment:
     # Per-currency score: monetary-policy keywords flip sign for XAU
     # (hawkish is dollar-bullish but gold-bearish).
     sentiment_by_currency: Dict[str, float] = field(default_factory=dict)
-    # Display-only enrichment (dashboard detail dialog). Optional with
-    # defaults so existing constructor calls keep working.
-    link: Optional[str] = None         # article URL (Google News = redirect)
-    description: Optional[str] = None  # plain text, tags stripped, ≤500 chars
-    publisher: Optional[str] = None    # Google News <source> text, else None
 
 
 @dataclass
@@ -212,7 +105,6 @@ class FundamentalContext:
     news_headlines: List[NewsSentiment] = field(default_factory=list)
     event_impact_on_pairs: Dict[str, str] = field(default_factory=dict)
     calendar_available: bool = False  # True if live calendar data was fetched
-    llm_summary: str = ""  # DeepSeek market-context blurb (display-only)
 
 
 class NewsEngine:
@@ -293,25 +185,6 @@ class NewsEngine:
     _headlines_cache: Tuple[float, List[NewsSentiment]] = (0.0, [])
     _EMPTY_RESULT_TTL = 60.0  # retry sooner after a failed/empty fetch
 
-    # DeepSeek result caches. Headlines and releases are immutable, so a
-    # scored one never needs re-scoring; bounded by dropping the oldest
-    # insertion. _llm_sent_cache is only touched under _cache_lock (its
-    # writer runs inside _cached_fetch); the other two take the lock
-    # around cache access but keep HTTP calls outside it.
-    _llm_sent_cache: Dict[str, Tuple[float, float]] = {}   # sha1 → (s, xau)
-    _llm_event_cache: Dict[Tuple[str, str, str], str] = {}
-    _llm_summary_cache: Tuple[float, str] = (0.0, "")
-    _LLM_SENT_CACHE_MAX = 400
-    _LLM_EVENT_CACHE_MAX = 100
-
-    # Per-source fetch health for status displays (telegram /status).
-    # name → {"ok": ts|None, "err": ts|None, "detail": str, "extra": str}.
-    # All writers run under _cache_lock (fetches happen inside
-    # _cached_fetch); readers render without the lock — worst case a
-    # display line is one cycle stale. A source is "failing" when its
-    # last error is newer than its last success.
-    _source_health: Dict[str, dict] = {}
-
     def __init__(self, config: NewsConfig):
         self.config = config
         self._session = requests.Session()
@@ -391,12 +264,6 @@ class NewsEngine:
         except Exception as e:
             logger.warning(f"News sentiment fetch failed: {e}")
 
-        # 3. LLM market-context summary (display-only)
-        try:
-            context.llm_summary = self._llm_market_summary(context)
-        except Exception as e:
-            logger.debug(f"LLM market summary failed: {e}")
-
         return context
 
     @staticmethod
@@ -444,17 +311,10 @@ class NewsEngine:
                 )
                 if event.minutes_until > -window:
                     if surprised:
-                        desc = (
+                        return "caution", (
                             f"{event.currency} {event.event} released — {surprise} surprise "
                             f"(actual {event.actual} vs forecast {event.forecast})"
                         )
-                        # LLM enrichment touches the DESCRIPTION only —
-                        # `surprise` is numeric-direction and does not know
-                        # whether a beat is gold-bullish; the LLM does.
-                        gold_dir = self._llm_event_direction(event)
-                        if gold_dir:
-                            desc += f" — gold {gold_dir}"
-                        return "caution", desc
                     return "caution", f"{event.currency} {event.event} just released"
 
         return "clear", "No significant events"
@@ -488,24 +348,6 @@ class NewsEngine:
 
         return net, desc
 
-    def get_week_events(
-        self, currencies: Optional[List[str]] = None
-    ) -> List[EconomicEvent]:
-        """Full cached calendar week (past and upcoming), filtered to
-        `currencies` (default: tracked_currencies), sorted by time.
-
-        Read-only accessor for display surfaces (the web dashboard) —
-        rides the same class-level cache as signal scans, so it adds no
-        network I/O. No production signal path calls this.
-        """
-        if currencies is None:
-            currencies = self.config.tracked_currencies
-        events = self._cached_fetch("_calendar_cache", self._fetch_calendar_raw)
-        return sorted(
-            (e for e in events if e.currency in currencies),
-            key=lambda e: e.time,
-        )
-
     # ── Private Methods ────────────────────────────────────────────
 
     # Maps country names in MyFXBook titles to currency codes
@@ -515,36 +357,6 @@ class NewsEngine:
         "canada": "CAD", "new zealand": "NZD",
     }
 
-    @classmethod
-    def _record_source(cls, name: str, ok: bool, detail: str = "",
-                       extra: str = "") -> None:
-        """Track one dependency's fetch outcome for status displays.
-        Logs state TRANSITIONS so the log file carries the story without
-        a line every 10-min cycle: first failure WARNING (with detail),
-        repeat failures DEBUG, recovery INFO."""
-        h = cls._source_health.setdefault(
-            name, {"ok": None, "err": None, "detail": "", "extra": ""})
-        was_failing = h["err"] is not None and (
-            h["ok"] is None or h["err"] > h["ok"])
-        if ok:
-            h["ok"] = time.time()
-            h["extra"] = extra
-            if was_failing:
-                logger.info("Source recovered: %s", name)
-        else:
-            h["err"] = time.time()
-            h["detail"] = detail
-            if was_failing:
-                logger.debug("Source still failing: %s (%s)", name, detail)
-            else:
-                logger.warning("Source failing: %s (%s)", name, detail)
-
-    @staticmethod
-    def source_is_failing(entry: Optional[dict]) -> bool:
-        """True when the entry's last error is newer than its last ok."""
-        return bool(entry and entry["err"] is not None and
-                    (entry["ok"] is None or entry["err"] > entry["ok"]))
-
     def _cached_fetch(self, cache_attr: str, fetch_fn) -> list:
         """Return cached data for `cache_attr` or refetch when the TTL
         expired. Successful results live for context_cache_ttl_seconds;
@@ -577,27 +389,14 @@ class NewsEngine:
         Returns an empty list (with a log warning) if unavailable —
         we never generate fake events since that causes constant false alarms.
         """
-        sources = (
+        for fetcher in (self._try_fetch_calendar_api, self._try_fetch_myfxbook_rss):
-            (self._try_fetch_calendar_api, "calendar:forexfactory",
-             "ForexFactory"),
-            (self._try_fetch_myfxbook_rss, "calendar:myfxbook", "MyFXBook"),
-        )
-        for fetcher, src, label in sources:
             try:
                 events = fetcher()
                 if events:
-                    self._record_source(src, True)
-                    self._record_source(
-                        "calendar", True,
-                        extra=f"{len(events)} events via {label}")
                     return events
-                self._record_source(src, False, "no events returned")
             except Exception as e:
-                self._record_source(
-                    src, False, f"{type(e).__name__}: {e}")
                 logger.debug(f"Calendar fetch exception ({fetcher.__name__}): {e}")
 
-        self._record_source("calendar", False, "all sources failed")
         logger.info(
             "Economic calendar data unavailable. "
             "Event risk checks will be skipped for this cycle."
@@ -744,16 +543,11 @@ class NewsEngine:
         """Fetch and analyze news headlines for sentiment."""
         headlines = []
         seen_titles = set()
-        ok_feeds = 0
 
         for feed_url in self.config.rss_feeds:
-            host = feed_url.split("/")[2] if "//" in feed_url else feed_url
-            kept_before = len(headlines)
             try:
                 resp = self._session.get(feed_url, timeout=10)
                 if resp.status_code != 200:
-                    self._record_source(
-                        f"rss:{host}", False, f"HTTP {resp.status_code}")
                     continue
 
                 root = ET.fromstring(resp.content)
@@ -770,8 +564,8 @@ class NewsEngine:
                         pub_date_el.text if pub_date_el is not None else None
                     )
 
-                    # Only recent headlines (configurable: headline_max_age_seconds)
+                    # Only recent headlines (last 24h)
-                    if pub_date and (datetime.now(timezone.utc) - pub_date).total_seconds() > self.config.headline_max_age_seconds:
+                    if pub_date and (datetime.now(timezone.utc) - pub_date).total_seconds() > 86400:
                         continue
 
                     # Dedup within and across feeds so a syndicated story
@@ -786,21 +580,6 @@ class NewsEngine:
                     currencies = self._extract_currencies(title)
 
                     if currencies:
-                        link_el = item.find("link")
-                        desc_el = item.find("description")
-                        src_el = item.find("source")
-                        # itertext() survives CDATA and stray child
-                        # elements where .text stops at the first child.
-                        desc = _clean_html_text(
-                            "".join(desc_el.itertext())
-                            if desc_el is not None else None)
-                        # Google News descriptions are often just the
-                        # headline again inside an anchor — worthless.
-                        if desc and desc.strip() == title.strip():
-                            desc = None
-                        publisher = None
-                        if src_el is not None:
-                            publisher = (src_el.text or "").strip() or None
                         headlines.append(NewsSentiment(
                             headline=title,
                             source=feed_url,
@@ -812,32 +591,11 @@ class NewsEngine:
                                 cur: self._analyze_headline_sentiment(title, cur)
                                 for cur in currencies
                             },
-                            link=_safe_link(
-                                link_el.text if link_el is not None else None),
-                            description=desc,
-                            publisher=publisher,
                         ))
 
-                ok_feeds += 1
-                self._record_source(
-                    f"rss:{host}", True,
-                    extra=f"{len(headlines) - kept_before} kept")
             except Exception as e:
-                self._record_source(
-                    f"rss:{host}", False, f"{type(e).__name__}: {e}")
                 logger.debug(f"Failed to fetch RSS from {feed_url}: {e}")
 
-        self._record_source(
-            "headlines", ok_feeds > 0,
-            detail="all feeds failed" if ok_feeds == 0 else "",
-            extra=(f"{len(headlines)} headlines from "
-                   f"{ok_feeds}/{len(self.config.rss_feeds)} feeds"))
-
-        try:
-            self._apply_llm_sentiment(headlines)
-        except Exception as e:
-            logger.debug(f"LLM sentiment failed, keyword scores stand: {e}")
-
         return headlines
 
     def _analyze_headline_sentiment(
@@ -879,142 +637,6 @@ class NewsEngine:
 
         return (bullish_score - bearish_score) / total
 
-    # ── DeepSeek enrichment (display/warning-only, keyword fallback) ──
-
-    def _llm_headline_scores(
-        self, texts: List[str]
-    ) -> Optional[List[Tuple[float, float]]]:
-        """One batched DeepSeek call → [(overall, xau), ...] aligned with
-        `texts`, or None on any failure/mismatch (caller keeps keyword
-        scores). Tests stub this method directly."""
-        result = get_llm_client().chat_json(
-            _LLM_SENTIMENT_SYSTEM, {"headlines": texts})
-        if not result:
-            return None
-        scores = result.get("scores")
-        if not isinstance(scores, list) or len(scores) != len(texts):
-            return None
-        out = []
-        for item in scores:
-            try:
-                s = max(-1.0, min(1.0, float(item[0])))
-                x = max(-1.0, min(1.0, float(item[1])))
-            except (TypeError, KeyError, ValueError, IndexError):
-                return None
-            out.append((s, x))
-        return out
-
-    def _apply_llm_sentiment(self, headlines: List[NewsSentiment]) -> None:
-        """Overwrite keyword sentiment on `headlines` with LLM scores.
-
-        Runs inside _cached_fetch, i.e. UNDER _cache_lock — cache access
-        is direct and the (rare) HTTP call blocks sibling scans at most
-        once per cache window, matching the existing shared-fetch design.
-        Only headlines missing from the cache are sent; a failed batch
-        changes nothing.
-        """
-        if not headlines or not get_llm_client().available:
-            return
-        cache = NewsEngine._llm_sent_cache
-        keyed = [(hashlib.sha1(h.headline.encode()).hexdigest(), h)
-                 for h in headlines]
-        # Cap the batch so the reply fits the client's max_tokens — the
-        # overflow stays keyword-scored this cycle and, still uncached,
-        # goes out with the next window's batch.
-        new = [(k, h) for k, h in keyed if k not in cache][:25]
-        if new:
-            scores = self._llm_headline_scores([h.headline for _, h in new])
-            if scores is not None:
-                for (k, _), sc in zip(new, scores):
-                    cache[k] = sc
-                while len(cache) > self._LLM_SENT_CACHE_MAX:
-                    cache.pop(next(iter(cache)))
-        for k, h in keyed:
-            sc = cache.get(k)
-            if sc is None:
-                continue
-            overall, xau = sc
-            h.sentiment_score = overall
-            h.sentiment_by_currency = {
-                cur: (xau if cur == "XAU" else overall)
-                for cur in h.currencies_mentioned
-            }
-
-    def _llm_event_direction(self, event: EconomicEvent) -> Optional[str]:
-        """Gold direction for a released surprise, e.g. "bearish: strong
-        jobs lift yields". None → caller keeps the numeric description.
-        Cached per release; failures are NOT cached (retry next scan).
-
-        The actual-vs-forecast comparison is precomputed and sent as
-        "vs_forecast": the 2026-08-06 validation found the model reads
-        close K/M-suffixed pairs backwards ~11% of the time (e.g. claims
-        202K vs 217K called "higher than forecast") when left to compare
-        the raw numbers itself."""
-        key = (event.event, event.currency, str(event.actual or ""))
-        with NewsEngine._cache_lock:
-            if key in NewsEngine._llm_event_cache:
-                return NewsEngine._llm_event_cache[key] or None
-        client = get_llm_client()
-        if not client.available:
-            return None
-        vs_forecast = {
-            "positive": "above forecast",
-            "negative": "below forecast",
-            "inline": "in line with forecast",
-        }.get(event.surprise or "", "unknown")
-        result = client.chat_json(_LLM_EVENT_SYSTEM, {
-            "event": event.event,
-            "currency": event.currency,
-            "actual": event.actual,
-            "forecast": event.forecast,
-            "previous": event.previous,
-            "vs_forecast": vs_forecast,
-        })
-        if not result:
-            return None
-        direction = result.get("gold_direction")
-        if direction not in ("bullish", "bearish", "neutral"):
-            return None
-        reason = str(result.get("reason", "")).strip()[:80]
-        text = "" if direction == "neutral" else (
-            f"{direction}: {reason}" if reason else direction)
-        with NewsEngine._cache_lock:
-            NewsEngine._llm_event_cache[key] = text
-            while len(NewsEngine._llm_event_cache) > self._LLM_EVENT_CACHE_MAX:
-                NewsEngine._llm_event_cache.pop(
-                    next(iter(NewsEngine._llm_event_cache)))
-        return text or None
-
-    def _llm_market_summary(self, context: FundamentalContext) -> str:
-        """1-2 sentence macro blurb for alerts; "" when unavailable.
-        Cached (success OR failure) for the context-cache TTL so at most
-        one call is made per window."""
-        with NewsEngine._cache_lock:
-            ts, cached = NewsEngine._llm_summary_cache
-            if time.time() - ts < self.config.context_cache_ttl_seconds:
-                return cached
-        high_impact = [e for e in context.events_today if e.is_high_impact]
-        if not context.news_headlines and not high_impact:
-            return ""
-        client = get_llm_client()
-        if not client.available:
-            return ""
-        result = client.chat_json(_LLM_SUMMARY_SYSTEM, {
-            "headlines": [h.headline for h in context.news_headlines[:15]],
-            "events": [
-                {"event": e.event, "currency": e.currency,
-                 "actual": e.actual, "forecast": e.forecast,
-                 "released": not e.is_upcoming}
-                for e in high_impact[:10]
-            ],
-        })
-        summary = ""
-        if result:
-            summary = str(result.get("summary", "")).strip()[:300]
-        with NewsEngine._cache_lock:
-            NewsEngine._llm_summary_cache = (time.time(), summary)
-        return summary
-
     def _extract_currencies(self, headline: str) -> List[str]:
         """Extract mentioned currencies from a headline."""
         headline_lower = headline.lower()

oanda_daemon.py

@@ -90,17 +90,8 @@ STATE_FILE = os.path.join(
 def _oanda_setting_int(env_key: str, json_key: str, default: int) -> int:
     env = os.environ.get(env_key)
     if env:
-        try:
         return int(env)
-        except ValueError:
-            raise SystemExit(
-                f"{env_key}={env!r} is not an integer — refusing to start")
-    try:
     return int(get_app_config().oanda_settings.get(json_key, default))
-    except (TypeError, ValueError):
-        raise SystemExit(
-            f"config.json oanda.{json_key} is not an integer — "
-            f"refusing to start")
 
 
 def _oanda_setting_str(env_key: str, json_key: str, default: str) -> str:
@@ -139,11 +130,8 @@ def _seconds_until_boundary(interval_minutes: int, buffer: int = 10) -> float:
     elapsed_seconds = elapsed_minutes * 60 + now.second + now.microsecond / 1e6
     interval_seconds = interval_minutes * 60
     remaining = interval_seconds - elapsed_seconds + buffer
-    # NOTE: remaining > buffer always (elapsed < interval by construction);
+    if remaining <= buffer:
-    # a call at the exact boundary instant targets the NEXT boundary —
+        remaining += interval_seconds
-    # steady-state loops wake at boundary+buffer, so only a process
-    # started in the boundary microsecond skips one scan (2026-08-04
-    # audit I42; the old `remaining <= buffer` guard was unreachable).
     return remaining
 
 # ── Trade state persistence ───────────────────────────────────────────
@@ -177,13 +165,7 @@ class TradingDaemon:
         base_url = oanda_base_url()
         lot_size = _oanda_setting_int("OANDA_LOT_SIZE", "lot_size", 1)
         self._quality_gate = _oanda_setting_str(
-            "OANDA_QUALITY_GATE", "quality_gate", "C").strip()
+            "OANDA_QUALITY_GATE", "quality_gate", "C")
-        if self._quality_gate not in QUALITY_RANK:
-            # Fail CLOSED: an unrecognized gate must never silently fall
-            # back to the most permissive rank (QUALITY_RANK.get default).
-            raise SystemExit(
-                f"OANDA_QUALITY_GATE/quality_gate {self._quality_gate!r} "
-                f"is not one of {sorted(QUALITY_RANK)} — refusing to start")
         self._max_hold_bars = _oanda_setting_int(
             "OANDA_MAX_HOLD_BARS", "max_hold_bars", 300)
         self._price_precision = PAIR_INFO[SYMBOL]["price_precision"]
@@ -293,30 +275,19 @@ class TradingDaemon:
     # ── Position discovery (restart recovery) ──────────────────────
 
     def _discover_existing_positions(self) -> None:
-        self._reconcile_positions("boot discovery")
-
-    def _reconcile_positions(self, reason: str) -> None:
-        """Re-sync tracked state with the broker's open trades. Safe to
-        call mid-run (lock-guarded); used at boot and whenever a fill or
-        placement error suggests state and broker may have diverged."""
         try:
             open_trades = self._executor.get_open_trades(INSTRUMENT)
         except OandaAPIError:
-            logger.exception(
+            logger.exception("Failed to query open trades at boot")
-                "Failed to query open trades (%s) — reconcile skipped",
-                reason)
             return
 
-        live_ids = {t["id"] for t in open_trades}
-
-        with self._state_lock:
         saved = self._state.get("trades", {})
+        live_ids = {t["id"] for t in open_trades}
 
         stale = [tid for tid in saved if tid not in live_ids]
         for tid in stale:
             logger.info(
-                    "Stale trade %s in state file (closed on Oanda), "
+                "Stale trade %s in state file (closed on Oanda), removing", tid)
-                    "removing", tid)
             del saved[tid]
 
         for trade in open_trades:
@@ -328,26 +299,11 @@ class TradingDaemon:
                     saved[tid].get("entry"), saved[tid].get("current_sl"))
                 continue
 
-                direction = ("BUY" if float(trade.get("currentUnits", 0)) > 0
+            direction = "BUY" if float(trade.get("currentUnits", 0)) > 0 else "SELL"
-                             else "SELL")
             entry = float(trade.get("price", 0))
-                sl_order = trade.get("stopLossOrder") or {}
+            sl_order = trade.get("stopLossOrder", {})
-                sl_price = sl_order.get("price")
+            sl = float(sl_order.get("price", entry))
-                if sl_price is None:
+            risk = abs(entry - sl) if sl else 1.0
-                    # No broker-side stop at all: risk is undefined, so the
-                    # trailing model cannot manage this trade. Scream — a
-                    # naked position needs a human (or a policy) to set a
-                    # stop; fabricating one here would invent trading
-                    # behavior the books never validated.
-                    logger.critical(
-                        "Adopted trade %s (%s @ %.2f) has NO STOP LOSS on "
-                        "the broker — it will NOT be trailed. Set a stop "
-                        "manually or close it.", tid, direction, entry)
-                    sl = entry
-                    risk = 0.0
-                else:
-                    sl = float(sl_price)
-                    risk = abs(entry - sl)
 
             saved[tid] = {
                 "tf": "H1",
@@ -372,7 +328,7 @@ class TradingDaemon:
         self._state["trades"] = saved
         _save_state(self._state)
         logger.info(
-            "Position reconcile (%s): %d active trade(s)", reason, len(saved))
+            "Position discovery: %d active trade(s)", len(saved))
 
     # ── Candle-boundary loop ───────────────────────────────────────
 
@@ -565,15 +521,9 @@ class TradingDaemon:
                     stop_loss=ts.stop_loss,
                 )
             except OandaAPIError:
-                # The order is never retried (a timeout after a server-side
-                # fill would double-up on retry), but the server MAY have
-                # filled before the error reached us — reconcile so a live
-                # position can't stay untracked until the next restart.
                 logger.exception(
-                    "Failed to place %s order for %s %s — reconciling "
+                    "Failed to place %s order for %s %s",
-                    "broker positions",
                     ts.direction, SYMBOL, tf.value)
-                self._reconcile_positions("order placement error")
                 return
 
             fill_tx = result.get("orderFillTransaction")
@@ -585,16 +535,7 @@ class TradingDaemon:
             trade_opened = fill_tx.get("tradeOpened", {})
             trade_id = trade_opened.get("tradeID")
             if not trade_id:
-                # A fill with no tradeOpened means the order interacted
+                logger.warning("No trade ID in fill response")
-                # with an existing broker position (tradesClosed /
-                # tradeReduced on a netting account) — state and broker
-                # have diverged. Surface it and re-sync instead of
-                # silently forgetting a real fill.
-                logger.critical(
-                    "Order FILLED but opened no trade (netting close/"
-                    "reduce?) — fill: closed=%s reduced=%s. Reconciling.",
-                    fill_tx.get("tradesClosed"), fill_tx.get("tradeReduced"))
-                self._reconcile_positions("fill without tradeOpened")
                 return
 
             fill_price = float(fill_tx.get("price", ts.entry_price))
@@ -646,12 +587,8 @@ class TradingDaemon:
     # ── Trailing stop management ───────────────────────────────────
 
     def _manage_trailing(self, tf: Timeframe) -> None:
-        # Copy the per-trade dicts too: the loop mutates info (peak,
-        # bars_held, last_bar_ts) and must not touch live state outside
-        # the lock — another thread may be serializing it in _save_state.
         with self._state_lock:
-            trades = {tid: dict(info)
+            trades = dict(self._state.get("trades", {}))
-                      for tid, info in self._state.get("trades", {}).items()}
 
         for trade_id, info in trades.items():
             if info.get("tf") != tf.value:
@@ -697,14 +634,11 @@ class TradingDaemon:
                     self._executor.close_trade(trade_id)
                 except OandaAPIError:
                     # Keep tracking — the trade is still live on the
-                    # broker (incl. a 200 whose close order was cancelled,
+                    # broker; we retry the close at the next bar.
-                    # e.g. MARKET_HALTED — close_trade raises on no-fill);
-                    # we retry the close at the next bar.
                     logger.exception(
                         "Failed to close timed-out trade %s — will retry",
                         trade_id)
                     with self._state_lock:
-                        if trade_id in self._state["trades"]:
                         self._state["trades"][trade_id] = info
                         _save_state(self._state)
                     continue
@@ -717,14 +651,6 @@ class TradingDaemon:
             bar_high = float(last_bar["High"])
             bar_low = float(last_bar["Low"])
 
-            if info.get("risk", 0) <= 0:
-                # Adopted naked position (no broker SL → risk undefined):
-                # never manageable by the trailing model — keep it loud
-                # instead of silently skipping forever.
-                logger.warning(
-                    "Trade %s has risk<=0 (no stop loss) — trailing "
-                    "skipped; set a stop manually", trade_id)
-
             new_sl = self._compute_trail(info, bar_high, bar_low)
 
             if new_sl is not None:
@@ -741,11 +667,6 @@ class TradingDaemon:
                             trade_id)
 
             with self._state_lock:
-                # Write back ONLY if the trade is still tracked — another
-                # loop's _check_closed_trades may have popped it while we
-                # iterated (broker-side close). Re-inserting would create
-                # a ghost that blocks new entries.
-                if trade_id in self._state["trades"]:
                 self._state["trades"][trade_id] = info
                 _save_state(self._state)
 

oanda_executor.py

@@ -69,13 +69,6 @@ class OandaExecutor:
                 "instrument": instrument,
                 "units": signed_units,
                 "timeInForce": "FOK",
-                # The book model is one position total; on a netting
-                # account an opposite-side order would silently CLOSE an
-                # existing (possibly untracked) position instead of opening
-                # the signal trade. OPEN_ONLY makes that case fail loudly
-                # (order cancelled) so the divergence surfaces instead of
-                # executing an unintended close.
-                "positionFill": "OPEN_ONLY",
                 "stopLossOnFill": {
                     "price": self._fmt_price(stop_loss),
                 },
@@ -163,18 +156,7 @@ class OandaExecutor:
                f"/trades/{trade_id}/close")
         resp = self._request_with_retry("PUT", url, json={"units": "ALL"})
         data = self._parse_json(resp)
-        close_tx = data.get("orderFillTransaction")
+        close_tx = data.get("orderFillTransaction", {})
-        if not close_tx:
-            # A 200 whose close order was created but NOT filled (e.g.
-            # cancelled with MARKET_HALTED during the settlement pause)
-            # means the position is STILL OPEN. Raising keeps the caller
-            # tracking it and retrying — silently returning here would
-            # orphan a live position.
-            reason = data.get("orderCancelTransaction", {}).get(
-                "reason", "no orderFillTransaction in response")
-            raise OandaAPIError(
-                f"Close of trade {trade_id} did not fill: {reason}",
-                resp.status_code, str(data)[:300])
         close_price = float(close_tx.get("price", 0))
         pl = float(close_tx.get("pl", 0))
         logger.info(

pair_signals/xauusd.py

@@ -117,54 +117,6 @@ campaign: an M5 hour-22 block (+4.8R on 12mo — the burst-heavy
 specific); M15 hour-16 unblock (+0.8R/+57 trades 12mo but -18.3R
 DD +34% on 5yr); every SELL-gate/vol/TP-size relaxation retested
 negative (see memory notes for the full grid).
-
-M5 override 4 — XAU_PB disabled (validated 2026-08-04, Oanda bars,
-$0.65 REAL spread — first campaign validated at measured cost on the
-live feed, per the 2026-08-04 re-pin baseline): m5_pullback_enabled
-False on M5. Loss forensics on the Oanda books identified XAU_PB as
-the largest wrong-direction loss source (12mo: -43.5R of the -156.6R
-wrong-direction bucket; 5yr: -274.5R). Disabling: 12mo +96.6R/588
-PF 1.44 max DD 13.9R halves +51.2/+45.4 vs +82.3R/788 PF 1.26 DD
-23.9R shipped (which FAILS the PF>=1.3 clause at real cost); OOS
-2024-06-01 -> 2025-03-30 +9.5R PF 1.34; 5yr +123.9R/956 PF 1.35 vs
-PF 1.16 with PB. PB's 5yr net contribution (+23.6R) was almost
-entirely the 2022 chop year (+36.1R vs -2.2R without) — a regime
-earner that has been a drag on trend-regime bars since; its tight
-stops are also the most cost-punished at $0.65. PB code and the
-pb_trail_* params STAY for future regime-gated experiments (same
-pattern as cp_enabled). Rejected in the same campaign (4 candidates
-total, N logged): hour-19 block (+3.7R 12mo but -3.3R OOS, -1.5R
-5yr — OOS gate, the classic trap); hours-14/15 block (-6.9R 12mo
-once PB is removed — their raw negativity WAS the PB losses);
-trail_dist_r 0.75 (-2.6R 12mo; the 0.88R average give-back is the
-validated price of riding trends, exits confirmed NOT the loss
-driver: exit-management accounts for 1-2% of lost R vs 50%
-wrong-direction / 49% entry-timing).
-
-M5 override 5 — EMA partial-alignment squash (validated 2026-08-04,
-Oanda bars, $0.65 real spread): ema_partial_neutral True on M5 —
-partial 4-EMA ribbon alignment (2-3 of 4 checks) scores NEUTRAL
-instead of a weak BUY/SELL. Motivated by the indicator audit: partial-
-alignment trades averaged -0.008R/84 vs +0.230R/341 fully aligned.
-Replay (knob mirrored in the backtester's precomputed EMA scorer in
-the same commit — live==backtest parity): 12mo +97.6R/556 PF 1.48
-DD 10.7R halves +47.8/+49.8 vs control +96.6R/588 PF 1.44 DD 13.9R;
-5yr +127.4R/912 PF 1.38 DD 12.8R vs +123.9R/956 PF 1.35 DD 13.2R,
-2022 improves -2.2 -> -1.1; OOS 2024-06-01 -> 2025-03-30 +9.4R
-PF 1.33 vs +9.5R PF 1.34 — a one-trade tie, documented as such (the
-OOS window holds up; it neither prefers nor punishes). Net effect:
-32 fewer marginal trades, every risk-adjusted measure better.
-Rejected in the same round (trial-logged): confirmation-TF swap
-M5->[M15,H4] (12mo +104.0R PF 1.51 DD 10.3 but OOS +7.8R PF 1.27
-vs control +9.5R PF 1.34 and 2022 3x worse — the H4 confirm keeps
-endorsing stale direction in chop; OOS-ordering rejection, same
-precedent as trail dist 1.0); confirm [H1,H4] (-31.8R 12mo);
-triple-confirm [M15,H1,H4] (-4.9R); divergence weight -1/3
-(-3.8R 12mo); candlestick weight +50% (-3.0R 12mo). The weight
-vector keeps proving calibrated as a system: divergence/candlestick/
-bollinger all look mis-weighted in-book and all replay worse when
-"fixed" (bollinger weight-0: +80.9R PF 1.29 DD 23.8 — its constant
-oppose vote is a load-bearing quality bar, like ROC).
 """
 
 from typing import Optional, List
@@ -197,22 +149,9 @@ def make_config(
         config.xau_strategy.blocked_hours_utc = [
             h for h in config.xau_strategy.blocked_hours_utc if h != 21
         ] + [9]
-        # Deliberately NOT mirrored into config.backtest.blocked_hours_utc:
-        # that copy is the generic non-XAU fallback carrying the JSON base
-        # only. Propagating a gold-validated per-TF delta there would leak
-        # XAU-calibrated hours to a future pair (pair-isolation rule).
-        # Every XAU consumer branches to the xau_strategy copy. Enforced
-        # by the Module 13 test "M5 factory appends hour 9 on top of a
-        # JSON base".
         # BUY-only chart patterns on M5 — see module docstring.
         config.xau_strategy.cp_enabled = True
         config.xau_strategy.cp_patterns = ["inv_hs", "double_bottom", "bull_flag"]
-        # XAU_PB pullback entries OFF — largest wrong-direction loss
-        # source at real cost on Oanda bars — see module docstring.
-        config.xau_strategy.m5_pullback_enabled = False
-        # Partial EMA-ribbon alignment scores NEUTRAL on M5 — its
-        # trades carried no edge — see module docstring.
-        config.technical.ema_partial_neutral = True
         # Wider trailing distance on M5 only — see module docstring.
         config.signals.trail_arm_r = 0.60
         config.signals.trail_dist_r = 0.85

providers/__init__.py

@@ -6,13 +6,12 @@ caching, validation, boundary guarantees, and range pagination — the
 behavior the signal pipeline depends on).
 
 Adding a provider = one module implementing `_fetch_raw` + one registry
-entry + one config.json `data_providers` block. The live candle feed is
+entry + one config.json `data_providers` block. The live feed is selected
-selected by config.json `data_providers.active`; the optional
+by config.json `data_providers.active`; registration alone changes
-`data_providers.streaming` scalar selects a different provider for the
+nothing.
-tick stream (level-breach monitor). Registration alone changes nothing.
 """
 
-from typing import Dict, Optional
+from typing import Dict
 
 from app_config import AppConfigError, get_app_config
 from config import PlatformConfig
@@ -30,10 +29,8 @@ PROVIDER_REGISTRY: Dict[str, type] = {
 }
 
 
-def make_data_provider(config: PlatformConfig,
+def make_data_provider(config: PlatformConfig) -> BaseDataProvider:
-                       name: Optional[str] = None) -> BaseDataProvider:
+    """Build the ACTIVE data provider (config.json: data_providers.active)."""
-    """Build a provider by name (default: config.json data_providers.active)."""
-    if name is None:
     name = get_app_config().active_provider
     cls = PROVIDER_REGISTRY.get(name)
     if cls is None:

providers/base.py

@@ -24,7 +24,6 @@ import time
 from abc import ABC, abstractmethod
 from datetime import datetime, timedelta
 from typing import AsyncIterator, Callable, Dict, Optional, Sequence, Tuple
-from zoneinfo import ZoneInfo
 
 import pandas as pd
 
@@ -32,36 +31,19 @@ from config import PAIR_INFO, PlatformConfig, Timeframe, XAUUSD
 
 logger = logging.getLogger(__name__)
 
-# The gold/FX market week is defined in New York time (see
-# _is_forex_weekend); a fixed UTC hour is wrong for half the year.
-_NY_TZ = ZoneInfo("America/New_York")
-
 
 def _is_forex_weekend(now=None) -> bool:
-    """Gold/FX market closed for the weekend.
+    """Forex market closed: Friday 22:00 UTC → Sunday 22:00 UTC."""
-
-    The market week is anchored to NEW YORK time — Friday 17:00 ET →
-    Sunday 18:00 ET — not a fixed UTC hour. During US DST the feed
-    actually stops at 21:00 UTC (measured on 5 years of Oanda bars:
-    159/162 summer Fridays; winter close 22:00, winter reopen Sunday
-    23:00 UTC). The old hardcoded Fri 22:00→Sun 22:00 UTC rule produced
-    two false-"STALE DATA/feed lagging" hours every week (2026-08-04
-    audit I43, fixed per owner decision).
-
-    Kept as three deliberate copies (here, signal_aggregator.py,
-    gpu_backtest._is_xau_blocked_by_calendar) — a Module 21 test asserts
-    they agree hour-by-hour across DST transitions.
-    """
     if now is None:
         now = pd.Timestamp.now(tz="UTC")
-    ny = now.astimezone(_NY_TZ)
+    wd = now.weekday()
-    wd = ny.weekday()
+    hour = now.hour
-    if wd == 4:                      # Friday
+    if wd == 4 and hour >= 22:
-        return ny.hour >= 17
+        return True
-    if wd == 5:                      # Saturday
+    if wd == 5:
+        return True
+    if wd == 6 and hour < 22:
         return True
-    if wd == 6:                      # Sunday
-        return ny.hour < 18
     return False
 
 
@@ -104,15 +86,9 @@ class BaseDataProvider(ABC):
         self.config = config
         self._cache: Dict[str, pd.DataFrame] = {}
         self._cache_timestamps: Dict[str, float] = {}
-        # Requested fetch depth per key: a cached short frame must not
-        # satisfy a caller asking for a longer lookback.
-        self._cache_outputsize: Dict[str, int] = {}
         # The bot shares one provider across a thread pool; cache eviction
         # is check-then-act and needs locking.
         self._cache_lock = threading.Lock()
-        # Single-flight: one API fetch per (symbol, TF) at a time; other
-        # threads wait and then read the cache.
-        self._inflight: Dict[str, threading.Event] = {}
 
     # ── Abstract: the only provider-specific piece ─────────────────
 
@@ -143,28 +119,17 @@ class BaseDataProvider(ABC):
             lookback_days = self.config.data_lookback_days
 
         cache_key = f"{symbol}_{timeframe.value}"
-        outputsize = max(
-            300,
-            min(int(lookback_days * self.BARS_PER_DAY[timeframe]) + 20,
-                self.MAX_BARS_PER_REQUEST),
-        )
-
-        while True:
         now = time.time()
-            cache_ttl = self.CANDLE_SECONDS.get(
+
-                timeframe, self.config.refresh_interval)
+        cache_ttl = self.CANDLE_SECONDS.get(timeframe, self.config.refresh_interval)
         with self._cache_lock:
             cached = self._cache.get(cache_key)
             cached_at = self._cache_timestamps.get(cache_key)
-                cached_size = self._cache_outputsize.get(cache_key, 0)
         if (
             cached is not None
             and cached_at is not None
             and (now - cached_at) < cache_ttl
             and not cached.empty
-                # A caller asking for MORE history than the cached fetch
-                # requested must not be served the short frame silently.
-                and cached_size >= outputsize
         ):
             if (self._is_data_fresh(cached, symbol, timeframe)
                     and self._has_latest_closed_bar(cached, timeframe)):
@@ -173,31 +138,21 @@ class BaseDataProvider(ABC):
             with self._cache_lock:
                 self._cache.pop(cache_key, None)
                 self._cache_timestamps.pop(cache_key, None)
-                    self._cache_outputsize.pop(cache_key, None)
 
-            # Single-flight: concurrent scans for the same (symbol, TF)
+        outputsize = max(
-            # must not each hit the API (the four TF loops fire together
+            300,
-            # at aligned boundaries and Oanda has no client throttle).
+            min(int(lookback_days * self.BARS_PER_DAY[timeframe]) + 20,
-            # First caller claims the key and fetches; the rest wait,
+                self.MAX_BARS_PER_REQUEST),
-            # then re-check the cache.
+        )
-            with self._cache_lock:
-                inflight = self._inflight.get(cache_key)
-                if inflight is None:
-                    self._inflight[cache_key] = threading.Event()
-                    break
-            inflight.wait(timeout=90)
-
-        try:
         df = self._fetch_raw(symbol, timeframe, outputsize)
         df = self._drop_forming_candle(df, timeframe)
         df = self._validate_and_clean(df, symbol, timeframe)
 
-            # The closed-bar guarantee must hold for FRESH responses too,
+        # The closed-bar guarantee must hold for FRESH responses too, not
-            # not just cache hits: the scanner fires seconds after the
+        # just cache hits: the scanner fires seconds after the boundary,
-            # boundary, and a lagging feed can answer without the
+        # and a lagging feed can answer without the just-closed bar — the
-            # just-closed bar — the scan would silently analyze (and
+        # scan would silently analyze (and session-check) the previous
-            # session-check) the previous bar. Give the feed one short
+        # bar. Give the feed one short grace retry, then proceed loudly.
-            # grace retry, then proceed loudly.
         if not df.empty and not self._has_latest_closed_bar(df, timeframe):
             time.sleep(self.FRESH_FETCH_RETRY_SECONDS)
             try:
@@ -225,15 +180,7 @@ class BaseDataProvider(ABC):
         with self._cache_lock:
             self._cache[cache_key] = df
             self._cache_timestamps[cache_key] = now
-                self._cache_outputsize[cache_key] = outputsize
         return df
-        finally:
-            # Release waiters even when the fetch raised — they re-check
-            # the cache and (on a miss) claim the key themselves.
-            with self._cache_lock:
-                ev = self._inflight.pop(cache_key, None)
-            if ev is not None:
-                ev.set()
 
     def fetch_range(
         self,
@@ -252,7 +199,6 @@ class BaseDataProvider(ABC):
         chunks = []
         cursor_end = end
         empty_streak = 0
-        error_streak = 0
         while cursor_end > start:
             cursor_start = max(start, cursor_end - timedelta(days=days_per_chunk))
             try:
@@ -261,27 +207,19 @@ class BaseDataProvider(ABC):
                     start=cursor_start, end=cursor_end,
                 )
             except Exception as e:
-                # A raising chunk is a PROVIDER ERROR (auth, 429, 5xx,
+                # Providers raise for windows with no data (weekends,
-                # malformed payload) — not "no data". Three in a row means
+                # holidays, asset gaps). Skip the window, step back, retry
-                # the rest of the history is unreachable: raise instead of
+                # — but bail after a few consecutive empties.
-                # returning a silently truncated frame (the backtester
-                # would treat it as complete).
                 logger.warning(
                     f"Chunk {cursor_start.date()}–{cursor_end.date()} failed: {e}"
                 )
-                error_streak += 1
+                empty_streak += 1
-                if error_streak >= 3:
+                if empty_streak >= 3:
-                    raise self.DATA_ERROR(
+                    break
-                        f"{symbol} {timeframe.value}: {error_streak} "
-                        f"consecutive chunk failures walking back from "
-                        f"{cursor_end.date()} — refusing to return "
-                        f"truncated history (last error: {e})") from e
                 cursor_end = cursor_start - timedelta(seconds=1)
                 continue
 
             if chunk is None or chunk.empty:
-                # Genuinely empty windows (weekends/holidays/asset gaps)
-                # are normal; several in a row = start of history.
                 empty_streak += 1
                 if empty_streak >= 3:
                     break
@@ -289,18 +227,9 @@ class BaseDataProvider(ABC):
                 continue
 
             empty_streak = 0
-            error_streak = 0
             chunks.append(chunk)
             # Step back; dedup on concat.
-            new_cursor = chunk.index[0] - timedelta(seconds=1)
+            cursor_end = chunk.index[0] - timedelta(seconds=1)
-            if new_cursor >= cursor_end:
-                # A provider ignoring start/end would pin the cursor and
-                # loop forever — refuse to spin.
-                logger.warning(
-                    f"{symbol} {timeframe.value}: pagination made no "
-                    f"progress at {cursor_end} — stopping")
-                break
-            cursor_end = new_cursor
             if len(chunk) < 2:
                 break
 
@@ -314,17 +243,7 @@ class BaseDataProvider(ABC):
         df = self._drop_forming_candle(df, timeframe)
         start_ts = pd.Timestamp(start)
         end_ts = pd.Timestamp(end)
-        df = df[(df.index >= start_ts) & (df.index <= end_ts)]
+        return df[(df.index >= start_ts) & (df.index <= end_ts)]
-        # Coverage sanity: a head materially later than the requested
-        # start means history was truncated (empty-streak bail or provider
-        # depth limit). Loud, not fatal — some assets genuinely start late.
-        if not df.empty and (df.index[0] - start_ts) > pd.Timedelta(days=7):
-            logger.error(
-                f"{symbol} {timeframe.value}: fetched history starts "
-                f"{df.index[0]} but {start_ts} was requested — "
-                f"~{(df.index[0] - start_ts).days} days missing from the "
-                f"head; treat downstream results accordingly")
-        return df
 
     def is_data_fresh(
         self, df: pd.DataFrame, symbol: str, timeframe: Timeframe
@@ -404,19 +323,8 @@ class BaseDataProvider(ABC):
             return df
         candle_seconds = self.CANDLE_SECONDS.get(timeframe, 3600)
         now = pd.Timestamp.now(tz="UTC")
-        # Loop, not a single slice: a misbehaving feed can return MORE
+        if df.index[-1] + pd.Timedelta(seconds=candle_seconds) > now:
-        # than one not-yet-closed bar, and a single iloc[:-1] would let
-        # the residual forming bar through cleaning (future-tolerance is
-        # +5 min) straight into the signal engine.
-        dropped = 0
-        while (not df.empty
-               and df.index[-1] + pd.Timedelta(seconds=candle_seconds) > now):
             df = df.iloc[:-1]
-            dropped += 1
-        if dropped > 1:
-            logger.warning(
-                "%s: feed returned %d forming/future bars — all dropped",
-                timeframe.value, dropped)
         return df
 
     def _is_data_fresh(self, df: pd.DataFrame, symbol: str, timeframe: Timeframe) -> bool:
@@ -464,20 +372,6 @@ class BaseDataProvider(ABC):
             logger.warning(f"{symbol}: Dropping {future_mask.sum()} future timestamps")
             df = df[~future_mask]
 
-        # Non-positive prices are corrupt, never real gold quotes. Reject
-        # the row BEFORE the consistency repair below — repairing High/Low
-        # from a corrupt 0.0 Open/Close would propagate the zero into
-        # previously-valid fields (2026-08-04 audit, I45).
-        nonpositive = (df[["Open", "High", "Low", "Close"]] <= 0).any(axis=1)
-        if nonpositive.any():
-            logger.warning(
-                f"{symbol}: Dropping {nonpositive.sum()} bars with "
-                f"non-positive prices (corrupt feed data)")
-            df = df[~nonpositive]
-            if df.empty:
-                raise self.DATA_ERROR(
-                    f"{symbol}: all bars had non-positive prices")
-
         invalid_ohlc = (
             (df["High"] < df["Low"]) |
             (df["High"] < df["Open"]) |
@@ -492,10 +386,7 @@ class BaseDataProvider(ABC):
                 df.loc[idx, "High"] = max(vals + [df.loc[idx, "High"]])
                 df.loc[idx, "Low"]  = min(vals + [df.loc[idx, "Low"]])
 
-        # inf returns (division by a zero/na Close) would NaN the std and
-        # silently disarm the whole 10σ pass — neutralize them first.
         returns = df["Close"].pct_change()
-        returns = returns.replace([float("inf"), float("-inf")], float("nan"))
         std_ret = returns.std()
         if std_ret > 0:
             dev = returns - returns.mean()

providers/oanda.py

@@ -8,7 +8,6 @@ Oanda's candle JSON, and exposes the real bid/ask spread from MBA candles.
 """
 
 import logging
-import time
 from datetime import datetime
 from typing import Dict, Optional
 
@@ -32,7 +31,6 @@ class OandaDataProvider(BaseDataProvider):
 
     name = "oanda"
     DATA_ERROR = OandaDataError
-    RETRY_DELAY_S = 2.0
 
     OANDA_GRANULARITY = {
         Timeframe.M1: "M1",
@@ -126,35 +124,11 @@ class OandaDataProvider(BaseDataProvider):
         else:
             params["count"] = min(count, self.MAX_BARS_PER_REQUEST)
 
-        # One retry absorbs Oanda's sporadic spurious 401s (~15-20/day on
-        # the practice API — the same token succeeds on adjacent requests)
-        # plus transient 5xx/network faults. A real credential problem
-        # fails both attempts and raises exactly as before. Other 4xx
-        # (bad request, 429) are never retried.
-        last_err: Optional[OandaDataError] = None
-        for attempt in range(2):
-            if attempt:
-                time.sleep(self.RETRY_DELAY_S)
         try:
             resp = self._session.get(url, params=params, timeout=30)
         except requests.RequestException as e:
-                last_err = OandaDataError(
+            raise OandaDataError(
-                    f"Oanda candle fetch failed for {instrument}: {e}")
+                f"Oanda candle fetch failed for {instrument}: {e}") from e
-                last_err.__cause__ = e
-                continue
-            if resp.status_code == 401 or resp.status_code >= 500:
-                last_err = OandaDataError(
-                    f"Oanda candle fetch failed for {instrument}: "
-                    f"HTTP {resp.status_code}: {resp.text[:300]}")
-                if attempt == 0:
-                    logger.warning(
-                        f"Oanda fetch for {instrument} got HTTP "
-                        f"{resp.status_code}; retrying in "
-                        f"{self.RETRY_DELAY_S}s")
-                continue
-            break
-        else:
-            raise last_err
 
         if resp.status_code != 200:
             raise OandaDataError(
@@ -175,37 +149,23 @@ class OandaDataProvider(BaseDataProvider):
         rows = []
         last_bid_close = None
         last_ask_close = None
-        skipped_malformed = 0
         for c in candles:
             if not c.get("complete", False):
                 continue
             mid = c.get("mid", {})
             bid = c.get("bid", {})
             ask = c.get("ask", {})
-            # A candle missing any OHLC field must be SKIPPED, never
-            # defaulted to 0.0: a zero price flows through cleaning as a
-            # "legit" bar and poisons ATR/Keltner/SL math — and a
-            # mid-frame zero Close makes pct_change() emit inf, which
-            # NaNs the std and disarms the 10σ outlier filter for the
-            # whole frame (2026-08-04 audit, I45).
-            if any(mid.get(k) is None for k in ("o", "h", "l", "c")):
-                skipped_malformed += 1
-                continue
             rows.append({
                 "Datetime": pd.Timestamp(c["time"]).tz_convert("UTC"),
-                "Open": float(mid["o"]),
+                "Open": float(mid.get("o", 0)),
-                "High": float(mid["h"]),
+                "High": float(mid.get("h", 0)),
-                "Low": float(mid["l"]),
+                "Low": float(mid.get("l", 0)),
-                "Close": float(mid["c"]),
+                "Close": float(mid.get("c", 0)),
                 "Volume": int(c.get("volume", 0)),
             })
             if bid and ask:
                 last_bid_close = float(bid.get("c", 0))
                 last_ask_close = float(ask.get("c", 0))
-        if skipped_malformed:
-            logger.warning(
-                "Oanda returned %d malformed candle(s) for %s (missing "
-                "OHLC fields) — skipped", skipped_malformed, instrument)
 
         if not rows:
             raise OandaDataError(

providers/tiingo.py

@@ -10,18 +10,15 @@ Tiingo's JSON into the normalized frame.
 
 Every failure mode raises `TiingoDataError` (a `DataIntegrityError`)
 with a specific message — the scan loops log it and carry on; nothing
-fails silently. NOTE: candles and mid-price ticks only. Tiingo's free
+fails silently. NOTE: this is the CANDLE feed only. Tiingo's free
 `/tiingo/fx/top` quote endpoint publishes indicative two-tier spreads
 and must not be used for spread display (removed 2026-07-23) —
-`get_spread` deliberately stays on the base-class default, and the
+`get_spread` deliberately stays on the base-class default.
-WebSocket stream (`iter_ticks`) exposes the mid price only, never
-bid/ask or a derived spread.
 """
 
-import json
 import logging
 from datetime import datetime
-from typing import AsyncIterator, Callable, Optional, Sequence, Tuple
+from typing import Optional
 
 import pandas as pd
 import requests
@@ -201,106 +198,3 @@ class TiingoDataProvider(BaseDataProvider):
             "Fetched %d bars for %s @ %s from Tiingo",
             len(df), ticker, resample)
         return df
-
-    # ── Streaming quotes ───────────────────────────────────────────
-
-    supports_streaming = True
-
-    WS_URL = "wss://api.tiingo.com/fx"
-
-    @staticmethod
-    def _parse_ws_message(raw: str) -> Optional[Tuple[str, float]]:
-        """Parse one WS message; return (tiingo_ticker, mid_price) for
-        quote updates, None for heartbeats ("H"), subscription acks
-        ("I"), malformed JSON, or quotes without a usable price.
-
-        Server error messages ("E", e.g. bad token) RAISE TiingoDataError:
-        Tiingo authenticates inside the socket via the subscribe message,
-        so an auth failure arrives as a data message followed by a clean
-        close — swallowing it would make the caller's reconnect loop
-        hammer the endpoint with no backoff (backoff only grows on
-        exceptions).
-        """
-        try:
-            msg = json.loads(raw)
-        except json.JSONDecodeError:
-            logger.warning("Tiingo WS: malformed JSON, skipping")
-            return None
-
-        mtype = msg.get("messageType")
-        if mtype == "H":
-            return None
-        if mtype == "I":
-            logger.info("Tiingo WS: subscription acknowledged")
-            return None
-        if mtype == "E":
-            raise TiingoDataError(
-                f"Tiingo WS server error: {str(msg.get('response'))[:200]}")
-        if mtype != "A":
-            return None
-
-        data = msg.get("data")
-        if not isinstance(data, list) or len(data) < 8 or data[0] != "Q":
-            return None
-        # Live-verified field order (2026-08-03): ["Q", ticker, timestamp,
-        # bidSize, bid, mid, askSize, ask] — ask is index 7, NOT 6 as the
-        # docs' table suggests (index 6 is askSize).
-        ticker = data[1]
-        bid, mid, ask = data[4], data[5], data[7]
-        if mid is None and bid is not None and ask is not None:
-            mid = (bid + ask) / 2
-        try:
-            return ticker, float(mid)
-        except (TypeError, ValueError):
-            logger.warning("Tiingo WS: quote for %s has no usable price",
-                           ticker)
-            return None
-
-    async def iter_ticks(
-        self,
-        symbols: Sequence[str],
-        on_connect: Optional[Callable[[], None]] = None,
-    ) -> AsyncIterator[Tuple[str, float]]:
-        """One connection's lifetime of (canonical_symbol, mid) ticks.
-
-        The caller owns the reconnect/backoff loop around this generator.
-        Divergences from the Twelve Data reference implementation, both
-        deliberate: `on_connect` fires on the FIRST RECEIVED message (the
-        "I" ack in practice) rather than after the subscribe send, so a
-        rejected token cannot reset the caller's backoff; and there is no
-        client heartbeat task — Tiingo sends "H" frames itself and the
-        websockets library's built-in ping keepalive detects dead sockets.
-        """
-        import websockets
-
-        if self._api_key is None:
-            raise TiingoDataError(
-                "No Tiingo API key. Set the TIINGO_API_KEY env var or "
-                "data_providers.tiingo.api_key in config.json.")
-
-        ws_to_pair = {self._to_tiingo_ticker(s): s for s in symbols}
-        async with websockets.connect(self.WS_URL) as ws:
-            await ws.send(json.dumps({
-                "eventName": "subscribe",
-                "authorization": self._api_key,
-                "eventData": {
-                    "thresholdLevel": 5,
-                    "tickers": sorted(ws_to_pair),
-                },
-            }))
-            logger.info(
-                "Tiingo WS connected, subscribing to %d tickers",
-                len(ws_to_pair))
-            connected = False
-            async for raw in ws:
-                parsed = self._parse_ws_message(raw)
-                if not connected:
-                    connected = True
-                    if on_connect is not None:
-                        on_connect()
-                if parsed is None:
-                    continue
-                pair_symbol = ws_to_pair.get(parsed[0])
-                if pair_symbol is None:
-                    continue
-                yield pair_symbol, parsed[1]

providers/twelve_data.py

@@ -210,12 +210,6 @@ class TwelveDataProvider(BaseDataProvider):
         The TD WS speaks display symbols ("XAU/USD"); ticks are mapped back
         to the canonical pair symbols passed in. The caller owns the
         reconnect/backoff loop around this generator.
-
-        `on_connect` fires on the FIRST RECEIVED frame, not after the
-        subscribe send (2026-08-04 audit H38b, mirroring the Tiingo
-        provider) — firing pre-receive let a rejected API key reset the
-        caller's reconnect backoff on every attempt, pinning it at the
-        minimum forever.
         """
         import websockets
 
@@ -228,14 +222,11 @@ class TwelveDataProvider(BaseDataProvider):
             }
             await ws.send(json.dumps(subscribe))
             logger.info(f"WebSocket connected, subscribed to {len(ws_to_pair)} symbols")
-            hb_task = asyncio.create_task(self._ws_heartbeat(ws))
-            connected = False
-            try:
-                async for raw in ws:
-                    if not connected:
-                        connected = True
             if on_connect is not None:
                 on_connect()
+            hb_task = asyncio.create_task(self._ws_heartbeat(ws))
+            try:
+                async for raw in ws:
                     parsed = self._parse_ws_price(raw)
                     if parsed is None:
                         continue

signal_aggregator.py

@@ -10,12 +10,9 @@ import os
 from dataclasses import dataclass, field
 from datetime import datetime, timezone
 from typing import Dict, List, Optional
-from zoneinfo import ZoneInfo
 
 import pandas as pd
 
-_NY_TZ = ZoneInfo("America/New_York")
-
 from chart_patterns import detect_chart_pattern
 from config import PlatformConfig, SignalStrength, Timeframe
 from providers import BaseDataProvider, make_data_provider
@@ -26,19 +23,15 @@ logger = logging.getLogger(__name__)
 
 
 def _is_forex_weekend(now_utc: datetime) -> bool:
-    """Gold/FX market closed for the weekend: Friday 17:00 → Sunday 18:00
+    """Forex market closed: Friday 22:00 UTC → Sunday 22:00 UTC (Sydney open)."""
-    NEW YORK time (= Fri 21:00 UTC in US summer, 22:00 in winter —
+    wd = now_utc.weekday()
-    matches measured Oanda feed behavior; 2026-08-04 audit I43).
+    hour = now_utc.hour
-    Deliberate copy of providers.base._is_forex_weekend — a Module 21
+    if wd == 4 and hour >= 22:  # Friday after 22:00
-    test asserts the copies agree across DST transitions."""
+        return True
-    ny = now_utc.astimezone(_NY_TZ)
+    if wd == 5:  # All Saturday
-    wd = ny.weekday()
+        return True
-    if wd == 4:                      # Friday
+    if wd == 6 and hour < 22:  # Sunday before 22:00
-        return ny.hour >= 17
-    if wd == 5:                      # Saturday
         return True
-    if wd == 6:                      # Sunday
-        return ny.hour < 18
     return False
 
 
@@ -86,16 +79,6 @@ class SignalReport:
     event_description: str = ""
     news_sentiment: float = 0.0
     news_description: str = ""
-    news_summary: str = ""  # LLM market-context blurb (display-only)
-
-    # LLM agent-panel consensus (advisory/DISPLAY-ONLY). Populated
-    # post-hoc by consumers via agent_panel.attach_panel — NEVER inside
-    # analyze_from_dataframes (backtest parity) and never read by any
-    # gate or scoring path.
-    panel_verdict: str = ""        # "AGREE" / "NEUTRAL" / "OPPOSE" / ""
-    panel_confidence: float = 0.0
-    panel_rationale: str = ""
-    panel_takes: List[Dict] = field(default_factory=list)
 
     final_signal: SignalStrength = SignalStrength.NEUTRAL
     final_score: float = 0.0
@@ -849,13 +832,11 @@ class SignalAggregator:
             sentiment, sent_desc = news.get_pair_sentiment(symbol, context)
             report.news_sentiment = sentiment
             report.news_description = sent_desc
-            report.news_summary = context.llm_summary
         else:
             report.event_risk = "clear"
             report.event_description = ""
             report.news_sentiment = 0.0
             report.news_description = ""
-            report.news_summary = ""
 
         # ── 5. Spread estimate from primary data (avoids extra API call) ─
         try:
@@ -1269,17 +1250,7 @@ class SignalAggregator:
             ema200 = primary.key_levels.get("ema_trend")
             if ema200 and ema200 > 0 and report.current_price > 0:
                 ema_dist_pct = abs(report.current_price - ema200) / ema200 * 100.0
-                # FB2_EMA_LO: research override for the EMA200-proximity
-                # penalty band; the tuned default is 0.5. An unparseable
-                # value must degrade to the default, never crash the scan
-                # (2026-08-04 audit; hardened per owner delegation).
-                try:
                 _lo = float(os.environ.get("FB2_EMA_LO", "0.5"))
-                except (TypeError, ValueError):
-                    logger.warning(
-                        "FB2_EMA_LO=%r is not a number — using tuned "
-                        "default 0.5", os.environ.get("FB2_EMA_LO"))
-                    _lo = 0.5
                 if ema_dist_pct < _lo:
                     report.warnings.append(f"NEAR EMA200 ({ema_dist_pct:.2f}%) — chop risk")
                     penalties.append(("ema200_near", 5))

static/news_summary.php

@@ -1,245 +0,0 @@
-<?php
-/**
- * On-demand gold-impact AI summary for one published headline.
- *
- * The ONLY dynamic piece of the otherwise-static dashboard. Serves
- * GET ?id=<sha1-of-headline> and returns {"summary": ..., "generated_at": ...}.
- *
- * Abuse model (the page is public):
- *   - WHITELIST: the id must match a headline in the local news.json —
- *     arbitrary text can never reach the LLM through this endpoint.
- *   - CACHE FOREVER: headlines are immutable, so each id costs at most
- *     one DeepSeek call, ever (ai/<id>.json). Cache is checked BEFORE
- *     the whitelist on purpose: summaries stay reachable after the
- *     headline ages out of the 24 h feed window.
- *   - DAILY CAP: file-based UTC-day counter, FAIL CLOSED on counter I/O
- *     errors (a perms mistake must not mean unmetered spend).
- *
- * DeepSeek doctrine (mirrors llm_client.py — deliberate local copy, the
- * dashboard never imports bot code and this file never reads the repo's
- * config.json):
- *   - STATIC byte-identical system prompt (prefix-cache pricing).
- *   - "thinking": disabled is MANDATORY for V4-Flash (else it burns the
- *     timeout reasoning), temperature 0, json response_format.
- *   - One attempt, no retries; every failure returns a JSON error the
- *     page renders as a graceful fallback.
- *
- * Key file lives OUTSIDE the docroot: /etc/gold-dashboard/deepseek.key
- * (root:apache 640). Override paths/caps via Apache SetEnv:
- * DEEPSEEK_KEY_FILE, AI_SUMMARY_DAILY_CAP.
- */
-
-const NEWS_FILE = __DIR__ . '/news.json';
-const CACHE_DIR = __DIR__ . '/ai';   // apache-owned; created at deploy time
-const API_URL   = 'https://api.deepseek.com/chat/completions';
-const MODEL     = 'deepseek-v4-flash';
-const TIMEOUT   = 12;
-const MAX_TOKENS = 200;
-const SUMMARY_MAX_CHARS = 400;
-
-// .ht_ prefix: Apache's default <Files ".ht*"> rule denies these even
-// though they sit in the public docroot.
-const CAP_FILE = CACHE_DIR . '/.ht_cap.json';
-
-const SYSTEM_PROMPT = <<<'PROMPT'
-You are a forex/gold news analyst. You receive one news item as JSON:
-{"headline", "description", "score", "xau_score", "keywords", "currencies"}.
-"xau_score" is the platform's automated gold-effect score for this item,
--1.0 (gold down) to 1.0 (gold up); "score" is overall market sentiment.
-Both are automated and can be wrong — judge the news yourself.
-In 1-2 plain-English sentences (max 45 words) explain what this news
-means for the GOLD (XAU/USD) price and WHY it is bullish, bearish, or
-neutral. If your reading clearly contradicts xau_score, add "(the
-automated score disagrees)". Never write the words "xau_score" or
-"score" — a general reader sees this text.
-Gold doctrine: hawkish Fed / rate hikes / rising real yields / strong
-dollar are gold-bearish; easing / rate cuts / safe-haven flows /
-geopolitical stress / weak dollar are gold-bullish.
-Reply with ONLY JSON: {"summary": "..."}
-PROMPT;
-
-function respond(int $status, array $obj, string $cache = 'no-store'): void
-{
-    http_response_code($status);
-    header('Content-Type: application/json; charset=utf-8');
-    header('Cache-Control: ' . $cache);
-    echo json_encode($obj, JSON_UNESCAPED_SLASHES | JSON_UNESCAPED_UNICODE);
-    exit;
-}
-
-function atomic_write(string $path, string $data): bool
-{
-    $tmp = $path . '.tmp.' . getmypid();
-    if (@file_put_contents($tmp, $data) === false) {
-        return false;
-    }
-    if (!@rename($tmp, $path)) {
-        @unlink($tmp);
-        return false;
-    }
-    return true;
-}
-
-// ── 1. id validation ────────────────────────────────────────────────
-$id = $_GET['id'] ?? '';
-if (!preg_match('/^[a-f0-9]{40}$/', $id)) {
-    respond(400, ['error' => 'bad_id']);
-}
-
-// ── 2. cache first (immutable headlines → summaries never change) ───
-$cachePath = CACHE_DIR . '/' . $id . '.json';
-$cached = @file_get_contents($cachePath);
-if ($cached !== false && $cached !== '') {
-    http_response_code(200);
-    header('Content-Type: application/json; charset=utf-8');
-    header('Cache-Control: max-age=3600');
-    echo $cached;
-    exit;
-}
-
-// ── 3. whitelist against the published feed ─────────────────────────
-$news = json_decode((string)@file_get_contents(NEWS_FILE), true);
-if (!is_array($news)) {
-    respond(503, ['error' => 'no_feed']);
-}
-$hit = null;
-foreach ($news['headlines'] ?? [] as $h) {
-    // Fallback sha1(): a publisher that predates the id field still
-    // serves a valid whitelist during the restart window.
-    $hid = $h['id'] ?? sha1($h['headline'] ?? '');
-    if ($hid === $id) {
-        $hit = $h;
-        break;
-    }
-}
-if ($hit === null) {
-    respond(404, ['error'   => 'unknown_id',
-                  'message' => 'Headline is no longer in the feed.']);
-}
-
-// ── 4. daily cap (fail closed) ──────────────────────────────────────
-$dailyCap = (int)(getenv('AI_SUMMARY_DAILY_CAP') ?: 150);
-$capFh = @fopen(CAP_FILE, 'c+');
-if ($capFh === false || !flock($capFh, LOCK_EX)) {
-    respond(503, ['error' => 'cap_io']);
-}
-$today = gmdate('Y-m-d');
-$cap = json_decode((string)stream_get_contents($capFh), true);
-if (!is_array($cap) || ($cap['day'] ?? '') !== $today) {
-    $cap = ['day' => $today, 'count' => 0];
-}
-if ($cap['count'] >= $dailyCap) {
-    flock($capFh, LOCK_UN);
-    fclose($capFh);
-    respond(429, ['error' => 'cap_exceeded']);
-}
-// Increment BEFORE the LLM call — attempts count against the cap
-// (same semantics as llm_client._under_cap).
-$cap['count']++;
-rewind($capFh);
-if (ftruncate($capFh, 0) === false
-        || fwrite($capFh, json_encode($cap)) === false) {
-    flock($capFh, LOCK_UN);
-    fclose($capFh);
-    respond(503, ['error' => 'cap_io']);
-}
-fflush($capFh);
-flock($capFh, LOCK_UN);
-fclose($capFh);
-
-// ── 5. API key ──────────────────────────────────────────────────────
-$keyFile = getenv('DEEPSEEK_KEY_FILE') ?: '/etc/gold-dashboard/deepseek.key';
-$apiKey = trim((string)@file_get_contents($keyFile));
-if ($apiKey === '') {
-    respond(503, ['error' => 'no_key']);
-}
-
-// ── 6. per-id stampede lock, then re-check the cache ────────────────
-$lockFh = @fopen(CACHE_DIR . '/.ht_lock_' . $id, 'c');
-if ($lockFh !== false) {
-    flock($lockFh, LOCK_EX);   // blocks while a concurrent winner generates
-    $cached = @file_get_contents($cachePath);
-    if ($cached !== false && $cached !== '') {
-        flock($lockFh, LOCK_UN);
-        fclose($lockFh);
-        http_response_code(200);
-        header('Content-Type: application/json; charset=utf-8');
-        header('Cache-Control: max-age=3600');
-        echo $cached;
-        exit;
-    }
-}
-
-// ── 7. DeepSeek call (exact chat_json body shape) ───────────────────
-$userPayload = json_encode([
-    'headline'    => $hit['headline'] ?? '',
-    'description' => $hit['description'] ?? null,
-    'score'       => $hit['score'] ?? null,
-    'xau_score'   => $hit['score_by_currency']['XAU'] ?? null,
-    'keywords'    => $hit['keywords'] ?? [],
-    'currencies'  => $hit['currencies'] ?? [],
-], JSON_UNESCAPED_UNICODE);
-
-$body = json_encode([
-    'model'    => MODEL,
-    'messages' => [
-        ['role' => 'system', 'content' => SYSTEM_PROMPT],
-        ['role' => 'user',   'content' => $userPayload],
-    ],
-    'response_format' => ['type' => 'json_object'],
-    'temperature'     => 0,
-    'thinking'        => ['type' => 'disabled'],
-    'max_tokens'      => MAX_TOKENS,
-]);
-
-$ch = curl_init(API_URL);
-curl_setopt_array($ch, [
-    CURLOPT_POST           => true,
-    CURLOPT_POSTFIELDS     => $body,
-    CURLOPT_HTTPHEADER     => ['Content-Type: application/json',
-                               'Authorization: Bearer ' . $apiKey],
-    CURLOPT_RETURNTRANSFER => true,
-    CURLOPT_TIMEOUT        => TIMEOUT,
-]);
-$raw = curl_exec($ch);
-$httpCode = curl_getinfo($ch, CURLINFO_RESPONSE_CODE);
-curl_close($ch);
-
-// ── 8. parse (any failure → 502, never cached) ──────────────────────
-$summary = null;
-if ($raw !== false && $httpCode === 200) {
-    $resp = json_decode($raw, true);
-    $content = $resp['choices'][0]['message']['content'] ?? '';
-    $parsed = json_decode((string)$content, true);
-    if (is_array($parsed) && is_string($parsed['summary'] ?? null)
-            && trim($parsed['summary']) !== '') {
-        $summary = trim($parsed['summary']);
-        if (mb_strlen($summary) > SUMMARY_MAX_CHARS) {
-            $summary = mb_substr($summary, 0, SUMMARY_MAX_CHARS) . '…';
-        }
-    }
-}
-if ($lockFh !== false && $summary === null) {
-    flock($lockFh, LOCK_UN);
-    fclose($lockFh);
-}
-if ($summary === null) {
-    respond(502, ['error' => 'llm_failed']);
-}
-
-// ── 9. cache + respond ──────────────────────────────────────────────
-$out = json_encode(['summary' => $summary, 'generated_at' => gmdate('c')],
-                   JSON_UNESCAPED_SLASHES | JSON_UNESCAPED_UNICODE);
-if (!atomic_write($cachePath, $out)) {
-    // Degrade to per-click generation rather than a broken feature —
-    // the daily cap still bounds spend.
-    error_log('news_summary.php: cache write failed for ' . $cachePath);
-}
-if ($lockFh !== false) {
-    flock($lockFh, LOCK_UN);
-    fclose($lockFh);
-}
-http_response_code(200);
-header('Content-Type: application/json; charset=utf-8');
-header('Cache-Control: max-age=3600');
-echo $out;

telegram_bot.py

@@ -12,15 +12,11 @@ Auto-scan:   Starts automatically on boot if TELEGRAM_CHAT_ID is set.
              M15 fires at :00/:15/:30/:45 + 10s buffer.
              H1  fires at :00 + 10s buffer.
              H4  fires at :00 every 4 hours + 10s buffer.
-WebSocket:   Streams live ticks from the streaming provider — config.json
+WebSocket:   Streams live ticks from TwelveData for XAU/USD.
-             `data_providers.streaming` when set (e.g. tiingo), else the
-             active provider if it streams; auto-disabled otherwise.
              When price crosses a key structure level (swing S/R, BB,
              EMA200) cached from the last boundary scan, triggers an
              immediate M15 scan and sends a Level Break Alert.
              5-minute cooldown prevents spam.
-News:        /news replies with the net XAU/USD news sentiment score
-             (display-only — news never gates signals).
 
 Owner lock:  All commands and buttons only respond to TELEGRAM_CHAT_ID
              (the owner). Everyone else gets a polite refusal — the bot
@@ -71,9 +67,6 @@ from app_config import (
     telegram_channel_id,
 )
 from providers import BaseDataProvider, make_data_provider
-from agent_panel import attach_panel, panel_health
-from llm_client import get_llm_client
-from news_engine import FundamentalContext, NewsEngine
 from pair_signals import PAIR_PRODUCERS
 from signal_aggregator import SignalAggregator, SignalReport
 
@@ -105,12 +98,6 @@ logging.basicConfig(level=getattr(logging, _APP.log_level, logging.INFO),
                     force=True)
 logging.getLogger("httpx").setLevel(logging.WARNING)
 logging.getLogger("httpcore").setLevel(logging.WARNING)
-# Never let DEBUG runs write credentials into the 30-day rotating log:
-# the websockets client logger emits the connect URI (Twelve Data puts
-# the API key in the query string) and frame contents (Tiingo's auth
-# frame carries the key) — 2026-08-04 audit, finding K47.
-logging.getLogger("websockets").setLevel(logging.INFO)
-logging.getLogger("websockets.client").setLevel(logging.INFO)
 logger = logging.getLogger("telegram_bot")
 
 TOKEN = telegram_token()
@@ -216,27 +203,6 @@ def _get_shared_dp() -> BaseDataProvider:
     return _shared_dp
 
 
-_streaming_dp: Optional[BaseDataProvider] = None
-
-
-def _get_streaming_dp() -> BaseDataProvider:
-    """Provider for the tick stream: config.json data_providers.streaming
-    when set (a SEPARATE instance — the candle cache stays on the active
-    provider), else the shared active provider. Reads app config at call
-    time, not the import-time _APP snapshot, so tests can swap it via
-    set_app_config()."""
-    global _streaming_dp
-    if _streaming_dp is None:
-        app = get_app_config()
-        if app.streaming_provider is None or \
-                app.streaming_provider == app.active_provider:
-            _streaming_dp = _get_shared_dp()
-        else:
-            _streaming_dp = make_data_provider(
-                PlatformConfig(), app.streaming_provider)
-    return _streaming_dp
-
-
 # ── Signal formatting ─────────────────────────────────────────────────
 
 _SIGNAL_EMOJI = {
@@ -406,19 +372,6 @@ def format_signal_message(report: SignalReport) -> str:
         extra.append(f"📋 {ind_summary}")
     for w in report.warnings:
         extra.append(f"⚠️ {_escape_md(w)}")
-    if report.news_summary:
-        extra.append(f"📰 {_escape_md(report.news_summary)}")
-    if report.panel_verdict:
-        mark = {"AGREE": "✅", "OPPOSE": "⛔", "NEUTRAL": "➖"}.get(
-            report.panel_verdict, "")
-        extra.append(
-            f"🤖 AI Panel {mark} *{_escape_md(report.panel_verdict)}* "
-            f"({report.panel_confidence:.0f}%) — "
-            f"{_escape_md(report.panel_rationale)}")
-        if report.panel_takes:
-            votes = " · ".join(
-                f"{t['persona']}: {t['stance']}" for t in report.panel_takes)
-            extra.append(f"   _{_escape_md(votes)}_")
     if extra:
         lines += [""] + extra
 
@@ -619,45 +572,6 @@ def _commit_alert(key: str, report: SignalReport) -> None:
         _last_alert_setup.pop(key, None)
 
 
-# Per-key locks serializing the _should_alert → send → _commit_alert
-# window. The auto-scan loop and a concurrent breach handler share the
-# "XAU/USD_15min" dedup key; without the lock both pass _should_alert
-# while the other's send is still awaiting Telegram, and the same signal
-# goes out twice (2026-08-04 audit H36). Per-key rather than one global
-# lock so a slow send on one timeframe cannot head-of-line-block another
-# timeframe's alert. Keys are symbol × TF, so the dict stays tiny.
-_alert_locks: Dict[str, asyncio.Lock] = {}
-
-
-def _alert_lock(key: str) -> asyncio.Lock:
-    lock = _alert_locks.get(key)
-    if lock is None:
-        lock = _alert_locks[key] = asyncio.Lock()
-    return lock
-
-
-async def _send_alert(bot, chat_id: int, text: str) -> bool:
-    """Markdown alert send with a plain-text fallback on parse failures,
-    mirroring _safe_edit (2026-08-04 audit H35 — alert paths previously
-    dropped the message on a BadRequest with only a warning logged).
-    Returns True when a message reached the chat by either attempt."""
-    try:
-        await bot.send_message(chat_id, text, parse_mode=ParseMode.MARKDOWN)
-        return True
-    except BadRequest as exc:
-        logger.warning(f"Markdown alert failed for chat {chat_id} ({exc}); "
-                       f"retrying as plain text")
-        try:
-            await bot.send_message(chat_id, text)
-            return True
-        except Exception as exc2:
-            logger.warning(f"Failed to send alert to chat {chat_id}: {exc2}")
-            return False
-    except Exception as exc:
-        logger.warning(f"Failed to send alert to chat {chat_id}: {exc}")
-        return False
-
-
 # ── Candle-boundary scheduler ────────────────────────────────────────
 
 _autoscan_chats: Set[int] = set()
@@ -686,11 +600,8 @@ def _seconds_until_boundary(interval_minutes: int, buffer: int = 10) -> float:
     elapsed_seconds = elapsed_minutes * 60 + now.second + now.microsecond / 1e6
     interval_seconds = interval_minutes * 60
     remaining = interval_seconds - elapsed_seconds + buffer
-    # NOTE: remaining > buffer always (elapsed < interval by construction);
+    if remaining <= buffer:
-    # a call at the exact boundary instant targets the NEXT boundary —
+        remaining += interval_seconds
-    # steady-state loops wake at boundary+buffer, so only a process
-    # started in the boundary microsecond skips one scan (2026-08-04
-    # audit I42; the old `remaining <= buffer` guard was unreachable).
     return remaining
 
 
@@ -705,13 +616,7 @@ async def _scan_timeframe(
     reports: List[SignalReport] = []
     for i, result in enumerate(results):
         if isinstance(result, Exception):
-            # gather(return_exceptions=True) means there is no active
+            logger.exception(f"Auto-scan failed for {SCAN_PAIRS[i]} {tf.value}: {result}")
-            # exception here — logger.exception would log "NoneType: None"
-            # instead of the real stack; pass the instance explicitly.
-            logger.error(
-                f"Auto-scan failed for {SCAN_PAIRS[i]} {tf.value}: {result}",
-                exc_info=result,
-            )
         else:
             reports.append(result)
 
@@ -720,58 +625,36 @@ async def _scan_timeframe(
 
     _update_level_cache(reports)
 
-    candidates = [
+    actionable = [
         r for r in reports
         if r.final_signal != SignalStrength.NEUTRAL   # same gate as breach path
         and _passes_quality(r.signal_quality)
+        and _should_alert(f"{r.symbol}_{tf.value}", r)
     ]
-    for report in candidates:
+    sent_ok: Set[int] = set()
-        key = f"{report.symbol}_{tf.value}"
-        # The lock spans check → send → commit so a concurrent breach
-        # handler on the same key cannot double-alert (H36).
-        async with _alert_lock(key):
-            if not _should_alert(key, report):
-                continue
-            # Panel AFTER the quality gate + dedup decision so no LLM
-            # spend on suppressed alerts. Runs on the scan executor to
-            # keep the event loop free; this key's alert lock is held
-            # for the panel's wall time (≤~25 s), which only delays a
-            # same-key duplicate that _should_alert would drop anyway.
-            await asyncio.get_running_loop().run_in_executor(
-                _executor, attach_panel, report)
-            detail = format_signal_message(report)
-            delivered = False
     for chat_id in _signal_targets():
-                if await _send_alert(bot, chat_id, detail):
+        for i, report in enumerate(actionable):
-                    delivered = True
+            detail = format_signal_message(report)
-            # Commit only when the alert reached at least one target;
+            try:
+                await bot.send_message(chat_id, detail, parse_mode=ParseMode.MARKDOWN)
+                sent_ok.add(i)
+            except Exception as exc:
+                logger.warning(f"Failed to send signal for chat {chat_id}: {exc}")
+    # Commit fingerprints only for alerts that reached at least one target;
     # a fully failed send stays eligible for the next scan.
-            if delivered:
+    for i, report in enumerate(actionable):
-                _commit_alert(key, report)
+        if i in sent_ok:
+            _commit_alert(f"{report.symbol}_{tf.value}", report)
 
 
 # ── WebSocket level-breach monitor ──────────────────────────────────
 
-# Level cache keyed symbol → timeframe → {level_name: price}, with a
+_level_cache: Dict[str, Dict[str, float]] = {}
-# matching per-TF write timestamp. Keyed per TF (2026-08-04 audit H34)
+_level_cache_ts: Dict[str, float] = {}
-# because every scan path writes here — the M5/M15/H1/H4 boundary loops,
-# the breach rescan, the manual Scan-All button — and a single flat dict
-# meant the last writer wiped the other timeframes' swing/EMA levels
-# within minutes. Breach checks scan the union of all fresh TF entries.
-_level_cache: Dict[str, Dict[str, Dict[str, float]]] = {}
-_level_cache_ts: Dict[str, Dict[str, float]] = {}
 _LEVEL_CACHE_TTL = 3600
 _last_prices: Dict[str, float] = {}
 _last_price_ts: Dict[str, float] = {}
 _scan_cooldowns: Dict[str, float] = {}
-# Breaches are edge-triggered (only the crossing tick fires), so one that
-# lands while the symbol is inside the rescan cooldown would otherwise be
-# dropped forever (H33b). Park it here; the tick loop fires it once the
-# cooldown expires. symbol → (price, level_name, recorded_at).
-_pending_breach: Dict[str, tuple] = {}
-# A failed breach rescan shrinks the just-set cooldown to this retry
-# window instead of silencing the symbol for the full cooldown (H33a).
-_BREACH_RETRY_SECONDS = 30
 
 _cfg = PlatformConfig()
 _WS_COOLDOWN = _cfg.ws_cooldown_seconds
@@ -780,8 +663,6 @@ _breach_tasks: Set[asyncio.Task] = set()
 
 
 def _update_level_cache(reports: List[SignalReport]) -> None:
-    """Each report updates only its own timeframe's slot — an M5 scan can
-    no longer overwrite the H1/H4 swing levels (H34)."""
     now = time.time()
     for r in reports:
         if r.primary_analysis and r.primary_analysis.key_levels:
@@ -790,31 +671,26 @@ def _update_level_cache(reports: List[SignalReport]) -> None:
                 if k in _LEVEL_KEYS
             }
             if levels:
-                tf = r.primary_analysis.timeframe
+                _level_cache[r.symbol] = levels
-                _level_cache.setdefault(r.symbol, {})[tf] = levels
+                _level_cache_ts[r.symbol] = now
-                _level_cache_ts.setdefault(r.symbol, {})[tf] = now
 
 
 def _check_level_breach(symbol: str, price: float) -> Optional[str]:
-    """Scan the union of every fresh timeframe's levels; returns
+    levels = _level_cache.get(symbol)
-    '<tf> <level_name>' (e.g. '15min swing_high_1') on a crossing so the
+    if not levels:
-    alert label says which timeframe's level broke."""
+        return None
-    per_tf = _level_cache.get(symbol)
+
-    if not per_tf:
+    cache_age = time.time() - _level_cache_ts.get(symbol, 0)
+    if cache_age > _LEVEL_CACHE_TTL:
         return None
 
     last = _last_prices.get(symbol)
     if last is None:
         return None
 
-    now = time.time()
-    ts_map = _level_cache_ts.get(symbol, {})
-    for tf, levels in per_tf.items():
-        if now - ts_map.get(tf, 0) > _LEVEL_CACHE_TTL:
-            continue   # stale TF entry — skip, fresher TFs still count
     for name, level in levels.items():
         if (last < level <= price) or (last > level >= price):
-                return f"{tf} {name}"
+            return name
 
     return None
 
@@ -829,17 +705,9 @@ def _cleanup_cooldowns() -> None:
 async def _handle_breach(bot, pair_symbol: str, price: float, level_name: str) -> None:
     now = time.time()
     if now - _scan_cooldowns.get(pair_symbol, 0) < _WS_COOLDOWN:
-        # Park the crossing instead of dropping it — price holding beyond
-        # the level never re-fires the edge trigger, so without this the
-        # breach would be lost forever (H33b). The tick loop fires it
-        # once the cooldown expires; a newer crossing overwrites it.
-        _pending_breach[pair_symbol] = (price, level_name, now)
         return
 
-    # Early-set on purpose: this is what stops concurrent breach tasks
-    # from stacking rescans. The failure path below rolls it back.
     _scan_cooldowns[pair_symbol] = now
-    _pending_breach.pop(pair_symbol, None)   # this crossing is being handled
     display = PAIR_INFO.get(pair_symbol, {}).get("display", pair_symbol)
     label = level_name.replace("_", " ").title()
     logger.info(f"Level breach: {display} crossed {label} at {price} — rescanning M15")
@@ -849,46 +717,29 @@ async def _handle_breach(bot, pair_symbol: str, price: float, level_name: str) -
         _update_level_cache([report])
     except Exception:
         logger.exception(f"Breach scan failed for {display}")
-        # Shrink the just-set cooldown to a short retry window so a
-        # transient fetch error doesn't silence the symbol for the full
-        # cooldown (H33a); the floor bounds the retry rate while the
-        # provider stays down. Re-park the breach so the tick loop
-        # retries it (setdefault: a newer crossing recorded meanwhile
-        # wins over this stale one).
-        retry = min(_BREACH_RETRY_SECONDS, _WS_COOLDOWN)
-        _scan_cooldowns[pair_symbol] = now - _WS_COOLDOWN + retry
-        _pending_breach.setdefault(pair_symbol, (price, level_name, now))
         return
 
     # Alert only when the rescan produced an actionable, changed signal —
     # same gates as the auto-scanner, so a breach can't spam the channel.
     if (report.final_signal == SignalStrength.NEUTRAL
-            or not _passes_quality(report.signal_quality)):
+            or not _passes_quality(report.signal_quality)
+            or not _should_alert(f"{pair_symbol}_{Timeframe.M15.value}", report)):
         return
 
-    key = f"{pair_symbol}_{Timeframe.M15.value}"
-    # Lock spans check → send → commit; without it this handler and an
-    # in-flight M15 auto-scan both pass _should_alert and double-alert
-    # the same signal (H36).
-    async with _alert_lock(key):
-        if not _should_alert(key, report):
-            return
-
-        # Same post-gate panel attach as the auto-scan path (see
-        # _scan_timeframe for the spend/lock rationale).
-        await asyncio.get_running_loop().run_in_executor(
-            _executor, attach_panel, report)
     header = (
         f"⚡ *LEVEL BREAK* — {_escape_md(display)} crossed "
         f"*{_escape_md(label)}* @ `{_fmt_price(price)}`\n\n"
     )
-        detail = header + format_signal_message(report)
     sent = False
     for chat_id in _signal_targets():
-            if await _send_alert(bot, chat_id, detail):
+        detail = header + format_signal_message(report)
+        try:
+            await bot.send_message(chat_id, detail, parse_mode=ParseMode.MARKDOWN)
             sent = True
+        except Exception as exc:
+            logger.warning(f"Failed to send breach alert to chat {chat_id}: {exc}")
     if sent:
-            _commit_alert(key, report)
+        _commit_alert(f"{pair_symbol}_{Timeframe.M15.value}", report)
 
 
 def _breach_task_done(task: asyncio.Task) -> None:
@@ -904,32 +755,22 @@ async def _ws_monitor(bot) -> None:
     """Level-breach tick monitor. The provider owns the connection and
     protocol (`iter_ticks`); this loop owns reconnect/backoff and breach
     dispatch. Providers without streaming quotes simply disable it."""
-    dp = _get_streaming_dp()
+    dp = _get_shared_dp()
     if not dp.supports_streaming:
         logger.info(
             "Data provider %r has no streaming quotes — "
             "level-breach monitor disabled", dp.name)
         return
-    if getattr(dp, "_api_key", "unused") is None:
-        # Guard, not a traceback loop: without a key every connect
-        # attempt would raise and re-log a full traceback every ≤60s.
-        logger.warning(
-            "Streaming provider %r has no API key — "
-            "level-breach monitor disabled", dp.name)
-        return
 
     backoff = 1
     while _feed_active():
-        got_tick = False
         try:
-            async for pair_symbol, price in dp.iter_ticks(SCAN_PAIRS):
+            def _on_connect() -> None:
-                if not got_tick:
+                nonlocal backoff
-                    # Reset backoff only once the connection actually
-                    # produced data — resetting on connect let a
-                    # rejected key or accept-then-close server pin the
-                    # backoff at minimum forever (H38a/H38b).
-                    got_tick = True
                 backoff = 1
+
+            async for pair_symbol, price in dp.iter_ticks(
+                    SCAN_PAIRS, on_connect=_on_connect):
                 if not _feed_active():
                     break
 
@@ -938,32 +779,11 @@ async def _ws_monitor(bot) -> None:
                 _last_price_ts[pair_symbol] = time.time()
 
                 if breached:
-                    # A fresh crossing supersedes any parked one.
-                    _pending_breach.pop(pair_symbol, None)
                     task = asyncio.create_task(
                         _handle_breach(bot, pair_symbol, price, breached)
                     )
                     _breach_tasks.add(task)
                     task.add_done_callback(_breach_task_done)
-                elif pair_symbol in _pending_breach:
-                    pend = _pending_breach[pair_symbol]
-                    if time.time() - pend[2] > _LEVEL_CACHE_TTL:
-                        # Ancient parked breach (feed gap/weekend) — an
-                        # alert about an hours-old crossing would only
-                        # mislead; the level cache it came from is stale
-                        # anyway.
-                        _pending_breach.pop(pair_symbol)
-                    elif (time.time() - _scan_cooldowns.get(pair_symbol, 0)
-                            >= _WS_COOLDOWN):
-                        # Crossing parked during the cooldown: fire it now
-                        # that the window expired — the edge trigger alone
-                        # would have dropped it forever (H33b).
-                        _pending_breach.pop(pair_symbol)
-                        task = asyncio.create_task(
-                            _handle_breach(bot, pair_symbol, pend[0], pend[1])
-                        )
-                        _breach_tasks.add(task)
-                        task.add_done_callback(_breach_task_done)
 
                 _cleanup_cooldowns()
 
@@ -973,16 +793,6 @@ async def _ws_monitor(bot) -> None:
             logger.exception(f"WebSocket error, reconnecting in {backoff}s")
             await asyncio.sleep(backoff)
             backoff = min(backoff * 2, 60)
-        else:
-            # Clean server close: the async-for ended without raising.
-            # This must back off and log like the error path — silently
-            # reconnecting with zero delay hot-loops against a server
-            # that accepts and immediately closes (H38a).
-            if _feed_active():
-                logger.warning(
-                    f"WebSocket closed cleanly, reconnecting in {backoff}s")
-                await asyncio.sleep(backoff)
-                backoff = min(backoff * 2, 60)
 
     logger.info("WebSocket monitor stopped")
 
@@ -1025,25 +835,15 @@ _ws_task: Optional[asyncio.Task] = None
 
 def _start_scheduler(bot) -> None:
     global _ws_task
-    # Per-TF respawn (2026-08-04 audit H37): drop finished loop tasks and
+    alive = [t for t in _autoscan_tasks if not t.done()]
-    # recreate only the missing ones. The old all-or-nothing alive check
+    if not alive:
-    # left a single dead TF loop unmonitored forever while its siblings
+        _autoscan_tasks.clear()
-    # lived — and with the channel feed active, /autoscan never reaches
-    # _stop_scheduler, so nothing short of a process restart recovered it.
-    _autoscan_tasks[:] = [t for t in _autoscan_tasks if not t.done()]
-    alive_names = {t.get_name() for t in _autoscan_tasks}
-    started = []
         for tf, interval in _AUTOSCAN_LOOPS:
-        name = f"candle_loop_{tf.name}"
-        if name in alive_names:
-            continue
             _autoscan_tasks.append(
-            asyncio.create_task(_candle_loop(bot, tf, interval), name=name)
+                asyncio.create_task(_candle_loop(bot, tf, interval))
             )
-        started.append(tf.name)
-    if started:
         logger.info("Candle-boundary scheduler started (%s)",
-                    "/".join(started))
+                    "/".join(tf.name for tf, _ in _AUTOSCAN_LOOPS))
 
     if _ws_task is None or _ws_task.done():
         _ws_task = asyncio.create_task(_ws_monitor(bot))
@@ -1119,7 +919,6 @@ _HELP_TEXT = (
     "/scan — Quick scan menu\n"
     "/autoscan — Toggle auto-scan on/off\n"
     "/status — Bot status, WS connection, live prices\n"
-    "/news — Net news sentiment score for XAU/USD\n"
     "/cache — Show data cache age per pair/TF\n"
     "/clearcache — Wipe data cache, force fresh fetch\n"
     "/help — This message\n\n"
@@ -1138,91 +937,6 @@ async def cmd_help(update: Update, context: ContextTypes.DEFAULT_TYPE) -> None:
     await update.message.reply_text(_HELP_TEXT, parse_mode=ParseMode.MARKDOWN)
 
 
-def _fmt_age(ts: Optional[float]) -> str:
-    """Compact age like '45s', '12m', '3.2h' for status lines."""
-    if not ts:
-        return "?"
-    s = time.time() - ts
-    if s < 90:
-        return f"{s:.0f}s"
-    if s < 5400:
-        return f"{s / 60:.0f}m"
-    return f"{s / 3600:.1f}h"
-
-
-def _news_health_lines() -> List[str]:
-    """News / calendar / DeepSeek source health for /status.
-
-    Reads the class-level `NewsEngine._source_health` written by the
-    scan-path fetches plus the LLM client's health stamps — display
-    only, never triggers a fetch. ⚠️ marks a degraded dependency;
-    details live in the rotating log (transition-logged WARNINGs)."""
-    lines = ["\n*News & AI:*"]
-    # Snapshot: a scan thread may insert a source's FIRST entry while we
-    # iterate (dict-changed-size RuntimeError); entry mutation is safe.
-    for _ in range(3):
-        try:
-            sh = dict(NewsEngine._source_health)
-            break
-        except RuntimeError:
-            continue
-    else:
-        sh = {}
-    if not sh:
-        lines.append("  _no news cycle yet (fills after first scan)_")
-    else:
-        cal = sh.get("calendar")
-        ff = sh.get("calendar:forexfactory")
-        if cal:
-            if NewsEngine.source_is_failing(cal):
-                lines.append(f"  ⚠️ Calendar: unavailable — {cal['detail']}"
-                             f" ({_fmt_age(cal['err'])})")
-            elif NewsEngine.source_is_failing(ff):
-                lines.append(f"  ⚠️ Calendar: {cal['extra']} — ForexFactory"
-                             f" down: {ff['detail']} ({_fmt_age(ff['err'])})")
-            else:
-                lines.append(f"  ✅ Calendar: {cal['extra']}"
-                             f" ({_fmt_age(cal['ok'])} ago)")
-        heads = sh.get("headlines")
-        if heads:
-            failing = NewsEngine.source_is_failing(heads)
-            mark = "⚠️" if failing else "✅"
-            detail = f" — {heads['detail']}" if failing else ""
-            ts = heads["err"] if failing else heads["ok"]
-            lines.append(f"  {mark} Headlines: {heads['extra']}{detail}"
-                         f" ({_fmt_age(ts)} ago)")
-        for name, entry in sorted(sh.items()):
-            if name.startswith("rss:") and NewsEngine.source_is_failing(entry):
-                lines.append(f"  ⚠️ {name[4:]}: {entry['detail']}"
-                             f" ({_fmt_age(entry['err'])})")
-    client = get_llm_client()
-    if not client.available:
-        lines.append("  DeepSeek: off — keyword scoring only")
-    else:
-        # The client's counter only resets on the first call of a new
-        # UTC day — show 0 (not yesterday's total, or a phantom
-        # cap-exhausted warning) until that reset happens.
-        today = datetime.now(timezone.utc).date()
-        used = client._count if client._count_day == today else 0
-        cap = client._daily_cap
-        scored = len(NewsEngine._llm_sent_cache)
-        err = client.last_error
-        if used >= cap:
-            lines.append(f"  ⚠️ DeepSeek: daily cap {cap} exhausted —"
-                         " keyword fallback until UTC midnight")
-        elif err and err[0] > client.last_ok:
-            lines.append(f"  ⚠️ DeepSeek: last call failed ({err[1]},"
-                         f" {_fmt_age(err[0])}) — keyword fallback")
-        else:
-            lines.append(f"  ✅ DeepSeek: {used}/{cap} calls today,"
-                         f" {scored} headlines scored")
-        if get_app_config().agent_panel_settings.get("enabled", False):
-            ph = panel_health()
-            lines.append(f"  🤖 AI panel: {ph['used_today']}/{ph['cap']}"
-                         " panels today")
-    return lines
-
-
 def _status_text() -> str:
     lines = ["📊 *ForexBot2 Status*\n"]
 
@@ -1237,8 +951,7 @@ def _status_text() -> str:
         lines.append(f"📢 Channel feed: *on* (`{CHANNEL_ID}`)")
 
     if _ws_task and not _ws_task.done():
-        provider = _streaming_dp.name if _streaming_dp else "?"
+        lines.append("📡 WebSocket: *connected*")
-        lines.append(f"📡 WebSocket: *connected* ({provider})")
     else:
         lines.append("❌ WebSocket: *disconnected*")
 
@@ -1256,16 +969,9 @@ def _status_text() -> str:
         lines.append("\n_No live prices yet (WS not streaming)_")
 
     cached_pairs = len(_level_cache)
-    cached_levels = sum(
+    cached_levels = sum(len(v) for v in _level_cache.values())
-        len(levels) for per_tf in _level_cache.values()
-        for levels in per_tf.values()
-    )
     if cached_levels:
-        all_ts = [
+        oldest_age = time.time() - min(_level_cache_ts.values()) if _level_cache_ts else 0
-            ts for per_tf in _level_cache_ts.values()
-            for ts in per_tf.values()
-        ]
-        oldest_age = time.time() - min(all_ts) if all_ts else 0
         lines.append(f"\n*Level Cache:* {cached_levels} levels across {cached_pairs} pairs (oldest {oldest_age:.0f}s)")
     else:
         lines.append("\n*Level Cache:* empty — breach alerts start after first candle scan")
@@ -1281,8 +987,6 @@ def _status_text() -> str:
     if active_breach:
         lines.append(f"*Breach tasks:* {active_breach} running")
 
-    lines.extend(_news_health_lines())
-
     lines.append(f"\n*Alert quality:* {_alert_min_quality}+ "
                  f"| re-arm {_ALERT_REARM_SECONDS // 60}m")
     return "\n".join(lines)
@@ -1293,38 +997,6 @@ async def cmd_status(update: Update, context: ContextTypes.DEFAULT_TYPE) -> None
     await update.message.reply_text(_status_text(), parse_mode=ParseMode.MARKDOWN)
 
 
-def _news_text(context: FundamentalContext, engine: NewsEngine) -> str:
-    net, desc = engine.get_pair_sentiment(XAUUSD, context)
-    if round(net, 2) == 0:
-        net = 0.0  # a tiny negative mean would render as "-0.00"
-    lines = [
-        "📰 *XAU/USD News Score*\n",
-        f"*Net score:* `{net:+.2f}` — {desc}",
-    ]
-    # A cold/failed feed yields 0.00 "Neutral" — flag it so a data outage
-    # can't be mistaken for a genuinely neutral read.
-    if not context.news_sentiment:
-        lines.append("_no scored headlines yet — score defaults to 0_")
-    return "\n".join(lines)
-
-
-def _news_report_sync() -> str:
-    engine = NewsEngine(PlatformConfig().news)
-    return _news_text(engine.get_fundamental_context(), engine)
-
-
-@_owner_only
-async def cmd_news(update: Update, context: ContextTypes.DEFAULT_TYPE) -> None:
-    msg = await update.message.reply_text("⏳ Fetching news…")
-    loop = asyncio.get_running_loop()
-    try:
-        text = await loop.run_in_executor(_executor, _news_report_sync)
-    except Exception as exc:
-        logger.exception("News fetch failed")
-        text = f"❌ News fetch failed: {_escape_md(str(exc))}"
-    await msg.edit_text(text, parse_mode=ParseMode.MARKDOWN)
-
-
 @_owner_only
 async def cmd_cache(update: Update, context: ContextTypes.DEFAULT_TYPE) -> None:
     dp = _get_shared_dp()
@@ -1540,7 +1212,6 @@ async def _post_init(app: Application) -> None:
         ("scan", "Quick scan buttons"),
         ("autoscan", "Toggle auto-scan on/off"),
         ("status", "Bot status, WS connection, live prices"),
-        ("news", "XAU/USD net news sentiment score"),
         ("cache", "Show data cache age per pair/TF"),
         ("clearcache", "Wipe data cache, force fresh fetch"),
         ("help", "All commands"),
@@ -1570,11 +1241,7 @@ async def _post_init(app: Application) -> None:
                 logger.error(f"Initial scan failed for {SCAN_PAIRS[i]}: {r}")
         if init_reports:
             _update_level_cache(init_reports)
-            n_levels = sum(
+            logger.info(f"Level cache populated: {sum(len(v) for v in _level_cache.values())} levels")
-                len(levels) for per_tf in _level_cache.values()
-                for levels in per_tf.values()
-            )
-            logger.info(f"Level cache populated: {n_levels} levels")
     except Exception:
         logger.exception("Initial scan failed — WS will wait for first candle scan")
 
@@ -1674,7 +1341,6 @@ def main() -> None:
     app.add_handler(CommandHandler("scan", cmd_scan))
     app.add_handler(CommandHandler("help", cmd_help))
     app.add_handler(CommandHandler("status", cmd_status))
-    app.add_handler(CommandHandler("news", cmd_news))
     app.add_handler(CommandHandler("autoscan", cmd_autoscan))
     app.add_handler(CommandHandler("cache", cmd_cache))
     app.add_handler(CommandHandler("clearcache", cmd_clearcache))