Provider abstraction: providers/ package with BaseDataProvider template

One class per provider (TwelveData, Oanda, Tiingo), all subclassing
BaseDataProvider which owns cache, validation/cleaning, freshness +
boundary guarantees, fetch_range pagination, and streaming stubs —
subclasses implement only _fetch_raw. config.json data_providers.active
selects the live feed; data_provider.py / oanda_provider.py stay as
back-compat shims. Module 16 provider-abstraction tests (253 total green).

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Mc7XKYyPySVXiUpcN82QEK

CLAUDE.md

@@ -15,7 +15,12 @@ This file is the authoritative developer/agent reference. The human-facing docs
 | `config.py` | ~374 | All config dataclasses, `PAIR_INFO` registry, per-pair strategy configs. **`PlatformConfig` defaults ARE the tuned XAU config.** |
 | `app_config.py` | ~290 | config.json loader/validation, env-precedence resolvers, repo-root `.env` loader (setdefault at import), `apply_pair_overrides` deployment overlay |
 | `config.json.example` | 24 | committed schema template; real gitignored config.json holds secrets + deployment settings |
-| `data_provider.py` | ~347 | Twelve Data SDK wrapper: fetch, rate-limit (55 credits/min), in-memory cache, validation/cleaning, freshness checks |
+| `providers/base.py` | ~400 | `BaseDataProvider`: shared cache, validation/cleaning, freshness + boundary guarantees, `fetch_range` pagination, streaming stubs. Subclasses implement only `_fetch_raw` |
+| `providers/twelve_data.py` | ~240 | `TwelveDataProvider` (alias `ForexDataProvider`): TD SDK fetch, rate-limit (55 credits/min), WS quote streaming |
+| `providers/oanda.py` | ~190 | `OandaDataProvider`: v20 REST MBA candles, real bid/ask spread, date-window fetch |
+| `providers/tiingo.py` | ~200 | `TiingoDataProvider`: FX endpoint mid-price candles (no volume, no spread — see Tiingo note), named errors for every failure shape |
+| `providers/__init__.py` | ~55 | `PROVIDER_REGISTRY` + `make_data_provider` (config.json `data_providers.active` selects) |
+| `data_provider.py` / `oanda_provider.py` | ~25 | back-compat shims re-exporting from `providers/` |
 | `engine.py` | ~1173 | `TechnicalEngine`: computes all indicators on one DataFrame → `TechnicalAnalysis` (composite score, regime, key levels) |
 | `chart_patterns.py` | ~280 | Chart-pattern detectors for the XAU_CP archetype (pure functions, ATR-unit geometry, first-close-beyond triggers) |
 | `signal_aggregator.py` | ~1447 | `SignalAggregator`: MTF analysis, news integration, gates/penalties, quality grade, trade-setup generation → `SignalReport` |
@@ -24,7 +29,7 @@ This file is the authoritative developer/agent reference. The human-facing docs
 | `pair_signals/__init__.py` | 52 | `PAIR_PRODUCERS` registry + `produce_signal()` |
 | `pair_signals/xauusd.py` | 71 | XAU config factory + per-timeframe overrides (docstring = validation record) |
 | `main.py` | ~348 | CLI: `python main.py --pair XAUUSD --timeframe H1 --detail/--json` |
-| `test_all.py` | ~3100 | QA suite: **210 tests in 14 modules**, no live API needed. Must stay green. |
+| `test_all.py` | ~4100 | QA suite: **253 tests in 16 modules**, no live API needed. Must stay green. |
 | `backtest/gpu_backtest.py` | ~2231 | Bar-replay backtester (CuPy→NumPy fallback) using the real `SignalAggregator` |
 | `backtest/diagnose_filters.py` | 609 | Filter-diagnosis tooling |
 | `backtest/research/` | — | Calendar research infra (`ff_calendar_scraper.py`, `calendar_store.py`) + result CSVs |
@@ -230,15 +235,25 @@ Two entry points:
 
 ---
 
-## data_provider.py — ForexDataProvider
+## providers/ — data-provider abstraction (refactored 2026-07-25)
 
-- Twelve Data SDK (`TDClient.time_series(...).as_pandas()`), symbols normalized to `"XAU/USD"` form; renames lowercase columns → capitalized; sorts oldest-first; localizes UTC.
-- **Rate limit**: 55 credits/min sliding window (`_throttle`), blocks with sleep when exhausted.
-- **In-memory cache**: key `{symbol}_{tf}`, TTL = candle duration (M5 300 s … H4 14400 s); a cached frame is evicted and refetched if it fails the freshness check **or the boundary check** (`_has_latest_closed_bar`: the frame must contain the most recent fully closed bar, weekend-exempt — a wall-clock TTL alone would serve mid-candle snapshots at the next boundary and scans would run one bar behind). Fresh fetches enforce the same boundary guarantee: a response missing the just-closed bar (feed latency at the boundary) gets one grace retry after `FRESH_FETCH_RETRY_SECONDS` (3 s), then proceeds with a loud warning. Cache and throttle are lock-protected (one provider is shared across the bot's thread pool).
-- **`_drop_forming_candle`**: Twelve Data includes the current partial candle; the scanner fires seconds after boundary, so the last bar is dropped if its close time is in the future. Without this every indicator would read a seconds-old stub and signals would repaint.
-- **Freshness** (`_is_data_fresh`): last bar older than 3× candle duration → stale (warns; live path suppresses signal if the PRIMARY TF is stale). Weekends exempt (Fri 22:00→Sun 22:00 UTC).
-- **`_validate_and_clean`**: drop all-NaN rows → ffill (limit 3) → drop NaN closes → sort + dedupe (keep last) → drop future timestamps (>now+5 min) → repair OHLC inconsistencies (High=max, Low=min of O/C/existing) → remove >10σ return outliers (spike bar only — the opposite-direction snap-back bar right after it is kept, else every spike punched a two-bar hole) → warn if < ~210 bars.
-- `outputsize = max(300, min(days×bars_per_day+20, 5000))`; TD caps at 5000/call (the backtester paginates by date).
+One class per provider, all subclassing **`providers/base.py BaseDataProvider`**, which owns every behavior the signal pipeline depends on — defined ONCE (a test enforces that neither subclass overrides shared methods):
+
+- **In-memory cache**: key `{symbol}_{tf}`, TTL = candle duration (M5 300 s … H4 14400 s); a cached frame is evicted and refetched if it fails the freshness check **or the boundary check** (`_has_latest_closed_bar`: the frame must contain the most recent fully closed bar, weekend-exempt — a wall-clock TTL alone would serve mid-candle snapshots at the next boundary and scans would run one bar behind). Fresh fetches enforce the same boundary guarantee: a response missing the just-closed bar (feed latency at the boundary) gets one grace retry after `FRESH_FETCH_RETRY_SECONDS` (3 s), then proceeds with a loud warning. Cache is lock-protected (one provider is shared across the bot's thread pool).
+- **`_drop_forming_candle`**: some feeds include the current partial candle; the scanner fires seconds after boundary, so the last bar is dropped if its close time is in the future. Without this every indicator would read a seconds-old stub and signals would repaint. (No-op on feeds that only return complete candles, e.g. Oanda.)
+- **Freshness** (`is_data_fresh`, public): last bar older than 3× candle duration → stale (warns; live path suppresses signal if the PRIMARY TF is stale). Weekends exempt (Fri 22:00→Sun 22:00 UTC).
+- **`_validate_and_clean`** (public wrapper `validate_and_clean`): drop all-NaN rows → ffill (limit 3) → drop NaN closes → sort + dedupe (keep last) → drop future timestamps (>now+5 min) → repair OHLC inconsistencies (High=max, Low=min of O/C/existing) → remove >10σ return outliers (spike bar only — the opposite-direction snap-back bar right after it is kept, else every spike punched a two-bar hole) → warn if < ~210 bars.
+- **`fetch_range(symbol, tf, start, end)`**: paginated RAW date-range fetch under `MAX_BARS_PER_REQUEST` (5000) — the backtester's `fetch_history` calls this, so ANY registered provider can feed the backtester.
+- `outputsize = max(300, min(days×bars_per_day+20, 5000))`; `CANDLE_SECONDS`/`BARS_PER_DAY` tables live on the base class (the aggregator's bar-close clock reads `BaseDataProvider.CANDLE_SECONDS`).
+- **Streaming capability (optional)**: `supports_streaming` flag + async generator `iter_ticks(symbols, on_connect)` yielding `(canonical_symbol, price)` for one connection's lifetime; the caller owns reconnect/backoff. Errors: each provider raises its own `DATA_ERROR`, always a subclass of `DataIntegrityError` — catch the base type provider-agnostically.
+
+**A subclass implements only `_fetch_raw(symbol, tf, outputsize, start=None, end=None)`** (normalized frame: capitalized OHLCV, UTC tz-aware index named "Datetime", oldest-first) plus symbol/interval maps and optional capabilities. To add a provider: one module in `providers/`, one `PROVIDER_REGISTRY` entry in `providers/__init__.py`, one config.json `data_providers` block. The live feed is selected by config.json `data_providers.active` (registration alone changes nothing).
+
+- **`providers/twelve_data.py` `TwelveDataProvider`** (alias `ForexDataProvider`): TD SDK (`TDClient.time_series(...).as_pandas()`), symbols normalized to `"XAU/USD"` form; lowercase→capitalized columns, newest-first→oldest-first; **rate limit** 55 credits/min sliding window (`_throttle`, lock-protected, blocks with sleep). Implements streaming (`ws.twelvedata.com` quotes WS, 10 s heartbeat, `_parse_ws_price`).
+- **`providers/oanda.py` `OandaDataProvider`**: v20 REST MBA candles (complete-only), real bid/ask spread via `get_spread` override; credentials default to the config.json `oanda` block so the registry can build it config-only (explicit `api_token`/`base_url` args still accepted — the daemon passes them). Date-window fetches send `from`/`to` and omit `count` (v20 rejects the combination).
+- **`providers/tiingo.py` `TiingoDataProvider`**: `/tiingo/fx/{ticker}/prices` resampled mid-price candles; no volume (filled 0.0), W1 unmapped (raises). Tiingo takes date windows, not bar counts — live fetches derive `startDate` from the requested count × candle duration × 1.6 weekend pad. Auth in the header (key never in URLs/logs). Every failure shape raises `TiingoDataError` with a named cause (429 quota, 401/403 auth, empty response, malformed candle, missing key). **Candles only** — `get_spread` stays on the base default; the free quote endpoint's indicative spread must not be re-added (see Tiingo spread-monitor removal note).
+- **`data_provider.py` / `oanda_provider.py` are back-compat shims** re-exporting from `providers/`; `ForexDataProvider` is the historical name of the TD provider.
+- **No silent failures**: provider errors all subclass `DataIntegrityError` and carry a named cause; the aggregator logs per-TF fetch failures and degrades the report (missing primary → NEUTRAL/F "data unavailable"); `make_data_provider` raises on an unknown `active` name; keyless providers warn at construction and raise at fetch.
 
 ---
 
@@ -270,7 +285,7 @@ Run: `python telegram_bot.py` (credentials resolve env `TELEGRAM_TOKEN`/`TELEGRA
 - **Subscriber channel** (`TELEGRAM_CHANNEL_ID`, `-100…`; `""` disables): auto-scan signal alerts + breach alerts also post there (bot needs channel-admin). Monetization = Telegram Stars subscription invite links; Telegram handles membership, the bot has zero payment code. The channel feed keeps running even when the owner toggles DM auto-scan off (`_feed_active()` = DM chats OR channel).
 - **Auto-scan**: boot-starts if `TELEGRAM_CHAT_ID` set (initial M15 scan populates the level cache first). Candle-boundary loops for M5/M15/H1/H4, firing at boundary + 10 s. Each scan runs all `scan_pairs` concurrently, updates the level cache, then broadcasts.
 - **Alert gating**: quality threshold (`_alert_min_quality`, default "C", menu cycles C→B→A) + fingerprint dedup per (symbol, TF) on (signal, quality, direction) with 1-hour re-arm — a persisting identical signal doesn't re-send every candle, but a setup whose entry moved ≥ half the last-ALERTED risk distance (new pullback/swing anchor; compared against the stored last-alert entry/risk, a stable reference immune to per-candle ATR drift) re-alerts. The fingerprint is committed only **after** a message reaches ≥1 target (`_commit_alert`), so a failed Telegram send doesn't consume the alert.
-- **WebSocket monitor**: subscribes every `scan_pairs` symbol to TD quotes WS; 10 s heartbeat; exponential reconnect (≤60 s). On tick crossing any cached key level (`swing_high/low_1..3`, `ema_trend`, `bb_upper/lower`, `kc_upper/lower`; cache TTL 1 h) → immediate M15 rescan → "LEVEL BREAK" alert if actionable+changed (same gates as auto-scan), 5-min per-pair cooldown.
+- **WebSocket monitor**: the provider owns the connection/protocol (`BaseDataProvider.iter_ticks` streaming capability — TD implements it; heartbeat + message parsing live in `providers/twelve_data.py`); the bot owns exponential reconnect (≤60 s) + breach dispatch, and disables the monitor with a log line if the active provider has `supports_streaming=False`. On tick crossing any cached key level (`swing_high/low_1..3`, `ema_trend`, `bb_upper/lower`, `kc_upper/lower`; cache TTL 1 h) → immediate M15 rescan → "LEVEL BREAK" alert if actionable+changed (same gates as auto-scan), 5-min per-pair cooldown.
 - **Commands**: `/start` (menu), `/scan` (buttons: per pair — M5/M15/H1/H4 + Scan All TFs), `/autoscan` (toggle DM copy), `/status` (loops, WS, live prices, level cache, cooldowns), `/cache` (age per pair/TF), `/clearcache`, `/help`.
 - **Tiingo spread monitor: REMOVED 2026-07-23** (shipped 2026-07-22, removed next day). Tiingo's free `/tiingo/fx/top` publishes a two-tier *indicative* spread for gold — exactly $0.10 in liquid hours and exactly $0.70 in the 21:00–00:00 UTC rollover window / news moments (verified: 273-sample production log + live probes; only those two values ever occur). Quotes are mid ± a posted increment, not executable depth, so the feature could never validate the $0.30 backtest cost assumption. Do not re-add a spread display from Tiingo's free FX feed.
 - Signal messages (redesigned 2026-07-24, `format_signal_message`): verdict header first, then trade setup, then a condensed one-line indicator summary (`EMA↑ RSI→ …`, `_indicator_summary`) + warnings. The setup block includes an **informational 100-pip TP line** (`_INFO_TP_PIPS`, display-only — entry ± 100 pips, ≈R vs stop; NOT part of the backtested exit model) and the **trailing-stop management plan** (arm price + trail distance, read from the `trail_arm_r`/`trail_dist_r` stamped on the setup by the producing per-TF config). Risk disclaimer on every outbound message.
@@ -314,13 +329,13 @@ Report: summary metrics (total R, PF, Sharpe on R, max DD, MFE/MAE), breakdowns
 
 **Comparability note (2026-07-16):** the parity/realism fixes of that date (EMA-ribbon port to the GPU engine, exact-pandas Bollinger precompute, raw-path entry re-anchoring, per-TF D1 warmup, live confirm-map CLI default, outlier snap-back retention) all shift books slightly vs earlier runs. Books produced before these fixes are not directly comparable — re-run the baseline before any before/after tuning comparison.
 
-Data: paginated TD fetches (5000-bar chunks) cached as parquet in `backtest/cache/` keyed by symbol/TF/window; warmup pad ≥60 days before the window for EMA200, extended per-TF so slow confirmation TFs get ≥210 bars at window start (D1 gets ~299 days — 62 days gave only ~61 bars: no EMA200, unlike live's 300-bar outputsize floor). `--confirm-tfs` defaults to the pair factory's live confirmation map (pass it explicitly to override); a hardcoded default here previously made the standard invocation backtest a different MTF setup than the bot scans.
+Data: paginated fetches via `provider.fetch_range` (5000-bar chunks; works with any registered provider) cached as parquet in `backtest/cache/` keyed by symbol/TF/window; warmup pad ≥60 days before the window for EMA200, extended per-TF so slow confirmation TFs get ≥210 bars at window start (D1 gets ~299 days — 62 days gave only ~61 bars: no EMA200, unlike live's 300-bar outputsize floor). `--confirm-tfs` defaults to the pair factory's live confirmation map (pass it explicitly to override); a hardcoded default here previously made the standard invocation backtest a different MTF setup than the bot scans.
 
 ---
 
 ## test_all.py — QA suite
 
-`python test_all.py` — **210 tests, 14 modules** (config, data_provider, engine, news_engine, signal_aggregator, main, telegram_bot, integration, pair_signals registry + engine-bug regressions, 2026-07-16 audit-fix regressions, chart patterns, app_config, .env loader + bot resilience). No network needed (mocked). Uses a custom `@test("name")` decorator that runs immediately (NOT pytest — don't grep for `def test_`). Exit code non-zero on failure. Includes: the xauusd-factory ≡ inline-config equivalence test, engine bug-fix regression tests (Module 9b), a live-vs-GPU engine parity sweep incl. the EMA-ribbon path (Module 11), config.json loading/precedence/validation coverage (Module 13), and `.env` loader + bot-resilience coverage (Module 14). **Any signal-logic change must keep this green and should add tests.**
+`python test_all.py` — **253 tests, 16 modules** (config, data_provider, engine, news_engine, signal_aggregator, main, telegram_bot, integration, pair_signals registry + engine-bug regressions, 2026-07-16 audit-fix regressions, chart patterns, app_config, .env loader + bot resilience, Oanda provider/executor/daemon, provider abstraction). No network needed (mocked). Uses a custom `@test("name")` decorator that runs immediately (NOT pytest — don't grep for `def test_`). Exit code non-zero on failure. Includes: the xauusd-factory ≡ inline-config equivalence test, engine bug-fix regression tests (Module 9b), a live-vs-GPU engine parity sweep incl. the EMA-ribbon path (Module 11), config.json loading/precedence/validation coverage (Module 13), and `.env` loader + bot-resilience coverage (Module 14). **Any signal-logic change must keep this green and should add tests.**
 
 ---
 
@@ -330,7 +345,8 @@ Precedence for every credential/deployment setting: **env var → config.json
 
 | Variable | Purpose | Default |
 |---|---|---|
-| `TWELVE_DATA_API_KEY` | market data key | none — falls through to config.json `data_providers.twelve_data.api_key`, then fails fast |
+| `TWELVE_DATA_API_KEY` | market data key (twelve_data only) | none — falls through to config.json `data_providers.twelve_data.api_key`, then fails fast |
+| `TIINGO_API_KEY` | market data key (tiingo only) | none — falls through to config.json `data_providers.tiingo.api_key`, then fails fast |
 | `TELEGRAM_TOKEN` | bot token | none — falls through to config.json `telegram.token`, then refuses to boot |
 | `TELEGRAM_CHAT_ID` | owner chat id (boot auto-scan + owner lock) | none — falls through to config.json `telegram.chat_id`, then refuses to boot |
 | `TELEGRAM_CHANNEL_ID` | subscriber channel (`-100…`); `""` disables the feed | none — falls through to config.json `telegram.channel_id` |