ForexBot2/docs/data_source_change_test_plan.md

Test Plan — Data Source Change Validation

Purpose: determine whether switching data_providers.active (e.g. Twelve Data → Oanda, or any future provider swap) changes signal calculations materially, and gate the switch on evidence.

Status: in progress, 2026-08-14

Progress (2026-08-14): Phase 0 ✅, Phase 1 ✅ (backtest/research/feed_divergence_2026-08-14.md), Phase 2 ✅ (backtest/research/signal_parity_2026-08-14.md), Phase 3 ✅ (backtest/research/repin_verification_2026-08-14.md — re-pin already existed), Phase 4 🔨 tool ready (scripts/shadow_scan.py, needs a 2-week run — launch manually per owner decision), Phase 5 owner decisions recorded (dual cost reporting; Tiingo streaming-only; see bottom). Context: Oanda has been the live feed since 2026-07-28. The backtest books were re-pinned on Oanda bars on 2026-08-04 (backtest/oanda_*.csv — 27 books, now the baseline; all TD-era books superseded). Cost reporting is dual: $0.30 retained for cross-book comparability, $0.65 is the realistic headline (measured Oanda round trip $0.65–0.72). Twelve Data was proven signal-incompatible in Phase 1 (~19% of M5 bars disagree on direction; systematic ~$1 bias) — it is NOT a valid candle fallback.

Why signals can differ between feeds (mechanisms to test):

• Feeds publish different mid prices → tiny OHLC diffs shift every indicator; near a threshold (RSI band, FB2_EMA_LO EMA200-proximity band, composite/grade cutoffs, M5 min-score gate) one bar's discrete score flips.
• Gate edge cases turn a tiny price diff into "setup generated" vs "no signal".
• ATR-derived SL/TP/trail levels differ even when the signal is identical.
• The shared 10σ cleaning pass can drop different bars per feed.
• Volume present (Oanda) vs absent (Tiingo FX); real spread (Oanda MBA) vs pair default (Tiingo).
• Known reference point: H4 seeding divergence — same provider, different fetch depth flipped ~2% of indicator signals (composite Δ up to 12.3). Cross-feed diffs act the same way.

---

Phase 0 — Recent-bar parity smoke test ✅ DONE

• Tool: scripts/compare_oanda_tiingo.py (removed 2026-09-06 once Phases 1-2 superseded it; the result is recorded here)
• Result 2026-08-14: 10/10 timestamp alignment on M5/M15/H1, Close max |Δ| ≤ $0.175 (≤ 0.004% of price), no directional bias.
• Verdict: recent-tail parity confirmed. Does NOT cover historical depth — proceed to Phase 1.

Phase 1 — Historical candle divergence study

Goal: quantify raw-data divergence over the full window used for validation, not just the recent tail.

• Extend the comparison script to use fetch_range (paginated, raw bars) for both providers over ≥ 12 months for M5, M15, H1 (and H4 if in scope).
• Record per TF:
  • % of bar timestamps present in both feeds; list of missing/extra bars (gaps, holidays, weekend boundary handling).
  • OHLC |Δ| distribution: p50 / p95 / p99 / max, in $ and % of price.
  • Count of bars dropped by validate_and_clean on each feed (10σ outliers, malformed).
  • Bar-close timestamp conventions (alignment, DST behavior around US transitions).
• Pass criteria (measured on the weekend-filtered bar grid — the aggregator's NY-anchored _is_forex_weekend decides which bars exist for signal purposes, so raw timestamp overlap includes coverage-only diffs like Tiingo's Sunday bars and is NOT the criterion): ≥ 99.5% overlap of signal-eligible bar timestamps; Close p95 |Δ| ≤ 0.01% of price; no systematic bias (mean Δ ≈ 0); no unexplained multi-hour gaps unique to one feed. (Raw-overlap result 2026-08-14: 94.6–96.6% vs Tiingo — entirely explained by Tiingo's ~22h/week of flat Sunday bars, which the weekend filter gates out.)
• Write findings to backtest/research/feed_divergence_<date>.md with tables and the exact commands used.

Phase 2 — Signal-level replay parity

Goal: measure how often candle divergence actually changes a decision.

• Replay bar-by-bar through the real analyze_from_dataframes (news off) for both feeds over the same ≥ 12-month window, per TF (M5, M15, H1). Reuse the gpu_backtest replay harness; disable the parquet cache (--no-cache) or use separate cache keys per provider.
• Compare per bar: signal direction, composite score, quality grade, gate outcomes (which gate fired), setup presence, strategy label (XAU ATR / XAU_PB / XAU_CP), entry/SL/TP prices.
• Metrics:
  • % bars with identical direction+grade.
  • Direction flips, grade changes, gate-only changes (setup appears/disappears).
  • Distribution of composite Δ on mismatched bars.
  • For matching signals: mean/max |Δ| of entry, SL, TP1/TP2 in $ and in ATR fractions.
• Result (Oanda↔Tiingo): 98.32% / 98.16% same direction (M15/M5), zero opposite-sign flips; 93.41% / 94.21% same direction+grade+setup. The original ≥97% direction+grade proposal is too strict for benign grade-letter changes and was not met; fallback acceptance is direction ≥98%, zero sign flips, and marginal/boundary-hour-only flips. See signal_parity_2026-08-14.md.

Phase 3 — Backtest re-pin on the new feed

Goal: a new pinned book on the new provider's bars, evaluated under rule 1 acceptance criteria.

• Provider selection: gpu_backtest.py has no --provider flag — it builds the provider from data_providers.active in config.json (gpu_backtest.py:1753). A cross-provider run requires flipping that config line. The parquet cache is already keyed per provider (gpu_backtest.py:984 — oanda_-prefixed files vs legacy unprefixed TD files), so TD and Oanda data cannot mix in cache; --no-cache is only needed to force a refetch.
• Run the full backtest on new-feed bars, both cost assumptions:

  python backtest/gpu_backtest.py --pair XAUUSD --days 365 --spread 0.30 --export-csv backtest/repin_<provider>_m15_030.csv
  python backtest/gpu_backtest.py --pair XAUUSD --days 365 --spread 0.65 --export-csv backtest/repin_<provider>_m15_065.csv
  

  (repeat for each TF under test; note the Backend: log line and provider in the report per backtest/instructions_for_backtest.txt).
• Split into train/validation halves; require positive total R in both halves and PF ≥ 1.3 at realistic cost (rule 1), with trailing exits and next-open fills untouched.
• Marginal configs additionally pass the 2024-06 → 2025-03 OOS window (within-year splits alone have approved configs that later failed OOS).
• Compare against the pinned Twelve-Data book trade-by-trade AMENDED 2026-08-14: skipped with justification — Phase 1 showed TD bars disagree on direction 19% of the time on M5 with a systematic ~$1 bias, so a TD-pinned book is a different instrument's book and a trade-level diff carries no decision value. Phantom-trade classes remain documented (e.g. the weekend-calendar fix already dropped exactly one live-impossible M5 trade).
• Deliverable: re-pinned book + report in backtest/research/; update the reference numbers in CLAUDE.md if the book is adopted. Current-HEAD M5 canonical book: backtest/oanda_m5_12mo_065_nopb_20260814.csv, +106.82R PF 1.51.

Phase 4 — Live shadow comparison

Goal: confirm parity under real fetch depths, caching, and boundary timing — not just history.

• Run a shadow scan loop (no orders, no Telegram) for ≥ 2 weeks: each candle close, scan with both providers, log direction/grade/gates/setup to a side-by-side file or SQLite table.
• Include at least one weekend transition and one news-heavy day in the window.
• Track: scan-to-scan agreement rate, feed-lag incidents ("feed lagging — just-closed bar missing"), staleness warnings, spread readings (Oanda real vs Tiingo default).
• Acceptance thresholds (baseline = Phase 2 replay parity): ≥ 98% direction agreement; zero opposite-sign (BUY↔SELL) flips; direction flips confined to actionable↔NEUTRAL at marginal composites, clustered only at Friday-close/Sunday-reopen (UTC 22–23) and Asian (03–04) hours; setup-level drift (SL/TP) within the Phase 2 envelope (mean |Δ| ≲ $3, p95 ≲ $17 on M15); every feed-lag/staleness incident explained.
• H4 caution: live H4 fetches only 380 bars vs unbounded backtest history (known ~2% indicator-signal flip from seeding alone) — keep H4 labeled "unproven" until a dedicated campaign closes this.

Phase 5 — Cutover checklist

• config.json data_providers.active: "oanda" and data_providers.streaming: "tiingo" set deliberately (verified 2026-08-14 — Oanda cannot stream; Tiingo streams).
• Cost model: dual reporting adopted (owner decision 2026-08-14) — $0.65 (measured Oanda round trip) is the realistic headline, $0.30 retained for cross-book comparability with TD-era books. ⚠️ Cost-gate coupling: config.py round_trip_cost (0.30) is NOT just a backtest knob — it feeds the live TP1 spread gate (min_tp = round_trip_cost × min_tp_cost_multiple, three sites in signal_aggregator.py). Raising it to 0.65 moves the TP1 floor from ~$9 to ~$19.50 and is a rule-1 revalidation event, not a config tweak. round_trip_cost stays 0.30.
• QA suite green: python test_all.py (381 tests; verified 2026-08-14).
• Week 1 post-cutover: daily review of logs/signals.db rows vs shadow log; alert on grade/direction disagreement beyond the Phase 2 baseline rate.
• Rollback: revert data_providers.active; pinned books of the previous feed remain valid for comparison. ⚠️ Twelve Data is a permanent registered fallback but is signal-incompatible (Phase 1) — on an Oanda candle outage, stop scanning rather than switching active to twelve_data; a TD-fed book cannot be compared to the Oanda baseline.
• Docs sync (CLAUDE.md rule): docs/user_guide.html backtest numbers reflect the Oanda-pinned books, not TD-era numbers.

Open questions — RESOLVED 2026-08-14 (owner decisions)

• Adopt which spread as canonical → dual reporting: $0.65 realistic headline + $0.30 cross-book comparability; round_trip_cost stays 0.30 (gate coupling above).
• Tiingo role → streaming-only. Tiingo remains registered for the WS tick stream and comparison research, but is not a designated candle fallback; candle fallback is a manual halt-and-decide, never an automatic TD failover.
• H4 seeding campaign (parked since 2026-08-04) → still needs an owner; H4 stays "unproven" until then.