ForexBot2/docs/data_source_change_test_plan.md

# Test Plan — Data Source Change Validation

**Purpose:** determine whether switching `data_providers.active` (e.g. Twelve Data → Oanda, or any future provider swap) changes signal calculations materially, and gate the switch on evidence.

**Status:** in progress, 2026-08-14

> **Progress (2026-08-14):** Phase 0 ✅, Phase 1 ✅ (`backtest/research/feed_divergence_2026-08-14.md`), Phase 2 ✅ (`backtest/research/signal_parity_2026-08-14.md`), Phase 3 ✅ (`backtest/research/repin_verification_2026-08-14.md` — re-pin already existed), Phase 4 🔨 tool ready (`scripts/shadow_scan.py`, needs a 2-week run — launch manually per owner decision), Phase 5 owner decisions recorded (dual cost reporting; Tiingo streaming-only; see bottom).
**Context:** Oanda has been the live feed since 2026-07-28. The backtest books were **re-pinned on Oanda bars on 2026-08-04** (`backtest/oanda_*.csv` — 27 books, now the baseline; all TD-era books superseded). Cost reporting is dual: $0.30 retained for cross-book comparability, $0.65 is the realistic headline (measured Oanda round trip $0.65–0.72). Twelve Data was proven signal-incompatible in Phase 1 (~19% of M5 bars disagree on direction; systematic ~$1 bias) — it is NOT a valid candle fallback.

**Why signals can differ between feeds** (mechanisms to test):

- Feeds publish different mid prices → tiny OHLC diffs shift every indicator; near a threshold (RSI band, `FB2_EMA_LO` EMA200-proximity band, composite/grade cutoffs, M5 min-score gate) one bar's discrete score flips.
- Gate edge cases turn a tiny price diff into "setup generated" vs "no signal".
- ATR-derived SL/TP/trail levels differ even when the signal is identical.
- The shared 10σ cleaning pass can drop different bars per feed.
- Volume present (Oanda) vs absent (Tiingo FX); real spread (Oanda MBA) vs pair default (Tiingo).
- Known reference point: H4 seeding divergence — same provider, different fetch depth flipped ~2% of indicator signals (composite Δ up to 12.3). Cross-feed diffs act the same way.

---

## Phase 0 — Recent-bar parity smoke test ✅ DONE

- **Tool:** `scripts/compare_oanda_tiingo.py` (removed 2026-09-06 once Phases 1-2 superseded it; the result is recorded here)
- **Result 2026-08-14:** 10/10 timestamp alignment on M5/M15/H1, Close max |Δ| ≤ $0.175 (≤ 0.004% of price), no directional bias.
- **Verdict:** recent-tail parity confirmed. Does NOT cover historical depth — proceed to Phase 1.

## Phase 1 — Historical candle divergence study

**Goal:** quantify raw-data divergence over the full window used for validation, not just the recent tail.

- [x] Extend the comparison script to use `fetch_range` (paginated, raw bars) for both providers over **≥ 12 months** for M5, M15, H1 (and H4 if in scope).
- [x] Record per TF:
  - % of bar timestamps present in both feeds; list of missing/extra bars (gaps, holidays, weekend boundary handling).
  - OHLC |Δ| distribution: p50 / p95 / p99 / max, in $ and % of price.
  - Count of bars dropped by `validate_and_clean` on each feed (10σ outliers, malformed).
  - Bar-close timestamp conventions (alignment, DST behavior around US transitions).
- [x] **Pass criteria (measured on the weekend-filtered bar grid — the aggregator's NY-anchored `_is_forex_weekend` decides which bars exist for signal purposes, so raw timestamp overlap includes coverage-only diffs like Tiingo's Sunday bars and is NOT the criterion):** ≥ 99.5% overlap of *signal-eligible* bar timestamps; Close p95 |Δ| ≤ 0.01% of price; no systematic bias (mean Δ ≈ 0); no unexplained multi-hour gaps unique to one feed. (Raw-overlap result 2026-08-14: 94.6–96.6% vs Tiingo — entirely explained by Tiingo's ~22h/week of flat Sunday bars, which the weekend filter gates out.)
- [x] Write findings to `backtest/research/feed_divergence_<date>.md` with tables and the exact commands used.

## Phase 2 — Signal-level replay parity

**Goal:** measure how often candle divergence actually changes a *decision*.

- [x] Replay bar-by-bar through the real `analyze_from_dataframes` (news off) for both feeds over the same ≥ 12-month window, per TF (M5, M15, H1). Reuse the gpu_backtest replay harness; disable the parquet cache (`--no-cache`) or use separate cache keys per provider.
- [x] Compare per bar: signal direction, composite score, quality grade, gate outcomes (which gate fired), setup presence, strategy label (XAU ATR / XAU_PB / XAU_CP), entry/SL/TP prices.
- [x] Metrics:
  - % bars with identical direction+grade.
  - Direction flips, grade changes, gate-only changes (setup appears/disappears).
  - Distribution of composite Δ on mismatched bars.
  - For matching signals: mean/max |Δ| of entry, SL, TP1/TP2 in $ and in ATR fractions.
- [x] **Result (Oanda↔Tiingo):** 98.32% / 98.16% same direction (M15/M5), zero opposite-sign flips; 93.41% / 94.21% same direction+grade+setup. The original ≥97% direction+grade proposal is too strict for benign grade-letter changes and was not met; fallback acceptance is direction ≥98%, zero sign flips, and marginal/boundary-hour-only flips. See `signal_parity_2026-08-14.md`.

## Phase 3 — Backtest re-pin on the new feed

**Goal:** a new pinned book on the new provider's bars, evaluated under rule 1 acceptance criteria.

- [x] **Provider selection:** `gpu_backtest.py` has **no `--provider` flag** — it builds the provider from `data_providers.active` in config.json (`gpu_backtest.py:1753`). A cross-provider run requires flipping that config line. The parquet cache is already keyed per provider (`gpu_backtest.py:984` — `oanda_`-prefixed files vs legacy unprefixed TD files), so TD and Oanda data cannot mix in cache; `--no-cache` is only needed to force a refetch.
- [x] Run the full backtest on new-feed bars, both cost assumptions:
  ```bash
  python backtest/gpu_backtest.py --pair XAUUSD --days 365 --spread 0.30 --export-csv backtest/repin_<provider>_m15_030.csv
  python backtest/gpu_backtest.py --pair XAUUSD --days 365 --spread 0.65 --export-csv backtest/repin_<provider>_m15_065.csv
  ```
  (repeat for each TF under test; note the `Backend:` log line and provider in the report per `backtest/instructions_for_backtest.txt`).
- [x] Split into train/validation halves; require **positive total R in both halves and PF ≥ 1.3 at realistic cost** (rule 1), with trailing exits and next-open fills untouched.
- [x] Marginal configs additionally pass the **2024-06 → 2025-03 OOS window** (within-year splits alone have approved configs that later failed OOS).
- [x] ~~Compare against the pinned Twelve-Data book trade-by-trade~~ **AMENDED 2026-08-14:** skipped with justification — Phase 1 showed TD bars disagree on direction 19% of the time on M5 with a systematic ~$1 bias, so a TD-pinned book is a different instrument's book and a trade-level diff carries no decision value. Phantom-trade classes remain documented (e.g. the weekend-calendar fix already dropped exactly one live-impossible M5 trade).
- [x] **Deliverable:** re-pinned book + report in `backtest/research/`; update the reference numbers in CLAUDE.md if the book is adopted. Current-HEAD M5 canonical book: `backtest/oanda_m5_12mo_065_nopb_20260814.csv`, +106.82R PF 1.51.

## Phase 4 — Live shadow comparison

**Goal:** confirm parity under real fetch depths, caching, and boundary timing — not just history.

- [ ] Run a shadow scan loop (no orders, no Telegram) for **≥ 2 weeks**: each candle close, scan with both providers, log direction/grade/gates/setup to a side-by-side file or SQLite table.
- [ ] Include at least one weekend transition and one news-heavy day in the window.
- [ ] Track: scan-to-scan agreement rate, feed-lag incidents ("feed lagging — just-closed bar missing"), staleness warnings, spread readings (Oanda real vs Tiingo default).
- [ ] **Acceptance thresholds (baseline = Phase 2 replay parity):** ≥ 98% direction agreement; **zero** opposite-sign (BUY↔SELL) flips; direction flips confined to actionable↔NEUTRAL at marginal composites, clustered only at Friday-close/Sunday-reopen (UTC 22–23) and Asian (03–04) hours; setup-level drift (SL/TP) within the Phase 2 envelope (mean |Δ| ≲ $3, p95 ≲ $17 on M15); every feed-lag/staleness incident explained.
- [ ] H4 caution: live H4 fetches only 380 bars vs unbounded backtest history (known ~2% indicator-signal flip from seeding alone) — keep H4 labeled "unproven" until a dedicated campaign closes this.

## Phase 5 — Cutover checklist

- [x] `config.json` `data_providers.active: "oanda"` and `data_providers.streaming: "tiingo"` set deliberately (verified 2026-08-14 — Oanda cannot stream; Tiingo streams).
- [x] Cost model: **dual reporting adopted (owner decision 2026-08-14)** — $0.65 (measured Oanda round trip) is the realistic headline, $0.30 retained for cross-book comparability with TD-era books. ⚠️ **Cost-gate coupling:** `config.py round_trip_cost` (0.30) is NOT just a backtest knob — it feeds the live TP1 spread gate (`min_tp = round_trip_cost × min_tp_cost_multiple`, three sites in `signal_aggregator.py`). Raising it to 0.65 moves the TP1 floor from ~$9 to ~$19.50 and is a rule-1 revalidation event, not a config tweak. `round_trip_cost` stays 0.30.
- [x] QA suite green: `python test_all.py` (381 tests; verified 2026-08-14).
- [ ] Week 1 post-cutover: daily review of `logs/signals.db` rows vs shadow log; alert on grade/direction disagreement beyond the Phase 2 baseline rate.
- [ ] Rollback: revert `data_providers.active`; pinned books of the previous feed remain valid for comparison. ⚠️ Twelve Data is a permanent *registered* fallback but is **signal-incompatible** (Phase 1) — on an Oanda candle outage, stop scanning rather than switching `active` to `twelve_data`; a TD-fed book cannot be compared to the Oanda baseline.
- [x] Docs sync (CLAUDE.md rule): `docs/user_guide.html` backtest numbers reflect the Oanda-pinned books, not TD-era numbers.

## Open questions — RESOLVED 2026-08-14 (owner decisions)

- ~~Adopt which spread as canonical~~ → **dual reporting**: $0.65 realistic headline + $0.30 cross-book comparability; `round_trip_cost` stays 0.30 (gate coupling above).
- ~~Tiingo role~~ → **streaming-only**. Tiingo remains registered for the WS tick stream and comparison research, but is not a designated candle fallback; candle fallback is a manual halt-and-decide, never an automatic TD failover.
- H4 seeding campaign (parked since 2026-08-04) → **still needs an owner**; H4 stays "unproven" until then.