ForexBot2/OANDA_MIGRATION_ROADMAP.md

Full Migration to Oanda v20 as Market-Data Source — Roadmap

Status: reference roadmap, not scheduled for implementation. Owner decision 2026-07-24: document only; when executed, Twelve Data stays as a permanent fallback (no removal phase).

Context

The oanda_demo branch already executes trades on an Oanda practice account, but the signal feed (Telegram bot, CLI, backtester) still runs on Twelve Data. Oanda's API is free with no metering (120 req/s vs TD's 55 credits/min), serves real bid/ask spreads, and matches the actual trading calendar. The 2026-07-24 divergence study (backtest/research/oanda_td_divergence.py) proved the switch is not plumbing-only: steady-state bars differ by median $0.69 (11% of M15 ATR), bar direction agrees only 89.9%, TD prints 4,000 phantom Saturday bars/yr + 21:00 UTC rollover bars that contaminate Monday indicator windows, and Oanda's real spread ($0.72 liquid / $0.84 rollover) is ~2.4× the $0.30 the validated books assume. So the migration = plumbing (Phases 1–3, inert) + shadow evidence (4) + book re-validation with owner gates (5) + reversible cutover (6). CLAUDE.md rules 1 (no tuned-value change without re-validation) and 4 (live == backtest parity) govern throughout.

Master sequencing rule: Phases 1–3 land inert behind data_providers.active == "twelve_data". Nothing changes live behavior until the Phase 6 config flip, which stays reversible forever (TD is a permanent fallback).

Phase 1 — Provider plumbing (inert)

Files: oanda_provider.py, data_provider.py, config.py, config.json.example.

1. Constructor self-resolution: OandaDataProvider.__init__(config, api_token=None, base_url=None) — defaults resolve via existing app_config.oanda_api_token() / oanda_base_url(); explicit args keep oanda_daemon.py:180 working. Fail lazily at first fetch (mirror ForexDataProvider's _td is None pattern) so credential-less construction doesn't break the hermetic test header.
2. Registry: add "oanda": OandaDataProvider to PROVIDER_REGISTRY (data_provider.py:392); import at bottom of data_provider.py (one-direction edge, no cycle — oanda_provider must never import from data_provider). Add PROVIDER_NAME class attr to both providers. Cutover config needs an "oanda": {} block under data_providers (creds stay in the existing top-level oanda section / env).
3. Dedupe shared constants: move CANDLE_SECONDS / BARS_PER_DAY to config.py, alias as class attrs in both providers — all existing call sites (signal_aggregator.py:18,217,640,942, telegram_bot.py:1062, gpu_backtest.py:975,1706, tests) keep working unchanged. Also move _is_forex_weekend to config.py eventually (currently duplicated + sync-tested).
4. _fetch_window(symbol, tf, start, end) on both providers — the backtester's common interface. Contract: raw closed bars labeled in [start, end], oldest-first, capitalized OHLCV, UTC index; empty DataFrame (not exception) for empty windows; caller keeps windows ≤ ~4500 bars. TD: thin wrapper over existing _fetch_twelve_data(start_date=, end_date=) (keep its exception behavior byte-identical). Oanda: new from/to RFC3339 request (price=MBA, dailyAlignment=0, alignmentTimezone=Etc/UTC); refactor _fetch_oanda_candles's parsing body (complete-filter, mid extraction, spread cache) into a shared _parse_candles(). The from+count quirk (Oanda's count includes the boundary bar; see backtest/research/fetch_oanda_m15.py) does not apply to from/to windows. Add a no-op-by-construction _drop_forming_candle to OandaDataProvider (gpu_backtest.py:1025 calls it unconditionally). No throttle needed for Oanda.
5. Tests (extend Module 15): registry selection; constructor self-resolution + explicit-args-win; _fetch_window both providers (mock _session.get / _fetch_twelve_data); CANDLE_SECONDS/BARS_PER_DAY cross-provider sync; provider-contract hasattr test covering everything callers reach for (fetch_ohlcv, get_pair_info, clear_cache, _is_data_fresh, _has_latest_closed_bar, _validate_and_clean, _drop_forming_candle, _fetch_window, CANDLE_SECONDS, BARS_PER_DAY, _cache, _cache_timestamps, PROVIDER_NAME); port Module 11's two cache/boundary tests (test_all.py:2278,2313) to Oanda variants patching _fetch_oanda_candles. Tests that swap set_app_config must restore _TEST_APP in finally.

Phase 2 — Backtester provider-agnosticism

File: backtest/gpu_backtest.py (fetch_history, lines 943–1036).

1. Line 984: provider._fetch_twelve_data(...) → provider._fetch_window(...). Keep the backward-stepping chunk loop, empty_streak bail, cursor stepping byte-identical (rule 6 — this loop produced every pinned cache).
2. Lines 975/1706: ForexDataProvider.BARS_PER_DAY → provider.BARS_PER_DAY.
3. Provider-namespaced cache keys in _cache_path (line 938): TD keeps legacy names (pinned parquets stay loadable, needed for A/B), Oanda gets oanda_ prefix (oanda_XAU_USD_15min_....parquet) — no collisions. Comment the asymmetry.
4. clean=True/False raw-path semantics untouched.
5. Verification before merge: one M15 12-month TD replay pre/post commit — trade books must be identical. Alignment audit: fetch one month of Oanda H4 + D1 via _fetch_window, assert label sets match TD's cached labels (H4 at 00/04/08 UTC, D1 at 00:00 UTC) — a mismatch would silently shift every MTF confirmation.
6. Tests: fetch_history dispatch with a stub provider (concat/dedupe/trim, clean=False skips cleaning); _cache_path namespacing regression.

Phase 3 — Telegram bot: TD WebSocket → Oanda pricing stream

File: telegram_bot.py; new resolver in app_config.py.

1. New oanda_stream_url() resolver: env OANDA_STREAM_URL → config.json oanda.stream_url (add to _KNOWN_OANDA) → derived oanda_base_url().replace("api-", "stream-", 1).
2. Pure parser _parse_stream_line(line) -> Optional[(pair_symbol, mid_price)]: type==PRICE → mid from closeoutBid/Ask (fallback bids[0]/asks[0]); HEARTBEAT/garbage → None. _STREAM_TO_PAIR from OandaDataProvider.OANDA_INSTRUMENT (replaces TD-display-keyed _WS_TO_PAIR).
3. Reader thread (no new deps): requests.get(.../v3/accounts/{id}/pricing/stream?instruments=..., stream=True, timeout=(10,30)), iterate iter_lines(chunk_size=1) — chunk_size=1 is load-bearing (urllib3 buffering vs 5s server heartbeats); push lines to asyncio queue via loop.call_soon_threadsafe; interrupt a blocked read with resp.close() from the async side.
4. Async monitor _oanda_stream_monitor: dedicated thread (NOT the 4-worker scan executor), await asyncio.wait_for(queue.get(), timeout=15) as staleness watchdog (3× server heartbeat) → reconnect with existing exponential-backoff pattern. PRICE lines feed the identical existing body: _check_level_breach → _handle_breach → M15 rescan, 5-min cooldown. No client heartbeat.
5. Provider-gated selection in _start_scheduler (line 889): active_provider == "oanda" → stream monitor, else existing _ws_monitor. This is the rollback hinge and stays permanently (TD fallback). _stop_scheduler must stop the reader thread. Generalize /status labels (_status_text iterates SCAN_PAIRS, not _WS_TO_PAIR).
6. Tests: _parse_stream_line variants (PRICE/closeout, PRICE/arrays, HEARTBEAT, malformed, unknown instrument); _STREAM_TO_PAIR round-trip; monitor selection under both active values; watchdog-reconnect (structure one connection lifecycle as a testable inner coroutine).

Phase 4 — Shadow / parity evidence (3–4 weeks, no hot-path code)

The daemon already runs the full signal stack on Oanda bars. Formalize evidence before any cutover:

1. Code parity (exact): replay the daemon's shadow window through the backtester on Oanda data (Phase 2 enables this); diff against daemon-logged decisions per (tf, bar_close_ts) — signal, quality, direction, entry/SL/TP. Acceptance: 100% modulo documented daemon divergences (single-position rule, quality gate). Any mismatch is a bug, not feed noise.
2. Feed divergence (measured, not zero-gated): backtest/research/shadow_compare.py parses logs/telegram.log + daemon log into per-scan rows: signal agreement rate, direction flips, grade drift, clustering by hour/weekday. Expect high-single-digit % divergence concentrated near grade thresholds and Monday sessions (TD Saturday/rollover bars). Deliverable: written note attributing every divergence class.
3. Operational telemetry: boundary grace-retry frequency (is Oanda's just-closed bar present at boundary+10s?), get_spread() time series (live input to the Phase 5 cost decision), stream disconnect counts, Monday-open indicator snapshots TD-vs-Oanda.
4. Owner note: at cutover the data feed host family should match the execution host (practice vs live pricing can micro-differ).

Phase 5 — Re-validation campaign (rule 1, owner gates)

Fetch Oanda history (M5/M15/H1/H4/D1, 2020-07 → present; Oanda XAU reaches ~2005). Run matrix per primary TF (M5/M15/H1, current factory configs): 12-mo train/val halves, 5-yr, OOS 2024-06→2025-03 — each at --spread 0.30 (pure feed-effect A/B vs pinned TD books), 0.70 (honest Oanda cost), 0.85 (rollover stress). Caution: Jul–Nov 2025 is provider-sensitive (weekly offsets −$6..+$18.5) — lean on 5-yr + OOS for decisions. Acceptance unchanged: positive both halves, PF ≥ 1.3, trailing exits, next-open fills, OOS positive.

Owner decision gates (never silent):

1. Canonical cost — (a) round_trip_cost=0.70 everywhere (moves the live TP1 spread gate $9→$21 — a strategy re-tune); (b) default_spread=0.70 for backtest PnL but retune min_tp_cost_multiple 30→~13 to preserve the validated $9 gate; (c) keep $0.30 with documented optimism (only if $0.70 books pass anyway). Full acceptance matrix at chosen values.
2. Degradation handling per failing TF: re-run tuning grids on Oanda data / drop TF from scanning.timeframes / keep Oanda execution-only (status quo).
3. Re-pin: passing books become the new pinned CSVs + factory-docstring evidence; CLAUDE.md tuned-value annotations update in the cutover commit, not before.

Files (per decisions): pair_signals/xauusd.py, config.py (PAIR_INFO.default_spread, SignalConfig.round_trip_cost/min_tp_cost_multiple), test_all.py tuned-constant assertions, CLAUDE.md.

Phase 6 — Cutover (reversible forever; no TD removal)

1. config.json: data_providers.active: "oanda" + "oanda": {} block; restart bot. Everything flips through two seams: make_data_provider (bot shared dp, CLI, backtester) and the Phase 3 monitor selector. Any Phase 5 tuned-value changes land in this commit with their validation records.
2. Bake ≥1 week (full weekly cycle incl. Monday open): grace-retry rate, stream stability, alert plausibility vs daemon, /cache freshness.
3. Rollback = flip active back — TD code, key, caches, and WS monitor all stay maintained indefinitely (owner decision). Dual-provider posture: cross-provider sync tests (Phase 1) keep both honest; TD subscription/key must stay valid to remain a real fallback.
4. Docs: CLAUDE.md (pipeline header, data_provider section, env-var table + OANDA_STREAM_URL, backtester cache note, comparability note: "pre-Oanda books are TD-feed books"), docs/ guides if they name the data source.

Key risks

1. Cost-gate coupling: naively setting round_trip_cost=0.70 silently moves the live $9 TP1 gate to $21 — owner gate 5.1 exists precisely for this.
2. Cutover before re-pinning — feed divergence is material (11% of ATR, 10% direction flips); sequencing rule prevents it.
3. Hermetic test header — _TEST_APP pins twelve_data; provider-swapping tests must restore it or later modules fail mysteriously.
4. Stream liveness triad — chunk_size=1, resp.close() interruption, 15s watchdog.
5. H4/D1 alignment — audit before trusting any Oanda backtest.
6. Monday/rollover divergence class — attribute in Phase 4 evidence; do not "fix".

Verification (per phase)

• Phases 1–3: python test_all.py green (235 + new); live behavior provably unchanged (TD active).
• Phase 2: pre/post TD book identity diff; Oanda H4/D1 alignment audit.
• Phase 3: soak a second bot instance with active=oanda scratch config; observe stream reconnects in log.
• Phase 5: full acceptance matrix, owner sign-offs recorded in factory docstrings.
• Phase 6: 1-week bake checklist above; rollback drill (flip back, confirm TD scan cycle) once during bake.