ForexBot2/OANDA_MIGRATION_ROADMAP.md

Full Migration to Oanda v20 as Market-Data Source — Roadmap

Status: partially executed (updated 2026-08-03). Phases 1–2 shipped 2026-07-28 via the providers/ package refactor and the live feed switched to Oanda (data_providers.active: "oanda"). Phases 3–6 remain unbuilt: the level-breach stream monitor is simply disabled for Oanda, and the pinned backtest books are still Twelve-Data-pinned at the $0.30 cost assumption. Owner decision 2026-07-24 stands: Twelve Data is a permanent fallback (no removal phase).

Context

The daemon already executes trades on an Oanda practice account, and since 2026-07-28 the signal feed (Telegram bot, CLI, backtester) also runs on Oanda. Oanda's API is free with no metering (120 req/s vs TD's 55 credits/min), serves real bid/ask spreads, and matches the actual trading calendar. The 2026-07-24 divergence study (backtest/research/oanda_td_divergence.py) proved the switch is not plumbing-only: steady-state bars differ by median $0.69 (11% of M15 ATR), bar direction agrees only 89.9%, TD prints 4,000 phantom Saturday bars/yr + 21:00 UTC rollover bars that contaminate Monday indicator windows, and Oanda's real spread ($0.72 liquid / $0.84 rollover) is ~2.4× the $0.30 the validated books assume. So the migration = plumbing (Phases 1–3, inert) + shadow evidence (4) + book re-validation with owner gates (5) + reversible cutover (6). CLAUDE.md rules 1 (no tuned-value change without re-validation) and 4 (live == backtest parity) govern throughout.

Master sequencing rule: Phases 1–3 land inert behind data_providers.active == "twelve_data". Nothing changes live behavior until the Phase 6 config flip, which stays reversible forever (TD is a permanent fallback).

Phase 1 — Provider plumbing (inert)

Files: oanda_provider.py, data_provider.py, config.py, config.json.example.

1. Constructor self-resolution: OandaDataProvider.__init__(config, api_token=None, base_url=None) — defaults resolve via existing app_config.oanda_api_token() / oanda_base_url(); explicit args keep oanda_daemon.py:180 working. Fail lazily at first fetch (mirror ForexDataProvider's _td is None pattern) so credential-less construction doesn't break the hermetic test header.
2. Registry: add "oanda": OandaDataProvider to PROVIDER_REGISTRY (data_provider.py:392); import at bottom of data_provider.py (one-direction edge, no cycle — oanda_provider must never import from data_provider). Add PROVIDER_NAME class attr to both providers. Cutover config needs an "oanda": {} block under data_providers (creds stay in the existing top-level oanda section / env).
3. Dedupe shared constants: move CANDLE_SECONDS / BARS_PER_DAY to config.py, alias as class attrs in both providers — all existing call sites (signal_aggregator.py:18,217,640,942, telegram_bot.py:1062, gpu_backtest.py:975,1706, tests) keep working unchanged. Also move _is_forex_weekend to config.py eventually (currently duplicated + sync-tested).
4. _fetch_window(symbol, tf, start, end) on both providers — the backtester's common interface. Contract: raw closed bars labeled in [start, end], oldest-first, capitalized OHLCV, UTC index; empty DataFrame (not exception) for empty windows; caller keeps windows ≤ ~4500 bars. TD: thin wrapper over existing _fetch_twelve_data(start_date=, end_date=) (keep its exception behavior byte-identical). Oanda: new from/to RFC3339 request (price=MBA, dailyAlignment=0, alignmentTimezone=Etc/UTC); refactor _fetch_oanda_candles's parsing body (complete-filter, mid extraction, spread cache) into a shared _parse_candles(). The from+count quirk (Oanda's count includes the boundary bar; see backtest/research/fetch_oanda_m15.py) does not apply to from/to windows. Add a no-op-by-construction _drop_forming_candle to OandaDataProvider (gpu_backtest.py:1025 calls it unconditionally). No throttle needed for Oanda.
5. Tests (extend Module 15): registry selection; constructor self-resolution + explicit-args-win; _fetch_window both providers (mock _session.get / _fetch_twelve_data); CANDLE_SECONDS/BARS_PER_DAY cross-provider sync; provider-contract hasattr test covering everything callers reach for (fetch_ohlcv, get_pair_info, clear_cache, _is_data_fresh, _has_latest_closed_bar, _validate_and_clean, _drop_forming_candle, _fetch_window, CANDLE_SECONDS, BARS_PER_DAY, _cache, _cache_timestamps, PROVIDER_NAME); port Module 11's two cache/boundary tests (test_all.py:2278,2313) to Oanda variants patching _fetch_oanda_candles. Tests that swap set_app_config must restore _TEST_APP in finally.

Phase 2 — Backtester provider-agnosticism

File: backtest/gpu_backtest.py (fetch_history, lines 943–1036).

1. Line 984: provider._fetch_twelve_data(...) → provider._fetch_window(...). Keep the backward-stepping chunk loop, empty_streak bail, cursor stepping byte-identical (rule 6 — this loop produced every pinned cache).
2. Lines 975/1706: ForexDataProvider.BARS_PER_DAY → provider.BARS_PER_DAY.
3. Provider-namespaced cache keys in _cache_path (line 938): TD keeps legacy names (pinned parquets stay loadable, needed for A/B), Oanda gets oanda_ prefix (oanda_XAU_USD_15min_....parquet) — no collisions. Comment the asymmetry.
4. clean=True/False raw-path semantics untouched.
5. Verification before merge: one M15 12-month TD replay pre/post commit — trade books must be identical. Alignment audit: fetch one month of Oanda H4 + D1 via _fetch_window, assert label sets match TD's cached labels (H4 at 00/04/08 UTC, D1 at 00:00 UTC) — a mismatch would silently shift every MTF confirmation.
6. Tests: fetch_history dispatch with a stub provider (concat/dedupe/trim, clean=False skips cleaning); _cache_path namespacing regression.

Phase 3 — Telegram bot: TD WebSocket → Oanda pricing stream

File: telegram_bot.py; new resolver in app_config.py.

1. New oanda_stream_url() resolver: env OANDA_STREAM_URL → config.json oanda.stream_url (add to _KNOWN_OANDA) → derived oanda_base_url().replace("api-", "stream-", 1).
2. Pure parser _parse_stream_line(line) -> Optional[(pair_symbol, mid_price)]: type==PRICE → mid from closeoutBid/Ask (fallback bids[0]/asks[0]); HEARTBEAT/garbage → None. _STREAM_TO_PAIR from OandaDataProvider.OANDA_INSTRUMENT (replaces TD-display-keyed _WS_TO_PAIR).
3. Reader thread (no new deps): requests.get(.../v3/accounts/{id}/pricing/stream?instruments=..., stream=True, timeout=(10,30)), iterate iter_lines(chunk_size=1) — chunk_size=1 is load-bearing (urllib3 buffering vs 5s server heartbeats); push lines to asyncio queue via loop.call_soon_threadsafe; interrupt a blocked read with resp.close() from the async side.
4. Async monitor _oanda_stream_monitor: dedicated thread (NOT the 4-worker scan executor), await asyncio.wait_for(queue.get(), timeout=15) as staleness watchdog (3× server heartbeat) → reconnect with existing exponential-backoff pattern. PRICE lines feed the identical existing body: _check_level_breach → _handle_breach → M15 rescan, 5-min cooldown. No client heartbeat.
5. Provider-gated selection in _start_scheduler (line 889): active_provider == "oanda" → stream monitor, else existing _ws_monitor. This is the rollback hinge and stays permanently (TD fallback). _stop_scheduler must stop the reader thread. Generalize /status labels (_status_text iterates SCAN_PAIRS, not _WS_TO_PAIR).
6. Tests: _parse_stream_line variants (PRICE/closeout, PRICE/arrays, HEARTBEAT, malformed, unknown instrument); _STREAM_TO_PAIR round-trip; monitor selection under both active values; watchdog-reconnect (structure one connection lifecycle as a testable inner coroutine).

Phase 4 — Shadow / parity evidence (3–4 weeks, no hot-path code)

The daemon already runs the full signal stack on Oanda bars. Formalize evidence before any cutover:

1. Code parity (exact): replay the daemon's shadow window through the backtester on Oanda data (Phase 2 enables this); diff against daemon-logged decisions per (tf, bar_close_ts) — signal, quality, direction, entry/SL/TP. Acceptance: 100% modulo documented daemon divergences (single-position rule, quality gate). Any mismatch is a bug, not feed noise.
2. Feed divergence (measured, not zero-gated): backtest/research/shadow_compare.py parses logs/telegram.log + daemon log into per-scan rows: signal agreement rate, direction flips, grade drift, clustering by hour/weekday. Expect high-single-digit % divergence concentrated near grade thresholds and Monday sessions (TD Saturday/rollover bars). Deliverable: written note attributing every divergence class.
3. Operational telemetry: boundary grace-retry frequency (is Oanda's just-closed bar present at boundary+10s?), get_spread() time series (live input to the Phase 5 cost decision), stream disconnect counts, Monday-open indicator snapshots TD-vs-Oanda.
4. Owner note: at cutover the data feed host family should match the execution host (practice vs live pricing can micro-differ).

Phase 5 — Re-validation campaign (rule 1, owner gates)

Fetch Oanda history (M5/M15/H1/H4/D1, 2020-07 → present; Oanda XAU reaches ~2005). Run matrix per primary TF (M5/M15/H1, current factory configs): 12-mo train/val halves, 5-yr, OOS 2024-06→2025-03 — each at --spread 0.30 (pure feed-effect A/B vs pinned TD books), 0.70 (honest Oanda cost), 0.85 (rollover stress). Caution: Jul–Nov 2025 is provider-sensitive (weekly offsets −$6..+$18.5) — lean on 5-yr + OOS for decisions. Acceptance unchanged: positive both halves, PF ≥ 1.3, trailing exits, next-open fills, OOS positive.

Owner decision gates (never silent):

1. Canonical cost — (a) round_trip_cost=0.70 everywhere (moves the live TP1 spread gate $9→$21 — a strategy re-tune); (b) default_spread=0.70 for backtest PnL but retune min_tp_cost_multiple 30→~13 to preserve the validated $9 gate; (c) keep $0.30 with documented optimism (only if $0.70 books pass anyway). Full acceptance matrix at chosen values.
2. Degradation handling per failing TF: re-run tuning grids on Oanda data / drop TF from scanning.timeframes / keep Oanda execution-only (status quo).
3. Re-pin: passing books become the new pinned CSVs + factory-docstring evidence; CLAUDE.md tuned-value annotations update in the cutover commit, not before.

Files (per decisions): pair_signals/xauusd.py, config.py (PAIR_INFO.default_spread, SignalConfig.round_trip_cost/min_tp_cost_multiple), test_all.py tuned-constant assertions, CLAUDE.md.

Phase 6 — Cutover (reversible forever; no TD removal)

1. config.json: data_providers.active: "oanda" + "oanda": {} block; restart bot. Everything flips through two seams: make_data_provider (bot shared dp, CLI, backtester) and the Phase 3 monitor selector. Any Phase 5 tuned-value changes land in this commit with their validation records.
2. Bake ≥1 week (full weekly cycle incl. Monday open): grace-retry rate, stream stability, alert plausibility vs daemon, /cache freshness.
3. Rollback = flip active back — TD code, key, caches, and WS monitor all stay maintained indefinitely (owner decision). Dual-provider posture: cross-provider sync tests (Phase 1) keep both honest; TD subscription/key must stay valid to remain a real fallback.
4. Docs: CLAUDE.md (pipeline header, data_provider section, env-var table + OANDA_STREAM_URL, backtester cache note, comparability note: "pre-Oanda books are TD-feed books"), docs/ guides if they name the data source.

Key risks

1. Cost-gate coupling: naively setting round_trip_cost=0.70 silently moves the live $9 TP1 gate to $21 — owner gate 5.1 exists precisely for this.
2. Cutover before re-pinning — feed divergence is material (11% of ATR, 10% direction flips); sequencing rule prevents it.
3. Hermetic test header — _TEST_APP pins twelve_data; provider-swapping tests must restore it or later modules fail mysteriously.
4. Stream liveness triad — chunk_size=1, resp.close() interruption, 15s watchdog.
5. H4/D1 alignment — audit before trusting any Oanda backtest.
6. Monday/rollover divergence class — attribute in Phase 4 evidence; do not "fix".

Verification (per phase)

• Phases 1–3: python test_all.py green (235 + new); live behavior provably unchanged (TD active).
• Phase 2: pre/post TD book identity diff; Oanda H4/D1 alignment audit.
• Phase 3: soak a second bot instance with active=oanda scratch config; observe stream reconnects in log.
• Phase 5: full acceptance matrix, owner sign-offs recorded in factory docstrings.
• Phase 6: 1-week bake checklist above; rollback drill (flip back, confirm TD scan cycle) once during bake.