diff --git a/CLAUDE.md b/CLAUDE.md index 6316622..8b550b7 100644 --- a/CLAUDE.md +++ b/CLAUDE.md @@ -11,7 +11,7 @@ Every numeric threshold in `config.py` and `pair_signals/xauusd.py` was backtest ## Commands ```bash -python test_all.py # QA suite: 268 tests, 18 modules, no network (mocked) +python test_all.py # QA suite: 266 tests, 18 modules, no network (mocked) python main.py --pair XAUUSD --timeframe H1 --detail # CLI signal report (--json for full dump) python telegram_bot.py # the bot (refuses to boot without credentials) python oanda_daemon.py # auto-execution daemon on Oanda practice account @@ -49,7 +49,7 @@ Signal currency everywhere is `SignalStrength` (−2..+2). DataFrames are UTC tz ### Layers -- **`providers/`** — one class per provider subclassing `BaseDataProvider` (`providers/base.py`), which owns everything the pipeline depends on: TTL+boundary cache, forming-candle drop, freshness check, validate/clean, paginated `fetch_range` (feeds the backtester from any provider), optional tick streaming. Subclasses implement only `_fetch_raw` + symbol/interval maps. Registry + `make_data_provider` in `providers/__init__.py`; config.json `data_providers.active` selects the live feed. `data_provider.py`/`oanda_provider.py` are back-compat shims. A test enforces that subclasses don't override shared base methods. Tiingo is candles-only — do not re-add spread display from its free FX feed (two-tier indicative quotes, removed 2026-07-23). +- **`providers/`** — one class per provider subclassing `BaseDataProvider` (`providers/base.py`), which owns everything the pipeline depends on: TTL+boundary cache, forming-candle drop, freshness check, validate/clean, paginated `fetch_range` (feeds the backtester from any provider), optional tick streaming. Subclasses implement only `_fetch_raw` + symbol/interval maps. Registry + `make_data_provider` in `providers/__init__.py`; config.json `data_providers.active` selects the live feed. A test enforces that subclasses don't override shared base methods. Tiingo is candles-only — do not re-add spread display from its free FX feed (two-tier indicative quotes, removed 2026-07-23). - **`engine.py`** — `TechnicalEngine`: 9 scored indicators (ema_trend, rsi, macd, bollinger, adx, roc, keltner, divergence, candlestick) + ATR as context. Regime classification (trending_up/down, `trending` = mixed/direction-UNKNOWN, transitioning, ranging) drives per-indicator weight multipliers and regime-aware RSI reinterpretation. Compute order matters (ATR first; MACD always computed even when not a pillar — divergence needs the histogram). Private series ride on the result as `_underscore` attrs; scorers/formatters skip anything `_`-prefixed or weight-0. - **`signal_aggregator.py`** — MTF agreement, diminishing-decay penalty system, exhaustion hard block, quality grading, then an ordered chain of post-grade gates (session, XAU SELL quality/vol/hour, regime blocks, M5 noise gates), then three setup generators: XAU ATR (A+/A), generic swing-based, plus the M5 pullback (`XAU_PB`) and chart-pattern (`XAU_CP`, `chart_patterns.py`) entry paths. `TradeSetup.strategy` selects trailing params; trail params are stamped on the setup from the *producing* config so per-TF overrides reach formatters. - **`pair_signals/`** — per-pair config factories. **Always use the factory (`produce_signal` / `make_config`), never bare `PlatformConfig()`** — the factory applies the config.json overlay first, then the validated per-TF deltas (M5 hour-9 block, M15 trail arm 0.70, M15 SELL-hour block, etc.; evidence in the docstring). A test enforces factory ≡ inline-config equivalence. Confirmation map: M5→[M15,H1], M15→[M5,H1], H1→[M15,H4], H4→[H1,D1]. diff --git a/OANDA_MIGRATION_ROADMAP.md b/OANDA_MIGRATION_ROADMAP.md index f13ce18..cc4e4ba 100644 --- a/OANDA_MIGRATION_ROADMAP.md +++ b/OANDA_MIGRATION_ROADMAP.md @@ -1,10 +1,10 @@ # Full Migration to Oanda v20 as Market-Data Source — Roadmap -**Status: reference roadmap, not scheduled for implementation.** Owner decision 2026-07-24: document only; when executed, Twelve Data stays as a **permanent fallback** (no removal phase). +**Status: partially executed (updated 2026-08-03).** Phases 1–2 shipped 2026-07-28 via the `providers/` package refactor and the live feed switched to Oanda (`data_providers.active: "oanda"`). Phases 3–6 remain unbuilt: the level-breach stream monitor is simply disabled for Oanda, and the pinned backtest books are still Twelve-Data-pinned at the $0.30 cost assumption. Owner decision 2026-07-24 stands: Twelve Data is a **permanent fallback** (no removal phase). ## Context -The `oanda_demo` branch already executes trades on an Oanda practice account, but the signal feed (Telegram bot, CLI, backtester) still runs on Twelve Data. Oanda's API is free with no metering (120 req/s vs TD's 55 credits/min), serves real bid/ask spreads, and matches the actual trading calendar. The 2026-07-24 divergence study (`backtest/research/oanda_td_divergence.py`) proved the switch is **not plumbing-only**: steady-state bars differ by median $0.69 (11% of M15 ATR), bar direction agrees only 89.9%, TD prints ~4,000 phantom Saturday bars/yr + 21:00 UTC rollover bars that contaminate Monday indicator windows, and Oanda's real spread (~$0.72 liquid / $0.84 rollover) is ~2.4× the $0.30 the validated books assume. So the migration = plumbing (Phases 1–3, inert) + shadow evidence (4) + book re-validation with owner gates (5) + reversible cutover (6). CLAUDE.md rules 1 (no tuned-value change without re-validation) and 4 (live == backtest parity) govern throughout. +The daemon already executes trades on an Oanda practice account, and since 2026-07-28 the signal feed (Telegram bot, CLI, backtester) also runs on Oanda. Oanda's API is free with no metering (120 req/s vs TD's 55 credits/min), serves real bid/ask spreads, and matches the actual trading calendar. The 2026-07-24 divergence study (`backtest/research/oanda_td_divergence.py`) proved the switch is **not plumbing-only**: steady-state bars differ by median $0.69 (11% of M15 ATR), bar direction agrees only 89.9%, TD prints ~4,000 phantom Saturday bars/yr + 21:00 UTC rollover bars that contaminate Monday indicator windows, and Oanda's real spread (~$0.72 liquid / $0.84 rollover) is ~2.4× the $0.30 the validated books assume. So the migration = plumbing (Phases 1–3, inert) + shadow evidence (4) + book re-validation with owner gates (5) + reversible cutover (6). CLAUDE.md rules 1 (no tuned-value change without re-validation) and 4 (live == backtest parity) govern throughout. **Master sequencing rule:** Phases 1–3 land inert behind `data_providers.active == "twelve_data"`. Nothing changes live behavior until the Phase 6 config flip, which stays reversible forever (TD is a permanent fallback). diff --git a/backtest/research/calendar_store.py b/backtest/research/calendar_store.py deleted file mode 100644 index 3327e1c..0000000 --- a/backtest/research/calendar_store.py +++ /dev/null @@ -1,94 +0,0 @@ -"""Load and query the historical economic calendar (backtest side). - -The store is built by `ff_calendar_scraper.py`. Timestamps are unix -epoch (UTC) — verified against known release times and price-volatility -alignment (see scraper docstring). - -Key API: - cal = load_calendar() # full DataFrame, `utc` column - ev = precise_events(cal, ["USD"]) # high-impact, minute-accurate - mask = event_window_mask(bar_index, ev.utc, before, after) -""" -from pathlib import Path -from typing import Iterable, Optional - -import numpy as np -import pandas as pd - -PARQUET = Path(__file__).resolve().parent.parent / "cache" / "calendar.parquet" - -# timeLabel values that mean "no intraday time" — these events cannot -# anchor an M5/M15 window (German Prelim CPI is the notable High one). -_UNTIMED = {"All Day", "Tentative", ""} - - -def _parse_release_value(raw) -> float: - """'0.3%' / '-1.7%' / '228K' / '1.1M' -> float (nan if unparseable).""" - import re - if raw is None or (isinstance(raw, float) and np.isnan(raw)): - return np.nan - m = re.search(r"(-?\d+(?:\.\d+)?)\s*([kmbt])?", - str(raw).replace(",", ""), re.IGNORECASE) - if not m: - return np.nan - mult = {"k": 1e3, "m": 1e6, "b": 1e9, "t": 1e12}.get( - (m.group(2) or "").lower(), 1.0) - return float(m.group(1)) * mult - - -def load_calendar(path: Path = PARQUET) -> pd.DataFrame: - df = pd.read_parquet(path) - df["utc"] = pd.to_datetime(df.dateline, unit="s", utc=True) - for col in ("actual", "forecast", "previous"): - df[f"{col}_num"] = df[col].map(_parse_release_value) - df["surprise"] = df.actual_num - df.forecast_num - return df - - -def precise_events( - cal: pd.DataFrame, - currencies: Iterable[str] = ("USD",), - impacts: Iterable[str] = ("high",), - require_values: bool = False, -) -> pd.DataFrame: - """Events with a minute-accurate timestamp, filtered by currency/impact.""" - out = cal[cal.currency.isin(list(currencies)) - & cal.impactName.isin(list(impacts)) - & ~cal.timeMasked.astype(bool) - & ~cal.timeLabel.fillna("").isin(_UNTIMED)] - if require_values: - out = out.dropna(subset=["actual_num", "forecast_num"]) - return out.sort_values("utc").reset_index(drop=True) - - -def event_window_mask( - bar_index: pd.DatetimeIndex, - event_times: pd.Series, - before: pd.Timedelta = pd.Timedelta(minutes=30), - after: pd.Timedelta = pd.Timedelta(minutes=30), -) -> np.ndarray: - """Boolean mask: bar OPEN time falls inside [event-before, event+after). - - Use on a backtest's entry times to identify trades entered around - releases (avoidance counterfactuals, event-window cost models). - """ - starts = pd.DatetimeIndex(event_times) - before - ends = pd.DatetimeIndex(event_times) + after - order = np.argsort(starts.values) - starts, ends = starts.values[order], ends.values[order] - # merge overlapping windows, then interval-stab each bar time - merged_s, merged_e = [], [] - for s, e in zip(starts, ends): - if merged_e and s <= merged_e[-1]: - merged_e[-1] = max(merged_e[-1], e) - else: - merged_s.append(s) - merged_e.append(e) - ms = np.array(merged_s) - me = np.array(merged_e) - ts = bar_index.values - pos = np.searchsorted(ms, ts, side="right") - 1 - ok = pos >= 0 - res = np.zeros(len(ts), dtype=bool) - res[ok] = ts[ok] < me[pos[ok]] - return res diff --git a/config.py b/config.py index a83f377..35b2c77 100755 --- a/config.py +++ b/config.py @@ -146,7 +146,7 @@ class NewsConfig: # against the validated 12mo XAU M5 book (547 trades, +139.9R) # showed they suppress 25-35 trades worth +4.6..+7.2R/yr — trades # entered near USD releases OUTPERFORM (+0.475R vs +0.256R avg). - # Audit 2026-07-11 via backtest/research/calendar_store. + # Audit 2026-07-11 (calendar-replay study; tooling since removed). event_risk_caps_signals: bool = False # Whether headline sentiment moves the SIGNAL (aligned: +5 confidence; # conflicting: 10-point penalty). Warnings are always emitted diff --git a/data_provider.py b/data_provider.py deleted file mode 100755 index bcbb4db..0000000 --- a/data_provider.py +++ /dev/null @@ -1,24 +0,0 @@ -""" -ForexBot2 — Data Provider (back-compat shim) -============================================= -The provider implementations live in the `providers/` package: - - providers/base.py BaseDataProvider (shared contract + behavior) - providers/twelve_data.py TwelveDataProvider (SDK fetch, throttle, WS) - providers/oanda.py OandaDataProvider (v20 REST, real spread) - -`ForexDataProvider` is the historical name of the Twelve Data provider and -is kept as an alias so existing imports keep working. -""" - -from providers import ( # noqa: F401 - BaseDataProvider, - DataIntegrityError, - PROVIDER_REGISTRY, - TwelveDataProvider, - make_data_provider, - _is_forex_weekend, -) - -# Historical name for the Twelve Data provider. -ForexDataProvider = TwelveDataProvider diff --git a/docs/ForexBot2_Beginner_Guide.pdf b/docs/ForexBot2_Beginner_Guide.pdf deleted file mode 100644 index 14d0b60..0000000 Binary files a/docs/ForexBot2_Beginner_Guide.pdf and /dev/null differ diff --git a/docs/ForexBot2_User_Guide.pdf b/docs/ForexBot2_User_Guide.pdf deleted file mode 100644 index 965a36e..0000000 Binary files a/docs/ForexBot2_User_Guide.pdf and /dev/null differ diff --git a/docs/superpowers/plans/2026-07-21-centralized-config.md b/docs/superpowers/plans/2026-07-21-centralized-config.md deleted file mode 100644 index 972ae45..0000000 --- a/docs/superpowers/plans/2026-07-21-centralized-config.md +++ /dev/null @@ -1,980 +0,0 @@ -# Centralized config.json Implementation Plan - -> **For agentic workers:** REQUIRED SUB-SKILL: Use superpowers:subagent-driven-development (recommended) or superpowers:executing-plans to implement this plan task-by-task. Steps use checkbox (`- [ ]`) syntax for tracking. - -**Goal:** Move deployment settings (Telegram credentials, data-provider API keys/selection, scanned pairs, scan timeframes, per-pair base blocked hours, alert quality boot default, log level) into a gitignored `config.json` loaded once at startup, and delete the hardcoded credentials from source. - -**Architecture:** A new `app_config.py` module loads and validates `config.json` into an `AppConfig` dataclass (cached singleton), with resolver functions applying the precedence **env var → config.json → code default**. The pair factory calls `apply_pair_overrides()` on every fresh `PlatformConfig` *before* its validated per-timeframe deltas, so the bot, CLI, and backtester all pick up the overlay identically (live == backtest parity by construction). Spec: `docs/superpowers/specs/2026-07-21-centralized-config-design.md`. - -**Tech Stack:** Python 3.12 stdlib only (`json`, `os`, `dataclasses`). No new dependencies. - -## Global Constraints - -- **Zero signal-logic change.** No indicator weight, threshold, gate, or geometry value may change. With no `config.json`, or one holding the example defaults, every produced `PlatformConfig` must be byte-identical (`dataclasses.asdict` equality) to today's factory output — pinned by a test. -- The test suite is `python test_all.py` — a custom harness (`@test("name")` decorator, tests execute at import time, exit code non-zero on failure). It is NOT pytest; there is no single-test selection. "Run the test" always means `python test_all.py` and checking the named line in the output. Baseline before this work: **184 passed, 0 failed**. -- The suite must be hermetic: it must pass identically whether or not the developer's machine has a `config.json`, and regardless of `TELEGRAM_*`/`TWELVE_DATA_API_KEY` env vars. -- Secrets get **no code default**. After Task 4 the only places credentials exist are `config.json` (gitignored) and env vars. -- `config.json.example` (already committed at repo root) is the schema reference. `config.json` is already in `.gitignore` — never commit it. -- Repo root: `/home/harwinder/Development/ForexBot2`. Use `venv/bin/python` if bare `python` lacks pandas/numpy. -- Every commit message ends with: - ``` - Co-Authored-By: Claude Fable 5 - Claude-Session: https://claude.ai/code/session_01VPNF37MP9Kuv62B1VBrZ1b - ``` -- Work on branch `centralized-config` (create from `main` at start of Task 1: `git checkout -b centralized-config`). - ---- - -### Task 1: `app_config.py` — loader, validation, resolvers - -**Files:** -- Create: `app_config.py` -- Modify: `test_all.py` (new Module 13 before the SUMMARY block, ~line 2495; hermetic header after line 19) -- Test: `test_all.py` (same file — this repo keeps all tests in one suite) - -**Interfaces:** -- Consumes: `config.PAIR_INFO`, `config.Timeframe`, `config.XAUUSD`, `config.PlatformConfig` (all existing). -- Produces (later tasks rely on these exact names): - - `class AppConfigError(Exception)` - - `@dataclass class AppConfig` — fields: `telegram_token: str = ""`, `telegram_chat_id: str = ""`, `telegram_channel_id: str = ""`, `alert_min_quality: str = "C"`, `active_provider: str = "twelve_data"`, `provider_settings: Dict[str, Dict[str, Any]] = {}`, `scan_pairs: List[str] = [XAUUSD]`, `scan_timeframes: List[Timeframe] = [M5, M15, H1, H4]`, `pair_overrides: Dict[str, Dict[str, Any]] = {}`, `log_level: str = "INFO"` - - `load_app_config(path: str = CONFIG_PATH) -> AppConfig` - - `get_app_config() -> AppConfig` (cached), `set_app_config(app: Optional[AppConfig]) -> None` (test hook; `None` re-enables lazy load) - - `telegram_token(app=None) -> str`, `telegram_chat_id(app=None) -> str`, `telegram_channel_id(app=None) -> str`, `provider_api_key(app=None) -> str` - - `apply_pair_overrides(cfg: PlatformConfig, app: Optional[AppConfig] = None) -> PlatformConfig` (implemented here, wired in Task 2) - -- [ ] **Step 1: Create branch** - -```bash -cd /home/harwinder/Development/ForexBot2 && git checkout -b centralized-config -``` - -- [ ] **Step 2: Add the hermetic test header and Module 13 failing tests to `test_all.py`** - -In `test_all.py`, directly after `sys.path.insert(0, os.path.dirname(__file__))` (line 19), add: - -```python -# ── Hermetic app-config: the suite must not depend on the developer's -# local config.json or TELEGRAM_*/TWELVE_DATA_API_KEY env vars. The -# dummy TD key mirrors the pre-config.json behavior where a key was -# always present (TDClient built, no network at construction). -import json -import app_config -for _k in ("TELEGRAM_TOKEN", "TELEGRAM_CHAT_ID", "TELEGRAM_CHANNEL_ID", - "TWELVE_DATA_API_KEY"): - os.environ.pop(_k, None) -_TEST_APP = app_config.AppConfig( - provider_settings={"twelve_data": {"api_key": "test-key"}} -) -app_config.set_app_config(_TEST_APP) -``` - -Then insert Module 13 immediately BEFORE the `# SUMMARY` comment block near the end of the file (search for `SUMMARY`): - -```python -# ══════════════════════════════════════════════════════════════════════ -# MODULE 13: app_config.py (config.json overlay) -# ══════════════════════════════════════════════════════════════════════ -print("\n═══ Module 13: app_config.py ═══") - -import tempfile -from app_config import ( - AppConfig, AppConfigError, load_app_config, set_app_config, - telegram_token, telegram_chat_id, telegram_channel_id, - provider_api_key, apply_pair_overrides, -) - - -def _write_json(payload) -> str: - fh = tempfile.NamedTemporaryFile( - "w", suffix=".json", delete=False, encoding="utf-8") - if isinstance(payload, str): - fh.write(payload) - else: - json.dump(payload, fh) - fh.close() - return fh.name - - -_FULL_CFG = { - "telegram": {"token": "t0k", "chat_id": "111", "channel_id": "-100222", - "alert_min_quality": "B"}, - "data_providers": {"active": "twelve_data", - "twelve_data": {"api_key": "json-key"}}, - "scanning": {"pairs": ["XAU/USD"], "timeframes": ["M15", "H1"]}, - "pairs": {"XAU/USD": {"blocked_hours_utc": [3, 4]}}, - "log_level": "WARNING", -} - - -@test("app_config: valid config.json maps every field") -def _(): - app = load_app_config(_write_json(_FULL_CFG)) - assert app.telegram_token == "t0k" - assert app.telegram_chat_id == "111" - assert app.telegram_channel_id == "-100222" - assert app.alert_min_quality == "B" - assert app.active_provider == "twelve_data" - assert app.provider_settings["twelve_data"]["api_key"] == "json-key" - assert app.scan_pairs == [XAUUSD] - assert app.scan_timeframes == [Timeframe.M15, Timeframe.H1] - assert app.pair_overrides[XAUUSD]["blocked_hours_utc"] == [3, 4] - assert app.log_level == "WARNING" - - -@test("app_config: missing file returns pure defaults") -def _(): - app = load_app_config("/nonexistent/config.json") - assert app == AppConfig(), "absent config.json must yield pure defaults" - - -@test("app_config: malformed JSON raises AppConfigError") -def _(): - path = _write_json("{not json") - try: - load_app_config(path) - assert False, "should have raised" - except AppConfigError as e: - assert "not valid JSON" in str(e) - - -@test("app_config: unknown keys raise AppConfigError") -def _(): - for payload in ( - {"telgram": {}}, # top-level typo - {"telegram": {"tokn": "x"}}, # section typo - {"pairs": {"XAU/USD": {"blocked_hours": [1]}}}, # pair-block typo - {"scanning": {"pair": ["XAU/USD"]}}, # scanning typo - ): - try: - load_app_config(_write_json(payload)) - assert False, f"should have raised for {payload}" - except AppConfigError as e: - assert "unknown key" in str(e).lower() - - -@test("app_config: invalid values raise AppConfigError") -def _(): - bad_payloads = [ - {"pairs": {"XAU/USD": {"blocked_hours_utc": [24]}}}, # hour range - {"pairs": {"XAU/USD": {"blocked_hours_utc": "3"}}}, # not a list - {"pairs": {"EUR/USD": {"blocked_hours_utc": [3]}}}, # unknown pair - {"scanning": {"pairs": ["EUR/USD"]}}, # unknown pair - {"scanning": {"timeframes": ["M7"]}}, # bad TF name - {"scanning": {"timeframes": ["W1"]}}, # unscannable TF - {"data_providers": {"active": "polygon", - "twelve_data": {"api_key": "k"}}}, # no block - {"telegram": {"alert_min_quality": "Z"}}, # bad quality - {"log_level": "CHATTY"}, # bad level - ] - for payload in bad_payloads: - try: - load_app_config(_write_json(payload)) - assert False, f"should have raised for {payload}" - except AppConfigError: - pass - - -@test("app_config: env precedence env → config.json → default") -def _(): - app = load_app_config(_write_json(_FULL_CFG)) - with patch.dict(os.environ, {}, clear=True): - assert telegram_token(app) == "t0k" - assert telegram_chat_id(app) == "111" - assert telegram_channel_id(app) == "-100222" - assert provider_api_key(app) == "json-key" - with patch.dict(os.environ, {"TELEGRAM_TOKEN": "env-tok", - "TELEGRAM_CHAT_ID": "999", - "TWELVE_DATA_API_KEY": "env-key"}): - assert telegram_token(app) == "env-tok" - assert telegram_chat_id(app) == "999" - assert provider_api_key(app) == "env-key" - # channel: env set-but-empty means "disable feed" and must beat JSON - with patch.dict(os.environ, {"TELEGRAM_CHANNEL_ID": ""}): - assert telegram_channel_id(app) == "" -``` - -- [ ] **Step 3: Run to verify the new tests fail (import error)** - -```bash -python test_all.py 2>&1 | head -5 -``` - -Expected: `ModuleNotFoundError: No module named 'app_config'` (the hermetic header imports it before any test runs). - -- [ ] **Step 4: Write `app_config.py`** - -Create `app_config.py` with exactly this content: - -```python -""" -app_config.py — deployment-settings overlay (config.json) -========================================================= - -Loads the gitignored config.json at the repo root ONCE per process. -Deployment surface ONLY: Telegram credentials, data-provider selection -and API keys, scanned pairs/timeframes, per-pair base blocked hours, -alert-quality boot default, log level. Signal logic, weights, -thresholds and gates stay in config.py / pair_signals. - -Precedence for every setting: env var → config.json → code default. -Secrets have NO code default — with neither env nor config.json the -bot refuses to start (telegram_bot.main) and data fetches fail with a -named error (data_provider). - -Parity contract (test-enforced): with no config.json, or one holding -the config.json.example defaults, every produced PlatformConfig is -byte-identical to the pure-code defaults. apply_pair_overrides() runs -in the pair factory BEFORE the validated per-timeframe deltas, so the -bot, CLI and backtester all see the same overlay (live == backtest). - -Spec: docs/superpowers/specs/2026-07-21-centralized-config-design.md -""" - -import json -import logging -import os -from dataclasses import dataclass, field -from typing import Any, Dict, List, Optional - -from config import PAIR_INFO, PlatformConfig, Timeframe, XAUUSD - -logger = logging.getLogger(__name__) - -CONFIG_PATH = os.path.join( - os.path.dirname(os.path.abspath(__file__)), "config.json") - -_VALID_QUALITIES = ("A+", "A", "B", "C") -_VALID_LOG_LEVELS = ("DEBUG", "INFO", "WARNING", "ERROR", "CRITICAL") -# Timeframes the bot's candle-boundary scheduler can run (W1 has no loop). -_SCANNABLE_TFS = ("M5", "M15", "M30", "H1", "H4", "D1") - -_KNOWN_TOP = {"telegram", "data_providers", "scanning", "pairs", "log_level"} -_KNOWN_TELEGRAM = {"token", "chat_id", "channel_id", "alert_min_quality"} -_KNOWN_SCANNING = {"pairs", "timeframes"} -_KNOWN_PAIR_KEYS = {"blocked_hours_utc"} - - -class AppConfigError(Exception): - """config.json is present but invalid — fail fast at boot.""" - - -@dataclass -class AppConfig: - telegram_token: str = "" - telegram_chat_id: str = "" - telegram_channel_id: str = "" # "" = channel feed disabled - alert_min_quality: str = "C" - active_provider: str = "twelve_data" - provider_settings: Dict[str, Dict[str, Any]] = field(default_factory=dict) - scan_pairs: List[str] = field(default_factory=lambda: [XAUUSD]) - scan_timeframes: List[Timeframe] = field(default_factory=lambda: [ - Timeframe.M5, Timeframe.M15, Timeframe.H1, Timeframe.H4]) - pair_overrides: Dict[str, Dict[str, Any]] = field(default_factory=dict) - log_level: str = "INFO" - - -def _check_keys(section: Dict[str, Any], known: set, where: str) -> None: - unknown = sorted(set(section) - known) - if unknown: - raise AppConfigError( - f"config.json: unknown key(s) {unknown} in {where}; " - f"known keys: {sorted(known)}") - - -def load_app_config(path: str = CONFIG_PATH) -> AppConfig: - """Parse + validate config.json. Missing file → pure defaults.""" - app = AppConfig() - if not os.path.exists(path): - logger.warning( - "config.json not found at %s — running on code defaults " - "(secrets must come from environment variables)", path) - return app - try: - with open(path, encoding="utf-8") as fh: - raw = json.load(fh) - except json.JSONDecodeError as e: - raise AppConfigError(f"config.json is not valid JSON: {e}") from e - if not isinstance(raw, dict): - raise AppConfigError("config.json top level must be a JSON object") - _check_keys(raw, _KNOWN_TOP, "top level") - - tg = raw.get("telegram", {}) - _check_keys(tg, _KNOWN_TELEGRAM, '"telegram"') - app.telegram_token = str(tg.get("token", app.telegram_token)) - app.telegram_chat_id = str(tg.get("chat_id", app.telegram_chat_id)) - app.telegram_channel_id = str( - tg.get("channel_id", app.telegram_channel_id)) - quality = tg.get("alert_min_quality", app.alert_min_quality) - if quality not in _VALID_QUALITIES: - raise AppConfigError( - f'config.json: telegram.alert_min_quality "{quality}" ' - f"not in {_VALID_QUALITIES}") - app.alert_min_quality = quality - - dp = raw.get("data_providers") - if dp is not None: - if not isinstance(dp, dict): - raise AppConfigError('config.json: "data_providers" must be an object') - app.active_provider = str(dp.get("active", app.active_provider)) - app.provider_settings = {} - for name, block in dp.items(): - if name == "active": - continue - if not isinstance(block, dict): - raise AppConfigError( - f'config.json: data_providers.{name} must be an object') - app.provider_settings[name] = dict(block) - if app.active_provider not in app.provider_settings: - raise AppConfigError( - f'config.json: data_providers.active "{app.active_provider}" ' - "has no matching provider block") - - scanning = raw.get("scanning", {}) - _check_keys(scanning, _KNOWN_SCANNING, '"scanning"') - if "pairs" in scanning: - pairs = scanning["pairs"] - bad = [p for p in pairs if p not in PAIR_INFO] - if bad: - raise AppConfigError( - f"config.json: unknown pair(s) {bad} in scanning.pairs; " - f"known: {sorted(PAIR_INFO)}") - app.scan_pairs = list(pairs) - if "timeframes" in scanning: - names = scanning["timeframes"] - bad = [n for n in names if n not in _SCANNABLE_TFS] - if bad: - raise AppConfigError( - f"config.json: unsupported timeframe(s) {bad} in " - f"scanning.timeframes; supported: {list(_SCANNABLE_TFS)}") - app.scan_timeframes = [Timeframe[n] for n in names] - - pair_section = raw.get("pairs", {}) - for symbol, overrides in pair_section.items(): - if symbol not in PAIR_INFO: - raise AppConfigError( - f'config.json: unknown pair "{symbol}" in pairs; ' - f"known: {sorted(PAIR_INFO)}") - _check_keys(overrides, _KNOWN_PAIR_KEYS, f'pairs."{symbol}"') - if "blocked_hours_utc" in overrides: - hours = overrides["blocked_hours_utc"] - valid = isinstance(hours, list) and all( - isinstance(h, int) and not isinstance(h, bool) - and 0 <= h <= 23 for h in hours) - if not valid: - raise AppConfigError( - f'config.json: pairs."{symbol}".blocked_hours_utc must ' - "be a list of integers in 0-23") - app.pair_overrides = {s: dict(o) for s, o in pair_section.items()} - - level = raw.get("log_level", app.log_level) - if level not in _VALID_LOG_LEVELS: - raise AppConfigError( - f'config.json: log_level "{level}" not in {_VALID_LOG_LEVELS}') - app.log_level = level - return app - - -# ── Process-wide singleton ──────────────────────────────────────────── - -_cached: Optional[AppConfig] = None - - -def get_app_config() -> AppConfig: - global _cached - if _cached is None: - _cached = load_app_config() - return _cached - - -def set_app_config(app: Optional[AppConfig]) -> None: - """Override the process-wide AppConfig (tests/experiments). - Pass None to re-enable lazy loading from CONFIG_PATH.""" - global _cached - _cached = app - - -# ── Resolution helpers: env var → config.json → default ─────────────── - -def telegram_token(app: Optional[AppConfig] = None) -> str: - app = app or get_app_config() - return os.environ.get("TELEGRAM_TOKEN") or app.telegram_token - - -def telegram_chat_id(app: Optional[AppConfig] = None) -> str: - app = app or get_app_config() - return os.environ.get("TELEGRAM_CHAT_ID") or app.telegram_chat_id - - -def telegram_channel_id(app: Optional[AppConfig] = None) -> str: - """Env set-but-empty ("") means "disable the feed" and beats JSON, - so this checks None-ness, not truthiness.""" - app = app or get_app_config() - env = os.environ.get("TELEGRAM_CHANNEL_ID") - return env if env is not None else app.telegram_channel_id - - -def provider_api_key(app: Optional[AppConfig] = None) -> str: - """API key for the ACTIVE provider. The env override is per-provider - (TWELVE_DATA_API_KEY applies only when twelve_data is active).""" - app = app or get_app_config() - if app.active_provider == "twelve_data": - env = os.environ.get("TWELVE_DATA_API_KEY") - if env: - return env - block = app.provider_settings.get(app.active_provider, {}) - return str(block.get("api_key", "")) - - -# ── PlatformConfig overlay ──────────────────────────────────────────── - -def apply_pair_overrides( - cfg: PlatformConfig, app: Optional[AppConfig] = None -) -> PlatformConfig: - """Stamp deployment settings from config.json onto a PlatformConfig. - - Called by the pair factories BEFORE their validated per-timeframe - deltas, so e.g. the M5 hour-9 append lands on top of the JSON base. - The pair's base blocked_hours_utc is set on BOTH mirrored copies - (xau_strategy + backtest) so they cannot drift. - """ - app = app or get_app_config() - overrides = app.pair_overrides.get(cfg.symbol, {}) - if "blocked_hours_utc" in overrides: - hours = list(overrides["blocked_hours_utc"]) - cfg.xau_strategy.blocked_hours_utc = hours - cfg.backtest.blocked_hours_utc = list(hours) - cfg.scan_pairs = list(app.scan_pairs) - cfg.scan_timeframes = list(app.scan_timeframes) - return cfg -``` - -- [ ] **Step 5: Run tests, verify Module 13 passes and total is 190** - -```bash -python test_all.py 2>&1 | tail -25 -``` - -Expected: `RESULTS: 190 passed, 0 failed, 190 total` (184 baseline + 6 new). All six `app_config:` lines show `✓`. - -- [ ] **Step 6: Commit** - -```bash -git add app_config.py test_all.py -git commit -m "Add app_config.py: config.json loader, validation, env-precedence resolvers" -``` - ---- - -### Task 2: Factory overlay wiring + parity pins - -**Files:** -- Modify: `pair_signals/xauusd.py:81-90` (make_config), `test_all.py` (Module 9 equivalence test ~line 1851; Module 13 additions) - -**Interfaces:** -- Consumes: `app_config.apply_pair_overrides`, `app_config.set_app_config`, `app_config.AppConfig`, `app_config.load_app_config` (Task 1). -- Produces: `make_config` now applies the overlay — every consumer (bot, CLI, backtester) inherits it with no further change. Test symbol `_TEST_APP` (Task 1) is the restore target for any test that swaps the singleton. - -- [ ] **Step 1: Add failing tests (Module 13 additions)** - -Append to Module 13 in `test_all.py`: - -```python -@test("app_config: apply_pair_overrides stamps both blocked-hours mirrors") -def _(): - app = AppConfig(pair_overrides={XAUUSD: {"blocked_hours_utc": [3, 4]}}, - scan_pairs=[XAUUSD], - scan_timeframes=[Timeframe.M15]) - cfg = apply_pair_overrides(PlatformConfig(), app) - assert cfg.xau_strategy.blocked_hours_utc == [3, 4] - assert cfg.backtest.blocked_hours_utc == [3, 4] - assert cfg.scan_timeframes == [Timeframe.M15] - # the two mirrors must be independent lists, not aliases - cfg.xau_strategy.blocked_hours_utc.append(5) - assert cfg.backtest.blocked_hours_utc == [3, 4] - - -@test("app_config: M5 factory appends hour 9 on top of a JSON base") -def _(): - from pair_signals import PAIR_PRODUCERS - app = AppConfig(pair_overrides={XAUUSD: {"blocked_hours_utc": [3, 4]}}, - provider_settings={"twelve_data": {"api_key": "test-key"}}) - set_app_config(app) - try: - cfg = PAIR_PRODUCERS[XAUUSD](Timeframe.M5, None) - finally: - set_app_config(_TEST_APP) - assert cfg.xau_strategy.blocked_hours_utc == [3, 4, 9], \ - "validated M5 hour-9 delta must land on top of the JSON base" - assert cfg.backtest.blocked_hours_utc == [3, 4], \ - "mirror carries the base only (per-TF delta is XAU-strategy-side)" - - -@test("app_config: config.json.example is valid and behavior-neutral") -def _(): - import dataclasses - from pair_signals import PAIR_PRODUCERS - example = load_app_config( - os.path.join(os.path.dirname(__file__), "config.json.example")) - for tf in (Timeframe.M5, Timeframe.M15, Timeframe.H1, Timeframe.H4): - baseline = PAIR_PRODUCERS[XAUUSD](tf, None) # under _TEST_APP - set_app_config(example) - try: - with_example = PAIR_PRODUCERS[XAUUSD](tf, None) - finally: - set_app_config(_TEST_APP) - assert dataclasses.asdict(baseline) == dataclasses.asdict(with_example), \ - f"example config.json must be byte-identical to defaults on {tf}" -``` - -- [ ] **Step 2: Run to verify the new tests fail** - -```bash -python test_all.py 2>&1 | grep "app_config: M5 factory" -``` - -Expected: `✗ app_config: M5 factory appends hour 9 on top of a JSON base` (the factory doesn't apply the overlay yet, so blocked hours are the defaults `[0,1,2,10,11,13,16,21,9]`, not `[3,4,9]`). - -- [ ] **Step 3: Wire the overlay into the factory** - -In `pair_signals/xauusd.py`, change the import line and the top of `make_config`: - -```python -from typing import Optional, List -from config import PlatformConfig, Timeframe -from app_config import apply_pair_overrides -``` - -and in `make_config`, immediately after `config = PlatformConfig()`: - -```python - config = PlatformConfig() - # Deployment overlay (config.json) BEFORE the validated per-TF - # deltas below, so e.g. the M5 hour-9 append lands on the JSON base. - apply_pair_overrides(config) - config.primary_timeframe = primary_tf -``` - -- [ ] **Step 4: Update the Module 9 equivalence test** - -In `test_all.py` (~line 1851, `"xauusd.make_config equals inline PlatformConfig construction"`), the inline construction must apply the same overlay in the same order. After the `inline_cfg = PlatformConfig(...)` construction and BEFORE the per-TF override lines, add: - -```python - app_config.apply_pair_overrides(inline_cfg) -``` - -(`app_config` is already imported at the top of `test_all.py` from Task 1. Under `_TEST_APP` this is a no-op on blocked hours, but it keeps the equivalence definition exact: factory ≡ inline + overlay + per-TF deltas, whatever the active AppConfig.) - -- [ ] **Step 5: Run tests, verify 193 passing** - -```bash -python test_all.py 2>&1 | tail -25 -``` - -Expected: `RESULTS: 193 passed, 0 failed, 193 total`. Watch specifically for `✓ xauusd.make_config equals inline PlatformConfig construction` (Module 9) and the three new Module 13 lines. - -- [ ] **Step 6: Commit** - -```bash -git add pair_signals/xauusd.py test_all.py -git commit -m "Wire config.json overlay into the pair factory (before per-TF deltas)" -``` - ---- - -### Task 3: Data provider — key from app_config, registry, delete hardcoded key - -**Files:** -- Modify: `config.py:346` (delete field), `data_provider.py:72-76` (key resolution), `data_provider.py:265-269` (error message), `data_provider.py` end (registry), `signal_aggregator.py:107`, `backtest/gpu_backtest.py:1682`, `test_all.py` (Module 13 additions) - -**Interfaces:** -- Consumes: `app_config.provider_api_key`, `app_config.get_app_config`, `app_config.AppConfigError` (Task 1). -- Produces: `data_provider.PROVIDER_REGISTRY: Dict[str, type]` and `data_provider.make_data_provider(config: PlatformConfig) -> ForexDataProvider` — Task 4 uses `make_data_provider` in `telegram_bot.py`. - -- [ ] **Step 1: Add failing tests (Module 13 additions)** - -```python -@test("data_provider: api key resolves env → config.json (no code default)") -def _(): - from data_provider import ForexDataProvider - app = AppConfig(provider_settings={"twelve_data": {"api_key": "json-key"}}) - set_app_config(app) - try: - with patch.dict(os.environ, {}, clear=True): - dp = ForexDataProvider(PlatformConfig()) - assert dp._api_key == "json-key" - with patch.dict(os.environ, {"TWELVE_DATA_API_KEY": "env-key"}): - dp = ForexDataProvider(PlatformConfig()) - assert dp._api_key == "env-key" - set_app_config(AppConfig()) # no key anywhere - with patch.dict(os.environ, {}, clear=True): - dp = ForexDataProvider(PlatformConfig()) - assert dp._api_key is None and dp._td is None - finally: - set_app_config(_TEST_APP) - - -@test("data_provider: make_data_provider dispatches on active provider") -def _(): - from data_provider import ForexDataProvider, make_data_provider - assert isinstance(make_data_provider(PlatformConfig()), ForexDataProvider) - set_app_config(AppConfig(active_provider="polygon", - provider_settings={"polygon": {"api_key": "k"}})) - try: - make_data_provider(PlatformConfig()) - assert False, "unregistered provider must raise" - except AppConfigError as e: - assert "polygon" in str(e) - finally: - set_app_config(_TEST_APP) - - -@test("config.py: hardcoded twelve_data_api_key field is gone") -def _(): - assert not hasattr(PlatformConfig(), "twelve_data_api_key"), \ - "credentials must not live in committed source" -``` - -- [ ] **Step 2: Run to verify they fail** - -```bash -python test_all.py 2>&1 | grep -A1 "hardcoded twelve_data" -``` - -Expected: `✗ config.py: hardcoded twelve_data_api_key field is gone` (plus the make_data_provider import failure). - -- [ ] **Step 3: Implement** - -`config.py`: delete line 346 (`twelve_data_api_key: str = ""`). - -`data_provider.py` — add to the local imports: - -```python -from app_config import AppConfigError, get_app_config, provider_api_key -``` - -In `ForexDataProvider.__init__` replace the key resolution: - -```python - self.config = config - self._api_key = provider_api_key() or None -``` - -In `_fetch_twelve_data`, update the no-key error: - -```python - if self._td is None: - raise DataIntegrityError( - "No data-provider API key. Set the TWELVE_DATA_API_KEY env " - "var or data_providers.twelve_data.api_key in config.json." - ) -``` - -At the bottom of `data_provider.py` (after the class): - -```python -# ── Provider registry ───────────────────────────────────────────────── -# config.json data_providers.active selects from here. Adding a provider -# later = one adapter class + one entry + one config block. -PROVIDER_REGISTRY: Dict[str, type] = {"twelve_data": ForexDataProvider} - - -def make_data_provider(config: PlatformConfig) -> ForexDataProvider: - """Build the ACTIVE data provider (config.json: data_providers.active).""" - name = get_app_config().active_provider - cls = PROVIDER_REGISTRY.get(name) - if cls is None: - raise AppConfigError( - f'Unknown data provider "{name}"; ' - f"registered: {sorted(PROVIDER_REGISTRY)}") - return cls(config) -``` - -`signal_aggregator.py` line 18 + 107 — import and use the registry: - -```python -from data_provider import ForexDataProvider, make_data_provider -``` -```python - self.data_provider = make_data_provider(config) -``` - -`backtest/gpu_backtest.py` line 1682 (and its import of `ForexDataProvider`): - -```python - provider = make_data_provider(platform_cfg) -``` - -- [ ] **Step 4: Run tests, verify 196 passing** - -```bash -python test_all.py 2>&1 | tail -25 -``` - -Expected: `RESULTS: 196 passed, 0 failed, 196 total`. If any Module 2 test fails on a missing key, it means it constructed `ForexDataProvider` outside `_TEST_APP` scope — fix the test to run under `_TEST_APP` (which supplies the dummy key), never by re-adding a code default. - -- [ ] **Step 5: Commit** - -```bash -git add config.py data_provider.py signal_aggregator.py backtest/gpu_backtest.py test_all.py -git commit -m "Data provider: key from config.json/env, provider registry, drop hardcoded key" -``` - ---- - -### Task 4: telegram_bot wiring + startup guard - -**Files:** -- Modify: `telegram_bot.py` (credentials block ~93-105, logging ~88, SCAN_PAIRS ~113, alert quality ~478, WS key ~611-614, `_AUTOSCAN_LOOPS` ~797-802, scheduler log ~817, `_get_shared_dp` ~195, `main()` ~1261), `test_all.py` (Module 13 addition) - -**Interfaces:** -- Consumes: `app_config.get_app_config`, resolvers, `data_provider.make_data_provider` (Tasks 1+3). -- Produces: nothing consumed later; this is the final wiring task. - -- [ ] **Step 1: Add failing test (Module 13 addition)** - -```python -@test("telegram_bot: credentials, quality, pairs, loops from app_config") -def _(): - import importlib - import telegram_bot - app = AppConfig(telegram_token="json-tok", telegram_chat_id="42", - telegram_channel_id="", alert_min_quality="B", - scan_timeframes=[Timeframe.M15, Timeframe.H1], - provider_settings={"twelve_data": {"api_key": "k"}}) - set_app_config(app) - try: - with patch.dict(os.environ, {}, clear=True): - tb = importlib.reload(telegram_bot) - assert tb.TOKEN == "json-tok" - assert tb.OWNER_ID == 42 - assert tb.CHANNEL_ID is None, '"" channel id must disable the feed' - assert tb._alert_min_quality == "B" - assert [tf for tf, _ in tb._AUTOSCAN_LOOPS] == \ - [Timeframe.M15, Timeframe.H1] - assert tb._AUTOSCAN_LOOPS == [(Timeframe.M15, 15), - (Timeframe.H1, 60)] - finally: - set_app_config(_TEST_APP) - importlib.reload(telegram_bot) - - -@test("telegram_bot: main() refuses to start without credentials") -def _(): - import telegram_bot - if telegram_bot.TOKEN: # only meaningful in the hermetic no-cred env - return - try: - telegram_bot.main() - assert False, "main() must refuse to start without token/chat_id" - except SystemExit as e: - assert "credentials" in str(e) -``` - -- [ ] **Step 2: Run to verify failure** - -```bash -python test_all.py 2>&1 | grep "telegram_bot: credentials" -``` - -Expected: `✗ telegram_bot: credentials, quality, pairs, loops from app_config` (TOKEN still comes from the hardcoded fallback, not "json-tok"). - -- [ ] **Step 3: Implement the wiring** - -`telegram_bot.py` — add to the local imports (after the `config` import block): - -```python -from app_config import ( - get_app_config, telegram_token, telegram_chat_id, telegram_channel_id, -) -from data_provider import ForexDataProvider, make_data_provider -``` - -Immediately after the imports (before the logging block), add: - -```python -_APP = get_app_config() -``` - -Change the `logging.basicConfig` line (~88) to honor the configured level: - -```python -logging.basicConfig(level=getattr(logging, _APP.log_level, logging.INFO), - handlers=[_console_handler, _file_handler]) -``` - -Replace the credentials block (lines ~93-105) — keep the existing comments about the owner lock and subscriber channel, change only the assignments: - -```python -TOKEN = telegram_token() -CHAT_ID = telegram_chat_id() - -# ... (owner-lock comment unchanged) -OWNER_ID: Optional[int] = int(CHAT_ID) if CHAT_ID else None - -# ... (subscriber-channel comment unchanged) -_CHANNEL_ID_RAW = telegram_channel_id() -CHANNEL_ID: Optional[int] = int(_CHANNEL_ID_RAW) if _CHANNEL_ID_RAW else None -``` - -`SCAN_PAIRS` (~113): - -```python -SCAN_PAIRS = list(_APP.scan_pairs) -``` - -`_get_shared_dp` (~195): - -```python - _shared_dp = make_data_provider(PlatformConfig()) -``` - -Alert quality boot default (~478): - -```python -_alert_min_quality = _APP.alert_min_quality -``` - -WS key (~611-614) — `provider_api_key` needs adding to the `app_config` import list: - -```python -_cfg = PlatformConfig() -_WS_API_KEY = provider_api_key() -_WS_URL = f"wss://ws.twelvedata.com/v1/quotes/price?apikey={_WS_API_KEY}" -_WS_COOLDOWN = _cfg.ws_cooldown_seconds -``` - -`_AUTOSCAN_LOOPS` (~797-802) — derive from configured timeframes: - -```python -_TF_INTERVAL_MINUTES = { - Timeframe.M5: 5, Timeframe.M15: 15, Timeframe.M30: 30, - Timeframe.H1: 60, Timeframe.H4: 240, Timeframe.D1: 1440, -} -_AUTOSCAN_LOOPS = [ - (tf, _TF_INTERVAL_MINUTES[tf]) for tf in _APP.scan_timeframes -] -``` - -Scheduler log line (~817): - -```python - logger.info("Candle-boundary scheduler started (%s)", - "/".join(tf.name for tf, _ in _AUTOSCAN_LOOPS)) -``` - -(The manual `/scan` buttons — `_TF_BUTTON_LABELS` — deliberately stay hardcoded M5/M15/H1/H4: they are the owner's on-demand UI, not the automated feed. Documented decision, not an omission.) - -`main()` (~1261) — first lines of the function body: - -```python -def main() -> None: - if not TOKEN or not CHAT_ID: - raise SystemExit( - "telegram_bot: missing Telegram credentials. Set telegram.token " - "and telegram.chat_id in config.json, or the TELEGRAM_TOKEN and " - "TELEGRAM_CHAT_ID environment variables.") -``` - -- [ ] **Step 4: Run tests, verify 198 passing** - -```bash -python test_all.py 2>&1 | tail -25 -``` - -Expected: `RESULTS: 198 passed, 0 failed, 198 total`. If Module 7 tests break on `OWNER_ID` being `None`, they were relying on the hardcoded chat id — fix by having those tests set `telegram_bot.OWNER_ID`/`CHAT_ID` explicitly in their setup, never by re-adding fallbacks. - -- [ ] **Step 5: Commit** - -```bash -git add telegram_bot.py test_all.py -git commit -m "telegram_bot: credentials/quality/scan-loops from config.json, startup guard" -``` - ---- - -### Task 5: Migration, docs, final verification - -**Files:** -- Create: `config.json` (LOCAL ONLY — gitignored, never committed) -- Modify: `CLAUDE.md` (repo map, env-var table, config section, test count) - -**Interfaces:** -- Consumes: everything above. -- Produces: the deployed machine's real `config.json`; updated authoritative docs. - -- [ ] **Step 1: Create the local config.json with the current live values** - -These are the exact values being deleted from source (from pre-change `telegram_bot.py:93-105` and `config.py:346`), so the running bot behaves identically: - -```bash -cat > /home/harwinder/Development/ForexBot2/config.json <<'EOF' -{ - "telegram": { - "token": "", - "chat_id": "", - "channel_id": "", - "alert_min_quality": "C" - }, - "data_providers": { - "active": "twelve_data", - "twelve_data": { "api_key": "" } - }, - "scanning": { - "pairs": ["XAU/USD"], - "timeframes": ["M5", "M15", "H1", "H4"] - }, - "pairs": { - "XAU/USD": { "blocked_hours_utc": [0, 1, 2, 10, 11, 13, 16, 21] } - }, - "log_level": "INFO" -} -EOF -git status --short # config.json must NOT appear (gitignored) -``` - -(Real values were scrubbed from this doc 2026-07-21 — take them from the previously deployed telegram_bot.py/config.py in git history, and rotate both credentials after merge.) - -- [ ] **Step 2: Boot smoke test (no network needed)** - -```bash -python - <<'EOF' -from app_config import get_app_config, telegram_token, provider_api_key, apply_pair_overrides -from pair_signals import PAIR_PRODUCERS -from config import Timeframe, XAUUSD -app = get_app_config() -assert telegram_token(app), "token missing" -assert provider_api_key(app), "api key missing" -cfg = PAIR_PRODUCERS[XAUUSD](Timeframe.M5, None) -assert cfg.xau_strategy.blocked_hours_utc == [0, 1, 2, 10, 11, 13, 16, 21, 9] -print("boot smoke OK — live config.json loads and factory output unchanged") -EOF -``` - -Expected: `boot smoke OK — live config.json loads and factory output unchanged`. - -- [ ] **Step 3: Update CLAUDE.md** - -- Repository-map table: add rows for `app_config.py` ("config.json loader/validation, env-precedence resolvers, `apply_pair_overrides` deployment overlay") and `config.json.example` ("committed schema template; real gitignored config.json holds secrets + deployment settings"). -- Environment-variables table: note precedence "env var → config.json → none"; state that hardcoded fallbacks were REMOVED 2026-07-21 and the bot refuses to boot without credentials. -- `PlatformConfig` section: delete the `twelve_data_api_key` mention ("hardcoded fallback"); add one line: "Deployment settings (credentials, scan pairs/TFs, per-pair base blocked hours, alert quality, log level) overlay from config.json via `app_config.apply_pair_overrides`, applied in the pair factory BEFORE per-TF deltas." -- `pair_signals` section: mention the overlay call order in `make_config`. -- Test-suite line: update "184 tests in 12 modules" → "198 tests in 13 modules" (verify against the actual final count from Step 4 before writing). - -- [ ] **Step 4: Full-suite final verification** - -```bash -python test_all.py 2>&1 | tail -15 -``` - -Expected: `RESULTS: 198 passed, 0 failed, 198 total` (use the real number if it differs, and make CLAUDE.md match). - -- [ ] **Step 5: Commit docs** - -```bash -git add CLAUDE.md -git commit -m "Docs: config.json deployment overlay (CLAUDE.md repo map, env table, test count)" -``` - -- [ ] **Step 6: Merge/PR decision** - -Implementation complete → follow superpowers:finishing-a-development-branch (options: merge `centralized-config` to main, or push the branch to `origin`/`gitea` for review). After merge, remind the user: **rotate the Telegram token and Twelve Data key** — the old values remain in git history on both remotes. diff --git a/oanda_provider.py b/oanda_provider.py deleted file mode 100644 index d840019..0000000 --- a/oanda_provider.py +++ /dev/null @@ -1,9 +0,0 @@ -""" -ForexBot2 — Oanda provider (back-compat shim) -============================================== -The implementation lives in `providers/oanda.py`; shared behavior is in -`providers/base.py`. This module keeps the historical import path working. -""" - -from providers.base import _is_forex_weekend # noqa: F401 -from providers.oanda import OandaDataError, OandaDataProvider # noqa: F401 diff --git a/test_all.py b/test_all.py index 72d8ac2..37deef3 100644 --- a/test_all.py +++ b/test_all.py @@ -214,29 +214,29 @@ def _(): # ══════════════════════════════════════════════════════════════════════ -# MODULE 2: data_provider.py +# MODULE 2: providers/twelve_data.py # ══════════════════════════════════════════════════════════════════════ -print("\n═══ Module 2: data_provider.py ═══") +print("\n═══ Module 2: providers/twelve_data.py ═══") -from data_provider import ForexDataProvider, DataIntegrityError +from providers import TwelveDataProvider, DataIntegrityError -@test("ForexDataProvider._to_td_symbol conversion") +@test("TwelveDataProvider._to_td_symbol conversion") def _(): - assert ForexDataProvider._to_td_symbol("XAU/USD") == "XAU/USD" + assert TwelveDataProvider._to_td_symbol("XAU/USD") == "XAU/USD" -@test("ForexDataProvider TD_INTERVALS complete mapping") +@test("TwelveDataProvider TD_INTERVALS complete mapping") def _(): - dp = ForexDataProvider(PlatformConfig()) + dp = TwelveDataProvider(PlatformConfig()) for tf in Timeframe: assert tf in dp.TD_INTERVALS, f"Missing TD_INTERVALS mapping for {tf}" -@test("ForexDataProvider BARS_PER_DAY mapping") +@test("TwelveDataProvider BARS_PER_DAY mapping") def _(): - dp = ForexDataProvider(PlatformConfig()) + dp = TwelveDataProvider(PlatformConfig()) for tf in Timeframe: assert tf in dp.BARS_PER_DAY, f"Missing BARS_PER_DAY for {tf}" assert dp.BARS_PER_DAY[Timeframe.H1] == 24 @@ -244,9 +244,9 @@ def _(): assert dp.BARS_PER_DAY[Timeframe.D1] == 1 -@test("ForexDataProvider CANDLE_SECONDS mapping") +@test("TwelveDataProvider CANDLE_SECONDS mapping") def _(): - dp = ForexDataProvider(PlatformConfig()) + dp = TwelveDataProvider(PlatformConfig()) for tf in Timeframe: assert tf in dp.CANDLE_SECONDS, f"Missing CANDLE_SECONDS for {tf}" assert dp.CANDLE_SECONDS[Timeframe.M1] == 60 @@ -254,9 +254,9 @@ def _(): assert dp.CANDLE_SECONDS[Timeframe.D1] == 86400 -@test("ForexDataProvider._validate_and_clean basic valid data") +@test("TwelveDataProvider._validate_and_clean basic valid data") def _(): - dp = ForexDataProvider(PlatformConfig()) + dp = TwelveDataProvider(PlatformConfig()) df = make_ohlcv(300) result = dp._validate_and_clean(df, "XAU/USD", Timeframe.H1) assert len(result) > 0 @@ -264,9 +264,9 @@ def _(): assert not result.index.has_duplicates -@test("ForexDataProvider._validate_and_clean empty df raises error") +@test("TwelveDataProvider._validate_and_clean empty df raises error") def _(): - dp = ForexDataProvider(PlatformConfig()) + dp = TwelveDataProvider(PlatformConfig()) try: dp._validate_and_clean(pd.DataFrame(), "XAU/USD", Timeframe.H1) assert False, "Should have raised DataIntegrityError" @@ -274,9 +274,9 @@ def _(): pass -@test("ForexDataProvider._validate_and_clean None df raises error") +@test("TwelveDataProvider._validate_and_clean None df raises error") def _(): - dp = ForexDataProvider(PlatformConfig()) + dp = TwelveDataProvider(PlatformConfig()) try: dp._validate_and_clean(None, "XAU/USD", Timeframe.H1) assert False, "Should have raised DataIntegrityError" @@ -284,18 +284,18 @@ def _(): pass -@test("ForexDataProvider._validate_and_clean fixes OHLC inconsistencies") +@test("TwelveDataProvider._validate_and_clean fixes OHLC inconsistencies") def _(): - dp = ForexDataProvider(PlatformConfig()) + dp = TwelveDataProvider(PlatformConfig()) df = make_ohlcv(300) df.iloc[10, df.columns.get_loc("High")] = df.iloc[10]["Low"] - 1 result = dp._validate_and_clean(df, "XAU/USD", Timeframe.H1) assert (result["High"] >= result["Low"]).all() -@test("ForexDataProvider._validate_and_clean drops future timestamps") +@test("TwelveDataProvider._validate_and_clean drops future timestamps") def _(): - dp = ForexDataProvider(PlatformConfig()) + dp = TwelveDataProvider(PlatformConfig()) df = make_ohlcv(300) future = pd.Timestamp.now(tz="UTC") + pd.Timedelta(hours=10) df.loc[future] = df.iloc[-1] @@ -303,9 +303,9 @@ def _(): assert (result.index <= pd.Timestamp.utcnow() + pd.Timedelta(minutes=5)).all() -@test("ForexDataProvider._validate_and_clean removes duplicates") +@test("TwelveDataProvider._validate_and_clean removes duplicates") def _(): - dp = ForexDataProvider(PlatformConfig()) + dp = TwelveDataProvider(PlatformConfig()) df = make_ohlcv(300) dup = pd.concat([df, df.iloc[-5:]]) assert dup.index.has_duplicates, "Setup: should have duplicates" @@ -313,9 +313,9 @@ def _(): assert not result.index.has_duplicates, "Should remove duplicate indices" -@test("ForexDataProvider._validate_and_clean forward fills NaN (limit 3)") +@test("TwelveDataProvider._validate_and_clean forward fills NaN (limit 3)") def _(): - dp = ForexDataProvider(PlatformConfig()) + dp = TwelveDataProvider(PlatformConfig()) df = make_ohlcv(300) df.iloc[100, df.columns.get_loc("Close")] = np.nan df.iloc[101, df.columns.get_loc("Close")] = np.nan @@ -323,9 +323,9 @@ def _(): assert not result["Close"].isna().any() -@test("ForexDataProvider._is_data_fresh weekend override") +@test("TwelveDataProvider._is_data_fresh weekend override") def _(): - dp = ForexDataProvider(PlatformConfig()) + dp = TwelveDataProvider(PlatformConfig()) # Create data that ends on a Friday, call _is_data_fresh on a Saturday friday = pd.Timestamp("2025-05-09 16:00:00", tz="UTC") old_dates = pd.date_range(end=friday, periods=300, freq="h") @@ -339,16 +339,16 @@ def _(): assert result is True, "Weekend data should always be considered fresh" -@test("ForexDataProvider._is_data_fresh returns False for empty df") +@test("TwelveDataProvider._is_data_fresh returns False for empty df") def _(): - dp = ForexDataProvider(PlatformConfig()) + dp = TwelveDataProvider(PlatformConfig()) assert dp._is_data_fresh(pd.DataFrame(), "XAU/USD", Timeframe.H1) is False assert dp._is_data_fresh(None, "XAU/USD", Timeframe.H1) is False -@test("ForexDataProvider.clear_cache clears all entries") +@test("TwelveDataProvider.clear_cache clears all entries") def _(): - dp = ForexDataProvider(PlatformConfig()) + dp = TwelveDataProvider(PlatformConfig()) dp._cache["test_key"] = make_ohlcv(10) dp._cache_timestamps["test_key"] = time.time() count = dp.clear_cache() @@ -357,9 +357,9 @@ def _(): assert len(dp._cache_timestamps) == 0 -@test("ForexDataProvider.get_pair_info returns correct info") +@test("TwelveDataProvider.get_pair_info returns correct info") def _(): - dp = ForexDataProvider(PlatformConfig()) + dp = TwelveDataProvider(PlatformConfig()) info = dp.get_pair_info(XAUUSD) assert info["symbol"] == XAUUSD assert info["display"] == "XAU/USD" @@ -369,18 +369,18 @@ def _(): assert info_unknown["display"] == "XAU/USD" -@test("ForexDataProvider._throttle rate limiting logic") +@test("TwelveDataProvider._throttle rate limiting logic") def _(): - dp = ForexDataProvider(PlatformConfig()) + dp = TwelveDataProvider(PlatformConfig()) dp._request_times.clear() for _ in range(10): dp._throttle(credits=1) assert len(dp._request_times) == 10 -@test("ForexDataProvider caching - in-memory cache returns same data") +@test("TwelveDataProvider caching - in-memory cache returns same data") def _(): - dp = ForexDataProvider(PlatformConfig()) + dp = TwelveDataProvider(PlatformConfig()) df = make_ohlcv(300) dp._cache["XAU/USD_1h"] = df dp._cache_timestamps["XAU/USD_1h"] = time.time() @@ -2267,7 +2267,7 @@ def _(): @test("Outlier filter drops the spike bar but keeps the snap-back bar") def _(): - provider = ForexDataProvider(PlatformConfig()) + provider = TwelveDataProvider(PlatformConfig()) df = make_ohlcv(n=300, seed=3) k = 150 spike_ts, after_ts = df.index[k], df.index[k + 1] @@ -2288,7 +2288,7 @@ def _(): "Close": v, "Volume": 1000.0}, index=idx) for _attempt in range(2): - provider = ForexDataProvider(PlatformConfig()) + provider = TwelveDataProvider(PlatformConfig()) started = pd.Timestamp.now(tz="UTC").floor("15min") last_closed = started - pd.Timedelta(minutes=15) stale = bars_ending(last_closed - pd.Timedelta(minutes=15)) @@ -2323,13 +2323,13 @@ def _(): "Close": v, "Volume": 1000.0}, index=idx) for _attempt in range(2): - provider = ForexDataProvider(PlatformConfig()) + provider = TwelveDataProvider(PlatformConfig()) started = pd.Timestamp.now(tz="UTC").floor("15min") last_closed = started - pd.Timedelta(minutes=15) lagging = bars_ending(last_closed - pd.Timedelta(minutes=15)) complete = bars_ending(last_closed) with patch("providers.base._is_forex_weekend", return_value=False), \ - patch.object(ForexDataProvider, "FRESH_FETCH_RETRY_SECONDS", 0.0): + patch.object(TwelveDataProvider, "FRESH_FETCH_RETRY_SECONDS", 0.0): # (a) lagging first response → one retry returns the closed bar. with patch.object(provider, "_fetch_twelve_data", side_effect=[lagging.copy(), complete.copy()]) as mf: @@ -2966,30 +2966,30 @@ def _(): f"example config.json must be byte-identical to defaults on {tf}" -@test("data_provider: api key resolves env → config.json (no code default)") +@test("providers: api key resolves env → config.json (no code default)") def _(): - from data_provider import ForexDataProvider + from providers import TwelveDataProvider app = AppConfig(provider_settings={"twelve_data": {"api_key": "json-key"}}) set_app_config(app) try: with patch.dict(os.environ, {}, clear=True): - dp = ForexDataProvider(PlatformConfig()) + dp = TwelveDataProvider(PlatformConfig()) assert dp._api_key == "json-key" with patch.dict(os.environ, {"TWELVE_DATA_API_KEY": "env-key"}): - dp = ForexDataProvider(PlatformConfig()) + dp = TwelveDataProvider(PlatformConfig()) assert dp._api_key == "env-key" set_app_config(AppConfig()) # no key anywhere with patch.dict(os.environ, {}, clear=True): - dp = ForexDataProvider(PlatformConfig()) + dp = TwelveDataProvider(PlatformConfig()) assert dp._api_key is None and dp._td is None finally: set_app_config(_TEST_APP) -@test("data_provider: make_data_provider dispatches on active provider") +@test("providers: make_data_provider dispatches on active provider") def _(): - from data_provider import ForexDataProvider, make_data_provider - assert isinstance(make_data_provider(PlatformConfig()), ForexDataProvider) + from providers import TwelveDataProvider, make_data_provider + assert isinstance(make_data_provider(PlatformConfig()), TwelveDataProvider) set_app_config(AppConfig(active_provider="polygon", provider_settings={"polygon": {"api_key": "k"}})) try: @@ -3132,30 +3132,12 @@ def _(): print("\n═══ Module 15: Oanda provider / executor / daemon ═══") -import oanda_provider +import providers.oanda import oanda_executor import oanda_daemon -@test("oanda_provider: _is_forex_weekend matches data_provider's") -def _(): - from data_provider import _is_forex_weekend as dp_weekend - from oanda_provider import _is_forex_weekend as oa_weekend - import pandas as pd - for ts_str in [ - "2026-07-17 12:00", # Friday noon - "2026-07-17 22:30", # Friday after close - "2026-07-18 15:00", # Saturday - "2026-07-19 12:00", # Sunday before open - "2026-07-19 22:30", # Sunday after open - "2026-07-20 08:00", # Monday - ]: - ts = pd.Timestamp(ts_str, tz="UTC") - assert dp_weekend(ts) == oa_weekend(ts), \ - f"Weekend check mismatch at {ts_str}" - - -@test("oanda_provider: OandaDataProvider parses Oanda candle JSON into correct DataFrame") +@test("providers.oanda: OandaDataProvider parses Oanda candle JSON into correct DataFrame") def _(): sample_candles = [ { @@ -3190,8 +3172,8 @@ def _(): "raise_for_status": lambda self: None, })() - provider2 = oanda_provider.OandaDataProvider.__new__( - oanda_provider.OandaDataProvider) + provider2 = providers.oanda.OandaDataProvider.__new__( + providers.oanda.OandaDataProvider) provider2.config = PlatformConfig() provider2._base_url = "https://test" provider2._cache = {} @@ -3219,25 +3201,25 @@ def _(): assert abs(spread - 0.30) < 0.01, f"Expected spread ~0.30, got {spread}" -@test("oanda_provider: instrument name mapping") +@test("providers.oanda: instrument name mapping") def _(): - provider = oanda_provider.OandaDataProvider.__new__( - oanda_provider.OandaDataProvider) + provider = providers.oanda.OandaDataProvider.__new__( + providers.oanda.OandaDataProvider) assert provider._to_oanda_instrument("XAU/USD") == "XAU_USD" assert provider._to_oanda_instrument("EUR/USD") == "EUR_USD" -@test("oanda_provider: granularity mapping covers all scan timeframes") +@test("providers.oanda: granularity mapping covers all scan timeframes") def _(): for tf in [Timeframe.M5, Timeframe.M15, Timeframe.H1, Timeframe.H4]: - assert tf in oanda_provider.OandaDataProvider.OANDA_GRANULARITY, \ + assert tf in providers.oanda.OandaDataProvider.OANDA_GRANULARITY, \ f"{tf} missing from OANDA_GRANULARITY" -@test("oanda_provider: get_spread returns default when no cache") +@test("providers.oanda: get_spread returns default when no cache") def _(): - provider = oanda_provider.OandaDataProvider.__new__( - oanda_provider.OandaDataProvider) + provider = providers.oanda.OandaDataProvider.__new__( + providers.oanda.OandaDataProvider) provider._spread_cache = {} spread = provider.get_spread("XAU/USD") assert spread == 0.30, f"Expected default spread 0.30, got {spread}" @@ -3820,15 +3802,15 @@ def _(): assert "oanda.com" in url, f"Expected Oanda URL, got {url}" -@test("oanda_provider: HTTP error surfaces status code and body") +@test("providers.oanda: HTTP error surfaces status code and body") def _(): mock_resp = type("MockResp", (), { "status_code": 401, "text": '{"errorMessage":"Insufficient authorization to perform request"}', })() - provider = oanda_provider.OandaDataProvider.__new__( - oanda_provider.OandaDataProvider) + provider = providers.oanda.OandaDataProvider.__new__( + providers.oanda.OandaDataProvider) provider._base_url = "https://test" provider._spread_cache = {} provider._session = type("S", (), { @@ -3839,17 +3821,17 @@ def _(): try: provider._fetch_oanda_candles("XAU/USD", Timeframe.H1, 300) assert False, "Should have raised OandaDataError" - except oanda_provider.OandaDataError as e: + except providers.oanda.OandaDataError as e: msg = str(e) assert "401" in msg, f"Status code missing from error: {msg}" assert "Insufficient authorization" in msg, \ f"Body detail missing from error: {msg}" -@test("oanda_provider: boundary-retry failure keeps original data") +@test("providers.oanda: boundary-retry failure keeps original data") def _(): - provider = oanda_provider.OandaDataProvider.__new__( - oanda_provider.OandaDataProvider) + provider = providers.oanda.OandaDataProvider.__new__( + providers.oanda.OandaDataProvider) provider.config = PlatformConfig() provider._cache = {} provider._cache_timestamps = {} @@ -3868,7 +3850,7 @@ def _(): call_count["n"] += 1 if call_count["n"] == 1: return original_df - raise oanda_provider.OandaDataError("boom") + raise providers.oanda.OandaDataError("boom") def _validate_and_clean(df, symbol, timeframe): return df @@ -4029,19 +4011,6 @@ def _(): assert TiingoDataProvider.CANDLE_SECONDS is BaseDataProvider.CANDLE_SECONDS -@test("providers: back-compat shims re-export the real classes") -def _(): - import data_provider as dp_shim - import oanda_provider as oa_shim - assert dp_shim.ForexDataProvider is TwelveDataProvider - assert dp_shim.make_data_provider is make_data_provider - assert oa_shim.OandaDataProvider is _POanda - from providers.base import DataIntegrityError as base_die - assert dp_shim.DataIntegrityError is base_die - # Provider-specific errors are catchable as the shared type. - assert issubclass(oa_shim.OandaDataError, base_die) - - @test("providers: fetch_range paginates, dedupes, and trims to the window") def _(): class FakeProvider(BaseDataProvider): @@ -4288,7 +4257,7 @@ def _(): # must never change the session clock. import signal_aggregator as sa assert sa.BaseDataProvider.CANDLE_SECONDS[Timeframe.M15] == 900 - assert not hasattr(sa, "ForexDataProvider"), \ + assert not hasattr(sa, "TwelveDataProvider"), \ "aggregator must not depend on a concrete provider class"