diff --git a/CLAUDE.md b/CLAUDE.md index 3fae88b..e22bac3 100644 --- a/CLAUDE.md +++ b/CLAUDE.md @@ -4,18 +4,17 @@ This file provides guidance to Claude Code (claude.ai/code) when working with co ## What this is -Signal-intelligence platform for **XAU/USD (gold — the only supported pair)**. It fetches OHLCV from the active data provider (**Oanda v20 since 2026-07-28**; Twelve Data and Tiingo remain registered fallbacks), scores 9 technical indicators per timeframe with regime-adaptive weights, checks multi-timeframe agreement and news risk, grades the signal (A+…F), and generates a full trade setup (entry, SL, TP1/TP2, trailing plan). Delivery: Telegram bot, CLI, and an Oanda practice-account auto-execution daemon. A realism-focused backtester replays history through the *exact same* signal code. +Signal-intelligence platform for **XAU/USD (gold — the only supported pair)**. It fetches OHLCV from the active data provider (**Oanda v20 since 2026-07-28**; Twelve Data and Tiingo remain registered fallbacks), scores 9 technical indicators per timeframe with regime-adaptive weights, checks multi-timeframe agreement and news risk, grades the signal (A+…F), and generates a full trade setup (entry, SL, TP1/TP2, trailing plan). Delivery: Telegram bot and CLI. A realism-focused backtester replays history through the *exact same* signal code. Every numeric threshold in `config.py` and `pair_signals/xauusd.py` was backtested or is load-bearing for a backtested result. Validation evidence lives in the factory/module docstrings and in `~/.claude/projects/.../memory/`. ## Commands ```bash -python test_all.py # QA suite: 430 tests, 27 modules, no network (mocked) +python test_all.py # QA suite: 402 tests, 27 modules, no network (mocked) python main.py --pair XAUUSD --timeframe H1 --detail # CLI signal report (--json for full dump, # --panel to run the advisory LLM agent panel on-demand) python telegram_bot.py # the bot (refuses to boot without credentials) -python oanda_daemon.py # auto-execution daemon on Oanda practice account python dashboard_publisher.py --once # web dashboard: one scan+publish cycle to $DASHBOARD_OUT_DIR # (omit --once for the candle-close daemon; deploy/README-dashboard.md) @@ -48,7 +47,7 @@ providers/.fetch_ohlcv() per TF: single-flight fetch, drop forming → signal_aggregator.analyze_pair() live entry point (weekend/day filters, fetch, staleness) └─ analyze_from_dataframes() replayable core: MTF agreement → news → bonuses/penalties → final signal + quality grade → gates → trade setup → reasons - → SignalReport consumed by telegram_bot.py, main.py, oanda_daemon.py, backtester + → SignalReport consumed by telegram_bot.py, main.py, backtester ``` Signal currency everywhere is `SignalStrength` (−2..+2). DataFrames are UTC tz-aware, oldest-first, capitalized OHLCV columns. Everything runs on **closed bars only** — the forming candle is dropped, and session/hour filters evaluate **bar-CLOSE time** (bar label + candle duration) identically in live and backtest. @@ -59,13 +58,12 @@ Signal currency everywhere is `SignalStrength` (−2..+2). DataFrames are UTC tz - **`engine.py`** — `TechnicalEngine`: 9 scored indicators (ema_trend, rsi, macd, bollinger, adx, roc, keltner, divergence, candlestick) + ATR as context. Regime classification (trending_up/down, `trending` = mixed/direction-UNKNOWN, transitioning, ranging) drives per-indicator weight multipliers and regime-aware RSI reinterpretation. Compute order matters (ATR first; MACD always computed even when not a pillar — divergence needs the histogram). Private series ride on the result as `_underscore` attrs; scorers/formatters skip anything `_`-prefixed or weight-0. - **`signal_aggregator.py`** — MTF agreement, diminishing-decay penalty system, exhaustion hard block, quality grading, then an ordered chain of post-grade gates (session, XAU SELL quality/vol/hour, regime blocks, M5 noise gates), then three setup generators: XAU ATR (A+/A), generic swing-based, plus the M5 pullback (`XAU_PB`) and chart-pattern (`XAU_CP`, `chart_patterns.py`) entry paths. The PB/CP generators run AFTER the gate chain and force quality to "B"; the CP path deliberately accepts grade-F composites and re-checks only session/day/big-candle (via warning prefixes) plus its own SELL/vol/RSI gates — it does NOT re-check exhaustion or the M5 min-score/MTF gates. **This is validated behavior, not a bug** (2026-08-04 audit: 226/586 trades in the shipped M5 12mo book are XAU_CP, +38.9R, 104 of them exhaustion-flagged and net-positive) — changing it is a rule-1 revalidation event. `TradeSetup.strategy` selects trailing params; trail params are stamped on the setup from the *producing* config so per-TF overrides reach formatters. - **`pair_signals/`** — per-pair config factories. **Always use the factory (`produce_signal` / `make_config`), never bare `PlatformConfig()`** — the factory applies the config.json overlay first, then the validated per-TF deltas (M5 hour-9 block, M15 trail arm 0.70, M15 SELL-hour block, M5 SELL-in-transitioning block, H1 all-shorts block (2026-09-03), etc.; evidence in the docstring). A test enforces factory ≡ inline-config equivalence. Confirmation map: M5→[M15,H1], M15→[M5,H1], H1→[M15,H4], H4→[H1,D1]. -- **`signal_log.py` / `advisory_outcome.py`** — SQLite signal-history logger (stdlib only). Hooked into `SignalAggregator.analyze_pair` only (the live entry point — the backtester never writes here, preserving parity), so every live scan from every surface (Telegram `telegram`, CLI `cli`, Oanda daemon `oanda_daemon`, dashboard `dashboard`) lands one row in `logs/signals.db`: signal/grade/price, MTF + trade-setup + reasons JSON, full per-TF indicator snapshots, the last 14 closed OHLCV candles per fetched TF, and the news at scan time (event risk + calendar events + scored headlines, re-fetched via the NewsEngine class-level caches). Config.json `signal_log` block (`enabled`, `path`, `log_blocked_scans`; env `SIGNAL_LOG_ENABLED`/`SIGNAL_LOG_PATH`/`SIGNAL_LOG_BLOCKED_SCANS`). **Fail-safe by contract** — a logging failure can never break a scan. Panel fields arrive post-hoc: `log_signal` stashes the rowid and `agent_panel.attach_panel` writes verdict + creation/cache provenance + payload hash onto the same row. **Advisory outcomes (2026-09):** EVERY row with a trade setup is seeded at log time (not only panel rows); same-setup re-logs within 60 min from any surface (`setup_key` = tf|direction|entry|stop|strategy) are `DUPLICATE` rows pointing at the canonical row via `advisory_event_id`, so sum `advisory_r_multiple` over `RESOLVED` rows only. The versioned model advances from later stored closed-candle snapshots (next-open reanchoring, intrabar peak with close-only ratchet, SL-first, scheduled weekend/21:00-UTC closures allowed, 300-bar timeout, $0.30/$0.65 views); when a snapshot no longer overlaps the outcome's last bar it rebuilds the stream from every stored snapshot of that TF before giving up. It is a cleaned-mid evidence model, not execution P&L. **Post-gate truth columns:** `gate_reason` (the warning that removed the setup), `actionable` (grade ≥ C AND setup — the `action` text is written BEFORE the gate chain and can say "STRONG SELL — High-quality setup" on a D row), `cleaned_warnings_json` (what the XAU_CP acceptance stripped), `bar_range_median` (= the misnamed `spread` column: median High-Low, NOT a broker spread), `scan_kind` (`blocked` weekend/day early returns carry no news/candles), `config_fingerprint` (gate config hash — surfaces config drift between processes). **News is content-addressed:** `headlines_json`/`events_json` are interned into `news_snapshots` (they were 73% of a 667 MB DB); read through the `signals_with_news` view or `signal_log.read_news`. Readers must open with `mode=ro`, never `immutable=1` (blind to WAL rows). Tools: `analyze_signal_log.py` (`--stored-candles` DB-only replay with event/opportunity/one-position books + persona scorecard, `--parity` telegram-vs-dashboard divergence report), `scripts/backfill_signal_outcomes.py` (seed/dedup/advance legacy rows), `scripts/migrate_signal_log_news.py` (intern legacy blobs; `--vacuum` with both writers stopped). +- **`signal_log.py` / `advisory_outcome.py`** — SQLite signal-history logger (stdlib only). Hooked into `SignalAggregator.analyze_pair` only (the live entry point — the backtester never writes here, preserving parity), so every live scan from every surface (Telegram `telegram`, CLI `cli`, dashboard `dashboard`) lands one row in `logs/signals.db`: signal/grade/price, MTF + trade-setup + reasons JSON, full per-TF indicator snapshots, the last 14 closed OHLCV candles per fetched TF, and the news at scan time (event risk + calendar events + scored headlines, re-fetched via the NewsEngine class-level caches). Config.json `signal_log` block (`enabled`, `path`, `log_blocked_scans`; env `SIGNAL_LOG_ENABLED`/`SIGNAL_LOG_PATH`/`SIGNAL_LOG_BLOCKED_SCANS`). **Fail-safe by contract** — a logging failure can never break a scan. Panel fields arrive post-hoc: `log_signal` stashes the rowid and `agent_panel.attach_panel` writes verdict + creation/cache provenance + payload hash onto the same row. **Advisory outcomes (2026-09):** EVERY row with a trade setup is seeded at log time (not only panel rows); same-setup re-logs within 60 min from any surface (`setup_key` = tf|direction|entry|stop|strategy) are `DUPLICATE` rows pointing at the canonical row via `advisory_event_id`, so sum `advisory_r_multiple` over `RESOLVED` rows only. The versioned model advances from later stored closed-candle snapshots (next-open reanchoring, intrabar peak with close-only ratchet, SL-first, scheduled weekend/21:00-UTC closures allowed, 300-bar timeout, $0.30/$0.65 views); when a snapshot no longer overlaps the outcome's last bar it rebuilds the stream from every stored snapshot of that TF before giving up. It is a cleaned-mid evidence model, not execution P&L. **Post-gate truth columns:** `gate_reason` (the warning that removed the setup), `actionable` (grade ≥ C AND setup — the `action` text is written BEFORE the gate chain and can say "STRONG SELL — High-quality setup" on a D row), `cleaned_warnings_json` (what the XAU_CP acceptance stripped), `bar_range_median` (= the misnamed `spread` column: median High-Low, NOT a broker spread), `scan_kind` (`blocked` weekend/day early returns carry no news/candles), `config_fingerprint` (gate config hash — surfaces config drift between processes). **News is content-addressed:** `headlines_json`/`events_json` are interned into `news_snapshots` (they were 73% of a 667 MB DB); read through the `signals_with_news` view or `signal_log.read_news`. Readers must open with `mode=ro`, never `immutable=1` (blind to WAL rows). Tools: `analyze_signal_log.py` (`--stored-candles` DB-only replay with event/opportunity/one-position books + persona scorecard, `--parity` telegram-vs-dashboard divergence report), `scripts/backfill_signal_outcomes.py` (seed/dedup/advance legacy rows), `scripts/migrate_signal_log_news.py` (intern legacy blobs; `--vacuum` with both writers stopped). - **`news_engine.py`** — Forex Factory calendar (MyFXBook RSS fallback) + RSS keyword sentiment with gold-inverse keywords. Class-level caches (fresh engine per scan). Optional DeepSeek enrichment via `llm_client.py` (requests-only, no SDK): batched headline sentiment overriding keyword scores, gold-direction on release surprises, and a market-summary blurb (`SignalReport.news_summary`) — **display/warning-only, keyword fallback on any failure, inert without a key**; prompts are static module constants (DeepSeek prefix-cache pricing) and requests send `thinking: disabled` (V4-Flash reasoning otherwise blows the timeout). By XAU-tuned default, event risk and sentiment **warn but do not gate signals** (`event_risk_caps_signals=False`, `sentiment_affects_signals=False`) — the LLM does not change that. - **`agent_panel.py`** — ai-hedge-fund-style advisory panel: 4 DeepSeek persona calls (macro/technical/risk/contrarian, concurrent on a panel-private pool) + 1 synthesizer producing an **ADVISORY-ONLY** consensus (`AGREE/NEUTRAL/OPPOSE` + confidence + rationale) written onto `SignalReport.panel_*` fields. Attached **post-hoc** by consumers (`telegram_bot` alert path, `main --panel`, `dashboard_publisher`) — never runs inside `analyze_from_dataframes` (backtest parity, test-locked) and no gate reads its fields. Persona payloads are role-filtered: macro gets high-impact events plus only code-scored directional news; technical gets strategy/pattern, primary indicators, MTF, levels, and ATR-normalized candles; risk gets setup strategy, strategy cost assumption, trailing semantics, volatility/events/warnings (never the median candle range as spread); contrarian gets directional news, setup, timestamped latest-closed price, indicators, reasons, and warnings. Spend is bounded by exact cache, full-evidence-identity TTL reuse, and `max_panels_per_day`; changed evidence never inherits a direction-only cached verdict (test-locked — which also means TTL reuse effectively never fires live because price/candles change every bar; the daily cap is the real spend bound). Verdict provenance logs creation time, `MISS`/`EXACT_HIT`/`TTL_REUSE`, and a SHA-256 payload hash. Prompts forbid outside knowledge and outputs remain short. **OPPOSE guard (2026-09):** the 3-week signal-log scorecard showed the contrarian/risk/technical personas oppose almost every trend entry (chair said OPPOSE 73% of the time; opposed trades made +30R, PF 2.6) while the macro persona was the only informative voice (macro=support → 88% win). `_apply_oppose_guard` therefore downgrades a chair OPPOSE to NEUTRAL unless macro opposes, or technical opposes while macro is not supportive; the synthesizer prompt states the same rule. Re-score with `analyze_signal_log.py --stored-candles` (persona scorecard) before changing it again. - **`market_outlook.py`** — hourly DeepSeek market brief for XAU/USD, using ATR-normalized M15/H1/H4 candles, visible indicator states, levels, news, and recent signal history (read `mode=ro`). It returns a written **ADVISORY-ONLY** bias/setup-forming assessment and persists it independently in `outlooks` within `logs/signals.db` together with the news, indicator/regime/level, and candle evidence sent to the model. It may be sent post-scan by Telegram or requested with `main.py --outlook`; it never imports or runs inside the signal path. Hourly dedup is by UTC hour BUCKET (the old `now − last < 3600` rule dropped every other hour because the H1 loop fires at boundary+10 s+scan time) plus its own UTC-day cap (default 12); every silent early return logs `market outlook: skipped (...)` and an empty LLM reply logs a warning — a dark feature must be visible in `logs/telegram.log`. Provider/news failures leave a safe partial brief while LLM/database failures degrade to no outlook. - **`telegram_bot.py`** — owner-locked bot + subscriber channel; candle-boundary auto-scan loops (M5/M15/H1/H4, boundary+10 s), alert quality threshold + fingerprint dedup (committed only after a successful send; the fingerprint is signal+direction — grade is NOT part of it, a grade UPGRADE re-alerts, an A↔B flicker does not), 📰 LLM market-context line on alerts when available, WebSocket level-breach monitor fed by the streaming provider (`data_providers.streaming`, falling back to the active provider; auto-disabled when the resolved provider can't stream or has no API key — Oanda does not stream, Tiingo does). The breach handler reuses this bar's M15 report (`_cached_or_scan`, keyed by closed-bar epoch) instead of rescanning — it used to log 3.5 M15 rows per candle. Rotating log: `logs/telegram.log`, 3-h rotation × 240 backups. -- **`oanda_daemon.py` / `oanda_executor.py`** — practice-account auto-execution. Book-parity rules (see daemon docstring): no broker-side TP (trailing model — TP1/TP2 advisory), 5-bar per-TF cooldown (wall-clock approximation, so it expires over weekends where the backtester would still count bars), trail ratchets at bar close once per *observed* bar (a missed/failed cycle skips those bars entirely — the intra-gap high/low never enters the peak). Accepted divergence: one position total; orders send `positionFill=OPEN_ONLY` so a state/broker divergence cancels the order instead of silently netting. State/broker re-sync runs at boot and after any suspicious fill (`_reconcile_positions`). -- **`dashboard_publisher.py` / `static/dashboard.html`** — web dashboard as a **static publisher** (live LAMP server): standalone daemon scans at candle closes via `produce_signal` and atomically writes `snapshot.json`/`news.json`/`dashboard.html` to `$DASHBOARD_OUT_DIR` (Apache serves them; page is public by owner decision). **Hard independence rule: never imports, and is never imported by, `telegram_bot`/`oanda_daemon`** — confirmation map and boundary math are deliberate local copies, so a web-side failure can only mean a stale page. Due-TF classification uses the RAW wake time (flooring wake−buffer skips every H1/H4 scan — 2026-08-02 bug). News panel reads `NewsEngine.get_fundamental_context()` plus the additive `get_week_events()` accessor (display-only; no signal path calls it); as a separate process it has its own news caches and DeepSeek daily cap. News/calendar rows are clickable (detail dialog); headlines carry `id` = sha1(headline) in news.json, and exactly one on-demand endpoint exists: `static/news_summary.php` (installed by `install_static`), which serves a DeepSeek gold-impact summary per headline — whitelist-gated against news.json ids, cache-forever in `ai/`, own fail-closed daily cap (150/day, file-based, separate from the publisher's), key at `/etc/gold-dashboard/deepseek.key`, never reads repo config. Deployment checklist: `deploy/README-dashboard.md`. +- **`dashboard_publisher.py` / `static/dashboard.html`** — web dashboard as a **static publisher** (live LAMP server): standalone daemon scans at candle closes via `produce_signal` and atomically writes `snapshot.json`/`news.json`/`dashboard.html` to `$DASHBOARD_OUT_DIR` (Apache serves them; page is public by owner decision). **Hard independence rule: never imports, and is never imported by, `telegram_bot`** — confirmation map and boundary math are deliberate local copies, so a web-side failure can only mean a stale page. Due-TF classification uses the RAW wake time (flooring wake−buffer skips every H1/H4 scan — 2026-08-02 bug). News panel reads `NewsEngine.get_fundamental_context()` plus the additive `get_week_events()` accessor (display-only; no signal path calls it); as a separate process it has its own news caches and DeepSeek daily cap. News/calendar rows are clickable (detail dialog); headlines carry `id` = sha1(headline) in news.json, and exactly one on-demand endpoint exists: `static/news_summary.php` (installed by `install_static`), which serves a DeepSeek gold-impact summary per headline — whitelist-gated against news.json ids, cache-forever in `ai/`, own fail-closed daily cap (150/day, file-based, separate from the publisher's), key at `/etc/gold-dashboard/deepseek.key`, never reads repo config. Deployment checklist: `deploy/README-dashboard.md`. - **`backtest/gpu_backtest.py`** — bar-by-bar replay through the real `analyze_from_dataframes` (news off). Realism model (do not regress): next-bar-open entries re-anchored to the raw price path, round-trip `--spread` charged per trade, trade resolution on RAW bars (cleaning applies only to signal-side data), same-bar SL+TP tie → SL first, gap fills at bar open, trail ratchet at bar close only. Indicators are precomputed once per TF (CuPy GPU or NumPy fallback — check the `Backend:` log line); replay fans out over worker processes and the 5-bar cooldown is applied in an ordered pass afterwards, so parallel and sequential books are identical. Parquet cache in `backtest/cache/` keyed by provider/symbol/TF/window. `backtest/*.csv` are gitignored pinned trade books from past campaigns — used for manual before/after comparison. `backtest/instructions_for_backtest.txt` lists what every backtest report must record. ## Rules for making changes (read before editing signal logic) @@ -85,13 +83,14 @@ Signal currency everywhere is `SignalStrength` (−2..+2). DataFrames are UTC tz - **Weekend calendar: FIXED (owner-approved 2026-08-04).** All three signal-path `_is_forex_weekend` copies are NY-anchored (Fri 17:00 → Sun 18:00 ET; during US DST that is Fri 21:00 UTC — measured on 5yr of Oanda bars). Kills the weekly false-"STALE DATA/feed lagging" hours. Book impact at the audit HEAD: M15 byte-identical; M5 12mo dropped exactly ONE trade — a phantom whose signal fired on the final summer-Friday bar (close 21:00 UTC) and "filled" at Sunday's reopen 49 h later (−1.08R, live-impossible), leaving 555 trades / +98.64R / PF≈1.50 @ $0.65 (was 556/+97.56/1.48). The current-HEAD canonical reference is recorded below. A Module 21 test sweeps those copies plus the advisory outcome's continuity copy hourly across both DST transitions. - **Spread in exit-trigger detection (rule 6): owner DECLINED 2026-08-04.** Backtest stop triggers stay on raw mid High/Low; the measured optimism (~4–6% of total R: −3.55R @ $0.30, −4.24R @ $0.65 on the M15 12mo book) is accepted and books stay comparable. Re-open only as a conscious re-pin campaign. - **`FB2_EMA_LO`: HARDENED 2026-08-04 (delegated).** Research override for the EMA200-proximity band, default = tuned 0.5; an unparseable value now degrades to the default with a warning instead of crashing every scan. Documented here — it is a knob that exists. -- **H4 seeding divergence: parked.** Live H4 fetches only 380 bars vs the backtest's unbounded history — ~2% of sampled H4 bars flip an indicator signal (composite Δ up to 12.3). M5/M15/H1 clean at their real fetch depths. Fold into any future H4 campaign; H4 remains "unproven for auto-execution". +- **H4 seeding divergence: parked.** Live H4 fetches only 380 bars vs the backtest's unbounded history — ~2% of sampled H4 bars flip an indicator signal (composite Δ up to 12.3). M5/M15/H1 clean at their real fetch depths. Fold into any future H4 campaign; H4 remains "unproven". ## Current state notes (2026-08-14) - Oanda v20 is the live feed (`data_providers.active: "oanda"`). The 2026-08-04 re-pin made the 27 `backtest/oanda_*.csv` books the baseline; all TD-era books are superseded. Current-HEAD M5 canonical book: `backtest/oanda_m5_12mo_065_nopb_20260814.csv`, +106.82R PF 1.51 at $0.65 (both halves positive; PF 1.46 / 1.55). M15 reference: +73.93R PF 1.53. See `backtest/research/repin_verification_2026-08-14.md`. - Cost reporting is dual: $0.65 is the measured-Oanda realistic headline ($0.65-$0.72 round trip), while $0.30 remains for cross-book comparability and as the live TP spread-gate input. Raising `round_trip_cost` from $0.30 would move the TP floor from about $9 to $19.50 and requires full rule-1 revalidation. -- Twelve Data is signal-incompatible with Oanda (Phase 1: about 19% M5 bar-direction disagreement and a systematic ~$1 high bias; see `backtest/research/feed_divergence_2026-08-14.md`). On an Oanda candle outage, halt auto-execution rather than switching to TD candles. +- Twelve Data is signal-incompatible with Oanda (Phase 1: about 19% M5 bar-direction disagreement and a systematic ~$1 high bias; see `backtest/research/feed_divergence_2026-08-14.md`). On an Oanda candle outage, stop scanning rather than switching to TD candles. - Tiingo is streaming-only (`data_providers.streaming: "tiingo"`), not a designated candle fallback. Oanda-Tiingo signal parity is documented in `backtest/research/signal_parity_2026-08-14.md`; Phase 4's two-week live shadow scan remains a manual operational step. - Human-facing docs in `docs/` (`user_guide.html`, `beginner_guide.html` + PDF builds) must be kept in sync when signal behavior changes. - 2026-09-03 signal-log campaign shipped two new per-TF gates (`backtest/research/signal_log_campaign_2026-09.md`): M5 `sell_blocked_regimes=["transitioning"]` and H1 `sell_min_quality=[]` (all H1 shorts blocked). Current-HEAD M5 canonical book is now `backtest/oanda_m5_12mo_065_nopb_20260903.csv` (557 / +112.21R / PF 1.56, halves +55.9/+56.4); H1 reference `backtest/oanda_h1_12mo_065_20260903.csv` (91 / +22.2R / PF 1.79; 5yr +49.1R PF 1.46; OOS +16.7R PF 1.74). +- Auto-execution (oanda_daemon.py / oanda_executor.py) was removed 2026-09-03 — the owner is not auto-trading; last commit containing it is 31002cb. Book-parity rules that lived in its docstring (no broker TP, 5-bar cooldown, once-per-bar trail) remain the backtester's model. diff --git a/app_config.py b/app_config.py index 3218006..43f4296 100644 --- a/app_config.py +++ b/app_config.py @@ -89,8 +89,7 @@ _KNOWN_TOP = {"telegram", "data_providers", "scanning", "pairs", "log_level", "oanda", "deepseek", "agent_panel", "market_outlook", "signal_log"} _KNOWN_OANDA = { - "account_id", "api_token", "base_url", "lot_size", - "quality_gate", "max_hold_bars", + "account_id", "api_token", "base_url", } _KNOWN_DEEPSEEK = { "api_key", "model", "enabled", "timeout_seconds", "daily_request_cap", @@ -256,20 +255,6 @@ def load_app_config(path: str = CONFIG_PATH) -> AppConfig: oanda = raw.get("oanda", {}) _check_keys(oanda, _KNOWN_OANDA, '"oanda"') - if "lot_size" in oanda: - ls = oanda["lot_size"] - if not isinstance(ls, int) or isinstance(ls, bool) or ls < 1: - raise AppConfigError( - "config.json: oanda.lot_size must be a positive integer") - if "quality_gate" in oanda and oanda["quality_gate"] not in _VALID_QUALITIES: - raise AppConfigError( - f'config.json: oanda.quality_gate "{oanda["quality_gate"]}" ' - f"not in {_VALID_QUALITIES}") - if "max_hold_bars" in oanda: - mhb = oanda["max_hold_bars"] - if not isinstance(mhb, int) or isinstance(mhb, bool) or mhb < 1: - raise AppConfigError( - "config.json: oanda.max_hold_bars must be a positive integer") app.oanda_settings = dict(oanda) deepseek = raw.get("deepseek", {}) diff --git a/config.json.example b/config.json.example index a08a08e..43761c2 100644 --- a/config.json.example +++ b/config.json.example @@ -28,10 +28,7 @@ "oanda": { "account_id": "REPLACE_WITH_ACCOUNT_ID", "api_token": "REPLACE_WITH_API_TOKEN", - "base_url": "https://api-fxpractice.oanda.com", - "lot_size": 1, - "quality_gate": "C", - "max_hold_bars": 300 + "base_url": "https://api-fxpractice.oanda.com" }, "deepseek": { "api_key": "REPLACE_WITH_YOUR_DEEPSEEK_API_KEY_OR_EMPTY", diff --git a/dashboard_publisher.py b/dashboard_publisher.py index fe1844b..945ddda 100644 --- a/dashboard_publisher.py +++ b/dashboard_publisher.py @@ -11,9 +11,9 @@ can only ever mean a stale page. Independence rules (owner requirement — the telegram bot must be immune to anything on the web side): * Runs as its own process; never imported by, and never imports, - `telegram_bot` / `oanda_daemon` (telegram_bot resolves credentials - at import). The confirmation map and boundary math are therefore - duplicated here as local constants. + `telegram_bot` (telegram_bot resolves credentials at import). The + confirmation map and boundary math are therefore duplicated here + as local constants. * The only shared-code dependency beyond the standard signal path is the additive `NewsEngine.get_week_events` accessor, which no production signal path calls. @@ -236,8 +236,8 @@ def due_timeframes(wake_dt: datetime) -> List[Timeframe]: def seconds_until_boundary(interval_minutes: int, buffer: int = 10) -> float: - # Same math as telegram_bot.py / oanda_daemon.py — kept local so the - # dashboard never imports bot modules (independence rules above). + # Same math as telegram_bot.py — kept local so the dashboard never + # imports bot modules (independence rules above). now = datetime.now(timezone.utc) total_minutes = now.hour * 60 + now.minute elapsed_minutes = total_minutes % interval_minutes diff --git a/docs/architecture_diagram.py b/docs/architecture_diagram.py index 53565cd..5711d13 100644 --- a/docs/architecture_diagram.py +++ b/docs/architecture_diagram.py @@ -130,13 +130,6 @@ box(31, 33.2, 38, 5.2, "SignalReport", band(1, 13.2, 98, 17.2, "DELIVERY SURFACES", "#4C8F50") box(2.5, 15.0, 12, 12.6, "main.py", ["CLI report", "--detail", "--json"], DEL_FC, DEL_EC) -box(16, 15.0, 24, 12.6, "oanda_daemon.py + executor", - ["practice auto-execution", - "book-parity rules: no broker TP,", - "5-bar cooldown, once-per-bar", - "trail, OPEN_ONLY fills,", - "boot reconcile · orders → Oanda"], - DEL_FC, DEL_EC, title_fs=7.8) box(42.5, 15.0, 31, 12.6, "dashboard_publisher.py → Apache", ["independent daemon · scans at candle closes", "atomic publish to /var/www/trade:", @@ -170,7 +163,6 @@ arrow((55, 69.8), (55, 63.1), label="warn-only context", lx=0, ly=1.0) arrow((86, 69.8), (86, 67.3), ls=(0, (3, 2)), lw=1.1) # config → core arrow((50, 40.5), (50, 38.8)) # core → report arrow((36, 34.5), (8.5, 27.95), conn="arc3,rad=0.12") # report → cli -arrow((40, 33.2), (28, 27.95), conn="arc3,rad=0.06") # report → daemon arrow((60, 33.2), (58, 27.95)) # report → dashboard arrow((66, 34.5), (86.5, 27.95), conn="arc3,rad=-0.12") # report → telegram @@ -181,7 +173,6 @@ arrow((95, 86.2), (95, 27.95), conn="arc3,rad=0.15", color=HAUL, ax.text(99.2, 57, "alerts / commands", fontsize=6.4, color=MUT, rotation=90, ha="center", va="center") # browser click → news_summary.php → DeepSeek -# (oanda_daemon's order flow is noted inside its box — no long arc) arrow((66, 27.95), (73, 86.2), conn="arc3,rad=-0.18", color=HAUL, lw=1.2, ls=(0, (4, 2)), label="on-demand AI summary\n(browser click → PHP)", lx=2.5, ly=11.6) diff --git a/oanda_daemon.py b/oanda_daemon.py deleted file mode 100644 index af76151..0000000 --- a/oanda_daemon.py +++ /dev/null @@ -1,855 +0,0 @@ -""" -ForexBot2 — Oanda Automated Trading Daemon -============================================ -Autonomous trading process for XAU/USD on an Oanda practice account. -Runs candle-boundary scan loops (M5/M15/H1/H4), executes trades via -the Oanda v20 API, and manages trailing stops at bar close — matching -the backtested bar-close ratchet model. - -Book-parity notes (matching backtest/gpu_backtest.py): -- No broker-side take-profit: in the trailing model TP1/TP2 are - advisory; the only exits are SL (broker), TRAIL (daemon ratchet), - and TIMEOUT (max_hold_bars). A hard TP would cap winners the trail - lets run and diverge the live book from the backtest. -- Entry cooldown is 5 bars per timeframe (the backtester's only - sequential rule) — not a fingerprint block. -- The trail ratchets at bar CLOSE only, and each closed bar is - processed exactly once (last_bar_ts guard), so weekends and feed - lag can't double-count bars or inflate the hold counter. - -Divergence from the backtest book (accepted for demo safety): one -position total. While a trade is open, other signals are logged and -skipped, so skipped entries can be compared against the book later. - -Usage: python oanda_daemon.py -Config: env vars OANDA_ACCOUNT_ID, OANDA_API_TOKEN, OANDA_BASE_URL - or config.json "oanda" section (env wins). -""" - -import asyncio -import json -import logging -import os -import signal -import sys -import threading -from concurrent.futures import ThreadPoolExecutor -from datetime import datetime, timezone -from logging.handlers import TimedRotatingFileHandler -from typing import Dict, Optional - -import pandas as pd - -from app_config import ( - get_app_config, oanda_account_id, oanda_api_token, oanda_base_url, -) -from config import PAIR_INFO, PlatformConfig, Timeframe, XAUUSD -from signal_aggregator import SignalAggregator, SignalReport, SignalStrength -from pair_signals import PAIR_PRODUCERS -from providers.oanda import OandaDataProvider -from oanda_executor import OandaExecutor, OandaAPIError - -logger = logging.getLogger(__name__) - -# ── Constants ────────────────────────────────────────────────────────── - -SYMBOL = XAUUSD -INSTRUMENT = "XAU_USD" - -SCAN_CONFIRMATIONS = { - Timeframe.M5: [Timeframe.M15, Timeframe.H1], - Timeframe.M15: [Timeframe.M5, Timeframe.H1], - Timeframe.H1: [Timeframe.M15, Timeframe.H4], - Timeframe.H4: [Timeframe.H1, Timeframe.D1], -} - -TF_INTERVAL_MINUTES = { - Timeframe.M5: 5, Timeframe.M15: 15, Timeframe.M30: 30, - Timeframe.H1: 60, Timeframe.H4: 240, Timeframe.D1: 1440, -} - -SCAN_TIMEFRAMES = [Timeframe.M5, Timeframe.M15, Timeframe.H1, Timeframe.H4] - -QUALITY_RANK = {"A+": 4, "A": 3, "B": 2, "C": 1, "D": 0, "F": -1} - -# Matches the backtester's --cooldown-bars default (the only sequential -# rule in the replay): no new entry within 5 bars of the previous entry -# on the same timeframe. -COOLDOWN_BARS = 5 - -# How often the heartbeat confirms the process is alive and the account -# is still reachable — independent of the candle-boundary loops, so a -# wedged scan or mid-run credential expiry still surfaces in the log. -HEARTBEAT_SECONDS = 3600 - -STATE_FILE = os.path.join( - os.path.dirname(os.path.abspath(__file__)), "oanda_state.json") - -# ── Config resolution (env → config.json → default) ────────────────── - -def _oanda_setting_int(env_key: str, json_key: str, default: int) -> int: - env = os.environ.get(env_key) - if env: - try: - return int(env) - except ValueError: - raise SystemExit( - f"{env_key}={env!r} is not an integer — refusing to start") - try: - return int(get_app_config().oanda_settings.get(json_key, default)) - except (TypeError, ValueError): - raise SystemExit( - f"config.json oanda.{json_key} is not an integer — " - f"refusing to start") - - -def _oanda_setting_str(env_key: str, json_key: str, default: str) -> str: - return (os.environ.get(env_key) - or str(get_app_config().oanda_settings.get(json_key, default))) - -# ── Logging setup ────────────────────────────────────────────────────── - -def _setup_logging(level: str = "INFO") -> None: - log_dir = os.path.join(os.path.dirname(os.path.abspath(__file__)), "logs") - os.makedirs(log_dir, exist_ok=True) - log_file = os.path.join(log_dir, "daemon.log") - - fmt = logging.Formatter( - "%(asctime)s [%(levelname)s] %(name)s: %(message)s", - datefmt="%Y-%m-%d %H:%M:%S") - - file_handler = TimedRotatingFileHandler( - log_file, when="h", interval=3, backupCount=240, utc=True) - file_handler.setFormatter(fmt) - - console_handler = logging.StreamHandler(sys.stdout) - console_handler.setFormatter(fmt) - - logging.basicConfig( - level=getattr(logging, level.upper(), logging.INFO), - handlers=[file_handler, console_handler], - force=True) - -# ── Boundary timing (same as telegram_bot.py) ───────────────────────── - -def _seconds_until_boundary(interval_minutes: int, buffer: int = 10) -> float: - now = datetime.now(timezone.utc) - total_minutes = now.hour * 60 + now.minute - elapsed_minutes = total_minutes % interval_minutes - elapsed_seconds = elapsed_minutes * 60 + now.second + now.microsecond / 1e6 - interval_seconds = interval_minutes * 60 - remaining = interval_seconds - elapsed_seconds + buffer - # NOTE: remaining > buffer always (elapsed < interval by construction); - # a call at the exact boundary instant targets the NEXT boundary — - # steady-state loops wake at boundary+buffer, so only a process - # started in the boundary microsecond skips one scan (2026-08-04 - # audit I42; the old `remaining <= buffer` guard was unreachable). - return remaining - -# ── Trade state persistence ─────────────────────────────────────────── - -def _load_state() -> Dict: - if os.path.exists(STATE_FILE): - try: - with open(STATE_FILE, encoding="utf-8") as fh: - return json.load(fh) - except (OSError, json.JSONDecodeError): - logger.warning("Could not load state file, starting fresh") - return {"trades": {}, "last_entries": {}} - - -def _save_state(state: Dict) -> None: - try: - tmp = STATE_FILE + ".tmp" - with open(tmp, "w", encoding="utf-8") as fh: - json.dump(state, fh, indent=2) - os.replace(tmp, STATE_FILE) - except OSError: - logger.exception("Failed to save state file") - -# ── Trading Daemon ───────────────────────────────────────────────────── - -class TradingDaemon: - - def __init__(self): - account_id = oanda_account_id() - api_token = oanda_api_token() - base_url = oanda_base_url() - lot_size = _oanda_setting_int("OANDA_LOT_SIZE", "lot_size", 1) - self._quality_gate = _oanda_setting_str( - "OANDA_QUALITY_GATE", "quality_gate", "C").strip() - if self._quality_gate not in QUALITY_RANK: - # Fail CLOSED: an unrecognized gate must never silently fall - # back to the most permissive rank (QUALITY_RANK.get default). - raise SystemExit( - f"OANDA_QUALITY_GATE/quality_gate {self._quality_gate!r} " - f"is not one of {sorted(QUALITY_RANK)} — refusing to start") - self._max_hold_bars = _oanda_setting_int( - "OANDA_MAX_HOLD_BARS", "max_hold_bars", 300) - self._price_precision = PAIR_INFO[SYMBOL]["price_precision"] - - if not account_id or not api_token: - msg = ( - "Missing Oanda credentials. Set OANDA_ACCOUNT_ID and " - "OANDA_API_TOKEN env vars or config.json oanda section.") - logger.critical(msg) - raise SystemExit(msg) - - self._provider = OandaDataProvider( - PlatformConfig(), api_token, base_url) - self._executor = OandaExecutor( - account_id, api_token, base_url, lot_size, - price_precision=self._price_precision) - self._thread_pool = ThreadPoolExecutor(max_workers=4) - - self._state = _load_state() - self._state.setdefault("trades", {}) - self._state.setdefault("last_entries", {}) - self._state_lock = threading.Lock() - # Serializes the position-check + order-placement critical section - # across concurrent TF loops (all four fire at aligned boundaries). - self._entry_lock = threading.Lock() - self._running = True - - # ── Main entry point ─────────────────────────────────────────── - - async def run(self) -> None: - loop = asyncio.get_running_loop() - loop.set_exception_handler(self._asyncio_exception_handler) - for sig in (signal.SIGINT, signal.SIGTERM): - loop.add_signal_handler(sig, self._request_shutdown) - - logger.info("=== Oanda Trading Daemon starting ===") - logger.info( - "Quality gate: %s | Max hold: %d bars | Cooldown: %d bars | " - "Timeframes: %s", - self._quality_gate, self._max_hold_bars, COOLDOWN_BARS, - "/".join(tf.name for tf in SCAN_TIMEFRAMES)) - - # Credentials first: with a bad token every later call fails, so - # refuse to start before touching positions or spawning loops. - self._verify_account_access() - - self._discover_existing_positions() - - tasks = [] - for tf in SCAN_TIMEFRAMES: - interval = TF_INTERVAL_MINUTES[tf] - tasks.append(asyncio.create_task( - self._candle_loop(tf, interval))) - tasks.append(asyncio.create_task(self._heartbeat_loop())) - - logger.info("Candle-boundary loops started") - try: - await asyncio.gather(*tasks) - except asyncio.CancelledError: - pass - finally: - self._shutdown() - - def _verify_account_access(self) -> None: - """Boot-time credential check. A 4xx here means the token/account - id is wrong — that never fixes itself, so refuse to start rather - than run a daemon that can't place or manage trades. A connection - failure or 5xx is a transient outage: log it and continue, since - broker-side stops already protect any open position and the - hourly heartbeat will keep reporting account health.""" - try: - self._executor.get_account_summary() - except OandaAPIError as e: - if 400 <= e.status_code < 500: - logger.critical( - "Oanda rejected the account credentials (HTTP %d): " - "%s — refusing to start", e.status_code, e) - raise SystemExit( - f"Oanda rejected the account credentials " - f"(HTTP {e.status_code}): {e}") - logger.error( - "Failed to fetch account summary at boot (HTTP %d): %s", - e.status_code, e) - - def _asyncio_exception_handler(self, loop, context) -> None: - exc = context.get("exception") - if exc is not None: - logger.critical( - "Uncaught asyncio exception: %s", - context.get("message", ""), exc_info=exc) - else: - logger.critical( - "Uncaught asyncio error: %s", context.get("message", "")) - - def _request_shutdown(self) -> None: - logger.info("Shutdown requested") - self._running = False - for task in asyncio.all_tasks(): - task.cancel() - - def _shutdown(self) -> None: - with self._state_lock: - _save_state(self._state) - self._thread_pool.shutdown(wait=False) - logger.info("=== Oanda Trading Daemon stopped ===") - - # ── Position discovery (restart recovery) ────────────────────── - - def _discover_existing_positions(self) -> None: - self._reconcile_positions("boot discovery") - - def _reconcile_positions(self, reason: str) -> None: - """Re-sync tracked state with the broker's open trades. Safe to - call mid-run (lock-guarded); used at boot and whenever a fill or - placement error suggests state and broker may have diverged.""" - try: - open_trades = self._executor.get_open_trades(INSTRUMENT) - except OandaAPIError: - logger.exception( - "Failed to query open trades (%s) — reconcile skipped", - reason) - return - - live_ids = {t["id"] for t in open_trades} - - with self._state_lock: - saved = self._state.get("trades", {}) - - stale = [tid for tid in saved if tid not in live_ids] - for tid in stale: - logger.info( - "Stale trade %s in state file (closed on Oanda), " - "removing", tid) - del saved[tid] - - for trade in open_trades: - tid = trade["id"] - if tid in saved: - logger.info( - "Recovered trade %s from state: %s %s @ %.2f, SL %.2f", - tid, saved[tid].get("direction"), INSTRUMENT, - saved[tid].get("entry"), saved[tid].get("current_sl")) - continue - - direction = ("BUY" if float(trade.get("currentUnits", 0)) > 0 - else "SELL") - entry = float(trade.get("price", 0)) - sl_order = trade.get("stopLossOrder") or {} - sl_price = sl_order.get("price") - if sl_price is None: - # No broker-side stop at all: risk is undefined, so the - # trailing model cannot manage this trade. Scream — a - # naked position needs a human (or a policy) to set a - # stop; fabricating one here would invent trading - # behavior the books never validated. - logger.critical( - "Adopted trade %s (%s @ %.2f) has NO STOP LOSS on " - "the broker — it will NOT be trailed. Set a stop " - "manually or close it.", tid, direction, entry) - sl = entry - risk = 0.0 - else: - sl = float(sl_price) - risk = abs(entry - sl) - - saved[tid] = { - "tf": "H1", - "direction": direction, - "entry": entry, - "sl_initial": sl, - "current_sl": sl, - "risk": risk, - "peak": entry, - "trail_armed": False, - "trail_arm_r": 0.6, - "trail_dist_r": 0.75, - "bars_held": 0, - "last_bar_ts": None, - "signal_bar_ts": None, - } - logger.info( - "Adopted unknown trade %s: %s @ %.2f, SL %.2f, risk %.2f " - "(conservative defaults)", - tid, direction, entry, sl, risk) - - self._state["trades"] = saved - _save_state(self._state) - logger.info( - "Position reconcile (%s): %d active trade(s)", reason, len(saved)) - - # ── Candle-boundary loop ─────────────────────────────────────── - - async def _candle_loop(self, tf: Timeframe, interval_minutes: int) -> None: - label = tf.value - try: - while self._running: - wait = _seconds_until_boundary(interval_minutes, buffer=10) - logger.info("%s loop: next scan in %.0fs", label, wait) - await asyncio.sleep(wait) - if not self._running: - break - logger.info("%s loop: firing scan", label) - try: - await self._scan_and_manage(tf) - except Exception: - logger.exception("%s loop: cycle failed", label) - except asyncio.CancelledError: - pass - finally: - logger.info("%s loop stopped", label) - - async def _scan_and_manage(self, tf: Timeframe) -> None: - loop = asyncio.get_running_loop() - - # Trade management FIRST: the SL ratchet must not wait on (or be - # starved by) the signal scan — news fetches can take seconds and - # a failed scan must never cost an open trade its trail update. - try: - await loop.run_in_executor( - self._thread_pool, self._manage_open_trades, tf) - except Exception: - logger.exception("%s: trade management failed", tf.value) - - report = await loop.run_in_executor( - self._thread_pool, self._run_signal, tf) - - if not self._should_execute(report, tf): - return - - await loop.run_in_executor( - self._thread_pool, self._execute_trade, report, tf) - - def _manage_open_trades(self, tf: Timeframe) -> None: - self._check_closed_trades() - self._manage_trailing(tf) - - # ── Heartbeat ───────────────────────────────────────────────── - - def _heartbeat_once(self) -> None: - with self._state_lock: - n = len(self._state.get("trades", {})) - try: - self._executor.get_account_summary() - except OandaAPIError as e: - logger.error("HEARTBEAT: account summary failed: %s", e) - # Logged unconditionally — even on a failed summary call the - # heartbeat line itself must reach the log, so a wedged process - # doesn't look identical to a healthy but silent one. - logger.info("HEARTBEAT: alive | tracked trades: %d", n) - - async def _heartbeat_loop(self) -> None: - loop = asyncio.get_running_loop() - try: - while self._running: - await asyncio.sleep(HEARTBEAT_SECONDS) - if not self._running: - break - try: - await loop.run_in_executor( - self._thread_pool, self._heartbeat_once) - except Exception: - logger.exception("heartbeat loop: cycle failed") - except asyncio.CancelledError: - pass - finally: - logger.info("heartbeat loop stopped") - - # ── Signal pipeline ──────────────────────────────────────────── - - def _run_signal(self, tf: Timeframe) -> SignalReport: - conf_tfs = SCAN_CONFIRMATIONS.get( - tf, [Timeframe.M15, Timeframe.H4]) - factory = PAIR_PRODUCERS.get(SYMBOL) - if factory is not None: - config = factory(tf, conf_tfs) - else: - config = PlatformConfig() - config.primary_timeframe = tf - config.confirmation_timeframes = conf_tfs - - aggregator = SignalAggregator(config) - aggregator.data_provider = self._provider - report = aggregator.analyze_pair(symbol=SYMBOL, - log_source="oanda_daemon") - - logger.info( - "%s scan: signal=%s score=%.1f conf=%.0f quality=%s " - "regime=%s vol=%s price=%.2f spread=%.4f", - tf.value, - report.final_signal.name, report.final_score, - report.final_confidence, report.signal_quality, - report.market_regime, report.volatility_regime, - report.current_price, - self._provider.get_spread(SYMBOL)) - - # Stale-data/suppression warnings must be visible at WARNING; - # routine session-filter notices go to DEBUG — the M5 loop fires - # 288x/day and would drown the log at WARNING otherwise. - for w in report.warnings: - if "STALE" in w.upper() or "FAIL" in w.upper(): - logger.warning("%s scan: %s", tf.value, w) - else: - logger.debug("%s scan: %s", tf.value, w) - - return report - - # ── Execution decision ───────────────────────────────────────── - - def _should_execute(self, report: SignalReport, tf: Timeframe) -> bool: - if report.final_signal == SignalStrength.NEUTRAL: - return False - - if report.trade_setup is None: - return False - - quality_rank = QUALITY_RANK.get(report.signal_quality, -1) - gate_rank = QUALITY_RANK.get(self._quality_gate, 1) - if quality_rank < gate_rank: - return False - - if self._in_cooldown(tf): - logger.info( - "%s: actionable signal (%s %s) but inside %d-bar entry " - "cooldown, skipping", - tf.value, report.trade_setup.direction, - report.signal_quality, COOLDOWN_BARS) - return False - - with self._state_lock: - if self._state.get("trades"): - logger.info( - "%s: actionable signal (%s %s) but position already " - "open, skipping", - tf.value, report.trade_setup.direction, - report.signal_quality) - return False - - return True - - def _in_cooldown(self, tf: Timeframe) -> bool: - """Backtest parity: no entry within COOLDOWN_BARS bars of the - previous entry on the same timeframe (wall-clock approximation).""" - with self._state_lock: - last = self._state.get("last_entries", {}).get(tf.value) - if not last: - return False - try: - last_ts = datetime.fromisoformat(last) - except ValueError: - return False - interval_s = TF_INTERVAL_MINUTES[tf] * 60 - elapsed = (datetime.now(timezone.utc) - last_ts).total_seconds() - return elapsed < COOLDOWN_BARS * interval_s - - # ── Trade execution ──────────────────────────────────────────── - - def _execute_trade(self, report: SignalReport, tf: Timeframe) -> None: - ts = report.trade_setup - if ts is None: - return - - # All four TF loops can fire at an aligned boundary (e.g. :00) and - # pass _should_execute before any of them has recorded a fill — - # re-check under the entry lock so only one order goes out. - with self._entry_lock: - with self._state_lock: - if self._state.get("trades"): - logger.info( - "%s: position opened by a concurrent scan, skipping", - tf.value) - return - - # NO take-profit on the order: the backtested trailing model - # exits via SL/TRAIL/TIMEOUT only — a hard TP would cap - # winners and diverge the live book from the backtest. - try: - result = self._executor.place_market_order( - instrument=INSTRUMENT, - direction=ts.direction, - stop_loss=ts.stop_loss, - ) - except OandaAPIError: - # The order is never retried (a timeout after a server-side - # fill would double-up on retry), but the server MAY have - # filled before the error reached us — reconcile so a live - # position can't stay untracked until the next restart. - logger.exception( - "Failed to place %s order for %s %s — reconciling " - "broker positions", - ts.direction, SYMBOL, tf.value) - self._reconcile_positions("order placement error") - return - - fill_tx = result.get("orderFillTransaction") - if not fill_tx: - logger.warning( - "Order not filled for %s %s", SYMBOL, tf.value) - return - - trade_opened = fill_tx.get("tradeOpened", {}) - trade_id = trade_opened.get("tradeID") - if not trade_id: - # A fill with no tradeOpened means the order interacted - # with an existing broker position (tradesClosed / - # tradeReduced on a netting account) — state and broker - # have diverged. Surface it and re-sync instead of - # silently forgetting a real fill. - logger.critical( - "Order FILLED but opened no trade (netting close/" - "reduce?) — fill: closed=%s reduced=%s. Reconciling.", - fill_tx.get("tradesClosed"), fill_tx.get("tradeReduced")) - self._reconcile_positions("fill without tradeOpened") - return - - fill_price = float(fill_tx.get("price", ts.entry_price)) - risk = abs(fill_price - ts.stop_loss) - - trail_arm = ts.trail_arm_r if ts.trail_arm_r is not None else 0.6 - trail_dist = ts.trail_dist_r if ts.trail_dist_r is not None else 0.75 - - now_iso = datetime.now(timezone.utc).isoformat() - trade_state = { - "tf": tf.value, - "direction": ts.direction, - "entry": fill_price, - "sl_initial": ts.stop_loss, - "current_sl": ts.stop_loss, - "tp1": ts.take_profit_1, - "tp2": ts.take_profit_2, - "risk": risk, - "peak": fill_price, - "trail_armed": False, - "trail_arm_r": trail_arm, - "trail_dist_r": trail_dist, - "bars_held": 0, - # The signal bar is already closed; the first bar this - # trade lives through is the one that just started. Guard - # so a lagging feed can't trail on the signal bar itself. - "signal_bar_ts": report.timestamp, - "last_bar_ts": None, - "strategy": ts.strategy, - "quality": report.signal_quality, - "rr": ts.risk_reward, - "opened_at": now_iso, - } - - with self._state_lock: - self._state["trades"][trade_id] = trade_state - self._state["last_entries"][tf.value] = now_iso - _save_state(self._state) - - logger.info( - "TRADE OPENED: %s %s @ %.2f | SL %.2f | TP1 %.2f (advisory) | " - "risk $%.2f | RR %.2f | quality %s | strategy %s | TF %s | " - "trail arm %.2fR dist %.2fR", - ts.direction, SYMBOL, fill_price, ts.stop_loss, - ts.take_profit_1, - risk, ts.risk_reward, report.signal_quality, - ts.strategy, tf.value, trail_arm, trail_dist) - - # ── Trailing stop management ─────────────────────────────────── - - def _manage_trailing(self, tf: Timeframe) -> None: - # Copy the per-trade dicts too: the loop mutates info (peak, - # bars_held, last_bar_ts) and must not touch live state outside - # the lock — another thread may be serializing it in _save_state. - with self._state_lock: - trades = {tid: dict(info) - for tid, info in self._state.get("trades", {}).items()} - - for trade_id, info in trades.items(): - if info.get("tf") != tf.value: - continue - - try: - df = self._provider.fetch_ohlcv(SYMBOL, tf) - except Exception: - logger.exception( - "Failed to fetch candle data for trail update") - continue - - if df.empty: - logger.warning( - "%s: no candle data for trail update of trade %s", - tf.value, trade_id) - continue - - bar_ts = df.index[-1].isoformat() - - # Process each closed bar exactly once. Weekends and feed lag - # re-serve the same last bar; without this guard bars_held - # inflates (an M5 trade would "time out" over a weekend) and - # the peak gets double-counted. - if bar_ts == info.get("last_bar_ts"): - continue - sig_ts = info.get("signal_bar_ts") - if sig_ts: - try: - if df.index[-1] <= pd.Timestamp(sig_ts): - continue - except (ValueError, TypeError): - pass - info["last_bar_ts"] = bar_ts - - info["bars_held"] = info.get("bars_held", 0) + 1 - - if info["bars_held"] >= self._max_hold_bars: - logger.info( - "TIMEOUT: trade %s held %d bars, closing", - trade_id, info["bars_held"]) - try: - self._executor.close_trade(trade_id) - except OandaAPIError: - # Keep tracking — the trade is still live on the - # broker (incl. a 200 whose close order was cancelled, - # e.g. MARKET_HALTED — close_trade raises on no-fill); - # we retry the close at the next bar. - logger.exception( - "Failed to close timed-out trade %s — will retry", - trade_id) - with self._state_lock: - if trade_id in self._state["trades"]: - self._state["trades"][trade_id] = info - _save_state(self._state) - continue - with self._state_lock: - self._state["trades"].pop(trade_id, None) - _save_state(self._state) - continue - - last_bar = df.iloc[-1] - bar_high = float(last_bar["High"]) - bar_low = float(last_bar["Low"]) - - if info.get("risk", 0) <= 0: - # Adopted naked position (no broker SL → risk undefined): - # never manageable by the trailing model — keep it loud - # instead of silently skipping forever. - logger.warning( - "Trade %s has risk<=0 (no stop loss) — trailing " - "skipped; set a stop manually", trade_id) - - new_sl = self._compute_trail(info, bar_high, bar_low) - - if new_sl is not None: - # Round to instrument precision and track exactly what the - # broker holds, so tracked vs broker SL can never drift. - new_sl = round(new_sl, self._price_precision) - if new_sl != info["current_sl"]: - try: - self._executor.modify_trade_sl(trade_id, new_sl) - info["current_sl"] = new_sl - except OandaAPIError: - logger.exception( - "Failed to update trailing SL for trade %s", - trade_id) - - with self._state_lock: - # Write back ONLY if the trade is still tracked — another - # loop's _check_closed_trades may have popped it while we - # iterated (broker-side close). Re-inserting would create - # a ghost that blocks new entries. - if trade_id in self._state["trades"]: - self._state["trades"][trade_id] = info - _save_state(self._state) - - def _compute_trail( - self, info: Dict, bar_high: float, bar_low: float - ) -> Optional[float]: - buy = info["direction"] == "BUY" - risk = info["risk"] - if risk <= 0: - return None - arm = info.get("trail_arm_r", 0.6) - dist = info.get("trail_dist_r", 0.75) - if arm <= 0: - return None - - fav = bar_high if buy else bar_low - if buy: - info["peak"] = max(info.get("peak", info["entry"]), fav) - else: - info["peak"] = min(info.get("peak", info["entry"]), fav) - - exc = ((info["peak"] - info["entry"]) if buy - else (info["entry"] - info["peak"])) / risk - - if exc >= arm: - if buy: - lvl = info["peak"] - dist * risk - cur_sl = info.get("current_sl", info["sl_initial"]) - if lvl > cur_sl: - info["trail_armed"] = True - return lvl - else: - lvl = info["peak"] + dist * risk - cur_sl = info.get("current_sl", info["sl_initial"]) - if lvl < cur_sl: - info["trail_armed"] = True - return lvl - - return None - - # ── Closed-trade detection ───────────────────────────────────── - - def _check_closed_trades(self) -> None: - with self._state_lock: - tracked = set(self._state.get("trades", {}).keys()) - - if not tracked: - return - - try: - open_trades = self._executor.get_open_trades(INSTRUMENT) - except OandaAPIError: - logger.exception("Failed to query open trades") - return - - live_ids = {t["id"] for t in open_trades} - closed = tracked - live_ids - - for tid in closed: - with self._state_lock: - info = self._state["trades"].pop(tid, {}) - entry = info.get("entry", 0) - direction = info.get("direction", "?") - risk = info.get("risk", 1) - sl = info.get("current_sl", 0) - was_trailed = info.get("trail_armed", False) - - logger.info( - "TRADE CLOSED BY BROKER: %s %s %s | entry %.2f | " - "SL was %.2f | risk $%.2f | trailed=%s | bars_held=%d", - tid, direction, SYMBOL, entry, sl, risk, - was_trailed, info.get("bars_held", 0)) - - if closed: - with self._state_lock: - _save_state(self._state) - -# ── Uncaught exception handlers ─────────────────────────────────────── - -def _excepthook(exc_type, exc_value, exc_tb): - if issubclass(exc_type, KeyboardInterrupt): - sys.__excepthook__(exc_type, exc_value, exc_tb) - return - logger.critical( - "Uncaught exception", exc_info=(exc_type, exc_value, exc_tb)) - -def _threading_excepthook(args): - logger.critical( - "Uncaught exception in thread %s", - args.thread.name if args.thread else "?", - exc_info=(args.exc_type, args.exc_value, args.exc_traceback)) - -# ── Entry point ──────────────────────────────────────────────────────── - -def main() -> None: - _setup_logging( - os.environ.get("OANDA_LOG_LEVEL") or get_app_config().log_level) - sys.excepthook = _excepthook - threading.excepthook = _threading_excepthook - - daemon = TradingDaemon() - asyncio.run(daemon.run()) - - -if __name__ == "__main__": - main() diff --git a/oanda_executor.py b/oanda_executor.py deleted file mode 100644 index f5eba31..0000000 --- a/oanda_executor.py +++ /dev/null @@ -1,230 +0,0 @@ -""" -ForexBot2 — Oanda v20 Order Executor -====================================== -Handles all Oanda account operations: market orders, trade queries, -SL/TP modification, and trade closure. Uses raw ``requests`` against -the Oanda v20 REST API — no extra dependencies. -""" - -import logging -import time -from typing import Dict, List, Optional - -import requests - -logger = logging.getLogger(__name__) - - -class OandaAPIError(Exception): - """Raised when an Oanda API call fails.""" - - def __init__(self, message: str, status_code: int = 0, body: str = ""): - super().__init__(message) - self.status_code = status_code - self.body = body - - -class OandaExecutor: - """Oanda v20 order execution and trade management.""" - - MAX_RETRIES = 1 - RETRY_DELAY = 2.0 - - def __init__( - self, - account_id: str, - api_token: str, - base_url: str = "https://api-fxpractice.oanda.com", - lot_size: int = 1, - price_precision: int = 2, - ): - self._account_id = account_id - self._base_url = base_url.rstrip("/") - self._lot_size = lot_size - self._price_precision = price_precision - self._session = requests.Session() - self._session.headers.update({ - "Authorization": f"Bearer {api_token}", - "Accept-Datetime-Format": "RFC3339", - "Content-Type": "application/json", - }) - - # ── Order Placement ──────────────────────────────────────────── - - def place_market_order( - self, - instrument: str, - direction: str, - stop_loss: float, - take_profit: Optional[float] = None, - units: Optional[int] = None, - ) -> Dict: - if units is None: - units = self._lot_size - signed_units = str(units) if direction.upper() == "BUY" else str(-units) - - order_body = { - "order": { - "type": "MARKET", - "instrument": instrument, - "units": signed_units, - "timeInForce": "FOK", - # The book model is one position total; on a netting - # account an opposite-side order would silently CLOSE an - # existing (possibly untracked) position instead of opening - # the signal trade. OPEN_ONLY makes that case fail loudly - # (order cancelled) so the divergence surfaces instead of - # executing an unintended close. - "positionFill": "OPEN_ONLY", - "stopLossOnFill": { - "price": self._fmt_price(stop_loss), - }, - } - } - if take_profit is not None: - order_body["order"]["takeProfitOnFill"] = { - "price": self._fmt_price(take_profit), - } - - url = f"{self._base_url}/v3/accounts/{self._account_id}/orders" - # Never retry order placement: a timeout after the server already - # filled would open a SECOND position on retry. Fail loudly instead; - # the next candle boundary re-evaluates. - resp = self._request_with_retry("POST", url, json=order_body, retries=0) - data = self._parse_json(resp) - - fill_tx = data.get("orderFillTransaction") - if fill_tx: - trade_id = fill_tx.get("tradeOpened", {}).get("tradeID", "unknown") - fill_price = float(fill_tx.get("price", 0)) - logger.info( - "ORDER FILLED: %s %s %s units @ %.2f — trade %s, SL %.2f%s", - direction.upper(), instrument, signed_units, fill_price, - trade_id, stop_loss, - f", TP {take_profit:.2f}" if take_profit else "") - else: - cancel_tx = data.get("orderCancelTransaction", {}) - reason = cancel_tx.get("reason", "unknown") - logger.warning( - "ORDER NOT FILLED: %s %s — reason: %s", - direction.upper(), instrument, reason) - - return data - - # ── Trade Queries ────────────────────────────────────────────── - - def get_open_trades(self, instrument: Optional[str] = None) -> List[Dict]: - url = f"{self._base_url}/v3/accounts/{self._account_id}/openTrades" - resp = self._request_with_retry("GET", url) - trades = self._parse_json(resp).get("trades", []) - if instrument: - trades = [t for t in trades if t.get("instrument") == instrument] - return trades - - def get_trade(self, trade_id: str) -> Dict: - url = (f"{self._base_url}/v3/accounts/{self._account_id}" - f"/trades/{trade_id}") - resp = self._request_with_retry("GET", url) - return self._parse_json(resp).get("trade", {}) - - def get_account_summary(self) -> Dict: - url = f"{self._base_url}/v3/accounts/{self._account_id}/summary" - resp = self._request_with_retry("GET", url) - account = self._parse_json(resp).get("account", {}) - logger.info( - "Account: balance=%.2f, NAV=%.2f, unrealizedPL=%.2f, openTrades=%s", - float(account.get("balance", 0)), - float(account.get("NAV", 0)), - float(account.get("unrealizedPL", 0)), - account.get("openTradeCount", 0)) - return account - - # ── Trade Modification ───────────────────────────────────────── - - def modify_trade_sl(self, trade_id: str, new_sl: float) -> Dict: - url = (f"{self._base_url}/v3/accounts/{self._account_id}" - f"/trades/{trade_id}/orders") - body = {"stopLoss": {"price": self._fmt_price(new_sl)}} - resp = self._request_with_retry("PUT", url, json=body) - logger.info( - "TRAIL UPDATE: trade %s — new SL %.2f", trade_id, new_sl) - return self._parse_json(resp) - - def modify_trade_tp(self, trade_id: str, new_tp: float) -> Dict: - url = (f"{self._base_url}/v3/accounts/{self._account_id}" - f"/trades/{trade_id}/orders") - body = {"takeProfit": {"price": self._fmt_price(new_tp)}} - resp = self._request_with_retry("PUT", url, json=body) - logger.info("TP UPDATE: trade %s — new TP %.2f", trade_id, new_tp) - return self._parse_json(resp) - - def close_trade(self, trade_id: str) -> Dict: - url = (f"{self._base_url}/v3/accounts/{self._account_id}" - f"/trades/{trade_id}/close") - resp = self._request_with_retry("PUT", url, json={"units": "ALL"}) - data = self._parse_json(resp) - close_tx = data.get("orderFillTransaction") - if not close_tx: - # A 200 whose close order was created but NOT filled (e.g. - # cancelled with MARKET_HALTED during the settlement pause) - # means the position is STILL OPEN. Raising keeps the caller - # tracking it and retrying — silently returning here would - # orphan a live position. - reason = data.get("orderCancelTransaction", {}).get( - "reason", "no orderFillTransaction in response") - raise OandaAPIError( - f"Close of trade {trade_id} did not fill: {reason}", - resp.status_code, str(data)[:300]) - close_price = float(close_tx.get("price", 0)) - pl = float(close_tx.get("pl", 0)) - logger.info( - "TRADE CLOSED: trade %s @ %.2f — PL %.2f", - trade_id, close_price, pl) - return data - - # ── Internals ────────────────────────────────────────────────── - - @staticmethod - def _parse_json(resp: requests.Response) -> Dict: - try: - return resp.json() - except ValueError as e: - raise OandaAPIError( - f"Oanda returned a non-JSON response (HTTP {resp.status_code})", - resp.status_code, resp.text[:300]) from e - - def _fmt_price(self, price: float) -> str: - # Oanda rejects prices with more decimals than the instrument's - # displayPrecision (XAU_USD: sub-cent precision is invalid). - return f"{price:.{self._price_precision}f}" - - def _request_with_retry( - self, method: str, url: str, retries: Optional[int] = None, **kwargs - ) -> requests.Response: - if retries is None: - retries = self.MAX_RETRIES - last_err = None - for attempt in range(1 + retries): - try: - resp = self._session.request(method, url, timeout=15, **kwargs) - if resp.status_code >= 500: - raise OandaAPIError( - f"Oanda server error {resp.status_code}", - resp.status_code, resp.text) - if resp.status_code >= 400: - raise OandaAPIError( - f"Oanda API error {resp.status_code}: {resp.text}", - resp.status_code, resp.text) - return resp - except (requests.ConnectionError, requests.Timeout, - OandaAPIError) as e: - last_err = e - if isinstance(e, OandaAPIError) and 400 <= e.status_code < 500: - raise - if attempt < retries: - logger.warning( - "Oanda request failed (attempt %d/%d): %s — retrying in %.0fs", - attempt + 1, 1 + retries, e, - self.RETRY_DELAY) - time.sleep(self.RETRY_DELAY) - raise OandaAPIError(f"Oanda request failed after retries: {last_err}") diff --git a/test_all.py b/test_all.py index ee49e37..5eb2299 100644 --- a/test_all.py +++ b/test_all.py @@ -3464,14 +3464,12 @@ def _(): # ══════════════════════════════════════════════════════════════════════ -# MODULE 15: Oanda provider / executor / daemon +# MODULE 15: Oanda provider # ══════════════════════════════════════════════════════════════════════ -print("\n═══ Module 15: Oanda provider / executor / daemon ═══") +print("\n═══ Module 15: Oanda provider ═══") import providers.oanda -import oanda_executor -import oanda_daemon @test("providers.oanda: OandaDataProvider parses Oanda candle JSON into correct DataFrame") @@ -3647,568 +3645,20 @@ def _(): assert spread == 0.30, f"Expected default spread 0.30, got {spread}" -@test("oanda_executor: market order body construction") -def _(): - executor = oanda_executor.OandaExecutor.__new__( - oanda_executor.OandaExecutor) - executor._account_id = "test-account" - executor._base_url = "https://test" - executor._lot_size = 1 - executor._price_precision = 2 - executor._session = None - - captured = {} - - def mock_request(self, method, url, **kwargs): - captured["method"] = method - captured["url"] = url - captured["json"] = kwargs.get("json") - return type("R", (), { - "status_code": 201, - "json": lambda self: { - "orderFillTransaction": { - "price": "2400.50", - "tradeOpened": {"tradeID": "123"}, - } - }, - })() - - executor._session = type("S", (), { - "request": mock_request, - "headers": {}, - })() - - result = executor.place_market_order( - instrument="XAU_USD", - direction="BUY", - stop_loss=2392.50, - take_profit=2420.00, - ) - - body = captured["json"] - order = body["order"] - assert order["type"] == "MARKET" - assert order["instrument"] == "XAU_USD" - assert order["units"] == "1" - # XAU price precision is 2 — Oanda rejects excess decimals. - assert order["stopLossOnFill"]["price"] == "2392.50" - assert order["takeProfitOnFill"]["price"] == "2420.00" - - fill = result.get("orderFillTransaction") - assert fill is not None - assert fill["tradeOpened"]["tradeID"] == "123" - - -@test("oanda_executor: SELL order uses negative units") -def _(): - executor = oanda_executor.OandaExecutor.__new__( - oanda_executor.OandaExecutor) - executor._account_id = "test-account" - executor._base_url = "https://test" - executor._lot_size = 1 - executor._price_precision = 2 - - captured = {} - - def mock_request(self, method, url, **kwargs): - captured["json"] = kwargs.get("json") - return type("R", (), { - "status_code": 201, - "json": lambda self: { - "orderFillTransaction": { - "price": "2400.50", - "tradeOpened": {"tradeID": "456"}, - } - }, - })() - - executor._session = type("S", (), { - "request": mock_request, - "headers": {}, - })() - - executor.place_market_order( - instrument="XAU_USD", direction="SELL", stop_loss=2410.00) - - order = captured["json"]["order"] - assert order["units"] == "-1", f"SELL should be -1, got {order['units']}" - assert "takeProfitOnFill" not in order, "No TP should be set when None" - - -@test("oanda_executor: 4xx error raises immediately (no retry)") -def _(): - executor = oanda_executor.OandaExecutor.__new__( - oanda_executor.OandaExecutor) - executor._account_id = "test" - executor._base_url = "https://test" - executor._lot_size = 1 - call_count = {"n": 0} - - def mock_request(self, method, url, **kwargs): - call_count["n"] += 1 - return type("R", (), { - "status_code": 400, - "text": "INVALID_ORDER", - })() - - executor._session = type("S", (), { - "request": mock_request, - "headers": {}, - })() - - try: - executor._request_with_retry("POST", "https://test/orders") - assert False, "Should have raised OandaAPIError" - except oanda_executor.OandaAPIError as e: - assert e.status_code == 400 - assert call_count["n"] == 1, "4xx should not retry" - - -@test("oanda_daemon: trailing stop ratchet matches backtester logic") -def _(): - daemon = oanda_daemon.TradingDaemon.__new__(oanda_daemon.TradingDaemon) - - info_buy = { - "direction": "BUY", - "entry": 2400.0, - "sl_initial": 2384.0, - "current_sl": 2384.0, - "risk": 16.0, - "peak": 2400.0, - "trail_armed": False, - "trail_arm_r": 0.6, - "trail_dist_r": 0.75, - } - - # Bar 1: price moves up but not enough to arm (< 0.6R = $9.60) - new_sl = daemon._compute_trail(info_buy, bar_high=2408.0, bar_low=2399.0) - assert new_sl is None, "Should not arm yet (excursion 0.5R)" - assert info_buy["peak"] == 2408.0 - - # Bar 2: excursion reaches 0.625R = $10 — arms, trail = 2408 - 0.75*16 = 2396 - new_sl = daemon._compute_trail(info_buy, bar_high=2410.0, bar_low=2403.0) - assert new_sl is not None, "Should arm at 0.625R" - assert abs(new_sl - 2398.0) < 0.01, f"Expected trail ~2398, got {new_sl}" - assert info_buy["trail_armed"] is True - - # Bar 3: price goes higher, trail ratchets up - info_buy["current_sl"] = new_sl - new_sl2 = daemon._compute_trail(info_buy, bar_high=2420.0, bar_low=2410.0) - assert new_sl2 is not None - assert new_sl2 > new_sl, "Trail must ratchet up" - expected = 2420.0 - 0.75 * 16.0 # 2408.0 - assert abs(new_sl2 - expected) < 0.01, f"Expected {expected}, got {new_sl2}" - - # Bar 4: price pulls back — trail must NOT ratchet down - info_buy["current_sl"] = new_sl2 - new_sl3 = daemon._compute_trail(info_buy, bar_high=2412.0, bar_low=2405.0) - assert new_sl3 is None, "Trail must not ratchet backwards" - - -@test("oanda_daemon: trailing stop for SELL direction") -def _(): - daemon = oanda_daemon.TradingDaemon.__new__(oanda_daemon.TradingDaemon) - - info_sell = { - "direction": "SELL", - "entry": 2400.0, - "sl_initial": 2416.0, - "current_sl": 2416.0, - "risk": 16.0, - "peak": 2400.0, - "trail_armed": False, - "trail_arm_r": 0.6, - "trail_dist_r": 0.75, - } - - # Not enough excursion - new_sl = daemon._compute_trail(info_sell, bar_high=2401.0, bar_low=2393.0) - assert new_sl is None, "Should not arm yet" - assert info_sell["peak"] == 2393.0 - - # Arms: peak at 2390, excursion = 10/16 = 0.625R - new_sl = daemon._compute_trail(info_sell, bar_high=2395.0, bar_low=2390.0) - assert new_sl is not None - expected = 2390.0 + 0.75 * 16.0 # 2402.0 - assert abs(new_sl - expected) < 0.01, f"Expected {expected}, got {new_sl}" - assert info_sell["trail_armed"] is True - - # Ratchets down - info_sell["current_sl"] = new_sl - new_sl2 = daemon._compute_trail(info_sell, bar_high=2392.0, bar_low=2380.0) - assert new_sl2 is not None - assert new_sl2 < new_sl, "SELL trail must ratchet down" - - -@test("oanda_daemon: _should_execute respects quality gate") -def _(): - daemon = oanda_daemon.TradingDaemon.__new__(oanda_daemon.TradingDaemon) - daemon._quality_gate = "B" - daemon._state = {"trades": {}, "last_entries": {}} - daemon._state_lock = threading.Lock() - - from signal_aggregator import TradeSetup - setup = TradeSetup( - pair="XAU/USD", display_name="XAU/USD", direction="BUY", - entry_price=2400.0, stop_loss=2384.0, - take_profit_1=2416.0, take_profit_2=2432.0, - risk_reward=1.0, risk_pips=160.0, reward_pips=160.0, - pip_value=0.10) - - report_c = SignalReport( - symbol="XAU/USD", display_name="XAU/USD", timestamp="2026-07-20") - report_c.final_signal = SignalStrength.BUY - report_c.signal_quality = "C" - report_c.trade_setup = setup - - assert not daemon._should_execute(report_c, Timeframe.H1), \ - "C should be blocked by gate=B" - - report_a = SignalReport( - symbol="XAU/USD", display_name="XAU/USD", timestamp="2026-07-20") - report_a.final_signal = SignalStrength.BUY - report_a.signal_quality = "A" - report_a.trade_setup = setup - - assert daemon._should_execute(report_a, Timeframe.H1), \ - "A should pass gate=B" - - -@test("oanda_daemon: _should_execute blocks when position exists") -def _(): - daemon = oanda_daemon.TradingDaemon.__new__(oanda_daemon.TradingDaemon) - daemon._quality_gate = "C" - daemon._state = {"trades": {"12345": {"direction": "BUY"}}, - "last_entries": {}} - daemon._state_lock = threading.Lock() - - from signal_aggregator import TradeSetup - setup = TradeSetup( - pair="XAU/USD", display_name="XAU/USD", direction="BUY", - entry_price=2400.0, stop_loss=2384.0, - take_profit_1=2416.0, take_profit_2=2432.0, - risk_reward=1.0, risk_pips=160.0, reward_pips=160.0, - pip_value=0.10) - - report = SignalReport( - symbol="XAU/USD", display_name="XAU/USD", timestamp="2026-07-20") - report.final_signal = SignalStrength.BUY - report.signal_quality = "A" - report.trade_setup = setup - - assert not daemon._should_execute(report, Timeframe.H1), \ - "Should block when a position already exists" - - -@test("oanda_daemon: NEUTRAL signal is never executed") -def _(): - daemon = oanda_daemon.TradingDaemon.__new__(oanda_daemon.TradingDaemon) - daemon._quality_gate = "C" - daemon._state = {"trades": {}, "last_entries": {}} - daemon._state_lock = threading.Lock() - - report = SignalReport( - symbol="XAU/USD", display_name="XAU/USD", timestamp="2026-07-20") - report.final_signal = SignalStrength.NEUTRAL - report.signal_quality = "F" - report.trade_setup = None - - assert not daemon._should_execute(report, Timeframe.H1) - - -def _make_daemon_report(quality="A", direction="BUY"): - from signal_aggregator import TradeSetup - setup = TradeSetup( - pair="XAU/USD", display_name="XAU/USD", direction=direction, - entry_price=2400.0, stop_loss=2384.0, - take_profit_1=2416.0, take_profit_2=2432.0, - risk_reward=1.0, risk_pips=160.0, reward_pips=160.0, - pip_value=0.10, trail_arm_r=0.6, trail_dist_r=0.85) - report = SignalReport( - symbol="XAU/USD", display_name="XAU/USD", - timestamp="2026-07-20 10:00:00+00:00") - report.final_signal = (SignalStrength.BUY if direction == "BUY" - else SignalStrength.SELL) - report.signal_quality = quality - report.trade_setup = setup - return report - - -@test("oanda_daemon: 5-bar entry cooldown blocks re-entry (backtest parity)") -def _(): - daemon = oanda_daemon.TradingDaemon.__new__(oanda_daemon.TradingDaemon) - daemon._quality_gate = "C" - daemon._state = {"trades": {}, "last_entries": {}} - daemon._state_lock = threading.Lock() - - report = _make_daemon_report() - - # Entry 2 minutes ago on M5 → inside 5×5min cooldown → blocked - recent = (datetime.now(timezone.utc) - timedelta(minutes=2)).isoformat() - daemon._state["last_entries"]["5min"] = recent - assert not daemon._should_execute(report, Timeframe.M5), \ - "Entry 2min ago must block M5 (cooldown 25min)" - - # Entry 30 minutes ago on M5 → past 5×5min cooldown → allowed - old = (datetime.now(timezone.utc) - timedelta(minutes=30)).isoformat() - daemon._state["last_entries"]["5min"] = old - assert daemon._should_execute(report, Timeframe.M5), \ - "Entry 30min ago must clear the 25min M5 cooldown" - - # M5 cooldown must not block H1 - daemon._state["last_entries"] = {"5min": recent} - assert daemon._should_execute(report, Timeframe.H1), \ - "Cooldown is per-timeframe" - - -@test("oanda_daemon: orders carry NO take-profit (trailing book parity)") +@test("app_config: unknown oanda key raises AppConfigError") def _(): import tempfile - daemon = oanda_daemon.TradingDaemon.__new__(oanda_daemon.TradingDaemon) - daemon._state = {"trades": {}, "last_entries": {}} - daemon._state_lock = threading.Lock() - daemon._entry_lock = threading.Lock() - - captured = {} - - class MockExecutor: - def place_market_order(self, instrument, direction, stop_loss, - take_profit=None, units=None): - captured["take_profit"] = take_profit - captured["stop_loss"] = stop_loss - return { - "orderFillTransaction": { - "price": "2400.60", - "tradeOpened": {"tradeID": "777"}, - } - } - - daemon._executor = MockExecutor() - - with tempfile.NamedTemporaryFile(suffix=".json", delete=False) as f: - tmp_path = f.name - orig_state_file = oanda_daemon.STATE_FILE - try: - oanda_daemon.STATE_FILE = tmp_path - daemon._execute_trade(_make_daemon_report(), Timeframe.M15) - finally: - oanda_daemon.STATE_FILE = orig_state_file - os.unlink(tmp_path) - - assert captured.get("take_profit") is None, \ - "Backtested trailing model has no hard TP — order must not carry one" - assert captured["stop_loss"] == 2384.0 - trade = daemon._state["trades"]["777"] - assert trade["entry"] == 2400.60, "Risk must be computed from actual fill" - assert abs(trade["risk"] - 16.60) < 0.01 - assert trade["trail_arm_r"] == 0.6 and trade["trail_dist_r"] == 0.85, \ - "Trail params must come from the setup stamp (per-TF factory values)" - assert daemon._state["last_entries"]["15min"], \ - "Entry must be recorded for the cooldown" - - -@test("oanda_daemon: entry-lock recheck prevents double entry from concurrent scans") -def _(): - import tempfile - daemon = oanda_daemon.TradingDaemon.__new__(oanda_daemon.TradingDaemon) - # A concurrent TF loop already opened a trade after our _should_execute - daemon._state = {"trades": {"111": {"direction": "BUY"}}, - "last_entries": {}} - daemon._state_lock = threading.Lock() - daemon._entry_lock = threading.Lock() - - placed = {"n": 0} - - class MockExecutor: - def place_market_order(self, *a, **k): - placed["n"] += 1 - return {} - - daemon._executor = MockExecutor() - - with tempfile.NamedTemporaryFile(suffix=".json", delete=False) as f: - tmp_path = f.name - orig_state_file = oanda_daemon.STATE_FILE - try: - oanda_daemon.STATE_FILE = tmp_path - daemon._execute_trade(_make_daemon_report(), Timeframe.M5) - finally: - oanda_daemon.STATE_FILE = orig_state_file - os.unlink(tmp_path) - - assert placed["n"] == 0, \ - "Recheck under the entry lock must skip when a position exists" - - -@test("oanda_daemon: same closed bar is never processed twice") -def _(): - import tempfile - daemon = oanda_daemon.TradingDaemon.__new__(oanda_daemon.TradingDaemon) - daemon._max_hold_bars = 300 - daemon._price_precision = 2 - daemon._state_lock = threading.Lock() - - bar_time = pd.Timestamp("2026-07-20 10:00:00", tz="UTC") - df = pd.DataFrame( - {"Open": [2400.0], "High": [2405.0], "Low": [2398.0], - "Close": [2403.0], "Volume": [100.0]}, - index=pd.DatetimeIndex([bar_time], name="Datetime")) - - class MockProvider: - def fetch_ohlcv(self, symbol, tf): - return df - - daemon._provider = MockProvider() - daemon._executor = None # trail must not arm on this bar - - daemon._state = { - "trades": { - "555": { - "tf": "5min", "direction": "BUY", "entry": 2400.0, - "sl_initial": 2384.0, "current_sl": 2384.0, "risk": 16.0, - "peak": 2400.0, "trail_armed": False, - "trail_arm_r": 0.6, "trail_dist_r": 0.85, - "bars_held": 0, "last_bar_ts": None, "signal_bar_ts": None, - } - }, - "last_entries": {}, - } - - with tempfile.NamedTemporaryFile(suffix=".json", delete=False) as f: - tmp_path = f.name - orig_state_file = oanda_daemon.STATE_FILE - try: - oanda_daemon.STATE_FILE = tmp_path - daemon._manage_trailing(Timeframe.M5) - daemon._manage_trailing(Timeframe.M5) # same bar re-served (weekend/lag) - daemon._manage_trailing(Timeframe.M5) - finally: - oanda_daemon.STATE_FILE = orig_state_file - os.unlink(tmp_path) - - assert daemon._state["trades"]["555"]["bars_held"] == 1, \ - "A re-served bar must not inflate bars_held (weekend timeout bug)" - - -@test("oanda_daemon: failed timeout-close keeps the trade tracked") -def _(): - import tempfile - daemon = oanda_daemon.TradingDaemon.__new__(oanda_daemon.TradingDaemon) - daemon._max_hold_bars = 1 - daemon._price_precision = 2 - daemon._state_lock = threading.Lock() - - bar_time = pd.Timestamp("2026-07-20 10:00:00", tz="UTC") - df = pd.DataFrame( - {"Open": [2400.0], "High": [2405.0], "Low": [2398.0], - "Close": [2403.0], "Volume": [100.0]}, - index=pd.DatetimeIndex([bar_time], name="Datetime")) - - class MockProvider: - def fetch_ohlcv(self, symbol, tf): - return df - - class MockExecutor: - def close_trade(self, trade_id): - raise oanda_executor.OandaAPIError("market halted", 503) - - daemon._provider = MockProvider() - daemon._executor = MockExecutor() - - daemon._state = { - "trades": { - "666": { - "tf": "5min", "direction": "BUY", "entry": 2400.0, - "sl_initial": 2384.0, "current_sl": 2384.0, "risk": 16.0, - "peak": 2400.0, "trail_armed": False, - "trail_arm_r": 0.6, "trail_dist_r": 0.85, - "bars_held": 0, "last_bar_ts": None, "signal_bar_ts": None, - } - }, - "last_entries": {}, - } - - with tempfile.NamedTemporaryFile(suffix=".json", delete=False) as f: - tmp_path = f.name - orig_state_file = oanda_daemon.STATE_FILE - try: - oanda_daemon.STATE_FILE = tmp_path - daemon._manage_trailing(Timeframe.M5) - finally: - oanda_daemon.STATE_FILE = orig_state_file - os.unlink(tmp_path) - - assert "666" in daemon._state["trades"], \ - "A failed broker close must keep the trade tracked for retry" - - -@test("app_config: invalid oanda section values raise AppConfigError") -def _(): - import tempfile - for bad in ( - {"oanda": {"quality_gate": "Z"}}, - {"oanda": {"lot_size": 0}}, - {"oanda": {"max_hold_bars": -5}}, - {"oanda": {"unknown_key": 1}}, - ): - with tempfile.NamedTemporaryFile( - mode="w", suffix=".json", delete=False) as f: - json.dump(bad, f) - tmp_path = f.name - try: - app_config.load_app_config(tmp_path) - assert False, f"Should have raised for {bad}" - except app_config.AppConfigError: - pass - finally: - os.unlink(tmp_path) - - -@test("oanda_daemon: boundary timing matches telegram_bot") -def _(): - from telegram_bot import _seconds_until_boundary as tb_boundary - from oanda_daemon import _seconds_until_boundary as od_boundary - for interval in (5, 15, 60, 240): - tb_val = tb_boundary(interval, buffer=10) - od_val = od_boundary(interval, buffer=10) - assert abs(tb_val - od_val) < 0.01, \ - f"Boundary timing mismatch for {interval}min: {tb_val} vs {od_val}" - - -@test("oanda_daemon: state file round-trip") -def _(): - import tempfile - state = { - "trades": { - "999": { - "tf": "M15", - "direction": "BUY", - "entry": 2400.0, - "sl_initial": 2384.0, - "current_sl": 2392.0, - "risk": 16.0, - "peak": 2412.0, - "trail_armed": True, - "trail_arm_r": 0.6, - "trail_dist_r": 0.75, - "bars_held": 25, - } - } - } + bad = {"oanda": {"unknown_key": 1}} with tempfile.NamedTemporaryFile( mode="w", suffix=".json", delete=False) as f: + json.dump(bad, f) tmp_path = f.name - - orig_state_file = oanda_daemon.STATE_FILE try: - oanda_daemon.STATE_FILE = tmp_path - oanda_daemon._save_state(state) - loaded = oanda_daemon._load_state() - assert loaded["trades"]["999"]["entry"] == 2400.0 - assert loaded["trades"]["999"]["trail_armed"] is True - assert loaded["trades"]["999"]["bars_held"] == 25 + app_config.load_app_config(tmp_path) + assert False, f"Should have raised for {bad}" + except app_config.AppConfigError: + pass finally: - oanda_daemon.STATE_FILE = orig_state_file os.unlink(tmp_path) @@ -4301,78 +3751,6 @@ def _(): f"Retry failure must keep the original 250-bar df, got {len(result)}" -@test("oanda_executor: non-JSON response raises OandaAPIError with status") -def _(): - def _bad_json(self): - raise ValueError("Expecting value: line 1 column 1 (char 0)") - - mock_resp = type("MockResp", (), { - "status_code": 200, - "text": "gateway error", - "json": _bad_json, - })() - - executor = oanda_executor.OandaExecutor.__new__( - oanda_executor.OandaExecutor) - executor._account_id = "test-account" - executor._base_url = "https://test" - executor._session = type("S", (), { - "request": lambda self, method, url, **kwargs: mock_resp, - "headers": {}, - })() - - try: - executor.get_open_trades() - assert False, "Should have raised OandaAPIError" - except oanda_executor.OandaAPIError as e: - assert e.status_code == 200, f"Expected status 200, got {e.status_code}" - assert "non-JSON" in str(e), f"Expected non-JSON in message: {e}" - - -@test("oanda_daemon: boot fails fast on rejected credentials (4xx)") -def _(): - daemon = oanda_daemon.TradingDaemon.__new__(oanda_daemon.TradingDaemon) - - class MockExecutor401: - def get_account_summary(self): - raise oanda_executor.OandaAPIError("unauthorized", 401) - - daemon._executor = MockExecutor401() - try: - daemon._verify_account_access() - assert False, "A 4xx credential rejection must raise SystemExit" - except SystemExit: - pass - - class MockExecutorConnFail: - def get_account_summary(self): - raise oanda_executor.OandaAPIError("conn refused", 0) - - daemon._executor = MockExecutorConnFail() - daemon._verify_account_access() # transient — must NOT raise - - -@test("oanda_daemon: heartbeat never raises and logs on account failure") -def _(): - daemon = oanda_daemon.TradingDaemon.__new__(oanda_daemon.TradingDaemon) - daemon._state = {"trades": {}, "last_entries": {}} - daemon._state_lock = threading.Lock() - - class MockExecutorDown: - def get_account_summary(self): - raise oanda_executor.OandaAPIError("down", 503) - - daemon._executor = MockExecutorDown() - daemon._heartbeat_once() # must not raise - - class MockExecutorUp: - def get_account_summary(self): - return {"balance": "1000"} - - daemon._executor = MockExecutorUp() - daemon._heartbeat_once() # must not raise - - # ══════════════════════════════════════════════════════════════════════ # MODULE 16: Provider abstraction (providers/ package) # ══════════════════════════════════════════════════════════════════════ @@ -5827,227 +5205,6 @@ def _(): # ══════════════════════════════════════════════════════════════════════ print("\n═══ Module 21: 2026-08-04 audit fixes — execution lifecycle ═══") -import threading as _thr21 - - -def _m21_daemon(state=None, quality_gate="C", max_hold=300): - d = oanda_daemon.TradingDaemon.__new__(oanda_daemon.TradingDaemon) - d._state = state if state is not None else {"trades": {}, "last_entries": {}} - d._state_lock = _thr21.Lock() - d._entry_lock = _thr21.Lock() - d._quality_gate = quality_gate - d._max_hold_bars = max_hold - d._price_precision = 2 - d._running = True - return d - - -def _m21_bar(ts="2026-08-04 10:00:00+00:00", high=2410.0, low=2395.0): - idx = pd.DatetimeIndex([pd.Timestamp(ts)]) - return pd.DataFrame({"Open": [2400.0], "High": [high], "Low": [low], - "Close": [2405.0], "Volume": [1.0]}, index=idx) - - -def _m21_trade(**over): - base = {"tf": "5min", "direction": "BUY", "entry": 2400.0, - "sl_initial": 2384.0, "current_sl": 2384.0, "risk": 16.0, - "peak": 2400.0, "trail_armed": False, "trail_arm_r": 0.6, - "trail_dist_r": 0.85, "bars_held": 0, - "signal_bar_ts": None, "last_bar_ts": None} - base.update(over) - return base - - -@test("executor: close_trade raises when the close order did not fill") -def _(): - ex = oanda_executor.OandaExecutor.__new__(oanda_executor.OandaExecutor) - ex._account_id, ex._base_url, ex._price_precision = "acc", "http://x", 2 - - class _Resp: - status_code = 200 - text = "{}" - def json(self): - return {"orderCreateTransaction": {"id": "1"}, - "orderCancelTransaction": {"reason": "MARKET_HALTED"}} - - ex._request_with_retry = lambda *a, **k: _Resp() - try: - ex.close_trade("T1") - assert False, "close_trade must raise on a no-fill close response" - except oanda_executor.OandaAPIError as e: - assert "MARKET_HALTED" in str(e) - - -@test("daemon: timeout close that did not fill keeps the trade tracked") -def _(): - d = _m21_daemon({"trades": {"T1": _m21_trade(bars_held=299)}, - "last_entries": {}}, max_hold=300) - - class _Exec: - def close_trade(self, tid): - raise oanda_executor.OandaAPIError( - "Close of trade T1 did not fill: MARKET_HALTED", 200, "") - - class _Prov: - def fetch_ohlcv(self, sym, tf): - return _m21_bar() - - d._executor, d._provider = _Exec(), _Prov() - d._manage_trailing(Timeframe.M5) - assert "T1" in d._state["trades"], \ - "un-filled close must NOT pop the trade from state" - - -@test("daemon: fill without tradeOpened triggers reconciliation, not silence") -def _(): - d = _m21_daemon() - recon = [] - d._reconcile_positions = lambda reason: recon.append(reason) - - class _Exec: - def place_market_order(self, **k): - return {"orderFillTransaction": { - "id": "5001", "price": "2400.5", - "tradesClosed": [{"tradeID": "OLD1"}]}} - - d._executor = _Exec() - ts = TradeSetup(pair="XAU_USD", display_name="XAU/USD", - direction="SELL", entry_price=2400.0, stop_loss=2416.0, - take_profit_1=2384.0, take_profit_2=2376.0, - risk_reward=1.0, risk_pips=160.0, reward_pips=160.0, - pip_value=0.1) - rep = SignalReport(symbol="XAU_USD", display_name="XAU/USD", - timestamp="2026-08-04 09:55:00+00:00") - rep.trade_setup, rep.signal_quality = ts, "B" - rep.final_signal = SignalStrength.SELL - d._execute_trade(rep, Timeframe.M5) - assert recon, "unexpected fill shape must trigger _reconcile_positions" - assert not d._state["trades"] - - -@test("daemon: placement OandaAPIError triggers reconciliation") -def _(): - d = _m21_daemon() - recon = [] - d._reconcile_positions = lambda reason: recon.append(reason) - - class _Exec: - def place_market_order(self, **k): - raise oanda_executor.OandaAPIError("timeout after retries") - - d._executor = _Exec() - ts = TradeSetup(pair="XAU_USD", display_name="XAU/USD", - direction="BUY", entry_price=2400.0, stop_loss=2384.0, - take_profit_1=2416.0, take_profit_2=2432.0, - risk_reward=1.0, risk_pips=160.0, reward_pips=160.0, - pip_value=0.1) - rep = SignalReport(symbol="XAU_USD", display_name="XAU/USD", - timestamp="2026-08-04 09:55:00+00:00") - rep.trade_setup, rep.signal_quality = ts, "B" - rep.final_signal = SignalStrength.BUY - d._execute_trade(rep, Timeframe.M5) - assert recon, "placement error must trigger _reconcile_positions " \ - "(server may have filled before the error)" - - -@test("daemon: reconcile adopts naked position with risk=0 and no fake SL") -def _(): - d = _m21_daemon() - - class _Exec: - def get_open_trades(self, instrument=None): - return [{"id": "N1", "instrument": "XAU_USD", - "currentUnits": "1", "price": "2400.0"}] # no stopLossOrder - - d._executor = _Exec() - d._reconcile_positions("test") - info = d._state["trades"]["N1"] - assert info["risk"] == 0.0, "risk must be 0 (undefined), not entry-diff" - assert d._compute_trail(dict(info), 2500.0, 2400.0) is None, \ - "risk=0 must never arm the trail (no invented stop)" - - -@test("daemon: trade popped mid-trailing is NOT resurrected by write-back") -def _(): - d = _m21_daemon({"trades": {"R1": _m21_trade()}, "last_entries": {}}) - - class _Exec: - def get_open_trades(self, instrument=None): - return [] - def modify_trade_sl(self, tid, sl): - return {} - - d._executor = _Exec() - - class _Prov: - def fetch_ohlcv(self, sym, tf): - d._check_closed_trades() # interleave: broker closed R1 - assert "R1" not in d._state["trades"] - return _m21_bar() - - d._provider = _Prov() - d._manage_trailing(Timeframe.M5) - assert "R1" not in d._state["trades"], \ - "write-back must not re-insert a trade closed by the broker" - - -@test("daemon: trailing snapshot no longer aliases live state dicts") -def _(): - d = _m21_daemon({"trades": {"S1": _m21_trade()}, "last_entries": {}}) - with d._state_lock: - snap = {tid: dict(info) - for tid, info in d._state.get("trades", {}).items()} - assert snap["S1"] is not d._state["trades"]["S1"] - - -@test("daemon: unknown quality gate refuses to start (fail closed)") -def _(): - real_init = oanda_daemon.TradingDaemon.__init__ - # Drive just the validation logic the ctor now applies. - gate = "Bb" - assert gate not in oanda_daemon.QUALITY_RANK - d = _m21_daemon(quality_gate="Bb") - # _should_execute is unreachable in prod with a bad gate (ctor exits); - # the ctor check is the contract: - src = open(os.path.join(os.path.dirname(__file__), - "oanda_daemon.py"), encoding="utf-8").read() - assert "refusing to start" in src and "not in QUALITY_RANK" in src - - -@test("executor: market orders request positionFill=OPEN_ONLY") -def _(): - ex = oanda_executor.OandaExecutor.__new__(oanda_executor.OandaExecutor) - ex._account_id, ex._base_url, ex._price_precision = "acc", "http://x", 2 - ex._lot_size = 1 - sent = {} - - class _Resp: - status_code = 200 - text = "{}" - def json(self): - return {"orderFillTransaction": { - "price": "2400.0", "tradeOpened": {"tradeID": "T9"}}} - - def _req(method, url, retries=None, **kw): - sent.update(kw.get("json", {})) - return _Resp() - - ex._request_with_retry = _req - ex.place_market_order("XAU_USD", "BUY", stop_loss=2384.0) - assert sent["order"]["positionFill"] == "OPEN_ONLY" - - -@test("daemon: _oanda_setting_int rejects a non-integer env value cleanly") -def _(): - os.environ["OANDA_LOT_SIZE"] = "one" - try: - oanda_daemon._oanda_setting_int("OANDA_LOT_SIZE", "lot_size", 1) - assert False, "must SystemExit on a non-integer env value" - except SystemExit as e: - assert "not an integer" in str(e) - finally: - os.environ.pop("OANDA_LOT_SIZE", None) - @test("GPU parity: ALL indicators + composite identical on every prefix") def _():