ForexBot2/CLAUDE.md

CLAUDE.md

This file provides guidance to Claude Code (claude.ai/code) when working with code in this repository.

What this is

Signal-intelligence platform for XAU/USD (gold — the only supported pair). It fetches OHLCV from the active data provider (Oanda v20 since 2026-07-28; Twelve Data and Tiingo remain registered fallbacks), scores 9 technical indicators per timeframe with regime-adaptive weights, checks multi-timeframe agreement and news risk, grades the signal (A+…F), and generates a full trade setup (entry, SL, TP1/TP2, trailing plan). Delivery: Telegram bot and CLI. A realism-focused backtester replays history through the exact same signal code.

Every numeric threshold in config.py and pair_signals/xauusd.py was backtested or is load-bearing for a backtested result. Validation evidence lives in the factory/module docstrings and in ~/.claude/projects/.../memory/.

Commands

python test_all.py                      # QA suite: 406 tests, 27 modules, no network (mocked)
python main.py --pair XAUUSD --timeframe H1 --detail   # CLI signal report (--json for full dump,
                                        #  --panel to run the advisory LLM agent panel on-demand)
python telegram_bot.py                  # the bot (refuses to boot without credentials)
python dashboard_publisher.py --once    # web dashboard: one scan+publish cycle to $DASHBOARD_OUT_DIR
                                        # (omit --once for the candle-close daemon; deploy/README-dashboard.md)

# Backtests
python backtest/gpu_backtest.py --pair XAUUSD --days 365
python backtest/gpu_backtest.py --pair XAUUSD --start 2025-01-01 --end 2025-06-01 \
    --primary-tf 15min --confirm-tfs 1h,4h
# key flags: --spread --quality A+,A,B,C --strategies --workers --max-hold-bars
#            --warmup-bars --cooldown-bars --no-cache --export-csv out.csv --detail
#            --fixed-rr-tp N (research-only: hard TP at N×risk, trail off; verdict
#            2026-08-05: NOT shipped — backtest/research/fixedtp_2026-08-05.md)

test_all.py is NOT pytest — it uses a custom @test("name") decorator that runs each test immediately at import; don't grep for def test_ or try pytest test_all.py. Run the whole file; it prints per-module results and exits non-zero on failure. There is no way to run a single test short of commenting modules out. Tests must stay green; every new gate/knob needs tests. Signal-log fixtures: use a WEEKDAY signal bar when a test exercises candle continuity — the outcome model treats weekend and 21:00 UTC gaps as scheduled, so Saturday-dated bars never produce a gap. _candles_json stores df.tail(14), so a test snapshot longer than 14 bars silently drops its head. When patching weekend logic in tests, patch providers.base._is_forex_weekend — but know there are THREE independent signal-path weekend implementations (providers/base.py, signal_aggregator.py, and an inline copy in backtest/gpu_backtest.py:_is_xau_blocked_by_calendar) plus the advisory-only continuity copy in advisory_outcome.py; patching one does not reach the others.

Python 3.12; a venv/ exists in-repo. Dependencies in requirements.txt (numpy, pandas, requests, twelvedata, python-telegram-bot ≥20, websockets).

Configuration & credentials

Precedence for every credential/deployment setting: env var → config.json → refuse to run (no hardcoded fallbacks). A gitignored repo-root .env is loaded at app_config import via os.environ.setdefault. config.json (gitignored; schema in config.json.example) holds secrets plus deployment settings: data_providers.active (the live-feed switch), data_providers.streaming (optional tick-stream provider for the level-breach monitor), scanning.pairs/timeframes, per-pair blocked_hours_utc, log_level. Key env vars: TELEGRAM_TOKEN, TELEGRAM_CHAT_ID, TELEGRAM_CHANNEL_ID, TWELVE_DATA_API_KEY, TIINGO_API_KEY, OANDA_ACCOUNT_ID/OANDA_API_TOKEN/OANDA_BASE_URL, DEEPSEEK_API_KEY (news LLM; the deepseek config.json block holds model/enabled/timeout/daily cap). The agent_panel config.json block (all-defaults = feature dark) controls the advisory LLM agent panel: enabled, per-surface telegram/dashboard toggles, min_quality, max_panels_per_day, reuse_ttl_seconds, plus the technical-payload enrichment knobs candles_enabled/candle_bars_primary/candle_bars_confirm. The dark-by-default market_outlook block controls the hourly advisory market brief: enabled, telegram, max_per_day, candles_bars; MARKET_OUTLOOK_ENABLED overrides its enabled setting.

Architecture

End-to-end pipeline (every scan)

providers/<active>.fetch_ohlcv()       per TF: single-flight fetch, drop forming candle, validate/clean, cache
                                       (only Twelve Data rate-throttles; Oanda has no client throttle)
  → engine.TechnicalEngine.analyze()   per TF: 9 scored indicators + ATR context → regime → composite score
  → signal_aggregator.analyze_pair()   live entry point (weekend/day filters, fetch, staleness)
      └─ analyze_from_dataframes()     replayable core: MTF agreement → news → bonuses/penalties →
                                       final signal + quality grade → gates → trade setup → reasons
  → SignalReport                       consumed by telegram_bot.py, main.py, backtester

Signal currency everywhere is SignalStrength (−2..+2). DataFrames are UTC tz-aware, oldest-first, capitalized OHLCV columns. Everything runs on closed bars only — the forming candle is dropped, and session/hour filters evaluate bar-CLOSE time (bar label + candle duration) identically in live and backtest.

Layers

• providers/ — one class per provider subclassing BaseDataProvider (providers/base.py), which owns everything the pipeline depends on: TTL+boundary cache, forming-candle drop, freshness check, validate/clean, paginated fetch_range (feeds the backtester from any provider), optional tick streaming. Subclasses implement only _fetch_raw + symbol/interval maps. Registry + make_data_provider(config, name=None) in providers/__init__.py; config.json data_providers.active selects the live candle feed and the optional data_providers.streaming scalar selects a different provider for the tick stream. A test enforces that subclasses don't override shared base methods (supports_streaming/iter_ticks are the sanctioned streaming override points). Tiingo streams mid-price ticks (wss://api.tiingo.com/fx) — do not re-add spread display from its free FX feed (two-tier indicative quotes, removed 2026-07-23); the no-spread rule extends to the WS stream.
• engine.py — TechnicalEngine: 9 scored indicators (ema_trend, rsi, macd, bollinger, adx, roc, keltner, divergence, candlestick) + ATR as context. Regime classification (trending_up/down, trending = mixed/direction-UNKNOWN, transitioning, ranging) drives per-indicator weight multipliers and regime-aware RSI reinterpretation. Compute order matters (ATR first; MACD always computed even when not a pillar — divergence needs the histogram). Private series ride on the result as _underscore attrs; scorers/formatters skip anything _-prefixed or weight-0.
• signal_aggregator.py — MTF agreement, diminishing-decay penalty system, exhaustion hard block, quality grading, then an ordered chain of post-grade gates (session, XAU SELL quality/vol/hour, regime blocks, M5 noise gates), then three setup generators: XAU ATR (A+/A), generic swing-based, plus the M5 pullback (XAU_PB) and chart-pattern (XAU_CP, chart_patterns.py) entry paths. The PB/CP generators run AFTER the gate chain and force quality to "B"; the CP path deliberately accepts grade-F composites and re-checks only session/day/big-candle (via warning prefixes) plus its own SELL/vol/RSI gates — it does NOT re-check exhaustion or the M5 min-score/MTF gates. This is validated behavior, not a bug (2026-08-04 audit: 226/586 trades in the shipped M5 12mo book are XAU_CP, +38.9R, 104 of them exhaustion-flagged and net-positive) — changing it is a rule-1 revalidation event. TradeSetup.strategy selects trailing params; trail params are stamped on the setup from the producing config so per-TF overrides reach formatters.
• pair_signals/ — per-pair config factories. Always use the factory (produce_signal / make_config), never bare PlatformConfig() — the factory applies the config.json overlay first, then the validated per-TF deltas (M5 hour-9 block, M15 trail arm 0.70, M15 SELL-hour block, M5 SELL-in-transitioning block, H1 all-shorts block (2026-09-03), etc.; evidence in the docstring). A test enforces factory ≡ inline-config equivalence. Confirmation map: M5→[M15,H1], M15→[M5,H1], H1→[M15,H4], H4→[H1,D1].
• signal_log.py / advisory_outcome.py — SQLite signal-history logger (stdlib only). Hooked into SignalAggregator.analyze_pair only (the live entry point — the backtester never writes here, preserving parity), so every live scan from every surface (Telegram telegram, CLI cli, dashboard dashboard) lands one row in logs/signals.db: signal/grade/price, MTF + trade-setup + reasons JSON, full per-TF indicator snapshots, the last 14 closed OHLCV candles per fetched TF, and the news at scan time (event risk + calendar events + scored headlines, re-fetched via the NewsEngine class-level caches). Config.json signal_log block (enabled, path, log_blocked_scans; env SIGNAL_LOG_ENABLED/SIGNAL_LOG_PATH/SIGNAL_LOG_BLOCKED_SCANS). Fail-safe by contract — a logging failure can never break a scan. Panel fields arrive post-hoc: log_signal stashes the rowid and agent_panel.attach_panel writes verdict + creation/cache provenance + payload hash onto the same row. Advisory outcomes (2026-09): EVERY row with a trade setup is seeded at log time (not only panel rows); same-setup re-logs within 60 min from any surface (setup_key = tf|direction|entry|stop|strategy) are DUPLICATE rows pointing at the canonical row via advisory_event_id, so sum advisory_r_multiple over RESOLVED rows only. The versioned model advances from later stored closed-candle snapshots (next-open reanchoring, intrabar peak with close-only ratchet, SL-first, scheduled weekend/21:00-UTC closures allowed, 300-bar timeout, $0.30/$0.65 views); when a snapshot no longer overlaps the outcome's last bar it rebuilds the stream from every stored snapshot of that TF before giving up. It is a cleaned-mid evidence model, not execution P&L. Post-gate truth columns: gate_reason (the warning that removed the setup), actionable (grade ≥ C AND setup — the action text is written BEFORE the gate chain and can say "STRONG SELL — High-quality setup" on a D row), cleaned_warnings_json (what the XAU_CP acceptance stripped), bar_range_median (= the misnamed spread column: median High-Low, NOT a broker spread), scan_kind (blocked weekend/day early returns carry no news/candles), config_fingerprint (gate config hash — surfaces config drift between processes). News is content-addressed: headlines_json/events_json are interned into news_snapshots (they were 73% of a 667 MB DB); read through the signals_with_news view or signal_log.read_news. Readers must open with mode=ro, never immutable=1 (blind to WAL rows). Tools: analyze_signal_log.py (--stored-candles DB-only replay with event/opportunity/one-position books + persona scorecard, --parity telegram-vs-dashboard divergence report), scripts/backfill_signal_outcomes.py (seed/dedup/advance legacy rows), scripts/migrate_signal_log_news.py (intern legacy blobs; --vacuum with both writers stopped).
• news_engine.py — Forex Factory calendar (MyFXBook RSS fallback) + RSS keyword sentiment with gold-inverse keywords. Class-level caches (fresh engine per scan). Optional DeepSeek enrichment via llm_client.py (requests-only, no SDK): batched headline sentiment overriding keyword scores, gold-direction on release surprises, and a market-summary blurb (SignalReport.news_summary) — display/warning-only, keyword fallback on any failure, inert without a key; prompts are static module constants (DeepSeek prefix-cache pricing) and requests send thinking: disabled (V4-Flash reasoning otherwise blows the timeout). By XAU-tuned default, event risk and sentiment warn but do not gate signals (event_risk_caps_signals=False, sentiment_affects_signals=False) — the LLM does not change that.
• agent_panel.py — ai-hedge-fund-style advisory panel: 4 DeepSeek persona calls (macro/technical/risk/contrarian, concurrent on a panel-private pool) + 1 synthesizer producing an ADVISORY-ONLY consensus (AGREE/NEUTRAL/OPPOSE + confidence + rationale) written onto SignalReport.panel_* fields. Attached post-hoc by consumers (telegram_bot alert path, main --panel, dashboard_publisher) — never runs inside analyze_from_dataframes (backtest parity, test-locked) and no gate reads its fields. Persona payloads are role-filtered: macro gets high-impact events plus only code-scored directional news; technical gets strategy/pattern, primary indicators, MTF, levels, and ATR-normalized candles; risk gets setup strategy, strategy cost assumption, trailing semantics, volatility/events/warnings (never the median candle range as spread); contrarian gets directional news, setup, timestamped latest-closed price, indicators, reasons, and warnings. Spend is bounded by exact cache, full-evidence-identity TTL reuse, and max_panels_per_day; changed evidence never inherits a direction-only cached verdict (test-locked — which also means TTL reuse effectively never fires live because price/candles change every bar; the daily cap is the real spend bound). Verdict provenance logs creation time, MISS/EXACT_HIT/TTL_REUSE, and a SHA-256 payload hash. Prompts forbid outside knowledge and outputs remain short. OPPOSE guard (2026-09): the 3-week signal-log scorecard showed the contrarian/risk/technical personas oppose almost every trend entry (chair said OPPOSE 73% of the time; opposed trades made +30R, PF 2.6) while the macro persona was the only informative voice (macro=support → 88% win). _apply_oppose_guard therefore downgrades a chair OPPOSE to NEUTRAL unless macro opposes, or technical opposes while macro is not supportive; the synthesizer prompt states the same rule. Re-score with analyze_signal_log.py --stored-candles (persona scorecard) before changing it again. Verdict source (2026-09-03, owner decision): all four personas still run and are logged (panel_takes_json), but agent_panel.verdict_source defaults to "macro" — the alert verdict is the macro persona's stance (support/neutral/oppose → AGREE/NEUTRAL/OPPOSE) and the synthesizer call is skipped (4 LLM calls per panel instead of 5); "chair" restores the four-voice synthesizer + OPPOSE guard. Re-score personas monthly with analyze_signal_log.py --stored-candles; weights are not to be tuned before ~300 graded outcomes with a held-out half.
• market_outlook.py — hourly DeepSeek market brief for XAU/USD, using ATR-normalized M15/H1/H4 candles, visible indicator states, levels, news, and recent signal history (read mode=ro). It returns a written ADVISORY-ONLY bias/setup-forming assessment and persists it independently in outlooks within logs/signals.db together with the news, indicator/regime/level, and candle evidence sent to the model. It may be sent post-scan by Telegram or requested with main.py --outlook; it never imports or runs inside the signal path. Hourly dedup is by UTC hour BUCKET (the old now − last < 3600 rule dropped every other hour because the H1 loop fires at boundary+10 s+scan time) plus its own UTC-day cap (default 12); every silent early return logs market outlook: skipped (...) and an empty LLM reply logs a warning — a dark feature must be visible in logs/telegram.log. Provider/news failures leave a safe partial brief while LLM/database failures degrade to no outlook.
• telegram_bot.py — owner-locked bot + subscriber channel; candle-boundary auto-scan loops (M5/M15/H1/H4, boundary+10 s), alert quality threshold + fingerprint dedup (committed only after a successful send; the fingerprint is signal+direction — grade is NOT part of it, a grade UPGRADE re-alerts, an A↔B flicker does not), 📰 LLM market-context line on alerts when available, WebSocket level-breach monitor fed by the streaming provider (data_providers.streaming, falling back to the active provider; auto-disabled when the resolved provider can't stream or has no API key — Oanda does not stream, Tiingo does). The breach handler reuses this bar's M15 report (_cached_or_scan, keyed by closed-bar epoch) instead of rescanning — it used to log 3.5 M15 rows per candle. Rotating log: logs/telegram.log, 3-h rotation × 240 backups.
• dashboard_publisher.py / static/dashboard.html — web dashboard as a static publisher (live LAMP server): standalone daemon scans at candle closes via produce_signal and atomically writes snapshot.json/news.json/dashboard.html to $DASHBOARD_OUT_DIR (Apache serves them; page is public by owner decision). Hard independence rule: never imports, and is never imported by, telegram_bot — confirmation map and boundary math are deliberate local copies, so a web-side failure can only mean a stale page. Due-TF classification uses the RAW wake time (flooring wake−buffer skips every H1/H4 scan — 2026-08-02 bug). News panel reads NewsEngine.get_fundamental_context() plus the additive get_week_events() accessor (display-only; no signal path calls it); as a separate process it has its own news caches and DeepSeek daily cap. News/calendar rows are clickable (detail dialog); headlines carry id = sha1(headline) in news.json, and exactly one on-demand endpoint exists: static/news_summary.php (installed by install_static), which serves a DeepSeek gold-impact summary per headline — whitelist-gated against news.json ids, cache-forever in ai/, own fail-closed daily cap (150/day, file-based, separate from the publisher's), key at /etc/gold-dashboard/deepseek.key, never reads repo config. Deployment checklist: deploy/README-dashboard.md.
• backtest/gpu_backtest.py — bar-by-bar replay through the real analyze_from_dataframes (news off). Realism model (do not regress): next-bar-open entries re-anchored to the raw price path, round-trip --spread charged per trade, trade resolution on RAW bars (cleaning applies only to signal-side data), same-bar SL+TP tie → SL first, gap fills at bar open, trail ratchet at bar close only. Indicators are precomputed once per TF (CuPy GPU or NumPy fallback — check the Backend: log line); replay fans out over worker processes and the 5-bar cooldown is applied in an ordered pass afterwards, so parallel and sequential books are identical. Parquet cache in backtest/cache/ keyed by provider/symbol/TF/window. backtest/*.csv are gitignored pinned trade books from past campaigns — used for manual before/after comparison. backtest/instructions_for_backtest.txt lists what every backtest report must record.

Rules for making changes (read before editing signal logic)

1. Never change a tuned value without re-validation. Acceptance rule: positive total R in BOTH train and validation halves of a ≥12-month window, PF ≥ 1.3, at realistic cost (XAU $0.65 round trip headline; retain $0.30 for TD-era cross-book comparability), trailing exits, next-open fills. round_trip_cost stays $0.30 because it also drives the live TP spread gate; changing it is a separate rule-1 event. Marginal edges must additionally pass the 2024-06→2025-03 out-of-sample window — within-year splits alone have approved configs that later failed OOS.
2. Per-timeframe overrides live in the pair factory, never in global defaults — the same knob has opposite effects on different TFs (proven repeatedly: tp_floor_rr, hour-9 block, trail params). Document validation evidence in the factory docstring.
3. Pair isolation is a hard invariant. XAU gates in the aggregator are display-name-gated so a future pair port cannot inherit gold-calibrated logic by accident.
4. Live == backtest parity. If you add a live gate, add it to the replay path in the same commit; all time filters evaluate bar-close time in both paths.
5. ROC stays weighted even though it's correlated with MACD/Keltner/EMA — zeroing it backtested 13R worse; composite thresholds are calibrated with it.
6. Backtester realism knobs are results-critical (raw-path resolution, spread charging, next-open entry, SL-first ties, close-only trail). Touching them invalidates comparability with all pinned books.
7. The forming candle must stay dropped — live signals repaint otherwise.
8. Don't add config fields nothing reads; don't disable indicators in production paths (disabled_indicators is for experiments only).
9. EUR/USD was removed 2026-07-16 after a no-edge verdict (cost-to-volatility; last commit with EUR support: 7d3fd95). Do not re-add without new data. BTC/ETH work lives on unmerged branches only.

2026-08-04 QA audit — owner decisions ledger

• Weekend calendar: FIXED (owner-approved 2026-08-04). All three signal-path _is_forex_weekend copies are NY-anchored (Fri 17:00 → Sun 18:00 ET; during US DST that is Fri 21:00 UTC — measured on 5yr of Oanda bars). Kills the weekly false-"STALE DATA/feed lagging" hours. Book impact at the audit HEAD: M15 byte-identical; M5 12mo dropped exactly ONE trade — a phantom whose signal fired on the final summer-Friday bar (close 21:00 UTC) and "filled" at Sunday's reopen 49 h later (−1.08R, live-impossible), leaving 555 trades / +98.64R / PF≈1.50 @ $0.65 (was 556/+97.56/1.48). The current-HEAD canonical reference is recorded below. A Module 21 test sweeps those copies plus the advisory outcome's continuity copy hourly across both DST transitions.
• Spread in exit-trigger detection (rule 6): owner DECLINED 2026-08-04. Backtest stop triggers stay on raw mid High/Low; the measured optimism (~4–6% of total R: −3.55R @ $0.30, −4.24R @ $0.65 on the M15 12mo book) is accepted and books stay comparable. Re-open only as a conscious re-pin campaign.
• FB2_EMA_LO: HARDENED 2026-08-04 (delegated). Research override for the EMA200-proximity band, default = tuned 0.5; an unparseable value now degrades to the default with a warning instead of crashing every scan. Documented here — it is a knob that exists.
• H4 seeding divergence: parked. Live H4 fetches only 380 bars vs the backtest's unbounded history — ~2% of sampled H4 bars flip an indicator signal (composite Δ up to 12.3). M5/M15/H1 clean at their real fetch depths. Fold into any future H4 campaign; H4 remains "unproven".

Current state notes (2026-08-14)

• Oanda v20 is the live feed (data_providers.active: "oanda"). The 2026-08-04 re-pin made the 27 backtest/oanda_*.csv books the baseline; all TD-era books are superseded. Current-HEAD M5 canonical book: backtest/oanda_m5_12mo_065_nopb_20260814.csv, +106.82R PF 1.51 at $0.65 (both halves positive; PF 1.46 / 1.55). M15 reference: +73.93R PF 1.53. See backtest/research/repin_verification_2026-08-14.md.
• Cost reporting is dual: $0.65 is the measured-Oanda realistic headline ($0.65-$0.72 round trip), while $0.30 remains for cross-book comparability and as the live TP spread-gate input. Raising round_trip_cost from $0.30 would move the TP floor from about $9 to $19.50 and requires full rule-1 revalidation.
• Twelve Data is signal-incompatible with Oanda (Phase 1: about 19% M5 bar-direction disagreement and a systematic ~$1 high bias; see backtest/research/feed_divergence_2026-08-14.md). On an Oanda candle outage, stop scanning rather than switching to TD candles.
• Tiingo is streaming-only (data_providers.streaming: "tiingo"), not a designated candle fallback. Oanda-Tiingo signal parity is documented in backtest/research/signal_parity_2026-08-14.md; Phase 4's two-week live shadow scan remains a manual operational step.
• Human-facing docs in docs/ (user_guide.html, beginner_guide.html + PDF builds) must be kept in sync when signal behavior changes.
• 2026-09-03 signal-log campaign shipped two new per-TF gates (backtest/research/signal_log_campaign_2026-09.md): M5 sell_blocked_regimes=["transitioning"] and H1 sell_min_quality=[] (all H1 shorts blocked). Current-HEAD M5 canonical book is now backtest/oanda_m5_12mo_065_nopb_20260903.csv (557 / +112.21R / PF 1.56, halves +55.9/+56.4); H1 reference backtest/oanda_h1_12mo_065_20260903.csv (91 / +22.2R / PF 1.79; 5yr +49.1R PF 1.46; OOS +16.7R PF 1.74).
• Auto-execution (oanda_daemon.py / oanda_executor.py) was removed 2026-09-03 — the owner is not auto-trading; last commit containing it is 31002cb. Book-parity rules that lived in its docstring (no broker TP, 5-bar cooldown, once-per-bar trail) remain the backtester's model.