diff --git a/CLAUDE.md b/CLAUDE.md index fefc11a..1159a8e 100644 --- a/CLAUDE.md +++ b/CLAUDE.md @@ -1,10 +1,225 @@ -# ForexBot2 — Dev Notes +# ForexBot2 — Working Context for Claude -## Notes +Comprehensive context to avoid re-deriving the codebase in every session. +Update this file when behavior, defaults, or file layout changes. -- Free tier: 8 credits/min, 800 credits/day. Each full scan = ~21 credits. -- Supported pairs (live data on free tier): EURUSD, GBPUSD, USDJPY, USDCHF, - AUDUSD, USDCAD, NZDUSD, EURGBP, EURJPY, GBPJPY, AUDJPY, EURCHF, CADJPY. -- XAU/XAG and other unsupported symbols fall back to yfinance. -- No synthetic data fallback — if both Twelve Data and yfinance fail, the - request errors out and no signal is generated. +--- + +## What it does + +Forex + gold signal bot. Two entry points: + +- **CLI**: `main.py --pairs EURUSD --timeframe H1 [--detail] [--min-grade C]` +- **Telegram bot**: `telegram_bot.py` (commands: `/scan`, `/signal `, `/help`) + +Pulls OHLCV → runs 8 indicators (RSI, MACD, EMA, Stoch, ADX, BB, Ichimoku, Pivots) → multi-timeframe agreement check → news sentiment → grades A+/A/B/C/D/F. + +## Run commands (Windows PowerShell host) + +```powershell +.venv/Scripts/python.exe main.py --pairs USDJPY --timeframe H1 --detail +.venv/Scripts/python.exe telegram_bot.py +``` + +On Linux server: `venv/bin/python ...`. Server requires `pip install curl_cffi` (see SSL section). + +--- + +## File roles + +| File | Role | +|---|---| +| `config.py` | Dataclasses for `PlatformConfig`, `TechnicalConfig`, `SignalConfig`, `NewsConfig`. Pair lists (`MAJOR_PAIRS`, `CROSS_PAIRS`, `SYMBOL_DISPLAY`, `PIP_VALUES`). `Timeframe` enum. | +| `data_provider.py` | OHLCV fetch (Twelve Data primary, yfinance for XAU/XAG). Cache, rate limiting, validation. | +| `engine.py` | `TechnicalEngine` — computes indicators, builds `TechnicalAnalysis` per (symbol, TF). | +| `news_engine.py` | RSS feed sentiment + economic calendar event risk. | +| `signal_aggregator.py` | The "brain". Iterates primary + confirmation TFs, runs MTF agreement check, applies news, builds `SignalReport` + `TradeSetup`. | +| `main.py` | CLI entry. Parses args, builds `PlatformConfig`, runs `analyze_all_pairs`, prints colored table. | +| `telegram_bot.py` | Telegram bot. Owns `SCAN_PAIRS`, `SCAN_TIMEFRAMES`, per-TF confirmation map. Renders cards. | +| `LOGIC.md` | Engine decision flow + indicator weights. Kept up-to-date manually. | +| `USAGE.md` | User-facing run guide. | + +--- + +## Data flow + +``` +fetch_ohlcv (per symbol/TF) → tech_engine.analyze → TechnicalAnalysis + ↓ + SignalAggregator.analyze_pair (primary + confirmation TFs) + ↓ + _check_mtf_agreement → SignalReport + ↓ + news_engine.get_pair_sentiment → final score + grade + ↓ + _build_trade_setup → TradeSetup (Entry/TP/SL) +``` + +`analyze_all_pairs` calls `prefetch_batch` once before iterating pairs — that's the optimization that makes the free Twelve Data tier viable. + +--- + +## Where defaults live (and what overrides them) + +**This is the #1 footgun.** The dataclass defaults in `config.py` are NOT authoritative for actual runs — both CLI and Telegram bot rebuild the config and override `confirmation_timeframes` based on the chosen primary TF. + +| Default | Defined in | Overridden by | +|---|---|---| +| `primary_timeframe = H1` | [config.py:188](config.py#L188) | `main.py` (`--timeframe`), `telegram_bot.py` (per scan) | +| `confirmation_timeframes = [M15, D1]` | [config.py:189-191](config.py#L189-L191) | `main.py:407-417` (`tf_confirmations` dict), `telegram_bot.py:53-56` (`_TF_CONFIRMATIONS`) | +| `mtf_agreement_threshold = 2` | [config.py:143](config.py#L143) | _Not_ overridden — single source of truth. | +| `weights` dict | [config.py:133-140](config.py#L133-L140) | _Not_ overridden. | +| `cache_ttl_per_tf` | [config.py:207-216](config.py#L207-L216) | _Not_ overridden. | + +**When you change confirmation TFs, update all three places.** Otherwise CLI/Telegram silently keep the old values. + +--- + +## Per-symbol TF overrides (XAU/XAG) + +Defined in `signal_aggregator.py`: + +```python +_PER_SYMBOL_TFS = { + "XAUUSD=X": [Timeframe.M15, Timeframe.H1, Timeframe.H4], + "XAGUSD=X": [Timeframe.M15, Timeframe.H1, Timeframe.H4], +} +``` + +`analyze_pair` calls `self._confirmation_tfs(symbol)` which returns the override (minus primary) when the symbol is in the map, else falls back to `config.confirmation_timeframes`. Forex pairs are unaffected. + +To add another symbol-specific TF set: add to `_PER_SYMBOL_TFS`. `analyze_all_pairs.prefetch_batch` already unions per-symbol TFs. + +--- + +## Data source rules + +### Twelve Data (forex) +- Free tier: **8 credits/min, 800 credits/day**. +- Each `time_series` call = 1 credit per symbol. Batch endpoint accepts comma-separated symbols → ~1 credit per timeframe regardless of pair count (when used). +- API key in [config.py:197](config.py#L197). Override via `TWELVE_DATA_API_KEY` env var. +- Rate limiter is **module-level** (`_rate_lock`, `_rate_request_times`) so concurrent scans share the budget. +- Supported on free tier: EURUSD, GBPUSD, USDJPY, USDCHF, AUDUSD, USDCAD, NZDUSD, EURGBP, EURJPY, GBPJPY, AUDJPY, EURCHF, CADJPY. +- **Not supported on free tier**: XAUUSD, XAGUSD, anything in `_YF_FALLBACK_MAP`. + +### yfinance (gold/silver) +- Routes via `curl_cffi.Session(impersonate="chrome", verify=False)` — see SSL section. +- Mapping in [data_provider.py:34-37](data_provider.py#L34-L37): `XAUUSD=X → GC=F`, `XAGUSD=X → SI=F` (CME futures). +- yfinance interval limits: 1m → 7d back; 5m/15m/30m/60m/1h → 60d back. +- **No native H4 candle.** XAU/XAG H4 is resampled from H1 in `_resample_h1_to_h4` ([data_provider.py:291-300](data_provider.py#L291-L300)). +- **D1 explicitly skipped** for XAU/XAG — futures daily data via yfinance is unreliable. Engine drops missing confirmation TFs cleanly via `except Exception` at `signal_aggregator.py:189-190`. + +### No synthetic fallback +The old `_generate_synthetic_data()` (GBM with a `pair_params` dict that defaulted XAU to ~$1.00, causing trade setups like Entry 1.01) is **removed**. If both Twelve Data and yfinance fail, `fetch_ohlcv` raises `DataUnavailableError`. The engine returns grade F + warning, Telegram shows `❌ Data unavailable`. + +### Cache +Per-symbol per-TF, in-memory dict. TTLs in `config.cache_ttl_per_tf`: +- M1: 60s · M5: 150s · M15: 300s · M30: 600s · H1: 1800s · H4: 7200s · D1: 14400s · W1: 86400s + +Shared across `SignalAggregator` instances when the same `ForexDataProvider` is reused (see Telegram `_analyze_all_tfs`). + +--- + +## Telegram /scan flow + +```python +SCAN_PAIRS = ["EURUSD=X", "USDJPY=X", "GBPUSD=X", "XAUUSD=X"] # 4 pairs only +SCAN_TIMEFRAMES = ["M15", "H1"] # 2 cards per pair +_TF_CONFIRMATIONS = { + "M15": [H1, D1], + "H1": [M15, D1], +} +``` + +`_analyze_all_tfs` ([telegram_bot.py:282-310](telegram_bot.py#L282-L310)): +1. Builds the union of all unique TFs across `SCAN_TIMEFRAMES` + their confirmations. +2. Single `prefetch_batch` for all of them on **one shared `ForexDataProvider` instance**. +3. Spawns one scan per `SCAN_TIMEFRAMES` entry concurrently — each reuses the warm cache, so no extra API credits. + +Per /scan API cost: ~9 Twelve Data credits (3 forex pairs × 3 TFs, fetched once). XAU is free via yfinance. + +To add/remove pairs: edit `SCAN_PAIRS` ([telegram_bot.py:47](telegram_bot.py#L47)). +To change which TFs get their own card: edit `SCAN_TIMEFRAMES` ([telegram_bot.py:43](telegram_bot.py#L43)). + +--- + +## SSL — verify=False on server + +Live server sits behind a TLS-inspecting proxy that re-signs HTTPS with an untrusted root CA. Fix in [data_provider.py:27-37](data_provider.py#L27-L37): + +```python +from curl_cffi import requests as _curl_cffi_requests +_YF_SESSION = _curl_cffi_requests.Session(impersonate="chrome", verify=False) +``` + +Passed into `yf.download(..., session=_YF_SESSION)`. yfinance ≥ 0.2.40 **requires** a `curl_cffi` session — it rejects `requests.Session`. + +Trade-off: yfinance traffic is no longer authenticated against MITM. Acceptable because the data is public market quotes and an attacker can at worst inject bad prices. **Do not** reuse `_YF_SESSION` for any authenticated/private requests. + +--- + +## Indicator weights & scoring + +In `config.SignalConfig.weights`: + +```python +{"trend": 0.25, "momentum": 0.20, "volatility": 0.15, + "price_action": 0.15, "multi_timeframe": 0.15, "sentiment": 0.10} +``` + +Per-indicator weights live in `engine.py` (each indicator function returns its own contribution). Composite score is a weighted sum, capped to [-100, +100]. Grade derived from `abs(score) + confidence + mtf_agreement` in `_compute_signal_quality` (`signal_aggregator.py`). + +--- + +## How to make common changes + +### Add a new tradeable pair (forex) +1. Add `"XYZUSD=X"` to `MAJOR_PAIRS` or `CROSS_PAIRS` in `config.py`. +2. Add display name to `SYMBOL_DISPLAY`. +3. Verify it's on Twelve Data free tier (else add to `_YF_FALLBACK_MAP`). +4. If Telegram /scan should include it, add to `SCAN_PAIRS` in `telegram_bot.py`. + +### Add a non-Twelve-Data symbol (commodity, index) +1. Add to `_YF_FALLBACK_MAP` in `data_provider.py` with the Yahoo symbol. +2. Add a `_PER_SYMBOL_TFS` entry in `signal_aggregator.py` (D1 unreliable, so drop it; H4 needs resample). +3. Add to `SYMBOL_DISPLAY` and `PIP_VALUES` in `config.py`. + +### Change confirmation TFs for forex +Edit **all three**: +- `config.py:189-191` — engine default. +- `main.py:407-417` — CLI per-primary map. +- `telegram_bot.py:53-56` — Telegram per-primary map. + +### Change MTF agreement threshold +Single edit: `config.py:143`. Affects all symbols. Note: with N TFs and threshold T, the rule is "T-of-N must agree" — keep T proportional to N if you change one. + +### Add a new indicator +Add to `engine.py` → `TechnicalEngine.analyze`. Set its `weight` in the `Indicator` instance. The composite score will pick it up automatically. + +### Switch primary TF +CLI: `--timeframe M15|M30|H1|H4|D1`. +Telegram: edit `SCAN_TIMEFRAMES`. + +--- + +## Common gotchas + +1. **The dataclass defaults in `config.py` are not the source of truth for `confirmation_timeframes`.** CLI and Telegram override them. See "Where defaults live" table. +2. **Per-minute API rate limit hits hard with `--detail`** — one detailed run on one pair burns ~3 credits (3 TFs). Multiple back-to-back runs without ~60s gap will hit the 8/min ceiling. +3. **Daily limit (800) is shared across the whole day** — heavy testing exhausts the budget; signal cache survives but new fetches fail until midnight UTC. +4. **`curl_cffi` is a hard dep on the server** — without it, `_YF_SESSION` is `None`, yfinance falls back to its default backend, which fails on TLS-inspecting servers. +5. **`__pycache__/` is committed** in this repo (legacy). Don't add it to `.gitignore` without coordinating; old `.pyc` will linger as untracked-modified. +6. **Telegram `SCAN_PAIRS` is hardcoded to 4 pairs** — editing `MAJOR_PAIRS` does not change /scan. Update both if intentional. +7. **`mtf_agreement_threshold = 2` means 2-of-N**, not 2-of-3 specifically. Adding a 4th TF to a pair makes "2-of-4" looser than "2-of-3"; consider per-symbol thresholds if this matters. +8. **The `MTF: ✅ Aligned` line in Telegram cards comes from the engine's `report.mtf_agreement` boolean** ([signal_aggregator.py:223-233](signal_aggregator.py#L223-L233)) — not displayed per-TF. The full per-TF breakdown is only in the CLI `--detail` output. + +--- + +## Recent architectural decisions (kept short) + +- **Removed synthetic data path** — explicit failure beats fake prices. +- **XAU/XAG → yfinance only** — Twelve Data free tier doesn't carry them; routing them through TD wastes credits and used to fall through to bogus synthetic data. +- **H1→H4 resample for gold** — yfinance has no native 4h, but H1 with 60d lookback gives ~360 H4 bars after resample, enough for indicator calculation. +- **/scan limited to 4 pairs** — credit budget on free tier doesn't sustain 13-pair scans. +- **Two SCAN_TIMEFRAMES (M15 + H1) with shared cache** — gives users two perspectives without doubling API cost (cache reuse). +- **No SSL verify on yfinance** — server-side TLS inspection breaks cert chain; disabled because data is public quotes. diff --git a/__pycache__/config.cpython-314.pyc b/__pycache__/config.cpython-314.pyc index 886d755..0c47b37 100644 Binary files a/__pycache__/config.cpython-314.pyc and b/__pycache__/config.cpython-314.pyc differ diff --git a/__pycache__/data_provider.cpython-314.pyc b/__pycache__/data_provider.cpython-314.pyc index 61a6bc3..9ae04cf 100644 Binary files a/__pycache__/data_provider.cpython-314.pyc and b/__pycache__/data_provider.cpython-314.pyc differ diff --git a/__pycache__/main.cpython-314.pyc b/__pycache__/main.cpython-314.pyc new file mode 100644 index 0000000..5b764f1 Binary files /dev/null and b/__pycache__/main.cpython-314.pyc differ diff --git a/__pycache__/signal_aggregator.cpython-314.pyc b/__pycache__/signal_aggregator.cpython-314.pyc index 1770de2..d657f74 100644 Binary files a/__pycache__/signal_aggregator.cpython-314.pyc and b/__pycache__/signal_aggregator.cpython-314.pyc differ diff --git a/__pycache__/telegram_bot.cpython-314.pyc b/__pycache__/telegram_bot.cpython-314.pyc new file mode 100644 index 0000000..ac0422f Binary files /dev/null and b/__pycache__/telegram_bot.cpython-314.pyc differ