updated guide

CLAUDE.md

@@ -1,10 +1,225 @@
-# ForexBot2 — Dev Notes
+# ForexBot2 — Working Context for Claude
 
-## Notes
+Comprehensive context to avoid re-deriving the codebase in every session.
+Update this file when behavior, defaults, or file layout changes.
 
-- Free tier: 8 credits/min, 800 credits/day. Each full scan = ~21 credits.
-- Supported pairs (live data on free tier): EURUSD, GBPUSD, USDJPY, USDCHF,
-  AUDUSD, USDCAD, NZDUSD, EURGBP, EURJPY, GBPJPY, AUDJPY, EURCHF, CADJPY.
-- XAU/XAG and other unsupported symbols fall back to yfinance.
-- No synthetic data fallback — if both Twelve Data and yfinance fail, the
-  request errors out and no signal is generated.
+---
+
+## What it does
+
+Forex + gold signal bot. Two entry points:
+
+- **CLI**: `main.py --pairs EURUSD --timeframe H1 [--detail] [--min-grade C]`
+- **Telegram bot**: `telegram_bot.py` (commands: `/scan`, `/signal <pair>`, `/help`)
+
+Pulls OHLCV → runs 8 indicators (RSI, MACD, EMA, Stoch, ADX, BB, Ichimoku, Pivots) → multi-timeframe agreement check → news sentiment → grades A+/A/B/C/D/F.
+
+## Run commands (Windows PowerShell host)
+
+```powershell
+.venv/Scripts/python.exe main.py --pairs USDJPY --timeframe H1 --detail
+.venv/Scripts/python.exe telegram_bot.py
+```
+
+On Linux server: `venv/bin/python ...`. Server requires `pip install curl_cffi` (see SSL section).
+
+---
+
+## File roles
+
+| File | Role |
+|---|---|
+| `config.py` | Dataclasses for `PlatformConfig`, `TechnicalConfig`, `SignalConfig`, `NewsConfig`. Pair lists (`MAJOR_PAIRS`, `CROSS_PAIRS`, `SYMBOL_DISPLAY`, `PIP_VALUES`). `Timeframe` enum. |
+| `data_provider.py` | OHLCV fetch (Twelve Data primary, yfinance for XAU/XAG). Cache, rate limiting, validation. |
+| `engine.py` | `TechnicalEngine` — computes indicators, builds `TechnicalAnalysis` per (symbol, TF). |
+| `news_engine.py` | RSS feed sentiment + economic calendar event risk. |
+| `signal_aggregator.py` | The "brain". Iterates primary + confirmation TFs, runs MTF agreement check, applies news, builds `SignalReport` + `TradeSetup`. |
+| `main.py` | CLI entry. Parses args, builds `PlatformConfig`, runs `analyze_all_pairs`, prints colored table. |
+| `telegram_bot.py` | Telegram bot. Owns `SCAN_PAIRS`, `SCAN_TIMEFRAMES`, per-TF confirmation map. Renders cards. |
+| `LOGIC.md` | Engine decision flow + indicator weights. Kept up-to-date manually. |
+| `USAGE.md` | User-facing run guide. |
+
+---
+
+## Data flow
+
+```
+fetch_ohlcv (per symbol/TF) → tech_engine.analyze → TechnicalAnalysis
+                                       ↓
+            SignalAggregator.analyze_pair (primary + confirmation TFs)
+                                       ↓
+                        _check_mtf_agreement → SignalReport
+                                       ↓
+                  news_engine.get_pair_sentiment → final score + grade
+                                       ↓
+                      _build_trade_setup → TradeSetup (Entry/TP/SL)
+```
+
+`analyze_all_pairs` calls `prefetch_batch` once before iterating pairs — that's the optimization that makes the free Twelve Data tier viable.
+
+---
+
+## Where defaults live (and what overrides them)
+
+**This is the #1 footgun.** The dataclass defaults in `config.py` are NOT authoritative for actual runs — both CLI and Telegram bot rebuild the config and override `confirmation_timeframes` based on the chosen primary TF.
+
+| Default | Defined in | Overridden by |
+|---|---|---|
+| `primary_timeframe = H1` | [config.py:188](config.py#L188) | `main.py` (`--timeframe`), `telegram_bot.py` (per scan) |
+| `confirmation_timeframes = [M15, D1]` | [config.py:189-191](config.py#L189-L191) | `main.py:407-417` (`tf_confirmations` dict), `telegram_bot.py:53-56` (`_TF_CONFIRMATIONS`) |
+| `mtf_agreement_threshold = 2` | [config.py:143](config.py#L143) | _Not_ overridden — single source of truth. |
+| `weights` dict | [config.py:133-140](config.py#L133-L140) | _Not_ overridden. |
+| `cache_ttl_per_tf` | [config.py:207-216](config.py#L207-L216) | _Not_ overridden. |
+
+**When you change confirmation TFs, update all three places.** Otherwise CLI/Telegram silently keep the old values.
+
+---
+
+## Per-symbol TF overrides (XAU/XAG)
+
+Defined in `signal_aggregator.py`:
+
+```python
+_PER_SYMBOL_TFS = {
+    "XAUUSD=X": [Timeframe.M15, Timeframe.H1, Timeframe.H4],
+    "XAGUSD=X": [Timeframe.M15, Timeframe.H1, Timeframe.H4],
+}
+```
+
+`analyze_pair` calls `self._confirmation_tfs(symbol)` which returns the override (minus primary) when the symbol is in the map, else falls back to `config.confirmation_timeframes`. Forex pairs are unaffected.
+
+To add another symbol-specific TF set: add to `_PER_SYMBOL_TFS`. `analyze_all_pairs.prefetch_batch` already unions per-symbol TFs.
+
+---
+
+## Data source rules
+
+### Twelve Data (forex)
+- Free tier: **8 credits/min, 800 credits/day**.
+- Each `time_series` call = 1 credit per symbol. Batch endpoint accepts comma-separated symbols → ~1 credit per timeframe regardless of pair count (when used).
+- API key in [config.py:197](config.py#L197). Override via `TWELVE_DATA_API_KEY` env var.
+- Rate limiter is **module-level** (`_rate_lock`, `_rate_request_times`) so concurrent scans share the budget.
+- Supported on free tier: EURUSD, GBPUSD, USDJPY, USDCHF, AUDUSD, USDCAD, NZDUSD, EURGBP, EURJPY, GBPJPY, AUDJPY, EURCHF, CADJPY.
+- **Not supported on free tier**: XAUUSD, XAGUSD, anything in `_YF_FALLBACK_MAP`.
+
+### yfinance (gold/silver)
+- Routes via `curl_cffi.Session(impersonate="chrome", verify=False)` — see SSL section.
+- Mapping in [data_provider.py:34-37](data_provider.py#L34-L37): `XAUUSD=X → GC=F`, `XAGUSD=X → SI=F` (CME futures).
+- yfinance interval limits: 1m → 7d back; 5m/15m/30m/60m/1h → 60d back.
+- **No native H4 candle.** XAU/XAG H4 is resampled from H1 in `_resample_h1_to_h4` ([data_provider.py:291-300](data_provider.py#L291-L300)).
+- **D1 explicitly skipped** for XAU/XAG — futures daily data via yfinance is unreliable. Engine drops missing confirmation TFs cleanly via `except Exception` at `signal_aggregator.py:189-190`.
+
+### No synthetic fallback
+The old `_generate_synthetic_data()` (GBM with a `pair_params` dict that defaulted XAU to ~$1.00, causing trade setups like Entry 1.01) is **removed**. If both Twelve Data and yfinance fail, `fetch_ohlcv` raises `DataUnavailableError`. The engine returns grade F + warning, Telegram shows `❌ Data unavailable`.
+
+### Cache
+Per-symbol per-TF, in-memory dict. TTLs in `config.cache_ttl_per_tf`:
+- M1: 60s · M5: 150s · M15: 300s · M30: 600s · H1: 1800s · H4: 7200s · D1: 14400s · W1: 86400s
+
+Shared across `SignalAggregator` instances when the same `ForexDataProvider` is reused (see Telegram `_analyze_all_tfs`).
+
+---
+
+## Telegram /scan flow
+
+```python
+SCAN_PAIRS      = ["EURUSD=X", "USDJPY=X", "GBPUSD=X", "XAUUSD=X"]   # 4 pairs only
+SCAN_TIMEFRAMES = ["M15", "H1"]                                       # 2 cards per pair
+_TF_CONFIRMATIONS = {
+    "M15": [H1, D1],
+    "H1":  [M15, D1],
+}
+```
+
+`_analyze_all_tfs` ([telegram_bot.py:282-310](telegram_bot.py#L282-L310)):
+1. Builds the union of all unique TFs across `SCAN_TIMEFRAMES` + their confirmations.
+2. Single `prefetch_batch` for all of them on **one shared `ForexDataProvider` instance**.
+3. Spawns one scan per `SCAN_TIMEFRAMES` entry concurrently — each reuses the warm cache, so no extra API credits.
+
+Per /scan API cost: ~9 Twelve Data credits (3 forex pairs × 3 TFs, fetched once). XAU is free via yfinance.
+
+To add/remove pairs: edit `SCAN_PAIRS` ([telegram_bot.py:47](telegram_bot.py#L47)).
+To change which TFs get their own card: edit `SCAN_TIMEFRAMES` ([telegram_bot.py:43](telegram_bot.py#L43)).
+
+---
+
+## SSL — verify=False on server
+
+Live server sits behind a TLS-inspecting proxy that re-signs HTTPS with an untrusted root CA. Fix in [data_provider.py:27-37](data_provider.py#L27-L37):
+
+```python
+from curl_cffi import requests as _curl_cffi_requests
+_YF_SESSION = _curl_cffi_requests.Session(impersonate="chrome", verify=False)
+```
+
+Passed into `yf.download(..., session=_YF_SESSION)`. yfinance ≥ 0.2.40 **requires** a `curl_cffi` session — it rejects `requests.Session`.
+
+Trade-off: yfinance traffic is no longer authenticated against MITM. Acceptable because the data is public market quotes and an attacker can at worst inject bad prices. **Do not** reuse `_YF_SESSION` for any authenticated/private requests.
+
+---
+
+## Indicator weights & scoring
+
+In `config.SignalConfig.weights`:
+
+```python
+{"trend": 0.25, "momentum": 0.20, "volatility": 0.15,
+ "price_action": 0.15, "multi_timeframe": 0.15, "sentiment": 0.10}
+```
+
+Per-indicator weights live in `engine.py` (each indicator function returns its own contribution). Composite score is a weighted sum, capped to [-100, +100]. Grade derived from `abs(score) + confidence + mtf_agreement` in `_compute_signal_quality` (`signal_aggregator.py`).
+
+---
+
+## How to make common changes
+
+### Add a new tradeable pair (forex)
+1. Add `"XYZUSD=X"` to `MAJOR_PAIRS` or `CROSS_PAIRS` in `config.py`.
+2. Add display name to `SYMBOL_DISPLAY`.
+3. Verify it's on Twelve Data free tier (else add to `_YF_FALLBACK_MAP`).
+4. If Telegram /scan should include it, add to `SCAN_PAIRS` in `telegram_bot.py`.
+
+### Add a non-Twelve-Data symbol (commodity, index)
+1. Add to `_YF_FALLBACK_MAP` in `data_provider.py` with the Yahoo symbol.
+2. Add a `_PER_SYMBOL_TFS` entry in `signal_aggregator.py` (D1 unreliable, so drop it; H4 needs resample).
+3. Add to `SYMBOL_DISPLAY` and `PIP_VALUES` in `config.py`.
+
+### Change confirmation TFs for forex
+Edit **all three**:
+- `config.py:189-191` — engine default.
+- `main.py:407-417` — CLI per-primary map.
+- `telegram_bot.py:53-56` — Telegram per-primary map.
+
+### Change MTF agreement threshold
+Single edit: `config.py:143`. Affects all symbols. Note: with N TFs and threshold T, the rule is "T-of-N must agree" — keep T proportional to N if you change one.
+
+### Add a new indicator
+Add to `engine.py` → `TechnicalEngine.analyze`. Set its `weight` in the `Indicator` instance. The composite score will pick it up automatically.
+
+### Switch primary TF
+CLI: `--timeframe M15|M30|H1|H4|D1`.
+Telegram: edit `SCAN_TIMEFRAMES`.
+
+---
+
+## Common gotchas
+
+1. **The dataclass defaults in `config.py` are not the source of truth for `confirmation_timeframes`.** CLI and Telegram override them. See "Where defaults live" table.
+2. **Per-minute API rate limit hits hard with `--detail`** — one detailed run on one pair burns ~3 credits (3 TFs). Multiple back-to-back runs without ~60s gap will hit the 8/min ceiling.
+3. **Daily limit (800) is shared across the whole day** — heavy testing exhausts the budget; signal cache survives but new fetches fail until midnight UTC.
+4. **`curl_cffi` is a hard dep on the server** — without it, `_YF_SESSION` is `None`, yfinance falls back to its default backend, which fails on TLS-inspecting servers.
+5. **`__pycache__/` is committed** in this repo (legacy). Don't add it to `.gitignore` without coordinating; old `.pyc` will linger as untracked-modified.
+6. **Telegram `SCAN_PAIRS` is hardcoded to 4 pairs** — editing `MAJOR_PAIRS` does not change /scan. Update both if intentional.
+7. **`mtf_agreement_threshold = 2` means 2-of-N**, not 2-of-3 specifically. Adding a 4th TF to a pair makes "2-of-4" looser than "2-of-3"; consider per-symbol thresholds if this matters.
+8. **The `MTF: ✅ Aligned` line in Telegram cards comes from the engine's `report.mtf_agreement` boolean** ([signal_aggregator.py:223-233](signal_aggregator.py#L223-L233)) — not displayed per-TF. The full per-TF breakdown is only in the CLI `--detail` output.
+
+---
+
+## Recent architectural decisions (kept short)
+
+- **Removed synthetic data path** — explicit failure beats fake prices.
+- **XAU/XAG → yfinance only** — Twelve Data free tier doesn't carry them; routing them through TD wastes credits and used to fall through to bogus synthetic data.
+- **H1→H4 resample for gold** — yfinance has no native 4h, but H1 with 60d lookback gives ~360 H4 bars after resample, enough for indicator calculation.
+- **/scan limited to 4 pairs** — credit budget on free tier doesn't sustain 13-pair scans.
+- **Two SCAN_TIMEFRAMES (M15 + H1) with shared cache** — gives users two perspectives without doubling API cost (cache reuse).
+- **No SSL verify on yfinance** — server-side TLS inspection breaks cert chain; disabled because data is public quotes.