ForexBot2/CLAUDE.md

# ForexBot2 — Working Context for Claude

Comprehensive context to avoid re-deriving the codebase in every session.
Update this file when behavior, defaults, or file layout changes.

---

## What it does

Forex + gold signal bot. Two entry points:

- **CLI**: `main.py --pairs EURUSD --timeframe H1 [--detail] [--min-grade C]`
- **Telegram bot**: `telegram_bot.py` (commands: `/scan`, `/signal <pair>`, `/help`)

Pulls OHLCV → runs 8 indicators (RSI, MACD, EMA, Stoch, ADX, BB, Ichimoku, Pivots) → multi-timeframe agreement check → news sentiment → grades A+/A/B/C/D/F.

## Run commands (Windows PowerShell host)

```powershell
.venv/Scripts/python.exe main.py --pairs USDJPY --timeframe H1 --detail
.venv/Scripts/python.exe telegram_bot.py
```

On Linux server: `venv/bin/python ...`. Server requires `pip install curl_cffi` (see SSL section).

---

## File roles

| File | Role |
|---|---|
| `config.py` | Dataclasses for `PlatformConfig`, `TechnicalConfig`, `SignalConfig`, `NewsConfig`. Pair lists (`MAJOR_PAIRS`, `CROSS_PAIRS`, `SYMBOL_DISPLAY`, `PIP_VALUES`). `Timeframe` enum. |
| `data_provider.py` | OHLCV fetch (Twelve Data primary, yfinance for XAU/XAG). Cache, rate limiting, validation. |
| `engine.py` | `TechnicalEngine` — computes indicators, builds `TechnicalAnalysis` per (symbol, TF). |
| `news_engine.py` | RSS feed sentiment + economic calendar event risk. |
| `signal_aggregator.py` | The "brain". Iterates primary + confirmation TFs, runs MTF agreement check, applies news, builds `SignalReport` + `TradeSetup`. |
| `main.py` | CLI entry. Parses args, builds `PlatformConfig`, runs `analyze_all_pairs`, prints colored table. |
| `telegram_bot.py` | Telegram bot. Owns `SCAN_PAIRS`, `SCAN_TIMEFRAMES`, per-TF confirmation map. Renders cards. |
| `LOGIC.md` | Engine decision flow + indicator weights. Kept up-to-date manually. |
| `USAGE.md` | User-facing run guide. |

---

## Data flow

```
fetch_ohlcv (per symbol/TF) → tech_engine.analyze → TechnicalAnalysis
                                       ↓
            SignalAggregator.analyze_pair (primary + confirmation TFs)
                                       ↓
                        _check_mtf_agreement → SignalReport
                                       ↓
                  news_engine.get_pair_sentiment → final score + grade
                                       ↓
                      _build_trade_setup → TradeSetup (Entry/TP/SL)
```

`analyze_all_pairs` calls `prefetch_batch` once before iterating pairs — that's the optimization that makes the free Twelve Data tier viable.

---

## Where defaults live (and what overrides them)

**This is the #1 footgun.** The dataclass defaults in `config.py` are NOT authoritative for actual runs — both CLI and Telegram bot rebuild the config and override `confirmation_timeframes` based on the chosen primary TF.

| Default | Defined in | Overridden by |
|---|---|---|
| `primary_timeframe = H1` | [config.py:188](config.py#L188) | `main.py` (`--timeframe`), `telegram_bot.py` (per scan) |
| `confirmation_timeframes = [M15, D1]` | [config.py:189-191](config.py#L189-L191) | `main.py:407-417` (`tf_confirmations` dict), `telegram_bot.py:53-56` (`_TF_CONFIRMATIONS`) |
| `mtf_agreement_threshold = 2` | [config.py:143](config.py#L143) | _Not_ overridden — single source of truth. |
| `weights` dict | [config.py:133-140](config.py#L133-L140) | _Not_ overridden. |
| `cache_ttl_per_tf` | [config.py:207-216](config.py#L207-L216) | _Not_ overridden. |

**When you change confirmation TFs, update all three places.** Otherwise CLI/Telegram silently keep the old values.

---

## Per-symbol TF overrides (XAU/XAG)

Defined in `signal_aggregator.py`:

```python
_PER_SYMBOL_TFS = {
    "XAUUSD=X": [Timeframe.M15, Timeframe.H1, Timeframe.H4],
    "XAGUSD=X": [Timeframe.M15, Timeframe.H1, Timeframe.H4],
}
```

`analyze_pair` calls `self._confirmation_tfs(symbol)` which returns the override (minus primary) when the symbol is in the map, else falls back to `config.confirmation_timeframes`. Forex pairs are unaffected.

To add another symbol-specific TF set: add to `_PER_SYMBOL_TFS`. `analyze_all_pairs.prefetch_batch` already unions per-symbol TFs.

---

## Data source rules

### Twelve Data (forex)
- Free tier: **8 credits/min, 800 credits/day**.
- Each `time_series` call = 1 credit per symbol. Batch endpoint accepts comma-separated symbols → ~1 credit per timeframe regardless of pair count (when used).
- API key in [config.py:197](config.py#L197). Override via `TWELVE_DATA_API_KEY` env var.
- Rate limiter is **module-level** (`_rate_lock`, `_rate_request_times`) so concurrent scans share the budget.
- Supported on free tier: EURUSD, GBPUSD, USDJPY, USDCHF, AUDUSD, USDCAD, NZDUSD, EURGBP, EURJPY, GBPJPY, AUDJPY, EURCHF, CADJPY.
- **Not supported on free tier**: XAUUSD, XAGUSD, anything in `_YF_FALLBACK_MAP`.

### yfinance (gold/silver)
- Routes via `curl_cffi.Session(impersonate="chrome", verify=False)` — see SSL section.
- Mapping in [data_provider.py:34-37](data_provider.py#L34-L37): `XAUUSD=X → GC=F`, `XAGUSD=X → SI=F` (CME futures).
- yfinance interval limits: 1m → 7d back; 5m/15m/30m/60m/1h → 60d back.
- **No native H4 candle.** XAU/XAG H4 is resampled from H1 in `_resample_h1_to_h4` ([data_provider.py:291-300](data_provider.py#L291-L300)).
- **D1 explicitly skipped** for XAU/XAG — futures daily data via yfinance is unreliable. Engine drops missing confirmation TFs cleanly via `except Exception` at `signal_aggregator.py:189-190`.

### No synthetic fallback
The old `_generate_synthetic_data()` (GBM with a `pair_params` dict that defaulted XAU to ~$1.00, causing trade setups like Entry 1.01) is **removed**. If both Twelve Data and yfinance fail, `fetch_ohlcv` raises `DataUnavailableError`. The engine returns grade F + warning, Telegram shows `❌ Data unavailable`.

### Cache
Per-symbol per-TF, in-memory dict. TTLs in `config.cache_ttl_per_tf`:
- M1: 60s · M5: 150s · M15: 300s · M30: 600s · H1: 1800s · H4: 7200s · D1: 14400s · W1: 86400s

Shared across `SignalAggregator` instances when the same `ForexDataProvider` is reused (see Telegram `_analyze_all_tfs`).

---

## Telegram /scan flow

```python
SCAN_PAIRS      = ["EURUSD=X", "USDJPY=X", "GBPUSD=X", "XAUUSD=X"]   # 4 pairs only
SCAN_TIMEFRAMES = ["M15", "H1"]                                       # 2 cards per pair
_TF_CONFIRMATIONS = {
    "M15": [H1, D1],
    "H1":  [M15, D1],
}
```

`_analyze_all_tfs` ([telegram_bot.py:282-310](telegram_bot.py#L282-L310)):
1. Builds the union of all unique TFs across `SCAN_TIMEFRAMES` + their confirmations.
2. Single `prefetch_batch` for all of them on **one shared `ForexDataProvider` instance**.
3. Spawns one scan per `SCAN_TIMEFRAMES` entry concurrently — each reuses the warm cache, so no extra API credits.

Per /scan API cost: ~9 Twelve Data credits (3 forex pairs × 3 TFs, fetched once). XAU is free via yfinance.

To add/remove pairs: edit `SCAN_PAIRS` ([telegram_bot.py:47](telegram_bot.py#L47)).
To change which TFs get their own card: edit `SCAN_TIMEFRAMES` ([telegram_bot.py:43](telegram_bot.py#L43)).

---

## SSL — verify=False on server

Live server sits behind a TLS-inspecting proxy that re-signs HTTPS with an untrusted root CA. Fix in [data_provider.py:27-37](data_provider.py#L27-L37):

```python
from curl_cffi import requests as _curl_cffi_requests
_YF_SESSION = _curl_cffi_requests.Session(impersonate="chrome", verify=False)
```

Passed into `yf.download(..., session=_YF_SESSION)`. yfinance ≥ 0.2.40 **requires** a `curl_cffi` session — it rejects `requests.Session`.

Trade-off: yfinance traffic is no longer authenticated against MITM. Acceptable because the data is public market quotes and an attacker can at worst inject bad prices. **Do not** reuse `_YF_SESSION` for any authenticated/private requests.

---

## Indicator weights & scoring

In `config.SignalConfig.weights`:

```python
{"trend": 0.25, "momentum": 0.20, "volatility": 0.15,
 "price_action": 0.15, "multi_timeframe": 0.15, "sentiment": 0.10}
```

Per-indicator weights live in `engine.py` (each indicator function returns its own contribution). Composite score is a weighted sum, capped to [-100, +100]. Grade derived from `abs(score) + confidence + mtf_agreement` in `_compute_signal_quality` (`signal_aggregator.py`).

---

## How to make common changes

### Add a new tradeable pair (forex)
1. Add `"XYZUSD=X"` to `MAJOR_PAIRS` or `CROSS_PAIRS` in `config.py`.
2. Add display name to `SYMBOL_DISPLAY`.
3. Verify it's on Twelve Data free tier (else add to `_YF_FALLBACK_MAP`).
4. If Telegram /scan should include it, add to `SCAN_PAIRS` in `telegram_bot.py`.

### Add a non-Twelve-Data symbol (commodity, index)
1. Add to `_YF_FALLBACK_MAP` in `data_provider.py` with the Yahoo symbol.
2. Add a `_PER_SYMBOL_TFS` entry in `signal_aggregator.py` (D1 unreliable, so drop it; H4 needs resample).
3. Add to `SYMBOL_DISPLAY` and `PIP_VALUES` in `config.py`.

### Change confirmation TFs for forex
Edit **all three**:
- `config.py:189-191` — engine default.
- `main.py:407-417` — CLI per-primary map.
- `telegram_bot.py:53-56` — Telegram per-primary map.

### Change MTF agreement threshold
Single edit: `config.py:143`. Affects all symbols. Note: with N TFs and threshold T, the rule is "T-of-N must agree" — keep T proportional to N if you change one.

### Add a new indicator
Add to `engine.py` → `TechnicalEngine.analyze`. Set its `weight` in the `Indicator` instance. The composite score will pick it up automatically.

### Switch primary TF
CLI: `--timeframe M15|M30|H1|H4|D1`.
Telegram: edit `SCAN_TIMEFRAMES`.

---

## Common gotchas

1. **The dataclass defaults in `config.py` are not the source of truth for `confirmation_timeframes`.** CLI and Telegram override them. See "Where defaults live" table.
2. **Per-minute API rate limit hits hard with `--detail`** — one detailed run on one pair burns ~3 credits (3 TFs). Multiple back-to-back runs without ~60s gap will hit the 8/min ceiling.
3. **Daily limit (800) is shared across the whole day** — heavy testing exhausts the budget; signal cache survives but new fetches fail until midnight UTC.
4. **`curl_cffi` is a hard dep on the server** — without it, `_YF_SESSION` is `None`, yfinance falls back to its default backend, which fails on TLS-inspecting servers.
5. **`__pycache__/` is committed** in this repo (legacy). Don't add it to `.gitignore` without coordinating; old `.pyc` will linger as untracked-modified.
6. **Telegram `SCAN_PAIRS` is hardcoded to 4 pairs** — editing `MAJOR_PAIRS` does not change /scan. Update both if intentional.
7. **`mtf_agreement_threshold = 2` means 2-of-N**, not 2-of-3 specifically. Adding a 4th TF to a pair makes "2-of-4" looser than "2-of-3"; consider per-symbol thresholds if this matters.
8. **The `MTF: ✅ Aligned` line in Telegram cards comes from the engine's `report.mtf_agreement` boolean** ([signal_aggregator.py:223-233](signal_aggregator.py#L223-L233)) — not displayed per-TF. The full per-TF breakdown is only in the CLI `--detail` output.

---

## News sentiment — Google Gemini (live)

`news_engine` uses **Gemini `gemini-2.5-flash-lite`** for headline sentiment, replacing the keyword lexicon (which still exists as a fallback).

- API key loaded from `.env` (`GEMINI_API_KEY=...`) via `python-dotenv`. `.env` is gitignored.
- One **batched** API call per `/scan` — `_score_headlines_batch` sends all uncached titles in a single prompt, returns a JSON array of `{id, score}` ([news_engine.py:355-410](news_engine.py#L355-L410)).
- In-memory cache `_sentiment_cache` keyed by `hash(headline)` — repeat calls for the same headline are free.
- Free-tier quota on `gemini-2.5-flash-lite`: **1000 RPD** (vs 20 on `gemini-2.5-flash` — do not use the non-lite variant).
- If Gemini fails (timeout, 429, parse error), falls back per-headline to `_fallback_keyword_sentiment` so signals still grade.

Observed quality lift: ~88% of fetched headlines now get non-zero sentiment (vs ~10% with keywords). Headlines like "RBA signals pause" or "inflation cools below forecast" score correctly bearish where keywords missed them entirely.

To swap models: change `NewsEngine.GEMINI_MODEL` ([news_engine.py:118](news_engine.py#L118)). Use the new `google-genai` SDK (`from google import genai; genai.Client(api_key=...)`); the older `google-generativeai` package is replaced.

---

## Recent architectural decisions (kept short)

- **Removed synthetic data path** — explicit failure beats fake prices.
- **XAU/XAG → yfinance only** — Twelve Data free tier doesn't carry them; routing them through TD wastes credits and used to fall through to bogus synthetic data.
- **H1→H4 resample for gold** — yfinance has no native 4h, but H1 with 60d lookback gives ~360 H4 bars after resample, enough for indicator calculation.
- **/scan limited to 4 pairs** — credit budget on free tier doesn't sustain 13-pair scans.
- **Two SCAN_TIMEFRAMES (M15 + H1) with shared cache** — gives users two perspectives without doubling API cost (cache reuse).
- **No SSL verify on yfinance** — server-side TLS inspection breaks cert chain; disabled because data is public quotes.