ForexBot2/CLAUDE.md

ForexBot2 — Working Context for Claude

Comprehensive context to avoid re-deriving the codebase in every session. Update this file when behavior, defaults, or file layout changes.

---

What it does

Forex + gold signal bot. Two entry points:

• CLI: main.py --pairs EURUSD --timeframe H1 [--detail] [--min-grade C]
• Telegram bot: telegram_bot.py (commands: /scan, /signal <pair>, /help)

Pulls OHLCV → runs 8 indicators (RSI, MACD, EMA, Stoch, ADX, BB, Ichimoku, Pivots) → multi-timeframe agreement check → news sentiment → grades A+/A/B/C/D/F.

Run commands (Windows PowerShell host)

.venv/Scripts/python.exe main.py --pairs USDJPY --timeframe H1 --detail
.venv/Scripts/python.exe telegram_bot.py

On Linux server: venv/bin/python .... Server requires pip install curl_cffi (see SSL section).

---

File roles

File	Role
config.py	Dataclasses for PlatformConfig, TechnicalConfig, SignalConfig, NewsConfig. Pair lists (MAJOR_PAIRS, CROSS_PAIRS, SYMBOL_DISPLAY, PIP_VALUES). Timeframe enum.
data_provider.py	OHLCV fetch (Twelve Data primary, yfinance for XAU/XAG). Cache, rate limiting, validation.
engine.py	TechnicalEngine — computes indicators, builds TechnicalAnalysis per (symbol, TF).
news_engine.py	RSS feed sentiment + economic calendar event risk.
signal_aggregator.py	The "brain". Iterates primary + confirmation TFs, runs MTF agreement check, applies news, builds SignalReport + TradeSetup.
main.py	CLI entry. Parses args, builds PlatformConfig, runs analyze_all_pairs, prints colored table.
telegram_bot.py	Telegram bot. Owns SCAN_PAIRS, SCAN_TIMEFRAMES, per-TF confirmation map. Renders cards.
LOGIC.md	Engine decision flow + indicator weights. Kept up-to-date manually.
USAGE.md	User-facing run guide.

---

Data flow

fetch_ohlcv (per symbol/TF) → tech_engine.analyze → TechnicalAnalysis
                                       ↓
            SignalAggregator.analyze_pair (primary + confirmation TFs)
                                       ↓
                        _check_mtf_agreement → SignalReport
                                       ↓
                  news_engine.get_pair_sentiment → final score + grade
                                       ↓
                      _build_trade_setup → TradeSetup (Entry/TP/SL)

analyze_all_pairs calls prefetch_batch once before iterating pairs — that's the optimization that makes the free Twelve Data tier viable.

---

Where defaults live (and what overrides them)

This is the #1 footgun. The dataclass defaults in config.py are NOT authoritative for actual runs — both CLI and Telegram bot rebuild the config and override confirmation_timeframes based on the chosen primary TF.

Default	Defined in	Overridden by
primary_timeframe = H1	config.py:188	main.py (--timeframe), telegram_bot.py (per scan)
confirmation_timeframes = [M15, D1]	config.py:189-191	main.py:407-417 (tf_confirmations dict), telegram_bot.py:53-56 (_TF_CONFIRMATIONS)
mtf_agreement_threshold = 2	config.py:143	Not overridden — single source of truth.
weights dict	config.py:133-140	Not overridden.
cache_ttl_per_tf	config.py:207-216	Not overridden.

When you change confirmation TFs, update all three places. Otherwise CLI/Telegram silently keep the old values.

---

Per-symbol TF overrides (XAU/XAG)

Defined in signal_aggregator.py:

_PER_SYMBOL_TFS = {
    "XAUUSD=X": [Timeframe.M15, Timeframe.H1, Timeframe.H4],
    "XAGUSD=X": [Timeframe.M15, Timeframe.H1, Timeframe.H4],
}

analyze_pair calls self._confirmation_tfs(symbol) which returns the override (minus primary) when the symbol is in the map, else falls back to config.confirmation_timeframes. Forex pairs are unaffected.

To add another symbol-specific TF set: add to _PER_SYMBOL_TFS. analyze_all_pairs.prefetch_batch already unions per-symbol TFs.

---

Data source rules

Twelve Data (forex)

• Free tier: 8 credits/min, 800 credits/day.
• Each time_series call = 1 credit per symbol. Batch endpoint accepts comma-separated symbols → ~1 credit per timeframe regardless of pair count (when used).
• API key in config.py:197. Override via TWELVE_DATA_API_KEY env var.
• Rate limiter is module-level (_rate_lock, _rate_request_times) so concurrent scans share the budget.
• Supported on free tier: EURUSD, GBPUSD, USDJPY, USDCHF, AUDUSD, USDCAD, NZDUSD, EURGBP, EURJPY, GBPJPY, AUDJPY, EURCHF, CADJPY.
• Not supported on free tier: XAUUSD, XAGUSD, anything in _YF_FALLBACK_MAP.

yfinance (gold/silver)

• Routes via curl_cffi.Session(impersonate="chrome", verify=False) — see SSL section.
• Mapping in data_provider.py:34-37: XAUUSD=X → GC=F, XAGUSD=X → SI=F (CME futures).
• yfinance interval limits: 1m → 7d back; 5m/15m/30m/60m/1h → 60d back.
• No native H4 candle. XAU/XAG H4 is resampled from H1 in _resample_h1_to_h4 (data_provider.py:291-300).
• D1 explicitly skipped for XAU/XAG — futures daily data via yfinance is unreliable. Engine drops missing confirmation TFs cleanly via except Exception at signal_aggregator.py:189-190.

No synthetic fallback

The old _generate_synthetic_data() (GBM with a pair_params dict that defaulted XAU to ~$1.00, causing trade setups like Entry 1.01) is removed. If both Twelve Data and yfinance fail, fetch_ohlcv raises DataUnavailableError. The engine returns grade F + warning, Telegram shows ❌ Data unavailable.

Cache

Per-symbol per-TF, in-memory dict. TTLs in config.cache_ttl_per_tf:

• M1: 60s · M5: 150s · M15: 300s · M30: 600s · H1: 1800s · H4: 7200s · D1: 14400s · W1: 86400s

Shared across SignalAggregator instances when the same ForexDataProvider is reused (see Telegram _analyze_all_tfs).

---

Telegram /scan flow

SCAN_PAIRS      = ["EURUSD=X", "USDJPY=X", "GBPUSD=X", "XAUUSD=X"]   # 4 pairs only
SCAN_TIMEFRAMES = ["M15", "H1"]                                       # 2 cards per pair
_TF_CONFIRMATIONS = {
    "M15": [H1, D1],
    "H1":  [M15, D1],
}

_analyze_all_tfs (telegram_bot.py:282-310):

1. Builds the union of all unique TFs across SCAN_TIMEFRAMES + their confirmations.
2. Single prefetch_batch for all of them on one shared ForexDataProvider instance.
3. Spawns one scan per SCAN_TIMEFRAMES entry concurrently — each reuses the warm cache, so no extra API credits.

Per /scan API cost: ~9 Twelve Data credits (3 forex pairs × 3 TFs, fetched once). XAU is free via yfinance.

To add/remove pairs: edit SCAN_PAIRS (telegram_bot.py:47). To change which TFs get their own card: edit SCAN_TIMEFRAMES (telegram_bot.py:43).

---

SSL — verify=False on server

Live server sits behind a TLS-inspecting proxy that re-signs HTTPS with an untrusted root CA. Fix in data_provider.py:27-37:

from curl_cffi import requests as _curl_cffi_requests
_YF_SESSION = _curl_cffi_requests.Session(impersonate="chrome", verify=False)

Passed into yf.download(..., session=_YF_SESSION). yfinance ≥ 0.2.40 requires a curl_cffi session — it rejects requests.Session.

Trade-off: yfinance traffic is no longer authenticated against MITM. Acceptable because the data is public market quotes and an attacker can at worst inject bad prices. Do not reuse _YF_SESSION for any authenticated/private requests.

---

Indicator weights & scoring

In config.SignalConfig.weights:

{"trend": 0.25, "momentum": 0.20, "volatility": 0.15,
 "price_action": 0.15, "multi_timeframe": 0.15, "sentiment": 0.10}

Per-indicator weights live in engine.py (each indicator function returns its own contribution). Composite score is a weighted sum, capped to [-100, +100]. Grade derived from abs(score) + confidence + mtf_agreement in _compute_signal_quality (signal_aggregator.py).

---

How to make common changes

Add a new tradeable pair (forex)

1. Add "XYZUSD=X" to MAJOR_PAIRS or CROSS_PAIRS in config.py.
2. Add display name to SYMBOL_DISPLAY.
3. Verify it's on Twelve Data free tier (else add to _YF_FALLBACK_MAP).
4. If Telegram /scan should include it, add to SCAN_PAIRS in telegram_bot.py.

Add a non-Twelve-Data symbol (commodity, index)

1. Add to _YF_FALLBACK_MAP in data_provider.py with the Yahoo symbol.
2. Add a _PER_SYMBOL_TFS entry in signal_aggregator.py (D1 unreliable, so drop it; H4 needs resample).
3. Add to SYMBOL_DISPLAY and PIP_VALUES in config.py.

Change confirmation TFs for forex

Edit all three:

• config.py:189-191 — engine default.
• main.py:407-417 — CLI per-primary map.
• telegram_bot.py:53-56 — Telegram per-primary map.

Change MTF agreement threshold

Single edit: config.py:143. Affects all symbols. Note: with N TFs and threshold T, the rule is "T-of-N must agree" — keep T proportional to N if you change one.

Add a new indicator

Add to engine.py → TechnicalEngine.analyze. Set its weight in the Indicator instance. The composite score will pick it up automatically.

Switch primary TF

CLI: --timeframe M15|M30|H1|H4|D1. Telegram: edit SCAN_TIMEFRAMES.

---

Common gotchas

1. The dataclass defaults in config.py are not the source of truth for confirmation_timeframes. CLI and Telegram override them. See "Where defaults live" table.
2. Per-minute API rate limit hits hard with --detail — one detailed run on one pair burns ~3 credits (3 TFs). Multiple back-to-back runs without ~60s gap will hit the 8/min ceiling.
3. Daily limit (800) is shared across the whole day — heavy testing exhausts the budget; signal cache survives but new fetches fail until midnight UTC.
4. curl_cffi is a hard dep on the server — without it, _YF_SESSION is None, yfinance falls back to its default backend, which fails on TLS-inspecting servers.
5. __pycache__/ is committed in this repo (legacy). Don't add it to .gitignore without coordinating; old .pyc will linger as untracked-modified.
6. Telegram SCAN_PAIRS is hardcoded to 4 pairs — editing MAJOR_PAIRS does not change /scan. Update both if intentional.
7. mtf_agreement_threshold = 2 means 2-of-N, not 2-of-3 specifically. Adding a 4th TF to a pair makes "2-of-4" looser than "2-of-3"; consider per-symbol thresholds if this matters.
8. The MTF: ✅ Aligned line in Telegram cards comes from the engine's report.mtf_agreement boolean (signal_aggregator.py:223-233) — not displayed per-TF. The full per-TF breakdown is only in the CLI --detail output.

---

Suggested improvements (not yet implemented)

Replace keyword sentiment with FinBERT

The current news_engine._analyze_headline_sentiment() uses keyword matching against BULLISH_KEYWORDS / BEARISH_KEYWORDS sets (news_engine.py:82-95). This misses obvious context — e.g. "RBA signals pause", "Fed patient on rate cuts", "inflation cools" all score 0.00 because the lexicon doesn't include "pause", "patient", or "cools". Around 80% of headlines fetched currently score 0.

Recommended fix: FinBERT (HuggingFace pretrained model on financial news).

• Drop-in replacement for _analyze_headline_sentiment().
• Pretrained on financial headlines, ~70-75% accuracy on this specific domain.
• One-time ~500MB model download; CPU inference ~50ms/headline (fine for ~30 headlines per fetch).
• No API cost, no external dependency at runtime.

Implementation sketch (~30 lines):

from transformers import pipeline
_finbert = pipeline("sentiment-analysis", model="ProsusAI/finbert")

def _analyze_headline_sentiment(self, headline: str) -> float:
    result = _finbert(headline[:512])[0]  # truncate to model max
    label, score = result["label"], result["score"]
    if label == "positive": return score
    if label == "negative": return -score
    return 0.0

Add transformers and torch (or torch-cpu) to requirements.txt. Cache scored headlines by hash(headline) to avoid re-scoring across pairs in the same fetch. Add a try/except fallback to the current keyword logic if model load fails (graceful degradation).

Realistic uplift on overall signal quality: 1-3% win-rate. Sentiment is only 10% of the composite score (config.py:139), so even doubling sentiment accuracy moves the needle modestly. Worth doing for technical correctness, not as a P&L optimization.

Higher-tier alternative: Claude/GPT API per headline. 80-85% accurate, $0.001/headline ($0.03 per /scan with caching). Adds external dependency and latency. Only worth it if FinBERT proves insufficient.

Cheap alternative without ML: Expand keyword sets. Add ~50-100 phrases to BULLISH_KEYWORDS / BEARISH_KEYWORDS ("pause", "patient", "data-dependent", "soft landing", "beat", "revised up" / "miss", "revised down", "warning", "fragile"). Gets ~50% → ~60% accuracy in 30 min, no new deps.

---

Recent architectural decisions (kept short)

• Removed synthetic data path — explicit failure beats fake prices.
• XAU/XAG → yfinance only — Twelve Data free tier doesn't carry them; routing them through TD wastes credits and used to fall through to bogus synthetic data.
• H1→H4 resample for gold — yfinance has no native 4h, but H1 with 60d lookback gives ~360 H4 bars after resample, enough for indicator calculation.
• /scan limited to 4 pairs — credit budget on free tier doesn't sustain 13-pair scans.
• Two SCAN_TIMEFRAMES (M15 + H1) with shared cache — gives users two perspectives without doubling API cost (cache reuse).
• No SSL verify on yfinance — server-side TLS inspection breaks cert chain; disabled because data is public quotes.