Fix scoring math; add 10-min auto-scan A-grade alerts with news block

- engine: composite-score denominator now confidence-weighted, so the
  output actually spans -100..+100 (the prior formula put confidence
  in the numerator only, capping the realistic score around ±25-30
  and making STRONG_BUY ≥40 essentially unreachable).
- engine: ATR percentile uses iloc[-50:-1] to exclude the current bar
  from its own comparison (off-by-one in volatility regime).
- signal_aggregator: always blend primary score with MTF average. The
  prior code only blended on agreement, leaving the primary unblended
  on disagreement — backwards. Disagreement is exactly when the score
  should be pulled toward consensus.
- telegram_bot: JobQueue auto-scan every 10 min that pushes A/A+ to
  ALERT_CHAT_ID with a prominent News & Events block (sentiment +
  headline summary + event-risk + alignment hint). Per-bar dedup so
  persistent setups alert once per new candle, not every 10 min.
  Suppresses alerts when event_risk = "danger".
- telegram_bot: /myid returns chat ID for ALERT_CHAT_ID setup. Auto-scan
  silently disabled when ALERT_CHAT_ID is unset.

Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>

.gitignore

@@ -0,0 +1,17 @@
+# Secrets
+.env
+.env.local
+
+# Editor / tooling
+.claude/
+.vscode/
+.idea/
+
+# Python (note: __pycache__/ is intentionally tracked in this repo — see CLAUDE.md)
+*.egg-info/
+.venv/
+venv/
+
+# OS
+.DS_Store
+Thumbs.db

CLAUDE.md

@@ -0,0 +1,241 @@
+# ForexBot2 — Working Context for Claude
+
+Comprehensive context to avoid re-deriving the codebase in every session.
+Update this file when behavior, defaults, or file layout changes.
+
+---
+
+## What it does
+
+Forex + gold signal bot. Two entry points:
+
+- **CLI**: `main.py --pairs EURUSD --timeframe H1 [--detail] [--min-grade C]`
+- **Telegram bot**: `telegram_bot.py` (commands: `/scan`, `/signal <pair>`, `/help`)
+
+Pulls OHLCV → runs 8 indicators (RSI, MACD, EMA, Stoch, ADX, BB, Ichimoku, Pivots) → multi-timeframe agreement check → news sentiment → grades A+/A/B/C/D/F.
+
+## Run commands (Windows PowerShell host)
+
+```powershell
+.venv/Scripts/python.exe main.py --pairs USDJPY --timeframe H1 --detail
+.venv/Scripts/python.exe telegram_bot.py
+```
+
+On Linux server: `venv/bin/python ...`. Server requires `pip install curl_cffi` (see SSL section).
+
+---
+
+## File roles
+
+| File | Role |
+|---|---|
+| `config.py` | Dataclasses for `PlatformConfig`, `TechnicalConfig`, `SignalConfig`, `NewsConfig`. Pair lists (`MAJOR_PAIRS`, `CROSS_PAIRS`, `SYMBOL_DISPLAY`, `PIP_VALUES`). `Timeframe` enum. |
+| `data_provider.py` | OHLCV fetch (Twelve Data primary, yfinance for XAU/XAG). Cache, rate limiting, validation. |
+| `engine.py` | `TechnicalEngine` — computes indicators, builds `TechnicalAnalysis` per (symbol, TF). |
+| `news_engine.py` | RSS feed sentiment + economic calendar event risk. |
+| `signal_aggregator.py` | The "brain". Iterates primary + confirmation TFs, runs MTF agreement check, applies news, builds `SignalReport` + `TradeSetup`. |
+| `main.py` | CLI entry. Parses args, builds `PlatformConfig`, runs `analyze_all_pairs`, prints colored table. |
+| `telegram_bot.py` | Telegram bot. Owns `SCAN_PAIRS`, `SCAN_TIMEFRAMES`, per-TF confirmation map. Renders cards. |
+| `LOGIC.md` | Engine decision flow + indicator weights. Kept up-to-date manually. |
+| `USAGE.md` | User-facing run guide. |
+
+---
+
+## Data flow
+
+```
+fetch_ohlcv (per symbol/TF) → tech_engine.analyze → TechnicalAnalysis
+                                       ↓
+            SignalAggregator.analyze_pair (primary + confirmation TFs)
+                                       ↓
+                        _check_mtf_agreement → SignalReport
+                                       ↓
+                  news_engine.get_pair_sentiment → final score + grade
+                                       ↓
+                      _build_trade_setup → TradeSetup (Entry/TP/SL)
+```
+
+`analyze_all_pairs` calls `prefetch_batch` once before iterating pairs — that's the optimization that makes the free Twelve Data tier viable.
+
+---
+
+## Where defaults live (and what overrides them)
+
+**This is the #1 footgun.** The dataclass defaults in `config.py` are NOT authoritative for actual runs — both CLI and Telegram bot rebuild the config and override `confirmation_timeframes` based on the chosen primary TF.
+
+| Default | Defined in | Overridden by |
+|---|---|---|
+| `primary_timeframe = H1` | [config.py:188](config.py#L188) | `main.py` (`--timeframe`), `telegram_bot.py` (per scan) |
+| `confirmation_timeframes = [M15, D1]` | [config.py:189-191](config.py#L189-L191) | `main.py:407-417` (`tf_confirmations` dict), `telegram_bot.py:53-56` (`_TF_CONFIRMATIONS`) |
+| `mtf_agreement_threshold = 2` | [config.py:143](config.py#L143) | _Not_ overridden — single source of truth. |
+| `weights` dict | [config.py:133-140](config.py#L133-L140) | _Not_ overridden. |
+| `cache_ttl_per_tf` | [config.py:207-216](config.py#L207-L216) | _Not_ overridden. |
+
+**When you change confirmation TFs, update all three places.** Otherwise CLI/Telegram silently keep the old values.
+
+---
+
+## Per-symbol TF overrides (XAU/XAG)
+
+Defined in `signal_aggregator.py`:
+
+```python
+_PER_SYMBOL_TFS = {
+    "XAUUSD=X": [Timeframe.M15, Timeframe.H1, Timeframe.H4],
+    "XAGUSD=X": [Timeframe.M15, Timeframe.H1, Timeframe.H4],
+}
+```
+
+`analyze_pair` calls `self._confirmation_tfs(symbol)` which returns the override (minus primary) when the symbol is in the map, else falls back to `config.confirmation_timeframes`. Forex pairs are unaffected.
+
+To add another symbol-specific TF set: add to `_PER_SYMBOL_TFS`. `analyze_all_pairs.prefetch_batch` already unions per-symbol TFs.
+
+---
+
+## Data source rules
+
+### Twelve Data (forex)
+- Free tier: **8 credits/min, 800 credits/day**.
+- Each `time_series` call = 1 credit per symbol. Batch endpoint accepts comma-separated symbols → ~1 credit per timeframe regardless of pair count (when used).
+- API key in [config.py:197](config.py#L197). Override via `TWELVE_DATA_API_KEY` env var.
+- Rate limiter is **module-level** (`_rate_lock`, `_rate_request_times`) so concurrent scans share the budget.
+- Supported on free tier: EURUSD, GBPUSD, USDJPY, USDCHF, AUDUSD, USDCAD, NZDUSD, EURGBP, EURJPY, GBPJPY, AUDJPY, EURCHF, CADJPY.
+- **Not supported on free tier**: XAUUSD, XAGUSD, anything in `_YF_FALLBACK_MAP`.
+
+### yfinance (gold/silver)
+- Routes via `curl_cffi.Session(impersonate="chrome", verify=False)` — see SSL section.
+- Mapping in [data_provider.py:34-37](data_provider.py#L34-L37): `XAUUSD=X → GC=F`, `XAGUSD=X → SI=F` (CME futures).
+- yfinance interval limits: 1m → 7d back; 5m/15m/30m/60m/1h → 60d back.
+- **No native H4 candle.** XAU/XAG H4 is resampled from H1 in `_resample_h1_to_h4` ([data_provider.py:291-300](data_provider.py#L291-L300)).
+- **D1 explicitly skipped** for XAU/XAG — futures daily data via yfinance is unreliable. Engine drops missing confirmation TFs cleanly via `except Exception` at `signal_aggregator.py:189-190`.
+
+### No synthetic fallback
+The old `_generate_synthetic_data()` (GBM with a `pair_params` dict that defaulted XAU to ~$1.00, causing trade setups like Entry 1.01) is **removed**. If both Twelve Data and yfinance fail, `fetch_ohlcv` raises `DataUnavailableError`. The engine returns grade F + warning, Telegram shows `❌ Data unavailable`.
+
+### Cache
+Per-symbol per-TF, in-memory dict. TTLs in `config.cache_ttl_per_tf`:
+- M1: 60s · M5: 150s · M15: 300s · M30: 600s · H1: 1800s · H4: 7200s · D1: 14400s · W1: 86400s
+
+Shared across `SignalAggregator` instances when the same `ForexDataProvider` is reused (see Telegram `_analyze_all_tfs`).
+
+---
+
+## Telegram /scan flow
+
+```python
+SCAN_PAIRS      = ["EURUSD=X", "USDJPY=X", "GBPUSD=X", "XAUUSD=X"]   # 4 pairs only
+SCAN_TIMEFRAMES = ["M15", "H1"]                                       # 2 cards per pair
+_TF_CONFIRMATIONS = {
+    "M15": [H1, D1],
+    "H1":  [M15, D1],
+}
+```
+
+`_analyze_all_tfs` ([telegram_bot.py:282-310](telegram_bot.py#L282-L310)):
+1. Builds the union of all unique TFs across `SCAN_TIMEFRAMES` + their confirmations.
+2. Single `prefetch_batch` for all of them on **one shared `ForexDataProvider` instance**.
+3. Spawns one scan per `SCAN_TIMEFRAMES` entry concurrently — each reuses the warm cache, so no extra API credits.
+
+Per /scan API cost: ~9 Twelve Data credits (3 forex pairs × 3 TFs, fetched once). XAU is free via yfinance.
+
+To add/remove pairs: edit `SCAN_PAIRS` ([telegram_bot.py:47](telegram_bot.py#L47)).
+To change which TFs get their own card: edit `SCAN_TIMEFRAMES` ([telegram_bot.py:43](telegram_bot.py#L43)).
+
+---
+
+## SSL — verify=False on server
+
+Live server sits behind a TLS-inspecting proxy that re-signs HTTPS with an untrusted root CA. Fix in [data_provider.py:27-37](data_provider.py#L27-L37):
+
+```python
+from curl_cffi import requests as _curl_cffi_requests
+_YF_SESSION = _curl_cffi_requests.Session(impersonate="chrome", verify=False)
+```
+
+Passed into `yf.download(..., session=_YF_SESSION)`. yfinance ≥ 0.2.40 **requires** a `curl_cffi` session — it rejects `requests.Session`.
+
+Trade-off: yfinance traffic is no longer authenticated against MITM. Acceptable because the data is public market quotes and an attacker can at worst inject bad prices. **Do not** reuse `_YF_SESSION` for any authenticated/private requests.
+
+---
+
+## Indicator weights & scoring
+
+In `config.SignalConfig.weights`:
+
+```python
+{"trend": 0.25, "momentum": 0.20, "volatility": 0.15,
+ "price_action": 0.15, "multi_timeframe": 0.15, "sentiment": 0.10}
+```
+
+Per-indicator weights live in `engine.py` (each indicator function returns its own contribution). Composite score is a weighted sum, capped to [-100, +100]. Grade derived from `abs(score) + confidence + mtf_agreement` in `_compute_signal_quality` (`signal_aggregator.py`).
+
+---
+
+## How to make common changes
+
+### Add a new tradeable pair (forex)
+1. Add `"XYZUSD=X"` to `MAJOR_PAIRS` or `CROSS_PAIRS` in `config.py`.
+2. Add display name to `SYMBOL_DISPLAY`.
+3. Verify it's on Twelve Data free tier (else add to `_YF_FALLBACK_MAP`).
+4. If Telegram /scan should include it, add to `SCAN_PAIRS` in `telegram_bot.py`.
+
+### Add a non-Twelve-Data symbol (commodity, index)
+1. Add to `_YF_FALLBACK_MAP` in `data_provider.py` with the Yahoo symbol.
+2. Add a `_PER_SYMBOL_TFS` entry in `signal_aggregator.py` (D1 unreliable, so drop it; H4 needs resample).
+3. Add to `SYMBOL_DISPLAY` and `PIP_VALUES` in `config.py`.
+
+### Change confirmation TFs for forex
+Edit **all three**:
+- `config.py:189-191` — engine default.
+- `main.py:407-417` — CLI per-primary map.
+- `telegram_bot.py:53-56` — Telegram per-primary map.
+
+### Change MTF agreement threshold
+Single edit: `config.py:143`. Affects all symbols. Note: with N TFs and threshold T, the rule is "T-of-N must agree" — keep T proportional to N if you change one.
+
+### Add a new indicator
+Add to `engine.py` → `TechnicalEngine.analyze`. Set its `weight` in the `Indicator` instance. The composite score will pick it up automatically.
+
+### Switch primary TF
+CLI: `--timeframe M15|M30|H1|H4|D1`.
+Telegram: edit `SCAN_TIMEFRAMES`.
+
+---
+
+## Common gotchas
+
+1. **The dataclass defaults in `config.py` are not the source of truth for `confirmation_timeframes`.** CLI and Telegram override them. See "Where defaults live" table.
+2. **Per-minute API rate limit hits hard with `--detail`** — one detailed run on one pair burns ~3 credits (3 TFs). Multiple back-to-back runs without ~60s gap will hit the 8/min ceiling.
+3. **Daily limit (800) is shared across the whole day** — heavy testing exhausts the budget; signal cache survives but new fetches fail until midnight UTC.
+4. **`curl_cffi` is a hard dep on the server** — without it, `_YF_SESSION` is `None`, yfinance falls back to its default backend, which fails on TLS-inspecting servers.
+5. **`__pycache__/` is committed** in this repo (legacy). Don't add it to `.gitignore` without coordinating; old `.pyc` will linger as untracked-modified.
+6. **Telegram `SCAN_PAIRS` is hardcoded to 4 pairs** — editing `MAJOR_PAIRS` does not change /scan. Update both if intentional.
+7. **`mtf_agreement_threshold = 2` means 2-of-N**, not 2-of-3 specifically. Adding a 4th TF to a pair makes "2-of-4" looser than "2-of-3"; consider per-symbol thresholds if this matters.
+8. **The `MTF: ✅ Aligned` line in Telegram cards comes from the engine's `report.mtf_agreement` boolean** ([signal_aggregator.py:223-233](signal_aggregator.py#L223-L233)) — not displayed per-TF. The full per-TF breakdown is only in the CLI `--detail` output.
+
+---
+
+## News sentiment — Google Gemini (live)
+
+`news_engine` uses **Gemini `gemini-2.5-flash-lite`** for headline sentiment, replacing the keyword lexicon (which still exists as a fallback).
+
+- API key loaded from `.env` (`GEMINI_API_KEY=...`) via `python-dotenv`. `.env` is gitignored.
+- One **batched** API call per `/scan` — `_score_headlines_batch` sends all uncached titles in a single prompt, returns a JSON array of `{id, score}` ([news_engine.py:355-410](news_engine.py#L355-L410)).
+- In-memory cache `_sentiment_cache` keyed by `hash(headline)` — repeat calls for the same headline are free.
+- Free-tier quota on `gemini-2.5-flash-lite`: **1000 RPD** (vs 20 on `gemini-2.5-flash` — do not use the non-lite variant).
+- If Gemini fails (timeout, 429, parse error), falls back per-headline to `_fallback_keyword_sentiment` so signals still grade.
+
+Observed quality lift: ~88% of fetched headlines now get non-zero sentiment (vs ~10% with keywords). Headlines like "RBA signals pause" or "inflation cools below forecast" score correctly bearish where keywords missed them entirely.
+
+To swap models: change `NewsEngine.GEMINI_MODEL` ([news_engine.py:118](news_engine.py#L118)). Use the new `google-genai` SDK (`from google import genai; genai.Client(api_key=...)`); the older `google-generativeai` package is replaced.
+
+---
+
+## Recent architectural decisions (kept short)
+
+- **Removed synthetic data path** — explicit failure beats fake prices.
+- **XAU/XAG → yfinance only** — Twelve Data free tier doesn't carry them; routing them through TD wastes credits and used to fall through to bogus synthetic data.
+- **H1→H4 resample for gold** — yfinance has no native 4h, but H1 with 60d lookback gives ~360 H4 bars after resample, enough for indicator calculation.
+- **/scan limited to 4 pairs** — credit budget on free tier doesn't sustain 13-pair scans.
+- **Two SCAN_TIMEFRAMES (M15 + H1) with shared cache** — gives users two perspectives without doubling API cost (cache reuse).
+- **No SSL verify on yfinance** — server-side TLS inspection breaks cert chain; disabled because data is public quotes.

LOGIC.md

@@ -0,0 +1,226 @@
+# ForexBot2 — Engine Logic Reference
+
+## 1. Data Provider (Twelve Data API)
+
+### What the API provides
+Each API call returns an OHLCV time series for a currency pair:
+
+| Field    | Description                        |
+|----------|------------------------------------|
+| Datetime | UTC timestamp of the candle open   |
+| Open     | Opening price                      |
+| High     | Highest price during the candle    |
+| Low      | Lowest price during the candle     |
+| Close    | Closing price                      |
+| Volume   | Tick volume (limited on Forex)     |
+
+Supported intervals: 1min, 5min, 15min, 30min, 1h, 4h, 1day, 1week.
+
+### Supported pairs (free tier — live data)
+EURUSD, GBPUSD, USDJPY, USDCHF, AUDUSD, USDCAD, NZDUSD,
+EURGBP, EURJPY, GBPJPY, AUDJPY, EURCHF, CADJPY.
+All other symbols (XAU, XAG, indices) return no data on the free plan.
+
+### API credit limits (free tier)
+- 8 credits per minute
+- 800 credits per day
+- 1 credit per symbol in a batch request
+  (7 pairs in one request = 7 credits, not 7 requests)
+
+### Data validation and cleaning
+After every fetch the data goes through `_validate_and_clean`:
+1. Drop rows where all of Open/High/Low/Close are NaN
+2. Forward-fill small gaps (max 3 consecutive bars)
+3. Drop rows still missing Close
+4. Remove bars with future timestamps (> 5 min ahead of now)
+5. Fix OHLC inconsistencies (e.g. High < Low) by clamping
+6. Remove extreme outlier bars (> 10 standard deviations return)
+7. Sort index and deduplicate (keep last on duplicates)
+
+---
+
+## 2. How Many API Calls Are Made
+
+### Per scan or /signal command
+
+The bot does **one shared prefetch** before spawning TF analyses.
+It collects all unique timeframes needed across scans, then makes
+one batch API call per timeframe (all pairs in a single request).
+
+Current SCAN_TIMEFRAMES: M15, H1 (only intraday trade signals)
+Confirmations:
+- M15 uses: H1, D1
+- H1  uses: M15, D1
+
+D1 is kept as a confirmation for higher-timeframe trend context but
+no D1 trade signals are produced. W1 has been dropped entirely.
+
+Unique timeframes: M15, H1, D1 = **3 batch calls**
+
+Each call: 7 pairs × 1 credit = **7 credits per call**
+
+Total per scan: **3 calls × 7 credits = 21 credits**
+
+With the 8 credits/minute limit:
+- Call 1 (7 credits): fires immediately — 1 credit remaining
+- Call 2 (7 credits): must wait ~60s for window to clear
+- Call 3 (7 credits): waits another ~60s
+
+**Minimum scan time: ~2 minutes.**
+
+After the shared prefetch, the 4 concurrent TF analyses read from
+the in-memory cache and make **zero additional API calls**.
+
+### Cache behaviour
+Results are cached per symbol+timeframe for `refresh_interval` seconds
+(default in PlatformConfig). A second scan within that window skips all
+API calls entirely and returns cached data instantly.
+
+---
+
+## 3. Technical Engine — How Signals Are Decided
+
+### Step 1 — Fetch OHLCV data
+Primary timeframe data is fetched first. If it fails, the report is
+immediately graded F and returned — no further analysis is attempted.
+Confirmation timeframe data is fetched next; failures are logged as
+warnings but do not abort the analysis.
+
+### Step 2 — Compute 11 indicators
+
+Each indicator returns: value, signal (STRONG_BUY / BUY / NEUTRAL /
+SELL / STRONG_SELL), confidence (0–100), and a weight.
+
+| Indicator        | Weight | What it measures                              |
+|------------------|--------|-----------------------------------------------|
+| EMA Trend        | 1.2    | Alignment of fast/medium/slow/trend EMAs      |
+| MACD             | 1.1    | Momentum crossover and histogram direction    |
+| Ichimoku         | 1.1    | Cloud position, TK cross, cloud colour        |
+| RSI              | 1.0    | Overbought/oversold momentum                  |
+| ADX              | 1.0    | Trend strength and directional index          |
+| Bollinger Bands  | 0.9    | Price position within volatility envelope     |
+| Stochastic       | 0.8    | Oscillator crossover in OB/OS zones           |
+| Candlestick      | 0.7    | Pattern detection (hammer, engulfing, doji…)  |
+| Divergence       | 1.5    | RSI/MACD divergence from price (reversal)     |
+| Pivot Points     | 0.6    | Classical pivot S/R relative to current price |
+| ATR / Volume     | 0.0    | Context only — not included in score          |
+
+Divergence has the highest weight (1.5) because divergences are strong
+reversal predictors. ATR and Volume have weight 0 — they inform the
+trade setup and volatility regime but do not vote on direction.
+
+### Step 3 — Classify market regime
+Using ADX value and EMA direction:
+- `trending_up` / `trending_down`: ADX ≥ strong threshold + EMA aligned
+- `ranging`: ADX below threshold
+
+The regime changes indicator weights for the composite score:
+- **Trending**: EMA (×1.5), ADX (×1.3), MACD (×1.2) weighted up;
+  RSI (×0.7) and Bollinger (×0.6) weighted down
+- **Ranging**: RSI (×1.4), Bollinger (×1.4), Stochastic (×1.3)
+  weighted up; EMA (×0.6) and ADX (×0.5) weighted down
+
+### Step 4 — Compute composite score (−100 to +100)
+For each indicator:
+```
+weighted_score = signal_value × weight × regime_multiplier × (confidence / 100)
+```
+Signal values: STRONG_BUY = +2, BUY = +1, NEUTRAL = 0, SELL = −1, STRONG_SELL = −2
+
+Final composite score = (sum of weighted scores / total weight) × 50
+
+Composite signal thresholds:
+- score ≥ +40 → STRONG BUY
+- score ≥ +15 → BUY
+- score ≤ −40 → STRONG SELL
+- score ≤ −15 → SELL
+- otherwise  → NEUTRAL
+
+### Step 5 — Multi-timeframe (MTF) agreement check
+All timeframe signals (primary + confirmations) are collected.
+MTF agreement = True if at least `mtf_agreement_threshold` timeframes
+all point bullish or all point bearish.
+
+Effect on confidence:
+- **Agreement**: confidence +15 (capped at 95); score blended with
+  average of all TF scores (60% primary, 40% MTF average)
+- **Disagreement**: confidence ×0.7
+- **Conflict** (some bullish, some bearish): confidence ×0.7 × 0.6 = ×0.42
+  + warning added to report
+
+### Step 6 — Fundamental adjustments
+Applied in order, each multiplying confidence:
+
+| Condition                        | Confidence adjustment |
+|----------------------------------|-----------------------|
+| News aligns with technical       | +5 (capped 95)        |
+| News conflicts with technical    | ×0.85                 |
+| Event risk: caution              | ×0.70                 |
+| Event risk: danger               | ×0.30 + block signal  |
+| Volatility: extreme              | ×0.80                 |
+
+### Step 7 — Final signal determination
+Uses the adjusted score and confidence:
+
+| Condition                        | Signal      |
+|----------------------------------|-------------|
+| score ≥ +40 and confidence ≥ 65 | STRONG BUY  |
+| score ≥ +15 and confidence ≥ 50 | BUY         |
+| score ≤ −40 and confidence ≥ 65 | STRONG SELL |
+| score ≤ −15 and confidence ≥ 50 | SELL        |
+| otherwise                        | NEUTRAL     |
+
+### Step 8 — Quality grade
+Combines score, confidence, and MTF agreement into a letter grade:
+
+| Grade | abs(score) | confidence | MTF agreement |
+|-------|-----------|------------|---------------|
+| A+    | ≥ 50      | ≥ 75       | required      |
+| A     | ≥ 35      | ≥ 65       | required      |
+| B     | ≥ 25      | ≥ 55       | not required  |
+| C     | ≥ 15      | ≥ 45       | not required  |
+| D     | ≥ 10      | any        | not required  |
+| F     | < 10      | any        | —             |
+
+A+ and A require MTF agreement as a hard condition — no matter how
+strong the primary score, without timeframe alignment the grade caps at B.
+
+### Step 9 — Trade setup (grades A+, A, B only)
+Entry = current close price
+Stop Loss = entry ± (ATR × 1.5)
+Take Profit 1 = entry ± (SL distance × R:R ratio)
+Take Profit 2 = entry ± (SL distance × R:R ratio × 1.5)
+
+ATR is used as the volatility measure so SL/TP adapt to current
+market conditions rather than using fixed pip distances.
+
+---
+
+## 4. Signal Flow Summary
+
+```
+Telegram /scan or /signal
+        │
+        ▼
+_analyze_all_tfs(pairs)
+        │
+        ├── prefetch_batch(pairs, [M15, H1, D1])
+        │       ├── Batch call 1: all pairs @ M15  (7 credits, wait if needed)
+        │       ├── Batch call 2: all pairs @ H1   (7 credits, wait if needed)
+        │       └── Batch call 3: all pairs @ D1   (7 credits, wait if needed)
+        │
+        └── 2 threads (M15 + H1), all reading from warm cache
+                │
+                └── per thread: SignalAggregator.analyze_all_pairs()
+                        │
+                        └── per pair: analyze_pair()
+                                ├── fetch_ohlcv (cache hit)
+                                ├── TechnicalEngine.analyze()
+                                │       ├── 11 indicators
+                                │       ├── Regime classification
+                                │       └── Composite score
+                                ├── MTF agreement check
+                                ├── News / event risk adjustments
+                                ├── Final signal + grade
+                                └── Trade setup (A+/A/B only)
+```

USAGE.md

@@ -0,0 +1,285 @@
+# ForexBot2 — Usage Guide
+
+## Setup
+
+```bash
+pip install -r requirements.txt
+```
+
+Set your Twelve Data API key (the bot will not work without one):
+
+```powershell
+$env:TWELVE_DATA_API_KEY="your_key_here"   # PowerShell
+```
+
+```bash
+export TWELVE_DATA_API_KEY=your_key_here   # macOS / Linux
+set TWELVE_DATA_API_KEY=your_key_here      # Windows CMD
+```
+
+Get a free key at <https://twelvedata.com/pricing>. Free tier covers most major Forex pairs; gold/silver fall through to Yahoo Finance automatically (see [Symbols](#supported-symbols)).
+
+---
+
+## Terminal — `main.py`
+
+```
+python main.py [OPTIONS]
+```
+
+### Options
+
+| Option                       | Default                  | Description                                              |
+|------------------------------|--------------------------|----------------------------------------------------------|
+| `--pairs PAIR [PAIR ...]`    | all major pairs          | Specific pairs to analyze                                |
+| `--timeframe TF` / `-tf TF`  | `H1`                     | Primary timeframe — see table below                      |
+| `--all`                      | off                      | Include cross pairs (EUR/GBP, EUR/JPY, GBP/JPY, etc.)    |
+| `--detail`                   | off                      | Full indicator breakdown for every pair                  |
+| `--top N`                    | all                      | Show only the top N signals by strength                  |
+| `--min-grade GRADE`          | `F`                      | Minimum grade to display: `A+ A B C D F`                 |
+| `--json`                     | off                      | Output results as JSON                                   |
+| `--watch`                    | off                      | Continuous mode — re-scans every 5 minutes               |
+| `--verbose` / `-v`           | off                      | Enable debug logging                                     |
+
+### Timeframes accepted by `--timeframe`
+
+| Flag  | Interval    | Best for                  |
+|-------|-------------|---------------------------|
+| `M5`  | 5 minutes   | Scalping                  |
+| `M15` | 15 minutes  | Short-term intraday       |
+| `M30` | 30 minutes  | Intraday                  |
+| `H1`  | 1 hour      | Intraday / swing (default)|
+| `H4`  | 4 hours     | Swing trading             |
+| `D1`  | Daily       | Position trading          |
+
+`main.py` automatically picks confirmation timeframes around the primary
+(e.g. `H1` confirms with `M15` and `H4`).
+
+### Examples
+
+```bash
+# Scan all major pairs on H1 (default)
+python main.py
+
+# One pair, full indicator detail
+python main.py --pairs EURUSD --detail
+
+# Gold on the 4-hour chart
+python main.py --pairs XAUUSD --timeframe H4 --detail
+
+# Multiple pairs
+python main.py --pairs EURUSD GBPUSD USDJPY
+
+# Only Grade B+ signals
+python main.py --min-grade B
+
+# Top 3 signals across all pairs including crosses
+python main.py --all --top 3
+
+# Continuous monitoring — refreshes every 5 minutes
+python main.py --watch
+python main.py --pairs XAUUSD --watch --detail
+
+# JSON output to a file
+python main.py --json
+python main.py --all --json > signals.json
+```
+
+### Terminal output
+
+Summary table:
+
+```
+  │ EUR/USD     │ STRONG BUY │ ░░░░░░░░░░░░░│███████████████ │ A+     │   82.3%  │
+```
+
+- **Score bar** — left of `│` = bearish, right = bullish; filled blocks show strength
+- **Grade** — overall quality of the signal setup
+- **⚠** — high-impact event imminent;  **△** — event approaching (caution)
+
+Detail block (shown automatically for Grade B+ signals, or for every pair
+with `--detail`) includes price, regime, multi-timeframe breakdown, every
+indicator value, bullish/bearish reasons, warnings, news, and the trade
+setup (entry / SL / TP1 / TP2).
+
+---
+
+## Telegram Bot — `telegram_bot.py`
+
+```powershell
+$env:TELEGRAM_BOT_TOKEN="your_token_here"
+python telegram_bot.py
+```
+
+Get a token from [@BotFather](https://t.me/BotFather) → `/newbot`.
+
+### Commands
+
+| Command            | Description                                                         |
+|--------------------|---------------------------------------------------------------------|
+| `/signal EURUSD`   | Full signal analysis for one pair across M15 and H1                 |
+| `/signal XAUUSD`   | Same — gold uses the yfinance fallback automatically                |
+| `/scan`            | Analyze all major pairs, return one card per Grade B+ pair           |
+| `/help`            | Command reference + grade legend                                     |
+| `/start`           | Same as `/help`                                                      |
+
+The bot **does not** accept a timeframe argument. It always scans M15 and
+H1 (using D1 as a confirmation timeframe internally for higher-timeframe
+trend context). To change timeframes, edit `SCAN_TIMEFRAMES` in
+[telegram_bot.py](telegram_bot.py).
+
+### Live progress
+
+Every command shows coarse progress through message edits while it runs:
+
+```
+🔄 Analyzing XAU/USD across M15 · H1…
+📡 Fetching market data (3 timeframes × 1 pair)…
+🧠 Analyzing pairs across timeframes…
+```
+
+The placeholder is then replaced by the result message (`/signal`) or new
+messages are sent (`/scan`).
+
+### Signal card format
+
+```
+🌐 🧠 EUR/USD — Signal Analysis
+🕐 03:45 PM EST · Mon 04 May 2026
+Grade scale: 🔥 A+ · ✅ A · 👍 B · ⚠️ C · 📉 D · ❌ F
+
+────────────────────────
+✅ M15 · BUY · Grade A
+
+Entry: 1.08423
+Take Profit: 1.08970  (+54.7 pips)
+Take Profit 2: 1.09517
+Stop Loss: 1.08150  (-27.3 pips)
+Risk:Reward: 1:2.0
+
+• EMA Trend: 1.00 (Golden cross)
+• MACD: 0.000082 (Bullish crossover)
+• RSI: 58.3 (Neutral)
+...
+
+MTF: ✅ Aligned | Score: +62.4 | trending_up | Vol: normal
+```
+
+### Scan format
+
+`/scan` produces one compact line per Grade B+ pair:
+
+```
+👍 EUR/USD · BUY · B · M15
+   Entry: 1.08423 | TP: 1.08970 (+55p) | SL: 1.08150 (-27p) | R:R 1:2.0
+```
+
+Pairs without an actionable Grade B+ signal are omitted from the scan output.
+
+---
+
+## Signal Grades
+
+| Grade | Symbol | abs(score) | confidence | MTF agreement | Meaning                              |
+|-------|--------|-----------:|-----------:|---------------|--------------------------------------|
+| A+    | 🔥     | ≥ 50       | ≥ 75       | required      | Strongest — entry candidate          |
+| A     | ✅     | ≥ 35       | ≥ 65       | required      | Strong signal                        |
+| B     | 👍     | ≥ 25       | ≥ 55       | not required  | Decent setup — confirm before entry  |
+| C     | ⚠️     | ≥ 15       | ≥ 45       | not required  | Weak — monitor only                  |
+| D     | 📉     | ≥ 10       | any        | not required  | Noise                                |
+| F     | ❌     | < 10       | any        | —             | No trade                             |
+
+Trade setup (entry / TP / SL) is generated only for Grade A+ / A / B.
+
+---
+
+## Supported Symbols
+
+Pass any symbol without slashes or the `=X` suffix.
+
+### Forex (live data on Twelve Data free tier)
+
+| Input    | Pair                                  |
+|----------|---------------------------------------|
+| `EURUSD` | Euro / US Dollar                      |
+| `GBPUSD` | British Pound / US Dollar             |
+| `USDJPY` | US Dollar / Japanese Yen              |
+| `USDCHF` | US Dollar / Swiss Franc               |
+| `AUDUSD` | Australian Dollar / US Dollar         |
+| `USDCAD` | US Dollar / Canadian Dollar           |
+| `NZDUSD` | New Zealand Dollar / US Dollar        |
+| `EURGBP`, `EURJPY`, `GBPJPY`, `AUDJPY`, `EURCHF`, `CADJPY` | Major crosses |
+
+### Metals (Yahoo Finance fallback — automatic)
+
+| Input    | Maps to | Source             |
+|----------|---------|--------------------|
+| `XAUUSD` | `GC=F`  | Yahoo gold futures |
+| `XAGUSD` | `SI=F`  | Yahoo silver futures |
+
+When Twelve Data returns no data for a symbol (e.g. gold on the free tier),
+the data provider transparently falls back to Yahoo Finance via `yfinance`.
+Logs are tagged `[live]` for Twelve Data and `[yf]` for Yahoo so you can
+verify the source.
+
+Note: `GC=F` is the futures contract, not spot XAU/USD. Direction is
+identical, but absolute price differs from your broker's spot quote by a
+small basis (typically $5–15). Use signal direction; execute at your
+broker's price.
+
+### Symbols not on free tier and not mapped to yfinance
+
+Any symbol Twelve Data doesn't return data for and isn't in the yfinance
+fallback map will fail with `❌ Data unavailable`. The bot does not
+generate synthetic data.
+
+---
+
+## API Limits
+
+### Twelve Data — Free Tier
+
+| Limit               | Value         |
+|---------------------|---------------|
+| Credits per minute  | 8             |
+| Credits per day     | 800           |
+| Credits per scan    | 21 (3 TFs × 7 pairs) |
+| Theoretical scans/day | ~38         |
+
+Each scan makes 3 batch API calls (one per unique timeframe: M15, H1, D1).
+The 8-credits/minute limit means subsequent calls wait ~60 s for the
+window to clear, so a full scan takes about **2 minutes**. Subsequent
+scans within `refresh_interval` are served from cache and make zero API
+calls.
+
+### Yahoo Finance — Unofficial
+
+No published rate limits. Used only for symbols mapped in
+`_YF_FALLBACK_MAP` (currently gold and silver futures). Daily/weekly
+intervals have unlimited history; intraday is capped (1m: last 7 days,
+15m–1h: last 60 days), which is more than enough for the engine's
+~210 bar requirement.
+
+### `--watch` mode
+
+`main.py --watch` re-scans every 5 minutes. With 7 pairs that's
+21 credits × 12 = 252 credits/hour — still within the daily limit if
+you run it for under ~3 hours. For longer runs, scope down with
+`--pairs` or stop and restart on demand.
+
+---
+
+## Configuration
+
+Most user-tunable values live in [config.py](config.py):
+
+- `MAJOR_PAIRS`, `CROSS_PAIRS` — which symbols `/scan` and `--all` cover
+- `SYMBOL_DISPLAY` — pretty names for messages
+- `PIP_VALUES` — per-symbol pip value override (gold = 0.01, JPY = 0.01)
+- `PlatformConfig` defaults — lookback days, refresh interval, indicator periods, MTF threshold, R:R ratio
+
+Bot-specific knobs in [telegram_bot.py](telegram_bot.py):
+
+- `SCAN_TIMEFRAMES` — which primary timeframes the bot reports on (default: `["M15", "H1"]`)
+- `_TF_CONFIRMATIONS` — confirmation TFs per primary
+- `_YF_FALLBACK_MAP` (in [data_provider.py](data_provider.py)) — Yahoo symbol mapping

config.py

@@ -48,7 +48,7 @@ CROSS_PAIRS = [
     "EURCHF=X", "CADJPY=X",
 ]
 
-# Yahoo Finance symbol mapping for display
+# Symbol display names
 SYMBOL_DISPLAY = {
     "EURUSD=X": "EUR/USD", "GBPUSD=X": "GBP/USD",
     "USDJPY=X": "USD/JPY", "USDCHF=X": "USD/CHF",
@@ -59,6 +59,11 @@ SYMBOL_DISPLAY = {
     "CADJPY=X": "CAD/JPY",
 }
 
+# Pip values per symbol (used for trade setup risk/reward in pips)
+# Forex defaults: 0.01 for JPY pairs, 0.0001 for everything else.
+# Override here for any pair that doesn't follow that rule.
+PIP_VALUES: Dict[str, float] = {}
+
 
 @dataclass
 class TechnicalConfig:
@@ -178,7 +183,7 @@ class PlatformConfig:
     # Analysis timeframes (primary + confirmation)
     primary_timeframe: Timeframe = Timeframe.H1
     confirmation_timeframes: List[Timeframe] = field(
-        default_factory=lambda: [Timeframe.M15, Timeframe.H4]
+        default_factory=lambda: [Timeframe.M15, Timeframe.D1]
     )
 
     # Data source
@@ -187,9 +192,25 @@ class PlatformConfig:
     # Twelve Data API key (overridden by TWELVE_DATA_API_KEY env var)
     twelve_data_api_key: str = "f563b22ffdfd4924955decbe1a8c8701"
 
-    # Dashboard refresh interval (seconds)
+    # Dashboard refresh interval (seconds) — used as a fallback when a
+    # timeframe isn't covered by cache_ttl_per_tf below.
     refresh_interval: int = 300  # 5 minutes
 
+    # Per-timeframe cache TTL (seconds). A 1-week candle changes less
+    # often than a 15-minute candle, so caching them all for the same
+    # 5 minutes wastes API credits. Tuned to roughly match each TF's
+    # bar interval so we re-fetch about once per new bar.
+    cache_ttl_per_tf: Dict[Timeframe, int] = field(default_factory=lambda: {
+        Timeframe.M1:  60,         # 1 minute
+        Timeframe.M5:  150,        # 2.5 minutes
+        Timeframe.M15: 300,        # 5 minutes
+        Timeframe.M30: 600,        # 10 minutes
+        Timeframe.H1:  1800,       # 30 minutes
+        Timeframe.H4:  7200,       # 2 hours
+        Timeframe.D1:  14400,      # 4 hours
+        Timeframe.W1:  86400,      # 24 hours
+    })
+
     # Logging
     log_level: str = "INFO"
     log_signals_to_file: bool = True

data_provider.py

@@ -1,25 +1,26 @@
 """
 Forex Signal Intelligence Platform - Data Provider
 ====================================================
-Retrieves and validates market data from Twelve Data API.
+Retrieves and validates currency-pair OHLCV from the Twelve Data API.
-Falls back to synthetic GBM data if the API is unavailable.
+Raises DataUnavailableError when the API is unreachable or credits exhausted.
+No fallback data source — forex pairs only.
 """
 
-import hashlib
 import logging
 import os
+import threading
 import time
 from collections import deque
 from datetime import datetime, timedelta, timezone
 from typing import Dict, List, Optional
 
-import numpy as np
 import pandas as pd
 import requests
 
-from config import PlatformConfig, Timeframe, SYMBOL_DISPLAY
+from config import PlatformConfig, Timeframe, SYMBOL_DISPLAY, PIP_VALUES
 
 logger = logging.getLogger(__name__)
+_no_data_warned: set = set()  # suppress duplicate "no data" warnings per symbol
 
 
 class DataIntegrityError(Exception):
@@ -27,6 +28,20 @@ class DataIntegrityError(Exception):
     pass
 
 
+class DataUnavailableError(Exception):
+    """Raised when live market data cannot be fetched (no key, rate limit, API error)."""
+    pass
+
+
+# ── Module-level shared rate limiter ───────────────────────────────
+# All ForexDataProvider instances share these so concurrent scans
+# don't each think they have a full 8-credit budget.
+_rate_lock          = threading.Lock()
+_rate_request_times: deque = deque()
+_rate_daily_credits: int   = 0
+_rate_daily_reset:   float = 0.0
+
+
 class ForexDataProvider:
     """
     Forex data provider backed by Twelve Data API.
@@ -60,9 +75,10 @@ class ForexDataProvider:
         Timeframe.W1:  0.143,
     }
 
-    # Free plan: 8 API credits per minute (each symbol = 1 credit)
+    # Free plan: 8 credits/minute, 800 credits/day
     RATE_LIMIT = 8
     RATE_WINDOW = 60.0
+    DAILY_LIMIT = 800
 
     def __init__(self, config: PlatformConfig):
         self.config = config
@@ -72,11 +88,12 @@ class ForexDataProvider:
         )
         self._cache: Dict[str, pd.DataFrame] = {}
         self._cache_timestamps: Dict[str, float] = {}
-        self._request_times: deque = deque()  # timestamps of recent API credits used
         self._session = requests.Session()
         self._session.headers.update({
-            "User-Agent": "Mozilla/5.0 (compatible; ForexIntel/1.0)"
+            "User-Agent": "Mozilla/5.0 (compatible; ForexIntel/1.0)",
         })
+        if self._api_key:
+            self._session.headers["Authorization"] = f"apikey {self._api_key}"
 
     @staticmethod
     def _to_td_symbol(symbol: str) -> str:
@@ -84,6 +101,10 @@ class ForexDataProvider:
         clean = symbol.replace("=X", "")
         return f"{clean[:3]}/{clean[3:]}"
 
+    def _cache_ttl(self, timeframe: Timeframe) -> int:
+        """Cache TTL for a given timeframe — falls back to refresh_interval."""
+        return self.config.cache_ttl_per_tf.get(timeframe, self.config.refresh_interval)
+
     def fetch_ohlcv(
         self,
         symbol: str,
@@ -101,10 +122,11 @@ class ForexDataProvider:
 
         cache_key = f"{symbol}_{timeframe.value}"
         now = time.time()
+        ttl = self._cache_ttl(timeframe)
 
         if (
             cache_key in self._cache_timestamps
-            and (now - self._cache_timestamps[cache_key]) < self.config.refresh_interval
+            and (now - self._cache_timestamps[cache_key]) < ttl
             and cache_key in self._cache
         ):
             cached = self._cache.get(cache_key)
@@ -135,9 +157,24 @@ class ForexDataProvider:
         if lookback_days is None:
             lookback_days = self.config.data_lookback_days
 
+        td_symbols = list(symbols)
+        if not td_symbols:
+            return
+
+        now = time.time()
         for tf in timeframes:
+            ttl = self._cache_ttl(tf)
+            cached = all(
+                f"{s}_{tf.value}" in self._cache_timestamps
+                and (now - self._cache_timestamps[f"{s}_{tf.value}"]) < ttl
+                and not self._cache.get(f"{s}_{tf.value}", pd.DataFrame()).empty
+                for s in td_symbols
+            )
+            if cached:
+                logger.debug(f"Skipping prefetch for {tf.value} — all {len(td_symbols)} symbols cached (TTL {ttl}s)")
+                continue
             try:
-                self._batch_fetch_timeframe(symbols, tf, lookback_days)
+                self._batch_fetch_timeframe(td_symbols, tf, lookback_days)
                 time.sleep(1.0)  # brief pause between TF requests
             except Exception as e:
                 logger.warning(f"Batch prefetch failed for {tf.value}: {e}")
@@ -189,10 +226,12 @@ class ForexDataProvider:
         """Return display name and pip value for a pair."""
         display = SYMBOL_DISPLAY.get(symbol, symbol)
         is_jpy = "JPY" in symbol
+        # Commodity-specific pip values take priority, then JPY pairs, then standard forex
+        pip_value = PIP_VALUES.get(symbol, 0.01 if is_jpy else 0.0001)
         return {
             "symbol": symbol,
             "display": display,
-            "pip_value": 0.01 if is_jpy else 0.0001,
+            "pip_value": pip_value,
             "is_jpy_pair": is_jpy,
         }
 
@@ -200,31 +239,64 @@ class ForexDataProvider:
 
     def _throttle(self, credits: int = 1) -> None:
         """
-        Block until `credits` API credits are available.
+        Block until `credits` per-minute API credits are available.
+        Raises DataUnavailableError if the daily limit is exhausted.
 
-        Twelve Data free tier: 8 credits/minute. Each symbol in any request
+        Uses module-level state so all ForexDataProvider instances (including
-        (single or batch) consumes 1 credit. This method enforces the limit
+        those running in parallel threads) share the same credit accounting.
-        regardless of whether requests are batched or individual.
+
+        Twelve Data free tier: 8 credits/minute, 800 credits/day.
         """
-        while True:
+        global _rate_daily_credits, _rate_daily_reset
+
+        with _rate_lock:
             now = time.time()
-            # Evict credits older than the rate window
-            while self._request_times and now - self._request_times[0] >= self.RATE_WINDOW:
-                self._request_times.popleft()
 
-            available = self.RATE_LIMIT - len(self._request_times)
+            # Reset daily counter after 24 hours
-            if available >= credits:
+            if _rate_daily_reset == 0.0:
-                for _ in range(credits):
+                _rate_daily_reset = now + 86400
-                    self._request_times.append(time.time())
+            if now >= _rate_daily_reset:
-                return
+                _rate_daily_credits = 0
+                _rate_daily_reset = now + 86400
 
-            # Wait until the oldest credit expires
+            # Daily limit check — raise immediately, no fallback
-            wait = (self._request_times[0] + self.RATE_WINDOW) - now + 0.2
+            if _rate_daily_credits + credits > self.DAILY_LIMIT:
-            logger.info(
+                raise DataUnavailableError(
-                f"Rate limit: {len(self._request_times)}/{self.RATE_LIMIT} credits used "
+                    f"Twelve Data daily API limit reached "
-                f"— waiting {wait:.1f}s for next window..."
+                    f"({_rate_daily_credits}/{self.DAILY_LIMIT} credits). "
-            )
+                    "Resets in 24 hours."
-            time.sleep(wait)
+                )
+
+            if _rate_daily_credits + credits > self.DAILY_LIMIT * 0.8:
+                logger.warning(
+                    f"Approaching daily API limit: "
+                    f"{_rate_daily_credits}/{self.DAILY_LIMIT} credits used."
+                )
+
+            # Per-minute rate limit — wait inside the lock so other threads
+            # queue up rather than over-counting available credits.
+            while True:
+                now = time.time()
+                while _rate_request_times and now - _rate_request_times[0] >= self.RATE_WINDOW:
+                    _rate_request_times.popleft()
+
+                available = self.RATE_LIMIT - len(_rate_request_times)
+                if available >= credits:
+                    ts = time.time()
+                    for _ in range(credits):
+                        _rate_request_times.append(ts)
+                    _rate_daily_credits += credits
+                    return
+
+                wait = (_rate_request_times[0] + self.RATE_WINDOW) - now + 0.2
+                logger.info(
+                    f"Rate limit: {len(_rate_request_times)}/{self.RATE_LIMIT} credits used "
+                    f"— waiting {wait:.1f}s for next window…"
+                )
+                # Release lock while sleeping so other threads can check state
+                _rate_lock.release()
+                time.sleep(wait)
+                _rate_lock.acquire()
 
     def _batch_fetch_timeframe(
         self, symbols: List[str], timeframe: Timeframe, lookback_days: int
@@ -244,7 +316,8 @@ class ForexDataProvider:
         )
         outputsize = max(outputsize, 300)
 
-        # Each symbol in a batch request costs 1 credit
+        # Each symbol in a batch request costs 1 credit — raises DataUnavailableError
+        # if daily limit reached; per-minute waits are handled inside _throttle.
         self._throttle(credits=len(symbols))
 
         resp = self._session.get(
@@ -253,7 +326,6 @@ class ForexDataProvider:
                 "symbol": ",".join(td_symbols),
                 "interval": interval,
                 "outputsize": outputsize,
-                "apikey": self._api_key,
                 "timezone": "UTC",
                 "format": "json",
             },
@@ -281,19 +353,22 @@ class ForexDataProvider:
 
                 values = sym_data.get("values")
                 if not values:
-                    logger.warning(f"No values in batch response for {td_sym}")
+                    if td_sym not in _no_data_warned:
+                        logger.warning(f"No data from Twelve Data for {td_sym} — symbol may require a paid plan.")
+                        _no_data_warned.add(td_sym)
                     continue
 
                 records = [
                     {
                         "Datetime": pd.Timestamp(v["datetime"], tz="UTC"),
-                        "Open":   float(v["open"]),
+                        "Open":   float(v["open"])   if v.get("open")   is not None else None,
-                        "High":   float(v["high"]),
+                        "High":   float(v["high"])   if v.get("high")   is not None else None,
-                        "Low":    float(v["low"]),
+                        "Low":    float(v["low"])    if v.get("low")    is not None else None,
-                        "Close":  float(v["close"]),
+                        "Close":  float(v["close"])  if v.get("close")  is not None else None,
-                        "Volume": float(v.get("volume") or 0),
+                        "Volume": float(v["volume"]) if v.get("volume") is not None else 0.0,
                     }
                     for v in reversed(values)
+                    if v.get("datetime")
                 ]
                 df = pd.DataFrame(records).set_index("Datetime")
                 df.index.name = "Datetime"
@@ -302,7 +377,7 @@ class ForexDataProvider:
                 cache_key = f"{symbol}_{timeframe.value}"
                 self._cache[cache_key] = df
                 self._cache_timestamps[cache_key] = now
-                logger.debug(f"Batch cached {len(df)} bars for {td_sym} @ {interval}")
+                logger.info(f"[live] {td_sym} {interval}: {len(df)} bars, last close={df['Close'].iloc[-1]:.5f}")
 
             except Exception as e:
                 logger.warning(f"Failed to parse batch data for {td_sym}: {e}")
@@ -312,12 +387,12 @@ class ForexDataProvider:
     ) -> pd.DataFrame:
         """
         Fetch OHLCV from Twelve Data REST API.
-        Falls back to synthetic GBM data if API is unavailable or key is missing.
+        Raises DataUnavailableError when data cannot be retrieved.
         """
         if not self._api_key:
-            logger.warning("No Twelve Data API key — using synthetic data. "
+            raise DataUnavailableError(
-                           "Set TWELVE_DATA_API_KEY env var or config.twelve_data_api_key.")
+                "No Twelve Data API key — set the TWELVE_DATA_API_KEY environment variable."
-            return self._generate_synthetic_data(symbol, timeframe, lookback_days)
+            )
 
         td_symbol = self._to_td_symbol(symbol)
         interval = self.TD_INTERVALS[timeframe]
@@ -327,6 +402,7 @@ class ForexDataProvider:
         )
         outputsize = max(outputsize, 300)
 
+        # Raises DataUnavailableError on daily limit; blocks on per-minute limit.
         self._throttle(credits=1)
 
         try:
@@ -336,7 +412,6 @@ class ForexDataProvider:
                     "symbol": td_symbol,
                     "interval": interval,
                     "outputsize": outputsize,
-                    "apikey": self._api_key,
                     "timezone": "UTC",
                     "format": "json",
                 },
@@ -346,138 +421,40 @@ class ForexDataProvider:
             data = resp.json()
 
             if data.get("status") == "error":
-                raise DataIntegrityError(
+                raise DataUnavailableError(
                     f"Twelve Data API error for {td_symbol}: {data.get('message', 'unknown')}"
                 )
 
             values = data.get("values")
             if not values:
-                raise DataIntegrityError(f"Empty response from Twelve Data for {td_symbol}")
+                raise DataUnavailableError(
+                    f"No data returned for {td_symbol} — symbol may require a paid plan."
+                )
 
-            # Values arrive newest-first; reverse to chronological order
             records = [
                 {
                     "Datetime": pd.Timestamp(v["datetime"], tz="UTC"),
-                    "Open":   float(v["open"]),
+                    "Open":   float(v["open"])   if v.get("open")   is not None else None,
-                    "High":   float(v["high"]),
+                    "High":   float(v["high"])   if v.get("high")   is not None else None,
-                    "Low":    float(v["low"]),
+                    "Low":    float(v["low"])    if v.get("low")    is not None else None,
-                    "Close":  float(v["close"]),
+                    "Close":  float(v["close"])  if v.get("close")  is not None else None,
-                    "Volume": float(v.get("volume") or 0),
+                    "Volume": float(v["volume"]) if v.get("volume") is not None else 0.0,
                 }
                 for v in reversed(values)
+                if v.get("datetime")
             ]
 
             df = pd.DataFrame(records).set_index("Datetime")
             df.index.name = "Datetime"
-            logger.debug(f"Fetched {len(df)} bars for {td_symbol} @ {interval}")
+            logger.info(f"[live] {td_symbol} {interval}: {len(df)} bars, last close={df['Close'].iloc[-1]:.5f}")
             return df
 
-        except DataIntegrityError:
+        except (DataUnavailableError, DataIntegrityError):
             raise
         except Exception as e:
-            logger.info(f"Twelve Data fetch failed for {symbol} ({e}), using synthetic data.")
+            raise DataUnavailableError(
-            return self._generate_synthetic_data(symbol, timeframe, lookback_days)
+                f"Failed to fetch {td_symbol}: {e}"
-
+            ) from e
-    def _generate_synthetic_data(
-        self, symbol: str, timeframe: Timeframe, lookback_days: int
-    ) -> pd.DataFrame:
-        """
-        Generate realistic synthetic Forex OHLCV using GBM with mean-reversion.
-        Used as a fallback when live data is unavailable.
-        """
-        pair_params = {
-            "EURUSD=X": {"price": 1.1350, "vol": 0.07, "drift": -0.001},
-            "GBPUSD=X": {"price": 1.3200, "vol": 0.08, "drift":  0.002},
-            "USDJPY=X": {"price": 144.50, "vol": 0.09, "drift":  0.003},
-            "USDCHF=X": {"price": 0.8850, "vol": 0.07, "drift": -0.001},
-            "AUDUSD=X": {"price": 0.6550, "vol": 0.10, "drift": -0.002},
-            "USDCAD=X": {"price": 1.3650, "vol": 0.07, "drift":  0.001},
-            "NZDUSD=X": {"price": 0.6100, "vol": 0.10, "drift": -0.001},
-            "EURGBP=X": {"price": 0.8590, "vol": 0.06, "drift":  0.000},
-            "EURJPY=X": {"price": 163.90, "vol": 0.09, "drift":  0.002},
-            "GBPJPY=X": {"price": 190.70, "vol": 0.11, "drift":  0.003},
-            "AUDJPY=X": {"price": 94.60,  "vol": 0.11, "drift":  0.001},
-            "EURCHF=X": {"price": 0.9380, "vol": 0.05, "drift":  0.000},
-            "CADJPY=X": {"price": 105.80, "vol": 0.09, "drift":  0.001},
-        }
-
-        params = pair_params.get(symbol, {"price": 1.0000, "vol": 0.08, "drift": 0.000})
-        base_price = params["price"]
-        annual_vol = params["vol"]
-        annual_drift = params["drift"]
-
-        tf_minutes = {
-            Timeframe.M1: 1, Timeframe.M5: 5, Timeframe.M15: 15,
-            Timeframe.M30: 30, Timeframe.H1: 60, Timeframe.H4: 240,
-            Timeframe.D1: 1440, Timeframe.W1: 10080,
-        }
-        bar_minutes = tf_minutes.get(timeframe, 60)
-        bars_per_year = (365.25 * 24 * 60) / bar_minutes
-        dt = 1.0 / bars_per_year
-
-        n_bars = min(int(lookback_days * 24 * 60 / bar_minutes), 2000)
-        n_bars = max(n_bars, 300)
-
-        seed_base = int(hashlib.md5(symbol.encode()).hexdigest()[:8], 16)
-        time_component = int(time.time()) // 3600
-        np.random.seed((seed_base + time_component) % (2**31))
-
-        bar_vol = annual_vol * np.sqrt(dt)
-        bar_drift = annual_drift * dt
-
-        regime_len = np.random.randint(20, 80)
-        trend_strength = np.random.normal(0, 0.3)
-
-        price = base_price
-        prices = [price]
-
-        for i in range(1, n_bars):
-            if i % regime_len == 0:
-                regime_len = np.random.randint(20, 80)
-                trend_strength = np.random.normal(0, 0.3)
-
-            mr_force = -0.001 * (price - base_price) / base_price
-            ret = bar_drift + trend_strength * dt + mr_force + bar_vol * np.random.randn()
-
-            if np.random.random() < 0.02:
-                ret *= np.random.uniform(2.0, 4.0)
-
-            price *= (1 + ret)
-            prices.append(price)
-
-        prices = np.array(prices)
-        opens = np.roll(prices, 1)
-        opens[0] = prices[0]
-
-        typical_range = bar_vol * base_price * 1.5
-        highs = np.maximum(prices, opens) + np.abs(np.random.normal(0, typical_range * 0.6, n_bars))
-        lows  = np.minimum(prices, opens) - np.abs(np.random.normal(0, typical_range * 0.6, n_bars))
-        highs = np.maximum(highs, np.maximum(opens, prices))
-        lows  = np.minimum(lows,  np.minimum(opens, prices))
-
-        volume = np.random.lognormal(mean=10, sigma=0.5, size=n_bars).astype(int)
-
-        freq_map = {
-            Timeframe.M1: "1min",  Timeframe.M5: "5min",
-            Timeframe.M15: "15min", Timeframe.M30: "30min",
-            Timeframe.H1: "1h",    Timeframe.H4: "4h",
-            Timeframe.D1: "D",     Timeframe.W1: "W",
-        }
-        freq = freq_map.get(timeframe, "1h")
-        end_time = pd.Timestamp.now(tz="UTC").floor(f"{bar_minutes}min")
-        index = pd.date_range(end=end_time, periods=n_bars, freq=freq, tz="UTC")
-
-        df = pd.DataFrame({
-            "Open": opens, "High": highs, "Low": lows,
-            "Close": prices, "Volume": volume,
-        }, index=index)
-        df.index.name = "Datetime"
-
-        logger.info(
-            f"Generated {n_bars} synthetic {timeframe.value} bars for "
-            f"{SYMBOL_DISPLAY.get(symbol, symbol)} @ {prices[-1]:.5f}"
-        )
-        return df
 
     def _validate_and_clean(
         self, df: pd.DataFrame, symbol: str, timeframe: Timeframe
@@ -522,7 +499,7 @@ class ForexDataProvider:
         if std_ret > 0:
             outlier_mask = (returns - returns.mean()).abs() > 10 * std_ret
             if outlier_mask.any():
-                logger.warning(f"{symbol}: Removing {outlier_mask.sum()} extreme outlier bars")
+                logger.debug(f"{symbol}: Removing {outlier_mask.sum()} extreme outlier bars")
                 df = df[~outlier_mask]
 
         min_required = max(

engine.py

@@ -499,7 +499,8 @@ class TechnicalEngine:
 
         # Compare to historical ATR
         if len(atr) >= 50:
-            atr_percentile = (atr.iloc[-1] > atr.iloc[-50:]).mean() * 100
+            # Exclude current bar so ATR isn't compared against itself
+            atr_percentile = (atr.iloc[-1] > atr.iloc[-50:-1]).mean() * 100
         else:
             atr_percentile = 50.0
 
@@ -844,7 +845,8 @@ class TechnicalEngine:
         - Confidence scaling
         """
         total_score = 0.0
-        total_weight = 0.0
+        score_weight = 0.0       # denominator for score: weight × confidence
+        weight_sum = 0.0         # denominator for confidence average: weight only
         total_confidence = 0.0
 
         regime_weights = {
@@ -863,16 +865,18 @@ class TechnicalEngine:
             regime_mult = active_regime_weights.get(name, 1.0)
             weight *= regime_mult
 
+            conf_factor = ind.confidence / 100.0
             signal_score = ind.signal.value  # -2 to +2
-            weighted_score = signal_score * weight * (ind.confidence / 100.0)
 
-            total_score += weighted_score
+            total_score += signal_score * weight * conf_factor
-            total_weight += weight
+            score_weight += weight * conf_factor
+            weight_sum += weight
             total_confidence += ind.confidence * weight
 
-        if total_weight > 0:
+        if score_weight > 0:
-            normalized_score = (total_score / total_weight) * 50  # scale to -100..+100
+            # Confidence-weighted average of signal_score (∈ [-2,+2]); ×50 → [-100,+100]
-            avg_confidence = total_confidence / total_weight
+            normalized_score = (total_score / score_weight) * 50
+            avg_confidence = total_confidence / weight_sum if weight_sum > 0 else 0
         else:
             normalized_score = 0
             avg_confidence = 0

main.py

@@ -380,11 +380,16 @@ Examples:
 
 
 def symbol_from_name(name: str) -> str:
-    """Convert pair name like 'EURUSD' to Yahoo symbol 'EURUSD=X'."""
+    """
-    name = name.upper().replace("/", "").replace("_", "")
+    Convert pair name like 'EURUSD' or 'XAUUSD' to internal symbol 'EURUSD=X'.
-    if not name.endswith("=X"):
+    Returns None for clearly invalid inputs (e.g. timeframe strings like '15M').
-        name += "=X"
+    """
-    return name
+    clean = name.upper().replace("/", "").replace("_", "").replace("=X", "")
+    if not clean.isalpha() or len(clean) < 5:
+        print(f"  Warning: '{name}' is not a valid pair symbol — skipping.")
+        print(f"  Hint: timeframes go in --timeframe (e.g. --timeframe M15), not --pairs.")
+        return None
+    return clean + "=X"
 
 
 def run_analysis(args) -> List[SignalReport]:
@@ -398,22 +403,27 @@ def run_analysis(args) -> List[SignalReport]:
     }
     config.primary_timeframe = tf_map[args.timeframe]
 
-    # Configure confirmation timeframes based on primary
+    # Configure confirmation timeframes based on primary.
+    # H1 uses M15 + D1 for high-conviction confluence (3-of-3 threshold).
     tf_confirmations = {
         "M5": [Timeframe.M1, Timeframe.M15],
-        "M15": [Timeframe.M5, Timeframe.H1],
+        "M15": [Timeframe.H1, Timeframe.D1],
-        "M30": [Timeframe.M15, Timeframe.H1],
+        "M30": [Timeframe.H1, Timeframe.D1],
-        "H1": [Timeframe.M15, Timeframe.H4],
+        "H1": [Timeframe.M15, Timeframe.D1],
         "H4": [Timeframe.H1, Timeframe.D1],
         "D1": [Timeframe.H4, Timeframe.W1],
     }
     config.confirmation_timeframes = tf_confirmations.get(
-        args.timeframe, [Timeframe.M15, Timeframe.H4]
+        args.timeframe, [Timeframe.M15, Timeframe.D1]
     )
 
     # Configure pairs
     if args.pairs:
-        config.watch_pairs = [symbol_from_name(p) for p in args.pairs]
+        symbols = [symbol_from_name(p) for p in args.pairs]
+        config.watch_pairs = [s for s in symbols if s is not None]
+        if not config.watch_pairs:
+            print("  Error: No valid pair symbols provided. Exiting.")
+            sys.exit(1)
     elif args.all:
         config.watch_pairs = MAJOR_PAIRS + CROSS_PAIRS
     else:

news_engine.py

@@ -1,6 +1,4 @@
 """
-Forex Signal Intelligence Platform - News & Sentiment Engine
-=============================================================
 Monitors economic calendar events and news sentiment to provide
 fundamental context for technical signals.
 
@@ -8,7 +6,9 @@ Key principle: High-impact news events override technical signals.
 A perfect technical setup can be destroyed by an unexpected NFP print.
 """
 
+import json
 import logging
+import os
 import xml.etree.ElementTree as ET
 from dataclasses import dataclass, field
 from datetime import datetime, timedelta, timezone
@@ -16,6 +16,23 @@ from typing import Dict, List, Optional, Tuple
 
 import requests
 
+# Load .env so GEMINI_API_KEY is available without exporting it manually.
+try:
+    from dotenv import load_dotenv
+    load_dotenv()
+except ImportError:
+    pass
+
+# New unified Google GenAI SDK (replaces google-generativeai).
+try:
+    from google import genai
+    from google.genai import types as genai_types
+    _GENAI_AVAILABLE = True
+except ImportError:
+    genai = None
+    genai_types = None
+    _GENAI_AVAILABLE = False
+
 from config import NewsConfig
 
 logger = logging.getLogger(__name__)
@@ -73,12 +90,12 @@ class FundamentalContext:
 
 class NewsEngine:
     """
-    Economic calendar and news sentiment analyzer.
+    Economic calendar and news sentiment analyzer using Google Gemini API.
 
     Provides fundamental context that modifies technical signal confidence.
     """
 
-    # Keywords that indicate directional sentiment
+    # Keywords that indicate directional sentiment (kept for fallback and keyword extraction)
     BULLISH_KEYWORDS = {
         "rate hike", "hawkish", "strong growth", "beat expectations",
         "above forecast", "surplus", "expansion", "employment gains",
@@ -114,6 +131,11 @@ class NewsEngine:
         "boe rate", "rba rate", "snb rate", "boc rate",
     }
 
+    # gemini-2.5-flash-lite has a much higher free-tier quota (1000 RPD vs 20
+    # for gemini-2.5-flash) and is more than capable for headline classification.
+    GEMINI_MODEL = "gemini-2.5-flash-lite"
+    BATCH_MAX_HEADLINES = 60  # cap per Gemini call to keep prompts small
+
     def __init__(self, config: NewsConfig):
         self.config = config
         self._session = requests.Session()
@@ -121,6 +143,25 @@ class NewsEngine:
             "User-Agent": "Mozilla/5.0 (compatible; ForexIntel/1.0)"
         })
 
+        # In-memory cache for Gemini-scored headlines so repeat fetches across
+        # /scan calls don't re-spend tokens on the same RSS items.
+        self._sentiment_cache: Dict[int, float] = {}
+
+        # Initialize Google Gemini via the new unified google-genai SDK.
+        api_key = os.environ.get("GEMINI_API_KEY")
+        self.gemini_client = None
+        if not _GENAI_AVAILABLE:
+            logger.warning("google-genai SDK not installed — falling back to keyword sentiment.")
+        elif not api_key:
+            logger.warning("GEMINI_API_KEY not set — falling back to keyword sentiment.")
+        else:
+            try:
+                self.gemini_client = genai.Client(api_key=api_key)
+                logger.info(f"Google Gemini ({self.GEMINI_MODEL}) configured.")
+            except Exception as e:
+                logger.error(f"Failed to configure Gemini: {e}")
+                self.gemini_client = None
+
     def get_fundamental_context(
         self, currencies: Optional[List[str]] = None
     ) -> FundamentalContext:
@@ -191,7 +232,6 @@ class NewsEngine:
         """
         Get event risk level for a specific pair.
         Returns (risk_level, description).
-        risk_level: "clear", "caution", "danger"
         """
         clean = pair_symbol.replace("=X", "")
         base = clean[:3]
@@ -206,7 +246,7 @@ class NewsEngine:
                 else:
                     return "caution", f"{event.currency} {event.event} in {mins:.0f}min"
 
-        # Check for recent high-impact releases (within post-event buffer)
+        # Check for recent high-impact releases
         for event in context.events_today:
             if event.is_high_impact and event.currency in (base, quote):
                 if not event.is_upcoming and event.minutes_until > -self.config.post_event_buffer_minutes:
@@ -220,7 +260,6 @@ class NewsEngine:
         """
         Get net sentiment for a pair based on news.
         Returns (score, description) where score is -1 to +1.
-        Positive = bullish for the pair, negative = bearish.
         """
         clean = pair_symbol.replace("=X", "")
         base = clean[:3]
@@ -248,11 +287,6 @@ class NewsEngine:
     # ── Private Methods ────────────────────────────────────────────
 
     def _fetch_economic_calendar(self, currencies: List[str]) -> List[EconomicEvent]:
-        """
-        Fetch economic calendar events from a free API.
-        Returns an empty list (with a log warning) if unavailable —
-        we never generate fake events since that causes constant false alarms.
-        """
         try:
             events = self._try_fetch_calendar_api()
             if events:
@@ -260,14 +294,10 @@ class NewsEngine:
         except Exception as e:
             logger.debug(f"Calendar API fetch exception: {e}")
 
-        logger.info(
+        logger.info("Economic calendar data unavailable. Risk checks skipped.")
-            "Economic calendar data unavailable. "
-            "Event risk checks will be skipped for this cycle."
-        )
         return []
 
     def _try_fetch_calendar_api(self) -> List[EconomicEvent]:
-        """Attempt to fetch from the Forex Factory free JSON calendar."""
         events = []
         url = "https://nfs.faireconomy.media/ff_calendar_thisweek.json"
         resp = self._session.get(url, timeout=10)
@@ -280,7 +310,6 @@ class NewsEngine:
         for item in data:
             try:
                 impact = impact_map.get(item.get("impact", ""), "low")
-
                 date_str = item.get("date", "")
                 if not date_str:
                     continue
@@ -301,61 +330,169 @@ class NewsEngine:
         return events
 
     def _fetch_news_sentiment(self) -> List[NewsSentiment]:
-        """Fetch and analyze news headlines for sentiment."""
+        # Stage 1: collect all candidate (title, pub_date, source) from RSS feeds
-        headlines = []
+        # without scoring yet.
-
+        candidates: List[Tuple[str, datetime, str]] = []
         for feed_url in self.config.rss_feeds:
             try:
                 resp = self._session.get(feed_url, timeout=10)
                 if resp.status_code != 200:
                     continue
-
                 root = ET.fromstring(resp.content)
-
                 for item in root.iter("item"):
                     title_el = item.find("title")
                     pub_date_el = item.find("pubDate")
-
                     if title_el is None:
                         continue
-
                     title = title_el.text or ""
                     pub_date = self._parse_rss_date(
                         pub_date_el.text if pub_date_el is not None else None
                     )
-
-                    # Only recent headlines (last 24h)
                     if pub_date and (datetime.now(timezone.utc) - pub_date).total_seconds() > 86400:
                         continue
-
+                    if not self._extract_currencies(title):
-                    sentiment = self._analyze_headline_sentiment(title)
+                        continue  # skip headlines that don't mention any tracked currency
-                    currencies = self._extract_currencies(title)
+                    candidates.append((title, pub_date or datetime.now(timezone.utc), feed_url))
-
-                    if currencies:
-                        headlines.append(NewsSentiment(
-                            headline=title,
-                            source=feed_url,
-                            timestamp=pub_date or datetime.now(timezone.utc),
-                            currencies_mentioned=currencies,
-                            sentiment_score=sentiment,
-                            keywords=self._extract_keywords(title),
-                        ))
-
             except Exception as e:
                 logger.debug(f"Failed to fetch RSS from {feed_url}: {e}")
 
-        return headlines
+        # Stage 2: batch-score all relevant titles in one Gemini call (or as
+        # few as possible). Cached titles are served from memory; only the
+        # uncached subset is sent to the API.
+        unique_titles = list({t for t, _, _ in candidates})
+        scores = self._score_headlines_batch(unique_titles)
+
+        # Stage 3: build NewsSentiment objects using the scored map.
+        return [
+            NewsSentiment(
+                headline=title,
+                source=source,
+                timestamp=ts,
+                currencies_mentioned=self._extract_currencies(title),
+                sentiment_score=scores.get(title, 0.0),
+                keywords=self._extract_keywords(title),
+            )
+            for title, ts, source in candidates
+        ]
+
+    def _score_headlines_batch(self, headlines: List[str]) -> Dict[str, float]:
+        """
+        Score a list of headlines with a single Gemini API call.
+        Cached titles are returned from memory; only the uncached subset is
+        sent to the model. Falls back to keyword scoring on API failure.
+        """
+        out: Dict[str, float] = {}
+        to_score: List[str] = []
+        for h in headlines:
+            key = hash(h)
+            if key in self._sentiment_cache:
+                out[h] = self._sentiment_cache[key]
+            else:
+                to_score.append(h)
+
+        if not to_score:
+            return out
+
+        if self.gemini_client is None:
+            for h in to_score:
+                s = self._fallback_keyword_sentiment(h)
+                self._sentiment_cache[hash(h)] = s
+                out[h] = s
+            return out
+
+        # Process in chunks to keep the prompt reasonable.
+        for i in range(0, len(to_score), self.BATCH_MAX_HEADLINES):
+            chunk = to_score[i : i + self.BATCH_MAX_HEADLINES]
+            chunk_scores = self._gemini_score_chunk(chunk)
+            for h in chunk:
+                s = chunk_scores.get(h)
+                if s is None:
+                    s = self._fallback_keyword_sentiment(h)
+                self._sentiment_cache[hash(h)] = s
+                out[h] = s
+        return out
+
+    def _gemini_score_chunk(self, headlines: List[str]) -> Dict[str, float]:
+        """Single Gemini call returning {headline: score} for the chunk."""
+        numbered = "\n".join(f'{i + 1}. "{h}"' for i, h in enumerate(headlines))
+        prompt = (
+            "You are an expert Forex fundamental analyst. Score each headline "
+            "from -1.0 (extremely bearish for its primary currency) to +1.0 "
+            "(extremely bullish). Use 0.0 if neutral or off-topic. Consider "
+            "context: surprises beating forecast = bullish; misses = bearish; "
+            'central bank "pause" or "patient" = dovish/bearish; "hawkish" or '
+            '"hike" = bullish.\n\n'
+            f"Headlines:\n{numbered}\n\n"
+            'Respond ONLY with a JSON array, one object per headline, in order:\n'
+            '[{"id": 1, "score": 0.5}, {"id": 2, "score": -0.3}, ...]'
+        )
+        try:
+            response = self.gemini_client.models.generate_content(
+                model=self.GEMINI_MODEL,
+                contents=prompt,
+                config=genai_types.GenerateContentConfig(
+                    response_mime_type="application/json",
+                    temperature=0.0,
+                ),
+            )
+            data = json.loads(response.text)
+            scores: Dict[str, float] = {}
+            for entry in data:
+                idx = int(entry.get("id", 0)) - 1
+                if 0 <= idx < len(headlines):
+                    raw = float(entry.get("score", 0.0))
+                    scores[headlines[idx]] = max(-1.0, min(1.0, raw))
+            return scores
+        except Exception as e:
+            logger.warning(f"Gemini batch scoring failed ({len(headlines)} headlines): {e}")
+            return {}
 
     def _analyze_headline_sentiment(self, headline: str) -> float:
         """
-        Simple keyword-based sentiment analysis.
+        Score a single headline -1.0 to +1.0 via Google Gemini.
-
+        Falls back to keyword sentiment if the API is unavailable or fails.
-        Returns -1.0 (very bearish) to +1.0 (very bullish).
+        Cached per-headline so repeat calls cost zero tokens.
-
-        Note: This is intentionally simple. NLP models like FinBERT
-        would be more accurate but require large dependencies. This catches
-        the most impactful sentiment signals from headlines.
         """
+        cache_key = hash(headline)
+        if cache_key in self._sentiment_cache:
+            return self._sentiment_cache[cache_key]
+
+        if self.gemini_client is None:
+            score = self._fallback_keyword_sentiment(headline)
+            self._sentiment_cache[cache_key] = score
+            return score
+
+        prompt = (
+            "You are an expert Forex fundamental analyst. "
+            "Analyze this financial news headline:\n"
+            f'"{headline}"\n\n'
+            "Determine the sentiment score on the primary currency mentioned.\n"
+            "-1.0 means extremely bearish (rate cuts, recession, weak data).\n"
+            " 1.0 means extremely bullish (rate hikes, strong growth, beats).\n"
+            " 0.0 means neutral or irrelevant.\n\n"
+            'Respond ONLY with valid JSON: {"score": 0.85}'
+        )
+
+        try:
+            response = self.gemini_client.models.generate_content(
+                model=self.GEMINI_MODEL,
+                contents=prompt,
+                config=genai_types.GenerateContentConfig(
+                    response_mime_type="application/json",
+                    temperature=0.0,
+                ),
+            )
+            data = json.loads(response.text)
+            score = max(-1.0, min(1.0, float(data.get("score", 0.0))))
+        except Exception as e:
+            logger.debug(f"Gemini API failed for '{headline[:60]}': {e}")
+            score = self._fallback_keyword_sentiment(headline)
+
+        self._sentiment_cache[cache_key] = score
+        return score
+
+    def _fallback_keyword_sentiment(self, headline: str) -> float:
+        """Original keyword-based fallback if Gemini API is unavailable or fails."""
         headline_lower = headline.lower()
         bullish_score = sum(1 for kw in self.BULLISH_KEYWORDS if kw in headline_lower)
         bearish_score = sum(1 for kw in self.BEARISH_KEYWORDS if kw in headline_lower)
@@ -367,7 +504,6 @@ class NewsEngine:
         return (bullish_score - bearish_score) / total
 
     def _extract_currencies(self, headline: str) -> List[str]:
-        """Extract mentioned currencies from a headline."""
         headline_lower = headline.lower()
         found = []
 
@@ -383,27 +519,23 @@ class NewsEngine:
         return list(set(found))
 
     def _extract_keywords(self, headline: str) -> List[str]:
-        """Extract relevant trading keywords from headline."""
         headline_lower = headline.lower()
         return [kw for kw in self.BULLISH_KEYWORDS | self.BEARISH_KEYWORDS if kw in headline_lower]
 
     @staticmethod
     def _parse_rss_date(date_str: Optional[str]) -> Optional[datetime]:
-        """Parse various RSS date formats into a timezone-aware datetime."""
         if not date_str:
             return None
 
         formats = [
-            "%a, %d %b %Y %H:%M:%S %z",   # RFC 822 with numeric tz offset
+            "%a, %d %b %Y %H:%M:%S %z",
-            "%a, %d %b %Y %H:%M:%S %Z",   # RFC 822 with tz abbreviation (GMT/UTC)
+            "%a, %d %b %Y %H:%M:%S %Z",
-            "%Y-%m-%dT%H:%M:%S%z",         # ISO 8601 with offset
+            "%Y-%m-%dT%H:%M:%S%z",
-            "%Y-%m-%d %H:%M:%S",           # ISO 8601 naive (assume UTC)
+            "%Y-%m-%d %H:%M:%S",
         ]
         for fmt in formats:
             try:
                 dt = datetime.strptime(date_str.strip(), fmt)
-                # If strptime already attached tzinfo (from %z), keep it.
-                # Otherwise the string had no timezone info — assume UTC.
                 if dt.tzinfo is not None:
                     return dt
                 return dt.replace(tzinfo=timezone.utc)
@@ -413,7 +545,6 @@ class NewsEngine:
 
     @staticmethod
     def _get_affected_pairs(currency: str) -> List[str]:
-        """Get all major pairs affected by a currency event."""
         pairs = {
             "USD": ["EURUSD=X", "GBPUSD=X", "USDJPY=X", "USDCHF=X", "AUDUSD=X", "USDCAD=X", "NZDUSD=X"],
             "EUR": ["EURUSD=X", "EURGBP=X", "EURJPY=X", "EURCHF=X"],

requirements.txt

@@ -0,0 +1,13 @@
+# Core data + math
+numpy>=1.24
+pandas>=2.0
+
+# HTTP
+requests>=2.28
+
+# News sentiment (Gemini LLM)
+google-genai>=1.0      # new unified Google GenAI SDK
+python-dotenv>=1.0     # loads GEMINI_API_KEY and TELEGRAM_BOT_TOKEN from .env
+
+# Telegram bot
+python-telegram-bot[job-queue]>=20.0

signal_aggregator.py

@@ -31,6 +31,12 @@ from news_engine import NewsEngine, FundamentalContext
 logger = logging.getLogger(__name__)
 
 
+# Per-symbol TF overrides. Currently empty — all forex pairs use the global
+# config.confirmation_timeframes. Add entries here only if a specific symbol
+# needs a different MTF set.
+_PER_SYMBOL_TFS: Dict[str, List[Timeframe]] = {}
+
+
 @dataclass
 class TradeSetup:
     """A complete trade setup recommendation."""
@@ -104,9 +110,9 @@ class SignalAggregator:
     fundamental analysis, and produces final graded signals.
     """
 
-    def __init__(self, config: PlatformConfig):
+    def __init__(self, config: PlatformConfig, data_provider: Optional[ForexDataProvider] = None):
         self.config = config
-        self.data_provider = ForexDataProvider(config)
+        self.data_provider = data_provider or ForexDataProvider(config)
         self.tech_engine = TechnicalEngine(config.technical)
         self.news_engine = NewsEngine(config.news)
 
@@ -118,11 +124,12 @@ class SignalAggregator:
         reports = []
         context = self.news_engine.get_fundamental_context()
 
-        # Batch-prefetch all pairs × all needed timeframes (3 API calls instead of 21)
+        # Batch-prefetch all pairs × all needed timeframes (3 API calls instead of 21).
-        all_timeframes = (
+        # Union of primary + per-symbol confirmation TFs across the watchlist.
-            [self.config.primary_timeframe] + self.config.confirmation_timeframes
+        tf_set = {self.config.primary_timeframe, *self.config.confirmation_timeframes}
-        )
+        for sym in self.config.watch_pairs:
-        self.data_provider.prefetch_batch(self.config.watch_pairs, all_timeframes)
+            tf_set.update(self._confirmation_tfs(sym))
+        self.data_provider.prefetch_batch(self.config.watch_pairs, list(tf_set))
 
         for symbol in self.config.watch_pairs:
             try:
@@ -179,7 +186,7 @@ class SignalAggregator:
             return report
 
         # ── 2. Confirmation Timeframe Analyses ─────────────────────
-        for tf in self.config.confirmation_timeframes:
+        for tf in self._confirmation_tfs(symbol):
             try:
                 cf_df = self.data_provider.fetch_ohlcv(symbol, tf)
                 cf_analysis = self.tech_engine.analyze(cf_df, symbol, tf.value)
@@ -202,7 +209,11 @@ class SignalAggregator:
         report.news_description = sent_desc
 
         # ── 5. Spread Estimate ─────────────────────────────────────
-        report.spread_estimate = self.data_provider.get_spread_estimate(symbol)
+        # Derive spread from already-fetched primary_df to avoid a separate M5 API call
+        if primary_df is not None and not primary_df.empty:
+            report.spread_estimate = float((primary_df["High"] - primary_df["Low"]).median())
+        else:
+            report.spread_estimate = self.data_provider.get_spread_estimate(symbol)
 
         # ── 6. Compute Final Signal ────────────────────────────────
         self._compute_final_signal(report)
@@ -216,6 +227,14 @@ class SignalAggregator:
 
         return report
 
+    def _confirmation_tfs(self, symbol: str) -> List[Timeframe]:
+        """Return confirmation TFs for a symbol, honouring per-symbol overrides."""
+        override = _PER_SYMBOL_TFS.get(symbol)
+        if override is None:
+            return self.config.confirmation_timeframes
+        primary = self.config.primary_timeframe
+        return [tf for tf in override if tf != primary]
+
     def _check_mtf_agreement(self, report: SignalReport) -> bool:
         """Check if multiple timeframes agree on direction."""
         signals = list(report.mtf_signals.values())
@@ -242,20 +261,21 @@ class SignalAggregator:
         confidence = report.primary_confidence
 
         # ── MTF adjustment ─────────────────────────────────────────
+        # Always blend primary with MTF average so the score is pulled toward
+        # consensus on disagreement (and barely moves on agreement, since the
+        # TFs already point the same way).
+        all_scores = list(report.mtf_scores.values())
+        if all_scores:
+            mtf_avg = sum(all_scores) / len(all_scores)
+            score = score * 0.6 + mtf_avg * 0.4
+
         if report.mtf_agreement:
-            # All timeframes agree — boost confidence and blend scores
             confidence = min(confidence + 15, 95)
-            all_scores = list(report.mtf_scores.values())
-            if all_scores:
-                mtf_avg = sum(all_scores) / len(all_scores)
-                score = score * 0.6 + mtf_avg * 0.4
         else:
-            # Timeframes disagree — reduce confidence
             confidence *= 0.7
             if len(report.mtf_scores) >= 2:
-                scores = list(report.mtf_scores.values())
+                has_bull = any(s > 10 for s in report.mtf_scores.values())
-                has_bull = any(s > 10 for s in scores)
+                has_bear = any(s < -10 for s in report.mtf_scores.values())
-                has_bear = any(s < -10 for s in scores)
                 if has_bull and has_bear:
                     report.warnings.append("Timeframes CONFLICT — mixed signals across timeframes")
                     confidence *= 0.6

telegram_bot.py

@@ -0,0 +1,728 @@
+#!/usr/bin/env python3
+"""
+Forex Signal Intelligence Platform - Telegram Bot
+==================================================
+Trade alert frontend for the signal engine.
+All timeframes are analyzed automatically — no TF parameter needed.
+
+Setup:
+    set TELEGRAM_BOT_TOKEN=your_token_here      (Windows CMD)
+    export TELEGRAM_BOT_TOKEN=your_token_here   (Linux/Mac)
+    python telegram_bot.py
+
+Commands:
+    /signal EURUSD  — Combined signal card across all timeframes
+    /scan           — Scan all pairs, compact Grade B+ summary
+    /help           — Command guide
+"""
+
+import asyncio
+import html
+import logging
+import os
+from concurrent.futures import ThreadPoolExecutor
+from datetime import datetime
+from typing import Dict, List, Optional, Tuple
+from zoneinfo import ZoneInfo
+
+# Load .env so TELEGRAM_BOT_TOKEN / GEMINI_API_KEY / TWELVE_DATA_API_KEY
+# don't need to be exported manually.
+try:
+    from dotenv import load_dotenv
+    load_dotenv()
+except ImportError:
+    pass
+
+from telegram import BotCommand, Update
+from telegram.constants import ParseMode
+from telegram.ext import Application, CommandHandler, ContextTypes
+
+from config import (
+    MAJOR_PAIRS, CROSS_PAIRS, SYMBOL_DISPLAY,
+    PlatformConfig, SignalStrength, Timeframe,
+)
+from data_provider import ForexDataProvider
+from signal_aggregator import SignalAggregator, SignalReport
+
+logger = logging.getLogger(__name__)
+
+# ── Config ──────────────────────────────────────────────────────────
+
+SCAN_TIMEFRAMES = ["M15", "H1"]  # two cards (M15 and H1) per /scan; cache shared
+
+# Pairs analyzed by the /scan command. Limited to the most liquid majors
+# to keep credits low and signal quality high.
+SCAN_PAIRS = ["EURUSD=X", "USDJPY=X", "GBPUSD=X", "AUDUSD=X"]
+
+# Auto-scan job: fires every AUTO_SCAN_INTERVAL seconds and pushes any A/A+
+# signal to ALERT_CHAT_ID (env var). 10 min cadence ≈ 595 credits/day on
+# cold-cache average — fits inside the 800/day Twelve Data free-tier budget.
+AUTO_SCAN_INTERVAL = 600
+ALERT_GRADES = {"A+", "A"}
+
+_TF_MAP = {
+    "M5":  Timeframe.M5,  "M15": Timeframe.M15,
+    "M30": Timeframe.M30, "H1":  Timeframe.H1,
+    "H4":  Timeframe.H4,  "D1":  Timeframe.D1,
+}
+
+# D1 kept as confirmation for higher-timeframe trend context.
+# W1 dropped — only the D1 scan used it, and we no longer scan D1.
+_TF_CONFIRMATIONS = {
+    "M15": [Timeframe.H1,  Timeframe.D1],
+    "H1":  [Timeframe.M15, Timeframe.D1],
+}
+
+_GRADE_RANK      = {"A+": 0, "A": 1, "B": 2, "C": 3, "D": 4, "F": 5}
+_GRADE_LEGEND    = "Grade scale: 🔥 A+ · ✅ A · 👍 B · ⚠️ C · 📉 D · ❌ F"
+_EST             = ZoneInfo("America/New_York")
+
+# ── Formatting helpers ──────────────────────────────────────────────
+
+def _now_est() -> str:
+    return datetime.now(_EST).strftime("%I:%M %p EST · %a %d %b %Y")
+
+
+
+def _signal_action(signal: SignalStrength) -> str:
+    return {
+        SignalStrength.STRONG_BUY:  "STRONG BUY",
+        SignalStrength.BUY:         "BUY",
+        SignalStrength.STRONG_SELL: "STRONG SELL",
+        SignalStrength.SELL:        "SELL",
+        SignalStrength.NEUTRAL:     "NEUTRAL",
+    }.get(signal, "NEUTRAL")
+
+
+def _grade_symbol(quality: str) -> str:
+    return {"A+": "🔥", "A": "✅", "B": "👍", "C": "⚠️", "D": "📉"}.get(quality, "❌")
+
+
+def _indicator_condition(ind) -> str:
+    desc = ind.description
+    if " — " in desc:
+        return desc.split(" — ", 1)[1].strip()
+    if " | " in desc:
+        return desc.split(" | ", 1)[0].strip()
+    return ind.signal.name.replace("_", " ").title()
+
+
+def _fmt_float(val: float) -> str:
+    if val == 0.0:
+        return "0"
+    av = abs(val)
+    if av >= 1000:
+        return f"{val:.1f}"
+    if av >= 10:
+        return f"{val:.2f}"
+    if av >= 0.01:
+        return f"{val:.4f}"
+    return f"{val:.6f}"
+
+
+def _h(text: str) -> str:
+    return html.escape(str(text))
+
+
+# ── Message builders ────────────────────────────────────────────────
+
+def _build_combined_signal(
+    results_by_tf: Dict[str, List[SignalReport]],
+    display: str,
+    now_str: str,
+) -> str:
+    """
+    Single message showing all timeframes stacked.
+    Entry/TP/SL always preserved; trimmed: per-card disclaimer,
+    MTF detail block (folded to one line), news unless non-neutral.
+    """
+    lines: List[str] = []
+    lines.append(f"🌐 🧠 <b>{_h(display)} — Signal Analysis</b>")
+    lines.append(f"🕐 {now_str}")
+    lines.append(f"<i>{_GRADE_LEGEND}</i>")
+
+    for tf in SCAN_TIMEFRAMES:
+        reports = results_by_tf.get(tf, [])
+        if not reports:
+            continue
+        r = reports[0]
+
+        # If primary TF data was unavailable the engine returns grade F with a warning.
+        # Show a clean "data unavailable" line instead of a garbled card.
+        if r.signal_quality == "F" and any(
+            "unavailable" in w.lower() or "api" in w.lower() or "credits" in w.lower()
+            for w in r.warnings
+        ):
+            lines.append("")
+            lines.append("────────────────────────")
+            lines.append(f"<b>{tf}</b> — ❌ Data unavailable")
+            if r.warnings:
+                lines.append(f"<i>{_h(r.warnings[0])}</i>")
+            continue
+
+        action = _signal_action(r.final_signal)
+        sym    = _grade_symbol(r.signal_quality)
+
+        lines.append("")
+        lines.append("────────────────────────")
+        lines.append(f"{sym} <b>{tf} · {action} · Grade {_h(r.signal_quality)}</b>")
+        lines.append("")
+
+        # ── Trade setup (always kept intact) ─────────────────────────
+        if r.trade_setup:
+            ts = r.trade_setup
+            lines.append(f"<b>Entry:</b> {ts.entry_price:.5f}")
+            lines.append(
+                f"<b>Take Profit:</b> {ts.take_profit_1:.5f}"
+                f"  <i>(+{ts.reward_pips:.1f} pips)</i>"
+            )
+            if abs(ts.take_profit_2 - ts.take_profit_1) > 1e-8:
+                lines.append(f"<b>Take Profit 2:</b> {ts.take_profit_2:.5f}")
+            lines.append(
+                f"<b>Stop Loss:</b> {ts.stop_loss:.5f}"
+                f"  <i>(-{ts.risk_pips:.1f} pips)</i>"
+            )
+            lines.append(f"<b>Risk:Reward:</b> 1:{ts.risk_reward:.1f}")
+            lines.append("")
+        else:
+            lines.append(f"<b>Current Price:</b> {r.current_price:.5f}")
+            lines.append(f"<b>Analysis:</b> {_h(r.action_recommendation)}")
+            lines.append("")
+
+        # ── Indicators ────────────────────────────────────────────────
+        if r.primary_analysis and r.primary_analysis.indicators:
+            for name, ind in r.primary_analysis.indicators.items():
+                if name.startswith("_") or ind.weight == 0:
+                    continue
+                lines.append(
+                    f"• {_h(ind.name)}: {_fmt_float(ind.value)}"
+                    f" ({_h(_indicator_condition(ind))})"
+                )
+
+        # ── MTF summary (one line) ────────────────────────────────────
+        agree_str = "✅ Aligned" if r.mtf_agreement else "⚠️ Divergent"
+        lines.append(
+            f"<i>MTF: {agree_str} | Score: {r.final_score:+.1f}"
+            f" | {_h(r.market_regime)} | Vol: {_h(r.volatility_regime)}</i>"
+        )
+
+        # ── Critical warnings only ────────────────────────────────────
+        for w in r.warnings:
+            if any(k in w for k in ("EVENT RISK", "CONFLICT", "EXTREME")):
+                lines.append(f"⚠️ {_h(w)}")
+
+        # ── Event / news only when relevant ──────────────────────────
+        if r.event_risk == "danger" and r.event_description:
+            lines.append(f"📅 <b>Event Risk:</b> {_h(r.event_description)}")
+        if r.news_description and r.news_description not in ("", "Neutral news sentiment"):
+            lines.append(f"📰 {_h(r.news_description)}")
+
+    lines.append("")
+    lines.append("<i>⚠️ Educational use only. Not financial advice.</i>")
+    return "\n".join(lines)
+
+
+def _build_scan_message(
+    actionable: List[Tuple[str, SignalReport]],
+    now_str: str,
+    tfs_str: str,
+) -> str:
+    """
+    Compact single message: one line header + one trade line per pair.
+    Full entry/TP/SL preserved; no indicator list (use /signal to drill in).
+    """
+    lines: List[str] = []
+    lines.append(f"📊 🧠 <b>Scan Complete</b> · {now_str}")
+    lines.append(f"Timeframes: <code>{_h(tfs_str)}</code> | Best signal per pair")
+    lines.append(f"<i>{_GRADE_LEGEND}</i>")
+    lines.append("")
+
+    for tf, r in actionable:
+        action = _signal_action(r.final_signal)
+        sym    = _grade_symbol(r.signal_quality)
+
+        lines.append(
+            f"{sym} <b>{_h(r.display_name)} · {action}</b>"
+            f" · {_h(r.signal_quality)} · {tf}"
+        )
+
+        if r.trade_setup:
+            ts = r.trade_setup
+            lines.append(
+                f"   Entry: {ts.entry_price:.5f} | "
+                f"TP: {ts.take_profit_1:.5f} (+{ts.reward_pips:.0f}p) | "
+                f"SL: {ts.stop_loss:.5f} (-{ts.risk_pips:.0f}p) | "
+                f"R:R 1:{ts.risk_reward:.1f}"
+            )
+        else:
+            lines.append("   Watch only")
+
+        # Show danger-level events inline
+        if r.event_risk == "danger" and r.event_description:
+            lines.append(f"   📅 {_h(r.event_description)}")
+
+        lines.append("")
+
+    if not actionable:
+        lines.append("No Grade B+ signals found across any timeframe.")
+        lines.append("")
+
+    lines.append("<i>⚠️ Educational use only. Not financial advice.</i>")
+    return "\n".join(lines)
+
+
+def _build_alert_card(tf: str, r: SignalReport, now_str: str) -> str:
+    """
+    Auto-scan alert message — focused on actionable trade info plus a
+    prominent news/event block (the user wants news context surfaced
+    on every A-grade alert).
+    """
+    lines: List[str] = []
+    sym    = _grade_symbol(r.signal_quality)
+    action = _signal_action(r.final_signal)
+
+    lines.append(
+        f"{sym} <b>{_h(r.signal_quality)} ALERT · {_h(r.display_name)}"
+        f" · {action} · {tf}</b>"
+    )
+    lines.append(f"🕐 {now_str}")
+    lines.append("")
+
+    # ── Trade setup ──────────────────────────────────────────────
+    if r.trade_setup:
+        ts = r.trade_setup
+        lines.append(f"<b>Entry:</b> {ts.entry_price:.5f}")
+        lines.append(
+            f"<b>Take Profit:</b> {ts.take_profit_1:.5f}"
+            f"  <i>(+{ts.reward_pips:.1f} pips)</i>"
+        )
+        if abs(ts.take_profit_2 - ts.take_profit_1) > 1e-8:
+            lines.append(f"<b>Take Profit 2:</b> {ts.take_profit_2:.5f}")
+        lines.append(
+            f"<b>Stop Loss:</b> {ts.stop_loss:.5f}"
+            f"  <i>(-{ts.risk_pips:.1f} pips)</i>"
+        )
+        lines.append(f"<b>Risk:Reward:</b> 1:{ts.risk_reward:.1f}")
+        lines.append("")
+    else:
+        lines.append(f"<b>Current Price:</b> {r.current_price:.5f}")
+        lines.append("")
+
+    # ── Top confluence reasons (top 3, direction-matched) ────────
+    if r.final_signal in (SignalStrength.BUY, SignalStrength.STRONG_BUY):
+        reasons = r.bullish_reasons[:3]
+    elif r.final_signal in (SignalStrength.SELL, SignalStrength.STRONG_SELL):
+        reasons = r.bearish_reasons[:3]
+    else:
+        reasons = []
+    if reasons:
+        lines.append("<b>📊 Confluence:</b>")
+        for reason in reasons:
+            lines.append(f"  • {_h(reason)}")
+        lines.append("")
+
+    # ── News & Events (always shown on alerts) ───────────────────
+    lines.append("<b>📰 News &amp; Events:</b>")
+
+    sentiment = r.news_sentiment
+    if sentiment > 0.2:
+        sent_label = f"📈 Bullish news bias <i>(score {sentiment:+.2f})</i>"
+    elif sentiment < -0.2:
+        sent_label = f"📉 Bearish news bias <i>(score {sentiment:+.2f})</i>"
+    else:
+        sent_label = f"📊 Neutral news <i>(score {sentiment:+.2f})</i>"
+    lines.append(f"  {sent_label}")
+    if r.news_description and r.news_description.strip().lower() not in (
+        "", "neutral news sentiment", "no recent news",
+    ):
+        lines.append(f"  <i>{_h(r.news_description)}</i>")
+
+    risk_label = {
+        "clear":   "✅ Clear (no major events nearby)",
+        "caution": "⚠️ Caution — upcoming event",
+        "danger":  "🚨 Danger — high-impact event imminent",
+    }.get(r.event_risk, _h(r.event_risk))
+    lines.append(f"  Event Risk: {risk_label}")
+    if r.event_description:
+        lines.append(f"  <i>{_h(r.event_description)}</i>")
+
+    # Sentiment vs technical alignment hint — useful one-liner for the trader
+    if abs(sentiment) > 0.2:
+        score_dir = 1 if r.final_score > 0 else -1 if r.final_score < 0 else 0
+        sent_dir  = 1 if sentiment > 0 else -1
+        if score_dir != 0 and score_dir == sent_dir:
+            lines.append("  ✅ News aligns with technical signal")
+        elif score_dir != 0:
+            lines.append("  ⚠️ News conflicts with technical signal")
+    lines.append("")
+
+    # ── MTF / regime / vol summary ────────────────────────────────
+    agree_str = "✅ Aligned" if r.mtf_agreement else "⚠️ Divergent"
+    lines.append(
+        f"<i>MTF: {agree_str} | Score: {r.final_score:+.1f}"
+        f" | Confidence: {r.final_confidence:.0f}%"
+        f" | {_h(r.market_regime)} | Vol: {_h(r.volatility_regime)}</i>"
+    )
+
+    # ── Critical warnings ────────────────────────────────────────
+    for w in r.warnings:
+        if any(k in w for k in ("EVENT RISK", "CONFLICT", "EXTREME")):
+            lines.append(f"⚠️ {_h(w)}")
+
+    lines.append("")
+    lines.append("<i>⚠️ Educational use only. Not financial advice.</i>")
+    return "\n".join(lines)
+
+
+# ══════════════════════════════════════════════════════════════════════
+#  Bot
+# ══════════════════════════════════════════════════════════════════════
+
+class ForexTelegramBot:
+    def __init__(self):
+        self._executor = ThreadPoolExecutor(
+            max_workers=len(SCAN_TIMEFRAMES), thread_name_prefix="forex"
+        )
+        # Persistent ForexDataProvider so the OHLCV cache survives across
+        # /scan and /signal calls. Without this each call burns API credits
+        # re-fetching the same bars and waits up to 60s on the rate limit.
+        self._provider = ForexDataProvider(PlatformConfig())
+
+        # Dedup keys for the auto-scan alert job: (symbol, tf, direction,
+        # candle_open_timestamp). New bar → new key → re-alert allowed if the
+        # setup persists. Bounded to avoid unbounded growth on long uptime.
+        self._alerted_keys: set = set()
+
+    # ── Engine helpers ────────────────────────────────────────────────
+
+    def _build_config(self, tf: str, pairs: List[str]) -> PlatformConfig:
+        cfg = PlatformConfig()
+        cfg.primary_timeframe       = _TF_MAP[tf]
+        cfg.confirmation_timeframes = _TF_CONFIRMATIONS.get(tf, [Timeframe.M15, Timeframe.D1])
+        cfg.watch_pairs             = list(pairs)
+        return cfg
+
+    def _run_scan(self, tf: str, pairs: List[str], provider: ForexDataProvider) -> List[SignalReport]:
+        return SignalAggregator(self._build_config(tf, pairs), data_provider=provider).analyze_all_pairs()
+
+    async def _analyze_all_tfs(
+        self, pairs: List[str], progress=None,
+    ) -> Dict[str, List[SignalReport]]:
+        """Single combined prefetch, then run all SCAN_TIMEFRAMES concurrently off the warm cache."""
+        # Collect all unique timeframes needed across every TF scan
+        all_tfs = list({
+            tf
+            for tf_name in SCAN_TIMEFRAMES
+            for tf in [_TF_MAP[tf_name]] + _TF_CONFIRMATIONS.get(tf_name, [])
+        })
+        provider = self._provider  # reuse across calls — see __init__
+
+        loop = asyncio.get_running_loop()
+        if progress:
+            await progress(f"📡 Fetching market data ({len(all_tfs)} timeframes × {len(pairs)} pairs)…")
+        # Run blocking prefetch off the event loop so we don't freeze the bot
+        await loop.run_in_executor(None, provider.prefetch_batch, list(pairs), all_tfs)
+
+        if progress:
+            await progress("🧠 Analyzing pairs across timeframes…")
+
+        tasks = {
+            tf: loop.run_in_executor(self._executor, self._run_scan, tf, list(pairs), provider)
+            for tf in SCAN_TIMEFRAMES
+        }
+        results: Dict[str, List[SignalReport]] = {}
+        for tf, task in tasks.items():
+            try:
+                results[tf] = await task
+            except Exception as e:
+                logger.error(f"TF {tf} scan failed: {e}", exc_info=True)
+                results[tf] = []
+        return results
+
+    @staticmethod
+    async def _send_html(update: Update, text: str):
+        """Send HTML, splitting into chunks at newlines if over 4096 chars."""
+        LIMIT = 4000
+        chunks: List[str] = []
+        while text:
+            if len(text) <= LIMIT:
+                chunks.append(text)
+                break
+            split_at = text.rfind("\n", 0, LIMIT)
+            if split_at == -1:
+                split_at = LIMIT
+            chunks.append(text[:split_at])
+            text = text[split_at:].lstrip("\n")
+        for chunk in chunks:
+            await update.message.reply_text(chunk, parse_mode=ParseMode.HTML)
+
+    # ── Commands ──────────────────────────────────────────────────────
+
+    async def cmd_start(self, update: Update, context: ContextTypes.DEFAULT_TYPE):
+        await self.cmd_help(update, context)
+
+    async def cmd_myid(self, update: Update, context: ContextTypes.DEFAULT_TYPE):
+        """
+        Returns the current chat's ID so the operator can set ALERT_CHAT_ID
+        in .env and receive auto-scan alerts here.
+        """
+        chat = update.effective_chat
+        text = (
+            f"<b>Chat ID:</b> <code>{chat.id}</code>\n"
+            f"<b>Type:</b> {chat.type}\n\n"
+            "Set <code>ALERT_CHAT_ID=&lt;id&gt;</code> in <code>.env</code> "
+            "and restart the bot to receive A-grade alerts here."
+        )
+        await update.message.reply_text(text, parse_mode=ParseMode.HTML)
+
+    # ── Auto-scan job (every AUTO_SCAN_INTERVAL seconds) ─────────────
+
+    async def _auto_scan_job(self, context: ContextTypes.DEFAULT_TYPE):
+        """
+        Periodic scan that pushes A/A+ alerts to the configured chat.
+        Dedup: one alert per (symbol, tf, direction, candle_open) — so a
+        persistent setup re-alerts once per new bar, not every 10 min.
+        """
+        chat_id = context.job.data["chat_id"]
+
+        try:
+            results_by_tf = await self._analyze_all_tfs(list(SCAN_PAIRS))
+        except Exception as e:
+            logger.error(f"Auto-scan failed: {e}", exc_info=True)
+            return
+
+        now_str = _now_est()
+        sent = 0
+        for tf, reports in results_by_tf.items():
+            for r in reports:
+                if r.signal_quality not in ALERT_GRADES:
+                    continue
+                # Skip if engine blocked it via event danger
+                if r.event_risk == "danger":
+                    continue
+                ts = r.primary_analysis.timestamp if r.primary_analysis else ""
+                key = (r.symbol, tf, r.final_signal.name, ts)
+                if key in self._alerted_keys:
+                    continue
+                self._alerted_keys.add(key)
+
+                text = _build_alert_card(tf, r, now_str)
+                try:
+                    await context.bot.send_message(
+                        chat_id=chat_id,
+                        text=text,
+                        parse_mode=ParseMode.HTML,
+                    )
+                    sent += 1
+                    logger.info(
+                        f"Alert sent: {r.symbol} {tf} {r.final_signal.name} "
+                        f"({r.signal_quality})"
+                    )
+                except Exception as e:
+                    logger.error(f"Failed to send alert for {r.symbol} {tf}: {e}")
+
+        # Bound dedup memory: keep most recent 256 keys
+        if len(self._alerted_keys) > 256:
+            self._alerted_keys = set(list(self._alerted_keys)[-256:])
+
+        if sent == 0:
+            logger.debug("Auto-scan complete: no new A-grade signals")
+
+    async def cmd_help(self, update: Update, context: ContextTypes.DEFAULT_TYPE):
+        tfs = " · ".join(SCAN_TIMEFRAMES)
+        text = (
+            "🌐 <b>Forex Signal Intelligence Platform</b>\n\n"
+            "<b>Commands:</b>\n"
+            "  /signal EURUSD — Full signal analysis across all timeframes\n"
+            "  /scan          — Scan all pairs, compact Grade B+ summary\n"
+            "  /help          — This message\n\n"
+            f"<b>Timeframes analyzed:</b> <code>{tfs}</code>\n\n"
+            "<b>Signal Grades:</b>\n"
+            "  🔥 A+ — Strongest signal, MTF aligned, high confidence\n"
+            "  ✅ A  — Strong signal, MTF aligned\n"
+            "  👍 B  — Decent setup, wait for confirmation\n"
+            "  ⚠️ C  — Weak signal, watch only\n"
+            "  📉 D  — Very weak, no action\n"
+            "  ❌ F  — No clear signal\n\n"
+            "<i>⚠️ Educational use only. Not financial advice.</i>"
+        )
+        await update.message.reply_text(text, parse_mode=ParseMode.HTML)
+
+    async def cmd_signal(self, update: Update, context: ContextTypes.DEFAULT_TYPE):
+        """
+        /signal EURUSD
+        Runs all timeframes concurrently, returns a single combined message
+        with one stacked section per TF. Entry/TP/SL always preserved.
+        """
+        args = context.args or []
+        if not args:
+            await update.message.reply_text(
+                "Usage: <b>/signal &lt;pair&gt;</b>\n"
+                "Example: <code>/signal EURUSD</code>",
+                parse_mode=ParseMode.HTML,
+            )
+            return
+
+        raw = args[0].upper().replace("/", "").replace("_", "")
+        if not raw.endswith("=X"):
+            raw += "=X"
+        display  = SYMBOL_DISPLAY.get(raw, raw.replace("=X", ""))
+        tfs_str  = " · ".join(SCAN_TIMEFRAMES)
+
+        msg = await update.message.reply_text(
+            f"🔄 Analyzing {_h(display)} across {_h(tfs_str)}…"
+        )
+
+        async def progress(text: str):
+            try:
+                await msg.edit_text(text)
+            except Exception:
+                pass  # ignore edit-rate errors / no-change errors
+
+        try:
+            results_by_tf = await self._analyze_all_tfs([raw], progress=progress)
+            now_str = _now_est()
+
+            # Check we got at least one result
+            if not any(results_by_tf.values()):
+                await msg.edit_text(f"❌ No data returned for {_h(display)}")
+                return
+
+            text = _build_combined_signal(results_by_tf, display, now_str)
+            await msg.delete()
+            await self._send_html(update, text)
+
+        except Exception as e:
+            logger.error(f"Signal error for {raw}: {e}", exc_info=True)
+            await msg.edit_text(f"❌ Analysis failed: {_h(str(e))}")
+
+    async def cmd_scan(self, update: Update, context: ContextTypes.DEFAULT_TYPE):
+        """
+        /scan
+        Analyzes all major pairs across all timeframes concurrently.
+        Picks the best-grade result per pair, returns a single compact message.
+        """
+        tfs_str = " · ".join(SCAN_TIMEFRAMES)
+
+        msg = await update.message.reply_text(
+            f"🔄 Scanning all pairs across {_h(tfs_str)}…"
+        )
+
+        async def progress(text: str):
+            try:
+                await msg.edit_text(text)
+            except Exception:
+                pass
+
+        try:
+            results_by_tf = await self._analyze_all_tfs(list(SCAN_PAIRS), progress=progress)
+            now_str = _now_est()
+
+            # Best grade per pair across all TFs
+            best_per_pair: Dict[str, Tuple[str, SignalReport]] = {}
+            for tf, reports in results_by_tf.items():
+                for r in reports:
+                    current = best_per_pair.get(r.symbol)
+                    if current is None or (
+                        _GRADE_RANK.get(r.signal_quality, 99)
+                        < _GRADE_RANK.get(current[1].signal_quality, 99)
+                    ):
+                        best_per_pair[r.symbol] = (tf, r)
+
+            # Filter B+ and sort best-first
+            actionable = [
+                (tf, r)
+                for tf, r in best_per_pair.values()
+                if r.signal_quality in ("A+", "A", "B")
+            ]
+            actionable.sort(key=lambda x: _GRADE_RANK.get(x[1].signal_quality, 99))
+
+            text = _build_scan_message(actionable, now_str, tfs_str)
+            await msg.edit_text(text, parse_mode=ParseMode.HTML)
+
+        except Exception as e:
+            logger.error(f"Scan error: {e}", exc_info=True)
+            await msg.edit_text(f"❌ Scan failed: {_h(str(e))}")
+
+    # ── Application builder ───────────────────────────────────────────
+
+    def build_application(self) -> Application:
+        token = os.environ.get("TELEGRAM_BOT_TOKEN")
+        if not token:
+            raise ValueError(
+                "TELEGRAM_BOT_TOKEN environment variable is not set.\n"
+                "Get a token from @BotFather on Telegram, then:\n"
+                "  Windows CMD:  set TELEGRAM_BOT_TOKEN=your_token\n"
+                "  Linux/Mac:    export TELEGRAM_BOT_TOKEN=your_token"
+            )
+
+        async def _register_commands(app: Application) -> None:
+            await app.bot.set_my_commands([
+                BotCommand("signal", "Signal analysis for one pair — /signal EURUSD"),
+                BotCommand("scan",   "Scan all pairs, best Grade B+ per pair"),
+                BotCommand("myid",   "Show this chat's ID (for ALERT_CHAT_ID setup)"),
+                BotCommand("help",   "Commands and grade guide"),
+            ])
+
+        app = (
+            Application.builder()
+            .token(token)
+            .post_init(_register_commands)
+            .build()
+        )
+
+        app.add_handler(CommandHandler("start",  self.cmd_start))
+        app.add_handler(CommandHandler("help",   self.cmd_help))
+        app.add_handler(CommandHandler("signal", self.cmd_signal))
+        app.add_handler(CommandHandler("scan",   self.cmd_scan))
+        app.add_handler(CommandHandler("myid",   self.cmd_myid))
+
+        # ── Auto-scan job: only enabled when ALERT_CHAT_ID is set ─────
+        alert_chat_raw = os.environ.get("ALERT_CHAT_ID")
+        if alert_chat_raw:
+            if app.job_queue is None:
+                logger.warning(
+                    "ALERT_CHAT_ID set but JobQueue is unavailable. "
+                    "Install with: pip install 'python-telegram-bot[job-queue]'"
+                )
+            else:
+                try:
+                    chat_id = int(alert_chat_raw)
+                except ValueError:
+                    chat_id = alert_chat_raw  # channel @username form
+                app.job_queue.run_repeating(
+                    self._auto_scan_job,
+                    interval=AUTO_SCAN_INTERVAL,
+                    first=15,  # first run 15s after startup
+                    name="auto_scan",
+                    data={"chat_id": chat_id},
+                )
+                logger.info(
+                    f"Auto-scan enabled: every {AUTO_SCAN_INTERVAL}s → "
+                    f"chat {chat_id} (grades: {sorted(ALERT_GRADES)})"
+                )
+        else:
+            logger.info(
+                "Auto-scan disabled — set ALERT_CHAT_ID in .env to enable. "
+                "Use /myid in your target chat to find the ID."
+            )
+
+        return app
+
+
+# ══════════════════════════════════════════════════════════════════════
+
+def main():
+    logging.basicConfig(
+        level=logging.WARNING,
+        format="%(asctime)s [%(levelname)s] %(name)s: %(message)s",
+        datefmt="%H:%M:%S",
+    )
+    bot = ForexTelegramBot()
+    app = bot.build_application()
+    print("🤖 Forex Signal Bot started. Press Ctrl+C to stop.")
+    app.run_polling(drop_pending_updates=True)
+
+
+if __name__ == "__main__":
+    main()