Narrow bot to USDCAD H4 Grade B + paper-trade tracker + /status command

Backtest-driven config change. The previous wide-scan setup (4 pairs ×
M15+H1, alert grades A+/A) was empirically losing money in every
backtested permutation. The 2-year backtest identified one robust niche
(USDCAD H4 Grade B: train +20.8 R / test +14.0 R / PF 1.52 / max DD
-9.6 R, positive in every 6-month sub-period). This commit restricts
the live bot to that niche and runs it in paper-trade mode for
validation before any real capital is risked.

Config:
  SCAN_PAIRS         ["USDCAD=X"]   (was 4 pairs)
  SCAN_TIMEFRAMES    ["H4"]          (was M15+H1)
  ALERT_GRADES       {"B"}           (was A+/A — those are anti-edge)
  AUTO_SCAN_INTERVAL 1800            (was 600 — H4 candles only print
                                       every 4h, polling more wastes credits)
  PAPER_TRADE_MODE   True            (alerts get [PAPER] prefix and
                                       are NOT for real money)

New machinery:
- _open_paper_position() / _check_paper_positions(): on every scan,
  walks any open paper trades forward against the latest H4 bars using
  the same first-touch SL/TP1 logic as backtest.simulate_trade.
- paper_open.json + paper_trades.csv: persistent state across restarts.
  Open positions survive bot restarts; closed trades append to the CSV.
- /status: uptime, scan count, last scan time, alert count, open paper
  positions, win rate, total R, profit factor.
- /paper: full ledger — open trades + last 10 closed.
- INFO-level logging in main() so the user sees scan activity in
  stdout instead of wondering whether the bot is alive.
- Help text updated to reflect the calibrated strategy and warn that
  A+/A grades are anti-edge in backtest.
- gitignore: paper_open.json + paper_trades.csv + backtest_results_*/

Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>

.gitignore

@@ -0,0 +1,26 @@
+# Secrets
+.env
+.env.local
+
+# Editor / tooling
+.claude/
+.vscode/
+.idea/
+
+# Python (note: __pycache__/ is intentionally tracked in this repo — see CLAUDE.md)
+*.egg-info/
+.venv/
+venv/
+
+# OS
+.DS_Store
+Thumbs.db
+
+# Backtest data + outputs (regenerable from seed_history.py / backtest.py)
+history/
+backtest_results/
+backtest_results_*/
+
+# Paper-trade ledger (live state — do not version control)
+paper_open.json
+paper_trades.csv

CLAUDE.md

@@ -0,0 +1,241 @@
+# ForexBot2 — Working Context for Claude
+
+Comprehensive context to avoid re-deriving the codebase in every session.
+Update this file when behavior, defaults, or file layout changes.
+
+---
+
+## What it does
+
+Forex + gold signal bot. Two entry points:
+
+- **CLI**: `main.py --pairs EURUSD --timeframe H1 [--detail] [--min-grade C]`
+- **Telegram bot**: `telegram_bot.py` (commands: `/scan`, `/signal <pair>`, `/help`)
+
+Pulls OHLCV → runs 8 indicators (RSI, MACD, EMA, Stoch, ADX, BB, Ichimoku, Pivots) → multi-timeframe agreement check → news sentiment → grades A+/A/B/C/D/F.
+
+## Run commands (Windows PowerShell host)
+
+```powershell
+.venv/Scripts/python.exe main.py --pairs USDJPY --timeframe H1 --detail
+.venv/Scripts/python.exe telegram_bot.py
+```
+
+On Linux server: `venv/bin/python ...`. Server requires `pip install curl_cffi` (see SSL section).
+
+---
+
+## File roles
+
+| File | Role |
+|---|---|
+| `config.py` | Dataclasses for `PlatformConfig`, `TechnicalConfig`, `SignalConfig`, `NewsConfig`. Pair lists (`MAJOR_PAIRS`, `CROSS_PAIRS`, `SYMBOL_DISPLAY`, `PIP_VALUES`). `Timeframe` enum. |
+| `data_provider.py` | OHLCV fetch (Twelve Data primary, yfinance for XAU/XAG). Cache, rate limiting, validation. |
+| `engine.py` | `TechnicalEngine` — computes indicators, builds `TechnicalAnalysis` per (symbol, TF). |
+| `news_engine.py` | RSS feed sentiment + economic calendar event risk. |
+| `signal_aggregator.py` | The "brain". Iterates primary + confirmation TFs, runs MTF agreement check, applies news, builds `SignalReport` + `TradeSetup`. |
+| `main.py` | CLI entry. Parses args, builds `PlatformConfig`, runs `analyze_all_pairs`, prints colored table. |
+| `telegram_bot.py` | Telegram bot. Owns `SCAN_PAIRS`, `SCAN_TIMEFRAMES`, per-TF confirmation map. Renders cards. |
+| `LOGIC.md` | Engine decision flow + indicator weights. Kept up-to-date manually. |
+| `USAGE.md` | User-facing run guide. |
+
+---
+
+## Data flow
+
+```
+fetch_ohlcv (per symbol/TF) → tech_engine.analyze → TechnicalAnalysis
+                                       ↓
+            SignalAggregator.analyze_pair (primary + confirmation TFs)
+                                       ↓
+                        _check_mtf_agreement → SignalReport
+                                       ↓
+                  news_engine.get_pair_sentiment → final score + grade
+                                       ↓
+                      _build_trade_setup → TradeSetup (Entry/TP/SL)
+```
+
+`analyze_all_pairs` calls `prefetch_batch` once before iterating pairs — that's the optimization that makes the free Twelve Data tier viable.
+
+---
+
+## Where defaults live (and what overrides them)
+
+**This is the #1 footgun.** The dataclass defaults in `config.py` are NOT authoritative for actual runs — both CLI and Telegram bot rebuild the config and override `confirmation_timeframes` based on the chosen primary TF.
+
+| Default | Defined in | Overridden by |
+|---|---|---|
+| `primary_timeframe = H1` | [config.py:188](config.py#L188) | `main.py` (`--timeframe`), `telegram_bot.py` (per scan) |
+| `confirmation_timeframes = [M15, D1]` | [config.py:189-191](config.py#L189-L191) | `main.py:407-417` (`tf_confirmations` dict), `telegram_bot.py:53-56` (`_TF_CONFIRMATIONS`) |
+| `mtf_agreement_threshold = 2` | [config.py:143](config.py#L143) | _Not_ overridden — single source of truth. |
+| `weights` dict | [config.py:133-140](config.py#L133-L140) | _Not_ overridden. |
+| `cache_ttl_per_tf` | [config.py:207-216](config.py#L207-L216) | _Not_ overridden. |
+
+**When you change confirmation TFs, update all three places.** Otherwise CLI/Telegram silently keep the old values.
+
+---
+
+## Per-symbol TF overrides (XAU/XAG)
+
+Defined in `signal_aggregator.py`:
+
+```python
+_PER_SYMBOL_TFS = {
+    "XAUUSD=X": [Timeframe.M15, Timeframe.H1, Timeframe.H4],
+    "XAGUSD=X": [Timeframe.M15, Timeframe.H1, Timeframe.H4],
+}
+```
+
+`analyze_pair` calls `self._confirmation_tfs(symbol)` which returns the override (minus primary) when the symbol is in the map, else falls back to `config.confirmation_timeframes`. Forex pairs are unaffected.
+
+To add another symbol-specific TF set: add to `_PER_SYMBOL_TFS`. `analyze_all_pairs.prefetch_batch` already unions per-symbol TFs.
+
+---
+
+## Data source rules
+
+### Twelve Data (forex)
+- Free tier: **8 credits/min, 800 credits/day**.
+- Each `time_series` call = 1 credit per symbol. Batch endpoint accepts comma-separated symbols → ~1 credit per timeframe regardless of pair count (when used).
+- API key in [config.py:197](config.py#L197). Override via `TWELVE_DATA_API_KEY` env var.
+- Rate limiter is **module-level** (`_rate_lock`, `_rate_request_times`) so concurrent scans share the budget.
+- Supported on free tier: EURUSD, GBPUSD, USDJPY, USDCHF, AUDUSD, USDCAD, NZDUSD, EURGBP, EURJPY, GBPJPY, AUDJPY, EURCHF, CADJPY.
+- **Not supported on free tier**: XAUUSD, XAGUSD, anything in `_YF_FALLBACK_MAP`.
+
+### yfinance (gold/silver)
+- Routes via `curl_cffi.Session(impersonate="chrome", verify=False)` — see SSL section.
+- Mapping in [data_provider.py:34-37](data_provider.py#L34-L37): `XAUUSD=X → GC=F`, `XAGUSD=X → SI=F` (CME futures).
+- yfinance interval limits: 1m → 7d back; 5m/15m/30m/60m/1h → 60d back.
+- **No native H4 candle.** XAU/XAG H4 is resampled from H1 in `_resample_h1_to_h4` ([data_provider.py:291-300](data_provider.py#L291-L300)).
+- **D1 explicitly skipped** for XAU/XAG — futures daily data via yfinance is unreliable. Engine drops missing confirmation TFs cleanly via `except Exception` at `signal_aggregator.py:189-190`.
+
+### No synthetic fallback
+The old `_generate_synthetic_data()` (GBM with a `pair_params` dict that defaulted XAU to ~$1.00, causing trade setups like Entry 1.01) is **removed**. If both Twelve Data and yfinance fail, `fetch_ohlcv` raises `DataUnavailableError`. The engine returns grade F + warning, Telegram shows `❌ Data unavailable`.
+
+### Cache
+Per-symbol per-TF, in-memory dict. TTLs in `config.cache_ttl_per_tf`:
+- M1: 60s · M5: 150s · M15: 300s · M30: 600s · H1: 1800s · H4: 7200s · D1: 14400s · W1: 86400s
+
+Shared across `SignalAggregator` instances when the same `ForexDataProvider` is reused (see Telegram `_analyze_all_tfs`).
+
+---
+
+## Telegram /scan flow
+
+```python
+SCAN_PAIRS      = ["EURUSD=X", "USDJPY=X", "GBPUSD=X", "XAUUSD=X"]   # 4 pairs only
+SCAN_TIMEFRAMES = ["M15", "H1"]                                       # 2 cards per pair
+_TF_CONFIRMATIONS = {
+    "M15": [H1, D1],
+    "H1":  [M15, D1],
+}
+```
+
+`_analyze_all_tfs` ([telegram_bot.py:282-310](telegram_bot.py#L282-L310)):
+1. Builds the union of all unique TFs across `SCAN_TIMEFRAMES` + their confirmations.
+2. Single `prefetch_batch` for all of them on **one shared `ForexDataProvider` instance**.
+3. Spawns one scan per `SCAN_TIMEFRAMES` entry concurrently — each reuses the warm cache, so no extra API credits.
+
+Per /scan API cost: ~9 Twelve Data credits (3 forex pairs × 3 TFs, fetched once). XAU is free via yfinance.
+
+To add/remove pairs: edit `SCAN_PAIRS` ([telegram_bot.py:47](telegram_bot.py#L47)).
+To change which TFs get their own card: edit `SCAN_TIMEFRAMES` ([telegram_bot.py:43](telegram_bot.py#L43)).
+
+---
+
+## SSL — verify=False on server
+
+Live server sits behind a TLS-inspecting proxy that re-signs HTTPS with an untrusted root CA. Fix in [data_provider.py:27-37](data_provider.py#L27-L37):
+
+```python
+from curl_cffi import requests as _curl_cffi_requests
+_YF_SESSION = _curl_cffi_requests.Session(impersonate="chrome", verify=False)
+```
+
+Passed into `yf.download(..., session=_YF_SESSION)`. yfinance ≥ 0.2.40 **requires** a `curl_cffi` session — it rejects `requests.Session`.
+
+Trade-off: yfinance traffic is no longer authenticated against MITM. Acceptable because the data is public market quotes and an attacker can at worst inject bad prices. **Do not** reuse `_YF_SESSION` for any authenticated/private requests.
+
+---
+
+## Indicator weights & scoring
+
+In `config.SignalConfig.weights`:
+
+```python
+{"trend": 0.25, "momentum": 0.20, "volatility": 0.15,
+ "price_action": 0.15, "multi_timeframe": 0.15, "sentiment": 0.10}
+```
+
+Per-indicator weights live in `engine.py` (each indicator function returns its own contribution). Composite score is a weighted sum, capped to [-100, +100]. Grade derived from `abs(score) + confidence + mtf_agreement` in `_compute_signal_quality` (`signal_aggregator.py`).
+
+---
+
+## How to make common changes
+
+### Add a new tradeable pair (forex)
+1. Add `"XYZUSD=X"` to `MAJOR_PAIRS` or `CROSS_PAIRS` in `config.py`.
+2. Add display name to `SYMBOL_DISPLAY`.
+3. Verify it's on Twelve Data free tier (else add to `_YF_FALLBACK_MAP`).
+4. If Telegram /scan should include it, add to `SCAN_PAIRS` in `telegram_bot.py`.
+
+### Add a non-Twelve-Data symbol (commodity, index)
+1. Add to `_YF_FALLBACK_MAP` in `data_provider.py` with the Yahoo symbol.
+2. Add a `_PER_SYMBOL_TFS` entry in `signal_aggregator.py` (D1 unreliable, so drop it; H4 needs resample).
+3. Add to `SYMBOL_DISPLAY` and `PIP_VALUES` in `config.py`.
+
+### Change confirmation TFs for forex
+Edit **all three**:
+- `config.py:189-191` — engine default.
+- `main.py:407-417` — CLI per-primary map.
+- `telegram_bot.py:53-56` — Telegram per-primary map.
+
+### Change MTF agreement threshold
+Single edit: `config.py:143`. Affects all symbols. Note: with N TFs and threshold T, the rule is "T-of-N must agree" — keep T proportional to N if you change one.
+
+### Add a new indicator
+Add to `engine.py` → `TechnicalEngine.analyze`. Set its `weight` in the `Indicator` instance. The composite score will pick it up automatically.
+
+### Switch primary TF
+CLI: `--timeframe M15|M30|H1|H4|D1`.
+Telegram: edit `SCAN_TIMEFRAMES`.
+
+---
+
+## Common gotchas
+
+1. **The dataclass defaults in `config.py` are not the source of truth for `confirmation_timeframes`.** CLI and Telegram override them. See "Where defaults live" table.
+2. **Per-minute API rate limit hits hard with `--detail`** — one detailed run on one pair burns ~3 credits (3 TFs). Multiple back-to-back runs without ~60s gap will hit the 8/min ceiling.
+3. **Daily limit (800) is shared across the whole day** — heavy testing exhausts the budget; signal cache survives but new fetches fail until midnight UTC.
+4. **`curl_cffi` is a hard dep on the server** — without it, `_YF_SESSION` is `None`, yfinance falls back to its default backend, which fails on TLS-inspecting servers.
+5. **`__pycache__/` is committed** in this repo (legacy). Don't add it to `.gitignore` without coordinating; old `.pyc` will linger as untracked-modified.
+6. **Telegram `SCAN_PAIRS` is hardcoded to 4 pairs** — editing `MAJOR_PAIRS` does not change /scan. Update both if intentional.
+7. **`mtf_agreement_threshold = 2` means 2-of-N**, not 2-of-3 specifically. Adding a 4th TF to a pair makes "2-of-4" looser than "2-of-3"; consider per-symbol thresholds if this matters.
+8. **The `MTF: ✅ Aligned` line in Telegram cards comes from the engine's `report.mtf_agreement` boolean** ([signal_aggregator.py:223-233](signal_aggregator.py#L223-L233)) — not displayed per-TF. The full per-TF breakdown is only in the CLI `--detail` output.
+
+---
+
+## News sentiment — Google Gemini (live)
+
+`news_engine` uses **Gemini `gemini-2.5-flash-lite`** for headline sentiment, replacing the keyword lexicon (which still exists as a fallback).
+
+- API key loaded from `.env` (`GEMINI_API_KEY=...`) via `python-dotenv`. `.env` is gitignored.
+- One **batched** API call per `/scan` — `_score_headlines_batch` sends all uncached titles in a single prompt, returns a JSON array of `{id, score}` ([news_engine.py:355-410](news_engine.py#L355-L410)).
+- In-memory cache `_sentiment_cache` keyed by `hash(headline)` — repeat calls for the same headline are free.
+- Free-tier quota on `gemini-2.5-flash-lite`: **1000 RPD** (vs 20 on `gemini-2.5-flash` — do not use the non-lite variant).
+- If Gemini fails (timeout, 429, parse error), falls back per-headline to `_fallback_keyword_sentiment` so signals still grade.
+
+Observed quality lift: ~88% of fetched headlines now get non-zero sentiment (vs ~10% with keywords). Headlines like "RBA signals pause" or "inflation cools below forecast" score correctly bearish where keywords missed them entirely.
+
+To swap models: change `NewsEngine.GEMINI_MODEL` ([news_engine.py:118](news_engine.py#L118)). Use the new `google-genai` SDK (`from google import genai; genai.Client(api_key=...)`); the older `google-generativeai` package is replaced.
+
+---
+
+## Recent architectural decisions (kept short)
+
+- **Removed synthetic data path** — explicit failure beats fake prices.
+- **XAU/XAG → yfinance only** — Twelve Data free tier doesn't carry them; routing them through TD wastes credits and used to fall through to bogus synthetic data.
+- **H1→H4 resample for gold** — yfinance has no native 4h, but H1 with 60d lookback gives ~360 H4 bars after resample, enough for indicator calculation.
+- **/scan limited to 4 pairs** — credit budget on free tier doesn't sustain 13-pair scans.
+- **Two SCAN_TIMEFRAMES (M15 + H1) with shared cache** — gives users two perspectives without doubling API cost (cache reuse).
+- **No SSL verify on yfinance** — server-side TLS inspection breaks cert chain; disabled because data is public quotes.

LOGIC.md

@@ -0,0 +1,226 @@
+# ForexBot2 — Engine Logic Reference
+
+## 1. Data Provider (Twelve Data API)
+
+### What the API provides
+Each API call returns an OHLCV time series for a currency pair:
+
+| Field    | Description                        |
+|----------|------------------------------------|
+| Datetime | UTC timestamp of the candle open   |
+| Open     | Opening price                      |
+| High     | Highest price during the candle    |
+| Low      | Lowest price during the candle     |
+| Close    | Closing price                      |
+| Volume   | Tick volume (limited on Forex)     |
+
+Supported intervals: 1min, 5min, 15min, 30min, 1h, 4h, 1day, 1week.
+
+### Supported pairs (free tier — live data)
+EURUSD, GBPUSD, USDJPY, USDCHF, AUDUSD, USDCAD, NZDUSD,
+EURGBP, EURJPY, GBPJPY, AUDJPY, EURCHF, CADJPY.
+All other symbols (XAU, XAG, indices) return no data on the free plan.
+
+### API credit limits (free tier)
+- 8 credits per minute
+- 800 credits per day
+- 1 credit per symbol in a batch request
+  (7 pairs in one request = 7 credits, not 7 requests)
+
+### Data validation and cleaning
+After every fetch the data goes through `_validate_and_clean`:
+1. Drop rows where all of Open/High/Low/Close are NaN
+2. Forward-fill small gaps (max 3 consecutive bars)
+3. Drop rows still missing Close
+4. Remove bars with future timestamps (> 5 min ahead of now)
+5. Fix OHLC inconsistencies (e.g. High < Low) by clamping
+6. Remove extreme outlier bars (> 10 standard deviations return)
+7. Sort index and deduplicate (keep last on duplicates)
+
+---
+
+## 2. How Many API Calls Are Made
+
+### Per scan or /signal command
+
+The bot does **one shared prefetch** before spawning TF analyses.
+It collects all unique timeframes needed across scans, then makes
+one batch API call per timeframe (all pairs in a single request).
+
+Current SCAN_TIMEFRAMES: M15, H1 (only intraday trade signals)
+Confirmations:
+- M15 uses: H1, D1
+- H1  uses: M15, D1
+
+D1 is kept as a confirmation for higher-timeframe trend context but
+no D1 trade signals are produced. W1 has been dropped entirely.
+
+Unique timeframes: M15, H1, D1 = **3 batch calls**
+
+Each call: 7 pairs × 1 credit = **7 credits per call**
+
+Total per scan: **3 calls × 7 credits = 21 credits**
+
+With the 8 credits/minute limit:
+- Call 1 (7 credits): fires immediately — 1 credit remaining
+- Call 2 (7 credits): must wait ~60s for window to clear
+- Call 3 (7 credits): waits another ~60s
+
+**Minimum scan time: ~2 minutes.**
+
+After the shared prefetch, the 4 concurrent TF analyses read from
+the in-memory cache and make **zero additional API calls**.
+
+### Cache behaviour
+Results are cached per symbol+timeframe for `refresh_interval` seconds
+(default in PlatformConfig). A second scan within that window skips all
+API calls entirely and returns cached data instantly.
+
+---
+
+## 3. Technical Engine — How Signals Are Decided
+
+### Step 1 — Fetch OHLCV data
+Primary timeframe data is fetched first. If it fails, the report is
+immediately graded F and returned — no further analysis is attempted.
+Confirmation timeframe data is fetched next; failures are logged as
+warnings but do not abort the analysis.
+
+### Step 2 — Compute 11 indicators
+
+Each indicator returns: value, signal (STRONG_BUY / BUY / NEUTRAL /
+SELL / STRONG_SELL), confidence (0–100), and a weight.
+
+| Indicator        | Weight | What it measures                              |
+|------------------|--------|-----------------------------------------------|
+| EMA Trend        | 1.2    | Alignment of fast/medium/slow/trend EMAs      |
+| MACD             | 1.1    | Momentum crossover and histogram direction    |
+| Ichimoku         | 1.1    | Cloud position, TK cross, cloud colour        |
+| RSI              | 1.0    | Overbought/oversold momentum                  |
+| ADX              | 1.0    | Trend strength and directional index          |
+| Bollinger Bands  | 0.9    | Price position within volatility envelope     |
+| Stochastic       | 0.8    | Oscillator crossover in OB/OS zones           |
+| Candlestick      | 0.7    | Pattern detection (hammer, engulfing, doji…)  |
+| Divergence       | 1.5    | RSI/MACD divergence from price (reversal)     |
+| Pivot Points     | 0.6    | Classical pivot S/R relative to current price |
+| ATR / Volume     | 0.0    | Context only — not included in score          |
+
+Divergence has the highest weight (1.5) because divergences are strong
+reversal predictors. ATR and Volume have weight 0 — they inform the
+trade setup and volatility regime but do not vote on direction.
+
+### Step 3 — Classify market regime
+Using ADX value and EMA direction:
+- `trending_up` / `trending_down`: ADX ≥ strong threshold + EMA aligned
+- `ranging`: ADX below threshold
+
+The regime changes indicator weights for the composite score:
+- **Trending**: EMA (×1.5), ADX (×1.3), MACD (×1.2) weighted up;
+  RSI (×0.7) and Bollinger (×0.6) weighted down
+- **Ranging**: RSI (×1.4), Bollinger (×1.4), Stochastic (×1.3)
+  weighted up; EMA (×0.6) and ADX (×0.5) weighted down
+
+### Step 4 — Compute composite score (−100 to +100)
+For each indicator:
+```
+weighted_score = signal_value × weight × regime_multiplier × (confidence / 100)
+```
+Signal values: STRONG_BUY = +2, BUY = +1, NEUTRAL = 0, SELL = −1, STRONG_SELL = −2
+
+Final composite score = (sum of weighted scores / total weight) × 50
+
+Composite signal thresholds:
+- score ≥ +40 → STRONG BUY
+- score ≥ +15 → BUY
+- score ≤ −40 → STRONG SELL
+- score ≤ −15 → SELL
+- otherwise  → NEUTRAL
+
+### Step 5 — Multi-timeframe (MTF) agreement check
+All timeframe signals (primary + confirmations) are collected.
+MTF agreement = True if at least `mtf_agreement_threshold` timeframes
+all point bullish or all point bearish.
+
+Effect on confidence:
+- **Agreement**: confidence +15 (capped at 95); score blended with
+  average of all TF scores (60% primary, 40% MTF average)
+- **Disagreement**: confidence ×0.7
+- **Conflict** (some bullish, some bearish): confidence ×0.7 × 0.6 = ×0.42
+  + warning added to report
+
+### Step 6 — Fundamental adjustments
+Applied in order, each multiplying confidence:
+
+| Condition                        | Confidence adjustment |
+|----------------------------------|-----------------------|
+| News aligns with technical       | +5 (capped 95)        |
+| News conflicts with technical    | ×0.85                 |
+| Event risk: caution              | ×0.70                 |
+| Event risk: danger               | ×0.30 + block signal  |
+| Volatility: extreme              | ×0.80                 |
+
+### Step 7 — Final signal determination
+Uses the adjusted score and confidence:
+
+| Condition                        | Signal      |
+|----------------------------------|-------------|
+| score ≥ +40 and confidence ≥ 65 | STRONG BUY  |
+| score ≥ +15 and confidence ≥ 50 | BUY         |
+| score ≤ −40 and confidence ≥ 65 | STRONG SELL |
+| score ≤ −15 and confidence ≥ 50 | SELL        |
+| otherwise                        | NEUTRAL     |
+
+### Step 8 — Quality grade
+Combines score, confidence, and MTF agreement into a letter grade:
+
+| Grade | abs(score) | confidence | MTF agreement |
+|-------|-----------|------------|---------------|
+| A+    | ≥ 50      | ≥ 75       | required      |
+| A     | ≥ 35      | ≥ 65       | required      |
+| B     | ≥ 25      | ≥ 55       | not required  |
+| C     | ≥ 15      | ≥ 45       | not required  |
+| D     | ≥ 10      | any        | not required  |
+| F     | < 10      | any        | —             |
+
+A+ and A require MTF agreement as a hard condition — no matter how
+strong the primary score, without timeframe alignment the grade caps at B.
+
+### Step 9 — Trade setup (grades A+, A, B only)
+Entry = current close price
+Stop Loss = entry ± (ATR × 1.5)
+Take Profit 1 = entry ± (SL distance × R:R ratio)
+Take Profit 2 = entry ± (SL distance × R:R ratio × 1.5)
+
+ATR is used as the volatility measure so SL/TP adapt to current
+market conditions rather than using fixed pip distances.
+
+---
+
+## 4. Signal Flow Summary
+
+```
+Telegram /scan or /signal
+        │
+        ▼
+_analyze_all_tfs(pairs)
+        │
+        ├── prefetch_batch(pairs, [M15, H1, D1])
+        │       ├── Batch call 1: all pairs @ M15  (7 credits, wait if needed)
+        │       ├── Batch call 2: all pairs @ H1   (7 credits, wait if needed)
+        │       └── Batch call 3: all pairs @ D1   (7 credits, wait if needed)
+        │
+        └── 2 threads (M15 + H1), all reading from warm cache
+                │
+                └── per thread: SignalAggregator.analyze_all_pairs()
+                        │
+                        └── per pair: analyze_pair()
+                                ├── fetch_ohlcv (cache hit)
+                                ├── TechnicalEngine.analyze()
+                                │       ├── 11 indicators
+                                │       ├── Regime classification
+                                │       └── Composite score
+                                ├── MTF agreement check
+                                ├── News / event risk adjustments
+                                ├── Final signal + grade
+                                └── Trade setup (A+/A/B only)
+```

USAGE.md

@@ -0,0 +1,285 @@
+# ForexBot2 — Usage Guide
+
+## Setup
+
+```bash
+pip install -r requirements.txt
+```
+
+Set your Twelve Data API key (the bot will not work without one):
+
+```powershell
+$env:TWELVE_DATA_API_KEY="your_key_here"   # PowerShell
+```
+
+```bash
+export TWELVE_DATA_API_KEY=your_key_here   # macOS / Linux
+set TWELVE_DATA_API_KEY=your_key_here      # Windows CMD
+```
+
+Get a free key at <https://twelvedata.com/pricing>. Free tier covers most major Forex pairs; gold/silver fall through to Yahoo Finance automatically (see [Symbols](#supported-symbols)).
+
+---
+
+## Terminal — `main.py`
+
+```
+python main.py [OPTIONS]
+```
+
+### Options
+
+| Option                       | Default                  | Description                                              |
+|------------------------------|--------------------------|----------------------------------------------------------|
+| `--pairs PAIR [PAIR ...]`    | all major pairs          | Specific pairs to analyze                                |
+| `--timeframe TF` / `-tf TF`  | `H1`                     | Primary timeframe — see table below                      |
+| `--all`                      | off                      | Include cross pairs (EUR/GBP, EUR/JPY, GBP/JPY, etc.)    |
+| `--detail`                   | off                      | Full indicator breakdown for every pair                  |
+| `--top N`                    | all                      | Show only the top N signals by strength                  |
+| `--min-grade GRADE`          | `F`                      | Minimum grade to display: `A+ A B C D F`                 |
+| `--json`                     | off                      | Output results as JSON                                   |
+| `--watch`                    | off                      | Continuous mode — re-scans every 5 minutes               |
+| `--verbose` / `-v`           | off                      | Enable debug logging                                     |
+
+### Timeframes accepted by `--timeframe`
+
+| Flag  | Interval    | Best for                  |
+|-------|-------------|---------------------------|
+| `M5`  | 5 minutes   | Scalping                  |
+| `M15` | 15 minutes  | Short-term intraday       |
+| `M30` | 30 minutes  | Intraday                  |
+| `H1`  | 1 hour      | Intraday / swing (default)|
+| `H4`  | 4 hours     | Swing trading             |
+| `D1`  | Daily       | Position trading          |
+
+`main.py` automatically picks confirmation timeframes around the primary
+(e.g. `H1` confirms with `M15` and `H4`).
+
+### Examples
+
+```bash
+# Scan all major pairs on H1 (default)
+python main.py
+
+# One pair, full indicator detail
+python main.py --pairs EURUSD --detail
+
+# Gold on the 4-hour chart
+python main.py --pairs XAUUSD --timeframe H4 --detail
+
+# Multiple pairs
+python main.py --pairs EURUSD GBPUSD USDJPY
+
+# Only Grade B+ signals
+python main.py --min-grade B
+
+# Top 3 signals across all pairs including crosses
+python main.py --all --top 3
+
+# Continuous monitoring — refreshes every 5 minutes
+python main.py --watch
+python main.py --pairs XAUUSD --watch --detail
+
+# JSON output to a file
+python main.py --json
+python main.py --all --json > signals.json
+```
+
+### Terminal output
+
+Summary table:
+
+```
+  │ EUR/USD     │ STRONG BUY │ ░░░░░░░░░░░░░│███████████████ │ A+     │   82.3%  │
+```
+
+- **Score bar** — left of `│` = bearish, right = bullish; filled blocks show strength
+- **Grade** — overall quality of the signal setup
+- **⚠** — high-impact event imminent;  **△** — event approaching (caution)
+
+Detail block (shown automatically for Grade B+ signals, or for every pair
+with `--detail`) includes price, regime, multi-timeframe breakdown, every
+indicator value, bullish/bearish reasons, warnings, news, and the trade
+setup (entry / SL / TP1 / TP2).
+
+---
+
+## Telegram Bot — `telegram_bot.py`
+
+```powershell
+$env:TELEGRAM_BOT_TOKEN="your_token_here"
+python telegram_bot.py
+```
+
+Get a token from [@BotFather](https://t.me/BotFather) → `/newbot`.
+
+### Commands
+
+| Command            | Description                                                         |
+|--------------------|---------------------------------------------------------------------|
+| `/signal EURUSD`   | Full signal analysis for one pair across M15 and H1                 |
+| `/signal XAUUSD`   | Same — gold uses the yfinance fallback automatically                |
+| `/scan`            | Analyze all major pairs, return one card per Grade B+ pair           |
+| `/help`            | Command reference + grade legend                                     |
+| `/start`           | Same as `/help`                                                      |
+
+The bot **does not** accept a timeframe argument. It always scans M15 and
+H1 (using D1 as a confirmation timeframe internally for higher-timeframe
+trend context). To change timeframes, edit `SCAN_TIMEFRAMES` in
+[telegram_bot.py](telegram_bot.py).
+
+### Live progress
+
+Every command shows coarse progress through message edits while it runs:
+
+```
+🔄 Analyzing XAU/USD across M15 · H1…
+📡 Fetching market data (3 timeframes × 1 pair)…
+🧠 Analyzing pairs across timeframes…
+```
+
+The placeholder is then replaced by the result message (`/signal`) or new
+messages are sent (`/scan`).
+
+### Signal card format
+
+```
+🌐 🧠 EUR/USD — Signal Analysis
+🕐 03:45 PM EST · Mon 04 May 2026
+Grade scale: 🔥 A+ · ✅ A · 👍 B · ⚠️ C · 📉 D · ❌ F
+
+────────────────────────
+✅ M15 · BUY · Grade A
+
+Entry: 1.08423
+Take Profit: 1.08970  (+54.7 pips)
+Take Profit 2: 1.09517
+Stop Loss: 1.08150  (-27.3 pips)
+Risk:Reward: 1:2.0
+
+• EMA Trend: 1.00 (Golden cross)
+• MACD: 0.000082 (Bullish crossover)
+• RSI: 58.3 (Neutral)
+...
+
+MTF: ✅ Aligned | Score: +62.4 | trending_up | Vol: normal
+```
+
+### Scan format
+
+`/scan` produces one compact line per Grade B+ pair:
+
+```
+👍 EUR/USD · BUY · B · M15
+   Entry: 1.08423 | TP: 1.08970 (+55p) | SL: 1.08150 (-27p) | R:R 1:2.0
+```
+
+Pairs without an actionable Grade B+ signal are omitted from the scan output.
+
+---
+
+## Signal Grades
+
+| Grade | Symbol | abs(score) | confidence | MTF agreement | Meaning                              |
+|-------|--------|-----------:|-----------:|---------------|--------------------------------------|
+| A+    | 🔥     | ≥ 50       | ≥ 75       | required      | Strongest — entry candidate          |
+| A     | ✅     | ≥ 35       | ≥ 65       | required      | Strong signal                        |
+| B     | 👍     | ≥ 25       | ≥ 55       | not required  | Decent setup — confirm before entry  |
+| C     | ⚠️     | ≥ 15       | ≥ 45       | not required  | Weak — monitor only                  |
+| D     | 📉     | ≥ 10       | any        | not required  | Noise                                |
+| F     | ❌     | < 10       | any        | —             | No trade                             |
+
+Trade setup (entry / TP / SL) is generated only for Grade A+ / A / B.
+
+---
+
+## Supported Symbols
+
+Pass any symbol without slashes or the `=X` suffix.
+
+### Forex (live data on Twelve Data free tier)
+
+| Input    | Pair                                  |
+|----------|---------------------------------------|
+| `EURUSD` | Euro / US Dollar                      |
+| `GBPUSD` | British Pound / US Dollar             |
+| `USDJPY` | US Dollar / Japanese Yen              |
+| `USDCHF` | US Dollar / Swiss Franc               |
+| `AUDUSD` | Australian Dollar / US Dollar         |
+| `USDCAD` | US Dollar / Canadian Dollar           |
+| `NZDUSD` | New Zealand Dollar / US Dollar        |
+| `EURGBP`, `EURJPY`, `GBPJPY`, `AUDJPY`, `EURCHF`, `CADJPY` | Major crosses |
+
+### Metals (Yahoo Finance fallback — automatic)
+
+| Input    | Maps to | Source             |
+|----------|---------|--------------------|
+| `XAUUSD` | `GC=F`  | Yahoo gold futures |
+| `XAGUSD` | `SI=F`  | Yahoo silver futures |
+
+When Twelve Data returns no data for a symbol (e.g. gold on the free tier),
+the data provider transparently falls back to Yahoo Finance via `yfinance`.
+Logs are tagged `[live]` for Twelve Data and `[yf]` for Yahoo so you can
+verify the source.
+
+Note: `GC=F` is the futures contract, not spot XAU/USD. Direction is
+identical, but absolute price differs from your broker's spot quote by a
+small basis (typically $5–15). Use signal direction; execute at your
+broker's price.
+
+### Symbols not on free tier and not mapped to yfinance
+
+Any symbol Twelve Data doesn't return data for and isn't in the yfinance
+fallback map will fail with `❌ Data unavailable`. The bot does not
+generate synthetic data.
+
+---
+
+## API Limits
+
+### Twelve Data — Free Tier
+
+| Limit               | Value         |
+|---------------------|---------------|
+| Credits per minute  | 8             |
+| Credits per day     | 800           |
+| Credits per scan    | 21 (3 TFs × 7 pairs) |
+| Theoretical scans/day | ~38         |
+
+Each scan makes 3 batch API calls (one per unique timeframe: M15, H1, D1).
+The 8-credits/minute limit means subsequent calls wait ~60 s for the
+window to clear, so a full scan takes about **2 minutes**. Subsequent
+scans within `refresh_interval` are served from cache and make zero API
+calls.
+
+### Yahoo Finance — Unofficial
+
+No published rate limits. Used only for symbols mapped in
+`_YF_FALLBACK_MAP` (currently gold and silver futures). Daily/weekly
+intervals have unlimited history; intraday is capped (1m: last 7 days,
+15m–1h: last 60 days), which is more than enough for the engine's
+~210 bar requirement.
+
+### `--watch` mode
+
+`main.py --watch` re-scans every 5 minutes. With 7 pairs that's
+21 credits × 12 = 252 credits/hour — still within the daily limit if
+you run it for under ~3 hours. For longer runs, scope down with
+`--pairs` or stop and restart on demand.
+
+---
+
+## Configuration
+
+Most user-tunable values live in [config.py](config.py):
+
+- `MAJOR_PAIRS`, `CROSS_PAIRS` — which symbols `/scan` and `--all` cover
+- `SYMBOL_DISPLAY` — pretty names for messages
+- `PIP_VALUES` — per-symbol pip value override (gold = 0.01, JPY = 0.01)
+- `PlatformConfig` defaults — lookback days, refresh interval, indicator periods, MTF threshold, R:R ratio
+
+Bot-specific knobs in [telegram_bot.py](telegram_bot.py):
+
+- `SCAN_TIMEFRAMES` — which primary timeframes the bot reports on (default: `["M15", "H1"]`)
+- `_TF_CONFIRMATIONS` — confirmation TFs per primary
+- `_YF_FALLBACK_MAP` (in [data_provider.py](data_provider.py)) — Yahoo symbol mapping

analyze_trades.py

@@ -0,0 +1,240 @@
+#!/usr/bin/env python3
+"""
+Phase 1 analysis: post-hoc filters on an existing trades.csv.
+
+Asks "which simple filters, applied to the trades the engine already
+produced, would have moved total R toward zero or positive?" — without
+re-running the backtest engine.
+
+Discipline: candidate filters are derived on the **first 70% of trades
+by entry_time** (train slice), then evaluated on the **last 30%** (test
+slice). This is the same time-based split the meta-model uses, so the
+numbers are comparable.
+
+Outputs (printed + saved to backtest_results_all/phase1_summary.txt):
+  - Session-of-day breakdown
+  - Day-of-week breakdown
+  - Pair × regime cross-tab
+  - Per-pair contribution
+  - Combined-filter sweep (drop-worst-hours / drop-worst-pairs / both)
+
+Usage:
+    python analyze_trades.py
+    python analyze_trades.py --trades backtest_results_all/trades.csv
+"""
+
+from __future__ import annotations
+
+import argparse
+import sys
+from pathlib import Path
+
+import pandas as pd
+
+
+def fmt(df: pd.DataFrame) -> pd.DataFrame:
+    return df.round(3)
+
+
+def split(df: pd.DataFrame, train_frac: float = 0.7):
+    df = df.sort_values("entry_time").reset_index(drop=True)
+    cutoff = int(len(df) * train_frac)
+    return df.iloc[:cutoff].copy(), df.iloc[cutoff:].copy()
+
+
+def stats(df: pd.DataFrame) -> dict:
+    if len(df) == 0:
+        return {"n": 0, "wr": 0.0, "avg_r": 0.0, "total_r": 0.0, "pf": 0.0}
+    wins = (df["r_multiple"] > 0).sum()
+    losses = (df["r_multiple"] <= 0).sum()
+    gross_win = df.loc[df["r_multiple"] > 0, "r_multiple"].sum()
+    gross_loss = -df.loc[df["r_multiple"] <= 0, "r_multiple"].sum()
+    return {
+        "n":       len(df),
+        "wr":      wins / len(df) * 100,
+        "avg_r":   df["r_multiple"].mean(),
+        "total_r": df["r_multiple"].sum(),
+        "pf":      gross_win / gross_loss if gross_loss > 0 else float("inf"),
+    }
+
+
+def stats_line(label: str, df: pd.DataFrame) -> str:
+    s = stats(df)
+    return (
+        f"{label:<40s}"
+        f"  n={s['n']:>5}  WR={s['wr']:5.1f}%"
+        f"  avgR={s['avg_r']:+6.3f}  totalR={s['total_r']:+9.2f}  PF={s['pf']:.2f}"
+    )
+
+
+def main() -> int:
+    ap = argparse.ArgumentParser(description="Phase 1 post-hoc analysis on backtest trades.")
+    ap.add_argument("--trades", default="backtest_results_all/trades.csv")
+    ap.add_argument("--out", default="backtest_results_all/phase1_summary.txt")
+    ap.add_argument("--train-frac", type=float, default=0.7)
+    args = ap.parse_args()
+
+    p = Path(args.trades)
+    if not p.exists():
+        print(f"❌ {p} not found")
+        return 1
+
+    df = pd.read_csv(p)
+    df["entry_time"] = pd.to_datetime(df["entry_time"])
+    df["hour"] = df["entry_time"].dt.hour
+    df["dow"] = df["entry_time"].dt.dayofweek
+    df["winner"] = (df["r_multiple"] > 0).astype(int)
+
+    train, test = split(df, args.train_frac)
+    out_lines = []
+
+    def emit(line: str = ""):
+        print(line)
+        out_lines.append(line)
+
+    emit("=" * 92)
+    emit(f"PHASE 1 ANALYSIS — {len(df):,} trades total")
+    emit(f"  full:  {df['entry_time'].min().date()} → {df['entry_time'].max().date()}")
+    emit(f"  train: {train['entry_time'].min().date()} → {train['entry_time'].max().date()}  ({len(train):,} trades)")
+    emit(f"  test:  {test['entry_time'].min().date()} → {test['entry_time'].max().date()}  ({len(test):,} trades)")
+    emit("=" * 92)
+    emit()
+    emit(stats_line("BASELINE  full",  df))
+    emit(stats_line("BASELINE  train", train))
+    emit(stats_line("BASELINE  test",  test))
+    emit()
+
+    # ── Per-hour (training set, used for filter selection) ──────────
+    emit("─" * 92)
+    emit("HOUR (UTC) — train slice  (used for filter selection)")
+    emit("─" * 92)
+    hr_stats = train.groupby("hour").apply(
+        lambda x: pd.Series(stats(x)), include_groups=False,
+    ).round(3)
+    emit(hr_stats.to_string())
+    emit()
+
+    # Bottom-K hours by avg_r
+    bad_hours_4 = hr_stats.sort_values("avg_r").head(4).index.tolist()
+    bad_hours_6 = hr_stats.sort_values("avg_r").head(6).index.tolist()
+    emit(f"  Worst 4 hours by avg_r (train): {sorted(bad_hours_4)}")
+    emit(f"  Worst 6 hours by avg_r (train): {sorted(bad_hours_6)}")
+    emit()
+
+    # ── Per-DOW (training set) ──────────────────────────────────────
+    emit("─" * 92)
+    emit("DAY OF WEEK — train slice  (0=Mon, 4=Fri)")
+    emit("─" * 92)
+    dow_stats = train.groupby("dow").apply(
+        lambda x: pd.Series(stats(x)), include_groups=False,
+    ).round(3)
+    emit(dow_stats.to_string())
+    emit()
+    bad_dow = dow_stats.sort_values("avg_r").head(2).index.tolist()
+    emit(f"  Worst 2 days by avg_r (train): {sorted(bad_dow)}")
+    emit()
+
+    # ── Per-pair (training set) ─────────────────────────────────────
+    emit("─" * 92)
+    emit("PAIR — train slice")
+    emit("─" * 92)
+    pair_stats = train.groupby("pair").apply(
+        lambda x: pd.Series(stats(x)), include_groups=False,
+    ).round(3)
+    emit(pair_stats.to_string())
+    emit()
+    bad_pairs = pair_stats.sort_values("total_r").head(3).index.tolist()
+    emit(f"  Worst 3 pairs by total_r (train): {bad_pairs}")
+    emit()
+
+    # ── Per-regime × direction (training set) ───────────────────────
+    emit("─" * 92)
+    emit("REGIME × DIRECTION — train slice")
+    emit("─" * 92)
+    reg_stats = train.groupby(["regime", "direction"]).apply(
+        lambda x: pd.Series(stats(x)), include_groups=False,
+    ).round(3)
+    emit(reg_stats.to_string())
+    emit()
+
+    # ── Filter sweep ON THE TEST SET (rule selected on train) ───────
+    emit("=" * 92)
+    emit("FILTER SWEEP — applied to TEST slice (rule chosen from train, no leakage)")
+    emit("=" * 92)
+    test_filtered_dropworst4hours = test[~test["hour"].isin(bad_hours_4)]
+    test_filtered_dropworst6hours = test[~test["hour"].isin(bad_hours_6)]
+    test_filtered_droppairs       = test[~test["pair"].isin(bad_pairs)]
+    test_filtered_droprangingsell = test[~((test["regime"] == "ranging") & (test["direction"] == "SELL"))]
+    test_filtered_only_ranging    = test[test["regime"] == "ranging"]
+    test_filtered_drop_dow        = test[~test["dow"].isin(bad_dow)]
+
+    test_combo_hr_pair = test[
+        (~test["hour"].isin(bad_hours_4)) & (~test["pair"].isin(bad_pairs))
+    ]
+    test_combo_hr_pair_dow = test[
+        (~test["hour"].isin(bad_hours_4))
+        & (~test["pair"].isin(bad_pairs))
+        & (~test["dow"].isin(bad_dow))
+    ]
+
+    emit()
+    emit(stats_line("test  baseline (no filter)",         test))
+    emit(stats_line("test  drop 4 worst hours",           test_filtered_dropworst4hours))
+    emit(stats_line("test  drop 6 worst hours",           test_filtered_dropworst6hours))
+    emit(stats_line("test  drop 3 worst pairs",           test_filtered_droppairs))
+    emit(stats_line("test  drop 2 worst DOWs",            test_filtered_drop_dow))
+    emit(stats_line("test  ONLY ranging regime",          test_filtered_only_ranging))
+    emit(stats_line("test  hours+pairs (combined)",       test_combo_hr_pair))
+    emit(stats_line("test  hours+pairs+DOW (combined)",   test_combo_hr_pair_dow))
+    emit()
+
+    # ── Same filters applied to TRAIN slice for reference ───────────
+    emit("─" * 92)
+    emit("Same filters — TRAIN slice for reference (these are the in-sample numbers)")
+    emit("─" * 92)
+    train_drop_hours = train[~train["hour"].isin(bad_hours_4)]
+    train_drop_pairs = train[~train["pair"].isin(bad_pairs)]
+    train_combo      = train[(~train["hour"].isin(bad_hours_4)) & (~train["pair"].isin(bad_pairs))]
+
+    emit(stats_line("train baseline (no filter)",         train))
+    emit(stats_line("train drop 4 worst hours",           train_drop_hours))
+    emit(stats_line("train drop 3 worst pairs",           train_drop_pairs))
+    emit(stats_line("train hours+pairs (combined)",       train_combo))
+    emit()
+
+    # ── Verdict ─────────────────────────────────────────────────────
+    emit("=" * 92)
+    emit("VERDICT")
+    emit("=" * 92)
+    test_base = stats(test)
+    test_combo = stats(test_combo_hr_pair_dow)
+    delta_r = test_combo["total_r"] - test_base["total_r"]
+    delta_avgr = test_combo["avg_r"] - test_base["avg_r"]
+    emit(f"  Best combined filter (hours+pairs+DOW) on TEST set:")
+    emit(f"    {test_base['n']} → {test_combo['n']} trades  "
+         f"({test_combo['n']/test_base['n']*100:.0f}% retained)")
+    emit(f"    total R: {test_base['total_r']:+.2f} → {test_combo['total_r']:+.2f}  "
+         f"(Δ {delta_r:+.2f})")
+    emit(f"    avg R:   {test_base['avg_r']:+.3f} → {test_combo['avg_r']:+.3f}  "
+         f"(Δ {delta_avgr:+.3f})")
+    emit(f"    win rate: {test_base['wr']:.1f}% → {test_combo['wr']:.1f}%")
+    emit()
+    if test_combo["total_r"] > 0:
+        emit("  ✅ Filtered subset is profitable on the held-out test set.")
+        emit("     → Phase 2 (geometry grid) may be unnecessary; this filter alone may suffice.")
+    elif test_combo["avg_r"] > test_base["avg_r"]:
+        emit("  ⚠️  Filtered subset improves but is still negative.")
+        emit("     → Phase 2 (geometry grid) needed to find profitable structure.")
+    else:
+        emit("  ❌ Filters do not survive the train/test split — rejected as noise.")
+        emit("     → Skip Phase 1's filter list; rely on Phase 2 (geometry) instead.")
+
+    Path(args.out).parent.mkdir(parents=True, exist_ok=True)
+    Path(args.out).write_text("\n".join(out_lines))
+    emit()
+    emit(f"📁 Saved: {args.out}")
+    return 0
+
+
+if __name__ == "__main__":
+    sys.exit(main())

backtest.py

@@ -0,0 +1,712 @@
+#!/usr/bin/env python3
+"""
+Walk-forward backtest harness for the Forex signal engine.
+
+Replays historical OHLCV (loaded from history/<PAIR>_<TF>.csv) through
+the existing SignalAggregator with strict no-look-ahead semantics: at
+each primary-TF bar close, the engine sees only bars that have *fully
+closed by that moment* — never the bar still in formation.
+
+Trade simulation: first-touch (SL vs TP1) on the primary timeframe,
+worst-case if both touched in the same bar (assume SL first), with
+per-pip spread cost deducted from the final R-multiple.
+
+Outputs:
+    backtest_results/trades.csv     — every simulated trade
+    backtest_results/summary.txt    — per-grade win rate / avg R / PF
+    backtest_results/equity.png     — cumulative R curve (if --plot)
+    backtest_results/by_grade.png   — win rate by grade (if --plot)
+
+Usage:
+    python backtest.py --primary M15 --pairs EURUSD
+    python backtest.py --primary H1 --pairs EURUSD GBPUSD USDJPY --plot
+    python backtest.py --primary M15 --grades A+ A
+"""
+
+from __future__ import annotations
+
+import argparse
+import logging
+import sys
+from dataclasses import dataclass, field
+from pathlib import Path
+from typing import Dict, List, Optional, Tuple
+
+import pandas as pd
+
+from config import PlatformConfig, SignalStrength, Timeframe, MAJOR_PAIRS
+from data_provider import DataUnavailableError, ForexDataProvider
+from signal_aggregator import SignalAggregator, SignalReport
+
+logger = logging.getLogger(__name__)
+
+HISTORY_DIR = Path("history")
+RESULTS_DIR = Path("backtest_results")
+
+_TF_MAP = {
+    "M1":  Timeframe.M1,  "M5":  Timeframe.M5,
+    "M15": Timeframe.M15, "M30": Timeframe.M30,
+    "H1":  Timeframe.H1,  "H4":  Timeframe.H4,
+    "D1":  Timeframe.D1,  "W1":  Timeframe.W1,
+}
+
+_TF_INTERVAL = {
+    Timeframe.M1:  pd.Timedelta(minutes=1),
+    Timeframe.M5:  pd.Timedelta(minutes=5),
+    Timeframe.M15: pd.Timedelta(minutes=15),
+    Timeframe.M30: pd.Timedelta(minutes=30),
+    Timeframe.H1:  pd.Timedelta(hours=1),
+    Timeframe.H4:  pd.Timedelta(hours=4),
+    Timeframe.D1:  pd.Timedelta(days=1),
+    Timeframe.W1:  pd.Timedelta(weeks=1),
+}
+
+# Confirmation TFs per primary — same shape Telegram bot uses
+_TF_CONFIRMATIONS: Dict[Timeframe, List[Timeframe]] = {
+    Timeframe.M15: [Timeframe.H1,  Timeframe.D1],
+    Timeframe.M30: [Timeframe.H1,  Timeframe.D1],
+    Timeframe.H1:  [Timeframe.M15, Timeframe.D1],
+    Timeframe.H4:  [Timeframe.H1,  Timeframe.D1],
+    Timeframe.D1:  [Timeframe.H4],
+}
+
+
+# ══════════════════════════════════════════════════════════════════════
+#  Stub news engine (no historical news data — returns neutral)
+# ══════════════════════════════════════════════════════════════════════
+
+class _StubFundamentalContext:
+    """Placeholder — backtest doesn't replay news, so context is empty."""
+    pass
+
+
+class StubNewsEngine:
+    """
+    Drop-in for `NewsEngine` in backtest mode. Returns neutral sentiment
+    and 'clear' event risk for every call. The signal aggregator's news
+    adjustments thus become no-ops, isolating the test to pure technicals.
+    """
+
+    def get_fundamental_context(self):
+        return _StubFundamentalContext()
+
+    def get_event_risk_level(self, symbol, context):
+        return ("clear", "")
+
+    def get_pair_sentiment(self, symbol, context):
+        return (0.0, "")
+
+
+# ══════════════════════════════════════════════════════════════════════
+#  Backtest data provider (slices history at "as of" timestamp)
+# ══════════════════════════════════════════════════════════════════════
+
+class BacktestProvider(ForexDataProvider):
+    """
+    Replaces network fetches with in-memory slices of preloaded OHLCV.
+
+    `set_time(ts)` sets the cut-off; `fetch_ohlcv` returns only bars whose
+    *close* time is ≤ ts. Bars that are still in formation (open ≤ ts but
+    close > ts) are excluded — this is the no-look-ahead guarantee.
+    """
+
+    def __init__(
+        self,
+        config: PlatformConfig,
+        history: Dict[Tuple[str, Timeframe], pd.DataFrame],
+    ):
+        super().__init__(config)
+        self._history = history
+        self._current_ts: Optional[pd.Timestamp] = None
+
+    def set_time(self, ts: pd.Timestamp) -> None:
+        if ts.tzinfo is None:
+            ts = ts.tz_localize("UTC")
+        self._current_ts = ts
+
+    def fetch_ohlcv(
+        self,
+        symbol: str,
+        timeframe: Timeframe,
+        lookback_days: Optional[int] = None,
+    ) -> pd.DataFrame:
+        df = self._history.get((symbol, timeframe))
+        if df is None:
+            raise DataUnavailableError(
+                f"No history loaded for {symbol} {timeframe.value}. "
+                f"Run seed_history.py with --tfs including this TF."
+            )
+        if self._current_ts is not None:
+            interval = _TF_INTERVAL[timeframe]
+            df = df[df.index + interval <= self._current_ts]
+        if df.empty:
+            raise DataUnavailableError(
+                f"No bars before {self._current_ts} for {symbol} {timeframe.value}"
+            )
+        return df
+
+    def prefetch_batch(self, symbols, timeframes, lookback_days=None):
+        return  # no-op: data is in memory
+
+    def get_spread_estimate(self, symbol: str) -> float:
+        # Backtest doesn't need a live spread estimate; return a fixed value
+        # so the aggregator's spread field is filled but does not affect SL/TP.
+        return 0.0
+
+
+# ══════════════════════════════════════════════════════════════════════
+#  History loader
+# ══════════════════════════════════════════════════════════════════════
+
+def load_history(
+    pairs: List[str],
+    timeframes: List[Timeframe],
+    history_dir: Path = HISTORY_DIR,
+) -> Dict[Tuple[str, Timeframe], pd.DataFrame]:
+    """
+    Load CSVs produced by seed_history.py into a dict keyed by (symbol, tf).
+
+    Drops CSVs that are missing — caller decides whether to abort or continue
+    with reduced TF coverage.
+    """
+    history: Dict[Tuple[str, Timeframe], pd.DataFrame] = {}
+    missing: List[str] = []
+
+    for symbol in pairs:
+        for tf in timeframes:
+            csv_p = history_dir / f"{symbol.replace('=X', '')}_{tf.value}.csv"
+            if not csv_p.exists():
+                missing.append(str(csv_p))
+                continue
+            df = pd.read_csv(csv_p, index_col="Datetime", parse_dates=["Datetime"])
+            if df.index.tz is None:
+                df.index = df.index.tz_localize("UTC")
+            history[(symbol, tf)] = df.sort_index()
+            logger.info(
+                f"loaded {symbol} {tf.value}: {len(df):,} bars "
+                f"({df.index.min().date()} → {df.index.max().date()})"
+            )
+
+    if missing:
+        logger.warning(f"missing CSVs ({len(missing)}): {missing[:5]}…")
+
+    return history
+
+
+# ══════════════════════════════════════════════════════════════════════
+#  Trade record + first-touch simulator
+# ══════════════════════════════════════════════════════════════════════
+
+@dataclass
+class Trade:
+    symbol: str
+    pair_display: str
+    primary_tf: str
+    grade: str
+    direction: str
+    entry_time: pd.Timestamp
+    entry_price: float
+    sl: float
+    tp1: float
+    tp2: float
+    risk_pips: float
+    reward_pips: float
+    score: float
+    confidence: float
+    regime: str
+    mtf_aligned: bool
+
+    # Filled by simulation
+    exit_time: Optional[pd.Timestamp] = None
+    exit_price: float = 0.0
+    outcome: str = "OPEN"     # WIN_TP1, LOSS_SL, TIMEOUT, INVALID
+    bars_held: int = 0
+    r_multiple: float = 0.0   # net R after spread cost
+
+
+def simulate_trade(
+    trade: Trade,
+    forward_bars: pd.DataFrame,
+    max_bars: int,
+    spread_pips: float,
+) -> Trade:
+    """
+    First-touch simulation on the primary timeframe.
+
+    Conservatives:
+    - If both SL and TP1 touch within the same bar, assume SL first.
+    - Spread cost (entry + exit round-trip) is deducted from final R.
+    - TP2 is recorded but not used in this baseline simulation — we cap at
+      TP1 to keep the R-multiple math clean. Extend with breakeven-trail
+      management later if needed.
+    """
+    if trade.risk_pips <= 0:
+        trade.outcome = "INVALID"
+        return trade
+
+    spread_r = spread_pips / trade.risk_pips
+    bars_examined = 0
+
+    for ts, bar in forward_bars.iterrows():
+        bars_examined += 1
+        if bars_examined > max_bars:
+            break
+
+        bar_high = bar["High"]
+        bar_low  = bar["Low"]
+
+        if trade.direction == "BUY":
+            sl_hit = bar_low  <= trade.sl
+            tp_hit = bar_high >= trade.tp1
+        else:
+            sl_hit = bar_high >= trade.sl
+            tp_hit = bar_low  <= trade.tp1
+
+        if sl_hit:
+            trade.outcome    = "LOSS_SL"
+            trade.exit_time  = ts
+            trade.exit_price = trade.sl
+            trade.bars_held  = bars_examined
+            trade.r_multiple = -1.0 - spread_r
+            return trade
+
+        if tp_hit:
+            trade.outcome    = "WIN_TP1"
+            trade.exit_time  = ts
+            trade.exit_price = trade.tp1
+            trade.bars_held  = bars_examined
+            # TP1 sits at risk × RR away from entry → R = reward_pips / risk_pips
+            trade.r_multiple = (trade.reward_pips / trade.risk_pips) - spread_r
+            return trade
+
+    # Timeout — exit at last seen close
+    if bars_examined == 0:
+        trade.outcome = "INVALID"
+        return trade
+
+    last_idx = bars_examined - 1
+    last = forward_bars.iloc[last_idx]
+    last_ts = forward_bars.index[last_idx]
+
+    if trade.direction == "BUY":
+        pnl_price = last["Close"] - trade.entry_price
+    else:
+        pnl_price = trade.entry_price - last["Close"]
+
+    sl_distance_price = abs(trade.entry_price - trade.sl)
+    pnl_r = pnl_price / sl_distance_price if sl_distance_price > 0 else 0.0
+
+    trade.outcome    = "TIMEOUT"
+    trade.exit_time  = last_ts
+    trade.exit_price = float(last["Close"])
+    trade.bars_held  = bars_examined
+    trade.r_multiple = pnl_r - spread_r
+    return trade
+
+
+# ══════════════════════════════════════════════════════════════════════
+#  Backtester (the replay loop)
+# ══════════════════════════════════════════════════════════════════════
+
+class Backtester:
+    def __init__(
+        self,
+        primary_tf: Timeframe,
+        pairs: List[str],
+        history: Dict[Tuple[str, Timeframe], pd.DataFrame],
+        target_grades: List[str],
+        max_hold_bars: int = 100,
+        spread_pips: float = 1.0,
+        warmup_bars: int = 250,
+    ):
+        self.primary_tf      = primary_tf
+        self.pairs           = pairs
+        self.history         = history
+        self.target_grades   = set(target_grades)
+        self.max_hold_bars   = max_hold_bars
+        self.spread_pips     = spread_pips
+        self.warmup_bars     = warmup_bars
+        self.confirmation_tfs = _TF_CONFIRMATIONS.get(primary_tf, [])
+
+    def _build_config(self, symbol: str) -> PlatformConfig:
+        cfg = PlatformConfig()
+        cfg.primary_timeframe       = self.primary_tf
+        cfg.confirmation_timeframes = list(self.confirmation_tfs)
+        cfg.watch_pairs             = [symbol]
+        return cfg
+
+    def _make_aggregator(self, config: PlatformConfig, provider: BacktestProvider) -> SignalAggregator:
+        agg = SignalAggregator(config, data_provider=provider)
+        # Replace live news engine with neutral stub (no historical news data)
+        agg.news_engine = StubNewsEngine()
+        return agg
+
+    def run(self) -> List[Trade]:
+        trades: List[Trade] = []
+        provider = BacktestProvider(PlatformConfig(), self.history)
+
+        for symbol in self.pairs:
+            primary_df = self.history.get((symbol, self.primary_tf))
+            if primary_df is None or primary_df.empty:
+                logger.warning(f"{symbol}: no primary {self.primary_tf.value} data, skipping")
+                continue
+
+            config = self._build_config(symbol)
+            aggregator = self._make_aggregator(config, provider)
+
+            interval = _TF_INTERVAL[self.primary_tf]
+            timestamps = primary_df.index
+            n_total = len(timestamps)
+
+            print(f"\n📊 {symbol} {self.primary_tf.value}: "
+                  f"{n_total:,} bars, warmup={self.warmup_bars}")
+
+            open_until: Optional[pd.Timestamp] = None  # block overlapping entries
+            n_signals = 0
+
+            for i, bar_open_ts in enumerate(timestamps):
+                if i < self.warmup_bars:
+                    continue
+                if open_until is not None and bar_open_ts < open_until:
+                    continue  # previous trade still open — skip
+
+                # Run engine "as if it's now this bar's close"
+                bar_close_ts = bar_open_ts + interval
+                provider.set_time(bar_close_ts)
+
+                try:
+                    report = aggregator.analyze_pair(symbol)
+                except DataUnavailableError:
+                    continue
+                except Exception as e:
+                    logger.debug(f"engine error at {bar_open_ts}: {e}")
+                    continue
+
+                if report.signal_quality not in self.target_grades:
+                    continue
+                if report.trade_setup is None:
+                    continue
+                if report.final_signal == SignalStrength.NEUTRAL:
+                    continue
+
+                ts = report.trade_setup
+                trade = Trade(
+                    symbol       = symbol,
+                    pair_display = report.display_name,
+                    primary_tf   = self.primary_tf.value,
+                    grade        = report.signal_quality,
+                    direction    = ts.direction,
+                    entry_time   = bar_open_ts,
+                    entry_price  = ts.entry_price,
+                    sl           = ts.stop_loss,
+                    tp1          = ts.take_profit_1,
+                    tp2          = ts.take_profit_2,
+                    risk_pips    = ts.risk_pips,
+                    reward_pips  = ts.reward_pips,
+                    score        = report.final_score,
+                    confidence   = report.final_confidence,
+                    regime       = report.market_regime,
+                    mtf_aligned  = report.mtf_agreement,
+                )
+
+                # Forward window for simulation
+                forward = primary_df.iloc[i + 1 : i + 1 + self.max_hold_bars]
+                if forward.empty:
+                    continue
+
+                trade = simulate_trade(
+                    trade,
+                    forward,
+                    max_bars=self.max_hold_bars,
+                    spread_pips=self.spread_pips,
+                )
+                trades.append(trade)
+                n_signals += 1
+
+                if trade.exit_time is not None:
+                    open_until = trade.exit_time
+
+                if n_signals % 25 == 0:
+                    print(f"   [{i:>6}/{n_total}]  {n_signals} trades so far "
+                          f"(latest: {trade.grade} {trade.direction} {trade.outcome} "
+                          f"R={trade.r_multiple:+.2f})")
+
+            print(f"   ✅ {symbol}: {n_signals} trades")
+
+        return trades
+
+
+# ══════════════════════════════════════════════════════════════════════
+#  Reporting
+# ══════════════════════════════════════════════════════════════════════
+
+def trades_to_dataframe(trades: List[Trade]) -> pd.DataFrame:
+    rows = [
+        {
+            "entry_time":   t.entry_time,
+            "exit_time":    t.exit_time,
+            "symbol":       t.symbol,
+            "pair":         t.pair_display,
+            "tf":           t.primary_tf,
+            "grade":        t.grade,
+            "direction":    t.direction,
+            "entry":        t.entry_price,
+            "sl":           t.sl,
+            "tp1":          t.tp1,
+            "exit_price":   t.exit_price,
+            "outcome":      t.outcome,
+            "bars_held":    t.bars_held,
+            "r_multiple":   t.r_multiple,
+            "score":        t.score,
+            "confidence":   t.confidence,
+            "regime":       t.regime,
+            "mtf_aligned":  t.mtf_aligned,
+            "risk_pips":    t.risk_pips,
+            "reward_pips":  t.reward_pips,
+        }
+        for t in trades
+    ]
+    return pd.DataFrame(rows)
+
+
+def summarize(df: pd.DataFrame) -> str:
+    """Per-grade win rate / average R / profit factor summary."""
+    if df.empty:
+        return "No trades recorded."
+
+    lines: List[str] = []
+    lines.append("=" * 78)
+    lines.append(f"BACKTEST SUMMARY  ({len(df)} trades)")
+    lines.append(f"  Date range: {df['entry_time'].min()} → {df['entry_time'].max()}")
+    lines.append("=" * 78)
+    lines.append("")
+
+    overall = _grade_block("ALL", df)
+    lines.append(overall)
+    lines.append("")
+
+    for grade in ["A+", "A", "B", "C", "D"]:
+        sub = df[df["grade"] == grade]
+        if sub.empty:
+            continue
+        lines.append(_grade_block(grade, sub))
+        lines.append("")
+
+    # Per-pair breakdown
+    lines.append("─" * 78)
+    lines.append("PER-PAIR SUMMARY")
+    lines.append("─" * 78)
+    by_pair = df.groupby("pair").agg(
+        trades=("r_multiple", "count"),
+        win_rate=("r_multiple", lambda s: (s > 0).mean() * 100),
+        avg_r=("r_multiple", "mean"),
+        total_r=("r_multiple", "sum"),
+    ).round(2)
+    lines.append(by_pair.to_string())
+    lines.append("")
+
+    # Per-direction
+    lines.append("─" * 78)
+    lines.append("BY DIRECTION")
+    lines.append("─" * 78)
+    by_dir = df.groupby("direction").agg(
+        trades=("r_multiple", "count"),
+        win_rate=("r_multiple", lambda s: (s > 0).mean() * 100),
+        avg_r=("r_multiple", "mean"),
+    ).round(2)
+    lines.append(by_dir.to_string())
+    lines.append("")
+
+    return "\n".join(lines)
+
+
+def _grade_block(label: str, sub: pd.DataFrame) -> str:
+    n        = len(sub)
+    wins     = (sub["r_multiple"] > 0).sum()
+    losses   = (sub["r_multiple"] <= 0).sum()
+    win_rate = wins / n * 100 if n else 0.0
+    avg_r    = sub["r_multiple"].mean()
+    sum_r    = sub["r_multiple"].sum()
+
+    gross_win  = sub.loc[sub["r_multiple"] > 0, "r_multiple"].sum()
+    gross_loss = -sub.loc[sub["r_multiple"] <= 0, "r_multiple"].sum()
+    profit_factor = (gross_win / gross_loss) if gross_loss > 0 else float("inf")
+
+    # Equity-curve drawdown (in R)
+    cum = sub["r_multiple"].cumsum()
+    running_peak = cum.cummax()
+    drawdown = (cum - running_peak).min()
+
+    return (
+        f"{label:>4}: {n:>4} trades | "
+        f"win {win_rate:5.1f}% ({wins}W/{losses}L) | "
+        f"avg R {avg_r:+.3f} | total R {sum_r:+.2f} | "
+        f"PF {profit_factor:.2f} | maxDD {drawdown:.2f}R"
+    )
+
+
+def plot_results(df: pd.DataFrame, out_dir: Path) -> None:
+    """Generate equity curve and per-grade bar chart. Requires matplotlib."""
+    try:
+        import matplotlib
+        matplotlib.use("Agg")  # headless
+        import matplotlib.pyplot as plt
+    except ImportError:
+        print("⚠️  matplotlib not installed — skipping plots. "
+              "pip install matplotlib")
+        return
+
+    if df.empty:
+        return
+
+    df = df.sort_values("entry_time").reset_index(drop=True)
+    df["cum_r"] = df["r_multiple"].cumsum()
+
+    # Equity curve
+    fig, ax = plt.subplots(figsize=(11, 5))
+    ax.plot(df["entry_time"], df["cum_r"], linewidth=1.5)
+    ax.axhline(0, color="gray", linewidth=0.5)
+    ax.fill_between(df["entry_time"], df["cum_r"], 0,
+                    where=df["cum_r"] >= 0, alpha=0.2, color="green")
+    ax.fill_between(df["entry_time"], df["cum_r"], 0,
+                    where=df["cum_r"] < 0, alpha=0.2, color="red")
+    ax.set_title(f"Cumulative R  ({len(df)} trades, total {df['cum_r'].iloc[-1]:+.2f}R)")
+    ax.set_xlabel("Trade entry time")
+    ax.set_ylabel("Cumulative R (1R = risked amount per trade)")
+    ax.grid(True, alpha=0.3)
+    fig.tight_layout()
+    fig.savefig(out_dir / "equity.png", dpi=120)
+    plt.close(fig)
+
+    # Win rate by grade
+    grades = ["A+", "A", "B", "C", "D"]
+    grade_stats = []
+    for g in grades:
+        sub = df[df["grade"] == g]
+        if sub.empty:
+            continue
+        wr = (sub["r_multiple"] > 0).mean() * 100
+        avg_r = sub["r_multiple"].mean()
+        grade_stats.append((g, len(sub), wr, avg_r))
+
+    if grade_stats:
+        fig, (ax1, ax2) = plt.subplots(1, 2, figsize=(11, 4.5))
+        labels = [f"{g}\n(n={n})" for g, n, _, _ in grade_stats]
+        wrs    = [w for _, _, w, _ in grade_stats]
+        avgs   = [r for _, _, _, r in grade_stats]
+
+        ax1.bar(labels, wrs, color=["#16a34a", "#22c55e", "#84cc16", "#eab308", "#f97316"][:len(labels)])
+        ax1.axhline(50, linestyle="--", color="gray", linewidth=0.8)
+        ax1.set_ylabel("Win rate (%)")
+        ax1.set_title("Win rate by grade")
+        for i, w in enumerate(wrs):
+            ax1.text(i, w + 1, f"{w:.0f}%", ha="center", fontsize=9)
+
+        ax2.bar(labels, avgs, color=["#16a34a", "#22c55e", "#84cc16", "#eab308", "#f97316"][:len(labels)])
+        ax2.axhline(0, color="black", linewidth=0.5)
+        ax2.set_ylabel("Average R per trade")
+        ax2.set_title("Average R by grade")
+        for i, a in enumerate(avgs):
+            ax2.text(i, a + (0.02 if a >= 0 else -0.05), f"{a:+.2f}",
+                     ha="center", fontsize=9)
+
+        fig.tight_layout()
+        fig.savefig(out_dir / "by_grade.png", dpi=120)
+        plt.close(fig)
+
+
+# ══════════════════════════════════════════════════════════════════════
+#  CLI
+# ══════════════════════════════════════════════════════════════════════
+
+def main() -> int:
+    parser = argparse.ArgumentParser(
+        description="Walk-forward backtest of the Forex signal engine.",
+    )
+    parser.add_argument("--primary", default="M15",
+                        help="Primary timeframe (default M15)")
+    parser.add_argument("--pairs", nargs="+", default=None,
+                        help="Pairs to test (default: all majors)")
+    parser.add_argument("--grades", nargs="+", default=["A+", "A"],
+                        help='Grades to take (default "A+ A")')
+    parser.add_argument("--history-dir", default=str(HISTORY_DIR))
+    parser.add_argument("--out-dir", default=str(RESULTS_DIR))
+    parser.add_argument("--max-hold-bars", type=int, default=100,
+                        help="Max bars to hold a trade before timeout (default 100)")
+    parser.add_argument("--spread-pips", type=float, default=1.0,
+                        help="Round-trip spread cost in pips (default 1.0)")
+    parser.add_argument("--warmup-bars", type=int, default=250,
+                        help="Bars to skip at start so indicators stabilize (default 250)")
+    parser.add_argument("--plot", action="store_true",
+                        help="Generate equity / by-grade PNGs (needs matplotlib)")
+    parser.add_argument("-v", "--verbose", action="store_true")
+    args = parser.parse_args()
+
+    logging.basicConfig(
+        level=logging.INFO if args.verbose else logging.WARNING,
+        format="%(asctime)s [%(levelname)s] %(message)s",
+        datefmt="%H:%M:%S",
+    )
+
+    primary_tf = _TF_MAP[args.primary.upper()]
+    confirmations = _TF_CONFIRMATIONS.get(primary_tf, [])
+    needed_tfs = [primary_tf] + confirmations
+
+    pairs = args.pairs if args.pairs else MAJOR_PAIRS
+    pairs = [p if p.endswith("=X") else f"{p}=X" for p in pairs]
+
+    history_dir = Path(args.history_dir)
+    out_dir = Path(args.out_dir)
+    out_dir.mkdir(parents=True, exist_ok=True)
+
+    print(f"🔬 Backtest: primary={primary_tf.value}  "
+          f"confirmations={[t.value for t in confirmations]}")
+    print(f"   Pairs:  {[p.replace('=X','') for p in pairs]}")
+    print(f"   Grades: {args.grades}")
+    print(f"   History dir: {history_dir}/")
+    print(f"   Results dir: {out_dir}/")
+    print()
+
+    history = load_history(pairs, needed_tfs, history_dir=history_dir)
+    if not history:
+        print(f"❌ No history loaded from {history_dir}/. "
+              f"Run: python seed_history.py --tfs {' '.join(t.value for t in needed_tfs)}")
+        return 1
+
+    bt = Backtester(
+        primary_tf     = primary_tf,
+        pairs          = pairs,
+        history        = history,
+        target_grades  = args.grades,
+        max_hold_bars  = args.max_hold_bars,
+        spread_pips    = args.spread_pips,
+        warmup_bars    = args.warmup_bars,
+    )
+
+    trades = bt.run()
+
+    if not trades:
+        print("\n❌ No trades emitted. Try lowering --grades (e.g., 'B') or "
+              "check that primary+confirmation TFs are in history/.")
+        return 0
+
+    df = trades_to_dataframe(trades)
+    df.to_csv(out_dir / "trades.csv", index=False)
+
+    summary = summarize(df)
+    (out_dir / "summary.txt").write_text(summary)
+
+    print("\n" + summary)
+    print(f"\n📁 trades.csv  → {out_dir/'trades.csv'}")
+    print(f"📁 summary.txt → {out_dir/'summary.txt'}")
+
+    if args.plot:
+        plot_results(df, out_dir)
+        print(f"📈 equity.png  → {out_dir/'equity.png'}")
+        print(f"📈 by_grade.png → {out_dir/'by_grade.png'}")
+
+    return 0
+
+
+if __name__ == "__main__":
+    sys.exit(main())

config.py

@@ -48,7 +48,7 @@ CROSS_PAIRS = [
     "EURCHF=X", "CADJPY=X",
 ]
 
-# Yahoo Finance symbol mapping for display
+# Symbol display names
 SYMBOL_DISPLAY = {
     "EURUSD=X": "EUR/USD", "GBPUSD=X": "GBP/USD",
     "USDJPY=X": "USD/JPY", "USDCHF=X": "USD/CHF",
@@ -59,6 +59,11 @@ SYMBOL_DISPLAY = {
     "CADJPY=X": "CAD/JPY",
 }
 
+# Pip values per symbol (used for trade setup risk/reward in pips)
+# Forex defaults: 0.01 for JPY pairs, 0.0001 for everything else.
+# Override here for any pair that doesn't follow that rule.
+PIP_VALUES: Dict[str, float] = {}
+
 
 @dataclass
 class TechnicalConfig:
@@ -178,7 +183,7 @@ class PlatformConfig:
     # Analysis timeframes (primary + confirmation)
     primary_timeframe: Timeframe = Timeframe.H1
     confirmation_timeframes: List[Timeframe] = field(
-        default_factory=lambda: [Timeframe.M15, Timeframe.H4]
+        default_factory=lambda: [Timeframe.M15, Timeframe.D1]
     )
 
     # Data source
@@ -187,9 +192,25 @@ class PlatformConfig:
     # Twelve Data API key (overridden by TWELVE_DATA_API_KEY env var)
     twelve_data_api_key: str = "f563b22ffdfd4924955decbe1a8c8701"
 
-    # Dashboard refresh interval (seconds)
+    # Dashboard refresh interval (seconds) — used as a fallback when a
+    # timeframe isn't covered by cache_ttl_per_tf below.
     refresh_interval: int = 300  # 5 minutes
 
+    # Per-timeframe cache TTL (seconds). A 1-week candle changes less
+    # often than a 15-minute candle, so caching them all for the same
+    # 5 minutes wastes API credits. Tuned to roughly match each TF's
+    # bar interval so we re-fetch about once per new bar.
+    cache_ttl_per_tf: Dict[Timeframe, int] = field(default_factory=lambda: {
+        Timeframe.M1:  60,         # 1 minute
+        Timeframe.M5:  150,        # 2.5 minutes
+        Timeframe.M15: 300,        # 5 minutes
+        Timeframe.M30: 600,        # 10 minutes
+        Timeframe.H1:  1800,       # 30 minutes
+        Timeframe.H4:  7200,       # 2 hours
+        Timeframe.D1:  14400,      # 4 hours
+        Timeframe.W1:  86400,      # 24 hours
+    })
+
     # Logging
     log_level: str = "INFO"
     log_signals_to_file: bool = True

data_provider.py

@@ -1,25 +1,26 @@
 """
 Forex Signal Intelligence Platform - Data Provider
 ====================================================
-Retrieves and validates market data from Twelve Data API.
+Retrieves and validates currency-pair OHLCV from the Twelve Data API.
-Falls back to synthetic GBM data if the API is unavailable.
+Raises DataUnavailableError when the API is unreachable or credits exhausted.
+No fallback data source — forex pairs only.
 """
 
-import hashlib
 import logging
 import os
+import threading
 import time
 from collections import deque
 from datetime import datetime, timedelta, timezone
 from typing import Dict, List, Optional
 
-import numpy as np
 import pandas as pd
 import requests
 
-from config import PlatformConfig, Timeframe, SYMBOL_DISPLAY
+from config import PlatformConfig, Timeframe, SYMBOL_DISPLAY, PIP_VALUES
 
 logger = logging.getLogger(__name__)
+_no_data_warned: set = set()  # suppress duplicate "no data" warnings per symbol
 
 
 class DataIntegrityError(Exception):
@@ -27,6 +28,20 @@ class DataIntegrityError(Exception):
     pass
 
 
+class DataUnavailableError(Exception):
+    """Raised when live market data cannot be fetched (no key, rate limit, API error)."""
+    pass
+
+
+# ── Module-level shared rate limiter ───────────────────────────────
+# All ForexDataProvider instances share these so concurrent scans
+# don't each think they have a full 8-credit budget.
+_rate_lock          = threading.Lock()
+_rate_request_times: deque = deque()
+_rate_daily_credits: int   = 0
+_rate_daily_reset:   float = 0.0
+
+
 class ForexDataProvider:
     """
     Forex data provider backed by Twelve Data API.
@@ -60,9 +75,10 @@ class ForexDataProvider:
         Timeframe.W1:  0.143,
     }
 
-    # Free plan: 8 API credits per minute (each symbol = 1 credit)
+    # Free plan: 8 credits/minute, 800 credits/day
     RATE_LIMIT = 8
     RATE_WINDOW = 60.0
+    DAILY_LIMIT = 800
 
     def __init__(self, config: PlatformConfig):
         self.config = config
@@ -72,11 +88,12 @@ class ForexDataProvider:
         )
         self._cache: Dict[str, pd.DataFrame] = {}
         self._cache_timestamps: Dict[str, float] = {}
-        self._request_times: deque = deque()  # timestamps of recent API credits used
         self._session = requests.Session()
         self._session.headers.update({
-            "User-Agent": "Mozilla/5.0 (compatible; ForexIntel/1.0)"
+            "User-Agent": "Mozilla/5.0 (compatible; ForexIntel/1.0)",
         })
+        if self._api_key:
+            self._session.headers["Authorization"] = f"apikey {self._api_key}"
 
     @staticmethod
     def _to_td_symbol(symbol: str) -> str:
@@ -84,6 +101,10 @@ class ForexDataProvider:
         clean = symbol.replace("=X", "")
         return f"{clean[:3]}/{clean[3:]}"
 
+    def _cache_ttl(self, timeframe: Timeframe) -> int:
+        """Cache TTL for a given timeframe — falls back to refresh_interval."""
+        return self.config.cache_ttl_per_tf.get(timeframe, self.config.refresh_interval)
+
     def fetch_ohlcv(
         self,
         symbol: str,
@@ -101,10 +122,11 @@ class ForexDataProvider:
 
         cache_key = f"{symbol}_{timeframe.value}"
         now = time.time()
+        ttl = self._cache_ttl(timeframe)
 
         if (
             cache_key in self._cache_timestamps
-            and (now - self._cache_timestamps[cache_key]) < self.config.refresh_interval
+            and (now - self._cache_timestamps[cache_key]) < ttl
             and cache_key in self._cache
         ):
             cached = self._cache.get(cache_key)
@@ -135,9 +157,24 @@ class ForexDataProvider:
         if lookback_days is None:
             lookback_days = self.config.data_lookback_days
 
+        td_symbols = list(symbols)
+        if not td_symbols:
+            return
+
+        now = time.time()
         for tf in timeframes:
+            ttl = self._cache_ttl(tf)
+            cached = all(
+                f"{s}_{tf.value}" in self._cache_timestamps
+                and (now - self._cache_timestamps[f"{s}_{tf.value}"]) < ttl
+                and not self._cache.get(f"{s}_{tf.value}", pd.DataFrame()).empty
+                for s in td_symbols
+            )
+            if cached:
+                logger.debug(f"Skipping prefetch for {tf.value} — all {len(td_symbols)} symbols cached (TTL {ttl}s)")
+                continue
             try:
-                self._batch_fetch_timeframe(symbols, tf, lookback_days)
+                self._batch_fetch_timeframe(td_symbols, tf, lookback_days)
                 time.sleep(1.0)  # brief pause between TF requests
             except Exception as e:
                 logger.warning(f"Batch prefetch failed for {tf.value}: {e}")
@@ -189,10 +226,12 @@ class ForexDataProvider:
         """Return display name and pip value for a pair."""
         display = SYMBOL_DISPLAY.get(symbol, symbol)
         is_jpy = "JPY" in symbol
+        # Commodity-specific pip values take priority, then JPY pairs, then standard forex
+        pip_value = PIP_VALUES.get(symbol, 0.01 if is_jpy else 0.0001)
         return {
             "symbol": symbol,
             "display": display,
-            "pip_value": 0.01 if is_jpy else 0.0001,
+            "pip_value": pip_value,
             "is_jpy_pair": is_jpy,
         }
 
@@ -200,31 +239,64 @@ class ForexDataProvider:
 
     def _throttle(self, credits: int = 1) -> None:
         """
-        Block until `credits` API credits are available.
+        Block until `credits` per-minute API credits are available.
+        Raises DataUnavailableError if the daily limit is exhausted.
 
-        Twelve Data free tier: 8 credits/minute. Each symbol in any request
+        Uses module-level state so all ForexDataProvider instances (including
-        (single or batch) consumes 1 credit. This method enforces the limit
+        those running in parallel threads) share the same credit accounting.
-        regardless of whether requests are batched or individual.
+
+        Twelve Data free tier: 8 credits/minute, 800 credits/day.
         """
-        while True:
+        global _rate_daily_credits, _rate_daily_reset
+
+        with _rate_lock:
             now = time.time()
-            # Evict credits older than the rate window
-            while self._request_times and now - self._request_times[0] >= self.RATE_WINDOW:
-                self._request_times.popleft()
 
-            available = self.RATE_LIMIT - len(self._request_times)
+            # Reset daily counter after 24 hours
-            if available >= credits:
+            if _rate_daily_reset == 0.0:
-                for _ in range(credits):
+                _rate_daily_reset = now + 86400
-                    self._request_times.append(time.time())
+            if now >= _rate_daily_reset:
-                return
+                _rate_daily_credits = 0
+                _rate_daily_reset = now + 86400
 
-            # Wait until the oldest credit expires
+            # Daily limit check — raise immediately, no fallback
-            wait = (self._request_times[0] + self.RATE_WINDOW) - now + 0.2
+            if _rate_daily_credits + credits > self.DAILY_LIMIT:
-            logger.info(
+                raise DataUnavailableError(
-                f"Rate limit: {len(self._request_times)}/{self.RATE_LIMIT} credits used "
+                    f"Twelve Data daily API limit reached "
-                f"— waiting {wait:.1f}s for next window..."
+                    f"({_rate_daily_credits}/{self.DAILY_LIMIT} credits). "
-            )
+                    "Resets in 24 hours."
-            time.sleep(wait)
+                )
+
+            if _rate_daily_credits + credits > self.DAILY_LIMIT * 0.8:
+                logger.warning(
+                    f"Approaching daily API limit: "
+                    f"{_rate_daily_credits}/{self.DAILY_LIMIT} credits used."
+                )
+
+            # Per-minute rate limit — wait inside the lock so other threads
+            # queue up rather than over-counting available credits.
+            while True:
+                now = time.time()
+                while _rate_request_times and now - _rate_request_times[0] >= self.RATE_WINDOW:
+                    _rate_request_times.popleft()
+
+                available = self.RATE_LIMIT - len(_rate_request_times)
+                if available >= credits:
+                    ts = time.time()
+                    for _ in range(credits):
+                        _rate_request_times.append(ts)
+                    _rate_daily_credits += credits
+                    return
+
+                wait = (_rate_request_times[0] + self.RATE_WINDOW) - now + 0.2
+                logger.info(
+                    f"Rate limit: {len(_rate_request_times)}/{self.RATE_LIMIT} credits used "
+                    f"— waiting {wait:.1f}s for next window…"
+                )
+                # Release lock while sleeping so other threads can check state
+                _rate_lock.release()
+                time.sleep(wait)
+                _rate_lock.acquire()
 
     def _batch_fetch_timeframe(
         self, symbols: List[str], timeframe: Timeframe, lookback_days: int
@@ -244,7 +316,8 @@ class ForexDataProvider:
         )
         outputsize = max(outputsize, 300)
 
-        # Each symbol in a batch request costs 1 credit
+        # Each symbol in a batch request costs 1 credit — raises DataUnavailableError
+        # if daily limit reached; per-minute waits are handled inside _throttle.
         self._throttle(credits=len(symbols))
 
         resp = self._session.get(
@@ -253,7 +326,6 @@ class ForexDataProvider:
                 "symbol": ",".join(td_symbols),
                 "interval": interval,
                 "outputsize": outputsize,
-                "apikey": self._api_key,
                 "timezone": "UTC",
                 "format": "json",
             },
@@ -281,19 +353,22 @@ class ForexDataProvider:
 
                 values = sym_data.get("values")
                 if not values:
-                    logger.warning(f"No values in batch response for {td_sym}")
+                    if td_sym not in _no_data_warned:
+                        logger.warning(f"No data from Twelve Data for {td_sym} — symbol may require a paid plan.")
+                        _no_data_warned.add(td_sym)
                     continue
 
                 records = [
                     {
                         "Datetime": pd.Timestamp(v["datetime"], tz="UTC"),
-                        "Open":   float(v["open"]),
+                        "Open":   float(v["open"])   if v.get("open")   is not None else None,
-                        "High":   float(v["high"]),
+                        "High":   float(v["high"])   if v.get("high")   is not None else None,
-                        "Low":    float(v["low"]),
+                        "Low":    float(v["low"])    if v.get("low")    is not None else None,
-                        "Close":  float(v["close"]),
+                        "Close":  float(v["close"])  if v.get("close")  is not None else None,
-                        "Volume": float(v.get("volume") or 0),
+                        "Volume": float(v["volume"]) if v.get("volume") is not None else 0.0,
                     }
                     for v in reversed(values)
+                    if v.get("datetime")
                 ]
                 df = pd.DataFrame(records).set_index("Datetime")
                 df.index.name = "Datetime"
@@ -302,7 +377,7 @@ class ForexDataProvider:
                 cache_key = f"{symbol}_{timeframe.value}"
                 self._cache[cache_key] = df
                 self._cache_timestamps[cache_key] = now
-                logger.debug(f"Batch cached {len(df)} bars for {td_sym} @ {interval}")
+                logger.info(f"[live] {td_sym} {interval}: {len(df)} bars, last close={df['Close'].iloc[-1]:.5f}")
 
             except Exception as e:
                 logger.warning(f"Failed to parse batch data for {td_sym}: {e}")
@@ -312,12 +387,12 @@ class ForexDataProvider:
     ) -> pd.DataFrame:
         """
         Fetch OHLCV from Twelve Data REST API.
-        Falls back to synthetic GBM data if API is unavailable or key is missing.
+        Raises DataUnavailableError when data cannot be retrieved.
         """
         if not self._api_key:
-            logger.warning("No Twelve Data API key — using synthetic data. "
+            raise DataUnavailableError(
-                           "Set TWELVE_DATA_API_KEY env var or config.twelve_data_api_key.")
+                "No Twelve Data API key — set the TWELVE_DATA_API_KEY environment variable."
-            return self._generate_synthetic_data(symbol, timeframe, lookback_days)
+            )
 
         td_symbol = self._to_td_symbol(symbol)
         interval = self.TD_INTERVALS[timeframe]
@@ -327,6 +402,7 @@ class ForexDataProvider:
         )
         outputsize = max(outputsize, 300)
 
+        # Raises DataUnavailableError on daily limit; blocks on per-minute limit.
         self._throttle(credits=1)
 
         try:
@@ -336,7 +412,6 @@ class ForexDataProvider:
                     "symbol": td_symbol,
                     "interval": interval,
                     "outputsize": outputsize,
-                    "apikey": self._api_key,
                     "timezone": "UTC",
                     "format": "json",
                 },
@@ -346,15 +421,120 @@ class ForexDataProvider:
             data = resp.json()
 
             if data.get("status") == "error":
-                raise DataIntegrityError(
+                raise DataUnavailableError(
                     f"Twelve Data API error for {td_symbol}: {data.get('message', 'unknown')}"
                 )
 
             values = data.get("values")
             if not values:
-                raise DataIntegrityError(f"Empty response from Twelve Data for {td_symbol}")
+                raise DataUnavailableError(
+                    f"No data returned for {td_symbol} — symbol may require a paid plan."
+                )
+
+            records = [
+                {
+                    "Datetime": pd.Timestamp(v["datetime"], tz="UTC"),
+                    "Open":   float(v["open"])   if v.get("open")   is not None else None,
+                    "High":   float(v["high"])   if v.get("high")   is not None else None,
+                    "Low":    float(v["low"])    if v.get("low")    is not None else None,
+                    "Close":  float(v["close"])  if v.get("close")  is not None else None,
+                    "Volume": float(v["volume"]) if v.get("volume") is not None else 0.0,
+                }
+                for v in reversed(values)
+                if v.get("datetime")
+            ]
+
+            df = pd.DataFrame(records).set_index("Datetime")
+            df.index.name = "Datetime"
+            logger.info(f"[live] {td_symbol} {interval}: {len(df)} bars, last close={df['Close'].iloc[-1]:.5f}")
+            return df
+
+        except (DataUnavailableError, DataIntegrityError):
+            raise
+        except Exception as e:
+            raise DataUnavailableError(
+                f"Failed to fetch {td_symbol}: {e}"
+            ) from e
+
+    def fetch_historical(
+        self,
+        symbol: str,
+        timeframe: Timeframe,
+        start: datetime,
+        end: datetime,
+    ) -> pd.DataFrame:
+        """
+        Paginated historical OHLCV fetch for backtesting seed.
+
+        Walks backward from `end` to `start` in 5000-bar chunks (the free-tier
+        per-call max), 1 credit per chunk. Honours the shared rate limiter so
+        it cooperates with any concurrent live scans.
+
+        Returns a single concatenated DataFrame indexed by UTC timestamp.
+        """
+        if not self._api_key:
+            raise DataUnavailableError("No Twelve Data API key set.")
+
+        if start.tzinfo is None:
+            start = start.replace(tzinfo=timezone.utc)
+        if end.tzinfo is None:
+            end = end.replace(tzinfo=timezone.utc)
+
+        bars_per_day = self.BARS_PER_DAY[timeframe]
+        # Stay well clear of the 5000-bar cap — leave a small margin for
+        # weekend gaps that don't decrement bars on calendar-day arithmetic.
+        chunk_days = max(1, int(4500 / bars_per_day))
+
+        td_symbol = self._to_td_symbol(symbol)
+        interval = self.TD_INTERVALS[timeframe]
+
+        chunks: List[pd.DataFrame] = []
+        cursor_end = end
+
+        while cursor_end > start:
+            cursor_start = max(start, cursor_end - timedelta(days=chunk_days))
+
+            self._throttle(credits=1)
+
+            resp = self._session.get(
+                f"{self.TD_BASE_URL}/time_series",
+                params={
+                    "symbol":     td_symbol,
+                    "interval":   interval,
+                    "start_date": cursor_start.strftime("%Y-%m-%d %H:%M:%S"),
+                    "end_date":   cursor_end.strftime("%Y-%m-%d %H:%M:%S"),
+                    "outputsize": 5000,
+                    "timezone":   "UTC",
+                    "format":     "json",
+                },
+                timeout=30,
+            )
+            resp.raise_for_status()
+            data = resp.json()
+
+            if isinstance(data, dict) and data.get("status") == "error":
+                msg = data.get("message", "unknown")
+                # If we already have chunks, an error on a deeper-history call
+                # almost always means "you've walked past the available start."
+                # Treat as graceful end-of-history instead of fatal.
+                if chunks:
+                    logger.info(
+                        f"  {td_symbol} {interval}: end of available history "
+                        f"reached ({msg})"
+                    )
+                    break
+                raise DataUnavailableError(
+                    f"Twelve Data error for {td_symbol}: {msg}"
+                )
+
+            values = data.get("values") or []
+            if not values:
+                logger.info(
+                    f"  {td_symbol} {interval}: empty range "
+                    f"{cursor_start.date()}..{cursor_end.date()}, stopping"
+                )
+                break
 
-            # Values arrive newest-first; reverse to chronological order
             records = [
                 {
                     "Datetime": pd.Timestamp(v["datetime"], tz="UTC"),
@@ -364,119 +544,35 @@ class ForexDataProvider:
                     "Close":  float(v["close"]),
                     "Volume": float(v.get("volume") or 0),
                 }
-                for v in reversed(values)
+                for v in values if v.get("datetime")
             ]
+            df_chunk = pd.DataFrame(records).set_index("Datetime").sort_index()
+            df_chunk.index.name = "Datetime"
+            chunks.append(df_chunk)
 
-            df = pd.DataFrame(records).set_index("Datetime")
+            earliest = df_chunk.index.min()
-            df.index.name = "Datetime"
+            logger.info(
-            logger.debug(f"Fetched {len(df)} bars for {td_symbol} @ {interval}")
+                f"  {td_symbol} {interval}: chunk "
-            return df
+                f"{earliest.date()} → {df_chunk.index.max().date()} "
+                f"({len(df_chunk):,} bars)"
+            )
 
-        except DataIntegrityError:
+            # Stop when:
-            raise
+            #   1. The current chunk's start window already hit the requested
-        except Exception as e:
+            #      `start` boundary — nothing earlier was asked for.
-            logger.info(f"Twelve Data fetch failed for {symbol} ({e}), using synthetic data.")
+            #   2. The chunk is unusually thin (< 100 bars), suggesting we've
-            return self._generate_synthetic_data(symbol, timeframe, lookback_days)
+            #      reached the edge of the symbol's available history.
+            if cursor_start <= start or len(df_chunk) < 100:
+                break
 
-    def _generate_synthetic_data(
+            cursor_end = earliest - timedelta(seconds=1)
-        self, symbol: str, timeframe: Timeframe, lookback_days: int
-    ) -> pd.DataFrame:
-        """
-        Generate realistic synthetic Forex OHLCV using GBM with mean-reversion.
-        Used as a fallback when live data is unavailable.
-        """
-        pair_params = {
-            "EURUSD=X": {"price": 1.1350, "vol": 0.07, "drift": -0.001},
-            "GBPUSD=X": {"price": 1.3200, "vol": 0.08, "drift":  0.002},
-            "USDJPY=X": {"price": 144.50, "vol": 0.09, "drift":  0.003},
-            "USDCHF=X": {"price": 0.8850, "vol": 0.07, "drift": -0.001},
-            "AUDUSD=X": {"price": 0.6550, "vol": 0.10, "drift": -0.002},
-            "USDCAD=X": {"price": 1.3650, "vol": 0.07, "drift":  0.001},
-            "NZDUSD=X": {"price": 0.6100, "vol": 0.10, "drift": -0.001},
-            "EURGBP=X": {"price": 0.8590, "vol": 0.06, "drift":  0.000},
-            "EURJPY=X": {"price": 163.90, "vol": 0.09, "drift":  0.002},
-            "GBPJPY=X": {"price": 190.70, "vol": 0.11, "drift":  0.003},
-            "AUDJPY=X": {"price": 94.60,  "vol": 0.11, "drift":  0.001},
-            "EURCHF=X": {"price": 0.9380, "vol": 0.05, "drift":  0.000},
-            "CADJPY=X": {"price": 105.80, "vol": 0.09, "drift":  0.001},
-        }
 
-        params = pair_params.get(symbol, {"price": 1.0000, "vol": 0.08, "drift": 0.000})
+        if not chunks:
-        base_price = params["price"]
+            raise DataUnavailableError(f"No historical data returned for {td_symbol}")
-        annual_vol = params["vol"]
-        annual_drift = params["drift"]
 
-        tf_minutes = {
+        df = pd.concat(chunks).sort_index()
-            Timeframe.M1: 1, Timeframe.M5: 5, Timeframe.M15: 15,
+        df = df[~df.index.duplicated(keep="last")]
-            Timeframe.M30: 30, Timeframe.H1: 60, Timeframe.H4: 240,
+        df = df[(df.index >= start) & (df.index <= end)]
-            Timeframe.D1: 1440, Timeframe.W1: 10080,
-        }
-        bar_minutes = tf_minutes.get(timeframe, 60)
-        bars_per_year = (365.25 * 24 * 60) / bar_minutes
-        dt = 1.0 / bars_per_year
-
-        n_bars = min(int(lookback_days * 24 * 60 / bar_minutes), 2000)
-        n_bars = max(n_bars, 300)
-
-        seed_base = int(hashlib.md5(symbol.encode()).hexdigest()[:8], 16)
-        time_component = int(time.time()) // 3600
-        np.random.seed((seed_base + time_component) % (2**31))
-
-        bar_vol = annual_vol * np.sqrt(dt)
-        bar_drift = annual_drift * dt
-
-        regime_len = np.random.randint(20, 80)
-        trend_strength = np.random.normal(0, 0.3)
-
-        price = base_price
-        prices = [price]
-
-        for i in range(1, n_bars):
-            if i % regime_len == 0:
-                regime_len = np.random.randint(20, 80)
-                trend_strength = np.random.normal(0, 0.3)
-
-            mr_force = -0.001 * (price - base_price) / base_price
-            ret = bar_drift + trend_strength * dt + mr_force + bar_vol * np.random.randn()
-
-            if np.random.random() < 0.02:
-                ret *= np.random.uniform(2.0, 4.0)
-
-            price *= (1 + ret)
-            prices.append(price)
-
-        prices = np.array(prices)
-        opens = np.roll(prices, 1)
-        opens[0] = prices[0]
-
-        typical_range = bar_vol * base_price * 1.5
-        highs = np.maximum(prices, opens) + np.abs(np.random.normal(0, typical_range * 0.6, n_bars))
-        lows  = np.minimum(prices, opens) - np.abs(np.random.normal(0, typical_range * 0.6, n_bars))
-        highs = np.maximum(highs, np.maximum(opens, prices))
-        lows  = np.minimum(lows,  np.minimum(opens, prices))
-
-        volume = np.random.lognormal(mean=10, sigma=0.5, size=n_bars).astype(int)
-
-        freq_map = {
-            Timeframe.M1: "1min",  Timeframe.M5: "5min",
-            Timeframe.M15: "15min", Timeframe.M30: "30min",
-            Timeframe.H1: "1h",    Timeframe.H4: "4h",
-            Timeframe.D1: "D",     Timeframe.W1: "W",
-        }
-        freq = freq_map.get(timeframe, "1h")
-        end_time = pd.Timestamp.now(tz="UTC").floor(f"{bar_minutes}min")
-        index = pd.date_range(end=end_time, periods=n_bars, freq=freq, tz="UTC")
-
-        df = pd.DataFrame({
-            "Open": opens, "High": highs, "Low": lows,
-            "Close": prices, "Volume": volume,
-        }, index=index)
-        df.index.name = "Datetime"
-
-        logger.info(
-            f"Generated {n_bars} synthetic {timeframe.value} bars for "
-            f"{SYMBOL_DISPLAY.get(symbol, symbol)} @ {prices[-1]:.5f}"
-        )
         return df
 
     def _validate_and_clean(
@@ -522,7 +618,7 @@ class ForexDataProvider:
         if std_ret > 0:
             outlier_mask = (returns - returns.mean()).abs() > 10 * std_ret
             if outlier_mask.any():
-                logger.warning(f"{symbol}: Removing {outlier_mask.sum()} extreme outlier bars")
+                logger.debug(f"{symbol}: Removing {outlier_mask.sum()} extreme outlier bars")
                 df = df[~outlier_mask]
 
         min_required = max(

engine.py

@@ -499,7 +499,8 @@ class TechnicalEngine:
 
         # Compare to historical ATR
         if len(atr) >= 50:
-            atr_percentile = (atr.iloc[-1] > atr.iloc[-50:]).mean() * 100
+            # Exclude current bar so ATR isn't compared against itself
+            atr_percentile = (atr.iloc[-1] > atr.iloc[-50:-1]).mean() * 100
         else:
             atr_percentile = 50.0
 
@@ -844,7 +845,8 @@ class TechnicalEngine:
         - Confidence scaling
         """
         total_score = 0.0
-        total_weight = 0.0
+        score_weight = 0.0       # denominator for score: weight × confidence
+        weight_sum = 0.0         # denominator for confidence average: weight only
         total_confidence = 0.0
 
         regime_weights = {
@@ -863,16 +865,18 @@ class TechnicalEngine:
             regime_mult = active_regime_weights.get(name, 1.0)
             weight *= regime_mult
 
+            conf_factor = ind.confidence / 100.0
             signal_score = ind.signal.value  # -2 to +2
-            weighted_score = signal_score * weight * (ind.confidence / 100.0)
 
-            total_score += weighted_score
+            total_score += signal_score * weight * conf_factor
-            total_weight += weight
+            score_weight += weight * conf_factor
+            weight_sum += weight
             total_confidence += ind.confidence * weight
 
-        if total_weight > 0:
+        if score_weight > 0:
-            normalized_score = (total_score / total_weight) * 50  # scale to -100..+100
+            # Confidence-weighted average of signal_score (∈ [-2,+2]); ×50 → [-100,+100]
-            avg_confidence = total_confidence / total_weight
+            normalized_score = (total_score / score_weight) * 50
+            avg_confidence = total_confidence / weight_sum if weight_sum > 0 else 0
         else:
             normalized_score = 0
             avg_confidence = 0

geometry_grid.py

@@ -0,0 +1,361 @@
+#!/usr/bin/env python3
+"""
+Phase 2: re-simulate the existing trade pool under different SL/TP geometries
+without re-running the engine.
+
+The engine already gave us, per trade: entry_time, entry_price, direction,
+and the original SL distance (= 1.5 × ATR by default). That distance lets
+us recover ATR and re-derive new SL/TP for any (sl_mult, rr) combo. We
+then walk forward through the saved OHLCV bars, applying first-touch
+simulation with worst-case (SL first if both touch same bar).
+
+This is faster than re-running the engine because the signal output (when
+to trade, in which direction, at what price) is independent of SL/TP
+geometry — only the exits change.
+
+Discipline: we report TRAIN and TEST stats for every cell, time-split at
+70/30 by entry_time. A cell wins only if it's positive on the held-out
+TEST slice.
+
+Usage:
+    python geometry_grid.py
+    python geometry_grid.py --apply-phase1-filters
+    python geometry_grid.py --pairs USDJPY  # single-pair grid
+"""
+
+from __future__ import annotations
+
+import argparse
+import sys
+from pathlib import Path
+from typing import Dict, Tuple
+
+import numpy as np
+import pandas as pd
+
+# Reuse history loader so we don't re-implement
+from backtest import load_history
+from config import Timeframe
+
+# trades.csv stores TF as Twelve-Data interval string ("15min", "1h", "1d", …)
+_TFSTR_TO_ENUM = {
+    "1min":  Timeframe.M1,  "5min":  Timeframe.M5,
+    "15min": Timeframe.M15, "30min": Timeframe.M30,
+    "1h":    Timeframe.H1,  "4h":    Timeframe.H4,
+    "1d":    Timeframe.D1,  "1wk":   Timeframe.W1,
+}
+
+# Filters carried over from Phase 1
+PHASE1_BAD_HOURS = {10, 17, 19, 21}
+PHASE1_BAD_PAIRS = {"NZD/USD", "AUD/USD", "GBP/USD"}
+PHASE1_BAD_DOWS  = {5, 6}  # Sat / Sun
+
+
+def simulate_one(
+    direction: str,
+    entry: float,
+    new_sl: float,
+    new_tp: float,
+    forward_bars: pd.DataFrame,
+    max_bars: int,
+    spread_pips: float,
+    pip_value: float,
+) -> Tuple[float, str]:
+    """First-touch simulation. Returns (r_multiple, outcome)."""
+    sl_distance = abs(entry - new_sl)
+    if sl_distance == 0:
+        return 0.0, "INVALID"
+    risk_pips = sl_distance / pip_value
+    spread_r = spread_pips / risk_pips if risk_pips > 0 else 0.0
+    rr_target = abs(new_tp - entry) / sl_distance
+
+    bars_examined = 0
+    for ts, bar in forward_bars.iterrows():
+        bars_examined += 1
+        if bars_examined > max_bars:
+            break
+        bar_high = bar["High"]
+        bar_low  = bar["Low"]
+        if direction == "BUY":
+            sl_hit = bar_low <= new_sl
+            tp_hit = bar_high >= new_tp
+        else:
+            sl_hit = bar_high >= new_sl
+            tp_hit = bar_low <= new_tp
+        if sl_hit:
+            return -1.0 - spread_r, "LOSS_SL"
+        if tp_hit:
+            return rr_target - spread_r, "WIN_TP1"
+
+    if bars_examined == 0:
+        return 0.0, "INVALID"
+    last = forward_bars.iloc[bars_examined - 1]
+    pnl = (last["Close"] - entry) if direction == "BUY" else (entry - last["Close"])
+    return (pnl / sl_distance) - spread_r, "TIMEOUT"
+
+
+def pip_value_for(symbol: str) -> float:
+    """Return pip value: 0.01 for JPY pairs, 0.0001 elsewhere."""
+    return 0.01 if "JPY" in symbol else 0.0001
+
+
+def grid_run(
+    df: pd.DataFrame,
+    history: Dict[Tuple[str, str], pd.DataFrame],
+    sl_mults: list,
+    rr_ratios: list,
+    max_hold: int = 100,
+    spread_pips: float = 1.0,
+) -> pd.DataFrame:
+    """For each (sl_mult, rr) cell, re-simulate every trade and report stats."""
+    # Pre-index history by (symbol, tf_str) for fast lookup
+    rows = []
+    n_total = len(df) * len(sl_mults) * len(rr_ratios)
+    print(f"   evaluating {n_total:,} simulations "
+          f"({len(df):,} trades × {len(sl_mults)} SL × {len(rr_ratios)} RR)…")
+
+    for sl_mult in sl_mults:
+        for rr in rr_ratios:
+            r_multiples = []
+            outcomes = []
+            entry_times = []
+            pairs = []
+            tfs = []
+            for trade in df.itertuples(index=False):
+                # Recover ATR from original SL distance (engine used 1.5×ATR)
+                atr = abs(trade.entry - trade.sl) / 1.5
+                new_sl_dist = sl_mult * atr
+                if trade.direction == "BUY":
+                    new_sl = trade.entry - new_sl_dist
+                    new_tp = trade.entry + new_sl_dist * rr
+                else:
+                    new_sl = trade.entry + new_sl_dist
+                    new_tp = trade.entry - new_sl_dist * rr
+
+                hist_df = history.get((trade.symbol, trade.tf))
+                if hist_df is None:
+                    continue
+                ts = pd.Timestamp(trade.entry_time)
+                # Forward bars (strictly after entry)
+                forward = hist_df[hist_df.index > ts].head(max_hold)
+                if forward.empty:
+                    continue
+
+                pip = pip_value_for(trade.symbol)
+                r, outcome = simulate_one(
+                    trade.direction, trade.entry,
+                    new_sl, new_tp, forward,
+                    max_bars=max_hold, spread_pips=spread_pips,
+                    pip_value=pip,
+                )
+                r_multiples.append(r)
+                outcomes.append(outcome)
+                entry_times.append(ts)
+                pairs.append(trade.pair)
+                tfs.append(trade.tf)
+
+            arr = np.array(r_multiples) if r_multiples else np.array([0.0])
+            wins = (arr > 0).sum()
+            losses = (arr <= 0).sum()
+            gross_win = arr[arr > 0].sum()
+            gross_loss = -arr[arr <= 0].sum()
+            pf = gross_win / gross_loss if gross_loss > 0 else float("inf")
+            cum = np.cumsum(arr)
+            running_peak = np.maximum.accumulate(cum)
+            max_dd = (cum - running_peak).min()
+
+            rows.append({
+                "sl_mult":  sl_mult,
+                "rr":       rr,
+                "n":        len(arr),
+                "wins":     int(wins),
+                "wr":       wins / len(arr) * 100 if len(arr) else 0,
+                "avg_r":    arr.mean() if len(arr) else 0,
+                "total_r":  arr.sum(),
+                "pf":       pf,
+                "max_dd":   max_dd,
+                # Carry the per-trade results so we can split train/test downstream
+                "_arr":         arr,
+                "_entry_times": entry_times,
+                "_pairs":       pairs,
+                "_outcomes":    outcomes,
+            })
+    return pd.DataFrame(rows)
+
+
+def split_summary(cell: pd.Series, train_frac: float = 0.7) -> dict:
+    """Split a cell's per-trade results into train/test by entry_time."""
+    arr = cell["_arr"]
+    times = cell["_entry_times"]
+    if len(arr) == 0:
+        return {"train_total": 0, "train_wr": 0, "test_total": 0, "test_wr": 0}
+    order = np.argsort(times)
+    cutoff = int(len(arr) * train_frac)
+    train_idx = order[:cutoff]
+    test_idx  = order[cutoff:]
+    train_arr = arr[train_idx]
+    test_arr  = arr[test_idx]
+    return {
+        "train_n":     len(train_arr),
+        "train_wr":    (train_arr > 0).mean() * 100 if len(train_arr) else 0,
+        "train_avg":   train_arr.mean() if len(train_arr) else 0,
+        "train_total": train_arr.sum(),
+        "test_n":      len(test_arr),
+        "test_wr":     (test_arr > 0).mean() * 100 if len(test_arr) else 0,
+        "test_avg":    test_arr.mean() if len(test_arr) else 0,
+        "test_total":  test_arr.sum(),
+    }
+
+
+def main() -> int:
+    ap = argparse.ArgumentParser()
+    ap.add_argument("--trades", default="backtest_results_all/trades.csv")
+    ap.add_argument("--out", default="backtest_results_all/geometry_grid.txt")
+    ap.add_argument("--max-hold", type=int, default=100)
+    ap.add_argument("--spread-pips", type=float, default=1.0)
+    ap.add_argument("--apply-phase1-filters", action="store_true",
+                    help="Drop weekend trades, worst 4 hours, worst 3 pairs before grid")
+    ap.add_argument("--sl-mults", nargs="+", type=float,
+                    default=[1.0, 1.5, 2.0, 2.5, 3.0])
+    ap.add_argument("--rr-ratios", nargs="+", type=float,
+                    default=[1.0, 1.5, 2.0, 2.5])
+    args = ap.parse_args()
+
+    p = Path(args.trades)
+    if not p.exists():
+        print(f"❌ {p} not found")
+        return 1
+
+    df = pd.read_csv(p)
+    df["entry_time"] = pd.to_datetime(df["entry_time"])
+    df["hour"] = df["entry_time"].dt.hour
+    df["dow"] = df["entry_time"].dt.dayofweek
+
+    print(f"📊 Loaded {len(df):,} trades from {p}")
+
+    if args.apply_phase1_filters:
+        before = len(df)
+        df = df[~df["hour"].isin(PHASE1_BAD_HOURS)]
+        df = df[~df["pair"].isin(PHASE1_BAD_PAIRS)]
+        df = df[~df["dow"].isin(PHASE1_BAD_DOWS)]
+        df = df.reset_index(drop=True)
+        print(f"   Phase 1 filters applied: {before:,} → {len(df):,} trades "
+              f"({len(df)/before*100:.0f}% retained)")
+
+    # Load history for forward simulation
+    print(f"📁 Loading history…")
+    pairs = sorted(df["symbol"].unique().tolist())
+    tfs   = sorted({_TFSTR_TO_ENUM[t] for t in df["tf"].unique() if t in _TFSTR_TO_ENUM},
+                   key=lambda x: x.value)
+    history_raw = load_history(pairs, tfs)
+    # Rekey by (symbol, tf_string) to match trades.csv
+    history = {}
+    for (sym, tf_enum), hdf in history_raw.items():
+        history[(sym, tf_enum.value)] = hdf
+
+    print(f"\n🧮 Grid search:")
+    print(f"   SL multipliers (× ATR): {args.sl_mults}")
+    print(f"   RR ratios:               {args.rr_ratios}\n")
+
+    grid = grid_run(
+        df, history, args.sl_mults, args.rr_ratios,
+        max_hold=args.max_hold, spread_pips=args.spread_pips,
+    )
+
+    # ── Heatmap (full-data total R) ─────────────────────────────────
+    pivot_total = grid.pivot(index="sl_mult", columns="rr", values="total_r").round(1)
+    pivot_wr    = grid.pivot(index="sl_mult", columns="rr", values="wr").round(1)
+    pivot_avg   = grid.pivot(index="sl_mult", columns="rr", values="avg_r").round(3)
+    pivot_pf    = grid.pivot(index="sl_mult", columns="rr", values="pf").round(2)
+
+    out_lines = []
+    def emit(line: str = ""):
+        print(line)
+        out_lines.append(line)
+
+    emit("=" * 80)
+    emit(f"GEOMETRY GRID — {len(df):,} trades")
+    emit(f"  filters applied: {args.apply_phase1_filters}")
+    emit(f"  spread cost:     {args.spread_pips} pip")
+    emit(f"  max hold:        {args.max_hold} bars")
+    emit("=" * 80)
+    emit()
+    emit("─── Total R (rows = SL multiplier × ATR; cols = RR ratio) ───")
+    emit(pivot_total.to_string())
+    emit()
+    emit("─── Win rate % ───")
+    emit(pivot_wr.to_string())
+    emit()
+    emit("─── Avg R per trade ───")
+    emit(pivot_avg.to_string())
+    emit()
+    emit("─── Profit factor ───")
+    emit(pivot_pf.to_string())
+    emit()
+
+    # ── Train/Test discipline on top cells ──────────────────────────
+    grid_sorted = grid.sort_values("total_r", ascending=False)
+    emit("=" * 80)
+    emit("TOP 5 CELLS (by full-period total R) — with TRAIN / TEST split discipline")
+    emit("=" * 80)
+    for _, cell in grid_sorted.head(5).iterrows():
+        s = split_summary(cell)
+        emit(
+            f"  SL={cell['sl_mult']:.1f}×ATR  RR=1:{cell['rr']:.1f}  "
+            f"|  full: {cell['n']} trades  WR {cell['wr']:.1f}%  "
+            f"avgR {cell['avg_r']:+.3f}  totalR {cell['total_r']:+.1f}  "
+            f"PF {cell['pf']:.2f}"
+        )
+        emit(
+            f"      train ({s['train_n']}):  "
+            f"WR {s['train_wr']:.1f}%  avgR {s['train_avg']:+.3f}  totalR {s['train_total']:+.1f}"
+        )
+        emit(
+            f"      test  ({s['test_n']}):   "
+            f"WR {s['test_wr']:.1f}%  avgR {s['test_avg']:+.3f}  totalR {s['test_total']:+.1f}"
+        )
+        emit()
+
+    # ── Bottom 3 for contrast ───────────────────────────────────────
+    emit("─── BOTTOM 3 cells (worst total R) ───")
+    for _, cell in grid_sorted.tail(3).iterrows():
+        emit(
+            f"  SL={cell['sl_mult']:.1f}×ATR  RR=1:{cell['rr']:.1f}  "
+            f"totalR {cell['total_r']:+.1f}  WR {cell['wr']:.1f}%"
+        )
+    emit()
+
+    # ── Verdict ─────────────────────────────────────────────────────
+    emit("=" * 80)
+    emit("VERDICT")
+    emit("=" * 80)
+    best = grid_sorted.iloc[0]
+    s = split_summary(best)
+    emit(
+        f"  Best cell: SL={best['sl_mult']:.1f}×ATR, RR=1:{best['rr']:.1f}  "
+        f"(full totalR {best['total_r']:+.1f})"
+    )
+    emit(
+        f"  Test slice: {s['test_n']} trades, WR {s['test_wr']:.1f}%, "
+        f"totalR {s['test_total']:+.1f}, avgR {s['test_avg']:+.3f}"
+    )
+    if s["test_total"] > 0:
+        emit("  ✅ Best geometry is profitable on held-out test set.")
+        emit("     → Phase 3A: hardcode in _generate_trade_setup, re-run end-to-end.")
+    elif s["test_avg"] > -0.05:
+        emit("  ⚠️  Best geometry is near-break-even on test set.")
+        emit("     → Phase 3A worth attempting; combine with Phase 1 filters for margin.")
+    else:
+        emit("  ❌ No geometry is profitable on held-out test set.")
+        emit("     → Phase 3B: M15 confluence is structurally not edge-bearing.")
+        emit("        Consider H4 primary, mean-reversion-only, or indicator-level meta-model.")
+
+    Path(args.out).parent.mkdir(parents=True, exist_ok=True)
+    Path(args.out).write_text("\n".join(out_lines))
+    emit()
+    emit(f"📁 Saved: {args.out}")
+    return 0
+
+
+if __name__ == "__main__":
+    sys.exit(main())

main.py

@@ -380,11 +380,16 @@ Examples:
 
 
 def symbol_from_name(name: str) -> str:
-    """Convert pair name like 'EURUSD' to Yahoo symbol 'EURUSD=X'."""
+    """
-    name = name.upper().replace("/", "").replace("_", "")
+    Convert pair name like 'EURUSD' or 'XAUUSD' to internal symbol 'EURUSD=X'.
-    if not name.endswith("=X"):
+    Returns None for clearly invalid inputs (e.g. timeframe strings like '15M').
-        name += "=X"
+    """
-    return name
+    clean = name.upper().replace("/", "").replace("_", "").replace("=X", "")
+    if not clean.isalpha() or len(clean) < 5:
+        print(f"  Warning: '{name}' is not a valid pair symbol — skipping.")
+        print(f"  Hint: timeframes go in --timeframe (e.g. --timeframe M15), not --pairs.")
+        return None
+    return clean + "=X"
 
 
 def run_analysis(args) -> List[SignalReport]:
@@ -398,22 +403,27 @@ def run_analysis(args) -> List[SignalReport]:
     }
     config.primary_timeframe = tf_map[args.timeframe]
 
-    # Configure confirmation timeframes based on primary
+    # Configure confirmation timeframes based on primary.
+    # H1 uses M15 + D1 for high-conviction confluence (3-of-3 threshold).
     tf_confirmations = {
         "M5": [Timeframe.M1, Timeframe.M15],
-        "M15": [Timeframe.M5, Timeframe.H1],
+        "M15": [Timeframe.H1, Timeframe.D1],
-        "M30": [Timeframe.M15, Timeframe.H1],
+        "M30": [Timeframe.H1, Timeframe.D1],
-        "H1": [Timeframe.M15, Timeframe.H4],
+        "H1": [Timeframe.M15, Timeframe.D1],
         "H4": [Timeframe.H1, Timeframe.D1],
         "D1": [Timeframe.H4, Timeframe.W1],
     }
     config.confirmation_timeframes = tf_confirmations.get(
-        args.timeframe, [Timeframe.M15, Timeframe.H4]
+        args.timeframe, [Timeframe.M15, Timeframe.D1]
     )
 
     # Configure pairs
     if args.pairs:
-        config.watch_pairs = [symbol_from_name(p) for p in args.pairs]
+        symbols = [symbol_from_name(p) for p in args.pairs]
+        config.watch_pairs = [s for s in symbols if s is not None]
+        if not config.watch_pairs:
+            print("  Error: No valid pair symbols provided. Exiting.")
+            sys.exit(1)
     elif args.all:
         config.watch_pairs = MAJOR_PAIRS + CROSS_PAIRS
     else:

meta_model.py

@@ -0,0 +1,368 @@
+#!/usr/bin/env python3
+"""
+Meta-model: learn an empirical "will this trade win?" probability from
+backtest output, replacing the engine's heuristic A+/A/B grades with a
+calibrated win-probability score.
+
+Inputs:
+    backtest_results/trades.csv  (or --trades <path>)
+
+Outputs (under backtest_results/meta/):
+    feature_importances.txt
+    classification_report.txt
+    calibration.png              — predicted prob vs actual win rate
+    decile_returns.png           — avg R per predicted-prob decile
+    equity_compare.png           — engine grades vs meta-top-N
+    summary.txt                  — headline numbers + recommended threshold
+
+Train/test split is **time-based** (first 70% train, last 30% test) — random
+splits leak future regime information into training and overstate test
+performance on financial time series.
+
+Usage:
+    python meta_model.py
+    python meta_model.py --trades backtest_results_h1/trades.csv --out meta_h1
+    python meta_model.py --top-decile 0.2   # take top 20% predictions
+"""
+
+from __future__ import annotations
+
+import argparse
+import logging
+import sys
+from pathlib import Path
+from typing import Tuple
+
+import numpy as np
+import pandas as pd
+
+logger = logging.getLogger(__name__)
+
+
+def engineer_features(df: pd.DataFrame) -> Tuple[pd.DataFrame, pd.Series]:
+    """
+    Build features available at trade entry time. Strict no-leakage:
+    nothing from exit_time, exit_price, outcome, bars_held, r_multiple.
+    """
+    df = df.copy()
+    df["entry_time"] = pd.to_datetime(df["entry_time"])
+
+    # Time-of-day / day-of-week
+    df["hour"] = df["entry_time"].dt.hour
+    df["dow"]  = df["entry_time"].dt.dayofweek
+    df["is_buy"] = (df["direction"] == "BUY").astype(int)
+
+    # MTF flag
+    df["mtf"] = df["mtf_aligned"].astype(int)
+
+    # Score / confidence — keep raw and signed/abs versions; model picks
+    df["abs_score"] = df["score"].abs()
+    df["score_sign"] = np.sign(df["score"])
+
+    # Risk geometry
+    df["rr"] = df["reward_pips"] / df["risk_pips"].replace(0, np.nan)
+    df["rr"] = df["rr"].fillna(2.0)
+
+    # Regime one-hot
+    regime_dummies = pd.get_dummies(df["regime"], prefix="regime").astype(int)
+    df = pd.concat([df, regime_dummies], axis=1)
+
+    # Pair one-hot (model can learn pair-specific behavior)
+    pair_dummies = pd.get_dummies(df["pair"], prefix="pair").astype(int)
+    df = pd.concat([df, pair_dummies], axis=1)
+
+    # Grade one-hot — included so the meta-model can re-weight rather
+    # than blindly disregard the engine's view
+    grade_dummies = pd.get_dummies(df["grade"], prefix="grade").astype(int)
+    df = pd.concat([df, grade_dummies], axis=1)
+
+    feature_cols = (
+        ["score", "abs_score", "score_sign", "confidence", "rr",
+         "is_buy", "mtf", "hour", "dow", "risk_pips", "reward_pips"]
+        + list(regime_dummies.columns)
+        + list(pair_dummies.columns)
+        + list(grade_dummies.columns)
+    )
+
+    X = df[feature_cols].astype(float)
+    y = (df["r_multiple"] > 0).astype(int)
+    return X, y
+
+
+def time_based_split(
+    X: pd.DataFrame, y: pd.Series, df: pd.DataFrame, train_frac: float = 0.7,
+):
+    """First 70% by entry_time = train, last 30% = test. Standard for time series."""
+    order = df["entry_time"].argsort().values
+    cutoff = int(len(order) * train_frac)
+    train_idx = order[:cutoff]
+    test_idx  = order[cutoff:]
+    return (
+        X.iloc[train_idx], X.iloc[test_idx],
+        y.iloc[train_idx], y.iloc[test_idx],
+        df.iloc[train_idx], df.iloc[test_idx],
+    )
+
+
+def evaluate_decile_strategy(
+    df_test: pd.DataFrame, p_win: np.ndarray, top_frac: float = 0.2,
+) -> dict:
+    """
+    Take only the top X% by predicted win probability. Compare to taking
+    every trade in the test set (the 'engine baseline').
+    """
+    n = len(df_test)
+    cutoff_n = max(1, int(n * top_frac))
+
+    df_eval = df_test.copy()
+    df_eval["p_win"] = p_win
+    df_eval = df_eval.sort_values("p_win", ascending=False)
+
+    top = df_eval.head(cutoff_n)
+    base = df_eval
+
+    return {
+        "top_frac":    top_frac,
+        "top_n":       len(top),
+        "top_wr":      (top["r_multiple"] > 0).mean() * 100,
+        "top_avg_r":   top["r_multiple"].mean(),
+        "top_total_r": top["r_multiple"].sum(),
+        "base_n":      len(base),
+        "base_wr":     (base["r_multiple"] > 0).mean() * 100,
+        "base_avg_r":  base["r_multiple"].mean(),
+        "base_total_r": base["r_multiple"].sum(),
+    }
+
+
+def main() -> int:
+    parser = argparse.ArgumentParser(description="Meta-model on backtest trades.")
+    parser.add_argument("--trades", default="backtest_results/trades.csv",
+                        help="Path to trades.csv (default backtest_results/trades.csv)")
+    parser.add_argument("--out", default="backtest_results/meta",
+                        help="Output directory")
+    parser.add_argument("--train-frac", type=float, default=0.7)
+    parser.add_argument("--top-decile", type=float, default=0.2,
+                        help="Top fraction of predictions to 'trade' (default 0.2)")
+    parser.add_argument("--min-trades", type=int, default=300,
+                        help="Refuse to train below this many trades")
+    args = parser.parse_args()
+
+    logging.basicConfig(level=logging.INFO, format="%(message)s")
+
+    try:
+        from sklearn.linear_model import LogisticRegression
+        from sklearn.ensemble import GradientBoostingClassifier
+        from sklearn.preprocessing import StandardScaler
+        from sklearn.pipeline import Pipeline
+        from sklearn.metrics import (
+            classification_report, roc_auc_score, brier_score_loss,
+        )
+    except ImportError:
+        print("❌ scikit-learn not installed.  pip install scikit-learn")
+        return 1
+
+    trades_path = Path(args.trades)
+    if not trades_path.exists():
+        print(f"❌ {trades_path} not found. Run backtest.py first.")
+        return 1
+
+    out_dir = Path(args.out)
+    out_dir.mkdir(parents=True, exist_ok=True)
+
+    df = pd.read_csv(trades_path)
+    df["entry_time"] = pd.to_datetime(df["entry_time"])
+    if len(df) < args.min_trades:
+        print(f"❌ Only {len(df)} trades — need ≥{args.min_trades} for training.")
+        print("   Seed more pairs / longer history, or lower --min-trades.")
+        return 1
+
+    print(f"📊 Loaded {len(df):,} trades from {trades_path}")
+    print(f"   Pairs: {sorted(df['pair'].unique())}")
+    print(f"   Date range: {df['entry_time'].min()} → {df['entry_time'].max()}")
+    print(f"   Baseline win rate: {(df['r_multiple'] > 0).mean()*100:.1f}%")
+    print(f"   Baseline avg R:   {df['r_multiple'].mean():+.3f}")
+    print()
+
+    X, y = engineer_features(df)
+    print(f"🔧 Features: {list(X.columns)}")
+    print(f"   Shape: {X.shape}, target balance: {y.mean()*100:.1f}% wins\n")
+
+    X_tr, X_te, y_tr, y_te, df_tr, df_te = time_based_split(X, y, df, args.train_frac)
+    print(f"📚 Train: {len(X_tr):,} trades  ({df_tr['entry_time'].min().date()} → {df_tr['entry_time'].max().date()})")
+    print(f"   Test:  {len(X_te):,} trades  ({df_te['entry_time'].min().date()} → {df_te['entry_time'].max().date()})\n")
+
+    # ── Model 1: Logistic regression (interpretable baseline) ─────────
+    lr = Pipeline([
+        ("scale", StandardScaler()),
+        ("lr", LogisticRegression(max_iter=2000, C=1.0)),
+    ])
+    lr.fit(X_tr, y_tr)
+    p_lr = lr.predict_proba(X_te)[:, 1]
+
+    # ── Model 2: Gradient boosting (non-linear, captures interactions) ─
+    gb = GradientBoostingClassifier(
+        n_estimators=200, max_depth=3, learning_rate=0.05,
+        subsample=0.8, random_state=42,
+    )
+    gb.fit(X_tr, y_tr)
+    p_gb = gb.predict_proba(X_te)[:, 1]
+
+    # ── Metrics ───────────────────────────────────────────────────────
+    print("══ Logistic Regression ══")
+    print(f"  AUC:    {roc_auc_score(y_te, p_lr):.3f}")
+    print(f"  Brier:  {brier_score_loss(y_te, p_lr):.3f}  (lower=better, 0.25=random for 50% prior)")
+    print()
+    print("══ Gradient Boosting ══")
+    print(f"  AUC:    {roc_auc_score(y_te, p_gb):.3f}")
+    print(f"  Brier:  {brier_score_loss(y_te, p_gb):.3f}")
+    print()
+
+    # Pick the better-AUC model for downstream evaluation
+    p_best = p_gb if roc_auc_score(y_te, p_gb) >= roc_auc_score(y_te, p_lr) else p_lr
+    best_name = "GradientBoosting" if p_best is p_gb else "LogisticRegression"
+    print(f"➡️  Selecting {best_name} for top-decile evaluation\n")
+
+    # ── Decile-strategy comparison ─────────────────────────────────────
+    res = evaluate_decile_strategy(df_te, p_best, args.top_decile)
+    print(f"══ {int(args.top_decile*100)}% top-pick strategy vs trade-everything baseline (test set) ══")
+    print(f"  Trade everything:  {res['base_n']:>5} trades  WR {res['base_wr']:5.1f}%  avgR {res['base_avg_r']:+.3f}  totalR {res['base_total_r']:+8.2f}")
+    print(f"  Top {int(args.top_decile*100)}% by p_win:    {res['top_n']:>5} trades  WR {res['top_wr']:5.1f}%  avgR {res['top_avg_r']:+.3f}  totalR {res['top_total_r']:+8.2f}")
+    print()
+
+    # ── Per-decile breakdown ──────────────────────────────────────────
+    df_eval = df_te.copy()
+    df_eval["p_win"] = p_best
+    df_eval["decile"] = pd.qcut(df_eval["p_win"], 10, labels=False, duplicates="drop")
+    decile_summary = df_eval.groupby("decile").agg(
+        n=("r_multiple", "count"),
+        avg_p_win=("p_win", "mean"),
+        actual_wr=("r_multiple", lambda s: (s > 0).mean()),
+        avg_r=("r_multiple", "mean"),
+        total_r=("r_multiple", "sum"),
+    ).round(3)
+    decile_summary.index = decile_summary.index.astype(int) + 1  # 1..10
+    print("══ Per-decile breakdown (1=lowest p_win, 10=highest) ══")
+    print(decile_summary.to_string())
+    print()
+
+    # ── Feature importances ────────────────────────────────────────────
+    print("══ Top features by importance (from gradient boosting) ══")
+    importances = pd.Series(gb.feature_importances_, index=X.columns)
+    top_features = importances.sort_values(ascending=False).head(15)
+    for name, imp in top_features.items():
+        print(f"  {name:30s}  {imp:.4f}")
+    print()
+
+    # ── Save full reports ──────────────────────────────────────────────
+    (out_dir / "classification_report.txt").write_text(
+        f"Logistic Regression\n{'-'*40}\n"
+        f"AUC:   {roc_auc_score(y_te, p_lr):.3f}\n"
+        f"Brier: {brier_score_loss(y_te, p_lr):.3f}\n\n"
+        f"{classification_report(y_te, (p_lr > 0.5).astype(int))}\n"
+        f"\nGradient Boosting\n{'-'*40}\n"
+        f"AUC:   {roc_auc_score(y_te, p_gb):.3f}\n"
+        f"Brier: {brier_score_loss(y_te, p_gb):.3f}\n\n"
+        f"{classification_report(y_te, (p_gb > 0.5).astype(int))}\n"
+    )
+    (out_dir / "feature_importances.txt").write_text(
+        importances.sort_values(ascending=False).to_string()
+    )
+    (out_dir / "decile_breakdown.csv").write_text(decile_summary.to_csv())
+    (out_dir / "summary.txt").write_text(
+        f"Meta-model on {len(df):,} trades  "
+        f"({df['entry_time'].min()} → {df['entry_time'].max()})\n\n"
+        f"Baseline (trade everything):\n"
+        f"  win rate: {(df['r_multiple'] > 0).mean()*100:.1f}%\n"
+        f"  avg R:    {df['r_multiple'].mean():+.3f}\n"
+        f"  total R:  {df['r_multiple'].sum():+.2f}\n\n"
+        f"Test set ({len(X_te):,} trades, last 30% by date):\n"
+        f"  Trade everything:  WR {res['base_wr']:.1f}%  avgR {res['base_avg_r']:+.3f}  totalR {res['base_total_r']:+.2f}\n"
+        f"  Top {int(args.top_decile*100)}% by p_win:  WR {res['top_wr']:.1f}%  avgR {res['top_avg_r']:+.3f}  totalR {res['top_total_r']:+.2f}\n\n"
+        f"Best model: {best_name}\n"
+        f"  AUC:   {roc_auc_score(y_te, p_best):.3f}  (0.5 = random, >0.55 = real signal)\n"
+        f"  Brier: {brier_score_loss(y_te, p_best):.3f}\n"
+    )
+
+    # ── Plots ─────────────────────────────────────────────────────────
+    import matplotlib
+    matplotlib.use("Agg")
+    import matplotlib.pyplot as plt
+
+    # Calibration curve
+    bins = pd.cut(p_best, bins=np.linspace(0, 1, 11))
+    cal = pd.DataFrame({"p_win": p_best, "y": y_te.values, "bin": bins})
+    grouped = cal.groupby("bin", observed=True).agg(
+        mean_pred=("p_win", "mean"),
+        actual=("y", "mean"),
+        n=("y", "count"),
+    )
+    fig, ax = plt.subplots(figsize=(6, 5))
+    ax.plot([0, 1], [0, 1], "--", color="gray", label="Perfect calibration")
+    ax.plot(grouped["mean_pred"], grouped["actual"], "o-", linewidth=2, markersize=8)
+    for _, row in grouped.iterrows():
+        ax.annotate(f"n={int(row['n'])}",
+                    (row["mean_pred"], row["actual"]),
+                    textcoords="offset points", xytext=(8, -3), fontsize=8)
+    ax.set_xlabel("Mean predicted win probability")
+    ax.set_ylabel("Actual win rate")
+    ax.set_title(f"Calibration ({best_name}, test set)")
+    ax.set_xlim(0, 1); ax.set_ylim(0, 1)
+    ax.legend(); ax.grid(alpha=0.3)
+    fig.tight_layout()
+    fig.savefig(out_dir / "calibration.png", dpi=120)
+    plt.close(fig)
+
+    # Decile returns
+    fig, ax = plt.subplots(figsize=(8, 5))
+    colors = ["#dc2626" if r < 0 else "#16a34a" for r in decile_summary["avg_r"]]
+    ax.bar(decile_summary.index.astype(str), decile_summary["avg_r"], color=colors)
+    ax.axhline(0, color="black", linewidth=0.5)
+    ax.set_xlabel("Predicted-probability decile (1=lowest, 10=highest)")
+    ax.set_ylabel("Average R per trade")
+    ax.set_title(f"Trade outcome by predicted-probability decile ({best_name})")
+    for i, (idx, row) in enumerate(decile_summary.iterrows()):
+        ax.text(i, row["avg_r"] + (0.02 if row["avg_r"] >= 0 else -0.05),
+                f"n={int(row['n'])}\nWR={row['actual_wr']*100:.0f}%",
+                ha="center", fontsize=8)
+    ax.grid(alpha=0.3, axis="y")
+    fig.tight_layout()
+    fig.savefig(out_dir / "decile_returns.png", dpi=120)
+    plt.close(fig)
+
+    # Equity comparison: trade-everything vs top-decile (test set, time-ordered)
+    df_compare = df_te.copy().sort_values("entry_time").reset_index(drop=True)
+    df_compare["p_win"] = p_best
+    threshold = np.quantile(p_best, 1 - args.top_decile)
+    df_compare["take_top"] = (df_compare["p_win"] >= threshold).astype(int)
+    df_compare["base_cum"] = df_compare["r_multiple"].cumsum()
+    df_compare["top_cum"] = (df_compare["r_multiple"] * df_compare["take_top"]).cumsum()
+
+    fig, ax = plt.subplots(figsize=(11, 5))
+    ax.plot(df_compare["entry_time"], df_compare["base_cum"],
+            label="Trade everything (engine baseline)", linewidth=1.5)
+    ax.plot(df_compare["entry_time"], df_compare["top_cum"],
+            label=f"Top {int(args.top_decile*100)}% by p_win (meta-model)",
+            linewidth=1.8)
+    ax.axhline(0, color="gray", linewidth=0.5)
+    ax.set_xlabel("Trade entry time")
+    ax.set_ylabel("Cumulative R")
+    ax.set_title(f"Cumulative R: engine baseline vs meta-model top-{int(args.top_decile*100)}%  (test set)")
+    ax.legend(); ax.grid(alpha=0.3)
+    fig.tight_layout()
+    fig.savefig(out_dir / "equity_compare.png", dpi=120)
+    plt.close(fig)
+
+    print(f"📁 Reports → {out_dir}/")
+    print(f"   • classification_report.txt")
+    print(f"   • feature_importances.txt")
+    print(f"   • decile_breakdown.csv")
+    print(f"   • summary.txt")
+    print(f"📈 Plots:")
+    print(f"   • calibration.png")
+    print(f"   • decile_returns.png")
+    print(f"   • equity_compare.png")
+    return 0
+
+
+if __name__ == "__main__":
+    sys.exit(main())

news_engine.py

@@ -1,6 +1,4 @@
 """
-Forex Signal Intelligence Platform - News & Sentiment Engine
-=============================================================
 Monitors economic calendar events and news sentiment to provide
 fundamental context for technical signals.
 
@@ -8,7 +6,9 @@ Key principle: High-impact news events override technical signals.
 A perfect technical setup can be destroyed by an unexpected NFP print.
 """
 
+import json
 import logging
+import os
 import xml.etree.ElementTree as ET
 from dataclasses import dataclass, field
 from datetime import datetime, timedelta, timezone
@@ -16,6 +16,23 @@ from typing import Dict, List, Optional, Tuple
 
 import requests
 
+# Load .env so GEMINI_API_KEY is available without exporting it manually.
+try:
+    from dotenv import load_dotenv
+    load_dotenv()
+except ImportError:
+    pass
+
+# New unified Google GenAI SDK (replaces google-generativeai).
+try:
+    from google import genai
+    from google.genai import types as genai_types
+    _GENAI_AVAILABLE = True
+except ImportError:
+    genai = None
+    genai_types = None
+    _GENAI_AVAILABLE = False
+
 from config import NewsConfig
 
 logger = logging.getLogger(__name__)
@@ -73,12 +90,12 @@ class FundamentalContext:
 
 class NewsEngine:
     """
-    Economic calendar and news sentiment analyzer.
+    Economic calendar and news sentiment analyzer using Google Gemini API.
 
     Provides fundamental context that modifies technical signal confidence.
     """
 
-    # Keywords that indicate directional sentiment
+    # Keywords that indicate directional sentiment (kept for fallback and keyword extraction)
     BULLISH_KEYWORDS = {
         "rate hike", "hawkish", "strong growth", "beat expectations",
         "above forecast", "surplus", "expansion", "employment gains",
@@ -114,6 +131,11 @@ class NewsEngine:
         "boe rate", "rba rate", "snb rate", "boc rate",
     }
 
+    # gemini-2.5-flash-lite has a much higher free-tier quota (1000 RPD vs 20
+    # for gemini-2.5-flash) and is more than capable for headline classification.
+    GEMINI_MODEL = "gemini-2.5-flash-lite"
+    BATCH_MAX_HEADLINES = 60  # cap per Gemini call to keep prompts small
+
     def __init__(self, config: NewsConfig):
         self.config = config
         self._session = requests.Session()
@@ -121,6 +143,25 @@ class NewsEngine:
             "User-Agent": "Mozilla/5.0 (compatible; ForexIntel/1.0)"
         })
 
+        # In-memory cache for Gemini-scored headlines so repeat fetches across
+        # /scan calls don't re-spend tokens on the same RSS items.
+        self._sentiment_cache: Dict[int, float] = {}
+
+        # Initialize Google Gemini via the new unified google-genai SDK.
+        api_key = os.environ.get("GEMINI_API_KEY")
+        self.gemini_client = None
+        if not _GENAI_AVAILABLE:
+            logger.warning("google-genai SDK not installed — falling back to keyword sentiment.")
+        elif not api_key:
+            logger.warning("GEMINI_API_KEY not set — falling back to keyword sentiment.")
+        else:
+            try:
+                self.gemini_client = genai.Client(api_key=api_key)
+                logger.info(f"Google Gemini ({self.GEMINI_MODEL}) configured.")
+            except Exception as e:
+                logger.error(f"Failed to configure Gemini: {e}")
+                self.gemini_client = None
+
     def get_fundamental_context(
         self, currencies: Optional[List[str]] = None
     ) -> FundamentalContext:
@@ -191,7 +232,6 @@ class NewsEngine:
         """
         Get event risk level for a specific pair.
         Returns (risk_level, description).
-        risk_level: "clear", "caution", "danger"
         """
         clean = pair_symbol.replace("=X", "")
         base = clean[:3]
@@ -206,7 +246,7 @@ class NewsEngine:
                 else:
                     return "caution", f"{event.currency} {event.event} in {mins:.0f}min"
 
-        # Check for recent high-impact releases (within post-event buffer)
+        # Check for recent high-impact releases
         for event in context.events_today:
             if event.is_high_impact and event.currency in (base, quote):
                 if not event.is_upcoming and event.minutes_until > -self.config.post_event_buffer_minutes:
@@ -220,7 +260,6 @@ class NewsEngine:
         """
         Get net sentiment for a pair based on news.
         Returns (score, description) where score is -1 to +1.
-        Positive = bullish for the pair, negative = bearish.
         """
         clean = pair_symbol.replace("=X", "")
         base = clean[:3]
@@ -248,11 +287,6 @@ class NewsEngine:
     # ── Private Methods ────────────────────────────────────────────
 
     def _fetch_economic_calendar(self, currencies: List[str]) -> List[EconomicEvent]:
-        """
-        Fetch economic calendar events from a free API.
-        Returns an empty list (with a log warning) if unavailable —
-        we never generate fake events since that causes constant false alarms.
-        """
         try:
             events = self._try_fetch_calendar_api()
             if events:
@@ -260,14 +294,10 @@ class NewsEngine:
         except Exception as e:
             logger.debug(f"Calendar API fetch exception: {e}")
 
-        logger.info(
+        logger.info("Economic calendar data unavailable. Risk checks skipped.")
-            "Economic calendar data unavailable. "
-            "Event risk checks will be skipped for this cycle."
-        )
         return []
 
     def _try_fetch_calendar_api(self) -> List[EconomicEvent]:
-        """Attempt to fetch from the Forex Factory free JSON calendar."""
         events = []
         url = "https://nfs.faireconomy.media/ff_calendar_thisweek.json"
         resp = self._session.get(url, timeout=10)
@@ -280,7 +310,6 @@ class NewsEngine:
         for item in data:
             try:
                 impact = impact_map.get(item.get("impact", ""), "low")
-
                 date_str = item.get("date", "")
                 if not date_str:
                     continue
@@ -301,61 +330,169 @@ class NewsEngine:
         return events
 
     def _fetch_news_sentiment(self) -> List[NewsSentiment]:
-        """Fetch and analyze news headlines for sentiment."""
+        # Stage 1: collect all candidate (title, pub_date, source) from RSS feeds
-        headlines = []
+        # without scoring yet.
-
+        candidates: List[Tuple[str, datetime, str]] = []
         for feed_url in self.config.rss_feeds:
             try:
                 resp = self._session.get(feed_url, timeout=10)
                 if resp.status_code != 200:
                     continue
-
                 root = ET.fromstring(resp.content)
-
                 for item in root.iter("item"):
                     title_el = item.find("title")
                     pub_date_el = item.find("pubDate")
-
                     if title_el is None:
                         continue
-
                     title = title_el.text or ""
                     pub_date = self._parse_rss_date(
                         pub_date_el.text if pub_date_el is not None else None
                     )
-
-                    # Only recent headlines (last 24h)
                     if pub_date and (datetime.now(timezone.utc) - pub_date).total_seconds() > 86400:
                         continue
-
+                    if not self._extract_currencies(title):
-                    sentiment = self._analyze_headline_sentiment(title)
+                        continue  # skip headlines that don't mention any tracked currency
-                    currencies = self._extract_currencies(title)
+                    candidates.append((title, pub_date or datetime.now(timezone.utc), feed_url))
-
-                    if currencies:
-                        headlines.append(NewsSentiment(
-                            headline=title,
-                            source=feed_url,
-                            timestamp=pub_date or datetime.now(timezone.utc),
-                            currencies_mentioned=currencies,
-                            sentiment_score=sentiment,
-                            keywords=self._extract_keywords(title),
-                        ))
-
             except Exception as e:
                 logger.debug(f"Failed to fetch RSS from {feed_url}: {e}")
 
-        return headlines
+        # Stage 2: batch-score all relevant titles in one Gemini call (or as
+        # few as possible). Cached titles are served from memory; only the
+        # uncached subset is sent to the API.
+        unique_titles = list({t for t, _, _ in candidates})
+        scores = self._score_headlines_batch(unique_titles)
+
+        # Stage 3: build NewsSentiment objects using the scored map.
+        return [
+            NewsSentiment(
+                headline=title,
+                source=source,
+                timestamp=ts,
+                currencies_mentioned=self._extract_currencies(title),
+                sentiment_score=scores.get(title, 0.0),
+                keywords=self._extract_keywords(title),
+            )
+            for title, ts, source in candidates
+        ]
+
+    def _score_headlines_batch(self, headlines: List[str]) -> Dict[str, float]:
+        """
+        Score a list of headlines with a single Gemini API call.
+        Cached titles are returned from memory; only the uncached subset is
+        sent to the model. Falls back to keyword scoring on API failure.
+        """
+        out: Dict[str, float] = {}
+        to_score: List[str] = []
+        for h in headlines:
+            key = hash(h)
+            if key in self._sentiment_cache:
+                out[h] = self._sentiment_cache[key]
+            else:
+                to_score.append(h)
+
+        if not to_score:
+            return out
+
+        if self.gemini_client is None:
+            for h in to_score:
+                s = self._fallback_keyword_sentiment(h)
+                self._sentiment_cache[hash(h)] = s
+                out[h] = s
+            return out
+
+        # Process in chunks to keep the prompt reasonable.
+        for i in range(0, len(to_score), self.BATCH_MAX_HEADLINES):
+            chunk = to_score[i : i + self.BATCH_MAX_HEADLINES]
+            chunk_scores = self._gemini_score_chunk(chunk)
+            for h in chunk:
+                s = chunk_scores.get(h)
+                if s is None:
+                    s = self._fallback_keyword_sentiment(h)
+                self._sentiment_cache[hash(h)] = s
+                out[h] = s
+        return out
+
+    def _gemini_score_chunk(self, headlines: List[str]) -> Dict[str, float]:
+        """Single Gemini call returning {headline: score} for the chunk."""
+        numbered = "\n".join(f'{i + 1}. "{h}"' for i, h in enumerate(headlines))
+        prompt = (
+            "You are an expert Forex fundamental analyst. Score each headline "
+            "from -1.0 (extremely bearish for its primary currency) to +1.0 "
+            "(extremely bullish). Use 0.0 if neutral or off-topic. Consider "
+            "context: surprises beating forecast = bullish; misses = bearish; "
+            'central bank "pause" or "patient" = dovish/bearish; "hawkish" or '
+            '"hike" = bullish.\n\n'
+            f"Headlines:\n{numbered}\n\n"
+            'Respond ONLY with a JSON array, one object per headline, in order:\n'
+            '[{"id": 1, "score": 0.5}, {"id": 2, "score": -0.3}, ...]'
+        )
+        try:
+            response = self.gemini_client.models.generate_content(
+                model=self.GEMINI_MODEL,
+                contents=prompt,
+                config=genai_types.GenerateContentConfig(
+                    response_mime_type="application/json",
+                    temperature=0.0,
+                ),
+            )
+            data = json.loads(response.text)
+            scores: Dict[str, float] = {}
+            for entry in data:
+                idx = int(entry.get("id", 0)) - 1
+                if 0 <= idx < len(headlines):
+                    raw = float(entry.get("score", 0.0))
+                    scores[headlines[idx]] = max(-1.0, min(1.0, raw))
+            return scores
+        except Exception as e:
+            logger.warning(f"Gemini batch scoring failed ({len(headlines)} headlines): {e}")
+            return {}
 
     def _analyze_headline_sentiment(self, headline: str) -> float:
         """
-        Simple keyword-based sentiment analysis.
+        Score a single headline -1.0 to +1.0 via Google Gemini.
-
+        Falls back to keyword sentiment if the API is unavailable or fails.
-        Returns -1.0 (very bearish) to +1.0 (very bullish).
+        Cached per-headline so repeat calls cost zero tokens.
-
-        Note: This is intentionally simple. NLP models like FinBERT
-        would be more accurate but require large dependencies. This catches
-        the most impactful sentiment signals from headlines.
         """
+        cache_key = hash(headline)
+        if cache_key in self._sentiment_cache:
+            return self._sentiment_cache[cache_key]
+
+        if self.gemini_client is None:
+            score = self._fallback_keyword_sentiment(headline)
+            self._sentiment_cache[cache_key] = score
+            return score
+
+        prompt = (
+            "You are an expert Forex fundamental analyst. "
+            "Analyze this financial news headline:\n"
+            f'"{headline}"\n\n'
+            "Determine the sentiment score on the primary currency mentioned.\n"
+            "-1.0 means extremely bearish (rate cuts, recession, weak data).\n"
+            " 1.0 means extremely bullish (rate hikes, strong growth, beats).\n"
+            " 0.0 means neutral or irrelevant.\n\n"
+            'Respond ONLY with valid JSON: {"score": 0.85}'
+        )
+
+        try:
+            response = self.gemini_client.models.generate_content(
+                model=self.GEMINI_MODEL,
+                contents=prompt,
+                config=genai_types.GenerateContentConfig(
+                    response_mime_type="application/json",
+                    temperature=0.0,
+                ),
+            )
+            data = json.loads(response.text)
+            score = max(-1.0, min(1.0, float(data.get("score", 0.0))))
+        except Exception as e:
+            logger.debug(f"Gemini API failed for '{headline[:60]}': {e}")
+            score = self._fallback_keyword_sentiment(headline)
+
+        self._sentiment_cache[cache_key] = score
+        return score
+
+    def _fallback_keyword_sentiment(self, headline: str) -> float:
+        """Original keyword-based fallback if Gemini API is unavailable or fails."""
         headline_lower = headline.lower()
         bullish_score = sum(1 for kw in self.BULLISH_KEYWORDS if kw in headline_lower)
         bearish_score = sum(1 for kw in self.BEARISH_KEYWORDS if kw in headline_lower)
@@ -367,7 +504,6 @@ class NewsEngine:
         return (bullish_score - bearish_score) / total
 
     def _extract_currencies(self, headline: str) -> List[str]:
-        """Extract mentioned currencies from a headline."""
         headline_lower = headline.lower()
         found = []
 
@@ -383,27 +519,23 @@ class NewsEngine:
         return list(set(found))
 
     def _extract_keywords(self, headline: str) -> List[str]:
-        """Extract relevant trading keywords from headline."""
         headline_lower = headline.lower()
         return [kw for kw in self.BULLISH_KEYWORDS | self.BEARISH_KEYWORDS if kw in headline_lower]
 
     @staticmethod
     def _parse_rss_date(date_str: Optional[str]) -> Optional[datetime]:
-        """Parse various RSS date formats into a timezone-aware datetime."""
         if not date_str:
             return None
 
         formats = [
-            "%a, %d %b %Y %H:%M:%S %z",   # RFC 822 with numeric tz offset
+            "%a, %d %b %Y %H:%M:%S %z",
-            "%a, %d %b %Y %H:%M:%S %Z",   # RFC 822 with tz abbreviation (GMT/UTC)
+            "%a, %d %b %Y %H:%M:%S %Z",
-            "%Y-%m-%dT%H:%M:%S%z",         # ISO 8601 with offset
+            "%Y-%m-%dT%H:%M:%S%z",
-            "%Y-%m-%d %H:%M:%S",           # ISO 8601 naive (assume UTC)
+            "%Y-%m-%d %H:%M:%S",
         ]
         for fmt in formats:
             try:
                 dt = datetime.strptime(date_str.strip(), fmt)
-                # If strptime already attached tzinfo (from %z), keep it.
-                # Otherwise the string had no timezone info — assume UTC.
                 if dt.tzinfo is not None:
                     return dt
                 return dt.replace(tzinfo=timezone.utc)
@@ -413,7 +545,6 @@ class NewsEngine:
 
     @staticmethod
     def _get_affected_pairs(currency: str) -> List[str]:
-        """Get all major pairs affected by a currency event."""
         pairs = {
             "USD": ["EURUSD=X", "GBPUSD=X", "USDJPY=X", "USDCHF=X", "AUDUSD=X", "USDCAD=X", "NZDUSD=X"],
             "EUR": ["EURUSD=X", "EURGBP=X", "EURJPY=X", "EURCHF=X"],
@@ -424,4 +555,4 @@ class NewsEngine:
             "CAD": ["USDCAD=X", "CADJPY=X"],
             "NZD": ["NZDUSD=X"],
         }
-        return pairs.get(currency, [])
+        return pairs.get(currency, [])

requirements.txt

@@ -0,0 +1,16 @@
+# Core data + math
+numpy>=1.24
+pandas>=2.0
+
+# HTTP
+requests>=2.28
+
+# News sentiment (Gemini LLM)
+google-genai>=1.0      # new unified Google GenAI SDK
+python-dotenv>=1.0     # loads GEMINI_API_KEY and TELEGRAM_BOT_TOKEN from .env
+
+# Telegram bot
+python-telegram-bot[job-queue]>=20.0
+
+# Backtest analysis (only needed for `python backtest.py --plot`)
+matplotlib>=3.7

seed_history.py

@@ -0,0 +1,133 @@
+#!/usr/bin/env python3
+"""
+One-shot historical OHLCV seeder for backtesting.
+
+Downloads bars from Twelve Data for the configured pairs/timeframes,
+paginated to stay within the 5000-bar per-call limit, and writes each
+(symbol, timeframe) to history/<PAIR>_<TF>.csv.
+
+Usage:
+    python seed_history.py --years 1
+    python seed_history.py --years 2 --pairs EURUSD GBPUSD USDJPY
+    python seed_history.py --tfs M15 H1 D1 --years 1 --force
+
+Cost (free tier, 800 credits/day):
+    1 year M15 + H1 + D1 × 7 majors  ≈ 56 credits
+    5 years same                     ≈ 231 credits
+"""
+
+import argparse
+import logging
+import sys
+from datetime import datetime, timedelta, timezone
+from pathlib import Path
+
+try:
+    from dotenv import load_dotenv
+    load_dotenv()
+except ImportError:
+    pass
+
+from config import MAJOR_PAIRS, PlatformConfig, Timeframe
+from data_provider import DataUnavailableError, ForexDataProvider
+
+DEFAULT_TFS = ["M15", "H1", "D1"]
+HISTORY_DIR = Path("history")
+
+_TF_MAP = {
+    "M1":  Timeframe.M1,  "M5":  Timeframe.M5,
+    "M15": Timeframe.M15, "M30": Timeframe.M30,
+    "H1":  Timeframe.H1,  "H4":  Timeframe.H4,
+    "D1":  Timeframe.D1,  "W1":  Timeframe.W1,
+}
+
+
+def csv_path(out_dir: Path, symbol: str, tf: Timeframe) -> Path:
+    return out_dir / f"{symbol.replace('=X', '')}_{tf.value}.csv"
+
+
+def main() -> int:
+    parser = argparse.ArgumentParser(
+        description="Seed historical OHLCV for backtesting (Twelve Data, free tier).",
+    )
+    parser.add_argument("--years", type=float, default=1.0,
+                        help="Years of history to download (default 1.0)")
+    parser.add_argument("--pairs", nargs="+", default=None,
+                        help="Pair symbols, e.g. EURUSD GBPUSD (default all 7 majors)")
+    parser.add_argument("--tfs", nargs="+", default=DEFAULT_TFS,
+                        help=f"Timeframes (default {' '.join(DEFAULT_TFS)})")
+    parser.add_argument("--out", default=str(HISTORY_DIR),
+                        help="Output directory (default history/)")
+    parser.add_argument("--force", action="store_true",
+                        help="Re-download even if a CSV already exists")
+    args = parser.parse_args()
+
+    logging.basicConfig(
+        level=logging.INFO,
+        format="%(asctime)s [%(levelname)s] %(message)s",
+        datefmt="%H:%M:%S",
+    )
+
+    out_dir = Path(args.out)
+    out_dir.mkdir(parents=True, exist_ok=True)
+
+    pairs = args.pairs if args.pairs else MAJOR_PAIRS
+    pairs = [p if p.endswith("=X") else f"{p}=X" for p in pairs]
+
+    try:
+        tfs = [_TF_MAP[t.upper()] for t in args.tfs]
+    except KeyError as e:
+        print(f"❌ Unknown timeframe: {e}. Valid: {list(_TF_MAP.keys())}")
+        return 2
+
+    end = datetime.now(timezone.utc)
+    start = end - timedelta(days=int(args.years * 365))
+
+    provider = ForexDataProvider(PlatformConfig())
+
+    total = len(pairs) * len(tfs)
+    print(f"📥 Seeding {total} pair × tf combinations from "
+          f"{start.date()} to {end.date()}  ({args.years} year(s))")
+    print(f"   Pairs: {[p.replace('=X', '') for p in pairs]}")
+    print(f"   TFs:   {[tf.value for tf in tfs]}")
+    print(f"   Output: {out_dir}/\n")
+
+    done = 0
+    failed: list = []
+    for symbol in pairs:
+        for tf in tfs:
+            done += 1
+            out_path = csv_path(out_dir, symbol, tf)
+
+            if out_path.exists() and not args.force:
+                print(f"[{done}/{total}] ⏭  {symbol} {tf.value} — exists, "
+                      f"skipping (--force to overwrite)")
+                continue
+
+            print(f"[{done}/{total}] 📥 {symbol} {tf.value}…", flush=True)
+            try:
+                df = provider.fetch_historical(symbol, tf, start, end)
+                df.to_csv(out_path)
+                print(f"   ✅ {len(df):,} bars  "
+                      f"({df.index.min().date()} → {df.index.max().date()})  "
+                      f"→ {out_path}")
+            except DataUnavailableError as e:
+                print(f"   ❌ {e}")
+                failed.append((symbol, tf.value, str(e)))
+            except Exception as e:
+                print(f"   ❌ unexpected: {e}")
+                failed.append((symbol, tf.value, str(e)))
+
+    print()
+    if failed:
+        print(f"⚠️  {len(failed)} fetch(es) failed:")
+        for sym, tfn, msg in failed:
+            print(f"     {sym} {tfn}: {msg}")
+        return 1
+
+    print("✅ Seeding complete. Run: python backtest.py --primary M15")
+    return 0
+
+
+if __name__ == "__main__":
+    sys.exit(main())

signal_aggregator.py

@@ -31,6 +31,12 @@ from news_engine import NewsEngine, FundamentalContext
 logger = logging.getLogger(__name__)
 
 
+# Per-symbol TF overrides. Currently empty — all forex pairs use the global
+# config.confirmation_timeframes. Add entries here only if a specific symbol
+# needs a different MTF set.
+_PER_SYMBOL_TFS: Dict[str, List[Timeframe]] = {}
+
+
 @dataclass
 class TradeSetup:
     """A complete trade setup recommendation."""
@@ -104,9 +110,9 @@ class SignalAggregator:
     fundamental analysis, and produces final graded signals.
     """
 
-    def __init__(self, config: PlatformConfig):
+    def __init__(self, config: PlatformConfig, data_provider: Optional[ForexDataProvider] = None):
         self.config = config
-        self.data_provider = ForexDataProvider(config)
+        self.data_provider = data_provider or ForexDataProvider(config)
         self.tech_engine = TechnicalEngine(config.technical)
         self.news_engine = NewsEngine(config.news)
 
@@ -118,11 +124,12 @@ class SignalAggregator:
         reports = []
         context = self.news_engine.get_fundamental_context()
 
-        # Batch-prefetch all pairs × all needed timeframes (3 API calls instead of 21)
+        # Batch-prefetch all pairs × all needed timeframes (3 API calls instead of 21).
-        all_timeframes = (
+        # Union of primary + per-symbol confirmation TFs across the watchlist.
-            [self.config.primary_timeframe] + self.config.confirmation_timeframes
+        tf_set = {self.config.primary_timeframe, *self.config.confirmation_timeframes}
-        )
+        for sym in self.config.watch_pairs:
-        self.data_provider.prefetch_batch(self.config.watch_pairs, all_timeframes)
+            tf_set.update(self._confirmation_tfs(sym))
+        self.data_provider.prefetch_batch(self.config.watch_pairs, list(tf_set))
 
         for symbol in self.config.watch_pairs:
             try:
@@ -179,7 +186,7 @@ class SignalAggregator:
             return report
 
         # ── 2. Confirmation Timeframe Analyses ─────────────────────
-        for tf in self.config.confirmation_timeframes:
+        for tf in self._confirmation_tfs(symbol):
             try:
                 cf_df = self.data_provider.fetch_ohlcv(symbol, tf)
                 cf_analysis = self.tech_engine.analyze(cf_df, symbol, tf.value)
@@ -202,7 +209,11 @@ class SignalAggregator:
         report.news_description = sent_desc
 
         # ── 5. Spread Estimate ─────────────────────────────────────
-        report.spread_estimate = self.data_provider.get_spread_estimate(symbol)
+        # Derive spread from already-fetched primary_df to avoid a separate M5 API call
+        if primary_df is not None and not primary_df.empty:
+            report.spread_estimate = float((primary_df["High"] - primary_df["Low"]).median())
+        else:
+            report.spread_estimate = self.data_provider.get_spread_estimate(symbol)
 
         # ── 6. Compute Final Signal ────────────────────────────────
         self._compute_final_signal(report)
@@ -216,6 +227,14 @@ class SignalAggregator:
 
         return report
 
+    def _confirmation_tfs(self, symbol: str) -> List[Timeframe]:
+        """Return confirmation TFs for a symbol, honouring per-symbol overrides."""
+        override = _PER_SYMBOL_TFS.get(symbol)
+        if override is None:
+            return self.config.confirmation_timeframes
+        primary = self.config.primary_timeframe
+        return [tf for tf in override if tf != primary]
+
     def _check_mtf_agreement(self, report: SignalReport) -> bool:
         """Check if multiple timeframes agree on direction."""
         signals = list(report.mtf_signals.values())
@@ -242,20 +261,21 @@ class SignalAggregator:
         confidence = report.primary_confidence
 
         # ── MTF adjustment ─────────────────────────────────────────
+        # Always blend primary with MTF average so the score is pulled toward
+        # consensus on disagreement (and barely moves on agreement, since the
+        # TFs already point the same way).
+        all_scores = list(report.mtf_scores.values())
+        if all_scores:
+            mtf_avg = sum(all_scores) / len(all_scores)
+            score = score * 0.6 + mtf_avg * 0.4
+
         if report.mtf_agreement:
-            # All timeframes agree — boost confidence and blend scores
             confidence = min(confidence + 15, 95)
-            all_scores = list(report.mtf_scores.values())
-            if all_scores:
-                mtf_avg = sum(all_scores) / len(all_scores)
-                score = score * 0.6 + mtf_avg * 0.4
         else:
-            # Timeframes disagree — reduce confidence
             confidence *= 0.7
             if len(report.mtf_scores) >= 2:
-                scores = list(report.mtf_scores.values())
+                has_bull = any(s > 10 for s in report.mtf_scores.values())
-                has_bull = any(s > 10 for s in scores)
+                has_bear = any(s < -10 for s in report.mtf_scores.values())
-                has_bear = any(s < -10 for s in scores)
                 if has_bull and has_bear:
                     report.warnings.append("Timeframes CONFLICT — mixed signals across timeframes")
                     confidence *= 0.6