Harwinder Sekhon
dde41fbeb8
Fix 1 — mixed-"trending" regime (ADX strong, EMAs unaligned) was hard-coded bullish in 5 sites: RSI reinterpretation used the uptrend playbook (RSI 38-55 -> STRONG_BUY conf 75), BUY-only confidence bonuses, BUY-only extension penalties, no counter-trend penalty. Now: extremes-only RSI (as transitioning), no directional bonuses, symmetric extension penalties. 6yr validation: H1 +67.2->+61.9R (mixed skew 16B/11S -> 4B/7S), H4 +31.6->+27.5R (the 4 bias BUYs), M15 +152.6->+154.7R, M5 +207.1->+206.1R with maxDD 28.7->24.3R. The ~8R aggregate give-back is bull-market subsidy from the bug. Fix 2 — live session filter now applies the EXACT replay-loop predicate (session window + blocked hours on the signal bar's CLOSE time, from the bar timestamp not wall clock). Replaces a wall-clock filter gated on a "XAU STRATEGY" marker that no code writes, which made live diverge from every validated book. Fix 3 — delete dead _check_xau_oversold_bounce and _generate_xau_trade_setup (unreachable; the latter hardcoded BUY and skipped the spread/RR gates). Backtester: parallel bar replay via forked workers (--workers, default auto = cores-2 capped 12). Per-bar evaluation is pure; the entry cooldown is applied in an ordered phase-2 pass. Proven byte-identical to sequential on the same window; ~300 -> 1030-1680 bars/s (6yr M5: ~26min -> 5-8min). --workers 1 = sequential path. Tests: 179 passed (new mixed-regime RSI regression test).
131 lines
8.8 KiB
Markdown
131 lines
8.8 KiB
Markdown
# ForexBot2
|
||
|
||
Multi-pair signal intelligence platform (XAU/USD + EUR/USD). Analyzes each pair across multiple timeframes using 9 technical indicators with regime-adaptive weights, news sentiment, and economic calendar data. Delivers signals via Telegram with full trade setups (entry, SL, TP).
|
||
|
||
## Architecture
|
||
|
||
```
|
||
Twelve Data REST API + WebSocket
|
||
|
|
||
data_provider.py (fetch, cache, validate OHLCV; rate-limited 55 calls/min)
|
||
|
|
||
engine.py (9 indicators per timeframe → composite score)
|
||
|
|
||
signal_aggregator.py (MTF agreement + news + final signal + trade setup)
|
||
|
|
||
pair_signals/ (per-pair config factories: xauusd.py, eurusd.py)
|
||
|
|
||
+-----+--------+
|
||
| |
|
||
telegram_bot.py main.py
|
||
(Telegram alerts, (CLI per-pair
|
||
auto-scanner, analysis)
|
||
WS monitor)
|
||
```
|
||
|
||
**Per-pair modularity:** each pair registers a `make_config(primary_tf, confirmation_tfs) -> PlatformConfig` factory in `pair_signals/`. `PlatformConfig.scan_pairs` is the single switch for which pairs the bot scans (single- or multi-pair). Market constants (pip value, price precision, backtest default spread) live in `PAIR_INFO`; strategy knobs live in per-pair dataclasses (`XauStrategyConfig`, `EurUsdStrategyConfig`). All XAU-specific gates in the aggregator are display-gated — nothing gold-calibrated touches the EUR path, and vice versa (enforced by pair-isolation tests + a pinned-window XAU regression CSV).
|
||
|
||
## Core Files
|
||
|
||
| File | Purpose |
|
||
|------|---------|
|
||
| `config.py` | All configuration dataclasses, `PAIR_INFO` registry, per-pair strategy configs |
|
||
| `data_provider.py` | Twelve Data SDK wrapper with in-memory cache and rate limiting |
|
||
| `engine.py` | `TechnicalEngine` — computes EMA, RSI, BB, ADX, ATR, MACD, ROC, Keltner, divergences, candlestick patterns, swing S/R |
|
||
| `signal_aggregator.py` | `SignalAggregator` — MTF analysis, news integration, diminishing penalty cap, trade setup generation |
|
||
| `news_engine.py` | Economic calendar (Forex Factory JSON) + RSS sentiment (keyword-based) |
|
||
| `telegram_bot.py` | Async Telegram bot with auto-scanner (M5/M15/H1/H4), WebSocket price monitor |
|
||
| `pair_signals/` | Per-pair producer registry (`PAIR_PRODUCERS`) + config factories |
|
||
| `main.py` | CLI entry point (`--pair XAUUSD\|EURUSD`) |
|
||
| `test_all.py` | Comprehensive QA suite (~176 tests, no live API needed) |
|
||
|
||
## Key Concepts
|
||
|
||
**Signal Pipeline:** OHLCV → indicators → regime classification → regime-weighted composite score → MTF agreement check → news/event adjustment → diminishing penalty cap → final signal (STRONG_BUY/BUY/NEUTRAL/SELL/STRONG_SELL) → quality grade (A+/A/B/C/D/F) → trade setup if grade C+.
|
||
|
||
**Regime-Adaptive Weights:** Market classified as `trending_up`, `trending_down`, `ranging`, or `transitioning` via ADX + EMA + ADX slope. Gold-specific weights per regime:
|
||
- **Trending:** MACD (2.0), EMA (2.0), Keltner (1.5), ROC (1.5), Divergence (1.0)
|
||
- **Ranging:** Divergence (2.5), Keltner (2.0), RSI (1.5), BB (1.5), MACD (1.5)
|
||
- **Transitioning:** Keltner (2.5), MACD (2.5), ROC (2.0), Divergence (2.0), EMA (1.5)
|
||
|
||
ROC is directionally redundant with MACD/Keltner/EMA (0.8–0.94 correlation) but its weight is load-bearing: composite thresholds are calibrated with it — removing it backtested 13R worse. Don't zero it without retuning thresholds.
|
||
|
||
**RSI (regime-aware, default):** In trends RSI buys pullback zones instead of fading strength (classic mode backtested worse for gold). In `transitioning` only extremes (≥70/≤30) signal.
|
||
|
||
**Divergence:** 40-bar lookback; price and indicator swings must align within ±2 bars to count as one divergence observation.
|
||
|
||
**Transitioning Regime:** ADX < 25 but ADX slope > 3.0 = trend emerging. Catches trend birth 5-10 bars before ADX crosses 25. Boosts quality grade for early entries.
|
||
|
||
**Diminishing Penalty Cap:** Penalties sorted by magnitude, applied with exponential decay (1st at 100%, 2nd at 50%, 3rd at 25%). Hard cap at 30% total. Replaces old additive stacking that could gut 30-50% confidence.
|
||
|
||
**Indicators (9 scored):** EMA Trend, RSI, Bollinger Bands, MACD, ADX, Keltner Channels (TTM Squeeze), ROC (Rate of Change), Divergence, Candlestick Patterns.
|
||
|
||
**Trade Setup:** SL = nearest swing low/high ± 1×ATR buffer (min 1.5×ATR). TP = nearest swing level if RR≥1.2, else calculated from ATR. Minimum RR 0.5:1.
|
||
|
||
## How to Run
|
||
|
||
```bash
|
||
# CLI
|
||
python main.py # XAU/USD H1 default
|
||
python main.py --pair EURUSD --timeframe M5 --detail
|
||
|
||
# Telegram bot (auto-scanner)
|
||
TELEGRAM_TOKEN=<token> TELEGRAM_CHAT_ID=<id> python telegram_bot.py
|
||
|
||
# Tests
|
||
python test_all.py
|
||
|
||
# Backtest (spread defaults per pair: XAU $0.30, EUR 0.0001 — see PAIR_INFO)
|
||
python backtest/gpu_backtest.py --pair XAUUSD --days 365
|
||
python backtest/gpu_backtest.py --pair EURUSD --primary-tf 5min --confirm-tfs 15min,1h --days 365
|
||
```
|
||
|
||
## Environment Variables
|
||
|
||
| Variable | Purpose | Default |
|
||
|----------|---------|---------|
|
||
| `TWELVE_DATA_API_KEY` | Market data API key | Hardcoded fallback in config |
|
||
| `TELEGRAM_TOKEN` | Telegram bot token | Hardcoded fallback |
|
||
| `TELEGRAM_CHAT_ID` | Owner chat ID (enables boot auto-scan; only this user can drive the bot) | Hardcoded fallback |
|
||
| `TELEGRAM_CHANNEL_ID` | Subscriber channel ID (`-100…`); signal + breach alerts mirror there; `""` disables | Hardcoded fallback |
|
||
|
||
## Dependencies
|
||
|
||
`numpy`, `pandas`, `requests`, `twelvedata`, `python-telegram-bot`, `websockets` — see `requirements.txt`.
|
||
|
||
## Data Provider
|
||
|
||
- **Source:** Twelve Data REST API via `twelvedata` SDK
|
||
- **Rate limit:** 55 credits/min (Grow plan), enforced by sliding-window throttle
|
||
- **Cache:** In-memory dict, TTL = candle duration of the timeframe
|
||
- **Validation:** Drops NaN rows, forward-fills gaps (limit 3), fixes OHLC inconsistencies, removes outliers (>10σ), strips future timestamps, deduplicates
|
||
- **WebSocket:** subscribes every pair in `scan_pairs`
|
||
|
||
## Telegram Bot
|
||
|
||
- **Auto-scan:** M5, M15, H1, H4 at candle boundaries + 10s buffer (all timeframes alert)
|
||
- **Manual scan:** Buttons for M15/H1/H4 individual scans + "Scan All TFs" multi-timeframe summary
|
||
- **WebSocket:** Streams live ticks for XAU/USD; triggers immediate M15 scan on key level breach (swing S/R, BB, KC, EMA200) with 5-min cooldown
|
||
- **Commands:** `/start`, `/scan`, `/autoscan`, `/status`, `/cache`, `/clearcache`, `/help`
|
||
- **Owner lock:** all commands/buttons respond only to `TELEGRAM_CHAT_ID`; others get a refusal message
|
||
- **Subscriber channel:** with `TELEGRAM_CHANNEL_ID` set, auto-scan + breach alerts also post to the channel (bot must be channel admin). Monetized via Telegram Stars subscription invite links — Telegram handles membership; the bot has no payment code. Channel feed keeps running even if the owner's DM auto-scan is off
|
||
|
||
## Backtesting
|
||
|
||
The `backtest/` directory contains `gpu_backtest.py` — a GPU-accelerated backtester (CuPy with NumPy fallback) that replays historical OHLCV through the real SignalAggregator. Pre-computes all indicator series including ROC, Keltner, ADX slope for 100x+ speedup. Uses parquet caching for data persistence.
|
||
|
||
Fill realism: charges a round-trip spread per trade (`--spread`; defaults to the pair's `default_spread` in PAIR_INFO), fills gap-through SL/TP at the bar open, and resolves trades against the RAW price path (the parquet cache stores raw bars; validation/cleaning runs on load — signals see cleaned data, trade resolution sees real spikes). Entry is at the next bar's open after the signal bar. Weekend/session filters evaluate bar-close (decision) time to match the live bot's wall clock.
|
||
|
||
GPU note: CuPy needs the NVIDIA runtime pip wheels (`nvidia-cuda-runtime-cu12`, `nvidia-cuda-nvrtc-cu12`, `nvidia-curand-cu12`, `nvidia-nvjitlink-cu12`) or it silently falls back to NumPy — check the "Backend:" line in the run log. GPU accelerates indicator precompute and SL/TP scanning; the bar-replay loop is Python (~300 bars/s/core) but fans out across forked worker processes (`--workers`, default auto = cores−2 capped at 12, ~1700 bars/s at 10). Per-bar signal evaluation is pure, so parallel and sequential books are identical (the entry cooldown is applied in an ordered pass afterwards); `--workers 1` forces the sequential path.
|
||
|
||
## Code Conventions
|
||
|
||
- Python 3.12, type hints on public methods
|
||
- Dataclasses for all config and output structures
|
||
- `SignalStrength` enum (-2 to +2) is the universal signal currency
|
||
- Indicators stored as `Dict[str, IndicatorSignal]` in `TechnicalAnalysis`
|
||
- Key levels stored as `Dict[str, float]` — prefixed `swing_high_`, `swing_low_`, `ema_`, `bb_`, `kc_`, `atr`
|
||
- Private indicator data attached as `result._rsi_series`, `result._macd_histogram`, `result._adx_slope`, `result._kc_squeeze` (underscore prefix, skipped by composite scorer)
|
||
- Blocking signal computation runs in `ThreadPoolExecutor(max_workers=4)` to avoid blocking the async Telegram event loop
|
||
- Shared `ForexDataProvider` instance across all scans for cache reuse
|