harry/ForexBot2
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

GPU backtester, optimised weights, local CSV data pipeline

Browse Source 
## What changed

### backtest_gpu.py (new — ~1,300 lines)
Standalone GPU-accelerated backtester completely separate from backtest.py.
Computes all indicator series in one vectorised O(N) pass over the full price
history, then replays positions with a sequential state machine.

Key design:
- VectorIndicators: pandas ewm/rolling (C-optimised) for all 7 indicators
- CuPy GPU backend with automatic NumPy fallback (no hard dependency)
- Two-phase API: prepare_arrays() once, score_and_run() many times
- --optimize flag: 36-combo weight sweep (1 baseline + 7 ablations + 28
  multiplier runs) — calls prepare_arrays() once, replays all combos in seconds
- Full CLI: --tf, --local, --start/--end, --regime-filter, --regime-tf,
  --session-filter, --optimize, --journal, --journal-csv

O(N) vs O(N^2): backtest.py recomputes all indicators per bar (O(N^2));
backtest_gpu.py computes each indicator series once and reads bar T in O(1).
On 100,000 H1 bars: ~4 seconds vs >10 minutes.

### data_provider.py
Added load_local_csv() method — reads MetaTrader tab-delimited CSV exports
from market_data/ directory. Format: Time|Open|High|Low|Close|Volume|Spread
(spread column dropped on load, timestamps localised to UTC). Applies the same
_validate_and_clean() pipeline as the Twelve Data path. LOCAL_CSV_MAP maps
each Timeframe enum to its filename.

market_data/ files added (MetaTrader exports, UTC):
  USDJPY_M15.csv  100,000 bars  2022-05-04 to 2026-05-08
  USDJPY_H1.csv   100,000 bars  2010-04-23 to 2026-05-08
  USDJPY_H4.csv    25,855 bars  2010-04-22 to 2026-05-08
  USDJPY_D1.csv     5,020 bars  2010-04-22 to 2026-05-08

16 years of H1/H4/D1 vs ~6 weeks of H1 on Twelve Data free plan. Use --local
for all backtests; Twelve Data reserved for live signal feed.

### engine.py — optimised indicator weights
36-combo sweep on H4 2022-2026 found MACD and EMA trend are the dominant
signals; RSI and Bollinger are net drag on this trending pair.

Old → new weights:
  ema_trend:  1.2 → 3.6  (×3 — strongest directional signal)
  macd:       1.1 → 3.3  (×3 — ablating it is the single worst move: -42%)
  stochastic: 0.8 → 1.6  (×2 — clean timing, 40% WR at this level)
  adx:        1.0 → 0.5  (÷2 — partially redundant with EMA stack)
  ichimoku:   1.1 → 0.55 (÷2 — lags too much at H4, hurts in ranging)
  rsi:        0.3 → 0.0  (ablated — net drag across all test windows)
  bollinger:  0.9 → 0.0  (ablated — counter-trend noise on trending pair)

RSI and Bollinger are still computed and displayed in live output for
reference; weight=0 means they contribute nothing to the composite score.

Combined result vs baseline (H4 2022-2026, local data):
  Return:   +7.12% → +21.30%  (+14.2pp)
  Max DD:   -9.05% → -8.52%   (tighter — gains did not come at expense of risk)
  Sharpe:    0.23  →  0.47    (2x improvement)
  Sortino:   1.38  →  1.91
  Win Rate: 36.3%  → 38.1%

### backtest.py
Added 16-thread CPU parallelism for signal pre-computation via
ThreadPoolExecutor. _precompute_all_signals() dispatches each bar to a worker;
pandas/NumPy C extensions release the GIL enabling true parallel execution.
Results are collected and replayed in the main loop, replacing the per-bar
recompute. Also added --regime-tf CLI flag so regime filter timeframe can be
specified independently of the primary TF.

### signal_aggregator.py
EMA200 distance gate: confidence *= 0.55 when price is within 0.5% of EMA200
(chop zone) or beyond 1.5% (overextended). Sweet spot is 0.5-1.5% from EMA200
where trending momentum is cleanest. Strongest single filter found: lifted
cumulative return +13.88% to +35.50% in ablation window.

### ENGINE_LOGIC.md (new)
Full reference document covering: architecture, data pipeline, all indicator
logic, composite scoring formula, optimised weights with rationale, regime
classification, EMA200 gate, MTF alignment, grade system, backtesting results,
filter catalogue, and configuration reference.

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
This commit is contained in:
Harwinder
2026-05-07 22:12:43 -04:00
parent a687db9413
commit e30535f5e9
10 changed files with 232766 additions and 20 deletions
Download Patch File Download Diff File Expand all files Collapse all files
ENGINE_LOGIC.md
+450
View File
@@ -0,0 +1,450 @@
# ForexBot2 — Engine Logic & Design Reference
> USD/JPY signal intelligence platform.
> Last updated: 2026-05-08
---
## Table of Contents
1. [Architecture Overview](#1-architecture-overview)
2. [Data Pipeline](#2-data-pipeline)
3. [Indicator Engine (`engine.py`)](#3-indicator-engine)
4. [Signal Aggregator (`signal_aggregator.py`)](#4-signal-aggregator)
5. [Composite Scoring Formula](#5-composite-scoring-formula)
6. [Indicator Weights — Optimised](#6-indicator-weights--optimised)
7. [Market Regime Classification](#7-market-regime-classification)
8. [EMA200 Distance Gate](#8-ema200-distance-gate)
9. [Multi-Timeframe Alignment](#9-multi-timeframe-alignment)
10. [Signal Grade System](#10-signal-grade-system)
11. [GPU Backtester (`backtest_gpu.py`)](#11-gpu-backtester)
12. [Backtesting Results Summary](#12-backtesting-results-summary)
13. [Filter Catalogue](#13-filter-catalogue)
14. [Configuration Reference](#14-configuration-reference)
---
## 1. Architecture Overview
```
market_data/ (local CSV)
│
▼
ForexDataProvider ← also pulls live via Twelve Data SDK
│
▼
TechnicalEngine ← computes all indicators bar-by-bar
│
▼
SignalAggregator ← MTF blending, EMA200 gate, grading
│
├──▶ Live output (main.py)
└──▶ GPUBacktester ← vectorised O(N) replay over full history
```
**Key design principle:** No single indicator is trusted alone. The composite score is a confidence-weighted, regime-adjusted blend of up to 7 indicators across 3 timeframes. A signal only fires when direction, strength, and confidence all agree.
---
## 2. Data Pipeline
### Live mode
`ForexDataProvider.fetch_ohlcv()` calls the Twelve Data SDK with in-memory caching (5-min TTL). Fetches primary TF + two confirmation TFs + optional regime TF.
### Backtest mode
`ForexDataProvider.load_local_csv()` reads MetaTrader tab-delimited exports from `market_data/`:
| File | TF | Rows | Date Range |
|------|----|------|------------|
| USDJPY_M15.csv | M15 | 100,000 | 2022-05-04 → 2026-05-08 |
| USDJPY_H1.csv | H1 | 100,000 | 2010-04-23 → 2026-05-08 |
| USDJPY_H4.csv | H4 | 25,855 | 2010-04-22 → 2026-05-08 |
| USDJPY_D1.csv | D1 | 5,020 | 2010-04-22 → 2026-05-08 |
CSV format: tab-delimited, columns `Time | Open | High | Low | Close | Volume | Spread`. Spread column is dropped on load. Timestamps are UTC.
**Data validation** (applied to both sources):
- Drop NaN OHLC rows; forward-fill up to 3 bars
- Drop future timestamps (>5 min ahead of UTC now)
- Fix OHLC inconsistencies (High < Low etc.)
- Remove outlier bars where `|return| > 10 × rolling_std`
---
## 3. Indicator Engine
`engine.py → TechnicalEngine.analyze(df, symbol, timeframe)`
All indicators are computed on a single OHLCV DataFrame. Each returns an `IndicatorSignal` with:
- `value` — raw numeric reading
- `signal` — `SignalStrength` enum: STRONG_BUY(+2) / BUY(+1) / NEUTRAL(0) / SELL(-1) / STRONG_SELL(-2)
- `confidence` — 0–100%
- `weight` — contribution to composite score (0 = excluded)
### Indicators computed
| Name | Key | Wilder smoothing? | Notes |
|------|-----|:-----------------:|-------|
| EMA Alignment | `ema_trend` | No | 4-EMA stack: 9/21/50/200; normalised to -1..+1 |
| RSI | `rsi` | Yes (`alpha=1/14`) | Overbought/oversold + momentum zones |
| MACD | `macd` | No | 12/26/9; crossover = STRONG signal |
| Bollinger Bands | `bollinger` | No | 20-period; position within band + squeeze detect |
| Stochastic | `stochastic` | No | 14/3; K+D both OB/OS required for signal |
| ADX | `adx` | Yes | DI+/DI- direction + ADX strength gate (≥25) |
| ATR | `atr` | Yes | Context only — sets SL/TP distance; weight=0 |
| Ichimoku | `ichimoku` | No | Price vs cloud (+2/-2) + TK cross (+1/-1) + cloud colour (+1/-1) |
| Divergence | `divergence` | — | RSI/MACD price vs indicator swing divergence; weight=1.5 |
| Candlestick | `candlestick` | — | **Disabled** (ablation: net -3.4%) |
| Pivot Points | `pivot` | — | **Disabled** (ablation: net -9.5%) |
| Volume | `volume` | — | Context filter; weight=0 |
**Disabled indicators** are controlled by `TechnicalConfig.disabled_indicators` (default: `["pivot", "candlestick"]`).
---
## 4. Signal Aggregator
`signal_aggregator.py → SignalAggregator.analyze()`
Orchestrates three timeframes (primary + 2 confirmation), blends their scores, applies gates, and emits a final graded signal.
### Flow
```
H1 (primary) ──┐
M15 (confirm) ──┼──▶ per-TF score ──▶ MTF blend ──▶ EMA200 gate ──▶ grade ──▶ output
H4 (confirm) ──┘
```
### MTF alignment (no lookahead)
Lower-frequency bars are aligned to the primary TF index by forward-filling at **close times**:
```
secondary close time = secondary bar open + secondary TF minutes
primary close time = primary bar open + primary TF minutes
For each primary bar T: use the most recent secondary close ≤ T_close
```
This ensures no future information leaks into the signal.
---
## 5. Composite Scoring Formula
Mirrors `engine._compute_composite` exactly:
```
composite_score = clip(
(Σ signal_i × weight_i × regime_mult_i × (confidence_i / 100))
────────────────────────────────────────────────────────────────── × 50,
(Σ weight_i × regime_mult_i)
-100, +100
)
avg_confidence = Σ(confidence_i × weight_i) / Σ weight_i
```
- `signal_i` ∈ {-2, -1, 0, +1, +2}
- `weight_i` — per-indicator base weight (see §6)
- `regime_mult_i` — boosted/dampened by market regime (see §7)
- Indicators with `weight == 0` are excluded entirely
---
## 6. Indicator Weights — Optimised
Optimised 2026-05-08 via 36-combo sweep (ablation + ×0.5/1.5/2.0/3.0 per indicator) on H4 data 2022–2026. Combined result: **+21.30% return / -8.52% max DD / Sharpe 0.47** vs baseline +7.12% / -9.05% DD / Sharpe 0.23.
| Indicator | Weight | Change from original | Reason |
|-----------|-------:|---------------------|--------|
| `ema_trend` | **3.6** | was 1.2 (×3) | Strongest directional signal on trending pair |
| `macd` | **3.3** | was 1.1 (×3) | Most impactful single indicator; ablation = worst result |
| `stochastic` | **1.6** | was 0.8 (×2) | Clean timing signal; 40% WR at ×2 |
| `adx` | **0.5** | was 1.0 (÷2) | Partially redundant with EMA stack |
| `ichimoku` | **0.55** | was 1.1 (÷2) | Lags too much at H4; hurts on ranging periods |
| `rsi` | **0.0** | was 0.3 (ablated) | Net drag across all test windows |
| `bollinger` | **0.0** | was 0.9 (ablated) | Counter-trend noise on trending pair |
| `divergence` | 1.5 | unchanged | Strong reversal signal; not in sweep scope |
| `atr` | 0.0 | unchanged | Context only; sets SL/TP |
> **Note:** RSI and Bollinger are still **computed and displayed** in live output for reference, but their weight=0 means they contribute nothing to the composite score.
### Regime weight multipliers
Applied on top of base weights:
| Indicator | Trending mult | Ranging mult |
|-----------|:-------------:|:------------:|
| ema_trend | ×1.5 | ×0.6 |
| adx | ×1.3 | ×0.5 |
| macd | ×1.2 | ×1.0 |
| ichimoku | ×1.3 | ×0.2 |
| rsi | ×0.7 | ×1.4 |
| bollinger | ×0.6 | ×1.4 |
| stochastic | ×1.0 | ×1.3 |
In trending regimes: EMA trend, MACD, and Ichimoku are boosted further.
In ranging regimes: RSI and Bollinger would be boosted — but since both are weight=0, ranging regime currently relies entirely on stochastic + reduced EMA/ADX.
---
## 7. Market Regime Classification
`engine._classify_regime()` — runs after all indicators are computed:
```
if ADX ≥ 25:
if EMA signal is BUY or STRONG_BUY → trending_up
if EMA signal is SELL or STRONG_SELL → trending_down
else → trending
else:
→ ranging
```
Regime feeds directly into the weight multipliers applied in `_compute_composite`.
---
## 8. EMA200 Distance Gate
**Strongest single filter discovered** — implemented in `signal_aggregator.py`.
```
ema_dist = |close - EMA200| / EMA200 × 100 (percent)
if ema_dist < 0.5% or ema_dist > 1.5%:
confidence *= 0.55 ← damped
else: ← sweet spot
no change
```
**Rationale:** Price within 0.5% of EMA200 is in a high-chop zone where the 200 EMA acts as contested support/resistance. Price more than 1.5% away has often already run far and is prone to reversion. The 0.5–1.5% band is where trending momentum is cleanest.
**Impact measured:** Lifted cumulative return +13.88% → +35.50% in the 2024-trend ablation window.
---
## 9. Multi-Timeframe Alignment
Primary TF: **H1** (default). Confirmation TFs: **M15** and **H4**.
### MTF blending (when 2 of 3 TFs agree)
```
mtf_agree = (bull_count ≥ 2) OR (bear_count ≥ 2)
if mtf_agree:
score = 0.6 × primary_score + 0.4 × mean(all_scores)
confidence = min(confidence + 15, 95)
else:
confidence = confidence × 0.7 ← penalised for disagreement
```
### MTF agreement bonus in grade assignment
`mtf_agree=True` is required for grades A and A+.
---
## 10. Signal Grade System
| Grade | Conditions |
|-------|-----------|
| **A+** | `|score| ≥ 50` AND `conf ≥ 75%` AND `mtf_agree` |
| **A** | `|score| ≥ 35` AND `conf ≥ 65%` AND `mtf_agree` |
| **B** | `|score| ≥ 25` AND `conf ≥ 55%` |
| **C** | `|score| ≥ 15` AND `conf ≥ 45%` |
| **D** | `|score| ≥ 10` |
| **F** | below all thresholds |
Default minimum grade for trade entry: **B** (`BacktestConfig.min_trade_grade`).
### Signal integer thresholds (for trade direction)
| Signal | Condition |
|--------|-----------|
| STRONG_BUY (+2) | score ≥ +40 AND conf ≥ 65% |
| BUY (+1) | score ≥ +15 AND conf ≥ 50% |
| STRONG_SELL (-2) | score ≤ -40 AND conf ≥ 65% |
| SELL (-1) | score ≤ -15 AND conf ≥ 50% |
| NEUTRAL | otherwise |
---
## 11. GPU Backtester
`backtest_gpu.py` — completely separate from `backtest.py`.
### Why O(N) instead of O(N²)
`backtest.py` slices the DataFrame and recomputes all indicators fresh each bar → N × O(N) = **O(N²)**.
`backtest_gpu.py` computes every indicator series end-to-end once, then reads bar T's value in O(1) → **O(N) total**.
On 100,000 H1 bars: ~4 seconds vs >10 minutes.
### Two-phase architecture
```
Phase 1 — prepare_arrays(dfs)
VectorIndicators.compute_all() ← pandas ewm/rolling, C-optimised
Align MTF signals to primary TF ← forward-fill at close times
Compute regime ADX series ← if regime filter enabled
Returns reusable bundle dict
Phase 2 — score_and_run(bundle, weights, disabled)
compute_bar_scores() ← CuPy GPU or NumPy CPU
compute_final_scores() ← MTF blend + EMA200 gate (vectorised)
compute_grades() ← vectorised grade assignment
run_state_machine() ← sequential position management (O(N))
```
`prepare_arrays()` is called once; `score_and_run()` can be called many times with different weights — this is what the `--optimize` sweep uses.
### Optimization sweep (`--optimize`)
36 combinations tested per run:
- **1** baseline (default weights)
- **7** ablation runs (each indicator disabled one at a time)
- **28** weight multiplier runs (each indicator at ×0.5, ×1.5, ×2.0, ×3.0)
Indicator arrays computed once → all 36 replays run in seconds.
### Trade execution model
```
Entry: next bar Open ± (spread/2 + slippage) [default: 0.5 pip slippage]
SL: entry ± ATR × atr_sl_multiplier [default: 1.5×]
TP: entry ± SL_dist × rr_ratio [default: 2.0× = 1:2 R:R]
Risk: 1% of current equity per trade
```
If SL and TP both touch within the same bar: SL assumed to hit first (`sl_first_on_tie=True`).
### CLI flags
```
python backtest_gpu.py --tf H4 --local # local CSV, H4
python backtest_gpu.py --tf H1 --local --start 2022-01-01
python backtest_gpu.py --tf H4 --local --optimize # 36-combo weight sweep
python backtest_gpu.py --regime-filter --regime-tf D1
python backtest_gpu.py --session-filter # UTC 07-16 only
python backtest_gpu.py --journal --journal-csv out.csv
```
---
## 12. Backtesting Results Summary
### H4 — local CSV 2022–2026 (optimised weights)
| Metric | Value |
|--------|-------|
| Trades | 160 |
| Win Rate | 38.1% |
| Profit Factor | 1.19 |
| Total Return | +21.30% |
| Max Drawdown | -8.52% |
| Sharpe | 0.47 |
| Sortino | 1.91 |
### D1 — Twelve Data 2007–2026 (optimised weights)
| Metric | Value |
|--------|-------|
| Trades | 62 |
| Win Rate | 40.3% |
| Profit Factor | 1.27 |
| Total Return | +10.51% |
| Max Drawdown | -6.84% |
| Sortino | 7.74 |
### Optimisation sweep delta (H4 2022–2026)
| | Baseline | Optimised |
|--|----------|-----------|
| Return | +7.12% | +21.30% |
| Max DD | -9.05% | -8.52% |
| Sharpe | 0.23 | 0.47 |
| Win Rate | 36.3% | 38.1% |
---
## 13. Filter Catalogue
### Regime filter (`--regime-filter`)
Gates trade entry on ADX of a **higher** timeframe than the primary. Default: H4 ADX gate when trading H1. Redundant if regime TF == primary TF.
```
regime TF ADX ≥ min_adx (default 25) → entry allowed
```
Sensible defaults: H1→H4, H4→D1, D1→D1.
### Session filter (`--session-filter`)
Only trade during UTC 07:00–16:00 (London + NY overlap).
2026 diagnostic: 07-15 UTC = 47% WR / +127 pips; off-hours = 0% WR / -571 pips.
### Auto-session toggle (`auto_session_filter`)
Reactive version of session filter. Activates automatically when mean P&L over last N closed trades drops below threshold:
```
if mean(pnl_pct[-10:]) < -0.3%:
session_filter = ON ← restrict to liquid hours
else:
session_filter = OFF ← all hours allowed
```
### EMA200 distance gate
Always active. See §8.
### Signal grade gate (`min_trade_grade`)
Default: **B**. Only trades with grade ≥ B are entered. Raise to A for fewer, higher-quality trades.
### Min absolute score (`min_abs_score`)
Default: **0** (no extra gate). Set to e.g. 15 to skip weak directional scores.
---
## 14. Configuration Reference
All knobs live in `config.py → PlatformConfig`.
```python
# Indicator parameters
TechnicalConfig.ema_fast = 9
TechnicalConfig.ema_medium = 21
TechnicalConfig.ema_slow = 50
TechnicalConfig.ema_trend = 200
TechnicalConfig.rsi_period = 14
TechnicalConfig.macd_fast / slow / signal = 12/26/9
TechnicalConfig.bb_period = 20
TechnicalConfig.atr_period = 14
TechnicalConfig.stoch_k / d = 14/3
TechnicalConfig.adx_period = 14
TechnicalConfig.adx_strong_trend = 25.0
TechnicalConfig.ichimoku_tenkan / kijun / senkou_b = 9/26/52
TechnicalConfig.disabled_indicators = ["pivot", "candlestick"]
# Signal scoring
SignalConfig.min_confidence = 55.0
SignalConfig.atr_sl_multiplier = 1.5
SignalConfig.default_rr_ratio = 2.0
SignalConfig.default_risk_pct = 1.0
# Backtest
BacktestConfig.initial_balance = 10_000
BacktestConfig.risk_per_trade_pct = 1.0
BacktestConfig.min_trade_grade = "B"
BacktestConfig.regime_filter = False
BacktestConfig.session_filter = False
BacktestConfig.auto_session_filter = False
BacktestConfig.cache_dir = ".cache"
```