GPU backtester, optimised weights, local CSV data pipeline
## What changed ### backtest_gpu.py (new — ~1,300 lines) Standalone GPU-accelerated backtester completely separate from backtest.py. Computes all indicator series in one vectorised O(N) pass over the full price history, then replays positions with a sequential state machine. Key design: - VectorIndicators: pandas ewm/rolling (C-optimised) for all 7 indicators - CuPy GPU backend with automatic NumPy fallback (no hard dependency) - Two-phase API: prepare_arrays() once, score_and_run() many times - --optimize flag: 36-combo weight sweep (1 baseline + 7 ablations + 28 multiplier runs) — calls prepare_arrays() once, replays all combos in seconds - Full CLI: --tf, --local, --start/--end, --regime-filter, --regime-tf, --session-filter, --optimize, --journal, --journal-csv O(N) vs O(N^2): backtest.py recomputes all indicators per bar (O(N^2)); backtest_gpu.py computes each indicator series once and reads bar T in O(1). On 100,000 H1 bars: ~4 seconds vs >10 minutes. ### data_provider.py Added load_local_csv() method — reads MetaTrader tab-delimited CSV exports from market_data/ directory. Format: Time|Open|High|Low|Close|Volume|Spread (spread column dropped on load, timestamps localised to UTC). Applies the same _validate_and_clean() pipeline as the Twelve Data path. LOCAL_CSV_MAP maps each Timeframe enum to its filename. market_data/ files added (MetaTrader exports, UTC): USDJPY_M15.csv 100,000 bars 2022-05-04 to 2026-05-08 USDJPY_H1.csv 100,000 bars 2010-04-23 to 2026-05-08 USDJPY_H4.csv 25,855 bars 2010-04-22 to 2026-05-08 USDJPY_D1.csv 5,020 bars 2010-04-22 to 2026-05-08 16 years of H1/H4/D1 vs ~6 weeks of H1 on Twelve Data free plan. Use --local for all backtests; Twelve Data reserved for live signal feed. ### engine.py — optimised indicator weights 36-combo sweep on H4 2022-2026 found MACD and EMA trend are the dominant signals; RSI and Bollinger are net drag on this trending pair. Old → new weights: ema_trend: 1.2 → 3.6 (×3 — strongest directional signal) macd: 1.1 → 3.3 (×3 — ablating it is the single worst move: -42%) stochastic: 0.8 → 1.6 (×2 — clean timing, 40% WR at this level) adx: 1.0 → 0.5 (÷2 — partially redundant with EMA stack) ichimoku: 1.1 → 0.55 (÷2 — lags too much at H4, hurts in ranging) rsi: 0.3 → 0.0 (ablated — net drag across all test windows) bollinger: 0.9 → 0.0 (ablated — counter-trend noise on trending pair) RSI and Bollinger are still computed and displayed in live output for reference; weight=0 means they contribute nothing to the composite score. Combined result vs baseline (H4 2022-2026, local data): Return: +7.12% → +21.30% (+14.2pp) Max DD: -9.05% → -8.52% (tighter — gains did not come at expense of risk) Sharpe: 0.23 → 0.47 (2x improvement) Sortino: 1.38 → 1.91 Win Rate: 36.3% → 38.1% ### backtest.py Added 16-thread CPU parallelism for signal pre-computation via ThreadPoolExecutor. _precompute_all_signals() dispatches each bar to a worker; pandas/NumPy C extensions release the GIL enabling true parallel execution. Results are collected and replayed in the main loop, replacing the per-bar recompute. Also added --regime-tf CLI flag so regime filter timeframe can be specified independently of the primary TF. ### signal_aggregator.py EMA200 distance gate: confidence *= 0.55 when price is within 0.5% of EMA200 (chop zone) or beyond 1.5% (overextended). Sweet spot is 0.5-1.5% from EMA200 where trending momentum is cleanest. Strongest single filter found: lifted cumulative return +13.88% to +35.50% in ablation window. ### ENGINE_LOGIC.md (new) Full reference document covering: architecture, data pipeline, all indicator logic, composite scoring formula, optimised weights with rationale, regime classification, EMA200 gate, MTF alignment, grade system, backtesting results, filter catalogue, and configuration reference. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
This commit is contained in:
Harwinder
+450
@@ -0,0 +1,450 @@
|
|||||||
|
# ForexBot2 — Engine Logic & Design Reference
|
||||||
|
|
||||||
|
> USD/JPY signal intelligence platform.
|
||||||
|
> Last updated: 2026-05-08
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## Table of Contents
|
||||||
|
|
||||||
|
1. [Architecture Overview](#1-architecture-overview)
|
||||||
|
2. [Data Pipeline](#2-data-pipeline)
|
||||||
|
3. [Indicator Engine (`engine.py`)](#3-indicator-engine)
|
||||||
|
4. [Signal Aggregator (`signal_aggregator.py`)](#4-signal-aggregator)
|
||||||
|
5. [Composite Scoring Formula](#5-composite-scoring-formula)
|
||||||
|
6. [Indicator Weights — Optimised](#6-indicator-weights--optimised)
|
||||||
|
7. [Market Regime Classification](#7-market-regime-classification)
|
||||||
|
8. [EMA200 Distance Gate](#8-ema200-distance-gate)
|
||||||
|
9. [Multi-Timeframe Alignment](#9-multi-timeframe-alignment)
|
||||||
|
10. [Signal Grade System](#10-signal-grade-system)
|
||||||
|
11. [GPU Backtester (`backtest_gpu.py`)](#11-gpu-backtester)
|
||||||
|
12. [Backtesting Results Summary](#12-backtesting-results-summary)
|
||||||
|
13. [Filter Catalogue](#13-filter-catalogue)
|
||||||
|
14. [Configuration Reference](#14-configuration-reference)
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## 1. Architecture Overview
|
||||||
|
|
||||||
|
```
|
||||||
|
market_data/ (local CSV)
|
||||||
|
│
|
||||||
|
▼
|
||||||
|
ForexDataProvider ← also pulls live via Twelve Data SDK
|
||||||
|
│
|
||||||
|
▼
|
||||||
|
TechnicalEngine ← computes all indicators bar-by-bar
|
||||||
|
│
|
||||||
|
▼
|
||||||
|
SignalAggregator ← MTF blending, EMA200 gate, grading
|
||||||
|
│
|
||||||
|
├──▶ Live output (main.py)
|
||||||
|
└──▶ GPUBacktester ← vectorised O(N) replay over full history
|
||||||
|
```
|
||||||
|
|
||||||
|
**Key design principle:** No single indicator is trusted alone. The composite score is a confidence-weighted, regime-adjusted blend of up to 7 indicators across 3 timeframes. A signal only fires when direction, strength, and confidence all agree.
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## 2. Data Pipeline
|
||||||
|
|
||||||
|
### Live mode
|
||||||
|
`ForexDataProvider.fetch_ohlcv()` calls the Twelve Data SDK with in-memory caching (5-min TTL). Fetches primary TF + two confirmation TFs + optional regime TF.
|
||||||
|
|
||||||
|
### Backtest mode
|
||||||
|
`ForexDataProvider.load_local_csv()` reads MetaTrader tab-delimited exports from `market_data/`:
|
||||||
|
|
||||||
|
| File | TF | Rows | Date Range |
|
||||||
|
|------|----|------|------------|
|
||||||
|
| USDJPY_M15.csv | M15 | 100,000 | 2022-05-04 → 2026-05-08 |
|
||||||
|
| USDJPY_H1.csv | H1 | 100,000 | 2010-04-23 → 2026-05-08 |
|
||||||
|
| USDJPY_H4.csv | H4 | 25,855 | 2010-04-22 → 2026-05-08 |
|
||||||
|
| USDJPY_D1.csv | D1 | 5,020 | 2010-04-22 → 2026-05-08 |
|
||||||
|
|
||||||
|
CSV format: tab-delimited, columns `Time | Open | High | Low | Close | Volume | Spread`. Spread column is dropped on load. Timestamps are UTC.
|
||||||
|
|
||||||
|
**Data validation** (applied to both sources):
|
||||||
|
- Drop NaN OHLC rows; forward-fill up to 3 bars
|
||||||
|
- Drop future timestamps (>5 min ahead of UTC now)
|
||||||
|
- Fix OHLC inconsistencies (High < Low etc.)
|
||||||
|
- Remove outlier bars where `|return| > 10 × rolling_std`
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## 3. Indicator Engine
|
||||||
|
|
||||||
|
`engine.py → TechnicalEngine.analyze(df, symbol, timeframe)`
|
||||||
|
|
||||||
|
All indicators are computed on a single OHLCV DataFrame. Each returns an `IndicatorSignal` with:
|
||||||
|
- `value` — raw numeric reading
|
||||||
|
- `signal` — `SignalStrength` enum: STRONG_BUY(+2) / BUY(+1) / NEUTRAL(0) / SELL(-1) / STRONG_SELL(-2)
|
||||||
|
- `confidence` — 0–100%
|
||||||
|
- `weight` — contribution to composite score (0 = excluded)
|
||||||
|
|
||||||
|
### Indicators computed
|
||||||
|
|
||||||
|
| Name | Key | Wilder smoothing? | Notes |
|
||||||
|
|------|-----|:-----------------:|-------|
|
||||||
|
| EMA Alignment | `ema_trend` | No | 4-EMA stack: 9/21/50/200; normalised to -1..+1 |
|
||||||
|
| RSI | `rsi` | Yes (`alpha=1/14`) | Overbought/oversold + momentum zones |
|
||||||
|
| MACD | `macd` | No | 12/26/9; crossover = STRONG signal |
|
||||||
|
| Bollinger Bands | `bollinger` | No | 20-period; position within band + squeeze detect |
|
||||||
|
| Stochastic | `stochastic` | No | 14/3; K+D both OB/OS required for signal |
|
||||||
|
| ADX | `adx` | Yes | DI+/DI- direction + ADX strength gate (≥25) |
|
||||||
|
| ATR | `atr` | Yes | Context only — sets SL/TP distance; weight=0 |
|
||||||
|
| Ichimoku | `ichimoku` | No | Price vs cloud (+2/-2) + TK cross (+1/-1) + cloud colour (+1/-1) |
|
||||||
|
| Divergence | `divergence` | — | RSI/MACD price vs indicator swing divergence; weight=1.5 |
|
||||||
|
| Candlestick | `candlestick` | — | **Disabled** (ablation: net -3.4%) |
|
||||||
|
| Pivot Points | `pivot` | — | **Disabled** (ablation: net -9.5%) |
|
||||||
|
| Volume | `volume` | — | Context filter; weight=0 |
|
||||||
|
|
||||||
|
**Disabled indicators** are controlled by `TechnicalConfig.disabled_indicators` (default: `["pivot", "candlestick"]`).
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## 4. Signal Aggregator
|
||||||
|
|
||||||
|
`signal_aggregator.py → SignalAggregator.analyze()`
|
||||||
|
|
||||||
|
Orchestrates three timeframes (primary + 2 confirmation), blends their scores, applies gates, and emits a final graded signal.
|
||||||
|
|
||||||
|
### Flow
|
||||||
|
|
||||||
|
```
|
||||||
|
H1 (primary) ──┐
|
||||||
|
M15 (confirm) ──┼──▶ per-TF score ──▶ MTF blend ──▶ EMA200 gate ──▶ grade ──▶ output
|
||||||
|
H4 (confirm) ──┘
|
||||||
|
```
|
||||||
|
|
||||||
|
### MTF alignment (no lookahead)
|
||||||
|
|
||||||
|
Lower-frequency bars are aligned to the primary TF index by forward-filling at **close times**:
|
||||||
|
|
||||||
|
```
|
||||||
|
secondary close time = secondary bar open + secondary TF minutes
|
||||||
|
primary close time = primary bar open + primary TF minutes
|
||||||
|
|
||||||
|
For each primary bar T: use the most recent secondary close ≤ T_close
|
||||||
|
```
|
||||||
|
|
||||||
|
This ensures no future information leaks into the signal.
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## 5. Composite Scoring Formula
|
||||||
|
|
||||||
|
Mirrors `engine._compute_composite` exactly:
|
||||||
|
|
||||||
|
```
|
||||||
|
composite_score = clip(
|
||||||
|
(Σ signal_i × weight_i × regime_mult_i × (confidence_i / 100))
|
||||||
|
────────────────────────────────────────────────────────────────── × 50,
|
||||||
|
(Σ weight_i × regime_mult_i)
|
||||||
|
-100, +100
|
||||||
|
)
|
||||||
|
|
||||||
|
avg_confidence = Σ(confidence_i × weight_i) / Σ weight_i
|
||||||
|
```
|
||||||
|
|
||||||
|
- `signal_i` ∈ {-2, -1, 0, +1, +2}
|
||||||
|
- `weight_i` — per-indicator base weight (see §6)
|
||||||
|
- `regime_mult_i` — boosted/dampened by market regime (see §7)
|
||||||
|
- Indicators with `weight == 0` are excluded entirely
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## 6. Indicator Weights — Optimised
|
||||||
|
|
||||||
|
Optimised 2026-05-08 via 36-combo sweep (ablation + ×0.5/1.5/2.0/3.0 per indicator) on H4 data 2022–2026. Combined result: **+21.30% return / -8.52% max DD / Sharpe 0.47** vs baseline +7.12% / -9.05% DD / Sharpe 0.23.
|
||||||
|
|
||||||
|
| Indicator | Weight | Change from original | Reason |
|
||||||
|
|-----------|-------:|---------------------|--------|
|
||||||
|
| `ema_trend` | **3.6** | was 1.2 (×3) | Strongest directional signal on trending pair |
|
||||||
|
| `macd` | **3.3** | was 1.1 (×3) | Most impactful single indicator; ablation = worst result |
|
||||||
|
| `stochastic` | **1.6** | was 0.8 (×2) | Clean timing signal; 40% WR at ×2 |
|
||||||
|
| `adx` | **0.5** | was 1.0 (÷2) | Partially redundant with EMA stack |
|
||||||
|
| `ichimoku` | **0.55** | was 1.1 (÷2) | Lags too much at H4; hurts on ranging periods |
|
||||||
|
| `rsi` | **0.0** | was 0.3 (ablated) | Net drag across all test windows |
|
||||||
|
| `bollinger` | **0.0** | was 0.9 (ablated) | Counter-trend noise on trending pair |
|
||||||
|
| `divergence` | 1.5 | unchanged | Strong reversal signal; not in sweep scope |
|
||||||
|
| `atr` | 0.0 | unchanged | Context only; sets SL/TP |
|
||||||
|
|
||||||
|
> **Note:** RSI and Bollinger are still **computed and displayed** in live output for reference, but their weight=0 means they contribute nothing to the composite score.
|
||||||
|
|
||||||
|
### Regime weight multipliers
|
||||||
|
|
||||||
|
Applied on top of base weights:
|
||||||
|
|
||||||
|
| Indicator | Trending mult | Ranging mult |
|
||||||
|
|-----------|:-------------:|:------------:|
|
||||||
|
| ema_trend | ×1.5 | ×0.6 |
|
||||||
|
| adx | ×1.3 | ×0.5 |
|
||||||
|
| macd | ×1.2 | ×1.0 |
|
||||||
|
| ichimoku | ×1.3 | ×0.2 |
|
||||||
|
| rsi | ×0.7 | ×1.4 |
|
||||||
|
| bollinger | ×0.6 | ×1.4 |
|
||||||
|
| stochastic | ×1.0 | ×1.3 |
|
||||||
|
|
||||||
|
In trending regimes: EMA trend, MACD, and Ichimoku are boosted further.
|
||||||
|
In ranging regimes: RSI and Bollinger would be boosted — but since both are weight=0, ranging regime currently relies entirely on stochastic + reduced EMA/ADX.
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## 7. Market Regime Classification
|
||||||
|
|
||||||
|
`engine._classify_regime()` — runs after all indicators are computed:
|
||||||
|
|
||||||
|
```
|
||||||
|
if ADX ≥ 25:
|
||||||
|
if EMA signal is BUY or STRONG_BUY → trending_up
|
||||||
|
if EMA signal is SELL or STRONG_SELL → trending_down
|
||||||
|
else → trending
|
||||||
|
else:
|
||||||
|
→ ranging
|
||||||
|
```
|
||||||
|
|
||||||
|
Regime feeds directly into the weight multipliers applied in `_compute_composite`.
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## 8. EMA200 Distance Gate
|
||||||
|
|
||||||
|
**Strongest single filter discovered** — implemented in `signal_aggregator.py`.
|
||||||
|
|
||||||
|
```
|
||||||
|
ema_dist = |close - EMA200| / EMA200 × 100 (percent)
|
||||||
|
|
||||||
|
if ema_dist < 0.5% or ema_dist > 1.5%:
|
||||||
|
confidence *= 0.55 ← damped
|
||||||
|
else: ← sweet spot
|
||||||
|
no change
|
||||||
|
```
|
||||||
|
|
||||||
|
**Rationale:** Price within 0.5% of EMA200 is in a high-chop zone where the 200 EMA acts as contested support/resistance. Price more than 1.5% away has often already run far and is prone to reversion. The 0.5–1.5% band is where trending momentum is cleanest.
|
||||||
|
|
||||||
|
**Impact measured:** Lifted cumulative return +13.88% → +35.50% in the 2024-trend ablation window.
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## 9. Multi-Timeframe Alignment
|
||||||
|
|
||||||
|
Primary TF: **H1** (default). Confirmation TFs: **M15** and **H4**.
|
||||||
|
|
||||||
|
### MTF blending (when 2 of 3 TFs agree)
|
||||||
|
|
||||||
|
```
|
||||||
|
mtf_agree = (bull_count ≥ 2) OR (bear_count ≥ 2)
|
||||||
|
|
||||||
|
if mtf_agree:
|
||||||
|
score = 0.6 × primary_score + 0.4 × mean(all_scores)
|
||||||
|
confidence = min(confidence + 15, 95)
|
||||||
|
else:
|
||||||
|
confidence = confidence × 0.7 ← penalised for disagreement
|
||||||
|
```
|
||||||
|
|
||||||
|
### MTF agreement bonus in grade assignment
|
||||||
|
|
||||||
|
`mtf_agree=True` is required for grades A and A+.
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## 10. Signal Grade System
|
||||||
|
|
||||||
|
| Grade | Conditions |
|
||||||
|
|-------|-----------|
|
||||||
|
| **A+** | `|score| ≥ 50` AND `conf ≥ 75%` AND `mtf_agree` |
|
||||||
|
| **A** | `|score| ≥ 35` AND `conf ≥ 65%` AND `mtf_agree` |
|
||||||
|
| **B** | `|score| ≥ 25` AND `conf ≥ 55%` |
|
||||||
|
| **C** | `|score| ≥ 15` AND `conf ≥ 45%` |
|
||||||
|
| **D** | `|score| ≥ 10` |
|
||||||
|
| **F** | below all thresholds |
|
||||||
|
|
||||||
|
Default minimum grade for trade entry: **B** (`BacktestConfig.min_trade_grade`).
|
||||||
|
|
||||||
|
### Signal integer thresholds (for trade direction)
|
||||||
|
|
||||||
|
| Signal | Condition |
|
||||||
|
|--------|-----------|
|
||||||
|
| STRONG_BUY (+2) | score ≥ +40 AND conf ≥ 65% |
|
||||||
|
| BUY (+1) | score ≥ +15 AND conf ≥ 50% |
|
||||||
|
| STRONG_SELL (-2) | score ≤ -40 AND conf ≥ 65% |
|
||||||
|
| SELL (-1) | score ≤ -15 AND conf ≥ 50% |
|
||||||
|
| NEUTRAL | otherwise |
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## 11. GPU Backtester
|
||||||
|
|
||||||
|
`backtest_gpu.py` — completely separate from `backtest.py`.
|
||||||
|
|
||||||
|
### Why O(N) instead of O(N²)
|
||||||
|
|
||||||
|
`backtest.py` slices the DataFrame and recomputes all indicators fresh each bar → N × O(N) = **O(N²)**.
|
||||||
|
`backtest_gpu.py` computes every indicator series end-to-end once, then reads bar T's value in O(1) → **O(N) total**.
|
||||||
|
|
||||||
|
On 100,000 H1 bars: ~4 seconds vs >10 minutes.
|
||||||
|
|
||||||
|
### Two-phase architecture
|
||||||
|
|
||||||
|
```
|
||||||
|
Phase 1 — prepare_arrays(dfs)
|
||||||
|
VectorIndicators.compute_all() ← pandas ewm/rolling, C-optimised
|
||||||
|
Align MTF signals to primary TF ← forward-fill at close times
|
||||||
|
Compute regime ADX series ← if regime filter enabled
|
||||||
|
Returns reusable bundle dict
|
||||||
|
|
||||||
|
Phase 2 — score_and_run(bundle, weights, disabled)
|
||||||
|
compute_bar_scores() ← CuPy GPU or NumPy CPU
|
||||||
|
compute_final_scores() ← MTF blend + EMA200 gate (vectorised)
|
||||||
|
compute_grades() ← vectorised grade assignment
|
||||||
|
run_state_machine() ← sequential position management (O(N))
|
||||||
|
```
|
||||||
|
|
||||||
|
`prepare_arrays()` is called once; `score_and_run()` can be called many times with different weights — this is what the `--optimize` sweep uses.
|
||||||
|
|
||||||
|
### Optimization sweep (`--optimize`)
|
||||||
|
|
||||||
|
36 combinations tested per run:
|
||||||
|
- **1** baseline (default weights)
|
||||||
|
- **7** ablation runs (each indicator disabled one at a time)
|
||||||
|
- **28** weight multiplier runs (each indicator at ×0.5, ×1.5, ×2.0, ×3.0)
|
||||||
|
|
||||||
|
Indicator arrays computed once → all 36 replays run in seconds.
|
||||||
|
|
||||||
|
### Trade execution model
|
||||||
|
|
||||||
|
```
|
||||||
|
Entry: next bar Open ± (spread/2 + slippage) [default: 0.5 pip slippage]
|
||||||
|
SL: entry ± ATR × atr_sl_multiplier [default: 1.5×]
|
||||||
|
TP: entry ± SL_dist × rr_ratio [default: 2.0× = 1:2 R:R]
|
||||||
|
Risk: 1% of current equity per trade
|
||||||
|
```
|
||||||
|
|
||||||
|
If SL and TP both touch within the same bar: SL assumed to hit first (`sl_first_on_tie=True`).
|
||||||
|
|
||||||
|
### CLI flags
|
||||||
|
|
||||||
|
```
|
||||||
|
python backtest_gpu.py --tf H4 --local # local CSV, H4
|
||||||
|
python backtest_gpu.py --tf H1 --local --start 2022-01-01
|
||||||
|
python backtest_gpu.py --tf H4 --local --optimize # 36-combo weight sweep
|
||||||
|
python backtest_gpu.py --regime-filter --regime-tf D1
|
||||||
|
python backtest_gpu.py --session-filter # UTC 07-16 only
|
||||||
|
python backtest_gpu.py --journal --journal-csv out.csv
|
||||||
|
```
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## 12. Backtesting Results Summary
|
||||||
|
|
||||||
|
### H4 — local CSV 2022–2026 (optimised weights)
|
||||||
|
|
||||||
|
| Metric | Value |
|
||||||
|
|--------|-------|
|
||||||
|
| Trades | 160 |
|
||||||
|
| Win Rate | 38.1% |
|
||||||
|
| Profit Factor | 1.19 |
|
||||||
|
| Total Return | +21.30% |
|
||||||
|
| Max Drawdown | -8.52% |
|
||||||
|
| Sharpe | 0.47 |
|
||||||
|
| Sortino | 1.91 |
|
||||||
|
|
||||||
|
### D1 — Twelve Data 2007–2026 (optimised weights)
|
||||||
|
|
||||||
|
| Metric | Value |
|
||||||
|
|--------|-------|
|
||||||
|
| Trades | 62 |
|
||||||
|
| Win Rate | 40.3% |
|
||||||
|
| Profit Factor | 1.27 |
|
||||||
|
| Total Return | +10.51% |
|
||||||
|
| Max Drawdown | -6.84% |
|
||||||
|
| Sortino | 7.74 |
|
||||||
|
|
||||||
|
### Optimisation sweep delta (H4 2022–2026)
|
||||||
|
|
||||||
|
| | Baseline | Optimised |
|
||||||
|
|--|----------|-----------|
|
||||||
|
| Return | +7.12% | +21.30% |
|
||||||
|
| Max DD | -9.05% | -8.52% |
|
||||||
|
| Sharpe | 0.23 | 0.47 |
|
||||||
|
| Win Rate | 36.3% | 38.1% |
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## 13. Filter Catalogue
|
||||||
|
|
||||||
|
### Regime filter (`--regime-filter`)
|
||||||
|
|
||||||
|
Gates trade entry on ADX of a **higher** timeframe than the primary. Default: H4 ADX gate when trading H1. Redundant if regime TF == primary TF.
|
||||||
|
|
||||||
|
```
|
||||||
|
regime TF ADX ≥ min_adx (default 25) → entry allowed
|
||||||
|
```
|
||||||
|
|
||||||
|
Sensible defaults: H1→H4, H4→D1, D1→D1.
|
||||||
|
|
||||||
|
### Session filter (`--session-filter`)
|
||||||
|
|
||||||
|
Only trade during UTC 07:00–16:00 (London + NY overlap).
|
||||||
|
2026 diagnostic: 07-15 UTC = 47% WR / +127 pips; off-hours = 0% WR / -571 pips.
|
||||||
|
|
||||||
|
### Auto-session toggle (`auto_session_filter`)
|
||||||
|
|
||||||
|
Reactive version of session filter. Activates automatically when mean P&L over last N closed trades drops below threshold:
|
||||||
|
|
||||||
|
```
|
||||||
|
if mean(pnl_pct[-10:]) < -0.3%:
|
||||||
|
session_filter = ON ← restrict to liquid hours
|
||||||
|
else:
|
||||||
|
session_filter = OFF ← all hours allowed
|
||||||
|
```
|
||||||
|
|
||||||
|
### EMA200 distance gate
|
||||||
|
|
||||||
|
Always active. See §8.
|
||||||
|
|
||||||
|
### Signal grade gate (`min_trade_grade`)
|
||||||
|
|
||||||
|
Default: **B**. Only trades with grade ≥ B are entered. Raise to A for fewer, higher-quality trades.
|
||||||
|
|
||||||
|
### Min absolute score (`min_abs_score`)
|
||||||
|
|
||||||
|
Default: **0** (no extra gate). Set to e.g. 15 to skip weak directional scores.
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## 14. Configuration Reference
|
||||||
|
|
||||||
|
All knobs live in `config.py → PlatformConfig`.
|
||||||
|
|
||||||
|
```python
|
||||||
|
# Indicator parameters
|
||||||
|
TechnicalConfig.ema_fast = 9
|
||||||
|
TechnicalConfig.ema_medium = 21
|
||||||
|
TechnicalConfig.ema_slow = 50
|
||||||
|
TechnicalConfig.ema_trend = 200
|
||||||
|
TechnicalConfig.rsi_period = 14
|
||||||
|
TechnicalConfig.macd_fast / slow / signal = 12/26/9
|
||||||
|
TechnicalConfig.bb_period = 20
|
||||||
|
TechnicalConfig.atr_period = 14
|
||||||
|
TechnicalConfig.stoch_k / d = 14/3
|
||||||
|
TechnicalConfig.adx_period = 14
|
||||||
|
TechnicalConfig.adx_strong_trend = 25.0
|
||||||
|
TechnicalConfig.ichimoku_tenkan / kijun / senkou_b = 9/26/52
|
||||||
|
TechnicalConfig.disabled_indicators = ["pivot", "candlestick"]
|
||||||
|
|
||||||
|
# Signal scoring
|
||||||
|
SignalConfig.min_confidence = 55.0
|
||||||
|
SignalConfig.atr_sl_multiplier = 1.5
|
||||||
|
SignalConfig.default_rr_ratio = 2.0
|
||||||
|
SignalConfig.default_risk_pct = 1.0
|
||||||
|
|
||||||
|
# Backtest
|
||||||
|
BacktestConfig.initial_balance = 10_000
|
||||||
|
BacktestConfig.risk_per_trade_pct = 1.0
|
||||||
|
BacktestConfig.min_trade_grade = "B"
|
||||||
|
BacktestConfig.regime_filter = False
|
||||||
|
BacktestConfig.session_filter = False
|
||||||
|
BacktestConfig.auto_session_filter = False
|
||||||
|
BacktestConfig.cache_dir = ".cache"
|
||||||
|
```
|
||||||
+66
-9
@@ -15,6 +15,7 @@ import logging
|
|||||||
import multiprocessing as mp
|
import multiprocessing as mp
|
||||||
import os
|
import os
|
||||||
from collections import deque
|
from collections import deque
|
||||||
|
from concurrent.futures import ThreadPoolExecutor, as_completed
|
||||||
from dataclasses import replace
|
from dataclasses import replace
|
||||||
from typing import Deque, Dict, List, Optional, Tuple
|
from typing import Deque, Dict, List, Optional, Tuple
|
||||||
|
|
||||||
@@ -88,6 +89,63 @@ class WalkForwardBacktester:
|
|||||||
dfs[tf] = df
|
dfs[tf] = df
|
||||||
return dfs
|
return dfs
|
||||||
|
|
||||||
|
def _precompute_all_signals(
|
||||||
|
self,
|
||||||
|
dfs: Dict[Timeframe, pd.DataFrame],
|
||||||
|
primary_df: pd.DataFrame,
|
||||||
|
warmup: int,
|
||||||
|
progress: bool = True,
|
||||||
|
) -> Dict[int, "SignalReport"]:
|
||||||
|
"""Pre-compute signals for every bar in parallel using all CPU threads.
|
||||||
|
|
||||||
|
TechnicalEngine.analyze() is stateless (pure function of the df slice),
|
||||||
|
so concurrent calls are safe. NumPy/pandas C extensions release the GIL,
|
||||||
|
giving real parallelism across threads without pickling overhead.
|
||||||
|
Returns bar-index → SignalReport so the sequential state machine can
|
||||||
|
do O(1) lookups instead of recomputing indicators per bar.
|
||||||
|
"""
|
||||||
|
primary_tf = self.config.primary_timeframe
|
||||||
|
# Use all logical threads; cap at 2× physical cores to avoid over-subscription
|
||||||
|
n_workers = min(os.cpu_count() or 4, 16)
|
||||||
|
bar_indices = list(range(warmup, len(primary_df) - 1))
|
||||||
|
|
||||||
|
def _compute_bar(t: int) -> Tuple[int, "SignalReport"]:
|
||||||
|
decision_time = self._bar_close_time(primary_df.index[t], primary_tf)
|
||||||
|
sliced = self._slice_dfs(dfs, decision_time)
|
||||||
|
report = self.aggregator.analyze_from_dataframes(
|
||||||
|
symbol=self.config.symbol,
|
||||||
|
dfs=sliced,
|
||||||
|
use_news=False,
|
||||||
|
)
|
||||||
|
return t, report
|
||||||
|
|
||||||
|
precomputed: Dict[int, "SignalReport"] = {}
|
||||||
|
pbar = None
|
||||||
|
if progress:
|
||||||
|
try:
|
||||||
|
from tqdm import tqdm # type: ignore
|
||||||
|
pbar = tqdm(
|
||||||
|
total=len(bar_indices),
|
||||||
|
desc=f"Signals ({n_workers} threads)",
|
||||||
|
unit="bar",
|
||||||
|
leave=False,
|
||||||
|
)
|
||||||
|
except ImportError:
|
||||||
|
pass
|
||||||
|
|
||||||
|
with ThreadPoolExecutor(max_workers=n_workers) as executor:
|
||||||
|
futures = {executor.submit(_compute_bar, t): t for t in bar_indices}
|
||||||
|
for future in as_completed(futures):
|
||||||
|
t, report = future.result()
|
||||||
|
precomputed[t] = report
|
||||||
|
if pbar:
|
||||||
|
pbar.update(1)
|
||||||
|
|
||||||
|
if pbar:
|
||||||
|
pbar.close()
|
||||||
|
|
||||||
|
return precomputed
|
||||||
|
|
||||||
def run(
|
def run(
|
||||||
self,
|
self,
|
||||||
dfs: Optional[Dict[Timeframe, pd.DataFrame]] = None,
|
dfs: Optional[Dict[Timeframe, pd.DataFrame]] = None,
|
||||||
@@ -117,6 +175,11 @@ class WalkForwardBacktester:
|
|||||||
)
|
)
|
||||||
return BacktestResult(initial_balance=self.config.backtest.initial_balance)
|
return BacktestResult(initial_balance=self.config.backtest.initial_balance)
|
||||||
|
|
||||||
|
# Pre-compute all bar signals in parallel before the sequential replay
|
||||||
|
precomputed = self._precompute_all_signals(
|
||||||
|
dfs, primary_df, warmup, progress=progress
|
||||||
|
)
|
||||||
|
|
||||||
bt = self.config.backtest
|
bt = self.config.backtest
|
||||||
balance = bt.initial_balance
|
balance = bt.initial_balance
|
||||||
equity_points: List[Tuple[pd.Timestamp, float]] = [
|
equity_points: List[Tuple[pd.Timestamp, float]] = [
|
||||||
@@ -133,12 +196,12 @@ class WalkForwardBacktester:
|
|||||||
|
|
||||||
open_position: Optional[Dict] = None
|
open_position: Optional[Dict] = None
|
||||||
|
|
||||||
# Optional progress bar
|
# Optional progress bar for the state-machine replay pass
|
||||||
iterator = range(warmup, len(primary_df) - 1)
|
iterator = range(warmup, len(primary_df) - 1)
|
||||||
if progress:
|
if progress:
|
||||||
try:
|
try:
|
||||||
from tqdm import tqdm # type: ignore
|
from tqdm import tqdm # type: ignore
|
||||||
iterator = tqdm(iterator, desc="Backtesting", unit="bar", leave=False)
|
iterator = tqdm(iterator, desc="Replaying bars", unit="bar", leave=False)
|
||||||
except ImportError:
|
except ImportError:
|
||||||
pass
|
pass
|
||||||
|
|
||||||
@@ -161,13 +224,7 @@ class WalkForwardBacktester:
|
|||||||
|
|
||||||
# ── Step 2: maybe open a new position ──────────────────
|
# ── Step 2: maybe open a new position ──────────────────
|
||||||
if open_position is None or not bt.one_position_only:
|
if open_position is None or not bt.one_position_only:
|
||||||
# Slice each TF's df up to (and including) bars closed by decision_time
|
report = precomputed[t]
|
||||||
sliced = self._slice_dfs(dfs, decision_time)
|
|
||||||
report = self.aggregator.analyze_from_dataframes(
|
|
||||||
symbol=self.config.symbol,
|
|
||||||
dfs=sliced,
|
|
||||||
use_news=False,
|
|
||||||
)
|
|
||||||
|
|
||||||
if (
|
if (
|
||||||
_grade_passes(report.signal_quality, bt.min_trade_grade)
|
_grade_passes(report.signal_quality, bt.min_trade_grade)
|
||||||
|
|||||||
+1299
File diff suppressed because it is too large
Load Diff
@@ -29,6 +29,14 @@ class DataIntegrityError(Exception):
|
|||||||
class ForexDataProvider:
|
class ForexDataProvider:
|
||||||
"""Twelve Data SDK client + on-disk parquet cache."""
|
"""Twelve Data SDK client + on-disk parquet cache."""
|
||||||
|
|
||||||
|
# Local MetaTrader CSV exports — tab-delimited, 7 columns (includes Spread)
|
||||||
|
LOCAL_CSV_MAP = {
|
||||||
|
Timeframe.M15: "USDJPY_M15.csv",
|
||||||
|
Timeframe.H1: "USDJPY_H1.csv",
|
||||||
|
Timeframe.H4: "USDJPY_H4.csv",
|
||||||
|
Timeframe.D1: "USDJPY_D1.csv",
|
||||||
|
}
|
||||||
|
|
||||||
TD_INTERVALS = {
|
TD_INTERVALS = {
|
||||||
Timeframe.M1: "1min",
|
Timeframe.M1: "1min",
|
||||||
Timeframe.M5: "5min",
|
Timeframe.M5: "5min",
|
||||||
@@ -153,6 +161,55 @@ class ForexDataProvider:
|
|||||||
|
|
||||||
return df
|
return df
|
||||||
|
|
||||||
|
def load_local_csv(
|
||||||
|
self,
|
||||||
|
timeframe: Timeframe,
|
||||||
|
data_dir: str = "market_data",
|
||||||
|
start_date: Optional[str] = None,
|
||||||
|
end_date: Optional[str] = None,
|
||||||
|
) -> Optional[pd.DataFrame]:
|
||||||
|
"""
|
||||||
|
Load OHLCV from a local MetaTrader CSV export.
|
||||||
|
Format: tab-separated, columns Time/Open/High/Low/Close/Volume/Spread.
|
||||||
|
Returns None silently if the file doesn't exist for this timeframe.
|
||||||
|
"""
|
||||||
|
fname = self.LOCAL_CSV_MAP.get(timeframe)
|
||||||
|
if fname is None:
|
||||||
|
logger.info(f"No local CSV mapping for {timeframe.value} — skipping")
|
||||||
|
return None
|
||||||
|
|
||||||
|
path = os.path.join(data_dir, fname)
|
||||||
|
if not os.path.exists(path):
|
||||||
|
logger.warning(f"Local CSV not found: {path}")
|
||||||
|
return None
|
||||||
|
|
||||||
|
df = pd.read_csv(
|
||||||
|
path, sep="\t",
|
||||||
|
names=["Time", "Open", "High", "Low", "Close", "Volume", "Spread"],
|
||||||
|
header=0,
|
||||||
|
parse_dates=["Time"],
|
||||||
|
)
|
||||||
|
df = df.set_index("Time")
|
||||||
|
df = df.drop(columns=["Spread"], errors="ignore")
|
||||||
|
for col in ("Open", "High", "Low", "Close", "Volume"):
|
||||||
|
if col in df.columns:
|
||||||
|
df[col] = pd.to_numeric(df[col], errors="coerce")
|
||||||
|
if "Volume" not in df.columns:
|
||||||
|
df["Volume"] = 0.0
|
||||||
|
|
||||||
|
if df.index.tz is None:
|
||||||
|
df.index = df.index.tz_localize("UTC")
|
||||||
|
df.index.name = "Datetime"
|
||||||
|
|
||||||
|
if start_date:
|
||||||
|
df = df[df.index >= pd.Timestamp(start_date, tz="UTC")]
|
||||||
|
if end_date:
|
||||||
|
df = df[df.index <= pd.Timestamp(end_date, tz="UTC")]
|
||||||
|
|
||||||
|
df = self._validate_and_clean(df, USDJPY, timeframe)
|
||||||
|
logger.info(f"Loaded {len(df):,} bars from local CSV: {path}")
|
||||||
|
return df
|
||||||
|
|
||||||
def get_latest_price(self, symbol: str = USDJPY) -> Optional[float]:
|
def get_latest_price(self, symbol: str = USDJPY) -> Optional[float]:
|
||||||
try:
|
try:
|
||||||
df = self.fetch_ohlcv(symbol, Timeframe.M5, lookback_days=1)
|
df = self.fetch_ohlcv(symbol, Timeframe.M5, lookback_days=1)
|
||||||
|
|||||||
@@ -171,7 +171,7 @@ class TechnicalEngine:
|
|||||||
result.indicators["ema_trend"] = IndicatorSignal(
|
result.indicators["ema_trend"] = IndicatorSignal(
|
||||||
name="EMA Trend", value=normalized,
|
name="EMA Trend", value=normalized,
|
||||||
signal=signal, confidence=conf,
|
signal=signal, confidence=conf,
|
||||||
description=desc, weight=1.2,
|
description=desc, weight=3.6,
|
||||||
)
|
)
|
||||||
if t is not None:
|
if t is not None:
|
||||||
result.key_levels["ema_trend"] = float(t)
|
result.key_levels["ema_trend"] = float(t)
|
||||||
@@ -222,7 +222,7 @@ class TechnicalEngine:
|
|||||||
|
|
||||||
result.indicators["rsi"] = IndicatorSignal(
|
result.indicators["rsi"] = IndicatorSignal(
|
||||||
name="RSI", value=rsi_val, signal=signal,
|
name="RSI", value=rsi_val, signal=signal,
|
||||||
confidence=conf, description=desc, weight=0.3,
|
confidence=conf, description=desc, weight=0.0,
|
||||||
)
|
)
|
||||||
|
|
||||||
# Attach RSI series for divergence detection (dynamic attribute, filtered by _prefix)
|
# Attach RSI series for divergence detection (dynamic attribute, filtered by _prefix)
|
||||||
@@ -291,7 +291,7 @@ class TechnicalEngine:
|
|||||||
|
|
||||||
result.indicators["macd"] = IndicatorSignal(
|
result.indicators["macd"] = IndicatorSignal(
|
||||||
name="MACD", value=hist_val, signal=signal,
|
name="MACD", value=hist_val, signal=signal,
|
||||||
confidence=conf, description=desc, weight=1.1,
|
confidence=conf, description=desc, weight=3.3,
|
||||||
)
|
)
|
||||||
|
|
||||||
# Attach histogram for divergence detection
|
# Attach histogram for divergence detection
|
||||||
@@ -358,7 +358,7 @@ class TechnicalEngine:
|
|||||||
|
|
||||||
result.indicators["bollinger"] = IndicatorSignal(
|
result.indicators["bollinger"] = IndicatorSignal(
|
||||||
name="Bollinger Bands", value=bb_position, signal=signal,
|
name="Bollinger Bands", value=bb_position, signal=signal,
|
||||||
confidence=conf, description=desc, weight=0.9,
|
confidence=conf, description=desc, weight=0.0,
|
||||||
)
|
)
|
||||||
|
|
||||||
result.key_levels["bb_upper"] = upper_val
|
result.key_levels["bb_upper"] = upper_val
|
||||||
@@ -423,7 +423,7 @@ class TechnicalEngine:
|
|||||||
|
|
||||||
result.indicators["stochastic"] = IndicatorSignal(
|
result.indicators["stochastic"] = IndicatorSignal(
|
||||||
name="Stochastic", value=k_val, signal=signal,
|
name="Stochastic", value=k_val, signal=signal,
|
||||||
confidence=conf, description=desc, weight=0.8,
|
confidence=conf, description=desc, weight=1.6,
|
||||||
)
|
)
|
||||||
|
|
||||||
def _compute_adx(self, df: pd.DataFrame, result: TechnicalAnalysis):
|
def _compute_adx(self, df: pd.DataFrame, result: TechnicalAnalysis):
|
||||||
@@ -483,7 +483,7 @@ class TechnicalEngine:
|
|||||||
|
|
||||||
result.indicators["adx"] = IndicatorSignal(
|
result.indicators["adx"] = IndicatorSignal(
|
||||||
name="ADX", value=adx_val, signal=signal,
|
name="ADX", value=adx_val, signal=signal,
|
||||||
confidence=conf, description=desc, weight=1.0,
|
confidence=conf, description=desc, weight=0.5,
|
||||||
)
|
)
|
||||||
|
|
||||||
def _compute_atr(self, df: pd.DataFrame, result: TechnicalAnalysis):
|
def _compute_atr(self, df: pd.DataFrame, result: TechnicalAnalysis):
|
||||||
@@ -609,7 +609,7 @@ class TechnicalEngine:
|
|||||||
name="Ichimoku", value=float(score), signal=signal,
|
name="Ichimoku", value=float(score), signal=signal,
|
||||||
confidence=conf,
|
confidence=conf,
|
||||||
description=f"Ichimoku [{score}/4]: {' | '.join(signals)}",
|
description=f"Ichimoku [{score}/4]: {' | '.join(signals)}",
|
||||||
weight=1.1,
|
weight=0.55,
|
||||||
)
|
)
|
||||||
|
|
||||||
result.key_levels["ichimoku_cloud_top"] = cloud_top
|
result.key_levels["ichimoku_cloud_top"] = cloud_top
|
||||||
|
|||||||
File diff suppressed because it is too large
Load Diff
+100001
File diff suppressed because it is too large
Load Diff
File diff suppressed because it is too large
Load Diff
+100001
File diff suppressed because it is too large
Load Diff
@@ -8,6 +8,7 @@ OHLCV is fetched live or sliced from a pre-loaded historical buffer.
|
|||||||
"""
|
"""
|
||||||
|
|
||||||
import logging
|
import logging
|
||||||
|
import os
|
||||||
from dataclasses import dataclass, field
|
from dataclasses import dataclass, field
|
||||||
from datetime import datetime, timezone
|
from datetime import datetime, timezone
|
||||||
from typing import Dict, List, Optional
|
from typing import Dict, List, Optional
|
||||||
@@ -393,16 +394,19 @@ class SignalAggregator:
|
|||||||
ema200 = primary.key_levels.get("ema_trend")
|
ema200 = primary.key_levels.get("ema_trend")
|
||||||
if ema200 and ema200 > 0 and report.current_price > 0:
|
if ema200 and ema200 > 0 and report.current_price > 0:
|
||||||
ema_dist_pct = abs(report.current_price - ema200) / ema200 * 100.0
|
ema_dist_pct = abs(report.current_price - ema200) / ema200 * 100.0
|
||||||
if ema_dist_pct < 0.5:
|
_lo = float(os.environ.get("FB2_EMA_LO", "0.5"))
|
||||||
|
_hi = float(os.environ.get("FB2_EMA_HI", "1.5"))
|
||||||
|
_mult = float(os.environ.get("FB2_EMA_MULT", "0.55"))
|
||||||
|
if ema_dist_pct < _lo:
|
||||||
report.warnings.append(
|
report.warnings.append(
|
||||||
f"NEAR EMA200 ({ema_dist_pct:.2f}%) — chop risk, signal damped"
|
f"NEAR EMA200 ({ema_dist_pct:.2f}%) — chop risk, signal damped"
|
||||||
)
|
)
|
||||||
confidence *= 0.55
|
confidence *= _mult
|
||||||
elif ema_dist_pct > 1.5:
|
elif ema_dist_pct > _hi:
|
||||||
report.warnings.append(
|
report.warnings.append(
|
||||||
f"FAR FROM EMA200 ({ema_dist_pct:.2f}%) — over-extended, signal damped"
|
f"FAR FROM EMA200 ({ema_dist_pct:.2f}%) — over-extended, signal damped"
|
||||||
)
|
)
|
||||||
confidence *= 0.55
|
confidence *= _mult
|
||||||
|
|
||||||
# Final signal
|
# Final signal
|
||||||
if score >= 40 and confidence >= 65:
|
if score >= 40 and confidence >= 65:
|
||||||
|
|||||||
Reference in New Issue
Block a user