Strip backtesting, rebalance indicator weights toward leading signals
Remove backtest.py, backtest_gpu.py, metrics.py, local CSV data, Dukascopy loader, and all backtest CLI modes from main.py. Rebalance engine weights: leading indicators (divergence, RSI, stochastic, BB) now carry 66% of signal weight vs 34% for lagging (EMA, MACD). Add ws_twelvedata.py for live tick streaming. Drop pyarrow and tqdm deps.
This commit is contained in:
Harwinder
-450
@@ -1,450 +0,0 @@
|
||||
# ForexBot2 — Engine Logic & Design Reference
|
||||
|
||||
> USD/JPY signal intelligence platform.
|
||||
> Last updated: 2026-05-08
|
||||
|
||||
---
|
||||
|
||||
## Table of Contents
|
||||
|
||||
1. [Architecture Overview](#1-architecture-overview)
|
||||
2. [Data Pipeline](#2-data-pipeline)
|
||||
3. [Indicator Engine (`engine.py`)](#3-indicator-engine)
|
||||
4. [Signal Aggregator (`signal_aggregator.py`)](#4-signal-aggregator)
|
||||
5. [Composite Scoring Formula](#5-composite-scoring-formula)
|
||||
6. [Indicator Weights — Optimised](#6-indicator-weights--optimised)
|
||||
7. [Market Regime Classification](#7-market-regime-classification)
|
||||
8. [EMA200 Distance Gate](#8-ema200-distance-gate)
|
||||
9. [Multi-Timeframe Alignment](#9-multi-timeframe-alignment)
|
||||
10. [Signal Grade System](#10-signal-grade-system)
|
||||
11. [GPU Backtester (`backtest_gpu.py`)](#11-gpu-backtester)
|
||||
12. [Backtesting Results Summary](#12-backtesting-results-summary)
|
||||
13. [Filter Catalogue](#13-filter-catalogue)
|
||||
14. [Configuration Reference](#14-configuration-reference)
|
||||
|
||||
---
|
||||
|
||||
## 1. Architecture Overview
|
||||
|
||||
```
|
||||
market_data/ (local CSV)
|
||||
│
|
||||
▼
|
||||
ForexDataProvider ← also pulls live via Twelve Data SDK
|
||||
│
|
||||
▼
|
||||
TechnicalEngine ← computes all indicators bar-by-bar
|
||||
│
|
||||
▼
|
||||
SignalAggregator ← MTF blending, EMA200 gate, grading
|
||||
│
|
||||
├──▶ Live output (main.py)
|
||||
└──▶ GPUBacktester ← vectorised O(N) replay over full history
|
||||
```
|
||||
|
||||
**Key design principle:** No single indicator is trusted alone. The composite score is a confidence-weighted, regime-adjusted blend of up to 7 indicators across 3 timeframes. A signal only fires when direction, strength, and confidence all agree.
|
||||
|
||||
---
|
||||
|
||||
## 2. Data Pipeline
|
||||
|
||||
### Live mode
|
||||
`ForexDataProvider.fetch_ohlcv()` calls the Twelve Data SDK with in-memory caching (5-min TTL). Fetches primary TF + two confirmation TFs + optional regime TF.
|
||||
|
||||
### Backtest mode
|
||||
`ForexDataProvider.load_local_csv()` reads MetaTrader tab-delimited exports from `market_data/`:
|
||||
|
||||
| File | TF | Rows | Date Range |
|
||||
|------|----|------|------------|
|
||||
| USDJPY_M15.csv | M15 | 100,000 | 2022-05-04 → 2026-05-08 |
|
||||
| USDJPY_H1.csv | H1 | 100,000 | 2010-04-23 → 2026-05-08 |
|
||||
| USDJPY_H4.csv | H4 | 25,855 | 2010-04-22 → 2026-05-08 |
|
||||
| USDJPY_D1.csv | D1 | 5,020 | 2010-04-22 → 2026-05-08 |
|
||||
|
||||
CSV format: tab-delimited, columns `Time | Open | High | Low | Close | Volume | Spread`. Spread column is dropped on load. Timestamps are UTC.
|
||||
|
||||
**Data validation** (applied to both sources):
|
||||
- Drop NaN OHLC rows; forward-fill up to 3 bars
|
||||
- Drop future timestamps (>5 min ahead of UTC now)
|
||||
- Fix OHLC inconsistencies (High < Low etc.)
|
||||
- Remove outlier bars where `|return| > 10 × rolling_std`
|
||||
|
||||
---
|
||||
|
||||
## 3. Indicator Engine
|
||||
|
||||
`engine.py → TechnicalEngine.analyze(df, symbol, timeframe)`
|
||||
|
||||
All indicators are computed on a single OHLCV DataFrame. Each returns an `IndicatorSignal` with:
|
||||
- `value` — raw numeric reading
|
||||
- `signal` — `SignalStrength` enum: STRONG_BUY(+2) / BUY(+1) / NEUTRAL(0) / SELL(-1) / STRONG_SELL(-2)
|
||||
- `confidence` — 0–100%
|
||||
- `weight` — contribution to composite score (0 = excluded)
|
||||
|
||||
### Indicators computed
|
||||
|
||||
| Name | Key | Wilder smoothing? | Notes |
|
||||
|------|-----|:-----------------:|-------|
|
||||
| EMA Alignment | `ema_trend` | No | 4-EMA stack: 9/21/50/200; normalised to -1..+1 |
|
||||
| RSI | `rsi` | Yes (`alpha=1/14`) | Overbought/oversold + momentum zones |
|
||||
| MACD | `macd` | No | 12/26/9; crossover = STRONG signal |
|
||||
| Bollinger Bands | `bollinger` | No | 20-period; position within band + squeeze detect |
|
||||
| Stochastic | `stochastic` | No | 14/3; K+D both OB/OS required for signal |
|
||||
| ADX | `adx` | Yes | DI+/DI- direction + ADX strength gate (≥25) |
|
||||
| ATR | `atr` | Yes | Context only — sets SL/TP distance; weight=0 |
|
||||
| Ichimoku | `ichimoku` | No | Price vs cloud (+2/-2) + TK cross (+1/-1) + cloud colour (+1/-1) |
|
||||
| Divergence | `divergence` | — | RSI/MACD price vs indicator swing divergence; weight=1.5 |
|
||||
| Candlestick | `candlestick` | — | **Disabled** (ablation: net -3.4%) |
|
||||
| Pivot Points | `pivot` | — | **Disabled** (ablation: net -9.5%) |
|
||||
| Volume | `volume` | — | Context filter; weight=0 |
|
||||
|
||||
**Disabled indicators** are controlled by `TechnicalConfig.disabled_indicators` (default: `["pivot", "candlestick"]`).
|
||||
|
||||
---
|
||||
|
||||
## 4. Signal Aggregator
|
||||
|
||||
`signal_aggregator.py → SignalAggregator.analyze()`
|
||||
|
||||
Orchestrates three timeframes (primary + 2 confirmation), blends their scores, applies gates, and emits a final graded signal.
|
||||
|
||||
### Flow
|
||||
|
||||
```
|
||||
H1 (primary) ──┐
|
||||
M15 (confirm) ──┼──▶ per-TF score ──▶ MTF blend ──▶ EMA200 gate ──▶ grade ──▶ output
|
||||
H4 (confirm) ──┘
|
||||
```
|
||||
|
||||
### MTF alignment (no lookahead)
|
||||
|
||||
Lower-frequency bars are aligned to the primary TF index by forward-filling at **close times**:
|
||||
|
||||
```
|
||||
secondary close time = secondary bar open + secondary TF minutes
|
||||
primary close time = primary bar open + primary TF minutes
|
||||
|
||||
For each primary bar T: use the most recent secondary close ≤ T_close
|
||||
```
|
||||
|
||||
This ensures no future information leaks into the signal.
|
||||
|
||||
---
|
||||
|
||||
## 5. Composite Scoring Formula
|
||||
|
||||
Mirrors `engine._compute_composite` exactly:
|
||||
|
||||
```
|
||||
composite_score = clip(
|
||||
(Σ signal_i × weight_i × regime_mult_i × (confidence_i / 100))
|
||||
────────────────────────────────────────────────────────────────── × 50,
|
||||
(Σ weight_i × regime_mult_i)
|
||||
-100, +100
|
||||
)
|
||||
|
||||
avg_confidence = Σ(confidence_i × weight_i) / Σ weight_i
|
||||
```
|
||||
|
||||
- `signal_i` ∈ {-2, -1, 0, +1, +2}
|
||||
- `weight_i` — per-indicator base weight (see §6)
|
||||
- `regime_mult_i` — boosted/dampened by market regime (see §7)
|
||||
- Indicators with `weight == 0` are excluded entirely
|
||||
|
||||
---
|
||||
|
||||
## 6. Indicator Weights — Optimised
|
||||
|
||||
Optimised 2026-05-08 via 36-combo sweep (ablation + ×0.5/1.5/2.0/3.0 per indicator) on H4 data 2022–2026. Combined result: **+21.30% return / -8.52% max DD / Sharpe 0.47** vs baseline +7.12% / -9.05% DD / Sharpe 0.23.
|
||||
|
||||
| Indicator | Weight | Change from original | Reason |
|
||||
|-----------|-------:|---------------------|--------|
|
||||
| `ema_trend` | **3.6** | was 1.2 (×3) | Strongest directional signal on trending pair |
|
||||
| `macd` | **3.3** | was 1.1 (×3) | Most impactful single indicator; ablation = worst result |
|
||||
| `stochastic` | **1.6** | was 0.8 (×2) | Clean timing signal; 40% WR at ×2 |
|
||||
| `adx` | **0.5** | was 1.0 (÷2) | Partially redundant with EMA stack |
|
||||
| `ichimoku` | **0.55** | was 1.1 (÷2) | Lags too much at H4; hurts on ranging periods |
|
||||
| `rsi` | **0.0** | was 0.3 (ablated) | Net drag across all test windows |
|
||||
| `bollinger` | **0.0** | was 0.9 (ablated) | Counter-trend noise on trending pair |
|
||||
| `divergence` | 1.5 | unchanged | Strong reversal signal; not in sweep scope |
|
||||
| `atr` | 0.0 | unchanged | Context only; sets SL/TP |
|
||||
|
||||
> **Note:** RSI and Bollinger are still **computed and displayed** in live output for reference, but their weight=0 means they contribute nothing to the composite score.
|
||||
|
||||
### Regime weight multipliers
|
||||
|
||||
Applied on top of base weights:
|
||||
|
||||
| Indicator | Trending mult | Ranging mult |
|
||||
|-----------|:-------------:|:------------:|
|
||||
| ema_trend | ×1.5 | ×0.6 |
|
||||
| adx | ×1.3 | ×0.5 |
|
||||
| macd | ×1.2 | ×1.0 |
|
||||
| ichimoku | ×1.3 | ×0.2 |
|
||||
| rsi | ×0.7 | ×1.4 |
|
||||
| bollinger | ×0.6 | ×1.4 |
|
||||
| stochastic | ×1.0 | ×1.3 |
|
||||
|
||||
In trending regimes: EMA trend, MACD, and Ichimoku are boosted further.
|
||||
In ranging regimes: RSI and Bollinger would be boosted — but since both are weight=0, ranging regime currently relies entirely on stochastic + reduced EMA/ADX.
|
||||
|
||||
---
|
||||
|
||||
## 7. Market Regime Classification
|
||||
|
||||
`engine._classify_regime()` — runs after all indicators are computed:
|
||||
|
||||
```
|
||||
if ADX ≥ 25:
|
||||
if EMA signal is BUY or STRONG_BUY → trending_up
|
||||
if EMA signal is SELL or STRONG_SELL → trending_down
|
||||
else → trending
|
||||
else:
|
||||
→ ranging
|
||||
```
|
||||
|
||||
Regime feeds directly into the weight multipliers applied in `_compute_composite`.
|
||||
|
||||
---
|
||||
|
||||
## 8. EMA200 Distance Gate
|
||||
|
||||
**Strongest single filter discovered** — implemented in `signal_aggregator.py`.
|
||||
|
||||
```
|
||||
ema_dist = |close - EMA200| / EMA200 × 100 (percent)
|
||||
|
||||
if ema_dist < 0.5% or ema_dist > 1.5%:
|
||||
confidence *= 0.55 ← damped
|
||||
else: ← sweet spot
|
||||
no change
|
||||
```
|
||||
|
||||
**Rationale:** Price within 0.5% of EMA200 is in a high-chop zone where the 200 EMA acts as contested support/resistance. Price more than 1.5% away has often already run far and is prone to reversion. The 0.5–1.5% band is where trending momentum is cleanest.
|
||||
|
||||
**Impact measured:** Lifted cumulative return +13.88% → +35.50% in the 2024-trend ablation window.
|
||||
|
||||
---
|
||||
|
||||
## 9. Multi-Timeframe Alignment
|
||||
|
||||
Primary TF: **H1** (default). Confirmation TFs: **M15** and **H4**.
|
||||
|
||||
### MTF blending (when 2 of 3 TFs agree)
|
||||
|
||||
```
|
||||
mtf_agree = (bull_count ≥ 2) OR (bear_count ≥ 2)
|
||||
|
||||
if mtf_agree:
|
||||
score = 0.6 × primary_score + 0.4 × mean(all_scores)
|
||||
confidence = min(confidence + 15, 95)
|
||||
else:
|
||||
confidence = confidence × 0.7 ← penalised for disagreement
|
||||
```
|
||||
|
||||
### MTF agreement bonus in grade assignment
|
||||
|
||||
`mtf_agree=True` is required for grades A and A+.
|
||||
|
||||
---
|
||||
|
||||
## 10. Signal Grade System
|
||||
|
||||
| Grade | Conditions |
|
||||
|-------|-----------|
|
||||
| **A+** | `|score| ≥ 50` AND `conf ≥ 75%` AND `mtf_agree` |
|
||||
| **A** | `|score| ≥ 35` AND `conf ≥ 65%` AND `mtf_agree` |
|
||||
| **B** | `|score| ≥ 25` AND `conf ≥ 55%` |
|
||||
| **C** | `|score| ≥ 15` AND `conf ≥ 45%` |
|
||||
| **D** | `|score| ≥ 10` |
|
||||
| **F** | below all thresholds |
|
||||
|
||||
Default minimum grade for trade entry: **B** (`BacktestConfig.min_trade_grade`).
|
||||
|
||||
### Signal integer thresholds (for trade direction)
|
||||
|
||||
| Signal | Condition |
|
||||
|--------|-----------|
|
||||
| STRONG_BUY (+2) | score ≥ +40 AND conf ≥ 65% |
|
||||
| BUY (+1) | score ≥ +15 AND conf ≥ 50% |
|
||||
| STRONG_SELL (-2) | score ≤ -40 AND conf ≥ 65% |
|
||||
| SELL (-1) | score ≤ -15 AND conf ≥ 50% |
|
||||
| NEUTRAL | otherwise |
|
||||
|
||||
---
|
||||
|
||||
## 11. GPU Backtester
|
||||
|
||||
`backtest_gpu.py` — completely separate from `backtest.py`.
|
||||
|
||||
### Why O(N) instead of O(N²)
|
||||
|
||||
`backtest.py` slices the DataFrame and recomputes all indicators fresh each bar → N × O(N) = **O(N²)**.
|
||||
`backtest_gpu.py` computes every indicator series end-to-end once, then reads bar T's value in O(1) → **O(N) total**.
|
||||
|
||||
On 100,000 H1 bars: ~4 seconds vs >10 minutes.
|
||||
|
||||
### Two-phase architecture
|
||||
|
||||
```
|
||||
Phase 1 — prepare_arrays(dfs)
|
||||
VectorIndicators.compute_all() ← pandas ewm/rolling, C-optimised
|
||||
Align MTF signals to primary TF ← forward-fill at close times
|
||||
Compute regime ADX series ← if regime filter enabled
|
||||
Returns reusable bundle dict
|
||||
|
||||
Phase 2 — score_and_run(bundle, weights, disabled)
|
||||
compute_bar_scores() ← CuPy GPU or NumPy CPU
|
||||
compute_final_scores() ← MTF blend + EMA200 gate (vectorised)
|
||||
compute_grades() ← vectorised grade assignment
|
||||
run_state_machine() ← sequential position management (O(N))
|
||||
```
|
||||
|
||||
`prepare_arrays()` is called once; `score_and_run()` can be called many times with different weights — this is what the `--optimize` sweep uses.
|
||||
|
||||
### Optimization sweep (`--optimize`)
|
||||
|
||||
36 combinations tested per run:
|
||||
- **1** baseline (default weights)
|
||||
- **7** ablation runs (each indicator disabled one at a time)
|
||||
- **28** weight multiplier runs (each indicator at ×0.5, ×1.5, ×2.0, ×3.0)
|
||||
|
||||
Indicator arrays computed once → all 36 replays run in seconds.
|
||||
|
||||
### Trade execution model
|
||||
|
||||
```
|
||||
Entry: next bar Open ± (spread/2 + slippage) [default: 0.5 pip slippage]
|
||||
SL: entry ± ATR × atr_sl_multiplier [default: 1.5×]
|
||||
TP: entry ± SL_dist × rr_ratio [default: 2.0× = 1:2 R:R]
|
||||
Risk: 1% of current equity per trade
|
||||
```
|
||||
|
||||
If SL and TP both touch within the same bar: SL assumed to hit first (`sl_first_on_tie=True`).
|
||||
|
||||
### CLI flags
|
||||
|
||||
```
|
||||
python backtest_gpu.py --tf H4 --local # local CSV, H4
|
||||
python backtest_gpu.py --tf H1 --local --start 2022-01-01
|
||||
python backtest_gpu.py --tf H4 --local --optimize # 36-combo weight sweep
|
||||
python backtest_gpu.py --regime-filter --regime-tf D1
|
||||
python backtest_gpu.py --session-filter # UTC 07-16 only
|
||||
python backtest_gpu.py --journal --journal-csv out.csv
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## 12. Backtesting Results Summary
|
||||
|
||||
### H4 — local CSV 2022–2026 (optimised weights)
|
||||
|
||||
| Metric | Value |
|
||||
|--------|-------|
|
||||
| Trades | 160 |
|
||||
| Win Rate | 38.1% |
|
||||
| Profit Factor | 1.19 |
|
||||
| Total Return | +21.30% |
|
||||
| Max Drawdown | -8.52% |
|
||||
| Sharpe | 0.47 |
|
||||
| Sortino | 1.91 |
|
||||
|
||||
### D1 — Twelve Data 2007–2026 (optimised weights)
|
||||
|
||||
| Metric | Value |
|
||||
|--------|-------|
|
||||
| Trades | 62 |
|
||||
| Win Rate | 40.3% |
|
||||
| Profit Factor | 1.27 |
|
||||
| Total Return | +10.51% |
|
||||
| Max Drawdown | -6.84% |
|
||||
| Sortino | 7.74 |
|
||||
|
||||
### Optimisation sweep delta (H4 2022–2026)
|
||||
|
||||
| | Baseline | Optimised |
|
||||
|--|----------|-----------|
|
||||
| Return | +7.12% | +21.30% |
|
||||
| Max DD | -9.05% | -8.52% |
|
||||
| Sharpe | 0.23 | 0.47 |
|
||||
| Win Rate | 36.3% | 38.1% |
|
||||
|
||||
---
|
||||
|
||||
## 13. Filter Catalogue
|
||||
|
||||
### Regime filter (`--regime-filter`)
|
||||
|
||||
Gates trade entry on ADX of a **higher** timeframe than the primary. Default: H4 ADX gate when trading H1. Redundant if regime TF == primary TF.
|
||||
|
||||
```
|
||||
regime TF ADX ≥ min_adx (default 25) → entry allowed
|
||||
```
|
||||
|
||||
Sensible defaults: H1→H4, H4→D1, D1→D1.
|
||||
|
||||
### Session filter (`--session-filter`)
|
||||
|
||||
Only trade during UTC 07:00–16:00 (London + NY overlap).
|
||||
2026 diagnostic: 07-15 UTC = 47% WR / +127 pips; off-hours = 0% WR / -571 pips.
|
||||
|
||||
### Auto-session toggle (`auto_session_filter`)
|
||||
|
||||
Reactive version of session filter. Activates automatically when mean P&L over last N closed trades drops below threshold:
|
||||
|
||||
```
|
||||
if mean(pnl_pct[-10:]) < -0.3%:
|
||||
session_filter = ON ← restrict to liquid hours
|
||||
else:
|
||||
session_filter = OFF ← all hours allowed
|
||||
```
|
||||
|
||||
### EMA200 distance gate
|
||||
|
||||
Always active. See §8.
|
||||
|
||||
### Signal grade gate (`min_trade_grade`)
|
||||
|
||||
Default: **B**. Only trades with grade ≥ B are entered. Raise to A for fewer, higher-quality trades.
|
||||
|
||||
### Min absolute score (`min_abs_score`)
|
||||
|
||||
Default: **0** (no extra gate). Set to e.g. 15 to skip weak directional scores.
|
||||
|
||||
---
|
||||
|
||||
## 14. Configuration Reference
|
||||
|
||||
All knobs live in `config.py → PlatformConfig`.
|
||||
|
||||
```python
|
||||
# Indicator parameters
|
||||
TechnicalConfig.ema_fast = 9
|
||||
TechnicalConfig.ema_medium = 21
|
||||
TechnicalConfig.ema_slow = 50
|
||||
TechnicalConfig.ema_trend = 200
|
||||
TechnicalConfig.rsi_period = 14
|
||||
TechnicalConfig.macd_fast / slow / signal = 12/26/9
|
||||
TechnicalConfig.bb_period = 20
|
||||
TechnicalConfig.atr_period = 14
|
||||
TechnicalConfig.stoch_k / d = 14/3
|
||||
TechnicalConfig.adx_period = 14
|
||||
TechnicalConfig.adx_strong_trend = 25.0
|
||||
TechnicalConfig.ichimoku_tenkan / kijun / senkou_b = 9/26/52
|
||||
TechnicalConfig.disabled_indicators = ["pivot", "candlestick"]
|
||||
|
||||
# Signal scoring
|
||||
SignalConfig.min_confidence = 55.0
|
||||
SignalConfig.atr_sl_multiplier = 1.5
|
||||
SignalConfig.default_rr_ratio = 2.0
|
||||
SignalConfig.default_risk_pct = 1.0
|
||||
|
||||
# Backtest
|
||||
BacktestConfig.initial_balance = 10_000
|
||||
BacktestConfig.risk_per_trade_pct = 1.0
|
||||
BacktestConfig.min_trade_grade = "B"
|
||||
BacktestConfig.regime_filter = False
|
||||
BacktestConfig.session_filter = False
|
||||
BacktestConfig.auto_session_filter = False
|
||||
BacktestConfig.cache_dir = ".cache"
|
||||
```
|
||||
@@ -1,42 +0,0 @@
|
||||
# Running ForexBot2 on Linux
|
||||
|
||||
## Install dependencies
|
||||
|
||||
```bash
|
||||
python -m venv .venv
|
||||
source .venv/bin/activate
|
||||
pip install -r requirements.txt
|
||||
```
|
||||
|
||||
## Telegram bot
|
||||
|
||||
```bash
|
||||
# Start (runs in background, logs to file)
|
||||
nohup .venv/bin/python telegram_bot.py >> bot.log 2>&1 &
|
||||
|
||||
# Follow logs
|
||||
tail -f bot.log
|
||||
|
||||
# Check it is running
|
||||
pgrep -a -f telegram_bot.py
|
||||
|
||||
# Stop
|
||||
pkill -f telegram_bot.py
|
||||
```
|
||||
|
||||
## Live signal (CLI)
|
||||
|
||||
```bash
|
||||
.venv/bin/python main.py # H1 summary
|
||||
.venv/bin/python main.py --detail # full indicator breakdown
|
||||
.venv/bin/python main.py --json # JSON output
|
||||
```
|
||||
|
||||
## Backtest
|
||||
|
||||
```bash
|
||||
.venv/bin/python main.py --backtest
|
||||
.venv/bin/python main.py --backtest --optimize
|
||||
```
|
||||
|
||||
> No GPU required — GPU code is only used by `backtest_gpu.py`.
|
||||
-667
@@ -1,667 +0,0 @@
|
||||
"""
|
||||
ForexBot2 — Walk-Forward Backtester + Parameter Optimizer
|
||||
==========================================================
|
||||
Replays Twelve Data USD/JPY history bar-by-bar through the same
|
||||
SignalAggregator that drives live mode. The same code path that
|
||||
trades USD/JPY in production is what produces these metrics — so
|
||||
the backtest validates the engine, not a parallel implementation.
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import copy
|
||||
import itertools
|
||||
import logging
|
||||
import multiprocessing as mp
|
||||
import os
|
||||
from collections import deque
|
||||
from concurrent.futures import ThreadPoolExecutor, as_completed
|
||||
from dataclasses import replace
|
||||
from typing import Deque, Dict, List, Optional, Tuple
|
||||
|
||||
import pandas as pd
|
||||
|
||||
from config import PlatformConfig, Timeframe, USDJPY, PIP_VALUE
|
||||
from data_provider import ForexDataProvider
|
||||
from metrics import BacktestResult, Trade, compute_metrics, trades_to_dataframe
|
||||
from signal_aggregator import SignalAggregator, SignalReport, TradeSetup
|
||||
|
||||
logger = logging.getLogger(__name__)
|
||||
|
||||
|
||||
# Period in minutes for each timeframe (used for close-time computation)
|
||||
TF_MINUTES: Dict[Timeframe, int] = {
|
||||
Timeframe.M1: 1,
|
||||
Timeframe.M5: 5,
|
||||
Timeframe.M15: 15,
|
||||
Timeframe.M30: 30,
|
||||
Timeframe.H1: 60,
|
||||
Timeframe.H4: 240,
|
||||
Timeframe.D1: 1440,
|
||||
Timeframe.W1: 10080,
|
||||
}
|
||||
|
||||
GRADE_ORDER = ["A+", "A", "B", "C", "D", "F"]
|
||||
|
||||
|
||||
def _grade_passes(grade: str, min_grade: str) -> bool:
|
||||
"""True if `grade` is at least as good as `min_grade`."""
|
||||
if grade not in GRADE_ORDER or min_grade not in GRADE_ORDER:
|
||||
return False
|
||||
return GRADE_ORDER.index(grade) <= GRADE_ORDER.index(min_grade)
|
||||
|
||||
|
||||
# ══════════════════════════════════════════════════════════════════
|
||||
# WALK-FORWARD BACKTESTER
|
||||
# ══════════════════════════════════════════════════════════════════
|
||||
|
||||
class WalkForwardBacktester:
|
||||
"""Single-pair, single-position event-driven backtester."""
|
||||
|
||||
def __init__(self, config: PlatformConfig):
|
||||
self.config = config
|
||||
self.aggregator = SignalAggregator(config)
|
||||
self.data_provider = self.aggregator.data_provider
|
||||
self._regime_adx: Optional[pd.Series] = None
|
||||
self._regime_tf: Optional[Timeframe] = None
|
||||
|
||||
# ── Public ─────────────────────────────────────────────────────
|
||||
|
||||
def load_data(self) -> Dict[Timeframe, pd.DataFrame]:
|
||||
"""Pull (or read from disk cache) all needed timeframes."""
|
||||
timeframes = [self.config.primary_timeframe] + list(
|
||||
self.config.confirmation_timeframes
|
||||
)
|
||||
bt = self.config.backtest
|
||||
if bt.regime_filter and bt.regime_filter_timeframe not in timeframes:
|
||||
timeframes.append(bt.regime_filter_timeframe)
|
||||
|
||||
dfs: Dict[Timeframe, pd.DataFrame] = {}
|
||||
for tf in timeframes:
|
||||
df = self.data_provider.fetch_full_history(
|
||||
symbol=self.config.symbol,
|
||||
timeframe=tf,
|
||||
max_bars=bt.max_bars,
|
||||
start_date=bt.start_date,
|
||||
end_date=bt.end_date,
|
||||
)
|
||||
df = self._apply_window(df)
|
||||
dfs[tf] = df
|
||||
return dfs
|
||||
|
||||
def _precompute_all_signals(
|
||||
self,
|
||||
dfs: Dict[Timeframe, pd.DataFrame],
|
||||
primary_df: pd.DataFrame,
|
||||
warmup: int,
|
||||
progress: bool = True,
|
||||
) -> Dict[int, "SignalReport"]:
|
||||
"""Pre-compute signals for every bar in parallel using all CPU threads.
|
||||
|
||||
TechnicalEngine.analyze() is stateless (pure function of the df slice),
|
||||
so concurrent calls are safe. NumPy/pandas C extensions release the GIL,
|
||||
giving real parallelism across threads without pickling overhead.
|
||||
Returns bar-index → SignalReport so the sequential state machine can
|
||||
do O(1) lookups instead of recomputing indicators per bar.
|
||||
"""
|
||||
primary_tf = self.config.primary_timeframe
|
||||
# Use all logical threads; cap at 2× physical cores to avoid over-subscription
|
||||
n_workers = min(os.cpu_count() or 4, 16)
|
||||
bar_indices = list(range(warmup, len(primary_df) - 1))
|
||||
|
||||
def _compute_bar(t: int) -> Tuple[int, "SignalReport"]:
|
||||
decision_time = self._bar_close_time(primary_df.index[t], primary_tf)
|
||||
sliced = self._slice_dfs(dfs, decision_time)
|
||||
report = self.aggregator.analyze_from_dataframes(
|
||||
symbol=self.config.symbol,
|
||||
dfs=sliced,
|
||||
use_news=False,
|
||||
)
|
||||
return t, report
|
||||
|
||||
precomputed: Dict[int, "SignalReport"] = {}
|
||||
pbar = None
|
||||
if progress:
|
||||
try:
|
||||
from tqdm import tqdm # type: ignore
|
||||
pbar = tqdm(
|
||||
total=len(bar_indices),
|
||||
desc=f"Signals ({n_workers} threads)",
|
||||
unit="bar",
|
||||
leave=False,
|
||||
)
|
||||
except ImportError:
|
||||
pass
|
||||
|
||||
with ThreadPoolExecutor(max_workers=n_workers) as executor:
|
||||
futures = {executor.submit(_compute_bar, t): t for t in bar_indices}
|
||||
for future in as_completed(futures):
|
||||
t, report = future.result()
|
||||
precomputed[t] = report
|
||||
if pbar:
|
||||
pbar.update(1)
|
||||
|
||||
if pbar:
|
||||
pbar.close()
|
||||
|
||||
return precomputed
|
||||
|
||||
def run(
|
||||
self,
|
||||
dfs: Optional[Dict[Timeframe, pd.DataFrame]] = None,
|
||||
progress: bool = True,
|
||||
) -> BacktestResult:
|
||||
if dfs is None:
|
||||
dfs = self.load_data()
|
||||
|
||||
self._precompute_regime_filter(dfs)
|
||||
|
||||
primary_tf = self.config.primary_timeframe
|
||||
primary_df = dfs.get(primary_tf)
|
||||
if primary_df is None or primary_df.empty:
|
||||
return BacktestResult(initial_balance=self.config.backtest.initial_balance)
|
||||
|
||||
# Warmup: enough bars for ema_trend & ichimoku_senkou_b on the primary TF
|
||||
warmup = max(
|
||||
self.config.technical.ema_trend,
|
||||
self.config.technical.ichimoku_senkou_b,
|
||||
200,
|
||||
) + 30
|
||||
|
||||
if len(primary_df) <= warmup + 1:
|
||||
logger.warning(
|
||||
f"Insufficient data: {len(primary_df)} primary bars, "
|
||||
f"need >{warmup}."
|
||||
)
|
||||
return BacktestResult(initial_balance=self.config.backtest.initial_balance)
|
||||
|
||||
# Pre-compute all bar signals in parallel before the sequential replay
|
||||
precomputed = self._precompute_all_signals(
|
||||
dfs, primary_df, warmup, progress=progress
|
||||
)
|
||||
|
||||
bt = self.config.backtest
|
||||
balance = bt.initial_balance
|
||||
equity_points: List[Tuple[pd.Timestamp, float]] = [
|
||||
(primary_df.index[warmup], balance)
|
||||
]
|
||||
trades: List[Trade] = []
|
||||
|
||||
# Auto-session feedback: rolling pnl_pct over last K closed trades.
|
||||
# When mean falls below threshold, the session filter activates;
|
||||
# once expectancy recovers, it releases.
|
||||
self._recent_pnl_pcts: Deque[float] = deque(
|
||||
maxlen=max(1, bt.auto_session_lookback_trades)
|
||||
)
|
||||
|
||||
open_position: Optional[Dict] = None
|
||||
|
||||
# Optional progress bar for the state-machine replay pass
|
||||
iterator = range(warmup, len(primary_df) - 1)
|
||||
if progress:
|
||||
try:
|
||||
from tqdm import tqdm # type: ignore
|
||||
iterator = tqdm(iterator, desc="Replaying bars", unit="bar", leave=False)
|
||||
except ImportError:
|
||||
pass
|
||||
|
||||
for t in iterator:
|
||||
decision_time = self._bar_close_time(primary_df.index[t], primary_tf)
|
||||
|
||||
# ── Step 1: manage any open position ───────────────────
|
||||
if open_position is not None:
|
||||
next_bar = primary_df.iloc[t]
|
||||
exit_info = self._check_exit(open_position, next_bar)
|
||||
if exit_info is not None:
|
||||
closed = self._close_trade(
|
||||
open_position, next_bar.name, exit_info, balance
|
||||
)
|
||||
trades.append(closed)
|
||||
balance += closed.pnl_currency
|
||||
equity_points.append((next_bar.name, balance))
|
||||
self._recent_pnl_pcts.append(float(closed.pnl_pct))
|
||||
open_position = None
|
||||
|
||||
# ── Step 2: maybe open a new position ──────────────────
|
||||
if open_position is None or not bt.one_position_only:
|
||||
report = precomputed[t]
|
||||
|
||||
if (
|
||||
_grade_passes(report.signal_quality, bt.min_trade_grade)
|
||||
and abs(float(report.final_score)) >= bt.min_abs_score
|
||||
and report.trade_setup is not None
|
||||
and t + 1 < len(primary_df)
|
||||
and self._regime_passes(decision_time)
|
||||
and self._session_passes(decision_time)
|
||||
):
|
||||
next_bar = primary_df.iloc[t + 1]
|
||||
open_position = self._open_trade(report, next_bar, balance)
|
||||
|
||||
# ── Force-close any still-open position at end of data ────
|
||||
if open_position is not None:
|
||||
last_bar = primary_df.iloc[-1]
|
||||
closed = self._close_trade(
|
||||
open_position,
|
||||
last_bar.name,
|
||||
{"price": float(last_bar["Close"]), "reason": "end_of_data"},
|
||||
balance,
|
||||
)
|
||||
trades.append(closed)
|
||||
balance += closed.pnl_currency
|
||||
equity_points.append((last_bar.name, balance))
|
||||
|
||||
# Build equity curve
|
||||
eq_index = [p[0] for p in equity_points]
|
||||
eq_values = [p[1] for p in equity_points]
|
||||
equity_curve = pd.Series(eq_values, index=pd.DatetimeIndex(eq_index))
|
||||
|
||||
elapsed_days: Optional[float] = None
|
||||
if len(equity_curve) >= 2:
|
||||
elapsed_days = (equity_curve.index[-1] - equity_curve.index[0]).total_seconds() / 86400
|
||||
|
||||
metrics = compute_metrics(
|
||||
trades, equity_curve, bt.initial_balance, elapsed_days=elapsed_days
|
||||
)
|
||||
|
||||
return BacktestResult(
|
||||
trades=trades,
|
||||
equity_curve=equity_curve,
|
||||
initial_balance=bt.initial_balance,
|
||||
final_balance=balance,
|
||||
metrics=metrics,
|
||||
)
|
||||
|
||||
# ── Private: regime gate ───────────────────────────────────────
|
||||
|
||||
def _precompute_regime_filter(self, dfs: Dict[Timeframe, pd.DataFrame]) -> None:
|
||||
"""Pre-compute ADX on the regime-filter TF; cached for the whole run."""
|
||||
bt = self.config.backtest
|
||||
if not bt.regime_filter:
|
||||
self._regime_adx = None
|
||||
self._regime_tf = None
|
||||
return
|
||||
|
||||
tf = bt.regime_filter_timeframe
|
||||
df = dfs.get(tf)
|
||||
if df is None or df.empty:
|
||||
logger.warning(
|
||||
f"Regime filter requested ({tf.value}) but no data available; "
|
||||
f"all signals will be skipped."
|
||||
)
|
||||
self._regime_adx = pd.Series(dtype=float)
|
||||
self._regime_tf = tf
|
||||
return
|
||||
|
||||
self._regime_tf = tf
|
||||
self._regime_adx = self._compute_adx_series(df, self.config.technical.adx_period)
|
||||
|
||||
def _regime_passes(self, decision_time: pd.Timestamp) -> bool:
|
||||
bt = self.config.backtest
|
||||
if not bt.regime_filter:
|
||||
return True
|
||||
if self._regime_adx is None or self._regime_adx.empty:
|
||||
return False
|
||||
period_min = TF_MINUTES[self._regime_tf]
|
||||
close_times = self._regime_adx.index + pd.Timedelta(minutes=period_min)
|
||||
valid = self._regime_adx[close_times <= decision_time]
|
||||
if valid.empty:
|
||||
return False
|
||||
last = valid.iloc[-1]
|
||||
if pd.isna(last):
|
||||
return False
|
||||
return float(last) >= bt.regime_filter_min_adx
|
||||
|
||||
def _session_passes(self, decision_time: pd.Timestamp) -> bool:
|
||||
"""UTC hour gate — decision_time = entry bar's open time.
|
||||
|
||||
Filter is active when EITHER manual `session_filter` is on, OR
|
||||
`auto_session_filter` is on and the rolling mean pnl_pct over the
|
||||
last N closed trades is below threshold.
|
||||
"""
|
||||
bt = self.config.backtest
|
||||
active = bt.session_filter
|
||||
if not active and bt.auto_session_filter:
|
||||
recent = getattr(self, "_recent_pnl_pcts", None)
|
||||
if recent is not None and len(recent) >= bt.auto_session_lookback_trades:
|
||||
mean_pnl = sum(recent) / len(recent)
|
||||
active = mean_pnl < bt.auto_session_threshold_pct
|
||||
if not active:
|
||||
return True
|
||||
h = int(decision_time.hour)
|
||||
return bt.session_start_hour <= h < bt.session_end_hour
|
||||
|
||||
@staticmethod
|
||||
def _compute_adx_series(df: pd.DataFrame, period: int = 14) -> pd.Series:
|
||||
"""Wilder's ADX series — same algorithm as engine._compute_adx."""
|
||||
import numpy as np
|
||||
high, low, close = df["High"], df["Low"], df["Close"]
|
||||
tr = pd.concat([
|
||||
high - low,
|
||||
(high - close.shift()).abs(),
|
||||
(low - close.shift()).abs(),
|
||||
], axis=1).max(axis=1)
|
||||
plus_dm = high.diff()
|
||||
minus_dm = -low.diff()
|
||||
plus_dm = plus_dm.where((plus_dm > minus_dm) & (plus_dm > 0), 0.0)
|
||||
minus_dm = minus_dm.where((minus_dm > plus_dm) & (minus_dm > 0), 0.0)
|
||||
atr_s = tr.ewm(alpha=1/period, min_periods=period).mean()
|
||||
plus_di = 100 * (plus_dm.ewm(alpha=1/period, min_periods=period).mean() / atr_s)
|
||||
minus_di = 100 * (minus_dm.ewm(alpha=1/period, min_periods=period).mean() / atr_s)
|
||||
dx = 100 * (plus_di - minus_di).abs() / (plus_di + minus_di).replace(0, np.nan)
|
||||
return dx.ewm(alpha=1/period, min_periods=period).mean()
|
||||
|
||||
# ── Private: data slicing ──────────────────────────────────────
|
||||
|
||||
def _apply_window(self, df: pd.DataFrame) -> pd.DataFrame:
|
||||
bt = self.config.backtest
|
||||
if bt.start_date:
|
||||
start = pd.Timestamp(bt.start_date, tz="UTC")
|
||||
df = df[df.index >= start]
|
||||
if bt.end_date:
|
||||
end = pd.Timestamp(bt.end_date, tz="UTC")
|
||||
df = df[df.index <= end]
|
||||
return df
|
||||
|
||||
@staticmethod
|
||||
def _bar_close_time(open_time: pd.Timestamp, tf: Timeframe) -> pd.Timestamp:
|
||||
return open_time + pd.Timedelta(minutes=TF_MINUTES[tf])
|
||||
|
||||
def _slice_dfs(
|
||||
self, dfs: Dict[Timeframe, pd.DataFrame], decision_time: pd.Timestamp
|
||||
) -> Dict[Timeframe, pd.DataFrame]:
|
||||
"""Return only bars whose close time ≤ decision_time (no lookahead)."""
|
||||
sliced: Dict[Timeframe, pd.DataFrame] = {}
|
||||
for tf, df in dfs.items():
|
||||
close_times = df.index + pd.Timedelta(minutes=TF_MINUTES[tf])
|
||||
sliced[tf] = df[close_times <= decision_time]
|
||||
return sliced
|
||||
|
||||
# ── Private: trade execution ───────────────────────────────────
|
||||
|
||||
def _open_trade(
|
||||
self, report: SignalReport, next_bar: pd.Series, balance: float
|
||||
) -> Dict:
|
||||
"""Open at next bar's open, applying spread + slippage."""
|
||||
ts: TradeSetup = report.trade_setup # type: ignore[assignment]
|
||||
bt = self.config.backtest
|
||||
|
||||
raw_entry = float(next_bar["Open"])
|
||||
|
||||
# Sanity-test inversion: flip direction and mirror SL/TP around entry
|
||||
direction = ts.direction
|
||||
sl = ts.stop_loss
|
||||
tp = ts.take_profit_1
|
||||
if bt.invert_signal:
|
||||
direction = "SELL" if direction == "BUY" else "BUY"
|
||||
sl, tp = (2 * raw_entry - sl), (2 * raw_entry - tp)
|
||||
|
||||
# Cost in price units = (half-spread + slippage) × pip_value
|
||||
half_spread_pips = 0.5 if bt.use_spread_cost else 0.0
|
||||
cost_pips = half_spread_pips + bt.slippage_pips
|
||||
cost_price = cost_pips * PIP_VALUE
|
||||
|
||||
if direction == "BUY":
|
||||
entry_price = raw_entry + cost_price # buy higher (worse)
|
||||
else:
|
||||
entry_price = raw_entry - cost_price # sell lower (worse)
|
||||
|
||||
# Position sizing — risk a fixed % of current balance
|
||||
risk_pct = bt.risk_per_trade_pct / 100.0
|
||||
risk_amount = balance * risk_pct
|
||||
risk_pips = max(ts.risk_pips, 1e-9)
|
||||
pip_currency_value = risk_amount / risk_pips # $ per pip
|
||||
|
||||
# ── Diagnostic snapshot at entry ─────────────────────────
|
||||
analysis = report.primary_analysis
|
||||
adx_val = 0.0
|
||||
atr_pct = 0.0
|
||||
rsi_val = 50.0
|
||||
ema_dist = 0.0
|
||||
if analysis is not None:
|
||||
adx_sig = analysis.indicators.get("adx")
|
||||
if adx_sig is not None:
|
||||
adx_val = float(adx_sig.value)
|
||||
atr_pct = float(analysis.key_levels.get("atr_pct", 0.0))
|
||||
rsi_sig = analysis.indicators.get("rsi")
|
||||
if rsi_sig is not None:
|
||||
rsi_val = float(rsi_sig.value)
|
||||
ema_trend = analysis.key_levels.get("ema_trend")
|
||||
if ema_trend and ema_trend > 0:
|
||||
ema_dist = (raw_entry - ema_trend) / ema_trend * 100.0
|
||||
|
||||
# Per-timeframe scores from MTF map (keys are tf.value strings)
|
||||
m15_s = float(report.mtf_scores.get("15min", 0.0))
|
||||
h1_s = float(report.mtf_scores.get("1h", 0.0))
|
||||
h4_s = float(report.mtf_scores.get("4h", 0.0))
|
||||
|
||||
entry_ts = next_bar.name
|
||||
entry_hour = int(entry_ts.hour) if hasattr(entry_ts, "hour") else -1
|
||||
entry_dow = int(entry_ts.dayofweek) if hasattr(entry_ts, "dayofweek") else -1
|
||||
|
||||
return {
|
||||
"direction": direction,
|
||||
"entry_time": next_bar.name,
|
||||
"entry_price": entry_price,
|
||||
"stop_loss": sl,
|
||||
"take_profit": tp,
|
||||
"risk_pips": risk_pips,
|
||||
"pip_currency_value": pip_currency_value,
|
||||
"balance_at_open": balance,
|
||||
"grade": report.signal_quality,
|
||||
"score_at_entry": float(report.final_score),
|
||||
"confidence_at_entry": float(report.final_confidence),
|
||||
"bar_index_open": next_bar.name,
|
||||
# Diagnostics
|
||||
"entry_hour": entry_hour,
|
||||
"entry_dow": entry_dow,
|
||||
"adx_at_entry": adx_val,
|
||||
"atr_pct_at_entry": atr_pct,
|
||||
"market_regime": report.market_regime or "",
|
||||
"volatility_regime": report.volatility_regime or "",
|
||||
"mtf_agreement": bool(report.mtf_agreement),
|
||||
"m15_score": m15_s,
|
||||
"h1_score": h1_s,
|
||||
"h4_score": h4_s,
|
||||
"rsi_at_entry": rsi_val,
|
||||
"ema_distance_pct": ema_dist,
|
||||
}
|
||||
|
||||
def _check_exit(
|
||||
self, position: Dict, bar: pd.Series
|
||||
) -> Optional[Dict]:
|
||||
"""Return {price, reason} if SL or TP hit, else None.
|
||||
|
||||
On the entry bar (bar with same timestamp as position's entry_time),
|
||||
we already filled at the open — only check subsequent bars' OHLC.
|
||||
"""
|
||||
if bar.name == position["bar_index_open"]:
|
||||
return None # entry bar — no exit check
|
||||
|
||||
bt = self.config.backtest
|
||||
high = float(bar["High"])
|
||||
low = float(bar["Low"])
|
||||
|
||||
sl = position["stop_loss"]
|
||||
tp = position["take_profit"]
|
||||
slip = bt.slippage_pips * PIP_VALUE
|
||||
half_spread = (0.5 if bt.use_spread_cost else 0.0) * PIP_VALUE
|
||||
|
||||
if position["direction"] == "BUY":
|
||||
sl_hit = low <= sl
|
||||
tp_hit = high >= tp
|
||||
if sl_hit and tp_hit:
|
||||
# Conservative: SL first
|
||||
return {"price": sl - slip - half_spread, "reason": "sl"} if bt.sl_first_on_tie else \
|
||||
{"price": tp - slip - half_spread, "reason": "tp"}
|
||||
if sl_hit:
|
||||
return {"price": sl - slip - half_spread, "reason": "sl"}
|
||||
if tp_hit:
|
||||
return {"price": tp - slip - half_spread, "reason": "tp"}
|
||||
else: # SELL
|
||||
sl_hit = high >= sl
|
||||
tp_hit = low <= tp
|
||||
if sl_hit and tp_hit:
|
||||
return {"price": sl + slip + half_spread, "reason": "sl"} if bt.sl_first_on_tie else \
|
||||
{"price": tp + slip + half_spread, "reason": "tp"}
|
||||
if sl_hit:
|
||||
return {"price": sl + slip + half_spread, "reason": "sl"}
|
||||
if tp_hit:
|
||||
return {"price": tp + slip + half_spread, "reason": "tp"}
|
||||
return None
|
||||
|
||||
def _close_trade(
|
||||
self, position: Dict, exit_time: pd.Timestamp, exit_info: Dict, balance: float
|
||||
) -> Trade:
|
||||
exit_price = exit_info["price"]
|
||||
reason = exit_info["reason"]
|
||||
|
||||
if position["direction"] == "BUY":
|
||||
pnl_pips = (exit_price - position["entry_price"]) / PIP_VALUE
|
||||
else:
|
||||
pnl_pips = (position["entry_price"] - exit_price) / PIP_VALUE
|
||||
|
||||
pnl_currency = pnl_pips * position["pip_currency_value"]
|
||||
pnl_pct = pnl_currency / position["balance_at_open"] * 100
|
||||
|
||||
# Duration in primary-TF bars (timestamp diff / period)
|
||||
period_min = TF_MINUTES[self.config.primary_timeframe]
|
||||
duration_minutes = (exit_time - position["entry_time"]).total_seconds() / 60
|
||||
duration_bars = int(round(duration_minutes / period_min))
|
||||
|
||||
return Trade(
|
||||
entry_time=position["entry_time"],
|
||||
exit_time=exit_time,
|
||||
direction=position["direction"],
|
||||
entry_price=position["entry_price"],
|
||||
exit_price=exit_price,
|
||||
stop_loss=position["stop_loss"],
|
||||
take_profit=position["take_profit"],
|
||||
exit_reason=reason,
|
||||
risk_pips=position["risk_pips"],
|
||||
pnl_pips=pnl_pips,
|
||||
pnl_pct=pnl_pct,
|
||||
pnl_currency=pnl_currency,
|
||||
duration_bars=duration_bars,
|
||||
grade=position["grade"],
|
||||
score_at_entry=position["score_at_entry"],
|
||||
confidence_at_entry=position["confidence_at_entry"],
|
||||
entry_hour=position.get("entry_hour", -1),
|
||||
entry_dow=position.get("entry_dow", -1),
|
||||
adx_at_entry=position.get("adx_at_entry", 0.0),
|
||||
atr_pct_at_entry=position.get("atr_pct_at_entry", 0.0),
|
||||
market_regime=position.get("market_regime", ""),
|
||||
volatility_regime=position.get("volatility_regime", ""),
|
||||
mtf_agreement=position.get("mtf_agreement", False),
|
||||
m15_score=position.get("m15_score", 0.0),
|
||||
h1_score=position.get("h1_score", 0.0),
|
||||
h4_score=position.get("h4_score", 0.0),
|
||||
rsi_at_entry=position.get("rsi_at_entry", 50.0),
|
||||
ema_distance_pct=position.get("ema_distance_pct", 0.0),
|
||||
)
|
||||
|
||||
|
||||
# ══════════════════════════════════════════════════════════════════
|
||||
# PARAMETER OPTIMIZER (PARALLEL)
|
||||
# ══════════════════════════════════════════════════════════════════
|
||||
|
||||
def _apply_overrides(config: PlatformConfig, overrides: Dict[str, float]) -> PlatformConfig:
|
||||
"""Return a deep-copied config with parameter overrides applied."""
|
||||
new_config = copy.deepcopy(config)
|
||||
|
||||
if "atr_sl_multiplier" in overrides:
|
||||
new_config.signals.atr_sl_multiplier = float(overrides["atr_sl_multiplier"])
|
||||
if "default_rr_ratio" in overrides:
|
||||
new_config.signals.default_rr_ratio = float(overrides["default_rr_ratio"])
|
||||
if "min_trade_grade" in overrides:
|
||||
new_config.backtest.min_trade_grade = str(overrides["min_trade_grade"])
|
||||
if "min_abs_score" in overrides:
|
||||
new_config.backtest.min_abs_score = float(overrides["min_abs_score"])
|
||||
if "invert_signal" in overrides:
|
||||
new_config.backtest.invert_signal = bool(overrides["invert_signal"])
|
||||
if "regime_filter" in overrides:
|
||||
new_config.backtest.regime_filter = bool(overrides["regime_filter"])
|
||||
if "regime_filter_min_adx" in overrides:
|
||||
new_config.backtest.regime_filter_min_adx = float(overrides["regime_filter_min_adx"])
|
||||
if "session_filter" in overrides:
|
||||
new_config.backtest.session_filter = bool(overrides["session_filter"])
|
||||
if "session_start_hour" in overrides:
|
||||
new_config.backtest.session_start_hour = int(overrides["session_start_hour"])
|
||||
if "session_end_hour" in overrides:
|
||||
new_config.backtest.session_end_hour = int(overrides["session_end_hour"])
|
||||
if "auto_session_filter" in overrides:
|
||||
new_config.backtest.auto_session_filter = bool(overrides["auto_session_filter"])
|
||||
if "auto_session_lookback_trades" in overrides:
|
||||
new_config.backtest.auto_session_lookback_trades = int(overrides["auto_session_lookback_trades"])
|
||||
if "auto_session_threshold_pct" in overrides:
|
||||
new_config.backtest.auto_session_threshold_pct = float(overrides["auto_session_threshold_pct"])
|
||||
if "risk_per_trade_pct" in overrides:
|
||||
new_config.backtest.risk_per_trade_pct = float(overrides["risk_per_trade_pct"])
|
||||
if "disable_indicator" in overrides:
|
||||
v = overrides["disable_indicator"]
|
||||
new_config.technical.disabled_indicators = [str(v)] if v else []
|
||||
|
||||
return new_config
|
||||
|
||||
|
||||
def _run_single_combo(args: Tuple[PlatformConfig, Dict[str, float]]) -> BacktestResult:
|
||||
"""Worker function — must be top-level for multiprocessing.Pool."""
|
||||
base_config, overrides = args
|
||||
config = _apply_overrides(base_config, overrides)
|
||||
backtester = WalkForwardBacktester(config)
|
||||
# Workers reuse the disk parquet cache, so no Twelve Data fetch needed
|
||||
result = backtester.run(progress=False)
|
||||
result.params = overrides
|
||||
return result
|
||||
|
||||
|
||||
class Optimizer:
|
||||
"""Embarrassingly parallel grid search over BacktestConfig.param_grid."""
|
||||
|
||||
def __init__(self, config: PlatformConfig):
|
||||
self.config = config
|
||||
|
||||
def run_grid(
|
||||
self,
|
||||
param_grid: Optional[Dict[str, List[float]]] = None,
|
||||
prewarm_cache: bool = True,
|
||||
) -> List[BacktestResult]:
|
||||
if param_grid is None:
|
||||
param_grid = self.config.backtest.param_grid
|
||||
|
||||
# Prewarm the disk cache once before fanning out — avoids each worker
|
||||
# re-fetching from Twelve Data.
|
||||
if prewarm_cache:
|
||||
logger.info("Prewarming disk cache for backtest data...")
|
||||
warmer = WalkForwardBacktester(self.config)
|
||||
warmer.load_data()
|
||||
|
||||
keys = list(param_grid.keys())
|
||||
value_lists = [param_grid[k] for k in keys]
|
||||
combos = [
|
||||
dict(zip(keys, vals))
|
||||
for vals in itertools.product(*value_lists)
|
||||
]
|
||||
|
||||
n_workers = min(
|
||||
self.config.backtest.parallel_workers,
|
||||
len(combos),
|
||||
os.cpu_count() or 1,
|
||||
)
|
||||
logger.info(
|
||||
f"Optimizer: {len(combos)} combos × {n_workers} workers"
|
||||
)
|
||||
|
||||
tasks = [(self.config, c) for c in combos]
|
||||
|
||||
if n_workers <= 1:
|
||||
results = [_run_single_combo(t) for t in tasks]
|
||||
else:
|
||||
with mp.Pool(processes=n_workers) as pool:
|
||||
results = pool.map(_run_single_combo, tasks)
|
||||
|
||||
# Sort by total return descending
|
||||
results.sort(
|
||||
key=lambda r: r.metrics.get("total_return_pct", -float("inf")),
|
||||
reverse=True,
|
||||
)
|
||||
return results
|
||||
-1323
File diff suppressed because it is too large
Load Diff
@@ -1,15 +1,11 @@
|
||||
"""
|
||||
ForexBot2 — Configuration (USD/JPY focus)
|
||||
==========================================
|
||||
Single-pair signal & backtest configuration.
|
||||
All knobs that govern indicator behavior, signal scoring, news handling,
|
||||
and the walk-forward backtester live here.
|
||||
"""
|
||||
|
||||
import os
|
||||
from dataclasses import dataclass, field
|
||||
from enum import Enum
|
||||
from typing import Dict, List, Optional
|
||||
from typing import Dict, List
|
||||
|
||||
|
||||
class Timeframe(Enum):
|
||||
@@ -85,23 +81,7 @@ class TechnicalConfig:
|
||||
pivot_lookback: int = 1
|
||||
divergence_lookback: int = 14
|
||||
|
||||
# Ablation control — names listed here are skipped during analyze().
|
||||
# Use lowercase keys: ema_trend, rsi, macd, bollinger, stochastic,
|
||||
# adx, ichimoku, divergence, candlestick, pivot, atr, volume.
|
||||
#
|
||||
# Defaults disable indicators that empirically hurt P&L on USD/JPY across
|
||||
# 2024-trend / 2023-chop / 2026-recent ablation sweeps (see project memory):
|
||||
# - pivot: net +9.5% to disable across 3 windows
|
||||
# - candlestick: net +3.4% to disable
|
||||
disabled_indicators: List[str] = field(
|
||||
default_factory=lambda: ["pivot", "candlestick"]
|
||||
)
|
||||
|
||||
|
||||
ALL_INDICATORS: List[str] = [
|
||||
"ema_trend", "rsi", "macd", "bollinger", "stochastic",
|
||||
"adx", "ichimoku", "divergence", "candlestick", "pivot",
|
||||
]
|
||||
disabled_indicators: List[str] = field(default_factory=list)
|
||||
|
||||
|
||||
@dataclass
|
||||
@@ -141,67 +121,15 @@ class NewsConfig:
|
||||
|
||||
|
||||
@dataclass
|
||||
class BacktestConfig:
|
||||
"""Walk-forward backtest configuration."""
|
||||
# Data window — None means "all available history"
|
||||
start_date: Optional[str] = None # "YYYY-MM-DD"
|
||||
end_date: Optional[str] = None # "YYYY-MM-DD"
|
||||
max_bars: int = 5000 # Twelve Data ceiling per request
|
||||
|
||||
# Account & sizing
|
||||
initial_balance: float = 10_000.0
|
||||
risk_per_trade_pct: float = 1.0 # % of equity risked per trade
|
||||
|
||||
# Execution realism
|
||||
use_spread_cost: bool = True
|
||||
slippage_pips: float = 0.5 # added to spread on entry & exit
|
||||
one_position_only: bool = True
|
||||
sl_first_on_tie: bool = True # if SL & TP both inside bar, SL hits first
|
||||
|
||||
# Signal gates
|
||||
min_trade_grade: str = "B" # "A+", "A", "B", "C", "D", "F"
|
||||
min_abs_score: float = 0.0 # |final_score| must exceed this to enter
|
||||
|
||||
# Sanity-test knob — flip BUY↔SELL to see if the engine has reversed edge
|
||||
invert_signal: bool = False
|
||||
|
||||
# Regime gate — only trade when a higher-TF trend is confirmed
|
||||
class FilterConfig:
|
||||
"""Regime and session filters for live signals."""
|
||||
regime_filter: bool = False
|
||||
regime_filter_timeframe: Timeframe = Timeframe.H4
|
||||
regime_filter_min_adx: float = 25.0
|
||||
|
||||
# Session gate — only trade during a specified UTC hour window.
|
||||
# 2026 diagnostic backtest showed 07-15 UTC (London + NY) had
|
||||
# 47% WR / +127 pips while off-hours were 0% WR / -571 pips.
|
||||
session_filter: bool = False
|
||||
session_start_hour: int = 7 # UTC, inclusive
|
||||
session_end_hour: int = 16 # UTC, exclusive — so trades fire on 07..15
|
||||
|
||||
# Auto-toggle: enable session filter ONLY when recent realised trades
|
||||
# show adverse expectancy (mean pnl_pct over the last N closed trades
|
||||
# below threshold). Releases automatically once expectancy recovers.
|
||||
# Designed so trending regimes keep all hours, while adverse/chop
|
||||
# regimes auto-restrict to liquid sessions.
|
||||
auto_session_filter: bool = False
|
||||
auto_session_lookback_trades: int = 10
|
||||
auto_session_threshold_pct: float = -0.3
|
||||
|
||||
# Parallelism (used by Optimizer; single-run backtest is sequential)
|
||||
parallel_workers: int = field(default_factory=lambda: os.cpu_count() or 4)
|
||||
|
||||
# Optimizer
|
||||
optimize: bool = False
|
||||
param_grid: Dict[str, list] = field(default_factory=lambda: {
|
||||
# 2 × 2 × 3 × 2 × 2 = 48 combos
|
||||
"atr_sl_multiplier": [1.5, 2.5],
|
||||
"default_rr_ratio": [1.5, 2.5],
|
||||
"min_abs_score": [15.0, 35.0, 60.0],
|
||||
"min_trade_grade": ["B", "A"],
|
||||
"invert_signal": [False, True],
|
||||
})
|
||||
|
||||
# Cache directory for fetched OHLCV (parquet)
|
||||
cache_dir: str = ".cache"
|
||||
session_start_hour: int = 7
|
||||
session_end_hour: int = 16
|
||||
|
||||
|
||||
@dataclass
|
||||
@@ -210,7 +138,7 @@ class PlatformConfig:
|
||||
technical: TechnicalConfig = field(default_factory=TechnicalConfig)
|
||||
signals: SignalConfig = field(default_factory=SignalConfig)
|
||||
news: NewsConfig = field(default_factory=NewsConfig)
|
||||
backtest: BacktestConfig = field(default_factory=BacktestConfig)
|
||||
backtest: FilterConfig = field(default_factory=FilterConfig)
|
||||
|
||||
symbol: str = USDJPY
|
||||
display_name: str = DISPLAY
|
||||
@@ -221,12 +149,10 @@ class PlatformConfig:
|
||||
default_factory=lambda: [Timeframe.M15, Timeframe.H4]
|
||||
)
|
||||
|
||||
data_lookback_days: int = 60 # enough for 200 EMA on H1
|
||||
data_lookback_days: int = 60
|
||||
|
||||
twelve_data_api_key: str = "f563b22ffdfd4924955decbe1a8c8701"
|
||||
|
||||
refresh_interval: int = 300 # 5 min cache TTL
|
||||
refresh_interval: int = 300
|
||||
|
||||
log_level: str = "INFO"
|
||||
log_signals_to_file: bool = True
|
||||
signal_log_path: str = "signals.log"
|
||||
|
||||
@@ -2,9 +2,6 @@
|
||||
ForexBot2 — Data Provider (Twelve Data SDK, USD/JPY only)
|
||||
==========================================================
|
||||
Fetches and validates OHLCV via the official `twelvedata` Python SDK.
|
||||
No synthetic fallback — if the API fails the caller learns about it.
|
||||
Adds a parquet on-disk cache so the backtest worker pool reuses one
|
||||
download instead of hammering Twelve Data during a parameter sweep.
|
||||
"""
|
||||
|
||||
import logging
|
||||
@@ -29,14 +26,6 @@ class DataIntegrityError(Exception):
|
||||
class ForexDataProvider:
|
||||
"""Twelve Data SDK client + on-disk parquet cache."""
|
||||
|
||||
# Local MetaTrader CSV exports — tab-delimited, 7 columns (includes Spread)
|
||||
LOCAL_CSV_MAP = {
|
||||
Timeframe.M15: "USDJPY_M15.csv",
|
||||
Timeframe.H1: "USDJPY_H1.csv",
|
||||
Timeframe.H4: "USDJPY_H4.csv",
|
||||
Timeframe.D1: "USDJPY_D1.csv",
|
||||
}
|
||||
|
||||
TD_INTERVALS = {
|
||||
Timeframe.M1: "1min",
|
||||
Timeframe.M5: "5min",
|
||||
@@ -48,18 +37,6 @@ class ForexDataProvider:
|
||||
Timeframe.W1: "1week",
|
||||
}
|
||||
|
||||
BARS_PER_DAY = {
|
||||
Timeframe.M1: 1440,
|
||||
Timeframe.M5: 288,
|
||||
Timeframe.M15: 96,
|
||||
Timeframe.M30: 48,
|
||||
Timeframe.H1: 24,
|
||||
Timeframe.H4: 6,
|
||||
Timeframe.D1: 1,
|
||||
Timeframe.W1: 0.143,
|
||||
}
|
||||
|
||||
# Free plan: 8 credits / minute
|
||||
RATE_LIMIT = 8
|
||||
RATE_WINDOW = 60.0
|
||||
|
||||
@@ -116,100 +93,6 @@ class ForexDataProvider:
|
||||
self._cache_timestamps[cache_key] = now
|
||||
return df
|
||||
|
||||
def fetch_full_history(
|
||||
self,
|
||||
symbol: str = USDJPY,
|
||||
timeframe: Timeframe = Timeframe.H1,
|
||||
max_bars: int = 5000,
|
||||
use_disk_cache: bool = True,
|
||||
start_date: Optional[str] = None,
|
||||
end_date: Optional[str] = None,
|
||||
) -> pd.DataFrame:
|
||||
"""
|
||||
Pull the largest window Twelve Data allows in one call.
|
||||
On-disk parquet cache prevents re-download across worker subprocesses
|
||||
during an optimizer sweep.
|
||||
"""
|
||||
cache_path = self._disk_cache_path(symbol, timeframe, start_date, end_date)
|
||||
|
||||
if use_disk_cache and os.path.exists(cache_path):
|
||||
try:
|
||||
df = pd.read_parquet(cache_path)
|
||||
if df.index.tz is None:
|
||||
df.index = df.index.tz_localize("UTC")
|
||||
logger.info(
|
||||
f"Loaded {len(df)} bars from disk cache: {cache_path}"
|
||||
)
|
||||
return df
|
||||
except Exception as e:
|
||||
logger.warning(f"Disk cache read failed ({e}); refetching.")
|
||||
|
||||
outputsize = min(max_bars, 5000)
|
||||
df = self._fetch_twelve_data(
|
||||
symbol, timeframe, outputsize,
|
||||
start_date=start_date, end_date=end_date,
|
||||
)
|
||||
df = self._validate_and_clean(df, symbol, timeframe)
|
||||
|
||||
if use_disk_cache:
|
||||
os.makedirs(os.path.dirname(cache_path), exist_ok=True)
|
||||
try:
|
||||
df.to_parquet(cache_path)
|
||||
logger.info(f"Cached {len(df)} bars → {cache_path}")
|
||||
except Exception as e:
|
||||
logger.warning(f"Disk cache write failed: {e}")
|
||||
|
||||
return df
|
||||
|
||||
def load_local_csv(
|
||||
self,
|
||||
timeframe: Timeframe,
|
||||
data_dir: str = "market_data",
|
||||
start_date: Optional[str] = None,
|
||||
end_date: Optional[str] = None,
|
||||
) -> Optional[pd.DataFrame]:
|
||||
"""
|
||||
Load OHLCV from a local MetaTrader CSV export.
|
||||
Format: tab-separated, columns Time/Open/High/Low/Close/Volume/Spread.
|
||||
Returns None silently if the file doesn't exist for this timeframe.
|
||||
"""
|
||||
fname = self.LOCAL_CSV_MAP.get(timeframe)
|
||||
if fname is None:
|
||||
logger.info(f"No local CSV mapping for {timeframe.value} — skipping")
|
||||
return None
|
||||
|
||||
path = os.path.join(data_dir, fname)
|
||||
if not os.path.exists(path):
|
||||
logger.warning(f"Local CSV not found: {path}")
|
||||
return None
|
||||
|
||||
df = pd.read_csv(
|
||||
path, sep="\t",
|
||||
names=["Time", "Open", "High", "Low", "Close", "Volume", "Spread"],
|
||||
header=0,
|
||||
parse_dates=["Time"],
|
||||
)
|
||||
df = df.set_index("Time")
|
||||
df = df.drop(columns=["Spread"], errors="ignore")
|
||||
for col in ("Open", "High", "Low", "Close", "Volume"):
|
||||
if col in df.columns:
|
||||
df[col] = pd.to_numeric(df[col], errors="coerce")
|
||||
if "Volume" not in df.columns:
|
||||
df["Volume"] = 0.0
|
||||
|
||||
if df.index.tz is None:
|
||||
df.index = df.index.tz_localize("UTC")
|
||||
df.index.name = "Datetime"
|
||||
|
||||
if start_date:
|
||||
df = df[df.index >= pd.Timestamp(start_date, tz="UTC")]
|
||||
if end_date:
|
||||
df = df[df.index <= pd.Timestamp(end_date, tz="UTC")]
|
||||
|
||||
df = self._validate_and_clean(df, USDJPY, timeframe)
|
||||
logger.info(f"Loaded {len(df):,} bars from local CSV: {path}")
|
||||
return df
|
||||
|
||||
def get_latest_price(self, symbol: str = USDJPY) -> Optional[float]:
|
||||
try:
|
||||
df = self.fetch_ohlcv(symbol, Timeframe.M5, lookback_days=1)
|
||||
@@ -240,22 +123,6 @@ class ForexDataProvider:
|
||||
|
||||
# ── Private ────────────────────────────────────────────────────
|
||||
|
||||
def _disk_cache_path(
|
||||
self,
|
||||
symbol: str,
|
||||
timeframe: Timeframe,
|
||||
start_date: Optional[str] = None,
|
||||
end_date: Optional[str] = None,
|
||||
) -> str:
|
||||
clean = symbol.replace("=X", "").lower()
|
||||
suffix = ""
|
||||
if start_date or end_date:
|
||||
suffix = f"_{start_date or 'min'}_{end_date or 'max'}"
|
||||
return os.path.join(
|
||||
self.config.backtest.cache_dir,
|
||||
f"{clean}_{timeframe.value}{suffix}.parquet",
|
||||
)
|
||||
|
||||
def _throttle(self, credits: int = 1) -> None:
|
||||
"""Free tier: 8 credits / minute. Block until budget allows."""
|
||||
while True:
|
||||
|
||||
@@ -171,7 +171,7 @@ class TechnicalEngine:
|
||||
result.indicators["ema_trend"] = IndicatorSignal(
|
||||
name="EMA Trend", value=normalized,
|
||||
signal=signal, confidence=conf,
|
||||
description=desc, weight=1.8,
|
||||
description=desc, weight=1.0,
|
||||
)
|
||||
if t is not None:
|
||||
result.key_levels["ema_trend"] = float(t)
|
||||
@@ -222,7 +222,7 @@ class TechnicalEngine:
|
||||
|
||||
result.indicators["rsi"] = IndicatorSignal(
|
||||
name="RSI", value=rsi_val, signal=signal,
|
||||
confidence=conf, description=desc, weight=0.0,
|
||||
confidence=conf, description=desc, weight=1.8,
|
||||
)
|
||||
|
||||
# Attach RSI series for divergence detection (dynamic attribute, filtered by _prefix)
|
||||
@@ -291,7 +291,7 @@ class TechnicalEngine:
|
||||
|
||||
result.indicators["macd"] = IndicatorSignal(
|
||||
name="MACD", value=hist_val, signal=signal,
|
||||
confidence=conf, description=desc, weight=3.3,
|
||||
confidence=conf, description=desc, weight=1.2,
|
||||
)
|
||||
|
||||
# Attach histogram for divergence detection
|
||||
@@ -358,7 +358,7 @@ class TechnicalEngine:
|
||||
|
||||
result.indicators["bollinger"] = IndicatorSignal(
|
||||
name="Bollinger Bands", value=bb_position, signal=signal,
|
||||
confidence=conf, description=desc, weight=0.0,
|
||||
confidence=conf, description=desc, weight=1.5,
|
||||
)
|
||||
|
||||
result.key_levels["bb_upper"] = upper_val
|
||||
@@ -423,7 +423,7 @@ class TechnicalEngine:
|
||||
|
||||
result.indicators["stochastic"] = IndicatorSignal(
|
||||
name="Stochastic", value=k_val, signal=signal,
|
||||
confidence=conf, description=desc, weight=2.4,
|
||||
confidence=conf, description=desc, weight=2.5,
|
||||
)
|
||||
|
||||
def _compute_adx(self, df: pd.DataFrame, result: TechnicalAnalysis):
|
||||
@@ -609,7 +609,7 @@ class TechnicalEngine:
|
||||
name="Ichimoku", value=float(score), signal=signal,
|
||||
confidence=conf,
|
||||
description=f"Ichimoku [{score}/4]: {' | '.join(signals)}",
|
||||
weight=0.55,
|
||||
weight=0.3,
|
||||
)
|
||||
|
||||
result.key_levels["ichimoku_cloud_top"] = cloud_top
|
||||
@@ -672,7 +672,7 @@ class TechnicalEngine:
|
||||
name="Divergence", value=len(divergences),
|
||||
signal=signal, confidence=70,
|
||||
description=" | ".join(divergences),
|
||||
weight=1.5, # divergences are strong reversal signals
|
||||
weight=3.5,
|
||||
)
|
||||
# Note: divergences are NOT added to pattern_signals to avoid
|
||||
# duplicate reason collection (the indicator entry covers them).
|
||||
@@ -742,7 +742,7 @@ class TechnicalEngine:
|
||||
name="Candlestick", value=float(bullish - bearish),
|
||||
signal=signal, confidence=50,
|
||||
description=" | ".join(patterns),
|
||||
weight=0.7,
|
||||
weight=1.2,
|
||||
)
|
||||
result.pattern_signals.extend(patterns)
|
||||
|
||||
@@ -784,7 +784,7 @@ class TechnicalEngine:
|
||||
result.indicators["pivot"] = IndicatorSignal(
|
||||
name="Pivot Points", value=pivot,
|
||||
signal=SignalStrength.BUY if latest > pivot else SignalStrength.SELL,
|
||||
confidence=45, description=desc, weight=0.6,
|
||||
confidence=45, description=desc, weight=0.8,
|
||||
)
|
||||
|
||||
def _compute_volume_analysis(self, df: pd.DataFrame, result: TechnicalAnalysis):
|
||||
@@ -854,12 +854,10 @@ class TechnicalEngine:
|
||||
total_weight = 0.0
|
||||
total_confidence = 0.0
|
||||
|
||||
# Regime-gate ichimoku: trend hero (-23% to disable in 2024 trend) but
|
||||
# chop villain (+18% to disable in 2023 chop & 2026 recent).
|
||||
regime_weights = {
|
||||
"trending_up": {"ema_trend": 1.5, "adx": 1.3, "macd": 1.2, "ichimoku": 1.3, "rsi": 0.7, "bollinger": 0.6},
|
||||
"trending_down": {"ema_trend": 1.5, "adx": 1.3, "macd": 1.2, "ichimoku": 1.3, "rsi": 0.7, "bollinger": 0.6},
|
||||
"ranging": {"rsi": 1.4, "stochastic": 1.3, "bollinger": 1.4, "ema_trend": 0.6, "adx": 0.5, "ichimoku": 0.2},
|
||||
"trending_up": {"divergence": 1.5, "rsi": 1.2, "stochastic": 1.1, "candlestick": 1.2, "ema_trend": 1.0, "macd": 1.0, "adx": 1.0, "ichimoku": 0.8, "bollinger": 0.7, "pivot": 0.8},
|
||||
"trending_down": {"divergence": 1.5, "rsi": 1.2, "stochastic": 1.1, "candlestick": 1.2, "ema_trend": 1.0, "macd": 1.0, "adx": 1.0, "ichimoku": 0.8, "bollinger": 0.7, "pivot": 0.8},
|
||||
"ranging": {"divergence": 1.3, "rsi": 1.4, "stochastic": 1.4, "bollinger": 1.5, "candlestick": 1.3, "pivot": 1.3, "ema_trend": 0.5, "macd": 0.6, "adx": 0.4, "ichimoku": 0.2},
|
||||
}
|
||||
|
||||
active_regime_weights = regime_weights.get(result.market_regime, {})
|
||||
|
||||
@@ -2,14 +2,10 @@
|
||||
"""
|
||||
ForexBot2 — USD/JPY Signal Intelligence (single-pair)
|
||||
=======================================================
|
||||
Entry point for both live signal analysis and walk-forward backtesting.
|
||||
|
||||
Usage:
|
||||
python main.py # Live signal on H1
|
||||
python main.py --timeframe H4 --detail # Detailed H4 signal
|
||||
python main.py --json # JSON output
|
||||
python main.py --backtest # Walk-forward backtest
|
||||
python main.py --backtest --optimize # Parallel grid search
|
||||
"""
|
||||
|
||||
import argparse
|
||||
@@ -233,54 +229,6 @@ def report_to_dict(report: SignalReport) -> dict:
|
||||
return d
|
||||
|
||||
|
||||
# ── Backtest renderer ──────────────────────────────────────────────
|
||||
|
||||
def print_backtest_report(result, params: dict = None, verbose_journal: bool = False):
|
||||
m = result.metrics
|
||||
print(f" {Colors.BOLD}{Colors.WHITE}Backtest Summary — USD/JPY{Colors.RESET}")
|
||||
if params:
|
||||
params_str = ", ".join(f"{k}={v}" for k, v in params.items())
|
||||
print(f" {Colors.GRAY}Params: {params_str}{Colors.RESET}")
|
||||
print(f" {Colors.GRAY}{'─' * 76}{Colors.RESET}")
|
||||
|
||||
initial = result.initial_balance
|
||||
final = result.final_balance
|
||||
ret_pct = m.get("total_return_pct", 0)
|
||||
ret_color = Colors.GREEN if ret_pct >= 0 else Colors.RED
|
||||
|
||||
print(f" Trades: {Colors.BOLD}{m.get('total_trades', 0)}{Colors.RESET}"
|
||||
f" ({m.get('wins', 0)}W / {m.get('losses', 0)}L)")
|
||||
print(f" Win Rate: {m.get('win_rate', 0):.1f}%")
|
||||
print(f" Profit Factor: {m.get('profit_factor', 0):.2f}")
|
||||
pf_inf = m.get('profit_factor', 0) == float('inf')
|
||||
if pf_inf:
|
||||
print(f" (no losing trades — infinite PF)")
|
||||
print(f" Expectancy: {m.get('expectancy_pips', 0):+.2f} pips/trade"
|
||||
f" ({m.get('expectancy_per_trade', 0):+.2f} per trade)")
|
||||
print(f" Initial Balance: ${initial:,.2f}")
|
||||
print(f" Final Balance: {ret_color}${final:,.2f}{Colors.RESET}")
|
||||
print(f" Total Return: {ret_color}{ret_pct:+.2f}%{Colors.RESET}")
|
||||
print(f" CAGR: {m.get('cagr', 0):+.2f}%")
|
||||
print(f" Max Drawdown: {Colors.RED}{m.get('max_drawdown_pct', 0):.2f}%{Colors.RESET}")
|
||||
print(f" Sharpe Ratio: {m.get('sharpe', 0):.2f}")
|
||||
print(f" Sortino Ratio: {m.get('sortino', 0):.2f}")
|
||||
print(f" Avg Duration: {m.get('avg_trade_duration_bars', 0):.1f} bars")
|
||||
print(f" Win Streak: {m.get('longest_winning_streak', 0)} | "
|
||||
f"Loss Streak: {m.get('longest_losing_streak', 0)}")
|
||||
print(f" {Colors.GRAY}{'─' * 76}{Colors.RESET}")
|
||||
|
||||
if verbose_journal and result.trades:
|
||||
print(f"\n {Colors.BOLD}Trade Journal (last 10):{Colors.RESET}")
|
||||
print(f" {'#':<4} {'Entry Time':<22} {'Dir':<5} {'Entry':<9} "
|
||||
f"{'Exit':<9} {'Reason':<6} {'Pips':>8} {'P&L%':>7}")
|
||||
for i, t in enumerate(result.trades[-10:], start=max(1, len(result.trades) - 9)):
|
||||
color = Colors.GREEN if t.pnl_currency > 0 else Colors.RED
|
||||
entry_str = str(t.entry_time)[:19]
|
||||
print(f" {i:<4} {entry_str:<22} {t.direction:<5} {t.entry_price:<9.3f} "
|
||||
f"{t.exit_price:<9.3f} {t.exit_reason:<6} "
|
||||
f"{color}{t.pnl_pips:>+8.1f}{Colors.RESET} "
|
||||
f"{color}{t.pnl_pct:>+6.2f}%{Colors.RESET}")
|
||||
|
||||
|
||||
# ══════════════════════════════════════════════════════════════════
|
||||
# CLI
|
||||
@@ -288,7 +236,7 @@ def print_backtest_report(result, params: dict = None, verbose_journal: bool = F
|
||||
|
||||
def parse_args():
|
||||
parser = argparse.ArgumentParser(
|
||||
description="ForexBot2 — USD/JPY Signal Intelligence + Backtester",
|
||||
description="ForexBot2 — USD/JPY Signal Intelligence",
|
||||
formatter_class=argparse.RawDescriptionHelpFormatter,
|
||||
epilog="""
|
||||
Examples:
|
||||
@@ -296,58 +244,18 @@ Examples:
|
||||
python main.py --detail Live signal with full breakdown
|
||||
python main.py --timeframe H4 --detail H4 timeframe
|
||||
python main.py --json JSON output
|
||||
python main.py --backtest Walk-forward backtest (full history)
|
||||
python main.py --backtest --bars 2000 Backtest on last 2000 bars
|
||||
python main.py --backtest --optimize Parallel grid search across CPU cores
|
||||
""",
|
||||
)
|
||||
parser.add_argument("--timeframe", "-tf", default="H1",
|
||||
choices=["M5", "M15", "M30", "H1", "H4", "D1"],
|
||||
help="Primary analysis timeframe (default: H1)")
|
||||
parser.add_argument("--detail", action="store_true",
|
||||
help="Show full indicator breakdown (live mode)")
|
||||
help="Show full indicator breakdown")
|
||||
parser.add_argument("--json", action="store_true",
|
||||
help="Output as JSON")
|
||||
parser.add_argument("--verbose", "-v", action="store_true",
|
||||
help="Debug logging")
|
||||
|
||||
# Backtest options
|
||||
parser.add_argument("--backtest", action="store_true",
|
||||
help="Run walk-forward backtest instead of live signal")
|
||||
parser.add_argument("--optimize", action="store_true",
|
||||
help="Run parallel parameter grid search (implies --backtest)")
|
||||
parser.add_argument("--bars", type=int, default=5000,
|
||||
help="Max bars of history (default 5000 — TD ceiling)")
|
||||
parser.add_argument("--start", default=None,
|
||||
help="Backtest window start (YYYY-MM-DD)")
|
||||
parser.add_argument("--end", default=None,
|
||||
help="Backtest window end (YYYY-MM-DD)")
|
||||
parser.add_argument("--min-grade", default="B",
|
||||
choices=["A+", "A", "B", "C", "D", "F"],
|
||||
help="Minimum grade to open backtest trade (default B)")
|
||||
parser.add_argument("--journal", action="store_true",
|
||||
help="Print last 10 trades from backtest journal")
|
||||
parser.add_argument("--journal-csv", default=None,
|
||||
help="Write full trade journal to CSV path")
|
||||
parser.add_argument("--regime-filter", action="store_true",
|
||||
help="Gate signals on higher-TF ADX trend confirmation")
|
||||
parser.add_argument("--regime-min-adx", type=float, default=30.0,
|
||||
help="Min ADX for regime filter (default 30, requires --regime-filter)")
|
||||
parser.add_argument("--session-filter", action="store_true",
|
||||
help="Only trade during a UTC hour window (default 07-16, London + NY)")
|
||||
parser.add_argument("--session-start", type=int, default=7,
|
||||
help="Session start hour UTC, inclusive (default 7)")
|
||||
parser.add_argument("--session-end", type=int, default=16,
|
||||
help="Session end hour UTC, exclusive (default 16)")
|
||||
parser.add_argument("--auto-session-filter", action="store_true",
|
||||
help="Auto-toggle session filter when recent trades show adverse mean pnl%%")
|
||||
parser.add_argument("--auto-session-lookback", type=int, default=10,
|
||||
help="Lookback in closed trades for auto-session detection (default 10)")
|
||||
parser.add_argument("--auto-session-threshold-pct", type=float, default=-0.3,
|
||||
help="Mean pnl%% threshold below which auto-session activates (default -0.3)")
|
||||
parser.add_argument("--ablate", action="store_true",
|
||||
help="Run leave-one-out indicator ablation sweep (parallel)")
|
||||
|
||||
return parser.parse_args()
|
||||
|
||||
|
||||
@@ -372,21 +280,6 @@ def build_config(args) -> PlatformConfig:
|
||||
args.timeframe, [Timeframe.M15, Timeframe.H4]
|
||||
)
|
||||
|
||||
# Backtest config
|
||||
config.backtest.start_date = args.start
|
||||
config.backtest.end_date = args.end
|
||||
config.backtest.max_bars = args.bars
|
||||
config.backtest.min_trade_grade = args.min_grade
|
||||
config.backtest.optimize = args.optimize
|
||||
config.backtest.regime_filter = args.regime_filter
|
||||
config.backtest.regime_filter_min_adx = args.regime_min_adx
|
||||
config.backtest.session_filter = args.session_filter
|
||||
config.backtest.session_start_hour = args.session_start
|
||||
config.backtest.session_end_hour = args.session_end
|
||||
config.backtest.auto_session_filter = args.auto_session_filter
|
||||
config.backtest.auto_session_lookback_trades = args.auto_session_lookback
|
||||
config.backtest.auto_session_threshold_pct = args.auto_session_threshold_pct
|
||||
|
||||
return config
|
||||
|
||||
|
||||
@@ -406,148 +299,6 @@ def run_live(args, config: PlatformConfig):
|
||||
print(f"{Colors.CYAN}{'═' * 80}{Colors.RESET}\n")
|
||||
|
||||
|
||||
def run_backtest(args, config: PlatformConfig):
|
||||
from backtest import WalkForwardBacktester
|
||||
from metrics import trades_to_dataframe
|
||||
|
||||
print_header("USD/JPY WALK-FORWARD BACKTEST")
|
||||
print(f"{Colors.GRAY} Loading data and replaying bars...{Colors.RESET}\n")
|
||||
|
||||
backtester = WalkForwardBacktester(config)
|
||||
result = backtester.run()
|
||||
|
||||
if args.json:
|
||||
out = {
|
||||
"metrics": result.metrics,
|
||||
"initial_balance": result.initial_balance,
|
||||
"final_balance": result.final_balance,
|
||||
"trades": [
|
||||
{
|
||||
"entry_time": str(t.entry_time),
|
||||
"exit_time": str(t.exit_time),
|
||||
"direction": t.direction,
|
||||
"entry_price": t.entry_price,
|
||||
"exit_price": t.exit_price,
|
||||
"exit_reason": t.exit_reason,
|
||||
"pnl_pips": t.pnl_pips,
|
||||
"pnl_pct": t.pnl_pct,
|
||||
"grade": t.grade,
|
||||
}
|
||||
for t in result.trades
|
||||
],
|
||||
}
|
||||
print(json.dumps(out, indent=2))
|
||||
else:
|
||||
print_backtest_report(result, verbose_journal=args.journal)
|
||||
|
||||
if args.journal_csv and result.trades:
|
||||
df = trades_to_dataframe(result.trades)
|
||||
df.to_csv(args.journal_csv, index=False)
|
||||
print(f"\n {Colors.GREEN}✓ Trade journal written to {args.journal_csv}{Colors.RESET}")
|
||||
|
||||
|
||||
def run_optimize(args, config: PlatformConfig):
|
||||
from backtest import Optimizer
|
||||
|
||||
print_header("USD/JPY BACKTEST PARAMETER OPTIMIZER")
|
||||
grid = config.backtest.param_grid
|
||||
n_combos = 1
|
||||
for v in grid.values():
|
||||
n_combos *= len(v)
|
||||
print(f"{Colors.GRAY} Grid: {n_combos} combos × {config.backtest.parallel_workers} workers{Colors.RESET}\n")
|
||||
|
||||
optimizer = Optimizer(config)
|
||||
results = optimizer.run_grid(grid)
|
||||
|
||||
print(f"\n {Colors.BOLD}Top 10 Parameter Combinations (by total return):{Colors.RESET}\n")
|
||||
print(f" {'#':<3} {'Return%':>8} {'PF':>6} {'WinR':>6} {'Trades':>7} "
|
||||
f"{'MaxDD%':>7} {'Sharpe':>7} Params")
|
||||
print(f" {Colors.GRAY}{'─' * 90}{Colors.RESET}")
|
||||
|
||||
for i, r in enumerate(results[:10], start=1):
|
||||
m = r.metrics
|
||||
ret = m.get("total_return_pct", 0)
|
||||
ret_color = Colors.GREEN if ret >= 0 else Colors.RED
|
||||
params_str = ", ".join(f"{k}={v}" for k, v in r.params.items())
|
||||
pf = m.get("profit_factor", 0)
|
||||
pf_str = "inf" if pf == float("inf") else f"{pf:.2f}"
|
||||
print(f" {i:<3} {ret_color}{ret:>+7.2f}%{Colors.RESET} {pf_str:>6} "
|
||||
f"{m.get('win_rate', 0):>5.1f}% {m.get('total_trades', 0):>7} "
|
||||
f"{m.get('max_drawdown_pct', 0):>+6.2f}% {m.get('sharpe', 0):>7.2f}"
|
||||
f" {Colors.GRAY}{params_str}{Colors.RESET}")
|
||||
|
||||
print(f"\n {Colors.BOLD}Best combination details:{Colors.RESET}\n")
|
||||
print_backtest_report(results[0], params=results[0].params, verbose_journal=False)
|
||||
|
||||
|
||||
def run_ablation(args, config: PlatformConfig):
|
||||
"""Leave-one-out ablation: baseline + each indicator disabled, in parallel."""
|
||||
from backtest import Optimizer
|
||||
from config import ALL_INDICATORS
|
||||
|
||||
print_header("USD/JPY INDICATOR ABLATION SWEEP")
|
||||
|
||||
grid = {"disable_indicator": [None] + list(ALL_INDICATORS)}
|
||||
n_combos = len(grid["disable_indicator"])
|
||||
print(f"{Colors.GRAY} Grid: {n_combos} runs (1 baseline + {n_combos-1} ablations) "
|
||||
f"× {config.backtest.parallel_workers} workers{Colors.RESET}\n")
|
||||
|
||||
optimizer = Optimizer(config)
|
||||
results = optimizer.run_grid(grid)
|
||||
|
||||
# Find baseline (disable_indicator=None) for delta computation
|
||||
baseline = next(
|
||||
(r for r in results if not r.params.get("disable_indicator")),
|
||||
None,
|
||||
)
|
||||
base_ret = baseline.metrics.get("total_return_pct", 0.0) if baseline else 0.0
|
||||
base_pf = baseline.metrics.get("profit_factor", 0.0) if baseline else 0.0
|
||||
base_sharpe = baseline.metrics.get("sharpe", 0.0) if baseline else 0.0
|
||||
|
||||
# Sort: baseline first, then ablations by impact (largest drop = most useful indicator)
|
||||
ablations = [r for r in results if r is not baseline]
|
||||
ablations.sort(
|
||||
key=lambda r: r.metrics.get("total_return_pct", 0.0) - base_ret
|
||||
)
|
||||
|
||||
print(f" {Colors.BOLD}Indicator Ablation Report{Colors.RESET}")
|
||||
print(f" {Colors.GRAY}{'─' * 88}{Colors.RESET}")
|
||||
print(f" {'Disabled':<14} {'Return%':>9} {'ΔRet%':>8} {'PF':>6} {'ΔPF':>7} "
|
||||
f"{'Sharpe':>7} {'ΔSh':>7} {'Trades':>7}")
|
||||
print(f" {Colors.GRAY}{'─' * 88}{Colors.RESET}")
|
||||
|
||||
def fmt_row(label, m, deltas=True):
|
||||
ret = m.get("total_return_pct", 0.0)
|
||||
pf = m.get("profit_factor", 0.0)
|
||||
sh = m.get("sharpe", 0.0)
|
||||
tr = m.get("total_trades", 0)
|
||||
ret_color = Colors.GREEN if ret >= 0 else Colors.RED
|
||||
pf_str = "inf" if pf == float("inf") else f"{pf:.2f}"
|
||||
if deltas:
|
||||
d_ret = ret - base_ret
|
||||
d_pf = pf - base_pf if pf != float("inf") else 0.0
|
||||
d_sh = sh - base_sharpe
|
||||
d_ret_color = Colors.RED if d_ret < 0 else Colors.GREEN
|
||||
print(f" {label:<14} {ret_color}{ret:>+8.2f}%{Colors.RESET} "
|
||||
f"{d_ret_color}{d_ret:>+7.2f}%{Colors.RESET} "
|
||||
f"{pf_str:>6} {d_pf:>+7.2f} "
|
||||
f"{sh:>7.2f} {d_sh:>+7.2f} {tr:>7}")
|
||||
else:
|
||||
print(f" {Colors.BOLD}{label:<14}{Colors.RESET} "
|
||||
f"{ret_color}{ret:>+8.2f}%{Colors.RESET} {'':>8} "
|
||||
f"{pf_str:>6} {'':>7} {sh:>7.2f} {'':>7} {tr:>7}")
|
||||
|
||||
if baseline:
|
||||
fmt_row("baseline", baseline.metrics, deltas=False)
|
||||
print(f" {Colors.GRAY}{'─' * 88}{Colors.RESET}")
|
||||
for r in ablations:
|
||||
fmt_row(str(r.params.get("disable_indicator")), r.metrics)
|
||||
|
||||
print(f"\n {Colors.GRAY}Indicators ranked by importance — most negative ΔRet means "
|
||||
f"removing it hurt most.{Colors.RESET}")
|
||||
print(f" {Colors.GRAY}Positive ΔRet means the indicator is *hurting* the system.{Colors.RESET}\n")
|
||||
|
||||
|
||||
def main():
|
||||
args = parse_args()
|
||||
|
||||
@@ -559,14 +310,6 @@ def main():
|
||||
)
|
||||
|
||||
config = build_config(args)
|
||||
|
||||
if args.ablate:
|
||||
run_ablation(args, config)
|
||||
elif args.optimize:
|
||||
run_optimize(args, config)
|
||||
elif args.backtest:
|
||||
run_backtest(args, config)
|
||||
else:
|
||||
run_live(args, config)
|
||||
|
||||
|
||||
|
||||
File diff suppressed because it is too large
Load Diff
-100001
File diff suppressed because it is too large
Load Diff
File diff suppressed because it is too large
Load Diff
-100001
File diff suppressed because it is too large
Load Diff
-169
@@ -1,169 +0,0 @@
|
||||
"""
|
||||
ForexBot2 — Backtest Metrics
|
||||
==============================
|
||||
Pure functions over Trade records & equity curves.
|
||||
"""
|
||||
|
||||
from dataclasses import dataclass, field, asdict
|
||||
from typing import Dict, List, Optional
|
||||
|
||||
import numpy as np
|
||||
import pandas as pd
|
||||
|
||||
|
||||
@dataclass
|
||||
class Trade:
|
||||
"""A single closed trade."""
|
||||
entry_time: pd.Timestamp
|
||||
exit_time: pd.Timestamp
|
||||
direction: str # "BUY" / "SELL"
|
||||
entry_price: float
|
||||
exit_price: float
|
||||
stop_loss: float
|
||||
take_profit: float
|
||||
exit_reason: str # "tp" / "sl" / "end_of_data"
|
||||
risk_pips: float
|
||||
pnl_pips: float
|
||||
pnl_pct: float # P&L as % of equity at trade open
|
||||
pnl_currency: float # P&L in account currency (units = balance)
|
||||
duration_bars: int
|
||||
grade: str # signal grade at entry
|
||||
score_at_entry: float
|
||||
confidence_at_entry: float
|
||||
# Diagnostic fields captured at entry — used to mine for loss-pattern
|
||||
# filters when a window underperforms.
|
||||
entry_hour: int = -1 # UTC hour 0-23
|
||||
entry_dow: int = -1 # 0=Mon, 6=Sun
|
||||
adx_at_entry: float = 0.0
|
||||
atr_pct_at_entry: float = 0.0
|
||||
market_regime: str = "" # trending_up / trending_down / ranging
|
||||
volatility_regime: str = "" # low / medium / high
|
||||
mtf_agreement: bool = False
|
||||
m15_score: float = 0.0
|
||||
h1_score: float = 0.0
|
||||
h4_score: float = 0.0
|
||||
rsi_at_entry: float = 50.0
|
||||
ema_distance_pct: float = 0.0 # (price - ema_trend) / ema_trend * 100
|
||||
|
||||
|
||||
@dataclass
|
||||
class BacktestResult:
|
||||
"""Complete backtest output."""
|
||||
trades: List[Trade] = field(default_factory=list)
|
||||
equity_curve: pd.Series = field(default_factory=lambda: pd.Series(dtype=float))
|
||||
initial_balance: float = 0.0
|
||||
final_balance: float = 0.0
|
||||
metrics: Dict[str, float] = field(default_factory=dict)
|
||||
params: Dict[str, float] = field(default_factory=dict)
|
||||
|
||||
|
||||
def compute_metrics(
|
||||
trades: List[Trade],
|
||||
equity_curve: pd.Series,
|
||||
initial_balance: float,
|
||||
elapsed_days: Optional[float] = None,
|
||||
) -> Dict[str, float]:
|
||||
"""All summary metrics from a list of closed trades + equity curve."""
|
||||
if not trades:
|
||||
return {
|
||||
"total_trades": 0,
|
||||
"win_rate": 0.0,
|
||||
"profit_factor": 0.0,
|
||||
"expectancy_per_trade": 0.0,
|
||||
"expectancy_pips": 0.0,
|
||||
"total_return_pct": 0.0,
|
||||
"cagr": 0.0,
|
||||
"max_drawdown_pct": 0.0,
|
||||
"sharpe": 0.0,
|
||||
"sortino": 0.0,
|
||||
"avg_trade_duration_bars": 0.0,
|
||||
"longest_winning_streak": 0,
|
||||
"longest_losing_streak": 0,
|
||||
"final_balance": initial_balance,
|
||||
}
|
||||
|
||||
pnl_currency = np.array([t.pnl_currency for t in trades])
|
||||
pnl_pips = np.array([t.pnl_pips for t in trades])
|
||||
|
||||
wins = pnl_currency > 0
|
||||
losses = pnl_currency < 0
|
||||
|
||||
total_wins = pnl_currency[wins].sum() if wins.any() else 0.0
|
||||
total_losses = -pnl_currency[losses].sum() if losses.any() else 0.0
|
||||
|
||||
win_rate = wins.sum() / len(trades) * 100
|
||||
profit_factor = (total_wins / total_losses) if total_losses > 0 else float("inf")
|
||||
expectancy_per_trade = pnl_currency.mean()
|
||||
expectancy_pips = pnl_pips.mean()
|
||||
|
||||
final_balance = float(equity_curve.iloc[-1]) if len(equity_curve) else initial_balance
|
||||
total_return_pct = (final_balance / initial_balance - 1) * 100
|
||||
|
||||
# CAGR
|
||||
if elapsed_days and elapsed_days > 0 and final_balance > 0:
|
||||
years = elapsed_days / 365.25
|
||||
cagr = ((final_balance / initial_balance) ** (1 / years) - 1) * 100 if years > 0 else 0.0
|
||||
else:
|
||||
cagr = 0.0
|
||||
|
||||
# Max drawdown
|
||||
if len(equity_curve):
|
||||
running_max = equity_curve.cummax()
|
||||
drawdown = (equity_curve - running_max) / running_max
|
||||
max_dd = float(drawdown.min()) * 100
|
||||
else:
|
||||
max_dd = 0.0
|
||||
|
||||
# Sharpe / Sortino — daily returns from equity curve
|
||||
sharpe, sortino = 0.0, 0.0
|
||||
if len(equity_curve) > 2:
|
||||
daily = equity_curve.resample("1D").last().ffill().pct_change().dropna()
|
||||
if len(daily) > 1 and daily.std() > 0:
|
||||
sharpe = float(daily.mean() / daily.std() * np.sqrt(252))
|
||||
downside = daily[daily < 0]
|
||||
if len(downside) > 1 and downside.std() > 0:
|
||||
sortino = float(daily.mean() / downside.std() * np.sqrt(252))
|
||||
|
||||
# Streaks
|
||||
win_loss_seq = [1 if t.pnl_currency > 0 else (-1 if t.pnl_currency < 0 else 0) for t in trades]
|
||||
longest_win = _longest_streak(win_loss_seq, 1)
|
||||
longest_loss = _longest_streak(win_loss_seq, -1)
|
||||
|
||||
avg_duration = float(np.mean([t.duration_bars for t in trades]))
|
||||
|
||||
return {
|
||||
"total_trades": len(trades),
|
||||
"wins": int(wins.sum()),
|
||||
"losses": int(losses.sum()),
|
||||
"win_rate": float(win_rate),
|
||||
"profit_factor": float(profit_factor),
|
||||
"expectancy_per_trade": float(expectancy_per_trade),
|
||||
"expectancy_pips": float(expectancy_pips),
|
||||
"total_return_pct": float(total_return_pct),
|
||||
"cagr": float(cagr),
|
||||
"max_drawdown_pct": float(max_dd),
|
||||
"sharpe": float(sharpe),
|
||||
"sortino": float(sortino),
|
||||
"avg_trade_duration_bars": avg_duration,
|
||||
"longest_winning_streak": longest_win,
|
||||
"longest_losing_streak": longest_loss,
|
||||
"final_balance": final_balance,
|
||||
}
|
||||
|
||||
|
||||
def _longest_streak(seq: List[int], target: int) -> int:
|
||||
best = current = 0
|
||||
for x in seq:
|
||||
if x == target:
|
||||
current += 1
|
||||
best = max(best, current)
|
||||
else:
|
||||
current = 0
|
||||
return best
|
||||
|
||||
|
||||
def trades_to_dataframe(trades: List[Trade]) -> pd.DataFrame:
|
||||
"""Convert trade journal to a DataFrame for CSV/JSON export."""
|
||||
if not trades:
|
||||
return pd.DataFrame()
|
||||
return pd.DataFrame([asdict(t) for t in trades])
|
||||
+1
-2
@@ -1,7 +1,6 @@
|
||||
numpy>=1.24
|
||||
pandas>=2.0
|
||||
requests>=2.28
|
||||
pyarrow>=14.0
|
||||
tqdm>=4.66
|
||||
twelvedata>=1.2.0
|
||||
python-telegram-bot>=20.0
|
||||
websockets>=12.0
|
||||
|
||||
@@ -2,9 +2,7 @@
|
||||
ForexBot2 — Signal Aggregator
|
||||
==============================
|
||||
Combines multi-timeframe technical analysis with optional news context
|
||||
into a graded actionable signal. Same code path is used for live mode
|
||||
and walk-forward backtesting; the only difference is whether the input
|
||||
OHLCV is fetched live or sliced from a pre-loaded historical buffer.
|
||||
into a graded actionable signal.
|
||||
"""
|
||||
|
||||
import logging
|
||||
@@ -79,10 +77,7 @@ class SignalReport:
|
||||
|
||||
|
||||
class SignalAggregator:
|
||||
"""
|
||||
Master aggregator. Designed so the backtester can drive it bar-by-bar
|
||||
via `analyze_from_dataframes` without re-fetching from the API.
|
||||
"""
|
||||
"""Master aggregator for live signal analysis."""
|
||||
|
||||
def __init__(self, config: PlatformConfig):
|
||||
self.config = config
|
||||
|
||||
@@ -0,0 +1,55 @@
|
||||
"""Live tick stream from Twelve Data WebSocket API."""
|
||||
|
||||
import json
|
||||
import asyncio
|
||||
import websockets
|
||||
from datetime import datetime, timezone
|
||||
|
||||
|
||||
API_KEY = "f563b22ffdfd4924955decbe1a8c8701"
|
||||
WS_URL = f"wss://ws.twelvedata.com/v1/quotes/price?apikey={API_KEY}"
|
||||
SYMBOLS = ["EUR/USD", "GBP/USD", "USD/JPY"]
|
||||
|
||||
|
||||
async def stream_ticks():
|
||||
async with websockets.connect(WS_URL) as ws:
|
||||
subscribe = {
|
||||
"action": "subscribe",
|
||||
"params": {
|
||||
"symbols": ",".join(SYMBOLS),
|
||||
},
|
||||
}
|
||||
await ws.send(json.dumps(subscribe))
|
||||
print(f"Subscribed to: {', '.join(SYMBOLS)}")
|
||||
print("-" * 60)
|
||||
|
||||
async for raw in ws:
|
||||
msg = json.loads(raw)
|
||||
|
||||
if msg.get("event") == "subscribe-status":
|
||||
print(f"Status: {msg.get('status')} — {msg.get('message', '')}")
|
||||
continue
|
||||
|
||||
if msg.get("event") == "heartbeat":
|
||||
continue
|
||||
|
||||
if msg.get("event") == "price":
|
||||
ts = datetime.fromtimestamp(msg["timestamp"], tz=timezone.utc)
|
||||
symbol = msg["symbol"]
|
||||
price = msg["price"]
|
||||
print(f"{ts:%H:%M:%S} | {symbol:<8} | {price}")
|
||||
|
||||
|
||||
async def main():
|
||||
print(f"Connecting to Twelve Data WebSocket...")
|
||||
print(f"API key: ...{API_KEY[-6:]}")
|
||||
try:
|
||||
await stream_ticks()
|
||||
except KeyboardInterrupt:
|
||||
print("\nStopped.")
|
||||
except websockets.exceptions.ConnectionClosed as e:
|
||||
print(f"\nConnection closed: {e}")
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
asyncio.run(main())
|
||||
Reference in New Issue
Block a user