backtest: walk-forward mode + extended-range data safety
--walk-forward slices ONE continuous replay into rolling train/test folds by entry time (default 12mo/3mo/3mo; no per-fold re-optimization) and reports per-fold PF / expectancy / max DD plus aggregated OOS stats, flagging test folds with PF < 1. Off by default — default runs are byte-identical (scripts/compare_baseline.py 32/32 PASS with this diff). Data safety for deep-history fetches: fetch_history refuses to recreate any baseline-pinned parquet filename and records the sha256 of every freshly written parquet in backtest/cache/data_manifest.json. Tests: Module 23 (fold builder, slicing, flagging, OOS dedup, CLI defaults, pin guard, manifest recording) — 367 total. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_017FBjfomAipxvPWwRJEDDyJ
This commit is contained in:
Harwinder Sekhon
@@ -11,7 +11,7 @@ Every numeric threshold in `config.py` and `pair_signals/xauusd.py` was backtest
|
||||
## Commands
|
||||
|
||||
```bash
|
||||
python test_all.py # QA suite: 359 tests, 22 modules, no network (mocked)
|
||||
python test_all.py # QA suite: 367 tests, 23 modules, no network (mocked)
|
||||
python main.py --pair XAUUSD --timeframe H1 --detail # CLI signal report (--json for full dump)
|
||||
python telegram_bot.py # the bot (refuses to boot without credentials)
|
||||
python oanda_daemon.py # auto-execution daemon on Oanda practice account
|
||||
@@ -26,6 +26,10 @@ python backtest/gpu_backtest.py --pair XAUUSD --start 2025-01-01 --end 2025-06-0
|
||||
# --warmup-bars --cooldown-bars --no-cache --export-csv out.csv --detail
|
||||
# --fixed-rr-tp N (research-only: hard TP at N×risk, trail off; verdict
|
||||
# 2026-08-05: NOT shipped — backtest/research/fixedtp_2026-08-05.md)
|
||||
# --walk-forward [--wf-train-months 12 --wf-test-months 3 --wf-step-months 3]
|
||||
# rolling-fold OOS report over the --start/--end window (one continuous
|
||||
# replay sliced by entry time; fixed config, no re-optimization; test
|
||||
# folds with PF<1 flagged). Off by default — default runs byte-identical.
|
||||
```
|
||||
|
||||
**`test_all.py` is NOT pytest** — it uses a custom `@test("name")` decorator that runs each test immediately at import; don't grep for `def test_` or try `pytest test_all.py`. Run the whole file; it prints per-module results and exits non-zero on failure. There is no way to run a single test short of commenting modules out. Tests must stay green; every new gate/knob needs tests. When patching weekend logic in tests, patch `providers.base._is_forex_weekend` — but know there are THREE independent weekend implementations (`providers/base.py`, `signal_aggregator.py`, and an inline copy in `backtest/gpu_backtest.py:_is_xau_blocked_by_calendar`); patching the provider one does not reach the other two.
|
||||
@@ -62,7 +66,7 @@ Signal currency everywhere is `SignalStrength` (−2..+2). DataFrames are UTC tz
|
||||
- **`telegram_bot.py`** — owner-locked bot + subscriber channel; candle-boundary auto-scan loops (M5/M15/H1/H4, boundary+10 s), alert quality threshold + fingerprint dedup (committed only after a successful send), 📰 LLM market-context line on alerts when available, WebSocket level-breach monitor fed by the streaming provider (`data_providers.streaming`, falling back to the active provider; auto-disabled when the resolved provider can't stream or has no API key — Oanda does not stream, Tiingo does). Rotating log: `logs/telegram.log`, 3-h rotation × 240 backups.
|
||||
- **`oanda_daemon.py` / `oanda_executor.py`** — practice-account auto-execution. Book-parity rules (see daemon docstring): no broker-side TP (trailing model — TP1/TP2 advisory), 5-bar per-TF cooldown (wall-clock approximation, so it expires over weekends where the backtester would still count bars), trail ratchets at bar close once per *observed* bar (a missed/failed cycle skips those bars entirely — the intra-gap high/low never enters the peak). Accepted divergence: one position total; orders send `positionFill=OPEN_ONLY` so a state/broker divergence cancels the order instead of silently netting. State/broker re-sync runs at boot and after any suspicious fill (`_reconcile_positions`).
|
||||
- **`dashboard_publisher.py` / `static/dashboard.html`** — web dashboard as a **static publisher** (live LAMP server): standalone daemon scans at candle closes via `produce_signal` and atomically writes `snapshot.json`/`news.json`/`dashboard.html` to `$DASHBOARD_OUT_DIR` (Apache serves them; page is public by owner decision). **Hard independence rule: never imports, and is never imported by, `telegram_bot`/`oanda_daemon`** — confirmation map and boundary math are deliberate local copies, so a web-side failure can only mean a stale page. Due-TF classification uses the RAW wake time (flooring wake−buffer skips every H1/H4 scan — 2026-08-02 bug). News panel reads `NewsEngine.get_fundamental_context()` plus the additive `get_week_events()` accessor (display-only; no signal path calls it); as a separate process it has its own news caches and DeepSeek daily cap. News/calendar rows are clickable (detail dialog); headlines carry `id` = sha1(headline) in news.json, and exactly one on-demand endpoint exists: `static/news_summary.php` (installed by `install_static`), which serves a DeepSeek gold-impact summary per headline — whitelist-gated against news.json ids, cache-forever in `ai/`, own fail-closed daily cap (150/day, file-based, separate from the publisher's), key at `/etc/gold-dashboard/deepseek.key`, never reads repo config. Deployment checklist: `deploy/README-dashboard.md`.
|
||||
- **`backtest/gpu_backtest.py`** — bar-by-bar replay through the real `analyze_from_dataframes` (news off). Realism model (do not regress): next-bar-open entries re-anchored to the raw price path, round-trip `--spread` charged per trade, trade resolution on RAW bars (cleaning applies only to signal-side data), same-bar SL+TP tie → SL first, gap fills at bar open, trail ratchet at bar close only. Indicators are precomputed once per TF (CuPy GPU or NumPy fallback — check the `Backend:` log line); replay fans out over worker processes and the 5-bar cooldown is applied in an ordered pass afterwards, so parallel and sequential books are identical. Parquet cache in `backtest/cache/` keyed by provider/symbol/TF/window. `backtest/*.csv` are gitignored pinned trade books from past campaigns — used for manual before/after comparison. `backtest/instructions_for_backtest.txt` lists what every backtest report must record.
|
||||
- **`backtest/gpu_backtest.py`** — bar-by-bar replay through the real `analyze_from_dataframes` (news off). Realism model (do not regress): next-bar-open entries re-anchored to the raw price path, round-trip `--spread` charged per trade, trade resolution on RAW bars (cleaning applies only to signal-side data), same-bar SL+TP tie → SL first, gap fills at bar open, trail ratchet at bar close only. Indicators are precomputed once per TF (CuPy GPU or NumPy fallback — check the `Backend:` log line); replay fans out over worker processes and the 5-bar cooldown is applied in an ordered pass afterwards, so parallel and sequential books are identical. Parquet cache in `backtest/cache/` keyed by provider/symbol/TF/window; `fetch_history` refuses to (re)write any filename pinned by a `baseline/*/manifest.json` (a cache miss there means the reference file was deleted — restore it, don't refetch) and records the sha256 of every freshly fetched parquet in `backtest/cache/data_manifest.json` (same `cache_files` format as the baseline manifest). `backtest/*.csv` are gitignored pinned trade books from past campaigns — used for manual before/after comparison. `backtest/instructions_for_backtest.txt` lists what every backtest report must record.
|
||||
|
||||
## Rules for making changes (read before editing signal logic)
|
||||
|
||||
|
||||
@@ -25,11 +25,14 @@ Usage:
|
||||
python backtest/gpu_backtest.py --pair XAUUSD --days 365
|
||||
python backtest/gpu_backtest.py --pair XAUUSD --start 2025-01-01 --end 2025-06-01
|
||||
python backtest/gpu_backtest.py --pair XAUUSD --days 180 --no-cache
|
||||
python backtest/gpu_backtest.py --pair XAUUSD --start 2021-07-30 --end 2026-07-30 \
|
||||
--walk-forward # rolling 12mo-train/3mo-test folds, step 3mo
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import argparse
|
||||
import hashlib
|
||||
import json
|
||||
import logging
|
||||
import multiprocessing as mp
|
||||
@@ -972,6 +975,65 @@ class GPUTechnicalEngine(TechnicalEngine):
|
||||
|
||||
CACHE_DIR = Path(__file__).resolve().parent / "cache"
|
||||
|
||||
#: Root of the byte-for-byte reference set (scripts/capture_baseline.py).
|
||||
BASELINE_ROOT = PROJECT_ROOT / "baseline"
|
||||
|
||||
|
||||
def _sha256_file(path: Path) -> str:
|
||||
h = hashlib.sha256()
|
||||
with open(path, "rb") as f:
|
||||
for chunk in iter(lambda: f.read(1 << 20), b""):
|
||||
h.update(chunk)
|
||||
return h.hexdigest()
|
||||
|
||||
|
||||
def _baseline_pinned_names() -> set:
|
||||
"""Parquet filenames claimed by any baseline/<commit>/manifest.json.
|
||||
|
||||
These are the pinned inputs of the byte-for-byte reference runs
|
||||
(scripts/compare_baseline.py). fetch_history must never write one:
|
||||
the write path only runs on a cache miss, so reaching it for a pinned
|
||||
name means the original file was deleted — and recreating it from a
|
||||
live fetch would put NEW bars under the baseline's filename, silently
|
||||
poisoning the reference set. Extended-range windows always resolve to
|
||||
different filenames (the cache key embeds start/end), so they land in
|
||||
separate parquets and never collide with the pinned set.
|
||||
"""
|
||||
names: set = set()
|
||||
if not BASELINE_ROOT.is_dir():
|
||||
return names
|
||||
for manifest in BASELINE_ROOT.glob("*/manifest.json"):
|
||||
try:
|
||||
names.update(json.loads(manifest.read_text()).get("cache_files", {}))
|
||||
except (OSError, json.JSONDecodeError):
|
||||
continue
|
||||
return names
|
||||
|
||||
|
||||
def _record_in_data_manifest(path: Path) -> None:
|
||||
"""Record a freshly written parquet's sha256 in cache/data_manifest.json.
|
||||
|
||||
Uses the baseline manifest's ``cache_files`` mapping format
|
||||
({filename: sha256}) so extended-range data files carry the same
|
||||
provenance as the pinned baseline set. Best-effort: a manifest write
|
||||
failure must not fail the fetch that produced the data.
|
||||
"""
|
||||
manifest_path = CACHE_DIR / "data_manifest.json"
|
||||
try:
|
||||
manifest = json.loads(manifest_path.read_text())
|
||||
except (OSError, json.JSONDecodeError):
|
||||
manifest = {}
|
||||
files = manifest.get("cache_files", {})
|
||||
files[path.name] = _sha256_file(path)
|
||||
manifest["cache_files"] = dict(sorted(files.items()))
|
||||
manifest["updated_utc"] = datetime.now(timezone.utc).strftime(
|
||||
"%Y-%m-%dT%H:%M:%SZ")
|
||||
try:
|
||||
manifest_path.write_text(
|
||||
json.dumps(manifest, indent=2, sort_keys=True) + "\n")
|
||||
except OSError as exc:
|
||||
logger.warning(f"data_manifest.json not updated: {exc}")
|
||||
|
||||
|
||||
def _cache_path(
|
||||
symbol: str, tf: Timeframe, start: datetime, end: datetime,
|
||||
@@ -1031,7 +1093,15 @@ def fetch_history(
|
||||
df = provider.fetch_range(symbol, timeframe, start, end)
|
||||
|
||||
if use_cache:
|
||||
if path.name in _baseline_pinned_names():
|
||||
raise RuntimeError(
|
||||
f"refusing to write {path.name}: this filename is pinned by a "
|
||||
f"baseline manifest under baseline/ and freshly fetched bars "
|
||||
f"are not the baseline's bars. Restore the original parquet "
|
||||
f"(git/backup) instead of refetching, or use a different "
|
||||
f"--start/--end window.")
|
||||
df.to_parquet(path)
|
||||
_record_in_data_manifest(path)
|
||||
logger.info(f"Cached {len(df)} raw bars to {path.name}")
|
||||
return provider.validate_and_clean(df, symbol, timeframe) if clean else df
|
||||
|
||||
@@ -2157,6 +2227,154 @@ def print_trade_reasoning(trades: List[Trade], limit: int = 5) -> None:
|
||||
print(f" MFE / MAE: {t.mfe_r:.2f}R / {t.mae_r:.2f}R")
|
||||
|
||||
|
||||
# ═════════════════════════════════════════════════════════════════════
|
||||
# WALK-FORWARD ANALYSIS (--walk-forward)
|
||||
# ═════════════════════════════════════════════════════════════════════
|
||||
#
|
||||
# Rolling-fold out-of-sample evaluation of ONE continuous replay: the
|
||||
# window is replayed exactly once with the run's fixed config (there is
|
||||
# no per-fold re-optimization — nothing here retunes anything), then the
|
||||
# trade book is sliced into rolling train/test folds by ENTRY time.
|
||||
# Slicing a single replay keeps the book identical to a normal run of the
|
||||
# same window: warmup happens once and the entry cooldown crosses fold
|
||||
# boundaries exactly as it would live. Train columns are in-sample
|
||||
# context; the aggregated stats over all test folds are the OOS read.
|
||||
|
||||
def _add_months(dt: datetime, months: int) -> datetime:
|
||||
"""Calendar-month arithmetic (tz preserved; clamps at month end)."""
|
||||
return (pd.Timestamp(dt) + pd.DateOffset(months=months)).to_pydatetime()
|
||||
|
||||
|
||||
def build_walkforward_folds(
|
||||
start: datetime, end: datetime,
|
||||
train_months: int, test_months: int, step_months: int,
|
||||
) -> List[Tuple[datetime, datetime, datetime, datetime]]:
|
||||
"""Rolling (train_start, train_end, test_start, test_end) folds.
|
||||
|
||||
test_start == train_end; a fold is emitted only if it fits entirely
|
||||
inside [start, end]. Returns [] when the window is too short.
|
||||
"""
|
||||
if min(train_months, test_months, step_months) <= 0:
|
||||
raise ValueError("train/test/step months must all be > 0")
|
||||
folds = []
|
||||
i = 0
|
||||
while True:
|
||||
tr_s = _add_months(start, i * step_months)
|
||||
tr_e = _add_months(tr_s, train_months)
|
||||
te_e = _add_months(tr_e, test_months)
|
||||
if te_e > end:
|
||||
break
|
||||
folds.append((tr_s, tr_e, tr_e, te_e))
|
||||
i += 1
|
||||
return folds
|
||||
|
||||
|
||||
def walkforward_stats(
|
||||
trades: List[Trade],
|
||||
folds: List[Tuple[datetime, datetime, datetime, datetime]],
|
||||
) -> Dict:
|
||||
"""Per-fold train/test Metrics + deduped aggregated OOS stats.
|
||||
|
||||
Fold membership is by entry time (half-open [lo, hi)); a trade's
|
||||
resolution may extend past its fold end, matching how a live account
|
||||
would hold through the boundary. With step < test length the test
|
||||
windows overlap — the OOS aggregate counts each trade exactly once,
|
||||
in chronological order.
|
||||
"""
|
||||
fold_stats: List[Dict] = []
|
||||
oos_trades: List[Trade] = []
|
||||
seen_ids: set = set()
|
||||
for k, (tr_s, tr_e, te_s, te_e) in enumerate(folds, 1):
|
||||
lo_tr, hi_tr = pd.Timestamp(tr_s), pd.Timestamp(tr_e)
|
||||
lo_te, hi_te = pd.Timestamp(te_s), pd.Timestamp(te_e)
|
||||
test_slice = [t for t in trades if lo_te <= t.entry_time < hi_te]
|
||||
train_m = compute_metrics(
|
||||
[t for t in trades if lo_tr <= t.entry_time < hi_tr])
|
||||
test_m = compute_metrics(test_slice)
|
||||
fold_stats.append({
|
||||
"fold": k,
|
||||
"train_start": tr_s, "train_end": tr_e,
|
||||
"test_start": te_s, "test_end": te_e,
|
||||
"train": train_m, "test": test_m,
|
||||
# PF < 1.0 on the fold's OOS slice; empty folds are not flagged
|
||||
# (no evidence either way) but are visible as n=0 in the table.
|
||||
"flagged": test_m.n_trades > 0 and test_m.profit_factor < 1.0,
|
||||
})
|
||||
for t in test_slice:
|
||||
if id(t) not in seen_ids:
|
||||
seen_ids.add(id(t))
|
||||
oos_trades.append(t)
|
||||
oos_trades.sort(key=lambda t: t.entry_time)
|
||||
return {
|
||||
"folds": fold_stats,
|
||||
"oos": compute_metrics(oos_trades),
|
||||
"n_flagged": sum(1 for f in fold_stats if f["flagged"]),
|
||||
}
|
||||
|
||||
|
||||
def _fmt_pf(m: Metrics) -> str:
|
||||
if m.n_trades == 0:
|
||||
return "—"
|
||||
return "inf" if np.isinf(m.profit_factor) else f"{m.profit_factor:.2f}"
|
||||
|
||||
|
||||
def print_walkforward_report(
|
||||
bt_cfg: BacktestConfig, stats: Dict,
|
||||
train_months: int, test_months: int, step_months: int,
|
||||
) -> None:
|
||||
bar = "═" * 78
|
||||
folds = stats["folds"]
|
||||
print(f"\n{bar}")
|
||||
print(f" WALK-FORWARD ANALYSIS "
|
||||
f"({bt_cfg.start.date()} → {bt_cfg.end.date()})")
|
||||
print(f" Train {train_months}mo / Test {test_months}mo / "
|
||||
f"Step {step_months}mo → {len(folds)} folds "
|
||||
f"(fixed config — no per-fold re-optimization)")
|
||||
print(f"{bar}")
|
||||
|
||||
rows = []
|
||||
for f in folds:
|
||||
tr, te = f["train"], f["test"]
|
||||
rows.append([
|
||||
str(f["fold"]),
|
||||
f"{f['train_start']:%Y-%m-%d}→{f['train_end']:%Y-%m-%d}",
|
||||
str(tr.n_trades), _fmt_pf(tr),
|
||||
f"{tr.avg_r:+.3f}" if tr.n_trades else "—",
|
||||
f"{tr.max_drawdown_r:.2f}" if tr.n_trades else "—",
|
||||
f"{f['test_start']:%Y-%m-%d}→{f['test_end']:%Y-%m-%d}",
|
||||
str(te.n_trades), _fmt_pf(te),
|
||||
f"{te.avg_r:+.3f}" if te.n_trades else "—",
|
||||
f"{te.max_drawdown_r:.2f}" if te.n_trades else "—",
|
||||
"⚠ PF<1" if f["flagged"] else "",
|
||||
])
|
||||
print("\n ── Rolling folds (expR = expectancy per trade, "
|
||||
"DD = max drawdown in R) ──")
|
||||
print(_fmt_table(
|
||||
["Fold", "Train window", "n", "PF", "expR", "DD",
|
||||
"Test window", "n", "PF", "expR", "DD", "Flag"], rows))
|
||||
|
||||
o = stats["oos"]
|
||||
print("\n ── Aggregated out-of-sample (union of test folds, "
|
||||
"each trade once) ──")
|
||||
if o.n_trades == 0:
|
||||
print(" No trades in any test fold.")
|
||||
else:
|
||||
print(_fmt_table(["Metric", "Value"], [
|
||||
["OOS trades", f"{o.n_trades}"],
|
||||
["Win rate", f"{o.win_rate:.1f}%"],
|
||||
["Total R", f"{o.total_r:+.2f}"],
|
||||
["Expectancy / trade", f"{o.avg_r:+.3f}"],
|
||||
["Profit factor", _fmt_pf(o)],
|
||||
["Max DD (R)", f"{o.max_drawdown_r:.2f}"],
|
||||
["Folds flagged PF<1", f"{stats['n_flagged']} / {len(folds)}"],
|
||||
]))
|
||||
if stats["n_flagged"]:
|
||||
flagged = [str(f["fold"]) for f in folds if f["flagged"]]
|
||||
print(f"\n ⚠ {stats['n_flagged']} fold(s) with out-of-sample "
|
||||
f"PF < 1.0: fold {', '.join(flagged)}")
|
||||
print(f"{bar}\n")
|
||||
|
||||
|
||||
def trades_to_dataframe(trades: List[Trade]) -> pd.DataFrame:
|
||||
"""Export trades for downstream analysis (e.g. parquet/CSV).
|
||||
|
||||
@@ -2217,6 +2435,19 @@ def _parse_args(argv: Optional[List[str]] = None) -> argparse.Namespace:
|
||||
p.add_argument("--days", type=int, default=180, help="Days of history (ignored if --start/--end set)")
|
||||
p.add_argument("--start", help="ISO date YYYY-MM-DD (overrides --days)")
|
||||
p.add_argument("--end", help="ISO date YYYY-MM-DD (defaults to today)")
|
||||
p.add_argument("--walk-forward", action="store_true",
|
||||
help="Walk-forward analysis: replay the window once with "
|
||||
"the run's fixed config, then slice the book into "
|
||||
"rolling train/test folds and report per-fold PF / "
|
||||
"expectancy / max DD / trade count plus aggregated "
|
||||
"out-of-sample stats (folds with test PF < 1.0 are "
|
||||
"flagged). No per-fold re-optimization.")
|
||||
p.add_argument("--wf-train-months", type=int, default=12,
|
||||
help="Walk-forward: train window length (calendar months)")
|
||||
p.add_argument("--wf-test-months", type=int, default=3,
|
||||
help="Walk-forward: test window length (calendar months)")
|
||||
p.add_argument("--wf-step-months", type=int, default=3,
|
||||
help="Walk-forward: step between fold starts (calendar months)")
|
||||
p.add_argument("--primary-tf", default="15min", help="Primary TF (1min/5min/15min/1h/4h)")
|
||||
p.add_argument("--confirm-tfs", default=None,
|
||||
help="Comma-separated confirmation TFs. Default: the pair "
|
||||
@@ -2398,6 +2629,24 @@ def main(argv: Optional[List[str]] = None) -> int:
|
||||
else:
|
||||
start = end - timedelta(days=args.days)
|
||||
|
||||
# Fail fast on an unusable walk-forward request — before any data
|
||||
# loading. With walk-forward off nothing below changes.
|
||||
wf_folds = None
|
||||
if args.walk_forward:
|
||||
try:
|
||||
wf_folds = build_walkforward_folds(
|
||||
start, end, args.wf_train_months,
|
||||
args.wf_test_months, args.wf_step_months)
|
||||
except ValueError as exc:
|
||||
raise SystemExit(f"--walk-forward: {exc}")
|
||||
if not wf_folds:
|
||||
raise SystemExit(
|
||||
f"--walk-forward: window {start.date()} → {end.date()} "
|
||||
f"({(end - start).days} days) is shorter than one fold "
|
||||
f"(train {args.wf_train_months}mo + test "
|
||||
f"{args.wf_test_months}mo); pass a longer --start/--end "
|
||||
f"range.")
|
||||
|
||||
spread = (args.spread if args.spread is not None
|
||||
else PAIR_INFO[pair].get("default_spread", 0.30))
|
||||
|
||||
@@ -2427,6 +2676,10 @@ def main(argv: Optional[List[str]] = None) -> int:
|
||||
trades, _ = run_backtest(bt_cfg, platform_cfg=platform_cfg)
|
||||
metrics = compute_metrics(trades)
|
||||
print_report(pair, bt_cfg, metrics, trades=trades, detail=args.detail)
|
||||
if wf_folds:
|
||||
print_walkforward_report(
|
||||
bt_cfg, walkforward_stats(trades, wf_folds),
|
||||
args.wf_train_months, args.wf_test_months, args.wf_step_months)
|
||||
print_trade_reasoning(trades, limit=args.reasoning_samples)
|
||||
|
||||
if args.export_csv:
|
||||
|
||||
+241
@@ -6943,6 +6943,247 @@ def _():
|
||||
assert "must be > 0" in str(e)
|
||||
|
||||
|
||||
# ══════════════════════════════════════════════════════════════════════
|
||||
# MODULE 23: walk-forward mode + extended-range data safety
|
||||
# ══════════════════════════════════════════════════════════════════════
|
||||
print("\n═══ Module 23: walk-forward mode + extended-range data safety ═══")
|
||||
|
||||
|
||||
def _wf_trade(ts, r):
|
||||
"""Resolved trade at ts (UTC) with realized R — for fold slicing tests."""
|
||||
t = _m21_sim_trade(ts=ts)
|
||||
t.exit_reason = "TP1" if r > 0 else "SL"
|
||||
t.exit_time = t.entry_time
|
||||
t.r_multiple = r
|
||||
t.bars_held = 3
|
||||
return t
|
||||
|
||||
|
||||
@test("walk-forward: fold builder — 12/3/3 over 24mo = 4 contiguous folds")
|
||||
def _():
|
||||
import gpu_backtest as _gb
|
||||
start = datetime(2021, 1, 1, tzinfo=timezone.utc)
|
||||
end = datetime(2023, 1, 1, tzinfo=timezone.utc)
|
||||
folds = _gb.build_walkforward_folds(start, end, 12, 3, 3)
|
||||
assert len(folds) == 4, f"expected 4 folds, got {len(folds)}"
|
||||
for i, (tr_s, tr_e, te_s, te_e) in enumerate(folds):
|
||||
assert te_s == tr_e, "test must start exactly at train end"
|
||||
assert tr_s == datetime(2021, 1 + 3 * i, 1, tzinfo=timezone.utc)
|
||||
assert folds[0][3] == datetime(2022, 4, 1, tzinfo=timezone.utc)
|
||||
assert folds[-1][3] == end, "last fold must end exactly at window end"
|
||||
|
||||
|
||||
@test("walk-forward: fold builder — month-end clamp, short window, bad months")
|
||||
def _():
|
||||
import gpu_backtest as _gb
|
||||
# Jan 31 + 1 month clamps to Feb 28 (calendar-month arithmetic).
|
||||
d = _gb._add_months(datetime(2021, 1, 31, tzinfo=timezone.utc), 1)
|
||||
assert (d.year, d.month, d.day) == (2021, 2, 28)
|
||||
# 14mo window can't fit train 12 + test 3 → no folds, no exception.
|
||||
start = datetime(2021, 1, 1, tzinfo=timezone.utc)
|
||||
assert _gb.build_walkforward_folds(
|
||||
start, datetime(2022, 3, 1, tzinfo=timezone.utc), 12, 3, 3) == []
|
||||
# 18mo window fits exactly two stepped folds.
|
||||
assert len(_gb.build_walkforward_folds(
|
||||
start, datetime(2022, 7, 1, tzinfo=timezone.utc), 12, 3, 3)) == 2
|
||||
for bad in ((0, 3, 3), (12, 0, 3), (12, 3, -1)):
|
||||
try:
|
||||
_gb.build_walkforward_folds(start, start, *bad)
|
||||
assert False, f"months {bad} must raise ValueError"
|
||||
except ValueError as e:
|
||||
assert "must all be > 0" in str(e)
|
||||
|
||||
|
||||
@test("walk-forward: fold slicing — half-open boundaries, per-fold metrics")
|
||||
def _():
|
||||
import gpu_backtest as _gb
|
||||
fold = (datetime(2024, 1, 1, tzinfo=timezone.utc),
|
||||
datetime(2025, 1, 1, tzinfo=timezone.utc),
|
||||
datetime(2025, 1, 1, tzinfo=timezone.utc),
|
||||
datetime(2025, 4, 1, tzinfo=timezone.utc))
|
||||
trades = [
|
||||
_wf_trade("2024-02-01 10:00", +1.0), # train
|
||||
_wf_trade("2024-03-01 10:00", -1.0), # train
|
||||
_wf_trade("2024-06-01 10:00", +2.0), # train
|
||||
_wf_trade("2025-01-01 00:00", -1.0), # exactly test_start → test
|
||||
_wf_trade("2025-02-15 10:00", +0.5), # test
|
||||
_wf_trade("2025-04-01 00:00", +9.0), # exactly test_end → excluded
|
||||
]
|
||||
unresolved = _wf_trade("2025-02-20 10:00", +1.0)
|
||||
unresolved.exit_reason = "" # never scored
|
||||
trades.append(unresolved)
|
||||
stats = _gb.walkforward_stats(trades, [fold])
|
||||
f = stats["folds"][0]
|
||||
tr, te = f["train"], f["test"]
|
||||
assert tr.n_trades == 3 and te.n_trades == 2
|
||||
assert abs(tr.profit_factor - 3.0) < 1e-9
|
||||
assert abs(tr.avg_r - 2.0 / 3.0) < 1e-9
|
||||
assert abs(tr.max_drawdown_r - 1.0) < 1e-9
|
||||
assert abs(te.profit_factor - 0.5) < 1e-9
|
||||
assert abs(te.avg_r - (-0.25)) < 1e-9
|
||||
assert abs(te.max_drawdown_r - 1.0) < 1e-9
|
||||
assert f["flagged"], "test PF 0.5 < 1.0 must be flagged"
|
||||
|
||||
|
||||
@test("walk-forward: flagging — PF≥1 and zero-trade folds are not flagged")
|
||||
def _():
|
||||
import gpu_backtest as _gb
|
||||
good = (datetime(2024, 1, 1, tzinfo=timezone.utc),
|
||||
datetime(2025, 1, 1, tzinfo=timezone.utc),
|
||||
datetime(2025, 1, 1, tzinfo=timezone.utc),
|
||||
datetime(2025, 4, 1, tzinfo=timezone.utc))
|
||||
empty = (datetime(2024, 4, 1, tzinfo=timezone.utc),
|
||||
datetime(2025, 4, 1, tzinfo=timezone.utc),
|
||||
datetime(2025, 4, 1, tzinfo=timezone.utc),
|
||||
datetime(2025, 7, 1, tzinfo=timezone.utc))
|
||||
trades = [_wf_trade("2025-02-01 10:00", +1.0),
|
||||
_wf_trade("2025-03-01 10:00", -0.5)]
|
||||
stats = _gb.walkforward_stats(trades, [good, empty])
|
||||
assert not stats["folds"][0]["flagged"], "PF 2.0 must not be flagged"
|
||||
assert stats["folds"][1]["test"].n_trades == 0
|
||||
assert not stats["folds"][1]["flagged"], "empty fold must not be flagged"
|
||||
assert stats["n_flagged"] == 0
|
||||
assert _gb._fmt_pf(stats["folds"][1]["test"]) == "—"
|
||||
|
||||
|
||||
@test("walk-forward: aggregated OOS dedupes overlapping test folds")
|
||||
def _():
|
||||
import gpu_backtest as _gb
|
||||
f1 = (datetime(2024, 1, 1, tzinfo=timezone.utc),
|
||||
datetime(2025, 1, 1, tzinfo=timezone.utc),
|
||||
datetime(2025, 1, 1, tzinfo=timezone.utc),
|
||||
datetime(2025, 4, 1, tzinfo=timezone.utc))
|
||||
f2 = (datetime(2024, 2, 1, tzinfo=timezone.utc),
|
||||
datetime(2025, 2, 1, tzinfo=timezone.utc),
|
||||
datetime(2025, 2, 1, tzinfo=timezone.utc),
|
||||
datetime(2025, 5, 1, tzinfo=timezone.utc))
|
||||
trades = [
|
||||
_wf_trade("2025-01-05 10:00", -1.0), # f1 test only
|
||||
_wf_trade("2025-02-15 10:00", +0.5), # BOTH test windows
|
||||
_wf_trade("2025-04-15 10:00", +2.0), # f2 test only
|
||||
]
|
||||
stats = _gb.walkforward_stats(trades, [f1, f2])
|
||||
o = stats["oos"]
|
||||
assert o.n_trades == 3, "overlapping-window trade must count exactly once"
|
||||
assert abs(o.total_r - 1.5) < 1e-9
|
||||
assert abs(o.max_drawdown_r - 1.0) < 1e-9 # chronological OOS equity
|
||||
assert stats["n_flagged"] == 1 # f1 PF 0.5; f2 has no losses
|
||||
|
||||
|
||||
@test("walk-forward: CLI — off by default, 12/3/3 defaults, short window refused")
|
||||
def _():
|
||||
import gpu_backtest as _gb
|
||||
a = _gb._parse_args(["--pair", "XAUUSD"])
|
||||
assert a.walk_forward is False
|
||||
assert (a.wf_train_months, a.wf_test_months, a.wf_step_months) == (12, 3, 3)
|
||||
a = _gb._parse_args(["--pair", "XAUUSD", "--walk-forward",
|
||||
"--wf-train-months", "6", "--wf-test-months", "2",
|
||||
"--wf-step-months", "1"])
|
||||
assert a.walk_forward is True
|
||||
assert (a.wf_train_months, a.wf_test_months, a.wf_step_months) == (6, 2, 1)
|
||||
# Default --days 180 can't fit train 12 + test 3 → refuse before any
|
||||
# data loading.
|
||||
try:
|
||||
_gb.main(["--pair", "XAUUSD", "--walk-forward"])
|
||||
assert False, "short window must raise SystemExit"
|
||||
except SystemExit as e:
|
||||
assert "shorter than one fold" in str(e)
|
||||
try:
|
||||
_gb.main(["--pair", "XAUUSD", "--walk-forward",
|
||||
"--wf-train-months", "0"])
|
||||
assert False, "zero train months must raise SystemExit"
|
||||
except SystemExit as e:
|
||||
assert "must all be > 0" in str(e)
|
||||
|
||||
|
||||
class _WFMockProvider:
|
||||
"""Offline provider stand-in for fetch_history cache-write tests."""
|
||||
name = "mockprov"
|
||||
|
||||
def __init__(self, df):
|
||||
self.df = df
|
||||
self.fetches = 0
|
||||
|
||||
def fetch_range(self, symbol, tf, start, end):
|
||||
self.fetches += 1
|
||||
return self.df
|
||||
|
||||
def validate_and_clean(self, df, symbol, tf):
|
||||
return df
|
||||
|
||||
|
||||
@test("fetch_history: refuses to recreate a baseline-pinned parquet")
|
||||
def _():
|
||||
import tempfile
|
||||
from pathlib import Path as _Path
|
||||
import gpu_backtest as _gb
|
||||
old_cache, old_base = _gb.CACHE_DIR, _gb.BASELINE_ROOT
|
||||
with tempfile.TemporaryDirectory() as td:
|
||||
_gb.CACHE_DIR = _Path(td) / "cache"
|
||||
_gb.BASELINE_ROOT = _Path(td) / "baseline"
|
||||
try:
|
||||
start = datetime(2020, 1, 1, tzinfo=timezone.utc)
|
||||
end = datetime(2020, 6, 1, tzinfo=timezone.utc)
|
||||
pinned = _gb._cache_path("XAU/USD", Timeframe.H1, start, end,
|
||||
"mockprov").name
|
||||
mdir = _gb.BASELINE_ROOT / "abc1234"
|
||||
mdir.mkdir(parents=True)
|
||||
(mdir / "manifest.json").write_text(
|
||||
json.dumps({"cache_files": {pinned: "0" * 64}}))
|
||||
prov = _WFMockProvider(make_ohlcv(30, seed=7))
|
||||
try:
|
||||
_gb.fetch_history(prov, "XAU/USD", Timeframe.H1, start, end)
|
||||
assert False, "must refuse to write a baseline-pinned name"
|
||||
except RuntimeError as e:
|
||||
assert "baseline" in str(e)
|
||||
assert not (_gb.CACHE_DIR / pinned).exists(), \
|
||||
"refused write must leave nothing behind"
|
||||
# A different window is a different filename → allowed.
|
||||
end2 = datetime(2020, 7, 1, tzinfo=timezone.utc)
|
||||
out = _gb.fetch_history(prov, "XAU/USD", Timeframe.H1, start, end2)
|
||||
assert len(out) == 30
|
||||
other = _gb._cache_path("XAU/USD", Timeframe.H1, start, end2,
|
||||
"mockprov").name
|
||||
assert (_gb.CACHE_DIR / other).exists()
|
||||
finally:
|
||||
_gb.CACHE_DIR, _gb.BASELINE_ROOT = old_cache, old_base
|
||||
|
||||
|
||||
@test("fetch_history: new parquet sha256 lands in data_manifest.json "
|
||||
"(baseline cache_files format); cache hits never refetch or rehash")
|
||||
def _():
|
||||
import hashlib
|
||||
import tempfile
|
||||
from pathlib import Path as _Path
|
||||
import gpu_backtest as _gb
|
||||
old_cache, old_base = _gb.CACHE_DIR, _gb.BASELINE_ROOT
|
||||
with tempfile.TemporaryDirectory() as td:
|
||||
_gb.CACHE_DIR = _Path(td) / "cache"
|
||||
_gb.BASELINE_ROOT = _Path(td) / "baseline" # no manifests → no pins
|
||||
try:
|
||||
start = datetime(2020, 1, 1, tzinfo=timezone.utc)
|
||||
end = datetime(2020, 6, 1, tzinfo=timezone.utc)
|
||||
prov = _WFMockProvider(make_ohlcv(30, seed=8))
|
||||
_gb.fetch_history(prov, "XAU/USD", Timeframe.H1, start, end)
|
||||
name = _gb._cache_path("XAU/USD", Timeframe.H1, start, end,
|
||||
"mockprov").name
|
||||
mpath = _gb.CACHE_DIR / "data_manifest.json"
|
||||
manifest = json.loads(mpath.read_text())
|
||||
assert "cache_files" in manifest and "updated_utc" in manifest
|
||||
want = hashlib.sha256(
|
||||
(_gb.CACHE_DIR / name).read_bytes()).hexdigest()
|
||||
assert manifest["cache_files"] == {name: want}, \
|
||||
"manifest must map filename → sha256 like the baseline format"
|
||||
# Second call: cache hit — no refetch, manifest untouched.
|
||||
before = mpath.read_text()
|
||||
_gb.fetch_history(prov, "XAU/USD", Timeframe.H1, start, end)
|
||||
assert prov.fetches == 1, "cache hit must not refetch"
|
||||
assert mpath.read_text() == before, \
|
||||
"cache hit must not rewrite the data manifest"
|
||||
finally:
|
||||
_gb.CACHE_DIR, _gb.BASELINE_ROOT = old_cache, old_base
|
||||
|
||||
|
||||
# ══════════════════════════════════════════════════════════════════════
|
||||
# SUMMARY
|
||||
# ══════════════════════════════════════════════════════════════════════
|
||||
|
||||
Reference in New Issue
Block a user