diff --git a/.gitignore b/.gitignore index 091d623..b2750c0 100644 --- a/.gitignore +++ b/.gitignore @@ -26,12 +26,7 @@ Thumbs.db # Backtesting data (keep scripts, ignore generated files) backtest/cache/ backtest/*.csv -backtest/research/*.parquet -backtest/research/duka_m5_boundary_spreads.csv backtest/__pycache__/ .env.* logs/ -backtest/research/commit_ladder_2026-08/ -backtest/research/llm_news_eval_2026-08/ - backtest/research/sizing_2026-09/ diff --git a/CLAUDE.md b/CLAUDE.md index 907ebed..b7b227f 100644 --- a/CLAUDE.md +++ b/CLAUDE.md @@ -37,7 +37,7 @@ python backtest/gpu_backtest.py --pair XAUUSD --start 2025-01-01 --end 2025-06-0 **`test_all.py` is NOT pytest** — it uses a custom `@test("name")` decorator that runs each test immediately at import; don't grep for `def test_` or try `pytest test_all.py`. Run the whole file; it prints per-module results and exits non-zero on failure. There is no way to run a single test short of commenting modules out. Tests must stay green; every new gate/knob needs tests. Signal-log fixtures: use a WEEKDAY signal bar when a test exercises candle continuity — the outcome model treats weekend and 21:00 UTC gaps as scheduled, so Saturday-dated bars never produce a gap. `_candles_json` stores `df.tail(14)`, so a test snapshot longer than 14 bars silently drops its head. When patching weekend logic in tests, patch `providers.base._is_forex_weekend` — but know there are THREE independent signal-path weekend implementations (`providers/base.py`, `signal_aggregator.py`, and an inline copy in `backtest/gpu_backtest.py:_is_xau_blocked_by_calendar`) plus the advisory-only continuity copy in `advisory_outcome.py`; patching one does not reach the others. -Python 3.12; a `venv/` exists in-repo. Dependencies in `requirements.txt` (numpy, pandas, requests, twelvedata, python-telegram-bot ≥20, websockets). scikit-learn and lightgbm are installed in the dev `venv/` for `backtest/research/` only — never add them to `requirements.txt` or import them from live code; anything shipped must be numpy-only. +Python 3.12; a `venv/` exists in-repo. Dependencies in `requirements.txt` (numpy, pandas, requests, twelvedata, python-telegram-bot ≥20, websockets). scikit-learn and lightgbm are installed in the dev `venv/` for ad-hoc research only — never add them to `requirements.txt` or import them from live code; anything shipped must be numpy-only. ## Configuration & credentials @@ -96,11 +96,11 @@ Signal currency everywhere is `SignalStrength` (−2..+2). DataFrames are UTC tz ## Current state notes (dated per entry) - Oanda v20 is the live feed (`data_providers.active: "oanda"`). The 2026-08-04 re-pin made the 27 `backtest/oanda_*.csv` books the baseline; all TD-era books are superseded. (2026-08-14) The re-pin M5 book `oanda_m5_12mo_065_nopb_20260814.csv` (+106.82R PF 1.51) is itself SUPERSEDED by the 2026-09-03 book in the entry below; the M15 reference is unchanged since 2026-08-04: `oanda_m15_12mo_065.csv` (397 / +73.93R / PF 1.53). See `backtest/research/repin_verification_2026-08-14.md`. -- Cost reporting is dual: $0.65 is the measured-Oanda realistic headline ($0.65-$0.72 round trip), while $0.30 remains for cross-book comparability and as the live TP spread-gate input. Raising `round_trip_cost` from $0.30 would move the TP floor from about $9 to $19.50 and requires full rule-1 revalidation. The $0.65 figure comes from the 2026-07-24 Dukascopy tick study (`backtest/research/duka_spread.py` + `duka_spread_by_hour.csv`, kept as the evidence driver although nothing imports it). +- Cost reporting is dual: $0.65 is the measured-Oanda realistic headline ($0.65-$0.72 round trip), while $0.30 remains for cross-book comparability and as the live TP spread-gate input. Raising `round_trip_cost` from $0.30 would move the TP floor from about $9 to $19.50 and requires full rule-1 revalidation. The $0.65 figure comes from the 2026-07-24 Dukascopy tick study (9M ticks over 12 months; its tooling was removed 2026-09-06 — the conclusion lives here and in memory). - Twelve Data is signal-incompatible with Oanda (Phase 1: about 19% M5 bar-direction disagreement and a systematic ~$1 high bias; see `backtest/research/feed_divergence_2026-08-14.md`). On an Oanda candle outage, stop scanning rather than switching to TD candles. -- Tiingo is streaming-only (`data_providers.streaming: "tiingo"`), not a designated candle fallback. Oanda-Tiingo signal parity is documented in `backtest/research/signal_parity_2026-08-14.md`; Phase 4's two-week live shadow scan remains a manual operational step. +- Tiingo is streaming-only (`data_providers.streaming: "tiingo"`), not a designated candle fallback. Oanda-Tiingo signal parity is documented in `backtest/research/signal_parity_2026-08-14.md`; Phase 4's two-week live shadow scan (`scripts/shadow_scan.py`) remains a manual operational step. - Human-facing docs in `docs/` (`user_guide.html`, `beginner_guide.html` + PDF builds) must be kept in sync when signal behavior changes. As of 2026-09-06 the PDF builds are BEHIND the HTML (violent-move watch entry added without a PDF rebuild). - 2026-09-03 signal-log campaign shipped two new per-TF gates (`backtest/research/signal_log_campaign_2026-09.md`): M5 `sell_blocked_regimes=["transitioning"]` and H1 `sell_min_quality=[]` (all H1 shorts blocked). Current-HEAD M5 canonical book is now `backtest/oanda_m5_12mo_065_nopb_20260903.csv` (557 / +112.21R / PF 1.56, halves +55.9/+56.4); H1 reference `backtest/oanda_h1_12mo_065_20260903.csv` (91 / +22.2R / PF 1.79; 5yr +49.1R PF 1.46; OOS +16.7R PF 1.74). -- 2026-09-06 stake-sizing campaign (`backtest/research/sizing_campaign_2026-09.md`): a walk-forward "second opinion" stake multiplier (meta-labelling on 65 decision-time features, 3-year Oanda books at $0.65) FAILED the pre-registered rule on M5 (clean null) and M15 (lift inside the label-shuffle null, explained by BUY-vs-SELL). Nothing that changes numbers shipped. Behaviour-neutral tooling remains: `position_sizing.build_features` + `--export-features` (default export stays 63 columns), `SignalReport.penalty_tags/penalty_total/momentum_fading`, `TradeSetup.stake_multiplier` (None; reserved), `backtest/research/{book_report,burst_map,sizing_model}.py`. Stage-B facts: M5 stretch measures are POSITIVE in both halves (do not re-propose exhaustion sizing on M5); M15 exhaustion tag is weakly negative (≈ +3R/yr, noise); bursts cluster at 12–15 UTC and 01 UTC bar-close and do NOT hurt entries. +- 2026-09-06 stake-sizing campaign (`backtest/research/sizing_campaign_2026-09.md`): a walk-forward "second opinion" stake multiplier (meta-labelling on 65 decision-time features, 3-year Oanda books at $0.65) FAILED the pre-registered rule on M5 (clean null) and M15 (lift inside the label-shuffle null, explained by BUY-vs-SELL). Nothing that changes numbers shipped. Behaviour-neutral tooling remains: `position_sizing.build_features` + `--export-features` (default export stays 63 columns), `SignalReport.penalty_tags/penalty_total/momentum_fading`, `TradeSetup.stake_multiplier` (None; reserved) (the campaign's research scripts were removed 2026-09-06; git history has them). Stage-B facts: M5 stretch measures are POSITIVE in both halves (do not re-propose exhaustion sizing on M5); M15 exhaustion tag is weakly negative (≈ +3R/yr, noise); bursts cluster at 12–15 UTC and 01 UTC bar-close and do NOT hurt entries. - Auto-execution (oanda_daemon.py / oanda_executor.py) was removed 2026-09-03 — the owner is not auto-trading; last commit containing it is 31002cb. Book-parity rules that lived in its docstring (no broker TP, 5-bar cooldown, once-per-bar trail) remain the backtester's model. -- 2026-09-06 stale-code cleanup (behaviour-neutral; suite green, M5/H1 12-month books byte-identical after every signal-side commit): `main.py --timeframe` now accepts M5/M15/H1/H4 and takes its confirmation TFs from the pair factory (the CLI's private table had paired M5 with M1+M15); `PlatformConfig.pip_value` is gone (`PAIR_INFO` is the source); removed as dead or superseded: `_get_affected_pairs`/`event_impact_on_pairs`, `SignalLogger.advance_with_bars`/`path`/`news_for`, `reset_llm_client`, the Telegram scan-summary/keyboard helpers and duplicate confirmation map, the 2026-07 Oanda-vs-TD research pager+study, `burst_profile.py`, `hour_analysis.py`, `scripts/compare_oanda_tiingo.py`, the never-implemented LLM-provider plan and the merged config.json spec. `docs/architecture_diagram.py` only writes the PNG when run as a script. +- 2026-09-06 stale-code cleanup (behaviour-neutral; suite green, M5/H1 12-month books byte-identical after every signal-side commit): `main.py --timeframe` now accepts M5/M15/H1/H4 and takes its confirmation TFs from the pair factory (the CLI's private table had paired M5 with M1+M15); `PlatformConfig.pip_value` is gone (`PAIR_INFO` is the source); removed as dead or superseded: `_get_affected_pairs`/`event_impact_on_pairs`, `SignalLogger.advance_with_bars`/`path`/`news_for`, `reset_llm_client`, the Telegram scan-summary/keyboard helpers and duplicate confirmation map, the 2026-07 Oanda-vs-TD research pager+study, `burst_profile.py`, `hour_analysis.py`, `scripts/compare_oanda_tiingo.py`, the never-implemented LLM-provider plan and the merged config.json spec. Second pass the same day (owner directive: keep only what the system needs): every research script and data file under `backtest/research/` (book_report, book_diff, the burst tools, sizing_model, commit_ladder, llm_news_eval + golden CSV, ff_calendar_scraper, calendar_impact, watch_rule_eval, the Dukascopy spread study + CSVs, the panel dry-test JSONL), the two completed study scripts in `scripts/`, the architecture diagram + generator and the data-source test plan were deleted; the campaign `.md` reports stay as the decision ledger. Git history before `953c5ad` has all of it. diff --git a/backtest/instructions_for_backtest.txt b/backtest/instructions_for_backtest.txt index 35003d8..eb63be6 100644 --- a/backtest/instructions_for_backtest.txt +++ b/backtest/instructions_for_backtest.txt @@ -9,21 +9,18 @@ Every backtest report must record: - Per trade (the exported CSV already carries these): entry/exit time and price, exit reason, R multiple, MFE/MAE, entry reasons, and every indicator's signal / value / agreement at decision time. -- Breakdowns by session/hour, weekday, regime and volatility regime - (backtest/research/book_report.py) and, for any candidate change, the - removed/added/shifted trade diff against the control book - (backtest/research/book_diff.py). +- Breakdowns by session/hour, weekday, regime and volatility regime, and, + for any candidate change, the removed/added/shifted trade diff against the + control book (the 5-bar cooldown displaces entries, so judge run-level + numbers, never the added-trade bucket alone). - The md5 of the exported book whenever the run is a parity or regression check. Replays go through the real signal code (analyze_from_dataframes) — never a re-implementation. - --- 2026-09-06: decision-time feature export --- - `--export-features` appends stake_multiplier, r_sized and the 65 `f_*` decision-time features (position_sizing.FEATURE_ORDER) AFTER the 63 pinned columns; default export is - byte-identical to the pinned books. Use backtest/research/book_report.py for weekday / - hour x weekday / session / streak / rolling-WR / feature-quintile (both halves) tables, - backtest/research/burst_map.py for the burst-by-hour map, and - backtest/research/sizing_model.py for the walk-forward + shuffle-control harness. + byte-identical to the pinned books. - Any sizing report must show the FLAT book and the SIZED book side by side and the - label-shuffle null; the 2026-09-06 campaign verdict (nothing ships) is the reference. + label-shuffle null; the 2026-09-06 campaign verdict (nothing ships, + backtest/research/sizing_campaign_2026-09.md) is the reference. diff --git a/backtest/research/__init__.py b/backtest/research/__init__.py deleted file mode 100644 index 575b2ff..0000000 --- a/backtest/research/__init__.py +++ /dev/null @@ -1,5 +0,0 @@ -"""Research tooling: calendar sourcing/store and edge-screening utilities. - -Nothing here is imported by the live bot; these modules support -backtesting and edge validation only. -""" diff --git a/backtest/research/book_diff.py b/backtest/research/book_diff.py deleted file mode 100644 index c143131..0000000 --- a/backtest/research/book_diff.py +++ /dev/null @@ -1,86 +0,0 @@ -"""Trade-book diff: relaxed/gated run vs its control. - -Usage: - python backtest/research/book_diff.py CONTROL.csv CANDIDATE.csv BAR_MINUTES - -Decision numbers are RUN-LEVEL (ΔtotalR, ΔPF, ΔmaxDD, Δtrades) — never -the added-trade bucket alone: the 5-bar entry cooldown means new -entries displace existing ones. The added/removed decomposition and the -per-decision-hour histogram of added trades are diagnostics. - -Shifted trades = removed+added pairs in the same direction within -±5 bars (cooldown displacement, not a genuinely new/lost signal). -""" -import sys - -import pandas as pd - - -def _metrics(df): - r = df["r_multiple"] - gains, losses = r[r > 0].sum(), -r[r <= 0].sum() - cum = r.cumsum() - return dict(trades=len(df), total_r=r.sum(), - pf=gains / max(losses, 1e-9), - wr=(r > 0).mean() * 100 if len(df) else 0.0, - max_dd=(cum.cummax() - cum).max() if len(cum) else 0.0) - - -def diff_books(control_csv, candidate_csv, bar_minutes, split="2026-01-29"): - ctrl, cand = pd.read_csv(control_csv), pd.read_csv(candidate_csv) - for df in (ctrl, cand): - df["_t"] = pd.to_datetime(df["entry_time"], utc=True, format="mixed") - df["_key"] = df["entry_time"].astype(str) + "|" + df["direction"] - - added = cand[~cand["_key"].isin(ctrl["_key"])].copy() - removed = ctrl[~ctrl["_key"].isin(cand["_key"])].copy() - - # cooldown displacement: same-direction removed/added within ±5 bars - tol = pd.Timedelta(minutes=5 * bar_minutes) - shifted = 0 - rem_pool = removed.copy() - for _, a in added.iterrows(): - m = rem_pool[(rem_pool["direction"] == a["direction"]) - & ((rem_pool["_t"] - a["_t"]).abs() <= tol)] - if len(m): - shifted += 1 - rem_pool = rem_pool.drop(m.index[0]) - - mc, mk = _metrics(ctrl), _metrics(cand) - split_ts = pd.Timestamp(split, tz="UTC") - halves = (cand[cand["_t"] < split_ts]["r_multiple"].sum(), - cand[cand["_t"] >= split_ts]["r_multiple"].sum()) - hour_hist = None - if len(added): - added["dec_hour"] = (added["_t"] - - pd.Timedelta(minutes=bar_minutes)).dt.hour - hour_hist = added.groupby("dec_hour")["r_multiple"] \ - .agg(["count", "sum"]).round(2) - return dict( - control=mc, candidate=mk, - delta_r=mk["total_r"] - mc["total_r"], - delta_pf=mk["pf"] - mc["pf"], - delta_dd=mk["max_dd"] - mc["max_dd"], - delta_trades=mk["trades"] - mc["trades"], - added_n=len(added), added_r=added["r_multiple"].sum(), - removed_n=len(removed), removed_r=removed["r_multiple"].sum(), - shifted=shifted, halves=halves, added_by_hour=hour_hist, - ) - - -if __name__ == "__main__": - d = diff_books(sys.argv[1], sys.argv[2], int(sys.argv[3])) - c, k = d["control"], d["candidate"] - print(f"control : {c['trades']:4d} trades {c['total_r']:+7.1f}R " - f"PF {c['pf']:.2f} WR {c['wr']:.1f}% DD {c['max_dd']:.1f}") - print(f"candidate: {k['trades']:4d} trades {k['total_r']:+7.1f}R " - f"PF {k['pf']:.2f} WR {k['wr']:.1f}% DD {k['max_dd']:.1f}") - print(f"ΔR {d['delta_r']:+.1f} ΔPF {d['delta_pf']:+.2f} " - f"ΔDD {d['delta_dd']:+.1f} Δtrades {d['delta_trades']:+d}") - print(f"added {d['added_n']} ({d['added_r']:+.1f}R) " - f"removed {d['removed_n']} ({d['removed_r']:+.1f}R) " - f"shifted≈{d['shifted']}") - print(f"candidate halves: H1 {d['halves'][0]:+.1f}R H2 {d['halves'][1]:+.1f}R") - if d["added_by_hour"] is not None: - print("\nadded trades by decision hour:") - print(d["added_by_hour"].to_string()) diff --git a/backtest/research/book_report.py b/backtest/research/book_report.py deleted file mode 100644 index 8855c33..0000000 --- a/backtest/research/book_report.py +++ /dev/null @@ -1,581 +0,0 @@ -"""Full descriptive report of a backtest trade book (research only). - -Usage: - python backtest/research/book_report.py --book BOOK.csv --bar-minutes N \ - [--split YYYY-MM-DD|median] [--parquet-4h PATH] [--parquet-1d PATH] \ - [--out DIR] - -Prints (and writes to DIR/report.txt) headline metrics for the whole book -and for each half, group tables (weekday / decision hour / session / quality -/ regime / vol_regime / direction / strategy / exit_reason / indicators_agree), -win-loss streaks, a rolling win-rate summary, and a quantile-bucket screen of -every numeric decision-time feature the book carries (`f_*` / `x_*` columns -plus score, confidence, rr_planned, indicators_agree/oppose and `ind_*_value`). - -The screen is deliberately half-split: a feature only "passes" when the -top-bucket-minus-bottom-bucket mean-R gap has the SAME SIGN in both halves of -the book and both extreme buckets are populated in both halves. Bin edges are -fit on the WHOLE book (so the halves share edges and stay comparable) — that -leaks a little cross-half information into the edges themselves, which is why -the sign test, not the size of the gap, is the thing to trust. - -Everything here is measurement, not simulation: it never re-runs the signal -engine, it only describes a book that gpu_backtest.py already produced. -Analysis tooling only — never imported by the live code. -""" -from __future__ import annotations - -import argparse -import math -import os -from typing import Iterable - -import numpy as np -import pandas as pd - -SESSIONS = ( - ("asian", 3, 6), - ("london", 7, 12), - ("overlap", 13, 16), - ("ny", 17, 22), -) - -# Numeric non-`f_`/`x_` columns that are known at DECISION time and are -# therefore legal screen candidates. bars_held / r_multiple / mfe_r / mae_r -# are post-entry outcomes and must never appear here. -BASE_SCREEN_COLS = ("score", "confidence", "rr_planned", - "indicators_agree", "indicators_oppose") - - -def session_of(hour: int) -> str: - for name, lo, hi in SESSIONS: - if lo <= hour <= hi: - return name - return "off" - - -def feature_columns(df: pd.DataFrame) -> list[str]: - """Decision-time feature columns carried by this book, generically.""" - extra = [c for c in df.columns if c.startswith(("f_", "x_"))] - base = [c for c in BASE_SCREEN_COLS if c in df.columns] - ind = [c for c in df.columns - if c.startswith("ind_") and c.endswith("_value")] - out, seen = [], set() - for c in extra + base + ind: - if c not in seen: - seen.add(c) - out.append(c) - return out - - -# ───────────────────────────── loading ────────────────────────────── - -def load_book(csv: str, bar_minutes: int = 5, - split: str = "median") -> pd.DataFrame: - """Read a gpu_backtest trade book and add the decision-time columns. - - `decision_time` = entry_time − one bar: entries fill at the NEXT bar's - open, so the bar the signal actually decided on closed one bar earlier. - That is the clock blocked_hours_utc and every session filter evaluate. - `half` is 1 before the split instant and 2 at/after it. - """ - df = pd.read_csv(csv) - for col in ("entry_time", "exit_time"): - if col in df.columns: - df[col] = pd.to_datetime(df[col], utc=True, format="mixed") - df["decision_time"] = df["entry_time"] - pd.Timedelta(minutes=bar_minutes) - df["decision_hour"] = df["decision_time"].dt.hour - df["weekday"] = df["decision_time"].dt.weekday - df["session"] = df["decision_hour"].map(session_of) - df = df.sort_values("entry_time", kind="mergesort").reset_index(drop=True) - df["half"] = _halves(df["entry_time"], split) - return df - - -def split_instant(entry_time: pd.Series, split: str) -> pd.Timestamp: - if split == "median": - return entry_time.median() - return pd.Timestamp(split, tz="UTC") - - -def _halves(entry_time: pd.Series, split: str) -> pd.Series: - cut = split_instant(entry_time, split) - return np.where(entry_time < cut, 1, 2) - - -# ───────────────────────────── metrics ────────────────────────────── - -def base_metrics(r: pd.Series, df: pd.DataFrame | None = None) -> dict: - """Headline stats for a series of R multiples. - - max_dd uses the gpu_backtest.compute_metrics convention: a 0.0 is - prepended to the cumulative-R curve so an opening losing streak counts - as drawdown from flat, not from an already-negative first trade. - pf/expectancy split on r > 0 / r < 0 (exact zeros are neither). - """ - r = pd.Series(r, dtype="float64").dropna() - n = int(len(r)) - if n == 0: - return {"n": 0, "total_r": 0.0, "mean_r": 0.0, "median_r": 0.0, - "wr": 0.0, "pf": 0.0, "max_dd": 0.0, "sharpe": 0.0, - "expectancy": 0.0, "expectancy_alt": 0.0, "recovery": 0.0, - "avg_bars": float("nan"), "avg_mfe": float("nan"), - "avg_mae": float("nan")} - wins, losses = r[r > 0], r[r < 0] - gross_win = float(wins.sum()) - gross_loss = float(-losses.sum()) - cum = np.concatenate(([0.0], np.cumsum(r.to_numpy()))) - max_dd = float((np.maximum.accumulate(cum) - cum).max()) - std = float(r.std(ddof=0)) - mean_r = float(r.mean()) - wr = len(wins) / n - lr = len(losses) / n - avg_win = float(wins.mean()) if len(wins) else 0.0 - avg_loss = float(-losses.mean()) if len(losses) else 0.0 - expectancy_alt = wr * avg_win - lr * avg_loss - # Identity check: the textbook expectancy form must equal the plain mean - # whenever no trade is exactly flat (and it does even then, since a 0 R - # trade contributes 0 to both sides). - assert math.isclose(expectancy_alt, mean_r, rel_tol=1e-9, abs_tol=1e-9), ( - f"expectancy mismatch: {expectancy_alt} vs mean {mean_r}") - out = { - "n": n, - "total_r": float(r.sum()), - "mean_r": mean_r, - "median_r": float(r.median()), - "wr": wr * 100.0, - "pf": (gross_win / gross_loss) if gross_loss > 0 else float("inf"), - "max_dd": max_dd, - "sharpe": (mean_r / std) if std > 0 else 0.0, - "expectancy": mean_r, - "expectancy_alt": expectancy_alt, - "recovery": (float(r.sum()) / max_dd) if max_dd > 0 else float("inf"), - "avg_bars": float("nan"), - "avg_mfe": float("nan"), - "avg_mae": float("nan"), - } - if df is not None: - for key, col in (("avg_bars", "bars_held"), ("avg_mfe", "mfe_r"), - ("avg_mae", "mae_r")): - if col in df.columns: - out[key] = float(pd.to_numeric(df[col], - errors="coerce").mean()) - return out - - -def _pf(r: pd.Series) -> float: - """Profit factor (loss side is r <= 0).""" - pos = r[r > 0].sum() - neg = -r[r <= 0].sum() - return float(pos / max(neg, 1e-9)) - - -def group_table(df: pd.DataFrame, by, min_n: int = 10) -> pd.DataFrame: - """Per-group P&L with a half-1 / half-2 sign-stability flag.""" - keys = [by] if isinstance(by, str) else list(by) - rows = [] - for key, sub in df.groupby(keys, dropna=False, sort=True): - r = sub["r_multiple"] - h1, h2 = sub[sub["half"] == 1]["r_multiple"], sub[sub["half"] == 2]["r_multiple"] - m1 = float(h1.mean()) if len(h1) else float("nan") - m2 = float(h2.mean()) if len(h2) else float("nan") - stable = bool(len(h1) >= min_n and len(h2) >= min_n - and np.sign(m1) != 0 and np.sign(m1) == np.sign(m2)) - rec = {} - # pandas >= 2.2 hands back a 1-tuple even for a single group key. - vals = key if isinstance(key, tuple) else (key,) - for k, v in zip(keys, vals): - rec[k] = v - rec.update({ - "n": int(len(sub)), - "total_r": round(float(r.sum()), 2), - "mean_r": round(float(r.mean()), 3), - "wr": round(float((r > 0).mean() * 100), 2), - "pf": round(_pf(r), 2), - "n_h1": int(len(h1)), - "total_r_h1": round(float(h1.sum()), 2), - "mean_r_h1": round(m1, 3) if len(h1) else float("nan"), - "n_h2": int(len(h2)), - "total_r_h2": round(float(h2.sum()), 2), - "mean_r_h2": round(m2, 3) if len(h2) else float("nan"), - "sign_stable": stable, - }) - rows.append(rec) - out = pd.DataFrame(rows) - if not out.empty: - out = out.set_index(keys) - return out - - -def streaks(df: pd.DataFrame) -> dict: - """Longest win/loss runs and the longest stretch under water (in trades).""" - d = df.sort_values("entry_time", kind="mergesort") - r = d["r_multiple"].to_numpy(dtype="float64") - best_w = best_l = cur_w = cur_l = 0 - for x in r: - if x > 0: - cur_w, cur_l = cur_w + 1, 0 - elif x < 0: - cur_l, cur_w = cur_l + 1, 0 - else: - cur_w = cur_l = 0 - best_w, best_l = max(best_w, cur_w), max(best_l, cur_l) - cum = np.concatenate(([0.0], np.cumsum(r))) - peak = np.maximum.accumulate(cum) - flat, longest = 0, 0 - for i in range(1, len(cum)): - flat = 0 if cum[i] >= peak[i] else flat + 1 - longest = max(longest, flat) - return {"max_win_streak": best_w, "max_loss_streak": best_l, - "longest_flat_trades": longest} - - -def rolling_wr(df: pd.DataFrame, window: int = 20) -> pd.Series: - d = df.sort_values("entry_time", kind="mergesort") - s = (d["r_multiple"] > 0).astype(float).rolling(window).mean() * 100 - s.index = d["entry_time"].to_numpy() - return s - - -def equity_curve(df: pd.DataFrame) -> pd.DataFrame: - d = df.sort_values("entry_time", kind="mergesort") - cum = d["r_multiple"].cumsum() - peak = pd.concat([pd.Series([0.0]), cum]).cummax().iloc[1:] - peak.index = cum.index - return pd.DataFrame({"entry_time": d["entry_time"].to_numpy(), - "cum_r": cum.to_numpy(), - "drawdown": (peak - cum).to_numpy()}) - - -# ─────────────────────── quantile bucket screen ───────────────────── - -def quantile_buckets(df: pd.DataFrame, col: str, q: int = 5, - min_n: int = 10) -> pd.DataFrame: - """Equal-frequency buckets of `col` with per-half mean R. - - Bin edges are fit on the WHOLE book (both halves share edges). - Result carries gap_h1 / gap_h2 / sign_stable in `.attrs` and as - repeated columns so the CSV is self-describing. - """ - v = pd.to_numeric(df.get(col), errors="coerce") - d = df.loc[v.notna()].copy() - d["_v"] = v.loc[v.notna()] - empty = pd.DataFrame(columns=["bin", "lo", "hi", "n", "mean_r", "pf", "wr", - "n_h1", "mean_r_h1", "n_h2", "mean_r_h2", - "gap_h1", "gap_h2", "sign_stable"]) - empty.attrs = {"feature": col, "gap_h1": float("nan"), - "gap_h2": float("nan"), "sign_stable": False, "n": 0, - "mode": "empty"} - if d.empty: - return empty - - nuniq = int(d["_v"].nunique()) - if nuniq < 3: - mode = "values" - d["_bin"] = d["_v"] - order = sorted(d["_v"].unique()) - edges = {b: (float(b), float(b)) for b in order} - else: - mode = "qcut" - binned = pd.qcut(d["_v"], q, duplicates="drop") - d["_bin"] = binned - order = list(binned.cat.categories) - edges = {b: (float(b.left), float(b.right)) for b in order} - - rows = [] - for b in order: - sub = d[d["_bin"] == b] - if sub.empty: - continue - r = sub["r_multiple"] - h1 = sub[sub["half"] == 1]["r_multiple"] - h2 = sub[sub["half"] == 2]["r_multiple"] - lo, hi = edges[b] - rows.append({ - "bin": str(b), "lo": round(lo, 4), "hi": round(hi, 4), - "n": int(len(sub)), "mean_r": round(float(r.mean()), 3), - "pf": round(_pf(r), 2), - "wr": round(float((r > 0).mean() * 100), 2), - "n_h1": int(len(h1)), - "mean_r_h1": round(float(h1.mean()), 3) if len(h1) else float("nan"), - "n_h2": int(len(h2)), - "mean_r_h2": round(float(h2.mean()), 3) if len(h2) else float("nan"), - }) - out = pd.DataFrame(rows) - if out.empty: - return empty - - bot, top = out.iloc[0], out.iloc[-1] - gap_h1 = float(top["mean_r_h1"] - bot["mean_r_h1"]) - gap_h2 = float(top["mean_r_h2"] - bot["mean_r_h2"]) - populated = min(bot["n_h1"], bot["n_h2"], top["n_h1"], top["n_h2"]) >= min_n - signs_ok = (not math.isnan(gap_h1) and not math.isnan(gap_h2) - and np.sign(gap_h1) != 0 - and np.sign(gap_h1) == np.sign(gap_h2)) - sign_stable = bool(populated and signs_ok and len(out) >= 2) - out["gap_h1"] = round(gap_h1, 3) if not math.isnan(gap_h1) else float("nan") - out["gap_h2"] = round(gap_h2, 3) if not math.isnan(gap_h2) else float("nan") - out["sign_stable"] = sign_stable - out.attrs = {"feature": col, "gap_h1": gap_h1, "gap_h2": gap_h2, - "sign_stable": sign_stable, "n": int(len(d)), "mode": mode} - return out - - -def feature_screen(df: pd.DataFrame, q: int = 5, min_n: int = 10, - columns: Iterable[str] | None = None) -> pd.DataFrame: - """Bucket screen over every decision-time numeric feature in the book.""" - cols = list(columns) if columns is not None else feature_columns(df) - rows = [] - for c in cols: - bk = quantile_buckets(df, c, q=q, min_n=min_n) - a = bk.attrs - stable = bool(a.get("sign_stable")) - g1, g2 = a.get("gap_h1", float("nan")), a.get("gap_h2", float("nan")) - min_abs = (min(abs(g1), abs(g2)) if stable else 0.0) - rows.append({"feature": c, - "gap_h1": round(g1, 3) if g1 == g1 else float("nan"), - "gap_h2": round(g2, 3) if g2 == g2 else float("nan"), - "min_abs_gap": round(float(min_abs), 3), - "sign_stable": stable, - "n": int(a.get("n", 0)), - "bins": int(len(bk))}) - out = pd.DataFrame(rows) - if out.empty: - return out - return out.sort_values(["min_abs_gap", "n"], - ascending=[False, False]).reset_index(drop=True) - - -# ────────────────────── higher-timeframe extension ────────────────── - -def wilder_atr(bars: pd.DataFrame, period: int = 14) -> pd.Series: - h, l, c = bars["High"], bars["Low"], bars["Close"] - tr = pd.concat([h - l, (h - c.shift()).abs(), (l - c.shift()).abs()], - axis=1).max(axis=1) - return tr.ewm(alpha=1 / period, min_periods=period, adjust=False).mean() - - -def _ext_series(parquet: str, minutes: int) -> pd.DataFrame: - bars = pd.read_parquet(parquet).sort_index() - bars = bars[~bars.index.duplicated(keep="last")] - ext = ((bars["Close"] - bars["Close"].rolling(20).mean()) - / wilder_atr(bars, 14)) - out = pd.DataFrame({ - "close_time": bars.index + pd.Timedelta(minutes=minutes), - "ext": ext.to_numpy(), - }).dropna().reset_index(drop=True) - # Books parse to datetime64[us, UTC] (format="mixed"), parquet indexes are - # [ns, UTC]; merge_asof refuses mismatched resolutions. - out["close_time"] = out["close_time"].astype("datetime64[ns, UTC]") - return out - - -def add_htf_extension(df: pd.DataFrame, parquet_4h: str | None, - parquet_1d: str | None) -> pd.DataFrame: - """Add direction-signed H4 / D1 stretch-from-mean in ATR units. - - ext = (close − 20-bar SMA of close) / Wilder ATR(14) on that timeframe, - taken from the LAST higher-TF bar whose CLOSE time (bar label + one bar) - is <= decision_time — i.e. only information the live scan could have had. - Signed +1 for BUY, −1 for SELL, so positive always means "extended in the - direction of the trade". - """ - out = df.copy() - sign = np.where(out["direction"].astype(str).str.upper().str.startswith("B"), - 1.0, -1.0) - for col, parquet, minutes in (("x_h4_ext_atr", parquet_4h, 240), - ("x_d1_ext_atr", parquet_1d, 1440)): - if not parquet: - continue - right = _ext_series(parquet, minutes) - left = out[["decision_time"]].reset_index() - left["decision_time"] = left["decision_time"].astype( - "datetime64[ns, UTC]") - left = left.sort_values("decision_time", kind="mergesort") - merged = pd.merge_asof(left, right, left_on="decision_time", - right_on="close_time", direction="backward") - merged["_stale_h"] = ((merged["decision_time"] - merged["close_time"]) - .dt.total_seconds() / 3600.0) - merged = merged.set_index("index").reindex(out.index) - out[col] = merged["ext"].to_numpy() * sign - out.attrs[f"{col}_max_stale_h"] = float( - merged["_stale_h"].max()) if len(merged) else float("nan") - return out - - -# ──────────────────────────── reporting ───────────────────────────── - -def _fmt(v, nd=3): - if isinstance(v, float): - if math.isinf(v): - return "inf" - if math.isnan(v): - return "n/a" - return f"{v:.{nd}f}" - return str(v) - - -def _metrics_block(name: str, m: dict) -> list[str]: - return [ - f" {name:<10} n={m['n']:<5} totalR={_fmt(m['total_r'], 2):>8} " - f"meanR={_fmt(m['mean_r'])} medR={_fmt(m['median_r'])} " - f"WR={_fmt(m['wr'], 2)}% PF={_fmt(m['pf'], 2)}", - f" {'':<10} maxDD={_fmt(m['max_dd'], 2)} recovery={_fmt(m['recovery'], 2)}" - f" sharpe={_fmt(m['sharpe'])} expectancy={_fmt(m['expectancy'])}" - f" (wr·avgW − lr·avgL = {_fmt(m['expectancy_alt'])})", - f" {'':<10} avg bars={_fmt(m['avg_bars'], 1)} " - f"avg MFE={_fmt(m['avg_mfe'])}R avg MAE={_fmt(m['avg_mae'])}R", - ] - - -def build_report(df: pd.DataFrame, book: str, bar_minutes: int, split: str, - out_dir: str, q: int = 5, min_n: int = 10) -> str: - lines: list[str] = [] - add = lines.append - cut = split_instant(df["entry_time"], split) - add("=" * 78) - add(f"BOOK REPORT {os.path.basename(book)}") - add("=" * 78) - add(f"trades : {len(df)}") - add(f"entry window : {df['entry_time'].min()} -> {df['entry_time'].max()}") - add(f"bar minutes : {bar_minutes} (decision_time = entry_time − 1 bar)") - add(f"split rule : {split} -> cut at {cut} " - f"(half 1 = before, half 2 = at/after)") - add(f"half sizes : h1={int((df['half'] == 1).sum())} " - f"h2={int((df['half'] == 2).sum())}") - add("session map : asian 3-6, london 7-12, overlap 13-16, ny 17-22, " - "else off (on DECISION hour, UTC)") - add("bucket edges : quantile bins are fit on the WHOLE book; the two " - "halves share the same edges.") - feats = feature_columns(df) - add(f"screen features : {len(feats)} " - f"({sum(1 for c in feats if c.startswith(('f_', 'x_')))} f_/x_ columns)") - for k, v in df.attrs.items(): - add(f"htf staleness : {k} = {_fmt(v, 1)} h (max age of the matched bar)") - - add("") - add("── HEADLINE METRICS ──") - lines.extend(_metrics_block("full", base_metrics(df["r_multiple"], df))) - for h in (1, 2): - sub = df[df["half"] == h] - lines.extend(_metrics_block(f"half {h}", - base_metrics(sub["r_multiple"], sub))) - - group_specs = [ - ("weekday", "weekday"), ("decision_hour", "decision_hour"), - ("session", "session"), ("quality", "quality"), ("regime", "regime"), - ("vol_regime", "vol_regime"), ("direction", "direction"), - ("strategy", "strategy"), ("exit_reason", "exit_reason"), - ("indicators_agree", "indicators_agree"), - ] - tables = {} - for name, col in group_specs: - if col not in df.columns: - continue - t = group_table(df, col, min_n=min_n) - tables[name] = t - add("") - add(f"── BY {name.upper()} ──") - add(t.to_string() if not t.empty else " (empty)") - - # (weekday, decision_hour) pivots - add("") - add("── weekday x decision_hour : TOTAL R ──") - wh = group_table(df, ["weekday", "decision_hour"], min_n=min_n) - if not wh.empty: - piv_r = wh["total_r"].unstack("decision_hour") - piv_n = wh["n"].unstack("decision_hour") - add(piv_r.fillna(0).round(2).to_string()) - add("") - add("── weekday x decision_hour : N ──") - add(piv_n.fillna(0).astype(int).to_string()) - tables["weekday_hour"] = wh - piv_r.to_csv(os.path.join(out_dir, "groups_weekday_hour_total_r.csv")) - piv_n.to_csv(os.path.join(out_dir, "groups_weekday_hour_n.csv")) - - st = streaks(df) - add("") - add("── STREAKS ──") - add(f" max consecutive wins : {st['max_win_streak']}") - add(f" max consecutive losses : {st['max_loss_streak']}") - add(f" longest flat stretch : {st['longest_flat_trades']} trades " - f"between equity highs") - - rw = rolling_wr(df, 20).dropna() - add("") - add("── ROLLING WIN RATE (20 trades) ──") - if len(rw): - add(f" min={_fmt(rw.min(), 2)}% median={_fmt(rw.median(), 2)}% " - f"max={_fmt(rw.max(), 2)}% n={len(rw)}") - add(" last 5: " + ", ".join( - f"{ts:%Y-%m-%d %H:%M}={v:.2f}%" for ts, v in rw.tail(5).items())) - else: - add(" (fewer than 20 trades)") - - scr = feature_screen(df, q=q, min_n=min_n) - add("") - add("── FEATURE SCREEN (top 25 by min |gap| across halves) ──") - add(" gap = mean R(top bucket) − mean R(bottom bucket), per half; " - "min_abs_gap is 0 unless the sign is stable.") - add(scr.head(25).to_string(index=False) if not scr.empty else " (none)") - - stable = scr[scr["sign_stable"]].head(10) if not scr.empty else scr - add("") - add(f"── BUCKET TABLES FOR THE TOP {len(stable)} SIGN-STABLE FEATURES ──") - if stable.empty: - add(" (no feature has a sign-stable top-vs-bottom gap)") - for feat in stable["feature"]: - bk = quantile_buckets(df, feat, q=q, min_n=min_n) - add("") - add(f" · {feat} (mode={bk.attrs['mode']}, n={bk.attrs['n']}, " - f"gap_h1={_fmt(bk.attrs['gap_h1'])}, " - f"gap_h2={_fmt(bk.attrs['gap_h2'])})") - add(bk.to_string(index=False)) - bk.to_csv(os.path.join(out_dir, f"buckets_{feat}.csv"), index=False) - - # CSV artefacts - for name, t in tables.items(): - t.to_csv(os.path.join(out_dir, f"groups_{name}.csv")) - scr.to_csv(os.path.join(out_dir, "feature_screen.csv"), index=False) - equity_curve(df).to_csv(os.path.join(out_dir, "equity.csv"), index=False) - all_buckets = [] - for c in feats: - bk = quantile_buckets(df, c, q=q, min_n=min_n) - if bk.empty: - continue - bk = bk.copy() - bk.insert(0, "feature", c) - all_buckets.append(bk) - if all_buckets: - pd.concat(all_buckets, ignore_index=True).to_csv( - os.path.join(out_dir, "buckets.csv"), index=False) - return "\n".join(lines) - - -def main() -> None: - ap = argparse.ArgumentParser(description=__doc__.split("\n")[0]) - ap.add_argument("--book", required=True) - ap.add_argument("--bar-minutes", type=int, required=True) - ap.add_argument("--split", default="median", - help="'median' (default) or YYYY-MM-DD") - ap.add_argument("--parquet-4h", default=None) - ap.add_argument("--parquet-1d", default=None) - ap.add_argument("--q", type=int, default=5) - ap.add_argument("--min-n", type=int, default=10) - ap.add_argument("--out", default="backtest/research/book_report_out") - args = ap.parse_args() - - os.makedirs(args.out, exist_ok=True) - df = load_book(args.book, args.bar_minutes, args.split) - if args.parquet_4h or args.parquet_1d: - df = add_htf_extension(df, args.parquet_4h, args.parquet_1d) - text = build_report(df, args.book, args.bar_minutes, args.split, - args.out, q=args.q, min_n=args.min_n) - print(text) - with open(os.path.join(args.out, "report.txt"), "w") as fh: - fh.write(text + "\n") - print(f"\n[written] {os.path.join(args.out, 'report.txt')}") - - -if __name__ == "__main__": - main() diff --git a/backtest/research/burst_indicator_map.py b/backtest/research/burst_indicator_map.py deleted file mode 100644 index 0450ee7..0000000 --- a/backtest/research/burst_indicator_map.py +++ /dev/null @@ -1,121 +0,0 @@ -"""What do the system's OWN indicators read right before a violent bar? Analysis tooling only. - -Uses backtest.gpu_backtest.precompute_indicators (the same series the backtester feeds the -engine) on 3 years of bars; every state is taken at t-1 (known before the bar). Reports burst -rate (range >= 3x prior ATR14) for the NEXT bar and within the NEXT HOUR by indicator state, -by regime, by volatility regime, by weekday x hour, day of month, calendar proximity — each in -both halves — and whether the state predicts the burst's DIRECTION. - - python backtest/research/burst_indicator_map.py PARQUET BAR_MINUTES [--start --end] -""" -import argparse, sys -from pathlib import Path -import numpy as np, pandas as pd -ROOT = Path(__file__).resolve().parents[2]; sys.path.insert(0, str(ROOT)); sys.path.insert(0, str(ROOT / "backtest")) -from gpu_backtest import precompute_indicators -from config import TechnicalConfig - -CAL = ROOT / "backtest" / "cache" / "calendar.parquet" - - -def wilder_atr(df, period=14): - pc = df["Close"].shift(1) - tr = pd.concat([df["High"] - df["Low"], (df["High"] - pc).abs(), (df["Low"] - pc).abs()], axis=1).max(axis=1) - return tr.ewm(alpha=1 / period, min_periods=period, adjust=False).mean() - - -def rate_table(d, col, target, half, order=None): - base1 = d.loc[half == 1, target].mean(); base2 = d.loc[half == 2, target].mean() - g = d.groupby(col, observed=True) - out = pd.DataFrame({"n": g.size(), - "h1_lift": d[half == 1].groupby(col, observed=True)[target].mean() / base1, - "h2_lift": d[half == 2].groupby(col, observed=True)[target].mean() / base2, - "rate%": 100 * g[target].mean()}) - if order is not None: out = out.reindex([o for o in order if o in out.index]) - out["stable"] = np.where((out.h1_lift > 1.15) & (out.h2_lift > 1.15), "HIGH", np.where((out.h1_lift < 0.85) & (out.h2_lift < 0.85), "low", "")) - return out.round(2) - - -def main(): - ap = argparse.ArgumentParser(); ap.add_argument("parquet"); ap.add_argument("bar_minutes", type=int) - ap.add_argument("--start", default="2023-09-05"); ap.add_argument("--end", default="2026-09-05") - a = ap.parse_args(); bm = a.bar_minutes - px = pd.read_parquet(a.parquet) - px = px[(px.index >= pd.Timestamp(a.start, tz="UTC") - pd.Timedelta(days=30)) & (px.index < pd.Timestamp(a.end, tz="UTC"))] - ic = precompute_indicators(px, TechnicalConfig()) - d = pd.DataFrame(index=px.index) - atr_prev = wilder_atr(px).shift(1) - rng = px["High"] - px["Low"] - d["burst"] = (rng >= 3.0 * atr_prev).astype(int) - H = max(1, 60 // bm) - d["next_hour"] = d["burst"][::-1].rolling(H, min_periods=1).max()[::-1].astype(int) - d["next_bar"] = d["burst"] - d["bdir"] = np.sign(px["Close"] - px["Open"]) - # states at t-1 - sh = lambda arr: pd.Series(np.asarray(arr, dtype=float), index=px.index).shift(1) - rsi = sh(ic.rsi); adx = sh(ic.adx); adxs = sh(ic.adx_slope); hist = sh(ic.macd_hist); bbp = sh(ic.bb_position) - bbw = sh(ic.bb_band_width); bbaw = sh(ic.bb_avg_width); kcsq = sh(ic.kc_squeeze); roc = sh(ic.roc) - ef, em, es = sh(ic.ema_fast), sh(ic.ema_medium), sh(ic.ema_slow); et = sh(ic.ema_trend) if ic.ema_trend is not None else None - close1 = px["Close"].shift(1); atr1 = sh(ic.atr) - kcu, kcl = sh(ic.kc_upper), sh(ic.kc_lower) - d["rsi"] = pd.cut(rsi, [0, 30, 40, 50, 60, 70, 100], labels=["<30", "30-40", "40-50", "50-60", "60-70", ">=70"]) - d["adx"] = pd.cut(adx, [0, 20, 25, 35, 50, 200], labels=["<20", "20-25", "25-35", "35-50", ">=50"]) - d["adx_slope"] = pd.cut(adxs, [-1e9, -3, 0, 3, 1e9], labels=["falling<-3", "-3..0", "0..3", "rising>3"]) - hist_prev = hist.shift(1) - d["macd"] = np.select([(hist > 0) & (hist > hist_prev), (hist > 0), (hist < 0) & (hist < hist_prev), (hist < 0)], - ["pos&rising", "pos&fading", "neg&falling", "neg&fading"], default="flat") - d["bb_pos"] = pd.cut(bbp, [-1e9, 0, 0.2, 0.8, 1.0, 1e9], labels=["<0 (below)", "0-0.2", "0.2-0.8", "0.8-1", ">1 (above)"]) - d["bb_squeeze"] = np.where(bbw < 0.75 * bbaw, "squeeze", "normal") - d["kc_squeeze"] = np.where(kcsq > 0, "TTM squeeze", "no") - d["kc_pos"] = np.select([close1 > kcu, close1 < kcl], ["above KC", "below KC"], default="inside") - align = np.sign(close1 - ef) + np.sign(ef - em) + np.sign(em - es) + (np.sign(es - et) if et is not None else 0) - d["ema_align"] = pd.cut(align, [-5, -3.5, -0.5, 0.5, 3.5, 5], labels=["fully bear(-4)", "mostly bear", "mixed", "mostly bull", "fully bull(+4)"]) - ribbon = (pd.concat([ef, em, es], axis=1).max(axis=1) - pd.concat([ef, em, es], axis=1).min(axis=1)) / atr1 - d["ribbon"] = np.where(ribbon < 1.0, "compressed(<1 ATR)", "expanded") - ema_dir = np.where(align > 0, 1, np.where(align < 0, -1, 0)) - d["regime"] = np.select([(adx >= 25) & (ema_dir > 0), (adx >= 25) & (ema_dir < 0), adx >= 25, adxs > 3.0], - ["trending_up", "trending_down", "trending(mixed)", "transitioning"], default="ranging") - atr_pct = atr1.rolling(50).apply(lambda w: (w[-1] > w).mean() * 100, raw=True) - d["vol_regime"] = pd.cut(atr_pct, [-1, 30, 60, 80, 101], labels=["low", "normal", "high", "extreme"]) - d["roc"] = pd.cut(roc, [-1e9, -1, -0.3, 0.3, 1, 1e9], labels=["<-1%", "-1..-0.3", "-0.3..0.3", "0.3..1", ">1%"]) - d["ema200_dist"] = pd.cut((close1 - et) / et * 100 if et is not None else pd.Series(np.nan, index=px.index), - [-1e9, -2.5, -0.5, 0.5, 2.5, 1e9], labels=["<-2.5%", "-2.5..-0.5", "near(±0.5%)", "0.5..2.5", ">2.5%"]) - # time - ct = px.index + pd.Timedelta(minutes=bm) - d["hour"] = ct.hour; d["weekday"] = ct.weekday; d["dom"] = pd.cut(ct.day, [0, 5, 10, 15, 20, 25, 31], labels=["1-5", "6-10", "11-15", "16-20", "21-25", "26-31"]) - d["first_friday"] = np.where((ct.weekday == 4) & (ct.day <= 7), "first Friday", "other") - # calendar proximity (high-impact USD) - cal = pd.read_parquet(CAL); cal["ts"] = pd.to_datetime(cal.dateline, unit="s", utc=True) - hi = np.sort(cal[(cal.currency == "USD") & (cal.impactName.str.lower() == "high")].ts.values.astype("datetime64[ns]")) - t = ct.values.astype("datetime64[ns]") - nxt = np.searchsorted(hi, t) # next event at/after bar close - mins_to = np.where(nxt < len(hi), (hi[np.minimum(nxt, len(hi) - 1)] - t) / np.timedelta64(1, "m"), 1e9) - prv = nxt - 1 - mins_since = np.where(prv >= 0, (t - hi[np.maximum(prv, 0)]) / np.timedelta64(1, "m"), 1e9) - d["calendar"] = np.select([mins_since <= 30, mins_to <= 15, mins_to <= 60, mins_to <= 180], ["0-30m AFTER hi-impact", "next 15m", "15-60m ahead", "1-3h ahead"], default="none within 3h") - d = d[d.index >= pd.Timestamp(a.start, tz="UTC")].dropna(subset=["rsi", "adx", "bb_pos", "vol_regime"]) - half = pd.Series(np.where(np.arange(len(d)) < len(d) // 2, 1, 2), index=d.index) - print(f"bars {len(d)} base next_bar {100*d.next_bar.mean():.3f}% next_hour {100*d.next_hour.mean():.2f}% (lift = rate / base; 'HIGH' = >1.15x in BOTH halves, 'low' = <0.85x in both)") - groups = ["calendar", "regime", "vol_regime", "rsi", "adx", "adx_slope", "macd", "bb_pos", "bb_squeeze", "kc_squeeze", "kc_pos", "ema_align", "ribbon", "roc", "ema200_dist", "dom", "first_friday"] - for target in ("next_bar", "next_hour"): - print(f"\n################ TARGET: {target} ################") - for g in groups: - print(f"-- {g}"); print(rate_table(d, g, target, half).to_string()) - # weekday x hour top cells (next_bar) - piv = d.pivot_table(index="weekday", columns="hour", values="next_bar", aggfunc="mean") * 100 - base = 100 * d.next_bar.mean() - cells = piv.stack().sort_values(ascending=False) - print(f"\n################ TOP 15 weekday x close-hour cells (next-bar burst pct, base {base:.2f}) ################") - names = {0: "Mon", 1: "Tue", 2: "Wed", 3: "Thu", 4: "Fri", 6: "Sun"} - for (wd, h), v in cells.head(15).items(): - n = int(((d.weekday == wd) & (d.hour == h)).sum()) - print(f" {names.get(wd, wd)} {h:02d}:xx {v:.2f}% ({v/base:.1f}x, n={n})") - # direction: does the state predict the burst direction? - b = d[d.burst == 1] - print("\n################ BURST DIRECTION vs prior state (share of bursts going UP) ################") - for g, up_states in (("ema_align", None), ("rsi", None), ("macd", None), ("regime", None)): - print(f"-- {g}: " + ", ".join(f"{k}: {100*(v>0).mean():.0f}% up (n={len(v)})" for k, v in b.groupby(g, observed=True).bdir if len(v) >= 20)) - - -if __name__ == "__main__": - main() diff --git a/backtest/research/burst_map.py b/backtest/research/burst_map.py deleted file mode 100644 index a665d2a..0000000 --- a/backtest/research/burst_map.py +++ /dev/null @@ -1,444 +0,0 @@ -"""Where and when gold moves violently — burst map of the raw bars. - -Usage: - python backtest/research/burst_map.py --parquet PATH --bar-minutes N \ - [--k 3,4,5] [--book BOOK.csv] [--within-bars 12] \ - [--start YYYY-MM-DD --end YYYY-MM-DD] [--out DIR] - -A "burst" is one bar whose range is >= K x the PRIOR bar's Wilder ATR(14) — -the same definition every burst study in this folder has used (spelled -out here bar-for-bar: TR = max(H-L, |H-Cprev|, |L-Cprev|), ATR = -TR.ewm(alpha=1/14, min_periods=14, adjust=False).mean(), ratio = -(H-L) / ATR.shift(1)). - -Reports: - * burst frequency by (weekday, UTC bar-CLOSE hour), plus mean |1 h| and - |4 h| absolute moves in $ and in ATR units on the same grid; - * post-burst drift in the burst's own direction over 6/12/24/48 bars, - overall and by session bucket — does a violent bar continue or snap back; - * the largest trailing-60-minute close-to-close moves, de-overlapped; - * if a trade book is given: how trades that were decided just after a burst, - or that were OPEN when one printed, performed against the rest. - -Every ATR-unit number here is normalised by the PRIOR bar's ATR — the same -denominator as the burst test, so a "4x burst that ran another 4 ATR" reads -directly. All hours are UTC and are the bar's CLOSE hour (label + one bar), -matching the clock the live gates evaluate. - -Analysis tooling only — never imported by the live code. -""" -from __future__ import annotations - -import argparse -import math -import os - -import numpy as np -import pandas as pd - -try: # package import (backtest.research.burst_map) - from backtest.research.book_report import base_metrics, load_book, session_of -except ImportError: # run directly as a script from anywhere - from book_report import base_metrics, load_book, session_of - -SLOT_UNKNOWN = "" - - -# ─────────────────────────── bar primitives ───────────────────────── - -def load_bars(parquet: str, start: str | None = None, - end: str | None = None) -> pd.DataFrame: - """Read a cached OHLCV parquet, de-duplicated and oldest-first.""" - df = pd.read_parquet(parquet).sort_index() - df = df[~df.index.duplicated(keep="last")] - if start: - df = df[df.index >= pd.Timestamp(start, tz="UTC")] - if end: - df = df[df.index <= pd.Timestamp(end, tz="UTC")] - return df - - -def wilder_atr(df: pd.DataFrame, period: int = 14) -> pd.Series: - """Wilder ATR (the definition every burst study in this folder uses).""" - h, l, c = df["High"], df["Low"], df["Close"] - tr = pd.concat([h - l, (h - c.shift()).abs(), (l - c.shift()).abs()], - axis=1).max(axis=1) - return tr.ewm(alpha=1 / period, min_periods=period, adjust=False).mean() - - -def burst_ratio(df: pd.DataFrame, period: int = 14) -> pd.Series: - """Bar range expressed in PRIOR-bar ATRs.""" - return (df["High"] - df["Low"]) / wilder_atr(df, period).shift(1) - - -def burst_flags(df: pd.DataFrame, k: float, period: int = 14) -> pd.Series: - """True where the bar's range is >= k x the prior bar's Wilder ATR(14).""" - return (burst_ratio(df, period) >= k).fillna(False) - - -def burst_dir(df: pd.DataFrame) -> pd.Series: - """+1 up bar, −1 down bar, 0 doji (sign of Close − Open).""" - return np.sign(df["Close"] - df["Open"]).astype(float) - - -def close_times(df: pd.DataFrame, bar_minutes: int) -> pd.DatetimeIndex: - """Bar CLOSE timestamps (the parquet index labels bar OPEN time).""" - return df.index + pd.Timedelta(minutes=bar_minutes) - - -# ───────────────────── when does gold move? tables ────────────────── - -def hour_weekday_table(df: pd.DataFrame, k: float, bar_minutes: int = 5, - ks: tuple[float, ...] | None = None) -> dict: - """Burst frequency and absolute-move size on a (weekday, close-hour) grid. - - Returns a dict of DataFrames (several tables are wanted, one frame each): - burst_pct : weekday rows x close-hour cols, % of bars that are >= k x - bursts (one frame; per-k frames live in `burst_pct_by_k`) - absmove : (weekday, close-hour) rows; bars, mean |1 h move| and - |4 h move| in $ and in prior-bar ATR units - by_hour : the same stats collapsed to close-hour - by_weekday : the same stats collapsed to weekday - """ - ks = tuple(ks) if ks else (k,) - ct = close_times(df, bar_minutes) - hour = pd.Series(ct.hour, index=df.index) - wd = pd.Series(ct.weekday, index=df.index) - atr_prev = wilder_atr(df).shift(1) - - per_hour = max(1, int(round(60 / bar_minutes))) - c = df["Close"] - mv1 = (c - c.shift(per_hour)).abs() - mv4 = (c - c.shift(4 * per_hour)).abs() - stats = pd.DataFrame({ - "weekday": wd, "hour": hour, - "abs1h_usd": mv1, "abs4h_usd": mv4, - "abs1h_atr": mv1 / atr_prev, "abs4h_atr": mv4 / atr_prev, - }) - for kk in ks: - stats[f"burst_{kk:g}x"] = burst_flags(df, kk).astype(float) * 100.0 - - agg = {"abs1h_usd": "mean", "abs4h_usd": "mean", - "abs1h_atr": "mean", "abs4h_atr": "mean"} - agg.update({f"burst_{kk:g}x": "mean" for kk in ks}) - - grid = stats.groupby(["weekday", "hour"]).agg(agg) - grid.insert(0, "bars", stats.groupby(["weekday", "hour"]).size()) - by_hour = stats.groupby("hour").agg(agg) - by_hour.insert(0, "bars", stats.groupby("hour").size()) - by_wd = stats.groupby("weekday").agg(agg) - by_wd.insert(0, "bars", stats.groupby("weekday").size()) - - pct_by_k = {kk: grid[f"burst_{kk:g}x"].unstack("hour") for kk in ks} - return { - "burst_pct": pct_by_k[k if k in pct_by_k else ks[0]].round(3), - "burst_pct_by_k": {kk: v.round(3) for kk, v in pct_by_k.items()}, - "absmove": grid.round(3), - "by_hour": by_hour.round(3), - "by_weekday": by_wd.round(3), - } - - -def post_burst_drift(df: pd.DataFrame, k: float, - horizons: tuple[int, ...] = (6, 12, 24, 48), - bar_minutes: int = 5) -> pd.DataFrame: - """Signed drift IN the burst's direction over the next h bars, in ATRs. - - A positive mean means violent bars continue; negative means they snap - back. Split by the session bucket of the burst bar's close hour. - """ - flags = burst_flags(df, k) - d = burst_dir(df) - atr_prev = wilder_atr(df).shift(1) - ct = close_times(df, bar_minutes) - bucket = pd.Series([session_of(h) for h in ct.hour], index=df.index) - c = df["Close"] - - rows = [] - for h in horizons: - fwd = (c.shift(-h) - c) * d / atr_prev - sub = fwd[flags].dropna() - buckets = bucket[flags].reindex(sub.index) - rows.append({"horizon_bars": h, "bucket": "ALL", "n": int(len(sub)), - "mean_atr": round(float(sub.mean()), 4) if len(sub) else float("nan"), - "median_atr": round(float(sub.median()), 4) if len(sub) else float("nan"), - "pct_positive": round(float((sub > 0).mean() * 100), 2) if len(sub) else float("nan")}) - for b, s in sub.groupby(buckets): - rows.append({"horizon_bars": h, "bucket": b, "n": int(len(s)), - "mean_atr": round(float(s.mean()), 4), - "median_atr": round(float(s.median()), 4), - "pct_positive": round(float((s > 0).mean() * 100), 2)}) - return pd.DataFrame(rows) - - -def slot_label(weekday: int, hour: int, day: int) -> str: - """Time-slot label for a big move. - - NOTE: there is no economic-calendar file in this repo, so these are - TIME SLOTS, not confirmed events — "payroll window" means "the clock - slot US payrolls normally print in", nothing more. - """ - if weekday == 4 and day <= 7 and hour in (12, 13): - return "first-Friday 12:30 slot (payroll-report window)" - if weekday == 2 and hour in (18, 19): - return "Wednesday 18:00 slot (central-bank window)" - if hour in (12, 13): - return "12:30 slot (US data window)" - if hour == 14: - return "14:00 slot" - if hour in (21, 22, 23): - return "settlement/after-hours" - if hour in (0, 1, 2): - return "Asia open" - return SLOT_UNKNOWN - - -def largest_moves(df: pd.DataFrame, n: int = 40, window_minutes: int = 60, - bar_minutes: int = 5) -> pd.DataFrame: - """The n biggest trailing-`window_minutes` close-to-close moves. - - Overlapping windows are de-duplicated greedily: the largest move wins and - anything whose close time is within +/- `window_minutes` of an already - kept move is dropped, so one violent hour contributes one row. - """ - bars = max(1, int(round(window_minutes / bar_minutes))) - c = df["Close"] - move = c - c.shift(bars) - atr_prev = wilder_atr(df).shift(1) - ct = close_times(df, bar_minutes) - tbl = pd.DataFrame({ - "close_time": ct, "move_usd": move.to_numpy(), - "move_atr": (move / atr_prev).to_numpy(), - }).dropna() - tbl["abs_usd"] = tbl["move_usd"].abs() - tbl = tbl.sort_values("abs_usd", ascending=False) - - keep, taken = [], [] - gap = pd.Timedelta(minutes=window_minutes) - for _, row in tbl.iterrows(): - t = row["close_time"] - if any(abs(t - u) <= gap for u in taken): - continue - taken.append(t) - keep.append(row) - if len(keep) >= n: - break - out = pd.DataFrame(keep).reset_index(drop=True) - if out.empty: - return out - out["weekday"] = out["close_time"].dt.weekday - out["hour"] = out["close_time"].dt.hour - out["direction"] = np.where(out["move_usd"] > 0, "UP", "DOWN") - out["slot"] = [slot_label(w, h, t.day) for w, h, t - in zip(out["weekday"], out["hour"], out["close_time"])] - out["move_usd"] = out["move_usd"].round(2) - out["move_atr"] = out["move_atr"].round(2) - return out[["close_time", "weekday", "hour", "direction", "move_usd", - "move_atr", "slot"]] - - -# ───────────────────────── book <-> burst join ────────────────────── - -def _stats(r: pd.Series) -> dict: - m = base_metrics(r) - return {"n": m["n"], "total_r": round(m["total_r"], 2), - "mean_r": round(m["mean_r"], 3), "wr": round(m["wr"], 2), - "pf": round(m["pf"], 2) if math.isfinite(m["pf"]) else float("inf")} - - -def join_book_to_bursts(book_df: pd.DataFrame, bars: pd.DataFrame, k: float, - within_bars: int = 12, - bar_minutes: int = 5) -> dict: - """Split a trade book by its relationship to burst bars. - - (a) `after_burst` : decision_time falls within `within_bars` bars after - the most recent burst's close (bar distance is counted POSITIONALLY on - the bar index, so weekend gaps cannot inflate it). - (b) `burst_during` : at least one burst closed inside [entry, exit). - (c) `burst_agrees` / `burst_opposes`: of (b), whether the FIRST burst that - printed during the trade pushed with or against the trade direction. - """ - ct = close_times(bars, bar_minutes) - ct_arr = ct.to_numpy() - flags = burst_flags(bars, k).to_numpy() - bdir = burst_dir(bars).to_numpy() - burst_pos = np.flatnonzero(flags) - - # Last burst position at or before each bar (−1 where none yet). - last_burst = np.where(flags, np.arange(len(flags)), -1) - last_burst = np.maximum.accumulate(last_burst) - - dec = book_df["decision_time"].astype("datetime64[ns, UTC]").to_numpy() - ent = book_df["entry_time"].astype("datetime64[ns, UTC]").to_numpy() - exi = book_df["exit_time"].astype("datetime64[ns, UTC]").to_numpy() - - # Position of the last bar that had CLOSED by decision_time. - idx = np.searchsorted(ct_arr, dec, side="right") - 1 - valid = idx >= 0 - lb = np.where(valid, last_burst[np.clip(idx, 0, len(flags) - 1)], -1) - bars_since = np.where((lb >= 0) & valid, idx - lb, np.iinfo(np.int32).max) - after = pd.Series((bars_since >= 0) & (bars_since <= within_bars), - index=book_df.index) - - # Bursts printing inside the open trade. - bct = ct_arr[burst_pos] if len(burst_pos) else np.array([], dtype=ct_arr.dtype) - lo = np.searchsorted(bct, ent, side="left") - hi = np.searchsorted(bct, exi, side="left") - during = pd.Series(hi > lo, index=book_df.index) - trade_sign = np.where( - book_df["direction"].astype(str).str.upper().str.startswith("B"), - 1.0, -1.0) - has = hi > lo - first_dir = np.zeros(len(book_df), dtype=float) - if len(burst_pos) and has.any(): - # hi > lo guarantees lo[has] indexes a real burst. - first_dir[has] = bdir[burst_pos[lo[has]]] - agrees = pd.Series(has & (first_dir * trade_sign > 0), - index=book_df.index) - opposes = pd.Series(has & (first_dir * trade_sign < 0), - index=book_df.index) - - r = book_df["r_multiple"] - out = { - "k": k, "within_bars": within_bars, - "bursts_in_window": int(len(burst_pos)), - "after_burst": _stats(r[after]), - "not_after_burst": _stats(r[~after]), - "burst_during": _stats(r[during]), - "no_burst_during": _stats(r[~during]), - "burst_agrees": _stats(r[agrees]), - "burst_opposes": _stats(r[opposes]), - "median_bars_since_burst": float(np.median( - bars_since[bars_since < np.iinfo(np.int32).max])) if ( - bars_since < np.iinfo(np.int32).max).any() else float("nan"), - } - return out - - -# ──────────────────────────── reporting ───────────────────────────── - -def _fmt_stats(name: str, s: dict) -> str: - pf = "inf" if math.isinf(s["pf"]) else f"{s['pf']:.2f}" - return (f" {name:<18} n={s['n']:<5} totalR={s['total_r']:>8.2f} " - f"meanR={s['mean_r']:>7.3f} WR={s['wr']:>6.2f}% PF={pf}") - - -def main() -> None: - ap = argparse.ArgumentParser(description=__doc__.split("\n")[0]) - ap.add_argument("--parquet", required=True) - ap.add_argument("--bar-minutes", type=int, required=True) - ap.add_argument("--k", default="3,4,5", - help="burst thresholds in prior-bar ATRs (default 3,4,5)") - ap.add_argument("--book", default=None) - ap.add_argument("--book-bar-minutes", type=int, default=None, - help="bar minutes of the BOOK (default: --bar-minutes)") - ap.add_argument("--within-bars", type=int, default=12) - ap.add_argument("--start", default=None) - ap.add_argument("--end", default=None) - ap.add_argument("--n-moves", type=int, default=40) - ap.add_argument("--out", default="backtest/research/burst_map_out") - args = ap.parse_args() - - ks = [float(x) for x in args.k.split(",") if x.strip()] - os.makedirs(args.out, exist_ok=True) - bars = load_bars(args.parquet, args.start, args.end) - bm = args.bar_minutes - primary = ks[0] - - lines: list[str] = [] - add = lines.append - add("=" * 78) - add(f"BURST MAP {os.path.basename(args.parquet)}") - add("=" * 78) - add(f"bars : {len(bars)} ({bars.index[0]} -> {bars.index[-1]}" - f", labels = bar OPEN time)") - add(f"bar minutes : {bm} hours below are bar CLOSE hours (UTC)") - add(f"burst definition : range >= K x PRIOR bar Wilder ATR(14); " - f"K = {', '.join(f'{k:g}' for k in ks)}") - add("ATR units : always normalised by the PRIOR bar's ATR") - for k in ks: - f = burst_flags(bars, k) - add(f" K={k:g}: {int(f.sum())} burst bars " - f"({float(f.mean()) * 100:.3f}% of bars)") - - tabs = hour_weekday_table(bars, primary, bm, ks=tuple(ks)) - add("") - add(f"── BURST % BY (weekday, close-hour), K={primary:g} ──") - add(tabs["burst_pct"].fillna(0).round(2).to_string()) - add("") - add("── MEAN ABSOLUTE MOVE BY CLOSE-HOUR (all weekdays) ──") - add(tabs["by_hour"].to_string()) - add("") - add("── MEAN ABSOLUTE MOVE BY WEEKDAY ──") - add(tabs["by_weekday"].to_string()) - - add("") - add(f"── POST-BURST DRIFT (in the burst's own direction), K={primary:g} ──") - drift = post_burst_drift(bars, primary, bar_minutes=bm) - add(drift.to_string(index=False)) - - add("") - add(f"── LARGEST {args.n_moves} TRAILING-60-MIN MOVES (de-overlapped) ──") - add(" slot labels are CLOCK SLOTS — no economic calendar exists in " - "this repo, so no event is confirmed.") - moves = largest_moves(bars, n=args.n_moves, bar_minutes=bm) - add(moves.to_string(index=False)) - - join_rows = [] - if args.book: - bbm = args.book_bar_minutes or bm - book = load_book(args.book, bbm) - add("") - add(f"── BOOK JOIN {os.path.basename(args.book)} " - f"({len(book)} trades, book bar = {bbm} min) ──") - first, last = close_times(bars, bm)[0], close_times(bars, bm)[-1] - outside = int(((book["decision_time"] < first) - | (book["exit_time"] > last)).sum()) - if outside: - add(f" !! {outside} trade(s) start before / end after the bar " - f"window ({first} .. {last}) — their burst context is " - f"partial. Widen --start/--end or use a longer parquet.") - add(" CAUTION: group (b) is survivorship-shaped — a trade only sees " - "a burst if it survived long enough to still be open, so its high " - "PF is mostly 'winners last longer', not a burst edge.") - for k in ks: - j = join_book_to_bursts(book, bars, k, args.within_bars, bm) - add(f" K={k:g} ({j['bursts_in_window']} bursts in window; " - f"median bars since last burst = " - f"{j['median_bars_since_burst']:.0f})") - add(f" (a) decided within {args.within_bars} bars of a burst") - add(_fmt_stats("after_burst", j["after_burst"])) - add(_fmt_stats("not_after_burst", j["not_after_burst"])) - add(" (b) a burst printed while the trade was open") - add(_fmt_stats("burst_during", j["burst_during"])) - add(_fmt_stats("no_burst_during", j["no_burst_during"])) - add(" (c) of (b), first burst vs the trade's direction") - add(_fmt_stats("burst_agrees", j["burst_agrees"])) - add(_fmt_stats("burst_opposes", j["burst_opposes"])) - for grp in ("after_burst", "not_after_burst", "burst_during", - "no_burst_during", "burst_agrees", "burst_opposes"): - join_rows.append({"k": k, "group": grp, - "within_bars": args.within_bars, - **j[grp]}) - - text = "\n".join(lines) - print(text) - with open(os.path.join(args.out, "burst_map.txt"), "w") as fh: - fh.write(text + "\n") - pd.concat([v.assign(k=kk) for kk, v in tabs["burst_pct_by_k"].items()] - ).to_csv(os.path.join(args.out, "hour_weekday_burst_pct.csv")) - tabs["absmove"].to_csv(os.path.join(args.out, "hour_weekday_absmove.csv")) - tabs["by_hour"].to_csv(os.path.join(args.out, "by_hour.csv")) - tabs["by_weekday"].to_csv(os.path.join(args.out, "by_weekday.csv")) - pd.concat([post_burst_drift(bars, k, bar_minutes=bm).assign(k=k) - for k in ks]).to_csv( - os.path.join(args.out, "post_burst_drift.csv"), index=False) - moves.to_csv(os.path.join(args.out, "largest_moves.csv"), index=False) - if join_rows: - pd.DataFrame(join_rows).to_csv( - os.path.join(args.out, "book_join.csv"), index=False) - print(f"\n[written] {os.path.join(args.out, 'burst_map.txt')}") - - -if __name__ == "__main__": - main() diff --git a/backtest/research/burst_precursors.py b/backtest/research/burst_precursors.py deleted file mode 100644 index 2669077..0000000 --- a/backtest/research/burst_precursors.py +++ /dev/null @@ -1,127 +0,0 @@ -"""Are there signs BEFORE a violent bar? Analysis tooling only — never imported by the live code. - -For every bar t, features are computed from bars <= t-1 only. Targets: - next_bar : bar t itself is a burst (range >= K x prior Wilder ATR14) - next_hour : any burst in bars t .. t+H-1 -Reports per-quintile burst rates (lift vs base rate) in both halves, the same -within/outside the scheduled-data hours, and an out-of-sample logistic model -(fit on half 1, scored on half 2): AUC + lift in the top 5% of scores. - - python backtest/research/burst_precursors.py PARQUET BAR_MINUTES [--k 3] [--start ... --end ...] -""" -import argparse -import numpy as np, pandas as pd - - -def wilder_atr(df, period=14): - pc = df["Close"].shift(1) - tr = pd.concat([df["High"] - df["Low"], (df["High"] - pc).abs(), (df["Low"] - pc).abs()], axis=1).max(axis=1) - return tr.ewm(alpha=1 / period, min_periods=period, adjust=False).mean() - - -def build(df, bar_minutes, k): - d = df.copy() - rng = d["High"] - d["Low"] - atr_prev = wilder_atr(d).shift(1) # ATR known at t-1 - d["burst"] = (rng >= k * atr_prev).astype(int) - H = max(1, 60 // bar_minutes) - # burst within next H bars (t..t+H-1) - d["next_hour"] = d["burst"][::-1].rolling(H, min_periods=1).max()[::-1].astype(int) - d["next_bar"] = d["burst"] - close_t = d.index + pd.Timedelta(minutes=bar_minutes) - d["hour"] = close_t.hour - r1 = rng.shift(1) - d["squeeze6"] = r1.rolling(6).mean() / atr_prev - d["squeeze12"] = r1.rolling(12).mean() / atr_prev - d["last_bar_range"] = r1 / atr_prev - sma = d["Close"].shift(1).rolling(20).mean(); sd = d["Close"].shift(1).rolling(20).std(ddof=0) - bbw = (4 * sd / sma) - d["bbw_pct500"] = bbw.rolling(500).rank(pct=True) - v1 = d["Volume"].shift(1) - d["vol_ratio"] = v1.rolling(3).mean() / v1.rolling(50).mean() - d["drift12"] = (d["Close"].shift(1) - d["Close"].shift(13)).abs() / atr_prev - d["atr_pct500"] = atr_prev.rolling(500).rank(pct=True) - b = d["burst"].shift(1).fillna(0).astype(int) - since = np.zeros(len(d)); cnt = 200 - bv = b.values - for i in range(len(d)): - cnt = 0 if bv[i] == 1 else min(200, cnt + 1) - since[i] = cnt - d["bars_since_burst"] = since - return d.dropna(subset=["squeeze6", "squeeze12", "bbw_pct500", "vol_ratio", "drift12", "atr_pct500"]) - - -FEATS = ["squeeze6", "squeeze12", "last_bar_range", "bbw_pct500", "vol_ratio", "drift12", "atr_pct500", "bars_since_burst"] - - -def quintile_table(d, feat, target, half): - q = pd.qcut(d[feat], 5, labels=False, duplicates="drop") - base1 = d.loc[half == 1, target].mean(); base2 = d.loc[half == 2, target].mean() - rows = [] - for b in sorted(q.dropna().unique()): - m = q == b - lo, hi = d.loc[m, feat].min(), d.loc[m, feat].max() - r1 = d.loc[m & (half == 1), target].mean(); r2 = d.loc[m & (half == 2), target].mean() - rows.append((int(b), lo, hi, int(m.sum()), 100 * r1, r1 / base1, 100 * r2, r2 / base2)) - t = pd.DataFrame(rows, columns=["bin", "lo", "hi", "n", "h1_rate%", "h1_lift", "h2_rate%", "h2_lift"]) - return t.round(3) - - -def main(): - ap = argparse.ArgumentParser(); ap.add_argument("parquet"); ap.add_argument("bar_minutes", type=int) - ap.add_argument("--k", type=float, default=3.0); ap.add_argument("--start"); ap.add_argument("--end") - a = ap.parse_args() - df = pd.read_parquet(a.parquet) - if a.start: df = df[df.index >= pd.Timestamp(a.start, tz="UTC")] - if a.end: df = df[df.index < pd.Timestamp(a.end, tz="UTC")] - d = build(df, a.bar_minutes, a.k) - half = pd.Series(np.where(np.arange(len(d)) < len(d) // 2, 1, 2), index=d.index) - print(f"bars {len(d)} K={a.k} base rate next_bar {100*d.next_bar.mean():.3f}% next_hour {100*d.next_hour.mean():.2f}%") - print(f"half 1 ends {d.index[len(d)//2 - 1]}") - # 1. the clock - hr = d.groupby("hour")[["next_bar", "next_hour"]].mean() * 100 - hr["lift_next_bar"] = hr["next_bar"] / (100 * d.next_bar.mean()) - print("\n== THE CLOCK (bar-close hour UTC): burst rate % and lift ==") - print(hr.round(2).T.to_string()) - sched = d["hour"].isin([1, 12, 13]) - print(f"\nscheduled hours (01,12,13) hold {100*d.loc[sched,'next_bar'].sum()/d.next_bar.sum():.1f}% of bursts in {100*sched.mean():.1f}% of bars") - # 2. per-feature quintiles, both halves - for target in ("next_bar", "next_hour"): - print(f"\n== TARGET {target}: quintile burst rate (lift vs base) — ALL hours ==") - for f in FEATS: - t = quintile_table(d, f, target, half) - print(f"-- {f}: lifts h1 {list(t.h1_lift)} h2 {list(t.h2_lift)} (bins lo..hi {t.lo.iloc[0]:.2f}..{t.hi.iloc[-1]:.2f})") - # 3. same, OUTSIDE the scheduled hours (does anything predict the unscheduled ones?) - u = d[~sched]; hu = half[~sched] - print(f"\n== next_bar lifts OUTSIDE scheduled hours (base {100*u.next_bar.mean():.3f}%) ==") - for f in FEATS: - t = quintile_table(u, f, "next_bar", hu) - print(f"-- {f}: h1 {list(t.h1_lift)} h2 {list(t.h2_lift)}") - # 4. OOS logistic model: fit half 1, score half 2 - try: - from sklearn.linear_model import LogisticRegression - from sklearn.metrics import roc_auc_score - X = d[FEATS].copy() - for h in range(24): X[f"h{h}"] = (d["hour"] == h).astype(float) - Xc = d[FEATS].copy() # clock-free variant - for name, XX in (("features+clock", X), ("features only", Xc), ("clock only", X[[f"h{h}" for h in range(24)]])): - mu, sd = XX[half == 1].mean(), XX[half == 1].std().replace(0, 1) - Z = (XX - mu) / sd - for target in ("next_bar", "next_hour"): - m = LogisticRegression(C=0.1, max_iter=2000, class_weight="balanced").fit(Z[half == 1], d.loc[half == 1, target]) - p = m.predict_proba(Z[half == 2])[:, 1]; y = d.loc[half == 2, target] - auc = roc_auc_score(y, p) - top = p >= np.quantile(p, 0.95) - print(f"OOS {name:15s} {target:9s}: AUC {auc:.3f} | top-5% of scores: burst rate {100*y[top].mean():.2f}% vs base {100*y.mean():.2f}% (lift {y[top].mean()/y.mean():.2f}x), captures {100*y[top].sum()/y.sum():.1f}% of bursts") - except ImportError: - print("sklearn missing — OOS model skipped") - # 5. direction: does the burst continue the prior 12-bar drift? - bt = d[d.burst == 1] - prior = np.sign(bt["Close"].shift(1) - bt["Close"].shift(13)) if False else np.sign(d["Close"].shift(1) - d["Close"].shift(13))[d.burst == 1] - bdir = np.sign(bt["Close"] - bt["Open"]) - agree = (prior == bdir)[(prior != 0) & (bdir != 0)] - print(f"\nburst direction == prior-hour drift direction: {100*agree.mean():.1f}% of {len(agree)} bursts (50% = no information)") - - -if __name__ == "__main__": - main() diff --git a/backtest/research/calendar_impact.py b/backtest/research/calendar_impact.py deleted file mode 100644 index 6ba339c..0000000 --- a/backtest/research/calendar_impact.py +++ /dev/null @@ -1,111 +0,0 @@ -"""Do scheduled economic events move gold violently, and do they hurt OPEN trades? Analysis tooling only. - -Needs backtest/cache/calendar.parquet (ff_calendar_scraper.py). Uses USD events by impact. - python backtest/research/calendar_impact.py PARQUET_BARS BAR_MINUTES BOOK.csv ARM DIST [--start --end] -""" -import argparse, sys -from pathlib import Path -import numpy as np, pandas as pd -sys.path.insert(0, str(Path(__file__).resolve().parent)) -from open_trade_bursts import wilder_atr, simulate, r_of, stats, K - -CAL = Path(__file__).resolve().parent.parent / "cache" / "calendar.parquet" - - -def main(): - ap = argparse.ArgumentParser(); ap.add_argument("parquet"); ap.add_argument("bar_minutes", type=int) - ap.add_argument("book"); ap.add_argument("arm", type=float); ap.add_argument("dist", type=float) - ap.add_argument("--start", default="2023-09-05"); ap.add_argument("--end", default="2026-09-05") - a = ap.parse_args(); bm = a.bar_minutes - cal = pd.read_parquet(CAL) - cal["ts"] = pd.to_datetime(cal["dateline"], unit="s", utc=True) - cal = cal[(cal.ts >= pd.Timestamp(a.start, tz="UTC")) & (cal.ts < pd.Timestamp(a.end, tz="UTC"))] - usd = cal[cal.currency == "USD"].copy() - usd["impact"] = usd["impactName"].str.lower() - print(f"calendar: {len(cal)} events in window, USD {len(usd)}: " + ", ".join(f"{k}={v}" for k, v in usd.impact.value_counts().items())) - px = pd.read_parquet(a.parquet); px = px[(px.index >= pd.Timestamp(a.start, tz="UTC")) & (px.index < pd.Timestamp(a.end, tz="UTC"))] - atr_prev = wilder_atr(px).shift(1) - rng = px["High"] - px["Low"]; burst = (rng >= K * atr_prev).fillna(False) - close = px["Close"]; idx = px.index - H1 = 60 // bm - move60 = (close.shift(-H1) - close).abs() / atr_prev # |move| over the next hour from bar OPEN-ish (label) - move15 = (close.shift(-max(1, 15 // bm)) - close).abs() / atr_prev - burst60 = burst[::-1].rolling(H1, min_periods=1).max()[::-1].astype(bool) - # event -> bar containing it (label <= ts < label + bm) - def bar_pos(ts): - p = idx.searchsorted(ts, side="right") - 1 - return p if 0 <= p < len(idx) and (ts - idx[p]) < pd.Timedelta(minutes=bm) else -1 - usd["pos"] = [bar_pos(t) for t in usd.ts] - usd = usd[usd.pos >= 0] - usd["bar_range_atr"] = rng.values[usd.pos] / atr_prev.values[usd.pos] - usd["burst_in_60m"] = burst60.values[usd.pos] - usd["move60_atr"] = move60.values[usd.pos]; usd["move60_usd"] = (close.shift(-H1) - close).abs().values[usd.pos] - usd["move15_atr"] = move15.values[usd.pos] - # baseline: all bars, and bars in the same close-hours without any USD high/medium event within the hour - chour = (idx + pd.Timedelta(minutes=bm)).hour - ev_bars = set() - for p in usd[usd.impact.isin(["high", "medium"])].pos: ev_bars.update(range(max(0, p - H1), min(len(idx), p + H1))) - quiet = ~np.isin(np.arange(len(idx)), list(ev_bars)) - def base_row(mask, label): - return dict(group=label, n=int(mask.sum()), bar_range_atr=float((rng / atr_prev)[mask].mean()), burst_in_60m_pct=100 * float(burst60[mask].mean()), - move15_atr=float(move15[mask].mean()), move60_atr=float(move60[mask].mean()), move60_usd=float((close.shift(-H1) - close).abs()[mask].mean())) - rows = [base_row(np.ones(len(idx), bool), "ALL bars"), base_row(quiet & np.isin(chour, [12, 13, 14]), "12-14 UTC bars with NO USD event")] - for imp in ("high", "medium", "low"): - g = usd[usd.impact == imp] - rows.append(dict(group=f"USD {imp}-impact events", n=len(g), bar_range_atr=g.bar_range_atr.mean(), burst_in_60m_pct=100 * g.burst_in_60m.mean(), - move15_atr=g.move15_atr.mean(), move60_atr=g.move60_atr.mean(), move60_usd=g.move60_usd.mean())) - print("\n== HOW MUCH GOLD MOVES AROUND USD EVENTS (event bar = the bar containing the release) ==") - print(pd.DataFrame(rows).round(2).to_string(index=False)) - hi = usd[usd.impact == "high"] - top = hi.groupby("name").agg(n=("ts", "size"), bar_range_atr=("bar_range_atr", "mean"), burst60_pct=("burst_in_60m", lambda s: 100 * s.mean()), - move60_usd=("move60_usd", "mean"), move60_atr=("move60_atr", "mean")).sort_values("n", ascending=False) - print("\n== HIGH-IMPACT USD EVENTS BY NAME (n >= 6) ==") - print(top[top.n >= 6].round(2).to_string()) - # what share of ALL bursts sit within ±30 min of a high/medium USD event? - near = np.zeros(len(idx), bool) - for p in usd[usd.impact.isin(["high", "medium"])].pos: near[max(0, p - 30 // bm):min(len(idx), p + 30 // bm + 1)] = True - nearh = np.zeros(len(idx), bool) - for p in hi.pos: nearh[max(0, p - 30 // bm):min(len(idx), p + 30 // bm + 1)] = True - print(f"\nshare of ALL violent bars within ±30 min of a USD high-impact event: {100*burst[nearh].sum()/burst.sum():.1f}% (those bars are {100*nearh.mean():.1f}% of time)") - print(f"share within ±30 min of a USD high- or medium-impact event: {100*burst[near].sum()/burst.sum():.1f}% (bars {100*near.mean():.1f}% of time)") - # open trades through high-impact events - book = pd.read_csv(a.book); book["entry_time"] = pd.to_datetime(book.entry_time, utc=True, format="mixed") - bars = dict(O=px.Open.values, H=px.High.values, L=px.Low.values, C=px.Close.values, burst=burst.values, chour=chour.values) - pos = idx.get_indexer(book.entry_time); ok = pos >= 0; book = book[ok].reset_index(drop=True); pos = pos[ok] - ev_pos = np.sort(hi.pos.values) - # flat mask: exit at the open of the bar that CONTAINS a high-impact event... use the bar before (exit ~5-15 min before) - flat_mask = np.zeros(len(idx), bool); tight_mask = np.zeros(len(idx), bool) - for p in ev_pos: - if p - 1 >= 0: flat_mask[p - 1] = True - tight_mask[max(0, p - 1):min(len(idx), p + H1)] = True - res = {} - for v, kw in (("base", {}), ("mask_flat", dict(flat_mask=flat_mask)), ("mask_tight", dict(tight_mask=tight_mask))): - rs, fl, thru = [], [], [] - for i, t in book.iterrows(): - xp, held, reason, flag = simulate(bars, pos[i], t.direction, t.entry, t.sl, a.arm, a.dist, v, **kw) - rs.append(r_of(t.direction, t.entry, t.sl, xp)); fl.append(flag) - if v == "base": - j0, j1 = pos[i], pos[i] + held - k = ev_pos[(ev_pos > j0) & (ev_pos <= j1)] - thru.append(k[0] if len(k) else -1) - res[v] = (np.array(rs), np.array(fl)) - if v == "base": book["ev_pos"] = thru - base = res["base"][0]; half = np.where(np.arange(len(book)) < len(book) // 2, 1, 2) - hit = book.ev_pos.values >= 0 - print(f"\n== OPEN TRADES THAT HELD THROUGH A HIGH-IMPACT USD EVENT ({a.book.split('/')[-1]}) ==") - print(f"trades through an event: {hit.sum()} of {len(book)} ({100*hit.mean():.1f}%) | mean R {base[hit].mean():+.3f} vs {base[~hit].mean():+.3f} for the rest | WR {100*(base[hit]>0).mean():.0f}% vs {100*(base[~hit]>0).mean():.0f}%") - if hit.sum(): - d = np.where(book.direction == "BUY", 1.0, -1.0) - m15 = np.array([ (close.shift(-max(1,15//bm)).values[p] - close.values[p]) if p >= 0 else np.nan for p in book.ev_pos]) - with_ = hit & (d * m15 > 0); against = hit & (d * m15 < 0) - print(f" event's first 15-min move WITH the trade: n={with_.sum()} mean R {base[with_].mean():+.3f} | AGAINST: n={against.sum()} mean R {base[against].mean():+.3f} (WR {100*(base[against]>0).mean():.0f}%)") - print("\n== PROTECTION AROUND HIGH-IMPACT USD EVENTS ==") - b = stats(base) - for v, (r, f) in res.items(): - s = stats(r); aff = f == 1 - d1 = r[half == 1].sum() - base[half == 1].sum(); d2 = r[half == 2].sum() - base[half == 2].sum() - print(f"{v:11s} total {s['total']:+8.2f} Δ {s['total']-b['total']:+7.2f} PF {s['pf']:.2f} DD {s['dd']:.2f} affected {aff.sum():4d} (affected: base {base[aff].sum():+.2f} -> {r[aff].sum():+.2f}) | Δ h1 {d1:+.2f} / h2 {d2:+.2f}") - - -if __name__ == "__main__": - main() diff --git a/backtest/research/commit_ladder.py b/backtest/research/commit_ladder.py deleted file mode 100644 index 9dc03ef..0000000 --- a/backtest/research/commit_ladder.py +++ /dev/null @@ -1,178 +0,0 @@ -"""Commit-ladder campaign driver (2026-08). - -Question: of all signal-code states shipped in the last two months, which -configuration is the most accurate? Historical in-era backtest numbers are -not comparable (the backtester's realism model changed on 07-08/07-16 and -the data feed switched to Oanda on 07-28), so every historical -configuration is recreated on TODAY's harness via --set overrides and -replayed on the same pinned Oanda bars. Bug-fix commits stay on; the -ladder compares shipped tuning decisions only. - -Usage: - python backtest/research/commit_ladder.py # full matrix - python backtest/research/commit_ladder.py --tf 5min # one timeframe - python backtest/research/commit_ladder.py --windows 12mo,oos - -Resumable: a run whose CSV already exists in the output dir is skipped. -Outputs: backtest/research/commit_ladder_2026-08/_.csv - + .log per run, summary.csv, and a printed ranking table. -""" -from __future__ import annotations - -import argparse -import subprocess -import sys -from pathlib import Path - -import pandas as pd - -ROOT = Path(__file__).resolve().parents[2] -BT = ROOT / "backtest" / "gpu_backtest.py" -OUT = ROOT / "backtest" / "research" / "commit_ladder_2026-08" - -WINDOWS = { - "12mo": ("2025-07-30", "2026-07-30"), - "oos": ("2024-06-01", "2025-03-30"), - "5yr": ("2021-07-30", "2026-07-30"), -} -SPLIT_12MO = pd.Timestamp("2026-01-29", tz="UTC") # book_diff.py's split - -# (variant, "commit date what-it-represents", [--set overrides]) -VARIANTS: dict[str, list[tuple[str, str, list[str]]]] = { - "5min": [ - ("M5-A", "dad2432 07-09 gates+trailing+pullback", - ["xau_strategy.blocked_hours_utc=[0,1,2,10,11,13,16,21]", - "xau_strategy.cp_enabled=false", - "signals.trail_dist_r=0.75", - "xau_strategy.m5_min_abs_score=30", - "xau_strategy.m5_min_confirmation_score=10"]), - ("M5-B", "f3ba4d5 07-11 +hour-9 block", - ["xau_strategy.blocked_hours_utc=[0,1,2,10,11,13,16,21,9]", - "xau_strategy.cp_enabled=false", - "signals.trail_dist_r=0.75", - "xau_strategy.m5_min_abs_score=30", - "xau_strategy.m5_min_confirmation_score=10"]), - ("M5-C", "de6424e 07-16 +chart patterns", - ["xau_strategy.blocked_hours_utc=[0,1,2,10,11,13,16,21,9]", - "signals.trail_dist_r=0.75", - "xau_strategy.m5_min_abs_score=30", - "xau_strategy.m5_min_confirmation_score=10"]), - ("M5-D", "3ee79b5 07-21 +trail dist 0.85", - ["xau_strategy.blocked_hours_utc=[0,1,2,10,11,13,16,21,9]", - "xau_strategy.m5_min_abs_score=30", - "xau_strategy.m5_min_confirmation_score=10"]), - ("M5-E", "0396970 07-30 relaxation (HEAD)", []), - ], - "15min": [ - ("M15-A", "987ec5b 07-09 pre-campaigns", - ["signals.tp_floor_rr=0.5", - "xau_strategy.blocked_regimes=[]", - "xau_strategy.sell_blocked_hours_utc=[]", - "signals.trail_arm_r=0.60"]), - ("M15-B", "f3ba4d5/f88e39f 07-11 1R floor + regime block", - ["xau_strategy.sell_blocked_hours_utc=[]", - "signals.trail_arm_r=0.60"]), - ("M15-C", "7d3fd95 07-16 +SELL hour 15/17 block", - ["signals.trail_arm_r=0.60"]), - ("M15-D", "bf8e5fb 07-21 trail arm 0.70 (HEAD)", []), - ], - "1h": [ - ("H1-A", "pre-d95d31a 07-08 TP-tightening model", - ["signals.min_rr_ratio=0.4", - "signals.tp_tightening_factor=0.8"]), - ("H1-B", "HEAD", []), - ], -} - - -def run_one(tf: str, name: str, overrides: list[str], wkey: str, - spread: str | None = None) -> Path: - suffix = f"_s{spread}" if spread else "" - csv_path = OUT / f"{name}_{wkey}{suffix}.csv" - log_path = OUT / f"{name}_{wkey}{suffix}.log" - if csv_path.exists(): - print(f" [skip] {csv_path.name} exists") - return csv_path - start, end = WINDOWS[wkey] - cmd = [sys.executable, str(BT), "--pair", "XAUUSD", - "--primary-tf", tf, "--start", start, "--end", end, - "--export-csv", str(csv_path)] - if spread: - cmd += ["--spread", spread] - for ov in overrides: - cmd += ["--set", ov] - print(f" [run ] {csv_path.name} ...", flush=True) - with open(log_path, "w") as lf: - rc = subprocess.run(cmd, stdout=lf, stderr=subprocess.STDOUT, - cwd=ROOT).returncode - if rc != 0: - raise SystemExit(f"backtest failed (rc={rc}) — see {log_path}") - log = log_path.read_text() - n_ov = log.count("--set override") - if n_ov != len(overrides): - raise SystemExit( - f"{name}_{wkey}: expected {len(overrides)} --set overrides, " - f"log shows {n_ov} — see {log_path}") - if "Loading cached" not in log and wkey != "sanity": - print(f" [warn] {log_path.name}: no 'Loading cached' line — " - f"data was fetched live for at least one timeframe") - return csv_path - - -def metrics(csv_path: Path, wkey: str) -> dict: - df = pd.read_csv(csv_path, parse_dates=["entry_time"]) - if df.empty: - return {"trades": 0, "total_r": 0.0, "wr": 0.0, "pf": 0.0, - "max_dd": 0.0, "h1_r": 0.0, "h2_r": 0.0, "years_pos": ""} - if df["entry_time"].dt.tz is None: - df["entry_time"] = df["entry_time"].dt.tz_localize("UTC") - r = df["r_multiple"] - pos, neg = r[r > 0].sum(), r[r < 0].sum() - cum = r.cumsum() - dd = (pd.concat([pd.Series([0.0]), cum]).cummax()[1:] - cum).max() - out = { - "trades": len(df), - "total_r": round(r.sum(), 1), - "wr": round((r > 0).mean() * 100, 1), - "pf": round(pos / abs(neg), 2) if neg else float("inf"), - "max_dd": round(dd, 1), - "h1_r": "", "h2_r": "", "years_pos": "", - } - if wkey == "12mo": - h1 = r[df["entry_time"] < SPLIT_12MO].sum() - out["h1_r"] = round(h1, 1) - out["h2_r"] = round(r.sum() - h1, 1) - if wkey == "5yr": - by_year = r.groupby(df["entry_time"].dt.year).sum() - out["years_pos"] = f"{(by_year > 0).sum()}/{len(by_year)}" - return out - - -def main() -> None: - ap = argparse.ArgumentParser() - ap.add_argument("--tf", choices=list(VARIANTS), default=None) - ap.add_argument("--windows", default="12mo,oos,5yr") - args = ap.parse_args() - OUT.mkdir(parents=True, exist_ok=True) - wkeys = [w.strip() for w in args.windows.split(",")] - tfs = [args.tf] if args.tf else list(VARIANTS) - - rows = [] - for tf in tfs: - for name, label, overrides in VARIANTS[tf]: - for wkey in wkeys: - csv_path = run_one(tf, name, overrides, wkey) - m = metrics(csv_path, wkey) - rows.append({"tf": tf, "variant": name, "represents": label, - "window": wkey, **m}) - print(f" {name} {wkey}: {m['trades']} trades, " - f"{m['total_r']}R, PF {m['pf']}, WR {m['wr']}%") - - summary = pd.DataFrame(rows) - summary.to_csv(OUT / "summary.csv", index=False) - print("\n=== SUMMARY (also in summary.csv) ===") - print(summary.to_string(index=False)) - - -if __name__ == "__main__": - main() diff --git a/backtest/research/duka_spread.py b/backtest/research/duka_spread.py deleted file mode 100644 index 12eddf5..0000000 --- a/backtest/research/duka_spread.py +++ /dev/null @@ -1,193 +0,0 @@ -"""Dukascopy XAU/USD tick download + bid/ask spread study. - -Validates ForexBot2's $0.30 round-trip cost assumption (config: -SignalConfig.round_trip_cost / PAIR_INFO default_spread). Round-trip -cost of a market entry+exit = one ask-bid spread width (buy at ask, -sell at bid), so the assumption holds iff typical spread ~ $0.30. - -bi5 format: LZMA 'alone' stream of 20-byte big-endian records -(ms_in_hour uint32, ask uint32, bid uint32, ask_vol f32, bid_vol f32); -XAUUSD prices scaled x1000. URL months are 0-indexed. -""" -import lzma -import sys -from concurrent.futures import ThreadPoolExecutor -from datetime import date, timedelta -from pathlib import Path - -import numpy as np -import pandas as pd -import requests - -SCALE = 1000.0 -CACHE = Path(__file__).parent / "tick_cache" -CACHE.mkdir(exist_ok=True) -URL = "https://datafeed.dukascopy.com/datafeed/XAUUSD/{y}/{m:02d}/{d:02d}/{h:02d}h_ticks.bi5" - -# Bot's tradable bar-close hours: session 03:00-23:00 minus XAU blocked -# hours [0,1,2,10,11,13,16,21] (M5 additionally blocks 9 — reported -# separately below). -BLOCKED = {0, 1, 2, 10, 11, 13, 16, 21} -TRADABLE = sorted(set(range(3, 23)) - BLOCKED) - -# ── Day sample ──────────────────────────────────────────────────────── -recent = [date(2026, 7, 6) + timedelta(days=i) for i in range(17)] -recent = [d for d in recent if d.weekday() < 5] # Mon-Fri Jul 6-22 -monthly = [date(2025, 8, 13), date(2025, 9, 10), date(2025, 10, 15), - date(2025, 11, 12), date(2025, 12, 10), date(2026, 1, 14), - date(2026, 2, 11), date(2026, 3, 11), date(2026, 4, 15), - date(2026, 5, 13), date(2026, 6, 10)] # mid-month Wednesdays -nfp = [date(2026, 6, 5), date(2026, 7, 3)] # NFP Fridays -sundays = [date(2026, 7, 12), date(2026, 7, 19)] # Sun 22-24 open -DAYS = sorted(set(recent + monthly + nfp + sundays)) -assert all(d.weekday() < 5 for d in recent + monthly + nfp) - - -_tls = __import__("threading").local() - - -def _session() -> requests.Session: - if not hasattr(_tls, "s"): - _tls.s = requests.Session() - _tls.s.headers["User-Agent"] = "Mozilla/5.0 (research; spread-study)" - return _tls.s - - -def fetch(day: date, hour: int) -> Path: - p = CACHE / f"{day.isoformat()}_{hour:02d}.bi5" - if p.exists(): - return p - u = URL.format(y=day.year, m=day.month - 1, d=day.day, h=hour) - last = None - for attempt in range(4): - try: - r = _session().get(u, timeout=30) - if r.status_code != 200: - raise RuntimeError(f"{u} -> HTTP {r.status_code}") - p.write_bytes(r.content) - return p - except (requests.ConnectionError, requests.Timeout, RuntimeError) as exc: - last = exc - __import__("time").sleep(1.5 * (attempt + 1)) - if hasattr(_tls, "s"): - _tls.s.close() - del _tls.s - raise RuntimeError(f"giving up on {u}: {last}") - - -def decode(p: Path, day: date, hour: int) -> pd.DataFrame: - blob = p.read_bytes() - if not blob: - return pd.DataFrame() - raw = lzma.decompress(blob) - n = len(raw) // 20 - a = np.frombuffer(raw[: n * 20], dtype=">u4").reshape(n, 5) - f = np.frombuffer(raw[: n * 20], dtype=">f4").reshape(n, 5) - df = pd.DataFrame({ - "ms": a[:, 0].astype(np.int64), - "ask": a[:, 1] / SCALE, - "bid": a[:, 2] / SCALE, - }) - df["spread"] = df["ask"] - df["bid"] - df["date"], df["hour"] = day.isoformat(), hour - df["weekday"] = day.weekday() - return df - - -def load_all() -> pd.DataFrame: - jobs = [(d, h) for d in DAYS for h in range(24)] - with ThreadPoolExecutor(max_workers=3) as ex: - paths = list(ex.map(lambda j: fetch(*j), jobs)) - frames = [decode(p, d, h) for p, (d, h) in zip(paths, jobs)] - df = pd.concat([f for f in frames if not f.empty], ignore_index=True) - return df - - -def wstats(g: pd.DataFrame) -> pd.Series: - """Tick stats + time-weighted mean (each tick's spread held until the - next tick; last tick held to hour end).""" - tw_num = tw_den = 0.0 - keys = [k for k in ("date", "hour") if k in g.columns] - for _, grp in g.groupby(keys, sort=False): - ms = grp["ms"].to_numpy() - sp = grp["spread"].to_numpy() - w = np.diff(np.append(ms, 3_600_000)).clip(min=0) - tw_num += float((sp * w).sum()) - tw_den += float(w.sum()) - s = g["spread"] - return pd.Series({ - "ticks": len(s), - "tw_mean": tw_num / tw_den if tw_den else np.nan, - "mean": s.mean(), "median": s.median(), - "p75": s.quantile(0.75), "p90": s.quantile(0.90), - "p99": s.quantile(0.99), "max": s.max(), - "pct_le_030": (s <= 0.30).mean() * 100, - "pct_ge_060": (s >= 0.60).mean() * 100, - }) - - -def m5_boundary_spreads(df: pd.DataFrame) -> pd.DataFrame: - """Spread of the FIRST tick at/after each M5 boundary — what a - next-open fill actually pays.""" - rows = [] - for (day, hour), grp in df.groupby(["date", "hour"], sort=False): - ms = grp["ms"].to_numpy() - sp = grp["spread"].to_numpy() - wd = grp["weekday"].iat[0] - for b in range(0, 3_600_000, 300_000): - i = np.searchsorted(ms, b) - if i < len(ms) and ms[i] - b <= 30_000: # tick within 30 s - rows.append((day, wd, hour, b // 60_000, sp[i])) - return pd.DataFrame(rows, columns=["date", "weekday", "hour", "minute", "spread"]) - - -def main() -> None: - df = load_all() - print(f"days={df['date'].nunique()} ticks={len(df):,} " - f"price range {df['bid'].min():.0f}-{df['ask'].max():.0f}") - - print("\n── Overall (all sampled hours) ──") - print(wstats(df).round(3).to_string()) - - trad = df[df["hour"].isin(TRADABLE) & (df["weekday"] < 5)] - print("\n── Bot-tradable hours (session 03-23 minus blocked, Mon-Fri) ──") - print(wstats(trad).round(3).to_string()) - - roll = df[df["hour"].isin([21, 22, 23])] - print("\n── Rollover window 21-24 UTC ──") - print(wstats(roll).round(3).to_string()) - - print("\n── Per UTC hour ──") - per_hour = df[df["weekday"] < 5].groupby("hour").apply(wstats, include_groups=False).round(3) - per_hour["tradable"] = ["Y" if h in TRADABLE else "-" for h in per_hour.index] - print(per_hour.to_string()) - per_hour.to_csv(Path(__file__).parent / "duka_spread_by_hour.csv") - - print("\n── M5 next-open boundary fills ──") - mb = m5_boundary_spreads(df) - mb_trad = mb[mb["hour"].isin(TRADABLE) & (mb["weekday"] < 5)] - for name, d in [("all hours", mb), ("tradable hours", mb_trad)]: - s = d["spread"] - print(f"{name}: n={len(s):,} mean={s.mean():.3f} median={s.median():.3f} " - f"p90={s.quantile(0.9):.3f} p99={s.quantile(0.99):.3f} " - f"%<=0.30={(s <= 0.30).mean()*100:.1f} %>=0.60={(s >= 0.60).mean()*100:.1f}") - mb.to_csv(Path(__file__).parent / "duka_m5_boundary_spreads.csv", index=False) - - print("\n── By month (tradable hours, tick median / tw_mean) ──") - trad2 = trad.copy() - trad2["month"] = trad2["date"].str[:7] - bym = trad2.groupby("month").apply(wstats, include_groups=False).round(3) - print(bym[["ticks", "tw_mean", "median", "p90", "pct_le_030", "pct_ge_060"]].to_string()) - - print("\n── NFP Fridays 12:00-15:00 UTC vs same hours other days ──") - nfp_days = {"2026-06-05", "2026-07-03"} - win = df[df["hour"].isin([12, 13, 14])] - for name, d in [("NFP", win[win["date"].isin(nfp_days)]), - ("non-NFP", win[~win["date"].isin(nfp_days) & (win["weekday"] < 5)])]: - s = d["spread"] - print(f"{name}: n={len(s):,} tick-median={s.median():.3f} p90={s.quantile(0.9):.3f} " - f"p99={s.quantile(0.99):.3f} max={s.max():.2f}") - - -if __name__ == "__main__": - sys.exit(main()) diff --git a/backtest/research/duka_spread_by_hour.csv b/backtest/research/duka_spread_by_hour.csv deleted file mode 100644 index a96a3ea..0000000 --- a/backtest/research/duka_spread_by_hour.csv +++ /dev/null @@ -1,25 +0,0 @@ -hour,ticks,tw_mean,mean,median,p75,p90,p99,max,pct_le_030,pct_ge_060,tradable -0,362893.0,0.749,0.768,0.79,0.82,0.86,1.1,7.69,0.053,94.561,- -1,540182.0,0.735,0.742,0.74,0.81,0.87,1.0,5.59,0.073,90.449,- -2,431589.0,0.716,0.722,0.73,0.8,0.86,0.95,2.96,0.1,85.27,- -3,343952.0,0.708,0.711,0.71,0.79,0.84,0.95,1.67,0.081,82.939,Y -4,246262.0,0.704,0.702,0.71,0.78,0.82,0.94,1.86,0.098,82.627,Y -5,388365.0,0.712,0.716,0.72,0.79,0.82,0.95,2.17,0.076,86.745,Y -6,368999.0,0.65,0.651,0.65,0.69,0.74,0.94,2.32,0.079,71.668,Y -7,313149.0,0.601,0.598,0.59,0.65,0.69,0.994,3.08,0.239,42.904,Y -8,347276.0,0.612,0.616,0.6,0.66,0.74,1.0,3.04,0.308,48.349,Y -9,321619.0,0.612,0.615,0.6,0.66,0.72,1.0,5.95,0.288,50.644,Y -10,295271.0,0.611,0.618,0.6,0.66,0.71,1.0,9.36,0.164,49.253,- -11,336963.0,0.619,0.624,0.6,0.67,0.79,1.0,4.97,0.203,50.178,- -12,476773.0,0.66,0.697,0.63,0.72,0.877,1.367,14.98,0.257,59.633,Y -13,651425.0,0.641,0.648,0.63,0.69,0.787,1.487,11.407,0.296,60.297,- -14,673391.0,0.625,0.625,0.63,0.68,0.77,0.9,11.98,0.458,59.085,Y -15,598398.0,0.636,0.638,0.64,0.7,0.78,0.91,4.49,0.433,66.296,Y -16,485023.0,0.648,0.644,0.64,0.7,0.757,0.917,4.09,0.118,68.496,- -17,450119.0,0.626,0.628,0.627,0.7,0.726,0.84,4.08,0.154,63.337,Y -18,411538.0,0.626,0.63,0.637,0.7,0.71,0.79,4.16,0.107,67.734,Y -19,435396.0,0.638,0.643,0.65,0.7,0.72,0.896,5.804,0.11,72.007,Y -20,212654.0,0.684,0.665,0.66,0.7,0.76,1.217,7.15,0.226,70.215,Y -21,31319.0,0.627,0.622,0.61,0.68,0.75,1.02,2.08,0.089,53.153,- -22,108879.0,0.829,0.834,0.797,0.84,0.98,2.05,10.11,0.056,95.434,Y -23,161079.0,1.032,0.913,0.8,0.89,1.307,2.77,13.174,0.092,95.41,- diff --git a/backtest/research/feed_divergence_2026-08-14.md b/backtest/research/feed_divergence_2026-08-14.md index 0558a96..f96e440 100644 --- a/backtest/research/feed_divergence_2026-08-14.md +++ b/backtest/research/feed_divergence_2026-08-14.md @@ -1,8 +1,8 @@ # Feed Divergence Study — Oanda vs Tiingo vs Twelve Data (Phase 1) **Date:** 2026-08-14 -**Test plan:** `docs/data_source_change_test_plan.md` -**Tool:** `scripts/feed_divergence_study.py` (fetch_range over 365d, parquet-cached) +**Test plan:** `docs/data_source_change_test_plan.md` (removed 2026-09-06, in git history; the pending Phase 4 shadow scan is noted in CLAUDE.md) +**Tool:** `scripts/feed_divergence_study.py` (script removed 2026-09-06, in git history) (fetch_range over 365d, parquet-cached) **Window:** 2025-08-14 → 2026-08-14 (12 months); Oanda-vs-TD truncated at TD cache head (2026-07-23) ## Headline diff --git a/backtest/research/feed_divergence_oanda_vs_tiingo.csv b/backtest/research/feed_divergence_oanda_vs_tiingo.csv deleted file mode 100644 index c6ceccd..0000000 --- a/backtest/research/feed_divergence_oanda_vs_tiingo.csv +++ /dev/null @@ -1,2 +0,0 @@ -tf,a,b,a_bars,b_bars,common,a_only,b_only,dir_agree_pct,ret_corr,mean_abs_dclose,p95_abs_dclose,max_abs_dclose,dclose_pct_atr_mean -5min,oanda,tiingo,70901,73241,70807,94,2434,97.674000,0.994970,0.123070,0.285000,26.105000,2.147942 diff --git a/backtest/research/ff_calendar_scraper.py b/backtest/research/ff_calendar_scraper.py deleted file mode 100644 index 6e8edac..0000000 --- a/backtest/research/ff_calendar_scraper.py +++ /dev/null @@ -1,144 +0,0 @@ -"""Forex Factory calendar scraper -> backtest/cache/calendar.parquet. - -Fetches weekly calendar pages and parses the event JSON embedded in -`window.calendarComponentStates` (the `days:` array). Event `dateline` -is a unix epoch, so timestamps carry no timezone ambiguity — verified -2026-07-11 against known release times (NFP 12:30/13:30 UTC across DST, -ECB 12:15/13:15) and against EUR/USD M15 volatility: the bar containing -a high-impact event averages 3.5x normal range, and shifting event -times by ±1h destroys that alignment. - -Usage: - python backtest/research/ff_calendar_scraper.py --start 2024-01-01 --end 2026-07-13 - python backtest/research/ff_calendar_scraper.py # refresh: last stored week -> today - -Merges into the existing parquet on event (id, dateline), so refreshing -is idempotent and also picks up revised `actual` values for recent weeks. -Be polite: ~1.6s between requests (a full year is ~52 requests). -""" -import argparse -import json -import time -from datetime import date, timedelta -from pathlib import Path -from typing import List, Optional - -import pandas as pd -import requests - -CACHE_DIR = Path(__file__).resolve().parent.parent / "cache" -PARQUET = CACHE_DIR / "calendar.parquet" - -UA = ("Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 " - "(KHTML, like Gecko) Chrome/126.0 Safari/537.36") - -FIELDS = ["id", "dateline", "name", "country", "currency", "impactName", - "timeLabel", "timeMasked", "actual", "forecast", "previous", - "revision", "actualBetterWorse"] - - -def week_url(d: date) -> str: - return (f"https://www.forexfactory.com/calendar?week=" - f"{d.strftime('%b').lower()}{d.day}.{d.year}") - - -def extract_days(html: str) -> Optional[list]: - """Balanced-bracket extraction of the `days:` JSON array.""" - i = html.find("days:") - if i < 0: - return None - j = html.find("[", i) - depth, in_str, esc = 0, False, False - for k in range(j, len(html)): - ch = html[k] - if in_str: - if esc: - esc = False - elif ch == "\\": - esc = True - elif ch == '"': - in_str = False - else: - if ch == '"': - in_str = True - elif ch == "[": - depth += 1 - elif ch == "]": - depth -= 1 - if depth == 0: - return json.loads(html[j:k + 1]) - return None - - -def fetch_week(sess: requests.Session, d: date, retries: int = 4) -> Optional[List[dict]]: - for attempt in range(retries): - try: - r = sess.get(week_url(d), timeout=30) - if r.status_code == 200: - days = extract_days(r.text) - if days is not None: - return [{k: e.get(k) for k in FIELDS} - for day in days for e in day.get("events", [])] - except requests.RequestException: - pass - time.sleep(5 * (attempt + 1)) - return None - - -def scrape(start: date, end: date) -> pd.DataFrame: - sess = requests.Session() - sess.headers["User-Agent"] = UA - rows: List[dict] = [] - failed: List[date] = [] - d = start - while d <= end: - week_rows = fetch_week(sess, d) - if week_rows is None: - failed.append(d) - print(f"FAILED week {d}") - else: - rows.extend(week_rows) - d += timedelta(days=7) - time.sleep(1.6) - if failed: - print(f"WARNING: {len(failed)} weeks failed: {failed}") - return pd.DataFrame(rows) - - -def merge_into_store(new: pd.DataFrame, path: Path = PARQUET) -> pd.DataFrame: - if path.exists(): - old = pd.read_parquet(path) - combined = pd.concat([old, new]) - else: - combined = new - # keep="last": a refresh overwrites stale rows (actuals fill in - # after release, revisions land weeks later). - combined = (combined.drop_duplicates(subset=["id", "dateline"], keep="last") - .sort_values("dateline").reset_index(drop=True)) - combined.to_parquet(path, index=False) - return combined - - -def main() -> None: - ap = argparse.ArgumentParser(description=__doc__.splitlines()[0]) - ap.add_argument("--start", type=date.fromisoformat, default=None) - ap.add_argument("--end", type=date.fromisoformat, default=date.today()) - args = ap.parse_args() - - start = args.start - if start is None: - if PARQUET.exists(): - last = pd.read_parquet(PARQUET).dateline.max() - start = pd.Timestamp(last, unit="s").date() - timedelta(days=7) - else: - start = date(2024, 1, 1) - print(f"Scraping {start} -> {args.end}") - new = scrape(start, args.end) - combined = merge_into_store(new) - print(f"Store: {len(combined)} events " - f"({pd.Timestamp(combined.dateline.min(), unit='s')} -> " - f"{pd.Timestamp(combined.dateline.max(), unit='s')}) -> {PARQUET}") - - -if __name__ == "__main__": - main() diff --git a/backtest/research/llm_news_eval.py b/backtest/research/llm_news_eval.py deleted file mode 100644 index de8d074..0000000 --- a/backtest/research/llm_news_eval.py +++ /dev/null @@ -1,697 +0,0 @@ -""" -DeepSeek news-analysis validation harness — XAU/USD. -==================================================== -Research-only (no production code changes). Answers: is the DeepSeek -layer's news analysis *correct*? - -Part A (events): replays every high-impact USD release with a genuine -surprise (2024-01 → 2026-07, from backtest/cache/calendar.parquet) -through the REAL `NewsEngine._llm_event_direction` — real prompt, real -payload, real validation — and scores its "gold bullish/bearish" calls -against gold's actual move (Oanda M15 mids) at +15m/30m/1h/4h/24h. -Baselines: mechanical surprise-sign doctrine, FF actualBetterWorse, -always-bullish/bearish (2024-26 is a gold bull regime), coin flip. - -Part B (headlines): a curated golden set (llm_news_golden.csv) through -the REAL `NewsEngine._llm_headline_scores` in production-sized batches, -vs the keyword scorer it overrides. Includes repeat-batch (determinism) -and shuffled-order (cross-contamination) probes. - -Implementation notes that keep the replay faithful to production: - * `_llm_event_direction`/`_llm_headline_scores` resolve the client via - `get_llm_client()`, so we seed the module singleton with a research - client (raised daily cap: 586 events > production cap 500). - * The event cache key omits the forecast, so a repeat `actual` would - silently re-serve a direction computed for a different forecast — - the cache is cleared before every call. - * A "neutral" reply and a hard failure both return None; they are - disambiguated by reading the event cache back after the call - (absent = failure, "" = neutral, "bullish:/bearish: …" = call). - * Raw request/response bytes are captured with a requests response - hook — the request-building code path is untouched. - -Usage (from repo root): - venv/bin/python backtest/research/llm_news_eval.py events --limit 5 - venv/bin/python backtest/research/llm_news_eval.py events --determinism 20 - venv/bin/python backtest/research/llm_news_eval.py headlines - venv/bin/python backtest/research/llm_news_eval.py report - -Artifacts land in backtest/research/llm_news_eval_2026-08/ (JSONL replay -logs are resume-safe: completed events are never re-paid). `report` is a -pure offline function of the artifacts. -""" - -import argparse -import json -import math -import random -import sys -import time -from datetime import datetime, timezone -from pathlib import Path - -import numpy as np -import pandas as pd - -REPO_ROOT = Path(__file__).resolve().parents[2] -sys.path.insert(0, str(REPO_ROOT)) - -import llm_client -from app_config import deepseek_api_key, get_app_config -from config import NewsConfig -from llm_client import DEFAULT_MODEL, DeepSeekClient -from news_engine import EconomicEvent, NewsEngine - -RESEARCH_DIR = Path(__file__).parent -ART_DIR = RESEARCH_DIR / "llm_news_eval_2026-08" -CALENDAR = REPO_ROOT / "backtest" / "cache" / "calendar.parquet" -BARS = (REPO_ROOT / "backtest" / "cache" / - "oanda_XAU_USD_15min_20210531_20260804.parquet") -GOLDEN = RESEARCH_DIR / "llm_news_golden.csv" -EVENTS_JSONL = ART_DIR / "events_replay.jsonl" -HEADLINES_JSONL = ART_DIR / "headlines_replay.jsonl" -SCORED_CSV = ART_DIR / "events_scored.csv" - -HORIZONS = [("15m", 15), ("30m", 30), ("1h", 60), ("4h", 240), - ("24h", 1440)] -NEUTRAL_BAND = 0.15 # |xau| below this = neutral prediction (Part B) -BATCH_CAP = 25 # production batch cap (news_engine.py:783) -EXPECTED_CORPUS = 586 - - -# ── shared plumbing ────────────────────────────────────────────────── - -class _Recorder: - """requests response hook capturing the exact wire traffic.""" - - def __init__(self): - self.last = None - - def hook(self, resp, **kwargs): - body = resp.request.body - if isinstance(body, bytes): - body = body.decode("utf-8", "replace") - self.last = { - "request_body": body, - "http_status": resp.status_code, - "response_body": resp.text, - "elapsed_s": round(resp.elapsed.total_seconds(), 3), - } - return resp - - def pop(self): - cap, self.last = self.last, None - return cap or {} - - -def install_research_client(timeout: float): - """Build a research DeepSeekClient (raised cap) and seed the module - singleton so the production code paths use it. Returns (client, - recorder).""" - app = get_app_config() - key = deepseek_api_key(app) - ds = app.deepseek_settings - if not key: - sys.exit("No DeepSeek API key (env DEEPSEEK_API_KEY or " - "config.json deepseek.api_key) — aborting before spend.") - if ds.get("enabled") is False: - sys.exit("deepseek.enabled is false in config.json — aborting.") - client = DeepSeekClient( - api_key=key, - model=str(ds.get("model", DEFAULT_MODEL)), - timeout=timeout, - daily_cap=10 ** 6, # research: corpus 586 > production cap 500 - ) - rec = _Recorder() - client._session.hooks["response"].append(rec.hook) - with llm_client._client_lock: - llm_client._client = client - return client, rec - - -def _s(x): - """Parquet str columns carry float NaN — EconomicEvent needs None.""" - return None if pd.isna(x) else str(x) - - -def load_corpus(): - """High-impact USD releases with a genuine surprise, through the - production `EconomicEvent.surprise` gate. Asserts run before any - API spend.""" - df = pd.read_parquet(CALENDAR) - df = df[(df["impactName"] == "high") & (df["currency"] == "USD")] - rows = [] - for r in df.itertuples(): - ev = EconomicEvent( - time=datetime.fromtimestamp(int(r.dateline), tz=timezone.utc), - currency="USD", impact="high", event=_s(r.name) or "", - forecast=_s(r.forecast), previous=_s(r.previous), - actual=_s(r.actual), - ) - if ev.surprise not in ("positive", "negative"): - continue - rows.append({ - "ffid": int(r.id), "dateline": int(r.dateline), - "utc_time": ev.time.isoformat(), "name": ev.event, - "actual": ev.actual, "forecast": ev.forecast, - "previous": ev.previous, "surprise": ev.surprise, - "better_worse": int(r.actualBetterWorse), "ev": ev, - }) - ids = [d["ffid"] for d in rows] - if len(rows) != EXPECTED_CORPUS: - sys.exit(f"corpus size {len(rows)} != {EXPECTED_CORPUS} — " - "calendar.parquet changed; re-verify before spending.") - if len(set(ids)) != len(ids): - sys.exit("FF ids not unique — resume keying would break.") - # Pin the actualBetterWorse mapping on two unambiguous NFP prints. - nfp = {d["utc_time"][:10]: d["better_worse"] for d in rows - if d["name"] == "Non-Farm Employment Change"} - if nfp.get("2024-01-05") != 1 or nfp.get("2024-05-03") != 2: - sys.exit("actualBetterWorse mapping pins failed (expected " - "1=better-for-USD on 2024-01-05 beat, 2=worse on " - "2024-05-03 miss).") - rows.sort(key=lambda d: (d["dateline"], d["ffid"])) - return rows - - -def _load_jsonl(path: Path): - recs = [] - if path.exists(): - with open(path) as f: - for line in f: - line = line.strip() - if line: - recs.append(json.loads(line)) - return recs - - -# ── Part A: event replay ───────────────────────────────────────────── - -def _extract_reason(cap): - try: - content = json.loads(cap["response_body"])[ - "choices"][0]["message"]["content"] - return json.loads(content).get("reason") - except Exception: - return None - - -def cmd_events(args): - ART_DIR.mkdir(exist_ok=True) - corpus = load_corpus() - if args.limit: - corpus = corpus[:args.limit] - engine = NewsEngine(NewsConfig()) - existing = _load_jsonl(EVENTS_JSONL) - done1 = {r["ffid"] for r in existing - if r.get("run_pass") == 1 and r.get("llm_direction")} - done2 = {r["ffid"] for r in existing - if r.get("run_pass") == 2 and r.get("llm_direction")} - pending = [d for d in corpus if d["ffid"] not in done1] - print(f"corpus {len(corpus)} | already recorded {len(done1)} | " - f"pending {len(pending)} | determinism re-calls " - f"{args.determinism} | model calls are ~100 tokens each " - f"(deepseek-v4-flash, static-prefix cached) → cost ≈ cents") - if not pending and not args.determinism: - print("nothing to do") - return - _, rec = install_research_client(args.timeout) - - out = open(EVENTS_JSONL, "a") - - def call_one(d, run_pass): - # Cache key omits forecast → cleared per call so 196 colliding - # corpus rows can't be served a stale direction. - with NewsEngine._cache_lock: - NewsEngine._llm_event_cache.clear() - ev = d["ev"] - engine._llm_event_direction(ev) - key = (ev.event, ev.currency, str(ev.actual or "")) - with NewsEngine._cache_lock: - cached = NewsEngine._llm_event_cache.get(key) - cap = rec.pop() - if cached is None: - direction, text = None, None # hard failure → retry - elif cached == "": - direction, text = "neutral", "" - else: - direction, text = cached.split(":", 1)[0], cached - recd = {k: d[k] for k in ( - "ffid", "dateline", "utc_time", "name", "actual", "forecast", - "previous", "surprise", "better_worse")} - recd.update({ - "llm_direction": direction, "llm_text": text, - "reason_raw": _extract_reason(cap), "run_pass": run_pass, - "ts": datetime.now(timezone.utc).isoformat(), **cap, - }) - out.write(json.dumps(recd) + "\n") - out.flush() - return direction - - remaining = pending - for npass in range(1, args.retry_passes + 1): - if not remaining: - break - failed = [] - for i, d in enumerate(remaining): - if call_one(d, 1) is None: - failed.append(d) - if args.sleep: - time.sleep(args.sleep) - if (i + 1) % 25 == 0: - print(f" pass {npass}: {i + 1}/{len(remaining)} " - f"({len(failed)} failed)") - print(f"pass {npass} done: {len(remaining) - len(failed)} ok, " - f"{len(failed)} failed") - remaining = failed - if remaining: - print(f"WARNING: {len(remaining)} events still unrecorded after " - f"{args.retry_passes} passes — re-run `events` to resume.") - - if args.determinism: - ok_ids = {x["ffid"] for x in remaining} - done_now = [d for d in corpus if d["ffid"] not in ok_ids] - done_now.sort(key=lambda d: (d["name"], d["dateline"])) - stride = max(1, len(done_now) // args.determinism) - sample = done_now[::stride][:args.determinism] - todo = [d for d in sample if d["ffid"] not in done2] - print(f"determinism probe: {len(todo)} re-calls " - f"({len(sample) - len(todo)} already recorded)") - for d in todo: - call_one(d, 2) - if args.sleep: - time.sleep(args.sleep) - out.close() - print("events replay complete →", EVENTS_JSONL) - - -# ── Part B: golden headlines ───────────────────────────────────────── - -def _load_golden(): - g = pd.read_csv(GOLDEN) - need = {"headline", "category", "expected_xau_sign", "rationale"} - if not need <= set(g.columns): - sys.exit(f"golden CSV missing columns {need - set(g.columns)}") - if not g["expected_xau_sign"].isin([-1, 0, 1]).all(): - sys.exit("expected_xau_sign must be -1/0/+1") - if not g["headline"].is_unique: - sys.exit("duplicate headlines in golden CSV") - return g - - -def cmd_headlines(args): - ART_DIR.mkdir(exist_ok=True) - g = _load_golden() - heads = list(g["headline"]) - _, rec = install_research_client(args.timeout) - engine = NewsEngine(NewsConfig()) - out = open(HEADLINES_JSONL, "w") # tiny (~9 calls): fresh each run - - def run_batch(texts, tag, batch_idx): - scores = None - for attempt in range(3): - scores = engine._llm_headline_scores(texts) - cap = rec.pop() - if scores is not None: - break - print(f" {tag} batch {batch_idx} attempt {attempt + 1} " - f"failed (http {cap.get('http_status')})") - time.sleep(2.0) - out.write(json.dumps({ - "run_tag": tag, "batch": batch_idx, "headlines": texts, - "scores": scores, - "ts": datetime.now(timezone.utc).isoformat(), **cap, - }) + "\n") - out.flush() - return scores - - batches = [heads[i:i + BATCH_CAP] for i in range(0, len(heads), - BATCH_CAP)] - for bi, b in enumerate(batches): - ok = run_batch(b, "main", bi) is not None - print(f"main batch {bi} ({len(b)} headlines): " - f"{'ok' if ok else 'FAILED'}") - time.sleep(args.sleep) - run_batch(batches[0], "repeat", 0) - print("repeat probe done") - shuffled = heads[:] - random.Random(20260806).shuffle(shuffled) - for bi in range(0, len(shuffled), BATCH_CAP): - run_batch(shuffled[bi:bi + BATCH_CAP], "shuffled", - bi // BATCH_CAP) - print("shuffled probe done →", HEADLINES_JSONL) - out.close() - - -# ── report: pure offline function of the artifacts ─────────────────── - -def _binom_p(k, n): - """Exact two-sided binomial p-value vs p=0.5 (no scipy).""" - if n == 0: - return float("nan") - dist = [math.comb(n, i) for i in range(n + 1)] - tot = float(2 ** n) - pk = dist[k] - return min(1.0, sum(c for c in dist if c <= pk) / tot) - - -def _mcnemar_p(hit_a, hit_b): - """Exact McNemar via binomial on discordant pairs. Inputs: aligned - boolean lists.""" - b = sum(1 for x, y in zip(hit_a, hit_b) if x and not y) - c = sum(1 for x, y in zip(hit_a, hit_b) if not x and y) - return b, c, _binom_p(min(b, c), b + c) - - -def _latest_events(recs): - """Last pass-1 record per ffid (retries append; last wins).""" - out = {} - for r in recs: - if r.get("run_pass") == 1: - out[r["ffid"]] = r - return list(out.values()) - - -def _fmt_rate(hits, n, label=""): - if n == 0: - return f"{label:<24} n=0" - p = _binom_p(hits, n) - return (f"{label:<24} {hits:>4}/{n:<4} = {hits / n * 100:5.1f}% " - f"(binom p={p:.3f})") - - -def report_events(): - recs = _latest_events(_load_jsonl(EVENTS_JSONL)) - if not recs: - print("no event replay records — run `events` first") - return - bars = pd.read_parquet(BARS) - closes = bars["Close"].to_numpy() - close_ns = (bars.index + pd.Timedelta(minutes=15)).asi8 - - def anchor(t_ns, h_min): - i = np.searchsorted(close_ns, t_ns, side="right") - 1 - if i < 0: - return None - stale_ns = max(45.0, h_min / 4.0) * 60e9 - if t_ns - close_ns[i] > stale_ns: - return None - return closes[i] - - rows, skip = [], {h: 0 for h, _ in HORIZONS} - base_skipped = 0 - for r in recs: - t_ns = int(r["dateline"]) * 10 ** 9 - base = anchor(t_ns, 0) - if base is None: - base_skipped += 1 - continue - row = dict(r) - row.pop("request_body", None) - row.pop("response_body", None) - row["year"] = r["utc_time"][:4] - row["base"] = base - for hname, hmin in HORIZONS: - p = anchor(t_ns + hmin * 60 * 10 ** 9, hmin) - if p is None: - skip[hname] += 1 - row[f"d_{hname}"] = np.nan - else: - row[f"d_{hname}"] = p - base - rows.append(row) - df = pd.DataFrame(rows) - df.to_csv(SCORED_CSV, index=False) - - n_fail = int((df["llm_direction"].isna()).sum()) - print("═" * 70) - print("PART A — event gold-direction vs actual XAU move") - print("═" * 70) - print(f"events replayed: {len(df)} (baseline-price skipped: " - f"{base_skipped}) | LLM hard failures: {n_fail}") - print("horizon skip counts (stale/closed):", - {k: v for k, v in skip.items() if v}) - dist = df["llm_direction"].value_counts(dropna=False).to_dict() - print("LLM call distribution:", dist) - - # Unconditional per-horizon noise scale (null): median |Δ| over all - # 2024+ bar times, position-shifted. - bars24 = bars[bars.index >= "2024-01-01"] - null_med = {} - for hname, hmin in HORIZONS: - k = hmin // 15 - d = (bars24["Close"].shift(-k) - bars24["Close"]).abs().dropna() - null_med[hname] = float(d.median()) - print("unconditional null median |Δ| ($):", - {k: round(v, 2) for k, v in null_med.items()}) - - # Strategy call columns: +1 gold up, −1 gold down, 0/NaN excluded. - df["c_llm"] = df["llm_direction"].map( - {"bullish": 1, "bearish": -1}).fillna(0) - df["c_doctrine"] = df["surprise"].map({"positive": -1, "negative": 1}) - df["c_ffbw"] = df["better_worse"].map({1: -1, 2: 1}).fillna(0) - df["c_bull"] = 1 - df["c_bear"] = -1 - - strategies = [("LLM", "c_llm"), ("doctrine(surprise)", "c_doctrine"), - ("FF betterWorse", "c_ffbw"), ("always-bullish", "c_bull"), - ("always-bearish", "c_bear")] - - for hname, _ in HORIZONS: - dcol = f"d_{hname}" - sub = df[df[dcol].notna()] - print(f"\n── horizon {hname} (n scored bars={len(sub)}) ──") - zeros = int((sub[dcol] == 0).sum()) - if zeros: - print(f" Δ==0 count (scored as miss): {zeros}") - for label, col in strategies: - s = sub[sub[col] != 0] - hits = int((np.sign(s[dcol]) == s[col]).sum()) - print(" " + _fmt_rate(hits, len(s), label)) - # noise-filtered LLM hit rate - s = sub[(sub["c_llm"] != 0) & - (sub[dcol].abs() >= null_med[hname])] - hits = int((np.sign(s[dcol]) == s["c_llm"]).sum()) - print(" " + _fmt_rate(hits, len(s), - f"LLM |Δ|≥null-med")) - # event-window magnitude vs null - mag = sub[dcol].abs().median() - print(f" event-window median |Δ| = ${mag:.2f} " - f"(null ${null_med[hname]:.2f})") - # paired McNemar: LLM vs each baseline on common directional set - for label, col in [("doctrine", "c_doctrine"), - ("FF betterWorse", "c_ffbw"), - ("always-bullish", "c_bull")]: - s = sub[(sub["c_llm"] != 0) & (sub[col] != 0)] - ha = list(np.sign(s[dcol]) == s["c_llm"]) - hb = list(np.sign(s[dcol]) == s[col]) - b, c, p = _mcnemar_p(ha, hb) - print(f" McNemar LLM vs {label:<15} b={b:>3} c={c:>3} " - f"p={p:.3f}") - - # neutral-call flatness (15m) - neu = df[(df["llm_direction"] == "neutral") & df["d_15m"].notna()] - dire = df[(df["c_llm"] != 0) & df["d_15m"].notna()] - if len(neu): - print(f"\nneutral calls (n={len(neu)}): median |Δ_15m| " - f"${neu['d_15m'].abs().median():.2f} vs directional " - f"(n={len(dire)}) ${dire['d_15m'].abs().median():.2f} " - f"(null ${null_med['15m']:.2f})") - - # breakdowns at 1h - dcol = "d_1h" - sub = df[df[dcol].notna() & (df["c_llm"] != 0)] - print("\n── LLM hit rate by event name (1h, top 10 by count) ──") - for name, grp in sorted(sub.groupby("name"), key=lambda kv: -len(kv[1]))[:10]: - hits = int((np.sign(grp[dcol]) == grp["c_llm"]).sum()) - print(" " + _fmt_rate(hits, len(grp), name[:24])) - print("── LLM hit rate by year (1h) ──") - for yr, grp in sub.groupby("year"): - hits = int((np.sign(grp[dcol]) == grp["c_llm"]).sum()) - print(" " + _fmt_rate(hits, len(grp), yr)) - - # cluster view: shared datelines - print("\n── same-timestamp clusters ──") - clus = df.groupby("dateline").filter(lambda g: len(g) > 1) - groups = list(clus.groupby("dateline")) - consistent = 0 - net_hits, net_n = 0, 0 - for _, g in groups: - signs = set(g.loc[g["c_llm"] != 0, "c_llm"]) - if len(signs) <= 1: - consistent += 1 - net = int(g["c_llm"].sum()) - d = g["d_1h"].iloc[0] - if net != 0 and pd.notna(d) and d != 0: - net_n += 1 - if np.sign(d) == np.sign(net): - net_hits += 1 - print(f"clusters: {len(groups)} datelines " - f"({len(clus)} events, {len(clus) / max(len(df), 1) * 100:.0f}% of " - f"corpus) | internally consistent: {consistent}/{len(groups)}") - print(" " + _fmt_rate(net_hits, net_n, "cluster net-call (1h)")) - - # determinism probe - p2 = {r["ffid"]: r for r in _load_jsonl(EVENTS_JSONL) - if r.get("run_pass") == 2 and r.get("llm_direction")} - if p2: - p1 = {r["ffid"]: r for r in recs} - agree = same_reason = n = 0 - for fid, r2 in p2.items(): - r1 = p1.get(fid) - if not r1 or not r1.get("llm_direction"): - continue - n += 1 - if r1["llm_direction"] == r2["llm_direction"]: - agree += 1 - if r1.get("reason_raw") == r2.get("reason_raw"): - same_reason += 1 - print(f"\ndeterminism probe: direction agreement {agree}/{n}, " - f"identical reason text {same_reason}/{n}") - - # stratified reason examples - print("\n── reason samples (1h hit/miss) ──") - sub = df[df[dcol].notna() & (df["c_llm"] != 0)].copy() - sub["hit"] = np.sign(sub[dcol]) == sub["c_llm"] - for verdict in (True, False): - picked, seen = [], set() - for _, r in sub[sub["hit"] == verdict].iterrows(): - if r["name"] in seen: - continue - seen.add(r["name"]) - picked.append(r) - if len(picked) == 10: - break - tag = "HIT " if verdict else "MISS" - for r in picked: - print(f" [{tag}] {r['utc_time'][:16]} {r['name'][:30]:<30} " - f"a={r['actual']} f={r['forecast']} → " - f"{r['llm_direction']:<7} Δ1h={r[dcol]:+7.2f} " - f"“{(r['reason_raw'] or '')[:60]}”") - print(f"\nscored table → {SCORED_CSV}") - - -def report_headlines(): - recs = _load_jsonl(HEADLINES_JSONL) - if not recs: - print("\n(no headline replay records — run `headlines` first)") - return - g = _load_golden() - engine = NewsEngine(NewsConfig()) - - def collect(tag): - m = {} - for r in recs: - if r["run_tag"] == tag and r["scores"]: - for h, sc in zip(r["headlines"], r["scores"]): - m[h] = tuple(sc) - return m - - main, shuf = collect("main"), collect("shuffled") - print("\n" + "═" * 70) - print("PART B — golden headline set: LLM vs keyword baseline") - print("═" * 70) - missing = [h for h in g["headline"] if h not in main] - if missing: - print(f"WARNING: {len(missing)} headlines missing from main run") - - def cls(x): - if x >= NEUTRAL_BAND: - return 1 - if x <= -NEUTRAL_BAND: - return -1 - return 0 - - rows = [] - for _, r in g.iterrows(): - h = r["headline"] - if h not in main: - continue - llm_xau = main[h][1] - kw = engine._analyze_headline_sentiment(h, for_currency="XAU") - rows.append({ - "headline": h, "category": r["category"], - "exp": int(r["expected_xau_sign"]), - "llm_xau": llm_xau, "llm_cls": cls(llm_xau), - "kw": kw, "kw_cls": cls(kw), - }) - t = pd.DataFrame(rows) - for who in ("llm", "kw"): - acc = (t[f"{who}_cls"] == t["exp"]).mean() - print(f"{who.upper():<4} 3-class accuracy (band ±{NEUTRAL_BAND}): " - f"{(t[f'{who}_cls'] == t['exp']).sum()}/{len(t)} " - f"= {acc * 100:.1f}%") - ha = list(t["llm_cls"] == t["exp"]) - hb = list(t["kw_cls"] == t["exp"]) - b, c, p = _mcnemar_p(ha, hb) - print(f"paired McNemar LLM vs keyword: b={b} c={c} p={p:.3f}") - - print("\nper-category accuracy (LLM | keyword):") - for cat, grp in t.groupby("category"): - a1 = (grp["llm_cls"] == grp["exp"]).sum() - a2 = (grp["kw_cls"] == grp["exp"]).sum() - print(f" {cat:<18} {a1:>2}/{len(grp):<2} | {a2:>2}/{len(grp)}") - - print("\nLLM confusion (rows=expected, cols=predicted -1/0/+1):") - cm = pd.crosstab(t["exp"], t["llm_cls"]).reindex( - index=[-1, 0, 1], columns=[-1, 0, 1], fill_value=0) - print(cm.to_string()) - - print("\nLLM misclassified headlines:") - for _, r in t[t["llm_cls"] != t["exp"]].iterrows(): - print(f" exp {r['exp']:+d} got {r['llm_xau']:+.2f} " - f"[{r['category']}] {r['headline'][:70]}") - - # repeat-batch determinism - m_b0 = [r for r in recs if r["run_tag"] == "main" and r["batch"] == 0] - r_b0 = [r for r in recs if r["run_tag"] == "repeat"] - if m_b0 and r_b0 and m_b0[0]["scores"] and r_b0[0]["scores"]: - a = np.array(m_b0[0]["scores"], dtype=float) - b_ = np.array(r_b0[0]["scores"], dtype=float) - exact = int((a == b_).all(axis=1).sum()) - print(f"\nrepeat-batch: {exact}/{len(a)} score pairs identical, " - f"max |diff| = {np.abs(a - b_).max():.3f}") - - # shuffled-order drift - if shuf: - both = [h for h in t["headline"] if h in shuf] - drift = [abs(main[h][1] - shuf[h][1]) for h in both] - if drift: - print(f"shuffled-order xau drift over {len(both)} headlines: " - f"mean {np.mean(drift):.3f}, max {np.max(drift):.3f}") - - -def cmd_report(args): - report_events() - report_headlines() - - -# ── CLI ────────────────────────────────────────────────────────────── - -def main(): - ap = argparse.ArgumentParser(description=__doc__.split("\n")[1]) - sub = ap.add_subparsers(dest="cmd", required=True) - - e = sub.add_parser("events", help="replay release surprises") - e.add_argument("--limit", type=int, default=0, - help="smoke-test on first N events") - e.add_argument("--determinism", type=int, default=0, - help="re-call N stratified events a second time") - e.add_argument("--sleep", type=float, default=0.3) - e.add_argument("--retry-passes", type=int, default=3) - e.add_argument("--timeout", type=float, default=30.0) - e.set_defaults(fn=cmd_events) - - h = sub.add_parser("headlines", help="golden-set probes") - h.add_argument("--sleep", type=float, default=0.3) - h.add_argument("--timeout", type=float, default=30.0) - h.set_defaults(fn=cmd_headlines) - - r = sub.add_parser("report", help="offline scoring report") - r.set_defaults(fn=cmd_report) - - args = ap.parse_args() - args.fn(args) - - -if __name__ == "__main__": - main() diff --git a/backtest/research/llm_news_golden.csv b/backtest/research/llm_news_golden.csv deleted file mode 100644 index 7c7c136..0000000 --- a/backtest/research/llm_news_golden.csv +++ /dev/null @@ -1,79 +0,0 @@ -headline,category,expected_xau_sign,rationale -"Powell says restrictive policy needed for longer as inflation proves sticky",hawkish_fed,-1,"higher-for-longer rates lift real yields, gold-bearish" -"Fed minutes show officials leaning toward two more rate hikes this year",hawkish_fed,-1,"more hikes = stronger dollar/yields" -"Hot CPI print fuels bets on additional Fed tightening",hawkish_fed,-1,"tightening expectations weigh on gold" -"Fed's Waller: higher-for-longer rates warranted by resilient economy",hawkish_fed,-1,"hawkish guidance" -"FOMC raises rates 50bp, flags further increases ahead",hawkish_fed,-1,"outsized hike plus hawkish guidance" -"Treasury yields spike as Fed pushes back on rate cut hopes",hawkish_fed,-1,"cut hopes dashed, yields up" -"Core inflation accelerates, cementing case for hawkish Fed stance",hawkish_fed,-1,"hawkish repricing" -"Fed dot plot shifts higher; markets price out easing for the year",hawkish_fed,-1,"easing priced out" -"Fed cuts rates 25bp and signals more easing to come",dovish_fed,1,"easing cycle is gold-bullish" -"Powell hints at rate cuts as inflation cools toward target",dovish_fed,1,"cut guidance" -"FOMC minutes reveal growing support for policy easing",dovish_fed,1,"dovish minutes" -"Fed's Williams says rate cuts appropriate later this year",dovish_fed,1,"cut guidance" -"Soft inflation data boosts odds of a dovish Fed pivot",dovish_fed,1,"pivot odds up" -"Fed pauses hikes, markets price aggressive easing cycle",dovish_fed,1,"easing priced in" -"Central banks turn dovish as global growth falters",dovish_fed,1,"global easing backdrop" -"Fed to restart bond purchases to support flagging economy",dovish_fed,1,"QE is gold-bullish" -"Dollar index hits 12-month high on robust US economic data",strong_usd,-1,"strong dollar pressures gold" -"US GDP grows 4.1% annualized, blowing past expectations",strong_usd,-1,"strong economy, hawkish implication" -"Blockbuster jobs report sends dollar surging across the board",strong_usd,-1,"strong jobs lift dollar and yields" -"US retail sales smash forecasts, underscoring consumer strength",strong_usd,-1,"strong data" -"Greenback rallies as US data outshines Europe and Asia",strong_usd,-1,"dollar strength" -"ISM services index jumps to two-year high on strong demand",strong_usd,-1,"strong data" -"Consumer confidence surges to post-pandemic high in the US",strong_usd,-1,"strong data" -"Dollar tumbles to eight-month low after dismal manufacturing data",weak_usd,1,"weak dollar lifts gold" -"US economy adds far fewer jobs than expected; dollar slides",weak_usd,1,"weak jobs, dovish implication" -"Greenback weakens as US retail sales unexpectedly contract",weak_usd,1,"weak data" -"US GDP growth revised down sharply, stoking slowdown fears",weak_usd,1,"slowdown is dovish" -"Dollar slumps as trade deficit balloons to record",weak_usd,1,"dollar weakness" -"Soft services PMI adds to evidence of cooling US economy",weak_usd,1,"cooling economy" -"US consumer sentiment plunges to two-year low",weak_usd,1,"weak data" -"Real yields climb to multi-year highs as TIPS sell off",real_yields,-1,"rising real yields raise gold's opportunity cost" -"10-year Treasury yield surges past 5% on supply glut",real_yields,-1,"yield spike" -"Rising real interest rates dim appeal of non-yielding assets",real_yields,-1,"explicit opportunity-cost story" -"Treasury yields sink as investors flock to bonds",real_yields,1,"falling yields support gold" -"Real yields turn negative as inflation outpaces rates",real_yields,1,"negative real yields are gold's best backdrop" -"Bond rally drives 10-year yield to lowest level in a year",real_yields,1,"falling yields" -"Geopolitical tensions escalate in Middle East, investors seek safety",safe_haven,1,"safe-haven bid" -"Stocks crater worldwide as banking crisis fears spread",safe_haven,1,"crisis flight to safety" -"Escalating conflict drives flight to safe-haven assets",safe_haven,1,"explicit haven flow" -"Investors dump risk assets after surprise sovereign default",safe_haven,1,"default shock" -"Military escalation on NATO border rattles global markets",safe_haven,1,"geopolitical stress" -"Wall Street plunges 4% in worst session since pandemic",safe_haven,1,"risk-off" -"Currency crisis in emerging markets sparks contagion fears",safe_haven,1,"contagion, haven bid" -"Nuclear rhetoric intensifies, sending investors to havens",safe_haven,1,"extreme geopolitical stress" -"Ceasefire agreement lifts global markets, haven demand fades",risk_on,-1,"haven unwind" -"Stocks roar to record highs as trade deal signed",risk_on,-1,"risk-on drains haven demand" -"Risk appetite returns as banking fears subside",risk_on,-1,"risk-on" -"Global equities rally on strong earnings and easing tensions",risk_on,-1,"risk-on" -"Gold hits record high as central banks keep buying bullion",gold_direct,1,"direct demand" -"Gold ETFs post biggest weekly inflows in three years",gold_direct,1,"investment inflows" -"China's central bank extends gold buying spree to 18th month",gold_direct,1,"official-sector demand" -"Miners struggle to meet gold demand as supply deficit widens",gold_direct,1,"supply deficit" -"Gold slides 2% as investors book profits after record run",gold_direct,-1,"profit-taking pressure" -"Gold tumbles below key support as ETF outflows accelerate",gold_direct,-1,"outflows" -"Bullion demand weakens as Indian jewellery buying stalls",gold_direct,-1,"physical demand soft" -"Speculators slash bullish gold bets as dollar strengthens",gold_direct,-1,"positioning unwind" -"Fed holds rates steady, signals no rate hike on the horizon",negation_traps,1,"dovish hold; keyword layer sees the phrase rate hike and flips it gold-bearish" -"Powell says no rate cuts until inflation shows sustained cooling",negation_traps,-1,"hawkish hold; keyword layer sees rate cut and calls it gold-bullish" -"Recession fears fade as US data streak beats estimates",negation_traps,-1,"fading fear is risk-on; keywords see recession plus beats and cancel out" -"Inflation fails to accelerate, easing pressure on the Fed",negation_traps,1,"disinflation opens door to cuts" -"Fed's Bowman: not yet confident inflation is beaten, cuts premature",negation_traps,-1,"hawkish pushback phrased around negations" -"Central bank stops short of hiking, but warns tightening may resume",negation_traps,-1,"hawkish hold with tightening bias" -"No safe-haven bid for gold despite equity wobble, analysts say",negation_traps,-1,"explicitly negated haven demand" -"Jobs report not as weak as feared, dollar recovers",negation_traps,-1,"less-bad data is dollar-positive" -"US manufacturing output plunges as new orders dry up",inverted_metrics,1,"weak economy is dovish/gold-bullish; keyword layer reads plunges as bearish" -"US factory activity slumps to three-year low",inverted_metrics,1,"weak data; keyword reads slumps as bearish" -"Consumer spending falls for third straight month",inverted_metrics,1,"weak data; keyword reads falls as bearish" -"Private payrolls growth slides to weakest pace since 2020",inverted_metrics,1,"weak jobs; keyword reads slides as bearish" -"Jobless claims hit two-year high as layoffs spread",inverted_metrics,1,"higher claims = weaker economy = gold-bullish" -"US unemployment rate hits 4.5%, highest since 2021",inverted_metrics,1,"higher unemployment = dovish = gold-bullish" -"Lakers clinch playoff berth with overtime win over Suns",distractors,0,"irrelevant" -"New Marvel film smashes box office records on opening weekend",distractors,0,"irrelevant" -"Apple unveils thinner MacBook with faster chip lineup",distractors,0,"irrelevant" -"Scientists discover new deep-sea species off Pacific coast",distractors,0,"irrelevant" -"Heavy rain forecast for the Midwest through the weekend",distractors,0,"irrelevant" -"Streaming service raises subscription prices in Europe",distractors,0,"irrelevant" -"Electric vehicle startup recalls 12,000 cars over software glitch",distractors,0,"irrelevant" -"Local elections see record turnout in several US states",distractors,0,"irrelevant" diff --git a/backtest/research/open_trade_bursts.py b/backtest/research/open_trade_bursts.py deleted file mode 100644 index 7ebef39..0000000 --- a/backtest/research/open_trade_bursts.py +++ /dev/null @@ -1,150 +0,0 @@ -"""What do violent bars do to OPEN trades, and can we protect them? Analysis tooling only. - -Replays every trade of a book on the raw bars with the backtester's exit model (SL-first, -gap-through at open, close-only trail ratchet, 300-bar timeout, $0.65 cost) and compares: - base : reproduce the book (validation: must match r_multiple for almost every trade) - flat_data : exit at the OPEN of the first bar whose close-hour enters {12,13} UTC (US data) - flat_asia : same for close-hour {0,1} UTC (Asian open) - tight_data : entering {12,13}, hold a protective stop 0.5R below the best price for the window - tight_asia : same for {0,1} - react_exit : a burst bar (range >= 3x prior ATR14) closes AGAINST the trade -> exit next open - react_tight : after an opposing burst, protective stop 0.5R below the best price for the rest -Plus a damage table: what happens to a trade when the first burst hits it (with vs against). - - python backtest/research/open_trade_bursts.py BOOK.csv PARQUET BAR_MINUTES ARM DIST -""" -import sys -import numpy as np, pandas as pd - -COST = 0.65; MAX_BARS = 300; K = 3.0; PROT = 0.5 - - -def wilder_atr(df, period=14): - pc = df["Close"].shift(1) - tr = pd.concat([df["High"] - df["Low"], (df["High"] - pc).abs(), (df["Low"] - pc).abs()], axis=1).max(axis=1) - return tr.ewm(alpha=1 / period, min_periods=period, adjust=False).mean() - - -def simulate(bars, i0, direction, entry, sl, arm, dist, variant, hours_flat=(), hours_tight=(), flat_mask=None, tight_mask=None): - """bars: dict of numpy arrays; i0 = index of the entry bar. Returns (exit_price, bars_held, reason, flag).""" - O, H, L, C, burst, chour = bars["O"], bars["H"], bars["L"], bars["C"], bars["burst"], bars["chour"] - d = 1.0 if direction == "BUY" else -1.0 - risk = abs(entry - sl); stop = sl; peak = entry; flag = 0 - n = len(O); tight = False; tight_until = -1; exit_next = False - for j in range(i0, min(n, i0 + MAX_BARS)): - if exit_next: - return O[j], j - i0, "REACT", flag - h = chour[j] - in_flat = h in hours_flat; in_tight = h in hours_tight - if variant.startswith("flat") and in_flat: - return O[j], j - i0, "FLAT", 1 - if variant == "mask_flat" and flat_mask is not None and flat_mask[j]: - return O[j], j - i0, "FLAT", 1 - if variant.startswith("tight") and in_tight: - tight = True; tight_until = j; flag = 1 - if variant == "mask_tight" and tight_mask is not None and tight_mask[j]: - tight = True; tight_until = j; flag = 1 - # stop check (SL first, gap-through at open) - if d > 0: - if O[j] <= stop: return O[j], j - i0, "SL", flag - if L[j] <= stop: return stop, j - i0, "SL", flag - else: - if O[j] >= stop: return O[j], j - i0, "SL", flag - if H[j] >= stop: return stop, j - i0, "SL", flag - # update peak / trail at bar close - peak = max(peak, H[j]) if d > 0 else min(peak, L[j]) - fav = d * (peak - entry) - if fav >= arm * risk: - stop = max(stop, peak - dist * risk) if d > 0 else min(stop, peak + dist * risk) - if tight and (variant == "react_tight" or j <= tight_until or in_tight): - stop = max(stop, peak - PROT * risk) if d > 0 else min(stop, peak + PROT * risk) - if variant.startswith("react") and burst[j] and d * (C[j] - O[j]) < 0: - flag = 1 - if variant == "react_exit": exit_next = True - else: tight = True - j = min(n - 1, i0 + MAX_BARS - 1) - return C[j], j - i0, "TIMEOUT", flag - - -def r_of(direction, entry, sl, exit_price): - d = 1.0 if direction == "BUY" else -1.0 - return (d * (exit_price - entry) - COST) / abs(entry - sl) - - -def stats(r): - r = np.asarray(r); eq = np.concatenate([[0.0], np.cumsum(r)]); dd = (np.maximum.accumulate(eq) - eq).max() - pf = r[r > 0].sum() / max(-r[r < 0].sum(), 1e-9) - return dict(n=len(r), total=r.sum(), mean=r.mean(), wr=100 * (r > 0).mean(), pf=pf, dd=dd) - - -def main(): - book_csv, pq, bm, arm, dist = sys.argv[1], sys.argv[2], int(sys.argv[3]), float(sys.argv[4]), float(sys.argv[5]) - book = pd.read_csv(book_csv); book["entry_time"] = pd.to_datetime(book["entry_time"], utc=True, format="mixed") - px = pd.read_parquet(pq) - atr_prev = wilder_atr(px).shift(1) - bars = dict(O=px["Open"].values, H=px["High"].values, L=px["Low"].values, C=px["Close"].values, - burst=((px["High"] - px["Low"]) >= K * atr_prev).fillna(False).values, - chour=(px.index + pd.Timedelta(minutes=bm)).hour.values) - idx = pd.Index(px.index) - pos = idx.get_indexer(book["entry_time"]) - ok = pos >= 0 - print(f"{book_csv}: {len(book)} trades, {ok.sum()} matched to bars; arm {arm} dist {dist}") - book = book[ok].reset_index(drop=True); pos = pos[ok] - half = np.where(np.arange(len(book)) < len(book) // 2, 1, 2) - variants = {"base": {}, "flat_data": dict(hours_flat=(12, 13)), "flat_asia": dict(hours_flat=(0, 1)), - "tight_data": dict(hours_tight=(12, 13)), "tight_asia": dict(hours_tight=(0, 1)), - "react_exit": {}, "react_tight": {}} - results = {} - for v, kw in variants.items(): - rs, flags = [], [] - for i, t in book.iterrows(): - xp, held, reason, flag = simulate(bars, pos[i], t.direction, t.entry, t.sl, arm, dist, v, **kw) - rs.append(r_of(t.direction, t.entry, t.sl, xp)); flags.append(flag) - results[v] = (np.array(rs), np.array(flags)) - base = results["base"][0] - match = np.isclose(base, book["r_multiple"].values, atol=0.02).mean() - print(f"validation: replay reproduces the book's r_multiple (±0.02) on {100*match:.1f}% of trades; " - f"sum replay {base.sum():+.2f} vs book {book.r_multiple.sum():+.2f}") - print("\n== PROTECTION VARIANTS (whole book; 'affected' = trades where the rule fired) ==") - print(f"{'variant':12s} {'n':>5s} {'affected':>8s} {'total R':>9s} {'Δ vs base':>9s} {'PF':>5s} {'maxDD':>6s} {'WR%':>5s} | affected only: base R -> variant R | Δ h1 / Δ h2") - b = stats(base) - for v, (r, f) in results.items(): - s = stats(r); aff = f == 1 - d1 = r[half == 1].sum() - base[half == 1].sum(); d2 = r[half == 2].sum() - base[half == 2].sum() - print(f"{v:12s} {s['n']:5d} {aff.sum():8d} {s['total']:+9.2f} {s['total']-b['total']:+9.2f} {s['pf']:5.2f} {s['dd']:6.2f} {s['wr']:5.1f} | " - f"{base[aff].sum():+8.2f} -> {r[aff].sum():+8.2f} (n={aff.sum()}) | {d1:+.2f} / {d2:+.2f}") - # damage table: first burst while open (base path), with vs against - print("\n== DAMAGE: first violent bar that hits an OPEN trade (base exits) ==") - rows = [] - for i, t in book.iterrows(): - d = 1.0 if t.direction == "BUY" else -1.0; risk = abs(t.entry - t.sl) - xp, held, reason, _ = simulate(bars, pos[i], t.direction, t.entry, t.sl, arm, dist, "base") - j0 = pos[i]; j1 = j0 + held - for j in range(j0 + 1, j1 + 1): # bursts after the entry bar, up to and including the exit bar - if bars["burst"][j]: - against = d * (bars["C"][j] - bars["O"][j]) < 0 - mtm_before = (d * (bars["C"][j - 1] - t.entry) - COST) / risk - mtm_after = (d * (bars["C"][j] - t.entry) - COST) / risk - rows.append(dict(against=against, bars_open=j - j0, mtm_before=mtm_before, mtm_after=mtm_after, - final=r_of(t.direction, t.entry, t.sl, xp), ended_here=(j == j1), hour=bars["chour"][j], - sched=bars["chour"][j] in (0, 1, 12, 13))) - break - dmg = pd.DataFrame(rows) - if len(dmg): - g = dmg.groupby("against").agg(n=("final", "size"), bars_open_med=("bars_open", "median"), - mtm_before=("mtm_before", "mean"), mtm_after=("mtm_after", "mean"), - final=("final", "mean"), ended_on_that_bar_pct=("ended_here", lambda s: 100 * s.mean()), - in_sched_hours_pct=("sched", lambda s: 100 * s.mean())) - print(g.round(3).to_string()) - ag = dmg[dmg.against] - print(f"\nopposing bursts: {len(ag)} trades ({100*len(ag)/len(book):.1f}% of the book); " - f"mean R just before {ag.mtm_before.mean():+.3f} -> just after {ag.mtm_after.mean():+.3f} -> final {ag.final.mean():+.3f}; " - f"{100*(ag.final<0).mean():.0f}% end as losers; {100*ag.ended_here.mean():.0f}% are stopped ON that bar") - print(f" those with an open PROFIT before the burst (mtm>0): n={int((ag.mtm_before>0).sum())}, final mean {ag[ag.mtm_before>0].final.mean():+.3f}, " - f"{100*(ag[ag.mtm_before>0].final<0).mean():.0f}% end as losers") - print(f" by hour of the opposing burst (n, mean final R): " + ", ".join(f"{h:02d}:{n}/{m:+.2f}" for h, n, m in - ag.groupby("hour").final.agg(["size", "mean"]).reset_index().itertuples(index=False))) - - -if __name__ == "__main__": - main() diff --git a/backtest/research/panel_payload_drytest_2026-08-15.jsonl b/backtest/research/panel_payload_drytest_2026-08-15.jsonl deleted file mode 100644 index 022f7e8..0000000 --- a/backtest/research/panel_payload_drytest_2026-08-15.jsonl +++ /dev/null @@ -1,7 +0,0 @@ -{"probe": "P1 coherent-bullish", "payload": {"brief": {"pair": "XAU/USD", "timeframe": "15min", "direction": "BUY", "quality": "A", "score": 38.5, "confidence": 62, "price": 2000.0, "market_regime": "trending_up", "volatility_regime": "normal", "mtf_agreement": true}, "indicators": {"ema_trend": {"signal": "STRONG_BUY", "confidence": 90, "value": 1.0}, "rsi": {"signal": "BUY", "confidence": 65, "value": 58.0}, "macd": {"signal": "STRONG_BUY", "confidence": 80, "value": 1.25}, "bollinger": {"signal": "BUY", "confidence": 60, "value": 0.7}, "adx": {"signal": "BUY", "confidence": 70, "value": 32.0}, "roc": {"signal": "BUY", "confidence": 55, "value": 0.35}, "keltner": {"signal": "BUY", "confidence": 65, "value": 3.1}, "candlestick": {"signal": "BUY", "confidence": 60, "value": 2.0}}, "mtf": {"5min": {"signal": "BUY", "score": 30.0}, "1h": {"signal": "STRONG_BUY", "score": 45.0}}, "key_levels": {"resistance": 2012.0, "support": 1985.0}, "exhaustion": 0, "candles": {"15min": {"ref_close": 2000.0, "atr": 5.0, "bars": [[-2.4, -2.2, -2.4, -2.3], [-2.3, -2.1, -2.3, -2.2], [-2.2, -2.0, -2.2, -2.0], [-2.0, -1.9, -2.1, -1.9], [-1.9, -1.8, -2.0, -1.8], [-1.8, -1.6, -1.8, -1.7], [-1.7, -1.5, -1.7, -1.6], [-1.6, -1.4, -1.6, -1.4], [-1.4, -1.3, -1.5, -1.3], [-1.3, -1.2, -1.4, -1.2], [-1.2, -1.0, -1.2, -1.1], [-1.1, -0.9, -1.1, -1.0], [-1.0, -0.8, -1.0, -0.8], [-0.8, -0.7, -0.9, -0.7], [-0.7, -0.6, -0.8, -0.6], [-0.6, -0.4, -0.6, -0.5], [-0.5, -0.3, -0.5, -0.4], [-0.4, -0.2, -0.4, -0.2], [-0.2, -0.1, -0.3, -0.1], [-0.1, 0.0, -0.2, 0.0]]}}}, "raw_reply": {"stance": "support", "confidence": 70, "note": "The trend is up and all signs agree, so buying fits the flow."}} -{"probe": "P2 bearish-vs-BUY-brief", "payload": {"brief": {"pair": "XAU/USD", "timeframe": "15min", "direction": "BUY", "quality": "A", "score": 38.5, "confidence": 62, "price": 2000.0, "market_regime": "trending_down", "volatility_regime": "normal", "mtf_agreement": false}, "indicators": {"ema_trend": {"signal": "STRONG_SELL", "confidence": 85, "value": 0.1}, "rsi": {"signal": "SELL", "confidence": 70, "value": 74.0}, "macd": {"signal": "STRONG_SELL", "confidence": 80, "value": -1.4}, "bollinger": {"signal": "SELL", "confidence": 60, "value": 0.95}, "adx": {"signal": "SELL", "confidence": 70, "value": 35.0}, "roc": {"signal": "SELL", "confidence": 60, "value": -0.5}, "keltner": {"signal": "SELL", "confidence": 65, "value": -2.8}, "candlestick": {"signal": "SELL", "confidence": 55, "value": -2.0}}, "mtf": {"5min": {"signal": "SELL", "score": -28.0}, "1h": {"signal": "STRONG_SELL", "score": -41.0}}, "key_levels": {"resistance": 2012.0, "support": 1985.0}, "exhaustion": 3, "candles": {"15min": {"ref_close": 2000.0, "atr": 5.0, "bars": [[2.4, 2.4, 2.2, 2.3], [2.3, 2.3, 2.1, 2.2], [2.2, 2.2, 2.0, 2.0], [2.0, 2.1, 1.9, 1.9], [1.9, 2.0, 1.8, 1.8], [1.8, 1.8, 1.6, 1.7], [1.7, 1.7, 1.5, 1.6], [1.6, 1.6, 1.4, 1.4], [1.4, 1.5, 1.3, 1.3], [1.3, 1.4, 1.2, 1.2], [1.2, 1.2, 1.0, 1.1], [1.1, 1.1, 0.9, 1.0], [1.0, 1.0, 0.8, 0.8], [0.8, 0.9, 0.7, 0.7], [0.7, 0.8, 0.6, 0.6], [0.6, 0.6, 0.4, 0.5], [0.5, 0.5, 0.3, 0.4], [0.4, 0.4, 0.2, 0.2], [0.2, 0.3, 0.1, 0.1], [0.1, 0.2, -0.0, 0.0]]}}}, "raw_reply": {"stance": "oppose", "confidence": 85, "note": "The price is falling and every indicator says sell, so buying fights the trend."}} -{"probe": "P3 conflicting", "payload": {"brief": {"pair": "XAU/USD", "timeframe": "15min", "direction": "BUY", "quality": "A", "score": 12.0, "confidence": 35, "price": 2000.0, "market_regime": "ranging", "volatility_regime": "normal", "mtf_agreement": false}, "indicators": {"ema_trend": {"signal": "BUY", "confidence": 55, "value": 0.6}, "rsi": {"signal": "NEUTRAL", "confidence": 40, "value": 51.0}, "macd": {"signal": "SELL", "confidence": 55, "value": -0.3}, "bollinger": {"signal": "NEUTRAL", "confidence": 35, "value": 0.5}, "adx": {"signal": "NEUTRAL", "confidence": 40, "value": 18.0}, "roc": {"signal": "SELL", "confidence": 50, "value": -0.1}, "keltner": {"signal": "BUY", "confidence": 50, "value": 0.8}, "candlestick": {"signal": "NEUTRAL", "confidence": 30, "value": 0.0}}, "mtf": {"5min": {"signal": "NEUTRAL", "score": 4.0}, "1h": {"signal": "NEUTRAL", "score": -3.0}}, "key_levels": {}, "exhaustion": 0, "candles": {"15min": {"ref_close": 2000.0, "atr": 5.0, "bars": [[0.0, 0.4, -0.1, 0.3], [0.3, 0.4, -0.1, 0.0], [0.0, 0.4, -0.1, 0.3], [0.3, 0.4, -0.1, 0.0], [0.0, 0.4, -0.1, 0.3], [0.3, 0.4, -0.1, 0.0], [0.0, 0.4, -0.1, 0.3], [0.3, 0.4, -0.1, 0.0], [0.0, 0.4, -0.1, 0.3], [0.3, 0.4, -0.1, 0.0], [0.0, 0.4, -0.1, 0.3], [0.3, 0.4, -0.1, 0.0], [0.0, 0.4, -0.1, 0.3], [0.3, 0.4, -0.1, 0.0], [0.0, 0.4, -0.1, 0.3], [0.3, 0.4, -0.1, 0.0], [0.0, 0.4, -0.1, 0.3], [0.3, 0.4, -0.1, 0.0], [0.0, 0.4, -0.1, 0.3], [0.3, 0.4, -0.1, 0.0]]}}}, "raw_reply": {"stance": "neutral", "confidence": 30, "note": "Mixed signals with no clear direction, so cannot strongly support or oppose."}} -{"probe": "P4 empty-evidence", "payload": {"brief": {"pair": "XAU/USD", "timeframe": "15min", "direction": "BUY", "quality": "A", "score": 38.5, "confidence": 62, "price": 2000.0, "market_regime": "trending_up", "volatility_regime": "normal", "mtf_agreement": true}, "indicators": {}, "mtf": {}, "key_levels": {}, "exhaustion": 0}, "raw_reply": {"stance": "neutral", "confidence": 20, "note": "Not enough data to judge the trade direction."}} -{"probe": "P5 hallucination-trap", "payload": {"brief": {"pair": "XAU/USD", "timeframe": "15min", "direction": "BUY", "quality": "A", "score": 38.5, "confidence": 62, "price": 2000.0, "market_regime": "trending_up", "volatility_regime": "normal", "mtf_agreement": true}, "indicators": {"ema_trend": {"signal": "STRONG_BUY", "confidence": 90, "value": 1.0}, "rsi": {"signal": "BUY", "confidence": 65, "value": 58.0}, "macd": {"signal": "STRONG_BUY", "confidence": 80, "value": 1.25}, "bollinger": {"signal": "BUY", "confidence": 60, "value": 0.7}, "adx": {"signal": "BUY", "confidence": 70, "value": 32.0}, "roc": {"signal": "BUY", "confidence": 55, "value": 0.35}, "keltner": {"signal": "BUY", "confidence": 65, "value": 3.1}, "candlestick": {"signal": "BUY", "confidence": 60, "value": 2.0}}, "mtf": {"5min": {"signal": "BUY", "score": 30.0}, "1h": {"signal": "STRONG_BUY", "score": 45.0}}, "key_levels": {}, "exhaustion": 0}, "raw_reply": {"stance": "support", "confidence": 70, "note": "The trend is up and all signs agree, so buying fits the flow."}} -{"probe": "P6 output-discipline", "payload": {}, "raw_reply": null} -{"probe": "P7 synthesizer", "payload": {"direction": "BUY", "takes": [{"persona": "macro", "stance": "support", "confidence": 70, "note": "evidence aligned"}, {"persona": "technical", "stance": "support", "confidence": 80, "note": "evidence aligned"}, {"persona": "risk", "stance": "support", "confidence": 65, "note": "evidence aligned"}, {"persona": "contrarian", "stance": "support", "confidence": 60, "note": "evidence aligned"}]}, "raw_reply": {"verdict": "AGREE", "confidence": 85, "rationale": "All four analysts support buying, with high confidence and consistent evidence. No one opposes, so the panel clearly backs the trade."}} diff --git a/backtest/research/signal_parity_2026-08-14.md b/backtest/research/signal_parity_2026-08-14.md index 7705890..48922fc 100644 --- a/backtest/research/signal_parity_2026-08-14.md +++ b/backtest/research/signal_parity_2026-08-14.md @@ -1,7 +1,7 @@ # Signal-Level Replay Parity — Oanda vs Tiingo (Phase 2) **Date:** 2026-08-14 -**Tool:** `scripts/signal_parity_study.py` (replays real `analyze_from_dataframes`, news off) +**Tool:** `scripts/signal_parity_study.py` (script removed 2026-09-06, in git history) (replays real `analyze_from_dataframes`, news off) **Window:** 2025-08-14 → 2026-08-14 (12 months); M15 stride 8, M5 stride 24 → ~2,900 sampled decisions each **Preceding:** Phase 1 price-diff study (`feed_divergence_2026-08-14.md`) diff --git a/backtest/research/sizing_model.py b/backtest/research/sizing_model.py deleted file mode 100644 index 47cd7f7..0000000 --- a/backtest/research/sizing_model.py +++ /dev/null @@ -1,961 +0,0 @@ -"""Stage-D stake-sizing model driver (2026-09 campaign). - -Walk-forward fit + evaluation of a per-signal stake multiplier on a trade -book exported by ``gpu_backtest.py --export-features`` (63 base columns + -``stake_multiplier``/``r_sized`` + 65 ``f_*`` decision-time features). - -The protocol implemented here is the one pre-registered in -``backtest/research/sizing_campaign_2026-09.md`` and is NOT to be varied to -rescue a result: - -* expanding walk-forward, 12-month burn-in, calendar-quarter refits, a - ``purge_bars × bar_minutes`` gap between the end of train and the start of - the test quarter, standardisation fit on train only; -* models: ridge on winsorised R (primary), L2 logistic on ``R > 0`` - (secondary), a one-feature ``f_is_buy`` simplicity baseline, an - exhaustion-block-only ablation, and an optional lightgbm upper bound; -* size map ``m = round_0.05(clip(normaliser × (0.5 + clip((z−lo)/(hi−lo),0,1)), - 0.5, 1.5))`` with ``lo``/``hi``/``normaliser`` calibrated on TRAIN - predictions only; -* a label-shuffle null (training labels permuted, test labels untouched); -* six acceptance clauses, printed PASS/FAIL individually. - -Usage: - python backtest/research/sizing_model.py --book BOOK.csv --bar-minutes 5 \ - --features f_a,f_b,... [--model ridge|logistic] [--out DIR] \ - [--n-shuffle 50] [--skip-lgbm] \ - [--write-artifact PATH --tf 5min --confirm-tfs 15min,1h] - -Analysis tooling only — never imported by the live code. -""" -from __future__ import annotations - -import argparse -import hashlib -import json -import math -import subprocess -import sys -import warnings -from pathlib import Path -from typing import Dict, List, Optional, Sequence, Tuple - -import numpy as np -import pandas as pd - -PROJECT_ROOT = Path(__file__).resolve().parents[2] -if str(PROJECT_ROOT) not in sys.path: - sys.path.insert(0, str(PROJECT_ROOT)) - -from position_sizing import FEATURE_ORDER, FEATURE_VERSION # noqa: E402 (read-only import) - -# ── Pre-registered constants (flag any change loudly) ────────────────── -WINSOR_LO, WINSOR_HI = -1.5, 4.0 -LAMBDA_GRID: Tuple[float, ...] = (1.0, 10.0, 100.0, 1000.0) -CV_FOLDS = 5 -MIN_TRAIN = 150 -MAP_Q_LO, MAP_Q_HI = 0.10, 0.90 -MAP_OUT_LO, MAP_OUT_HI = 0.5, 1.5 -MAP_CLIP_LO, MAP_CLIP_HI = 0.5, 1.5 -MAP_ROUND_TO = 0.05 -MEAN_MULT_TOL = 0.05 -DD_TOL = 1.10 -PF_FLOOR = 1.3 -NEWTON_MAX_ITER = 50 -NEWTON_TOL = 1e-8 - -#: The exhaustion-block ablation's fixed feature list. -EXHAUSTION_FEATURES: Tuple[str, ...] = ( - "f_swing_dist_atr", - "f_ema21_dist_atr", - "f_ema200_dist_atr", - "f_exhaustion_count", - "f_penalty_total", - "f_tag_bb_overextended", - "f_tag_rsi_extended", - "f_tag_no_pullback", - "f_tag_exhaustion", -) -ISBUY_FEATURES: Tuple[str, ...] = ("f_is_buy",) - -#: lightgbm never ships; running the full shuffle null through 200-tree fits -#: costs minutes for a number nobody acts on, so its null is capped here. -#: DEVIATION FROM THE PROTOCOL CONSTANT — printed in the summary when used. -LGBM_MAX_SHUFFLE = 10 - -LGBM_PARAMS = dict( - n_estimators=200, - learning_rate=0.03, - num_leaves=7, - min_child_samples=30, - subsample=0.8, - colsample_bytree=0.8, - reg_lambda=10, -) - -_LINES: List[str] = [] - - -def say(msg: str = "") -> None: - """Print and tee into the summary buffer.""" - print(msg) - _LINES.append(msg) - - -# ══════════════════════════════════════════════════════════════════════ -# Book loading -# ══════════════════════════════════════════════════════════════════════ -def load_book(path: Path) -> pd.DataFrame: - """Read a feature book, parse ``entry_time`` UTC, drop NaN-R rows.""" - df = pd.read_csv(path) - if "entry_time" not in df.columns or "r_multiple" not in df.columns: - raise SystemExit(f"{path}: not a trade book (no entry_time/r_multiple)") - df["entry_time"] = pd.to_datetime(df["entry_time"], utc=True, format="mixed") - n0 = len(df) - bad = ~np.isfinite(pd.to_numeric(df["r_multiple"], errors="coerce").to_numpy(float)) - if bad.any(): - say(f"WARNING: dropping {int(bad.sum())}/{n0} rows with non-finite r_multiple") - df = df.loc[~bad].copy() - df["r_multiple"] = df["r_multiple"].astype(float) - df = df.sort_values("entry_time", kind="mergesort").reset_index(drop=True) - return df - - -def resolve_features(df: pd.DataFrame, names: Sequence[str], label: str) -> List[str]: - """Validate a requested feature list against the book and FEATURE_ORDER.""" - missing = [n for n in names if n not in df.columns] - if missing: - raise SystemExit(f"{label}: columns absent from book: {', '.join(missing)}") - unknown = [n for n in names if n.startswith("f_") and n not in FEATURE_ORDER] - if unknown: - say(f"NOTE: {label}: {', '.join(unknown)} not in position_sizing.FEATURE_ORDER") - return list(names) - - -def sha256_file(path: Path) -> str: - h = hashlib.sha256() - with open(path, "rb") as fh: - for chunk in iter(lambda: fh.read(1 << 20), b""): - h.update(chunk) - return h.hexdigest() - - -def git_commit() -> str: - try: - return subprocess.check_output( - ["git", "rev-parse", "HEAD"], cwd=PROJECT_ROOT, text=True).strip() - except Exception: # pragma: no cover - research tool - return "unknown" - - -# ══════════════════════════════════════════════════════════════════════ -# Walk-forward geometry -# ══════════════════════════════════════════════════════════════════════ -def quarter_starts(first: pd.Timestamp, last: pd.Timestamp) -> List[pd.Timestamp]: - """Every Jan/Apr/Jul/Oct 1 00:00 UTC in ``[first, last]`` inclusive.""" - out: List[pd.Timestamp] = [] - year = first.year - while True: - for month in (1, 4, 7, 10): - ts = pd.Timestamp(year=year, month=month, day=1, tz="UTC") - if ts < first: - continue - if ts > last: - return out - out.append(ts) - year += 1 - - -def quarter_label(ts: pd.Timestamp) -> str: - return f"{ts.year}Q{(ts.month - 1) // 3 + 1}" - - -def build_folds(times: np.ndarray, n_folds: int) -> List[np.ndarray]: - """Contiguous-by-time index blocks (rows are already time-sorted).""" - n = len(times) - edges = np.linspace(0, n, n_folds + 1).astype(int) - return [np.arange(edges[i], edges[i + 1]) for i in range(n_folds) - if edges[i + 1] > edges[i]] - - -# ══════════════════════════════════════════════════════════════════════ -# Standardisation -# ══════════════════════════════════════════════════════════════════════ -def fit_standardiser(X: np.ndarray) -> Tuple[np.ndarray, np.ndarray, np.ndarray]: - """Return (mean, scale, keep) from a training matrix. - - NaNs are ignored when computing the mean (they are later filled with it). - ``scale`` is the population std with zeros replaced by 1.0; ``keep`` is - False for zero-variance-in-train columns, which are dropped from the fit - and carry a coefficient of 0. - """ - with warnings.catch_warnings(): - warnings.simplefilter("ignore", category=RuntimeWarning) - mean = np.nanmean(X, axis=0) - std = np.nanstd(X, axis=0) - mean = np.where(np.isfinite(mean), mean, 0.0) - std = np.where(np.isfinite(std), std, 0.0) - keep = std > 0 - scale = np.where(keep, std, 1.0) - return mean, scale, keep - - -def apply_standardiser(X: np.ndarray, mean: np.ndarray, scale: np.ndarray) -> np.ndarray: - """Fill NaN with the TRAIN mean, then standardise.""" - Z = np.array(X, dtype=float, copy=True) - bad = ~np.isfinite(Z) - if bad.any(): - Z[bad] = np.take(mean, np.where(bad)[1]) - return (Z - mean) / scale - - -# ══════════════════════════════════════════════════════════════════════ -# Models (numpy only) -# ══════════════════════════════════════════════════════════════════════ -def _solve(A: np.ndarray, b: np.ndarray) -> np.ndarray: - try: - return np.linalg.solve(A, b) - except np.linalg.LinAlgError: # pragma: no cover - degenerate design - return np.linalg.lstsq(A, b, rcond=None)[0] - - -def fit_ridge(X: np.ndarray, y: np.ndarray, lam: float) -> Tuple[np.ndarray, float]: - """Closed-form ridge with an UNPENALISED intercept. - - Centring X and y is exactly equivalent to appending a 1-column carrying - zero penalty. Returns ``(coef, intercept)`` with ``pred = intercept + X @ coef``. - """ - if X.shape[1] == 0: - return np.zeros(0), float(y.mean()) if len(y) else 0.0 - xm = X.mean(axis=0) - ym = float(y.mean()) - Xc = X - xm - w = _solve(Xc.T @ Xc + lam * np.eye(X.shape[1]), Xc.T @ (y - ym)) - return w, ym - float(xm @ w) - - -def fit_logistic(X: np.ndarray, y: np.ndarray, lam: float, - max_iter: int = NEWTON_MAX_ITER, - tol: float = NEWTON_TOL) -> Tuple[np.ndarray, float, float]: - """L2 logistic via Newton-Raphson, intercept unpenalised. - - Returns ``(coef, intercept, grad_norm)`` where ``grad_norm`` is the - infinity-norm of the penalised gradient at the returned point. - """ - n, p = X.shape - A = np.hstack([np.ones((n, 1)), X]) - pen = np.zeros(p + 1) - pen[1:] = lam - beta = np.zeros(p + 1) - # Warm-start the intercept at the empirical log-odds. - frac = float(np.clip(y.mean(), 1e-6, 1 - 1e-6)) - beta[0] = math.log(frac / (1 - frac)) - grad = np.zeros(p + 1) - for _ in range(max_iter): - eta = A @ beta - prob = 1.0 / (1.0 + np.exp(-np.clip(eta, -35, 35))) - grad = A.T @ (y - prob) - pen * beta - W = np.clip(prob * (1 - prob), 1e-10, None) - H = A.T @ (A * W[:, None]) + np.diag(pen) - step = _solve(H, grad) - beta = beta + step - if np.max(np.abs(step)) < tol: - break - eta = A @ beta - prob = 1.0 / (1.0 + np.exp(-np.clip(eta, -35, 35))) - grad = A.T @ (y - prob) - pen * beta - return beta[1:], float(beta[0]), float(np.max(np.abs(grad))) - - -def predict_linear(X: np.ndarray, coef: np.ndarray, intercept: float) -> np.ndarray: - if X.shape[1] == 0: - return np.full(len(X), intercept, dtype=float) - return intercept + X @ coef - - -def sigmoid(z: np.ndarray) -> np.ndarray: - return 1.0 / (1.0 + np.exp(-np.clip(z, -35, 35))) - - -def log_loss(y: np.ndarray, p: np.ndarray) -> float: - p = np.clip(p, 1e-12, 1 - 1e-12) - return float(-np.mean(y * np.log(p) + (1 - y) * np.log(1 - p))) - - -def choose_lambda(X: np.ndarray, y: np.ndarray, kind: str, - grid: Sequence[float] = LAMBDA_GRID, - n_folds: int = CV_FOLDS) -> Tuple[float, Dict[float, float]]: - """5-fold time-contiguous CV inside the training window.""" - folds = build_folds(np.arange(len(y)), n_folds) - if X.shape[1] == 0 or len(folds) < 2: - return float(grid[0]), {} - scores: Dict[float, float] = {} - for lam in grid: - vals: List[float] = [] - for hold in folds: - mask = np.ones(len(y), dtype=bool) - mask[hold] = False - if mask.sum() < 2: - continue - if kind == "logistic": - ytr = y[mask] - if ytr.min() == ytr.max(): # single-class fold - p = np.full(len(hold), float(np.clip(ytr.mean(), 1e-6, 1 - 1e-6))) - else: - c, b, _ = fit_logistic(X[mask], ytr, lam) - p = sigmoid(predict_linear(X[hold], c, b)) - vals.append(log_loss(y[hold], p)) - else: - c, b = fit_ridge(X[mask], y[mask], lam) - resid = y[hold] - predict_linear(X[hold], c, b) - vals.append(float(np.mean(resid ** 2))) - scores[float(lam)] = float(np.mean(vals)) if vals else float("inf") - best = min(scores, key=lambda k: scores[k]) - return best, scores - - -def lgbm_available() -> bool: - try: - import lightgbm # noqa: F401 - return True - except Exception: - return False - - -def fit_lgbm(X: np.ndarray, y: np.ndarray, seed: int): - import lightgbm as lgb - model = lgb.LGBMRegressor(random_state=seed, verbose=-1, **LGBM_PARAMS) - model.fit(X, y) - return model - - -# ══════════════════════════════════════════════════════════════════════ -# Size map -# ══════════════════════════════════════════════════════════════════════ -def round_to(x, step: float = MAP_ROUND_TO) -> np.ndarray: - """Round to the nearest ``step`` and kill float dust (0.05 → 2 dp).""" - decimals = max(0, int(round(-math.log10(step)))) + 1 - return np.round(np.round(np.asarray(x, dtype=float) / step) * step, decimals) - - -def calibrate_map(z_train: np.ndarray) -> Dict[str, float]: - """lo/hi/normaliser from TRAIN predictions only.""" - lo = float(np.quantile(z_train, MAP_Q_LO)) - hi = float(np.quantile(z_train, MAP_Q_HI)) - if not np.isfinite(lo) or not np.isfinite(hi) or hi <= lo: - return {"lo": lo, "hi": hi, "normalizer": 1.0, "degenerate": True} - raw = MAP_OUT_LO + (MAP_OUT_HI - MAP_OUT_LO) * np.clip((z_train - lo) / (hi - lo), 0, 1) - mean_raw = float(np.mean(raw)) - norm = 1.0 / mean_raw if mean_raw > 0 else 1.0 - return {"lo": lo, "hi": hi, "normalizer": norm, "degenerate": False} - - -def apply_map(z: np.ndarray, cal: Dict[str, float]) -> np.ndarray: - z = np.asarray(z, dtype=float) - if cal.get("degenerate"): - return np.ones(len(z)) - lo, hi, norm = cal["lo"], cal["hi"], cal["normalizer"] - raw = MAP_OUT_LO + (MAP_OUT_HI - MAP_OUT_LO) * np.clip((z - lo) / (hi - lo), 0, 1) - return round_to(np.clip(raw * norm, MAP_CLIP_LO, MAP_CLIP_HI)) - - -# ══════════════════════════════════════════════════════════════════════ -# Metrics -# ══════════════════════════════════════════════════════════════════════ -def max_drawdown(r: np.ndarray) -> float: - """0-prepended cumulative-R drawdown (gpu_backtest.compute_metrics).""" - r = np.asarray(r, dtype=float) - if len(r) == 0: - return 0.0 - cum = np.concatenate(([0.0], np.cumsum(r))) - return float((np.maximum.accumulate(cum) - cum).max()) - - -def profit_factor(r: np.ndarray) -> float: - r = np.asarray(r, dtype=float) - gp = float(r[r > 0].sum()) if (r > 0).any() else 0.0 - gl = float(-r[r < 0].sum()) if (r < 0).any() else 0.0 - return gp / gl if gl > 0 else float("inf") - - -def multiplier_hist(m: np.ndarray) -> pd.DataFrame: - """Counts in 0.1-wide bins from 0.5 to 1.5. - - ``np.histogram`` already closes the final bin on the right, so an - exactly-1.5 multiplier lands in [1.4, 1.5] once and the counts sum to - ``len(m)``. - """ - edges = np.round(np.arange(MAP_OUT_LO, MAP_OUT_HI + 1e-9, 0.1), 2) - counts, _ = np.histogram(np.asarray(m, dtype=float), bins=edges) - counts = counts.astype(int) - return pd.DataFrame({ - "bin_lo": edges[:-1], "bin_hi": edges[1:], "count": counts, - "pct": np.where(len(m), counts / max(len(m), 1) * 100, 0.0), - }) - - -# ══════════════════════════════════════════════════════════════════════ -# Walk-forward driver -# ══════════════════════════════════════════════════════════════════════ -def walk_forward(df: pd.DataFrame, features: Sequence[str], model: str, - bar_minutes: int, burn_in_months: int, purge_bars: int, - seed: int, y_train_override: Optional[np.ndarray] = None, - verbose: bool = True) -> Dict: - """One full expanding walk-forward pass. - - ``y_train_override`` (full-book length, aligned to ``df``) replaces the - TRAINING labels only — this is how the shuffle null is run. Test labels - always come from the real ``r_multiple``. - """ - times = df["entry_time"].to_numpy() - r_all = df["r_multiple"].to_numpy(float) - r_train_src = r_all if y_train_override is None else np.asarray(y_train_override, float) - X_all = df[list(features)].apply(pd.to_numeric, errors="coerce").to_numpy(float) \ - if features else np.zeros((len(df), 0)) - - book_start = df["entry_time"].iloc[0] - book_end = df["entry_time"].iloc[-1] - burn_end = book_start + pd.DateOffset(months=burn_in_months) - purge = pd.Timedelta(minutes=purge_bars * bar_minutes) - - qs = [q for q in quarter_starts(book_start, book_end) if q >= burn_end] - rows: List[Dict] = [] - per_q: List[Dict] = [] - coef_rows: List[Dict] = [] - skipped: List[str] = [] - dropped_seen: set = set() - - for q_start in qs: - q_end = q_start + pd.DateOffset(months=3) - train_mask = df["entry_time"] < (q_start - purge) - test_mask = (df["entry_time"] >= q_start) & (df["entry_time"] < q_end) - n_tr, n_te = int(train_mask.sum()), int(test_mask.sum()) - if n_tr < MIN_TRAIN or n_te == 0: - skipped.append(f"{quarter_label(q_start)} (train={n_tr}, test={n_te})") - continue - - tr = np.flatnonzero(train_mask.to_numpy()) - te = np.flatnonzero(test_mask.to_numpy()) - Xtr_raw, Xte_raw = X_all[tr], X_all[te] - mean, scale, keep = fit_standardiser(Xtr_raw) - dropped = [features[i] for i in np.flatnonzero(~keep)] - if verbose and dropped: - tag = (quarter_label(q_start), tuple(dropped)) - if tag not in dropped_seen: - dropped_seen.add(tag) - say(f" WARNING {quarter_label(q_start)}: zero variance in train, " - f"dropped from fit (coef 0): {', '.join(dropped)}") - Xtr = apply_standardiser(Xtr_raw, mean, scale)[:, keep] - Xte = apply_standardiser(Xte_raw, mean, scale)[:, keep] - - r_tr = r_train_src[tr] - lam: Optional[float] = None - if model == "logistic": - y_tr = (r_tr > 0).astype(float) - lam, _ = choose_lambda(Xtr, y_tr, "logistic") - coef, b, _ = fit_logistic(Xtr, y_tr, lam) - z_tr = sigmoid(predict_linear(Xtr, coef, b)) - z_te = sigmoid(predict_linear(Xte, coef, b)) - elif model == "lgbm": - y_tr = np.clip(r_tr, WINSOR_LO, WINSOR_HI) - gbm = fit_lgbm(Xtr, y_tr, seed) - z_tr = np.asarray(gbm.predict(Xtr), dtype=float) - z_te = np.asarray(gbm.predict(Xte), dtype=float) - coef, b = np.zeros(int(keep.sum())), 0.0 - else: # ridge / isbuy / exhaustion - y_tr = np.clip(r_tr, WINSOR_LO, WINSOR_HI) - lam, _ = choose_lambda(Xtr, y_tr, "ridge") - coef, b = fit_ridge(Xtr, y_tr, lam) - z_tr = predict_linear(Xtr, coef, b) - z_te = predict_linear(Xte, coef, b) - - cal = calibrate_map(z_tr) - m_te = apply_map(z_te, cal) - - if model != "lgbm": - full_coef = np.zeros(len(features)) - full_coef[np.flatnonzero(keep)] = coef - for name, c in zip(features, full_coef): - coef_rows.append({"quarter": quarter_label(q_start), "feature": name, - "coef_std": float(c), "lambda": lam, - "dropped": name in dropped}) - coef_rows.append({"quarter": quarter_label(q_start), "feature": "_intercept", - "coef_std": float(b), "lambda": lam, "dropped": False}) - - r_te = r_all[te] - for i, idx in enumerate(te): - rows.append({ - "entry_time": df["entry_time"].iloc[idx], - "r_multiple": float(r_te[i]), - "mult": float(m_te[i]), - "r_sized": float(r_te[i] * m_te[i]), - "quarter": quarter_label(q_start), - "z": float(z_te[i]), - }) - per_q.append({ - "quarter": quarter_label(q_start), - "q_start": q_start, "n_train": n_tr, "n_test": n_te, - "flat_r": float(r_te.sum()), "sized_r": float((r_te * m_te).sum()), - "mean_m": float(m_te.mean()), "lambda": lam, - "map_lo": cal["lo"], "map_hi": cal["hi"], "normalizer": cal["normalizer"], - }) - - oos = pd.DataFrame(rows) - return {"oos": oos, "quarters": pd.DataFrame(per_q), - "coefs": pd.DataFrame(coef_rows), "skipped": skipped, - "quarter_starts": qs} - - -def evaluate(oos: pd.DataFrame) -> Dict: - """Headline OOS numbers for one model's walk-forward output.""" - if oos.empty: - return {"n": 0, "flat_r": 0.0, "sized_r": 0.0, "lift": 0.0} - r = oos["r_multiple"].to_numpy(float) - rs = oos["r_sized"].to_numpy(float) - m = oos["mult"].to_numpy(float) - med = oos["entry_time"].median() - first = (oos["entry_time"] < med).to_numpy() - second = ~first - return { - "n": int(len(oos)), - "flat_r": float(r.sum()), "sized_r": float(rs.sum()), - "lift": float(rs.sum() - r.sum()), - "flat_pf": profit_factor(r), "sized_pf": profit_factor(rs), - "flat_dd": max_drawdown(r), "sized_dd": max_drawdown(rs), - "mean_m": float(m.mean()), "min_m": float(m.min()), "max_m": float(m.max()), - "median_time": med, - "n_h1": int(first.sum()), "n_h2": int(second.sum()), - "flat_h1": float(r[first].sum()), "flat_h2": float(r[second].sum()), - "sized_h1": float(rs[first].sum()), "sized_h2": float(rs[second].sum()), - } - - -def acceptance(ev: Dict, real_lift: float, shuffle_p95: float) -> Dict: - """The six pre-registered clauses.""" - c1 = ev["sized_h1"] > ev["flat_h1"] and ev["sized_h2"] > ev["flat_h2"] - c2 = ev["sized_r"] > ev["flat_r"] - c3 = ev["sized_dd"] <= DD_TOL * ev["flat_dd"] - c4 = (ev["sized_pf"] >= PF_FLOOR) and ev["sized_h1"] > 0 and ev["sized_h2"] > 0 - c5 = real_lift > shuffle_p95 if np.isfinite(shuffle_p95) else False - c6 = (abs(ev["mean_m"] - 1.0) <= MEAN_MULT_TOL - and ev["min_m"] >= MAP_CLIP_LO - 1e-9 and ev["max_m"] <= MAP_CLIP_HI + 1e-9) - clauses = {"1_both_halves": bool(c1), "2_overall": bool(c2), "3_drawdown": bool(c3), - "4_pf_and_halves": bool(c4), "5_beats_shuffle": bool(c5), - "6_multiplier_bounds": bool(c6)} - clauses["overall"] = bool(all(clauses.values())) - return clauses - - -def shuffle_null(df: pd.DataFrame, features: Sequence[str], model: str, - bar_minutes: int, burn_in_months: int, purge_bars: int, - n_shuffle: int, seed: int) -> np.ndarray: - """Lift distribution under permuted TRAINING labels (test labels real).""" - rng = np.random.default_rng(seed) - r_all = df["r_multiple"].to_numpy(float) - lifts: List[float] = [] - for _ in range(n_shuffle): - r_perm = r_all[rng.permutation(len(r_all))] - res = walk_forward(df, features, model, bar_minutes, burn_in_months, - purge_bars, seed, y_train_override=r_perm, verbose=False) - ev = evaluate(res["oos"]) - lifts.append(ev["lift"]) - return np.asarray(lifts, dtype=float) - - -# ══════════════════════════════════════════════════════════════════════ -# Final fit + artifact -# ══════════════════════════════════════════════════════════════════════ -def final_fit(df: pd.DataFrame, features: Sequence[str], model: str, - seed: int) -> Dict: - """Fit on ALL trades; calibrate the map on in-sample predictions.""" - X_raw = df[list(features)].apply(pd.to_numeric, errors="coerce").to_numpy(float) - r = df["r_multiple"].to_numpy(float) - mean, scale, keep = fit_standardiser(X_raw) - X = apply_standardiser(X_raw, mean, scale)[:, keep] - if model == "logistic": - y = (r > 0).astype(float) - lam, _ = choose_lambda(X, y, "logistic") - coef, b, gnorm = fit_logistic(X, y, lam) - z = sigmoid(predict_linear(X, coef, b)) - target = "r_positive" - else: - y = np.clip(r, WINSOR_LO, WINSOR_HI) - lam, _ = choose_lambda(X, y, "ridge") - coef, b = fit_ridge(X, y, lam) - gnorm = float("nan") - z = predict_linear(X, coef, b) - target = "r_winsorised" - full_coef = np.zeros(len(features)) - full_coef[np.flatnonzero(keep)] = coef - cal = calibrate_map(z) - return {"mean": mean, "scale": scale, "keep": keep, "coef": full_coef, - "intercept": float(b), "lambda": float(lam), "z": z, - "mult": apply_map(z, cal), "map": cal, "target": target, - "grad_norm": gnorm} - - -def predict_multiplier(feature_dict: Dict[str, float], artifact: Dict) -> float: - """Runtime formula: standardise → linear score → size map → 0.05 grid.""" - names = artifact["features"] - mean = artifact["mean"] - scale = artifact["scale"] - coef = artifact["coef"] - z = float(artifact["intercept"]) - for i, name in enumerate(names): - raw = feature_dict.get(name, mean[i]) - if raw is None or not np.isfinite(float(raw)): - raw = mean[i] - z += float(coef[i]) * (float(raw) - float(mean[i])) / float(scale[i]) - if artifact["model"] == "logistic": - z = 1.0 / (1.0 + math.exp(-max(-35.0, min(35.0, z)))) - mp = artifact["map"] - if mp["hi"] <= mp["lo"]: - return 1.0 - frac = min(1.0, max(0.0, (z - mp["lo"]) / (mp["hi"] - mp["lo"]))) - raw_m = mp["out_lo"] + (mp["out_hi"] - mp["out_lo"]) * frac - val = min(mp["clip_hi"], max(mp["clip_lo"], raw_m * mp["normalizer"])) - step = mp["round_to"] - return float(round(round(val / step) * step, 2)) - - -def build_artifact(fit: Dict, features: Sequence[str], model: str, df: pd.DataFrame, - ev: Dict, clauses: Dict, real_lift: float, shuffle: np.ndarray, - n_shuffle: int, book: Path, tf: str, confirm_tfs: List[str], - burn_in_months: int, purge_bars: int, spread: float) -> Dict: - return { - "schema_version": 1, - "feature_version": FEATURE_VERSION, - "pair": "XAU/USD", - "primary_tf": tf, - "confirmation_tfs": confirm_tfs, - "model": model, - "target": fit["target"], - "features": list(features), - "mean": [float(v) for v in fit["mean"]], - "scale": [float(v) for v in fit["scale"]], - "coef": [float(v) for v in fit["coef"]], - "intercept": float(fit["intercept"]), - "map": { - "kind": "clipped_linear_on_quantiles", - "lo": float(fit["map"]["lo"]), "hi": float(fit["map"]["hi"]), - "out_lo": MAP_OUT_LO, "out_hi": MAP_OUT_HI, - "normalizer": float(fit["map"]["normalizer"]), - "clip_lo": MAP_CLIP_LO, "clip_hi": MAP_CLIP_HI, - "round_to": MAP_ROUND_TO, - }, - "training": { - "window": [df["entry_time"].iloc[0].isoformat(), - df["entry_time"].iloc[-1].isoformat()], - "spread": spread, - "n_train_trades": int(len(df)), - "n_oos_trades": int(ev.get("n", 0)), - "burn_in_months": burn_in_months, - "purge_bars": purge_bars, - "refit": "calendar_quarter", - "final_fit": "full_window", - "lambda": float(fit["lambda"]), - "git_commit": git_commit(), - "book_sha256": sha256_file(book), - }, - "acceptance": { - **{k: v for k, v in clauses.items()}, - "flat_r": ev.get("flat_r"), "sized_r": ev.get("sized_r"), - "sized_pf": ev.get("sized_pf"), "flat_dd": ev.get("flat_dd"), - "sized_dd": ev.get("sized_dd"), - "sized_h1": ev.get("sized_h1"), "sized_h2": ev.get("sized_h2"), - "flat_h1": ev.get("flat_h1"), "flat_h2": ev.get("flat_h2"), - "real_lift": float(real_lift), - "shuffle_p95_lift": float(np.percentile(shuffle, 95)) if len(shuffle) else None, - "n_shuffle": int(n_shuffle), - "mean_multiplier": ev.get("mean_m"), - }, - } - - -# ══════════════════════════════════════════════════════════════════════ -# Reporting -# ══════════════════════════════════════════════════════════════════════ -def fmt(x, nd=2, width=0) -> str: - if x is None: - return "n/a".rjust(width) - if isinstance(x, float) and not np.isfinite(x): - return ("inf" if x > 0 else "-inf").rjust(width) - s = f"{x:.{nd}f}" if isinstance(x, (int, float, np.floating)) else str(x) - return s.rjust(width) if width else s - - -def print_quarters(q: pd.DataFrame) -> None: - say(f"{'quarter':<9}{'n_tr':>6}{'n_te':>6}{'flat R':>10}{'sized R':>10}" - f"{'mean m':>9}{'lambda':>9}") - for _, row in q.iterrows(): - say(f"{row['quarter']:<9}{int(row['n_train']):>6}{int(row['n_test']):>6}" - f"{fmt(row['flat_r'], 2, 10)}{fmt(row['sized_r'], 2, 10)}" - f"{fmt(row['mean_m'], 3, 9)}" - f"{('n/a' if row['lambda'] is None else fmt(row['lambda'], 0)):>9}") - - -def print_hist(m: np.ndarray) -> None: - h = multiplier_hist(m) - say(f"{'bin':<12}{'count':>8}{'pct':>8}") - for _, row in h.iterrows(): - bar = "#" * int(round(row["pct"] / 2)) - say(f"{row['bin_lo']:.1f}–{row['bin_hi']:.1f}{'':<4}{int(row['count']):>8}" - f"{row['pct']:>7.1f}% {bar}") - - -def print_coef_signs(coefs: pd.DataFrame, label: str) -> None: - """Mean standardised coefficient + sign stability across quarters.""" - if coefs.empty: - return - d = coefs[coefs["feature"] != "_intercept"] - say(f" standardised coefficients across quarters ({label}):") - say(f" {'feature':<24}{'mean':>10}{'min':>10}{'max':>10}{'+q':>5}{'-q':>5} sign-stable") - for name, grp in d.groupby("feature", sort=False): - c = grp["coef_std"].to_numpy(float) - pos, neg = int((c > 0).sum()), int((c < 0).sum()) - stable = "yes" if (pos == 0 or neg == 0) and (pos + neg) > 0 else "no" - say(f" {name:<24}{c.mean():>10.4f}{c.min():>10.4f}{c.max():>10.4f}" - f"{pos:>5}{neg:>5} {stable}") - - -def print_comparison(table: List[Dict]) -> None: - say(f"{'model':<12}{'n':>6}{'flat R':>10}{'sized R':>10}{'lift':>9}" - f"{'PF':>7}{'DD':>8}{'mean m':>8}{'p95 null':>10}{'rank%':>7} pass") - for row in table: - say(f"{row['model']:<12}{row['n']:>6}{fmt(row['flat_r'], 2, 10)}" - f"{fmt(row['sized_r'], 2, 10)}{fmt(row['lift'], 2, 9)}" - f"{fmt(row['sized_pf'], 2, 7)}{fmt(row['sized_dd'], 2, 8)}" - f"{fmt(row['mean_m'], 3, 8)}{fmt(row['shuffle_p95'], 2, 10)}" - f"{fmt(row['rank_pct'], 0, 7)} {'PASS' if row['pass'] else 'FAIL'}") - - -# ══════════════════════════════════════════════════════════════════════ -# main -# ══════════════════════════════════════════════════════════════════════ -def main() -> None: - ap = argparse.ArgumentParser(description=__doc__, - formatter_class=argparse.RawDescriptionHelpFormatter) - ap.add_argument("--book", required=True, type=Path) - ap.add_argument("--bar-minutes", required=True, type=int) - ap.add_argument("--features", required=True, - help="comma-separated f_* column names for the main model") - ap.add_argument("--model", default="ridge", choices=("ridge", "logistic")) - ap.add_argument("--burn-in-months", type=int, default=12) - ap.add_argument("--purge-bars", type=int, default=300) - ap.add_argument("--n-shuffle", type=int, default=50) - ap.add_argument("--seed", type=int, default=20260905) - ap.add_argument("--skip-lgbm", action="store_true") - ap.add_argument("--out", type=Path, default=None) - ap.add_argument("--write-artifact", type=Path, default=None) - ap.add_argument("--tf", default=None, help="primary TF label for the artifact") - ap.add_argument("--confirm-tfs", default="15min,1h") - ap.add_argument("--spread", type=float, default=0.65, - help="cost the book was produced at (recorded in the artifact)") - args = ap.parse_args() - - if args.write_artifact and not args.tf: - raise SystemExit("--write-artifact requires --tf") - - say("=" * 96) - say("Stage-D stake-sizing walk-forward — RESEARCH ONLY") - say("=" * 96) - say(f"book : {args.book}") - df = load_book(args.book) - say(f"trades : {len(df)} " - f"({df['entry_time'].iloc[0]} → {df['entry_time'].iloc[-1]})") - say(f"bar minutes : {args.bar_minutes} purge = {args.purge_bars} bars = " - f"{args.purge_bars * args.bar_minutes} min " - f"({args.purge_bars * args.bar_minutes / 1440:.2f} d)") - say(f"burn-in : {args.burn_in_months} months " - f"(train-only until {df['entry_time'].iloc[0] + pd.DateOffset(months=args.burn_in_months)})") - say(f"model / seed : {args.model} / {args.seed} shuffles: {args.n_shuffle}") - say(f"winsor / lambda : [{WINSOR_LO}, {WINSOR_HI}] / {list(LAMBDA_GRID)} " - f"CV folds {CV_FOLDS} (time-contiguous)") - say(f"map : m = round_{MAP_ROUND_TO}(clip(norm × (0.5 + clip((z−lo)/(hi−lo),0,1)" - f"), {MAP_CLIP_LO}, {MAP_CLIP_HI})) lo/hi = train q{MAP_Q_LO}/q{MAP_Q_HI}") - say("") - - main_feats = resolve_features(df, [f.strip() for f in args.features.split(",") if f.strip()], - "--features") - say(f"main features ({len(main_feats)}): {', '.join(main_feats)}") - - specs: List[Tuple[str, List[str], str]] = [(args.model, main_feats, args.model)] - for label, feats in (("isbuy", ISBUY_FEATURES), ("exhaustion", EXHAUSTION_FEATURES)): - have = [f for f in feats if f in df.columns] - if len(have) != len(feats): - say(f"NOTE: {label} baseline missing " - f"{', '.join(f for f in feats if f not in df.columns)} — using {len(have)}") - if have: - specs.append((label, have, "ridge")) - use_lgbm = (not args.skip_lgbm) and lgbm_available() - if not args.skip_lgbm and not use_lgbm: - say("lightgbm not installed — upper bound skipped") - if use_lgbm: - all_f = [c for c in df.columns if c.startswith("f_")] - specs.append(("lgbm", all_f, "lgbm")) - say(f"lightgbm upper bound: {len(all_f)} f_* columns") - say("") - - results: Dict[str, Dict] = {} - table: List[Dict] = [] - for label, feats, kind in specs: - say("-" * 96) - say(f"MODEL: {label} ({kind}, {len(feats)} feature(s))") - say("-" * 96) - res = walk_forward(df, feats, kind, args.bar_minutes, args.burn_in_months, - args.purge_bars, args.seed, verbose=True) - if res["skipped"]: - say(f" skipped quarters: {'; '.join(res['skipped'])}") - if res["oos"].empty: - say(" no OOS quarters — nothing to evaluate") - continue - ev = evaluate(res["oos"]) - n_sh = args.n_shuffle if kind != "lgbm" else min(args.n_shuffle, LGBM_MAX_SHUFFLE) - if kind == "lgbm" and n_sh != args.n_shuffle: - say(f" NOTE (protocol deviation, flagged): lgbm shuffle null capped at " - f"{n_sh} repeats (never ships)") - sh = shuffle_null(df, feats, kind, args.bar_minutes, args.burn_in_months, - args.purge_bars, n_sh, args.seed) if n_sh > 0 else np.array([]) - p95 = float(np.percentile(sh, 95)) if len(sh) else float("nan") - rank = float((sh < ev["lift"]).mean() * 100) if len(sh) else float("nan") - cl = acceptance(ev, ev["lift"], p95) - - say(f" OOS quarters : {len(res['quarters'])} " - f"({res['quarters']['quarter'].iloc[0]} → {res['quarters']['quarter'].iloc[-1]}), " - f"{ev['n']} trades") - say(f" flat R : {ev['flat_r']:+.2f} PF {fmt(ev['flat_pf'])} " - f"maxDD {ev['flat_dd']:.2f}") - say(f" sized R : {ev['sized_r']:+.2f} PF {fmt(ev['sized_pf'])} " - f"maxDD {ev['sized_dd']:.2f}") - say(f" lift : {ev['lift']:+.2f} R") - say(f" halves (split {ev['median_time']}): " - f"h1 flat {ev['flat_h1']:+.2f} / sized {ev['sized_h1']:+.2f} (n={ev['n_h1']}) | " - f"h2 flat {ev['flat_h2']:+.2f} / sized {ev['sized_h2']:+.2f} (n={ev['n_h2']})") - say(f" multiplier : mean {ev['mean_m']:.3f} min {ev['min_m']:.2f} " - f"max {ev['max_m']:.2f}") - say(f" shuffle null : n={len(sh)} mean {fmt(float(sh.mean()) if len(sh) else None)} " - f"p95 {fmt(p95)} real-lift percentile rank {fmt(rank, 0)}%") - say(" acceptance:") - for k, v in cl.items(): - if k == "overall": - continue - say(f" {k:<22} {'PASS' if v else 'FAIL'}") - say(f" {'OVERALL':<22} {'PASS' if cl['overall'] else 'FAIL'}") - say(" per-quarter:") - print_quarters(res["quarters"]) - print_coef_signs(res["coefs"], label) - say("") - - results[label] = {"res": res, "ev": ev, "shuffle": sh, "p95": p95, - "rank": rank, "clauses": cl, "features": feats, "kind": kind} - table.append({"model": label, "n": ev["n"], "flat_r": ev["flat_r"], - "sized_r": ev["sized_r"], "lift": ev["lift"], - "sized_pf": ev["sized_pf"], "sized_dd": ev["sized_dd"], - "mean_m": ev["mean_m"], "shuffle_p95": p95, "rank_pct": rank, - "pass": cl["overall"]}) - - if args.model not in results: - raise SystemExit("the main model produced no OOS quarters — check burn-in / " - "book window / MIN_TRAIN") - - say("=" * 96) - say("COMPARISON") - say("=" * 96) - print_comparison(table) - say("") - say(f"OOS multiplier histogram ({args.model}):") - print_hist(results[args.model]["res"]["oos"]["mult"].to_numpy()) - say("") - - main_label = args.model - beats = None - if main_label in results and "isbuy" in results: - beats = results[main_label]["ev"]["lift"] > results["isbuy"]["ev"]["lift"] - say(f"beats isbuy baseline: {'yes' if beats else 'no'} " - f"({results[main_label]['ev']['lift']:+.2f} R vs " - f"{results['isbuy']['ev']['lift']:+.2f} R)") - say(f"OVERALL ACCEPTANCE ({main_label}): " - f"{'PASS' if results[main_label]['clauses']['overall'] else 'FAIL'}") - say("") - - # ── artifact ────────────────────────────────────────────────────── - artifact = None - if args.write_artifact: - say("-" * 96) - say("FINAL FIT (all trades) + ARTIFACT") - say("-" * 96) - fit = final_fit(df, main_feats, args.model, args.seed) - say(f" lambda (CV on full window): {fit['lambda']:g}") - if np.isfinite(fit["grad_norm"]): - say(f" logistic |grad|_inf at convergence: {fit['grad_norm']:.3e}") - say(f" map lo/hi/normaliser: {fit['map']['lo']:.5f} / {fit['map']['hi']:.5f} / " - f"{fit['map']['normalizer']:.5f}") - artifact = build_artifact( - fit, main_feats, args.model, df, results[main_label]["ev"], - results[main_label]["clauses"], results[main_label]["ev"]["lift"], - results[main_label]["shuffle"], args.n_shuffle, args.book, args.tf, - [t.strip() for t in args.confirm_tfs.split(",") if t.strip()], - args.burn_in_months, args.purge_bars, args.spread) - # Self-check: the runtime formula must reproduce the in-sample map exactly. - rows = df[main_feats].to_dict("records") - repro = np.array([predict_multiplier(r, artifact) for r in rows]) - exact = bool(np.array_equal(repro, fit["mult"])) - say(f" predict_multiplier reproduces in-sample multipliers exactly: " - f"{'YES' if exact else 'NO'} (max |diff| " - f"{float(np.max(np.abs(repro - fit['mult']))):.3e})") - if not exact: - raise SystemExit("predict_multiplier does not reproduce the final fit — abort") - args.write_artifact.parent.mkdir(parents=True, exist_ok=True) - args.write_artifact.write_text(json.dumps(artifact, indent=2, default=str)) - say(f" wrote {args.write_artifact}") - say("") - - # ── outputs ─────────────────────────────────────────────────────── - if args.out: - out = args.out - out.mkdir(parents=True, exist_ok=True) - main_res = results[main_label] - main_res["res"]["oos"].to_csv(out / "oos_trades.csv", index=False) - main_res["res"]["quarters"].to_csv(out / "quarters.csv", index=False) - coefs = pd.concat( - [r["res"]["coefs"].assign(model=lbl) for lbl, r in results.items() - if not r["res"]["coefs"].empty], - ignore_index=True) if results else pd.DataFrame() - coefs.to_csv(out / "coefs_by_quarter.csv", index=False) - pd.DataFrame({ - "model": np.repeat([lbl for lbl in results], [len(r["shuffle"]) for r in results.values()]), - "lift": np.concatenate([r["shuffle"] for r in results.values()]) - if results else np.array([]), - }).to_csv(out / "shuffle_lifts.csv", index=False) - multiplier_hist(main_res["res"]["oos"]["mult"].to_numpy()).to_csv( - out / "multiplier_hist.csv", index=False) - payload = { - "book": str(args.book), "book_sha256": sha256_file(args.book), - "git_commit": git_commit(), "bar_minutes": args.bar_minutes, - "model": args.model, "features": main_feats, "seed": args.seed, - "n_shuffle": args.n_shuffle, "burn_in_months": args.burn_in_months, - "purge_bars": args.purge_bars, - "beats_isbuy": beats, - "models": { - lbl: { - "kind": r["kind"], "features": r["features"], - "evaluation": {k: (None if isinstance(v, float) and not np.isfinite(v) - else (str(v) if isinstance(v, pd.Timestamp) else v)) - for k, v in r["ev"].items()}, - "shuffle": {"n": int(len(r["shuffle"])), - "mean": float(r["shuffle"].mean()) if len(r["shuffle"]) else None, - "p95": None if not np.isfinite(r["p95"]) else r["p95"], - "real_lift_percentile": None if not np.isfinite(r["rank"]) - else r["rank"]}, - "acceptance": r["clauses"], - } for lbl, r in results.items() - }, - "artifact": artifact, - } - (out / "results.json").write_text(json.dumps(payload, indent=2, default=str)) - (out / "summary.txt").write_text("\n".join(_LINES) + "\n") - print(f"\nwrote outputs to {out}") - - -if __name__ == "__main__": - main() diff --git a/backtest/research/watch_rule_eval.py b/backtest/research/watch_rule_eval.py deleted file mode 100644 index 67f99fe..0000000 --- a/backtest/research/watch_rule_eval.py +++ /dev/null @@ -1,78 +0,0 @@ -"""Evaluate the violent-move WATCH rule (as shipped in violent_watch.py) bar by bar on history. Analysis tooling only. -Level per bar from: calendar (high-impact USD within 30 min ahead / 30 min after), clock slots, indicator readings. -Reports share of bars per level, burst rate next bar / next hour per level (both halves), and hot-hours per day. - python backtest/research/watch_rule_eval.py PARQUET BAR_MINUTES -""" -import sys -from pathlib import Path -import numpy as np, pandas as pd -ROOT = Path(__file__).resolve().parents[2]; sys.path.insert(0, str(ROOT)); sys.path.insert(0, str(ROOT / "backtest")) -from gpu_backtest import precompute_indicators -from config import TechnicalConfig - - -def wilder_atr(df, period=14): - pc = df["Close"].shift(1) - tr = pd.concat([df["High"] - df["Low"], (df["High"] - pc).abs(), (df["Low"] - pc).abs()], axis=1).max(axis=1) - return tr.ewm(alpha=1 / period, min_periods=period, adjust=False).mean() - - -def main(): - pq, bm = sys.argv[1], int(sys.argv[2]) - px = pd.read_parquet(pq); px = px[(px.index >= pd.Timestamp("2023-08-05", tz="UTC")) & (px.index < pd.Timestamp("2026-09-05", tz="UTC"))] - ic = precompute_indicators(px, TechnicalConfig()) - sh = lambda a: pd.Series(np.asarray(a, dtype=float), index=px.index).shift(1) - rsi, adxs, hist, bbp, kcu, kcl = sh(ic.rsi), sh(ic.adx_slope), sh(ic.macd_hist), sh(ic.bb_position), sh(ic.kc_upper), sh(ic.kc_lower) - ef, em, es, et, adx, atr1 = sh(ic.ema_fast), sh(ic.ema_medium), sh(ic.ema_slow), sh(ic.ema_trend), sh(ic.adx), sh(ic.atr) - c1 = px["Close"].shift(1); hp = hist.shift(1) - atr_prev = wilder_atr(px).shift(1); rng = px["High"] - px["Low"] - burst = (rng >= 3.0 * atr_prev).astype(int); H = max(1, 60 // bm) - next_hour = burst[::-1].rolling(H, min_periods=1).max()[::-1].astype(int) - align = np.sign(c1 - ef) + np.sign(ef - em) + np.sign(em - es) + np.sign(es - et) - regime = np.select([(adx >= 25) & (align > 0), (adx >= 25) & (align < 0), adx >= 25, adxs > 3.0], ["trending_up", "trending_down", "trending", "transitioning"], default="ranging") - atr_pct = atr1.rolling(50).apply(lambda w: (w[-1] > w).mean() * 100, raw=True); extreme = atr_pct > 80 - # readings - bear = ((bbp < 0).astype(int) + (rsi < 30).astype(int) + (c1 < kcl).astype(int) + ((hist < 0) & (hist < hp) & (hp < 0)).astype(int)) - bull = ((bbp > 1).astype(int) + (rsi >= 70).astype(int) + (c1 > kcu).astype(int) + ((hist > 0) & (hist > hp) & (hp > 0)).astype(int)) - neutral = (adxs > 3).astype(int) + extreme.astype(int) + (regime == "transitioning").astype(int) - readings = bear + bull + neutral - # clock (now = bar close time) - ct = px.index + pd.Timedelta(minutes=bm); wd = ct.weekday; hr = ct.hour - clock = ((wd <= 4) & np.isin(hr, [12, 13, 14])) | (hr == 1) | ((wd == 6) & np.isin(hr, [22, 23])) - # calendar - cal = pd.read_parquet(ROOT / "backtest/cache/calendar.parquet"); cal["ts"] = pd.to_datetime(cal.dateline, unit="s", utc=True) - hi = np.sort(cal[(cal.currency == "USD") & (cal.impactName.str.lower() == "high")].ts.values.astype("datetime64[ns]")) - t = ct.values.astype("datetime64[ns]"); nx = np.searchsorted(hi, t) - to_next = np.where(nx < len(hi), (hi[np.minimum(nx, len(hi) - 1)] - t) / np.timedelta64(1, "m"), 1e9) - since = np.where(nx - 1 >= 0, (t - hi[np.maximum(nx - 1, 0)]) / np.timedelta64(1, "m"), 1e9) - calendar = ((to_next > 0) & (to_next <= 30)) | (since <= 30) - # FINAL RULE (2026-09-06, chosen from the variant sweep recorded in sizing_campaign_2026-09.md): - # extreme = price-at-extreme readings only (outside Bollinger, RSI beyond 30/70, outside Keltner) - # hot = calendar | (clock & extreme >= 1) -> ~8.6% of bars, next-bar odds 3-5x normal, ~4 hot hours/day - # watch = clock | extreme >= 2 -> ~2x normal - extreme = ((bbp < 0) | (bbp > 1)).astype(int) + ((rsi < 30) | (rsi >= 70)).astype(int) + ((c1 < kcl) | (c1 > kcu)).astype(int) - level = np.where(calendar | (clock & (extreme >= 1)), "hot", np.where(clock | (extreme >= 2), "watch", "none")) - d = pd.DataFrame({"level": level, "burst": burst.values, "next_hour": next_hour.values, "readings": readings.values, "clock": clock, "calendar": calendar}, index=px.index) - d = d[d.index >= pd.Timestamp("2023-09-05", tz="UTC")] - half = np.where(np.arange(len(d)) < len(d) // 2, 1, 2) - b1, b2 = d.burst[half == 1].mean(), d.burst[half == 2].mean(); h1, h2 = d.next_hour[half == 1].mean(), d.next_hour[half == 2].mean() - print(f"{Path(pq).name}: {len(d)} bars; base next-bar {100*d.burst.mean():.2f}%, next-hour {100*d.next_hour.mean():.1f}%") - rows = [] - for lv in ("none", "watch", "hot"): - m = d.level == lv - rows.append(dict(level=lv, share_of_bars=100 * m.mean(), next_bar_pct=100 * d.burst[m].mean(), lift_h1=d.burst[m & (half == 1)].mean() / b1, lift_h2=d.burst[m & (half == 2)].mean() / b2, - next_hour_pct=100 * d.next_hour[m].mean(), hour_lift_h1=d.next_hour[m & (half == 1)].mean() / h1, hour_lift_h2=d.next_hour[m & (half == 2)].mean() / h2, - share_of_all_bursts=100 * d.burst[m].sum() / d.burst.sum())) - print(pd.DataFrame(rows).round(2).to_string(index=False)) - hot = d[d.level == "hot"] - kinds = pd.Series(np.where(hot.calendar, "calendar", "clock+extreme reading"), index=hot.index) - print("hot bars by cause:", (100 * kinds.value_counts(normalize=True)).round(1).to_dict()) - hot_hours = hot.index.floor("h").unique() - days = pd.Series(hot_hours.date).value_counts() - print(f"distinct HOT hours per calendar day: mean {days.mean():.1f}, median {days.median():.0f}, max {days.max()} (≈ heads-up messages/day before the daily cap)") - hb = d.burst.groupby(d.index.floor("h")).max(); hh = pd.Series(1, index=hot_hours) - print(f"of all hours with a violent bar, the hour was flagged HOT at its start or during: {100*hb[hb==1].index.isin(hot_hours).mean():.1f}%") - - -if __name__ == "__main__": - main() diff --git a/docs/architecture.png b/docs/architecture.png deleted file mode 100644 index 9a70cb0..0000000 Binary files a/docs/architecture.png and /dev/null differ diff --git a/docs/architecture_diagram.py b/docs/architecture_diagram.py deleted file mode 100644 index 1bf6648..0000000 --- a/docs/architecture_diagram.py +++ /dev/null @@ -1,191 +0,0 @@ -"""ForexBot2 architecture diagram → docs/architecture.png. - -Regenerate after architecture changes: venv/bin/python docs/architecture_diagram.py -(matplotlib only — no graphviz on the dev box).""" -import os -import matplotlib -matplotlib.use("Agg") -import matplotlib.pyplot as plt -from matplotlib.patches import FancyBboxPatch, FancyArrowPatch - -W, H = 100.0, 100.0 -fig, ax = plt.subplots(figsize=(16, 11.5)) -ax.set_xlim(0, W); ax.set_ylim(0, H); ax.axis("off") - -INK = "#263238"; MUT = "#607D8B"; ARR = "#455A64"; HAUL = "#7B8FA1" - -def box(x, y, w, h, title, lines, fc, ec, title_fs=8.2, body_fs=6.6, lw=1.3): - ax.add_patch(FancyBboxPatch((x, y), w, h, - boxstyle="round,pad=0.35,rounding_size=0.9", - facecolor=fc, edgecolor=ec, linewidth=lw, zorder=2)) - cy = y + h - 1.9 - ax.text(x + w / 2, cy, title, ha="center", va="center", fontsize=title_fs, - fontweight="bold", color=INK, zorder=3) - cy -= 2.3 - for ln in lines: - ax.text(x + w / 2, cy, ln, ha="center", va="center", fontsize=body_fs, - color=INK, zorder=3) - cy -= 1.85 - -def band(x, y, w, h, label, color): - ax.add_patch(FancyBboxPatch((x, y), w, h, - boxstyle="round,pad=0.3,rounding_size=1.2", - facecolor="none", edgecolor=color, linewidth=1.0, - linestyle=(0, (4, 3)), zorder=1)) - ax.text(x + 0.9, y + h - 1.4, label, ha="left", va="center", - fontsize=7.2, fontweight="bold", color=color, zorder=3) - -def arrow(p1, p2, color=ARR, lw=1.5, style="-|>", conn="arc3,rad=0.0", - label=None, lx=0, ly=0, ls="solid", fs=6.4, rot=0): - ax.add_patch(FancyArrowPatch(p1, p2, arrowstyle=style, mutation_scale=11, - color=color, linewidth=lw, linestyle=ls, - connectionstyle=conn, zorder=4)) - if label: - mx, my = (p1[0] + p2[0]) / 2 + lx, (p1[1] + p2[1]) / 2 + ly - ax.text(mx, my, label, fontsize=fs, color=MUT, ha="center", - va="center", zorder=5, rotation=rot, - bbox=dict(boxstyle="round,pad=0.15", fc="white", ec="none", alpha=0.9)) - -# ── title ──────────────────────────────────────────────────────────── -ax.text(2, 98.4, "ForexBot2 — XAU/USD Signal-Intelligence Platform", - fontsize=15, fontweight="bold", color=INK, ha="left") -ax.text(2, 96.0, "one pipeline, four delivery surfaces · closed bars only · live == backtest parity", - fontsize=8.5, color=MUT, ha="left") - -EXT_FC, EXT_EC = "#ECEFF1", "#90A4AE" -DAT_FC, DAT_EC = "#E3F2FD", "#5C9BD1" -NWS_FC, NWS_EC = "#E0F2F1", "#3D9C8F" -COR_FC, COR_EC = "#FFFDF5", "#C9A227" -REP_FC, REP_EC = "#FFE9A8", "#B8860B" -DEL_FC, DEL_EC = "#E8F5E9", "#5FAF63" -BTS_FC, BTS_EC = "#EDE7F6", "#8E6BC6" - -# ── external services (top band) ───────────────────────────────────── -band(1, 85.0, 98, 9.4, "EXTERNAL SERVICES", "#78909C") -box(2.5, 86.2, 15.5, 6.4, "Oanda v20 API", - ["candles (live feed)", "practice orders"], EXT_FC, EXT_EC) -box(19.5, 86.2, 15.5, 6.4, "Twelve Data · Tiingo", - ["registered fallbacks", "(Tiingo: tick stream)"], EXT_FC, EXT_EC) -box(36.5, 86.2, 15.5, 6.4, "Forex Factory", - ["economic calendar", "MyFXBook RSS fallback"], EXT_FC, EXT_EC) -box(53.5, 86.2, 12.5, 6.4, "News RSS ×4", - ["gold / macro", "headlines"], EXT_FC, EXT_EC) -box(67.5, 86.2, 13.5, 6.4, "DeepSeek API", - ["V4-Flash LLM", "thinking: disabled"], EXT_FC, EXT_EC) -box(82.5, 86.2, 15, 6.4, "Telegram API", - ["owner bot +", "subscriber channel"], EXT_FC, EXT_EC) - -# ── data & context layer ───────────────────────────────────────────── -box(2.5, 69.8, 33, 11.0, "providers/ (BaseDataProvider)", - ["single-flight fetch · TTL+boundary cache", - "forming-candle DROP · validate/clean", - "paginated fetch_range · optional tick stream", - "active feed knob: data_providers.active = oanda"], - DAT_FC, DAT_EC) -box(38.5, 69.8, 33, 11.0, "news_engine.py + llm_client.py", - ["calendar events + keyword sentiment (gold-inverse)", - "DeepSeek enrichment: batch headline scores,", - "event gold-direction, market blurb — display-only,", - "keyword fallback, daily request cap"], - NWS_FC, NWS_EC) -box(74.5, 69.8, 23, 11.0, "app_config.py", - ["credentials & deployment:", - "env → config.json → refuse", - "(no hardcoded fallbacks)", - ".env loaded at import"], - "#FAFAFA", "#9E9E9E") - -# ── signal core container ──────────────────────────────────────────── -ax.add_patch(FancyBboxPatch((2.5, 40.5), 95, 26.5, - boxstyle="round,pad=0.35,rounding_size=1.2", - facecolor=COR_FC, edgecolor=COR_EC, linewidth=1.6, zorder=1)) -ax.text(50, 65.2, "SIGNAL CORE — runs identically in live scans and backtests (closed bars only)", - ha="center", fontsize=8.4, fontweight="bold", color="#8a6d1a", zorder=3) - -box(4.5, 51.8, 28, 12.0, "engine.py — TechnicalEngine", - ["9 scored indicators + ATR context:", - "EMA · RSI · MACD · BB · ADX · ROC ·", - "Keltner · divergence · candlestick", - "regime → adaptive weights → composite"], - "white", COR_EC) -box(4.5, 42.0, 28, 8.4, "pair_signals/xauusd.py — factory", - ["ALWAYS used: config.json overlay +", - "validated per-TF deltas (hour blocks, trail…)"], - "white", COR_EC) -box(35.5, 42.0, 60, 20.7, "signal_aggregator.py — analyze_pair → analyze_from_dataframes", - ["multi-TF agreement (M5→[M15,H1] · M15→[M5,H1] · H1→[M15,H4] · H4→[H1,D1])", - "→ news context (warn-only by default) → diminishing-decay penalties", - "→ exhaustion hard block → quality grade A+…F", - "→ ordered post-grade gate chain (session · XAU SELL vol/hour · regime · M5 noise)", - "→ setup generators: XAU ATR (A+/A) · generic swing · XAU_PB · XAU_CP", - "→ entry · SL · TP1/TP2 · trailing plan"], - "white", COR_EC, body_fs=6.9) - -# ── SignalReport hub ───────────────────────────────────────────────── -box(31, 33.2, 38, 5.2, "SignalReport", - ["signal (−2…+2) · grade · trade setup · reasons · news summary"], - REP_FC, REP_EC, title_fs=8.8) - -# ── delivery surfaces ──────────────────────────────────────────────── -band(1, 13.2, 98, 17.2, "DELIVERY SURFACES", "#4C8F50") -box(2.5, 15.0, 12, 12.6, "main.py", - ["CLI report", "--detail", "--json"], DEL_FC, DEL_EC) -box(42.5, 15.0, 31, 12.6, "dashboard_publisher.py → Apache", - ["independent daemon · scans at candle closes", - "atomic publish to /var/www/trade:", - "dashboard.html · snapshot.json · news.json", - "+ news_summary.php — on-demand AI", - "gold-impact summary (whitelist · cache · cap)"], - DEL_FC, DEL_EC) -box(76, 15.0, 21.5, 12.6, "telegram_bot.py", - ["candle-boundary auto-scans", "quality threshold + dedup", - "alerts + LLM context line", "WS level-breach monitor"], - DEL_FC, DEL_EC) - -# ── backtest / QA ──────────────────────────────────────────────────── -box(2.5, 3.2, 71.8, 8.0, "backtest/gpu_backtest.py — bar-by-bar replay through the SAME analyze_from_dataframes", - ["realism (results-critical): next-open entries · raw-path resolution · spread charged · SL-first ties · close-only trail", - "CuPy GPU / NumPy fallback · parquet cache · fed by providers.fetch_range (news off)"], - BTS_FC, BTS_EC) -box(77.5, 3.2, 20, 8.0, "test_all.py", - ["461 tests · 31 modules", "no network (mocked)"], "#FAFAFA", "#9E9E9E") - -# ── arrows: short structural hops (solid dark) ─────────────────────── -arrow((10, 86.2), (14, 81.2), label="OHLCV", lx=-3.4, ly=0.2) # oanda → providers -arrow((27, 86.2), (24, 81.2), ls=(0, (3, 2)), lw=1.1) # fallbacks → providers -arrow((44, 86.2), (48, 81.2)) # FF → news -arrow((59.5, 86.2), (56, 81.2)) # RSS → news -arrow((72, 86.2), (62, 81.2), label="LLM", lx=3.4, ly=0.6) # deepseek → news -arrow((19, 69.8), (19, 64.2), label="DataFrames (UTC, closed bars)", lx=0.2, ly=0.9) -arrow((33.5, 57.5), (35.5, 55.5)) # engine → aggregator -arrow((32.5, 46.0), (35.5, 48.0)) # factory → aggregator -arrow((55, 69.8), (55, 63.1), label="warn-only context", lx=0, ly=1.0) -arrow((86, 69.8), (86, 67.3), ls=(0, (3, 2)), lw=1.1) # config → core -arrow((50, 40.5), (50, 38.8)) # core → report -arrow((36, 34.5), (8.5, 27.95), conn="arc3,rad=0.12") # report → cli -arrow((60, 33.2), (58, 27.95)) # report → dashboard -arrow((66, 34.5), (86.5, 27.95), conn="arc3,rad=-0.12") # report → telegram - -# ── long-haul edges (thin dashed, routed through gutters) ──────────── -# telegram bot ↔ Telegram API — right-margin arc, clear of app_config -arrow((95, 86.2), (95, 27.95), conn="arc3,rad=0.15", color=HAUL, - lw=1.2, ls=(0, (4, 2))) -ax.text(99.2, 57, "alerts / commands", fontsize=6.4, color=MUT, - rotation=90, ha="center", va="center") -# browser click → news_summary.php → DeepSeek -arrow((66, 27.95), (73, 86.2), conn="arc3,rad=-0.18", color=HAUL, - lw=1.2, ls=(0, (4, 2)), - label="on-demand AI summary\n(browser click → PHP)", lx=2.5, ly=11.6) -# providers.fetch_range → backtester — far-left margin channel -arrow((1.5, 69.8), (1.5, 11.0), color=BTS_EC, lw=1.1, ls=(0, (3, 2)), - label="fetch_range\n(history)", lx=3.6, ly=-1.2) -# backtester replays the signal core — right gutter between boxes -arrow((74.0, 11.6), (74.9, 40.3), color=HAUL, lw=1.3) -ax.text(81.8, 38.9, "replays the exact signal code", fontsize=6.4, - color=MUT, ha="center", va="center", zorder=5, - bbox=dict(boxstyle="round,pad=0.15", fc="white", ec="none", alpha=0.9)) - -if __name__ == "__main__": # importing the module must never rewrite the PNG - out = os.path.join(os.path.dirname(os.path.abspath(__file__)), "architecture.png") - fig.savefig(out, dpi=200, bbox_inches="tight", facecolor="white") - print(f"saved {out}") diff --git a/docs/data_source_change_test_plan.md b/docs/data_source_change_test_plan.md deleted file mode 100644 index c1480cf..0000000 --- a/docs/data_source_change_test_plan.md +++ /dev/null @@ -1,92 +0,0 @@ -# Test Plan — Data Source Change Validation - -**Purpose:** determine whether switching `data_providers.active` (e.g. Twelve Data → Oanda, or any future provider swap) changes signal calculations materially, and gate the switch on evidence. - -**Status:** in progress, 2026-08-14 - -> **Progress (2026-08-14):** Phase 0 ✅, Phase 1 ✅ (`backtest/research/feed_divergence_2026-08-14.md`), Phase 2 ✅ (`backtest/research/signal_parity_2026-08-14.md`), Phase 3 ✅ (`backtest/research/repin_verification_2026-08-14.md` — re-pin already existed), Phase 4 🔨 tool ready (`scripts/shadow_scan.py`, needs a 2-week run — launch manually per owner decision), Phase 5 owner decisions recorded (dual cost reporting; Tiingo streaming-only; see bottom). -**Context:** Oanda has been the live feed since 2026-07-28. The backtest books were **re-pinned on Oanda bars on 2026-08-04** (`backtest/oanda_*.csv` — 27 books, now the baseline; all TD-era books superseded). Cost reporting is dual: $0.30 retained for cross-book comparability, $0.65 is the realistic headline (measured Oanda round trip $0.65–0.72). Twelve Data was proven signal-incompatible in Phase 1 (~19% of M5 bars disagree on direction; systematic ~$1 bias) — it is NOT a valid candle fallback. - -**Why signals can differ between feeds** (mechanisms to test): - -- Feeds publish different mid prices → tiny OHLC diffs shift every indicator; near a threshold (RSI band, `FB2_EMA_LO` EMA200-proximity band, composite/grade cutoffs, M5 min-score gate) one bar's discrete score flips. -- Gate edge cases turn a tiny price diff into "setup generated" vs "no signal". -- ATR-derived SL/TP/trail levels differ even when the signal is identical. -- The shared 10σ cleaning pass can drop different bars per feed. -- Volume present (Oanda) vs absent (Tiingo FX); real spread (Oanda MBA) vs pair default (Tiingo). -- Known reference point: H4 seeding divergence — same provider, different fetch depth flipped ~2% of indicator signals (composite Δ up to 12.3). Cross-feed diffs act the same way. - ---- - -## Phase 0 — Recent-bar parity smoke test ✅ DONE - -- **Tool:** `scripts/compare_oanda_tiingo.py` (removed 2026-09-06 once Phases 1-2 superseded it; the result is recorded here) -- **Result 2026-08-14:** 10/10 timestamp alignment on M5/M15/H1, Close max |Δ| ≤ $0.175 (≤ 0.004% of price), no directional bias. -- **Verdict:** recent-tail parity confirmed. Does NOT cover historical depth — proceed to Phase 1. - -## Phase 1 — Historical candle divergence study - -**Goal:** quantify raw-data divergence over the full window used for validation, not just the recent tail. - -- [x] Extend the comparison script to use `fetch_range` (paginated, raw bars) for both providers over **≥ 12 months** for M5, M15, H1 (and H4 if in scope). -- [x] Record per TF: - - % of bar timestamps present in both feeds; list of missing/extra bars (gaps, holidays, weekend boundary handling). - - OHLC |Δ| distribution: p50 / p95 / p99 / max, in $ and % of price. - - Count of bars dropped by `validate_and_clean` on each feed (10σ outliers, malformed). - - Bar-close timestamp conventions (alignment, DST behavior around US transitions). -- [x] **Pass criteria (measured on the weekend-filtered bar grid — the aggregator's NY-anchored `_is_forex_weekend` decides which bars exist for signal purposes, so raw timestamp overlap includes coverage-only diffs like Tiingo's Sunday bars and is NOT the criterion):** ≥ 99.5% overlap of *signal-eligible* bar timestamps; Close p95 |Δ| ≤ 0.01% of price; no systematic bias (mean Δ ≈ 0); no unexplained multi-hour gaps unique to one feed. (Raw-overlap result 2026-08-14: 94.6–96.6% vs Tiingo — entirely explained by Tiingo's ~22h/week of flat Sunday bars, which the weekend filter gates out.) -- [x] Write findings to `backtest/research/feed_divergence_.md` with tables and the exact commands used. - -## Phase 2 — Signal-level replay parity - -**Goal:** measure how often candle divergence actually changes a *decision*. - -- [x] Replay bar-by-bar through the real `analyze_from_dataframes` (news off) for both feeds over the same ≥ 12-month window, per TF (M5, M15, H1). Reuse the gpu_backtest replay harness; disable the parquet cache (`--no-cache`) or use separate cache keys per provider. -- [x] Compare per bar: signal direction, composite score, quality grade, gate outcomes (which gate fired), setup presence, strategy label (XAU ATR / XAU_PB / XAU_CP), entry/SL/TP prices. -- [x] Metrics: - - % bars with identical direction+grade. - - Direction flips, grade changes, gate-only changes (setup appears/disappears). - - Distribution of composite Δ on mismatched bars. - - For matching signals: mean/max |Δ| of entry, SL, TP1/TP2 in $ and in ATR fractions. -- [x] **Result (Oanda↔Tiingo):** 98.32% / 98.16% same direction (M15/M5), zero opposite-sign flips; 93.41% / 94.21% same direction+grade+setup. The original ≥97% direction+grade proposal is too strict for benign grade-letter changes and was not met; fallback acceptance is direction ≥98%, zero sign flips, and marginal/boundary-hour-only flips. See `signal_parity_2026-08-14.md`. - -## Phase 3 — Backtest re-pin on the new feed - -**Goal:** a new pinned book on the new provider's bars, evaluated under rule 1 acceptance criteria. - -- [x] **Provider selection:** `gpu_backtest.py` has **no `--provider` flag** — it builds the provider from `data_providers.active` in config.json (`gpu_backtest.py:1753`). A cross-provider run requires flipping that config line. The parquet cache is already keyed per provider (`gpu_backtest.py:984` — `oanda_`-prefixed files vs legacy unprefixed TD files), so TD and Oanda data cannot mix in cache; `--no-cache` is only needed to force a refetch. -- [x] Run the full backtest on new-feed bars, both cost assumptions: - ```bash - python backtest/gpu_backtest.py --pair XAUUSD --days 365 --spread 0.30 --export-csv backtest/repin__m15_030.csv - python backtest/gpu_backtest.py --pair XAUUSD --days 365 --spread 0.65 --export-csv backtest/repin__m15_065.csv - ``` - (repeat for each TF under test; note the `Backend:` log line and provider in the report per `backtest/instructions_for_backtest.txt`). -- [x] Split into train/validation halves; require **positive total R in both halves and PF ≥ 1.3 at realistic cost** (rule 1), with trailing exits and next-open fills untouched. -- [x] Marginal configs additionally pass the **2024-06 → 2025-03 OOS window** (within-year splits alone have approved configs that later failed OOS). -- [x] ~~Compare against the pinned Twelve-Data book trade-by-trade~~ **AMENDED 2026-08-14:** skipped with justification — Phase 1 showed TD bars disagree on direction 19% of the time on M5 with a systematic ~$1 bias, so a TD-pinned book is a different instrument's book and a trade-level diff carries no decision value. Phantom-trade classes remain documented (e.g. the weekend-calendar fix already dropped exactly one live-impossible M5 trade). -- [x] **Deliverable:** re-pinned book + report in `backtest/research/`; update the reference numbers in CLAUDE.md if the book is adopted. Current-HEAD M5 canonical book: `backtest/oanda_m5_12mo_065_nopb_20260814.csv`, +106.82R PF 1.51. - -## Phase 4 — Live shadow comparison - -**Goal:** confirm parity under real fetch depths, caching, and boundary timing — not just history. - -- [ ] Run a shadow scan loop (no orders, no Telegram) for **≥ 2 weeks**: each candle close, scan with both providers, log direction/grade/gates/setup to a side-by-side file or SQLite table. -- [ ] Include at least one weekend transition and one news-heavy day in the window. -- [ ] Track: scan-to-scan agreement rate, feed-lag incidents ("feed lagging — just-closed bar missing"), staleness warnings, spread readings (Oanda real vs Tiingo default). -- [ ] **Acceptance thresholds (baseline = Phase 2 replay parity):** ≥ 98% direction agreement; **zero** opposite-sign (BUY↔SELL) flips; direction flips confined to actionable↔NEUTRAL at marginal composites, clustered only at Friday-close/Sunday-reopen (UTC 22–23) and Asian (03–04) hours; setup-level drift (SL/TP) within the Phase 2 envelope (mean |Δ| ≲ $3, p95 ≲ $17 on M15); every feed-lag/staleness incident explained. -- [ ] H4 caution: live H4 fetches only 380 bars vs unbounded backtest history (known ~2% indicator-signal flip from seeding alone) — keep H4 labeled "unproven" until a dedicated campaign closes this. - -## Phase 5 — Cutover checklist - -- [x] `config.json` `data_providers.active: "oanda"` and `data_providers.streaming: "tiingo"` set deliberately (verified 2026-08-14 — Oanda cannot stream; Tiingo streams). -- [x] Cost model: **dual reporting adopted (owner decision 2026-08-14)** — $0.65 (measured Oanda round trip) is the realistic headline, $0.30 retained for cross-book comparability with TD-era books. ⚠️ **Cost-gate coupling:** `config.py round_trip_cost` (0.30) is NOT just a backtest knob — it feeds the live TP1 spread gate (`min_tp = round_trip_cost × min_tp_cost_multiple`, three sites in `signal_aggregator.py`). Raising it to 0.65 moves the TP1 floor from ~$9 to ~$19.50 and is a rule-1 revalidation event, not a config tweak. `round_trip_cost` stays 0.30. -- [x] QA suite green: `python test_all.py` (381 tests; verified 2026-08-14). -- [ ] Week 1 post-cutover: daily review of `logs/signals.db` rows vs shadow log; alert on grade/direction disagreement beyond the Phase 2 baseline rate. -- [ ] Rollback: revert `data_providers.active`; pinned books of the previous feed remain valid for comparison. ⚠️ Twelve Data is a permanent *registered* fallback but is **signal-incompatible** (Phase 1) — on an Oanda candle outage, stop scanning rather than switching `active` to `twelve_data`; a TD-fed book cannot be compared to the Oanda baseline. -- [x] Docs sync (CLAUDE.md rule): `docs/user_guide.html` backtest numbers reflect the Oanda-pinned books, not TD-era numbers. - -## Open questions — RESOLVED 2026-08-14 (owner decisions) - -- ~~Adopt which spread as canonical~~ → **dual reporting**: $0.65 realistic headline + $0.30 cross-book comparability; `round_trip_cost` stays 0.30 (gate coupling above). -- ~~Tiingo role~~ → **streaming-only**. Tiingo remains registered for the WS tick stream and comparison research, but is not a designated candle fallback; candle fallback is a manual halt-and-decide, never an automatic TD failover. -- H4 seeding campaign (parked since 2026-08-04) → **still needs an owner**; H4 stays "unproven" until then. diff --git a/scripts/feed_divergence_study.py b/scripts/feed_divergence_study.py deleted file mode 100644 index 0488ef8..0000000 --- a/scripts/feed_divergence_study.py +++ /dev/null @@ -1,259 +0,0 @@ -#!/usr/bin/env python3 -""" -Phase 1 — historical candle divergence study (data source change test) -======================================================================= -Fetches the RAW closed-bar series for XAU/USD from two providers over a -multi-month window via the normal provider stack (`fetch_range`, which -paginates under the per-request bar cap and drops the forming candle), -then quantifies how the feeds diverge: - - * coverage: % shared timestamps, feed-only bars, where they cluster - * OHLC |Δ| distribution (mean/median/p90/p95/p99/max, $ and bias) - * |ΔClose| vs mean ATR14 (the scale the signal engine cares about) - * threshold exceedance (how many bars differ by more than X $) - * bar-direction agreement (sign of close−open) - * close-return correlation on consecutive bars - * per-hour and per-month drift - -Both feeds are mid-price (Oanda mid of MBA candles; Tiingo resampled mid; -Twelve Data its own mid). Frames are cached to parquet so re-runs are -instant; pass --no-cache to force a fresh fetch. - -Usage: - venv/bin/python scripts/feed_divergence_study.py --a oanda --b tiingo \ - --tfs M15,H1,H4 --days 365 -""" - -import argparse -import sys -import time -from pathlib import Path - -import numpy as np -import pandas as pd - -sys.path.insert(0, str(Path(__file__).resolve().parent.parent)) - -from config import PlatformConfig, Timeframe, XAUUSD -from providers import make_data_provider - -REPO = Path(__file__).resolve().parent.parent -CACHE_DIR = REPO / "backtest" / "cache" -RESEARCH_DIR = REPO / "backtest" / "research" - -_TF_LOOKUP = {t.value: t for t in Timeframe} -_TF_LOOKUP.update({ - "M1": Timeframe.M1, "M5": Timeframe.M5, "M15": Timeframe.M15, - "M30": Timeframe.M30, "H1": Timeframe.H1, "H4": Timeframe.H4, - "D1": Timeframe.D1, "W1": Timeframe.W1, -}) - - -def wilder_atr(df: pd.DataFrame, period: int = 14) -> pd.Series: - hl = df["High"] - df["Low"] - hc = (df["High"] - df["Close"].shift(1)).abs() - lc = (df["Low"] - df["Close"].shift(1)).abs() - tr = pd.concat([hl, hc, lc], axis=1).max(axis=1) - return tr.ewm(alpha=1.0 / period, adjust=False).mean() - - -def fmt_stats(name: str, s: pd.Series) -> str: - a = s.abs() - return (f" {name:<6} mean ${a.mean():.3f} med ${a.median():.3f} " - f"p90 ${a.quantile(0.90):.3f} p95 ${a.quantile(0.95):.3f} " - f"p99 ${a.quantile(0.99):.3f} max ${a.max():.2f} " - f"bias {s.mean():+.4f}") - - -def cache_path(name: str, tf: Timeframe, start: pd.Timestamp, - end: pd.Timestamp) -> Path: - return (CACHE_DIR / - f"{name}_XAU_USD_{tf.value}_{start:%Y%m%d}_{end:%Y%m%d}.parquet") - - -def load_or_fetch(provider, name: str, tf: Timeframe, start, end, - no_cache: bool): - cp = cache_path(name, tf, start, end) - if not no_cache and cp.exists(): - df = pd.read_parquet(cp) - print(f" [{name}] loaded {len(df):,} bars from cache " - f"{cp.name}") - return df - t0 = time.time() - print(f" [{name}] fetching {tf.value} {start.date()} → {end.date()} ...") - df = provider.fetch_range(XAUUSD, tf, start, end) - print(f" [{name}] fetched {len(df):,} bars in {time.time() - t0:.1f}s") - df.to_parquet(cp) - print(f" [{name}] cached → {cp.name}") - return df - - -def compare(tf: Timeframe, a_name: str, b_name: str, a: pd.DataFrame, - b: pd.DataFrame): - print("\n" + "=" * 80) - print(f"{tf.value} — {a_name.upper()} vs {b_name.upper()}, XAU/USD") - print("=" * 80) - - common = a.index.intersection(b.index) - a_only = a.index.difference(b.index) - b_only = b.index.difference(a.index) - - print(f"\nCOVERAGE") - print(f" {a_name} bars : {len(a):>8,} head {a.index[0]} tail {a.index[-1]}") - print(f" {b_name} bars : {len(b):>8,} head {b.index[0]} tail {b.index[-1]}") - print(f" shared : {len(common):>8,} " - f"({100 * len(common) / max(len(a.index.union(b.index)), 1):.2f}% of union)") - print(f" {a_name}-only : {len(a_only):>8,} " - f"({100 * len(a_only) / max(len(a), 1):.2f}% of {a_name})") - print(f" {b_name}-only : {len(b_only):>8,} " - f"({100 * len(b_only) / max(len(b), 1):.2f}% of {b_name})") - - if len(common) == 0: - print(" !! no shared timestamps — feeds on different bar grids") - return - - for label, idx in ((f"{a_name}-only", a_only), - (f"{b_name}-only", b_only)): - if not len(idx): - continue - by_date = pd.Series(1, index=idx).groupby(idx.date).sum() - top = by_date.sort_values(ascending=False).head(6) - print(f" {label} clusters on: " - + ", ".join(f"{d} ({n})" for d, n in top.items())) - - x = a.loc[common] - y = b.loc[common] - - atr = wilder_atr(x).dropna() - mean_price = x["Close"].mean() - print(f"\nSCALE REFERENCES ({a_name} feed)") - print(f" mean close : ${mean_price:,.2f}") - print(f" mean bar range: ${(x['High'] - x['Low']).mean():.3f}") - print(f" mean ATR14 : ${atr.mean():.3f} median ${atr.median():.3f}") - - print(f"\nPER-BAR DIFFS ({a_name} − {b_name}, $)") - diffs = {} - for col in ("Open", "High", "Low", "Close"): - d = x[col] - y[col] - diffs[col] = d - print(fmt_stats(col, d)) - - dclose = diffs["Close"].abs() - print(f"\n |ΔClose| as % of mean ATR14 : " - f"mean {100 * dclose.mean() / atr.mean():.1f}% " - f"p95 {100 * dclose.quantile(0.95) / atr.mean():.1f}%") - - print(f"\nTHRESHOLD EXCEEDANCE (|ΔClose|)") - for thr in (0.10, 0.30, 0.65, 1.00, 2.00, 5.00): - pct = 100 * (dclose > thr).mean() - print(f" > ${thr:<5.2f}: {pct:6.2f}% of bars ({(dclose > thr).sum():,})") - - # direction agreement - a_dir = np.sign(x["Close"] - x["Open"]) - b_dir = np.sign(y["Close"] - y["Open"]) - both_nonflat = (a_dir != 0) & (b_dir != 0) - dir_agree = 100 * (a_dir[both_nonflat] == b_dir[both_nonflat]).mean() - print(f"\nAGREEMENT") - print(f" bar direction (sign of close−open) agree: {dir_agree:.2f}%" - f" (n={both_nonflat.sum():,})") - - step = pd.Series(common, index=common).diff().median() - a_ret = x["Close"].pct_change() - b_ret = y["Close"].pct_change() - step_ok = pd.Series(common, index=common).diff().le(step + pd.Timedelta(seconds=1)) - corr = a_ret[step_ok].corr(b_ret[step_ok]) - print(f" close-return correlation (consecutive bars): {corr:.5f}") - - # by-hour - hours = common.hour - rows = [] - for h in range(24): - m = hours == h - if not m.any(): - continue - rows.append({"hour_utc": h, "bars": int(m.sum()), - "mean_abs_dclose": dclose[m].mean(), - "p95_abs_dclose": dclose[m].quantile(0.95)}) - by_hour = pd.DataFrame(rows) - print(f"\nBY UTC HOUR (mean |ΔClose|, $)") - print(by_hour.to_string(index=False, - float_format=lambda v: f"{v:.3f}")) - - print(f"\nBY MONTH (mean |ΔClose|, $)") - months = dclose.groupby(common.tz_localize(None).to_period("M")).mean() - for p, v in months.items(): - print(f" {p}: {v:.3f}") - - return { - "tf": tf.value, "a": a_name, "b": b_name, - "a_bars": len(a), "b_bars": len(b), "common": len(common), - "a_only": len(a_only), "b_only": len(b_only), - "dir_agree_pct": round(dir_agree, 3), - "ret_corr": round(corr, 5), - "mean_abs_dclose": dclose.mean(), - "p95_abs_dclose": dclose.quantile(0.95), - "max_abs_dclose": dclose.max(), - "dclose_pct_atr_mean": 100 * dclose.mean() / atr.mean(), - } - - -def main(): - ap = argparse.ArgumentParser() - ap.add_argument("--a", required=True) - ap.add_argument("--b", required=True) - ap.add_argument("--tfs", default="M15,H1,H4", - help="comma-separated timeframe values (default M15,H1,H4)") - ap.add_argument("--days", type=int, default=365) - ap.add_argument("--end", default=None, help="UTC end date (default now)") - ap.add_argument("--no-cache", action="store_true") - args = ap.parse_args() - - tfs = [] - for s in args.tfs.split(","): - s = s.strip() - tf = _TF_LOOKUP.get(s) - if tf is None: - sys.exit(f"unknown timeframe '{s}'; use one of " - f"{sorted(_TF_LOOKUP)}") - tfs.append(tf) - - end = pd.Timestamp(args.end, tz="UTC") if args.end \ - else pd.Timestamp.now(tz="UTC") - start = end - pd.Timedelta(days=args.days) - - cfg = PlatformConfig() - pa = make_data_provider(cfg, args.a) - pb = make_data_provider(cfg, args.b) - - summary = [] - for tf in tfs: - print(f"\n{'─' * 80}\nWINDOW {start.date()} → {end.date()} " - f"({args.days} days), {tf.value}") - try: - a = load_or_fetch(pa, args.a, tf, start, end, args.no_cache) - except Exception as e: - print(f" !! {args.a} {tf.value} fetch failed: {e}") - continue - try: - b = load_or_fetch(pb, args.b, tf, start, end, args.no_cache) - except Exception as e: - print(f" !! {args.b} {tf.value} fetch failed: {e}") - continue - r = compare(tf, args.a, args.b, a, b) - if r: - summary.append(r) - - print("\n" + "=" * 80) - print("SUMMARY TABLE") - print("=" * 80) - sdf = pd.DataFrame(summary) - print(sdf.to_string(index=False, - float_format=lambda v: f"{v:.4f}")) - out = RESEARCH_DIR / f"feed_divergence_{args.a}_vs_{args.b}.csv" - if len(sdf): - sdf.to_csv(out, index=False, float_format="%.6f") - print(f"\nSummary written to {out}") - - -if __name__ == "__main__": - main() diff --git a/scripts/signal_parity_study.py b/scripts/signal_parity_study.py deleted file mode 100644 index c7dd7da..0000000 --- a/scripts/signal_parity_study.py +++ /dev/null @@ -1,246 +0,0 @@ -#!/usr/bin/env python3 -""" -Phase 2 — signal-level replay parity (data source change test) -=============================================================== -Replays the REAL decision core (`SignalAggregator.analyze_from_dataframes`, -news off, use_news=False) over the same window for two providers and -compares the *decisions* bar-by-bar: direction, composite/final score, -quality grade, gate/setup presence, strategy tag, and entry/SL/TP levels. - -This turns the Phase-1 price-diff stats into flip rates. Feeds are read -from the Phase-1 parquet cache (scripts/feed_divergence_study.py writes -them); pass --no-cache-fetch only if you want to fetch fresh via -fetch_range first. - -Slicing matches gpu_backtest.py exactly: decision at close of bar `pos`, -each TF sliced to bars fully closed by that decision time, primary at the -shared grid, so the two feeds are compared on identical timestamps. - -Usage: - venv/bin/python scripts/signal_parity_study.py --a oanda --b tiingo \ - --tf M15 --stride 8 --days 365 -""" - -import argparse -import sys -import time -from pathlib import Path - -import numpy as np -import pandas as pd - -sys.path.insert(0, str(Path(__file__).resolve().parent.parent)) - -from config import Timeframe, XAUUSD -from pair_signals.xauusd import make_config -from signal_aggregator import SignalAggregator - -REPO = Path(__file__).resolve().parent.parent -CACHE_DIR = REPO / "backtest" / "cache" -RESEARCH_DIR = REPO / "backtest" / "research" - -BAR_SECONDS = { - Timeframe.M1: 60, Timeframe.M5: 300, Timeframe.M15: 900, - Timeframe.M30: 1800, Timeframe.H1: 3600, Timeframe.H4: 14400, - Timeframe.D1: 86400, Timeframe.W1: 604800, -} -_TF_LOOKUP = {t.value: t for t in Timeframe} -_TF_LOOKUP.update({"M1": Timeframe.M1, "M5": Timeframe.M5, - "M15": Timeframe.M15, "M30": Timeframe.M30, - "H1": Timeframe.H1, "H4": Timeframe.H4, - "D1": Timeframe.D1, "W1": Timeframe.W1}) -_CONFIRMS = { - Timeframe.M5: [Timeframe.M15, Timeframe.H1], - Timeframe.M15: [Timeframe.M5, Timeframe.H1], - Timeframe.H1: [Timeframe.M15, Timeframe.H4], - Timeframe.H4: [Timeframe.H1, Timeframe.D1], -} - - -def load_cached(name: str, tf: Timeframe, start: pd.Timestamp, - end: pd.Timestamp) -> pd.DataFrame: - cp = CACHE_DIR / f"{name}_XAU_USD_{tf.value}_{start:%Y%m%d}_{end:%Y%m%d}.parquet" - if not cp.exists(): - sys.exit(f"missing cache {cp}; run scripts/feed_divergence_study.py " - f"--a oanda --b {name} --tfs {tf.value} --days ... first") - return pd.read_parquet(cp) - - -def tag_of(report) -> str: - first_bull = report.bullish_reasons[0] if report.bullish_reasons else "" - first_bear = report.bearish_reasons[0] if report.bearish_reasons else "" - if "XAU PULLBACK" in first_bull or "XAU PULLBACK" in first_bear: - return "XAU_PB" - if "XAU CHART PATTERN" in first_bull or "XAU CHART PATTERN" in first_bear: - return "XAU_CP" - if "XAU STRATEGY" in first_bull: - return "XAU" - return "GENERIC" - - -def decide(agg, dfs, tss, stride, warmup) -> list: - """Replay decisions over the shared grid, returning one row per bar.""" - primary = dfs[agg.config.primary_timeframe] - dur = pd.Timedelta(seconds=BAR_SECONDS[agg.config.primary_timeframe]) - - # positions of sampled bars in the primary frame - primary_pos = primary.index.get_indexer(tss) - rows = [] - for i in range(0, len(tss), stride): - ts = tss[i] - pos = int(primary_pos[i]) - if pos < warmup: - continue - slices = {} - for tf, full in dfs.items(): - cutoff = ts # decision_time = ts + dur; bars closed by ts - cut = int(full.index.searchsorted(cutoff, side="right")) - if cut <= 0: - continue - slices[tf] = full.iloc[:cut] - if agg.config.primary_timeframe not in slices: - continue - r = agg.analyze_from_dataframes(XAUUSD, slices, use_news=False) - ts_setup = r.trade_setup - rows.append({ - "ts": ts, - "direction": r.final_signal.value, - "final_score": r.final_score, - "primary_score": r.primary_score, - "grade": r.signal_quality, - "has_setup": ts_setup is not None, - "tag": tag_of(r) if ts_setup else "", - "entry": ts_setup.entry_price if ts_setup else np.nan, - "sl": ts_setup.stop_loss if ts_setup else np.nan, - "tp1": ts_setup.take_profit_1 if ts_setup else np.nan, - "tp2": ts_setup.take_profit_2 if ts_setup else np.nan, - }) - return rows - - -def main(): - ap = argparse.ArgumentParser() - ap.add_argument("--a", required=True) - ap.add_argument("--b", required=True) - ap.add_argument("--tf", default="M15") - ap.add_argument("--stride", type=int, default=1, - help="sample every Nth shared bar (default 1)") - ap.add_argument("--days", type=int, default=365) - ap.add_argument("--end", default=None) - ap.add_argument("--warmup", type=int, default=300) - args = ap.parse_args() - - tf = _TF_LOOKUP.get(args.tf) - if tf is None: - sys.exit(f"unknown tf {args.tf}") - confirms = _CONFIRMS[tf] - tfs = [tf] + confirms - - end = pd.Timestamp(args.end, tz="UTC") if args.end \ - else pd.Timestamp.now(tz="UTC") - start = end - pd.Timedelta(days=args.days) - - cfg = make_config(tf, confirms) - agg = SignalAggregator(cfg) - - a_dfs, b_dfs = {}, {} - for t in tfs: - a_dfs[t] = load_cached(args.a, t, start, end) - b_dfs[t] = load_cached(args.b, t, start, end) - - # shared grid = primary timestamps present in both feeds - shared = a_dfs[tf].index.intersection(b_dfs[tf].index) - - t0 = time.time() - print(f"replaying {len(shared)} shared {tf.value} bars " - f"(stride {args.stride} → ~{len(shared)//args.stride}) on {args.a}...") - a_rows = decide(agg, a_dfs, shared, args.stride, args.warmup) - print(f" {args.a}: {len(a_rows)} decisions in {time.time()-t0:.1f}s") - t0 = time.time() - b_rows = decide(agg, b_dfs, shared, args.stride, args.warmup) - print(f" {args.b}: {len(b_rows)} decisions in {time.time()-t0:.1f}s") - - A = pd.DataFrame(a_rows).set_index("ts") - B = pd.DataFrame(b_rows).set_index("ts") - common = A.index.intersection(B.index) - A, B = A.loc[common], B.loc[common] - n = len(common) - - print(f"\n{'=' * 78}") - print(f"SIGNAL-LEVEL PARITY — {tf.value} — {args.a} vs {args.b} " - f"({n:,} bars)") - print("=" * 78) - - def pct(x): return f"{100 * x:.2f}%" - - same_dir = (A["direction"] == B["direction"]) - same_grade = (A["grade"] == B["grade"]) - same_setup = (A["has_setup"] == B["has_setup"]) - same_all = same_dir & same_grade & same_setup - - print(f"\nIDENTITY") - print(f" same direction : {pct(same_dir.mean())}") - print(f" same grade : {pct(same_grade.mean())}") - print(f" same setup presence : {pct(same_setup.mean())}") - print(f" same dir+grade+setup : {pct(same_all.mean())}") - - both_sig = (A["direction"] != 0) & (B["direction"] != 0) - flips = both_sig & (A["direction"] != B["direction"]) - opp = flips & (A["direction"] == -B["direction"]) - print(f"\nDIRECTION") - print(f" bars both actionable : {both_sig.sum():,} " - f"({pct(both_sig.mean())})") - print(f" direction flips : {flips.sum():,} " - f"({pct(flips.sum()/max(both_sig.sum(),1))} of actionable)") - print(f" opposite-sign flips : {opp.sum():,}") - - grade_a = set(A.loc[A["has_setup"], "grade"]) - print(f"\nGRADE (of {args.a} setups, grades seen: {sorted(grade_a)})") - changed = A.loc[both_sig, "grade"] != B.loc[both_sig, "grade"] - print(f" grade changed on actionable bars: " - f"{changed.sum():,} ({pct(changed.mean())})") - - # score deltas on shared bars - ds = (A["final_score"] - B["final_score"]).abs() - dp = (A["primary_score"] - B["primary_score"]).abs() - print(f"\nSCORE Δ ({args.a} − {args.b})") - print(f" final_score mean {ds.mean():6.2f} p95 {ds.quantile(.95):6.2f} " - f"max {ds.max():6.2f}") - print(f" primary_score mean {dp.mean():6.2f} p95 {dp.quantile(.95):6.2f} " - f"max {dp.max():6.2f}") - - # setup price diffs on bars where BOTH generated a setup - both_setup = A["has_setup"] & B["has_setup"] - if both_setup.any(): - print(f"\nSETUP LEVELS (both-setup bars: {both_setup.sum():,})") - for col, label in (("entry", "entry"), ("sl", "SL"), ("tp1", "TP1"), - ("tp2", "TP2")): - d = (A.loc[both_setup, col] - B.loc[both_setup, col]).abs() - print(f" {label:<6} mean ${d.mean():6.3f} p95 ${d.quantile(.95):7.3f} " - f"max ${d.max():7.2f}") - - # tag disagreement on both-setup bars - if both_setup.any(): - same_tag = (A.loc[both_setup, "tag"] == B.loc[both_setup, "tag"]) - print(f" same strategy tag : {pct(same_tag.mean())}") - - # where do flips cluster (hour)? - flip_hours = A.loc[flips].index.hour.value_counts().sort_index() - if len(flip_hours): - print(f"\nFLIPS BY UTC HOUR") - for h, c in flip_hours.items(): - print(f" {h:02d}: {c}") - - out = { - "tf": tf.value, "a": args.a, "b": args.b, "bars": n, - "same_dir": same_dir.mean(), "same_grade": same_grade.mean(), - "same_setup": same_setup.mean(), "same_all": same_all.mean(), - "flip_rate_actionable": flips.sum() / max(both_sig.sum(), 1), - "mean_abs_final_score_delta": ds.mean(), - "mean_abs_primary_score_delta": dp.mean(), - } - print(f"\n{out}") - - -if __name__ == "__main__": - main()