From 2a85f1e23c166f16a242232c03bb398b3529e2d3 Mon Sep 17 00:00:00 2001 From: Harwinder Sekhon Date: Thu, 16 Jul 2026 09:30:50 -0400 Subject: [PATCH] Docs: delete stale USAGE.md and superseded pullback plan MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit USAGE.md documented a different product: USD/JPY (unsupported — the code is XAU/USD + EUR/USD), 16 main.py flags that don't exist (--backtest, --optimize, --auto-session-filter, …; main.py has five), and backtest.py/metrics.py, which aren't in the repo. Two claims were actively misleading: it called the session filter "default off" (it is True by default, 03:00-23:00 not 07-16) and gave regime-min-adx as 30 (real default 25). CLAUDE.md now covers the CLI, bot and backtester correctly, so this is a delete rather than a rewrite. PULLBACK_ENTRY_PLAN.txt was a pre-implementation draft whose spec no longer matches what shipped — it specifies a 0.5-2.5 ATR retrace window and a detector in engine.py, while XAU_PB ships 0.8-6.0 (the drafted window rejected 81% of valid pullbacks) with the detector in signal_aggregator.py so live and backtest share one code path. Its validated outcome (+38.3R / 254 trades / 12mo M5) and every shipped gate are recorded in CLAUDE.md and the config docstrings. Also drops the now-dangling references to both files from CLAUDE.md and the config.py XAU_PB comment. Co-Authored-By: Claude Fable 5 --- CLAUDE.md | 2 +- PULLBACK_ENTRY_PLAN.txt | 112 ----------------------- USAGE.md | 198 ---------------------------------------- config.py | 2 +- 4 files changed, 2 insertions(+), 312 deletions(-) delete mode 100644 PULLBACK_ENTRY_PLAN.txt delete mode 100644 USAGE.md diff --git a/CLAUDE.md b/CLAUDE.md index 354571f..66636b9 100644 --- a/CLAUDE.md +++ b/CLAUDE.md @@ -2,7 +2,7 @@ Signal-intelligence platform for **XAU/USD (gold — live, validated)** and **EUR/USD (functional but OFF by default — no validated edge, see below)**. It fetches OHLCV from Twelve Data, scores 9 technical indicators per timeframe with regime-adaptive weights, checks multi-timeframe agreement and news/calendar risk, grades the signal (A+…F), and generates a full trade setup (entry, SL, TP1/TP2, trailing plan). Delivery: Telegram bot (auto-scanner + subscriber channel) and a CLI. A realism-focused backtester replays history through the *exact same* signal code. -This file is the authoritative documentation. `DOCUMENTATION.md` and `USAGE.md` are stale (May 2026) — do not trust them. `PULLBACK_ENTRY_PLAN.txt` documents the XAU_PB pullback strategy design. `backtest/instructions_for_backtest.txt` lists what every backtest report must record. +This file is the authoritative documentation. `DOCUMENTATION.md` is stale (May 2026) — do not trust it. `backtest/instructions_for_backtest.txt` lists what every backtest report must record. **Read this whole file before changing signal logic.** Every numeric threshold below was either backtested or is load-bearing for a backtested result. Rules for changes are at the bottom ("Rules for making changes"). diff --git a/PULLBACK_ENTRY_PLAN.txt b/PULLBACK_ENTRY_PLAN.txt deleted file mode 100644 index a4e5b03..0000000 --- a/PULLBACK_ENTRY_PLAN.txt +++ /dev/null @@ -1,112 +0,0 @@ -PULLBACK ENTRY PATTERN — IMPLEMENTATION PLAN -============================================= -Drafted 2026-07-09. Status: IMPLEMENTED & VALIDATED (same day). - -OUTCOME: shipped as XAU_PB. 12mo M5 validation: 254 PB trades, -+38.3R, avgR +0.151, WR 58%, halves +20.1/+18.1, all four full -quarters positive. Core breakout trades undisturbed (-3.1R -displacement). Combined book: 549 trades (~10.6/wk), +140.1R, -PF 1.78, DD 11.3R. Calibration lesson: M5 gold trend legs measure -3-4 ATR swing-to-trough — the drafted 0.5-2.5 ATR retrace window -rejected 81% of valid pullbacks (fixed: 0.8-6.0). PB stops are -~$8 vs ~$25 core, so PB got its own trail arm (0.8R vs 0.6R). -Detector lives in signal_aggregator (NOT engine.py) so the GPU -backtest engine exercises the same code path as live. - -WHY (evidence from 12mo M5 backtests, $0.30 spread, trailing exits) -------------------------------------------------------------------- -All 311 current trades are one archetype: trend-continuation breakout -(EMA 4/4 aligned + ADX>25 + MACD accelerating + price through Keltner -band). We systematically enter at maximum extension: - -- Trades carrying the "NO PULLBACK — price >2.5 ATR from EMA21" warning - average +0.063R vs +0.112R without it (145 vs 103 trades, 6mo set). -- 28% of stopped-out trades never moved toward the target — instant - reversals from buying the local top / selling the local bottom. -- "Extended — RSI in trend" warning trades: +0.057R vs +0.183R without. - -A pullback entry catches the SAME trends at better prices: enter on the -retrace toward EMA21 / Keltner midline, stop below the retrace swing -instead of 2×ATR behind an extended price. Tighter risk, same trend. -This ADDS trades (new signal source) — it does not filter existing ones. - -DETECTOR SPEC (engine.py — new module-level check in M5 path) -------------------------------------------------------------- -Trend context (all required): - 1. EMA alignment >= 3/4 in trend direction (allow 9<21 wobble mid-pullback) - 2. ADX >= 20 on M5 (relaxed vs breakout's 25: ADX decays during retrace) - 3. H1 confirmation score agrees with trend direction (>= +10 / <= -10), - reuse Gate-3 machinery. - -Pullback trigger (all required): - 4. Price touched/undercut EMA21 or KC midline within last 3 bars - (low <= band for BUY; high >= band for SELL). - 5. Retrace depth 0.5–2.5 ATR from the pre-pullback swing extreme - (deeper = trend likely broken; shallower = still extended). - 6. Resumption bar: close back in trend direction (close > open for BUY) - AND close above EMA9 (below for SELL) — do not catch the falling knife. - 7. RSI in the pullback zone: BUY 35–55, SELL 45–65 (NOT the extended - zones the breakout entries fire in — that is the whole point). - -Suppression: - 8. Skip if a breakout signal fired within cooldown (no double entry - on the same leg — cd2/cd3 tests proved same-direction pyramiding - is train-half-only noise). - 9. All existing M5 gates 1–6 still apply where meaningful; Gate 1 - (|score|>=30) will need a variant — pullback bars have weaker - composite scores BY DESIGN (price below the momentum bands), so - validate a separate m5_pullback_min_score knob (start ~20). - -GEOMETRY --------- -- SL: below the pullback swing low (BUY) − 0.5×ATR buffer, floored at - min_sl_pips ($8). Expect typical SL 40–60% tighter than breakout SLs. -- Exit: same trailing stop (arm +0.6R, trail 0.75R, max_hold 300). - NOTE: tighter SL means R units are smaller in dollars — re-validate - trail params on pullback trades separately; the arm/dist plateau was - measured on breakout-sized risk. If needed give pullback its own - trail knobs. -- Tag trades strategy="XAU_PB" in the backtester so every report - splits breakout vs pullback cleanly. - -VALIDATION PROTOCOL (same discipline as 2026-07-09 sessions) ------------------------------------------------------------- -1. Implement detector behind config flag m5_pullback_enabled=False. -2. 365d backtest, flag on; export CSV; analyze XAU_PB trades alone: - - REQUIRE: positive totalR in BOTH data halves - - REQUIRE: positive in >= 3 of 4 full quarters - - REQUIRE: does not degrade breakout-trade totals (displacement - check vs baseline entry_time set, like the cooldown analysis) - - TARGET: >= +0.15R/trade average, >= 1 trade/week added -3. Threshold sweeps (retrace depth, RSI zone, ADX floor) via the - offline exit-sim harness pattern where possible; full reruns where - entries change. -4. Reject-and-document if it fails — same as blocked-hours/cooldown - experiments (see memory: entry-side experiments rejected). - -BASELINE TO BEAT (do not regress) ---------------------------------- -12mo M5: 311 trades, +105.0R, PF 2.27, DD 7.9R, WR 61.4%, -train/val +54.6/+50.4, every quarter positive, ~6-8 trades/week. -(commit 987ec5b, CSV: m5_365d_trail300 in session scratchpad) - -IMPLEMENTATION ORDER --------------------- -1. config.py: m5_pullback_enabled, m5_pullback_min_score, - pullback RSI zones, retrace depth bounds. -2. engine.py: pullback detector producing a candidate flag + swing - levels (attach as result._pullback_setup, underscore-private). -3. signal_aggregator.py: consume flag, build XAU_PB trade setup with - pullback geometry; route through gates with pullback score variant. -4. gpu_backtest.py: ensure strategy tag flows to CSV (already does via - strategy_tag) and pre-filter loop does not drop pullback candidates. -5. Backtest + validate per protocol; tune; commit only on pass. - -RISKS ------ -- Pullback entries in a trend that IS reversing = catching the knife; - the resumption-bar condition (6) is the main guard — do not weaken it. -- Score-gate variant (9) is the most likely source of overfitting; - sweep it last, keep the grid coarse. -- M5 pullbacks during news spikes: calendar buffer already applies via - existing pipeline; verify it fires for XAU_PB too. diff --git a/USAGE.md b/USAGE.md deleted file mode 100644 index 8f25fc9..0000000 --- a/USAGE.md +++ /dev/null @@ -1,198 +0,0 @@ -# ForexBot2 — USD/JPY Signal Intelligence - -A focused USD/JPY signal generator + walk-forward backtester. Live signals -and backtests run through the **same** technical engine, so what you measure -in a backtest is exactly what you'd execute live. - -## Setup - -```bash -pip install -r requirements.txt -``` - -API key (optional — a default key is in `config.py`): - -```bash -export TWELVE_DATA_API_KEY=your_key_here # Linux/Mac -$env:TWELVE_DATA_API_KEY="your_key_here" # PowerShell -``` - ---- - -## Live Signal - -```bash -python main.py # H1, summary view -python main.py --detail # Full indicator breakdown -python main.py --timeframe H4 --detail # H4 instead of H1 -python main.py --json # JSON output (for piping) -``` - -| Option | Default | Description | -|---------------|---------|--------------------------------------------| -| `--timeframe` | `H1` | `M5 M15 M30 H1 H4 D1` | -| `--detail` | off | Indicator breakdown, key levels, reasons | -| `--json` | off | Machine-readable output | -| `--verbose` | off | Debug logging | - -### Output - -``` -USD/JPY │ Price: 149.230 │ Regime: trending_up │ Vol: normal - -SIGNAL: BUY │ Score: +28.4 │ Grade: B │ Confidence: 62% -Action: BUY — Decent setup, wait for confirmation - -Multi-Timeframe: ✓ ALIGNED - M15: BUY (+18.2) - H1: BUY (+28.4) - H4: STRONG BUY (+45.1) - -Trade Setup: - Direction: BUY - Entry: 149.230 - Stop Loss: 148.960 (27.0 pips) - Take Profit 1: 149.770 (54.0 pips) - Take Profit 2: 150.040 - Risk:Reward: 1:2.0 -``` - ---- - -## Backtest (Walk-Forward) - -The backtester replays USD/JPY history bar-by-bar through the live signal -pipeline. No lookahead, no parallel implementation — what passes in -backtest passes live. - -```bash -python main.py --backtest # Full available history -python main.py --backtest --bars 2000 # Last 2000 H1 bars -python main.py --backtest --start 2024-01-01 --end 2024-12-31 -python main.py --backtest --timeframe H4 # Backtest on H4 -python main.py --backtest --min-grade A # Only A/A+ entries -python main.py --backtest --journal # Last 10 trades printed -python main.py --backtest --journal-csv trades.csv # Full journal to CSV -python main.py --backtest --regime-filter # Skip choppy/ranging bars -python main.py --backtest --regime-filter --regime-min-adx 35 -python main.py --backtest --session-filter # Trade only 07-16 UTC -python main.py --backtest --session-filter --session-start 8 --session-end 17 -python main.py --backtest --auto-session-filter # Reactive session toggle -python main.py --backtest --auto-session-filter --auto-session-lookback 15 \ - --auto-session-threshold-pct -0.5 -``` - -### Regime filter - -The engine is a confluence-based trend-follower — it makes money in trending -regimes and loses in chop. `--regime-filter` gates entries on H4 ADX: -signals only fire when ADX ≥ `--regime-min-adx` (default 30). Sweet spot is -ADX 30–35; tighter cuts trades faster than it improves quality. - -### Session filter - -`--session-filter` restricts entries to a UTC hour window (default 07-16, -covering London + NY). Diagnostic on 2025-11 → 2026-05 showed 47% WR / -+127 pips inside that window vs 0% WR / -571 pips outside it. Override -bounds with `--session-start` (inclusive) and `--session-end` (exclusive). - -Tradeoff: it's regime-specific medicine. In trending regimes (e.g. 2024) -it cuts winning off-hours trades and gives back ~33% of the edge. Ships -**default off** because the pure regime filter still wins cumulative -across the 2024 / 2023 / 2026 windows. - -### Auto session toggle - -`--auto-session-filter` is a reactive variant: it tracks the rolling mean -`pnl_pct` over the last N closed trades (`--auto-session-lookback`, -default 10) and activates the 07-16 UTC window only when that mean drops -below `--auto-session-threshold-pct` (default `-0.3`). Releases -automatically once expectancy recovers. - -Less aggressive than the manual flag — preserves more trend edge but -reacts slowly because the deque must fill with losses before kicking in. -Use this when you don't want to manually toggle the regime medicine. - -### Execution model - -| Aspect | Behavior | -|--------------------|-------------------------------------------------------------------| -| Entry | Next bar's open (no current-bar lookahead) | -| Stop Loss | 1.5 × ATR from entry (configurable) | -| Take Profit | RR × stop distance (default 1:2) | -| Same-bar SL & TP | Conservative: SL fills first | -| Spread cost | Half-spread on entry + half-spread on exit (full round-trip) | -| Slippage | 0.5 pips per fill (entry + exit) | -| Position size | 1% of current equity risked per trade | -| Concurrency | One open position at a time | -| News in backtest | Disabled — RSS/calendar can't be replayed without lookahead | - -### Output - -``` -Backtest Summary — USD/JPY -───────────────────────────────────────────────── -Trades: 87 (52W / 35L) -Win Rate: 59.8% -Profit Factor: 1.84 -Expectancy: +12.4 pips/trade (+12.42 per trade) -Initial Balance: $10,000.00 -Final Balance: $11,084.00 -Total Return: +10.84% -CAGR: +18.92% -Max Drawdown: -4.21% -Sharpe Ratio: 1.42 -Sortino Ratio: 2.18 -Avg Duration: 14.3 bars -Win Streak: 7 | Loss Streak: 4 -───────────────────────────────────────────────── -``` - ---- - -## Parameter Optimization (parallel) - -```bash -python main.py --backtest --optimize -``` - -Runs a 3×3×3 grid (27 combos) over `atr_sl_multiplier`, `default_rr_ratio`, -and `min_score` (signal-score threshold). Combos run in parallel across all -CPU cores via `multiprocessing.Pool`. Twelve Data is hit once — workers -share an on-disk parquet cache (`.cache/usdjpy_.parquet`). - -Output ranks combos by total return: - -``` -Top 10 Parameter Combinations (by total return): - -# Return% PF WinR Trades MaxDD% Sharpe Params -────────────────────────────────────────────────────────────────────── -1 +14.21% 2.10 62.5% 64 -3.84 1.78 atr_sl_multiplier=1.5, default_rr_ratio=2.5, min_score=25.0 -2 +11.93% 1.92 58.7% 71 -4.11 1.55 atr_sl_multiplier=1.0, default_rr_ratio=2.0, min_score=15.0 -... -``` - -Customize the grid in `config.py → BacktestConfig.param_grid`. - ---- - -## Files - -| File | Role | -|-----------------------|---------------------------------------------------| -| `main.py` | CLI entry point (live + backtest + optimize) | -| `config.py` | All parameters in one place | -| `data_provider.py` | Twelve Data fetcher + parquet cache | -| `engine.py` | Technical indicators (8 + divergence + patterns) | -| `signal_aggregator.py`| Multi-TF aggregation, grading, trade setup | -| `news_engine.py` | RSS sentiment + economic calendar (live only) | -| `backtest.py` | Walk-forward replay + parallel optimizer | -| `metrics.py` | Win rate, PF, Sharpe, max DD, trade journal | - ---- - -## Disclaimer - -For educational and research use. Past performance does not guarantee -future results. Always use proper risk management. diff --git a/config.py b/config.py index 563f29e..4ba24ec 100755 --- a/config.py +++ b/config.py @@ -251,7 +251,7 @@ class XauStrategyConfig: # halves of 6mo and 12mo backtests). m5_sell_min_rsi: float = 25.0 - # ── M5 pullback entry (XAU_PB — see PULLBACK_ENTRY_PLAN.txt) ── + # ── M5 pullback entry (XAU_PB) ── # Second entry archetype: enter on the retrace to EMA21/KC-mid in # an established trend instead of at Keltner-breakout extension. # Validated 2026-07: +38.3R/254 trades on 12mo, positive both