harry/ForexBot2
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

Docs: delete stale USAGE.md and superseded pullback plan

Browse Source 
USAGE.md documented a different product: USD/JPY (unsupported — the
code is XAU/USD + EUR/USD), 16 main.py flags that don't exist
(--backtest, --optimize, --auto-session-filter, …; main.py has five),
and backtest.py/metrics.py, which aren't in the repo. Two claims were
actively misleading: it called the session filter "default off" (it is
True by default, 03:00-23:00 not 07-16) and gave regime-min-adx as 30
(real default 25). CLAUDE.md now covers the CLI, bot and backtester
correctly, so this is a delete rather than a rewrite.

PULLBACK_ENTRY_PLAN.txt was a pre-implementation draft whose spec no
longer matches what shipped — it specifies a 0.5-2.5 ATR retrace window
and a detector in engine.py, while XAU_PB ships 0.8-6.0 (the drafted
window rejected 81% of valid pullbacks) with the detector in
signal_aggregator.py so live and backtest share one code path. Its
validated outcome (+38.3R / 254 trades / 12mo M5) and every shipped
gate are recorded in CLAUDE.md and the config docstrings.

Also drops the now-dangling references to both files from CLAUDE.md
and the config.py XAU_PB comment.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
This commit is contained in:
Harwinder Sekhon
2026-07-16 09:30:50 -04:00
parent 79ea70c8d7
commit 2a85f1e23c
4 changed files with 2 additions and 312 deletions
Download Patch File Download Diff File Expand all files Collapse all files
CLAUDE.md
+1 -1
View File
@@ -2,7 +2,7 @@
Signal-intelligence platform for **XAU/USD (gold — live, validated)** and **EUR/USD (functional but OFF by default — no validated edge, see below)**. It fetches OHLCV from Twelve Data, scores 9 technical indicators per timeframe with regime-adaptive weights, checks multi-timeframe agreement and news/calendar risk, grades the signal (A+…F), and generates a full trade setup (entry, SL, TP1/TP2, trailing plan). Delivery: Telegram bot (auto-scanner + subscriber channel) and a CLI. A realism-focused backtester replays history through the *exact same* signal code. Signal-intelligence platform for **XAU/USD (gold — live, validated)** and **EUR/USD (functional but OFF by default — no validated edge, see below)**. It fetches OHLCV from Twelve Data, scores 9 technical indicators per timeframe with regime-adaptive weights, checks multi-timeframe agreement and news/calendar risk, grades the signal (A+…F), and generates a full trade setup (entry, SL, TP1/TP2, trailing plan). Delivery: Telegram bot (auto-scanner + subscriber channel) and a CLI. A realism-focused backtester replays history through the *exact same* signal code.
This file is the authoritative documentation. `DOCUMENTATION.md` and `USAGE.md` are stale (May 2026) — do not trust them. `PULLBACK_ENTRY_PLAN.txt` documents the XAU_PB pullback strategy design. `backtest/instructions_for_backtest.txt` lists what every backtest report must record. This file is the authoritative documentation. `DOCUMENTATION.md` is stale (May 2026) — do not trust it. `backtest/instructions_for_backtest.txt` lists what every backtest report must record.
**Read this whole file before changing signal logic.** Every numeric threshold below was either backtested or is load-bearing for a backtested result. Rules for changes are at the bottom ("Rules for making changes"). **Read this whole file before changing signal logic.** Every numeric threshold below was either backtested or is load-bearing for a backtested result. Rules for changes are at the bottom ("Rules for making changes").
PULLBACK_ENTRY_PLAN.txt
-112
View File
@@ -1,112 +0,0 @@
PULLBACK ENTRY PATTERN — IMPLEMENTATION PLAN
=============================================
Drafted 2026-07-09. Status: IMPLEMENTED & VALIDATED (same day).
OUTCOME: shipped as XAU_PB. 12mo M5 validation: 254 PB trades,
+38.3R, avgR +0.151, WR 58%, halves +20.1/+18.1, all four full
quarters positive. Core breakout trades undisturbed (-3.1R
displacement). Combined book: 549 trades (~10.6/wk), +140.1R,
PF 1.78, DD 11.3R. Calibration lesson: M5 gold trend legs measure
3-4 ATR swing-to-trough — the drafted 0.5-2.5 ATR retrace window
rejected 81% of valid pullbacks (fixed: 0.8-6.0). PB stops are
~$8 vs ~$25 core, so PB got its own trail arm (0.8R vs 0.6R).
Detector lives in signal_aggregator (NOT engine.py) so the GPU
backtest engine exercises the same code path as live.
WHY (evidence from 12mo M5 backtests, $0.30 spread, trailing exits)
-------------------------------------------------------------------
All 311 current trades are one archetype: trend-continuation breakout
(EMA 4/4 aligned + ADX>25 + MACD accelerating + price through Keltner
band). We systematically enter at maximum extension:
- Trades carrying the "NO PULLBACK — price >2.5 ATR from EMA21" warning
average +0.063R vs +0.112R without it (145 vs 103 trades, 6mo set).
- 28% of stopped-out trades never moved toward the target — instant
reversals from buying the local top / selling the local bottom.
- "Extended — RSI in trend" warning trades: +0.057R vs +0.183R without.
A pullback entry catches the SAME trends at better prices: enter on the
retrace toward EMA21 / Keltner midline, stop below the retrace swing
instead of 2×ATR behind an extended price. Tighter risk, same trend.
This ADDS trades (new signal source) — it does not filter existing ones.
DETECTOR SPEC (engine.py — new module-level check in M5 path)
-------------------------------------------------------------
Trend context (all required):
1. EMA alignment >= 3/4 in trend direction (allow 9<21 wobble mid-pullback)
2. ADX >= 20 on M5 (relaxed vs breakout's 25: ADX decays during retrace)
3. H1 confirmation score agrees with trend direction (>= +10 / <= -10),
reuse Gate-3 machinery.
Pullback trigger (all required):
4. Price touched/undercut EMA21 or KC midline within last 3 bars
(low <= band for BUY; high >= band for SELL).
5. Retrace depth 0.5–2.5 ATR from the pre-pullback swing extreme
(deeper = trend likely broken; shallower = still extended).
6. Resumption bar: close back in trend direction (close > open for BUY)
AND close above EMA9 (below for SELL) — do not catch the falling knife.
7. RSI in the pullback zone: BUY 35–55, SELL 45–65 (NOT the extended
zones the breakout entries fire in — that is the whole point).
Suppression:
8. Skip if a breakout signal fired within cooldown (no double entry
on the same leg — cd2/cd3 tests proved same-direction pyramiding
is train-half-only noise).
9. All existing M5 gates 1–6 still apply where meaningful; Gate 1
(|score|>=30) will need a variant — pullback bars have weaker
composite scores BY DESIGN (price below the momentum bands), so
validate a separate m5_pullback_min_score knob (start ~20).
GEOMETRY
--------
- SL: below the pullback swing low (BUY) − 0.5×ATR buffer, floored at
min_sl_pips ($8). Expect typical SL 40–60% tighter than breakout SLs.
- Exit: same trailing stop (arm +0.6R, trail 0.75R, max_hold 300).
NOTE: tighter SL means R units are smaller in dollars — re-validate
trail params on pullback trades separately; the arm/dist plateau was
measured on breakout-sized risk. If needed give pullback its own
trail knobs.
- Tag trades strategy="XAU_PB" in the backtester so every report
splits breakout vs pullback cleanly.
VALIDATION PROTOCOL (same discipline as 2026-07-09 sessions)
------------------------------------------------------------
1. Implement detector behind config flag m5_pullback_enabled=False.
2. 365d backtest, flag on; export CSV; analyze XAU_PB trades alone:
- REQUIRE: positive totalR in BOTH data halves
- REQUIRE: positive in >= 3 of 4 full quarters
- REQUIRE: does not degrade breakout-trade totals (displacement
check vs baseline entry_time set, like the cooldown analysis)
- TARGET: >= +0.15R/trade average, >= 1 trade/week added
3. Threshold sweeps (retrace depth, RSI zone, ADX floor) via the
offline exit-sim harness pattern where possible; full reruns where
entries change.
4. Reject-and-document if it fails — same as blocked-hours/cooldown
experiments (see memory: entry-side experiments rejected).
BASELINE TO BEAT (do not regress)
---------------------------------
12mo M5: 311 trades, +105.0R, PF 2.27, DD 7.9R, WR 61.4%,
train/val +54.6/+50.4, every quarter positive, ~6-8 trades/week.
(commit 987ec5b, CSV: m5_365d_trail300 in session scratchpad)
IMPLEMENTATION ORDER
--------------------
1. config.py: m5_pullback_enabled, m5_pullback_min_score,
pullback RSI zones, retrace depth bounds.
2. engine.py: pullback detector producing a candidate flag + swing
levels (attach as result._pullback_setup, underscore-private).
3. signal_aggregator.py: consume flag, build XAU_PB trade setup with
pullback geometry; route through gates with pullback score variant.
4. gpu_backtest.py: ensure strategy tag flows to CSV (already does via
strategy_tag) and pre-filter loop does not drop pullback candidates.
5. Backtest + validate per protocol; tune; commit only on pass.
RISKS
-----
- Pullback entries in a trend that IS reversing = catching the knife;
the resumption-bar condition (6) is the main guard — do not weaken it.
- Score-gate variant (9) is the most likely source of overfitting;
sweep it last, keep the grid coarse.
- M5 pullbacks during news spikes: calendar buffer already applies via
existing pipeline; verify it fires for XAU_PB too.
USAGE.md
-198
View File
@@ -1,198 +0,0 @@
# ForexBot2 — USD/JPY Signal Intelligence
A focused USD/JPY signal generator + walk-forward backtester. Live signals
and backtests run through the **same** technical engine, so what you measure
in a backtest is exactly what you'd execute live.
## Setup
```bash
pip install -r requirements.txt
```
API key (optional — a default key is in `config.py`):
```bash
export TWELVE_DATA_API_KEY=your_key_here # Linux/Mac
$env:TWELVE_DATA_API_KEY="your_key_here" # PowerShell
```
---
## Live Signal
```bash
python main.py # H1, summary view
python main.py --detail # Full indicator breakdown
python main.py --timeframe H4 --detail # H4 instead of H1
python main.py --json # JSON output (for piping)
```
| Option | Default | Description |
|---------------|---------|--------------------------------------------|
| `--timeframe` | `H1` | `M5 M15 M30 H1 H4 D1` |
| `--detail` | off | Indicator breakdown, key levels, reasons |
| `--json` | off | Machine-readable output |
| `--verbose` | off | Debug logging |
### Output
```
USD/JPY │ Price: 149.230 │ Regime: trending_up │ Vol: normal
SIGNAL: BUY │ Score: +28.4 │ Grade: B │ Confidence: 62%
Action: BUY — Decent setup, wait for confirmation
Multi-Timeframe: ✓ ALIGNED
M15: BUY (+18.2)
H1: BUY (+28.4)
H4: STRONG BUY (+45.1)
Trade Setup:
Direction: BUY
Entry: 149.230
Stop Loss: 148.960 (27.0 pips)
Take Profit 1: 149.770 (54.0 pips)
Take Profit 2: 150.040
Risk:Reward: 1:2.0
```
---
## Backtest (Walk-Forward)
The backtester replays USD/JPY history bar-by-bar through the live signal
pipeline. No lookahead, no parallel implementation — what passes in
backtest passes live.
```bash
python main.py --backtest # Full available history
python main.py --backtest --bars 2000 # Last 2000 H1 bars
python main.py --backtest --start 2024-01-01 --end 2024-12-31
python main.py --backtest --timeframe H4 # Backtest on H4
python main.py --backtest --min-grade A # Only A/A+ entries
python main.py --backtest --journal # Last 10 trades printed
python main.py --backtest --journal-csv trades.csv # Full journal to CSV
python main.py --backtest --regime-filter # Skip choppy/ranging bars
python main.py --backtest --regime-filter --regime-min-adx 35
python main.py --backtest --session-filter # Trade only 07-16 UTC
python main.py --backtest --session-filter --session-start 8 --session-end 17
python main.py --backtest --auto-session-filter # Reactive session toggle
python main.py --backtest --auto-session-filter --auto-session-lookback 15 \
--auto-session-threshold-pct -0.5
```
### Regime filter
The engine is a confluence-based trend-follower — it makes money in trending
regimes and loses in chop. `--regime-filter` gates entries on H4 ADX:
signals only fire when ADX ≥ `--regime-min-adx` (default 30). Sweet spot is
ADX 30–35; tighter cuts trades faster than it improves quality.
### Session filter
`--session-filter` restricts entries to a UTC hour window (default 07-16,
covering London + NY). Diagnostic on 2025-11 → 2026-05 showed 47% WR /
+127 pips inside that window vs 0% WR / -571 pips outside it. Override
bounds with `--session-start` (inclusive) and `--session-end` (exclusive).
Tradeoff: it's regime-specific medicine. In trending regimes (e.g. 2024)
it cuts winning off-hours trades and gives back ~33% of the edge. Ships
**default off** because the pure regime filter still wins cumulative
across the 2024 / 2023 / 2026 windows.
### Auto session toggle
`--auto-session-filter` is a reactive variant: it tracks the rolling mean
`pnl_pct` over the last N closed trades (`--auto-session-lookback`,
default 10) and activates the 07-16 UTC window only when that mean drops
below `--auto-session-threshold-pct` (default `-0.3`). Releases
automatically once expectancy recovers.
Less aggressive than the manual flag — preserves more trend edge but
reacts slowly because the deque must fill with losses before kicking in.
Use this when you don't want to manually toggle the regime medicine.
### Execution model
| Aspect | Behavior |
|--------------------|-------------------------------------------------------------------|
| Entry | Next bar's open (no current-bar lookahead) |
| Stop Loss | 1.5 × ATR from entry (configurable) |
| Take Profit | RR × stop distance (default 1:2) |
| Same-bar SL & TP | Conservative: SL fills first |
| Spread cost | Half-spread on entry + half-spread on exit (full round-trip) |
| Slippage | 0.5 pips per fill (entry + exit) |
| Position size | 1% of current equity risked per trade |
| Concurrency | One open position at a time |
| News in backtest | Disabled — RSS/calendar can't be replayed without lookahead |
### Output
```
Backtest Summary — USD/JPY
─────────────────────────────────────────────────
Trades: 87 (52W / 35L)
Win Rate: 59.8%
Profit Factor: 1.84
Expectancy: +12.4 pips/trade (+12.42 per trade)
Initial Balance: $10,000.00
Final Balance: $11,084.00
Total Return: +10.84%
CAGR: +18.92%
Max Drawdown: -4.21%
Sharpe Ratio: 1.42
Sortino Ratio: 2.18
Avg Duration: 14.3 bars
Win Streak: 7 | Loss Streak: 4
─────────────────────────────────────────────────
```
---
## Parameter Optimization (parallel)
```bash
python main.py --backtest --optimize
```
Runs a 3×3×3 grid (27 combos) over `atr_sl_multiplier`, `default_rr_ratio`,
and `min_score` (signal-score threshold). Combos run in parallel across all
CPU cores via `multiprocessing.Pool`. Twelve Data is hit once — workers
share an on-disk parquet cache (`.cache/usdjpy_<tf>.parquet`).
Output ranks combos by total return:
```
Top 10 Parameter Combinations (by total return):
# Return% PF WinR Trades MaxDD% Sharpe Params
──────────────────────────────────────────────────────────────────────
1 +14.21% 2.10 62.5% 64 -3.84 1.78 atr_sl_multiplier=1.5, default_rr_ratio=2.5, min_score=25.0
2 +11.93% 1.92 58.7% 71 -4.11 1.55 atr_sl_multiplier=1.0, default_rr_ratio=2.0, min_score=15.0
...
```
Customize the grid in `config.py → BacktestConfig.param_grid`.
---
## Files
| File | Role |
|-----------------------|---------------------------------------------------|
| `main.py` | CLI entry point (live + backtest + optimize) |
| `config.py` | All parameters in one place |
| `data_provider.py` | Twelve Data fetcher + parquet cache |
| `engine.py` | Technical indicators (8 + divergence + patterns) |
| `signal_aggregator.py`| Multi-TF aggregation, grading, trade setup |
| `news_engine.py` | RSS sentiment + economic calendar (live only) |
| `backtest.py` | Walk-forward replay + parallel optimizer |
| `metrics.py` | Win rate, PF, Sharpe, max DD, trade journal |
---
## Disclaimer
For educational and research use. Past performance does not guarantee
future results. Always use proper risk management.
config.py
+1 -1
View File
@@ -251,7 +251,7 @@ class XauStrategyConfig:
# halves of 6mo and 12mo backtests). # halves of 6mo and 12mo backtests).
m5_sell_min_rsi: float = 25.0 m5_sell_min_rsi: float = 25.0
# ── M5 pullback entry (XAU_PB — see PULLBACK_ENTRY_PLAN.txt) ── # ── M5 pullback entry (XAU_PB) ──
# Second entry archetype: enter on the retrace to EMA21/KC-mid in # Second entry archetype: enter on the retrace to EMA21/KC-mid in
# an established trend instead of at Keltner-breakout extension. # an established trend instead of at Keltner-breakout extension.
# Validated 2026-07: +38.3R/254 trades on 12mo, positive both # Validated 2026-07: +38.3R/254 trades on 12mo, positive both