Docs: delete stale USAGE.md and superseded pullback plan
USAGE.md documented a different product: USD/JPY (unsupported — the code is XAU/USD + EUR/USD), 16 main.py flags that don't exist (--backtest, --optimize, --auto-session-filter, …; main.py has five), and backtest.py/metrics.py, which aren't in the repo. Two claims were actively misleading: it called the session filter "default off" (it is True by default, 03:00-23:00 not 07-16) and gave regime-min-adx as 30 (real default 25). CLAUDE.md now covers the CLI, bot and backtester correctly, so this is a delete rather than a rewrite. PULLBACK_ENTRY_PLAN.txt was a pre-implementation draft whose spec no longer matches what shipped — it specifies a 0.5-2.5 ATR retrace window and a detector in engine.py, while XAU_PB ships 0.8-6.0 (the drafted window rejected 81% of valid pullbacks) with the detector in signal_aggregator.py so live and backtest share one code path. Its validated outcome (+38.3R / 254 trades / 12mo M5) and every shipped gate are recorded in CLAUDE.md and the config docstrings. Also drops the now-dangling references to both files from CLAUDE.md and the config.py XAU_PB comment.
This commit is contained in:
Harwinder Sekhon
@@ -2,7 +2,7 @@
|
|||||||
|
|
||||||
Signal-intelligence platform for **XAU/USD (gold — live, validated)** and **EUR/USD (functional but OFF by default — no validated edge, see below)**. It fetches OHLCV from Twelve Data, scores 9 technical indicators per timeframe with regime-adaptive weights, checks multi-timeframe agreement and news/calendar risk, grades the signal (A+…F), and generates a full trade setup (entry, SL, TP1/TP2, trailing plan). Delivery: Telegram bot (auto-scanner + subscriber channel) and a CLI. A realism-focused backtester replays history through the *exact same* signal code.
|
Signal-intelligence platform for **XAU/USD (gold — live, validated)** and **EUR/USD (functional but OFF by default — no validated edge, see below)**. It fetches OHLCV from Twelve Data, scores 9 technical indicators per timeframe with regime-adaptive weights, checks multi-timeframe agreement and news/calendar risk, grades the signal (A+…F), and generates a full trade setup (entry, SL, TP1/TP2, trailing plan). Delivery: Telegram bot (auto-scanner + subscriber channel) and a CLI. A realism-focused backtester replays history through the *exact same* signal code.
|
||||||
|
|
||||||
This file is the authoritative documentation. `DOCUMENTATION.md` and `USAGE.md` are stale (May 2026) — do not trust them. `PULLBACK_ENTRY_PLAN.txt` documents the XAU_PB pullback strategy design. `backtest/instructions_for_backtest.txt` lists what every backtest report must record.
|
This file is the authoritative documentation. `DOCUMENTATION.md` is stale (May 2026) — do not trust it. `backtest/instructions_for_backtest.txt` lists what every backtest report must record.
|
||||||
|
|
||||||
**Read this whole file before changing signal logic.** Every numeric threshold below was either backtested or is load-bearing for a backtested result. Rules for changes are at the bottom ("Rules for making changes").
|
**Read this whole file before changing signal logic.** Every numeric threshold below was either backtested or is load-bearing for a backtested result. Rules for changes are at the bottom ("Rules for making changes").
|
||||||
|
|
||||||
|
|||||||
@@ -1,112 +0,0 @@
|
|||||||
PULLBACK ENTRY PATTERN — IMPLEMENTATION PLAN
|
|
||||||
=============================================
|
|
||||||
Drafted 2026-07-09. Status: IMPLEMENTED & VALIDATED (same day).
|
|
||||||
|
|
||||||
OUTCOME: shipped as XAU_PB. 12mo M5 validation: 254 PB trades,
|
|
||||||
+38.3R, avgR +0.151, WR 58%, halves +20.1/+18.1, all four full
|
|
||||||
quarters positive. Core breakout trades undisturbed (-3.1R
|
|
||||||
displacement). Combined book: 549 trades (~10.6/wk), +140.1R,
|
|
||||||
PF 1.78, DD 11.3R. Calibration lesson: M5 gold trend legs measure
|
|
||||||
3-4 ATR swing-to-trough — the drafted 0.5-2.5 ATR retrace window
|
|
||||||
rejected 81% of valid pullbacks (fixed: 0.8-6.0). PB stops are
|
|
||||||
~$8 vs ~$25 core, so PB got its own trail arm (0.8R vs 0.6R).
|
|
||||||
Detector lives in signal_aggregator (NOT engine.py) so the GPU
|
|
||||||
backtest engine exercises the same code path as live.
|
|
||||||
|
|
||||||
WHY (evidence from 12mo M5 backtests, $0.30 spread, trailing exits)
|
|
||||||
-------------------------------------------------------------------
|
|
||||||
All 311 current trades are one archetype: trend-continuation breakout
|
|
||||||
(EMA 4/4 aligned + ADX>25 + MACD accelerating + price through Keltner
|
|
||||||
band). We systematically enter at maximum extension:
|
|
||||||
|
|
||||||
- Trades carrying the "NO PULLBACK — price >2.5 ATR from EMA21" warning
|
|
||||||
average +0.063R vs +0.112R without it (145 vs 103 trades, 6mo set).
|
|
||||||
- 28% of stopped-out trades never moved toward the target — instant
|
|
||||||
reversals from buying the local top / selling the local bottom.
|
|
||||||
- "Extended — RSI in trend" warning trades: +0.057R vs +0.183R without.
|
|
||||||
|
|
||||||
A pullback entry catches the SAME trends at better prices: enter on the
|
|
||||||
retrace toward EMA21 / Keltner midline, stop below the retrace swing
|
|
||||||
instead of 2×ATR behind an extended price. Tighter risk, same trend.
|
|
||||||
This ADDS trades (new signal source) — it does not filter existing ones.
|
|
||||||
|
|
||||||
DETECTOR SPEC (engine.py — new module-level check in M5 path)
|
|
||||||
-------------------------------------------------------------
|
|
||||||
Trend context (all required):
|
|
||||||
1. EMA alignment >= 3/4 in trend direction (allow 9<21 wobble mid-pullback)
|
|
||||||
2. ADX >= 20 on M5 (relaxed vs breakout's 25: ADX decays during retrace)
|
|
||||||
3. H1 confirmation score agrees with trend direction (>= +10 / <= -10),
|
|
||||||
reuse Gate-3 machinery.
|
|
||||||
|
|
||||||
Pullback trigger (all required):
|
|
||||||
4. Price touched/undercut EMA21 or KC midline within last 3 bars
|
|
||||||
(low <= band for BUY; high >= band for SELL).
|
|
||||||
5. Retrace depth 0.5–2.5 ATR from the pre-pullback swing extreme
|
|
||||||
(deeper = trend likely broken; shallower = still extended).
|
|
||||||
6. Resumption bar: close back in trend direction (close > open for BUY)
|
|
||||||
AND close above EMA9 (below for SELL) — do not catch the falling knife.
|
|
||||||
7. RSI in the pullback zone: BUY 35–55, SELL 45–65 (NOT the extended
|
|
||||||
zones the breakout entries fire in — that is the whole point).
|
|
||||||
|
|
||||||
Suppression:
|
|
||||||
8. Skip if a breakout signal fired within cooldown (no double entry
|
|
||||||
on the same leg — cd2/cd3 tests proved same-direction pyramiding
|
|
||||||
is train-half-only noise).
|
|
||||||
9. All existing M5 gates 1–6 still apply where meaningful; Gate 1
|
|
||||||
(|score|>=30) will need a variant — pullback bars have weaker
|
|
||||||
composite scores BY DESIGN (price below the momentum bands), so
|
|
||||||
validate a separate m5_pullback_min_score knob (start ~20).
|
|
||||||
|
|
||||||
GEOMETRY
|
|
||||||
--------
|
|
||||||
- SL: below the pullback swing low (BUY) − 0.5×ATR buffer, floored at
|
|
||||||
min_sl_pips ($8). Expect typical SL 40–60% tighter than breakout SLs.
|
|
||||||
- Exit: same trailing stop (arm +0.6R, trail 0.75R, max_hold 300).
|
|
||||||
NOTE: tighter SL means R units are smaller in dollars — re-validate
|
|
||||||
trail params on pullback trades separately; the arm/dist plateau was
|
|
||||||
measured on breakout-sized risk. If needed give pullback its own
|
|
||||||
trail knobs.
|
|
||||||
- Tag trades strategy="XAU_PB" in the backtester so every report
|
|
||||||
splits breakout vs pullback cleanly.
|
|
||||||
|
|
||||||
VALIDATION PROTOCOL (same discipline as 2026-07-09 sessions)
|
|
||||||
------------------------------------------------------------
|
|
||||||
1. Implement detector behind config flag m5_pullback_enabled=False.
|
|
||||||
2. 365d backtest, flag on; export CSV; analyze XAU_PB trades alone:
|
|
||||||
- REQUIRE: positive totalR in BOTH data halves
|
|
||||||
- REQUIRE: positive in >= 3 of 4 full quarters
|
|
||||||
- REQUIRE: does not degrade breakout-trade totals (displacement
|
|
||||||
check vs baseline entry_time set, like the cooldown analysis)
|
|
||||||
- TARGET: >= +0.15R/trade average, >= 1 trade/week added
|
|
||||||
3. Threshold sweeps (retrace depth, RSI zone, ADX floor) via the
|
|
||||||
offline exit-sim harness pattern where possible; full reruns where
|
|
||||||
entries change.
|
|
||||||
4. Reject-and-document if it fails — same as blocked-hours/cooldown
|
|
||||||
experiments (see memory: entry-side experiments rejected).
|
|
||||||
|
|
||||||
BASELINE TO BEAT (do not regress)
|
|
||||||
---------------------------------
|
|
||||||
12mo M5: 311 trades, +105.0R, PF 2.27, DD 7.9R, WR 61.4%,
|
|
||||||
train/val +54.6/+50.4, every quarter positive, ~6-8 trades/week.
|
|
||||||
(commit 987ec5b, CSV: m5_365d_trail300 in session scratchpad)
|
|
||||||
|
|
||||||
IMPLEMENTATION ORDER
|
|
||||||
--------------------
|
|
||||||
1. config.py: m5_pullback_enabled, m5_pullback_min_score,
|
|
||||||
pullback RSI zones, retrace depth bounds.
|
|
||||||
2. engine.py: pullback detector producing a candidate flag + swing
|
|
||||||
levels (attach as result._pullback_setup, underscore-private).
|
|
||||||
3. signal_aggregator.py: consume flag, build XAU_PB trade setup with
|
|
||||||
pullback geometry; route through gates with pullback score variant.
|
|
||||||
4. gpu_backtest.py: ensure strategy tag flows to CSV (already does via
|
|
||||||
strategy_tag) and pre-filter loop does not drop pullback candidates.
|
|
||||||
5. Backtest + validate per protocol; tune; commit only on pass.
|
|
||||||
|
|
||||||
RISKS
|
|
||||||
-----
|
|
||||||
- Pullback entries in a trend that IS reversing = catching the knife;
|
|
||||||
the resumption-bar condition (6) is the main guard — do not weaken it.
|
|
||||||
- Score-gate variant (9) is the most likely source of overfitting;
|
|
||||||
sweep it last, keep the grid coarse.
|
|
||||||
- M5 pullbacks during news spikes: calendar buffer already applies via
|
|
||||||
existing pipeline; verify it fires for XAU_PB too.
|
|
||||||
@@ -1,198 +0,0 @@
|
|||||||
# ForexBot2 — USD/JPY Signal Intelligence
|
|
||||||
|
|
||||||
A focused USD/JPY signal generator + walk-forward backtester. Live signals
|
|
||||||
and backtests run through the **same** technical engine, so what you measure
|
|
||||||
in a backtest is exactly what you'd execute live.
|
|
||||||
|
|
||||||
## Setup
|
|
||||||
|
|
||||||
```bash
|
|
||||||
pip install -r requirements.txt
|
|
||||||
```
|
|
||||||
|
|
||||||
API key (optional — a default key is in `config.py`):
|
|
||||||
|
|
||||||
```bash
|
|
||||||
export TWELVE_DATA_API_KEY=your_key_here # Linux/Mac
|
|
||||||
$env:TWELVE_DATA_API_KEY="your_key_here" # PowerShell
|
|
||||||
```
|
|
||||||
|
|
||||||
---
|
|
||||||
|
|
||||||
## Live Signal
|
|
||||||
|
|
||||||
```bash
|
|
||||||
python main.py # H1, summary view
|
|
||||||
python main.py --detail # Full indicator breakdown
|
|
||||||
python main.py --timeframe H4 --detail # H4 instead of H1
|
|
||||||
python main.py --json # JSON output (for piping)
|
|
||||||
```
|
|
||||||
|
|
||||||
| Option | Default | Description |
|
|
||||||
|---------------|---------|--------------------------------------------|
|
|
||||||
| `--timeframe` | `H1` | `M5 M15 M30 H1 H4 D1` |
|
|
||||||
| `--detail` | off | Indicator breakdown, key levels, reasons |
|
|
||||||
| `--json` | off | Machine-readable output |
|
|
||||||
| `--verbose` | off | Debug logging |
|
|
||||||
|
|
||||||
### Output
|
|
||||||
|
|
||||||
```
|
|
||||||
USD/JPY │ Price: 149.230 │ Regime: trending_up │ Vol: normal
|
|
||||||
|
|
||||||
SIGNAL: BUY │ Score: +28.4 │ Grade: B │ Confidence: 62%
|
|
||||||
Action: BUY — Decent setup, wait for confirmation
|
|
||||||
|
|
||||||
Multi-Timeframe: ✓ ALIGNED
|
|
||||||
M15: BUY (+18.2)
|
|
||||||
H1: BUY (+28.4)
|
|
||||||
H4: STRONG BUY (+45.1)
|
|
||||||
|
|
||||||
Trade Setup:
|
|
||||||
Direction: BUY
|
|
||||||
Entry: 149.230
|
|
||||||
Stop Loss: 148.960 (27.0 pips)
|
|
||||||
Take Profit 1: 149.770 (54.0 pips)
|
|
||||||
Take Profit 2: 150.040
|
|
||||||
Risk:Reward: 1:2.0
|
|
||||||
```
|
|
||||||
|
|
||||||
---
|
|
||||||
|
|
||||||
## Backtest (Walk-Forward)
|
|
||||||
|
|
||||||
The backtester replays USD/JPY history bar-by-bar through the live signal
|
|
||||||
pipeline. No lookahead, no parallel implementation — what passes in
|
|
||||||
backtest passes live.
|
|
||||||
|
|
||||||
```bash
|
|
||||||
python main.py --backtest # Full available history
|
|
||||||
python main.py --backtest --bars 2000 # Last 2000 H1 bars
|
|
||||||
python main.py --backtest --start 2024-01-01 --end 2024-12-31
|
|
||||||
python main.py --backtest --timeframe H4 # Backtest on H4
|
|
||||||
python main.py --backtest --min-grade A # Only A/A+ entries
|
|
||||||
python main.py --backtest --journal # Last 10 trades printed
|
|
||||||
python main.py --backtest --journal-csv trades.csv # Full journal to CSV
|
|
||||||
python main.py --backtest --regime-filter # Skip choppy/ranging bars
|
|
||||||
python main.py --backtest --regime-filter --regime-min-adx 35
|
|
||||||
python main.py --backtest --session-filter # Trade only 07-16 UTC
|
|
||||||
python main.py --backtest --session-filter --session-start 8 --session-end 17
|
|
||||||
python main.py --backtest --auto-session-filter # Reactive session toggle
|
|
||||||
python main.py --backtest --auto-session-filter --auto-session-lookback 15 \
|
|
||||||
--auto-session-threshold-pct -0.5
|
|
||||||
```
|
|
||||||
|
|
||||||
### Regime filter
|
|
||||||
|
|
||||||
The engine is a confluence-based trend-follower — it makes money in trending
|
|
||||||
regimes and loses in chop. `--regime-filter` gates entries on H4 ADX:
|
|
||||||
signals only fire when ADX ≥ `--regime-min-adx` (default 30). Sweet spot is
|
|
||||||
ADX 30–35; tighter cuts trades faster than it improves quality.
|
|
||||||
|
|
||||||
### Session filter
|
|
||||||
|
|
||||||
`--session-filter` restricts entries to a UTC hour window (default 07-16,
|
|
||||||
covering London + NY). Diagnostic on 2025-11 → 2026-05 showed 47% WR /
|
|
||||||
+127 pips inside that window vs 0% WR / -571 pips outside it. Override
|
|
||||||
bounds with `--session-start` (inclusive) and `--session-end` (exclusive).
|
|
||||||
|
|
||||||
Tradeoff: it's regime-specific medicine. In trending regimes (e.g. 2024)
|
|
||||||
it cuts winning off-hours trades and gives back ~33% of the edge. Ships
|
|
||||||
**default off** because the pure regime filter still wins cumulative
|
|
||||||
across the 2024 / 2023 / 2026 windows.
|
|
||||||
|
|
||||||
### Auto session toggle
|
|
||||||
|
|
||||||
`--auto-session-filter` is a reactive variant: it tracks the rolling mean
|
|
||||||
`pnl_pct` over the last N closed trades (`--auto-session-lookback`,
|
|
||||||
default 10) and activates the 07-16 UTC window only when that mean drops
|
|
||||||
below `--auto-session-threshold-pct` (default `-0.3`). Releases
|
|
||||||
automatically once expectancy recovers.
|
|
||||||
|
|
||||||
Less aggressive than the manual flag — preserves more trend edge but
|
|
||||||
reacts slowly because the deque must fill with losses before kicking in.
|
|
||||||
Use this when you don't want to manually toggle the regime medicine.
|
|
||||||
|
|
||||||
### Execution model
|
|
||||||
|
|
||||||
| Aspect | Behavior |
|
|
||||||
|--------------------|-------------------------------------------------------------------|
|
|
||||||
| Entry | Next bar's open (no current-bar lookahead) |
|
|
||||||
| Stop Loss | 1.5 × ATR from entry (configurable) |
|
|
||||||
| Take Profit | RR × stop distance (default 1:2) |
|
|
||||||
| Same-bar SL & TP | Conservative: SL fills first |
|
|
||||||
| Spread cost | Half-spread on entry + half-spread on exit (full round-trip) |
|
|
||||||
| Slippage | 0.5 pips per fill (entry + exit) |
|
|
||||||
| Position size | 1% of current equity risked per trade |
|
|
||||||
| Concurrency | One open position at a time |
|
|
||||||
| News in backtest | Disabled — RSS/calendar can't be replayed without lookahead |
|
|
||||||
|
|
||||||
### Output
|
|
||||||
|
|
||||||
```
|
|
||||||
Backtest Summary — USD/JPY
|
|
||||||
─────────────────────────────────────────────────
|
|
||||||
Trades: 87 (52W / 35L)
|
|
||||||
Win Rate: 59.8%
|
|
||||||
Profit Factor: 1.84
|
|
||||||
Expectancy: +12.4 pips/trade (+12.42 per trade)
|
|
||||||
Initial Balance: $10,000.00
|
|
||||||
Final Balance: $11,084.00
|
|
||||||
Total Return: +10.84%
|
|
||||||
CAGR: +18.92%
|
|
||||||
Max Drawdown: -4.21%
|
|
||||||
Sharpe Ratio: 1.42
|
|
||||||
Sortino Ratio: 2.18
|
|
||||||
Avg Duration: 14.3 bars
|
|
||||||
Win Streak: 7 | Loss Streak: 4
|
|
||||||
─────────────────────────────────────────────────
|
|
||||||
```
|
|
||||||
|
|
||||||
---
|
|
||||||
|
|
||||||
## Parameter Optimization (parallel)
|
|
||||||
|
|
||||||
```bash
|
|
||||||
python main.py --backtest --optimize
|
|
||||||
```
|
|
||||||
|
|
||||||
Runs a 3×3×3 grid (27 combos) over `atr_sl_multiplier`, `default_rr_ratio`,
|
|
||||||
and `min_score` (signal-score threshold). Combos run in parallel across all
|
|
||||||
CPU cores via `multiprocessing.Pool`. Twelve Data is hit once — workers
|
|
||||||
share an on-disk parquet cache (`.cache/usdjpy_<tf>.parquet`).
|
|
||||||
|
|
||||||
Output ranks combos by total return:
|
|
||||||
|
|
||||||
```
|
|
||||||
Top 10 Parameter Combinations (by total return):
|
|
||||||
|
|
||||||
# Return% PF WinR Trades MaxDD% Sharpe Params
|
|
||||||
──────────────────────────────────────────────────────────────────────
|
|
||||||
1 +14.21% 2.10 62.5% 64 -3.84 1.78 atr_sl_multiplier=1.5, default_rr_ratio=2.5, min_score=25.0
|
|
||||||
2 +11.93% 1.92 58.7% 71 -4.11 1.55 atr_sl_multiplier=1.0, default_rr_ratio=2.0, min_score=15.0
|
|
||||||
...
|
|
||||||
```
|
|
||||||
|
|
||||||
Customize the grid in `config.py → BacktestConfig.param_grid`.
|
|
||||||
|
|
||||||
---
|
|
||||||
|
|
||||||
## Files
|
|
||||||
|
|
||||||
| File | Role |
|
|
||||||
|-----------------------|---------------------------------------------------|
|
|
||||||
| `main.py` | CLI entry point (live + backtest + optimize) |
|
|
||||||
| `config.py` | All parameters in one place |
|
|
||||||
| `data_provider.py` | Twelve Data fetcher + parquet cache |
|
|
||||||
| `engine.py` | Technical indicators (8 + divergence + patterns) |
|
|
||||||
| `signal_aggregator.py`| Multi-TF aggregation, grading, trade setup |
|
|
||||||
| `news_engine.py` | RSS sentiment + economic calendar (live only) |
|
|
||||||
| `backtest.py` | Walk-forward replay + parallel optimizer |
|
|
||||||
| `metrics.py` | Win rate, PF, Sharpe, max DD, trade journal |
|
|
||||||
|
|
||||||
---
|
|
||||||
|
|
||||||
## Disclaimer
|
|
||||||
|
|
||||||
For educational and research use. Past performance does not guarantee
|
|
||||||
future results. Always use proper risk management.
|
|
||||||
@@ -251,7 +251,7 @@ class XauStrategyConfig:
|
|||||||
# halves of 6mo and 12mo backtests).
|
# halves of 6mo and 12mo backtests).
|
||||||
m5_sell_min_rsi: float = 25.0
|
m5_sell_min_rsi: float = 25.0
|
||||||
|
|
||||||
# ── M5 pullback entry (XAU_PB — see PULLBACK_ENTRY_PLAN.txt) ──
|
# ── M5 pullback entry (XAU_PB) ──
|
||||||
# Second entry archetype: enter on the retrace to EMA21/KC-mid in
|
# Second entry archetype: enter on the retrace to EMA21/KC-mid in
|
||||||
# an established trend instead of at Keltner-breakout extension.
|
# an established trend instead of at Keltner-breakout extension.
|
||||||
# Validated 2026-07: +38.3R/254 trades on 12mo, positive both
|
# Validated 2026-07: +38.3R/254 trades on 12mo, positive both
|
||||||
|
|||||||
Reference in New Issue
Block a user