ForexBot2/CLAUDE.md

ForexBot2 — Codebase Reference

Signal-intelligence platform for XAU/USD (gold — live, validated). It fetches OHLCV from Twelve Data, scores 9 technical indicators per timeframe with regime-adaptive weights, checks multi-timeframe agreement and news/calendar risk, grades the signal (A+…F), and generates a full trade setup (entry, SL, TP1/TP2, trailing plan). Delivery: Telegram bot (auto-scanner + subscriber channel) and a CLI. A realism-focused backtester replays history through the exact same signal code.

This file is the authoritative developer/agent reference. The human-facing docs live in docs/ (user_guide.html + beginner_guide.html, and their PDF builds) — keep them in sync when signal behavior changes. backtest/instructions_for_backtest.txt lists what every backtest report must record.

Read this whole file before changing signal logic. Every numeric threshold below was either backtested or is load-bearing for a backtested result. Rules for changes are at the bottom ("Rules for making changes").

---

Repository map

Path	Lines	Purpose
config.py	~374	All config dataclasses, PAIR_INFO registry, per-pair strategy configs. PlatformConfig defaults ARE the tuned XAU config.
app_config.py	~244	config.json loader/validation, env-precedence resolvers, apply_pair_overrides deployment overlay
config.json.example	24	committed schema template; real gitignored config.json holds secrets + deployment settings
data_provider.py	~347	Twelve Data SDK wrapper: fetch, rate-limit (55 credits/min), in-memory cache, validation/cleaning, freshness checks
engine.py	~1173	TechnicalEngine: computes all indicators on one DataFrame → TechnicalAnalysis (composite score, regime, key levels)
chart_patterns.py	~280	Chart-pattern detectors for the XAU_CP archetype (pure functions, ATR-unit geometry, first-close-beyond triggers)
signal_aggregator.py	~1447	SignalAggregator: MTF analysis, news integration, gates/penalties, quality grade, trade-setup generation → SignalReport
news_engine.py	~680	Economic calendar (Forex Factory JSON + MyFXBook RSS fallback) + keyword RSS sentiment
telegram_bot.py	~1259	Async Telegram bot: candle-boundary auto-scanner, WebSocket level-breach monitor, owner lock, subscriber channel
pair_signals/__init__.py	52	PAIR_PRODUCERS registry + produce_signal()
pair_signals/xauusd.py	71	XAU config factory + per-timeframe overrides (docstring = validation record)
main.py	~348	CLI: python main.py --pair XAUUSD --timeframe H1 --detail/--json
test_all.py	~2800	QA suite: 199 tests in 13 modules, no live API needed. Must stay green.
backtest/gpu_backtest.py	~2231	Bar-replay backtester (CuPy→NumPy fallback) using the real SignalAggregator
backtest/diagnose_filters.py	609	Filter-diagnosis tooling
backtest/research/	—	Calendar research infra (ff_calendar_scraper.py, calendar_store.py) + result CSVs
backtest/*.csv	—	Verification trade books from past tuning campaigns (xau_*.csv, …). Gitignored, local-only — used for manual before/after comparison, not by tests
backtest/cache/	—	Parquet OHLCV cache (raw bars)
logs/telegram.log	—	Bot log (midnight rotation, 30 backups)

---

Pair status

• XAU/USD: the tuned, validated product — and the ONLY supported pair. All defaults in PlatformConfig/XauStrategyConfig are its tuned values.
• EUR/USD: REMOVED from the codebase (2026-07-16, branch xau-only-chart-patterns). It never had a validated edge: the 2026-07-11/12 verdict found no configuration passing validation on M5, M15, or H1 at a 1-pip round-trip cost; root cause was cost-to-volatility (EUR M15 ATR ≈ 10 pips → 1-pip spread taxes ~10% of ATR; gold's ATR/spread ratio is several times larger). Calendar-surprise and H1-hybrid avenues were also tested and closed. The full EUR implementation and verdict history live in git history (last commit with EUR support: 7d3fd95). Do not re-add EUR without new data (tick fills or H4/D1 horizons).

---

End-to-end pipeline (what happens on every scan)

Twelve Data REST (+ WebSocket ticks in the bot)
  → data_provider.fetch_ohlcv()        per timeframe: throttle, fetch, drop forming candle, validate/clean, cache
  → engine.TechnicalEngine.analyze()   per timeframe: 9 scored indicators + ATR context → regime → composite score
  → signal_aggregator.analyze_pair()   live entry point (weekend/day filters, fetch, staleness)
      └─ analyze_from_dataframes()     replayable core: MTF agreement → news → bonuses/penalties →
                                       final signal + quality grade → gates (session, SELL, M5, regime) →
                                       trade setup (3 generators) → reasons
  → SignalReport                       consumed by telegram_bot.py, main.py, backtest/gpu_backtest.py

Signal currency everywhere is the SignalStrength enum: STRONG_BUY=2, BUY=1, NEUTRAL=0, SELL=-1, STRONG_SELL=-2.

---

config.py — every knob

PAIR_INFO (market constants; key is the canonical symbol "XAU/USD")

Field	XAU/USD	Meaning
pip_value	0.10	price units per pip
price_precision	2	decimals for SL/TP rounding & display
default_spread	0.30	round-trip cost charged by the backtester when --spread not given

TechnicalConfig (indicator parameters)

EMA 9/21/50/200 (ema_mode: "4ema" default or "2ema" = 50/200 only); RSI 14 (OB 70 / OS 30 / neutral zone 40–60; rsi_mode: "regime_aware" default or "classic"); MACD 12/26/9 (macd_as_pillar=True — MACD is a scored indicator, not just divergence input); BB 20/2.0σ; ATR 14; ADX 14 (strong ≥25, very strong ≥50); divergence lookback 40; swing-S/R lookback 50; ROC period 10, threshold 1.0%; Keltner: EMA 20 mid ± 1.5 × ATR(10); ADX slope lookback 5, "emerging" threshold 3.0; disabled_indicators list (removes the pillar from scoring; for MACD the histogram is still computed — divergence needs it); regime_weights_override dict (replaces the gold regime-multiplier table for experiments).

SignalConfig (scoring & trade geometry — XAU-calibrated)

Field	Default	Meaning
mtf_agreement_threshold	2	timeframes (of primary + confirmations) that must agree
min_rr_ratio	0.5	reject setups below this RR
tp_tightening_factor	1.0	TP scale factor. Leave at 1.0: 0.8 was re-tested 2026-07-08 with spread+gap fills and nearly halved total R
round_trip_cost	0.30	XAU $ round trip
min_tp_cost_multiple	30.0	spread gate: TP1 distance must be ≥ cost × this (= $9 on XAU). Validated: sub-$10 targets lose
min_sl_pips	80.0	SL floor in pips ($8 on XAU)
tp_floor_rr	0.5	TP1 floor as fraction of SL distance (M15 factory overrides to 1.0)
trail_arm_r / trail_dist_r	0.6 / 0.75	trailing exit: once favorable excursion ≥ arm×risk, stop trails dist×risk behind best price (bar-close updates). TP1 becomes advisory. Validated on 12mo M5: +105R vs +48R static. 0 disables
pb_trail_arm_r / pb_trail_dist_r	0.8 / 0.75	trailing params for XAU_PB pullback trades (tighter stops → arm later)

NewsConfig

Tracked currencies ["USD","XAU"]; pre-event buffer 30 min, post-event 15 min (45 min after a surprise print); event_risk_caps_signals=False — XAU default: event risk warns but does NOT cap confidence (2026-07-11 audit: the caps suppressed trades worth +4.6..+7.2R/yr; XAU trades near USD releases outperform); sentiment_affects_signals=False — headline sentiment is informational only (warnings still emitted; the +5 alignment bonus and 10-pt news_conflict penalty are gated off, so live decisions are pure technical, matching backtests). Context cache TTL 600 s (must stay < pre-event buffer). 4 RSS feeds (investinglive, FXStreet, Google-News gold query, CNBC economy).

XauStrategyConfig (all XAU gates; enabled=True)

Field	Default	Notes
blocked_hours_utc	[0,1,2,10,11,13,16,21]	bar-close hours never traded. M5 factory appends 9 (post-London-open whipsaw, −28.9R/143 over 6 yr)
allowed_weekdays	[0,1,2,3,4,6]	Mon–Fri + Sun (Sydney open). weekend_filter=True: Fri 22:00 → Sun 22:00 UTC closed
min_atr_threshold	1.0	$1 ATR floor for swing-based setups
v2_sl_atr_multiplier	2.0	SL ATR floor in the generic setup generator
sl_atr_multiplier / tp_atr_multiplier	2.0 / 1.0	used by the XAU ATR setup generator (A+/A only)
sell_min_quality	["A+","A","B"]	SELLs below B are blocked
sell_vol_regimes	["high","extreme"]	SELLs only in these volatility regimes (quiet shorts fade vs gold's macro uptrend)
blocked_regimes / sell_blocked_regimes	[] / []	per-TF factory opt-ins. M15 factory sets blocked_regimes=["trending"] (mixed regime: strong ADX, EMAs unaligned — lost −6.8R/24)
sell_blocked_hours_utc	[]	bar-CLOSE UTC hours where SELLs are blocked (BUYs unaffected). Per-TF factory opt-in: M15 sets [15, 17] (NY-session shorts, −33.5R/165 over 5 yr, negative 6/6 years both halves; BUYs there +34.3R)
M5 gates		m5_require_mtf=True; m5_min_abs_score=30; m5_min_confirmation_score=10; m5_sell_max_quality=["B","C"] (A-grade M5 SELLs enter at dip extremes and lose); m5_blocked_vol_regimes=["high"] (chop transition — 61% WR, negative total); m5_sell_min_rsi=25
M5 pullback (XAU_PB)		m5_pullback_enabled=True; min score 15 (lower than gate 1 BY DESIGN — mid-pullback scores weaker); ADX ≥ 20; RSI zones BUY (35,62) / SELL (38,65); retrace depth 0.8–6.0 ATR from the 40-bar swing extreme; band touch within 3 bars. Validated +38.3R/254 trades on 12mo
Chart patterns (XAU_CP)	cp_enabled=False	cp_* fields: pattern list, lookback 72, swing window 3, recency 20, double tolerance 0.5 ATR, height 1.5–8 ATR, H&S head ≥0.5 ATR / shoulder tol 1.0 ATR, flag pole ≥3 ATR in ≤12 bars, flag 3–12 bars retracing 10–50%. Textbook defaults, NOT tuned. Default OFF — M5 factory enables BUY-only (inv_hs, double_bottom, bull_flag); M15/H1/H4 remain disabled (all failed OOS validation 2026-07-16). Bearish patterns (hs_top, double_top, bear_flag) dropped: SELL CP lost −3.21R on OOS, −4.52R on M15 5yr

FilterConfig (config.backtest — despite the name it applies LIVE too)

session_filter=True, window 03:00–23:00 UTC, blocked_hours_utc mirrors XAU's list (non-XAU pairs read this copy), allowed_weekdays Mon–Fri+Sun. regime_filter=False (optional H4-ADX≥25 gate, off).

PlatformConfig (top level)

symbol="XAU/USD", primary TF H1, confirmations [M15, H4], data_lookback_days=60, scan_pairs=[XAUUSD] (mirrors config.json scanning.pairs — see below), scan_timeframes=[M5,M15,H1,H4], ws_cooldown_seconds=300.

Per-pair base blocked hours plus scan_pairs/scan_timeframes overlay from config.json via app_config.apply_pair_overrides, applied in the pair factory BEFORE per-TF deltas. Credentials, alert quality, and log level never touch PlatformConfig — telegram_bot.py and data_provider.py read them directly from app_config.get_app_config() and its resolvers. scanning.pairs in config.json is the deployment switch for what the bot scans — PlatformConfig.scan_pairs just carries the resolved value.

---

engine.py — TechnicalEngine

Input contract: pandas DataFrame, UTC tz-aware DatetimeIndex, oldest-first, columns Open High Low Close Volume (capitalized). < 30 bars → empty NEUTRAL result. EMA200 needs ≥ 200 bars (otherwise EMA-trend indicator degrades gracefully).

Compute order matters: ATR first (EMA-ribbon and candlestick proximity use it) → EMA → RSI → BB → ADX → MACD (always computed, even if not a pillar — divergence needs the histogram) → ROC → Keltner (needs BB levels + MACD histogram) → divergence → structure S/R → candlesticks → regime classification → regime-aware RSI reinterpretation → composite.

Results land in TechnicalAnalysis.indicators: Dict[str, IndicatorSignal] (fields: name, value, signal, confidence 0-100, description, weight). Private series ride on the result object as underscore attrs (_rsi_series, _macd_histogram, _adx_slope, _kc_squeeze, _kc_squeeze_released) and are skipped by the composite scorer and all formatters (skip rule: name starts with _ or weight == 0).

The 9 scored indicators (+ ATR as context)

Key	Base weight	Signal logic (exact)
ema_trend	1.0	4-EMA mode: alignment score over checks (price>EMA9, 9>21, 21>50, 50>200), normalized to [-1,1]. >0.6 STRONG_BUY (conf ≤90), >0.2 BUY, mirrored sells, else NEUTRAL (40). Fast/medium cross ±10 conf. Ribbon (9/21/50 spread < 1 ATR) = "COMPRESSED"; compressed→expanded = +10 conf. Emits ema_fast/medium/slow/trend key levels
rsi	1.5	Wilder RSI (ewm α=1/14). Classic pass first: ≥70 SELL, ≤30 BUY, ≥60 BUY-momentum (45), ≤40 SELL-momentum (45), else NEUTRAL (30). Then overwritten by regime reinterpretation (below) when rsi_mode="regime_aware"
macd	1.5	Histogram = MACD line − signal line. Cross above 0 → BUY (conf 55+); accelerating (hist>prev>0) → STRONG_BUY; mirrored down; positive-but-fading / negative-but-fading → NEUTRAL (35); flat → NEUTRAL (30). Confidence scaled by
bollinger	1.5	Position = (close−lower)/(upper−lower). >1.0 SELL (65), <0.0 BUY (65), >0.8 SELL (45), <0.2 BUY (45), else NEUTRAL (30). Squeeze = width < 0.75 × 20-bar avg width → +10 conf. Emits bb_upper/lower/mid
adx	0.5	Wilder ADX/DI (ties zero BOTH DMs — fixes a systematic bearish nudge). ADX ≥ 25 → BUY/SELL by DI dominance, else NEUTRAL. Conf 40/65/80 by ADX band. ADX slope (5-bar diff): <25 with slope > 3.0 → "TREND EMERGING" +15 conf; also strengthening/weakening adjustments. Sets _adx_slope
roc	1.5	10-period %change. Positive & rising → BUY (STRONG if >2× threshold=2%); negative & falling → SELL; zero-cross → BUY/SELL (65); else NEUTRAL. Directionally redundant with MACD/Keltner/EMA (0.8–0.94 corr) but its weight is load-bearing: zeroing it backtested 13R WORSE — composite thresholds are calibrated with it. Do not remove without retuning thresholds
keltner	1.5	EMA20 mid ± 1.5×ATR(10). TTM squeeze = BB fully inside KC. Squeeze released (was→not) + MACD hist sign → STRONG_BUY/STRONG_SELL (80); squeeze active → NEUTRAL (60); close above/below channel → BUY/SELL continuation (65); inside → NEUTRAL (30). Emits kc_upper/lower/mid
divergence	1.5	RSI + MACD-hist vs price over 40 bars. Swing points (window 3) must align within ±2 bars to pair up; last two pairs: price LL + indicator HL = bullish, price HH + indicator LH = bearish. Any found → BUY/SELL/NEUTRAL-if-both, conf 70, value = count
candlestick	1.0	Last bar: Doji (body<10% range), Hammer/Hanging Man (lower wick>2×body), Shooting Star/Inverted Hammer, Engulfing (vs prev bar), Marubozu (body>85%). Majority vote → signal, conf 50; +20 if pattern sits within 0.5 ATR of a swing S/R level
atr	0.0 (context)	Wilder ATR. Volatility regime from percentile vs last 50 ATR values: >80 extreme, >60 high, >30 normal, else low. Emits atr, atr_pct

Structure S/R (not scored): swing highs/lows (window 3) over swing_sr_lookback (min 120 bars on 1min/5min TFs); up to 3 above and 3 below stored as swing_high_1..3 / swing_low_1..3 key levels.

Market regime classification (exact rules)

• ADX ≥ 25 and EMA-trend signal bullish → trending_up; bearish → trending_down; EMA mixed → trending (the "mixed" regime — momentum without directional structure; treated as direction-UNKNOWN everywhere; a past bullish-bias bug here cost −6.8R and is fixed)
• else ADX slope > 3.0 → transitioning (trend birth, ~5–10 bars before ADX crosses 25)
• else → ranging

Regime-aware RSI reinterpretation (rsi_mode="regime_aware", the default — classic mode backtested worse for gold)

Regime	RSI zones
trending_up	38–55 STRONG_BUY 75 ("pullback zone, ideal entry"); 30–38 BUY 60; 55–65 BUY 50; >65 NEUTRAL ("extended, wait"); <30 NEUTRAL ("possible reversal")
trending_down	mirrored: 45–62 STRONG_SELL 75; 62–70 SELL 60; 35–45 SELL 50; <35 / >70 NEUTRAL
transitioning / trending (mixed)	ONLY extremes: ≥70 SELL 50, ≤30 BUY 50, else NEUTRAL 30
ranging	≥70 SELL 70, ≤30 BUY 70, ≥60 SELL 45, ≤40 BUY 45, else NEUTRAL

Composite score

For each scored indicator: effective_weight = base_weight × regime_multiplier; contribution = signal.value × effective_weight × confidence/100. Composite = (Σ contributions / Σ effective_weights) × 50, clamped to ±100. Confidence = weight-averaged indicator confidence. Composite signal: ≥40 STRONG_BUY, ≥15 BUY, ≤−40 STRONG_SELL, ≤−15 SELL, else NEUTRAL.

Gold regime multipliers (defaults in _compute_composite; technical.regime_weights_override replaces the whole dict):

Indicator	trending_up/down/trending	ranging	transitioning
ema_trend	2.0	0.3	1.5
rsi	1.0	1.5	1.0
macd	2.0	1.5	2.5
bollinger	0.5	1.5	1.0
keltner	1.5	2.0	2.5
roc	1.5	0.5	2.0
candlestick	0.8	1.0	1.0
divergence	1.0	2.5	2.0
adx	0.8	0.5	1.0

---

signal_aggregator.py — SignalAggregator

Two entry points:

• analyze_pair(symbol) — LIVE: applies XAU weekend + allowed-weekday filters on wall clock, fetches all TFs (primary + confirmations), flags stale data (primary stale → signal suppressed to NEUTRAL/F), delegates to analyze_from_dataframes, then applies the live session check on the bar-CLOSE time (bar timestamp + candle duration): outside 03:00–23:00 or in blocked hours → NO TRADE. This is the exact predicate the backtester replays, so live == backtest.
• analyze_from_dataframes(symbol, dfs, context, use_news) — replayable core; treats each df as the visible history at decision time (caller must slice; no weekend filter here — the backtester applies it).

Pipeline inside analyze_from_dataframes (numbered as in the source)

1. Primary TF analysis → seeds report (score, confidence, regime, vol regime, price, timestamp = last bar's index label).
2. Confirmation TF analyses → mtf_signals / mtf_scores keyed by TF string ("5min", "1h", …).
3. MTF agreement: count BUY-ish vs SELL-ish across ALL TFs (primary included); agreement = count ≥ 2 in the primary's direction (two bullish confirms cannot "confirm" a bearish primary).
4. News (if use_news): event risk (clear/caution/danger) + pair sentiment (−1..+1). Backtests pass use_news=False.
5. Spread estimate = median(High−Low) of primary df.
6. _compute_final_signal (bonuses → event cap → penalties → thresholds → grade):
   • Bonuses: MTF agreement → +10 conf and score = 0.6×primary + 0.4×mean(all TF scores); news sentiment aligned (|sent|>0.2) → +5 (gated by news.sentiment_affects_signals, False by default); in a directional trend, RSI in pullback zone → +5, BB position favorable → +5 (mixed trending regime gets NO directional bonus).
   • Event risk danger: warning always; confidence hard-capped at 30 ONLY if news.event_risk_caps_signals (False by default — XAU-tuned).
   • Penalty system (diminishing, replaces old additive stacking): collect, sort by size desc, apply with decay ×1.0, ×0.5, ×0.25…, total capped at 30. Penalties: mtf_disagreement 15; news_conflict 10 (gated by sentiment_affects_signals, off by default — warning still emitted); event_caution 15 (config-gated); counter_trend 15; extreme_vol 10; ema200_near 5 (price within 0.5% of EMA200 — env FB2_EMA_LO overrides); ema200_far 5 (>2.5%); exhaustion 8 (price >3.5 ATR beyond nearest swing); no_pullback 8 (>2.5 ATR from EMA21 in trend) / weak_pullback 5 (>2.0); rsi_extended 5 (RSI>65 with BUY / <35 with SELL in any strong-ADX regime, both directions in mixed trending); bb_overextended 8 (BB pos >0.85 BUY / <0.15 SELL).
   • Exhaustion hard block: ≥2 exhaustion-tagged penalties ({exhaustion, no_pullback, rsi_extended, bb_overextended} → report.exhaustion_count) AND momentum fading (ROC sign against score, or MACD signal against score) → confidence = 0.
   • Confidence clamped [0, 95]. Final signal: score ≥35 & conf ≥55 STRONG_BUY; ≥25 & ≥50 BUY; mirrored sells; else NEUTRAL.
   • Quality grade: |score|≥40 & conf≥65 & MTF → A+; ≥33 & ≥55 & MTF → A; ≥28 & ≥50 → B; ≥25 & ≥45 → C; ≥15 → D; else F.
7. Post-grade gates, in order:
   • 6b session filter (bar-label hour + weekday; XAU reads xau_strategy.blocked_hours_utc/allowed_weekdays, others read config.backtest.*) → NEUTRAL/F.
   • 6c transitioning boost: regime transitioning and grade B/C/D → A if |score|≥30 & conf≥50 & MTF; else B if |score|≥25 & conf≥45 (early trend entries).
   • 6d exhausted-trend cap: trending regime + exhaustion_count ≥2 + grade C/D → D, setup killed.
   • 6e XAU SELL quality gate: sell-direction & grade actionable (A+..C) & not in sell_min_quality → D. The actionable guard is load-bearing: without it a session-blocked F report was upgraded to D, re-opening the M5 pullback path (its entry check is != "F").
   • 6f MTF opposition: ≥2 TFs pointing against the trade direction → D (never trade against consensus).
   • 6g XAU SELL volatility gate: sell-direction & vol regime not in {high, extreme} → D.
   • 6g2 XAU regime gates: regime in blocked_regimes (M15: mixed trending) → D; sell-direction & regime in sell_blocked_regimes → D.
   • 6g3 XAU SELL hour gate: sell-direction & bar-CLOSE hour (label + candle duration — the entry hour under next-open fills) in sell_blocked_hours_utc → D. Per-TF opt-in; M15 factory blocks 15/17 UTC.
   • 6i XAU M5 noise gates (_apply_m5_filters, primary TF = M5 only): (1) |score| ≥ 30; (2) MTF agreement required; (3) ≥1 confirmation TF with score ≥ ±10 in trade direction (NEUTRAL confirms don't count); (4) SELL grade must be B/C (A-grade M5 shorts lose −0.017R/trade); (5) vol regime high blocked; (6) SELL blocked when RSI < 25. Each failure → D, setup killed.
8. Trade setup: XAU + grade A+/A → _generate_xau_atr_trade_setup first; any pair + grade A+/A/B/C and no setup yet → _generate_trade_setup (swing-based).
9. Reasons: every non-neutral indicator description → bullish_reasons/bearish_reasons (the backtester tags strategy from reasons[0] markers).
10. XAU_PB pullback (M5 XAU only, m5_pullback_enabled, no setup yet, grade ≠ F): _try_m5_pullback — see below.
11. XAU_CP chart pattern (any TF, XAU only, cp_enabled, no setup yet, NOT session/day-blocked — weak-composite grade F is allowed BY DESIGN, reversals complete on weak composites): _try_chart_pattern — see below.

Trade setup generators (exact geometry)

A. _generate_xau_atr_trade_setup (XAU, grade A+/A, ATR>0): TP quality scale A+ 2.0 / A 1.5 (A with exhaustion≥2 → 1.0). TP1 = entry ± tp_atr_multiplier(1.0)×scale×ATR; TP2 = 1.5×that; SL = entry ∓ 2.0×ATR. Gates: RR ≥ 0.5; TP1 distance ≥ $9 (spread gate). Prices rounded to 2 dp; pip 0.10 hardcoded — this generator is XAU-only by construction.

B. _generate_trade_setup (swing-based, any pair, grade A+..C): XAU requires ATR ≥ $1 (LOW ATR FILTER). Per-side:

• SL anchor = nearest swing low below price (BUY) / swing high above (SELL) ∓/± sl_swing_buffer×ATR (1.0); fallback (no swing) = (1+buffer)×ATR. SL distance floored at max(ATR × sl_atr_floor, min_sl_pips×pip) — XAU: max(2.0×ATR, $8).
• TP1 = nearest swing beyond entry ± 0.3 ATR if it's within tp_swing_cap ATR (2.5) AND ≥ SL×RR-floor; else entry ± max(tp_mult×ATR, SL×RR-floor). tp_mult 1.5, RR-floor = max(min_rr 0.5, tp_floor_rr — 1.0 on M15).
• TP2 = second swing if within 5 ATR, else entry ± max((tp_mult+1.5)×ATR, SL×floor×1.5); forced beyond TP1 by ≥1 ATR.
• Gates: RR ≥ min_rr; spread gate TP1 distance ≥ cost×multiple ($9 on XAU).

C. _try_m5_pullback (XAU_PB): detection = EMA alignment ≥3 of 4 checks (9>21, 21>50, 50>200, close>200 — mirrored for SELL), ADX ≥ 20, price touched max(EMA21, KC-mid) within last 3 bars, resumption bar closed back in trend direction beyond EMA9, retrace depth 0.8–6.0 ATR from the 40-bar extreme, RSI in pullback zone (BUY 35–62 / SELL 38–65). Then gates: directional score ≥ 15; ≥1 confirmation TF ≥ ±10; not against MTF consensus; vol regime gates (same as breakout path). Geometry: SL = pullback 8-bar extreme ∓ 0.5 ATR (min $8); TP1 = 1.5×risk, TP2 = 2.5×risk; spread gate $9. Sets strategy="XAU_PB", grade min B, marker "XAU PULLBACK STRATEGY: …" at reasons[0]. Rejection funnel counters in class attr SignalAggregator.pb_funnel.

D. _try_chart_pattern (XAU_CP): detection in chart_patterns.py — 6 classical patterns (double_bottom/double_top, inv_hs/hs_top, bull_flag/bear_flag), swing-point based (strict local extrema, window cp_swing_window=3), all thresholds in ATR units, scanned over the last cp_lookback=72 closed bars. Trigger = FIRST close beyond the neckline/boundary since the pattern formed (one breakout → one signal, no repaint); reversal patterns are invalidated by a new extreme beyond the pattern. No composite-score gate BY DESIGN. Gates: MTF non-opposition (6f mirror); SELL patterns respect sell_vol_regimes, sell_blocked_hours_utc (bar-close hour), M5 RSI floor; M5 blocks m5_blocked_vol_regimes both ways; ATR ≥ $1. Geometry: SL = pattern anchor ∓ 0.5 ATR (min $8); TP1 = measured move (pattern height), TP2 = 1.5×height; RR ≥ 0.5; spread gate $9. Sets strategy="XAU_CP" (global trailing params), grade min B, marker "XAU CHART PATTERN: …" at reasons[0]. Funnel in SignalAggregator.cp_funnel.

TradeSetup fields: pair, direction, entry, SL, TP1, TP2, RR, risk/reward pips, pip_value, strategy ("GENERIC", "XAU_PB", or "XAU_CP" — selects trailing params; XAU_CP uses the global pair).

---

news_engine.py — NewsEngine

• Calendar: Forex Factory free JSON (nfs.faireconomy.media/ff_calendar_thisweek.json), 3 retries, honors 429 Retry-After with class-level backoff (≤600 s); fallback MyFXBook RSS. Never fabricates events — unavailable calendar = risk checks skipped that cycle.
• Caches are class-level (a fresh NewsEngine is built per scan): calendar + headlines, TTL = 600 s on success, 60 s on empty result; lock held across fetch so concurrent scans share one request.
• Event risk (get_event_risk_level): high-impact event for base/quote within 30 min → danger; within 60 min → caution; already-released high-impact → caution for 15 min after (45 min if actual ≠ forecast — "surprise"). EconomicEvent.surprise compares parsed numeric actual vs forecast (handles "%", "K/M/B" suffixes); direction is numeric-only (doesn't know if higher is bullish).
• Sentiment: keyword sets over headlines from 4 RSS feeds, deduped by title (Google News " - Publisher" suffix stripped), last 24 h only. Per-currency scores; GOLD_INVERSE_KEYWORDS (rate hike/hawkish/tightening/rate cut/dovish/easing/stimulus) flip sign for XAU (hawkish = dollar-bullish = gold-bearish). Currency detection via compiled word-boundary-ish regexes (bare "us"/"eu"/"uk" deliberately excluded). Pair sentiment = base − quote.

---

data_provider.py — ForexDataProvider

• Twelve Data SDK (TDClient.time_series(...).as_pandas()), symbols normalized to "XAU/USD" form; renames lowercase columns → capitalized; sorts oldest-first; localizes UTC.
• Rate limit: 55 credits/min sliding window (_throttle), blocks with sleep when exhausted.
• In-memory cache: key {symbol}_{tf}, TTL = candle duration (M5 300 s … H4 14400 s); a cached frame is evicted and refetched if it fails the freshness check or the boundary check (_has_latest_closed_bar: the frame must contain the most recent fully closed bar, weekend-exempt — a wall-clock TTL alone would serve mid-candle snapshots at the next boundary and scans would run one bar behind). Cache and throttle are lock-protected (one provider is shared across the bot's thread pool).
• _drop_forming_candle: Twelve Data includes the current partial candle; the scanner fires seconds after boundary, so the last bar is dropped if its close time is in the future. Without this every indicator would read a seconds-old stub and signals would repaint.
• Freshness (_is_data_fresh): last bar older than 3× candle duration → stale (warns; live path suppresses signal if the PRIMARY TF is stale). Weekends exempt (Fri 22:00→Sun 22:00 UTC).
• _validate_and_clean: drop all-NaN rows → ffill (limit 3) → drop NaN closes → sort + dedupe (keep last) → drop future timestamps (>now+5 min) → repair OHLC inconsistencies (High=max, Low=min of O/C/existing) → remove >10σ return outliers (spike bar only — the opposite-direction snap-back bar right after it is kept, else every spike punched a two-bar hole) → warn if < ~210 bars.
• outputsize = max(300, min(days×bars_per_day+20, 5000)); TD caps at 5000/call (the backtester paginates by date).

---

pair_signals/ — per-pair factories (ALWAYS use these, never bare PlatformConfig for a pair)

PAIR_PRODUCERS: Dict[symbol, factory]; factory signature make_config(primary_tf, confirmation_tfs=None) -> PlatformConfig. produce_signal(symbol, tf, data_provider) builds config → aggregator → analyze_pair. Registered: XAU/USD.

Inside make_config, call order is: PlatformConfig() defaults → app_config.apply_pair_overrides(config) (stamps config.json's non-credential deployment settings onto the config: base xau_strategy/backtest blocked_hours_utc, scan_pairs, scan_timeframes) → the validated per-TF deltas below. The overlay runs first so e.g. the M5 hour-9 append lands on top of the JSON base, not the other way around.

A bare PlatformConfig() silently runs any pair with XAU defaults — always go through the factory (the bot, CLI, and backtester all do).

Default confirmation map (same in the factory, the bot, and backtest usage): M5→[M15,H1], M15→[M5,H1], H1→[M15,H4], H4→[H1,D1]. (Exception: main.py's CLI map uses M5→[M1,M15] — a CLI-only quirk.)

xauusd.py: PlatformConfig() defaults + per-TF overrides (each validated; full evidence in the module docstring):

• M5: append hour 9 UTC to blocked hours (6-yr replay: +207.1R vs +178.3R; M15-and-up keep hour 9 — it's positive there). Enable BUY-only chart patterns (cp_enabled=True, cp_patterns=["inv_hs","double_bottom","bull_flag"]): 12mo +41.8R PF 1.60 (train/val both positive), 5yr +65.2R PF 1.48; bearish CP dropped (OOS −3.21R). M15/H1 CP remain disabled (all failed OOS).
• M15: signals.tp_floor_rr = 1.0 (1R minimum targets: +116.6R vs +105.4R, better in train/val and 2024 OOS; hurts M5 — per-TF only) and xau_strategy.blocked_regimes = ["trending"] (mixed regime block: +65.7R vs +59.0R). A transitioning-SELL block was tested and REJECTED (bull-market artifact — failed 2024 OOS).
• M15: xau_strategy.sell_blocked_hours_utc = [15, 17] (SELL-only hour block, validated 2026-07-16 on the 5-yr replay: +176.7R PF 1.31 vs +144.9R PF 1.22, better in all 6 years, both 12-mo halves, and 2024-06→2025-03 OOS; M5 cross-TF pocket confirms; BUYs at those hours are +34.3R so blocked_hours_utc would be wrong).
• The factory must stay equivalent to inline PlatformConfig(symbol=XAUUSD, …) + overrides — enforced by a test in test_all.py.

---

telegram_bot.py — the bot

Run: python telegram_bot.py (credentials resolve env TELEGRAM_TOKEN/TELEGRAM_CHAT_ID/TELEGRAM_CHANNEL_ID → config.json telegram.* → refuses to boot; see Environment variables).

• Owner lock: every command/button is wrapped in _owner_only — only TELEGRAM_CHAT_ID gets responses; others get a refusal. The bot is the owner's control panel; the product is the channel.
• Subscriber channel (TELEGRAM_CHANNEL_ID, -100…; "" disables): auto-scan signal alerts + breach alerts also post there (bot needs channel-admin). Monetization = Telegram Stars subscription invite links; Telegram handles membership, the bot has zero payment code. The channel feed keeps running even when the owner toggles DM auto-scan off (_feed_active() = DM chats OR channel).
• Auto-scan: boot-starts if TELEGRAM_CHAT_ID set (initial M15 scan populates the level cache first). Candle-boundary loops for M5/M15/H1/H4, firing at boundary + 10 s. Each scan runs all scan_pairs concurrently, updates the level cache, then broadcasts.
• Alert gating: quality threshold (_alert_min_quality, default "C", menu cycles C→B→A) + fingerprint dedup per (symbol, TF) on (signal, quality, direction, entry bucket) with 1-hour re-arm — a persisting identical signal doesn't re-send every candle, but a setup that moved ≥ half its risk distance (new pullback/swing anchor) re-alerts. The fingerprint is committed only after a message reaches ≥1 target (_commit_alert), so a failed Telegram send doesn't consume the alert.
• WebSocket monitor: subscribes every scan_pairs symbol to TD quotes WS; 10 s heartbeat; exponential reconnect (≤60 s). On tick crossing any cached key level (swing_high/low_1..3, ema_trend, bb_upper/lower, kc_upper/lower; cache TTL 1 h) → immediate M15 rescan → "LEVEL BREAK" alert if actionable+changed (same gates as auto-scan), 5-min per-pair cooldown.
• Commands: /start (menu), /scan (buttons: per pair — M5/M15/H1/H4 + Scan All TFs), /autoscan (toggle DM copy), /status (loops, WS, live prices, level cache, cooldowns), /cache (age per pair/TF), /clearcache, /help.
• Signal messages include the full indicator breakdown, warnings, trade setup, trailing-stop management plan (arm price + trail distance, from Signal/pb params by strategy), and a risk disclaimer on every outbound message.
• Blocking scan work runs in ThreadPoolExecutor(max_workers=4); one shared ForexDataProvider across all scans (cache reuse). Logs to logs/telegram.log.

---

main.py — CLI

python main.py [--pair XAUUSD] [--timeframe M5|M15|M30|H1|H4|D1] [--detail] [--json] [--verbose]. Uses the pair factory, renders a compact terminal report (decision, setup, context, top-3 reasons per side, warnings) or full JSON via report_to_dict.

---

backtest/gpu_backtest.py — the backtester

Replays history bar-by-bar through the real SignalAggregator.analyze_from_dataframes — same gates, same setup math. News is disabled (use_news=False): backtests are pure technical.

Realism model (all validated fixes — do not regress):

• Entry at the next bar's open after the signal bar; SL/TP distances preserved relative to actual entry. Entries are re-anchored to the raw price path (_reanchor_entries_to_raw): if cleaning dropped a >10σ spike bar right after the signal bar, the trade enters at the spike bar's open, not the bar after it — skipping it was a favorable bias.
• Round-trip --spread subtracted from every trade's PnL before computing R (default = pair's default_spread: XAU $0.30). Results before 2026-07-08 (+59.5R baseline era) charged no spread — not comparable.
• Trade resolution runs against the RAW price path (fetch_history(clean=False)): the parquet cache stores raw bars; cleaning (which drops >10σ news spikes that hit stops in reality) is applied only to the signal-side data. Caches written before this fix contain cleaned data — refetch to restore spike bars.
• Same-bar SL+TP tie → SL first (conservative). Gap through SL/TP → filled at the bar open.
• Trailing exits (default on, from signals.trail_arm_r/trail_dist_r; XAU_PB trades carry per-trade 0.8/0.75 overrides): stop ratchets at bar close only, so a new stop can only fill from the next bar (conservative). Exit reasons: SL / TRAIL / TP1 (static mode) / TIMEOUT / NO_DATA.
• Session + weekend + weekday filters evaluate the decision time = bar close (ts + primary_duration) — exactly matching the live bot's wall-clock check.
• Confirmation TFs are sliced to bars fully closed by decision time (no look-ahead).

Performance: precompute_indicators builds every indicator series once per TF (CuPy on GPU or NumPy — check the log line Backend: CuPy GPU vs NumPy CPU fallback; CuPy needs the nvidia-*-cu12 pip wheels or it silently falls back). GPUTechnicalEngine subclasses the live engine and overrides only the _compute_* methods with cache lookups — outputs are byte-identical to the live engine (regression-tested). Bar replay: pure per-bar evaluation fans out over forked worker processes (--workers, 0 = auto: cores−2 capped 12; <5000 bars forces 1); the entry cooldown (5 bars) is the only sequential rule, applied in an ordered pass afterwards — parallel and sequential books are identical. ~300 bars/s/core, ~1700 bars/s at 10 workers.

Strategy tags (from reasons[0] markers): XAU_PB, XAU_CP, GENERIC (plus legacy XAU recognized). --strategies filters the book by tag. --cp force-enables the chart-pattern entry for candidate validation; --cp-patterns narrows the detector set.

CLI:

python backtest/gpu_backtest.py --pair XAUUSD --days 365
python backtest/gpu_backtest.py --pair XAUUSD --start 2025-01-01 --end 2025-06-01 --primary-tf 15min --confirm-tfs 1h,4h
# key flags: --spread --quality A+,A,B,C --strategies --workers --max-hold-bars 300
#            --warmup-bars 250 --cooldown-bars 5 --no-cache --export-csv out.csv --detail

Report: summary metrics (total R, PF, Sharpe on R, max DD, MFE/MAE), breakdowns by quality/strategy/regime, trade log, and per-trade reasoning dumps (indicator-by-indicator agreement at entry — required by instructions_for_backtest.txt). --export-csv flattens every indicator's signal/value/agreement into columns.

Comparability note (2026-07-16): the parity/realism fixes of that date (EMA-ribbon port to the GPU engine, exact-pandas Bollinger precompute, raw-path entry re-anchoring, per-TF D1 warmup, live confirm-map CLI default, outlier snap-back retention) all shift books slightly vs earlier runs. Books produced before these fixes are not directly comparable — re-run the baseline before any before/after tuning comparison.

Data: paginated TD fetches (5000-bar chunks) cached as parquet in backtest/cache/ keyed by symbol/TF/window; warmup pad ≥60 days before the window for EMA200, extended per-TF so slow confirmation TFs get ≥210 bars at window start (D1 gets ~299 days — 62 days gave only ~61 bars: no EMA200, unlike live's 300-bar outputsize floor). --confirm-tfs defaults to the pair factory's live confirmation map (pass it explicitly to override); a hardcoded default here previously made the standard invocation backtest a different MTF setup than the bot scans.

---

test_all.py — QA suite

python test_all.py — 200 tests, 13 modules (config, data_provider, engine, news_engine, signal_aggregator, main, telegram_bot, integration, pair_signals registry + engine-bug regressions, 2026-07-16 audit-fix regressions, chart patterns, app_config). No network needed (mocked). Uses a custom @test("name") decorator that runs immediately (NOT pytest — don't grep for def test_). Exit code non-zero on failure. Includes: the xauusd-factory ≡ inline-config equivalence test, engine bug-fix regression tests (Module 9b), a live-vs-GPU engine parity sweep incl. the EMA-ribbon path (Module 11), and config.json loading/precedence/validation coverage (Module 13). Any signal-logic change must keep this green and should add tests.

---

Environment variables

Precedence for every credential/deployment setting: env var → config.json → none. Hardcoded fallbacks (previously in config.py / telegram_bot.py) were REMOVED 2026-07-21; the bot now refuses to boot (telegram_bot.main) and data fetches fail with a named error (data_provider) if a secret resolves to neither source. See app_config.py and config.json.example.

Variable	Purpose	Default
TWELVE_DATA_API_KEY	market data key	none — falls through to config.json data_providers.twelve_data.api_key, then fails fast
TELEGRAM_TOKEN	bot token	none — falls through to config.json telegram.token, then refuses to boot
TELEGRAM_CHAT_ID	owner chat id (boot auto-scan + owner lock)	none — falls through to config.json telegram.chat_id, then refuses to boot
TELEGRAM_CHANNEL_ID	subscriber channel (-100…); "" disables the feed	none — falls through to config.json telegram.channel_id
FB2_EMA_LO	EMA200 "near" penalty threshold in %	0.5

Dependencies (requirements.txt): numpy, pandas, requests, twelvedata, python-telegram-bot ≥20, websockets. Python 3.12. A venv/ exists in-repo.

---

Code conventions

• Type hints on public methods; dataclasses for all config and outputs.
• SignalStrength (−2..+2) is the universal signal currency.
• Key levels dict keys: swing_high_*, swing_low_*, ema_*, bb_*, kc_*, atr, atr_pct.
• Private per-indicator series ride on TechnicalAnalysis as _underscore attributes; anything _-prefixed or weight-0 is skipped by scorers/formatters.
• UTC everywhere; DataFrames indexed by tz-aware Datetime, oldest-first, capitalized OHLCV columns.
• Logging via module loggers; the bot adds a rotating file handler.

---

Rules for making changes (READ BEFORE EDITING SIGNAL LOGIC)

1. Never change a tuned value without re-validation. The acceptance rule used throughout: positive total R in BOTH the train and validation halves of a ≥12-month window, PF ≥ 1.3, at realistic cost (XAU $0.30 round trip), with trailing exits and next-open fills. For any marginal edge additionally re-test on the 2024-06→2025-03 out-of-sample window — within-year splits alone have approved configs that failed OOS.
2. Per-timeframe overrides live in the pair factory (pair_signals/*.py), never in global defaults — the same knob has opposite effects on different TFs (proven twice: tp_floor_rr, hour-9 block). Document the validation evidence in the factory docstring.
3. Pair isolation is a hard invariant. XAU gates in the aggregator are display-gated (report.display_name == "XAU/USD") so a future pair port cannot inherit gold-calibrated logic by accident.
4. Live == backtest parity: session/weekend/hour filters must evaluate bar-CLOSE time in both paths; xauusd.make_config must stay equivalent to the inline config the backtester/bot would build (test-enforced). If you add a live gate, add it to the replay path (or analyze_from_dataframes) in the same commit.
5. ROC stays weighted (see engine table) unless you retune composite thresholds and re-validate — removing it lost 13R.
6. Backtester realism knobs are results-critical: raw-path resolution, spread charging, next-open entry, SL-first ties, close-only trail ratchet. Any "optimization" that touches these invalidates comparability with the pinned books (backtest/*.csv, memory notes).
7. Don't zero/disable indicators via config in production paths — disabled_indicators exists for experiments; composite thresholds assume the full set.
8. The forming candle must stay dropped and analysis must only ever see closed bars, or live signals repaint.
9. Keep test_all.py green (python test_all.py); extend it for every new gate/knob.
10. Dead/legacy config fields were removed 2026-07-21 (MarketSession, is_jpy_pair, candle_buffer_seconds, the unused SignalConfig/XauStrategyConfig legacy knobs). Do not add config fields that nothing reads; delete dead fields only with test updates.