diff --git a/CLAUDE.md b/CLAUDE.md index 0117b50..ec65cc0 100644 --- a/CLAUDE.md +++ b/CLAUDE.md @@ -130,7 +130,7 @@ Results land in `TechnicalAnalysis.indicators: Dict[str, IndicatorSignal]` (fiel | `macd` | 1.5 | Histogram = MACD line − signal line. Cross above 0 → BUY (conf 55+); accelerating (hist>prev>0) → STRONG_BUY; mirrored down; positive-but-fading / negative-but-fading → NEUTRAL (35); flat → NEUTRAL (30). Confidence scaled by |hist| vs its 20-bar mean | | `bollinger` | 1.5 | Position = (close−lower)/(upper−lower). >1.0 SELL (65), <0.0 BUY (65), >0.8 SELL (45), <0.2 BUY (45), else NEUTRAL (30). Squeeze = width < 0.75 × 20-bar avg width → +10 conf. Emits `bb_upper/lower/mid` | | `adx` | 0.5 | Wilder ADX/DI (ties zero BOTH DMs — fixes a systematic bearish nudge). ADX ≥ 25 → BUY/SELL by DI dominance, else NEUTRAL. Conf 40/65/80 by ADX band. ADX slope (5-bar diff): <25 with slope > 3.0 → "TREND EMERGING" +15 conf; also strengthening/weakening adjustments. Sets `_adx_slope` | -| `roc` | 1.5 | 10-period %change. Positive & rising → BUY (STRONG if >2× threshold=2%); negative & falling → SELL; zero-cross → BUY/SELL (65); else NEUTRAL. **Directionally redundant with MACD/Keltner/EMA (0.8–0.94 corr) but its weight is load-bearing: zeroing it backtested 13R WORSE — composite thresholds are calibrated with it. Do not remove without retuning thresholds** | +| `roc` | 1.5 | 10-period %change. Positive & rising → BUY (STRONG if >2× threshold=2%); negative & falling → SELL (zero-crosses land in these branches by definition — a separate cross branch was provably dead and removed 2026-07-23); else NEUTRAL. **Directionally redundant with MACD/Keltner/EMA (0.8–0.94 corr) but its weight is load-bearing: zeroing it backtested 13R WORSE — composite thresholds are calibrated with it. Do not remove without retuning thresholds** | | `keltner` | 1.5 | EMA20 mid ± 1.5×ATR(10). TTM squeeze = BB fully inside KC. Squeeze released (was→not) + MACD hist sign → STRONG_BUY/STRONG_SELL (80); squeeze active → NEUTRAL (60); close above/below channel → BUY/SELL continuation (65); inside → NEUTRAL (30). Emits `kc_upper/lower/mid` | | `divergence` | 1.5 | RSI + MACD-hist vs price over 40 bars. Swing points (window 3) must align within ±2 bars to pair up; last two pairs: price LL + indicator HL = bullish, price HH + indicator LH = bearish. Any found → BUY/SELL/NEUTRAL-if-both, conf 70, value = count | | `candlestick` | 1.0 | Last bar: Doji (body<10% range), Hammer/Hanging Man (lower wick>2×body), Shooting Star/Inverted Hammer, Engulfing (vs prev bar), Marubozu (body>85%). Majority vote → signal, conf 50; +20 if pattern sits within 0.5 ATR of a swing S/R level | @@ -185,7 +185,7 @@ Two entry points: 6. **`_compute_final_signal`** (bonuses → event cap → penalties → thresholds → grade): - Bonuses: MTF agreement → +10 conf and `score = 0.6×primary + 0.4×mean(all TF scores)`; news sentiment aligned (|sent|>0.2) → +5 (gated by `news.sentiment_affects_signals`, False by default); in a *directional* trend, RSI in pullback zone → +5, BB position favorable → +5 (mixed `trending` regime gets NO directional bonus). - Event risk `danger`: warning always; confidence hard-capped at 30 ONLY if `news.event_risk_caps_signals` (False by default — XAU-tuned). - - **Penalty system (diminishing, replaces old additive stacking):** collect, sort by size desc, apply with decay ×1.0, ×0.5, ×0.25…, total capped at 30. Penalties: `mtf_disagreement` 15; `news_conflict` 10 (gated by `sentiment_affects_signals`, off by default — warning still emitted); `event_caution` 15 (config-gated); `counter_trend` 15; `extreme_vol` 10; `ema200_near` 5 (price within 0.5% of EMA200 — env `FB2_EMA_LO` overrides); `ema200_far` 5 (>2.5%); `exhaustion` 8 (price >3.5 ATR beyond nearest swing); `no_pullback` 8 (>2.5 ATR from EMA21 in trend) / `weak_pullback` 5 (>2.0); `rsi_extended` 5 (RSI>65 with BUY / <35 with SELL in any strong-ADX regime, both directions in mixed `trending`); `bb_overextended` 8 (BB pos >0.85 BUY / <0.15 SELL). + - **Penalty system (diminishing, replaces old additive stacking):** collect, sort by size desc, apply with decay ×1.0, ×0.5, ×0.25…, total capped at 30. Penalties: `mtf_disagreement` 15; `news_conflict` 10 (gated by `sentiment_affects_signals`, off by default — warning still emitted); `event_caution` 15 (config-gated); `counter_trend` 15; `extreme_vol` 10; `ema200_near` 5 (price within 0.5% of EMA200 — env `FB2_EMA_LO` overrides); `ema200_far` 5 (>2.5%); `exhaustion` 8 (price >3.5 ATR beyond the FARTHEST of the 3 tracked swing levels — fires earlier than a nearest-swing rule would; calibrated as-is in every validated book); `no_pullback` 8 (>2.5 ATR from EMA21 in trend) / `weak_pullback` 5 (>2.0); `rsi_extended` 5 (RSI>65 with BUY / <35 with SELL in any strong-ADX regime, both directions in mixed `trending`); `bb_overextended` 8 (BB pos >0.85 BUY / <0.15 SELL). - **Exhaustion hard block**: ≥2 exhaustion-tagged penalties ({exhaustion, no_pullback, rsi_extended, bb_overextended} → `report.exhaustion_count`) AND momentum fading (ROC sign against score, or MACD signal against score) → confidence = 0. - Confidence clamped [0, 95]. **Final signal**: score ≥35 & conf ≥55 STRONG_BUY; ≥25 & ≥50 BUY; mirrored sells; else NEUTRAL. - **Quality grade**: |score|≥40 & conf≥65 & MTF → **A+**; ≥33 & ≥55 & MTF → **A**; ≥28 & ≥50 → **B**; ≥25 & ≥45 → **C**; ≥15 → **D**; else **F**. @@ -218,7 +218,7 @@ Two entry points: **D. `_try_chart_pattern` (XAU_CP)**: detection in `chart_patterns.py` — 6 classical patterns (double_bottom/double_top, inv_hs/hs_top, bull_flag/bear_flag), swing-point based (strict local extrema, window `cp_swing_window`=3), all thresholds in ATR units, scanned over the last `cp_lookback`=72 closed bars. Trigger = FIRST close beyond the neckline/boundary since the pattern formed (one breakout → one signal, no repaint); reversal patterns are invalidated by a new extreme beyond the pattern. No composite-score gate BY DESIGN. Gates: MTF non-opposition (6f mirror); SELL patterns respect `sell_vol_regimes`, `sell_blocked_hours_utc` (bar-close hour), M5 RSI floor; M5 blocks `m5_blocked_vol_regimes` both ways; ATR ≥ $1. Geometry: SL = pattern anchor ∓ 0.5 ATR (min $8); TP1 = measured move (pattern height), TP2 = 1.5×height; RR ≥ 0.5; spread gate $9. Sets `strategy="XAU_CP"` (global trailing params), grade min B, marker `"XAU CHART PATTERN: …"` at reasons[0]. Funnel in `SignalAggregator.cp_funnel`. -`TradeSetup` fields: pair, direction, entry, SL, TP1, TP2, RR, risk/reward pips, pip_value, `strategy` ("GENERIC", "XAU_PB", or "XAU_CP" — selects trailing params; XAU_CP uses the global pair). +`TradeSetup` fields: pair, direction, entry, SL, TP1, TP2, RR, risk/reward pips, pip_value, `strategy` ("GENERIC", "XAU_PB", or "XAU_CP" — selects trailing params; XAU_CP uses the global pair), `trail_arm_r`/`trail_dist_r` (stamped from the PRODUCING config at generation so per-TF factory overrides reach formatters; `None` = legacy unstamped → display falls back to platform defaults). --- @@ -235,7 +235,7 @@ Two entry points: - Twelve Data SDK (`TDClient.time_series(...).as_pandas()`), symbols normalized to `"XAU/USD"` form; renames lowercase columns → capitalized; sorts oldest-first; localizes UTC. - **Rate limit**: 55 credits/min sliding window (`_throttle`), blocks with sleep when exhausted. -- **In-memory cache**: key `{symbol}_{tf}`, TTL = candle duration (M5 300 s … H4 14400 s); a cached frame is evicted and refetched if it fails the freshness check **or the boundary check** (`_has_latest_closed_bar`: the frame must contain the most recent fully closed bar, weekend-exempt — a wall-clock TTL alone would serve mid-candle snapshots at the next boundary and scans would run one bar behind). Cache and throttle are lock-protected (one provider is shared across the bot's thread pool). +- **In-memory cache**: key `{symbol}_{tf}`, TTL = candle duration (M5 300 s … H4 14400 s); a cached frame is evicted and refetched if it fails the freshness check **or the boundary check** (`_has_latest_closed_bar`: the frame must contain the most recent fully closed bar, weekend-exempt — a wall-clock TTL alone would serve mid-candle snapshots at the next boundary and scans would run one bar behind). Fresh fetches enforce the same boundary guarantee: a response missing the just-closed bar (feed latency at the boundary) gets one grace retry after `FRESH_FETCH_RETRY_SECONDS` (3 s), then proceeds with a loud warning. Cache and throttle are lock-protected (one provider is shared across the bot's thread pool). - **`_drop_forming_candle`**: Twelve Data includes the current partial candle; the scanner fires seconds after boundary, so the last bar is dropped if its close time is in the future. Without this every indicator would read a seconds-old stub and signals would repaint. - **Freshness** (`_is_data_fresh`): last bar older than 3× candle duration → stale (warns; live path suppresses signal if the PRIMARY TF is stale). Weekends exempt (Fri 22:00→Sun 22:00 UTC). - **`_validate_and_clean`**: drop all-NaN rows → ffill (limit 3) → drop NaN closes → sort + dedupe (keep last) → drop future timestamps (>now+5 min) → repair OHLC inconsistencies (High=max, Low=min of O/C/existing) → remove >10σ return outliers (spike bar only — the opposite-direction snap-back bar right after it is kept, else every spike punched a two-bar hole) → warn if < ~210 bars. @@ -270,11 +270,11 @@ Run: `python telegram_bot.py` (credentials resolve env `TELEGRAM_TOKEN`/`TELEGRA - **Owner lock**: every command/button is wrapped in `_owner_only` — only `TELEGRAM_CHAT_ID` gets responses; others get a refusal. The bot is the owner's control panel; the product is the channel. - **Subscriber channel** (`TELEGRAM_CHANNEL_ID`, `-100…`; `""` disables): auto-scan signal alerts + breach alerts also post there (bot needs channel-admin). Monetization = Telegram Stars subscription invite links; Telegram handles membership, the bot has zero payment code. The channel feed keeps running even when the owner toggles DM auto-scan off (`_feed_active()` = DM chats OR channel). - **Auto-scan**: boot-starts if `TELEGRAM_CHAT_ID` set (initial M15 scan populates the level cache first). Candle-boundary loops for M5/M15/H1/H4, firing at boundary + 10 s. Each scan runs all `scan_pairs` concurrently, updates the level cache, then broadcasts. -- **Alert gating**: quality threshold (`_alert_min_quality`, default "C", menu cycles C→B→A) + fingerprint dedup per (symbol, TF) on (signal, quality, direction, entry bucket) with 1-hour re-arm — a persisting identical signal doesn't re-send every candle, but a setup that moved ≥ half its risk distance (new pullback/swing anchor) re-alerts. The fingerprint is committed only **after** a message reaches ≥1 target (`_commit_alert`), so a failed Telegram send doesn't consume the alert. +- **Alert gating**: quality threshold (`_alert_min_quality`, default "C", menu cycles C→B→A) + fingerprint dedup per (symbol, TF) on (signal, quality, direction) with 1-hour re-arm — a persisting identical signal doesn't re-send every candle, but a setup whose entry moved ≥ half the last-ALERTED risk distance (new pullback/swing anchor; compared against the stored last-alert entry/risk, a stable reference immune to per-candle ATR drift) re-alerts. The fingerprint is committed only **after** a message reaches ≥1 target (`_commit_alert`), so a failed Telegram send doesn't consume the alert. - **WebSocket monitor**: subscribes every `scan_pairs` symbol to TD quotes WS; 10 s heartbeat; exponential reconnect (≤60 s). On tick crossing any cached key level (`swing_high/low_1..3`, `ema_trend`, `bb_upper/lower`, `kc_upper/lower`; cache TTL 1 h) → immediate M15 rescan → "LEVEL BREAK" alert if actionable+changed (same gates as auto-scan), 5-min per-pair cooldown. - **Commands**: `/start` (menu), `/scan` (buttons: per pair — M5/M15/H1/H4 + Scan All TFs), `/autoscan` (toggle DM copy), `/status` (loops, WS, live prices, level cache, cooldowns), `/cache` (age per pair/TF), `/clearcache`, `/help`. - **Tiingo spread monitor** (optional; on only when a Tiingo key resolves — see Environment variables): a 5-min sampler task fetches top-of-book bid/ask per scanned pair (`tiingo_provider.py`), logs to `logs/spread_log.csv` (skipping the weekend-closed window), and caches the latest quote; alert sends refresh the quote first, and signal messages then show live bid/ask + a warning when spread ≥ 2× the pair's assumed round-trip cost (`WIDE_SPREAD_MULTIPLE`). **Warn-only BY DESIGN** — no historical bid/ask exists, so a suppressing gate could never be validated (live == backtest rule); signals never consume bid/ask. `/status` shows the live spread per pair. **License (free tier is internal-use-only):** raw bid/ask/spread numbers go to owner DMs always; the channel gets them ONLY while config.json `telegram.channel_spread_detail=true` (the channel is internal/testing as of 2026-07-22 — **flip to false before the channel goes public**), otherwise a number-free "spread unusually wide" caution via `format_signal_message(report, spread_detail=False)` — routing in `_spread_detail_for`, test-enforced, committed default false. **Rate limits (free tier 50 req/h, 1000/day, checked 2026-07-22):** sampler 12/h + alert-batch refreshes ≤17/h + breach refreshes ≤12/h = worst case ≈41/h with ONE pair; adding pairs multiplies this — recheck before growing `scan_pairs`. -- Signal messages include the full indicator breakdown, warnings, trade setup, **trailing-stop management plan** (arm price + trail distance, from Signal/pb params by `strategy`), and a risk disclaimer on every outbound message. +- Signal messages include the full indicator breakdown, warnings, trade setup, **trailing-stop management plan** (arm price + trail distance, read from the `trail_arm_r`/`trail_dist_r` stamped on the setup by the producing per-TF config), and a risk disclaimer on every outbound message. - Blocking scan work runs in `ThreadPoolExecutor(max_workers=4)`; one shared `ForexDataProvider` across all scans (cache reuse). - **Logging (hardened 2026-07-22)**: root-logger handlers (`force=True`) → console + `logs/telegram.log` (3-hour rotation, 240 backups = 30 days), full-date timestamps; every module logger (data_provider, tiingo_provider, news_engine, websockets, telegram.ext) lands in the same file. Failure surfaces all covered: candle loops and the spread sampler catch-and-continue per cycle (one bad scan can't kill a loop), breach tasks log exceptions via done-callback, PTB `add_error_handler` registered, and `sys.excepthook`/`threading.excepthook` (installed in `main()`) route uncaught crashes to the log. WS monitor keeps its exponential-backoff reconnect + `logger.exception`. @@ -297,7 +297,7 @@ Replays history bar-by-bar through the **real `SignalAggregator.analyze_from_dat - Same-bar SL+TP tie → SL first (conservative). Gap through SL/TP → filled at the bar open. - **Trailing exits** (default on, from `signals.trail_arm_r/trail_dist_r`; XAU_PB trades carry per-trade 0.8/0.75 overrides): stop ratchets at bar close only, so a new stop can only fill from the next bar (conservative). Exit reasons: SL / TRAIL / TP1 (static mode) / TIMEOUT / NO_DATA. - Session + weekend + weekday filters evaluate the **decision time = bar close** (`ts + primary_duration`) — exactly matching the live bot's wall-clock check. -- The optional H4-ADX regime filter (`config.backtest.regime_filter`, default off) is applied in the replay by calling the live `_apply_regime_filter_live` method itself under the same guard — live and backtest cannot diverge if it's enabled (2026-07-21 parity fix; the unregistered-pair config fallback also applies the config.json overlay now). +- The optional H4-ADX regime filter (`config.backtest.regime_filter`, default off) is applied in the replay by calling the live `_apply_regime_filter_live` method itself under the same guard, and the replay loads/slices the regime-filter TF whenever the gate is on (like live `analyze_pair`) — previously M5/M15 primaries silently no-opped the filter because H4 wasn't in their confirm maps. Live and backtest cannot diverge if it's enabled (2026-07-21 parity fix; the unregistered-pair config fallback also applies the config.json overlay now). - Confirmation TFs are sliced to bars **fully closed** by decision time (no look-ahead). **Performance:** `precompute_indicators` builds every indicator series once per TF (CuPy on GPU or NumPy — check the log line `Backend: CuPy GPU` vs `NumPy CPU fallback`; CuPy needs the `nvidia-*-cu12` pip wheels or it silently falls back). `GPUTechnicalEngine` subclasses the live engine and overrides only the `_compute_*` methods with cache lookups — outputs are byte-identical to the live engine (regression-tested). Bar replay: pure per-bar evaluation fans out over forked worker processes (`--workers`, 0 = auto: cores−2 capped 12; <5000 bars forces 1); the entry **cooldown (5 bars) is the only sequential rule**, applied in an ordered pass afterwards — parallel and sequential books are identical. ~300 bars/s/core, ~1700 bars/s at 10 workers. diff --git a/app_config.py b/app_config.py index c9052e4..7bb4f33 100644 --- a/app_config.py +++ b/app_config.py @@ -65,6 +65,13 @@ def load_dotenv(path: str = ENV_PATH) -> None: key, value = key.strip(), value.strip() if len(value) >= 2 and value[0] == value[-1] and value[0] in "\"'": value = value[1:-1] + elif value[:1] not in "\"'": + # Unquoted value: whitespace-then-# starts an inline comment + # (KEY=abc # prod). Quote the value to keep a literal " #". + for sep in (" #", "\t#"): + cut = value.find(sep) + if cut != -1: + value = value[:cut].rstrip() if key: os.environ.setdefault(key, value) diff --git a/backtest/gpu_backtest.py b/backtest/gpu_backtest.py index 8f530d6..4676d6d 100644 --- a/backtest/gpu_backtest.py +++ b/backtest/gpu_backtest.py @@ -851,15 +851,10 @@ class GPUTechnicalEngine(TechnicalEngine): signal = SignalStrength.SELL conf = min(70, 50 + roc_abs * 10) desc = f"ROC {roc_val:.2f}% — Falling momentum" - elif roc_prev <= 0 < roc_val: - signal = SignalStrength.BUY - conf = 65 - desc = f"ROC crossed above zero (+{roc_val:.2f}%)" - elif roc_prev >= 0 > roc_val: - signal = SignalStrength.SELL - conf = 65 - desc = f"ROC crossed below zero ({roc_val:.2f}%)" else: + # Zero-crosses land in the branches above by definition (a + # cross up is positive and rising, a cross down negative and + # falling) — only flat/fading momentum reaches here. signal = SignalStrength.NEUTRAL conf = 30 desc = f"ROC {roc_val:+.2f}% — No clear momentum" @@ -1690,6 +1685,13 @@ def run_backtest( # ── Load all required timeframes ── timeframes = [bt_cfg.primary_tf] + list(bt_cfg.confirmation_tfs) + # Live parity: analyze_pair also fetches the regime-filter TF when the + # optional H4-ADX gate is enabled. Without it here, M5/M15 primaries + # (whose confirm maps lack H4) would make _apply_regime_filter_live + # silently no-op in the replay while live applies the gate. + flt = platform_cfg.backtest + if flt.regime_filter and flt.regime_filter_timeframe not in timeframes: + timeframes.append(flt.regime_filter_timeframe) # Pad lookback so earliest TF has enough warmup for EMA200, ADX, etc. pad_days = max(60, bt_cfg.warmup_bars // 4) fetch_start = bt_cfg.start - timedelta(days=pad_days) @@ -1879,7 +1881,9 @@ def compute_metrics(trades: List[Trade]) -> Metrics: gross_loss = float(-rs[losses_mask].sum()) if losses_mask.any() else 0.0 pf = gross_profit / gross_loss if gross_loss > 0 else float("inf") - cum = np.cumsum(rs) + # Prepend the flat starting equity so an opening losing streak counts + # as drawdown from 0, not from the first trade's already-negative cum. + cum = np.concatenate(([0.0], np.cumsum(rs))) high_water = np.maximum.accumulate(cum) drawdown = high_water - cum max_dd = float(drawdown.max()) if len(drawdown) else 0.0 diff --git a/chart_patterns.py b/chart_patterns.py index 84895db..9d0b72f 100644 --- a/chart_patterns.py +++ b/chart_patterns.py @@ -12,9 +12,10 @@ Chart Patterns*) that are unambiguous from OHLC at bar close: the interim peak (neckline) double_top bearish reversal mirrored inv_hs bullish reversal inverse head & shoulders, break of - the neckline (max of the two peaks - flanking the head — conservative - horizontal line, no sloped fits) + the neckline (the LOWER of the two + peaks flanking the head — the earlier + horizontal trigger, no sloped fits; + validated as-is, see detector note) hs_top bearish reversal head & shoulders top, mirrored bull_flag bullish continuation strong pole, shallow ordered retrace, close at a new high @@ -166,8 +167,10 @@ def _detect_hs(o, h, l, c, atr, cfg, bullish: bool) -> Optional[Dict]: return None if abs(s1 - s2) > cfg.cp_hs_shoulder_tol_atr * atr: return None - # Neckline: the LOWER of the two interim peaks — the price must - # clear both, so this is the conservative horizontal trigger. + # Neckline: the LOWER of the two interim peaks — the EARLIER of + # the two horizontal triggers (max() would demand clearing both + # peaks). The whole validated CP book (M5 BUY-only) ran on this + # trigger — do not flip to max() without full re-validation. p1 = float(h[i1 : i2 + 1].max()) p2 = float(h[i2 : i3 + 1].max()) neckline = min(p1, p2) diff --git a/data_provider.py b/data_provider.py index 250ed7b..1119cd1 100755 --- a/data_provider.py +++ b/data_provider.py @@ -67,6 +67,9 @@ class ForexDataProvider: RATE_LIMIT = 55 RATE_WINDOW = 60.0 + # Grace wait before the one retry when a fresh response at a candle + # boundary is missing the just-closed bar (feed latency). + FRESH_FETCH_RETRY_SECONDS = 3.0 def __init__(self, config: PlatformConfig): self.config = config @@ -130,6 +133,24 @@ class ForexDataProvider: df = self._drop_forming_candle(df, timeframe) df = self._validate_and_clean(df, symbol, timeframe) + # The closed-bar guarantee must hold for FRESH responses too, not + # just cache hits: the scanner fires seconds after the boundary, + # and a lagging feed can answer without the just-closed bar — the + # scan would silently analyze (and session-check) the previous + # bar. Give the feed one short grace retry, then proceed loudly. + if (not df.empty and not self._has_latest_closed_bar(df, timeframe)): + time.sleep(self.FRESH_FETCH_RETRY_SECONDS) + retry = self._fetch_twelve_data(symbol, timeframe, outputsize) + retry = self._drop_forming_candle(retry, timeframe) + retry = self._validate_and_clean(retry, symbol, timeframe) + if retry is not None and not retry.empty: + df = retry + if not self._has_latest_closed_bar(df, timeframe): + logger.warning( + f"{symbol} {timeframe.value}: feed lagging — just-closed bar " + f"still missing after retry; this scan may run one bar behind" + ) + if not self._is_data_fresh(df, symbol, timeframe): logger.warning( f"STALE DATA from API: {symbol} {timeframe.value} — " @@ -198,9 +219,10 @@ class ForexDataProvider: def clear_cache(self) -> int: """Wipe all cached data. Returns number of entries cleared.""" - count = len(self._cache) - self._cache.clear() - self._cache_timestamps.clear() + with self._cache_lock: + count = len(self._cache) + self._cache.clear() + self._cache_timestamps.clear() logger.info(f"Cache cleared: {count} entries wiped") return count diff --git a/engine.py b/engine.py index 5eb85ec..5372685 100755 --- a/engine.py +++ b/engine.py @@ -638,15 +638,10 @@ class TechnicalEngine: signal = SignalStrength.SELL conf = min(70, 50 + roc_abs * 10) desc = f"ROC {roc_val:.2f}% — Falling momentum" - elif roc_prev <= 0 < roc_val: - signal = SignalStrength.BUY - conf = 65 - desc = f"ROC crossed above zero (+{roc_val:.2f}%)" - elif roc_prev >= 0 > roc_val: - signal = SignalStrength.SELL - conf = 65 - desc = f"ROC crossed below zero ({roc_val:.2f}%)" else: + # Zero-crosses land in the branches above by definition (a + # cross up is positive and rising, a cross down negative and + # falling) — only flat/fading momentum reaches here. signal = SignalStrength.NEUTRAL conf = 30 desc = f"ROC {roc_val:+.2f}% — No clear momentum" diff --git a/main.py b/main.py index f2cfc0c..89d1142 100755 --- a/main.py +++ b/main.py @@ -221,6 +221,9 @@ def report_to_dict(report: SignalReport) -> dict: "risk_reward": ts.risk_reward, "risk_pips": round(ts.risk_pips, 1), "reward_pips": round(ts.reward_pips, 1), + "strategy": ts.strategy, + "trail_arm_r": ts.trail_arm_r, + "trail_dist_r": ts.trail_dist_r, } if report.primary_analysis: d["indicators"] = { @@ -230,7 +233,7 @@ def report_to_dict(report: SignalReport) -> dict: "description": ind.description, } for name, ind in report.primary_analysis.indicators.items() - if not name.startswith("_") + if not name.startswith("_") and ind.weight != 0 } d["key_levels"] = { k: round(v, 5) for k, v in report.primary_analysis.key_levels.items() diff --git a/news_engine.py b/news_engine.py index 769ab9d..21109f0 100755 --- a/news_engine.py +++ b/news_engine.py @@ -15,6 +15,7 @@ import time import xml.etree.ElementTree as ET from dataclasses import dataclass, field from datetime import datetime, timezone +from email.utils import parsedate_to_datetime from typing import Dict, List, Optional, Tuple import requests @@ -281,14 +282,20 @@ class NewsEngine: """ base, quote = self._parse_pair(pair_symbol) - # Check upcoming high-impact events + # Check upcoming high-impact events. Grade from the SOONEST + # matching event, not the first listed — an unsorted feed could + # otherwise report "caution" for a far event while a nearer one + # warrants "danger". + soonest = None for event in context.events_upcoming_1h: if event.is_high_impact and event.currency in (base, quote): - mins = event.minutes_until - if mins <= self.config.pre_event_buffer_minutes: - return "danger", f"{event.currency} {event.event} in {mins:.0f}min" - else: - return "caution", f"{event.currency} {event.event} in {mins:.0f}min" + if soonest is None or event.minutes_until < soonest.minutes_until: + soonest = event + if soonest is not None: + mins = soonest.minutes_until + if mins <= self.config.pre_event_buffer_minutes: + return "danger", f"{soonest.currency} {soonest.event} in {mins:.0f}min" + return "caution", f"{soonest.currency} {soonest.event} in {mins:.0f}min" # Check for recent high-impact releases. A surprise print (actual # away from forecast) keeps the market unstable longer than an @@ -650,11 +657,20 @@ class NewsEngine: if not date_str: return None + # RFC 822/2822 forms, with or without seconds, numeric offsets AND + # named zones (GMT, EST/EDT, PST…). The old strptime "%Z" path + # accepted ANY abbreviation but attached no offset, so e.g. an EST + # pubDate was silently stamped UTC — a 5-hour error. + try: + dt = parsedate_to_datetime(date_str.strip()) + if dt is not None: + if dt.tzinfo is None: # unknown zone name — assume UTC + return dt.replace(tzinfo=timezone.utc) + return dt.astimezone(timezone.utc) + except (TypeError, ValueError): + pass + formats = [ - "%a, %d %b %Y %H:%M:%S %z", # RFC 822 with numeric tz offset - "%a, %d %b %Y %H:%M:%S %Z", # RFC 822 with tz abbreviation (GMT/UTC) - "%a, %d %b %Y %H:%M %Z", # RFC 822 without seconds (MyFXBook) - "%a, %d %b %Y %H:%M %z", # RFC 822 without seconds, numeric offset "%Y-%m-%dT%H:%M:%S%z", # ISO 8601 with offset "%Y-%m-%d %H:%M:%S", # ISO 8601 naive (assume UTC) ] diff --git a/signal_aggregator.py b/signal_aggregator.py index d24dc7d..9afb7b5 100755 --- a/signal_aggregator.py +++ b/signal_aggregator.py @@ -50,6 +50,12 @@ class TradeSetup: pip_value: float # Entry archetype ("GENERIC", "XAU_PB") — selects trail params. strategy: str = "GENERIC" + # Trailing exit params stamped from the PRODUCING config, so per-TF + # factory overrides (e.g. M15 arm 0.70, M5 dist 0.85) reach every + # consumer. None = legacy/unstamped setup — formatters fall back to + # platform defaults; 0.0 means trailing is genuinely disabled. + trail_arm_r: Optional[float] = None + trail_dist_r: Optional[float] = None @dataclass @@ -228,7 +234,12 @@ class SignalAggregator: report.action_recommendation = "NO TRADE — Outside tradeable session" report.trade_setup = None except Exception: - pass + # Fail-open by design (an unparseable timestamp shouldn't + # kill the scan) but never silently: this filter guards + # real money hours. + logger.exception( + f"{symbol}: live session filter errored — filter SKIPPED this scan" + ) return report @@ -538,6 +549,7 @@ class SignalAggregator: min_tp = (self.config.signals.round_trip_cost * self.config.signals.min_tp_cost_multiple) if 1.5 * sl_dist < min_tp: + _reject("spread_gate") return report.trade_setup = TradeSetup( @@ -553,7 +565,10 @@ class SignalAggregator: reward_pips=1.5 * sl_dist / pip_value, pip_value=pip_value, strategy="XAU_PB", + trail_arm_r=self.config.signals.pb_trail_arm_r, + trail_dist_r=self.config.signals.pb_trail_dist_r, ) + _reject("accepted") # funnel tick, mirrors the CP path if report.signal_quality not in ("A+", "A", "B"): report.signal_quality = "B" report.final_signal = (SignalStrength.BUY if is_buy @@ -629,6 +644,10 @@ class SignalAggregator: ).hour except Exception: close_h = None + logger.warning( + f"{report.symbol}: bar-close hour underivable — " + f"SELL hour gate skipped" + ) if close_h in cfg.sell_blocked_hours_utc: _reject("sell_hour") return @@ -683,6 +702,8 @@ class SignalAggregator: reward_pips=height / pip_value, pip_value=pip_value, strategy="XAU_CP", + trail_arm_r=self.config.signals.trail_arm_r, + trail_dist_r=self.config.signals.trail_dist_r, ) if report.signal_quality not in ("A+", "A", "B"): report.signal_quality = "B" @@ -814,7 +835,10 @@ class SignalAggregator: f"DAY FILTER: {day_names.get(bar_wd, '?')} not in allowed days" ) except Exception: - pass + # Fail-open by design, but never silently (see live wrapper). + logger.exception( + f"{symbol}: session/day filter errored — filter SKIPPED" + ) # ── 6c. Transitioning regime quality boost ────────────────── if report.market_regime == "transitioning" and report.signal_quality in ("B", "C", "D"): @@ -922,6 +946,10 @@ class SignalAggregator: ).hour except Exception: close_h = None + logger.warning( + f"{report.symbol}: bar-close hour underivable — " + f"SELL hour gate skipped" + ) if close_h in self.config.xau_strategy.sell_blocked_hours_utc: report.warnings.append( f"SELL FILTER: {close_h:02}:00 UTC entry blocked for shorts" @@ -1048,6 +1076,8 @@ class SignalAggregator: risk_pips=round(sl_distance / pip_value, 1), reward_pips=round(tp_distance / pip_value, 1), pip_value=pip_value, + trail_arm_r=self.config.signals.trail_arm_r, + trail_dist_r=self.config.signals.trail_dist_r, ) def _check_mtf_agreement(self, report: SignalReport) -> bool: @@ -1165,6 +1195,11 @@ class SignalAggregator: atr = primary.key_levels.get("atr", 0) if atr > 0 and score != 0: + # Distance is measured from the FARTHEST of the 3 tracked + # swings (sorted()[0] = deepest low / highest high), so the + # penalty fires earlier than a nearest-swing rule would. + # Calibrated as-is in every validated book — changing the + # anchor is a signal-logic change requiring re-validation. if score > 0: swing_lows = sorted( v for k, v in primary.key_levels.items() if k.startswith("swing_low_") @@ -1445,6 +1480,8 @@ class SignalAggregator: risk_pips=risk_pips, reward_pips=reward_pips, pip_value=pip_value, + trail_arm_r=self.config.signals.trail_arm_r, + trail_dist_r=self.config.signals.trail_dist_r, ) def _collect_reasons(self, report: SignalReport): diff --git a/telegram_bot.py b/telegram_bot.py index cd4ae62..551cc7b 100755 --- a/telegram_bot.py +++ b/telegram_bot.py @@ -398,12 +398,17 @@ def format_signal_message(report: SignalReport, spread_detail: bool = True) -> s lines.append(f"⚖️ RR: 1:{ts.risk_reward:.1f}") # Trailing-stop management plan (matches backtested exit model: - # TP1 is advisory, the trail is the exit). Pullback setups run - # tighter stops and arm later. - sig_cfg = PlatformConfig().signals + # TP1 is advisory, the trail is the exit). Params are stamped on + # the setup by the producing config so per-TF factory overrides + # (M15 arm 0.70, M5 dist 0.85) are what subscribers see; only a + # legacy unstamped setup falls back to platform defaults. is_pb = getattr(ts, "strategy", "GENERIC").endswith("_PB") - arm_r = sig_cfg.pb_trail_arm_r if is_pb else sig_cfg.trail_arm_r - dist_r = sig_cfg.pb_trail_dist_r if is_pb else sig_cfg.trail_dist_r + arm_r = getattr(ts, "trail_arm_r", None) + dist_r = getattr(ts, "trail_dist_r", None) + if arm_r is None or dist_r is None: + sig_cfg = PlatformConfig().signals + arm_r = sig_cfg.pb_trail_arm_r if is_pb else sig_cfg.trail_arm_r + dist_r = sig_cfg.pb_trail_dist_r if is_pb else sig_cfg.trail_dist_r if arm_r > 0: risk_dist = abs(ts.entry_price - ts.stop_loss) if ts.direction == "BUY": @@ -574,6 +579,7 @@ _alert_min_quality = _APP.alert_min_quality # signal doesn't re-send an identical message every candle. _last_alert_fp: Dict[str, tuple] = {} _last_alert_ts: Dict[str, float] = {} +_last_alert_setup: Dict[str, tuple] = {} # key → (entry, risk) of last ALERTED setup _ALERT_REARM_SECONDS = 3600 # identical signal re-alerts after 1h @@ -582,40 +588,54 @@ def _passes_quality(quality: str) -> bool: def _alert_fingerprint(report: SignalReport) -> tuple: - """Alert identity for dedup. Includes a risk-relative entry bucket: - a persisting signal whose entry drifts a few ticks between candles - keeps the same fingerprint, while a setup that moved by half its risk - distance or more (new pullback, new swing anchor) re-alerts instead - of being swallowed for the full re-arm hour.""" + """Alert identity for dedup: signal, quality, setup direction. + + Entry movement is checked separately in _should_alert against the + last ALERTED entry/risk (a stable reference). Deriving a price bucket + from the CURRENT SL distance here made the fingerprint churn on + normal ATR/anchor drift — near-duplicate alerts every candle.""" setup = report.trade_setup - entry_bucket = None - if setup is not None: - sl_dist = abs(setup.entry_price - setup.stop_loss) - entry_bucket = round(setup.entry_price / max(sl_dist * 0.5, 1e-9)) return ( report.final_signal.name, report.signal_quality, setup.direction if setup else None, - entry_bucket, ) def _should_alert(key: str, report: SignalReport) -> bool: """True when this signal is new/changed for the key, or re-armed. + A persisting identical signal dedups for the re-arm hour, but a setup + whose entry moved >= half the last-alerted risk distance (new + pullback/swing anchor) re-alerts immediately. + Pure check — call _commit_alert only after the message actually went out, so a failed Telegram send doesn't consume the alert for an hour. """ fp = _alert_fingerprint(report) - return not ( - _last_alert_fp.get(key) == fp - and time.time() - _last_alert_ts.get(key, 0) < _ALERT_REARM_SECONDS - ) + if _last_alert_fp.get(key) != fp: + return True + if time.time() - _last_alert_ts.get(key, 0) >= _ALERT_REARM_SECONDS: + return True + setup = report.trade_setup + last = _last_alert_setup.get(key) + if setup is not None and last is not None: + last_entry, last_risk = last + if abs(setup.entry_price - last_entry) >= 0.5 * max(last_risk, 1e-9): + return True + return False def _commit_alert(key: str, report: SignalReport) -> None: _last_alert_fp[key] = _alert_fingerprint(report) _last_alert_ts[key] = time.time() + setup = report.trade_setup + if setup is not None: + _last_alert_setup[key] = ( + setup.entry_price, abs(setup.entry_price - setup.stop_loss) + ) + else: + _last_alert_setup.pop(key, None) # ── Candle-boundary scheduler ──────────────────────────────────────── @@ -673,7 +693,8 @@ async def _scan_timeframe( actionable = [ r for r in reports - if _passes_quality(r.signal_quality) + if r.final_signal != SignalStrength.NEUTRAL # same gate as breach path + and _passes_quality(r.signal_quality) and _should_alert(f"{r.symbol}_{tf.value}", r) ] if actionable: diff --git a/test_all.py b/test_all.py index 36c65b3..021e12e 100644 --- a/test_all.py +++ b/test_all.py @@ -814,6 +814,18 @@ def _(): assert dt.tzinfo is not None +@test("NewsEngine._parse_rss_date named zones carry their real offset") +def _(): + # EST is UTC−5: 12:00 EST == 17:00 UTC. The old "%Z" path accepted the + # abbreviation but stamped UTC — a silent 5-hour shift. + dt = NewsEngine._parse_rss_date("Mon, 20 Jul 2026 12:00:00 EST") + assert dt is not None and dt.tzinfo is not None + assert dt.astimezone(timezone.utc).hour == 17 + # GMT stays a no-op; without-seconds (MyFXBook) form still parses. + dt2 = NewsEngine._parse_rss_date("Mon, 20 Jul 2026 12:00 GMT") + assert dt2 is not None and dt2.astimezone(timezone.utc).hour == 12 + + @test("NewsEngine._parse_rss_date ISO format") def _(): dt = NewsEngine._parse_rss_date("2025-05-12T14:30:00+00:00") @@ -899,6 +911,27 @@ def _(): assert risk == "caution" +@test("NewsEngine.get_event_risk_level grades from the soonest event") +def _(): + # Far event listed FIRST: an unsorted feed must still yield danger + # for the nearer event, not caution for the first match. + ne = NewsEngine(NewsConfig()) + context = FundamentalContext() + context.events_upcoming_1h = [ + EconomicEvent( + time=datetime.now(timezone.utc) + timedelta(minutes=55), + currency="USD", impact="high", event="GDP", + ), + EconomicEvent( + time=datetime.now(timezone.utc) + timedelta(minutes=10), + currency="USD", impact="high", event="NFP", + ), + ] + risk, desc = ne.get_event_risk_level("XAU/USD", context) + assert risk == "danger" + assert "NFP" in desc + + @test("NewsEngine.get_event_risk_level clear when no events") def _(): ne = NewsEngine(NewsConfig()) @@ -1113,6 +1146,24 @@ def _(): assert ts.direction == "BUY" assert ts.risk_reward == 2.0 assert ts.entry_price > ts.stop_loss + assert ts.trail_arm_r is None and ts.trail_dist_r is None # unstamped default + + +@test("trade setups carry trail params from the producing config") +def _(): + config = PlatformConfig() + config.signals.trail_arm_r = 0.70 + config.signals.trail_dist_r = 0.85 + agg = SignalAggregator(config) + report = SignalReport(symbol="XAU/USD", display_name="XAU/USD", timestamp="now") + report.final_signal = SignalStrength.BUY + report.final_score = 40.0 + report.signal_quality = "A+" + report.current_price = 3300.0 + ts = agg._generate_xau_atr_trade_setup(report, 10.0) + assert ts is not None, "A+ setup with ATR 10 should generate" + assert ts.trail_arm_r == 0.70, "setup must carry the producing config's arm" + assert ts.trail_dist_r == 0.85, "setup must carry the producing config's dist" @test("SignalAggregator._check_mtf_agreement with 2 bullish") @@ -1569,6 +1620,30 @@ def _(): assert "Indicators" in msg +@test("format_signal_message renders stamped per-TF trail plan") +def _(): + def _setup(**extra): + return TradeSetup( + pair="XAU/USD", display_name="XAU/USD", direction="BUY", + entry_price=3300.0, stop_loss=3290.0, take_profit_1=3315.0, + take_profit_2=3325.0, risk_reward=1.5, risk_pips=100, + reward_pips=150, pip_value=0.10, **extra, + ) + report = SignalReport(symbol="XAU/USD", display_name="XAU/USD", timestamp="now") + report.final_signal = SignalStrength.BUY + report.signal_quality = "A" + # Stamped setup (M15-style overrides) must render the stamped values… + report.trade_setup = _setup(trail_arm_r=0.70, trail_dist_r=0.85) + msg = format_signal_message(report) + assert "+0.7R" in msg, "must render the stamped arm, not the 0.6 default" + assert "$8.50" in msg, "must render the stamped dist × risk, not the default" + # …while a legacy unstamped setup falls back to platform defaults. + report.trade_setup = _setup() + msg = format_signal_message(report) + assert "+0.6R" in msg + assert "$7.50" in msg + + @test("_format_scan_summary produces valid text") def _(): config = PlatformConfig() @@ -2116,6 +2191,10 @@ def _(): src = open(_gb.__file__, encoding="utf-8").read() assert "_apply_regime_filter_live(report, slice_dfs)" in src assert "backtest.regime_filter" in src + # …and loads the regime-filter TF like live analyze_pair does, or the + # method silently no-ops for M5/M15 primaries (confirm maps lack H4). + assert "regime_filter_timeframe not in timeframes" in src, \ + "replay must fetch/slice the regime-filter TF when the gate is on" @test("disabled_indicators=['macd'] removes the pillar but keeps the histogram") @@ -2181,6 +2260,40 @@ def _(): break +@test("Fresh fetch missing the just-closed bar retries once, then warns") +def _(): + def bars_ending(last): + idx = pd.date_range(end=last, periods=300, freq="15min", tz="UTC") + v = np.linspace(3000.0, 3100.0, 300) + return pd.DataFrame({"Open": v, "High": v + 1, "Low": v - 1, + "Close": v, "Volume": 1000.0}, index=idx) + + for _attempt in range(2): + provider = ForexDataProvider(PlatformConfig()) + started = pd.Timestamp.now(tz="UTC").floor("15min") + last_closed = started - pd.Timedelta(minutes=15) + lagging = bars_ending(last_closed - pd.Timedelta(minutes=15)) + complete = bars_ending(last_closed) + with patch("data_provider._is_forex_weekend", return_value=False), \ + patch.object(ForexDataProvider, "FRESH_FETCH_RETRY_SECONDS", 0.0): + # (a) lagging first response → one retry returns the closed bar. + with patch.object(provider, "_fetch_twelve_data", + side_effect=[lagging.copy(), complete.copy()]) as mf: + served = provider.fetch_ohlcv("XAU/USD", Timeframe.M15) + # (b) complete first response → no retry at all. + provider._cache.clear() + provider._cache_timestamps.clear() + with patch.object(provider, "_fetch_twelve_data", + return_value=complete.copy()) as mf2: + provider.fetch_ohlcv("XAU/USD", Timeframe.M15) + if pd.Timestamp.now(tz="UTC").floor("15min") != started: + continue # boundary crossed mid-test — retry once + assert mf.call_count == 2, "lagging response must trigger exactly one retry" + assert served.index[-1] == last_closed, "retry result must be served" + assert mf2.call_count == 1, "boundary-complete response must not retry" + break + + @test("Alert dedup: drift dedups, materially-moved setup re-alerts, no early commit") def _(): import telegram_bot as tb @@ -2201,17 +2314,23 @@ def _(): key = "TEST_ALERT_FP" tb._last_alert_fp.pop(key, None) tb._last_alert_ts.pop(key, None) - r1 = rpt(3400.0, 3392.0) # risk $8 → $4 buckets + tb._last_alert_setup.pop(key, None) + r1 = rpt(3400.0, 3392.0) # risk $8 → re-alert threshold $4 assert tb._should_alert(key, r1) assert tb._should_alert(key, r1), "_should_alert must not commit state itself" tb._commit_alert(key, r1) assert not tb._should_alert(key, r1), "identical signal must dedup" assert not tb._should_alert(key, rpt(3400.5, 3392.5)), \ - "sub-bucket drift must dedup" + "sub-threshold drift must dedup" + assert not tb._should_alert(key, rpt(3401.0, 3392.6)), \ + "SL-distance drift (ATR/anchor wobble) must not re-fingerprint" assert tb._should_alert(key, rpt(3412.0, 3404.0)), \ "setup moved 1.5x risk — must re-alert" + assert tb._should_alert(key, rpt(3395.5, 3387.5)), \ + "setup moved half risk AGAINST the entry — must re-alert too" tb._last_alert_fp.pop(key, None) tb._last_alert_ts.pop(key, None) + tb._last_alert_setup.pop(key, None) @test("Backtester --confirm-tfs defaults to None (factory's live map)") @@ -2250,6 +2369,31 @@ def _(): assert t2.entry_time == idx[2] and t2.entry_price == 110.0 and t2.sl == 102.0 +@test("Max drawdown counts the decline from flat starting equity") +def _(): + from gpu_backtest import Trade, compute_metrics + + def mk(r): + t = Trade(pair="XAU/USD", + entry_time=pd.Timestamp("2026-01-05 10:00", tz="UTC"), + entry_price=3300.0, sl=3292.0, tp1=3312.0, tp2=3320.0, + direction="BUY", pip_value=0.10, risk_pips=80.0, + reward_pips=120.0, rr_planned=1.5, quality="B", + final_score=30.0, final_confidence=60.0, + market_regime="trending_up") + t.exit_reason = "SL" if r < 0 else "TP1" + t.r_multiple = r + return t + + # Opening losing streak: equity 0 → −3R with no prior peak. The old + # high-water started at the first trade's cum (−1) and reported 2R. + m = compute_metrics([mk(-1.0), mk(-1.0), mk(-1.0)]) + assert m.max_drawdown_r == 3.0, f"expected 3.0, got {m.max_drawdown_r}" + # Unchanged when a peak precedes the streak. + m2 = compute_metrics([mk(2.0), mk(-1.0), mk(-1.0), mk(-1.0)]) + assert m2.max_drawdown_r == 3.0, f"expected 3.0, got {m2.max_drawdown_r}" + + # ══════════════════════════════════════════════════════════════════════ # MODULE 12: Chart-pattern entry (XAU_CP) # ══════════════════════════════════════════════════════════════════════ @@ -2997,7 +3141,10 @@ def _(): "w", suffix=".env", delete=False, encoding="utf-8") fh.write("# comment\n\nPLAIN=abc\nexport EXPORTED=def\n" 'QUOTED="with spaces"\nSINGLE=\'sq\'\nEXISTING=from-dotenv\n' - "NOT A KV LINE\n") + "NOT A KV LINE\n" + "COMMENTED=abc123 # prod key\n" + 'QUOTED_HASH="keep #this"\n' + "HASH_IN_VALUE=abc#123\n") fh.close() with patch.dict(os.environ, {"EXISTING": "from-shell"}, clear=True): load_dotenv(fh.name) @@ -3007,6 +3154,12 @@ def _(): assert os.environ["SINGLE"] == "sq" assert os.environ["EXISTING"] == "from-shell", \ "a real shell export must beat .env" + assert os.environ["COMMENTED"] == "abc123", \ + "inline comment must be stripped, not absorbed into the value" + assert os.environ["QUOTED_HASH"] == "keep #this", \ + "quoted values keep their hashes" + assert os.environ["HASH_IN_VALUE"] == "abc#123", \ + "a hash with no preceding whitespace is part of the value" load_dotenv("/nonexistent/.env") # missing file is a silent no-op