diff --git a/CLAUDE.md b/CLAUDE.md index 78d50a6..ef914df 100644 --- a/CLAUDE.md +++ b/CLAUDE.md @@ -12,7 +12,7 @@ Every numeric threshold in `config.py` and `pair_signals/xauusd.py` was backtest ```bash python test_all.py # QA suite, no network (mocked); prints its own count - # (460 tests / 31 modules as of 2026-09-06) + # (468 tests / 31 modules as of 2026-09-08) python main.py --pair XAUUSD --timeframe H1 --detail # CLI signal report (--json for full dump, # --panel for the advisory LLM agent panel, --outlook for the # hourly AI market brief; the violent-move watch line appears @@ -63,14 +63,14 @@ Signal currency everywhere is `SignalStrength` (−2..+2). DataFrames are UTC tz - **`providers/`** — one class per provider subclassing `BaseDataProvider` (`providers/base.py`), which owns everything the pipeline depends on: TTL+boundary cache, forming-candle drop, freshness check, validate/clean, paginated `fetch_range` (feeds the backtester from any provider), optional tick streaming. Subclasses implement only `_fetch_raw` + symbol/interval maps. **Cleaning rule (2026-09-07):** the signal-side outlier filter is a bad-print filter, not a volatility filter — a >10σ close-to-close bar is dropped only when the NEXT close snaps back ≥80% of the way (`OUTLIER_SIGMA`/`OUTLIER_REVERT_FRACTION` on the base class); persisting moves (NFP/CPI/Sunday-gap bars) and the newest bar are always kept. The previous unconditional 10σ rule deleted ~36 real M5 bars/yr in the backtester and a *different* ~24/yr live (window-dependent σ), hiding e.g. the 2026-09-04 12:30 UTC NFP bar from ATR/exhaustion math. Oanda fetches retry 401/5xx/network faults along `RETRY_DELAYS_S` (2/5/10 s) under a 45 s `RETRY_BUDGET_S`; a scan that still loses its primary frame is logged with `scan_kind='failed'`. Registry + `make_data_provider(config, name=None)` in `providers/__init__.py`; config.json `data_providers.active` selects the live candle feed and the optional `data_providers.streaming` scalar selects a different provider for the tick stream. A test enforces that subclasses don't override shared base methods (`supports_streaming`/`iter_ticks` are the sanctioned streaming override points). Tiingo streams mid-price ticks (`wss://api.tiingo.com/fx`) — do not re-add spread display from its free FX feed (two-tier indicative quotes, removed 2026-07-23); the no-spread rule extends to the WS stream. - **`engine.py`** — `TechnicalEngine`: 9 scored indicators (ema_trend, rsi, macd, bollinger, adx, roc, keltner, divergence, candlestick) + ATR as context. Regime classification (trending_up/down, `trending` = mixed/direction-UNKNOWN, transitioning, ranging) drives per-indicator weight multipliers and regime-aware RSI reinterpretation. Compute order matters (ATR first; MACD always computed even when not a pillar — divergence needs the histogram). Private series ride on the result as `_underscore` attrs; scorers/formatters skip anything `_`-prefixed or weight-0. -- **`signal_aggregator.py`** — MTF agreement, diminishing-decay penalty system, exhaustion hard block, quality grading, then an ordered chain of post-grade gates (session, XAU SELL quality/vol/hour, regime blocks, M5 noise gates), then three setup generators: XAU ATR (A+/A), generic swing-based, plus the M5 pullback (`XAU_PB`) and chart-pattern (`XAU_CP`, `chart_patterns.py`) entry paths. The PB/CP generators run AFTER the gate chain and force quality to "B"; the CP path deliberately accepts grade-F composites and re-checks only session/day/big-candle (via warning prefixes) plus its own SELL/vol/RSI gates — it does NOT re-check exhaustion or the M5 min-score/MTF gates. **This is validated behavior, not a bug** (2026-08-04 audit: 226/586 trades in the shipped M5 12mo book are XAU_CP, +38.9R, 104 of them exhaustion-flagged and net-positive) — changing it is a rule-1 revalidation event. `TradeSetup.strategy` selects trailing params; trail params are stamped on the setup from the *producing* config so per-TF overrides reach formatters. +- **`signal_aggregator.py`** — MTF agreement, diminishing-decay penalty system, exhaustion hard block, quality grading, then an ordered chain of post-grade gates (session, XAU SELL quality/vol/hour, regime blocks, M5 noise gates), then three setup generators: XAU ATR (A+/A), generic swing-based, plus the M5 pullback (`XAU_PB`) and chart-pattern (`XAU_CP`, `chart_patterns.py`) entry paths. Step 7 lives in `_attach_core_setup`; a final pass after the PB/CP archetypes (`_drop_stale_spread_gate_vetoes`, 2026-09-08) moves a core generator's `SPREAD GATE:` veto to `cleaned_warnings` whenever a later generator shipped a setup (metadata only — books unchanged; the exported `warnings` text column can differ; the prefix is the `_SPREAD_GATE_PREFIX` constant shared by the emitters and both cleanups). The PB/CP generators run AFTER the gate chain and force quality to "B"; the CP path deliberately accepts grade-F composites and re-checks only session/day/big-candle (via warning prefixes) plus its own SELL/vol/RSI gates — it does NOT re-check exhaustion or the M5 min-score/MTF gates. **This is validated behavior, not a bug** (2026-08-04 audit: 226/586 trades in the shipped M5 12mo book are XAU_CP, +38.9R, 104 of them exhaustion-flagged and net-positive) — changing it is a rule-1 revalidation event. `TradeSetup.strategy` selects trailing params; trail params are stamped on the setup from the *producing* config so per-TF overrides reach formatters. - **`pair_signals/`** — per-pair config factories. **Always use the factory (`produce_signal` / `make_config`), never bare `PlatformConfig()`** — the factory applies the config.json overlay first, then the validated per-TF deltas (M5 hour-9 block, M15 trail arm 0.70, M15 SELL-hour block, M5 SELL-in-transitioning block, H1 all-shorts block (2026-09-03), etc.; evidence in the docstring). A test enforces factory ≡ inline-config equivalence. Confirmation map: M5→[M15,H1], M15→[M5,H1], H1→[M15,H4], H4→[H1,D1]. -- **`signal_log.py` / `advisory_outcome.py`** — SQLite signal-history logger (stdlib only). Hooked into `SignalAggregator.analyze_pair` only (the live entry point — the backtester never writes here, preserving parity), so every live scan from every surface (Telegram `telegram`, CLI `cli`, dashboard `dashboard`) lands one row in `logs/signals.db`: signal/grade/price, MTF + trade-setup + reasons JSON, full per-TF indicator snapshots, the last 14 closed OHLCV candles per fetched TF, and the news at scan time (event risk + calendar events + scored headlines, re-fetched via the NewsEngine class-level caches). Config.json `signal_log` block (`enabled`, `path`, `log_blocked_scans`; env `SIGNAL_LOG_ENABLED`/`SIGNAL_LOG_PATH`/`SIGNAL_LOG_BLOCKED_SCANS`). **Fail-safe by contract** — a logging failure can never break a scan. Panel fields arrive post-hoc: `log_signal` stashes the rowid and `agent_panel.attach_panel` writes verdict + creation/cache provenance + payload hash onto the same row. **Advisory outcomes (2026-09):** EVERY row with a trade setup is seeded at log time (not only panel rows); same-setup re-logs within 60 min from any surface (`setup_key` = tf|direction|entry|stop|strategy) are `DUPLICATE` rows pointing at the canonical row via `advisory_event_id`, so sum `advisory_r_multiple` over `RESOLVED` rows only. The versioned model advances from later stored closed-candle snapshots (next-open reanchoring, intrabar peak with close-only ratchet, SL-first, scheduled weekend/21:00-UTC closures allowed, 300-bar timeout, $0.30/$0.65 views); when a snapshot no longer overlaps the outcome's last bar it rebuilds the stream from every stored snapshot of that TF before giving up. It is a cleaned-mid evidence model, not execution P&L. **Post-gate truth columns:** `gate_reason` (the warning that removed the setup), `actionable` (grade ≥ C AND setup — the `action` text is written BEFORE the gate chain and can say "STRONG SELL — High-quality setup" on a D row), `cleaned_warnings_json` (what the XAU_CP acceptance stripped), `bar_range_median` (= the misnamed `spread` column: median High-Low, NOT a broker spread), `scan_kind` (`blocked` weekend/day early returns carry no news/candles; `failed` = the primary fetch failed — price 0, no signal — readers skip it like `blocked`), `config_fingerprint` (gate config hash — surfaces config drift between processes). **News is content-addressed:** `headlines_json`/`events_json` are interned into `news_snapshots` (they were 73% of a 667 MB DB); read through the `signals_with_news` view or `signal_log.read_news`. Readers must open with `mode=ro`, never `immutable=1` (blind to WAL rows). **Weight-research columns (2026-09):** `bar_ts` (the primary bar LABEL the scan closed on — same string as `indicators_json`, indexed with `timeframe`), `engine_fingerprint` (hash of the SCORING config: indicator params, thresholds, the regime weight table in force, TFs — `SignalAggregator.engine_config_fingerprint`, the gate fingerprint's twin), `code_version` (short git SHA), `penalty_tags_json`/`penalty_total`/`momentum_fading`/`exhaustion_count`, and the post-hoc `watch_level`/`watch_kinds`/`watch_line` written by `signal_log.update_watch` from `violent_watch.attach_watch` (level and kinds are stored even when the display line is off). Every `indicators_json` entry also carries `eff_weight` (base weight × regime multiplier = the weight the composite actually used; `engine.regime_weight_table` is the single source of truth) and `wscore` (its contribution) — null for skipped indicators (weight 0, `_`-prefixed); **divergence and candlestick entries exist only when they fire, so absence is "no vote", not a zero vote**. **Table `scan_outcomes`:** model-free forward moves per scan (`fwd_1h/4h/24h` in $ and ATRs, `mfe_24h_atr`/`mae_24h_atr`, `burst_1h` = a ≥3×ATR bar within the hour, `bars_used`, `complete`), measured from the row's own last closed bar on the stored candle stream; a horizon stays NULL until it has elapsed, and `complete` needs the full 24 h with no gap over 3 bars beyond the scheduled weekend/17:00-NY closures (`advisory_outcome._has_continuity`). `SignalLogger.label_scan_outcomes` runs opportunistically at the end of `log()` (at most once per 60 s per process, 150 rows and ~2 s per pass, idempotent — a row is rewritten only when its numbers change), so no daemon is needed; `scripts/label_scan_outcomes.py` is the backlog driver (`--dry-run` counts read-only). Readers must dedup by the FIRST row per (`timeframe`, `bar_ts`) — several surfaces log the same bar. Tools: `analyze_signal_log.py` (`--stored-candles` DB-only replay with event/opportunity/one-position books + persona scorecard, `--news` news-sentiment lead/lag + extremes + outcome buckets, `--parity` telegram-vs-dashboard divergence report, `--scan-outcomes` forward-move + hit-rate table by TF×signal/grade with a NEUTRAL baseline and burst rate by watch level), `scripts/backfill_signal_outcomes.py` (seed/dedup/advance legacy rows), `scripts/label_scan_outcomes.py` (label/relabel scan outcomes), `scripts/migrate_signal_log_news.py` (intern legacy blobs; `--vacuum` with both writers stopped). +- **`signal_log.py` / `advisory_outcome.py`** — SQLite signal-history logger (stdlib only). Hooked into `SignalAggregator.analyze_pair` only (the live entry point — the backtester never writes here, preserving parity), so every live scan from every surface (Telegram `telegram`, CLI `cli`, dashboard `dashboard`) lands one row in `logs/signals.db`: signal/grade/price, MTF + trade-setup + reasons JSON, full per-TF indicator snapshots, the last 14 closed OHLCV candles per fetched TF, and the news at scan time (event risk + calendar events + scored headlines, re-fetched via the NewsEngine class-level caches). Config.json `signal_log` block (`enabled`, `path`, `log_blocked_scans`; env `SIGNAL_LOG_ENABLED`/`SIGNAL_LOG_PATH`/`SIGNAL_LOG_BLOCKED_SCANS`). **Fail-safe by contract** — a logging failure can never break a scan. Panel fields arrive post-hoc: `log_signal` stashes the rowid and `agent_panel.attach_panel` writes verdict + creation/cache provenance + payload hash onto the same row. **Advisory outcomes (2026-09):** EVERY row with a trade setup is seeded at log time (not only panel rows); same-setup re-logs within 60 min from any surface (`setup_key` = tf|direction|entry|stop|strategy) are `DUPLICATE` rows pointing at the canonical row via `advisory_event_id`, so sum `advisory_r_multiple` over `RESOLVED` rows only. The versioned model advances from later stored closed-candle snapshots (next-open reanchoring, intrabar peak with close-only ratchet, SL-first, scheduled weekend/21:00-UTC closures allowed, 300-bar timeout, $0.30/$0.65 views); when a snapshot no longer overlaps the outcome's last bar it rebuilds the stream from every stored snapshot of that TF before giving up. It is a cleaned-mid evidence model, not execution P&L. **Post-gate truth columns:** `gate_reason` (the warning that removed the setup), `actionable` (grade ≥ C AND setup — the `action` text is written BEFORE the gate chain and can say "STRONG SELL — High-quality setup" on a D row), `cleaned_warnings_json` (warnings a shipped setup superseded — the XAU_CP acceptance's stripped vetoes and, since 2026-09-08, a stale core spread-gate veto; non-empty is NOT a CP marker, use `setup_json.strategy`), `bar_range_median` (= the misnamed `spread` column: median High-Low, NOT a broker spread), `scan_kind` (`blocked` weekend/day early returns carry no news/candles; `failed` = the primary fetch failed — price 0, no signal — readers skip it like `blocked`), `config_fingerprint` (gate config hash — surfaces config drift between processes). **News is content-addressed:** `headlines_json`/`events_json` are interned into `news_snapshots` (they were 73% of a 667 MB DB); read through the `signals_with_news` view or `signal_log.read_news`. Readers must open with `mode=ro`, never `immutable=1` (blind to WAL rows). **Weight-research columns (2026-09):** `bar_ts` (the primary bar LABEL the scan closed on — same string as `indicators_json`, indexed with `timeframe`), `engine_fingerprint` (hash of the SCORING config: indicator params, thresholds, the regime weight table in force, TFs — `SignalAggregator.engine_config_fingerprint`, the gate fingerprint's twin), `code_version` (short git SHA), `penalty_tags_json`/`penalty_total`/`momentum_fading`/`exhaustion_count`, and the post-hoc `watch_level`/`watch_kinds`/`watch_line` written by `signal_log.update_watch` from `violent_watch.attach_watch` (level and kinds are stored even when the display line is off). Every `indicators_json` entry also carries `eff_weight` (base weight × regime multiplier = the weight the composite actually used; `engine.regime_weight_table` is the single source of truth) and `wscore` (its contribution) — null for skipped indicators (weight 0, `_`-prefixed); **divergence and candlestick entries exist only when they fire, so absence is "no vote", not a zero vote**. **Table `scan_outcomes`:** model-free forward moves per scan (`fwd_1h/4h/24h` in $ and ATRs, `mfe_24h_atr`/`mae_24h_atr`, `burst_1h` = a ≥3×ATR bar within the hour, `bars_used`, `complete`), measured from the row's own last closed bar on the stored candle stream; a horizon stays NULL until it has elapsed, and `complete` needs the full 24 h with no gap over 3 bars beyond the scheduled weekend/17:00-NY closures (`advisory_outcome._has_continuity`). `SignalLogger.label_scan_outcomes` runs opportunistically at the end of `log()` (at most once per 60 s per process, 150 rows and ~2 s per pass, idempotent — a row is rewritten only when its numbers change), so no daemon is needed; `scripts/label_scan_outcomes.py` is the backlog driver (`--dry-run` counts read-only). Readers must dedup by the FIRST row per (`timeframe`, `bar_ts`) — several surfaces log the same bar. Tools: `analyze_signal_log.py` (`--stored-candles` DB-only replay with event/opportunity/one-position books + persona scorecard, `--news` news-sentiment lead/lag + extremes + outcome buckets, `--parity` telegram-vs-dashboard divergence report, `--scan-outcomes` forward-move + hit-rate table by TF×signal/grade with a NEUTRAL baseline and burst rate by watch level), `scripts/backfill_signal_outcomes.py` (seed/dedup/advance legacy rows), `scripts/label_scan_outcomes.py` (label/relabel scan outcomes), `scripts/migrate_signal_log_news.py` (intern legacy blobs; `--vacuum` with both writers stopped). - **`news_engine.py`** — Forex Factory calendar (MyFXBook RSS fallback) + RSS keyword sentiment with gold-inverse keywords. Class-level caches (fresh engine per scan). Optional DeepSeek enrichment via `llm_client.py` (requests-only, no SDK): batched headline sentiment overriding keyword scores, gold-direction on release surprises, and a market-summary blurb (`SignalReport.news_summary`) — **display/warning-only, keyword fallback on any failure, inert without a key**; prompts are static module constants (DeepSeek prefix-cache pricing) and requests send `thinking: disabled` (V4-Flash reasoning otherwise blows the timeout). By XAU-tuned default, event risk and sentiment **warn but do not gate signals** (`event_risk_caps_signals=False`, `sentiment_affects_signals=False`) — the LLM does not change that. **Headline pool (2026-09-04):** only headlines ≤ 4 h old with a parsable publish time (`headline_max_age_seconds=14400`, undated items dropped), and `get_fundamental_context` drops headlines whose every relevant score is inside ±0.1 (`_is_neutral_headline`, `NEUTRAL_HEADLINE_THRESHOLD`) before the per-currency mean — so `news_sentiment` from that date is a mean over directional headlines only (bolder values; log comparisons must split at the deploy date). The market summary prompt writes for someone who has never traded (no acronyms), sends headlines newest-first with the platform's own gold direction, and caps at 400 chars. **Verdict 2026-09-04** (`backtest/research/news_sentiment_value_2026-09-04.md`): `news_sentiment` is a trailing-24h echo of price (r≈0.67 with the past day, ≈0.03 with the next; flips 44 h after tops / 16 h after bottoms) — both gating flags stay False; re-check with `analyze_signal_log.py --stored-candles --news`. - **`agent_panel.py`** — ai-hedge-fund-style advisory panel: 4 DeepSeek persona calls (macro/technical/risk/contrarian, concurrent on a panel-private pool) + 1 synthesizer producing an **ADVISORY-ONLY** consensus (`AGREE/NEUTRAL/OPPOSE` + confidence + rationale) written onto `SignalReport.panel_*` fields. Attached **post-hoc** by consumers (`telegram_bot` alert path, `main --panel`, `dashboard_publisher`) — never runs inside `analyze_from_dataframes` (backtest parity, test-locked) and no gate reads its fields. Persona payloads are role-filtered: macro gets high-impact events plus only code-scored directional news; technical gets strategy/pattern, primary indicators, MTF, levels, and ATR-normalized candles; risk gets setup strategy, strategy cost assumption, trailing semantics, volatility/events/warnings (never the median candle range as spread); contrarian gets directional news, setup, timestamped latest-closed price, indicators, reasons, and warnings. Spend is bounded by exact cache, full-evidence-identity TTL reuse, and `max_panels_per_day`; changed evidence never inherits a direction-only cached verdict (test-locked — which also means TTL reuse effectively never fires live because price/candles change every bar; the daily cap is the real spend bound). Verdict provenance logs creation time, `MISS`/`EXACT_HIT`/`TTL_REUSE`, and a SHA-256 payload hash. Prompts forbid outside knowledge and outputs remain short. **OPPOSE guard (2026-09):** the 3-week signal-log scorecard showed the contrarian/risk/technical personas oppose almost every trend entry (chair said OPPOSE 73% of the time; opposed trades made +30R, PF 2.6) while the macro persona was the only informative voice (macro=support → 88% win). `_apply_oppose_guard` therefore downgrades a chair OPPOSE to NEUTRAL unless macro opposes, or technical opposes while macro is not supportive; the synthesizer prompt states the same rule. Re-score with `analyze_signal_log.py --stored-candles` (persona scorecard) before changing it again. **Verdict source (2026-09-03, owner decision):** all four personas still run and are logged (`panel_takes_json`), but `agent_panel.verdict_source` defaults to `"macro"` — the alert verdict is the macro persona's stance (support/neutral/oppose → AGREE/NEUTRAL/OPPOSE) and the synthesizer call is skipped (4 LLM calls per panel instead of 5); `"chair"` restores the four-voice synthesizer + OPPOSE guard. Re-score personas monthly with `analyze_signal_log.py --stored-candles`; weights are not to be tuned before ~300 graded outcomes with a held-out half. - **`market_outlook.py`** — hourly DeepSeek market brief for XAU/USD, using ATR-normalized M15/H1/H4 candles, visible indicator states, levels, news, and recent signal history (read `mode=ro`). It returns a written **ADVISORY-ONLY** bias/setup-forming assessment and persists it independently in `outlooks` within `logs/signals.db` together with the news, indicator/regime/level, and candle evidence sent to the model. It may be sent post-scan by Telegram or requested with `main.py --outlook`; it never imports or runs inside the signal path. Hourly dedup is by UTC hour BUCKET (the old `now − last < 3600` rule dropped every other hour because the H1 loop fires at boundary+10 s+scan time) plus its own UTC-day cap (default 12); every silent early return logs `market outlook: skipped (...)` and an empty LLM reply logs a warning — a dark feature must be visible in `logs/telegram.log`. Provider/news failures leave a safe partial brief while LLM/database failures degrade to no outlook. - **`violent_watch.py`** — **display-only** violent-move watch, dark by default (`violent_watch` config block: `enabled`, `alert_line`, `headsup`, `max_headsup_per_day`, `headsup_min_level`; env `VIOLENT_WATCH_ENABLED`). Flags when a "violent bar" (range ≥ 3× prior ATR14) is unusually likely and stamps `SignalReport.watch_line` **post-hoc** via `attach_watch` (telegram alert path, `main.py`, `dashboard_publisher`) — same contract as the agent panel: never imported by `signal_aggregator`/`engine`/`backtest/`, never raises, never read by a gate (test-locked, Module 30). Triggers: a high-impact **USD** calendar release within ±30 min (hot), hot clock slots (weekday 12–14 UTC, weekday 01 UTC, Sunday reopen 22–23 UTC), and **price-extreme** indicator readings (outside the Bollinger bands, RSI <30 / ≥70, outside the Keltner channel). Level = hot when the calendar fires or a clock slot plus ≥1 price extreme; watch on a clock slot alone or ≥2 price extremes. Context readings (ADX slope >3, volatility "extreme", regime "transitioning", MACD STRONG_BUY/STRONG_SELL) are printed but **never raise the level** — the first draft that let them count flagged 27% of bars "hot" at only ~2× odds; the shipped rule flags 8.6% at 3–5× (evidence: `backtest/research/sizing_campaign_2026-09.md` addenda). Direction is called DOWN on bearish readings (70–78% on M5 and M15) and UP only on M5. Telegram also sends a standalone heads-up on the fastest scanned TF (`_HEADSUP_TF`), rate-limited per UTC hour+trigger-set and per day; it explicitly tells the owner **not** to touch open trades — every protection tested lost money, the stop is the protection. -- **`telegram_bot.py`** — owner-locked bot + subscriber channel; candle-boundary auto-scan loops (M5/M15/H1/H4, boundary+10 s), alert quality threshold + fingerprint dedup (committed only after a successful send; the fingerprint is signal+direction — grade is NOT part of it, a grade UPGRADE re-alerts, an A↔B flicker does not), 📰 LLM market-context line on alerts when available, WebSocket level-breach monitor fed by the streaming provider (`data_providers.streaming`, falling back to the active provider; auto-disabled when the resolved provider can't stream or has no API key — Oanda does not stream, Tiingo does). The breach handler reuses this bar's M15 report (`_cached_or_scan`, keyed by closed-bar epoch) instead of rescanning — it used to log 3.5 M15 rows per candle. Rotating log: `logs/telegram.log`, 3-h rotation × 240 backups. +- **`telegram_bot.py`** — owner-locked bot + subscriber channel; candle-boundary auto-scan loops (M5/M15/H1/H4, boundary+10 s), alert quality threshold + fingerprint dedup (committed only after a successful send; the fingerprint is signal+direction — grade is NOT part of it, a grade UPGRADE re-alerts, an A↔B flicker does not), 📰 LLM market-context line on alerts when available, WebSocket level-breach monitor fed by the streaming provider (`data_providers.streaming`, falling back to the active provider; auto-disabled when the resolved provider can't stream or has no API key — Oanda does not stream, Tiingo does). The breach handler reuses this bar's M15 report (`_cached_or_scan`, keyed by closed-bar epoch) instead of rescanning — it used to log 3.5 M15 rows per candle. **Alerts require a trade setup (2026-09-08):** `_is_alert_candidate` (shared by the scan and breach paths) needs non-neutral + grade ≥ threshold + `trade_setup`; a signal every generator declined used to go out anyway (a B-grade no-setup SELL under a "costs would eat the edge" warning). **Sends are logged:** every delivered alert / breach alert / violent-move heads-up writes an `… sent:` INFO line carrying the signal-log row id, and a no-setup candidate writes `alert skipped: … (gate reason)` — before this only failures were logged and a review had to reconstruct sends from panel rows. Rotating log: `logs/telegram.log`, one file per server-local day × 30 backups (was 3-h × 240 until 2026-09-08; the old `_HH`-suffixed files on the live box are not matched by the daily cleanup and need one manual delete). - **`dashboard_publisher.py` / `static/dashboard.html`** — web dashboard as a **static publisher** (live LAMP server): standalone daemon scans at candle closes via `produce_signal` and atomically writes `snapshot.json`/`news.json`/`dashboard.html` to `$DASHBOARD_OUT_DIR` (Apache serves them; page is public by owner decision). **Hard independence rule: never imports, and is never imported by, `telegram_bot`** — confirmation map and boundary math are deliberate local copies, so a web-side failure can only mean a stale page. Due-TF classification uses the RAW wake time (flooring wake−buffer skips every H1/H4 scan — 2026-08-02 bug). News panel reads `NewsEngine.get_fundamental_context()` plus the additive `get_week_events()` accessor (display-only; no signal path calls it); as a separate process it has its own news caches and DeepSeek daily cap. News/calendar rows are clickable (detail dialog); headlines carry `id` = sha1(headline) in news.json, and exactly one on-demand endpoint exists: `static/news_summary.php` (installed by `install_static`), which serves a DeepSeek gold-impact summary per headline — whitelist-gated against news.json ids, cache-forever in `ai/`, own fail-closed daily cap (150/day, file-based, separate from the publisher's), key at `/etc/gold-dashboard/deepseek.key`, never reads repo config. Deployment checklist: `deploy/README-dashboard.md`. - **`backtest/gpu_backtest.py`** — bar-by-bar replay through the real `analyze_from_dataframes` (news off). Realism model (do not regress): next-bar-open entries re-anchored to the raw price path, round-trip `--spread` charged per trade, trade resolution on RAW bars (cleaning applies only to signal-side data), same-bar SL+TP tie → SL first, gap fills at bar open, trail ratchet at bar close only. Indicators are precomputed once per TF (CuPy GPU or NumPy fallback — check the `Backend:` log line); replay fans out over worker processes and the 5-bar cooldown is applied in an ordered pass afterwards, so parallel and sequential books are identical. Parquet cache in `backtest/cache/` keyed by provider/symbol/TF/window. `backtest/*.csv` are gitignored pinned trade books from past campaigns — used for manual before/after comparison. `backtest/instructions_for_backtest.txt` lists what every backtest report must record. @@ -104,4 +104,5 @@ Signal currency everywhere is `SignalStrength` (−2..+2). DataFrames are UTC tz - 2026-09-06 stake-sizing campaign (`backtest/research/sizing_campaign_2026-09.md`): a walk-forward "second opinion" stake multiplier (meta-labelling on 65 decision-time features, 3-year Oanda books at $0.65) FAILED the pre-registered rule on M5 (clean null) and M15 (lift inside the label-shuffle null, explained by BUY-vs-SELL). Nothing that changes numbers shipped. Behaviour-neutral tooling remains: `position_sizing.build_features` + `--export-features` (default export stays 63 columns), `SignalReport.penalty_tags/penalty_total/momentum_fading`, `TradeSetup.stake_multiplier` (None; reserved) (the campaign's research scripts were removed 2026-09-06; git history has them). Stage-B facts: M5 stretch measures are POSITIVE in both halves (do not re-propose exhaustion sizing on M5); M15 exhaustion tag is weakly negative (≈ +3R/yr, noise); bursts cluster at 12–15 UTC and 01 UTC bar-close and do NOT hurt entries. - Auto-execution (oanda_daemon.py / oanda_executor.py) was removed 2026-09-03 — the owner is not auto-trading; last commit containing it is 31002cb. Book-parity rules that lived in its docstring (no broker TP, 5-bar cooldown, once-per-bar trail) remain the backtester's model. - 2026-09-07 signal-log inspection (fresh 2-day `logs/signals.db` import: NFP Friday + Monday; 5 setups, all BUY, 4 stopped out at local tops — two days is not evidence). Three sample-size-independent fixes shipped: (1) the signal-side 10σ outlier filter now drops only bars whose NEXT close snaps back ≥80% (it had been deleting ~36 real M5 news bars/yr in the backtester and a different ~24/yr live, e.g. the 2026-09-04 12:30 UTC NFP bar; 3-yr drops are now M5 2 / M15 1 / H1 0). Book impact at $0.65, 12mo: M5 557/+112.21R/PF 1.56 → 560/+108.81R/PF 1.54 (halves +55.6/+53.2), M15 397/+73.93/1.53 → 398/+74.61/1.53 (max DD 10.8→9.1R), H1 byte-identical; 3-yr (2023-09-05→2026-09-05): M5 857/+161.3/1.53 → 870/+153.6/1.50, M15 1086/+148.7/1.37 → 1092/+151.9/1.38, H1 255/+70.5/1.98 → 253/+70.0/1.98, H4 113/+22.7/1.63 → 114/+21.7/1.59 — only trades on the flagged event days move (M5 gives back a few Jan–Mar 2026 crash-period winners); the pinned M5/M15 CSVs are therefore no longer byte-reproducible (regenerate with the parity recipe when re-pinning). (2) Oanda candle fetches retry along a 2/5/10 s ladder under a 45 s budget (11 boundary scans in two days had lost their primary frame). (3) A scan that still loses its primary frame logs `scan_kind='failed'` and readers skip it. The live dashboard publisher was found dead (0 rows Friday, stopped Mon 02:20 UTC): install `deploy/gold-dashboard.service` on the live server. +- 2026-09-08 delivery-side fixes from the 3-day signal-log + telegram-log review (nothing in the signal core changed; 468 tests): (1) `_drop_stale_spread_gate_vetoes` (after every setup archetype, so the XAU_PB path is covered too) moves a core generator's stale `SPREAD GATE:` veto to `cleaned_warnings` when a later generator ships a setup — the 3-yr H1 and M15 books were regenerated before/after and are identical on every trade column (H1 253/+70.02R/PF 1.98, M15 1092/+151.90R/PF 1.38); only the exported `warnings` text differs, on 50/253 H1 and 358/1092 M15 trades, i.e. a fifth to a third of all A-grade alerts had been carrying "costs would eat the edge" beside a valid target; (2) Telegram alerts now require a trade setup (`_is_alert_candidate`, scan + breach paths; no-setup candidates log `alert skipped: … (gate reason)`); (3) every delivered alert / breach alert / violent-move heads-up logs an `… sent:` INFO line with the signal-log row id (the review had to reconstruct sends from panel rows); (4) `telegram.log` rotates daily × 30 instead of 3-hourly × 240 (old `_HH` files on the live box need one manual delete). Review facts (not evidence — three days): 8 setups, 1 right; strong readings (|score| ≥ 20) were followed by reversals in 16 of 18 episodes; hot watch level 0/25 bursts on M5 (calendar trigger never fired, 15 of 33 hot stamps were the 12–14 UTC slot on Labor Day — no holiday awareness anywhere); Oanda published no XAU bars 18:30–22:00 UTC on Labor Day and the bot scanned the stale bar 43×; the retry ladder recovered all 25 post-deploy 401 bursts (41 of 58 incidents sit on the :00 boundary); the live dashboard publisher ran only 01:00–02:20 UTC Sep 7 and is dead again. - 2026-09-06 stale-code cleanup (behaviour-neutral; suite green, M5/H1 12-month books byte-identical after every signal-side commit): `main.py --timeframe` now accepts M5/M15/H1/H4 and takes its confirmation TFs from the pair factory (the CLI's private table had paired M5 with M1+M15); `PlatformConfig.pip_value` is gone (`PAIR_INFO` is the source); removed as dead or superseded: `_get_affected_pairs`/`event_impact_on_pairs`, `SignalLogger.advance_with_bars`/`path`/`news_for`, `reset_llm_client`, the Telegram scan-summary/keyboard helpers and duplicate confirmation map, the 2026-07 Oanda-vs-TD research pager+study, `burst_profile.py`, `hour_analysis.py`, `scripts/compare_oanda_tiingo.py`, the never-implemented LLM-provider plan and the merged config.json spec. Second pass the same day (owner directive: keep only what the system needs): every research script and data file under `backtest/research/` (book_report, book_diff, the burst tools, sizing_model, commit_ladder, llm_news_eval + golden CSV, ff_calendar_scraper, calendar_impact, watch_rule_eval, the Dukascopy spread study + CSVs, the panel dry-test JSONL), the two completed study scripts in `scripts/`, the architecture diagram + generator and the data-source test plan were deleted; the campaign `.md` reports stay as the decision ledger. Git history before `953c5ad` has all of it. Third pass (owner: "remove dead functions as well"): the write-only pb/cp rejection counters, the unreachable `_TP_QUALITY_SCALE` B/C entries, the engine's unused DataFrame argument and M1 arm, the providers' generic-symbol guesses (unknown symbols now raise), the sub-$10 price format, the `=X` pair parsing and the unread `minutes_until`/`is_high_impact` news.json keys are gone. diff --git a/deploy/README-dashboard.md b/deploy/README-dashboard.md index 64f7a32..6fc37a9 100644 --- a/deploy/README-dashboard.md +++ b/deploy/README-dashboard.md @@ -72,7 +72,8 @@ tail -f logs/dashboard.log The daemon scans all four timeframes at boot, then wakes at every M5 boundary (+10 s) and rescans whichever timeframes just closed. Logs rotate -every 3 h (240 backups), same scheme as the bot. +every 3 h (240 backups); the bot's own `telegram.log` rotates daily (30 +backups) since 2026-09-08. ## 5. Optional Apache polish diff --git a/signal_aggregator.py b/signal_aggregator.py index e98db43..c80df90 100755 --- a/signal_aggregator.py +++ b/signal_aggregator.py @@ -18,6 +18,11 @@ import pandas as pd _NY_TZ = ZoneInfo("America/New_York") +# Prefix of the cost-floor veto both core setup generators emit; the XAU_CP +# cleanup and the post-archetype stale-veto pass match on it, so it is +# spelled once. +_SPREAD_GATE_PREFIX = "SPREAD GATE:" + from chart_patterns import detect_chart_pattern from config import PlatformConfig, SignalStrength, Timeframe from providers import BaseDataProvider, make_data_provider @@ -129,8 +134,11 @@ class SignalReport: bullish_reasons: List[str] = field(default_factory=list) bearish_reasons: List[str] = field(default_factory=list) warnings: List[str] = field(default_factory=list) - # Warnings the XAU_CP acceptance removed from `warnings` (metadata - # only — logged so the gates a CP trade bypassed stay auditable). + # Warnings a shipped setup superseded, removed from `warnings` but + # logged so they stay auditable (metadata only): the vetoes the + # XAU_CP acceptance stripped and, since 2026-09-08, a core + # generator's stale spread-gate veto when a later generator produced + # the setup (`_drop_stale_spread_gate_vetoes`). cleaned_warnings: List[str] = field(default_factory=list) current_price: float = 0.0 @@ -889,7 +897,7 @@ class SignalAggregator: final when this runs. """ stale_prefixes = ( - "SPREAD GATE:", + _SPREAD_GATE_PREFIX, "EXHAUSTION BLOCK", "EXHAUSTED TREND FILTER:", "MTF OPPOSITION:", @@ -1225,16 +1233,7 @@ class SignalAggregator: self._apply_m5_filters(report) # ── 7. Trade setup for actionable signals ────────────────── - # ATR generator is XAU-only: it hardcodes gold pip value and - # 2-decimal price rounding. - if (report.trade_setup is None and report.signal_quality in ("A+", "A") - and report.display_name == "XAU/USD"): - atr = (report.primary_analysis.key_levels.get("atr", 0) - if report.primary_analysis else 0) - if atr > 0: - report.trade_setup = self._generate_xau_atr_trade_setup(report, atr) - if report.trade_setup is None and report.signal_quality in ("A+", "A", "B", "C"): - report.trade_setup = self._generate_trade_setup(report, pip_value) + self._attach_core_setup(report, pip_value) # ── 8. Reasons ───────────────────────────────────────────── self._collect_reasons(report) @@ -1271,6 +1270,11 @@ class SignalAggregator: for w in report.warnings)): self._try_chart_pattern(report, primary_df, pip_value) + # ── 11. Stale spread-gate vetoes ───────────────────────────── + # Whichever archetype shipped the setup, a core generator's + # cost-floor veto no longer describes it. + self._drop_stale_spread_gate_vetoes(report) + return report # ── Internals ────────────────────────────────────────────────── @@ -1278,6 +1282,49 @@ class SignalAggregator: # Only A+/A reach the XAU ATR generator (gate at the call site). _TP_QUALITY_SCALE = {"A+": 2.0, "A": 1.5} + def _attach_core_setup(self, report: SignalReport, pip_value: float) -> None: + """Step 7: the XAU ATR generator (A+/A) first, the swing generator + (GENERIC) for anything it declined. A veto either one leaves + behind is reconciled after every archetype has run + (`_drop_stale_spread_gate_vetoes`).""" + # ATR generator is XAU-only: it hardcodes gold pip value and + # 2-decimal price rounding. + if (report.trade_setup is None and report.signal_quality in ("A+", "A") + and report.display_name == "XAU/USD"): + atr = (report.primary_analysis.key_levels.get("atr", 0) + if report.primary_analysis else 0) + if atr > 0: + report.trade_setup = self._generate_xau_atr_trade_setup(report, atr) + if report.trade_setup is None and report.signal_quality in ("A+", "A", "B", "C"): + report.trade_setup = self._generate_trade_setup(report, pip_value) + + @staticmethod + def _drop_stale_spread_gate_vetoes(report: SignalReport) -> None: + """Step 11: runs once after every setup archetype (core ATR/swing, + XAU_PB, XAU_CP) has had its turn. + + Both core generators veto a target under the cost floor with a + `_SPREAD_GATE_PREFIX` warning. When a later generator then ships + a setup with a bigger target, that veto is stale — the 2026-09-08 + 07:45 UTC A-grade alert said "costs would eat the edge" beside a + $26.49 TP1. A stale veto moves to `cleaned_warnings` (metadata + only, the XAU_CP cleanup's contract; that path has already + stripped its own copy by the time this runs, and the PB path is + covered here). Nothing downstream reads warnings while a setup + exists, so trade books are unchanged — verified 2026-09-08 on the + 3-yr H1/M15 books; only the exported `warnings` text column + differs. + """ + if report.trade_setup is None: + return + kept: List[str] = [] + stale: List[str] = [] + for warning in report.warnings: + (stale if warning.startswith(_SPREAD_GATE_PREFIX) else kept).append(warning) + if stale: + report.warnings[:] = kept + report.cleaned_warnings.extend(stale) + def _generate_xau_atr_trade_setup( self, report: SignalReport, atr: float ) -> Optional[TradeSetup]: @@ -1325,7 +1372,7 @@ class SignalAggregator: min_tp = self.config.signals.round_trip_cost * self.config.signals.min_tp_cost_multiple if tp_distance < min_tp: report.warnings.append( - f"SPREAD GATE: target ${tp_distance:.2f} < ${min_tp:.2f} minimum — " + f"{_SPREAD_GATE_PREFIX} target ${tp_distance:.2f} < ${min_tp:.2f} minimum — " f"costs would eat the edge" ) return None @@ -1744,7 +1791,7 @@ class SignalAggregator: min_tp = self.config.signals.round_trip_cost * self.config.signals.min_tp_cost_multiple if tp1_distance < min_tp: report.warnings.append( - f"SPREAD GATE: target ${tp1_distance:.2f} < ${min_tp:.2f} minimum — " + f"{_SPREAD_GATE_PREFIX} target ${tp1_distance:.2f} < ${min_tp:.2f} minimum — " f"costs would eat the edge" ) return None diff --git a/signal_log.py b/signal_log.py index 183de23..0261b8b 100644 --- a/signal_log.py +++ b/signal_log.py @@ -49,8 +49,11 @@ or news_engine) so this module stays at the bottom of the import graph. * Post-gate truth columns: ``gate_reason`` (the warning that suppressed the setup), ``actionable`` (grade >= C AND setup present — what an alert surface would act on; the ``action`` text is written BEFORE the - gate chain and can disagree), ``cleaned_warnings_json`` (warnings the - XAU_CP acceptance removed from ``warnings_json``), ``bar_range_median`` + gate chain and can disagree), ``cleaned_warnings_json`` (warnings a + shipped setup superseded, removed from ``warnings_json``: the vetoes + the XAU_CP acceptance stripped and, since 2026-09-08, a core + generator's stale spread-gate veto when a later generator produced + the setup — so non-empty does NOT mean "XAU_CP row"), ``bar_range_median`` (the value historically stored as ``spread`` — it is the median High-Low of the primary frame, NOT a broker spread), ``scan_kind`` (``scan``; ``blocked`` for weekend/day early returns, which carry no diff --git a/telegram_bot.py b/telegram_bot.py index 68f4ea2..56c9c74 100755 --- a/telegram_bot.py +++ b/telegram_bot.py @@ -77,6 +77,7 @@ from market_outlook import maybe_run_outlook from news_engine import FundamentalContext, NewsEngine from pair_signals import PAIR_PRODUCERS from signal_aggregator import SignalAggregator, SignalReport +from signal_log import gate_reason from violent_watch import attach_watch, should_send_headsup if TYPE_CHECKING: # type hints only — ViolentWatch is never needed at runtime here @@ -96,13 +97,17 @@ _log_formatter = logging.Formatter( ) _console_handler = logging.StreamHandler() _console_handler.setFormatter(_log_formatter) -# 3-hour rotation × 240 backups = 30 days retention. Handlers sit on the -# ROOT logger (force=True guarantees ownership even if something logged -# first), so every module logger — data_provider, news_engine, -# signal_aggregator, websockets, telegram.ext — lands in the same file. +# One file per (server-local) day × 30 backups = 30 days retention. +# 2026-09-08: was 3-hourly × 240 — a 4-day review meant 18 files. The old +# `telegram.log.YYYY-MM-DD_HH` files do not match the daily handler's +# cleanup pattern and need one manual delete on the live box. +# Handlers sit on the ROOT logger (force=True guarantees ownership even +# if something logged first), so every module logger — data_provider, +# news_engine, signal_aggregator, websockets, telegram.ext — lands in the +# same file. _file_handler = logging.handlers.TimedRotatingFileHandler( os.path.join(_LOG_DIR, "telegram.log"), - when="h", interval=3, backupCount=240, encoding="utf-8", + when="midnight", backupCount=30, encoding="utf-8", ) _file_handler.setFormatter(_log_formatter) logging.basicConfig(level=getattr(logging, _APP.log_level, logging.INFO), @@ -575,6 +580,44 @@ def _passes_quality(quality: str) -> bool: return _QUALITY_RANK.get(quality, 0) >= _QUALITY_RANK.get(_alert_min_quality, 1) +def _is_alert_candidate(report: SignalReport, tf_label: str) -> bool: + """An alert is a trade call: non-neutral, grade at/above the + threshold, AND a trade setup. + + Until 2026-09-08 the setup was not required, so a signal every + generator had declined (spread gate, RR floor, swing geometry) still + went out — 07:50 UTC that day sent a B-grade SELL with no setup under + a "costs would eat the edge" warning. Skips are logged at INFO so the + decision stays visible in telegram.log.""" + if report.final_signal == SignalStrength.NEUTRAL: + return False + if not _passes_quality(report.signal_quality): + return False + if report.trade_setup is None: + why = gate_reason(report.warnings, report.cleaned_warnings, + report.signal_quality, False) + logger.info( + f"alert skipped: {report.display_name} {tf_label} " + f"{report.final_signal.name} grade {report.signal_quality} " + f"has no trade setup ({why})") + return False + return True + + +def _alert_sent_line(report: SignalReport, tf_label: str, delivered_to: int, + kind: str = "alert") -> str: + """One INFO line per alert that reached a chat. Only failures were + logged before 2026-09-08, so that day's log review had to reconstruct + what was sent from panel rows and dedup skips.""" + s = report.trade_setup + setup = (f"{s.direction} entry {s.entry_price:.2f} sl {s.stop_loss:.2f} " + f"tp1 {s.take_profit_1:.2f} [{s.strategy}]" if s else "no setup") + rowid = getattr(report, "_signal_log_rowid", None) + return (f"{kind} sent: {report.display_name} {tf_label} " + f"{report.final_signal.name} grade {report.signal_quality} — " + f"{setup} — signal-log row #{rowid} → {delivered_to} chat(s)") + + _last_alert_quality: Dict[str, str] = {} # key → grade of last ALERTED signal @@ -677,6 +720,17 @@ async def _send_alert(bot, chat_id: int, text: str) -> bool: return False +async def _broadcast(bot, text: str) -> int: + """Send `text` to every signal target; returns how many chats got it. + Shared by the scan and breach paths so the delivery rule cannot + drift between them.""" + delivered = 0 + for chat_id in _signal_targets(): + if await _send_alert(bot, chat_id, text): + delivered += 1 + return delivered + + # ── Candle-boundary scheduler ──────────────────────────────────────── _autoscan_chats: Set[int] = set() @@ -786,8 +840,16 @@ async def _scan_timeframe( _watch_state, now, w.kinds, int(vw_settings.get("max_headsup_per_day", 12))) if ok: - await _send_alert(bot, OWNER_ID, - _format_headsup(report, w)) + if await _send_alert(bot, OWNER_ID, + _format_headsup(report, w)): + logger.info( + f"violent watch: heads-up sent ({w.level}: " + f"{', '.join(w.kinds)}) signal-log row " + f"#{getattr(report, '_signal_log_rowid', None)}") + else: + logger.warning( + "violent watch: heads-up send failed " + "(this hour's slot is consumed)") else: logger.info( f"violent watch: heads-up skipped ({why})") @@ -798,11 +860,9 @@ async def _scan_timeframe( except Exception: logger.exception("violent watch failed") - candidates = [ - r for r in reports - if r.final_signal != SignalStrength.NEUTRAL # same gate as breach path - and _passes_quality(r.signal_quality) - ] + # Same gate as the breach path: non-neutral, grade ≥ threshold, and a + # trade setup (_is_alert_candidate logs the no-setup skips). + candidates = [r for r in reports if _is_alert_candidate(r, tf.value)] for report in candidates: key = f"{report.symbol}_{tf.value}" # The lock spans check → send → commit so a concurrent breach @@ -818,14 +878,12 @@ async def _scan_timeframe( await asyncio.get_running_loop().run_in_executor( _executor, attach_panel, report) detail = format_signal_message(report) - delivered = False - for chat_id in _signal_targets(): - if await _send_alert(bot, chat_id, detail): - delivered = True + delivered = await _broadcast(bot, detail) # Commit only when the alert reached at least one target; # a fully failed send stays eligible for the next scan. if delivered: _commit_alert(key, report) + logger.info(_alert_sent_line(report, tf.value, delivered)) # This independent advisory call runs only after the H1 scan completes. # Offloading the LLM request keeps the event loop responsive. @@ -962,8 +1020,7 @@ async def _handle_breach(bot, pair_symbol: str, price: float, level_name: str) - # Alert only when the rescan produced an actionable, changed signal — # same gates as the auto-scanner, so a breach can't spam the channel. - if (report.final_signal == SignalStrength.NEUTRAL - or not _passes_quality(report.signal_quality)): + if not _is_alert_candidate(report, Timeframe.M15.value): return key = f"{pair_symbol}_{Timeframe.M15.value}" @@ -983,12 +1040,11 @@ async def _handle_breach(bot, pair_symbol: str, price: float, level_name: str) - f"*{_escape_md(label)}* @ `{_fmt_price(price)}`\n\n" ) detail = header + format_signal_message(report) - sent = False - for chat_id in _signal_targets(): - if await _send_alert(bot, chat_id, detail): - sent = True + sent = await _broadcast(bot, detail) if sent: _commit_alert(key, report) + logger.info(_alert_sent_line(report, Timeframe.M15.value, sent, + kind="breach alert")) def _breach_task_done(task: asyncio.Task) -> None: diff --git a/test_all.py b/test_all.py index 9a9835a..99b3f5d 100644 --- a/test_all.py +++ b/test_all.py @@ -2579,6 +2579,55 @@ def _(): d.pop(key, None) +@test("Alert candidates: a quality-passing signal without a trade setup is skipped and logged") +def _(): + import logging + import telegram_bot as tb + from signal_aggregator import SignalReport, TradeSetup + + def rpt(setup: bool, signal=SignalStrength.SELL, q="B"): + r = SignalReport(symbol="XAU/USD", display_name="XAU/USD", timestamp="t") + r.final_signal = signal + r.signal_quality = q + r.warnings = ["SPREAD GATE: target $8.31 < $9.00 minimum — " + "costs would eat the edge"] + if setup: + r.trade_setup = TradeSetup( + pair="XAU/USD", display_name="XAU/USD", direction="SELL", + entry_price=4388.0, stop_loss=4410.0, take_profit_1=4377.0, + take_profit_2=4372.0, risk_reward=0.5, risk_pips=220.0, + reward_pips=110.0, pip_value=0.10) + return r + + records = [] + handler = logging.Handler() + handler.emit = records.append + tb.logger.addHandler(handler) + saved_threshold = tb._alert_min_quality + tb._alert_min_quality = "C" + try: + assert tb._is_alert_candidate(rpt(True), "5min") + assert not tb._is_alert_candidate(rpt(False), "5min"), \ + "no trade setup → no alert (the 2026-09-08 07:50 UTC no-setup SELL)" + assert not tb._is_alert_candidate(rpt(True, SignalStrength.NEUTRAL), "5min") + assert not tb._is_alert_candidate(rpt(True, q="F"), "5min") + finally: + tb._alert_min_quality = saved_threshold + tb.logger.removeHandler(handler) + skips = [r.getMessage() for r in records + if r.getMessage().startswith("alert skipped:")] + assert len(skips) == 1, skips + assert "5min SELL grade B" in skips[0] and "SPREAD GATE" in skips[0], skips[0] + + line = tb._alert_sent_line(rpt(True), "5min", 2) + assert line.startswith("alert sent: XAU/USD 5min SELL grade B"), line + assert "SELL entry 4388.00 sl 4410.00 tp1 4377.00" in line and "2 chat(s)" in line, line + r = rpt(True) + r._signal_log_rowid = 917 + breach = tb._alert_sent_line(r, "15min", 1, kind="breach alert") + assert breach.startswith("breach alert sent:") and "row #917" in breach, breach + + @test("Backtester --confirm-tfs defaults to None (factory's live map)") def _(): from gpu_backtest import _parse_args @@ -2861,6 +2910,65 @@ def _(): ], rep.warnings +@test("Core setup: a stale ATR spread-gate veto is cleaned when the swing generator ships a setup") +def _(): + from types import SimpleNamespace + from signal_aggregator import SignalAggregator, SignalReport, TradeSetup + + agg = SignalAggregator(PlatformConfig()) + price = 4400.0 + + def rpt(levels): + r = SignalReport(symbol="XAU/USD", display_name="XAU/USD", timestamp="t") + r.final_signal = SignalStrength.STRONG_BUY + r.final_score = 40.0 + r.signal_quality = "A" + r.current_price = price + r.primary_analysis = SimpleNamespace(key_levels=dict(levels)) + return r + + # ATR $2 → the A-grade ATR target is 1.5 × $2 = $3, under the $9 cost + # floor (veto + warning). The swing generator's stop sits under the + # $20-deep swing low, so its RR-floor target is $11 and a GENERIC + # setup ships — the ATR veto is then stale (alert #915, 2026-09-08). + r = rpt({"atr": 2.0, "swing_low_1": price - 20.0}) + agg._attach_core_setup(r, 0.10) + assert r.trade_setup is not None and r.trade_setup.strategy == "GENERIC", r.trade_setup + assert sum(w.startswith("SPREAD GATE:") for w in r.warnings) == 1, \ + "the generators themselves leave the veto in place" + agg._drop_stale_spread_gate_vetoes(r) + assert not any(w.startswith("SPREAD GATE:") for w in r.warnings), r.warnings + assert sum(w.startswith("SPREAD GATE:") for w in r.cleaned_warnings) == 1, \ + "the veto must stay auditable in cleaned_warnings" + + # No swing structure: the swing generator's fallback target is + # max(1.5 × ATR, RR floor × the $8 minimum stop) = $4 — also gated — + # so no setup ships and BOTH vetoes stay visible. + r2 = rpt({"atr": 2.0}) + agg._attach_core_setup(r2, 0.10) + agg._drop_stale_spread_gate_vetoes(r2) + assert r2.trade_setup is None + assert sum(w.startswith("SPREAD GATE:") for w in r2.warnings) == 2, r2.warnings + assert not r2.cleaned_warnings + + # The pass runs after the PB/CP archetypes too: a setup shipped by a + # later generator (XAU_PB shape here) supersedes BOTH core vetoes, + # and unrelated warnings keep their order. + r3 = rpt({"atr": 2.0}) + agg._attach_core_setup(r3, 0.10) + r3.warnings.insert(0, "NEAR EMA200 (0.2%) — chop risk") + r3.warnings.append("TREND EXHAUSTION — price extended >3.5 ATR past swing low") + r3.trade_setup = TradeSetup( + pair="XAU/USD", display_name="XAU/USD", direction="BUY", + entry_price=price, stop_loss=price - 8.0, take_profit_1=price + 12.0, + take_profit_2=price + 18.0, risk_reward=1.5, risk_pips=80.0, + reward_pips=120.0, pip_value=0.10, strategy="XAU_PB") + agg._drop_stale_spread_gate_vetoes(r3) + assert r3.warnings == ["NEAR EMA200 (0.2%) — chop risk", + "TREND EXHAUSTION — price extended >3.5 ATR past swing low"], r3.warnings + assert sum(w.startswith("SPREAD GATE:") for w in r3.cleaned_warnings) == 2 + + @test("CP gate: MTF opposition blocks the pattern trade") def _(): cfg = _cp_config() @@ -3392,15 +3500,16 @@ def _(): load_dotenv("/nonexistent/.env") # missing file is a silent no-op -@test("telegram_bot: log rotation is 3-hourly with 30-day retention") +@test("telegram_bot: log rotation is daily with 30-day retention") def _(): import logging.handlers import telegram_bot as tb h = tb._file_handler assert isinstance(h, logging.handlers.TimedRotatingFileHandler) - assert h.when.upper() == "H" and h.interval == 3 * 3600, \ - "rotation must fire every 3 hours" - assert h.backupCount == 240, "240 × 3h = 30-day retention" + assert h.when.upper() == "MIDNIGHT", \ + "one file per day (2026-09-08: was 3-hourly × 240)" + assert h.interval == 86400, "plain daily — an interval=N would stretch retention" + assert h.backupCount == 30, "30 daily files = 30-day retention" @test("telegram_bot: candle loop survives a failing scan cycle")