diff --git a/CLAUDE.md b/CLAUDE.md index 3acd699..547be50 100644 --- a/CLAUDE.md +++ b/CLAUDE.md @@ -23,7 +23,7 @@ cp config.json.example config.json # then fill i venv/bin/pip install cupy-cuda12x "nvidia-cuda-runtime-cu12==12.9.*" # optional, dev box: backtester GPU backend # (CuPy from pip needs that header wheel to compile kernels) venv/bin/python test_all.py # QA suite, no network; prints its own count - # (581 tests / 38 modules as of 2026-09-27) + # (632 tests / 40 modules as of 2026-10-01) venv/bin/python main.py --pair XAUUSD --timeframe H1 --detail # one signal report in the terminal # --json full dump, --panel adds the advisory LLM panel, # --timeframe M5 | M15 | H1 | H4 @@ -48,7 +48,10 @@ unknown path is a hard exit), `--export-features` (appends the 65 decision-time it on a parity run because it changes the file), `--fixed-rr-tp N` (research only, not shipped), `--set signals.grading_mode=min|product` (research only, 2026-09-27: grade on one conviction number instead of the score + confidence floors; both FAILED rule 1 on M5 and M15, default `pair` is the -shipped rule and byte-identical), `--record DIR [--record-bars]` (research recorder, see section 9), `--backend auto|gpu|cpu` (`auto` +shipped rule and byte-identical), `--set xau_strategy.daily_trend_veto=off|both|buy|sell` (2026-09-30: +the M15 factory ships `both`; pass `off` to reproduce a pre-veto M15 book; `buy` / `sell` and every +mode on M5 / H1 FAILED their rule), `--set xau_strategy.wide_stop_cap_atr=K` (research only: FAILED +on 2026-09-30, ships 0 = off), `--record DIR [--record-bars]` (research recorder, see section 9), `--backend auto|gpu|cpu` (`auto` uses the CuPy GPU backend when CUDA works and falls back to NumPy with the reason in the `Backend:` line; `gpu` refuses to fall back; `cpu` forces the NumPy twins for a parity run; the env var `FB2_BACKEND` sets the default). Both backends produce byte-identical books (section 9). @@ -75,6 +78,8 @@ same schedule. `main.py` runs once when you call it. (`app_config.py`), then applies the validated per-timeframe overrides (section 8). Never build a bare `PlatformConfig()` for a real scan; always go through the factory. The confirmation timeframes come from the factory map: M5 → M15 + H1, M15 → M5 + H1, H1 → M15 + H4, H4 → H1 + D1. +Since 2026-09-30 an M15 scan also needs the daily (D1) frame, not as a confirmation but as the +input of the daily-trend veto (step 8); it is a required frame like the others. **3. Fetch candles.** `signal_aggregator.analyze_pair` (the live entry point) first checks the calendar: on the forex weekend or a blocked weekday it returns a "blocked" scan and does nothing @@ -146,6 +151,16 @@ M5 six extra noise gates (minimum score, required agreement, confirmation streng cap, chop-transition volatility, SELL RSI floor). Every gate that removes a setup writes a warning that names itself. +One more gate runs at the very end, after the setup generators and the late entry paths of step 9 +(`_apply_daily_trend_veto`, step 12 in the code; M15 only since 2026-09-30): the **daily-trend +veto**. The daily trend is read from closed D1 bars with the provider's Sunday stub bars dropped +(`signal_aggregator.daily_trend_series`): +1 when the last daily close is above its 50-day EMA and +above the close 60 kept days earlier, −1 when below both, else 0. A finished setup that goes +against a non-zero state is removed (grade D, no setup, warning `DAILY TREND FILTER: BUY against a +falling daily trend — no trade`). It runs after every archetype on purpose, so a chart-pattern +setup is vetoed too. Where the veto is off (M5, H1, H4) the live wrapper still stamps +`report.daily_trend` for the alert text and the signal log; nothing reads it back there. + **9. Trade setup.** Grades A+ and A get the gold ATR setup (`_generate_xau_atr_trade_setup`): stop 2 × ATR; target 1 at 2 × ATR for A+ and 1.5 × ATR for A (1 × ATR for an A carrying two or more exhaustion tags); target 2 at 1.5 × the target-1 distance. Other tradeable grades get the generic swing-based setup, @@ -175,7 +190,12 @@ later generator shipped a setup. reaches `alert_min_quality`, **and** a setup exists. Since 2026-09-26 no second alert goes out on a timeframe within 5 bars of the last alerted setup's bar (`_ALERT_COOLDOWN_BARS`, the backtester's entry cooldown; consecutive-bar re-entries were −25.1R of the −37.8R live book of 2026-09-04 → - 09-25). After that, a fingerprint (signal + direction) stops repeats; a grade upgrade re-alerts. + 09-25). Since 2026-10-01 the five bars are COUNTED in the scan's own primary frame + (`_primary_bars_since`, the backtester's position difference; clock time is only the fallback + for a report without frames) and the last alerted bar per timeframe is read back from + `signals.db` at start-up (`signal_log.last_alerted_bars` → `_restore_alert_cooldowns`), so + neither the settlement pause, the weekend nor a restart lets live alert where the books do not + enter. After that, a fingerprint (signal + direction) stops repeats; a grade upgrade re-alerts. Since 2026-09-27 the header leads with the grade (`Grade A · BUY · XAU/USD · M15`) and the raw score / confidence line is replaced by words from `main.plain_words` (`Direction: strong buy · Reliability: good · timeframes agree`; the cut-offs are the grade table's own, so the words never @@ -187,12 +207,29 @@ later generator shipped a setup. (`_book_rate_line`, from `agent_panel_base_rates.json`). The fingerprint is committed only after a successful send, so a failed send stays eligible next scan. Every send is logged with the signal-log row id. The level-breach monitor reuses the bar's cached M15 report (`_cached_or_scan`) instead of rescanning. -- *CLI* (`main.py`): prints the report; `--json` dumps everything. + Since 2026-09-30 three more display lines: the daily trend in words under the regime line + (`🧭 Daily trend: rising — this buy goes with it`, `main.daily_trend_words`), and inside the setup + block the lot size for the owner's risk limit (`📏`, off until `trade_size.risk_per_trade` is set) + and the count of alerted same-direction plans still open (`🧱`). The size line is private: + `format_signal_message(report, channel=True)` builds the copy without it unless + `trade_size.telegram_channel` is true, and since 2026-10-01 `_broadcast` sends the owner copy to + the owner's own chat only (`_is_owner_chat`): the channel and any other target (a group the owner + used `/autoscan` or a scan button in; the owner lock is per user) get the copy without it. + Both alert paths (candle scan, level break) end in one `_deliver_alert`: the panel, then on the + one-worker `alert-db` pool the exposure lines are refreshed and the row is stamped `alerted_at` + (`signal_log.mark_alerted`), then the send. Stamping BEFORE the send is what lets two alerts at + the same candle close count each other; an alert that reaches no chat clears its own stamp + (`unmark_alerted`) and commits nothing. No database write runs on the event loop. +- *CLI* (`main.py`): prints the report; `--json` dumps everything (including `daily_trend`, + `daily_trend_words`, `size_line`, `open_line`). - *Dashboard* (`dashboard_publisher.py`): writes `snapshot.json`, `news.json` and the static page - into the Apache folder. + into the Apache folder. Each card shows the daily-trend words; a plan shows the already-open + line, and the size line only when `trade_size.dashboard` is true (the page is public). -Two advisory add-ons are attached **after** the scan by the surface, never inside the signal -code: the LLM agent panel (`agent_panel.py`; since 2026-09-17 only the **macro** persona runs by +Three display add-ons are attached **after** the scan by the surface, never inside the signal +code. The first is `exposure.attach_exposure` (2026-09-30): the lot-size line and the +already-open line, a leaf module that imports only `app_config` and `signal_log`. The other two are +advisory: the LLM agent panel (`agent_panel.py`; since 2026-09-17 only the **macro** persona runs by default, `agent_panel.personas: ["macro"]`, one DeepSeek call per panel — the technical, risk and contrarian personas carried no information in two live evaluations and stay available for research; the persona returns a probability that the trade closes profitably, anchored to the setup class's @@ -207,22 +244,30 @@ replayable core never sets it. **11. Logging.** `signal_log.py` writes one row per live scan (from the bot and the CLI; the dashboard process never writes either database, section 3) into `logs/signals.db` (signal, grade, price, indicators, the last 14 candles per timeframe, the news at scan time, the setup, the -gate that removed a setup, a gate-config fingerprint and an engine fingerprint). The engine +gate that removed a setup, a gate-config fingerprint and an engine fingerprint; since 2026-09-30 +also `daily_trend` on every real scan and `alerted_at` on rows whose alert reached a chat). The engine fingerprint hashes the scoring config, so it changes whenever a weight or threshold ships; signal-log comparisons across such a deploy must be split at the deploy time (the 2026-09-09 M15 RSI change is -one). `news_archive.py` writes every headline and +one). The gate fingerprint changes when a gate ships: the M15 daily-trend veto moved M15 rows from +`f9b2f801de54` to `8a8075688d67` (engine fingerprint unchanged), so M15 comparisons split there too. +`news_archive.py` writes every headline and its score into `logs/newsdata.db`. Both are fail-safe: a logging error can never break a scan. **12. Measurement, later.** Two models read the stored candles and score what happened next: advisory outcomes (`advisory_outcome.py`: replays each logged setup with next-open entry, stop first, close-only trailing; a cleaned-mid evidence model, not execution P&L) and scan outcomes (model-free forward moves 1 h / 4 h / 24 h after every scan). `analyze_signal_log.py` turns the -database into reports. +database into reports. Since 2026-10-01 the outcome model checks candle continuity bar by bar as +it walks: a hole in the stored candles AFTER a trade's exit no longer voids the trade, and a hole +reached while it is open ends the walk with the progress kept (so the locked-profit floor is the +same whether the stream arrives in one pass or many). **The backtester is the same pipeline on history.** `backtest/gpu_backtest.py` loads candles from `backtest/cache/*.parquet` (fetched once through the same provider code), precomputes every indicator series over the whole window, and calls `analyze_from_dataframes` bar by bar with news -off. Trades are resolved on the raw price path with next-open entries, spread charged per trade, +off. When the daily-trend veto is on it also loads the D1 frame (1,400-day pad) and slices it as-of +like every other frame, so the veto sees exactly the closed daily bars a live scan would. Trades +are resolved on the raw price path with next-open entries, spread charged per trade, stop-first ties, gap fills at the open, and the trailing stop moved at bar close only. Time filters use bar-close time in both paths, so live and backtest agree. The replay fans out over worker processes; the 5-bar entry cooldown, the only cross-bar state, is applied in an ordered pass @@ -238,9 +283,10 @@ Arrows show who imports whom (generated from the code on 2026-09-15). | File | Role | Uses | Used by | |---|---|---|---| -| `main.py` | Command-line signal report (`--detail`, `--json`, `--panel`) | pair_signals, signal_aggregator, agent_panel, violent_watch, app_config | dashboard_publisher (formatting helpers) | -| `telegram_bot.py` | Owner-locked bot: candle-close scan loops, alerts, breach monitor, heads-ups, `/scan /status /news /autoscan /cache /clearcache /help /start` | pair_signals, signal_aggregator, providers, agent_panel, violent_watch, news_engine, llm_client, signal_log, app_config | nobody (a top-level process) | -| `dashboard_publisher.py` | Static web publisher daemon; writes JSON + HTML into the Apache folder. Its own process with its own news caches and its own DeepSeek daily cap. Never writes `signals.db` or `newsdata.db`: `main()` calls `app_config.set_db_writes_for_process(False)`, a process-wide switch every SQLite writer honours (owner decision 2026-09-17; the bot is the single writer because the dashboard is not always running) | pair_signals, providers, main, agent_panel, violent_watch, news_engine, app_config | nobody. Hard rule: never imports and is never imported by `telegram_bot` (test-enforced) | +| `main.py` | Command-line signal report (`--detail`, `--json`, `--panel`); the display helpers every surface shares (`plain_words`, `daily_trend_words`) | pair_signals, signal_aggregator, agent_panel, violent_watch, exposure, app_config | dashboard_publisher and telegram_bot (formatting helpers) | +| `telegram_bot.py` | Owner-locked bot: candle-close scan loops, alerts, breach monitor, heads-ups, `/scan /status /news /autoscan /cache /clearcache /help /start` | pair_signals, signal_aggregator, providers, agent_panel, violent_watch, exposure, news_engine, llm_client, signal_log, app_config | nobody (a top-level process) | +| `exposure.py` | Display-only lot-size line and already-open line (`attach_exposure`, 2026-09-30). `lot_size` floors to the lot step in Decimal so the risk limit is never exceeded; `size_shared` answers whether config.json shares the size with a public audience. A leaf: never raises, never read by the core | app_config, signal_log | main, telegram_bot, dashboard_publisher | +| `dashboard_publisher.py` | Static web publisher daemon; writes JSON + HTML into the Apache folder. Its own process with its own news caches and its own DeepSeek daily cap. Never writes `signals.db` or `newsdata.db`: `main()` calls `app_config.set_db_writes_for_process(False)`, a process-wide switch every SQLite writer honours (owner decision 2026-09-17; the bot is the single writer because the dashboard is not always running). It does READ `signals.db` (`mode=ro`) for the already-open line since 2026-09-30 | pair_signals, providers, main, agent_panel, violent_watch, exposure, news_engine, app_config | nobody. Hard rule: never imports and is never imported by `telegram_bot` (test-enforced) | | `static/dashboard.html`, `.css`, `.js` | The public dashboard page (reads `snapshot.json` / `news.json`). Bootstrap 5 + jQuery + Chart.js + Font Awesome straight from a CDN, no build step; M5/M15/H1/H4 each get a card in one column grid, with a diverging Chart.js bar chart for direction and one per card for the indicator votes | | copied by the publisher | | `static/news_summary.php` | The one dynamic endpoint: on-demand DeepSeek summary for a clicked headline, whitelist-gated against `news.json`, own fail-closed daily cap, key read from `/etc/gold-dashboard/deepseek.key` (never repo config) | | installed by the publisher | | `docs/user_guide.html`, `docs/beginner_guide.html` | Human guides. Keep them in sync with signal behaviour. There are no PDF builds | | | @@ -253,7 +299,7 @@ Arrows show who imports whom (generated from the code on 2026-09-15). | `pair_signals/xauusd.py` | The config factory: overlay + validated per-timeframe overrides + confirmation map. Its docstring is the evidence log for every override | config, app_config, engine | main, telegram_bot, dashboard_publisher, backtester | | `pair_signals/__init__.py` | Registry of pair producers (`PAIR_PRODUCERS`) and `produce_signal` | config, signal_aggregator, providers, xauusd | the surfaces, backtester | | `engine.py` | Indicators, regime, composite score (`TechnicalEngine`) | config | signal_aggregator, backtester, replay_record, pair factory | -| `signal_aggregator.py` | Multi-timeframe merge, penalties, grade, gate chain, setup generators, reasons; `analyze_pair` (live) and `analyze_from_dataframes` (replayable core) | engine, chart_patterns, news_engine, providers, signal_log, config | main, telegram_bot, backtester, pair_signals, analyze_signal_log (agent_panel only for type hints) | +| `signal_aggregator.py` | Multi-timeframe merge, penalties, grade, gate chain, setup generators, reasons, the daily-trend veto and its one definition (`daily_trend_series` / `daily_trend` / `daily_trend_needed`, shared by live, the replay and the twin's host side); `analyze_pair` (live) and `analyze_from_dataframes` (replayable core) | engine, chart_patterns, news_engine, providers, signal_log, config | main, telegram_bot, backtester, pair_signals, analyze_signal_log (agent_panel only for type hints) | | `chart_patterns.py` | The XAU_CP pattern detector | | signal_aggregator | | `position_sizing.py` | Decision-time feature builder (65 columns) and the reserved stake multiplier (always 1.0). Bounded lookbacks keep live == backtest | config | backtester (`--export-features`) | @@ -272,9 +318,9 @@ Arrows show who imports whom (generated from the code on 2026-09-15). | File | Role | |---|---| | `news_engine.py` | Calendar + RSS headline sentiment with gold-specific keywords; class-level caches; headline scores upgraded by the local XAU model (`news_model.py`, mode `primary` live since 2026-09-27) or by DeepSeek (modes `off` / `fallback`, `_enrich_headline_scores`); DeepSeek keeps event direction and the summary; display and warnings only | -| `llm_client.py` | Thin `requests` client for DeepSeek (`deepseek-v4-flash`, `thinking: disabled`, one attempt, UTC-day request cap). Every caller has a non-LLM fallback | +| `llm_client.py` | Thin `requests` client for DeepSeek (`deepseek-v4-flash`, `thinking: disabled`, one attempt, UTC-day request cap). Every caller has a non-LLM fallback. `chat_json` never raises and never logs the API key (2026-10-01: a key with a line break made `requests` quote the `Authorization` value in its error text); an unserialisable payload is refused before the cap is counted | | `news_archive.py` | Durable headline corpus in `logs/newsdata.db` (one row per distinct headline, upgrade-only scores, provenance of the scorer). Its own database on purpose: a `signals.db` rotation in 2026-09 destroyed a 3-week corpus of 4,054 headlines and `logs/` is gitignored. Hooked at the end of `news_engine._fetch_news_sentiment_raw`, before the ±0.1 neutral-headline filter, so neutral headlines are kept for training; do not move the hook. Since 2026-09-17 the corpus is complete for a gold scorer: `xau_score` holds DeepSeek's gold score for every LLM-scored row (it used to survive only for gold-keyword headlines, so 930 of the first 1,546 rows had no gold label; older rows read NULL), headlines naming no tracked currency are scored and archived too (`currencies_json` `[]`; they are never returned to the platform), and every new headline goes to DeepSeek in the cycle it arrives, 25 per call. **Frozen in `news_model.mode` `primary` (live since 2026-09-27)**: the local model's output is not training data, so nothing is archived except the optional DeepSeek audit copies (`deepseek_audit_per_day`, shipped 0); `off` / `fallback` archive as before, a `fallback` model row carrying `teacher_model` `xau_transformer@` | -| `agent_panel.py` | Advisory persona panel (macro by default; technical / risk / contrarian and the chair available through `agent_panel.personas` for research). Attached post-hoc. v2 (2026-09-17, `PANEL_PROMPT_VERSION`): every persona sees a code-computed `context` block (prior move in ATR, distance from the 24 h extreme, candle size, stop/target in ATR, the class base rate from `agent_panel_base_rates.json`, event timing against the expected hold) and returns `p_win` plus one reason each way; stance is derived symmetrically (≥ 60 support, ≤ 40 oppose); the verdict is the equal-weight mean (quorum 3, `verdict_source: "mean"`, default). `macro` (stance verbatim) and `chair` (synthesizer + `_apply_oppose_guard`) remain for comparison. The prompt version is logged and hashed. Persona weights are not to be tuned before about 300 graded outcomes with a held-out half | +| `agent_panel.py` | Advisory persona panel (macro by default; technical / risk / contrarian and the chair available through `agent_panel.personas` for research). Attached post-hoc. v2 (2026-09-17, `PANEL_PROMPT_VERSION`): every persona sees a code-computed `context` block (prior move in ATR, distance from the 24 h extreme, candle size, stop/target in ATR, the class base rate from `agent_panel_base_rates.json`, event timing against the expected hold) and returns `p_win` plus one reason each way; stance is derived symmetrically (≥ 60 support, ≤ 40 oppose); the verdict is the equal-weight mean (quorum 3, `verdict_source: "mean"`, default). `macro` (stance verbatim) and `chair` (synthesizer + `_apply_oppose_guard`) remain for comparison. The prompt version is logged and hashed. v2.2 (2026-10-01) changed the persona's INPUT, not the prompt text: the event timing reads the cached calendar week `attach_panel` hands to `_price_context` (`NewsEngine.peek_week_events`, which never fetches; a hold past UTC midnight sees tomorrow's release), and the ten headlines are the newest ten, not the first ten in feed order. Persona weights are not to be tuned before about 300 graded outcomes with a held-out half | | `violent_watch.py` | Display-only violent-move warning and heads-up. Attached post-hoc | | `news_model.py` | The local headline scorer (2026-09-27): vendored CPU runtime of the owner's ML-NEWS XAU transformer (DeBERTa-v3-small, two heads: moves gold? / up or down?, ONNX; `onnxruntime` + `tokenizers` imported lazily inside `load_model`, so the bot starts without them) plus a load-once, never-raising singleton (`get_news_model`, `teacher_tag`, `warm_up_in_background`). Gold head only: `news_engine._apply_model_sentiment` writes `xau_score` and XAU's per-currency entry, never USD. Bundle (280 MB) lives outside the tree at `news_model.path`. Spec and deviations: `docs/news_model_integration_spec_2026-09-27.md`; ops: `deploy/README-news-model.md` | @@ -282,14 +328,14 @@ Arrows show who imports whom (generated from the code on 2026-09-15). | File | Role | |---|---| -| `signal_log.py` | SQLite logger for every live scan (`signals`, `news_snapshots`, `scan_outcomes`; view `signals_with_news`); opportunistic scan-outcome labelling; advisory outcome seeding | -| `advisory_outcome.py` | The deterministic outcome model for logged setups | +| `signal_log.py` | SQLite logger for every live scan (`signals`, `news_snapshots`, `scan_outcomes`; view `signals_with_news`); opportunistic scan-outcome labelling; advisory outcome seeding; `mark_alerted` (stamps `alerted_at` just before the alert is sent and returns the stamp it wrote), `unmark_alerted` (clears that stamp when no chat received the alert), the read-only `open_alerted_setups` (alerted plans whose outcome is still `PENDING`, one per event, stale rows dropped) and the read-only `last_alerted_bars` (newest alerted bar per timeframe, for the bot's cooldown after a restart) | +| `advisory_outcome.py` | The deterministic outcome model for logged setups. Continuity of the stored candles is judged one step at a time inside the walk (`_next_bar_follows`); `_has_continuity` stays all-or-nothing for the scan-outcome labeller | | `analyze_signal_log.py` | Read-only reports on `signals.db`: trade replay, panel scorecard, parity between surfaces, news lead/lag, scan outcomes (`--stored-candles`, `--news`, `--parity`, `--scan-outcomes`) | | `scripts/backfill_signal_outcomes.py` | Batch seeding / advancing of advisory outcomes on an imported database (`--dry-run`, `--reseed-unresolvable`) | | `scripts/label_scan_outcomes.py` | Batch labelling of scan outcomes for a backlog (`--dry-run`) | -| `scripts/panel_replay.py` | Offline replay of the agent panel on logged, resolved setups: rebuilds each persona's payload (exact `panel_payloads_json` when present, else from the stored JSON blobs, interned news and the kept Oanda cache), runs the CURRENT prompts, and scores Brier / calibration / constant-voter rate / counterfactual books against the logged v1 takes and the base-rate predictor. `--dry-run`, `--max-calls`, `--limit`, `--out`. Read-only on the DB; 4 DeepSeek calls per event | +| `scripts/panel_replay.py` | Offline replay of the agent panel on logged, resolved setups: rebuilds each persona's payload (the exact logged one from `panel_payloads_json` per persona when the row has it, macro alone on live rows since 2026-09-17; else from the stored JSON blobs, interned news and the kept Oanda cache; the CSV column `stored_payloads` names the personas that replayed a logged payload), runs the CURRENT prompts, and scores Brier / calibration / constant-voter rate / counterfactual books against the logged v1 takes and the base-rate predictor. `--dry-run`, `--max-calls`, `--limit`, `--out`. Read-only on the DB; 4 DeepSeek calls per event | | `scripts/build_panel_base_rates.py` | Builds `agent_panel_base_rates.json` (per timeframe × strategy × grade × direction: n, win rate, mean R, TP1 reach, median hold, book MD5) from the exported reference books. Re-run after any re-pin | -| `agent_panel_base_rates.json` | The base-rate table the panel anchors its probabilities to (built 2026-09-17 from the relabelled W2/W3 books) | +| `agent_panel_base_rates.json` | The base-rate table the panel anchors its probabilities to (rebuilt 2026-10-01 from the reference books of section 9, the M15 one with the daily-trend veto; `generated_at` and each book's MD5 are in the file) | ### Backtesting and research @@ -313,7 +359,7 @@ Arrows show who imports whom (generated from the code on 2026-09-15). | `app_config.py` | Loads `config.json` once per process; env var → config.json → refuse; validates keys | | `config.json.example` | The schema of `config.json` (which is gitignored) | | `deploy/README-dashboard.md`, `deploy/gold-dashboard.service` | Live-server checklist and systemd unit for the dashboard daemon | -| `test_all.py` | The QA suite (38 modules; Module 36 is the GPU backend parity module, Module 37 the CUDA decision-stage parity module, Module 38 the local news model) | +| `test_all.py` | The QA suite (40 modules; Module 36 is the GPU backend parity module, Module 37 the CUDA decision-stage parity module, Module 38 the local news model, Module 39 the daily-trend veto and the wide-stop cap, Module 40 the trade-size / already-open / daily-trend display lines) | --- @@ -350,7 +396,20 @@ symbol and interval maps. A test enforces that subclasses do not override the sh scheduled closure since 2026-09-13. A "stale data" warning at any other weekday hour is a real feed stall. - The backtester fetches history through `fetch_range` (paginated, retries a failed page, logs an - error for interior gaps longer than 3 bars) and stores it under `backtest/cache/`. + error for interior gaps longer than 3 bars) and stores it under `backtest/cache/`. A DATE window + with no bars in it (a closed day at the head of a padded window) is an empty page, not a failure: + until 2026-09-30 the Oanda provider raised on it and `fetch_range` aborted after three tries, so + a D1 pad starting on a Saturday could not be fetched at all. Only a closure-sized window gets + that reading (`EMPTY_WINDOW_MAX`, 5 days: a weekend joined to a holiday, and no malformed candle + skipped); an empty answer for a longer window (a normal 15-day M5 page) raises, so `fetch_range` + retries it and refuses the history after three tries instead of caching a hole (2026-10-01). + A `count` request (the live path) with nothing in it still raises. +- Daily (D1) candles are UTC days (`dailyAlignment=0`). Oanda also emits a 1–2 hour stub bar + labelled Sunday for the reopening (about 600 of them since 2015); it is a real bar for the H4 + confirmation engine, and the daily-trend definition drops Sunday labels before it computes + anything. Since 2026-09-30 D1 is a required frame of every M15 scan (live and backtest) and is + fetched for display on M5 and H1 (cached until the next daily close; a failure there leaves the + line empty and never fails the scan). --- @@ -377,6 +436,7 @@ gates live in `config.py` and the pair factory, never in `config.json`. Blocks ( | `signal_log` | `enabled`, `path`, `log_blocked_scans` (env `SIGNAL_LOG_ENABLED`, `SIGNAL_LOG_PATH`, `SIGNAL_LOG_BLOCKED_SCANS`) | | `news_archive` | `enabled` (default true), `path` (env `NEWS_ARCHIVE_ENABLED`, `NEWS_ARCHIVE_PATH`) | | `news_model` | `path` (bundle directory, required when mode ≠ off), `threads` (default 4), `mode` (`off` default = DeepSeek/keyword path; `fallback` = model only where DeepSeek failed; `primary` = model scores every headline, no DeepSeek headline calls), `deepseek_audit_per_day` (default 0; primary only, archive-only DeepSeek labels). Env `NEWS_MODEL_MODE` overrides `mode` per process (the dashboard unit). Live: `primary` on both processes | +| `trade_size` | Lot-size guidance on the alert (display only, 2026-09-30): `risk_per_trade` (US dollars; 0 or absent = the size line is off), `contract_size` (ounces per lot, default 100), `lot_step` (default 0.01), `telegram_channel` and `dashboard` (default false: the size shows the owner's risk, so every Telegram chat other than the owner's own (`telegram.chat_id`) — the channel, a group — and the public page get it only when these are true). No env override | | `log_level` | Logging level | A legacy top-level `market_outlook` block is tolerated and ignored with one warning (the feature @@ -392,7 +452,7 @@ day, 30 kept), `logs/dashboard.log`. Read the databases with `mode=ro`, never `i ## 6. Testing - `venv/bin/python test_all.py` runs everything with mocked network. Expect - `RESULTS: 581 passed, 0 failed, 581 total` (2026-09-27). Module 38's last test loads the + `RESULTS: 632 passed, 0 failed, 632 total` (2026-10-01). Module 38's last test loads the real news-model bundle when `NEWS_MODEL_DIR` (or the configured `news_model.path`) exists and prints SKIP otherwise. On a box with a visible CUDA device Module 36 fails if the GPU backend falls back (the reason is in the message); with @@ -403,6 +463,12 @@ day, 30 kept), `logs/dashboard.log`. Read the databases with `mode=ro`, never `i both database writers at a suite-private temp dir; before 2026-09-17 the RSS fixture tests wrote fixture headlines into the real `logs/newsdata.db`, and a Module 32 test now pins the isolation. One Twelve Data interval-map test reaches the network and can time out on a slow connection; a rerun clears it. +- The suite is meant to run on the live box too (Python 3.9, SQLite 3.34, no `pyarrow`, no CUDA): + the four tests that must write a Parquet fixture print a notice and return when no Parquet + engine is installed, and a test that needs an older database builds it forwards + (`_legacy_signals_db39`: the base schema plus every additive column but the one under test), + never with `ALTER TABLE … DROP COLUMN` (SQLite 3.35). Verified 2026-10-01 with both Parquet + engines blocked and `FB2_BACKEND=cpu`: 0 failures. - Test module numbers are cited from this file, so never renumber them. Module 10 and 26 are gaps. - When a test needs the weekend rule, patch `providers.base._is_forex_weekend` and remember the other copies. Signal-log fixtures that exercise candle continuity must use a weekday bar. @@ -423,7 +489,13 @@ day, 30 kept), `logs/dashboard.log`. Read the databases with `mode=ro`, never `i fills. Marginal edges must also pass the 2024-06-01 → 2025-03-30 out-of-sample window. Backtest windows go back at most 3 years (owner preference, 2026-09-05). `round_trip_cost` stays at 0.30 because it also drives the live target floor (0.65 would move target 1's floor from about $9 to - $19.50); changing it is its own rule-1 event. + $19.50); changing it is its own rule-1 event. A change whose purpose is to cut the losing runs + rather than add R may instead be judged by a **risk-first rule** (owner decision 2026-09-30), + written before the runs: 3-year R at least 97% of the control, profit factor up by 0.10 or more, + max drawdown not deeper, the 2019 → 2026 window better on R, on drawdown and in 2021 + 2022, an + unseen holdout, both costs, and at least 3 of 4 neighbouring definitions holding. Under it the + halves of a purely rising window are reported, not clauses. The owner chooses the rule per + campaign; the default stays the acceptance above. 2. **Per-timeframe overrides live in the pair factory**, never in the global defaults. The same knob has had opposite effects on different timeframes several times. Record the evidence in the factory docstring. @@ -466,7 +538,7 @@ overrides (each one has its validation record in the `pair_signals/xauusd.py` do | Timeframe | Confirmations | Overrides on top of the defaults | |---|---|---| | M5 | M15, H1 | hour 9 UTC blocked and hour 21 unblocked; chart-pattern entries on (buy-side only: inverse head and shoulders, double bottom, bull flag); pullback entries off; partial EMA alignment scores neutral; trailing stop arms at 0.60R and trails 0.85R; shorts blocked in the `transitioning` regime | -| M15 | M5, H1 | target floor 1R; the mixed `trending` regime blocked; shorts blocked at 15:00 and 17:00 UTC; trailing stop arms at 0.70R and trails 0.75R; RSI weight halved in every regime; grade-A shorts blocked (`sell_min_quality` `A+, B`) | +| M15 | M5, H1 (+ D1 for the veto) | target floor 1R; the mixed `trending` regime blocked; shorts blocked at 15:00 and 17:00 UTC; trailing stop arms at 0.70R and trails 0.75R; RSI weight halved in every regime; grade-A shorts blocked (`sell_min_quality` `A+, B`); daily-trend veto in both directions (`daily_trend_veto` `both`, since 2026-09-30) | | H1 | M15, H4 | every short blocked (`sell_min_quality` empty) | | H4 | H1, D1 | none. H4 remains "unproven" (few trades) | @@ -475,7 +547,13 @@ to Friday plus Sunday evening; shorts need grade A+, A or B and high-or-extreme trailing stop arms at 0.6R and trails 0.75R; gated target floor 0.5R and printed target floor 1R; round-trip cost $0.30 with target 1 at least 30 × cost; the generic and chart-pattern setups floor the stop at $8; the gold -ATR setup uses a 1 × ATR target before the grade scale (2× at A+, 1.5× at A) and a 2 × ATR stop; the H4 regime filter is off; news never gates. +ATR setup uses a 1 × ATR target before the grade scale (2× at A+, 1.5× at A) and a 2 × ATR stop; the H4 regime filter is off; news never gates; +the daily-trend veto is off (M15 alone turns it on) and the wide-stop cap is off everywhere +(`wide_stop_cap_atr` 0.0, a research knob that failed its rule). + +The daily-trend veto failed the same rule on M5 (it deepens the 3-year losing run from 10.1R to +15.2R and loses the 2019 → 2026 window) and on H1 (−9.7R of +70.0R). Do not turn it on there +without a new campaign; the daily trend is printed on those alerts as context only. One semantic to know: `blocked_hours_utc` is applied to the bar **label** hour inside the core and to the bar **close** hour in the live wrapper and the backtester. So H1 can only trade bars @@ -501,11 +579,19 @@ CuPy GPU backend and `--backend cpu` both reproduce every MD5 below; earlier rep | Book | Window | Trades / R / PF | MD5 of the CSV | |---|---|---|---| | M5 W2 | 2024-09-05 → 2026-09-05 | 781 / +146.07 / 1.52 | `e599c8a8652f95b7fcdf5a7ef2b564f5` | -| M15 W2 | 2024-09-05 → 2026-09-05 | 813 / +154.15 / 1.53 | `5b65ecf6fad9a8da412749066fd50d36` | -| M15 OOS | 2024-06-01 → 2025-03-30 | 313 / +68.50 / 1.67 | `09b8a8e7f5833436c20aef60ca0a7ff7` | +| M15 W2 | 2024-09-05 → 2026-09-05 | 597 / +155.89 / 1.78 | `dfc647ee02afd179dd9e031f344ed380` | +| M15 OOS | 2024-06-01 → 2025-03-30 | 258 / +63.78 / 1.80 | `e87b6de12f90f3b5a5d41a953ee9ff7a` | | H1 W3 | 2023-09-05 → 2026-09-05 | 253 / +70.0 / 1.98 | `a44fd19b4067f46e5bd98d50cc3eff92` | | H4 W3 | 2023-09-05 → 2026-09-05 | 114 / +21.7 / 1.59 | `f3587bae3c5dc3eda468ea3588b17dcb` | +Re-pinned 2026-09-30. M15 only: the daily-trend veto (M15 override 7). W2 216 trades removed, ++1.7R, max drawdown 15.6R → 11.4R; OOS 55 removed, −4.7R (the accepted cost of the risk-first rule +in a rising window), drawdown 12.4R → 9.1R; previous pins `5b65ecf6…` (813 / +154.15 / 1.53) and +`09b8a8e7…` (313 / +68.50 / 1.67). M15 W3 with the veto 824 / +205.12 / 1.73, max drawdown 12.4R, +`ad3b0dbda0bf77b659caea1c53be6a50` (was `70195fb0…`, 1105 / +195.79 / 1.49, drawdown 20.9R). The +M5, H1 and H4 pins are unchanged. An M15 book needs the D1 frame of its window (1,400-day pad) in +the cache; `--set xau_strategy.daily_trend_veto=off` reproduces the previous M15 pins byte for byte. + Re-pinned 2026-09-26. M15: the grade-A SELL block (M15 override 6, 162 trades removed, +8.0R); previous pins `b4aa07e3…` (931 / +146.14 / 1.43) and `9fd624fb…` (355 / +60.67 / 1.49); M15 W3 (3-year) with the block 1105 / +195.79 / 1.49, `70195fb0e6865b3a46ee8a9aa8844e51`. All four @@ -520,18 +606,28 @@ columns of all five books are byte-identical to the previous pins (`29120a1e…` `6f8f24b4…`, `3b45d12b…`, `e75adae1…`); trades, R and PF are unchanged. `--strategies` accepts the live names (`XAU` still means `XAU_ATR`). -Other current-HEAD figures at $0.65: M5 W3 (3-year) 870 / +153.6R / PF 1.50; M15 W3 1273 / -+186.6R / PF 1.39; M5 12-month (2025-08-14 → 2026-08-14) 560 / +108.81R / PF 1.54; -H1 12-month (2025-08-04 → 2026-08-04) 91 / +22.2R / PF 1.79. All five MD5s above were -reproduced on 2026-09-15 after the cache prune and on 2026-09-18 on both backends -(`backtest/research/gpu_backend_2026-09-18.md`). +Other current figures at $0.65 (2026-10-01): M5 W3 (3-year) 870 / +153.64R / PF 1.50 / max +drawdown 10.1R (`d20ec8a3…`); M15 W3 824 / +205.12R / PF 1.73 / 12.4R; M5 12-month (2025-08-14 → +2026-08-14) 560 / +108.81R / PF 1.54; M15 12-month (2025-08-04 → 2026-08-04) 266 / +92.3R / +PF 2.12 / 8.3R; H1 12-month (2025-08-04 → 2026-08-04) 91 / +22.2R / PF 1.79. The long robustness +window 2019-01-01 → 2026-08-07 (allowed as supporting evidence, owner decision 2026-09-30): M15 +1473 / +223.33R / PF 1.42 / max drawdown 41.5R (`aa5253aa…`; by year +14.0, +45.1, −27.3, +3.2, ++1.8, +53.5, +92.5, +40.6), M5 1052 / +128.10R / PF 1.34 / 18.4R (`3f0c091c…`; −1.0, −2.8, −1.7, +−2.2, −2.8, +1.0, +40.5, +97.0: the M5 rules take few trades and earn nothing before 2025). The +unseen holdout 2026-09-05 → 2026-10-01: M15 18 / −4.62R, M5 33 / −2.45R, H1 3 / −3.04R. The pins of +the table above were reproduced on 2026-09-15 after the cache prune, on 2026-09-18 on both +backends (`backtest/research/gpu_backend_2026-09-18.md`) and on 2026-10-01 with both scripts. The CUDA research twin (`backtest/cudaBasedBacktesting.py`, section 3) reproduced all five MD5s above on 2026-09-18 in 0.8–2.4 s per book, plus the four 6-month acceptance books (2026-03-18 → 2026-09-18, M5 and M15 at $0.65 and $0.30: `86111c36…`, `e70d0252…`, `b8c98083…`, `e66b5fbd…`, each equal to the reference script's), the M15 OOS `--export-features` CSV (`6239d197…`) and `--record --record-bars` tables, and ten research-flag variants -(`backtest/research/cuda_backtester_2026-09-18.md`). +(`backtest/research/cuda_backtester_2026-09-18.md`). Parity was re-proved for the 2026-09-30 change: +the five pins above with both scripts, six `--set` books on W3 (veto both / buy / sell and the stop +cap on M15, veto and cap on M5), the M15 OOS `--export-features` CSV with the veto on +(`732f396bb3ee6036b62750a8c755194e`) and its `--record --record-bars` tables +(`backtest/research/daily_trend_campaign_2026-09-30.md`). Kept cache windows (`backtest/cache/oanda_XAU_USD___.parquet`; the padded start is the window start minus 62 days for M5/M15/H1, 234 for H4 and 1,400 for D1, so the D1 @@ -539,7 +635,12 @@ and H4 files of one run have different start dates than its M5 file): 3-year `20 2-year `20240705_20260905`, OOS `20240331_20250330`, M5 12-month `20250613_20260814`, M15/H1 12-month `20250603_20260804`, 5-year `20210531_20260804`, 7.6-year `20181031_20260807`, the 2026-08-14 parity window `20250814_20260814`, the 6-month CUDA acceptance window -`20260115_20260918` (M5/M15/H1), their D1/H4 pads, and `calendar.parquet`. +`20260115_20260918` (M5/M15/H1), their D1/H4 pads, and `calendar.parquet`. Added 2026-09-30 for +the daily-trend veto (an M15 run needs the D1 frame of its window): `1d_20201105_20260905` (W2), +`1d_20200801_20250330` (OOS), `1d_20150303_20260807` (the 7.6-year window), `1d_20211004_20260804` +and `4h_20241213_20260804` (the M15 / H1 12-month window), and the holdout window +`20260705_20261001` (M5 / M15 / H1) with `1d_20221105_20261001` and `4h_20260114_20261001`. The +M15 3-year run reads `1d_20191105_20260905`, which H4 W3 already kept. The research recorder (`--record DIR [--record-bars]`) writes Parquet files describing every accepted trade (indicator votes on all three timeframes, penalties, setup, outcome, calendar @@ -567,7 +668,9 @@ MD5s. Scorecards from it are in-book statistics, never decisions. there too since 2026-09-08); use `setup_json.strategy`. - The `--stored-candles` scorecard in `analyze_signal_log.py` still advances outcomes without the locked-profit floor step that the logger and the backfill script use (owner decision pending), so - the two measurement paths can disagree on gap rows. + the two measurement paths can disagree on gap rows. Those are now only the trades still OPEN at + a hole (3 of 111 setups on the 2026-10-01 database copy; until then the replay also threw away + every trade that had exited before a later hole, 12 of 111). - There is no holiday awareness anywhere. On Labor Day 2026 Oanda published no bars for hours, the bot rescanned the stale bar 43 times, and the violent-move watch called the 12–14 UTC slot hot. Interior gaps in history are logged, never fatal, for the same reason. @@ -584,6 +687,29 @@ MD5s. Scorecards from it are in-book statistics, never decisions. ≈ 0.03 with the next). Both news gating flags stay off. - The level-breach rescan and the candle loop both skip the settlement-pause window (`_closed_bar_in_settlement`); a rescan there only re-reads the last pre-pause bar. +- The alert cooldown counts bars of `report._frames` (set by `analyze_pair`). A report built by + hand has no frames and falls back to clock time, which releases the first bars after the + settlement pause or the weekend too early; that is the fallback, not the rule. +- An alert's row is stamped `alerted_at` BEFORE the send and the stamp is cleared when no chat + received it, also when the alert's task is cancelled while the stamp is being written (a + shutdown at that moment: the worker job cannot be stopped, so its future's done-callback + undoes it). A cancellation in the MIDDLE of the send keeps the stamp, because a chat may + already hold the message. Two residues, both accepted: if a send fails completely after + another alert already counted the row, the other alert's "already open" line is one too + high; and a process KILLED between the stamp and the send leaves a stamp on an alert nobody + received (it also seeds the cooldown at the next start, which errs on the quiet side). +- `news_engine._record_source("news_model", …)` is the only sign that the local headline model + failed a batch (WARNING once, `/status` line `News model: not scoring`); in `primary` mode + nothing else scores those headlines for the cycle. The line describes the latest cycle that + needed the model: a cycle with nothing for it to score (fallback mode with DeepSeek healthy) + drops an older failure instead of leaving it on `/status`. `/status` failure details are + exception text and are Markdown-escaped; an unpaired underscore made Telegram reject the reply. +- The agent panel's scorecards split at `PANEL_PROMPT_VERSION` v2.2 (2026-10-01): same prompt + text, different input (calendar-week event timing, headlines newest first). `_price_context` + never fetches the calendar itself; the offline replay passes an as-of context and must not see + today's events. +- The Oanda empty-window rule cannot catch an anomalous empty answer for a window of 5 days or + less (a short head page; every 3-day M1 page, a timeframe nothing fetches). - The agent panel never runs inside `analyze_from_dataframes`, and its cached verdict is never reused when the evidence changed; the daily cap is the real spend bound. - Live == backtest also means the backtester never writes to `signals.db` and never builds a @@ -624,6 +750,45 @@ MD5s. Scorecards from it are in-book statistics, never decisions. and never feed the printed target back into a gate. - "Stop hunting" was tested on the live log (2026-09-26): 71% of live stop-outs ran a full stop distance or more past the stop, the same as the books' 70–72%. The losers were real moves. +- The daily-trend veto has ONE definition (`signal_aggregator.daily_trend_series`; EMA span 50 and + lookback 60 are module constants, validated literals). Live, the bar replay and the twin's host + side (`trend_flags`) all call it; the kernel only receives a flag per bar. Changing the + literals, the Sunday filter or the as-of rule (a daily bar labelled D counts from D + 1 day ≤ the + primary bar's close) is a rule-1 event and a parity event. The live frame holds 1,000 daily bars + and a backtest pad 1,400 days; the EMA seed has decayed below float precision in both, so the + states agree. +- With the veto on, D1 is a REQUIRED frame of the M15 scan: a failed D1 fetch fails the scan + (`DATA UNAVAILABLE: 1d — scan failed`), it does not fall back to "no veto". After a deploy watch + that rate. Inside the core a missing D1 slice only adds the notice `Daily trend filter requested + but 1d data unavailable` and ships the setup; the replay reaches that state only before the + first daily bar of its pad. +- The veto is the LAST step of the core (after the chart-pattern path and the stale-veto cleanup), + so `gate_reason` lists `DAILY TREND FILTER` ahead of the gate chain. The rule itself only + removes a plan. In a book a removed entry also moves the 5-bar cooldown, which could free a + later entry: that is possible in principle and happened 0 times in every book of the campaign + (an observation, not a guarantee; count the added trades in any new comparison). The wide-stop cap never + changes the trade set either (it is applied after the min-RR and spread gates); a smaller stop + can lower the printed 1R target, never raise it. +- The control for any M15 candidate is now the factory config WITH the veto. Comparing against a + pre-2026-09-30 M15 number needs `--set xau_strategy.daily_trend_veto=off` (reproduces + `5b65ecf6…` / `09b8a8e7…` / `70195fb0…`). +- The veto does not find big losers. The trades it removes are break-even after costs (−0.03R to + −0.05R each against +0.15R to +0.25R for the kept ones); the gain is fewer flat trades and + shorter losing runs. The SELL half is carried by the rising 3-year window and the BUY half by the + falling years of 2019 → 2026; `buy`-only and `sell`-only both fail. 2021 is not improved and 2023 + is worse: in a sideways year the daily trend flips late. +- The lot-size line reveals the owner's risk per trade. It is stamped per surface + (`exposure.attach_exposure`) and the channel copy of an alert is built separately + (`format_signal_message(report, channel=True)`); never print `report.size_line` on a public + surface without checking `exposure.size_shared`. The dashboard snapshot is public. +- "Already open" is the advisory outcome model's view of ALERTED plans (`alerted_at` set, outcome + `PENDING`), not a broker position. Rows alerted before the 2026-09-30 deploy carry no + `alerted_at`, so the line starts empty and fills as alerts go out; a `PENDING` row older than the + model's own time-out (300 bars, with a weekend allowance) is not shown. Stacked entries were not + worse trades in the books, so the line is information, never a reason the code skips an alert. +- `signals.daily_trend` is the core's value on M15 and the live wrapper's display stamp on M5, H1 + and H4 (`_stamp_daily_trend_for_display`, one extra cached D1 fetch on M5 / H1); it is NULL on + blocked and failed scans. The stamp sits in `analyze_pair`, never in `analyze_from_dataframes`. - The dashboard's due-timeframe check uses the raw wake time; flooring `wake − buffer` skipped every H1 and H4 scan once (2026-08-02). - The agent panel's price context reads the frames `analyze_pair` stashes on `report._frames`. @@ -644,6 +809,10 @@ One line per validated change or verdict, newest first. Full evidence in the nam | Date | Decision | Where the evidence is | |---|---|---| +| 2026-10-01 | Bug review of the whole codebase (owner request), then the fixes. Signal path clean: indicators, grading, gates, the daily-trend veto, the as-of logic and both backtesters checked, and no live module needs a newer Python or SQLite than the server has. 17 defects elsewhere, all FIXED, each with a test; books byte-identical (the five pins reproduced on both scripts, 0 cache fallbacks). **Alert path**: the "already open" line was stamped at scan time, so two alerts at one candle close never counted each other (the M5 and M15 books enter together 42 times in three years); the row is now stamped just before the send on a one-worker `alert-db` pool and un-stamped when no chat received it (`_deliver_alert`, one path for the candle scan and the level break; no SQLite write on the event loop any more). The 5-bar cooldown counted clock time and forgot its state at every restart: it now counts the scan's own bars like `_apply_cooldown` (the old rule released the 22:00 UTC bar after the settlement pause: 6 M15 and 3 M5 entries the books never take in three years) and is restored from `signals.db` at start-up. The lot-size line went to every target that was not the channel (a group the owner ran `/autoscan` in): now the owner's own chat only. **Measurement**: the outcome model rejected a whole candle stream for a hole anywhere in it; continuity is now judged bar by bar in the walk, so the stored-candle replay resolves 108 of 111 logged setups instead of 99, all 108 equal to the R the live resolver stored. **History fetch**: an empty answer is accepted only for a window of 5 days or less with no malformed candle (a longer one is retried, then refused). **News**: one unreadable headline no longer costs the local model its whole batch (verified on the real bundle), a failing batch is a WARNING and a `/status` line, no DeepSeek audit call in a process that does not archive, the event-direction cache is per release. **LLM client**: `chat_json` never raises and never logs the key; the scan's news block is guarded (M15 OOS pair re-proved, Module 37 green). **Panel** (`PANEL_PROMPT_VERSION` v2.2, a new scorecard era): event timing over the cached calendar week, headlines newest first; the replay script uses the logged macro payloads it used to ignore (39 of 110 events). **Tests**: no `DROP COLUMN` (SQLite 3.35), Parquet-writing tests skip without `pyarrow`, the suite passes with both Parquet engines blocked and no GPU. Also fixed on the way: `/status` failure details are Markdown-escaped. **Independent review of these fixes** (a second agent, 20,000-case differential fuzz of the outcome model among its checks): four further LOW defects, fixed the same day — an alert cancelled while its row was being stamped kept the stamp; the panel's calendar read could trigger a 40 s refetch (now `peek_week_events`); a stale model-failure line could stay on `/status`; an HTTP error body was logged without redaction. **Independent verification of the 2026-09-30 direction change**: every figure reproduced, no signal changed by this pass, caveats recorded in section 5.13 of the campaign report. Suite 632 / 40 modules | plan file `~/.claude/plans/you-are-a-quant-velvety-wave.md`; sections 2, 4, 6 and 10 of this file | +| 2026-10-01 | New data sources, report only (owner request). Eight outside series tested against gold's own daily trend: the dollar (EUR/USD), silver, the US 10-year note, the real yield and the broad dollar index (FRED), the gold fund's share count, CFTC managed-money positioning, and Oanda's own client positioning (the position-book endpoint DOES answer on the practice API, with history to 2017; the plan had expected it gone). None shows a forward move of gold beyond gold's own trend (largest t-values 1.2 and 1.8); on the books only one flag, M5 trades against the broad dollar index (−11.2R over 435, losing in both eras), which is the 2025–2026 rally seen from another angle and does nothing on M15: WATCH, a hypothesis. Nothing ships, no feed added, all data stayed in the scratchpad | `backtest/research/new_inputs_2026-10.md` | +| 2026-10-01 | Context screens after the veto (owner request): weekly trend, four-hour trend, daily-trend strength, daily volatility state, place in the 20-day range, tick-volume surge, time to a high-impact US release; rule written first (a state needs 8% of both books and a loss in all four slices to earn a replay). NO state nominated on M15 or M5, no replay, nothing ships. Description: M15 has no losing pocket left in these views; M5 earns only where everything agrees (with the weekly and four-hour trend +0.16R to +0.24R per trade, against them about 0) and near the 20-day extreme in the trade's direction, so a "do not chase" filter would cut the part that pays | `backtest/research/context_screens_2026-10.md` | +| 2026-09-30 | Direction and setup-size campaign (owner request; RISK-FIRST rule approved by the owner, 2019 → 2026 allowed as supporting evidence). **Daily-trend veto SHIPPED on M15** (override 7, `daily_trend_veto` `both`): a finished setup against a clear daily trend (close vs 50-day EMA and vs 60 days earlier, closed D1 bars, Sunday stubs dropped) is removed. PASSED every clause: W3 824 / +205.12R / PF 1.73 / max DD 12.4R vs 1105 / +195.79R / 1.49 / 20.9R; 2019 → 2026 1473 / +223.33R / 1.42 / 41.5R vs 2091 / +192.53R / 1.24 / 64.8R; unseen holdout −4.62R vs −6.63R; both costs; 3 of 4 neighbouring definitions. Accepted cost: W2 first half +61.8R vs +70.1R, old OOS window −4.7R. FAILED: `buy`-only and `sell`-only on M15, every mode on M5 (W3 drawdown 10.1R → 15.2R, long window worse) and on H1 (−9.7R). **Wide-stop cap FAILED** rule C (K = 6 +11.8R on W3 but K = 5 −18.6R on the long window: a peak, not a plateau); knob kept at 0.0. M15 books re-pinned (W2 `dfc647ee…`, OOS `e87b6de1…`), M5 / H1 / H4 pins unchanged, twin MD5-equal throughout, recorder parity re-proved. **Display lines SHIPPED** (books byte-identical): lot size for the owner's risk limit (`trade_size` in config.json, private by default), alerted same-direction plans still open (`signals.alerted_at`, read-only `open_alerted_setups`), the daily trend in words on every surface and in every logged row. Measured on the way: taking every M5 + M15 alert, two entries in three already have a same-direction plan open and up to 13 are open at once; those stacked entries are not worse trades. FIXED: the Oanda provider raised on an empty date window, so `fetch_range` aborted on a pad starting on a closed day. Stops and exits re-screened (25 variants): at a plateau, tighter is far worse. Suite 607 / 40 modules | `backtest/research/daily_trend_campaign_2026-09-30.md`; factory docstring M15 override 7 | | 2026-09-27 | Local news scorer, round two: the owner's ML-NEWS model is now a DeBERTa-v3-small two-head transformer (gold-relevant? then bullish/bearish), distilled from 17,245 DeepSeek gold labels (Sep 8–22 archive + a 2019–2026 Reuters/CNBC/MarketWatch crawl labelled by DeepSeek, 560 calls). Golden set 72/78 (DeepSeek 78, keywords 24, the 09-17 TF-IDF 39), agreement with DeepSeek on unseen live days ~72% (ceiling 90%), 19 ms/headline on one CPU thread as int8-embedding ONNX; passed the four promotion criteria fixed on 09-22. OWNER DECISIONS: it is to score headlines here in mode `primary` with NO DeepSeek for headline scoring (DeepSeek stays for event direction, summaries, the dashboard endpoint); model files live OUTSIDE this tree at a configured path (280 MB, never in git); shadow mode declined (the archive already allows offline replay). IMPLEMENTED the same day (branch `sql-integration`): `news_model.py` (vendored runtime + never-raising singleton, lazy `onnxruntime` / `tokenizers`), `news_model` config section + `NEWS_MODEL_MODE` env, `news_engine._enrich_headline_scores` / `_apply_model_sentiment` (gold-only rules A–F) / `_audit_with_deepseek`, archive provenance for `model` rows, `deploy/README-news-model.md`, Module 38 (15 tests, one against the real bundle: bullish / neutral labels, 7 ms per headline on the dev box). Two further owner decisions during implementation: **`primary` mode stops writing `newsdata.db`** (only the DeepSeek audit sample, shipped 0, is archived) and **both processes load the model**. Books unaffected (news never enters signals); no signal file touched, no parity event. ML-NEWS `news_transformer.py` received the same two `load_model` edits (commit `9496cd9` there) | `docs/news_model_integration_spec_2026-09-27.md`; `~/Development/ML-NEWS/eval/RESULTS_2026-09-27.md` | | 2026-09-27 | Single "conviction" number vs the two-number grade (owner question: one number that is easier to understand and to decide on). Pre-registered; switch `signals.grading_mode` (`pair` default = the shipped rule, byte-identical on both scripts, all five pins reproduced; `min` = min(\|score\|, 0.6 × conf); `product` = \|score\| × conf / 100; thresholds from the control's corner points, no tuning; mirrored in K2, twin MD5-equal to the reference on both new paths). FAILED on M5 and M15: K-MIN moves 43–54 trades for −1.2R / −2.9R (a, b, e / a, b, d); K-PROD −29R on M15 with PF 1.29 and W3 DD 28.9R, neutral on M5. H1 / H4 K-PROD +11.9R / +17.0R on W3 by taking 40–100 extra low-score trades (H1 PF 1.98 → 1.75, DD doubles): a hypothesis, not evidence. Nothing ships; the knob stays research-only. Code review (high) before the runs: 6 findings fixed, 1 recorded. SHIPPED the display half of the question (owner request the same day): alerts lead with the grade and print score / confidence as words (`main.plain_words`, same line on Telegram, `/status`, CLI and the dashboard card); books untouched. Suite 566 | `backtest/research/conviction_campaign_2026-09-27.md` | | 2026-09-26 | Quant review + live log 2026-09-04 → 09-25 (100 setups replayed on the scans' stored candles): stop-hunt claim REFUTED (71% of stop-outs ran ≥ 1R past the stop, same as the books); −25.1R of the −37.8R raw live book were consecutive-bar re-entries the backtester's cooldown never takes; the cooldown view sits at the 3rd–5th percentile of the books' three-week windows. SHIPPED: 5-bar alert cooldown; alert wording (trail first as the tested exit, TP1 a guide with its R, planned RR, validated-book line per class); M15 grade-A SELL block (`sell_min_quality` `A+, B`, PASSED a–f: W2 +154.15R/813 PF 1.53 vs +146.14/931 PF 1.43, OOS +68.5 vs +60.67, W3 DD +8%); 1R printed-target floor in every generator after the gates (`tp_print_floor_rr`, owner rule "TP1 at least 1:1"; label-only re-pin of all four timeframes). FAILED: M15 `min_rr_ratio` 1.0 (trade-identical, −2.0R); a pre-gate target floor (−24.6R). OWNER'S CALL: M5 `tp_floor_rr` 1.0 adds +48.9R on W2 with both halves up but max DD 10.1 → 15.9R (ladder 0.6–1.0 non-monotone); not shipped. Telegram / dashboard AI block rewritten for a new trader (owner rule): one plain headline against the usual odds, the chance "in 100", one reason for and one against, no verdict word or confidence figure; persona prompts ban trading jargon (`PANEL_PROMPT_VERSION` v2.1). Two delivery bugs from the DB checks fixed: the macro-source panel verdict never stored its probability (all v2 rows NULL); the level-breach rescan re-logged the 20:45 M15 bar three times every evening after the settlement pause (guard added to `_handle_breach`). Indicator grid and volume / daily-direction screens (Part C) not run | `backtest/research/rr_and_sell_campaign_2026-09-26.md` | diff --git a/advisory_outcome.py b/advisory_outcome.py index ac30f70..9e14b21 100644 --- a/advisory_outcome.py +++ b/advisory_outcome.py @@ -176,10 +176,31 @@ def _valid_bars(bars: Iterable[Dict[str, Any]]) -> List[Dict[str, Any]]: return valid +def _next_bar_follows( + previous: datetime, stamp: datetime, step: timedelta +) -> bool: + """True when ``stamp`` is the next tradable bar after ``previous``: every + bar slot between the two lies inside a scheduled XAU closure (the weekend + or the daily 17:00 New York maintenance hour) and ``stamp`` is on the bar + grid. ``step`` must be positive.""" + expected = previous + step + while expected < stamp: + ny = expected.astimezone(_NY_TZ) + daily_maintenance = ny.weekday() < 4 and ny.hour == 17 + if not (_is_forex_weekend(expected) or daily_maintenance): + return False + expected += step + return expected == stamp + + def _has_continuity( bars: List[Dict[str, Any]], last: datetime, timeframe: str ) -> bool: - """Reject missing tradable bars while allowing scheduled XAU closures.""" + """Reject missing tradable bars while allowing scheduled XAU closures. + + All-or-nothing over the whole of ``bars`` (the scan-outcome labeller + calls it one bar at a time). advance_outcome does not use it: it checks + each step as it walks, so a hole AFTER a trade's exit cannot void it.""" seconds = _TF_SECONDS.get(normalize_timeframe(timeframe)) if seconds is None: return False @@ -190,14 +211,7 @@ def _has_continuity( stamp = bar["_timestamp"] if stamp <= previous or stamp in seen: continue - expected = previous + step - while expected < stamp: - ny = expected.astimezone(_NY_TZ) - daily_maintenance = ny.weekday() < 4 and ny.hour == 17 - if not (_is_forex_weekend(expected) or daily_maintenance): - return False - expected += step - if expected != stamp: + if not _next_bar_follows(previous, stamp, step): return False previous = stamp seen.add(stamp) @@ -256,18 +270,36 @@ def advance_outcome( state["result"] = None state["reason"] = "stored candle snapshots lost continuity" return state - if has_future and not _has_continuity(valid_bars, last, - state["timeframe"]): + if not has_future: + return state + seconds = _TF_SECONDS.get(normalize_timeframe(state["timeframe"])) + if seconds is None: state["status"] = "UNRESOLVABLE" state["result"] = None state["reason"] = "stored candle snapshots contain an internal gap" return state + step = timedelta(seconds=seconds) for bar in valid_bars: stamp = bar["_timestamp"] if stamp <= signal_bar or stamp <= last: continue + # Continuity is judged ONE STEP AT A TIME, only as far as the + # trade is still open. Until 2026-10-01 the whole stream was + # checked before the walk, so a trade that had exited days + # before a later hole (a holiday, an outage) was thrown away: + # 9 of 111 setups in the September log, 7 of them with an exit + # the live resolver had recorded. A hole reached while the + # trade is open still ends the walk; the progress made up to + # the bar before it stays, so resolve_locked_floor floors from + # the latest stop. + if not _next_bar_follows(last, stamp, step): + state["status"] = "UNRESOLVABLE" + state["result"] = None + state["reason"] = "stored candle snapshots contain an internal gap" + return state + if progress["entry_price"] is None: entry = bar["open"] initial_stop = entry - risk if buy else entry + risk diff --git a/agent_panel.py b/agent_panel.py index 36fbc7c..f547560 100644 --- a/agent_panel.py +++ b/agent_panel.py @@ -44,6 +44,10 @@ stop/target in ATR, the setup class's base rate from the two-year book, event timing against the expected hold) so the model judges facts instead of narrative; ``PANEL_PROMPT_VERSION`` is logged with every verdict and is part of the payload hash so prompt eras never mix. +Since v2.2 (2026-10-01) the event timing reads the cached calendar week +(a hold that runs past UTC midnight sees tomorrow's release) and the +headline list is cut newest first; both change what the persona reads, +hence the new era. Two invariants, both load-bearing (CLAUDE.md rules 1 and 4): @@ -126,7 +130,10 @@ VERDICT_SOURCES = ("mean", "macro", "chair") #: Logged with every verdict and folded into the payload hash. Bump on #: ANY prompt or context-schema change so the scorecard can split eras #: (analyze_signal_log splits by panel_prompt_version). -PANEL_PROMPT_VERSION = "v2.1-2026-09-26" # v2.1: beginner wording rules in _TAKE_CONTRACT +# v2.1: beginner wording rules in _TAKE_CONTRACT. v2.2: no prompt text +# changed, the persona's INPUT did (event timing from the cached calendar +# week, headlines newest first), so the scorecard starts a new era. +PANEL_PROMPT_VERSION = "v2.2-2026-10-01" #: Symmetric probability bands (percent). A persona's stance and the #: mean verdict are DERIVED from p_win with the same thresholds, so @@ -606,13 +613,31 @@ def _headline_gold_score(headline: Any) -> Optional[float]: return max(-1.0, min(1.0, score)) if math.isfinite(score) else None +def _published_key(headline: Any) -> float: + """Sort key for "newest first": the publish time as epoch seconds, + -inf when there is none. Never raises.""" + stamp = getattr(headline, "timestamp", None) + try: + if isinstance(stamp, str): + stamp = _dt.datetime.fromisoformat(stamp.replace(" ", "T")) + value = float(stamp.timestamp()) + return value if math.isfinite(value) else float("-inf") + except Exception: + return float("-inf") + + def _directional_news_payload( context: Optional["FundamentalContext"], limit: int = 10 ) -> List[Dict[str, Any]]: if context is None: return [] out: List[Dict[str, Any]] = [] - for headline in context.news_headlines: + # Newest first, THEN the cut: the context lists headlines feed by + # feed, so the first `limit` were whatever the first feed carried and + # a newer story from a later feed never reached the persona. + ordered = sorted(context.news_headlines or [], key=_published_key, + reverse=True) + for headline in ordered: score = _headline_gold_score(headline) if score is None or abs(score) < 0.1: continue @@ -906,9 +931,14 @@ def _primary_frame(report: "SignalReport", frames: Any) -> Any: def _price_context(report: "SignalReport", frames: Any = None, - context: Optional["FundamentalContext"] = None + context: Optional["FundamentalContext"] = None, + week_events: Optional[List[Any]] = None ) -> Dict[str, Any]: - """The shared "context" block (see module docstring). {} on failure.""" + """The shared "context" block (see module docstring). {} on failure. + + ``week_events`` is the cached calendar week (attach_panel hands it + in); it is never fetched here, because the offline replay calls this + function with an as-of context and must not see today's calendar.""" out: Dict[str, Any] = {} try: direction = _direction(report) @@ -992,10 +1022,16 @@ def _price_context(report: "SignalReport", frames: Any = None, hold_bars = (out.get("base_rate") or {}).get("median_bars_held") if hold_bars and secs: out["expected_hold_minutes"] = int(hold_bars * secs // 60) - if context is not None: + # The context alone knows today's events (UTC date) and the next + # hour: an H1 or H4 hold running past midnight into tomorrow's + # release read as event-free. The calendar week covers it. + pool = list(week_events or []) + if not pool and context is not None: + pool = list(getattr(context, "events_today", None) or []) + \ + list(getattr(context, "events_upcoming_1h", None) or []) + if pool: minutes = [] - for ev in list(getattr(context, "events_today", None) or []) + \ - list(getattr(context, "events_upcoming_1h", None) or []): + for ev in pool: try: if ev.is_high_impact and ev.is_upcoming: minutes.append(float(ev.minutes_until)) @@ -1466,21 +1502,35 @@ def attach_panel(report: "SignalReport", if direction is None: return False + week_events: Optional[List[Any]] = None if context is None: # Best-effort; the news engine's class-level 600 s caches make # this nearly free when warm in both bot and dashboard. try: from config import PlatformConfig from news_engine import NewsEngine - context = NewsEngine( - PlatformConfig().news).get_fundamental_context() + engine = NewsEngine(PlatformConfig().news) + context = engine.get_fundamental_context() except Exception as e: logger.debug("agent panel: news context unavailable: %s", e) context = None + else: + # The calendar week for the event-timing check, read from + # the class-level cache the context was just built from. + # peek, not get: get_week_events refreshes an expired + # cache, and with the calendar sources down that is a 40 s + # wait under the news lock while this alert's lock is held. + # Only on this path — a caller that passes its own context + # (the replay) passes its own time. + try: + week_events = engine.peek_week_events() + except Exception as e: + logger.debug("agent panel: calendar week unavailable: %s", e) personas = normalize_personas(settings.get("personas", DEFAULT_PERSONAS)) candles = _candle_context(report, settings) - ctx = _price_context(report, getattr(report, "_frames", None), context) + ctx = _price_context(report, getattr(report, "_frames", None), context, + week_events) jobs = _panel_jobs(report, context, candles, ctx, personas) payload_hash = _panel_payload_hash(jobs) reuse_hash = _panel_payload_hash(jobs, ignore_bar_timestamp=True) diff --git a/agent_panel_base_rates.json b/agent_panel_base_rates.json index 96e25c6..0344613 100644 --- a/agent_panel_base_rates.json +++ b/agent_panel_base_rates.json @@ -1,6 +1,6 @@ { "schema": "agent-panel-base-rates-v1", - "generated_at": "2026-09-27T00:45:18+00:00", + "generated_at": "2026-10-01T05:49:09+00:00", "key": "timeframe|strategy|grade|direction ('*' = any)", "min_n_for_exact_class": 30, "books": { @@ -13,8 +13,8 @@ }, "15min": { "file": "m15_w2.csv", - "md5": "5b65ecf6fad9a8da412749066fd50d36", - "trades": 813, + "md5": "dfc647ee02afd179dd9e031f344ed380", + "trades": 597, "first_entry": "2024-09-05 06:15:00+00:00", "last_entry": "2026-09-04 08:00:00+00:00" }, @@ -35,27 +35,27 @@ }, "classes": { "15min|*|*|*": { - "n": 813, - "win_rate": 62.24, - "mean_r": 0.1896, - "tp1_reach_rate": 50.43, - "median_bars_held": 32, + "n": 597, + "win_rate": 64.32, + "mean_r": 0.2611, + "tp1_reach_rate": 53.6, + "median_bars_held": 34, "book": "m15_w2.csv" }, "15min|*|*|BUY": { - "n": 635, - "win_rate": 63.31, - "mean_r": 0.2309, - "tp1_reach_rate": 51.65, - "median_bars_held": 33, + "n": 540, + "win_rate": 64.44, + "mean_r": 0.2725, + "tp1_reach_rate": 53.52, + "median_bars_held": 35, "book": "m15_w2.csv" }, "15min|*|*|SELL": { - "n": 178, - "win_rate": 58.43, - "mean_r": 0.0424, - "tp1_reach_rate": 46.07, - "median_bars_held": 28, + "n": 57, + "win_rate": 63.16, + "mean_r": 0.1536, + "tp1_reach_rate": 54.39, + "median_bars_held": 27, "book": "m15_w2.csv" }, "15min|GENERIC|A+|BUY": { @@ -66,68 +66,60 @@ "median_bars_held": 32, "book": "m15_w2.csv" }, - "15min|GENERIC|A+|SELL": { - "n": 1, - "win_rate": 0.0, - "mean_r": -1.0276, - "tp1_reach_rate": 0.0, - "median_bars_held": 34, - "book": "m15_w2.csv" - }, "15min|GENERIC|A|BUY": { - "n": 208, - "win_rate": 67.31, - "mean_r": 0.3158, - "tp1_reach_rate": 59.13, - "median_bars_held": 44, + "n": 191, + "win_rate": 67.02, + "mean_r": 0.3152, + "tp1_reach_rate": 59.16, + "median_bars_held": 45, "book": "m15_w2.csv" }, "15min|GENERIC|B|BUY": { - "n": 282, - "win_rate": 62.41, - "mean_r": 0.1869, - "tp1_reach_rate": 46.1, + "n": 238, + "win_rate": 62.18, + "mean_r": 0.2096, + "tp1_reach_rate": 47.06, "median_bars_held": 38, "book": "m15_w2.csv" }, "15min|GENERIC|B|SELL": { - "n": 161, - "win_rate": 58.39, - "mean_r": 0.0273, - "tp1_reach_rate": 45.34, - "median_bars_held": 32, + "n": 52, + "win_rate": 63.46, + "mean_r": 0.1569, + "tp1_reach_rate": 53.85, + "median_bars_held": 28, "book": "m15_w2.csv" }, "15min|GENERIC|C|BUY": { - "n": 60, - "win_rate": 65.0, - "mean_r": 0.395, - "tp1_reach_rate": 60.0, - "median_bars_held": 24, + "n": 49, + "win_rate": 71.43, + "mean_r": 0.5518, + "tp1_reach_rate": 67.35, + "median_bars_held": 27, "book": "m15_w2.csv" }, "15min|XAU_ATR|A+|BUY": { - "n": 3, - "win_rate": 66.67, - "mean_r": 0.2684, - "tp1_reach_rate": 33.33, - "median_bars_held": 13, + "n": 2, + "win_rate": 50.0, + "mean_r": -0.4203, + "tp1_reach_rate": 0.0, + "median_bars_held": 12, "book": "m15_w2.csv" }, "15min|XAU_ATR|A+|SELL": { - "n": 16, - "win_rate": 62.5, - "mean_r": 0.2612, - "tp1_reach_rate": 56.25, - "median_bars_held": 5, + "n": 5, + "win_rate": 60.0, + "mean_r": 0.1192, + "tp1_reach_rate": 60.0, + "median_bars_held": 6, "book": "m15_w2.csv" }, "15min|XAU_ATR|A|BUY": { - "n": 80, - "win_rate": 56.25, - "mean_r": 0.06, - "tp1_reach_rate": 47.5, - "median_bars_held": 10, + "n": 58, + "win_rate": 62.07, + "mean_r": 0.2058, + "tp1_reach_rate": 53.45, + "median_bars_held": 11, "book": "m15_w2.csv" }, "1h|*|*|*": { diff --git a/app_config.py b/app_config.py index 55a916e..03f78eb 100644 --- a/app_config.py +++ b/app_config.py @@ -25,6 +25,7 @@ bot, CLI and backtester all see the same overlay (live == backtest). import json import logging +import math import os from dataclasses import dataclass, field from typing import Any, Dict, List, Optional @@ -90,7 +91,8 @@ _KNOWN_TOP = {"telegram", "data_providers", "scanning", "pairs", "log_level", # config.json files still carry the block and must keep # loading; load_app_config logs one warning when it is there. "market_outlook", - "violent_watch", "signal_log", "news_archive", "news_model"} + "violent_watch", "signal_log", "news_archive", "news_model", + "trade_size"} _KNOWN_OANDA = { # "account_id" is TOLERATED but ignored: it belonged to the auto- # execution daemon removed 2026-09-03 (last commit 31002cb). The candle @@ -113,6 +115,19 @@ _KNOWN_SIGNAL_LOG = {"enabled", "path", "log_blocked_scans"} _KNOWN_NEWS_ARCHIVE = {"enabled", "path"} _KNOWN_NEWS_MODEL = {"path", "threads", "mode", "deepseek_audit_per_day"} _NEWS_MODEL_MODES = ("off", "fallback", "primary") +# Lot-size guidance on the alert (exposure.py, display only). Off until +# the owner sets risk_per_trade. The size reveals the owner's risk per +# trade, so it is printed in the owner's own chat (telegram.chat_id) and +# the CLI only; telegram_channel switches it on for every other Telegram +# chat (the subscriber channel, a group), dashboard for the public page. +_KNOWN_TRADE_SIZE = {"risk_per_trade", "contract_size", "lot_step", + "telegram_channel", "dashboard"} +_TRADE_SIZE_DEFAULTS = { + "contract_size": 100.0, # ounces per 1.00 lot of XAU/USD + "lot_step": 0.01, # smallest size increment the broker accepts + "telegram_channel": False, + "dashboard": False, +} _KNOWN_TELEGRAM = {"token", "chat_id", "channel_id", "alert_min_quality"} _KNOWN_SCANNING = {"pairs", "timeframes"} _KNOWN_PAIR_KEYS = {"blocked_hours_utc"} @@ -153,6 +168,7 @@ class AppConfig: signal_log_settings: Dict[str, Any] = field(default_factory=dict) news_archive_settings: Dict[str, Any] = field(default_factory=dict) news_model_settings: Dict[str, Any] = field(default_factory=dict) + trade_size_settings: Dict[str, Any] = field(default_factory=dict) log_level: str = "INFO" @@ -432,6 +448,27 @@ def load_app_config(path: str = CONFIG_PATH) -> AppConfig: "is not off") app.news_model_settings = dict(nm) + size = raw.get("trade_size", {}) + _check_keys(size, _KNOWN_TRADE_SIZE, '"trade_size"') + for key in ("risk_per_trade", "contract_size", "lot_step"): + if key in size: + v = size[key] + if (isinstance(v, bool) or not isinstance(v, (int, float)) + or not math.isfinite(v)): + raise AppConfigError( + f"config.json: trade_size.{key} must be a number") + # risk_per_trade 0 = the size line is off (the default) + if v < 0 or (v == 0 and key != "risk_per_trade"): + raise AppConfigError( + f"config.json: trade_size.{key} must be " + + ("zero (off) or a positive number" + if key == "risk_per_trade" else "a positive number")) + for flag in ("telegram_channel", "dashboard"): + if flag in size and not isinstance(size[flag], bool): + raise AppConfigError( + f"config.json: trade_size.{flag} must be true or false") + app.trade_size_settings = dict(size) + level = raw.get("log_level", app.log_level) if level not in _VALID_LOG_LEVELS: raise AppConfigError( @@ -685,6 +722,21 @@ def news_archive_path(app: Optional[AppConfig] = None) -> str: return path +# ── Lot-size guidance (config.json only; display) ──────────────────── + +def trade_size_settings(app: Optional[AppConfig] = None) -> Optional[Dict[str, Any]]: + """The lot-size guidance settings with their defaults filled in, or + None when the owner has not set `trade_size.risk_per_trade` (the size + line is then off on every surface).""" + app = app or get_app_config() + block = app.trade_size_settings or {} + if not block.get("risk_per_trade"): + return None + out = dict(_TRADE_SIZE_DEFAULTS) + out.update(block) + return out + + # ── PlatformConfig overlay ──────────────────────────────────────────── def apply_pair_overrides( diff --git a/backtest/cudaBasedBacktesting.py b/backtest/cudaBasedBacktesting.py index f8bf75b..8627818 100644 --- a/backtest/cudaBasedBacktesting.py +++ b/backtest/cudaBasedBacktesting.py @@ -45,7 +45,9 @@ if str(PROJECT_ROOT) not in sys.path: from config import PAIR_INFO, PlatformConfig, Timeframe # noqa: E402 from providers import make_data_provider # noqa: E402 from providers.base import BaseDataProvider # noqa: E402 -from signal_aggregator import SignalAggregator # noqa: E402 (fingerprints + the two static cleanups only) +from signal_aggregator import ( # noqa: E402 (fingerprints, the two static cleanups, the daily-trend definition) + SignalAggregator, daily_trend_needed, daily_trend_series, +) import position_sizing # noqa: E402 from backtest import gpu_kernels as _gk # noqa: E402 from backtest import cuda_signal_kernels as ck # noqa: E402 @@ -318,6 +320,12 @@ def format_warning(code: int, arg: float, ctx: Dict) -> str: if n == "rf_blocked": return (f"REGIME FILTER: {ctx['rf_tf']} ADX={arg:.1f} < " f"{ctx['regime_filter_min_adx']:.0f} — market choppy, skipping signal") + if n == "trend_veto": + is_buy = arg > 0 + return (f"DAILY TREND FILTER: {'BUY' if is_buy else 'SELL'} against a " + f"{'falling' if is_buy else 'rising'} daily trend — no trade") + if n == "trend_unavailable": + return "Daily trend filter requested but 1d data unavailable" raise ValueError(f"unknown warning code {code}") @@ -379,6 +387,34 @@ def host_flags(primary_index: pd.DatetimeIndex, platform_cfg: PlatformConfig, } +def trend_flags(primary_index: pd.DatetimeIndex, d1: Optional[pd.DataFrame], + bt_cfg: BacktestConfig) -> np.ndarray: + """Per primary bar: the daily trend the live code reads on the D1 slice + _replay_span hands it (+1 / 0 / -1), or ck.DAILY_TREND_UNAVAILABLE when + that slice holds no usable bar. + + The definition is signal_aggregator.daily_trend_series, evaluated once on + the whole frame (every step is causal, so the value at a bar equals the + value on any prefix ending there). The as-of rule is the replay's: a D1 + bar is visible once label + 1 day <= the primary bar's close, compared in + int64 nanoseconds; the Sunday stubs daily_trend_series drops are skipped + by searching its own (kept-bar) index.""" + n = len(primary_index) + out = np.full(n, ck.DAILY_TREND_UNAVAILABLE, dtype=np.int8) + if d1 is None or len(d1) == 0: + return out + series = daily_trend_series(d1) + if len(series) == 0: + return out + primary_ns = primary_index.as_unit("ns").asi8 + cutoff = primary_ns + (BAR_SECONDS[bt_cfg.primary_tf] - BAR_SECONDS[Timeframe.D1]) * 1_000_000_000 + labels_ns = series.index.as_unit("ns").asi8 + k = np.searchsorted(labels_ns, cutoff, side="right") + seen = k > 0 + out[seen] = series.to_numpy()[k[seen] - 1] + return out + + def replay_bounds(primary_df: pd.DataFrame, bt_cfg: BacktestConfig) -> Tuple[int, int]: bt_start_ts = pd.Timestamp(bt_cfg.start) bt_end_ts = pd.Timestamp(bt_cfg.end) @@ -421,6 +457,8 @@ def refuse_unsupported(platform_cfg: PlatformConfig, bt_cfg: BacktestConfig) -> problems.append(f"technical.ema_mode={t.ema_mode!r}") if platform_cfg.signals.grading_mode not in ck.GRADING_MODE_CODES: problems.append(f"signals.grading_mode={platform_cfg.signals.grading_mode!r}") + if x.daily_trend_veto not in ck.DAILY_TREND_CODES: + problems.append(f"xau_strategy.daily_trend_veto={x.daily_trend_veto!r}") if problems: raise SystemExit("cudaBasedBacktesting.py cannot run this configuration: " + "; ".join(problems) + ". Use backtest/gpu_backtest.py.") @@ -456,6 +494,11 @@ def analyze_frames(bt_cfg: BacktestConfig, platform_cfg: PlatformConfig, rf_cut = cuts[rtf] if rtf != bt_cfg.primary_tf else cp.arange(1, primary_dev["n"] + 1, dtype=cp.int64) rf = (dev[rtf]["adx"], rf_cut) flags = host_flags(primary_df.index, platform_cfg, bt_cfg) + # The daily trend per primary bar (read by K2 only when the veto is on). + flags["daily_trend"] = ( + trend_flags(primary_df.index, dfs_full.get(Timeframe.D1), bt_cfg) + if daily_trend_needed(platform_cfg) + else np.zeros(len(primary_df), dtype=np.int8)) first_eligible_pos, last_pos = replay_bounds(primary_df, bt_cfg) logger.info(f"Replaying {bt_cfg.pair} {bt_cfg.primary_tf.value}: " f"bars {first_eligible_pos}–{last_pos} of {len(primary_df)}") @@ -508,6 +551,9 @@ def run_backtest_cuda(bt_cfg: BacktestConfig, platform_cfg: PlatformConfig) -> T flt = platform_cfg.backtest if flt.regime_filter and flt.regime_filter_timeframe not in timeframes: timeframes.append(flt.regime_filter_timeframe) + # the D1 frame of the daily-trend veto, appended exactly like the reference script + if daily_trend_needed(platform_cfg) and Timeframe.D1 not in timeframes: + timeframes.append(Timeframe.D1) dfs_full: Dict[Timeframe, pd.DataFrame] = {} cache_names: Dict[str, str] = {} tf_starts = {} @@ -615,27 +661,30 @@ def _warning_ctx(ctx, pos: int, r2i: Dict, r2f: Dict) -> Dict: _RF_CODES = {ck.WARN_CODES["rf_unavailable"], ck.WARN_CODES["rf_no_adx"], ck.WARN_CODES["rf_blocked"]} +_TREND_CODES = {ck.WARN_CODES["trend_veto"], ck.WARN_CODES["trend_unavailable"]} def build_warnings(ctx, pos: int, r2i: Dict, r2f: Dict, codes, args) -> Tuple[List[str], List[str]]: """(warnings, cleaned_warnings) of one analysed bar, after the XAU_CP cleanup (when a pattern setup was accepted) and the stale spread-gate - step (when a setup existed at step 11), with the regime-filter lines - appended last like the replay does.""" + step (when a setup existed at step 11), with the daily-trend line of + step 12 and then the regime-filter lines appended last, in the order + the live code and the replay write them.""" wctx = _warning_ctx(ctx, pos, r2i, r2f) n = r2i["warn_n"] main: List[str] = [] + trend: List[str] = [] rf: List[str] = [] for q in range(n): code = int(codes[q]) text = format_warning(code, float(args[q]), wctx) - (rf if code in _RF_CODES else main).append(text) + (rf if code in _RF_CODES else trend if code in _TREND_CODES else main).append(text) ns = types.SimpleNamespace(warnings=main, cleaned_warnings=[], trade_setup=object() if r2i["strategy_pre_rf"] != 0 else None) if r2i["cp_accepted"]: SignalAggregator._clean_accepted_cp_warnings(ns) SignalAggregator._drop_stale_spread_gate_vetoes(ns) - return list(ns.warnings) + rf, list(ns.cleaned_warnings) + return list(ns.warnings) + trend + rf, list(ns.cleaned_warnings) def build_reasons(ctx, r2i: Dict, r2f: Dict, inds: Dict[str, Dict]) -> Tuple[List[str], List[str]]: diff --git a/backtest/cuda_signal_kernels.py b/backtest/cuda_signal_kernels.py index c4f0c8b..a408016 100644 --- a/backtest/cuda_signal_kernels.py +++ b/backtest/cuda_signal_kernels.py @@ -12,9 +12,13 @@ the repo and stays the reference: thread analyses the prefix [0..j]. K2 fb2_aggregate signal_aggregator.analyze_from_dataframes end to end (MTF merge, penalties, grade, gates 6b-6i, the three - setup generators, XAU_PB, XAU_CP, the stale-veto step) + setup generators with the wide-stop cap, XAU_PB, + XAU_CP, the stale-veto step, the daily-trend veto) plus the per-bar rules of gpu_backtest._replay_span. - One thread per primary bar. + One thread per primary bar. The daily trend itself + arrives as a host flag (cudaBasedBacktesting. + trend_flags): its pandas definition lives in + signal_aggregator.daily_trend_series only. K3 fb2_features position_sizing.build_features (65 features), one thread per accepted trade (--export-features). K4 fb2_cooldown gpu_backtest._apply_cooldown (single thread). @@ -155,6 +159,8 @@ WARN_NAMES = ( "rf_unavailable", # 33 regime filter: TF missing "rf_no_adx", # 34 regime filter: ADX None "rf_blocked", # 35 arg adx + "trend_veto", # 36 daily-trend veto: arg setup direction (+1 / -1) + "trend_unavailable", # 37 daily-trend veto on, no usable D1 bar ) WARN_CODES = {n: i for i, n in enumerate(WARN_NAMES)} @@ -206,6 +212,13 @@ SKIP_NAMES = {1: "WEEKEND_FILTER", 2: "DAY_FILTER", 3: "SESSION_WINDOW"} # 1 = min conviction, 2 = product conviction). Pinned by name, never by the # position in signal_aggregator.GRADING_MODES; pack_params checks the two agree. GRADING_MODE_CODES = {"pair": 0, "min": 1, "product": 2} +# xau_strategy.daily_trend_veto (signal_aggregator.DAILY_TREND_MODES), pinned +# by name like the grading modes; pack_params checks the two agree. The +# per-bar trend itself is computed on the host (cudaBasedBacktesting. +# trend_flags, from signal_aggregator.daily_trend_series): +1 / 0 / -1, and +# DAILY_TREND_UNAVAILABLE where the D1 slice holds no usable bar. +DAILY_TREND_CODES = {"off": 0, "both": 1, "buy": 2, "sell": 3} +DAILY_TREND_UNAVAILABLE = 2 # ── Config packing ──────────────────────────────────────────────────── PARAMS_F = ( @@ -228,6 +241,7 @@ PARAMS_F = ( # all 0.0 under "pair"): aplus, a, b, c, strong, buy, boost_a, boost_b "conv_aplus", "conv_a", "conv_b", "conv_c", "conv_strong", "conv_buy", "conv_boost_a", "conv_boost_b", + "wide_stop_cap_atr", # 0.0 = off (GENERIC and XAU_CP stop cap, after the gates) ) PARAMS_I = ( "ema_trend_period", "ema_mode_2ema", "ema_partial_neutral", "rsi_regime_aware", @@ -245,6 +259,7 @@ PARAMS_I = ( "quality_filter_mask", "strategies_mask", "primary_is_m5", "primary_bar_secs", "regime_filter", "n_confirm", "grading_mode", # 0 pair (shipped) / 1 min / 2 product + "daily_trend_veto", # 0 off / 1 both / 2 buy / 3 sell (DAILY_TREND_CODES) ) # Fields of the four config dataclasses the kernels do NOT read, with the @@ -377,6 +392,12 @@ def pack_params(platform_cfg, bt_cfg) -> Tuple[np.ndarray, np.ndarray, np.ndarra for name, val in zip(("conv_aplus", "conv_a", "conv_b", "conv_c", "conv_strong", "conv_buy", "conv_boost_a", "conv_boost_b"), conv_t): f[name] = val + f["wide_stop_cap_atr"] = x.wide_stop_cap_atr + # daily-trend veto: the live module owns the mode list (same local import) + from signal_aggregator import DAILY_TREND_MODES, daily_trend_mode_of + assert set(DAILY_TREND_MODES) == set(DAILY_TREND_CODES), ( + f"signal_aggregator.DAILY_TREND_MODES {DAILY_TREND_MODES} != the kernel's {sorted(DAILY_TREND_CODES)}") + daily_trend_veto = DAILY_TREND_CODES[daily_trend_mode_of(x)] from providers.base import BaseDataProvider # CANDLE_SECONDS (the aggregator's clock) bar_secs = BaseDataProvider.CANDLE_SECONDS[platform_cfg.primary_timeframe] strategies = list(bt_cfg.strategies_only or []) @@ -431,6 +452,7 @@ def pack_params(platform_cfg, bt_cfg) -> Tuple[np.ndarray, np.ndarray, np.ndarra "regime_filter": 1 if b.regime_filter else 0, "n_confirm": len(platform_cfg.confirmation_timeframes), "grading_mode": GRADING_MODE_CODES[grading_mode], + "daily_trend_veto": daily_trend_veto, } pf = np.array([float(f[name]) for name in PARAMS_F], dtype=np.float64) pi = np.array([int(i[name]) for name in PARAMS_I], dtype=np.int32) @@ -480,6 +502,9 @@ def _defines() -> str: lines.append(f"#define Q_{name.replace('+', 'PLUS')} {code}") for name, code in STRATEGY_CODES.items(): lines.append(f"#define STRAT_{name or 'NONE'} {code}") + for name, code in DAILY_TREND_CODES.items(): + lines.append(f"#define TREND_{name} {code}") + lines.append(f"#define FB2_TREND_UNAVAILABLE {DAILY_TREND_UNAVAILABLE}") return "\n".join(lines) + "\n" @@ -1512,6 +1537,7 @@ void fb2_aggregate(const double* __restrict__ open_p, const double* __restrict__ const double* __restrict__ rf_adx, const long long* __restrict__ rf_cut, const signed char* __restrict__ skip_reason, const signed char* __restrict__ label_hour, const signed char* __restrict__ label_wd, const signed char* __restrict__ close_hour, + const signed char* __restrict__ daily_trend_p, const double* __restrict__ PF, const int* __restrict__ PI, const int first_pos, const int n_bars, double* __restrict__ OF2, int* __restrict__ OI2, @@ -1933,6 +1959,16 @@ void fb2_aggregate(const double* __restrict__ open_p, const double* __restrict__ if (!(rr < min_rr - 1e-9)) { if (tp1_distance < min_tp) { WARN(W_spread_gate_generic, tp1_distance); } else { + // wide-stop cap AFTER the gates (signal_aggregator._wide_stop_cap, 0 = off) + if (PF[PF_wide_stop_cap_atr] > 0.0 && xau_enabled && is_xau) { + const double stop_floor = PYMAX(atr * sl_atr_floor, min_sl_price); + const double cap_dist = PYMAX(PF[PF_wide_stop_cap_atr] * atr, stop_floor); + if (sl_distance > cap_dist) { + sl_distance = cap_dist; + sl_price = is_buy ? (entry - cap_dist) : (entry + cap_dist); + rr = tp1_distance / sl_distance; + } + } { // printed-target floor AFTER the gates (signal_aggregator 2026-09-26) const double floor_dist = sl_distance * PF[PF_tp_print_floor_rr]; if (tp1_distance < floor_dist) { @@ -2079,6 +2115,15 @@ void fb2_aggregate(const double* __restrict__ open_p, const double* __restrict__ } double rr = (sl_dist > 0.0) ? height / sl_dist : 0.0; if (!(rr < min_rr - 1e-9) && !(height < min_tp)) { + // wide-stop cap AFTER the gates (signal_aggregator._wide_stop_cap, 0 = off) + if (PF[PF_wide_stop_cap_atr] > 0.0 && xau_enabled && is_xau) { + const double cap_dist = PYMAX(PF[PF_wide_stop_cap_atr] * atr, min_sl_price); + if (sl_dist > cap_dist) { + sl_dist = cap_dist; + sl_ = is_buy ? (entry - cap_dist) : (entry + cap_dist); + rr = height / sl_dist; + } + } { // printed-target floor AFTER the gates (signal_aggregator 2026-09-26) double tp1_dist = height; const double floor_dist = sl_dist * PF[PF_tp_print_floor_rr]; @@ -2103,7 +2148,22 @@ void fb2_aggregate(const double* __restrict__ open_p, const double* __restrict__ } // ── replay: the optional higher-TF regime filter (after analyze_from_dataframes) ── + // strategy_pre_rf = the setup at step 11 (the stale spread-gate cleanup on + // the host), i.e. BEFORE the daily-trend veto of step 12 and the filter. K2I(strategy_pre_rf) = strategy; + + // ── 12. daily-trend veto (signal_aggregator._apply_daily_trend_veto) ── + if (PI[PI_daily_trend_veto] != 0 && xau_enabled && is_xau) { + const int td = daily_trend_p[pos]; + if (td == FB2_TREND_UNAVAILABLE) { WARN(W_trend_unavailable, 0.0); } + else if (strategy != STRAT_NONE && td != 0) { + const int mode = PI[PI_daily_trend_veto]; + const bool against = (direction > 0 && td < 0 && (mode == TREND_both || mode == TREND_buy)) + || (direction < 0 && td > 0 && (mode == TREND_both || mode == TREND_sell)); + if (against) { WARN(W_trend_veto, (double)direction); quality = Q_D; strategy = STRAT_NONE; } + } + } + if (PI[PI_regime_filter]) { const long long cut = rf_cut[pos]; if (cut == 0) { K2I(rf_state) = 1; WARN(W_rf_unavailable, 0.0); } @@ -2780,6 +2840,7 @@ def aggregate(primary_dev: Dict, primary_an: TFAnalysis, rf_adx, rf_cut, _dev(flags["skip_reason"], np.int8), _dev(flags["label_hour"], np.int8), _dev(flags["label_wd"], np.int8), _dev(flags["close_hour"], np.int8), + _dev(flags["daily_trend"], np.int8), _dev(pf, np.float64), _dev(pi, np.int32), np.int32(first_pos), np.int32(n_bars), OF2, OI2, warn_code, warn_arg)) cp.cuda.Device().synchronize() diff --git a/backtest/gpu_backtest.py b/backtest/gpu_backtest.py index c108f07..18f9e9a 100644 --- a/backtest/gpu_backtest.py +++ b/backtest/gpu_backtest.py @@ -63,7 +63,9 @@ from config import ( # noqa: E402 ) from providers import BaseDataProvider, make_data_provider # noqa: E402 from engine import IndicatorSignal, TechnicalAnalysis, TechnicalEngine # noqa: E402 -from signal_aggregator import SignalAggregator, SignalReport, TradeSetup # noqa: E402 +from signal_aggregator import ( # noqa: E402 + SignalAggregator, SignalReport, TradeSetup, daily_trend_needed, +) import position_sizing # noqa: E402 from backtest import replay_record # noqa: E402 from backtest import gpu_kernels as _gk # noqa: E402 @@ -2084,6 +2086,11 @@ def run_backtest( flt = platform_cfg.backtest if flt.regime_filter and flt.regime_filter_timeframe not in timeframes: timeframes.append(flt.regime_filter_timeframe) + # Same for the daily-trend veto: analyze_pair fetches D1 as a required + # frame when it is on, and the core reads it from the slices (the + # as-of slicing in _replay_span hands it closed daily bars only). + if daily_trend_needed(platform_cfg) and Timeframe.D1 not in timeframes: + timeframes.append(Timeframe.D1) dfs_full: Dict[Timeframe, pd.DataFrame] = {} # Which parquet each TF came from (recorded in --record's manifest so # a study can prove two runs read the same bars). diff --git a/backtest/instructions_for_backtest.txt b/backtest/instructions_for_backtest.txt index e43d084..444718e 100644 --- a/backtest/instructions_for_backtest.txt +++ b/backtest/instructions_for_backtest.txt @@ -65,3 +65,26 @@ re-implementation. and it refuses configurations beyond its compile-time limits (more than 4 confirmation timeframes, divergence lookback >= 64, swing lookback >= 256, cp_lookback >= 128) with a message that points back to gpu_backtest.py. + +--- 2026-09-30: daily-trend veto (M15) and the D1 frame --- +- The M15 factory config carries xau_strategy.daily_trend_veto = "both" (M15 override 7). Both + backtesters then load the D1 frame as an EXTRA frame next to the primary and confirmation + frames (the "Confirm TFs:" line does not list it), cached under its own pad: + backtest/cache/oanda_XAU_USD_1d__.parquet. A new M15 window therefore + needs that file (one network fetch the first time; "GPU cache fallbacks: 0" is unaffected). +- The veto reads CLOSED daily bars only (a bar labelled D counts from D + 1 day <= the primary + bar's close), drops the provider's Sunday stub bars, and removes a finished setup that goes + against a non-zero state. Definition and constants: signal_aggregator.daily_trend_series. +- Research switches, both default-neutral elsewhere: --set xau_strategy.daily_trend_veto= + off|both|buy|sell and --set xau_strategy.wide_stop_cap_atr=K (K = 0 is off; the cap FAILED its + rule on 2026-09-30 and ships nowhere). A control for an M15 candidate is the factory config, + i.e. WITH the veto; to reproduce a pre-2026-09-30 M15 book pass --set + xau_strategy.daily_trend_veto=off. +- A candidate that is a FINAL veto (it only removes finished setups) may be screened on a research + harness (K1 + K2 of the twin, a mask on the final candidates before the cooldown, the shared + resolver) once that harness is shown to reproduce a shipped book byte for byte; the formal + numbers still come from the real code path with the paired gpu_backtest.py MD5. +- The 2026-09-30 report (backtest/research/daily_trend_campaign_2026-09-30.md) is the template for + a risk-first rule: rules, windows and neighbours written before the runs; the long window + (2019-01-01 -> 2026-08-07) and an unseen holdout as clauses; the rising-market cost reported, + not hidden. diff --git a/backtest/research/context_screens_2026-10.md b/backtest/research/context_screens_2026-10.md new file mode 100644 index 0000000..24bf9c7 --- /dev/null +++ b/backtest/research/context_screens_2026-10.md @@ -0,0 +1,235 @@ +# Context screens after the daily-trend veto — declared before looking, 2026-10-01 + +Follow-up to `daily_trend_campaign_2026-09-30.md` (owner decision 2026-09-30: "more context +screens"). The daily chart was information the engine had never seen, and it paid on M15. This +file asks the same question of seven more pieces of context. Sections 1 to 3 were written before +any number in section 4 existed. + +## 1. Books and slices + +The books as shipped on 2026-09-30, `--spread 0.65`, CUDA twin (byte-identical to +`gpu_backtest.py` on the paired runs of the campaign report): + +| Book | Window | Trades / R | MD5 | +|---|---|---|---| +| M15 LONG (veto on) | 2019-01-01 → 2026-08-07 | 1473 / +223.33 | `aa5253aa…` | +| M15 W3 (veto on) | 2023-09-05 → 2026-09-05 | 824 / +205.12 | `ad3b0dbd…` | +| M5 LONG | 2019-01-01 → 2026-08-07 | 1052 / +128.10 | `3f0c091c…` | +| M5 W3 | 2023-09-05 → 2026-09-05 | 870 / +153.64 | `d20ec8a3…` | + +Four slices per timeframe: LONG era A (entries 2019–2023), LONG era B (2024 → 2026-08-07), W3 +first half (before 2025-03-05), W3 second half. W3 overlaps era B, so the four slices are not +independent; that is known and accepted for a screen. + +## 2. The seven features (closed bars only, as of the trade's decision time) + +Decision time = the trade's entry time (the entry is the open of the bar after the signal bar, so +it equals the signal bar's close). A bar counts only when it had closed by then. + +Daily bars: provider D1 candles (UTC days), Sunday stub bars dropped, usable from label + 1 day. + +1. **Weekly trend.** Weekly bars = the kept daily bars grouped into weeks ending Friday; a week is + usable once its Friday bar is. State +1 when the weekly close is above its 20-week EMA + (`ewm(span=20, adjust=False)`) AND above the close 12 weeks earlier; −1 when below both; else 0. + Trade state: `with` / `against` / `unclear`. +2. **Four-hour trend.** Provider H4 candles (UTC-aligned), usable from label + 4 h. State +1 when + the close is above its 50-bar EMA AND above the close 30 bars earlier; −1 when below both; else + 0. Trade state: `with` / `against` / `unclear`. +3. **Daily-trend strength.** `(Close − Close 60 kept days earlier) / ATR14`, ATR14 = the simple + 14-day mean of the true range, taken in the trade's direction (multiplied by −1 for a SELL). + Trade state by fixed cut-offs: `counter` (< 0), `weak` (0 to 5), `medium` (5 to 12), + `strong` (> 12 daily ATRs). +4. **Daily volatility state.** ATR14 / Close, ranked within its own trailing 250 kept days + (percentile, the current day included). Trade state: `quiet` (< 33%), `normal`, `wild` (> 67%). +5. **Place in the 20-day range.** `(Close − lowest Low of 20 kept days) / (highest High − lowest + Low)`, mirrored for a SELL (1 − value). Trade state: `near the far edge` (< 1/3: buying near + the 20-day low or selling near the high), `middle`, `near the chasing edge` (> 2/3: buying near + the 20-day high or selling near the low). +6. **Tick-volume surge.** Volume of the signal bar divided by the median volume of the same clock + slot (same hour and minute, UTC) over that slot's previous 20 bars. Trade state: `thin` + (< 0.8), `usual` (0.8 to 1.5), `surge` (> 1.5). +7. **Time to a high-impact US release** (`backtest/cache/calendar.parquet`, currency USD, impact + high, time not masked; the file starts 2023-09-01, so W3 only). Minutes from the decision time + to the nearest such release. Trade state: `0–30 min before`, `0–30 min after`, + `30–120 min before`, `30–120 min after`, `clear` (more than 2 hours either side). + +Rows whose feature cannot be computed (not enough history) are counted as `n/a` and never flagged. + +## 3. Rule + +A screen is an in-book statistic: it cannot see the trades a change would add and it ignores the +5-bar cooldown. It can only nominate a hypothesis. + +- A state is **nominated** only if, on one timeframe, it holds at least 8% of the trades of BOTH + books (LONG and W3) and its total R is negative in ALL FOUR slices. +- For each nominated state the mechanism is written down before its replay. +- At most **three** replays in total, on the final-veto research harness of the campaign (K1 + K2 + of the twin, a mask on the final candidates before the cooldown, the shared resolver), judged by + the same risk-first rule T against the shipped book: W3 R ≥ 97% of control, W3 PF ≥ control + + 0.10, W3 max drawdown ≤ control, LONG R ≥ control, LONG max drawdown ≤ control, 2021 + 2022 + better, HOLDOUT R ≥ control − 0.5R, the same at $0.30, at least 20 trades differ. (No + neighbour-definition clause here: a pass only earns a full Stage-1/Stage-2 build, where the + neighbours are tested.) +- No nomination, no replay. No second cut-off, no other bins, no "best of seven" story afterwards. + With 7 features, about 25 states and 2 timeframes, a state that is negative in four overlapping + slices by luck alone is not rare; that is why a nomination is only a ticket to a replay. + +## 4. Results + +Run 2026-10-01 on the four books of section 1 (read from the campaign's exported CSVs; the feature +frames are the kept Oanda cache windows: daily `1d_20150303_20260807` / `1d_20191105_20260905`, +four-hour `4h_20181031_20260807` / `4h_20230114_20260905`, the books' own primary frames for +volume, `calendar.parquet` for releases). Every trade's decision time equals its entry time in +all four books (no entry sat behind a market closure). Cells are total R (trades); "mean" is R per +trade. + +### 4.1 M15 (daily-trend veto on): LONG 1473 trades / +223.33R, W3 824 trades / +205.12R + +**M15 — 1. weekly trend** + +| State | LONG trades (share) | LONG R / mean | W3 trades (share) | W3 R / mean | LONG era A (2019–2023) | LONG era B (2024–2026) | W3 first half | W3 second half | Nominated | +|---|---|---|---|---|---|---|---|---|---| +| against | 78 (5%) | -15.0 / -0.193 | 25 (3%) | +3.9 / +0.157 | -18.3 (58) | +3.2 (20) | +3.8 (7) | +0.1 (18) | | +| unclear | 176 (12%) | +28.1 / +0.160 | 101 (12%) | +37.8 / +0.375 | +1.9 (102) | +26.2 (74) | +25.4 (61) | +12.5 (40) | | +| with | 1219 (83%) | +210.3 / +0.173 | 698 (85%) | +163.4 / +0.234 | +53.1 (568) | +157.2 (651) | +47.6 (316) | +115.8 (382) | | + +**M15 — 2. four-hour trend** + +| State | LONG trades (share) | LONG R / mean | W3 trades (share) | W3 R / mean | LONG era A (2019–2023) | LONG era B (2024–2026) | W3 first half | W3 second half | Nominated | +|---|---|---|---|---|---|---|---|---|---| +| against | 260 (18%) | +23.3 / +0.090 | 134 (16%) | +14.8 / +0.111 | +12.4 (137) | +10.9 (123) | +3.4 (59) | +11.4 (75) | | +| unclear | 167 (11%) | +16.8 / +0.100 | 84 (10%) | +16.7 / +0.199 | +0.3 (90) | +16.5 (77) | +1.8 (41) | +14.9 (43) | | +| with | 1046 (71%) | +183.3 / +0.175 | 606 (74%) | +173.6 / +0.286 | +24.1 (501) | +159.2 (545) | +71.5 (284) | +102.1 (322) | | + +**M15 — 3. daily-trend strength** + +| State | LONG trades (share) | LONG R / mean | W3 trades (share) | W3 R / mean | LONG era A (2019–2023) | LONG era B (2024–2026) | W3 first half | W3 second half | Nominated | +|---|---|---|---|---|---|---|---|---|---| +| counter | 145 (10%) | -4.8 / -0.033 | 72 (9%) | +16.9 / +0.234 | -13.1 (92) | +8.3 (53) | +15.2 (39) | +1.7 (33) | | +| medium | 586 (40%) | +106.6 / +0.182 | 393 (48%) | +94.6 / +0.241 | +16.7 (199) | +89.9 (387) | +9.5 (162) | +85.1 (231) | | +| strong | 17 (1%) | +4.6 / +0.268 | 10 (1%) | +4.8 / +0.483 | -0.3 (7) | +4.8 (10) | -1.1 (1) | +5.9 (9) | | +| weak | 725 (49%) | +117.0 / +0.161 | 349 (42%) | +88.8 / +0.254 | +33.5 (430) | +83.5 (295) | +53.1 (182) | +35.7 (167) | | + +**M15 — 4. daily volatility state** + +| State | LONG trades (share) | LONG R / mean | W3 trades (share) | W3 R / mean | LONG era A (2019–2023) | LONG era B (2024–2026) | W3 first half | W3 second half | Nominated | +|---|---|---|---|---|---|---|---|---|---| +| normal | 445 (30%) | +54.8 / +0.123 | 299 (36%) | +77.6 / +0.259 | -11.8 (194) | +66.6 (251) | +6.7 (118) | +70.8 (181) | | +| quiet | 373 (25%) | +64.1 / +0.172 | 152 (18%) | +51.7 / +0.340 | +24.2 (235) | +39.9 (138) | +38.9 (109) | +12.8 (43) | | +| wild | 655 (44%) | +104.4 / +0.159 | 373 (45%) | +75.9 / +0.203 | +24.3 (299) | +80.1 (356) | +31.1 (157) | +44.8 (216) | | + +**M15 — 5. place in the 20-day range** + +| State | LONG trades (share) | LONG R / mean | W3 trades (share) | W3 R / mean | LONG era A (2019–2023) | LONG era B (2024–2026) | W3 first half | W3 second half | Nominated | +|---|---|---|---|---|---|---|---|---|---| +| middle | 420 (29%) | +54.9 / +0.131 | 230 (28%) | +32.3 / +0.140 | +25.3 (217) | +29.7 (203) | +30.9 (107) | +1.4 (123) | | +| near the chasing edge | 835 (57%) | +143.5 / +0.172 | 483 (59%) | +147.1 / +0.305 | +8.8 (400) | +134.7 (435) | +40.6 (227) | +106.5 (256) | | +| near the far edge | 218 (15%) | +24.8 / +0.114 | 111 (13%) | +25.7 / +0.231 | +2.6 (111) | +22.2 (107) | +5.3 (50) | +20.4 (61) | | + +**M15 — 6. tick-volume surge** + +| State | LONG trades (share) | LONG R / mean | W3 trades (share) | W3 R / mean | LONG era A (2019–2023) | LONG era B (2024–2026) | W3 first half | W3 second half | Nominated | +|---|---|---|---|---|---|---|---|---|---| +| surge | 556 (38%) | +83.1 / +0.149 | 226 (27%) | +45.7 / +0.202 | +45.8 (355) | +37.3 (201) | +22.5 (110) | +23.1 (116) | | +| thin | 220 (15%) | +15.6 / +0.071 | 125 (15%) | +28.5 / +0.228 | -10.4 (99) | +26.0 (121) | +11.5 (48) | +17.0 (77) | | +| usual | 697 (47%) | +124.7 / +0.179 | 473 (57%) | +130.9 / +0.277 | +1.4 (274) | +123.2 (423) | +42.7 (226) | +88.3 (247) | | + +**M15 — 7. time to a high-impact US release (W3 only)** + +| State | LONG trades (share) | LONG R / mean | W3 trades (share) | W3 R / mean | LONG era A (2019–2023) | LONG era B (2024–2026) | W3 first half | W3 second half | Nominated | +|---|---|---|---|---|---|---|---|---|---| +| 0–30 min after | | | 42 (5%) | +8.9 / +0.212 | | | +4.7 (27) | +4.2 (15) | | +| 0–30 min before | | | 26 (3%) | +8.7 / +0.336 | | | +7.8 (18) | +0.9 (8) | | +| 30–120 min after | | | 75 (9%) | +28.4 / +0.378 | | | +11.7 (51) | +16.6 (24) | | +| 30–120 min before | | | 22 (3%) | +16.1 / +0.732 | | | +12.9 (18) | +3.2 (4) | | +| clear | | | 659 (80%) | +143.0 / +0.217 | | | +39.6 (270) | +103.5 (389) | | + +### 4.2 M5: LONG 1052 trades / +128.10R, W3 870 trades / +153.64R + +**M5 — 1. weekly trend** + +| State | LONG trades (share) | LONG R / mean | W3 trades (share) | W3 R / mean | LONG era A (2019–2023) | LONG era B (2024–2026) | W3 first half | W3 second half | Nominated | +|---|---|---|---|---|---|---|---|---|---| +| against | 258 (25%) | +11.0 / +0.043 | 202 (23%) | +6.6 / +0.033 | +9.4 (69) | +1.6 (189) | +3.8 (10) | +2.8 (192) | | +| unclear | 84 (8%) | +6.3 / +0.075 | 88 (10%) | +24.6 / +0.279 | -7.5 (30) | +13.8 (54) | +8.2 (12) | +16.4 (76) | | +| with | 710 (67%) | +110.8 / +0.156 | 580 (67%) | +122.5 / +0.211 | -12.3 (159) | +123.1 (551) | +4.7 (114) | +117.8 (466) | | + +**M5 — 2. four-hour trend** + +| State | LONG trades (share) | LONG R / mean | W3 trades (share) | W3 R / mean | LONG era A (2019–2023) | LONG era B (2024–2026) | W3 first half | W3 second half | Nominated | +|---|---|---|---|---|---|---|---|---|---| +| against | 332 (32%) | +3.0 / +0.009 | 255 (29%) | +11.2 / +0.044 | -5.5 (98) | +8.4 (234) | -0.9 (35) | +12.2 (220) | | +| unclear | 126 (12%) | +13.7 / +0.109 | 105 (12%) | +17.8 / +0.170 | +0.6 (28) | +13.1 (98) | +0.8 (13) | +17.0 (92) | | +| with | 594 (56%) | +111.4 / +0.188 | 510 (59%) | +124.6 / +0.244 | -5.5 (132) | +116.9 (462) | +16.9 (88) | +107.7 (422) | | + +**M5 — 3. daily-trend strength** + +| State | LONG trades (share) | LONG R / mean | W3 trades (share) | W3 R / mean | LONG era A (2019–2023) | LONG era B (2024–2026) | W3 first half | W3 second half | Nominated | +|---|---|---|---|---|---|---|---|---|---| +| counter | 296 (28%) | +0.7 / +0.003 | 241 (28%) | -2.6 / -0.011 | +8.8 (87) | -8.0 (209) | +3.7 (18) | -6.3 (223) | | +| medium | 367 (35%) | +77.3 / +0.211 | 318 (37%) | +78.4 / +0.247 | -1.2 (49) | +78.4 (318) | +6.9 (53) | +71.6 (265) | | +| strong | 14 (1%) | +16.6 / +1.186 | 13 (1%) | +15.6 / +1.199 | +1.0 (1) | +15.6 (13) | +0.0 (0) | +15.6 (13) | | +| weak | 375 (36%) | +33.5 / +0.089 | 298 (34%) | +62.2 / +0.209 | -19.0 (121) | +52.5 (254) | +6.2 (65) | +56.1 (233) | | + +**M5 — 4. daily volatility state** + +| State | LONG trades (share) | LONG R / mean | W3 trades (share) | W3 R / mean | LONG era A (2019–2023) | LONG era B (2024–2026) | W3 first half | W3 second half | Nominated | +|---|---|---|---|---|---|---|---|---|---| +| normal | 302 (29%) | +33.2 / +0.110 | 305 (35%) | +45.6 / +0.150 | -0.8 (63) | +34.0 (239) | -3.0 (42) | +48.6 (263) | | +| quiet | 114 (11%) | +18.8 / +0.165 | 72 (8%) | +17.9 / +0.249 | +3.0 (45) | +15.8 (69) | +10.9 (37) | +7.0 (35) | | +| wild | 636 (60%) | +76.1 / +0.120 | 493 (57%) | +90.1 / +0.183 | -12.6 (150) | +88.8 (486) | +8.8 (57) | +81.3 (436) | | + +**M5 — 5. place in the 20-day range** + +| State | LONG trades (share) | LONG R / mean | W3 trades (share) | W3 R / mean | LONG era A (2019–2023) | LONG era B (2024–2026) | W3 first half | W3 second half | Nominated | +|---|---|---|---|---|---|---|---|---|---| +| middle | 323 (31%) | +11.8 / +0.037 | 271 (31%) | +18.8 / +0.069 | -6.8 (76) | +18.6 (247) | -1.8 (34) | +20.6 (237) | | +| near the chasing edge | 439 (42%) | +101.5 / +0.231 | 380 (44%) | +115.0 / +0.303 | -2.6 (103) | +104.1 (336) | +17.2 (77) | +97.8 (303) | | +| near the far edge | 290 (28%) | +14.8 / +0.051 | 219 (25%) | +19.9 / +0.091 | -1.0 (79) | +15.9 (211) | +1.4 (25) | +18.5 (194) | | + +**M5 — 6. tick-volume surge** + +| State | LONG trades (share) | LONG R / mean | W3 trades (share) | W3 R / mean | LONG era A (2019–2023) | LONG era B (2024–2026) | W3 first half | W3 second half | Nominated | +|---|---|---|---|---|---|---|---|---|---| +| surge | 479 (46%) | +46.2 / +0.097 | 334 (38%) | +65.3 / +0.195 | -5.9 (176) | +52.1 (303) | +13.2 (64) | +52.1 (270) | | +| thin | 152 (14%) | +9.3 / +0.061 | 134 (15%) | +18.1 / +0.135 | -8.0 (22) | +17.3 (130) | -0.7 (7) | +18.8 (127) | | +| usual | 421 (40%) | +72.6 / +0.172 | 402 (46%) | +70.3 / +0.175 | +3.4 (60) | +69.2 (361) | +4.2 (65) | +66.1 (337) | | + +**M5 — 7. time to a high-impact US release (W3 only)** + +| State | LONG trades (share) | LONG R / mean | W3 trades (share) | W3 R / mean | LONG era A (2019–2023) | LONG era B (2024–2026) | W3 first half | W3 second half | Nominated | +|---|---|---|---|---|---|---|---|---|---| +| 0–30 min after | | | 46 (5%) | -3.8 / -0.082 | | | -0.6 (15) | -3.2 (31) | | +| 0–30 min before | | | 18 (2%) | +1.8 / +0.098 | | | +0.7 (5) | +1.1 (13) | | +| 30–120 min after | | | 87 (10%) | +32.8 / +0.377 | | | +8.3 (31) | +24.4 (56) | | +| 30–120 min before | | | 18 (2%) | +3.4 / +0.187 | | | +2.7 (7) | +0.7 (11) | | +| clear | | | 701 (81%) | +119.5 / +0.171 | | | +5.6 (78) | +114.0 (623) | | + +### 4.3 Verdict + +**Nominated states: none.** No state on either timeframe holds 8% of both books and loses in all +four slices. By the rule of section 3 there is no replay and nothing ships. + +What the tables do say (in-book description, not a decision): + +- **M15, after the daily-trend veto, has no losing pocket left in these seven views.** The only + clearly negative cell is trades against the weekly trend in 2019–2023 (−18.3R over 58 trades), + 5% of the book and positive since 2024. Trades with the four-hour trend earn +0.18R to +0.29R + each against +0.09R to +0.11R against it: weaker, still positive. +- **M5 earns its R where everything already agrees.** With the weekly trend +0.16R to +0.21R per + trade, against it +0.03R to +0.04R; with the four-hour trend +0.19R to +0.24R, against it + +0.01R to +0.04R; when the 60-day move runs against the trade, 0.00R (about 28% of the book). + Those against-the-tide M5 trades are flat, not losing, which is the same picture the formal + daily-trend test gave on M5: removing them removes no loss, and it made the worst losing run + deeper. They stay. +- **M5's profit sits near the 20-day extreme in the trade's direction** (+0.23R to +0.30R per + trade against +0.04R to +0.09R elsewhere). This repeats the 2026-08-04 finding that the best + trades are the stretched runs: a "do not chase" filter would cut the part of the book that pays. +- **Volume and news timing separate nothing reliably.** A volume surge is the best state in + 2019–2023 on M15 and the middle state since; trades 30 to 120 minutes after a major US release + look good on both timeframes (+0.38R each) on 75 and 87 trades, and trades in the half hour + after a release are slightly negative on M5 (−3.8R over 46 trades, 5% of the book, below the + 8% bar). Neither is a rule. + +So the daily chart was the one piece of missing context that paid, on M15. Weekly and four-hour +context add nothing on top of it there, and nothing on M5 that survives the rule. diff --git a/backtest/research/daily_trend_campaign_2026-09-30.md b/backtest/research/daily_trend_campaign_2026-09-30.md new file mode 100644 index 0000000..1e7064e --- /dev/null +++ b/backtest/research/daily_trend_campaign_2026-09-30.md @@ -0,0 +1,639 @@ +# Daily-trend veto and wide-stop cap — pre-registered campaign, 2026-09-30 + +Owner request (2026-09-30): improve the direction call and the trade setup size; anything may +change if the evidence supports it. This file holds the rules first, then the results. Sections 1 +to 4 were written before any formal run. Section 2 records what was already looked at while +planning, so the reader knows which numbers are a confirmation and which are a first look. + +## 1. What is tested + +**Candidate T — daily-trend veto.** The engine sees at most the H1 chart (M5 confirms on M15 and +H1, M15 on M5 and H1). None of the 65 decision-time features of the earlier campaigns reads the +daily chart. The candidate drops a finished trade setup that goes against a clear daily trend. + +Definition (fixed; `signal_aggregator.daily_trend_series`): +- daily candles from the provider (UTC-aligned, label = 00:00 UTC of the day), closed bars only: + a bar labelled D is used once D + 1 day ≤ the close of the primary bar; +- bars labelled Sunday are dropped first (Oanda's 1 to 2 hour reopening stub); +- `ema = Close.ewm(span=50, adjust=False).mean()`, `change = Close / Close.shift(60) − 1`; +- state +1 when `Close > ema` and `change > 0`; −1 when `Close < ema` and `change < 0`; else 0; +- after every setup archetype has run (gold ATR, swing, pullback, chart pattern) a setup against a + non-zero state is removed: grade D, no setup, warning `DAILY TREND FILTER: …`. +Switch `xau_strategy.daily_trend_veto`: `off` (default, books byte-identical), `both`, `buy` +(only buys against a falling daily trend are removed), `sell`. + +**Candidate C — wide-stop cap.** `xau_strategy.wide_stop_cap_atr = K` caps the stop distance of +the swing (GENERIC) and chart-pattern setups at `max(K × ATR, the existing floor)`, applied after +the min-RR and spread gates so the set of trades cannot change. K = 6, with 5, 7 and 8 declared +as neighbours. M15 only (the exit-only screen was negative on M5). + +## 2. What was already seen (planning phase, 2026-09-30, read-only, in memory) + +Prototype: K1 and K2 of the CUDA twin, then a host-side mask on the final candidates before the +5-bar cooldown, then the shared resolver. The control rows of the prototype reproduce the pinned +books (M15 W3 1105 / +195.79R, M5 W3 870 / +153.64R, H1 W3 253 / +70.02R). + +| Book ($0.65) | Control | Veto, both directions | +|---|---|---| +| M15 W3 2023-09-05 → 2026-09-05 | 1105 / +195.8R / PF 1.49 / DD 20.9 | 824 / +205.1R / PF 1.73 / DD 12.4 | +| M15 LONG 2019-01-01 → 2026-08-07 | 2091 / +192.5R / PF 1.24 / DD 64.8 | 1473 / +223.3R / PF 1.42 / DD 41.5 | +| M15 rising window 2024-06-01 → 2025-03-30 | 313 / +68.5R / PF 1.67 | 258 / +63.8R / PF 1.80 | +| M5 W3 | 870 / +153.6R / PF 1.50 / DD 10.1 | 668 / +159.1R / PF 1.70 / DD 15.2 | +| M5 W3, buy side only | | 741 / +156.5R / PF 1.60 / DD 9.6 | +| M5 LONG | 1052 / +128.1R / PF 1.34 / DD 18.4 | 790 / +127.5R / PF 1.45 / DD 25.2 | +| H1 W3 | 253 / +70.0R / PF 1.98 / DD 4.4 | 231 / +60.3R / PF 1.90 / DD 4.4 | + +Also seen: neighbouring definitions on M15 (60-day change alone +188.3R W3 / +228.1R LONG; +90-day +196.1 / +228.8; 120-day +199.4 / +233.2; 100-day average +201.2 / +227.3); the share of +M15 W3 trades reaching +1R before −1R (58.6% → 61.4%); the live September 2026 setups with the +veto applied after the fact (cooldown view −11.6R → −6.0R to −8.8R); an exit-only re-simulation +of 25 stop and exit variants (cap 6 × ATR on M15 W3 +14.0R, LONG +11.8R; cap 5 negative on LONG); +two variants dropped at this stage and NOT tested formally: lifting the SELL gates when the daily +trend is down (worse on LONG on both timeframes), and removing the existing M15 short blocks +under the veto (costs 4 to 22R). + +So the windows W3, W2, the rising window and LONG are a CONFIRMATION through the real code path. +First looks are: the HOLDOUT window through the backtester, the $0.30 numbers on LONG, and every +number produced by the shipped code rather than the prototype. + +## 3. Harness + +`venv/bin/python backtest/cudaBasedBacktesting.py --pair XAUUSD --primary-tf --start +--end --spread <0.65|0.30> [--set xau_strategy.daily_trend_veto=] +[--set xau_strategy.wide_stop_cap_atr=] --export-csv `; factory configuration otherwise. +Twin numbers count only with the paired `backtest/gpu_backtest.py` MD5 (same flags plus +`--workers 12`). Neighbouring trend definitions are not shipped code: they run on the research +harness (the prototype), after the harness book for the shipped definition is shown identical to +the twin's. + +Windows: **W3** 2023-09-05 → 2026-09-05 (halves at 2025-03-05); **W2** 2024-09-05 → 2026-09-05 +(halves at 2025-09-05); **RISING** 2024-06-01 → 2025-03-30 (the old out-of-sample window; a +rising market throughout); **LONG** 2019-01-01 → 2026-08-07; **HOLDOUT** 2026-09-05 → 2026-10-01 +(fetched for this campaign, never seen by any tuning or by the prototype). + +## 4. Acceptance rules (written before the runs) + +**Rule T (risk-first; owner decision 2026-09-30).** Candidate T on a timeframe PASSES only if, +against the same-harness control: +- (a) W3 total R ≥ 97% of the control's at $0.65; +- (b) W3 profit factor ≥ control + 0.10; +- (c) W3 max drawdown ≤ the control's; +- (d) LONG: total R ≥ control, max drawdown ≤ control, and 2021 + 2022 combined R better; +- (e) HOLDOUT total R ≥ control − 0.5R; +- (f) clauses (a) to (c) also hold at $0.30; +- (g) at least 3 of the 4 neighbouring definitions (60-day change alone, 90-day, 120-day, + 100-day average alone) satisfy (a) and (c): a plateau, not a spike; +- (h) at least 20 trades differ on W3. +Reported, NOT clauses (the owner accepted this cost on 2026-09-30): the W2 halves and the RISING +window, where the veto is expected to give up some R. The 2026-09-26 template (every half and the +old out-of-sample window must not lose) would reject this candidate by construction; that is the +difference between the two rules and it is deliberate. +Decision per timeframe: `both` if it passes; else `buy` if that passes; else no veto (the daily +trend is then shown on the alert as information only). H1 is run for the record. + +**Rule C (standard).** K = 6 on M15 against the post-T control PASSES only if: W3 ΔR ≥ +5R; both +W3 halves ≥ control − 1R; PF ≥ control − 0.02; W3 max drawdown ≤ control + 10%; LONG total R ≥ +control; the same sign of ΔR at $0.30; at least 20 trades differ; and the neighbours K = 5 and +K = 7 are not negative on W3 or on LONG. + +No rescue for either rule: no second definition, no other K, no window shopping after the result. + +## 5. Results + +All runs 2026-09-30 on the dev box, branch `sql-integration`, working tree = HEAD `0c56b4e` plus +this campaign's code. Every twin log reads +`Backend: CuPy GPU (NVIDIA GeForce RTX 4070 SUPER, cupy 14.0.1)` and +`GPU cache fallbacks: 0 (no per-bar recompute path in cudaBasedBacktesting)`; every reference log +reads the same `Backend:` line and `GPU cache fallbacks: 0`. 86 formal twin books (5 windows × 2 +costs × the variants below, plus the fixed-1R-target books). R is per trade in units of the +initial risk; "DD" is the largest peak-to-trough fall of the running total, in R. + +### 5.1 Proofs before any formal number + +- **Baseline.** Before any code change the five reference MD5s of CLAUDE.md §9 were reproduced on + both scripts (ten books). +- **Default-off neutrality.** With the new code and both knobs at their defaults the same ten + books are byte-identical to the pins (`e599c8a8…` M5 W2, `5b65ecf6…` M15 W2, `09b8a8e7…` M15 + OOS, `a44fd19b…` H1 W3, `f3587bae…` H4 W3). +- **Twin = reference on every new path** (`--spread 0.65`, W3 2023-09-05 → 2026-09-05, equal MD5 + from `backtest/gpu_backtest.py --workers 12` and `backtest/cudaBasedBacktesting.py`): + +| Flags | Trades / R / DD | MD5 (both scripts) | +|---|---|---| +| M15 `--set xau_strategy.daily_trend_veto=both` | 824 / +205.12 / 12.39 | `ad3b0dbda0bf77b659caea1c53be6a50` | +| M15 `…=buy` | 999 / +188.21 / 20.87 | `6e0f3b742d9690f78c97319d1b1c8a43` | +| M15 `…=sell` | 930 / +212.70 / 13.72 | `31aa5762964d846f15d111cd4a718adc` | +| M15 `--set xau_strategy.wide_stop_cap_atr=6` | 1105 / +209.78 / 19.32 | `4ec0e4450d5dab143a4ed37e80545626` | +| M5 `--set xau_strategy.daily_trend_veto=both` | 668 / +159.07 / 15.20 | `39cc0f198d38b532720dce4f98d99452` | +| M5 `--set xau_strategy.wide_stop_cap_atr=6` | 870 / +141.57 / 13.95 | `4da0997256b272ffcd11a90bf9658eef` | + + The M15 `both` book equals the planning prototype trade for trade (824 / +205.1R / PF 1.73 / + DD 12.4), so the shipped definition is the one that was screened. +- **Suite.** 595 passed / 0 failed after Stage 1 (Module 39 new, Module 37 extended). + +### 5.2 Candidate T — every window, both costs + +Command per row: `venv/bin/python backtest/cudaBasedBacktesting.py --pair XAUUSD --primary-tf +<15min|5min> --start --end --spread <0.65|0.30> [--set xau_strategy.daily_trend_veto=] +--export-csv `. "control" = no `--set`. + +**M15** + +| Window | Cost | Variant | Trades | Total R | Win rate | PF | Max DD | First half | Second half | MD5 (8) | +|---|---|---|---|---|---|---|---|---|---|---| +| W3 | $0.65 | control | 1105 | +195.79 | 61.1% | 1.49 | 20.9 | +76.87 | +118.92 | 70195fb0 | +| W3 | $0.65 | both | 824 | +205.12 | 63.1% | 1.73 | 12.4 | +76.72 | +128.40 | ad3b0dbd | +| W3 | $0.65 | buy | 999 | +188.21 | 61.3% | 1.52 | 20.9 | +60.40 | +127.81 | 6e0f3b74 | +| W3 | $0.65 | sell | 930 | +212.70 | 62.7% | 1.66 | 13.7 | +93.19 | +119.52 | 31aa5762 | +| W3 | $0.30 | control | 1105 | +216.53 | 61.4% | 1.55 | 19.6 | +88.86 | +127.67 | ce074a1b | +| W3 | $0.30 | both | 824 | +221.18 | 63.6% | 1.81 | 11.4 | +86.41 | +134.77 | 96a87454 | +| W3 | $0.30 | buy | 999 | +207.23 | 61.7% | 1.59 | 19.6 | +71.83 | +135.41 | 2ada14e4 | +| W3 | $0.30 | sell | 930 | +230.48 | 63.1% | 1.73 | 12.8 | +103.45 | +127.03 | 49665c41 | +| W2 | $0.65 | control | 813 | +154.15 | 62.2% | 1.53 | 15.6 | +70.10 | +84.04 | 5b65ecf6 | +| W2 | $0.65 | both | 597 | +155.89 | 64.3% | 1.78 | 11.4 | +61.83 | +94.06 | dfc647ee | +| W2 | $0.65 | buy | 718 | +154.67 | 63.0% | 1.62 | 14.4 | +65.03 | +89.65 | f23bfda4 | +| W2 | $0.65 | sell | 692 | +155.36 | 63.3% | 1.65 | 13.7 | +66.91 | +88.46 | 1a071ac7 | +| W2 | $0.30 | control | 813 | +167.53 | 62.6% | 1.59 | 14.5 | +78.61 | +88.91 | 15486607 | +| W2 | $0.30 | both | 597 | +166.22 | 64.8% | 1.85 | 11.0 | +68.82 | +97.39 | d2f678bf | +| W2 | $0.30 | buy | 718 | +166.64 | 63.4% | 1.68 | 14.0 | +73.12 | +93.52 | f3abc1e8 | +| W2 | $0.30 | sell | 692 | +167.10 | 63.7% | 1.71 | 12.8 | +74.31 | +92.79 | d7d038fd | +| RISING | $0.65 | control | 313 | +68.50 | 63.6% | 1.67 | 12.4 | +18.87 | +49.63 | 09b8a8e7 | +| RISING | $0.65 | both | 258 | +63.78 | 64.3% | 1.80 | 9.1 | +23.97 | +39.81 | e87b6de1 | +| RISING | $0.65 | buy | 303 | +60.15 | 63.0% | 1.60 | 12.4 | +18.87 | +41.28 | f001a0a6 | +| RISING | $0.65 | sell | 268 | +72.13 | 64.9% | 1.89 | 9.1 | +23.97 | +48.16 | cbe4cb24 | +| RISING | $0.30 | control | 313 | +76.21 | 64.5% | 1.77 | 11.1 | +22.92 | +53.29 | 5fc34cef | +| RISING | $0.30 | both | 258 | +70.24 | 65.5% | 1.91 | 7.9 | +27.31 | +42.92 | 5de32f51 | +| RISING | $0.30 | buy | 303 | +67.59 | 64.0% | 1.70 | 11.1 | +22.92 | +44.67 | 69b41026 | +| RISING | $0.30 | sell | 268 | +78.86 | 66.0% | 2.00 | 7.9 | +27.31 | +51.54 | 3631466e | +| LONG | $0.65 | control | 2091 | +192.53 | 57.8% | 1.24 | 64.8 | | | c2228398 | +| LONG | $0.65 | both | 1473 | +223.33 | 59.7% | 1.42 | 41.5 | | | aa5253aa | +| LONG | $0.65 | buy | 1762 | +214.62 | 58.9% | 1.33 | 34.5 | | | 952d64d8 | +| LONG | $0.65 | sell | 1802 | +201.23 | 58.3% | 1.30 | 76.4 | | | d3965ab8 | +| LONG | $0.30 | control | 2091 | +240.00 | 58.8% | 1.31 | 49.6 | | | 0c58b300 | +| LONG | $0.30 | both | 1473 | +257.25 | 60.8% | 1.50 | 37.2 | | | 1835b341 | +| LONG | $0.30 | buy | 1762 | +254.33 | 59.9% | 1.40 | 29.5 | | | a2b519bd | +| LONG | $0.30 | sell | 1802 | +242.92 | 59.4% | 1.37 | 62.3 | | | f6edb469 | +| HOLDOUT | $0.65 | control | 24 | −6.63 | 50.0% | 0.42 | 7.0 | | | c42470a0 | +| HOLDOUT | $0.65 | both | 18 | −4.62 | 50.0% | 0.50 | 4.8 | | | fa50bcd0 | +| HOLDOUT | $0.65 | buy | 22 | −6.81 | 45.5% | 0.41 | 7.0 | | | 548a4bf4 | +| HOLDOUT | $0.65 | sell | 20 | −4.44 | 55.0% | 0.52 | 4.8 | | | 1459ef5f | +| HOLDOUT | $0.30 | control | 24 | −6.29 | 50.0% | 0.44 | 6.7 | | | d7fbee9d | +| HOLDOUT | $0.30 | both | 18 | −4.35 | 50.0% | 0.52 | 4.5 | | | 2179c08c | +| HOLDOUT | $0.30 | buy | 22 | −6.49 | 45.5% | 0.43 | 6.7 | | | 0b90faeb | +| HOLDOUT | $0.30 | sell | 20 | −4.15 | 55.0% | 0.54 | 4.5 | | | 70932b7e | + +**M5** + +| Window | Cost | Variant | Trades | Total R | Win rate | PF | Max DD | First half | Second half | MD5 (8) | +|---|---|---|---|---|---|---|---|---|---|---| +| W3 | $0.65 | control | 870 | +153.64 | 53.3% | 1.50 | 10.1 | +16.73 | +136.91 | d20ec8a3 | +| W3 | $0.65 | both | 668 | +159.07 | 53.9% | 1.70 | 15.2 | +14.01 | +145.06 | 39cc0f19 | +| W3 | $0.65 | buy | 741 | +156.53 | 53.4% | 1.60 | 9.6 | +15.68 | +140.85 | 631f25ac | +| W3 | $0.30 | control | 870 | +172.40 | 54.3% | 1.57 | 8.9 | +21.08 | +151.32 | 5acafcba | +| W3 | $0.30 | both | 668 | +174.43 | 54.8% | 1.79 | 14.3 | +18.09 | +156.34 | 34f2e282 | +| W3 | $0.30 | buy | 741 | +172.72 | 54.3% | 1.69 | 8.9 | +19.87 | +152.86 | c8bdc9e2 | +| W2 | $0.65 | control | 781 | +146.07 | 52.9% | 1.52 | 10.1 | +26.42 | +119.65 | e599c8a8 | +| W2 | $0.65 | both | 585 | +153.82 | 53.7% | 1.78 | 15.2 | +33.16 | +120.66 | b9fa356c | +| W2 | $0.65 | buy | 656 | +151.09 | 53.0% | 1.66 | 9.6 | +27.42 | +123.66 | 30e243b0 | +| W2 | $0.30 | control | 781 | +162.09 | 53.9% | 1.60 | 8.9 | +31.71 | +130.37 | 7ab48db9 | +| W2 | $0.30 | both | 585 | +166.59 | 54.7% | 1.87 | 14.3 | +38.05 | +128.54 | 3427b933 | +| W2 | $0.30 | buy | 656 | +164.66 | 54.0% | 1.74 | 8.9 | +32.58 | +132.08 | 7dd9e2a2 | +| RISING | $0.65 | control | 86 | +9.47 | 53.5% | 1.33 | 8.2 | +0.01 | +9.46 | 65c46e62 | +| RISING | $0.65 | both | 81 | +9.07 | 54.3% | 1.34 | 7.5 | +0.67 | +8.40 | e20fdef2 | +| RISING | $0.65 | buy | 85 | +10.55 | 54.1% | 1.39 | 7.2 | +0.01 | +10.54 | de24b2e8 | +| RISING | $0.30 | control | 86 | +12.42 | 54.7% | 1.46 | 6.7 | +1.65 | +10.77 | 5c154360 | +| RISING | $0.30 | both | 81 | +11.91 | 55.6% | 1.47 | 6.1 | +2.28 | +9.63 | 3b1a8f9e | +| RISING | $0.30 | buy | 85 | +13.46 | 55.3% | 1.52 | 5.7 | +1.65 | +11.81 | 21ffab88 | +| LONG | $0.65 | control | 1052 | +128.10 | 50.5% | 1.34 | 18.4 | | | 3f0c091c | +| LONG | $0.65 | both | 790 | +127.47 | 50.4% | 1.45 | 25.2 | | | 9872c068 | +| LONG | $0.65 | buy | 871 | +123.37 | 49.9% | 1.39 | 27.1 | | | 613dcf0e | +| LONG | $0.30 | control | 1052 | +153.02 | 51.7% | 1.41 | 13.0 | | | 6708d8cc | +| LONG | $0.30 | both | 790 | +147.13 | 51.5% | 1.54 | 19.6 | | | 201d968f | +| LONG | $0.30 | buy | 871 | +144.02 | 51.1% | 1.47 | 22.3 | | | d9cba459 | +| HOLDOUT | $0.65 | control | 33 | −2.45 | 51.5% | 0.82 | 6.1 | | | 351f7149 | +| HOLDOUT | $0.65 | both | 26 | −2.04 | 50.0% | 0.81 | 5.4 | | | b9c8cb82 | +| HOLDOUT | $0.65 | buy | 33 | −2.45 | 51.5% | 0.82 | 6.1 | | | 351f7149 | +| HOLDOUT | $0.30 | control | 33 | −1.85 | 51.5% | 0.86 | 5.6 | | | 5a6aea9c | +| HOLDOUT | $0.30 | both | 26 | −1.56 | 50.0% | 0.85 | 5.0 | | | ddd29068 | +| HOLDOUT | $0.30 | buy | 33 | −1.85 | 51.5% | 0.86 | 5.6 | | | 5a6aea9c | + +**H1, for the record** ($0.65): W3 control 253 / +70.02R / PF 1.98 / DD 4.4 (`a44fd19b`), `both` +231 / +60.28R / PF 1.90 / DD 4.4 (`cdac563b`); HOLDOUT 3 / −3.04R in both (no trade removed). +Fails (a) (60.28 < 67.92) and (b). H4 was not run: it confirms on the daily chart already. + +### 5.3 Per calendar year ($0.65), total R / trades + +M15, LONG 2019-01-01 → 2026-08-07: + +| Variant | 2019 | 2020 | 2021 | 2022 | 2023 | 2024 | 2025 | 2026 | +|---|---|---|---|---|---|---|---|---| +| control | +9.0 / 117 | +48.8 / 283 | −25.6 / 210 | −24.6 / 267 | +11.7 / 211 | +53.3 / 343 | +81.2 / 425 | +38.7 / 235 | +| both | +14.0 / 88 | +45.1 / 207 | −27.3 / 121 | +3.2 / 149 | +1.8 / 163 | +53.5 / 275 | +92.5 / 341 | +40.6 / 129 | +| buy | +7.6 / 111 | +44.2 / 253 | −21.4 / 135 | +7.4 / 166 | +1.6 / 189 | +46.4 / 335 | +83.1 / 416 | +45.9 / 157 | +| sell | +15.4 / 94 | +49.7 / 237 | −31.5 / 196 | −28.8 / 250 | +11.9 / 185 | +60.4 / 283 | +90.6 / 350 | +33.4 / 207 | + +M5, LONG: + +| Variant | 2019 | 2020 | 2021 | 2022 | 2023 | 2024 | 2025 | 2026 | +|---|---|---|---|---|---|---|---|---| +| control | −1.0 / 20 | −2.8 / 95 | −1.7 / 32 | −2.2 / 50 | −2.8 / 61 | +1.0 / 102 | +40.5 / 332 | +97.0 / 360 | +| both | +0.2 / 17 | −6.9 / 76 | −3.4 / 14 | −4.7 / 28 | −3.5 / 54 | +1.5 / 96 | +43.2 / 297 | +101.0 / 208 | +| buy | +0.2 / 17 | −6.7 / 83 | −3.2 / 16 | −6.0 / 30 | −4.6 / 56 | +2.1 / 101 | +40.5 / 329 | +101.0 / 239 | + +M15, W3 per calendar year (total R / trades / max DD inside the year): + +| Variant | 2023 (from 09-05) | 2024 | 2025 | 2026 (to 09-05) | +|---|---|---|---|---| +| control | +10.8 / 66 / 9.0 | +53.3 / 343 / 16.3 | +81.2 / 425 / 15.6 | +50.5 / 271 / 10.3 | +| both | +9.3 / 47 / 6.9 | +53.5 / 275 / 11.0 | +92.5 / 341 / 11.4 | +49.9 / 161 / 8.3 | + +M15, R per month from 2025-09 (W3): + +| Variant | 2025-09 | 10 | 11 | 12 | 2026-01 | 02 | 03 | 04 | 05 | 06 | 07 | 08 | 09 (to 09-05) | +|---|---|---|---|---|---|---|---|---|---|---|---|---|---| +| control | +14.7 | −1.4 | +3.7 | +19.4 | +29.1 | −2.2 | +9.5 | +4.4 | +0.0 | +1.7 | −4.6 | +12.4 | +0.2 | +| both | +19.4 | +0.6 | +8.4 | +18.7 | +32.4 | −2.4 | +1.5 | +7.2 | −2.4 | +5.1 | +2.3 | +5.9 | +0.2 | + +Losing runs on M15: longest run of losing trades 9 → 8 (W3 and LONG); worst calendar month +−10.1R → −6.1R (W3, 2024-01) and −11.7R → −10.0R (LONG); losing months 11 → 8 of 37 on W3, +39 of 92 → 37 of 91 on LONG. + +### 5.4 Rule T, clause by clause + +**M15 `both` — PASS** + +| Clause | Result | Numbers (candidate vs control) | +|---|---|---| +| (a) W3 R ≥ 97% of control | PASS | +205.12 vs +195.79 (97% = +189.92) | +| (b) W3 PF ≥ control + 0.10 | PASS | 1.734 vs 1.488 | +| (c) W3 max DD ≤ control | PASS | 12.39 vs 20.87 | +| (d) LONG R ≥ control | PASS | +223.33 vs +192.53 | +| (d) LONG max DD ≤ control | PASS | 41.54 vs 64.77 | +| (d) 2021 + 2022 better | PASS | −24.12 vs −50.21 | +| (e) HOLDOUT R ≥ control − 0.5 | PASS | −4.62 (18 trades) vs −6.63 (24) | +| (f) (a) at $0.30 | PASS | +221.18 vs +216.53 (97% = +210.04) | +| (f) (b) at $0.30 | PASS | 1.809 vs 1.551 | +| (f) (c) at $0.30 | PASS | 11.44 vs 19.56 | +| (g) ≥ 3 of 4 neighbours pass (a) and (c) | PASS | 3 of 4 (table 5.5) | +| (h) ≥ 20 trades differ on W3 | PASS | 281 removed, 0 added | + +**M15 `buy` — FAIL**: (a) +188.21 < +189.92; (b) 1.523 < 1.588; the same two at $0.30. +**M15 `sell` — FAIL**: (d) LONG max DD 76.39 > 64.77 and 2021 + 2022 −60.29 < −50.21. It has the +best W3 total (+212.70R) and that is exactly the trap the long window was there to catch: in the +falling years the buys against the daily trend are the losers, and `sell` keeps them. +**M5 `both` — FAIL**: (c) W3 DD 15.20 > 10.11 (also at $0.30: 14.30 > 8.88); (d) LONG R ++127.47 < +128.10, LONG DD 25.25 > 18.39, 2021 + 2022 −8.02 < −3.91; (g) 0 of 4. +**M5 `buy` — FAIL**: (d) LONG R +123.37 < +128.10, LONG DD 27.14 > 18.39, 2021 + 2022 −9.13 < +−3.91; (g) 1 of 4. +**H1 `both` — FAIL** (a), (b). + +### 5.5 Clause (g): neighbouring definitions (research harness) + +The harness (K1 + K2 of the twin with the veto OFF, a host-side mask on the final candidates +before the cooldown, the shared resolver) reproduces the twin's `--set` books byte for byte for +the shipped definition, so it is the same thing as the shipped path for a final veto: + +| Book ($0.65) | Harness CSV MD5 | Twin CSV MD5 | +|---|---|---| +| M15 W3 control | `70195fb0e6865b3a46ee8a9aa8844e51` | same | +| M15 W3 `both` | `ad3b0dbda0bf77b659caea1c53be6a50` | same | +| M15 LONG control | `c222839878f791e980b2dbc99fb6dfbd` | same | +| M15 LONG `both` | `aa5253aab206fd21fa6a8d5ba25b75b5` | same | +| M5 W3 control / `both` / `buy` | `d20ec8a3…` / `39cc0f19…` / `631f25ac…` | same | +| M5 LONG control / `both` / `buy` | `3f0c091c…` / `9872c068…` / `613dcf0e…` | same | + +M15, veto in both directions, each neighbour replacing the definition of "clear daily trend": + +| Definition | W3 trades | W3 R | W3 PF | W3 DD | (a) | (c) | LONG trades | LONG R | LONG PF | LONG DD | +|---|---|---|---|---|---|---|---|---|---|---| +| control | 1105 | +195.79 | 1.49 | 20.87 | (97% = +189.92) | | 2091 | +192.53 | 1.24 | 64.77 | +| **shipped: 50-day average AND 60-day change** | 824 | +205.12 | 1.73 | 12.39 | PASS | PASS | 1473 | +223.33 | 1.42 | 41.54 | +| 60-day change alone | 752 | +188.25 | 1.73 | 11.89 | FAIL | PASS | 1328 | +228.11 | 1.49 | 27.30 | +| 90-day change alone | 782 | +196.14 | 1.72 | 12.46 | PASS | PASS | 1412 | +228.75 | 1.46 | 37.36 | +| 120-day change alone | 781 | +199.36 | 1.75 | 12.26 | PASS | PASS | 1397 | +233.19 | 1.47 | 34.79 | +| 100-day average alone | 796 | +201.23 | 1.74 | 12.26 | PASS | PASS | 1375 | +227.30 | 1.47 | 32.56 | + +3 of 4 pass. Every neighbour beats the control on PF and on drawdown in both windows; the one +miss is 1.7R under the 97% line. It is a plateau. (Each neighbour has a lower LONG drawdown than +the shipped definition; switching to one now would be the rescue the rule forbids.) + +M5: `both` 0 of 4 (every neighbour keeps the 15.2R drawdown), `buy` 1 of 4 (90-day, 120-day and +100-day average all fall under the 97% line). No plateau on M5. + +### 5.6 What the veto removes (M15, $0.65) + +Zero trades are added in any window (W3 and LONG, every variant, both timeframes): the setups +inside a vetoed setup's 5-bar cooldown point the same way under the same daily state and are +vetoed with it, so the cooldown displaces nothing. + +| Slice | W3 trades | W3 R | W3 mean R | LONG trades | LONG R | LONG mean R | +|---|---|---|---|---|---|---| +| all removed | 281 | −9.32 | −0.033 | 618 | −30.80 | −0.050 | +| BUY against a falling daily trend | 106 | +7.59 | +0.072 | 329 | −22.10 | −0.067 | +| SELL against a rising daily trend | 175 | −16.91 | −0.097 | 289 | −8.70 | −0.030 | +| removed, grade B | 210 | −11.53 | −0.055 | 408 | −25.69 | −0.063 | +| removed, grade A | 46 | +1.28 | +0.028 | 164 | −1.12 | −0.007 | +| removed, grade A+ | 14 | +4.26 | +0.304 | 26 | +1.81 | +0.069 | +| removed, grade C | 11 | −3.34 | −0.304 | 20 | −5.80 | −0.290 | +| kept (the new book) | 824 | +205.12 | +0.249 | 1473 | +223.33 | +0.152 | +| kept BUY | 763 | +194.20 | +0.255 | 1302 | +222.50 | +0.171 | +| kept SELL | 61 | +10.91 | +0.179 | 171 | +0.83 | +0.005 | + +Reading: the removed trades are close to break-even after costs (−0.03 to −0.05R each) while the +kept trades earn +0.15 to +0.25R each. The veto does not find big losers; it drops about a quarter +of the trades, the ones with no edge, and with them the long flat-to-down stretches. The two +halves of the veto are carried by different windows: the SELL half by the rising 3-year window, +the BUY half by the falling years of the long window. Only the symmetric rule holds on both. + +Share of trades that reach +1R before −1R (same entries, fixed 1R target, `--fixed-rr-tp 1.0`, +$0.65): M15 W3 58.6% → 61.4% (1105 → 824 trades), M15 LONG 55.9% → 58.2%; M5 W3 56.8% → 58.2%, +M5 LONG 55.3% → 56.1%. + +Checklist breakdowns, M15 W3 → with the veto (trades / R), $0.65: + +| Slice | Control | Veto | Change in R | +|---|---|---|---| +| regime ranging | 140 / +46.75 | 106 / +49.23 | +2.48 | +| regime transitioning | 264 / +47.05 | 199 / +49.47 | +2.42 | +| regime trending_down | 165 / −4.39 | 43 / +9.71 | +14.10 | +| regime trending_up | 536 / +106.38 | 476 / +96.70 | −9.68 | +| volatility extreme | 508 / +103.91 | 361 / +95.86 | −8.05 | +| volatility high | 228 / +4.00 | 135 / +18.32 | +14.32 | +| volatility normal | 202 / +58.96 | 177 / +57.01 | −1.96 | +| volatility low | 167 / +28.92 | 151 / +33.94 | +5.01 | +| BUY | 869 / +201.79 | 763 / +194.20 | −7.59 | +| SELL | 236 / −6.00 | 61 / +10.91 | +16.91 | +| setup GENERIC | 1004 / +184.73 | 757 / +192.15 | +7.42 | +| setup XAU_ATR | 101 / +11.07 | 67 / +12.97 | +1.90 | +| Monday | 188 / +36.54 | 145 / +52.46 | +15.92 | +| Tuesday | 240 / +44.89 | 186 / +41.82 | −3.07 | +| Wednesday | 234 / +12.08 | 165 / +13.36 | +1.28 | +| Thursday | 216 / +63.84 | 157 / +70.36 | +6.51 | +| Friday | 214 / +33.51 | 162 / +21.04 | −12.47 | +| Sunday evening | 13 / +4.94 | 9 / +6.08 | +1.14 | + +By entry hour (UTC), change in R on W3 / on LONG: 03 +0.3 / +1.5; 04 −2.1 / −0.1; 05 −0.2 / +2.6; +06 +1.9 / −1.5; 07 +2.9 / +0.9; 08 −5.3 / −0.3; 09 −6.6 / −4.0; 12 −2.9 / +2.7; 14 +11.8 / +12.9; +15 −4.0 / +1.8; 17 −0.5 / −2.4; 18 +11.4 / +11.3; 19 +0.9 / +2.7; 20 +0.8 / +2.9; 22 +0.8 / −0.1. +The weekday and hour changes are not stable in sign between the two windows except hours 14 and 18 +(positive in both) and hour 9 (negative in both); none of it is a reason to act. + +### 5.7 Verdict T + +- **M15: `both` PASSES every clause and SHIPS** (`pair_signals/xauusd.py`, M15 override 7). +- **M5: no veto.** `both` deepens the worst losing run (10.1R → 15.2R) and loses the long window; + `buy` passes the 3-year window and fails the long one, with no plateau. The M5 book earns + almost all of its R in 2025–2026 and the daily trend adds nothing reliable to it. +- **H1 / H4: no veto.** H1 loses 10R of 70R; H4 already confirms on the daily chart. +- On the timeframes without the veto the daily trend is shown on the alert as information only. + +What the owner gives up, as agreed: in the purely rising stretch the veto costs R. W2 first half +(2024-09 → 2025-09) +61.83R vs +70.10R; the RISING window +63.78R vs +68.50R; W2 at $0.30 ++166.22R vs +167.53R. What it buys: the worst losing run on the 3-year window falls from 20.9R to +12.4R, on the 7.6-year window from 64.8R to 41.5R, 2022 turns from −24.6R to +3.2R, and the three +weeks after the test window (HOLDOUT, never seen by any tuning) lose 4.6R instead of 6.6R. 2021 +does not improve (−27.3R vs −25.6R) and 2023 is worse (+1.8R vs +11.7R): in a year that chops +sideways the daily trend flips late and the veto cannot help. + +### 5.8 Candidate C — wide-stop cap on M15 (post-T control) + +Control = the factory configuration after the M15 flip (daily-trend veto on). Command per row: +`venv/bin/python backtest/cudaBasedBacktesting.py --pair XAUUSD --primary-tf 15min --start +--end --spread <0.65|0.30> [--set xau_strategy.wide_stop_cap_atr=] --export-csv `. +Paired proof on this control: W3, K = 6, `gpu_backtest.py --workers 12` and the twin both give +`f94f1429fb4c9172427ea34c2f4d92a1` (824 / +216.87R / DD 11.37). + +| Window | Cost | K | Trades | Total R | Win rate | PF | Max DD | First half | Second half | MD5 (8) | +|---|---|---|---|---|---|---|---|---|---|---| +| W3 | $0.65 | off | 824 | +205.12 | 63.1% | 1.73 | 12.4 | +76.72 | +128.40 | ad3b0dbd | +| W3 | $0.65 | 5 | 824 | +211.26 | 61.4% | 1.74 | 15.0 | +81.77 | +129.48 | 055d72a4 | +| W3 | $0.65 | **6** | 824 | +216.87 | 62.9% | 1.78 | 11.4 | +80.67 | +136.20 | f94f1429 | +| W3 | $0.65 | 7 | 824 | +213.15 | 63.1% | 1.76 | 11.8 | +79.08 | +134.07 | 5ea4ab4b | +| W3 | $0.65 | 8 | 824 | +208.01 | 62.9% | 1.74 | 12.3 | +77.18 | +130.84 | 372ac298 | +| W3 | $0.30 | off | 824 | +221.18 | 63.6% | 1.81 | 11.4 | +86.41 | +134.77 | 96a87454 | +| W3 | $0.30 | 5 | 824 | +228.46 | 62.1% | 1.82 | 14.0 | +92.27 | +136.18 | 7453b5f7 | +| W3 | $0.30 | **6** | 824 | +233.37 | 63.6% | 1.86 | 11.0 | +90.67 | +142.70 | bf7e4157 | +| W3 | $0.30 | 7 | 824 | +229.37 | 63.7% | 1.84 | 11.0 | +88.88 | +140.48 | e44cb5b5 | +| W3 | $0.30 | 8 | 824 | +224.12 | 63.5% | 1.82 | 11.3 | +86.91 | +137.22 | c214455b | +| W2 | $0.65 | off | 597 | +155.89 | 64.3% | 1.78 | 11.4 | +61.83 | +94.06 | dfc647ee | +| W2 | $0.65 | 5 | 597 | +156.59 | 63.3% | 1.79 | 11.4 | +62.63 | +93.96 | 5e5c0207 | +| W2 | $0.65 | **6** | 597 | +160.79 | 64.0% | 1.81 | 11.4 | +61.31 | +99.48 | 3614c98a | +| W2 | $0.65 | 7 | 597 | +158.70 | 64.3% | 1.79 | 11.4 | +60.19 | +98.51 | afa7eb01 | +| W2 | $0.65 | 8 | 597 | +156.58 | 64.2% | 1.78 | 11.4 | +60.54 | +96.04 | ba29cc41 | +| RISING | $0.65 | off | 258 | +63.78 | 64.3% | 1.80 | 9.1 | +23.97 | +39.81 | e87b6de1 | +| RISING | $0.65 | 5 | 258 | +57.68 | 61.6% | 1.71 | 8.0 | +22.85 | +34.83 | 295731a3 | +| RISING | $0.65 | **6** | 258 | +58.03 | 64.0% | 1.72 | 8.6 | +23.84 | +34.19 | 8dba71d7 | +| RISING | $0.65 | 7 | 258 | +62.18 | 64.3% | 1.78 | 8.8 | +26.44 | +35.74 | c461380f | +| RISING | $0.65 | 8 | 258 | +62.12 | 63.6% | 1.77 | 9.1 | +24.06 | +38.07 | fce21577 | +| LONG | $0.65 | off | 1473 | +223.33 | 59.7% | 1.42 | 41.5 | | | aa5253aa | +| LONG | $0.65 | 5 | 1473 | +204.76 | 58.4% | 1.38 | 53.1 | | | b365f28f | +| LONG | $0.65 | **6** | 1473 | +233.56 | 59.7% | 1.44 | 47.2 | | | 4aa12918 | +| LONG | $0.65 | 7 | 1473 | +228.83 | 59.8% | 1.43 | 46.5 | | | 6e91e0dd | +| LONG | $0.65 | 8 | 1473 | +223.79 | 59.7% | 1.42 | 43.2 | | | 36267a43 | +| LONG | $0.30 | off | 1473 | +257.25 | 60.8% | 1.50 | 37.2 | | | 1835b341 | +| LONG | $0.30 | 5 | 1473 | +241.64 | 59.8% | 1.46 | 44.9 | | | 6a346020 | +| LONG | $0.30 | **6** | 1473 | +268.63 | 60.8% | 1.52 | 41.6 | | | 526ffa4a | +| LONG | $0.30 | 7 | 1473 | +263.16 | 60.9% | 1.51 | 42.1 | | | 3f42a60e | +| LONG | $0.30 | 8 | 1473 | +257.84 | 60.8% | 1.50 | 38.9 | | | ffbba3e4 | +| HOLDOUT | $0.65 | off, 6, 7, 8 | 18 | −4.62 | 50.0% | 0.50 | 4.8 | | | fa50bcd0 | +| HOLDOUT | $0.65 | 5 | 18 | −4.62 | 50.0% | 0.50 | 4.8 | | | 3ff32a0d | + +(W2 and RISING at $0.30 and the HOLDOUT at $0.30 are in the same direction as the $0.65 rows: +W2 +171.35 vs +166.22 for K = 6; RISING +64.66 vs +70.24; HOLDOUT unchanged.) + +LONG per calendar year ($0.65), total R (trades are identical: 88 / 207 / 121 / 149 / 163 / 275 / +341 / 129): + +| K | 2019 | 2020 | 2021 | 2022 | 2023 | 2024 | 2025 | 2026 | +|---|---|---|---|---|---|---|---|---| +| off | +14.0 | +45.1 | −27.3 | +3.2 | +1.8 | +53.5 | +92.5 | +40.6 | +| 5 | +19.0 | +27.9 | −32.4 | −1.0 | +2.5 | +63.1 | +89.2 | +36.6 | +| 6 | +19.5 | +46.7 | −31.8 | +1.8 | +4.7 | +59.8 | +88.1 | +44.8 | +| 7 | +16.5 | +45.4 | −32.2 | +4.3 | +1.8 | +59.2 | +91.1 | +42.7 | +| 8 | +14.5 | +43.9 | −29.0 | +3.4 | +2.9 | +54.4 | +92.3 | +41.4 | + +Trades the cap touches (the trade set is identical by construction; only stops, and through them +results, move). All of them are swing (GENERIC) setups; the gold ATR setup is never capped: + +| Window | K | Stops changed | R of those trades: control → capped | +|---|---|---|---| +| W3 | 5 / 6 / 7 / 8 | 281 / 168 / 89 / 44 | +66.4 → +72.5 / +49.2 → +60.9 / +25.6 → +33.6 / +12.2 → +15.1 | +| LONG | 5 / 6 / 7 / 8 | 597 / 350 / 170 / 83 | +97.5 → +78.9 / +56.7 → +66.9 / +37.0 → +42.5 / +12.5 → +13.0 | +| RISING | 5 / 6 / 7 / 8 | 97 / 58 / 25 / 12 | +28.1 → +22.0 / +21.0 → +15.3 / +13.1 → +11.5 / +5.7 → +4.0 | +| HOLDOUT | 5 / 6 / 7 / 8 | 3 / 0 / 0 / 0 | −3.03 → −3.04 | + +**Rule C, K = 6** + +| Clause | Result | Numbers | +|---|---|---| +| W3 change in R ≥ +5R | PASS | +11.75 (+216.87 vs +205.12) | +| W3 first half ≥ control − 1R | PASS | +80.67 vs +76.72 | +| W3 second half ≥ control − 1R | PASS | +136.20 vs +128.40 | +| W3 PF ≥ control − 0.02 | PASS | 1.777 vs 1.734 | +| W3 max DD ≤ control + 10% | PASS | 11.37 vs 12.39 | +| LONG total R ≥ control | PASS | +233.56 vs +223.33 (but max DD 47.19 vs 41.54) | +| same sign at $0.30 | PASS | +12.19 (+233.37 vs +221.18) | +| ≥ 20 trades differ | PASS | 168 stops and results changed | +| neighbour K = 5 not negative on W3 | PASS | +6.14 | +| **neighbour K = 5 not negative on LONG** | **FAIL** | **−18.57** | +| neighbour K = 7 not negative on W3 | PASS | +8.04 | +| neighbour K = 7 not negative on LONG | PASS | +5.50 | + +**Verdict C: FAIL. Nothing ships; `wide_stop_cap_atr` stays 0.0 (research knob only).** K = 6 is a +narrow peak, not a plateau: one step tighter loses 18.6R over the long window (2020 alone −17R), +one step wider keeps half the gain, two steps wider keeps none. The cap also costs R in the rising +window at every K (−1.6R to −6.1R), deepens the long-window drawdown at every K (41.5R → 43.2R to +53.1R) and changes nothing in the unseen three weeks. This agrees with the planning-phase screen +of 25 stop and exit variants: the stop geometry is at a plateau, and tightening it is the one +direction that reliably loses. No second K is tried (the rule's no-rescue clause). + +### 5.9 What shipped, and the re-pinned books + +Shipped: `pair_signals/xauusd.py` M15 override 7, `config.xau_strategy.daily_trend_veto = "both"` +on M15 only. Not shipped: any veto on M5 / H1 / H4; the wide-stop cap (knob kept at 0.0). + +Reference books after the flip, factory configuration, `--spread 0.65`, regenerated with BOTH +scripts (`gpu_backtest.py --workers 12` and `cudaBasedBacktesting.py`), `Backend: CuPy GPU +(NVIDIA GeForce RTX 4070 SUPER, cupy 14.0.1)`, `GPU cache fallbacks: 0`: + +| Book | Window | Trades / R / PF / DD | MD5 (reference = twin) | | +|---|---|---|---|---| +| M5 W2 | 2024-09-05 → 2026-09-05 | 781 / +146.07 / 1.52 / 10.1 | `e599c8a8652f95b7fcdf5a7ef2b564f5` | unchanged | +| M15 W2 | 2024-09-05 → 2026-09-05 | 597 / +155.89 / 1.78 / 11.4 | `dfc647ee02afd179dd9e031f344ed380` | re-pinned (was `5b65ecf6…`, 813 / +154.15 / 1.53) | +| M15 OOS | 2024-06-01 → 2025-03-30 | 258 / +63.78 / 1.80 / 9.1 | `e87b6de12f90f3b5a5d41a953ee9ff7a` | re-pinned (was `09b8a8e7…`, 313 / +68.50 / 1.67) | +| H1 W3 | 2023-09-05 → 2026-09-05 | 253 / +70.02 / 1.98 / 4.4 | `a44fd19b4067f46e5bd98d50cc3eff92` | unchanged | +| H4 W3 | 2023-09-05 → 2026-09-05 | 114 / +21.7 / 1.59 | `f3587bae3c5dc3eda468ea3588b17dcb` | unchanged | +| M15 W3 | 2023-09-05 → 2026-09-05 | 824 / +205.12 / 1.73 / 12.4 | `ad3b0dbda0bf77b659caea1c53be6a50` | was `70195fb0…`, 1105 / +195.79 / 1.49 | + +The factory books are byte-identical to the `--set xau_strategy.daily_trend_veto=both` books of +section 5.2, as they must be. + +### 5.10 A fetch bug found on the way (fixed, not a signal change) + +The first HOLDOUT and RISING runs with the daily frame failed with `OandaDataError: XAU/USD 1d: +3 consecutive failures fetching window 2020-08-01–2020-08-01 … (Empty candle response)`. Since +2026-09-13 `fetch_range` retries a failing page and aborts after three tries, while the Oanda +provider raised on an EMPTY date window. A padded window that starts on a closed day (the daily +pad is 1,400 days = exactly 200 weeks, so a window starting on a Saturday pads to a Saturday) +begins with a page nothing traded in, and the whole fetch aborted. Fix in `providers/oanda.py`: +a DATE-WINDOW request with no candles returns an empty frame (a `count` request, the live path, +still raises); two Module 15 tests. Books are unaffected (every pinned window was already cached; +the ten MD5s reproduce). New cache files fetched for this campaign: the daily frames +`oanda_XAU_USD_1d_20201105_20260905`, `_20150303_20260807`, `_20200801_20250330`, +`_20221105_20261001`, and the HOLDOUT window `5min` / `15min` / `1h` `_20260705_20261001` and +`4h_20260114_20261001`. The two interior gaps the HOLDOUT fetch logged are market closures +(Labor Day 2026-09-07, Good Friday 2026-04-02), not dropped pages. + +### 5.11 Setup size: what the evidence supports, and the display lines that shipped + +"Trade setup size" was tested two ways. + +**The stop itself.** Candidate C (section 5.8) fails its rule, and the planning-phase screen of +25 stop and exit variants found nothing better than the shipped geometry (a flat 2 × ATR stop on +M15 turns +196R into +3R). The stops stay as they are. + +**How much is at risk at once.** If every alert is taken, the plans pile up. Shipped books, W3, +$0.65 (`M15` = veto on): + +| Books taken together | Trades | Same-direction plans already open at an entry: mean | median | max | Entries with ≥ 1 open | with ≥ 3 open | Open at once while anything is open: mean | max | Share of the time something is open | +|---|---|---|---|---|---|---|---|---|---| +| M15 alone | 824 | 1.17 | 1 | 6 | 62% | 15% | 1.70 | 7 | 41% | +| M5 alone | 870 | 0.69 | 0 | 6 | 40% | 7% | 1.36 | 7 | 23% | +| M5 + M15 | 1694 | 1.70 | 1 | 12 | 66% | 26% | 1.97 | 13 | 51% | +| M5 + M15 + H1 | 1947 | 2.75 | 2 | 15 | 78% | 43% | 2.60 | 16 | 65% | + +The stacked entries are not worse trades (M5 + M15, by the number of same-direction plans already +open at the entry: 0 → +0.154R per trade over 577; 1 → +0.218R over 410; 2 → +0.241R over 271; +3–4 → +0.282R over 267; 5 or more → +0.236R over 169). So there is no evidence for skipping them, +and the 2026-09-06 sizing campaign already showed that varying the stake by signal features does +not pay. What the numbers do say is that "1% per alert" is in practice several percent on one move +of gold. That is a sizing fact for the reader, so it ships as information, not as a gate: + +- **Size line** (`exposure.py`, off until `config.json` `trade_size.risk_per_trade` is set): the + number of lots that keeps the loss at the printed stop inside the owner's risk limit, rounded + DOWN to the lot step. Private by default (owner chats and the CLI); the channel and the public + dashboard get it only with `trade_size.telegram_channel` / `trade_size.dashboard`. +- **Already-open line**: how many ALERTED plans in the same direction are still open in the + advisory outcome model (`signal_log.open_alerted_setups`, a read-only query; the bot stamps + `alerted_at` on a row once its alert reached a chat). +- **Daily-trend line** on every surface (`main.daily_trend_words`): on M15 the veto has already + removed a plan against a clear daily trend; on M5 / H1 / H4, where the veto failed its rule, the + line is context ("Daily trend: falling — this buy goes against it"). The live wrapper stamps the + trend on every logged scan row (`signals.daily_trend`), so the live record can answer the M5 + question again after a few months. + +None of the three is read by a gate, the backtester or the twin. + +### 5.12 Final verification (2026-10-01) + +- `venv/bin/python test_all.py`: **607 passed, 0 failed** (40 modules: Module 39 daily-trend veto + and wide-stop cap, Module 40 the display lines, Module 37 extended, two Module 15 tests for the + empty-window fetch). +- The five reference books regenerated with both scripts after the last code change: ten MD5s + equal the table in section 5.9. +- Recorder and feature export with the veto on, M15 2024-06-01 → 2025-03-30, `--export-features + --record DIR --record-bars` on both scripts: feature CSV `732f396bb3ee6036b62750a8c755194e` on + both; `trades` 258 × 137, `indicators` 5900 × 15, `events` 5098 × 14 and `bars` 12315 × 172 + identical; both manifests carry book MD5 `e87b6de12f90f3b5a5d41a953ee9ff7a` and differ only in + `cli`, `created_at`, `replay_seconds`, `workers`; `replay_scorecard.py DIR --check`: all + invariants held. +- `agent_panel_base_rates.json` rebuilt from the re-pinned books (M15 timeframe row 597 trades, + 64.3% winners, +0.261R; was 813 / 62.2% / +0.190R). + +### 5.13 Independent verification (2026-10-01) + +An agent that had not done the work re-derived the change list from `git diff HEAD` and +re-ran the books itself (24 on the twin, 9 on the reference script, all offline, 0 cache +fallbacks). Every figure of sections 5.2 to 5.9 it was given matched to the cent, on both +scripts where it ran both: the M15 books with the veto on and off (W3, W2, the 2024-06-01 → +2025-03-30 window, LONG with its per-year R, the holdout), the unchanged M5 / H1 / H4 pins, the +failed M5, H1, `buy` and `sell` variants, and the removed-trade tables (281 removed on W3, 0 +added, all 63 columns of the 824 kept trades identical). Its own implementation of the rule +from closed daily bars predicts the removed trades exactly (281 / 281, 618 / 618, 55 / 55); the +same rule fed the still-forming day mis-predicts 52, 129 and 14 of them, so there is no +look-ahead. The scoring configuration is identical to HEAD on all four timeframes; the only +decision-side difference is the M15 veto. + +It also raised four points this report did not make clearly enough. They change no verdict; +they are recorded so the size of the claim is not overstated. + +1. **Clause (g) is thinner than "3 of 4" reads.** The "100-day average alone" neighbour of + table 5.5 is an EXPONENTIAL average, the same kind as the shipped 50-day one (harness key + `e100`). With a SIMPLE 100-day average the M15 W3 book is 781 / +188.68R / PF 1.70 / + drawdown 12.27R: it beats the control on profit factor and drawdown like the others, but + sits 1.2R under the 97% line (+189.92R), so with that reading the count would be 2 of 4. + The four definitions were fixed before the runs in the harness's own form and the clause + passes as written; the honest summary is "five alternative definitions, all better than the + control on profit factor and drawdown, two of them 1.2R and 1.7R short on total R". +2. **"The veto never adds a trade" is an observation, not a guarantee.** The rule itself only + removes a plan. In a book, removing an entry also moves the 5-bar cooldown, which could + free a later entry; that happened 0 times in every control / candidate pair compared (13). +3. **It is a risk filter, not a better direction call.** Nothing in how direction is scored + changed, and a vetoed bar still shows its buy or sell reading. The removed plans were close + to break-even (−0.03R each on W3, −0.05R on LONG; 55% winners against 63% for the kept + ones), so total R barely moves while profit factor and the losing runs improve. +4. **The help is concentrated.** Effect of the rule by year on LONG (shipped minus off): + 2019 +4.9R, 2020 −3.7R, 2021 −1.8R, 2022 **+27.8R**, 2023 **−10.0R**, 2024 +0.1R, 2025 + +11.4R, 2026 +1.9R; total +30.8R, of which 2022 (buys against a falling daily trend) is + nine tenths. On W3 the whole gain in R comes from sells removed in 2025, and the drawdown + cut from 20.9R to 12.4R is one episode: 13 sells between 2023-12-01 and 2024-02-23 + (−8.5R). In a year where the daily trend flips often (2023) the rule costs. The multi-year + windows had been looked at with the prototype before rule T was written (section 2), and + the only period nobody had seen is four weeks long and differs by six trades. + +Cross-checked here from the formal books (a filter on the control book reproduces the shipped +book and the exponential neighbour exactly, and gives the simple-average figure above). + +## 6. Summary of decisions + +| Item | Verdict | +|---|---| +| Daily-trend veto, M15, both directions | **PASSED rule T on every clause, SHIPPED** (factory M15 override 7) | +| Daily-trend veto, M15, buy-only / sell-only | FAILED (W3 R and PF / LONG drawdown and 2021–22) | +| Daily-trend veto, M5 (both, buy-only) | FAILED (W3 drawdown; LONG R, drawdown, 2021–22; no plateau) | +| Daily-trend veto, H1 | FAILED (W3 −9.7R); H4 not run | +| Wide-stop cap K = 6 on M15 | FAILED rule C (neighbour K = 5 loses 18.6R on LONG); knob kept at 0.0, research-only | +| Lot-size, already-open and daily-trend lines | SHIPPED, display only (books byte-identical) | +| Oanda empty date window | FIXED (fetch_range no longer aborts on a closed-day page) | +| Seven further context screens | no state nominated, nothing shipped (`context_screens_2026-10.md`) | +| Eight outside data sources (dollar, silver, bonds, real yield, fund flows, futures and Oanda client positioning) | none adds a forecast to gold's own daily trend; broad dollar index on M5 = WATCH; nothing shipped, no feed added (`new_inputs_2026-10.md`) | diff --git a/backtest/research/new_inputs_2026-10.md b/backtest/research/new_inputs_2026-10.md new file mode 100644 index 0000000..9e50915 --- /dev/null +++ b/backtest/research/new_inputs_2026-10.md @@ -0,0 +1,304 @@ +# New data sources for the direction call — a report-only study, 2026-10-01 + +Owner decision 2026-09-30: after the daily-trend work, study new data sources and report; nothing +ships from this file. The engine reads gold's own price and nothing else. The question here is +whether anything OUTSIDE gold's own chart tells the direction better than gold's chart already +does. Sections 1 to 3 were written before any data was fetched. + +## 1. Sources + +| # | Source | What it is | Where from | When a value is usable | +|---|---|---|---|---| +| 1 | Dollar | EUR/USD daily candles, read upside down (a falling EUR/USD is a rising dollar) | Oanda v20 candles, the live feed's own token | daily close + 0 (same rule as gold's daily bars) | +| 2 | Silver | XAG/USD daily candles | Oanda v20 | daily close | +| 3 | US 10-year note | USB10Y_USD daily candles (price up = yields down), if Oanda offers it | Oanda v20 | daily close | +| 4 | Real yield | US 10-year inflation-indexed yield, `DFII10` | FRED (public CSV) | the day's value from two days later (it is published the next US afternoon) | +| 5 | Broad dollar index | `DTWEXBGS` | FRED | same two-day rule | +| 6 | Gold fund holdings | SPDR Gold Shares, shares outstanding (each share is a fixed slice of the vault, so creations and redemptions are the fund's gold flow) | the fund manager's public history file (`navhist-us-en-gld.xlsx`); the older tonnes archive now serves a PDF | the day's value from two days later | +| 7 | Futures positioning | CFTC disaggregated report, gold (code 088691), managed-money net long | CFTC public reporting site | a Tuesday report from the following Monday | +| 8 | Oanda client positioning | position book for XAU_USD: the share of Oanda clients' open gold positions that are long (by count) | Oanda v20 `/v3/instruments/XAU_USD/positionBook`, one snapshot per day at 00:00 UTC | at once (the snapshot is taken at the daily close) | + +Everything fetched stays in the session scratchpad. Nothing is added to the repository's cache. + +Amendment before any analysis (2026-10-01, after the access probes and nothing else): the plan +expected source 8 to be gone. It is not: on the practice API the position book answers for every +date probed back to 2017, so it joins the study with the definition below. The probe printed ten +long-share readings (between 60% and 75% long, higher at higher gold prices) and no forward +returns; that is everything that was seen before the definition was fixed. Source 6 uses shares +outstanding instead of tonnes because that is the file that can still be downloaded. + +## 2. One state per source + +The same shape as the shipped daily trend, so the comparison is like for like. For a daily series +`x` (Sunday stub bars dropped for the Oanda candles): `up` when `x` is above its 50-day EMA AND +above its value 60 days earlier, `down` when below both, else `unclear`. Read for gold: + +| Source | Gold-friendly state | Gold-hostile state | +|---|---|---| +| Dollar (EUR/USD) | EUR/USD up (dollar falling) | EUR/USD down | +| Silver | up | down | +| 10-year note price | up (yields falling) | down | +| Real yield | down | up | +| Broad dollar index | down | up | +| Fund holdings | shares outstanding above the level 20 trading days earlier | below | +| Futures positioning | managed-money net long above its level 4 reports earlier | below | +| Oanda client positioning | long share `down` (the crowd is cutting longs or adding shorts) | long share `up` (the crowd is adding longs) | + +The positioning state is read the contrarian way on purpose and before looking: small traders +tend to buy dips and sell rallies, so a growing long share is expected to go with a falling price. +Test A is two-sided, so if the truth is the other way round it will show as a real spread with the +opposite sign and is reported as such; Test B is judged on the declared reading only. + +## 3. Two tests and the verdict rule + +**Test A — forward move of gold.** Every closed gold daily bar from 2019-01-01 to 2026-08-07. +Forward move = gold's close 5 days later minus today's close, in units of gold's own 14-day ATR +(and the same for 20 days). For each source: mean forward move on gold-friendly days minus mean +on gold-hostile days, (i) over all days and (ii) inside each of gold's own three daily-trend +states (rising / falling / unclear), in two eras (2019–2023, 2024–2026). The spread is called real +only if it has the same sign in both eras and its t-value, computed on every fifth day so the +5-day windows do not overlap, is at least 2 in each era. + +**Test B — the shipped books.** M15 LONG (daily-trend veto on, 1473 trades) and M5 LONG (1052 +trades), $0.65. Each trade is labelled `friendly` / `hostile` / `unclear` from the source state at +its decision time, mirrored for a SELL. A source is interesting on a timeframe only if its +`hostile` trades are at least 8% of the book and lose (total R < 0) in BOTH eras. + +**Verdict per source.** +- `GO` (worth a pre-registered campaign and the feed work it needs) only if Test A calls the + spread real INSIDE gold's own trend states (that is the "beyond gold's own chart" part) AND + Test B finds the hostile trades losing in both eras on at least one timeframe. +- `WATCH` if exactly one of the two holds. +- `NO-GO` otherwise, or when the source cannot be fetched reliably. +A `GO` is a ticket to a campaign, never a shipping decision: a live feed for a second instrument +needs its own provider path, cache, staleness rule and live == backtest proof. + +Two points of method were fixed while the analysis script was written, before it produced any +output. (1) The broad dollar index is published weekly (the H.10 release, on Monday, for the week +before), not daily, so the "two-day rule" of the table would have looked ahead: each week's final +state is used from the Tuesday after that week. (2) "Inside gold's own trend states" is computed +as one stratified number per era: the forward move minus the mean forward move of that day's gold +trend state (rising / unclear / falling), friendly days against hostile days, t-value on every +fifth day. That is the column the verdict reads; the unstratified spread is printed beside it. + +## 4. Results + +Run 2026-10-01. Fetched into the session scratchpad: Oanda daily candles for EUR_USD, XAG_USD, +USB10Y_USD and XAU_USD (2015-01-01 → 2026-09-30, about 3,650 bars each, about 600 Sunday stubs +dropped); the Oanda XAU_USD position book, one snapshot per day at 00:00 UTC from 2018-06-01 +(2,609 days asked, 97.7% answered at exactly 00:00, the rest from the last snapshot up to six +hours earlier, 45 days with none, all before mid-2022); FRED `DFII10` and `DTWEXBGS`; the gold +fund's history file (5,540 days since 2004); 1,059 weekly CFTC reports since 2006. + +### 4.1 What was fetched + +| Source | Days with a state, 2019-01-01 → 2026-08-07 | friendly / unclear / hostile | Same side as gold's own daily trend (days where both are clear) | +|---|---|---|---| +| 1 dollar (EUR/USD) | 1973 | 37% / 18% / 44% | 67% of 1363 | +| 2 silver | 1973 | 49% / 21% / 31% | 88% of 1387 | +| 3 10-year note price | 1973 | 39% / 16% / 46% | 73% of 1398 | +| 4 real yield (DFII10) | 1973 | 42% / 20% / 39% | 73% of 1353 | +| 5 broad dollar index | 1973 | 39% / 19% / 42% | 62% of 1362 | +| 6 gold fund shares | 1973 | 51% / 0% / 49% | 77% of 1639 | +| 7 futures positioning | 1973 | 51% / 0% / 49% | 59% of 1644 | +| 8 Oanda client positioning | 1973 | 37% / 15% / 48% | 36% of 1414 | + +Gold itself, same days: rising 58%, unclear 17%, falling 25%; mean 5-day forward move +0.27 ATR (rising +0.32, unclear +0.30, falling +0.14). + +### 4.2 Test A — gold's forward move, friendly days minus hostile days (in daily ATRs) + +| Source | Era | Days friendly / hostile | 5-day spread, all days (t) | 5-day spread inside gold's trend states (t) | 20-day spread inside gold's trend states (t) | Real? | +|---|---|---|---|---|---|---| +| 1 dollar (EUR/USD) | 2019–2023 | 443 / 623 | -0.04 (-0.3) | -0.05 (-0.4) | +0.35 (+0.7) | | +| 1 dollar (EUR/USD) | 2024–2026 | 296 / 249 | +0.21 (+0.7) | +0.20 (+0.7) | +0.80 (+0.5) | | +| 1 dollar (EUR/USD) | verdict | | | same sign in both eras: no; |t| ≥ 2 in both: no | | not real | +| 2 silver | 2019–2023 | 548 / 449 | -0.21 (-0.9) | -0.25 (-1.0) | -1.00 (-1.2) | | +| 2 silver | 2024–2026 | 411 / 160 | -0.10 (-0.7) | -0.20 (-0.7) | -0.64 (-0.2) | | +| 2 silver | verdict | | | same sign in both eras: yes; |t| ≥ 2 in both: no | | not real | +| 3 10-year note price | 2019–2023 | 515 / 603 | +0.08 (+0.3) | +0.04 (+0.1) | +0.46 (+0.5) | | +| 3 10-year note price | 2024–2026 | 253 / 295 | +0.23 (+0.4) | +0.17 (+0.4) | +1.02 (-0.2) | | +| 3 10-year note price | verdict | | | same sign in both eras: yes; |t| ≥ 2 in both: no | | not real | +| 4 real yield (DFII10) | 2019–2023 | 583 / 468 | +0.04 (+0.4) | +0.01 (+0.2) | -0.20 (-0.5) | | +| 4 real yield (DFII10) | 2024–2026 | 242 / 294 | +0.20 (+0.7) | +0.14 (+0.6) | +1.22 (+0.5) | | +| 4 real yield (DFII10) | verdict | | | same sign in both eras: yes; |t| ≥ 2 in both: no | | not real | +| 5 broad dollar index | 2019–2023 | 497 / 536 | +0.03 (-0.1) | +0.02 (-0.1) | -0.12 (-0.5) | | +| 5 broad dollar index | 2024–2026 | 269 / 293 | +0.02 (-0.2) | +0.04 (-0.0) | +0.71 (+0.1) | | +| 5 broad dollar index | verdict | | | same sign in both eras: yes; |t| ≥ 2 in both: no | | not real | +| 6 gold fund shares | 2019–2023 | 609 / 681 | +0.20 (+1.4) | +0.17 (+1.2) | +0.37 (+1.0) | | +| 6 gold fund shares | 2024–2026 | 393 / 281 | +0.40 (+1.7) | +0.36 (+1.8) | +1.09 (-0.1) | | +| 6 gold fund shares | verdict | | | same sign in both eras: yes; |t| ≥ 2 in both: no | | not real | +| 7 futures positioning | 2019–2023 | 640 / 656 | +0.06 (+0.0) | +0.04 (-0.1) | +1.03 (+0.6) | | +| 7 futures positioning | 2024–2026 | 369 / 308 | -0.01 (+0.3) | +0.11 (+0.8) | +1.23 (+1.7) | | +| 7 futures positioning | verdict | | | same sign in both eras: yes; |t| ≥ 2 in both: no | | not real | +| 8 Oanda client positioning | 2019–2023 | 636 / 428 | +0.20 (+1.2) | +0.19 (+1.1) | +0.53 (+0.1) | | +| 8 Oanda client positioning | 2024–2026 | 97 / 516 | -0.01 (-0.6) | +0.33 (+0.3) | n/a (too few friendly days) | | +| 8 Oanda client positioning | verdict | | | same sign in both eras: yes; |t| ≥ 2 in both: no | | not real | + +### 4.3 Test B — the shipped books, by the source's state at the trade's decision time + + +**M15 LONG (daily-trend veto on): 1473 trades, +223.3R** (era A = 2019–2023: 728 trades, +36.8R; era B = 2024–2026: 745 trades, +186.6R) + +| Source | Trade state | Trades (share) | Total R / mean | Era A R (trades) | Era B R (trades) | Hostile ≥ 8% and losing in both eras | +|---|---|---|---|---|---|---| +| 1 dollar (EUR/USD) | friendly | 725 (49%) | +127.3 / +0.176 | +32.0 (345) | +95.4 (380) | | +| 1 dollar (EUR/USD) | unclear | 252 (17%) | +13.0 / +0.052 | -15.3 (127) | +28.3 (125) | | +| 1 dollar (EUR/USD) | hostile | 496 (34%) | +83.0 / +0.167 | +20.1 (256) | +62.9 (240) | no | +| 2 silver | friendly | 937 (64%) | +154.9 / +0.165 | +26.8 (406) | +128.0 (531) | | +| 2 silver | unclear | 304 (21%) | +1.3 / +0.004 | -18.2 (196) | +19.5 (108) | | +| 2 silver | hostile | 232 (16%) | +67.1 / +0.289 | +28.1 (126) | +39.0 (106) | no | +| 3 10-year note price | friendly | 753 (51%) | +107.0 / +0.142 | +34.6 (397) | +72.4 (356) | | +| 3 10-year note price | unclear | 254 (17%) | +39.1 / +0.154 | +11.3 (121) | +27.8 (133) | | +| 3 10-year note price | hostile | 466 (32%) | +77.2 / +0.166 | -9.2 (210) | +86.4 (256) | no | +| 4 real yield (DFII10) | friendly | 723 (49%) | +97.7 / +0.135 | +45.5 (405) | +52.2 (318) | | +| 4 real yield (DFII10) | unclear | 313 (21%) | +30.8 / +0.098 | -12.5 (142) | +43.2 (171) | | +| 4 real yield (DFII10) | hostile | 437 (30%) | +94.9 / +0.217 | +3.8 (181) | +91.1 (256) | no | +| 5 broad dollar index | friendly | 647 (44%) | +122.5 / +0.189 | +37.5 (315) | +85.0 (332) | | +| 5 broad dollar index | unclear | 277 (19%) | +17.2 / +0.062 | -4.7 (168) | +21.8 (109) | | +| 5 broad dollar index | hostile | 549 (37%) | +83.7 / +0.152 | +3.9 (245) | +79.8 (304) | no | +| 6 gold fund shares | friendly | 1031 (70%) | +183.6 / +0.178 | +52.7 (519) | +130.9 (512) | | +| 6 gold fund shares | unclear | 7 (0%) | -4.7 / -0.668 | -2.9 (4) | -1.8 (3) | | +| 6 gold fund shares | hostile | 435 (30%) | +44.4 / +0.102 | -13.1 (205) | +57.5 (230) | no | +| 7 futures positioning | friendly | 789 (54%) | +115.5 / +0.146 | +22.3 (422) | +93.3 (367) | | +| 7 futures positioning | unclear | 0 (0%) | +0.0 / | +0.0 (0) | +0.0 (0) | | +| 7 futures positioning | hostile | 684 (46%) | +107.8 / +0.158 | +14.5 (306) | +93.3 (378) | no | +| 8 Oanda client positioning | friendly | 394 (27%) | +41.7 / +0.106 | +15.9 (326) | +25.9 (68) | | +| 8 Oanda client positioning | unclear | 198 (13%) | +26.9 / +0.136 | +13.7 (122) | +13.2 (76) | | +| 8 Oanda client positioning | hostile | 881 (60%) | +154.7 / +0.176 | +7.2 (280) | +147.5 (601) | no | + +**M5 LONG: 1052 trades, +128.1R** (era A = 2019–2023: 258 trades, -10.4R; era B = 2024–2026: 794 trades, +138.5R) + +| Source | Trade state | Trades (share) | Total R / mean | Era A R (trades) | Era B R (trades) | Hostile ≥ 8% and losing in both eras | +|---|---|---|---|---|---|---| +| 1 dollar (EUR/USD) | friendly | 439 (42%) | +85.3 / +0.194 | -11.5 (103) | +96.8 (336) | | +| 1 dollar (EUR/USD) | unclear | 211 (20%) | +28.6 / +0.136 | -0.9 (34) | +29.5 (177) | | +| 1 dollar (EUR/USD) | hostile | 402 (38%) | +14.2 / +0.035 | +2.0 (121) | +12.2 (281) | no | +| 2 silver | friendly | 556 (53%) | +89.3 / +0.161 | -7.6 (113) | +96.8 (443) | | +| 2 silver | unclear | 143 (14%) | +7.3 / +0.051 | -9.9 (45) | +17.2 (98) | | +| 2 silver | hostile | 353 (34%) | +31.5 / +0.089 | +7.1 (100) | +24.4 (253) | no | +| 3 10-year note price | friendly | 432 (41%) | +51.6 / +0.120 | -0.6 (119) | +52.2 (313) | | +| 3 10-year note price | unclear | 169 (16%) | -9.5 / -0.056 | -13.7 (49) | +4.2 (120) | | +| 3 10-year note price | hostile | 451 (43%) | +86.0 / +0.191 | +3.9 (90) | +82.1 (361) | no | +| 4 real yield (DFII10) | friendly | 377 (36%) | +4.2 / +0.011 | -15.0 (115) | +19.2 (262) | | +| 4 real yield (DFII10) | unclear | 186 (18%) | +32.3 / +0.174 | -7.0 (45) | +39.3 (141) | | +| 4 real yield (DFII10) | hostile | 489 (46%) | +91.6 / +0.187 | +11.6 (98) | +80.0 (391) | no | +| 5 broad dollar index | friendly | 447 (42%) | +131.2 / +0.294 | -0.7 (91) | +131.9 (356) | | +| 5 broad dollar index | unclear | 170 (16%) | +8.1 / +0.047 | -6.0 (40) | +14.1 (130) | | +| 5 broad dollar index | hostile | 435 (41%) | -11.2 / -0.026 | -3.7 (127) | -7.5 (308) | YES | +| 6 gold fund shares | friendly | 600 (57%) | +86.9 / +0.145 | -15.8 (142) | +102.7 (458) | | +| 6 gold fund shares | unclear | 3 (0%) | -1.8 / -0.608 | +0.0 (0) | -1.8 (3) | | +| 6 gold fund shares | hostile | 449 (43%) | +43.0 / +0.096 | +5.4 (116) | +37.6 (333) | no | +| 7 futures positioning | friendly | 493 (47%) | +41.1 / +0.083 | -17.0 (120) | +58.1 (373) | | +| 7 futures positioning | unclear | 0 (0%) | +0.0 / | +0.0 (0) | +0.0 (0) | | +| 7 futures positioning | hostile | 559 (53%) | +87.0 / +0.156 | +6.6 (138) | +80.4 (421) | no | +| 8 Oanda client positioning | friendly | 313 (30%) | +17.9 / +0.057 | -1.9 (99) | +19.8 (214) | | +| 8 Oanda client positioning | unclear | 92 (9%) | +2.3 / +0.025 | +0.5 (40) | +1.9 (52) | | +| 8 Oanda client positioning | hostile | 647 (62%) | +107.9 / +0.167 | -8.9 (119) | +116.9 (528) | no | + +### 4.4 Verdict per source + +| Source | Test A: real spread beyond gold's own trend | Test B: hostile trades ≥ 8% and losing in both eras (M15 / M5) | Verdict | +|---|---|---|---| +| 1 dollar (EUR/USD) | no | no / no | NO-GO | +| 2 silver | no | no / no | NO-GO | +| 3 10-year note price | no | no / no | NO-GO | +| 4 real yield (DFII10) | no | no / no | NO-GO | +| 5 broad dollar index | no | no / yes | WATCH | +| 6 gold fund shares | no | no / no | NO-GO | +| 7 futures positioning | no | no / no | NO-GO | +| 8 Oanda client positioning | no | no / no | NO-GO | + +### 4.5 The one flag, looked at more closely (after the fact: description, not a test) + +**M5 LONG: gold's own daily trend (rows) x 5 broad dollar index (columns)** — total R (trades) + +| Gold's own daily trend vs the trade | dollar friendly | dollar unclear | dollar hostile | +|---|---|---|---| +| with | +118.8 (352) | +1.9 (105) | -0.1 (215) | +| unclear | +6.8 (33) | -2.1 (12) | +2.3 (73) | +| against | +5.6 (62) | +8.3 (53) | -13.3 (147) | + +**M5 LONG: gold's own daily trend (rows) x 1 dollar (EUR/USD) (columns)** — total R (trades) + +| Gold's own daily trend vs the trade | dollar friendly | dollar unclear | dollar hostile | +|---|---|---|---| +| with | +76.8 (372) | +23.0 (109) | +20.7 (191) | +| unclear | +4.5 (37) | -5.5 (19) | +7.9 (62) | +| against | +4.0 (30) | +11.1 (83) | -14.4 (149) | + +**M5 LONG: broad-dollar hostile trades by year** — 2019: +1.4 (14); 2020: -3.4 (45); 2021: -0.6 (21); 2022: -1.1 (36); 2023: -0.1 (11); 2024: +3.5 (68); 2025: -8.5 (107); 2026: -2.4 (133) +**M5 LONG: broad-dollar friendly trades by year** — 2019: -1.5 (3); 2020: +0.0 (45); 2021: -0.0 (9); 2022: +0.0 (7); 2023: +0.7 (27); 2024: -0.1 (19); 2025: +40.3 (160); 2026: +91.7 (177) + W3 friendly: 397 trades (46%), +145.1R, halves +8.2 / +136.9 + W3 unclear: 150 trades (17%), +15.6R, halves -3.0 / +18.7 + W3 hostile: 323 trades (37%), -7.1R, halves +11.6 / -18.7 + +**M15 LONG (veto on): gold's own daily trend (rows) x 5 broad dollar index (columns)** — total R (trades) + +| Gold's own daily trend vs the trade | dollar friendly | dollar unclear | dollar hostile | +|---|---|---|---| +| with | +93.2 (561) | +48.0 (196) | +68.5 (416) | +| unclear | +29.3 (86) | -30.8 (81) | +15.2 (133) | +| against | +0.0 (0) | +0.0 (0) | +0.0 (0) | + +**M15 LONG (veto on): gold's own daily trend (rows) x 1 dollar (EUR/USD) (columns)** — total R (trades) + +| Gold's own daily trend vs the trade | dollar friendly | dollar unclear | dollar hostile | +|---|---|---|---| +| with | +99.4 (635) | +41.8 (176) | +68.5 (362) | +| unclear | +27.9 (90) | -28.8 (76) | +14.5 (134) | +| against | +0.0 (0) | +0.0 (0) | +0.0 (0) | + +**M15 LONG (veto on): broad-dollar hostile trades by year** — 2019: +12.5 (57); 2020: +11.5 (74); 2021: -6.7 (44); 2022: -9.2 (43); 2023: -4.1 (27); 2024: +65.1 (179); 2025: +9.9 (95); 2026: +4.8 (30) +**M15 LONG (veto on): broad-dollar friendly trades by year** — 2019: +4.7 (16); 2020: +25.0 (114); 2021: -8.9 (46); 2022: +8.2 (51); 2023: +8.5 (88); 2024: -4.4 (56); 2025: +52.1 (192); 2026: +37.3 (84) + W3 friendly: 373 trades (45%), +103.5R, halves +4.0 / +99.4 + W3 unclear: 130 trades (16%), +28.2R, halves +6.2 / +22.1 + W3 hostile: 321 trades (39%), +73.4R, halves +66.5 / +6.9 + +### 4.6 Reading + +- **Nothing outside gold's own chart passes Test A.** For all eight sources the 5-day forward move + of gold on "friendly" days is not reliably different from "hostile" days once gold's own daily + trend is accounted for (largest t-values 1.2 and 1.8, the gold fund's share count; everything + else is under 1.2 in at least one era). The familiar stories (a falling dollar, falling real + yields, rising silver, fund inflows) describe gold's past move well: they sit on the same side + as gold's own daily trend on 59% to 88% of clear days. They do not add a forecast to it. +- **Oanda client positioning is available but not useful in this form.** The plan expected the + endpoint to be gone; it answers, with history back to 2017. Read the contrarian way (a growing + long share is hostile), the spread has the expected sign and no weight (t 1.1 and 0.3), and in + 2024–2026 the crowd was adding longs on 516 of 613 clear days while gold rose. The trades taken + on those "hostile" days are the best part of both books. Only the daily state was tested; the + intraday snapshots (one per hour) were not. +- **The one flag is the broad dollar index on M5** (`WATCH`): the 435 M5 trades taken against the + dollar's trend lost 11.2R in total and lost in both eras, while the 447 taken with it made + +131.2R. The closer look shows what that is. All of the M5 profit sits in one cell: trades with + gold's own daily trend AND with the dollar's trend (+118.8R over 352 trades); the same trades + against the dollar's trend made nothing (−0.1R over 215). But every R of the friendly side was + earned in 2025 and 2026 (+40.3R and +91.7R), that is inside the one long rally from 2,600 to the + March 2026 peak (in 2025 the broad dollar fell 7.5% while gold rose 65%); before 2025 the + dollar's state separates nothing on M5 because the M5 book earned nothing at all. In the 3-year + window the hostile trades are positive in the first half (+11.6R) and negative only in the + second (−18.7R). On M15 the same state separates nothing: the hostile + trades made +65R in 2024 alone, a year when gold and the dollar rose together. The timelier + dollar measure (EUR/USD) shows the same lean on M5 without losing in both eras. So this is one + market phase seen from another angle, not a second source of information; it stays a + hypothesis, and it is the same picture as the context screens: M5 pays only when everything + agrees and has paid only since 2025. + +### 4.7 Verdict + +| Source | Verdict | Why | +|---|---|---| +| Dollar via EUR/USD | NO-GO | no forward spread; hostile trades are not losers in both eras | +| Silver | NO-GO | 88% the same information as gold's own trend; the spread has the wrong sign | +| US 10-year note | NO-GO | no forward spread | +| Real yield | NO-GO | no forward spread; on M5 the "hostile" trades are the better ones | +| Broad dollar index | WATCH | M5 trades against it lose in both eras, but the forward test finds nothing and the effect is the 2025–2026 phase; weekly data, four to ten days old when used | +| Gold fund share count | NO-GO | the nearest miss in the forward test (t 1.2 and 1.8), nothing on the books | +| Futures positioning | NO-GO | no forward spread, no difference on the books | +| Oanda client positioning | NO-GO in this daily form | available, but the contrarian reading failed in the rising years | + +**Nothing ships and no feed is added.** The direction call was improved by the one input that +passed a formal test, gold's own daily chart on M15. If the M5 question is taken up again, the +honest test is forward: `signals.daily_trend` is now logged on every scan, so a few months of live +M5 rows can be split by the daily trend, and by the dollar's trend after the fact, without touching +the signal code. diff --git a/config.json.example b/config.json.example index a831a20..25aab3a 100644 --- a/config.json.example +++ b/config.json.example @@ -71,5 +71,12 @@ "mode": "off", "deepseek_audit_per_day": 0 }, + "trade_size": { + "risk_per_trade": 0, + "contract_size": 100, + "lot_step": 0.01, + "telegram_channel": false, + "dashboard": false + }, "log_level": "INFO" } diff --git a/config.py b/config.py index 965729f..283bf57 100755 --- a/config.py +++ b/config.py @@ -325,6 +325,29 @@ class XauStrategyConfig: big_candle_atr_multiple: float = 0.0 big_candle_lookback_hours: float = 24.0 + # ── Daily-trend veto (2026-09-30) ────────────────────────────── + # The engine sees at most the H1 chart, and on M15 trades against a + # CLEAR daily trend earn nothing after costs. The daily trend is +1 / + # -1 when the last closed D1 close sits above / below its 50-bar EMA + # AND above / below the close 60 daily bars earlier, else 0 + # (signal_aggregator.daily_trend_series; Sunday stub bars dropped). + # "both" removes any finished setup that goes against a non-zero + # daily trend, "buy" only BUYs against a falling one, "sell" only + # SELLs against a rising one, "off" nothing. Off by default; the M15 + # factory sets "both" (override 7). It FAILED its rule on M5 and H1 + # and as "buy" / "sell" on M15 + # (backtest/research/daily_trend_campaign_2026-09-30.md). + daily_trend_veto: str = "off" + + # ── Wide-stop cap (2026-09-30, research knob) ────────────────── + # Caps the stop distance of the swing (GENERIC) and chart-pattern + # setups at max(K x ATR, the setup's own stop floor). Applied AFTER + # the min-RR and spread gates, like the printed-target floor, so + # the set of trades cannot change. 0.0 = off, and it ships off + # everywhere: K = 6 FAILED its rule on M15 (a narrow peak; K = 5 + # loses 18.6R over 2019-2026). Same campaign report. + wide_stop_cap_atr: float = 0.0 + @dataclass class FilterConfig: diff --git a/dashboard_publisher.py b/dashboard_publisher.py index 44e5a3c..c237433 100644 --- a/dashboard_publisher.py +++ b/dashboard_publisher.py @@ -65,6 +65,7 @@ from agent_panel import attach_panel from app_config import (get_app_config, set_db_writes_for_process, violent_watch_enabled) from config import PlatformConfig, Timeframe, XAUUSD +from exposure import attach_exposure from main import report_to_dict from news_engine import NewsEngine from pair_signals import produce_signal @@ -437,6 +438,11 @@ class DashboardPublisher: if violent_watch_enabled(): attach_watch(report, tf, get_app_config().violent_watch_settings) + # Display-only "already open" line (a read-only query on the + # bot's signals.db — this process still never writes it) and, + # only when config.json sets trade_size.dashboard, the lot + # size: the page is public and the size shows the owner's risk. + attach_exposure(report, surface="dashboard") # Which boundary this report belongs to: the bar LABEL it closed # on plus one interval. Recorded so a boundary missed by an # overrunning cycle or a restart is caught up on the next wake diff --git a/deploy/README-dashboard.md b/deploy/README-dashboard.md index 7ff76a1..605fe37 100644 --- a/deploy/README-dashboard.md +++ b/deploy/README-dashboard.md @@ -77,6 +77,17 @@ boundary (+10 s) and rescans whichever timeframes just closed. Logs rotate every 3 h (240 backups); the bot's own `telegram.log` rotates daily (30 backups) since 2026-09-08. +Since 2026-09-30 each card also shows the daily trend, and a trade plan can +carry an "Already open" line. That line is one **read-only** query on the +bot's `logs/signals.db` (`mode=ro`; this process still never writes either +database), so the daemon user needs read access to that file and its `-wal` +/ `-shm` companions. If it cannot read them the line is simply absent. The +lot-size line is NOT published here unless `config.json` sets +`"trade_size": {"dashboard": true}`: the page is public and the size shows +the owner's risk per trade. The M15 scan now needs the daily (D1) candles as +a required frame, so a D1 fetch failure fails that card's scan like any other +missing frame. + ## 5. Optional Apache polish The page polls its JSON with conditional GETs (`cache: "no-cache"` → Apache diff --git a/docs/beginner_guide.html b/docs/beginner_guide.html index b90d0b4..dae8e8b 100644 --- a/docs/beginner_guide.html +++ b/docs/beginner_guide.html @@ -40,7 +40,7 @@

ForexBot2 — Beginner's Guide

-
Everything explained in plain words. No experience needed.  ·  July 2026
+
Everything explained in plain words. No experience needed.  ·  updated October 2026

1 · What is this bot?

ForexBot2 is a computer program that watches the price of gold all day and all night. It studies the price charts using nine different mathematical checks. When the checks agree that the price is likely to move in one direction, it sends you a message on Telegram telling you:

@@ -74,12 +74,17 @@ 🟢 Grade A · BUY · XAU/USD · H1 🥇 ← how good the setup is + what to do 📐 Direction: strong buy · Reliability: good · timeframes agree ← the bot's two numbers, in words +🧭 Daily trend: rising — this buy + goes with it ← which way gold has been heading for weeks 📋 Indicators … ← the bot's 9 checks (you can skip this part) ━━━━━━━━━━━━━━━━━━━━ 🚀 TRADE SETUP ← ↓ THIS is the part you act on ↓ 📈 Direction: BUY ▶ Entry: 4,132.50 ← open the trade near this price 🛑 SL: 4,088.20 ← your safety brake (stop loss) +🧱 Already open: 1 buy plan alerted + earlier (H1) — taking this one adds + to the same position ← you may already be in this move 🔄 Exit rule (tested): once price reaches 4,159.08 (+0.6R), trail the SL $33.23 behind the best price, moved at @@ -91,6 +96,8 @@
  • Grade (A+, A, B, C): the bot's own mark for how clean the setup is. A+ is best. It never sends anything worse than C.
  • Direction and reliability: plain words for the bot's two internal numbers. Direction says how strongly the checks lean one way (very strong, strong, clear, weak). Reliability says how much the bot trusts that reading (high, good, fair, low). "Timeframes agree" means the bigger and smaller charts point the same way.
    • +
  • 🧭 Daily trend: which way the gold price has been heading over the last weeks, read from the daily chart: rising, falling, or no clear direction. A trade that goes the same way as the daily trend is swimming with the tide. On the 15-minute signals the bot no longer sends a trade that goes against a clear daily trend at all, because years of tests showed those trades earned nothing. On the other signals the line is simply something to know.
    • +
  • 🧱 Already open: appears when the bot has already sent buy (or sell) plans that are still running. A new signal in the same direction is usually the same gold move seen again. The new signal is not a worse one, but if you took the earlier one, taking this one too means you are risking more on a single move. As a beginner, skip it, or split your usual risk between the two.
  • The indicator list: extra detail for curious people. You don't need to understand it to trade the signal.
  • ⚠️ Warning lines: things the bot noticed, like high volatility. They are "be aware," not "don't trade."
  • 🌪 Violent-move watch: now and then the bot adds a line saying the next hour could produce very big candles — usually because an important American economic number is about to come out, or the market is in an hour that is historically wild, or the price has stretched a long way from normal. It is only a "hold on tight" note. It does not change the signal, and it is not a reason to close or move your stop loss early — the bot was tested doing exactly that and it lost money. Your stop loss is already your protection; leave it where it is.
    • @@ -106,7 +113,7 @@ Your account: $2,000. Your risk: 1% = $20. So for this trade, 1R = The signal says: Entry 4,132.50, Stop loss 4,088.20. The gap between them is $44.30 per ounce of gold.
    How much gold to buy: $20 ÷ $44.30 = 0.45 ounces.
    Now, whatever happens: if the price falls to the stop loss, you lose about $20 — and no more. If the trade wins 2R, you make about $40. -

    Most broker apps have a built-in calculator: enter the stop-loss price and the amount you want to risk ($20), and it works out the position size for you. Use it — it avoids mistakes. (On many platforms gold is traded in "lots," where 0.01 lot = 1 ounce.)

    +

    Most broker apps have a built-in calculator: enter the stop-loss price and the amount you want to risk ($20), and it works out the position size for you. Use it — it avoids mistakes. (On many platforms gold is traded in "lots," where 0.01 lot = 1 ounce.) The bot's owner can also switch on a 📏 Size line that does this sum for their own risk limit; it is shown only in the owner's private chat, so if you do not see it, use the calculator.

    5 · What to do when a signal arrives — step by step

      @@ -135,7 +142,7 @@ The message says: "once price hits 4,159, trail SL $33 behind best price."The bot sends signals from several chart speeds. As a beginner, follow only the 1-hour (H1) signals and ignore the others. Here is why:

      - + @@ -146,6 +153,7 @@ The message says: "once price hits 4,159, trail SL $33 behind best price."H1 is BUY-only. All H1 SELL signals are blocked (since 2026-09-03) because they were a losing side on this timeframe — expect to see BUY signals here, never SELL.
    1. M5 also holds back some SELL signals — specifically while the market is in a choppy "transitioning" phase, since those shorts tested poorly.
    2. M15 got a small tune-up on 2026-09-09 — one of its nine indicators (RSI) now counts for half as much, which means a few more M15 signals per week. Nothing changes in how you act on a signal.
      3. +
    3. M15 now respects the daily trend (since 2026-09-30). A 15-minute trade that goes against a clear daily trend is no longer sent. That removes about one signal in four, the ones that made no money in the tests, and makes the bad stretches shorter. Nothing changes in how you act on the signals you do receive.
    4. Never take the same signal on two timeframes. They are usually the same gold move seen at different zoom levels. Taking both doubles your risk, not your profit.
      5. diff --git a/docs/user_guide.html b/docs/user_guide.html index e471815..1c573f9 100644 --- a/docs/user_guide.html +++ b/docs/user_guide.html @@ -35,42 +35,48 @@

      ForexBot2 — User Guide

      -
      How to read, act on, and survive the signals  ·  July 2026
      +
      How to read, act on, and survive the signals  ·  updated October 2026

      1 · What the bot does

      ForexBot2 analyzes gold (XAU/USD) around the clock on several chart timeframes at once. On every candle close it scores nine technical indicators (trend, momentum, volatility, divergences, candlestick patterns), adapts the weighting to the current market regime (trending / ranging / transitioning), checks that higher timeframes agree, adjusts for news and economic-calendar events, and — only when everything lines up — sends a signal with a complete trade plan to Telegram: direction, entry, stop loss, target, and a trailing-stop management plan.

      Most of the time it sends nothing. Silence means no edge was found; that is the bot working correctly, not a malfunction.

      2 · Anatomy of a signal

      -
      🔍 XAU/USD Signal Analysis (M15) -🕐 2026-07-10 14:00 UTC -💰 Close: 4,132.50 -📊 Regime: Trending Up | Vol: normal -━━━━━━━━━━━━━━━━━━━━ -🟢 BUY 🅰 A | Score: +38.2 | 64% conf -✅ MTF aligned -📈 MTF: M5 ✅ H1 ✅ -📋 Indicators … (9 rows) -⚠️ EXTREME volatility — wider stops needed -━━━━━━━━━━━━━━━━━━━━ +
      🟢 Grade A · STRONG BUY · XAU/USD · 15min 🥇 +🕐 2026-09-04 08:00 UTC · 💰 4483.41 +📐 Direction: strong buy · Reliability: high · timeframes agree +📊 Regime: Transitioning · Vol: high +🧭 Daily trend: rising — this buy goes with it +STRONG BUY — High-quality setup, consider entry + 🚀 TRADE SETUP 📈 Direction: BUY -▶ Entry: 4,132.50 -🛑 SL: 4,103.10 ($29.40 / 294 pips = 1R) +▶ Entry 4483.41 +🛑 SL 4468.92 ($14.49 / 145p = 1R) +📏 Size: 0.06 lots risks $86.94 at the stop + (your limit $100.00 per trade) ← owner's chats only +🧱 Already open: 2 buy plans alerted earlier (2 on M15) + — taking this one adds to the same position 🔄 Exit rule (tested): once price reaches - 4,150.14 (+0.6R), trail the SL $22.05 + 4493.55 (+0.7R), trail the SL $10.87 behind the best price, moving it at candle close only. Otherwise the SL is the exit. -🎯 TP1 guide 4,161.90 ($29.40 / 294 pips = 1.0R) -🎯 TP2 guide 4,176.60 (1.5R) +🎯 TP1 guide 4497.90 ($14.49 / 145p = 1.0R) +🎯 TP2 guide 4505.15 (1.5R) ⚖️ Planned TP1 RR 1:1.0 (guide only — the trail is the exit) -📚 Book (this setup class): 312 trades, 58% winners, avg +0.19R per trade
      +📚 Book (this setup class): 58 trades, 62% winners, avg +0.21R per trade + +📈 Other timeframes: 15min: STRONG BUY +46.5 | 5min: BUY +22.8 | 1h: NEUTRAL +14.2 +📋 EMA⇈ RSI→ BB↓ ADX→ MACD⇈ ROC→ KC⇈
      M5 (5-minute)M15 (15-minute)H1 (1-hour) ★ start here
      Signals per weekabout 10about 10about 3
      Signals per weekabout 10about 5about 2
      A trade usually lastsa few hoursabout half a dayhalf a day to a few days
      Time you have to react1–2 minutesup to 15 minutesup to an hour
      Suitsfull-time / automatedexperienced, quickanyone with a job and a life
      - + + + + @@ -78,33 +84,34 @@ - +
      FieldWhat it means
      Signal + gradeDirection (BUY / SELL / STRONG_*) plus a quality grade A+ → F. A trade setup is only attached at grade C or better. Higher grades = more indicator agreement and cleaner regime.
      Direction / reliability lineSince 2026-09-27 the alert prints the two raw numbers in words. Direction is the regime-weighted composite of all nine indicators: very strong (|score| ≥ 40), strong (≥ 33), clear (≥ 25), weak (≥ 15) or no clear direction, with buy or sell from its sign. Reliability is the confidence after the penalties: high (≥ 65), good (≥ 55), fair (≥ 45) or low. The cut-offs are the grade table's own, so the words never disagree with the grade. The numbers themselves remain in --json, --detail and the dashboard meters.
      MTF alignedThe two confirmation timeframes agree with the trade direction. Signals without alignment are filtered out on M5.
      Timeframes agreeThe two confirmation timeframes agree with the trade direction. Signals without alignment are filtered out on M5.
      Daily trend 🧭The direction of the daily chart, from closed daily candles only: rising when the last daily close is above its 50-day average and above the close 60 trading days earlier, falling when it is below both, otherwise no clear direction. The line also says whether the plan goes with it or against it. On M15 a plan against a clear daily trend is not sent at all (since 2026-09-30): in the three-year test those plans earned nothing after costs, and dropping them cut the worst losing run from 20.9R to 12.4R while the total rose from +195.8R to +205.1R; over 2019–2026 the worst run fell from 64.8R to 41.5R. On M5, H1 and H4 the same rule was tested and did not pass, so there the line is context only — a plan "against" the daily trend is still a valid signal on those timeframes.
      Size 📏How many lots keep the loss at the printed stop inside the risk limit set in config.json (trade_size.risk_per_trade, in US dollars; 1 lot = 100 ounces unless contract_size says otherwise, and the lot step is 0.01 unless lot_step says otherwise). The size is rounded down to the broker's lot step, so the money at risk is at or under the limit, never over. When even the smallest lot would break the limit the line says so and gives the amount; skip that trade or accept the larger risk knowingly. The line appears only when the limit is set, and only in the owner's own chat with the bot and on the command line: the channel, any group the bot posts in and the public dashboard do not show it unless trade_size.telegram_channel / trade_size.dashboard are switched on.
      Already open 🧱How many plans in the same direction the bot alerted earlier are still open in its own outcome tracking (stop not hit, trail not closed), and on which timeframes. Taking every alert stacks positions on one move of gold: in the three-year M5 + M15 test two entries in three already had a same-direction plan open and one in four had three or more. Those stacked entries were not worse trades, so the line is not a reason to skip; it is a reminder that the total at risk is the sum of every open plan, not the risk of this one alert. The count comes from the bot's model of the plans it sent, not from your broker account. Alerts sent at the same candle close are counted too: when the 5-minute and the 15-minute alert go out together, the second one already shows the first.
      Warnings ⚠️Context, not veto. Backtesting shows warned trades perform about the same — strong momentum often triggers warnings.
      Violent-move watch 🌪An occasional extra line saying the next hour looks unusually likely to produce a very large candle — because a high-impact US release is due (or just landed), because the clock is in a historically wild slot, or because price is stretched outside its bands. It is a heads-up about volatility, not a direction call, and it never changes the signal, the grade or the trade setup — the same alert would be sent without it. Do not close or tighten an open trade because of it: every such protective action was tested over three years and lost money. Your stop is the protection.
      AI check 🤖An advisory block written for a new trader, such as "AI check: the odds look worse than usual. New data just came out against this buy. Chance it pays off: about 45 in 100 (usual for this kind of setup: 60 in 100). For: Before the news, the price was climbing. Against: Fresh data just came out against gold. This check is an experiment. Do not trade on it alone." The first sentence compares the estimate with the usual odds for this kind of setup (8 points either way counts as a difference); there is no verdict word and no confidence percentage. One AI reviewer (the news reader) receives the recent headlines together with facts computed by code — how far the move has already run, whether a very large candle just printed, how the stop and target compare with the usual, how trades of this exact kind fared in the two-year test, and whether major scheduled news lands before the trade would normally finish — and gives a probability that the trade ends in profit plus one reason for and one against. (Three further reviewers, chart, risk and sceptic, were switched off on 2026-09-17: in two evaluations they added nothing beyond the base rate and mostly voted the same way on every setup.) It is not a recommendation and it never changes the signal, the grade or the setup. Until about 300 of these estimates have been graded against real outcomes, treat it as context, not as a filter: the earlier AGREE/OPPOSE version of this line opposed target-reaching trades as often as it agreed.
      SL / TP1 guide / planned RRStop loss sits behind the nearest swing level with an ATR buffer (grade A+/A setups use a 2 × ATR stop). 1R = the entry-to-SL distance = the amount you risk. TP1 is a guide (the nearest opposing level or a multiple of ATR, never printed closer than 1R since 2026-09-26); the "planned TP1 RR" is TP1 ÷ 1R and is always at least 1:1. It is not the expected payoff: the tested exit is the trail, and the validated books earn their profit from trailed runners (M5 average win about +1.0R against an average loss of −0.76R; M15 +0.87R against −0.93R).
      Exit rule 🔄The real, backtested exit. Once price moves the stated R in your favor (0.6R on M5, 0.7R on M15), move the stop to trail the stated $ distance behind the best price and update it at each candle close only. Until then the original SL is the exit. TP1 is advisory — let winners run.
      Book line 📚What this class of setup (timeframe, setup type, grade, direction) actually did in the validated two-year backtests: number of trades, share of winners and average R per trade. When the exact class has fewer than 30 trades the line falls back to all setups of that direction on the timeframe, then to the whole timeframe.
      One alert per moveAfter an alert, the bot sends no new alert on that timeframe for the next 5 candles (25 min on M5, 75 min on M15), the same entry cooldown the backtester applies. Consecutive-bar re-entries into the same move were the largest live-only loss in September 2026.
      One alert per moveAfter an alert, the bot sends no new alert on that timeframe for the next 5 candles (25 min on M5, 75 min on M15), the same entry cooldown the backtester applies. Consecutive-bar re-entries into the same move were the largest live-only loss in September 2026. The five candles are candles that actually traded: the daily one-hour pause and the weekend do not use them up, so the first candle after the pause is still held if fewer than five have printed since the alert. The bot also remembers its last alert across a restart.

      3 · Acting on a signal — step by step

        -
      1. Size the position first. Decide your risk per trade — 0.5–1% of account, never more. Position size = (account × risk%) ÷ stop distance in $. The stop distance varies per signal, so the size changes every trade; the dollar risk must not.
        2. +
      2. Size the position first. Decide your risk per trade — 0.5–1% of account, never more. Position size = (account × risk%) ÷ stop distance in $. The stop distance varies per signal, so the size changes every trade; the dollar risk must not. With trade_size.risk_per_trade set in config.json the alert does this sum for you (the 📏 line). If the 🧱 line shows plans already open in the same direction, the new trade adds to them: keep the total at risk across all open plans inside what you can afford on one move.
      3. Enter promptly at market. All backtested results assume entry at the open of the candle after the signal. On M5, minutes matter (see §4). If you are very late — price already moved a large part of the way to the arm level — skip the trade. There is always another signal.
      4. Place the stop loss immediately at the given SL. No mental stops — gold gaps and spikes.
      5. Manage with the trail, not TP1. When price reaches the arm level printed in the message, start trailing the stop the stated $ distance behind the best price, tightening only at candle close. This is what produced the backtested results; taking fixed TP1 profits gives most of the edge away.
        6. -
      6. One signal, one trade. The M5, M15 and H1 books confirm off each other and are usually long or short gold at the same time. Taking the same move on multiple timeframes multiplies risk, not diversification. Pick your timeframe (§4) and ignore the others.
        7. +
      7. One signal, one trade. The M5, M15 and H1 books confirm off each other and are usually long or short gold at the same time. Taking the same move on multiple timeframes multiplies risk, not diversification. Pick your timeframe (§4) and ignore the others. The 🧱 "already open" line tells you when an alert would add to plans that are still running.

      4 · Choosing your timeframe

      -

      Twelve-month Oanda-candle backtest (Aug 2025 – Aug 2026, $0.65 round-trip cost, entry at next candle open):

      +

      Twelve-month Oanda-candle backtest with the rules as they run today (Aug 2025 – Aug 2026, $0.65 round-trip cost, entry at next candle open; regenerated 2026-10-01):

      - - - - - - - + + + + + + +
      M5M15H1
      Trades / week~11.3~7.6~2.7
      Total result (12 mo)+106.8R+73.9R+24.0R
      Win rate / profit factor52.9% / 1.5162.2% / 1.5360.9% / 1.55
      Typical (median) hold2.4 h7 h15.5 h
      90% of trades close within10.5 h29.8 h3.2 days
      Typical stop distance~$21~$32~$52
      Worst drawdown10.7R10.8R8.0R
      Trades / week~10.7~5.1~1.7
      Total result (12 mo)+108.8R+92.3R+22.2R
      Win rate / profit factor53.6% / 1.5468.0% / 2.1262.6% / 1.79
      Typical (median) hold2.5 h7.8 h21 h
      90% of trades close within11.2 h29.4 h3.2 days
      Typical stop distance~$21~$32~$48
      Worst drawdown10.1R8.3R4.1R
      +

      This was a strongly rising year for gold until March 2026 and a falling, choppy one after. Four fifths of the M5 total (+89.6R of +108.8R) came in January–March 2026; M15 earned in every quarter but only +2.3R since July. Over 2019–2026 the M15 rules made +223R with one losing year (2021, −27R) and two years close to flat (2022 and 2023); the M5 rules took few trades and made nothing before 2025. Judge the next twelve months by the long record, not by this table.

      • M5 — intraday, execution-critical. Highest total edge, but stops are tight: a couple of dollars of slippage or a late entry eats a meaningful share of every trade. Suitable only if you can act within a minute or two of the alert (or automate). Trades open and close within a session; you sleep flat. SELL signals are held back while the market regime reads "transitioning" — a validated filter, not a bug if you notice fewer shorts there.
        • -
      • M15 — the middle path. Same frequency as M5, wider stops, up to 15 minutes to act. About half your trades will still be open overnight. Since 2026-09-09 the M15 scoring gives its RSI reading half the weight it used to (RSI mostly voted against the trend on this timeframe), so expect roughly 9 signals a week instead of 7–8; in the two-year test (Sep 2024 – Sep 2026) that change lifted M15 from +117.6R to +146.1R with the same win rate and a slightly better profit factor, and it also improved the separate out-of-sample period. The M15 column above predates it.
        • +
      • M15 — the middle path. Same frequency as M5, wider stops, up to 15 minutes to act. About half your trades will still be open overnight. Since 2026-09-09 the M15 scoring gives its RSI reading half the weight it used to (RSI mostly voted against the trend on this timeframe), so expect roughly 9 signals a week instead of 7–8; in the two-year test (Sep 2024 – Sep 2026) that change lifted M15 from +117.6R to +146.1R with the same win rate and a slightly better profit factor, and it also improved the separate out-of-sample period. Since 2026-09-26 grade-A shorts are blocked on M15, and since 2026-09-30 any M15 plan against a clear daily trend is dropped (the 🧭 line, §2). Together these leave about 5 signals a week. The daily-trend rule is a trade of profit for safety: it removes about a quarter of the trades, the ones that earned nothing after costs, and shortens the losing runs; in a stretch where gold only rises it gives up a little (Sep 2024 – Sep 2025: +61.8R instead of +70.1R).
      • H1 — swing trading around a normal life. Best quality per trade and very forgiving of slow manual execution (you have up to an hour). The costs: positions held for days, through news and overnight sessions, and a lumpy equity curve — one quarter of the test year was negative and most of the profit came in a single quarter. Judge it on quarters, not weeks. H1 no longer issues SELL signals at all (blocked since 2026-09-03) — it is BUY-only on this timeframe.
      If you trade manually from the Telegram alerts, H1 (or M15) is the honest choice. M5's backtested edge assumes near-instant fills that a human copying signals rarely achieves.
      @@ -113,7 +120,7 @@
      • You will lose roughly 38–47% of trades. A 53–62% win rate means losing streaks of 5+ are routine over a year. The system's edge is the average across hundreds of trades, not any single one.
      • Stops hit fast, winners take time. Losing trades die in roughly half the time winners take. An early stop-out is the system doing its job.
        • -
      • Drawdowns of ~10R are normal. At 1% risk per trade, expect to be ~10% down from a peak at some point in a normal year. This is why risking more than 1% per trade is how accounts die.
        • +
      • Drawdowns of ~10R are normal, and worse has happened. At 1% risk per trade, expect to be ~10% down from a peak at some point in a normal year. Over 2019–2026 the M15 rules as they run today had one losing run of 41.5R that took from late 2020 to early 2022 to bottom out, while gold drifted sideways and down. This is why risking more than 1% per trade is how accounts die.
      • Backtest ≠ live. Results above charge real spreads and realistic fills, but cannot model your personal latency, broker slippage, or requotes. Assume live results will be somewhat worse — especially on M5.
      • Quiet periods are normal. Filters block bad hours, bad regimes and misaligned timeframes. Days can pass without a signal.
      @@ -136,7 +143,7 @@ /startShow the button menu (manual scans per timeframe, Scan All TFs). /scanRun an immediate analysis of the primary timeframe. /autoscanToggle automatic scanning (M5 / M15 / H1 / H4 at each candle close). -/statusUptime, auto-scan state, WebSocket price-monitor state. +/statusUptime, auto-scan state, WebSocket price-monitor state, and the health of the news sources: the calendar, the headline feeds, the local headline model (a ⚠️ line "News model: not scoring" means this cycle's headlines carry the simple keyword scores) and DeepSeek. /newsNet XAU/USD news score from the last 4 hours of headlines (neutral headlines are removed before averaging; informational only — never gates signals). /cache · /clearcacheInspect / reset the market-data cache. /helpCommand reference. diff --git a/exposure.py b/exposure.py new file mode 100644 index 0000000..c5437dc --- /dev/null +++ b/exposure.py @@ -0,0 +1,160 @@ +""" +ForexBot2 — trade size and open-exposure lines (display only) +============================================================== +Two lines a surface prints next to a trade setup. Nothing here is read by +the signal core, a gate or the backtester: + +* size line — how many lots keep the loss at the stop inside the risk the + owner set in config.json (`trade_size.risk_per_trade`). Off until that + is set. The size reveals the owner's risk per trade, so it goes to the + owner's own chat (telegram.chat_id) and the CLI only, unless + `trade_size.telegram_channel` (every other Telegram chat: the channel, + a group) / `trade_size.dashboard` switch it on for the public surfaces. +* open line — how many ALERTED plans in the same direction are still open + in the advisory outcome model (logs/signals.db, read-only). Taking every + alert stacks positions: on the 3-year M5 + M15 books two entries in + three already have a same-direction plan open, one in four has three or + more, and up to 13 are open at once + (backtest/research/daily_trend_campaign_2026-09-30.md, section 5.11). + Those stacked entries did not pay worse; the point of the line is that + the full risk on each alert is several times that risk on one move. + +Attached post-hoc by the surface (Telegram bot, CLI, dashboard), the same +pattern as violent_watch.attach_watch. Never raises: a display line must +not break a scan or an alert. +""" + +import logging +from decimal import ROUND_FLOOR, Decimal, InvalidOperation +from typing import Any, Dict, List, Optional, Tuple + +import app_config +import signal_log + +logger = logging.getLogger(__name__) + +# Surfaces a size line may be attached for, and the config flag that +# shares it with a public audience (None = the owner's own surface). +SURFACES = {"telegram": None, "cli": None, "dashboard": "dashboard"} + +_TF_LABEL = {"5min": "M5", "15min": "M15", "1h": "H1", "4h": "H4"} + + +def lot_size(stop_distance: float, risk: float, contract_size: float = 100.0, + lot_step: float = 0.01) -> Tuple[float, float]: + """(lots, money at risk with those lots): the largest multiple of + `lot_step` whose loss at the stop stays within `risk`. + + One lot is `contract_size` ounces, so a $1 move is worth + `contract_size` dollars per lot. Decimal arithmetic with a floor, so + the size never rounds UP past the risk limit and 0.07 never prints as + 0.0699. (0.0, 0.0) when any input is unusable.""" + try: + d, r = Decimal(str(stop_distance)), Decimal(str(risk)) + c, s = Decimal(str(contract_size)), Decimal(str(lot_step)) + except (InvalidOperation, ValueError): + return 0.0, 0.0 + if not (d.is_finite() and r.is_finite() and c.is_finite() and s.is_finite()): + return 0.0, 0.0 + if d <= 0 or r <= 0 or c <= 0 or s <= 0: + return 0.0, 0.0 + steps = (r / (d * c * s)).to_integral_value(rounding=ROUND_FLOOR) + lots = steps * s + return float(lots), float(lots * d * c) + + +def _decimals(step: float) -> int: + text = format(Decimal(str(step)).normalize(), "f") + return len(text.split(".")[1]) if "." in text else 0 + + +def size_line(setup: Any, settings: Optional[Dict[str, Any]]) -> str: + """"Size: 0.05 lots risks $99.00 at the stop (your limit $100.00 per + trade)", or "" when the size is not configured or the setup has no + usable stop. + + The stop distance is taken from the prices as they are printed (two + decimals, subtracted exactly), so the line agrees with the entry and + stop above it and a $20.00 stop is never read as $20.000000000000004.""" + if setup is None or not settings: + return "" + try: + risk = float(settings.get("risk_per_trade") or 0.0) + contract = float(settings.get("contract_size", 100.0)) + step = float(settings.get("lot_step", 0.01)) + distance = abs(Decimal(f"{float(setup.entry_price):.2f}") + - Decimal(f"{float(setup.stop_loss):.2f}")) + except (TypeError, ValueError, InvalidOperation): + return "" + if not risk > 0 or not distance.is_finite() or not distance > 0: + return "" + lots, at_risk = lot_size(distance, risk, contract, step) + places = _decimals(step) + if lots <= 0: + smallest = float(distance) * contract * step + if not smallest > 0: + return "" + return (f"Size: the stop is too wide for your ${risk:,.2f} limit — the smallest " + f"size ({step:.{places}f} lots) would risk ${smallest:,.2f}") + return (f"Size: {lots:.{places}f} lots risks ${at_risk:,.2f} at the stop " + f"(your limit ${risk:,.2f} per trade)") + + +def open_line(rows: Optional[List[Dict[str, Any]]], direction: str) -> str: + """"Already open: 2 buy plans alerted earlier (M5, M15) — taking this + one adds to the same position", or "" when no alerted plan in + `direction` is still open.""" + same = [r for r in (rows or []) if r.get("direction") == direction] + if not same: + return "" + counts: Dict[str, int] = {} + for r in same: + label = _TF_LABEL.get(str(r.get("timeframe")), str(r.get("timeframe"))) + counts[label] = counts.get(label, 0) + 1 + where = ", ".join(f"{n} on {tf}" if n > 1 else tf for tf, n in counts.items()) + n = len(same) + return (f"Already open: {n} {direction.lower()} plan{'s' if n != 1 else ''} alerted " + f"earlier ({where}) — taking this one adds to the same position") + + +def size_shared(flag: str) -> bool: + """True when config.json shares the size line with a public audience + (`flag` = "telegram_channel" or "dashboard"). Never raises.""" + try: + settings = app_config.trade_size_settings() + return bool(settings and settings.get(flag)) + except Exception: + return False + + +def attach_exposure(report: Any, surface: str = "telegram") -> Any: + """Stamp report.size_line and report.open_line for `surface` + ("telegram" / "cli": the owner's own view; "dashboard": the size only + when trade_size.dashboard is true). Display only; never raises.""" + report.size_line = "" + report.open_line = "" + setup = getattr(report, "trade_setup", None) + if setup is None: + return report + try: + settings = app_config.trade_size_settings() + flag = SURFACES.get(surface, "unknown-surface") + if settings and (flag is None or settings.get(flag)): + report.size_line = size_line(setup, settings) + except Exception: + logger.debug("exposure: size line unavailable", exc_info=True) + try: + # The same identity the logger stores for this scan's row, so the + # plan being alerted now never counts itself (a re-alert of an + # unchanged plan is the same position, not a second one). + analysis = getattr(report, "primary_analysis", None) + key = signal_log.setup_key( + {"direction": setup.direction, "entry": setup.entry_price, + "stop_loss": setup.stop_loss, + "strategy": getattr(setup, "strategy", None)}, + analysis.timeframe if analysis is not None else "") + rows = signal_log.open_alerted_setups(report.symbol, exclude_key=key) + report.open_line = open_line(rows, setup.direction) + except Exception: + logger.debug("exposure: open line unavailable", exc_info=True) + return report diff --git a/llm_client.py b/llm_client.py index d128caa..04bbc64 100644 --- a/llm_client.py +++ b/llm_client.py @@ -53,7 +53,10 @@ class DeepSeekClient: daily_cap: int = DEFAULT_DAILY_CAP, enabled: bool = True, ): - self._api_key = api_key or "" + # Whitespace around a key is never part of it, and a trailing line + # break (a key pasted from a file) is not a legal header value: + # every call would fail before it left the process. + self._api_key = (api_key or "").strip() self._model = model self._timeout = timeout self._daily_cap = daily_cap @@ -104,17 +107,36 @@ class DeepSeekClient: self._count += 1 return True + def _redact(self, text: Any) -> str: + """Exception text that is safe to log: never contains the key.""" + out = str(text) + return out.replace(self._api_key, "") if self._api_key else out + def chat_json(self, system_prompt: str, user_payload: Dict[str, Any]) -> Optional[Dict[str, Any]]: - """One chat call; returns the model's JSON object or None.""" - if not self.available or not self._under_cap(): + """One chat call; returns the model's JSON object or None. + + Never raises: every caller reads None as "use the fallback", and + the news enrichment runs inside a live scan.""" + if not self.available: + return None + try: + content = json.dumps(user_payload, sort_keys=True) + except (TypeError, ValueError) as e: + # Before the cap is counted: a payload that cannot be sent + # must not spend one of the day's requests. + self.last_error = (time.time(), "payload") + self._warn_once( + "payload", "DeepSeek payload is not JSON-serialisable " + "(%s) — call skipped", type(e).__name__) + return None + if not self._under_cap(): return None body = { "model": self._model, "messages": [ {"role": "system", "content": system_prompt}, - {"role": "user", - "content": json.dumps(user_payload, sort_keys=True)}, + {"role": "user", "content": content}, ], "response_format": {"type": "json_object"}, "temperature": 0, @@ -133,19 +155,44 @@ class DeepSeekClient: try: resp = self._session.post( DEEPSEEK_URL, json=body, timeout=self._timeout) + except requests.exceptions.InvalidHeader: + # requests quotes the offending header VALUE in this error, and + # the only header built from configuration is Authorization: + # the text would put the API key in the log. Fixed wording. + self.last_error = (time.time(), "header") + self._warn_once( + "header", "DeepSeek call not sent: the API key holds a " + "character that is not allowed in an HTTP header (a line " + "break?) — check deepseek.api_key / DEEPSEEK_API_KEY") + return None except requests.RequestException as e: self.last_error = (time.time(), "network") - self._warn_once("network", "DeepSeek call failed: %s", e) + self._warn_once("network", "DeepSeek call failed: %s", + self._redact(e)) + return None + except Exception as e: + # Anything else raised inside the HTTP stack (a header that + # cannot be encoded, a transport bug) must not reach a scan + # either. Type name only: the text may quote the request. + self.last_error = (time.time(), "client") + self._warn_once("client", "DeepSeek call failed inside the " + "HTTP client: %s", type(e).__name__) return None if resp.status_code != 200: self.last_error = (time.time(), f"http_{resp.status_code}") self._warn_once( f"http_{resp.status_code}", - "DeepSeek HTTP %s: %s", resp.status_code, resp.text[:200]) + "DeepSeek HTTP %s: %s", resp.status_code, + # Redact BEFORE cutting: a gateway error page can echo the + # request headers, and a key cut at the 200-character + # mark would be left partly visible. + self._redact(resp.text)[:200]) return None try: parsed = self._parse_response(resp.json()) - except ValueError: + except Exception: + # ValueError for a body that is not JSON; anything else (a + # pathological nesting depth) is just as unusable. parsed = None if parsed is None: self.last_error = (time.time(), "parse") diff --git a/main.py b/main.py index 39d2e15..392e74a 100755 --- a/main.py +++ b/main.py @@ -23,6 +23,7 @@ if hasattr(sys.stderr, "reconfigure"): from agent_panel import attach_panel from app_config import get_app_config, violent_watch_enabled from config import PlatformConfig, Timeframe, SignalStrength, XAUUSD +from exposure import attach_exposure from pair_signals import PAIR_PRODUCERS from signal_aggregator import SignalAggregator, SignalReport from violent_watch import attach_watch @@ -143,6 +144,25 @@ def plain_words(report: SignalReport) -> str: f"Reliability: {reliability_words(report.final_confidence)} · {agree}") +def daily_trend_words(report: SignalReport) -> str: + """The daily trend in words, and whether the trade plan goes with it + ("Daily trend: rising — this buy goes with it"). "" when the scan + carries no daily trend. Display only (2026-09-30): on M15 the signal + core already drops a plan against a clear daily trend; on the other + timeframes the line is information for the reader.""" + trend = getattr(report, "daily_trend", None) + if trend is None: + return "" + if trend == 0: + return "Daily trend: no clear direction" + word = "rising" if trend > 0 else "falling" + setup = report.trade_setup + if setup is None: + return f"Daily trend: {word}" + side = "with" if (setup.direction == "BUY") == (trend > 0) else "against" + return f"Daily trend: {word} — this {setup.direction.lower()} goes {side} it" + + def print_signal_report(report: SignalReport, detail: bool = False): sc = signal_color(report.final_signal) qc = quality_color(report.signal_quality) @@ -156,6 +176,9 @@ def print_signal_report(report: SignalReport, detail: bool = False): f"{sc}{report.final_score:+.1f}{Colors.RESET} · " f"{report.final_confidence:.0f}% · {mtf_tag}") print(f" {plain_words(report)}") + trend_words = daily_trend_words(report) + if trend_words: + print(f" {trend_words}") print(f" {Colors.GRAY}{report.action_recommendation}{Colors.RESET}") # ── 2. TRADE SETUP (if signal is actionable) ─────────────────── @@ -182,6 +205,11 @@ def print_signal_report(report: SignalReport, detail: bool = False): f"{_fmt_price(arm_price)} (+{arm_r:.1f}R), trail the SL " f"${dist_r * risk_dist:.2f} behind the best price at candle close; " f"TP1 is a guide only.{Colors.RESET}") + # Lot size for the owner's risk limit and the alerted plans still + # open in this direction (exposure.attach_exposure; display only). + for line in (report.size_line, report.open_line): + if line: + print(f" {Colors.BOLD}{line}{Colors.RESET}") # ── 3. CONTEXT (price · regime · vol · MTF breakdown) ────────── print(f"\n {Colors.WHITE}{report.display_name}{Colors.RESET} " @@ -304,6 +332,10 @@ def report_to_dict(report: SignalReport) -> dict: "panel_probability": round(report.panel_probability, 1), "panel_context": report.panel_context, "watch_line": report.watch_line, + "daily_trend": report.daily_trend, + "daily_trend_words": daily_trend_words(report), + "size_line": report.size_line, + "open_line": report.open_line, "bullish_reasons": report.bullish_reasons, "bearish_reasons": report.bearish_reasons, "warnings": report.warnings, @@ -422,6 +454,10 @@ def run_live(args, config: PlatformConfig): attach_watch(report, config.primary_timeframe, get_app_config().violent_watch_settings) + # Display-only lot size (off until config.json sets trade_size) and + # the alerted plans still open in this direction. + attach_exposure(report, surface="cli") + if args.json: payload = report_to_dict(report) print(json.dumps(payload, indent=2)) diff --git a/news_engine.py b/news_engine.py index 86d68ad..3eec9af 100755 --- a/news_engine.py +++ b/news_engine.py @@ -384,7 +384,7 @@ class NewsEngine: # writer runs inside _cached_fetch); the other two take the lock # around cache access but keep HTTP calls outside it. _llm_sent_cache: Dict[str, Tuple[float, float]] = {} # sha1 → (s, xau) - _llm_event_cache: Dict[Tuple[str, str, str], str] = {} + _llm_event_cache: Dict[Tuple[str, str, str, str, str], str] = {} _llm_summary_cache: Tuple[float, str] = (0.0, "") _LLM_SENT_CACHE_MAX = 400 _LLM_EVENT_CACHE_MAX = 100 @@ -609,6 +609,23 @@ class NewsEngine: key=lambda e: e.time, ) + def peek_week_events( + self, currencies: Optional[List[str]] = None + ) -> List[EconomicEvent]: + """get_week_events WITHOUT the refresh: whatever calendar the + class-level cache holds right now, however old, filtered and sorted + the same way. Never fetches and never waits on the cache lock, for + a caller that must not add a network wait: the agent panel runs + while an alert is waiting to go out, and a refetch with both + calendar sources down takes about 40 seconds.""" + if currencies is None: + currencies = self.config.tracked_currencies + _stamp, events = NewsEngine._calendar_cache + return sorted( + (e for e in events if e.currency in currencies), + key=lambda e: e.time, + ) + # ── Private Methods ──────────────────────────────────────────── # Maps country names in MyFXBook titles to currency codes @@ -871,6 +888,11 @@ class NewsEngine: continue title = title_el.text or "" + # A blank title carries nothing to score, show or + # archive, and one such item made the local model + # reject the whole batch (2026-10-01). + if not title.strip(): + continue pub_date = self._parse_rss_date( pub_date_el.text if pub_date_el is not None else None ) @@ -1173,11 +1195,27 @@ class NewsEngine: is re-sent to DeepSeek (fallback mode retries, audit sample); F. only score_source == "keyword" rows change; an llm row stands. """ - bundle = news_model.get_news_model() - if bundle is None or not headlines: + if not headlines: return 0 - todo = [h for h in headlines if h.score_source == "keyword"] + bundle = news_model.get_news_model() + if bundle is None: + self._record_source("news_model", False, + "not loaded in this process") + return 0 + # score_batch cleans every text first and raises on one that is + # empty afterwards (" " is), which used to cost the whole + # batch its scores; such a row keeps its keyword score instead. + todo = [h for h in headlines if h.score_source == "keyword" + and self._model_can_score(h.headline)] if not todo: + # Nothing for the model this cycle (fallback mode with DeepSeek + # healthy, or no readable headline). A failure an earlier batch + # recorded says nothing about now and must not sit on /status + # as if it were current; the next batch records its own outcome + # and warns again if it fails. + if self.source_is_failing( + NewsEngine._source_health.get("news_model")): + NewsEngine._source_health.pop("news_model", None) return 0 try: results = news_model.score_batch(bundle, [h.headline for h in todo]) @@ -1185,8 +1223,13 @@ class NewsEngine: raise ValueError( f"{len(results)} results for {len(todo)} headlines") except Exception as e: - logger.debug(f"news model scoring failed, keyword scores stand: {e}") + # Keyword scores stand for the cycle. Loud once (WARNING on the + # first failure, INFO on recovery) and shown by /status: in + # primary mode nothing else scores these headlines. + self._record_source("news_model", False, + f"{type(e).__name__}: {e}") return 0 + self._record_source("news_model", True, extra=f"{len(todo)} scored") for h, r in zip(todo, results): raw = float(r["score"]) # archive value shown = raw if r["relevant"] else 0.0 # display value @@ -1198,6 +1241,15 @@ class NewsEngine: h.score_source = "model" return len(todo) + @staticmethod + def _model_can_score(text: str) -> bool: + """True when the local model's own cleaner accepts the text.""" + try: + news_model.clean_headline(text) + return True + except Exception: + return False + def _audit_with_deepseek( self, headlines: List[NewsSentiment] ) -> List[NewsSentiment]: @@ -1207,10 +1259,15 @@ class NewsEngine: numbers. One batch per cycle at most; the pairs go into _llm_sent_cache so a headline is never audited twice. This is the only place DeepSeek is called for headlines in primary mode; with - the cap at 0 (the shipped value) it returns [] without a call.""" + the cap at 0 (the shipped value) it returns [] without a call. + The copies exist only to be archived, so a process that does not + write the archive (the dashboard daemon, NEWS_ARCHIVE_ENABLED=0) + makes no call either.""" cap = app_config.news_model_audit_per_day() if cap <= 0 or not headlines or not get_llm_client().available: return [] + if not app_config.news_archive_enabled(): + return [] today = datetime.now(timezone.utc).date().isoformat() if NewsEngine._audit_day != today: NewsEngine._audit_day, NewsEngine._audit_sent_today = today, 0 @@ -1253,7 +1310,17 @@ class NewsEngine: close K/M-suffixed pairs backwards ~11% of the time (e.g. claims 202K vs 217K called "higher than forecast") when left to compare the raw numbers itself.""" - key = (event.event, event.currency, str(event.actual or "")) + # Forecast and release time are part of the key: the direction is + # a reading of actual AGAINST forecast, and next month's release + # can print the same actual against a different forecast. The time + # is keyed in UTC so one instant is one key whatever offset the + # feed printed it with. + try: + when = event.time.astimezone(timezone.utc).isoformat() + except Exception: + when = str(event.time) + key = (event.event, event.currency, str(event.actual or ""), + str(event.forecast or ""), when) with NewsEngine._cache_lock: if key in NewsEngine._llm_event_cache: return NewsEngine._llm_event_cache[key] or None diff --git a/pair_signals/xauusd.py b/pair_signals/xauusd.py index 36d25d0..2d4eba3 100644 --- a/pair_signals/xauusd.py +++ b/pair_signals/xauusd.py @@ -130,6 +130,45 @@ on M15 (trade-identical, -2.0R W2, -2.8R W3: the swing-anchored GENERIC stop is not better than the 2 x ATR stop on the same grade-A signals); the combination with this block (-0.7R vs the block alone). +M15 override 7 — daily-trend veto (validated 2026-09-30, +backtest/research/daily_trend_campaign_2026-09-30.md, pre-registered +RISK-FIRST rule approved by the owner that day, CUDA twin runs paired +with gpu_backtest.py MD5s, $0.65): daily_trend_veto="both". The engine +sees at most the H1 chart. The daily trend is +1 when the last CLOSED +daily bar closes above its 50-day EMA AND above the close 60 daily +bars earlier, -1 when below both, else 0 (Oanda's Sunday stub bars are +dropped; signal_aggregator.daily_trend_series). After every setup +archetype has run, a setup AGAINST a non-zero state is dropped (grade +D, no setup, warning "DAILY TREND FILTER: ..."). Candidate vs control: +3-year W3 2023-09-05 -> 2026-09-05 824/+205.12R PF 1.73 DD 12.4R vs +1105/+195.79R PF 1.49 DD 20.9R (halves +76.7/+128.4 vs +76.9/+118.9; +at $0.30 +221.18R PF 1.81 DD 11.4R vs +216.53R PF 1.55 DD 19.6R); +7.6-year LONG 2019-01-01 -> 2026-08-07 1473/+223.33R PF 1.42 DD 41.5R +vs 2091/+192.53R PF 1.24 DD 64.8R (2022 +3.2R vs -24.6R; 2021 NOT +improved, -27.3R vs -25.6R; 2023 worse, +1.8R vs +11.7R); the unseen +HOLDOUT 2026-09-05 -> 2026-10-01 18/-4.62R vs 24/-6.63R; 3 of 4 +neighbouring definitions pass (90-day change, 120-day change, 100-day +average; the 60-day change alone misses the 97% line by 1.7R). Trade +diff W3: 281 removed (-9.3R: BUY 106/+7.6R, SELL 175/-16.9R), 0 added; +the removed trades are break-even after costs (-0.03R each) against ++0.25R for the kept ones, so the veto drops the no-edge quarter of the +book rather than a pocket of big losers. ACCEPTED COST (owner decision +2026-09-30): R given up while gold only rises — 2-year W2 597/+155.89R +PF 1.78 DD 11.4R vs 813/+154.15R PF 1.53 DD 15.6R with the first half ++61.8R vs +70.1R, and the old OOS window 2024-06-01 -> 2025-03-30 +258/+63.78R PF 1.80 DD 9.1R vs 313/+68.50R PF 1.67 DD 12.4R. The +2026-09-26 rule template (no half and no OOS window may lose) would +have rejected it; the risk-first rule (3-year R >= 97%, PF +0.10, DD +not deeper, long window better on R, DD and 2021+2022, holdout, both +costs, plateau) is the owner's deliberate choice. M15 ONLY: the same +veto FAILED on M5 ("both": W3 DD 10.1 -> 15.2R and the long window +worse on R and DD; "buy": passes W3, loses the long window, no +plateau) and on H1 (W3 +60.3R vs +70.0R). "buy"-only and "sell"-only +FAILED on M15 as well ("sell" has the best W3 total, +212.7R, and a +LONG DD of 76.4R vs 64.8R: in falling years the buys against the daily +trend are the losers). The live wrapper and both backtesters load D1 +as an extra REQUIRED frame while the veto is on. + M5 override (validated 2026-07-11, full-history replay 2020-03-17 -> 2026-07-11 + 12mo book, $0.30 spread): block hour 9 UTC (post-London- open whipsaw). Hour-9 entries lost -28.9R/143 over 6 years, negative @@ -332,6 +371,12 @@ def make_config( # actionable-grade short whose grade is not listed) — see module # docstring (M15 override 6, validated 2026-09-26). config.xau_strategy.sell_min_quality = ["A+", "B"] + # Daily-trend veto on M15 only: a finished setup against a clear + # daily trend (price vs its 50-day average AND the 60-day change) + # is dropped. The live wrapper and both backtesters load the D1 + # frame for it — see module docstring (M15 override 7, validated + # 2026-09-30). M5, H1 and H4 FAILED the same rule and stay "off". + config.xau_strategy.daily_trend_veto = "both" if primary_tf == Timeframe.H1: # Block every H1 SELL — the SELL quality gate (6e in # signal_aggregator.py) downgrades any actionable-grade short diff --git a/providers/oanda.py b/providers/oanda.py index b640490..bc495e7 100644 --- a/providers/oanda.py +++ b/providers/oanda.py @@ -37,6 +37,10 @@ class OandaDataProvider(BaseDataProvider): RETRY_DELAYS_S = (2.0, 5.0, 10.0) RETRY_BUDGET_S = 45.0 RETRY_DELAY_S = RETRY_DELAYS_S[0] # first delay; kept for callers/tests + # The longest date window that may come back empty and still mean + # "nothing traded here": a weekend joined to a holiday. A longer window + # with no bars is an anomaly, not a closure (see _fetch_oanda_candles). + EMPTY_WINDOW_MAX = pd.Timedelta(days=5) OANDA_GRANULARITY = { Timeframe.M1: "M1", @@ -96,6 +100,26 @@ class OandaDataProvider(BaseDataProvider): symbol, timeframe, outputsize, start, end) return self._fetch_oanda_candles(symbol, timeframe, outputsize) + @staticmethod + def _empty_frame() -> pd.DataFrame: + """A normalized frame with no bars (a date window nothing traded in).""" + df = pd.DataFrame( + {col: pd.Series(dtype="float64") + for col in ("Open", "High", "Low", "Close", "Volume")}, + index=pd.DatetimeIndex([], tz="UTC", name="Datetime")) + return df + + @classmethod + def _closure_sized(cls, start: Optional[datetime], + end: Optional[datetime]) -> bool: + """True for a date window short enough to be one scheduled closure.""" + if start is None or end is None: + return False + try: + return pd.Timestamp(end) - pd.Timestamp(start) <= cls.EMPTY_WINDOW_MAX + except (TypeError, ValueError): + return False + def _to_oanda_instrument(self, symbol: str) -> str: try: return self.OANDA_INSTRUMENT[symbol] @@ -192,8 +216,25 @@ class OandaDataProvider(BaseDataProvider): raise OandaDataError( f"Oanda candle response for {instrument} is not valid " f"JSON: {e}") from e + # A DATE WINDOW with no bars in it (a weekend, a holiday, the closed + # days before a window's first session) is not a failure: fetch_range + # walks history in windows and counts an empty one as "nothing traded + # here". Raising for it made every backtest window whose padded start + # falls on a closed day unfetchable once fetch_range began retrying a + # failing window and aborting after three tries (2026-09-13); found + # 2026-09-30 on the D1 pad of a window starting on a Saturday. A + # `count` request (the live path) with nothing in it stays an error. + # Only a CLOSURE-SIZED window gets that reading (2026-10-01 review): + # an empty answer for a longer one (a 15-day M5 page, a 46-day M15 + # page) cannot be a closure, and accepting it let fetch_range step + # over the window and cache history with a hole in it. Raising makes + # fetch_range retry the same window and refuse after three tries. + # (An M1 page is 3 days, so it would still pass; M1 is not fetched.) + closure_sized = self._closure_sized(start, end) candles = data.get("candles", []) if not candles: + if closure_sized: + return self._empty_frame() raise OandaDataError( f"Empty candle response from Oanda for {instrument}") @@ -233,6 +274,9 @@ class OandaDataProvider(BaseDataProvider): "OHLC fields) — skipped", skipped_malformed, instrument) if not rows: + # Malformed candles mean the window DID trade: never "no bars". + if closure_sized and not skipped_malformed: + return self._empty_frame() raise OandaDataError( f"No complete candles from Oanda for {instrument}") diff --git a/scripts/label_scan_outcomes.py b/scripts/label_scan_outcomes.py index 5d4e32c..d8ad265 100755 --- a/scripts/label_scan_outcomes.py +++ b/scripts/label_scan_outcomes.py @@ -23,6 +23,8 @@ import argparse import os import sqlite3 import sys +from datetime import datetime, timedelta, timezone +from typing import Optional sys.path.insert(0, os.path.dirname(os.path.dirname(os.path.abspath(__file__)))) @@ -33,18 +35,27 @@ CHUNK = 2000 def _counts(con: sqlite3.Connection, symbol: str, - labelled: bool = True) -> dict: + labelled: bool = True, now: Optional[datetime] = None) -> dict: """Candidate / labelled / complete / pending counts (read-only). The age filter uses logged_at (always an ISO string) as a proxy for - the bar close — the labeller itself parses the real bar label.""" + the bar close — the labeller itself parses the real bar label. The + cutoff is built here in logged_at's own format (UTC, 'T' separator), + the labeller's convention: SQLite's datetime() prints a space, which + sorts before the 'T', so every row logged on the cutoff's calendar + day was left out of the count.""" + moment = now or datetime.now(timezone.utc) + if moment.tzinfo is None: + moment = moment.replace(tzinfo=timezone.utc) + cutoff = moment.astimezone(timezone.utc) - timedelta( + hours=SCAN_LABEL_MIN_AGE_H) scans = ("SELECT COUNT(*) FROM signals s WHERE s.symbol = ?" " AND s.indicators_json IS NOT NULL AND s.price > 0") - due = (scans + " AND s.logged_at <" - f" datetime('now', '-{int(SCAN_LABEL_MIN_AGE_H)} hours')") + due = scans + " AND s.logged_at < ?" out = { "scan rows with indicators": con.execute(scans, (symbol,)).fetchone()[0], - "of those, window elapsed": con.execute(due, (symbol,)).fetchone()[0], + "of those, window elapsed": con.execute( + due, (symbol, cutoff.isoformat())).fetchone()[0], } if labelled: out["labelled"] = con.execute( diff --git a/scripts/panel_replay.py b/scripts/panel_replay.py index ea21bf9..7ec80df 100644 --- a/scripts/panel_replay.py +++ b/scripts/panel_replay.py @@ -275,15 +275,26 @@ def build_report(con, rowid: int) -> Tuple[Optional[SignalReport], Optional[Fund return report, context, frames, _loads(payloads_json, None) +def stored_personas(stored: Any) -> List[str]: + """The requested personas whose exact payload the row logged.""" + if not isinstance(stored, dict): + return [] + return [p for p in PERSONAS if isinstance(stored.get(p), dict)] + + def build_jobs(report: SignalReport, context: FundamentalContext, frames: Dict[Timeframe, pd.DataFrame], stored: Optional[Dict[str, Any]], settings: Dict[str, Any]) -> Tuple[Tuple[str, str, Dict[str, Any]], ...]: - """Persona jobs on the CURRENT prompts: stored payloads when the row - has them, else rebuilt through the live payload builders.""" + """Persona jobs on the CURRENT prompts. A persona whose exact payload + the row logged replays that payload; every other persona is rebuilt + through the live payload builders. Per persona, not all-or-nothing: + the live panel stores only the personas it ran (macro alone since + 2026-09-17), so "all four or none" never used a stored payload.""" systems = {"macro": ap._MACRO_SYSTEM, "technical": ap._TECH_SYSTEM, "risk": ap._RISK_SYSTEM, "contrarian": ap._CONTRARIAN_SYSTEM} - if stored and all(p in stored for p in PERSONAS): - return tuple((p, systems[p], stored[p]) for p in PERSONAS) + kept = {p: stored[p] for p in stored_personas(stored)} + if len(kept) == len(PERSONAS): + return tuple((p, systems[p], kept[p]) for p in PERSONAS) candles: Dict[str, Any] = {} analyses = [report.primary_analysis] + list(report.confirmation_analyses) counts = [int(settings.get("candle_bars_primary", ap.DEFAULT_CANDLE_BARS_PRIMARY))] + \ @@ -297,7 +308,8 @@ def build_jobs(report: SignalReport, context: FundamentalContext, if atr and atr > 0 and df is not None and len(df) >= 2: candles[analysis.timeframe] = ap._normalize_candles(df, atr, n) ctx = ap._price_context(report, frames, context) - return ap._panel_jobs(report, context, candles or None, ctx, PERSONAS) + rebuilt = ap._panel_jobs(report, context, candles or None, ctx, PERSONAS) + return tuple((p, system, kept.get(p, payload)) for p, system, payload in rebuilt) # ── Scoring ─────────────────────────────────────────────────────────── @@ -445,7 +457,9 @@ def main(argv: Optional[Sequence[str]] = None) -> int: "event_row": rep.id, "bar": report.timestamp, "tf": rep.timeframe, "direction": report.trade_setup.direction, "grade": rep.quality, "strategy": report.trade_setup.strategy, - "R": res.advisory_r if res else None, "stored_payloads": bool(stored), + # which personas replayed the exact logged payload ("" = all rebuilt) + "R": res.advisory_r if res else None, + "stored_payloads": ",".join(stored_personas(stored)), "base_p": base_p, "prior_move_4h_atr": ctx.get("prior_move_4h_atr"), "move_from_start_atr": ctx.get("move_from_24h_start_atr"), "big_candle_last3": ctx.get("big_candle_last3"), diff --git a/signal_aggregator.py b/signal_aggregator.py index f93bf2a..7e87f3c 100755 --- a/signal_aggregator.py +++ b/signal_aggregator.py @@ -14,6 +14,7 @@ from datetime import datetime, timezone from typing import Dict, List, Optional from zoneinfo import ZoneInfo +import numpy as np import pandas as pd _NY_TZ = ZoneInfo("America/New_York") @@ -155,6 +156,65 @@ def _transitioning_boost(abs_score: float, confidence: float, mtf: bool, mode: s return None +# ── Daily trend (campaign 2026-09-30) ──────────────────────────────── +# One definition for the live scan, the bar replay and the CUDA twin's host +# side (backtest/cudaBasedBacktesting.trend_flags calls daily_trend_series on +# the whole D1 frame and indexes it). Every step is causal (the Sunday +# filter, the EMA recursion, the shift), so the value on a prefix slice is +# bit-identical to the value at the same bar of the full series. +DAILY_TREND_MODES = ("off", "both", "buy", "sell") +DAILY_TREND_EMA_SPAN = 50 # validated literals (do not edit): the +DAILY_TREND_LOOKBACK = 60 # campaign tested exactly this definition +_DAILY_TREND_PREFIX = "DAILY TREND FILTER:" + + +def daily_trend_mode_of(xau_cfg) -> str: + """The validated `xau_strategy.daily_trend_veto`; an unknown value is a hard error.""" + mode = xau_cfg.daily_trend_veto + if mode not in DAILY_TREND_MODES: + raise ValueError( + f"xau_strategy.daily_trend_veto={mode!r}; expected one of {DAILY_TREND_MODES}" + ) + return mode + + +def daily_trend_needed(config: PlatformConfig) -> bool: + """True when the scan needs the D1 frame: the veto is on for a gold config.""" + return (daily_trend_mode_of(config.xau_strategy) != "off" + and config.xau_strategy.enabled + and config.display_name == "XAU/USD") + + +def daily_trend_series(d1: Optional[pd.DataFrame]) -> pd.Series: + """Daily trend per CLOSED D1 bar: +1 when the close is above its 50-bar + EMA and above the close 60 bars earlier, -1 when below both, else 0. + + `d1` must hold closed bars only (the provider drops the forming candle; + the replay slices as-of). Bars labelled Sunday are dropped first: the + provider's UTC-aligned daily candles include a 1-2 hour stub for the + Sunday reopening, which is not a trading day. The first 60 kept bars + read 0 (no 60-bar change yet).""" + if d1 is None or len(d1) == 0: + return pd.Series([], dtype="int8") + kept = d1[d1.index.weekday != 6] + close = kept["Close"].astype("float64") + ema = close.ewm(span=DAILY_TREND_EMA_SPAN, adjust=False).mean() + change = close / close.shift(DAILY_TREND_LOOKBACK) - 1.0 + up = (close > ema) & (change > 0) + down = (close < ema) & (change < 0) + state = np.where(up, 1, np.where(down, -1, 0)).astype("int8") + return pd.Series(state, index=kept.index) + + +def daily_trend(d1: Optional[pd.DataFrame]) -> Optional[int]: + """The daily trend at the last closed D1 bar of `d1` (+1 / 0 / -1), or + None when the frame holds no usable bar.""" + series = daily_trend_series(d1) + if len(series) == 0: + return None + return int(series.iloc[-1]) + + def _is_forex_weekend(now_utc: datetime) -> bool: """Gold/FX market closed for the weekend: Friday 17:00 → Sunday 18:00 NEW YORK time (= Fri 21:00 UTC in US summer, 22:00 in winter — @@ -284,6 +344,17 @@ class SignalReport: penalty_tags: Dict[str, float] = field(default_factory=dict) penalty_total: float = 0.0 momentum_fading: bool = False + # Daily trend at the last closed D1 bar (+1 up / 0 unclear / -1 down); + # None = not computed. Set by the core when the daily-trend veto is on + # (_apply_daily_trend_veto); where the veto is off the LIVE wrapper + # stamps it for the alert text and the signal log only + # (_stamp_daily_trend_for_display) — no gate reads it there. + daily_trend: Optional[int] = None + # Lot-size and open-exposure lines (advisory/DISPLAY-ONLY). Stamped + # post-hoc by surfaces via exposure.attach_exposure — never populated + # inside analyze_from_dataframes and never read by any gate. + size_line: str = "" + open_line: str = "" class SignalAggregator: @@ -292,6 +363,11 @@ class SignalAggregator: def __init__(self, config: PlatformConfig): self.config = config grading_mode_of(config.signals) # fail at start, not at the first bar + daily_trend_mode_of(config.xau_strategy) + if not config.xau_strategy.wide_stop_cap_atr >= 0: + raise ValueError( + f"xau_strategy.wide_stop_cap_atr={config.xau_strategy.wide_stop_cap_atr!r}; " + f"expected a number >= 0 (0 = off)") self.data_provider = make_data_provider(config) self.tech_engine = TechnicalEngine(config.technical) self._news_engine: Optional[NewsEngine] = None # lazy @@ -362,6 +438,12 @@ class SignalAggregator: bt = self.config.backtest if bt.regime_filter and bt.regime_filter_timeframe not in timeframes: timeframes.append(bt.regime_filter_timeframe) + # The daily-trend veto reads the D1 frame inside the core, so it is + # a REQUIRED frame when the veto is on (a failed fetch fails the + # scan below, as for any other frame). The backtesters append it + # under the same guard (rule 4). + if daily_trend_needed(self.config) and Timeframe.D1 not in timeframes: + timeframes.append(Timeframe.D1) dfs: Dict[Timeframe, pd.DataFrame] = {} stale_tfs = [] @@ -465,9 +547,34 @@ class SignalAggregator: f"{symbol}: live session filter errored — filter SKIPPED this scan" ) + self._stamp_daily_trend_for_display(report, dfs) self._log_signal(report, log_source, dfs) return report + def _stamp_daily_trend_for_display( + self, report: SignalReport, dfs: Dict[Timeframe, pd.DataFrame] + ) -> None: + """Daily trend for the alert text and the signal log on the + timeframes where the daily-trend veto is OFF (the core stamps it + where the veto is on). Display and record-keeping only: the frame + is never added to `dfs`, nothing reads the value back, and a failed + fetch leaves it None instead of failing the scan. The scan's own D1 + frame is used when it has one (H4 confirms on D1); otherwise the + provider's D1 frame, cached until the next daily close and + refreshed by the M15 scan, which needs it as a required frame. + Live wrapper only — the backtesters never come through here.""" + if report.daily_trend is not None or report.display_name != "XAU/USD": + return + if report.primary_analysis is None: + return + try: + d1 = dfs.get(Timeframe.D1) + if d1 is None: + d1 = self.data_provider.fetch_ohlcv(report.symbol, Timeframe.D1) + report.daily_trend = daily_trend(d1) + except Exception: + logger.debug("daily trend (display) unavailable", exc_info=True) + def _log_signal( self, report: SignalReport, source: Optional[str], dataframes: Optional[Dict[Timeframe, pd.DataFrame]] = None, @@ -541,6 +648,13 @@ class SignalAggregator: bt.session_end_hour], "regime_filter": bool(bt.regime_filter), } + # 2026-09-30 knobs enter the hash only when they are in force (the + # grading_mode precedent in engine_config_fingerprint): a config + # that leaves them off keeps the fingerprint its live rows carry. + if xs.daily_trend_veto != "off": + payload["daily_trend_veto"] = xs.daily_trend_veto + if xs.wide_stop_cap_atr > 0: + payload["wide_stop_cap_atr"] = xs.wide_stop_cap_atr return hashlib.sha1( json.dumps(payload, sort_keys=True, default=str).encode("utf-8") ).hexdigest()[:12] @@ -1032,6 +1146,14 @@ class SignalAggregator: if height < min_tp: return + # Wide-stop cap AFTER the gates (xau_strategy.wide_stop_cap_atr, + # 0 = off), same contract as the swing generator's. + cap_dist = self._wide_stop_cap(report, atr, min_sl_price) + if cap_dist is not None and sl_dist > cap_dist: + sl_dist = cap_dist + sl = entry - cap_dist if is_buy else entry + cap_dist + rr = height / sl_dist + # Printed-target floor AFTER the gates (2026-09-26, see # config.SignalConfig.tp_print_floor_rr): the measured move stays # the gated target; the printed guide is at least the floor. @@ -1215,23 +1337,26 @@ class SignalAggregator: report.mtf_agreement = self._check_mtf_agreement(report) # ── 4. Fundamental context (optional) ────────────────────── + # News is display and warning only (it never gates), so a failure + # anywhere in it must not cost the scan: the report then carries + # no news at all, the values the replay path (use_news=False) + # always has. All five fields are set together or not at all. + news_fields = ("clear", "", 0.0, "", "") if use_news: - news = self._get_news_engine() - if context is None: - context = news.get_fundamental_context() - event_risk, event_desc = news.get_event_risk_level(symbol, context) - report.event_risk = event_risk - report.event_description = event_desc - sentiment, sent_desc = news.get_pair_sentiment(symbol, context) - report.news_sentiment = sentiment - report.news_description = sent_desc - report.news_summary = context.llm_summary - else: - report.event_risk = "clear" - report.event_description = "" - report.news_sentiment = 0.0 - report.news_description = "" - report.news_summary = "" + try: + news = self._get_news_engine() + if context is None: + context = news.get_fundamental_context() + event_risk, event_desc = news.get_event_risk_level(symbol, context) + sentiment, sent_desc = news.get_pair_sentiment(symbol, context) + news_fields = (event_risk, event_desc, sentiment, sent_desc, + context.llm_summary) + except Exception: + logger.warning( + f"News context failed for {symbol} — the scan continues " + f"without news", exc_info=True) + (report.event_risk, report.event_description, report.news_sentiment, + report.news_description, report.news_summary) = news_fields # ── 5. Spread estimate from primary data (avoids extra API call) ─ try: @@ -1471,6 +1596,12 @@ class SignalAggregator: # cost-floor veto no longer describes it. self._drop_stale_spread_gate_vetoes(report) + # ── 12. Daily-trend veto (opt-in via per-TF factories) ─────── + # Last on purpose: a FINAL veto on whatever archetype shipped the + # setup (core, XAU_PB or XAU_CP), which is exactly what the + # campaign replayed. Mirrored in cuda_signal_kernels.py K2. + self._apply_daily_trend_veto(report, dfs) + return report # ── Internals ────────────────────────────────────────────────── @@ -1543,6 +1674,57 @@ class SignalAggregator: report.warnings[:] = kept report.cleaned_warnings.extend(stale) + def _apply_daily_trend_veto( + self, report: SignalReport, dfs: Dict[Timeframe, pd.DataFrame] + ) -> None: + """Step 12: remove a finished setup that goes against a clear daily + trend (xau_strategy.daily_trend_veto, off by default). + + Reads the D1 frame from `dfs` (closed bars only: the live wrapper + fetches it as a required frame, the replay slices it as-of), stamps + report.daily_trend, and follows the gate-chain convention when it + fires: grade D, no setup, a warning that names the gate. The final + signal and the action text are left as they are.""" + xcfg = self.config.xau_strategy + mode = daily_trend_mode_of(xcfg) + if mode == "off" or not xcfg.enabled or report.display_name != "XAU/USD": + return + state = daily_trend(dfs.get(Timeframe.D1)) + report.daily_trend = state + if state is None: + # Mixed case like the regime filter's twin: never a gate prefix. + report.warnings.append( + "Daily trend filter requested but 1d data unavailable") + return + setup = report.trade_setup + if setup is None or state == 0: + return + is_buy = setup.direction == "BUY" + against = ((is_buy and state < 0 and mode in ("both", "buy")) + or (not is_buy and state > 0 and mode in ("both", "sell"))) + if not against: + return + report.warnings.append( + f"{_DAILY_TREND_PREFIX} {setup.direction} against a " + f"{'falling' if is_buy else 'rising'} daily trend — no trade" + ) + report.signal_quality = "D" + report.trade_setup = None + + def _wide_stop_cap(self, report: SignalReport, atr: float, + stop_floor: float) -> Optional[float]: + """The largest stop distance xau_strategy.wide_stop_cap_atr allows + for a swing or chart-pattern setup: max(K x ATR, the setup's own + stop floor); None when the cap is off (K = 0) or the pair is not + gold. Applied by the generators AFTER their min-RR and spread + gates, so the gates keep judging the uncapped geometry and the + set of trades cannot change (the printed-target floor precedent).""" + xcfg = self.config.xau_strategy + k = xcfg.wide_stop_cap_atr + if not k > 0 or not xcfg.enabled or report.display_name != "XAU/USD": + return None + return max(k * atr, stop_floor) + def _printed_target_floor(self) -> float: """The floor every generator applies to the PRINTED target 1, as a fraction of the stop: max(tp_floor_rr, tp_print_floor_rr). Applied @@ -2024,6 +2206,18 @@ class SignalAggregator: ) return None + # Wide-stop cap AFTER the gates (xau_strategy.wide_stop_cap_atr, + # 0 = off): the gates above judged the swing-anchored stop, so the + # same trades ship; only the stop (the risk unit) is pulled in, + # never inside the generator's own floor. Mirrored in + # cuda_signal_kernels.py K2. + cap_dist = self._wide_stop_cap( + report, atr, max(atr * sl_atr_floor, min_sl_price)) + if cap_dist is not None and sl_distance > cap_dist: + sl_distance = cap_dist + sl = entry - cap_dist if is_buy else entry + cap_dist + rr = tp1_distance / sl_distance + # Printed-target floor AFTER the gates (2026-09-26, see # config.SignalConfig.tp_print_floor_rr): the gates above judged # the swing/ATR target, so the trade is unchanged; only the guide diff --git a/signal_log.py b/signal_log.py index 183e326..cfba623 100644 --- a/signal_log.py +++ b/signal_log.py @@ -84,6 +84,23 @@ or news_engine) so this module stays at the bottom of the import graph. time-boxed; batch driver: scripts/label_scan_outcomes.py). Readers must dedup by the FIRST row per (timeframe, bar_ts) — several surfaces log the same bar. + +2026-09-30 daily-trend veto: + +* Row column ``daily_trend`` (+1 / 0 / -1 from closed D1 bars: computed + by the core where the veto is on, stamped by the live wrapper for + display where it is off; NULL on blocked / failed scans and when the + daily frame could not be read) and the gate prefix ``DAILY TREND + FILTER`` in ``gate_reason``. +* Row column ``alerted_at`` (UTC time of the Telegram alert for this row; + ``mark_alerted``) and the read-only helper ``open_alerted_setups`` — + alerted plans whose advisory outcome is still PENDING — behind the + "already open" line on the alert (exposure.py). Since 2026-10-01 the + stamp is written just BEFORE the send and cleared again + (``unmark_alerted``) when no chat received the alert, so two alerts + sent at the same candle close count each other; ``last_alerted_bars`` + reads the newest alerted bar per timeframe so the bot's entry cooldown + survives a restart. """ import hashlib @@ -100,6 +117,7 @@ from typing import Any, Dict, Iterable, List, Optional, Tuple import app_config from advisory_outcome import ( + MAX_HOLD_BARS, advance_outcome, canonical_r, initial_outcome, @@ -240,6 +258,14 @@ _ADDITIVE_SIGNAL_COLS = ( ("panel_probs_json", "TEXT"), ("panel_payloads_json", "TEXT"), ("panel_context_json", "TEXT"), + # 2026-09-30 daily-trend veto (see module docstring). + ("daily_trend", "INTEGER"), + # 2026-09-30: UTC time of the Telegram alert for THIS row (mark_alerted: + # written by the bot just before the send, cleared by unmark_alerted + # when no chat received it). NULL on every row that was scanned but not + # alerted — the "which plans did the reader actually receive" key the + # open-exposure line reads (open_alerted_setups). + ("alerted_at", "TEXT"), ) # Read-side compatibility view `signals_with_news`: legacy rows keep @@ -262,6 +288,11 @@ _GATE_PREFIXES = ( "Primary timeframe stale", "Primary timeframe data unavailable", "DATA UNAVAILABLE:", + # The daily-trend veto removes a setup that every earlier gate had let + # through, so on such a row it IS the reason: listed ahead of the chain + # because the vetoed setup's stale warnings (a cleaned SPREAD GATE, the + # chart-pattern path's cleaned vetoes) are still in the pool. + "DAILY TREND FILTER", "REGIME FILTER", "EXHAUSTION BLOCK", "EXHAUSTED TREND FILTER", @@ -313,6 +344,7 @@ _SIGNAL_INSERT_COLS = ( "config_fingerprint", "bar_ts", "engine_fingerprint", "code_version", "penalty_tags_json", "penalty_total", "momentum_fading", "exhaustion_count", + "daily_trend", ) _INSERT = ( "INSERT INTO signals (" + ", ".join(_SIGNAL_INSERT_COLS) + ") VALUES (" @@ -330,6 +362,12 @@ def _finite(x: Any) -> Optional[float]: return f if math.isfinite(f) else None +def _daily_trend_value(x: Any) -> Optional[int]: + """The report's daily trend as -1 / 0 / +1, or None when it carries none.""" + f = _finite(x) + return None if f is None else int(f) + + def _iso(ts: Any) -> Optional[str]: if ts is None: return None @@ -815,6 +853,7 @@ def _report_row( "momentum_fading": int(bool(getattr(report, "momentum_fading", False))), "exhaustion_count": int( _finite(getattr(report, "exhaustion_count", 0)) or 0), + "daily_trend": _daily_trend_value(getattr(report, "daily_trend", None)), } @@ -903,6 +942,11 @@ class SignalLogger: self._conn.execute( "CREATE INDEX IF NOT EXISTS idx_signals_tf_bar" " ON signals (timeframe, bar_ts)") + # Partial index: only alerted rows (a few per day), so the + # open-exposure read on every scan never walks the table. + self._conn.execute( + "CREATE INDEX IF NOT EXISTS idx_signals_alerted" + " ON signals (symbol, alerted_at) WHERE alerted_at IS NOT NULL") self._conn.commit() def log( @@ -1106,6 +1150,31 @@ class SignalLogger: (str(level or ""), kinds_text, str(line or ""), int(rowid))) self._conn.commit() + def mark_alerted(self, rowid: int, + when_iso: Optional[str] = None) -> Optional[str]: + """Stamp the time of the alert for this row. The first stamp wins: + a re-alert of the same row keeps the original time. Returns the + stamp when THIS call wrote it and None when the row already had + one (or does not exist), so a caller that must undo its own stamp + can never clear an earlier alert's.""" + when_iso = when_iso or datetime.now(timezone.utc).isoformat() + with self._lock: + cur = self._conn.execute( + "UPDATE signals SET alerted_at=?" + " WHERE id=? AND alerted_at IS NULL", (when_iso, int(rowid))) + self._conn.commit() + return when_iso if cur.rowcount == 1 else None + + def unmark_alerted(self, rowid: int, when_iso: str) -> bool: + """Clear the stamp mark_alerted returned (the alert reached no + chat). Only that exact stamp is cleared; True when it was.""" + with self._lock: + cur = self._conn.execute( + "UPDATE signals SET alerted_at=NULL" + " WHERE id=? AND alerted_at=?", (int(rowid), str(when_iso))) + self._conn.commit() + return cur.rowcount == 1 + def resolve_pending( self, symbol: str, candles_json: Optional[str], limit: int = 100 ) -> int: @@ -1488,11 +1557,12 @@ class SignalLogger: old = json.loads(state_json) if state_json else {} if not _is_gap_reason(old.get("reason")): continue - # The locked trailing stop lives ONLY in this progress - # block, and a re-seed rebuilds the walk from scratch over - # a stream whose hole is still there (the continuity check - # is all-or-nothing), so the floor would be lost for good. - # Finish such a row here; re-seed only what has nothing. + # The locked trailing stop lives in this progress block. A + # re-seed would rebuild the walk from scratch only to stop + # at the same hole (advance_outcome checks continuity bar + # by bar and keeps the progress up to the hole) and be + # floored at the same stop, so finish such a row here; + # re-seed only what has nothing locked. floored = resolve_locked_floor(json.loads(_dumps(old))) if floored.get("status") == "RESOLVED": self._conn.execute( @@ -1673,6 +1743,151 @@ def log_watch_update(report: Any, watch: Any) -> None: exc_info=True) +def _logged_row(report: Any) -> Optional[Tuple["SignalLogger", int]]: + """(logger, rowid) of the row log_signal wrote for this scan; None for + a report that was never logged or a process with logging off.""" + rowid = getattr(report, "_signal_log_rowid", None) + if rowid is None or not app_config.signal_log_enabled(): + return None + path = app_config.signal_log_path() + with _loggers_lock: + lg = _loggers.get(path) + return None if lg is None else (lg, int(rowid)) + + +def mark_alerted(report: Any) -> Optional[str]: + """Stamp `alerted_at` on the row logged for this scan. The bot calls + this just BEFORE it sends the alert, so an alert sent a moment later + on another timeframe already counts this plan, and undoes it with + unmark_alerted when no chat received the message. Same hand-off as + log_panel_update (the private rowid from log_signal). Returns the + stamp this call wrote; None when nothing was written (never logged, + logging off, or the row already carried a stamp). NEVER raises.""" + try: + hit = _logged_row(report) + return None if hit is None else hit[0].mark_alerted(hit[1]) + except Exception: + logger.warning("signal_log: failed to mark the alerted row", + exc_info=True) + return None + + +def unmark_alerted(report: Any, stamp: Optional[str]) -> bool: + """Clear the stamp mark_alerted(report) returned: the alert reached no + chat, so the row must not count as a plan the reader holds. A no-op + without a stamp; only that exact stamp is cleared. NEVER raises.""" + if not stamp: + return False + try: + hit = _logged_row(report) + return False if hit is None else hit[0].unmark_alerted(hit[1], stamp) + except Exception: + logger.warning("signal_log: failed to clear the alerted stamp", + exc_info=True) + return False + + +# A PENDING outcome older than this is not shown as "open": the advisory +# model times a trade out after MAX_HOLD_BARS bars, so an older PENDING row +# is one the resolver never reached (bot outage, candle gap), not a trade +# still running. Calendar allowance: 7/5 for weekends plus three days. +def _open_setup_max_age_seconds(timeframe: Any) -> float: + bar = _TF_SECONDS.get(normalize_timeframe(timeframe), 3600) + return MAX_HOLD_BARS * bar * 1.4 + 3 * 86400 + + +def open_alerted_setups(symbol: str, exclude_key: Optional[str] = None, + now: Optional[datetime] = None, + path: Optional[str] = None) -> List[Dict[str, Any]]: + """ALERTED trade plans whose advisory outcome is still open, oldest + first: ``[{id, event_id, timeframe, direction, entry, stop_loss, + alerted_at, setup_key}]``, one entry per advisory event. + + Read-only (a ``mode=ro`` connection of its own, so the dashboard + process, which never writes this database, can call it). "Open" means + the row's advisory event is still PENDING in the cleaned-mid outcome + model: next-open entry, stop first, close-only trailing. That is the + model's view of a plan the reader MAY have taken, not a broker + position. ``exclude_key`` drops the plan being alerted now from its + own count. NEVER raises: any failure (no database yet, an older schema + without ``alerted_at``) returns an empty list.""" + try: + db_path = path or app_config.signal_log_path() + if not os.path.exists(db_path): + return [] + now = now or datetime.now(timezone.utc) + con = sqlite3.connect(f"file:{db_path}?mode=ro", uri=True, timeout=2.0) + try: + rows = con.execute( + "SELECT a.id, COALESCE(a.advisory_event_id, a.id), a.timeframe," + " a.setup_json, a.alerted_at, a.setup_key" + " FROM signals a JOIN signals e" + " ON e.id = COALESCE(a.advisory_event_id, a.id)" + " WHERE a.symbol=? AND a.alerted_at IS NOT NULL" + " AND e.advisory_outcome_status='PENDING'" + " ORDER BY a.id", (symbol,)).fetchall() + finally: + con.close() + out: List[Dict[str, Any]] = [] + seen = set() + for rowid, event_id, timeframe, setup_json, alerted_at, key in rows: + if event_id in seen or (exclude_key and key == exclude_key): + continue + try: + age = (now - datetime.fromisoformat(alerted_at)).total_seconds() + except (TypeError, ValueError): + continue + if age > _open_setup_max_age_seconds(timeframe): + continue + setup = json.loads(setup_json) if setup_json else {} + direction = setup.get("direction") + if direction not in ("BUY", "SELL"): + continue + seen.add(event_id) + out.append({ + "id": int(rowid), "event_id": int(event_id), + "timeframe": normalize_timeframe(timeframe), + "direction": direction, + "entry": _finite(setup.get("entry")), + "stop_loss": _finite(setup.get("stop_loss")), + "alerted_at": alerted_at, "setup_key": key, + }) + return out + except Exception: + logger.debug("signal_log: open alerted setups unavailable", + exc_info=True) + return [] + + +def last_alerted_bars(path: Optional[str] = None + ) -> Dict[Tuple[str, str], str]: + """``{(symbol, timeframe): bar label}`` of the newest ALERTED row per + symbol and timeframe (newest = highest id). The bot seeds its 5-bar + entry cooldown from this at start-up, so a restart does not forget the + last alert. Read-only (``mode=ro``); NEVER raises: no database, an + older schema or a locked file returns ``{}``.""" + try: + db_path = path or app_config.signal_log_path() + if not os.path.exists(db_path): + return {} + con = sqlite3.connect(f"file:{db_path}?mode=ro", uri=True, timeout=2.0) + try: + rows = con.execute( + "SELECT s.symbol, s.timeframe, s.bar_ts FROM signals s" + " JOIN (SELECT MAX(id) AS id FROM signals" + " WHERE alerted_at IS NOT NULL" + " GROUP BY symbol, timeframe) m ON m.id = s.id" + ).fetchall() + finally: + con.close() + return {(str(symbol), normalize_timeframe(timeframe)): str(bar) + for symbol, timeframe, bar in rows if bar} + except Exception: + logger.debug("signal_log: last alerted bars unavailable", + exc_info=True) + return {} + + def reset_loggers_for_tests() -> None: """Close and drop all cached loggers (test isolation).""" with _loggers_lock: diff --git a/static/dashboard.js b/static/dashboard.js index 6c4b12f..e6f8a28 100644 --- a/static/dashboard.js +++ b/static/dashboard.js @@ -549,6 +549,7 @@ function cardShell(tf) {

      +

      @@ -589,7 +590,7 @@ function buildCards() { TF_ORDER.forEach(tf => indChart(tf)); } -function planHTML(ts) { +function planHTML(ts, r) { if (!ts) { return `
      @@ -601,6 +602,11 @@ function planHTML(ts) { const trail = (ts.trail_arm_r != null && ts.trail_dist_r != null) ? `Exit rule (tested): once the trade is ${ts.trail_arm_r}R in profit, the stop trails ${ts.trail_dist_r}R behind the best price, moved at candle close. Until then the stop is the exit; targets are guides.` : "Exit rule: the trailing stop uses the platform default; targets are guides."; + // Lot size (present only when the owner shares it on this page) and the + // alerted plans still open in this direction. Both lines are written by + // exposure.py; the page prints them as they come, no arithmetic here. + const notes = [r && r.size_line, r && r.open_line].filter(Boolean) + .map(t => `
      ${esc(t)}
      `).join(""); return `
      @@ -618,7 +624,7 @@ function planHTML(ts) {
      Target 2
      ${fmt$(ts.take_profit_2)}
      -
      ${esc(trail)}
      +
      ${esc(trail)}
      ${notes}
      `; } @@ -673,6 +679,9 @@ function renderCard(tf) { : actionTail(r.action)); // Plain words for the two numbers below (from main.plain_words, same on every surface). f("plain").toggleClass("d-none", !r.plain_words).text(r.plain_words || ""); + // The daily trend in words (main.daily_trend_words). On M15 a plan against + // a clear daily trend is removed by the signal core; elsewhere this is context. + f("trend").toggleClass("d-none", !r.daily_trend_words).text(r.daily_trend_words || ""); const score = r.score == null ? 0 : r.score; f("score").text(signed(r.score, 1)) @@ -692,7 +701,7 @@ function renderCard(tf) { r.mtf_agreement ? "chip-up" : "") + (r.exhaustion_count ? chip("fa-battery-quarter", `${r.exhaustion_count} exhaustion signs`, "chip-warn") : "")); - f("plan").html(planHTML(r.trade_setup)); + f("plan").html(planHTML(r.trade_setup, r)); f("flags").html(flagsHTML(r, 2)); updateIndChart(tf); } @@ -838,11 +847,13 @@ function detailHTML(tf) { ${tile("Event risk", esc(r.event_risk || "—"), r.event_risk === "clear" ? "up" : r.event_risk ? "text-warning" : "")} ${tile("News tone", ns != null ? signed(ns, 2) : "—", ns > 0.05 ? "up" : ns < -0.05 ? "down" : "")} ${tile("Timeframes", r.mtf_agreement ? "agree" : "mixed", r.mtf_agreement ? "up" : "")} + ${tile("Daily trend", r.daily_trend == null ? "—" : r.daily_trend > 0 ? "rising" : r.daily_trend < 0 ? "falling" : "no clear direction", + r.daily_trend > 0 ? "up" : r.daily_trend < 0 ? "down" : "")} ${tile("This timeframe alone", signed(r.primary_score, 1), toneClass(r.primary_signal))} ${tile("Exhaustion signs", esc(r.exhaustion_count == null ? "—" : r.exhaustion_count), r.exhaustion_count ? "text-warning" : "")}
      `); - out += sec("Trade plan", planHTML(r.trade_setup)); + out += sec("Trade plan", planHTML(r.trade_setup, r)); out += sec("Why this signal", reasonsBlock(r) || `
      Nothing was recorded on this scan.
      `); out += sec(`${TF_LABEL[tf]} indicators`, analysisBlock(pa, price) || indTable(r.indicators) + diff --git a/telegram_bot.py b/telegram_bot.py index 162756b..4f59c27 100755 --- a/telegram_bot.py +++ b/telegram_bot.py @@ -74,11 +74,13 @@ from app_config import ( from providers import BaseDataProvider, make_data_provider from providers.base import _is_daily_maintenance from agent_panel import attach_panel, panel_health +from exposure import attach_exposure, size_shared from llm_client import get_llm_client from news_engine import FundamentalContext, NewsEngine from pair_signals import PAIR_PRODUCERS from signal_aggregator import SignalAggregator, SignalReport -from signal_log import gate_reason +from signal_log import (gate_reason, last_alerted_bars, mark_alerted, + unmark_alerted) from violent_watch import attach_watch, should_send_headsup if TYPE_CHECKING: # type hints only — ViolentWatch is never needed at runtime here @@ -199,6 +201,16 @@ TF_CONFIRMATIONS = { } _executor = ThreadPoolExecutor(max_workers=4) +# ONE worker for an alert's database steps (refresh the "already open" line, +# stamp the row as alerted, undo the stamp). A single worker is the mutual +# exclusion: one alert's read and write run back to back, in arrival order, +# so of two alerts sent at the same candle close the second counts the +# first. It also keeps SQLite off the event loop (the logger's lock can be +# held for seconds by the outcome labeller) and never queues behind the +# scan pool above. Not an asyncio.Lock: on Python 3.9 one created at +# import binds to a loop that is not the running one. +_alert_db_executor = ThreadPoolExecutor( + max_workers=1, thread_name_prefix="alert-db") # ── Markdown escaping ───────────────────────────────────────────────── @@ -409,7 +421,23 @@ def _plain_words_line(report: SignalReport) -> str: f"Reliability: *{reliability_words(report.final_confidence)}* · {agree}") -def format_signal_message(report: SignalReport) -> str: +def _daily_trend_line(report: SignalReport) -> str: + """"🧭 Daily trend: rising — this buy goes with it" (main.daily_trend_words, + the same words on every surface), or "" when the scan carries no daily + trend. On M15 a plan against a clear daily trend never reaches an + alert (the core removes it); on the other timeframes the line is + information only.""" + from main import daily_trend_words # display helper, no side effects + words = daily_trend_words(report) + return f"🧭 {_escape_md(words)}" if words else "" + + +def format_signal_message(report: SignalReport, channel: bool = False) -> str: + """The alert / scan text. `channel=True` builds the copy for every + audience but the owner's own chat (the subscriber channel, a group + the bot was switched on in): identical, except that the lot-size line + (which reveals the owner's risk per trade) is left out unless + config.json sets trade_size.telegram_channel.""" sig = report.final_signal emoji = _SIGNAL_EMOJI.get(sig, "⬜") qe = _QUALITY_EMOJI.get(report.signal_quality, "") @@ -431,6 +459,9 @@ def format_signal_message(report: SignalReport) -> str: _plain_words_line(report), f"📊 Regime: {regime_str} · Vol: {vol_str}", ] + trend_line = _daily_trend_line(report) + if trend_line: + lines.append(trend_line) if report.action_recommendation: # Pattern identifiers use underscores, which legacy Telegram Markdown # can misparse even when escaped inside this italicized action line. @@ -457,6 +488,15 @@ def format_signal_message(report: SignalReport) -> str: f"▶ Entry `{_fmt_price(ts.entry_price)}`", f"🛑 SL `{_fmt_price(ts.stop_loss)}` (${risk_usd:.2f} / {ts.risk_pips:.0f}p = 1R)", ] + # Lot size for the owner's risk limit and the alerted plans still + # open in this direction (exposure.attach_exposure stamps both + # post-hoc; empty unless configured / unless something is open). + # The size is private: the channel copy carries it only when + # config.json says so. + if report.size_line and (not channel or size_shared("telegram_channel")): + lines.append(f"📏 {_escape_md(report.size_line)}") + if report.open_line: + lines.append(f"🧱 {_escape_md(report.open_line)}") # Trailing-stop management plan (matches backtested exit model: # TP1 is advisory, the trail is the exit). Params are stamped on @@ -611,7 +651,13 @@ def _run_signal_sync( config = PAIR_PRODUCERS[symbol](timeframe, confirmation_tfs) aggregator = SignalAggregator(config) aggregator.data_provider = _get_shared_dp() - return aggregator.analyze_pair(symbol=symbol, log_source="telegram") + report = aggregator.analyze_pair(symbol=symbol, log_source="telegram") + # Display-only lot-size and "already open" lines (never raises; one + # read-only query on signals.db). Stamped here, on the scan executor, + # for the manual scans. An ALERT refreshes them just before it is sent + # (_deliver_alert): every timeframe scans at the same candle close, so + # the line stamped here cannot know about an alert going out beside it. + return attach_exposure(report, surface="telegram") async def run_signal( @@ -761,11 +807,10 @@ def _key_bar_seconds(key: str) -> Optional[int]: return None -def _report_bar_ts(report: SignalReport) -> Optional[pd.Timestamp]: - """The report's primary bar label as a UTC timestamp; None when it - cannot be parsed (the cooldown then does not apply).""" +def _bar_label_ts(label: Any) -> Optional[pd.Timestamp]: + """A bar label as a UTC timestamp; None when it cannot be parsed.""" try: - ts = pd.Timestamp(report.timestamp) + ts = pd.Timestamp(label) except Exception: return None if pd.isna(ts): @@ -773,9 +818,45 @@ def _report_bar_ts(report: SignalReport) -> Optional[pd.Timestamp]: return ts.tz_localize("UTC") if ts.tzinfo is None else ts.tz_convert("UTC") +def _report_bar_ts(report: SignalReport) -> Optional[pd.Timestamp]: + """The report's primary bar label as a UTC timestamp; None when it + cannot be parsed (the cooldown then does not apply).""" + return _bar_label_ts(report.timestamp) + + +def _primary_bars_since(key: str, report: SignalReport, last: pd.Timestamp, + now_bar: pd.Timestamp) -> Optional[int]: + """How many primary bars lie after `last`, up to and including the + report's own bar, in the frame the scan analysed (report._frames): + the backtester's position difference `pos - last_entry_pos`. None when + the report carries no frame for the key's timeframe or its bar is not + in it; the caller then falls back to clock time.""" + try: + tf = Timeframe(key.rsplit("_", 1)[-1]) + df = (getattr(report, "_frames", None) or {}).get(tf) + if df is None or len(df) == 0: + return None + idx = df.index + if getattr(idx, "tz", None) is None: + idx = idx.tz_localize("UTC") + if now_bar not in idx: + return None + return int(((idx > last) & (idx <= now_bar)).sum()) + except Exception: + return None + + def _inside_alert_cooldown(key: str, report: SignalReport) -> bool: """True while this key sits within _ALERT_COOLDOWN_BARS primary bars - of its last ALERTED setup's bar (the backtester's entry cooldown).""" + of its last ALERTED setup's bar (the backtester's entry cooldown). + + Bars are COUNTED in the scan's own primary frame, like the + backtester's bar positions. Clock time is not the same thing: across + the daily settlement pause or the weekend an hour or two days pass + while no bar prints, and a clock rule released the 22:00 UTC bar that + no book ever enters (six M15 and three M5 entries in three years, + 2026-10-01 review). Clock time remains the fallback for a report + without frames.""" last = _last_alert_bar.get(key) if last is None or report.trade_setup is None: return False @@ -783,8 +864,30 @@ def _inside_alert_cooldown(key: str, report: SignalReport) -> bool: now_bar = _report_bar_ts(report) if secs is None or now_bar is None: return False - elapsed = (now_bar - last).total_seconds() - return 0 <= elapsed < _ALERT_COOLDOWN_BARS * secs + if now_bar < last: + return False + bars = _primary_bars_since(key, report, last, now_bar) + if bars is not None: + return bars < _ALERT_COOLDOWN_BARS + return (now_bar - last).total_seconds() < _ALERT_COOLDOWN_BARS * secs + + +def _restore_alert_cooldowns(bars: Dict[Any, str]) -> int: + """Seed the cooldown memory from signal_log.last_alerted_bars() after a + start-up: {(symbol, timeframe): bar label} of the newest alerted row. + Without it every restart forgot the last alert and the next bar could + alert again inside the 5 bars. Only the cooldown is restored; the + dedup fingerprints start empty, as before. A key already in memory is + left alone. Returns how many keys were seeded.""" + restored = 0 + for (symbol, timeframe), label in (bars or {}).items(): + bar = _bar_label_ts(label) + key = f"{symbol}_{timeframe}" + if bar is None or key in _last_alert_bar: + continue + _last_alert_bar[key] = bar + restored += 1 + return restored @@ -844,6 +947,10 @@ def _should_alert(key: str, report: SignalReport) -> bool: def _commit_alert(key: str, report: SignalReport) -> None: + # In-memory dedup state only. The signal-log stamp of the alert is + # written (and, for an undelivered alert, undone) by _deliver_alert on + # the alert-db worker: a database write does not belong on the event + # loop this function runs on. _last_alert_fp[key] = _alert_fingerprint(report) _last_alert_ts[key] = time.time() _last_alert_quality[key] = report.signal_quality @@ -898,17 +1005,123 @@ async def _send_alert(bot, chat_id: int, text: str) -> bool: return False -async def _broadcast(bot, text: str) -> int: +def _is_owner_chat(chat_id: Any) -> bool: + """True for the owner's own chat with the bot, the one audience of the + private lot-size line. The owner lock is per USER, so /autoscan or a + scan button used inside a group makes that group a target: it is not + the owner's chat and gets the copy without the size.""" + return OWNER_ID is not None and chat_id == OWNER_ID + + +async def _broadcast(bot, text: str, channel_text: Optional[str] = None) -> int: """Send `text` to every signal target; returns how many chats got it. Shared by the scan and breach paths so the delivery rule cannot - drift between them.""" + drift between them. `channel_text`, when given, is what every target + except the owner's own chat receives instead (the copy without the + owner's private lot-size line).""" delivered = 0 for chat_id in _signal_targets(): - if await _send_alert(bot, chat_id, text): + body = (text if channel_text is None or _is_owner_chat(chat_id) + else channel_text) + if await _send_alert(bot, chat_id, body): delivered += 1 return delivered +def _stamp_exposure_and_mark(report: SignalReport) -> Optional[str]: + """On the alert-db worker: refresh the lot-size and "already open" + lines, then stamp the report's signal-log row as alerted. Read first, + write second, nothing in between: the next alert on the worker sees + this stamp. Returns the stamp to undo if no chat receives the alert + (None when nothing was written). Never raises.""" + attach_exposure(report, surface="telegram") + return mark_alerted(report) + + +def _undo_cancelled_stamp(report: SignalReport, done: Any) -> None: + """Done-callback of the stamp job when the alert's task was cancelled + while that job was queued or running (a shutdown, or auto-scan switched + off, at the wrong moment). Cancelling a task cannot stop a job the + worker has already started, so its stamp lands although nothing will be + sent: clear it. Runs on the alert-db worker right after the job, also + when the pool has already been told to shut down. Never raises.""" + try: + if done.cancelled(): + return # never ran: nothing was written + stamp = done.result() + if stamp: + unmark_alerted(report, stamp) + except Exception: + logger.warning("alert: could not clear the stamp of a cancelled " + "alert", exc_info=True) + + +async def _undo_alert_mark(report: SignalReport, stamp: Optional[str]) -> None: + """The alert reached no chat: clear the stamp this attempt wrote (and + only that one), so the row does not count as a plan the reader holds. + Never raises.""" + if not stamp: + return + try: + await asyncio.get_running_loop().run_in_executor( + _alert_db_executor, unmark_alerted, report, stamp) + except Exception: + logger.warning("alert: could not clear the alerted stamp of an " + "undelivered alert", exc_info=True) + + +async def _deliver_alert(bot, key: str, report: SignalReport, tf_label: str, + header: str = "", kind: str = "alert") -> int: + """Send one alert: panel → exposure lines + alerted stamp → both + copies → commit. The ONE delivery path of the candle scan and the + level break; the caller holds _alert_lock(key) and has passed + _should_alert. Returns how many chats received it. + + The row is stamped BEFORE the send so that an alert going out at the + same candle close on another timeframe already counts this plan in its + "already open" line (2026-10-01: stamped after the send, two alerts at + one boundary never counted each other). A send that reaches no chat + undoes the stamp and commits nothing, so the setup stays eligible for + the next scan; so does a cancellation before the send has started. + Residue: if this send fails completely after another alert has already + counted it, that alert's line is one too high.""" + loop = asyncio.get_running_loop() + # Panel AFTER the quality gate + dedup decision so no LLM spend on + # suppressed alerts. Runs on the scan executor to keep the event loop + # free; this key's alert lock is held for the panel's wall time + # (≤~25 s), which only delays a same-key duplicate that _should_alert + # would drop anyway. + await loop.run_in_executor(_executor, attach_panel, report) + # submit(), not run_in_executor(): the job's own future is kept so that + # a cancellation HERE (nothing has been sent yet) can still undo the + # stamp the job writes. + job = _alert_db_executor.submit(_stamp_exposure_and_mark, report) + try: + stamp = await asyncio.wrap_future(job, loop=loop) + except asyncio.CancelledError: + job.add_done_callback( + functools.partial(_undo_cancelled_stamp, report)) + raise + delivered = 0 + try: + detail = header + format_signal_message(report) + delivered = await _broadcast( + bot, detail, header + format_signal_message(report, channel=True)) + except Exception: + # (A cancellation is not an Exception: on shutdown mid-send a chat + # may already hold the message, so the stamp stays.) + await _undo_alert_mark(report, stamp) + raise + if not delivered: + await _undo_alert_mark(report, stamp) + return 0 + # Commit only when the alert reached at least one target; a fully + # failed send stays eligible for the next scan. + _commit_alert(key, report) + logger.info(_alert_sent_line(report, tf_label, delivered, kind=kind)) + return delivered + + # ── Candle-boundary scheduler ──────────────────────────────────────── _autoscan_chats: Set[int] = set() @@ -1062,20 +1275,7 @@ async def _scan_timeframe( async with _alert_lock(key): if not _should_alert(key, report): continue - # Panel AFTER the quality gate + dedup decision so no LLM - # spend on suppressed alerts. Runs on the scan executor to - # keep the event loop free; this key's alert lock is held - # for the panel's wall time (≤~25 s), which only delays a - # same-key duplicate that _should_alert would drop anyway. - await asyncio.get_running_loop().run_in_executor( - _executor, attach_panel, report) - detail = format_signal_message(report) - delivered = await _broadcast(bot, detail) - # Commit only when the alert reached at least one target; - # a fully failed send stays eligible for the next scan. - if delivered: - _commit_alert(key, report) - logger.info(_alert_sent_line(report, tf.value, delivered)) + await _deliver_alert(bot, key, report, tf.value) # ── WebSocket level-breach monitor ────────────────────────────────── @@ -1222,20 +1422,16 @@ async def _handle_breach(bot, pair_symbol: str, price: float, level_name: str) - if not _should_alert(key, report): return - # Same post-gate panel attach as the auto-scan path (see - # _scan_timeframe for the spend/lock rationale). - await asyncio.get_running_loop().run_in_executor( - _executor, attach_panel, report) + # The same delivery as the auto-scan path (panel, the refreshed + # "already open" line, stamp, send, commit), under the level-break + # header. The cached M15 report this path reuses can be up to a + # bar old; _deliver_alert refreshes its exposure lines. header = ( f"⚡ *LEVEL BREAK* — {_escape_md(display)} crossed " f"*{_escape_md(label)}* @ `{_fmt_price(price)}`\n\n" ) - detail = header + format_signal_message(report) - sent = await _broadcast(bot, detail) - if sent: - _commit_alert(key, report) - logger.info(_alert_sent_line(report, Timeframe.M15.value, sent, - kind="breach alert")) + await _deliver_alert(bot, key, report, Timeframe.M15.value, + header=header, kind="breach alert") def _breach_task_done(task: asyncio.Task) -> None: @@ -1517,7 +1713,9 @@ def _news_health_lines() -> List[str]: Reads the class-level `NewsEngine._source_health` written by the scan-path fetches plus the LLM client's health stamps — display only, never triggers a fetch. ⚠️ marks a degraded dependency; - details live in the rotating log (transition-logged WARNINGs).""" + details live in the rotating log (transition-logged WARNINGs). + Failure details are exception text, so they are Markdown-escaped: an + unpaired underscore in one would make Telegram reject the reply.""" lines = ["\n*News & AI:*"] # Snapshot: a scan thread may insert a source's FIRST entry while we # iterate (dict-changed-size RuntimeError); entry mutation is safe. @@ -1536,11 +1734,13 @@ def _news_health_lines() -> List[str]: ff = sh.get("calendar:forexfactory") if cal: if NewsEngine.source_is_failing(cal): - lines.append(f" ⚠️ Calendar: unavailable — {cal['detail']}" + lines.append(" ⚠️ Calendar: unavailable — " + f"{_escape_md(str(cal['detail']))}" f" ({_fmt_age(cal['err'])})") elif NewsEngine.source_is_failing(ff): lines.append(f" ⚠️ Calendar: {cal['extra']} — ForexFactory" - f" down: {ff['detail']} ({_fmt_age(ff['err'])})") + f" down: {_escape_md(str(ff['detail']))}" + f" ({_fmt_age(ff['err'])})") else: lines.append(f" ✅ Calendar: {cal['extra']}" f" ({_fmt_age(cal['ok'])} ago)") @@ -1548,13 +1748,27 @@ def _news_health_lines() -> List[str]: if heads: failing = NewsEngine.source_is_failing(heads) mark = "⚠️" if failing else "✅" - detail = f" — {heads['detail']}" if failing else "" + detail = (f" — {_escape_md(str(heads['detail']))}" + if failing else "") ts = heads["err"] if failing else heads["ok"] lines.append(f" {mark} Headlines: {heads['extra']}{detail}" f" ({_fmt_age(ts)} ago)") + # The local headline scorer (news_model modes fallback / primary). + # Present only in a process that ran it; a failing batch leaves the + # cycle's headlines on keyword scores, which nothing else showed. + model = sh.get("news_model") + if model: + if NewsEngine.source_is_failing(model): + lines.append(" ⚠️ News model: not scoring — " + f"{_escape_md(str(model['detail']))}" + f" ({_fmt_age(model['err'])})") + else: + lines.append(f" ✅ News model: {model['extra']}" + f" ({_fmt_age(model['ok'])} ago)") for name, entry in sorted(sh.items()): if name.startswith("rss:") and NewsEngine.source_is_failing(entry): - lines.append(f" ⚠️ {name[4:]}: {entry['detail']}" + lines.append(f" ⚠️ {_escape_md(name[4:])}: " + f"{_escape_md(str(entry['detail']))}" f" ({_fmt_age(entry['err'])})") client = get_llm_client() if not client.available: @@ -1786,7 +2000,12 @@ async def cb_scan(update: Update, context: ContextTypes.DEFAULT_TYPE) -> None: try: report = await run_signal(btn["symbol"], btn["tf"]) - text = format_signal_message(report) + # The reply lands in the chat the button was pressed in. Only the + # owner's own chat gets the private lot-size line; the owner + # pressing it inside a group must not show the group their risk. + chat = update.effective_chat + text = format_signal_message( + report, channel=not _is_owner_chat(chat.id if chat else None)) except Exception as exc: logger.exception("Signal scan failed") text = f"❌ Scan failed: {_escape_md(str(exc))}" @@ -1914,6 +2133,18 @@ async def _post_init(app: Application) -> None: if CHAT_ID: _autoscan_chats.add(int(CHAT_ID)) + # The 5-bar entry cooldown lives in memory; seed it from the signal log + # so a restart (a deploy, the weekly timer) cannot re-alert inside it. + # Before the feed check: /autoscan can start the loops later. + try: + bars = await asyncio.get_running_loop().run_in_executor( + _alert_db_executor, last_alerted_bars) + restored = _restore_alert_cooldowns(bars) + logger.info(f"Alert cooldown restored from the signal log: " + f"{restored} timeframe(s)") + except Exception: + logger.exception("Alert cooldown restore failed — starting without it") + if not _feed_active(): logger.info("Neither TELEGRAM_CHAT_ID nor TELEGRAM_CHANNEL_ID set — " "auto-scan disabled on boot. Use /autoscan in chat to enable.") @@ -1980,6 +2211,7 @@ async def _post_shutdown(app: Application) -> None: logger.info("Shutting down ForexBot2...") _stop_scheduler() _executor.shutdown(wait=False) + _alert_db_executor.shutdown(wait=False) logger.info("Shutdown complete") diff --git a/test_all.py b/test_all.py index 1d47f2f..708668f 100644 --- a/test_all.py +++ b/test_all.py @@ -206,6 +206,9 @@ def _(): assert isinstance(xc.blocked_hours_utc, list) assert xc.sell_blocked_hours_utc == [], \ "sell hour gate must be off by default (per-TF factories opt in)" + assert xc.daily_trend_veto == "off", \ + "daily-trend veto must be off by default (per-TF factories opt in)" + assert xc.wide_stop_cap_atr == 0.0, "wide-stop cap must be off by default" @test("FilterConfig defaults") @@ -1153,6 +1156,20 @@ def _(): wide = ne.get_week_events(currencies=["USD", "EUR"]) assert calls["n"] == 1, f"Expected 1 fetch, got {calls['n']}" assert [e.event for e in wide] == ["Past Event", "ECB Rate", "NFP"] + + # peek_week_events: the same view, but it NEVER fetches. With the + # cache expired get_week_events refetches; the peek returns what is + # held, however old (the agent panel must not wait on the network). + stamp, held = NewsEngine._calendar_cache + NewsEngine._calendar_cache = (stamp - 10 * ne.config.context_cache_ttl_seconds, held) + assert [e.event for e in ne.peek_week_events()] == ["Past Event", "NFP"] + assert [e.event for e in ne.peek_week_events(currencies=["USD", "EUR"])] == \ + ["Past Event", "ECB Rate", "NFP"] + assert calls["n"] == 1, "the peek must not have fetched" + ne.get_week_events() + assert calls["n"] == 2, "the ordinary accessor does refresh an expired cache" + NewsEngine._calendar_cache = (0.0, []) + assert ne.peek_week_events() == [] and calls["n"] == 2, "an empty cache is an empty list, not a fetch" finally: NewsEngine._calendar_cache = (0.0, []) @@ -1358,8 +1375,12 @@ def _(): assert _transitioning_boost(abs(s), c, m, "pair") == want, (s, c, m) n += 1 assert n > 800 - # the real report's grade is the helper of its own final values + # the real report's grade is the helper of its own final values. Recomputed + # like the 'min' / 'product' tests below: the fixture ends at the wall-clock + # hour, so a report straight out of analyze_from_dataframes is forced to F + # by gate 6b whenever the suite runs in a blocked hour (2026-09-30). config, agg, report = _grading_report() + agg._compute_final_signal(report) assert report.signal_quality == _quality_pair(abs(report.final_score), report.final_confidence, report.mtf_agreement) assert report.final_signal == _signal_pair(report.final_score, report.final_confidence) @@ -2345,6 +2366,8 @@ def _(): } # Grade-A SELLs blocked on M15 (2026-09-26 campaign). inline_cfg.xau_strategy.sell_min_quality = ["A+", "B"] + # Daily-trend veto on M15 (2026-09-30 campaign, override 7). + inline_cfg.xau_strategy.daily_trend_veto = "both" if tf == Timeframe.M5: # Documented per-TF overrides (see xauusd.py docstring): # hour 9 UTC blocked, hour 21 UNBLOCKED (2026-07-30 @@ -2910,6 +2933,170 @@ def _(): d.clear(); d.update(old) +def _m11_cooldown_report(bar, frame=None, tf=Timeframe.M15): + """A SELL setup on `bar`; with `frame`, the bars the scan analysed (up to its own bar).""" + from signal_aggregator import SignalReport, TradeSetup + r = SignalReport(symbol="XAU/USD", display_name="XAU/USD", timestamp=str(bar)) + r.final_signal = SignalStrength.SELL + r.signal_quality = "A" + r.trade_setup = TradeSetup( + pair="XAU/USD", display_name="XAU/USD", direction="SELL", + entry_price=4300.0, stop_loss=4330.0, take_profit_1=4270.0, + take_profit_2=4255.0, risk_reward=1.0, risk_pips=300.0, + reward_pips=300.0, pip_value=0.10, strategy="GENERIC") + if frame is not None: + r._frames = {tf: frame.loc[:pd.Timestamp(bar)]} + return r + + +def _m11_frame(labels): + idx = pd.DatetimeIndex(labels) + return pd.DataFrame({"Open": 1.0, "High": 1.0, "Low": 1.0, "Close": 1.0, "Volume": 1.0}, index=idx) + + +@test("Alert cooldown counts the scan's own bars, not clock time: held across the settlement pause and " + "the weekend exactly as long as the backtester holds it (2026-10-01)") +def _(): + import telegram_bot as tb + dicts = (tb._last_alert_fp, tb._last_alert_ts, tb._last_alert_setup, + tb._last_alert_quality, tb._last_alert_bar) + saved = [dict(d) for d in dicts] + try: + for d in dicts: + d.clear() + # Tuesday 2026-09-08 (summer): M15 labels up to 20:45, the pause, then 22:00 onward + m15 = _m11_frame(list(pd.date_range("2026-09-08 19:00", "2026-09-08 20:45", freq="15min", tz="UTC")) + + list(pd.date_range("2026-09-08 22:00", "2026-09-08 23:00", freq="15min", tz="UTC"))) + key = "XAU/USD_15min" + tb._commit_alert(key, _m11_cooldown_report("2026-09-08 20:15:00+00:00", m15)) + held = {} + for bar in ("20:30", "20:45", "22:00", "22:15", "22:30", "22:45"): + label = f"2026-09-08 {bar}:00+00:00" + held[bar] = tb._inside_alert_cooldown(key, _m11_cooldown_report(label, m15)) + # 20:30, 20:45, 22:00, 22:15 are bars 1-4 after the alert; 22:30 is the fifth + assert held == {"20:30": True, "20:45": True, "22:00": True, "22:15": True, + "22:30": False, "22:45": False}, held + assert tb._primary_bars_since(key, _m11_cooldown_report("2026-09-08 22:00:00+00:00", m15), + tb._last_alert_bar[key], pd.Timestamp("2026-09-08 22:00", tz="UTC")) == 3 + # the clock rule (still the fallback for a report without frames) released 22:00: + # 105 minutes had passed, but only three bars + assert not tb._inside_alert_cooldown(key, _m11_cooldown_report("2026-09-08 22:00:00+00:00")) + # the same bar again, and an older bar: never "inside" by accident + assert tb._inside_alert_cooldown(key, _m11_cooldown_report("2026-09-08 20:15:00+00:00", m15)) + assert not tb._inside_alert_cooldown(key, _m11_cooldown_report("2026-09-08 20:00:00+00:00", m15)) + + # H1 across the weekend: Friday 19:00 alert; Friday 20:00 is the week's last bar, + # Sunday 22:00 the first of the next + h1 = _m11_frame(list(pd.date_range("2026-09-11 12:00", "2026-09-11 20:00", freq="1h", tz="UTC")) + + list(pd.date_range("2026-09-13 22:00", "2026-09-14 06:00", freq="1h", tz="UTC"))) + hkey = "XAU/USD_1h" + tb._commit_alert(hkey, _m11_cooldown_report("2026-09-11 19:00:00+00:00", h1, Timeframe.H1)) + got = [tb._inside_alert_cooldown(hkey, _m11_cooldown_report(str(bar), h1, Timeframe.H1)) + for bar in h1.index[h1.index > pd.Timestamp("2026-09-11 19:00", tz="UTC")][:6]] + assert got == [True, True, True, True, False, False], got # Fri 20:00, Sun 22:00-Mon 00:00 | Mon 01:00 + # a naive index, a frame for another timeframe, a bar missing from the frame: + # the count is unavailable and the clock rule decides, never an exception + last = tb._last_alert_bar[key] + now_bar = pd.Timestamp("2026-09-08 22:00", tz="UTC") + naive = _m11_cooldown_report("2026-09-08 22:00:00+00:00") + naive._frames = {Timeframe.M15: m15.tz_localize(None)} + assert tb._primary_bars_since(key, naive, last, now_bar) == 3 + wrong_tf = _m11_cooldown_report("2026-09-08 22:00:00+00:00") + wrong_tf._frames = {Timeframe.H1: h1} + assert tb._primary_bars_since(key, wrong_tf, last, now_bar) is None + missing = _m11_cooldown_report("2026-09-08 22:00:00+00:00") + missing._frames = {Timeframe.M15: m15.drop(now_bar)} + assert tb._primary_bars_since(key, missing, last, now_bar) is None + assert tb._primary_bars_since("XAU/USD_nonsense", naive, last, now_bar) is None + empty = _m11_cooldown_report("2026-09-08 22:00:00+00:00") + empty._frames = {Timeframe.M15: m15.iloc[:0]} + assert tb._primary_bars_since(key, empty, last, now_bar) is None + finally: + for d, old in zip(dicts, saved): + d.clear(); d.update(old) + + +@test("Alert cooldown == gpu_backtest._apply_cooldown: on a week of M15 bars the bars live would alert " + "are the bars the backtester enters; the old clock rule added entries after the settlement pause") +def _(): + import telegram_bot as tb + from backtest import gpu_backtest as _gb11 + from providers.base import _is_scheduled_closure + dicts = (tb._last_alert_fp, tb._last_alert_ts, tb._last_alert_setup, + tb._last_alert_quality, tb._last_alert_bar) + saved = [dict(d) for d in dicts] + key = "XAU/USD_15min" + labels = [ts for ts in pd.date_range("2026-09-07 00:00", "2026-09-11 20:45", freq="15min", tz="UTC") + if not _is_scheduled_closure(ts)] + frame = _m11_frame(labels) + assert not any(ts.hour == 21 for ts in labels[:300]), "the fixture must carry the settlement pause" + # a setup on every bar from 19:30 to 23:45 each day (the hours around the pause), + # plus a sparse sprinkle over the rest of the day + cands = [i for i, ts in enumerate(labels) + if (ts.hour, ts.minute) >= (19, 30) or (i * 7919) % 11 == 0] + + def live(with_frames): + for d in dicts: + d.pop(key, None) + accepted = [] + for pos in cands: + report = _m11_cooldown_report(labels[pos], frame if with_frames else None) + if not tb._inside_alert_cooldown(key, report): + accepted.append(pos) + tb._commit_alert(key, report) + return accepted + + try: + book = [pos for pos, _trade in _gb11._apply_cooldown([(p, None) for p in cands], + tb._ALERT_COOLDOWN_BARS)] + assert live(with_frames=True) == book, "live must alert exactly the bars the backtester enters" + clock = live(with_frames=False) + extra = sorted(set(clock) - set(book)) + assert extra, "the clock rule differs on this fixture (that was the defect)" + assert clock != book and len(book) >= 40 + # every disagreement starts at the first bars after a pause: clock time ran, bars did not + first_diff = next(i for i, (a, b) in enumerate(zip(clock, book)) if a != b) + assert labels[clock[first_diff]].hour == 22, labels[clock[first_diff]] + finally: + for d, old in zip(dicts, saved): + d.clear(); d.update(old) + + +@test("Alert cooldown survives a restart: _restore_alert_cooldowns seeds the last alerted bar; " + "fingerprints are not restored (the database half is tested in Module 40)") +def _(): + import telegram_bot as tb + dicts = (tb._last_alert_fp, tb._last_alert_ts, tb._last_alert_setup, + tb._last_alert_quality, tb._last_alert_bar) + saved = [dict(d) for d in dicts] + try: + for d in dicts: + d.clear() + kept = pd.Timestamp("2026-09-15 04:00", tz="UTC") + tb._last_alert_bar["XAU/USD_4h"] = kept + n = tb._restore_alert_cooldowns({ + ("XAU/USD", "15min"): "2026-09-15 08:00:00+00:00", + ("XAU/USD", "1h"): "2026-09-15 07:00:00", # naive: read as UTC + ("XAU/USD", "5min"): "not a bar label", # skipped + ("XAU/USD", "4h"): "2026-09-14 20:00:00+00:00"}) # already in memory: left alone + assert n == 2, n + assert tb._last_alert_bar["XAU/USD_15min"] == pd.Timestamp("2026-09-15 08:00", tz="UTC") + assert tb._last_alert_bar["XAU/USD_1h"] == pd.Timestamp("2026-09-15 07:00", tz="UTC") + assert "XAU/USD_5min" not in tb._last_alert_bar and tb._last_alert_bar["XAU/USD_4h"] == kept + assert tb._restore_alert_cooldowns({}) == 0 and tb._restore_alert_cooldowns(None) == 0 + # the restored bar holds the next four bars; the fifth after it is free again + key = "XAU/USD_15min" + assert tb._inside_alert_cooldown(key, _m11_cooldown_report("2026-09-15 08:15:00+00:00")) + assert not tb._should_alert(key, _m11_cooldown_report("2026-09-15 09:00:00+00:00")) + free = _m11_cooldown_report("2026-09-15 09:15:00+00:00") + assert not tb._inside_alert_cooldown(key, free) + assert tb._should_alert(key, free), "no fingerprint was restored: the first alert after the cooldown goes out" + assert tb._last_alert_fp == {} and tb._last_alert_ts == {} + finally: + for d, old in zip(dicts, saved): + d.clear(); d.update(old) + + @test("format_signal_message: trail first as the tested exit, TP1 a guide with its R, book line") def _(): import telegram_bot as tb @@ -4357,6 +4544,155 @@ def _(): f"Retry failure must keep the original 250-bar df, got {len(result)}" +@test("providers.oanda: a DATE WINDOW with no bars is an empty frame, not an error; " + "a count request with nothing in it still raises (2026-09-30)") +def _(): + start = datetime(2020, 8, 1, tzinfo=timezone.utc) # a Saturday + end = datetime(2020, 8, 1, 23, 59, 59, tzinfo=timezone.utc) + forming = dict(_OANDA_OK_CANDLE, complete=False) + for body in ([], [forming]): + provider = _make_oanda_provider([_oanda_resp(200, body)]) + df = provider._fetch_raw("XAU/USD", Timeframe.D1, 5000, start=start, end=end) + assert df.empty and list(df.columns) == ["Open", "High", "Low", "Close", "Volume"], df + assert df.index.name == "Datetime" and str(df.index.tz) == "UTC" + for body, needle in (([], "Empty candle response"), ([forming], "No complete candles")): + provider = _make_oanda_provider([_oanda_resp(200, body)]) + try: + provider._fetch_raw("XAU/USD", Timeframe.D1, 300) + raise AssertionError("an empty live fetch must stay an error") + except providers.oanda.OandaDataError as e: + assert needle in str(e), e + + +@test("providers.oanda: fetch_range over a window whose head is a closed day returns the " + "bars instead of aborting (D1 pad starting on a Saturday)") +def _(): + def candle(day): + return dict(_OANDA_OK_CANDLE, time=f"2020-08-{day:02d}T00:00:00.000000000Z") + + # request 1 = the whole window (Sun 2nd stub, Mon 3rd, Tue 4th); request 2 = + # the leftover head [Sat 1st 00:00, Sat 1st 23:59:59], where nothing traded + provider = _make_oanda_provider([ + _oanda_resp(200, [candle(2), candle(3), candle(4)]), + _oanda_resp(200, []), + ]) + with patch("providers.base.BaseDataProvider._drop_forming_candle", lambda self, df, tf: df): + df = provider.fetch_range("XAU/USD", Timeframe.D1, + datetime(2020, 8, 1, tzinfo=timezone.utc), + datetime(2020, 8, 5, tzinfo=timezone.utc)) + assert [ts.day for ts in df.index] == [2, 3, 4], df.index + # a real provider error on that head window still aborts after three tries + provider = _make_oanda_provider( + [_oanda_resp(200, [candle(2), candle(3), candle(4)])] + + [_oanda_resp(400, text="bad request")] * 3) + try: + with patch("providers.base.BaseDataProvider._drop_forming_candle", lambda self, df, tf: df): + provider.fetch_range("XAU/USD", Timeframe.D1, + datetime(2020, 8, 1, tzinfo=timezone.utc), + datetime(2020, 8, 5, tzinfo=timezone.utc)) + raise AssertionError("three failures on one window must abort the fetch") + except providers.oanda.OandaDataError as e: + assert "consecutive failures" in str(e), e + + +@test("providers.oanda: only a closure-sized date window (<= 5 days, no malformed candle) may come back empty; " + "a longer empty window or an all-malformed one raises (2026-10-01)") +def _(): + import logging + P = providers.oanda.OandaDataProvider + assert P.EMPTY_WINDOW_MAX == pd.Timedelta(days=5) + start = datetime(2025, 12, 24, 18, 0, tzinfo=timezone.utc) + forming = dict(_OANDA_OK_CANDLE, complete=False) + malformed = dict(_OANDA_OK_CANDLE, mid={"o": "2400.00", "h": "2405.00", "l": "2398.00"}) # no close + + def fetch(body, span, tf=Timeframe.M5, begin=start): + provider = _make_oanda_provider([_oanda_resp(200, body)]) + return provider._fetch_raw("XAU/USD", tf, 5000, start=begin, end=begin + span) + + def raises(body, span, needle, **kw): + try: + fetch(body, span, **kw) + except providers.oanda.OandaDataError as e: + assert needle in str(e), e + return True + return False + + # exactly five days with nothing in it: a closure (a weekend joined to a holiday) + for body in ([], [forming]): + df = fetch(body, timedelta(days=5)) + assert df.empty and str(df.index.tz) == "UTC", df + # one second longer: no closure is that long, so the empty answer is an anomaly + assert raises([], timedelta(days=5, seconds=1), "Empty candle response") + assert raises([forming], timedelta(days=5, seconds=1), "No complete candles") + # the normal page sizes of the history fetch (15 days of M5, 46 of M15, 187 of H1) + for tf, days in ((Timeframe.M5, 15), (Timeframe.M15, 46), (Timeframe.H1, 187)): + assert raises([], timedelta(days=days), "Empty candle response", tf=tf), (tf, days) + # malformed candles mean the window DID trade: never "nothing traded here" + logging.disable(logging.CRITICAL) + try: + assert raises([malformed], timedelta(days=1), "No complete candles") + assert raises([malformed, forming], timedelta(hours=2), "No complete candles") + # ... while a good candle beside a malformed one is returned as before + df = fetch([malformed, _OANDA_OK_CANDLE], timedelta(days=1)) + assert len(df) == 1 and df["Close"].iloc[0] == 2403.0 + finally: + logging.disable(logging.NOTSET) + # the window test itself: both ends needed, mixed naive/aware never counts as a closure + assert P._closure_sized(start, start + timedelta(days=5)) is True + assert P._closure_sized(start, start + timedelta(days=5, seconds=1)) is False + assert P._closure_sized(None, start) is False and P._closure_sized(start, None) is False + assert P._closure_sized(start.replace(tzinfo=None), start + timedelta(days=1)) is False + assert P._closure_sized(pd.Timestamp(start), pd.Timestamp(start) + pd.Timedelta(days=2)) is True + + +@test("providers.oanda: fetch_range refuses a history page that comes back empty (three tries, then " + "'consecutive failures'); an empty closure-sized head is still skipped") +def _(): + import logging + start = datetime(2026, 3, 2, tzinfo=timezone.utc) + end = start + timedelta(days=20) + calls = [] + + def provider_for(responses): + provider = _make_oanda_provider(responses) + queue = list(responses) + + def get(self, url, params=None, timeout=None): + calls.append((params["from"], params["to"])) + return queue.pop(0) + provider._session = type("S", (), {"get": get, "headers": {}})() + return provider + + # the first page is the newest 15 days of M5: an empty answer for it is retried on + # the SAME window and refused, never stepped over + logging.disable(logging.CRITICAL) + try: + provider = provider_for([_oanda_resp(200, [])] * 3) + try: + provider.fetch_range("XAU/USD", Timeframe.M5, start, end) + raise AssertionError("an empty 15-day page must abort the fetch") + except providers.oanda.OandaDataError as e: + assert "consecutive failures" in str(e) and "Empty candle response" in str(e), e + finally: + logging.disable(logging.NOTSET) + assert len(calls) == 3 and len(set(calls)) == 1, calls + first_from, first_to = calls[0] + assert pd.Timestamp(first_to) - pd.Timestamp(first_from) == pd.Timedelta(days=15), calls[0] + + # a page with bars, then the leftover five-day head with none: the head is skipped + # and the bars are returned (the 2026-09-30 behaviour, kept for closure-sized windows) + del calls[:] + bar_time = (pd.Timestamp(start) + pd.Timedelta(days=5)).strftime("%Y-%m-%dT%H:%M:%S.000000000Z") + later = (pd.Timestamp(start) + pd.Timedelta(days=5, minutes=5)).strftime("%Y-%m-%dT%H:%M:%S.000000000Z") + page = [dict(_OANDA_OK_CANDLE, time=bar_time), dict(_OANDA_OK_CANDLE, time=later)] + provider = provider_for([_oanda_resp(200, page), _oanda_resp(200, [])]) + with patch("providers.base.BaseDataProvider._drop_forming_candle", lambda self, df, tf: df): + df = provider.fetch_range("XAU/USD", Timeframe.M5, start, end) + assert len(df) == 2 and len(calls) == 2, (len(df), calls) + head_from, head_to = calls[1] + assert pd.Timestamp(head_to) - pd.Timestamp(head_from) <= pd.Timedelta(days=5), calls[1] + + # ══════════════════════════════════════════════════════════════════════ # MODULE 16: Provider abstraction (providers/ package) # ══════════════════════════════════════════════════════════════════════ @@ -5006,6 +5342,15 @@ def _counting_session(payload, status=200): return S() +class _LogCapture(logging.Handler): + def __init__(self): + super().__init__() + self.records = [] + + def emit(self, record): + self.records.append(record) + + @test("llm_client: response parsing and availability gating") def _(): parse = DeepSeekClient._parse_response @@ -5102,6 +5447,45 @@ def _(): NewsEngine._source_health = saved +@test("NewsEngine: an RSS item with a blank title is skipped — nothing to show, score or archive (2026-10-01)") +def _(): + cfg = NewsConfig() + cfg.rss_feeds = ["https://feed.example/rss"] + ne = NewsEngine(cfg) + saved = NewsEngine._source_health + NewsEngine._source_health = {} + stamp = b"" + _m19_pubdate(1) + b"" + rss = (b"" + b"Gold surges as Fed signals rate cuts" + stamp + b"" + b"" + stamp + b"" + b" \n " + stamp + b"" + b" " + stamp + b"" + b"Stocks crater worldwide" + stamp + b"" + b"") + + class _Sess: + def get(self, url, timeout=None): + return type("R", (), {"status_code": 200, "content": rss})() + + ne._session = _Sess() + scored_texts = [] + + def enrich(scored, mode): + scored_texts.extend(h.headline for h in scored) + return [] + + ne._enrich_headline_scores = enrich + try: + heads = ne._fetch_news_sentiment_raw() + assert [h.headline for h in heads] == ["Gold surges as Fed signals rate cuts"], heads + # the scorers (and the archive) see the two real headlines only + assert scored_texts == ["Gold surges as Fed signals rate cuts", + "Stocks crater worldwide"], scored_texts + assert "1 kept" in NewsEngine._source_health["rss:feed.example"]["extra"] + finally: + NewsEngine._source_health = saved + + @test("NewsEngine: neutral headlines are dropped before the per-currency mean") def _(): from news_engine import _is_neutral_headline, NEUTRAL_HEADLINE_THRESHOLD @@ -5308,6 +5692,48 @@ def _(): lc._client = None +@test("NewsEngine: the event-direction cache is per release — the same name and actual against " + "another forecast, or on another date, is asked again (2026-10-01)") +def _(): + now = _dt_m19.datetime.now(_dt_m19.timezone.utc) + ne = NewsEngine(NewsConfig()) + NewsEngine._llm_event_cache = {} + + def release(when, forecast, actual="3.1%"): + return EconomicEvent(time=when, currency="USD", impact="high", + event="CPI y/y", forecast=forecast, actual=actual) + + first_time = now - _dt_m19.timedelta(minutes=10) + try: + stub = _StubLLM({"gold_direction": "bearish", "reason": "hot inflation"}) + lc._client = stub + assert ne._llm_event_direction(release(first_time, "2.9%")) == "bearish: hot inflation" + assert stub.last_payload["vs_forecast"] == "above forecast" + # the same release seen by the next scan (a new but equal object): served from the cache + assert ne._llm_event_direction(release(first_time, "2.9%")) == "bearish: hot inflation" + assert stub.calls == 1 + # next month: same name, same actual, but the forecast now sits ABOVE it + stub.reply = {"gold_direction": "bullish", "reason": "cooler than expected"} + next_month = first_time + _dt_m19.timedelta(days=30) + assert ne._llm_event_direction(release(next_month, "3.3%")) == "bullish: cooler than expected" + assert stub.calls == 2 and stub.last_payload["vs_forecast"] == "below forecast" + # same forecast and actual on another date is still another release + assert ne._llm_event_direction(release(next_month, "2.9%")) == "bullish: cooler than expected" + assert stub.calls == 3 + # the first answer is untouched + assert ne._llm_event_direction(release(first_time, "2.9%")) == "bearish: hot inflation" + assert stub.calls == 3 + assert len(NewsEngine._llm_event_cache) == 3 + assert all(len(key) == 5 for key in NewsEngine._llm_event_cache) + # one instant is one key, whatever offset the feed printed it with + eastern = _dt_m19.timezone(_dt_m19.timedelta(hours=-4)) + assert ne._llm_event_direction(release(first_time.astimezone(eastern), "2.9%")) == "bearish: hot inflation" + assert stub.calls == 3 and len(NewsEngine._llm_event_cache) == 3 + finally: + NewsEngine._llm_event_cache = {} + lc._client = None + + @test("NewsEngine: market summary cached across engine instances") def _(): now = _dt_m19.datetime.now(_dt_m19.timezone.utc) @@ -5363,6 +5789,164 @@ def _(): assert cfg.sentiment_affects_signals is False +@test("signal_aggregator: a failing news engine never fails the scan — the report is the technical one, " + "with no news text (2026-10-01)") +def _(): + import signal_aggregator as sa_mod + config = PlatformConfig() + config.primary_timeframe = Timeframe.H1 + config.confirmation_timeframes = [Timeframe.M15, Timeframe.H4] + agg = SignalAggregator(config) + dfs = { + Timeframe.H1: make_ohlcv(300, seed=42), + Timeframe.M15: make_ohlcv(300, seed=43), + Timeframe.H4: make_ohlcv(300, seed=44), + } + plain = agg.analyze_from_dataframes("XAU/USD", dfs, use_news=False) + + class _News: + def __init__(self, fail_at): + self.fail_at = fail_at + + def _maybe(self, name): + if name == self.fail_at: + raise KeyError(f"boom in {name}") + + def get_fundamental_context(self): + self._maybe("context") + return FundamentalContext(llm_summary="A summary that must not survive a failure.") + + def get_event_risk_level(self, symbol, context): + self._maybe("events") + return "danger", "NFP in 5 minutes" + + def get_pair_sentiment(self, symbol, context): + self._maybe("sentiment") + return 0.6, "Bullish headlines" + + def news_fields(r): + return (r.event_risk, r.event_description, r.news_sentiment, r.news_description, r.news_summary) + + def technical(r): + return (r.final_signal, r.final_score, r.final_confidence, r.signal_quality, + r.trade_setup, r.mtf_agreement, r.market_regime) + + cap = _LogCapture() + sa_mod.logger.addHandler(cap) + old_propagate = sa_mod.logger.propagate + sa_mod.logger.propagate = False # asserted below, not printed + try: + for fail_at in ("context", "events", "sentiment"): + agg._news_engine = _News(fail_at) + report = agg.analyze_from_dataframes("XAU/USD", dfs, use_news=True) + # all five fields together or none: a half-filled news block would show + # an event warning beside an empty summary + assert news_fields(report) == ("clear", "", 0.0, "", "") == news_fields(plain), \ + (fail_at, news_fields(report)) + assert technical(report) == technical(plain), fail_at + warns = [r for r in cap.records if r.levelno == logging.WARNING] + assert len(warns) == 3 and all("News context failed for XAU/USD" in r.getMessage() for r in warns) + assert all(r.exc_info for r in warns), "the traceback is logged" + # the healthy engine still fills the block + agg._news_engine = _News(None) + report = agg.analyze_from_dataframes("XAU/USD", dfs, use_news=True) + assert news_fields(report) == ("danger", "NFP in 5 minutes", 0.6, "Bullish headlines", + "A summary that must not survive a failure.") + finally: + sa_mod.logger.removeHandler(cap) + sa_mod.logger.propagate = old_propagate + + +@test("llm_client: chat_json never raises and never logs the API key — bad header, transport error, " + "unserialisable payload, pathological body (2026-10-01)") +def _(): + secret = "sk-TOPSECRET123" + cap = _LogCapture() + lc.logger.addHandler(cap) + old_level, old_propagate = lc.logger.level, lc.logger.propagate + lc.logger.setLevel(logging.DEBUG) + lc.logger.propagate = False + + def raising_session(exc): + class S: + calls = 0 + + def post(self, url, json=None, timeout=None): + S.calls += 1 + raise exc + return S() + + try: + # 1. a key with a line break in the middle: the REAL requests session refuses the + # header before any network I/O, and its error text quotes the header value + client = DeepSeekClient(api_key="sk-TOP\nSECRET123") + assert client.chat_json("sys", {"q": 1}) is None + assert client.last_error[1] == "header", client.last_error + # 2. the same failure with the key quoted verbatim in the exception text + client = DeepSeekClient(api_key=secret) + client._session = raising_session(requests.exceptions.InvalidHeader( + f"Invalid leading whitespace, reserved character(s), or return character(s) " + f"in header value: 'Bearer {secret}'")) + assert client.chat_json("sys", {"q": 1}) is None and client.last_error[1] == "header" + # 3. any other requests error: its text is logged with the key blanked + client = DeepSeekClient(api_key=secret) + client._session = raising_session(requests.ConnectionError( + f"HTTPSConnectionPool: request with Authorization Bearer {secret} refused")) + assert client.chat_json("sys", {"q": 1}) is None and client.last_error[1] == "network" + # 4. a non-requests exception from inside the HTTP stack does not escape + client = DeepSeekClient(api_key=secret) + client._session = raising_session( + UnicodeEncodeError("latin-1", f"Bearer {secret}\u2019", 21, 22, "ordinal not in range(256)")) + assert client.chat_json("sys", {"q": 1}) is None and client.last_error[1] == "client" + # 5. a payload that cannot be serialised: no request sent, no cap spent + client = DeepSeekClient(api_key=secret, daily_cap=1) + s = _counting_session(_chat_body({"ok": True})) + client._session = s + assert client.chat_json("sys", {"x": object()}) is None + assert client.chat_json("sys", {1: "a", "b": 2}) is None, "mixed keys cannot be sorted" + assert type(s).calls == 0 and client._count == 0 and client.last_error[1] == "payload" + assert client.chat_json("sys", {"q": 1}) == {"ok": True}, "the day's one request is still there" + # 6. a body whose .json() blows up with something that is not a ValueError + client = DeepSeekClient(api_key=secret) + + class _Resp: + status_code = 200 + text = "x" + + def json(self): + raise RecursionError("maximum recursion depth exceeded") + + class _S: + def post(self, url, json=None, timeout=None): + return _Resp() + client._session = _S() + assert client.chat_json("sys", {"q": 1}) is None and client.last_error[1] == "parse" + # 7. an error body that echoes the request headers (a gateway page): redacted + # before it is cut, so a key at the 200-character mark is not left half visible + client = DeepSeekClient(api_key=secret) + for body in (f"401 Unauthorized. Authorization: Bearer {secret} was rejected", + "x" * 193 + secret + " trailing"): + client._warned.clear() + client._session = type("S", (), {"post": lambda self, url, json=None, timeout=None, _b=body: type( + "R", (), {"status_code": 401, "text": _b, "json": lambda self: {}})()})() + assert client.chat_json("sys", {"q": 1}) is None and client.last_error[1] == "http_401" + # whitespace around a key is not part of it (a key pasted with its line break) + tidy = DeepSeekClient(api_key=f" {secret}\r\n") + assert tidy._api_key == secret and tidy._session.headers["Authorization"] == f"Bearer {secret}" + assert DeepSeekClient(api_key=" \n").available is False + finally: + lc.logger.removeHandler(cap) + lc.logger.setLevel(old_level) + lc.logger.propagate = old_propagate + logged = "\n".join(r.getMessage() for r in cap.records) + assert "SECRET123" not in logged and "sk-TOP" not in logged, logged + assert "" in logged, "the transport error is still logged, with the key blanked" + assert "UnicodeEncodeError" in logged and "not JSON-serialisable (TypeError)" in logged, logged + assert "DeepSeek HTTP 401: 401 Unauthorized. Authorization: Bearer was rejected" in logged, logged + assert "TOPSEC" not in logged, "not even a part of the key at the cut" + assert "check deepseek.api_key / DEEPSEEK_API_KEY" in logged, logged + + @test("llm_client: health stamps track last success and failure") def _(): client = DeepSeekClient(api_key="k", daily_cap=10) @@ -5377,15 +5961,6 @@ def _(): assert kind == "http_429" and ts >= client.last_ok -class _LogCapture(logging.Handler): - def __init__(self): - super().__init__() - self.records = [] - - def emit(self, record): - self.records.append(record) - - @test("NewsEngine: _record_source logs transitions only, tracks state") def _(): import news_engine as ne_mod @@ -5622,6 +6197,49 @@ def _(): lc._client = None +@test("/status: failure details are Markdown-escaped — exception text with an underscore or a " + "bracket must not make Telegram reject the reply (2026-10-01)") +def _(): + import telegram_bot as tb + saved = NewsEngine._source_health + t0 = time.time() + + def failing(detail, extra=""): + return {"ok": None, "err": t0, "detail": detail, "extra": extra} + + lc._client = _StubLLM(None) + lc._client.last_ok, lc._client.last_error = t0, None + lc._client._count, lc._client._daily_cap = 0, 500 + lc._client._count_day = _dt_m19.datetime.now(_dt_m19.timezone.utc).date() + try: + NewsEngine._source_health = { + "calendar": failing("ConnectionError: /ff_calendar_thisweek.json [Errno -2]"), + "headlines": failing("all feeds failed", "0 headlines from 0/1 feeds"), + "rss:feeds.my_news.example": failing("ParseError: not well-formed (invalid token) `x`"), + "news_model": failing("ValueError: input_ids has 1 * 70 > max_length"), + } + lines = tb._news_health_lines() + text = "\n".join(lines) + assert "/ff\\_calendar\\_thisweek.json \\[Errno -2\\]" in text, text + assert "⚠️ feeds.my\\_news.example: ParseError: not well-formed (invalid token) \\`x\\`" in text, text + assert "⚠️ News model: not scoring — ValueError: input\\_ids has 1 \\* 70 > max\\_length" in text, text + # no raw control character is left outside the fixed frame of the section + body = text.replace("*News & AI:*", "") + for ch in "*_`[]": + body = body.replace("\\" + ch, "") + assert not any(ch in body for ch in "*_`[]"), body + # the forexfactory-down variant escapes its detail too + NewsEngine._source_health = { + "calendar": {"ok": t0, "err": None, "detail": "", "extra": "42 events via MyFXBook"}, + "calendar:forexfactory": failing("HTTPError: 503 for /ff_calendar"), + } + assert any("ForexFactory down: HTTPError: 503 for /ff\\_calendar" in x + for x in tb._news_health_lines()) + finally: + NewsEngine._source_health = saved + lc._client = None + + # ══════════════════════════════════════════════════════════════════════ # MODULE 20: dashboard_publisher.py (static web dashboard) # ══════════════════════════════════════════════════════════════════════ @@ -6672,10 +7290,17 @@ def _(): with patch.object(_tb21, "run_signal", rs): t1 = _aio21.create_task(_tb21._scan_timeframe( bot, Timeframe.M15, [Timeframe.M5, Timeframe.H1])) - await _aio21.sleep(0.02) # scan holds the key lock, blocked in send + # The scan holds the key lock and blocks in the send. Delivery + # hops through two worker threads first (panel, alert-db), so + # poll for the send instead of trusting a fixed sleep. + for _ in range(400): + if bot.attempts: + break + await _aio21.sleep(0.005) + assert len(bot.attempts) == 1, "the scan's alert never reached the send" t2 = _aio21.create_task(_tb21._handle_breach( bot, "XAU/USD", 3301.0, "swing_high_1")) - await _aio21.sleep(0.02) # breach is queued on the lock + await _aio21.sleep(0.05) # breach is queued on the lock assert len(bot.attempts) == 1, \ "breach must wait on the alert lock, not race into send" gate.set() @@ -8792,6 +9417,51 @@ def _(): == [1, 2], "a signal inside the cooldown window is skipped" +@test("analyze_signal_log: stored-candle replay keeps a trade that exited before a later hole in the stream; " + "a trade still open at the hole stays UNRESOLVABLE (2026-10-01)") +def _(): + t0 = datetime(2026, 8, 19, 12, 0, 10, tzinfo=timezone.utc) # Wednesday + bars = [] + price = 3400.0 + for i in range(40): + ts = (pd.Timestamp("2026-08-19 12:00", tz="UTC") + + pd.Timedelta(minutes=15 * i)) + o = price + c = price + (2.0 if i < 20 else -2.0) + bars.append({"timestamp": ts.isoformat(), "open": o, + "high": max(o, c) + 0.5, "low": min(o, c) - 0.5, + "close": c, "volume": 1.0}) + price = c + # 19:45-20:30 UTC (four tradable bars) never reached the database + holed_stream = bars[:31] + bars[35:] + + def rows(): + return [_asl_row(1, t0, bar_ts="2026-08-19T12:00:00+00:00"), + _asl_row(2, t0 + timedelta(hours=7), signal="BUY", entry=3440.0, sl=3436.0, + bar_ts="2026-08-19T19:00:00+00:00")] + + def replay(stream, items): + events = asl.cluster_setup_rows(items) + asl.replay_stored(events, {"15min": stream}) + return events + + full, holed = replay(bars, rows()), replay(holed_stream, rows()) + assert [e.exit_reason for e in full] == ["TRAIL", "SL"], [e.exit_reason for e in full] + for a, b in zip(full, holed): + # both exits (17:30 and 19:30) came before the hole: the same trade, to the cent + assert (b.exit_reason, b.exit_time, b.r_headline, b.r_compare, b.bars_held, b.entry_fill) \ + == (a.exit_reason, a.exit_time, a.r_headline, a.r_compare, a.bars_held, a.entry_fill), \ + (a.representative.id, b.exit_reason) + # a wide-stop buy entered at 19:30 is still open when the stream breaks: not guessed + wide = [_asl_row(3, t0 + timedelta(hours=7, minutes=15), signal="BUY", entry=3440.0, sl=3300.0, + bar_ts="2026-08-19T19:15:00+00:00")] + (open_ev,), (lost_ev,) = replay(bars, wide), replay(holed_stream, wide) + assert open_ev.exit_reason == "OPEN" and open_ev.r_headline is None, open_ev.exit_reason + assert lost_ev.exit_reason == "UNRESOLVABLE" and lost_ev.r_headline is None, lost_ev.exit_reason + assert lost_ev.entry_fill == bars[30]["open"] and lost_ev.bars_held == 1, \ + "the walk up to the hole is kept" + + @test("analyze_signal_log: parity report flags a divergent bar and rows-per-bar") def _(): t0 = datetime(2026, 8, 19, 12, 45, 10, tzinfo=timezone.utc) @@ -9252,6 +9922,73 @@ def _(): os.remove(path) +@test("signal_log: a hole AFTER the exit does not cost the outcome — resolve_pending settles it on the " + "first pass (no stream rebuild) and resolve_from_store settles it from the stored stream (2026-10-01)") +def _(): + import signal_log as slog + fd, path = tempfile.mkstemp(suffix=".db") + os.close(fd) + try: + lg = slog.SignalLogger(path) + # Wednesday; a steady fall: the BUY filled at 12:05 is stopped on the 12:15 bar + bars = _m27_bars("2026-08-19T12:00:00+00:00", 10, up=-2.0) + holed = bars[0:4] + bars[6:10] # 12:20 and 12:25 never reached the database + + def outcome(rowid): + status, r, blob = lg._conn.execute( + "SELECT advisory_outcome_status, advisory_r_multiple, advisory_outcome_json" + " FROM signals WHERE id=?", (rowid,)).fetchone() + return status, r, json.loads(blob) + + # 1. live resolver: the scan's own snapshot carries the hole + calls = [] + real_stream = lg._stream_since + + def _spy(symbol, timeframe, last_bar, max_rows=4000): + calls.append(last_bar) + return real_stream(symbol, timeframe, last_bar, max_rows) + + lg._stream_since = _spy + a = lg.log(_m27_report(), source="telegram", timeframe="5min") + del calls[:] # log()'s scan-outcome labeller streams bars too + assert lg.resolve_pending("XAU/USD", slog._dumps({"5min": holed})) == 1 + assert calls == [], "the trade was over before the hole: no union rebuild" + status, r, state = outcome(a) + assert status == "RESOLVED" and state["result"]["exit_reason"] == "SL", state + assert state["result"]["exit_at"] == bars[3]["timestamp"], state["result"] + assert state["progress"]["entry_price"] == bars[1]["open"] == 2448.0 + assert abs(r - (-5.5 - 0.65) / 5.5) < 1e-9, r + lg._stream_since = real_stream + + # 2. backfill path: the stream is already in the database (stored by rows that + # carry no setup), then a setup row is seeded and resolved from the store alone + for chunk in (holed[0:4], holed[4:8]): + lg.log(_m27_report(setup=False, quality="D", ts=chunk[-1]["timestamp"]), + source="dashboard", timeframe="5min", + dataframes={Timeframe.M5: _m27_df(chunk)}) + b = lg.log(_m27_report(sl=2444.5), source="telegram", timeframe="5min") + assert outcome(b)[0] == "PENDING" + assert lg.resolve_from_store("XAU/USD") == 1 + status, r, state = outcome(b) + assert status == "RESOLVED" and state["result"]["exit_reason"] == "SL", state + assert state["result"]["exit_at"] == bars[3]["timestamp"], state["result"] + assert abs(r - (-6.0 - 0.65) / 6.0) < 1e-9, r + + # 3. a trade still open at the hole is NOT guessed: the fill is kept, the row + # is UNRESOLVABLE (no locked stop to floor it at) + c = lg.log(_m27_report(sl=2400.0), source="telegram", timeframe="5min") + assert lg.resolve_from_store("XAU/USD") == 1 + status, r, state = outcome(c) + assert status == "UNRESOLVABLE" and r is None, (status, r) + assert state["reason"] == "stored candle snapshots contain an internal gap" + assert state["progress"]["last_processed_bar"] == bars[3]["timestamp"], state["progress"] + assert state["progress"]["bars_held"] == 3 and state["progress"]["entry_price"] == 2448.0 + lg.close() + finally: + slog.reset_loggers_for_tests() + os.remove(path) + + @test("signal_log: backfill_seeds keys legacy rows, dedups, seeds, keeps panel outcomes") def _(): import signal_log as slog @@ -9352,6 +10089,15 @@ def _(): "MTF OPPOSITION"), "cleaned (CP) warnings count too" assert gr(["NO PULLBACK — x"], [], "B", False).startswith( "SETUP GENERATOR") + # the daily-trend veto removes a setup every earlier gate let through, so + # it is the reason even when the vetoed setup's stale warnings are pooled + assert gr(["MTF OPPOSITION: majority", + "DAILY TREND FILTER: BUY against a falling daily trend — no trade"], + ["SPREAD GATE: target $3.00 < $9.00 minimum — x"], "D", + False).startswith("DAILY TREND FILTER") + assert gr(["Daily trend filter requested but 1d data unavailable"], [], + "B", False).startswith("SETUP GENERATOR"), \ + "the unavailable notice is not a gate" assert gr([], [], "D", False) == "GRADE D: below setup threshold" assert gr([], [], "", False) is None fd, path = tempfile.mkstemp(suffix=".db") @@ -10851,6 +11597,73 @@ def _(): os.remove(path) +@test("scripts/label_scan_outcomes: the 'window elapsed' count uses an ISO cutoff, so rows logged on the cutoff's own day are counted") +def _(): + import contextlib + import importlib.util as _ilu31 + import io as _io + import sqlite3 + import signal_log as slog + spec = _ilu31.spec_from_file_location( + "label_scan_outcomes_script", + os.path.join(os.path.dirname(os.path.abspath(__file__)), + "scripts", "label_scan_outcomes.py")) + ls = _ilu31.module_from_spec(spec) + spec.loader.exec_module(ls) + assert slog.SCAN_LABEL_MIN_AGE_H == 25, "the stamps below assume a 25 h window" + fd, path = tempfile.mkstemp(suffix=".db") + os.close(fd) + try: + slog.SignalLogger(path).close() + slog.reset_loggers_for_tests() + con = sqlite3.connect(path) + rows = ( + # (logged_at, symbol, indicators_json, price) + ("2026-09-30T23:00:00.250000+00:00", "XAU/USD", "{}", 2450.0), # the day before: due + ("2026-10-01T10:00:00.250000+00:00", "XAU/USD", "{}", 2450.0), # the cutoff's own day, 1 h before it: due + ("2026-10-01T10:59:59+00:00", "XAU/USD", "{}", 2450.0), # no microseconds, 1 s before: due + ("2026-10-01T11:00:00+00:00", "XAU/USD", "{}", 2450.0), # the cutoff itself: not yet + ("2026-10-01T12:00:00.250000+00:00", "XAU/USD", "{}", 2450.0), # after it: not yet + ("2026-09-30T23:00:00.250000+00:00", "EUR/USD", "{}", 1.1), # another symbol + ("2026-09-30T23:00:00.250000+00:00", "XAU/USD", None, 2450.0), # blocked scan: no indicators + ("2026-09-30T23:00:00.250000+00:00", "XAU/USD", "{}", 0.0), # failed scan: no price + ) + con.executemany( + "INSERT INTO signals (logged_at, source, symbol, timeframe, signal," + " indicators_json, price) VALUES (?, 'telegram', ?, '5min', 'NEUTRAL', ?, ?)", + rows) + con.commit() + now = datetime(2026, 10, 2, 12, 0, tzinfo=timezone.utc) # cutoff 2026-10-01T11:00Z + got = ls._counts(con, "XAU/USD", now=now) + assert got["scan rows with indicators"] == 5, got + assert got["of those, window elapsed"] == 3, got + assert got["labelled"] == 0 and got["complete"] == 0, got + # a naive `now` is read as UTC; another zone gives the same instant + assert ls._counts(con, "XAU/USD", now=datetime(2026, 10, 2, 12, 0)) == got + tokyo = timezone(timedelta(hours=9)) + assert ls._counts(con, "XAU/USD", now=datetime(2026, 10, 2, 21, 0, tzinfo=tokyo)) == got + # one second later the row logged at the cutoff is due as well + late = ls._counts(con, "XAU/USD", labelled=False, now=now + timedelta(seconds=1)) + assert late == {"scan rows with indicators": 5, "of those, window elapsed": 4}, late + con.close() + # no SQLite clock in the query any more (its datetime() prints a space) + with open(os.path.join(os.path.dirname(os.path.abspath(__file__)), + "scripts", "label_scan_outcomes.py"), encoding="utf-8") as fh: + body = fh.read().split("def _counts")[1].split("\ndef ")[0] + assert "datetime('now'" not in body + # the read-only preview prints both counts and writes nothing + buf = _io.StringIO() + with contextlib.redirect_stdout(buf): + assert ls.main(["--db", path, "--dry-run"]) == 0 + out = buf.getvalue() + assert "scan rows with indicators: 5" in out and "of those, window elapsed:" in out, out + finally: + slog.reset_loggers_for_tests() + for suffix in ("", "-wal", "-shm"): + if os.path.exists(path + suffix): + os.remove(path + suffix) + + @test("backtest parity: the replay path knows nothing about the new writers") def _(): root = os.path.dirname(os.path.abspath(__file__)) @@ -11768,6 +12581,23 @@ from backtest import replay_record as _rr _M33_CACHE = {} +def _has_parquet33() -> bool: + """True when pandas can write Parquet here. pyarrow lives in the dev venv + only (CLAUDE.md section 10: live code never touches Parquet), so on the + live box the few tests that must write a Parquet fixture print a notice + and return, the convention of the CUDA tests.""" + for name in ("pyarrow", "fastparquet"): + try: + __import__(name) + return True + except ImportError: + continue + return False + + +_NO_PARQUET33 = " (no Parquet engine here — pyarrow is dev-only: fixture not written, test not exercised)" + + def _m33_report(): """One REAL M5 analyze_from_dataframes run + a hand-built setup. @@ -12024,6 +12854,9 @@ def _(): @test("replay record: calendar join — window edges, phases, all-day, holiday, offsets") def _(): + if not _has_parquet33(): + print(_NO_PARQUET33) + return entry = pd.Timestamp("2026-09-07 13:00", tz="UTC") # Monday exit_ts = entry + pd.Timedelta(hours=1) # Window = [entry-4h, max(exit, entry+4h)] = [09:00, 17:00], inclusive. @@ -12123,6 +12956,9 @@ def _(): @test("replay record: write_run + load_run round-trip with a full manifest") def _(): + if not _has_parquet33(): + print(_NO_PARQUET33) + return cfg, agg, _report, _setup = _m33_report() snap = _m33_snapshot() trades = [_m29_trade(r_multiple=1.5), _m29_trade(r_multiple=-1.0)] @@ -12712,6 +13548,9 @@ def _(): @test("backtester: cache-miss fallbacks are counted per span and surfaced by run_backtest (audit 2026-09-13)") def _(): + if not _has_parquet33(): + print(_NO_PARQUET33) + return # A GPU cache miss recomputes the indicator with the live engine's # pandas math on the sliced frame (different EWM seeding than the # pre-computed cache) — a live-vs-backtest parity hazard. The counter @@ -12893,6 +13732,90 @@ def _(): assert state["reason"] == before["reason"], "no floor, no rewrite" +@test("advisory_outcome: a hole AFTER the exit does not void the trade; a hole reached while it is open " + "ends the walk with the progress kept, so one pass and many passes floor alike (2026-10-01)") +def _(): + from advisory_outcome import (_has_continuity, _next_bar_follows, _valid_bars, + advance_outcome, canonical_r, resolve_locked_floor) + head = [ + _m34_bar("07:30", 4396.0, 4397.0, 4394.0, 4395.5), # signal bar + _m34_bar("07:45", 4395.64, 4396.0, 4388.0, 4389.0), # entry + _m34_bar("08:00", 4389.0, 4390.0, 4345.06, 4350.0), # arms; the stop ratchets below the entry + _m34_bar("08:15", 4350.0, 4356.0, 4348.0, 4352.0), + ] + exit_bar = _m34_bar("08:30", 4352.0, 4390.0, 4351.0, 4388.0) # runs through the trailed stop + tail = [_m34_bar("10:45", 4380.0, 4381.0, 4379.0, 4380.5), # eight tradable bars missing before it + _m34_bar("11:00", 4380.5, 4382.0, 4379.5, 4381.0)] + risk = _m34_sell_seed()["setup"]["stop_distance"] + + # 1. The trade is over at 08:30: the hole at 08:45 is none of its business. + clean = advance_outcome(_m34_sell_seed(), head + [exit_bar]) + holed = advance_outcome(_m34_sell_seed(), head + [exit_bar] + tail) + assert clean["status"] == "RESOLVED" and clean["result"]["exit_reason"] == "TRAIL", clean + assert holed == clean, "a hole after the exit must not change the outcome" + assert holed["result"]["exit_at"] == "2026-09-08T08:30:00+00:00", holed["result"] + stop = holed["result"]["exit_price"] + assert abs(stop - (4345.06 + 0.75 * risk)) < 1e-9, stop + assert abs(canonical_r(holed) - (4395.64 - stop - 0.65) / risk) < 1e-9 + # ... while the all-or-nothing check the scan-outcome labeller builds on is unchanged + seed_bar = _valid_bars(head[:1])[0]["_timestamp"] + assert _has_continuity(_valid_bars(head + [exit_bar]), seed_bar, "15min") is True + assert _has_continuity(_valid_bars(head + [exit_bar] + tail), seed_bar, "15min") is False + assert _has_continuity(_valid_bars(head), seed_bar, "2h") is False + + # 2. A hole reached while the trade is open: UNRESOLVABLE, the progress kept up to + # the bar before it (the old up-front check left a one-pass walk with no entry at all). + one_pass = advance_outcome(_m34_sell_seed(), head + tail) + assert one_pass["status"] == "UNRESOLVABLE" and one_pass["result"] is None, one_pass + assert one_pass["reason"] == "stored candle snapshots contain an internal gap" + progress = one_pass["progress"] + assert progress["last_processed_bar"] == "2026-09-08T08:15:00+00:00", progress + assert progress["bars_held"] == 3 and progress["entry_price"] == 4395.64, progress + assert abs(progress["current_stop"] - stop) < 1e-9, progress + stepwise = advance_outcome(_m34_sell_seed(), head) + assert stepwise["status"] == "PENDING" + stepwise = advance_outcome(stepwise, head[-1:] + tail) + assert stepwise == one_pass, "the same stream in one call or two must end in the same state" + floored = resolve_locked_floor(one_pass) + assert floored["status"] == "RESOLVED" and floored["result"]["exit_reason"] == "GAP_LOCKED", floored + assert floored["result"]["exit_at"] == "2026-09-08T08:15:00+00:00", floored["result"] + assert abs(floored["result"]["exit_price"] - stop) < 1e-9 + + # 3. A hole before the stop is locked: nothing to floor, the fill is still recorded. + early = advance_outcome(_m34_sell_seed(), head[:2] + tail) + assert early["status"] == "UNRESOLVABLE", early + assert early["progress"]["bars_held"] == 1 and early["progress"]["entry_price"] == 4395.64 + assert early["progress"]["last_processed_bar"] == "2026-09-08T07:45:00+00:00" + assert resolve_locked_floor(early)["status"] == "UNRESOLVABLE" + + # 4. An unknown bar size cannot be walked; with nothing new to walk the state is untouched. + odd = _m34_sell_seed() + odd["timeframe"] = "2h" + untouched = json.loads(json.dumps(odd)) + assert advance_outcome(odd, head[:1]) == untouched + odd = advance_outcome(odd, head) + assert odd["status"] == "UNRESOLVABLE" and "internal gap" in odd["reason"], odd + assert odd["progress"]["entry_price"] is None + + # 5. The step rule: the next bar, or one separated from it by scheduled closures only. + def at(text): + return datetime.fromisoformat(text + "+00:00") + step = timedelta(minutes=15) + assert _next_bar_follows(at("2026-09-08T08:15:00"), at("2026-09-08T08:30:00"), step) + assert not _next_bar_follows(at("2026-09-08T08:15:00"), at("2026-09-08T08:45:00"), step) # one bar missing + assert not _next_bar_follows(at("2026-09-08T08:15:00"), at("2026-09-08T08:37:00"), step) # off the grid + # Tuesday's settlement pause (21:00-21:59 UTC in summer) and the weekend are not holes + assert _next_bar_follows(at("2026-09-08T20:45:00"), at("2026-09-08T22:00:00"), step) + assert _next_bar_follows(at("2026-09-11T20:45:00"), at("2026-09-13T22:00:00"), step) + assert not _next_bar_follows(at("2026-09-11T20:30:00"), at("2026-09-13T22:00:00"), step) # Friday 20:45 missing + # the model no longer calls the all-or-nothing check + import advisory_outcome as _ao34 + import inspect as _inspect34 + body = _inspect34.getsource(_ao34.advance_outcome) + assert "_next_bar_follows(last, stamp, step)" in body + assert "_has_continuity(" not in body.split('"""', 2)[2], "the docstring aside, no up-front stream check" + + @test("signal_log: the outcome resolver rebuilds the stream for both gap " "reasons (audit 2026-09-13)") def _(): @@ -13834,6 +14757,201 @@ def _(): ap.reset_panel_state() +@test("agent panel v2.2: event timing reads the calendar week attach_panel hands in — a hold that runs past " + "midnight sees tomorrow's release; a caller with its own context never triggers a calendar read") +def _(): + assert ap.PANEL_PROMPT_VERSION.startswith("v2.2-"), ap.PANEL_PROMPT_VERSION + orig = ap._BASE_RATES_FILE + fd, path = tempfile.mkstemp(suffix=".json") + os.close(fd) + now = _dt_m19.datetime.now(_dt_m19.timezone.utc) + + def event(minutes, impact="high", actual=None, name="NFP"): + return _M35Event(time=now + _dt_m19.timedelta(minutes=minutes), currency="USD", + impact=impact, event=name, actual=actual, forecast="150K") + + try: + with open(path, "w") as fh: + json.dump({"classes": {"1h|XAU_ATR|A|BUY": { + "n": 50, "win_rate": 61.0, "mean_r": 0.2, "tp1_reach_rate": 55.0, + "median_bars_held": 12, "book": "h1_w3.csv"}}}, fh) + ap.reset_base_rates_for_tests(path) + ap._dp = _NullDP() + report, dfs = _m35_report_and_frames("BUY") + empty = FundamentalContext() # nothing today (UTC date), nothing in the next hour + # the week: a past release, a medium event in 2 h, the next high-impact one in 9 h + week = [event(-300, actual="140K"), event(120, impact="medium", name="Housing"), + event(540), event(2000, name="CPI")] + # the context alone (the old view): the 12-hour hold looks event-free + ctx = ap._price_context(report, dfs, empty) + assert ctx["expected_hold_minutes"] == 720 and "next_high_impact_event_minutes" not in ctx + # with the week: the release in 9 h sits inside the hold + ctx = ap._price_context(report, dfs, empty, week) + assert 539.0 <= ctx["next_high_impact_event_minutes"] <= 540.1, ctx + assert ctx["event_inside_expected_hold"] is True + ctx = ap._price_context(report, dfs, None, week) # no context at all + assert 539.0 <= ctx["next_high_impact_event_minutes"] <= 540.1 + assert ap._price_context(report, dfs, empty, [event(900)])["event_inside_expected_hold"] is False + # an empty week (calendar down) falls back to what the context knows + soon = FundamentalContext(events_today=[event(30)]) + assert 29.0 <= ap._price_context(report, dfs, soon, [])["next_high_impact_event_minutes"] <= 30.1 + # a broken entry in the week is skipped, never fatal + ctx = ap._price_context(report, dfs, empty, [object(), event(540)]) + assert 539.0 <= ctx["next_high_impact_event_minutes"] <= 540.1 + json.dumps(ctx, allow_nan=False) + + # attach_panel hands the week in ONLY when it builds the context itself + calls = {"context": 0, "week": 0} + + def fake_context(self, currencies=None): + calls["context"] += 1 + return FundamentalContext() + + def fake_week(self, currencies=None): + calls["week"] += 1 + return week + + def dead_week(self, currencies=None): + raise RuntimeError("calendar down") + + settings = {"enabled": True, "telegram": True, "min_quality": "B", + "max_panels_per_day": 20, "reuse_ttl_seconds": 0} + lc._client = _PanelStub(_m35_v2_replies(macro=58)) + def must_not_fetch(self, currencies=None): + raise AssertionError("attach_panel must not call the refreshing accessor") + + with patch("news_engine.NewsEngine.get_fundamental_context", fake_context), \ + patch("news_engine.NewsEngine.peek_week_events", fake_week), \ + patch("news_engine.NewsEngine.get_week_events", must_not_fetch): + report._frames = dfs + assert ap.attach_panel(report, settings) is True + assert calls == {"context": 1, "week": 1}, calls + assert 538.0 <= report.panel_context["next_high_impact_event_minutes"] <= 540.1 + assert report.panel_context["event_inside_expected_hold"] is True + assert report.panel_payloads["macro"]["context"]["event_inside_expected_hold"] is True, \ + "the persona reads the same block that is logged" + assert report.panel_prompt_version == ap.PANEL_PROMPT_VERSION + # a caller that passes its own context (the replay): no calendar read + report2, dfs2 = _m35_report_and_frames("BUY") + report2._frames = dfs2 + assert ap.attach_panel(report2, settings, context=FundamentalContext()) is True + assert calls == {"context": 1, "week": 1}, calls + assert "next_high_impact_event_minutes" not in report2.panel_context + # a calendar failure costs the timing line only, never the panel + with patch("news_engine.NewsEngine.get_fundamental_context", fake_context), \ + patch("news_engine.NewsEngine.peek_week_events", dead_week): + report3, dfs3 = _m35_report_and_frames("SELL") + report3._frames = dfs3 + assert ap.attach_panel(report3, settings) is True + assert "next_high_impact_event_minutes" not in report3.panel_context + # the offline replay calls _price_context with three arguments: still no week + src = Path("scripts/panel_replay.py").read_text(encoding="utf-8") + assert "ap._price_context(report, frames, context)" in src + assert "get_week_events" not in src + finally: + lc._client = None + ap._dp = None + ap.reset_panel_state() + ap.reset_base_rates_for_tests(orig) + os.remove(path) + + +@test("agent panel v2.2: the persona's headline list is the newest ten across all feeds, not the first ten in feed order") +def _(): + now = _dt_m19.datetime.now(_dt_m19.timezone.utc) + + unset = object() + + def head(title, hours_ago, score=0.4, stamp=unset): + return NewsSentiment( + headline=title, source="rss", + timestamp=(now - _dt_m19.timedelta(hours=hours_ago)) if stamp is unset else stamp, + currencies_mentioned=["XAU"], sentiment_score=score, keywords=[]) + + # the context lists feed A first (twelve older stories), then feed B (three fresh ones) + feed_a = [head(f"old {i}", 4.0 - i * 0.25) for i in range(12)] + feed_b = [head("fresh 1", 0.2), head("fresh 2", 0.1), head("fresh 3", 0.05)] + got = ap._directional_news_payload(FundamentalContext(news_headlines=feed_a + feed_b)) + titles = [h["headline"] for h in got] + assert len(titles) == 10, titles + assert titles[:3] == ["fresh 3", "fresh 2", "fresh 1"], titles + assert titles[3:] == [f"old {i}" for i in range(11, 4, -1)], titles + stamps = [h["published_at"] for h in got] + assert stamps == sorted(stamps, reverse=True) + # both personas that read headlines get the same list + report = _panel_report() + ctx = FundamentalContext(news_headlines=feed_a + feed_b) + assert ap._macro_payload(report, ctx)["headlines"] == got + assert ap._contrarian_payload(report, ctx)["news"] == got + # a fresh story that leans neither way takes no slot (the filter runs before the cut) + with_neutral = feed_a + feed_b + [head("fresh neutral", 0.01, score=0.02)] + assert [h["headline"] for h in ap._directional_news_payload( + FundamentalContext(news_headlines=with_neutral))] == titles + # equal publish times keep the feed order (a stable sort) + same = [head(f"same {i}", 1.0, stamp=now) for i in range(4)] + assert [h["headline"] for h in ap._directional_news_payload( + FundamentalContext(news_headlines=same))] == [f"same {i}" for i in range(4)] + # the sort key never raises: text stamps are read, anything else sorts last + utc = _dt_m19.timezone.utc + assert ap._published_key(head("x", 0, stamp="2026-01-05T11:00:00+00:00")) == \ + _dt_m19.datetime(2026, 1, 5, 11, 0, tzinfo=utc).timestamp() + for junk in (None, "garbage", 12345, object()): + assert ap._published_key(head("x", 0, stamp=junk)) == float("-inf"), junk + assert ap._published_key(object()) == float("-inf") + mixed = [head("undated", 0, stamp="n/a"), head("dated", 2.0)] + assert [h["headline"] for h in ap._directional_news_payload( + FundamentalContext(news_headlines=mixed))] == ["dated", "undated"] + assert ap._directional_news_payload(FundamentalContext()) == [] + + +@test("scripts/panel_replay: a stored payload is replayed per persona (a macro-only row too) and the CSV names them") +def _(): + spec = _ilu35.spec_from_file_location( + "panel_replay_stored", os.path.join(os.path.dirname(os.path.abspath(__file__)), + "scripts", "panel_replay.py")) + pr = _ilu35.module_from_spec(spec) + spec.loader.exec_module(pr) + report, dfs = _m35_report_and_frames() + ctx = FundamentalContext() + settings = {"candle_bars_primary": ap.DEFAULT_CANDLE_BARS_PRIMARY, + "candle_bars_confirm": ap.DEFAULT_CANDLE_BARS_CONFIRM} + logged_macro = {"brief": {"marker": "exactly what the live panel sent"}, + "context": {"base_rate": {"win_rate": 61.0}}} + systems = [ap._MACRO_SYSTEM, ap._TECH_SYSTEM, ap._RISK_SYSTEM, ap._CONTRARIAN_SYSTEM] + try: + ap._dp = _NullDP() + assert pr.PERSONAS == ap.ALL_PERSONAS, "the research default scores every persona" + # what live writes since 2026-09-17: the macro payload only + stored = {"macro": logged_macro} + jobs = pr.build_jobs(report, ctx, dfs, stored, settings) + assert [j[0] for j in jobs] == list(ap.ALL_PERSONAS) and [j[1] for j in jobs] == systems + by = {name: payload for name, _system, payload in jobs} + assert by["macro"] is logged_macro, "the logged macro payload must be replayed, not rebuilt" + for name in ("technical", "risk", "contrarian"): + assert by[name]["brief"]["direction"] == "BUY" and "marker" not in by[name]["brief"], name + assert pr.stored_personas(stored) == ["macro"] + # a four-persona row replays all four; nothing is rebuilt + full = {name: {"brief": {"marker": name}} for name in ap.ALL_PERSONAS} + jobs = pr.build_jobs(report, ctx, dfs, full, settings) + assert all(payload is full[name] for name, _system, payload in jobs) + assert [j[1] for j in jobs] == systems, "always the CURRENT prompts" + assert pr.stored_personas(full) == list(ap.ALL_PERSONAS) + # nothing usable stored: everything rebuilt, and the CSV cell is empty + for junk in (None, {}, [], "text", {"macro": "not a payload"}, {"someone": {}}): + jobs = pr.build_jobs(report, ctx, dfs, junk, settings) + assert pr.stored_personas(junk) == [], junk + assert len(jobs) == 4 and all("marker" not in payload["brief"] for _n, _s, payload in jobs) + # --personas macro on a macro-only row: one job, the logged payload + pr.PERSONAS = ("macro",) + jobs = pr.build_jobs(report, ctx, dfs, stored, settings) + assert len(jobs) == 1 and jobs[0][2] is logged_macro + src = Path("scripts/panel_replay.py").read_text(encoding="utf-8") + assert '"stored_payloads": ",".join(stored_personas(stored))' in src + assert "bool(stored)" not in src, "the CSV must not claim a stored payload that was not used" + finally: + ap._dp = None + + @test("app_config: agent_panel.personas validated (names, chair needs two, macro source needs macro)") def _(): assert load_app_config(_write_json( @@ -14329,11 +15447,40 @@ def _m37_resample(df, rule): return agg.dropna(subset=["Open"]) +def _m37_d1_trend_fixture(end="2026-03-23", days=420, turn="2026-03-10"): + """A long daily frame for the daily-trend veto: UTC-midnight labels with + the Sunday stub bars Oanda emits (no Saturdays), a steady rise until + `turn` and a fall afterwards, so the trend reads +1, then 0 for a few + days (close under its 50-bar EMA but still above the close 60 bars + back), then -1 inside the 21-day window of the 'main' M5 fixture. The + Sunday stubs carry an absurd close: the definition must ignore them.""" + idx = pd.date_range(end=pd.Timestamp(end, tz="UTC"), periods=days, freq="D") + idx = idx[idx.weekday != 5] + t = np.arange(len(idx), dtype=float) + k = int(idx.searchsorted(pd.Timestamp(turn, tz="UTC"))) + close = 3000.0 + 1.2 * t + 4.0 * np.sin(t / 3.0) + close[k:] = close[k - 1] - 22.0 * (t[k:] - t[k - 1]) + close[idx.weekday == 6] = 9999.0 + df = pd.DataFrame({"Open": close - 1.0, "High": close + 6.0, "Low": close - 6.0, + "Close": close, "Volume": 1000.0}, index=idx) + df.index.name = "Datetime" + return df + + def _m37_frames(key="main"): """M5/M15/H1/H4/D1 frames. 'main' is the 21-day fixture (M5 and M15 primaries); 'slow' a 30-day fixture whose D1 frame has fewer than 30 - bars (H1 and H4 primaries; the empty-analysis path on a confirmation).""" + bars (H1 and H4 primaries; the empty-analysis path on a confirmation); + 'trend' is 'main' with the long daily frame of the daily-trend veto; + 'trend_late' the same with a daily frame that only starts mid-window + (no visible daily bar at first, then too few for a 60-bar change).""" if key not in _M37: + if key in ("trend", "trend_late"): + frames = dict(_m37_frames("main")) + d1 = _m37_d1_trend_fixture() + frames[Timeframe.D1] = d1 if key == "trend" else d1[d1.index >= pd.Timestamp("2026-03-12", tz="UTC")] + _M37[key] = frames + return _M37[key] m5 = (_m37_m5_fixture() if key == "main" else _m37_m5_fixture(days=30, seed=12, start="2026-04-06")) _M37[key] = { @@ -14356,10 +15503,13 @@ def _m37_bt(cfg, dfs, **kw): def _m37_timeframes(cfg, bt): + from signal_aggregator import daily_trend_needed tfs = [bt.primary_tf] + list(bt.confirmation_tfs) flt = cfg.backtest if flt.regime_filter and flt.regime_filter_timeframe not in tfs: tfs.append(flt.regime_filter_timeframe) + if daily_trend_needed(cfg) and Timeframe.D1 not in tfs: # both loaders append it like this + tfs.append(Timeframe.D1) return tfs @@ -14665,8 +15815,40 @@ def _(): raise AssertionError("the packer must refuse an unknown grading_mode") except ValueError as e: assert "grading_mode" in str(e), e + # daily-trend veto and wide-stop cap (2026-09-30): the packed codes follow + # the live module's mode list by NAME; M5, H1 and H4 ship both knobs off + from signal_aggregator import DAILY_TREND_MODES as _DTM + assert _ck37.DAILY_TREND_CODES == {"off": 0, "both": 1, "buy": 2, "sell": 3} and set(_DTM) == set(_ck37.DAILY_TREND_CODES) + for tf in (Timeframe.M5, Timeframe.M15, Timeframe.H1, Timeframe.H4): + cfg = _mk37(tf) + pf0, pi0, _ = _ck37.pack_params(cfg, _m37_bt(cfg, dfs)) + f0, i0 = dict(zip(_ck37.PARAMS_F, pf0.tolist())), dict(zip(_ck37.PARAMS_I, pi0.tolist())) + assert i0["daily_trend_veto"] == _ck37.DAILY_TREND_CODES[cfg.xau_strategy.daily_trend_veto], tf + assert f0["wide_stop_cap_atr"] == cfg.xau_strategy.wide_stop_cap_atr, tf + if tf != Timeframe.M15: + assert i0["daily_trend_veto"] == 0 and f0["wide_stop_cap_atr"] == 0.0, tf + else: + assert i0["daily_trend_veto"] == _ck37.DAILY_TREND_CODES["both"], "M15 override 7 (2026-09-30)" + for mode, code in _ck37.DAILY_TREND_CODES.items(): + cfg = _mk37(Timeframe.M5) + cfg.xau_strategy.daily_trend_veto = mode + cfg.xau_strategy.wide_stop_cap_atr = 6.0 + pfm, pim, _ = _ck37.pack_params(cfg, _m37_bt(cfg, dfs)) + assert dict(zip(_ck37.PARAMS_I, pim.tolist()))["daily_trend_veto"] == code, mode + assert dict(zip(_ck37.PARAMS_F, pfm.tolist()))["wide_stop_cap_atr"] == 6.0 + bad = _mk37(Timeframe.M15) + bad.xau_strategy.daily_trend_veto = "weekly" + try: + _ck37.pack_params(bad, _m37_bt(bad, dfs)) + raise AssertionError("the packer must refuse an unknown daily_trend_veto") + except ValueError as e: + assert "daily_trend_veto" in str(e), e # the kernel #defines are generated from the same name lists src = _ck37._defines() + for mode, code in _ck37.DAILY_TREND_CODES.items(): + assert f"#define TREND_{mode} {code}\n" in src, mode + assert f"#define FB2_TREND_UNAVAILABLE {_ck37.DAILY_TREND_UNAVAILABLE}\n" in src + assert _ck37.DAILY_TREND_UNAVAILABLE not in (-1, 0, 1) for k, name in enumerate(_ck37.PARAMS_F): assert f"#define PF_{name} {k}\n" in src, name for k, name in enumerate(_ck37.PARAMS_I): @@ -15053,6 +16235,180 @@ def _(): f"{t4['short_confirm']} with an empty D1 analysis)") +@test("CUDA stage: trend_flags == daily_trend(the replay's D1 slice) on every primary bar (Sunday stubs, no visible bar, warm-up; M5 and H4 clocks; no CUDA needed)") +def _(): + from signal_aggregator import daily_trend as _dt, daily_trend_series as _dts + d1 = _m37_d1_trend_fixture() + assert int((d1.index.weekday == 6).sum()) > 40 and not (d1.index.weekday == 5).any() + series = _dts(d1) + assert {-1, 0, 1} == set(series.unique().tolist()), "the fixture must visit all three states" + assert not (series.index.weekday == 6).any() + cases = (("trend", Timeframe.M5), ("trend", Timeframe.M15), ("trend_late", Timeframe.M5), ("trend", Timeframe.H4)) + seen = set() + for key, tf in cases: + dfs = _m37_frames(key) + cfg = _mk37(tf) + bt = _m37_bt(cfg, dfs) + frame = dfs[Timeframe.D1] + primary = dfs[tf] + got = _cb37.trend_flags(primary.index, frame, bt) + assert got.dtype == np.int8 and len(got) == len(primary) + dur = pd.Timedelta(seconds=_gb37.BAR_SECONDS[tf]) + day = pd.Timedelta(seconds=_gb37.BAR_SECONDS[Timeframe.D1]) + step = 1 if tf == Timeframe.H4 else 7 + for pos in range(0, len(primary), step): + cut = int(frame.index.searchsorted(primary.index[pos] + dur - day, side="right")) # _replay_span's slice + want = _dt(frame.iloc[:cut]) if cut else None + want = _ck37.DAILY_TREND_UNAVAILABLE if want is None else want + assert int(got[pos]) == want, (key, tf, pos, primary.index[pos], int(got[pos]), want) + seen.add((key, want)) + assert {("trend", 1), ("trend", 0), ("trend", -1), ("trend_late", _ck37.DAILY_TREND_UNAVAILABLE), + ("trend_late", 0)} <= seen, seen + # no frame / an empty frame / a frame of Sunday stubs only: unavailable everywhere + m5 = _m37_frames("main")[Timeframe.M5] + bt = _m37_bt(_mk37(Timeframe.M5), _m37_frames("main")) + for frame in (None, d1.iloc[:0], d1[d1.index.weekday == 6]): + assert (_cb37.trend_flags(m5.index, frame, bt) == _ck37.DAILY_TREND_UNAVAILABLE).all() + # a microsecond-resolution index must give the same flags (the mixed-unit trap of align_cut) + us = d1.copy() + us.index = us.index.as_unit("us") + assert np.array_equal(_cb37.trend_flags(m5.index, us, bt), _cb37.trend_flags(m5.index, d1, bt)) + + +def _tweak_trend37(mode, cap=0.0): + def tweak(cfg): + cfg.xau_strategy.daily_trend_veto = mode + cfg.xau_strategy.wide_stop_cap_atr = cap + return tweak + + +def _trend_vetoes37(cfg_tf, mode, key="trend"): + """(vetoed BUY rows, vetoed SELL rows) the live code writes on the fixture.""" + dfs = _m37_frames(key) + cfg = _mk37(cfg_tf) + _tweak_trend37(mode)(cfg) + bt = _m37_bt(cfg, dfs) + got, ctx = _m37_device(cfg, bt, dfs) + buys = sum(1 for r in got.values() if r[0] == "rep" + and any(w.startswith("DAILY TREND FILTER: BUY") for w in r[1]["warnings"])) + sells = sum(1 for r in got.values() if r[0] == "rep" + and any(w.startswith("DAILY TREND FILTER: SELL") for w in r[1]["warnings"])) + return buys, sells, got + + +@test("CUDA stage: K2 == analyze_from_dataframes — daily-trend veto on M15 (both / buy / sell, both directions vetoed, a vetoed row is grade D without a setup)") +def _(): + if not _GPU37: + print(" (no CUDA: kernel not exercised)") + return + t = _k2_case37("M15 trend both", Timeframe.M15, _tweak_trend37("both"), {}, "trend", 1, None) + assert "trend_veto" in t["warn"] and "trend_unavailable" not in t["warn"], sorted(t["warn"]) + buys, sells, got = _trend_vetoes37(Timeframe.M15, "both") + assert buys >= 3 and sells >= 3, (buys, sells) + for pos, r in got.items(): + if r[0] == "rep" and any(w.startswith("DAILY TREND FILTER:") for w in r[1]["warnings"]): + assert r[1]["quality"] == "D" and r[1]["setup"] is None and r[1]["cand"] is None, (pos, r[1]["quality"]) + assert r[1]["warnings"][-1].startswith("DAILY TREND FILTER:"), r[1]["warnings"] + # the one-sided modes veto one side only, and exactly the rows 'both' vetoes on that side + for mode, want in (("buy", (buys, 0)), ("sell", (0, sells))): + tm = _k2_case37(f"M15 trend {mode}", Timeframe.M15, _tweak_trend37(mode), {}, "trend", 3, None) + assert "trend_veto" in tm["warn"], (mode, sorted(tm["warn"])) + b, s, _ = _trend_vetoes37(Timeframe.M15, mode) + assert (b, s) == want, (mode, b, s, want) + # off: the trend frame is not even loaded, no row carries a trend warning, and + # the veto is the only difference: a row 'both' left alone is the same row, a + # vetoed row is the 'off' row minus its setup (grade D, one warning appended) + off_dfs = _m37_frames("trend") + cfg = _mk37(Timeframe.M15) + cfg.xau_strategy.daily_trend_veto = "off" + bt = _m37_bt(cfg, off_dfs) + assert Timeframe.D1 not in _m37_timeframes(cfg, bt) + t_off = _k2_case37("M15 trend off", Timeframe.M15, _tweak_trend37("off"), {}, "trend", 3, None) + assert not ({"trend_veto", "trend_unavailable"} & t_off["warn"]), sorted(t_off["warn"]) + b_off, s_off, got_off = _trend_vetoes37(Timeframe.M15, "off") + assert (b_off, s_off) == (0, 0), (b_off, s_off) + assert set(got_off) == set(got) + veto_fields = {"quality", "setup", "cand", "warnings", "_warn_names"} + restored = 0 + for pos, r in got.items(): + o = got_off[pos] + if r[0] == "rep" and any(w.startswith("DAILY TREND FILTER:") for w in r[1]["warnings"]): + assert o[0] == "rep" and o[1]["setup"] is not None, (pos, "the veto only removes a finished setup") + assert o[1]["warnings"] == r[1]["warnings"][:-1], (pos, o[1]["warnings"], r[1]["warnings"]) + assert all(_same37(o[1][k], r[1][k]) for k in r[1] if k not in veto_fields), pos + restored += 1 + else: + assert _same37(o, r), (pos, "a row the veto did not touch must not change") + assert restored == buys + sells, (restored, buys, sells) + cands_both = sum(1 for r in got.values() if r[0] == "rep" and r[1]["cand"]) + cands_off = sum(1 for r in got_off.values() if r[0] == "rep" and r[1]["cand"]) + assert cands_off > cands_both, (cands_off, cands_both) + print(f" ({t['n']} bars: {buys} BUY and {sells} SELL setups vetoed, {t['cands']} candidates left; " + f"veto off: {cands_off} candidates, every other row identical)") + + +@test("CUDA stage: K2 == analyze_from_dataframes — daily-trend veto on M5 (chart-pattern setups vetoed too) and a daily frame with no visible bar") +def _(): + if not _GPU37: + print(" (no CUDA: kernel not exercised)") + return + t = _k2_case37("M5 trend both", Timeframe.M5, _tweak_trend37("both"), {}, "trend", 2, None) + assert "trend_veto" in t["warn"], sorted(t["warn"]) + assert {"GENERIC", "XAU_CP"} <= t["strategy"], t["strategy"] + # a vetoed chart-pattern setup: its marker stays in the reasons, the plan is gone + _, _, got = _trend_vetoes37(Timeframe.M5, "both") + cp_vetoed = sum(1 for r in got.values() if r[0] == "rep" and r[1]["setup"] is None + and any(w.startswith("DAILY TREND FILTER:") for w in r[1]["warnings"]) + and any(x.startswith("XAU CHART PATTERN") for x in r[1]["bull"] + r[1]["bear"])) + assert cp_vetoed >= 1, "the fixture must veto at least one chart-pattern setup" + late = _k2_case37("M5 trend, late daily frame", Timeframe.M5, _tweak_trend37("both"), {}, "trend_late", 3, None) + assert "trend_unavailable" in late["warn"] and "trend_veto" not in late["warn"], sorted(late["warn"]) + assert late["cands"] > 0, "an unavailable or unclear trend must not remove setups" + print(f" ({t['n']} bars, {cp_vetoed} chart-pattern setups vetoed; late frame: {late['cands']} candidates kept)") + + +@test("CUDA stage: K2 == analyze_from_dataframes — wide-stop cap (M5 GENERIC + XAU_CP, M15 GENERIC): same candidates, capped stops; veto + cap together") +def _(): + if not _GPU37: + print(" (no CUDA: kernel not exercised)") + return + for tf, cap in ((Timeframe.M5, 2.5), (Timeframe.M15, 2.5)): + t = _k2_case37(f"{tf.value} cap {cap}", tf, _tweak_trend37("off", cap), {}, "main", 2, None) + assert t["cands"] > 0 + dfs = _m37_frames("main") + base_cfg, cap_cfg = _mk37(tf), _mk37(tf) + cap_cfg.xau_strategy.wide_stop_cap_atr = cap + base, _ = _m37_device(base_cfg, _m37_bt(base_cfg, dfs), dfs) + capped, _ = _m37_device(cap_cfg, _m37_bt(cap_cfg, dfs), dfs) + changed = {"GENERIC": 0, "XAU_CP": 0, "XAU_ATR": 0} + for pos, b in base.items(): + c = capped[pos] + assert b[0] == c[0], pos + if b[0] != "rep": + continue + bs, cs = b[1]["setup"], c[1]["setup"] + assert (bs is None) == (cs is None), (pos, "the cap must not add or remove a setup") + assert (b[1]["cand"] is None) == (c[1]["cand"] is None), pos + assert b[1]["quality"] == c[1]["quality"] and b[1]["warnings"] == c[1]["warnings"], pos + if bs is None: + continue + assert bs[:3] == cs[:3] and bs[7] == cs[7], (pos, "generator, direction, entry and pattern are untouched") + b_dist, c_dist = abs(bs[2] - bs[3]), abs(cs[2] - cs[3]) + assert c_dist <= b_dist + 1e-9, (pos, b_dist, c_dist) + # the printed target is max(the gated target, 1R of the stop): a smaller + # stop can only lower it, never raise it + assert abs(cs[4] - cs[2]) <= abs(bs[4] - bs[2]) + 1e-9, (pos, bs, cs) + if c_dist < b_dist - 1e-9: + changed[bs[0]] += 1 + assert changed["XAU_ATR"] == 0, "the gold ATR setup (2 x ATR stop) is never capped" + assert changed["GENERIC"] >= 5, (tf, changed) + if tf == Timeframe.M5: + assert changed["XAU_CP"] >= 1, (tf, changed) + both = _k2_case37("M15 trend both + cap", Timeframe.M15, _tweak_trend37("both", 3.0), {}, "trend", 3, None) + assert "trend_veto" in both["warn"] and both["cands"] > 0 + print(f" (cap leaves the candidate set alone; M15 veto + cap: {both['cands']} candidates)") + + @test("CUDA stage: device chart-pattern detector == detect_chart_pattern (six patterns, mirrored series, zero ATR); big candle == _detect_big_candle") def _(): if not _GPU37: @@ -15197,6 +16553,9 @@ def _(): if not _GPU37: print(" (no CUDA: script not exercised)") return + if not _has_parquet33(): + print(_NO_PARQUET33) + return dfs = _m37_frames("main") cfg = _mk37(Timeframe.M15) start = datetime(2026, 3, 9, tzinfo=timezone.utc) @@ -15209,6 +16568,11 @@ def _(): for tf in [Timeframe.M15] + list(cfg.confirmation_timeframes): tf_start = start - timedelta(days=_gb37._history_pad_days(tf, 250)) dfs[tf].to_parquet(_gb37._cache_path(XAUUSD, tf, tf_start, end, provider_name)) + # The M15 factory ships the daily-trend veto (override 7), so every + # run below reads the D1 frame from the cache under its own + # (1400-day pad) name; a missing file would be a network fetch. + d1_start = start - timedelta(days=_gb37._history_pad_days(Timeframe.D1, 250)) + _m37_d1_trend_fixture().to_parquet(_gb37._cache_path(XAUUSD, Timeframe.D1, d1_start, end, provider_name)) cal = Path(d) / "cal.parquet" pd.DataFrame([_m33_cal_row(1, pd.Timestamp("2026-03-06 13:30", tz="UTC"), "Non-Farm Employment Change")]).to_parquet(cal, index=False) @@ -15221,11 +16585,30 @@ def _(): assert rc == 0 return _hashlib37.md5(out.read_bytes()).hexdigest() + # plain = the factory config, which on M15 carries the daily-trend veto md5_ref = run(_gb37, Path(d) / "ctrl.csv", []) md5_dev = run(_cb37, Path(d) / "cuda.csv", []) book = pd.read_csv(Path(d) / "cuda.csv") - assert len(book) >= 20 and len(book.columns) == 63, book.shape + assert len(book) > 0 and len(book.columns) == 63, book.shape assert md5_ref == md5_dev, f"plain book differs: {md5_ref} vs {md5_dev}" + # the veto switched off through both CLIs: byte-identical books that + # hold the plain book's trades plus the ones the veto removes + off = ["--set", "xau_strategy.daily_trend_veto=off"] + md5_ref_off = run(_gb37, Path(d) / "ctrl_off.csv", off) + md5_dev_off = run(_cb37, Path(d) / "cuda_off.csv", off) + assert md5_ref_off == md5_dev_off, f"veto-off book differs: {md5_ref_off} vs {md5_dev_off}" + assert md5_ref_off != md5_ref, "the veto must change the book" + book_off = pd.read_csv(Path(d) / "cuda_off.csv") + assert len(book_off) >= 20 and len(book) < len(book_off), (len(book), len(book_off)) + # the wide-stop cap on top of the veto: byte-identical on both + # scripts, the same trades as the plain book with different stops + cap = ["--set", "xau_strategy.wide_stop_cap_atr=3"] + md5_ref_t = run(_gb37, Path(d) / "ctrl_t.csv", cap) + md5_dev_t = run(_cb37, Path(d) / "cuda_t.csv", cap) + assert md5_ref_t == md5_dev_t, f"veto + cap book differs: {md5_ref_t} vs {md5_dev_t}" + assert md5_ref_t != md5_ref, "the cap must change the book" + book_t = pd.read_csv(Path(d) / "cuda_t.csv") + assert list(book_t.entry_time) == list(book.entry_time), "the cap never changes the trade set" saved = _rr37.CALENDAR_PARQUET try: _rr37.CALENDAR_PARQUET = cal @@ -15286,6 +16669,7 @@ def _(): (lambda c: setattr(c.xau_strategy, "cp_lookback", _ck37.MAX_CP_LOOKBACK), "cp_lookback"), (lambda c: setattr(c.technical, "ema_mode", "3ema"), "ema_mode"), (lambda c: setattr(c.signals, "grading_mode", "geomean"), "grading_mode"), + (lambda c: setattr(c.xau_strategy, "daily_trend_veto", "weekly"), "daily_trend_veto"), (lambda c: setattr(c, "confirmation_timeframes", [Timeframe.M5, Timeframe.H1, Timeframe.H4, Timeframe.D1, Timeframe.M30]), "confirmation timeframes"), (lambda c: (setattr(c.xau_strategy, "big_candle_atr_multiple", 3.0), @@ -15441,6 +16825,10 @@ class _Model38: def __enter__(self): self._saved = (_nm38.get_news_model, _nm38.score_batch, _nm38.warm_up_in_background) + # _apply_model_sentiment records "news_model" in the class-level + # health dict (2026-10-01): keep it out of the other tests' state. + self._saved_health = _NE38._source_health + _NE38._source_health = {} _nm38.get_news_model = lambda: self.bundle _nm38._bundle = self.bundle # teacher_tag() reads it directly _nm38.warm_up_in_background = lambda: None @@ -15459,6 +16847,8 @@ class _Model38: def __exit__(self, *exc): (_nm38.get_news_model, _nm38.score_batch, _nm38.warm_up_in_background) = self._saved + self.health = _NE38._source_health # what the block recorded + _NE38._source_health = self._saved_health _nm38.reset_for_tests() return False @@ -15547,21 +16937,126 @@ def _(): @test("news model: a scoring failure or a missing model leaves keyword scores standing") def _(): rows = [_h38("Gold up"), _h38("Gold down")] - with _Model38(raise_exc=RuntimeError("ORT exploded")): - n = _NE38(_NC38())._apply_model_sentiment(rows) - assert n == 0 and all(h.score_source == "keyword" for h in rows) - assert all(h.xau_score is None for h in rows) - with _Model38(bundle=None) as m: - n = _NE38(_NC38())._apply_model_sentiment(rows) - assert n == 0 and m.calls == [], "no scoring call without a bundle" - assert all(h.score_source == "keyword" for h in rows) - # a length mismatch is a failure too, not a silent misalignment - saved = _nm38.score_batch - with _Model38() as m: - _nm38.score_batch = lambda b, texts: [{"relevant": True, "score": 0.5}] - n = _NE38(_NC38())._apply_model_sentiment(rows) - assert n == 0 and all(h.score_source == "keyword" for h in rows) - assert _nm38.score_batch is saved + cap = _LogCapture() + _ne38.logger.addHandler(cap) + old_level, old_propagate = _ne38.logger.level, _ne38.logger.propagate + _ne38.logger.setLevel(_log38.INFO) + _ne38.logger.propagate = False # asserted below, not printed + try: + with _Model38(raise_exc=RuntimeError("ORT exploded")): + n = _NE38(_NC38())._apply_model_sentiment(rows) + assert n == 0 and all(h.score_source == "keyword" for h in rows) + assert all(h.xau_score is None for h in rows) + with _Model38(bundle=None) as m: + n = _NE38(_NC38())._apply_model_sentiment(rows) + assert n == 0 and m.calls == [], "no scoring call without a bundle" + assert all(h.score_source == "keyword" for h in rows) + assert _NE38.source_is_failing(m.health["news_model"]), m.health + assert m.health["news_model"]["detail"] == "not loaded in this process" + # a length mismatch is a failure too, not a silent misalignment + saved = _nm38.score_batch + with _Model38() as m: + _nm38.score_batch = lambda b, texts: [{"relevant": True, "score": 0.5}] + n = _NE38(_NC38())._apply_model_sentiment(rows) + assert n == 0 and all(h.score_source == "keyword" for h in rows) + assert _nm38.score_batch is saved + assert "1 results for 2 headlines" in m.health["news_model"]["detail"], m.health + finally: + _ne38.logger.removeHandler(cap) + _ne38.logger.setLevel(old_level) + _ne38.logger.propagate = old_propagate + warns = [r.getMessage() for r in cap.records if r.levelno == _log38.WARNING] + assert len(warns) == 3 and all("news_model" in w for w in warns), warns + assert "ORT exploded" in warns[0], warns + + +@test("news model: a headline the model cannot read keeps its keyword score and no longer costs " + "the batch; failure and recovery are logged once and shown by /status (2026-10-01)") +def _(): + import telegram_bot as _tb38 + + def strict(text): + # the real scorer cleans every text first and raises on an empty one + _nm38.clean_headline(text) + return {"relevant": True, "score": 0.6} + + assert _NE38._model_can_score("Gold up") is True + for bad in ("", " ", " ", "   \n", None): + assert _NE38._model_can_score(bad) is False, repr(bad) + rows = [_h38("Gold up"), _h38(" ", currencies=(), by={}), _h38(" ", currencies=(), by={}), + _h38("Gold down")] + cap = _LogCapture() + _ne38.logger.addHandler(cap) + old_level, old_propagate = _ne38.logger.level, _ne38.logger.propagate + _ne38.logger.setLevel(_log38.INFO) + _ne38.logger.propagate = False + saved_client = _lc38._client + _lc38._client = None + try: + with _Model38(strict) as m: + ne = _NE38(_NC38()) + assert ne._apply_model_sentiment(rows) == 2 + assert m.calls == [["Gold up", "Gold down"]], m.calls + assert [h.score_source for h in rows] == ["model", "keyword", "keyword", "model"] + assert rows[0].xau_score == 0.6 and rows[1].xau_score is None + entry = _NE38._source_health["news_model"] + assert not _NE38.source_is_failing(entry) and entry["extra"] == "2 scored", entry + assert any("✅ News model: 2 scored" in line for line in _tb38._news_health_lines()) + # the batch fails: WARNING once, DEBUG after, the /status line turns + more = [_h38("Silver up"), _h38("Silver down")] + m.raise_exc = RuntimeError("ORT_exploded [code 7]") + assert ne._apply_model_sentiment(more) == 0 + assert ne._apply_model_sentiment(more) == 0 + assert all(h.score_source == "keyword" for h in more) + entry = _NE38._source_health["news_model"] + assert _NE38.source_is_failing(entry), entry + line = [x for x in _tb38._news_health_lines() if "News model" in x] + assert len(line) == 1 and line[0].startswith(" ⚠️ News model: not scoring — "), line + # Markdown-safe: the exception text is escaped, or /status would not render + assert "ORT\\_exploded \\[code 7\\]" in line[0], line + # ... and recovers + m.raise_exc = None + assert ne._apply_model_sentiment(more) == 2 + assert not _NE38.source_is_failing(_NE38._source_health["news_model"]) + # a later cycle in which the model is not needed (fallback mode, DeepSeek scored + # every row): the old failure is dropped, not left on /status as if it were current + with _Model38(strict) as m: + ne = _NE38(_NC38()) + m.raise_exc = RuntimeError("boom") + assert ne._apply_model_sentiment([_h38("Zinc up")]) == 0 + assert any("News model: not scoring" in x for x in _tb38._news_health_lines()) + scored_already = [_h38("Zinc up", source="llm"), _h38("Zinc down", source="llm")] + calls_before = len(m.calls) + assert ne._apply_model_sentiment(scored_already) == 0 and len(m.calls) == calls_before + assert "news_model" not in _NE38._source_health + assert not any("News model" in x for x in _tb38._news_health_lines()) + # a healthy record is left alone by an idle cycle + m.raise_exc = None + assert ne._apply_model_sentiment([_h38("Tin up")]) == 1 + assert ne._apply_model_sentiment(scored_already) == 0 + assert _NE38._source_health["news_model"]["extra"] == "1 scored" + # a process that never ran the model shows no line for it + saved_health = _NE38._source_health + _NE38._source_health = {"headlines": {"ok": time.time(), "err": None, "detail": "", + "extra": "3 headlines from 1/1 feeds"}} + try: + assert not any("News model" in x for x in _tb38._news_health_lines()) + finally: + _NE38._source_health = saved_health + finally: + _lc38._client = saved_client + _ne38.logger.removeHandler(cap) + _ne38.logger.setLevel(old_level) + _ne38.logger.propagate = old_propagate + msgs = [(r.levelno, r.getMessage()) for r in cap.records + if "news_model" in r.getMessage() and r.levelno >= _log38.INFO] + # first failure WARNING, its repeat silent, the recovery INFO; then the second + # failure WARNING again. The idle cycle that dropped it logged nothing: the model + # did not run, so no "recovered" line may claim it did. + assert [lvl for lvl, _ in msgs] == [_log38.WARNING, _log38.INFO, _log38.WARNING], msgs + assert "Source failing: news_model" in msgs[0][1] and "ORT_exploded" in msgs[0][1], msgs + assert msgs[1][1] == "Source recovered: news_model", msgs + assert "Source failing: news_model (RuntimeError: boom)" in msgs[2][1], msgs @test("news model: primary mode with no model degrades to DeepSeek, not to keywords") @@ -15694,6 +17189,59 @@ def _(): _lc38._client = None +@test("news model: the audit sample makes no DeepSeek call in a process that does not write the archive (2026-10-01)") +def _(): + """The audit copies exist only to be archived. The dashboard daemon's + database writes are off (and NEWS_ARCHIVE_ENABLED=0 switches the + archive off anywhere): a call there buys labels nobody keeps.""" + saved_cache = _NE38._llm_sent_cache + saved_app = _ac38.get_app_config() + saved_day = (_NE38._audit_day, _NE38._audit_sent_today) + saved_env = os.environ.get("NEWS_ARCHIVE_ENABLED") + saved_writes = _ac38.db_writes_enabled() + _NE38._llm_sent_cache = {} + _NE38._audit_day, _NE38._audit_sent_today = "", 0 + _lc38._client = _StubLLM({}) + sent = [] + try: + ne = _NE38(_NC38()) + + def scores(texts): + sent.append(list(texts)) + return [(0.4, -0.6)] * len(texts) + ne._llm_headline_scores = scores + _ac38.set_app_config(_app38("primary", audit=5)) + os.environ.pop("NEWS_ARCHIVE_ENABLED", None) + + def cycle(tag): + rows = [_h38(f"Gold headline {tag}{i}") for i in range(4)] + with _Model38({"relevant": True, "score": 0.7}): + copies = ne._enrich_headline_scores(rows, "primary") + assert all(h.score_source == "model" for h in rows), "the model still scores" + return copies + + _ac38.set_db_writes_for_process(False) # the dashboard daemon + assert cycle("a") == [] and sent == [] + _ac38.set_db_writes_for_process(True) + os.environ["NEWS_ARCHIVE_ENABLED"] = "0" # archive switched off by hand + assert cycle("b") == [] and sent == [] + assert _NE38._audit_sent_today == 0 and _NE38._llm_sent_cache == {}, \ + "no budget spent, nothing marked as audited" + os.environ["NEWS_ARCHIVE_ENABLED"] = "1" # the bot: the audit runs as before + copies = cycle("c") + assert len(copies) == 4 and [len(b) for b in sent] == [4], (copies, sent) + finally: + _ac38.set_db_writes_for_process(saved_writes) + if saved_env is None: + os.environ.pop("NEWS_ARCHIVE_ENABLED", None) + else: + os.environ["NEWS_ARCHIVE_ENABLED"] = saved_env + _ac38.set_app_config(saved_app) + _NE38._llm_sent_cache = saved_cache + _NE38._audit_day, _NE38._audit_sent_today = saved_day + _lc38._client = None + + def _cycle38(mode, audit, model_results, llm_scores): """Run one fake RSS cycle under `mode` and return (returned headlines, archived rows by headline). Restores every piece of shared state.""" @@ -16012,6 +17560,1442 @@ def _(): _ac38.set_app_config(saved_app) +# ══════════════════════════════════════════════════════════════════════ +# MODULE 39: daily-trend veto + wide-stop cap (2026-09-30) +# The veto is the last step of analyze_from_dataframes: a finished setup +# against a clear daily trend (closed D1 bars; close vs its 50-bar EMA +# AND vs the close 60 bars earlier) is removed. The cap pulls a very wide +# swing / chart-pattern stop in AFTER the gates. Both are off by default. +# CPU only; the CUDA mirrors are pinned in Module 37. +# ══════════════════════════════════════════════════════════════════════ + +print("\n═══ Module 39: daily-trend veto + wide-stop cap ═══") + +import dataclasses as _dc39 +import hashlib as _hashlib39 + +import signal_aggregator as _sa39 +from signal_aggregator import TradeSetup as _TS39 + + +def _d1_39(end="2026-07-07", days=320, slope=1.5, sunday_close=None): + """Daily bars labelled 00:00 UTC ending at `end` (no Saturdays, WITH the + Sunday stub bars the provider emits). slope > 0 rises, < 0 falls.""" + idx = pd.date_range(end=pd.Timestamp(end, tz="UTC"), periods=days, freq="D") + idx = idx[idx.weekday != 5] + t = np.arange(len(idx), dtype=float) + close = 3300.0 + slope * t + 3.0 * np.sin(t / 2.0) + if sunday_close is not None: + close[idx.weekday == 6] = sunday_close + df = pd.DataFrame({"Open": close - 1.0, "High": close + 5.0, "Low": close - 5.0, + "Close": close, "Volume": 1000.0}, index=idx) + df.index.name = "Datetime" + return df + + +def _trend_agg39(mode="both", cap=0.0, tf=Timeframe.M5, confirms=(Timeframe.M15, Timeframe.H1)): + from pair_signals.xauusd import make_config as xau_make_config + cfg = xau_make_config(tf, list(confirms)) + cfg.xau_strategy.daily_trend_veto = mode + cfg.xau_strategy.wide_stop_cap_atr = cap + return SignalAggregator(cfg) + + +def _setup39(direction="BUY", strategy="GENERIC"): + entry = 3400.0 + sign = 1.0 if direction == "BUY" else -1.0 + return _TS39(pair=XAUUSD, display_name="XAU/USD", direction=direction, entry_price=entry, + stop_loss=entry - sign * 20.0, take_profit_1=entry + sign * 20.0, + take_profit_2=entry + sign * 30.0, risk_reward=1.0, risk_pips=200.0, + reward_pips=200.0, pip_value=0.10, strategy=strategy) + + +def _report39(direction="BUY", strategy="GENERIC", quality="A"): + r = SignalReport(symbol=XAUUSD, display_name="XAU/USD", timestamp="2026-07-08 14:55:00+00:00") + r.signal_quality = quality + r.final_signal = SignalStrength.BUY if direction == "BUY" else SignalStrength.SELL + r.action_recommendation = "kept as written" + r.trade_setup = _setup39(direction, strategy) + return r + + +@test("daily trend: series == the campaign formula; Sunday stubs ignored; warm-up reads 0; value on a prefix == value in the full series") +def _(): + for slope in (1.5, -1.5, 0.0): + d1 = _d1_39(slope=slope) + s = _sa39.daily_trend_series(d1) + kept = d1[d1.index.weekday != 6] + c = kept["Close"] + ema = c.ewm(span=50, adjust=False).mean() + chg = c / c.shift(60) - 1 + want = np.where((c > ema) & (chg > 0), 1, np.where((c < ema) & (chg < 0), -1, 0)) + assert s.dtype == np.int8 and list(s.index) == list(kept.index) + assert np.array_equal(s.to_numpy(), want), slope + assert (s.iloc[:60] == 0).all(), "no 60-bar change yet -> unclear" + assert not (s.index.weekday == 6).any() + up, down = _sa39.daily_trend_series(_d1_39(slope=1.5)), _sa39.daily_trend_series(_d1_39(slope=-1.5)) + assert (up.iloc[80:] == 1).all() and (down.iloc[80:] == -1).all() + assert _sa39.DAILY_TREND_EMA_SPAN == 50 and _sa39.DAILY_TREND_LOOKBACK == 60, "validated literals" + # Sunday stubs never count, whatever they print + clean = _d1_39(slope=-1.5) + wild = _d1_39(slope=-1.5, sunday_close=99999.0) + assert np.array_equal(_sa39.daily_trend_series(clean).to_numpy(), _sa39.daily_trend_series(wild).to_numpy()) + # prefix-stable (the replay slices the frame per bar; the twin evaluates it once) + d1 = pd.concat([_d1_39(end="2026-03-20", days=200, slope=1.5).iloc[:-1], + _d1_39(end="2026-07-07", days=110, slope=-6.0)]) + d1 = d1[~d1.index.duplicated()].sort_index() + full = _sa39.daily_trend_series(d1) + assert {-1, 0, 1} <= set(full.unique().tolist()) + for n in range(1, len(d1) + 1): + head = d1.iloc[:n] + got = _sa39.daily_trend(head) + kept_n = int((head.index.weekday != 6).sum()) + assert got == (int(full.iloc[kept_n - 1]) if kept_n else None), n + # nothing usable -> None, never an exception + assert _sa39.daily_trend(None) is None and _sa39.daily_trend(d1.iloc[:0]) is None + assert _sa39.daily_trend(d1[d1.index.weekday == 6]) is None + assert len(_sa39.daily_trend_series(None)) == 0 + + +@test("daily trend: mode validation — unknown veto mode / negative cap fail at construction; the D1 frame is needed only for a gold config with the veto on") +def _(): + from pair_signals.xauusd import make_config as xau_make_config + assert _sa39.DAILY_TREND_MODES == ("off", "both", "buy", "sell") + for tf in (Timeframe.M5, Timeframe.H1, Timeframe.H4): + cfg = xau_make_config(tf) + assert cfg.xau_strategy.daily_trend_veto == "off" and cfg.xau_strategy.wide_stop_cap_atr == 0.0, tf + assert _sa39.daily_trend_needed(cfg) is False, tf + # M15 override 7 (2026-09-30): the veto ships on M15 only, both directions + m15 = xau_make_config(Timeframe.M15) + assert m15.xau_strategy.daily_trend_veto == "both", "M15 ships the daily-trend veto (override 7)" + assert _sa39.daily_trend_needed(m15) is True + cfg = xau_make_config(Timeframe.M5) + for mode in ("both", "buy", "sell"): + cfg.xau_strategy.daily_trend_veto = mode + assert _sa39.daily_trend_needed(cfg) is True, mode + cfg.xau_strategy.enabled = False + assert _sa39.daily_trend_needed(cfg) is False, "the gold strategy switch gates it" + cfg.xau_strategy.enabled = True + cfg.display_name = "EUR/USD" + assert _sa39.daily_trend_needed(cfg) is False, "gold only (pair isolation)" + bad = xau_make_config(Timeframe.M5) + bad.xau_strategy.daily_trend_veto = "weekly" + for call in (lambda: SignalAggregator(bad), lambda: _sa39.daily_trend_needed(bad), + lambda: _sa39.daily_trend_mode_of(bad.xau_strategy)): + try: + call() + raise AssertionError("an unknown daily_trend_veto must raise") + except ValueError as e: + assert "daily_trend_veto" in str(e) and "weekly" in str(e), e + for value in (-1.0, float("nan")): + neg = xau_make_config(Timeframe.M5) + neg.xau_strategy.wide_stop_cap_atr = value + try: + SignalAggregator(neg) + raise AssertionError("a negative / NaN wide_stop_cap_atr must raise") + except ValueError as e: + assert "wide_stop_cap_atr" in str(e), e + + +@test("daily trend veto: direction x mode x trend matrix — only a setup AGAINST a clear trend is removed (grade D, no setup, one named warning)") +def _(): + frames = {1: _d1_39(slope=1.5), -1: _d1_39(slope=-1.5), 0: _d1_39(days=40, slope=1.5)} + assert [_sa39.daily_trend(frames[k]) for k in (1, -1, 0)] == [1, -1, 0] + for mode in ("both", "buy", "sell", "off"): + agg = _trend_agg39(mode) + for direction in ("BUY", "SELL"): + for trend, d1 in frames.items(): + for strategy in ("GENERIC", "XAU_ATR", "XAU_CP", "XAU_PB"): + r = _report39(direction, strategy, quality="B") + agg._apply_daily_trend_veto(r, {Timeframe.D1: d1}) + against = ((direction == "BUY" and trend < 0 and mode in ("both", "buy")) + or (direction == "SELL" and trend > 0 and mode in ("both", "sell"))) + label = (mode, direction, trend, strategy) + if mode == "off": + assert r.daily_trend is None and r.trade_setup is not None and r.warnings == [], label + continue + assert r.daily_trend == trend, label + if against: + assert r.trade_setup is None and r.signal_quality == "D", label + word = "falling" if direction == "BUY" else "rising" + assert r.warnings == [f"DAILY TREND FILTER: {direction} against a {word} daily trend — no trade"], r.warnings + else: + assert r.trade_setup is not None and r.signal_quality == "B" and r.warnings == [], label + # the gate-chain convention: signal and action text stay as written + assert r.action_recommendation == "kept as written", label + assert r.final_signal == (SignalStrength.BUY if direction == "BUY" else SignalStrength.SELL), label + # no setup: nothing to veto, the trend is still stamped + agg = _trend_agg39("both") + r = _report39("BUY") + r.trade_setup = None + agg._apply_daily_trend_veto(r, {Timeframe.D1: frames[-1]}) + assert r.daily_trend == -1 and r.warnings == [] and r.signal_quality == "A" + # veto requested but no daily frame in the slices: a notice, never a veto and never a gate prefix + for dfs in ({}, {Timeframe.D1: frames[-1].iloc[:0]}): + r = _report39("BUY") + agg._apply_daily_trend_veto(r, dfs) + assert r.daily_trend is None and r.trade_setup is not None and r.signal_quality == "A" + assert r.warnings == ["Daily trend filter requested but 1d data unavailable"], r.warnings + # gold only: another pair's report is never touched + r = _report39("BUY") + r.display_name = "EUR/USD" + agg._apply_daily_trend_veto(r, {Timeframe.D1: frames[-1]}) + assert r.trade_setup is not None and r.daily_trend is None and r.warnings == [] + + +@test("daily trend veto: through analyze_from_dataframes — an A-grade BUY against a falling daily trend is removed; rising / unclear / off leave the report identical") +def _(): + dfs = _bc_actionable_dfs() + control = _trend_agg39("off").analyze_from_dataframes(XAUUSD, dfs, use_news=False) + assert control.signal_quality == "A" and control.trade_setup is not None \ + and control.trade_setup.direction == "BUY", \ + f"fixture drifted: expected an actionable A-grade BUY, got {control.signal_quality}" + assert control.daily_trend is None + + def run(mode, d1): + frames = dict(dfs) + if d1 is not None: + frames[Timeframe.D1] = d1 + return _trend_agg39(mode).analyze_from_dataframes(XAUUSD, frames, use_news=False) + + falling, rising, unclear = _d1_39(slope=-1.5), _d1_39(slope=1.5), _d1_39(days=40) + for mode in ("both", "buy"): + gated = run(mode, falling) + assert gated.trade_setup is None and gated.signal_quality == "D", (mode, gated.signal_quality) + assert gated.warnings[-1] == "DAILY TREND FILTER: BUY against a falling daily trend — no trade", gated.warnings + assert gated.warnings[:-1] == control.warnings, "the veto only APPENDS its own line" + assert gated.daily_trend == -1 + assert gated.final_signal == control.final_signal and gated.final_score == control.final_score + assert gated.action_recommendation == control.action_recommendation + assert gated.bullish_reasons == control.bullish_reasons and gated.cleaned_warnings == control.cleaned_warnings + for mode, d1, trend in (("sell", falling, -1), ("both", rising, 1), ("both", unclear, 0), ("buy", rising, 1)): + kept = run(mode, d1) + assert kept.daily_trend == trend, (mode, trend) + assert kept.signal_quality == "A" and kept.warnings == control.warnings, (mode, kept.warnings) + assert _dc39.asdict(kept.trade_setup) == _dc39.asdict(control.trade_setup), mode + # off: a daily frame in the slices is ignored and nothing is stamped + off = run("off", falling) + assert off.daily_trend is None and off.warnings == control.warnings + assert _dc39.asdict(off.trade_setup) == _dc39.asdict(control.trade_setup) + # veto on but the frame missing: the setup ships with the notice + missing = run("both", None) + assert missing.trade_setup is not None and missing.signal_quality == "A" + assert missing.warnings == control.warnings + ["Daily trend filter requested but 1d data unavailable"] + + +@test("daily trend veto: runs AFTER every archetype — a chart-pattern setup is vetoed too, and its stale-veto cleanup has already happened") +def _(): + dfs = dict(_bc_actionable_dfs()) + dfs[Timeframe.D1] = _d1_39(slope=-1.5) + agg = _trend_agg39("both") + assert agg.config.xau_strategy.cp_enabled is True + order = [] + core = agg._attach_core_setup + + def no_core_setup(report, pip_value): + core(report, pip_value) + report.trade_setup = None # leave the slot empty for the pattern path + report.warnings.append("SPREAD GATE: target $3.00 < $9.00 minimum — costs would eat the edge") + order.append("core") + + def pattern_setup(report, primary_df, pip_value): + report.trade_setup = _setup39("BUY", "XAU_CP") + report.signal_quality = "B" + order.append("cp") + + stale = SignalAggregator._drop_stale_spread_gate_vetoes + + def stale_spy(report): + order.append("stale:" + ("setup" if report.trade_setup is not None else "none")) + stale(report) + + veto = agg._apply_daily_trend_veto + + def veto_spy(report, frames): + order.append("veto") + veto(report, frames) + + with patch.object(agg, "_attach_core_setup", no_core_setup), \ + patch.object(agg, "_try_chart_pattern", pattern_setup), \ + patch.object(agg, "_drop_stale_spread_gate_vetoes", stale_spy), \ + patch.object(agg, "_apply_daily_trend_veto", veto_spy): + report = agg.analyze_from_dataframes(XAUUSD, dfs, use_news=False) + assert order == ["core", "cp", "stale:setup", "veto"], order + assert report.trade_setup is None and report.signal_quality == "D" + assert report.warnings[-1].startswith("DAILY TREND FILTER: BUY") + assert any(w.startswith("SPREAD GATE:") for w in report.cleaned_warnings), \ + "step 11 ran while the pattern setup still existed" + assert not any(w.startswith("SPREAD GATE:") for w in report.warnings) + import signal_log as slog + assert slog.gate_reason(report.warnings, report.cleaned_warnings, report.signal_quality, + False).startswith("DAILY TREND FILTER") + + +@test("daily trend veto: live wrapper fetches D1 as a REQUIRED frame only when the veto is on; a failed D1 fetch fails the scan; both backtest loaders append it under the same guard") +def _(): + import sqlite3 + import signal_log as slog + from pair_signals import PAIR_PRODUCERS + + class _QuietNews(_NoNews): + def get_event_risk_level(self, symbol, context): + return "clear", "" + + def get_pair_sentiment(self, symbol, context): + return 0.0, "" + + def _run(mode, raise_for=None): + cfg = PAIR_PRODUCERS[XAUUSD](Timeframe.M15, [Timeframe.M5, Timeframe.H1]) + cfg.xau_strategy.daily_trend_veto = mode + cfg.xau_strategy.allowed_weekdays = [] # deterministic on any day + agg = SignalAggregator(cfg) + agg._get_news_engine = lambda: _QuietNews() + fetched = [] + + class _Stub: + def get_pair_info(self, symbol): + return PAIR_INFO[XAUUSD] + + def fetch_ohlcv(self, symbol, timeframe, lookback_days=None): + fetched.append(timeframe) + if timeframe is raise_for: + raise RuntimeError("boom") + if timeframe is Timeframe.D1: + return _d1_39(end=str(pd.Timestamp.now(tz="UTC").date() - pd.Timedelta(days=1))) + return make_xau_ohlcv(300, seed=42) + + def is_data_fresh(self, df, symbol, timeframe): + return True + + agg.data_provider = _Stub() + fd, path = tempfile.mkstemp(suffix=".db") + os.close(fd) + os.remove(path) + try: + os.environ["SIGNAL_LOG_PATH"] = path + os.environ.pop("SIGNAL_LOG_ENABLED", None) + with patch("signal_aggregator._is_forex_weekend", return_value=False): + report = agg.analyze_pair(symbol=XAUUSD, log_source="telegram") + conn = sqlite3.connect(path) + row = conn.execute( + "SELECT scan_kind, gate_reason, daily_trend, config_fingerprint FROM signals").fetchone() + conn.close() + return report, row, fetched + finally: + os.environ.pop("SIGNAL_LOG_PATH", None) + os.environ.pop("SIGNAL_LOG_ENABLED", None) + slog.reset_loggers_for_tests() + if os.path.exists(path): + os.remove(path) + + # veto off: D1 is NOT a required frame. The live wrapper still reads it + # once, after the analysis, for the alert text and the log (display + # only): it never joins the scan's frames and losing it never fails + # the scan. + report, row, fetched = _run("off") + assert fetched == [Timeframe.M15, Timeframe.M5, Timeframe.H1, Timeframe.D1], fetched + assert Timeframe.D1 not in report._frames, "the display frame never joins the scan's frames" + assert row[0] == "scan" and report.daily_trend == 1 and row[2] == 1, (row, report.daily_trend) + fp_off = row[3] + report, row, fetched = _run("off", raise_for=Timeframe.D1) + assert row[0] == "scan" and report.primary_analysis is not None, \ + "a failed DISPLAY fetch must not fail the scan" + assert report.daily_trend is None and row[2] is None + assert not any("DATA UNAVAILABLE" in w or "aily trend" in w for w in report.warnings), report.warnings + report, row, fetched = _run("both") + assert fetched == [Timeframe.M15, Timeframe.M5, Timeframe.H1, Timeframe.D1], fetched + assert row[0] == "scan" and report.daily_trend == 1 and row[2] == 1, (row, report.daily_trend) + assert row[3] != fp_off, "the gate fingerprint names the veto when it is on" + assert Timeframe.D1 in report._frames, "the scan's own frames are handed to the add-ons unchanged" + report, row, fetched = _run("both", raise_for=Timeframe.D1) + assert report.signal_quality == "F" and report.trade_setup is None and report.primary_analysis is None + assert len([w for w in report.warnings if w.startswith("DATA UNAVAILABLE: 1d")]) == 1, report.warnings + assert row[0] == "failed" and row[1].startswith("DATA UNAVAILABLE:"), row + # rule 4: the reference replay and the twin load the frame under the same guard + for script in ("backtest/gpu_backtest.py", "backtest/cudaBasedBacktesting.py"): + src = Path(script).read_text(encoding="utf-8") + assert "if daily_trend_needed(platform_cfg) and Timeframe.D1 not in timeframes:" in src, script + assert "timeframes.append(Timeframe.D1)" in src, script + live = Path("signal_aggregator.py").read_text(encoding="utf-8") + assert "if daily_trend_needed(self.config) and Timeframe.D1 not in timeframes:" in live + + +@test("wide-stop cap: GENERIC stop pulled in to K x ATR after the gates, never inside the generator's floor; K = 0 / a loose K change nothing; entry and gated target untouched") +def _(): + dfs = _bc_actionable_dfs() + control = _trend_agg39("off").analyze_from_dataframes(XAUUSD, dfs, use_news=False) + base = control.trade_setup + assert base is not None and base.strategy == "GENERIC" and base.direction == "BUY", base + atr = control.primary_analysis.key_levels["atr"] + base_dist = base.entry_price - base.stop_loss + floor = max(2.0 * atr, 8.0) + assert base_dist > floor * 1.2, f"fixture drifted: the swing stop must sit beyond the floor ({base_dist:.2f} vs {floor:.2f})" + + def run(cap): + return _trend_agg39("off", cap).analyze_from_dataframes(XAUUSD, dfs, use_news=False) + + for cap in (0.0, 1000.0, base_dist / atr + 0.01): + same = run(cap) + assert _dc39.asdict(same.trade_setup) == _dc39.asdict(base), cap + assert same.warnings == control.warnings and same.signal_quality == control.signal_quality + k_mid = (base_dist / atr + floor / atr) / 2.0 # between the floor and the swing stop + mid = run(k_mid).trade_setup + assert mid is not None and mid.strategy == "GENERIC" and mid.direction == "BUY" + assert mid.entry_price == base.entry_price + assert abs((mid.entry_price - mid.stop_loss) - k_mid * atr) < 1e-9, (mid.stop_loss, k_mid * atr) + assert abs(mid.risk_pips - (k_mid * atr) / mid.pip_value) < 1e-6 + tight = run(0.5).trade_setup # K below the floor: the floor wins + assert abs((tight.entry_price - tight.stop_loss) - floor) < 1e-9, (tight.stop_loss, floor) + for capped in (mid, tight): + dist = capped.entry_price - capped.stop_loss + tp1 = capped.take_profit_1 - capped.entry_price + assert tp1 >= dist - 1e-9, "printed target >= 1R of the capped stop" + assert tp1 <= (base.take_profit_1 - base.entry_price) + 1e-9, "a smaller stop never raises the printed target" + assert capped.risk_reward == round(tp1 / dist, 2) + assert capped.take_profit_2 > capped.take_profit_1 + assert capped.trail_arm_r == base.trail_arm_r and capped.trail_dist_r == base.trail_dist_r + # the gold ATR setup (2 x ATR stop) is never capped: M15 grade A through the ATR generator + m15 = {Timeframe.M15: make_xau_bars(end="2026-07-08 14:45", freq="15min", drift=0.8, seed=100), + Timeframe.M5: make_xau_bars(end="2026-07-08 14:55", drift=0.8, seed=0), + Timeframe.H1: make_xau_bars(end="2026-07-08 14:00", freq="60min", drift=0.8, seed=200)} + a0 = _trend_agg39("off", 0.0, Timeframe.M15, (Timeframe.M5, Timeframe.H1)).analyze_from_dataframes(XAUUSD, m15, use_news=False) + a1 = _trend_agg39("off", 0.5, Timeframe.M15, (Timeframe.M5, Timeframe.H1)).analyze_from_dataframes(XAUUSD, m15, use_news=False) + assert (a0.trade_setup is None) == (a1.trade_setup is None) + if a0.trade_setup is not None and a0.trade_setup.strategy == "XAU_ATR": + assert _dc39.asdict(a0.trade_setup) == _dc39.asdict(a1.trade_setup) + + +@test("wide-stop cap: chart-pattern setup — the stop is capped after its RR and spread gates; a pattern the gates reject is still rejected") +def _(): + dfs = _bc_actionable_dfs() + primary = dfs[Timeframe.M5] + + def run(cap, anchor_atr, height_atr): + agg = _trend_agg39("off", cap) + report = agg.analyze_from_dataframes(XAUUSD, dfs, use_news=False) + atr = report.primary_analysis.key_levels["atr"] + report.trade_setup = None + report.volatility_regime = "normal" + entry = report.current_price + hit = {"pattern": "double_bottom", "direction": "BUY", "anchor": entry - anchor_atr * atr, + "trigger": entry - 0.2 * atr, "height": height_atr * atr} + with patch("signal_aggregator.detect_chart_pattern", return_value=hit): + agg._try_chart_pattern(report, primary, 0.10) + return report.trade_setup, atr, entry + + base, atr, entry = run(0.0, 10.0, 12.0) + assert base is not None and base.strategy == "XAU_CP", base + assert abs((entry - base.stop_loss) - 10.5 * atr) < 1e-9 # anchor - 0.5 ATR + capped, _, _ = run(6.0, 10.0, 12.0) + assert capped is not None and capped.strategy == "XAU_CP" + assert abs((entry - capped.stop_loss) - 6.0 * atr) < 1e-9 + assert capped.take_profit_1 == base.take_profit_1 and capped.take_profit_2 == base.take_profit_2 + assert capped.risk_reward == round(12.0 / 6.0, 2) and base.risk_reward == round(12.0 / 10.5, 2) + loose, _, _ = run(11.0, 10.0, 12.0) + assert _dc39.asdict(loose) == _dc39.asdict(base), "a cap beyond the stop changes nothing" + # the measured move fails the RR gate on the UNCAPPED stop: still no setup with the cap + rejected, _, _ = run(6.0, 10.0, 4.0) + assert rejected is None, "the gates judge the uncapped geometry (the trade set cannot change)" + assert run(0.0, 10.0, 4.0)[0] is None + + +@test("fingerprints: the two knobs enter the gate fingerprint only when in force; default-off configs keep the hash their live rows carry; the engine fingerprint never moves") +def _(): + from pair_signals.xauusd import make_config as xau_make_config + + def legacy_gate_fp(cfg): + """The pre-2026-09-30 payload, key for key.""" + xs, bt = cfg.xau_strategy, cfg.backtest + payload = { + "primary": cfg.primary_timeframe.value, + "confirm": [tf.value for tf in cfg.confirmation_timeframes], + "blocked_hours": list(xs.blocked_hours_utc), + "sell_blocked_hours": list(xs.sell_blocked_hours_utc), + "allowed_weekdays": list(xs.allowed_weekdays), + "blocked_regimes": list(xs.blocked_regimes), + "sell_blocked_regimes": list(xs.sell_blocked_regimes), + "sell_min_quality": list(xs.sell_min_quality), + "sell_vol_regimes": list(xs.sell_vol_regimes), + "big_candle": xs.big_candle_atr_multiple, + "cp": bool(xs.cp_enabled), + "pb": bool(xs.m5_pullback_enabled), + "session": [bt.session_filter, bt.session_start_hour, bt.session_end_hour], + "regime_filter": bool(bt.regime_filter), + } + return _hashlib39.sha1(json.dumps(payload, sort_keys=True, default=str).encode("utf-8")).hexdigest()[:12] + + for tf in (Timeframe.M5, Timeframe.M15, Timeframe.H1, Timeframe.H4): + cfg = xau_make_config(tf) + # as shipped: only M15 (override 7) carries a new key, so only its + # live rows change hash at the 2026-09-30 deploy + shipped = SignalAggregator(cfg).gate_config_fingerprint() + assert (shipped != legacy_gate_fp(cfg)) == (tf == Timeframe.M15), tf + cfg.xau_strategy.daily_trend_veto = "off" + cfg.xau_strategy.wide_stop_cap_atr = 0.0 + agg = SignalAggregator(cfg) + fp0, eng0 = agg.gate_config_fingerprint(), agg.engine_config_fingerprint() + assert fp0 == legacy_gate_fp(cfg), tf + seen = {fp0} + for mode in ("both", "buy", "sell"): + cfg.xau_strategy.daily_trend_veto = mode + seen.add(agg.gate_config_fingerprint()) + assert len(seen) == 4, (tf, seen) + cfg.xau_strategy.daily_trend_veto = "off" + cfg.xau_strategy.wide_stop_cap_atr = 6.0 + fp_cap = agg.gate_config_fingerprint() + assert fp_cap not in seen + cfg.xau_strategy.wide_stop_cap_atr = 5.0 + assert agg.gate_config_fingerprint() not in seen | {fp_cap} + cfg.xau_strategy.daily_trend_veto = "both" + assert agg.engine_config_fingerprint() == eng0, "gold-strategy knobs are not scoring config" + cfg.xau_strategy.daily_trend_veto = "off" + cfg.xau_strategy.wide_stop_cap_atr = 0.0 + assert agg.gate_config_fingerprint() == fp0 + + +def _legacy_signals_db39(path, without): + """An older signals.db that lacks the columns in `without`: the base + schema plus every additive column except those. Built forwards on + purpose: ALTER TABLE ... DROP COLUMN needs SQLite 3.35 and the live + box runs 3.34 (2026-10-01 review).""" + import sqlite3 + import signal_log as slog + conn = sqlite3.connect(path) + try: + conn.executescript(slog._SCHEMA) + have = {row[1] for row in conn.execute("PRAGMA table_info(signals)")} + for name, col_type in slog._ADDITIVE_SIGNAL_COLS: + if name not in have and name not in without: + conn.execute(f"ALTER TABLE signals ADD COLUMN {name} {col_type}") + conn.commit() + cols = {row[1] for row in conn.execute("PRAGMA table_info(signals)")} + assert not (set(without) & cols), f"fixture guard: {without} must be absent from the legacy table" + finally: + conn.close() + + +def _rm_db39(path): + for suffix in ("", "-wal", "-shm"): + if os.path.exists(path + suffix): + os.remove(path + suffix) + + +@test("signal_log: daily_trend is stored (-1 / 0 / +1, NULL when the report carries none) and the column is added to an existing database") +def _(): + import signal_log as slog + fd, path = tempfile.mkstemp(suffix=".db") + os.close(fd) + legacy = path + ".legacy" + try: + lg = slog.SignalLogger(path) + ids = {} + for value in (-1, 0, 1, None): + r = _m27_report(setup=False, quality="D") + r.daily_trend = value + ids[value] = lg.log(r, source="telegram", timeframe="15min") + got = {v: lg._conn.execute("SELECT daily_trend FROM signals WHERE id=?", (i,)).fetchone()[0] + for v, i in ids.items()} + assert got == {-1: -1, 0: 0, 1: 1, None: None}, got + lg.close() + slog.reset_loggers_for_tests() + # an older database without the column gains it on the next open + _legacy_signals_db39(legacy, without=("daily_trend",)) + lg = slog.SignalLogger(legacy) + cols = [row[1] for row in lg._conn.execute("PRAGMA table_info(signals)")] + assert "daily_trend" in cols, "the additive migration must add the column" + r = _m27_report(setup=False, quality="D") + r.daily_trend = 1 + rowid = lg.log(r, source="telegram", timeframe="15min") + assert lg._conn.execute("SELECT daily_trend FROM signals WHERE id=?", (rowid,)).fetchone()[0] == 1 + lg.close() + finally: + slog.reset_loggers_for_tests() + _rm_db39(path) + _rm_db39(legacy) + + + +# ══════════════════════════════════════════════════════════════════════ +# MODULE 40: trade size, open exposure and daily-trend lines (2026-09-30) +# Display only: exposure.py, signal_log.alerted_at, main.daily_trend_words, +# the config.json trade_size block and the three surfaces that print them. +# ══════════════════════════════════════════════════════════════════════ + +print("\n═══ Module 40: trade size, open exposure and daily-trend lines ═══") + +import asyncio as _aio40 +import contextlib as _ctx40 +import io as _io40 +import sqlite3 as _sq40 + +import exposure as _ex40 +import signal_log as _slog40 + + +@_ctx40.contextmanager +def _trade_size40(**block): + """config.json's trade_size block on the suite's hermetic app config.""" + saved = dict(_TEST_APP.trade_size_settings) + _TEST_APP.trade_size_settings = dict(block) + try: + yield + finally: + _TEST_APP.trade_size_settings = saved + + +def _rm_db40(path): + for suffix in ("", "-wal", "-shm"): + if os.path.exists(path + suffix): + os.remove(path + suffix) + + +def _no_clock40(message): + """An alert text without its wall-clock line (two builds may straddle a minute).""" + return "\n".join(line for line in message.splitlines() if not line.startswith("🕐")) + + +@test("trade_size: config block — off by default and at risk 0, defaults filled in, bad values refused, the example file documents it off") +def _(): + assert "trade_size" in app_config._KNOWN_TOP + empty = load_app_config(_write_json({})) + assert empty.trade_size_settings == {} and app_config.trade_size_settings(empty) is None + zero = load_app_config(_write_json({"trade_size": {"risk_per_trade": 0}})) + assert app_config.trade_size_settings(zero) is None, "risk 0 = off" + on = app_config.trade_size_settings(load_app_config(_write_json({"trade_size": {"risk_per_trade": 150}}))) + assert on == {"risk_per_trade": 150, "contract_size": 100.0, "lot_step": 0.01, + "telegram_channel": False, "dashboard": False}, on + full_block = {"risk_per_trade": 75.5, "contract_size": 10, "lot_step": 0.1, + "telegram_channel": True, "dashboard": True} + assert app_config.trade_size_settings(load_app_config(_write_json({"trade_size": full_block}))) == full_block + for bad in ({"risk_per_trade": -1}, {"risk_per_trade": "100"}, {"risk_per_trade": True}, + {"risk_per_trade": float("nan")}, {"contract_size": 0}, {"contract_size": "100"}, + {"lot_step": 0}, {"lot_step": -0.01}, {"telegram_channel": "yes"}, {"dashboard": 1}, + {"risk": 100}): + try: + load_app_config(_write_json({"trade_size": bad})) + raise AssertionError(f"trade_size {bad} must be refused") + except AppConfigError as e: + assert "trade_size" in str(e), e + example = load_app_config(os.path.join(os.path.dirname(__file__), "config.json.example")) + assert example.trade_size_settings == {"risk_per_trade": 0, "contract_size": 100, "lot_step": 0.01, + "telegram_channel": False, "dashboard": False} + assert app_config.trade_size_settings(example) is None, "the example ships the size line off" + assert app_config.trade_size_settings() is None, "the suite's hermetic config sets no size" + + +@test("trade size: the lot size never rounds up past the risk limit; the line uses the printed prices; a stop too wide for the limit says so; off or unusable inputs give no line") +def _(): + assert _ex40.lot_size(20.0, 100.0) == (0.05, 100.0) + assert _ex40.lot_size(21.37, 100.0) == (0.04, 85.48) + assert _ex40.lot_size(14.29, 100.0) == (0.06, 85.74), "6.998 steps floor to 6, never 7" + assert _ex40.lot_size(10.0, 70.0) == (0.07, 70.0), "an exact multiple is kept (no 0.0699 float artefact)" + assert _ex40.lot_size(150.0, 100.0) == (0.0, 0.0) + assert _ex40.lot_size(20.0, 1000.0, 100.0, 0.1) == (0.5, 1000.0) + assert _ex40.lot_size(20.0, 100.0, 10.0, 0.01) == (0.5, 100.0) + for bad in ((0.0, 100.0), (-5.0, 100.0), (20.0, 0.0), (float("nan"), 100.0), + (20.0, float("inf")), ("x", 100.0), (20.0, 100.0, 0.0), (20.0, 100.0, 100.0, 0.0)): + assert _ex40.lot_size(*bad) == (0.0, 0.0), bad + rng = np.random.default_rng(40) + for _i in range(500): + dist = round(float(rng.uniform(2.0, 80.0)), 2) + risk = round(float(rng.uniform(10.0, 2000.0)), 2) + lots, at_risk = _ex40.lot_size(dist, risk) + assert at_risk <= risk + 1e-9, (dist, risk, lots, at_risk) + assert (round(lots * 100) + 1) * dist > risk - 1e-9, (dist, risk, lots, "one more step would break the limit") + assert abs(lots * 100 - round(lots * 100)) < 1e-9, "a whole number of 0.01 steps" + setup = _setup39("BUY") # entry 3400.00, stop 3380.00 + st = {"risk_per_trade": 100.0, "contract_size": 100.0, "lot_step": 0.01} + assert _ex40.size_line(setup, st) == "Size: 0.05 lots risks $100.00 at the stop (your limit $100.00 per trade)" + setup.stop_loss = 3378.634 # printed 3378.63: a $21.37 stop + assert _ex40.size_line(setup, st) == "Size: 0.04 lots risks $85.48 at the stop (your limit $100.00 per trade)" + drift = _setup39("BUY") + drift.entry_price, drift.stop_loss = 4150.07, 4130.07 # 20.0 exactly as printed, whatever the floats say + assert _ex40.size_line(drift, st).startswith("Size: 0.05 lots risks $100.00") + setup.stop_loss = 3250.0 + assert _ex40.size_line(setup, st) == ("Size: the stop is too wide for your $100.00 limit — the smallest " + "size (0.01 lots) would risk $150.00") + sell = _setup39("SELL") # entry 3400.00, stop 3420.00 + assert _ex40.size_line(sell, {"risk_per_trade": 2500, "contract_size": 100.0, "lot_step": 0.1}) == \ + "Size: 1.2 lots risks $2,400.00 at the stop (your limit $2,500.00 per trade)" + for off in (None, {}, {"risk_per_trade": 0}): + assert _ex40.size_line(sell, off) == "", off + assert _ex40.size_line(None, st) == "" + flat = _setup39("BUY") + flat.stop_loss = flat.entry_price + assert _ex40.size_line(flat, st) == "" + broken = _setup39("BUY") + broken.stop_loss = float("nan") + assert _ex40.size_line(broken, st) == "" + + +@test("signal_log: alerted_at is stamped once; open_alerted_setups returns the alerted plans whose outcome is still open — one per event, own plan excluded, stale rows dropped, read-only, never raising") +def _(): + fd, path = tempfile.mkstemp(suffix=".db") + os.close(fd) + os.remove(path) + try: + assert _slog40.open_alerted_setups(XAUUSD, path=path) == [], "no database yet" + lg = _slog40.SignalLogger(path) + assert "alerted_at" in [r[1] for r in lg._conn.execute("PRAGMA table_info(signals)")] + assert "idx_signals_alerted" in [r[1] for r in lg._conn.execute("PRAGMA index_list(signals)")] + + def log(entry, sl, direction, tf): + return lg.log(_m27_report(entry=entry, sl=sl, direction=direction, tf=tf), + source="telegram", timeframe=tf.value) + + a = log(2450.5, 2445.0, "BUY", Timeframe.M5) + b = log(2452.0, 2440.0, "BUY", Timeframe.M15) + c = log(2449.0, 2460.0, "SELL", Timeframe.H1) + d = log(2455.0, 2447.0, "BUY", Timeframe.M5) + status = dict(lg._conn.execute("SELECT id, advisory_outcome_status FROM signals").fetchall()) + assert all(status[i] == "PENDING" for i in (a, b, c, d)), status + assert _slog40.open_alerted_setups(XAUUSD, path=path) == [], "scanned but never alerted" + for rowid in (a, b, c): + lg.mark_alerted(rowid) + stamp = lambda i: lg._conn.execute("SELECT alerted_at FROM signals WHERE id=?", (i,)).fetchone()[0] + first = stamp(a) + assert first and datetime.fromisoformat(first).tzinfo is not None + lg.mark_alerted(a, "2030-01-01T00:00:00+00:00") + assert stamp(a) == first, "the first stamp wins" + assert stamp(d) is None + rows = _slog40.open_alerted_setups(XAUUSD, path=path) + assert [(r["id"], r["timeframe"], r["direction"]) for r in rows] == \ + [(a, "5min", "BUY"), (b, "15min", "BUY"), (c, "1h", "SELL")], rows + assert rows[0]["entry"] == 2450.5 and rows[0]["stop_loss"] == 2445.0 and rows[0]["event_id"] == a + assert rows[0]["setup_key"] == "5min|BUY|2450.50|2445.00|GENERIC" + assert _slog40.open_alerted_setups("EUR/USD", path=path) == [] + own = _slog40.open_alerted_setups(XAUUSD, exclude_key=rows[0]["setup_key"], path=path) + assert [r["id"] for r in own] == [b, c], "the plan being alerted now never counts itself" + # a re-log of the same setup inside the dedup gap is the same event: + # one entry, even when both rows were alerted + dup = log(2450.5, 2445.0, "BUY", Timeframe.M5) + assert lg._conn.execute("SELECT advisory_outcome_status, advisory_event_id FROM signals WHERE id=?", + (dup,)).fetchone() == ("DUPLICATE", a) + lg.mark_alerted(dup) + assert [r["id"] for r in _slog40.open_alerted_setups(XAUUSD, path=path)] == [a, b, c] + # a duplicate row alerted while its canonical row was not: counted once, through the event + dup2 = log(2455.0, 2447.0, "BUY", Timeframe.M5) + lg.mark_alerted(dup2) + rows = _slog40.open_alerted_setups(XAUUSD, path=path) + assert [(r["id"], r["event_id"]) for r in rows] == [(a, a), (b, b), (c, c), (dup2, d)], rows + # a resolved outcome is no longer open + lg._conn.execute("UPDATE signals SET advisory_outcome_status='RESOLVED' WHERE id=?", (b,)) + lg._conn.commit() + assert [r["id"] for r in _slog40.open_alerted_setups(XAUUSD, path=path)] == [a, c, dup2] + # a PENDING row the resolver never reached is not shown as open for ever: + # M5 drops after about 4.5 days, H1 after about 20 + now = datetime.now(timezone.utc) + assert [r["id"] for r in _slog40.open_alerted_setups(XAUUSD, now=now + timedelta(days=10), path=path)] == [c] + assert _slog40.open_alerted_setups(XAUUSD, now=now + timedelta(days=30), path=path) == [] + # read-only users are unaffected by the process write switch (the dashboard) + app_config.set_db_writes_for_process(False) + try: + assert [r["id"] for r in _slog40.open_alerted_setups(XAUUSD, path=path)] == [a, c, dup2] + finally: + app_config.set_db_writes_for_process(True) + lg.close() + body = Path("signal_log.py").read_text(encoding="utf-8").split("def open_alerted_setups")[1] \ + .split("\ndef ")[0] + assert '?mode=ro", uri=True' in body and "immutable" not in body, "CLAUDE.md section 5: mode=ro, never immutable" + assert "UPDATE" not in body and "INSERT" not in body + # an older database without the column: an empty list, never an exception; + # the next writer open adds the column and its index (built forwards, + # no DROP COLUMN: the live box's SQLite 3.34 has none) + legacy = path + ".legacy" + _legacy_signals_db39(legacy, without=("alerted_at",)) + assert _slog40.open_alerted_setups(XAUUSD, path=legacy) == [] + lg = _slog40.SignalLogger(legacy) + assert "alerted_at" in [r[1] for r in lg._conn.execute("PRAGMA table_info(signals)")] + assert "idx_signals_alerted" in [r[1] for r in lg._conn.execute("PRAGMA index_list(signals)")] + lg.close() + finally: + _slog40.reset_loggers_for_tests() + _rm_db40(path) + _rm_db40(path + ".legacy") + + +@test("signal_log.mark_alerted(report): stamps the row log_signal wrote for this scan; a report that was never logged, or a process with writes off, is a silent no-op") +def _(): + fd, path = tempfile.mkstemp(suffix=".db") + os.close(fd) + os.remove(path) + try: + os.environ["SIGNAL_LOG_PATH"] = path + os.environ.pop("SIGNAL_LOG_ENABLED", None) + _slog40.reset_loggers_for_tests() + r = _m27_report(tf=Timeframe.M15) + _slog40.log_signal(r, source="telegram", timeframe="15min") + rowid = r._signal_log_rowid + _slog40.mark_alerted(r) + conn = _sq40.connect(path) + assert conn.execute("SELECT alerted_at FROM signals WHERE id=?", (rowid,)).fetchone()[0] + conn.close() + assert [x["id"] for x in _slog40.open_alerted_setups(XAUUSD)] == [rowid], \ + "the default path is the configured database" + _slog40.mark_alerted(_m27_report()) # never logged: no rowid + _slog40.mark_alerted(object()) # not even a report + r2 = _m27_report(entry=2460.0, sl=2452.0, tf=Timeframe.M15) + _slog40.log_signal(r2, source="telegram", timeframe="15min") + app_config.set_db_writes_for_process(False) + try: + _slog40.mark_alerted(r2) + finally: + app_config.set_db_writes_for_process(True) + conn = _sq40.connect(path) + assert conn.execute("SELECT alerted_at FROM signals WHERE id=?", + (r2._signal_log_rowid,)).fetchone()[0] is None + conn.close() + finally: + os.environ.pop("SIGNAL_LOG_PATH", None) + _slog40.reset_loggers_for_tests() + _rm_db40(path) + + +@test("exposure.attach_exposure: size line on the owner's surfaces only (the dashboard needs trade_size.dashboard); the open line counts same-direction plans and excludes this scan's own; no setup, the feature off or a failing reader leave the lines empty") +def _(): + r = _m27_report(entry=2450.5, sl=2440.5, direction="BUY", tf=Timeframe.M15) # a $10.00 stop + seen = {} + + def fake_open(symbol, exclude_key=None, **kw): + seen["args"] = (symbol, exclude_key) + return [{"direction": "BUY", "timeframe": "5min"}, {"direction": "BUY", "timeframe": "15min"}, + {"direction": "SELL", "timeframe": "1h"}] + + two_buys = "Already open: 2 buy plans alerted earlier (M5, M15) — taking this one adds to the same position" + with patch.object(_slog40, "open_alerted_setups", fake_open): + for surface in ("telegram", "cli", "dashboard"): # no trade_size in the suite's config + assert _ex40.attach_exposure(r, surface=surface) is r + assert r.size_line == "" and r.open_line == two_buys, surface + assert seen["args"] == (XAUUSD, "15min|BUY|2450.50|2440.50|GENERIC") + assert seen["args"][1] == _slog40._report_row(r, None, "telegram", "15min")["setup_key"], \ + "the logger's own identity for this scan's row" + want = "Size: 0.10 lots risks $100.00 at the stop (your limit $100.00 per trade)" + with _trade_size40(risk_per_trade=100): + for surface in ("telegram", "cli"): + assert _ex40.attach_exposure(r, surface=surface).size_line == want, surface + assert _ex40.attach_exposure(r, surface="dashboard").size_line == "", "private by default" + assert _ex40.attach_exposure(r, surface="somewhere-new").size_line == "", "an unknown surface is never the owner's" + assert not _ex40.size_shared("telegram_channel") and not _ex40.size_shared("dashboard") + with _trade_size40(risk_per_trade=100, dashboard=True, telegram_channel=True): + assert _ex40.attach_exposure(r, surface="dashboard").size_line == want + assert _ex40.size_shared("telegram_channel") and _ex40.size_shared("dashboard") + assert not _ex40.size_shared("dashboard"), "nothing configured, nothing shared" + sell = _m27_report(entry=2450.5, sl=2460.5, direction="SELL", tf=Timeframe.H1) + assert _ex40.attach_exposure(sell).open_line == \ + "Already open: 1 sell plan alerted earlier (H1) — taking this one adds to the same position" + none = _m27_report(setup=False) + none.size_line, none.open_line = "stale", "stale" + with _trade_size40(risk_per_trade=100): + _ex40.attach_exposure(none) + assert none.size_line == "" and none.open_line == "" + assert _ex40.open_line([{"direction": "BUY", "timeframe": "5min"}] * 3, "BUY") == \ + "Already open: 3 buy plans alerted earlier (3 on M5) — taking this one adds to the same position" + assert _ex40.open_line([{"direction": "SELL", "timeframe": "1h"}], "BUY") == "" and _ex40.open_line(None, "BUY") == "" + + def boom(*a, **k): + raise RuntimeError("database is locked") + + with patch.object(_slog40, "open_alerted_setups", boom), patch.object(app_config, "trade_size_settings", boom): + _ex40.attach_exposure(r) + assert not _ex40.size_shared("dashboard") + assert r.size_line == "" and r.open_line == "", "a display line never raises" + + +@test("daily trend in words: one wording for every surface; the CLI prints the trend, the size and the open line; the JSON dump carries all four keys") +def _(): + import main as _main40 + r = _m27_report(direction="BUY", tf=Timeframe.M5) + assert r.daily_trend is None and _main40.daily_trend_words(r) == "" + cases = {(1, "BUY"): "Daily trend: rising — this buy goes with it", + (-1, "BUY"): "Daily trend: falling — this buy goes against it", + (1, "SELL"): "Daily trend: rising — this sell goes against it", + (-1, "SELL"): "Daily trend: falling — this sell goes with it"} + for (trend, direction), want in cases.items(): + rr = _m27_report(direction=direction, tf=Timeframe.M5) + rr.daily_trend = trend + assert _main40.daily_trend_words(rr) == want, (trend, direction) + r.daily_trend = 0 + assert _main40.daily_trend_words(r) == "Daily trend: no clear direction" + flat = _m27_report(setup=False) + flat.daily_trend = -1 + assert _main40.daily_trend_words(flat) == "Daily trend: falling" + r.daily_trend = 1 + r.size_line = "Size: 0.05 lots risks $100.00 at the stop (your limit $100.00 per trade)" + r.open_line = "Already open: 1 buy plan alerted earlier (M15) — taking this one adds to the same position" + d = _main40.report_to_dict(r) + assert d["daily_trend"] == 1 and d["daily_trend_words"] == cases[(1, "BUY")] + assert d["size_line"] == r.size_line and d["open_line"] == r.open_line + bare = _main40.report_to_dict(_m27_report()) + assert bare["daily_trend"] is None and bare["daily_trend_words"] == "" + assert bare["size_line"] == "" and bare["open_line"] == "" + buf = _io40.StringIO() + with _ctx40.redirect_stdout(buf): + _main40.print_signal_report(r) + text = buf.getvalue() + for line in (cases[(1, "BUY")], r.size_line, r.open_line): + assert line in text, line + buf = _io40.StringIO() + with _ctx40.redirect_stdout(buf): + _main40.print_signal_report(_m27_report()) + quiet = buf.getvalue() + assert "Daily trend" not in quiet and "Size:" not in quiet and "Already open" not in quiet + assert 'attach_exposure(report, surface="cli")' in Path("main.py").read_text(encoding="utf-8") + + +@test("telegram alert: daily-trend line; the size line is private (only the owner's own chat gets it unless shared); the open line reaches everyone; _broadcast sends every other target the copy without it; _commit_alert is in-memory only") +def _(): + import telegram_bot as tb + r = _m27_report(entry=2450.5, sl=2440.5, direction="BUY", tf=Timeframe.M15) + r.warnings = [] + bare = tb.format_signal_message(r) + assert "🧭" not in bare and "📏" not in bare and "🧱" not in bare, "nothing stamped, nothing printed" + assert _no_clock40(tb.format_signal_message(r, channel=True)) == _no_clock40(bare) + r.daily_trend = 1 + r.size_line = "Size: 0.10 lots risks $100.00 at the stop (your limit $100.00 per trade)" + r.open_line = "Already open: 2 buy plans alerted earlier (M5, M15) — taking this one adds to the same position" + owner = tb.format_signal_message(r) + channel = tb.format_signal_message(r, channel=True) + for message in (owner, channel): + assert "🧭 Daily trend: rising — this buy goes with it" in message + assert f"🧱 {r.open_line}" in message + assert f"📏 {r.size_line}" in owner + assert "📏" not in channel and "Size:" not in channel and "your limit" not in channel, \ + "the owner's risk per trade never reaches the channel by default" + lines = owner.splitlines() + sl = next(i for i, line in enumerate(lines) if line.startswith("🛑 SL")) + assert lines[sl + 1].startswith("📏 ") and lines[sl + 2].startswith("🧱 "), lines[sl:sl + 3] + regime = next(i for i, line in enumerate(lines) if line.startswith("📊 Regime")) + assert lines[regime + 1].startswith("🧭 ") + assert _no_clock40(owner).replace(f"📏 {r.size_line}\n", "") == _no_clock40(channel), \ + "the two copies differ by the size line only" + with _trade_size40(risk_per_trade=100, telegram_channel=True): + assert _no_clock40(tb.format_signal_message(r, channel=True)) == _no_clock40(owner) + # a setup-less scan prints the trend and nothing else + flat = _m27_report(setup=False, quality="D") + flat.daily_trend = -1 + flat.size_line, flat.open_line = "Size: must not print", "Already open: must not print" + text = tb.format_signal_message(flat) + assert "🧭 Daily trend: falling" in text and "must not print" not in text + # delivery: the owner's own chat gets the owner copy; the channel AND any other + # target (a group the owner ran /autoscan in: the owner lock is per user) get + # the copy without the size line + sent = [] + + async def fake_send(bot, chat_id, text): + sent.append((chat_id, text)) + return True + + with patch.object(tb, "_send_alert", fake_send), \ + patch.object(tb, "_signal_targets", lambda: [111, -555, -100222]), \ + patch.object(tb, "OWNER_ID", 111), \ + patch.object(tb, "CHANNEL_ID", -100222): + assert _aio40.run(tb._broadcast(None, "owner text", "channel text")) == 3 + assert sent == [(111, "owner text"), (-555, "channel text"), (-100222, "channel text")], sent + sent.clear() + _aio40.run(tb._broadcast(None, "same for all")) + assert sent == [(111, "same for all"), (-555, "same for all"), (-100222, "same for all")] + assert tb._is_owner_chat(111) and not tb._is_owner_chat(-555) and not tb._is_owner_chat(None) + with patch.object(tb, "OWNER_ID", None): + assert not tb._is_owner_chat(111), "no owner configured: nobody is the owner" + src = Path("telegram_bot.py").read_text(encoding="utf-8") + assert src.count("format_signal_message(report, channel=True)") == 1, \ + "one delivery path (_deliver_alert) builds the two copies" + for fn in ("_scan_timeframe", "_handle_breach"): + body = src.split(f"async def {fn}(")[1].split("\nasync def ")[0].split("\ndef ")[0] + assert "_deliver_alert(" in body and "_broadcast(" not in body and "_commit_alert(" not in body, fn + assert 'return attach_exposure(report, surface="telegram")' in src, "every bot scan stamps the lines" + # the commit is the in-memory dedup state and nothing else: the signal-log + # stamp belongs to _deliver_alert, off the event loop + import inspect as _inspect40 + assert "mark_alerted" not in _inspect40.getsource(tb._commit_alert) + marked = [] + names = ("_last_alert_fp", "_last_alert_ts", "_last_alert_quality", "_last_alert_setup", "_last_alert_bar") + saved = [dict(getattr(tb, n)) for n in names] + try: + with patch.object(tb, "mark_alerted", marked.append): + tb._commit_alert("XAU/USD_test40", r) + assert marked == [], "no database write on the event loop" + assert tb._last_alert_setup["XAU/USD_test40"] == (2450.5, 10.0) + finally: + for n, val in zip(names, saved): + getattr(tb, n).clear() + getattr(tb, n).update(val) + + +class _Bot40: + """send_message stub that can fail for chosen chats.""" + + def __init__(self, fail_for=()): + self.fail_for = set(fail_for) + self.sent = [] + + async def send_message(self, chat_id, text, parse_mode=None, **kw): + if chat_id in self.fail_for: + raise RuntimeError("telegram is down") + self.sent.append((chat_id, text)) + + +@_ctx40.contextmanager +def _alert_env40(targets=(111,), owner=111, channel=None): + """A private signals.db plus the bot's alert state saved and restored; + yields (telegram_bot, path).""" + import telegram_bot as tb + fd, path = tempfile.mkstemp(suffix=".db") + os.close(fd) + os.remove(path) + names = ("_last_alert_fp", "_last_alert_ts", "_last_alert_quality", "_last_alert_setup", "_last_alert_bar") + saved = [dict(getattr(tb, n)) for n in names] + saved_env = {k: os.environ.get(k) for k in ("SIGNAL_LOG_PATH", "SIGNAL_LOG_ENABLED")} + os.environ["SIGNAL_LOG_PATH"] = path + os.environ.pop("SIGNAL_LOG_ENABLED", None) + _slog40.reset_loggers_for_tests() + logging.disable(logging.CRITICAL) # undelivered sends log warnings by design + try: + with patch.object(tb, "_signal_targets", lambda: list(targets)), \ + patch.object(tb, "OWNER_ID", owner), \ + patch.object(tb, "CHANNEL_ID", channel), \ + patch.object(tb, "attach_panel", lambda report: False): + yield tb, path + finally: + logging.disable(logging.NOTSET) + for n, val in zip(names, saved): + getattr(tb, n).clear() + getattr(tb, n).update(val) + for k, v in saved_env.items(): + if v is None: + os.environ.pop(k, None) + else: + os.environ[k] = v + _slog40.reset_loggers_for_tests() + _rm_db40(path) + + +def _logged40(entry, sl, direction, tf): + """A report with a setup, logged the way a bot scan logs it (rowid + PENDING outcome).""" + r = _m27_report(entry=entry, sl=sl, direction=direction, tf=tf) + r.warnings = [] + _slog40.log_signal(r, source="telegram", timeframe=tf.value) + assert getattr(r, "_signal_log_rowid", None) is not None + return _ex40.attach_exposure(r, surface="telegram") # what _run_signal_sync does + + +def _stamp40(path, report): + con = _sq40.connect(f"file:{path}?mode=ro", uri=True) + try: + return con.execute("SELECT alerted_at FROM signals WHERE id=?", + (report._signal_log_rowid,)).fetchone()[0] + finally: + con.close() + + +@test("alert delivery: two alerts sent at the same candle close count each other — the row is stamped " + "before the send, on the one-worker alert-db pool, never on the event loop (2026-10-01)") +def _(): + threads = [] + with _alert_env40() as (tb, path): + m5 = _logged40(2450.5, 2445.0, "BUY", Timeframe.M5) + m15 = _logged40(2452.0, 2440.0, "BUY", Timeframe.M15) + assert m5.open_line == "" and m15.open_line == "", "at scan time nothing has been alerted yet" + real_mark, real_exposure = tb.mark_alerted, tb.attach_exposure + + def mark(report): + threads.append(("mark", threading.current_thread().name)) + return real_mark(report) + + def exposure(report, surface="telegram"): + threads.append(("exposure", threading.current_thread().name)) + return real_exposure(report, surface=surface) + + bot = _Bot40() + + async def main(): + loop_thread = threading.current_thread().name + got = await _aio40.gather( + tb._deliver_alert(bot, "XAU/USD_5min", m5, "5min"), + tb._deliver_alert(bot, "XAU/USD_15min", m15, "15min")) + return loop_thread, got + + with patch.object(tb, "mark_alerted", mark), patch.object(tb, "attach_exposure", exposure): + loop_thread, got = _aio40.run(main()) + assert got == [1, 1], got + assert _stamp40(path, m5) and _stamp40(path, m15), "both rows are stamped" + assert len(bot.sent) == 2 + lines = [m5.open_line, m15.open_line] + assert sorted(bool(x) for x in lines) == [False, True], lines + # whichever reached the worker second names the other one's timeframe + second, where = (m15, "M5") if m15.open_line else (m5, "M15") + assert second.open_line == (f"Already open: 1 buy plan alerted earlier ({where}) — " + f"taking this one adds to the same position"), second.open_line + with_line = [text for _chat, text in bot.sent if "🧱" in text] + assert len(with_line) == 1 and f"🧱 {second.open_line}" in with_line[0] + # read and write of one alert run back to back on the single worker, off the loop + assert [kind for kind, _ in threads] == ["exposure", "mark", "exposure", "mark"], threads + assert all(name.startswith("alert-db") and name != loop_thread for _kind, name in threads), threads + assert tb._alert_db_executor._max_workers == 1 + # in-memory dedup state was committed for both keys + assert "XAU/USD_5min" in tb._last_alert_fp and "XAU/USD_15min" in tb._last_alert_fp + # a later alert counts both + h1 = _logged40(2456.0, 2441.0, "BUY", Timeframe.H1) + assert h1.open_line.startswith("Already open: 2 buy plans alerted earlier (M5, M15)"), h1.open_line + + +@test("alert delivery: an alert that reaches no chat leaves no stamp and commits nothing; a partial delivery " + "keeps both; an earlier alert's stamp is never cleared; a formatting error undoes the stamp") +def _(): + with _alert_env40(targets=(111, -100222), channel=-100222) as (tb, path): + key = "XAU/USD_15min" + # 1. every target fails + r = _logged40(2452.0, 2440.0, "BUY", Timeframe.M15) + undone = [] + real_unmark = tb.unmark_alerted + + def unmark(report, stamp): + undone.append((threading.current_thread().name, stamp)) + return real_unmark(report, stamp) + + with patch.object(tb, "unmark_alerted", unmark): + sent = _aio40.run(tb._deliver_alert(_Bot40(fail_for=(111, -100222)), key, r, "15min")) + assert sent == 0 and _stamp40(path, r) is None, "no chat received it: the row must not count" + assert key not in tb._last_alert_fp, "and the setup stays eligible for the next scan" + assert len(undone) == 1 and undone[0][0].startswith("alert-db") and undone[0][1], undone + assert _slog40.open_alerted_setups(XAUUSD) == [] + # 2. the owner's chat fails, the channel gets it: delivered + bot = _Bot40(fail_for=(111,)) + assert _aio40.run(tb._deliver_alert(bot, key, r, "15min", header="HEAD\n", kind="breach alert")) == 1 + first = _stamp40(path, r) + assert first and key in tb._last_alert_fp + assert [chat for chat, _ in bot.sent] == [-100222] and bot.sent[0][1].startswith("HEAD\n") + # 3. the same row alerted again (already stamped) and nobody receives it: + # the first alert's stamp stays + assert _aio40.run(tb._deliver_alert(_Bot40(fail_for=(111, -100222)), key, r, "15min")) == 0 + assert _stamp40(path, r) == first, "only the stamp this attempt wrote may be undone" + # 4. a formatting error: the stamp is undone and the error surfaces + r2 = _logged40(2460.0, 2452.0, "BUY", Timeframe.M15) + + def boom(report, channel=False): + raise ValueError("cannot format") + + with patch.object(tb, "format_signal_message", boom): + try: + _aio40.run(tb._deliver_alert(_Bot40(), key, r2, "15min")) + raise AssertionError("the formatting error must surface") + except ValueError: + pass + assert _stamp40(path, r2) is None + # 5. a report that was never logged still goes out (no row, no stamp, no error) + loose = _m27_report(entry=2470.0, sl=2462.0, direction="BUY", tf=Timeframe.M15) + loose.warnings = [] + assert _aio40.run(tb._deliver_alert(_Bot40(), key, loose, "15min")) == 2 + + +@test("alert delivery: cancelled while its row is being stamped (a shutdown at the wrong moment), the alert " + "leaves no stamp — nothing was sent; cancelled in the middle of the send, the stamp stays") +def _(): + with _alert_env40() as (tb, path): + key = "XAU/USD_15min" + real_mark = tb.mark_alerted + marks = [] + + def slow_mark(report): + time.sleep(0.3) # waiting on the logger's lock + marks.append(report) + return real_mark(report) + + def idle(): # the one worker has drained its queue + tb._alert_db_executor.submit(lambda: None).result(timeout=10) + + async def cancel_after(coro, delay): + task = _aio40.ensure_future(coro) + await _aio40.sleep(delay) + task.cancel() + try: + await task + except _aio40.CancelledError: + return "cancelled" + return "finished" + + # 1. cancelled while the stamp job RUNS: the job cannot be stopped, its stamp is undone + r = _logged40(2452.0, 2440.0, "BUY", Timeframe.M15) + bot = _Bot40() + with patch.object(tb, "mark_alerted", slow_mark): + assert _aio40.run(cancel_after(tb._deliver_alert(bot, key, r, "15min"), 0.1)) == "cancelled" + idle() + assert marks == [r], "the job did run to its end" + assert _stamp40(path, r) is None, "nothing was sent: the row must not count as alerted" + assert bot.sent == [] and key not in tb._last_alert_fp + assert _slog40.open_alerted_setups(XAUUSD) == [] and _slog40.last_alerted_bars() == {} + + # 2. cancelled while the job is still QUEUED behind another alert's: it never runs + del marks[:] + r2 = _logged40(2460.0, 2452.0, "BUY", Timeframe.M15) + blocker = tb._alert_db_executor.submit(time.sleep, 0.3) + with patch.object(tb, "mark_alerted", slow_mark): + assert _aio40.run(cancel_after(tb._deliver_alert(bot, key, r2, "15min"), 0.1)) == "cancelled" + blocker.result(timeout=10) + idle() + assert marks == [] and _stamp40(path, r2) is None and bot.sent == [] + + # 3. cancelled in the middle of the SEND: a chat may already hold the message, + # so the stamp stays (the documented choice); nothing is committed in memory + class _Hanging: + def __init__(self): + self.started = 0 + + async def send_message(self, chat_id, text, parse_mode=None, **kw): + self.started += 1 + await _aio40.sleep(30) + + r3 = _logged40(2470.0, 2462.0, "BUY", Timeframe.M15) + hanging = _Hanging() + assert _aio40.run(cancel_after(tb._deliver_alert(hanging, key, r3, "15min"), 0.5)) == "cancelled" + idle() + assert hanging.started == 1 and _stamp40(path, r3), "mid-send: the stamp is kept" + assert key not in tb._last_alert_fp + # the delivery that is NOT cancelled is unchanged + r4 = _logged40(2480.0, 2472.0, "BUY", Timeframe.M15) + assert _aio40.run(tb._deliver_alert(_Bot40(), key, r4, "15min")) == 1 and _stamp40(path, r4) + + +@test("signal_log: mark_alerted says whether it wrote the stamp; unmark_alerted clears only that stamp; " + "last_alerted_bars reads the newest alerted bar per timeframe, read-only, never raising") +def _(): + fd, path = tempfile.mkstemp(suffix=".db") + os.close(fd) + os.remove(path) + try: + assert _slog40.last_alerted_bars(path) == {}, "no database yet" + lg = _slog40.SignalLogger(path) + + def log(entry, sl, tf, bar): + return lg.log(_m27_report(entry=entry, sl=sl, direction="BUY", tf=tf, ts=bar), + source="telegram", timeframe=tf.value) + + a = log(2450.5, 2445.0, Timeframe.M15, "2026-08-19 12:00:00+00:00") + b = log(2452.0, 2440.0, Timeframe.M15, "2026-08-19 12:15:00+00:00") + c = log(2449.0, 2441.0, Timeframe.M5, "2026-08-19 12:20:00+00:00") + d = log(2455.0, 2447.0, Timeframe.H1, "2026-08-19 12:00:00+00:00") # scanned, never alerted + assert _slog40.last_alerted_bars(path) == {}, "nothing alerted yet" + stamp = lg.mark_alerted(a) + assert stamp and datetime.fromisoformat(stamp).tzinfo is not None + assert lg.mark_alerted(a) is None, "already stamped: this call wrote nothing" + assert lg.mark_alerted(10 ** 9) is None, "no such row" + assert lg.mark_alerted(b, "2026-08-19T12:30:10+00:00") == "2026-08-19T12:30:10+00:00" + assert lg.mark_alerted(c) + assert _slog40.last_alerted_bars(path) == { + ("XAU/USD", "15min"): "2026-08-19 12:15:00+00:00", # the NEWEST alerted M15 row + ("XAU/USD", "5min"): "2026-08-19 12:20:00+00:00"} + # undo: a foreign stamp changes nothing, the right one clears the row + assert lg.unmark_alerted(b, "2000-01-01T00:00:00+00:00") is False + assert lg.unmark_alerted(b, "2026-08-19T12:30:10+00:00") is True + assert lg.unmark_alerted(b, "2026-08-19T12:30:10+00:00") is False, "nothing left to clear" + assert _slog40.last_alerted_bars(path)[("XAU/USD", "15min")] == "2026-08-19 12:00:00+00:00" + assert lg.mark_alerted(b), "a cleared row can be stamped again" + lg.close() + # module-level wrappers never raise + assert _slog40.unmark_alerted(object(), None) is False + assert _slog40.unmark_alerted(object(), "x") is False and _slog40.mark_alerted(object()) is None + body = Path("signal_log.py").read_text(encoding="utf-8").split("def last_alerted_bars")[1] \ + .split("\ndef ")[0] + assert '?mode=ro", uri=True' in body and "immutable" not in body + assert "UPDATE" not in body and "INSERT" not in body + # an older database without the column: {} (the live box before its first writer open) + legacy = path + ".legacy" + _legacy_signals_db39(legacy, without=("alerted_at",)) + assert _slog40.last_alerted_bars(legacy) == {} + finally: + _slog40.reset_loggers_for_tests() + _rm_db40(path) + _rm_db40(path + ".legacy") + + +@test("alert cooldown after a restart: the bot reads the newest alerted bar per timeframe back from the " + "database it wrote and seeds the cooldown before the feed check, off the event loop") +def _(): + with _alert_env40() as (tb, path): + tb._last_alert_bar.clear() + early = _m27_report(tf=Timeframe.M15, ts="2026-09-15 08:00:00+00:00") + late = _m27_report(entry=2460.0, sl=2452.0, tf=Timeframe.M15, ts="2026-09-15 09:30:00+00:00") + unsent = _m27_report(entry=2470.0, sl=2462.0, tf=Timeframe.M15, ts="2026-09-15 09:45:00+00:00") + for r in (early, late, unsent): + r.warnings = [] + _slog40.log_signal(r, source="telegram", timeframe="15min") + bot = _Bot40() + assert _aio40.run(tb._deliver_alert(bot, "XAU/USD_15min", early, "15min")) == 1 + assert _aio40.run(tb._deliver_alert(bot, "XAU/USD_15min", late, "15min")) == 1 + # ... the process restarts: memory is gone, the database is not + for name in ("_last_alert_fp", "_last_alert_ts", "_last_alert_quality", "_last_alert_setup", + "_last_alert_bar"): + getattr(tb, name).clear() + _slog40.reset_loggers_for_tests() + bars = _slog40.last_alerted_bars() + assert bars == {("XAU/USD", "15min"): "2026-09-15 09:30:00+00:00"}, bars + assert tb._restore_alert_cooldowns(bars) == 1 + assert tb._last_alert_bar == {"XAU/USD_15min": pd.Timestamp("2026-09-15 09:30", tz="UTC")} + assert not tb._should_alert("XAU/USD_15min", unsent), "09:45 is one bar after the last alert" + src = Path("telegram_bot.py").read_text(encoding="utf-8") + body = src.split("async def _post_init(")[1].split("\nasync def ")[0] + assert "_alert_db_executor, last_alerted_bars)" in body + assert body.index("_restore_alert_cooldowns(bars)") < body.index("if not _feed_active():") + assert "_alert_db_executor.shutdown(wait=False)" in src + + +@test("telegram scan button: the reply carries the lot-size line only in the owner's own chat — " + "pressed inside a group it prints the copy without it") +def _(): + import telegram_bot as tb + r = _m27_report(entry=2450.5, sl=2440.5, direction="BUY", tf=Timeframe.M15) + r.warnings = [] + r.size_line = "Size: 0.10 lots risks $100.00 at the stop (your limit $100.00 per trade)" + r.open_line = "Already open: 1 buy plan alerted earlier (M5) — taking this one adds to the same position" + button = next(b for b in tb.SCAN_BUTTONS if b["tf"] is not None) + edits = [] + + async def fake_run(symbol, timeframe, confirmation_tfs=None): + return r + + async def fake_edit(query, text, reply_markup=None): + edits.append(text) + + class _Query: + data = button["cb"] + + async def answer(self, *a, **k): + return None + + def update_for(chat_id): + return type("U", (), { + "callback_query": _Query(), "message": None, + "effective_user": type("X", (), {"id": 111})(), + "effective_chat": type("C", (), {"id": chat_id})()})() + + with patch.object(tb, "OWNER_ID", 111), patch.object(tb, "run_signal", fake_run), \ + patch.object(tb, "_safe_edit", fake_edit): + _aio40.run(tb.cb_scan(update_for(111), None)) # the owner's private chat + _aio40.run(tb.cb_scan(update_for(-555), None)) # the owner, inside a group + owner_text, group_text = edits[1], edits[3] # edits[0], [2] are the "Scanning…" notes + assert "📏 Size: 0.10 lots" in owner_text + assert "📏" not in group_text and "your limit" not in group_text, "a group must not see the owner's risk" + assert "🧱 Already open: 1 buy plan" in owner_text and "🧱 Already open: 1 buy plan" in group_text + with _trade_size40(risk_per_trade=100, telegram_channel=True): + del edits[:] + with patch.object(tb, "OWNER_ID", 111), patch.object(tb, "run_signal", fake_run), \ + patch.object(tb, "_safe_edit", fake_edit): + _aio40.run(tb.cb_scan(update_for(-555), None)) + assert "📏 Size: 0.10 lots" in edits[1], "trade_size.telegram_channel shares it with every chat" + + +def _live_scan40(tf, confirms, raise_for=None, weekend=False): + """One analyze_pair through a stub provider; returns (report, fetched timeframes).""" + from pair_signals import PAIR_PRODUCERS + + class _QuietNews(_NoNews): + def get_event_risk_level(self, symbol, context): + return "clear", "" + + def get_pair_sentiment(self, symbol, context): + return 0.0, "" + + cfg = PAIR_PRODUCERS[XAUUSD](tf, confirms) + cfg.xau_strategy.allowed_weekdays = [] # deterministic on any day + agg = SignalAggregator(cfg) + agg._get_news_engine = lambda: _QuietNews() + fetched = [] + + class _Stub: + def get_pair_info(self, symbol): + return PAIR_INFO[XAUUSD] + + def fetch_ohlcv(self, symbol, timeframe, lookback_days=None): + fetched.append(timeframe) + if timeframe is raise_for: + raise RuntimeError("boom") + if timeframe is Timeframe.D1: + return _d1_39(end=str(pd.Timestamp.now(tz="UTC").date() - pd.Timedelta(days=1)), slope=-1.5) + return make_xau_ohlcv(300, seed=42) + + def is_data_fresh(self, df, symbol, timeframe): + return True + + agg.data_provider = _Stub() + os.environ["SIGNAL_LOG_ENABLED"] = "0" + try: + with patch("signal_aggregator._is_forex_weekend", return_value=weekend): + report = agg.analyze_pair(symbol=XAUUSD, log_source="telegram") + finally: + os.environ.pop("SIGNAL_LOG_ENABLED", None) + return report, fetched + + +@test("live wrapper: where the veto is off the daily trend is stamped for display — from the scan's own D1 frame on H4 (no extra fetch), from one extra fetch on M5 and H1; a failed display fetch or a blocked scan leaves it empty and never fails the scan") +def _(): + report, fetched = _live_scan40(Timeframe.H4, [Timeframe.H1, Timeframe.D1]) + assert fetched == [Timeframe.H4, Timeframe.H1, Timeframe.D1], fetched + assert report.daily_trend == -1 and report.primary_analysis is not None + for tf, confirms in ((Timeframe.M5, [Timeframe.M15, Timeframe.H1]), (Timeframe.H1, [Timeframe.M15, Timeframe.H4])): + report, fetched = _live_scan40(tf, confirms) + assert fetched == [tf] + confirms + [Timeframe.D1], (tf, fetched) + assert report.daily_trend == -1, tf + assert Timeframe.D1 not in report._frames, "the display frame never joins the scan's frames" + assert not any("DAILY TREND" in w for w in report.warnings), "no veto on this timeframe" + report, fetched = _live_scan40(tf, confirms, raise_for=Timeframe.D1) + assert report.daily_trend is None and report.primary_analysis is not None, tf + assert not any("DATA UNAVAILABLE" in w for w in report.warnings), report.warnings + report, fetched = _live_scan40(Timeframe.M5, [Timeframe.M15, Timeframe.H1], weekend=True) + assert fetched == [] and report.daily_trend is None, "a blocked scan fetches nothing" + # display only: the replayable core never stamps the trend where the veto is off, + # and never touches the size / open lines anywhere + frames = dict(_bc_actionable_dfs()) + frames[Timeframe.D1] = _d1_39(slope=-1.5) + core = _trend_agg39("off").analyze_from_dataframes(XAUUSD, frames, use_news=False) + assert core.daily_trend is None and core.size_line == "" and core.open_line == "" + src = Path("signal_aggregator.py").read_text(encoding="utf-8") + assert ".size_line" not in src and ".open_line" not in src, "no core code reads or writes the display lines" + body = src.split("def analyze_from_dataframes")[1].split("\n def ")[0] + assert "_stamp_daily_trend_for_display" not in body, "the display stamp lives in the live wrapper only" + + +@test("dashboard: the snapshot carries the daily trend and the open line, the size line only when trade_size.dashboard is set; the page prints the lines as written; the publisher attaches exposure") +def _(): + import dashboard_publisher as dp + r = _m27_report(entry=2450.5, sl=2440.5, direction="BUY", tf=Timeframe.M15) + r.daily_trend = -1 + rows = [{"direction": "BUY", "timeframe": "15min"}] + with patch.object(_slog40, "open_alerted_setups", lambda *a, **k: rows): + with _trade_size40(risk_per_trade=100): + d = dp.report_to_webapp_dict(_ex40.attach_exposure(r, surface="dashboard")) + assert d["size_line"] == "", "the public page never shows the owner's risk by default" + assert d["open_line"] == "Already open: 1 buy plan alerted earlier (M15) — taking this one adds to the same position" + assert d["daily_trend"] == -1 + assert d["daily_trend_words"] == "Daily trend: falling — this buy goes against it" + with _trade_size40(risk_per_trade=100, dashboard=True): + d = dp.report_to_webapp_dict(_ex40.attach_exposure(r, surface="dashboard")) + assert d["size_line"] == "Size: 0.10 lots risks $100.00 at the stop (your limit $100.00 per trade)" + assert 'attach_exposure(report, surface="dashboard")' in Path("dashboard_publisher.py").read_text(encoding="utf-8") + js = Path("static/dashboard.js").read_text(encoding="utf-8") + for needle in ("function planHTML(ts, r)", "r.size_line", "r.open_line", "r.daily_trend_words", + 'data-f="trend"', 'tile("Daily trend"'): + assert needle in js, needle + assert js.count("planHTML(r.trade_setup, r)") == 2 and "planHTML(r.trade_setup)" not in js + + +@test("exposure: a display-only leaf — it imports only app_config and signal_log, and neither the signal core nor a backtester imports it") +def _(): + tree = ast.parse(Path("exposure.py").read_text(encoding="utf-8")) + imported = set() + for node in ast.walk(tree): + if isinstance(node, ast.Import): + imported |= {a.name.split(".")[0] for a in node.names} + elif isinstance(node, ast.ImportFrom): + imported.add((node.module or "").split(".")[0]) + assert imported == {"logging", "decimal", "typing", "app_config", "signal_log"}, imported + for path in ("signal_aggregator.py", "engine.py", "chart_patterns.py", "position_sizing.py", "signal_log.py", + "app_config.py", "pair_signals/xauusd.py", "backtest/gpu_backtest.py", + "backtest/cudaBasedBacktesting.py", "backtest/cuda_signal_kernels.py", "backtest/replay_record.py"): + src = Path(path).read_text(encoding="utf-8") + assert "import exposure" not in src and "from exposure" not in src, path + for surface in ("telegram_bot.py", "main.py", "dashboard_publisher.py"): + assert "from exposure import attach_exposure" in Path(surface).read_text(encoding="utf-8"), surface + # ══════════════════════════════════════════════════════════════════════ # SUMMARY