harry/ForexBot2
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

Daily-trend veto on M15 + lot-size / already-open lines (campaign 2026-09-30), and the 2026-10-01 whole-codebase bug review

Browse Source 
Two bodies of work that touch the same files, so one commit.

Direction and setup-size campaign (2026-09-30 → 10-01, owner's RISK-FIRST rule,
written before the runs):
- SIGNAL: M15 daily-trend veto (factory override 7, xau_strategy.daily_trend_veto
  "both"). A finished setup against a clear daily trend (closed D1 bars, Sunday
  stubs dropped; close vs its 50-bar EMA AND vs the close 60 kept days earlier)
  is removed as the last step of the core. One definition
  (signal_aggregator.daily_trend_series) for live, the bar replay and the twin's
  host side; D1 is a required frame of the M15 scan; mirrored in
  cuda_signal_kernels.py K2. W3 824 / +205.12R / PF 1.73 / max DD 12.4R vs
  1105 / +195.79R / 1.49 / 20.9R; 2019 → 2026 1473 / +223.33R / 1.42 / 41.5R vs
  2091 / +192.53R / 1.24 / 64.8R; unseen holdout −4.62R vs −6.63R.
- FAILED, not shipped: the veto on M5 (W3 drawdown 10.1R → 15.2R) and on H1
  (−9.7R), buy-only and sell-only on M15, the wide-stop cap
  (xau_strategy.wide_stop_cap_atr stays 0.0, a research knob).
- New M15 pins: W2 dfc647ee…, OOS e87b6de1…, W3 ad3b0dbd…. M5 / H1 / H4 pins
  unchanged. --set xau_strategy.daily_trend_veto=off reproduces the previous M15
  pins byte for byte. Twin and reference script MD5-equal on every window;
  recorder and feature-export parity re-proved; base-rate table rebuilt.
- DISPLAY (books byte-identical): exposure.py — the lot size for
  trade_size.risk_per_trade (new config.json block, off by default, private to
  the owner) and the count of alerted same-direction plans still open
  (signals.alerted_at, read-only open_alerted_setups); the daily trend in words
  on Telegram, the CLI and the dashboard; signals.daily_trend logged.
- providers/oanda: a date window with no bars is an empty page, not an error
  (a D1 pad starting on a Saturday could not be fetched).
- Reports only, nothing shipped: seven context screens (no state nominated) and
  eight outside data sources (none adds a forecast to gold's own daily trend).
- Independent verification (report section 5.13): every figure reproduced on
  both scripts, scoring identical to the previous commit on all four
  timeframes, no look-ahead. Caveats recorded there: this is a risk filter, not
  a more accurate direction call; nine tenths of the 2019 → 2026 gain is 2022;
  the neighbour clause is thin (a simple 100-day average misses the R line by
  1.2R); "never adds a trade" is an observation, not a guarantee.

Bug review of the whole codebase (2026-10-01). Signal path clean. 17 defects
elsewhere, plus 4 found by an independent review of the fixes; all fixed, each
with a test; no trade book changed:
- telegram_bot: one _deliver_alert for the candle scan and the level break. The
  row is stamped alerted_at BEFORE the send, on a one-worker alert-db pool, and
  un-stamped when no chat received it or the task is cancelled mid-stamp, so
  two alerts at one candle close count each other and no SQLite write runs on
  the event loop. The 5-bar cooldown counts bars of the scan's own frame
  (== gpu_backtest._apply_cooldown; clock time released the 22:00 UTC bar after
  the settlement pause) and is restored from signals.db at start-up. The
  lot-size line goes to the owner's own chat only (a group the owner ran
  /autoscan in used to receive it). /status gains a news-model line and escapes
  failure details.
- signal_log: mark_alerted returns the stamp it wrote; unmark_alerted;
  last_alerted_bars (read-only).
- advisory_outcome: candle continuity is judged bar by bar inside the walk; a
  hole after a trade's exit no longer voids it. The stored-candle replay
  resolves 108 of 111 logged setups instead of 99, all equal to the live
  resolver's R.
- providers/oanda: an empty date window is accepted only when it is 5 days or
  shorter and no malformed candle was skipped; a longer one is retried, then
  refused, instead of being cached as history with a hole.
- news_engine: blank RSS titles skipped; one unreadable headline no longer
  costs the local model its whole batch (checked on the real bundle); model
  health recorded; no DeepSeek audit call in a process that does not archive;
  event-direction cache keyed per release; peek_week_events never fetches.
- llm_client: chat_json never raises and never logs the API key (requests
  quotes the Authorization value in InvalidHeader; error bodies are redacted).
- signal_aggregator: the scan's news block is guarded; the replay path
  (use_news=False) assigns exactly the values it did before.
- agent_panel: PANEL_PROMPT_VERSION v2.2 — same prompt text, corrected input
  (event timing over the cached calendar week, headlines newest first).
  scripts/panel_replay.py replays the logged payload per persona;
  scripts/label_scan_outcomes.py counts with an ISO cutoff.
- test_all.py: no ALTER TABLE … DROP COLUMN (the live box has SQLite 3.34);
  tests that write Parquet print a notice and return without pyarrow.

Verification after the last code change: suite 632 passed, 0 failed (also with
both Parquet engines blocked and FB2_BACKEND=cpu, the live box's view); the
five reference MD5s reproduce on both backtesters, 0 cache fallbacks; Module 37
green; static Python 3.9 / SQLite 3.34 check clean on every touched file.

Evidence: backtest/research/daily_trend_campaign_2026-09-30.md,
context_screens_2026-10.md, new_inputs_2026-10.md; CLAUDE.md sections 2, 4, 6,
8, 9, 10 and 11; pair_signals/xauusd.py docstring (M15 override 7).
This commit is contained in:
Harwinder Sekhon
2026-10-01 10:28:28 -04:00
parent 42025da0b0
commit 2cc5915b00
31 changed files with 6021 additions and 286 deletions
Download Patch File Download Diff File Expand all files Collapse all files
CLAUDE.md
+206 -37
View File
@@ -23,7 +23,7 @@ cp config.json.example config.json # then fill i
venv/bin/pip install cupy-cuda12x "nvidia-cuda-runtime-cu12==12.9.*" # optional, dev box: backtester GPU backend
# (CuPy from pip needs that header wheel to compile kernels)
venv/bin/python test_all.py # QA suite, no network; prints its own count
# (581 tests / 38 modules as of 2026-09-27)
# (632 tests / 40 modules as of 2026-10-01)
venv/bin/python main.py --pair XAUUSD --timeframe H1 --detail # one signal report in the terminal
# --json full dump, --panel adds the advisory LLM panel,
# --timeframe M5 | M15 | H1 | H4
@@ -48,7 +48,10 @@ unknown path is a hard exit), `--export-features` (appends the 65 decision-time
it on a parity run because it changes the file), `--fixed-rr-tp N` (research only, not shipped),
`--set signals.grading_mode=min|product` (research only, 2026-09-27: grade on one conviction number
instead of the score + confidence floors; both FAILED rule 1 on M5 and M15, default `pair` is the
shipped rule and byte-identical), `--record DIR [--record-bars]` (research recorder, see section 9), `--backend auto|gpu|cpu` (`auto`
shipped rule and byte-identical), `--set xau_strategy.daily_trend_veto=off|both|buy|sell` (2026-09-30:
the M15 factory ships `both`; pass `off` to reproduce a pre-veto M15 book; `buy` / `sell` and every
mode on M5 / H1 FAILED their rule), `--set xau_strategy.wide_stop_cap_atr=K` (research only: FAILED
on 2026-09-30, ships 0 = off), `--record DIR [--record-bars]` (research recorder, see section 9), `--backend auto|gpu|cpu` (`auto`
uses the CuPy GPU backend when CUDA works and falls back to NumPy with the reason in the `Backend:`
line; `gpu` refuses to fall back; `cpu` forces the NumPy twins for a parity run; the env var
`FB2_BACKEND` sets the default). Both backends produce byte-identical books (section 9).
@@ -75,6 +78,8 @@ same schedule. `main.py` runs once when you call it.
(`app_config.py`), then applies the validated per-timeframe overrides (section 8). Never build a
bare `PlatformConfig()` for a real scan; always go through the factory. The confirmation
timeframes come from the factory map: M5 → M15 + H1, M15 → M5 + H1, H1 → M15 + H4, H4 → H1 + D1.
Since 2026-09-30 an M15 scan also needs the daily (D1) frame, not as a confirmation but as the
input of the daily-trend veto (step 8); it is a required frame like the others.
**3. Fetch candles.** `signal_aggregator.analyze_pair` (the live entry point) first checks the
calendar: on the forex weekend or a blocked weekday it returns a "blocked" scan and does nothing
@@ -146,6 +151,16 @@ M5 six extra noise gates (minimum score, required agreement, confirmation streng
cap, chop-transition volatility, SELL RSI floor). Every gate that removes a setup writes a warning
that names itself.
One more gate runs at the very end, after the setup generators and the late entry paths of step 9
(`_apply_daily_trend_veto`, step 12 in the code; M15 only since 2026-09-30): the **daily-trend
veto**. The daily trend is read from closed D1 bars with the provider's Sunday stub bars dropped
(`signal_aggregator.daily_trend_series`): +1 when the last daily close is above its 50-day EMA and
above the close 60 kept days earlier, −1 when below both, else 0. A finished setup that goes
against a non-zero state is removed (grade D, no setup, warning `DAILY TREND FILTER: BUY against a
falling daily trend — no trade`). It runs after every archetype on purpose, so a chart-pattern
setup is vetoed too. Where the veto is off (M5, H1, H4) the live wrapper still stamps
`report.daily_trend` for the alert text and the signal log; nothing reads it back there.
**9. Trade setup.** Grades A+ and A get the gold ATR setup (`_generate_xau_atr_trade_setup`): stop
2 × ATR; target 1 at 2 × ATR for A+ and 1.5 × ATR for A (1 × ATR for an A carrying two or more
exhaustion tags); target 2 at 1.5 × the target-1 distance. Other tradeable grades get the generic swing-based setup,
@@ -175,7 +190,12 @@ later generator shipped a setup.
reaches `alert_min_quality`, **and** a setup exists. Since 2026-09-26 no second alert goes out on a
timeframe within 5 bars of the last alerted setup's bar (`_ALERT_COOLDOWN_BARS`, the backtester's
entry cooldown; consecutive-bar re-entries were −25.1R of the −37.8R live book of 2026-09-04 →
09-25). After that, a fingerprint (signal + direction) stops repeats; a grade upgrade re-alerts.
09-25). Since 2026-10-01 the five bars are COUNTED in the scan's own primary frame
(`_primary_bars_since`, the backtester's position difference; clock time is only the fallback
for a report without frames) and the last alerted bar per timeframe is read back from
`signals.db` at start-up (`signal_log.last_alerted_bars` → `_restore_alert_cooldowns`), so
neither the settlement pause, the weekend nor a restart lets live alert where the books do not
enter. After that, a fingerprint (signal + direction) stops repeats; a grade upgrade re-alerts.
Since 2026-09-27 the header leads with the grade (`Grade A · BUY · XAU/USD · M15`) and the raw
score / confidence line is replaced by words from `main.plain_words` (`Direction: strong buy ·
Reliability: good · timeframes agree`; the cut-offs are the grade table's own, so the words never
@@ -187,12 +207,29 @@ later generator shipped a setup.
(`_book_rate_line`, from `agent_panel_base_rates.json`). The fingerprint is committed only after a successful send,
so a failed send stays eligible next scan. Every send is logged with the signal-log row id. The
level-breach monitor reuses the bar's cached M15 report (`_cached_or_scan`) instead of rescanning.
- *CLI* (`main.py`): prints the report; `--json` dumps everything.
Since 2026-09-30 three more display lines: the daily trend in words under the regime line
(`🧭 Daily trend: rising — this buy goes with it`, `main.daily_trend_words`), and inside the setup
block the lot size for the owner's risk limit (`📏`, off until `trade_size.risk_per_trade` is set)
and the count of alerted same-direction plans still open (`🧱`). The size line is private:
`format_signal_message(report, channel=True)` builds the copy without it unless
`trade_size.telegram_channel` is true, and since 2026-10-01 `_broadcast` sends the owner copy to
the owner's own chat only (`_is_owner_chat`): the channel and any other target (a group the owner
used `/autoscan` or a scan button in; the owner lock is per user) get the copy without it.
Both alert paths (candle scan, level break) end in one `_deliver_alert`: the panel, then on the
one-worker `alert-db` pool the exposure lines are refreshed and the row is stamped `alerted_at`
(`signal_log.mark_alerted`), then the send. Stamping BEFORE the send is what lets two alerts at
the same candle close count each other; an alert that reaches no chat clears its own stamp
(`unmark_alerted`) and commits nothing. No database write runs on the event loop.
- *CLI* (`main.py`): prints the report; `--json` dumps everything (including `daily_trend`,
`daily_trend_words`, `size_line`, `open_line`).
- *Dashboard* (`dashboard_publisher.py`): writes `snapshot.json`, `news.json` and the static page
into the Apache folder.
into the Apache folder. Each card shows the daily-trend words; a plan shows the already-open
line, and the size line only when `trade_size.dashboard` is true (the page is public).
Two advisory add-ons are attached **after** the scan by the surface, never inside the signal
code: the LLM agent panel (`agent_panel.py`; since 2026-09-17 only the **macro** persona runs by
Three display add-ons are attached **after** the scan by the surface, never inside the signal
code. The first is `exposure.attach_exposure` (2026-09-30): the lot-size line and the
already-open line, a leaf module that imports only `app_config` and `signal_log`. The other two are
advisory: the LLM agent panel (`agent_panel.py`; since 2026-09-17 only the **macro** persona runs by
default, `agent_panel.personas: ["macro"]`, one DeepSeek call per panel — the technical, risk and
contrarian personas carried no information in two live evaluations and stay available for research;
the persona returns a probability that the trade closes profitably, anchored to the setup class's
@@ -207,22 +244,30 @@ replayable core never sets it.
**11. Logging.** `signal_log.py` writes one row per live scan (from the bot and the CLI; the
dashboard process never writes either database, section 3) into `logs/signals.db` (signal,
grade, price, indicators, the last 14 candles per timeframe, the news at scan time, the setup, the
gate that removed a setup, a gate-config fingerprint and an engine fingerprint). The engine
gate that removed a setup, a gate-config fingerprint and an engine fingerprint; since 2026-09-30
also `daily_trend` on every real scan and `alerted_at` on rows whose alert reached a chat). The engine
fingerprint hashes the scoring config, so it changes whenever a weight or threshold ships; signal-log
comparisons across such a deploy must be split at the deploy time (the 2026-09-09 M15 RSI change is
one). `news_archive.py` writes every headline and
one). The gate fingerprint changes when a gate ships: the M15 daily-trend veto moved M15 rows from
`f9b2f801de54` to `8a8075688d67` (engine fingerprint unchanged), so M15 comparisons split there too.
`news_archive.py` writes every headline and
its score into `logs/newsdata.db`. Both are fail-safe: a logging error can never break a scan.
**12. Measurement, later.** Two models read the stored candles and score what happened next:
advisory outcomes (`advisory_outcome.py`: replays each logged setup with next-open entry, stop
first, close-only trailing; a cleaned-mid evidence model, not execution P&L) and scan outcomes
(model-free forward moves 1 h / 4 h / 24 h after every scan). `analyze_signal_log.py` turns the
database into reports.
database into reports. Since 2026-10-01 the outcome model checks candle continuity bar by bar as
it walks: a hole in the stored candles AFTER a trade's exit no longer voids the trade, and a hole
reached while it is open ends the walk with the progress kept (so the locked-profit floor is the
same whether the stream arrives in one pass or many).
**The backtester is the same pipeline on history.** `backtest/gpu_backtest.py` loads candles from
`backtest/cache/*.parquet` (fetched once through the same provider code), precomputes every
indicator series over the whole window, and calls `analyze_from_dataframes` bar by bar with news
off. Trades are resolved on the raw price path with next-open entries, spread charged per trade,
off. When the daily-trend veto is on it also loads the D1 frame (1,400-day pad) and slices it as-of
like every other frame, so the veto sees exactly the closed daily bars a live scan would. Trades
are resolved on the raw price path with next-open entries, spread charged per trade,
stop-first ties, gap fills at the open, and the trailing stop moved at bar close only. Time filters
use bar-close time in both paths, so live and backtest agree. The replay fans out over worker
processes; the 5-bar entry cooldown, the only cross-bar state, is applied in an ordered pass
@@ -238,9 +283,10 @@ Arrows show who imports whom (generated from the code on 2026-09-15).
| File | Role | Uses | Used by |
|---|---|---|---|
| `main.py` | Command-line signal report (`--detail`, `--json`, `--panel`) | pair_signals, signal_aggregator, agent_panel, violent_watch, app_config | dashboard_publisher (formatting helpers) |
| `telegram_bot.py` | Owner-locked bot: candle-close scan loops, alerts, breach monitor, heads-ups, `/scan /status /news /autoscan /cache /clearcache /help /start` | pair_signals, signal_aggregator, providers, agent_panel, violent_watch, news_engine, llm_client, signal_log, app_config | nobody (a top-level process) |
| `dashboard_publisher.py` | Static web publisher daemon; writes JSON + HTML into the Apache folder. Its own process with its own news caches and its own DeepSeek daily cap. Never writes `signals.db` or `newsdata.db`: `main()` calls `app_config.set_db_writes_for_process(False)`, a process-wide switch every SQLite writer honours (owner decision 2026-09-17; the bot is the single writer because the dashboard is not always running) | pair_signals, providers, main, agent_panel, violent_watch, news_engine, app_config | nobody. Hard rule: never imports and is never imported by `telegram_bot` (test-enforced) |
| `main.py` | Command-line signal report (`--detail`, `--json`, `--panel`); the display helpers every surface shares (`plain_words`, `daily_trend_words`) | pair_signals, signal_aggregator, agent_panel, violent_watch, exposure, app_config | dashboard_publisher and telegram_bot (formatting helpers) |
| `telegram_bot.py` | Owner-locked bot: candle-close scan loops, alerts, breach monitor, heads-ups, `/scan /status /news /autoscan /cache /clearcache /help /start` | pair_signals, signal_aggregator, providers, agent_panel, violent_watch, exposure, news_engine, llm_client, signal_log, app_config | nobody (a top-level process) |
| `exposure.py` | Display-only lot-size line and already-open line (`attach_exposure`, 2026-09-30). `lot_size` floors to the lot step in Decimal so the risk limit is never exceeded; `size_shared` answers whether config.json shares the size with a public audience. A leaf: never raises, never read by the core | app_config, signal_log | main, telegram_bot, dashboard_publisher |
| `dashboard_publisher.py` | Static web publisher daemon; writes JSON + HTML into the Apache folder. Its own process with its own news caches and its own DeepSeek daily cap. Never writes `signals.db` or `newsdata.db`: `main()` calls `app_config.set_db_writes_for_process(False)`, a process-wide switch every SQLite writer honours (owner decision 2026-09-17; the bot is the single writer because the dashboard is not always running). It does READ `signals.db` (`mode=ro`) for the already-open line since 2026-09-30 | pair_signals, providers, main, agent_panel, violent_watch, exposure, news_engine, app_config | nobody. Hard rule: never imports and is never imported by `telegram_bot` (test-enforced) |
| `static/dashboard.html`, `.css`, `.js` | The public dashboard page (reads `snapshot.json` / `news.json`). Bootstrap 5 + jQuery + Chart.js + Font Awesome straight from a CDN, no build step; M5/M15/H1/H4 each get a card in one column grid, with a diverging Chart.js bar chart for direction and one per card for the indicator votes | | copied by the publisher |
| `static/news_summary.php` | The one dynamic endpoint: on-demand DeepSeek summary for a clicked headline, whitelist-gated against `news.json`, own fail-closed daily cap, key read from `/etc/gold-dashboard/deepseek.key` (never repo config) | | installed by the publisher |
| `docs/user_guide.html`, `docs/beginner_guide.html` | Human guides. Keep them in sync with signal behaviour. There are no PDF builds | | |
@@ -253,7 +299,7 @@ Arrows show who imports whom (generated from the code on 2026-09-15).
| `pair_signals/xauusd.py` | The config factory: overlay + validated per-timeframe overrides + confirmation map. Its docstring is the evidence log for every override | config, app_config, engine | main, telegram_bot, dashboard_publisher, backtester |
| `pair_signals/__init__.py` | Registry of pair producers (`PAIR_PRODUCERS`) and `produce_signal` | config, signal_aggregator, providers, xauusd | the surfaces, backtester |
| `engine.py` | Indicators, regime, composite score (`TechnicalEngine`) | config | signal_aggregator, backtester, replay_record, pair factory |
| `signal_aggregator.py` | Multi-timeframe merge, penalties, grade, gate chain, setup generators, reasons; `analyze_pair` (live) and `analyze_from_dataframes` (replayable core) | engine, chart_patterns, news_engine, providers, signal_log, config | main, telegram_bot, backtester, pair_signals, analyze_signal_log (agent_panel only for type hints) |
| `signal_aggregator.py` | Multi-timeframe merge, penalties, grade, gate chain, setup generators, reasons, the daily-trend veto and its one definition (`daily_trend_series` / `daily_trend` / `daily_trend_needed`, shared by live, the replay and the twin's host side); `analyze_pair` (live) and `analyze_from_dataframes` (replayable core) | engine, chart_patterns, news_engine, providers, signal_log, config | main, telegram_bot, backtester, pair_signals, analyze_signal_log (agent_panel only for type hints) |
| `chart_patterns.py` | The XAU_CP pattern detector | | signal_aggregator |
| `position_sizing.py` | Decision-time feature builder (65 columns) and the reserved stake multiplier (always 1.0). Bounded lookbacks keep live == backtest | config | backtester (`--export-features`) |
@@ -272,9 +318,9 @@ Arrows show who imports whom (generated from the code on 2026-09-15).
| File | Role |
|---|---|
| `news_engine.py` | Calendar + RSS headline sentiment with gold-specific keywords; class-level caches; headline scores upgraded by the local XAU model (`news_model.py`, mode `primary` live since 2026-09-27) or by DeepSeek (modes `off` / `fallback`, `_enrich_headline_scores`); DeepSeek keeps event direction and the summary; display and warnings only |
| `llm_client.py` | Thin `requests` client for DeepSeek (`deepseek-v4-flash`, `thinking: disabled`, one attempt, UTC-day request cap). Every caller has a non-LLM fallback |
| `llm_client.py` | Thin `requests` client for DeepSeek (`deepseek-v4-flash`, `thinking: disabled`, one attempt, UTC-day request cap). Every caller has a non-LLM fallback. `chat_json` never raises and never logs the API key (2026-10-01: a key with a line break made `requests` quote the `Authorization` value in its error text); an unserialisable payload is refused before the cap is counted |
| `news_archive.py` | Durable headline corpus in `logs/newsdata.db` (one row per distinct headline, upgrade-only scores, provenance of the scorer). Its own database on purpose: a `signals.db` rotation in 2026-09 destroyed a 3-week corpus of 4,054 headlines and `logs/` is gitignored. Hooked at the end of `news_engine._fetch_news_sentiment_raw`, before the ±0.1 neutral-headline filter, so neutral headlines are kept for training; do not move the hook. Since 2026-09-17 the corpus is complete for a gold scorer: `xau_score` holds DeepSeek's gold score for every LLM-scored row (it used to survive only for gold-keyword headlines, so 930 of the first 1,546 rows had no gold label; older rows read NULL), headlines naming no tracked currency are scored and archived too (`currencies_json` `[]`; they are never returned to the platform), and every new headline goes to DeepSeek in the cycle it arrives, 25 per call. **Frozen in `news_model.mode` `primary` (live since 2026-09-27)**: the local model's output is not training data, so nothing is archived except the optional DeepSeek audit copies (`deepseek_audit_per_day`, shipped 0); `off` / `fallback` archive as before, a `fallback` model row carrying `teacher_model` `xau_transformer@<trained_at>` |
| `agent_panel.py` | Advisory persona panel (macro by default; technical / risk / contrarian and the chair available through `agent_panel.personas` for research). Attached post-hoc. v2 (2026-09-17, `PANEL_PROMPT_VERSION`): every persona sees a code-computed `context` block (prior move in ATR, distance from the 24 h extreme, candle size, stop/target in ATR, the class base rate from `agent_panel_base_rates.json`, event timing against the expected hold) and returns `p_win` plus one reason each way; stance is derived symmetrically (≥ 60 support, ≤ 40 oppose); the verdict is the equal-weight mean (quorum 3, `verdict_source: "mean"`, default). `macro` (stance verbatim) and `chair` (synthesizer + `_apply_oppose_guard`) remain for comparison. The prompt version is logged and hashed. Persona weights are not to be tuned before about 300 graded outcomes with a held-out half |
| `agent_panel.py` | Advisory persona panel (macro by default; technical / risk / contrarian and the chair available through `agent_panel.personas` for research). Attached post-hoc. v2 (2026-09-17, `PANEL_PROMPT_VERSION`): every persona sees a code-computed `context` block (prior move in ATR, distance from the 24 h extreme, candle size, stop/target in ATR, the class base rate from `agent_panel_base_rates.json`, event timing against the expected hold) and returns `p_win` plus one reason each way; stance is derived symmetrically (≥ 60 support, ≤ 40 oppose); the verdict is the equal-weight mean (quorum 3, `verdict_source: "mean"`, default). `macro` (stance verbatim) and `chair` (synthesizer + `_apply_oppose_guard`) remain for comparison. The prompt version is logged and hashed. v2.2 (2026-10-01) changed the persona's INPUT, not the prompt text: the event timing reads the cached calendar week `attach_panel` hands to `_price_context` (`NewsEngine.peek_week_events`, which never fetches; a hold past UTC midnight sees tomorrow's release), and the ten headlines are the newest ten, not the first ten in feed order. Persona weights are not to be tuned before about 300 graded outcomes with a held-out half |
| `violent_watch.py` | Display-only violent-move warning and heads-up. Attached post-hoc |
| `news_model.py` | The local headline scorer (2026-09-27): vendored CPU runtime of the owner's ML-NEWS XAU transformer (DeBERTa-v3-small, two heads: moves gold? / up or down?, ONNX; `onnxruntime` + `tokenizers` imported lazily inside `load_model`, so the bot starts without them) plus a load-once, never-raising singleton (`get_news_model`, `teacher_tag`, `warm_up_in_background`). Gold head only: `news_engine._apply_model_sentiment` writes `xau_score` and XAU's per-currency entry, never USD. Bundle (280 MB) lives outside the tree at `news_model.path`. Spec and deviations: `docs/news_model_integration_spec_2026-09-27.md`; ops: `deploy/README-news-model.md` |
@@ -282,14 +328,14 @@ Arrows show who imports whom (generated from the code on 2026-09-15).
| File | Role |
|---|---|
| `signal_log.py` | SQLite logger for every live scan (`signals`, `news_snapshots`, `scan_outcomes`; view `signals_with_news`); opportunistic scan-outcome labelling; advisory outcome seeding |
| `advisory_outcome.py` | The deterministic outcome model for logged setups |
| `signal_log.py` | SQLite logger for every live scan (`signals`, `news_snapshots`, `scan_outcomes`; view `signals_with_news`); opportunistic scan-outcome labelling; advisory outcome seeding; `mark_alerted` (stamps `alerted_at` just before the alert is sent and returns the stamp it wrote), `unmark_alerted` (clears that stamp when no chat received the alert), the read-only `open_alerted_setups` (alerted plans whose outcome is still `PENDING`, one per event, stale rows dropped) and the read-only `last_alerted_bars` (newest alerted bar per timeframe, for the bot's cooldown after a restart) |
| `advisory_outcome.py` | The deterministic outcome model for logged setups. Continuity of the stored candles is judged one step at a time inside the walk (`_next_bar_follows`); `_has_continuity` stays all-or-nothing for the scan-outcome labeller |
| `analyze_signal_log.py` | Read-only reports on `signals.db`: trade replay, panel scorecard, parity between surfaces, news lead/lag, scan outcomes (`--stored-candles`, `--news`, `--parity`, `--scan-outcomes`) |
| `scripts/backfill_signal_outcomes.py` | Batch seeding / advancing of advisory outcomes on an imported database (`--dry-run`, `--reseed-unresolvable`) |
| `scripts/label_scan_outcomes.py` | Batch labelling of scan outcomes for a backlog (`--dry-run`) |
| `scripts/panel_replay.py` | Offline replay of the agent panel on logged, resolved setups: rebuilds each persona's payload (exact `panel_payloads_json` when present, else from the stored JSON blobs, interned news and the kept Oanda cache), runs the CURRENT prompts, and scores Brier / calibration / constant-voter rate / counterfactual books against the logged v1 takes and the base-rate predictor. `--dry-run`, `--max-calls`, `--limit`, `--out`. Read-only on the DB; 4 DeepSeek calls per event |
| `scripts/panel_replay.py` | Offline replay of the agent panel on logged, resolved setups: rebuilds each persona's payload (the exact logged one from `panel_payloads_json` per persona when the row has it, macro alone on live rows since 2026-09-17; else from the stored JSON blobs, interned news and the kept Oanda cache; the CSV column `stored_payloads` names the personas that replayed a logged payload), runs the CURRENT prompts, and scores Brier / calibration / constant-voter rate / counterfactual books against the logged v1 takes and the base-rate predictor. `--dry-run`, `--max-calls`, `--limit`, `--out`. Read-only on the DB; 4 DeepSeek calls per event |
| `scripts/build_panel_base_rates.py` | Builds `agent_panel_base_rates.json` (per timeframe × strategy × grade × direction: n, win rate, mean R, TP1 reach, median hold, book MD5) from the exported reference books. Re-run after any re-pin |
| `agent_panel_base_rates.json` | The base-rate table the panel anchors its probabilities to (built 2026-09-17 from the relabelled W2/W3 books) |
| `agent_panel_base_rates.json` | The base-rate table the panel anchors its probabilities to (rebuilt 2026-10-01 from the reference books of section 9, the M15 one with the daily-trend veto; `generated_at` and each book's MD5 are in the file) |
### Backtesting and research
@@ -313,7 +359,7 @@ Arrows show who imports whom (generated from the code on 2026-09-15).
| `app_config.py` | Loads `config.json` once per process; env var → config.json → refuse; validates keys |
| `config.json.example` | The schema of `config.json` (which is gitignored) |
| `deploy/README-dashboard.md`, `deploy/gold-dashboard.service` | Live-server checklist and systemd unit for the dashboard daemon |
| `test_all.py` | The QA suite (38 modules; Module 36 is the GPU backend parity module, Module 37 the CUDA decision-stage parity module, Module 38 the local news model) |
| `test_all.py` | The QA suite (40 modules; Module 36 is the GPU backend parity module, Module 37 the CUDA decision-stage parity module, Module 38 the local news model, Module 39 the daily-trend veto and the wide-stop cap, Module 40 the trade-size / already-open / daily-trend display lines) |
---
@@ -350,7 +396,20 @@ symbol and interval maps. A test enforces that subclasses do not override the sh
scheduled closure since 2026-09-13. A "stale data" warning at any other weekday hour is a real
feed stall.
- The backtester fetches history through `fetch_range` (paginated, retries a failed page, logs an
error for interior gaps longer than 3 bars) and stores it under `backtest/cache/`.
error for interior gaps longer than 3 bars) and stores it under `backtest/cache/`. A DATE window
with no bars in it (a closed day at the head of a padded window) is an empty page, not a failure:
until 2026-09-30 the Oanda provider raised on it and `fetch_range` aborted after three tries, so
a D1 pad starting on a Saturday could not be fetched at all. Only a closure-sized window gets
that reading (`EMPTY_WINDOW_MAX`, 5 days: a weekend joined to a holiday, and no malformed candle
skipped); an empty answer for a longer window (a normal 15-day M5 page) raises, so `fetch_range`
retries it and refuses the history after three tries instead of caching a hole (2026-10-01).
A `count` request (the live path) with nothing in it still raises.
- Daily (D1) candles are UTC days (`dailyAlignment=0`). Oanda also emits a 1–2 hour stub bar
labelled Sunday for the reopening (about 600 of them since 2015); it is a real bar for the H4
confirmation engine, and the daily-trend definition drops Sunday labels before it computes
anything. Since 2026-09-30 D1 is a required frame of every M15 scan (live and backtest) and is
fetched for display on M5 and H1 (cached until the next daily close; a failure there leaves the
line empty and never fails the scan).
---
@@ -377,6 +436,7 @@ gates live in `config.py` and the pair factory, never in `config.json`. Blocks (
| `signal_log` | `enabled`, `path`, `log_blocked_scans` (env `SIGNAL_LOG_ENABLED`, `SIGNAL_LOG_PATH`, `SIGNAL_LOG_BLOCKED_SCANS`) |
| `news_archive` | `enabled` (default true), `path` (env `NEWS_ARCHIVE_ENABLED`, `NEWS_ARCHIVE_PATH`) |
| `news_model` | `path` (bundle directory, required when mode ≠ off), `threads` (default 4), `mode` (`off` default = DeepSeek/keyword path; `fallback` = model only where DeepSeek failed; `primary` = model scores every headline, no DeepSeek headline calls), `deepseek_audit_per_day` (default 0; primary only, archive-only DeepSeek labels). Env `NEWS_MODEL_MODE` overrides `mode` per process (the dashboard unit). Live: `primary` on both processes |
| `trade_size` | Lot-size guidance on the alert (display only, 2026-09-30): `risk_per_trade` (US dollars; 0 or absent = the size line is off), `contract_size` (ounces per lot, default 100), `lot_step` (default 0.01), `telegram_channel` and `dashboard` (default false: the size shows the owner's risk, so every Telegram chat other than the owner's own (`telegram.chat_id`) — the channel, a group — and the public page get it only when these are true). No env override |
| `log_level` | Logging level |
A legacy top-level `market_outlook` block is tolerated and ignored with one warning (the feature
@@ -392,7 +452,7 @@ day, 30 kept), `logs/dashboard.log`. Read the databases with `mode=ro`, never `i
## 6. Testing
- `venv/bin/python test_all.py` runs everything with mocked network. Expect
`RESULTS: 581 passed, 0 failed, 581 total` (2026-09-27). Module 38's last test loads the
`RESULTS: 632 passed, 0 failed, 632 total` (2026-10-01). Module 38's last test loads the
real news-model bundle when `NEWS_MODEL_DIR` (or the configured `news_model.path`) exists and
prints SKIP otherwise. On a box with a visible CUDA device
Module 36 fails if the GPU backend falls back (the reason is in the message); with
@@ -403,6 +463,12 @@ day, 30 kept), `logs/dashboard.log`. Read the databases with `mode=ro`, never `i
both database writers at a suite-private temp dir; before 2026-09-17 the RSS fixture tests wrote
fixture headlines into the real `logs/newsdata.db`, and a Module 32 test now pins the isolation. One Twelve Data interval-map test
reaches the network and can time out on a slow connection; a rerun clears it.
- The suite is meant to run on the live box too (Python 3.9, SQLite 3.34, no `pyarrow`, no CUDA):
the four tests that must write a Parquet fixture print a notice and return when no Parquet
engine is installed, and a test that needs an older database builds it forwards
(`_legacy_signals_db39`: the base schema plus every additive column but the one under test),
never with `ALTER TABLE … DROP COLUMN` (SQLite 3.35). Verified 2026-10-01 with both Parquet
engines blocked and `FB2_BACKEND=cpu`: 0 failures.
- Test module numbers are cited from this file, so never renumber them. Module 10 and 26 are gaps.
- When a test needs the weekend rule, patch `providers.base._is_forex_weekend` and remember the
other copies. Signal-log fixtures that exercise candle continuity must use a weekday bar.
@@ -423,7 +489,13 @@ day, 30 kept), `logs/dashboard.log`. Read the databases with `mode=ro`, never `i
fills. Marginal edges must also pass the 2024-06-01 → 2025-03-30 out-of-sample window. Backtest
windows go back at most 3 years (owner preference, 2026-09-05). `round_trip_cost` stays at 0.30
because it also drives the live target floor (0.65 would move target 1's floor from about $9 to
$19.50); changing it is its own rule-1 event.
$19.50); changing it is its own rule-1 event. A change whose purpose is to cut the losing runs
rather than add R may instead be judged by a **risk-first rule** (owner decision 2026-09-30),
written before the runs: 3-year R at least 97% of the control, profit factor up by 0.10 or more,
max drawdown not deeper, the 2019 → 2026 window better on R, on drawdown and in 2021 + 2022, an
unseen holdout, both costs, and at least 3 of 4 neighbouring definitions holding. Under it the
halves of a purely rising window are reported, not clauses. The owner chooses the rule per
campaign; the default stays the acceptance above.
2. **Per-timeframe overrides live in the pair factory**, never in the global defaults. The same
knob has had opposite effects on different timeframes several times. Record the evidence in the
factory docstring.
@@ -466,7 +538,7 @@ overrides (each one has its validation record in the `pair_signals/xauusd.py` do
| Timeframe | Confirmations | Overrides on top of the defaults |
|---|---|---|
| M5 | M15, H1 | hour 9 UTC blocked and hour 21 unblocked; chart-pattern entries on (buy-side only: inverse head and shoulders, double bottom, bull flag); pullback entries off; partial EMA alignment scores neutral; trailing stop arms at 0.60R and trails 0.85R; shorts blocked in the `transitioning` regime |
| M15 | M5, H1 | target floor 1R; the mixed `trending` regime blocked; shorts blocked at 15:00 and 17:00 UTC; trailing stop arms at 0.70R and trails 0.75R; RSI weight halved in every regime; grade-A shorts blocked (`sell_min_quality` `A+, B`) |
| M15 | M5, H1 (+ D1 for the veto) | target floor 1R; the mixed `trending` regime blocked; shorts blocked at 15:00 and 17:00 UTC; trailing stop arms at 0.70R and trails 0.75R; RSI weight halved in every regime; grade-A shorts blocked (`sell_min_quality` `A+, B`); daily-trend veto in both directions (`daily_trend_veto` `both`, since 2026-09-30) |
| H1 | M15, H4 | every short blocked (`sell_min_quality` empty) |
| H4 | H1, D1 | none. H4 remains "unproven" (few trades) |
@@ -475,7 +547,13 @@ to Friday plus Sunday evening; shorts need grade A+, A or B and high-or-extreme
trailing stop arms at 0.6R and trails 0.75R; gated target floor 0.5R and printed target floor 1R;
round-trip cost $0.30 with
target 1 at least 30 × cost; the generic and chart-pattern setups floor the stop at $8; the gold
ATR setup uses a 1 × ATR target before the grade scale (2× at A+, 1.5× at A) and a 2 × ATR stop; the H4 regime filter is off; news never gates.
ATR setup uses a 1 × ATR target before the grade scale (2× at A+, 1.5× at A) and a 2 × ATR stop; the H4 regime filter is off; news never gates;
the daily-trend veto is off (M15 alone turns it on) and the wide-stop cap is off everywhere
(`wide_stop_cap_atr` 0.0, a research knob that failed its rule).
The daily-trend veto failed the same rule on M5 (it deepens the 3-year losing run from 10.1R to
15.2R and loses the 2019 → 2026 window) and on H1 (−9.7R of +70.0R). Do not turn it on there
without a new campaign; the daily trend is printed on those alerts as context only.
One semantic to know: `blocked_hours_utc` is applied to the bar **label** hour inside the core
and to the bar **close** hour in the live wrapper and the backtester. So H1 can only trade bars
@@ -501,11 +579,19 @@ CuPy GPU backend and `--backend cpu` both reproduce every MD5 below; earlier rep
| Book | Window | Trades / R / PF | MD5 of the CSV |
|---|---|---|---|
| M5 W2 | 2024-09-05 → 2026-09-05 | 781 / +146.07 / 1.52 | `e599c8a8652f95b7fcdf5a7ef2b564f5` |
| M15 W2 | 2024-09-05 → 2026-09-05 | 813 / +154.15 / 1.53 | `5b65ecf6fad9a8da412749066fd50d36` |
| M15 OOS | 2024-06-01 → 2025-03-30 | 313 / +68.50 / 1.67 | `09b8a8e7f5833436c20aef60ca0a7ff7` |
| M15 W2 | 2024-09-05 → 2026-09-05 | 597 / +155.89 / 1.78 | `dfc647ee02afd179dd9e031f344ed380` |
| M15 OOS | 2024-06-01 → 2025-03-30 | 258 / +63.78 / 1.80 | `e87b6de12f90f3b5a5d41a953ee9ff7a` |
| H1 W3 | 2023-09-05 → 2026-09-05 | 253 / +70.0 / 1.98 | `a44fd19b4067f46e5bd98d50cc3eff92` |
| H4 W3 | 2023-09-05 → 2026-09-05 | 114 / +21.7 / 1.59 | `f3587bae3c5dc3eda468ea3588b17dcb` |
Re-pinned 2026-09-30. M15 only: the daily-trend veto (M15 override 7). W2 216 trades removed,
+1.7R, max drawdown 15.6R → 11.4R; OOS 55 removed, −4.7R (the accepted cost of the risk-first rule
in a rising window), drawdown 12.4R → 9.1R; previous pins `5b65ecf6…` (813 / +154.15 / 1.53) and
`09b8a8e7…` (313 / +68.50 / 1.67). M15 W3 with the veto 824 / +205.12 / 1.73, max drawdown 12.4R,
`ad3b0dbda0bf77b659caea1c53be6a50` (was `70195fb0…`, 1105 / +195.79 / 1.49, drawdown 20.9R). The
M5, H1 and H4 pins are unchanged. An M15 book needs the D1 frame of its window (1,400-day pad) in
the cache; `--set xau_strategy.daily_trend_veto=off` reproduces the previous M15 pins byte for byte.
Re-pinned 2026-09-26. M15: the grade-A SELL block (M15 override 6, 162 trades removed, +8.0R);
previous pins `b4aa07e3…` (931 / +146.14 / 1.43) and `9fd624fb…` (355 / +60.67 / 1.49); M15 W3
(3-year) with the block 1105 / +195.79 / 1.49, `70195fb0e6865b3a46ee8a9aa8844e51`. All four
@@ -520,18 +606,28 @@ columns of all five books are byte-identical to the previous pins (`29120a1e…`
`6f8f24b4…`, `3b45d12b…`, `e75adae1…`); trades, R and PF are unchanged. `--strategies` accepts the
live names (`XAU` still means `XAU_ATR`).
Other current-HEAD figures at $0.65: M5 W3 (3-year) 870 / +153.6R / PF 1.50; M15 W3 1273 /
+186.6R / PF 1.39; M5 12-month (2025-08-14 → 2026-08-14) 560 / +108.81R / PF 1.54;
H1 12-month (2025-08-04 → 2026-08-04) 91 / +22.2R / PF 1.79. All five MD5s above were
reproduced on 2026-09-15 after the cache prune and on 2026-09-18 on both backends
(`backtest/research/gpu_backend_2026-09-18.md`).
Other current figures at $0.65 (2026-10-01): M5 W3 (3-year) 870 / +153.64R / PF 1.50 / max
drawdown 10.1R (`d20ec8a3…`); M15 W3 824 / +205.12R / PF 1.73 / 12.4R; M5 12-month (2025-08-14 →
2026-08-14) 560 / +108.81R / PF 1.54; M15 12-month (2025-08-04 → 2026-08-04) 266 / +92.3R /
PF 2.12 / 8.3R; H1 12-month (2025-08-04 → 2026-08-04) 91 / +22.2R / PF 1.79. The long robustness
window 2019-01-01 → 2026-08-07 (allowed as supporting evidence, owner decision 2026-09-30): M15
1473 / +223.33R / PF 1.42 / max drawdown 41.5R (`aa5253aa…`; by year +14.0, +45.1, −27.3, +3.2,
+1.8, +53.5, +92.5, +40.6), M5 1052 / +128.10R / PF 1.34 / 18.4R (`3f0c091c…`; −1.0, −2.8, −1.7,
−2.2, −2.8, +1.0, +40.5, +97.0: the M5 rules take few trades and earn nothing before 2025). The
unseen holdout 2026-09-05 → 2026-10-01: M15 18 / −4.62R, M5 33 / −2.45R, H1 3 / −3.04R. The pins of
the table above were reproduced on 2026-09-15 after the cache prune, on 2026-09-18 on both
backends (`backtest/research/gpu_backend_2026-09-18.md`) and on 2026-10-01 with both scripts.
The CUDA research twin (`backtest/cudaBasedBacktesting.py`, section 3) reproduced all five MD5s
above on 2026-09-18 in 0.8–2.4 s per book, plus the four 6-month acceptance books (2026-03-18 →
2026-09-18, M5 and M15 at $0.65 and $0.30: `86111c36…`, `e70d0252…`, `b8c98083…`, `e66b5fbd…`,
each equal to the reference script's), the M15 OOS `--export-features` CSV (`6239d197…`) and
`--record --record-bars` tables, and ten research-flag variants
(`backtest/research/cuda_backtester_2026-09-18.md`).
(`backtest/research/cuda_backtester_2026-09-18.md`). Parity was re-proved for the 2026-09-30 change:
the five pins above with both scripts, six `--set` books on W3 (veto both / buy / sell and the stop
cap on M15, veto and cap on M5), the M15 OOS `--export-features` CSV with the veto on
(`732f396bb3ee6036b62750a8c755194e`) and its `--record --record-bars` tables
(`backtest/research/daily_trend_campaign_2026-09-30.md`).
Kept cache windows (`backtest/cache/oanda_XAU_USD_<tf>_<paddedStart>_<end>.parquet`; the padded
start is the window start minus 62 days for M5/M15/H1, 234 for H4 and 1,400 for D1, so the D1
@@ -539,7 +635,12 @@ and H4 files of one run have different start dates than its M5 file): 3-year `20
2-year `20240705_20260905`, OOS `20240331_20250330`, M5 12-month `20250613_20260814`, M15/H1
12-month `20250603_20260804`, 5-year `20210531_20260804`, 7.6-year `20181031_20260807`, the
2026-08-14 parity window `20250814_20260814`, the 6-month CUDA acceptance window
`20260115_20260918` (M5/M15/H1), their D1/H4 pads, and `calendar.parquet`.
`20260115_20260918` (M5/M15/H1), their D1/H4 pads, and `calendar.parquet`. Added 2026-09-30 for
the daily-trend veto (an M15 run needs the D1 frame of its window): `1d_20201105_20260905` (W2),
`1d_20200801_20250330` (OOS), `1d_20150303_20260807` (the 7.6-year window), `1d_20211004_20260804`
and `4h_20241213_20260804` (the M15 / H1 12-month window), and the holdout window
`20260705_20261001` (M5 / M15 / H1) with `1d_20221105_20261001` and `4h_20260114_20261001`. The
M15 3-year run reads `1d_20191105_20260905`, which H4 W3 already kept.
The research recorder (`--record DIR [--record-bars]`) writes Parquet files describing every
accepted trade (indicator votes on all three timeframes, penalties, setup, outcome, calendar
@@ -567,7 +668,9 @@ MD5s. Scorecards from it are in-book statistics, never decisions.
there too since 2026-09-08); use `setup_json.strategy`.
- The `--stored-candles` scorecard in `analyze_signal_log.py` still advances outcomes without the
locked-profit floor step that the logger and the backfill script use (owner decision pending), so
the two measurement paths can disagree on gap rows.
the two measurement paths can disagree on gap rows. Those are now only the trades still OPEN at
a hole (3 of 111 setups on the 2026-10-01 database copy; until then the replay also threw away
every trade that had exited before a later hole, 12 of 111).
- There is no holiday awareness anywhere. On Labor Day 2026 Oanda published no bars for hours, the
bot rescanned the stale bar 43 times, and the violent-move watch called the 12–14 UTC slot hot.
Interior gaps in history are logged, never fatal, for the same reason.
@@ -584,6 +687,29 @@ MD5s. Scorecards from it are in-book statistics, never decisions.
≈ 0.03 with the next). Both news gating flags stay off.
- The level-breach rescan and the candle loop both skip the settlement-pause window
(`_closed_bar_in_settlement`); a rescan there only re-reads the last pre-pause bar.
- The alert cooldown counts bars of `report._frames` (set by `analyze_pair`). A report built by
hand has no frames and falls back to clock time, which releases the first bars after the
settlement pause or the weekend too early; that is the fallback, not the rule.
- An alert's row is stamped `alerted_at` BEFORE the send and the stamp is cleared when no chat
received it, also when the alert's task is cancelled while the stamp is being written (a
shutdown at that moment: the worker job cannot be stopped, so its future's done-callback
undoes it). A cancellation in the MIDDLE of the send keeps the stamp, because a chat may
already hold the message. Two residues, both accepted: if a send fails completely after
another alert already counted the row, the other alert's "already open" line is one too
high; and a process KILLED between the stamp and the send leaves a stamp on an alert nobody
received (it also seeds the cooldown at the next start, which errs on the quiet side).
- `news_engine._record_source("news_model", …)` is the only sign that the local headline model
failed a batch (WARNING once, `/status` line `News model: not scoring`); in `primary` mode
nothing else scores those headlines for the cycle. The line describes the latest cycle that
needed the model: a cycle with nothing for it to score (fallback mode with DeepSeek healthy)
drops an older failure instead of leaving it on `/status`. `/status` failure details are
exception text and are Markdown-escaped; an unpaired underscore made Telegram reject the reply.
- The agent panel's scorecards split at `PANEL_PROMPT_VERSION` v2.2 (2026-10-01): same prompt
text, different input (calendar-week event timing, headlines newest first). `_price_context`
never fetches the calendar itself; the offline replay passes an as-of context and must not see
today's events.
- The Oanda empty-window rule cannot catch an anomalous empty answer for a window of 5 days or
less (a short head page; every 3-day M1 page, a timeframe nothing fetches).
- The agent panel never runs inside `analyze_from_dataframes`, and its cached verdict is never
reused when the evidence changed; the daily cap is the real spend bound.
- Live == backtest also means the backtester never writes to `signals.db` and never builds a
@@ -624,6 +750,45 @@ MD5s. Scorecards from it are in-book statistics, never decisions.
and never feed the printed target back into a gate.
- "Stop hunting" was tested on the live log (2026-09-26): 71% of live stop-outs ran a full stop
distance or more past the stop, the same as the books' 70–72%. The losers were real moves.
- The daily-trend veto has ONE definition (`signal_aggregator.daily_trend_series`; EMA span 50 and
lookback 60 are module constants, validated literals). Live, the bar replay and the twin's host
side (`trend_flags`) all call it; the kernel only receives a flag per bar. Changing the
literals, the Sunday filter or the as-of rule (a daily bar labelled D counts from D + 1 day ≤ the
primary bar's close) is a rule-1 event and a parity event. The live frame holds 1,000 daily bars
and a backtest pad 1,400 days; the EMA seed has decayed below float precision in both, so the
states agree.
- With the veto on, D1 is a REQUIRED frame of the M15 scan: a failed D1 fetch fails the scan
(`DATA UNAVAILABLE: 1d — scan failed`), it does not fall back to "no veto". After a deploy watch
that rate. Inside the core a missing D1 slice only adds the notice `Daily trend filter requested
but 1d data unavailable` and ships the setup; the replay reaches that state only before the
first daily bar of its pad.
- The veto is the LAST step of the core (after the chart-pattern path and the stale-veto cleanup),
so `gate_reason` lists `DAILY TREND FILTER` ahead of the gate chain. The rule itself only
removes a plan. In a book a removed entry also moves the 5-bar cooldown, which could free a
later entry: that is possible in principle and happened 0 times in every book of the campaign
(an observation, not a guarantee; count the added trades in any new comparison). The wide-stop cap never
changes the trade set either (it is applied after the min-RR and spread gates); a smaller stop
can lower the printed 1R target, never raise it.
- The control for any M15 candidate is now the factory config WITH the veto. Comparing against a
pre-2026-09-30 M15 number needs `--set xau_strategy.daily_trend_veto=off` (reproduces
`5b65ecf6…` / `09b8a8e7…` / `70195fb0…`).
- The veto does not find big losers. The trades it removes are break-even after costs (−0.03R to
−0.05R each against +0.15R to +0.25R for the kept ones); the gain is fewer flat trades and
shorter losing runs. The SELL half is carried by the rising 3-year window and the BUY half by the
falling years of 2019 → 2026; `buy`-only and `sell`-only both fail. 2021 is not improved and 2023
is worse: in a sideways year the daily trend flips late.
- The lot-size line reveals the owner's risk per trade. It is stamped per surface
(`exposure.attach_exposure`) and the channel copy of an alert is built separately
(`format_signal_message(report, channel=True)`); never print `report.size_line` on a public
surface without checking `exposure.size_shared`. The dashboard snapshot is public.
- "Already open" is the advisory outcome model's view of ALERTED plans (`alerted_at` set, outcome
`PENDING`), not a broker position. Rows alerted before the 2026-09-30 deploy carry no
`alerted_at`, so the line starts empty and fills as alerts go out; a `PENDING` row older than the
model's own time-out (300 bars, with a weekend allowance) is not shown. Stacked entries were not
worse trades in the books, so the line is information, never a reason the code skips an alert.
- `signals.daily_trend` is the core's value on M15 and the live wrapper's display stamp on M5, H1
and H4 (`_stamp_daily_trend_for_display`, one extra cached D1 fetch on M5 / H1); it is NULL on
blocked and failed scans. The stamp sits in `analyze_pair`, never in `analyze_from_dataframes`.
- The dashboard's due-timeframe check uses the raw wake time; flooring `wake − buffer` skipped
every H1 and H4 scan once (2026-08-02).
- The agent panel's price context reads the frames `analyze_pair` stashes on `report._frames`.
@@ -644,6 +809,10 @@ One line per validated change or verdict, newest first. Full evidence in the nam
| Date | Decision | Where the evidence is |
|---|---|---|
| 2026-10-01 | Bug review of the whole codebase (owner request), then the fixes. Signal path clean: indicators, grading, gates, the daily-trend veto, the as-of logic and both backtesters checked, and no live module needs a newer Python or SQLite than the server has. 17 defects elsewhere, all FIXED, each with a test; books byte-identical (the five pins reproduced on both scripts, 0 cache fallbacks). **Alert path**: the "already open" line was stamped at scan time, so two alerts at one candle close never counted each other (the M5 and M15 books enter together 42 times in three years); the row is now stamped just before the send on a one-worker `alert-db` pool and un-stamped when no chat received it (`_deliver_alert`, one path for the candle scan and the level break; no SQLite write on the event loop any more). The 5-bar cooldown counted clock time and forgot its state at every restart: it now counts the scan's own bars like `_apply_cooldown` (the old rule released the 22:00 UTC bar after the settlement pause: 6 M15 and 3 M5 entries the books never take in three years) and is restored from `signals.db` at start-up. The lot-size line went to every target that was not the channel (a group the owner ran `/autoscan` in): now the owner's own chat only. **Measurement**: the outcome model rejected a whole candle stream for a hole anywhere in it; continuity is now judged bar by bar in the walk, so the stored-candle replay resolves 108 of 111 logged setups instead of 99, all 108 equal to the R the live resolver stored. **History fetch**: an empty answer is accepted only for a window of 5 days or less with no malformed candle (a longer one is retried, then refused). **News**: one unreadable headline no longer costs the local model its whole batch (verified on the real bundle), a failing batch is a WARNING and a `/status` line, no DeepSeek audit call in a process that does not archive, the event-direction cache is per release. **LLM client**: `chat_json` never raises and never logs the key; the scan's news block is guarded (M15 OOS pair re-proved, Module 37 green). **Panel** (`PANEL_PROMPT_VERSION` v2.2, a new scorecard era): event timing over the cached calendar week, headlines newest first; the replay script uses the logged macro payloads it used to ignore (39 of 110 events). **Tests**: no `DROP COLUMN` (SQLite 3.35), Parquet-writing tests skip without `pyarrow`, the suite passes with both Parquet engines blocked and no GPU. Also fixed on the way: `/status` failure details are Markdown-escaped. **Independent review of these fixes** (a second agent, 20,000-case differential fuzz of the outcome model among its checks): four further LOW defects, fixed the same day — an alert cancelled while its row was being stamped kept the stamp; the panel's calendar read could trigger a 40 s refetch (now `peek_week_events`); a stale model-failure line could stay on `/status`; an HTTP error body was logged without redaction. **Independent verification of the 2026-09-30 direction change**: every figure reproduced, no signal changed by this pass, caveats recorded in section 5.13 of the campaign report. Suite 632 / 40 modules | plan file `~/.claude/plans/you-are-a-quant-velvety-wave.md`; sections 2, 4, 6 and 10 of this file |
| 2026-10-01 | New data sources, report only (owner request). Eight outside series tested against gold's own daily trend: the dollar (EUR/USD), silver, the US 10-year note, the real yield and the broad dollar index (FRED), the gold fund's share count, CFTC managed-money positioning, and Oanda's own client positioning (the position-book endpoint DOES answer on the practice API, with history to 2017; the plan had expected it gone). None shows a forward move of gold beyond gold's own trend (largest t-values 1.2 and 1.8); on the books only one flag, M5 trades against the broad dollar index (−11.2R over 435, losing in both eras), which is the 2025–2026 rally seen from another angle and does nothing on M15: WATCH, a hypothesis. Nothing ships, no feed added, all data stayed in the scratchpad | `backtest/research/new_inputs_2026-10.md` |
| 2026-10-01 | Context screens after the veto (owner request): weekly trend, four-hour trend, daily-trend strength, daily volatility state, place in the 20-day range, tick-volume surge, time to a high-impact US release; rule written first (a state needs 8% of both books and a loss in all four slices to earn a replay). NO state nominated on M15 or M5, no replay, nothing ships. Description: M15 has no losing pocket left in these views; M5 earns only where everything agrees (with the weekly and four-hour trend +0.16R to +0.24R per trade, against them about 0) and near the 20-day extreme in the trade's direction, so a "do not chase" filter would cut the part that pays | `backtest/research/context_screens_2026-10.md` |
| 2026-09-30 | Direction and setup-size campaign (owner request; RISK-FIRST rule approved by the owner, 2019 → 2026 allowed as supporting evidence). **Daily-trend veto SHIPPED on M15** (override 7, `daily_trend_veto` `both`): a finished setup against a clear daily trend (close vs 50-day EMA and vs 60 days earlier, closed D1 bars, Sunday stubs dropped) is removed. PASSED every clause: W3 824 / +205.12R / PF 1.73 / max DD 12.4R vs 1105 / +195.79R / 1.49 / 20.9R; 2019 → 2026 1473 / +223.33R / 1.42 / 41.5R vs 2091 / +192.53R / 1.24 / 64.8R; unseen holdout −4.62R vs −6.63R; both costs; 3 of 4 neighbouring definitions. Accepted cost: W2 first half +61.8R vs +70.1R, old OOS window −4.7R. FAILED: `buy`-only and `sell`-only on M15, every mode on M5 (W3 drawdown 10.1R → 15.2R, long window worse) and on H1 (−9.7R). **Wide-stop cap FAILED** rule C (K = 6 +11.8R on W3 but K = 5 −18.6R on the long window: a peak, not a plateau); knob kept at 0.0. M15 books re-pinned (W2 `dfc647ee…`, OOS `e87b6de1…`), M5 / H1 / H4 pins unchanged, twin MD5-equal throughout, recorder parity re-proved. **Display lines SHIPPED** (books byte-identical): lot size for the owner's risk limit (`trade_size` in config.json, private by default), alerted same-direction plans still open (`signals.alerted_at`, read-only `open_alerted_setups`), the daily trend in words on every surface and in every logged row. Measured on the way: taking every M5 + M15 alert, two entries in three already have a same-direction plan open and up to 13 are open at once; those stacked entries are not worse trades. FIXED: the Oanda provider raised on an empty date window, so `fetch_range` aborted on a pad starting on a closed day. Stops and exits re-screened (25 variants): at a plateau, tighter is far worse. Suite 607 / 40 modules | `backtest/research/daily_trend_campaign_2026-09-30.md`; factory docstring M15 override 7 |
| 2026-09-27 | Local news scorer, round two: the owner's ML-NEWS model is now a DeBERTa-v3-small two-head transformer (gold-relevant? then bullish/bearish), distilled from 17,245 DeepSeek gold labels (Sep 8–22 archive + a 2019–2026 Reuters/CNBC/MarketWatch crawl labelled by DeepSeek, 560 calls). Golden set 72/78 (DeepSeek 78, keywords 24, the 09-17 TF-IDF 39), agreement with DeepSeek on unseen live days ~72% (ceiling 90%), 19 ms/headline on one CPU thread as int8-embedding ONNX; passed the four promotion criteria fixed on 09-22. OWNER DECISIONS: it is to score headlines here in mode `primary` with NO DeepSeek for headline scoring (DeepSeek stays for event direction, summaries, the dashboard endpoint); model files live OUTSIDE this tree at a configured path (280 MB, never in git); shadow mode declined (the archive already allows offline replay). IMPLEMENTED the same day (branch `sql-integration`): `news_model.py` (vendored runtime + never-raising singleton, lazy `onnxruntime` / `tokenizers`), `news_model` config section + `NEWS_MODEL_MODE` env, `news_engine._enrich_headline_scores` / `_apply_model_sentiment` (gold-only rules A–F) / `_audit_with_deepseek`, archive provenance for `model` rows, `deploy/README-news-model.md`, Module 38 (15 tests, one against the real bundle: bullish / neutral labels, 7 ms per headline on the dev box). Two further owner decisions during implementation: **`primary` mode stops writing `newsdata.db`** (only the DeepSeek audit sample, shipped 0, is archived) and **both processes load the model**. Books unaffected (news never enters signals); no signal file touched, no parity event. ML-NEWS `news_transformer.py` received the same two `load_model` edits (commit `9496cd9` there) | `docs/news_model_integration_spec_2026-09-27.md`; `~/Development/ML-NEWS/eval/RESULTS_2026-09-27.md` |
| 2026-09-27 | Single "conviction" number vs the two-number grade (owner question: one number that is easier to understand and to decide on). Pre-registered; switch `signals.grading_mode` (`pair` default = the shipped rule, byte-identical on both scripts, all five pins reproduced; `min` = min(\|score\|, 0.6 × conf); `product` = \|score\| × conf / 100; thresholds from the control's corner points, no tuning; mirrored in K2, twin MD5-equal to the reference on both new paths). FAILED on M5 and M15: K-MIN moves 43–54 trades for −1.2R / −2.9R (a, b, e / a, b, d); K-PROD −29R on M15 with PF 1.29 and W3 DD 28.9R, neutral on M5. H1 / H4 K-PROD +11.9R / +17.0R on W3 by taking 40–100 extra low-score trades (H1 PF 1.98 → 1.75, DD doubles): a hypothesis, not evidence. Nothing ships; the knob stays research-only. Code review (high) before the runs: 6 findings fixed, 1 recorded. SHIPPED the display half of the question (owner request the same day): alerts lead with the grade and print score / confidence as words (`main.plain_words`, same line on Telegram, `/status`, CLI and the dashboard card); books untouched. Suite 566 | `backtest/research/conviction_campaign_2026-09-27.md` |
| 2026-09-26 | Quant review + live log 2026-09-04 → 09-25 (100 setups replayed on the scans' stored candles): stop-hunt claim REFUTED (71% of stop-outs ran ≥ 1R past the stop, same as the books); −25.1R of the −37.8R raw live book were consecutive-bar re-entries the backtester's cooldown never takes; the cooldown view sits at the 3rd–5th percentile of the books' three-week windows. SHIPPED: 5-bar alert cooldown; alert wording (trail first as the tested exit, TP1 a guide with its R, planned RR, validated-book line per class); M15 grade-A SELL block (`sell_min_quality` `A+, B`, PASSED a–f: W2 +154.15R/813 PF 1.53 vs +146.14/931 PF 1.43, OOS +68.5 vs +60.67, W3 DD +8%); 1R printed-target floor in every generator after the gates (`tp_print_floor_rr`, owner rule "TP1 at least 1:1"; label-only re-pin of all four timeframes). FAILED: M15 `min_rr_ratio` 1.0 (trade-identical, −2.0R); a pre-gate target floor (−24.6R). OWNER'S CALL: M5 `tp_floor_rr` 1.0 adds +48.9R on W2 with both halves up but max DD 10.1 → 15.9R (ladder 0.6–1.0 non-monotone); not shipped. Telegram / dashboard AI block rewritten for a new trader (owner rule): one plain headline against the usual odds, the chance "in 100", one reason for and one against, no verdict word or confidence figure; persona prompts ban trading jargon (`PANEL_PROMPT_VERSION` v2.1). Two delivery bugs from the DB checks fixed: the macro-source panel verdict never stored its probability (all v2 rows NULL); the level-breach rescan re-logged the 20:45 M15 bar three times every evening after the settlement pause (guard added to `_handle_breach`). Indicator grid and volume / daily-direction screens (Part C) not run | `backtest/research/rr_and_sell_campaign_2026-09-26.md` |