Files
ForexBot2/backtest/research/context_screens_2026-10.md
Harwinder Sekhon 2cc5915b00 Daily-trend veto on M15 + lot-size / already-open lines (campaign 2026-09-30), and the 2026-10-01 whole-codebase bug review
Two bodies of work that touch the same files, so one commit.

Direction and setup-size campaign (2026-09-30 → 10-01, owner's RISK-FIRST rule,
written before the runs):
- SIGNAL: M15 daily-trend veto (factory override 7, xau_strategy.daily_trend_veto
  "both"). A finished setup against a clear daily trend (closed D1 bars, Sunday
  stubs dropped; close vs its 50-bar EMA AND vs the close 60 kept days earlier)
  is removed as the last step of the core. One definition
  (signal_aggregator.daily_trend_series) for live, the bar replay and the twin's
  host side; D1 is a required frame of the M15 scan; mirrored in
  cuda_signal_kernels.py K2. W3 824 / +205.12R / PF 1.73 / max DD 12.4R vs
  1105 / +195.79R / 1.49 / 20.9R; 2019 → 2026 1473 / +223.33R / 1.42 / 41.5R vs
  2091 / +192.53R / 1.24 / 64.8R; unseen holdout −4.62R vs −6.63R.
- FAILED, not shipped: the veto on M5 (W3 drawdown 10.1R → 15.2R) and on H1
  (−9.7R), buy-only and sell-only on M15, the wide-stop cap
  (xau_strategy.wide_stop_cap_atr stays 0.0, a research knob).
- New M15 pins: W2 dfc647ee…, OOS e87b6de1…, W3 ad3b0dbd…. M5 / H1 / H4 pins
  unchanged. --set xau_strategy.daily_trend_veto=off reproduces the previous M15
  pins byte for byte. Twin and reference script MD5-equal on every window;
  recorder and feature-export parity re-proved; base-rate table rebuilt.
- DISPLAY (books byte-identical): exposure.py — the lot size for
  trade_size.risk_per_trade (new config.json block, off by default, private to
  the owner) and the count of alerted same-direction plans still open
  (signals.alerted_at, read-only open_alerted_setups); the daily trend in words
  on Telegram, the CLI and the dashboard; signals.daily_trend logged.
- providers/oanda: a date window with no bars is an empty page, not an error
  (a D1 pad starting on a Saturday could not be fetched).
- Reports only, nothing shipped: seven context screens (no state nominated) and
  eight outside data sources (none adds a forecast to gold's own daily trend).
- Independent verification (report section 5.13): every figure reproduced on
  both scripts, scoring identical to the previous commit on all four
  timeframes, no look-ahead. Caveats recorded there: this is a risk filter, not
  a more accurate direction call; nine tenths of the 2019 → 2026 gain is 2022;
  the neighbour clause is thin (a simple 100-day average misses the R line by
  1.2R); "never adds a trade" is an observation, not a guarantee.

Bug review of the whole codebase (2026-10-01). Signal path clean. 17 defects
elsewhere, plus 4 found by an independent review of the fixes; all fixed, each
with a test; no trade book changed:
- telegram_bot: one _deliver_alert for the candle scan and the level break. The
  row is stamped alerted_at BEFORE the send, on a one-worker alert-db pool, and
  un-stamped when no chat received it or the task is cancelled mid-stamp, so
  two alerts at one candle close count each other and no SQLite write runs on
  the event loop. The 5-bar cooldown counts bars of the scan's own frame
  (== gpu_backtest._apply_cooldown; clock time released the 22:00 UTC bar after
  the settlement pause) and is restored from signals.db at start-up. The
  lot-size line goes to the owner's own chat only (a group the owner ran
  /autoscan in used to receive it). /status gains a news-model line and escapes
  failure details.
- signal_log: mark_alerted returns the stamp it wrote; unmark_alerted;
  last_alerted_bars (read-only).
- advisory_outcome: candle continuity is judged bar by bar inside the walk; a
  hole after a trade's exit no longer voids it. The stored-candle replay
  resolves 108 of 111 logged setups instead of 99, all equal to the live
  resolver's R.
- providers/oanda: an empty date window is accepted only when it is 5 days or
  shorter and no malformed candle was skipped; a longer one is retried, then
  refused, instead of being cached as history with a hole.
- news_engine: blank RSS titles skipped; one unreadable headline no longer
  costs the local model its whole batch (checked on the real bundle); model
  health recorded; no DeepSeek audit call in a process that does not archive;
  event-direction cache keyed per release; peek_week_events never fetches.
- llm_client: chat_json never raises and never logs the API key (requests
  quotes the Authorization value in InvalidHeader; error bodies are redacted).
- signal_aggregator: the scan's news block is guarded; the replay path
  (use_news=False) assigns exactly the values it did before.
- agent_panel: PANEL_PROMPT_VERSION v2.2 — same prompt text, corrected input
  (event timing over the cached calendar week, headlines newest first).
  scripts/panel_replay.py replays the logged payload per persona;
  scripts/label_scan_outcomes.py counts with an ISO cutoff.
- test_all.py: no ALTER TABLE … DROP COLUMN (the live box has SQLite 3.34);
  tests that write Parquet print a notice and return without pyarrow.

Verification after the last code change: suite 632 passed, 0 failed (also with
both Parquet engines blocked and FB2_BACKEND=cpu, the live box's view); the
five reference MD5s reproduce on both backtesters, 0 cache fallbacks; Module 37
green; static Python 3.9 / SQLite 3.34 check clean on every touched file.

Evidence: backtest/research/daily_trend_campaign_2026-09-30.md,
context_screens_2026-10.md, new_inputs_2026-10.md; CLAUDE.md sections 2, 4, 6,
8, 9, 10 and 11; pair_signals/xauusd.py docstring (M15 override 7).
2026-10-01 10:28:28 -04:00

17 KiB
Raw Permalink Blame History

Context screens after the daily-trend veto — declared before looking, 2026-10-01

Follow-up to daily_trend_campaign_2026-09-30.md (owner decision 2026-09-30: "more context screens"). The daily chart was information the engine had never seen, and it paid on M15. This file asks the same question of seven more pieces of context. Sections 1 to 3 were written before any number in section 4 existed.

1. Books and slices

The books as shipped on 2026-09-30, --spread 0.65, CUDA twin (byte-identical to gpu_backtest.py on the paired runs of the campaign report):

Book Window Trades / R MD5
M15 LONG (veto on) 2019-01-01 → 2026-08-07 1473 / +223.33 aa5253aa…
M15 W3 (veto on) 2023-09-05 → 2026-09-05 824 / +205.12 ad3b0dbd…
M5 LONG 2019-01-01 → 2026-08-07 1052 / +128.10 3f0c091c…
M5 W3 2023-09-05 → 2026-09-05 870 / +153.64 d20ec8a3…

Four slices per timeframe: LONG era A (entries 2019–2023), LONG era B (2024 → 2026-08-07), W3 first half (before 2025-03-05), W3 second half. W3 overlaps era B, so the four slices are not independent; that is known and accepted for a screen.

2. The seven features (closed bars only, as of the trade's decision time)

Decision time = the trade's entry time (the entry is the open of the bar after the signal bar, so it equals the signal bar's close). A bar counts only when it had closed by then.

Daily bars: provider D1 candles (UTC days), Sunday stub bars dropped, usable from label + 1 day.

  1. Weekly trend. Weekly bars = the kept daily bars grouped into weeks ending Friday; a week is usable once its Friday bar is. State +1 when the weekly close is above its 20-week EMA (ewm(span=20, adjust=False)) AND above the close 12 weeks earlier; −1 when below both; else 0. Trade state: with / against / unclear.
  2. Four-hour trend. Provider H4 candles (UTC-aligned), usable from label + 4 h. State +1 when the close is above its 50-bar EMA AND above the close 30 bars earlier; −1 when below both; else 0. Trade state: with / against / unclear.
  3. Daily-trend strength. (Close − Close 60 kept days earlier) / ATR14, ATR14 = the simple 14-day mean of the true range, taken in the trade's direction (multiplied by −1 for a SELL). Trade state by fixed cut-offs: counter (< 0), weak (0 to 5), medium (5 to 12), strong (> 12 daily ATRs).
  4. Daily volatility state. ATR14 / Close, ranked within its own trailing 250 kept days (percentile, the current day included). Trade state: quiet (< 33%), normal, wild (> 67%).
  5. Place in the 20-day range. (Close − lowest Low of 20 kept days) / (highest High − lowest Low), mirrored for a SELL (1 − value). Trade state: near the far edge (< 1/3: buying near the 20-day low or selling near the high), middle, near the chasing edge (> 2/3: buying near the 20-day high or selling near the low).
  6. Tick-volume surge. Volume of the signal bar divided by the median volume of the same clock slot (same hour and minute, UTC) over that slot's previous 20 bars. Trade state: thin (< 0.8), usual (0.8 to 1.5), surge (> 1.5).
  7. Time to a high-impact US release (backtest/cache/calendar.parquet, currency USD, impact high, time not masked; the file starts 2023-09-01, so W3 only). Minutes from the decision time to the nearest such release. Trade state: 0–30 min before, 0–30 min after, 30–120 min before, 30–120 min after, clear (more than 2 hours either side).

Rows whose feature cannot be computed (not enough history) are counted as n/a and never flagged.

3. Rule

A screen is an in-book statistic: it cannot see the trades a change would add and it ignores the 5-bar cooldown. It can only nominate a hypothesis.

  • A state is nominated only if, on one timeframe, it holds at least 8% of the trades of BOTH books (LONG and W3) and its total R is negative in ALL FOUR slices.
  • For each nominated state the mechanism is written down before its replay.
  • At most three replays in total, on the final-veto research harness of the campaign (K1 + K2 of the twin, a mask on the final candidates before the cooldown, the shared resolver), judged by the same risk-first rule T against the shipped book: W3 R ≥ 97% of control, W3 PF ≥ control + 0.10, W3 max drawdown ≤ control, LONG R ≥ control, LONG max drawdown ≤ control, 2021 + 2022 better, HOLDOUT R ≥ control − 0.5R, the same at $0.30, at least 20 trades differ. (No neighbour-definition clause here: a pass only earns a full Stage-1/Stage-2 build, where the neighbours are tested.)
  • No nomination, no replay. No second cut-off, no other bins, no "best of seven" story afterwards. With 7 features, about 25 states and 2 timeframes, a state that is negative in four overlapping slices by luck alone is not rare; that is why a nomination is only a ticket to a replay.

4. Results

Run 2026-10-01 on the four books of section 1 (read from the campaign's exported CSVs; the feature frames are the kept Oanda cache windows: daily 1d_20150303_20260807 / 1d_20191105_20260905, four-hour 4h_20181031_20260807 / 4h_20230114_20260905, the books' own primary frames for volume, calendar.parquet for releases). Every trade's decision time equals its entry time in all four books (no entry sat behind a market closure). Cells are total R (trades); "mean" is R per trade.

4.1 M15 (daily-trend veto on): LONG 1473 trades / +223.33R, W3 824 trades / +205.12R

M15 — 1. weekly trend

State LONG trades (share) LONG R / mean W3 trades (share) W3 R / mean LONG era A (2019–2023) LONG era B (2024–2026) W3 first half W3 second half Nominated
against 78 (5%) -15.0 / -0.193 25 (3%) +3.9 / +0.157 -18.3 (58) +3.2 (20) +3.8 (7) +0.1 (18)
unclear 176 (12%) +28.1 / +0.160 101 (12%) +37.8 / +0.375 +1.9 (102) +26.2 (74) +25.4 (61) +12.5 (40)
with 1219 (83%) +210.3 / +0.173 698 (85%) +163.4 / +0.234 +53.1 (568) +157.2 (651) +47.6 (316) +115.8 (382)

M15 — 2. four-hour trend

State LONG trades (share) LONG R / mean W3 trades (share) W3 R / mean LONG era A (2019–2023) LONG era B (2024–2026) W3 first half W3 second half Nominated
against 260 (18%) +23.3 / +0.090 134 (16%) +14.8 / +0.111 +12.4 (137) +10.9 (123) +3.4 (59) +11.4 (75)
unclear 167 (11%) +16.8 / +0.100 84 (10%) +16.7 / +0.199 +0.3 (90) +16.5 (77) +1.8 (41) +14.9 (43)
with 1046 (71%) +183.3 / +0.175 606 (74%) +173.6 / +0.286 +24.1 (501) +159.2 (545) +71.5 (284) +102.1 (322)

M15 — 3. daily-trend strength

State LONG trades (share) LONG R / mean W3 trades (share) W3 R / mean LONG era A (2019–2023) LONG era B (2024–2026) W3 first half W3 second half Nominated
counter 145 (10%) -4.8 / -0.033 72 (9%) +16.9 / +0.234 -13.1 (92) +8.3 (53) +15.2 (39) +1.7 (33)
medium 586 (40%) +106.6 / +0.182 393 (48%) +94.6 / +0.241 +16.7 (199) +89.9 (387) +9.5 (162) +85.1 (231)
strong 17 (1%) +4.6 / +0.268 10 (1%) +4.8 / +0.483 -0.3 (7) +4.8 (10) -1.1 (1) +5.9 (9)
weak 725 (49%) +117.0 / +0.161 349 (42%) +88.8 / +0.254 +33.5 (430) +83.5 (295) +53.1 (182) +35.7 (167)

M15 — 4. daily volatility state

State LONG trades (share) LONG R / mean W3 trades (share) W3 R / mean LONG era A (2019–2023) LONG era B (2024–2026) W3 first half W3 second half Nominated
normal 445 (30%) +54.8 / +0.123 299 (36%) +77.6 / +0.259 -11.8 (194) +66.6 (251) +6.7 (118) +70.8 (181)
quiet 373 (25%) +64.1 / +0.172 152 (18%) +51.7 / +0.340 +24.2 (235) +39.9 (138) +38.9 (109) +12.8 (43)
wild 655 (44%) +104.4 / +0.159 373 (45%) +75.9 / +0.203 +24.3 (299) +80.1 (356) +31.1 (157) +44.8 (216)

M15 — 5. place in the 20-day range

State LONG trades (share) LONG R / mean W3 trades (share) W3 R / mean LONG era A (2019–2023) LONG era B (2024–2026) W3 first half W3 second half Nominated
middle 420 (29%) +54.9 / +0.131 230 (28%) +32.3 / +0.140 +25.3 (217) +29.7 (203) +30.9 (107) +1.4 (123)
near the chasing edge 835 (57%) +143.5 / +0.172 483 (59%) +147.1 / +0.305 +8.8 (400) +134.7 (435) +40.6 (227) +106.5 (256)
near the far edge 218 (15%) +24.8 / +0.114 111 (13%) +25.7 / +0.231 +2.6 (111) +22.2 (107) +5.3 (50) +20.4 (61)

M15 — 6. tick-volume surge

State LONG trades (share) LONG R / mean W3 trades (share) W3 R / mean LONG era A (2019–2023) LONG era B (2024–2026) W3 first half W3 second half Nominated
surge 556 (38%) +83.1 / +0.149 226 (27%) +45.7 / +0.202 +45.8 (355) +37.3 (201) +22.5 (110) +23.1 (116)
thin 220 (15%) +15.6 / +0.071 125 (15%) +28.5 / +0.228 -10.4 (99) +26.0 (121) +11.5 (48) +17.0 (77)
usual 697 (47%) +124.7 / +0.179 473 (57%) +130.9 / +0.277 +1.4 (274) +123.2 (423) +42.7 (226) +88.3 (247)

M15 — 7. time to a high-impact US release (W3 only)

State LONG trades (share) LONG R / mean W3 trades (share) W3 R / mean LONG era A (2019–2023) LONG era B (2024–2026) W3 first half W3 second half Nominated
0–30 min after 42 (5%) +8.9 / +0.212 +4.7 (27) +4.2 (15)
0–30 min before 26 (3%) +8.7 / +0.336 +7.8 (18) +0.9 (8)
30–120 min after 75 (9%) +28.4 / +0.378 +11.7 (51) +16.6 (24)
30–120 min before 22 (3%) +16.1 / +0.732 +12.9 (18) +3.2 (4)
clear 659 (80%) +143.0 / +0.217 +39.6 (270) +103.5 (389)

4.2 M5: LONG 1052 trades / +128.10R, W3 870 trades / +153.64R

M5 — 1. weekly trend

State LONG trades (share) LONG R / mean W3 trades (share) W3 R / mean LONG era A (2019–2023) LONG era B (2024–2026) W3 first half W3 second half Nominated
against 258 (25%) +11.0 / +0.043 202 (23%) +6.6 / +0.033 +9.4 (69) +1.6 (189) +3.8 (10) +2.8 (192)
unclear 84 (8%) +6.3 / +0.075 88 (10%) +24.6 / +0.279 -7.5 (30) +13.8 (54) +8.2 (12) +16.4 (76)
with 710 (67%) +110.8 / +0.156 580 (67%) +122.5 / +0.211 -12.3 (159) +123.1 (551) +4.7 (114) +117.8 (466)

M5 — 2. four-hour trend

State LONG trades (share) LONG R / mean W3 trades (share) W3 R / mean LONG era A (2019–2023) LONG era B (2024–2026) W3 first half W3 second half Nominated
against 332 (32%) +3.0 / +0.009 255 (29%) +11.2 / +0.044 -5.5 (98) +8.4 (234) -0.9 (35) +12.2 (220)
unclear 126 (12%) +13.7 / +0.109 105 (12%) +17.8 / +0.170 +0.6 (28) +13.1 (98) +0.8 (13) +17.0 (92)
with 594 (56%) +111.4 / +0.188 510 (59%) +124.6 / +0.244 -5.5 (132) +116.9 (462) +16.9 (88) +107.7 (422)

M5 — 3. daily-trend strength

State LONG trades (share) LONG R / mean W3 trades (share) W3 R / mean LONG era A (2019–2023) LONG era B (2024–2026) W3 first half W3 second half Nominated
counter 296 (28%) +0.7 / +0.003 241 (28%) -2.6 / -0.011 +8.8 (87) -8.0 (209) +3.7 (18) -6.3 (223)
medium 367 (35%) +77.3 / +0.211 318 (37%) +78.4 / +0.247 -1.2 (49) +78.4 (318) +6.9 (53) +71.6 (265)
strong 14 (1%) +16.6 / +1.186 13 (1%) +15.6 / +1.199 +1.0 (1) +15.6 (13) +0.0 (0) +15.6 (13)
weak 375 (36%) +33.5 / +0.089 298 (34%) +62.2 / +0.209 -19.0 (121) +52.5 (254) +6.2 (65) +56.1 (233)

M5 — 4. daily volatility state

State LONG trades (share) LONG R / mean W3 trades (share) W3 R / mean LONG era A (2019–2023) LONG era B (2024–2026) W3 first half W3 second half Nominated
normal 302 (29%) +33.2 / +0.110 305 (35%) +45.6 / +0.150 -0.8 (63) +34.0 (239) -3.0 (42) +48.6 (263)
quiet 114 (11%) +18.8 / +0.165 72 (8%) +17.9 / +0.249 +3.0 (45) +15.8 (69) +10.9 (37) +7.0 (35)
wild 636 (60%) +76.1 / +0.120 493 (57%) +90.1 / +0.183 -12.6 (150) +88.8 (486) +8.8 (57) +81.3 (436)

M5 — 5. place in the 20-day range

State LONG trades (share) LONG R / mean W3 trades (share) W3 R / mean LONG era A (2019–2023) LONG era B (2024–2026) W3 first half W3 second half Nominated
middle 323 (31%) +11.8 / +0.037 271 (31%) +18.8 / +0.069 -6.8 (76) +18.6 (247) -1.8 (34) +20.6 (237)
near the chasing edge 439 (42%) +101.5 / +0.231 380 (44%) +115.0 / +0.303 -2.6 (103) +104.1 (336) +17.2 (77) +97.8 (303)
near the far edge 290 (28%) +14.8 / +0.051 219 (25%) +19.9 / +0.091 -1.0 (79) +15.9 (211) +1.4 (25) +18.5 (194)

M5 — 6. tick-volume surge

State LONG trades (share) LONG R / mean W3 trades (share) W3 R / mean LONG era A (2019–2023) LONG era B (2024–2026) W3 first half W3 second half Nominated
surge 479 (46%) +46.2 / +0.097 334 (38%) +65.3 / +0.195 -5.9 (176) +52.1 (303) +13.2 (64) +52.1 (270)
thin 152 (14%) +9.3 / +0.061 134 (15%) +18.1 / +0.135 -8.0 (22) +17.3 (130) -0.7 (7) +18.8 (127)
usual 421 (40%) +72.6 / +0.172 402 (46%) +70.3 / +0.175 +3.4 (60) +69.2 (361) +4.2 (65) +66.1 (337)

M5 — 7. time to a high-impact US release (W3 only)

State LONG trades (share) LONG R / mean W3 trades (share) W3 R / mean LONG era A (2019–2023) LONG era B (2024–2026) W3 first half W3 second half Nominated
0–30 min after 46 (5%) -3.8 / -0.082 -0.6 (15) -3.2 (31)
0–30 min before 18 (2%) +1.8 / +0.098 +0.7 (5) +1.1 (13)
30–120 min after 87 (10%) +32.8 / +0.377 +8.3 (31) +24.4 (56)
30–120 min before 18 (2%) +3.4 / +0.187 +2.7 (7) +0.7 (11)
clear 701 (81%) +119.5 / +0.171 +5.6 (78) +114.0 (623)

4.3 Verdict

Nominated states: none. No state on either timeframe holds 8% of both books and loses in all four slices. By the rule of section 3 there is no replay and nothing ships.

What the tables do say (in-book description, not a decision):

  • M15, after the daily-trend veto, has no losing pocket left in these seven views. The only clearly negative cell is trades against the weekly trend in 2019–2023 (−18.3R over 58 trades), 5% of the book and positive since 2024. Trades with the four-hour trend earn +0.18R to +0.29R each against +0.09R to +0.11R against it: weaker, still positive.
  • M5 earns its R where everything already agrees. With the weekly trend +0.16R to +0.21R per trade, against it +0.03R to +0.04R; with the four-hour trend +0.19R to +0.24R, against it +0.01R to +0.04R; when the 60-day move runs against the trade, 0.00R (about 28% of the book). Those against-the-tide M5 trades are flat, not losing, which is the same picture the formal daily-trend test gave on M5: removing them removes no loss, and it made the worst losing run deeper. They stay.
  • M5's profit sits near the 20-day extreme in the trade's direction (+0.23R to +0.30R per trade against +0.04R to +0.09R elsewhere). This repeats the 2026-08-04 finding that the best trades are the stretched runs: a "do not chase" filter would cut the part of the book that pays.
  • Volume and news timing separate nothing reliably. A volume surge is the best state in 2019–2023 on M15 and the middle state since; trades 30 to 120 minutes after a major US release look good on both timeframes (+0.38R each) on 75 and 87 trades, and trades in the half hour after a release are slightly negative on M5 (−3.8R over 46 trades, 5% of the book, below the 8% bar). Neither is a rule.

So the daily chart was the one piece of missing context that paid, on M15. Weekly and four-hour context add nothing on top of it there, and nothing on M5 that survives the rule.