CuPy from pip had never compiled a kernel on the dev box (no CUDA_PATH for the
splayed CUDA-12 wheel layout); the probe swallowed the error and every book to
date was built on the NumPy path under a "GPU backtester" name.
- backtest/gpu_kernels.py (new): lazily probed, never-silent BACKEND state
(CUDA_PATH pointed at the nvidia-cuda-runtime-cu12 header wheel; the probe
compiles a kernel and keeps the failure reason), --backend auto|gpu|cpu and
FB2_BACKEND, and three CUDA kernels compiled with --fmad=false: ewm_batch
(the _gpu_ewm_alpha recurrence, one thread per series), roll_mean_var
(pandas 3.0.2 roll_mean/roll_var replica, Kahan + Welford + unstable
recompute), resolve_trades (trailing and static exits, one thread per trade).
- gpu_backtest.py: precompute_indicators and simulate_trades dispatch to the
GPU twins or to the untouched NumPy paths; caches are host NumPy and device
memory is released before the replay workers fork; the Backend: line names
the device or the fallback reason.
- test_all.py Module 36 (7 tests): a visible CUDA device must never fall back
silently; kernels vs their CPU references on synthetic edge cases; backend
selection; import guards (cupy banned from the live roots). Suite 536.
- CLAUDE.md, instructions_for_backtest.txt, research note with the evidence.
Verification: all five reference MD5s reproduced on the GPU and with
--backend cpu (ten runs, GPU cache fallbacks 0); precompute GPU == CPU on all
42 cached windows (2.34 M bars); suite green with GPU, with
CUDA_VISIBLE_DEVICES="" and with FB2_BACKEND=cpu; --workers 1 and --record
pairs equal. The per-bar replay stays the live Python signal code on CPU
workers by design; wall time is roughly unchanged (precompute ~3x faster).
Agent panel v2: personas return p_win anchored to the book base rates with a
code-computed decision context; verdict = equal-weight mean (verdict_source
"mean"); only the macro persona runs live (agent_panel.personas ["macro"]);
prompt version and exact payloads logged; offline replay harness
scripts/panel_replay.py; base-rate table agent_panel_base_rates.json
(scripts/build_panel_base_rates.py). Backtester `strategy` column now carries
the generator's own stamp (label-only; five reference books re-pinned, all
62 other columns identical).
News corpus (after evaluating the owner's ML-NEWS scikit model against
DeepSeek: golden set 39/78 vs 78/78, out-of-sample agreement 64% against a
90% self-agreement ceiling; verdict: fallback only, not a replacement):
- NewsSentiment.xau_score + archive column xau_score keep DeepSeek's gold
score for every headline (930 of the first 1,546 DeepSeek rows had none);
an existing newsdata.db gains the column in place on first open
- headlines naming no tracked currency are scored and archived
(currencies_json []) but never returned to the platform
- _apply_llm_sentiment sends every new headline in its cycle, 25 per call;
a failed call ends the cycle's sending
Dashboard publisher never writes a database: main() calls
app_config.set_db_writes_for_process(False), honoured by signal_log_enabled()
and news_archive_enabled() (owner decision: the bot is the single writer).
Test suite: the hermetic app config now points both writers at a
suite-private temp dir — the RSS fixture tests were writing fixture
headlines into the real logs/newsdata.db — with a Module 32 test pinning it.
529 tests.
Same facts, new shape: quick start; the scan pipeline from trigger to
delivery to logging (A to Z) with the real thresholds and gate order;
a file-by-file guide with the import graph; data providers; config;
testing; the nine change rules; the validated per-timeframe overrides;
reference books with MD5s and the kept cache windows; a traps list;
and the decision ledger condensed to one row per dated decision with a
pointer to its research report.
Every claim was checked against the code by a reviewer pass; the
mechanism invariants the old file carried (MACD always computed, the
_-prefixed convention, PB/CP force grade B, penalties hit confidence,
gate 6b label-hour vs close-hour, the news-archive hook position, the
ordered cooldown pass, engine_fingerprint split rule, oppose guard,
holiday blindness) are kept. The previous 58 KB version is in git
history at b0b71e1.
Discussion-first cleanup of backtest/ and scripts/. Nothing in the signal
core or the backtester changed.
Removed (last commit containing them: 931b265):
- backtest/diagnose_filters.py + its two Module 18 tests
- scripts/shadow_scan.py (Phase 4 shadow scan, never run)
- scripts/backfill_news_archive.py, scripts/migrate_signal_log_news.py
(one-time scripts; the interning test keeps its live-path half, the
prune test goes)
Kept: the replay-recording stack (replay_record.py, replay_scorecard.py,
ff_calendar_scraper.py, cache/calendar.parquet, Module 33).
Local gitignored data was moved to ~/Development/ForexBot2_trash_2026-09-15
(346 MB) for the owner to delete: all 35 pinned backtest/*.csv books, the
three research output folders, the BTC/ETH cache snapshots, the Tiingo
parquets and 13 uncited Oanda windows. The three research-folder ignore
rules are dropped with the folders.
Verification: all five gate books regenerate offline from the pruned cache
with the recorded MD5s (M5 W2 29120a1e, M15 W2 d185f099, M15 OOS 6f8f24b4,
H1 W3 3b45d12b, H4 W3 e75adae1; NumPy CPU, 0 cache fallbacks). Suite
506 passed, 0 failed, 33 modules.
Owner decision 2026-09-15. Gone: market_outlook.py, the post-H1-scan Telegram
outlook DM, main.py --outlook, the outlooks table DDL/migration/writer in
signal_log.py, the market_outlook config block + MARKET_OUTLOOK_ENABLED, and
test Module 26. Last commit containing the feature: 2836024.
- app_config: "market_outlook" stays in _KNOWN_TOP as a TOLERATED, ignored
legacy key (oanda.account_id precedent) with one warning, so a live
config.json that still carries the block keeps booting.
- signal_log: existing DBs keep their orphan outlooks table and rows (no
DROP TABLE); fresh DBs never create it.
- tests: Module 26's 4-thread concurrent legacy-DB migration test (the
suite's only one) moved to Module 31; new test for the tolerated block.
519 -> 509 tests, 34 -> 33 modules, 0 failed.
- docs: CLAUDE.md usage/config/file-map lines and a dated state note.
Verification: W2 books (2024-09-05 -> 2026-09-05, $0.65, NumPy CPU, 0 cache
fallbacks) byte-identical before/after — M5 29120a1ee32979431d062f210d41a9d1
(781 / +146.07R / PF 1.52), M15 d185f099482a07f29502cb5a371a9bbf
(931 / +146.14R / PF 1.43).
Signal path
- A shipped setup always names a direction: grade C needs confidence
>= 45 (gate 6c grants B at >= 45) while BUY/SELL needs >= 50, so C/B
reports reached the alert gate as NEUTRAL with a full setup. The
backtester traded them (the replay never reads final_signal) while
Telegram rejected NEUTRAL first — 5/91 trades, +3.7R of +22.2R, in
the H1 12-month book. _attach_core_setup stamps the setup's direction
and repairs the pre-gate action text; no-setup reports stay NEUTRAL.
- analyze_pair fails the scan when ANY required frame is missing
("DATA UNAVAILABLE:", scan_kind='failed'); a lost confirmation frame
used to run with an unearned -15 MTF penalty.
Outcomes and labels
- advisory_outcome.resolve_locked_floor: a gap the database cannot fill
resolves at the locked trailing stop (GAP_LOCKED, result.floor) when
the stop had crossed the entry; the rebuild fires on both gap reasons
(rows 915/920: +1.14R / +0.84R floors; resolved sum -2.46R -> -0.48R).
- label_scan_outcomes: SCAN_LABEL_MAX_AGE_H = 72 h, stale rows never
anchor the shared bar stream, labeled_at bumped on re-examination,
_stream_since warns at its row cap (the Labor-Day pile would have
blinded scan_outcomes around 2026-09-14).
Providers and scheduling
- The daily settlement pause (Mon-Thu 17:00 New York) is a scheduled
closure for _has_latest_closed_bar (whole-span label walk-back) and
_is_data_fresh (closed seconds are not age); bot and dashboard skip
scans whose just-closed label sits in the pause (61 false stale rows
in three nights).
- fetch_range retries a failed page in place (three failures on one
window raise) and logs an ERROR for interior gaps of >3 unscheduled
bars.
- pd.Timestamp.utcnow() replaced in providers/base.py and engine.py;
requirements pin pandas<4, numpy<3.
- telegram_bot: boundary loops share one scan per bar with the breach
handler (_remember_scan keys on the analysed bar; breach defers 20 s
after an M15 boundary); the rotating file handler attaches in main()
only (the test suite used to write into logs/telegram.log).
- dashboard_publisher: catch_up_timeframes re-scans boundaries missed
while a cycle overran or the process was down.
Backtester and engine guards
- _replay_span returns the per-span GPU cache-miss count; run_backtest
sums, logs it (must be 0) and writes cache_fallbacks to the manifest.
- regime_weight_table raises on a partial regime_weights_override.
Verification: 519 tests; M5/M15 2-yr, M15 OOS, H1/H4 3-yr books
byte-identical before and after; live signals.db untouched (checked on
a copy). CLAUDE.md carries the dated entry and the blocked-hours union
semantics that every book was validated under.
Claude-Session: https://claude.ai/code/session_0139xz1J6YccndA2be6yFnEo
backtest/research/cycle_2026-09-10.md: pre-registered rules, HEAD baseline
on the 3-year window (three pinned MD5s reproduced), cost ladder and
concentration facts, bucket screen (one cycle-2 candidate: M15 grade-A
SELLs), C1 H1 divergence->0 REJECTED, C2 M15 RSI-multiplier ladder =
stable plateau, seeding-depth measurement and the two shipped parity
fixes. CLAUDE.md: live fetch depth, backtester pad, 501 tests, the
2026-09-10 state entry; corrected the docs note (no PDF builds exist)
and recorded pyarrow as a dev-venv-only dependency.
Claude-Session: https://claude.ai/code/session_01NQDjW8rXYrJR4JDWMWKUij
Owner-approved 2026-09-09 after the pre-registered weight test
(backtest/research/weight_test_2026-09-09.md). M15-only
technical.regime_weights_override = the shipped table with rsi x 0.5
(0.5 trending family / 0.75 ranging / 0.5 transitioning), derived in the
factory and pinned literally by the Module 9 equivalence test; a Module 33
test locks that M5/H1/H4 carry no override.
Evidence, 2-year Oanda window 2024-09-05..2026-09-05 at $0.65: 793/+117.62R
PF 1.41 -> 931/+146.14R PF 1.43 (fixed halves +27.9/+0.6R vs control); OOS
2024-06-01..2025-03-30 300/+40.59R PF 1.38 -> 355/+60.67R PF 1.49; 3-year
+151.90R -> +186.61R (max DD 15.5 -> 19.3R, 2026 -4R). Parity: the factory
run with no --set flag reproduces the validated candidate books byte-for-
byte (2-yr MD5 d185f099..., OOS 6f8f24b4...). M15 canonical book re-pinned
to backtest/oanda_m15_2yr_065_20260909.csv. Docs carry a one-line note.
Claude-Session: https://claude.ai/code/session_01Nc9Ninzyi18QkWezrxeBvB
First use of the --record evidence: a drift-adjusted per-bar screen ranked
regime-multiplier candidates, the top 3 per timeframe went through the full
2-year replay against same-harness controls plus the 2024-06..2025-03 OOS
window under a rule written before any run. M5: all three MACD candidates
fail (-10..-46R). M15: ema_trend x2 and divergence->0 (trending) fail;
RSI x0.5 in every regime passes every check (2-yr +117.6 -> +146.1R PF 1.43,
OOS +40.6 -> +60.7R PF 1.49, 3-yr +151.9 -> +186.6R) with the caveats recorded
(2026 -4R, 3-yr max DD 15.5 -> 19.3R, added trades mostly B-grade BUYs).
Nothing shipped; the passing candidate is an owner decision.
Claude-Session: https://claude.ai/code/session_01Nc9Ninzyi18QkWezrxeBvB
Adds an opt-in recorder to the backtester so a replay keeps the evidence each
trade setup was decided on: every indicator's vote, value, confidence, base
weight, regime multiplier, effective weight and contribution on the primary and
both confirmation timeframes, penalties, the setup and outcome, and the Forex
Factory calendar events around the trade (4 h before entry to exit, all-day rows
on the New-York date). --record-bars adds one float32 row per analyzed bar with
the same votes plus 1 h/4 h/24 h forward moves in ATRs. Output is Parquet in a
folder per run with a manifest carrying the engine/gate fingerprints, the regime
weight table in force and the book MD5.
Nothing in the signal core changes: the trade book is byte-identical with
recording on or off (control-vs-recorded MD5 equal on both 2-year runs, M5
781 / +146.07R / PF 1.52, M15 793 / +117.62R / PF 1.41 at $0.65). Module 33
adds 12 tests (498 total): snapshot reproduces the engine's composite, absence
is no vote, calendar join edges/phases/holiday, write/load round-trip,
_replay_span parity with recording on, CLI flags, import and source guards.
Also: backtest/replay_scorecard.py (first-look scorecards, calendar buckets,
in-book and per-bar what-ifs, --check invariants), scripts/ff_calendar_scraper.py
restored from git history (the only way to regenerate the gitignored calendar
file), report in backtest/research/replay_recording_2026-09-09.md.
Claude-Session: https://claude.ai/code/session_01Nc9Ninzyi18QkWezrxeBvB
Headlines only ever survived as a side effect of signal logging, and
logs/signals.db was rotated around 2026-09-04 — destroying the 3-week,
4,054-headline corpus news_sentiment_value_2026-09-04.md was written on.
logs/ is gitignored, so git could not recover it and nobody noticed for
four days. There is no way to backfill a fetch window once it passes.
news_archive.py writes to its OWN database (logs/newsdata.db) so a
signal-log rotation cannot take the news with it. One row per distinct
headline (sha256 key) with the text, description, link, publisher,
publish time, the score, and — new — which scorer produced that score.
Provenance is the load-bearing part. DeepSeek scores 78/78 on the
2026-08-06 golden set; the keyword scorer it overrides scores 24/78
(31%), and that is what silently stands on no key, disabled, daily cap,
timeout, HTTP/parse failure, or the >25-new-headlines-per-cycle batch
overflow. Until now both landed in the same column, indistinguishable,
so a model trained on the mix would inherit the keyword scorer's errors.
NewsSentiment.score_source now records it, mirrored into signals.db via
signal_log._headline_to_dict, and teacher_model/teacher_prompt_sha1 keep
labels made under a changed prompt from being pooled with older ones.
Writes are one atomic UPSERT per row, upgrade-only: an llm score always
replaces a keyword one (the batch-overflow tail gets labelled on the
next window in a long-running process — verified 13/13), a keyword score
never overwrites an llm one, and re-seeing a headline always counts. A
read-then-write pair lost increments when the bot, the dashboard
publisher and the tests first saw a headline together, and the WAL
switch itself needed _with_lock_retry; both are regression-tested with
three concurrent writers.
The hook sits at the end of _fetch_news_sentiment_raw, BEFORE
_is_neutral_headline runs in get_fundamental_context. Archiving
downstream of that ±0.1 filter is why signals.db holds almost no neutral
headlines and would have taught a trained scorer that "neutral" barely
exists.
scripts/backfill_news_archive.py seeds it from both DB shapes on disk,
read-only and idempotent, recomputing provenance by comparing each
stored score against the keyword scorer (neither source recorded it):
1,016 distinct headlines, 846 DeepSeek-labelled, growing ~1,400/week.
Nothing in the signal core changed. Both news gating flags stay False,
no gate reads any of this, and the backtester never builds a NewsEngine
so replays never archive (test-locked). 468 -> 483 tests, including a
new guard that AST-walks live code for sklearn/scipy/lightgbm imports —
that check did not exist before.
Do not train yet: ~800 labels gives a holdout too small to grade
(SE +/-0.032). Target is ~4,000 deduplicated headlines over >=3 weeks.
Claude-Session: https://claude.ai/code/session_01Bzv4ETVapFNauFDQht7QtA
Delivery-side fixes from the 2026-09-08 three-day signal-log + telegram-log
review. Nothing in the signal core changes: the 3-year H1 and M15 books at
$0.65 were regenerated before/after and are identical on every trade column
(H1 253 / +70.02R / PF 1.98, M15 1092 / +151.90R / PF 1.38); only the
exported `warnings` text differs, on 50/253 H1 and 358/1092 M15 trades.
- signal_aggregator: step 7 moves into `_attach_core_setup`; a final pass
after the PB/CP archetypes (`_drop_stale_spread_gate_vetoes`) moves a core
generator's "SPREAD GATE: ... costs would eat the edge" veto to
`cleaned_warnings` whenever a later generator shipped a setup. The
2026-09-08 07:45 UTC A-grade alert carried that line beside a $26.49 TP1;
a fifth of H1 and a third of M15 A-grade alerts had been doing the same.
The prefix is now the `_SPREAD_GATE_PREFIX` constant shared by both
emitters and both cleanups.
- telegram_bot: alerts require a trade setup (`_is_alert_candidate`, shared
by the scan and breach paths; no-setup candidates log
`alert skipped: ... (gate reason)`). The 07:50 UTC B-grade SELL with no
setup had gone out under the same warning.
- telegram_bot: every delivered alert / breach alert / violent-move heads-up
logs an `... sent:` INFO line with the signal-log row id (only failures
were logged before, so the review had to reconstruct sends from panel
rows). Send fan-out is one `_broadcast` helper for both paths.
- telegram_bot: `telegram.log` rotates daily x 30 instead of 3-hourly x 240
(old `_HH` files on the live box need one manual delete).
- signal_log / CLAUDE.md: `cleaned_warnings_json` is no longer a CP-only
marker; docs say so.
- test_all: 468 tests (+2): stale-veto cleanup incl. the PB-shaped case and
the both-vetoed case, and the alert-candidate rule with its skip line; the
rotation test asserts MIDNIGHT / interval 86400 / 30 backups.
Claude-Session: https://claude.ai/code/session_018xY4NPPrx6NJoG6QWHzwpd
2026-09-07 signal-log inspection follow-up — three fixes that do not depend on sample size.
- providers/base.py: the signal-side 10σ outlier filter is now a bad-print filter. A >10σ
close-to-close bar is dropped only when the NEXT close snaps back ≥80% of the way
(OUTLIER_SIGMA / OUTLIER_REVERT_FRACTION); persisting moves and the newest bar are kept.
The unconditional rule deleted ~36 real M5 news bars/yr in the backtester (whole-frame σ)
and a different ~24/yr live (fetch-window σ) — e.g. the 2026-09-04 12:30 UTC NFP bar
vanished from M5+M15 and the 12:35 scan was a byte-identical repeat of 12:30. 3-yr Oanda
drops are now M5 2 / M15 1 / H1 0.
- providers/oanda.py: transient 401/5xx/network faults retry along RETRY_DELAYS_S=(2,5,10)s
under RETRY_BUDGET_S=45s. 11 boundary scans in two days had lost their primary frame at
xx:00:22–:26 UTC; one 2 s retry could not outlast the top-of-hour blip.
- signal_aggregator/_log_signal: a scan whose primary fetch failed logs scan_kind='failed'
(price 0, no signal); analyze_signal_log.parity_report skips it like 'blocked'.
- test_all.py: 466 tests (+6): retry ladder walks all delays, 5xx blip recovers on the third
attempt, retry budget stops the ladder, persisting/newest/partially-retraced 10σ bars kept,
failed-scan row kind, parity skip.
- CLAUDE.md: cleaning rule, retry ladder, failed scan kind, dated current-state note.
Book impact at $0.65 (rule 1 still passes everywhere, both halves positive):
12mo M5 557/+112.21/PF1.56 → 560/+108.81/1.54 M15 397/+73.93/1.53 → 398/+74.61/1.53 H1 identical
3yr M5 857/+161.3/1.53 → 870/+153.6/1.50 M15 1086/+148.7/1.37 → 1092/+151.9/1.38
H1 255/+70.5/1.98 → 253/+70.0/1.98 H4 113/+22.7/1.63 → 114/+21.7/1.59
Only trades on the previously-deleted event bars move; the pinned M5/M15 CSVs are no longer
byte-reproducible and should be regenerated at the next re-pin.
Claude-Session: https://claude.ai/code/session_01NDnBxJ2y8GRUM6ZNFJs1hZ
- providers: _to_oanda_instrument / _to_tiingo_ticker / _to_td_symbol no longer
guess a wire symbol for pairs that are not registered — the registries are
the only source, so an unknown symbol raises the provider's DataIntegrityError
instead of silently fetching something else. Every scanned symbol is validated
against PAIR_INFO at import, so nothing reachable changes.
- main / telegram_bot _fmt_price: the sub-$10 five-decimal branch could never
fire for gold.
- news_engine._parse_pair: BASE/QUOTE only; the Yahoo "=X" fallback was
unreachable (its test goes with it).
- dashboard_publisher.event_to_dict: drop minutes_until (frozen at publish
time, misleading) and is_high_impact — dashboard.js never read either.
- static/dashboard.html: unused id="cal-table".
Tried and reverted: trimming the LogRow fields nothing in analyze_signal_log
reads — the tests read them to verify the loader round-trips those DB columns,
so they stay.
Suite: 460 passed, 0 failed.
Claude-Session: https://claude.ai/code/session_0156YhogLytzj9P2iH1skGbL
Owner directive 2026-09-06: Dukascopy and similar research material is irrelevant to
the running system; keep only code and files with a role in system functioning.
Deleted (all in git history before this commit):
- backtest/research/ tooling and data: book_report, book_diff, burst_map,
burst_precursors, burst_indicator_map, open_trade_bursts, calendar_impact,
watch_rule_eval, sizing_model, commit_ladder, llm_news_eval + llm_news_golden.csv,
ff_calendar_scraper, duka_spread + duka_spread_by_hour.csv (the Dukascopy spread
study), feed_divergence_oanda_vs_tiingo.csv, the panel dry-test JSONL, __init__.py
- scripts/feed_divergence_study.py, scripts/signal_parity_study.py (completed studies;
their reports keep the results)
- docs/architecture.png + architecture_diagram.py (developer illustration)
- docs/data_source_change_test_plan.md (Phase 4 shadow scan stays noted in CLAUDE.md)
Kept: the campaign .md reports in backtest/research/ as the decision ledger CLAUDE.md
cites (rule 1 evidence); scripts/shadow_scan.py (pending operational step);
backtest/diagnose_filters.py (test-locked backtester diagnostic).
Local, untracked material removed on this box: .superpowers/, walkforward/, baseline/,
the commit-ladder and LLM-eval result folders, the Dukascopy boundary CSV; the
sizing_2026-09 folder is pruned to the four 3-year baseline books CLAUDE.md names.
.gitignore no longer lists the removed research outputs; the backtest instructions
and CLAUDE.md no longer point at deleted tools.
Suite: 461 passed, 0 failed (nothing imported the removed files).
Claude-Session: https://claude.ai/code/session_0156YhogLytzj9P2iH1skGbL
- test count 461 / 31 modules
- main.py --timeframe accepts M5/M15/H1/H4 and takes confirmation TFs from the
pair factory's map
- point the dual-cost note at the Dukascopy study that produced the $0.65
figure (duka_spread.py is evidence, not dead code)
- current-state note listing what the cleanup removed
Claude-Session: https://claude.ai/code/session_0156YhogLytzj9P2iH1skGbL
The candle provider needs only oanda.api_token + base_url; account_id belonged to the
auto-execution daemon removed 2026-09-03 (31002cb). Removed from config.json.example and
the unused oanda_account_id() accessor deleted; the key stays in _KNOWN_OANDA so existing
config.json files keep loading. CLAUDE.md env-var list updated. Test locks all three.
Suite: 464 passed.
Claude-Session: https://claude.ai/code/session_01RWV2yLt3YXHHGVYHSQU2EE
- test count no longer hardcoded (463 / 31 modules as of 2026-09-06)
- main.py --outlook, backtest --set / --export-features listed
- delivery surfaces include the web dashboard; SignalReport consumers include the publisher
- scikit-learn / lightgbm are research-only venv installs, never requirements
- rule 1: owner preference (2026-09-05) that backtests go back at most 3 years, with the
3-year baseline books and where they live
- "Current state notes" header un-dated; the 2026-08-14 canonical-book line marked superseded
by the 2026-09-03 book (M15 reference unchanged since 2026-08-04)
- docs: PDF builds noted as behind the HTML guides
Claude-Session: https://claude.ai/code/session_01RWV2yLt3YXHHGVYHSQU2EE
New additive columns on every logged scan: bar_ts (primary closed-bar label, indexed with
timeframe), engine_fingerprint (hash of the scoring config incl. the regime weight table in
force - SignalAggregator.engine_config_fingerprint, twin of the gate fingerprint), code_version
(short git SHA), penalty_tags_json / penalty_total / momentum_fading / exhaustion_count, and
post-hoc watch_level / watch_kinds / watch_line written by violent_watch.attach_watch via
signal_log.update_watch. Each indicators_json entry now also carries eff_weight (base weight x
regime multiplier, the weight the composite actually used) and wscore.
engine: IndicatorSignal.effective_weight / weighted_score are stamped inside _compute_composite
(record-keeping only); regime_weight_table(config) is the single source of truth for the
multiplier table (override honoured). Composite output byte-identical (120-report hash proof).
New table scan_outcomes: model-free forward moves per scan (fwd_1h/4h/24h in $ and ATRs,
mfe/mae over 24h, burst_1h, bars_used, complete) measured on the stored candle stream with the
advisory model's continuity rule; labelled opportunistically at the end of log() (time-boxed,
throttled, idempotent) - no daemon; scripts/label_scan_outcomes.py drives the backlog
(--dry-run is read-only). analyze_signal_log --scan-outcomes: forward move + hit rate by
TF x signal/grade with a NEUTRAL baseline and burst rate by watch level. Readers dedup by the
first row per (timeframe, bar_ts).
Module 31 (10 tests). Suite: 463 passed. Backtester untouched (test-locked).
Claude-Session: https://claude.ai/code/session_01RWV2yLt3YXHHGVYHSQU2EE
New violent_watch.py assesses each scan from the cached Forex Factory week events, the
UTC clock and the report's own indicator states, and stamps SignalReport.watch_line
post-hoc (telegram alerts, CLI, dashboard). A standalone heads-up goes to the owner on
the fastest scanned timeframe when the level is "hot", once per UTC hour per trigger
set, capped per day. Config block `violent_watch` (enabled False, alert_line, headsup,
max_headsup_per_day, headsup_min_level); env VIOLENT_WATCH_ENABLED.
Rule chosen from a 3-year bar-by-bar sweep (backtest/research/watch_rule_eval.py,
recorded in sizing_campaign_2026-09.md): hot = high-impact USD release within 30 min
(before or after) OR a hot clock slot (weekday 12-14 UTC, 01 UTC, Sunday 22-23 UTC)
with at least one price-extreme reading (outside Bollinger bands, RSI <30/>=70, outside
Keltner); ~8.6% of bars, next-bar odds 3-5x normal, ~4 hot hours/day. watch = clock
slot alone or two price-extreme readings (~2x). Context readings (ADX rising, extreme
volatility, transitioning, MACD acceleration) are phrases only - letting them raise the
level flagged 27% of bars at ~2x and would have sent ~13 heads-ups/day (rejected).
Direction sentence: DOWN when bearish readings dominate (70-78% on both TFs), UP on M5
only. Never a gate or exit: every open-trade protection tested lost money.
Test-locked out of analyze_from_dataframes / engine / backtester. Module 30 (18 tests).
Suite: 453 passed. Guide PDFs not rebuilt.
Claude-Session: https://claude.ai/code/session_01RWV2yLt3YXHHGVYHSQU2EE
Walk-forward "second opinion" stake multiplier (meta-labelling on 65 decision-time
features, 3yr Oanda books @ $0.65) FAILED the pre-registered rule on M5 (clean null)
and M15 (lift inside the label-shuffle null, explained by BUY-vs-SELL). Nothing that
changes numbers ships; the 12-month M5 book re-run is byte-identical to the pinned one.
Behaviour-neutral plumbing:
- position_sizing.build_features (65 features; multiplier path returns 1.0, no artifact)
- SignalReport.penalty_tags / penalty_total / momentum_fading (record-keeping)
- TradeSetup.stake_multiplier = None (reserved)
- gpu_backtest --export-features (default export stays the pinned 63 columns; extras
forced to the tail even when the first trade lacks an indicator)
Research tools (backtest/research): book_report, burst_map, burst_precursors,
open_trade_bursts, calendar_impact, burst_indicator_map, sizing_model, watch_rule_eval.
Findings recorded in backtest/research/sizing_campaign_2026-09.md: M5 stretch measures
are positive in both halves; every open-trade protection against violent bars loses on
both TFs; high-impact USD releases cause 20-31% of violent bars but trades held through
them do better; violent bars come when price is already at an extreme, not from calm.
Module 29 tests (13). Suite: 435 passed.
Claude-Session: https://claude.ai/code/session_01RWV2yLt3YXHHGVYHSQU2EE
Verdict from the 3-week signal log (backtest/research/news_sentiment_value_2026-09-04.md):
news_sentiment is a trailing-24h echo of price (r +0.67 with the past day, +0.03 with
the next; most bullish 2 h after the 08-25 top, bearish 44 h later; most bearish at the
09-02 low, bullish 16 h later). Per-headline LLM labels are correct; the 12 h pool and
the averaging are what destroy the information. Both gating flags stay False.
- news_engine: _LLM_SUMMARY_SYSTEM rewritten for someone who has never traded (no
acronyms, two worked examples); summary payload newest-first with the platform's own
gold direction; reply cap 300 -> 400 chars.
- config: headline_max_age_seconds 43200 -> 14400; RSS items without a parsable
pubDate are dropped (they used to be stamped "now" and never aged out).
- news_engine.get_fundamental_context drops headlines whose every relevant score is
inside +-0.1 (_is_neutral_headline) before the per-currency mean; news_sentiment
from this date is a mean over directional <=4 h headlines (log comparisons split here).
- static: dashboard copy for the 4 h pool; news_summary.php prompt gets the same style rule.
- analyze_signal_log: --news (under --stored-candles) prints sentiment lead/lag with
split halves, sentiment at the window high/low with flip lags, and outcome buckets by
direction x news x macro stance; LogRow gains news_sentiment/event_risk; _row_index shared.
- tests: 406 -> 414 (age window, undated drop, neutral filter, summary prompt/payload/cap,
--news helpers, load_rows round-trip); RSS fixtures now carry a pubDate.
- docs: user/beginner guides, CLAUDE.md, deploy README.
Claude-Session: https://claude.ai/code/session_0169BgFtWjp1SEAa5392Q9jH
agent_panel.verdict_source defaults to "macro": the macro take's stance is the
verdict, the synthesizer call is skipped (4 LLM calls per panel instead of 5);
"chair" restores the four-voice consensus + OPPOSE guard. Verdict source is
cached per mode and logged (panel_verdict_source). 406 tests.
Claude-Session: https://claude.ai/code/session_01FC4s6VCURFte4dub4xKxYL
Deletes oanda_daemon.py, oanda_executor.py, their 28 tests and the daemon-only
config keys (oanda.lot_size/quality_gate/max_hold_bars). The Oanda data
provider and credentials stay — it is the live candle feed. 402 tests.
Claude-Session: https://claude.ai/code/session_01FC4s6VCURFte4dub4xKxYL
Add panel_verdict, panel_confidence, panel_rationale, panel_takes_json
columns to the signals table (additive, idempotent ALTER TABLE migration
for existing DBs). _report_row now reads the panel fields attach_panel
writes onto the SignalReport, so LLM panel verdicts stop evaporating on
process exit. Also documents the Oanda re-pin (round_trip_cost stays
$0.30 for cross-book comparability and the live TP gate; $0.65 is the
measured headline) and adds the feed-divergence / parity / shadow-scan
study tooling referenced by CLAUDE.md.
New signal_log.py logs each SignalReport to logs/signals.db via a
fail-safe hook in SignalAggregator.analyze_pair — the live-only entry
point used by the bot, daemon, CLI and dashboard. Backtester calls
analyze_from_dataframes directly, so live==backtest parity is preserved
(test-locked). Rows carry signal/grade/price/spread, MTF + trade-setup +
reasons/warnings JSON, and news-at-scan-time (calendar events + scored
headlines JSON via NewsEngine class-level caches).
Config: signal_log block (enabled, path; env SIGNAL_LOG_ENABLED/PATH),
source tagged per consumer (telegram/oanda_daemon/cli/dashboard).
381 tests pass (24 modules).
Hard TP at N×risk beats the trail on all 2024+ windows but the grid is
monotone to 3R (trend artifact) and M15 rr20 goes negative in 2021+2023
with doubled DD — trailing exits stay. Flag is research-only; its
Module 22 tests land with the next commit (shared test_all.py).
Claude-Session: https://claude.ai/code/session_017mbXs9HRawZhD1uZ3ze7zQ
Owner decisions from the 2026-08-04 audit report:
- spread-in-exit-triggers: DECLINED (books stay comparable; ledger in
CLAUDE.md records the measured ~4-6% optimism).
- weekend clock: FIXED. The market week is Fri 17:00 -> Sun 18:00 New
York time, not Fri/Sun 22:00 UTC — during US DST the Oanda feed stops
at 21:00 UTC (measured: 159/162 summer weekends). All three weekend
implementations (providers/base, signal_aggregator, gpu_backtest
calendar filter) now share the NY-anchored rule; a new test sweeps
them hourly across both DST transitions and asserts agreement. No NTP
needed — the host clock was fine, the constant was wrong.
Ends the weekly false-"STALE DATA / feed lagging" hours (summer Fri
21-22 UTC, winter Sun 22-23 UTC).
- FB2_EMA_LO (delegated): kept as a research knob but crash-proofed —
an unparseable value degrades to the tuned default 0.5 with a warning
instead of killing every scan. Documented in CLAUDE.md.
Book gates: M15 12mo pinned book byte-identical (+78.7111R, 397
trades). M5 12mo drops exactly ONE trade (556 -> 555, +97.56 ->
+98.64R): a phantom whose signal fired on the final summer-Friday bar
(decision 21:00 UTC = NY 17:00 market close) and was "filled" at
Sunday's reopen 49 hours later, -1.08R — an entry live trading can
never take. The backtest no longer contains it; live == backtest
parity improved. M15 unaffected because hour 21 is in its blocked set.
Suite: 341/341 green.
Claude-Session: https://claude.ai/code/session_01UnZkzeFcTu9agFg8BDh2tn
2026-08-04 Fable 5 QA audit, final code batch.
- Boundary math: removed the provably-dead `remaining <= buffer` guard
from all three scheduler copies (518,400-instant frozen-clock sweep:
never fired) and documented the exact-boundary-instant semantics.
- websockets/websockets.client loggers pinned to INFO so a DEBUG run
can never write the TD key (connect URI) or Tiingo auth frame into
the 30-day rotating log. (.env also chmod 600, untracked.)
- Cooldown pass factored into _apply_cooldown (pure, testable) — the
only sequential replay rule finally has direct tests.
- New simulator behavior tests (previously ZERO coverage of the trade
resolver): same-bar SL+TP tie → SL-first, gap-through-SL fills at
the open, TIMEOUT pricing + round-trip spread charging, trailing
ratchet arms at bar close and fills next-bar-only, cooldown ordering.
- CLAUDE.md synced to verified reality: 338 tests/21 modules; three
weekend-implementation copies caveat; throttle claim corrected
(single-flight, TD-only throttle); daemon trail/cooldown parity
caveats (once-per-OBSERVED-bar, wall-clock cooldown) + OPEN_ONLY and
reconcile notes; CP-path gate semantics documented as validated
behavior; new "known open items" section for the owner-decision
findings (weekend DST constant, spread-in-trigger rule-6 item, H4
seeding divergence, FB2_EMA_LO).
Gates: suite 338/338 green; pinned M15 book and M5 HEAD baseline both
reproduced identically after the cooldown refactor.
Claude-Session: https://claude.ai/code/session_01UnZkzeFcTu9agFg8BDh2tn
- dashboard_publisher.py: standalone candle-close daemon that atomically
writes snapshot.json/news.json/dashboard.html to an Apache-served dir
(DASHBOARD_OUT_DIR). Fully independent of telegram_bot/oanda_daemon —
local confirmation map + boundary math, no imports either way, no new
deps, no open port. Due-TF classification from the RAW wake time.
- static/dashboard.html: self-contained dark page — hero rail, four TF
cards (score bar, 3x3 indicator grid, MTF, setup, reasons), DeepSeek
market context, week economic calendar, scored newswire. Clock offset
from the HTTP Date header (publish-time epoch is stale between closes).
- news_engine.get_week_events(): additive read-only accessor over the
cached FF week; no signal path calls it.
- deploy/: systemd unit + live-server (LAMP/SELinux) checklist.
- test_all.py: Module 20 (9 tests) + get_week_events test; 296 green.
Claude-Session: https://claude.ai/code/session_01Wc9ZoqKLYBT2QJcpRWKoGm
Three display/warning-only upgrades to the news engine, all behind the
existing keyword path as fallback — signals are never gated on LLM
output (event_risk_caps_signals / sentiment_affects_signals stay False
per the 2026-07 audit; backtester use_news=False untouched):
- Headline sentiment: one batched call per 600s cache window (capped at
25 new headlines, sha1-cached so repeats are never re-sent) overrides
the keyword scores per currency with the gold-inverse doctrine encoded
in the prompt.
- Release surprises: get_event_risk_level's description now says which
way the print pulls GOLD ("— gold bearish: strong jobs lift yields"),
fixing the documented numeric-only gap; risk level and windows
unchanged.
- Market summary: 1-2 sentence macro blurb on SignalReport.news_summary,
rendered in Telegram alerts (📰) and the CLI (🧠).
New llm_client.py: requests-only OpenAI-compatible client (no SDK), one
attempt per call, UTC-day request cap, singleton so the cap survives
per-scan NewsEngine reconstruction. Config: deepseek block in
config.json + DEEPSEEK_API_KEY env; absent key = fully inert.
Live-measured API behavior baked in: V4-Flash is a hybrid thinking model
— requests send thinking:disabled (reasoning otherwise blew the timeout
and could exhaust max_tokens before the answer); prompts are static
module constants so DeepSeek's prefix cache prices repeat calls at the
cache-hit rate; scores use compact [s, xau] pairs; timeout 12s.
Tests 273 → 286 (new Module 19 + deepseek config tests).
Claude-Session: https://claude.ai/code/session_01Nsd8HcrV67GbCZXxZEn1Ma
The monitor has been dead since the Oanda feed switch (Oanda's provider
does not stream). New config.json scalar data_providers.streaming selects
a tick-stream provider independent of the candle feed; telegram_bot
resolves it via _get_streaming_dp() (separate instance, candle cache
stays on the active provider) and falls back to the active provider when
unset — behavior is byte-identical to before when the key is absent.
TiingoDataProvider grows the two sanctioned streaming overrides:
supports_streaming=True and iter_ticks() on wss://api.tiingo.com/fx
(mid price only — the no-spread rule extends to the stream). Two
deliberate divergences from the TD reference implementation, both
because Tiingo authenticates inside the socket: server "E" messages
raise TiingoDataError (a swallowed auth error would end the generator
cleanly and reconnect-hammer with zero backoff), and on_connect fires on
the first received message rather than after the subscribe send. Field
order of quote frames was live-verified: ask is data[7], not data[6] as
the docs table suggests (data[6] is askSize).
make_data_provider(config, name=None) now takes an explicit provider
name; keyless streaming providers disable the monitor with one warning
instead of a traceback-per-minute loop.
Tests 266 → 273 (WS parser, ticker round-trip, streaming config scalar,
explicit-name dispatch, streaming-dp resolution, monitor auto-disable).
Claude-Session: https://claude.ai/code/session_01Nsd8HcrV67GbCZXxZEn1Ma
- Delete back-compat shims data_provider.py / oanda_provider.py;
test_all.py now imports providers.* directly (ForexDataProvider name
retired; the two shim-only tests removed, 268 -> 266 tests)
- Delete backtest/research/calendar_store.py (no importer, no entry
point); config.py audit comment reworded
- Delete executed plan doc (2026-07-21 centralized-config) and the two
unbuildable guide PDFs; HTML guides remain the source of truth
- OANDA_MIGRATION_ROADMAP.md status header corrected: phases 1-2
shipped 2026-07-28, feed is live on Oanda; phases 3-6 still unbuilt
- Branches deleted locally + on both remotes: oanda_demo (merged),
web-dashboard, eurusd-modular, reversal-exit-alert (owner-approved
permanent deletion); July stash dropped
Claude-Session: https://claude.ai/code/session_019zR58p8ACW1oJ5QXVUZ5es
config.json.example: data_providers.active = "oanda" (credentials stay
in env/.env or the top-level oanda block; TD and Tiingo remain
registered fallbacks). Backtest parquet cache is now keyed per provider
— Oanda and TD bars diverge (2026-07-24 study), so a feed switch must
never serve another feed's cached bars; twelve_data keeps the legacy
un-prefixed filename so pinned-book caches stay valid. 254 tests green;
live smoke-tested against the practice API (66 contract checks: all
scanned TFs, boundary/freshness guarantees, real spread $0.71,
fetch_range pagination, typed error paths, CLI + backtest end-to-end).
Claude-Session: https://claude.ai/code/session_01Mc7XKYyPySVXiUpcN82QEK
One class per provider (TwelveData, Oanda, Tiingo), all subclassing
BaseDataProvider which owns cache, validation/cleaning, freshness +
boundary guarantees, fetch_range pagination, and streaming stubs —
subclasses implement only _fetch_raw. config.json data_providers.active
selects the live feed; data_provider.py / oanda_provider.py stay as
back-compat shims. Module 16 provider-abstraction tests (253 total green).
Claude-Session: https://claude.ai/code/session_01Mc7XKYyPySVXiUpcN82QEK
format_signal_message restyled (display-only, zero signal-logic impact):
verdict header first (signal - pair - TF - quality), trade setup moved
above the detail, per-indicator breakdown condensed to one arrow line
(_indicator_summary), heavy dividers dropped. New TP 100p line in every
setup block: entry +/- 100 pips ($10.00 on XAU) with its R-multiple vs
the stop (_INFO_TP_PIPS, formatter-only - not stamped on TradeSetup,
not in CLI/JSON/backtester). Trailing-plan logic unchanged (stamped
per-TF params, pullback fallback). Removed dead _indicator_line and
_SIGNAL_TAG. Tests 210 -> 212 (100p TP math BUY/SELL/no-setup, one-line
indicator summary).
Claude-Session: https://claude.ai/code/session_01BoZKi67Ct6QTeiFJmybXSw
Tiingo's free /tiingo/fx/top publishes only a two-tier indicative
spread for gold ($0.10 liquid hours / $0.70 rollover+news; verified on
273 production samples + live probes) — mid ± a posted increment, not
executable depth, so it can never validate the $0.30 backtest cost
assumption. Removed one day after shipping: tiingo_provider.py, the
sampler/quote plumbing in telegram_bot.py, the channel_spread_detail
config knob and tiingo_api_key resolver, and Module 14's Tiingo tests
(221 -> 210). CLAUDE.md documents the verdict: do not re-add a spread
display from Tiingo's free FX feed.
Claude-Session: https://claude.ai/code/session_01BoZKi67Ct6QTeiFJmybXSw
Five-agent audit of the signal path; all fixes are delivery/measurement
side. Regression-proven behavior-neutral on validated signals: the 5yr
M15 replay reproduces the pinned book exactly (+210.65R PF 1.35
WR 57.8% MaxDD 33.90R). Tests 215 -> 221.
- TradeSetup carries trail_arm_r/trail_dist_r stamped from the producing
config; Telegram trail plan reads them (was showing default 0.6/0.75
instead of the validated M15 arm 0.70 / M5 dist 0.85)
- Alert dedup: fingerprint = (signal, quality, direction); entry movement
compared against last-ALERTED entry/risk (old SL-dist bucket churned on
ATR drift -> near-duplicate alerts every candle)
- Fresh fetches enforce the closed-bar boundary check with one 3s grace
retry, then warn (feed lag no longer silently analyzes the prior bar)
- Backtest max DD includes the zero baseline (opening losing streaks
were undercounted; pinned M15 DD unchanged)
- Replay loads the regime-filter TF when the H4-ADX gate is on (was a
silent no-op for M5/M15 primaries)
- clear_cache under _cache_lock; dead ROC zero-cross branches removed
from both engines (provably unreachable); RSS dates via
parsedate_to_datetime (named zones no longer stamped UTC); event risk
grades from the soonest event; .env inline comments stripped;
auto-scan skips NEUTRAL like the breach path; report_to_dict honors
the weight-0 skip rule and emits strategy/trail params; PB funnel
counts spread_gate/accepted; session-filter fail-open paths log loudly
- Docs corrected to match validated code (H&S neckline = lower peak,
exhaustion anchors farthest tracked swing) with re-validation warnings
Claude-Session: https://claude.ai/code/session_01R5soRUeRBi8kHMB99csrJy
Same-day sibling of the M15 campaign, opposite finding: on M5 the arm
point barely matters — winners need a looser leash. 5yr grid plateau
arm 0.5-0.7 x dist 0.85-1.0; shipped the OOS-preferred edge 0.60/0.85:
+231.2R PF 1.30 vs +218.0R PF 1.29 baseline, both halves better, DD
flat (19.2 vs 19.1), OOS +14.5R vs +11.1R. dist 1.0 rejected at the
OOS gate despite best in-sample total. XAU_PB pb_trail unaffected.
Factory replay reproduces the grid cell; 201/201 tests.
Claude-Session: https://claude.ai/code/session_01VPNF37MP9Kuv62B1VBrZ1b
Parity: replay now applies the live _apply_regime_filter_live method
under the live guard (function-identity, default-off); unregistered-pair
fallback applies the config.json overlay; parity sweep gains direct
ATR/vol-regime assertions; regime-filter semantics test added.
Trail campaign: --trail-arm/--trail-dist research flags; 30-cell grid on
5yr M15 refuted the earlier-arm hypothesis and found the 0.70-0.80 arm
plateau. Shipped M15-only trail_arm_r=0.70 (factory override):
+210.7R PF 1.35 WR 57.8% vs +175.7R PF 1.30 WR 53.9% baseline; both
halves and 2024-06->2025-03 OOS better (+57.4 vs +45.4); 5yr max-DD
clause (33.9R vs 29.1R) explicitly waived by owner. M5 unchanged.
Factory replay reproduces the grid cell trade-for-trade; 201/201 tests.
Claude-Session: https://claude.ai/code/session_01VPNF37MP9Kuv62B1VBrZ1b
Harwinder Sekhon