End-to-end map: external services → providers/news layer → signal core
(engine + factory + aggregator) → SignalReport → four delivery surfaces,
plus the backtester replaying the same core. Regenerate with
venv/bin/python docs/architecture_diagram.py after architecture changes.
Claude-Session: https://claude.ai/code/session_01Aq1jZ6jBsBdEC2xT3u46SA
The practice API rejects ~15-20 requests/day with a spurious 401
"Insufficient authorization" while the same token succeeds on adjacent
requests; each one silently cost a full scan cycle on that timeframe.
One retry after RETRY_DELAY_S (2 s) absorbs these plus transient
5xx/network faults. Real credential failures fail both attempts and
raise as before; other 4xx (bad request, 429) are never retried.
Tests: transient-401 recovery, persistent-401 raise, 400 not retried.
Claude-Session: https://claude.ai/code/session_01Tujtmmhxj9mxKjPFSoVjmP
Owner-supplied page design replacing the dark-only v1: Bootstrap 5
colour tokens with a luma-inverted dark mode (toggle + system-pref +
localStorage), compact clickable TF cards opening a full-detail modal
(all-nine indicator table with value/conf/weight, price ladder of key
levels vs spot, per-confirmation-TF drilldowns, provenance), newswire
aggregate sentiment box, Inter/JetBrains Mono webfonts.
Re-applied the c70f58a sync fixes the source copy predated: 5 s
conditional-GET polling and the +2 s boundary refetch sniper. Data
logic otherwise unchanged; all fields verified against
report_to_webapp_dict output.
Claude-Session: https://claude.ai/code/session_017mbXs9HRawZhD1uZ3ze7zQ
Three display-side fixes, no signal-logic changes:
- run_cycle publishes snapshot.json BEFORE the news refresh — news I/O
(FF + 4 RSS + possible DeepSeek calls) took 20-30 s on TTL-expired
cycles and kept fresh signals off the page (publish_all split into
publish_snapshot/publish_news, kept for boot)
- page polls every 5 s (was 20 s) using conditional GETs: dropped the
?t= cache-buster + no-store so Apache answers unchanged files with
bodyless 304s
- boundary refetch sniper re-armed: fires 2 s after next_scan_epoch
regardless of distance (the old delay<POLL_MS guard would have been
dead code at 5 s polling)
New Module 20 test locks the snapshot-before-news ordering. 352 tests.
Claude-Session: https://claude.ai/code/session_017mbXs9HRawZhD1uZ3ze7zQ
Validation (llm_news_eval.py harness + 78-headline golden CSV + verdict
doc): headline scoring 78/78 vs keyword baseline 24/78 — validated;
event gold-direction calls misread close K/M numeric comparisons ~11%
of the time (both claims cases in sample, JOLTS 8.79M vs 8.84M).
Fix: event payload now carries code-computed "vs_forecast" (from
EconomicEvent.surprise) and the prompt gains an explicit inverted-
metrics rule + worked claims example — iteration 1 (field alone) fixed
the arithmetic but exposed a masked inversion error. 18/18 on live
re-verification (4 misreads fixed, 11 regression cases clean, 3 fresh
both-side probes).
/status: new "News & AI" section — per-source health for the calendar
chain (which of FF/MyFXBook served), each RSS feed, and DeepSeek
(daily count, last error kind), written passively by the existing
fetch paths under _cache_lock. Failures WARN once on transition
(repeats DEBUG, recovery INFO) so the rotating log carries outage
stories without 10-min spam.
test_all.py: +5 Module 19 tests (351 total); also carries the Module 22
fixed-RR-TP tests from the prior research session (shared file).
Replay artifacts dir gitignored like commit_ladder.
Claude-Session: https://claude.ai/code/session_017mbXs9HRawZhD1uZ3ze7zQ
Hard TP at N×risk beats the trail on all 2024+ windows but the grid is
monotone to 3R (trend artifact) and M15 rr20 goes negative in 2021+2023
with doubled DD — trailing exits stay. Flag is research-only; its
Module 22 tests land with the next commit (shared test_all.py).
Claude-Session: https://claude.ai/code/session_017mbXs9HRawZhD1uZ3ze7zQ
Deletion was already staged in the working tree by the owner; the
roadmap was approved 2026-07-24 as reference only and its plan file
lives outside the repo. Content remains in git history.
Claude-Session: https://claude.ai/code/session_01UnZkzeFcTu9agFg8BDh2tn
Owner decisions from the 2026-08-04 audit report:
- spread-in-exit-triggers: DECLINED (books stay comparable; ledger in
CLAUDE.md records the measured ~4-6% optimism).
- weekend clock: FIXED. The market week is Fri 17:00 -> Sun 18:00 New
York time, not Fri/Sun 22:00 UTC — during US DST the Oanda feed stops
at 21:00 UTC (measured: 159/162 summer weekends). All three weekend
implementations (providers/base, signal_aggregator, gpu_backtest
calendar filter) now share the NY-anchored rule; a new test sweeps
them hourly across both DST transitions and asserts agreement. No NTP
needed — the host clock was fine, the constant was wrong.
Ends the weekly false-"STALE DATA / feed lagging" hours (summer Fri
21-22 UTC, winter Sun 22-23 UTC).
- FB2_EMA_LO (delegated): kept as a research knob but crash-proofed —
an unparseable value degrades to the tuned default 0.5 with a warning
instead of killing every scan. Documented in CLAUDE.md.
Book gates: M15 12mo pinned book byte-identical (+78.7111R, 397
trades). M5 12mo drops exactly ONE trade (556 -> 555, +97.56 ->
+98.64R): a phantom whose signal fired on the final summer-Friday bar
(decision 21:00 UTC = NY 17:00 market close) and was "filled" at
Sunday's reopen 49 hours later, -1.08R — an entry live trading can
never take. The backtest no longer contains it; live == backtest
parity improved. M15 unaffected because hour 21 is in its blocked set.
Suite: 341/341 green.
Claude-Session: https://claude.ai/code/session_01UnZkzeFcTu9agFg8BDh2tn
2026-08-04 Fable 5 QA audit, final code batch.
- Boundary math: removed the provably-dead `remaining <= buffer` guard
from all three scheduler copies (518,400-instant frozen-clock sweep:
never fired) and documented the exact-boundary-instant semantics.
- websockets/websockets.client loggers pinned to INFO so a DEBUG run
can never write the TD key (connect URI) or Tiingo auth frame into
the 30-day rotating log. (.env also chmod 600, untracked.)
- Cooldown pass factored into _apply_cooldown (pure, testable) — the
only sequential replay rule finally has direct tests.
- New simulator behavior tests (previously ZERO coverage of the trade
resolver): same-bar SL+TP tie → SL-first, gap-through-SL fills at
the open, TIMEOUT pricing + round-trip spread charging, trailing
ratchet arms at bar close and fills next-bar-only, cooldown ordering.
- CLAUDE.md synced to verified reality: 338 tests/21 modules; three
weekend-implementation copies caveat; throttle claim corrected
(single-flight, TD-only throttle); daemon trail/cooldown parity
caveats (once-per-OBSERVED-bar, wall-clock cooldown) + OPEN_ONLY and
reconcile notes; CP-path gate semantics documented as validated
behavior; new "known open items" section for the owner-decision
findings (weekend DST constant, spread-in-trigger rule-6 item, H4
seeding divergence, FB2_EMA_LO).
Gates: suite 338/338 green; pinned M15 book and M5 HEAD baseline both
reproduced identically after the cooldown refactor.
Claude-Session: https://claude.ai/code/session_01UnZkzeFcTu9agFg8BDh2tn
2026-08-04 Fable 5 QA audit. Three batches sharing one test file, all
verified by runnable repros before fixing, all gated below.
Batch C — telegram alert delivery (verified by h4 harness):
- Failed breach rescan shrinks the cooldown to 30 s instead of
silencing the symbol for 5 min; breach during cooldown is PARKED and
fired on the first tick after expiry (was dropped forever).
- Level cache keyed symbol→TF (M5 scans no longer wipe H1/H4 swing
levels within minutes); breach labels TF-qualified.
- Markdown alert sends get a plain-text fallback (mirroring _safe_edit);
fingerprint commits only when a message actually landed.
- Per-key asyncio.Lock closes the dedup TOCTOU between auto-scan and
breach sends (forced-interleave repro produced duplicate alerts).
- _start_scheduler respawns dead per-TF loops (was all-or-nothing).
- Clean WS close now backs off (was a zero-delay reconnect hot loop);
backoff resets only after a connection that received ticks; TD
iter_ticks fires on_connect on the first RECEIVED frame (invalid key
can no longer pin the backoff at minimum).
- Scan-failure logs carry the real traceback (was "NoneType: None").
Batch E — providers (verified by h5 harness):
- fetch_range: 3 consecutive chunk ERRORS raise instead of returning
silently truncated history (verified 731-day silent gap repro; also
covers 429 storms); pagination no-progress guard; head-truncation
ERROR log.
- Single-flight per (symbol,TF): 4 concurrent scans = 1 API call (was
4); cache records requested depth so a longer-lookback call refetches.
- Multi-bar forming/future tail fully dropped (was iloc[:-1] once).
- Oanda: malformed candles skipped, never 0.0-defaulted; cleaning drops
non-positive-price rows BEFORE OHLC repair; inf returns neutralized so
one corrupt bar can no longer disarm the whole 10-sigma filter.
Batch F — backtester internals (verified by h6 harness):
- NaN bar in a trailing window exits at the last valid bar (was NaN R
poisoning every metric); static-path TIMEOUT NaN guard;
compute_metrics excludes non-finite R loudly. Zero real-data exposure
(0 NaN rows in all 10 Oanda parquets) — defensive.
- fetch_history ERROR-logs pre-2026-07-08 cache files (cleaned-bars era)
when loaded for raw-path resolution.
Gates: suite 333/333 green; pinned M15 12mo book reproduced identically
(397 trades, +78.7111R) after E and after F; M5 HEAD-config baseline
reproduced identically (556 trades, +97.5604R) after F.
Claude-Session: https://claude.ai/code/session_01UnZkzeFcTu9agFg8BDh2tn
2026-08-04 audit, Batch D. Verification outcomes (no behavior changes —
comment + tests only):
- CP gate-bypass (quality forced to B after the gate chain, exhaustion/
M5-score gates not re-checked) is VALIDATED behavior, not a bug: the
pinned shipped-config M5 12mo book carries 226/586 XAU_CP trades
(+38.9R), 104 with EXHAUST warnings (+15.3R), 175 below the M5
min-score gate (+28.0R). Removing the "holes" removes validated
profit. New test pins the shipped state (PB off, CP BUY-only on M5,
ema_partial_neutral M5-only) so any flip is a conscious revalidated
decision.
- E18 (xau_strategy vs backtest blocked-hours "drift") REFUTED as a
defect: the divergence is required by pair isolation — the backtest
copy is the generic non-XAU fallback and must not inherit
gold-validated deltas. Documented in the factory + a test asserting
the delta stays XAU-side.
- 6c-upgrade re-arming an exhaustion-D refuted (exhaustion zeroes
confidence below 6c's floor; regimes mutually exclusive). "WEEKEND
FILTER" CP-suppression branch is defense-in-depth for direct
analyze_from_dataframes callers, kept.
Suite: 316 green.
Claude-Session: https://claude.ai/code/session_01UnZkzeFcTu9agFg8BDh2tn
2026-08-04 audit, Batch B. Verified first (h1 harness): the two engines
are byte-identical on every reachable input — 370-prefix flat-bar deck,
warmup band, 322 real-data M15 positions incl. all 7 zero-range bars,
zero diffs. The claimed ADX NaN desync / warmup-guard divergences are
unreachable below the shared 30-bar analyze() floor. Changes here are
therefore observability + hygiene, proven behavior-neutral by exact
reproduction of the pinned M15 12mo book (397 trades, +78.7111R,
byte-identical trade rows).
- _resolve_pos: cache-miss fallback to live-engine recompute now warns
on first miss and counts every miss (was fully silent — a run could
switch math with no trace).
- ROC/Keltner warmup guards textually aligned with the live engine so a
future period/floor change can't open a silent divergence window.
- New regression: full-sweep parity test comparing ALL indicator
signals/confidences, composite, and regimes on every prefix incl. a
flat zero-DM/zero-TR run (the old parity test skipped
ADX/MACD/ROC/Keltner/RSI entirely); plus a loud-fallback test.
Suite: 314 green. Book gate: oanda_m15_12mo_030 reproduced identically.
Claude-Session: https://claude.ai/code/session_01UnZkzeFcTu9agFg8BDh2tn
2026-08-04 Fable 5 QA audit, Batch A. All fixes are execution-robustness
only — no signal code touched (git diff scope: daemon/executor/tests).
Findings verified by runnable repros before fixing (h3 harness):
- close_trade: a 200 whose close order was created but NOT filled (e.g.
MARKET_HALTED during the settlement pause) now raises instead of
returning; the daemon's timeout path keeps tracking and retries. The
old path popped state and orphaned a live position forever
(reconciliation was boot-only).
- place_market_order: positionFill=OPEN_ONLY — on a netting account an
opposite-side order can no longer silently close an untracked
position; the divergence surfaces as an order cancel.
- _execute_trade: a fill without tradeOpened (netting close/reduce) and
any placement OandaAPIError (server may have filled before a network
timeout) now trigger _reconcile_positions instead of silent return.
- _discover_existing_positions → lock-guarded _reconcile_positions,
callable mid-run; adoption of a position with NO broker stop now
screams (CRITICAL + per-bar warning) instead of computing sl=entry,
risk=0 and silently never trailing.
- _manage_trailing: snapshot deep-copies per-trade dicts (no mutation of
live state outside the lock); write-back skips trades popped by
_check_closed_trades mid-iteration (no ghost resurrection).
- Fail-closed config: unknown OANDA_QUALITY_GATE refuses to start
(previously a typo silently became the most permissive gate "C");
non-integer OANDA_LOT_SIZE/MAX_HOLD_BARS exit with a clear message.
Suite: 302 → 312 tests (Module 21), all green.
Claude-Session: https://claude.ai/code/session_01UnZkzeFcTu9agFg8BDh2tn
A balanced feed (XAU and USD means nearly cancelling) yields a tiny
negative net that "+.2f" rendered as "-0.00" — seen live 2026-08-03.
Regression test uses that day's actual per-currency means (300 tests).
Claude-Session: https://claude.ai/code/session_017975c1TufZ3xRYAq8iWTCS
Owner-locked /news replies with the existing get_pair_sentiment() net
score (XAU mean - USD mean, -1..+1) and its label; a caveat line marks
the no-headlines case so a feed outage can't read as neutral. Fetch runs
in the shared executor with a placeholder-then-edit reply. Display-only:
news still never gates signals. Registered in handler block, command
menu, help text, docstring, and user guide; 3 new builder tests (299).
Claude-Session: https://claude.ai/code/session_017975c1TufZ3xRYAq8iWTCS
- dashboard_publisher.py: standalone candle-close daemon that atomically
writes snapshot.json/news.json/dashboard.html to an Apache-served dir
(DASHBOARD_OUT_DIR). Fully independent of telegram_bot/oanda_daemon —
local confirmation map + boundary math, no imports either way, no new
deps, no open port. Due-TF classification from the RAW wake time.
- static/dashboard.html: self-contained dark page — hero rail, four TF
cards (score bar, 3x3 indicator grid, MTF, setup, reasons), DeepSeek
market context, week economic calendar, scored newswire. Clock offset
from the HTTP Date header (publish-time epoch is stale between closes).
- news_engine.get_week_events(): additive read-only accessor over the
cached FF week; no signal path calls it.
- deploy/: systemd unit + live-server (LAMP/SELinux) checklist.
- test_all.py: Module 20 (9 tests) + get_week_events test; 296 green.
Claude-Session: https://claude.ai/code/session_01Wc9ZoqKLYBT2QJcpRWKoGm
Added: backtest/research/commit_ladder.py (recreates historical configs
via --set on the fixed harness; 2026-08-03 campaign confirmed HEAD is
the best config on all timeframes). Results stay local like all pinned
trade books.
Claude-Session: https://claude.ai/code/session_01Nsd8HcrV67GbCZXxZEn1Ma
Three display/warning-only upgrades to the news engine, all behind the
existing keyword path as fallback — signals are never gated on LLM
output (event_risk_caps_signals / sentiment_affects_signals stay False
per the 2026-07 audit; backtester use_news=False untouched):
- Headline sentiment: one batched call per 600s cache window (capped at
25 new headlines, sha1-cached so repeats are never re-sent) overrides
the keyword scores per currency with the gold-inverse doctrine encoded
in the prompt.
- Release surprises: get_event_risk_level's description now says which
way the print pulls GOLD ("— gold bearish: strong jobs lift yields"),
fixing the documented numeric-only gap; risk level and windows
unchanged.
- Market summary: 1-2 sentence macro blurb on SignalReport.news_summary,
rendered in Telegram alerts (📰) and the CLI (🧠).
New llm_client.py: requests-only OpenAI-compatible client (no SDK), one
attempt per call, UTC-day request cap, singleton so the cap survives
per-scan NewsEngine reconstruction. Config: deepseek block in
config.json + DEEPSEEK_API_KEY env; absent key = fully inert.
Live-measured API behavior baked in: V4-Flash is a hybrid thinking model
— requests send thinking:disabled (reasoning otherwise blew the timeout
and could exhaust max_tokens before the answer); prompts are static
module constants so DeepSeek's prefix cache prices repeat calls at the
cache-hit rate; scores use compact [s, xau] pairs; timeout 12s.
Tests 273 → 286 (new Module 19 + deepseek config tests).
Claude-Session: https://claude.ai/code/session_01Nsd8HcrV67GbCZXxZEn1Ma
The monitor has been dead since the Oanda feed switch (Oanda's provider
does not stream). New config.json scalar data_providers.streaming selects
a tick-stream provider independent of the candle feed; telegram_bot
resolves it via _get_streaming_dp() (separate instance, candle cache
stays on the active provider) and falls back to the active provider when
unset — behavior is byte-identical to before when the key is absent.
TiingoDataProvider grows the two sanctioned streaming overrides:
supports_streaming=True and iter_ticks() on wss://api.tiingo.com/fx
(mid price only — the no-spread rule extends to the stream). Two
deliberate divergences from the TD reference implementation, both
because Tiingo authenticates inside the socket: server "E" messages
raise TiingoDataError (a swallowed auth error would end the generator
cleanly and reconnect-hammer with zero backoff), and on_connect fires on
the first received message rather than after the subscribe send. Field
order of quote frames was live-verified: ask is data[7], not data[6] as
the docs table suggests (data[6] is askSize).
make_data_provider(config, name=None) now takes an explicit provider
name; keyless streaming providers disable the monitor with one warning
instead of a traceback-per-minute loop.
Tests 266 → 273 (WS parser, ticker round-trip, streaming config scalar,
explicit-name dispatch, streaming-dp resolution, monitor auto-disable).
Claude-Session: https://claude.ai/code/session_01Nsd8HcrV67GbCZXxZEn1Ma
- Delete back-compat shims data_provider.py / oanda_provider.py;
test_all.py now imports providers.* directly (ForexDataProvider name
retired; the two shim-only tests removed, 268 -> 266 tests)
- Delete backtest/research/calendar_store.py (no importer, no entry
point); config.py audit comment reworded
- Delete executed plan doc (2026-07-21 centralized-config) and the two
unbuildable guide PDFs; HTML guides remain the source of truth
- OANDA_MIGRATION_ROADMAP.md status header corrected: phases 1-2
shipped 2026-07-28, feed is live on Oanda; phases 3-6 still unbuilt
- Branches deleted locally + on both remotes: oanda_demo (merged),
web-dashboard, eurusd-modular, reversal-exit-alert (owner-approved
permanent deletion); July stash dropped
Claude-Session: https://claude.ai/code/session_019zR58p8ACW1oJ5QXVUZ5es
Gate 6h (big-candle entry gate): blocks all XAU entries (core, XAU_PB,
XAU_CP) when the primary TF's last 24h contains a bar with range >= K x
prior-bar ATR. Ships default-OFF: validation rejected every threshold
(K 3-6 on M5/M15 12mo) — removed trades were net-POSITIVE at every cell;
M5 quiet-window trades are the only losing bucket. Config knobs
big_candle_atr_multiple/lookback_hours, --big-candle backtest flag,
Module 17 tests.
M5 relaxation SHIPPED (validated 2026-07-30, Oanda bars, $0.30 spread):
m5_min_abs_score 30->25, m5_min_confirmation_score 10->5, hour-21 UTC
unblocked M5-only. Beats control in ALL 6 calendar years of the 5yr
replay (+230.9R/2319 PF 1.26 DD 21.3 vs +168.0R/2048 PF 1.21 DD 27.0),
12mo +107.4R PF 1.36 vs +84.6R PF 1.33, OOS +26.1R vs +17.2R, $0.70
spread +86.9R vs +66.4R. M15: every relaxation retested negative —
config unchanged. Declined-but-open: M5 hour-22 block, M15 hour-16
unblock (both fail 5yr/OOS). Evidence in the factory docstring.
Tooling: generic --set PATH=JSON research override in gpu_backtest
(typo-proof, logged); diagnose_filters filter_tags refreshed for gates
6d-6i with a staleness-guard test (Module 18); research scripts
hour_analysis/burst_profile/book_diff. Suite: 268 tests.
Claude-Session: https://claude.ai/code/session_014jqHpmNYmwYMowF6bPZyeD
config.json.example: data_providers.active = "oanda" (credentials stay
in env/.env or the top-level oanda block; TD and Tiingo remain
registered fallbacks). Backtest parquet cache is now keyed per provider
— Oanda and TD bars diverge (2026-07-24 study), so a feed switch must
never serve another feed's cached bars; twelve_data keeps the legacy
un-prefixed filename so pinned-book caches stay valid. 254 tests green;
live smoke-tested against the practice API (66 contract checks: all
scanned TFs, boundary/freshness guarantees, real spread $0.71,
fetch_range pagination, typed error paths, CLI + backtest end-to-end).
Claude-Session: https://claude.ai/code/session_01Mc7XKYyPySVXiUpcN82QEK
One class per provider (TwelveData, Oanda, Tiingo), all subclassing
BaseDataProvider which owns cache, validation/cleaning, freshness +
boundary guarantees, fetch_range pagination, and streaming stubs —
subclasses implement only _fetch_raw. config.json data_providers.active
selects the live feed; data_provider.py / oanda_provider.py stay as
back-compat shims. Module 16 provider-abstraction tests (253 total green).
Claude-Session: https://claude.ai/code/session_01Mc7XKYyPySVXiUpcN82QEK
Scripts + summary CSVs for the 2026-07-24 studies (real XAU spread ~$0.65
tradable hours; Oanda vs TD bars diverge ~11% of ATR with 10% direction
flips). Large raw data files (parquet, per-boundary samples) gitignored.
Claude-Session: https://claude.ai/code/session_01Mc7XKYyPySVXiUpcN82QEK
Approved 2026-07-24: 6-phase plan for fully switching the signal feed
from Twelve Data to Oanda v20 (inert plumbing -> backtester agnosticism
-> pricing-stream monitor -> shadow parity -> book re-validation with
owner cost gates -> reversible cutover). TD remains a permanent
fallback. Grounded in the 2026-07-24 divergence study (10% bar-direction
flips, 11%-of-ATR noise, real Oanda spread ~$0.72 vs assumed $0.30).
Claude-Session: https://claude.ai/code/session_01YNvxXtgz62F7kTKZZsAAAE
New independent trading daemon (oanda_daemon.py) that places orders on
an Oanda v20 practice account instead of sending Telegram alerts.
Oanda-only market data (oanda_provider.py, mid candles + real bid/ask
spread) through the unchanged signal pipeline; order execution and
trade management via oanda_executor.py (raw requests, no new deps).
Book parity with the backtester: no broker-side TP (exits = SL/TRAIL/
TIMEOUT only), bar-close-only trail ratchet with each closed bar
processed exactly once, 5-bar per-TF entry cooldown, structural SL
from the setup. Accepted divergence: one position total (skipped
signals logged for book reconciliation).
Safety/ops: fail-fast boot on rejected credentials (transient outages
continue), market orders never retried (double-fill risk), entry lock
against aligned-boundary races, restart recovery + adoption via
oanda_state.json, hourly heartbeat, asyncio exception handler, every
failure path logged. Prices formatted at instrument precision.
Config: env OANDA_* -> config.json "oanda" section (validated in
app_config). Tests: Module 15 (30 tests), suite 240 green.
Claude-Session: https://claude.ai/code/session_01VUnCqjWBuvKRSjLwHeB9BJ
format_signal_message restyled (display-only, zero signal-logic impact):
verdict header first (signal - pair - TF - quality), trade setup moved
above the detail, per-indicator breakdown condensed to one arrow line
(_indicator_summary), heavy dividers dropped. New TP 100p line in every
setup block: entry +/- 100 pips ($10.00 on XAU) with its R-multiple vs
the stop (_INFO_TP_PIPS, formatter-only - not stamped on TradeSetup,
not in CLI/JSON/backtester). Trailing-plan logic unchanged (stamped
per-TF params, pullback fallback). Removed dead _indicator_line and
_SIGNAL_TAG. Tests 210 -> 212 (100p TP math BUY/SELL/no-setup, one-line
indicator summary).
Claude-Session: https://claude.ai/code/session_01BoZKi67Ct6QTeiFJmybXSw
Tiingo's free /tiingo/fx/top publishes only a two-tier indicative
spread for gold ($0.10 liquid hours / $0.70 rollover+news; verified on
273 production samples + live probes) — mid ± a posted increment, not
executable depth, so it can never validate the $0.30 backtest cost
assumption. Removed one day after shipping: tiingo_provider.py, the
sampler/quote plumbing in telegram_bot.py, the channel_spread_detail
config knob and tiingo_api_key resolver, and Module 14's Tiingo tests
(221 -> 210). CLAUDE.md documents the verdict: do not re-add a spread
display from Tiingo's free FX feed.
Claude-Session: https://claude.ai/code/session_01BoZKi67Ct6QTeiFJmybXSw
Five-agent audit of the signal path; all fixes are delivery/measurement
side. Regression-proven behavior-neutral on validated signals: the 5yr
M15 replay reproduces the pinned book exactly (+210.65R PF 1.35
WR 57.8% MaxDD 33.90R). Tests 215 -> 221.
- TradeSetup carries trail_arm_r/trail_dist_r stamped from the producing
config; Telegram trail plan reads them (was showing default 0.6/0.75
instead of the validated M15 arm 0.70 / M5 dist 0.85)
- Alert dedup: fingerprint = (signal, quality, direction); entry movement
compared against last-ALERTED entry/risk (old SL-dist bucket churned on
ATR drift -> near-duplicate alerts every candle)
- Fresh fetches enforce the closed-bar boundary check with one 3s grace
retry, then warn (feed lag no longer silently analyzes the prior bar)
- Backtest max DD includes the zero baseline (opening losing streaks
were undercounted; pinned M15 DD unchanged)
- Replay loads the regime-filter TF when the H4-ADX gate is on (was a
silent no-op for M5/M15 primaries)
- clear_cache under _cache_lock; dead ROC zero-cross branches removed
from both engines (provably unreachable); RSS dates via
parsedate_to_datetime (named zones no longer stamped UTC); event risk
grades from the soonest event; .env inline comments stripped;
auto-scan skips NEUTRAL like the breach path; report_to_dict honors
the weight-0 skip rule and emits strategy/trail params; PB funnel
counts spread_gate/accepted; session-filter fail-open paths log loudly
- Docs corrected to match validated code (H&S neckline = lower peak,
exhaustion anchors farthest tracked swing) with re-validation warnings
Claude-Session: https://claude.ai/code/session_01R5soRUeRBi8kHMB99csrJy
Same-day sibling of the M15 campaign, opposite finding: on M5 the arm
point barely matters — winners need a looser leash. 5yr grid plateau
arm 0.5-0.7 x dist 0.85-1.0; shipped the OOS-preferred edge 0.60/0.85:
+231.2R PF 1.30 vs +218.0R PF 1.29 baseline, both halves better, DD
flat (19.2 vs 19.1), OOS +14.5R vs +11.1R. dist 1.0 rejected at the
OOS gate despite best in-sample total. XAU_PB pb_trail unaffected.
Factory replay reproduces the grid cell; 201/201 tests.
Claude-Session: https://claude.ai/code/session_01VPNF37MP9Kuv62B1VBrZ1b
Parity: replay now applies the live _apply_regime_filter_live method
under the live guard (function-identity, default-off); unregistered-pair
fallback applies the config.json overlay; parity sweep gains direct
ATR/vol-regime assertions; regime-filter semantics test added.
Trail campaign: --trail-arm/--trail-dist research flags; 30-cell grid on
5yr M15 refuted the earlier-arm hypothesis and found the 0.70-0.80 arm
plateau. Shipped M15-only trail_arm_r=0.70 (factory override):
+210.7R PF 1.35 WR 57.8% vs +175.7R PF 1.30 WR 53.9% baseline; both
halves and 2024-06->2025-03 OOS better (+57.4 vs +45.4); 5yr max-DD
clause (33.9R vs 29.1R) explicitly waived by owner. M5 unchanged.
Factory replay reproduces the grid cell trade-for-trade; 201/201 tests.
Claude-Session: https://claude.ai/code/session_01VPNF37MP9Kuv62B1VBrZ1b
- Delete test_ws_dual.py (standalone probe, unused) and a stale tracked log
- config.py: drop MarketSession enum, is_jpy_pair, candle_buffer_seconds,
and the never-read legacy SignalConfig/XauStrategyConfig fields
- data_provider.py: drop uncalled get_latest_price/get_spread_estimate
- Remove unused imports (engine, telegram_bot, main, news_engine)
- test_all.py: drop/trim the corresponding defaults assertions (199 tests)
- CLAUDE.md updated to match
Zero behavior change: no production code read any removed field
(verified reference-by-reference); suite green at 199/199.
Claude-Session: https://claude.ai/code/session_01VPNF37MP9Kuv62B1VBrZ1b
Documents the app_config.py / config.json overlay shipped in Tasks 1-4:
repo-map rows for app_config.py and config.json.example, env-var table
precedence (env -> config.json -> none, hardcoded fallbacks removed
2026-07-21), PlatformConfig/pair_signals overlay call order, and the
real 199-tests/13-modules suite count. Also fixes a stale SCAN_PAIRS
comment in telegram_bot.py that still named PlatformConfig.scan_pairs
as the deployment switch instead of config.json's scanning.pairs.
Claude-Session: https://claude.ai/code/session_01VPNF37MP9Kuv62B1VBrZ1b
- Credentials, alert quality, scan pairs/timeframes, log level and
candle-loop intervals now resolve via app_config (env -> config.json
-> code default) instead of hardcoded fallbacks.
- _get_shared_dp() uses make_data_provider() for provider-registry
dispatch.
- main() refuses to start without a token/chat_id, raising SystemExit
with an actionable message instead of a bare sys.exit string.
- test_all.py Module 13: two new tests covering credential/quality/
loop wiring via a reload, and the startup guard.
Claude-Session: https://claude.ai/code/session_01VPNF37MP9Kuv62B1VBrZ1b
Resolves the Twelve Data API key through app_config.provider_api_key()
(env → config.json → no default) instead of a hardcoded field on
PlatformConfig. Adds data_provider.PROVIDER_REGISTRY + make_data_provider()
so signal_aggregator and the backtester build the active provider through
one seam; telegram_bot.py gets the same one-line WS-key fix now (per Task 4's
already-planned edit for that line) so the suite doesn't crash on import
between this commit and Task 4's full wiring pass.
Claude-Session: https://claude.ai/code/session_01VPNF37MP9Kuv62B1VBrZ1b
Implements Task 2: apply_pair_overrides in xauusd.make_config before
the validated per-TF deltas, so e.g. M5 hour-9 appends on top of the
JSON base. Updated Module 9 equivalence test and added three parity pins.
Claude-Session: https://claude.ai/code/session_01VPNF37MP9Kuv62B1VBrZ1b
Add NewsConfig.sentiment_affects_signals (default False), gating the +5
aligned-sentiment confidence bonus and the 10-pt news_conflict penalty.
With event_risk_caps_signals already False, live decisions are now pure
technical, matching the backtester (use_news=False). Warnings and the
sentiment display are unchanged. Adds a gating regression test.
Claude-Session: https://claude.ai/code/session_01FMaaPdrrrQuh1VvGQH1icN
M5 factory: cp_enabled=True, patterns limited to inv_hs, double_bottom,
bull_flag. SELL CP lost -3.21R on OOS; BUY CP: +41.8R PF 1.60 on 12mo,
+65.2R PF 1.48 on 5yr. Adds +22R to the M5 book (708 vs 513 trades).
M15/H1 remain disabled — all failed OOS validation.
Claude-Session: https://claude.ai/code/session_015kby7M9JwpYi8NQT2yoxAi
Third entry archetype: completion of classical chart patterns (double
bottom/top, H&S top / inverse H&S, bull/bear flag) — the tight canon
with the strongest documented stats. New chart_patterns.py detector
module (pure functions, ATR-unit geometry, closed-bar slices only,
first-close-beyond trigger discipline so a breakout fires exactly
once and never repaints).
Pipeline step 10 in analyze_from_dataframes: fills only empty setup
slots, allows weak-composite F grades (normal at reversal completion)
but inherits session/day blocks, mirrors MTF-opposition + SELL
vol/hour + M5 protections. Geometry: SL = pattern anchor -/+ 0.5 ATR
(min $8), TP1 = measured move, $9 spread gate, trailing exit does the
real work. Strategy tag XAU_CP.
cp_enabled defaults to False everywhere — per-TF adoption happens in
pair_signals/xauusd.py only after the standard validation rule passes.
Backtester: --cp / --cp-patterns flags to test the candidate, XAU_CP
tag recognition.
Suite: 183/183 (21 new tests: detectors, trigger discipline,
invalidation, gates, pipeline, default-off safety).
Claude-Session: https://claude.ai/code/session_015kby7M9JwpYi8NQT2yoxAi
Fixes all high+medium findings from the 2026-07-16 full-codebase audit.
High:
- data_provider: cache now rejects snapshots missing the latest closed
bar (_has_latest_closed_bar, weekend-exempt). The wall-clock TTL served
mid-candle snapshots at the next boundary, so live M15/H1 scans ran one
full bar behind and the bar-close session check evaluated the wrong hour.
- signal_aggregator: gate 6e (SELL quality) only acts on actionable
grades. It was re-grading session-blocked F reports to D, which
re-opened the M5 pullback path (entry check: quality != "F") on
session-blocked bars.
- gpu_backtest: port the live EMA-ribbon compression/expansion logic
(+10 conf on expansion) into the GPU engine. Its absence diverged
composite score/conf on 7-9% of bars and flipped signals on ~0.2%.
Medium:
- gpu_backtest: Bollinger precompute uses the exact pandas rolling calls
(cumsum E[x2]-E[x]2 lost precision at gold price levels; bb_position
error up to 7.6e-3 crossed signal thresholds, worst during squeezes).
- gpu_backtest: --confirm-tfs defaults to the pair factory's live
confirmation map instead of a hardcoded 1h,4h.
- gpu_backtest: entries re-anchor to the first RAW bar after the signal
bar, so >10-sigma spike bars dropped by cleaning are no longer skipped
at entry (favorable bias).
- gpu_backtest: per-TF warmup pad — D1 confirms get >=210 bars at window
start (62-day pad gave ~61 bars: no EMA200, unlike live).
- data_provider: thread-safety locks for cache + throttle (shared
provider across the bot's 4-thread pool); outlier filter keeps the
snap-back bar after a spike instead of punching a two-bar hole.
- telegram_bot: alert fingerprint gains a risk-relative entry bucket
(materially-moved setups re-alert) and is committed only after a
successful send (failed sends no longer consume the alert for 1h).
- engine: disabled_indicators=["macd"] now removes the MACD pillar from
scoring (histogram still computed for divergence).
Verification: test_all.py 187/187 (new Module 11: 8 audit-fix
regressions incl. a live-vs-GPU parity sweep with ribbon coverage);
real-data parity sweep over 813 slices of 14mo XAU M15: zero divergence
(pre-fix: 7-9% of bars diverged); both original bug repros no longer
reproduce; cached end-to-end backtest run passes with the live confirm
map. Books shift slightly vs pre-fix runs — comparability note added to
CLAUDE.md; pinned tuned numbers need re-validation under the fixed
backtester.
Unlike USAGE.md (a USD/JPY-era fossil), this doc was structurally sound:
right pair, real architecture, correct composite formula, and 8 of its 9
regime-weight cells match engine.py. It is removed because both of its
sections now have live successors, and because the numbers it does state
are wrong in ways that mislead:
- Section 1 (trader-facing) → docs/user_guide.html + beginner_guide.html,
which are newer and cover the trailing-stop exit model this doc never
mentions.
- Section 2 (developer reference) → CLAUDE.md.
Stale/incorrect content it carried: the entire quality-grade table was
wrong (claimed A+ 45/70, A 30/60, B 20/50, C 12/40, D 8+; the code grades
40/65, 33/55, 28/50, 25/45, 15), Divergence's trending weight was listed
2.0 vs 1.0 in code, and it predates the 2026-07 work entirely — zero
mentions of XAU_PB or trailing exits (the actual default exit model), and
near-zero coverage of the EUR/USD port.
CLAUDE.md's header now points at docs/ as the human-facing docs instead
of disclaiming this file.
USAGE.md documented a different product: USD/JPY (unsupported — the
code is XAU/USD + EUR/USD), 16 main.py flags that don't exist
(--backtest, --optimize, --auto-session-filter, …; main.py has five),
and backtest.py/metrics.py, which aren't in the repo. Two claims were
actively misleading: it called the session filter "default off" (it is
True by default, 03:00-23:00 not 07-16) and gave regime-min-adx as 30
(real default 25). CLAUDE.md now covers the CLI, bot and backtester
correctly, so this is a delete rather than a rewrite.
PULLBACK_ENTRY_PLAN.txt was a pre-implementation draft whose spec no
longer matches what shipped — it specifies a 0.5-2.5 ATR retrace window
and a detector in engine.py, while XAU_PB ships 0.8-6.0 (the drafted
window rejected 81% of valid pullbacks) with the detector in
signal_aggregator.py so live and backtest share one code path. Its
validated outcome (+38.3R / 254 trades / 12mo M5) and every shipped
gate are recorded in CLAUDE.md and the config docstrings.
Also drops the now-dangling references to both files from CLAUDE.md
and the config.py XAU_PB comment.
Replaces the summary-level doc with a full spec derived from a read of
every core module, so contributors (and low-context models) don't have
to infer behavior or rediscover tuned values:
- repo map with line counts; flags DOCUMENTATION.md/USAGE.md as stale
- every config knob with its real default and the backtest evidence
behind it; marks the legacy/unused fields (SignalConfig.weights,
min_confidence, cooldown_bars, XAU h4/h1 gates) so nothing new is
built on dead knobs
- exact indicator logic for all 9 scored indicators + ATR context,
regime classification rules, regime-aware RSI zone tables, and the
regime-multiplier table copied from _compute_composite (the old
table did not match the code)
- aggregator pipeline in source order: MTF agreement, the 12 penalties
and the diminishing cap, exhaustion block, signal/grade thresholds,
and gates 6b-6i with exact numbers
- all three trade-setup generators with SL/TP geometry, spread gates,
and the trailing-exit model
- bot/CLI/news/data-provider/backtester mechanics incl. realism model
(raw-path resolution, next-open entry, SL-first ties, parallel replay)
- "Rules for making changes": validation acceptance rule, pair
isolation, live==backtest parity, ROC's load-bearing weight
Also corrects details the old doc got wrong: test count is 179 (not
~176), EUR ships off by default, backtest/*.csv are gitignored local
artifacts rather than test-enforced pins.
Two printable guides in docs/ (HTML source + weasyprint-rendered A4
PDFs): a trader-facing user guide (signal anatomy, trail management,
M5/M15/H1 comparison from the 12mo books, expectations, commands) and
a plain-language beginner edition (glossary, worked sizing example,
step-by-step trailing-stop walkthrough, H1-first guidance).
Fix 1 — mixed-"trending" regime (ADX strong, EMAs unaligned) was
hard-coded bullish in 5 sites: RSI reinterpretation used the uptrend
playbook (RSI 38-55 -> STRONG_BUY conf 75), BUY-only confidence
bonuses, BUY-only extension penalties, no counter-trend penalty.
Now: extremes-only RSI (as transitioning), no directional bonuses,
symmetric extension penalties. 6yr validation: H1 +67.2->+61.9R
(mixed skew 16B/11S -> 4B/7S), H4 +31.6->+27.5R (the 4 bias BUYs),
M15 +152.6->+154.7R, M5 +207.1->+206.1R with maxDD 28.7->24.3R.
The ~8R aggregate give-back is bull-market subsidy from the bug.
Fix 2 — live session filter now applies the EXACT replay-loop
predicate (session window + blocked hours on the signal bar's CLOSE
time, from the bar timestamp not wall clock). Replaces a wall-clock
filter gated on a "XAU STRATEGY" marker that no code writes, which
made live diverge from every validated book.
Fix 3 — delete dead _check_xau_oversold_bounce and
_generate_xau_trade_setup (unreachable; the latter hardcoded BUY and
skipped the spread/RR gates).
Backtester: parallel bar replay via forked workers (--workers,
default auto = cores-2 capped 12). Per-bar evaluation is pure; the
entry cooldown is applied in an ordered phase-2 pass. Proven
byte-identical to sequential on the same window; ~300 -> 1030-1680
bars/s (6yr M5: ~26min -> 5-8min). --workers 1 = sequential path.
Tests: 179 passed (new mixed-regime RSI regression test).
M15 mixed-"trending" regime block (ADX strong, EMAs unaligned =
momentum without direction): -6.8R/24 trades on 12mo, negative both
halves; blocking scores +65.7R/499 PF 1.38 DD 10.4R vs +59.0R/521
PF 1.32 DD 12.6R, better both halves, OOS 2024-06->2025-03 neutral
(-1.4R = the 4 such trades there). M5 never trades this regime.
New XauStrategyConfig.blocked_regimes / sell_blocked_regimes knobs
(default empty) + aggregator gate 6g2; M15 factory opts in.
M5 hour-9 UTC block (post-London-open whipsaw): -28.9R/143 over the
6-year book, negative 5/7 calendar years. Full-history replay with
block: +207.1R vs +178.3R, max DD 28.7R vs 35.5R, 2023 -13.0R ->
breakeven; tuned 12mo book unchanged (139.0 vs 139.3). Hour 9 is
positive on M15 (+1.7R 6yr, +8.6R OOS) so the block is factory
M5-only.
Rejected after validation (documented in xauusd.py + session memory):
transitioning-SELL block (bull-market artifact, +5.7R in 2024 OOS),
M15 trending_down block (M5 cross-TF contradiction), losing-streak
circuit breakers, 30d efficiency-ratio trend gate, and all entry-level
indicator filters (winners == losers at entry on every feature).
Full-history context (Twelve Data 5min starts 2020-03-16): M5 6.3yr
+207R PF ~1.3 post-fix with losing year 2021; M15 +152.6R PF 1.19
DD 49.7R — edge is regime-dependent, chop years are structural.
Tests: 178 passed (new gate test + factory-equivalence updates).
- TELEGRAM_CHANNEL_ID mirrors auto-scan signal + breach alerts to a
subscriber channel (Stars-subscription monetization; Telegram manages
membership). Feed keeps running even when the owner's DM auto-scan
is toggled off.
- _owner_only decorator locks all commands/buttons to TELEGRAM_CHAT_ID;
others get a refusal pointing at the channel.
- Every signal-bearing message (detailed signal, scan summary, all-TF
summary, breach alerts) now ends with a not-financial-advice /
educational-use disclaimer via a single _DISCLAIMER constant.
Surprise-conditioned entries lose 2-5p at every horizon both years;
avoidance filter marginal for EUR and profit-removing for XAU; the
H1-direction + M15-pullback prototype failed its pre-registered kill
criteria (2024-25 OOS negative). EUR stays informational-only.
Finishes PENDING_STEPS Step 1 under the current trailing-exit model.
M15 window 2025-03-30->2026-05-31 ($0.30 spread): +116.6R/460 PF 1.83
vs +105.4R/424 PF 1.82 baseline, better in both halves; 2024-06->
2025-03 OOS +30.3R vs +10.6R control. Same knob on the pinned M5 book
is WORSE (+131.3R/699 vs +139.9R/547) so the override is M15-only in
the xauusd factory, per the pre-registered decision rule. M5 path
unchanged (factory override gated on primary_tf == M15).
Replaying the live danger/caution gates against the validated 12mo XAU
M5 book showed they suppress 25-35 trades worth +4.6..+7.2R/yr — trades
entered near USD releases outperform the book average (+0.475R vs
+0.256R). New NewsConfig.event_risk_caps_signals (default False =
XAU-tuned) keeps EVENT RISK warnings in alerts but no longer guts
confidence; EUR factory re-enables the caps (evidence is gold-specific).
Pinned XAU M5 regression: 547 trades, +139.94R, trade-identical.
Tests: 177 passed.
Scrapes Forex Factory weekly pages (epoch datelines, actual/forecast/
impact/betterWorse) into backtest/cache/calendar.parquet (gitignored;
rebuild with the scraper). Timestamps verified: NFP/CPI/ECB times exact
across DST, and high-impact event bars average 3.5x ATR with the spike
vanishing at +/-1h shifts. calendar_store provides precise_events and
event_window_mask for screens and counterfactuals.
Five independent tests at 1-pip cost (M15 wide-open both windows,
transitioning-slice OOS, mean-reversion trade sims, forward-return
screens on M15+M5, H1 both windows) — nothing clears cost in both
2024-25 and 2025-26. Root cause: spread is ~10% of M15 ATR on EUR.
EUR stays out of scan_pairs; open avenues documented in docstring.
- telegram_bot: SCAN_PAIRS/_WS_TO_PAIR/SCAN_BUTTONS all derive from
PlatformConfig.scan_pairs (the single pair on/off switch); buttons and
keyboards are generated per pair; _run_signal_sync builds per-pair
configs via pair_signals factories (was: XAU defaults for any symbol);
scan-all works per pair; PB badge matches any *_PB tag; daily-rotating
file logs in logs/telegram.log (30-day retention)
- main.py: --pair argument (XAUUSD | EURUSD + aliases) routed through
pair_signals factories; price display is precision-aware (2dp gold,
5dp FX)
scan_pairs default stays [XAUUSD] until the EUR M5 re-tune validates.
- EURUSD aliases in PAIR_ALIASES
- main() builds PlatformConfig via pair_signals.PAIR_PRODUCERS factories
(per-pair regime weights/gates/sessions); unregistered pairs keep the
inline construction. XAU factory is asdict-identical to inline
(enforced by test) so XAU runs are unchanged.
- --spread default now resolves from PAIR_INFO default_spread
(XAU $0.30, EUR 0.0001) instead of a hardcoded gold value
- EUR strategy tag: renamed EUR_PB→EUR, detected in both bullish and
bearish reasons[0] (SELL setups mark the bearish list)
Smoke-validated: 30-day EUR M5 run produces EUR-tagged trades with
1-pip round-trip cost charged.
The SELL quality gate (6e), SELL volatility gate (6g), M5 noise gates
(6h), ATR trade-setup generator, and M5 pullback archetype are all
gold-tuned but previously keyed only on xau_strategy.enabled or the
timeframe, so they would have silently applied gold constants to any
other pair. Each now also requires display_name == "XAU/USD".
No behavior change for XAU (conditions are tautologically true for the
only registered pair today) — verified by test suite and pinned-window
backtest CSV diff.
Ported from feature/eurusd-support: PAIR_PRODUCERS registry with
make_config() factories per pair. xauusd.make_config is a pure
PlatformConfig-defaults wrapper (equality with the inline construction
is enforced by test); eurusd.make_config carries the donor branch's
regime weights and gates as a pre-realism baseline.
Adds 4 registry tests. 154/154 pass.
- EURUSD in PAIR_INFO; new per-pair price_precision and default_spread
fields on both pairs (backtester consumes default_spread when --spread
is omitted)
- EurUsdStrategyConfig ported from feature/eurusd-support plus EUR-local
cost knobs (round_trip_cost=1 pip, min_tp_cost_multiple=10,
min_sl_pips=8) so SignalConfig's gold-dollar values never apply to EUR;
donor-tuned values are a pre-realism baseline pending re-tune
- PlatformConfig.eur_strategy field; track EUR in news currencies
Ported from feature/eurusd-support (e6e09c5), pair-neutral fixes:
- _find_swing_points: all(<empty generator>) is vacuously True, so a value
surrounded by NaNs was reported as a swing point; now requires at least
one valid neighbor on each side
- ROC: guard against zero close at the lookback positions (inf scores)
- ADX: epsilon denominator instead of NaN-replace to avoid NaN cascade
Adds 8 regression tests (Module 9). 150/150 pass.
Improvements from a live-verified audit of the news/calendar pipeline:
- Currency tagging: keyword matching now uses letter-boundary regexes
and drops ambiguous "us "/"eu "/"uk " tokens — "versus" no longer
tags headlines as USD (confirmed live false positive polluting the
XAU/USD sentiment average).
- Fetch caching: calendar events and headlines are cached class-level
(10 min TTL, 60s retry after empty/failed fetch, lock-guarded for
concurrent scans). Stops per-scan refetching that was tripping the
Forex Factory rate limit; TTL stays under the 30-min pre-event
buffer so the danger window can't go stale.
- Surprise detection: EconomicEvent.surprise is computed from parsed
actual vs forecast (handles %, K/M/B/T suffixes, negatives). A
surprise print widens the post-release caution window to 45 min;
inline releases keep the original 15 min.
- Gold-aware sentiment: headlines are scored per currency, with
monetary-policy keywords (hawkish/rate hike/tightening vs dovish/
rate cut/easing/stimulus) inverted for XAU — "hawkish Fed" headlines
scored +1.0 for gold before, -1.0 now (verified on live feed).
- New sentiment sources: Google News gold-focused query and CNBC
Economy RSS (verified live; 40 → 74 unique headlines). Added
cross-feed title dedup so syndicated stories count once.
- events_today now buckets by UTC date (feed timestamps carry US-
Eastern offsets); --json output includes news_description.
- Removed dead NewsConfig.min_impact and unused HIGH_IMPACT_EVENTS.
Tests: 142 passed (7 new covering boundary matching, surprise
classification, post-event windows, cache reuse, gold inversion).
Second entry pattern alongside the Keltner-breakout: enter on the
retrace to EMA21/KC-mid in an established trend (EMA >=3/4 aligned,
ADX>=20, retrace 0.8-6.0 ATR, resumption bar closed back in trend
direction, RSI in the pullback zone not the extended zone). Detector
lives in the aggregator so live and GPU-backtest paths share one
implementation. Own gates (composite floor 15, confirmation-TF
agreement, MTF opposition, vol regimes) with a diagnostic funnel.
PB stops anchor to the pullback swing (~$8 vs ~$25 breakout), so PB
trades get their own trail arm (0.8R vs 0.6R global) via per-trade
trail overrides in the backtester and strategy-aware Telegram alerts.
12mo M5 validation: PB alone +38.3R/254 trades, positive both halves
and all four full quarters; core breakout trades undisturbed
(+101.9R vs +105.0 baseline). Combined book: 549 trades (~10.6/wk),
+140.1R, PF 1.78, DD 11.3R vs 311 trades/+105.0R before.
Design doc for a second M5 entry archetype: enter on retrace to
EMA21/KC-mid in an established trend instead of at Keltner-breakout
extension. Includes detector spec, geometry, validation protocol
(both-halves + quarterly + displacement checks), and the +105R
baseline to beat. Session/frequency knobs (blocked hours, cooldown)
were tested 2026-07-09 and confirmed already optimal — this is the
remaining evidence-backed entry-side direction.
Gates 5-6 in the M5 filter: block 'high' vol regime entries (chop
transition — 61% WR, negative total while low/normal/extreme all
positive) and block SELLs with RSI<25 (dip already extended, mean-
reverts before target). 365d: 419→311 trades, +41→+48.4R, DD 11.2→8.0R.
Replace the fixed RR-0.5 take-profit with a trailing stop: arm at
+0.6R favorable excursion, trail 0.75R behind best price, stop levels
update at bar close (conservative). The RR-0.5 target was the binding
constraint — winners kept running after TP. Whole trail parameter grid
(arm 0.4-1.2 × dist 0.5-1.25) lands +90-103R, a plateau not a spike;
also +91.7R on the pre-gate 419-trade population. TP1 stays advisory;
Telegram alerts include the arm price and trail distance. max_hold
200→300 bars (trailing exits need room; +10R).
12mo M5, same 311 trades: +105.0R total, PF 2.27, DD 7.9R, 61% WR,
train/val +54.6/+50.4, every quarter positive.
ATR was computed after EMAs so ribbon compression detection was dead code
on all timeframes — moved ATR first. Extended M5 swing S/R lookback from
50 to 120 bars. Added four M5-specific filters: min score, MTF agreement,
confirmation TF conviction, and SELL quality cap (A-grade M5 shorts enter
at dip extremes and lose). Fixed backtester blocked-hour check to use bar
close time matching the live bot. Added trade geometry config (spread gate,
SL floor, TP floor RR) and primary_tf mismatch guard in backtester.
Backtest: 1133 trades/+11.5R → 419 trades/+41.0R, PF 1.03→1.37, DD 21→11R.
Train/val split positive in both halves (+19.8R/+21.3R).
A/B on M15 2025-03-30→2026-05-31 with $0.30 spread + gap-through fills
(511 identical entries, only exits differ):
factor 0.8 / min_rr 0.4: +36.1R (avg +0.071R, SELL +1.2R)
factor 1.0 / min_rr 0.5: +66.8R (avg +0.131R, SELL +9.6R)
Worse in both train and validation halves and both directions. The 20%
tighter targets buy +1.8pt win rate but shrink every win while the fixed
spread stays constant. The knob remains for re-tuning with costs included.
WebSocket:
- Send heartbeat every 10s — Twelve Data drops idle connections, so the
monitor previously died on weekends/quiet hours and churned reconnects
- Log failed symbol subscriptions from subscribe-status events
- Stamp fresh live tick price + age into every alert message
Alerts:
- Re-enable level-breach alerts: WS breach triggers M15 rescan and sends
the report when actionable (was scanning and discarding the result)
- Dedup: identical (signal, quality, direction) per pair/TF is sent once,
re-armed after 1h — no more repeat alerts every candle
- Configurable minimum alert quality (C+/B+/A), cycled from the menu
- Trade setup shows dollar risk/reward alongside pips
Interactive menu (/start, /scan):
- Timeframe scans, Scan All, auto-scan ON/OFF toggle, alert-quality
cycle, status, and help as inline buttons
- Markdown-safe edits: 'message not modified' swallowed, parse failures
fall back to plain text; exception text escaped
Live pipeline:
- Drop still-forming candle before analysis (signals no longer repaint)
- Direction-aware MTF agreement + opposition gate (no trade against 2-of-3 TF consensus)
- SELL volatility gate: block shorts in low/normal vol regimes (config sell_vol_regimes)
- Divergence: 40-bar lookback, price/indicator swings must align within ±2 bars
(fired on 0.5% of trades before, 29% after), trending weight 2.0→1.0
- RSI regime_aware by default; transitioning regime only signals at extremes
- ADX ±DM tie no longer nudges bearish; TP2 clamped beyond TP1
Backtester realism:
- Charge $0.30 round-trip spread (--spread), gap-through SL/TP fills at bar open
- Resolve trades on RAW price path (cache stores raw bars, cleaning runs on load)
- Weekend/session filters use bar-close decision time to match live wall clock
Validated per-step on M15 2025-03-30→2026-05-31 under identical pricing:
+46.9R → +66.8R, win rate 69.9% → 73.0%, SELL side -2.3R → +9.6R.
Tested-and-reverted: ROC weight removal (-13R), BB fade suppression (-11.6R),
candlestick de-weight (-2R) — composite thresholds are co-tuned with weight mass.
The TP floor calculations used raw min_rr (0.4) but after 20% tightening
the effective RR dropped to ~0.32, failing the min_rr check and returning
None for all trade setups. Pre-compensate the floor with min_rr/tf so
post-tightening RR still meets the minimum.
M5 already broadcast through the shared _scan_timeframe path but was
documented as data-only (docstring, CLAUDE.md, DOCUMENTATION.md, plus a
phantom `if tf == M5: return` step that never existed in code). Align all
docs/comments so M5 alerting at C+ grade is the intended, documented behavior.
Remove the live dashboard entirely per request: the 30s refresh loop,
per-scan refresh, boot send, and the dashboard-only stats layer
(_signal_stats / _stats_updated / _update_signal_stats and its call sites,
_format_stats_dashboard and its helper constants). Drop now-unused ZoneInfo
import and dashboard mentions from help/autoscan/boot messages and the test
suite. The WebSocket level-breach monitor and level cache are untouched.
Tests: 136 passed, 0 failed.
The old `weekday() >= 5` check blocked all of Sunday, but the forex
market reopens Sunday 22:00 UTC (Sydney open). Now uses proper time-based
check so the bot fetches data and generates signals on Sunday evening.
A-grade trades entered exhausted moves with TP too far (0.75R), causing
57.7% WR and +0.25R over 30 days. Store exhaustion_count on SignalReport
and scale TP to B-level (1.0×) when A-grade has 2+ exhaustion tags.
Backtest: A-grade 73.1% WR / +3.25R, B/C unchanged.
- Blocked hours reduced to 0-2 UTC only (daily rollover)
- M5 signals now broadcast to Telegram like M15/H1/H4
- Live dashboard shows New York time (auto EDT/EST via zoneinfo)
News engine: expand sentiment keywords (18%→60% headline coverage), add
MyFXBook RSS as fallback calendar source when Forex Factory returns 429,
add retry/backoff on calendar API rate limits, update stale ForexLive URL
to investinglive.com, add date formats without seconds for MyFXBook RSS.
Telegram bot: remove flood control, rate limiter, and dashboard failure
tracking — all sends go directly through bot.send_message/edit_message_text.
- Move index dedup before OHLC consistency fix in _validate_and_clean
to prevent crash when df.loc[dup_idx] returns Series instead of scalar
- Add global flood backoff: detect RetryAfter, pause all sends/edits
until ban expires instead of retrying and escalating the ban
- Stop dashboard edit-fail → new-message spam cascade: track consecutive
failures, only resend after 3 failed edits
- Increase dashboard refresh from 5s to 30s, min send interval to 1.5s
- Add comprehensive QA test suite (140 tests across all 7 modules)
Timeframe keys stored as enum values (15min/1h/4h) but dashboard
looked for M15/1h/4h — GBP/USD showed no price or signal text.
Now uses Timeframe enum directly with display labels. Reduced
dashboard refresh from 60s to 5s with throttle guard.
Replace per-scan Telegram messages with a single dashboard message
that edits in-place every 60 seconds. Each pair shows all TFs
horizontally with short signal labels (Buy/Sell/Neutral). WebSocket
live prices feed into the dashboard. Level-breach alerts disabled
(WS still updates cache silently). Removed /stats command.
The general session filter ran after analyze_from_dataframes() and
wiped out XAU oversold bounce signals at most hours. Now skips the
session filter when the XAU strategy has already fired — the strategy
has its own backtested hour filter.
BUY-only strategy for gold: enters when M15 RSI drops below oversold
threshold while price remains above EMA200 on all three timeframes
(M15, H1, H4). Uses ATR-based TP/SL (0.7x/2.0x) sized from M15 ATR.
Isolated to XAU/USD only — other pairs completely unaffected.
- Drop USD/JPY from scan pairs (zero signals across all timeframes)
- Disable divergence indicator (zero impact in 14-config lab test)
- Parallelize evaluate.py with ProcessPoolExecutor (16 cores)
- Add --pair CLI filter to evaluate.py for single-pair backtests
- Add data freshness guard: stale cache auto-invalidated, primary TF
stale data suppresses signals to prevent trading on old data
- Add /cache command: shows fetch age and last bar per pair/TF
- Add /clearcache command: wipes in-memory cache for fresh API fetch
- Remove BB from WebSocket breach alerts (keep structural levels only)
- Show live prices only for WS-monitored pairs (EUR/USD, XAU/USD)
- Replace arrow emojis with text tags in indicator lines
DOCUMENTATION.md: Two-section guide covering the full project —
plain-English trader guide (indicators, scoring, trade setups) and
developer reference (architecture, data flow, every threshold/weight).
mt5.md: Architecture plan for automated order execution via MT5 bridge
(REST API on Windows, async client on Linux, risk-based lot sizing).
- Trade setup TP now scales to SL distance ensuring min RR is met when
no viable swing target exists; skip swing targets with poor RR
- Add float tolerance to RR gate to prevent edge-case rejection
- M5 auto-scan runs for level cache only, no Telegram broadcast
- WebSocket limited to EUR/USD + XAU/USD (Grow plan constraint)
- Register bot command menu via set_my_commands on startup
- Fix H4 boundary calculation using total minutes (hour*60+min) not just minute
- Add /help command listing all bot commands
- Add /status command showing WS connection, live prices, level cache, cooldowns
- Run initial M15 scan on boot to populate level cache before WS starts
- Update /start message to mention new commands
XAU/USD is not available in TwelveData's symbol list.
Swapped to USD/CAD across config, scan pairs, Telegram buttons,
and added CAD to tracked news currencies.
Expand from 2 to 4 pairs (EUR/USD, USD/JPY, GBP/USD, XAU/USD) with
automatic scanning at M15 and H1 candle close boundaries. Scheduler
staggers H1 by 60s to stay within 8 credits/min; daily budget is
480 of 800 allowed. Cache TTL is now candle-duration-aware so cross-TF
confirmation data is reused without extra API calls. Spread estimate
uses primary-TF data instead of fetching M5 separately.
Add EURUSD pair with dynamic pair info lookup. Telegram bot now shows
4 scan buttons: EUR/USD 15M, EUR/USD 1H, USD/JPY 15M, USD/JPY 1H.
Each indicator displays directional arrows in the message output.
Price formatting adapts per pair (5dp for EUR, 3dp for JPY).
Remove backtest.py, backtest_gpu.py, metrics.py, local CSV data,
Dukascopy loader, and all backtest CLI modes from main.py. Rebalance
engine weights: leading indicators (divergence, RSI, stochastic, BB)
now carry 66% of signal weight vs 34% for lagging (EMA, MACD). Add
ws_twelvedata.py for live tick streaming. Drop pyarrow and tqdm deps.
Introduces telegram_bot.py: python-telegram-bot v20+ async bot that
presents a persistent Scan Signal inline button. Each press fetches a
live USD/JPY signal via SignalAggregator and formats it with emoji,
MTF breakdown, top bull/bear reasons, and warnings. STRONG_BUY /
STRONG_SELL signals additionally render the full trade setup (entry,
SL, TP1, TP2, RR). Adds python-telegram-bot>=20.0 to requirements.txt.
Install nvidia pip packages for CUDA 12 runtime DLLs (no full toolkit needed):
nvidia-cuda-nvrtc-cu12, nvidia-cuda-runtime-cu12,
nvidia-curand-cu12, nvidia-cublas-cu12
Add _register_nvidia_dlls() in backtest_gpu.py that auto-discovers and
registers nvidia package DLL directories (site-packages/nvidia/*/bin/)
via os.add_dll_directory() before CuPy is imported. Suppresses the
spurious "CUDA path could not be detected" warning from cupy._environment.
Backend line now reads: CuPy 14.0.1 (RTX GPU)
Benchmark: scoring pass on 100k bars — NumPy 1.1ms vs CuPy 0.18ms = 6.3x.
Results are identical to CPU path (deterministic).
## What changed
### backtest_gpu.py (new — ~1,300 lines)
Standalone GPU-accelerated backtester completely separate from backtest.py.
Computes all indicator series in one vectorised O(N) pass over the full price
history, then replays positions with a sequential state machine.
Key design:
- VectorIndicators: pandas ewm/rolling (C-optimised) for all 7 indicators
- CuPy GPU backend with automatic NumPy fallback (no hard dependency)
- Two-phase API: prepare_arrays() once, score_and_run() many times
- --optimize flag: 36-combo weight sweep (1 baseline + 7 ablations + 28
multiplier runs) — calls prepare_arrays() once, replays all combos in seconds
- Full CLI: --tf, --local, --start/--end, --regime-filter, --regime-tf,
--session-filter, --optimize, --journal, --journal-csv
O(N) vs O(N^2): backtest.py recomputes all indicators per bar (O(N^2));
backtest_gpu.py computes each indicator series once and reads bar T in O(1).
On 100,000 H1 bars: ~4 seconds vs >10 minutes.
### data_provider.py
Added load_local_csv() method — reads MetaTrader tab-delimited CSV exports
from market_data/ directory. Format: Time|Open|High|Low|Close|Volume|Spread
(spread column dropped on load, timestamps localised to UTC). Applies the same
_validate_and_clean() pipeline as the Twelve Data path. LOCAL_CSV_MAP maps
each Timeframe enum to its filename.
market_data/ files added (MetaTrader exports, UTC):
USDJPY_M15.csv 100,000 bars 2022-05-04 to 2026-05-08
USDJPY_H1.csv 100,000 bars 2010-04-23 to 2026-05-08
USDJPY_H4.csv 25,855 bars 2010-04-22 to 2026-05-08
USDJPY_D1.csv 5,020 bars 2010-04-22 to 2026-05-08
16 years of H1/H4/D1 vs ~6 weeks of H1 on Twelve Data free plan. Use --local
for all backtests; Twelve Data reserved for live signal feed.
### engine.py — optimised indicator weights
36-combo sweep on H4 2022-2026 found MACD and EMA trend are the dominant
signals; RSI and Bollinger are net drag on this trending pair.
Old → new weights:
ema_trend: 1.2 → 3.6 (×3 — strongest directional signal)
macd: 1.1 → 3.3 (×3 — ablating it is the single worst move: -42%)
stochastic: 0.8 → 1.6 (×2 — clean timing, 40% WR at this level)
adx: 1.0 → 0.5 (÷2 — partially redundant with EMA stack)
ichimoku: 1.1 → 0.55 (÷2 — lags too much at H4, hurts in ranging)
rsi: 0.3 → 0.0 (ablated — net drag across all test windows)
bollinger: 0.9 → 0.0 (ablated — counter-trend noise on trending pair)
RSI and Bollinger are still computed and displayed in live output for
reference; weight=0 means they contribute nothing to the composite score.
Combined result vs baseline (H4 2022-2026, local data):
Return: +7.12% → +21.30% (+14.2pp)
Max DD: -9.05% → -8.52% (tighter — gains did not come at expense of risk)
Sharpe: 0.23 → 0.47 (2x improvement)
Sortino: 1.38 → 1.91
Win Rate: 36.3% → 38.1%
### backtest.py
Added 16-thread CPU parallelism for signal pre-computation via
ThreadPoolExecutor. _precompute_all_signals() dispatches each bar to a worker;
pandas/NumPy C extensions release the GIL enabling true parallel execution.
Results are collected and replayed in the main loop, replacing the per-bar
recompute. Also added --regime-tf CLI flag so regime filter timeframe can be
specified independently of the primary TF.
### signal_aggregator.py
EMA200 distance gate: confidence *= 0.55 when price is within 0.5% of EMA200
(chop zone) or beyond 1.5% (overextended). Sweet spot is 0.5-1.5% from EMA200
where trending momentum is cleanest. Strongest single filter found: lifted
cumulative return +13.88% to +35.50% in ablation window.
### ENGINE_LOGIC.md (new)
Full reference document covering: architecture, data pipeline, all indicator
logic, composite scoring formula, optimised weights with rationale, regime
classification, EMA200 gate, MTF alignment, grade system, backtesting results,
filter catalogue, and configuration reference.
Across 2024 trend / 2023 chop / 2026 hostile windows, distance from the
H1 EMA200 was empirically the most discriminative variable. Sweet spot
0.5-1.5% from EMA200; closer = false-breakout chop, farther = chase.
Outside the band, confidence is multiplied by 0.55 so the trade falls
below the B-grade threshold and is rejected.
Cumulative return across the 3 windows: +13.88% baseline -> +35.50%.
Win rate improved in every regime; 2023 chop turned positive (-2.56% to
+14.94%).
Also extends the trade journal with per-timeframe MTF scores, RSI at
entry, and EMA200 distance for future filter mining; surfaces ema_trend
in key_levels so the gate can read it.
Session filter restricts trades to a UTC hour window (default 07-16,
London + NY) — opt-in via --session-filter, with --session-start /
--session-end overrides. Wired into both the backtester (cheap hour
check next to the regime gate) and live mode (downgrades signal to
NEUTRAL/F when current UTC hour is outside the window).
Auto-toggle (--auto-session-filter) tracks rolling mean pnl_pct over
the last N closed trades; when it falls below threshold, the session
filter activates, releasing once expectancy recovers. Default lookback
10 trades, threshold -0.3%.
Trade dataclass extended with diagnostic fields captured at entry:
entry_hour, entry_dow, ADX, ATR%, market_regime, volatility_regime,
mtf_agreement. These let --journal-csv exports surface loss-pattern
clusters when a window underperforms.
Validated on 2024 trend / 2023 chop / 2026 recent windows. Auto-toggle
preserves ~50% of trend edge while still cutting ~25% of recent-regime
losses — pure regime filter is still best on cumulative, so both
session flags ship default-off.
data_provider now uses TDClient.time_series().as_pandas() instead of
hand-rolled REST calls. Throttle, validation, and parquet caching are
unchanged. Adds twelvedata>=1.2.0 to requirements.txt.
Restructure print_signal_report so decision (signal · grade · score ·
confidence · MTF) sits at the top, followed by trade setup, context line,
MTF breakdown, top supporting/opposing reasons in two columns, warnings,
and an optional detail block under --detail. Adds _compact_reason helper
that strips known indicator name prefixes via _INDICATOR_PREFIXES so
reason lines stay short.
--regime-filter previously affected only backtests. Live mode now also
checks higher-TF ADX after analysis: signals are downgraded to NEUTRAL
with a warning when ADX < threshold, and confirming when above.
This brings live behavior into parity with what the backtester measures.
Framework:
- TechnicalConfig.disabled_indicators skips named indicators in engine.analyze
- Optimizer supports disable_indicator override for parallel ablation sweeps
- main.py --ablate runs leave-one-out across all 10 directional indicators
Cleanup applied (validated across 2024 trend / 2023 chop / 2026 recent):
- Drop pivot by default (cumulative +9.5% to disable)
- Drop candlestick by default (+3.4% cumulative)
- Regime-gate ichimoku: 1.3x in trending, 0.2x in ranging (was a chop villain)
- Trim RSI weight 1.0 -> 0.3 (cumulative noise across regimes)
Result: +10.60% on 2024 trend (+33.24% -> +43.84%, Sharpe 2.17 -> 2.77),
+15.71% on 2023 chop (-18.27% -> -2.56%). Recent regime still adverse.
script.txt was superseded by requirements.txt. Python bytecode should
not be version-controlled — added .gitignore to keep cache and venv
out of the repo going forward.
Harwinder Sekhon
harry