136 Commits

Author SHA1 Message Date
Harwinder Sekhon da61cff509 docs: architecture diagram (PNG + matplotlib generator)
End-to-end map: external services → providers/news layer → signal core
(engine + factory + aggregator) → SignalReport → four delivery surfaces,
plus the backtester replaying the same core. Regenerate with
venv/bin/python docs/architecture_diagram.py after architecture changes.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Aq1jZ6jBsBdEC2xT3u46SA
2026-08-07 09:21:52 -04:00
Harwinder Sekhon 15a4ed7831 dashboard: clickable news — detail dialogs + on-demand AI summary endpoint
- news_engine: RSS parser captures link/description/publisher (three new
  optional NewsSentiment fields); _clean_html_text/_safe_link helpers
- news.json: headlines carry id = sha1(headline) + enrichment fields
- static/news_summary.php (NEW): on-demand DeepSeek gold-impact summary,
  whitelist-gated against news.json ids, cache-forever in ai/, fail-closed
  file-based daily cap (150/day), key at /etc/gold-dashboard/deepseek.key;
  installed to the docroot by install_static
- dashboard.html: newswire + calendar detail dialogs (score tiles, source
  description, article link, AI summary with loading/error states);
  delegated on stable containers, dialog keyed by id across polls
- tests: +6 (357 total); docs: README-dashboard §6 one-time setup
  (key file, apache-owned ai/, SELinux httpd_can_network_connect), CLAUDE.md

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Aq1jZ6jBsBdEC2xT3u46SA
2026-08-07 09:08:49 -04:00
root a42ef16113 oanda: retry once on transient 401/5xx/network faults in candle fetch
The practice API rejects ~15-20 requests/day with a spurious 401
"Insufficient authorization" while the same token succeeds on adjacent
requests; each one silently cost a full scan cycle on that timeframe.
One retry after RETRY_DELAY_S (2 s) absorbs these plus transient
5xx/network faults. Real credential failures fail both attempts and
raise as before; other 4xx (bad request, 429) are never retried.

Tests: transient-401 recovery, persistent-401 raise, 400 not retried.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Tujtmmhxj9mxKjPFSoVjmP
2026-08-06 23:06:29 -04:00
Harwinder Sekhon b3022ef5c5 dashboard: Gold Desk v2 UI — light/dark theme, TF detail dialog, wire score
Owner-supplied page design replacing the dark-only v1: Bootstrap 5
colour tokens with a luma-inverted dark mode (toggle + system-pref +
localStorage), compact clickable TF cards opening a full-detail modal
(all-nine indicator table with value/conf/weight, price ladder of key
levels vs spot, per-confirmation-TF drilldowns, provenance), newswire
aggregate sentiment box, Inter/JetBrains Mono webfonts.

Re-applied the c70f58a sync fixes the source copy predated: 5 s
conditional-GET polling and the +2 s boundary refetch sniper. Data
logic otherwise unchanged; all fields verified against
report_to_webapp_dict output.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_017mbXs9HRawZhD1uZ3ze7zQ
2026-08-06 22:44:48 -04:00
Harwinder Sekhon c70f58a572 dashboard: cut page sync latency to ~scan-time + ≤5 s
Three display-side fixes, no signal-logic changes:
- run_cycle publishes snapshot.json BEFORE the news refresh — news I/O
  (FF + 4 RSS + possible DeepSeek calls) took 20-30 s on TTL-expired
  cycles and kept fresh signals off the page (publish_all split into
  publish_snapshot/publish_news, kept for boot)
- page polls every 5 s (was 20 s) using conditional GETs: dropped the
  ?t= cache-buster + no-store so Apache answers unchanged files with
  bodyless 304s
- boundary refetch sniper re-armed: fires 2 s after next_scan_epoch
  regardless of distance (the old delay<POLL_MS guard would have been
  dead code at 5 s polling)

New Module 20 test locks the snapshot-before-news ordering. 352 tests.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_017mbXs9HRawZhD1uZ3ze7zQ
2026-08-06 22:35:13 -04:00
Harwinder Sekhon b831001cfb news: DeepSeek validation, vs_forecast fix, /status source health
Validation (llm_news_eval.py harness + 78-headline golden CSV + verdict
doc): headline scoring 78/78 vs keyword baseline 24/78 — validated;
event gold-direction calls misread close K/M numeric comparisons ~11%
of the time (both claims cases in sample, JOLTS 8.79M vs 8.84M).

Fix: event payload now carries code-computed "vs_forecast" (from
EconomicEvent.surprise) and the prompt gains an explicit inverted-
metrics rule + worked claims example — iteration 1 (field alone) fixed
the arithmetic but exposed a masked inversion error. 18/18 on live
re-verification (4 misreads fixed, 11 regression cases clean, 3 fresh
both-side probes).

/status: new "News & AI" section — per-source health for the calendar
chain (which of FF/MyFXBook served), each RSS feed, and DeepSeek
(daily count, last error kind), written passively by the existing
fetch paths under _cache_lock. Failures WARN once on transition
(repeats DEBUG, recovery INFO) so the rotating log carries outage
stories without 10-min spam.

test_all.py: +5 Module 19 tests (351 total); also carries the Module 22
fixed-RR-TP tests from the prior research session (shared file).
Replay artifacts dir gitignored like commit_ladder.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_017mbXs9HRawZhD1uZ3ze7zQ
2026-08-06 22:21:02 -04:00
Harwinder Sekhon daaed8e6c7 research: --fixed-rr-tp backtest flag + 2026-08-05 verdict (NOT shipped)
Hard TP at N×risk beats the trail on all 2024+ windows but the grid is
monotone to 3R (trend artifact) and M15 rr20 goes negative in 2021+2023
with doubled DD — trailing exits stay. Flag is research-only; its
Module 22 tests land with the next commit (shared test_all.py).

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_017mbXs9HRawZhD1uZ3ze7zQ
2026-08-06 22:20:43 -04:00
Harwinder Sekhon fc5ce85493 Remove OANDA_MIGRATION_ROADMAP.md (reference-only doc, superseded)
Deletion was already staged in the working tree by the owner; the
roadmap was approved 2026-07-24 as reference only and its plan file
lives outside the repo. Content remains in git history.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01UnZkzeFcTu9agFg8BDh2tn
2026-08-04 20:26:54 -04:00
Harwinder Sekhon 45ce97ed13 audit: NY-anchored weekend calendar (all 3 copies) + FB2_EMA_LO crash-proofing
Owner decisions from the 2026-08-04 audit report:
- spread-in-exit-triggers: DECLINED (books stay comparable; ledger in
  CLAUDE.md records the measured ~4-6% optimism).
- weekend clock: FIXED. The market week is Fri 17:00 -> Sun 18:00 New
  York time, not Fri/Sun 22:00 UTC — during US DST the Oanda feed stops
  at 21:00 UTC (measured: 159/162 summer weekends). All three weekend
  implementations (providers/base, signal_aggregator, gpu_backtest
  calendar filter) now share the NY-anchored rule; a new test sweeps
  them hourly across both DST transitions and asserts agreement. No NTP
  needed — the host clock was fine, the constant was wrong.
  Ends the weekly false-"STALE DATA / feed lagging" hours (summer Fri
  21-22 UTC, winter Sun 22-23 UTC).
- FB2_EMA_LO (delegated): kept as a research knob but crash-proofed —
  an unparseable value degrades to the tuned default 0.5 with a warning
  instead of killing every scan. Documented in CLAUDE.md.

Book gates: M15 12mo pinned book byte-identical (+78.7111R, 397
trades). M5 12mo drops exactly ONE trade (556 -> 555, +97.56 ->
+98.64R): a phantom whose signal fired on the final summer-Friday bar
(decision 21:00 UTC = NY 17:00 market close) and was "filled" at
Sunday's reopen 49 hours later, -1.08R — an entry live trading can
never take. The backtest no longer contains it; live == backtest
parity improved. M15 unaffected because hour 21 is in its blocked set.

Suite: 341/341 green.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01UnZkzeFcTu9agFg8BDh2tn
2026-08-04 20:21:34 -04:00
Harwinder Sekhon 8eca2f4044 audit: scheduler cleanup, WS-log secret hygiene, simulator test coverage, CLAUDE.md sync (Batch G)
2026-08-04 Fable 5 QA audit, final code batch.

- Boundary math: removed the provably-dead `remaining <= buffer` guard
  from all three scheduler copies (518,400-instant frozen-clock sweep:
  never fired) and documented the exact-boundary-instant semantics.
- websockets/websockets.client loggers pinned to INFO so a DEBUG run
  can never write the TD key (connect URI) or Tiingo auth frame into
  the 30-day rotating log. (.env also chmod 600, untracked.)
- Cooldown pass factored into _apply_cooldown (pure, testable) — the
  only sequential replay rule finally has direct tests.
- New simulator behavior tests (previously ZERO coverage of the trade
  resolver): same-bar SL+TP tie → SL-first, gap-through-SL fills at
  the open, TIMEOUT pricing + round-trip spread charging, trailing
  ratchet arms at bar close and fills next-bar-only, cooldown ordering.
- CLAUDE.md synced to verified reality: 338 tests/21 modules; three
  weekend-implementation copies caveat; throttle claim corrected
  (single-flight, TD-only throttle); daemon trail/cooldown parity
  caveats (once-per-OBSERVED-bar, wall-clock cooldown) + OPEN_ONLY and
  reconcile notes; CP-path gate semantics documented as validated
  behavior; new "known open items" section for the owner-decision
  findings (weekend DST constant, spread-in-trigger rule-6 item, H4
  seeding divergence, FB2_EMA_LO).

Gates: suite 338/338 green; pinned M15 book and M5 HEAD baseline both
reproduced identically after the cooldown refactor.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01UnZkzeFcTu9agFg8BDh2tn
2026-08-04 20:08:37 -04:00
Harwinder Sekhon fc40ba081f audit: alert delivery, provider robustness, backtester NaN/cache guards (Batches C+E+F)
2026-08-04 Fable 5 QA audit. Three batches sharing one test file, all
verified by runnable repros before fixing, all gated below.

Batch C — telegram alert delivery (verified by h4 harness):
- Failed breach rescan shrinks the cooldown to 30 s instead of
  silencing the symbol for 5 min; breach during cooldown is PARKED and
  fired on the first tick after expiry (was dropped forever).
- Level cache keyed symbol→TF (M5 scans no longer wipe H1/H4 swing
  levels within minutes); breach labels TF-qualified.
- Markdown alert sends get a plain-text fallback (mirroring _safe_edit);
  fingerprint commits only when a message actually landed.
- Per-key asyncio.Lock closes the dedup TOCTOU between auto-scan and
  breach sends (forced-interleave repro produced duplicate alerts).
- _start_scheduler respawns dead per-TF loops (was all-or-nothing).
- Clean WS close now backs off (was a zero-delay reconnect hot loop);
  backoff resets only after a connection that received ticks; TD
  iter_ticks fires on_connect on the first RECEIVED frame (invalid key
  can no longer pin the backoff at minimum).
- Scan-failure logs carry the real traceback (was "NoneType: None").

Batch E — providers (verified by h5 harness):
- fetch_range: 3 consecutive chunk ERRORS raise instead of returning
  silently truncated history (verified 731-day silent gap repro; also
  covers 429 storms); pagination no-progress guard; head-truncation
  ERROR log.
- Single-flight per (symbol,TF): 4 concurrent scans = 1 API call (was
  4); cache records requested depth so a longer-lookback call refetches.
- Multi-bar forming/future tail fully dropped (was iloc[:-1] once).
- Oanda: malformed candles skipped, never 0.0-defaulted; cleaning drops
  non-positive-price rows BEFORE OHLC repair; inf returns neutralized so
  one corrupt bar can no longer disarm the whole 10-sigma filter.

Batch F — backtester internals (verified by h6 harness):
- NaN bar in a trailing window exits at the last valid bar (was NaN R
  poisoning every metric); static-path TIMEOUT NaN guard;
  compute_metrics excludes non-finite R loudly. Zero real-data exposure
  (0 NaN rows in all 10 Oanda parquets) — defensive.
- fetch_history ERROR-logs pre-2026-07-08 cache files (cleaned-bars era)
  when loaded for raw-path resolution.

Gates: suite 333/333 green; pinned M15 12mo book reproduced identically
(397 trades, +78.7111R) after E and after F; M5 HEAD-config baseline
reproduced identically (556 trades, +97.5604R) after F.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01UnZkzeFcTu9agFg8BDh2tn
2026-08-04 20:02:57 -04:00
Harwinder Sekhon f7924af3b0 audit: gate-chain verdicts — lock shipped M5 entry-path state, document hour-delta isolation
2026-08-04 audit, Batch D. Verification outcomes (no behavior changes —
comment + tests only):

- CP gate-bypass (quality forced to B after the gate chain, exhaustion/
  M5-score gates not re-checked) is VALIDATED behavior, not a bug: the
  pinned shipped-config M5 12mo book carries 226/586 XAU_CP trades
  (+38.9R), 104 with EXHAUST warnings (+15.3R), 175 below the M5
  min-score gate (+28.0R). Removing the "holes" removes validated
  profit. New test pins the shipped state (PB off, CP BUY-only on M5,
  ema_partial_neutral M5-only) so any flip is a conscious revalidated
  decision.
- E18 (xau_strategy vs backtest blocked-hours "drift") REFUTED as a
  defect: the divergence is required by pair isolation — the backtest
  copy is the generic non-XAU fallback and must not inherit
  gold-validated deltas. Documented in the factory + a test asserting
  the delta stays XAU-side.
- 6c-upgrade re-arming an exhaustion-D refuted (exhaustion zeroes
  confidence below 6c's floor; regimes mutually exclusive). "WEEKEND
  FILTER" CP-suppression branch is defense-in-depth for direct
  analyze_from_dataframes callers, kept.

Suite: 316 green.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01UnZkzeFcTu9agFg8BDh2tn
2026-08-04 19:52:07 -04:00
Harwinder Sekhon 7395aa5473 audit: GPU-engine cache-miss observability + dead-guard alignment + full parity test
2026-08-04 audit, Batch B. Verified first (h1 harness): the two engines
are byte-identical on every reachable input — 370-prefix flat-bar deck,
warmup band, 322 real-data M15 positions incl. all 7 zero-range bars,
zero diffs. The claimed ADX NaN desync / warmup-guard divergences are
unreachable below the shared 30-bar analyze() floor. Changes here are
therefore observability + hygiene, proven behavior-neutral by exact
reproduction of the pinned M15 12mo book (397 trades, +78.7111R,
byte-identical trade rows).

- _resolve_pos: cache-miss fallback to live-engine recompute now warns
  on first miss and counts every miss (was fully silent — a run could
  switch math with no trace).
- ROC/Keltner warmup guards textually aligned with the live engine so a
  future period/floor change can't open a silent divergence window.
- New regression: full-sweep parity test comparing ALL indicator
  signals/confidences, composite, and regimes on every prefix incl. a
  flat zero-DM/zero-TR run (the old parity test skipped
  ADX/MACD/ROC/Keltner/RSI entirely); plus a loud-fallback test.

Suite: 314 green. Book gate: oanda_m15_12mo_030 reproduced identically.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01UnZkzeFcTu9agFg8BDh2tn
2026-08-04 19:49:08 -04:00
Harwinder Sekhon c9829a2a67 audit: harden Oanda execution lifecycle (close-fill, netting, races, fail-closed config)
2026-08-04 Fable 5 QA audit, Batch A. All fixes are execution-robustness
only — no signal code touched (git diff scope: daemon/executor/tests).
Findings verified by runnable repros before fixing (h3 harness):

- close_trade: a 200 whose close order was created but NOT filled (e.g.
  MARKET_HALTED during the settlement pause) now raises instead of
  returning; the daemon's timeout path keeps tracking and retries. The
  old path popped state and orphaned a live position forever
  (reconciliation was boot-only).
- place_market_order: positionFill=OPEN_ONLY — on a netting account an
  opposite-side order can no longer silently close an untracked
  position; the divergence surfaces as an order cancel.
- _execute_trade: a fill without tradeOpened (netting close/reduce) and
  any placement OandaAPIError (server may have filled before a network
  timeout) now trigger _reconcile_positions instead of silent return.
- _discover_existing_positions → lock-guarded _reconcile_positions,
  callable mid-run; adoption of a position with NO broker stop now
  screams (CRITICAL + per-bar warning) instead of computing sl=entry,
  risk=0 and silently never trailing.
- _manage_trailing: snapshot deep-copies per-trade dicts (no mutation of
  live state outside the lock); write-back skips trades popped by
  _check_closed_trades mid-iteration (no ghost resurrection).
- Fail-closed config: unknown OANDA_QUALITY_GATE refuses to start
  (previously a typo silently became the most permissive gate "C");
  non-integer OANDA_LOT_SIZE/MAX_HOLD_BARS exit with a clear message.

Suite: 302 → 312 tests (Module 21), all green.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01UnZkzeFcTu9agFg8BDh2tn
2026-08-04 19:46:37 -04:00
Harwinder Sekhon 3c9bb2f017 M5 at real cost: disable XAU_PB, squash partial EMA alignment
Both validated at the measured $0.65 spread on Oanda bars (12mo halves +
PF, OOS 2024-06->2025-03, 5yr; full evidence in pair_signals/xauusd.py
M5 overrides 4-5):

- m5_pullback_enabled False on M5: loss forensics identified XAU_PB as
  the largest wrong-direction loss source; 12mo PF 1.26 (failing the
  acceptance bar) -> 1.44, max DD 23.9R -> 13.9R. PB code stays for
  future regime-gated experiments.
- New TechnicalConfig.ema_partial_neutral (default False), True on M5
  only: partial 4-EMA ribbon alignment (2-3 of 4 checks) scores NEUTRAL
  instead of a weak BUY/SELL (-0.008R/84 vs +0.230R/341 fully aligned).
  Squash mirrored in the backtester's precomputed EMA scorer in this
  same commit (live==backtest parity). 12mo +97.6R/556 PF 1.48 DD
  10.7R halves +47.8/+49.8; 5yr +127.4R PF 1.38; OOS one-trade tie.
- Rejected and logged (memory + docstring): hour-19 block (OOS fail),
  hours-14/15, trail dist 0.75, confirm-TF swaps incl M15+H4 (OOS
  ordering), divergence/candlestick weight changes, bollinger removal.
- Tests 296 -> 302: partial-alignment fixture test, factory-equivalence
  mirrors, per-TF isolation test, BC-gate fixture force-enables PB.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01H1eKWZsAVapJ1C4winYLqX
2026-08-04 16:54:09 -04:00
Harwinder Sekhon 90801d665d /news: normalize rounded-zero score so "-0.00" renders as "+0.00"
A balanced feed (XAU and USD means nearly cancelling) yields a tiny
negative net that "+.2f" rendered as "-0.00" — seen live 2026-08-03.
Regression test uses that day's actual per-currency means (300 tests).

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_017975c1TufZ3xRYAq8iWTCS
2026-08-03 18:06:44 -04:00
Harwinder Sekhon cf84f7b1a7 Telegram /news command: net XAU/USD news sentiment score on demand
Owner-locked /news replies with the existing get_pair_sentiment() net
score (XAU mean - USD mean, -1..+1) and its label; a caveat line marks
the no-headlines case so a feed outage can't read as neutral. Fetch runs
in the shared executor with a placeholder-then-edit reply. Display-only:
news still never gates signals. Registered in handler block, command
menu, help text, docstring, and user guide; 3 new builder tests (299).

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_017975c1TufZ3xRYAq8iWTCS
2026-08-03 18:01:42 -04:00
Harwinder Sekhon 21c16977e8 Dashboard publishes directly into /var/www/trade (no subfolder)
Adds exactly three files next to the portal: dashboard.html,
snapshot.json, news.json. README covers chown vs setfacl for write access.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Wc9ZoqKLYBT2QJcpRWKoGm
2026-08-03 17:17:37 -04:00
Harwinder Sekhon 7a281125e9 Dashboard out-dir default → /var/www/trade/gold-dashboard (live portal docroot)
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Wc9ZoqKLYBT2QJcpRWKoGm
2026-08-03 17:14:08 -04:00
Harwinder Sekhon 4efe30508b Web dashboard v2: static publisher + Apache, with calendar & news panels
- dashboard_publisher.py: standalone candle-close daemon that atomically
  writes snapshot.json/news.json/dashboard.html to an Apache-served dir
  (DASHBOARD_OUT_DIR). Fully independent of telegram_bot/oanda_daemon —
  local confirmation map + boundary math, no imports either way, no new
  deps, no open port. Due-TF classification from the RAW wake time.
- static/dashboard.html: self-contained dark page — hero rail, four TF
  cards (score bar, 3x3 indicator grid, MTF, setup, reasons), DeepSeek
  market context, week economic calendar, scored newswire. Clock offset
  from the HTTP Date header (publish-time epoch is stale between closes).
- news_engine.get_week_events(): additive read-only accessor over the
  cached FF week; no signal path calls it.
- deploy/: systemd unit + live-server (LAMP/SELinux) checklist.
- test_all.py: Module 20 (9 tests) + get_week_events test; 296 green.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Wc9ZoqKLYBT2QJcpRWKoGm
2026-08-03 17:10:29 -04:00
Harwinder Sekhon 356a09e5ab Add commit-ladder campaign driver; ignore its 26MB results folder
Added: backtest/research/commit_ladder.py (recreates historical configs
via --set on the fixed harness; 2026-08-03 campaign confirmed HEAD is
the best config on all timeframes). Results stay local like all pinned
trade books.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Nsd8HcrV67GbCZXxZEn1Ma
2026-08-03 13:29:18 -04:00
Harwinder Sekhon 8521ab909a DeepSeek V4-Flash news enrichment: LLM sentiment, surprise direction, summary
Three display/warning-only upgrades to the news engine, all behind the
existing keyword path as fallback — signals are never gated on LLM
output (event_risk_caps_signals / sentiment_affects_signals stay False
per the 2026-07 audit; backtester use_news=False untouched):

- Headline sentiment: one batched call per 600s cache window (capped at
  25 new headlines, sha1-cached so repeats are never re-sent) overrides
  the keyword scores per currency with the gold-inverse doctrine encoded
  in the prompt.
- Release surprises: get_event_risk_level's description now says which
  way the print pulls GOLD ("— gold bearish: strong jobs lift yields"),
  fixing the documented numeric-only gap; risk level and windows
  unchanged.
- Market summary: 1-2 sentence macro blurb on SignalReport.news_summary,
  rendered in Telegram alerts (📰) and the CLI (🧠).

New llm_client.py: requests-only OpenAI-compatible client (no SDK), one
attempt per call, UTC-day request cap, singleton so the cap survives
per-scan NewsEngine reconstruction. Config: deepseek block in
config.json + DEEPSEEK_API_KEY env; absent key = fully inert.

Live-measured API behavior baked in: V4-Flash is a hybrid thinking model
— requests send thinking:disabled (reasoning otherwise blew the timeout
and could exhaust max_tokens before the answer); prompts are static
module constants so DeepSeek's prefix cache prices repeat calls at the
cache-hit rate; scores use compact [s, xau] pairs; timeout 12s.

Tests 273 → 286 (new Module 19 + deepseek config tests).

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Nsd8HcrV67GbCZXxZEn1Ma
2026-08-03 13:25:02 -04:00
Harwinder Sekhon f0d76021d1 Tiingo WebSocket streaming: re-enable the level-breach monitor
The monitor has been dead since the Oanda feed switch (Oanda's provider
does not stream). New config.json scalar data_providers.streaming selects
a tick-stream provider independent of the candle feed; telegram_bot
resolves it via _get_streaming_dp() (separate instance, candle cache
stays on the active provider) and falls back to the active provider when
unset — behavior is byte-identical to before when the key is absent.

TiingoDataProvider grows the two sanctioned streaming overrides:
supports_streaming=True and iter_ticks() on wss://api.tiingo.com/fx
(mid price only — the no-spread rule extends to the stream). Two
deliberate divergences from the TD reference implementation, both
because Tiingo authenticates inside the socket: server "E" messages
raise TiingoDataError (a swallowed auth error would end the generator
cleanly and reconnect-hammer with zero backoff), and on_connect fires on
the first received message rather than after the subscribe send. Field
order of quote frames was live-verified: ask is data[7], not data[6] as
the docs table suggests (data[6] is askSize).

make_data_provider(config, name=None) now takes an explicit provider
name; keyless streaming providers disable the monitor with one warning
instead of a traceback-per-minute loop.

Tests 266 → 273 (WS parser, ticker round-trip, streaming config scalar,
explicit-name dispatch, streaming-dp resolution, monitor auto-disable).

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Nsd8HcrV67GbCZXxZEn1Ma
2026-08-03 13:11:36 -04:00
Harwinder Sekhon b936d443b8 Cleanup: remove dead code, stale docs, and all non-main branches
- Delete back-compat shims data_provider.py / oanda_provider.py;
  test_all.py now imports providers.* directly (ForexDataProvider name
  retired; the two shim-only tests removed, 268 -> 266 tests)
- Delete backtest/research/calendar_store.py (no importer, no entry
  point); config.py audit comment reworded
- Delete executed plan doc (2026-07-21 centralized-config) and the two
  unbuildable guide PDFs; HTML guides remain the source of truth
- OANDA_MIGRATION_ROADMAP.md status header corrected: phases 1-2
  shipped 2026-07-28, feed is live on Oanda; phases 3-6 still unbuilt
- Branches deleted locally + on both remotes: oanda_demo (merged),
  web-dashboard, eurusd-modular, reversal-exit-alert (owner-approved
  permanent deletion); July stash dropped

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_019zR58p8ACW1oJ5QXVUZ5es
2026-08-03 11:08:48 -04:00
Harwinder Sekhon 0396970d29 Big-candle gate (default-OFF) + M5 relaxation campaign + --set tooling
Gate 6h (big-candle entry gate): blocks all XAU entries (core, XAU_PB,
XAU_CP) when the primary TF's last 24h contains a bar with range >= K x
prior-bar ATR. Ships default-OFF: validation rejected every threshold
(K 3-6 on M5/M15 12mo) — removed trades were net-POSITIVE at every cell;
M5 quiet-window trades are the only losing bucket. Config knobs
big_candle_atr_multiple/lookback_hours, --big-candle backtest flag,
Module 17 tests.

M5 relaxation SHIPPED (validated 2026-07-30, Oanda bars, $0.30 spread):
m5_min_abs_score 30->25, m5_min_confirmation_score 10->5, hour-21 UTC
unblocked M5-only. Beats control in ALL 6 calendar years of the 5yr
replay (+230.9R/2319 PF 1.26 DD 21.3 vs +168.0R/2048 PF 1.21 DD 27.0),
12mo +107.4R PF 1.36 vs +84.6R PF 1.33, OOS +26.1R vs +17.2R, $0.70
spread +86.9R vs +66.4R. M15: every relaxation retested negative —
config unchanged. Declined-but-open: M5 hour-22 block, M15 hour-16
unblock (both fail 5yr/OOS). Evidence in the factory docstring.

Tooling: generic --set PATH=JSON research override in gpu_backtest
(typo-proof, logged); diagnose_filters filter_tags refreshed for gates
6d-6i with a staleness-guard test (Module 18); research scripts
hour_analysis/burst_profile/book_diff. Suite: 268 tests.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_014jqHpmNYmwYMowF6bPZyeD
2026-07-30 11:30:20 -04:00
Harwinder Sekhon a46aa9ccd9 Rebuild CLAUDE.md: lean architecture reference
Replaces the deleted exhaustive knob-by-knob reference with a compact
guide: commands, config precedence, pipeline architecture, rules for
changes, and current-state notes (Oanda feed, TD-pinned books).

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_014jqHpmNYmwYMowF6bPZyeD
2026-07-30 11:30:07 -04:00
Harwinder Sekhon d2d4fb4ac6 Switch active data source to Oanda v20 (practice feed)
config.json.example: data_providers.active = "oanda" (credentials stay
in env/.env or the top-level oanda block; TD and Tiingo remain
registered fallbacks). Backtest parquet cache is now keyed per provider
— Oanda and TD bars diverge (2026-07-24 study), so a feed switch must
never serve another feed's cached bars; twelve_data keeps the legacy
un-prefixed filename so pinned-book caches stay valid. 254 tests green;
live smoke-tested against the practice API (66 contract checks: all
scanned TFs, boundary/freshness guarantees, real spread $0.71,
fetch_range pagination, typed error paths, CLI + backtest end-to-end).

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Mc7XKYyPySVXiUpcN82QEK
2026-07-28 20:12:59 -04:00
Harwinder Sekhon c8beb7e3ae Provider abstraction: providers/ package with BaseDataProvider template
One class per provider (TwelveData, Oanda, Tiingo), all subclassing
BaseDataProvider which owns cache, validation/cleaning, freshness +
boundary guarantees, fetch_range pagination, and streaming stubs —
subclasses implement only _fetch_raw. config.json data_providers.active
selects the live feed; data_provider.py / oanda_provider.py stay as
back-compat shims. Module 16 provider-abstraction tests (253 total green).

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Mc7XKYyPySVXiUpcN82QEK
2026-07-28 20:09:00 -04:00
Harwinder Sekhon 95f8c47d4f Research: Dukascopy real-spread study + Oanda/TD bar-divergence study
Scripts + summary CSVs for the 2026-07-24 studies (real XAU spread ~$0.65
tradable hours; Oanda vs TD bars diverge ~11% of ATR with 10% direction
flips). Large raw data files (parquet, per-boundary samples) gitignored.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Mc7XKYyPySVXiUpcN82QEK
2026-07-28 20:08:53 -04:00
Harwinder Sekhon e254f068fd Add Oanda migration roadmap (reference, not scheduled)
Approved 2026-07-24: 6-phase plan for fully switching the signal feed
from Twelve Data to Oanda v20 (inert plumbing -> backtester agnosticism
-> pricing-stream monitor -> shadow parity -> book re-validation with
owner cost gates -> reversible cutover). TD remains a permanent
fallback. Grounded in the 2026-07-24 divergence study (10% bar-direction
flips, 11%-of-ATR noise, real Oanda spread ~$0.72 vs assumed $0.30).

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01YNvxXtgz62F7kTKZZsAAAE
2026-07-24 15:12:37 -04:00
Harwinder Sekhon 5259e6e039 Oanda demo daemon: auto-execution on practice account (XAU/USD)
New independent trading daemon (oanda_daemon.py) that places orders on
an Oanda v20 practice account instead of sending Telegram alerts.
Oanda-only market data (oanda_provider.py, mid candles + real bid/ask
spread) through the unchanged signal pipeline; order execution and
trade management via oanda_executor.py (raw requests, no new deps).

Book parity with the backtester: no broker-side TP (exits = SL/TRAIL/
TIMEOUT only), bar-close-only trail ratchet with each closed bar
processed exactly once, 5-bar per-TF entry cooldown, structural SL
from the setup. Accepted divergence: one position total (skipped
signals logged for book reconciliation).

Safety/ops: fail-fast boot on rejected credentials (transient outages
continue), market orders never retried (double-fill risk), entry lock
against aligned-boundary races, restart recovery + adoption via
oanda_state.json, hourly heartbeat, asyncio exception handler, every
failure path logged. Prices formatted at instrument precision.

Config: env OANDA_* -> config.json "oanda" section (validated in
app_config). Tests: Module 15 (30 tests), suite 240 green.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01VUnCqjWBuvKRSjLwHeB9BJ
2026-07-24 14:10:45 -04:00
Harwinder Sekhon 0fc9000934 Telegram alert redesign: verdict-first layout + informational 100-pip TP
format_signal_message restyled (display-only, zero signal-logic impact):
verdict header first (signal - pair - TF - quality), trade setup moved
above the detail, per-indicator breakdown condensed to one arrow line
(_indicator_summary), heavy dividers dropped. New TP 100p line in every
setup block: entry +/- 100 pips ($10.00 on XAU) with its R-multiple vs
the stop (_INFO_TP_PIPS, formatter-only - not stamped on TradeSetup,
not in CLI/JSON/backtester). Trailing-plan logic unchanged (stamped
per-TF params, pullback fallback). Removed dead _indicator_line and
_SIGNAL_TAG. Tests 210 -> 212 (100p TP math BUY/SELL/no-setup, one-line
indicator summary).

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01BoZKi67Ct6QTeiFJmybXSw
2026-07-24 10:58:22 -04:00
Harwinder Sekhon 08eae15e11 Remove Tiingo spread monitor (indicative two-tier feed, unvalidatable)
Tiingo's free /tiingo/fx/top publishes only a two-tier indicative
spread for gold ($0.10 liquid hours / $0.70 rollover+news; verified on
273 production samples + live probes) — mid ± a posted increment, not
executable depth, so it can never validate the $0.30 backtest cost
assumption. Removed one day after shipping: tiingo_provider.py, the
sampler/quote plumbing in telegram_bot.py, the channel_spread_detail
config knob and tiingo_api_key resolver, and Module 14's Tiingo tests
(221 -> 210). CLAUDE.md documents the verdict: do not re-add a spread
display from Tiingo's free FX feed.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01BoZKi67Ct6QTeiFJmybXSw
2026-07-24 10:56:55 -04:00
Harwinder Sekhon 89734c3a87 Accuracy audit fixes: alerts carry validated per-TF trail plans + 13 more
Five-agent audit of the signal path; all fixes are delivery/measurement
side. Regression-proven behavior-neutral on validated signals: the 5yr
M15 replay reproduces the pinned book exactly (+210.65R PF 1.35
WR 57.8% MaxDD 33.90R). Tests 215 -> 221.

- TradeSetup carries trail_arm_r/trail_dist_r stamped from the producing
  config; Telegram trail plan reads them (was showing default 0.6/0.75
  instead of the validated M15 arm 0.70 / M5 dist 0.85)
- Alert dedup: fingerprint = (signal, quality, direction); entry movement
  compared against last-ALERTED entry/risk (old SL-dist bucket churned on
  ATR drift -> near-duplicate alerts every candle)
- Fresh fetches enforce the closed-bar boundary check with one 3s grace
  retry, then warn (feed lag no longer silently analyzes the prior bar)
- Backtest max DD includes the zero baseline (opening losing streaks
  were undercounted; pinned M15 DD unchanged)
- Replay loads the regime-filter TF when the H4-ADX gate is on (was a
  silent no-op for M5/M15 primaries)
- clear_cache under _cache_lock; dead ROC zero-cross branches removed
  from both engines (provably unreachable); RSS dates via
  parsedate_to_datetime (named zones no longer stamped UTC); event risk
  grades from the soonest event; .env inline comments stripped;
  auto-scan skips NEUTRAL like the breach path; report_to_dict honors
  the weight-0 skip rule and emits strategy/trail params; PB funnel
  counts spread_gate/accepted; session-filter fail-open paths log loudly
- Docs corrected to match validated code (H&S neckline = lower peak,
  exhaustion anchors farthest tracked swing) with re-validation warnings

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01R5soRUeRBi8kHMB99csrJy
2026-07-23 10:31:13 -04:00
Harwinder Sekhon 91b4a6bb5e Tiingo bid/ask spread monitor + logging hardening
- tiingo_provider.py: /tiingo/fx/top quotes (10s TTL cache, graceful
  None on any failure); spread log writer. Signals never consume
  bid/ask; warn-only by design (no historical bid/ask to validate a
  gate against — live == backtest rule).
- app_config: repo-root .env loader (setdefault; shell env still wins),
  tiingo_api_key resolver (TIINGO_API_KEY/TIINGO → config.json),
  telegram.channel_spread_detail flag (default false).
- telegram_bot: 5-min spread sampler → logs/spread_log.csv (weekend-
  aware), alert-time quote refresh, bid/ask + wide-spread warning in
  signal messages, /status spread line. Raw numbers to owner DM always;
  channel only while channel_spread_detail=true (Tiingo free tier is
  internal-use-only; flip before public launch). Free-tier limit math:
  worst case ~41 req/h vs 50/h cap, one pair.
- Logging: 3h rotation x 240 backups (30-day retention), full-date
  stamps, force=True root handlers; candle loops + sampler survive
  failed cycles; breach-task exceptions logged via done-callback;
  PTB error handler; sys/threading excepthooks in main().
- test_all.py: Module 14 (tiingo, dotenv, spread routing, loop
  resilience, rotation) — 215 tests green.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01HVRmy6KspMs96o5PTPYFAB
2026-07-22 17:18:14 -04:00
Harwinder Sekhon 0823c05868 gitignore: cover M5 trail-grid research CSVs
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01VPNF37MP9Kuv62B1VBrZ1b
2026-07-21 19:39:08 -04:00
Harwinder Sekhon 3ee79b5202 M5 trail dist 0.85 (validated grid, arm unchanged)
Same-day sibling of the M15 campaign, opposite finding: on M5 the arm
point barely matters — winners need a looser leash. 5yr grid plateau
arm 0.5-0.7 x dist 0.85-1.0; shipped the OOS-preferred edge 0.60/0.85:
+231.2R PF 1.30 vs +218.0R PF 1.29 baseline, both halves better, DD
flat (19.2 vs 19.1), OOS +14.5R vs +11.1R. dist 1.0 rejected at the
OOS gate despite best in-sample total. XAU_PB pb_trail unaffected.
Factory replay reproduces the grid cell; 201/201 tests.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01VPNF37MP9Kuv62B1VBrZ1b
2026-07-21 19:13:01 -04:00
Harwinder Sekhon bf8e5fbf39 M15 trail arm 0.70 (validated grid) + backtest/live parity hardening
Parity: replay now applies the live _apply_regime_filter_live method
under the live guard (function-identity, default-off); unregistered-pair
fallback applies the config.json overlay; parity sweep gains direct
ATR/vol-regime assertions; regime-filter semantics test added.

Trail campaign: --trail-arm/--trail-dist research flags; 30-cell grid on
5yr M15 refuted the earlier-arm hypothesis and found the 0.70-0.80 arm
plateau. Shipped M15-only trail_arm_r=0.70 (factory override):
+210.7R PF 1.35 WR 57.8% vs +175.7R PF 1.30 WR 53.9% baseline; both
halves and 2024-06->2025-03 OOS better (+57.4 vs +45.4); 5yr max-DD
clause (33.9R vs 29.1R) explicitly waived by owner. M5 unchanged.
Factory replay reproduces the grid cell trade-for-trade; 201/201 tests.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01VPNF37MP9Kuv62B1VBrZ1b
2026-07-21 16:40:33 -04:00
Harwinder Sekhon 11ff49befb Remove dead code: legacy config fields, unused helpers/imports, stray files
- Delete test_ws_dual.py (standalone probe, unused) and a stale tracked log
- config.py: drop MarketSession enum, is_jpy_pair, candle_buffer_seconds,
  and the never-read legacy SignalConfig/XauStrategyConfig fields
- data_provider.py: drop uncalled get_latest_price/get_spread_estimate
- Remove unused imports (engine, telegram_bot, main, news_engine)
- test_all.py: drop/trim the corresponding defaults assertions (199 tests)
- CLAUDE.md updated to match

Zero behavior change: no production code read any removed field
(verified reference-by-reference); suite green at 199/199.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01VPNF37MP9Kuv62B1VBrZ1b
2026-07-21 14:06:55 -04:00
Harwinder Sekhon 847fa18053 Merge branch 'centralized-config': config.json deployment overlay
Centralized deployment settings (Telegram credentials, data-provider
keys/selection, scan pairs/timeframes, per-pair base blocked hours,
alert quality, log level) into a gitignored config.json with env-var
precedence. Hardcoded credentials removed from source. Zero signal-
logic change — parity pinned by tests (200/200).

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01VPNF37MP9Kuv62B1VBrZ1b
2026-07-21 13:44:34 -04:00
Harwinder Sekhon f24009a784 Final review fixes: scrub tracked credentials, trim scannable TFs, absolute parity pin, scanning type-checks
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01VPNF37MP9Kuv62B1VBrZ1b
2026-07-21 13:42:00 -04:00
Harwinder Sekhon 00380448c3 gitignore: cover .env.* swap files and the whole logs/ directory
Rotated logs (telegram.log.YYYY-MM-DD) don't match *.log and a vim
.env.swp was nearly committed by a bulk add.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01VPNF37MP9Kuv62B1VBrZ1b
2026-07-21 13:30:21 -04:00
Harwinder Sekhon 798f4dec99 Docs: correct PlatformConfig overlay scope (credentials never pass through the factory)
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01VPNF37MP9Kuv62B1VBrZ1b
2026-07-21 13:30:15 -04:00
Harwinder Sekhon 3f46505af4 Docs: config.json deployment overlay (CLAUDE.md repo map, env table, test count)
Documents the app_config.py / config.json overlay shipped in Tasks 1-4:
repo-map rows for app_config.py and config.json.example, env-var table
precedence (env -> config.json -> none, hardcoded fallbacks removed
2026-07-21), PlatformConfig/pair_signals overlay call order, and the
real 199-tests/13-modules suite count. Also fixes a stale SCAN_PAIRS
comment in telegram_bot.py that still named PlatformConfig.scan_pairs
as the deployment switch instead of config.json's scanning.pairs.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01VPNF37MP9Kuv62B1VBrZ1b
2026-07-21 13:25:21 -04:00
Harwinder Sekhon a283806a5e telegram_bot: credentials/quality/scan-loops from config.json, startup guard
- Credentials, alert quality, scan pairs/timeframes, log level and
  candle-loop intervals now resolve via app_config (env -> config.json
  -> code default) instead of hardcoded fallbacks.
- _get_shared_dp() uses make_data_provider() for provider-registry
  dispatch.
- main() refuses to start without a token/chat_id, raising SystemExit
  with an actionable message instead of a bare sys.exit string.
- test_all.py Module 13: two new tests covering credential/quality/
  loop wiring via a reload, and the startup guard.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01VPNF37MP9Kuv62B1VBrZ1b
2026-07-21 13:18:05 -04:00
Harwinder Sekhon 44cadf7a9f Data provider: key from config.json/env, provider registry, drop hardcoded key
Resolves the Twelve Data API key through app_config.provider_api_key()
(env → config.json → no default) instead of a hardcoded field on
PlatformConfig. Adds data_provider.PROVIDER_REGISTRY + make_data_provider()
so signal_aggregator and the backtester build the active provider through
one seam; telegram_bot.py gets the same one-line WS-key fix now (per Task 4's
already-planned edit for that line) so the suite doesn't crash on import
between this commit and Task 4's full wiring pass.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01VPNF37MP9Kuv62B1VBrZ1b
2026-07-21 13:10:58 -04:00
Harwinder Sekhon 3669f5e33e Wire config.json overlay into the pair factory (before per-TF deltas)
Implements Task 2: apply_pair_overrides in xauusd.make_config before
the validated per-TF deltas, so e.g. M5 hour-9 appends on top of the
JSON base. Updated Module 9 equivalence test and added three parity pins.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01VPNF37MP9Kuv62B1VBrZ1b
2026-07-21 12:59:12 -04:00
Harwinder Sekhon 7455fcaa6c app_config: reject non-object config.json sections with AppConfigError
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01VPNF37MP9Kuv62B1VBrZ1b
2026-07-21 12:54:46 -04:00
Harwinder Sekhon 2c070fb9b3 Add app_config.py: config.json loader, validation, env-precedence resolvers
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01VPNF37MP9Kuv62B1VBrZ1b
2026-07-21 12:47:48 -04:00
Harwinder Sekhon 8de77deadd Plan: centralized config.json implementation (5 TDD tasks)
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01VPNF37MP9Kuv62B1VBrZ1b
2026-07-21 12:44:24 -04:00
Harwinder Sekhon bd506752ec Add config.json.example template; gitignore config.json
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01VPNF37MP9Kuv62B1VBrZ1b
2026-07-21 12:32:28 -04:00
Harwinder Sekhon bd59b5cd5b Spec: centralized config.json design (deployment settings overlay)
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01VPNF37MP9Kuv62B1VBrZ1b
2026-07-21 12:25:08 -04:00
Harwinder Sekhon a3305c81b9 News informational-only: gate sentiment bonus/penalty off by default
Add NewsConfig.sentiment_affects_signals (default False), gating the +5
aligned-sentiment confidence bonus and the 10-pt news_conflict penalty.
With event_risk_caps_signals already False, live decisions are now pure
technical, matching the backtester (use_news=False). Warnings and the
sentiment display are unchanged. Adds a gating regression test.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01FMaaPdrrrQuh1VvGQH1icN
2026-07-17 16:03:31 -04:00
harry 85b7567677 Merge pull request 'Remove EUR/USD support — XAU/USD-only codebase' (#1) from xau-only-chart-patterns into main
Reviewed-on: http://192.168.55.11:3000/harry/ForexBot2/pulls/1
2026-07-16 20:09:21 +00:00
Harwinder Sekhon de6424ec3f Enable BUY-only chart patterns on M5 — bearish patterns dropped (OOS negative)
M5 factory: cp_enabled=True, patterns limited to inv_hs, double_bottom,
bull_flag. SELL CP lost -3.21R on OOS; BUY CP: +41.8R PF 1.60 on 12mo,
+65.2R PF 1.48 on 5yr. Adds +22R to the M5 book (708 vs 513 trades).
M15/H1 remain disabled — all failed OOS validation.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_015kby7M9JwpYi8NQT2yoxAi
2026-07-16 16:08:13 -04:00
Harwinder Sekhon ca37a7ab5a Docs: CLAUDE.md — document XAU_CP archetype (pipeline step 11, generator D, cp_* knobs)
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_015kby7M9JwpYi8NQT2yoxAi
2026-07-16 14:31:45 -04:00
Harwinder Sekhon a70fc91f96 Add chart-pattern entry archetype (XAU_CP) — disabled by default
Third entry archetype: completion of classical chart patterns (double
bottom/top, H&S top / inverse H&S, bull/bear flag) — the tight canon
with the strongest documented stats. New chart_patterns.py detector
module (pure functions, ATR-unit geometry, closed-bar slices only,
first-close-beyond trigger discipline so a breakout fires exactly
once and never repaints).

Pipeline step 10 in analyze_from_dataframes: fills only empty setup
slots, allows weak-composite F grades (normal at reversal completion)
but inherits session/day blocks, mirrors MTF-opposition + SELL
vol/hour + M5 protections. Geometry: SL = pattern anchor -/+ 0.5 ATR
(min $8), TP1 = measured move, $9 spread gate, trailing exit does the
real work. Strategy tag XAU_CP.

cp_enabled defaults to False everywhere — per-TF adoption happens in
pair_signals/xauusd.py only after the standard validation rule passes.
Backtester: --cp / --cp-patterns flags to test the candidate, XAU_CP
tag recognition.

Suite: 183/183 (21 new tests: detectors, trigger discipline,
invalidation, gates, pipeline, default-off safety).

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_015kby7M9JwpYi8NQT2yoxAi
2026-07-16 14:23:17 -04:00
Harwinder Sekhon 68a62b0a53 Remove EUR/USD support — XAU/USD-only codebase
EUR/USD never had a validated edge (2026-07-11/12 verdict: no config
passed validation on M5/M15/H1 at 1-pip cost; cost-to-volatility root
cause). Removed: EurUsdStrategyConfig, pair_signals/eurusd.py, EUR
gates 6h + weekend/day filters + strategy marker + trade geometry in
signal_aggregator, EUR pair registry/aliases in main.py and
gpu_backtest.py, EUR/USD news pair mappings, test_all.py Module 10.
Generic multi-currency news keyword infra kept (GBP/JPY/EUR keyword
table is not pair code). Full EUR history remains at 7d3fd95.

Suite: 162/162 green (was 187; 25 EUR-specific tests removed).

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_015kby7M9JwpYi8NQT2yoxAi
2026-07-16 14:06:27 -04:00
Harwinder Sekhon 7d3fd9502e Fix on Sell order lossing by blocking time 2026-07-16 12:17:56 -04:00
Harwinder Sekhon f118ade3cf Audit fixes: cache boundary staleness, 6e session bypass, GPU engine parity
Fixes all high+medium findings from the 2026-07-16 full-codebase audit.

High:
- data_provider: cache now rejects snapshots missing the latest closed
  bar (_has_latest_closed_bar, weekend-exempt). The wall-clock TTL served
  mid-candle snapshots at the next boundary, so live M15/H1 scans ran one
  full bar behind and the bar-close session check evaluated the wrong hour.
- signal_aggregator: gate 6e (SELL quality) only acts on actionable
  grades. It was re-grading session-blocked F reports to D, which
  re-opened the M5 pullback path (entry check: quality != "F") on
  session-blocked bars.
- gpu_backtest: port the live EMA-ribbon compression/expansion logic
  (+10 conf on expansion) into the GPU engine. Its absence diverged
  composite score/conf on 7-9% of bars and flipped signals on ~0.2%.

Medium:
- gpu_backtest: Bollinger precompute uses the exact pandas rolling calls
  (cumsum E[x2]-E[x]2 lost precision at gold price levels; bb_position
  error up to 7.6e-3 crossed signal thresholds, worst during squeezes).
- gpu_backtest: --confirm-tfs defaults to the pair factory's live
  confirmation map instead of a hardcoded 1h,4h.
- gpu_backtest: entries re-anchor to the first RAW bar after the signal
  bar, so >10-sigma spike bars dropped by cleaning are no longer skipped
  at entry (favorable bias).
- gpu_backtest: per-TF warmup pad — D1 confirms get >=210 bars at window
  start (62-day pad gave ~61 bars: no EMA200, unlike live).
- data_provider: thread-safety locks for cache + throttle (shared
  provider across the bot's 4-thread pool); outlier filter keeps the
  snap-back bar after a spike instead of punching a two-bar hole.
- telegram_bot: alert fingerprint gains a risk-relative entry bucket
  (materially-moved setups re-alert) and is committed only after a
  successful send (failed sends no longer consume the alert for 1h).
- engine: disabled_indicators=["macd"] now removes the MACD pillar from
  scoring (histogram still computed for divergence).

Verification: test_all.py 187/187 (new Module 11: 8 audit-fix
regressions incl. a live-vs-GPU parity sweep with ribbon coverage);
real-data parity sweep over 813 slices of 14mo XAU M15: zero divergence
(pre-fix: 7-9% of bars diverged); both original bug repros no longer
reproduce; cached end-to-end backtest run passes with the live confirm
map. Books shift slightly vs pre-fix runs — comparability note added to
CLAUDE.md; pinned tuned numbers need re-validation under the fixed
backtester.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
2026-07-16 10:24:39 -04:00
Harwinder Sekhon bdcd886722 Docs: delete DOCUMENTATION.md — both halves superseded
Unlike USAGE.md (a USD/JPY-era fossil), this doc was structurally sound:
right pair, real architecture, correct composite formula, and 8 of its 9
regime-weight cells match engine.py. It is removed because both of its
sections now have live successors, and because the numbers it does state
are wrong in ways that mislead:

- Section 1 (trader-facing) → docs/user_guide.html + beginner_guide.html,
  which are newer and cover the trailing-stop exit model this doc never
  mentions.
- Section 2 (developer reference) → CLAUDE.md.

Stale/incorrect content it carried: the entire quality-grade table was
wrong (claimed A+ 45/70, A 30/60, B 20/50, C 12/40, D 8+; the code grades
40/65, 33/55, 28/50, 25/45, 15), Divergence's trending weight was listed
2.0 vs 1.0 in code, and it predates the 2026-07 work entirely — zero
mentions of XAU_PB or trailing exits (the actual default exit model), and
near-zero coverage of the EUR/USD port.

CLAUDE.md's header now points at docs/ as the human-facing docs instead
of disclaiming this file.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
2026-07-16 09:34:04 -04:00
Harwinder Sekhon 2a85f1e23c Docs: delete stale USAGE.md and superseded pullback plan
USAGE.md documented a different product: USD/JPY (unsupported — the
code is XAU/USD + EUR/USD), 16 main.py flags that don't exist
(--backtest, --optimize, --auto-session-filter, …; main.py has five),
and backtest.py/metrics.py, which aren't in the repo. Two claims were
actively misleading: it called the session filter "default off" (it is
True by default, 03:00-23:00 not 07-16) and gave regime-min-adx as 30
(real default 25). CLAUDE.md now covers the CLI, bot and backtester
correctly, so this is a delete rather than a rewrite.

PULLBACK_ENTRY_PLAN.txt was a pre-implementation draft whose spec no
longer matches what shipped — it specifies a 0.5-2.5 ATR retrace window
and a detector in engine.py, while XAU_PB ships 0.8-6.0 (the drafted
window rejected 81% of valid pullbacks) with the detector in
signal_aggregator.py so live and backtest share one code path. Its
validated outcome (+38.3R / 254 trades / 12mo M5) and every shipped
gate are recorded in CLAUDE.md and the config docstrings.

Also drops the now-dangling references to both files from CLAUDE.md
and the config.py XAU_PB comment.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
2026-07-16 09:30:50 -04:00
Harwinder Sekhon 79ea70c8d7 Docs: rewrite CLAUDE.md as a complete codebase reference
Replaces the summary-level doc with a full spec derived from a read of
every core module, so contributors (and low-context models) don't have
to infer behavior or rediscover tuned values:

- repo map with line counts; flags DOCUMENTATION.md/USAGE.md as stale
- every config knob with its real default and the backtest evidence
  behind it; marks the legacy/unused fields (SignalConfig.weights,
  min_confidence, cooldown_bars, XAU h4/h1 gates) so nothing new is
  built on dead knobs
- exact indicator logic for all 9 scored indicators + ATR context,
  regime classification rules, regime-aware RSI zone tables, and the
  regime-multiplier table copied from _compute_composite (the old
  table did not match the code)
- aggregator pipeline in source order: MTF agreement, the 12 penalties
  and the diminishing cap, exhaustion block, signal/grade thresholds,
  and gates 6b-6i with exact numbers
- all three trade-setup generators with SL/TP geometry, spread gates,
  and the trailing-exit model
- bot/CLI/news/data-provider/backtester mechanics incl. realism model
  (raw-path resolution, next-open entry, SL-first ties, parallel replay)
- "Rules for making changes": validation acceptance rule, pair
  isolation, live==backtest parity, ROC's load-bearing weight

Also corrects details the old doc got wrong: test count is 179 (not
~176), EUR ships off by default, backtest/*.csv are gitignored local
artifacts rather than test-enforced pins.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
2026-07-16 09:26:06 -04:00
Harwinder Sekhon a6c9493406 Docs: user guide + beginner guide PDFs
Two printable guides in docs/ (HTML source + weasyprint-rendered A4
PDFs): a trader-facing user guide (signal anatomy, trail management,
M5/M15/H1 comparison from the 12mo books, expectations, commands) and
a plain-language beginner edition (glossary, worked sizing example,
step-by-step trailing-stop walkthrough, H1-first guidance).

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
2026-07-12 13:42:21 -04:00
Harwinder Sekhon 53d8bc4390 Audit fixes: mixed-regime bias, live==backtest session parity, parallel replay
Fix 1 — mixed-"trending" regime (ADX strong, EMAs unaligned) was
hard-coded bullish in 5 sites: RSI reinterpretation used the uptrend
playbook (RSI 38-55 -> STRONG_BUY conf 75), BUY-only confidence
bonuses, BUY-only extension penalties, no counter-trend penalty.
Now: extremes-only RSI (as transitioning), no directional bonuses,
symmetric extension penalties. 6yr validation: H1 +67.2->+61.9R
(mixed skew 16B/11S -> 4B/7S), H4 +31.6->+27.5R (the 4 bias BUYs),
M15 +152.6->+154.7R, M5 +207.1->+206.1R with maxDD 28.7->24.3R.
The ~8R aggregate give-back is bull-market subsidy from the bug.

Fix 2 — live session filter now applies the EXACT replay-loop
predicate (session window + blocked hours on the signal bar's CLOSE
time, from the bar timestamp not wall clock). Replaces a wall-clock
filter gated on a "XAU STRATEGY" marker that no code writes, which
made live diverge from every validated book.

Fix 3 — delete dead _check_xau_oversold_bounce and
_generate_xau_trade_setup (unreachable; the latter hardcoded BUY and
skipped the spread/RR gates).

Backtester: parallel bar replay via forked workers (--workers,
default auto = cores-2 capped 12). Per-bar evaluation is pure; the
entry cooldown is applied in an ordered phase-2 pass. Proven
byte-identical to sequential on the same window; ~300 -> 1030-1680
bars/s (6yr M5: ~26min -> 5-8min). --workers 1 = sequential path.

Tests: 179 passed (new mixed-regime RSI regression test).

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
2026-07-11 22:57:37 -04:00
Harwinder Sekhon f3ba4d56a3 XAU: two validated per-TF gates — M15 mixed-regime block, M5 hour-9 block
M15 mixed-"trending" regime block (ADX strong, EMAs unaligned =
momentum without direction): -6.8R/24 trades on 12mo, negative both
halves; blocking scores +65.7R/499 PF 1.38 DD 10.4R vs +59.0R/521
PF 1.32 DD 12.6R, better both halves, OOS 2024-06->2025-03 neutral
(-1.4R = the 4 such trades there). M5 never trades this regime.
New XauStrategyConfig.blocked_regimes / sell_blocked_regimes knobs
(default empty) + aggregator gate 6g2; M15 factory opts in.

M5 hour-9 UTC block (post-London-open whipsaw): -28.9R/143 over the
6-year book, negative 5/7 calendar years. Full-history replay with
block: +207.1R vs +178.3R, max DD 28.7R vs 35.5R, 2023 -13.0R ->
breakeven; tuned 12mo book unchanged (139.0 vs 139.3). Hour 9 is
positive on M15 (+1.7R 6yr, +8.6R OOS) so the block is factory
M5-only.

Rejected after validation (documented in xauusd.py + session memory):
transitioning-SELL block (bull-market artifact, +5.7R in 2024 OOS),
M15 trending_down block (M5 cross-TF contradiction), losing-streak
circuit breakers, 30d efficiency-ratio trend gate, and all entry-level
indicator filters (winners == losers at entry on every feature).

Full-history context (Twelve Data 5min starts 2020-03-16): M5 6.3yr
+207R PF ~1.3 post-fix with losing year 2021; M15 +152.6R PF 1.19
DD 49.7R — edge is regime-dependent, chop years are structural.

Tests: 178 passed (new gate test + factory-equivalence updates).

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
2026-07-11 21:38:42 -04:00
Harwinder Sekhon 43d9064d1e Telegram: subscriber channel feed, owner lock, full risk disclaimer
- TELEGRAM_CHANNEL_ID mirrors auto-scan signal + breach alerts to a
  subscriber channel (Stars-subscription monetization; Telegram manages
  membership). Feed keeps running even when the owner's DM auto-scan
  is toggled off.
- _owner_only decorator locks all commands/buttons to TELEGRAM_CHAT_ID;
  others get a refusal pointing at the channel.
- Every signal-bearing message (detailed signal, scan summary, all-TF
  summary, breach alerts) now ends with a not-financial-advice /
  educational-use disclaimer via a single _DISCLAIMER constant.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
2026-07-11 18:37:50 -04:00
Harwinder Sekhon ec13cd4c07 EUR: document closure of calendar + H1-hybrid avenues
Surprise-conditioned entries lose 2-5p at every horizon both years;
avoidance filter marginal for EUR and profit-removing for XAU; the
H1-direction + M15-pullback prototype failed its pre-registered kill
criteria (2024-25 OOS negative). EUR stays informational-only.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
2026-07-11 10:26:32 -04:00
Harwinder Sekhon f88e39f5be XAU M15: adopt 1R minimum targets (per-timeframe, validated)
Finishes PENDING_STEPS Step 1 under the current trailing-exit model.
M15 window 2025-03-30->2026-05-31 ($0.30 spread): +116.6R/460 PF 1.83
vs +105.4R/424 PF 1.82 baseline, better in both halves; 2024-06->
2025-03 OOS +30.3R vs +10.6R control. Same knob on the pinned M5 book
is WORSE (+131.3R/699 vs +139.9R/547) so the override is M15-only in
the xauusd factory, per the pre-registered decision rule. M5 path
unchanged (factory override gated on primary_tf == M15).

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
2026-07-11 10:22:33 -04:00
Harwinder Sekhon 02bd61f378 Live news gate: stop capping signal confidence near events (XAU default)
Replaying the live danger/caution gates against the validated 12mo XAU
M5 book showed they suppress 25-35 trades worth +4.6..+7.2R/yr — trades
entered near USD releases outperform the book average (+0.475R vs
+0.256R). New NewsConfig.event_risk_caps_signals (default False =
XAU-tuned) keeps EVENT RISK warnings in alerts but no longer guts
confidence; EUR factory re-enables the caps (evidence is gold-specific).
Pinned XAU M5 regression: 547 trades, +139.94R, trade-identical.
Tests: 177 passed.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
2026-07-11 10:02:27 -04:00
Harwinder Sekhon 2842351f9b Add calendar research infra: FF scraper + verified event store
Scrapes Forex Factory weekly pages (epoch datelines, actual/forecast/
impact/betterWorse) into backtest/cache/calendar.parquet (gitignored;
rebuild with the scraper). Timestamps verified: NFP/CPI/ECB times exact
across DST, and high-impact event bars average 3.5x ATR with the spike
vanishing at +/-1h shifts. calendar_store provides precise_events and
event_window_mask for screens and counterfactuals.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
2026-07-11 09:47:11 -04:00
Harwinder Sekhon 58597de6d7 EUR M15/H1 investigation: document no-validated-edge verdict
Five independent tests at 1-pip cost (M15 wide-open both windows,
transitioning-slice OOS, mean-reversion trade sims, forward-return
screens on M15+M5, H1 both windows) — nothing clears cost in both
2024-25 and 2025-26. Root cause: spread is ~10% of M15 ATR on EUR.
EUR stays out of scan_pairs; open avenues documented in docstring.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
2026-07-11 09:02:24 -04:00
Harwinder Sekhon 27bccf3bf5 EUR M5: commit best-found config after realism re-tune — no validated edge
12mo grid campaign (2025-07-08 → 2026-07-08, 1-pip round-trip cost,
train/val split at 2026-01-08). Donor config scored -6.7R/34 trades.
Findings: transitioning regime -27R/88 wide-open; A-grade entries lose
(-26R) while B-grade wins; Fridays -17R; hours 8/12/13/18/19 UTC bleed;
score 30-35 band holds the train-half losses; trailing (0.45R arm/0.5R
dist) beats static TP1 and gold's 0.6/0.75.

Best-found config: -1.06R / 53 trades / PF 0.93 / 62.3% WR — breakeven,
FAILS acceptance (positive both halves, PF>=1.3). Per plan, EUR ships
functional but stays out of default scan_pairs. Grid results and
methodology documented in the module docstring; full numbers in
backtest/cache/eurv2_*.csv.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
2026-07-10 22:14:49 -04:00
Harwinder Sekhon 2d704753d6 Update CLAUDE.md for multi-pair architecture
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
2026-07-10 21:16:34 -04:00
Harwinder Sekhon 538977115b Live path: config-driven multi-pair scanning (Telegram bot + CLI)
- telegram_bot: SCAN_PAIRS/_WS_TO_PAIR/SCAN_BUTTONS all derive from
  PlatformConfig.scan_pairs (the single pair on/off switch); buttons and
  keyboards are generated per pair; _run_signal_sync builds per-pair
  configs via pair_signals factories (was: XAU defaults for any symbol);
  scan-all works per pair; PB badge matches any *_PB tag; daily-rotating
  file logs in logs/telegram.log (30-day retention)
- main.py: --pair argument (XAUUSD | EURUSD + aliases) routed through
  pair_signals factories; price display is precision-aware (2dp gold,
  5dp FX)

scan_pairs default stays [XAUUSD] until the EUR M5 re-tune validates.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
2026-07-10 21:13:42 -04:00
Harwinder Sekhon e052c7aa3e Add EUR/USD test suite + map EUR news events to EUR/USD
Module 10 (21 tests): EUR config/registry/factory, EUR filter pipeline
(regime/quality/score/confidence/RSI-extended/weekend), EUR trade
geometry (swing SL buffers, pip-denominated spread gate, per-pair gate
contrast), and cross-pair isolation regressions (XAU report bit-equal
with eur_strategy flipped; XAU gates never fire on EUR reports).

news_engine: _get_affected_pairs now maps USD events to EUR/USD too and
adds an EUR entry (calendar warnings reach EUR scans).

175/175 pass.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
2026-07-10 21:06:55 -04:00
Harwinder Sekhon e0789457dc Backtester: EUR/USD support with per-pair configs and default spread
- EURUSD aliases in PAIR_ALIASES
- main() builds PlatformConfig via pair_signals.PAIR_PRODUCERS factories
  (per-pair regime weights/gates/sessions); unregistered pairs keep the
  inline construction. XAU factory is asdict-identical to inline
  (enforced by test) so XAU runs are unchanged.
- --spread default now resolves from PAIR_INFO default_spread
  (XAU $0.30, EUR 0.0001) instead of a hardcoded gold value
- EUR strategy tag: renamed EUR_PB→EUR, detected in both bullish and
  bearish reasons[0] (SELL setups mark the bearish list)

Smoke-validated: 30-day EUR M5 run produces EUR-tagged trades with
1-pip round-trip cost charged.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
2026-07-10 21:02:44 -04:00
Harwinder Sekhon 3b5c0b27ed Add EUR/USD signal gates, trade geometry, and weekend/session filters
Re-grafted from feature/eurusd-support onto the evolved aggregator:
- 6h EUR gate chain (regime, quality, exhaustion, EMA200, session,
  score/confidence, volatility, indicator agreement, RSI-extended),
  active only for EUR/USD with eur_strategy.enabled
- _generate_trade_setup: EUR branch reads EurUsdStrategyConfig only
  (min_sl_pips, min_rr, mature-move + quality-scaled TP, sl_atr_floor,
  swing-buffer SL) with a pip-denominated spread gate; adds a
  pip_value<=0 guard; non-EUR branch is arithmetically unchanged
  (sl_swing_buffer=1.0 extraction only)
- EUR weekend/weekday pre-filter in analyze_pair mirroring the XAU block
- "EUR STRATEGY" reasons[0] marker so the backtester tags EUR trades

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
2026-07-10 20:57:54 -04:00
Harwinder Sekhon 77604960f5 Gate all XAU-only aggregator logic by pair
The SELL quality gate (6e), SELL volatility gate (6g), M5 noise gates
(6h), ATR trade-setup generator, and M5 pullback archetype are all
gold-tuned but previously keyed only on xau_strategy.enabled or the
timeframe, so they would have silently applied gold constants to any
other pair. Each now also requires display_name == "XAU/USD".

No behavior change for XAU (conditions are tautologically true for the
only registered pair today) — verified by test suite and pinned-window
backtest CSV diff.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
2026-07-10 20:53:23 -04:00
Harwinder Sekhon 668b7dad51 Add pair_signals per-pair producer registry
Ported from feature/eurusd-support: PAIR_PRODUCERS registry with
make_config() factories per pair. xauusd.make_config is a pure
PlatformConfig-defaults wrapper (equality with the inline construction
is enforced by test); eurusd.make_config carries the donor branch's
regime weights and gates as a pre-realism baseline.

Adds 4 registry tests. 154/154 pass.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
2026-07-10 20:52:06 -04:00
Harwinder Sekhon 88970e750b Add EUR/USD pair constants and EurUsdStrategyConfig
- EURUSD in PAIR_INFO; new per-pair price_precision and default_spread
  fields on both pairs (backtester consumes default_spread when --spread
  is omitted)
- EurUsdStrategyConfig ported from feature/eurusd-support plus EUR-local
  cost knobs (round_trip_cost=1 pip, min_tp_cost_multiple=10,
  min_sl_pips=8) so SignalConfig's gold-dollar values never apply to EUR;
  donor-tuned values are a pre-realism baseline pending re-tune
- PlatformConfig.eur_strategy field; track EUR in news currencies

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
2026-07-10 20:50:08 -04:00
Harwinder Sekhon 7470e093e1 Fix 3 engine bugs: swing-point NaN false positives, ROC div-by-zero, ADX epsilon
Ported from feature/eurusd-support (e6e09c5), pair-neutral fixes:
- _find_swing_points: all(<empty generator>) is vacuously True, so a value
  surrounded by NaNs was reported as a swing point; now requires at least
  one valid neighbor on each side
- ROC: guard against zero close at the lookback positions (inf scores)
- ADX: epsilon denominator instead of NaN-replace to avoid NaN cascade

Adds 8 regression tests (Module 9). 150/150 pass.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
2026-07-10 20:48:56 -04:00
Harwinder Sekhon ec33f8b718 Overhaul news engine: fix false signals, add surprise detection + caching
Improvements from a live-verified audit of the news/calendar pipeline:

- Currency tagging: keyword matching now uses letter-boundary regexes
  and drops ambiguous "us "/"eu "/"uk " tokens — "versus" no longer
  tags headlines as USD (confirmed live false positive polluting the
  XAU/USD sentiment average).

- Fetch caching: calendar events and headlines are cached class-level
  (10 min TTL, 60s retry after empty/failed fetch, lock-guarded for
  concurrent scans). Stops per-scan refetching that was tripping the
  Forex Factory rate limit; TTL stays under the 30-min pre-event
  buffer so the danger window can't go stale.

- Surprise detection: EconomicEvent.surprise is computed from parsed
  actual vs forecast (handles %, K/M/B/T suffixes, negatives). A
  surprise print widens the post-release caution window to 45 min;
  inline releases keep the original 15 min.

- Gold-aware sentiment: headlines are scored per currency, with
  monetary-policy keywords (hawkish/rate hike/tightening vs dovish/
  rate cut/easing/stimulus) inverted for XAU — "hawkish Fed" headlines
  scored +1.0 for gold before, -1.0 now (verified on live feed).

- New sentiment sources: Google News gold-focused query and CNBC
  Economy RSS (verified live; 40 → 74 unique headlines). Added
  cross-feed title dedup so syndicated stories count once.

- events_today now buckets by UTC date (feed timestamps carry US-
  Eastern offsets); --json output includes news_description.

- Removed dead NewsConfig.min_impact and unused HIGH_IMPACT_EVENTS.

Tests: 142 passed (7 new covering boundary matching, surprise
classification, post-event windows, cache reuse, gold inversion).

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
2026-07-10 09:43:11 -04:00
harwinder dad2432d4d Add M5 pullback entry archetype (XAU_PB): +140R combined on 12mo
Second entry pattern alongside the Keltner-breakout: enter on the
retrace to EMA21/KC-mid in an established trend (EMA >=3/4 aligned,
ADX>=20, retrace 0.8-6.0 ATR, resumption bar closed back in trend
direction, RSI in the pullback zone not the extended zone). Detector
lives in the aggregator so live and GPU-backtest paths share one
implementation. Own gates (composite floor 15, confirmation-TF
agreement, MTF opposition, vol regimes) with a diagnostic funnel.

PB stops anchor to the pullback swing (~$8 vs ~$25 breakout), so PB
trades get their own trail arm (0.8R vs 0.6R global) via per-trade
trail overrides in the backtester and strategy-aware Telegram alerts.

12mo M5 validation: PB alone +38.3R/254 trades, positive both halves
and all four full quarters; core breakout trades undisturbed
(+101.9R vs +105.0 baseline). Combined book: 549 trades (~10.6/wk),
+140.1R, PF 1.78, DD 11.3R vs 311 trades/+105.0R before.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
2026-07-09 14:21:59 -04:00
harwinder 3ac36525b0 Add pullback entry pattern implementation plan
Design doc for a second M5 entry archetype: enter on retrace to
EMA21/KC-mid in an established trend instead of at Keltner-breakout
extension. Includes detector spec, geometry, validation protocol
(both-halves + quarterly + displacement checks), and the +105R
baseline to beat. Session/frequency knobs (blocked hours, cooldown)
were tested 2026-07-09 and confirmed already optimal — this is the
remaining evidence-backed entry-side direction.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
2026-07-09 13:51:00 -04:00
harwinder 987ec5bbb0 Add M5 gates 5-6 and trailing-stop exits (+48R → +105R on 12mo)
Gates 5-6 in the M5 filter: block 'high' vol regime entries (chop
transition — 61% WR, negative total while low/normal/extreme all
positive) and block SELLs with RSI<25 (dip already extended, mean-
reverts before target). 365d: 419→311 trades, +41→+48.4R, DD 11.2→8.0R.

Replace the fixed RR-0.5 take-profit with a trailing stop: arm at
+0.6R favorable excursion, trail 0.75R behind best price, stop levels
update at bar close (conservative). The RR-0.5 target was the binding
constraint — winners kept running after TP. Whole trail parameter grid
(arm 0.4-1.2 × dist 0.5-1.25) lands +90-103R, a plateau not a spike;
also +91.7R on the pre-gate 419-trade population. TP1 stays advisory;
Telegram alerts include the arm price and trail distance. max_hold
200→300 bars (trailing exits need room; +10R).

12mo M5, same 311 trades: +105.0R total, PF 2.27, DD 7.9R, 61% WR,
train/val +54.6/+50.4, every quarter positive.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
2026-07-09 13:34:57 -04:00
harwinder db16f209bb Fix M5 signal quality: ATR ordering bug, noise gates, blocked-hour leak (+41R)
ATR was computed after EMAs so ribbon compression detection was dead code
on all timeframes — moved ATR first. Extended M5 swing S/R lookback from
50 to 120 bars. Added four M5-specific filters: min score, MTF agreement,
confirmation TF conviction, and SELL quality cap (A-grade M5 shorts enter
at dip extremes and lose). Fixed backtester blocked-hour check to use bar
close time matching the live bot. Added trade geometry config (spread gate,
SL floor, TP floor RR) and primary_tf mismatch guard in backtester.

Backtest: 1133 trades/+11.5R → 419 trades/+41.0R, PF 1.03→1.37, DD 21→11R.
Train/val split positive in both halves (+19.8R/+21.3R).

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
2026-07-09 12:48:36 -04:00
harwinder d95d31a852 Disable TP tightening: tuned pre-spread, halves total R under real fills
A/B on M15 2025-03-30→2026-05-31 with $0.30 spread + gap-through fills
(511 identical entries, only exits differ):
  factor 0.8 / min_rr 0.4: +36.1R (avg +0.071R, SELL +1.2R)
  factor 1.0 / min_rr 0.5: +66.8R (avg +0.131R, SELL +9.6R)
Worse in both train and validation halves and both directions. The 20%
tighter targets buy +1.8pt win rate but shrink every win while the fixed
spread stays constant. The knob remains for re-tuning with costs included.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
2026-07-08 21:35:59 -04:00
harwinder 5a0af663c1 Add WebSocket dual-connection test script
Probes whether Twelve Data allows two simultaneous WS connections
for one symbol (SDK-based, 30s run, prints per-connection verdict).

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
2026-07-08 21:32:46 -04:00
harwinder b191cee27e Telegram bot: WS heartbeat, breach alerts, alert dedup, interactive menu
WebSocket:
- Send heartbeat every 10s — Twelve Data drops idle connections, so the
  monitor previously died on weekends/quiet hours and churned reconnects
- Log failed symbol subscriptions from subscribe-status events
- Stamp fresh live tick price + age into every alert message

Alerts:
- Re-enable level-breach alerts: WS breach triggers M15 rescan and sends
  the report when actionable (was scanning and discarding the result)
- Dedup: identical (signal, quality, direction) per pair/TF is sent once,
  re-armed after 1h — no more repeat alerts every candle
- Configurable minimum alert quality (C+/B+/A), cycled from the menu
- Trade setup shows dollar risk/reward alongside pips

Interactive menu (/start, /scan):
- Timeframe scans, Scan All, auto-scan ON/OFF toggle, alert-quality
  cycle, status, and help as inline buttons
- Markdown-safe edits: 'message not modified' swallowed, parse failures
  fall back to plain text; exception text escaped

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
2026-07-08 21:32:46 -04:00
harwinder 6a96ed6006 Fix live/backtest parity, SELL-side losses, and indicator repairs (+42% total R)
Live pipeline:
- Drop still-forming candle before analysis (signals no longer repaint)
- Direction-aware MTF agreement + opposition gate (no trade against 2-of-3 TF consensus)
- SELL volatility gate: block shorts in low/normal vol regimes (config sell_vol_regimes)
- Divergence: 40-bar lookback, price/indicator swings must align within ±2 bars
  (fired on 0.5% of trades before, 29% after), trending weight 2.0→1.0
- RSI regime_aware by default; transitioning regime only signals at extremes
- ADX ±DM tie no longer nudges bearish; TP2 clamped beyond TP1

Backtester realism:
- Charge $0.30 round-trip spread (--spread), gap-through SL/TP fills at bar open
- Resolve trades on RAW price path (cache stores raw bars, cleaning runs on load)
- Weekend/session filters use bar-close decision time to match live wall clock

Validated per-step on M15 2025-03-30→2026-05-31 under identical pricing:
+46.9R → +66.8R, win rate 69.9% → 73.0%, SELL side -2.3R → +9.6R.
Tested-and-reverted: ROC weight removal (-13R), BB fade suppression (-11.6R),
candlestick de-weight (-2R) — composite thresholds are co-tuned with weight mass.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
2026-07-08 21:32:46 -04:00
harwinder 998f5e8616 Fix trade setup generation broken by TP tightening factor
The TP floor calculations used raw min_rr (0.4) but after 20% tightening
the effective RR dropped to ~0.32, failing the min_rr check and returning
None for all trade setups. Pre-compensate the floor with min_rr/tf so
post-tightening RR still meets the minimum.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
2026-05-24 18:22:57 -04:00
harwinder a731b7c8df Tighten TP targets by 20% for faster hits (configurable tp_tightening_factor=0.8)
Applied uniformly across all 3 trade setup functions. Lowered min_rr_ratio
0.5→0.4 proportionally to preserve trade acceptance rate. SL unchanged.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
2026-05-22 10:44:52 -04:00
Harwinder 643952a8e2 Enable M5 auto-scan alerts and remove live dashboard
M5 already broadcast through the shared _scan_timeframe path but was
documented as data-only (docstring, CLAUDE.md, DOCUMENTATION.md, plus a
phantom `if tf == M5: return` step that never existed in code). Align all
docs/comments so M5 alerting at C+ grade is the intended, documented behavior.

Remove the live dashboard entirely per request: the 30s refresh loop,
per-scan refresh, boot send, and the dashboard-only stats layer
(_signal_stats / _stats_updated / _update_signal_stats and its call sites,
_format_stats_dashboard and its helper constants). Drop now-unused ZoneInfo
import and dashboard mentions from help/autoscan/boot messages and the test
suite. The WebSocket level-breach monitor and level cache are untouched.

Tests: 136 passed, 0 failed.

Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
2026-05-20 12:45:27 -04:00
harwinder b61818330b Fix weekend filter: use precise forex market hours (Fri 22:00–Sun 22:00 UTC)
The old `weekday() >= 5` check blocked all of Sunday, but the forex
market reopens Sunday 22:00 UTC (Sydney open). Now uses proper time-based
check so the bot fetches data and generates signals on Sunday evening.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
2026-05-17 19:23:21 -04:00
harwinder 0e6f5d74a7 Add backtest scripts to version control
Track gpu_backtest.py, diagnose_filters.py, and instructions.
Data files (CSVs, parquet cache) remain gitignored.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
2026-05-15 19:49:30 -04:00
harwinder c505829dfb Reduce A-grade TP when exhausted: RR 0.75→0.50 on 2+ exhaustion tags
A-grade trades entered exhausted moves with TP too far (0.75R), causing
57.7% WR and +0.25R over 30 days. Store exhaustion_count on SignalReport
and scale TP to B-level (1.0×) when A-grade has 2+ exhaustion tags.
Backtest: A-grade 73.1% WR / +3.25R, B/C unchanged.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
2026-05-15 19:46:31 -04:00
harwinder 08742c5063 Align FilterConfig with XAU strategy: blocked hours [0,1,2], session 3-23 UTC, Mon-Fri
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
2026-05-15 15:36:03 -04:00
harwinder 3f7d24734d Add M5 to Telegram scan buttons and Scan All TFs
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
2026-05-15 15:26:00 -04:00
harwinder 5047601d46 Reduce blocked hours, enable M5 alerts, show NY time on dashboard
- Blocked hours reduced to 0-2 UTC only (daily rollover)
- M5 signals now broadcast to Telegram like M15/H1/H4
- Live dashboard shows New York time (auto EDT/EST via zoneinfo)

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
2026-05-15 15:23:48 -04:00
harwinder 8ed6454ea5 Refocus on XAU/USD: add momentum indicators, tune TP, exhaustion block
Strip EUR/USD, GBP/USD, USD/JPY pairs and strategies from entire codebase.
Add 3 new scored indicators (ROC, Keltner/TTM Squeeze, ADX slope) and
enable MACD-as-pillar + divergence detection for early trend entry.
Introduce "transitioning" regime (ADX < 25, slope > 3.0) to catch trend
births before ADX confirms. Gold-specific composite weights per regime.

Tune TP multipliers: quality-tiered scaling (A+/A get 1.5-2.0x ATR via
XAU ATR method, B/C use swing-level targeting). Backtest: +24R total,
65% WR, PF 1.12 over 365 days.

Add momentum-confirmed exhaustion block: hard-reject trades when 2+
static exhaustion signals fire AND ROC/MACD confirm fading momentum.
Prevents late trend entries without killing strong trend trades.

Telegram: simplify buttons to timeframe-only labels, add "Scan All TFs"
multi-timeframe summary, add /scan command, fix BB/KC level breach keys,
fix price formatting (.2f for gold), fix deprecated asyncio call, fix
"1 pairs" grammar and 60s/30s dashboard refresh mismatch.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
2026-05-15 10:51:51 -04:00
harwinder 98daf71a52 Add web dashboard (Redis + FastAPI + WebSocket), scope XAU penalties
Signal daemon (signal_web.py): candle-boundary scanner mirroring
telegram_bot.py, publishes serialized SignalReport JSON to Redis.
Web API (web/app.py): thin FastAPI reader behind Apache, serves
/api/signals REST + /ws WebSocket from Redis pub/sub.
Frontend (web/static/index.html): 3-pair grid, color-coded signals,
indicator breakdown per timeframe, trade setup display.

Also: XAU-specific penalty scoping in signal_aggregator (softer
confidence penalties for gold, original strict values for forex),
pair-aware session filter routing XAU to its own blocked hours.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
2026-05-14 23:15:37 -04:00
harwinder 57769967d1 Fix news engine (keywords, calendar fallback, RSS URL), remove Telegram guards
News engine: expand sentiment keywords (18%→60% headline coverage), add
MyFXBook RSS as fallback calendar source when Forex Factory returns 429,
add retry/backoff on calendar API rate limits, update stale ForexLive URL
to investinglive.com, add date formats without seconds for MyFXBook RSS.

Telegram bot: remove flood control, rate limiter, and dashboard failure
tracking — all sends go directly through bot.send_message/edit_message_text.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
2026-05-14 13:42:32 -04:00
harwinder bf9609708c Fix duplicate-index crash in data validation, add Telegram flood control
- Move index dedup before OHLC consistency fix in _validate_and_clean
  to prevent crash when df.loc[dup_idx] returns Series instead of scalar
- Add global flood backoff: detect RetryAfter, pause all sends/edits
  until ban expires instead of retrying and escalating the ban
- Stop dashboard edit-fail → new-message spam cascade: track consecutive
  failures, only resend after 3 failed edits
- Increase dashboard refresh from 5s to 30s, min send interval to 1.5s
- Add comprehensive QA test suite (140 tests across all 7 modules)

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
2026-05-14 13:04:28 -04:00
harwinder 235241f95a Harden Telegram bot, add EUR/GBP strategies, parallelize scans
Telegram reliability: rate-limited sends (1 msg/s), dashboard lock,
WebSocket JSON/price error handling, markdown escaping, level cache
TTL, cooldown cleanup, breach task tracking, graceful shutdown.

Telegram speed: parallelize 3-pair scans with asyncio.gather(),
increase thread pool 2→4, parallelize boot-up initial scan.

Strategy: add EUR/USD and GBP/USD pullback-buy strategies with
per-pair configs, optimize XAU TP to 0.9×ATR, block hours 8/9
for XAU and hour 17 for GBP.

Backtest: add --max-hours flag to evaluate.py for time-exit support.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
2026-05-13 21:07:49 -04:00
Harwinder 1f3f69c21d Fix dashboard TF key mismatch, increase refresh to 5s
Timeframe keys stored as enum values (15min/1h/4h) but dashboard
looked for M15/1h/4h — GBP/USD showed no price or signal text.
Now uses Timeframe enum directly with display labels. Reduced
dashboard refresh from 60s to 5s with throttle guard.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
2026-05-13 18:51:02 -04:00
Harwinder 439e179128 Add live dashboard with 60s auto-refresh, disable breakout alerts
Replace per-scan Telegram messages with a single dashboard message
that edits in-place every 60 seconds. Each pair shows all TFs
horizontally with short signal labels (Buy/Sell/Neutral). WebSocket
live prices feed into the dashboard. Level-breach alerts disabled
(WS still updates cache silently). Removed /stats command.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
2026-05-13 18:34:44 -04:00
harwinder 6b1ffff66a Fix live session filter overriding XAU strategy signals
The general session filter ran after analyze_from_dataframes() and
wiped out XAU oversold bounce signals at most hours. Now skips the
session filter when the XAU strategy has already fired — the strategy
has its own backtested hour filter.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
2026-05-13 17:31:46 -04:00
harwinder d63544b75d Add XAU/USD oversold-bounce strategy (79.7% WR, PF 2.06)
BUY-only strategy for gold: enters when M15 RSI drops below oversold
threshold while price remains above EMA200 on all three timeframes
(M15, H1, H4). Uses ATR-based TP/SL (0.7x/2.0x) sized from M15 ATR.
Isolated to XAU/USD only — other pairs completely unaffected.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
2026-05-13 17:25:49 -04:00
harwinder 1e5047e20f Backtest tuning, data freshness guard, Telegram improvements
- Drop USD/JPY from scan pairs (zero signals across all timeframes)
- Disable divergence indicator (zero impact in 14-config lab test)
- Parallelize evaluate.py with ProcessPoolExecutor (16 cores)
- Add --pair CLI filter to evaluate.py for single-pair backtests
- Add data freshness guard: stale cache auto-invalidated, primary TF
  stale data suppresses signals to prevent trading on old data
- Add /cache command: shows fetch age and last bar per pair/TF
- Add /clearcache command: wipes in-memory cache for fresh API fetch
- Remove BB from WebSocket breach alerts (keep structural levels only)
- Show live prices only for WS-monitored pairs (EUR/USD, XAU/USD)
- Replace arrow emojis with text tags in indicator lines

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
2026-05-12 22:56:55 -04:00
harwinder 6fe1a3e0f0 Add project documentation and MT5 integration plan
DOCUMENTATION.md: Two-section guide covering the full project —
plain-English trader guide (indicators, scoring, trade setups) and
developer reference (architecture, data flow, every threshold/weight).

mt5.md: Architecture plan for automated order execution via MT5 bridge
(REST API on Windows, async client on Linux, risk-based lot sizing).

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
2026-05-11 19:50:21 -04:00
harwinder 4db6121ca7 Fix trade setup generation, M5 data-only mode, WS symbol limits, command menu
- Trade setup TP now scales to SL distance ensuring min RR is met when
  no viable swing target exists; skip swing targets with poor RR
- Add float tolerance to RR gate to prevent edge-case rejection
- M5 auto-scan runs for level cache only, no Telegram broadcast
- WebSocket limited to EUR/USD + XAU/USD (Grow plan constraint)
- Register bot command menu via set_my_commands on startup

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
2026-05-11 18:58:40 -04:00
harwinder 5e12307b1f Suppress Telegram messages when no actionable signals found
Auto-scan still runs and updates the level cache every cycle,
but only sends to Telegram when quality A+/A/B/C signals exist.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
2026-05-11 18:03:49 -04:00
harwinder 4be7077c0a QA fixes: H4 timing bug, /status + /help commands, WS cold-start
- Fix H4 boundary calculation using total minutes (hour*60+min) not just minute
- Add /help command listing all bot commands
- Add /status command showing WS connection, live prices, level cache, cooldowns
- Run initial M15 scan on boot to populate level cache before WS starts
- Update /start message to mention new commands

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
2026-05-11 17:16:07 -04:00
harwinder 736e33302b Tune engine from backtesting: wider SL, capped TP, counter-trend filter
Backtest-driven fixes (0% win → 33% win, PF 0.00 → 1.05 over 7 days):
- SL: swing buffer 0.5→1.0 ATR, min distance 1.0→1.5 ATR (more room to breathe)
- TP: cap swing targets at ATR×3.0, fallback ATR×2.0 (was targeting unreachable levels)
- Mature-move TP: when score>30 (A/B grade), tighten TP to ATR×1.5 (move already extended)
- Counter-trend penalty: -15 confidence when signal fights regime direction
- EMA200 penalties halved: -20→-10 (was killing EUR/USD signals)
- Score thresholds lowered: BUY/SELL ±15→±12, STRONG ±40→±35
- Quality grades relaxed, C-grade now gets trade setups and Telegram alerts
- min_rr_ratio: 1.5→1.2
- Gitignore backtest/ directory

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
2026-05-11 17:00:12 -04:00
harwinder a7bed99caa Buy signal included 2026-05-11 16:37:45 -04:00
harwinder dc9c76a716 Pro tier upgrade: quant rearchitect + WebSocket level-breach monitor
- Restore XAU/USD (Pro plan), drop USD/CAD, raise rate limit to 55/min
- Reduce signal engine to 4 uncorrelated pillars (EMA, RSI, BB, Candlestick)
  with regime-adaptive weights (trending boosts EMA, ranging boosts RSI+BB)
- Add structure-based S/R (swing highs/lows) for trade setup targeting
- Fix confidence death spiral: all adjustments now additive, not multiplicative
- Dynamic R:R trade setup: SL behind swing level, TP at next structure level,
  reject if R:R < 1.5
- Fix news engine for all 4 pairs (was hardcoded to USDJPY)
- Add WebSocket monitor: streams live ticks, triggers M15 scan on level breach
  with 5-min cooldown per pair — catches breakouts between candle boundaries
- Add correlation filter (EUR/USD+GBP/USD, XAU/USD+USD/JPY exposure warnings)

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
2026-05-11 16:33:23 -04:00
harwinder a15dc40050 Replace XAU/USD with USD/CAD — TwelveData has no gold data
XAU/USD is not available in TwelveData's symbol list.
Swapped to USD/CAD across config, scan pairs, Telegram buttons,
and added CAD to tracked news currencies.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
2026-05-11 15:50:48 -04:00
harwinder b3e0da7455 Add 4-pair candle-boundary auto-scanner (free tier safe)
Expand from 2 to 4 pairs (EUR/USD, USD/JPY, GBP/USD, XAU/USD) with
automatic scanning at M15 and H1 candle close boundaries. Scheduler
staggers H1 by 60s to stay within 8 credits/min; daily budget is
480 of 800 allowed. Cache TTL is now candle-duration-aware so cross-TF
confirmation data is reused without extra API calls. Spread estimate
uses primary-TF data instead of fetching M5 separately.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
2026-05-11 15:30:11 -04:00
Harwinder 037dec643e Add EUR/USD support and 4-button Telegram scanner
Add EURUSD pair with dynamic pair info lookup. Telegram bot now shows
4 scan buttons: EUR/USD 15M, EUR/USD 1H, USD/JPY 15M, USD/JPY 1H.
Each indicator displays directional arrows in the message output.
Price formatting adapts per pair (5dp for EUR, 3dp for JPY).

Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
2026-05-10 23:49:36 -04:00
Harwinder 3a09dfd62f Strip backtesting, rebalance indicator weights toward leading signals
Remove backtest.py, backtest_gpu.py, metrics.py, local CSV data,
Dukascopy loader, and all backtest CLI modes from main.py. Rebalance
engine weights: leading indicators (divergence, RSI, stochastic, BB)
now carry 66% of signal weight vs 34% for lagging (EMA, MACD). Add
ws_twelvedata.py for live tick streaming. Drop pyarrow and tqdm deps.

Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
2026-05-10 23:40:15 -04:00
Harwinder 353ff05267 Add USAGE_LINUX.md — bot and CLI run instructions for Linux server
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
2026-05-07 23:12:55 -04:00
Harwinder a596e7cab2 Add async Telegram bot — live scan button with trade setup
Introduces telegram_bot.py: python-telegram-bot v20+ async bot that
presents a persistent Scan Signal inline button. Each press fetches a
live USD/JPY signal via SignalAggregator and formats it with emoji,
MTF breakdown, top bull/bear reasons, and warnings. STRONG_BUY /
STRONG_SELL signals additionally render the full trade setup (entry,
SL, TP1, TP2, RR). Adds python-telegram-bot>=20.0 to requirements.txt.

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
2026-05-07 23:05:01 -04:00
Harwinder c72e8e669b Round-2 weight optimisation — reduce EMA trend, boost stochastic
GPU sweep (36 combos, RTX 4070 SUPER) on H4 2022-2026 relative to the
round-1 baseline revealed EMA trend was over-weighted at 3.6. Halving
it and nudging stochastic up yields cleaner, higher-quality signals.

Weight changes (backtest_gpu.py _BASE_W + engine.py):
  ema_trend:  3.6 -> 1.8  (over-weighted; halving tightens signal filter)
  stochastic: 1.6 -> 2.4  (x1.5; improved WR to 40.4% in sweep)
  macd:       3.3          (unchanged — dominant signal, kept as anchor)
  adx/ichimoku/rsi/bollinger: unchanged

Combined result vs round-1 (H4 local data 2022-2026):
  Trades:      160 -> 73   (-87 — filters out marginal setups)
  Win Rate:   38.1% -> 43.8%  (+5.7pp)
  Profit Factor: 1.19 -> 1.52
  Return:    +21.30% -> +23.86%  (+2.57pp)
  Max DD:     -8.52% -> -9.32%
  Sharpe:      0.47  ->  0.71   (+51%)
  Sortino:     1.91  ->  3.06   (+60%)

Fewer trades at much higher quality. The lower EMA weight means only
strong, multi-indicator confluences clear the grade threshold — the
73 remaining trades have 43.8% WR vs 38.1% and expectancy nearly 3x.

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
2026-05-07 22:24:48 -04:00
Harwinder 3d9f4dc201 Enable GPU scoring via CuPy — RTX 4070 SUPER confirmed 6.3x speedup
Install nvidia pip packages for CUDA 12 runtime DLLs (no full toolkit needed):
  nvidia-cuda-nvrtc-cu12, nvidia-cuda-runtime-cu12,
  nvidia-curand-cu12, nvidia-cublas-cu12

Add _register_nvidia_dlls() in backtest_gpu.py that auto-discovers and
registers nvidia package DLL directories (site-packages/nvidia/*/bin/)
via os.add_dll_directory() before CuPy is imported. Suppresses the
spurious "CUDA path could not be detected" warning from cupy._environment.

Backend line now reads: CuPy 14.0.1 (RTX GPU)
Benchmark: scoring pass on 100k bars — NumPy 1.1ms vs CuPy 0.18ms = 6.3x.
Results are identical to CPU path (deterministic).

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
2026-05-07 22:22:21 -04:00
Harwinder e30535f5e9 GPU backtester, optimised weights, local CSV data pipeline
## What changed

### backtest_gpu.py (new — ~1,300 lines)
Standalone GPU-accelerated backtester completely separate from backtest.py.
Computes all indicator series in one vectorised O(N) pass over the full price
history, then replays positions with a sequential state machine.

Key design:
- VectorIndicators: pandas ewm/rolling (C-optimised) for all 7 indicators
- CuPy GPU backend with automatic NumPy fallback (no hard dependency)
- Two-phase API: prepare_arrays() once, score_and_run() many times
- --optimize flag: 36-combo weight sweep (1 baseline + 7 ablations + 28
  multiplier runs) — calls prepare_arrays() once, replays all combos in seconds
- Full CLI: --tf, --local, --start/--end, --regime-filter, --regime-tf,
  --session-filter, --optimize, --journal, --journal-csv

O(N) vs O(N^2): backtest.py recomputes all indicators per bar (O(N^2));
backtest_gpu.py computes each indicator series once and reads bar T in O(1).
On 100,000 H1 bars: ~4 seconds vs >10 minutes.

### data_provider.py
Added load_local_csv() method — reads MetaTrader tab-delimited CSV exports
from market_data/ directory. Format: Time|Open|High|Low|Close|Volume|Spread
(spread column dropped on load, timestamps localised to UTC). Applies the same
_validate_and_clean() pipeline as the Twelve Data path. LOCAL_CSV_MAP maps
each Timeframe enum to its filename.

market_data/ files added (MetaTrader exports, UTC):
  USDJPY_M15.csv  100,000 bars  2022-05-04 to 2026-05-08
  USDJPY_H1.csv   100,000 bars  2010-04-23 to 2026-05-08
  USDJPY_H4.csv    25,855 bars  2010-04-22 to 2026-05-08
  USDJPY_D1.csv     5,020 bars  2010-04-22 to 2026-05-08

16 years of H1/H4/D1 vs ~6 weeks of H1 on Twelve Data free plan. Use --local
for all backtests; Twelve Data reserved for live signal feed.

### engine.py — optimised indicator weights
36-combo sweep on H4 2022-2026 found MACD and EMA trend are the dominant
signals; RSI and Bollinger are net drag on this trending pair.

Old → new weights:
  ema_trend:  1.2 → 3.6  (×3 — strongest directional signal)
  macd:       1.1 → 3.3  (×3 — ablating it is the single worst move: -42%)
  stochastic: 0.8 → 1.6  (×2 — clean timing, 40% WR at this level)
  adx:        1.0 → 0.5  (÷2 — partially redundant with EMA stack)
  ichimoku:   1.1 → 0.55 (÷2 — lags too much at H4, hurts in ranging)
  rsi:        0.3 → 0.0  (ablated — net drag across all test windows)
  bollinger:  0.9 → 0.0  (ablated — counter-trend noise on trending pair)

RSI and Bollinger are still computed and displayed in live output for
reference; weight=0 means they contribute nothing to the composite score.

Combined result vs baseline (H4 2022-2026, local data):
  Return:   +7.12% → +21.30%  (+14.2pp)
  Max DD:   -9.05% → -8.52%   (tighter — gains did not come at expense of risk)
  Sharpe:    0.23  →  0.47    (2x improvement)
  Sortino:   1.38  →  1.91
  Win Rate: 36.3%  → 38.1%

### backtest.py
Added 16-thread CPU parallelism for signal pre-computation via
ThreadPoolExecutor. _precompute_all_signals() dispatches each bar to a worker;
pandas/NumPy C extensions release the GIL enabling true parallel execution.
Results are collected and replayed in the main loop, replacing the per-bar
recompute. Also added --regime-tf CLI flag so regime filter timeframe can be
specified independently of the primary TF.

### signal_aggregator.py
EMA200 distance gate: confidence *= 0.55 when price is within 0.5% of EMA200
(chop zone) or beyond 1.5% (overextended). Sweet spot is 0.5-1.5% from EMA200
where trending momentum is cleanest. Strongest single filter found: lifted
cumulative return +13.88% to +35.50% in ablation window.

### ENGINE_LOGIC.md (new)
Full reference document covering: architecture, data pipeline, all indicator
logic, composite scoring formula, optimised weights with rationale, regime
classification, EMA200 gate, MTF alignment, grade system, backtesting results,
filter catalogue, and configuration reference.

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
2026-05-07 22:12:43 -04:00
Harwinder a687db9413 Add EMA200 distance gate — strongest single signal-quality filter
Across 2024 trend / 2023 chop / 2026 hostile windows, distance from the
H1 EMA200 was empirically the most discriminative variable. Sweet spot
0.5-1.5% from EMA200; closer = false-breakout chop, farther = chase.

Outside the band, confidence is multiplied by 0.55 so the trade falls
below the B-grade threshold and is rejected.

Cumulative return across the 3 windows: +13.88% baseline -> +35.50%.
Win rate improved in every regime; 2023 chop turned positive (-2.56% to
+14.94%).

Also extends the trade journal with per-timeframe MTF scores, RSI at
entry, and EMA200 distance for future filter mining; surfaces ema_trend
in key_levels so the gate can read it.
2026-05-07 19:28:19 -04:00
Harwinder b7d03a9b61 Document session filter and auto-toggle flags in USAGE
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
2026-05-07 18:44:41 -04:00
Harwinder b38ed17c96 Add session filter + auto-toggle, extend trade journal diagnostics
Session filter restricts trades to a UTC hour window (default 07-16,
London + NY) — opt-in via --session-filter, with --session-start /
--session-end overrides. Wired into both the backtester (cheap hour
check next to the regime gate) and live mode (downgrades signal to
NEUTRAL/F when current UTC hour is outside the window).

Auto-toggle (--auto-session-filter) tracks rolling mean pnl_pct over
the last N closed trades; when it falls below threshold, the session
filter activates, releasing once expectancy recovers. Default lookback
10 trades, threshold -0.3%.

Trade dataclass extended with diagnostic fields captured at entry:
entry_hour, entry_dow, ADX, ATR%, market_regime, volatility_regime,
mtf_agreement. These let --journal-csv exports surface loss-pattern
clusters when a window underperforms.

Validated on 2024 trend / 2023 chop / 2026 recent windows. Auto-toggle
preserves ~50% of trend edge while still cutting ~25% of recent-regime
losses — pure regime filter is still best on cumulative, so both
session flags ship default-off.

Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
2026-05-07 18:38:49 -04:00
Harwinder dcd98fffa2 Swap raw requests for Twelve Data Python SDK
data_provider now uses TDClient.time_series().as_pandas() instead of
hand-rolled REST calls. Throttle, validation, and parquet caching are
unchanged. Adds twelvedata>=1.2.0 to requirements.txt.

Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
2026-05-07 18:38:38 -04:00
Harwinder 70990532a3 Compact hierarchical live signal output
Restructure print_signal_report so decision (signal · grade · score ·
confidence · MTF) sits at the top, followed by trade setup, context line,
MTF breakdown, top supporting/opposing reasons in two columns, warnings,
and an optional detail block under --detail. Adds _compact_reason helper
that strips known indicator name prefixes via _INDICATOR_PREFIXES so
reason lines stay short.

Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
2026-05-07 17:53:50 -04:00
Harwinder 53e6e07829 Wire regime filter into live mode
--regime-filter previously affected only backtests. Live mode now also
checks higher-TF ADX after analysis: signals are downgraded to NEUTRAL
with a warning when ADX < threshold, and confirming when above.

This brings live behavior into parity with what the backtester measures.

Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
2026-05-07 17:40:39 -04:00
Harwinder 7699e6dd1d Add indicator ablation framework + apply ablation-driven cleanup
Framework:
- TechnicalConfig.disabled_indicators skips named indicators in engine.analyze
- Optimizer supports disable_indicator override for parallel ablation sweeps
- main.py --ablate runs leave-one-out across all 10 directional indicators

Cleanup applied (validated across 2024 trend / 2023 chop / 2026 recent):
- Drop pivot by default (cumulative +9.5% to disable)
- Drop candlestick by default (+3.4% cumulative)
- Regime-gate ichimoku: 1.3x in trending, 0.2x in ranging (was a chop villain)
- Trim RSI weight 1.0 -> 0.3 (cumulative noise across regimes)

Result: +10.60% on 2024 trend (+33.24% -> +43.84%, Sharpe 2.17 -> 2.77),
+15.71% on 2023 chop (-18.27% -> -2.56%). Recent regime still adverse.

Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
2026-05-07 17:33:07 -04:00
Harwinder 636f0304ae Refocus on USD/JPY and add walk-forward backtester with regime filter
- Strip multi-pair plumbing; drop synthetic-data fallback
- Add BacktestConfig + walk-forward replay through the live signal pipeline
- Add metrics module (PF, Sharpe, Sortino, max DD, trade journal)
- Add parallel grid optimizer with disk parquet cache
- Add H4 ADX regime gate (--regime-filter, --regime-min-adx) — cuts chop
  losses ~84% in 2023 H1 while preserving trend edge

Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
2026-05-07 17:15:41 -04:00
Harwinder 6ebdc34cd3 Remove redundant script.txt and tracked __pycache__
script.txt was superseded by requirements.txt. Python bytecode should
not be version-controlled — added .gitignore to keep cache and venv
out of the repo going forward.

Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
2026-05-06 22:25:43 -04:00
Harwinder bc1844b233 Add requirements.txt and usage guide
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
2026-05-04 10:51:03 -04:00
13 changed files with 31 additions and 1414 deletions
+3 -5
View File
@@ -11,9 +11,8 @@ Every numeric threshold in `config.py` and `pair_signals/xauusd.py` was backtest
## Commands
```bash
python test_all.py # QA suite: 374 tests, 23 modules, no network (mocked)
python main.py --pair XAUUSD --timeframe H1 --detail # CLI signal report (--json for full dump,
# --panel to run the advisory LLM agent panel on-demand)
python test_all.py # QA suite: 359 tests, 22 modules, no network (mocked)
python main.py --pair XAUUSD --timeframe H1 --detail # CLI signal report (--json for full dump)
python telegram_bot.py # the bot (refuses to boot without credentials)
python oanda_daemon.py # auto-execution daemon on Oanda practice account
python dashboard_publisher.py --once # web dashboard: one scan+publish cycle to $DASHBOARD_OUT_DIR
@@ -35,7 +34,7 @@ Python 3.12; a `venv/` exists in-repo. Dependencies in `requirements.txt` (numpy
## Configuration & credentials
Precedence for every credential/deployment setting: **env var → config.json → refuse to run** (no hardcoded fallbacks). A gitignored repo-root `.env` is loaded at `app_config` import via `os.environ.setdefault`. `config.json` (gitignored; schema in `config.json.example`) holds secrets plus deployment settings: `data_providers.active` (the live-feed switch), `data_providers.streaming` (optional tick-stream provider for the level-breach monitor), `scanning.pairs`/`timeframes`, per-pair `blocked_hours_utc`, `log_level`. Key env vars: `TELEGRAM_TOKEN`, `TELEGRAM_CHAT_ID`, `TELEGRAM_CHANNEL_ID`, `TWELVE_DATA_API_KEY`, `TIINGO_API_KEY`, `OANDA_ACCOUNT_ID`/`OANDA_API_TOKEN`/`OANDA_BASE_URL`, `DEEPSEEK_API_KEY` (news LLM; the `deepseek` config.json block holds model/enabled/timeout/daily cap). The `agent_panel` config.json block (all-defaults = feature dark) controls the advisory LLM agent panel: `enabled`, per-surface `telegram`/`dashboard` toggles, `min_quality`, `max_panels_per_day`, `reuse_ttl_seconds`.
Precedence for every credential/deployment setting: **env var → config.json → refuse to run** (no hardcoded fallbacks). A gitignored repo-root `.env` is loaded at `app_config` import via `os.environ.setdefault`. `config.json` (gitignored; schema in `config.json.example`) holds secrets plus deployment settings: `data_providers.active` (the live-feed switch), `data_providers.streaming` (optional tick-stream provider for the level-breach monitor), `scanning.pairs`/`timeframes`, per-pair `blocked_hours_utc`, `log_level`. Key env vars: `TELEGRAM_TOKEN`, `TELEGRAM_CHAT_ID`, `TELEGRAM_CHANNEL_ID`, `TWELVE_DATA_API_KEY`, `TIINGO_API_KEY`, `OANDA_ACCOUNT_ID`/`OANDA_API_TOKEN`/`OANDA_BASE_URL`, `DEEPSEEK_API_KEY` (news LLM; the `deepseek` config.json block holds model/enabled/timeout/daily cap).
## Architecture
@@ -60,7 +59,6 @@ Signal currency everywhere is `SignalStrength` (−2..+2). DataFrames are UTC tz
- **`signal_aggregator.py`** — MTF agreement, diminishing-decay penalty system, exhaustion hard block, quality grading, then an ordered chain of post-grade gates (session, XAU SELL quality/vol/hour, regime blocks, M5 noise gates), then three setup generators: XAU ATR (A+/A), generic swing-based, plus the M5 pullback (`XAU_PB`) and chart-pattern (`XAU_CP`, `chart_patterns.py`) entry paths. The PB/CP generators run AFTER the gate chain and force quality to "B"; the CP path deliberately accepts grade-F composites and re-checks only session/day/big-candle (via warning prefixes) plus its own SELL/vol/RSI gates — it does NOT re-check exhaustion or the M5 min-score/MTF gates. **This is validated behavior, not a bug** (2026-08-04 audit: 226/586 trades in the shipped M5 12mo book are XAU_CP, +38.9R, 104 of them exhaustion-flagged and net-positive) — changing it is a rule-1 revalidation event. `TradeSetup.strategy` selects trailing params; trail params are stamped on the setup from the *producing* config so per-TF overrides reach formatters.
- **`pair_signals/`** — per-pair config factories. **Always use the factory (`produce_signal` / `make_config`), never bare `PlatformConfig()`** — the factory applies the config.json overlay first, then the validated per-TF deltas (M5 hour-9 block, M15 trail arm 0.70, M15 SELL-hour block, etc.; evidence in the docstring). A test enforces factory ≡ inline-config equivalence. Confirmation map: M5→[M15,H1], M15→[M5,H1], H1→[M15,H4], H4→[H1,D1].
- **`news_engine.py`** — Forex Factory calendar (MyFXBook RSS fallback) + RSS keyword sentiment with gold-inverse keywords. Class-level caches (fresh engine per scan). Optional DeepSeek enrichment via `llm_client.py` (requests-only, no SDK): batched headline sentiment overriding keyword scores, gold-direction on release surprises, and a market-summary blurb (`SignalReport.news_summary`) — **display/warning-only, keyword fallback on any failure, inert without a key**; prompts are static module constants (DeepSeek prefix-cache pricing) and requests send `thinking: disabled` (V4-Flash reasoning otherwise blows the timeout). By XAU-tuned default, event risk and sentiment **warn but do not gate signals** (`event_risk_caps_signals=False`, `sentiment_affects_signals=False`) — the LLM does not change that.
- **`agent_panel.py`** — ai-hedge-fund-style advisory panel: 4 DeepSeek persona calls (macro/technical/risk/contrarian, concurrent on a panel-private pool) + 1 synthesizer producing an **ADVISORY-ONLY** consensus (`AGREE/NEUTRAL/OPPOSE` + confidence + rationale) written onto `SignalReport.panel_*` fields. Attached **post-hoc** by consumers (`telegram_bot` alert path, `main --panel`, `dashboard_publisher`) — never runs inside `analyze_from_dataframes` (backtest parity, test-locked) and no gate reads its fields. Static prompts (prefix caching); spend bounded by an exact per-bar cache, a reuse TTL keyed (symbol, TF, direction) so persisting signals don't re-spend every bar, and a `max_panels_per_day` UTC budget (default 20 ≈ ≤100 requests) on top of the shared 500/day client cap. `<2` valid persona takes skips the synthesizer; any failure degrades to no panel, never a changed signal.
- **`telegram_bot.py`** — owner-locked bot + subscriber channel; candle-boundary auto-scan loops (M5/M15/H1/H4, boundary+10 s), alert quality threshold + fingerprint dedup (committed only after a successful send), 📰 LLM market-context line on alerts when available, WebSocket level-breach monitor fed by the streaming provider (`data_providers.streaming`, falling back to the active provider; auto-disabled when the resolved provider can't stream or has no API key — Oanda does not stream, Tiingo does). Rotating log: `logs/telegram.log`, 3-h rotation × 240 backups.
- **`oanda_daemon.py` / `oanda_executor.py`** — practice-account auto-execution. Book-parity rules (see daemon docstring): no broker-side TP (trailing model — TP1/TP2 advisory), 5-bar per-TF cooldown (wall-clock approximation, so it expires over weekends where the backtester would still count bars), trail ratchets at bar close once per *observed* bar (a missed/failed cycle skips those bars entirely — the intra-gap high/low never enters the peak). Accepted divergence: one position total; orders send `positionFill=OPEN_ONLY` so a state/broker divergence cancels the order instead of silently netting. State/broker re-sync runs at boot and after any suspicious fill (`_reconcile_positions`).
- **`dashboard_publisher.py` / `static/dashboard.html`** — web dashboard as a **static publisher** (live LAMP server): standalone daemon scans at candle closes via `produce_signal` and atomically writes `snapshot.json`/`news.json`/`dashboard.html` to `$DASHBOARD_OUT_DIR` (Apache serves them; page is public by owner decision). **Hard independence rule: never imports, and is never imported by, `telegram_bot`/`oanda_daemon`** — confirmation map and boundary math are deliberate local copies, so a web-side failure can only mean a stale page. Due-TF classification uses the RAW wake time (flooring wake−buffer skips every H1/H4 scan — 2026-08-02 bug). News panel reads `NewsEngine.get_fundamental_context()` plus the additive `get_week_events()` accessor (display-only; no signal path calls it); as a separate process it has its own news caches and DeepSeek daily cap. News/calendar rows are clickable (detail dialog); headlines carry `id` = sha1(headline) in news.json, and exactly one on-demand endpoint exists: `static/news_summary.php` (installed by `install_static`), which serves a DeepSeek gold-impact summary per headline — whitelist-gated against news.json ids, cache-forever in `ai/`, own fail-closed daily cap (150/day, file-based, separate from the publisher's), key at `/etc/gold-dashboard/deepseek.key`, never reads repo config. Deployment checklist: `deploy/README-dashboard.md`.
-582
View File
@@ -1,582 +0,0 @@
"""
ForexBot2 — LLM agent panel (advisory consensus)
=================================================
ai-hedge-fund-style persona panel for XAU/USD signals: four DeepSeek
personas (macro, technical, risk, contrarian) each review a finished
SignalReport from their own angle, then a synthesizer chair weighs the
takes and issues a consensus verdict (AGREE / NEUTRAL / OPPOSE with
confidence and a short rationale).
Two invariants, both load-bearing (CLAUDE.md rules 1 and 4):
- ADVISORY / DISPLAY-ONLY. The verdict is written onto SignalReport
fields that no gate and no scoring path ever reads. Any failure —
LLM down, cap exhausted, garbage reply — degrades to "no panel",
never to a changed or blocked signal.
- NEVER runs inside ``analyze_from_dataframes``. The backtester replays
that function; consumers (telegram_bot, main --panel,
dashboard_publisher) attach the panel POST-HOC via ``attach_panel``,
so replayed books can never see a panel field. Backtest parity holds
by construction.
Cost model: every full panel spends 5 requests (4 personas + 1
synthesizer) against the shared DeepSeek daily cap (llm_client). Two
layers bound the spend on top of that cap:
- ``max_panels_per_day`` (config, default 20) — a UTC-day budget of
panel *attempts*, mirroring the client's request cap.
- A reuse TTL (default 3600 s) keyed on (symbol, timeframe, direction):
a persisting signal re-serves its cached verdict across bars instead
of re-spending. Without this, one sticky M5 dashboard signal would
burn 5 calls per 5-minute bar — 1,440/day against the 500 cap.
Prompts are STATIC module constants (DeepSeek prefix-cache pricing —
see news_engine.py); all variable content travels in the user payload.
"""
import datetime as _dt
import logging
import threading
import time
from collections import OrderedDict
from concurrent.futures import ThreadPoolExecutor, wait as _fwait
from dataclasses import dataclass, field
from typing import TYPE_CHECKING, Any, Dict, List, Optional, Tuple
from app_config import get_app_config
from llm_client import DEFAULT_TIMEOUT, get_llm_client
if TYPE_CHECKING: # type hints only — keep the coupling one-way
from news_engine import FundamentalContext
from signal_aggregator import SignalReport
logger = logging.getLogger(__name__)
DEFAULT_MIN_QUALITY = "B"
DEFAULT_MAX_PANELS_PER_DAY = 20
DEFAULT_REUSE_TTL_SECONDS = 3600.0
# Quality ranks for the min_quality gate. Deliberately a local copy —
# telegram_bot's ranking is private and dashboard_publisher must not
# import telegram_bot (hard independence rule).
_QUALITY_RANK = {"A+": 4, "A": 3, "B": 2, "C": 1}
# ── DeepSeek prompts ─────────────────────────────────────────────────
# STATIC constants on purpose: DeepSeek prices a repeated byte-identical
# prompt prefix at the cache-hit rate (~50x cheaper), so nothing here may
# be interpolated or timestamped. All variable content travels in the
# user payload. Panel output is display-only — it never gates signals.
_GOLD_DOCTRINE = """\
Gold doctrine: hawkish Fed / rate hikes / rising real yields / strong
dollar are gold-bearish; easing / rate cuts / safe-haven flows /
geopolitical stress / weak dollar are gold-bullish.
"""
_TAKE_CONTRACT = """\
Your review is advisory research on an already-generated signal; you are
one voice on a panel, not the decision maker.
Hard rules: use ONLY the data in the JSON you receive — do not invent
numbers, events, or price levels. If the evidence your role needs is
thin or missing, answer "neutral" with confidence below 30, unless your
role's instructions say otherwise.
Confidence scale (0-100): 80-100 strong one-sided evidence; 50-79 solid
case with some doubt; 30-49 mixed evidence; below 30 little basis to
judge.
Write the note in plain everyday English for someone who has never
traded: short simple sentences, no trading jargon, no acronyms, no
abbreviations. Say "profit versus loss", not "RR"; "the 4-hour chart",
not "H4"; describe what an indicator MEANS ("the price has climbed
unusually fast"), never its name or number.
Reply with ONLY JSON:
{"stance": "support"|"oppose"|"neutral", "confidence": <0-100>,
"note": "<max 25 words>"}
"stance" says whether YOUR analysis supports the proposed trade
direction, opposes it, or is genuinely mixed; "confidence" is how sure
you are of your stance; "note" is your single most important point.
"""
_MACRO_SYSTEM = """\
You are the macro analyst on a gold (XAU/USD) trade-review panel. You
receive JSON: a proposed trade brief ("brief": direction, timeframe,
quality, price, regime) plus the current fundamental picture — recent
"headlines", today's high-impact "events" (released: true means the
number is out), a news "sentiment" score for gold, and an optional
"news_summary" blurb.
Judge ONE thing: does the macro backdrop support the proposed direction?
Weigh released surprises over stale headlines; an imminent unreleased
high-impact event is a reason for lower confidence, not a direction.
Stance rules:
- support: the released data and news flow clearly push gold in the
proposed direction.
- oppose: they clearly push gold the other way.
- neutral: the picture is mixed or stale — or there are no meaningful
headlines or events to judge (score that low confidence).
""" + _GOLD_DOCTRINE + """\
Example: brief direction BUY, headlines dominated by hawkish Fed
repricing and a hot CPI print -> {"stance": "oppose", "confidence": 70,
"note": "Inflation is running hot, so interest rates should stay high — that usually pushes gold down."}
Example: brief direction BUY, headlines led by an escalating conflict
and a soft jobs report -> {"stance": "support", "confidence": 75,
"note": "Worries about war and a cooling economy usually send money into gold."}
""" + _TAKE_CONTRACT
_TECH_SYSTEM = """\
You are the technical analyst on a gold (XAU/USD) trade-review panel.
You receive JSON: a proposed trade brief ("brief") plus the technical
evidence — per-"indicators" signal/confidence, multi-timeframe
"mtf" signals and scores, "key_levels" versus the current price, and an
"exhaustion" count of consecutive same-direction bars flagged by the
engine.
Judge ONE thing: is the technical picture internally coherent for the
proposed direction? Look for indicators fighting each other, the signal
fighting the regime (e.g. a BUY in trending_down), higher timeframes
disagreeing, price pressed into a key level, or an exhausted move being
chased.
Stance rules:
- support: the indicators, higher timeframes, and regime mostly agree
with the proposed direction and none of the conflicts above appear.
- oppose: one or more of those conflicts is present and serious.
- neutral: the evidence genuinely cuts both ways, or too much of it is
missing to judge (score that low confidence).
Example: brief direction SELL in regime trending_up with H1 and H4 both
BUY -> {"stance": "oppose", "confidence": 80,
"note": "This bets on gold falling while the bigger picture still points up."}
Example: brief direction BUY in regime trending_up with M15 and H1 both
BUY and exhaustion 0 -> {"stance": "support", "confidence": 70,
"note": "Every way of looking at the chart points up, so buying goes with the flow."}
""" + _TAKE_CONTRACT
_RISK_SYSTEM = """\
You are the risk manager on a gold (XAU/USD) trade-review panel. You
receive JSON: a proposed trade brief ("brief") plus the risk facts —
the trade "setup" (entry, stop_loss, take_profit_1/2, risk_reward,
risk_dollars), the "volatility" regime, engine "warnings", the calendar
"event_risk" level, and the assumed "spread".
Judge ONE thing: is this setup's risk acceptable as constructed? Check
reward-to-risk adequacy, stop distance versus the volatility regime
(a tight stop in extreme vol dies to noise; a huge stop in low vol is
oversized), spread as a fraction of the stop, and event landmines about
to hit.
Stance rules:
- support: the likely reward clearly justifies the risk, the stop fits
the current volatility, and no event landmine is imminent.
- oppose: reward too thin for the risk, a stop mismatched to the
volatility, spread eating a big share of the stop, an imminent
high-impact event — or no setup at all (a missing setup means the
signal graded too low for one; oppose unless the rest is
exceptional).
- neutral: acceptable but unremarkable — nothing dangerous, nothing
attractive.
Example: RR 1.4 with volatility extreme and a high-impact event due ->
{"stance": "oppose", "confidence": 65,
"note": "The possible profit is small for such wild price swings, and big economic news is due soon."}
Example: RR 2.4 with normal volatility and no events due ->
{"stance": "support", "confidence": 70,
"note": "The possible profit is more than double the possible loss, and conditions look calm."}
""" + _TAKE_CONTRACT
_CONTRARIAN_SYSTEM = """\
You are the contrarian on a gold (XAU/USD) trade-review panel — the
designated skeptic. You receive JSON: a proposed trade brief ("brief"),
the engine's own "bullish_reasons" and "bearish_reasons", its
"warnings", and an optional "news_summary".
Your job: construct the strongest honest case AGAINST the proposed
trade, then score how damaging that case is. Use "oppose" only when the
counter-case is genuinely strong, "neutral" when it is real but modest,
and "support" when even the skeptic finds nothing that bites.
""" + _GOLD_DOCTRINE + """\
Example: a BUY whose own engine lists three bearish reasons and a
counter-trend warning -> {"stance": "oppose", "confidence": 60,
"note": "The system's own list of negatives looks stronger than its reason to buy."}
""" + _TAKE_CONTRACT
_SYNTH_SYSTEM = """\
You are the chair of a gold (XAU/USD) trade-review panel. You receive
JSON: the proposed trade "direction" and the panel's "takes" — one per
analyst ("persona", "stance", "confidence", "note").
Weigh the takes into ONE consensus on whether to act on the signal:
"AGREE" (the panel backs the trade), "OPPOSE" (the panel is against it),
or "NEUTRAL" (genuinely split or low conviction). Weigh a high-
confidence oppose from any single analyst seriously — one strong veto
can outweigh two lukewarm supports. Do not average blindly.
Base the verdict and rationale ONLY on the takes given — do not add
outside analysis.
Confidence scale (0-100): 80-100 the panel clearly leans one way; 50-79
a lean with real dissent; below 50 split or low conviction.
Write the rationale in plain everyday English for someone who has never
traded: short simple sentences, no trading jargon, no acronyms, no
abbreviations.
Example rationale: "Three of four analysts are against this: the
possible profit is too small and the price has already climbed
unusually fast. Only the news backdrop supports buying."
Reply with ONLY JSON:
{"verdict": "AGREE"|"NEUTRAL"|"OPPOSE", "confidence": <0-100>,
"rationale": "<max 40 words>"}
"""
# ── Result types ─────────────────────────────────────────────────────
@dataclass
class PersonaTake:
persona: str # "macro" | "technical" | "risk" | "contrarian"
stance: str # "support" | "oppose" | "neutral"
confidence: float # 0-100
note: str
@dataclass
class PanelVerdict:
verdict: str # "AGREE" | "NEUTRAL" | "OPPOSE"
confidence: float # 0-100
rationale: str
takes: List[PersonaTake] = field(default_factory=list)
created_ts: float = 0.0
# ── Payload builders (pure functions; llm_client json-dumps sorted) ──
def _direction(report: "SignalReport") -> Optional[str]:
"""BUY/SELL the panel should review; None = nothing to review."""
if report.trade_setup is not None:
return report.trade_setup.direction
if report.final_signal.value > 0:
return "BUY"
if report.final_signal.value < 0:
return "SELL"
return None
def _brief(report: "SignalReport") -> Dict[str, Any]:
analysis = report.primary_analysis
return {
"pair": report.display_name,
"timeframe": analysis.timeframe if analysis else "",
"direction": _direction(report),
"quality": report.signal_quality,
"score": round(report.final_score, 1),
"confidence": round(report.final_confidence),
"price": round(report.current_price, 2),
"market_regime": report.market_regime,
"volatility_regime": report.volatility_regime,
"mtf_agreement": report.mtf_agreement,
}
def _macro_payload(report: "SignalReport",
context: Optional["FundamentalContext"]) -> Dict[str, Any]:
headlines: List[str] = []
events: List[Dict[str, Any]] = []
if context is not None:
headlines = [h.headline for h in context.news_headlines[:10]]
events = [
{"event": e.event, "currency": e.currency,
"actual": e.actual, "forecast": e.forecast,
"released": not e.is_upcoming}
for e in context.events_today if e.is_high_impact
][:8]
return {
"brief": _brief(report),
"headlines": headlines,
"events": events,
"sentiment": round(report.news_sentiment, 3),
"news_summary": report.news_summary,
}
def _technical_payload(report: "SignalReport") -> Dict[str, Any]:
indicators: Dict[str, Dict[str, Any]] = {}
key_levels: Dict[str, float] = {}
analysis = report.primary_analysis
if analysis is not None:
# Same visibility filter as main.report_to_dict: private series
# ride as _-prefixed pseudo-indicators, weight-0 = context-only.
indicators = {
name: {"signal": ind.signal.name,
"confidence": round(ind.confidence)}
for name, ind in analysis.indicators.items()
if not name.startswith("_") and ind.weight != 0
}
key_levels = {k: round(v, 2) for k, v in analysis.key_levels.items()}
return {
"brief": _brief(report),
"indicators": indicators,
"mtf": {tf: {"signal": sig.name,
"score": round(report.mtf_scores.get(tf, 0.0), 1)}
for tf, sig in report.mtf_signals.items()},
"key_levels": key_levels,
"exhaustion": report.exhaustion_count,
}
def _risk_payload(report: "SignalReport") -> Dict[str, Any]:
payload: Dict[str, Any] = {
"brief": _brief(report),
"volatility": report.volatility_regime,
"warnings": list(report.warnings),
"event_risk": report.event_risk,
"event_description": report.event_description,
"spread": round(report.spread_estimate, 2),
}
ts = report.trade_setup
if ts is not None:
payload["setup"] = {
"direction": ts.direction,
"entry": round(ts.entry_price, 2),
"stop_loss": round(ts.stop_loss, 2),
"take_profit_1": round(ts.take_profit_1, 2),
"take_profit_2": round(ts.take_profit_2, 2),
"risk_reward": round(ts.risk_reward, 2),
"risk_dollars": round(ts.risk_pips * ts.pip_value, 2),
}
return payload
def _contrarian_payload(report: "SignalReport") -> Dict[str, Any]:
return {
"brief": _brief(report),
"bullish_reasons": report.bullish_reasons[:3],
"bearish_reasons": report.bearish_reasons[:3],
"warnings": list(report.warnings),
"news_summary": report.news_summary,
}
# ── Orchestration ────────────────────────────────────────────────────
# Panel-private pool: the 4 persona calls run concurrently WITHOUT
# occupying telegram_bot's shared scan executor (which has 4 workers —
# borrowing them would stall every other scan for the panel's duration).
_pool = ThreadPoolExecutor(max_workers=4, thread_name_prefix="agent-panel")
_VALID_STANCES = ("support", "oppose", "neutral")
_VALID_VERDICTS = ("AGREE", "NEUTRAL", "OPPOSE")
def _clamp_confidence(raw: Any) -> Optional[float]:
try:
return max(0.0, min(100.0, float(raw)))
except (TypeError, ValueError):
return None
def _validate_take(persona: str, reply: Any) -> Optional[PersonaTake]:
if not isinstance(reply, dict):
return None
stance = reply.get("stance")
if stance not in _VALID_STANCES:
return None
confidence = _clamp_confidence(reply.get("confidence"))
if confidence is None:
return None
note = str(reply.get("note", "")).strip()[:160]
return PersonaTake(persona=persona, stance=stance,
confidence=confidence, note=note)
def _ask_persona(persona: str, system: str,
payload: Dict[str, Any]) -> Optional[PersonaTake]:
# Client resolved at call time: tests install stubs via lc._client.
reply = get_llm_client().chat_json(system, payload)
return _validate_take(persona, reply)
def run_panel(report: "SignalReport",
context: Optional["FundamentalContext"] = None,
) -> Optional[PanelVerdict]:
"""Run 4 personas + synthesizer; None = no verdict (fail-safe).
Requires ≥2 valid persona takes before spending the synthesizer
request; the synthesizer's reply is validated strictly and there is
deliberately NO programmatic fallback verdict — a panel that could
not really convene reports nothing rather than something invented.
"""
direction = _direction(report)
if direction is None:
return None
jobs = (
("macro", _MACRO_SYSTEM, _macro_payload(report, context)),
("technical", _TECH_SYSTEM, _technical_payload(report)),
("risk", _RISK_SYSTEM, _risk_payload(report)),
("contrarian", _CONTRARIAN_SYSTEM, _contrarian_payload(report)),
)
futures = [_pool.submit(_ask_persona, name, system, payload)
for name, system, payload in jobs]
# The HTTP timeout inside the client bounds each call; the margin
# covers pool queueing. Stragglers past the wait are abandoned (their
# threads finish harmlessly) and simply don't vote.
client_timeout = getattr(get_llm_client(), "_timeout", DEFAULT_TIMEOUT)
done, _ = _fwait(futures, timeout=client_timeout + 8.0)
takes = [f.result() for f in done if not f.exception()]
takes = [t for t in takes if t is not None]
if len(takes) < 2:
logger.debug("agent panel: %d/4 persona takes — skipping synthesizer",
len(takes))
return None
order = {"macro": 0, "technical": 1, "risk": 2, "contrarian": 3}
takes.sort(key=lambda t: order.get(t.persona, 9))
reply = get_llm_client().chat_json(_SYNTH_SYSTEM, {
"direction": direction,
"takes": [{"persona": t.persona, "stance": t.stance,
"confidence": t.confidence, "note": t.note}
for t in takes],
})
if not isinstance(reply, dict):
return None
verdict = reply.get("verdict")
confidence = _clamp_confidence(reply.get("confidence"))
if verdict not in _VALID_VERDICTS or confidence is None:
return None
rationale = str(reply.get("rationale", "")).strip()[:240]
return PanelVerdict(verdict=verdict, confidence=confidence,
rationale=rationale, takes=takes,
created_ts=time.time())
# ── Cache, budget, attach ────────────────────────────────────────────
_state_lock = threading.Lock()
# Exact result per reviewed bar: repeated attaches on the same signal
# (re-scan, multi-consumer) are free.
_cache: "OrderedDict[Tuple[str, str, str, str], PanelVerdict]" = OrderedDict()
_CACHE_MAX = 32
# Reuse layer: a persisting signal (same symbol/tf/direction, newer bar)
# re-serves its last verdict for reuse_ttl_seconds instead of re-spending
# 5 requests per bar (see module docstring for the M5 dashboard math).
_last_by_dir: Dict[Tuple[str, str, str], PanelVerdict] = {}
# UTC-day budget of panel attempts (mirrors llm_client._under_cap).
_panel_day: Optional[_dt.date] = None
_panel_count: int = 0
_warned_budget = False
def reset_panel_state() -> None:
"""Drop caches and counters (tests / config reloads)."""
global _panel_day, _panel_count, _warned_budget
with _state_lock:
_cache.clear()
_last_by_dir.clear()
_panel_day = None
_panel_count = 0
_warned_budget = False
def panel_health() -> Dict[str, Any]:
"""Read-only counters for status displays."""
settings = get_app_config().agent_panel_settings
cap = int(settings.get("max_panels_per_day", DEFAULT_MAX_PANELS_PER_DAY))
today = _dt.datetime.now(_dt.timezone.utc).date()
with _state_lock:
used = _panel_count if _panel_day == today else 0
return {"used_today": used, "cap": cap}
def _under_panel_budget(cap: int) -> bool:
"""Count one panel attempt against the UTC-day budget."""
global _panel_day, _panel_count, _warned_budget
today = _dt.datetime.now(_dt.timezone.utc).date()
with _state_lock:
if _panel_day != today:
_panel_day = today
_panel_count = 0
_warned_budget = False
if _panel_count >= cap:
if not _warned_budget:
_warned_budget = True
logger.warning(
"agent panel: daily budget (%d) exhausted — no more "
"panels until UTC midnight", cap)
return False
_panel_count += 1
return True
def _write_fields(report: "SignalReport", v: PanelVerdict) -> None:
report.panel_verdict = v.verdict
report.panel_confidence = float(v.confidence)
report.panel_rationale = v.rationale
report.panel_takes = [
{"persona": t.persona, "stance": t.stance,
"confidence": t.confidence, "note": t.note} for t in v.takes]
def attach_panel(report: "SignalReport",
settings: Optional[Dict[str, Any]] = None,
surface: str = "telegram",
force: bool = False,
context: Optional["FundamentalContext"] = None) -> bool:
"""Best-effort panel attach; True iff panel fields were populated.
Advisory feature contract: NEVER raises — an exploding panel must
not cost an alert send or a dashboard publish. ``force`` (CLI
--panel) bypasses the enabled/surface/quality toggles but never the
daily budget or the shared LLM cap.
"""
try:
if settings is None:
settings = get_app_config().agent_panel_settings
if not force:
if not settings.get("enabled", False):
return False
surface_on = {
"telegram": settings.get("telegram", True),
"dashboard": settings.get("dashboard", False),
}.get(surface, True)
if not surface_on:
return False
min_quality = settings.get("min_quality", DEFAULT_MIN_QUALITY)
if (_QUALITY_RANK.get(report.signal_quality, -1)
< _QUALITY_RANK.get(min_quality, 2)):
return False
direction = _direction(report)
if direction is None:
return False
analysis = report.primary_analysis
tf = analysis.timeframe if analysis else ""
exact_key = (report.symbol, tf, report.timestamp, direction)
dir_key = (report.symbol, tf, direction)
ttl = float(settings.get("reuse_ttl_seconds",
DEFAULT_REUSE_TTL_SECONDS))
with _state_lock:
hit = _cache.get(exact_key)
if hit is None:
prior = _last_by_dir.get(dir_key)
if prior is not None and time.time() - prior.created_ts < ttl:
hit = prior
if hit is not None:
_write_fields(report, hit)
return True
cap = int(settings.get("max_panels_per_day",
DEFAULT_MAX_PANELS_PER_DAY))
if not _under_panel_budget(cap):
return False
if context is None:
# Best-effort; the news engine's class-level 600 s caches make
# this nearly free when warm in both bot and dashboard.
try:
from config import PlatformConfig
from news_engine import NewsEngine
context = NewsEngine(
PlatformConfig().news).get_fundamental_context()
except Exception as e:
logger.debug("agent panel: news context unavailable: %s", e)
context = None
verdict = run_panel(report, context)
if verdict is None:
return False
_write_fields(report, verdict)
with _state_lock:
_cache[exact_key] = verdict
while len(_cache) > _CACHE_MAX:
_cache.popitem(last=False)
_last_by_dir[dir_key] = verdict
return True
except Exception:
logger.exception("agent panel: attach failed — signal unaffected")
return False
+1 -31
View File
@@ -86,7 +86,7 @@ _VALID_LOG_LEVELS = ("DEBUG", "INFO", "WARNING", "ERROR", "CRITICAL")
_SCANNABLE_TFS = ("M5", "M15", "H1", "H4")
_KNOWN_TOP = {"telegram", "data_providers", "scanning", "pairs", "log_level",
"oanda", "deepseek", "agent_panel"}
"oanda", "deepseek"}
_KNOWN_OANDA = {
"account_id", "api_token", "base_url", "lot_size",
"quality_gate", "max_hold_bars",
@@ -94,10 +94,6 @@ _KNOWN_OANDA = {
_KNOWN_DEEPSEEK = {
"api_key", "model", "enabled", "timeout_seconds", "daily_request_cap",
}
_KNOWN_AGENT_PANEL = {
"enabled", "telegram", "dashboard", "min_quality",
"max_panels_per_day", "reuse_ttl_seconds",
}
_KNOWN_TELEGRAM = {"token", "chat_id", "channel_id", "alert_min_quality"}
_KNOWN_SCANNING = {"pairs", "timeframes"}
_KNOWN_PAIR_KEYS = {"blocked_hours_utc"}
@@ -122,7 +118,6 @@ class AppConfig:
pair_overrides: Dict[str, Dict[str, Any]] = field(default_factory=dict)
oanda_settings: Dict[str, Any] = field(default_factory=dict)
deepseek_settings: Dict[str, Any] = field(default_factory=dict)
agent_panel_settings: Dict[str, Any] = field(default_factory=dict)
log_level: str = "INFO"
@@ -281,31 +276,6 @@ def load_app_config(path: str = CONFIG_PATH) -> AppConfig:
"config.json: deepseek.model must be a non-empty string")
app.deepseek_settings = dict(deepseek)
panel = raw.get("agent_panel", {})
_check_keys(panel, _KNOWN_AGENT_PANEL, '"agent_panel"')
for flag in ("enabled", "telegram", "dashboard"):
if flag in panel and not isinstance(panel[flag], bool):
raise AppConfigError(
f"config.json: agent_panel.{flag} must be true or false")
if "min_quality" in panel and panel["min_quality"] not in _VALID_QUALITIES:
raise AppConfigError(
f'config.json: agent_panel.min_quality "{panel["min_quality"]}" '
f"not in {_VALID_QUALITIES}")
if "max_panels_per_day" in panel:
mpd = panel["max_panels_per_day"]
if isinstance(mpd, bool) or not isinstance(mpd, int) or mpd < 1:
raise AppConfigError(
"config.json: agent_panel.max_panels_per_day must be a "
"positive integer")
if "reuse_ttl_seconds" in panel:
ttl = panel["reuse_ttl_seconds"]
if isinstance(ttl, bool) or not isinstance(ttl, (int, float)) \
or ttl < 0:
raise AppConfigError(
"config.json: agent_panel.reuse_ttl_seconds must be a "
"non-negative number")
app.agent_panel_settings = dict(panel)
level = raw.get("log_level", app.log_level)
if level not in _VALID_LOG_LEVELS:
raise AppConfigError(
-8
View File
@@ -40,13 +40,5 @@
"timeout_seconds": 12,
"daily_request_cap": 500
},
"agent_panel": {
"enabled": false,
"telegram": true,
"dashboard": false,
"min_quality": "B",
"max_panels_per_day": 20,
"reuse_ttl_seconds": 3600
},
"log_level": "INFO"
}
-3
View File
@@ -166,9 +166,6 @@ class NewsConfig:
# Must stay below pre_event_buffer_minutes so an event can't cross
# into the danger window while a stale cache says it's clear.
context_cache_ttl_seconds: int = 600
# Maximum age of RSS headlines accepted for scoring. Older headlines
# are discarded before keyword analysis and LLM enrichment.
headline_max_age_seconds: int = 43200 # 12 hours
rss_feeds: List[str] = field(default_factory=lambda: [
"https://investinglive.com/feed/",
-8
View File
@@ -49,7 +49,6 @@ from datetime import datetime, timezone
from logging.handlers import TimedRotatingFileHandler
from typing import Dict, List, Optional
from agent_panel import attach_panel
from config import PlatformConfig, Timeframe, XAUUSD
from main import report_to_dict
from news_engine import NewsEngine
@@ -358,13 +357,6 @@ class DashboardPublisher:
try:
report = produce_signal(XAUUSD, tf, self._provider(),
CONFIRMATIONS[tf])
# Advisory AI panel, post-hoc and display-only. Off unless
# config.json agent_panel.dashboard is true. The reuse-TTL
# cache + max_panels_per_day budget bound the spend against
# this process's own DeepSeek cap (a persisting M5 signal
# re-serves its cached verdict instead of re-running every
# bar); a fresh panel adds ≤~25 s before this TF publishes.
attach_panel(report, surface="dashboard")
self.snapshots[tf.value] = {
"status": "ok",
"scanned_at": datetime.now(timezone.utc).isoformat(),
-252
View File
@@ -1,252 +0,0 @@
================================================================================
MULTI-PROVIDER LLM LAYER - PLAN
(DeepSeek for news summary, Anthropic Claude for agent panel)
Written 2026-08-08. Plan only - no code has been changed.
================================================================================
WHY
---
Today every LLM feature (news enrichment + agent panel) goes through one
hardwired client: llm_client.py -> DeepSeekClient, a module singleton reached
via get_llm_client(). This plan describes how to route different features to
different providers - e.g. keep DeepSeek for cheap high-volume news scoring,
put the agent panel on Anthropic Claude for better reasoning.
The repo already solved this problem once for market data: the providers/
package (BaseDataProvider owns shared machinery, subclasses implement only
_fetch_raw, a registry + one config knob selects the provider). This plan
mirrors that proven pattern at the LLM layer, scaled down to a single module.
BLAST RADIUS (verified by grep)
-------------------------------
get_llm_client() call sites:
- news_engine.py ....... 4 sites
- agent_panel.py ....... 3 sites (+1 timeout getattr)
- telegram_bot.py ...... 1 site (/status health; reads private _count,
_count_day, _daily_cap fields)
- test_all.py .......... 41 locations stub via lc._client = _StubLLM(...)
Out of scope: static/news_summary.php (own PHP DeepSeek call + key file,
separate deployment).
DESIGN INVARIANTS (carried over unchanged)
------------------------------------------
1. Fail-safe contract: every call returns Optional[Dict]; any failure -> None;
callers keep their non-LLM fallbacks; LLM output never gates signals.
2. Single attempt, no retries (news cache cadence is the retry); per-provider
UTC daily cap; warn-once logging; health stamps (last_ok, last_error).
3. Static system prompts stay byte-identical constants - provider-neutral,
and they map to prompt caching on BOTH providers (DeepSeek prefix cache;
Anthropic cache_control ephemeral).
4. Test compatibility: all 41 existing lc._client stub sites keep working
with zero edits.
================================================================================
1. llm_client.py REFACTOR (~195 -> ~430 lines, no new files)
================================================================================
1a. BaseLLMClient - shared machinery (extracted from today's DeepSeekClient)
- __init__(api_key, model, timeout, daily_cap, enabled)
- public: name ("deepseek"/"anthropic"), timeout property,
usage() -> (used_today, cap) # so telegram_bot stops reading privates
- owns: available property, _warn_once, _under_cap() (UTC-day counter,
lock), health stamps, and the template method:
def chat_json(self, system_prompt, user_payload):
if not self.available or not self._under_cap():
return None
try:
return self._request(system_prompt, user_payload) # hook
except <provider-agnostic error mapping>:
# stamps last_error, returns None
- subclasses implement ONLY _request(system_prompt, payload) -> dict|None.
1b. DeepSeekClient(BaseLLMClient) - behavior-identical move
Current requests-based logic relocates into _request: same URL,
thinking:disabled, response_format:json_object, temperature 0,
max_tokens 2000, sort_keys=True payload, _parse_response.
Review criterion: Module 19 tests pass UNMODIFIED (they patch
client._session, which stays).
1c. AnthropicClient(BaseLLMClient) - new, official SDK
- Official `anthropic` Python SDK (add to requirements.txt), NOT raw
requests. LAZY IMPORT inside __init__: package missing -> warn-once +
client unavailable, so DeepSeek-only deployments never install it.
- Constructor: anthropic.Anthropic(api_key=..., timeout=<configured>,
max_retries=0) # house rule: single attempt
- _request:
resp = self._client.messages.create(
model=self._model, # default "claude-opus-5"
max_tokens=4000, # covers thinking + JSON reply
output_config={"effort": "low"}, # cheap structured takes
system=[{"type": "text", "text": system_prompt,
"cache_control": {"type": "ephemeral"}}],
messages=[{"role": "user",
"content": json.dumps(user_payload, sort_keys=True)}],
)
if resp.stop_reason == "refusal": # Opus 5: HTTP 200, check first
return None
text = first text block in resp.content
return self._parse_json(text) # tolerant: strip ```json fences,
# json.loads, dict check
- DELIBERATELY ABSENT (would 400 on Claude Opus 5): temperature (removed
on Opus 5 - do NOT copy from the DeepSeek client), budget_tokens,
prefills, response_format.
- Error mapping: APIConnectionError -> "network",
APIStatusError -> "http_<code>", parse fail -> "parse"
(same last_error kinds the /status display already understands).
- Defaults: model claude-opus-5 (config-overridable, e.g. claude-haiku-4-5
for ~5x cheaper), timeout 30s (Claude slower than V4-Flash), own
daily_request_cap 500.
1d. Registry + per-consumer routing
_PROVIDERS = {"deepseek": DeepSeekClient, "anthropic": AnthropicClient}
LLM_CONSUMERS = ("news", "agent_panel")
_clients = {} # keyed by PROVIDER name -> two consumers on the
# same provider share one instance and one cap
_client = None # kept: GLOBAL TEST OVERRIDE
def get_llm_client(consumer="news"):
if _client is not None: # tests: lc._client = stub
return _client # -> shadows ALL consumers
name = get_app_config().llm_routing.get(consumer, "deepseek")
create-once per provider, return _clients[name]
def reset_llm_client(): # clears _client AND _clients
Keeping _client as a documented test-override global is what makes the
41 existing stub sites work untouched.
================================================================================
2. CONFIG (app_config.py + config.json.example)
================================================================================
- New top-level "anthropic" block, mirroring "deepseek":
{api_key, model, enabled, timeout_seconds, daily_request_cap}
plus anthropic_api_key() resolver: ANTHROPIC_API_KEY env -> config.json -> "".
- New top-level "llm" ROUTING block:
{"news": "deepseek", "agent_panel": "deepseek"}
Validation: keys within {news, agent_panel}; values within registered
provider names. Absent block -> {} -> everything on DeepSeek (today's
behavior; zero-migration default). AppConfig.llm_routing: Dict[str, str].
- The target state is then PURE CONFIG - no code changes on switch day:
"llm": { "news": "deepseek", "agent_panel": "anthropic" },
"anthropic": { "api_key": "", "model": "claude-opus-5", "enabled": true,
"timeout_seconds": 30, "daily_request_cap": 500 }
================================================================================
3. CONSUMER EDITS (mechanical, ~10 lines total)
================================================================================
- news_engine.py: 4 call sites -> get_llm_client("news")
- agent_panel.py: 3 call sites -> get_llm_client("agent_panel");
getattr(client, "_timeout", ...) -> public client.timeout
- telegram_bot.py _news_health_lines: iterate the DISTINCT providers actually
routed (news + panel); per provider print "<name>: used/cap calls today"
via the new usage() accessor instead of private fields. DeepSeek keeps its
"headlines scored" extra; cap/error warnings work off base-class stamps.
- NO changes to prompts, payload builders, caches, or the panel's own
max_panels_per_day budget - all provider-agnostic already.
================================================================================
4. TESTS (new "Module 24: LLM provider routing", ~12 tests, ~200 lines)
================================================================================
Reuse _StubLLM; new _FakeAnthropic (fake messages.create returning a stub
response object) injected as client._client.
1. app_config: anthropic block parses/validates/defaults empty.
2. app_config: llm routing block - valid map round-trips; unknown consumer
key, unknown provider name, non-dict -> AppConfigError; absent -> {}.
3. anthropic_api_key resolves env -> config.json -> empty.
4. Base contract on BOTH subclasses (parametrized): keyless/disabled -> no
call; daily cap blocks + UTC-roll reset; failure stamps last_error,
success stamps last_ok.
5. AnthropicClient parse: clean JSON -> dict; ```json-fenced -> dict;
non-JSON -> None; stop_reason "refusal" -> None; no text block -> None.
6. AnthropicClient request shape: NO temperature key, max_retries=0,
cache_control present on system block (assert on captured kwargs).
7. AnthropicClient with `anthropic` package missing (simulated) ->
available False, chat_json None, no crash at import.
8. Routing: no llm block -> both consumers get the SAME DeepSeek singleton
(shared cap); routed config -> news gets DeepSeekClient while
agent_panel gets AnthropicClient; same-provider consumers share one
instance.
9. Global lc._client override shadows BOTH consumers (locks the 41 legacy
stub sites' semantics).
10. reset_llm_client() clears both the override and the per-provider cache.
11. End-to-end: routing agent_panel->anthropic, attach_panel reaches the
anthropic fake (5 calls) while a news call reaches the deepseek stub.
12. telegram _news_health_lines renders per-provider lines from usage()
without touching private fields.
GATE: Modules 19 and 23 must pass UNMODIFIED - the no-regression proof for
the refactor.
================================================================================
5. DOCS
================================================================================
- CLAUDE.md: llm_client bullet -> provider-abstracted (BaseLLMClient;
deepseek + anthropic registered); config.json "llm" block routes consumers;
absent = all-DeepSeek. Add ANTHROPIC_API_KEY to env-var list. Test counts.
- config.json.example: both new blocks.
================================================================================
COMMIT SEQUENCE (each commit green)
================================================================================
1. llm_client: extract BaseLLMClient; DeepSeekClient unchanged behavior
(+ base-contract tests) - pure refactor, Modules 19/23 green untouched.
2. config: anthropic block + llm consumer-routing block (+ config tests).
3. llm_client: AnthropicClient (official SDK, lazy import) + requirements.txt
(+ parse/shape tests).
4. routing: per-consumer get_llm_client + health lines - consumer call-site
edits, telegram usage() lines, routing/e2e tests.
5. docs: CLAUDE.md + example config.
================================================================================
VERIFICATION
================================================================================
- python test_all.py green at every commit (~374 -> ~386).
- Live smoke A (no behavior change): default config, main.py --panel ->
still DeepSeek, /status unchanged.
- Live smoke B (the goal): llm.agent_panel="anthropic" + real
ANTHROPIC_API_KEY, run --panel -> one real Claude panel (5 requests,
validates JSON contract + latency + refusal-free path) while a news
refresh still hits DeepSeek; /status shows two provider lines.
- Backtest untouched by construction (no LLM in replay path); one 30-day
run as a formality.
================================================================================
EFFORT ESTIMATE
================================================================================
0 new files, 8 modified, ~520 lines, est. 3-5 hours.
llm_client.py base class + AnthropicClient + routing +235
test_all.py Module 24 +200
app_config.py anthropic + llm blocks, resolver +55
telegram_bot.py per-provider health lines +15
news_engine.py consumer tags on call sites +4
agent_panel.py consumer tags on call sites +4
config.json.example blocks +14
requirements.txt anthropic +1
CLAUDE.md docs +10
================================================================================
KEY DECISIONS EMBEDDED IN THIS PLAN
================================================================================
- Official Anthropic SDK, not raw requests (current API guidance for Python;
lazy import keeps it optional for DeepSeek-only installs).
- Default Claude model: claude-opus-5 ($5/$25 per MTok; a full 5-call panel
~ 8k in / 500 out ~ $0.05). Config-overridable to claude-haiku-4-5.
- Two Claude gotchas handled: temperature is REMOVED on Opus 5 (sending it
400s - must not be copied from the DeepSeek request body), and refusals
arrive as HTTP 200 with stop_reason "refusal" (checked before parsing,
mapped to the normal None/fallback path).
- Caps are per-provider, not per-consumer: two consumers on one provider
share its cap (today's semantics); splitting across two providers doubles
total headroom naturally.
- The panel's own max_panels_per_day budget is untouched - it sits above
whichever provider serves it.
- Not in scope: static/news_summary.php (own PHP client/key), streaming,
tool use, per-consumer model overrides (add a "model" key per consumer
later if ever needed).
================================================================================
-20
View File
@@ -20,7 +20,6 @@ if hasattr(sys.stdout, "reconfigure"):
if hasattr(sys.stderr, "reconfigure"):
sys.stderr.reconfigure(encoding="utf-8")
from agent_panel import attach_panel
from config import PlatformConfig, Timeframe, SignalStrength, XAUUSD
from pair_signals import PAIR_PRODUCERS
from signal_aggregator import SignalAggregator, SignalReport
@@ -164,13 +163,6 @@ def print_signal_report(report: SignalReport, detail: bool = False):
print(f" {Colors.MAGENTA}📰{Colors.RESET} {report.news_description}")
if report.news_summary:
print(f" {Colors.MAGENTA}🧠{Colors.RESET} {report.news_summary}")
if report.panel_verdict:
print(f" {Colors.CYAN}🤖{Colors.RESET} AI Panel: "
f"{Colors.BOLD}{report.panel_verdict}{Colors.RESET} "
f"({report.panel_confidence:.0f}%) — {report.panel_rationale}")
for t in report.panel_takes:
print(f" {Colors.GRAY}{t['persona']:<11} {t['stance']:<8} "
f"{t['note']}{Colors.RESET}")
# ── 6. DETAIL (only with --detail) ─────────────────────────────
if detail and report.primary_analysis:
@@ -217,10 +209,6 @@ def report_to_dict(report: SignalReport) -> dict:
"news_sentiment": round(report.news_sentiment, 3),
"news_description": report.news_description,
"news_summary": report.news_summary,
"panel_verdict": report.panel_verdict,
"panel_confidence": round(report.panel_confidence, 1),
"panel_rationale": report.panel_rationale,
"panel_takes": report.panel_takes,
"bullish_reasons": report.bullish_reasons,
"bearish_reasons": report.bearish_reasons,
"warnings": report.warnings,
@@ -280,9 +268,6 @@ Examples:
help="Primary analysis timeframe (default: H1)")
parser.add_argument("--detail", action="store_true",
help="Show full indicator breakdown")
parser.add_argument("--panel", action="store_true",
help="Run the LLM agent panel on the result "
"(advisory only; needs a DeepSeek key)")
parser.add_argument("--json", action="store_true",
help="Output as JSON")
parser.add_argument("--verbose", "-v", action="store_true",
@@ -338,11 +323,6 @@ def run_live(args, config: PlatformConfig):
aggregator = SignalAggregator(config)
report = aggregator.analyze_pair(symbol=config.symbol)
# Explicit user intent: force bypasses the enabled/surface/quality
# toggles, but the daily panel budget and LLM cap still apply.
if getattr(args, "panel", False):
attach_panel(report, surface="cli", force=True)
if args.json:
print(json.dumps(report_to_dict(report), indent=2))
return
+2 -2
View File
@@ -770,8 +770,8 @@ class NewsEngine:
pub_date_el.text if pub_date_el is not None else None
)
# Only recent headlines (configurable: headline_max_age_seconds)
if pub_date and (datetime.now(timezone.utc) - pub_date).total_seconds() > self.config.headline_max_age_seconds:
# Only recent headlines (last 24h)
if pub_date and (datetime.now(timezone.utc) - pub_date).total_seconds() > 86400:
continue
# Dedup within and across feeds so a syndicated story
-9
View File
@@ -88,15 +88,6 @@ class SignalReport:
news_description: str = ""
news_summary: str = "" # LLM market-context blurb (display-only)
# LLM agent-panel consensus (advisory/DISPLAY-ONLY). Populated
# post-hoc by consumers via agent_panel.attach_panel — NEVER inside
# analyze_from_dataframes (backtest parity) and never read by any
# gate or scoring path.
panel_verdict: str = "" # "AGREE" / "NEUTRAL" / "OPPOSE" / ""
panel_confidence: float = 0.0
panel_rationale: str = ""
panel_takes: List[Dict] = field(default_factory=list)
final_signal: SignalStrength = SignalStrength.NEUTRAL
final_score: float = 0.0
final_confidence: float = 0.0
+25 -85
View File
@@ -630,7 +630,6 @@ const IND_SHORT = {ema_trend:"EMA", rsi:"RSI", macd:"MACD", bollinger:"BB",
const ARROW = {STRONG_BUY:"⇈", BUY:"↑", NEUTRAL:"→", SELL:"↓", STRONG_SELL:"⇊"};
const TF_LABEL = {"5min":"M5", "15min":"M15", "1h":"H1", "4h":"H4"};
const TF_ORDER = ["5min","15min","1h","4h"];
const TF_SECONDS = {"5min": 300, "15min": 900, "1h": 3600, "4h": 14400};
/* 5 s polling is cheap because fetchJSON uses conditional GETs — Apache
answers unchanged files with 304, so most polls transfer no body. */
const POLL_MS = 5000;
@@ -784,8 +783,7 @@ function renderCards() {
const foot = `<div class="tf-foot">
<span>${entry.scanned_at ? "scanned " + fmtAge(entry.scanned_at) : ""}</span>
<span class="cd">closes in
<span class="num" data-close="${entry.next_close_epoch || ""}"
data-interval="${TF_SECONDS[tf]}">--:--</span></span></div>`;
<span class="num" data-close="${entry.next_close_epoch || ""}">--:--</span></span></div>`;
if (!r) {
return `${open}<div class="tf-head"><span class="tf-name">${TF_LABEL[tf]}</span>
<span class="spacer"></span><span class="open-hint">details ›</span></div>
@@ -983,22 +981,6 @@ function detailHTML(tf) {
${esc(r.news_summary || "")}</div>`);
}
/* AI agent panel (advisory) */
if (r.panel_verdict) {
const vcls = r.panel_verdict === "AGREE" ? "buy"
: r.panel_verdict === "OPPOSE" ? "sell" : "neutral";
const takes = (r.panel_takes || []).map(t =>
`<tr><td class="ind-k">${esc(t.persona)}</td>
<td>${esc(t.stance)}</td>
<td class="num">${t.confidence != null ? Math.round(t.confidence) : "—"}</td>
<td>${esc(t.note || "")}</td></tr>`).join("");
out += sec("AI panel (advisory)", `<div class="d-note" style="margin-bottom:8px">
<b class="sig-txt ${vcls}">${esc(r.panel_verdict)}</b> ·
conf ${Math.round(r.panel_confidence || 0)}%<br>
${esc(r.panel_rationale || "")}</div>
${takes ? `<table class="d-table"><tbody>${takes}</tbody></table>` : ""}`);
}
/* provenance */
out += sec("Scan detail", `<table class="d-table"><tbody>
<tr><td class="ind-k">Instrument</td><td colspan="5">${esc(r.display_name || r.symbol || "—")}</td></tr>
@@ -1023,8 +1005,7 @@ function renderDetail() {
$("detail-title").textContent = TF_LABEL[detailTf] || detailTf;
$("detail-price").textContent = r?.current_price ? fmt$(r.current_price) : "—";
$("detail-sub").innerHTML = `${snapshot?.symbol ? esc(snapshot.symbol) + " · " : ""}closes in
<span class="num" data-close="${entry.next_close_epoch || ""}"
data-interval="${TF_SECONDS[detailTf]}">--:--</span>`;
<span class="num" data-close="${entry.next_close_epoch || ""}">--:--</span>`;
$("detail-body").innerHTML = detailHTML(detailTf);
dlg.querySelectorAll("details[data-dkey]").forEach(d => {
@@ -1163,12 +1144,8 @@ function renderWire() {
};
const head = items.slice(0, 12).map(item).join("");
const rest = items.slice(12);
/* preserve the disclosure across poll re-renders — otherwise an expanded
list snaps shut within one poll cycle (renderDetail does the same via
data-dkey; there is only one of these, so a class lookup suffices) */
const moreOpen = $("wire").querySelector("details.more")?.open ?? false;
$("wire").innerHTML = head + (rest.length
? `<details class="more" style="padding-top:6px"${moreOpen ? " open" : ""}><summary>${rest.length} more headlines…</summary>
? `<details class="more" style="padding-top:6px"><summary>${rest.length} more headlines…</summary>
<div class="wire">${rest.map(item).join("")}</div></details>` : "");
}
@@ -1349,62 +1326,43 @@ wireNewsOpeners($("cal-body"), "data-ev-key", openCalDetail);
async function fetchJSON(name) {
/* no-cache = revalidate with the server every time (If-Modified-Since);
unchanged files come back 304 with no body, so aggressive polling
stays nearly free. Returns raw TEXT, not parsed JSON: a 304 is
transparent to fetch (the browser answers with the cached body and a
200), so poll() detects "unchanged" by comparing the payload itself
and skips the DOM rebuild entirely. */
stays nearly free. */
const res = await fetch(name, {cache: "no-cache"});
if (!res.ok) throw new Error(`${name}: HTTP ${res.status}`);
/* The Date header is stamped at RESPONSE time — snapshot.server_time_epoch
is stamped at publish time, minutes stale between candle closes. */
const dateHdr = res.headers.get("Date");
const text = await res.text();
return {text, serverNow: dateHdr ? Date.parse(dateHdr) / 1000 : null};
const data = await res.json();
return {data, serverNow: dateHdr ? Date.parse(dateHdr) / 1000 : null};
}
let refetchTimer = null;
let lastSnapText = null, lastNewsText = null; // change detection for the render-skip below
async function poll() {
try {
const [snap, nw] = await Promise.all([
fetchJSON("snapshot.json"),
fetchJSON("news.json").catch(() => null), // news outage ≠ page failure
]);
const snapChanged = snap.text !== lastSnapText;
const newsChanged = nw != null && nw.text !== lastNewsText;
if (snapChanged) { snapshot = JSON.parse(snap.text); lastSnapText = snap.text; }
if (newsChanged) { news = JSON.parse(nw.text); lastNewsText = nw.text; }
clockOffset = (snap.serverNow ?? snapshot.server_time_epoch) - Date.now() / 1000;
snapshot = snap.data;
news = (nw && nw.data) || news;
clockOffset = (snap.serverNow ?? snap.data.server_time_epoch) - Date.now() / 1000;
lastPollOk = new Date();
$("banner-fail").style.display = "none";
/* re-render only what new data can actually change — an unchanged
payload keeps the existing DOM (selection, disclosure state, scroll)
intact; countdowns and the freshness line tick from it regardless.
Trade-off: "Xm ago" ages in the wire/calendar only advance when the
payload changes. */
if (snapChanged) {
renderHero(); renderCards();
renderDetail(); // keep an open detail view in sync with the new scan
}
if (newsChanged) {
renderCalendar(); renderWire();
renderNewsDlg(); // same for an open news/calendar dialog
}
if (snapChanged || newsChanged) renderContext(); // blurb reads news, risk chip reads snapshot
renderHero(); renderContext(); renderCards(); renderCalendar(); renderWire();
renderDetail(); // keep an open detail view in sync with the new scan
renderNewsDlg(); // same for an open news/calendar dialog
tick(); // countdowns appear immediately, not after the first 1 s tick
/* one-shot refetch shortly after the next expected publish, so a
fast scan appears ~2 s after the publisher wakes instead of
waiting out the poll interval (the regular POLL_MS loop covers
slower scans) */
if (snapChanged) {
const nextScan = Math.min(...TF_ORDER.map(tf =>
snapshot.timeframes[tf]?.next_scan_epoch || Infinity));
if (isFinite(nextScan)) {
clearTimeout(refetchTimer);
const delay = (nextScan - nowServer() + 2) * 1000;
if (delay > 0) refetchTimer = setTimeout(poll, delay);
}
const nextScan = Math.min(...TF_ORDER.map(tf =>
snapshot.timeframes[tf]?.next_scan_epoch || Infinity));
if (isFinite(nextScan)) {
clearTimeout(refetchTimer);
const delay = (nextScan - nowServer() + 2) * 1000;
if (delay > 0) refetchTimer = setTimeout(poll, delay);
}
} catch (err) {
const last = lastPollOk ? ` Last update ${lastPollOk.toLocaleTimeString()}.` : "";
@@ -1414,46 +1372,28 @@ async function poll() {
}
}
/* 1 s countdown ticker (server-clock anchored). The wrap interval comes
from data-interval stamped at render time — reverse-looking the epoch up
in snapshot.timeframes mis-resolves shared boundaries (M5/M15/H1 all
close on the hour) to the first matching TF in TF_ORDER. */
/* 1 s countdown ticker (server-clock anchored) */
function tick() {
if (!snapshot) return;
const now = nowServer();
document.querySelectorAll("[data-close]").forEach(el => {
const t = parseFloat(el.dataset.close);
let t = parseFloat(el.dataset.close);
if (!t) return;
const tf = TF_ORDER.find(k => snapshot.timeframes[k]?.next_close_epoch === parseFloat(el.dataset.close));
let secs = t - now;
if (secs < 0) { // boundary passed while waiting for next publish
const interval = parseFloat(el.dataset.interval);
/* one-shot countdowns (calendar releases) carry no data-interval:
clamp at zero until the next poll re-renders them as "released …" */
secs = interval > 0 ? ((secs % interval) + interval) % interval : 0;
const interval = {"5min": 300, "15min": 900, "1h": 3600, "4h": 14400}[tf] || 300;
secs = ((secs % interval) + interval) % interval;
}
el.textContent = fmtCountdown(secs);
});
const m5 = snapshot.timeframes["5min"]?.next_close_epoch;
if (m5) {
let secs = m5 - now;
if (secs < 0) secs = ((secs % TF_SECONDS["5min"]) + TF_SECONDS["5min"]) % TF_SECONDS["5min"];
if (secs < 0) secs = ((secs % 300) + 300) % 300;
$("hero-cd").textContent = fmtCountdown(secs);
}
/* publish-time freshness, not poll health — a dead publisher behind a
healthy Apache would otherwise read as "updated just now" forever */
const pubEpoch = snapshot.server_time_epoch;
if (pubEpoch) {
const age = now - pubEpoch;
const el = $("hero-updated");
/* M5 publishes every 5 min while the market is open, so anything older
than 15 min means the pipeline is down, not between scans. The
weekend case is already covered by the market-closed banner. */
const stale = !snapshot.market_closed && age > 900;
el.textContent = (stale ? "STALE — published " : "published ") +
fmtAge(new Date(pubEpoch * 1000).toISOString());
el.style.color = stale ? "var(--amber)" : "";
el.style.fontWeight = stale ? "600" : "";
}
if (lastPollOk) $("hero-updated").textContent = `updated ${fmtAge(lastPollOk.toISOString())}`;
}
setInterval(tick, 1000);
-27
View File
@@ -71,7 +71,6 @@ from app_config import (
telegram_channel_id,
)
from providers import BaseDataProvider, make_data_provider
from agent_panel import attach_panel, panel_health
from llm_client import get_llm_client
from news_engine import FundamentalContext, NewsEngine
from pair_signals import PAIR_PRODUCERS
@@ -408,17 +407,6 @@ def format_signal_message(report: SignalReport) -> str:
extra.append(f"⚠️ {_escape_md(w)}")
if report.news_summary:
extra.append(f"📰 {_escape_md(report.news_summary)}")
if report.panel_verdict:
mark = {"AGREE": "✅", "OPPOSE": "⛔", "NEUTRAL": "➖"}.get(
report.panel_verdict, "")
extra.append(
f"🤖 AI Panel {mark} *{_escape_md(report.panel_verdict)}* "
f"({report.panel_confidence:.0f}%) — "
f"{_escape_md(report.panel_rationale)}")
if report.panel_takes:
votes = " · ".join(
f"{t['persona']}: {t['stance']}" for t in report.panel_takes)
extra.append(f" _{_escape_md(votes)}_")
if extra:
lines += [""] + extra
@@ -732,13 +720,6 @@ async def _scan_timeframe(
async with _alert_lock(key):
if not _should_alert(key, report):
continue
# Panel AFTER the quality gate + dedup decision so no LLM
# spend on suppressed alerts. Runs on the scan executor to
# keep the event loop free; this key's alert lock is held
# for the panel's wall time (≤~25 s), which only delays a
# same-key duplicate that _should_alert would drop anyway.
await asyncio.get_running_loop().run_in_executor(
_executor, attach_panel, report)
detail = format_signal_message(report)
delivered = False
for chat_id in _signal_targets():
@@ -874,10 +855,6 @@ async def _handle_breach(bot, pair_symbol: str, price: float, level_name: str) -
if not _should_alert(key, report):
return
# Same post-gate panel attach as the auto-scan path (see
# _scan_timeframe for the spend/lock rationale).
await asyncio.get_running_loop().run_in_executor(
_executor, attach_panel, report)
header = (
f"⚡ *LEVEL BREAK* — {_escape_md(display)} crossed "
f"*{_escape_md(label)}* @ `{_fmt_price(price)}`\n\n"
@@ -1216,10 +1193,6 @@ def _news_health_lines() -> List[str]:
else:
lines.append(f" ✅ DeepSeek: {used}/{cap} calls today,"
f" {scored} headlines scored")
if get_app_config().agent_panel_settings.get("enabled", False):
ph = panel_health()
lines.append(f" 🤖 AI panel: {ph['used_today']}/{ph['cap']}"
" panels today")
return lines
-382
View File
@@ -6943,388 +6943,6 @@ def _():
assert "must be > 0" in str(e)
print("\n═══ Module 23: LLM agent panel ═══")
@test("app_config: agent_panel block parses; absent block → {}")
def _():
app = load_app_config(_write_json({"agent_panel": {
"enabled": True, "telegram": True, "dashboard": False,
"min_quality": "A", "max_panels_per_day": 10,
"reuse_ttl_seconds": 600}}))
assert app.agent_panel_settings["enabled"] is True
assert app.agent_panel_settings["min_quality"] == "A"
assert app.agent_panel_settings["max_panels_per_day"] == 10
assert load_app_config(_write_json({})).agent_panel_settings == {}
assert AppConfig().agent_panel_settings == {}
@test("app_config: agent_panel rejects unknown keys and bad types")
def _():
for payload in (
{"agent_panel": {"enabld": True}}, # unknown key
{"agent_panel": {"enabled": "yes"}}, # not a bool
{"agent_panel": {"telegram": 1}}, # not a bool
{"agent_panel": {"dashboard": "no"}}, # not a bool
{"agent_panel": {"min_quality": "Z"}}, # invalid grade
{"agent_panel": {"max_panels_per_day": 0}}, # not positive
{"agent_panel": {"max_panels_per_day": True}}, # bool is not an int
{"agent_panel": {"reuse_ttl_seconds": -5}}, # negative
{"agent_panel": "oops"}, # not an object
):
try:
load_app_config(_write_json(payload))
assert False, f"should have raised for {payload}"
except AppConfigError:
pass
import agent_panel as ap
from signal_aggregator import SignalReport, TradeSetup
class _PanelStub:
"""llm_client-singleton drop-in that routes replies by system prompt."""
available = True
def __init__(self, replies):
self.replies = replies # system prompt -> reply dict | None
self.calls = [] # (system_prompt, payload) in call order
def chat_json(self, system_prompt, user_payload):
self.calls.append((system_prompt, user_payload))
reply = self.replies.get(system_prompt)
return reply(user_payload) if callable(reply) else reply
_TAKE_OK = {"stance": "support", "confidence": 70, "note": "fine"}
_SYNTH_OK = {"verdict": "AGREE", "confidence": 75, "rationale": "panel backs it"}
_ALL_OK = {ap._MACRO_SYSTEM: _TAKE_OK, ap._TECH_SYSTEM: _TAKE_OK,
ap._RISK_SYSTEM: _TAKE_OK, ap._CONTRARIAN_SYSTEM: _TAKE_OK,
ap._SYNTH_SYSTEM: _SYNTH_OK}
_PANEL_ON = {"enabled": True, "telegram": True, "dashboard": False,
"min_quality": "B", "max_panels_per_day": 20,
"reuse_ttl_seconds": 3600}
def _panel_report(quality="A", signal=SignalStrength.BUY):
"""Real report via the pipeline, then forced directional/quality."""
config = PlatformConfig()
config.primary_timeframe = Timeframe.H1
config.confirmation_timeframes = [Timeframe.M15, Timeframe.H4]
agg = SignalAggregator(config)
dfs = {Timeframe.H1: make_ohlcv(300, seed=42),
Timeframe.M15: make_ohlcv(300, seed=43),
Timeframe.H4: make_ohlcv(300, seed=44)}
report = agg.analyze_from_dataframes("XAU/USD", dfs, use_news=False)
report.final_signal = signal
report.signal_quality = quality
return report
@test("agent_panel: payload builders compact, filtered, JSON-safe")
def _():
report = _panel_report()
now = _dt_m19.datetime.now(_dt_m19.timezone.utc)
ctx = FundamentalContext(
news_headlines=[NewsSentiment(
headline=f"h{i}", source="rss", timestamp=now,
currencies_mentioned=["USD"], sentiment_score=0.1, keywords=[])
for i in range(15)],
events_today=[EconomicEvent(
time=now, currency="USD", impact="high", event=f"e{i}",
forecast="1.0", actual="2.0") for i in range(12)],
)
macro = ap._macro_payload(report, ctx)
assert len(macro["headlines"]) == 10, "headline cap"
assert len(macro["events"]) == 8, "event cap"
assert macro["events"][0]["released"] is True
assert ap._macro_payload(report, None)["headlines"] == []
tech = ap._technical_payload(report)
assert tech["indicators"], "real analysis must yield indicators"
hidden = [n for n in tech["indicators"]
if n.startswith("_")
or report.primary_analysis.indicators[n].weight == 0]
assert not hidden, f"private/weight-0 leaked: {hidden}"
assert tech["brief"]["direction"] == "BUY"
assert tech["brief"]["timeframe"] == report.primary_analysis.timeframe
assert "setup" not in ap._risk_payload(report), "no setup when None"
report.trade_setup = TradeSetup(
pair="XAU/USD", display_name="XAU/USD", direction="SELL",
entry_price=2000.0, stop_loss=2010.0, take_profit_1=1985.0,
take_profit_2=1970.0, risk_reward=1.5, risk_pips=100.0,
reward_pips=150.0, pip_value=0.1)
risk = ap._risk_payload(report)
assert risk["setup"]["risk_reward"] == 1.5
assert risk["brief"]["direction"] == "SELL", "setup direction wins"
contra = ap._contrarian_payload(report)
assert len(contra["bullish_reasons"]) <= 3
for payload in (macro, tech, risk, contra):
json.dumps(payload, sort_keys=True, allow_nan=False)
@test("agent_panel: persona reply validation clamps and rejects")
def _():
v = ap._validate_take
assert v("macro", {"stance": "moon", "confidence": 50, "note": "x"}) is None
assert v("macro", {"stance": "support", "confidence": "hot"}) is None
assert v("macro", {"stance": "support"}) is None, "missing confidence"
assert v("macro", None) is None
assert v("macro", "support") is None
take = v("risk", {"stance": "oppose", "confidence": 250, "note": "n" * 300})
assert take.confidence == 100.0 and len(take.note) == 160
assert v("risk", {"stance": "neutral", "confidence": -3, "note": ""}).confidence == 0.0
@test("agent_panel: <2 valid takes → None without spending synthesizer")
def _():
try:
stub = _PanelStub({ap._MACRO_SYSTEM: _TAKE_OK}) # others → None
lc._client = stub
assert ap.run_panel(_panel_report()) is None
assert ap._SYNTH_SYSTEM not in [s for s, _ in stub.calls], \
"synthesizer request must not be spent on a failed panel"
assert len(stub.calls) == 4
finally:
lc._client = None
ap.reset_panel_state()
@test("agent_panel: synthesizer failure or garbage verdict → None")
def _():
try:
for synth in (None, {"verdict": "MAYBE", "confidence": 50},
{"verdict": "AGREE", "confidence": "hot"}):
lc._client = _PanelStub({**_ALL_OK, ap._SYNTH_SYSTEM: synth})
assert ap.run_panel(_panel_report()) is None, f"synth={synth}"
finally:
lc._client = None
ap.reset_panel_state()
@test("agent_panel: full run yields verdict + 4 ordered takes")
def _():
try:
stub = _PanelStub(dict(_ALL_OK))
stub.replies[ap._SYNTH_SYSTEM] = {
"verdict": "OPPOSE", "confidence": 72.4,
"rationale": "r" * 300}
lc._client = stub
v = ap.run_panel(_panel_report())
assert v.verdict == "OPPOSE" and v.confidence == 72.4
assert len(v.rationale) == 240, "rationale truncated"
assert [t.persona for t in v.takes] == \
["macro", "technical", "risk", "contrarian"]
synth_payload = dict(stub.calls)[ap._SYNTH_SYSTEM]
assert synth_payload["direction"] == "BUY"
assert len(synth_payload["takes"]) == 4
finally:
lc._client = None
ap.reset_panel_state()
@test("agent_panel: NEUTRAL signal → no panel even forced")
def _():
try:
stub = _PanelStub(dict(_ALL_OK))
lc._client = stub
report = _panel_report(signal=SignalStrength.NEUTRAL)
assert ap.run_panel(report) is None
assert ap.attach_panel(report, _PANEL_ON, force=True,
context=FundamentalContext()) is False
assert stub.calls == [], "NEUTRAL must spend nothing"
finally:
lc._client = None
ap.reset_panel_state()
@test("agent_panel: exact + reuse-TTL cache re-attach with zero spend")
def _():
try:
stub = _PanelStub(dict(_ALL_OK))
lc._client = stub
report = _panel_report()
assert ap.attach_panel(report, _PANEL_ON,
context=FundamentalContext()) is True
assert report.panel_verdict == "AGREE"
assert len(stub.calls) == 5
# same bar again → exact-key hit
report2 = _panel_report()
assert ap.attach_panel(report2, _PANEL_ON,
context=FundamentalContext()) is True
assert report2.panel_verdict == "AGREE" and len(stub.calls) == 5
# newer bar, same symbol/tf/direction → reuse-TTL hit
report3 = _panel_report()
report3.timestamp = "2099-01-01 01:00:00+00:00"
assert ap.attach_panel(report3, _PANEL_ON,
context=FundamentalContext()) is True
assert len(stub.calls) == 5, "persisting signal must not re-spend"
# TTL 0 → the reuse layer is off; a new bar re-runs the panel
report4 = _panel_report()
report4.timestamp = "2099-01-01 02:00:00+00:00"
assert ap.attach_panel(report4, {**_PANEL_ON, "reuse_ttl_seconds": 0},
context=FundamentalContext()) is True
assert len(stub.calls) == 10
finally:
lc._client = None
ap.reset_panel_state()
@test("agent_panel: daily budget exhausts and re-arms on UTC roll")
def _():
try:
stub = _PanelStub(dict(_ALL_OK))
lc._client = stub
settings = {**_PANEL_ON, "max_panels_per_day": 1,
"reuse_ttl_seconds": 0}
assert ap.attach_panel(_panel_report(), settings,
context=FundamentalContext()) is True
assert len(stub.calls) == 5
report = _panel_report()
report.timestamp = "2099-01-01 01:00:00+00:00"
assert ap.attach_panel(report, settings,
context=FundamentalContext()) is False
assert len(stub.calls) == 5, "budget-blocked attempt must spend nothing"
ap._panel_day = (_dt_m19.datetime.now(_dt_m19.timezone.utc).date()
- _dt_m19.timedelta(days=1))
assert ap.attach_panel(report, settings,
context=FundamentalContext()) is True
assert len(stub.calls) == 10
assert ap.panel_health()["used_today"] == 1
finally:
lc._client = None
ap.reset_panel_state()
@test("agent_panel: enabled/surface/min_quality gates; force bypasses")
def _():
try:
stub = _PanelStub(dict(_ALL_OK))
lc._client = stub
report = _panel_report()
ctx = FundamentalContext()
assert ap.attach_panel(report, {}, context=ctx) is False, \
"absent config block must keep the feature dark"
assert ap.attach_panel(report, {**_PANEL_ON, "enabled": False},
context=ctx) is False
assert ap.attach_panel(report, {**_PANEL_ON, "telegram": False},
context=ctx) is False
assert ap.attach_panel(report, _PANEL_ON, surface="dashboard",
context=ctx) is False, "dashboard defaults off"
assert ap.attach_panel(_panel_report(quality="C"), _PANEL_ON,
context=ctx) is False, "C < min_quality B"
assert ap.attach_panel(_panel_report(quality="F"), _PANEL_ON,
context=ctx) is False
assert stub.calls == [], "every gate must block before any spend"
assert ap.attach_panel(report, {**_PANEL_ON, "dashboard": True},
surface="dashboard", context=ctx) is True
ap.reset_panel_state()
assert ap.attach_panel(_panel_report(quality="C"),
{**_PANEL_ON, "enabled": False},
force=True, context=ctx) is True, \
"force bypasses enabled/surface/quality"
finally:
lc._client = None
ap.reset_panel_state()
@test("panel verdict flows to Telegram message; absent verdict silent")
def _():
import telegram_bot as tb
report = _panel_report()
assert "AI Panel" not in tb.format_signal_message(report)
report.panel_verdict = "OPPOSE"
report.panel_confidence = 64.0
report.panel_rationale = "counter-trend against aligned H4"
report.panel_takes = [
{"persona": "macro", "stance": "support", "confidence": 60,
"note": "n"},
{"persona": "technical", "stance": "oppose", "confidence": 80,
"note": "n"},
]
msg = tb.format_signal_message(report)
assert "AI Panel" in msg and "OPPOSE" in msg and "(64%)" in msg
assert "counter-trend against aligned H4" in msg
assert "macro: support" in msg and "technical: oppose" in msg
@test("backtest parity: analyze_from_dataframes never populates panel")
def _():
try:
stub = _PanelStub(dict(_ALL_OK))
lc._client = stub
config = PlatformConfig()
config.primary_timeframe = Timeframe.H1
config.confirmation_timeframes = [Timeframe.M15, Timeframe.H4]
agg = SignalAggregator(config)
dfs = {Timeframe.H1: make_ohlcv(300, seed=42),
Timeframe.M15: make_ohlcv(300, seed=43),
Timeframe.H4: make_ohlcv(300, seed=44)}
for use_news in (True, False):
report = agg.analyze_from_dataframes(
"XAU/USD", dfs, context=FundamentalContext(),
use_news=use_news)
assert report.panel_verdict == ""
assert report.panel_confidence == 0.0
assert report.panel_rationale == ""
assert report.panel_takes == []
assert stub.calls == [], \
"the replayable core must never invoke the panel"
finally:
lc._client = None
ap.reset_panel_state()
@test("report_to_dict carries panel keys → dashboard snapshot propagation")
def _():
from main import report_to_dict as _rtd
from dashboard_publisher import report_to_webapp_dict as _rtw
report = _panel_report()
d = _rtd(report)
assert d["panel_verdict"] == "" and d["panel_takes"] == []
report.panel_verdict = "AGREE"
report.panel_confidence = 77.5
report.panel_rationale = "solid confluence"
report.panel_takes = [{"persona": "risk", "stance": "support",
"confidence": 66.0, "note": "rr fine"}]
wd = _rtw(report)
assert wd["panel_verdict"] == "AGREE"
assert wd["panel_confidence"] == 77.5
assert wd["panel_takes"][0]["persona"] == "risk"
json.dumps(wd, allow_nan=False)
@test("agent_panel: attach never raises (LLM explosion, garbage settings)")
def _():
class _Bomb:
available = True
def chat_json(self, system_prompt, user_payload):
raise RuntimeError("boom")
try:
lc._client = _Bomb()
report = _panel_report()
assert ap.attach_panel(report, _PANEL_ON,
context=FundamentalContext()) is False
assert report.panel_verdict == "" and report.panel_takes == []
logging.disable(logging.CRITICAL) # silence the expected traceback
try:
assert ap.attach_panel(report, 42,
context=FundamentalContext()) is False
finally:
logging.disable(logging.NOTSET)
finally:
lc._client = None
ap.reset_panel_state()
# ══════════════════════════════════════════════════════════════════════
# SUMMARY
# ══════════════════════════════════════════════════════════════════════