146 Commits

Author SHA1 Message Date
Harwinder Sekhon 887aaed6f2 dashboard: skip DOM rebuild on unchanged poll, publish-time staleness, fix boundary countdowns
poll() now compares raw payload text and re-renders only what a changed
file can affect, preserving selection, disclosure, and scroll state. Hero
freshness reflects publish time (STALE warning past 15 min while market
open), and countdowns resolve their wrap interval from data-interval
instead of reverse-looking shared TF boundaries in snapshot.timeframes.
2026-08-09 18:15:31 -04:00
Harwinder Sekhon 7e8ed754ff news: reduce headline max age from 24h to 12h, make configurable
- Added headline_max_age_seconds to NewsConfig (default 43200 = 12h)
- Replaced hardcoded 86400 in _fetch_news_sentiment_raw with config ref
- All downstream consumers (keyword scoring, LLM scoring, LLM summary,
  agent panel macro/contrarian) auto-filter through the same path
2026-08-09 16:49:48 -04:00
Harwinder Sekhon 9769a0c84d agent_panel: port ai-hedge-fund prompt rubric — stance rules, anchored confidence, no-invention
Prompt-only change (static constants; plain-language output contract
byte-identical). Ported from virattt/ai-hedge-fund's persona rubric:

- _TAKE_CONTRACT hard rules: reason only from the JSON given, never
  invent numbers/events/levels; thin or missing evidence -> neutral
  with confidence below 30 (deferring to role-specific rules).
- Anchored confidence bands in the contract and synthesizer, replacing
  the unanchored "how sure you are".
- Per-stance support/oppose/neutral criteria for macro, technical, and
  risk (risk's missing-setup->oppose folded into its oppose bullet).
- One plain-language support example beside each oppose example for
  macro/technical/risk, fixing the all-oppose few-shot bias.
  Contrarian's designed skeptic tilt untouched.

News-engine prompts untouched (pinned by 78/78 + 18/18 validations).
374 tests green; real-DeepSeek smoke: macro correctly answered
neutral/20 on empty news context, confidences spread 20-60 across the
bands, all notes <=25 words, no jargon.

Claude-Session: https://claude.ai/code/session_01TyZx9iRJwfbPTPnCBqcwTR
2026-08-08 10:48:10 -04:00
Harwinder Sekhon 964239e67d docs: multi-provider LLM layer plan (DeepSeek news / Anthropic panel)
Reference plan only - no code changes. Describes the BaseLLMClient
refactor, AnthropicClient on the official SDK, per-consumer routing via
a config.json "llm" block, tests, commit sequence, and effort estimate
for the day we route the agent panel to Claude while news stays on
DeepSeek.

Claude-Session: https://claude.ai/code/session_01SisdfkxnXNL9iZsXxN3T15
2026-08-07 23:02:08 -04:00
Harwinder Sekhon 5e3758bd53 agent_panel: plain-language output — simple, short, no jargon/acronyms
Owner request: panel notes and rationale must read like plain everyday
English for a non-trader. Style rule added to the shared take contract
and synthesizer prompt, all few-shot examples rewritten in the target
voice (examples carry the style), and the Telegram votes line now spells
out full stances ("macro: support") instead of "macro:sup".

Before: "BUY against Bollinger SELL and price above upper band; 4h
neutral, mixed MTF"  →  After: "The price is above its usual range, and
the bigger picture is not clearly rising, so buying now is risky."

Claude-Session: https://claude.ai/code/session_01SisdfkxnXNL9iZsXxN3T15
2026-08-07 22:50:10 -04:00
Harwinder Sekhon c146fffb5b docs: agent panel module + config block
Claude-Session: https://claude.ai/code/session_01SisdfkxnXNL9iZsXxN3T15
2026-08-07 22:35:15 -04:00
Harwinder Sekhon 38c81251e8 cli+dashboard: --panel flag, panel dict keys, TF-dialog panel section
main.py: --panel runs the agent panel on demand (force bypasses
enabled/surface/quality toggles; daily budget + LLM cap still apply),
prints the verdict + persona takes, and report_to_dict carries the four
panel keys — which auto-propagates them into the dashboard's
snapshot.json via report_to_webapp_dict. dashboard_publisher attaches
in scan_tf gated by agent_panel.dashboard (default off); dashboard.html
renders an "AI panel (advisory)" section in the TF detail dialog.

Claude-Session: https://claude.ai/code/session_01SisdfkxnXNL9iZsXxN3T15
2026-08-07 22:34:30 -04:00
Harwinder Sekhon a7c2bfc7da telegram: attach AI panel to alert-quality sends (advisory block)
Auto-scan and level-breach alert paths attach the panel AFTER the
quality gate and dedup decision (no LLM spend on suppressed alerts),
via run_in_executor so the event loop stays free. Alert message gains a
two-line 🤖 AI Panel block (verdict + confidence + rationale, persona
votes); /status shows the daily panel budget when the feature is on.

Claude-Session: https://claude.ai/code/session_01SisdfkxnXNL9iZsXxN3T15
2026-08-07 22:32:57 -04:00
Harwinder Sekhon 7be2231eb3 agent_panel: persona/synthesizer advisory consensus module
ai-hedge-fund-style panel: 4 DeepSeek personas (macro, technical, risk,
contrarian) review a finished SignalReport concurrently on a
panel-private thread pool, then a synthesizer chair issues
AGREE/NEUTRAL/OPPOSE + confidence + rationale. Advisory/display-only by
construction: consumers attach post-hoc via attach_panel; the fields
live on SignalReport with empty defaults and analyze_from_dataframes
never touches them (backtest parity, locked by test).

Spend control on top of the shared 500/day client cap: max_panels_per_day
UTC budget (default 20), exact per-bar result cache, and a reuse TTL
keyed (symbol, tf, direction) so a persisting signal re-serves its
verdict across bars instead of burning 5 requests per bar. <2 valid
takes skips the synthesizer; any failure degrades to no panel.

Claude-Session: https://claude.ai/code/session_01SisdfkxnXNL9iZsXxN3T15
2026-08-07 22:31:50 -04:00
Harwinder Sekhon 0b604f1ad0 config: agent_panel block (enabled/surfaces/min_quality/daily budget)
New optional config.json block for the upcoming LLM agent-panel feature:
enabled, telegram, dashboard (per-surface toggles), min_quality,
max_panels_per_day, reuse_ttl_seconds. Absent block → {} → feature dark.
Validation mirrors the deepseek block. Module 23 opened with parse +
reject tests.

Claude-Session: https://claude.ai/code/session_01SisdfkxnXNL9iZsXxN3T15
2026-08-07 22:26:43 -04:00
Harwinder Sekhon 250078c987 docs: architecture diagram (PNG + matplotlib generator)
End-to-end map: external services → providers/news layer → signal core
(engine + factory + aggregator) → SignalReport → four delivery surfaces,
plus the backtester replaying the same core. Regenerate with
venv/bin/python docs/architecture_diagram.py after architecture changes.

Claude-Session: https://claude.ai/code/session_01Aq1jZ6jBsBdEC2xT3u46SA
2026-08-07 09:21:52 -04:00
Harwinder Sekhon d1d002a4f7 dashboard: clickable news — detail dialogs + on-demand AI summary endpoint
- news_engine: RSS parser captures link/description/publisher (three new
  optional NewsSentiment fields); _clean_html_text/_safe_link helpers
- news.json: headlines carry id = sha1(headline) + enrichment fields
- static/news_summary.php (NEW): on-demand DeepSeek gold-impact summary,
  whitelist-gated against news.json ids, cache-forever in ai/, fail-closed
  file-based daily cap (150/day), key at /etc/gold-dashboard/deepseek.key;
  installed to the docroot by install_static
- dashboard.html: newswire + calendar detail dialogs (score tiles, source
  description, article link, AI summary with loading/error states);
  delegated on stable containers, dialog keyed by id across polls
- tests: +6 (357 total); docs: README-dashboard §6 one-time setup
  (key file, apache-owned ai/, SELinux httpd_can_network_connect), CLAUDE.md

Claude-Session: https://claude.ai/code/session_01Aq1jZ6jBsBdEC2xT3u46SA
2026-08-07 09:08:49 -04:00
root edefd90aaa oanda: retry once on transient 401/5xx/network faults in candle fetch
The practice API rejects ~15-20 requests/day with a spurious 401
"Insufficient authorization" while the same token succeeds on adjacent
requests; each one silently cost a full scan cycle on that timeframe.
One retry after RETRY_DELAY_S (2 s) absorbs these plus transient
5xx/network faults. Real credential failures fail both attempts and
raise as before; other 4xx (bad request, 429) are never retried.

Tests: transient-401 recovery, persistent-401 raise, 400 not retried.

Claude-Session: https://claude.ai/code/session_01Tujtmmhxj9mxKjPFSoVjmP
2026-08-06 23:06:29 -04:00
Harwinder Sekhon b21c75a1f8 dashboard: Gold Desk v2 UI — light/dark theme, TF detail dialog, wire score
Owner-supplied page design replacing the dark-only v1: Bootstrap 5
colour tokens with a luma-inverted dark mode (toggle + system-pref +
localStorage), compact clickable TF cards opening a full-detail modal
(all-nine indicator table with value/conf/weight, price ladder of key
levels vs spot, per-confirmation-TF drilldowns, provenance), newswire
aggregate sentiment box, Inter/JetBrains Mono webfonts.

Re-applied the c70f58a sync fixes the source copy predated: 5 s
conditional-GET polling and the +2 s boundary refetch sniper. Data
logic otherwise unchanged; all fields verified against
report_to_webapp_dict output.

Claude-Session: https://claude.ai/code/session_017mbXs9HRawZhD1uZ3ze7zQ
2026-08-06 22:44:48 -04:00
Harwinder Sekhon a2f951f5c9 dashboard: cut page sync latency to ~scan-time + ≤5 s
Three display-side fixes, no signal-logic changes:
- run_cycle publishes snapshot.json BEFORE the news refresh — news I/O
  (FF + 4 RSS + possible DeepSeek calls) took 20-30 s on TTL-expired
  cycles and kept fresh signals off the page (publish_all split into
  publish_snapshot/publish_news, kept for boot)
- page polls every 5 s (was 20 s) using conditional GETs: dropped the
  ?t= cache-buster + no-store so Apache answers unchanged files with
  bodyless 304s
- boundary refetch sniper re-armed: fires 2 s after next_scan_epoch
  regardless of distance (the old delay<POLL_MS guard would have been
  dead code at 5 s polling)

New Module 20 test locks the snapshot-before-news ordering. 352 tests.

Claude-Session: https://claude.ai/code/session_017mbXs9HRawZhD1uZ3ze7zQ
2026-08-06 22:35:13 -04:00
Harwinder Sekhon 858c12280a news: DeepSeek validation, vs_forecast fix, /status source health
Validation (llm_news_eval.py harness + 78-headline golden CSV + verdict
doc): headline scoring 78/78 vs keyword baseline 24/78 — validated;
event gold-direction calls misread close K/M numeric comparisons ~11%
of the time (both claims cases in sample, JOLTS 8.79M vs 8.84M).

Fix: event payload now carries code-computed "vs_forecast" (from
EconomicEvent.surprise) and the prompt gains an explicit inverted-
metrics rule + worked claims example — iteration 1 (field alone) fixed
the arithmetic but exposed a masked inversion error. 18/18 on live
re-verification (4 misreads fixed, 11 regression cases clean, 3 fresh
both-side probes).

/status: new "News & AI" section — per-source health for the calendar
chain (which of FF/MyFXBook served), each RSS feed, and DeepSeek
(daily count, last error kind), written passively by the existing
fetch paths under _cache_lock. Failures WARN once on transition
(repeats DEBUG, recovery INFO) so the rotating log carries outage
stories without 10-min spam.

test_all.py: +5 Module 19 tests (351 total); also carries the Module 22
fixed-RR-TP tests from the prior research session (shared file).
Replay artifacts dir gitignored like commit_ladder.

Claude-Session: https://claude.ai/code/session_017mbXs9HRawZhD1uZ3ze7zQ
2026-08-06 22:21:02 -04:00
Harwinder Sekhon 525a1cd1c1 research: --fixed-rr-tp backtest flag + 2026-08-05 verdict (NOT shipped)
Hard TP at N×risk beats the trail on all 2024+ windows but the grid is
monotone to 3R (trend artifact) and M15 rr20 goes negative in 2021+2023
with doubled DD — trailing exits stay. Flag is research-only; its
Module 22 tests land with the next commit (shared test_all.py).

Claude-Session: https://claude.ai/code/session_017mbXs9HRawZhD1uZ3ze7zQ
2026-08-06 22:20:43 -04:00
Harwinder Sekhon b693f757f8 Remove OANDA_MIGRATION_ROADMAP.md (reference-only doc, superseded)
Deletion was already staged in the working tree by the owner; the
roadmap was approved 2026-07-24 as reference only and its plan file
lives outside the repo. Content remains in git history.

Claude-Session: https://claude.ai/code/session_01UnZkzeFcTu9agFg8BDh2tn
2026-08-04 20:26:54 -04:00
Harwinder Sekhon 7dbcbc6c82 audit: NY-anchored weekend calendar (all 3 copies) + FB2_EMA_LO crash-proofing
Owner decisions from the 2026-08-04 audit report:
- spread-in-exit-triggers: DECLINED (books stay comparable; ledger in
  CLAUDE.md records the measured ~4-6% optimism).
- weekend clock: FIXED. The market week is Fri 17:00 -> Sun 18:00 New
  York time, not Fri/Sun 22:00 UTC — during US DST the Oanda feed stops
  at 21:00 UTC (measured: 159/162 summer weekends). All three weekend
  implementations (providers/base, signal_aggregator, gpu_backtest
  calendar filter) now share the NY-anchored rule; a new test sweeps
  them hourly across both DST transitions and asserts agreement. No NTP
  needed — the host clock was fine, the constant was wrong.
  Ends the weekly false-"STALE DATA / feed lagging" hours (summer Fri
  21-22 UTC, winter Sun 22-23 UTC).
- FB2_EMA_LO (delegated): kept as a research knob but crash-proofed —
  an unparseable value degrades to the tuned default 0.5 with a warning
  instead of killing every scan. Documented in CLAUDE.md.

Book gates: M15 12mo pinned book byte-identical (+78.7111R, 397
trades). M5 12mo drops exactly ONE trade (556 -> 555, +97.56 ->
+98.64R): a phantom whose signal fired on the final summer-Friday bar
(decision 21:00 UTC = NY 17:00 market close) and was "filled" at
Sunday's reopen 49 hours later, -1.08R — an entry live trading can
never take. The backtest no longer contains it; live == backtest
parity improved. M15 unaffected because hour 21 is in its blocked set.

Suite: 341/341 green.

Claude-Session: https://claude.ai/code/session_01UnZkzeFcTu9agFg8BDh2tn
2026-08-04 20:21:34 -04:00
Harwinder Sekhon 5b3f47f3e1 audit: scheduler cleanup, WS-log secret hygiene, simulator test coverage, CLAUDE.md sync (Batch G)
2026-08-04 Fable 5 QA audit, final code batch.

- Boundary math: removed the provably-dead `remaining <= buffer` guard
  from all three scheduler copies (518,400-instant frozen-clock sweep:
  never fired) and documented the exact-boundary-instant semantics.
- websockets/websockets.client loggers pinned to INFO so a DEBUG run
  can never write the TD key (connect URI) or Tiingo auth frame into
  the 30-day rotating log. (.env also chmod 600, untracked.)
- Cooldown pass factored into _apply_cooldown (pure, testable) — the
  only sequential replay rule finally has direct tests.
- New simulator behavior tests (previously ZERO coverage of the trade
  resolver): same-bar SL+TP tie → SL-first, gap-through-SL fills at
  the open, TIMEOUT pricing + round-trip spread charging, trailing
  ratchet arms at bar close and fills next-bar-only, cooldown ordering.
- CLAUDE.md synced to verified reality: 338 tests/21 modules; three
  weekend-implementation copies caveat; throttle claim corrected
  (single-flight, TD-only throttle); daemon trail/cooldown parity
  caveats (once-per-OBSERVED-bar, wall-clock cooldown) + OPEN_ONLY and
  reconcile notes; CP-path gate semantics documented as validated
  behavior; new "known open items" section for the owner-decision
  findings (weekend DST constant, spread-in-trigger rule-6 item, H4
  seeding divergence, FB2_EMA_LO).

Gates: suite 338/338 green; pinned M15 book and M5 HEAD baseline both
reproduced identically after the cooldown refactor.

Claude-Session: https://claude.ai/code/session_01UnZkzeFcTu9agFg8BDh2tn
2026-08-04 20:08:37 -04:00
Harwinder Sekhon 99eb268cf3 audit: alert delivery, provider robustness, backtester NaN/cache guards (Batches C+E+F)
2026-08-04 Fable 5 QA audit. Three batches sharing one test file, all
verified by runnable repros before fixing, all gated below.

Batch C — telegram alert delivery (verified by h4 harness):
- Failed breach rescan shrinks the cooldown to 30 s instead of
  silencing the symbol for 5 min; breach during cooldown is PARKED and
  fired on the first tick after expiry (was dropped forever).
- Level cache keyed symbol→TF (M5 scans no longer wipe H1/H4 swing
  levels within minutes); breach labels TF-qualified.
- Markdown alert sends get a plain-text fallback (mirroring _safe_edit);
  fingerprint commits only when a message actually landed.
- Per-key asyncio.Lock closes the dedup TOCTOU between auto-scan and
  breach sends (forced-interleave repro produced duplicate alerts).
- _start_scheduler respawns dead per-TF loops (was all-or-nothing).
- Clean WS close now backs off (was a zero-delay reconnect hot loop);
  backoff resets only after a connection that received ticks; TD
  iter_ticks fires on_connect on the first RECEIVED frame (invalid key
  can no longer pin the backoff at minimum).
- Scan-failure logs carry the real traceback (was "NoneType: None").

Batch E — providers (verified by h5 harness):
- fetch_range: 3 consecutive chunk ERRORS raise instead of returning
  silently truncated history (verified 731-day silent gap repro; also
  covers 429 storms); pagination no-progress guard; head-truncation
  ERROR log.
- Single-flight per (symbol,TF): 4 concurrent scans = 1 API call (was
  4); cache records requested depth so a longer-lookback call refetches.
- Multi-bar forming/future tail fully dropped (was iloc[:-1] once).
- Oanda: malformed candles skipped, never 0.0-defaulted; cleaning drops
  non-positive-price rows BEFORE OHLC repair; inf returns neutralized so
  one corrupt bar can no longer disarm the whole 10-sigma filter.

Batch F — backtester internals (verified by h6 harness):
- NaN bar in a trailing window exits at the last valid bar (was NaN R
  poisoning every metric); static-path TIMEOUT NaN guard;
  compute_metrics excludes non-finite R loudly. Zero real-data exposure
  (0 NaN rows in all 10 Oanda parquets) — defensive.
- fetch_history ERROR-logs pre-2026-07-08 cache files (cleaned-bars era)
  when loaded for raw-path resolution.

Gates: suite 333/333 green; pinned M15 12mo book reproduced identically
(397 trades, +78.7111R) after E and after F; M5 HEAD-config baseline
reproduced identically (556 trades, +97.5604R) after F.

Claude-Session: https://claude.ai/code/session_01UnZkzeFcTu9agFg8BDh2tn
2026-08-04 20:02:57 -04:00
Harwinder Sekhon 18e4fd0403 audit: gate-chain verdicts — lock shipped M5 entry-path state, document hour-delta isolation
2026-08-04 audit, Batch D. Verification outcomes (no behavior changes —
comment + tests only):

- CP gate-bypass (quality forced to B after the gate chain, exhaustion/
  M5-score gates not re-checked) is VALIDATED behavior, not a bug: the
  pinned shipped-config M5 12mo book carries 226/586 XAU_CP trades
  (+38.9R), 104 with EXHAUST warnings (+15.3R), 175 below the M5
  min-score gate (+28.0R). Removing the "holes" removes validated
  profit. New test pins the shipped state (PB off, CP BUY-only on M5,
  ema_partial_neutral M5-only) so any flip is a conscious revalidated
  decision.
- E18 (xau_strategy vs backtest blocked-hours "drift") REFUTED as a
  defect: the divergence is required by pair isolation — the backtest
  copy is the generic non-XAU fallback and must not inherit
  gold-validated deltas. Documented in the factory + a test asserting
  the delta stays XAU-side.
- 6c-upgrade re-arming an exhaustion-D refuted (exhaustion zeroes
  confidence below 6c's floor; regimes mutually exclusive). "WEEKEND
  FILTER" CP-suppression branch is defense-in-depth for direct
  analyze_from_dataframes callers, kept.

Suite: 316 green.

Claude-Session: https://claude.ai/code/session_01UnZkzeFcTu9agFg8BDh2tn
2026-08-04 19:52:07 -04:00
Harwinder Sekhon ebb83749ef audit: GPU-engine cache-miss observability + dead-guard alignment + full parity test
2026-08-04 audit, Batch B. Verified first (h1 harness): the two engines
are byte-identical on every reachable input — 370-prefix flat-bar deck,
warmup band, 322 real-data M15 positions incl. all 7 zero-range bars,
zero diffs. The claimed ADX NaN desync / warmup-guard divergences are
unreachable below the shared 30-bar analyze() floor. Changes here are
therefore observability + hygiene, proven behavior-neutral by exact
reproduction of the pinned M15 12mo book (397 trades, +78.7111R,
byte-identical trade rows).

- _resolve_pos: cache-miss fallback to live-engine recompute now warns
  on first miss and counts every miss (was fully silent — a run could
  switch math with no trace).
- ROC/Keltner warmup guards textually aligned with the live engine so a
  future period/floor change can't open a silent divergence window.
- New regression: full-sweep parity test comparing ALL indicator
  signals/confidences, composite, and regimes on every prefix incl. a
  flat zero-DM/zero-TR run (the old parity test skipped
  ADX/MACD/ROC/Keltner/RSI entirely); plus a loud-fallback test.

Suite: 314 green. Book gate: oanda_m15_12mo_030 reproduced identically.

Claude-Session: https://claude.ai/code/session_01UnZkzeFcTu9agFg8BDh2tn
2026-08-04 19:49:08 -04:00
Harwinder Sekhon 8acc1e18f8 audit: harden Oanda execution lifecycle (close-fill, netting, races, fail-closed config)
2026-08-04 Fable 5 QA audit, Batch A. All fixes are execution-robustness
only — no signal code touched (git diff scope: daemon/executor/tests).
Findings verified by runnable repros before fixing (h3 harness):

- close_trade: a 200 whose close order was created but NOT filled (e.g.
  MARKET_HALTED during the settlement pause) now raises instead of
  returning; the daemon's timeout path keeps tracking and retries. The
  old path popped state and orphaned a live position forever
  (reconciliation was boot-only).
- place_market_order: positionFill=OPEN_ONLY — on a netting account an
  opposite-side order can no longer silently close an untracked
  position; the divergence surfaces as an order cancel.
- _execute_trade: a fill without tradeOpened (netting close/reduce) and
  any placement OandaAPIError (server may have filled before a network
  timeout) now trigger _reconcile_positions instead of silent return.
- _discover_existing_positions → lock-guarded _reconcile_positions,
  callable mid-run; adoption of a position with NO broker stop now
  screams (CRITICAL + per-bar warning) instead of computing sl=entry,
  risk=0 and silently never trailing.
- _manage_trailing: snapshot deep-copies per-trade dicts (no mutation of
  live state outside the lock); write-back skips trades popped by
  _check_closed_trades mid-iteration (no ghost resurrection).
- Fail-closed config: unknown OANDA_QUALITY_GATE refuses to start
  (previously a typo silently became the most permissive gate "C");
  non-integer OANDA_LOT_SIZE/MAX_HOLD_BARS exit with a clear message.

Suite: 302 → 312 tests (Module 21), all green.

Claude-Session: https://claude.ai/code/session_01UnZkzeFcTu9agFg8BDh2tn
2026-08-04 19:46:37 -04:00
Harwinder Sekhon 28a1c375bd M5 at real cost: disable XAU_PB, squash partial EMA alignment
Both validated at the measured $0.65 spread on Oanda bars (12mo halves +
PF, OOS 2024-06->2025-03, 5yr; full evidence in pair_signals/xauusd.py
M5 overrides 4-5):

- m5_pullback_enabled False on M5: loss forensics identified XAU_PB as
  the largest wrong-direction loss source; 12mo PF 1.26 (failing the
  acceptance bar) -> 1.44, max DD 23.9R -> 13.9R. PB code stays for
  future regime-gated experiments.
- New TechnicalConfig.ema_partial_neutral (default False), True on M5
  only: partial 4-EMA ribbon alignment (2-3 of 4 checks) scores NEUTRAL
  instead of a weak BUY/SELL (-0.008R/84 vs +0.230R/341 fully aligned).
  Squash mirrored in the backtester's precomputed EMA scorer in this
  same commit (live==backtest parity). 12mo +97.6R/556 PF 1.48 DD
  10.7R halves +47.8/+49.8; 5yr +127.4R PF 1.38; OOS one-trade tie.
- Rejected and logged (memory + docstring): hour-19 block (OOS fail),
  hours-14/15, trail dist 0.75, confirm-TF swaps incl M15+H4 (OOS
  ordering), divergence/candlestick weight changes, bollinger removal.
- Tests 296 -> 302: partial-alignment fixture test, factory-equivalence
  mirrors, per-TF isolation test, BC-gate fixture force-enables PB.

Claude-Session: https://claude.ai/code/session_01H1eKWZsAVapJ1C4winYLqX
2026-08-04 16:54:09 -04:00
Harwinder Sekhon 775330ff16 /news: normalize rounded-zero score so "-0.00" renders as "+0.00"
A balanced feed (XAU and USD means nearly cancelling) yields a tiny
negative net that "+.2f" rendered as "-0.00" — seen live 2026-08-03.
Regression test uses that day's actual per-currency means (300 tests).

Claude-Session: https://claude.ai/code/session_017975c1TufZ3xRYAq8iWTCS
2026-08-03 18:06:44 -04:00
Harwinder Sekhon 1f54452007 Telegram /news command: net XAU/USD news sentiment score on demand
Owner-locked /news replies with the existing get_pair_sentiment() net
score (XAU mean - USD mean, -1..+1) and its label; a caveat line marks
the no-headlines case so a feed outage can't read as neutral. Fetch runs
in the shared executor with a placeholder-then-edit reply. Display-only:
news still never gates signals. Registered in handler block, command
menu, help text, docstring, and user guide; 3 new builder tests (299).

Claude-Session: https://claude.ai/code/session_017975c1TufZ3xRYAq8iWTCS
2026-08-03 18:01:42 -04:00
Harwinder Sekhon be5dd4a25b Dashboard publishes directly into /var/www/trade (no subfolder)
Adds exactly three files next to the portal: dashboard.html,
snapshot.json, news.json. README covers chown vs setfacl for write access.

Claude-Session: https://claude.ai/code/session_01Wc9ZoqKLYBT2QJcpRWKoGm
2026-08-03 17:17:37 -04:00
Harwinder Sekhon 0445a20a76 Dashboard out-dir default → /var/www/trade/gold-dashboard (live portal docroot)
Claude-Session: https://claude.ai/code/session_01Wc9ZoqKLYBT2QJcpRWKoGm
2026-08-03 17:14:08 -04:00
Harwinder Sekhon 4e654b5edb Web dashboard v2: static publisher + Apache, with calendar & news panels
- dashboard_publisher.py: standalone candle-close daemon that atomically
  writes snapshot.json/news.json/dashboard.html to an Apache-served dir
  (DASHBOARD_OUT_DIR). Fully independent of telegram_bot/oanda_daemon —
  local confirmation map + boundary math, no imports either way, no new
  deps, no open port. Due-TF classification from the RAW wake time.
- static/dashboard.html: self-contained dark page — hero rail, four TF
  cards (score bar, 3x3 indicator grid, MTF, setup, reasons), DeepSeek
  market context, week economic calendar, scored newswire. Clock offset
  from the HTTP Date header (publish-time epoch is stale between closes).
- news_engine.get_week_events(): additive read-only accessor over the
  cached FF week; no signal path calls it.
- deploy/: systemd unit + live-server (LAMP/SELinux) checklist.
- test_all.py: Module 20 (9 tests) + get_week_events test; 296 green.

Claude-Session: https://claude.ai/code/session_01Wc9ZoqKLYBT2QJcpRWKoGm
2026-08-03 17:10:29 -04:00
Harwinder Sekhon 9e1c321f05 Add commit-ladder campaign driver; ignore its 26MB results folder
Added: backtest/research/commit_ladder.py (recreates historical configs
via --set on the fixed harness; 2026-08-03 campaign confirmed HEAD is
the best config on all timeframes). Results stay local like all pinned
trade books.

Claude-Session: https://claude.ai/code/session_01Nsd8HcrV67GbCZXxZEn1Ma
2026-08-03 13:29:18 -04:00
Harwinder Sekhon 6fff06ff90 DeepSeek V4-Flash news enrichment: LLM sentiment, surprise direction, summary
Three display/warning-only upgrades to the news engine, all behind the
existing keyword path as fallback — signals are never gated on LLM
output (event_risk_caps_signals / sentiment_affects_signals stay False
per the 2026-07 audit; backtester use_news=False untouched):

- Headline sentiment: one batched call per 600s cache window (capped at
  25 new headlines, sha1-cached so repeats are never re-sent) overrides
  the keyword scores per currency with the gold-inverse doctrine encoded
  in the prompt.
- Release surprises: get_event_risk_level's description now says which
  way the print pulls GOLD ("— gold bearish: strong jobs lift yields"),
  fixing the documented numeric-only gap; risk level and windows
  unchanged.
- Market summary: 1-2 sentence macro blurb on SignalReport.news_summary,
  rendered in Telegram alerts (📰) and the CLI (🧠).

New llm_client.py: requests-only OpenAI-compatible client (no SDK), one
attempt per call, UTC-day request cap, singleton so the cap survives
per-scan NewsEngine reconstruction. Config: deepseek block in
config.json + DEEPSEEK_API_KEY env; absent key = fully inert.

Live-measured API behavior baked in: V4-Flash is a hybrid thinking model
— requests send thinking:disabled (reasoning otherwise blew the timeout
and could exhaust max_tokens before the answer); prompts are static
module constants so DeepSeek's prefix cache prices repeat calls at the
cache-hit rate; scores use compact [s, xau] pairs; timeout 12s.

Tests 273 → 286 (new Module 19 + deepseek config tests).

Claude-Session: https://claude.ai/code/session_01Nsd8HcrV67GbCZXxZEn1Ma
2026-08-03 13:25:02 -04:00
Harwinder Sekhon f7d8f1113c Tiingo WebSocket streaming: re-enable the level-breach monitor
The monitor has been dead since the Oanda feed switch (Oanda's provider
does not stream). New config.json scalar data_providers.streaming selects
a tick-stream provider independent of the candle feed; telegram_bot
resolves it via _get_streaming_dp() (separate instance, candle cache
stays on the active provider) and falls back to the active provider when
unset — behavior is byte-identical to before when the key is absent.

TiingoDataProvider grows the two sanctioned streaming overrides:
supports_streaming=True and iter_ticks() on wss://api.tiingo.com/fx
(mid price only — the no-spread rule extends to the stream). Two
deliberate divergences from the TD reference implementation, both
because Tiingo authenticates inside the socket: server "E" messages
raise TiingoDataError (a swallowed auth error would end the generator
cleanly and reconnect-hammer with zero backoff), and on_connect fires on
the first received message rather than after the subscribe send. Field
order of quote frames was live-verified: ask is data[7], not data[6] as
the docs table suggests (data[6] is askSize).

make_data_provider(config, name=None) now takes an explicit provider
name; keyless streaming providers disable the monitor with one warning
instead of a traceback-per-minute loop.

Tests 266 → 273 (WS parser, ticker round-trip, streaming config scalar,
explicit-name dispatch, streaming-dp resolution, monitor auto-disable).

Claude-Session: https://claude.ai/code/session_01Nsd8HcrV67GbCZXxZEn1Ma
2026-08-03 13:11:36 -04:00
Harwinder Sekhon bcda6e0ebd Cleanup: remove dead code, stale docs, and all non-main branches
- Delete back-compat shims data_provider.py / oanda_provider.py;
  test_all.py now imports providers.* directly (ForexDataProvider name
  retired; the two shim-only tests removed, 268 -> 266 tests)
- Delete backtest/research/calendar_store.py (no importer, no entry
  point); config.py audit comment reworded
- Delete executed plan doc (2026-07-21 centralized-config) and the two
  unbuildable guide PDFs; HTML guides remain the source of truth
- OANDA_MIGRATION_ROADMAP.md status header corrected: phases 1-2
  shipped 2026-07-28, feed is live on Oanda; phases 3-6 still unbuilt
- Branches deleted locally + on both remotes: oanda_demo (merged),
  web-dashboard, eurusd-modular, reversal-exit-alert (owner-approved
  permanent deletion); July stash dropped

Claude-Session: https://claude.ai/code/session_019zR58p8ACW1oJ5QXVUZ5es
2026-08-03 11:08:48 -04:00
Harwinder Sekhon 042b079861 Big-candle gate (default-OFF) + M5 relaxation campaign + --set tooling
Gate 6h (big-candle entry gate): blocks all XAU entries (core, XAU_PB,
XAU_CP) when the primary TF's last 24h contains a bar with range >= K x
prior-bar ATR. Ships default-OFF: validation rejected every threshold
(K 3-6 on M5/M15 12mo) — removed trades were net-POSITIVE at every cell;
M5 quiet-window trades are the only losing bucket. Config knobs
big_candle_atr_multiple/lookback_hours, --big-candle backtest flag,
Module 17 tests.

M5 relaxation SHIPPED (validated 2026-07-30, Oanda bars, $0.30 spread):
m5_min_abs_score 30->25, m5_min_confirmation_score 10->5, hour-21 UTC
unblocked M5-only. Beats control in ALL 6 calendar years of the 5yr
replay (+230.9R/2319 PF 1.26 DD 21.3 vs +168.0R/2048 PF 1.21 DD 27.0),
12mo +107.4R PF 1.36 vs +84.6R PF 1.33, OOS +26.1R vs +17.2R, $0.70
spread +86.9R vs +66.4R. M15: every relaxation retested negative —
config unchanged. Declined-but-open: M5 hour-22 block, M15 hour-16
unblock (both fail 5yr/OOS). Evidence in the factory docstring.

Tooling: generic --set PATH=JSON research override in gpu_backtest
(typo-proof, logged); diagnose_filters filter_tags refreshed for gates
6d-6i with a staleness-guard test (Module 18); research scripts
hour_analysis/burst_profile/book_diff. Suite: 268 tests.

Claude-Session: https://claude.ai/code/session_014jqHpmNYmwYMowF6bPZyeD
2026-07-30 11:30:20 -04:00
Harwinder Sekhon dc7beb2877 Rebuild CLAUDE.md: lean architecture reference
Replaces the deleted exhaustive knob-by-knob reference with a compact
guide: commands, config precedence, pipeline architecture, rules for
changes, and current-state notes (Oanda feed, TD-pinned books).

Claude-Session: https://claude.ai/code/session_014jqHpmNYmwYMowF6bPZyeD
2026-07-30 11:30:07 -04:00
Harwinder Sekhon 637c39e201 Switch active data source to Oanda v20 (practice feed)
config.json.example: data_providers.active = "oanda" (credentials stay
in env/.env or the top-level oanda block; TD and Tiingo remain
registered fallbacks). Backtest parquet cache is now keyed per provider
— Oanda and TD bars diverge (2026-07-24 study), so a feed switch must
never serve another feed's cached bars; twelve_data keeps the legacy
un-prefixed filename so pinned-book caches stay valid. 254 tests green;
live smoke-tested against the practice API (66 contract checks: all
scanned TFs, boundary/freshness guarantees, real spread $0.71,
fetch_range pagination, typed error paths, CLI + backtest end-to-end).

Claude-Session: https://claude.ai/code/session_01Mc7XKYyPySVXiUpcN82QEK
2026-07-28 20:12:59 -04:00
Harwinder Sekhon 083a4e8d63 Provider abstraction: providers/ package with BaseDataProvider template
One class per provider (TwelveData, Oanda, Tiingo), all subclassing
BaseDataProvider which owns cache, validation/cleaning, freshness +
boundary guarantees, fetch_range pagination, and streaming stubs —
subclasses implement only _fetch_raw. config.json data_providers.active
selects the live feed; data_provider.py / oanda_provider.py stay as
back-compat shims. Module 16 provider-abstraction tests (253 total green).

Claude-Session: https://claude.ai/code/session_01Mc7XKYyPySVXiUpcN82QEK
2026-07-28 20:09:00 -04:00
Harwinder Sekhon f760e9070a Research: Dukascopy real-spread study + Oanda/TD bar-divergence study
Scripts + summary CSVs for the 2026-07-24 studies (real XAU spread ~$0.65
tradable hours; Oanda vs TD bars diverge ~11% of ATR with 10% direction
flips). Large raw data files (parquet, per-boundary samples) gitignored.

Claude-Session: https://claude.ai/code/session_01Mc7XKYyPySVXiUpcN82QEK
2026-07-28 20:08:53 -04:00
Harwinder Sekhon 9a384935bf Add Oanda migration roadmap (reference, not scheduled)
Approved 2026-07-24: 6-phase plan for fully switching the signal feed
from Twelve Data to Oanda v20 (inert plumbing -> backtester agnosticism
-> pricing-stream monitor -> shadow parity -> book re-validation with
owner cost gates -> reversible cutover). TD remains a permanent
fallback. Grounded in the 2026-07-24 divergence study (10% bar-direction
flips, 11%-of-ATR noise, real Oanda spread ~$0.72 vs assumed $0.30).

Claude-Session: https://claude.ai/code/session_01YNvxXtgz62F7kTKZZsAAAE
2026-07-24 15:12:37 -04:00
Harwinder Sekhon d669227ddc Oanda demo daemon: auto-execution on practice account (XAU/USD)
New independent trading daemon (oanda_daemon.py) that places orders on
an Oanda v20 practice account instead of sending Telegram alerts.
Oanda-only market data (oanda_provider.py, mid candles + real bid/ask
spread) through the unchanged signal pipeline; order execution and
trade management via oanda_executor.py (raw requests, no new deps).

Book parity with the backtester: no broker-side TP (exits = SL/TRAIL/
TIMEOUT only), bar-close-only trail ratchet with each closed bar
processed exactly once, 5-bar per-TF entry cooldown, structural SL
from the setup. Accepted divergence: one position total (skipped
signals logged for book reconciliation).

Safety/ops: fail-fast boot on rejected credentials (transient outages
continue), market orders never retried (double-fill risk), entry lock
against aligned-boundary races, restart recovery + adoption via
oanda_state.json, hourly heartbeat, asyncio exception handler, every
failure path logged. Prices formatted at instrument precision.

Config: env OANDA_* -> config.json "oanda" section (validated in
app_config). Tests: Module 15 (30 tests), suite 240 green.

Claude-Session: https://claude.ai/code/session_01VUnCqjWBuvKRSjLwHeB9BJ
2026-07-24 14:10:45 -04:00
Harwinder Sekhon c3329d6cd5 Telegram alert redesign: verdict-first layout + informational 100-pip TP
format_signal_message restyled (display-only, zero signal-logic impact):
verdict header first (signal - pair - TF - quality), trade setup moved
above the detail, per-indicator breakdown condensed to one arrow line
(_indicator_summary), heavy dividers dropped. New TP 100p line in every
setup block: entry +/- 100 pips ($10.00 on XAU) with its R-multiple vs
the stop (_INFO_TP_PIPS, formatter-only - not stamped on TradeSetup,
not in CLI/JSON/backtester). Trailing-plan logic unchanged (stamped
per-TF params, pullback fallback). Removed dead _indicator_line and
_SIGNAL_TAG. Tests 210 -> 212 (100p TP math BUY/SELL/no-setup, one-line
indicator summary).

Claude-Session: https://claude.ai/code/session_01BoZKi67Ct6QTeiFJmybXSw
2026-07-24 10:58:22 -04:00
Harwinder Sekhon 7a810c4a4c Remove Tiingo spread monitor (indicative two-tier feed, unvalidatable)
Tiingo's free /tiingo/fx/top publishes only a two-tier indicative
spread for gold ($0.10 liquid hours / $0.70 rollover+news; verified on
273 production samples + live probes) — mid ± a posted increment, not
executable depth, so it can never validate the $0.30 backtest cost
assumption. Removed one day after shipping: tiingo_provider.py, the
sampler/quote plumbing in telegram_bot.py, the channel_spread_detail
config knob and tiingo_api_key resolver, and Module 14's Tiingo tests
(221 -> 210). CLAUDE.md documents the verdict: do not re-add a spread
display from Tiingo's free FX feed.

Claude-Session: https://claude.ai/code/session_01BoZKi67Ct6QTeiFJmybXSw
2026-07-24 10:56:55 -04:00
Harwinder Sekhon 3a95a37a0c Accuracy audit fixes: alerts carry validated per-TF trail plans + 13 more
Five-agent audit of the signal path; all fixes are delivery/measurement
side. Regression-proven behavior-neutral on validated signals: the 5yr
M15 replay reproduces the pinned book exactly (+210.65R PF 1.35
WR 57.8% MaxDD 33.90R). Tests 215 -> 221.

- TradeSetup carries trail_arm_r/trail_dist_r stamped from the producing
  config; Telegram trail plan reads them (was showing default 0.6/0.75
  instead of the validated M15 arm 0.70 / M5 dist 0.85)
- Alert dedup: fingerprint = (signal, quality, direction); entry movement
  compared against last-ALERTED entry/risk (old SL-dist bucket churned on
  ATR drift -> near-duplicate alerts every candle)
- Fresh fetches enforce the closed-bar boundary check with one 3s grace
  retry, then warn (feed lag no longer silently analyzes the prior bar)
- Backtest max DD includes the zero baseline (opening losing streaks
  were undercounted; pinned M15 DD unchanged)
- Replay loads the regime-filter TF when the H4-ADX gate is on (was a
  silent no-op for M5/M15 primaries)
- clear_cache under _cache_lock; dead ROC zero-cross branches removed
  from both engines (provably unreachable); RSS dates via
  parsedate_to_datetime (named zones no longer stamped UTC); event risk
  grades from the soonest event; .env inline comments stripped;
  auto-scan skips NEUTRAL like the breach path; report_to_dict honors
  the weight-0 skip rule and emits strategy/trail params; PB funnel
  counts spread_gate/accepted; session-filter fail-open paths log loudly
- Docs corrected to match validated code (H&S neckline = lower peak,
  exhaustion anchors farthest tracked swing) with re-validation warnings

Claude-Session: https://claude.ai/code/session_01R5soRUeRBi8kHMB99csrJy
2026-07-23 10:31:13 -04:00
Harwinder Sekhon f814485615 Tiingo bid/ask spread monitor + logging hardening
- tiingo_provider.py: /tiingo/fx/top quotes (10s TTL cache, graceful
  None on any failure); spread log writer. Signals never consume
  bid/ask; warn-only by design (no historical bid/ask to validate a
  gate against — live == backtest rule).
- app_config: repo-root .env loader (setdefault; shell env still wins),
  tiingo_api_key resolver (TIINGO_API_KEY/TIINGO → config.json),
  telegram.channel_spread_detail flag (default false).
- telegram_bot: 5-min spread sampler → logs/spread_log.csv (weekend-
  aware), alert-time quote refresh, bid/ask + wide-spread warning in
  signal messages, /status spread line. Raw numbers to owner DM always;
  channel only while channel_spread_detail=true (Tiingo free tier is
  internal-use-only; flip before public launch). Free-tier limit math:
  worst case ~41 req/h vs 50/h cap, one pair.
- Logging: 3h rotation x 240 backups (30-day retention), full-date
  stamps, force=True root handlers; candle loops + sampler survive
  failed cycles; breach-task exceptions logged via done-callback;
  PTB error handler; sys/threading excepthooks in main().
- test_all.py: Module 14 (tiingo, dotenv, spread routing, loop
  resilience, rotation) — 215 tests green.

Claude-Session: https://claude.ai/code/session_01HVRmy6KspMs96o5PTPYFAB
2026-07-22 17:18:14 -04:00
Harwinder Sekhon 65a1debc4e gitignore: cover M5 trail-grid research CSVs
Claude-Session: https://claude.ai/code/session_01VPNF37MP9Kuv62B1VBrZ1b
2026-07-21 19:39:08 -04:00
Harwinder Sekhon 1499bb7cf2 M5 trail dist 0.85 (validated grid, arm unchanged)
Same-day sibling of the M15 campaign, opposite finding: on M5 the arm
point barely matters — winners need a looser leash. 5yr grid plateau
arm 0.5-0.7 x dist 0.85-1.0; shipped the OOS-preferred edge 0.60/0.85:
+231.2R PF 1.30 vs +218.0R PF 1.29 baseline, both halves better, DD
flat (19.2 vs 19.1), OOS +14.5R vs +11.1R. dist 1.0 rejected at the
OOS gate despite best in-sample total. XAU_PB pb_trail unaffected.
Factory replay reproduces the grid cell; 201/201 tests.

Claude-Session: https://claude.ai/code/session_01VPNF37MP9Kuv62B1VBrZ1b
2026-07-21 19:13:01 -04:00
Harwinder Sekhon 902122dbfe M15 trail arm 0.70 (validated grid) + backtest/live parity hardening
Parity: replay now applies the live _apply_regime_filter_live method
under the live guard (function-identity, default-off); unregistered-pair
fallback applies the config.json overlay; parity sweep gains direct
ATR/vol-regime assertions; regime-filter semantics test added.

Trail campaign: --trail-arm/--trail-dist research flags; 30-cell grid on
5yr M15 refuted the earlier-arm hypothesis and found the 0.70-0.80 arm
plateau. Shipped M15-only trail_arm_r=0.70 (factory override):
+210.7R PF 1.35 WR 57.8% vs +175.7R PF 1.30 WR 53.9% baseline; both
halves and 2024-06->2025-03 OOS better (+57.4 vs +45.4); 5yr max-DD
clause (33.9R vs 29.1R) explicitly waived by owner. M5 unchanged.
Factory replay reproduces the grid cell trade-for-trade; 201/201 tests.

Claude-Session: https://claude.ai/code/session_01VPNF37MP9Kuv62B1VBrZ1b
2026-07-21 16:40:33 -04:00
Harwinder Sekhon f0f9dcb169 Remove dead code: legacy config fields, unused helpers/imports, stray files
- Delete test_ws_dual.py (standalone probe, unused) and a stale tracked log
- config.py: drop MarketSession enum, is_jpy_pair, candle_buffer_seconds,
  and the never-read legacy SignalConfig/XauStrategyConfig fields
- data_provider.py: drop uncalled get_latest_price/get_spread_estimate
- Remove unused imports (engine, telegram_bot, main, news_engine)
- test_all.py: drop/trim the corresponding defaults assertions (199 tests)
- CLAUDE.md updated to match

Zero behavior change: no production code read any removed field
(verified reference-by-reference); suite green at 199/199.

Claude-Session: https://claude.ai/code/session_01VPNF37MP9Kuv62B1VBrZ1b
2026-07-21 14:06:55 -04:00
Harwinder Sekhon c8a2c83820 Merge branch 'centralized-config': config.json deployment overlay
Centralized deployment settings (Telegram credentials, data-provider
keys/selection, scan pairs/timeframes, per-pair base blocked hours,
alert quality, log level) into a gitignored config.json with env-var
precedence. Hardcoded credentials removed from source. Zero signal-
logic change — parity pinned by tests (200/200).

Claude-Session: https://claude.ai/code/session_01VPNF37MP9Kuv62B1VBrZ1b
2026-07-21 13:44:34 -04:00
Harwinder Sekhon 2d3e524f1c Final review fixes: scrub tracked credentials, trim scannable TFs, absolute parity pin, scanning type-checks
Claude-Session: https://claude.ai/code/session_01VPNF37MP9Kuv62B1VBrZ1b
2026-07-21 13:42:00 -04:00
Harwinder Sekhon f017f5aef5 gitignore: cover .env.* swap files and the whole logs/ directory
Rotated logs (telegram.log.YYYY-MM-DD) don't match *.log and a vim
.env.swp was nearly committed by a bulk add.

Claude-Session: https://claude.ai/code/session_01VPNF37MP9Kuv62B1VBrZ1b
2026-07-21 13:30:21 -04:00
Harwinder Sekhon a56a622846 Docs: correct PlatformConfig overlay scope (credentials never pass through the factory)
Claude-Session: https://claude.ai/code/session_01VPNF37MP9Kuv62B1VBrZ1b
2026-07-21 13:30:15 -04:00
Harwinder Sekhon b9633389fa Docs: config.json deployment overlay (CLAUDE.md repo map, env table, test count)
Documents the app_config.py / config.json overlay shipped in Tasks 1-4:
repo-map rows for app_config.py and config.json.example, env-var table
precedence (env -> config.json -> none, hardcoded fallbacks removed
2026-07-21), PlatformConfig/pair_signals overlay call order, and the
real 199-tests/13-modules suite count. Also fixes a stale SCAN_PAIRS
comment in telegram_bot.py that still named PlatformConfig.scan_pairs
as the deployment switch instead of config.json's scanning.pairs.

Claude-Session: https://claude.ai/code/session_01VPNF37MP9Kuv62B1VBrZ1b
2026-07-21 13:25:21 -04:00
Harwinder Sekhon 6e2fdbd684 telegram_bot: credentials/quality/scan-loops from config.json, startup guard
- Credentials, alert quality, scan pairs/timeframes, log level and
  candle-loop intervals now resolve via app_config (env -> config.json
  -> code default) instead of hardcoded fallbacks.
- _get_shared_dp() uses make_data_provider() for provider-registry
  dispatch.
- main() refuses to start without a token/chat_id, raising SystemExit
  with an actionable message instead of a bare sys.exit string.
- test_all.py Module 13: two new tests covering credential/quality/
  loop wiring via a reload, and the startup guard.

Claude-Session: https://claude.ai/code/session_01VPNF37MP9Kuv62B1VBrZ1b
2026-07-21 13:18:05 -04:00
Harwinder Sekhon 80a2b89769 Data provider: key from config.json/env, provider registry, drop hardcoded key
Resolves the Twelve Data API key through app_config.provider_api_key()
(env → config.json → no default) instead of a hardcoded field on
PlatformConfig. Adds data_provider.PROVIDER_REGISTRY + make_data_provider()
so signal_aggregator and the backtester build the active provider through
one seam; telegram_bot.py gets the same one-line WS-key fix now (per Task 4's
already-planned edit for that line) so the suite doesn't crash on import
between this commit and Task 4's full wiring pass.

Claude-Session: https://claude.ai/code/session_01VPNF37MP9Kuv62B1VBrZ1b
2026-07-21 13:10:58 -04:00
Harwinder Sekhon 1610507d57 Wire config.json overlay into the pair factory (before per-TF deltas)
Implements Task 2: apply_pair_overrides in xauusd.make_config before
the validated per-TF deltas, so e.g. M5 hour-9 appends on top of the
JSON base. Updated Module 9 equivalence test and added three parity pins.

Claude-Session: https://claude.ai/code/session_01VPNF37MP9Kuv62B1VBrZ1b
2026-07-21 12:59:12 -04:00
Harwinder Sekhon 1dab14d298 app_config: reject non-object config.json sections with AppConfigError
Claude-Session: https://claude.ai/code/session_01VPNF37MP9Kuv62B1VBrZ1b
2026-07-21 12:54:46 -04:00
Harwinder Sekhon cbc01565a2 Add app_config.py: config.json loader, validation, env-precedence resolvers
Claude-Session: https://claude.ai/code/session_01VPNF37MP9Kuv62B1VBrZ1b
2026-07-21 12:47:48 -04:00
Harwinder Sekhon 53b65e1bbb Plan: centralized config.json implementation (5 TDD tasks)
Claude-Session: https://claude.ai/code/session_01VPNF37MP9Kuv62B1VBrZ1b
2026-07-21 12:44:24 -04:00
Harwinder Sekhon 1a141abc7c Add config.json.example template; gitignore config.json
Claude-Session: https://claude.ai/code/session_01VPNF37MP9Kuv62B1VBrZ1b
2026-07-21 12:32:28 -04:00
Harwinder Sekhon 096b69686d Spec: centralized config.json design (deployment settings overlay)
Claude-Session: https://claude.ai/code/session_01VPNF37MP9Kuv62B1VBrZ1b
2026-07-21 12:25:08 -04:00
Harwinder Sekhon c21c102536 News informational-only: gate sentiment bonus/penalty off by default
Add NewsConfig.sentiment_affects_signals (default False), gating the +5
aligned-sentiment confidence bonus and the 10-pt news_conflict penalty.
With event_risk_caps_signals already False, live decisions are now pure
technical, matching the backtester (use_news=False). Warnings and the
sentiment display are unchanged. Adds a gating regression test.

Claude-Session: https://claude.ai/code/session_01FMaaPdrrrQuh1VvGQH1icN
2026-07-17 16:03:31 -04:00
harry c87bb14830 Merge pull request 'Remove EUR/USD support — XAU/USD-only codebase' (#1) from xau-only-chart-patterns into main
Reviewed-on: http://192.168.55.11:3000/harry/ForexBot2/pulls/1
2026-07-16 20:09:21 +00:00
Harwinder Sekhon a3279e54fa Enable BUY-only chart patterns on M5 — bearish patterns dropped (OOS negative)
M5 factory: cp_enabled=True, patterns limited to inv_hs, double_bottom,
bull_flag. SELL CP lost -3.21R on OOS; BUY CP: +41.8R PF 1.60 on 12mo,
+65.2R PF 1.48 on 5yr. Adds +22R to the M5 book (708 vs 513 trades).
M15/H1 remain disabled — all failed OOS validation.

Claude-Session: https://claude.ai/code/session_015kby7M9JwpYi8NQT2yoxAi
2026-07-16 16:08:13 -04:00
Harwinder Sekhon 1bce289132 Docs: CLAUDE.md — document XAU_CP archetype (pipeline step 11, generator D, cp_* knobs)
Claude-Session: https://claude.ai/code/session_015kby7M9JwpYi8NQT2yoxAi
2026-07-16 14:31:45 -04:00
Harwinder Sekhon f3d3e56aa8 Add chart-pattern entry archetype (XAU_CP) — disabled by default
Third entry archetype: completion of classical chart patterns (double
bottom/top, H&S top / inverse H&S, bull/bear flag) — the tight canon
with the strongest documented stats. New chart_patterns.py detector
module (pure functions, ATR-unit geometry, closed-bar slices only,
first-close-beyond trigger discipline so a breakout fires exactly
once and never repaints).

Pipeline step 10 in analyze_from_dataframes: fills only empty setup
slots, allows weak-composite F grades (normal at reversal completion)
but inherits session/day blocks, mirrors MTF-opposition + SELL
vol/hour + M5 protections. Geometry: SL = pattern anchor -/+ 0.5 ATR
(min $8), TP1 = measured move, $9 spread gate, trailing exit does the
real work. Strategy tag XAU_CP.

cp_enabled defaults to False everywhere — per-TF adoption happens in
pair_signals/xauusd.py only after the standard validation rule passes.
Backtester: --cp / --cp-patterns flags to test the candidate, XAU_CP
tag recognition.

Suite: 183/183 (21 new tests: detectors, trigger discipline,
invalidation, gates, pipeline, default-off safety).

Claude-Session: https://claude.ai/code/session_015kby7M9JwpYi8NQT2yoxAi
2026-07-16 14:23:17 -04:00
Harwinder Sekhon 14448c10e8 Remove EUR/USD support — XAU/USD-only codebase
EUR/USD never had a validated edge (2026-07-11/12 verdict: no config
passed validation on M5/M15/H1 at 1-pip cost; cost-to-volatility root
cause). Removed: EurUsdStrategyConfig, pair_signals/eurusd.py, EUR
gates 6h + weekend/day filters + strategy marker + trade geometry in
signal_aggregator, EUR pair registry/aliases in main.py and
gpu_backtest.py, EUR/USD news pair mappings, test_all.py Module 10.
Generic multi-currency news keyword infra kept (GBP/JPY/EUR keyword
table is not pair code). Full EUR history remains at 7d3fd95.

Suite: 162/162 green (was 187; 25 EUR-specific tests removed).

Claude-Session: https://claude.ai/code/session_015kby7M9JwpYi8NQT2yoxAi
2026-07-16 14:06:27 -04:00
Harwinder Sekhon 665ffdda48 Fix on Sell order lossing by blocking time 2026-07-16 12:17:56 -04:00
Harwinder Sekhon fe7d4c5d4a Audit fixes: cache boundary staleness, 6e session bypass, GPU engine parity
Fixes all high+medium findings from the 2026-07-16 full-codebase audit.

High:
- data_provider: cache now rejects snapshots missing the latest closed
  bar (_has_latest_closed_bar, weekend-exempt). The wall-clock TTL served
  mid-candle snapshots at the next boundary, so live M15/H1 scans ran one
  full bar behind and the bar-close session check evaluated the wrong hour.
- signal_aggregator: gate 6e (SELL quality) only acts on actionable
  grades. It was re-grading session-blocked F reports to D, which
  re-opened the M5 pullback path (entry check: quality != "F") on
  session-blocked bars.
- gpu_backtest: port the live EMA-ribbon compression/expansion logic
  (+10 conf on expansion) into the GPU engine. Its absence diverged
  composite score/conf on 7-9% of bars and flipped signals on ~0.2%.

Medium:
- gpu_backtest: Bollinger precompute uses the exact pandas rolling calls
  (cumsum E[x2]-E[x]2 lost precision at gold price levels; bb_position
  error up to 7.6e-3 crossed signal thresholds, worst during squeezes).
- gpu_backtest: --confirm-tfs defaults to the pair factory's live
  confirmation map instead of a hardcoded 1h,4h.
- gpu_backtest: entries re-anchor to the first RAW bar after the signal
  bar, so >10-sigma spike bars dropped by cleaning are no longer skipped
  at entry (favorable bias).
- gpu_backtest: per-TF warmup pad — D1 confirms get >=210 bars at window
  start (62-day pad gave ~61 bars: no EMA200, unlike live).
- data_provider: thread-safety locks for cache + throttle (shared
  provider across the bot's 4-thread pool); outlier filter keeps the
  snap-back bar after a spike instead of punching a two-bar hole.
- telegram_bot: alert fingerprint gains a risk-relative entry bucket
  (materially-moved setups re-alert) and is committed only after a
  successful send (failed sends no longer consume the alert for 1h).
- engine: disabled_indicators=["macd"] now removes the MACD pillar from
  scoring (histogram still computed for divergence).

Verification: test_all.py 187/187 (new Module 11: 8 audit-fix
regressions incl. a live-vs-GPU parity sweep with ribbon coverage);
real-data parity sweep over 813 slices of 14mo XAU M15: zero divergence
(pre-fix: 7-9% of bars diverged); both original bug repros no longer
reproduce; cached end-to-end backtest run passes with the live confirm
map. Books shift slightly vs pre-fix runs — comparability note added to
CLAUDE.md; pinned tuned numbers need re-validation under the fixed
backtester.
2026-07-16 10:24:39 -04:00
Harwinder Sekhon 4d54fddf80 Docs: delete DOCUMENTATION.md — both halves superseded
Unlike USAGE.md (a USD/JPY-era fossil), this doc was structurally sound:
right pair, real architecture, correct composite formula, and 8 of its 9
regime-weight cells match engine.py. It is removed because both of its
sections now have live successors, and because the numbers it does state
are wrong in ways that mislead:

- Section 1 (trader-facing) → docs/user_guide.html + beginner_guide.html,
  which are newer and cover the trailing-stop exit model this doc never
  mentions.
- Section 2 (developer reference) → CLAUDE.md.

Stale/incorrect content it carried: the entire quality-grade table was
wrong (claimed A+ 45/70, A 30/60, B 20/50, C 12/40, D 8+; the code grades
40/65, 33/55, 28/50, 25/45, 15), Divergence's trending weight was listed
2.0 vs 1.0 in code, and it predates the 2026-07 work entirely — zero
mentions of XAU_PB or trailing exits (the actual default exit model), and
near-zero coverage of the EUR/USD port.

CLAUDE.md's header now points at docs/ as the human-facing docs instead
of disclaiming this file.
2026-07-16 09:34:04 -04:00
Harwinder Sekhon aaf1d37b14 Docs: delete stale USAGE.md and superseded pullback plan
USAGE.md documented a different product: USD/JPY (unsupported — the
code is XAU/USD + EUR/USD), 16 main.py flags that don't exist
(--backtest, --optimize, --auto-session-filter, …; main.py has five),
and backtest.py/metrics.py, which aren't in the repo. Two claims were
actively misleading: it called the session filter "default off" (it is
True by default, 03:00-23:00 not 07-16) and gave regime-min-adx as 30
(real default 25). CLAUDE.md now covers the CLI, bot and backtester
correctly, so this is a delete rather than a rewrite.

PULLBACK_ENTRY_PLAN.txt was a pre-implementation draft whose spec no
longer matches what shipped — it specifies a 0.5-2.5 ATR retrace window
and a detector in engine.py, while XAU_PB ships 0.8-6.0 (the drafted
window rejected 81% of valid pullbacks) with the detector in
signal_aggregator.py so live and backtest share one code path. Its
validated outcome (+38.3R / 254 trades / 12mo M5) and every shipped
gate are recorded in CLAUDE.md and the config docstrings.

Also drops the now-dangling references to both files from CLAUDE.md
and the config.py XAU_PB comment.
2026-07-16 09:30:50 -04:00
Harwinder Sekhon 18b5ef27d2 Docs: rewrite CLAUDE.md as a complete codebase reference
Replaces the summary-level doc with a full spec derived from a read of
every core module, so contributors (and low-context models) don't have
to infer behavior or rediscover tuned values:

- repo map with line counts; flags DOCUMENTATION.md/USAGE.md as stale
- every config knob with its real default and the backtest evidence
  behind it; marks the legacy/unused fields (SignalConfig.weights,
  min_confidence, cooldown_bars, XAU h4/h1 gates) so nothing new is
  built on dead knobs
- exact indicator logic for all 9 scored indicators + ATR context,
  regime classification rules, regime-aware RSI zone tables, and the
  regime-multiplier table copied from _compute_composite (the old
  table did not match the code)
- aggregator pipeline in source order: MTF agreement, the 12 penalties
  and the diminishing cap, exhaustion block, signal/grade thresholds,
  and gates 6b-6i with exact numbers
- all three trade-setup generators with SL/TP geometry, spread gates,
  and the trailing-exit model
- bot/CLI/news/data-provider/backtester mechanics incl. realism model
  (raw-path resolution, next-open entry, SL-first ties, parallel replay)
- "Rules for making changes": validation acceptance rule, pair
  isolation, live==backtest parity, ROC's load-bearing weight

Also corrects details the old doc got wrong: test count is 179 (not
~176), EUR ships off by default, backtest/*.csv are gitignored local
artifacts rather than test-enforced pins.
2026-07-16 09:26:06 -04:00
Harwinder Sekhon 26f0681ad1 Docs: user guide + beginner guide PDFs
Two printable guides in docs/ (HTML source + weasyprint-rendered A4
PDFs): a trader-facing user guide (signal anatomy, trail management,
M5/M15/H1 comparison from the 12mo books, expectations, commands) and
a plain-language beginner edition (glossary, worked sizing example,
step-by-step trailing-stop walkthrough, H1-first guidance).
2026-07-12 13:42:21 -04:00
Harwinder Sekhon dde41fbeb8 Audit fixes: mixed-regime bias, live==backtest session parity, parallel replay
Fix 1 — mixed-"trending" regime (ADX strong, EMAs unaligned) was
hard-coded bullish in 5 sites: RSI reinterpretation used the uptrend
playbook (RSI 38-55 -> STRONG_BUY conf 75), BUY-only confidence
bonuses, BUY-only extension penalties, no counter-trend penalty.
Now: extremes-only RSI (as transitioning), no directional bonuses,
symmetric extension penalties. 6yr validation: H1 +67.2->+61.9R
(mixed skew 16B/11S -> 4B/7S), H4 +31.6->+27.5R (the 4 bias BUYs),
M15 +152.6->+154.7R, M5 +207.1->+206.1R with maxDD 28.7->24.3R.
The ~8R aggregate give-back is bull-market subsidy from the bug.

Fix 2 — live session filter now applies the EXACT replay-loop
predicate (session window + blocked hours on the signal bar's CLOSE
time, from the bar timestamp not wall clock). Replaces a wall-clock
filter gated on a "XAU STRATEGY" marker that no code writes, which
made live diverge from every validated book.

Fix 3 — delete dead _check_xau_oversold_bounce and
_generate_xau_trade_setup (unreachable; the latter hardcoded BUY and
skipped the spread/RR gates).

Backtester: parallel bar replay via forked workers (--workers,
default auto = cores-2 capped 12). Per-bar evaluation is pure; the
entry cooldown is applied in an ordered phase-2 pass. Proven
byte-identical to sequential on the same window; ~300 -> 1030-1680
bars/s (6yr M5: ~26min -> 5-8min). --workers 1 = sequential path.

Tests: 179 passed (new mixed-regime RSI regression test).
2026-07-11 22:57:37 -04:00
Harwinder Sekhon 6f69b9dd90 XAU: two validated per-TF gates — M15 mixed-regime block, M5 hour-9 block
M15 mixed-"trending" regime block (ADX strong, EMAs unaligned =
momentum without direction): -6.8R/24 trades on 12mo, negative both
halves; blocking scores +65.7R/499 PF 1.38 DD 10.4R vs +59.0R/521
PF 1.32 DD 12.6R, better both halves, OOS 2024-06->2025-03 neutral
(-1.4R = the 4 such trades there). M5 never trades this regime.
New XauStrategyConfig.blocked_regimes / sell_blocked_regimes knobs
(default empty) + aggregator gate 6g2; M15 factory opts in.

M5 hour-9 UTC block (post-London-open whipsaw): -28.9R/143 over the
6-year book, negative 5/7 calendar years. Full-history replay with
block: +207.1R vs +178.3R, max DD 28.7R vs 35.5R, 2023 -13.0R ->
breakeven; tuned 12mo book unchanged (139.0 vs 139.3). Hour 9 is
positive on M15 (+1.7R 6yr, +8.6R OOS) so the block is factory
M5-only.

Rejected after validation (documented in xauusd.py + session memory):
transitioning-SELL block (bull-market artifact, +5.7R in 2024 OOS),
M15 trending_down block (M5 cross-TF contradiction), losing-streak
circuit breakers, 30d efficiency-ratio trend gate, and all entry-level
indicator filters (winners == losers at entry on every feature).

Full-history context (Twelve Data 5min starts 2020-03-16): M5 6.3yr
+207R PF ~1.3 post-fix with losing year 2021; M15 +152.6R PF 1.19
DD 49.7R — edge is regime-dependent, chop years are structural.

Tests: 178 passed (new gate test + factory-equivalence updates).
2026-07-11 21:38:42 -04:00
Harwinder Sekhon a2bf80054c Telegram: subscriber channel feed, owner lock, full risk disclaimer
- TELEGRAM_CHANNEL_ID mirrors auto-scan signal + breach alerts to a
  subscriber channel (Stars-subscription monetization; Telegram manages
  membership). Feed keeps running even when the owner's DM auto-scan
  is toggled off.
- _owner_only decorator locks all commands/buttons to TELEGRAM_CHAT_ID;
  others get a refusal pointing at the channel.
- Every signal-bearing message (detailed signal, scan summary, all-TF
  summary, breach alerts) now ends with a not-financial-advice /
  educational-use disclaimer via a single _DISCLAIMER constant.
2026-07-11 18:37:50 -04:00
Harwinder Sekhon 60fb526fab EUR: document closure of calendar + H1-hybrid avenues
Surprise-conditioned entries lose 2-5p at every horizon both years;
avoidance filter marginal for EUR and profit-removing for XAU; the
H1-direction + M15-pullback prototype failed its pre-registered kill
criteria (2024-25 OOS negative). EUR stays informational-only.
2026-07-11 10:26:32 -04:00
Harwinder Sekhon 18abff85a3 XAU M15: adopt 1R minimum targets (per-timeframe, validated)
Finishes PENDING_STEPS Step 1 under the current trailing-exit model.
M15 window 2025-03-30->2026-05-31 ($0.30 spread): +116.6R/460 PF 1.83
vs +105.4R/424 PF 1.82 baseline, better in both halves; 2024-06->
2025-03 OOS +30.3R vs +10.6R control. Same knob on the pinned M5 book
is WORSE (+131.3R/699 vs +139.9R/547) so the override is M15-only in
the xauusd factory, per the pre-registered decision rule. M5 path
unchanged (factory override gated on primary_tf == M15).
2026-07-11 10:22:33 -04:00
Harwinder Sekhon 183f148bb4 Live news gate: stop capping signal confidence near events (XAU default)
Replaying the live danger/caution gates against the validated 12mo XAU
M5 book showed they suppress 25-35 trades worth +4.6..+7.2R/yr — trades
entered near USD releases outperform the book average (+0.475R vs
+0.256R). New NewsConfig.event_risk_caps_signals (default False =
XAU-tuned) keeps EVENT RISK warnings in alerts but no longer guts
confidence; EUR factory re-enables the caps (evidence is gold-specific).
Pinned XAU M5 regression: 547 trades, +139.94R, trade-identical.
Tests: 177 passed.
2026-07-11 10:02:27 -04:00
Harwinder Sekhon 528a308d36 Add calendar research infra: FF scraper + verified event store
Scrapes Forex Factory weekly pages (epoch datelines, actual/forecast/
impact/betterWorse) into backtest/cache/calendar.parquet (gitignored;
rebuild with the scraper). Timestamps verified: NFP/CPI/ECB times exact
across DST, and high-impact event bars average 3.5x ATR with the spike
vanishing at +/-1h shifts. calendar_store provides precise_events and
event_window_mask for screens and counterfactuals.
2026-07-11 09:47:11 -04:00
Harwinder Sekhon 9c72e90ce2 EUR M15/H1 investigation: document no-validated-edge verdict
Five independent tests at 1-pip cost (M15 wide-open both windows,
transitioning-slice OOS, mean-reversion trade sims, forward-return
screens on M15+M5, H1 both windows) — nothing clears cost in both
2024-25 and 2025-26. Root cause: spread is ~10% of M15 ATR on EUR.
EUR stays out of scan_pairs; open avenues documented in docstring.
2026-07-11 09:02:24 -04:00
Harwinder Sekhon 8b086f4f49 EUR M5: commit best-found config after realism re-tune — no validated edge
12mo grid campaign (2025-07-08 → 2026-07-08, 1-pip round-trip cost,
train/val split at 2026-01-08). Donor config scored -6.7R/34 trades.
Findings: transitioning regime -27R/88 wide-open; A-grade entries lose
(-26R) while B-grade wins; Fridays -17R; hours 8/12/13/18/19 UTC bleed;
score 30-35 band holds the train-half losses; trailing (0.45R arm/0.5R
dist) beats static TP1 and gold's 0.6/0.75.

Best-found config: -1.06R / 53 trades / PF 0.93 / 62.3% WR — breakeven,
FAILS acceptance (positive both halves, PF>=1.3). Per plan, EUR ships
functional but stays out of default scan_pairs. Grid results and
methodology documented in the module docstring; full numbers in
backtest/cache/eurv2_*.csv.
2026-07-10 22:14:49 -04:00
Harwinder Sekhon f02f4b4d37 Update CLAUDE.md for multi-pair architecture 2026-07-10 21:16:34 -04:00
Harwinder Sekhon 74829541d6 Live path: config-driven multi-pair scanning (Telegram bot + CLI)
- telegram_bot: SCAN_PAIRS/_WS_TO_PAIR/SCAN_BUTTONS all derive from
  PlatformConfig.scan_pairs (the single pair on/off switch); buttons and
  keyboards are generated per pair; _run_signal_sync builds per-pair
  configs via pair_signals factories (was: XAU defaults for any symbol);
  scan-all works per pair; PB badge matches any *_PB tag; daily-rotating
  file logs in logs/telegram.log (30-day retention)
- main.py: --pair argument (XAUUSD | EURUSD + aliases) routed through
  pair_signals factories; price display is precision-aware (2dp gold,
  5dp FX)

scan_pairs default stays [XAUUSD] until the EUR M5 re-tune validates.
2026-07-10 21:13:42 -04:00
Harwinder Sekhon 6546b89d6b Add EUR/USD test suite + map EUR news events to EUR/USD
Module 10 (21 tests): EUR config/registry/factory, EUR filter pipeline
(regime/quality/score/confidence/RSI-extended/weekend), EUR trade
geometry (swing SL buffers, pip-denominated spread gate, per-pair gate
contrast), and cross-pair isolation regressions (XAU report bit-equal
with eur_strategy flipped; XAU gates never fire on EUR reports).

news_engine: _get_affected_pairs now maps USD events to EUR/USD too and
adds an EUR entry (calendar warnings reach EUR scans).

175/175 pass.
2026-07-10 21:06:55 -04:00
Harwinder Sekhon 9ed07d4b65 Backtester: EUR/USD support with per-pair configs and default spread
- EURUSD aliases in PAIR_ALIASES
- main() builds PlatformConfig via pair_signals.PAIR_PRODUCERS factories
  (per-pair regime weights/gates/sessions); unregistered pairs keep the
  inline construction. XAU factory is asdict-identical to inline
  (enforced by test) so XAU runs are unchanged.
- --spread default now resolves from PAIR_INFO default_spread
  (XAU $0.30, EUR 0.0001) instead of a hardcoded gold value
- EUR strategy tag: renamed EUR_PB→EUR, detected in both bullish and
  bearish reasons[0] (SELL setups mark the bearish list)

Smoke-validated: 30-day EUR M5 run produces EUR-tagged trades with
1-pip round-trip cost charged.
2026-07-10 21:02:44 -04:00
Harwinder Sekhon 0da5ed03af Add EUR/USD signal gates, trade geometry, and weekend/session filters
Re-grafted from feature/eurusd-support onto the evolved aggregator:
- 6h EUR gate chain (regime, quality, exhaustion, EMA200, session,
  score/confidence, volatility, indicator agreement, RSI-extended),
  active only for EUR/USD with eur_strategy.enabled
- _generate_trade_setup: EUR branch reads EurUsdStrategyConfig only
  (min_sl_pips, min_rr, mature-move + quality-scaled TP, sl_atr_floor,
  swing-buffer SL) with a pip-denominated spread gate; adds a
  pip_value<=0 guard; non-EUR branch is arithmetically unchanged
  (sl_swing_buffer=1.0 extraction only)
- EUR weekend/weekday pre-filter in analyze_pair mirroring the XAU block
- "EUR STRATEGY" reasons[0] marker so the backtester tags EUR trades
2026-07-10 20:57:54 -04:00
Harwinder Sekhon db50bf02d7 Gate all XAU-only aggregator logic by pair
The SELL quality gate (6e), SELL volatility gate (6g), M5 noise gates
(6h), ATR trade-setup generator, and M5 pullback archetype are all
gold-tuned but previously keyed only on xau_strategy.enabled or the
timeframe, so they would have silently applied gold constants to any
other pair. Each now also requires display_name == "XAU/USD".

No behavior change for XAU (conditions are tautologically true for the
only registered pair today) — verified by test suite and pinned-window
backtest CSV diff.
2026-07-10 20:53:23 -04:00
Harwinder Sekhon 1bc48915bd Add pair_signals per-pair producer registry
Ported from feature/eurusd-support: PAIR_PRODUCERS registry with
make_config() factories per pair. xauusd.make_config is a pure
PlatformConfig-defaults wrapper (equality with the inline construction
is enforced by test); eurusd.make_config carries the donor branch's
regime weights and gates as a pre-realism baseline.

Adds 4 registry tests. 154/154 pass.
2026-07-10 20:52:06 -04:00
Harwinder Sekhon 9d3729ea73 Add EUR/USD pair constants and EurUsdStrategyConfig
- EURUSD in PAIR_INFO; new per-pair price_precision and default_spread
  fields on both pairs (backtester consumes default_spread when --spread
  is omitted)
- EurUsdStrategyConfig ported from feature/eurusd-support plus EUR-local
  cost knobs (round_trip_cost=1 pip, min_tp_cost_multiple=10,
  min_sl_pips=8) so SignalConfig's gold-dollar values never apply to EUR;
  donor-tuned values are a pre-realism baseline pending re-tune
- PlatformConfig.eur_strategy field; track EUR in news currencies
2026-07-10 20:50:08 -04:00
Harwinder Sekhon f3b90d3862 Fix 3 engine bugs: swing-point NaN false positives, ROC div-by-zero, ADX epsilon
Ported from feature/eurusd-support (e6e09c5), pair-neutral fixes:
- _find_swing_points: all(<empty generator>) is vacuously True, so a value
  surrounded by NaNs was reported as a swing point; now requires at least
  one valid neighbor on each side
- ROC: guard against zero close at the lookback positions (inf scores)
- ADX: epsilon denominator instead of NaN-replace to avoid NaN cascade

Adds 8 regression tests (Module 9). 150/150 pass.
2026-07-10 20:48:56 -04:00
Harwinder Sekhon ea306df484 Overhaul news engine: fix false signals, add surprise detection + caching
Improvements from a live-verified audit of the news/calendar pipeline:

- Currency tagging: keyword matching now uses letter-boundary regexes
  and drops ambiguous "us "/"eu "/"uk " tokens — "versus" no longer
  tags headlines as USD (confirmed live false positive polluting the
  XAU/USD sentiment average).

- Fetch caching: calendar events and headlines are cached class-level
  (10 min TTL, 60s retry after empty/failed fetch, lock-guarded for
  concurrent scans). Stops per-scan refetching that was tripping the
  Forex Factory rate limit; TTL stays under the 30-min pre-event
  buffer so the danger window can't go stale.

- Surprise detection: EconomicEvent.surprise is computed from parsed
  actual vs forecast (handles %, K/M/B/T suffixes, negatives). A
  surprise print widens the post-release caution window to 45 min;
  inline releases keep the original 15 min.

- Gold-aware sentiment: headlines are scored per currency, with
  monetary-policy keywords (hawkish/rate hike/tightening vs dovish/
  rate cut/easing/stimulus) inverted for XAU — "hawkish Fed" headlines
  scored +1.0 for gold before, -1.0 now (verified on live feed).

- New sentiment sources: Google News gold-focused query and CNBC
  Economy RSS (verified live; 40 → 74 unique headlines). Added
  cross-feed title dedup so syndicated stories count once.

- events_today now buckets by UTC date (feed timestamps carry US-
  Eastern offsets); --json output includes news_description.

- Removed dead NewsConfig.min_impact and unused HIGH_IMPACT_EVENTS.

Tests: 142 passed (7 new covering boundary matching, surprise
classification, post-event windows, cache reuse, gold inversion).
2026-07-10 09:43:11 -04:00
harwinder 1038a53bac Add M5 pullback entry archetype (XAU_PB): +140R combined on 12mo
Second entry pattern alongside the Keltner-breakout: enter on the
retrace to EMA21/KC-mid in an established trend (EMA >=3/4 aligned,
ADX>=20, retrace 0.8-6.0 ATR, resumption bar closed back in trend
direction, RSI in the pullback zone not the extended zone). Detector
lives in the aggregator so live and GPU-backtest paths share one
implementation. Own gates (composite floor 15, confirmation-TF
agreement, MTF opposition, vol regimes) with a diagnostic funnel.

PB stops anchor to the pullback swing (~$8 vs ~$25 breakout), so PB
trades get their own trail arm (0.8R vs 0.6R global) via per-trade
trail overrides in the backtester and strategy-aware Telegram alerts.

12mo M5 validation: PB alone +38.3R/254 trades, positive both halves
and all four full quarters; core breakout trades undisturbed
(+101.9R vs +105.0 baseline). Combined book: 549 trades (~10.6/wk),
+140.1R, PF 1.78, DD 11.3R vs 311 trades/+105.0R before.
2026-07-09 14:21:59 -04:00
harwinder 305f1d1857 Add pullback entry pattern implementation plan
Design doc for a second M5 entry archetype: enter on retrace to
EMA21/KC-mid in an established trend instead of at Keltner-breakout
extension. Includes detector spec, geometry, validation protocol
(both-halves + quarterly + displacement checks), and the +105R
baseline to beat. Session/frequency knobs (blocked hours, cooldown)
were tested 2026-07-09 and confirmed already optimal — this is the
remaining evidence-backed entry-side direction.
2026-07-09 13:51:00 -04:00
harwinder 0393087168 Add M5 gates 5-6 and trailing-stop exits (+48R → +105R on 12mo)
Gates 5-6 in the M5 filter: block 'high' vol regime entries (chop
transition — 61% WR, negative total while low/normal/extreme all
positive) and block SELLs with RSI<25 (dip already extended, mean-
reverts before target). 365d: 419→311 trades, +41→+48.4R, DD 11.2→8.0R.

Replace the fixed RR-0.5 take-profit with a trailing stop: arm at
+0.6R favorable excursion, trail 0.75R behind best price, stop levels
update at bar close (conservative). The RR-0.5 target was the binding
constraint — winners kept running after TP. Whole trail parameter grid
(arm 0.4-1.2 × dist 0.5-1.25) lands +90-103R, a plateau not a spike;
also +91.7R on the pre-gate 419-trade population. TP1 stays advisory;
Telegram alerts include the arm price and trail distance. max_hold
200→300 bars (trailing exits need room; +10R).

12mo M5, same 311 trades: +105.0R total, PF 2.27, DD 7.9R, 61% WR,
train/val +54.6/+50.4, every quarter positive.
2026-07-09 13:34:57 -04:00
harwinder 8687f98b39 Fix M5 signal quality: ATR ordering bug, noise gates, blocked-hour leak (+41R)
ATR was computed after EMAs so ribbon compression detection was dead code
on all timeframes — moved ATR first. Extended M5 swing S/R lookback from
50 to 120 bars. Added four M5-specific filters: min score, MTF agreement,
confirmation TF conviction, and SELL quality cap (A-grade M5 shorts enter
at dip extremes and lose). Fixed backtester blocked-hour check to use bar
close time matching the live bot. Added trade geometry config (spread gate,
SL floor, TP floor RR) and primary_tf mismatch guard in backtester.

Backtest: 1133 trades/+11.5R → 419 trades/+41.0R, PF 1.03→1.37, DD 21→11R.
Train/val split positive in both halves (+19.8R/+21.3R).
2026-07-09 12:48:36 -04:00
harwinder 2291fc3936 Disable TP tightening: tuned pre-spread, halves total R under real fills
A/B on M15 2025-03-30→2026-05-31 with $0.30 spread + gap-through fills
(511 identical entries, only exits differ):
  factor 0.8 / min_rr 0.4: +36.1R (avg +0.071R, SELL +1.2R)
  factor 1.0 / min_rr 0.5: +66.8R (avg +0.131R, SELL +9.6R)
Worse in both train and validation halves and both directions. The 20%
tighter targets buy +1.8pt win rate but shrink every win while the fixed
spread stays constant. The knob remains for re-tuning with costs included.
2026-07-08 21:35:59 -04:00
harwinder 3a85c6e23b Add WebSocket dual-connection test script
Probes whether Twelve Data allows two simultaneous WS connections
for one symbol (SDK-based, 30s run, prints per-connection verdict).
2026-07-08 21:32:46 -04:00
harwinder 09ee5208d4 Telegram bot: WS heartbeat, breach alerts, alert dedup, interactive menu
WebSocket:
- Send heartbeat every 10s — Twelve Data drops idle connections, so the
  monitor previously died on weekends/quiet hours and churned reconnects
- Log failed symbol subscriptions from subscribe-status events
- Stamp fresh live tick price + age into every alert message

Alerts:
- Re-enable level-breach alerts: WS breach triggers M15 rescan and sends
  the report when actionable (was scanning and discarding the result)
- Dedup: identical (signal, quality, direction) per pair/TF is sent once,
  re-armed after 1h — no more repeat alerts every candle
- Configurable minimum alert quality (C+/B+/A), cycled from the menu
- Trade setup shows dollar risk/reward alongside pips

Interactive menu (/start, /scan):
- Timeframe scans, Scan All, auto-scan ON/OFF toggle, alert-quality
  cycle, status, and help as inline buttons
- Markdown-safe edits: 'message not modified' swallowed, parse failures
  fall back to plain text; exception text escaped
2026-07-08 21:32:46 -04:00
harwinder 1f076c0b54 Fix live/backtest parity, SELL-side losses, and indicator repairs (+42% total R)
Live pipeline:
- Drop still-forming candle before analysis (signals no longer repaint)
- Direction-aware MTF agreement + opposition gate (no trade against 2-of-3 TF consensus)
- SELL volatility gate: block shorts in low/normal vol regimes (config sell_vol_regimes)
- Divergence: 40-bar lookback, price/indicator swings must align within ±2 bars
  (fired on 0.5% of trades before, 29% after), trending weight 2.0→1.0
- RSI regime_aware by default; transitioning regime only signals at extremes
- ADX ±DM tie no longer nudges bearish; TP2 clamped beyond TP1

Backtester realism:
- Charge $0.30 round-trip spread (--spread), gap-through SL/TP fills at bar open
- Resolve trades on RAW price path (cache stores raw bars, cleaning runs on load)
- Weekend/session filters use bar-close decision time to match live wall clock

Validated per-step on M15 2025-03-30→2026-05-31 under identical pricing:
+46.9R → +66.8R, win rate 69.9% → 73.0%, SELL side -2.3R → +9.6R.
Tested-and-reverted: ROC weight removal (-13R), BB fade suppression (-11.6R),
candlestick de-weight (-2R) — composite thresholds are co-tuned with weight mass.
2026-07-08 21:32:46 -04:00
harwinder baeee4997a Fix trade setup generation broken by TP tightening factor
The TP floor calculations used raw min_rr (0.4) but after 20% tightening
the effective RR dropped to ~0.32, failing the min_rr check and returning
None for all trade setups. Pre-compensate the floor with min_rr/tf so
post-tightening RR still meets the minimum.
2026-05-24 18:22:57 -04:00
harwinder bb40bceb4c Tighten TP targets by 20% for faster hits (configurable tp_tightening_factor=0.8)
Applied uniformly across all 3 trade setup functions. Lowered min_rr_ratio
0.5→0.4 proportionally to preserve trade acceptance rate. SL unchanged.
2026-05-22 10:44:52 -04:00
Harwinder c46bb85012 Enable M5 auto-scan alerts and remove live dashboard
M5 already broadcast through the shared _scan_timeframe path but was
documented as data-only (docstring, CLAUDE.md, DOCUMENTATION.md, plus a
phantom `if tf == M5: return` step that never existed in code). Align all
docs/comments so M5 alerting at C+ grade is the intended, documented behavior.

Remove the live dashboard entirely per request: the 30s refresh loop,
per-scan refresh, boot send, and the dashboard-only stats layer
(_signal_stats / _stats_updated / _update_signal_stats and its call sites,
_format_stats_dashboard and its helper constants). Drop now-unused ZoneInfo
import and dashboard mentions from help/autoscan/boot messages and the test
suite. The WebSocket level-breach monitor and level cache are untouched.

Tests: 136 passed, 0 failed.
2026-05-20 12:45:27 -04:00
harwinder cc53448c34 Fix weekend filter: use precise forex market hours (Fri 22:00–Sun 22:00 UTC)
The old `weekday() >= 5` check blocked all of Sunday, but the forex
market reopens Sunday 22:00 UTC (Sydney open). Now uses proper time-based
check so the bot fetches data and generates signals on Sunday evening.
2026-05-17 19:23:21 -04:00
harwinder 15a7074574 Add backtest scripts to version control
Track gpu_backtest.py, diagnose_filters.py, and instructions.
Data files (CSVs, parquet cache) remain gitignored.
2026-05-15 19:49:30 -04:00
harwinder 35e9b3750d Reduce A-grade TP when exhausted: RR 0.75→0.50 on 2+ exhaustion tags
A-grade trades entered exhausted moves with TP too far (0.75R), causing
57.7% WR and +0.25R over 30 days. Store exhaustion_count on SignalReport
and scale TP to B-level (1.0×) when A-grade has 2+ exhaustion tags.
Backtest: A-grade 73.1% WR / +3.25R, B/C unchanged.
2026-05-15 19:46:31 -04:00
harwinder 68cc5b28a5 Align FilterConfig with XAU strategy: blocked hours [0,1,2], session 3-23 UTC, Mon-Fri 2026-05-15 15:36:03 -04:00
harwinder 5c0adfab27 Add M5 to Telegram scan buttons and Scan All TFs 2026-05-15 15:26:00 -04:00
harwinder b7bfd9c830 Reduce blocked hours, enable M5 alerts, show NY time on dashboard
- Blocked hours reduced to 0-2 UTC only (daily rollover)
- M5 signals now broadcast to Telegram like M15/H1/H4
- Live dashboard shows New York time (auto EDT/EST via zoneinfo)
2026-05-15 15:23:48 -04:00
harwinder 04328eaeab Refocus on XAU/USD: add momentum indicators, tune TP, exhaustion block
Strip EUR/USD, GBP/USD, USD/JPY pairs and strategies from entire codebase.
Add 3 new scored indicators (ROC, Keltner/TTM Squeeze, ADX slope) and
enable MACD-as-pillar + divergence detection for early trend entry.
Introduce "transitioning" regime (ADX < 25, slope > 3.0) to catch trend
births before ADX confirms. Gold-specific composite weights per regime.

Tune TP multipliers: quality-tiered scaling (A+/A get 1.5-2.0x ATR via
XAU ATR method, B/C use swing-level targeting). Backtest: +24R total,
65% WR, PF 1.12 over 365 days.

Add momentum-confirmed exhaustion block: hard-reject trades when 2+
static exhaustion signals fire AND ROC/MACD confirm fading momentum.
Prevents late trend entries without killing strong trend trades.

Telegram: simplify buttons to timeframe-only labels, add "Scan All TFs"
multi-timeframe summary, add /scan command, fix BB/KC level breach keys,
fix price formatting (.2f for gold), fix deprecated asyncio call, fix
"1 pairs" grammar and 60s/30s dashboard refresh mismatch.
2026-05-15 10:51:51 -04:00
harwinder bb3c475f22 Add web dashboard (Redis + FastAPI + WebSocket), scope XAU penalties
Signal daemon (signal_web.py): candle-boundary scanner mirroring
telegram_bot.py, publishes serialized SignalReport JSON to Redis.
Web API (web/app.py): thin FastAPI reader behind Apache, serves
/api/signals REST + /ws WebSocket from Redis pub/sub.
Frontend (web/static/index.html): 3-pair grid, color-coded signals,
indicator breakdown per timeframe, trade setup display.

Also: XAU-specific penalty scoping in signal_aggregator (softer
confidence penalties for gold, original strict values for forex),
pair-aware session filter routing XAU to its own blocked hours.
2026-05-14 23:15:37 -04:00
harwinder 31ecdf4292 Fix news engine (keywords, calendar fallback, RSS URL), remove Telegram guards
News engine: expand sentiment keywords (18%→60% headline coverage), add
MyFXBook RSS as fallback calendar source when Forex Factory returns 429,
add retry/backoff on calendar API rate limits, update stale ForexLive URL
to investinglive.com, add date formats without seconds for MyFXBook RSS.

Telegram bot: remove flood control, rate limiter, and dashboard failure
tracking — all sends go directly through bot.send_message/edit_message_text.
2026-05-14 13:42:32 -04:00
harwinder 39c91c1aa4 Fix duplicate-index crash in data validation, add Telegram flood control
- Move index dedup before OHLC consistency fix in _validate_and_clean
  to prevent crash when df.loc[dup_idx] returns Series instead of scalar
- Add global flood backoff: detect RetryAfter, pause all sends/edits
  until ban expires instead of retrying and escalating the ban
- Stop dashboard edit-fail → new-message spam cascade: track consecutive
  failures, only resend after 3 failed edits
- Increase dashboard refresh from 5s to 30s, min send interval to 1.5s
- Add comprehensive QA test suite (140 tests across all 7 modules)
2026-05-14 13:04:28 -04:00
harwinder cd69980f1f Harden Telegram bot, add EUR/GBP strategies, parallelize scans
Telegram reliability: rate-limited sends (1 msg/s), dashboard lock,
WebSocket JSON/price error handling, markdown escaping, level cache
TTL, cooldown cleanup, breach task tracking, graceful shutdown.

Telegram speed: parallelize 3-pair scans with asyncio.gather(),
increase thread pool 2→4, parallelize boot-up initial scan.

Strategy: add EUR/USD and GBP/USD pullback-buy strategies with
per-pair configs, optimize XAU TP to 0.9×ATR, block hours 8/9
for XAU and hour 17 for GBP.

Backtest: add --max-hours flag to evaluate.py for time-exit support.
2026-05-13 21:07:49 -04:00
Harwinder 67a9169eb3 Fix dashboard TF key mismatch, increase refresh to 5s
Timeframe keys stored as enum values (15min/1h/4h) but dashboard
looked for M15/1h/4h — GBP/USD showed no price or signal text.
Now uses Timeframe enum directly with display labels. Reduced
dashboard refresh from 60s to 5s with throttle guard.
2026-05-13 18:51:02 -04:00
Harwinder f99fcc60b9 Add live dashboard with 60s auto-refresh, disable breakout alerts
Replace per-scan Telegram messages with a single dashboard message
that edits in-place every 60 seconds. Each pair shows all TFs
horizontally with short signal labels (Buy/Sell/Neutral). WebSocket
live prices feed into the dashboard. Level-breach alerts disabled
(WS still updates cache silently). Removed /stats command.
2026-05-13 18:34:44 -04:00
harwinder 4d3f05e2ba Fix live session filter overriding XAU strategy signals
The general session filter ran after analyze_from_dataframes() and
wiped out XAU oversold bounce signals at most hours. Now skips the
session filter when the XAU strategy has already fired — the strategy
has its own backtested hour filter.
2026-05-13 17:31:46 -04:00
harwinder 2deea1ed7f Add XAU/USD oversold-bounce strategy (79.7% WR, PF 2.06)
BUY-only strategy for gold: enters when M15 RSI drops below oversold
threshold while price remains above EMA200 on all three timeframes
(M15, H1, H4). Uses ATR-based TP/SL (0.7x/2.0x) sized from M15 ATR.
Isolated to XAU/USD only — other pairs completely unaffected.
2026-05-13 17:25:49 -04:00
harwinder 89030c96b6 Backtest tuning, data freshness guard, Telegram improvements
- Drop USD/JPY from scan pairs (zero signals across all timeframes)
- Disable divergence indicator (zero impact in 14-config lab test)
- Parallelize evaluate.py with ProcessPoolExecutor (16 cores)
- Add --pair CLI filter to evaluate.py for single-pair backtests
- Add data freshness guard: stale cache auto-invalidated, primary TF
  stale data suppresses signals to prevent trading on old data
- Add /cache command: shows fetch age and last bar per pair/TF
- Add /clearcache command: wipes in-memory cache for fresh API fetch
- Remove BB from WebSocket breach alerts (keep structural levels only)
- Show live prices only for WS-monitored pairs (EUR/USD, XAU/USD)
- Replace arrow emojis with text tags in indicator lines
2026-05-12 22:56:55 -04:00
harwinder 59a18b8bc6 Add project documentation and MT5 integration plan
DOCUMENTATION.md: Two-section guide covering the full project —
plain-English trader guide (indicators, scoring, trade setups) and
developer reference (architecture, data flow, every threshold/weight).

mt5.md: Architecture plan for automated order execution via MT5 bridge
(REST API on Windows, async client on Linux, risk-based lot sizing).
2026-05-11 19:50:21 -04:00
harwinder 49260abaa2 Fix trade setup generation, M5 data-only mode, WS symbol limits, command menu
- Trade setup TP now scales to SL distance ensuring min RR is met when
  no viable swing target exists; skip swing targets with poor RR
- Add float tolerance to RR gate to prevent edge-case rejection
- M5 auto-scan runs for level cache only, no Telegram broadcast
- WebSocket limited to EUR/USD + XAU/USD (Grow plan constraint)
- Register bot command menu via set_my_commands on startup
2026-05-11 18:58:40 -04:00
harwinder 4ce09c2f79 Suppress Telegram messages when no actionable signals found
Auto-scan still runs and updates the level cache every cycle,
but only sends to Telegram when quality A+/A/B/C signals exist.
2026-05-11 18:03:49 -04:00
harwinder 02eedcc875 QA fixes: H4 timing bug, /status + /help commands, WS cold-start
- Fix H4 boundary calculation using total minutes (hour*60+min) not just minute
- Add /help command listing all bot commands
- Add /status command showing WS connection, live prices, level cache, cooldowns
- Run initial M15 scan on boot to populate level cache before WS starts
- Update /start message to mention new commands
2026-05-11 17:16:07 -04:00
harwinder 8ead05432e Tune engine from backtesting: wider SL, capped TP, counter-trend filter
Backtest-driven fixes (0% win → 33% win, PF 0.00 → 1.05 over 7 days):
- SL: swing buffer 0.5→1.0 ATR, min distance 1.0→1.5 ATR (more room to breathe)
- TP: cap swing targets at ATR×3.0, fallback ATR×2.0 (was targeting unreachable levels)
- Mature-move TP: when score>30 (A/B grade), tighten TP to ATR×1.5 (move already extended)
- Counter-trend penalty: -15 confidence when signal fights regime direction
- EMA200 penalties halved: -20→-10 (was killing EUR/USD signals)
- Score thresholds lowered: BUY/SELL ±15→±12, STRONG ±40→±35
- Quality grades relaxed, C-grade now gets trade setups and Telegram alerts
- min_rr_ratio: 1.5→1.2
- Gitignore backtest/ directory
2026-05-11 17:00:12 -04:00
harwinder 90b7e0c54e Buy signal included 2026-05-11 16:37:45 -04:00
harwinder 058ef57d8f Pro tier upgrade: quant rearchitect + WebSocket level-breach monitor
- Restore XAU/USD (Pro plan), drop USD/CAD, raise rate limit to 55/min
- Reduce signal engine to 4 uncorrelated pillars (EMA, RSI, BB, Candlestick)
  with regime-adaptive weights (trending boosts EMA, ranging boosts RSI+BB)
- Add structure-based S/R (swing highs/lows) for trade setup targeting
- Fix confidence death spiral: all adjustments now additive, not multiplicative
- Dynamic R:R trade setup: SL behind swing level, TP at next structure level,
  reject if R:R < 1.5
- Fix news engine for all 4 pairs (was hardcoded to USDJPY)
- Add WebSocket monitor: streams live ticks, triggers M15 scan on level breach
  with 5-min cooldown per pair — catches breakouts between candle boundaries
- Add correlation filter (EUR/USD+GBP/USD, XAU/USD+USD/JPY exposure warnings)
2026-05-11 16:33:23 -04:00
harwinder cc66e65f2d Replace XAU/USD with USD/CAD — TwelveData has no gold data
XAU/USD is not available in TwelveData's symbol list.
Swapped to USD/CAD across config, scan pairs, Telegram buttons,
and added CAD to tracked news currencies.
2026-05-11 15:50:48 -04:00
harwinder 873cf5487e Add 4-pair candle-boundary auto-scanner (free tier safe)
Expand from 2 to 4 pairs (EUR/USD, USD/JPY, GBP/USD, XAU/USD) with
automatic scanning at M15 and H1 candle close boundaries. Scheduler
staggers H1 by 60s to stay within 8 credits/min; daily budget is
480 of 800 allowed. Cache TTL is now candle-duration-aware so cross-TF
confirmation data is reused without extra API calls. Spread estimate
uses primary-TF data instead of fetching M5 separately.
2026-05-11 15:30:11 -04:00
Harwinder a708609f8b Add EUR/USD support and 4-button Telegram scanner
Add EURUSD pair with dynamic pair info lookup. Telegram bot now shows
4 scan buttons: EUR/USD 15M, EUR/USD 1H, USD/JPY 15M, USD/JPY 1H.
Each indicator displays directional arrows in the message output.
Price formatting adapts per pair (5dp for EUR, 3dp for JPY).
2026-05-10 23:49:36 -04:00
Harwinder 7ccd8c4b04 Strip backtesting, rebalance indicator weights toward leading signals
Remove backtest.py, backtest_gpu.py, metrics.py, local CSV data,
Dukascopy loader, and all backtest CLI modes from main.py. Rebalance
engine weights: leading indicators (divergence, RSI, stochastic, BB)
now carry 66% of signal weight vs 34% for lagging (EMA, MACD). Add
ws_twelvedata.py for live tick streaming. Drop pyarrow and tqdm deps.
2026-05-10 23:40:15 -04:00
Harwinder 2e09237671 Add USAGE_LINUX.md — bot and CLI run instructions for Linux server 2026-05-07 23:12:55 -04:00
Harwinder a12084537d Add async Telegram bot — live scan button with trade setup
Introduces telegram_bot.py: python-telegram-bot v20+ async bot that
presents a persistent Scan Signal inline button. Each press fetches a
live USD/JPY signal via SignalAggregator and formats it with emoji,
MTF breakdown, top bull/bear reasons, and warnings. STRONG_BUY /
STRONG_SELL signals additionally render the full trade setup (entry,
SL, TP1, TP2, RR). Adds python-telegram-bot>=20.0 to requirements.txt.
2026-05-07 23:05:01 -04:00
Harwinder a55aca21b1 Round-2 weight optimisation — reduce EMA trend, boost stochastic
GPU sweep (36 combos, RTX 4070 SUPER) on H4 2022-2026 relative to the
round-1 baseline revealed EMA trend was over-weighted at 3.6. Halving
it and nudging stochastic up yields cleaner, higher-quality signals.

Weight changes (backtest_gpu.py _BASE_W + engine.py):
  ema_trend:  3.6 -> 1.8  (over-weighted; halving tightens signal filter)
  stochastic: 1.6 -> 2.4  (x1.5; improved WR to 40.4% in sweep)
  macd:       3.3          (unchanged — dominant signal, kept as anchor)
  adx/ichimoku/rsi/bollinger: unchanged

Combined result vs round-1 (H4 local data 2022-2026):
  Trades:      160 -> 73   (-87 — filters out marginal setups)
  Win Rate:   38.1% -> 43.8%  (+5.7pp)
  Profit Factor: 1.19 -> 1.52
  Return:    +21.30% -> +23.86%  (+2.57pp)
  Max DD:     -8.52% -> -9.32%
  Sharpe:      0.47  ->  0.71   (+51%)
  Sortino:     1.91  ->  3.06   (+60%)

Fewer trades at much higher quality. The lower EMA weight means only
strong, multi-indicator confluences clear the grade threshold — the
73 remaining trades have 43.8% WR vs 38.1% and expectancy nearly 3x.
2026-05-07 22:24:48 -04:00
Harwinder e57f4a1ecd Enable GPU scoring via CuPy — RTX 4070 SUPER confirmed 6.3x speedup
Install nvidia pip packages for CUDA 12 runtime DLLs (no full toolkit needed):
  nvidia-cuda-nvrtc-cu12, nvidia-cuda-runtime-cu12,
  nvidia-curand-cu12, nvidia-cublas-cu12

Add _register_nvidia_dlls() in backtest_gpu.py that auto-discovers and
registers nvidia package DLL directories (site-packages/nvidia/*/bin/)
via os.add_dll_directory() before CuPy is imported. Suppresses the
spurious "CUDA path could not be detected" warning from cupy._environment.

Backend line now reads: CuPy 14.0.1 (RTX GPU)
Benchmark: scoring pass on 100k bars — NumPy 1.1ms vs CuPy 0.18ms = 6.3x.
Results are identical to CPU path (deterministic).
2026-05-07 22:22:21 -04:00
Harwinder 17cb63e86e GPU backtester, optimised weights, local CSV data pipeline
## What changed

### backtest_gpu.py (new — ~1,300 lines)
Standalone GPU-accelerated backtester completely separate from backtest.py.
Computes all indicator series in one vectorised O(N) pass over the full price
history, then replays positions with a sequential state machine.

Key design:
- VectorIndicators: pandas ewm/rolling (C-optimised) for all 7 indicators
- CuPy GPU backend with automatic NumPy fallback (no hard dependency)
- Two-phase API: prepare_arrays() once, score_and_run() many times
- --optimize flag: 36-combo weight sweep (1 baseline + 7 ablations + 28
  multiplier runs) — calls prepare_arrays() once, replays all combos in seconds
- Full CLI: --tf, --local, --start/--end, --regime-filter, --regime-tf,
  --session-filter, --optimize, --journal, --journal-csv

O(N) vs O(N^2): backtest.py recomputes all indicators per bar (O(N^2));
backtest_gpu.py computes each indicator series once and reads bar T in O(1).
On 100,000 H1 bars: ~4 seconds vs >10 minutes.

### data_provider.py
Added load_local_csv() method — reads MetaTrader tab-delimited CSV exports
from market_data/ directory. Format: Time|Open|High|Low|Close|Volume|Spread
(spread column dropped on load, timestamps localised to UTC). Applies the same
_validate_and_clean() pipeline as the Twelve Data path. LOCAL_CSV_MAP maps
each Timeframe enum to its filename.

market_data/ files added (MetaTrader exports, UTC):
  USDJPY_M15.csv  100,000 bars  2022-05-04 to 2026-05-08
  USDJPY_H1.csv   100,000 bars  2010-04-23 to 2026-05-08
  USDJPY_H4.csv    25,855 bars  2010-04-22 to 2026-05-08
  USDJPY_D1.csv     5,020 bars  2010-04-22 to 2026-05-08

16 years of H1/H4/D1 vs ~6 weeks of H1 on Twelve Data free plan. Use --local
for all backtests; Twelve Data reserved for live signal feed.

### engine.py — optimised indicator weights
36-combo sweep on H4 2022-2026 found MACD and EMA trend are the dominant
signals; RSI and Bollinger are net drag on this trending pair.

Old → new weights:
  ema_trend:  1.2 → 3.6  (×3 — strongest directional signal)
  macd:       1.1 → 3.3  (×3 — ablating it is the single worst move: -42%)
  stochastic: 0.8 → 1.6  (×2 — clean timing, 40% WR at this level)
  adx:        1.0 → 0.5  (÷2 — partially redundant with EMA stack)
  ichimoku:   1.1 → 0.55 (÷2 — lags too much at H4, hurts in ranging)
  rsi:        0.3 → 0.0  (ablated — net drag across all test windows)
  bollinger:  0.9 → 0.0  (ablated — counter-trend noise on trending pair)

RSI and Bollinger are still computed and displayed in live output for
reference; weight=0 means they contribute nothing to the composite score.

Combined result vs baseline (H4 2022-2026, local data):
  Return:   +7.12% → +21.30%  (+14.2pp)
  Max DD:   -9.05% → -8.52%   (tighter — gains did not come at expense of risk)
  Sharpe:    0.23  →  0.47    (2x improvement)
  Sortino:   1.38  →  1.91
  Win Rate: 36.3%  → 38.1%

### backtest.py
Added 16-thread CPU parallelism for signal pre-computation via
ThreadPoolExecutor. _precompute_all_signals() dispatches each bar to a worker;
pandas/NumPy C extensions release the GIL enabling true parallel execution.
Results are collected and replayed in the main loop, replacing the per-bar
recompute. Also added --regime-tf CLI flag so regime filter timeframe can be
specified independently of the primary TF.

### signal_aggregator.py
EMA200 distance gate: confidence *= 0.55 when price is within 0.5% of EMA200
(chop zone) or beyond 1.5% (overextended). Sweet spot is 0.5-1.5% from EMA200
where trending momentum is cleanest. Strongest single filter found: lifted
cumulative return +13.88% to +35.50% in ablation window.

### ENGINE_LOGIC.md (new)
Full reference document covering: architecture, data pipeline, all indicator
logic, composite scoring formula, optimised weights with rationale, regime
classification, EMA200 gate, MTF alignment, grade system, backtesting results,
filter catalogue, and configuration reference.
2026-05-07 22:12:43 -04:00
Harwinder 5a392a578d Add EMA200 distance gate — strongest single signal-quality filter
Across 2024 trend / 2023 chop / 2026 hostile windows, distance from the
H1 EMA200 was empirically the most discriminative variable. Sweet spot
0.5-1.5% from EMA200; closer = false-breakout chop, farther = chase.

Outside the band, confidence is multiplied by 0.55 so the trade falls
below the B-grade threshold and is rejected.

Cumulative return across the 3 windows: +13.88% baseline -> +35.50%.
Win rate improved in every regime; 2023 chop turned positive (-2.56% to
+14.94%).

Also extends the trade journal with per-timeframe MTF scores, RSI at
entry, and EMA200 distance for future filter mining; surfaces ema_trend
in key_levels so the gate can read it.
2026-05-07 19:28:19 -04:00
Harwinder be42736223 Document session filter and auto-toggle flags in USAGE 2026-05-07 18:44:41 -04:00
Harwinder 2e5d66db1b Add session filter + auto-toggle, extend trade journal diagnostics
Session filter restricts trades to a UTC hour window (default 07-16,
London + NY) — opt-in via --session-filter, with --session-start /
--session-end overrides. Wired into both the backtester (cheap hour
check next to the regime gate) and live mode (downgrades signal to
NEUTRAL/F when current UTC hour is outside the window).

Auto-toggle (--auto-session-filter) tracks rolling mean pnl_pct over
the last N closed trades; when it falls below threshold, the session
filter activates, releasing once expectancy recovers. Default lookback
10 trades, threshold -0.3%.

Trade dataclass extended with diagnostic fields captured at entry:
entry_hour, entry_dow, ADX, ATR%, market_regime, volatility_regime,
mtf_agreement. These let --journal-csv exports surface loss-pattern
clusters when a window underperforms.

Validated on 2024 trend / 2023 chop / 2026 recent windows. Auto-toggle
preserves ~50% of trend edge while still cutting ~25% of recent-regime
losses — pure regime filter is still best on cumulative, so both
session flags ship default-off.
2026-05-07 18:38:49 -04:00
Harwinder fb1eb5b73f Swap raw requests for Twelve Data Python SDK
data_provider now uses TDClient.time_series().as_pandas() instead of
hand-rolled REST calls. Throttle, validation, and parquet caching are
unchanged. Adds twelvedata>=1.2.0 to requirements.txt.
2026-05-07 18:38:38 -04:00
Harwinder 28a00b7da0 Compact hierarchical live signal output
Restructure print_signal_report so decision (signal · grade · score ·
confidence · MTF) sits at the top, followed by trade setup, context line,
MTF breakdown, top supporting/opposing reasons in two columns, warnings,
and an optional detail block under --detail. Adds _compact_reason helper
that strips known indicator name prefixes via _INDICATOR_PREFIXES so
reason lines stay short.
2026-05-07 17:53:50 -04:00
Harwinder 3e078cfe7d Wire regime filter into live mode
--regime-filter previously affected only backtests. Live mode now also
checks higher-TF ADX after analysis: signals are downgraded to NEUTRAL
with a warning when ADX < threshold, and confirming when above.

This brings live behavior into parity with what the backtester measures.
2026-05-07 17:40:39 -04:00
Harwinder ce76684a62 Add indicator ablation framework + apply ablation-driven cleanup
Framework:
- TechnicalConfig.disabled_indicators skips named indicators in engine.analyze
- Optimizer supports disable_indicator override for parallel ablation sweeps
- main.py --ablate runs leave-one-out across all 10 directional indicators

Cleanup applied (validated across 2024 trend / 2023 chop / 2026 recent):
- Drop pivot by default (cumulative +9.5% to disable)
- Drop candlestick by default (+3.4% cumulative)
- Regime-gate ichimoku: 1.3x in trending, 0.2x in ranging (was a chop villain)
- Trim RSI weight 1.0 -> 0.3 (cumulative noise across regimes)

Result: +10.60% on 2024 trend (+33.24% -> +43.84%, Sharpe 2.17 -> 2.77),
+15.71% on 2023 chop (-18.27% -> -2.56%). Recent regime still adverse.
2026-05-07 17:33:07 -04:00
Harwinder 045f4248f3 Refocus on USD/JPY and add walk-forward backtester with regime filter
- Strip multi-pair plumbing; drop synthetic-data fallback
- Add BacktestConfig + walk-forward replay through the live signal pipeline
- Add metrics module (PF, Sharpe, Sortino, max DD, trade journal)
- Add parallel grid optimizer with disk parquet cache
- Add H4 ADX regime gate (--regime-filter, --regime-min-adx) — cuts chop
  losses ~84% in 2023 H1 while preserving trend edge
2026-05-07 17:15:41 -04:00
Harwinder c04f451c0c Remove redundant script.txt and tracked __pycache__
script.txt was superseded by requirements.txt. Python bytecode should
not be version-controlled — added .gitignore to keep cache and venv
out of the repo going forward.
2026-05-06 22:25:43 -04:00
Harwinder 0a374cfea7 Add requirements.txt and usage guide 2026-05-04 10:51:03 -04:00
5 changed files with 369 additions and 29 deletions
+43 -2
View File
@@ -77,6 +77,13 @@ geopolitical stress / weak dollar are gold-bullish.
_TAKE_CONTRACT = """\
Your review is advisory research on an already-generated signal; you are
one voice on a panel, not the decision maker.
Hard rules: use ONLY the data in the JSON you receive — do not invent
numbers, events, or price levels. If the evidence your role needs is
thin or missing, answer "neutral" with confidence below 30, unless your
role's instructions say otherwise.
Confidence scale (0-100): 80-100 strong one-sided evidence; 50-79 solid
case with some doubt; 30-49 mixed evidence; below 30 little basis to
judge.
Write the note in plain everyday English for someone who has never
traded: short simple sentences, no trading jargon, no acronyms, no
abbreviations. Say "profit versus loss", not "RR"; "the 4-hour chart",
@@ -100,10 +107,19 @@ number is out), a news "sentiment" score for gold, and an optional
Judge ONE thing: does the macro backdrop support the proposed direction?
Weigh released surprises over stale headlines; an imminent unreleased
high-impact event is a reason for lower confidence, not a direction.
Stance rules:
- support: the released data and news flow clearly push gold in the
proposed direction.
- oppose: they clearly push gold the other way.
- neutral: the picture is mixed or stale — or there are no meaningful
headlines or events to judge (score that low confidence).
""" + _GOLD_DOCTRINE + """\
Example: brief direction BUY, headlines dominated by hawkish Fed
repricing and a hot CPI print -> {"stance": "oppose", "confidence": 70,
"note": "Inflation is running hot, so interest rates should stay high — that usually pushes gold down."}
Example: brief direction BUY, headlines led by an escalating conflict
and a soft jobs report -> {"stance": "support", "confidence": 75,
"note": "Worries about war and a cooling economy usually send money into gold."}
""" + _TAKE_CONTRACT
_TECH_SYSTEM = """\
@@ -118,9 +134,18 @@ proposed direction? Look for indicators fighting each other, the signal
fighting the regime (e.g. a BUY in trending_down), higher timeframes
disagreeing, price pressed into a key level, or an exhausted move being
chased.
Stance rules:
- support: the indicators, higher timeframes, and regime mostly agree
with the proposed direction and none of the conflicts above appear.
- oppose: one or more of those conflicts is present and serious.
- neutral: the evidence genuinely cuts both ways, or too much of it is
missing to judge (score that low confidence).
Example: brief direction SELL in regime trending_up with H1 and H4 both
BUY -> {"stance": "oppose", "confidence": 80,
"note": "This bets on gold falling while the bigger picture still points up."}
Example: brief direction BUY in regime trending_up with M15 and H1 both
BUY and exhaustion 0 -> {"stance": "support", "confidence": 70,
"note": "Every way of looking at the chart points up, so buying goes with the flow."}
""" + _TAKE_CONTRACT
_RISK_SYSTEM = """\
@@ -133,11 +158,23 @@ Judge ONE thing: is this setup's risk acceptable as constructed? Check
reward-to-risk adequacy, stop distance versus the volatility regime
(a tight stop in extreme vol dies to noise; a huge stop in low vol is
oversized), spread as a fraction of the stop, and event landmines about
to hit. A missing setup means the signal graded too low for one —
oppose unless the rest is exceptional.
to hit.
Stance rules:
- support: the likely reward clearly justifies the risk, the stop fits
the current volatility, and no event landmine is imminent.
- oppose: reward too thin for the risk, a stop mismatched to the
volatility, spread eating a big share of the stop, an imminent
high-impact event — or no setup at all (a missing setup means the
signal graded too low for one; oppose unless the rest is
exceptional).
- neutral: acceptable but unremarkable — nothing dangerous, nothing
attractive.
Example: RR 1.4 with volatility extreme and a high-impact event due ->
{"stance": "oppose", "confidence": 65,
"note": "The possible profit is small for such wild price swings, and big economic news is due soon."}
Example: RR 2.4 with normal volatility and no events due ->
{"stance": "support", "confidence": 70,
"note": "The possible profit is more than double the possible loss, and conditions look calm."}
""" + _TAKE_CONTRACT
_CONTRARIAN_SYSTEM = """\
@@ -164,6 +201,10 @@ Weigh the takes into ONE consensus on whether to act on the signal:
or "NEUTRAL" (genuinely split or low conviction). Weigh a high-
confidence oppose from any single analyst seriously — one strong veto
can outweigh two lukewarm supports. Do not average blindly.
Base the verdict and rationale ONLY on the takes given — do not add
outside analysis.
Confidence scale (0-100): 80-100 the panel clearly leans one way; 50-79
a lean with real dissent; below 50 split or low conviction.
Write the rationale in plain everyday English for someone who has never
traded: short simple sentences, no trading jargon, no acronyms, no
abbreviations.
+3
View File
@@ -166,6 +166,9 @@ class NewsConfig:
# Must stay below pre_event_buffer_minutes so an event can't cross
# into the danger window while a stale cache says it's clear.
context_cache_ttl_seconds: int = 600
# Maximum age of RSS headlines accepted for scoring. Older headlines
# are discarded before keyword analysis and LLM enrichment.
headline_max_age_seconds: int = 43200 # 12 hours
rss_feeds: List[str] = field(default_factory=lambda: [
"https://investinglive.com/feed/",
+252
View File
@@ -0,0 +1,252 @@
================================================================================
MULTI-PROVIDER LLM LAYER - PLAN
(DeepSeek for news summary, Anthropic Claude for agent panel)
Written 2026-08-08. Plan only - no code has been changed.
================================================================================
WHY
---
Today every LLM feature (news enrichment + agent panel) goes through one
hardwired client: llm_client.py -> DeepSeekClient, a module singleton reached
via get_llm_client(). This plan describes how to route different features to
different providers - e.g. keep DeepSeek for cheap high-volume news scoring,
put the agent panel on Anthropic Claude for better reasoning.
The repo already solved this problem once for market data: the providers/
package (BaseDataProvider owns shared machinery, subclasses implement only
_fetch_raw, a registry + one config knob selects the provider). This plan
mirrors that proven pattern at the LLM layer, scaled down to a single module.
BLAST RADIUS (verified by grep)
-------------------------------
get_llm_client() call sites:
- news_engine.py ....... 4 sites
- agent_panel.py ....... 3 sites (+1 timeout getattr)
- telegram_bot.py ...... 1 site (/status health; reads private _count,
_count_day, _daily_cap fields)
- test_all.py .......... 41 locations stub via lc._client = _StubLLM(...)
Out of scope: static/news_summary.php (own PHP DeepSeek call + key file,
separate deployment).
DESIGN INVARIANTS (carried over unchanged)
------------------------------------------
1. Fail-safe contract: every call returns Optional[Dict]; any failure -> None;
callers keep their non-LLM fallbacks; LLM output never gates signals.
2. Single attempt, no retries (news cache cadence is the retry); per-provider
UTC daily cap; warn-once logging; health stamps (last_ok, last_error).
3. Static system prompts stay byte-identical constants - provider-neutral,
and they map to prompt caching on BOTH providers (DeepSeek prefix cache;
Anthropic cache_control ephemeral).
4. Test compatibility: all 41 existing lc._client stub sites keep working
with zero edits.
================================================================================
1. llm_client.py REFACTOR (~195 -> ~430 lines, no new files)
================================================================================
1a. BaseLLMClient - shared machinery (extracted from today's DeepSeekClient)
- __init__(api_key, model, timeout, daily_cap, enabled)
- public: name ("deepseek"/"anthropic"), timeout property,
usage() -> (used_today, cap) # so telegram_bot stops reading privates
- owns: available property, _warn_once, _under_cap() (UTC-day counter,
lock), health stamps, and the template method:
def chat_json(self, system_prompt, user_payload):
if not self.available or not self._under_cap():
return None
try:
return self._request(system_prompt, user_payload) # hook
except <provider-agnostic error mapping>:
# stamps last_error, returns None
- subclasses implement ONLY _request(system_prompt, payload) -> dict|None.
1b. DeepSeekClient(BaseLLMClient) - behavior-identical move
Current requests-based logic relocates into _request: same URL,
thinking:disabled, response_format:json_object, temperature 0,
max_tokens 2000, sort_keys=True payload, _parse_response.
Review criterion: Module 19 tests pass UNMODIFIED (they patch
client._session, which stays).
1c. AnthropicClient(BaseLLMClient) - new, official SDK
- Official `anthropic` Python SDK (add to requirements.txt), NOT raw
requests. LAZY IMPORT inside __init__: package missing -> warn-once +
client unavailable, so DeepSeek-only deployments never install it.
- Constructor: anthropic.Anthropic(api_key=..., timeout=<configured>,
max_retries=0) # house rule: single attempt
- _request:
resp = self._client.messages.create(
model=self._model, # default "claude-opus-5"
max_tokens=4000, # covers thinking + JSON reply
output_config={"effort": "low"}, # cheap structured takes
system=[{"type": "text", "text": system_prompt,
"cache_control": {"type": "ephemeral"}}],
messages=[{"role": "user",
"content": json.dumps(user_payload, sort_keys=True)}],
)
if resp.stop_reason == "refusal": # Opus 5: HTTP 200, check first
return None
text = first text block in resp.content
return self._parse_json(text) # tolerant: strip ```json fences,
# json.loads, dict check
- DELIBERATELY ABSENT (would 400 on Claude Opus 5): temperature (removed
on Opus 5 - do NOT copy from the DeepSeek client), budget_tokens,
prefills, response_format.
- Error mapping: APIConnectionError -> "network",
APIStatusError -> "http_<code>", parse fail -> "parse"
(same last_error kinds the /status display already understands).
- Defaults: model claude-opus-5 (config-overridable, e.g. claude-haiku-4-5
for ~5x cheaper), timeout 30s (Claude slower than V4-Flash), own
daily_request_cap 500.
1d. Registry + per-consumer routing
_PROVIDERS = {"deepseek": DeepSeekClient, "anthropic": AnthropicClient}
LLM_CONSUMERS = ("news", "agent_panel")
_clients = {} # keyed by PROVIDER name -> two consumers on the
# same provider share one instance and one cap
_client = None # kept: GLOBAL TEST OVERRIDE
def get_llm_client(consumer="news"):
if _client is not None: # tests: lc._client = stub
return _client # -> shadows ALL consumers
name = get_app_config().llm_routing.get(consumer, "deepseek")
create-once per provider, return _clients[name]
def reset_llm_client(): # clears _client AND _clients
Keeping _client as a documented test-override global is what makes the
41 existing stub sites work untouched.
================================================================================
2. CONFIG (app_config.py + config.json.example)
================================================================================
- New top-level "anthropic" block, mirroring "deepseek":
{api_key, model, enabled, timeout_seconds, daily_request_cap}
plus anthropic_api_key() resolver: ANTHROPIC_API_KEY env -> config.json -> "".
- New top-level "llm" ROUTING block:
{"news": "deepseek", "agent_panel": "deepseek"}
Validation: keys within {news, agent_panel}; values within registered
provider names. Absent block -> {} -> everything on DeepSeek (today's
behavior; zero-migration default). AppConfig.llm_routing: Dict[str, str].
- The target state is then PURE CONFIG - no code changes on switch day:
"llm": { "news": "deepseek", "agent_panel": "anthropic" },
"anthropic": { "api_key": "", "model": "claude-opus-5", "enabled": true,
"timeout_seconds": 30, "daily_request_cap": 500 }
================================================================================
3. CONSUMER EDITS (mechanical, ~10 lines total)
================================================================================
- news_engine.py: 4 call sites -> get_llm_client("news")
- agent_panel.py: 3 call sites -> get_llm_client("agent_panel");
getattr(client, "_timeout", ...) -> public client.timeout
- telegram_bot.py _news_health_lines: iterate the DISTINCT providers actually
routed (news + panel); per provider print "<name>: used/cap calls today"
via the new usage() accessor instead of private fields. DeepSeek keeps its
"headlines scored" extra; cap/error warnings work off base-class stamps.
- NO changes to prompts, payload builders, caches, or the panel's own
max_panels_per_day budget - all provider-agnostic already.
================================================================================
4. TESTS (new "Module 24: LLM provider routing", ~12 tests, ~200 lines)
================================================================================
Reuse _StubLLM; new _FakeAnthropic (fake messages.create returning a stub
response object) injected as client._client.
1. app_config: anthropic block parses/validates/defaults empty.
2. app_config: llm routing block - valid map round-trips; unknown consumer
key, unknown provider name, non-dict -> AppConfigError; absent -> {}.
3. anthropic_api_key resolves env -> config.json -> empty.
4. Base contract on BOTH subclasses (parametrized): keyless/disabled -> no
call; daily cap blocks + UTC-roll reset; failure stamps last_error,
success stamps last_ok.
5. AnthropicClient parse: clean JSON -> dict; ```json-fenced -> dict;
non-JSON -> None; stop_reason "refusal" -> None; no text block -> None.
6. AnthropicClient request shape: NO temperature key, max_retries=0,
cache_control present on system block (assert on captured kwargs).
7. AnthropicClient with `anthropic` package missing (simulated) ->
available False, chat_json None, no crash at import.
8. Routing: no llm block -> both consumers get the SAME DeepSeek singleton
(shared cap); routed config -> news gets DeepSeekClient while
agent_panel gets AnthropicClient; same-provider consumers share one
instance.
9. Global lc._client override shadows BOTH consumers (locks the 41 legacy
stub sites' semantics).
10. reset_llm_client() clears both the override and the per-provider cache.
11. End-to-end: routing agent_panel->anthropic, attach_panel reaches the
anthropic fake (5 calls) while a news call reaches the deepseek stub.
12. telegram _news_health_lines renders per-provider lines from usage()
without touching private fields.
GATE: Modules 19 and 23 must pass UNMODIFIED - the no-regression proof for
the refactor.
================================================================================
5. DOCS
================================================================================
- CLAUDE.md: llm_client bullet -> provider-abstracted (BaseLLMClient;
deepseek + anthropic registered); config.json "llm" block routes consumers;
absent = all-DeepSeek. Add ANTHROPIC_API_KEY to env-var list. Test counts.
- config.json.example: both new blocks.
================================================================================
COMMIT SEQUENCE (each commit green)
================================================================================
1. llm_client: extract BaseLLMClient; DeepSeekClient unchanged behavior
(+ base-contract tests) - pure refactor, Modules 19/23 green untouched.
2. config: anthropic block + llm consumer-routing block (+ config tests).
3. llm_client: AnthropicClient (official SDK, lazy import) + requirements.txt
(+ parse/shape tests).
4. routing: per-consumer get_llm_client + health lines - consumer call-site
edits, telegram usage() lines, routing/e2e tests.
5. docs: CLAUDE.md + example config.
================================================================================
VERIFICATION
================================================================================
- python test_all.py green at every commit (~374 -> ~386).
- Live smoke A (no behavior change): default config, main.py --panel ->
still DeepSeek, /status unchanged.
- Live smoke B (the goal): llm.agent_panel="anthropic" + real
ANTHROPIC_API_KEY, run --panel -> one real Claude panel (5 requests,
validates JSON contract + latency + refusal-free path) while a news
refresh still hits DeepSeek; /status shows two provider lines.
- Backtest untouched by construction (no LLM in replay path); one 30-day
run as a formality.
================================================================================
EFFORT ESTIMATE
================================================================================
0 new files, 8 modified, ~520 lines, est. 3-5 hours.
llm_client.py base class + AnthropicClient + routing +235
test_all.py Module 24 +200
app_config.py anthropic + llm blocks, resolver +55
telegram_bot.py per-provider health lines +15
news_engine.py consumer tags on call sites +4
agent_panel.py consumer tags on call sites +4
config.json.example blocks +14
requirements.txt anthropic +1
CLAUDE.md docs +10
================================================================================
KEY DECISIONS EMBEDDED IN THIS PLAN
================================================================================
- Official Anthropic SDK, not raw requests (current API guidance for Python;
lazy import keeps it optional for DeepSeek-only installs).
- Default Claude model: claude-opus-5 ($5/$25 per MTok; a full 5-call panel
~ 8k in / 500 out ~ $0.05). Config-overridable to claude-haiku-4-5.
- Two Claude gotchas handled: temperature is REMOVED on Opus 5 (sending it
400s - must not be copied from the DeepSeek request body), and refusals
arrive as HTTP 200 with stop_reason "refusal" (checked before parsing,
mapped to the normal None/fallback path).
- Caps are per-provider, not per-consumer: two consumers on one provider
share its cap (today's semantics); splitting across two providers doubles
total headroom naturally.
- The panel's own max_panels_per_day budget is untouched - it sits above
whichever provider serves it.
- Not in scope: static/news_summary.php (own PHP client/key), streaming,
tool use, per-consumer model overrides (add a "model" key per consumer
later if ever needed).
================================================================================
+2 -2
View File
@@ -770,8 +770,8 @@ class NewsEngine:
pub_date_el.text if pub_date_el is not None else None
)
# Only recent headlines (last 24h)
if pub_date and (datetime.now(timezone.utc) - pub_date).total_seconds() > 86400:
# Only recent headlines (configurable: headline_max_age_seconds)
if pub_date and (datetime.now(timezone.utc) - pub_date).total_seconds() > self.config.headline_max_age_seconds:
continue
# Dedup within and across feeds so a syndicated story
+61 -17
View File
@@ -630,6 +630,7 @@ const IND_SHORT = {ema_trend:"EMA", rsi:"RSI", macd:"MACD", bollinger:"BB",
const ARROW = {STRONG_BUY:"⇈", BUY:"↑", NEUTRAL:"→", SELL:"↓", STRONG_SELL:"⇊"};
const TF_LABEL = {"5min":"M5", "15min":"M15", "1h":"H1", "4h":"H4"};
const TF_ORDER = ["5min","15min","1h","4h"];
const TF_SECONDS = {"5min": 300, "15min": 900, "1h": 3600, "4h": 14400};
/* 5 s polling is cheap because fetchJSON uses conditional GETs — Apache
answers unchanged files with 304, so most polls transfer no body. */
const POLL_MS = 5000;
@@ -783,7 +784,8 @@ function renderCards() {
const foot = `<div class="tf-foot">
<span>${entry.scanned_at ? "scanned " + fmtAge(entry.scanned_at) : ""}</span>
<span class="cd">closes in
<span class="num" data-close="${entry.next_close_epoch || ""}">--:--</span></span></div>`;
<span class="num" data-close="${entry.next_close_epoch || ""}"
data-interval="${TF_SECONDS[tf]}">--:--</span></span></div>`;
if (!r) {
return `${open}<div class="tf-head"><span class="tf-name">${TF_LABEL[tf]}</span>
<span class="spacer"></span><span class="open-hint">details ›</span></div>
@@ -1021,7 +1023,8 @@ function renderDetail() {
$("detail-title").textContent = TF_LABEL[detailTf] || detailTf;
$("detail-price").textContent = r?.current_price ? fmt$(r.current_price) : "—";
$("detail-sub").innerHTML = `${snapshot?.symbol ? esc(snapshot.symbol) + " · " : ""}closes in
<span class="num" data-close="${entry.next_close_epoch || ""}">--:--</span>`;
<span class="num" data-close="${entry.next_close_epoch || ""}"
data-interval="${TF_SECONDS[detailTf]}">--:--</span>`;
$("detail-body").innerHTML = detailHTML(detailTf);
dlg.querySelectorAll("details[data-dkey]").forEach(d => {
@@ -1160,8 +1163,12 @@ function renderWire() {
};
const head = items.slice(0, 12).map(item).join("");
const rest = items.slice(12);
/* preserve the disclosure across poll re-renders — otherwise an expanded
list snaps shut within one poll cycle (renderDetail does the same via
data-dkey; there is only one of these, so a class lookup suffices) */
const moreOpen = $("wire").querySelector("details.more")?.open ?? false;
$("wire").innerHTML = head + (rest.length
? `<details class="more" style="padding-top:6px"><summary>${rest.length} more headlines…</summary>
? `<details class="more" style="padding-top:6px"${moreOpen ? " open" : ""}><summary>${rest.length} more headlines…</summary>
<div class="wire">${rest.map(item).join("")}</div></details>` : "");
}
@@ -1342,37 +1349,55 @@ wireNewsOpeners($("cal-body"), "data-ev-key", openCalDetail);
async function fetchJSON(name) {
/* no-cache = revalidate with the server every time (If-Modified-Since);
unchanged files come back 304 with no body, so aggressive polling
stays nearly free. */
stays nearly free. Returns raw TEXT, not parsed JSON: a 304 is
transparent to fetch (the browser answers with the cached body and a
200), so poll() detects "unchanged" by comparing the payload itself
and skips the DOM rebuild entirely. */
const res = await fetch(name, {cache: "no-cache"});
if (!res.ok) throw new Error(`${name}: HTTP ${res.status}`);
/* The Date header is stamped at RESPONSE time — snapshot.server_time_epoch
is stamped at publish time, minutes stale between candle closes. */
const dateHdr = res.headers.get("Date");
const data = await res.json();
return {data, serverNow: dateHdr ? Date.parse(dateHdr) / 1000 : null};
const text = await res.text();
return {text, serverNow: dateHdr ? Date.parse(dateHdr) / 1000 : null};
}
let refetchTimer = null;
let lastSnapText = null, lastNewsText = null; // change detection for the render-skip below
async function poll() {
try {
const [snap, nw] = await Promise.all([
fetchJSON("snapshot.json"),
fetchJSON("news.json").catch(() => null), // news outage ≠ page failure
]);
snapshot = snap.data;
news = (nw && nw.data) || news;
clockOffset = (snap.serverNow ?? snap.data.server_time_epoch) - Date.now() / 1000;
const snapChanged = snap.text !== lastSnapText;
const newsChanged = nw != null && nw.text !== lastNewsText;
if (snapChanged) { snapshot = JSON.parse(snap.text); lastSnapText = snap.text; }
if (newsChanged) { news = JSON.parse(nw.text); lastNewsText = nw.text; }
clockOffset = (snap.serverNow ?? snapshot.server_time_epoch) - Date.now() / 1000;
lastPollOk = new Date();
$("banner-fail").style.display = "none";
renderHero(); renderContext(); renderCards(); renderCalendar(); renderWire();
/* re-render only what new data can actually change — an unchanged
payload keeps the existing DOM (selection, disclosure state, scroll)
intact; countdowns and the freshness line tick from it regardless.
Trade-off: "Xm ago" ages in the wire/calendar only advance when the
payload changes. */
if (snapChanged) {
renderHero(); renderCards();
renderDetail(); // keep an open detail view in sync with the new scan
}
if (newsChanged) {
renderCalendar(); renderWire();
renderNewsDlg(); // same for an open news/calendar dialog
}
if (snapChanged || newsChanged) renderContext(); // blurb reads news, risk chip reads snapshot
tick(); // countdowns appear immediately, not after the first 1 s tick
/* one-shot refetch shortly after the next expected publish, so a
fast scan appears ~2 s after the publisher wakes instead of
waiting out the poll interval (the regular POLL_MS loop covers
slower scans) */
if (snapChanged) {
const nextScan = Math.min(...TF_ORDER.map(tf =>
snapshot.timeframes[tf]?.next_scan_epoch || Infinity));
if (isFinite(nextScan)) {
@@ -1380,6 +1405,7 @@ async function poll() {
const delay = (nextScan - nowServer() + 2) * 1000;
if (delay > 0) refetchTimer = setTimeout(poll, delay);
}
}
} catch (err) {
const last = lastPollOk ? ` Last update ${lastPollOk.toLocaleTimeString()}.` : "";
const b = $("banner-fail");
@@ -1388,28 +1414,46 @@ async function poll() {
}
}
/* 1 s countdown ticker (server-clock anchored) */
/* 1 s countdown ticker (server-clock anchored). The wrap interval comes
from data-interval stamped at render time — reverse-looking the epoch up
in snapshot.timeframes mis-resolves shared boundaries (M5/M15/H1 all
close on the hour) to the first matching TF in TF_ORDER. */
function tick() {
if (!snapshot) return;
const now = nowServer();
document.querySelectorAll("[data-close]").forEach(el => {
let t = parseFloat(el.dataset.close);
const t = parseFloat(el.dataset.close);
if (!t) return;
const tf = TF_ORDER.find(k => snapshot.timeframes[k]?.next_close_epoch === parseFloat(el.dataset.close));
let secs = t - now;
if (secs < 0) { // boundary passed while waiting for next publish
const interval = {"5min": 300, "15min": 900, "1h": 3600, "4h": 14400}[tf] || 300;
secs = ((secs % interval) + interval) % interval;
const interval = parseFloat(el.dataset.interval);
/* one-shot countdowns (calendar releases) carry no data-interval:
clamp at zero until the next poll re-renders them as "released …" */
secs = interval > 0 ? ((secs % interval) + interval) % interval : 0;
}
el.textContent = fmtCountdown(secs);
});
const m5 = snapshot.timeframes["5min"]?.next_close_epoch;
if (m5) {
let secs = m5 - now;
if (secs < 0) secs = ((secs % 300) + 300) % 300;
if (secs < 0) secs = ((secs % TF_SECONDS["5min"]) + TF_SECONDS["5min"]) % TF_SECONDS["5min"];
$("hero-cd").textContent = fmtCountdown(secs);
}
if (lastPollOk) $("hero-updated").textContent = `updated ${fmtAge(lastPollOk.toISOString())}`;
/* publish-time freshness, not poll health — a dead publisher behind a
healthy Apache would otherwise read as "updated just now" forever */
const pubEpoch = snapshot.server_time_epoch;
if (pubEpoch) {
const age = now - pubEpoch;
const el = $("hero-updated");
/* M5 publishes every 5 min while the market is open, so anything older
than 15 min means the pipeline is down, not between scans. The
weekend case is already covered by the market-closed banner. */
const stale = !snapshot.market_closed && age > 900;
el.textContent = (stale ? "STALE — published " : "published ") +
fmtAge(new Date(pubEpoch * 1000).toISOString());
el.style.color = stale ? "var(--amber)" : "";
el.style.fontWeight = stale ? "600" : "";
}
}
setInterval(tick, 1000);