Commit Graph

194 Commits

Author SHA1 Message Date
Harwinder Sekhon d1d002a4f7 dashboard: clickable news — detail dialogs + on-demand AI summary endpoint
- news_engine: RSS parser captures link/description/publisher (three new
  optional NewsSentiment fields); _clean_html_text/_safe_link helpers
- news.json: headlines carry id = sha1(headline) + enrichment fields
- static/news_summary.php (NEW): on-demand DeepSeek gold-impact summary,
  whitelist-gated against news.json ids, cache-forever in ai/, fail-closed
  file-based daily cap (150/day), key at /etc/gold-dashboard/deepseek.key;
  installed to the docroot by install_static
- dashboard.html: newswire + calendar detail dialogs (score tiles, source
  description, article link, AI summary with loading/error states);
  delegated on stable containers, dialog keyed by id across polls
- tests: +6 (357 total); docs: README-dashboard §6 one-time setup
  (key file, apache-owned ai/, SELinux httpd_can_network_connect), CLAUDE.md

Claude-Session: https://claude.ai/code/session_01Aq1jZ6jBsBdEC2xT3u46SA
2026-08-07 09:08:49 -04:00
root edefd90aaa oanda: retry once on transient 401/5xx/network faults in candle fetch
The practice API rejects ~15-20 requests/day with a spurious 401
"Insufficient authorization" while the same token succeeds on adjacent
requests; each one silently cost a full scan cycle on that timeframe.
One retry after RETRY_DELAY_S (2 s) absorbs these plus transient
5xx/network faults. Real credential failures fail both attempts and
raise as before; other 4xx (bad request, 429) are never retried.

Tests: transient-401 recovery, persistent-401 raise, 400 not retried.

Claude-Session: https://claude.ai/code/session_01Tujtmmhxj9mxKjPFSoVjmP
2026-08-06 23:06:29 -04:00
Harwinder Sekhon b21c75a1f8 dashboard: Gold Desk v2 UI — light/dark theme, TF detail dialog, wire score
Owner-supplied page design replacing the dark-only v1: Bootstrap 5
colour tokens with a luma-inverted dark mode (toggle + system-pref +
localStorage), compact clickable TF cards opening a full-detail modal
(all-nine indicator table with value/conf/weight, price ladder of key
levels vs spot, per-confirmation-TF drilldowns, provenance), newswire
aggregate sentiment box, Inter/JetBrains Mono webfonts.

Re-applied the c70f58a sync fixes the source copy predated: 5 s
conditional-GET polling and the +2 s boundary refetch sniper. Data
logic otherwise unchanged; all fields verified against
report_to_webapp_dict output.

Claude-Session: https://claude.ai/code/session_017mbXs9HRawZhD1uZ3ze7zQ
2026-08-06 22:44:48 -04:00
Harwinder Sekhon a2f951f5c9 dashboard: cut page sync latency to ~scan-time + ≤5 s
Three display-side fixes, no signal-logic changes:
- run_cycle publishes snapshot.json BEFORE the news refresh — news I/O
  (FF + 4 RSS + possible DeepSeek calls) took 20-30 s on TTL-expired
  cycles and kept fresh signals off the page (publish_all split into
  publish_snapshot/publish_news, kept for boot)
- page polls every 5 s (was 20 s) using conditional GETs: dropped the
  ?t= cache-buster + no-store so Apache answers unchanged files with
  bodyless 304s
- boundary refetch sniper re-armed: fires 2 s after next_scan_epoch
  regardless of distance (the old delay<POLL_MS guard would have been
  dead code at 5 s polling)

New Module 20 test locks the snapshot-before-news ordering. 352 tests.

Claude-Session: https://claude.ai/code/session_017mbXs9HRawZhD1uZ3ze7zQ
2026-08-06 22:35:13 -04:00
Harwinder Sekhon 858c12280a news: DeepSeek validation, vs_forecast fix, /status source health
Validation (llm_news_eval.py harness + 78-headline golden CSV + verdict
doc): headline scoring 78/78 vs keyword baseline 24/78 — validated;
event gold-direction calls misread close K/M numeric comparisons ~11%
of the time (both claims cases in sample, JOLTS 8.79M vs 8.84M).

Fix: event payload now carries code-computed "vs_forecast" (from
EconomicEvent.surprise) and the prompt gains an explicit inverted-
metrics rule + worked claims example — iteration 1 (field alone) fixed
the arithmetic but exposed a masked inversion error. 18/18 on live
re-verification (4 misreads fixed, 11 regression cases clean, 3 fresh
both-side probes).

/status: new "News & AI" section — per-source health for the calendar
chain (which of FF/MyFXBook served), each RSS feed, and DeepSeek
(daily count, last error kind), written passively by the existing
fetch paths under _cache_lock. Failures WARN once on transition
(repeats DEBUG, recovery INFO) so the rotating log carries outage
stories without 10-min spam.

test_all.py: +5 Module 19 tests (351 total); also carries the Module 22
fixed-RR-TP tests from the prior research session (shared file).
Replay artifacts dir gitignored like commit_ladder.

Claude-Session: https://claude.ai/code/session_017mbXs9HRawZhD1uZ3ze7zQ
2026-08-06 22:21:02 -04:00
Harwinder Sekhon 525a1cd1c1 research: --fixed-rr-tp backtest flag + 2026-08-05 verdict (NOT shipped)
Hard TP at N×risk beats the trail on all 2024+ windows but the grid is
monotone to 3R (trend artifact) and M15 rr20 goes negative in 2021+2023
with doubled DD — trailing exits stay. Flag is research-only; its
Module 22 tests land with the next commit (shared test_all.py).

Claude-Session: https://claude.ai/code/session_017mbXs9HRawZhD1uZ3ze7zQ
2026-08-06 22:20:43 -04:00
Harwinder Sekhon b693f757f8 Remove OANDA_MIGRATION_ROADMAP.md (reference-only doc, superseded)
Deletion was already staged in the working tree by the owner; the
roadmap was approved 2026-07-24 as reference only and its plan file
lives outside the repo. Content remains in git history.

Claude-Session: https://claude.ai/code/session_01UnZkzeFcTu9agFg8BDh2tn
2026-08-04 20:26:54 -04:00
Harwinder Sekhon 7dbcbc6c82 audit: NY-anchored weekend calendar (all 3 copies) + FB2_EMA_LO crash-proofing
Owner decisions from the 2026-08-04 audit report:
- spread-in-exit-triggers: DECLINED (books stay comparable; ledger in
  CLAUDE.md records the measured ~4-6% optimism).
- weekend clock: FIXED. The market week is Fri 17:00 -> Sun 18:00 New
  York time, not Fri/Sun 22:00 UTC — during US DST the Oanda feed stops
  at 21:00 UTC (measured: 159/162 summer weekends). All three weekend
  implementations (providers/base, signal_aggregator, gpu_backtest
  calendar filter) now share the NY-anchored rule; a new test sweeps
  them hourly across both DST transitions and asserts agreement. No NTP
  needed — the host clock was fine, the constant was wrong.
  Ends the weekly false-"STALE DATA / feed lagging" hours (summer Fri
  21-22 UTC, winter Sun 22-23 UTC).
- FB2_EMA_LO (delegated): kept as a research knob but crash-proofed —
  an unparseable value degrades to the tuned default 0.5 with a warning
  instead of killing every scan. Documented in CLAUDE.md.

Book gates: M15 12mo pinned book byte-identical (+78.7111R, 397
trades). M5 12mo drops exactly ONE trade (556 -> 555, +97.56 ->
+98.64R): a phantom whose signal fired on the final summer-Friday bar
(decision 21:00 UTC = NY 17:00 market close) and was "filled" at
Sunday's reopen 49 hours later, -1.08R — an entry live trading can
never take. The backtest no longer contains it; live == backtest
parity improved. M15 unaffected because hour 21 is in its blocked set.

Suite: 341/341 green.

Claude-Session: https://claude.ai/code/session_01UnZkzeFcTu9agFg8BDh2tn
2026-08-04 20:21:34 -04:00
Harwinder Sekhon 5b3f47f3e1 audit: scheduler cleanup, WS-log secret hygiene, simulator test coverage, CLAUDE.md sync (Batch G)
2026-08-04 Fable 5 QA audit, final code batch.

- Boundary math: removed the provably-dead `remaining <= buffer` guard
  from all three scheduler copies (518,400-instant frozen-clock sweep:
  never fired) and documented the exact-boundary-instant semantics.
- websockets/websockets.client loggers pinned to INFO so a DEBUG run
  can never write the TD key (connect URI) or Tiingo auth frame into
  the 30-day rotating log. (.env also chmod 600, untracked.)
- Cooldown pass factored into _apply_cooldown (pure, testable) — the
  only sequential replay rule finally has direct tests.
- New simulator behavior tests (previously ZERO coverage of the trade
  resolver): same-bar SL+TP tie → SL-first, gap-through-SL fills at
  the open, TIMEOUT pricing + round-trip spread charging, trailing
  ratchet arms at bar close and fills next-bar-only, cooldown ordering.
- CLAUDE.md synced to verified reality: 338 tests/21 modules; three
  weekend-implementation copies caveat; throttle claim corrected
  (single-flight, TD-only throttle); daemon trail/cooldown parity
  caveats (once-per-OBSERVED-bar, wall-clock cooldown) + OPEN_ONLY and
  reconcile notes; CP-path gate semantics documented as validated
  behavior; new "known open items" section for the owner-decision
  findings (weekend DST constant, spread-in-trigger rule-6 item, H4
  seeding divergence, FB2_EMA_LO).

Gates: suite 338/338 green; pinned M15 book and M5 HEAD baseline both
reproduced identically after the cooldown refactor.

Claude-Session: https://claude.ai/code/session_01UnZkzeFcTu9agFg8BDh2tn
2026-08-04 20:08:37 -04:00
Harwinder Sekhon 99eb268cf3 audit: alert delivery, provider robustness, backtester NaN/cache guards (Batches C+E+F)
2026-08-04 Fable 5 QA audit. Three batches sharing one test file, all
verified by runnable repros before fixing, all gated below.

Batch C — telegram alert delivery (verified by h4 harness):
- Failed breach rescan shrinks the cooldown to 30 s instead of
  silencing the symbol for 5 min; breach during cooldown is PARKED and
  fired on the first tick after expiry (was dropped forever).
- Level cache keyed symbol→TF (M5 scans no longer wipe H1/H4 swing
  levels within minutes); breach labels TF-qualified.
- Markdown alert sends get a plain-text fallback (mirroring _safe_edit);
  fingerprint commits only when a message actually landed.
- Per-key asyncio.Lock closes the dedup TOCTOU between auto-scan and
  breach sends (forced-interleave repro produced duplicate alerts).
- _start_scheduler respawns dead per-TF loops (was all-or-nothing).
- Clean WS close now backs off (was a zero-delay reconnect hot loop);
  backoff resets only after a connection that received ticks; TD
  iter_ticks fires on_connect on the first RECEIVED frame (invalid key
  can no longer pin the backoff at minimum).
- Scan-failure logs carry the real traceback (was "NoneType: None").

Batch E — providers (verified by h5 harness):
- fetch_range: 3 consecutive chunk ERRORS raise instead of returning
  silently truncated history (verified 731-day silent gap repro; also
  covers 429 storms); pagination no-progress guard; head-truncation
  ERROR log.
- Single-flight per (symbol,TF): 4 concurrent scans = 1 API call (was
  4); cache records requested depth so a longer-lookback call refetches.
- Multi-bar forming/future tail fully dropped (was iloc[:-1] once).
- Oanda: malformed candles skipped, never 0.0-defaulted; cleaning drops
  non-positive-price rows BEFORE OHLC repair; inf returns neutralized so
  one corrupt bar can no longer disarm the whole 10-sigma filter.

Batch F — backtester internals (verified by h6 harness):
- NaN bar in a trailing window exits at the last valid bar (was NaN R
  poisoning every metric); static-path TIMEOUT NaN guard;
  compute_metrics excludes non-finite R loudly. Zero real-data exposure
  (0 NaN rows in all 10 Oanda parquets) — defensive.
- fetch_history ERROR-logs pre-2026-07-08 cache files (cleaned-bars era)
  when loaded for raw-path resolution.

Gates: suite 333/333 green; pinned M15 12mo book reproduced identically
(397 trades, +78.7111R) after E and after F; M5 HEAD-config baseline
reproduced identically (556 trades, +97.5604R) after F.

Claude-Session: https://claude.ai/code/session_01UnZkzeFcTu9agFg8BDh2tn
2026-08-04 20:02:57 -04:00
Harwinder Sekhon 18e4fd0403 audit: gate-chain verdicts — lock shipped M5 entry-path state, document hour-delta isolation
2026-08-04 audit, Batch D. Verification outcomes (no behavior changes —
comment + tests only):

- CP gate-bypass (quality forced to B after the gate chain, exhaustion/
  M5-score gates not re-checked) is VALIDATED behavior, not a bug: the
  pinned shipped-config M5 12mo book carries 226/586 XAU_CP trades
  (+38.9R), 104 with EXHAUST warnings (+15.3R), 175 below the M5
  min-score gate (+28.0R). Removing the "holes" removes validated
  profit. New test pins the shipped state (PB off, CP BUY-only on M5,
  ema_partial_neutral M5-only) so any flip is a conscious revalidated
  decision.
- E18 (xau_strategy vs backtest blocked-hours "drift") REFUTED as a
  defect: the divergence is required by pair isolation — the backtest
  copy is the generic non-XAU fallback and must not inherit
  gold-validated deltas. Documented in the factory + a test asserting
  the delta stays XAU-side.
- 6c-upgrade re-arming an exhaustion-D refuted (exhaustion zeroes
  confidence below 6c's floor; regimes mutually exclusive). "WEEKEND
  FILTER" CP-suppression branch is defense-in-depth for direct
  analyze_from_dataframes callers, kept.

Suite: 316 green.

Claude-Session: https://claude.ai/code/session_01UnZkzeFcTu9agFg8BDh2tn
2026-08-04 19:52:07 -04:00
Harwinder Sekhon ebb83749ef audit: GPU-engine cache-miss observability + dead-guard alignment + full parity test
2026-08-04 audit, Batch B. Verified first (h1 harness): the two engines
are byte-identical on every reachable input — 370-prefix flat-bar deck,
warmup band, 322 real-data M15 positions incl. all 7 zero-range bars,
zero diffs. The claimed ADX NaN desync / warmup-guard divergences are
unreachable below the shared 30-bar analyze() floor. Changes here are
therefore observability + hygiene, proven behavior-neutral by exact
reproduction of the pinned M15 12mo book (397 trades, +78.7111R,
byte-identical trade rows).

- _resolve_pos: cache-miss fallback to live-engine recompute now warns
  on first miss and counts every miss (was fully silent — a run could
  switch math with no trace).
- ROC/Keltner warmup guards textually aligned with the live engine so a
  future period/floor change can't open a silent divergence window.
- New regression: full-sweep parity test comparing ALL indicator
  signals/confidences, composite, and regimes on every prefix incl. a
  flat zero-DM/zero-TR run (the old parity test skipped
  ADX/MACD/ROC/Keltner/RSI entirely); plus a loud-fallback test.

Suite: 314 green. Book gate: oanda_m15_12mo_030 reproduced identically.

Claude-Session: https://claude.ai/code/session_01UnZkzeFcTu9agFg8BDh2tn
2026-08-04 19:49:08 -04:00
Harwinder Sekhon 8acc1e18f8 audit: harden Oanda execution lifecycle (close-fill, netting, races, fail-closed config)
2026-08-04 Fable 5 QA audit, Batch A. All fixes are execution-robustness
only — no signal code touched (git diff scope: daemon/executor/tests).
Findings verified by runnable repros before fixing (h3 harness):

- close_trade: a 200 whose close order was created but NOT filled (e.g.
  MARKET_HALTED during the settlement pause) now raises instead of
  returning; the daemon's timeout path keeps tracking and retries. The
  old path popped state and orphaned a live position forever
  (reconciliation was boot-only).
- place_market_order: positionFill=OPEN_ONLY — on a netting account an
  opposite-side order can no longer silently close an untracked
  position; the divergence surfaces as an order cancel.
- _execute_trade: a fill without tradeOpened (netting close/reduce) and
  any placement OandaAPIError (server may have filled before a network
  timeout) now trigger _reconcile_positions instead of silent return.
- _discover_existing_positions → lock-guarded _reconcile_positions,
  callable mid-run; adoption of a position with NO broker stop now
  screams (CRITICAL + per-bar warning) instead of computing sl=entry,
  risk=0 and silently never trailing.
- _manage_trailing: snapshot deep-copies per-trade dicts (no mutation of
  live state outside the lock); write-back skips trades popped by
  _check_closed_trades mid-iteration (no ghost resurrection).
- Fail-closed config: unknown OANDA_QUALITY_GATE refuses to start
  (previously a typo silently became the most permissive gate "C");
  non-integer OANDA_LOT_SIZE/MAX_HOLD_BARS exit with a clear message.

Suite: 302 → 312 tests (Module 21), all green.

Claude-Session: https://claude.ai/code/session_01UnZkzeFcTu9agFg8BDh2tn
2026-08-04 19:46:37 -04:00
Harwinder Sekhon 28a1c375bd M5 at real cost: disable XAU_PB, squash partial EMA alignment
Both validated at the measured $0.65 spread on Oanda bars (12mo halves +
PF, OOS 2024-06->2025-03, 5yr; full evidence in pair_signals/xauusd.py
M5 overrides 4-5):

- m5_pullback_enabled False on M5: loss forensics identified XAU_PB as
  the largest wrong-direction loss source; 12mo PF 1.26 (failing the
  acceptance bar) -> 1.44, max DD 23.9R -> 13.9R. PB code stays for
  future regime-gated experiments.
- New TechnicalConfig.ema_partial_neutral (default False), True on M5
  only: partial 4-EMA ribbon alignment (2-3 of 4 checks) scores NEUTRAL
  instead of a weak BUY/SELL (-0.008R/84 vs +0.230R/341 fully aligned).
  Squash mirrored in the backtester's precomputed EMA scorer in this
  same commit (live==backtest parity). 12mo +97.6R/556 PF 1.48 DD
  10.7R halves +47.8/+49.8; 5yr +127.4R PF 1.38; OOS one-trade tie.
- Rejected and logged (memory + docstring): hour-19 block (OOS fail),
  hours-14/15, trail dist 0.75, confirm-TF swaps incl M15+H4 (OOS
  ordering), divergence/candlestick weight changes, bollinger removal.
- Tests 296 -> 302: partial-alignment fixture test, factory-equivalence
  mirrors, per-TF isolation test, BC-gate fixture force-enables PB.

Claude-Session: https://claude.ai/code/session_01H1eKWZsAVapJ1C4winYLqX
2026-08-04 16:54:09 -04:00
Harwinder Sekhon 775330ff16 /news: normalize rounded-zero score so "-0.00" renders as "+0.00"
A balanced feed (XAU and USD means nearly cancelling) yields a tiny
negative net that "+.2f" rendered as "-0.00" — seen live 2026-08-03.
Regression test uses that day's actual per-currency means (300 tests).

Claude-Session: https://claude.ai/code/session_017975c1TufZ3xRYAq8iWTCS
2026-08-03 18:06:44 -04:00
Harwinder Sekhon 1f54452007 Telegram /news command: net XAU/USD news sentiment score on demand
Owner-locked /news replies with the existing get_pair_sentiment() net
score (XAU mean - USD mean, -1..+1) and its label; a caveat line marks
the no-headlines case so a feed outage can't read as neutral. Fetch runs
in the shared executor with a placeholder-then-edit reply. Display-only:
news still never gates signals. Registered in handler block, command
menu, help text, docstring, and user guide; 3 new builder tests (299).

Claude-Session: https://claude.ai/code/session_017975c1TufZ3xRYAq8iWTCS
2026-08-03 18:01:42 -04:00
Harwinder Sekhon be5dd4a25b Dashboard publishes directly into /var/www/trade (no subfolder)
Adds exactly three files next to the portal: dashboard.html,
snapshot.json, news.json. README covers chown vs setfacl for write access.

Claude-Session: https://claude.ai/code/session_01Wc9ZoqKLYBT2QJcpRWKoGm
2026-08-03 17:17:37 -04:00
Harwinder Sekhon 0445a20a76 Dashboard out-dir default → /var/www/trade/gold-dashboard (live portal docroot)
Claude-Session: https://claude.ai/code/session_01Wc9ZoqKLYBT2QJcpRWKoGm
2026-08-03 17:14:08 -04:00
Harwinder Sekhon 4e654b5edb Web dashboard v2: static publisher + Apache, with calendar & news panels
- dashboard_publisher.py: standalone candle-close daemon that atomically
  writes snapshot.json/news.json/dashboard.html to an Apache-served dir
  (DASHBOARD_OUT_DIR). Fully independent of telegram_bot/oanda_daemon —
  local confirmation map + boundary math, no imports either way, no new
  deps, no open port. Due-TF classification from the RAW wake time.
- static/dashboard.html: self-contained dark page — hero rail, four TF
  cards (score bar, 3x3 indicator grid, MTF, setup, reasons), DeepSeek
  market context, week economic calendar, scored newswire. Clock offset
  from the HTTP Date header (publish-time epoch is stale between closes).
- news_engine.get_week_events(): additive read-only accessor over the
  cached FF week; no signal path calls it.
- deploy/: systemd unit + live-server (LAMP/SELinux) checklist.
- test_all.py: Module 20 (9 tests) + get_week_events test; 296 green.

Claude-Session: https://claude.ai/code/session_01Wc9ZoqKLYBT2QJcpRWKoGm
2026-08-03 17:10:29 -04:00
Harwinder Sekhon 9e1c321f05 Add commit-ladder campaign driver; ignore its 26MB results folder
Added: backtest/research/commit_ladder.py (recreates historical configs
via --set on the fixed harness; 2026-08-03 campaign confirmed HEAD is
the best config on all timeframes). Results stay local like all pinned
trade books.

Claude-Session: https://claude.ai/code/session_01Nsd8HcrV67GbCZXxZEn1Ma
2026-08-03 13:29:18 -04:00
Harwinder Sekhon 6fff06ff90 DeepSeek V4-Flash news enrichment: LLM sentiment, surprise direction, summary
Three display/warning-only upgrades to the news engine, all behind the
existing keyword path as fallback — signals are never gated on LLM
output (event_risk_caps_signals / sentiment_affects_signals stay False
per the 2026-07 audit; backtester use_news=False untouched):

- Headline sentiment: one batched call per 600s cache window (capped at
  25 new headlines, sha1-cached so repeats are never re-sent) overrides
  the keyword scores per currency with the gold-inverse doctrine encoded
  in the prompt.
- Release surprises: get_event_risk_level's description now says which
  way the print pulls GOLD ("— gold bearish: strong jobs lift yields"),
  fixing the documented numeric-only gap; risk level and windows
  unchanged.
- Market summary: 1-2 sentence macro blurb on SignalReport.news_summary,
  rendered in Telegram alerts (📰) and the CLI (🧠).

New llm_client.py: requests-only OpenAI-compatible client (no SDK), one
attempt per call, UTC-day request cap, singleton so the cap survives
per-scan NewsEngine reconstruction. Config: deepseek block in
config.json + DEEPSEEK_API_KEY env; absent key = fully inert.

Live-measured API behavior baked in: V4-Flash is a hybrid thinking model
— requests send thinking:disabled (reasoning otherwise blew the timeout
and could exhaust max_tokens before the answer); prompts are static
module constants so DeepSeek's prefix cache prices repeat calls at the
cache-hit rate; scores use compact [s, xau] pairs; timeout 12s.

Tests 273 → 286 (new Module 19 + deepseek config tests).

Claude-Session: https://claude.ai/code/session_01Nsd8HcrV67GbCZXxZEn1Ma
2026-08-03 13:25:02 -04:00
Harwinder Sekhon f7d8f1113c Tiingo WebSocket streaming: re-enable the level-breach monitor
The monitor has been dead since the Oanda feed switch (Oanda's provider
does not stream). New config.json scalar data_providers.streaming selects
a tick-stream provider independent of the candle feed; telegram_bot
resolves it via _get_streaming_dp() (separate instance, candle cache
stays on the active provider) and falls back to the active provider when
unset — behavior is byte-identical to before when the key is absent.

TiingoDataProvider grows the two sanctioned streaming overrides:
supports_streaming=True and iter_ticks() on wss://api.tiingo.com/fx
(mid price only — the no-spread rule extends to the stream). Two
deliberate divergences from the TD reference implementation, both
because Tiingo authenticates inside the socket: server "E" messages
raise TiingoDataError (a swallowed auth error would end the generator
cleanly and reconnect-hammer with zero backoff), and on_connect fires on
the first received message rather than after the subscribe send. Field
order of quote frames was live-verified: ask is data[7], not data[6] as
the docs table suggests (data[6] is askSize).

make_data_provider(config, name=None) now takes an explicit provider
name; keyless streaming providers disable the monitor with one warning
instead of a traceback-per-minute loop.

Tests 266 → 273 (WS parser, ticker round-trip, streaming config scalar,
explicit-name dispatch, streaming-dp resolution, monitor auto-disable).

Claude-Session: https://claude.ai/code/session_01Nsd8HcrV67GbCZXxZEn1Ma
2026-08-03 13:11:36 -04:00
Harwinder Sekhon bcda6e0ebd Cleanup: remove dead code, stale docs, and all non-main branches
- Delete back-compat shims data_provider.py / oanda_provider.py;
  test_all.py now imports providers.* directly (ForexDataProvider name
  retired; the two shim-only tests removed, 268 -> 266 tests)
- Delete backtest/research/calendar_store.py (no importer, no entry
  point); config.py audit comment reworded
- Delete executed plan doc (2026-07-21 centralized-config) and the two
  unbuildable guide PDFs; HTML guides remain the source of truth
- OANDA_MIGRATION_ROADMAP.md status header corrected: phases 1-2
  shipped 2026-07-28, feed is live on Oanda; phases 3-6 still unbuilt
- Branches deleted locally + on both remotes: oanda_demo (merged),
  web-dashboard, eurusd-modular, reversal-exit-alert (owner-approved
  permanent deletion); July stash dropped

Claude-Session: https://claude.ai/code/session_019zR58p8ACW1oJ5QXVUZ5es
2026-08-03 11:08:48 -04:00
Harwinder Sekhon 042b079861 Big-candle gate (default-OFF) + M5 relaxation campaign + --set tooling
Gate 6h (big-candle entry gate): blocks all XAU entries (core, XAU_PB,
XAU_CP) when the primary TF's last 24h contains a bar with range >= K x
prior-bar ATR. Ships default-OFF: validation rejected every threshold
(K 3-6 on M5/M15 12mo) — removed trades were net-POSITIVE at every cell;
M5 quiet-window trades are the only losing bucket. Config knobs
big_candle_atr_multiple/lookback_hours, --big-candle backtest flag,
Module 17 tests.

M5 relaxation SHIPPED (validated 2026-07-30, Oanda bars, $0.30 spread):
m5_min_abs_score 30->25, m5_min_confirmation_score 10->5, hour-21 UTC
unblocked M5-only. Beats control in ALL 6 calendar years of the 5yr
replay (+230.9R/2319 PF 1.26 DD 21.3 vs +168.0R/2048 PF 1.21 DD 27.0),
12mo +107.4R PF 1.36 vs +84.6R PF 1.33, OOS +26.1R vs +17.2R, $0.70
spread +86.9R vs +66.4R. M15: every relaxation retested negative —
config unchanged. Declined-but-open: M5 hour-22 block, M15 hour-16
unblock (both fail 5yr/OOS). Evidence in the factory docstring.

Tooling: generic --set PATH=JSON research override in gpu_backtest
(typo-proof, logged); diagnose_filters filter_tags refreshed for gates
6d-6i with a staleness-guard test (Module 18); research scripts
hour_analysis/burst_profile/book_diff. Suite: 268 tests.

Claude-Session: https://claude.ai/code/session_014jqHpmNYmwYMowF6bPZyeD
2026-07-30 11:30:20 -04:00
Harwinder Sekhon dc7beb2877 Rebuild CLAUDE.md: lean architecture reference
Replaces the deleted exhaustive knob-by-knob reference with a compact
guide: commands, config precedence, pipeline architecture, rules for
changes, and current-state notes (Oanda feed, TD-pinned books).

Claude-Session: https://claude.ai/code/session_014jqHpmNYmwYMowF6bPZyeD
2026-07-30 11:30:07 -04:00
Harwinder Sekhon 637c39e201 Switch active data source to Oanda v20 (practice feed)
config.json.example: data_providers.active = "oanda" (credentials stay
in env/.env or the top-level oanda block; TD and Tiingo remain
registered fallbacks). Backtest parquet cache is now keyed per provider
— Oanda and TD bars diverge (2026-07-24 study), so a feed switch must
never serve another feed's cached bars; twelve_data keeps the legacy
un-prefixed filename so pinned-book caches stay valid. 254 tests green;
live smoke-tested against the practice API (66 contract checks: all
scanned TFs, boundary/freshness guarantees, real spread $0.71,
fetch_range pagination, typed error paths, CLI + backtest end-to-end).

Claude-Session: https://claude.ai/code/session_01Mc7XKYyPySVXiUpcN82QEK
2026-07-28 20:12:59 -04:00
Harwinder Sekhon 083a4e8d63 Provider abstraction: providers/ package with BaseDataProvider template
One class per provider (TwelveData, Oanda, Tiingo), all subclassing
BaseDataProvider which owns cache, validation/cleaning, freshness +
boundary guarantees, fetch_range pagination, and streaming stubs —
subclasses implement only _fetch_raw. config.json data_providers.active
selects the live feed; data_provider.py / oanda_provider.py stay as
back-compat shims. Module 16 provider-abstraction tests (253 total green).

Claude-Session: https://claude.ai/code/session_01Mc7XKYyPySVXiUpcN82QEK
2026-07-28 20:09:00 -04:00
Harwinder Sekhon f760e9070a Research: Dukascopy real-spread study + Oanda/TD bar-divergence study
Scripts + summary CSVs for the 2026-07-24 studies (real XAU spread ~$0.65
tradable hours; Oanda vs TD bars diverge ~11% of ATR with 10% direction
flips). Large raw data files (parquet, per-boundary samples) gitignored.

Claude-Session: https://claude.ai/code/session_01Mc7XKYyPySVXiUpcN82QEK
2026-07-28 20:08:53 -04:00
Harwinder Sekhon 9a384935bf Add Oanda migration roadmap (reference, not scheduled)
Approved 2026-07-24: 6-phase plan for fully switching the signal feed
from Twelve Data to Oanda v20 (inert plumbing -> backtester agnosticism
-> pricing-stream monitor -> shadow parity -> book re-validation with
owner cost gates -> reversible cutover). TD remains a permanent
fallback. Grounded in the 2026-07-24 divergence study (10% bar-direction
flips, 11%-of-ATR noise, real Oanda spread ~$0.72 vs assumed $0.30).

Claude-Session: https://claude.ai/code/session_01YNvxXtgz62F7kTKZZsAAAE
2026-07-24 15:12:37 -04:00
Harwinder Sekhon d669227ddc Oanda demo daemon: auto-execution on practice account (XAU/USD)
New independent trading daemon (oanda_daemon.py) that places orders on
an Oanda v20 practice account instead of sending Telegram alerts.
Oanda-only market data (oanda_provider.py, mid candles + real bid/ask
spread) through the unchanged signal pipeline; order execution and
trade management via oanda_executor.py (raw requests, no new deps).

Book parity with the backtester: no broker-side TP (exits = SL/TRAIL/
TIMEOUT only), bar-close-only trail ratchet with each closed bar
processed exactly once, 5-bar per-TF entry cooldown, structural SL
from the setup. Accepted divergence: one position total (skipped
signals logged for book reconciliation).

Safety/ops: fail-fast boot on rejected credentials (transient outages
continue), market orders never retried (double-fill risk), entry lock
against aligned-boundary races, restart recovery + adoption via
oanda_state.json, hourly heartbeat, asyncio exception handler, every
failure path logged. Prices formatted at instrument precision.

Config: env OANDA_* -> config.json "oanda" section (validated in
app_config). Tests: Module 15 (30 tests), suite 240 green.

Claude-Session: https://claude.ai/code/session_01VUnCqjWBuvKRSjLwHeB9BJ
2026-07-24 14:10:45 -04:00
Harwinder Sekhon c3329d6cd5 Telegram alert redesign: verdict-first layout + informational 100-pip TP
format_signal_message restyled (display-only, zero signal-logic impact):
verdict header first (signal - pair - TF - quality), trade setup moved
above the detail, per-indicator breakdown condensed to one arrow line
(_indicator_summary), heavy dividers dropped. New TP 100p line in every
setup block: entry +/- 100 pips ($10.00 on XAU) with its R-multiple vs
the stop (_INFO_TP_PIPS, formatter-only - not stamped on TradeSetup,
not in CLI/JSON/backtester). Trailing-plan logic unchanged (stamped
per-TF params, pullback fallback). Removed dead _indicator_line and
_SIGNAL_TAG. Tests 210 -> 212 (100p TP math BUY/SELL/no-setup, one-line
indicator summary).

Claude-Session: https://claude.ai/code/session_01BoZKi67Ct6QTeiFJmybXSw
2026-07-24 10:58:22 -04:00
Harwinder Sekhon 7a810c4a4c Remove Tiingo spread monitor (indicative two-tier feed, unvalidatable)
Tiingo's free /tiingo/fx/top publishes only a two-tier indicative
spread for gold ($0.10 liquid hours / $0.70 rollover+news; verified on
273 production samples + live probes) — mid ± a posted increment, not
executable depth, so it can never validate the $0.30 backtest cost
assumption. Removed one day after shipping: tiingo_provider.py, the
sampler/quote plumbing in telegram_bot.py, the channel_spread_detail
config knob and tiingo_api_key resolver, and Module 14's Tiingo tests
(221 -> 210). CLAUDE.md documents the verdict: do not re-add a spread
display from Tiingo's free FX feed.

Claude-Session: https://claude.ai/code/session_01BoZKi67Ct6QTeiFJmybXSw
2026-07-24 10:56:55 -04:00
Harwinder Sekhon 3a95a37a0c Accuracy audit fixes: alerts carry validated per-TF trail plans + 13 more
Five-agent audit of the signal path; all fixes are delivery/measurement
side. Regression-proven behavior-neutral on validated signals: the 5yr
M15 replay reproduces the pinned book exactly (+210.65R PF 1.35
WR 57.8% MaxDD 33.90R). Tests 215 -> 221.

- TradeSetup carries trail_arm_r/trail_dist_r stamped from the producing
  config; Telegram trail plan reads them (was showing default 0.6/0.75
  instead of the validated M15 arm 0.70 / M5 dist 0.85)
- Alert dedup: fingerprint = (signal, quality, direction); entry movement
  compared against last-ALERTED entry/risk (old SL-dist bucket churned on
  ATR drift -> near-duplicate alerts every candle)
- Fresh fetches enforce the closed-bar boundary check with one 3s grace
  retry, then warn (feed lag no longer silently analyzes the prior bar)
- Backtest max DD includes the zero baseline (opening losing streaks
  were undercounted; pinned M15 DD unchanged)
- Replay loads the regime-filter TF when the H4-ADX gate is on (was a
  silent no-op for M5/M15 primaries)
- clear_cache under _cache_lock; dead ROC zero-cross branches removed
  from both engines (provably unreachable); RSS dates via
  parsedate_to_datetime (named zones no longer stamped UTC); event risk
  grades from the soonest event; .env inline comments stripped;
  auto-scan skips NEUTRAL like the breach path; report_to_dict honors
  the weight-0 skip rule and emits strategy/trail params; PB funnel
  counts spread_gate/accepted; session-filter fail-open paths log loudly
- Docs corrected to match validated code (H&S neckline = lower peak,
  exhaustion anchors farthest tracked swing) with re-validation warnings

Claude-Session: https://claude.ai/code/session_01R5soRUeRBi8kHMB99csrJy
2026-07-23 10:31:13 -04:00
Harwinder Sekhon f814485615 Tiingo bid/ask spread monitor + logging hardening
- tiingo_provider.py: /tiingo/fx/top quotes (10s TTL cache, graceful
  None on any failure); spread log writer. Signals never consume
  bid/ask; warn-only by design (no historical bid/ask to validate a
  gate against — live == backtest rule).
- app_config: repo-root .env loader (setdefault; shell env still wins),
  tiingo_api_key resolver (TIINGO_API_KEY/TIINGO → config.json),
  telegram.channel_spread_detail flag (default false).
- telegram_bot: 5-min spread sampler → logs/spread_log.csv (weekend-
  aware), alert-time quote refresh, bid/ask + wide-spread warning in
  signal messages, /status spread line. Raw numbers to owner DM always;
  channel only while channel_spread_detail=true (Tiingo free tier is
  internal-use-only; flip before public launch). Free-tier limit math:
  worst case ~41 req/h vs 50/h cap, one pair.
- Logging: 3h rotation x 240 backups (30-day retention), full-date
  stamps, force=True root handlers; candle loops + sampler survive
  failed cycles; breach-task exceptions logged via done-callback;
  PTB error handler; sys/threading excepthooks in main().
- test_all.py: Module 14 (tiingo, dotenv, spread routing, loop
  resilience, rotation) — 215 tests green.

Claude-Session: https://claude.ai/code/session_01HVRmy6KspMs96o5PTPYFAB
2026-07-22 17:18:14 -04:00
Harwinder Sekhon 65a1debc4e gitignore: cover M5 trail-grid research CSVs
Claude-Session: https://claude.ai/code/session_01VPNF37MP9Kuv62B1VBrZ1b
2026-07-21 19:39:08 -04:00
Harwinder Sekhon 1499bb7cf2 M5 trail dist 0.85 (validated grid, arm unchanged)
Same-day sibling of the M15 campaign, opposite finding: on M5 the arm
point barely matters — winners need a looser leash. 5yr grid plateau
arm 0.5-0.7 x dist 0.85-1.0; shipped the OOS-preferred edge 0.60/0.85:
+231.2R PF 1.30 vs +218.0R PF 1.29 baseline, both halves better, DD
flat (19.2 vs 19.1), OOS +14.5R vs +11.1R. dist 1.0 rejected at the
OOS gate despite best in-sample total. XAU_PB pb_trail unaffected.
Factory replay reproduces the grid cell; 201/201 tests.

Claude-Session: https://claude.ai/code/session_01VPNF37MP9Kuv62B1VBrZ1b
2026-07-21 19:13:01 -04:00
Harwinder Sekhon 902122dbfe M15 trail arm 0.70 (validated grid) + backtest/live parity hardening
Parity: replay now applies the live _apply_regime_filter_live method
under the live guard (function-identity, default-off); unregistered-pair
fallback applies the config.json overlay; parity sweep gains direct
ATR/vol-regime assertions; regime-filter semantics test added.

Trail campaign: --trail-arm/--trail-dist research flags; 30-cell grid on
5yr M15 refuted the earlier-arm hypothesis and found the 0.70-0.80 arm
plateau. Shipped M15-only trail_arm_r=0.70 (factory override):
+210.7R PF 1.35 WR 57.8% vs +175.7R PF 1.30 WR 53.9% baseline; both
halves and 2024-06->2025-03 OOS better (+57.4 vs +45.4); 5yr max-DD
clause (33.9R vs 29.1R) explicitly waived by owner. M5 unchanged.
Factory replay reproduces the grid cell trade-for-trade; 201/201 tests.

Claude-Session: https://claude.ai/code/session_01VPNF37MP9Kuv62B1VBrZ1b
2026-07-21 16:40:33 -04:00
Harwinder Sekhon f0f9dcb169 Remove dead code: legacy config fields, unused helpers/imports, stray files
- Delete test_ws_dual.py (standalone probe, unused) and a stale tracked log
- config.py: drop MarketSession enum, is_jpy_pair, candle_buffer_seconds,
  and the never-read legacy SignalConfig/XauStrategyConfig fields
- data_provider.py: drop uncalled get_latest_price/get_spread_estimate
- Remove unused imports (engine, telegram_bot, main, news_engine)
- test_all.py: drop/trim the corresponding defaults assertions (199 tests)
- CLAUDE.md updated to match

Zero behavior change: no production code read any removed field
(verified reference-by-reference); suite green at 199/199.

Claude-Session: https://claude.ai/code/session_01VPNF37MP9Kuv62B1VBrZ1b
2026-07-21 14:06:55 -04:00
Harwinder Sekhon c8a2c83820 Merge branch 'centralized-config': config.json deployment overlay
Centralized deployment settings (Telegram credentials, data-provider
keys/selection, scan pairs/timeframes, per-pair base blocked hours,
alert quality, log level) into a gitignored config.json with env-var
precedence. Hardcoded credentials removed from source. Zero signal-
logic change — parity pinned by tests (200/200).

Claude-Session: https://claude.ai/code/session_01VPNF37MP9Kuv62B1VBrZ1b
2026-07-21 13:44:34 -04:00
Harwinder Sekhon 2d3e524f1c Final review fixes: scrub tracked credentials, trim scannable TFs, absolute parity pin, scanning type-checks
Claude-Session: https://claude.ai/code/session_01VPNF37MP9Kuv62B1VBrZ1b
2026-07-21 13:42:00 -04:00
Harwinder Sekhon f017f5aef5 gitignore: cover .env.* swap files and the whole logs/ directory
Rotated logs (telegram.log.YYYY-MM-DD) don't match *.log and a vim
.env.swp was nearly committed by a bulk add.

Claude-Session: https://claude.ai/code/session_01VPNF37MP9Kuv62B1VBrZ1b
2026-07-21 13:30:21 -04:00
Harwinder Sekhon a56a622846 Docs: correct PlatformConfig overlay scope (credentials never pass through the factory)
Claude-Session: https://claude.ai/code/session_01VPNF37MP9Kuv62B1VBrZ1b
2026-07-21 13:30:15 -04:00
Harwinder Sekhon b9633389fa Docs: config.json deployment overlay (CLAUDE.md repo map, env table, test count)
Documents the app_config.py / config.json overlay shipped in Tasks 1-4:
repo-map rows for app_config.py and config.json.example, env-var table
precedence (env -> config.json -> none, hardcoded fallbacks removed
2026-07-21), PlatformConfig/pair_signals overlay call order, and the
real 199-tests/13-modules suite count. Also fixes a stale SCAN_PAIRS
comment in telegram_bot.py that still named PlatformConfig.scan_pairs
as the deployment switch instead of config.json's scanning.pairs.

Claude-Session: https://claude.ai/code/session_01VPNF37MP9Kuv62B1VBrZ1b
2026-07-21 13:25:21 -04:00
Harwinder Sekhon 6e2fdbd684 telegram_bot: credentials/quality/scan-loops from config.json, startup guard
- Credentials, alert quality, scan pairs/timeframes, log level and
  candle-loop intervals now resolve via app_config (env -> config.json
  -> code default) instead of hardcoded fallbacks.
- _get_shared_dp() uses make_data_provider() for provider-registry
  dispatch.
- main() refuses to start without a token/chat_id, raising SystemExit
  with an actionable message instead of a bare sys.exit string.
- test_all.py Module 13: two new tests covering credential/quality/
  loop wiring via a reload, and the startup guard.

Claude-Session: https://claude.ai/code/session_01VPNF37MP9Kuv62B1VBrZ1b
2026-07-21 13:18:05 -04:00
Harwinder Sekhon 80a2b89769 Data provider: key from config.json/env, provider registry, drop hardcoded key
Resolves the Twelve Data API key through app_config.provider_api_key()
(env → config.json → no default) instead of a hardcoded field on
PlatformConfig. Adds data_provider.PROVIDER_REGISTRY + make_data_provider()
so signal_aggregator and the backtester build the active provider through
one seam; telegram_bot.py gets the same one-line WS-key fix now (per Task 4's
already-planned edit for that line) so the suite doesn't crash on import
between this commit and Task 4's full wiring pass.

Claude-Session: https://claude.ai/code/session_01VPNF37MP9Kuv62B1VBrZ1b
2026-07-21 13:10:58 -04:00
Harwinder Sekhon 1610507d57 Wire config.json overlay into the pair factory (before per-TF deltas)
Implements Task 2: apply_pair_overrides in xauusd.make_config before
the validated per-TF deltas, so e.g. M5 hour-9 appends on top of the
JSON base. Updated Module 9 equivalence test and added three parity pins.

Claude-Session: https://claude.ai/code/session_01VPNF37MP9Kuv62B1VBrZ1b
2026-07-21 12:59:12 -04:00
Harwinder Sekhon 1dab14d298 app_config: reject non-object config.json sections with AppConfigError
Claude-Session: https://claude.ai/code/session_01VPNF37MP9Kuv62B1VBrZ1b
2026-07-21 12:54:46 -04:00
Harwinder Sekhon cbc01565a2 Add app_config.py: config.json loader, validation, env-precedence resolvers
Claude-Session: https://claude.ai/code/session_01VPNF37MP9Kuv62B1VBrZ1b
2026-07-21 12:47:48 -04:00
Harwinder Sekhon 53b65e1bbb Plan: centralized config.json implementation (5 TDD tasks)
Claude-Session: https://claude.ai/code/session_01VPNF37MP9Kuv62B1VBrZ1b
2026-07-21 12:44:24 -04:00
Harwinder Sekhon 1a141abc7c Add config.json.example template; gitignore config.json
Claude-Session: https://claude.ai/code/session_01VPNF37MP9Kuv62B1VBrZ1b
2026-07-21 12:32:28 -04:00