146 Commits

Author SHA1 Message Date
Harwinder Sekhon 887aaed6f2 dashboard: skip DOM rebuild on unchanged poll, publish-time staleness, fix boundary countdowns
poll() now compares raw payload text and re-renders only what a changed
file can affect, preserving selection, disclosure, and scroll state. Hero
freshness reflects publish time (STALE warning past 15 min while market
open), and countdowns resolve their wrap interval from data-interval
instead of reverse-looking shared TF boundaries in snapshot.timeframes.
2026-08-09 18:15:31 -04:00
Harwinder Sekhon 7e8ed754ff news: reduce headline max age from 24h to 12h, make configurable
- Added headline_max_age_seconds to NewsConfig (default 43200 = 12h)
- Replaced hardcoded 86400 in _fetch_news_sentiment_raw with config ref
- All downstream consumers (keyword scoring, LLM scoring, LLM summary,
  agent panel macro/contrarian) auto-filter through the same path
2026-08-09 16:49:48 -04:00
Harwinder Sekhon 9769a0c84d agent_panel: port ai-hedge-fund prompt rubric — stance rules, anchored confidence, no-invention
Prompt-only change (static constants; plain-language output contract
byte-identical). Ported from virattt/ai-hedge-fund's persona rubric:

- _TAKE_CONTRACT hard rules: reason only from the JSON given, never
  invent numbers/events/levels; thin or missing evidence -> neutral
  with confidence below 30 (deferring to role-specific rules).
- Anchored confidence bands in the contract and synthesizer, replacing
  the unanchored "how sure you are".
- Per-stance support/oppose/neutral criteria for macro, technical, and
  risk (risk's missing-setup->oppose folded into its oppose bullet).
- One plain-language support example beside each oppose example for
  macro/technical/risk, fixing the all-oppose few-shot bias.
  Contrarian's designed skeptic tilt untouched.

News-engine prompts untouched (pinned by 78/78 + 18/18 validations).
374 tests green; real-DeepSeek smoke: macro correctly answered
neutral/20 on empty news context, confidences spread 20-60 across the
bands, all notes <=25 words, no jargon.

Claude-Session: https://claude.ai/code/session_01TyZx9iRJwfbPTPnCBqcwTR
2026-08-08 10:48:10 -04:00
Harwinder Sekhon 964239e67d docs: multi-provider LLM layer plan (DeepSeek news / Anthropic panel)
Reference plan only - no code changes. Describes the BaseLLMClient
refactor, AnthropicClient on the official SDK, per-consumer routing via
a config.json "llm" block, tests, commit sequence, and effort estimate
for the day we route the agent panel to Claude while news stays on
DeepSeek.

Claude-Session: https://claude.ai/code/session_01SisdfkxnXNL9iZsXxN3T15
2026-08-07 23:02:08 -04:00
Harwinder Sekhon 5e3758bd53 agent_panel: plain-language output — simple, short, no jargon/acronyms
Owner request: panel notes and rationale must read like plain everyday
English for a non-trader. Style rule added to the shared take contract
and synthesizer prompt, all few-shot examples rewritten in the target
voice (examples carry the style), and the Telegram votes line now spells
out full stances ("macro: support") instead of "macro:sup".

Before: "BUY against Bollinger SELL and price above upper band; 4h
neutral, mixed MTF"  →  After: "The price is above its usual range, and
the bigger picture is not clearly rising, so buying now is risky."

Claude-Session: https://claude.ai/code/session_01SisdfkxnXNL9iZsXxN3T15
2026-08-07 22:50:10 -04:00
Harwinder Sekhon c146fffb5b docs: agent panel module + config block
Claude-Session: https://claude.ai/code/session_01SisdfkxnXNL9iZsXxN3T15
2026-08-07 22:35:15 -04:00
Harwinder Sekhon 38c81251e8 cli+dashboard: --panel flag, panel dict keys, TF-dialog panel section
main.py: --panel runs the agent panel on demand (force bypasses
enabled/surface/quality toggles; daily budget + LLM cap still apply),
prints the verdict + persona takes, and report_to_dict carries the four
panel keys — which auto-propagates them into the dashboard's
snapshot.json via report_to_webapp_dict. dashboard_publisher attaches
in scan_tf gated by agent_panel.dashboard (default off); dashboard.html
renders an "AI panel (advisory)" section in the TF detail dialog.

Claude-Session: https://claude.ai/code/session_01SisdfkxnXNL9iZsXxN3T15
2026-08-07 22:34:30 -04:00
Harwinder Sekhon a7c2bfc7da telegram: attach AI panel to alert-quality sends (advisory block)
Auto-scan and level-breach alert paths attach the panel AFTER the
quality gate and dedup decision (no LLM spend on suppressed alerts),
via run_in_executor so the event loop stays free. Alert message gains a
two-line 🤖 AI Panel block (verdict + confidence + rationale, persona
votes); /status shows the daily panel budget when the feature is on.

Claude-Session: https://claude.ai/code/session_01SisdfkxnXNL9iZsXxN3T15
2026-08-07 22:32:57 -04:00
Harwinder Sekhon 7be2231eb3 agent_panel: persona/synthesizer advisory consensus module
ai-hedge-fund-style panel: 4 DeepSeek personas (macro, technical, risk,
contrarian) review a finished SignalReport concurrently on a
panel-private thread pool, then a synthesizer chair issues
AGREE/NEUTRAL/OPPOSE + confidence + rationale. Advisory/display-only by
construction: consumers attach post-hoc via attach_panel; the fields
live on SignalReport with empty defaults and analyze_from_dataframes
never touches them (backtest parity, locked by test).

Spend control on top of the shared 500/day client cap: max_panels_per_day
UTC budget (default 20), exact per-bar result cache, and a reuse TTL
keyed (symbol, tf, direction) so a persisting signal re-serves its
verdict across bars instead of burning 5 requests per bar. <2 valid
takes skips the synthesizer; any failure degrades to no panel.

Claude-Session: https://claude.ai/code/session_01SisdfkxnXNL9iZsXxN3T15
2026-08-07 22:31:50 -04:00
Harwinder Sekhon 0b604f1ad0 config: agent_panel block (enabled/surfaces/min_quality/daily budget)
New optional config.json block for the upcoming LLM agent-panel feature:
enabled, telegram, dashboard (per-surface toggles), min_quality,
max_panels_per_day, reuse_ttl_seconds. Absent block → {} → feature dark.
Validation mirrors the deepseek block. Module 23 opened with parse +
reject tests.

Claude-Session: https://claude.ai/code/session_01SisdfkxnXNL9iZsXxN3T15
2026-08-07 22:26:43 -04:00
Harwinder Sekhon 250078c987 docs: architecture diagram (PNG + matplotlib generator)
End-to-end map: external services → providers/news layer → signal core
(engine + factory + aggregator) → SignalReport → four delivery surfaces,
plus the backtester replaying the same core. Regenerate with
venv/bin/python docs/architecture_diagram.py after architecture changes.

Claude-Session: https://claude.ai/code/session_01Aq1jZ6jBsBdEC2xT3u46SA
2026-08-07 09:21:52 -04:00
Harwinder Sekhon d1d002a4f7 dashboard: clickable news — detail dialogs + on-demand AI summary endpoint
- news_engine: RSS parser captures link/description/publisher (three new
  optional NewsSentiment fields); _clean_html_text/_safe_link helpers
- news.json: headlines carry id = sha1(headline) + enrichment fields
- static/news_summary.php (NEW): on-demand DeepSeek gold-impact summary,
  whitelist-gated against news.json ids, cache-forever in ai/, fail-closed
  file-based daily cap (150/day), key at /etc/gold-dashboard/deepseek.key;
  installed to the docroot by install_static
- dashboard.html: newswire + calendar detail dialogs (score tiles, source
  description, article link, AI summary with loading/error states);
  delegated on stable containers, dialog keyed by id across polls
- tests: +6 (357 total); docs: README-dashboard §6 one-time setup
  (key file, apache-owned ai/, SELinux httpd_can_network_connect), CLAUDE.md

Claude-Session: https://claude.ai/code/session_01Aq1jZ6jBsBdEC2xT3u46SA
2026-08-07 09:08:49 -04:00
root edefd90aaa oanda: retry once on transient 401/5xx/network faults in candle fetch
The practice API rejects ~15-20 requests/day with a spurious 401
"Insufficient authorization" while the same token succeeds on adjacent
requests; each one silently cost a full scan cycle on that timeframe.
One retry after RETRY_DELAY_S (2 s) absorbs these plus transient
5xx/network faults. Real credential failures fail both attempts and
raise as before; other 4xx (bad request, 429) are never retried.

Tests: transient-401 recovery, persistent-401 raise, 400 not retried.

Claude-Session: https://claude.ai/code/session_01Tujtmmhxj9mxKjPFSoVjmP
2026-08-06 23:06:29 -04:00
Harwinder Sekhon b21c75a1f8 dashboard: Gold Desk v2 UI — light/dark theme, TF detail dialog, wire score
Owner-supplied page design replacing the dark-only v1: Bootstrap 5
colour tokens with a luma-inverted dark mode (toggle + system-pref +
localStorage), compact clickable TF cards opening a full-detail modal
(all-nine indicator table with value/conf/weight, price ladder of key
levels vs spot, per-confirmation-TF drilldowns, provenance), newswire
aggregate sentiment box, Inter/JetBrains Mono webfonts.

Re-applied the c70f58a sync fixes the source copy predated: 5 s
conditional-GET polling and the +2 s boundary refetch sniper. Data
logic otherwise unchanged; all fields verified against
report_to_webapp_dict output.

Claude-Session: https://claude.ai/code/session_017mbXs9HRawZhD1uZ3ze7zQ
2026-08-06 22:44:48 -04:00
Harwinder Sekhon a2f951f5c9 dashboard: cut page sync latency to ~scan-time + ≤5 s
Three display-side fixes, no signal-logic changes:
- run_cycle publishes snapshot.json BEFORE the news refresh — news I/O
  (FF + 4 RSS + possible DeepSeek calls) took 20-30 s on TTL-expired
  cycles and kept fresh signals off the page (publish_all split into
  publish_snapshot/publish_news, kept for boot)
- page polls every 5 s (was 20 s) using conditional GETs: dropped the
  ?t= cache-buster + no-store so Apache answers unchanged files with
  bodyless 304s
- boundary refetch sniper re-armed: fires 2 s after next_scan_epoch
  regardless of distance (the old delay<POLL_MS guard would have been
  dead code at 5 s polling)

New Module 20 test locks the snapshot-before-news ordering. 352 tests.

Claude-Session: https://claude.ai/code/session_017mbXs9HRawZhD1uZ3ze7zQ
2026-08-06 22:35:13 -04:00
Harwinder Sekhon 858c12280a news: DeepSeek validation, vs_forecast fix, /status source health
Validation (llm_news_eval.py harness + 78-headline golden CSV + verdict
doc): headline scoring 78/78 vs keyword baseline 24/78 — validated;
event gold-direction calls misread close K/M numeric comparisons ~11%
of the time (both claims cases in sample, JOLTS 8.79M vs 8.84M).

Fix: event payload now carries code-computed "vs_forecast" (from
EconomicEvent.surprise) and the prompt gains an explicit inverted-
metrics rule + worked claims example — iteration 1 (field alone) fixed
the arithmetic but exposed a masked inversion error. 18/18 on live
re-verification (4 misreads fixed, 11 regression cases clean, 3 fresh
both-side probes).

/status: new "News & AI" section — per-source health for the calendar
chain (which of FF/MyFXBook served), each RSS feed, and DeepSeek
(daily count, last error kind), written passively by the existing
fetch paths under _cache_lock. Failures WARN once on transition
(repeats DEBUG, recovery INFO) so the rotating log carries outage
stories without 10-min spam.

test_all.py: +5 Module 19 tests (351 total); also carries the Module 22
fixed-RR-TP tests from the prior research session (shared file).
Replay artifacts dir gitignored like commit_ladder.

Claude-Session: https://claude.ai/code/session_017mbXs9HRawZhD1uZ3ze7zQ
2026-08-06 22:21:02 -04:00
Harwinder Sekhon 525a1cd1c1 research: --fixed-rr-tp backtest flag + 2026-08-05 verdict (NOT shipped)
Hard TP at N×risk beats the trail on all 2024+ windows but the grid is
monotone to 3R (trend artifact) and M15 rr20 goes negative in 2021+2023
with doubled DD — trailing exits stay. Flag is research-only; its
Module 22 tests land with the next commit (shared test_all.py).

Claude-Session: https://claude.ai/code/session_017mbXs9HRawZhD1uZ3ze7zQ
2026-08-06 22:20:43 -04:00
Harwinder Sekhon b693f757f8 Remove OANDA_MIGRATION_ROADMAP.md (reference-only doc, superseded)
Deletion was already staged in the working tree by the owner; the
roadmap was approved 2026-07-24 as reference only and its plan file
lives outside the repo. Content remains in git history.

Claude-Session: https://claude.ai/code/session_01UnZkzeFcTu9agFg8BDh2tn
2026-08-04 20:26:54 -04:00
Harwinder Sekhon 7dbcbc6c82 audit: NY-anchored weekend calendar (all 3 copies) + FB2_EMA_LO crash-proofing
Owner decisions from the 2026-08-04 audit report:
- spread-in-exit-triggers: DECLINED (books stay comparable; ledger in
  CLAUDE.md records the measured ~4-6% optimism).
- weekend clock: FIXED. The market week is Fri 17:00 -> Sun 18:00 New
  York time, not Fri/Sun 22:00 UTC — during US DST the Oanda feed stops
  at 21:00 UTC (measured: 159/162 summer weekends). All three weekend
  implementations (providers/base, signal_aggregator, gpu_backtest
  calendar filter) now share the NY-anchored rule; a new test sweeps
  them hourly across both DST transitions and asserts agreement. No NTP
  needed — the host clock was fine, the constant was wrong.
  Ends the weekly false-"STALE DATA / feed lagging" hours (summer Fri
  21-22 UTC, winter Sun 22-23 UTC).
- FB2_EMA_LO (delegated): kept as a research knob but crash-proofed —
  an unparseable value degrades to the tuned default 0.5 with a warning
  instead of killing every scan. Documented in CLAUDE.md.

Book gates: M15 12mo pinned book byte-identical (+78.7111R, 397
trades). M5 12mo drops exactly ONE trade (556 -> 555, +97.56 ->
+98.64R): a phantom whose signal fired on the final summer-Friday bar
(decision 21:00 UTC = NY 17:00 market close) and was "filled" at
Sunday's reopen 49 hours later, -1.08R — an entry live trading can
never take. The backtest no longer contains it; live == backtest
parity improved. M15 unaffected because hour 21 is in its blocked set.

Suite: 341/341 green.

Claude-Session: https://claude.ai/code/session_01UnZkzeFcTu9agFg8BDh2tn
2026-08-04 20:21:34 -04:00
Harwinder Sekhon 5b3f47f3e1 audit: scheduler cleanup, WS-log secret hygiene, simulator test coverage, CLAUDE.md sync (Batch G)
2026-08-04 Fable 5 QA audit, final code batch.

- Boundary math: removed the provably-dead `remaining <= buffer` guard
  from all three scheduler copies (518,400-instant frozen-clock sweep:
  never fired) and documented the exact-boundary-instant semantics.
- websockets/websockets.client loggers pinned to INFO so a DEBUG run
  can never write the TD key (connect URI) or Tiingo auth frame into
  the 30-day rotating log. (.env also chmod 600, untracked.)
- Cooldown pass factored into _apply_cooldown (pure, testable) — the
  only sequential replay rule finally has direct tests.
- New simulator behavior tests (previously ZERO coverage of the trade
  resolver): same-bar SL+TP tie → SL-first, gap-through-SL fills at
  the open, TIMEOUT pricing + round-trip spread charging, trailing
  ratchet arms at bar close and fills next-bar-only, cooldown ordering.
- CLAUDE.md synced to verified reality: 338 tests/21 modules; three
  weekend-implementation copies caveat; throttle claim corrected
  (single-flight, TD-only throttle); daemon trail/cooldown parity
  caveats (once-per-OBSERVED-bar, wall-clock cooldown) + OPEN_ONLY and
  reconcile notes; CP-path gate semantics documented as validated
  behavior; new "known open items" section for the owner-decision
  findings (weekend DST constant, spread-in-trigger rule-6 item, H4
  seeding divergence, FB2_EMA_LO).

Gates: suite 338/338 green; pinned M15 book and M5 HEAD baseline both
reproduced identically after the cooldown refactor.

Claude-Session: https://claude.ai/code/session_01UnZkzeFcTu9agFg8BDh2tn
2026-08-04 20:08:37 -04:00
Harwinder Sekhon 99eb268cf3 audit: alert delivery, provider robustness, backtester NaN/cache guards (Batches C+E+F)
2026-08-04 Fable 5 QA audit. Three batches sharing one test file, all
verified by runnable repros before fixing, all gated below.

Batch C — telegram alert delivery (verified by h4 harness):
- Failed breach rescan shrinks the cooldown to 30 s instead of
  silencing the symbol for 5 min; breach during cooldown is PARKED and
  fired on the first tick after expiry (was dropped forever).
- Level cache keyed symbol→TF (M5 scans no longer wipe H1/H4 swing
  levels within minutes); breach labels TF-qualified.
- Markdown alert sends get a plain-text fallback (mirroring _safe_edit);
  fingerprint commits only when a message actually landed.
- Per-key asyncio.Lock closes the dedup TOCTOU between auto-scan and
  breach sends (forced-interleave repro produced duplicate alerts).
- _start_scheduler respawns dead per-TF loops (was all-or-nothing).
- Clean WS close now backs off (was a zero-delay reconnect hot loop);
  backoff resets only after a connection that received ticks; TD
  iter_ticks fires on_connect on the first RECEIVED frame (invalid key
  can no longer pin the backoff at minimum).
- Scan-failure logs carry the real traceback (was "NoneType: None").

Batch E — providers (verified by h5 harness):
- fetch_range: 3 consecutive chunk ERRORS raise instead of returning
  silently truncated history (verified 731-day silent gap repro; also
  covers 429 storms); pagination no-progress guard; head-truncation
  ERROR log.
- Single-flight per (symbol,TF): 4 concurrent scans = 1 API call (was
  4); cache records requested depth so a longer-lookback call refetches.
- Multi-bar forming/future tail fully dropped (was iloc[:-1] once).
- Oanda: malformed candles skipped, never 0.0-defaulted; cleaning drops
  non-positive-price rows BEFORE OHLC repair; inf returns neutralized so
  one corrupt bar can no longer disarm the whole 10-sigma filter.

Batch F — backtester internals (verified by h6 harness):
- NaN bar in a trailing window exits at the last valid bar (was NaN R
  poisoning every metric); static-path TIMEOUT NaN guard;
  compute_metrics excludes non-finite R loudly. Zero real-data exposure
  (0 NaN rows in all 10 Oanda parquets) — defensive.
- fetch_history ERROR-logs pre-2026-07-08 cache files (cleaned-bars era)
  when loaded for raw-path resolution.

Gates: suite 333/333 green; pinned M15 12mo book reproduced identically
(397 trades, +78.7111R) after E and after F; M5 HEAD-config baseline
reproduced identically (556 trades, +97.5604R) after F.

Claude-Session: https://claude.ai/code/session_01UnZkzeFcTu9agFg8BDh2tn
2026-08-04 20:02:57 -04:00
Harwinder Sekhon 18e4fd0403 audit: gate-chain verdicts — lock shipped M5 entry-path state, document hour-delta isolation
2026-08-04 audit, Batch D. Verification outcomes (no behavior changes —
comment + tests only):

- CP gate-bypass (quality forced to B after the gate chain, exhaustion/
  M5-score gates not re-checked) is VALIDATED behavior, not a bug: the
  pinned shipped-config M5 12mo book carries 226/586 XAU_CP trades
  (+38.9R), 104 with EXHAUST warnings (+15.3R), 175 below the M5
  min-score gate (+28.0R). Removing the "holes" removes validated
  profit. New test pins the shipped state (PB off, CP BUY-only on M5,
  ema_partial_neutral M5-only) so any flip is a conscious revalidated
  decision.
- E18 (xau_strategy vs backtest blocked-hours "drift") REFUTED as a
  defect: the divergence is required by pair isolation — the backtest
  copy is the generic non-XAU fallback and must not inherit
  gold-validated deltas. Documented in the factory + a test asserting
  the delta stays XAU-side.
- 6c-upgrade re-arming an exhaustion-D refuted (exhaustion zeroes
  confidence below 6c's floor; regimes mutually exclusive). "WEEKEND
  FILTER" CP-suppression branch is defense-in-depth for direct
  analyze_from_dataframes callers, kept.

Suite: 316 green.

Claude-Session: https://claude.ai/code/session_01UnZkzeFcTu9agFg8BDh2tn
2026-08-04 19:52:07 -04:00
Harwinder Sekhon ebb83749ef audit: GPU-engine cache-miss observability + dead-guard alignment + full parity test
2026-08-04 audit, Batch B. Verified first (h1 harness): the two engines
are byte-identical on every reachable input — 370-prefix flat-bar deck,
warmup band, 322 real-data M15 positions incl. all 7 zero-range bars,
zero diffs. The claimed ADX NaN desync / warmup-guard divergences are
unreachable below the shared 30-bar analyze() floor. Changes here are
therefore observability + hygiene, proven behavior-neutral by exact
reproduction of the pinned M15 12mo book (397 trades, +78.7111R,
byte-identical trade rows).

- _resolve_pos: cache-miss fallback to live-engine recompute now warns
  on first miss and counts every miss (was fully silent — a run could
  switch math with no trace).
- ROC/Keltner warmup guards textually aligned with the live engine so a
  future period/floor change can't open a silent divergence window.
- New regression: full-sweep parity test comparing ALL indicator
  signals/confidences, composite, and regimes on every prefix incl. a
  flat zero-DM/zero-TR run (the old parity test skipped
  ADX/MACD/ROC/Keltner/RSI entirely); plus a loud-fallback test.

Suite: 314 green. Book gate: oanda_m15_12mo_030 reproduced identically.

Claude-Session: https://claude.ai/code/session_01UnZkzeFcTu9agFg8BDh2tn
2026-08-04 19:49:08 -04:00
Harwinder Sekhon 8acc1e18f8 audit: harden Oanda execution lifecycle (close-fill, netting, races, fail-closed config)
2026-08-04 Fable 5 QA audit, Batch A. All fixes are execution-robustness
only — no signal code touched (git diff scope: daemon/executor/tests).
Findings verified by runnable repros before fixing (h3 harness):

- close_trade: a 200 whose close order was created but NOT filled (e.g.
  MARKET_HALTED during the settlement pause) now raises instead of
  returning; the daemon's timeout path keeps tracking and retries. The
  old path popped state and orphaned a live position forever
  (reconciliation was boot-only).
- place_market_order: positionFill=OPEN_ONLY — on a netting account an
  opposite-side order can no longer silently close an untracked
  position; the divergence surfaces as an order cancel.
- _execute_trade: a fill without tradeOpened (netting close/reduce) and
  any placement OandaAPIError (server may have filled before a network
  timeout) now trigger _reconcile_positions instead of silent return.
- _discover_existing_positions → lock-guarded _reconcile_positions,
  callable mid-run; adoption of a position with NO broker stop now
  screams (CRITICAL + per-bar warning) instead of computing sl=entry,
  risk=0 and silently never trailing.
- _manage_trailing: snapshot deep-copies per-trade dicts (no mutation of
  live state outside the lock); write-back skips trades popped by
  _check_closed_trades mid-iteration (no ghost resurrection).
- Fail-closed config: unknown OANDA_QUALITY_GATE refuses to start
  (previously a typo silently became the most permissive gate "C");
  non-integer OANDA_LOT_SIZE/MAX_HOLD_BARS exit with a clear message.

Suite: 302 → 312 tests (Module 21), all green.

Claude-Session: https://claude.ai/code/session_01UnZkzeFcTu9agFg8BDh2tn
2026-08-04 19:46:37 -04:00
Harwinder Sekhon 28a1c375bd M5 at real cost: disable XAU_PB, squash partial EMA alignment
Both validated at the measured $0.65 spread on Oanda bars (12mo halves +
PF, OOS 2024-06->2025-03, 5yr; full evidence in pair_signals/xauusd.py
M5 overrides 4-5):

- m5_pullback_enabled False on M5: loss forensics identified XAU_PB as
  the largest wrong-direction loss source; 12mo PF 1.26 (failing the
  acceptance bar) -> 1.44, max DD 23.9R -> 13.9R. PB code stays for
  future regime-gated experiments.
- New TechnicalConfig.ema_partial_neutral (default False), True on M5
  only: partial 4-EMA ribbon alignment (2-3 of 4 checks) scores NEUTRAL
  instead of a weak BUY/SELL (-0.008R/84 vs +0.230R/341 fully aligned).
  Squash mirrored in the backtester's precomputed EMA scorer in this
  same commit (live==backtest parity). 12mo +97.6R/556 PF 1.48 DD
  10.7R halves +47.8/+49.8; 5yr +127.4R PF 1.38; OOS one-trade tie.
- Rejected and logged (memory + docstring): hour-19 block (OOS fail),
  hours-14/15, trail dist 0.75, confirm-TF swaps incl M15+H4 (OOS
  ordering), divergence/candlestick weight changes, bollinger removal.
- Tests 296 -> 302: partial-alignment fixture test, factory-equivalence
  mirrors, per-TF isolation test, BC-gate fixture force-enables PB.

Claude-Session: https://claude.ai/code/session_01H1eKWZsAVapJ1C4winYLqX
2026-08-04 16:54:09 -04:00
Harwinder Sekhon 775330ff16 /news: normalize rounded-zero score so "-0.00" renders as "+0.00"
A balanced feed (XAU and USD means nearly cancelling) yields a tiny
negative net that "+.2f" rendered as "-0.00" — seen live 2026-08-03.
Regression test uses that day's actual per-currency means (300 tests).

Claude-Session: https://claude.ai/code/session_017975c1TufZ3xRYAq8iWTCS
2026-08-03 18:06:44 -04:00
Harwinder Sekhon 1f54452007 Telegram /news command: net XAU/USD news sentiment score on demand
Owner-locked /news replies with the existing get_pair_sentiment() net
score (XAU mean - USD mean, -1..+1) and its label; a caveat line marks
the no-headlines case so a feed outage can't read as neutral. Fetch runs
in the shared executor with a placeholder-then-edit reply. Display-only:
news still never gates signals. Registered in handler block, command
menu, help text, docstring, and user guide; 3 new builder tests (299).

Claude-Session: https://claude.ai/code/session_017975c1TufZ3xRYAq8iWTCS
2026-08-03 18:01:42 -04:00
Harwinder Sekhon be5dd4a25b Dashboard publishes directly into /var/www/trade (no subfolder)
Adds exactly three files next to the portal: dashboard.html,
snapshot.json, news.json. README covers chown vs setfacl for write access.

Claude-Session: https://claude.ai/code/session_01Wc9ZoqKLYBT2QJcpRWKoGm
2026-08-03 17:17:37 -04:00
Harwinder Sekhon 0445a20a76 Dashboard out-dir default → /var/www/trade/gold-dashboard (live portal docroot)
Claude-Session: https://claude.ai/code/session_01Wc9ZoqKLYBT2QJcpRWKoGm
2026-08-03 17:14:08 -04:00
Harwinder Sekhon 4e654b5edb Web dashboard v2: static publisher + Apache, with calendar & news panels
- dashboard_publisher.py: standalone candle-close daemon that atomically
  writes snapshot.json/news.json/dashboard.html to an Apache-served dir
  (DASHBOARD_OUT_DIR). Fully independent of telegram_bot/oanda_daemon —
  local confirmation map + boundary math, no imports either way, no new
  deps, no open port. Due-TF classification from the RAW wake time.
- static/dashboard.html: self-contained dark page — hero rail, four TF
  cards (score bar, 3x3 indicator grid, MTF, setup, reasons), DeepSeek
  market context, week economic calendar, scored newswire. Clock offset
  from the HTTP Date header (publish-time epoch is stale between closes).
- news_engine.get_week_events(): additive read-only accessor over the
  cached FF week; no signal path calls it.
- deploy/: systemd unit + live-server (LAMP/SELinux) checklist.
- test_all.py: Module 20 (9 tests) + get_week_events test; 296 green.

Claude-Session: https://claude.ai/code/session_01Wc9ZoqKLYBT2QJcpRWKoGm
2026-08-03 17:10:29 -04:00
Harwinder Sekhon 9e1c321f05 Add commit-ladder campaign driver; ignore its 26MB results folder
Added: backtest/research/commit_ladder.py (recreates historical configs
via --set on the fixed harness; 2026-08-03 campaign confirmed HEAD is
the best config on all timeframes). Results stay local like all pinned
trade books.

Claude-Session: https://claude.ai/code/session_01Nsd8HcrV67GbCZXxZEn1Ma
2026-08-03 13:29:18 -04:00
Harwinder Sekhon 6fff06ff90 DeepSeek V4-Flash news enrichment: LLM sentiment, surprise direction, summary
Three display/warning-only upgrades to the news engine, all behind the
existing keyword path as fallback — signals are never gated on LLM
output (event_risk_caps_signals / sentiment_affects_signals stay False
per the 2026-07 audit; backtester use_news=False untouched):

- Headline sentiment: one batched call per 600s cache window (capped at
  25 new headlines, sha1-cached so repeats are never re-sent) overrides
  the keyword scores per currency with the gold-inverse doctrine encoded
  in the prompt.
- Release surprises: get_event_risk_level's description now says which
  way the print pulls GOLD ("— gold bearish: strong jobs lift yields"),
  fixing the documented numeric-only gap; risk level and windows
  unchanged.
- Market summary: 1-2 sentence macro blurb on SignalReport.news_summary,
  rendered in Telegram alerts (📰) and the CLI (🧠).

New llm_client.py: requests-only OpenAI-compatible client (no SDK), one
attempt per call, UTC-day request cap, singleton so the cap survives
per-scan NewsEngine reconstruction. Config: deepseek block in
config.json + DEEPSEEK_API_KEY env; absent key = fully inert.

Live-measured API behavior baked in: V4-Flash is a hybrid thinking model
— requests send thinking:disabled (reasoning otherwise blew the timeout
and could exhaust max_tokens before the answer); prompts are static
module constants so DeepSeek's prefix cache prices repeat calls at the
cache-hit rate; scores use compact [s, xau] pairs; timeout 12s.

Tests 273 → 286 (new Module 19 + deepseek config tests).

Claude-Session: https://claude.ai/code/session_01Nsd8HcrV67GbCZXxZEn1Ma
2026-08-03 13:25:02 -04:00
Harwinder Sekhon f7d8f1113c Tiingo WebSocket streaming: re-enable the level-breach monitor
The monitor has been dead since the Oanda feed switch (Oanda's provider
does not stream). New config.json scalar data_providers.streaming selects
a tick-stream provider independent of the candle feed; telegram_bot
resolves it via _get_streaming_dp() (separate instance, candle cache
stays on the active provider) and falls back to the active provider when
unset — behavior is byte-identical to before when the key is absent.

TiingoDataProvider grows the two sanctioned streaming overrides:
supports_streaming=True and iter_ticks() on wss://api.tiingo.com/fx
(mid price only — the no-spread rule extends to the stream). Two
deliberate divergences from the TD reference implementation, both
because Tiingo authenticates inside the socket: server "E" messages
raise TiingoDataError (a swallowed auth error would end the generator
cleanly and reconnect-hammer with zero backoff), and on_connect fires on
the first received message rather than after the subscribe send. Field
order of quote frames was live-verified: ask is data[7], not data[6] as
the docs table suggests (data[6] is askSize).

make_data_provider(config, name=None) now takes an explicit provider
name; keyless streaming providers disable the monitor with one warning
instead of a traceback-per-minute loop.

Tests 266 → 273 (WS parser, ticker round-trip, streaming config scalar,
explicit-name dispatch, streaming-dp resolution, monitor auto-disable).

Claude-Session: https://claude.ai/code/session_01Nsd8HcrV67GbCZXxZEn1Ma
2026-08-03 13:11:36 -04:00
Harwinder Sekhon bcda6e0ebd Cleanup: remove dead code, stale docs, and all non-main branches
- Delete back-compat shims data_provider.py / oanda_provider.py;
  test_all.py now imports providers.* directly (ForexDataProvider name
  retired; the two shim-only tests removed, 268 -> 266 tests)
- Delete backtest/research/calendar_store.py (no importer, no entry
  point); config.py audit comment reworded
- Delete executed plan doc (2026-07-21 centralized-config) and the two
  unbuildable guide PDFs; HTML guides remain the source of truth
- OANDA_MIGRATION_ROADMAP.md status header corrected: phases 1-2
  shipped 2026-07-28, feed is live on Oanda; phases 3-6 still unbuilt
- Branches deleted locally + on both remotes: oanda_demo (merged),
  web-dashboard, eurusd-modular, reversal-exit-alert (owner-approved
  permanent deletion); July stash dropped

Claude-Session: https://claude.ai/code/session_019zR58p8ACW1oJ5QXVUZ5es
2026-08-03 11:08:48 -04:00
Harwinder Sekhon 042b079861 Big-candle gate (default-OFF) + M5 relaxation campaign + --set tooling
Gate 6h (big-candle entry gate): blocks all XAU entries (core, XAU_PB,
XAU_CP) when the primary TF's last 24h contains a bar with range >= K x
prior-bar ATR. Ships default-OFF: validation rejected every threshold
(K 3-6 on M5/M15 12mo) — removed trades were net-POSITIVE at every cell;
M5 quiet-window trades are the only losing bucket. Config knobs
big_candle_atr_multiple/lookback_hours, --big-candle backtest flag,
Module 17 tests.

M5 relaxation SHIPPED (validated 2026-07-30, Oanda bars, $0.30 spread):
m5_min_abs_score 30->25, m5_min_confirmation_score 10->5, hour-21 UTC
unblocked M5-only. Beats control in ALL 6 calendar years of the 5yr
replay (+230.9R/2319 PF 1.26 DD 21.3 vs +168.0R/2048 PF 1.21 DD 27.0),
12mo +107.4R PF 1.36 vs +84.6R PF 1.33, OOS +26.1R vs +17.2R, $0.70
spread +86.9R vs +66.4R. M15: every relaxation retested negative —
config unchanged. Declined-but-open: M5 hour-22 block, M15 hour-16
unblock (both fail 5yr/OOS). Evidence in the factory docstring.

Tooling: generic --set PATH=JSON research override in gpu_backtest
(typo-proof, logged); diagnose_filters filter_tags refreshed for gates
6d-6i with a staleness-guard test (Module 18); research scripts
hour_analysis/burst_profile/book_diff. Suite: 268 tests.

Claude-Session: https://claude.ai/code/session_014jqHpmNYmwYMowF6bPZyeD
2026-07-30 11:30:20 -04:00
Harwinder Sekhon dc7beb2877 Rebuild CLAUDE.md: lean architecture reference
Replaces the deleted exhaustive knob-by-knob reference with a compact
guide: commands, config precedence, pipeline architecture, rules for
changes, and current-state notes (Oanda feed, TD-pinned books).

Claude-Session: https://claude.ai/code/session_014jqHpmNYmwYMowF6bPZyeD
2026-07-30 11:30:07 -04:00
Harwinder Sekhon 637c39e201 Switch active data source to Oanda v20 (practice feed)
config.json.example: data_providers.active = "oanda" (credentials stay
in env/.env or the top-level oanda block; TD and Tiingo remain
registered fallbacks). Backtest parquet cache is now keyed per provider
— Oanda and TD bars diverge (2026-07-24 study), so a feed switch must
never serve another feed's cached bars; twelve_data keeps the legacy
un-prefixed filename so pinned-book caches stay valid. 254 tests green;
live smoke-tested against the practice API (66 contract checks: all
scanned TFs, boundary/freshness guarantees, real spread $0.71,
fetch_range pagination, typed error paths, CLI + backtest end-to-end).

Claude-Session: https://claude.ai/code/session_01Mc7XKYyPySVXiUpcN82QEK
2026-07-28 20:12:59 -04:00
Harwinder Sekhon 083a4e8d63 Provider abstraction: providers/ package with BaseDataProvider template
One class per provider (TwelveData, Oanda, Tiingo), all subclassing
BaseDataProvider which owns cache, validation/cleaning, freshness +
boundary guarantees, fetch_range pagination, and streaming stubs —
subclasses implement only _fetch_raw. config.json data_providers.active
selects the live feed; data_provider.py / oanda_provider.py stay as
back-compat shims. Module 16 provider-abstraction tests (253 total green).

Claude-Session: https://claude.ai/code/session_01Mc7XKYyPySVXiUpcN82QEK
2026-07-28 20:09:00 -04:00
Harwinder Sekhon f760e9070a Research: Dukascopy real-spread study + Oanda/TD bar-divergence study
Scripts + summary CSVs for the 2026-07-24 studies (real XAU spread ~$0.65
tradable hours; Oanda vs TD bars diverge ~11% of ATR with 10% direction
flips). Large raw data files (parquet, per-boundary samples) gitignored.

Claude-Session: https://claude.ai/code/session_01Mc7XKYyPySVXiUpcN82QEK
2026-07-28 20:08:53 -04:00
Harwinder Sekhon 9a384935bf Add Oanda migration roadmap (reference, not scheduled)
Approved 2026-07-24: 6-phase plan for fully switching the signal feed
from Twelve Data to Oanda v20 (inert plumbing -> backtester agnosticism
-> pricing-stream monitor -> shadow parity -> book re-validation with
owner cost gates -> reversible cutover). TD remains a permanent
fallback. Grounded in the 2026-07-24 divergence study (10% bar-direction
flips, 11%-of-ATR noise, real Oanda spread ~$0.72 vs assumed $0.30).

Claude-Session: https://claude.ai/code/session_01YNvxXtgz62F7kTKZZsAAAE
2026-07-24 15:12:37 -04:00
Harwinder Sekhon d669227ddc Oanda demo daemon: auto-execution on practice account (XAU/USD)
New independent trading daemon (oanda_daemon.py) that places orders on
an Oanda v20 practice account instead of sending Telegram alerts.
Oanda-only market data (oanda_provider.py, mid candles + real bid/ask
spread) through the unchanged signal pipeline; order execution and
trade management via oanda_executor.py (raw requests, no new deps).

Book parity with the backtester: no broker-side TP (exits = SL/TRAIL/
TIMEOUT only), bar-close-only trail ratchet with each closed bar
processed exactly once, 5-bar per-TF entry cooldown, structural SL
from the setup. Accepted divergence: one position total (skipped
signals logged for book reconciliation).

Safety/ops: fail-fast boot on rejected credentials (transient outages
continue), market orders never retried (double-fill risk), entry lock
against aligned-boundary races, restart recovery + adoption via
oanda_state.json, hourly heartbeat, asyncio exception handler, every
failure path logged. Prices formatted at instrument precision.

Config: env OANDA_* -> config.json "oanda" section (validated in
app_config). Tests: Module 15 (30 tests), suite 240 green.

Claude-Session: https://claude.ai/code/session_01VUnCqjWBuvKRSjLwHeB9BJ
2026-07-24 14:10:45 -04:00
Harwinder Sekhon c3329d6cd5 Telegram alert redesign: verdict-first layout + informational 100-pip TP
format_signal_message restyled (display-only, zero signal-logic impact):
verdict header first (signal - pair - TF - quality), trade setup moved
above the detail, per-indicator breakdown condensed to one arrow line
(_indicator_summary), heavy dividers dropped. New TP 100p line in every
setup block: entry +/- 100 pips ($10.00 on XAU) with its R-multiple vs
the stop (_INFO_TP_PIPS, formatter-only - not stamped on TradeSetup,
not in CLI/JSON/backtester). Trailing-plan logic unchanged (stamped
per-TF params, pullback fallback). Removed dead _indicator_line and
_SIGNAL_TAG. Tests 210 -> 212 (100p TP math BUY/SELL/no-setup, one-line
indicator summary).

Claude-Session: https://claude.ai/code/session_01BoZKi67Ct6QTeiFJmybXSw
2026-07-24 10:58:22 -04:00
Harwinder Sekhon 7a810c4a4c Remove Tiingo spread monitor (indicative two-tier feed, unvalidatable)
Tiingo's free /tiingo/fx/top publishes only a two-tier indicative
spread for gold ($0.10 liquid hours / $0.70 rollover+news; verified on
273 production samples + live probes) — mid ± a posted increment, not
executable depth, so it can never validate the $0.30 backtest cost
assumption. Removed one day after shipping: tiingo_provider.py, the
sampler/quote plumbing in telegram_bot.py, the channel_spread_detail
config knob and tiingo_api_key resolver, and Module 14's Tiingo tests
(221 -> 210). CLAUDE.md documents the verdict: do not re-add a spread
display from Tiingo's free FX feed.

Claude-Session: https://claude.ai/code/session_01BoZKi67Ct6QTeiFJmybXSw
2026-07-24 10:56:55 -04:00
Harwinder Sekhon 3a95a37a0c Accuracy audit fixes: alerts carry validated per-TF trail plans + 13 more
Five-agent audit of the signal path; all fixes are delivery/measurement
side. Regression-proven behavior-neutral on validated signals: the 5yr
M15 replay reproduces the pinned book exactly (+210.65R PF 1.35
WR 57.8% MaxDD 33.90R). Tests 215 -> 221.

- TradeSetup carries trail_arm_r/trail_dist_r stamped from the producing
  config; Telegram trail plan reads them (was showing default 0.6/0.75
  instead of the validated M15 arm 0.70 / M5 dist 0.85)
- Alert dedup: fingerprint = (signal, quality, direction); entry movement
  compared against last-ALERTED entry/risk (old SL-dist bucket churned on
  ATR drift -> near-duplicate alerts every candle)
- Fresh fetches enforce the closed-bar boundary check with one 3s grace
  retry, then warn (feed lag no longer silently analyzes the prior bar)
- Backtest max DD includes the zero baseline (opening losing streaks
  were undercounted; pinned M15 DD unchanged)
- Replay loads the regime-filter TF when the H4-ADX gate is on (was a
  silent no-op for M5/M15 primaries)
- clear_cache under _cache_lock; dead ROC zero-cross branches removed
  from both engines (provably unreachable); RSS dates via
  parsedate_to_datetime (named zones no longer stamped UTC); event risk
  grades from the soonest event; .env inline comments stripped;
  auto-scan skips NEUTRAL like the breach path; report_to_dict honors
  the weight-0 skip rule and emits strategy/trail params; PB funnel
  counts spread_gate/accepted; session-filter fail-open paths log loudly
- Docs corrected to match validated code (H&S neckline = lower peak,
  exhaustion anchors farthest tracked swing) with re-validation warnings

Claude-Session: https://claude.ai/code/session_01R5soRUeRBi8kHMB99csrJy
2026-07-23 10:31:13 -04:00
Harwinder Sekhon f814485615 Tiingo bid/ask spread monitor + logging hardening
- tiingo_provider.py: /tiingo/fx/top quotes (10s TTL cache, graceful
  None on any failure); spread log writer. Signals never consume
  bid/ask; warn-only by design (no historical bid/ask to validate a
  gate against — live == backtest rule).
- app_config: repo-root .env loader (setdefault; shell env still wins),
  tiingo_api_key resolver (TIINGO_API_KEY/TIINGO → config.json),
  telegram.channel_spread_detail flag (default false).
- telegram_bot: 5-min spread sampler → logs/spread_log.csv (weekend-
  aware), alert-time quote refresh, bid/ask + wide-spread warning in
  signal messages, /status spread line. Raw numbers to owner DM always;
  channel only while channel_spread_detail=true (Tiingo free tier is
  internal-use-only; flip before public launch). Free-tier limit math:
  worst case ~41 req/h vs 50/h cap, one pair.
- Logging: 3h rotation x 240 backups (30-day retention), full-date
  stamps, force=True root handlers; candle loops + sampler survive
  failed cycles; breach-task exceptions logged via done-callback;
  PTB error handler; sys/threading excepthooks in main().
- test_all.py: Module 14 (tiingo, dotenv, spread routing, loop
  resilience, rotation) — 215 tests green.

Claude-Session: https://claude.ai/code/session_01HVRmy6KspMs96o5PTPYFAB
2026-07-22 17:18:14 -04:00
Harwinder Sekhon 65a1debc4e gitignore: cover M5 trail-grid research CSVs
Claude-Session: https://claude.ai/code/session_01VPNF37MP9Kuv62B1VBrZ1b
2026-07-21 19:39:08 -04:00
Harwinder Sekhon 1499bb7cf2 M5 trail dist 0.85 (validated grid, arm unchanged)
Same-day sibling of the M15 campaign, opposite finding: on M5 the arm
point barely matters — winners need a looser leash. 5yr grid plateau
arm 0.5-0.7 x dist 0.85-1.0; shipped the OOS-preferred edge 0.60/0.85:
+231.2R PF 1.30 vs +218.0R PF 1.29 baseline, both halves better, DD
flat (19.2 vs 19.1), OOS +14.5R vs +11.1R. dist 1.0 rejected at the
OOS gate despite best in-sample total. XAU_PB pb_trail unaffected.
Factory replay reproduces the grid cell; 201/201 tests.

Claude-Session: https://claude.ai/code/session_01VPNF37MP9Kuv62B1VBrZ1b
2026-07-21 19:13:01 -04:00
Harwinder Sekhon 902122dbfe M15 trail arm 0.70 (validated grid) + backtest/live parity hardening
Parity: replay now applies the live _apply_regime_filter_live method
under the live guard (function-identity, default-off); unregistered-pair
fallback applies the config.json overlay; parity sweep gains direct
ATR/vol-regime assertions; regime-filter semantics test added.

Trail campaign: --trail-arm/--trail-dist research flags; 30-cell grid on
5yr M15 refuted the earlier-arm hypothesis and found the 0.70-0.80 arm
plateau. Shipped M15-only trail_arm_r=0.70 (factory override):
+210.7R PF 1.35 WR 57.8% vs +175.7R PF 1.30 WR 53.9% baseline; both
halves and 2024-06->2025-03 OOS better (+57.4 vs +45.4); 5yr max-DD
clause (33.9R vs 29.1R) explicitly waived by owner. M5 unchanged.
Factory replay reproduces the grid cell trade-for-trade; 201/201 tests.

Claude-Session: https://claude.ai/code/session_01VPNF37MP9Kuv62B1VBrZ1b
2026-07-21 16:40:33 -04:00
Harwinder Sekhon f0f9dcb169 Remove dead code: legacy config fields, unused helpers/imports, stray files
- Delete test_ws_dual.py (standalone probe, unused) and a stale tracked log
- config.py: drop MarketSession enum, is_jpy_pair, candle_buffer_seconds,
  and the never-read legacy SignalConfig/XauStrategyConfig fields
- data_provider.py: drop uncalled get_latest_price/get_spread_estimate
- Remove unused imports (engine, telegram_bot, main, news_engine)
- test_all.py: drop/trim the corresponding defaults assertions (199 tests)
- CLAUDE.md updated to match

Zero behavior change: no production code read any removed field
(verified reference-by-reference); suite green at 199/199.

Claude-Session: https://claude.ai/code/session_01VPNF37MP9Kuv62B1VBrZ1b
2026-07-21 14:06:55 -04:00
Harwinder Sekhon c8a2c83820 Merge branch 'centralized-config': config.json deployment overlay
Centralized deployment settings (Telegram credentials, data-provider
keys/selection, scan pairs/timeframes, per-pair base blocked hours,
alert quality, log level) into a gitignored config.json with env-var
precedence. Hardcoded credentials removed from source. Zero signal-
logic change — parity pinned by tests (200/200).

Claude-Session: https://claude.ai/code/session_01VPNF37MP9Kuv62B1VBrZ1b
2026-07-21 13:44:34 -04:00
Harwinder Sekhon 2d3e524f1c Final review fixes: scrub tracked credentials, trim scannable TFs, absolute parity pin, scanning type-checks
Claude-Session: https://claude.ai/code/session_01VPNF37MP9Kuv62B1VBrZ1b
2026-07-21 13:42:00 -04:00
Harwinder Sekhon f017f5aef5 gitignore: cover .env.* swap files and the whole logs/ directory
Rotated logs (telegram.log.YYYY-MM-DD) don't match *.log and a vim
.env.swp was nearly committed by a bulk add.

Claude-Session: https://claude.ai/code/session_01VPNF37MP9Kuv62B1VBrZ1b
2026-07-21 13:30:21 -04:00
Harwinder Sekhon a56a622846 Docs: correct PlatformConfig overlay scope (credentials never pass through the factory)
Claude-Session: https://claude.ai/code/session_01VPNF37MP9Kuv62B1VBrZ1b
2026-07-21 13:30:15 -04:00
Harwinder Sekhon b9633389fa Docs: config.json deployment overlay (CLAUDE.md repo map, env table, test count)
Documents the app_config.py / config.json overlay shipped in Tasks 1-4:
repo-map rows for app_config.py and config.json.example, env-var table
precedence (env -> config.json -> none, hardcoded fallbacks removed
2026-07-21), PlatformConfig/pair_signals overlay call order, and the
real 199-tests/13-modules suite count. Also fixes a stale SCAN_PAIRS
comment in telegram_bot.py that still named PlatformConfig.scan_pairs
as the deployment switch instead of config.json's scanning.pairs.

Claude-Session: https://claude.ai/code/session_01VPNF37MP9Kuv62B1VBrZ1b
2026-07-21 13:25:21 -04:00
Harwinder Sekhon 6e2fdbd684 telegram_bot: credentials/quality/scan-loops from config.json, startup guard
- Credentials, alert quality, scan pairs/timeframes, log level and
  candle-loop intervals now resolve via app_config (env -> config.json
  -> code default) instead of hardcoded fallbacks.
- _get_shared_dp() uses make_data_provider() for provider-registry
  dispatch.
- main() refuses to start without a token/chat_id, raising SystemExit
  with an actionable message instead of a bare sys.exit string.
- test_all.py Module 13: two new tests covering credential/quality/
  loop wiring via a reload, and the startup guard.

Claude-Session: https://claude.ai/code/session_01VPNF37MP9Kuv62B1VBrZ1b
2026-07-21 13:18:05 -04:00
Harwinder Sekhon 80a2b89769 Data provider: key from config.json/env, provider registry, drop hardcoded key
Resolves the Twelve Data API key through app_config.provider_api_key()
(env → config.json → no default) instead of a hardcoded field on
PlatformConfig. Adds data_provider.PROVIDER_REGISTRY + make_data_provider()
so signal_aggregator and the backtester build the active provider through
one seam; telegram_bot.py gets the same one-line WS-key fix now (per Task 4's
already-planned edit for that line) so the suite doesn't crash on import
between this commit and Task 4's full wiring pass.

Claude-Session: https://claude.ai/code/session_01VPNF37MP9Kuv62B1VBrZ1b
2026-07-21 13:10:58 -04:00
Harwinder Sekhon 1610507d57 Wire config.json overlay into the pair factory (before per-TF deltas)
Implements Task 2: apply_pair_overrides in xauusd.make_config before
the validated per-TF deltas, so e.g. M5 hour-9 appends on top of the
JSON base. Updated Module 9 equivalence test and added three parity pins.

Claude-Session: https://claude.ai/code/session_01VPNF37MP9Kuv62B1VBrZ1b
2026-07-21 12:59:12 -04:00
Harwinder Sekhon 1dab14d298 app_config: reject non-object config.json sections with AppConfigError
Claude-Session: https://claude.ai/code/session_01VPNF37MP9Kuv62B1VBrZ1b
2026-07-21 12:54:46 -04:00
Harwinder Sekhon cbc01565a2 Add app_config.py: config.json loader, validation, env-precedence resolvers
Claude-Session: https://claude.ai/code/session_01VPNF37MP9Kuv62B1VBrZ1b
2026-07-21 12:47:48 -04:00
Harwinder Sekhon 53b65e1bbb Plan: centralized config.json implementation (5 TDD tasks)
Claude-Session: https://claude.ai/code/session_01VPNF37MP9Kuv62B1VBrZ1b
2026-07-21 12:44:24 -04:00
Harwinder Sekhon 1a141abc7c Add config.json.example template; gitignore config.json
Claude-Session: https://claude.ai/code/session_01VPNF37MP9Kuv62B1VBrZ1b
2026-07-21 12:32:28 -04:00
Harwinder Sekhon 096b69686d Spec: centralized config.json design (deployment settings overlay)
Claude-Session: https://claude.ai/code/session_01VPNF37MP9Kuv62B1VBrZ1b
2026-07-21 12:25:08 -04:00
Harwinder Sekhon c21c102536 News informational-only: gate sentiment bonus/penalty off by default
Add NewsConfig.sentiment_affects_signals (default False), gating the +5
aligned-sentiment confidence bonus and the 10-pt news_conflict penalty.
With event_risk_caps_signals already False, live decisions are now pure
technical, matching the backtester (use_news=False). Warnings and the
sentiment display are unchanged. Adds a gating regression test.

Claude-Session: https://claude.ai/code/session_01FMaaPdrrrQuh1VvGQH1icN
2026-07-17 16:03:31 -04:00
harry c87bb14830 Merge pull request 'Remove EUR/USD support — XAU/USD-only codebase' (#1) from xau-only-chart-patterns into main
Reviewed-on: http://192.168.55.11:3000/harry/ForexBot2/pulls/1
2026-07-16 20:09:21 +00:00
Harwinder Sekhon a3279e54fa Enable BUY-only chart patterns on M5 — bearish patterns dropped (OOS negative)
M5 factory: cp_enabled=True, patterns limited to inv_hs, double_bottom,
bull_flag. SELL CP lost -3.21R on OOS; BUY CP: +41.8R PF 1.60 on 12mo,
+65.2R PF 1.48 on 5yr. Adds +22R to the M5 book (708 vs 513 trades).
M15/H1 remain disabled — all failed OOS validation.

Claude-Session: https://claude.ai/code/session_015kby7M9JwpYi8NQT2yoxAi
2026-07-16 16:08:13 -04:00
Harwinder Sekhon 1bce289132 Docs: CLAUDE.md — document XAU_CP archetype (pipeline step 11, generator D, cp_* knobs)
Claude-Session: https://claude.ai/code/session_015kby7M9JwpYi8NQT2yoxAi
2026-07-16 14:31:45 -04:00
Harwinder Sekhon f3d3e56aa8 Add chart-pattern entry archetype (XAU_CP) — disabled by default
Third entry archetype: completion of classical chart patterns (double
bottom/top, H&S top / inverse H&S, bull/bear flag) — the tight canon
with the strongest documented stats. New chart_patterns.py detector
module (pure functions, ATR-unit geometry, closed-bar slices only,
first-close-beyond trigger discipline so a breakout fires exactly
once and never repaints).

Pipeline step 10 in analyze_from_dataframes: fills only empty setup
slots, allows weak-composite F grades (normal at reversal completion)
but inherits session/day blocks, mirrors MTF-opposition + SELL
vol/hour + M5 protections. Geometry: SL = pattern anchor -/+ 0.5 ATR
(min $8), TP1 = measured move, $9 spread gate, trailing exit does the
real work. Strategy tag XAU_CP.

cp_enabled defaults to False everywhere — per-TF adoption happens in
pair_signals/xauusd.py only after the standard validation rule passes.
Backtester: --cp / --cp-patterns flags to test the candidate, XAU_CP
tag recognition.

Suite: 183/183 (21 new tests: detectors, trigger discipline,
invalidation, gates, pipeline, default-off safety).

Claude-Session: https://claude.ai/code/session_015kby7M9JwpYi8NQT2yoxAi
2026-07-16 14:23:17 -04:00
Harwinder Sekhon 14448c10e8 Remove EUR/USD support — XAU/USD-only codebase
EUR/USD never had a validated edge (2026-07-11/12 verdict: no config
passed validation on M5/M15/H1 at 1-pip cost; cost-to-volatility root
cause). Removed: EurUsdStrategyConfig, pair_signals/eurusd.py, EUR
gates 6h + weekend/day filters + strategy marker + trade geometry in
signal_aggregator, EUR pair registry/aliases in main.py and
gpu_backtest.py, EUR/USD news pair mappings, test_all.py Module 10.
Generic multi-currency news keyword infra kept (GBP/JPY/EUR keyword
table is not pair code). Full EUR history remains at 7d3fd95.

Suite: 162/162 green (was 187; 25 EUR-specific tests removed).

Claude-Session: https://claude.ai/code/session_015kby7M9JwpYi8NQT2yoxAi
2026-07-16 14:06:27 -04:00
Harwinder Sekhon 665ffdda48 Fix on Sell order lossing by blocking time 2026-07-16 12:17:56 -04:00
Harwinder Sekhon fe7d4c5d4a Audit fixes: cache boundary staleness, 6e session bypass, GPU engine parity
Fixes all high+medium findings from the 2026-07-16 full-codebase audit.

High:
- data_provider: cache now rejects snapshots missing the latest closed
  bar (_has_latest_closed_bar, weekend-exempt). The wall-clock TTL served
  mid-candle snapshots at the next boundary, so live M15/H1 scans ran one
  full bar behind and the bar-close session check evaluated the wrong hour.
- signal_aggregator: gate 6e (SELL quality) only acts on actionable
  grades. It was re-grading session-blocked F reports to D, which
  re-opened the M5 pullback path (entry check: quality != "F") on
  session-blocked bars.
- gpu_backtest: port the live EMA-ribbon compression/expansion logic
  (+10 conf on expansion) into the GPU engine. Its absence diverged
  composite score/conf on 7-9% of bars and flipped signals on ~0.2%.

Medium:
- gpu_backtest: Bollinger precompute uses the exact pandas rolling calls
  (cumsum E[x2]-E[x]2 lost precision at gold price levels; bb_position
  error up to 7.6e-3 crossed signal thresholds, worst during squeezes).
- gpu_backtest: --confirm-tfs defaults to the pair factory's live
  confirmation map instead of a hardcoded 1h,4h.
- gpu_backtest: entries re-anchor to the first RAW bar after the signal
  bar, so >10-sigma spike bars dropped by cleaning are no longer skipped
  at entry (favorable bias).
- gpu_backtest: per-TF warmup pad — D1 confirms get >=210 bars at window
  start (62-day pad gave ~61 bars: no EMA200, unlike live).
- data_provider: thread-safety locks for cache + throttle (shared
  provider across the bot's 4-thread pool); outlier filter keeps the
  snap-back bar after a spike instead of punching a two-bar hole.
- telegram_bot: alert fingerprint gains a risk-relative entry bucket
  (materially-moved setups re-alert) and is committed only after a
  successful send (failed sends no longer consume the alert for 1h).
- engine: disabled_indicators=["macd"] now removes the MACD pillar from
  scoring (histogram still computed for divergence).

Verification: test_all.py 187/187 (new Module 11: 8 audit-fix
regressions incl. a live-vs-GPU parity sweep with ribbon coverage);
real-data parity sweep over 813 slices of 14mo XAU M15: zero divergence
(pre-fix: 7-9% of bars diverged); both original bug repros no longer
reproduce; cached end-to-end backtest run passes with the live confirm
map. Books shift slightly vs pre-fix runs — comparability note added to
CLAUDE.md; pinned tuned numbers need re-validation under the fixed
backtester.
2026-07-16 10:24:39 -04:00
Harwinder Sekhon 4d54fddf80 Docs: delete DOCUMENTATION.md — both halves superseded
Unlike USAGE.md (a USD/JPY-era fossil), this doc was structurally sound:
right pair, real architecture, correct composite formula, and 8 of its 9
regime-weight cells match engine.py. It is removed because both of its
sections now have live successors, and because the numbers it does state
are wrong in ways that mislead:

- Section 1 (trader-facing) → docs/user_guide.html + beginner_guide.html,
  which are newer and cover the trailing-stop exit model this doc never
  mentions.
- Section 2 (developer reference) → CLAUDE.md.

Stale/incorrect content it carried: the entire quality-grade table was
wrong (claimed A+ 45/70, A 30/60, B 20/50, C 12/40, D 8+; the code grades
40/65, 33/55, 28/50, 25/45, 15), Divergence's trending weight was listed
2.0 vs 1.0 in code, and it predates the 2026-07 work entirely — zero
mentions of XAU_PB or trailing exits (the actual default exit model), and
near-zero coverage of the EUR/USD port.

CLAUDE.md's header now points at docs/ as the human-facing docs instead
of disclaiming this file.
2026-07-16 09:34:04 -04:00
Harwinder Sekhon aaf1d37b14 Docs: delete stale USAGE.md and superseded pullback plan
USAGE.md documented a different product: USD/JPY (unsupported — the
code is XAU/USD + EUR/USD), 16 main.py flags that don't exist
(--backtest, --optimize, --auto-session-filter, …; main.py has five),
and backtest.py/metrics.py, which aren't in the repo. Two claims were
actively misleading: it called the session filter "default off" (it is
True by default, 03:00-23:00 not 07-16) and gave regime-min-adx as 30
(real default 25). CLAUDE.md now covers the CLI, bot and backtester
correctly, so this is a delete rather than a rewrite.

PULLBACK_ENTRY_PLAN.txt was a pre-implementation draft whose spec no
longer matches what shipped — it specifies a 0.5-2.5 ATR retrace window
and a detector in engine.py, while XAU_PB ships 0.8-6.0 (the drafted
window rejected 81% of valid pullbacks) with the detector in
signal_aggregator.py so live and backtest share one code path. Its
validated outcome (+38.3R / 254 trades / 12mo M5) and every shipped
gate are recorded in CLAUDE.md and the config docstrings.

Also drops the now-dangling references to both files from CLAUDE.md
and the config.py XAU_PB comment.
2026-07-16 09:30:50 -04:00
Harwinder Sekhon 18b5ef27d2 Docs: rewrite CLAUDE.md as a complete codebase reference
Replaces the summary-level doc with a full spec derived from a read of
every core module, so contributors (and low-context models) don't have
to infer behavior or rediscover tuned values:

- repo map with line counts; flags DOCUMENTATION.md/USAGE.md as stale
- every config knob with its real default and the backtest evidence
  behind it; marks the legacy/unused fields (SignalConfig.weights,
  min_confidence, cooldown_bars, XAU h4/h1 gates) so nothing new is
  built on dead knobs
- exact indicator logic for all 9 scored indicators + ATR context,
  regime classification rules, regime-aware RSI zone tables, and the
  regime-multiplier table copied from _compute_composite (the old
  table did not match the code)
- aggregator pipeline in source order: MTF agreement, the 12 penalties
  and the diminishing cap, exhaustion block, signal/grade thresholds,
  and gates 6b-6i with exact numbers
- all three trade-setup generators with SL/TP geometry, spread gates,
  and the trailing-exit model
- bot/CLI/news/data-provider/backtester mechanics incl. realism model
  (raw-path resolution, next-open entry, SL-first ties, parallel replay)
- "Rules for making changes": validation acceptance rule, pair
  isolation, live==backtest parity, ROC's load-bearing weight

Also corrects details the old doc got wrong: test count is 179 (not
~176), EUR ships off by default, backtest/*.csv are gitignored local
artifacts rather than test-enforced pins.
2026-07-16 09:26:06 -04:00
Harwinder Sekhon 26f0681ad1 Docs: user guide + beginner guide PDFs
Two printable guides in docs/ (HTML source + weasyprint-rendered A4
PDFs): a trader-facing user guide (signal anatomy, trail management,
M5/M15/H1 comparison from the 12mo books, expectations, commands) and
a plain-language beginner edition (glossary, worked sizing example,
step-by-step trailing-stop walkthrough, H1-first guidance).
2026-07-12 13:42:21 -04:00
Harwinder Sekhon dde41fbeb8 Audit fixes: mixed-regime bias, live==backtest session parity, parallel replay
Fix 1 — mixed-"trending" regime (ADX strong, EMAs unaligned) was
hard-coded bullish in 5 sites: RSI reinterpretation used the uptrend
playbook (RSI 38-55 -> STRONG_BUY conf 75), BUY-only confidence
bonuses, BUY-only extension penalties, no counter-trend penalty.
Now: extremes-only RSI (as transitioning), no directional bonuses,
symmetric extension penalties. 6yr validation: H1 +67.2->+61.9R
(mixed skew 16B/11S -> 4B/7S), H4 +31.6->+27.5R (the 4 bias BUYs),
M15 +152.6->+154.7R, M5 +207.1->+206.1R with maxDD 28.7->24.3R.
The ~8R aggregate give-back is bull-market subsidy from the bug.

Fix 2 — live session filter now applies the EXACT replay-loop
predicate (session window + blocked hours on the signal bar's CLOSE
time, from the bar timestamp not wall clock). Replaces a wall-clock
filter gated on a "XAU STRATEGY" marker that no code writes, which
made live diverge from every validated book.

Fix 3 — delete dead _check_xau_oversold_bounce and
_generate_xau_trade_setup (unreachable; the latter hardcoded BUY and
skipped the spread/RR gates).

Backtester: parallel bar replay via forked workers (--workers,
default auto = cores-2 capped 12). Per-bar evaluation is pure; the
entry cooldown is applied in an ordered phase-2 pass. Proven
byte-identical to sequential on the same window; ~300 -> 1030-1680
bars/s (6yr M5: ~26min -> 5-8min). --workers 1 = sequential path.

Tests: 179 passed (new mixed-regime RSI regression test).
2026-07-11 22:57:37 -04:00
Harwinder Sekhon 6f69b9dd90 XAU: two validated per-TF gates — M15 mixed-regime block, M5 hour-9 block
M15 mixed-"trending" regime block (ADX strong, EMAs unaligned =
momentum without direction): -6.8R/24 trades on 12mo, negative both
halves; blocking scores +65.7R/499 PF 1.38 DD 10.4R vs +59.0R/521
PF 1.32 DD 12.6R, better both halves, OOS 2024-06->2025-03 neutral
(-1.4R = the 4 such trades there). M5 never trades this regime.
New XauStrategyConfig.blocked_regimes / sell_blocked_regimes knobs
(default empty) + aggregator gate 6g2; M15 factory opts in.

M5 hour-9 UTC block (post-London-open whipsaw): -28.9R/143 over the
6-year book, negative 5/7 calendar years. Full-history replay with
block: +207.1R vs +178.3R, max DD 28.7R vs 35.5R, 2023 -13.0R ->
breakeven; tuned 12mo book unchanged (139.0 vs 139.3). Hour 9 is
positive on M15 (+1.7R 6yr, +8.6R OOS) so the block is factory
M5-only.

Rejected after validation (documented in xauusd.py + session memory):
transitioning-SELL block (bull-market artifact, +5.7R in 2024 OOS),
M15 trending_down block (M5 cross-TF contradiction), losing-streak
circuit breakers, 30d efficiency-ratio trend gate, and all entry-level
indicator filters (winners == losers at entry on every feature).

Full-history context (Twelve Data 5min starts 2020-03-16): M5 6.3yr
+207R PF ~1.3 post-fix with losing year 2021; M15 +152.6R PF 1.19
DD 49.7R — edge is regime-dependent, chop years are structural.

Tests: 178 passed (new gate test + factory-equivalence updates).
2026-07-11 21:38:42 -04:00
Harwinder Sekhon a2bf80054c Telegram: subscriber channel feed, owner lock, full risk disclaimer
- TELEGRAM_CHANNEL_ID mirrors auto-scan signal + breach alerts to a
  subscriber channel (Stars-subscription monetization; Telegram manages
  membership). Feed keeps running even when the owner's DM auto-scan
  is toggled off.
- _owner_only decorator locks all commands/buttons to TELEGRAM_CHAT_ID;
  others get a refusal pointing at the channel.
- Every signal-bearing message (detailed signal, scan summary, all-TF
  summary, breach alerts) now ends with a not-financial-advice /
  educational-use disclaimer via a single _DISCLAIMER constant.
2026-07-11 18:37:50 -04:00
Harwinder Sekhon 60fb526fab EUR: document closure of calendar + H1-hybrid avenues
Surprise-conditioned entries lose 2-5p at every horizon both years;
avoidance filter marginal for EUR and profit-removing for XAU; the
H1-direction + M15-pullback prototype failed its pre-registered kill
criteria (2024-25 OOS negative). EUR stays informational-only.
2026-07-11 10:26:32 -04:00
Harwinder Sekhon 18abff85a3 XAU M15: adopt 1R minimum targets (per-timeframe, validated)
Finishes PENDING_STEPS Step 1 under the current trailing-exit model.
M15 window 2025-03-30->2026-05-31 ($0.30 spread): +116.6R/460 PF 1.83
vs +105.4R/424 PF 1.82 baseline, better in both halves; 2024-06->
2025-03 OOS +30.3R vs +10.6R control. Same knob on the pinned M5 book
is WORSE (+131.3R/699 vs +139.9R/547) so the override is M15-only in
the xauusd factory, per the pre-registered decision rule. M5 path
unchanged (factory override gated on primary_tf == M15).
2026-07-11 10:22:33 -04:00
Harwinder Sekhon 183f148bb4 Live news gate: stop capping signal confidence near events (XAU default)
Replaying the live danger/caution gates against the validated 12mo XAU
M5 book showed they suppress 25-35 trades worth +4.6..+7.2R/yr — trades
entered near USD releases outperform the book average (+0.475R vs
+0.256R). New NewsConfig.event_risk_caps_signals (default False =
XAU-tuned) keeps EVENT RISK warnings in alerts but no longer guts
confidence; EUR factory re-enables the caps (evidence is gold-specific).
Pinned XAU M5 regression: 547 trades, +139.94R, trade-identical.
Tests: 177 passed.
2026-07-11 10:02:27 -04:00
Harwinder Sekhon 528a308d36 Add calendar research infra: FF scraper + verified event store
Scrapes Forex Factory weekly pages (epoch datelines, actual/forecast/
impact/betterWorse) into backtest/cache/calendar.parquet (gitignored;
rebuild with the scraper). Timestamps verified: NFP/CPI/ECB times exact
across DST, and high-impact event bars average 3.5x ATR with the spike
vanishing at +/-1h shifts. calendar_store provides precise_events and
event_window_mask for screens and counterfactuals.
2026-07-11 09:47:11 -04:00
Harwinder Sekhon 9c72e90ce2 EUR M15/H1 investigation: document no-validated-edge verdict
Five independent tests at 1-pip cost (M15 wide-open both windows,
transitioning-slice OOS, mean-reversion trade sims, forward-return
screens on M15+M5, H1 both windows) — nothing clears cost in both
2024-25 and 2025-26. Root cause: spread is ~10% of M15 ATR on EUR.
EUR stays out of scan_pairs; open avenues documented in docstring.
2026-07-11 09:02:24 -04:00
Harwinder Sekhon 8b086f4f49 EUR M5: commit best-found config after realism re-tune — no validated edge
12mo grid campaign (2025-07-08 → 2026-07-08, 1-pip round-trip cost,
train/val split at 2026-01-08). Donor config scored -6.7R/34 trades.
Findings: transitioning regime -27R/88 wide-open; A-grade entries lose
(-26R) while B-grade wins; Fridays -17R; hours 8/12/13/18/19 UTC bleed;
score 30-35 band holds the train-half losses; trailing (0.45R arm/0.5R
dist) beats static TP1 and gold's 0.6/0.75.

Best-found config: -1.06R / 53 trades / PF 0.93 / 62.3% WR — breakeven,
FAILS acceptance (positive both halves, PF>=1.3). Per plan, EUR ships
functional but stays out of default scan_pairs. Grid results and
methodology documented in the module docstring; full numbers in
backtest/cache/eurv2_*.csv.
2026-07-10 22:14:49 -04:00
Harwinder Sekhon f02f4b4d37 Update CLAUDE.md for multi-pair architecture 2026-07-10 21:16:34 -04:00
Harwinder Sekhon 74829541d6 Live path: config-driven multi-pair scanning (Telegram bot + CLI)
- telegram_bot: SCAN_PAIRS/_WS_TO_PAIR/SCAN_BUTTONS all derive from
  PlatformConfig.scan_pairs (the single pair on/off switch); buttons and
  keyboards are generated per pair; _run_signal_sync builds per-pair
  configs via pair_signals factories (was: XAU defaults for any symbol);
  scan-all works per pair; PB badge matches any *_PB tag; daily-rotating
  file logs in logs/telegram.log (30-day retention)
- main.py: --pair argument (XAUUSD | EURUSD + aliases) routed through
  pair_signals factories; price display is precision-aware (2dp gold,
  5dp FX)

scan_pairs default stays [XAUUSD] until the EUR M5 re-tune validates.
2026-07-10 21:13:42 -04:00
Harwinder Sekhon 6546b89d6b Add EUR/USD test suite + map EUR news events to EUR/USD
Module 10 (21 tests): EUR config/registry/factory, EUR filter pipeline
(regime/quality/score/confidence/RSI-extended/weekend), EUR trade
geometry (swing SL buffers, pip-denominated spread gate, per-pair gate
contrast), and cross-pair isolation regressions (XAU report bit-equal
with eur_strategy flipped; XAU gates never fire on EUR reports).

news_engine: _get_affected_pairs now maps USD events to EUR/USD too and
adds an EUR entry (calendar warnings reach EUR scans).

175/175 pass.
2026-07-10 21:06:55 -04:00
Harwinder Sekhon 9ed07d4b65 Backtester: EUR/USD support with per-pair configs and default spread
- EURUSD aliases in PAIR_ALIASES
- main() builds PlatformConfig via pair_signals.PAIR_PRODUCERS factories
  (per-pair regime weights/gates/sessions); unregistered pairs keep the
  inline construction. XAU factory is asdict-identical to inline
  (enforced by test) so XAU runs are unchanged.
- --spread default now resolves from PAIR_INFO default_spread
  (XAU $0.30, EUR 0.0001) instead of a hardcoded gold value
- EUR strategy tag: renamed EUR_PB→EUR, detected in both bullish and
  bearish reasons[0] (SELL setups mark the bearish list)

Smoke-validated: 30-day EUR M5 run produces EUR-tagged trades with
1-pip round-trip cost charged.
2026-07-10 21:02:44 -04:00
Harwinder Sekhon 0da5ed03af Add EUR/USD signal gates, trade geometry, and weekend/session filters
Re-grafted from feature/eurusd-support onto the evolved aggregator:
- 6h EUR gate chain (regime, quality, exhaustion, EMA200, session,
  score/confidence, volatility, indicator agreement, RSI-extended),
  active only for EUR/USD with eur_strategy.enabled
- _generate_trade_setup: EUR branch reads EurUsdStrategyConfig only
  (min_sl_pips, min_rr, mature-move + quality-scaled TP, sl_atr_floor,
  swing-buffer SL) with a pip-denominated spread gate; adds a
  pip_value<=0 guard; non-EUR branch is arithmetically unchanged
  (sl_swing_buffer=1.0 extraction only)
- EUR weekend/weekday pre-filter in analyze_pair mirroring the XAU block
- "EUR STRATEGY" reasons[0] marker so the backtester tags EUR trades
2026-07-10 20:57:54 -04:00
Harwinder Sekhon db50bf02d7 Gate all XAU-only aggregator logic by pair
The SELL quality gate (6e), SELL volatility gate (6g), M5 noise gates
(6h), ATR trade-setup generator, and M5 pullback archetype are all
gold-tuned but previously keyed only on xau_strategy.enabled or the
timeframe, so they would have silently applied gold constants to any
other pair. Each now also requires display_name == "XAU/USD".

No behavior change for XAU (conditions are tautologically true for the
only registered pair today) — verified by test suite and pinned-window
backtest CSV diff.
2026-07-10 20:53:23 -04:00
Harwinder Sekhon 1bc48915bd Add pair_signals per-pair producer registry
Ported from feature/eurusd-support: PAIR_PRODUCERS registry with
make_config() factories per pair. xauusd.make_config is a pure
PlatformConfig-defaults wrapper (equality with the inline construction
is enforced by test); eurusd.make_config carries the donor branch's
regime weights and gates as a pre-realism baseline.

Adds 4 registry tests. 154/154 pass.
2026-07-10 20:52:06 -04:00
Harwinder Sekhon 9d3729ea73 Add EUR/USD pair constants and EurUsdStrategyConfig
- EURUSD in PAIR_INFO; new per-pair price_precision and default_spread
  fields on both pairs (backtester consumes default_spread when --spread
  is omitted)
- EurUsdStrategyConfig ported from feature/eurusd-support plus EUR-local
  cost knobs (round_trip_cost=1 pip, min_tp_cost_multiple=10,
  min_sl_pips=8) so SignalConfig's gold-dollar values never apply to EUR;
  donor-tuned values are a pre-realism baseline pending re-tune
- PlatformConfig.eur_strategy field; track EUR in news currencies
2026-07-10 20:50:08 -04:00
Harwinder Sekhon f3b90d3862 Fix 3 engine bugs: swing-point NaN false positives, ROC div-by-zero, ADX epsilon
Ported from feature/eurusd-support (e6e09c5), pair-neutral fixes:
- _find_swing_points: all(<empty generator>) is vacuously True, so a value
  surrounded by NaNs was reported as a swing point; now requires at least
  one valid neighbor on each side
- ROC: guard against zero close at the lookback positions (inf scores)
- ADX: epsilon denominator instead of NaN-replace to avoid NaN cascade

Adds 8 regression tests (Module 9). 150/150 pass.
2026-07-10 20:48:56 -04:00
Harwinder Sekhon ea306df484 Overhaul news engine: fix false signals, add surprise detection + caching
Improvements from a live-verified audit of the news/calendar pipeline:

- Currency tagging: keyword matching now uses letter-boundary regexes
  and drops ambiguous "us "/"eu "/"uk " tokens — "versus" no longer
  tags headlines as USD (confirmed live false positive polluting the
  XAU/USD sentiment average).

- Fetch caching: calendar events and headlines are cached class-level
  (10 min TTL, 60s retry after empty/failed fetch, lock-guarded for
  concurrent scans). Stops per-scan refetching that was tripping the
  Forex Factory rate limit; TTL stays under the 30-min pre-event
  buffer so the danger window can't go stale.

- Surprise detection: EconomicEvent.surprise is computed from parsed
  actual vs forecast (handles %, K/M/B/T suffixes, negatives). A
  surprise print widens the post-release caution window to 45 min;
  inline releases keep the original 15 min.

- Gold-aware sentiment: headlines are scored per currency, with
  monetary-policy keywords (hawkish/rate hike/tightening vs dovish/
  rate cut/easing/stimulus) inverted for XAU — "hawkish Fed" headlines
  scored +1.0 for gold before, -1.0 now (verified on live feed).

- New sentiment sources: Google News gold-focused query and CNBC
  Economy RSS (verified live; 40 → 74 unique headlines). Added
  cross-feed title dedup so syndicated stories count once.

- events_today now buckets by UTC date (feed timestamps carry US-
  Eastern offsets); --json output includes news_description.

- Removed dead NewsConfig.min_impact and unused HIGH_IMPACT_EVENTS.

Tests: 142 passed (7 new covering boundary matching, surprise
classification, post-event windows, cache reuse, gold inversion).
2026-07-10 09:43:11 -04:00
harwinder 1038a53bac Add M5 pullback entry archetype (XAU_PB): +140R combined on 12mo
Second entry pattern alongside the Keltner-breakout: enter on the
retrace to EMA21/KC-mid in an established trend (EMA >=3/4 aligned,
ADX>=20, retrace 0.8-6.0 ATR, resumption bar closed back in trend
direction, RSI in the pullback zone not the extended zone). Detector
lives in the aggregator so live and GPU-backtest paths share one
implementation. Own gates (composite floor 15, confirmation-TF
agreement, MTF opposition, vol regimes) with a diagnostic funnel.

PB stops anchor to the pullback swing (~$8 vs ~$25 breakout), so PB
trades get their own trail arm (0.8R vs 0.6R global) via per-trade
trail overrides in the backtester and strategy-aware Telegram alerts.

12mo M5 validation: PB alone +38.3R/254 trades, positive both halves
and all four full quarters; core breakout trades undisturbed
(+101.9R vs +105.0 baseline). Combined book: 549 trades (~10.6/wk),
+140.1R, PF 1.78, DD 11.3R vs 311 trades/+105.0R before.
2026-07-09 14:21:59 -04:00
harwinder 305f1d1857 Add pullback entry pattern implementation plan
Design doc for a second M5 entry archetype: enter on retrace to
EMA21/KC-mid in an established trend instead of at Keltner-breakout
extension. Includes detector spec, geometry, validation protocol
(both-halves + quarterly + displacement checks), and the +105R
baseline to beat. Session/frequency knobs (blocked hours, cooldown)
were tested 2026-07-09 and confirmed already optimal — this is the
remaining evidence-backed entry-side direction.
2026-07-09 13:51:00 -04:00
harwinder 0393087168 Add M5 gates 5-6 and trailing-stop exits (+48R → +105R on 12mo)
Gates 5-6 in the M5 filter: block 'high' vol regime entries (chop
transition — 61% WR, negative total while low/normal/extreme all
positive) and block SELLs with RSI<25 (dip already extended, mean-
reverts before target). 365d: 419→311 trades, +41→+48.4R, DD 11.2→8.0R.

Replace the fixed RR-0.5 take-profit with a trailing stop: arm at
+0.6R favorable excursion, trail 0.75R behind best price, stop levels
update at bar close (conservative). The RR-0.5 target was the binding
constraint — winners kept running after TP. Whole trail parameter grid
(arm 0.4-1.2 × dist 0.5-1.25) lands +90-103R, a plateau not a spike;
also +91.7R on the pre-gate 419-trade population. TP1 stays advisory;
Telegram alerts include the arm price and trail distance. max_hold
200→300 bars (trailing exits need room; +10R).

12mo M5, same 311 trades: +105.0R total, PF 2.27, DD 7.9R, 61% WR,
train/val +54.6/+50.4, every quarter positive.
2026-07-09 13:34:57 -04:00
harwinder 8687f98b39 Fix M5 signal quality: ATR ordering bug, noise gates, blocked-hour leak (+41R)
ATR was computed after EMAs so ribbon compression detection was dead code
on all timeframes — moved ATR first. Extended M5 swing S/R lookback from
50 to 120 bars. Added four M5-specific filters: min score, MTF agreement,
confirmation TF conviction, and SELL quality cap (A-grade M5 shorts enter
at dip extremes and lose). Fixed backtester blocked-hour check to use bar
close time matching the live bot. Added trade geometry config (spread gate,
SL floor, TP floor RR) and primary_tf mismatch guard in backtester.

Backtest: 1133 trades/+11.5R → 419 trades/+41.0R, PF 1.03→1.37, DD 21→11R.
Train/val split positive in both halves (+19.8R/+21.3R).
2026-07-09 12:48:36 -04:00
harwinder 2291fc3936 Disable TP tightening: tuned pre-spread, halves total R under real fills
A/B on M15 2025-03-30→2026-05-31 with $0.30 spread + gap-through fills
(511 identical entries, only exits differ):
  factor 0.8 / min_rr 0.4: +36.1R (avg +0.071R, SELL +1.2R)
  factor 1.0 / min_rr 0.5: +66.8R (avg +0.131R, SELL +9.6R)
Worse in both train and validation halves and both directions. The 20%
tighter targets buy +1.8pt win rate but shrink every win while the fixed
spread stays constant. The knob remains for re-tuning with costs included.
2026-07-08 21:35:59 -04:00
harwinder 3a85c6e23b Add WebSocket dual-connection test script
Probes whether Twelve Data allows two simultaneous WS connections
for one symbol (SDK-based, 30s run, prints per-connection verdict).
2026-07-08 21:32:46 -04:00
harwinder 09ee5208d4 Telegram bot: WS heartbeat, breach alerts, alert dedup, interactive menu
WebSocket:
- Send heartbeat every 10s — Twelve Data drops idle connections, so the
  monitor previously died on weekends/quiet hours and churned reconnects
- Log failed symbol subscriptions from subscribe-status events
- Stamp fresh live tick price + age into every alert message

Alerts:
- Re-enable level-breach alerts: WS breach triggers M15 rescan and sends
  the report when actionable (was scanning and discarding the result)
- Dedup: identical (signal, quality, direction) per pair/TF is sent once,
  re-armed after 1h — no more repeat alerts every candle
- Configurable minimum alert quality (C+/B+/A), cycled from the menu
- Trade setup shows dollar risk/reward alongside pips

Interactive menu (/start, /scan):
- Timeframe scans, Scan All, auto-scan ON/OFF toggle, alert-quality
  cycle, status, and help as inline buttons
- Markdown-safe edits: 'message not modified' swallowed, parse failures
  fall back to plain text; exception text escaped
2026-07-08 21:32:46 -04:00
harwinder 1f076c0b54 Fix live/backtest parity, SELL-side losses, and indicator repairs (+42% total R)
Live pipeline:
- Drop still-forming candle before analysis (signals no longer repaint)
- Direction-aware MTF agreement + opposition gate (no trade against 2-of-3 TF consensus)
- SELL volatility gate: block shorts in low/normal vol regimes (config sell_vol_regimes)
- Divergence: 40-bar lookback, price/indicator swings must align within ±2 bars
  (fired on 0.5% of trades before, 29% after), trending weight 2.0→1.0
- RSI regime_aware by default; transitioning regime only signals at extremes
- ADX ±DM tie no longer nudges bearish; TP2 clamped beyond TP1

Backtester realism:
- Charge $0.30 round-trip spread (--spread), gap-through SL/TP fills at bar open
- Resolve trades on RAW price path (cache stores raw bars, cleaning runs on load)
- Weekend/session filters use bar-close decision time to match live wall clock

Validated per-step on M15 2025-03-30→2026-05-31 under identical pricing:
+46.9R → +66.8R, win rate 69.9% → 73.0%, SELL side -2.3R → +9.6R.
Tested-and-reverted: ROC weight removal (-13R), BB fade suppression (-11.6R),
candlestick de-weight (-2R) — composite thresholds are co-tuned with weight mass.
2026-07-08 21:32:46 -04:00
harwinder baeee4997a Fix trade setup generation broken by TP tightening factor
The TP floor calculations used raw min_rr (0.4) but after 20% tightening
the effective RR dropped to ~0.32, failing the min_rr check and returning
None for all trade setups. Pre-compensate the floor with min_rr/tf so
post-tightening RR still meets the minimum.
2026-05-24 18:22:57 -04:00
harwinder bb40bceb4c Tighten TP targets by 20% for faster hits (configurable tp_tightening_factor=0.8)
Applied uniformly across all 3 trade setup functions. Lowered min_rr_ratio
0.5→0.4 proportionally to preserve trade acceptance rate. SL unchanged.
2026-05-22 10:44:52 -04:00
Harwinder c46bb85012 Enable M5 auto-scan alerts and remove live dashboard
M5 already broadcast through the shared _scan_timeframe path but was
documented as data-only (docstring, CLAUDE.md, DOCUMENTATION.md, plus a
phantom `if tf == M5: return` step that never existed in code). Align all
docs/comments so M5 alerting at C+ grade is the intended, documented behavior.

Remove the live dashboard entirely per request: the 30s refresh loop,
per-scan refresh, boot send, and the dashboard-only stats layer
(_signal_stats / _stats_updated / _update_signal_stats and its call sites,
_format_stats_dashboard and its helper constants). Drop now-unused ZoneInfo
import and dashboard mentions from help/autoscan/boot messages and the test
suite. The WebSocket level-breach monitor and level cache are untouched.

Tests: 136 passed, 0 failed.
2026-05-20 12:45:27 -04:00
harwinder cc53448c34 Fix weekend filter: use precise forex market hours (Fri 22:00–Sun 22:00 UTC)
The old `weekday() >= 5` check blocked all of Sunday, but the forex
market reopens Sunday 22:00 UTC (Sydney open). Now uses proper time-based
check so the bot fetches data and generates signals on Sunday evening.
2026-05-17 19:23:21 -04:00
harwinder 15a7074574 Add backtest scripts to version control
Track gpu_backtest.py, diagnose_filters.py, and instructions.
Data files (CSVs, parquet cache) remain gitignored.
2026-05-15 19:49:30 -04:00
harwinder 35e9b3750d Reduce A-grade TP when exhausted: RR 0.75→0.50 on 2+ exhaustion tags
A-grade trades entered exhausted moves with TP too far (0.75R), causing
57.7% WR and +0.25R over 30 days. Store exhaustion_count on SignalReport
and scale TP to B-level (1.0×) when A-grade has 2+ exhaustion tags.
Backtest: A-grade 73.1% WR / +3.25R, B/C unchanged.
2026-05-15 19:46:31 -04:00
harwinder 68cc5b28a5 Align FilterConfig with XAU strategy: blocked hours [0,1,2], session 3-23 UTC, Mon-Fri 2026-05-15 15:36:03 -04:00
harwinder 5c0adfab27 Add M5 to Telegram scan buttons and Scan All TFs 2026-05-15 15:26:00 -04:00
harwinder b7bfd9c830 Reduce blocked hours, enable M5 alerts, show NY time on dashboard
- Blocked hours reduced to 0-2 UTC only (daily rollover)
- M5 signals now broadcast to Telegram like M15/H1/H4
- Live dashboard shows New York time (auto EDT/EST via zoneinfo)
2026-05-15 15:23:48 -04:00
harwinder 04328eaeab Refocus on XAU/USD: add momentum indicators, tune TP, exhaustion block
Strip EUR/USD, GBP/USD, USD/JPY pairs and strategies from entire codebase.
Add 3 new scored indicators (ROC, Keltner/TTM Squeeze, ADX slope) and
enable MACD-as-pillar + divergence detection for early trend entry.
Introduce "transitioning" regime (ADX < 25, slope > 3.0) to catch trend
births before ADX confirms. Gold-specific composite weights per regime.

Tune TP multipliers: quality-tiered scaling (A+/A get 1.5-2.0x ATR via
XAU ATR method, B/C use swing-level targeting). Backtest: +24R total,
65% WR, PF 1.12 over 365 days.

Add momentum-confirmed exhaustion block: hard-reject trades when 2+
static exhaustion signals fire AND ROC/MACD confirm fading momentum.
Prevents late trend entries without killing strong trend trades.

Telegram: simplify buttons to timeframe-only labels, add "Scan All TFs"
multi-timeframe summary, add /scan command, fix BB/KC level breach keys,
fix price formatting (.2f for gold), fix deprecated asyncio call, fix
"1 pairs" grammar and 60s/30s dashboard refresh mismatch.
2026-05-15 10:51:51 -04:00
harwinder bb3c475f22 Add web dashboard (Redis + FastAPI + WebSocket), scope XAU penalties
Signal daemon (signal_web.py): candle-boundary scanner mirroring
telegram_bot.py, publishes serialized SignalReport JSON to Redis.
Web API (web/app.py): thin FastAPI reader behind Apache, serves
/api/signals REST + /ws WebSocket from Redis pub/sub.
Frontend (web/static/index.html): 3-pair grid, color-coded signals,
indicator breakdown per timeframe, trade setup display.

Also: XAU-specific penalty scoping in signal_aggregator (softer
confidence penalties for gold, original strict values for forex),
pair-aware session filter routing XAU to its own blocked hours.
2026-05-14 23:15:37 -04:00
harwinder 31ecdf4292 Fix news engine (keywords, calendar fallback, RSS URL), remove Telegram guards
News engine: expand sentiment keywords (18%→60% headline coverage), add
MyFXBook RSS as fallback calendar source when Forex Factory returns 429,
add retry/backoff on calendar API rate limits, update stale ForexLive URL
to investinglive.com, add date formats without seconds for MyFXBook RSS.

Telegram bot: remove flood control, rate limiter, and dashboard failure
tracking — all sends go directly through bot.send_message/edit_message_text.
2026-05-14 13:42:32 -04:00
harwinder 39c91c1aa4 Fix duplicate-index crash in data validation, add Telegram flood control
- Move index dedup before OHLC consistency fix in _validate_and_clean
  to prevent crash when df.loc[dup_idx] returns Series instead of scalar
- Add global flood backoff: detect RetryAfter, pause all sends/edits
  until ban expires instead of retrying and escalating the ban
- Stop dashboard edit-fail → new-message spam cascade: track consecutive
  failures, only resend after 3 failed edits
- Increase dashboard refresh from 5s to 30s, min send interval to 1.5s
- Add comprehensive QA test suite (140 tests across all 7 modules)
2026-05-14 13:04:28 -04:00
harwinder cd69980f1f Harden Telegram bot, add EUR/GBP strategies, parallelize scans
Telegram reliability: rate-limited sends (1 msg/s), dashboard lock,
WebSocket JSON/price error handling, markdown escaping, level cache
TTL, cooldown cleanup, breach task tracking, graceful shutdown.

Telegram speed: parallelize 3-pair scans with asyncio.gather(),
increase thread pool 2→4, parallelize boot-up initial scan.

Strategy: add EUR/USD and GBP/USD pullback-buy strategies with
per-pair configs, optimize XAU TP to 0.9×ATR, block hours 8/9
for XAU and hour 17 for GBP.

Backtest: add --max-hours flag to evaluate.py for time-exit support.
2026-05-13 21:07:49 -04:00
Harwinder 67a9169eb3 Fix dashboard TF key mismatch, increase refresh to 5s
Timeframe keys stored as enum values (15min/1h/4h) but dashboard
looked for M15/1h/4h — GBP/USD showed no price or signal text.
Now uses Timeframe enum directly with display labels. Reduced
dashboard refresh from 60s to 5s with throttle guard.
2026-05-13 18:51:02 -04:00
Harwinder f99fcc60b9 Add live dashboard with 60s auto-refresh, disable breakout alerts
Replace per-scan Telegram messages with a single dashboard message
that edits in-place every 60 seconds. Each pair shows all TFs
horizontally with short signal labels (Buy/Sell/Neutral). WebSocket
live prices feed into the dashboard. Level-breach alerts disabled
(WS still updates cache silently). Removed /stats command.
2026-05-13 18:34:44 -04:00
harwinder 4d3f05e2ba Fix live session filter overriding XAU strategy signals
The general session filter ran after analyze_from_dataframes() and
wiped out XAU oversold bounce signals at most hours. Now skips the
session filter when the XAU strategy has already fired — the strategy
has its own backtested hour filter.
2026-05-13 17:31:46 -04:00
harwinder 2deea1ed7f Add XAU/USD oversold-bounce strategy (79.7% WR, PF 2.06)
BUY-only strategy for gold: enters when M15 RSI drops below oversold
threshold while price remains above EMA200 on all three timeframes
(M15, H1, H4). Uses ATR-based TP/SL (0.7x/2.0x) sized from M15 ATR.
Isolated to XAU/USD only — other pairs completely unaffected.
2026-05-13 17:25:49 -04:00
harwinder 89030c96b6 Backtest tuning, data freshness guard, Telegram improvements
- Drop USD/JPY from scan pairs (zero signals across all timeframes)
- Disable divergence indicator (zero impact in 14-config lab test)
- Parallelize evaluate.py with ProcessPoolExecutor (16 cores)
- Add --pair CLI filter to evaluate.py for single-pair backtests
- Add data freshness guard: stale cache auto-invalidated, primary TF
  stale data suppresses signals to prevent trading on old data
- Add /cache command: shows fetch age and last bar per pair/TF
- Add /clearcache command: wipes in-memory cache for fresh API fetch
- Remove BB from WebSocket breach alerts (keep structural levels only)
- Show live prices only for WS-monitored pairs (EUR/USD, XAU/USD)
- Replace arrow emojis with text tags in indicator lines
2026-05-12 22:56:55 -04:00
harwinder 59a18b8bc6 Add project documentation and MT5 integration plan
DOCUMENTATION.md: Two-section guide covering the full project —
plain-English trader guide (indicators, scoring, trade setups) and
developer reference (architecture, data flow, every threshold/weight).

mt5.md: Architecture plan for automated order execution via MT5 bridge
(REST API on Windows, async client on Linux, risk-based lot sizing).
2026-05-11 19:50:21 -04:00
harwinder 49260abaa2 Fix trade setup generation, M5 data-only mode, WS symbol limits, command menu
- Trade setup TP now scales to SL distance ensuring min RR is met when
  no viable swing target exists; skip swing targets with poor RR
- Add float tolerance to RR gate to prevent edge-case rejection
- M5 auto-scan runs for level cache only, no Telegram broadcast
- WebSocket limited to EUR/USD + XAU/USD (Grow plan constraint)
- Register bot command menu via set_my_commands on startup
2026-05-11 18:58:40 -04:00
harwinder 4ce09c2f79 Suppress Telegram messages when no actionable signals found
Auto-scan still runs and updates the level cache every cycle,
but only sends to Telegram when quality A+/A/B/C signals exist.
2026-05-11 18:03:49 -04:00
harwinder 02eedcc875 QA fixes: H4 timing bug, /status + /help commands, WS cold-start
- Fix H4 boundary calculation using total minutes (hour*60+min) not just minute
- Add /help command listing all bot commands
- Add /status command showing WS connection, live prices, level cache, cooldowns
- Run initial M15 scan on boot to populate level cache before WS starts
- Update /start message to mention new commands
2026-05-11 17:16:07 -04:00
harwinder 8ead05432e Tune engine from backtesting: wider SL, capped TP, counter-trend filter
Backtest-driven fixes (0% win → 33% win, PF 0.00 → 1.05 over 7 days):
- SL: swing buffer 0.5→1.0 ATR, min distance 1.0→1.5 ATR (more room to breathe)
- TP: cap swing targets at ATR×3.0, fallback ATR×2.0 (was targeting unreachable levels)
- Mature-move TP: when score>30 (A/B grade), tighten TP to ATR×1.5 (move already extended)
- Counter-trend penalty: -15 confidence when signal fights regime direction
- EMA200 penalties halved: -20→-10 (was killing EUR/USD signals)
- Score thresholds lowered: BUY/SELL ±15→±12, STRONG ±40→±35
- Quality grades relaxed, C-grade now gets trade setups and Telegram alerts
- min_rr_ratio: 1.5→1.2
- Gitignore backtest/ directory
2026-05-11 17:00:12 -04:00
harwinder 90b7e0c54e Buy signal included 2026-05-11 16:37:45 -04:00
harwinder 058ef57d8f Pro tier upgrade: quant rearchitect + WebSocket level-breach monitor
- Restore XAU/USD (Pro plan), drop USD/CAD, raise rate limit to 55/min
- Reduce signal engine to 4 uncorrelated pillars (EMA, RSI, BB, Candlestick)
  with regime-adaptive weights (trending boosts EMA, ranging boosts RSI+BB)
- Add structure-based S/R (swing highs/lows) for trade setup targeting
- Fix confidence death spiral: all adjustments now additive, not multiplicative
- Dynamic R:R trade setup: SL behind swing level, TP at next structure level,
  reject if R:R < 1.5
- Fix news engine for all 4 pairs (was hardcoded to USDJPY)
- Add WebSocket monitor: streams live ticks, triggers M15 scan on level breach
  with 5-min cooldown per pair — catches breakouts between candle boundaries
- Add correlation filter (EUR/USD+GBP/USD, XAU/USD+USD/JPY exposure warnings)
2026-05-11 16:33:23 -04:00
harwinder cc66e65f2d Replace XAU/USD with USD/CAD — TwelveData has no gold data
XAU/USD is not available in TwelveData's symbol list.
Swapped to USD/CAD across config, scan pairs, Telegram buttons,
and added CAD to tracked news currencies.
2026-05-11 15:50:48 -04:00
harwinder 873cf5487e Add 4-pair candle-boundary auto-scanner (free tier safe)
Expand from 2 to 4 pairs (EUR/USD, USD/JPY, GBP/USD, XAU/USD) with
automatic scanning at M15 and H1 candle close boundaries. Scheduler
staggers H1 by 60s to stay within 8 credits/min; daily budget is
480 of 800 allowed. Cache TTL is now candle-duration-aware so cross-TF
confirmation data is reused without extra API calls. Spread estimate
uses primary-TF data instead of fetching M5 separately.
2026-05-11 15:30:11 -04:00
Harwinder a708609f8b Add EUR/USD support and 4-button Telegram scanner
Add EURUSD pair with dynamic pair info lookup. Telegram bot now shows
4 scan buttons: EUR/USD 15M, EUR/USD 1H, USD/JPY 15M, USD/JPY 1H.
Each indicator displays directional arrows in the message output.
Price formatting adapts per pair (5dp for EUR, 3dp for JPY).
2026-05-10 23:49:36 -04:00
Harwinder 7ccd8c4b04 Strip backtesting, rebalance indicator weights toward leading signals
Remove backtest.py, backtest_gpu.py, metrics.py, local CSV data,
Dukascopy loader, and all backtest CLI modes from main.py. Rebalance
engine weights: leading indicators (divergence, RSI, stochastic, BB)
now carry 66% of signal weight vs 34% for lagging (EMA, MACD). Add
ws_twelvedata.py for live tick streaming. Drop pyarrow and tqdm deps.
2026-05-10 23:40:15 -04:00
Harwinder 2e09237671 Add USAGE_LINUX.md — bot and CLI run instructions for Linux server 2026-05-07 23:12:55 -04:00
Harwinder a12084537d Add async Telegram bot — live scan button with trade setup
Introduces telegram_bot.py: python-telegram-bot v20+ async bot that
presents a persistent Scan Signal inline button. Each press fetches a
live USD/JPY signal via SignalAggregator and formats it with emoji,
MTF breakdown, top bull/bear reasons, and warnings. STRONG_BUY /
STRONG_SELL signals additionally render the full trade setup (entry,
SL, TP1, TP2, RR). Adds python-telegram-bot>=20.0 to requirements.txt.
2026-05-07 23:05:01 -04:00
Harwinder a55aca21b1 Round-2 weight optimisation — reduce EMA trend, boost stochastic
GPU sweep (36 combos, RTX 4070 SUPER) on H4 2022-2026 relative to the
round-1 baseline revealed EMA trend was over-weighted at 3.6. Halving
it and nudging stochastic up yields cleaner, higher-quality signals.

Weight changes (backtest_gpu.py _BASE_W + engine.py):
  ema_trend:  3.6 -> 1.8  (over-weighted; halving tightens signal filter)
  stochastic: 1.6 -> 2.4  (x1.5; improved WR to 40.4% in sweep)
  macd:       3.3          (unchanged — dominant signal, kept as anchor)
  adx/ichimoku/rsi/bollinger: unchanged

Combined result vs round-1 (H4 local data 2022-2026):
  Trades:      160 -> 73   (-87 — filters out marginal setups)
  Win Rate:   38.1% -> 43.8%  (+5.7pp)
  Profit Factor: 1.19 -> 1.52
  Return:    +21.30% -> +23.86%  (+2.57pp)
  Max DD:     -8.52% -> -9.32%
  Sharpe:      0.47  ->  0.71   (+51%)
  Sortino:     1.91  ->  3.06   (+60%)

Fewer trades at much higher quality. The lower EMA weight means only
strong, multi-indicator confluences clear the grade threshold — the
73 remaining trades have 43.8% WR vs 38.1% and expectancy nearly 3x.
2026-05-07 22:24:48 -04:00
Harwinder e57f4a1ecd Enable GPU scoring via CuPy — RTX 4070 SUPER confirmed 6.3x speedup
Install nvidia pip packages for CUDA 12 runtime DLLs (no full toolkit needed):
  nvidia-cuda-nvrtc-cu12, nvidia-cuda-runtime-cu12,
  nvidia-curand-cu12, nvidia-cublas-cu12

Add _register_nvidia_dlls() in backtest_gpu.py that auto-discovers and
registers nvidia package DLL directories (site-packages/nvidia/*/bin/)
via os.add_dll_directory() before CuPy is imported. Suppresses the
spurious "CUDA path could not be detected" warning from cupy._environment.

Backend line now reads: CuPy 14.0.1 (RTX GPU)
Benchmark: scoring pass on 100k bars — NumPy 1.1ms vs CuPy 0.18ms = 6.3x.
Results are identical to CPU path (deterministic).
2026-05-07 22:22:21 -04:00
Harwinder 17cb63e86e GPU backtester, optimised weights, local CSV data pipeline
## What changed

### backtest_gpu.py (new — ~1,300 lines)
Standalone GPU-accelerated backtester completely separate from backtest.py.
Computes all indicator series in one vectorised O(N) pass over the full price
history, then replays positions with a sequential state machine.

Key design:
- VectorIndicators: pandas ewm/rolling (C-optimised) for all 7 indicators
- CuPy GPU backend with automatic NumPy fallback (no hard dependency)
- Two-phase API: prepare_arrays() once, score_and_run() many times
- --optimize flag: 36-combo weight sweep (1 baseline + 7 ablations + 28
  multiplier runs) — calls prepare_arrays() once, replays all combos in seconds
- Full CLI: --tf, --local, --start/--end, --regime-filter, --regime-tf,
  --session-filter, --optimize, --journal, --journal-csv

O(N) vs O(N^2): backtest.py recomputes all indicators per bar (O(N^2));
backtest_gpu.py computes each indicator series once and reads bar T in O(1).
On 100,000 H1 bars: ~4 seconds vs >10 minutes.

### data_provider.py
Added load_local_csv() method — reads MetaTrader tab-delimited CSV exports
from market_data/ directory. Format: Time|Open|High|Low|Close|Volume|Spread
(spread column dropped on load, timestamps localised to UTC). Applies the same
_validate_and_clean() pipeline as the Twelve Data path. LOCAL_CSV_MAP maps
each Timeframe enum to its filename.

market_data/ files added (MetaTrader exports, UTC):
  USDJPY_M15.csv  100,000 bars  2022-05-04 to 2026-05-08
  USDJPY_H1.csv   100,000 bars  2010-04-23 to 2026-05-08
  USDJPY_H4.csv    25,855 bars  2010-04-22 to 2026-05-08
  USDJPY_D1.csv     5,020 bars  2010-04-22 to 2026-05-08

16 years of H1/H4/D1 vs ~6 weeks of H1 on Twelve Data free plan. Use --local
for all backtests; Twelve Data reserved for live signal feed.

### engine.py — optimised indicator weights
36-combo sweep on H4 2022-2026 found MACD and EMA trend are the dominant
signals; RSI and Bollinger are net drag on this trending pair.

Old → new weights:
  ema_trend:  1.2 → 3.6  (×3 — strongest directional signal)
  macd:       1.1 → 3.3  (×3 — ablating it is the single worst move: -42%)
  stochastic: 0.8 → 1.6  (×2 — clean timing, 40% WR at this level)
  adx:        1.0 → 0.5  (÷2 — partially redundant with EMA stack)
  ichimoku:   1.1 → 0.55 (÷2 — lags too much at H4, hurts in ranging)
  rsi:        0.3 → 0.0  (ablated — net drag across all test windows)
  bollinger:  0.9 → 0.0  (ablated — counter-trend noise on trending pair)

RSI and Bollinger are still computed and displayed in live output for
reference; weight=0 means they contribute nothing to the composite score.

Combined result vs baseline (H4 2022-2026, local data):
  Return:   +7.12% → +21.30%  (+14.2pp)
  Max DD:   -9.05% → -8.52%   (tighter — gains did not come at expense of risk)
  Sharpe:    0.23  →  0.47    (2x improvement)
  Sortino:   1.38  →  1.91
  Win Rate: 36.3%  → 38.1%

### backtest.py
Added 16-thread CPU parallelism for signal pre-computation via
ThreadPoolExecutor. _precompute_all_signals() dispatches each bar to a worker;
pandas/NumPy C extensions release the GIL enabling true parallel execution.
Results are collected and replayed in the main loop, replacing the per-bar
recompute. Also added --regime-tf CLI flag so regime filter timeframe can be
specified independently of the primary TF.

### signal_aggregator.py
EMA200 distance gate: confidence *= 0.55 when price is within 0.5% of EMA200
(chop zone) or beyond 1.5% (overextended). Sweet spot is 0.5-1.5% from EMA200
where trending momentum is cleanest. Strongest single filter found: lifted
cumulative return +13.88% to +35.50% in ablation window.

### ENGINE_LOGIC.md (new)
Full reference document covering: architecture, data pipeline, all indicator
logic, composite scoring formula, optimised weights with rationale, regime
classification, EMA200 gate, MTF alignment, grade system, backtesting results,
filter catalogue, and configuration reference.
2026-05-07 22:12:43 -04:00
Harwinder 5a392a578d Add EMA200 distance gate — strongest single signal-quality filter
Across 2024 trend / 2023 chop / 2026 hostile windows, distance from the
H1 EMA200 was empirically the most discriminative variable. Sweet spot
0.5-1.5% from EMA200; closer = false-breakout chop, farther = chase.

Outside the band, confidence is multiplied by 0.55 so the trade falls
below the B-grade threshold and is rejected.

Cumulative return across the 3 windows: +13.88% baseline -> +35.50%.
Win rate improved in every regime; 2023 chop turned positive (-2.56% to
+14.94%).

Also extends the trade journal with per-timeframe MTF scores, RSI at
entry, and EMA200 distance for future filter mining; surfaces ema_trend
in key_levels so the gate can read it.
2026-05-07 19:28:19 -04:00
Harwinder be42736223 Document session filter and auto-toggle flags in USAGE 2026-05-07 18:44:41 -04:00
Harwinder 2e5d66db1b Add session filter + auto-toggle, extend trade journal diagnostics
Session filter restricts trades to a UTC hour window (default 07-16,
London + NY) — opt-in via --session-filter, with --session-start /
--session-end overrides. Wired into both the backtester (cheap hour
check next to the regime gate) and live mode (downgrades signal to
NEUTRAL/F when current UTC hour is outside the window).

Auto-toggle (--auto-session-filter) tracks rolling mean pnl_pct over
the last N closed trades; when it falls below threshold, the session
filter activates, releasing once expectancy recovers. Default lookback
10 trades, threshold -0.3%.

Trade dataclass extended with diagnostic fields captured at entry:
entry_hour, entry_dow, ADX, ATR%, market_regime, volatility_regime,
mtf_agreement. These let --journal-csv exports surface loss-pattern
clusters when a window underperforms.

Validated on 2024 trend / 2023 chop / 2026 recent windows. Auto-toggle
preserves ~50% of trend edge while still cutting ~25% of recent-regime
losses — pure regime filter is still best on cumulative, so both
session flags ship default-off.
2026-05-07 18:38:49 -04:00
Harwinder fb1eb5b73f Swap raw requests for Twelve Data Python SDK
data_provider now uses TDClient.time_series().as_pandas() instead of
hand-rolled REST calls. Throttle, validation, and parquet caching are
unchanged. Adds twelvedata>=1.2.0 to requirements.txt.
2026-05-07 18:38:38 -04:00
Harwinder 28a00b7da0 Compact hierarchical live signal output
Restructure print_signal_report so decision (signal · grade · score ·
confidence · MTF) sits at the top, followed by trade setup, context line,
MTF breakdown, top supporting/opposing reasons in two columns, warnings,
and an optional detail block under --detail. Adds _compact_reason helper
that strips known indicator name prefixes via _INDICATOR_PREFIXES so
reason lines stay short.
2026-05-07 17:53:50 -04:00
Harwinder 3e078cfe7d Wire regime filter into live mode
--regime-filter previously affected only backtests. Live mode now also
checks higher-TF ADX after analysis: signals are downgraded to NEUTRAL
with a warning when ADX < threshold, and confirming when above.

This brings live behavior into parity with what the backtester measures.
2026-05-07 17:40:39 -04:00
Harwinder ce76684a62 Add indicator ablation framework + apply ablation-driven cleanup
Framework:
- TechnicalConfig.disabled_indicators skips named indicators in engine.analyze
- Optimizer supports disable_indicator override for parallel ablation sweeps
- main.py --ablate runs leave-one-out across all 10 directional indicators

Cleanup applied (validated across 2024 trend / 2023 chop / 2026 recent):
- Drop pivot by default (cumulative +9.5% to disable)
- Drop candlestick by default (+3.4% cumulative)
- Regime-gate ichimoku: 1.3x in trending, 0.2x in ranging (was a chop villain)
- Trim RSI weight 1.0 -> 0.3 (cumulative noise across regimes)

Result: +10.60% on 2024 trend (+33.24% -> +43.84%, Sharpe 2.17 -> 2.77),
+15.71% on 2023 chop (-18.27% -> -2.56%). Recent regime still adverse.
2026-05-07 17:33:07 -04:00
Harwinder 045f4248f3 Refocus on USD/JPY and add walk-forward backtester with regime filter
- Strip multi-pair plumbing; drop synthetic-data fallback
- Add BacktestConfig + walk-forward replay through the live signal pipeline
- Add metrics module (PF, Sharpe, Sortino, max DD, trade journal)
- Add parallel grid optimizer with disk parquet cache
- Add H4 ADX regime gate (--regime-filter, --regime-min-adx) — cuts chop
  losses ~84% in 2023 H1 while preserving trend edge
2026-05-07 17:15:41 -04:00
Harwinder c04f451c0c Remove redundant script.txt and tracked __pycache__
script.txt was superseded by requirements.txt. Python bytecode should
not be version-controlled — added .gitignore to keep cache and venv
out of the repo going forward.
2026-05-06 22:25:43 -04:00
Harwinder 0a374cfea7 Add requirements.txt and usage guide 2026-05-04 10:51:03 -04:00
27 changed files with 1621 additions and 6229 deletions
+8 -13
View File
@@ -11,7 +11,7 @@ Every numeric threshold in `config.py` and `pair_signals/xauusd.py` was backtest
## Commands ## Commands
```bash ```bash
python test_all.py # QA suite: 406 tests, 24 modules, no network (mocked) python test_all.py # QA suite: 374 tests, 23 modules, no network (mocked)
python main.py --pair XAUUSD --timeframe H1 --detail # CLI signal report (--json for full dump, python main.py --pair XAUUSD --timeframe H1 --detail # CLI signal report (--json for full dump,
# --panel to run the advisory LLM agent panel on-demand) # --panel to run the advisory LLM agent panel on-demand)
python telegram_bot.py # the bot (refuses to boot without credentials) python telegram_bot.py # the bot (refuses to boot without credentials)
@@ -29,13 +29,13 @@ python backtest/gpu_backtest.py --pair XAUUSD --start 2025-01-01 --end 2025-06-0
# 2026-08-05: NOT shipped — backtest/research/fixedtp_2026-08-05.md) # 2026-08-05: NOT shipped — backtest/research/fixedtp_2026-08-05.md)
``` ```
**`test_all.py` is NOT pytest** — it uses a custom `@test("name")` decorator that runs each test immediately at import; don't grep for `def test_` or try `pytest test_all.py`. Run the whole file; it prints per-module results and exits non-zero on failure. There is no way to run a single test short of commenting modules out. Tests must stay green; every new gate/knob needs tests. When patching weekend logic in tests, patch `providers.base._is_forex_weekend` — but know there are THREE independent signal-path weekend implementations (`providers/base.py`, `signal_aggregator.py`, and an inline copy in `backtest/gpu_backtest.py:_is_xau_blocked_by_calendar`) plus the advisory-only continuity copy in `advisory_outcome.py`; patching one does not reach the others. **`test_all.py` is NOT pytest** — it uses a custom `@test("name")` decorator that runs each test immediately at import; don't grep for `def test_` or try `pytest test_all.py`. Run the whole file; it prints per-module results and exits non-zero on failure. There is no way to run a single test short of commenting modules out. Tests must stay green; every new gate/knob needs tests. When patching weekend logic in tests, patch `providers.base._is_forex_weekend` — but know there are THREE independent weekend implementations (`providers/base.py`, `signal_aggregator.py`, and an inline copy in `backtest/gpu_backtest.py:_is_xau_blocked_by_calendar`); patching the provider one does not reach the other two.
Python 3.12; a `venv/` exists in-repo. Dependencies in `requirements.txt` (numpy, pandas, requests, twelvedata, python-telegram-bot ≥20, websockets). Python 3.12; a `venv/` exists in-repo. Dependencies in `requirements.txt` (numpy, pandas, requests, twelvedata, python-telegram-bot ≥20, websockets).
## Configuration & credentials ## Configuration & credentials
Precedence for every credential/deployment setting: **env var → config.json → refuse to run** (no hardcoded fallbacks). A gitignored repo-root `.env` is loaded at `app_config` import via `os.environ.setdefault`. `config.json` (gitignored; schema in `config.json.example`) holds secrets plus deployment settings: `data_providers.active` (the live-feed switch), `data_providers.streaming` (optional tick-stream provider for the level-breach monitor), `scanning.pairs`/`timeframes`, per-pair `blocked_hours_utc`, `log_level`. Key env vars: `TELEGRAM_TOKEN`, `TELEGRAM_CHAT_ID`, `TELEGRAM_CHANNEL_ID`, `TWELVE_DATA_API_KEY`, `TIINGO_API_KEY`, `OANDA_ACCOUNT_ID`/`OANDA_API_TOKEN`/`OANDA_BASE_URL`, `DEEPSEEK_API_KEY` (news LLM; the `deepseek` config.json block holds model/enabled/timeout/daily cap). The `agent_panel` config.json block (all-defaults = feature dark) controls the advisory LLM agent panel: `enabled`, per-surface `telegram`/`dashboard` toggles, `min_quality`, `max_panels_per_day`, `reuse_ttl_seconds`, plus the technical-payload enrichment knobs `candles_enabled`/`candle_bars_primary`/`candle_bars_confirm`. The dark-by-default `market_outlook` block controls the hourly advisory market brief: `enabled`, `telegram`, `max_per_day`, `candles_bars`; `MARKET_OUTLOOK_ENABLED` overrides its enabled setting. Precedence for every credential/deployment setting: **env var → config.json → refuse to run** (no hardcoded fallbacks). A gitignored repo-root `.env` is loaded at `app_config` import via `os.environ.setdefault`. `config.json` (gitignored; schema in `config.json.example`) holds secrets plus deployment settings: `data_providers.active` (the live-feed switch), `data_providers.streaming` (optional tick-stream provider for the level-breach monitor), `scanning.pairs`/`timeframes`, per-pair `blocked_hours_utc`, `log_level`. Key env vars: `TELEGRAM_TOKEN`, `TELEGRAM_CHAT_ID`, `TELEGRAM_CHANNEL_ID`, `TWELVE_DATA_API_KEY`, `TIINGO_API_KEY`, `OANDA_ACCOUNT_ID`/`OANDA_API_TOKEN`/`OANDA_BASE_URL`, `DEEPSEEK_API_KEY` (news LLM; the `deepseek` config.json block holds model/enabled/timeout/daily cap). The `agent_panel` config.json block (all-defaults = feature dark) controls the advisory LLM agent panel: `enabled`, per-surface `telegram`/`dashboard` toggles, `min_quality`, `max_panels_per_day`, `reuse_ttl_seconds`.
## Architecture ## Architecture
@@ -59,10 +59,8 @@ Signal currency everywhere is `SignalStrength` (−2..+2). DataFrames are UTC tz
- **`engine.py`** — `TechnicalEngine`: 9 scored indicators (ema_trend, rsi, macd, bollinger, adx, roc, keltner, divergence, candlestick) + ATR as context. Regime classification (trending_up/down, `trending` = mixed/direction-UNKNOWN, transitioning, ranging) drives per-indicator weight multipliers and regime-aware RSI reinterpretation. Compute order matters (ATR first; MACD always computed even when not a pillar — divergence needs the histogram). Private series ride on the result as `_underscore` attrs; scorers/formatters skip anything `_`-prefixed or weight-0. - **`engine.py`** — `TechnicalEngine`: 9 scored indicators (ema_trend, rsi, macd, bollinger, adx, roc, keltner, divergence, candlestick) + ATR as context. Regime classification (trending_up/down, `trending` = mixed/direction-UNKNOWN, transitioning, ranging) drives per-indicator weight multipliers and regime-aware RSI reinterpretation. Compute order matters (ATR first; MACD always computed even when not a pillar — divergence needs the histogram). Private series ride on the result as `_underscore` attrs; scorers/formatters skip anything `_`-prefixed or weight-0.
- **`signal_aggregator.py`** — MTF agreement, diminishing-decay penalty system, exhaustion hard block, quality grading, then an ordered chain of post-grade gates (session, XAU SELL quality/vol/hour, regime blocks, M5 noise gates), then three setup generators: XAU ATR (A+/A), generic swing-based, plus the M5 pullback (`XAU_PB`) and chart-pattern (`XAU_CP`, `chart_patterns.py`) entry paths. The PB/CP generators run AFTER the gate chain and force quality to "B"; the CP path deliberately accepts grade-F composites and re-checks only session/day/big-candle (via warning prefixes) plus its own SELL/vol/RSI gates — it does NOT re-check exhaustion or the M5 min-score/MTF gates. **This is validated behavior, not a bug** (2026-08-04 audit: 226/586 trades in the shipped M5 12mo book are XAU_CP, +38.9R, 104 of them exhaustion-flagged and net-positive) — changing it is a rule-1 revalidation event. `TradeSetup.strategy` selects trailing params; trail params are stamped on the setup from the *producing* config so per-TF overrides reach formatters. - **`signal_aggregator.py`** — MTF agreement, diminishing-decay penalty system, exhaustion hard block, quality grading, then an ordered chain of post-grade gates (session, XAU SELL quality/vol/hour, regime blocks, M5 noise gates), then three setup generators: XAU ATR (A+/A), generic swing-based, plus the M5 pullback (`XAU_PB`) and chart-pattern (`XAU_CP`, `chart_patterns.py`) entry paths. The PB/CP generators run AFTER the gate chain and force quality to "B"; the CP path deliberately accepts grade-F composites and re-checks only session/day/big-candle (via warning prefixes) plus its own SELL/vol/RSI gates — it does NOT re-check exhaustion or the M5 min-score/MTF gates. **This is validated behavior, not a bug** (2026-08-04 audit: 226/586 trades in the shipped M5 12mo book are XAU_CP, +38.9R, 104 of them exhaustion-flagged and net-positive) — changing it is a rule-1 revalidation event. `TradeSetup.strategy` selects trailing params; trail params are stamped on the setup from the *producing* config so per-TF overrides reach formatters.
- **`pair_signals/`** — per-pair config factories. **Always use the factory (`produce_signal` / `make_config`), never bare `PlatformConfig()`** — the factory applies the config.json overlay first, then the validated per-TF deltas (M5 hour-9 block, M15 trail arm 0.70, M15 SELL-hour block, etc.; evidence in the docstring). A test enforces factory ≡ inline-config equivalence. Confirmation map: M5→[M15,H1], M15→[M5,H1], H1→[M15,H4], H4→[H1,D1]. - **`pair_signals/`** — per-pair config factories. **Always use the factory (`produce_signal` / `make_config`), never bare `PlatformConfig()`** — the factory applies the config.json overlay first, then the validated per-TF deltas (M5 hour-9 block, M15 trail arm 0.70, M15 SELL-hour block, etc.; evidence in the docstring). A test enforces factory ≡ inline-config equivalence. Confirmation map: M5→[M15,H1], M15→[M5,H1], H1→[M15,H4], H4→[H1,D1].
- **`signal_log.py` / `advisory_outcome.py`** — SQLite signal-history logger (stdlib only). Hooked into `SignalAggregator.analyze_pair` only (the live entry point — the backtester never writes here, preserving parity), so every live scan from every surface (Telegram `telegram`, CLI `cli`, Oanda daemon `oanda_daemon`, dashboard `dashboard`) lands one row in `logs/signals.db`: signal/grade/price, MTF + trade-setup + reasons JSON, full per-TF indicator snapshots, the last 14 closed OHLCV candles per fetched TF, and the news at scan time (event risk + calendar events + scored headlines JSON, re-fetched via the NewsEngine class-level caches). Config.json `signal_log` block (`enabled`, `path`; env `SIGNAL_LOG_ENABLED`/`SIGNAL_LOG_PATH`). **Fail-safe by contract** — a logging failure can never break a scan. Panel fields arrive post-hoc: `log_signal` stashes the rowid and `agent_panel.attach_panel` writes verdict + creation/cache provenance + payload hash onto the same row. A versioned advisory-only outcome then advances from later stored closed-candle snapshots using next-open reanchoring, close-only trailing, continuity checks that allow only scheduled weekend/maintenance closures, a 300-bar timeout, and both $0.30/$0.65 cost views; it is explicitly a cleaned-mid evidence model, not account execution or raw-path backtest P&L. No rowid means silent no-op.
- **`news_engine.py`** — Forex Factory calendar (MyFXBook RSS fallback) + RSS keyword sentiment with gold-inverse keywords. Class-level caches (fresh engine per scan). Optional DeepSeek enrichment via `llm_client.py` (requests-only, no SDK): batched headline sentiment overriding keyword scores, gold-direction on release surprises, and a market-summary blurb (`SignalReport.news_summary`) — **display/warning-only, keyword fallback on any failure, inert without a key**; prompts are static module constants (DeepSeek prefix-cache pricing) and requests send `thinking: disabled` (V4-Flash reasoning otherwise blows the timeout). By XAU-tuned default, event risk and sentiment **warn but do not gate signals** (`event_risk_caps_signals=False`, `sentiment_affects_signals=False`) — the LLM does not change that. - **`news_engine.py`** — Forex Factory calendar (MyFXBook RSS fallback) + RSS keyword sentiment with gold-inverse keywords. Class-level caches (fresh engine per scan). Optional DeepSeek enrichment via `llm_client.py` (requests-only, no SDK): batched headline sentiment overriding keyword scores, gold-direction on release surprises, and a market-summary blurb (`SignalReport.news_summary`) — **display/warning-only, keyword fallback on any failure, inert without a key**; prompts are static module constants (DeepSeek prefix-cache pricing) and requests send `thinking: disabled` (V4-Flash reasoning otherwise blows the timeout). By XAU-tuned default, event risk and sentiment **warn but do not gate signals** (`event_risk_caps_signals=False`, `sentiment_affects_signals=False`) — the LLM does not change that.
- **`agent_panel.py`** — ai-hedge-fund-style advisory panel: 4 DeepSeek persona calls (macro/technical/risk/contrarian, concurrent on a panel-private pool) + 1 synthesizer producing an **ADVISORY-ONLY** consensus (`AGREE/NEUTRAL/OPPOSE` + confidence + rationale) written onto `SignalReport.panel_*` fields. Attached **post-hoc** by consumers (`telegram_bot` alert path, `main --panel`, `dashboard_publisher`) — never runs inside `analyze_from_dataframes` (backtest parity, test-locked) and no gate reads its fields. Persona payloads are role-filtered: macro gets high-impact events plus only code-scored directional news; technical gets strategy/pattern, primary indicators, MTF, levels, and ATR-normalized candles; risk gets setup strategy, strategy cost assumption, trailing semantics, volatility/events/warnings (never the median candle range as spread); contrarian gets directional news, setup, timestamped latest-closed price, indicators, reasons, and warnings. Spend is bounded by exact cache, full-evidence-identity TTL reuse, and `max_panels_per_day`; changed evidence never inherits a direction-only cached verdict. Verdict provenance logs creation time, `MISS`/`EXACT_HIT`/`TTL_REUSE`, and a SHA-256 payload hash. Prompts forbid outside knowledge and outputs remain short. - **`agent_panel.py`** — ai-hedge-fund-style advisory panel: 4 DeepSeek persona calls (macro/technical/risk/contrarian, concurrent on a panel-private pool) + 1 synthesizer producing an **ADVISORY-ONLY** consensus (`AGREE/NEUTRAL/OPPOSE` + confidence + rationale) written onto `SignalReport.panel_*` fields. Attached **post-hoc** by consumers (`telegram_bot` alert path, `main --panel`, `dashboard_publisher`) — never runs inside `analyze_from_dataframes` (backtest parity, test-locked) and no gate reads its fields. Static prompts (prefix caching); spend bounded by an exact per-bar cache, a reuse TTL keyed (symbol, TF, direction) so persisting signals don't re-spend every bar, and a `max_panels_per_day` UTC budget (default 20 ≈ ≤100 requests) on top of the shared 500/day client cap. `<2` valid persona takes skips the synthesizer; any failure degrades to no panel, never a changed signal.
- **`market_outlook.py`** — hourly DeepSeek market brief for XAU/USD, using ATR-normalized M15/H1/H4 candles, visible indicator states, levels, news, and immutable recent signal history. It returns a written **ADVISORY-ONLY** bias/setup-forming assessment and persists it independently in `outlooks` within `logs/signals.db` together with the news, indicator/regime/level, and candle evidence sent to the model. It may be sent post-scan by Telegram or requested with `main.py --outlook`; it never imports or runs inside the signal path. A lock enforces hourly dedup plus its own UTC-day cap (default 12), and provider/news failures leave a safe partial brief while LLM/database failures degrade to no outlook.
- **`telegram_bot.py`** — owner-locked bot + subscriber channel; candle-boundary auto-scan loops (M5/M15/H1/H4, boundary+10 s), alert quality threshold + fingerprint dedup (committed only after a successful send), 📰 LLM market-context line on alerts when available, WebSocket level-breach monitor fed by the streaming provider (`data_providers.streaming`, falling back to the active provider; auto-disabled when the resolved provider can't stream or has no API key — Oanda does not stream, Tiingo does). Rotating log: `logs/telegram.log`, 3-h rotation × 240 backups. - **`telegram_bot.py`** — owner-locked bot + subscriber channel; candle-boundary auto-scan loops (M5/M15/H1/H4, boundary+10 s), alert quality threshold + fingerprint dedup (committed only after a successful send), 📰 LLM market-context line on alerts when available, WebSocket level-breach monitor fed by the streaming provider (`data_providers.streaming`, falling back to the active provider; auto-disabled when the resolved provider can't stream or has no API key — Oanda does not stream, Tiingo does). Rotating log: `logs/telegram.log`, 3-h rotation × 240 backups.
- **`oanda_daemon.py` / `oanda_executor.py`** — practice-account auto-execution. Book-parity rules (see daemon docstring): no broker-side TP (trailing model — TP1/TP2 advisory), 5-bar per-TF cooldown (wall-clock approximation, so it expires over weekends where the backtester would still count bars), trail ratchets at bar close once per *observed* bar (a missed/failed cycle skips those bars entirely — the intra-gap high/low never enters the peak). Accepted divergence: one position total; orders send `positionFill=OPEN_ONLY` so a state/broker divergence cancels the order instead of silently netting. State/broker re-sync runs at boot and after any suspicious fill (`_reconcile_positions`). - **`oanda_daemon.py` / `oanda_executor.py`** — practice-account auto-execution. Book-parity rules (see daemon docstring): no broker-side TP (trailing model — TP1/TP2 advisory), 5-bar per-TF cooldown (wall-clock approximation, so it expires over weekends where the backtester would still count bars), trail ratchets at bar close once per *observed* bar (a missed/failed cycle skips those bars entirely — the intra-gap high/low never enters the peak). Accepted divergence: one position total; orders send `positionFill=OPEN_ONLY` so a state/broker divergence cancels the order instead of silently netting. State/broker re-sync runs at boot and after any suspicious fill (`_reconcile_positions`).
- **`dashboard_publisher.py` / `static/dashboard.html`** — web dashboard as a **static publisher** (live LAMP server): standalone daemon scans at candle closes via `produce_signal` and atomically writes `snapshot.json`/`news.json`/`dashboard.html` to `$DASHBOARD_OUT_DIR` (Apache serves them; page is public by owner decision). **Hard independence rule: never imports, and is never imported by, `telegram_bot`/`oanda_daemon`** — confirmation map and boundary math are deliberate local copies, so a web-side failure can only mean a stale page. Due-TF classification uses the RAW wake time (flooring wake−buffer skips every H1/H4 scan — 2026-08-02 bug). News panel reads `NewsEngine.get_fundamental_context()` plus the additive `get_week_events()` accessor (display-only; no signal path calls it); as a separate process it has its own news caches and DeepSeek daily cap. News/calendar rows are clickable (detail dialog); headlines carry `id` = sha1(headline) in news.json, and exactly one on-demand endpoint exists: `static/news_summary.php` (installed by `install_static`), which serves a DeepSeek gold-impact summary per headline — whitelist-gated against news.json ids, cache-forever in `ai/`, own fail-closed daily cap (150/day, file-based, separate from the publisher's), key at `/etc/gold-dashboard/deepseek.key`, never reads repo config. Deployment checklist: `deploy/README-dashboard.md`. - **`dashboard_publisher.py` / `static/dashboard.html`** — web dashboard as a **static publisher** (live LAMP server): standalone daemon scans at candle closes via `produce_signal` and atomically writes `snapshot.json`/`news.json`/`dashboard.html` to `$DASHBOARD_OUT_DIR` (Apache serves them; page is public by owner decision). **Hard independence rule: never imports, and is never imported by, `telegram_bot`/`oanda_daemon`** — confirmation map and boundary math are deliberate local copies, so a web-side failure can only mean a stale page. Due-TF classification uses the RAW wake time (flooring wake−buffer skips every H1/H4 scan — 2026-08-02 bug). News panel reads `NewsEngine.get_fundamental_context()` plus the additive `get_week_events()` accessor (display-only; no signal path calls it); as a separate process it has its own news caches and DeepSeek daily cap. News/calendar rows are clickable (detail dialog); headlines carry `id` = sha1(headline) in news.json, and exactly one on-demand endpoint exists: `static/news_summary.php` (installed by `install_static`), which serves a DeepSeek gold-impact summary per headline — whitelist-gated against news.json ids, cache-forever in `ai/`, own fail-closed daily cap (150/day, file-based, separate from the publisher's), key at `/etc/gold-dashboard/deepseek.key`, never reads repo config. Deployment checklist: `deploy/README-dashboard.md`.
@@ -70,7 +68,7 @@ Signal currency everywhere is `SignalStrength` (−2..+2). DataFrames are UTC tz
## Rules for making changes (read before editing signal logic) ## Rules for making changes (read before editing signal logic)
1. **Never change a tuned value without re-validation.** Acceptance rule: positive total R in BOTH train and validation halves of a ≥12-month window, PF ≥ 1.3, at realistic cost (XAU $0.65 round trip headline; retain $0.30 for TD-era cross-book comparability), trailing exits, next-open fills. `round_trip_cost` stays $0.30 because it also drives the live TP spread gate; changing it is a separate rule-1 event. Marginal edges must additionally pass the 2024-06→2025-03 out-of-sample window — within-year splits alone have approved configs that later failed OOS. 1. **Never change a tuned value without re-validation.** Acceptance rule: positive total R in BOTH train and validation halves of a ≥12-month window, PF ≥ 1.3, at realistic cost (XAU $0.30 round trip), trailing exits, next-open fills. Marginal edges must additionally pass the 2024-06→2025-03 out-of-sample window — within-year splits alone have approved configs that later failed OOS.
2. **Per-timeframe overrides live in the pair factory**, never in global defaults — the same knob has opposite effects on different TFs (proven repeatedly: `tp_floor_rr`, hour-9 block, trail params). Document validation evidence in the factory docstring. 2. **Per-timeframe overrides live in the pair factory**, never in global defaults — the same knob has opposite effects on different TFs (proven repeatedly: `tp_floor_rr`, hour-9 block, trail params). Document validation evidence in the factory docstring.
3. **Pair isolation is a hard invariant.** XAU gates in the aggregator are display-name-gated so a future pair port cannot inherit gold-calibrated logic by accident. 3. **Pair isolation is a hard invariant.** XAU gates in the aggregator are display-name-gated so a future pair port cannot inherit gold-calibrated logic by accident.
4. **Live == backtest parity.** If you add a live gate, add it to the replay path in the same commit; all time filters evaluate bar-close time in both paths. 4. **Live == backtest parity.** If you add a live gate, add it to the replay path in the same commit; all time filters evaluate bar-close time in both paths.
@@ -82,15 +80,12 @@ Signal currency everywhere is `SignalStrength` (−2..+2). DataFrames are UTC tz
## 2026-08-04 QA audit — owner decisions ledger ## 2026-08-04 QA audit — owner decisions ledger
- **Weekend calendar: FIXED (owner-approved 2026-08-04).** All three signal-path `_is_forex_weekend` copies are NY-anchored (Fri 17:00 → Sun 18:00 ET; during US DST that is Fri 21:00 UTC — measured on 5yr of Oanda bars). Kills the weekly false-"STALE DATA/feed lagging" hours. Book impact at the audit HEAD: M15 byte-identical; M5 12mo dropped exactly ONE trade — a phantom whose signal fired on the final summer-Friday bar (close 21:00 UTC) and "filled" at Sunday's reopen 49 h later (−1.08R, live-impossible), leaving 555 trades / +98.64R / PF≈1.50 @ $0.65 (was 556/+97.56/1.48). The current-HEAD canonical reference is recorded below. A Module 21 test sweeps those copies plus the advisory outcome's continuity copy hourly across both DST transitions. - **Weekend calendar: FIXED (owner-approved 2026-08-04).** All three `_is_forex_weekend` copies are NY-anchored (Fri 17:00 → Sun 18:00 ET; during US DST that is Fri 21:00 UTC — measured on 5yr of Oanda bars). Kills the weekly false-"STALE DATA/feed lagging" hours. Book impact, fully accounted: M15 byte-identical; M5 12mo dropped exactly ONE trade — a phantom whose signal fired on the final summer-Friday bar (close 21:00 UTC) and "filled" at Sunday's reopen 49 h later (−1.08R, live-impossible). M5 12mo reference at HEAD: 555 trades, +98.64R PF≈1.50 @ $0.65 (was 556/+97.56/1.48). A Module 21 test sweeps all three copies hourly across both DST transitions.
- **Spread in exit-trigger detection (rule 6): owner DECLINED 2026-08-04.** Backtest stop triggers stay on raw mid High/Low; the measured optimism (~4–6% of total R: −3.55R @ $0.30, −4.24R @ $0.65 on the M15 12mo book) is accepted and books stay comparable. Re-open only as a conscious re-pin campaign. - **Spread in exit-trigger detection (rule 6): owner DECLINED 2026-08-04.** Backtest stop triggers stay on raw mid High/Low; the measured optimism (~4–6% of total R: −3.55R @ $0.30, −4.24R @ $0.65 on the M15 12mo book) is accepted and books stay comparable. Re-open only as a conscious re-pin campaign.
- **`FB2_EMA_LO`: HARDENED 2026-08-04 (delegated).** Research override for the EMA200-proximity band, default = tuned 0.5; an unparseable value now degrades to the default with a warning instead of crashing every scan. Documented here — it is a knob that exists. - **`FB2_EMA_LO`: HARDENED 2026-08-04 (delegated).** Research override for the EMA200-proximity band, default = tuned 0.5; an unparseable value now degrades to the default with a warning instead of crashing every scan. Documented here — it is a knob that exists.
- **H4 seeding divergence: parked.** Live H4 fetches only 380 bars vs the backtest's unbounded history — ~2% of sampled H4 bars flip an indicator signal (composite Δ up to 12.3). M5/M15/H1 clean at their real fetch depths. Fold into any future H4 campaign; H4 remains "unproven for auto-execution". - **H4 seeding divergence: parked.** Live H4 fetches only 380 bars vs the backtest's unbounded history — ~2% of sampled H4 bars flip an indicator signal (composite Δ up to 12.3). M5/M15/H1 clean at their real fetch depths. Fold into any future H4 campaign; H4 remains "unproven for auto-execution".
## Current state notes (2026-08-14) ## Current state notes (2026-07-30)
- Oanda v20 is the live feed (`data_providers.active: "oanda"`). The 2026-08-04 re-pin made the 27 `backtest/oanda_*.csv` books the baseline; all TD-era books are superseded. Current-HEAD M5 canonical book: `backtest/oanda_m5_12mo_065_nopb_20260814.csv`, +106.82R PF 1.51 at $0.65 (both halves positive; PF 1.46 / 1.55). M15 reference: +73.93R PF 1.53. See `backtest/research/repin_verification_2026-08-14.md`. - Oanda v20 is the live feed (`data_providers.active: "oanda"`), but the pinned backtest books are still Twelve-Data-pinned and the backtest cost assumption is $0.30 while measured real spreads are ~$0.65–0.72 — re-pin books on Oanda bars before treating feed-switch comparisons as valid (see Oanda/TD divergence study in memory).
- Cost reporting is dual: $0.65 is the measured-Oanda realistic headline ($0.65-$0.72 round trip), while $0.30 remains for cross-book comparability and as the live TP spread-gate input. Raising `round_trip_cost` from $0.30 would move the TP floor from about $9 to $19.50 and requires full rule-1 revalidation.
- Twelve Data is signal-incompatible with Oanda (Phase 1: about 19% M5 bar-direction disagreement and a systematic ~$1 high bias; see `backtest/research/feed_divergence_2026-08-14.md`). On an Oanda candle outage, halt auto-execution rather than switching to TD candles.
- Tiingo is streaming-only (`data_providers.streaming: "tiingo"`), not a designated candle fallback. Oanda-Tiingo signal parity is documented in `backtest/research/signal_parity_2026-08-14.md`; Phase 4's two-week live shadow scan remains a manual operational step.
- Human-facing docs in `docs/` (`user_guide.html`, `beginner_guide.html` + PDF builds) must be kept in sync when signal behavior changes. - Human-facing docs in `docs/` (`user_guide.html`, `beginner_guide.html` + PDF builds) must be kept in sync when signal behavior changes.
-343
View File
@@ -1,343 +0,0 @@
"""Deterministic advisory trade outcomes from stored closed candles.
This module measures the hypothetical setup reviewed by the LLM panel. It
never changes a signal, panel verdict, order, or daemon state. The model uses
the same next-open reanchoring and close-only trailing rules as the backtester,
but its evidence is the cleaned mid-candle snapshots stored in signals.db.
"""
from __future__ import annotations
import math
from datetime import datetime, timedelta
from typing import Any, Dict, Iterable, List, Optional, Tuple
from zoneinfo import ZoneInfo
MODEL = "stored-closed-mid-next-open-trail-v1"
MAX_HOLD_BARS = 300
REFERENCE_COST = 0.30
REALISTIC_COST = 0.65
_TF_ALIASES = {
"M1": "1min", "1MIN": "1min", "1min": "1min",
"M5": "5min", "5MIN": "5min", "5min": "5min",
"M15": "15min", "15MIN": "15min", "15min": "15min",
"M30": "30min", "30MIN": "30min", "30min": "30min",
"H1": "1h", "1H": "1h", "1h": "1h",
"H4": "4h", "4H": "4h", "4h": "4h",
"D1": "1day", "1D": "1day", "1d": "1day",
"1DAY": "1day", "1day": "1day",
}
_TF_SECONDS = {
"1min": 60,
"5min": 5 * 60,
"15min": 15 * 60,
"30min": 30 * 60,
"1h": 60 * 60,
"4h": 4 * 60 * 60,
"1day": 24 * 60 * 60,
}
_NY_TZ = ZoneInfo("America/New_York")
def _is_forex_weekend(now: datetime) -> bool:
"""NY-anchored XAU weekend closure; test-locked to the signal copies."""
ny = now.astimezone(_NY_TZ)
weekday = ny.weekday()
if weekday == 4:
return ny.hour >= 17
if weekday == 5:
return True
if weekday == 6:
return ny.hour < 18
return False
def normalize_timeframe(value: Any) -> str:
text = str(getattr(value, "value", value) or "")
return _TF_ALIASES.get(text, _TF_ALIASES.get(text.upper(), text))
def timeframe_value(mapping: Any, timeframe: Any) -> Any:
"""Read a timeframe-keyed snapshot regardless of enum/storage alias."""
if not isinstance(mapping, dict):
return None
wanted = normalize_timeframe(timeframe)
for key, value in mapping.items():
if normalize_timeframe(key) == wanted:
return value
return None
def _finite(value: Any) -> Optional[float]:
try:
number = float(value)
except (TypeError, ValueError):
return None
return number if math.isfinite(number) else None
def _timestamp(value: Any) -> Optional[datetime]:
try:
return datetime.fromisoformat(str(value).replace(" ", "T"))
except (TypeError, ValueError):
return None
def initial_outcome(
setup: Optional[Dict[str, Any]],
indicators: Optional[Dict[str, Any]],
timeframe: Any,
) -> Tuple[str, Dict[str, Any]]:
"""Freeze the setup and deterministic evaluation policy."""
if not setup:
return "NOT_APPLICABLE", {
"model": MODEL,
"status": "NOT_APPLICABLE",
"reason": "panel row has no trade setup",
}
tf = normalize_timeframe(timeframe)
analysis = timeframe_value(indicators, tf)
signal_bar = analysis.get("timestamp") if isinstance(analysis, dict) else None
direction = str(setup.get("direction", ""))
entry = _finite(setup.get("entry"))
stop = _finite(setup.get("stop_loss"))
tp1 = _finite(setup.get("take_profit_1"))
arm = _finite(setup.get("trail_arm_r"))
distance = _finite(setup.get("trail_dist_r"))
risk = abs(entry - stop) if entry is not None and stop is not None else 0.0
if (direction not in ("BUY", "SELL") or not signal_bar
or _timestamp(signal_bar) is None or risk <= 0 or tp1 is None
or arm is None or distance is None or arm < 0 or distance < 0):
return "UNRESOLVABLE", {
"model": MODEL,
"status": "UNRESOLVABLE",
"reason": "invalid or incomplete setup/outcome seed",
}
state = {
"model": MODEL,
"status": "PENDING",
"timeframe": tf,
"signal_bar_at": str(signal_bar),
"setup": {
"direction": direction,
"reported_entry": entry,
"stop_distance": risk,
"tp1_distance": abs(tp1 - entry),
"strategy": setup.get("strategy"),
"setup_pattern": setup.get("setup_pattern"),
},
"policy": {
"entry": "next_primary_bar_open_reanchored",
"max_hold_bars": MAX_HOLD_BARS,
"trail_arm_r": arm,
"trail_dist_r": distance,
"tp1_is_advisory": arm > 0,
"reference_cost": REFERENCE_COST,
"realistic_cost": REALISTIC_COST,
},
"progress": {
"last_processed_bar": str(signal_bar),
"entry_at": None,
"entry_price": None,
"initial_stop": None,
"current_stop": None,
"peak": None,
"bars_held": 0,
"mfe_r": 0.0,
"mae_r": 0.0,
},
"result": None,
}
return "PENDING", state
def _valid_bars(bars: Iterable[Dict[str, Any]]) -> List[Dict[str, Any]]:
valid = []
for bar in bars:
stamp = _timestamp(bar.get("timestamp")) if isinstance(bar, dict) else None
values = [_finite(bar.get(key)) for key in ("open", "high", "low", "close")]
if stamp is None or any(value is None for value in values):
continue
valid.append({
"timestamp": str(bar["timestamp"]),
"_timestamp": stamp,
"open": values[0], "high": values[1],
"low": values[2], "close": values[3],
})
valid.sort(key=lambda item: item["_timestamp"])
return valid
def _has_continuity(
bars: List[Dict[str, Any]], last: datetime, timeframe: str
) -> bool:
"""Reject missing tradable bars while allowing scheduled XAU closures."""
seconds = _TF_SECONDS.get(normalize_timeframe(timeframe))
if seconds is None:
return False
step = timedelta(seconds=seconds)
previous = last
seen = {last}
for bar in bars:
stamp = bar["_timestamp"]
if stamp <= previous or stamp in seen:
continue
expected = previous + step
while expected < stamp:
ny = expected.astimezone(_NY_TZ)
daily_maintenance = ny.weekday() < 4 and ny.hour == 17
if not (_is_forex_weekend(expected) or daily_maintenance):
return False
expected += step
if expected != stamp:
return False
previous = stamp
seen.add(stamp)
return True
def _finish(
state: Dict[str, Any], reason: str, exit_at: str, exit_price: float
) -> None:
setup = state["setup"]
progress = state["progress"]
entry = float(progress["entry_price"])
risk = float(setup["stop_distance"])
pnl = ((exit_price - entry) if setup["direction"] == "BUY"
else (entry - exit_price))
gross_r = pnl / risk
state["status"] = "RESOLVED"
state["result"] = {
"exit_at": exit_at,
"exit_price": exit_price,
"exit_reason": reason,
"bars_held": progress["bars_held"],
"gross_r": gross_r,
"net_r_030": (pnl - REFERENCE_COST) / risk,
"net_r_065": (pnl - REALISTIC_COST) / risk,
"mfe_r": progress["mfe_r"],
"mae_r": progress["mae_r"],
}
def advance_outcome(
state: Dict[str, Any], bars: Iterable[Dict[str, Any]]
) -> Dict[str, Any]:
"""Advance one pending outcome with overlapping closed-bar snapshots."""
if state.get("status") != "PENDING":
return state
try:
progress = state["progress"]
setup = state["setup"]
policy = state["policy"]
last = _timestamp(progress["last_processed_bar"])
signal_bar = _timestamp(state["signal_bar_at"])
if last is None or signal_bar is None:
raise ValueError("invalid progress timestamp")
buy = setup["direction"] == "BUY"
risk = float(setup["stop_distance"])
trail_arm = float(policy["trail_arm_r"])
trail_distance = float(policy["trail_dist_r"])
valid_bars = _valid_bars(bars)
has_future = any(bar["_timestamp"] > last for bar in valid_bars)
has_overlap = any(bar["_timestamp"] == last for bar in valid_bars)
if has_future and not has_overlap:
state["status"] = "UNRESOLVABLE"
state["result"] = None
state["reason"] = "stored candle snapshots lost continuity"
return state
if has_future and not _has_continuity(valid_bars, last,
state["timeframe"]):
state["status"] = "UNRESOLVABLE"
state["result"] = None
state["reason"] = "stored candle snapshots contain an internal gap"
return state
for bar in valid_bars:
stamp = bar["_timestamp"]
if stamp <= signal_bar or stamp <= last:
continue
if progress["entry_price"] is None:
entry = bar["open"]
initial_stop = entry - risk if buy else entry + risk
progress.update({
"entry_at": bar["timestamp"],
"entry_price": entry,
"initial_stop": initial_stop,
"current_stop": initial_stop,
"peak": entry,
})
entry = float(progress["entry_price"])
current_stop = float(progress["current_stop"])
initial_stop = float(progress["initial_stop"])
progress["bars_held"] += 1
progress["last_processed_bar"] = bar["timestamp"]
last = stamp
if buy:
mfe = (bar["high"] - entry) / risk
mae = (entry - bar["low"]) / risk
stop_hit = bar["low"] <= current_stop
else:
mfe = (entry - bar["low"]) / risk
mae = (bar["high"] - entry) / risk
stop_hit = bar["high"] >= current_stop
progress["mfe_r"] = max(float(progress["mfe_r"]), max(0.0, mfe))
progress["mae_r"] = max(float(progress["mae_r"]), max(0.0, mae))
if stop_hit:
exit_price = current_stop
if (buy and bar["open"] < current_stop) or \
(not buy and bar["open"] > current_stop):
exit_price = bar["open"]
reason = "SL" if current_stop == initial_stop else "TRAIL"
_finish(state, reason, bar["timestamp"], exit_price)
return state
if trail_arm <= 0:
target = entry + setup["tp1_distance"] if buy \
else entry - setup["tp1_distance"]
target_hit = bar["high"] >= target if buy else bar["low"] <= target
if target_hit:
exit_price = target
if (buy and bar["open"] > target) or \
(not buy and bar["open"] < target):
exit_price = bar["open"]
_finish(state, "TP1", bar["timestamp"], exit_price)
return state
else:
favorable = bar["high"] if buy else bar["low"]
peak = max(float(progress["peak"]), favorable) if buy \
else min(float(progress["peak"]), favorable)
progress["peak"] = peak
excursion = ((peak - entry) if buy else (entry - peak)) / risk
if excursion >= trail_arm:
level = peak - trail_distance * risk if buy \
else peak + trail_distance * risk
progress["current_stop"] = max(current_stop, level) if buy \
else min(current_stop, level)
if progress["bars_held"] >= int(policy["max_hold_bars"]):
_finish(state, "TIMEOUT", bar["timestamp"], bar["close"])
return state
return state
except Exception as exc:
state["status"] = "UNRESOLVABLE"
state["result"] = None
state["reason"] = f"outcome evaluator error: {type(exc).__name__}"
return state
def canonical_r(state: Dict[str, Any]) -> Optional[float]:
result = state.get("result")
value = result.get("net_r_065") if isinstance(result, dict) else None
return _finite(value)
+108 -495
View File
@@ -25,39 +25,17 @@ layers bound the spend on top of that cap:
- ``max_panels_per_day`` (config, default 20) — a UTC-day budget of - ``max_panels_per_day`` (config, default 20) — a UTC-day budget of
panel *attempts*, mirroring the client's request cap. panel *attempts*, mirroring the client's request cap.
- A reuse TTL (default 3600 s) keyed on the complete reviewed evidence - A reuse TTL (default 3600 s) keyed on (symbol, timeframe, direction):
payload, ignoring only the bar timestamp. Changed price, setup, news, a persisting signal re-serves its cached verdict across bars instead
warnings, indicators, or candles always trigger a fresh review (subject of re-spending. Without this, one sticky M5 dashboard signal would
to the daily budget); genuinely identical evidence can be re-served. burn 5 calls per 5-minute bar — 1,440/day against the 500 cap.
Prompts are STATIC module constants (DeepSeek prefix-cache pricing — Prompts are STATIC module constants (DeepSeek prefix-cache pricing —
see news_engine.py); all variable content travels in the user payload. see news_engine.py); all variable content travels in the user payload.
Payload enrichment (2026-08-15/23, owner-approved, advisory-only): the
technical persona receives strategy/pattern context, raw indicator VALUES
(with a units legend in the prompt), and ATR-NORMALIZED candles per timeframe
(code does the arithmetic — raw prices are the model's weakest input;
see backtest/research/llm_news_eval_2026-08-06.md). Sized by config
``candle_bars_primary`` (default 20) / ``candle_bars_confirm``
(default 10); ``candles_enabled: false`` reverts to the pre-enrichment
payload. Prompts forbid outside/training-data knowledge and cap notes
at 15 words / rationales at 30 (output tokens are the pricey side).
Macro and contrarian receive only code-scored directional news; contrarian
also receives the timestamped closed-market snapshot, complete setup, and
visible indicators. Risk receives the stamped trailing policy and strategy
cost assumption, never the candle range mislabeled as a spread.
Verdict logging writes the verdict, creation time, cache provenance, payload
hash, and a pending deterministic advisory outcome onto the scan's signals.db
row via signal_log's rowid hand-off. No field is read by live behavior.
""" """
import datetime as _dt import datetime as _dt
import hashlib
import json
import logging import logging
import math
import threading import threading
import time import time
from collections import OrderedDict from collections import OrderedDict
@@ -66,7 +44,6 @@ from dataclasses import dataclass, field
from typing import TYPE_CHECKING, Any, Dict, List, Optional, Tuple from typing import TYPE_CHECKING, Any, Dict, List, Optional, Tuple
from app_config import get_app_config from app_config import get_app_config
from config import Timeframe
from llm_client import DEFAULT_TIMEOUT, get_llm_client from llm_client import DEFAULT_TIMEOUT, get_llm_client
if TYPE_CHECKING: # type hints only — keep the coupling one-way if TYPE_CHECKING: # type hints only — keep the coupling one-way
@@ -78,8 +55,6 @@ logger = logging.getLogger(__name__)
DEFAULT_MIN_QUALITY = "B" DEFAULT_MIN_QUALITY = "B"
DEFAULT_MAX_PANELS_PER_DAY = 20 DEFAULT_MAX_PANELS_PER_DAY = 20
DEFAULT_REUSE_TTL_SECONDS = 3600.0 DEFAULT_REUSE_TTL_SECONDS = 3600.0
DEFAULT_CANDLE_BARS_PRIMARY = 20
DEFAULT_CANDLE_BARS_CONFIRM = 10
# Quality ranks for the min_quality gate. Deliberately a local copy — # Quality ranks for the min_quality gate. Deliberately a local copy —
# telegram_bot's ranking is private and dashboard_publisher must not # telegram_bot's ranking is private and dashboard_publisher must not
@@ -103,11 +78,9 @@ _TAKE_CONTRACT = """\
Your review is advisory research on an already-generated signal; you are Your review is advisory research on an already-generated signal; you are
one voice on a panel, not the decision maker. one voice on a panel, not the decision maker.
Hard rules: use ONLY the data in the JSON you receive — do not invent Hard rules: use ONLY the data in the JSON you receive — do not invent
numbers, events, or price levels. Your training data about gold, the numbers, events, or price levels. If the evidence your role needs is
markets, and current events is stale and OFF-LIMITS: never use outside thin or missing, answer "neutral" with confidence below 30, unless your
knowledge; if a fact is not in the JSON, treat it as unknown. If the role's instructions say otherwise.
evidence your role needs is thin or missing, answer "neutral" with
confidence below 30, unless your role's instructions say otherwise.
Confidence scale (0-100): 80-100 strong one-sided evidence; 50-79 solid Confidence scale (0-100): 80-100 strong one-sided evidence; 50-79 solid
case with some doubt; 30-49 mixed evidence; below 30 little basis to case with some doubt; 30-49 mixed evidence; below 30 little basis to
judge. judge.
@@ -118,7 +91,7 @@ not "H4"; describe what an indicator MEANS ("the price has climbed
unusually fast"), never its name or number. unusually fast"), never its name or number.
Reply with ONLY JSON: Reply with ONLY JSON:
{"stance": "support"|"oppose"|"neutral", "confidence": <0-100>, {"stance": "support"|"oppose"|"neutral", "confidence": <0-100>,
"note": "<max 15 words>"} "note": "<max 25 words>"}
"stance" says whether YOUR analysis supports the proposed trade "stance" says whether YOUR analysis supports the proposed trade
direction, opposes it, or is genuinely mixed; "confidence" is how sure direction, opposes it, or is genuinely mixed; "confidence" is how sure
you are of your stance; "note" is your single most important point. you are of your stance; "note" is your single most important point.
@@ -128,11 +101,9 @@ _MACRO_SYSTEM = """\
You are the macro analyst on a gold (XAU/USD) trade-review panel. You You are the macro analyst on a gold (XAU/USD) trade-review panel. You
receive JSON: a proposed trade brief ("brief": direction, timeframe, receive JSON: a proposed trade brief ("brief": direction, timeframe,
quality, price, regime) plus the current fundamental picture — recent quality, price, regime) plus the current fundamental picture — recent
directional "headlines", today's high-impact "events" (released: true "headlines", today's high-impact "events" (released: true means the
means the number is out), and optional aggregate "sentiment" for gold. number is out), a news "sentiment" score for gold, and an optional
Neutral and unrelated headlines were removed by code. Each remaining "news_summary" blurb.
headline has a code-computed "gold_influence" and score; trust that
direction and do not reinterpret it from the wording alone.
Judge ONE thing: does the macro backdrop support the proposed direction? Judge ONE thing: does the macro backdrop support the proposed direction?
Weigh released surprises over stale headlines; an imminent unreleased Weigh released surprises over stale headlines; an imminent unreleased
high-impact event is a reason for lower confidence, not a direction. high-impact event is a reason for lower confidence, not a direction.
@@ -154,32 +125,15 @@ and a soft jobs report -> {"stance": "support", "confidence": 75,
_TECH_SYSTEM = """\ _TECH_SYSTEM = """\
You are the technical analyst on a gold (XAU/USD) trade-review panel. You are the technical analyst on a gold (XAU/USD) trade-review panel.
You receive JSON: a proposed trade brief ("brief") plus the technical You receive JSON: a proposed trade brief ("brief") plus the technical
evidence — the setup "strategy" and optional "setup_pattern", evidence — per-"indicators" signal/confidence, multi-timeframe
per-"indicators" signal/confidence/value, multi-timeframe "mtf" signals "mtf" signals and scores, "key_levels" versus the current price, and an
and scores, "key_levels" versus the current price, an "exhaustion" "exhaustion" count of consecutive same-direction bars flagged by the
count of consecutive same-direction bars flagged by the engine, and engine.
optionally "candles".
Indicator "value" units: rsi 0-100 (above 70 overbought, below 30
oversold); adx 0-100 (above 25 = strong trend); macd = histogram in
dollars (momentum); roc = percent; bollinger = position 0-1 (0 = lower
band, 1 = upper band, values may extend outside the bands); keltner =
dollar distance from the mid-line; ema_trend = -1 to +1 alignment score;
divergence and candlestick = pattern counts. RSI is interpreted in the
supplied market regime, not as an automatic reversal at 70/30.
"candles" is per timeframe: {"ref_close", "atr", "bars"} — bars are
oldest-first, each [open, high, low, close] expressed in ATR units
relative to ref_close: 0.0 = the latest close, +1.0 = one ATR above it,
-2.0 = two ATR below it. Read the SHAPE (drift, swings, wicks); never
quote these as prices.
Judge ONE thing: is the technical picture internally coherent for the Judge ONE thing: is the technical picture internally coherent for the
proposed direction? Look for indicators fighting each other, the signal proposed direction? Look for indicators fighting each other, the signal
fighting the regime (e.g. a BUY in trending_down), higher timeframes fighting the regime (e.g. a BUY in trending_down), higher timeframes
disagreeing, price pressed into a key level, an exhausted move being disagreeing, price pressed into a key level, or an exhausted move being
chased, or the candle path contradicting the indicator readings. chased.
For XAU_CP, the named completed reversal/continuation pattern is the
setup trigger and may lead lagging indicators; indicator lag alone is
not sufficient reason to oppose it. Judge whether the pattern and the
remaining evidence are coherent together.
Stance rules: Stance rules:
- support: the indicators, higher timeframes, and regime mostly agree - support: the indicators, higher timeframes, and regime mostly agree
with the proposed direction and none of the conflicts above appear. with the proposed direction and none of the conflicts above appear.
@@ -197,22 +151,19 @@ BUY and exhaustion 0 -> {"stance": "support", "confidence": 70,
_RISK_SYSTEM = """\ _RISK_SYSTEM = """\
You are the risk manager on a gold (XAU/USD) trade-review panel. You You are the risk manager on a gold (XAU/USD) trade-review panel. You
receive JSON: a proposed trade brief ("brief") plus the risk facts — receive JSON: a proposed trade brief ("brief") plus the risk facts —
the trade "setup" (strategy, optional setup pattern, entry, stop, the trade "setup" (entry, stop_loss, take_profit_1/2, risk_reward,
advisory targets, risk, strategy cost assumption, and trailing policy), risk_dollars), the "volatility" regime, engine "warnings", the calendar
the "volatility" regime, engine "warnings", and calendar "event_risk". "event_risk" level, and the assumed "spread".
When trailing.enabled is true, TP1/TP2 are reference levels rather than
hard exits: profit can run beyond them, and the stop starts ratcheting
after arm_r using distance_r. Never evaluate that setup as a static TP1
trade. "round_trip_cost" is a strategy assumption, not a broker or data-
provider quote.
Judge ONE thing: is this setup's risk acceptable as constructed? Check Judge ONE thing: is this setup's risk acceptable as constructed? Check
stop distance versus volatility, cost as a fraction of risk, trailing reward-to-risk adequacy, stop distance versus the volatility regime
geometry, and event landmines about to hit. (a tight stop in extreme vol dies to noise; a huge stop in low vol is
oversized), spread as a fraction of the stop, and event landmines about
to hit.
Stance rules: Stance rules:
- support: the trailing opportunity justifies the risk, the stop fits - support: the likely reward clearly justifies the risk, the stop fits
the current volatility, and no event landmine is imminent. the current volatility, and no event landmine is imminent.
- oppose: a stop/trailing policy mismatched to volatility, strategy cost - oppose: reward too thin for the risk, a stop mismatched to the
eating a big share of the stop, an imminent volatility, spread eating a big share of the stop, an imminent
high-impact event — or no setup at all (a missing setup means the high-impact event — or no setup at all (a missing setup means the
signal graded too low for one; oppose unless the rest is signal graded too low for one; oppose unless the rest is
exceptional). exceptional).
@@ -220,21 +171,17 @@ Stance rules:
attractive. attractive.
Example: RR 1.4 with volatility extreme and a high-impact event due -> Example: RR 1.4 with volatility extreme and a high-impact event due ->
{"stance": "oppose", "confidence": 65, {"stance": "oppose", "confidence": 65,
"note": "The stop and trailing plan do not suit wild price swings before major news."} "note": "The possible profit is small for such wild price swings, and big economic news is due soon."}
Example: trailing enabled with normal volatility, low cost versus risk, Example: RR 2.4 with normal volatility and no events due ->
and no events due ->
{"stance": "support", "confidence": 70, {"stance": "support", "confidence": 70,
"note": "The stop and trailing plan fit calm conditions, with low costs and no major news."} "note": "The possible profit is more than double the possible loss, and conditions look calm."}
""" + _TAKE_CONTRACT """ + _TAKE_CONTRACT
_CONTRARIAN_SYSTEM = """\ _CONTRARIAN_SYSTEM = """\
You are the contrarian on a gold (XAU/USD) trade-review panel — the You are the contrarian on a gold (XAU/USD) trade-review panel — the
designated skeptic. You receive JSON: a proposed trade brief ("brief"), designated skeptic. You receive JSON: a proposed trade brief ("brief"),
the timestamped latest-closed-bar "market_snapshot", the complete trade the engine's own "bullish_reasons" and "bearish_reasons", its
"setup" and strategy context, visible primary-timeframe "indicators", "warnings", and an optional "news_summary".
filtered directional "news", the engine's bullish/bearish reasons, and
its warnings. Neutral and unrelated news was removed by code; trust each
headline's code-computed gold_influence instead of reinterpreting it.
Your job: construct the strongest honest case AGAINST the proposed Your job: construct the strongest honest case AGAINST the proposed
trade, then score how damaging that case is. Use "oppose" only when the trade, then score how damaging that case is. Use "oppose" only when the
counter-case is genuinely strong, "neutral" when it is real but modest, counter-case is genuinely strong, "neutral" when it is real but modest,
@@ -251,11 +198,9 @@ JSON: the proposed trade "direction" and the panel's "takes" — one per
analyst ("persona", "stance", "confidence", "note"). analyst ("persona", "stance", "confidence", "note").
Weigh the takes into ONE consensus on whether to act on the signal: Weigh the takes into ONE consensus on whether to act on the signal:
"AGREE" (the panel backs the trade), "OPPOSE" (the panel is against it), "AGREE" (the panel backs the trade), "OPPOSE" (the panel is against it),
or "NEUTRAL" (genuinely split or low conviction). The risk manager and or "NEUTRAL" (genuinely split or low conviction). Weigh a high-
contrarian have intentionally defensive roles; their objections are not confidence oppose from any single analyst seriously — one strong veto
automatic vetoes. Do not count repeated versions of the same warning as can outweigh two lukewarm supports. Do not average blindly.
independent evidence. Weigh role relevance, evidence quality, and
confidence rather than averaging blindly.
Base the verdict and rationale ONLY on the takes given — do not add Base the verdict and rationale ONLY on the takes given — do not add
outside analysis. outside analysis.
Confidence scale (0-100): 80-100 the panel clearly leans one way; 50-79 Confidence scale (0-100): 80-100 the panel clearly leans one way; 50-79
@@ -268,7 +213,7 @@ possible profit is too small and the price has already climbed
unusually fast. Only the news backdrop supports buying." unusually fast. Only the news backdrop supports buying."
Reply with ONLY JSON: Reply with ONLY JSON:
{"verdict": "AGREE"|"NEUTRAL"|"OPPOSE", "confidence": <0-100>, {"verdict": "AGREE"|"NEUTRAL"|"OPPOSE", "confidence": <0-100>,
"rationale": "<max 30 words>"} "rationale": "<max 40 words>"}
""" """
@@ -289,21 +234,10 @@ class PanelVerdict:
rationale: str rationale: str
takes: List[PersonaTake] = field(default_factory=list) takes: List[PersonaTake] = field(default_factory=list)
created_ts: float = 0.0 created_ts: float = 0.0
payload_hash: str = ""
# ── Payload builders (pure functions; llm_client json-dumps sorted) ── # ── Payload builders (pure functions; llm_client json-dumps sorted) ──
def _finite(x: Any) -> Optional[float]:
"""float(x) or None for non-numeric / non-finite (NaN/inf breaks the
strict-JSON wire format the client sends)."""
try:
f = float(x)
except (TypeError, ValueError):
return None
return f if math.isfinite(f) else None
def _direction(report: "SignalReport") -> Optional[str]: def _direction(report: "SignalReport") -> Optional[str]:
"""BUY/SELL the panel should review; None = nothing to review.""" """BUY/SELL the panel should review; None = nothing to review."""
if report.trade_setup is not None: if report.trade_setup is not None:
@@ -331,190 +265,50 @@ def _brief(report: "SignalReport") -> Dict[str, Any]:
} }
def _primary_indicator_payload(report: "SignalReport") -> Dict[str, Dict[str, Any]]:
indicators: Dict[str, Dict[str, Any]] = {}
analysis = report.primary_analysis
if analysis is None:
return indicators
for name, ind in analysis.indicators.items():
if name.startswith("_") or ind.weight == 0:
continue
entry: Dict[str, Any] = {
"signal": ind.signal.name,
"confidence": round(ind.confidence),
}
value = _finite(getattr(ind, "value", None))
if value is not None:
entry["value"] = round(value, 2)
indicators[name] = entry
return indicators
def _strategy_context(report: "SignalReport") -> Dict[str, Any]:
setup = report.trade_setup
code = getattr(setup, "strategy", None) if setup is not None else None
meanings = {
"XAU_ATR": "gold ATR setup produced by the scored technical signal",
"XAU_PB": "trend-continuation pullback to EMA21 or Keltner mid-line",
"XAU_CP": "completed classical chart-pattern entry",
"GENERIC": "swing and ATR setup produced by the scored technical signal",
}
out: Dict[str, Any] = {
"code": code or "NO_SETUP",
"meaning": meanings.get(code, "no executable trade setup was produced"),
}
pattern = getattr(setup, "setup_pattern", None) if setup is not None else None
if pattern:
out["setup_pattern"] = str(pattern)
return out
def _trade_setup_payload(report: "SignalReport") -> Optional[Dict[str, Any]]:
setup = report.trade_setup
if setup is None:
return None
entry = _finite(getattr(setup, "entry_price", None))
stop = _finite(getattr(setup, "stop_loss", None))
risk = abs(entry - stop) if entry is not None and stop is not None else None
cost = _finite(getattr(setup, "round_trip_cost", None))
arm = _finite(getattr(setup, "trail_arm_r", None))
distance = _finite(getattr(setup, "trail_dist_r", None))
out: Dict[str, Any] = {
"direction": getattr(setup, "direction", None),
"strategy": _strategy_context(report),
"entry": round(entry, 2) if entry is not None else None,
"stop_loss": round(stop, 2) if stop is not None else None,
"take_profit_1": round(float(setup.take_profit_1), 2),
"take_profit_2": round(float(setup.take_profit_2), 2),
"risk_reward_to_advisory_tp1": round(float(setup.risk_reward), 2),
"risk_price_distance": round(risk, 2) if risk is not None else None,
"round_trip_cost": round(cost, 2) if cost is not None else None,
"trailing": {
"enabled": bool(arm is not None and arm > 0),
"arm_r": round(arm, 2) if arm is not None else None,
"distance_r": round(distance, 2) if distance is not None else None,
"tp1_tp2_are_advisory": bool(arm is not None and arm > 0),
},
}
analysis = report.primary_analysis
atr = _finite((analysis.key_levels or {}).get("atr")) \
if analysis is not None else None
if atr is not None and atr > 0 and risk is not None:
out["atr"] = round(atr, 2)
out["stop_distance_atr"] = round(risk / atr, 2)
if risk and cost is not None:
out["cost_to_risk_pct"] = round(100.0 * cost / risk, 2)
return out
def _market_snapshot(report: "SignalReport") -> Dict[str, Any]:
return {
"price": round(report.current_price, 2),
"bar_timestamp": report.timestamp,
"basis": "latest_closed_candle",
}
def _headline_gold_score(headline: Any) -> Optional[float]:
"""Deterministic XAU/USD influence from an already-scored headline."""
by_currency = getattr(headline, "sentiment_by_currency", None) or {}
xau = _finite(by_currency.get("XAU"))
usd = _finite(by_currency.get("USD"))
if xau is not None or usd is not None:
score = (xau or 0.0) - (usd or 0.0)
else:
mentioned = set(getattr(headline, "currencies_mentioned", None) or [])
raw = _finite(getattr(headline, "sentiment_score", None))
if raw is None:
return None
if "XAU" in mentioned and "USD" in mentioned:
return None # pair direction is ambiguous without per-currency scores
if "XAU" in mentioned:
score = raw
elif "USD" in mentioned:
score = -raw
else:
return None
return max(-1.0, min(1.0, score)) if math.isfinite(score) else None
def _directional_news_payload(
context: Optional["FundamentalContext"], limit: int = 10
) -> List[Dict[str, Any]]:
if context is None:
return []
out: List[Dict[str, Any]] = []
for headline in context.news_headlines:
score = _headline_gold_score(headline)
if score is None or abs(score) < 0.1:
continue
timestamp = getattr(headline, "timestamp", None)
out.append({
"headline": str(getattr(headline, "headline", "")),
"published_at": (timestamp.isoformat()
if hasattr(timestamp, "isoformat") else str(timestamp or "")),
"gold_influence": "bullish" if score > 0 else "bearish",
"score": round(score, 3),
})
if len(out) >= limit:
break
return out
def _aggregate_sentiment_payload(report: "SignalReport") -> Optional[Dict[str, Any]]:
score = _finite(report.news_sentiment)
if score is None or abs(score) < 0.1:
return None
return {
"gold_influence": "bullish" if score > 0 else "bearish",
"score": round(score, 3),
}
def _macro_payload(report: "SignalReport", def _macro_payload(report: "SignalReport",
context: Optional["FundamentalContext"]) -> Dict[str, Any]: context: Optional["FundamentalContext"]) -> Dict[str, Any]:
headlines: List[str] = []
events: List[Dict[str, Any]] = [] events: List[Dict[str, Any]] = []
if context is not None: if context is not None:
headlines = [h.headline for h in context.news_headlines[:10]]
events = [ events = [
{"event": e.event, "currency": e.currency, {"event": e.event, "currency": e.currency,
"time": e.time.isoformat(), "actual": e.actual, "actual": e.actual, "forecast": e.forecast,
"forecast": e.forecast, "previous": e.previous,
"surprise": e.surprise,
"minutes_until": round(e.minutes_until, 1),
"released": not e.is_upcoming} "released": not e.is_upcoming}
for e in context.events_today if e.is_high_impact for e in context.events_today if e.is_high_impact
][:8] ][:8]
payload: Dict[str, Any] = { return {
"brief": _brief(report), "brief": _brief(report),
"headlines": _directional_news_payload(context), "headlines": headlines,
"events": events, "events": events,
"sentiment": round(report.news_sentiment, 3),
"news_summary": report.news_summary,
} }
sentiment = _aggregate_sentiment_payload(report)
if sentiment is not None:
payload["sentiment"] = sentiment
return payload
def _technical_payload(report: "SignalReport", def _technical_payload(report: "SignalReport") -> Dict[str, Any]:
candles: Optional[Dict[str, Any]] = None indicators: Dict[str, Dict[str, Any]] = {}
) -> Dict[str, Any]:
key_levels: Dict[str, float] = {} key_levels: Dict[str, float] = {}
analysis = report.primary_analysis analysis = report.primary_analysis
if analysis is not None: if analysis is not None:
# Same visibility filter as main.report_to_dict: private series
# ride as _-prefixed pseudo-indicators, weight-0 = context-only.
indicators = {
name: {"signal": ind.signal.name,
"confidence": round(ind.confidence)}
for name, ind in analysis.indicators.items()
if not name.startswith("_") and ind.weight != 0
}
key_levels = {k: round(v, 2) for k, v in analysis.key_levels.items()} key_levels = {k: round(v, 2) for k, v in analysis.key_levels.items()}
payload: Dict[str, Any] = { return {
"brief": _brief(report), "brief": _brief(report),
"strategy": _strategy_context(report), "indicators": indicators,
"indicators": _primary_indicator_payload(report),
"mtf": {tf: {"signal": sig.name, "mtf": {tf: {"signal": sig.name,
"score": round(report.mtf_scores.get(tf, 0.0), 1)} "score": round(report.mtf_scores.get(tf, 0.0), 1)}
for tf, sig in report.mtf_signals.items()}, for tf, sig in report.mtf_signals.items()},
"key_levels": key_levels, "key_levels": key_levels,
"exhaustion": report.exhaustion_count, "exhaustion": report.exhaustion_count,
} }
if candles:
payload["candles"] = candles
return payload
def _risk_payload(report: "SignalReport") -> Dict[str, Any]: def _risk_payload(report: "SignalReport") -> Dict[str, Any]:
@@ -524,103 +318,32 @@ def _risk_payload(report: "SignalReport") -> Dict[str, Any]:
"warnings": list(report.warnings), "warnings": list(report.warnings),
"event_risk": report.event_risk, "event_risk": report.event_risk,
"event_description": report.event_description, "event_description": report.event_description,
"spread": round(report.spread_estimate, 2),
}
ts = report.trade_setup
if ts is not None:
payload["setup"] = {
"direction": ts.direction,
"entry": round(ts.entry_price, 2),
"stop_loss": round(ts.stop_loss, 2),
"take_profit_1": round(ts.take_profit_1, 2),
"take_profit_2": round(ts.take_profit_2, 2),
"risk_reward": round(ts.risk_reward, 2),
"risk_dollars": round(ts.risk_pips * ts.pip_value, 2),
} }
setup = _trade_setup_payload(report)
if setup is not None:
payload["setup"] = setup
return payload return payload
def _contrarian_payload( def _contrarian_payload(report: "SignalReport") -> Dict[str, Any]:
report: "SignalReport", context: Optional["FundamentalContext"] = None
) -> Dict[str, Any]:
return { return {
"brief": _brief(report), "brief": _brief(report),
"market_snapshot": _market_snapshot(report),
"setup": _trade_setup_payload(report),
"indicators": _primary_indicator_payload(report),
"news": _directional_news_payload(context),
"bullish_reasons": report.bullish_reasons[:3], "bullish_reasons": report.bullish_reasons[:3],
"bearish_reasons": report.bearish_reasons[:3], "bearish_reasons": report.bearish_reasons[:3],
"warnings": list(report.warnings), "warnings": list(report.warnings),
"news_summary": report.news_summary,
} }
# ── Candle context (ATR-normalized; technical persona only) ─────────
#
# Raw prices are the LLM's weakest input modality (measured in-repo:
# numeric-comparison misreads, backtest/research/llm_news_eval_2026-08-06.md),
# so code does the arithmetic and the model reads a normalized SHAPE:
# each bar's [o,h,l,c] as offsets from the latest close in ATR units.
# A panel-private provider singleton keeps the TTL cache warm across
# attaches; any failure degrades to "no candles", never a failed panel.
_TF_BY_VALUE = {tf.value: tf for tf in Timeframe}
# Patchable for tests (mirrors the llm_client._client stub pattern).
_dp: Any = None
def _panel_provider() -> Any:
global _dp
if _dp is None:
from config import PlatformConfig
from providers import make_data_provider
_dp = make_data_provider(PlatformConfig())
return _dp
def _normalize_candles(df: Any, atr: float, bars: int) -> Dict[str, Any]:
"""Last `bars` closed candles as ATR-unit offsets from the final
close: 0.0 = latest close, +1.0 = one ATR above it. Oldest-first."""
tail = df.tail(bars)
ref = float(tail["Close"].iloc[-1])
out = []
for _, row in tail.iterrows():
out.append([round((float(row["Open"]) - ref) / atr, 1),
round((float(row["High"]) - ref) / atr, 1),
round((float(row["Low"]) - ref) / atr, 1),
round((float(row["Close"]) - ref) / atr, 1)])
return {"ref_close": round(ref, 2), "atr": round(float(atr), 2),
"bars": out}
def _candle_context(report: "SignalReport",
settings: Dict[str, Any]) -> Optional[Dict[str, Any]]:
"""Best-effort {"<tf>": _normalize_candles(...)} per analysis TF;
None = omit the key from the payload. NEVER raises."""
try:
if not settings.get("candles_enabled", True):
return None
confirmations = list(report.confirmation_analyses or [])
analyses = [report.primary_analysis] + confirmations
counts = [int(settings.get("candle_bars_primary",
DEFAULT_CANDLE_BARS_PRIMARY))] + [
int(settings.get("candle_bars_confirm",
DEFAULT_CANDLE_BARS_CONFIRM))
] * len(confirmations)
dp = _panel_provider()
out: Dict[str, Any] = {}
for analysis, n in zip(analyses, counts):
if analysis is None:
continue
atr = _finite(getattr(
(analysis.indicators or {}).get("atr"), "value", None))
if not atr or atr <= 0:
continue
tf = _TF_BY_VALUE.get(analysis.timeframe)
if tf is None:
continue
df = dp.fetch_ohlcv(report.symbol, tf)
if df is None or len(df) < 2:
continue
out[analysis.timeframe] = _normalize_candles(df, atr, n)
return out or None
except Exception as e:
logger.debug("agent panel: candle context unavailable: %s", e)
return None
# ── Orchestration ──────────────────────────────────────────────────── # ── Orchestration ────────────────────────────────────────────────────
# Panel-private pool: the 4 persona calls run concurrently WITHOUT # Panel-private pool: the 4 persona calls run concurrently WITHOUT
@@ -660,52 +383,8 @@ def _ask_persona(persona: str, system: str,
return _validate_take(persona, reply) return _validate_take(persona, reply)
def _panel_jobs(
report: "SignalReport",
context: Optional["FundamentalContext"],
candles: Optional[Dict[str, Any]],
) -> Tuple[Tuple[str, str, Dict[str, Any]], ...]:
return (
("macro", _MACRO_SYSTEM, _macro_payload(report, context)),
("technical", _TECH_SYSTEM, _technical_payload(report, candles)),
("risk", _RISK_SYSTEM, _risk_payload(report)),
("contrarian", _CONTRARIAN_SYSTEM,
_contrarian_payload(report, context)),
)
def _panel_payload_hash(
jobs: Tuple[Tuple[str, str, Dict[str, Any]], ...],
ignore_bar_timestamp: bool = False,
) -> str:
client = get_llm_client()
personas = []
for persona, system, payload in jobs:
payload_copy = json.loads(json.dumps(payload, allow_nan=False))
if ignore_bar_timestamp and persona == "contrarian":
snapshot = payload_copy.get("market_snapshot")
if isinstance(snapshot, dict):
snapshot.pop("bar_timestamp", None)
personas.append({
"persona": persona, "system": system, "payload": payload_copy,
})
envelope = {
"contract": "agent-panel-input-v2",
"model": getattr(client, "_model", client.__class__.__name__),
"personas": personas,
"synthesizer_system": _SYNTH_SYSTEM,
}
raw = json.dumps(
envelope, sort_keys=True, separators=(",", ":"),
ensure_ascii=True, allow_nan=False,
).encode("utf-8")
return "sha256:" + hashlib.sha256(raw).hexdigest()
def run_panel(report: "SignalReport", def run_panel(report: "SignalReport",
context: Optional["FundamentalContext"] = None, context: Optional["FundamentalContext"] = None,
candles: Optional[Dict[str, Any]] = None,
jobs: Optional[Tuple[Tuple[str, str, Dict[str, Any]], ...]] = None,
) -> Optional[PanelVerdict]: ) -> Optional[PanelVerdict]:
"""Run 4 personas + synthesizer; None = no verdict (fail-safe). """Run 4 personas + synthesizer; None = no verdict (fail-safe).
@@ -717,8 +396,12 @@ def run_panel(report: "SignalReport",
direction = _direction(report) direction = _direction(report)
if direction is None: if direction is None:
return None return None
jobs = jobs or _panel_jobs(report, context, candles) jobs = (
payload_hash = _panel_payload_hash(jobs) ("macro", _MACRO_SYSTEM, _macro_payload(report, context)),
("technical", _TECH_SYSTEM, _technical_payload(report)),
("risk", _RISK_SYSTEM, _risk_payload(report)),
("contrarian", _CONTRARIAN_SYSTEM, _contrarian_payload(report)),
)
futures = [_pool.submit(_ask_persona, name, system, payload) futures = [_pool.submit(_ask_persona, name, system, payload)
for name, system, payload in jobs] for name, system, payload in jobs]
# The HTTP timeout inside the client bounds each call; the margin # The HTTP timeout inside the client bounds each call; the margin
@@ -749,7 +432,7 @@ def run_panel(report: "SignalReport",
rationale = str(reply.get("rationale", "")).strip()[:240] rationale = str(reply.get("rationale", "")).strip()[:240]
return PanelVerdict(verdict=verdict, confidence=confidence, return PanelVerdict(verdict=verdict, confidence=confidence,
rationale=rationale, takes=takes, rationale=rationale, takes=takes,
created_ts=time.time(), payload_hash=payload_hash) created_ts=time.time())
# ── Cache, budget, attach ──────────────────────────────────────────── # ── Cache, budget, attach ────────────────────────────────────────────
@@ -757,14 +440,13 @@ def run_panel(report: "SignalReport",
_state_lock = threading.Lock() _state_lock = threading.Lock()
# Exact result per reviewed bar: repeated attaches on the same signal # Exact result per reviewed bar: repeated attaches on the same signal
# (re-scan, multi-consumer) are free. # (re-scan, multi-consumer) are free.
_cache: "OrderedDict[Tuple[str, str, str, str, str], PanelVerdict]" = OrderedDict() _cache: "OrderedDict[Tuple[str, str, str, str], PanelVerdict]" = OrderedDict()
_CACHE_MAX = 32 _CACHE_MAX = 32
# Reuse only when the full prompt + evidence payload is byte-equivalent. # Reuse layer: a persisting signal (same symbol/tf/direction, newer bar)
# Direction-only reuse served stale warnings/setups to materially changed # re-serves its last verdict for reuse_ttl_seconds instead of re-spending
# signals; the daily panel budget remains the hard spend bound. # 5 requests per bar (see module docstring for the M5 dashboard math).
_last_by_payload: Dict[str, PanelVerdict] = {} _last_by_dir: Dict[Tuple[str, str, str], PanelVerdict] = {}
_inflight: Dict[str, threading.Event] = {} # UTC-day budget of panel attempts (mirrors llm_client._under_cap).
# Per-process UTC-day budget of panel attempts (mirrors llm_client._under_cap).
_panel_day: Optional[_dt.date] = None _panel_day: Optional[_dt.date] = None
_panel_count: int = 0 _panel_count: int = 0
_warned_budget = False _warned_budget = False
@@ -775,10 +457,7 @@ def reset_panel_state() -> None:
global _panel_day, _panel_count, _warned_budget global _panel_day, _panel_count, _warned_budget
with _state_lock: with _state_lock:
_cache.clear() _cache.clear()
_last_by_payload.clear() _last_by_dir.clear()
for event in _inflight.values():
event.set()
_inflight.clear()
_panel_day = None _panel_day = None
_panel_count = 0 _panel_count = 0
_warned_budget = False _warned_budget = False
@@ -814,33 +493,13 @@ def _under_panel_budget(cap: int) -> bool:
return True return True
def _write_fields( def _write_fields(report: "SignalReport", v: PanelVerdict) -> None:
report: "SignalReport", v: PanelVerdict, cache_status: str
) -> None:
report.panel_verdict = v.verdict report.panel_verdict = v.verdict
report.panel_confidence = float(v.confidence) report.panel_confidence = float(v.confidence)
report.panel_rationale = v.rationale report.panel_rationale = v.rationale
report.panel_takes = [ report.panel_takes = [
{"persona": t.persona, "stance": t.stance, {"persona": t.persona, "stance": t.stance,
"confidence": t.confidence, "note": t.note} for t in v.takes] "confidence": t.confidence, "note": t.note} for t in v.takes]
report.panel_created_at = _dt.datetime.fromtimestamp(
v.created_ts, tz=_dt.timezone.utc).isoformat()
report.panel_cache_status = cache_status
report.panel_payload_hash = v.payload_hash
def _log_panel_verdict(report: "SignalReport") -> None:
"""Persist the verdict onto the scan's signals.db row (fail-safe).
The row is inserted inside analyze_pair — BEFORE the panel attaches
post-hoc — so panel columns arrive via this update. A report that was
never logged (backtest replay, logging disabled) is a silent no-op.
"""
try:
from signal_log import log_panel_update
log_panel_update(report)
except Exception:
logger.debug("agent panel: verdict logging failed", exc_info=True)
def attach_panel(report: "SignalReport", def attach_panel(report: "SignalReport",
@@ -875,6 +534,27 @@ def attach_panel(report: "SignalReport",
if direction is None: if direction is None:
return False return False
analysis = report.primary_analysis
tf = analysis.timeframe if analysis else ""
exact_key = (report.symbol, tf, report.timestamp, direction)
dir_key = (report.symbol, tf, direction)
ttl = float(settings.get("reuse_ttl_seconds",
DEFAULT_REUSE_TTL_SECONDS))
with _state_lock:
hit = _cache.get(exact_key)
if hit is None:
prior = _last_by_dir.get(dir_key)
if prior is not None and time.time() - prior.created_ts < ttl:
hit = prior
if hit is not None:
_write_fields(report, hit)
return True
cap = int(settings.get("max_panels_per_day",
DEFAULT_MAX_PANELS_PER_DAY))
if not _under_panel_budget(cap):
return False
if context is None: if context is None:
# Best-effort; the news engine's class-level 600 s caches make # Best-effort; the news engine's class-level 600 s caches make
# this nearly free when warm in both bot and dashboard. # this nearly free when warm in both bot and dashboard.
@@ -887,83 +567,16 @@ def attach_panel(report: "SignalReport",
logger.debug("agent panel: news context unavailable: %s", e) logger.debug("agent panel: news context unavailable: %s", e)
context = None context = None
candles = _candle_context(report, settings) verdict = run_panel(report, context)
jobs = _panel_jobs(report, context, candles)
payload_hash = _panel_payload_hash(jobs)
reuse_hash = _panel_payload_hash(jobs, ignore_bar_timestamp=True)
analysis = report.primary_analysis
tf = analysis.timeframe if analysis else ""
exact_key = (report.symbol, tf, report.timestamp, direction,
payload_hash)
ttl = float(settings.get("reuse_ttl_seconds",
DEFAULT_REUSE_TTL_SECONDS))
inflight_key = reuse_hash if ttl > 0 else payload_hash
hit = None
cache_status = ""
leader = False
event: Optional[threading.Event] = None
with _state_lock:
hit = _cache.get(exact_key)
if hit is not None:
cache_status = "EXACT_HIT"
elif ttl > 0:
prior = _last_by_payload.get(reuse_hash)
if prior is not None and time.time() - prior.created_ts < ttl:
hit = prior
cache_status = "TTL_REUSE"
if hit is None:
event = _inflight.get(inflight_key)
if event is None:
event = threading.Event()
_inflight[inflight_key] = event
leader = True
if hit is not None:
_write_fields(report, hit, cache_status)
_log_panel_verdict(report)
return True
if not leader:
# A successful leader can spend one client timeout on personas
# and another on synthesis; followers must cover both phases.
client_timeout = getattr(
get_llm_client(), "_timeout", DEFAULT_TIMEOUT)
timeout = 2.0 * client_timeout + 12.0
if event is None or not event.wait(timeout=timeout):
return False
with _state_lock:
hit = _cache.get(exact_key)
cache_status = "EXACT_HIT"
if hit is None and ttl > 0:
hit = _last_by_payload.get(reuse_hash)
cache_status = "TTL_REUSE"
if hit is None:
return False
_write_fields(report, hit, cache_status)
_log_panel_verdict(report)
return True
try:
cap = int(settings.get("max_panels_per_day",
DEFAULT_MAX_PANELS_PER_DAY))
if not _under_panel_budget(cap):
return False
verdict = run_panel(report, context, candles=candles, jobs=jobs)
if verdict is None: if verdict is None:
return False return False
_write_fields(report, verdict, "MISS") _write_fields(report, verdict)
with _state_lock: with _state_lock:
_cache[exact_key] = verdict _cache[exact_key] = verdict
while len(_cache) > _CACHE_MAX: while len(_cache) > _CACHE_MAX:
_cache.popitem(last=False) _cache.popitem(last=False)
_last_by_payload[reuse_hash] = verdict _last_by_dir[dir_key] = verdict
_log_panel_verdict(report)
return True return True
finally:
with _state_lock:
current = _inflight.pop(inflight_key, None)
if current is not None:
current.set()
except Exception: except Exception:
logger.exception("agent panel: attach failed — signal unaffected") logger.exception("agent panel: attach failed — signal unaffected")
return False return False
+1 -88
View File
@@ -86,8 +86,7 @@ _VALID_LOG_LEVELS = ("DEBUG", "INFO", "WARNING", "ERROR", "CRITICAL")
_SCANNABLE_TFS = ("M5", "M15", "H1", "H4") _SCANNABLE_TFS = ("M5", "M15", "H1", "H4")
_KNOWN_TOP = {"telegram", "data_providers", "scanning", "pairs", "log_level", _KNOWN_TOP = {"telegram", "data_providers", "scanning", "pairs", "log_level",
"oanda", "deepseek", "agent_panel", "market_outlook", "oanda", "deepseek", "agent_panel"}
"signal_log"}
_KNOWN_OANDA = { _KNOWN_OANDA = {
"account_id", "api_token", "base_url", "lot_size", "account_id", "api_token", "base_url", "lot_size",
"quality_gate", "max_hold_bars", "quality_gate", "max_hold_bars",
@@ -98,21 +97,11 @@ _KNOWN_DEEPSEEK = {
_KNOWN_AGENT_PANEL = { _KNOWN_AGENT_PANEL = {
"enabled", "telegram", "dashboard", "min_quality", "enabled", "telegram", "dashboard", "min_quality",
"max_panels_per_day", "reuse_ttl_seconds", "max_panels_per_day", "reuse_ttl_seconds",
"candles_enabled", "candle_bars_primary", "candle_bars_confirm",
} }
_KNOWN_MARKET_OUTLOOK = {"enabled", "telegram", "max_per_day", "candles_bars"}
_KNOWN_SIGNAL_LOG = {"enabled", "path"}
_KNOWN_TELEGRAM = {"token", "chat_id", "channel_id", "alert_min_quality"} _KNOWN_TELEGRAM = {"token", "chat_id", "channel_id", "alert_min_quality"}
_KNOWN_SCANNING = {"pairs", "timeframes"} _KNOWN_SCANNING = {"pairs", "timeframes"}
_KNOWN_PAIR_KEYS = {"blocked_hours_utc"} _KNOWN_PAIR_KEYS = {"blocked_hours_utc"}
_MARKET_OUTLOOK_DEFAULTS = {
"enabled": False,
"telegram": False,
"max_per_day": 12,
"candles_bars": 20,
}
class AppConfigError(Exception): class AppConfigError(Exception):
"""config.json is present but invalid — fail fast at boot.""" """config.json is present but invalid — fail fast at boot."""
@@ -134,9 +123,6 @@ class AppConfig:
oanda_settings: Dict[str, Any] = field(default_factory=dict) oanda_settings: Dict[str, Any] = field(default_factory=dict)
deepseek_settings: Dict[str, Any] = field(default_factory=dict) deepseek_settings: Dict[str, Any] = field(default_factory=dict)
agent_panel_settings: Dict[str, Any] = field(default_factory=dict) agent_panel_settings: Dict[str, Any] = field(default_factory=dict)
market_outlook_settings: Dict[str, Any] = field(
default_factory=lambda: dict(_MARKET_OUTLOOK_DEFAULTS))
signal_log_settings: Dict[str, Any] = field(default_factory=dict)
log_level: str = "INFO" log_level: str = "INFO"
@@ -318,45 +304,8 @@ def load_app_config(path: str = CONFIG_PATH) -> AppConfig:
raise AppConfigError( raise AppConfigError(
"config.json: agent_panel.reuse_ttl_seconds must be a " "config.json: agent_panel.reuse_ttl_seconds must be a "
"non-negative number") "non-negative number")
if "candles_enabled" in panel \
and not isinstance(panel["candles_enabled"], bool):
raise AppConfigError(
"config.json: agent_panel.candles_enabled must be true or false")
for key in ("candle_bars_primary", "candle_bars_confirm"):
if key in panel:
n = panel[key]
if isinstance(n, bool) or not isinstance(n, int) \
or not 1 <= n <= 100:
raise AppConfigError(
f"config.json: agent_panel.{key} must be an integer "
"between 1 and 100")
app.agent_panel_settings = dict(panel) app.agent_panel_settings = dict(panel)
outlook = raw.get("market_outlook", {})
_check_keys(outlook, _KNOWN_MARKET_OUTLOOK, '"market_outlook"')
for flag in ("enabled", "telegram"):
if flag in outlook and not isinstance(outlook[flag], bool):
raise AppConfigError(
f"config.json: market_outlook.{flag} must be true or false")
for key in ("max_per_day", "candles_bars"):
if key in outlook:
value = outlook[key]
if isinstance(value, bool) or not isinstance(value, int) or value < 1:
raise AppConfigError(
f"config.json: market_outlook.{key} must be a positive integer")
app.market_outlook_settings.update(outlook)
slog = raw.get("signal_log", {})
_check_keys(slog, _KNOWN_SIGNAL_LOG, '"signal_log"')
if "enabled" in slog and not isinstance(slog["enabled"], bool):
raise AppConfigError(
"config.json: signal_log.enabled must be true or false")
if "path" in slog and (not isinstance(slog["path"], str)
or not slog["path"]):
raise AppConfigError(
"config.json: signal_log.path must be a non-empty string")
app.signal_log_settings = dict(slog)
level = raw.get("log_level", app.log_level) level = raw.get("log_level", app.log_level)
if level not in _VALID_LOG_LEVELS: if level not in _VALID_LOG_LEVELS:
raise AppConfigError( raise AppConfigError(
@@ -456,42 +405,6 @@ def deepseek_api_key(app: Optional[AppConfig] = None) -> str:
app.deepseek_settings.get("api_key", "")) app.deepseek_settings.get("api_key", ""))
# ── AI market-outlook resolution helper ──────────────────────────────
def market_outlook_enabled(app: Optional[AppConfig] = None) -> bool:
"""Whether the advisory outlook feature is enabled (env wins)."""
env = os.environ.get("MARKET_OUTLOOK_ENABLED")
if env:
return env.strip().lower() in ("1", "true", "yes", "on")
app = app or get_app_config()
return bool(app.market_outlook_settings.get("enabled", False))
# ── Signal-log resolution helpers (env → config.json → default) ──────
def signal_log_enabled(app: Optional[AppConfig] = None) -> bool:
"""Default ON — signal history is the platform's audit trail."""
env = os.environ.get("SIGNAL_LOG_ENABLED")
if env:
return env.strip().lower() in ("1", "true", "yes", "on")
app = app or get_app_config()
return bool(app.signal_log_settings.get("enabled", True))
def signal_log_path(app: Optional[AppConfig] = None) -> str:
"""Absolute path to the SQLite signal-history DB. A relative config
value resolves against the repo root so CWD never matters."""
env = os.environ.get("SIGNAL_LOG_PATH")
if env:
path = env
else:
app = app or get_app_config()
path = str(app.signal_log_settings.get("path", "logs/signals.db"))
if not os.path.isabs(path):
path = os.path.join(os.path.dirname(os.path.abspath(__file__)), path)
return path
# ── PlatformConfig overlay ──────────────────────────────────────────── # ── PlatformConfig overlay ────────────────────────────────────────────
def apply_pair_overrides( def apply_pair_overrides(
@@ -1,116 +0,0 @@
# AI signal enrichment — analysis + build record (2026-08-15)
Owner questions:
1. If the AI panel also received indicator values and (limited) candle data, would the system be more accurate?
2. If AI reviewed every candle alongside the local score system, would prediction accuracy justify the token burn — or is there a middle ground?
## What the panel already received (pre-change baseline)
The four personas (`agent_panel.py`) already got a *distilled* picture:
per-indicator signal+confidence, MTF signals/scores, key levels,
exhaustion count, full trade-setup math, warnings, headlines/events/
sentiment. Missing: raw indicator **values** and any **candle** data.
Cost guardrails already in place: 5 requests/panel, `max_panels_per_day`
20 (≤100 requests/day), 3600 s reuse-TTL, per-bar exact cache, shared
500 req/day DeepSeek cap, static prompts on the ~50× prefix cache.
## Evidence used
- **LLM is strong at text, weak at raw numerics** (measured in-repo,
`llm_news_eval_2026-08-06.md`): headline sentiment 100% (78/78) vs
keyword scorer 31%; but ~11% of event-direction calls misread close
numeric comparisons until code pre-computed the comparison
(`vs_forecast`) and the model only mapped it to doctrine.
- **The book any filter would act on** (M5 canonical,
`oanda_m5_12mo_065_nopb_20260814.csv`): 577 trades, +106.82R,
PF 1.51, win rate 52.9%; A-grade 59.5%, B-grade 51.4% (+67.8R of the
total), C-grade 43.8% (−0.23R, 16 trades). Average edge ≈ 0.185R/trade.
## Answer 1 — enrich the existing panel? YES (cheap, safe)
- A compact candle is ~15–25 tokens; 20 primary-TF + 10 confirm-TF bars
≈ 500–1,500 tokens per persona call → cents/day at the 20-panel cap.
**Tokens are not the constraint; the request cap is.**
- Measured after the build (dry-test wire log): enriched technical
payload with 20 candles ≈ **~380 tokens**; replies ≈ 25–35 tokens
under the new 15-word note cap.
- Caveats: (a) the panel is advisory-only — richer payloads improve the
human's context, not the book's P&L; (b) raw prices are the model's
weakest input, so candles are sent **ATR-normalized** (code does the
arithmetic, LLM reads the shape) — the same pattern that fixed the
event-direction misreads.
## Answer 2 — AI on every candle? NO
- **Feasibility**: M5+M15+H1+H4 ≈ 414 bars/day → one call/bar nearly
exhausts the shared 500/day cap (news needs it too); a panel per bar
≈ 2,070/day = 4× over. Most of that spend reviews HOLD bars.
- **Parity (rule 4)**: an AI gate cannot be replayed by the backtester
(news off; no recorded verdicts) — every pinned book becomes
non-comparable, and model-version drift silently mutates the strategy.
- **Edge**: the engine already compresses 380 bars into 9 tuned
indicators + regime + MTF validated over 5 years; an LLM's generic
prior on raw candles won't beat fitted thresholds, and the margin it
would filter is ~0.185R/trade. B-grade holds +67.8R at 51.4% win —
a blunt veto is net-negative.
## The middle ground (adopted)
1. **Enrich payloads** (built today, see below) — near-zero cost, no
rule-1 exposure.
2. **Log panel verdicts to signals.db** (built today) — creates the
"does OPPOSE predict losers?" dataset with zero behavior change.
3. **After 4–8 weeks**: join logged verdicts to outcomes; simulate an
OPPOSE-veto against the pinned book offline. Only then consider any
gating, as a full rule-1 revalidation.
## Build record (2026-08-15, owner-approved)
- `agent_panel.py`
- Technical persona payload now carries indicator **values** (units
legend added to the prompt) and **ATR-normalized candles** per TF:
`{"ref_close", "atr", "bars": [[o,h,l,c]…]}` oldest-first, each price
as `(price − ref_close) / atr`, 1 decimal. Window: 20 bars primary /
10 per confirmation TF.
- New config keys (validated in `app_config.py`, in
`config.json.example`): `candles_enabled` (default true),
`candle_bars_primary` (20), `candle_bars_confirm` (10).
- Panel-private provider singleton (`_dp`, patchable in tests);
TTL cache keeps fetches near-free; any failure → candles silently
omitted, panel unaffected.
- Prompt hardening: training data declared stale and OFF-LIMITS;
"if a fact is not in the JSON, treat it as unknown"; thin evidence
→ neutral < 30 (unchanged); note cap 25→**15 words**, rationale
40→**30 words** (output-token savings).
- `attach_panel` now calls `signal_log.log_panel_update(report)` after
every successful attach (fresh, exact-cache, and reuse-TTL paths).
- `signal_log.py`
- `log()` returns the rowid; `log_signal` stashes it on the report as
`_signal_log_rowid`; new `log_panel_update()` fills the 4 panel
columns on that row post-hoc. No rowid (backtest replay, logging
disabled, other process) → silent no-op. Fail-safe contract kept.
- Tests: 381 → **388 passed** (7 new: candle normalization math, payload
values/candles, context windows + ATR gating + disable flag, attach
injection (technical persona only), prompt-contract assertions,
panel row update, rowid hand-off + no-op paths). Existing attach tests
stub `ap._dp` to stay off the network.
- **Live dry test** (`panel_payload_drytest_2026-08-15.py`, fabricated
numbers only, 13 real DeepSeek requests total): **7/7 probes pass** —
coherent-bullish → support@70; bearish-vs-BUY → oppose@85; genuinely
mixed → neutral@30; empty payload → neutral@20 (contract); no invented
numbers in any note; all notes ≤ 15 words; synthesizer AGREE with a
21-word rationale. Raw wire log:
`backtest/research/panel_payload_drytest_2026-08-15.jsonl`.
- Notable: an early P3 variant with H1 SELL against a BUY brief got
oppose@60 — *correct* per the prompt (higher-TF disagreement is a
listed conflict), so the probe was tightened to a genuinely mixed
scenario. Documented so nobody "fixes" the prompt for it later.
## What this deliberately did NOT change
- No gate reads panel fields; the panel remains advisory-only.
- No backtest-path change (`analyze_from_dataframes` untouched; parity
test still locks panel fields empty in replay).
- No verdict-based filtering — that decision waits for the signals.db
dataset (4–8 weeks) and an offline veto replay against the pinned book.
@@ -1,7 +0,0 @@
{"probe": "P1 coherent-bullish", "payload": {"brief": {"pair": "XAU/USD", "timeframe": "15min", "direction": "BUY", "quality": "A", "score": 38.5, "confidence": 62, "price": 2000.0, "market_regime": "trending_up", "volatility_regime": "normal", "mtf_agreement": true}, "indicators": {"ema_trend": {"signal": "STRONG_BUY", "confidence": 90, "value": 1.0}, "rsi": {"signal": "BUY", "confidence": 65, "value": 58.0}, "macd": {"signal": "STRONG_BUY", "confidence": 80, "value": 1.25}, "bollinger": {"signal": "BUY", "confidence": 60, "value": 0.7}, "adx": {"signal": "BUY", "confidence": 70, "value": 32.0}, "roc": {"signal": "BUY", "confidence": 55, "value": 0.35}, "keltner": {"signal": "BUY", "confidence": 65, "value": 3.1}, "candlestick": {"signal": "BUY", "confidence": 60, "value": 2.0}}, "mtf": {"5min": {"signal": "BUY", "score": 30.0}, "1h": {"signal": "STRONG_BUY", "score": 45.0}}, "key_levels": {"resistance": 2012.0, "support": 1985.0}, "exhaustion": 0, "candles": {"15min": {"ref_close": 2000.0, "atr": 5.0, "bars": [[-2.4, -2.2, -2.4, -2.3], [-2.3, -2.1, -2.3, -2.2], [-2.2, -2.0, -2.2, -2.0], [-2.0, -1.9, -2.1, -1.9], [-1.9, -1.8, -2.0, -1.8], [-1.8, -1.6, -1.8, -1.7], [-1.7, -1.5, -1.7, -1.6], [-1.6, -1.4, -1.6, -1.4], [-1.4, -1.3, -1.5, -1.3], [-1.3, -1.2, -1.4, -1.2], [-1.2, -1.0, -1.2, -1.1], [-1.1, -0.9, -1.1, -1.0], [-1.0, -0.8, -1.0, -0.8], [-0.8, -0.7, -0.9, -0.7], [-0.7, -0.6, -0.8, -0.6], [-0.6, -0.4, -0.6, -0.5], [-0.5, -0.3, -0.5, -0.4], [-0.4, -0.2, -0.4, -0.2], [-0.2, -0.1, -0.3, -0.1], [-0.1, 0.0, -0.2, 0.0]]}}}, "raw_reply": {"stance": "support", "confidence": 70, "note": "The trend is up and all signs agree, so buying fits the flow."}}
{"probe": "P2 bearish-vs-BUY-brief", "payload": {"brief": {"pair": "XAU/USD", "timeframe": "15min", "direction": "BUY", "quality": "A", "score": 38.5, "confidence": 62, "price": 2000.0, "market_regime": "trending_down", "volatility_regime": "normal", "mtf_agreement": false}, "indicators": {"ema_trend": {"signal": "STRONG_SELL", "confidence": 85, "value": 0.1}, "rsi": {"signal": "SELL", "confidence": 70, "value": 74.0}, "macd": {"signal": "STRONG_SELL", "confidence": 80, "value": -1.4}, "bollinger": {"signal": "SELL", "confidence": 60, "value": 0.95}, "adx": {"signal": "SELL", "confidence": 70, "value": 35.0}, "roc": {"signal": "SELL", "confidence": 60, "value": -0.5}, "keltner": {"signal": "SELL", "confidence": 65, "value": -2.8}, "candlestick": {"signal": "SELL", "confidence": 55, "value": -2.0}}, "mtf": {"5min": {"signal": "SELL", "score": -28.0}, "1h": {"signal": "STRONG_SELL", "score": -41.0}}, "key_levels": {"resistance": 2012.0, "support": 1985.0}, "exhaustion": 3, "candles": {"15min": {"ref_close": 2000.0, "atr": 5.0, "bars": [[2.4, 2.4, 2.2, 2.3], [2.3, 2.3, 2.1, 2.2], [2.2, 2.2, 2.0, 2.0], [2.0, 2.1, 1.9, 1.9], [1.9, 2.0, 1.8, 1.8], [1.8, 1.8, 1.6, 1.7], [1.7, 1.7, 1.5, 1.6], [1.6, 1.6, 1.4, 1.4], [1.4, 1.5, 1.3, 1.3], [1.3, 1.4, 1.2, 1.2], [1.2, 1.2, 1.0, 1.1], [1.1, 1.1, 0.9, 1.0], [1.0, 1.0, 0.8, 0.8], [0.8, 0.9, 0.7, 0.7], [0.7, 0.8, 0.6, 0.6], [0.6, 0.6, 0.4, 0.5], [0.5, 0.5, 0.3, 0.4], [0.4, 0.4, 0.2, 0.2], [0.2, 0.3, 0.1, 0.1], [0.1, 0.2, -0.0, 0.0]]}}}, "raw_reply": {"stance": "oppose", "confidence": 85, "note": "The price is falling and every indicator says sell, so buying fights the trend."}}
{"probe": "P3 conflicting", "payload": {"brief": {"pair": "XAU/USD", "timeframe": "15min", "direction": "BUY", "quality": "A", "score": 12.0, "confidence": 35, "price": 2000.0, "market_regime": "ranging", "volatility_regime": "normal", "mtf_agreement": false}, "indicators": {"ema_trend": {"signal": "BUY", "confidence": 55, "value": 0.6}, "rsi": {"signal": "NEUTRAL", "confidence": 40, "value": 51.0}, "macd": {"signal": "SELL", "confidence": 55, "value": -0.3}, "bollinger": {"signal": "NEUTRAL", "confidence": 35, "value": 0.5}, "adx": {"signal": "NEUTRAL", "confidence": 40, "value": 18.0}, "roc": {"signal": "SELL", "confidence": 50, "value": -0.1}, "keltner": {"signal": "BUY", "confidence": 50, "value": 0.8}, "candlestick": {"signal": "NEUTRAL", "confidence": 30, "value": 0.0}}, "mtf": {"5min": {"signal": "NEUTRAL", "score": 4.0}, "1h": {"signal": "NEUTRAL", "score": -3.0}}, "key_levels": {}, "exhaustion": 0, "candles": {"15min": {"ref_close": 2000.0, "atr": 5.0, "bars": [[0.0, 0.4, -0.1, 0.3], [0.3, 0.4, -0.1, 0.0], [0.0, 0.4, -0.1, 0.3], [0.3, 0.4, -0.1, 0.0], [0.0, 0.4, -0.1, 0.3], [0.3, 0.4, -0.1, 0.0], [0.0, 0.4, -0.1, 0.3], [0.3, 0.4, -0.1, 0.0], [0.0, 0.4, -0.1, 0.3], [0.3, 0.4, -0.1, 0.0], [0.0, 0.4, -0.1, 0.3], [0.3, 0.4, -0.1, 0.0], [0.0, 0.4, -0.1, 0.3], [0.3, 0.4, -0.1, 0.0], [0.0, 0.4, -0.1, 0.3], [0.3, 0.4, -0.1, 0.0], [0.0, 0.4, -0.1, 0.3], [0.3, 0.4, -0.1, 0.0], [0.0, 0.4, -0.1, 0.3], [0.3, 0.4, -0.1, 0.0]]}}}, "raw_reply": {"stance": "neutral", "confidence": 30, "note": "Mixed signals with no clear direction, so cannot strongly support or oppose."}}
{"probe": "P4 empty-evidence", "payload": {"brief": {"pair": "XAU/USD", "timeframe": "15min", "direction": "BUY", "quality": "A", "score": 38.5, "confidence": 62, "price": 2000.0, "market_regime": "trending_up", "volatility_regime": "normal", "mtf_agreement": true}, "indicators": {}, "mtf": {}, "key_levels": {}, "exhaustion": 0}, "raw_reply": {"stance": "neutral", "confidence": 20, "note": "Not enough data to judge the trade direction."}}
{"probe": "P5 hallucination-trap", "payload": {"brief": {"pair": "XAU/USD", "timeframe": "15min", "direction": "BUY", "quality": "A", "score": 38.5, "confidence": 62, "price": 2000.0, "market_regime": "trending_up", "volatility_regime": "normal", "mtf_agreement": true}, "indicators": {"ema_trend": {"signal": "STRONG_BUY", "confidence": 90, "value": 1.0}, "rsi": {"signal": "BUY", "confidence": 65, "value": 58.0}, "macd": {"signal": "STRONG_BUY", "confidence": 80, "value": 1.25}, "bollinger": {"signal": "BUY", "confidence": 60, "value": 0.7}, "adx": {"signal": "BUY", "confidence": 70, "value": 32.0}, "roc": {"signal": "BUY", "confidence": 55, "value": 0.35}, "keltner": {"signal": "BUY", "confidence": 65, "value": 3.1}, "candlestick": {"signal": "BUY", "confidence": 60, "value": 2.0}}, "mtf": {"5min": {"signal": "BUY", "score": 30.0}, "1h": {"signal": "STRONG_BUY", "score": 45.0}}, "key_levels": {}, "exhaustion": 0}, "raw_reply": {"stance": "support", "confidence": 70, "note": "The trend is up and all signs agree, so buying fits the flow."}}
{"probe": "P6 output-discipline", "payload": {}, "raw_reply": null}
{"probe": "P7 synthesizer", "payload": {"direction": "BUY", "takes": [{"persona": "macro", "stance": "support", "confidence": 70, "note": "evidence aligned"}, {"persona": "technical", "stance": "support", "confidence": 80, "note": "evidence aligned"}, {"persona": "risk", "stance": "support", "confidence": 65, "note": "evidence aligned"}, {"persona": "contrarian", "stance": "support", "confidence": 60, "note": "evidence aligned"}]}, "raw_reply": {"verdict": "AGREE", "confidence": 85, "rationale": "All four analysts support buying, with high confidence and consistent evidence. No one opposes, so the panel clearly backs the trade."}}
@@ -1,271 +0,0 @@
#!/usr/bin/env python3
"""
panel_payload_drytest_2026-08-15.py — live DeepSeek validation of the
enriched agent-panel payloads + hardened prompts (2026-08-15).
Spends ~6 REAL DeepSeek requests (against the shared 500/day cap) with
FABRICATED payloads — made-up prices/indicators, no real market data —
to verify the new technical-persona contract end to end:
P1 coherent bullish evidence -> technical persona SUPPORTS
P2 bearish evidence vs BUY brief -> technical persona OPPOSES
P3 conflicting evidence -> NEUTRAL or low confidence
P4 empty evidence -> NEUTRAL with confidence < 30
(the anti-hallucination contract)
P5 hallucination trap -> note invents no price levels
P6 output discipline -> <=20-word notes, exact JSON keys
P7 synthesizer over 4 supports -> AGREE verdict
Usage: venv/bin/python backtest/research/panel_payload_drytest_2026-08-15.py
Exit code 0 = all probes passed. Raw replies land in
backtest/research/panel_drytest_2026-08-15.jsonl
"""
import json
import re
import sys
from pathlib import Path
import pandas as pd
REPO_ROOT = Path(__file__).resolve().parents[2]
sys.path.insert(0, str(REPO_ROOT))
import agent_panel as ap # noqa: E402
from app_config import deepseek_api_key, get_app_config # noqa: E402
from llm_client import get_llm_client # noqa: E402
ARTIFACT = Path(__file__).with_suffix(".jsonl")
# ── Fabrication helpers (all numbers are invented) ───────────────────
def _staircase(start: float, step: float, n: int) -> pd.DataFrame:
"""n synthetic bars drifting `step` per bar; last close = start+n*step."""
rows, price = [], start
for _ in range(n):
o, c = price, price + step
rows.append({"Open": o, "Close": c,
"High": max(o, c) + abs(step) * 0.4,
"Low": min(o, c) - abs(step) * 0.4})
price = c
return pd.DataFrame(rows)
def _choppy(center: float, n: int) -> pd.DataFrame:
rows, price, sign = [], center, 1
for _ in range(n):
step = sign * 1.5
o, c = price, price + step
rows.append({"Open": o, "Close": c,
"High": max(o, c) + 0.6, "Low": min(o, c) - 0.6})
price, sign = c, -sign
return pd.DataFrame(rows)
_FAKE_PRICE = 2000.00 # deliberately made-up
_FAKE_ATR = 5.0
_BRIEF_BUY = {"pair": "XAU/USD", "timeframe": "15min", "direction": "BUY",
"quality": "A", "score": 38.5, "confidence": 62,
"price": _FAKE_PRICE, "market_regime": "trending_up",
"volatility_regime": "normal", "mtf_agreement": True}
_BULL_INDICATORS = {
"ema_trend": {"signal": "STRONG_BUY", "confidence": 90, "value": 1.0},
"rsi": {"signal": "BUY", "confidence": 65, "value": 58.0},
"macd": {"signal": "STRONG_BUY", "confidence": 80, "value": 1.25},
"bollinger": {"signal": "BUY", "confidence": 60, "value": 0.7},
"adx": {"signal": "BUY", "confidence": 70, "value": 32.0},
"roc": {"signal": "BUY", "confidence": 55, "value": 0.35},
"keltner": {"signal": "BUY", "confidence": 65, "value": 3.1},
"candlestick": {"signal": "BUY", "confidence": 60, "value": 2.0},
}
_BEAR_INDICATORS = {
"ema_trend": {"signal": "STRONG_SELL", "confidence": 85, "value": 0.1},
"rsi": {"signal": "SELL", "confidence": 70, "value": 74.0},
"macd": {"signal": "STRONG_SELL", "confidence": 80, "value": -1.4},
"bollinger": {"signal": "SELL", "confidence": 60, "value": 0.95},
"adx": {"signal": "SELL", "confidence": 70, "value": 35.0},
"roc": {"signal": "SELL", "confidence": 60, "value": -0.5},
"keltner": {"signal": "SELL", "confidence": 65, "value": -2.8},
"candlestick": {"signal": "SELL", "confidence": 55, "value": -2.0},
}
_MIXED_INDICATORS = {
"ema_trend": {"signal": "BUY", "confidence": 55, "value": 0.6},
"rsi": {"signal": "NEUTRAL", "confidence": 40, "value": 51.0},
"macd": {"signal": "SELL", "confidence": 55, "value": -0.3},
"bollinger": {"signal": "NEUTRAL", "confidence": 35, "value": 0.5},
"adx": {"signal": "NEUTRAL", "confidence": 40, "value": 18.0},
"roc": {"signal": "SELL", "confidence": 50, "value": -0.1},
"keltner": {"signal": "BUY", "confidence": 50, "value": 0.8},
"candlestick": {"signal": "NEUTRAL", "confidence": 30, "value": 0.0},
}
def _tech_payload(indicators, mtf, candles_df=None, key_levels=None,
brief=None, exhaustion=0):
payload = {"brief": brief or dict(_BRIEF_BUY),
"indicators": indicators, "mtf": mtf,
"key_levels": key_levels or {}, "exhaustion": exhaustion}
if candles_df is not None:
# the REAL production normalizer, not a hand-rolled copy
payload["candles"] = {
"15min": ap._normalize_candles(candles_df, _FAKE_ATR,
len(candles_df))}
return payload
# ── Probe runner ─────────────────────────────────────────────────────
_RESULTS = []
def _record(probe, payload, reply, take, passed, detail):
_RESULTS.append({"probe": probe, "pass": bool(passed),
"detail": detail,
"stance": getattr(take, "stance", None),
"confidence": getattr(take, "confidence", None),
"note": getattr(take, "note", None)})
with ARTIFACT.open("a") as fh:
fh.write(json.dumps({"probe": probe, "payload": payload,
"raw_reply": reply}, default=str) + "\n")
mark = "PASS" if passed else "FAIL"
print(f"[{mark}] {probe}: {detail}")
if take is not None:
print(f" -> {take.stance} @ {take.confidence:.0f} "
f"| {take.note}")
def _ask(payload):
"""Real API call through the production client + production
validation (the same path run_panel uses)."""
reply = get_llm_client().chat_json(ap._TECH_SYSTEM, payload)
return reply, ap._validate_take("technical", reply)
def _want(probe: str) -> bool:
"""--only P3 runs just that probe (saves requests on re-runs)."""
if "--only" not in sys.argv:
return True
return sys.argv[sys.argv.index("--only") + 1] == probe
def main() -> int:
app = get_app_config()
if not deepseek_api_key(app):
print("ABORT: no DeepSeek API key (env DEEPSEEK_API_KEY / "
"config.json deepseek.api_key).")
return 2
if app.deepseek_settings.get("enabled", True) is False:
print("ABORT: deepseek.enabled is false in config.json.")
return 2
if ARTIFACT.exists() and "--only" not in sys.argv:
ARTIFACT.unlink() # full run = fresh wire log; --only appends
mtf_agree = {"5min": {"signal": "BUY", "score": 30.0},
"1h": {"signal": "STRONG_BUY", "score": 45.0}}
levels = {"resistance": 2012.0, "support": 1985.0}
# P1 — coherent bullish, with rising ATR staircase (last close 2000)
if _want("P1"):
p1 = _tech_payload(_BULL_INDICATORS, mtf_agree,
candles_df=_staircase(_FAKE_PRICE - 20 * 0.6,
0.6, 20),
key_levels=levels)
reply, take = _ask(p1)
_record("P1 coherent-bullish", p1, reply, take,
take is not None and take.stance == "support"
and take.confidence >= 50,
"expect support @ conf>=50")
# P2 — bearish everything against a BUY brief, falling staircase
if _want("P2"):
brief2 = dict(_BRIEF_BUY, market_regime="trending_down",
mtf_agreement=False)
mtf_sell = {"5min": {"signal": "SELL", "score": -28.0},
"1h": {"signal": "STRONG_SELL", "score": -41.0}}
p2 = _tech_payload(_BEAR_INDICATORS, mtf_sell, brief=brief2,
candles_df=_staircase(_FAKE_PRICE + 20 * 0.6,
-0.6, 20),
key_levels=levels, exhaustion=3)
reply, take = _ask(p2)
_record("P2 bearish-vs-BUY-brief", p2, reply, take,
take is not None and take.stance == "oppose",
"expect oppose (BUY brief fights all evidence)")
# P3 — genuinely mixed evidence: MTF neutral both ways, ranging
# regime, choppy candles. (An H1 SELL against a BUY is a *listed
# conflict* in the prompt — oppose is correct there, not mixed.)
if _want("P3"):
brief3 = dict(_BRIEF_BUY, market_regime="ranging", score=12.0,
confidence=35, mtf_agreement=False)
mtf_mixed = {"5min": {"signal": "NEUTRAL", "score": 4.0},
"1h": {"signal": "NEUTRAL", "score": -3.0}}
p3 = _tech_payload(_MIXED_INDICATORS, mtf_mixed, brief=brief3,
candles_df=_choppy(_FAKE_PRICE, 20))
reply, take = _ask(p3)
_record("P3 conflicting", p3, reply, take,
take is not None and (take.stance == "neutral"
or take.confidence < 60),
"expect neutral or conf<60 on genuinely mixed evidence")
# P4 — empty evidence: contract demands neutral @ conf<30
if _want("P4"):
p4 = _tech_payload({}, {}, key_levels=None)
reply, take = _ask(p4)
_record("P4 empty-evidence", p4, reply, take,
take is not None and take.stance == "neutral"
and take.confidence < 30,
"contract: thin/missing evidence -> neutral, conf<30")
# P5 — hallucination trap: no key_levels, no candles; the note must
# not cite any 3+-digit number absent from the payload
if _want("P5"):
p5 = _tech_payload(_BULL_INDICATORS, mtf_agree, key_levels=None)
reply, take = _ask(p5)
invented = []
if take is not None:
payload_blob = json.dumps(p5)
for num in re.findall(r"\d{3,}", take.note):
if num not in payload_blob:
invented.append(num)
_record("P5 hallucination-trap", p5, reply, take,
take is not None and not invented,
f"invented numbers in note: {invented or 'none'}")
# P6 — output discipline across the persona replies gathered above
notes = [r for r in _RESULTS if r["note"]]
wordy = [r["probe"] for r in notes if len(r["note"].split()) > 20]
_record("P6 output-discipline", {}, None, None,
notes and not wordy,
f"notes >20 words: {wordy or 'none'} "
f"(cap is 15; 20 = fail threshold)")
# P7 — synthesizer over four fabricated SUPPORT takes
if _want("P7"):
takes = [{"persona": p, "stance": "support", "confidence": c,
"note": "evidence aligned"}
for p, c in (("macro", 70), ("technical", 80),
("risk", 65), ("contrarian", 60))]
synth_payload = {"direction": "BUY", "takes": takes}
reply = get_llm_client().chat_json(ap._SYNTH_SYSTEM, synth_payload)
verdict = reply.get("verdict") if isinstance(reply, dict) else None
rationale = (reply or {}).get("rationale", "")
_record("P7 synthesizer", synth_payload, reply, None,
verdict == "AGREE" and len(str(rationale).split()) <= 40,
f"expect AGREE + <=40-word rationale; got {verdict!r} "
f"({len(str(rationale).split())} words)")
n_pass = sum(1 for r in _RESULTS if r["pass"])
n_api = sum(1 for r in _RESULTS if r["probe"] != "P6 output-discipline")
print(f"\n{n_pass}/{len(_RESULTS)} probes passed "
f"({n_api} API requests spent; "
f"raw wire log: {ARTIFACT.name})")
return 0 if n_pass == len(_RESULTS) else 1
if __name__ == "__main__":
sys.exit(main())
+1 -14
View File
@@ -46,20 +46,7 @@
"dashboard": false, "dashboard": false,
"min_quality": "B", "min_quality": "B",
"max_panels_per_day": 20, "max_panels_per_day": 20,
"reuse_ttl_seconds": 3600, "reuse_ttl_seconds": 3600
"candles_enabled": true,
"candle_bars_primary": 20,
"candle_bars_confirm": 10
},
"market_outlook": {
"enabled": false,
"telegram": false,
"max_per_day": 12,
"candles_bars": 20
},
"signal_log": {
"enabled": true,
"path": "logs/signals.db"
}, },
"log_level": "INFO" "log_level": "INFO"
} }
+3 -8
View File
@@ -31,10 +31,8 @@ class SignalStrength(Enum):
XAUUSD = "XAU/USD" XAUUSD = "XAU/USD"
# price_precision: decimals for SL/TP rounding and display. # price_precision: decimals for SL/TP rounding and display.
# default_spread: $0.30 round-trip cost retained for TD-era cross-book # default_spread: round-trip cost per trade in price units, charged by
# comparability and used when the backtester receives no --spread override. # the backtester when --spread is not given.
# Oanda's measured round trip is $0.65-$0.72; research reports headline it
# separately with --spread 0.65.
PAIR_INFO = { PAIR_INFO = {
XAUUSD: {"display": "XAU/USD", "pip_value": 0.10, XAUUSD: {"display": "XAU/USD", "pip_value": 0.10,
"price_precision": 2, "default_spread": 0.30}, "price_precision": 2, "default_spread": 0.30},
@@ -112,10 +110,7 @@ class SignalConfig:
tp_tightening_factor: float = 1.0 tp_tightening_factor: float = 1.0
# ── Trade geometry ──────────────────────────────────────────── # ── Trade geometry ────────────────────────────────────────────
# Deliberately $0.30 for TD-era cross-book comparability. Oanda's measured # Round-trip transaction cost estimate in price units (XAU ~$0.30).
# round trip is $0.65-$0.72, reported separately in re-pin campaigns.
# This is also a LIVE gate input: increasing it to $0.65 raises the TP1
# floor from about $9 to $19.50, so it requires rule-1 revalidation.
round_trip_cost: float = 0.30 round_trip_cost: float = 0.30
# Spread gate: reject setups whose TP1 distance is below # Spread gate: reject setups whose TP1 distance is below
# round_trip_cost × this multiple (caps cost at ~3% of target). # round_trip_cost × this multiple (caps cost at ~3% of target).
+14 -34
View File
@@ -82,7 +82,7 @@ SUMMARY_PHP_FILE = "news_summary.php"
# Repo files copied into the out dir at daemon start (git pull + restart # Repo files copied into the out dir at daemon start (git pull + restart
# updates them). The PHP endpoint reads news.json in place, so it lives # updates them). The PHP endpoint reads news.json in place, so it lives
# next to the page. # next to the page.
STATIC_FILES = [PAGE_FILE, "dashboard.css", "dashboard.js", SUMMARY_PHP_FILE] STATIC_FILES = [PAGE_FILE, SUMMARY_PHP_FILE]
# ══════════════════════════════════════════════════════════════════ # ══════════════════════════════════════════════════════════════════
@@ -322,7 +322,6 @@ class DashboardPublisher:
def __init__(self, out_dir: str): def __init__(self, out_dir: str):
self.out_dir = out_dir self.out_dir = out_dir
self.snapshots: Dict[str, dict] = {} self.snapshots: Dict[str, dict] = {}
self._reports = {}
self.news_payload: Optional[dict] = None self.news_payload: Optional[dict] = None
self._dp = None self._dp = None
self._news_engine: Optional[NewsEngine] = None self._news_engine: Optional[NewsEngine] = None
@@ -359,7 +358,13 @@ class DashboardPublisher:
try: try:
report = produce_signal(XAUUSD, tf, self._provider(), report = produce_signal(XAUUSD, tf, self._provider(),
CONFIRMATIONS[tf]) CONFIRMATIONS[tf])
self._reports[tf.value] = report # Advisory AI panel, post-hoc and display-only. Off unless
# config.json agent_panel.dashboard is true. The reuse-TTL
# cache + max_panels_per_day budget bound the spend against
# this process's own DeepSeek cap (a persisting M5 signal
# re-serves its cached verdict instead of re-running every
# bar); a fresh panel adds ≤~25 s before this TF publishes.
attach_panel(report, surface="dashboard")
self.snapshots[tf.value] = { self.snapshots[tf.value] = {
"status": "ok", "status": "ok",
"scanned_at": datetime.now(timezone.utc).isoformat(), "scanned_at": datetime.now(timezone.utc).isoformat(),
@@ -371,41 +376,16 @@ class DashboardPublisher:
logger.info(f"{tf.value}: {report.final_signal.name} " logger.info(f"{tf.value}: {report.final_signal.name} "
f"q={report.signal_quality or '-'} " f"q={report.signal_quality or '-'} "
f"({time.time() - started:.1f}s)") f"({time.time() - started:.1f}s)")
except Exception: except Exception as e:
# Keep the last good report; the page shows it with an # Keep the last good report; the page shows it with an
# error badge instead of going blank. # error badge instead of going blank.
logger.exception(f"{tf.value} scan failed") logger.exception(f"{tf.value} scan failed")
self._reports.pop(tf.value, None)
entry = self.snapshots.get(tf.value) or _empty_tf_entry() entry = self.snapshots.get(tf.value) or _empty_tf_entry()
entry["status"] = "error" entry["status"] = "error"
# The public snapshot carries a stable message; exception entry["error"] = f"{type(e).__name__}: {e}"
# detail remains in dashboard.log for the operator.
entry["error"] = "Scan unavailable"
entry["error_at"] = datetime.now(timezone.utc).isoformat() entry["error_at"] = datetime.now(timezone.utc).isoformat()
self.snapshots[tf.value] = entry self.snapshots[tf.value] = entry
def attach_panels(self, tfs: List[Timeframe]) -> bool:
"""Add advisory panels after publishing the technical snapshot.
A panel is display-only and can take tens of seconds on a cold LLM
request. Keeping its report object locally lets the page receive the
candle-close signal first, then a second atomic snapshot only when a
panel produced a result.
"""
updated = False
for tf in tfs:
report = self._reports.get(tf.value)
if report is None:
continue
if not attach_panel(report, surface="dashboard"):
continue
entry = self.snapshots.get(tf.value)
if entry is None or entry.get("status") != "ok":
continue
entry["report"] = report_to_webapp_dict(report)
updated = True
return updated
def refresh_news(self) -> None: def refresh_news(self) -> None:
try: try:
ne = self._news() ne = self._news()
@@ -432,10 +412,10 @@ class DashboardPublisher:
def run_cycle(self, tfs: List[Timeframe]) -> None: def run_cycle(self, tfs: List[Timeframe]) -> None:
for tf in tfs: # sequential — see module docstring (cache dedup) for tf in tfs: # sequential — see module docstring (cache dedup)
self.scan_tf(tf) self.scan_tf(tf)
# Technical signals land before any optional slow work. News and the # Snapshot lands BEFORE the news refresh: when the news TTL has
# advisory panel are display-only and may take 20-30 seconds cold. # expired, refresh_news does FF + 4 RSS fetches (and possibly
self.publish_snapshot() # DeepSeek calls) — 20-30 s that must not keep fresh signals off
if self.attach_panels(tfs): # the page.
self.publish_snapshot() self.publish_snapshot()
self.refresh_news() self.refresh_news()
self.publish_news() self.publish_news()
+9 -9
View File
@@ -84,16 +84,16 @@
</ol> </ol>
<h2>4 · Choosing your timeframe</h2> <h2>4 · Choosing your timeframe</h2>
<p>Twelve-month Oanda-candle backtest (Aug 2025 – Aug 2026, $0.65 round-trip cost, entry at next candle open):</p> <p>Twelve-month backtest (Jul 2025 – Jul 2026, real spreads, entry at next candle open):</p>
<table> <table>
<tr><th></th><th>M5</th><th>M15</th><th>H1</th></tr> <tr><th></th><th>M5</th><th>M15</th><th>H1</th></tr>
<tr><td>Trades / week</td><td>~11.3</td><td>~7.6</td><td>~2.7</td></tr> <tr><td>Trades / week</td><td>~10</td><td>~9.5</td><td>~3</td></tr>
<tr><td>Total result (12 mo)</td><td>+106.8R</td><td>+73.9R</td><td>+24.0R</td></tr> <tr><td>Total result (12 mo)</td><td>+135.6R</td><td>+67.2R</td><td>+49.9R</td></tr>
<tr><td>Win rate / profit factor</td><td>52.9% / 1.51</td><td>62.2% / 1.53</td><td>60.9% / 1.55</td></tr> <tr><td>Win rate / profit factor</td><td>59.7% / 1.81</td><td>55.8% / 1.40</td><td>59.4% / 1.94</td></tr>
<tr><td>Typical (median) hold</td><td>2.4 h</td><td>7 h</td><td>15.5 h</td></tr> <tr><td>Typical (median) hold</td><td>2.4 h</td><td>6 h</td><td>12.5 h</td></tr>
<tr><td>90% of trades close within</td><td>10.5 h</td><td>29.8 h</td><td>3.2 days</td></tr> <tr><td>90% of trades close within</td><td>16 h</td><td>40 h</td><td>3.7 days</td></tr>
<tr><td>Typical stop distance</td><td>~$21</td><td>~$32</td><td>~$52</td></tr> <tr><td>Typical stop distance</td><td>~$19</td><td>~$29</td><td>~$44</td></tr>
<tr><td>Worst drawdown</td><td>10.7R</td><td>10.8R</td><td>8.0R</td></tr> <tr><td>Worst drawdown</td><td>11.2R</td><td>9.4R</td><td>9.5R</td></tr>
</table> </table>
<ul> <ul>
<li><b>M5 — intraday, execution-critical.</b> Highest total edge, but stops are tight: a couple of dollars of slippage or a late entry eats a meaningful share of every trade. Suitable only if you can act within a minute or two of the alert (or automate). Trades open and close within a session; you sleep flat.</li> <li><b>M5 — intraday, execution-critical.</b> Highest total edge, but stops are tight: a couple of dollars of slippage or a late entry eats a meaningful share of every trade. Suitable only if you can act within a minute or two of the alert (or automate). Trades open and close within a session; you sleep flat.</li>
@@ -104,7 +104,7 @@
<h2>5 · What to expect (and what not to)</h2> <h2>5 · What to expect (and what not to)</h2>
<ul> <ul>
<li><b>You will lose roughly 38–47% of trades.</b> A 53–62% win rate means losing streaks of 5+ are routine over a year. The system's edge is the average across hundreds of trades, not any single one.</li> <li><b>You will lose ~40% of trades.</b> A 56–60% win rate means losing streaks of 5+ are routine over a year. The system's edge is the average across hundreds of trades, not any single one.</li>
<li><b>Stops hit fast, winners take time.</b> Losing trades die in roughly half the time winners take. An early stop-out is the system doing its job.</li> <li><b>Stops hit fast, winners take time.</b> Losing trades die in roughly half the time winners take. An early stop-out is the system doing its job.</li>
<li><b>Drawdowns of ~10R are normal.</b> At 1% risk per trade, expect to be ~10% down from a peak at some point in a normal year. This is why risking more than 1% per trade is how accounts die.</li> <li><b>Drawdowns of ~10R are normal.</b> At 1% risk per trade, expect to be ~10% down from a peak at some point in a normal year. This is why risking more than 1% per trade is how accounts die.</li>
<li><b>Backtest ≠ live.</b> Results above charge real spreads and realistic fills, but cannot model your personal latency, broker slippage, or requotes. Assume live results will be somewhat worse — especially on M5.</li> <li><b>Backtest ≠ live.</b> Results above charge real spreads and realistic fills, but cannot model your personal latency, broker slippage, or requotes. Assume live results will be somewhat worse — especially on M5.</li>
+2 -48
View File
@@ -13,7 +13,6 @@ import json
import logging import logging
import sys import sys
from datetime import datetime, timezone from datetime import datetime, timezone
from typing import Optional
# Force UTF-8 output on Windows so box-drawing chars render # Force UTF-8 output on Windows so box-drawing chars render
if hasattr(sys.stdout, "reconfigure"): if hasattr(sys.stdout, "reconfigure"):
@@ -23,7 +22,6 @@ if hasattr(sys.stderr, "reconfigure"):
from agent_panel import attach_panel from agent_panel import attach_panel
from config import PlatformConfig, Timeframe, SignalStrength, XAUUSD from config import PlatformConfig, Timeframe, SignalStrength, XAUUSD
from market_outlook import OutlookResult, maybe_run_outlook
from pair_signals import PAIR_PRODUCERS from pair_signals import PAIR_PRODUCERS
from signal_aggregator import SignalAggregator, SignalReport from signal_aggregator import SignalAggregator, SignalReport
@@ -199,24 +197,6 @@ def print_signal_report(report: SignalReport, detail: bool = False):
print(f" {label:>20} {color}{val:.3f}{Colors.RESET}") print(f" {label:>20} {color}{val:.3f}{Colors.RESET}")
def print_market_outlook(outlook: Optional[OutlookResult]) -> None:
"""Render the independent advisory outlook for explicit CLI use."""
print(f"\n {Colors.BOLD}AI Market Outlook{Colors.RESET}")
if outlook is None:
print(f" {Colors.GRAY}Unavailable (disabled, market closed, or no LLM response).{Colors.RESET}")
return
forming = {
"buy": "possible buy forming",
"sell": "possible sell forming",
"none": "no clear setup forming",
}[outlook.setup_forming]
print(f" {outlook.bias.title()} · {forming} · {outlook.confidence:.0f}%")
if outlook.watch:
print(f" Watch: {outlook.watch}")
if outlook.note:
print(f" {Colors.GRAY}{outlook.note}{Colors.RESET}")
def report_to_dict(report: SignalReport) -> dict: def report_to_dict(report: SignalReport) -> dict:
d = { d = {
"symbol": report.symbol, "symbol": report.symbol,
@@ -241,9 +221,6 @@ def report_to_dict(report: SignalReport) -> dict:
"panel_confidence": round(report.panel_confidence, 1), "panel_confidence": round(report.panel_confidence, 1),
"panel_rationale": report.panel_rationale, "panel_rationale": report.panel_rationale,
"panel_takes": report.panel_takes, "panel_takes": report.panel_takes,
"panel_created_at": report.panel_created_at,
"panel_cache_status": report.panel_cache_status,
"panel_payload_hash": report.panel_payload_hash,
"bullish_reasons": report.bullish_reasons, "bullish_reasons": report.bullish_reasons,
"bearish_reasons": report.bearish_reasons, "bearish_reasons": report.bearish_reasons,
"warnings": report.warnings, "warnings": report.warnings,
@@ -260,8 +237,6 @@ def report_to_dict(report: SignalReport) -> dict:
"risk_pips": round(ts.risk_pips, 1), "risk_pips": round(ts.risk_pips, 1),
"reward_pips": round(ts.reward_pips, 1), "reward_pips": round(ts.reward_pips, 1),
"strategy": ts.strategy, "strategy": ts.strategy,
"setup_pattern": ts.setup_pattern,
"round_trip_cost": ts.round_trip_cost,
"trail_arm_r": ts.trail_arm_r, "trail_arm_r": ts.trail_arm_r,
"trail_dist_r": ts.trail_dist_r, "trail_dist_r": ts.trail_dist_r,
} }
@@ -308,9 +283,6 @@ Examples:
parser.add_argument("--panel", action="store_true", parser.add_argument("--panel", action="store_true",
help="Run the LLM agent panel on the result " help="Run the LLM agent panel on the result "
"(advisory only; needs a DeepSeek key)") "(advisory only; needs a DeepSeek key)")
parser.add_argument("--outlook", action="store_true",
help="Run the hourly AI market outlook now "
"(advisory only; needs a DeepSeek key)")
parser.add_argument("--json", action="store_true", parser.add_argument("--json", action="store_true",
help="Output as JSON") help="Output as JSON")
parser.add_argument("--verbose", "-v", action="store_true", parser.add_argument("--verbose", "-v", action="store_true",
@@ -364,37 +336,19 @@ def build_config(args) -> PlatformConfig:
def run_live(args, config: PlatformConfig): def run_live(args, config: PlatformConfig):
print(f"{Colors.GRAY} Fetching market data and running analysis...{Colors.RESET}\n") print(f"{Colors.GRAY} Fetching market data and running analysis...{Colors.RESET}\n")
aggregator = SignalAggregator(config) aggregator = SignalAggregator(config)
report = aggregator.analyze_pair(symbol=config.symbol, log_source="cli") report = aggregator.analyze_pair(symbol=config.symbol)
# Explicit user intent: force bypasses the enabled/surface/quality # Explicit user intent: force bypasses the enabled/surface/quality
# toggles, but the daily panel budget and LLM cap still apply. # toggles, but the daily panel budget and LLM cap still apply.
if getattr(args, "panel", False): if getattr(args, "panel", False):
attach_panel(report, surface="cli", force=True) attach_panel(report, surface="cli", force=True)
outlook = None
if getattr(args, "outlook", False):
outlook = maybe_run_outlook(surface="cli", force=True)
if args.json: if args.json:
payload = report_to_dict(report) print(json.dumps(report_to_dict(report), indent=2))
if getattr(args, "outlook", False):
payload["market_outlook"] = (
None if outlook is None else {
"bias": outlook.bias,
"setup_forming": outlook.setup_forming,
"confidence": outlook.confidence,
"watch": outlook.watch,
"note": outlook.note,
"created_ts": outlook.created_ts,
}
)
print(json.dumps(payload, indent=2))
return return
print_header(f"{config.display_name} SIGNAL INTELLIGENCE") print_header(f"{config.display_name} SIGNAL INTELLIGENCE")
print_signal_report(report, detail=args.detail) print_signal_report(report, detail=args.detail)
if getattr(args, "outlook", False):
print_market_outlook(outlook)
print(f"\n{Colors.CYAN}{'═' * 80}{Colors.RESET}") print(f"\n{Colors.CYAN}{'═' * 80}{Colors.RESET}")
print(f"{Colors.GRAY} Disclaimer: For educational/research purposes only.{Colors.RESET}") print(f"{Colors.GRAY} Disclaimer: For educational/research purposes only.{Colors.RESET}")
print(f"{Colors.CYAN}{'═' * 80}{Colors.RESET}\n") print(f"{Colors.CYAN}{'═' * 80}{Colors.RESET}\n")
-410
View File
@@ -1,410 +0,0 @@
"""
ForexBot2 — AI market outlook (advisory hourly assessment)
===========================================================
Builds a broad XAU/USD market brief and asks DeepSeek whether a good setup
appears to be forming. The result is written to the signal-history database
and may be displayed by consumers, but it is ADVISORY-ONLY: it never changes
signal scoring, gates, grades, or trade setup generation.
The outlook runs outside the replayable signal path. Individual candle/news
failures produce a partial brief; an unavailable LLM, malformed reply, or
locked database never reaches a scan caller as an exception.
The static prompt is intentional: DeepSeek prices byte-identical prompt
prefixes at its cache-hit rate, so timestamps and all other changing evidence
live only in the JSON user payload.
"""
import datetime as _dt
import logging
import math
import os
import sqlite3
import threading
import time
from dataclasses import dataclass
from typing import Any, Dict, List, Optional
from agent_panel import _normalize_candles
from app_config import (get_app_config, market_outlook_enabled,
signal_log_path)
from config import PlatformConfig, Timeframe, XAUUSD
from engine import TechnicalEngine
from llm_client import get_llm_client
from providers import base as _provider_base
logger = logging.getLogger(__name__)
DEFAULT_MAX_PER_DAY = 12
DEFAULT_CANDLE_BARS = 20
_OUTLOOK_TFS = (Timeframe.M15, Timeframe.H1, Timeframe.H4)
_VALID_BIASES = ("bullish", "bearish", "neutral")
_VALID_SETUPS = ("buy", "sell", "none")
# STATIC prompt: repeated prompt prefixes are much cheaper with DeepSeek's
# server-side cache. Variable market evidence belongs exclusively in payload.
SYSTEM_PROMPT = """\
You are writing an advisory market outlook for gold (XAU/USD). You receive a
JSON market brief covering recent candle shape, indicator states, price
levels, news, and recent technical signals. Assess what may be forming soon;
you are not a signal generator and must not provide an entry price, stop, or
target.
Gold doctrine: hawkish Fed / rate hikes / rising real yields / strong
dollar are gold-bearish; easing / rate cuts / safe-haven flows /
geopolitical stress / weak dollar are gold-bullish.
Indicator "value" units: rsi 0-100 (above 70 overbought, below 30
oversold); adx 0-100 (above 25 = strong trend); macd = histogram in
dollars (momentum); roc = percent; bollinger = position 0-1 (0 = lower
band, 1 = upper band); keltner = dollar distance from the mid-line;
ema_trend = 0-1 alignment score; divergence and candlestick = pattern
counts. "candles" is per timeframe: {"ref_close", "atr", "bars"}; bars
are oldest-first [open, high, low, close] in ATR units relative to the
latest close. Read their shape, never quote them as prices.
Hard rules: use ONLY the JSON provided. Do not invent numbers, events, or
price levels. Your training data about gold, markets, and current events is
stale and OFF-LIMITS: never use outside knowledge; if a fact is not in the
JSON, treat it as unknown. If evidence is thin or missing, answer "neutral"
with confidence below 30. Use plain everyday English for someone unfamiliar
with markets: short simple sentences, no acronyms or jargon. Keep "watch"
and "note" to at most 25 words each.
Reply with ONLY JSON:
{"bias": "bullish"|"bearish"|"neutral",
"setup_forming": "buy"|"sell"|"none",
"confidence": 0-100,
"watch": "<what to watch for, max 25 words>",
"note": "<plain-English summary, max 25 words>"}
"""
@dataclass
class OutlookResult:
bias: str
setup_forming: str
confidence: float
watch: str
note: str
created_ts: float
def _finite(x: Any) -> Optional[float]:
"""float(x) or None for non-numeric / non-finite JSON values."""
try:
value = float(x)
except (TypeError, ValueError):
return None
return value if math.isfinite(value) else None
def _round_finite(x: Any, digits: int) -> Optional[float]:
value = _finite(x)
return round(value, digits) if value is not None else None
def _json_safe(value: Any) -> Any:
"""Replace any accidental NaN/inf nested in provider data with null."""
if isinstance(value, dict):
return {str(key): _json_safe(item) for key, item in value.items()}
if isinstance(value, (list, tuple)):
return [_json_safe(item) for item in value]
if isinstance(value, bool) or value is None or isinstance(value, str):
return value
if isinstance(value, (int, float)):
return _finite(value)
return value
def _clamp_confidence(raw: Any) -> Optional[float]:
value = _finite(raw)
return max(0.0, min(100.0, value)) if value is not None else None
def _word_limit(text: Any, words: int = 25) -> str:
"""Bound a response field by words as well as a practical char limit."""
clean = str(text or "").strip()
return " ".join(clean.split()[:words])[:200]
def _validate_outlook(reply: Any) -> Optional[OutlookResult]:
"""Validate only model replies that fully match the requested contract."""
if not isinstance(reply, dict):
return None
bias = reply.get("bias")
setup_forming = reply.get("setup_forming")
confidence = _clamp_confidence(reply.get("confidence"))
if (bias not in _VALID_BIASES or setup_forming not in _VALID_SETUPS
or confidence is None):
return None
return OutlookResult(
bias=bias,
setup_forming=setup_forming,
confidence=confidence,
watch=_word_limit(reply.get("watch")),
note=_word_limit(reply.get("note")),
created_ts=time.time(),
)
# ── Data collection ──────────────────────────────────────────────────
_dp: Any = None # Patchable singleton, mirroring agent_panel._dp.
def _outlook_provider() -> Any:
"""Lazy provider singleton so its closed-bar cache stays warm hourly."""
global _dp
if _dp is None:
from providers import make_data_provider
_dp = make_data_provider(PlatformConfig())
return _dp
def _indicator_payload(analysis: Any) -> Dict[str, Dict[str, Any]]:
"""Visible scored indicators with finite, compact values only."""
out: Dict[str, Dict[str, Any]] = {}
for name, indicator in (getattr(analysis, "indicators", None) or {}).items():
if name.startswith("_") or getattr(indicator, "weight", 0) == 0:
continue
entry: Dict[str, Any] = {
"signal": getattr(getattr(indicator, "signal", None), "name", ""),
"confidence": _round_finite(
getattr(indicator, "confidence", None), 1),
"value": _round_finite(getattr(indicator, "value", None), 2),
}
out[name] = entry
return out
def _analysis_payload(analyses: Dict[str, Any]) -> Dict[str, Any]:
"""Pure formatter for the technical portion of the market brief."""
indicators: Dict[str, Any] = {}
market_regime: Dict[str, Optional[str]] = {}
volatility_regime: Dict[str, Optional[str]] = {}
key_levels: Dict[str, Dict[str, Optional[float]]] = {}
for tf in _OUTLOOK_TFS:
name = tf.value
analysis = analyses.get(name)
indicators[name] = _indicator_payload(analysis)
market_regime[name] = getattr(analysis, "market_regime", None)
volatility_regime[name] = getattr(analysis, "volatility_regime", None)
key_levels[name] = {
level: _round_finite(value, 2)
for level, value in (getattr(analysis, "key_levels", None) or {}).items()
}
return {
"indicators": indicators,
"market_regime": market_regime,
"volatility_regime": volatility_regime,
"key_levels": key_levels,
}
def _empty_news_payload() -> Dict[str, Any]:
return {
"headlines": None,
"events": None,
"gold_sentiment": None,
"remaining_high_impact_events": None,
}
def _news_payload() -> Dict[str, Any]:
"""Collect current news without allowing any news failure to block us."""
try:
from news_engine import NewsEngine
news = NewsEngine(PlatformConfig().news)
context = news.get_fundamental_context()
headlines = [str(getattr(h, "headline", ""))
for h in (context.news_headlines or [])[:10]
if getattr(h, "headline", None)]
events = [
{
"event": getattr(event, "event", None),
"currency": getattr(event, "currency", None),
"actual": getattr(event, "actual", None),
"forecast": getattr(event, "forecast", None),
"released": not event.is_upcoming,
}
for event in (context.events_today or [])
if event.is_high_impact
][:8]
gold_sentiment, _ = news.get_pair_sentiment(XAUUSD, context)
remaining = sum(
1 for event in news.get_week_events()
if event.is_high_impact and event.is_upcoming
)
return {
"headlines": headlines,
"events": events,
"gold_sentiment": _round_finite(gold_sentiment, 2),
"remaining_high_impact_events": remaining,
}
except Exception as exc:
logger.debug("market outlook: news context unavailable: %s", exc)
return _empty_news_payload()
def _recent_signals() -> List[Dict[str, Any]]:
"""Read the latest signal history through SQLite's immutable URI."""
try:
path = os.path.abspath(signal_log_path())
uri = f"file:{path}?immutable=1"
con = sqlite3.connect(uri, uri=True)
try:
rows = con.execute(
"SELECT logged_at, timeframe, signal, quality, panel_verdict "
"FROM signals ORDER BY id DESC LIMIT 10").fetchall()
finally:
con.close()
return [
{
"logged_at": row[0],
"timeframe": row[1],
"signal": row[2],
"quality": row[3],
"panel_verdict": row[4],
}
for row in reversed(rows)
]
except Exception as exc:
logger.debug("market outlook: signal history unavailable: %s", exc)
return []
def _build_payload(
analyses: Dict[str, Any],
candles: Dict[str, Any],
news: Dict[str, Any],
recent_signals: List[Dict[str, Any]],
as_of: _dt.datetime,
) -> Dict[str, Any]:
"""Pure market-brief formatter. Every numeric field is finite or null."""
technical = _analysis_payload(analyses)
return _json_safe({
"as_of": as_of.astimezone(_dt.timezone.utc).isoformat(),
"candles": candles,
**technical,
"news": news,
"recent_signals": recent_signals,
})
def _collect_payload(settings: Dict[str, Any]) -> Dict[str, Any]:
"""Fetch closed bars and turn them into a structured market brief."""
analyses: Dict[str, Any] = {}
candles: Dict[str, Any] = {tf.value: None for tf in _OUTLOOK_TFS}
bars = int(settings.get("candles_bars", DEFAULT_CANDLE_BARS))
provider = _outlook_provider()
engine = TechnicalEngine(PlatformConfig().technical)
for tf in _OUTLOOK_TFS:
try:
df = provider.fetch_ohlcv(XAUUSD, tf)
analysis = engine.analyze(df, XAUUSD, tf.value)
analyses[tf.value] = analysis
atr = _finite(getattr(
(getattr(analysis, "indicators", None) or {}).get("atr"),
"value", None))
if atr is not None and atr > 0 and df is not None and len(df) >= 2:
candles[tf.value] = _normalize_candles(df, atr, bars)
except Exception as exc:
logger.debug("market outlook: %s data unavailable: %s", tf.value, exc)
return _build_payload(
analyses=analyses,
candles=candles,
news=_news_payload(),
recent_signals=_recent_signals(),
as_of=_dt.datetime.now(_dt.timezone.utc),
)
# ── Orchestration, hourly dedup, and request budget ──────────────────
_state_lock = threading.Lock()
_last_run_ts: Optional[float] = None
_outlook_day: Optional[_dt.date] = None
_outlook_count = 0
_warned_budget = False
def reset_outlook_state() -> None:
"""Drop hourly and daily state (tests / deployment config reloads)."""
global _last_run_ts, _outlook_day, _outlook_count, _warned_budget
with _state_lock:
_last_run_ts = None
_outlook_day = None
_outlook_count = 0
_warned_budget = False
def _begin_attempt(cap: int, force: bool) -> bool:
"""Atomically apply hourly dedup, then reserve a daily attempt."""
global _last_run_ts, _outlook_day, _outlook_count, _warned_budget
now = time.time()
today = _dt.datetime.now(_dt.timezone.utc).date()
with _state_lock:
if not force and _last_run_ts is not None and now - _last_run_ts < 3600:
return False
if _outlook_day != today:
_outlook_day = today
_outlook_count = 0
_warned_budget = False
if _outlook_count >= cap:
if not _warned_budget:
_warned_budget = True
logger.warning(
"market outlook: daily budget (%d) exhausted — no more "
"outlooks until UTC midnight", cap)
return False
_outlook_count += 1
_last_run_ts = now
return True
def run_outlook(settings: Dict[str, Any],
surface: str = "live") -> Optional[OutlookResult]:
"""Build, ask, validate, and persist one advisory outlook."""
try:
payload = _collect_payload(settings)
result = _validate_outlook(get_llm_client().chat_json(SYSTEM_PROMPT, payload))
if result is None:
return None
try:
from signal_log import log_outlook
log_outlook(result, surface=surface, payload=payload)
except Exception:
logger.debug("market outlook: persistence unavailable", exc_info=True)
return result
except Exception:
logger.warning("market outlook: run failed", exc_info=True)
return None
def maybe_run_outlook(surface: str, force: bool = False) -> Optional[OutlookResult]:
"""Best-effort public entry point. It NEVER raises to a scan caller.
``force`` is for explicit CLI use: it bypasses enabled, surface, and
hourly gates, but not the independent outlook budget or shared LLM cap.
"""
try:
settings = get_app_config().market_outlook_settings
if not force:
if not market_outlook_enabled():
return None
if surface == "telegram" and not settings.get("telegram", False):
return None
if _provider_base._is_forex_weekend():
return None
cap = int(settings.get("max_per_day", DEFAULT_MAX_PER_DAY))
if cap < 1 or not _begin_attempt(cap, force):
return None
return run_outlook(settings, surface=surface)
except Exception:
logger.warning("market outlook: unavailable", exc_info=True)
return None
+1 -2
View File
@@ -466,8 +466,7 @@ class TradingDaemon:
aggregator = SignalAggregator(config) aggregator = SignalAggregator(config)
aggregator.data_provider = self._provider aggregator.data_provider = self._provider
report = aggregator.analyze_pair(symbol=SYMBOL, report = aggregator.analyze_pair(symbol=SYMBOL)
log_source="oanda_daemon")
logger.info( logger.info(
"%s scan: signal=%s score=%.1f conf=%.0f quality=%s " "%s scan: signal=%s score=%.1f conf=%.0f quality=%s "
+1 -2
View File
@@ -30,7 +30,6 @@ def produce_signal(
primary_tf: Timeframe, primary_tf: Timeframe,
data_provider: BaseDataProvider, data_provider: BaseDataProvider,
confirmation_tfs: Optional[List[Timeframe]] = None, confirmation_tfs: Optional[List[Timeframe]] = None,
log_source: str = "dashboard",
) -> SignalReport: ) -> SignalReport:
config_factory = PAIR_PRODUCERS.get(symbol) config_factory = PAIR_PRODUCERS.get(symbol)
if config_factory is None: if config_factory is None:
@@ -42,7 +41,7 @@ def produce_signal(
config = config_factory(primary_tf, confirmation_tfs) config = config_factory(primary_tf, confirmation_tfs)
aggregator = SignalAggregator(config) aggregator = SignalAggregator(config)
aggregator.data_provider = data_provider aggregator.data_provider = data_provider
return aggregator.analyze_pair(symbol=symbol, log_source=log_source) return aggregator.analyze_pair(symbol=symbol)
from pair_signals import xauusd as _xau_mod from pair_signals import xauusd as _xau_mod
-2
View File
@@ -3,8 +3,6 @@ ForexBot2 — Tiingo data provider
================================= =================================
OHLCV candles from Tiingo's FX endpoint OHLCV candles from Tiingo's FX endpoint
(`/tiingo/fx/{ticker}/prices`, resampled mid-price bars, no volume). (`/tiingo/fx/{ticker}/prices`, resampled mid-price bars, no volume).
Tiingo is streaming-only in production: its REST candles remain available
for comparison research, but it is not a designated candle fallback.
All shared behavior (cache, validation, boundary guarantees, range All shared behavior (cache, validation, boundary guarantees, range
pagination) lives in `BaseDataProvider`; this module only maps pagination) lives in `BaseDataProvider`; this module only maps
symbols/resample frequencies, sizes the request window, and parses symbols/resample frequencies, sizes the request window, and parses
-116
View File
@@ -1,116 +0,0 @@
#!/usr/bin/env python3
"""
Scratchpad — Oanda vs Tiingo candle parity check
=================================================
Fetches the last N closed XAU/USD candles from Oanda and Tiingo for
M5/M15/H1 via the normal provider stack (make_data_provider →
fetch_ohlcv: forming-candle drop + validate/clean included, i.e. exactly
what the signal pipeline consumes) and reports how well they line up:
* last-bar timestamp per provider (feed-lag check)
* timestamp mismatches (bars present in one feed only)
* per-bar OHLC diff table for the common tail
* max/mean |diff| per column, with the Close diff also in % of price
Both feeds are mid-price (Oanda mid of MBA candles; Tiingo resampled
mid), so small diffs are expected — this measures how small.
Usage:
venv/bin/python scripts/compare_oanda_tiingo.py [--n 10]
"""
import argparse
import sys
from pathlib import Path
import pandas as pd
sys.path.insert(0, str(Path(__file__).resolve().parent.parent))
from config import PlatformConfig, Timeframe, XAUUSD
from providers import make_data_provider
TIMEFRAMES = [Timeframe.M5, Timeframe.M15, Timeframe.H1]
OHLC = ["Open", "High", "Low", "Close"]
def fetch_tail(provider, tf, n):
"""Last n closed bars from one provider, or None on failure."""
try:
df = provider.fetch_ohlcv(symbol=XAUUSD, timeframe=tf)
except Exception as e: # scratchpad: report, don't crash
print(f" !! {provider.name} {tf.value} fetch failed: {e}")
return None
return df.tail(n)
def compare_tf(tf, oanda_df, tiingo_df, n):
print(f"\n{'=' * 78}\n{tf.value} — last {n} closed bars\n{'=' * 78}")
if oanda_df is None or tiingo_df is None:
print(" skipped (a provider failed)")
return
print(f" Oanda last bar: {oanda_df.index[-1]} "
f"Tiingo last bar: {tiingo_df.index[-1]}")
common = oanda_df.index.intersection(tiingo_df.index)
only_o = oanda_df.index.difference(tiingo_df.index)
only_t = tiingo_df.index.difference(oanda_df.index)
if len(only_o):
print(f" bars only in Oanda : {[str(t) for t in only_o]}")
if len(only_t):
print(f" bars only in Tiingo: {[str(t) for t in only_t]}")
if not len(common):
print(" NO common timestamps — feeds are on different bar grids!")
return
o = oanda_df.loc[common, OHLC]
t = tiingo_df.loc[common, OHLC]
diff = o - t
print(f"\n {len(common)}/{n} bars share timestamps "
f"(window {common[0]} → {common[-1]})\n")
# Per-bar Close table — the number that matters most.
hdr = f" {'bar close (UTC)':<20}{'Oanda C':>12}{'Tiingo C':>12}{'ΔC ($)':>10}"
print(hdr + f"{'ΔO':>8}{'ΔH':>8}{'ΔL':>8}")
for ts in common:
row = (f" {str(ts):<20}{o.loc[ts, 'Close']:>12.2f}"
f"{t.loc[ts, 'Close']:>12.2f}{diff.loc[ts, 'Close']:>10.3f}")
row += (f"{diff.loc[ts, 'Open']:>8.3f}{diff.loc[ts, 'High']:>8.3f}"
f"{diff.loc[ts, 'Low']:>8.3f}")
print(row)
print("\n summary over common bars:")
for col in OHLC:
d = diff[col].abs()
print(f" {col:<6} max |Δ| = {d.max():8.4f} mean |Δ| = {d.mean():8.4f}")
px = o["Close"].iloc[-1]
dc = diff["Close"].abs()
print(f" Close Δ as % of price: max {dc.max() / px * 100:.4f}% "
f"mean {dc.mean() / px * 100:.4f}%")
exact = (diff[OHLC].abs() < 1e-9).all(axis=1).sum()
print(f" bars with identical OHLC: {exact}/{len(common)}")
def main():
ap = argparse.ArgumentParser()
ap.add_argument("--n", type=int, default=10,
help="bars per timeframe to compare (default 10)")
args = ap.parse_args()
cfg = PlatformConfig()
oanda = make_data_provider(cfg, "oanda")
tiingo = make_data_provider(cfg, "tiingo")
for tf in TIMEFRAMES:
o_df = fetch_tail(oanda, tf, args.n)
t_df = fetch_tail(tiingo, tf, args.n)
compare_tf(tf, o_df, t_df, args.n)
print("\nDone. Note: Oanda=mid of MBA candles, Tiingo=resampled mid — "
"sub-$1 diffs on gold are normal feed divergence, not a bug.")
if __name__ == "__main__":
sys.exit(main())
-259
View File
@@ -1,259 +0,0 @@
#!/usr/bin/env python3
"""
Phase 1 — historical candle divergence study (data source change test)
=======================================================================
Fetches the RAW closed-bar series for XAU/USD from two providers over a
multi-month window via the normal provider stack (`fetch_range`, which
paginates under the per-request bar cap and drops the forming candle),
then quantifies how the feeds diverge:
* coverage: % shared timestamps, feed-only bars, where they cluster
* OHLC |Δ| distribution (mean/median/p90/p95/p99/max, $ and bias)
* |ΔClose| vs mean ATR14 (the scale the signal engine cares about)
* threshold exceedance (how many bars differ by more than X $)
* bar-direction agreement (sign of close−open)
* close-return correlation on consecutive bars
* per-hour and per-month drift
Both feeds are mid-price (Oanda mid of MBA candles; Tiingo resampled mid;
Twelve Data its own mid). Frames are cached to parquet so re-runs are
instant; pass --no-cache to force a fresh fetch.
Usage:
venv/bin/python scripts/feed_divergence_study.py --a oanda --b tiingo \
--tfs M15,H1,H4 --days 365
"""
import argparse
import sys
import time
from pathlib import Path
import numpy as np
import pandas as pd
sys.path.insert(0, str(Path(__file__).resolve().parent.parent))
from config import PlatformConfig, Timeframe, XAUUSD
from providers import make_data_provider
REPO = Path(__file__).resolve().parent.parent
CACHE_DIR = REPO / "backtest" / "cache"
RESEARCH_DIR = REPO / "backtest" / "research"
_TF_LOOKUP = {t.value: t for t in Timeframe}
_TF_LOOKUP.update({
"M1": Timeframe.M1, "M5": Timeframe.M5, "M15": Timeframe.M15,
"M30": Timeframe.M30, "H1": Timeframe.H1, "H4": Timeframe.H4,
"D1": Timeframe.D1, "W1": Timeframe.W1,
})
def wilder_atr(df: pd.DataFrame, period: int = 14) -> pd.Series:
hl = df["High"] - df["Low"]
hc = (df["High"] - df["Close"].shift(1)).abs()
lc = (df["Low"] - df["Close"].shift(1)).abs()
tr = pd.concat([hl, hc, lc], axis=1).max(axis=1)
return tr.ewm(alpha=1.0 / period, adjust=False).mean()
def fmt_stats(name: str, s: pd.Series) -> str:
a = s.abs()
return (f" {name:<6} mean ${a.mean():.3f} med ${a.median():.3f} "
f"p90 ${a.quantile(0.90):.3f} p95 ${a.quantile(0.95):.3f} "
f"p99 ${a.quantile(0.99):.3f} max ${a.max():.2f} "
f"bias {s.mean():+.4f}")
def cache_path(name: str, tf: Timeframe, start: pd.Timestamp,
end: pd.Timestamp) -> Path:
return (CACHE_DIR /
f"{name}_XAU_USD_{tf.value}_{start:%Y%m%d}_{end:%Y%m%d}.parquet")
def load_or_fetch(provider, name: str, tf: Timeframe, start, end,
no_cache: bool):
cp = cache_path(name, tf, start, end)
if not no_cache and cp.exists():
df = pd.read_parquet(cp)
print(f" [{name}] loaded {len(df):,} bars from cache "
f"{cp.name}")
return df
t0 = time.time()
print(f" [{name}] fetching {tf.value} {start.date()} → {end.date()} ...")
df = provider.fetch_range(XAUUSD, tf, start, end)
print(f" [{name}] fetched {len(df):,} bars in {time.time() - t0:.1f}s")
df.to_parquet(cp)
print(f" [{name}] cached → {cp.name}")
return df
def compare(tf: Timeframe, a_name: str, b_name: str, a: pd.DataFrame,
b: pd.DataFrame):
print("\n" + "=" * 80)
print(f"{tf.value} — {a_name.upper()} vs {b_name.upper()}, XAU/USD")
print("=" * 80)
common = a.index.intersection(b.index)
a_only = a.index.difference(b.index)
b_only = b.index.difference(a.index)
print(f"\nCOVERAGE")
print(f" {a_name} bars : {len(a):>8,} head {a.index[0]} tail {a.index[-1]}")
print(f" {b_name} bars : {len(b):>8,} head {b.index[0]} tail {b.index[-1]}")
print(f" shared : {len(common):>8,} "
f"({100 * len(common) / max(len(a.index.union(b.index)), 1):.2f}% of union)")
print(f" {a_name}-only : {len(a_only):>8,} "
f"({100 * len(a_only) / max(len(a), 1):.2f}% of {a_name})")
print(f" {b_name}-only : {len(b_only):>8,} "
f"({100 * len(b_only) / max(len(b), 1):.2f}% of {b_name})")
if len(common) == 0:
print(" !! no shared timestamps — feeds on different bar grids")
return
for label, idx in ((f"{a_name}-only", a_only),
(f"{b_name}-only", b_only)):
if not len(idx):
continue
by_date = pd.Series(1, index=idx).groupby(idx.date).sum()
top = by_date.sort_values(ascending=False).head(6)
print(f" {label} clusters on: "
+ ", ".join(f"{d} ({n})" for d, n in top.items()))
x = a.loc[common]
y = b.loc[common]
atr = wilder_atr(x).dropna()
mean_price = x["Close"].mean()
print(f"\nSCALE REFERENCES ({a_name} feed)")
print(f" mean close : ${mean_price:,.2f}")
print(f" mean bar range: ${(x['High'] - x['Low']).mean():.3f}")
print(f" mean ATR14 : ${atr.mean():.3f} median ${atr.median():.3f}")
print(f"\nPER-BAR DIFFS ({a_name} − {b_name}, $)")
diffs = {}
for col in ("Open", "High", "Low", "Close"):
d = x[col] - y[col]
diffs[col] = d
print(fmt_stats(col, d))
dclose = diffs["Close"].abs()
print(f"\n |ΔClose| as % of mean ATR14 : "
f"mean {100 * dclose.mean() / atr.mean():.1f}% "
f"p95 {100 * dclose.quantile(0.95) / atr.mean():.1f}%")
print(f"\nTHRESHOLD EXCEEDANCE (|ΔClose|)")
for thr in (0.10, 0.30, 0.65, 1.00, 2.00, 5.00):
pct = 100 * (dclose > thr).mean()
print(f" > ${thr:<5.2f}: {pct:6.2f}% of bars ({(dclose > thr).sum():,})")
# direction agreement
a_dir = np.sign(x["Close"] - x["Open"])
b_dir = np.sign(y["Close"] - y["Open"])
both_nonflat = (a_dir != 0) & (b_dir != 0)
dir_agree = 100 * (a_dir[both_nonflat] == b_dir[both_nonflat]).mean()
print(f"\nAGREEMENT")
print(f" bar direction (sign of close−open) agree: {dir_agree:.2f}%"
f" (n={both_nonflat.sum():,})")
step = pd.Series(common, index=common).diff().median()
a_ret = x["Close"].pct_change()
b_ret = y["Close"].pct_change()
step_ok = pd.Series(common, index=common).diff().le(step + pd.Timedelta(seconds=1))
corr = a_ret[step_ok].corr(b_ret[step_ok])
print(f" close-return correlation (consecutive bars): {corr:.5f}")
# by-hour
hours = common.hour
rows = []
for h in range(24):
m = hours == h
if not m.any():
continue
rows.append({"hour_utc": h, "bars": int(m.sum()),
"mean_abs_dclose": dclose[m].mean(),
"p95_abs_dclose": dclose[m].quantile(0.95)})
by_hour = pd.DataFrame(rows)
print(f"\nBY UTC HOUR (mean |ΔClose|, $)")
print(by_hour.to_string(index=False,
float_format=lambda v: f"{v:.3f}"))
print(f"\nBY MONTH (mean |ΔClose|, $)")
months = dclose.groupby(common.tz_localize(None).to_period("M")).mean()
for p, v in months.items():
print(f" {p}: {v:.3f}")
return {
"tf": tf.value, "a": a_name, "b": b_name,
"a_bars": len(a), "b_bars": len(b), "common": len(common),
"a_only": len(a_only), "b_only": len(b_only),
"dir_agree_pct": round(dir_agree, 3),
"ret_corr": round(corr, 5),
"mean_abs_dclose": dclose.mean(),
"p95_abs_dclose": dclose.quantile(0.95),
"max_abs_dclose": dclose.max(),
"dclose_pct_atr_mean": 100 * dclose.mean() / atr.mean(),
}
def main():
ap = argparse.ArgumentParser()
ap.add_argument("--a", required=True)
ap.add_argument("--b", required=True)
ap.add_argument("--tfs", default="M15,H1,H4",
help="comma-separated timeframe values (default M15,H1,H4)")
ap.add_argument("--days", type=int, default=365)
ap.add_argument("--end", default=None, help="UTC end date (default now)")
ap.add_argument("--no-cache", action="store_true")
args = ap.parse_args()
tfs = []
for s in args.tfs.split(","):
s = s.strip()
tf = _TF_LOOKUP.get(s)
if tf is None:
sys.exit(f"unknown timeframe '{s}'; use one of "
f"{sorted(_TF_LOOKUP)}")
tfs.append(tf)
end = pd.Timestamp(args.end, tz="UTC") if args.end \
else pd.Timestamp.now(tz="UTC")
start = end - pd.Timedelta(days=args.days)
cfg = PlatformConfig()
pa = make_data_provider(cfg, args.a)
pb = make_data_provider(cfg, args.b)
summary = []
for tf in tfs:
print(f"\n{'─' * 80}\nWINDOW {start.date()} → {end.date()} "
f"({args.days} days), {tf.value}")
try:
a = load_or_fetch(pa, args.a, tf, start, end, args.no_cache)
except Exception as e:
print(f" !! {args.a} {tf.value} fetch failed: {e}")
continue
try:
b = load_or_fetch(pb, args.b, tf, start, end, args.no_cache)
except Exception as e:
print(f" !! {args.b} {tf.value} fetch failed: {e}")
continue
r = compare(tf, args.a, args.b, a, b)
if r:
summary.append(r)
print("\n" + "=" * 80)
print("SUMMARY TABLE")
print("=" * 80)
sdf = pd.DataFrame(summary)
print(sdf.to_string(index=False,
float_format=lambda v: f"{v:.4f}"))
out = RESEARCH_DIR / f"feed_divergence_{args.a}_vs_{args.b}.csv"
if len(sdf):
sdf.to_csv(out, index=False, float_format="%.6f")
print(f"\nSummary written to {out}")
if __name__ == "__main__":
main()
-194
View File
@@ -1,194 +0,0 @@
#!/usr/bin/env python3
"""
Phase 4 — live shadow scan (data source change test)
=====================================================
Runs TWO data providers side-by-side at every candle close (M5/M15/H1)
and logs how their signals diverge, so the feed switch can be validated
under real fetch depths / caching / boundary timing for ≥2 weeks.
Unlike the history studies, this uses the LIVE fetch path (fetch_ohlcv,
forming-candle drop, freshness checks) for both providers, but calls
`analyze_from_dataframes(use_news=False)` directly so the (provider-
independent) news/DeepSeek budget is not burned on shadow scans. The
weekend filter is applied here (skip the whole cycle), and the session/
day/hour gates run inside analyze_from_dataframes as in the backtest.
No orders, no Telegram, no signal-log writes — output is a side-by-side
CSV plus a rolling agreement tally.
Usage:
venv/bin/python scripts/shadow_scan.py --a oanda --b tiingo \
--tfs M5,M15,H1 --out logs/shadow_oanda_tiingo.csv --once
# production (2+ weeks):
nohup venv/bin/python scripts/shadow_scan.py --a oanda --b tiingo \
--out logs/shadow_oanda_tiingo.csv >> logs/shadow_scan.log 2>&1 &
CSV columns (one row per TF per boundary):
ts, tf, a_<field>, b_<field> for direction/grade/setup/tag/entry/sl/tp1,
a_price, b_price, a_spread, b_spread, a_warnings, b_warnings,
dir_agree, grade_agree, setup_agree
"""
import argparse
import csv
import sys
import time
from datetime import datetime, timezone
from pathlib import Path
import pandas as pd
sys.path.insert(0, str(Path(__file__).resolve().parent.parent))
from config import PlatformConfig, Timeframe, XAUUSD
from pair_signals.xauusd import make_config
from providers import make_data_provider
from providers.base import _is_forex_weekend
from signal_aggregator import SignalAggregator, SignalReport
_TF_LOOKUP = {"M5": Timeframe.M5, "M15": Timeframe.M15, "H1": Timeframe.H1,
"H4": Timeframe.H4}
_TF_MINUTES = {Timeframe.M5: 5, Timeframe.M15: 15, Timeframe.H1: 60,
Timeframe.H4: 240}
_CONFIRMS = {Timeframe.M5: [Timeframe.M15, Timeframe.H1],
Timeframe.M15: [Timeframe.M5, Timeframe.H1],
Timeframe.H1: [Timeframe.M15, Timeframe.H4],
Timeframe.H4: [Timeframe.H1, Timeframe.D1]}
FIELDS = ["direction", "grade", "setup", "tag", "entry", "sl", "tp1"]
def seconds_until_boundary(interval_minutes: int, buffer: int = 10) -> float:
now = datetime.now(timezone.utc)
elapsed = (now.minute % interval_minutes) * 60 + now.second \
+ now.microsecond / 1e6
return interval_minutes * 60 - elapsed + buffer
def due_timeframes(wake: datetime, tfs) -> list:
minutes = wake.hour * 60 + wake.minute
floored = minutes - (minutes % 5)
return [tf for tf in tfs if floored % _TF_MINUTES[tf] == 0]
def tag_of(report) -> str:
fb = report.bullish_reasons[0] if report.bullish_reasons else ""
fbe = report.bearish_reasons[0] if report.bearish_reasons else ""
if "XAU PULLBACK" in fb or "XAU PULLBACK" in fbe:
return "XAU_PB"
if "XAU CHART PATTERN" in fb or "XAU CHART PATTERN" in fbe:
return "XAU_CP"
if "XAU STRATEGY" in fb:
return "XAU"
return "GENERIC"
def scan(provider, symbol, tf) -> SignalReport:
confirms = _CONFIRMS[tf]
cfg = make_config(tf, confirms)
agg = SignalAggregator(cfg)
agg.data_provider = provider
dfs = {}
for t in [tf] + confirms:
dfs[t] = provider.fetch_ohlcv(symbol, t)
return agg.analyze_from_dataframes(symbol, dfs, use_news=False)
def extract(provider, report) -> dict:
s = report.trade_setup
return {
"direction": report.final_signal.value,
"grade": report.signal_quality,
"setup": 1 if s is not None else 0,
"tag": tag_of(report) if s else "",
"entry": round(s.entry_price, 3) if s else "",
"sl": round(s.stop_loss, 3) if s else "",
"tp1": round(s.take_profit_1, 3) if s else "",
"price": round(report.current_price, 3),
"spread": round(provider.get_spread(XAUUSD), 3),
"warnings": " | ".join(report.warnings)[:200],
}
def main():
ap = argparse.ArgumentParser()
ap.add_argument("--a", default="oanda")
ap.add_argument("--b", default="tiingo")
ap.add_argument("--tfs", default="M5,M15,H1")
ap.add_argument("--out", default="logs/shadow_scan.csv")
ap.add_argument("--once", action="store_true",
help="one immediate scan cycle, then exit")
args = ap.parse_args()
tfs = [_TF_LOOKUP[s.strip()] for s in args.tfs.split(",")]
pa = make_data_provider(PlatformConfig(), args.a)
pb = make_data_provider(PlatformConfig(), args.b)
out_path = Path(args.out)
out_path.parent.mkdir(parents=True, exist_ok=True)
new_file = not out_path.exists()
tally = {"cycles": 0, "dir": 0, "grade": 0, "setup": 0}
with open(out_path, "a", newline="") as fh:
w = csv.writer(fh)
if new_file:
w.writerow(["ts", "tf"] + [f"a_{x}" for x in FIELDS + ["price", "spread", "warnings"]]
+ [f"b_{x}" for x in FIELDS + ["price", "spread", "warnings"]]
+ ["dir_agree", "grade_agree", "setup_agree"])
while True:
if args.once:
wake = datetime.now(timezone.utc)
else:
wait = seconds_until_boundary(5, buffer=10)
time.sleep(wait)
wake = datetime.now(timezone.utc)
if _is_forex_weekend():
if args.once:
print("weekend — nothing to scan")
return
continue
for tf in due_timeframes(wake, tfs):
try:
ra = scan(pa, XAUUSD, tf)
except Exception as e:
print(f"[{wake:%H:%M:%S}] {args.a} {tf.value} FAILED: {e}")
continue
try:
rb = scan(pb, XAUUSD, tf)
except Exception as e:
print(f"[{wake:%H:%M:%S}] {args.b} {tf.value} FAILED: {e}")
continue
ea, eb = extract(pa, ra), extract(pb, rb)
da = ea["direction"] == eb["direction"]
ga = ea["grade"] == eb["grade"]
sa = ea["setup"] == eb["setup"]
tally["cycles"] += 1
tally["dir"] += da; tally["grade"] += ga; tally["setup"] += sa
row = [wake.isoformat(), tf.value]
row += [ea[x] for x in FIELDS + ["price", "spread", "warnings"]]
row += [eb[x] for x in FIELDS + ["price", "spread", "warnings"]]
row += [int(da), int(ga), int(sa)]
w.writerow(row)
fh.flush()
print(f"[{wake:%Y-%m-%d %H:%M:%S}] {tf.value:<5} "
f"{args.a}={ea['direction']:+d}/{ea['grade'] or '-':>2} "
f"{args.b}={eb['direction']:+d}/{eb['grade'] or '-':>2} "
f"dir={'OK' if da else 'FLIP'} "
f"| Δprice {abs(ea['price']-eb['price']):.2f} "
f"| agree dir {tally['dir']}/{tally['cycles']} "
f"grade {tally['grade']}/{tally['cycles']} "
f"setup {tally['setup']}/{tally['cycles']}")
if args.once:
print("shadow scan complete (--once)")
return
if __name__ == "__main__":
main()
-246
View File
@@ -1,246 +0,0 @@
#!/usr/bin/env python3
"""
Phase 2 — signal-level replay parity (data source change test)
===============================================================
Replays the REAL decision core (`SignalAggregator.analyze_from_dataframes`,
news off, use_news=False) over the same window for two providers and
compares the *decisions* bar-by-bar: direction, composite/final score,
quality grade, gate/setup presence, strategy tag, and entry/SL/TP levels.
This turns the Phase-1 price-diff stats into flip rates. Feeds are read
from the Phase-1 parquet cache (scripts/feed_divergence_study.py writes
them); pass --no-cache-fetch only if you want to fetch fresh via
fetch_range first.
Slicing matches gpu_backtest.py exactly: decision at close of bar `pos`,
each TF sliced to bars fully closed by that decision time, primary at the
shared grid, so the two feeds are compared on identical timestamps.
Usage:
venv/bin/python scripts/signal_parity_study.py --a oanda --b tiingo \
--tf M15 --stride 8 --days 365
"""
import argparse
import sys
import time
from pathlib import Path
import numpy as np
import pandas as pd
sys.path.insert(0, str(Path(__file__).resolve().parent.parent))
from config import Timeframe, XAUUSD
from pair_signals.xauusd import make_config
from signal_aggregator import SignalAggregator
REPO = Path(__file__).resolve().parent.parent
CACHE_DIR = REPO / "backtest" / "cache"
RESEARCH_DIR = REPO / "backtest" / "research"
BAR_SECONDS = {
Timeframe.M1: 60, Timeframe.M5: 300, Timeframe.M15: 900,
Timeframe.M30: 1800, Timeframe.H1: 3600, Timeframe.H4: 14400,
Timeframe.D1: 86400, Timeframe.W1: 604800,
}
_TF_LOOKUP = {t.value: t for t in Timeframe}
_TF_LOOKUP.update({"M1": Timeframe.M1, "M5": Timeframe.M5,
"M15": Timeframe.M15, "M30": Timeframe.M30,
"H1": Timeframe.H1, "H4": Timeframe.H4,
"D1": Timeframe.D1, "W1": Timeframe.W1})
_CONFIRMS = {
Timeframe.M5: [Timeframe.M15, Timeframe.H1],
Timeframe.M15: [Timeframe.M5, Timeframe.H1],
Timeframe.H1: [Timeframe.M15, Timeframe.H4],
Timeframe.H4: [Timeframe.H1, Timeframe.D1],
}
def load_cached(name: str, tf: Timeframe, start: pd.Timestamp,
end: pd.Timestamp) -> pd.DataFrame:
cp = CACHE_DIR / f"{name}_XAU_USD_{tf.value}_{start:%Y%m%d}_{end:%Y%m%d}.parquet"
if not cp.exists():
sys.exit(f"missing cache {cp}; run scripts/feed_divergence_study.py "
f"--a oanda --b {name} --tfs {tf.value} --days ... first")
return pd.read_parquet(cp)
def tag_of(report) -> str:
first_bull = report.bullish_reasons[0] if report.bullish_reasons else ""
first_bear = report.bearish_reasons[0] if report.bearish_reasons else ""
if "XAU PULLBACK" in first_bull or "XAU PULLBACK" in first_bear:
return "XAU_PB"
if "XAU CHART PATTERN" in first_bull or "XAU CHART PATTERN" in first_bear:
return "XAU_CP"
if "XAU STRATEGY" in first_bull:
return "XAU"
return "GENERIC"
def decide(agg, dfs, tss, stride, warmup) -> list:
"""Replay decisions over the shared grid, returning one row per bar."""
primary = dfs[agg.config.primary_timeframe]
dur = pd.Timedelta(seconds=BAR_SECONDS[agg.config.primary_timeframe])
# positions of sampled bars in the primary frame
primary_pos = primary.index.get_indexer(tss)
rows = []
for i in range(0, len(tss), stride):
ts = tss[i]
pos = int(primary_pos[i])
if pos < warmup:
continue
slices = {}
for tf, full in dfs.items():
cutoff = ts # decision_time = ts + dur; bars closed by ts
cut = int(full.index.searchsorted(cutoff, side="right"))
if cut <= 0:
continue
slices[tf] = full.iloc[:cut]
if agg.config.primary_timeframe not in slices:
continue
r = agg.analyze_from_dataframes(XAUUSD, slices, use_news=False)
ts_setup = r.trade_setup
rows.append({
"ts": ts,
"direction": r.final_signal.value,
"final_score": r.final_score,
"primary_score": r.primary_score,
"grade": r.signal_quality,
"has_setup": ts_setup is not None,
"tag": tag_of(r) if ts_setup else "",
"entry": ts_setup.entry_price if ts_setup else np.nan,
"sl": ts_setup.stop_loss if ts_setup else np.nan,
"tp1": ts_setup.take_profit_1 if ts_setup else np.nan,
"tp2": ts_setup.take_profit_2 if ts_setup else np.nan,
})
return rows
def main():
ap = argparse.ArgumentParser()
ap.add_argument("--a", required=True)
ap.add_argument("--b", required=True)
ap.add_argument("--tf", default="M15")
ap.add_argument("--stride", type=int, default=1,
help="sample every Nth shared bar (default 1)")
ap.add_argument("--days", type=int, default=365)
ap.add_argument("--end", default=None)
ap.add_argument("--warmup", type=int, default=300)
args = ap.parse_args()
tf = _TF_LOOKUP.get(args.tf)
if tf is None:
sys.exit(f"unknown tf {args.tf}")
confirms = _CONFIRMS[tf]
tfs = [tf] + confirms
end = pd.Timestamp(args.end, tz="UTC") if args.end \
else pd.Timestamp.now(tz="UTC")
start = end - pd.Timedelta(days=args.days)
cfg = make_config(tf, confirms)
agg = SignalAggregator(cfg)
a_dfs, b_dfs = {}, {}
for t in tfs:
a_dfs[t] = load_cached(args.a, t, start, end)
b_dfs[t] = load_cached(args.b, t, start, end)
# shared grid = primary timestamps present in both feeds
shared = a_dfs[tf].index.intersection(b_dfs[tf].index)
t0 = time.time()
print(f"replaying {len(shared)} shared {tf.value} bars "
f"(stride {args.stride} → ~{len(shared)//args.stride}) on {args.a}...")
a_rows = decide(agg, a_dfs, shared, args.stride, args.warmup)
print(f" {args.a}: {len(a_rows)} decisions in {time.time()-t0:.1f}s")
t0 = time.time()
b_rows = decide(agg, b_dfs, shared, args.stride, args.warmup)
print(f" {args.b}: {len(b_rows)} decisions in {time.time()-t0:.1f}s")
A = pd.DataFrame(a_rows).set_index("ts")
B = pd.DataFrame(b_rows).set_index("ts")
common = A.index.intersection(B.index)
A, B = A.loc[common], B.loc[common]
n = len(common)
print(f"\n{'=' * 78}")
print(f"SIGNAL-LEVEL PARITY — {tf.value} — {args.a} vs {args.b} "
f"({n:,} bars)")
print("=" * 78)
def pct(x): return f"{100 * x:.2f}%"
same_dir = (A["direction"] == B["direction"])
same_grade = (A["grade"] == B["grade"])
same_setup = (A["has_setup"] == B["has_setup"])
same_all = same_dir & same_grade & same_setup
print(f"\nIDENTITY")
print(f" same direction : {pct(same_dir.mean())}")
print(f" same grade : {pct(same_grade.mean())}")
print(f" same setup presence : {pct(same_setup.mean())}")
print(f" same dir+grade+setup : {pct(same_all.mean())}")
both_sig = (A["direction"] != 0) & (B["direction"] != 0)
flips = both_sig & (A["direction"] != B["direction"])
opp = flips & (A["direction"] == -B["direction"])
print(f"\nDIRECTION")
print(f" bars both actionable : {both_sig.sum():,} "
f"({pct(both_sig.mean())})")
print(f" direction flips : {flips.sum():,} "
f"({pct(flips.sum()/max(both_sig.sum(),1))} of actionable)")
print(f" opposite-sign flips : {opp.sum():,}")
grade_a = set(A.loc[A["has_setup"], "grade"])
print(f"\nGRADE (of {args.a} setups, grades seen: {sorted(grade_a)})")
changed = A.loc[both_sig, "grade"] != B.loc[both_sig, "grade"]
print(f" grade changed on actionable bars: "
f"{changed.sum():,} ({pct(changed.mean())})")
# score deltas on shared bars
ds = (A["final_score"] - B["final_score"]).abs()
dp = (A["primary_score"] - B["primary_score"]).abs()
print(f"\nSCORE Δ ({args.a} − {args.b})")
print(f" final_score mean {ds.mean():6.2f} p95 {ds.quantile(.95):6.2f} "
f"max {ds.max():6.2f}")
print(f" primary_score mean {dp.mean():6.2f} p95 {dp.quantile(.95):6.2f} "
f"max {dp.max():6.2f}")
# setup price diffs on bars where BOTH generated a setup
both_setup = A["has_setup"] & B["has_setup"]
if both_setup.any():
print(f"\nSETUP LEVELS (both-setup bars: {both_setup.sum():,})")
for col, label in (("entry", "entry"), ("sl", "SL"), ("tp1", "TP1"),
("tp2", "TP2")):
d = (A.loc[both_setup, col] - B.loc[both_setup, col]).abs()
print(f" {label:<6} mean ${d.mean():6.3f} p95 ${d.quantile(.95):7.3f} "
f"max ${d.max():7.2f}")
# tag disagreement on both-setup bars
if both_setup.any():
same_tag = (A.loc[both_setup, "tag"] == B.loc[both_setup, "tag"])
print(f" same strategy tag : {pct(same_tag.mean())}")
# where do flips cluster (hour)?
flip_hours = A.loc[flips].index.hour.value_counts().sort_index()
if len(flip_hours):
print(f"\nFLIPS BY UTC HOUR")
for h, c in flip_hours.items():
print(f" {h:02d}: {c}")
out = {
"tf": tf.value, "a": args.a, "b": args.b, "bars": n,
"same_dir": same_dir.mean(), "same_grade": same_grade.mean(),
"same_setup": same_setup.mean(), "same_all": same_all.mean(),
"flip_rate_actionable": flips.sum() / max(both_sig.sum(), 1),
"mean_abs_final_score_delta": ds.mean(),
"mean_abs_primary_score_delta": dp.mean(),
}
print(f"\n{out}")
if __name__ == "__main__":
main()
+2 -116
View File
@@ -21,7 +21,6 @@ from config import PlatformConfig, SignalStrength, Timeframe
from providers import BaseDataProvider, make_data_provider from providers import BaseDataProvider, make_data_provider
from engine import TechnicalEngine, TechnicalAnalysis from engine import TechnicalEngine, TechnicalAnalysis
from news_engine import NewsEngine, FundamentalContext from news_engine import NewsEngine, FundamentalContext
import signal_log
logger = logging.getLogger(__name__) logger = logging.getLogger(__name__)
@@ -56,12 +55,8 @@ class TradeSetup:
risk_pips: float risk_pips: float
reward_pips: float reward_pips: float
pip_value: float pip_value: float
# Entry archetype — selects trail params and explains the setup to # Entry archetype ("GENERIC", "XAU_PB") — selects trail params.
# advisory consumers. setup_pattern is populated only by XAU_CP.
strategy: str = "GENERIC" strategy: str = "GENERIC"
setup_pattern: Optional[str] = None
# Strategy cost assumption, not a broker/data-provider quote.
round_trip_cost: Optional[float] = None
# Trailing exit params stamped from the PRODUCING config, so per-TF # Trailing exit params stamped from the PRODUCING config, so per-TF
# factory overrides (e.g. M15 arm 0.70, M5 dist 0.85) reach every # factory overrides (e.g. M15 arm 0.70, M5 dist 0.85) reach every
# consumer. None = legacy/unstamped setup — formatters fall back to # consumer. None = legacy/unstamped setup — formatters fall back to
@@ -101,9 +96,6 @@ class SignalReport:
panel_confidence: float = 0.0 panel_confidence: float = 0.0
panel_rationale: str = "" panel_rationale: str = ""
panel_takes: List[Dict] = field(default_factory=list) panel_takes: List[Dict] = field(default_factory=list)
panel_created_at: str = ""
panel_cache_status: str = ""
panel_payload_hash: str = ""
final_signal: SignalStrength = SignalStrength.NEUTRAL final_signal: SignalStrength = SignalStrength.NEUTRAL
final_score: float = 0.0 final_score: float = 0.0
@@ -150,14 +142,8 @@ class SignalAggregator:
self, self,
symbol: Optional[str] = None, symbol: Optional[str] = None,
context: Optional[FundamentalContext] = None, context: Optional[FundamentalContext] = None,
log_source: Optional[str] = None,
) -> SignalReport: ) -> SignalReport:
"""Live-mode entry: fetch all needed timeframes then delegate. """Live-mode entry: fetch all needed timeframes then delegate."""
Every report produced here (including weekend/day-filter early
returns) is persisted to the signal-history DB via _log_signal —
fail-safe, and never reached by the backtester, which calls
analyze_from_dataframes directly (live == backtest parity)."""
if symbol is None: if symbol is None:
symbol = self.config.symbol symbol = self.config.symbol
@@ -178,7 +164,6 @@ class SignalAggregator:
report.trade_setup = None report.trade_setup = None
report.warnings.append("WEEKEND FILTER: XAU/USD market closed — no liquidity") report.warnings.append("WEEKEND FILTER: XAU/USD market closed — no liquidity")
report.action_recommendation = "NO TRADE — Weekend (market closed)" report.action_recommendation = "NO TRADE — Weekend (market closed)"
self._log_signal(report, log_source)
return report return report
if xau_cfg.allowed_weekdays and now_utc.weekday() not in xau_cfg.allowed_weekdays: if xau_cfg.allowed_weekdays and now_utc.weekday() not in xau_cfg.allowed_weekdays:
@@ -196,7 +181,6 @@ class SignalAggregator:
f"DAY FILTER: {day_names.get(now_utc.weekday(), '?')} not in allowed days ({allowed})" f"DAY FILTER: {day_names.get(now_utc.weekday(), '?')} not in allowed days ({allowed})"
) )
report.action_recommendation = f"NO TRADE — Only trading {allowed}" report.action_recommendation = f"NO TRADE — Only trading {allowed}"
self._log_signal(report, log_source)
return report return report
timeframes = [self.config.primary_timeframe] + list( timeframes = [self.config.primary_timeframe] + list(
@@ -274,34 +258,8 @@ class SignalAggregator:
f"{symbol}: live session filter errored — filter SKIPPED this scan" f"{symbol}: live session filter errored — filter SKIPPED this scan"
) )
self._log_signal(report, log_source, dfs)
return report return report
def _log_signal(
self, report: SignalReport, source: Optional[str],
dataframes: Optional[Dict[Timeframe, pd.DataFrame]] = None,
) -> None:
"""Persist the scan to the signal-history DB (signal_log.py).
Fail-safe by contract — a logging failure must never break a
live scan. The news context re-fetch rides the NewsEngine's
class-level caches, so it costs nothing within a scan and
matches the context the signal just used; if it fails, the row
is still written, just without the news JSON blobs."""
context = None
try:
context = self._get_news_engine().get_fundamental_context()
except Exception:
logger.debug("signal log: news context unavailable",
exc_info=True)
signal_log.log_signal(
report,
context=context,
source=source or "live",
timeframe=self.config.primary_timeframe.value,
dataframes=dataframes,
)
def _apply_regime_filter_live( def _apply_regime_filter_live(
self, report: SignalReport, dfs: Dict[Timeframe, pd.DataFrame] self, report: SignalReport, dfs: Dict[Timeframe, pd.DataFrame]
) -> None: ) -> None:
@@ -662,7 +620,6 @@ class SignalAggregator:
reward_pips=1.5 * sl_dist / pip_value, reward_pips=1.5 * sl_dist / pip_value,
pip_value=pip_value, pip_value=pip_value,
strategy="XAU_PB", strategy="XAU_PB",
round_trip_cost=self.config.signals.round_trip_cost,
trail_arm_r=self.config.signals.pb_trail_arm_r, trail_arm_r=self.config.signals.pb_trail_arm_r,
trail_dist_r=self.config.signals.pb_trail_dist_r, trail_dist_r=self.config.signals.pb_trail_dist_r,
) )
@@ -800,8 +757,6 @@ class SignalAggregator:
reward_pips=height / pip_value, reward_pips=height / pip_value,
pip_value=pip_value, pip_value=pip_value,
strategy="XAU_CP", strategy="XAU_CP",
setup_pattern=pat["pattern"],
round_trip_cost=self.config.signals.round_trip_cost,
trail_arm_r=self.config.signals.trail_arm_r, trail_arm_r=self.config.signals.trail_arm_r,
trail_dist_r=self.config.signals.trail_dist_r, trail_dist_r=self.config.signals.trail_dist_r,
) )
@@ -817,74 +772,8 @@ class SignalAggregator:
f"height {height / atr:.1f} ATR") f"height {height / atr:.1f} ATR")
target = report.bullish_reasons if is_buy else report.bearish_reasons target = report.bullish_reasons if is_buy else report.bearish_reasons
target.insert(0, marker) target.insert(0, marker)
self._clean_accepted_cp_warnings(report)
_reject("accepted") _reject("accepted")
@staticmethod
def _clean_accepted_cp_warnings(report: SignalReport) -> None:
"""Drop superseded composite-path warnings after XAU_CP accepts.
This is metadata-only. Pattern detection, gates, confidence,
exhaustion state, setup geometry, and signal selection are already
final when this runs.
"""
stale_prefixes = (
"SPREAD GATE:",
"EXHAUSTION BLOCK",
"EXHAUSTED TREND FILTER:",
"MTF OPPOSITION:",
"SELL FILTER: quality",
"SELL FILTER: volatility",
"SELL FILTER: regime",
"SELL FILTER: ",
"M5 FILTER: |score|",
"M5 FILTER: no MTF agreement",
"M5 FILTER: no confirmation TF",
"M5 FILTER: SELL quality",
"M5 FILTER: SELL blocked",
)
extension_prefixes = (
"TREND EXHAUSTION",
"NO PULLBACK",
"FAR FROM EMA200",
"Extended — RSI",
"Weak pullback",
)
filtered: List[str] = []
for warning in report.warnings:
if warning.startswith(stale_prefixes):
continue
if (warning.startswith("REGIME FILTER: '")
and warning.endswith("blocked for XAU")):
continue
if (warning.startswith("M5 FILTER: '")
and "volatility" in warning):
continue
filtered.append(warning)
extension_items = [
(index, warning)
for index, warning in enumerate(filtered)
if warning.startswith(extension_prefixes)
]
if not extension_items:
report.warnings[:] = filtered
return
first_index = extension_items[0][0]
priority = {prefix: rank for rank, prefix in enumerate(extension_prefixes)}
strongest = min(
(warning for _, warning in extension_items),
key=lambda warning: next(
rank for prefix, rank in priority.items()
if warning.startswith(prefix)),
)
kept = [
warning for warning in filtered
if not warning.startswith(extension_prefixes)
]
kept.insert(first_index, strongest)
report.warnings[:] = kept
# ── Public: bar-replayable entry point ───────────────────────── # ── Public: bar-replayable entry point ─────────────────────────
def analyze_from_dataframes( def analyze_from_dataframes(
@@ -1271,8 +1160,6 @@ class SignalAggregator:
risk_pips=round(sl_distance / pip_value, 1), risk_pips=round(sl_distance / pip_value, 1),
reward_pips=round(tp_distance / pip_value, 1), reward_pips=round(tp_distance / pip_value, 1),
pip_value=pip_value, pip_value=pip_value,
strategy="XAU_ATR",
round_trip_cost=self.config.signals.round_trip_cost,
trail_arm_r=self.config.signals.trail_arm_r, trail_arm_r=self.config.signals.trail_arm_r,
trail_dist_r=self.config.signals.trail_dist_r, trail_dist_r=self.config.signals.trail_dist_r,
) )
@@ -1687,7 +1574,6 @@ class SignalAggregator:
risk_pips=risk_pips, risk_pips=risk_pips,
reward_pips=reward_pips, reward_pips=reward_pips,
pip_value=pip_value, pip_value=pip_value,
round_trip_cost=self.config.signals.round_trip_cost,
trail_arm_r=self.config.signals.trail_arm_r, trail_arm_r=self.config.signals.trail_arm_r,
trail_dist_r=self.config.signals.trail_dist_r, trail_dist_r=self.config.signals.trail_dist_r,
) )
-697
View File
@@ -1,697 +0,0 @@
"""
signal_log.py — SQLite signal-history logger
============================================
Persists every live scan's SignalReport (signal, grade, price at scan
time, news at scan time) to a SQLite database so the full decision
history can be queried later.
Hooked ONLY into SignalAggregator.analyze_pair (the live entry point);
the backtester calls analyze_from_dataframes directly and therefore
never writes here (live == backtest parity — CLAUDE.md rule 4).
Panel fields arrive POST-HOC: the row is inserted during analyze_pair
(before consumers attach the advisory panel), so log_signal stashes the
rowid on the report and agent_panel.attach_panel calls
log_panel_update() to fill the verdict, takes, creation/cache provenance,
payload hash, and advisory-outcome seed on that same row. Later closed-candle
snapshots advance the hypothetical next-open/trailing outcome. A report
without a rowid (backtest replay, logging disabled) is a silent no-op.
Deployment settings come from the config.json "signal_log" block via
app_config (env vars SIGNAL_LOG_ENABLED / SIGNAL_LOG_PATH win):
"signal_log": {"enabled": true, "path": "logs/signals.db"}
Fail-safe contract: logging must NEVER break a scan. Every public entry
point catches all exceptions and downgrades them to a warning log line.
Report/context objects are duck-typed (no imports from signal_aggregator
or news_engine) so this module stays at the bottom of the import graph.
"""
import json
import logging
import math
import os
import sqlite3
import threading
from datetime import datetime, timezone
from typing import Any, Dict, List, Optional
import app_config
from advisory_outcome import (
advance_outcome,
canonical_r,
initial_outcome,
timeframe_value,
)
logger = logging.getLogger(__name__)
_SCHEMA = """
CREATE TABLE IF NOT EXISTS signals (
id INTEGER PRIMARY KEY AUTOINCREMENT,
logged_at TEXT NOT NULL,
source TEXT NOT NULL,
symbol TEXT NOT NULL,
timeframe TEXT NOT NULL,
signal TEXT NOT NULL,
score REAL,
confidence REAL,
quality TEXT,
action TEXT,
price REAL,
spread REAL,
market_regime TEXT,
volatility_regime TEXT,
mtf_json TEXT,
setup_json TEXT,
reasons_json TEXT,
warnings_json TEXT,
event_risk TEXT,
event_description TEXT,
news_sentiment REAL,
news_description TEXT,
news_summary TEXT,
events_json TEXT,
headlines_json TEXT,
indicators_json TEXT,
candles_json TEXT,
panel_verdict TEXT,
panel_confidence REAL,
panel_rationale TEXT,
panel_takes_json TEXT,
panel_created_at TEXT,
panel_cache_status TEXT,
panel_payload_hash TEXT,
advisory_outcome_status TEXT,
advisory_r_multiple REAL,
advisory_outcome_json TEXT
);
CREATE INDEX IF NOT EXISTS idx_signals_sym_tf_ts
ON signals (symbol, timeframe, logged_at);
CREATE TABLE IF NOT EXISTS outlooks (
id INTEGER PRIMARY KEY AUTOINCREMENT,
logged_at TEXT NOT NULL,
surface TEXT NOT NULL,
bias TEXT,
setup_forming TEXT,
confidence REAL,
watch TEXT,
note TEXT,
news_json TEXT,
indicators_json TEXT,
candles_json TEXT
);
"""
# Additive migration for columns introduced after the original schema.
# Idempotent — skips columns already present.
_ADDITIVE_SIGNAL_COLS = (
("panel_verdict", "TEXT"),
("panel_confidence", "REAL"),
("panel_rationale", "TEXT"),
("panel_takes_json", "TEXT"),
("indicators_json", "TEXT"),
("candles_json", "TEXT"),
("panel_created_at", "TEXT"),
("panel_cache_status", "TEXT"),
("panel_payload_hash", "TEXT"),
("advisory_outcome_status", "TEXT"),
("advisory_r_multiple", "REAL"),
("advisory_outcome_json", "TEXT"),
)
_ADDITIVE_OUTLOOK_COLS = (
("news_json", "TEXT"),
("indicators_json", "TEXT"),
("candles_json", "TEXT"),
)
_INSERT = """
INSERT INTO signals (
logged_at, source, symbol, timeframe, signal, score, confidence,
quality, action, price, spread, market_regime, volatility_regime,
mtf_json, setup_json, reasons_json, warnings_json,
event_risk, event_description, news_sentiment, news_description,
news_summary, events_json, headlines_json,
indicators_json, candles_json,
panel_verdict, panel_confidence, panel_rationale, panel_takes_json
) VALUES (?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?)
"""
_OUTLOOK_INSERT = """
INSERT INTO outlooks (
logged_at, surface, bias, setup_forming, confidence, watch, note,
news_json, indicators_json, candles_json
) VALUES (?, ?, ?, ?, ?, ?, ?, ?, ?, ?)
"""
def _finite(x: Any) -> Optional[float]:
"""float(x) or None for non-numeric / non-finite values (NaN/inf —
SQLite has no NaN and strict JSON rejects it)."""
try:
f = float(x)
except (TypeError, ValueError):
return None
return f if math.isfinite(f) else None
def _iso(ts: Any) -> Optional[str]:
if ts is None:
return None
try:
if ts.tzinfo is None:
ts = ts.replace(tzinfo=timezone.utc)
return ts.astimezone(timezone.utc).isoformat()
except (AttributeError, ValueError):
return str(ts)
def _dumps(obj: Any) -> str:
return json.dumps(obj, default=str, allow_nan=False)
# ── Serializers (duck-typed; deliberately NOT imported from ──
# ── dashboard_publisher — the dashboard's hard independence rule) ──
def _event_to_dict(ev: Any) -> Dict[str, Any]:
try:
surprise = ev.surprise
except Exception:
surprise = None
return {
"time": _iso(getattr(ev, "time", None)),
"currency": getattr(ev, "currency", None),
"impact": getattr(ev, "impact", None),
"event": getattr(ev, "event", None),
"forecast": getattr(ev, "forecast", None),
"previous": getattr(ev, "previous", None),
"actual": getattr(ev, "actual", None),
"surprise": surprise,
}
def _headline_to_dict(h: Any) -> Dict[str, Any]:
return {
"headline": getattr(h, "headline", None),
"source": getattr(h, "source", None),
"timestamp": _iso(getattr(h, "timestamp", None)),
"score": _finite(getattr(h, "sentiment_score", None)),
"score_by_currency": {
str(currency): _finite(score)
for currency, score in (
getattr(h, "sentiment_by_currency", None) or {}).items()
},
"currencies": list(getattr(h, "currencies_mentioned", None) or []),
"keywords": list(getattr(h, "keywords", None) or []),
"link": getattr(h, "link", None),
"description": getattr(h, "description", None),
"publisher": getattr(h, "publisher", None),
}
def _context_news_json(context: Any) -> "tuple[Optional[str], Optional[str]]":
"""(events_json, headlines_json) from a FundamentalContext, or
(None, None) when no context was captured."""
if context is None:
return None, None
events = {
"high_impact_warning": bool(
getattr(context, "high_impact_warning", False)),
"warning_message": getattr(context, "warning_message", "") or "",
"calendar_available": bool(
getattr(context, "calendar_available", False)),
"sentiment_by_currency": dict(
getattr(context, "news_sentiment", None) or {}),
"events_today": [
_event_to_dict(e)
for e in getattr(context, "events_today", None) or []
],
"events_upcoming_1h": [
_event_to_dict(e)
for e in getattr(context, "events_upcoming_1h", None) or []
],
}
headlines = [
_headline_to_dict(h)
for h in getattr(context, "news_headlines", None) or []
]
return _dumps(events), _dumps(headlines)
def _key_str(k: Any) -> str:
return getattr(k, "value", str(k))
def _analysis_to_dict(analysis: Any) -> Optional[Dict[str, Any]]:
"""Compact, JSON-safe snapshot of one TechnicalAnalysis."""
if analysis is None:
return None
indicators = {}
for name, indicator in (
getattr(analysis, "indicators", None) or {}).items():
if str(name).startswith("_"):
continue
signal = getattr(indicator, "signal", None)
indicators[str(name)] = {
"signal": getattr(signal, "name", str(signal)),
"value": _finite(getattr(indicator, "value", None)),
"confidence": _finite(getattr(indicator, "confidence", None)),
"weight": _finite(getattr(indicator, "weight", None)),
"description": getattr(indicator, "description", "") or "",
}
composite = getattr(analysis, "composite_signal", None)
return {
"timestamp": str(getattr(analysis, "timestamp", "")),
"composite_signal": getattr(composite, "name", str(composite)),
"composite_score": _finite(
getattr(analysis, "composite_score", None)),
"confidence": _finite(getattr(analysis, "confidence", None)),
"market_regime": getattr(analysis, "market_regime", None),
"volatility_regime": getattr(analysis, "volatility_regime", None),
"key_levels": {
str(name): _finite(value)
for name, value in (
getattr(analysis, "key_levels", None) or {}).items()
},
"pattern_signals": list(
getattr(analysis, "pattern_signals", None) or []),
"indicators": indicators,
}
def _report_indicators_json(report: Any) -> Optional[str]:
analyses = [getattr(report, "primary_analysis", None)]
analyses.extend(getattr(report, "confirmation_analyses", None) or [])
snapshot = {}
for analysis in analyses:
if analysis is None:
continue
timeframe = str(getattr(analysis, "timeframe", ""))
if timeframe:
snapshot[timeframe] = _analysis_to_dict(analysis)
return _dumps(snapshot) if snapshot else None
def _candles_json(dataframes: Any, limit: int = 14) -> Optional[str]:
"""Last closed OHLCV bars per fetched timeframe, oldest first."""
snapshot = {}
for timeframe, df in (dataframes or {}).items():
if df is None or getattr(df, "empty", True):
continue
bars = []
for timestamp, row in df.tail(limit).iterrows():
bars.append({
"timestamp": _iso(timestamp),
"open": _finite(row.get("Open")),
"high": _finite(row.get("High")),
"low": _finite(row.get("Low")),
"close": _finite(row.get("Close")),
"volume": _finite(row.get("Volume")),
})
if bars:
snapshot[_key_str(timeframe)] = bars
return _dumps(snapshot) if snapshot else None
def _report_row(
report: Any,
context: Any,
source: str,
timeframe: str,
dataframes: Any = None,
) -> tuple:
mtf_signals = getattr(report, "mtf_signals", None) or {}
mtf_json = _dumps({
"agreement": bool(getattr(report, "mtf_agreement", False)),
"signals": {_key_str(k): getattr(v, "name", str(v))
for k, v in mtf_signals.items()},
"scores": {_key_str(k): _finite(v)
for k, v in (getattr(report, "mtf_scores", None)
or {}).items()},
})
setup = getattr(report, "trade_setup", None)
setup_json = None
if setup is not None:
setup_json = _dumps({
"direction": getattr(setup, "direction", None),
"entry": _finite(getattr(setup, "entry_price", None)),
"stop_loss": _finite(getattr(setup, "stop_loss", None)),
"take_profit_1": _finite(getattr(setup, "take_profit_1", None)),
"take_profit_2": _finite(getattr(setup, "take_profit_2", None)),
"risk_reward": _finite(getattr(setup, "risk_reward", None)),
"risk_pips": _finite(getattr(setup, "risk_pips", None)),
"reward_pips": _finite(getattr(setup, "reward_pips", None)),
"strategy": getattr(setup, "strategy", None),
"setup_pattern": getattr(setup, "setup_pattern", None),
"round_trip_cost": _finite(
getattr(setup, "round_trip_cost", None)),
"trail_arm_r": _finite(getattr(setup, "trail_arm_r", None)),
"trail_dist_r": _finite(getattr(setup, "trail_dist_r", None)),
})
reasons_json = _dumps({
"bullish": list(getattr(report, "bullish_reasons", None) or []),
"bearish": list(getattr(report, "bearish_reasons", None) or []),
})
warnings_json = _dumps(list(getattr(report, "warnings", None) or []))
events_json, headlines_json = _context_news_json(context)
signal = getattr(report, "final_signal", None)
panel_takes = getattr(report, "panel_takes", None) or []
panel_takes_json = _dumps(panel_takes) if panel_takes else None
return (
datetime.now(timezone.utc).isoformat(),
source,
str(getattr(report, "symbol", "")),
timeframe,
getattr(signal, "name", str(signal)),
_finite(getattr(report, "final_score", None)),
_finite(getattr(report, "final_confidence", None)),
getattr(report, "signal_quality", "") or "",
getattr(report, "action_recommendation", "") or "",
_finite(getattr(report, "current_price", None)),
_finite(getattr(report, "spread_estimate", None)),
getattr(report, "market_regime", "") or "",
getattr(report, "volatility_regime", "") or "",
mtf_json,
setup_json,
reasons_json,
warnings_json,
getattr(report, "event_risk", "") or "",
getattr(report, "event_description", "") or "",
_finite(getattr(report, "news_sentiment", None)),
getattr(report, "news_description", "") or "",
getattr(report, "news_summary", "") or "",
events_json,
headlines_json,
_report_indicators_json(report),
_candles_json(dataframes),
getattr(report, "panel_verdict", None),
_finite(getattr(report, "panel_confidence", None)),
getattr(report, "panel_rationale", None),
panel_takes_json,
)
class SignalLogger:
"""Thread-safe SQLite writer for signal history. One instance per
database path; safe to share across the bot's executor threads and
the daemon's thread pool."""
def __init__(self, path: str):
self._path = path
parent = os.path.dirname(path)
if parent:
os.makedirs(parent, exist_ok=True)
self._conn = sqlite3.connect(path, check_same_thread=False)
self._conn.execute("PRAGMA journal_mode=WAL")
self._conn.execute("PRAGMA busy_timeout=5000")
self._lock = threading.Lock()
with self._lock:
self._conn.executescript(_SCHEMA)
self._migrate_signal_columns()
self._migrate_outlook_columns()
@property
def path(self) -> str:
return self._path
def _migrate_signal_columns(self) -> None:
"""Add newer signal columns to pre-existing tables (idempotent)."""
existing = {r[1] for r in self._conn.execute("PRAGMA table_info(signals)")}
for name, col_type in _ADDITIVE_SIGNAL_COLS:
if name not in existing:
try:
self._conn.execute(
f"ALTER TABLE signals ADD COLUMN {name} {col_type}")
except sqlite3.OperationalError as exc:
# Another dashboard/bot process may have migrated the
# shared database after our PRAGMA snapshot.
if "duplicate column name" not in str(exc).lower():
raise
self._conn.execute(
"CREATE INDEX IF NOT EXISTS idx_signals_advisory_pending"
" ON signals (symbol, advisory_outcome_status, id)")
self._conn.commit()
def _migrate_outlook_columns(self) -> None:
"""Add snapshot columns to pre-existing outlook tables."""
existing = {
r[1] for r in self._conn.execute("PRAGMA table_info(outlooks)")
}
for name, col_type in _ADDITIVE_OUTLOOK_COLS:
if name not in existing:
try:
self._conn.execute(
f"ALTER TABLE outlooks ADD COLUMN {name} {col_type}")
except sqlite3.OperationalError as exc:
if "duplicate column name" not in str(exc).lower():
raise
self._conn.commit()
def log(
self,
report: Any,
context: Any = None,
source: str = "live",
timeframe: str = "",
dataframes: Any = None,
) -> int:
"""Insert one scan row; returns the rowid (panel-update key)."""
row = _report_row(report, context, source, timeframe, dataframes)
with self._lock:
cur = self._conn.execute(_INSERT, row)
self._conn.commit()
rowid = int(cur.lastrowid)
try:
self.resolve_pending(str(row[2]), row[25])
except Exception:
logger.warning("signal_log: advisory outcome resolver failed",
exc_info=True)
return rowid
def update_panel(
self,
rowid: int,
verdict: str,
confidence: Optional[float],
rationale: str,
takes_json: str,
*,
created_at: Optional[str] = None,
cache_status: Optional[str] = None,
payload_hash: Optional[str] = None,
) -> None:
"""Fill panel provenance and initialize its advisory outcome."""
with self._lock:
row = self._conn.execute(
"SELECT setup_json, indicators_json, timeframe,"
" advisory_outcome_status FROM signals WHERE id=?",
(rowid,),
).fetchone()
if row is None:
return
advisory_status = row[3]
advisory_json = None
if advisory_status is None:
setup = json.loads(row[0]) if row[0] else None
indicators = json.loads(row[1]) if row[1] else None
advisory_status, state = initial_outcome(
setup, indicators, row[2])
advisory_json = _dumps(state)
self._conn.execute(
"UPDATE signals SET panel_verdict=?, panel_confidence=?,"
" panel_rationale=?, panel_takes_json=?,"
" panel_created_at=COALESCE(?, panel_created_at),"
" panel_cache_status=COALESCE(?, panel_cache_status),"
" panel_payload_hash=COALESCE(?, panel_payload_hash),"
" advisory_outcome_status=COALESCE(advisory_outcome_status, ?),"
" advisory_outcome_json=COALESCE(advisory_outcome_json, ?)"
" WHERE id=?",
(verdict, confidence, rationale, takes_json,
created_at, cache_status, payload_hash,
advisory_status, advisory_json, rowid))
self._conn.commit()
def resolve_pending(
self, symbol: str, candles_json: Optional[str], limit: int = 100
) -> int:
"""Advance pending advisory outcomes from a new candle snapshot."""
if not candles_json:
return 0
snapshots = json.loads(candles_json)
if not isinstance(snapshots, dict):
return 0
changed = 0
with self._lock:
self._conn.execute("BEGIN IMMEDIATE")
try:
rows = self._conn.execute(
"SELECT id, timeframe, advisory_outcome_json FROM signals"
" WHERE symbol=? AND advisory_outcome_status='PENDING'"
" ORDER BY id",
(symbol,),
).fetchall()
eligible = 0
for rowid, timeframe, state_json in rows:
bars = timeframe_value(snapshots, timeframe)
if not bars or not state_json:
continue
if eligible >= max(1, int(limit)):
break
eligible += 1
state = json.loads(state_json)
before = _dumps(state)
state = advance_outcome(state, bars)
after = _dumps(state)
if after == before:
continue
status = state.get("status", "UNRESOLVABLE")
self._conn.execute(
"UPDATE signals SET advisory_outcome_status=?,"
" advisory_r_multiple=?, advisory_outcome_json=?"
" WHERE id=? AND advisory_outcome_status='PENDING'",
(status, canonical_r(state), after, rowid),
)
changed += 1
self._conn.commit()
except Exception:
self._conn.rollback()
raise
return changed
def log_outlook(self, outlook: Any, surface: str,
payload: Any = None) -> int:
"""Insert one advisory market outlook; returns its rowid."""
payload = payload if isinstance(payload, dict) else {}
technical = {
key: payload.get(key)
for key in ("indicators", "market_regime",
"volatility_regime", "key_levels")
if key in payload
}
row = (
datetime.now(timezone.utc).isoformat(),
surface,
getattr(outlook, "bias", None),
getattr(outlook, "setup_forming", None),
_finite(getattr(outlook, "confidence", None)),
getattr(outlook, "watch", None),
getattr(outlook, "note", None),
_dumps(payload.get("news")) if "news" in payload else None,
_dumps(technical) if technical else None,
_dumps(payload.get("candles")) if "candles" in payload else None,
)
with self._lock:
cur = self._conn.execute(_OUTLOOK_INSERT, row)
self._conn.commit()
return int(cur.lastrowid)
def close(self) -> None:
with self._lock:
self._conn.close()
# ── Module-level entry point (config-driven, fail-safe) ───────────────
_loggers: Dict[str, SignalLogger] = {}
_loggers_lock = threading.Lock()
def log_signal(
report: Any,
context: Any = None,
source: str = "live",
timeframe: str = "",
dataframes: Any = None,
) -> None:
"""Persist one scan if the signal_log config block enables it.
Stashes the rowid on the report (``_signal_log_rowid``) so a post-hoc
panel attach can update this same row. NEVER raises — a logging
failure must not break a live scan."""
try:
if not app_config.signal_log_enabled():
return
path = app_config.signal_log_path()
with _loggers_lock:
lg = _loggers.get(path)
if lg is None:
lg = SignalLogger(path)
_loggers[path] = lg
rowid = lg.log(report, context=context, source=source,
timeframe=timeframe, dataframes=dataframes)
try:
report._signal_log_rowid = rowid
except Exception:
pass # exotic report object — panel update just won't happen
except Exception:
logger.warning("signal_log: failed to persist signal",
exc_info=True)
def log_panel_update(report: Any) -> None:
"""Write post-hoc panel fields onto the row logged for this scan.
Matches on the private rowid hand-off from log_signal; a report that
was never logged (backtest replay, logging disabled, logger in
another process) is a silent no-op. NEVER raises."""
try:
rowid = getattr(report, "_signal_log_rowid", None)
if rowid is None or not app_config.signal_log_enabled():
return
verdict = getattr(report, "panel_verdict", "") or ""
if not verdict:
return
path = app_config.signal_log_path()
with _loggers_lock:
lg = _loggers.get(path)
if lg is None:
return
lg.update_panel(
int(rowid), verdict,
_finite(getattr(report, "panel_confidence", None)),
getattr(report, "panel_rationale", "") or "",
_dumps(getattr(report, "panel_takes", None) or []),
created_at=getattr(report, "panel_created_at", "") or None,
cache_status=getattr(report, "panel_cache_status", "") or None,
payload_hash=getattr(report, "panel_payload_hash", "") or None)
except Exception:
logger.warning("signal_log: failed to persist panel verdict",
exc_info=True)
def log_outlook(outlook: Any, surface: str = "live",
payload: Any = None) -> None:
"""Persist one advisory outlook when signal logging is enabled.
Outlook persistence is deliberately independent of a signal row: the
feature runs on its own hourly cadence and must never affect a scan.
"""
try:
if not app_config.signal_log_enabled():
return
path = app_config.signal_log_path()
with _loggers_lock:
lg = _loggers.get(path)
if lg is None:
lg = SignalLogger(path)
_loggers[path] = lg
lg.log_outlook(outlook, surface, payload=payload)
except Exception:
logger.warning("signal_log: failed to persist market outlook",
exc_info=True)
def reset_loggers_for_tests() -> None:
"""Close and drop all cached loggers (test isolation)."""
with _loggers_lock:
for lg in _loggers.values():
try:
lg.close()
except Exception:
pass
_loggers.clear()
-351
View File
@@ -1,351 +0,0 @@
/* Gold Desk — XAU/USD dashboard styles
*
* Builds on Bootstrap 5.3 color modes (data-bs-theme). Only app-specific
* tokens and components live here; layout, typography, tables, badges,
* cards, alerts, modals, etc. come from Bootstrap.
*/
:root {
--xau-font-sans: "Inter", system-ui, -apple-system, "Segoe UI", sans-serif;
--xau-font-mono: "JetBrains Mono", ui-monospace, "SF Mono", Consolas, Menlo, monospace;
--xau-gold: #f5b301;
}
/* Forced dark theme (owner decision 2026-08-16): dark background, pure
* white text (no gray), vivid red/green for data accents. Loaded after
* Bootstrap so these overrides win. */
[data-bs-theme="dark"] {
--bs-body-bg: #0a0e14;
--bs-secondary-bg: #111823;
--bs-tertiary-bg: #0e141d;
--bs-body-color: #ffffff;
--bs-emphasis-color: #ffffff;
--bs-heading-color: #ffffff;
--bs-secondary-color: rgba(255, 255, 255, 0.92);
--bs-tertiary-color: rgba(255, 255, 255, 0.8);
--bs-border-color: rgba(255, 255, 255, 0.14);
--bs-card-bg: #0e141d;
--bs-card-border-color: rgba(255, 255, 255, 0.12);
--bs-modal-bg: #0e141d;
--bs-success: #22c55e;
--bs-success-rgb: 34, 197, 94;
--bs-success-text-emphasis: #4ade80;
--bs-success-bg-subtle: rgba(34, 197, 94, 0.16);
--bs-success-border-subtle: rgba(34, 197, 94, 0.45);
--bs-danger: #ef4444;
--bs-danger-rgb: 239, 68, 68;
--bs-danger-text-emphasis: #f87171;
--bs-danger-bg-subtle: rgba(239, 68, 68, 0.16);
--bs-danger-border-subtle: rgba(239, 68, 68, 0.45);
}
html, body {
font-family: var(--xau-font-sans);
letter-spacing: -0.005em;
-webkit-font-smoothing: antialiased;
background-color: var(--bs-body-bg);
color: var(--bs-body-color);
}
.num, .xau-price, .xau-symbol {
font-family: var(--xau-font-mono);
font-variant-numeric: tabular-nums;
}
.xau-symbol {
font-size: 0.95rem;
font-weight: 700;
letter-spacing: 0.08em;
color: var(--xau-gold);
}
.xau-price {
line-height: 1;
color: #ffffff;
}
.live-dot {
display: inline-block;
width: 8px;
height: 8px;
border-radius: 50%;
background: var(--bs-success);
margin-right: 5px;
animation: pulse 2.4s ease-in-out infinite;
}
.live-dot.closed {
background: var(--bs-warning);
animation: none;
}
@keyframes pulse {
50% { opacity: 0.35; }
}
/* signal-coloured accents */
.sig-buy { --xau-signal: var(--bs-success); }
.sig-sell { --xau-signal: var(--bs-danger); }
.sig-neutral { --xau-signal: var(--bs-secondary); }
.border-signal {
border-left: 3px solid var(--xau-signal, var(--bs-secondary)) !important;
}
.text-signal {
color: var(--xau-signal, var(--bs-secondary)) !important;
}
.tf-name {
font-family: var(--xau-font-mono);
font-weight: 700;
font-size: 1.1rem;
color: var(--bs-primary);
letter-spacing: 0.05em;
}
.tf-selector {
display: grid;
grid-template-columns: repeat(4, minmax(0, 1fr));
gap: 0.5rem;
margin-bottom: 0.75rem;
}
.tf-select {
min-width: 0;
padding: 0.7rem 0.8rem;
border: 1px solid var(--bs-border-color);
border-radius: var(--bs-border-radius);
background: var(--bs-tertiary-bg);
color: var(--bs-body-color);
text-align: left;
transition: border-color 0.15s ease, background-color 0.15s ease, box-shadow 0.15s ease;
}
.tf-select:hover {
border-color: rgba(var(--bs-primary-rgb), 0.65);
}
.tf-select[aria-pressed="true"] {
border-color: var(--xau-gold);
background: rgba(245, 179, 1, 0.09);
box-shadow: inset 0 0 0 1px rgba(245, 179, 1, 0.16);
}
.tf-select .tf-name {
font-size: 1rem;
}
.workspace-card {
overflow: hidden;
}
.workspace-signal {
font-family: var(--xau-font-mono);
font-size: clamp(1.55rem, 3vw, 2.35rem);
font-weight: 700;
letter-spacing: 0.03em;
line-height: 1;
}
.workspace-meta {
min-width: 0;
}
.workspace-reasons > div + div {
margin-top: 0.35rem;
}
/* score bar: zero-centered diverging with midpoint tick */
.scorebar {
display: flex;
align-items: center;
gap: 0.75rem;
}
.scorebar .gauge-container {
position: relative;
flex: 1;
min-width: 60px;
height: 10px;
border-radius: 5px;
background: var(--bs-secondary-bg);
overflow: hidden;
}
.scorebar .gauge-container::after {
content: "";
position: absolute;
left: 50%;
top: 0;
bottom: 0;
width: 2px;
background: var(--bs-secondary);
transform: translateX(-1px);
}
.scorebar .fill {
position: absolute;
top: 0;
bottom: 0;
border-radius: 4px;
width: var(--score-width);
transition: width 0.25s ease;
}
.scorebar .fill.pos {
left: 50%;
background: var(--bs-success);
}
.scorebar .fill.neg {
right: 50%;
background: var(--bs-danger);
}
.scorebar .val {
width: 58px;
text-align: right;
font-size: 0.95rem;
font-weight: 500;
}
.scorebar .lbl {
font-size: 0.8rem;
color: var(--bs-secondary);
width: 44px;
}
/* rail chips in hero */
.rail-chip {
border: 1px solid var(--bs-border-color);
border-radius: var(--bs-border-radius);
background: var(--bs-tertiary-bg);
color: var(--bs-body-color);
font-family: var(--xau-font-mono);
font-size: 0.9rem;
line-height: 1.3;
padding: 0.3rem 0.5rem;
}
.rail-chip:hover, .rail-chip[aria-current="true"] {
border-color: var(--xau-gold);
}
.rail-chip:focus-visible, .tf-select:focus-visible, .wire-item:focus-visible,
.cal-event:focus-visible {
outline: 2px solid var(--xau-gold);
outline-offset: 2px;
}
/* detail dialog sections */
.d-stat .k {
display: block;
font-size: 0.7rem;
letter-spacing: 0.08em;
text-transform: uppercase;
color: var(--bs-secondary);
font-weight: 600;
}
.d-stat .v {
font-size: 1.15rem;
font-weight: 600;
}
.d-table {
font-size: 0.9rem;
}
.detail-table {
overflow-x: auto;
}
.d-table .ind-k {
font-family: var(--xau-font-mono);
font-weight: 600;
white-space: nowrap;
}
.wbar {
display: block;
height: 4px;
border-radius: 2px;
background: var(--bs-secondary-bg);
margin-top: 3px;
}
.wbar > i {
display: block;
height: 100%;
border-radius: 2px;
background: var(--bs-primary);
}
/* news */
.wire-item {
display: flex;
width: 100%;
border: 0;
border-radius: var(--bs-border-radius);
background: transparent;
color: inherit;
cursor: pointer;
font: inherit;
text-align: start;
}
.wire-item:hover {
background: rgba(var(--bs-primary-rgb), 0.08);
}
.cal-event {
display: block;
width: 100%;
padding: 0;
border: 0;
background: transparent;
color: inherit;
text-align: inherit;
}
.cal-event:hover {
color: var(--bs-primary);
text-decoration: underline;
}
.wire-score {
font-family: var(--xau-font-mono);
font-size: 0.85rem;
font-weight: 700;
padding: 2px 7px;
border-radius: 5px;
white-space: nowrap;
}
/* calendar highlight for next upcoming event */
.cal-next-up td:first-child {
box-shadow: inset 4px 0 0 var(--bs-primary);
padding-left: 0.75rem !important;
}
/* reduced motion */
@media (prefers-reduced-motion: reduce) {
.live-dot { animation: none; }
.scorebar .fill { transition: none; }
* { transition: none !important; }
}
@media (max-width: 575.98px) {
.tf-selector {
grid-template-columns: repeat(2, minmax(0, 1fr));
}
.workspace-card .card-body {
padding: 1rem;
}
.rail-chip {
flex: 1 1 calc(50% - 0.5rem);
}
}
+1433 -117
View File
File diff suppressed because it is too large Load Diff
-953
View File
@@ -1,953 +0,0 @@
"use strict";
/* Gold Desk — XAU/USD dashboard frontend
*
* Reads snapshot.json + news.json published by dashboard_publisher.py and
* renders a responsive, keyboard-accessible dashboard. Uses Bootstrap 5 for
* layout/components/modals and a small custom CSS file for app-specific
* visuals (score bar, signal colours, mono fonts).
*
* This file is intentionally self-contained: it mirrors a few constants from
* the Python signal pipeline so the HTML does not depend on telegram_bot.
*/
const IND_ORDER = ["ema_trend", "rsi", "macd", "bollinger", "adx", "roc", "keltner", "divergence", "candlestick"];
const IND_SHORT = {
ema_trend: "EMA", rsi: "RSI", macd: "MACD", bollinger: "BB",
adx: "ADX", roc: "ROC", keltner: "KC", divergence: "Div", candlestick: "Cndl"
};
const ARROW = { STRONG_BUY: "⇈", BUY: "↑", NEUTRAL: "→", SELL: "↓", STRONG_SELL: "⇊" };
const TF_LABEL = { "5min": "M5", "15min": "M15", "1h": "H1", "4h": "H4" };
const TF_ORDER = ["5min", "15min", "1h", "4h"];
const TF_SECONDS = { "5min": 300, "15min": 900, "1h": 3600, "4h": 14400 };
const FALLBACK_POLL_MS = 60000;
const RETRY_MS = 15000;
const HIGH_IMPACT_PREF = "goldDeskHighImpactOnly";
const TIMEFRAME_PREF = "goldDeskSelectedTimeframe";
const LEVEL_LABEL = {
ema_fast: "EMA fast", ema_medium: "EMA medium", ema_slow: "EMA slow", ema_trend: "EMA trend",
bb_upper: "Bollinger upper", bb_mid: "Bollinger mid", bb_lower: "Bollinger lower",
kc_upper: "Keltner upper", kc_mid: "Keltner mid", kc_lower: "Keltner lower",
swing_high_1: "Swing high 1", swing_high_2: "Swing high 2", swing_high_3: "Swing high 3",
swing_low_1: "Swing low 1", swing_low_2: "Swing low 2", swing_low_3: "Swing low 3"
};
let snapshot = null, news = null;
let clockOffset = 0;
let lastPollOk = null;
let lastSnapText = null, lastNewsText = null;
let refetchTimer = null;
let fallbackTimer = null;
let pollInFlight = false;
let pollQueued = false;
let selectedTf = "5min";
let detailTf = null;
let ndState = null;
const aiMemo = new Map();
const aiErr = new Map();
const aiPending = new Set();
let tfModal = null;
let newsModal = null;
let tfModalOpen = false;
let newsModalOpen = false;
const $ = id => document.getElementById(id);
/* ── helpers ──────────────────────────────────────────────────── */
const esc = s => String(s ?? "").replace(/[&<>"']/g,
c => ({"&": "&amp;", "<": "&lt;", ">": "&gt;", '"': "&quot;", "'": "&#39;"}[c]));
const fmt$ = v => v == null ? "—" : Number(v).toLocaleString("en-US",
{ minimumFractionDigits: 2, maximumFractionDigits: 2 });
const sigClass = s => s?.includes("BUY") ? "buy" : s?.includes("SELL") ? "sell" : "neutral";
const sigLabel = s => (s || "NEUTRAL").replace("_", " ");
const nowServer = () => Date.now() / 1000 + clockOffset;
function localGet(key) {
try { return window.localStorage.getItem(key); } catch { return null; }
}
function localSet(key, value) {
try { window.localStorage.setItem(key, value); } catch { /* storage is optional */ }
}
function announce(message) {
const live = $("dashboard-live");
if (live) live.textContent = message;
}
function fmtCountdown(secs) {
if (secs == null || !isFinite(secs)) return "--:--";
secs = Math.max(0, Math.floor(secs));
const h = Math.floor(secs / 3600), m = Math.floor((secs % 3600) / 60), s = secs % 60;
const mm = String(m).padStart(2, "0"), ss = String(s).padStart(2, "0");
return h > 0 ? `${h}:${mm}:${ss}` : `${mm}:${ss}`;
}
function fmtAge(iso) {
if (!iso) return "";
const secs = nowServer() - Date.parse(iso) / 1000;
if (secs < 90) return `${Math.max(0, Math.round(secs))}s ago`;
if (secs < 5400) return `${Math.round(secs / 60)}m ago`;
if (secs < 172800) return `${Math.round(secs / 3600)}h ago`;
return `${Math.round(secs / 86400)}d ago`;
}
const signed = (v, dp = 2) => v == null ? "—" : (v >= 0 ? "+" : "") + Number(v).toFixed(dp);
const badgeCls = s => s === "buy" ? "bg-success-subtle text-success-emphasis border-success-subtle"
: s === "sell" ? "bg-danger-subtle text-danger-emphasis border-danger-subtle"
: "bg-secondary-subtle text-secondary-emphasis border-secondary-subtle";
/* ── theme ────────────────────────────────────────────────────── */
// Dark theme is forced site-wide (owner decision 2026-08-16): dark
// background, pure-white text, red/green data accents.
document.documentElement.setAttribute("data-bs-theme", "dark");
/* ── hero ─────────────────────────────────────────────────────── */
function renderHero() {
if (!snapshot) return;
let price = null;
for (const tf of TF_ORDER) {
const p = snapshot.timeframes[tf]?.report?.current_price;
if (p) { price = p; break; }
}
$("hero-price").textContent = price ? fmt$(price) : "—";
$("hero-rail").innerHTML = TF_ORDER.map(tf => {
const r = snapshot.timeframes[tf]?.report;
const sig = r?.signal || null;
const cls = sigClass(sig);
const arrow = sig ? (ARROW[sig] || "→") : "·";
const score = r?.score != null ? r.score.toFixed(0) : "–";
const textColor = cls === "buy" ? "text-success" : cls === "sell" ? "text-danger" : "text-secondary";
return `<button type="button" class="rail-chip" data-select-tf="${tf}"
aria-current="${tf === selectedTf ? "true" : "false"}"
title="${esc(sigLabel(sig))} (score ${esc(score)}) — select ${TF_LABEL[tf]} workspace">
<span class="tf-name">${TF_LABEL[tf]}</span>
<span class="${textColor} fw-bold">${arrow}</span>
<span>${esc(score)}</span>
</button>`;
}).join("");
const closed = snapshot.market_closed;
$("hero-status").innerHTML =
`<span class="live-dot${closed ? " closed" : ""}"></span>${closed ? "CLOSED" : "OPEN"}`;
$("banner-closed").classList.toggle("d-none", !closed);
}
/* ── market context strip ─────────────────────────────────────── */
function renderContext() {
if (!news) return;
$("context").classList.remove("d-none");
$("ctx-blurb").innerHTML = news.llm_summary
? `<strong>Market context:</strong> ${esc(news.llm_summary)}`
: `<strong>Market context:</strong> <span class="text-secondary">no summary available</span>`;
let chips = "";
for (const cur of ["XAU", "USD"]) {
const v = news.sentiment?.[cur];
if (v == null) continue;
const cls = v > 0.05 ? "text-bg-success" : v < -0.05 ? "text-bg-danger" : "text-bg-secondary";
chips += `<span class="badge ${cls}" title="Mean headline sentiment for ${cur} (−1…+1)">
${cur} news <span class="num">${v >= 0 ? "+" : ""}${v.toFixed(2)}</span></span>`;
}
let risk = "clear", riskDesc = "No significant events";
if (snapshot) {
for (const tf of TF_ORDER) {
const r = snapshot.timeframes[tf]?.report;
if (r?.event_risk && r.event_risk !== "clear") { risk = r.event_risk; riskDesc = r.event_description; break; }
if (r?.event_description) riskDesc = r.event_description;
}
}
if (news.high_impact_warning) { risk = "danger"; riskDesc = news.warning_message || riskDesc; }
const riskCls = risk === "clear" ? "text-bg-success"
: risk === "danger" ? "text-bg-danger"
: "text-bg-warning";
chips += `<span class="badge ${riskCls}" title="${esc(riskDesc)}">event risk: ${risk.toUpperCase()}</span>`;
$("ctx-chips").innerHTML = chips;
}
/* ── timeframe cards ──────────────────────────────────────────── */
function scoreBar(label, score) {
const v = Math.max(-100, Math.min(100, score ?? 0));
const width = (Math.abs(v) / 2).toFixed(1);
return `<div class="scorebar">
<span class="lbl">${label}</span>
<div class="gauge-container" aria-hidden="true">
<i class="fill ${v >= 0 ? "pos" : "neg"}" style="--score-width:${width}%"></i>
</div>
<span class="val num ${v > 0 ? "text-success" : v < 0 ? "text-danger" : ""}">${v >= 0 ? "+" : ""}${v.toFixed(1)}</span>
</div>`;
}
function setupCard(ts) {
if (!ts) return "";
const dir = ts.direction === "BUY" ? "buy" : "sell";
const dirIcon = ts.direction === "BUY" ? "bi-arrow-up-circle-fill" : "bi-arrow-down-circle-fill";
const trail = (ts.trail_arm_r != null && ts.trail_dist_r != null)
? `arm ${ts.trail_arm_r}R · dist ${ts.trail_dist_r}R` : "platform default";
return `<div class="card border-signal sig-${dir} bg-body-tertiary">
<div class="card-body py-2 px-3">
<div class="d-flex align-items-baseline gap-2 mb-2">
<span class="fw-bold text-signal sig-${dir}"><i class="bi ${dirIcon} me-1"></i>${ts.direction}</span>
<span class="small text-secondary font-monospace">${esc(ts.strategy || "")}</span>
<span class="small text-secondary ms-auto font-monospace">R:R ${(+ts.risk_reward).toFixed(1)}</span>
</div>
<div class="table-responsive"><table class="table table-sm table-borderless mb-0">
<tbody><tr><td class="text-secondary"><i class="bi bi-box-arrow-in-right me-1"></i>Entry</td><td class="num">${fmt$(ts.entry)}</td>
<td class="text-secondary ps-4"><i class="bi bi-flag-fill me-1 text-success"></i>TP1</td><td class="num text-success">${fmt$(ts.take_profit_1)}</td></tr>
<tr><td class="text-secondary"><i class="bi bi-shield-x me-1 text-danger"></i>Stop</td><td class="num text-danger">${fmt$(ts.stop_loss)}</td>
<td class="text-secondary ps-4"><i class="bi bi-flag-fill me-1 text-success"></i>TP2</td><td class="num text-success">${fmt$(ts.take_profit_2)}</td></tr>
<tr><td class="text-secondary"><i class="bi bi-signpost-split me-1"></i>Trail</td><td class="num" colspan="3">${esc(trail)}</td></tr></tbody>
</table></div>
</div>
</div>`;
}
function workspacePanel(tf) {
const entry = snapshot?.timeframes?.[tf] || {};
const r = entry.report;
if (!r) {
return `<article class="card workspace-card border-signal sig-neutral" id="workspace-panel" aria-labelledby="workspace-panel-heading">
<div class="card-body py-4 text-center text-secondary">
<h3 class="tf-name mb-2" id="workspace-panel-heading">${TF_LABEL[tf]}</h3>
${entry.status === "error"
? `<span class="badge text-bg-danger">SCAN UNAVAILABLE</span><div class="small mt-2">Showing again when the next scan succeeds.</div>`
: "Warming up — no completed scan for this timeframe yet."}
</div>
</article>`;
}
const cls = sigClass(r.signal);
const textCls = cls === "buy" ? "text-success" : cls === "sell" ? "text-danger" : "text-secondary";
const qHi = r.quality === "A+" || r.quality === "A";
const pa = r.primary_analysis;
const meta = [r.market_regime, r.volatility_regime && r.volatility_regime + " volatility",
r.mtf_agreement ? "MTF aligned" : "MTF mixed"].filter(Boolean).join(" · ");
const facts = `<div class="row row-cols-2 row-cols-md-4 g-2 mt-1">
${tile("Confidence", r.confidence != null ? Math.round(r.confidence) + "%" : "—")}
${tile("Quality", esc(r.quality || "—"), qHi ? "text-primary" : "")}
${tile("Event risk", esc((r.event_risk || "—").toUpperCase()), r.event_risk === "clear" ? "text-success" : r.event_risk ? "text-warning" : "")}
${tile("MTF", r.mtf_agreement ? "aligned" : "mixed", r.mtf_agreement ? "text-success" : "")}
</div>`;
const setup = r.trade_setup
? setupCard(r.trade_setup)
: `<div class="text-secondary small">No setup published. The signal has not cleared its setup gate.</div>`;
return `<article class="card workspace-card border-signal sig-${cls}" id="workspace-panel" aria-labelledby="workspace-panel-heading">
<div class="card-body p-3 p-lg-4">
<div class="row g-4">
<div class="col-lg-7 workspace-meta">
<div class="d-flex justify-content-between align-items-start gap-3 flex-wrap mb-3">
<div>
<div class="d-flex align-items-center gap-2 flex-wrap mb-2">
<h3 class="tf-name mb-0" id="workspace-panel-heading">${TF_LABEL[tf]} signal</h3>
<span class="badge ${badgeCls(cls)}">${esc(sigLabel(r.signal))}</span>
${r.quality ? `<span class="badge ${qHi ? "text-bg-primary" : "text-bg-secondary"}">quality ${esc(r.quality)}</span>` : ""}
${entry.status === "error" ? `<span class="badge text-bg-danger">STALE</span>` : ""}
</div>
<div class="workspace-signal ${textCls}">${esc(r.action || sigLabel(r.signal))}</div>
</div>
<button type="button" class="btn btn-sm btn-outline-primary" data-open-detail="${tf}">
Full analysis <i class="bi bi-arrow-up-right ms-1"></i>
</button>
</div>
${scoreBar("score", pa ? pa.composite_score : r.score)}
<div class="small text-secondary mt-2">${esc(meta)}</div>
${facts}
<div class="workspace-reasons mt-3 pt-3 border-top">${listBlock(r) || "<div class=\"small text-secondary\">No supporting reasons recorded on this scan.</div>"}</div>
<div class="small text-secondary mt-3">${entry.scanned_at ? "scanned " + fmtAge(entry.scanned_at) : ""} · closes in <span class="num" data-close="${entry.next_close_epoch || ""}" data-interval="${TF_SECONDS[tf]}">--:--</span></div>
</div>
<div class="col-lg-5">
<div class="small text-uppercase text-secondary fw-bold mb-2">Trade plan</div>
${setup}
</div>
</div>
</div>
</article>`;
}
function renderCards() {
if (!snapshot) return;
if (!TF_LABEL[selectedTf]) selectedTf = "5min";
const selector = TF_ORDER.map(tf => {
const entry = snapshot.timeframes[tf] || {};
const r = entry.report;
const cls = r ? sigClass(r.signal) : "neutral";
const arrow = r ? (ARROW[r.signal] || "→") : "·";
const score = r?.score != null ? signed(r.score, 0) : "—";
const signal = r ? sigLabel(r.signal) : entry.status === "error" ? "UNAVAILABLE" : "WARMING";
return `<button type="button" class="tf-select" data-select-tf="${tf}" aria-pressed="${tf === selectedTf}">
<span class="d-flex align-items-center justify-content-between gap-2">
<span class="tf-name">${TF_LABEL[tf]}</span>
<span class="${cls === "buy" ? "text-success" : cls === "sell" ? "text-danger" : "text-secondary"} fw-bold">${arrow}</span>
</span>
<span class="small ${cls === "buy" ? "text-success" : cls === "sell" ? "text-danger" : "text-secondary"}">${esc(signal)}</span>
<span class="small num float-end">${esc(score)}</span>
</button>`;
}).join("");
$("tf-grid").innerHTML = `<div class="tf-selector">${selector}</div>${workspacePanel(selectedTf)}`;
tick();
}
/* ── timeframe detail dialog ──────────────────────────────────── */
const tile = (k, v, cls = "") =>
`<div class="col"><div class="card d-stat bg-body-tertiary h-100"><div class="card-body p-2">
<span class="k">${esc(k)}</span>
<span class="v ${cls}">${v}</span>
</div></div></div>`;
const SEC_ICON = {
"At a glance": "bi-speedometer2",
"Trade setup": "bi-bullseye",
"Why this signal": "bi-lightbulb",
"Multi-timeframe confirmation": "bi-layers",
"News context": "bi-newspaper",
"AI panel (advisory)": "bi-robot",
"Scan detail": "bi-clock-history",
"Score": "bi-graph-up",
"From the source": "bi-quote",
"AI gold-impact summary": "bi-stars",
"Timing": "bi-alarm",
"Numbers": "bi-calculator"
};
const sec = (title, html) => {
if (!html) return "";
const icon = SEC_ICON[title];
return `<section class="mb-4"><h6 class="text-uppercase text-secondary small fw-bold mb-2 d-flex align-items-center gap-2">${icon ? `<i class="bi ${icon} text-primary"></i>` : ""}<span>${esc(title)}</span><span class="flex-fill border-bottom"></span></h6>${html}</section>`;
};
function indTable(src) {
const rows = IND_ORDER.map(name => {
const i = src?.[name];
const k = `<td class="ind-k">${IND_SHORT[name]}</td>`;
if (!i) return `<tr>${k}<td class="text-secondary">— absent</td>
<td class="num text-end">—</td><td class="num text-end">—</td><td class="num text-end">—</td>
<td class="text-secondary small">no signal this scan</td></tr>`;
const w = i.weight == null ? null : +i.weight;
return `<tr>${k}
<td class="fw-bold ${sigClass(i.signal) === "buy" ? "text-success" : sigClass(i.signal) === "sell" ? "text-danger" : "text-secondary"}">${esc(sigLabel(i.signal))}</td>
<td class="num text-end">${i.value != null ? (+i.value).toFixed(2) : "—"}</td>
<td class="num text-end">${i.confidence != null ? Math.round(i.confidence) : "—"}</td>
<td class="num text-end">${w != null ? w.toFixed(1) : "—"}${w != null
? `<i class="wbar"><i style="width:${Math.min(100, w / 2 * 100)}%"></i></i>` : ""}</td>
<td class="text-secondary small">${esc(i.description || "")}</td></tr>`;
}).join("");
return `<div class="detail-table"><table class="table table-sm d-table"><thead><tr>
<th>Indicator</th><th>Signal</th><th class="text-end">Value</th><th class="text-end">Conf</th>
<th class="text-end">Weight</th><th>Reading</th></tr></thead><tbody>${rows}</tbody></table></div>`;
}
function ladder(levels, price) {
if (!levels) return "";
const atr = levels.atr;
const rows = Object.keys(LEVEL_LABEL)
.filter(k => levels[k] != null)
.map(k => ({ label: LEVEL_LABEL[k], v: +levels[k] }));
if (!rows.length) return "";
if (price != null) rows.push({ here: true, v: +price });
rows.sort((a, b) => b.v - a.v);
const body = rows.map(row => {
if (row.here) return `<tr class="table-primary fw-bold"><td><i class="bi bi-geo-alt-fill me-1"></i>current price</td>
<td class="num text-end">${fmt$(row.v)}</td><td class="num text-end">—</td><td class="small text-secondary">spot</td></tr>`;
const d = price != null ? row.v - price : null;
const cls = d == null ? "" : d > 0 ? "text-danger" : "text-success";
const icon = d == null ? "" : d > 0 ? "bi-arrow-up-short" : "bi-arrow-down-short";
return `<tr>
<td class="text-secondary">${esc(row.label)}</td>
<td class="num text-end ${cls}">${fmt$(row.v)}</td>
<td class="num text-end ${cls}">${d == null ? "" : signed(d) + (atr ? ` · ${signed(d / atr, 2)} ATR` : "")}</td>
<td class="small ${cls}">${d == null ? "" : `<i class="bi ${icon}"></i>${d > 0 ? "resistance" : "support"}`}</td></tr>`;
}).join("");
return `<div class="detail-table"><table class="table table-sm ladder"><thead><tr><th>Level</th><th class="text-end">Price</th>
<th class="text-end">vs. spot</th><th>Role</th></tr></thead><tbody>${body}</tbody></table></div>`;
}
function analysisBlock(a, price) {
if (!a) return "";
const pats = a.pattern_signals?.length
? `<div class="d-flex flex-wrap gap-1 mb-2">${a.pattern_signals.map(p =>
`<span class="badge text-bg-primary bg-opacity-10">${esc(p)}</span>`).join("")}</div>` : "";
const atr = a.atr_context
? `<div class="small text-secondary mb-2">${esc(a.atr_context.description ||
"ATR " + (+a.atr_context.value).toFixed(2))}</div>` : "";
const lv = ladder(a.key_levels, price);
return indTable(a.indicators) + atr + pats +
(lv ? `<div class="small text-secondary fw-semibold mt-2 mb-1">Key levels</div>${lv}` : "");
}
function listBlock(r) {
const group = (title, arr, cls, icon) => arr?.length
? `<div class="small text-secondary fw-semibold mt-2 mb-1">${title} (${arr.length})</div>
<div class="small">${arr.map(t => `<div class="${cls}"><i class="bi ${icon} me-1"></i>${esc(t)}</div>`).join("")}</div>` : "";
return group("Warnings", r.warnings, "text-warning", "bi-exclamation-triangle-fill")
+ group("Bullish", r.bullish_reasons, "text-success", "bi-arrow-up-circle-fill")
+ group("Bearish", r.bearish_reasons, "text-danger", "bi-arrow-down-circle-fill");
}
function detailHTML(tf) {
const entry = snapshot?.timeframes?.[tf] || {};
const r = entry.report;
if (!r) return `<div class="text-center text-secondary py-4">${entry.status === "error"
? `Scan error — ${esc(entry.error || "unknown")}`
: "Warming up — no completed scan for this timeframe yet."}</div>`;
const pa = r.primary_analysis;
const price = r.current_price;
const cls = sigClass(r.signal);
const textCls = cls === "buy" ? "text-success" : cls === "sell" ? "text-danger" : "text-secondary";
const borderCls = cls === "buy" ? "border-start border-success border-4"
: cls === "sell" ? "border-start border-danger border-4"
: "border-start border-secondary border-4";
let out = `<div class="card ${borderCls} mb-3"><div class="card-body">
<div class="fw-bold fs-5 ${textCls}">${esc(r.action || sigLabel(r.signal))}</div>
${r.news_description || r.event_description
? `<div class="small text-secondary">${esc([r.event_description, r.news_description].filter(Boolean).join(" · "))}</div>`
: ""}
</div></div>`;
const ns = r.news_sentiment;
out += sec("At a glance", `<div class="row row-cols-2 row-cols-md-3 row-cols-lg-4 g-2">
${tile("Signal", esc(sigLabel(r.signal)), textCls)}
${tile("Score", signed(r.score, 1), textCls)}
${tile("Confidence", r.confidence != null ? Math.round(r.confidence) + "%" : "—")}
${tile("Quality", esc(r.quality || "—"))}
${tile("Regime", esc(r.market_regime || "—"))}
${tile("Volatility", esc(r.volatility_regime || "—"))}
${tile("Event risk", esc((r.event_risk || "—").toUpperCase()),
r.event_risk === "clear" ? "text-success" : r.event_risk ? "text-warning" : "")}
${tile("News sentiment", ns != null ? signed(ns, 2) : "—",
ns > 0.05 ? "text-success" : ns < -0.05 ? "text-danger" : "")}
${tile("MTF", r.mtf_agreement ? "aligned" : "mixed", r.mtf_agreement ? "text-success" : "")}
${tile("Primary score", signed(r.primary_score, 1), sigClass(r.primary_signal) === "buy" ? "text-success" : sigClass(r.primary_signal) === "sell" ? "text-danger" : "")}
${tile("Exhaustion", esc(r.exhaustion_count ?? "—"), r.exhaustion_count ? "text-warning" : "")}
</div>`);
out += sec("Trade setup", r.trade_setup
? setupCard(r.trade_setup)
: `<div class="text-secondary small">No setup published for this timeframe — the engine only emits entry/stop/target once a signal clears its quality gate.</div>`);
out += sec("Why this signal", listBlock(r) || `<div class="text-secondary small">No reasons recorded on this scan.</div>`);
out += sec(`${TF_LABEL[tf]} indicators (primary)`,
analysisBlock(pa, price) || indTable(r.indicators) +
(r.key_levels ? `<div class="small text-secondary fw-semibold mt-2 mb-1">Key levels</div>${ladder(r.key_levels, price)}` : ""));
const mtf = Object.entries(r.mtf_signals || {});
if (mtf.length) {
const chips = mtf.map(([k, sig]) => {
const s = r.mtf_scores?.[k];
const sc = sigClass(sig);
return `<span class="badge ${badgeCls(sc)}">${esc(TF_LABEL[k] || k)}
<span class="${sc === "buy" ? "text-success" : sc === "sell" ? "text-danger" : "text-secondary"}">${esc(sigLabel(sig))}</span>
<span class="num">${s != null ? signed(s, 1) : ""}</span></span>`;
}).join("");
const confs = (r.confirmation_analyses || []).filter(Boolean).map(a => {
const lbl = TF_LABEL[a.timeframe] || a.timeframe;
const sc = sigClass(a.composite_signal);
return `<details data-dkey="conf-${esc(a.timeframe)}">
<summary><strong>${esc(lbl)}</strong>
<span class="${sc === "buy" ? "text-success" : sc === "sell" ? "text-danger" : "text-secondary"} fw-bold">${esc(sigLabel(a.composite_signal))}</span>
<span class="text-secondary small">${signed(a.composite_score, 1)} · conf ${a.confidence != null ? Math.round(a.confidence) : "—"} · ${esc(a.market_regime || "")}</span></summary>
${analysisBlock(a, price)}</details>`;
}).join("");
out += sec("Multi-timeframe confirmation",
`<div class="d-flex flex-wrap gap-2 align-items-center mb-2">${chips}
<span class="fw-bold ${r.mtf_agreement ? "text-success" : "text-secondary"}">${r.mtf_agreement ? "✓ aligned" : "✗ mixed"}</span></div>${confs}`);
}
if (r.news_summary || r.news_description) {
out += sec("News context", `<div class="text-secondary small">
${r.news_description ? `<strong>${esc(r.news_description)}</strong>${ns != null ? ` (${signed(ns, 2)})` : ""}<br>` : ""}
${esc(r.news_summary || "")}</div>`);
}
if (r.panel_verdict) {
const vcls = r.panel_verdict === "AGREE" ? "text-success"
: r.panel_verdict === "OPPOSE" ? "text-danger" : "text-secondary";
const takes = (r.panel_takes || []).map(t =>
`<tr><td class="ind-k">${esc(t.persona)}</td>
<td>${esc(t.stance)}</td>
<td class="num text-end">${t.confidence != null ? Math.round(t.confidence) : "—"}</td>
<td class="small text-secondary">${esc(t.note || "")}</td></tr>`).join("");
out += sec("AI panel (advisory)", `<div class="text-secondary small mb-2">
<strong class="${vcls}">${esc(r.panel_verdict)}</strong> ·
conf ${Math.round(r.panel_confidence || 0)}%<br>
${esc(r.panel_rationale || "")}</div>
${takes ? `<div class="detail-table"><table class="table table-sm d-table"><tbody>${takes}</tbody></table></div>` : ""}`);
}
out += sec("Scan detail", `<div class="detail-table"><table class="table table-sm d-table"><tbody>
<tr><td class="ind-k">Instrument</td><td colspan="5">${esc(r.display_name || r.symbol || "—")}</td></tr>
<tr><td class="ind-k">Candle</td><td colspan="5" class="num">${esc(r.timestamp || "—")}</td></tr>
<tr><td class="ind-k">Scanned</td><td colspan="5">${esc(entry.scanned_at || "—")}
(${fmtAge(entry.scanned_at)}${entry.scan_seconds ? `, took ${entry.scan_seconds}s` : ""})</td></tr>
<tr><td class="ind-k">Status</td><td colspan="5">${esc(entry.status || "—")}${entry.error
? ` — <span class="text-danger">${esc(entry.error)}</span>
${entry.error_at ? `(${fmtAge(entry.error_at)})` : ""}` : ""}</td></tr>
</tbody></table></div>`);
return out;
}
function renderDetail() {
if (!detailTf || !tfModalOpen) return;
const entry = snapshot?.timeframes?.[detailTf] || {};
const r = entry.report;
const openKeys = new Set([...document.querySelectorAll("#detail-body details[open][data-dkey]")]
.map(d => d.dataset.dkey));
const scroll = $("detail-body").scrollTop;
$("detail-title").textContent = TF_LABEL[detailTf] || detailTf;
$("detail-price").textContent = r?.current_price ? fmt$(r.current_price) : "—";
$("detail-sub").innerHTML = `${snapshot?.symbol ? esc(snapshot.symbol) + " · " : ""}closes in
<span class="num" data-close="${entry.next_close_epoch || ""}" data-interval="${TF_SECONDS[detailTf]}">--:--</span>`;
$("detail-body").innerHTML = detailHTML(detailTf);
document.querySelectorAll("#detail-body details[data-dkey]").forEach(d => {
if (openKeys.has(d.dataset.dkey)) d.open = true;
});
$("detail-body").scrollTop = scroll;
tick();
}
function openDetail(tf) {
if (!TF_LABEL[tf]) return;
detailTf = tf;
tfModal.show();
renderDetail();
$("detail-body").scrollTop = 0;
$("detail-close").focus();
}
$("detail-close")?.addEventListener("click", () => tfModal.hide());
/* ── calendar ─────────────────────────────────────────────────── */
function renderCalendar() {
const body = $("cal-body");
if (!news) return;
const zone = Intl.DateTimeFormat().resolvedOptions().timeZone || "local time";
$("calendar-note").textContent = `times local (${zone}) · USD events${news.generated_at ? " · updated " + fmtAge(news.generated_at) : ""}`;
const hiOnly = $("hi-only").checked;
let events = (news.events || []);
if (hiOnly) events = events.filter(e => e.impact === "high");
if (!events.length) {
body.innerHTML = `<tr><td colspan="7" class="text-secondary text-center py-3">${
news.calendar_available ? "No matching events this week." : "Calendar unavailable — source could not be reached."}</td></tr>`;
return;
}
const now = nowServer();
let nextUpMarked = false, lastDay = "", rows = "";
for (const e of events) {
const t = Date.parse(e.time) / 1000;
const d = new Date(t * 1000);
const day = d.toLocaleDateString(undefined, { weekday: "short", month: "short", day: "numeric" });
if (day !== lastDay) {
rows += `<tr class="table-group-divider"><td colspan="7" class="text-primary small fw-bold text-uppercase">${esc(day)}</td></tr>`;
lastDay = day;
}
const past = t < now;
const nextUp = !past && !nextUpMarked && (nextUpMarked = true);
const time = d.toLocaleTimeString(undefined, { hour: "2-digit", minute: "2-digit" });
const surp = e.surprise
? `<span class="${e.surprise === "positive" ? "text-success" : e.surprise === "negative" ? "text-danger" : "text-secondary"}" title="actual ${e.surprise === "positive" ? "above" : e.surprise === "negative" ? "below" : "matches"} forecast (numeric only — not gold direction)">${e.surprise === "positive" ? "▲" : e.surprise === "negative" ? "▼" : "="}</span>` : "";
rows += `<tr class="${past ? "text-secondary" : ""}${nextUp ? " cal-next-up" : ""}">
<td class="num text-nowrap">${esc(time)}</td>
<td class="text-secondary">${esc(e.currency)}</td>
<td><span class="badge ${e.impact === "high" ? "text-bg-danger" : e.impact === "medium" ? "text-bg-warning" : "text-bg-secondary"}">${e.impact === "medium" ? "Med" : e.impact.charAt(0).toUpperCase() + e.impact.slice(1)}</span></td>
<td><button type="button" class="cal-event" data-ev-key="${esc(evKey(e))}" aria-haspopup="dialog">${esc(e.event)}</button></td>
<td class="num text-end">${esc(e.forecast ?? "—")}</td>
<td class="num text-end">${esc(e.previous ?? "—")}</td>
<td class="num text-end">${esc(e.actual ?? "—")} ${surp}</td></tr>`;
}
body.innerHTML = rows;
}
/* ── newswire ─────────────────────────────────────────────────── */
function renderWireTotal() {
const box = $("wire-total");
const scored = (news?.headlines || []).filter(h => typeof h.score === "number");
if (!scored.length) { box.classList.add("d-none"); return; }
box.classList.remove("d-none");
const sum = scored.reduce((a, h) => a + h.score, 0);
const mean = sum / scored.length;
const up = scored.filter(h => h.score > 0.05).length;
const dn = scored.filter(h => h.score < -0.05).length;
const cls = mean > 0.05 ? "text-success" : mean < -0.05 ? "text-danger" : "text-secondary";
const dir = mean > 0.05 ? "BULLISH" : mean < -0.05 ? "BEARISH" : "NEUTRAL";
const wrapCls = mean > 0.05 ? "bg-success-subtle border-success-subtle"
: mean < -0.05 ? "bg-danger-subtle border-danger-subtle"
: "bg-body-tertiary";
box.className = `card mb-3 ${wrapCls}`;
box.innerHTML = `<div class="card-body p-2">
<div class="d-flex align-items-baseline gap-2 flex-wrap">
<span class="small text-secondary text-uppercase fw-bold">Total news score</span>
<span class="display-6 fw-bold ${cls}">${signed(mean, 2)}</span>
<span class="fw-bold ${cls}">${dir}</span>
<span class="ms-auto text-end small text-secondary">${scored.length} headlines · sum ${signed(sum, 1)}<br>
${up} bullish · ${dn} bearish · ${scored.length - up - dn} neutral</span>
</div>
</div>`;
box.title = `Mean sentiment of all ${scored.length} scored headlines (−1…+1). Sum ${signed(sum, 1)}. Bullish/bearish counted at |score| > 0.05.`;
}
function renderWire() {
if (!news) return;
const items = (news.headlines || []).slice(0, 40);
$("wire-note").textContent = news.generated_at ? `updated ${fmtAge(news.generated_at)}` : "";
renderWireTotal();
if (!items.length) {
$("wire").innerHTML = `<div class="text-secondary small py-2">No recent gold/dollar headlines.</div>`;
return;
}
const item = h => {
const s = h.score;
const cls = s == null || Math.abs(s) < 0.05 ? "zero" : s > 0 ? "pos" : "neg";
const scoreCls = cls === "pos" ? "bg-success-subtle text-success"
: cls === "neg" ? "bg-danger-subtle text-danger"
: "bg-body-tertiary text-secondary";
const scoreTxt = s == null ? "·" : (s > 0 ? "+" : "") + s.toFixed(1);
let host = "";
try { host = new URL(h.source).hostname.replace(/^www\./, ""); } catch {}
const content = `<span class="wire-score ${scoreCls}" title="Headline sentiment ${esc(String(s ?? "n/a"))} (−1…+1)${h.currencies?.length ? " · " + esc(h.currencies.join(", ")) : ""}">${scoreTxt}</span>
<span class="small text-secondary flex-fill">${esc(h.headline)}</span>
<span class="small text-secondary text-nowrap">${esc(h.publisher || host)}${h.timestamp ? " · " + fmtAge(h.timestamp) : ""}</span>`;
if (!h.id) return `<div class="wire-item gap-2 align-items-baseline py-1 px-1">${content}</div>`;
return `<button type="button" class="wire-item gap-2 align-items-baseline py-1 px-1" data-news-id="${h.id}" aria-haspopup="dialog">
${content}
</button>`;
};
const head = items.slice(0, 12).map(item).join("");
const rest = items.slice(12);
const moreOpen = $("wire")?.querySelector("details.more")?.open ?? false;
$("wire").innerHTML = head + (rest.length
? `<details class="more mt-1"${moreOpen ? " open" : ""}><summary class="small text-secondary">${rest.length} more headlines…</summary>
<div class="mt-1">${rest.map(item).join("")}</div></details>` : "");
}
/* ── news / calendar detail dialog ────────────────────────────── */
const evKey = e => `${e.time}|${e.currency}|${e.event}`;
const findHeadline = id => (news?.headlines || []).find(h => h.id === id);
const findEvent = key => (news?.events || []).find(e => evKey(e) === key);
const AI_ERR_MSG = {
unknown_id: "This headline has left the live feed — no summary available.",
cap_exceeded: "Daily AI budget reached — try again tomorrow.",
no_key: "AI summaries are not configured on this server."
};
async function fetchAiSummary(id) {
aiPending.add(id);
try {
const res = await fetch("news_summary.php?id=" + encodeURIComponent(id));
const data = await res.json().catch(() => null);
if (res.ok && data?.summary) aiMemo.set(id, data);
else aiErr.set(id, data?.error || `http_${res.status}`);
} catch {
aiErr.set(id, "network");
} finally {
aiPending.delete(id);
if (ndState?.kind === "wire" && ndState.id === id) renderNewsDlg();
}
}
function scoreTiles(h) {
const cls = v => v > 0.05 ? "text-success" : v < -0.05 ? "text-danger" : "text-secondary";
let tiles = tile("Overall sentiment", signed(h.score, 2), cls(h.score));
const xau = h.score_by_currency?.XAU;
if (xau != null) tiles += tile("Gold (XAU) effect", signed(xau, 2), cls(xau));
for (const [cur, v] of Object.entries(h.score_by_currency || {})) {
if (cur !== "XAU") tiles += tile(cur, signed(v, 2), cls(v));
}
let out = `<div class="row row-cols-2 row-cols-md-3 g-2">${tiles}</div>`;
if (h.keywords?.length) {
out += `<div class="small text-secondary mt-2">Matched keywords: ${esc(h.keywords.join(", "))}</div>`;
}
out += `<div class="small text-secondary">Scores range −1 (bearish) to +1 (bullish); the gold score flips monetary-policy keywords (hawkish = dollar-bullish, gold-bearish).</div>`;
return out;
}
function aiSection(id) {
if (aiMemo.has(id)) {
const m = aiMemo.get(id);
return `<div class="text-secondary small">${esc(m.summary)}</div>` + (m.generated_at
? `<div class="small text-secondary">generated ${fmtAge(m.generated_at)}</div>` : "");
}
if (aiPending.has(id)) return `<div class="text-secondary small">Generating gold-impact summary…</div>`;
if (aiErr.has(id)) {
return `<div class="text-secondary small">${esc(AI_ERR_MSG[aiErr.get(id)] || "AI summary unavailable right now.")}</div>`;
}
return `<div class="text-secondary small">Generating gold-impact summary…</div>`;
}
function wireDlgHTML() {
const fresh = findHeadline(ndState.id);
const h = fresh || ndState.snap;
if (!h) return "";
let host = "";
try { host = new URL(h.source).hostname.replace(/^www\./, ""); } catch {}
const when = h.timestamp
? `${new Date(h.timestamp).toLocaleString(undefined, { dateStyle: "medium", timeStyle: "short" })} · ${fmtAge(h.timestamp)}` : "";
let out = `<div class="fs-5 fw-semibold mb-1">${esc(h.headline)}</div>
<div class="small text-secondary mb-3">${esc(h.publisher || host)}${when ? " · " + when : ""}</div>`;
if (!fresh) out += `<div class="small text-secondary mb-2">No longer in the live feed — showing saved copy.</div>`;
out += sec("Score", scoreTiles(h));
if (h.description) out += sec("From the source", `<div class="text-secondary small">${esc(h.description)}</div>`);
out += sec("AI gold-impact summary", aiSection(ndState.id));
if (h.link && /^https?:\/\//.test(h.link)) {
out += `<a class="btn btn-outline-primary" href="${esc(h.link)}" target="_blank" rel="noopener noreferrer">Read full article ↗</a>`;
}
return out;
}
function calDlgHTML() {
const fresh = findEvent(ndState.key);
const e = fresh || ndState.snap;
if (!e) return "";
const t = Date.parse(e.time) / 1000;
const when = new Date(e.time).toLocaleString(undefined, { dateStyle: "full", timeStyle: "short" });
const impact = `<span class="badge ${e.impact === "high" ? "text-bg-danger" : e.impact === "medium" ? "text-bg-warning" : "text-bg-secondary"}">${e.impact === "medium" ? "Med" : e.impact.charAt(0).toUpperCase() + e.impact.slice(1)}</span>`;
let out = `<div class="fs-5 fw-semibold mb-1">${esc(e.event)}</div>
<div class="small text-secondary mb-3">${esc(e.currency)} · ${when}</div>`;
const timing = t > nowServer()
? `releases in <span class="num" data-close="${t}">--:--</span>`
: `released ${fmtAge(e.time)}`;
out += sec("Timing", `<div class="text-secondary small">${impact} impact · ${timing}</div>`);
const surpTxt = { positive: "actual above forecast", negative: "actual below forecast", inline: "actual matches forecast" }[e.surprise];
out += sec("Numbers", `<div class="row row-cols-3 g-2">${
tile("Forecast", esc(e.forecast ?? "—")) +
tile("Previous", esc(e.previous ?? "—")) +
tile("Actual", esc(e.actual ?? "—"), e.surprise === "positive" ? "text-success" : e.surprise === "negative" ? "text-danger" : "")
}</div>` + (surpTxt
? `<div class="small text-secondary mt-2">${surpTxt} (numeric only — not gold direction)</div>` : ""));
return out;
}
function renderNewsDlg() {
if (!ndState || !newsModalOpen) return;
const scroll = $("nd-body").scrollTop;
$("nd-title").textContent = ndState.kind === "wire" ? "Newswire" : "Economic calendar";
$("nd-body").innerHTML = ndState.kind === "wire" ? wireDlgHTML() : calDlgHTML();
$("nd-body").scrollTop = scroll;
tick();
}
function openNewsDetail(id) {
const h = findHeadline(id);
if (!h) return;
ndState = { kind: "wire", id, snap: h };
aiErr.delete(id);
newsModal.show();
if (!aiMemo.has(id) && !aiPending.has(id)) fetchAiSummary(id);
renderNewsDlg();
$("nd-body").scrollTop = 0;
$("nd-close").focus();
}
function openCalDetail(key) {
const e = findEvent(key);
if (!e) return;
ndState = { kind: "cal", key, snap: e };
newsModal.show();
renderNewsDlg();
$("nd-body").scrollTop = 0;
$("nd-close").focus();
}
$("nd-close")?.addEventListener("click", () => newsModal.hide());
/* ── polling ──────────────────────────────────────────────────── */
async function fetchJSON(name) {
const res = await fetch(name, { cache: "no-cache" });
if (!res.ok) throw new Error(`${name}: HTTP ${res.status}`);
const dateHdr = res.headers.get("Date");
const text = await res.text();
return { text, serverNow: dateHdr ? Date.parse(dateHdr) / 1000 : null };
}
function tfFingerprint(entry) {
if (!entry) return "";
return JSON.stringify({
status: entry.status,
scanned_at: entry.scanned_at,
error: entry.error,
error_at: entry.error_at,
report: entry.report
});
}
function scheduleRefresh(delay) {
clearTimeout(refetchTimer);
refetchTimer = setTimeout(() => {
refetchTimer = null;
poll();
}, Math.max(0, delay));
}
function scheduleFallbackRefresh() {
clearTimeout(fallbackTimer);
fallbackTimer = setTimeout(() => {
fallbackTimer = null;
poll();
scheduleFallbackRefresh();
}, FALLBACK_POLL_MS);
}
function scheduleBoundaryRefresh() {
if (!snapshot) return;
const nextScan = Math.min(...TF_ORDER.map(tf =>
snapshot.timeframes[tf]?.next_scan_epoch || Infinity));
if (!isFinite(nextScan)) return;
const delay = (nextScan - nowServer() + 2) * 1000;
scheduleRefresh(delay > 0 ? delay : RETRY_MS);
}
async function poll() {
if (pollInFlight) {
pollQueued = true;
return;
}
pollInFlight = true;
try {
const [snap, nw] = await Promise.all([
fetchJSON("snapshot.json"),
fetchJSON("news.json").catch(() => null)
]);
const priorSnapshot = snapshot;
let snapChanged = false;
let selectedChanged = false;
let detailChanged = false;
if (snap.text !== lastSnapText) {
const incoming = JSON.parse(snap.text);
if (!priorSnapshot || incoming.server_time_epoch >= priorSnapshot.server_time_epoch) {
snapChanged = true;
selectedChanged = tfFingerprint(priorSnapshot?.timeframes?.[selectedTf]) !==
tfFingerprint(incoming.timeframes?.[selectedTf]);
detailChanged = detailTf != null && tfFingerprint(priorSnapshot?.timeframes?.[detailTf]) !==
tfFingerprint(incoming.timeframes?.[detailTf]);
snapshot = incoming;
lastSnapText = snap.text;
}
}
const newsChanged = nw != null && nw.text !== lastNewsText;
if (newsChanged) { news = JSON.parse(nw.text); lastNewsText = nw.text; }
clockOffset = (snap.serverNow ?? snapshot.server_time_epoch) - Date.now() / 1000;
lastPollOk = new Date();
$("banner-fail").classList.add("d-none");
if (snapChanged) {
renderHero();
if (selectedChanged) renderCards();
if (detailChanged) renderDetail();
if (selectedChanged) {
const signal = snapshot.timeframes[selectedTf]?.report?.signal;
if (signal) announce(`${TF_LABEL[selectedTf]} signal updated: ${sigLabel(signal)}.`);
}
}
if (newsChanged) {
renderCalendar();
renderWire();
renderNewsDlg();
}
if (snapChanged || newsChanged) renderContext();
tick();
scheduleBoundaryRefresh();
} catch (err) {
const last = lastPollOk ? ` Last update ${lastPollOk.toLocaleTimeString()}.` : "";
const b = $("banner-fail");
b.textContent = `Dashboard data is temporarily unavailable.${last} Retrying…`;
b.classList.remove("d-none");
scheduleRefresh(RETRY_MS);
} finally {
pollInFlight = false;
if (pollQueued) {
pollQueued = false;
poll();
}
}
}
function tick() {
if (!snapshot) return;
const now = nowServer();
document.querySelectorAll("[data-close]").forEach(el => {
const t = parseFloat(el.dataset.close);
if (!t) return;
let secs = t - now;
if (secs < 0) {
const interval = parseFloat(el.dataset.interval);
secs = interval > 0 ? ((secs % interval) + interval) % interval : 0;
}
el.textContent = fmtCountdown(secs);
});
const m5 = snapshot.timeframes["5min"]?.next_close_epoch;
if (m5) {
let secs = m5 - now;
if (secs < 0) secs = ((secs % TF_SECONDS["5min"]) + TF_SECONDS["5min"]) % TF_SECONDS["5min"];
$("hero-cd").textContent = fmtCountdown(secs);
}
const pubEpoch = snapshot.server_time_epoch;
if (pubEpoch) {
const age = now - pubEpoch;
const el = $("hero-updated");
const stale = !snapshot.market_closed && age > 900;
el.textContent = (stale ? "STALE — published " : "published ") +
fmtAge(new Date(pubEpoch * 1000).toISOString());
el.classList.toggle("text-warning", stale);
el.classList.toggle("fw-semibold", stale);
}
}
/* ── boot ─────────────────────────────────────────────────────── */
document.addEventListener("DOMContentLoaded", () => {
tfModal = new bootstrap.Modal($("tf-detail"));
newsModal = new bootstrap.Modal($("news-detail"));
const storedTf = localGet(TIMEFRAME_PREF);
if (TF_LABEL[storedTf]) selectedTf = storedTf;
$("hi-only").checked = localGet(HIGH_IMPACT_PREF) === "true";
$("tf-detail")?.addEventListener("shown.bs.modal", () => { tfModalOpen = true; renderDetail(); });
$("tf-detail")?.addEventListener("hidden.bs.modal", () => { tfModalOpen = false; detailTf = null; });
$("news-detail")?.addEventListener("shown.bs.modal", () => { newsModalOpen = true; renderNewsDlg(); });
$("news-detail")?.addEventListener("hidden.bs.modal", () => { newsModalOpen = false; ndState = null; });
const selectTf = tf => {
if (!TF_LABEL[tf]) return;
const changed = selectedTf !== tf;
selectedTf = tf;
localSet(TIMEFRAME_PREF, tf);
renderHero();
renderCards();
if (changed) announce(`${TF_LABEL[tf]} workspace selected.`);
};
$("hero-rail")?.addEventListener("click", e => {
const button = e.target.closest("[data-select-tf]");
if (button) selectTf(button.dataset.selectTf);
});
$("tf-grid")?.addEventListener("click", e => {
const detail = e.target.closest("[data-open-detail]");
if (detail) return openDetail(detail.dataset.openDetail);
const button = e.target.closest("[data-select-tf]");
if (button) selectTf(button.dataset.selectTf);
});
$("wire")?.addEventListener("click", e => {
const button = e.target.closest("[data-news-id]");
if (button) openNewsDetail(button.dataset.newsId);
});
$("cal-body")?.addEventListener("click", e => {
const button = e.target.closest("[data-ev-key]");
if (button) openCalDetail(button.dataset.evKey);
});
$("hi-only")?.addEventListener("change", () => {
localSet(HIGH_IMPACT_PREF, String($("hi-only").checked));
renderCalendar();
});
poll();
scheduleFallbackRefresh();
setInterval(tick, 1000);
});
+2 -28
View File
@@ -73,7 +73,6 @@ from app_config import (
from providers import BaseDataProvider, make_data_provider from providers import BaseDataProvider, make_data_provider
from agent_panel import attach_panel, panel_health from agent_panel import attach_panel, panel_health
from llm_client import get_llm_client from llm_client import get_llm_client
from market_outlook import maybe_run_outlook
from news_engine import FundamentalContext, NewsEngine from news_engine import FundamentalContext, NewsEngine
from pair_signals import PAIR_PRODUCERS from pair_signals import PAIR_PRODUCERS
from signal_aggregator import SignalAggregator, SignalReport from signal_aggregator import SignalAggregator, SignalReport
@@ -341,10 +340,7 @@ def format_signal_message(report: SignalReport) -> str:
f"📊 Regime: {regime_str} · Vol: {vol_str}", f"📊 Regime: {regime_str} · Vol: {vol_str}",
] ]
if report.action_recommendation: if report.action_recommendation:
# Pattern identifiers use underscores, which legacy Telegram Markdown lines.append(f"_{_escape_md(report.action_recommendation)}_")
# can misparse even when escaped inside this italicized action line.
action = report.action_recommendation.replace("_", " ")
lines.append(f"_{_escape_md(action)}_")
if report.trade_setup and sig not in (SignalStrength.NEUTRAL,): if report.trade_setup and sig not in (SignalStrength.NEUTRAL,):
ts = report.trade_setup ts = report.trade_setup
@@ -540,7 +536,7 @@ def _run_signal_sync(
config.confirmation_timeframes = confirmation_tfs config.confirmation_timeframes = confirmation_tfs
aggregator = SignalAggregator(config) aggregator = SignalAggregator(config)
aggregator.data_provider = _get_shared_dp() aggregator.data_provider = _get_shared_dp()
return aggregator.analyze_pair(symbol=symbol, log_source="telegram") return aggregator.analyze_pair(symbol=symbol)
async def run_signal( async def run_signal(
@@ -753,28 +749,6 @@ async def _scan_timeframe(
if delivered: if delivered:
_commit_alert(key, report) _commit_alert(key, report)
# This independent advisory call runs only after the H1 scan completes.
# Offloading the LLM request keeps the event loop responsive.
if tf == Timeframe.H1:
try:
outlook = await asyncio.get_running_loop().run_in_executor(
_executor, maybe_run_outlook, "telegram")
settings = get_app_config().market_outlook_settings
if (outlook is not None and settings.get("telegram", False)
and OWNER_ID is not None):
forming = {
"buy": "possible buy forming",
"sell": "possible sell forming",
"none": "no clear setup forming",
}[outlook.setup_forming]
message = (
f"🔭 AI outlook: {outlook.bias}, {forming} "
f"({outlook.confidence:.0f}%) — {_escape_md(outlook.note)}"
)
await _send_alert(bot, OWNER_ID, message)
except Exception:
logger.exception("Hourly AI market outlook failed")
# ── WebSocket level-breach monitor ────────────────────────────────── # ── WebSocket level-breach monitor ──────────────────────────────────
+19 -1273
View File
File diff suppressed because it is too large Load Diff