signal_log: outcomes for every setup, event dedup, interned news; fix over-scan, dead outlook, panel OPPOSE bias
- seed an advisory outcome on every setup row; same-setup re-logs within 60 min are DUPLICATE rows (advisory_event_id); resolver rebuilds bar streams across gaps - headlines/events interned into news_snapshots (DB 667 MB -> 262 MB); readers use signals_with_news / read_news and mode=ro (immutable=1 missed WAL rows) - post-gate truth columns: gate_reason, actionable, cleaned_warnings, scan_kind, config_fingerprint; SELL-hour gates fail closed - telegram: breach rescan reuses this bar's report; grade dropped from alert fingerprint (upgrade-only re-alert) - market_outlook: hour-bucket dedup (old rule skipped alternate hours), skip heartbeats - agent_panel: macro-anchored OPPOSE guard (opposed trades were +30R over 3 weeks) - analyze_signal_log --stored-candles/--parity; backfill + news migration scripts - tests: Module 25 (analyze_signal_log) + 27 (observability); 429 total Claude-Session: https://claude.ai/code/session_01FC4s6VCURFte4dub4xKxYL
This commit is contained in:
Harwinder Sekhon
@@ -11,7 +11,7 @@ Every numeric threshold in `config.py` and `pair_signals/xauusd.py` was backtest
|
||||
## Commands
|
||||
|
||||
```bash
|
||||
python test_all.py # QA suite: 406 tests, 24 modules, no network (mocked)
|
||||
python test_all.py # QA suite: 430 tests, 27 modules, no network (mocked)
|
||||
python main.py --pair XAUUSD --timeframe H1 --detail # CLI signal report (--json for full dump,
|
||||
# --panel to run the advisory LLM agent panel on-demand)
|
||||
python telegram_bot.py # the bot (refuses to boot without credentials)
|
||||
@@ -29,7 +29,7 @@ python backtest/gpu_backtest.py --pair XAUUSD --start 2025-01-01 --end 2025-06-0
|
||||
# 2026-08-05: NOT shipped — backtest/research/fixedtp_2026-08-05.md)
|
||||
```
|
||||
|
||||
**`test_all.py` is NOT pytest** — it uses a custom `@test("name")` decorator that runs each test immediately at import; don't grep for `def test_` or try `pytest test_all.py`. Run the whole file; it prints per-module results and exits non-zero on failure. There is no way to run a single test short of commenting modules out. Tests must stay green; every new gate/knob needs tests. When patching weekend logic in tests, patch `providers.base._is_forex_weekend` — but know there are THREE independent signal-path weekend implementations (`providers/base.py`, `signal_aggregator.py`, and an inline copy in `backtest/gpu_backtest.py:_is_xau_blocked_by_calendar`) plus the advisory-only continuity copy in `advisory_outcome.py`; patching one does not reach the others.
|
||||
**`test_all.py` is NOT pytest** — it uses a custom `@test("name")` decorator that runs each test immediately at import; don't grep for `def test_` or try `pytest test_all.py`. Run the whole file; it prints per-module results and exits non-zero on failure. There is no way to run a single test short of commenting modules out. Tests must stay green; every new gate/knob needs tests. Signal-log fixtures: use a WEEKDAY signal bar when a test exercises candle continuity — the outcome model treats weekend and 21:00 UTC gaps as scheduled, so Saturday-dated bars never produce a gap. `_candles_json` stores `df.tail(14)`, so a test snapshot longer than 14 bars silently drops its head. When patching weekend logic in tests, patch `providers.base._is_forex_weekend` — but know there are THREE independent signal-path weekend implementations (`providers/base.py`, `signal_aggregator.py`, and an inline copy in `backtest/gpu_backtest.py:_is_xau_blocked_by_calendar`) plus the advisory-only continuity copy in `advisory_outcome.py`; patching one does not reach the others.
|
||||
|
||||
Python 3.12; a `venv/` exists in-repo. Dependencies in `requirements.txt` (numpy, pandas, requests, twelvedata, python-telegram-bot ≥20, websockets).
|
||||
|
||||
@@ -58,12 +58,12 @@ Signal currency everywhere is `SignalStrength` (−2..+2). DataFrames are UTC tz
|
||||
- **`providers/`** — one class per provider subclassing `BaseDataProvider` (`providers/base.py`), which owns everything the pipeline depends on: TTL+boundary cache, forming-candle drop, freshness check, validate/clean, paginated `fetch_range` (feeds the backtester from any provider), optional tick streaming. Subclasses implement only `_fetch_raw` + symbol/interval maps. Registry + `make_data_provider(config, name=None)` in `providers/__init__.py`; config.json `data_providers.active` selects the live candle feed and the optional `data_providers.streaming` scalar selects a different provider for the tick stream. A test enforces that subclasses don't override shared base methods (`supports_streaming`/`iter_ticks` are the sanctioned streaming override points). Tiingo streams mid-price ticks (`wss://api.tiingo.com/fx`) — do not re-add spread display from its free FX feed (two-tier indicative quotes, removed 2026-07-23); the no-spread rule extends to the WS stream.
|
||||
- **`engine.py`** — `TechnicalEngine`: 9 scored indicators (ema_trend, rsi, macd, bollinger, adx, roc, keltner, divergence, candlestick) + ATR as context. Regime classification (trending_up/down, `trending` = mixed/direction-UNKNOWN, transitioning, ranging) drives per-indicator weight multipliers and regime-aware RSI reinterpretation. Compute order matters (ATR first; MACD always computed even when not a pillar — divergence needs the histogram). Private series ride on the result as `_underscore` attrs; scorers/formatters skip anything `_`-prefixed or weight-0.
|
||||
- **`signal_aggregator.py`** — MTF agreement, diminishing-decay penalty system, exhaustion hard block, quality grading, then an ordered chain of post-grade gates (session, XAU SELL quality/vol/hour, regime blocks, M5 noise gates), then three setup generators: XAU ATR (A+/A), generic swing-based, plus the M5 pullback (`XAU_PB`) and chart-pattern (`XAU_CP`, `chart_patterns.py`) entry paths. The PB/CP generators run AFTER the gate chain and force quality to "B"; the CP path deliberately accepts grade-F composites and re-checks only session/day/big-candle (via warning prefixes) plus its own SELL/vol/RSI gates — it does NOT re-check exhaustion or the M5 min-score/MTF gates. **This is validated behavior, not a bug** (2026-08-04 audit: 226/586 trades in the shipped M5 12mo book are XAU_CP, +38.9R, 104 of them exhaustion-flagged and net-positive) — changing it is a rule-1 revalidation event. `TradeSetup.strategy` selects trailing params; trail params are stamped on the setup from the *producing* config so per-TF overrides reach formatters.
|
||||
- **`pair_signals/`** — per-pair config factories. **Always use the factory (`produce_signal` / `make_config`), never bare `PlatformConfig()`** — the factory applies the config.json overlay first, then the validated per-TF deltas (M5 hour-9 block, M15 trail arm 0.70, M15 SELL-hour block, etc.; evidence in the docstring). A test enforces factory ≡ inline-config equivalence. Confirmation map: M5→[M15,H1], M15→[M5,H1], H1→[M15,H4], H4→[H1,D1].
|
||||
- **`signal_log.py` / `advisory_outcome.py`** — SQLite signal-history logger (stdlib only). Hooked into `SignalAggregator.analyze_pair` only (the live entry point — the backtester never writes here, preserving parity), so every live scan from every surface (Telegram `telegram`, CLI `cli`, Oanda daemon `oanda_daemon`, dashboard `dashboard`) lands one row in `logs/signals.db`: signal/grade/price, MTF + trade-setup + reasons JSON, full per-TF indicator snapshots, the last 14 closed OHLCV candles per fetched TF, and the news at scan time (event risk + calendar events + scored headlines JSON, re-fetched via the NewsEngine class-level caches). Config.json `signal_log` block (`enabled`, `path`; env `SIGNAL_LOG_ENABLED`/`SIGNAL_LOG_PATH`). **Fail-safe by contract** — a logging failure can never break a scan. Panel fields arrive post-hoc: `log_signal` stashes the rowid and `agent_panel.attach_panel` writes verdict + creation/cache provenance + payload hash onto the same row. A versioned advisory-only outcome then advances from later stored closed-candle snapshots using next-open reanchoring, close-only trailing, continuity checks that allow only scheduled weekend/maintenance closures, a 300-bar timeout, and both $0.30/$0.65 cost views; it is explicitly a cleaned-mid evidence model, not account execution or raw-path backtest P&L. No rowid means silent no-op.
|
||||
- **`pair_signals/`** — per-pair config factories. **Always use the factory (`produce_signal` / `make_config`), never bare `PlatformConfig()`** — the factory applies the config.json overlay first, then the validated per-TF deltas (M5 hour-9 block, M15 trail arm 0.70, M15 SELL-hour block, M5 SELL-in-transitioning block, H1 all-shorts block (2026-09-03), etc.; evidence in the docstring). A test enforces factory ≡ inline-config equivalence. Confirmation map: M5→[M15,H1], M15→[M5,H1], H1→[M15,H4], H4→[H1,D1].
|
||||
- **`signal_log.py` / `advisory_outcome.py`** — SQLite signal-history logger (stdlib only). Hooked into `SignalAggregator.analyze_pair` only (the live entry point — the backtester never writes here, preserving parity), so every live scan from every surface (Telegram `telegram`, CLI `cli`, Oanda daemon `oanda_daemon`, dashboard `dashboard`) lands one row in `logs/signals.db`: signal/grade/price, MTF + trade-setup + reasons JSON, full per-TF indicator snapshots, the last 14 closed OHLCV candles per fetched TF, and the news at scan time (event risk + calendar events + scored headlines, re-fetched via the NewsEngine class-level caches). Config.json `signal_log` block (`enabled`, `path`, `log_blocked_scans`; env `SIGNAL_LOG_ENABLED`/`SIGNAL_LOG_PATH`/`SIGNAL_LOG_BLOCKED_SCANS`). **Fail-safe by contract** — a logging failure can never break a scan. Panel fields arrive post-hoc: `log_signal` stashes the rowid and `agent_panel.attach_panel` writes verdict + creation/cache provenance + payload hash onto the same row. **Advisory outcomes (2026-09):** EVERY row with a trade setup is seeded at log time (not only panel rows); same-setup re-logs within 60 min from any surface (`setup_key` = tf|direction|entry|stop|strategy) are `DUPLICATE` rows pointing at the canonical row via `advisory_event_id`, so sum `advisory_r_multiple` over `RESOLVED` rows only. The versioned model advances from later stored closed-candle snapshots (next-open reanchoring, intrabar peak with close-only ratchet, SL-first, scheduled weekend/21:00-UTC closures allowed, 300-bar timeout, $0.30/$0.65 views); when a snapshot no longer overlaps the outcome's last bar it rebuilds the stream from every stored snapshot of that TF before giving up. It is a cleaned-mid evidence model, not execution P&L. **Post-gate truth columns:** `gate_reason` (the warning that removed the setup), `actionable` (grade ≥ C AND setup — the `action` text is written BEFORE the gate chain and can say "STRONG SELL — High-quality setup" on a D row), `cleaned_warnings_json` (what the XAU_CP acceptance stripped), `bar_range_median` (= the misnamed `spread` column: median High-Low, NOT a broker spread), `scan_kind` (`blocked` weekend/day early returns carry no news/candles), `config_fingerprint` (gate config hash — surfaces config drift between processes). **News is content-addressed:** `headlines_json`/`events_json` are interned into `news_snapshots` (they were 73% of a 667 MB DB); read through the `signals_with_news` view or `signal_log.read_news`. Readers must open with `mode=ro`, never `immutable=1` (blind to WAL rows). Tools: `analyze_signal_log.py` (`--stored-candles` DB-only replay with event/opportunity/one-position books + persona scorecard, `--parity` telegram-vs-dashboard divergence report), `scripts/backfill_signal_outcomes.py` (seed/dedup/advance legacy rows), `scripts/migrate_signal_log_news.py` (intern legacy blobs; `--vacuum` with both writers stopped).
|
||||
- **`news_engine.py`** — Forex Factory calendar (MyFXBook RSS fallback) + RSS keyword sentiment with gold-inverse keywords. Class-level caches (fresh engine per scan). Optional DeepSeek enrichment via `llm_client.py` (requests-only, no SDK): batched headline sentiment overriding keyword scores, gold-direction on release surprises, and a market-summary blurb (`SignalReport.news_summary`) — **display/warning-only, keyword fallback on any failure, inert without a key**; prompts are static module constants (DeepSeek prefix-cache pricing) and requests send `thinking: disabled` (V4-Flash reasoning otherwise blows the timeout). By XAU-tuned default, event risk and sentiment **warn but do not gate signals** (`event_risk_caps_signals=False`, `sentiment_affects_signals=False`) — the LLM does not change that.
|
||||
- **`agent_panel.py`** — ai-hedge-fund-style advisory panel: 4 DeepSeek persona calls (macro/technical/risk/contrarian, concurrent on a panel-private pool) + 1 synthesizer producing an **ADVISORY-ONLY** consensus (`AGREE/NEUTRAL/OPPOSE` + confidence + rationale) written onto `SignalReport.panel_*` fields. Attached **post-hoc** by consumers (`telegram_bot` alert path, `main --panel`, `dashboard_publisher`) — never runs inside `analyze_from_dataframes` (backtest parity, test-locked) and no gate reads its fields. Persona payloads are role-filtered: macro gets high-impact events plus only code-scored directional news; technical gets strategy/pattern, primary indicators, MTF, levels, and ATR-normalized candles; risk gets setup strategy, strategy cost assumption, trailing semantics, volatility/events/warnings (never the median candle range as spread); contrarian gets directional news, setup, timestamped latest-closed price, indicators, reasons, and warnings. Spend is bounded by exact cache, full-evidence-identity TTL reuse, and `max_panels_per_day`; changed evidence never inherits a direction-only cached verdict. Verdict provenance logs creation time, `MISS`/`EXACT_HIT`/`TTL_REUSE`, and a SHA-256 payload hash. Prompts forbid outside knowledge and outputs remain short.
|
||||
- **`market_outlook.py`** — hourly DeepSeek market brief for XAU/USD, using ATR-normalized M15/H1/H4 candles, visible indicator states, levels, news, and immutable recent signal history. It returns a written **ADVISORY-ONLY** bias/setup-forming assessment and persists it independently in `outlooks` within `logs/signals.db` together with the news, indicator/regime/level, and candle evidence sent to the model. It may be sent post-scan by Telegram or requested with `main.py --outlook`; it never imports or runs inside the signal path. A lock enforces hourly dedup plus its own UTC-day cap (default 12), and provider/news failures leave a safe partial brief while LLM/database failures degrade to no outlook.
|
||||
- **`telegram_bot.py`** — owner-locked bot + subscriber channel; candle-boundary auto-scan loops (M5/M15/H1/H4, boundary+10 s), alert quality threshold + fingerprint dedup (committed only after a successful send), 📰 LLM market-context line on alerts when available, WebSocket level-breach monitor fed by the streaming provider (`data_providers.streaming`, falling back to the active provider; auto-disabled when the resolved provider can't stream or has no API key — Oanda does not stream, Tiingo does). Rotating log: `logs/telegram.log`, 3-h rotation × 240 backups.
|
||||
- **`agent_panel.py`** — ai-hedge-fund-style advisory panel: 4 DeepSeek persona calls (macro/technical/risk/contrarian, concurrent on a panel-private pool) + 1 synthesizer producing an **ADVISORY-ONLY** consensus (`AGREE/NEUTRAL/OPPOSE` + confidence + rationale) written onto `SignalReport.panel_*` fields. Attached **post-hoc** by consumers (`telegram_bot` alert path, `main --panel`, `dashboard_publisher`) — never runs inside `analyze_from_dataframes` (backtest parity, test-locked) and no gate reads its fields. Persona payloads are role-filtered: macro gets high-impact events plus only code-scored directional news; technical gets strategy/pattern, primary indicators, MTF, levels, and ATR-normalized candles; risk gets setup strategy, strategy cost assumption, trailing semantics, volatility/events/warnings (never the median candle range as spread); contrarian gets directional news, setup, timestamped latest-closed price, indicators, reasons, and warnings. Spend is bounded by exact cache, full-evidence-identity TTL reuse, and `max_panels_per_day`; changed evidence never inherits a direction-only cached verdict (test-locked — which also means TTL reuse effectively never fires live because price/candles change every bar; the daily cap is the real spend bound). Verdict provenance logs creation time, `MISS`/`EXACT_HIT`/`TTL_REUSE`, and a SHA-256 payload hash. Prompts forbid outside knowledge and outputs remain short. **OPPOSE guard (2026-09):** the 3-week signal-log scorecard showed the contrarian/risk/technical personas oppose almost every trend entry (chair said OPPOSE 73% of the time; opposed trades made +30R, PF 2.6) while the macro persona was the only informative voice (macro=support → 88% win). `_apply_oppose_guard` therefore downgrades a chair OPPOSE to NEUTRAL unless macro opposes, or technical opposes while macro is not supportive; the synthesizer prompt states the same rule. Re-score with `analyze_signal_log.py --stored-candles` (persona scorecard) before changing it again.
|
||||
- **`market_outlook.py`** — hourly DeepSeek market brief for XAU/USD, using ATR-normalized M15/H1/H4 candles, visible indicator states, levels, news, and recent signal history (read `mode=ro`). It returns a written **ADVISORY-ONLY** bias/setup-forming assessment and persists it independently in `outlooks` within `logs/signals.db` together with the news, indicator/regime/level, and candle evidence sent to the model. It may be sent post-scan by Telegram or requested with `main.py --outlook`; it never imports or runs inside the signal path. Hourly dedup is by UTC hour BUCKET (the old `now − last < 3600` rule dropped every other hour because the H1 loop fires at boundary+10 s+scan time) plus its own UTC-day cap (default 12); every silent early return logs `market outlook: skipped (...)` and an empty LLM reply logs a warning — a dark feature must be visible in `logs/telegram.log`. Provider/news failures leave a safe partial brief while LLM/database failures degrade to no outlook.
|
||||
- **`telegram_bot.py`** — owner-locked bot + subscriber channel; candle-boundary auto-scan loops (M5/M15/H1/H4, boundary+10 s), alert quality threshold + fingerprint dedup (committed only after a successful send; the fingerprint is signal+direction — grade is NOT part of it, a grade UPGRADE re-alerts, an A↔B flicker does not), 📰 LLM market-context line on alerts when available, WebSocket level-breach monitor fed by the streaming provider (`data_providers.streaming`, falling back to the active provider; auto-disabled when the resolved provider can't stream or has no API key — Oanda does not stream, Tiingo does). The breach handler reuses this bar's M15 report (`_cached_or_scan`, keyed by closed-bar epoch) instead of rescanning — it used to log 3.5 M15 rows per candle. Rotating log: `logs/telegram.log`, 3-h rotation × 240 backups.
|
||||
- **`oanda_daemon.py` / `oanda_executor.py`** — practice-account auto-execution. Book-parity rules (see daemon docstring): no broker-side TP (trailing model — TP1/TP2 advisory), 5-bar per-TF cooldown (wall-clock approximation, so it expires over weekends where the backtester would still count bars), trail ratchets at bar close once per *observed* bar (a missed/failed cycle skips those bars entirely — the intra-gap high/low never enters the peak). Accepted divergence: one position total; orders send `positionFill=OPEN_ONLY` so a state/broker divergence cancels the order instead of silently netting. State/broker re-sync runs at boot and after any suspicious fill (`_reconcile_positions`).
|
||||
- **`dashboard_publisher.py` / `static/dashboard.html`** — web dashboard as a **static publisher** (live LAMP server): standalone daemon scans at candle closes via `produce_signal` and atomically writes `snapshot.json`/`news.json`/`dashboard.html` to `$DASHBOARD_OUT_DIR` (Apache serves them; page is public by owner decision). **Hard independence rule: never imports, and is never imported by, `telegram_bot`/`oanda_daemon`** — confirmation map and boundary math are deliberate local copies, so a web-side failure can only mean a stale page. Due-TF classification uses the RAW wake time (flooring wake−buffer skips every H1/H4 scan — 2026-08-02 bug). News panel reads `NewsEngine.get_fundamental_context()` plus the additive `get_week_events()` accessor (display-only; no signal path calls it); as a separate process it has its own news caches and DeepSeek daily cap. News/calendar rows are clickable (detail dialog); headlines carry `id` = sha1(headline) in news.json, and exactly one on-demand endpoint exists: `static/news_summary.php` (installed by `install_static`), which serves a DeepSeek gold-impact summary per headline — whitelist-gated against news.json ids, cache-forever in `ai/`, own fail-closed daily cap (150/day, file-based, separate from the publisher's), key at `/etc/gold-dashboard/deepseek.key`, never reads repo config. Deployment checklist: `deploy/README-dashboard.md`.
|
||||
- **`backtest/gpu_backtest.py`** — bar-by-bar replay through the real `analyze_from_dataframes` (news off). Realism model (do not regress): next-bar-open entries re-anchored to the raw price path, round-trip `--spread` charged per trade, trade resolution on RAW bars (cleaning applies only to signal-side data), same-bar SL+TP tie → SL first, gap fills at bar open, trail ratchet at bar close only. Indicators are precomputed once per TF (CuPy GPU or NumPy fallback — check the `Backend:` log line); replay fans out over worker processes and the 5-bar cooldown is applied in an ordered pass afterwards, so parallel and sequential books are identical. Parquet cache in `backtest/cache/` keyed by provider/symbol/TF/window. `backtest/*.csv` are gitignored pinned trade books from past campaigns — used for manual before/after comparison. `backtest/instructions_for_backtest.txt` lists what every backtest report must record.
|
||||
@@ -94,3 +94,4 @@ Signal currency everywhere is `SignalStrength` (−2..+2). DataFrames are UTC tz
|
||||
- Twelve Data is signal-incompatible with Oanda (Phase 1: about 19% M5 bar-direction disagreement and a systematic ~$1 high bias; see `backtest/research/feed_divergence_2026-08-14.md`). On an Oanda candle outage, halt auto-execution rather than switching to TD candles.
|
||||
- Tiingo is streaming-only (`data_providers.streaming: "tiingo"`), not a designated candle fallback. Oanda-Tiingo signal parity is documented in `backtest/research/signal_parity_2026-08-14.md`; Phase 4's two-week live shadow scan remains a manual operational step.
|
||||
- Human-facing docs in `docs/` (`user_guide.html`, `beginner_guide.html` + PDF builds) must be kept in sync when signal behavior changes.
|
||||
- 2026-09-03 signal-log campaign shipped two new per-TF gates (`backtest/research/signal_log_campaign_2026-09.md`): M5 `sell_blocked_regimes=["transitioning"]` and H1 `sell_min_quality=[]` (all H1 shorts blocked). Current-HEAD M5 canonical book is now `backtest/oanda_m5_12mo_065_nopb_20260903.csv` (557 / +112.21R / PF 1.56, halves +55.9/+56.4); H1 reference `backtest/oanda_h1_12mo_065_20260903.csv` (91 / +22.2R / PF 1.79; 5yr +49.1R PF 1.46; OOS +16.7R PF 1.74).
|
||||
|
||||
+53
-3
@@ -253,9 +253,13 @@ Weigh the takes into ONE consensus on whether to act on the signal:
|
||||
"AGREE" (the panel backs the trade), "OPPOSE" (the panel is against it),
|
||||
or "NEUTRAL" (genuinely split or low conviction). The risk manager and
|
||||
contrarian have intentionally defensive roles; their objections are not
|
||||
automatic vetoes. Do not count repeated versions of the same warning as
|
||||
independent evidence. Weigh role relevance, evidence quality, and
|
||||
confidence rather than averaging blindly.
|
||||
automatic vetoes: "OPPOSE" requires that the macro analyst opposes, or
|
||||
that the technical analyst opposes while the macro analyst is not
|
||||
supportive. When only the defensive roles object, or the macro analyst
|
||||
supports the trade, the most the panel can say is "NEUTRAL". Do not
|
||||
count repeated versions
|
||||
of the same warning as independent evidence. Weigh role relevance,
|
||||
evidence quality, and confidence rather than averaging blindly.
|
||||
Base the verdict and rationale ONLY on the takes given — do not add
|
||||
outside analysis.
|
||||
Confidence scale (0-100): 80-100 the panel clearly leans one way; 50-79
|
||||
@@ -674,6 +678,15 @@ def _panel_jobs(
|
||||
)
|
||||
|
||||
|
||||
# NOTE (2026-09 audit): the TTL reuse key differs from the exact key
|
||||
# only by the contrarian's bar timestamp, and every other block (price,
|
||||
# ATR-normalized candles, indicator readings, drifting entry/stop) moves
|
||||
# each closed bar — so TTL_REUSE never fired live (30/30 attaches MISS).
|
||||
# That is the documented, test-locked decision ("changed evidence never
|
||||
# inherits a direction-only cached verdict"); spend is bounded by
|
||||
# max_panels_per_day instead. Relaxing the identity is an owner call.
|
||||
|
||||
|
||||
def _panel_payload_hash(
|
||||
jobs: Tuple[Tuple[str, str, Dict[str, Any]], ...],
|
||||
ignore_bar_timestamp: bool = False,
|
||||
@@ -747,11 +760,48 @@ def run_panel(report: "SignalReport",
|
||||
if verdict not in _VALID_VERDICTS or confidence is None:
|
||||
return None
|
||||
rationale = str(reply.get("rationale", "")).strip()[:240]
|
||||
verdict, confidence, rationale = _apply_oppose_guard(
|
||||
verdict, confidence, rationale, takes)
|
||||
return PanelVerdict(verdict=verdict, confidence=confidence,
|
||||
rationale=rationale, takes=takes,
|
||||
created_ts=time.time(), payload_hash=payload_hash)
|
||||
|
||||
|
||||
# Deterministic backstop for the chair (ADVISORY-ONLY, no gate reads it).
|
||||
#
|
||||
# Evidence (2026-09 signal-log audit, 133 live verdicts, 107 with a
|
||||
# replayed outcome): the contrarian and risk personas opposed 117/133
|
||||
# alerts each and the technical persona 88/133 — they object to the
|
||||
# trend extension the engine already discounted — so the chair said
|
||||
# OPPOSE 73% of the time while the OPPOSED trades made +30.1R (PF 2.6).
|
||||
# The macro persona was the only informative voice: macro=support
|
||||
# trades won 88% (+0.56R mean, n=69); macro=oppose won 25%; macro=neutral
|
||||
# 38%. With this rule the OPPOSE bucket becomes the weakest one
|
||||
# (n=29, +0.08R mean, 41% win) instead of the best. Small sample, one
|
||||
# bullish episode — re-score with analyze_signal_log --stored-candles
|
||||
# after more history before tightening further.
|
||||
#
|
||||
# Rule: OPPOSE stands only if the macro analyst opposes, or the technical
|
||||
# analyst opposes while the macro analyst is not supportive. Otherwise
|
||||
# the verdict is NEUTRAL (confidence capped below 50).
|
||||
|
||||
|
||||
def _apply_oppose_guard(verdict: str, confidence: float, rationale: str,
|
||||
takes: List[PersonaTake]):
|
||||
if verdict != "OPPOSE":
|
||||
return verdict, confidence, rationale
|
||||
stance = {t.persona: t.stance for t in takes}
|
||||
macro = stance.get("macro")
|
||||
technical = stance.get("technical")
|
||||
if macro == "oppose":
|
||||
return verdict, confidence, rationale
|
||||
if technical == "oppose" and macro != "support":
|
||||
return verdict, confidence, rationale
|
||||
note = ("The big-picture view does not object, so the panel stays "
|
||||
"neutral rather than against. ")
|
||||
return "NEUTRAL", min(float(confidence), 49.0), (note + rationale)[:240]
|
||||
|
||||
|
||||
# ── Cache, budget, attach ────────────────────────────────────────────
|
||||
|
||||
_state_lock = threading.Lock()
|
||||
|
||||
@@ -0,0 +1,951 @@
|
||||
#!/usr/bin/env python3
|
||||
"""
|
||||
analyze_signal_log.py — signal_log performance evaluator
|
||||
=========================================================
|
||||
|
||||
READ-ONLY analysis of logs/signals.db: how are the logged live signals
|
||||
and the advisory LLM panel actually performing?
|
||||
|
||||
Three lenses:
|
||||
|
||||
1. TRADE REPLAY — every logged row carrying a trade_setup is deduped
|
||||
(telegram + dashboard log the same scan; a persisting signal re-fires
|
||||
every bar) into unique signal EVENTS, then replayed against the RAW
|
||||
Oanda price path with the backtester's own resolution code
|
||||
(backtest.gpu_backtest.simulate_trades): entry at the next bar open
|
||||
after the scan boundary, SL/TP distances preserved, SL-first ties,
|
||||
gap fills at bar open, trail ratchet at bar close using the trail
|
||||
params stamped on the setup. Dual cost reporting per repo convention:
|
||||
$0.65 round trip headline, $0.30 derived analytically.
|
||||
|
||||
2. PANEL SCORECARD — for every row with a panel verdict, the
|
||||
counterfactual: what did/would the underlying trade do? OPPOSE on a
|
||||
loser = R saved; OPPOSE on a winner = R cost.
|
||||
|
||||
3. DIRECTIONAL HIT-RATE — non-setup BUY/SELL rows scored by forward
|
||||
close-to-close return at +1h/+4h/+24h vs signal direction.
|
||||
|
||||
4. SURFACE PARITY (--parity) — telegram vs dashboard rows on the same
|
||||
(timeframe, signal bar): signal / grade / setup / action / gate config
|
||||
fingerprint must agree; a divergence means the two processes ran
|
||||
different configs (2026-08-19: the dashboard applied a different
|
||||
session filter for a day). Also reports rows-per-bar per source (the
|
||||
over-scanning symptom).
|
||||
|
||||
5. STORED-CANDLE REPLAY (--stored-candles) — the same event replay, but
|
||||
resolved on the closed-candle streams stored in the DB itself (no
|
||||
network, no parquet): identical model to advisory_outcome.py, plus
|
||||
the "opportunity" (first event per tf+direction cluster) and
|
||||
one-position-per-timeframe views.
|
||||
|
||||
The DB is opened read-only (mode=ro — WAL-visible; the older immutable=1
|
||||
URI silently missed the newest rows). Bar data for the default replay
|
||||
comes from the active provider via the backtester's fetch_history
|
||||
(parquet-cached). Sample-size caveat: the log started 2026-08-13, so
|
||||
results are directional, not statistically significant.
|
||||
|
||||
Usage:
|
||||
python analyze_signal_log.py # full report
|
||||
python analyze_signal_log.py --csv out.csv # + per-event CSV
|
||||
python analyze_signal_log.py --no-fetch # cache only
|
||||
python analyze_signal_log.py --parity # surface parity only
|
||||
python analyze_signal_log.py --stored-candles # DB-only replay
|
||||
"""
|
||||
|
||||
import argparse
|
||||
import json
|
||||
import math
|
||||
import os
|
||||
import sqlite3
|
||||
import sys
|
||||
from dataclasses import dataclass, field
|
||||
from datetime import datetime, timedelta, timezone
|
||||
from typing import Dict, List, Optional, Sequence, Tuple
|
||||
|
||||
import pandas as pd
|
||||
|
||||
sys.path.insert(0, os.path.dirname(os.path.abspath(__file__)))
|
||||
|
||||
from config import Timeframe # noqa: E402
|
||||
|
||||
# Dual-cost convention (CLAUDE.md): $0.65 measured-Oanda headline,
|
||||
# $0.30 retained for cross-book comparability.
|
||||
COST_HEADLINE = 0.65
|
||||
COST_COMPARE = 0.30
|
||||
|
||||
DEFAULT_DB = os.path.join("logs", "signals.db")
|
||||
MAX_BARS_DEFAULT = 300 # matches the backtester's trailing default
|
||||
CLUSTER_GAP_DEFAULT_MIN = 60 # same setup re-logged within this = one event
|
||||
|
||||
# ── Data model ────────────────────────────────────────────────────────
|
||||
|
||||
|
||||
@dataclass
|
||||
class LogRow:
|
||||
"""One signals.db row, decoded."""
|
||||
id: int
|
||||
logged_at: datetime
|
||||
source: str
|
||||
timeframe: str
|
||||
signal: str
|
||||
score: Optional[float]
|
||||
quality: str
|
||||
price: Optional[float]
|
||||
market_regime: str
|
||||
setup: Optional[dict]
|
||||
panel_verdict: str
|
||||
panel_confidence: Optional[float]
|
||||
action: str = ""
|
||||
bar_ts: Optional[str] = None # primary-TF signal bar label
|
||||
gate_reason: str = ""
|
||||
actionable: Optional[int] = None
|
||||
config_fingerprint: str = ""
|
||||
advisory_event_id: Optional[int] = None
|
||||
advisory_status: str = ""
|
||||
advisory_r: Optional[float] = None
|
||||
scan_kind: str = ""
|
||||
|
||||
|
||||
@dataclass
|
||||
class SignalEvent:
|
||||
"""A deduped actionable signal (cluster of identical re-logs)."""
|
||||
rows: List[LogRow]
|
||||
representative: LogRow # FIRST occurrence — when a trader acts
|
||||
panel_verdict: str = ""
|
||||
panel_confidence: Optional[float] = None
|
||||
# Filled by replay:
|
||||
exit_reason: str = ""
|
||||
exit_time: Optional[pd.Timestamp] = None
|
||||
bars_held: int = 0
|
||||
r_headline: Optional[float] = None # R at $0.65 round trip
|
||||
r_compare: Optional[float] = None # R at $0.30 round trip
|
||||
mfe_r: Optional[float] = None
|
||||
mae_r: Optional[float] = None
|
||||
entry_fill: Optional[float] = None # actual next-open fill
|
||||
|
||||
|
||||
# ── Loading (immutable — never writes to the DB) ─────────────────────
|
||||
|
||||
_COLS = ("id, logged_at, source, timeframe, signal, score, quality,"
|
||||
" price, market_regime, setup_json, panel_verdict,"
|
||||
" panel_confidence, action")
|
||||
# Columns added by the 2026-09 observability work; absent on older DBs.
|
||||
_OPTIONAL_COLS = ("gate_reason", "actionable", "config_fingerprint",
|
||||
"advisory_event_id", "advisory_outcome_status",
|
||||
"advisory_r_multiple", "scan_kind")
|
||||
|
||||
|
||||
def open_readonly(db_path: str) -> sqlite3.Connection:
|
||||
"""Read-only connection that still sees WAL-resident rows.
|
||||
|
||||
``immutable=1`` (the original choice) ignores the -wal file, so the
|
||||
rows the live loggers wrote since the last checkpoint were invisible.
|
||||
``mode=ro`` needs the -shm/-wal files to be readable, which they are
|
||||
while a writer exists; fall back to immutable only if that fails."""
|
||||
path = os.path.abspath(db_path)
|
||||
try:
|
||||
con = sqlite3.connect(f"file:{path}?mode=ro", uri=True)
|
||||
con.execute("SELECT 1 FROM sqlite_master LIMIT 1").fetchall()
|
||||
return con
|
||||
except sqlite3.OperationalError:
|
||||
print("WARNING: mode=ro open failed — falling back to immutable=1 "
|
||||
"(newest WAL rows may be missing)")
|
||||
return sqlite3.connect(f"file:{path}?immutable=1", uri=True)
|
||||
|
||||
|
||||
def load_rows(db_path: str) -> List[LogRow]:
|
||||
"""Read every signals row (read-only, WAL-visible)."""
|
||||
con = open_readonly(db_path)
|
||||
try:
|
||||
have = {r[1] for r in con.execute("PRAGMA table_info(signals)")}
|
||||
opt = [c for c in _OPTIONAL_COLS if c in have]
|
||||
cols = _COLS + ("".join(", " + c for c in opt))
|
||||
bar_expr = (", json_extract(indicators_json, '$.\"' || timeframe"
|
||||
" || '\".timestamp')" if "indicators_json" in have
|
||||
else ", NULL")
|
||||
out: List[LogRow] = []
|
||||
for r in con.execute(f"SELECT {cols}{bar_expr} FROM signals"
|
||||
" ORDER BY id"):
|
||||
setup = json.loads(r[9]) if r[9] else None
|
||||
extra = dict(zip(opt, r[13:13 + len(opt)]))
|
||||
out.append(LogRow(
|
||||
id=r[0],
|
||||
logged_at=datetime.fromisoformat(r[1]),
|
||||
source=r[2], timeframe=r[3], signal=r[4], score=r[5],
|
||||
quality=r[6] or "", price=r[7], market_regime=r[8] or "",
|
||||
setup=setup,
|
||||
panel_verdict=r[10] or "", panel_confidence=r[11],
|
||||
action=r[12] or "",
|
||||
bar_ts=r[-1],
|
||||
gate_reason=extra.get("gate_reason") or "",
|
||||
actionable=extra.get("actionable"),
|
||||
config_fingerprint=extra.get("config_fingerprint") or "",
|
||||
advisory_event_id=extra.get("advisory_event_id"),
|
||||
advisory_status=extra.get("advisory_outcome_status") or "",
|
||||
advisory_r=extra.get("advisory_r_multiple"),
|
||||
scan_kind=extra.get("scan_kind") or "",
|
||||
))
|
||||
return out
|
||||
finally:
|
||||
con.close()
|
||||
|
||||
|
||||
# ── Dedup / clustering ────────────────────────────────────────────────
|
||||
|
||||
|
||||
def _setup_key(row: LogRow) -> Tuple:
|
||||
"""Delegates to signal_log.setup_key — the SAME identity the logger
|
||||
uses for advisory-event dedup, so both tools cluster identically."""
|
||||
from signal_log import setup_key
|
||||
return (row.timeframe, setup_key(row.setup, row.timeframe))
|
||||
|
||||
|
||||
def cluster_setup_rows(
|
||||
rows: Sequence[LogRow],
|
||||
gap: timedelta = timedelta(minutes=CLUSTER_GAP_DEFAULT_MIN),
|
||||
) -> List[SignalEvent]:
|
||||
"""Collapse re-logs of the same setup into one SignalEvent.
|
||||
|
||||
Two layers of duplication exist in the log:
|
||||
- telegram AND dashboard both log the same scan (seconds apart);
|
||||
- a persisting signal re-fires every bar with an identical setup
|
||||
(live, the one-position rule means it is taken at most once).
|
||||
Rows with an identical (tf, direction, entry, SL, strategy) key and
|
||||
consecutive-log gaps <= `gap` are ONE event; the representative is
|
||||
the FIRST occurrence (when a trader/daemon would have acted).
|
||||
"""
|
||||
events: List[SignalEvent] = []
|
||||
last_by_key: Dict[Tuple, SignalEvent] = {}
|
||||
for row in sorted((r for r in rows if r.setup), key=lambda r: r.id):
|
||||
key = _setup_key(row)
|
||||
ev = last_by_key.get(key)
|
||||
if ev is not None and row.logged_at - ev.rows[-1].logged_at <= gap:
|
||||
ev.rows.append(row)
|
||||
else:
|
||||
ev = SignalEvent(rows=[row], representative=row)
|
||||
events.append(ev)
|
||||
last_by_key[key] = ev
|
||||
# Any panel verdict on any member row attaches to the event.
|
||||
if row.panel_verdict and not ev.panel_verdict:
|
||||
ev.panel_verdict = row.panel_verdict
|
||||
ev.panel_confidence = row.panel_confidence
|
||||
return events
|
||||
|
||||
|
||||
# ── Trade construction + replay (backtester realism, reused code) ────
|
||||
|
||||
|
||||
def entry_bar_open(ts: datetime, tf: Timeframe) -> pd.Timestamp:
|
||||
"""The bar opening at the scan boundary just before/at `ts`.
|
||||
|
||||
Live scans run at boundary+buffer, so flooring the log time to the
|
||||
TF boundary gives the entry bar — identical to the backtester's
|
||||
"first raw bar strictly after the signal bar".
|
||||
"""
|
||||
secs = {Timeframe.M5: 300, Timeframe.M15: 900, Timeframe.M30: 1800,
|
||||
Timeframe.H1: 3600, Timeframe.H4: 14400,
|
||||
Timeframe.D1: 86400}[tf]
|
||||
epoch = int(ts.timestamp())
|
||||
return pd.Timestamp(epoch - epoch % secs, unit="s", tz="UTC")
|
||||
|
||||
|
||||
def build_trade(spec: LogRow, raw_df: pd.DataFrame):
|
||||
"""LogRow with setup -> gpu_backtest.Trade at the next-open fill.
|
||||
|
||||
SL/TP distances from the logged setup are preserved relative to the
|
||||
actual fill, exactly like the backtester's replay loop. Returns
|
||||
(trade, risk_distance) or (None, 0) when no bar exists at/after the
|
||||
entry boundary (signal too recent).
|
||||
"""
|
||||
from backtest.gpu_backtest import Trade
|
||||
|
||||
s = spec.setup
|
||||
tf = Timeframe(spec.timeframe)
|
||||
want = entry_bar_open(spec.logged_at, tf)
|
||||
pos = int(raw_df.index.searchsorted(want))
|
||||
if pos >= len(raw_df):
|
||||
return None, 0.0
|
||||
entry_time = raw_df.index[pos]
|
||||
entry_price = float(raw_df["Open"].iloc[pos])
|
||||
|
||||
direction = s["direction"]
|
||||
log_entry = float(s["entry"])
|
||||
sl_dist = abs(log_entry - float(s["stop_loss"]))
|
||||
tp1_dist = abs(float(s["take_profit_1"]) - log_entry)
|
||||
tp2_dist = abs(float(s["take_profit_2"]) - log_entry)
|
||||
if sl_dist <= 0:
|
||||
return None, 0.0
|
||||
sign = 1.0 if direction == "BUY" else -1.0
|
||||
risk_pips = float(s.get("risk_pips") or 0)
|
||||
pip_value = sl_dist / risk_pips if risk_pips > 0 else 0.1
|
||||
|
||||
trade = Trade(
|
||||
pair="XAU/USD",
|
||||
entry_time=entry_time,
|
||||
entry_price=entry_price,
|
||||
sl=entry_price - sign * sl_dist,
|
||||
tp1=entry_price + sign * tp1_dist,
|
||||
tp2=entry_price + sign * tp2_dist,
|
||||
direction=direction,
|
||||
pip_value=pip_value,
|
||||
risk_pips=risk_pips,
|
||||
reward_pips=float(s.get("reward_pips") or 0),
|
||||
rr_planned=float(s.get("risk_reward") or 0),
|
||||
quality=spec.quality,
|
||||
final_score=float(spec.score or 0),
|
||||
final_confidence=0.0,
|
||||
market_regime=spec.market_regime,
|
||||
strategy_tag=s.get("strategy") or "GENERIC",
|
||||
trail_arm_r=float(s.get("trail_arm_r") or 0),
|
||||
trail_dist_r=float(s.get("trail_dist_r") or 0),
|
||||
)
|
||||
return trade, sl_dist
|
||||
|
||||
|
||||
def replay_events(
|
||||
events: List[SignalEvent],
|
||||
raw_by_tf: Dict[Timeframe, pd.DataFrame],
|
||||
max_bars: int = MAX_BARS_DEFAULT,
|
||||
spread: float = COST_HEADLINE,
|
||||
) -> None:
|
||||
"""Resolve every event on the raw price path and stamp R multiples.
|
||||
|
||||
Reuses backtest.gpu_backtest.simulate_trades verbatim: raw-path
|
||||
resolution, SL-first ties, gap fills at open, close-only trail
|
||||
ratchet with the per-setup trail params as per-trade overrides.
|
||||
The $0.30 book is derived analytically (R is linear in cost).
|
||||
"""
|
||||
from backtest.gpu_backtest import simulate_trades
|
||||
|
||||
trades, owners, risks = [], [], []
|
||||
for ev in events:
|
||||
tf = Timeframe(ev.representative.timeframe)
|
||||
raw = raw_by_tf.get(tf)
|
||||
if raw is None or raw.empty:
|
||||
ev.exit_reason = "NO_DATA"
|
||||
continue
|
||||
trade, risk = build_trade(ev.representative, raw)
|
||||
if trade is None:
|
||||
ev.exit_reason = "PENDING" # no bar at/after the boundary
|
||||
continue
|
||||
trades.append(trade)
|
||||
owners.append(ev)
|
||||
risks.append(risk)
|
||||
|
||||
# simulate per-TF so each trade's window indexes the right frame.
|
||||
# Run-level trail params are placeholders: every trailing trade
|
||||
# carries its own stamped overrides, which win inside
|
||||
# _simulate_trailing. Static trades (arm=0) resolve SL/TP1.
|
||||
# Group by (TF, exit mode): a static trade inside a trailing call
|
||||
# would wrongly inherit the run-level arm.
|
||||
groups: Dict[Tuple[Timeframe, bool], list] = {}
|
||||
for t, ev in zip(trades, owners):
|
||||
key = (Timeframe(ev.representative.timeframe), t.trail_arm_r > 0)
|
||||
groups.setdefault(key, []).append(t)
|
||||
for (gtf, is_trailing), gtrades in groups.items():
|
||||
simulate_trades(gtrades, raw_by_tf[gtf],
|
||||
max_bars=max_bars, spread=spread,
|
||||
trail_arm_r=0.6 if is_trailing else 0.0,
|
||||
trail_dist_r=0.75)
|
||||
|
||||
for trade, ev, risk in zip(trades, owners, risks):
|
||||
ev.entry_fill = trade.entry_price
|
||||
ev.exit_time = trade.exit_time
|
||||
ev.bars_held = trade.bars_held
|
||||
ev.mfe_r = trade.mfe_r
|
||||
ev.mae_r = trade.mae_r
|
||||
if trade.exit_reason in ("NO_DATA", ""):
|
||||
ev.exit_reason = "NO_DATA"
|
||||
continue
|
||||
# Data ran out before max_bars → position still open, marked to
|
||||
# market; distinct from a true 300-bar TIMEOUT.
|
||||
if trade.exit_reason == "TIMEOUT" and trade.bars_held < max_bars:
|
||||
ev.exit_reason = "OPEN"
|
||||
else:
|
||||
ev.exit_reason = trade.exit_reason
|
||||
ev.r_headline = trade.r_multiple
|
||||
ev.r_compare = trade.r_multiple + (spread - COST_COMPARE) / risk
|
||||
|
||||
|
||||
# ── Panel scorecard ───────────────────────────────────────────────────
|
||||
|
||||
|
||||
def panel_scorecard(
|
||||
rows: Sequence[LogRow],
|
||||
events: Sequence[SignalEvent],
|
||||
m5: Optional[pd.DataFrame],
|
||||
) -> List[dict]:
|
||||
"""Counterfactual per panel verdict: did it save or cost R?
|
||||
|
||||
Verdicted rows with a setup join their SignalEvent outcome; rows
|
||||
without a setup are scored on the +4h forward return instead.
|
||||
"""
|
||||
by_row: Dict[int, SignalEvent] = {}
|
||||
for ev in events:
|
||||
for r in ev.rows:
|
||||
by_row[r.id] = ev
|
||||
|
||||
out = []
|
||||
for row in rows:
|
||||
if not row.panel_verdict:
|
||||
continue
|
||||
entry = {"id": row.id, "logged_at": row.logged_at,
|
||||
"timeframe": row.timeframe, "signal": row.signal,
|
||||
"quality": row.quality, "verdict": row.panel_verdict,
|
||||
"confidence": row.panel_confidence,
|
||||
"basis": "", "r_headline": None, "note": ""}
|
||||
ev = by_row.get(row.id)
|
||||
if ev is not None and ev.r_headline is not None:
|
||||
entry["basis"] = "trade_replay"
|
||||
entry["r_headline"] = ev.r_headline
|
||||
entry["note"] = f"exit={ev.exit_reason}"
|
||||
elif m5 is not None and row.price:
|
||||
fwd = _forward_return(m5, row.logged_at, row.price, hours=4)
|
||||
if fwd is not None:
|
||||
sign = 1 if row.signal in ("BUY", "STRONG_BUY") else -1
|
||||
entry["basis"] = "fwd_4h_return"
|
||||
entry["r_headline"] = None
|
||||
entry["note"] = f"4h move {sign * fwd:+.2f}$ in dir"
|
||||
out.append(entry)
|
||||
return out
|
||||
|
||||
|
||||
# ── Directional hit-rate (non-setup signals) ──────────────────────────
|
||||
|
||||
|
||||
def _forward_return(
|
||||
df: pd.DataFrame, ts: datetime, price: float, hours: float
|
||||
) -> Optional[float]:
|
||||
"""Close-to-close move from `price` to the last close at ts+hours."""
|
||||
target = pd.Timestamp(ts) + pd.Timedelta(hours=hours)
|
||||
closes = df["Close"]
|
||||
pos = int(df.index.searchsorted(target, side="right")) - 1
|
||||
if pos < 0:
|
||||
return None
|
||||
return float(closes.iloc[pos]) - price
|
||||
|
||||
|
||||
def directional_hit_rate(
|
||||
rows: Sequence[LogRow],
|
||||
m5: pd.DataFrame,
|
||||
horizons_h: Sequence[float] = (1, 4, 24),
|
||||
) -> List[dict]:
|
||||
"""Forward-return direction check for non-setup BUY/SELL rows.
|
||||
|
||||
Cross-source dupes collapse on (tf, signal, 5-minute bucket).
|
||||
Returns per-horizon aggregates plus the per-row detail under 'rows'.
|
||||
"""
|
||||
seen = set()
|
||||
detail = []
|
||||
for row in sorted(rows, key=lambda r: r.id):
|
||||
if row.setup or row.signal not in (
|
||||
"BUY", "STRONG_BUY", "SELL", "STRONG_SELL"):
|
||||
continue
|
||||
if row.price is None:
|
||||
continue
|
||||
bucket = int(row.logged_at.timestamp()) // 300
|
||||
key = (row.timeframe, row.signal, bucket)
|
||||
if key in seen:
|
||||
continue
|
||||
seen.add(key)
|
||||
sign = 1 if row.signal in ("BUY", "STRONG_BUY") else -1
|
||||
rec = {"id": row.id, "logged_at": row.logged_at,
|
||||
"timeframe": row.timeframe, "signal": row.signal,
|
||||
"quality": row.quality, "regime": row.market_regime}
|
||||
for h in horizons_h:
|
||||
fwd = _forward_return(m5, row.logged_at, row.price, h)
|
||||
rec[f"fwd_{h}h"] = None if fwd is None else sign * fwd
|
||||
detail.append(rec)
|
||||
|
||||
agg = []
|
||||
for h in horizons_h:
|
||||
k = f"fwd_{h}h"
|
||||
vals = [d[k] for d in detail if d[k] is not None]
|
||||
wins = [v for v in vals if v > 0]
|
||||
agg.append({"horizon_h": h, "n": len(vals),
|
||||
"hit_rate": (len(wins) / len(vals)) if vals else None,
|
||||
"mean_move": (sum(vals) / len(vals)) if vals else None})
|
||||
return {"aggregate": agg, "rows": detail}
|
||||
|
||||
|
||||
# ── Data fetching (backtester fetch_history: provider + parquet) ─────
|
||||
|
||||
|
||||
def fetch_raw_frames(
|
||||
tfs: Sequence[Timeframe],
|
||||
start: datetime,
|
||||
end: datetime,
|
||||
no_fetch: bool = False,
|
||||
) -> Dict[Timeframe, pd.DataFrame]:
|
||||
"""RAW (uncleaned) bars per TF — trade resolution must see spikes."""
|
||||
from backtest.gpu_backtest import _cache_path, fetch_history
|
||||
from pair_signals.xauusd import make_config
|
||||
from providers import make_data_provider
|
||||
|
||||
# Factory config (never bare PlatformConfig) — fetch_range and
|
||||
# validate_and_clean read lookback/refresh knobs off it.
|
||||
provider = make_data_provider(make_config(Timeframe.M15))
|
||||
out = {}
|
||||
for tf in tfs:
|
||||
if no_fetch and not _cache_path(
|
||||
"XAU/USD", tf, start, end, provider.name).exists():
|
||||
raise SystemExit(
|
||||
f"--no-fetch: no cached {tf.value} bars for "
|
||||
f"{start.date()}→{end.date()}; run without --no-fetch once")
|
||||
df = fetch_history(provider, "XAU/USD", tf, start, end,
|
||||
use_cache=True, clean=False)
|
||||
out[tf] = df
|
||||
return out
|
||||
|
||||
|
||||
# ── Surface parity ────────────────────────────────────────────────────
|
||||
|
||||
|
||||
def parity_report(rows: Sequence[LogRow]) -> dict:
|
||||
"""Compare every surface's row on the same (timeframe, signal bar).
|
||||
|
||||
Groups rows by (tf, bar label). For groups with 2+ sources, the first
|
||||
row per source is compared on signal / quality / setup presence /
|
||||
action / config fingerprint. Returns coverage, rows-per-bar per
|
||||
source (over-scanning symptom), and the divergence list."""
|
||||
groups: Dict[Tuple[str, str], List[LogRow]] = {}
|
||||
for r in rows:
|
||||
if not r.bar_ts or r.scan_kind == "blocked":
|
||||
continue
|
||||
groups.setdefault((r.timeframe, str(r.bar_ts)), []).append(r)
|
||||
per_bar: Dict[Tuple[str, str], List[int]] = {}
|
||||
divergences = []
|
||||
both = 0
|
||||
for (tf, bar), members in groups.items():
|
||||
by_src: Dict[str, LogRow] = {}
|
||||
counts: Dict[str, int] = {}
|
||||
for m in sorted(members, key=lambda x: x.id):
|
||||
counts[m.source] = counts.get(m.source, 0) + 1
|
||||
by_src.setdefault(m.source, m)
|
||||
for src, n in counts.items():
|
||||
per_bar.setdefault((src, tf), []).append(n)
|
||||
if len(by_src) < 2:
|
||||
continue
|
||||
both += 1
|
||||
srcs = sorted(by_src)
|
||||
a = by_src[srcs[0]]
|
||||
for src in srcs[1:]:
|
||||
b = by_src[src]
|
||||
diffs = []
|
||||
if a.signal != b.signal:
|
||||
diffs.append(f"signal {a.signal}/{b.signal}")
|
||||
if a.quality != b.quality:
|
||||
diffs.append(f"quality {a.quality}/{b.quality}")
|
||||
if (a.setup is None) != (b.setup is None):
|
||||
diffs.append("setup presence")
|
||||
if a.action != b.action:
|
||||
diffs.append("action")
|
||||
if (a.config_fingerprint and b.config_fingerprint
|
||||
and a.config_fingerprint != b.config_fingerprint):
|
||||
diffs.append("config fingerprint")
|
||||
if diffs:
|
||||
divergences.append({
|
||||
"timeframe": tf, "bar": bar, "sources": (srcs[0], src),
|
||||
"ids": (a.id, b.id), "diffs": diffs,
|
||||
"a_action": a.action, "b_action": b.action,
|
||||
})
|
||||
rows_per_bar = {
|
||||
f"{src}/{tf}": sum(v) / len(v)
|
||||
for (src, tf), v in per_bar.items() if v
|
||||
}
|
||||
return {"bars_with_2_sources": both, "n_groups": len(groups),
|
||||
"rows_per_bar": rows_per_bar, "divergences": divergences}
|
||||
|
||||
|
||||
def print_parity(rep: dict) -> None:
|
||||
print("\n── Surface parity (same tf + signal bar) " + "─" * 30)
|
||||
print(f" bars seen by 2+ sources: {rep['bars_with_2_sources']} "
|
||||
f"of {rep['n_groups']}")
|
||||
print(" rows per bar (>1.05 = the same bar was scanned more than once):")
|
||||
for k, v in sorted(rep["rows_per_bar"].items()):
|
||||
flag = " <-- over-scanning" if v > 1.05 else ""
|
||||
print(f" {k:20} {v:.2f}{flag}")
|
||||
divs = rep["divergences"]
|
||||
print(f" divergences: {len(divs)}")
|
||||
by_day: Dict[str, int] = {}
|
||||
for d in divs:
|
||||
by_day[d["bar"][:10]] = by_day.get(d["bar"][:10], 0) + 1
|
||||
for day, n in sorted(by_day.items()):
|
||||
print(f" {day}: {n}")
|
||||
for d in divs[:25]:
|
||||
print(f" {d['timeframe']:5} {d['bar']} ids {d['ids']} "
|
||||
f"{'; '.join(d['diffs'])} | {d['a_action'][:32]!r} vs "
|
||||
f"{d['b_action'][:32]!r}")
|
||||
|
||||
|
||||
# ── Stored-candle replay (no network) ────────────────────────────────
|
||||
|
||||
_TF_SECS = {"1min": 60, "5min": 300, "15min": 900, "30min": 1800,
|
||||
"1h": 3600, "4h": 14400, "1d": 86400, "1day": 86400}
|
||||
|
||||
|
||||
def stored_streams(db_path: str) -> Dict[str, List[dict]]:
|
||||
"""Union of every closed bar in candles_json per timeframe."""
|
||||
con = open_readonly(db_path)
|
||||
bars: Dict[str, Dict[str, dict]] = {}
|
||||
try:
|
||||
for (blob,) in con.execute(
|
||||
"SELECT candles_json FROM signals WHERE candles_json"
|
||||
" IS NOT NULL ORDER BY id"):
|
||||
try:
|
||||
snap = json.loads(blob)
|
||||
except (TypeError, ValueError):
|
||||
continue
|
||||
for tf, arr in (snap or {}).items():
|
||||
store = bars.setdefault(tf, {})
|
||||
for b in arr or []:
|
||||
ts = b.get("timestamp")
|
||||
if ts:
|
||||
store.setdefault(str(ts), b)
|
||||
finally:
|
||||
con.close()
|
||||
return {tf: [v[k] for k in sorted(v)] for tf, v in bars.items()}
|
||||
|
||||
|
||||
def replay_stored(events: List[SignalEvent],
|
||||
streams: Dict[str, List[dict]],
|
||||
max_bars: int = MAX_BARS_DEFAULT) -> None:
|
||||
"""Resolve events with advisory_outcome's model on the stored bars.
|
||||
|
||||
Same next-open reanchoring, intrabar peak, close-only ratchet, SL-
|
||||
first, 300-bar timeout and dual cost as the DB's own outcome column
|
||||
— extended to EVERY event, not only panel rows."""
|
||||
from advisory_outcome import advance_outcome, initial_outcome
|
||||
for ev in events:
|
||||
rep_row = ev.representative
|
||||
bars = streams.get(rep_row.timeframe) or []
|
||||
if not bars or not rep_row.bar_ts:
|
||||
ev.exit_reason = "NO_DATA"
|
||||
continue
|
||||
indicators = {rep_row.timeframe: {"timestamp": rep_row.bar_ts}}
|
||||
status, state = initial_outcome(rep_row.setup, indicators,
|
||||
rep_row.timeframe)
|
||||
if status != "PENDING":
|
||||
ev.exit_reason = "NO_DATA"
|
||||
continue
|
||||
state["policy"]["max_hold_bars"] = max_bars
|
||||
state = advance_outcome(state, bars)
|
||||
prog = state.get("progress") or {}
|
||||
ev.entry_fill = prog.get("entry_price")
|
||||
ev.bars_held = int(prog.get("bars_held") or 0)
|
||||
ev.mfe_r = prog.get("mfe_r")
|
||||
ev.mae_r = prog.get("mae_r")
|
||||
res = state.get("result")
|
||||
if state.get("status") == "RESOLVED" and res:
|
||||
ev.exit_reason = res["exit_reason"]
|
||||
ev.exit_time = pd.Timestamp(res["exit_at"])
|
||||
ev.r_headline = res["net_r_065"]
|
||||
ev.r_compare = res["net_r_030"]
|
||||
elif state.get("status") == "PENDING":
|
||||
ev.exit_reason = "OPEN" if ev.entry_fill else "PENDING"
|
||||
else:
|
||||
ev.exit_reason = "UNRESOLVABLE"
|
||||
|
||||
|
||||
def opportunity_view(events: Sequence[SignalEvent],
|
||||
gap_min: float = 60) -> List[SignalEvent]:
|
||||
"""First event per (tf, direction) cluster — consecutive same-direction
|
||||
events on a timeframe within ``gap_min`` are one opportunity."""
|
||||
out: List[SignalEvent] = []
|
||||
last: Dict[Tuple[str, str], datetime] = {}
|
||||
for ev in sorted(events, key=lambda e: e.representative.logged_at):
|
||||
r = ev.representative
|
||||
key = (r.timeframe, r.setup["direction"])
|
||||
prev = last.get(key)
|
||||
if prev is None or r.logged_at - prev > timedelta(minutes=gap_min):
|
||||
out.append(ev)
|
||||
last[key] = r.logged_at
|
||||
return out
|
||||
|
||||
|
||||
def one_position_view(events: Sequence[SignalEvent],
|
||||
cooldown_bars: int = 5) -> List[SignalEvent]:
|
||||
"""Daemon-style book: one open position per timeframe, next signal
|
||||
only after exit + cooldown (wall-clock bars)."""
|
||||
out: List[SignalEvent] = []
|
||||
busy: Dict[str, datetime] = {}
|
||||
for ev in sorted(events, key=lambda e: e.representative.logged_at):
|
||||
if ev.r_headline is None or ev.exit_time is None:
|
||||
continue
|
||||
r = ev.representative
|
||||
until = busy.get(r.timeframe)
|
||||
if until is not None and r.logged_at <= until:
|
||||
continue
|
||||
out.append(ev)
|
||||
secs = _TF_SECS.get(r.timeframe, 900) * cooldown_bars
|
||||
busy[r.timeframe] = (ev.exit_time.to_pydatetime()
|
||||
+ timedelta(seconds=secs))
|
||||
return out
|
||||
|
||||
|
||||
def _book_line(label: str, evs: Sequence[SignalEvent]) -> str:
|
||||
closed = [e for e in evs if e.r_headline is not None]
|
||||
if not closed:
|
||||
return f" {label:28} n=0"
|
||||
tot = sum(e.r_headline for e in closed)
|
||||
wins = [e for e in closed if e.r_headline > 0]
|
||||
gp = sum(e.r_headline for e in wins)
|
||||
gl = -sum(e.r_headline for e in closed if e.r_headline < 0)
|
||||
pf = gp / gl if gl > 0 else math.inf
|
||||
return (f" {label:28} n={len(closed):3d} wins {len(wins):3d} "
|
||||
f"({100 * len(wins) / len(closed):3.0f}%) "
|
||||
f"total {tot:+7.2f}R @0.65 PF {pf:.2f}")
|
||||
|
||||
|
||||
def print_stored_books(events: Sequence[SignalEvent]) -> None:
|
||||
print("\n── Stored-candle replay books ($0.65 rt) " + "─" * 30)
|
||||
print(_book_line("all unique events", events))
|
||||
print(_book_line("opportunities (60-min)", opportunity_view(events)))
|
||||
print(_book_line("one-position-per-TF", one_position_view(events)))
|
||||
by_dir: Dict[str, List[SignalEvent]] = {}
|
||||
for e in events:
|
||||
by_dir.setdefault(e.representative.setup["direction"], []).append(e)
|
||||
for d, evs in sorted(by_dir.items()):
|
||||
print(_book_line(f" direction {d}", evs))
|
||||
by_tf: Dict[str, List[SignalEvent]] = {}
|
||||
for e in events:
|
||||
by_tf.setdefault(e.representative.timeframe, []).append(e)
|
||||
for tf, evs in sorted(by_tf.items()):
|
||||
print(_book_line(f" timeframe {tf}", evs))
|
||||
|
||||
|
||||
# ── Reporting ─────────────────────────────────────────────────────────
|
||||
|
||||
|
||||
def _fmt_r(r: Optional[float]) -> str:
|
||||
return " n/a " if r is None else f"{r:+.2f}R"
|
||||
|
||||
|
||||
def persona_scorecard(rows: Sequence[LogRow], events: Sequence[SignalEvent],
|
||||
db_path: str) -> List[dict]:
|
||||
"""Per persona × stance: n, mean R and win rate of the underlying
|
||||
event (needs panel_takes_json — read straight from the DB)."""
|
||||
by_row: Dict[int, SignalEvent] = {}
|
||||
for ev in events:
|
||||
for r in ev.rows:
|
||||
by_row[r.id] = ev
|
||||
con = open_readonly(db_path)
|
||||
agg: Dict[Tuple[str, str], List[float]] = {}
|
||||
try:
|
||||
for rid, takes in con.execute(
|
||||
"SELECT id, panel_takes_json FROM signals"
|
||||
" WHERE panel_takes_json IS NOT NULL"):
|
||||
ev = by_row.get(rid)
|
||||
if ev is None or ev.r_headline is None:
|
||||
continue
|
||||
try:
|
||||
for t in json.loads(takes):
|
||||
agg.setdefault((t.get("persona", "?"),
|
||||
t.get("stance", "?")), []).append(
|
||||
ev.r_headline)
|
||||
except (TypeError, ValueError):
|
||||
continue
|
||||
finally:
|
||||
con.close()
|
||||
return [{"persona": p, "stance": s, "n": len(v),
|
||||
"mean_r": sum(v) / len(v),
|
||||
"win_rate": sum(1 for x in v if x > 0) / len(v)}
|
||||
for (p, s), v in sorted(agg.items())]
|
||||
|
||||
|
||||
def print_report(
|
||||
rows: List[LogRow],
|
||||
events: List[SignalEvent],
|
||||
panel: List[dict],
|
||||
hit: dict,
|
||||
) -> None:
|
||||
print("=" * 72)
|
||||
print("SIGNAL LOG PERFORMANCE REPORT")
|
||||
print("=" * 72)
|
||||
if rows:
|
||||
print(f"Window: {rows[0].logged_at:%Y-%m-%d %H:%M} → "
|
||||
f"{rows[-1].logged_at:%Y-%m-%d %H:%M} UTC "
|
||||
f"({(rows[-1].logged_at - rows[0].logged_at).days + 1} days)")
|
||||
by_src: Dict[str, int] = {}
|
||||
by_tf: Dict[str, int] = {}
|
||||
for r in rows:
|
||||
by_src[r.source] = by_src.get(r.source, 0) + 1
|
||||
by_tf[r.timeframe] = by_tf.get(r.timeframe, 0) + 1
|
||||
print(f"Rows: {len(rows)} by source {by_src} by tf {by_tf}")
|
||||
n_setup = sum(1 for r in rows if r.setup)
|
||||
print(f"Rows with trade setup: {n_setup} → "
|
||||
f"{len(events)} unique signal events after dedup")
|
||||
|
||||
print("\n── Trade replay (next-open fill, raw path, trail per setup, "
|
||||
f"cost ${COST_HEADLINE:.2f} rt) " + "─" * 8)
|
||||
hdr = (f"{'first logged':16} {'tf':5} {'dir':11} {'q':2} {'strategy':8}"
|
||||
f" {'fill':>9} {'exit':7} {'bars':>4} {'R@0.65':>8} {'R@0.30':>8}"
|
||||
f" {'MFE':>6} {'MAE':>6} panel")
|
||||
print(hdr)
|
||||
print("-" * len(hdr))
|
||||
for ev in events:
|
||||
rep = ev.representative
|
||||
print(f"{rep.logged_at:%m-%d %H:%M} "
|
||||
f"{rep.timeframe:5} {rep.signal:11} {rep.quality:2} "
|
||||
f"{(rep.setup.get('strategy') or ''):8} "
|
||||
f"{ev.entry_fill if ev.entry_fill else float('nan'):9.2f} "
|
||||
f"{ev.exit_reason:7} {ev.bars_held:4} "
|
||||
f"{_fmt_r(ev.r_headline):>8} {_fmt_r(ev.r_compare):>8} "
|
||||
f"{(f'{ev.mfe_r:.2f}' if ev.mfe_r is not None else 'n/a'):>6} "
|
||||
f"{(f'{ev.mae_r:.2f}' if ev.mae_r is not None else 'n/a'):>6} "
|
||||
f"{ev.panel_verdict or ''}")
|
||||
|
||||
closed = [e for e in events if e.r_headline is not None]
|
||||
if closed:
|
||||
tot = sum(e.r_headline for e in closed)
|
||||
tot30 = sum(e.r_compare for e in closed)
|
||||
wins = [e for e in closed if e.r_headline > 0]
|
||||
gp = sum(e.r_headline for e in wins)
|
||||
gl = -sum(e.r_headline for e in closed if e.r_headline < 0)
|
||||
pf = gp / gl if gl > 0 else math.inf
|
||||
print(f"\nClosed: {len(closed)} wins: {len(wins)} "
|
||||
f"({100 * len(wins) / len(closed):.0f}%) "
|
||||
f"total {tot:+.2f}R @0.65 / {tot30:+.2f}R @0.30 PF {pf:.2f}")
|
||||
by_q: Dict[str, List[SignalEvent]] = {}
|
||||
for e in closed:
|
||||
by_q.setdefault(e.representative.quality, []).append(e)
|
||||
for q, grp in sorted(by_q.items()):
|
||||
print(f" grade {q}: n={len(grp)} "
|
||||
f"total {sum(e.r_headline for e in grp):+.2f}R")
|
||||
n_open = sum(1 for e in events if e.exit_reason == "OPEN")
|
||||
n_pend = sum(1 for e in events if e.exit_reason in ("PENDING", "NO_DATA"))
|
||||
if n_open or n_pend:
|
||||
print(f" ({n_open} still open / mark-to-market, "
|
||||
f"{n_pend} pending — insufficient bars yet)")
|
||||
|
||||
print("\n── Advisory panel scorecard " + "─" * 42)
|
||||
if not panel:
|
||||
print(" no panel verdicts logged yet")
|
||||
for p in panel:
|
||||
r = p["r_headline"]
|
||||
if r is None:
|
||||
impact = p["note"] or "no outcome yet"
|
||||
elif p["verdict"] == "OPPOSE":
|
||||
impact = f"saved {-r:+.2f}R" if r < 0 else f"cost {r:+.2f}R"
|
||||
elif p["verdict"] == "AGREE":
|
||||
impact = f"confirmed {r:+.2f}R"
|
||||
else:
|
||||
impact = f"passed, trade {r:+.2f}R"
|
||||
print(f" #{p['id']:<5} {p['logged_at']:%m-%d %H:%M} "
|
||||
f"{p['timeframe']:5} {p['signal']:11} {p['quality']:2} "
|
||||
f"{p['verdict']:7} @{p['confidence'] or 0:3.0f}% "
|
||||
f"[{p['basis'] or 'pending'}] {impact}")
|
||||
|
||||
print("\n── Directional hit-rate (non-setup signals, M5 closes) " + "─" * 12)
|
||||
for a in hit["aggregate"]:
|
||||
if a["n"] == 0:
|
||||
print(f" +{a['horizon_h']}h: no data")
|
||||
continue
|
||||
print(f" +{a['horizon_h']}h: n={a['n']} "
|
||||
f"hit-rate {100 * a['hit_rate']:.0f}% "
|
||||
f"mean move in dir {a['mean_move']:+.2f}$")
|
||||
print("\nCaveat: log started 2026-08-13 — small sample, directional "
|
||||
"read only.")
|
||||
|
||||
|
||||
def export_csv(events: Sequence[SignalEvent], path: str) -> None:
|
||||
recs = []
|
||||
for ev in events:
|
||||
rep = ev.representative
|
||||
s = rep.setup
|
||||
recs.append({
|
||||
"row_ids": ",".join(str(r.id) for r in ev.rows),
|
||||
"first_logged": rep.logged_at.isoformat(),
|
||||
"timeframe": rep.timeframe, "signal": rep.signal,
|
||||
"quality": rep.quality, "score": rep.score,
|
||||
"strategy": s.get("strategy"),
|
||||
"logged_entry": s.get("entry"), "fill": ev.entry_fill,
|
||||
"sl": s.get("stop_loss"), "tp1": s.get("take_profit_1"),
|
||||
"trail_arm_r": s.get("trail_arm_r"),
|
||||
"trail_dist_r": s.get("trail_dist_r"),
|
||||
"exit_reason": ev.exit_reason,
|
||||
"exit_time": ev.exit_time.isoformat() if ev.exit_time is not None else "",
|
||||
"bars_held": ev.bars_held,
|
||||
"r_at_0.65": ev.r_headline, "r_at_0.30": ev.r_compare,
|
||||
"mfe_r": ev.mfe_r, "mae_r": ev.mae_r,
|
||||
"panel_verdict": ev.panel_verdict,
|
||||
"panel_confidence": ev.panel_confidence,
|
||||
})
|
||||
pd.DataFrame(recs).to_csv(path, index=False)
|
||||
print(f"\nWrote {len(recs)} events to {path}")
|
||||
|
||||
|
||||
# ── main ──────────────────────────────────────────────────────────────
|
||||
|
||||
|
||||
def main(argv: Optional[Sequence[str]] = None) -> int:
|
||||
ap = argparse.ArgumentParser(description=__doc__.splitlines()[0])
|
||||
ap.add_argument("--db", default=DEFAULT_DB)
|
||||
ap.add_argument("--csv", default=None, help="per-event CSV export")
|
||||
ap.add_argument("--max-bars", type=int, default=MAX_BARS_DEFAULT)
|
||||
ap.add_argument("--cluster-gap-min", type=float,
|
||||
default=CLUSTER_GAP_DEFAULT_MIN)
|
||||
ap.add_argument("--no-fetch", action="store_true",
|
||||
help="parquet cache only, never hit the network")
|
||||
ap.add_argument("--parity", action="store_true",
|
||||
help="surface parity report only (no bar data needed)")
|
||||
ap.add_argument("--stored-candles", action="store_true",
|
||||
help="replay on the candle streams stored in the DB "
|
||||
"(no network) instead of provider history")
|
||||
args = ap.parse_args(argv)
|
||||
|
||||
rows = load_rows(args.db)
|
||||
if not rows:
|
||||
print(f"No rows in {args.db}")
|
||||
return 0
|
||||
|
||||
if args.parity:
|
||||
print_parity(parity_report(rows))
|
||||
return 0
|
||||
|
||||
events = cluster_setup_rows(
|
||||
rows, gap=timedelta(minutes=args.cluster_gap_min))
|
||||
|
||||
if args.stored_candles:
|
||||
streams = stored_streams(args.db)
|
||||
replay_stored(events, streams, max_bars=args.max_bars)
|
||||
panel = panel_scorecard(rows, events, None)
|
||||
m5 = streams.get(Timeframe.M5.value) or []
|
||||
m5_df = pd.DataFrame(m5)
|
||||
if not m5_df.empty:
|
||||
m5_df.index = pd.to_datetime(m5_df["timestamp"], utc=True)
|
||||
m5_df = m5_df.rename(columns={"close": "Close"})
|
||||
hit = directional_hit_rate(rows, m5_df) if not m5_df.empty else \
|
||||
{"aggregate": [], "rows": []}
|
||||
print_report(rows, events, panel, hit)
|
||||
print_stored_books(events)
|
||||
print("\n── Persona scorecard (stance × event outcome) " + "─" * 22)
|
||||
for p in persona_scorecard(rows, events, args.db):
|
||||
print(f" {p['persona']:11} {p['stance']:8} n={p['n']:3d} "
|
||||
f"mean {p['mean_r']:+.3f}R win {100 * p['win_rate']:.0f}%")
|
||||
print_parity(parity_report(rows))
|
||||
if args.csv:
|
||||
export_csv(events, args.csv)
|
||||
return 0
|
||||
|
||||
tfs = {Timeframe(r.timeframe) for r in rows if r.setup}
|
||||
tfs.add(Timeframe.M5) # hit-rate + panel fallback frame
|
||||
start = (rows[0].logged_at - timedelta(days=1)).replace(
|
||||
hour=0, minute=0, second=0, microsecond=0)
|
||||
end = datetime.now(timezone.utc)
|
||||
raw = fetch_raw_frames(sorted(tfs, key=lambda t: t.value),
|
||||
start, end, no_fetch=args.no_fetch)
|
||||
|
||||
replay_events(events, raw, max_bars=args.max_bars)
|
||||
panel = panel_scorecard(rows, events, raw.get(Timeframe.M5))
|
||||
hit = directional_hit_rate(rows, raw[Timeframe.M5])
|
||||
|
||||
print_report(rows, events, panel, hit)
|
||||
print_parity(parity_report(rows))
|
||||
if args.csv:
|
||||
export_csv(events, args.csv)
|
||||
return 0
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
sys.exit(main())
|
||||
+16
-1
@@ -101,7 +101,7 @@ _KNOWN_AGENT_PANEL = {
|
||||
"candles_enabled", "candle_bars_primary", "candle_bars_confirm",
|
||||
}
|
||||
_KNOWN_MARKET_OUTLOOK = {"enabled", "telegram", "max_per_day", "candles_bars"}
|
||||
_KNOWN_SIGNAL_LOG = {"enabled", "path"}
|
||||
_KNOWN_SIGNAL_LOG = {"enabled", "path", "log_blocked_scans"}
|
||||
_KNOWN_TELEGRAM = {"token", "chat_id", "channel_id", "alert_min_quality"}
|
||||
_KNOWN_SCANNING = {"pairs", "timeframes"}
|
||||
_KNOWN_PAIR_KEYS = {"blocked_hours_utc"}
|
||||
@@ -355,6 +355,10 @@ def load_app_config(path: str = CONFIG_PATH) -> AppConfig:
|
||||
or not slog["path"]):
|
||||
raise AppConfigError(
|
||||
"config.json: signal_log.path must be a non-empty string")
|
||||
if ("log_blocked_scans" in slog
|
||||
and not isinstance(slog["log_blocked_scans"], bool)):
|
||||
raise AppConfigError(
|
||||
"config.json: signal_log.log_blocked_scans must be true or false")
|
||||
app.signal_log_settings = dict(slog)
|
||||
|
||||
level = raw.get("log_level", app.log_level)
|
||||
@@ -478,6 +482,17 @@ def signal_log_enabled(app: Optional[AppConfig] = None) -> bool:
|
||||
return bool(app.signal_log_settings.get("enabled", True))
|
||||
|
||||
|
||||
def signal_log_blocked_scans(app: Optional[AppConfig] = None) -> bool:
|
||||
"""Whether weekend/day-filter early returns are logged (as minimal
|
||||
rows without news/candles). Default ON — they are liveness evidence;
|
||||
turn off to keep the history table to real scans only."""
|
||||
env = os.environ.get("SIGNAL_LOG_BLOCKED_SCANS")
|
||||
if env:
|
||||
return env.strip().lower() in ("1", "true", "yes", "on")
|
||||
app = app or get_app_config()
|
||||
return bool(app.signal_log_settings.get("log_blocked_scans", True))
|
||||
|
||||
|
||||
def signal_log_path(app: Optional[AppConfig] = None) -> str:
|
||||
"""Absolute path to the SQLite signal-history DB. A relative config
|
||||
value resolves against the repo root so CWD never matters."""
|
||||
|
||||
@@ -0,0 +1,113 @@
|
||||
# Signal-log live performance evaluation — 2026-08-18
|
||||
|
||||
**Tool:** `analyze_signal_log.py` (new, repo root). Read-only on
|
||||
`logs/signals.db` (sqlite `immutable=1`); replays every logged trade
|
||||
setup through the backtester's own `simulate_trades` on the RAW Oanda
|
||||
price path (next-open fill, SL/TP distances preserved, SL-first ties,
|
||||
gap-at-open fills, close-only trail ratchet with the per-setup stamped
|
||||
trail params). Reproduce:
|
||||
|
||||
```bash
|
||||
python analyze_signal_log.py --csv backtest/signal_log_eval_20260818.csv
|
||||
```
|
||||
|
||||
## Window & sample
|
||||
|
||||
- Log window: **2026-08-13 14:31 → 2026-08-18 14:13 UTC (5 days)** —
|
||||
the entire signal_log history to date.
|
||||
- 2,654 rows (telegram 2,180 / dashboard 474); M5 1,308, M15 1,189,
|
||||
H1 119, H4 38.
|
||||
- 39 rows carry a `trade_setup` → **17 unique signal events** after
|
||||
dedup (cross-source double-logs + persisting-signal re-fires collapse
|
||||
on (tf, direction, entry, SL, strategy) with a 60-min gap).
|
||||
- **No real executions exist** (`oanda_state.json` empty, no
|
||||
`oanda_daemon` rows) — every outcome below is a hypothetical replayed
|
||||
fill, not a broker fill.
|
||||
- Cost: dual convention — **$0.65 rt headline**, $0.30 comparative
|
||||
(derived analytically, R is linear in cost).
|
||||
|
||||
## Trade replay results
|
||||
|
||||
| First logged | TF | Signal | Grade | Strategy | Fill | Exit | Bars | R@0.65 |
|
||||
|---|---|---|---|---|---|---|---|---|
|
||||
| 08-13 15:30 | 5min | SELL | B | GENERIC | 4365.39 | TRAIL | 19 | −0.20 |
|
||||
| 08-13 18:00 | 15min | SELL | B | GENERIC | 4356.05 | TRAIL | 44 | +0.91 |
|
||||
| 08-14 07:45 | 5min | BUY | B | XAU_CP | 4339.72 | TRAIL | 60 | +2.56 |
|
||||
| 08-14 12:15 | 15min | BUY | B | GENERIC | 4371.91 | TRAIL | 27 | +0.06 |
|
||||
| 08-14 12:25 | 5min | STRONG_BUY | A | GENERIC | 4379.12 | TRAIL | 277 | +0.20 |
|
||||
| 08-14 12:30 | 5min | STRONG_BUY | A | GENERIC | 4379.87 | TRAIL | 276 | +0.16 |
|
||||
| 08-14 12:30 | 15min | BUY | B | GENERIC | 4379.87 | TRAIL | 63 | +0.18 |
|
||||
| 08-17 08:00 | 1h | BUY | C | GENERIC | 4405.19 | TRAIL | 18 | −0.13 |
|
||||
| 08-17 14:15 | 5min | STRONG_BUY | A | GENERIC | 4412.35 | TRAIL | 46 | −0.10 |
|
||||
| 08-17 14:20 | 5min | STRONG_BUY | A | GENERIC | 4418.55 | TRAIL | 125 | −0.23 |
|
||||
| 08-17 14:25 | 5min | STRONG_BUY | A | GENERIC | 4424.10 | SL | 151 | −1.02 |
|
||||
| 08-17 14:30 | 15min | BUY | A | GENERIC | 4418.23 | SL | 17 | −1.03 |
|
||||
| 08-17 14:45 | 15min | STRONG_BUY | A | GENERIC | 4417.49 | SL | 16 | −1.03 |
|
||||
| 08-17 15:00 | 15min | STRONG_BUY | A | GENERIC | 4417.48 | SL | 15 | −1.03 |
|
||||
| 08-17 15:00 | 5min | NEUTRAL | C | GENERIC | 4417.48 | TRAIL | 117 | −0.16 |
|
||||
| 08-17 15:15 | 15min | STRONG_BUY | A | GENERIC | 4421.30 | SL | 14 | −1.04 |
|
||||
| 08-17 15:30 | 15min | BUY | B | GENERIC | 4425.70 | SL | 85 | −1.02 |
|
||||
|
||||
**Aggregate: 17 closed, 6 wins (35%), total −2.91R @0.65 / −2.67R
|
||||
@0.30, PF 0.58.** By grade: A n=9 → −5.12R; B n=6 → +2.50R; C n=2 →
|
||||
−0.30R.
|
||||
|
||||
Reading: the book is two regimes. Everything through the 08-14 rally
|
||||
made money (+4.7R across 8 events, incl. the XAU_CP +2.56R); the 08-17
|
||||
afternoon BUY/STRONG_BUY cluster chased the spike to ~4425 and got
|
||||
stopped as price faded — 9 events, −7.6R combined. One bad afternoon
|
||||
dominates the 5-day sample. Notable: a NEUTRAL-graded row carried a
|
||||
setup (post-gate CP/PB-style path) and lost small.
|
||||
|
||||
## Advisory panel scorecard
|
||||
|
||||
Only **7 verdicts** logged so far (2 NEUTRAL, 5 OPPOSE, 0 AGREE) — the
|
||||
panel's min-quality gate + daily budget keep it dark on most scans.
|
||||
|
||||
| Row | When | Signal | Verdict | Outcome basis | Impact |
|
||||
|---|---|---|---|---|---|
|
||||
| #1358 | 08-17 07:40 M5 | BUY B | NEUTRAL @55% | +4h move | −$8.19 in dir (pass was right) |
|
||||
| #1361 | 08-17 07:45 M5 | BUY A | NEUTRAL @55% | +4h move | −$12.43 in dir (pass was right) |
|
||||
| #1550 | 08-17 14:15 M5 | STRONG_BUY A | OPPOSE @80% | replay | saved 0.10R |
|
||||
| #1553 | 08-17 14:25 M5 | STRONG_BUY A | OPPOSE @80% | replay | saved 1.02R |
|
||||
| #1555 | 08-17 14:30 M15 | BUY A | OPPOSE @75% | replay | saved 1.03R |
|
||||
| #1561 | 08-17 14:45 M15 | STRONG_BUY A | OPPOSE @75% | replay | saved 1.03R |
|
||||
| #1588 | 08-17 15:30 M15 | BUY B | OPPOSE @75% | replay | saved 1.02R |
|
||||
|
||||
**Every OPPOSE was correct: ≈ +4.2R would have been lost taking those
|
||||
five trades.** The panel's rationale ("price climbed too fast, profit
|
||||
too small") correctly identified the 08-17 exhaustion spike that the
|
||||
composite scored A. Both NEUTRAL passes also preceded adverse moves.
|
||||
Caveat: n=7, one market episode, zero AGREE verdicts — the panel has
|
||||
only proven itself as a veto so far, never as a confirmation.
|
||||
|
||||
## Directional hit-rate (non-setup signals)
|
||||
|
||||
27 deduped non-setup BUY/SELL rows, forward close-to-close on M5:
|
||||
|
||||
| Horizon | Hit rate | Mean move in direction |
|
||||
|---|---|---|
|
||||
| +1h | 48% | −$0.72 |
|
||||
| +4h | 44% | −$2.30 |
|
||||
| +24h | 7% | −$27.45 |
|
||||
|
||||
The 24h figure is dominated by the 08-17 buy-cluster reversal; not
|
||||
independently meaningful at n=27.
|
||||
|
||||
## Method notes / caveats
|
||||
|
||||
- **Small sample**: 5 days, 17 events, one trending episode + one
|
||||
reversal. Directional read only; do not tune anything on this.
|
||||
- Entry model = next bar open after the scan boundary (live scans run
|
||||
at boundary+10s), SL/TP distances preserved from the logged setup —
|
||||
identical to the backtester replay loop.
|
||||
- Cross-source dedup verified: telegram+dashboard double-logs and
|
||||
5-min re-fires of a persisting signal collapse to one event each.
|
||||
- The stray repo-root `signals.db` (828 rows, Aug 13–14, pre-panel
|
||||
schema) is an early duplicate subset of `logs/signals.db` — excluded.
|
||||
- Tests: Module 25 in `test_all.py` was only added on 2026-09-03 (the
|
||||
2026-08-18 claim of 7 tests was never true at HEAD). It now covers
|
||||
boundary flooring, clustering (shared `signal_log.setup_key`), trade
|
||||
building, the stored-candle replay books, surface parity, WAL-visible
|
||||
loading (`mode=ro`; `immutable=1` missed the newest rows) and hit-rate
|
||||
signing.
|
||||
+2
-1
@@ -59,7 +59,8 @@
|
||||
},
|
||||
"signal_log": {
|
||||
"enabled": true,
|
||||
"path": "logs/signals.db"
|
||||
"path": "logs/signals.db",
|
||||
"log_blocked_scans": true
|
||||
},
|
||||
"log_level": "INFO"
|
||||
}
|
||||
|
||||
+39
-7
@@ -253,7 +253,9 @@ def _recent_signals() -> List[Dict[str, Any]]:
|
||||
"""Read the latest signal history through SQLite's immutable URI."""
|
||||
try:
|
||||
path = os.path.abspath(signal_log_path())
|
||||
uri = f"file:{path}?immutable=1"
|
||||
# mode=ro (not immutable=1): immutable ignores the WAL, so the
|
||||
# newest rows written by the live loggers were invisible here.
|
||||
uri = f"file:{path}?mode=ro"
|
||||
con = sqlite3.connect(uri, uri=True)
|
||||
try:
|
||||
rows = con.execute(
|
||||
@@ -328,28 +330,44 @@ def _collect_payload(settings: Dict[str, Any]) -> Dict[str, Any]:
|
||||
|
||||
_state_lock = threading.Lock()
|
||||
_last_run_ts: Optional[float] = None
|
||||
_last_run_hour: Optional[int] = None # UTC hour bucket of the last attempt
|
||||
_outlook_day: Optional[_dt.date] = None
|
||||
_outlook_count = 0
|
||||
_warned_budget = False
|
||||
|
||||
|
||||
def _skip(reason: str) -> None:
|
||||
"""Heartbeat for every silent early return — a dark outlook feature
|
||||
(zero rows for 15 days, 2026-08/09) must be visible in the log."""
|
||||
logger.info("market outlook: skipped (%s)", reason)
|
||||
|
||||
|
||||
def reset_outlook_state() -> None:
|
||||
"""Drop hourly and daily state (tests / deployment config reloads)."""
|
||||
global _last_run_ts, _outlook_day, _outlook_count, _warned_budget
|
||||
global _last_run_ts, _last_run_hour, _outlook_day, _outlook_count
|
||||
global _warned_budget
|
||||
with _state_lock:
|
||||
_last_run_ts = None
|
||||
_last_run_hour = None
|
||||
_outlook_day = None
|
||||
_outlook_count = 0
|
||||
_warned_budget = False
|
||||
|
||||
|
||||
def _begin_attempt(cap: int, force: bool) -> bool:
|
||||
"""Atomically apply hourly dedup, then reserve a daily attempt."""
|
||||
global _last_run_ts, _outlook_day, _outlook_count, _warned_budget
|
||||
"""Atomically apply hourly dedup, then reserve a daily attempt.
|
||||
|
||||
Hourly dedup is by UTC hour BUCKET, not ``now - last < 3600``: the
|
||||
H1 loop calls this at boundary+10 s+scan time, so consecutive hours
|
||||
were routinely 3,590 s apart and every other hour was dropped."""
|
||||
global _last_run_ts, _last_run_hour, _outlook_day, _outlook_count
|
||||
global _warned_budget
|
||||
now = time.time()
|
||||
hour_bucket = int(now // 3600)
|
||||
today = _dt.datetime.now(_dt.timezone.utc).date()
|
||||
with _state_lock:
|
||||
if not force and _last_run_ts is not None and now - _last_run_ts < 3600:
|
||||
if not force and _last_run_hour == hour_bucket:
|
||||
_skip(f"hour bucket {hour_bucket} already attempted")
|
||||
return False
|
||||
if _outlook_day != today:
|
||||
_outlook_day = today
|
||||
@@ -361,9 +379,12 @@ def _begin_attempt(cap: int, force: bool) -> bool:
|
||||
logger.warning(
|
||||
"market outlook: daily budget (%d) exhausted — no more "
|
||||
"outlooks until UTC midnight", cap)
|
||||
else:
|
||||
_skip("daily budget exhausted")
|
||||
return False
|
||||
_outlook_count += 1
|
||||
_last_run_ts = now
|
||||
_last_run_hour = hour_bucket
|
||||
return True
|
||||
|
||||
|
||||
@@ -372,8 +393,13 @@ def run_outlook(settings: Dict[str, Any],
|
||||
"""Build, ask, validate, and persist one advisory outlook."""
|
||||
try:
|
||||
payload = _collect_payload(settings)
|
||||
result = _validate_outlook(get_llm_client().chat_json(SYSTEM_PROMPT, payload))
|
||||
reply = get_llm_client().chat_json(SYSTEM_PROMPT, payload)
|
||||
result = _validate_outlook(reply)
|
||||
if result is None:
|
||||
logger.warning(
|
||||
"market outlook: no outlook produced (%s)",
|
||||
"LLM returned nothing — cap/timeout/disabled" if reply is None
|
||||
else "reply failed validation")
|
||||
return None
|
||||
try:
|
||||
from signal_log import log_outlook
|
||||
@@ -396,13 +422,19 @@ def maybe_run_outlook(surface: str, force: bool = False) -> Optional[OutlookResu
|
||||
settings = get_app_config().market_outlook_settings
|
||||
if not force:
|
||||
if not market_outlook_enabled():
|
||||
_skip("disabled")
|
||||
return None
|
||||
if surface == "telegram" and not settings.get("telegram", False):
|
||||
_skip("telegram surface off")
|
||||
return None
|
||||
if _provider_base._is_forex_weekend():
|
||||
_skip("weekend")
|
||||
return None
|
||||
cap = int(settings.get("max_per_day", DEFAULT_MAX_PER_DAY))
|
||||
if cap < 1 or not _begin_attempt(cap, force):
|
||||
if cap < 1:
|
||||
_skip("max_per_day < 1")
|
||||
return None
|
||||
if not _begin_attempt(cap, force):
|
||||
return None
|
||||
return run_outlook(settings, surface=surface)
|
||||
except Exception:
|
||||
|
||||
@@ -0,0 +1,91 @@
|
||||
#!/usr/bin/env python3
|
||||
"""
|
||||
backfill_signal_outcomes.py — seed and advance advisory outcomes for every
|
||||
logged trade setup (not only panel rows), one per advisory event.
|
||||
|
||||
Pass A (offline, no network): assign setup_key / advisory_event_id to all
|
||||
legacy setup rows, mark same-setup re-logs within 60 min as DUPLICATE, and
|
||||
seed a PENDING outcome on each canonical row that has none. Existing
|
||||
(panel-seeded) outcomes are never overwritten.
|
||||
|
||||
Pass B: advance PENDING rows from the closed-candle streams already stored
|
||||
in the DB (union across every snapshot of the timeframe — no provider).
|
||||
|
||||
Pass C (--reseed-unresolvable): bounded re-seed of continuity-broken rows,
|
||||
then Pass B again.
|
||||
|
||||
Safe beside the live writers: short per-row transactions with a busy
|
||||
timeout. Read-only preview with --dry-run.
|
||||
|
||||
python scripts/backfill_signal_outcomes.py --dry-run
|
||||
python scripts/backfill_signal_outcomes.py
|
||||
python scripts/backfill_signal_outcomes.py --reseed-unresolvable
|
||||
"""
|
||||
import argparse
|
||||
import os
|
||||
import sqlite3
|
||||
import sys
|
||||
|
||||
sys.path.insert(0, os.path.dirname(os.path.dirname(os.path.abspath(__file__))))
|
||||
|
||||
import app_config # noqa: E402
|
||||
from signal_log import SignalLogger # noqa: E402
|
||||
|
||||
|
||||
def _status_counts(con: sqlite3.Connection) -> dict:
|
||||
return dict(con.execute(
|
||||
"SELECT COALESCE(advisory_outcome_status, 'NULL'), COUNT(*)"
|
||||
" FROM signals WHERE setup_json IS NOT NULL AND setup_json != 'null'"
|
||||
" GROUP BY 1").fetchall())
|
||||
|
||||
|
||||
def main(argv=None) -> int:
|
||||
ap = argparse.ArgumentParser(description=__doc__.splitlines()[1])
|
||||
ap.add_argument("--db", default=None, help="defaults to signal_log.path")
|
||||
ap.add_argument("--dry-run", action="store_true")
|
||||
ap.add_argument("--reseed-unresolvable", action="store_true")
|
||||
ap.add_argument("--symbol", default="XAU/USD")
|
||||
args = ap.parse_args(argv)
|
||||
path = args.db or app_config.signal_log_path()
|
||||
if not os.path.exists(path):
|
||||
print(f"no database at {path}")
|
||||
return 1
|
||||
if args.dry_run:
|
||||
con = sqlite3.connect(f"file:{os.path.abspath(path)}?mode=ro", uri=True)
|
||||
print("setup rows by outcome status:", _status_counts(con))
|
||||
have = {r[1] for r in con.execute("PRAGMA table_info(signals)")}
|
||||
if "setup_key" in have:
|
||||
n_keyless = con.execute(
|
||||
"SELECT COUNT(*) FROM signals WHERE setup_json IS NOT NULL"
|
||||
" AND setup_json != 'null' AND setup_key IS NULL").fetchone()[0]
|
||||
print(f"setup rows without setup_key: {n_keyless}")
|
||||
else:
|
||||
print("setup_key column not present yet (schema migrates on"
|
||||
" the first non-dry run)")
|
||||
con.close()
|
||||
return 0
|
||||
lg = SignalLogger(path)
|
||||
try:
|
||||
print("before:", _status_counts(lg._conn))
|
||||
print("pass A (seed/dedup):", lg.backfill_seeds())
|
||||
total = 0
|
||||
for _ in range(50):
|
||||
n = lg.resolve_from_store(args.symbol)
|
||||
total += n
|
||||
if n == 0:
|
||||
break
|
||||
print(f"pass B (advance from stored bars): {total} rows changed")
|
||||
if args.reseed_unresolvable:
|
||||
reseeded = lg.reseed_unresolvable()
|
||||
print(f"pass C (re-seed): {len(reseeded)} rows re-seeded")
|
||||
for _ in range(50):
|
||||
if lg.resolve_from_store(args.symbol) == 0:
|
||||
break
|
||||
print("after:", _status_counts(lg._conn))
|
||||
finally:
|
||||
lg.close()
|
||||
return 0
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
sys.exit(main())
|
||||
@@ -0,0 +1,171 @@
|
||||
#!/usr/bin/env python3
|
||||
"""
|
||||
migrate_signal_log_news.py — move inline news blobs into news_snapshots.
|
||||
|
||||
Legacy rows store the full RSS headline list (~33 KB) and calendar events
|
||||
(~1.2 KB) inline on EVERY scan row; the same blob repeats ~5× because the
|
||||
news caches outlive a candle. This script interns each distinct blob once
|
||||
(content hash), stamps `headlines_hash` / `events_hash` on the row and
|
||||
NULLs the inline columns. Readers use the `signals_with_news` view or
|
||||
`signal_log.read_news`.
|
||||
|
||||
Idempotent and resumable (batched transactions). Run --vacuum with BOTH
|
||||
live writers (telegram bot, dashboard publisher) stopped — VACUUM needs
|
||||
exclusive access and ~1× the final size of free disk.
|
||||
|
||||
python scripts/migrate_signal_log_news.py --dry-run
|
||||
python scripts/migrate_signal_log_news.py
|
||||
python scripts/migrate_signal_log_news.py --vacuum # bots stopped
|
||||
"""
|
||||
import argparse
|
||||
import hashlib
|
||||
import os
|
||||
import sqlite3
|
||||
import sys
|
||||
from datetime import datetime, timezone
|
||||
|
||||
sys.path.insert(0, os.path.dirname(os.path.dirname(os.path.abspath(__file__))))
|
||||
|
||||
import app_config # noqa: E402
|
||||
from signal_log import SignalLogger # noqa: E402
|
||||
|
||||
|
||||
def _hash(text: str) -> str:
|
||||
return hashlib.sha256(text.encode("utf-8")).hexdigest()
|
||||
|
||||
|
||||
def migrate(con: sqlite3.Connection, batch: int = 500) -> dict:
|
||||
"""Intern legacy inline blobs; returns counters. Caller owns the
|
||||
connection (schema must already exist — open a SignalLogger once)."""
|
||||
stats = {"rows": 0, "headlines_interned": 0, "events_interned": 0}
|
||||
now = datetime.now(timezone.utc).isoformat()
|
||||
while True:
|
||||
rows = con.execute(
|
||||
"SELECT id, headlines_json, events_json FROM signals"
|
||||
" WHERE (headlines_json IS NOT NULL AND headlines_hash IS NULL)"
|
||||
" OR (events_json IS NOT NULL AND events_hash IS NULL)"
|
||||
" ORDER BY id LIMIT ?", (batch,)).fetchall()
|
||||
if not rows:
|
||||
break
|
||||
con.execute("BEGIN IMMEDIATE")
|
||||
try:
|
||||
for rowid, headlines, events in rows:
|
||||
sets, params = [], []
|
||||
if headlines is not None:
|
||||
h = _hash(headlines)
|
||||
cur = con.execute(
|
||||
"INSERT OR IGNORE INTO news_snapshots"
|
||||
" (hash, kind, body, first_seen, last_seen)"
|
||||
" VALUES (?, 'headlines', ?, ?, ?)",
|
||||
(h, headlines, now, now))
|
||||
stats["headlines_interned"] += cur.rowcount
|
||||
sets += ["headlines_hash=?", "headlines_json=NULL"]
|
||||
params.append(h)
|
||||
if events is not None:
|
||||
h = _hash(events)
|
||||
cur = con.execute(
|
||||
"INSERT OR IGNORE INTO news_snapshots"
|
||||
" (hash, kind, body, first_seen, last_seen)"
|
||||
" VALUES (?, 'events', ?, ?, ?)",
|
||||
(h, events, now, now))
|
||||
stats["events_interned"] += cur.rowcount
|
||||
sets += ["events_hash=?", "events_json=NULL"]
|
||||
params.append(h)
|
||||
params.append(rowid)
|
||||
con.execute(
|
||||
f"UPDATE signals SET {', '.join(sets)} WHERE id=?",
|
||||
params)
|
||||
stats["rows"] += 1
|
||||
con.commit()
|
||||
except Exception:
|
||||
con.rollback()
|
||||
raise
|
||||
return stats
|
||||
|
||||
|
||||
def prune_orphans(con: sqlite3.Connection) -> int:
|
||||
cur = con.execute(
|
||||
"DELETE FROM news_snapshots WHERE hash NOT IN"
|
||||
" (SELECT headlines_hash FROM signals WHERE headlines_hash IS NOT NULL"
|
||||
" UNION SELECT events_hash FROM signals WHERE events_hash IS NOT NULL)")
|
||||
con.commit()
|
||||
return cur.rowcount
|
||||
|
||||
|
||||
def prune_news_older_than(con: sqlite3.Connection, days: int) -> int:
|
||||
"""Drop the news reference (not the row) from rows older than N days,
|
||||
then delete snapshots nothing references any more."""
|
||||
cur = con.execute(
|
||||
"UPDATE signals SET headlines_hash=NULL, events_hash=NULL"
|
||||
" WHERE logged_at < datetime('now', ?)", (f"-{int(days)} days",))
|
||||
con.commit()
|
||||
prune_orphans(con)
|
||||
return cur.rowcount
|
||||
|
||||
|
||||
def main(argv=None) -> int:
|
||||
ap = argparse.ArgumentParser(description=__doc__.splitlines()[1])
|
||||
ap.add_argument("--db", default=None)
|
||||
ap.add_argument("--dry-run", action="store_true")
|
||||
ap.add_argument("--batch", type=int, default=500)
|
||||
ap.add_argument("--prune-news-older-than", type=int, default=None,
|
||||
metavar="DAYS")
|
||||
ap.add_argument("--prune-orphans", action="store_true")
|
||||
ap.add_argument("--vacuum", action="store_true",
|
||||
help="checkpoint + VACUUM (all writers must be stopped)")
|
||||
ap.add_argument("--no-backup", action="store_true")
|
||||
args = ap.parse_args(argv)
|
||||
path = args.db or app_config.signal_log_path()
|
||||
if not os.path.exists(path):
|
||||
print(f"no database at {path}")
|
||||
return 1
|
||||
size0 = os.path.getsize(path)
|
||||
if args.dry_run:
|
||||
con = sqlite3.connect(f"file:{os.path.abspath(path)}?mode=ro", uri=True)
|
||||
n = con.execute(
|
||||
"SELECT COUNT(*) FROM signals WHERE headlines_json IS NOT NULL"
|
||||
" OR events_json IS NOT NULL").fetchone()[0]
|
||||
d = con.execute(
|
||||
"SELECT COUNT(DISTINCT headlines_json) FROM signals"
|
||||
" WHERE headlines_json IS NOT NULL").fetchone()[0]
|
||||
print(f"{n} rows carry inline news blobs ({d} distinct headline "
|
||||
f"blobs); file size {size0 / 1e6:.1f} MB")
|
||||
con.close()
|
||||
return 0
|
||||
if not args.no_backup:
|
||||
backup = f"{path}.pre-news-{datetime.now(timezone.utc):%Y%m%d}"
|
||||
if not os.path.exists(backup):
|
||||
src = sqlite3.connect(path)
|
||||
dst = sqlite3.connect(backup)
|
||||
src.backup(dst)
|
||||
dst.close()
|
||||
src.close()
|
||||
print(f"backup written: {backup}")
|
||||
lg = SignalLogger(path) # applies the additive schema
|
||||
lg.close()
|
||||
con = sqlite3.connect(path)
|
||||
con.execute("PRAGMA busy_timeout=10000")
|
||||
try:
|
||||
print("migrate:", migrate(con, batch=args.batch))
|
||||
if args.prune_news_older_than is not None:
|
||||
print("rows de-referenced:",
|
||||
prune_news_older_than(con, args.prune_news_older_than))
|
||||
if args.prune_orphans:
|
||||
print("orphan snapshots removed:", prune_orphans(con))
|
||||
if args.vacuum:
|
||||
try:
|
||||
con.execute("PRAGMA wal_checkpoint(TRUNCATE)")
|
||||
con.execute("VACUUM")
|
||||
except sqlite3.OperationalError as exc:
|
||||
print(f"VACUUM failed ({exc}) — stop the bot and dashboard "
|
||||
"publisher first, then re-run with --vacuum")
|
||||
return 2
|
||||
finally:
|
||||
con.close()
|
||||
print(f"size {size0 / 1e6:.1f} MB -> {os.path.getsize(path) / 1e6:.1f} MB"
|
||||
+ ("" if args.vacuum else " (run --vacuum to reclaim space)"))
|
||||
return 0
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
sys.exit(main())
|
||||
+68
-10
@@ -5,6 +5,8 @@ Combines multi-timeframe technical analysis with optional news context
|
||||
into a graded actionable signal.
|
||||
"""
|
||||
|
||||
import hashlib
|
||||
import json
|
||||
import logging
|
||||
import os
|
||||
from dataclasses import dataclass, field
|
||||
@@ -116,6 +118,9 @@ class SignalReport:
|
||||
bullish_reasons: List[str] = field(default_factory=list)
|
||||
bearish_reasons: List[str] = field(default_factory=list)
|
||||
warnings: List[str] = field(default_factory=list)
|
||||
# Warnings the XAU_CP acceptance removed from `warnings` (metadata
|
||||
# only — logged so the gates a CP trade bypassed stay auditable).
|
||||
cleaned_warnings: List[str] = field(default_factory=list)
|
||||
|
||||
current_price: float = 0.0
|
||||
spread_estimate: float = 0.0
|
||||
@@ -178,7 +183,7 @@ class SignalAggregator:
|
||||
report.trade_setup = None
|
||||
report.warnings.append("WEEKEND FILTER: XAU/USD market closed — no liquidity")
|
||||
report.action_recommendation = "NO TRADE — Weekend (market closed)"
|
||||
self._log_signal(report, log_source)
|
||||
self._log_signal(report, log_source, blocked=True)
|
||||
return report
|
||||
|
||||
if xau_cfg.allowed_weekdays and now_utc.weekday() not in xau_cfg.allowed_weekdays:
|
||||
@@ -196,7 +201,7 @@ class SignalAggregator:
|
||||
f"DAY FILTER: {day_names.get(now_utc.weekday(), '?')} not in allowed days ({allowed})"
|
||||
)
|
||||
report.action_recommendation = f"NO TRADE — Only trading {allowed}"
|
||||
self._log_signal(report, log_source)
|
||||
self._log_signal(report, log_source, blocked=True)
|
||||
return report
|
||||
|
||||
timeframes = [self.config.primary_timeframe] + list(
|
||||
@@ -280,6 +285,7 @@ class SignalAggregator:
|
||||
def _log_signal(
|
||||
self, report: SignalReport, source: Optional[str],
|
||||
dataframes: Optional[Dict[Timeframe, pd.DataFrame]] = None,
|
||||
blocked: bool = False,
|
||||
) -> None:
|
||||
"""Persist the scan to the signal-history DB (signal_log.py).
|
||||
|
||||
@@ -287,12 +293,21 @@ class SignalAggregator:
|
||||
live scan. The news context re-fetch rides the NewsEngine's
|
||||
class-level caches, so it costs nothing within a scan and
|
||||
matches the context the signal just used; if it fails, the row
|
||||
is still written, just without the news JSON blobs."""
|
||||
is still written, just without the news JSON blobs. ``blocked``
|
||||
marks weekend/day early returns: no news fetch, no snapshots —
|
||||
a ~1 KB liveness row (or nothing, per signal_log.log_blocked_scans)."""
|
||||
context = None
|
||||
if not blocked:
|
||||
try:
|
||||
context = self._get_news_engine().get_fundamental_context()
|
||||
except Exception:
|
||||
logger.debug("signal log: news context unavailable",
|
||||
exc_info=True)
|
||||
extra = {"scan_kind": "blocked" if blocked else "scan"}
|
||||
try:
|
||||
context = self._get_news_engine().get_fundamental_context()
|
||||
extra["config_fingerprint"] = self.gate_config_fingerprint()
|
||||
except Exception:
|
||||
logger.debug("signal log: news context unavailable",
|
||||
logger.debug("signal log: config fingerprint unavailable",
|
||||
exc_info=True)
|
||||
signal_log.log_signal(
|
||||
report,
|
||||
@@ -300,8 +315,38 @@ class SignalAggregator:
|
||||
source=source or "live",
|
||||
timeframe=self.config.primary_timeframe.value,
|
||||
dataframes=dataframes,
|
||||
extra=extra,
|
||||
)
|
||||
|
||||
def gate_config_fingerprint(self) -> str:
|
||||
"""Short hash of the gate configuration this process runs with.
|
||||
|
||||
Logged on every row so two surfaces disagreeing on the same bar
|
||||
(2026-08-19: the dashboard process applied a different blocked-
|
||||
hours list than the bot for a day) is diagnosable from the DB."""
|
||||
xs = self.config.xau_strategy
|
||||
bt = self.config.backtest
|
||||
payload = {
|
||||
"primary": self.config.primary_timeframe.value,
|
||||
"confirm": [tf.value for tf in self.config.confirmation_timeframes],
|
||||
"blocked_hours": list(xs.blocked_hours_utc),
|
||||
"sell_blocked_hours": list(xs.sell_blocked_hours_utc),
|
||||
"allowed_weekdays": list(xs.allowed_weekdays),
|
||||
"blocked_regimes": list(xs.blocked_regimes),
|
||||
"sell_blocked_regimes": list(xs.sell_blocked_regimes),
|
||||
"sell_min_quality": list(xs.sell_min_quality),
|
||||
"sell_vol_regimes": list(xs.sell_vol_regimes),
|
||||
"big_candle": xs.big_candle_atr_multiple,
|
||||
"cp": bool(xs.cp_enabled),
|
||||
"pb": bool(xs.m5_pullback_enabled),
|
||||
"session": [bt.session_filter, bt.session_start_hour,
|
||||
bt.session_end_hour],
|
||||
"regime_filter": bool(bt.regime_filter),
|
||||
}
|
||||
return hashlib.sha1(
|
||||
json.dumps(payload, sort_keys=True, default=str).encode("utf-8")
|
||||
).hexdigest()[:12]
|
||||
|
||||
def _apply_regime_filter_live(
|
||||
self, report: SignalReport, dfs: Dict[Timeframe, pd.DataFrame]
|
||||
) -> None:
|
||||
@@ -744,9 +789,9 @@ class SignalAggregator:
|
||||
close_h = None
|
||||
logger.warning(
|
||||
f"{report.symbol}: bar-close hour underivable — "
|
||||
f"SELL hour gate skipped"
|
||||
f"CP SELL hour gate FAILS CLOSED (short rejected)"
|
||||
)
|
||||
if close_h in cfg.sell_blocked_hours_utc:
|
||||
if close_h is None or close_h in cfg.sell_blocked_hours_utc:
|
||||
_reject("sell_hour")
|
||||
return
|
||||
if self.config.primary_timeframe == Timeframe.M5:
|
||||
@@ -851,14 +896,18 @@ class SignalAggregator:
|
||||
"Weak pullback",
|
||||
)
|
||||
filtered: List[str] = []
|
||||
dropped: List[str] = []
|
||||
for warning in report.warnings:
|
||||
if warning.startswith(stale_prefixes):
|
||||
dropped.append(warning)
|
||||
continue
|
||||
if (warning.startswith("REGIME FILTER: '")
|
||||
and warning.endswith("blocked for XAU")):
|
||||
dropped.append(warning)
|
||||
continue
|
||||
if (warning.startswith("M5 FILTER: '")
|
||||
and "volatility" in warning):
|
||||
dropped.append(warning)
|
||||
continue
|
||||
filtered.append(warning)
|
||||
|
||||
@@ -869,6 +918,7 @@ class SignalAggregator:
|
||||
]
|
||||
if not extension_items:
|
||||
report.warnings[:] = filtered
|
||||
report.cleaned_warnings.extend(dropped)
|
||||
return
|
||||
first_index = extension_items[0][0]
|
||||
priority = {prefix: rank for rank, prefix in enumerate(extension_prefixes)}
|
||||
@@ -883,7 +933,10 @@ class SignalAggregator:
|
||||
if not warning.startswith(extension_prefixes)
|
||||
]
|
||||
kept.insert(first_index, strongest)
|
||||
dropped.extend(
|
||||
warning for _, warning in extension_items if warning != strongest)
|
||||
report.warnings[:] = kept
|
||||
report.cleaned_warnings.extend(dropped)
|
||||
|
||||
# ── Public: bar-replayable entry point ─────────────────────────
|
||||
|
||||
@@ -1116,11 +1169,16 @@ class SignalAggregator:
|
||||
close_h = None
|
||||
logger.warning(
|
||||
f"{report.symbol}: bar-close hour underivable — "
|
||||
f"SELL hour gate skipped"
|
||||
f"SELL hour gate FAILS CLOSED (short blocked)"
|
||||
)
|
||||
if close_h in self.config.xau_strategy.sell_blocked_hours_utc:
|
||||
# Fail CLOSED: an underivable close hour blocks the short
|
||||
# (`None in [15, 17]` used to pass silently). Never fires
|
||||
# on parseable timestamps, so pinned books are unchanged.
|
||||
if (close_h is None
|
||||
or close_h in self.config.xau_strategy.sell_blocked_hours_utc):
|
||||
hour_txt = "??" if close_h is None else f"{close_h:02}"
|
||||
report.warnings.append(
|
||||
f"SELL FILTER: {close_h:02}:00 UTC entry blocked for shorts"
|
||||
f"SELL FILTER: {hour_txt}:00 UTC entry blocked for shorts"
|
||||
)
|
||||
report.signal_quality = "D"
|
||||
report.trade_setup = None
|
||||
|
||||
+632
-85
@@ -27,16 +27,47 @@ Fail-safe contract: logging must NEVER break a scan. Every public entry
|
||||
point catches all exceptions and downgrades them to a warning log line.
|
||||
Report/context objects are duck-typed (no imports from signal_aggregator
|
||||
or news_engine) so this module stays at the bottom of the import graph.
|
||||
|
||||
2026-09 observability additions (signal-log audit):
|
||||
|
||||
* Every row with a trade setup is an advisory-outcome candidate, not only
|
||||
panel rows. Re-logs of the same setup (same ``setup_key`` — tf,
|
||||
direction, entry, stop, strategy — within OUTCOME_DEDUP_GAP_SECONDS,
|
||||
from ANY surface) are marked ``DUPLICATE`` and point at the canonical
|
||||
row through ``advisory_event_id``; only the canonical row is seeded and
|
||||
advanced, so summing ``advisory_r_multiple`` over
|
||||
``advisory_outcome_status='RESOLVED'`` counts each trade once.
|
||||
* Pending outcomes advance from the inserting scan's own snapshot first;
|
||||
when that snapshot no longer overlaps the outcome's last processed bar
|
||||
(downtime, weekend straddle) the resolver rebuilds the bar stream from
|
||||
every stored snapshot of that timeframe, so a >14-bar gap in one
|
||||
process's scans is not a terminal ``UNRESOLVABLE``.
|
||||
* News blobs are content-addressed: ``headlines_json``/``events_json`` are
|
||||
interned once into ``news_snapshots`` and rows carry ``headlines_hash``
|
||||
/ ``events_hash`` (the legacy columns stay NULL on new rows; the
|
||||
``signals_with_news`` view and ``read_news`` resolve both eras).
|
||||
* Post-gate truth columns: ``gate_reason`` (the warning that suppressed
|
||||
the setup), ``actionable`` (grade >= C AND setup present — what an
|
||||
alert surface would act on; the ``action`` text is written BEFORE the
|
||||
gate chain and can disagree), ``cleaned_warnings_json`` (warnings the
|
||||
XAU_CP acceptance removed from ``warnings_json``), ``bar_range_median``
|
||||
(the value historically stored as ``spread`` — it is the median
|
||||
High-Low of the primary frame, NOT a broker spread), ``scan_kind``
|
||||
(``scan`` / ``blocked`` for weekend/day early returns, which carry no
|
||||
news or candles) and ``config_fingerprint`` (hash of the gate config the
|
||||
producing process ran with — surfaces config drift between processes).
|
||||
"""
|
||||
|
||||
import hashlib
|
||||
import json
|
||||
import logging
|
||||
import math
|
||||
import os
|
||||
import sqlite3
|
||||
import threading
|
||||
import time
|
||||
from datetime import datetime, timezone
|
||||
from typing import Any, Dict, List, Optional
|
||||
from typing import Any, Dict, Iterable, List, Optional, Tuple
|
||||
|
||||
import app_config
|
||||
from advisory_outcome import (
|
||||
@@ -90,6 +121,13 @@ CREATE TABLE IF NOT EXISTS signals (
|
||||
);
|
||||
CREATE INDEX IF NOT EXISTS idx_signals_sym_tf_ts
|
||||
ON signals (symbol, timeframe, logged_at);
|
||||
CREATE TABLE IF NOT EXISTS news_snapshots (
|
||||
hash TEXT PRIMARY KEY,
|
||||
kind TEXT NOT NULL,
|
||||
body TEXT NOT NULL,
|
||||
first_seen TEXT NOT NULL,
|
||||
last_seen TEXT
|
||||
);
|
||||
CREATE TABLE IF NOT EXISTS outlooks (
|
||||
id INTEGER PRIMARY KEY AUTOINCREMENT,
|
||||
logged_at TEXT NOT NULL,
|
||||
@@ -120,25 +158,72 @@ _ADDITIVE_SIGNAL_COLS = (
|
||||
("advisory_outcome_status", "TEXT"),
|
||||
("advisory_r_multiple", "REAL"),
|
||||
("advisory_outcome_json", "TEXT"),
|
||||
# 2026-09 observability columns (see module docstring).
|
||||
("setup_key", "TEXT"),
|
||||
("advisory_event_id", "INTEGER"),
|
||||
("headlines_hash", "TEXT"),
|
||||
("events_hash", "TEXT"),
|
||||
("gate_reason", "TEXT"),
|
||||
("actionable", "INTEGER"),
|
||||
("cleaned_warnings_json", "TEXT"),
|
||||
("bar_range_median", "REAL"),
|
||||
("scan_kind", "TEXT"),
|
||||
("config_fingerprint", "TEXT"),
|
||||
)
|
||||
|
||||
# Read-side compatibility view `signals_with_news`: legacy rows keep
|
||||
# their inline news blobs, new rows resolve them through news_snapshots.
|
||||
# Recreated on every logger init (SignalLogger._ensure_news_view) so it
|
||||
# always exposes the current column set.
|
||||
|
||||
# Same-setup re-logs (cross-surface double logs, persisting-signal
|
||||
# re-fires) within this gap are ONE advisory event. Mirrors
|
||||
# analyze_signal_log's 60-minute clustering.
|
||||
OUTCOME_DEDUP_GAP_SECONDS = 3600
|
||||
|
||||
# Warning prefixes that mean "a gate removed the setup", checked in gate
|
||||
# order (live wrapper first, then the post-grade chain, then the setup
|
||||
# generators). Kept here — signal_log must not import signal_aggregator.
|
||||
_GATE_PREFIXES = (
|
||||
"WEEKEND FILTER",
|
||||
"DAY FILTER",
|
||||
"SESSION FILTER",
|
||||
"Primary timeframe stale",
|
||||
"Primary timeframe data unavailable",
|
||||
"REGIME FILTER",
|
||||
"EXHAUSTION BLOCK",
|
||||
"EXHAUSTED TREND FILTER",
|
||||
"SELL FILTER",
|
||||
"MTF OPPOSITION",
|
||||
"BIG CANDLE FILTER",
|
||||
"M5 FILTER",
|
||||
"LOW ATR FILTER",
|
||||
"SPREAD GATE",
|
||||
)
|
||||
_ACTIONABLE_GRADES = ("A+", "A", "B", "C")
|
||||
|
||||
_ADDITIVE_OUTLOOK_COLS = (
|
||||
("news_json", "TEXT"),
|
||||
("indicators_json", "TEXT"),
|
||||
("candles_json", "TEXT"),
|
||||
)
|
||||
|
||||
_INSERT = """
|
||||
INSERT INTO signals (
|
||||
logged_at, source, symbol, timeframe, signal, score, confidence,
|
||||
quality, action, price, spread, market_regime, volatility_regime,
|
||||
mtf_json, setup_json, reasons_json, warnings_json,
|
||||
event_risk, event_description, news_sentiment, news_description,
|
||||
news_summary, events_json, headlines_json,
|
||||
indicators_json, candles_json,
|
||||
panel_verdict, panel_confidence, panel_rationale, panel_takes_json
|
||||
) VALUES (?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?)
|
||||
"""
|
||||
_SIGNAL_INSERT_COLS = (
|
||||
"logged_at", "source", "symbol", "timeframe", "signal", "score",
|
||||
"confidence", "quality", "action", "price", "spread", "market_regime",
|
||||
"volatility_regime", "mtf_json", "setup_json", "reasons_json",
|
||||
"warnings_json", "event_risk", "event_description", "news_sentiment",
|
||||
"news_description", "news_summary", "events_json", "headlines_json",
|
||||
"indicators_json", "candles_json", "panel_verdict", "panel_confidence",
|
||||
"panel_rationale", "panel_takes_json",
|
||||
"setup_key", "headlines_hash", "events_hash", "gate_reason",
|
||||
"actionable", "cleaned_warnings_json", "bar_range_median", "scan_kind",
|
||||
"config_fingerprint",
|
||||
)
|
||||
_INSERT = (
|
||||
"INSERT INTO signals (" + ", ".join(_SIGNAL_INSERT_COLS) + ") VALUES ("
|
||||
+ ", ".join("?" for _ in _SIGNAL_INSERT_COLS) + ")"
|
||||
)
|
||||
|
||||
_OUTLOOK_INSERT = """
|
||||
INSERT INTO outlooks (
|
||||
@@ -173,6 +258,57 @@ def _dumps(obj: Any) -> str:
|
||||
return json.dumps(obj, default=str, allow_nan=False)
|
||||
|
||||
|
||||
def _blob_hash(text: str) -> str:
|
||||
return hashlib.sha256(text.encode("utf-8")).hexdigest()
|
||||
|
||||
|
||||
def setup_key(setup: Optional[Dict[str, Any]], timeframe: Any) -> Optional[str]:
|
||||
"""Identity of a logged trade setup for event dedup.
|
||||
|
||||
``"<tf>|<direction>|<entry>|<stop>|<strategy>"`` — the same tuple
|
||||
analyze_signal_log clusters on (tf, direction, round(entry, 2),
|
||||
round(SL, 2), strategy). None when the setup lacks a usable price."""
|
||||
if not isinstance(setup, dict):
|
||||
return None
|
||||
entry = _finite(setup.get("entry"))
|
||||
stop = _finite(setup.get("stop_loss"))
|
||||
if entry is None or stop is None:
|
||||
return None
|
||||
return "|".join((
|
||||
str(getattr(timeframe, "value", timeframe) or ""),
|
||||
str(setup.get("direction") or ""),
|
||||
f"{entry:.2f}",
|
||||
f"{stop:.2f}",
|
||||
str(setup.get("strategy") or ""),
|
||||
))
|
||||
|
||||
|
||||
def gate_reason(
|
||||
warnings: Iterable[str],
|
||||
cleaned_warnings: Iterable[str],
|
||||
quality: str,
|
||||
has_setup: bool,
|
||||
) -> Optional[str]:
|
||||
"""The warning that explains why a row carries no setup, or None.
|
||||
|
||||
Rows WITH a setup have no gate reason. Rows without one report the
|
||||
first gate-prefixed warning (in gate order); an actionable grade with
|
||||
no setup and no gate warning is the setup generator returning None
|
||||
(RR floor / geometry), which the aggregator does not warn about."""
|
||||
if has_setup:
|
||||
return None
|
||||
pool = list(warnings or []) + list(cleaned_warnings or [])
|
||||
for prefix in _GATE_PREFIXES:
|
||||
for warning in pool:
|
||||
if str(warning).startswith(prefix):
|
||||
return str(warning)
|
||||
if quality in _ACTIONABLE_GRADES:
|
||||
return "SETUP GENERATOR: no setup (RR floor / geometry)"
|
||||
if quality:
|
||||
return f"GRADE {quality}: below setup threshold"
|
||||
return None
|
||||
|
||||
|
||||
# ── Serializers (duck-typed; deliberately NOT imported from ──
|
||||
# ── dashboard_publisher — the dashboard's hard independence rule) ──
|
||||
|
||||
@@ -322,7 +458,12 @@ def _report_row(
|
||||
source: str,
|
||||
timeframe: str,
|
||||
dataframes: Any = None,
|
||||
) -> tuple:
|
||||
extra: Optional[Dict[str, Any]] = None,
|
||||
) -> Dict[str, Any]:
|
||||
"""Column-name → value mapping for one scan row (see
|
||||
_SIGNAL_INSERT_COLS). ``extra`` carries producer-side metadata
|
||||
(``scan_kind``, ``config_fingerprint``)."""
|
||||
extra = extra or {}
|
||||
mtf_signals = getattr(report, "mtf_signals", None) or {}
|
||||
mtf_json = _dumps({
|
||||
"agreement": bool(getattr(report, "mtf_agreement", False)),
|
||||
@@ -363,38 +504,56 @@ def _report_row(
|
||||
signal = getattr(report, "final_signal", None)
|
||||
panel_takes = getattr(report, "panel_takes", None) or []
|
||||
panel_takes_json = _dumps(panel_takes) if panel_takes else None
|
||||
return (
|
||||
datetime.now(timezone.utc).isoformat(),
|
||||
source,
|
||||
str(getattr(report, "symbol", "")),
|
||||
timeframe,
|
||||
getattr(signal, "name", str(signal)),
|
||||
_finite(getattr(report, "final_score", None)),
|
||||
_finite(getattr(report, "final_confidence", None)),
|
||||
getattr(report, "signal_quality", "") or "",
|
||||
getattr(report, "action_recommendation", "") or "",
|
||||
_finite(getattr(report, "current_price", None)),
|
||||
_finite(getattr(report, "spread_estimate", None)),
|
||||
getattr(report, "market_regime", "") or "",
|
||||
getattr(report, "volatility_regime", "") or "",
|
||||
mtf_json,
|
||||
setup_json,
|
||||
reasons_json,
|
||||
warnings_json,
|
||||
getattr(report, "event_risk", "") or "",
|
||||
getattr(report, "event_description", "") or "",
|
||||
_finite(getattr(report, "news_sentiment", None)),
|
||||
getattr(report, "news_description", "") or "",
|
||||
getattr(report, "news_summary", "") or "",
|
||||
events_json,
|
||||
headlines_json,
|
||||
_report_indicators_json(report),
|
||||
_candles_json(dataframes),
|
||||
getattr(report, "panel_verdict", None),
|
||||
_finite(getattr(report, "panel_confidence", None)),
|
||||
getattr(report, "panel_rationale", None),
|
||||
panel_takes_json,
|
||||
)
|
||||
quality = getattr(report, "signal_quality", "") or ""
|
||||
warnings = list(getattr(report, "warnings", None) or [])
|
||||
cleaned = list(getattr(report, "cleaned_warnings", None) or [])
|
||||
setup_dict = json.loads(setup_json) if setup_json else None
|
||||
bar_range = _finite(getattr(report, "spread_estimate", None))
|
||||
return {
|
||||
"logged_at": datetime.now(timezone.utc).isoformat(),
|
||||
"source": source,
|
||||
"symbol": str(getattr(report, "symbol", "")),
|
||||
"timeframe": timeframe,
|
||||
"signal": getattr(signal, "name", str(signal)),
|
||||
"score": _finite(getattr(report, "final_score", None)),
|
||||
"confidence": _finite(getattr(report, "final_confidence", None)),
|
||||
"quality": quality,
|
||||
"action": getattr(report, "action_recommendation", "") or "",
|
||||
"price": _finite(getattr(report, "current_price", None)),
|
||||
"spread": bar_range,
|
||||
"market_regime": getattr(report, "market_regime", "") or "",
|
||||
"volatility_regime": getattr(report, "volatility_regime", "") or "",
|
||||
"mtf_json": mtf_json,
|
||||
"setup_json": setup_json,
|
||||
"reasons_json": reasons_json,
|
||||
"warnings_json": warnings_json,
|
||||
"event_risk": getattr(report, "event_risk", "") or "",
|
||||
"event_description": getattr(report, "event_description", "") or "",
|
||||
"news_sentiment": _finite(getattr(report, "news_sentiment", None)),
|
||||
"news_description": getattr(report, "news_description", "") or "",
|
||||
"news_summary": getattr(report, "news_summary", "") or "",
|
||||
# Interned by SignalLogger.log into news_snapshots; the inline
|
||||
# columns stay NULL on new rows (legacy rows keep theirs).
|
||||
"events_json": events_json,
|
||||
"headlines_json": headlines_json,
|
||||
"indicators_json": _report_indicators_json(report),
|
||||
"candles_json": _candles_json(dataframes),
|
||||
"panel_verdict": getattr(report, "panel_verdict", None),
|
||||
"panel_confidence": _finite(getattr(report, "panel_confidence", None)),
|
||||
"panel_rationale": getattr(report, "panel_rationale", None),
|
||||
"panel_takes_json": panel_takes_json,
|
||||
"setup_key": setup_key(setup_dict, timeframe),
|
||||
"headlines_hash": None,
|
||||
"events_hash": None,
|
||||
"gate_reason": gate_reason(warnings, cleaned, quality,
|
||||
setup_dict is not None),
|
||||
"actionable": int(setup_dict is not None
|
||||
and quality in _ACTIONABLE_GRADES),
|
||||
"cleaned_warnings_json": _dumps(cleaned) if cleaned else None,
|
||||
"bar_range_median": bar_range,
|
||||
"scan_kind": str(extra.get("scan_kind") or "scan"),
|
||||
"config_fingerprint": extra.get("config_fingerprint"),
|
||||
}
|
||||
|
||||
|
||||
class SignalLogger:
|
||||
@@ -408,13 +567,53 @@ class SignalLogger:
|
||||
if parent:
|
||||
os.makedirs(parent, exist_ok=True)
|
||||
self._conn = sqlite3.connect(path, check_same_thread=False)
|
||||
self._conn.execute("PRAGMA journal_mode=WAL")
|
||||
# busy_timeout BEFORE the journal-mode switch: several processes
|
||||
# (bot, dashboard, tests) open the same file at once and the WAL
|
||||
# switch plus the additive migration need brief exclusive locks.
|
||||
self._conn.execute("PRAGMA busy_timeout=5000")
|
||||
self._lock = threading.Lock()
|
||||
with self._lock:
|
||||
self._conn.executescript(_SCHEMA)
|
||||
self._migrate_signal_columns()
|
||||
self._migrate_outlook_columns()
|
||||
self._with_lock_retry(
|
||||
lambda: self._conn.execute("PRAGMA journal_mode=WAL"))
|
||||
self._with_lock_retry(
|
||||
lambda: self._conn.executescript(_SCHEMA))
|
||||
self._with_lock_retry(self._migrate_signal_columns)
|
||||
self._with_lock_retry(self._migrate_outlook_columns)
|
||||
self._with_lock_retry(self._ensure_news_view)
|
||||
|
||||
def _ensure_news_view(self) -> None:
|
||||
"""(Re)create the read-side view atomically. Several processes
|
||||
open the DB at once; DROP+CREATE inside one IMMEDIATE transaction
|
||||
serialises them, and a view another opener created in between is
|
||||
not an error."""
|
||||
try:
|
||||
self._conn.execute("BEGIN IMMEDIATE")
|
||||
self._conn.execute("DROP VIEW IF EXISTS signals_with_news")
|
||||
self._conn.execute(
|
||||
"CREATE VIEW signals_with_news AS"
|
||||
" SELECT s.*,"
|
||||
" COALESCE(s.headlines_json, nh.body) AS headlines_resolved,"
|
||||
" COALESCE(s.events_json, ne.body) AS events_resolved"
|
||||
" FROM signals s"
|
||||
" LEFT JOIN news_snapshots nh ON nh.hash = s.headlines_hash"
|
||||
" LEFT JOIN news_snapshots ne ON ne.hash = s.events_hash")
|
||||
self._conn.commit()
|
||||
except sqlite3.OperationalError as exc:
|
||||
self._conn.rollback()
|
||||
if "already exists" not in str(exc).lower():
|
||||
raise
|
||||
|
||||
@staticmethod
|
||||
def _with_lock_retry(fn, attempts: int = 8, delay: float = 0.05):
|
||||
"""Run ``fn`` again on 'database is locked' (concurrent openers
|
||||
racing the same schema step); other errors propagate at once."""
|
||||
for attempt in range(attempts):
|
||||
try:
|
||||
return fn()
|
||||
except sqlite3.OperationalError as exc:
|
||||
if "locked" not in str(exc).lower() or attempt == attempts - 1:
|
||||
raise
|
||||
time.sleep(delay * (attempt + 1))
|
||||
|
||||
@property
|
||||
def path(self) -> str:
|
||||
@@ -436,6 +635,12 @@ class SignalLogger:
|
||||
self._conn.execute(
|
||||
"CREATE INDEX IF NOT EXISTS idx_signals_advisory_pending"
|
||||
" ON signals (symbol, advisory_outcome_status, id)")
|
||||
self._conn.execute(
|
||||
"CREATE INDEX IF NOT EXISTS idx_signals_setup_key"
|
||||
" ON signals (symbol, setup_key, id)")
|
||||
self._conn.execute(
|
||||
"CREATE INDEX IF NOT EXISTS idx_signals_event"
|
||||
" ON signals (advisory_event_id)")
|
||||
self._conn.commit()
|
||||
|
||||
def _migrate_outlook_columns(self) -> None:
|
||||
@@ -460,20 +665,109 @@ class SignalLogger:
|
||||
source: str = "live",
|
||||
timeframe: str = "",
|
||||
dataframes: Any = None,
|
||||
extra: Optional[Dict[str, Any]] = None,
|
||||
) -> int:
|
||||
"""Insert one scan row; returns the rowid (panel-update key)."""
|
||||
row = _report_row(report, context, source, timeframe, dataframes)
|
||||
"""Insert one scan row; returns the rowid (panel-update key).
|
||||
|
||||
One transaction interns the news blobs, inserts the row and seeds
|
||||
(or dedups) its advisory outcome; a seeding failure is logged and
|
||||
the row still commits. Pending outcomes are then advanced from
|
||||
this scan's candle snapshot."""
|
||||
row = _report_row(report, context, source, timeframe, dataframes,
|
||||
extra=extra)
|
||||
now_iso = row["logged_at"]
|
||||
with self._lock:
|
||||
cur = self._conn.execute(_INSERT, row)
|
||||
self._conn.commit()
|
||||
rowid = int(cur.lastrowid)
|
||||
self._conn.execute("BEGIN IMMEDIATE")
|
||||
try:
|
||||
row["headlines_hash"] = self._intern_news(
|
||||
"headlines", row.pop("headlines_json"), now_iso)
|
||||
row["events_hash"] = self._intern_news(
|
||||
"events", row.pop("events_json"), now_iso)
|
||||
row["headlines_json"] = None
|
||||
row["events_json"] = None
|
||||
cur = self._conn.execute(
|
||||
_INSERT, tuple(row[c] for c in _SIGNAL_INSERT_COLS))
|
||||
rowid = int(cur.lastrowid)
|
||||
try:
|
||||
self._seed_outcome(rowid, row)
|
||||
except Exception:
|
||||
logger.warning("signal_log: outcome seeding failed",
|
||||
exc_info=True)
|
||||
self._conn.commit()
|
||||
except Exception:
|
||||
self._conn.rollback()
|
||||
raise
|
||||
try:
|
||||
self.resolve_pending(str(row[2]), row[25])
|
||||
self.resolve_pending(str(row["symbol"]), row["candles_json"])
|
||||
except Exception:
|
||||
logger.warning("signal_log: advisory outcome resolver failed",
|
||||
exc_info=True)
|
||||
return rowid
|
||||
|
||||
# ── news interning ────────────────────────────────────────────
|
||||
|
||||
def _intern_news(self, kind: str, body: Optional[str],
|
||||
now_iso: str) -> Optional[str]:
|
||||
"""Store one news blob once; returns its content hash."""
|
||||
if not body:
|
||||
return None
|
||||
digest = _blob_hash(body)
|
||||
self._conn.execute(
|
||||
"INSERT OR IGNORE INTO news_snapshots"
|
||||
" (hash, kind, body, first_seen, last_seen)"
|
||||
" VALUES (?, ?, ?, ?, ?)",
|
||||
(digest, kind, body, now_iso, now_iso))
|
||||
self._conn.execute(
|
||||
"UPDATE news_snapshots SET last_seen=? WHERE hash=?",
|
||||
(now_iso, digest))
|
||||
return digest
|
||||
|
||||
def news_for(self, rowid: int) -> Tuple[Optional[str], Optional[str]]:
|
||||
"""(events_json, headlines_json) for a row, legacy or interned."""
|
||||
with self._lock:
|
||||
return read_news(self._conn, rowid)
|
||||
|
||||
# ── advisory-event seeding / dedup ────────────────────────────
|
||||
|
||||
def _seed_outcome(self, rowid: int, row: Dict[str, Any]) -> None:
|
||||
"""Seed a PENDING outcome for a new setup row, or mark it a
|
||||
DUPLICATE of the most recent same-setup row within the gap.
|
||||
Runs inside log()'s transaction."""
|
||||
key = row.get("setup_key")
|
||||
if not key or not row.get("setup_json"):
|
||||
return
|
||||
prior = self._conn.execute(
|
||||
"SELECT id, advisory_event_id, logged_at FROM signals"
|
||||
" WHERE symbol=? AND setup_key=? AND id<?"
|
||||
" ORDER BY id DESC LIMIT 1",
|
||||
(row["symbol"], key, rowid)).fetchone()
|
||||
if prior is not None:
|
||||
try:
|
||||
gap = (datetime.fromisoformat(row["logged_at"])
|
||||
- datetime.fromisoformat(prior[2])).total_seconds()
|
||||
except (TypeError, ValueError):
|
||||
gap = None
|
||||
if gap is not None and 0 <= gap <= OUTCOME_DEDUP_GAP_SECONDS:
|
||||
event_id = int(prior[1] or prior[0])
|
||||
self._conn.execute(
|
||||
"UPDATE signals SET advisory_event_id=?,"
|
||||
" advisory_outcome_status='DUPLICATE',"
|
||||
" advisory_outcome_json=? WHERE id=?",
|
||||
(event_id,
|
||||
_dumps({"model": "advisory-event-dedup-v1",
|
||||
"status": "DUPLICATE", "event_id": event_id}),
|
||||
rowid))
|
||||
return
|
||||
setup = json.loads(row["setup_json"])
|
||||
indicators = (json.loads(row["indicators_json"])
|
||||
if row.get("indicators_json") else None)
|
||||
status, state = initial_outcome(setup, indicators, row["timeframe"])
|
||||
self._conn.execute(
|
||||
"UPDATE signals SET advisory_event_id=?,"
|
||||
" advisory_outcome_status=?, advisory_outcome_json=?"
|
||||
" WHERE id=?",
|
||||
(rowid, status, _dumps(state), rowid))
|
||||
|
||||
def update_panel(
|
||||
self,
|
||||
rowid: int,
|
||||
@@ -486,7 +780,8 @@ class SignalLogger:
|
||||
cache_status: Optional[str] = None,
|
||||
payload_hash: Optional[str] = None,
|
||||
) -> None:
|
||||
"""Fill panel provenance and initialize its advisory outcome."""
|
||||
"""Fill panel provenance; seeds an outcome only for rows that
|
||||
log() did not already seed (rows written by an older process)."""
|
||||
with self._lock:
|
||||
row = self._conn.execute(
|
||||
"SELECT setup_json, indicators_json, timeframe,"
|
||||
@@ -520,7 +815,15 @@ class SignalLogger:
|
||||
def resolve_pending(
|
||||
self, symbol: str, candles_json: Optional[str], limit: int = 100
|
||||
) -> int:
|
||||
"""Advance pending advisory outcomes from a new candle snapshot."""
|
||||
"""Advance pending advisory outcomes from a new candle snapshot.
|
||||
|
||||
The inserting scan's own snapshot is tried first (cheap, covers the
|
||||
steady state). When it does not overlap an outcome's last processed
|
||||
bar — that process missed >14 bars of this timeframe, a weekend
|
||||
straddle, a restart — the resolver rebuilds the bar stream for the
|
||||
timeframe from every stored snapshot after that bar (any surface)
|
||||
and retries, so the outcome is not marked UNRESOLVABLE for a gap
|
||||
the database can fill. Each row commits on its own."""
|
||||
if not candles_json:
|
||||
return 0
|
||||
snapshots = json.loads(candles_json)
|
||||
@@ -528,42 +831,248 @@ class SignalLogger:
|
||||
return 0
|
||||
changed = 0
|
||||
with self._lock:
|
||||
self._conn.execute("BEGIN IMMEDIATE")
|
||||
try:
|
||||
rows = self._conn.execute(
|
||||
"SELECT id, timeframe, advisory_outcome_json FROM signals"
|
||||
" WHERE symbol=? AND advisory_outcome_status='PENDING'"
|
||||
" ORDER BY id",
|
||||
(symbol,),
|
||||
).fetchall()
|
||||
eligible = 0
|
||||
for rowid, timeframe, state_json in rows:
|
||||
bars = timeframe_value(snapshots, timeframe)
|
||||
if not bars or not state_json:
|
||||
rows = self._conn.execute(
|
||||
"SELECT id, timeframe FROM signals"
|
||||
" WHERE symbol=? AND advisory_outcome_status='PENDING'"
|
||||
" ORDER BY id",
|
||||
(symbol,),
|
||||
).fetchall()
|
||||
eligible = 0
|
||||
for rowid, timeframe in rows:
|
||||
bars = timeframe_value(snapshots, timeframe)
|
||||
if not bars:
|
||||
continue
|
||||
if eligible >= max(1, int(limit)):
|
||||
break
|
||||
eligible += 1
|
||||
# One IMMEDIATE transaction per row: the state is re-read
|
||||
# under the write lock, so the bot and the dashboard (each
|
||||
# advancing from its own snapshot) serialise instead of
|
||||
# overwriting each other's progress.
|
||||
self._conn.execute("BEGIN IMMEDIATE")
|
||||
try:
|
||||
hit = self._conn.execute(
|
||||
"SELECT advisory_outcome_json FROM signals"
|
||||
" WHERE id=? AND advisory_outcome_status='PENDING'",
|
||||
(rowid,)).fetchone()
|
||||
if hit is None or not hit[0]:
|
||||
self._conn.rollback()
|
||||
continue
|
||||
if eligible >= max(1, int(limit)):
|
||||
break
|
||||
eligible += 1
|
||||
state = json.loads(state_json)
|
||||
before = _dumps(state)
|
||||
state = advance_outcome(state, bars)
|
||||
after = _dumps(state)
|
||||
before = _dumps(json.loads(hit[0]))
|
||||
advanced = advance_outcome(json.loads(before), bars)
|
||||
if (advanced.get("status") == "UNRESOLVABLE"
|
||||
and "continuity" in str(advanced.get("reason", ""))):
|
||||
union = self._stream_since(
|
||||
symbol, timeframe,
|
||||
json.loads(before)["progress"]["last_processed_bar"])
|
||||
if union:
|
||||
advanced = advance_outcome(json.loads(before), union)
|
||||
after = _dumps(advanced)
|
||||
if after == before:
|
||||
self._conn.rollback()
|
||||
continue
|
||||
status = state.get("status", "UNRESOLVABLE")
|
||||
status = advanced.get("status", "UNRESOLVABLE")
|
||||
self._conn.execute(
|
||||
"UPDATE signals SET advisory_outcome_status=?,"
|
||||
" advisory_r_multiple=?, advisory_outcome_json=?"
|
||||
" WHERE id=? AND advisory_outcome_status='PENDING'",
|
||||
(status, canonical_r(state), after, rowid),
|
||||
" WHERE id=?",
|
||||
(status, canonical_r(advanced), after, rowid),
|
||||
)
|
||||
self._conn.commit()
|
||||
changed += 1
|
||||
self._conn.commit()
|
||||
except Exception:
|
||||
self._conn.rollback()
|
||||
raise
|
||||
except Exception:
|
||||
self._conn.rollback()
|
||||
raise
|
||||
return changed
|
||||
|
||||
def _stream_since(self, symbol: str, timeframe: str,
|
||||
last_bar: str, max_rows: int = 4000) -> List[Dict[str, Any]]:
|
||||
"""Union of every stored closed bar of ``timeframe`` from
|
||||
``last_bar`` onward, across all rows/surfaces (oldest first).
|
||||
Only the requested timeframe's array is pulled out of each
|
||||
candles_json (json_extract), so the cost is bounded by the number
|
||||
of rows logged since that bar, not by blob size."""
|
||||
tf = str(getattr(timeframe, "value", timeframe))
|
||||
try:
|
||||
since = datetime.fromisoformat(
|
||||
str(last_bar).replace(" ", "T")).isoformat()
|
||||
except (TypeError, ValueError):
|
||||
return []
|
||||
bars: Dict[str, Dict[str, Any]] = {}
|
||||
for key in _tf_storage_keys(tf):
|
||||
rows = self._conn.execute(
|
||||
"SELECT json_extract(candles_json, ?) FROM signals"
|
||||
" WHERE symbol=? AND candles_json IS NOT NULL"
|
||||
" AND logged_at >= ? ORDER BY id LIMIT ?",
|
||||
(f'$."{key}"', symbol, since, int(max_rows))).fetchall()
|
||||
for (blob,) in rows:
|
||||
if not blob:
|
||||
continue
|
||||
try:
|
||||
for bar in json.loads(blob):
|
||||
stamp = bar.get("timestamp")
|
||||
if stamp and stamp >= since:
|
||||
bars.setdefault(str(stamp), bar)
|
||||
except (TypeError, ValueError, AttributeError):
|
||||
continue
|
||||
return [bars[k] for k in sorted(bars)]
|
||||
|
||||
def advance_with_bars(self, rowid: int, bars: Iterable[Dict[str, Any]],
|
||||
evidence: str = "external_bars") -> str:
|
||||
"""Advance ONE pending outcome from caller-supplied bars (backfill
|
||||
from provider history). Returns the resulting status."""
|
||||
with self._lock:
|
||||
row = self._conn.execute(
|
||||
"SELECT advisory_outcome_status, advisory_outcome_json"
|
||||
" FROM signals WHERE id=?", (rowid,)).fetchone()
|
||||
if row is None or row[0] != "PENDING" or not row[1]:
|
||||
return str(row[0]) if row else "MISSING"
|
||||
state = advance_outcome(json.loads(row[1]), list(bars))
|
||||
state["evidence"] = evidence
|
||||
status = state.get("status", "UNRESOLVABLE")
|
||||
self._conn.execute(
|
||||
"UPDATE signals SET advisory_outcome_status=?,"
|
||||
" advisory_r_multiple=?, advisory_outcome_json=?"
|
||||
" WHERE id=? AND advisory_outcome_status='PENDING'",
|
||||
(status, canonical_r(state), _dumps(state), rowid))
|
||||
self._conn.commit()
|
||||
return status
|
||||
|
||||
def backfill_seeds(self, limit: Optional[int] = None) -> Dict[str, int]:
|
||||
"""Assign setup_key / advisory_event_id to legacy setup rows and
|
||||
seed the canonical ones that carry no outcome yet. Existing
|
||||
outcomes (panel-seeded rows) are never overwritten. Safe to run
|
||||
beside live writers (short per-row transactions)."""
|
||||
stats = {"keyed": 0, "duplicates": 0, "seeded": 0}
|
||||
with self._lock:
|
||||
rows = self._conn.execute(
|
||||
"SELECT id, symbol, timeframe, logged_at, setup_json,"
|
||||
" indicators_json, advisory_outcome_status, advisory_event_id"
|
||||
" FROM signals WHERE setup_json IS NOT NULL"
|
||||
" AND setup_json != 'null' ORDER BY id"
|
||||
+ (f" LIMIT {int(limit)}" if limit else "")).fetchall()
|
||||
last_by_key: Dict[Tuple[str, str], Tuple[int, int, datetime]] = {}
|
||||
for (rowid, symbol, tf, logged_at, setup_json, ind_json,
|
||||
status, event_id) in rows:
|
||||
setup = json.loads(setup_json)
|
||||
key = setup_key(setup, tf)
|
||||
if not key:
|
||||
continue
|
||||
try:
|
||||
ts = datetime.fromisoformat(logged_at)
|
||||
except (TypeError, ValueError):
|
||||
continue
|
||||
prior = last_by_key.get((symbol, key))
|
||||
if (prior is not None
|
||||
and 0 <= (ts - prior[2]).total_seconds()
|
||||
<= OUTCOME_DEDUP_GAP_SECONDS):
|
||||
# Same setup re-logged inside the gap: a duplicate of
|
||||
# the canonical event regardless of its legacy status
|
||||
# (a panel-seeded RESOLVED re-log measured the very
|
||||
# same trade — keeping it would double count R).
|
||||
canonical = prior[1]
|
||||
if status != "DUPLICATE" or event_id != canonical:
|
||||
self._conn.execute(
|
||||
"UPDATE signals SET setup_key=?,"
|
||||
" advisory_event_id=?,"
|
||||
" advisory_outcome_status='DUPLICATE',"
|
||||
" advisory_r_multiple=NULL,"
|
||||
" advisory_outcome_json=? WHERE id=?",
|
||||
(key, canonical,
|
||||
_dumps({"model": "advisory-event-dedup-v1",
|
||||
"status": "DUPLICATE",
|
||||
"event_id": canonical,
|
||||
"superseded_status": status}), rowid))
|
||||
stats["duplicates"] += 1
|
||||
last_by_key[(symbol, key)] = (rowid, canonical, ts)
|
||||
continue
|
||||
canonical = int(event_id or rowid)
|
||||
if status is None:
|
||||
indicators = json.loads(ind_json) if ind_json else None
|
||||
new_status, state = initial_outcome(setup, indicators, tf)
|
||||
self._conn.execute(
|
||||
"UPDATE signals SET setup_key=?, advisory_event_id=?,"
|
||||
" advisory_outcome_status=?, advisory_outcome_json=?"
|
||||
" WHERE id=?",
|
||||
(key, canonical, new_status, _dumps(state), rowid))
|
||||
stats["seeded"] += 1
|
||||
else:
|
||||
self._conn.execute(
|
||||
"UPDATE signals SET setup_key=?, advisory_event_id=?"
|
||||
" WHERE id=?", (key, canonical, rowid))
|
||||
stats["keyed"] += 1
|
||||
last_by_key[(symbol, key)] = (rowid, canonical, ts)
|
||||
self._conn.commit()
|
||||
return stats
|
||||
|
||||
def resolve_from_store(self, symbol: str, limit: int = 500) -> int:
|
||||
"""Advance every PENDING outcome from the stored bar streams alone
|
||||
(no provider). Used by the backfill script after seeding."""
|
||||
changed = 0
|
||||
with self._lock:
|
||||
rows = self._conn.execute(
|
||||
"SELECT id, timeframe, advisory_outcome_json FROM signals"
|
||||
" WHERE symbol=? AND advisory_outcome_status='PENDING'"
|
||||
" ORDER BY id LIMIT ?", (symbol, int(limit))).fetchall()
|
||||
for rowid, timeframe, state_json in rows:
|
||||
if not state_json:
|
||||
continue
|
||||
state = json.loads(state_json)
|
||||
union = self._stream_since(
|
||||
symbol, timeframe, state["progress"]["last_processed_bar"])
|
||||
if not union:
|
||||
continue
|
||||
before = _dumps(state)
|
||||
advanced = advance_outcome(json.loads(before), union)
|
||||
after = _dumps(advanced)
|
||||
if after == before:
|
||||
continue
|
||||
self._conn.execute(
|
||||
"UPDATE signals SET advisory_outcome_status=?,"
|
||||
" advisory_r_multiple=?, advisory_outcome_json=?"
|
||||
" WHERE id=? AND advisory_outcome_status='PENDING'",
|
||||
(advanced.get("status", "UNRESOLVABLE"),
|
||||
canonical_r(advanced), after, rowid))
|
||||
self._conn.commit()
|
||||
changed += 1
|
||||
return changed
|
||||
|
||||
def reseed_unresolvable(self, max_reseeds: int = 1) -> List[int]:
|
||||
"""Bounded retry for continuity-broken outcomes: re-freeze the
|
||||
seed from the row's own setup/indicators (at most ``max_reseeds``
|
||||
times per row) so the next resolver pass can walk the now-filled
|
||||
stream. Invalid-seed rows stay terminal."""
|
||||
reseeded: List[int] = []
|
||||
with self._lock:
|
||||
rows = self._conn.execute(
|
||||
"SELECT id, timeframe, setup_json, indicators_json,"
|
||||
" advisory_outcome_json FROM signals"
|
||||
" WHERE advisory_outcome_status='UNRESOLVABLE'"
|
||||
" AND advisory_event_id = id").fetchall()
|
||||
for rowid, tf, setup_json, ind_json, state_json in rows:
|
||||
old = json.loads(state_json) if state_json else {}
|
||||
if "continuity" not in str(old.get("reason", "")) and \
|
||||
"gap" not in str(old.get("reason", "")):
|
||||
continue
|
||||
count = int((old.get("reseed") or {}).get("count", 0))
|
||||
if count >= max_reseeds or not setup_json:
|
||||
continue
|
||||
indicators = json.loads(ind_json) if ind_json else None
|
||||
status, state = initial_outcome(
|
||||
json.loads(setup_json), indicators, tf)
|
||||
if status != "PENDING":
|
||||
continue
|
||||
state["reseed"] = {
|
||||
"count": count + 1,
|
||||
"at": datetime.now(timezone.utc).isoformat()}
|
||||
self._conn.execute(
|
||||
"UPDATE signals SET advisory_outcome_status=?,"
|
||||
" advisory_r_multiple=NULL, advisory_outcome_json=?"
|
||||
" WHERE id=?", (status, _dumps(state), rowid))
|
||||
reseeded.append(int(rowid))
|
||||
self._conn.commit()
|
||||
return reseeded
|
||||
|
||||
def log_outlook(self, outlook: Any, surface: str,
|
||||
payload: Any = None) -> int:
|
||||
"""Insert one advisory market outlook; returns its rowid."""
|
||||
@@ -596,6 +1105,37 @@ class SignalLogger:
|
||||
self._conn.close()
|
||||
|
||||
|
||||
def _tf_storage_keys(tf: str) -> List[str]:
|
||||
"""Keys under which a timeframe's bars may be stored in candles_json
|
||||
(enum value plus the D1 alias used by older rows)."""
|
||||
keys = [tf]
|
||||
if tf in ("1d", "1day"):
|
||||
keys = ["1d", "1day"]
|
||||
return keys
|
||||
|
||||
|
||||
def read_news(con: sqlite3.Connection, rowid: int
|
||||
) -> Tuple[Optional[str], Optional[str]]:
|
||||
"""(events_json, headlines_json) for ``rowid`` on ANY connection —
|
||||
resolves interned snapshots and falls back to the legacy inline
|
||||
columns. Read-only."""
|
||||
row = con.execute(
|
||||
"SELECT events_json, headlines_json, events_hash, headlines_hash"
|
||||
" FROM signals WHERE id=?", (rowid,)).fetchone()
|
||||
if row is None:
|
||||
return None, None
|
||||
events, headlines, ev_hash, hl_hash = row
|
||||
if events is None and ev_hash:
|
||||
hit = con.execute("SELECT body FROM news_snapshots WHERE hash=?",
|
||||
(ev_hash,)).fetchone()
|
||||
events = hit[0] if hit else None
|
||||
if headlines is None and hl_hash:
|
||||
hit = con.execute("SELECT body FROM news_snapshots WHERE hash=?",
|
||||
(hl_hash,)).fetchone()
|
||||
headlines = hit[0] if hit else None
|
||||
return events, headlines
|
||||
|
||||
|
||||
# ── Module-level entry point (config-driven, fail-safe) ───────────────
|
||||
|
||||
_loggers: Dict[str, SignalLogger] = {}
|
||||
@@ -608,14 +1148,20 @@ def log_signal(
|
||||
source: str = "live",
|
||||
timeframe: str = "",
|
||||
dataframes: Any = None,
|
||||
extra: Optional[Dict[str, Any]] = None,
|
||||
) -> None:
|
||||
"""Persist one scan if the signal_log config block enables it.
|
||||
Stashes the rowid on the report (``_signal_log_rowid``) so a post-hoc
|
||||
panel attach can update this same row. NEVER raises — a logging
|
||||
failure must not break a live scan."""
|
||||
failure must not break a live scan. ``extra`` = producer metadata
|
||||
(``scan_kind``, ``config_fingerprint``); blocked scans are skipped
|
||||
when ``signal_log.log_blocked_scans`` is false."""
|
||||
try:
|
||||
if not app_config.signal_log_enabled():
|
||||
return
|
||||
if ((extra or {}).get("scan_kind") == "blocked"
|
||||
and not app_config.signal_log_blocked_scans()):
|
||||
return
|
||||
path = app_config.signal_log_path()
|
||||
with _loggers_lock:
|
||||
lg = _loggers.get(path)
|
||||
@@ -623,7 +1169,8 @@ def log_signal(
|
||||
lg = SignalLogger(path)
|
||||
_loggers[path] = lg
|
||||
rowid = lg.log(report, context=context, source=source,
|
||||
timeframe=timeframe, dataframes=dataframes)
|
||||
timeframe=timeframe, dataframes=dataframes,
|
||||
extra=extra)
|
||||
try:
|
||||
report._signal_log_rowid = rowid
|
||||
except Exception:
|
||||
|
||||
+58
-5
@@ -549,9 +549,50 @@ async def run_signal(
|
||||
confirmation_tfs: Optional[List[Timeframe]] = None,
|
||||
) -> SignalReport:
|
||||
loop = asyncio.get_running_loop()
|
||||
return await loop.run_in_executor(
|
||||
report = await loop.run_in_executor(
|
||||
_executor, _run_signal_sync, symbol, timeframe, confirmation_tfs
|
||||
)
|
||||
_remember_scan(symbol, timeframe, report)
|
||||
return report
|
||||
|
||||
|
||||
# Most recent report per (symbol, timeframe) keyed by the closed-bar epoch
|
||||
# it was computed for. Closed-bar inputs are identical within a bar, so a
|
||||
# rescan inside the same bar (the WS level-breach handler fired up to
|
||||
# ~5 min apart all day, logging 3.5 M15 rows per candle — 2026-09 audit)
|
||||
# can reuse it instead of re-fetching, re-scoring and re-logging.
|
||||
_last_scan: Dict[str, tuple] = {}
|
||||
_TF_SECONDS_CACHE = {
|
||||
Timeframe.M1: 60, Timeframe.M5: 300, Timeframe.M15: 900,
|
||||
Timeframe.M30: 1800, Timeframe.H1: 3600, Timeframe.H4: 14400,
|
||||
Timeframe.D1: 86400,
|
||||
}
|
||||
|
||||
|
||||
def _bar_epoch(tf: Timeframe, now: Optional[float] = None) -> int:
|
||||
"""Epoch of the most recent candle boundary for ``tf`` (UTC-anchored,
|
||||
the same grid _seconds_until_boundary sleeps to)."""
|
||||
secs = _TF_SECONDS_CACHE.get(tf, 900)
|
||||
t = int(time.time() if now is None else now)
|
||||
return t - t % secs
|
||||
|
||||
|
||||
def _remember_scan(symbol: str, tf: Timeframe, report: SignalReport) -> None:
|
||||
_last_scan[f"{symbol}_{tf.value}"] = (_bar_epoch(tf), report)
|
||||
|
||||
|
||||
async def _cached_or_scan(
|
||||
symbol: str,
|
||||
timeframe: Timeframe,
|
||||
confirmation_tfs: Optional[List[Timeframe]] = None,
|
||||
) -> SignalReport:
|
||||
"""run_signal, unless this bar already has a report for the key."""
|
||||
cached = _last_scan.get(f"{symbol}_{timeframe.value}")
|
||||
if cached is not None and cached[0] == _bar_epoch(timeframe):
|
||||
logger.info(
|
||||
f"{symbol} {timeframe.value}: reusing this bar's scan — no rescan")
|
||||
return cached[1]
|
||||
return await run_signal(symbol, timeframe, confirmation_tfs)
|
||||
|
||||
|
||||
# ── Alert gating (quality threshold + dedup) ─────────────────────────
|
||||
@@ -572,17 +613,22 @@ def _passes_quality(quality: str) -> bool:
|
||||
return _QUALITY_RANK.get(quality, 0) >= _QUALITY_RANK.get(_alert_min_quality, 1)
|
||||
|
||||
|
||||
_last_alert_quality: Dict[str, str] = {} # key → grade of last ALERTED signal
|
||||
|
||||
|
||||
def _alert_fingerprint(report: SignalReport) -> tuple:
|
||||
"""Alert identity for dedup: signal, quality, setup direction.
|
||||
"""Alert identity for dedup: signal and setup direction.
|
||||
|
||||
Entry movement is checked separately in _should_alert against the
|
||||
last ALERTED entry/risk (a stable reference). Deriving a price bucket
|
||||
from the CURRENT SL distance here made the fingerprint churn on
|
||||
normal ATR/anchor drift — near-duplicate alerts every candle."""
|
||||
normal ATR/anchor drift — near-duplicate alerts every candle. The
|
||||
grade is deliberately NOT part of the identity (2026-09 audit: an
|
||||
A↔B flicker on an unchanged idea re-sent the full alert ~4× in 45
|
||||
min); a grade UPGRADE re-alerts via _should_alert instead."""
|
||||
setup = report.trade_setup
|
||||
return (
|
||||
report.final_signal.name,
|
||||
report.signal_quality,
|
||||
setup.direction if setup else None,
|
||||
)
|
||||
|
||||
@@ -602,6 +648,12 @@ def _should_alert(key: str, report: SignalReport) -> bool:
|
||||
return True
|
||||
if time.time() - _last_alert_ts.get(key, 0) >= _ALERT_REARM_SECONDS:
|
||||
return True
|
||||
# Same idea at a HIGHER grade than last alerted → worth re-sending;
|
||||
# a downgrade or flicker back is not.
|
||||
last_q = _last_alert_quality.get(key)
|
||||
if last_q is not None and (_QUALITY_RANK.get(report.signal_quality, 0)
|
||||
> _QUALITY_RANK.get(last_q, 0)):
|
||||
return True
|
||||
setup = report.trade_setup
|
||||
last = _last_alert_setup.get(key)
|
||||
if setup is not None and last is not None:
|
||||
@@ -614,6 +666,7 @@ def _should_alert(key: str, report: SignalReport) -> bool:
|
||||
def _commit_alert(key: str, report: SignalReport) -> None:
|
||||
_last_alert_fp[key] = _alert_fingerprint(report)
|
||||
_last_alert_ts[key] = time.time()
|
||||
_last_alert_quality[key] = report.signal_quality
|
||||
setup = report.trade_setup
|
||||
if setup is not None:
|
||||
_last_alert_setup[key] = (
|
||||
@@ -871,7 +924,7 @@ async def _handle_breach(bot, pair_symbol: str, price: float, level_name: str) -
|
||||
logger.info(f"Level breach: {display} crossed {label} at {price} — rescanning M15")
|
||||
|
||||
try:
|
||||
report = await run_signal(pair_symbol, Timeframe.M15)
|
||||
report = await _cached_or_scan(pair_symbol, Timeframe.M15)
|
||||
_update_level_cache([report])
|
||||
except Exception:
|
||||
logger.exception(f"Breach scan failed for {display}")
|
||||
|
||||
+918
-2
File diff suppressed because it is too large
Load Diff
Reference in New Issue
Block a user