""" signal_log.py — SQLite signal-history logger ============================================ Persists every live scan's SignalReport (signal, grade, price at scan time, news at scan time) to a SQLite database so the full decision history can be queried later. Hooked ONLY into SignalAggregator.analyze_pair (the live entry point); the backtester calls analyze_from_dataframes directly and therefore never writes here (live == backtest parity — CLAUDE.md rule 4). Panel fields arrive POST-HOC: the row is inserted during analyze_pair (before consumers attach the advisory panel), so log_signal stashes the rowid on the report and agent_panel.attach_panel calls log_panel_update() to fill the verdict, takes, creation/cache provenance, payload hash, and advisory-outcome seed on that same row. Later closed-candle snapshots advance the hypothetical next-open/trailing outcome. A report without a rowid (backtest replay, logging disabled) is a silent no-op. Deployment settings come from the config.json "signal_log" block via app_config (env vars SIGNAL_LOG_ENABLED / SIGNAL_LOG_PATH win): "signal_log": {"enabled": true, "path": "logs/signals.db"} Fail-safe contract: logging must NEVER break a scan. Every public entry point catches all exceptions and downgrades them to a warning log line. Report/context objects are duck-typed (no imports from signal_aggregator or news_engine) so this module stays at the bottom of the import graph. 2026-09 observability additions (signal-log audit): * Every row with a trade setup is an advisory-outcome candidate, not only panel rows. Re-logs of the same setup (same ``setup_key`` — tf, direction, entry, stop, strategy — within OUTCOME_DEDUP_GAP_SECONDS, from ANY surface) are marked ``DUPLICATE`` and point at the canonical row through ``advisory_event_id``; only the canonical row is seeded and advanced, so summing ``advisory_r_multiple`` over ``advisory_outcome_status='RESOLVED'`` counts each trade once. * Pending outcomes advance from the inserting scan's own snapshot first; when that snapshot no longer overlaps the outcome's last processed bar (downtime, weekend straddle) the resolver rebuilds the bar stream from every stored snapshot of that timeframe, so a >14-bar gap in one process's scans is not a terminal ``UNRESOLVABLE``. * News blobs are content-addressed: ``headlines_json``/``events_json`` are interned once into ``news_snapshots`` and rows carry ``headlines_hash`` / ``events_hash`` (the legacy columns stay NULL on new rows; the ``signals_with_news`` view and ``read_news`` resolve both eras). * Post-gate truth columns: ``gate_reason`` (the warning that suppressed the setup), ``actionable`` (grade >= C AND setup present — what an alert surface would act on; the ``action`` text is written BEFORE the gate chain and can disagree), ``cleaned_warnings_json`` (warnings a shipped setup superseded, removed from ``warnings_json``: the vetoes the XAU_CP acceptance stripped and, since 2026-09-08, a core generator's stale spread-gate veto when a later generator produced the setup — so non-empty does NOT mean "XAU_CP row"), ``bar_range_median`` (the value historically stored as ``spread`` — it is the median High-Low of the primary frame, NOT a broker spread), ``scan_kind`` (``scan``; ``blocked`` for weekend/day early returns, which carry no news or candles; ``failed`` when the primary frame could not be fetched — price 0, no signal — readers skip it like ``blocked``) and ``config_fingerprint`` (hash of the gate config the producing process ran with — surfaces config drift between processes). 2026-09 weight-research additions: * Row columns ``bar_ts`` (the primary bar LABEL this scan closed on, stored verbatim so it joins to indicators_json), ``engine_fingerprint`` (hash of the SCORING config — indicator params, thresholds, regime weight multipliers, timeframes), ``code_version`` (short git SHA), ``penalty_tags_json`` / ``penalty_total`` / ``momentum_fading`` / ``exhaustion_count`` (the decay/exhaustion record), and the post-hoc ``watch_level`` / ``watch_kinds`` / ``watch_line`` written by ``update_watch`` from violent_watch.attach_watch. * Every per-indicator entry in ``indicators_json`` also carries ``eff_weight`` (base weight x regime multiplier — the weight the composite actually used) and ``wscore`` (its contribution). Both are null for indicators the composite skipped. Divergence and candlestick entries exist only when they fire: absence is "no vote", not zero. * Table ``scan_outcomes`` — model-free forward returns per scan (1/4/24 h, $ and ATR-normalized, plus MFE/MAE, a 3xATR burst flag and a completeness flag), labelled from the stored candle streams by ``label_scan_outcomes`` (called opportunistically at the end of log(), time-boxed; batch driver: scripts/label_scan_outcomes.py). Readers must dedup by the FIRST row per (timeframe, bar_ts) — several surfaces log the same bar. 2026-09-30 daily-trend veto: * Row column ``daily_trend`` (+1 / 0 / -1 from closed D1 bars: computed by the core where the veto is on, stamped by the live wrapper for display where it is off; NULL on blocked / failed scans and when the daily frame could not be read) and the gate prefix ``DAILY TREND FILTER`` in ``gate_reason``. * Row column ``alerted_at`` (UTC time of the Telegram alert for this row; ``mark_alerted``) and the read-only helper ``open_alerted_setups`` — alerted plans whose advisory outcome is still PENDING — behind the "already open" line on the alert (exposure.py). Since 2026-10-01 the stamp is written just BEFORE the send and cleared again (``unmark_alerted``) when no chat received the alert, so two alerts sent at the same candle close count each other; ``last_alerted_bars`` reads the newest alerted bar per timeframe so the bot's entry cooldown survives a restart. """ import hashlib import json import logging import math import os import sqlite3 import subprocess import threading import time from datetime import datetime, timedelta, timezone from typing import Any, Dict, Iterable, List, Optional, Tuple import app_config from advisory_outcome import ( MAX_HOLD_BARS, advance_outcome, canonical_r, initial_outcome, normalize_timeframe, resolve_locked_floor, timeframe_value, ) # Private on purpose — the scan-outcome labeller must apply EXACTLY the # advisory model's bar validation and scheduled-closure rule (NY-anchored # weekend + the daily 17:00 NY maintenance hour), never a second copy. from advisory_outcome import _TF_SECONDS, _has_continuity, _valid_bars logger = logging.getLogger(__name__) _SCHEMA = """ CREATE TABLE IF NOT EXISTS signals ( id INTEGER PRIMARY KEY AUTOINCREMENT, logged_at TEXT NOT NULL, source TEXT NOT NULL, symbol TEXT NOT NULL, timeframe TEXT NOT NULL, signal TEXT NOT NULL, score REAL, confidence REAL, quality TEXT, action TEXT, price REAL, spread REAL, market_regime TEXT, volatility_regime TEXT, mtf_json TEXT, setup_json TEXT, reasons_json TEXT, warnings_json TEXT, event_risk TEXT, event_description TEXT, news_sentiment REAL, news_description TEXT, news_summary TEXT, events_json TEXT, headlines_json TEXT, indicators_json TEXT, candles_json TEXT, panel_verdict TEXT, panel_confidence REAL, panel_rationale TEXT, panel_takes_json TEXT, panel_created_at TEXT, panel_cache_status TEXT, panel_payload_hash TEXT, advisory_outcome_status TEXT, advisory_r_multiple REAL, advisory_outcome_json TEXT, panel_prompt_version TEXT, panel_probability REAL, panel_probs_json TEXT, panel_payloads_json TEXT, panel_context_json TEXT ); CREATE INDEX IF NOT EXISTS idx_signals_sym_tf_ts ON signals (symbol, timeframe, logged_at); CREATE TABLE IF NOT EXISTS news_snapshots ( hash TEXT PRIMARY KEY, kind TEXT NOT NULL, body TEXT NOT NULL, first_seen TEXT NOT NULL, last_seen TEXT ); CREATE TABLE IF NOT EXISTS scan_outcomes ( signal_id INTEGER PRIMARY KEY REFERENCES signals(id), timeframe TEXT, bar_ts TEXT, atr REAL, close0 REAL, fwd_1h REAL, fwd_4h REAL, fwd_24h REAL, fwd_1h_atr REAL, fwd_4h_atr REAL, fwd_24h_atr REAL, mfe_24h_atr REAL, mae_24h_atr REAL, burst_1h INTEGER, bars_used INTEGER, complete INTEGER, labeled_at TEXT ); CREATE INDEX IF NOT EXISTS idx_scan_outcomes_tf_bar ON scan_outcomes (timeframe, bar_ts); """ # Additive migration for columns introduced after the original schema. # Idempotent — skips columns already present. _ADDITIVE_SIGNAL_COLS = ( ("panel_verdict", "TEXT"), ("panel_confidence", "REAL"), ("panel_rationale", "TEXT"), ("panel_takes_json", "TEXT"), ("indicators_json", "TEXT"), ("candles_json", "TEXT"), ("panel_created_at", "TEXT"), ("panel_cache_status", "TEXT"), ("panel_payload_hash", "TEXT"), ("advisory_outcome_status", "TEXT"), ("advisory_r_multiple", "REAL"), ("advisory_outcome_json", "TEXT"), # 2026-09 observability columns (see module docstring). ("setup_key", "TEXT"), ("advisory_event_id", "INTEGER"), ("headlines_hash", "TEXT"), ("events_hash", "TEXT"), ("gate_reason", "TEXT"), ("actionable", "INTEGER"), ("cleaned_warnings_json", "TEXT"), ("bar_range_median", "REAL"), ("scan_kind", "TEXT"), ("config_fingerprint", "TEXT"), ("panel_verdict_source", "TEXT"), # 2026-09 weight-research columns (see module docstring). ("bar_ts", "TEXT"), ("engine_fingerprint", "TEXT"), ("code_version", "TEXT"), ("penalty_tags_json", "TEXT"), ("penalty_total", "REAL"), ("momentum_fading", "INTEGER"), ("exhaustion_count", "INTEGER"), # Violent-move watch, written post-hoc by update_watch. ("watch_level", "TEXT"), ("watch_kinds", "TEXT"), ("watch_line", "TEXT"), # 2026-09-17 agent-panel v2 (see agent_panel.py): prompt era, the mean # persona probability, per-persona probabilities + reasons, the exact # per-persona payloads and the shared context — written post-hoc by # update_panel so scripts/panel_replay.py can re-score any prompt # version on the same evidence. ("panel_prompt_version", "TEXT"), ("panel_probability", "REAL"), ("panel_probs_json", "TEXT"), ("panel_payloads_json", "TEXT"), ("panel_context_json", "TEXT"), # 2026-09-30 daily-trend veto (see module docstring). ("daily_trend", "INTEGER"), # 2026-09-30: UTC time of the Telegram alert for THIS row (mark_alerted: # written by the bot just before the send, cleared by unmark_alerted # when no chat received it). NULL on every row that was scanned but not # alerted — the "which plans did the reader actually receive" key the # open-exposure line reads (open_alerted_setups). ("alerted_at", "TEXT"), ) # Read-side compatibility view `signals_with_news`: legacy rows keep # their inline news blobs, new rows resolve them through news_snapshots. # Recreated on every logger init (SignalLogger._ensure_news_view) so it # always exposes the current column set. # Same-setup re-logs (cross-surface double logs, persisting-signal # re-fires) within this gap are ONE advisory event. Mirrors # analyze_signal_log's 60-minute clustering. OUTCOME_DEDUP_GAP_SECONDS = 3600 # Warning prefixes that mean "a gate removed the setup", checked in gate # order (live wrapper first, then the post-grade chain, then the setup # generators). Kept here — signal_log must not import signal_aggregator. _GATE_PREFIXES = ( "WEEKEND FILTER", "DAY FILTER", "SESSION FILTER", "Primary timeframe stale", "Primary timeframe data unavailable", "DATA UNAVAILABLE:", # The daily-trend veto removes a setup that every earlier gate had let # through, so on such a row it IS the reason: listed ahead of the chain # because the vetoed setup's stale warnings (a cleaned SPREAD GATE, the # chart-pattern path's cleaned vetoes) are still in the pool. "DAILY TREND FILTER", "REGIME FILTER", "EXHAUSTION BLOCK", "EXHAUSTED TREND FILTER", "SELL FILTER", "MTF OPPOSITION", "BIG CANDLE FILTER", "M5 FILTER", "LOW ATR FILTER", "SPREAD GATE", ) _ACTIONABLE_GRADES = ("A+", "A", "B", "C") # ── Scan outcomes (forward returns for indicator-weight research) ───── # Horizons are CALENDAR hours from the signal bar's close; a bar label is # its OPEN time, so the bar closing at label+interval is the one counted. SCAN_OUTCOME_HORIZONS_H = (1, 4, 24) SCAN_OUTCOME_WINDOW_H = 24 # A 24 h window can only be judged once it has elapsed; the extra hour # leaves room for the last bar of the window to be stored. SCAN_LABEL_MIN_AGE_H = 25 # An incomplete row older than this is FINAL: its window had a real hole # (a feed outage), which no later bar can fill. It is neither re-examined # nor allowed to anchor the shared bar stream for the rest of the batch. SCAN_LABEL_MAX_AGE_H = 72 # Missing bars tolerated inside a window ON TOP of the scheduled closures # advisory_outcome._has_continuity already allows. SCAN_LABEL_MAX_GAP_BARS = 3 # "Burst" bar: range >= this multiple of the scan's ATR (violent_watch). SCAN_BURST_ATR_MULTIPLE = 3.0 # Soft wall-clock cap for ONE opportunistic (in-scan) labelling pass. SCAN_LABEL_TIME_BUDGET = 2.0 # Minimum spacing between opportunistic passes in ONE process. Rows that # can never complete (a real hole in the stored stream) are re-examined # on every pass, and that read costs ~0.5 s on the live database — too # much to pay on the alert path for every logged scan, and pointless more # often than a candle close. Explicit calls (batch script, tests) ignore it. SCAN_LABEL_MIN_INTERVAL = 60.0 _SIGNAL_INSERT_COLS = ( "logged_at", "source", "symbol", "timeframe", "signal", "score", "confidence", "quality", "action", "price", "spread", "market_regime", "volatility_regime", "mtf_json", "setup_json", "reasons_json", "warnings_json", "event_risk", "event_description", "news_sentiment", "news_description", "news_summary", "events_json", "headlines_json", "indicators_json", "candles_json", "panel_verdict", "panel_confidence", "panel_rationale", "panel_takes_json", "setup_key", "headlines_hash", "events_hash", "gate_reason", "actionable", "cleaned_warnings_json", "bar_range_median", "scan_kind", "config_fingerprint", "bar_ts", "engine_fingerprint", "code_version", "penalty_tags_json", "penalty_total", "momentum_fading", "exhaustion_count", "daily_trend", ) _INSERT = ( "INSERT INTO signals (" + ", ".join(_SIGNAL_INSERT_COLS) + ") VALUES (" + ", ".join("?" for _ in _SIGNAL_INSERT_COLS) + ")" ) def _finite(x: Any) -> Optional[float]: """float(x) or None for non-numeric / non-finite values (NaN/inf — SQLite has no NaN and strict JSON rejects it).""" try: f = float(x) except (TypeError, ValueError): return None return f if math.isfinite(f) else None def _daily_trend_value(x: Any) -> Optional[int]: """The report's daily trend as -1 / 0 / +1, or None when it carries none.""" f = _finite(x) return None if f is None else int(f) def _iso(ts: Any) -> Optional[str]: if ts is None: return None try: if ts.tzinfo is None: ts = ts.replace(tzinfo=timezone.utc) return ts.astimezone(timezone.utc).isoformat() except (AttributeError, ValueError): return str(ts) def _dumps(obj: Any) -> str: return json.dumps(obj, default=str, allow_nan=False) def _blob_hash(text: str) -> str: return hashlib.sha256(text.encode("utf-8")).hexdigest() _code_version_cache: Optional[str] = None def _code_version() -> str: """Short git SHA of the tree this process runs from (computed once). Record-keeping only — "" whenever git is unavailable, slow (2 s cap) or the tree is not a checkout. There is no VERSION/__version__ convention in this repo, so the commit is the version.""" global _code_version_cache if _code_version_cache is None: _code_version_cache = "" try: done = subprocess.run( ["git", "rev-parse", "--short", "HEAD"], cwd=os.path.dirname(os.path.abspath(__file__)), capture_output=True, text=True, timeout=2.0) if done.returncode == 0: _code_version_cache = done.stdout.strip() except Exception: logger.debug("signal_log: code version unavailable", exc_info=True) return _code_version_cache def setup_key(setup: Optional[Dict[str, Any]], timeframe: Any) -> Optional[str]: """Identity of a logged trade setup for event dedup. ``"||||"`` — the same tuple analyze_signal_log clusters on (tf, direction, round(entry, 2), round(SL, 2), strategy). None when the setup lacks a usable price.""" if not isinstance(setup, dict): return None entry = _finite(setup.get("entry")) stop = _finite(setup.get("stop_loss")) if entry is None or stop is None: return None return "|".join(( str(getattr(timeframe, "value", timeframe) or ""), str(setup.get("direction") or ""), f"{entry:.2f}", f"{stop:.2f}", str(setup.get("strategy") or ""), )) def _is_gap_reason(reason: Any) -> bool: """True when an outcome was left UNRESOLVABLE by a candle gap. advance_outcome writes TWO such reasons — "stored candle snapshots lost continuity" (no overlap with the last processed bar) and "stored candle snapshots contain an internal gap" — and each call site used to match only one of them, so the stream rebuild and the locked-floor rescue skipped half the gap rows. The substring clauses keep states written by older code inside the same net.""" text = str(reason or "") return (text.startswith("stored candle snapshots") or "continuity" in text or "gap" in text) def gate_reason( warnings: Iterable[str], cleaned_warnings: Iterable[str], quality: str, has_setup: bool, ) -> Optional[str]: """The warning that explains why a row carries no setup, or None. Rows WITH a setup have no gate reason. Rows without one report the first gate-prefixed warning (in gate order); an actionable grade with no setup and no gate warning is the setup generator returning None (RR floor / geometry), which the aggregator does not warn about.""" if has_setup: return None pool = list(warnings or []) + list(cleaned_warnings or []) for prefix in _GATE_PREFIXES: for warning in pool: if str(warning).startswith(prefix): return str(warning) if quality in _ACTIONABLE_GRADES: return "SETUP GENERATOR: no setup (RR floor / geometry)" if quality: return f"GRADE {quality}: below setup threshold" return None _SCAN_OUTCOME_COLS = ( "timeframe", "bar_ts", "atr", "close0", "fwd_1h", "fwd_4h", "fwd_24h", "fwd_1h_atr", "fwd_4h_atr", "fwd_24h_atr", "mfe_24h_atr", "mae_24h_atr", "burst_1h", "bars_used", "complete", ) _SCAN_OUTCOME_INSERT = ( "INSERT OR REPLACE INTO scan_outcomes (signal_id, " + ", ".join(_SCAN_OUTCOME_COLS) + ", labeled_at) VALUES (" + ", ".join("?" for _ in range(len(_SCAN_OUTCOME_COLS) + 2)) + ")" ) def _utc(stamp: datetime) -> datetime: """Tz-aware UTC (stored bar labels are UTC; a naive one is assumed).""" if stamp.tzinfo is None: return stamp.replace(tzinfo=timezone.utc) return stamp.astimezone(timezone.utc) def _window_continuous(bars: List[Dict[str, Any]], anchor: datetime, timeframe: Any, max_gap_bars: int = SCAN_LABEL_MAX_GAP_BARS) -> bool: """True when ``bars`` walk forward from ``anchor`` without a real hole. The scheduled-closure judgement is advisory_outcome._has_continuity's (NY-anchored weekend + the daily maintenance hour) applied one step at a time; on top of it a gap of at most ``max_gap_bars`` missing bars is tolerated, because a scan-outcome window is measured from closes, not replayed bar by bar like an advisory trade.""" seconds = _TF_SECONDS.get(normalize_timeframe(timeframe)) if not seconds: return False previous = anchor for bar in bars: stamp = bar["_timestamp"] if stamp <= previous: continue if not _has_continuity([bar], previous, timeframe): missing = int((stamp - previous).total_seconds() // seconds) - 1 if missing > max_gap_bars: return False previous = stamp return True def scan_outcome(bar_ts: Any, timeframe: Any, bars: Iterable[Dict[str, Any]], atr: Optional[float], price: Optional[float] ) -> Optional[Dict[str, Any]]: """Forward-return label for ONE scan, from stored closed candles. ``bars`` is the union of stored bars for the row's primary timeframe. ``close0`` is the close of the bar labelled ``bar_ts`` (the row's own last closed bar), falling back to the logged scan price. A horizon is reported only once it has ELAPSED inside the stream (an unfinished window reports NULLs rather than a truncated move), and ``complete`` additionally requires an unbroken window. Pure: no DB, no clock.""" seconds = _TF_SECONDS.get(normalize_timeframe(timeframe)) anchor = None try: anchor = _utc(datetime.fromisoformat(str(bar_ts).replace(" ", "T"))) except (TypeError, ValueError): anchor = None if not seconds or anchor is None: return None valid = _valid_bars(bars) for bar in valid: bar["_timestamp"] = _utc(bar["_timestamp"]) close0 = next((b["close"] for b in valid if b["_timestamp"] == anchor), None) if close0 is None: close0 = _finite(price) if close0 is None: return None atr = atr if (atr is not None and atr > 0) else None forward = [b for b in valid if b["_timestamp"] > anchor] reach = forward[-1]["_timestamp"] if forward else anchor out: Dict[str, Any] = { "timeframe": str(getattr(timeframe, "value", timeframe)), "bar_ts": str(bar_ts), "atr": atr, "close0": close0, "mfe_24h_atr": None, "mae_24h_atr": None, "burst_1h": None, } window: List[Dict[str, Any]] = [] for hours in SCAN_OUTCOME_HORIZONS_H: edge = anchor + timedelta(hours=hours) upto = [b for b in forward if b["_timestamp"] <= edge] elapsed = reach >= edge and bool(upto) move = (upto[-1]["close"] - close0) if elapsed else None out[f"fwd_{hours}h"] = move out[f"fwd_{hours}h_atr"] = (move / atr) if (move is not None and atr) \ else None if hours == 1 and elapsed and atr: out["burst_1h"] = int(any( (b["high"] - b["low"]) >= SCAN_BURST_ATR_MULTIPLE * atr for b in upto)) if hours == SCAN_OUTCOME_WINDOW_H: window = upto if elapsed and atr: out["mfe_24h_atr"] = (max(b["high"] for b in upto) - close0) / atr out["mae_24h_atr"] = (close0 - min(b["low"] for b in upto)) / atr complete = (reach >= anchor + timedelta(hours=SCAN_OUTCOME_WINDOW_H) and bool(window) and _window_continuous(window, anchor, timeframe)) out["bars_used"] = len(window) out["complete"] = int(complete) return out # ── Serializers (duck-typed; deliberately NOT imported from ── # ── dashboard_publisher — the dashboard's hard independence rule) ── def _event_to_dict(ev: Any) -> Dict[str, Any]: try: surprise = ev.surprise except Exception: surprise = None return { "time": _iso(getattr(ev, "time", None)), "currency": getattr(ev, "currency", None), "impact": getattr(ev, "impact", None), "event": getattr(ev, "event", None), "forecast": getattr(ev, "forecast", None), "previous": getattr(ev, "previous", None), "actual": getattr(ev, "actual", None), "surprise": surprise, } def _headline_to_dict(h: Any) -> Dict[str, Any]: return { "headline": getattr(h, "headline", None), "source": getattr(h, "source", None), "timestamp": _iso(getattr(h, "timestamp", None)), "score": _finite(getattr(h, "sentiment_score", None)), "score_by_currency": { str(currency): _finite(score) for currency, score in ( getattr(h, "sentiment_by_currency", None) or {}).items() }, "currencies": list(getattr(h, "currencies_mentioned", None) or []), "keywords": list(getattr(h, "keywords", None) or []), "link": getattr(h, "link", None), "description": getattr(h, "description", None), "publisher": getattr(h, "publisher", None), # Which scorer produced "score": "keyword" | "llm" | "model". # The durable corpus lives in news_archive.py; this mirrors the # provenance here so signals.db rows are self-describing too. "score_source": getattr(h, "score_source", None), } def _context_news_json(context: Any) -> "tuple[Optional[str], Optional[str]]": """(events_json, headlines_json) from a FundamentalContext, or (None, None) when no context was captured.""" if context is None: return None, None events = { "high_impact_warning": bool( getattr(context, "high_impact_warning", False)), "warning_message": getattr(context, "warning_message", "") or "", "calendar_available": bool( getattr(context, "calendar_available", False)), "sentiment_by_currency": dict( getattr(context, "news_sentiment", None) or {}), "events_today": [ _event_to_dict(e) for e in getattr(context, "events_today", None) or [] ], "events_upcoming_1h": [ _event_to_dict(e) for e in getattr(context, "events_upcoming_1h", None) or [] ], } headlines = [ _headline_to_dict(h) for h in getattr(context, "news_headlines", None) or [] ] return _dumps(events), _dumps(headlines) def _key_str(k: Any) -> str: return getattr(k, "value", str(k)) def _analysis_to_dict(analysis: Any) -> Optional[Dict[str, Any]]: """Compact, JSON-safe snapshot of one TechnicalAnalysis.""" if analysis is None: return None indicators = {} for name, indicator in ( getattr(analysis, "indicators", None) or {}).items(): if str(name).startswith("_"): continue signal = getattr(indicator, "signal", None) indicators[str(name)] = { "signal": getattr(signal, "name", str(signal)), "value": _finite(getattr(indicator, "value", None)), "confidence": _finite(getattr(indicator, "confidence", None)), # `weight` is the BASE weight; `eff_weight` is that weight # times the regime multiplier the composite actually used and # `wscore` its contribution (null for skipped indicators). "weight": _finite(getattr(indicator, "weight", None)), "eff_weight": _finite(getattr(indicator, "effective_weight", None)), "wscore": _finite(getattr(indicator, "weighted_score", None)), "description": getattr(indicator, "description", "") or "", } composite = getattr(analysis, "composite_signal", None) return { "timestamp": str(getattr(analysis, "timestamp", "")), "composite_signal": getattr(composite, "name", str(composite)), "composite_score": _finite( getattr(analysis, "composite_score", None)), "confidence": _finite(getattr(analysis, "confidence", None)), "market_regime": getattr(analysis, "market_regime", None), "volatility_regime": getattr(analysis, "volatility_regime", None), "key_levels": { str(name): _finite(value) for name, value in ( getattr(analysis, "key_levels", None) or {}).items() }, "pattern_signals": list( getattr(analysis, "pattern_signals", None) or []), "indicators": indicators, } def _report_indicators_json(report: Any) -> Optional[str]: analyses = [getattr(report, "primary_analysis", None)] analyses.extend(getattr(report, "confirmation_analyses", None) or []) snapshot = {} for analysis in analyses: if analysis is None: continue timeframe = str(getattr(analysis, "timeframe", "")) if timeframe: snapshot[timeframe] = _analysis_to_dict(analysis) return _dumps(snapshot) if snapshot else None def _candles_json(dataframes: Any, limit: int = 14) -> Optional[str]: """Last closed OHLCV bars per fetched timeframe, oldest first.""" snapshot = {} for timeframe, df in (dataframes or {}).items(): if df is None or getattr(df, "empty", True): continue bars = [] for timestamp, row in df.tail(limit).iterrows(): bars.append({ "timestamp": _iso(timestamp), "open": _finite(row.get("Open")), "high": _finite(row.get("High")), "low": _finite(row.get("Low")), "close": _finite(row.get("Close")), "volume": _finite(row.get("Volume")), }) if bars: snapshot[_key_str(timeframe)] = bars return _dumps(snapshot) if snapshot else None def _report_row( report: Any, context: Any, source: str, timeframe: str, dataframes: Any = None, extra: Optional[Dict[str, Any]] = None, ) -> Dict[str, Any]: """Column-name → value mapping for one scan row (see _SIGNAL_INSERT_COLS). ``extra`` carries producer-side metadata (``scan_kind``, ``config_fingerprint``).""" extra = extra or {} mtf_signals = getattr(report, "mtf_signals", None) or {} mtf_json = _dumps({ "agreement": bool(getattr(report, "mtf_agreement", False)), "signals": {_key_str(k): getattr(v, "name", str(v)) for k, v in mtf_signals.items()}, "scores": {_key_str(k): _finite(v) for k, v in (getattr(report, "mtf_scores", None) or {}).items()}, }) setup = getattr(report, "trade_setup", None) setup_json = None if setup is not None: setup_json = _dumps({ "direction": getattr(setup, "direction", None), "entry": _finite(getattr(setup, "entry_price", None)), "stop_loss": _finite(getattr(setup, "stop_loss", None)), "take_profit_1": _finite(getattr(setup, "take_profit_1", None)), "take_profit_2": _finite(getattr(setup, "take_profit_2", None)), "risk_reward": _finite(getattr(setup, "risk_reward", None)), "risk_pips": _finite(getattr(setup, "risk_pips", None)), "reward_pips": _finite(getattr(setup, "reward_pips", None)), "strategy": getattr(setup, "strategy", None), "setup_pattern": getattr(setup, "setup_pattern", None), "round_trip_cost": _finite( getattr(setup, "round_trip_cost", None)), "trail_arm_r": _finite(getattr(setup, "trail_arm_r", None)), "trail_dist_r": _finite(getattr(setup, "trail_dist_r", None)), }) reasons_json = _dumps({ "bullish": list(getattr(report, "bullish_reasons", None) or []), "bearish": list(getattr(report, "bearish_reasons", None) or []), }) warnings_json = _dumps(list(getattr(report, "warnings", None) or [])) events_json, headlines_json = _context_news_json(context) signal = getattr(report, "final_signal", None) panel_takes = getattr(report, "panel_takes", None) or [] panel_takes_json = _dumps(panel_takes) if panel_takes else None quality = getattr(report, "signal_quality", "") or "" warnings = list(getattr(report, "warnings", None) or []) cleaned = list(getattr(report, "cleaned_warnings", None) or []) setup_dict = json.loads(setup_json) if setup_json else None bar_range = _finite(getattr(report, "spread_estimate", None)) # The primary bar LABEL this scan closed on, stored verbatim (the # same string indicators_json carries, so readers can join on it). primary = getattr(report, "primary_analysis", None) bar_ts = str(getattr(primary, "timestamp", "") or "") or None penalty_tags = getattr(report, "penalty_tags", None) or {} return { "logged_at": datetime.now(timezone.utc).isoformat(), "source": source, "symbol": str(getattr(report, "symbol", "")), "timeframe": timeframe, "signal": getattr(signal, "name", str(signal)), "score": _finite(getattr(report, "final_score", None)), "confidence": _finite(getattr(report, "final_confidence", None)), "quality": quality, "action": getattr(report, "action_recommendation", "") or "", "price": _finite(getattr(report, "current_price", None)), "spread": bar_range, "market_regime": getattr(report, "market_regime", "") or "", "volatility_regime": getattr(report, "volatility_regime", "") or "", "mtf_json": mtf_json, "setup_json": setup_json, "reasons_json": reasons_json, "warnings_json": warnings_json, "event_risk": getattr(report, "event_risk", "") or "", "event_description": getattr(report, "event_description", "") or "", "news_sentiment": _finite(getattr(report, "news_sentiment", None)), "news_description": getattr(report, "news_description", "") or "", "news_summary": getattr(report, "news_summary", "") or "", # Interned by SignalLogger.log into news_snapshots; the inline # columns stay NULL on new rows (legacy rows keep theirs). "events_json": events_json, "headlines_json": headlines_json, "indicators_json": _report_indicators_json(report), "candles_json": _candles_json(dataframes), "panel_verdict": getattr(report, "panel_verdict", None), "panel_confidence": _finite(getattr(report, "panel_confidence", None)), "panel_rationale": getattr(report, "panel_rationale", None), "panel_takes_json": panel_takes_json, "setup_key": setup_key(setup_dict, timeframe), "headlines_hash": None, "events_hash": None, "gate_reason": gate_reason(warnings, cleaned, quality, setup_dict is not None), "actionable": int(setup_dict is not None and quality in _ACTIONABLE_GRADES), "cleaned_warnings_json": _dumps(cleaned) if cleaned else None, "bar_range_median": bar_range, "scan_kind": str(extra.get("scan_kind") or "scan"), "config_fingerprint": extra.get("config_fingerprint"), # Weight-research record keeping (nothing live reads these back). "bar_ts": bar_ts, "engine_fingerprint": extra.get("engine_fingerprint"), "code_version": _code_version() or None, "penalty_tags_json": (_dumps({str(k): _finite(v) for k, v in penalty_tags.items()}) if penalty_tags else None), "penalty_total": _finite(getattr(report, "penalty_total", None)), "momentum_fading": int(bool(getattr(report, "momentum_fading", False))), "exhaustion_count": int( _finite(getattr(report, "exhaustion_count", 0)) or 0), "daily_trend": _daily_trend_value(getattr(report, "daily_trend", None)), } class SignalLogger: """Thread-safe SQLite writer for signal history. One instance per database path; safe to share across the bot's executor threads and the daemon's thread pool.""" def __init__(self, path: str): parent = os.path.dirname(path) if parent: os.makedirs(parent, exist_ok=True) self._conn = sqlite3.connect(path, check_same_thread=False) # busy_timeout BEFORE the journal-mode switch: several processes # (bot, dashboard, tests) open the same file at once and the WAL # switch plus the additive migration need brief exclusive locks. self._conn.execute("PRAGMA busy_timeout=5000") self._lock = threading.Lock() self._next_scan_label = 0.0 # opportunistic labeller throttle with self._lock: self._with_lock_retry( lambda: self._conn.execute("PRAGMA journal_mode=WAL")) self._with_lock_retry( lambda: self._conn.executescript(_SCHEMA)) self._with_lock_retry(self._migrate_signal_columns) self._with_lock_retry(self._ensure_news_view) def _ensure_news_view(self) -> None: """(Re)create the read-side view atomically. Several processes open the DB at once; DROP+CREATE inside one IMMEDIATE transaction serialises them, and a view another opener created in between is not an error.""" try: self._conn.execute("BEGIN IMMEDIATE") self._conn.execute("DROP VIEW IF EXISTS signals_with_news") self._conn.execute( "CREATE VIEW signals_with_news AS" " SELECT s.*," " COALESCE(s.headlines_json, nh.body) AS headlines_resolved," " COALESCE(s.events_json, ne.body) AS events_resolved" " FROM signals s" " LEFT JOIN news_snapshots nh ON nh.hash = s.headlines_hash" " LEFT JOIN news_snapshots ne ON ne.hash = s.events_hash") self._conn.commit() except sqlite3.OperationalError as exc: self._conn.rollback() if "already exists" not in str(exc).lower(): raise @staticmethod def _with_lock_retry(fn, attempts: int = 8, delay: float = 0.05): """Run ``fn`` again on 'database is locked' (concurrent openers racing the same schema step); other errors propagate at once.""" for attempt in range(attempts): try: return fn() except sqlite3.OperationalError as exc: if "locked" not in str(exc).lower() or attempt == attempts - 1: raise time.sleep(delay * (attempt + 1)) def _migrate_signal_columns(self) -> None: """Add newer signal columns to pre-existing tables (idempotent).""" existing = {r[1] for r in self._conn.execute("PRAGMA table_info(signals)")} for name, col_type in _ADDITIVE_SIGNAL_COLS: if name not in existing: try: self._conn.execute( f"ALTER TABLE signals ADD COLUMN {name} {col_type}") except sqlite3.OperationalError as exc: # Another dashboard/bot process may have migrated the # shared database after our PRAGMA snapshot. if "duplicate column name" not in str(exc).lower(): raise self._conn.execute( "CREATE INDEX IF NOT EXISTS idx_signals_advisory_pending" " ON signals (symbol, advisory_outcome_status, id)") self._conn.execute( "CREATE INDEX IF NOT EXISTS idx_signals_setup_key" " ON signals (symbol, setup_key, id)") self._conn.execute( "CREATE INDEX IF NOT EXISTS idx_signals_event" " ON signals (advisory_event_id)") # bar_ts is an additive column, so its index cannot live in # _SCHEMA (which runs before the ALTERs on a legacy database). self._conn.execute( "CREATE INDEX IF NOT EXISTS idx_signals_tf_bar" " ON signals (timeframe, bar_ts)") # Partial index: only alerted rows (a few per day), so the # open-exposure read on every scan never walks the table. self._conn.execute( "CREATE INDEX IF NOT EXISTS idx_signals_alerted" " ON signals (symbol, alerted_at) WHERE alerted_at IS NOT NULL") self._conn.commit() def log( self, report: Any, context: Any = None, source: str = "live", timeframe: str = "", dataframes: Any = None, extra: Optional[Dict[str, Any]] = None, ) -> int: """Insert one scan row; returns the rowid (panel-update key). One transaction interns the news blobs, inserts the row and seeds (or dedups) its advisory outcome; a seeding failure is logged and the row still commits. Pending outcomes are then advanced from this scan's candle snapshot.""" row = _report_row(report, context, source, timeframe, dataframes, extra=extra) now_iso = row["logged_at"] with self._lock: self._conn.execute("BEGIN IMMEDIATE") try: row["headlines_hash"] = self._intern_news( "headlines", row.pop("headlines_json"), now_iso) row["events_hash"] = self._intern_news( "events", row.pop("events_json"), now_iso) row["headlines_json"] = None row["events_json"] = None cur = self._conn.execute( _INSERT, tuple(row[c] for c in _SIGNAL_INSERT_COLS)) rowid = int(cur.lastrowid) try: self._seed_outcome(rowid, row) except Exception: logger.warning("signal_log: outcome seeding failed", exc_info=True) self._conn.commit() except Exception: self._conn.rollback() raise try: self.resolve_pending(str(row["symbol"]), row["candles_json"]) except Exception: logger.warning("signal_log: advisory outcome resolver failed", exc_info=True) # Opportunistic, time-boxed and throttled: the same "advance what # the stored bars now allow" pass, for scan outcomes. Keeps the # research table current without a daemon or a cron entry. try: if time.monotonic() >= self._next_scan_label: self._next_scan_label = (time.monotonic() + SCAN_LABEL_MIN_INTERVAL) self.label_scan_outcomes() except Exception: logger.warning("signal_log: scan-outcome labeller failed", exc_info=True) return rowid # ── news interning ──────────────────────────────────────────── def _intern_news(self, kind: str, body: Optional[str], now_iso: str) -> Optional[str]: """Store one news blob once; returns its content hash.""" if not body: return None digest = _blob_hash(body) self._conn.execute( "INSERT OR IGNORE INTO news_snapshots" " (hash, kind, body, first_seen, last_seen)" " VALUES (?, ?, ?, ?, ?)", (digest, kind, body, now_iso, now_iso)) self._conn.execute( "UPDATE news_snapshots SET last_seen=? WHERE hash=?", (now_iso, digest)) return digest # ── advisory-event seeding / dedup ──────────────────────────── def _seed_outcome(self, rowid: int, row: Dict[str, Any]) -> None: """Seed a PENDING outcome for a new setup row, or mark it a DUPLICATE of the most recent same-setup row within the gap. Runs inside log()'s transaction.""" key = row.get("setup_key") if not key or not row.get("setup_json"): return prior = self._conn.execute( "SELECT id, advisory_event_id, logged_at FROM signals" " WHERE symbol=? AND setup_key=? AND id None: """Fill panel provenance; seeds an outcome only for rows that log() did not already seed (rows written by an older process).""" with self._lock: row = self._conn.execute( "SELECT setup_json, indicators_json, timeframe," " advisory_outcome_status FROM signals WHERE id=?", (rowid,), ).fetchone() if row is None: return advisory_status = row[3] advisory_json = None if advisory_status is None: setup = json.loads(row[0]) if row[0] else None indicators = json.loads(row[1]) if row[1] else None advisory_status, state = initial_outcome( setup, indicators, row[2]) advisory_json = _dumps(state) self._conn.execute( "UPDATE signals SET panel_verdict=?, panel_confidence=?," " panel_rationale=?, panel_takes_json=?," " panel_created_at=COALESCE(?, panel_created_at)," " panel_cache_status=COALESCE(?, panel_cache_status)," " panel_payload_hash=COALESCE(?, panel_payload_hash)," " panel_verdict_source=COALESCE(?, panel_verdict_source)," " panel_prompt_version=COALESCE(?, panel_prompt_version)," " panel_probability=COALESCE(?, panel_probability)," " panel_probs_json=COALESCE(?, panel_probs_json)," " panel_payloads_json=COALESCE(?, panel_payloads_json)," " panel_context_json=COALESCE(?, panel_context_json)," " advisory_outcome_status=COALESCE(advisory_outcome_status, ?)," " advisory_outcome_json=COALESCE(advisory_outcome_json, ?)" " WHERE id=?", (verdict, confidence, rationale, takes_json, created_at, cache_status, payload_hash, verdict_source, prompt_version, probability, probs_json, payloads_json, context_json, advisory_status, advisory_json, rowid)) self._conn.commit() def update_watch(self, rowid: int, level: str, kinds: Any, line: str) -> None: """Fill the violent-move watch fields on an already-logged row. Post-hoc writer mirroring update_panel: violent_watch.attach_watch runs on the surfaces AFTER analyze_pair inserted the row. Level and kinds are written even when the display line is switched off — the evaluation ("did a burst follow a 'hot' scan?") needs the assessment, not the text.""" if isinstance(kinds, (list, tuple, set)): kinds_text = ",".join(str(k) for k in sorted(kinds)) else: kinds_text = str(kinds or "") with self._lock: self._conn.execute( "UPDATE signals SET watch_level=?, watch_kinds=?," " watch_line=? WHERE id=?", (str(level or ""), kinds_text, str(line or ""), int(rowid))) self._conn.commit() def mark_alerted(self, rowid: int, when_iso: Optional[str] = None) -> Optional[str]: """Stamp the time of the alert for this row. The first stamp wins: a re-alert of the same row keeps the original time. Returns the stamp when THIS call wrote it and None when the row already had one (or does not exist), so a caller that must undo its own stamp can never clear an earlier alert's.""" when_iso = when_iso or datetime.now(timezone.utc).isoformat() with self._lock: cur = self._conn.execute( "UPDATE signals SET alerted_at=?" " WHERE id=? AND alerted_at IS NULL", (when_iso, int(rowid))) self._conn.commit() return when_iso if cur.rowcount == 1 else None def unmark_alerted(self, rowid: int, when_iso: str) -> bool: """Clear the stamp mark_alerted returned (the alert reached no chat). Only that exact stamp is cleared; True when it was.""" with self._lock: cur = self._conn.execute( "UPDATE signals SET alerted_at=NULL" " WHERE id=? AND alerted_at=?", (int(rowid), str(when_iso))) self._conn.commit() return cur.rowcount == 1 def resolve_pending( self, symbol: str, candles_json: Optional[str], limit: int = 100 ) -> int: """Advance pending advisory outcomes from a new candle snapshot. The inserting scan's own snapshot is tried first (cheap, covers the steady state). When it does not overlap an outcome's last processed bar — that process missed >14 bars of this timeframe, a weekend straddle, a restart — the resolver rebuilds the bar stream for the timeframe from every stored snapshot after that bar (any surface) and retries, so the outcome is not marked UNRESOLVABLE for a gap the database can fill. Each row commits on its own.""" if not candles_json: return 0 snapshots = json.loads(candles_json) if not isinstance(snapshots, dict): return 0 changed = 0 with self._lock: rows = self._conn.execute( "SELECT id, timeframe FROM signals" " WHERE symbol=? AND advisory_outcome_status='PENDING'" " ORDER BY id", (symbol,), ).fetchall() eligible = 0 for rowid, timeframe in rows: bars = timeframe_value(snapshots, timeframe) if not bars: continue if eligible >= max(1, int(limit)): break eligible += 1 # One IMMEDIATE transaction per row: the state is re-read # under the write lock, so the bot and the dashboard (each # advancing from its own snapshot) serialise instead of # overwriting each other's progress. self._conn.execute("BEGIN IMMEDIATE") try: hit = self._conn.execute( "SELECT advisory_outcome_json FROM signals" " WHERE id=? AND advisory_outcome_status='PENDING'", (rowid,)).fetchone() if hit is None or not hit[0]: self._conn.rollback() continue before = _dumps(json.loads(hit[0])) advanced = advance_outcome(json.loads(before), bars) if (advanced.get("status") == "UNRESOLVABLE" and _is_gap_reason(advanced.get("reason"))): union = self._stream_since( symbol, timeframe, json.loads(before)["progress"]["last_processed_bar"]) if union: advanced = advance_outcome(json.loads(before), union) # The database could not fill the hole. A trade # whose trail had already locked profit still has a # real answer — its floor — so record that instead # of dropping the row out of the scorecard. if (advanced.get("status") == "UNRESOLVABLE" and _is_gap_reason(advanced.get("reason"))): advanced = resolve_locked_floor(advanced) after = _dumps(advanced) if after == before: self._conn.rollback() continue status = advanced.get("status", "UNRESOLVABLE") self._conn.execute( "UPDATE signals SET advisory_outcome_status=?," " advisory_r_multiple=?, advisory_outcome_json=?" " WHERE id=?", (status, canonical_r(advanced), after, rowid), ) self._conn.commit() changed += 1 except Exception: self._conn.rollback() raise return changed def _stream_since(self, symbol: str, timeframe: str, last_bar: str, max_rows: int = 4000) -> List[Dict[str, Any]]: """Union of every stored closed bar of ``timeframe`` from ``last_bar`` onward, across all rows/surfaces (oldest first). Only the requested timeframe's array is pulled out of each candles_json (json_extract), so the cost is bounded by the number of rows logged since that bar, not by blob size.""" tf = str(getattr(timeframe, "value", timeframe)) try: since = datetime.fromisoformat( str(last_bar).replace(" ", "T")).isoformat() except (TypeError, ValueError): return [] bars: Dict[str, Dict[str, Any]] = {} for key in _tf_storage_keys(tf): rows = self._conn.execute( "SELECT json_extract(candles_json, ?) FROM signals" " WHERE symbol=? AND candles_json IS NOT NULL" " AND logged_at >= ? ORDER BY id LIMIT ?", (f'$."{key}"', symbol, since, int(max_rows))).fetchall() if len(rows) == int(max_rows): logger.warning( "signal_log: bar stream truncated for %s %s (key %s)" " since %s — hit the %s-row cap, later bars are missing", symbol, tf, key, since, int(max_rows)) for (blob,) in rows: if not blob: continue try: for bar in json.loads(blob): stamp = bar.get("timestamp") if stamp and stamp >= since: bars.setdefault(str(stamp), bar) except (TypeError, ValueError, AttributeError): continue return [bars[k] for k in sorted(bars)] def label_scan_outcomes(self, budget: int = 150, now: Any = None, time_budget: Optional[float] = None) -> int: """Label forward returns for scans whose 24 h window has closed. Candidates are rows that carry indicator snapshots (a real scan) and a price, whose signal bar closed more than 25 h ago and which have no COMPLETE label yet. An incomplete row is re-examined only while it is younger than SCAN_LABEL_MAX_AGE_H (counted from the newest logged row when the database is behind the clock) — beyond that its window had a real hole that no later bar can fill, so the label it already carries is final — and a stale row never anchors the shared bar stream either (a key that has nothing else due starts at the age bound plus one window). Both bounds are load-bearing: a handful of permanently incomplete rows (a feed outage) sort to the front of every pass, and anchoring the stream at their bar pushes its row cap past today, which silently labels every genuinely due row NULL and incomplete — the scan_outcomes table then goes blind. Returns the number of rows actually WRITTEN — a row whose numbers did not change is not rewritten (only its labeled_at is bumped, so it rotates to the back of the queue), which makes repeated passes idempotent and lets a batch driver stop at 0. Bounded on purpose: at most ``budget`` candidates are examined and the pass stops after ``time_budget`` seconds (default 2 s) so the opportunistic call inside log() stays cheap. Per-row failures are logged and skipped.""" moment = now or datetime.now(timezone.utc) moment_utc = _utc(moment) cutoff = moment_utc - timedelta(hours=SCAN_LABEL_MIN_AGE_H) deadline = time.monotonic() + float( SCAN_LABEL_TIME_BUDGET if time_budget is None else time_budget) written = 0 with self._lock: # Age is measured from the newest row when the database itself # is behind the clock (an imported snapshot on the dev box), # otherwise every row of a 3-day-old import would be declared # final on the first pass. On a live logger the two coincide. horizon = moment_utc newest = self._conn.execute( "SELECT logged_at FROM signals ORDER BY id DESC" " LIMIT 1").fetchone() try: horizon = min(horizon, _utc(datetime.fromisoformat( str(newest[0]).replace(" ", "T")))) except (TypeError, ValueError, IndexError): pass stale_before = horizon - timedelta(hours=SCAN_LABEL_MAX_AGE_H) # Deepest a stream may start when every row due for that key is # stale: one window before the age bound, so such a row still # gets its single final pass without dragging the stream back. anchor_floor = stale_before - timedelta(hours=SCAN_LABEL_MIN_AGE_H) # Row ids are monotonic in logged_at (a uniform ISO string), so # an id bound is the cheap, index-friendly way to leave stale # incomplete rows out of the re-examination set. bar_ts mixes # separators and must never be compared as a string. min_id = self._conn.execute( "SELECT MIN(id) FROM signals WHERE logged_at >= ?", (stale_before.isoformat(),)).fetchone()[0] if min_id is None: min_id = 2 ** 62 # nothing recent: re-examine none rows = self._conn.execute( "SELECT s.id, s.symbol, s.timeframe, s.price," " COALESCE(s.bar_ts, json_extract(s.indicators_json," " '$.\"' || s.timeframe || '\".timestamp')) AS bar," " json_extract(s.indicators_json," " '$.\"' || s.timeframe || '\".key_levels.atr') AS atr," " (o.signal_id IS NULL) AS unlabelled" " FROM signals s" " LEFT JOIN scan_outcomes o ON o.signal_id = s.id" " WHERE s.indicators_json IS NOT NULL AND s.price > 0" " AND (o.signal_id IS NULL OR (o.complete = 0 AND s.id >= ?))" " AND bar IS NOT NULL" " ORDER BY COALESCE(o.labeled_at, '') ASC, s.id ASC" " LIMIT ?", (int(min_id), int(budget))).fetchall() # One bar stream per (symbol, timeframe) for the whole batch, # built from the OLDEST bar in it — _stream_since is a scan # over every row logged since that bar, far too expensive to # repeat per candidate. due: List[Tuple[Any, ...]] = [] oldest: Dict[Tuple[str, str], Tuple[datetime, str]] = {} for rowid, symbol, tf, price, bar, atr, unlabelled in rows: try: stamp = _utc(datetime.fromisoformat( str(bar).replace(" ", "T"))) except (TypeError, ValueError): continue if stamp > cutoff: continue # window has not elapsed yet # Stale = old by the bar it closed on AND by when it was # written (on a live database the two agree within minutes; # requiring both keeps a freshly imported or backfilled DB, # whose rows all carry today's logged_at, out of it). stale = stamp < stale_before and rowid < min_id if stale and not unlabelled: continue # final: a hole no later bar can ever fill key = (str(symbol), str(tf)) if not stale and (key not in oldest or stamp < oldest[key][0]): oldest[key] = (stamp, str(bar)) due.append((rowid, symbol, tf, price, bar, atr, key)) streams: Dict[Tuple[str, str], List[Dict[str, Any]]] = {} for rowid, symbol, tf, price, bar, atr, key in due: if time.monotonic() > deadline: break try: if key not in streams: # 3 days of anchor is ~2,100 rows on the live DB; # the cap is a safety net, not the working limit. anchor = oldest.get(key) streams[key] = self._stream_since( symbol, tf, anchor[1] if anchor else anchor_floor.isoformat(), max_rows=10000) label = scan_outcome(bar, tf, streams[key], _finite(atr), price) if label is None: continue values = tuple(label[col] for col in _SCAN_OUTCOME_COLS) prior = self._conn.execute( "SELECT " + ", ".join(_SCAN_OUTCOME_COLS) + " FROM scan_outcomes WHERE signal_id=?", (rowid,)).fetchone() if prior is not None and tuple(prior) == values: # Unchanged, so NOT counted as written — but its # queue position moves, or the same short-stream # rows would head every pass for ever. now_iso = datetime.now(timezone.utc).isoformat() self._conn.execute( "UPDATE scan_outcomes SET labeled_at=?" " WHERE signal_id=?", (now_iso, rowid)) self._conn.commit() continue self._conn.execute( _SCAN_OUTCOME_INSERT, (int(rowid),) + values + (datetime.now(timezone.utc).isoformat(),)) self._conn.commit() written += 1 except Exception: logger.warning( "signal_log: scan-outcome labelling failed for row %s", rowid, exc_info=True) return written def backfill_seeds(self, limit: Optional[int] = None) -> Dict[str, int]: """Assign setup_key / advisory_event_id to legacy setup rows and seed the canonical ones that carry no outcome yet. Existing outcomes (panel-seeded rows) are never overwritten. Safe to run beside live writers (short per-row transactions).""" stats = {"keyed": 0, "duplicates": 0, "seeded": 0} with self._lock: rows = self._conn.execute( "SELECT id, symbol, timeframe, logged_at, setup_json," " indicators_json, advisory_outcome_status, advisory_event_id" " FROM signals WHERE setup_json IS NOT NULL" " AND setup_json != 'null' ORDER BY id" + (f" LIMIT {int(limit)}" if limit else "")).fetchall() last_by_key: Dict[Tuple[str, str], Tuple[int, int, datetime]] = {} for (rowid, symbol, tf, logged_at, setup_json, ind_json, status, event_id) in rows: setup = json.loads(setup_json) key = setup_key(setup, tf) if not key: continue try: ts = datetime.fromisoformat(logged_at) except (TypeError, ValueError): continue prior = last_by_key.get((symbol, key)) if (prior is not None and abs((ts - prior[2]).total_seconds()) <= OUTCOME_DEDUP_GAP_SECONDS): # Same setup re-logged inside the gap: a duplicate of # the canonical event regardless of its legacy status # (a panel-seeded RESOLVED re-log measured the very # same trade — keeping it would double count R). canonical = prior[1] if status != "DUPLICATE" or event_id != canonical: self._conn.execute( "UPDATE signals SET setup_key=?," " advisory_event_id=?," " advisory_outcome_status='DUPLICATE'," " advisory_r_multiple=NULL," " advisory_outcome_json=? WHERE id=?", (key, canonical, _dumps({"model": "advisory-event-dedup-v1", "status": "DUPLICATE", "event_id": canonical, "superseded_status": status}), rowid)) stats["duplicates"] += 1 last_by_key[(symbol, key)] = (rowid, canonical, ts) continue canonical = int(event_id or rowid) if status is None: indicators = json.loads(ind_json) if ind_json else None new_status, state = initial_outcome(setup, indicators, tf) self._conn.execute( "UPDATE signals SET setup_key=?, advisory_event_id=?," " advisory_outcome_status=?, advisory_outcome_json=?" " WHERE id=?", (key, canonical, new_status, _dumps(state), rowid)) stats["seeded"] += 1 else: self._conn.execute( "UPDATE signals SET setup_key=?, advisory_event_id=?" " WHERE id=?", (key, canonical, rowid)) stats["keyed"] += 1 last_by_key[(symbol, key)] = (rowid, canonical, ts) self._conn.commit() return stats def resolve_from_store(self, symbol: str, limit: int = 500) -> int: """Advance every PENDING outcome from the stored bar streams alone (no provider). Used by the backfill script after seeding.""" changed = 0 with self._lock: rows = self._conn.execute( "SELECT id, timeframe, advisory_outcome_json FROM signals" " WHERE symbol=? AND advisory_outcome_status='PENDING'" " ORDER BY id LIMIT ?", (symbol, int(limit))).fetchall() for rowid, timeframe, state_json in rows: if not state_json: continue state = json.loads(state_json) union = self._stream_since( symbol, timeframe, state["progress"]["last_processed_bar"]) if not union: continue before = _dumps(state) advanced = advance_outcome(json.loads(before), union) if (advanced.get("status") == "UNRESOLVABLE" and _is_gap_reason(advanced.get("reason"))): advanced = resolve_locked_floor(advanced) after = _dumps(advanced) if after == before: continue self._conn.execute( "UPDATE signals SET advisory_outcome_status=?," " advisory_r_multiple=?, advisory_outcome_json=?" " WHERE id=? AND advisory_outcome_status='PENDING'", (advanced.get("status", "UNRESOLVABLE"), canonical_r(advanced), after, rowid)) self._conn.commit() changed += 1 return changed def reseed_unresolvable(self, max_reseeds: int = 1) -> List[int]: """Bounded retry for continuity-broken outcomes: re-freeze the seed from the row's own setup/indicators (at most ``max_reseeds`` times per row) so the next resolver pass can walk the now-filled stream. Invalid-seed rows stay terminal. A gap row that had already trailed its stop past the entry is finished at that locked floor instead of re-seeded; the returned list holds re-seeded ids only.""" reseeded: List[int] = [] with self._lock: rows = self._conn.execute( "SELECT id, timeframe, setup_json, indicators_json," " advisory_outcome_json FROM signals" " WHERE advisory_outcome_status='UNRESOLVABLE'" " AND advisory_event_id = id").fetchall() for rowid, tf, setup_json, ind_json, state_json in rows: old = json.loads(state_json) if state_json else {} if not _is_gap_reason(old.get("reason")): continue # The locked trailing stop lives in this progress block. A # re-seed would rebuild the walk from scratch only to stop # at the same hole (advance_outcome checks continuity bar # by bar and keeps the progress up to the hole) and be # floored at the same stop, so finish such a row here; # re-seed only what has nothing locked. floored = resolve_locked_floor(json.loads(_dumps(old))) if floored.get("status") == "RESOLVED": self._conn.execute( "UPDATE signals SET advisory_outcome_status=?," " advisory_r_multiple=?, advisory_outcome_json=?" " WHERE id=?", ("RESOLVED", canonical_r(floored), _dumps(floored), rowid)) continue count = int((old.get("reseed") or {}).get("count", 0)) if count >= max_reseeds or not setup_json: continue indicators = json.loads(ind_json) if ind_json else None status, state = initial_outcome( json.loads(setup_json), indicators, tf) if status != "PENDING": continue state["reseed"] = { "count": count + 1, "at": datetime.now(timezone.utc).isoformat()} self._conn.execute( "UPDATE signals SET advisory_outcome_status=?," " advisory_r_multiple=NULL, advisory_outcome_json=?" " WHERE id=?", (status, _dumps(state), rowid)) reseeded.append(int(rowid)) self._conn.commit() return reseeded def close(self) -> None: with self._lock: self._conn.close() def _tf_storage_keys(tf: str) -> List[str]: """Keys under which a timeframe's bars may be stored in candles_json (enum value plus the D1 alias used by older rows).""" keys = [tf] if tf in ("1d", "1day"): keys = ["1d", "1day"] return keys def read_news(con: sqlite3.Connection, rowid: int ) -> Tuple[Optional[str], Optional[str]]: """(events_json, headlines_json) for ``rowid`` on ANY connection — resolves interned snapshots and falls back to the legacy inline columns. Read-only.""" row = con.execute( "SELECT events_json, headlines_json, events_hash, headlines_hash" " FROM signals WHERE id=?", (rowid,)).fetchone() if row is None: return None, None events, headlines, ev_hash, hl_hash = row if events is None and ev_hash: hit = con.execute("SELECT body FROM news_snapshots WHERE hash=?", (ev_hash,)).fetchone() events = hit[0] if hit else None if headlines is None and hl_hash: hit = con.execute("SELECT body FROM news_snapshots WHERE hash=?", (hl_hash,)).fetchone() headlines = hit[0] if hit else None return events, headlines # ── Module-level entry point (config-driven, fail-safe) ─────────────── _loggers: Dict[str, SignalLogger] = {} _loggers_lock = threading.Lock() def log_signal( report: Any, context: Any = None, source: str = "live", timeframe: str = "", dataframes: Any = None, extra: Optional[Dict[str, Any]] = None, ) -> None: """Persist one scan if the signal_log config block enables it. Stashes the rowid on the report (``_signal_log_rowid``) so a post-hoc panel attach can update this same row. NEVER raises — a logging failure must not break a live scan. ``extra`` = producer metadata (``scan_kind``, ``config_fingerprint``); blocked scans are skipped when ``signal_log.log_blocked_scans`` is false.""" try: if not app_config.signal_log_enabled(): return if ((extra or {}).get("scan_kind") == "blocked" and not app_config.signal_log_blocked_scans()): return path = app_config.signal_log_path() with _loggers_lock: lg = _loggers.get(path) if lg is None: lg = SignalLogger(path) _loggers[path] = lg rowid = lg.log(report, context=context, source=source, timeframe=timeframe, dataframes=dataframes, extra=extra) try: report._signal_log_rowid = rowid except Exception: pass # exotic report object — panel update just won't happen except Exception: logger.warning("signal_log: failed to persist signal", exc_info=True) def log_panel_update(report: Any) -> None: """Write post-hoc panel fields onto the row logged for this scan. Matches on the private rowid hand-off from log_signal; a report that was never logged (backtest replay, logging disabled, logger in another process) is a silent no-op. NEVER raises.""" try: rowid = getattr(report, "_signal_log_rowid", None) if rowid is None or not app_config.signal_log_enabled(): return verdict = getattr(report, "panel_verdict", "") or "" if not verdict: return path = app_config.signal_log_path() with _loggers_lock: lg = _loggers.get(path) if lg is None: return takes = getattr(report, "panel_takes", None) or [] probs = [{"persona": t.get("persona"), "p_win": _finite(t.get("p_win")), "stance": t.get("stance"), "confidence": _finite(t.get("confidence")), "reason_for": t.get("reason_for") or "", "reason_against": t.get("reason_against") or ""} for t in takes if isinstance(t, dict)] payloads = getattr(report, "panel_payloads", None) or {} context = getattr(report, "panel_context", None) or {} probability = _finite(getattr(report, "panel_probability", None)) lg.update_panel( int(rowid), verdict, _finite(getattr(report, "panel_confidence", None)), getattr(report, "panel_rationale", "") or "", _dumps(takes), created_at=getattr(report, "panel_created_at", "") or None, cache_status=getattr(report, "panel_cache_status", "") or None, payload_hash=getattr(report, "panel_payload_hash", "") or None, verdict_source=getattr(report, "panel_verdict_source", "") or None, prompt_version=getattr(report, "panel_prompt_version", "") or None, probability=probability if probability else None, probs_json=_dumps(probs) if probs else None, payloads_json=_dumps(payloads) if payloads else None, context_json=_dumps(context) if context else None) except Exception: logger.warning("signal_log: failed to persist panel verdict", exc_info=True) def log_watch_update(report: Any, watch: Any) -> None: """Write the post-hoc violent-move watch fields onto this scan's row. Mirrors log_panel_update: matches on the private rowid hand-off from log_signal, and a report that was never logged (backtest replay, logging disabled, another process) is a silent no-op. NEVER raises.""" try: rowid = getattr(report, "_signal_log_rowid", None) if rowid is None or not app_config.signal_log_enabled(): return path = app_config.signal_log_path() with _loggers_lock: lg = _loggers.get(path) if lg is None: return lg.update_watch( int(rowid), str(getattr(watch, "level", "") or ""), getattr(watch, "kinds", None) or [], getattr(report, "watch_line", "") or "") except Exception: logger.warning("signal_log: failed to persist violent-move watch", exc_info=True) def _logged_row(report: Any) -> Optional[Tuple["SignalLogger", int]]: """(logger, rowid) of the row log_signal wrote for this scan; None for a report that was never logged or a process with logging off.""" rowid = getattr(report, "_signal_log_rowid", None) if rowid is None or not app_config.signal_log_enabled(): return None path = app_config.signal_log_path() with _loggers_lock: lg = _loggers.get(path) return None if lg is None else (lg, int(rowid)) def mark_alerted(report: Any) -> Optional[str]: """Stamp `alerted_at` on the row logged for this scan. The bot calls this just BEFORE it sends the alert, so an alert sent a moment later on another timeframe already counts this plan, and undoes it with unmark_alerted when no chat received the message. Same hand-off as log_panel_update (the private rowid from log_signal). Returns the stamp this call wrote; None when nothing was written (never logged, logging off, or the row already carried a stamp). NEVER raises.""" try: hit = _logged_row(report) return None if hit is None else hit[0].mark_alerted(hit[1]) except Exception: logger.warning("signal_log: failed to mark the alerted row", exc_info=True) return None def unmark_alerted(report: Any, stamp: Optional[str]) -> bool: """Clear the stamp mark_alerted(report) returned: the alert reached no chat, so the row must not count as a plan the reader holds. A no-op without a stamp; only that exact stamp is cleared. NEVER raises.""" if not stamp: return False try: hit = _logged_row(report) return False if hit is None else hit[0].unmark_alerted(hit[1], stamp) except Exception: logger.warning("signal_log: failed to clear the alerted stamp", exc_info=True) return False # A PENDING outcome older than this is not shown as "open": the advisory # model times a trade out after MAX_HOLD_BARS bars, so an older PENDING row # is one the resolver never reached (bot outage, candle gap), not a trade # still running. Calendar allowance: 7/5 for weekends plus three days. def _open_setup_max_age_seconds(timeframe: Any) -> float: bar = _TF_SECONDS.get(normalize_timeframe(timeframe), 3600) return MAX_HOLD_BARS * bar * 1.4 + 3 * 86400 def open_alerted_setups(symbol: str, exclude_key: Optional[str] = None, now: Optional[datetime] = None, path: Optional[str] = None) -> List[Dict[str, Any]]: """ALERTED trade plans whose advisory outcome is still open, oldest first: ``[{id, event_id, timeframe, direction, entry, stop_loss, alerted_at, setup_key}]``, one entry per advisory event. Read-only (a ``mode=ro`` connection of its own, so the dashboard process, which never writes this database, can call it). "Open" means the row's advisory event is still PENDING in the cleaned-mid outcome model: next-open entry, stop first, close-only trailing. That is the model's view of a plan the reader MAY have taken, not a broker position. ``exclude_key`` drops the plan being alerted now from its own count. NEVER raises: any failure (no database yet, an older schema without ``alerted_at``) returns an empty list.""" try: db_path = path or app_config.signal_log_path() if not os.path.exists(db_path): return [] now = now or datetime.now(timezone.utc) con = sqlite3.connect(f"file:{db_path}?mode=ro", uri=True, timeout=2.0) try: rows = con.execute( "SELECT a.id, COALESCE(a.advisory_event_id, a.id), a.timeframe," " a.setup_json, a.alerted_at, a.setup_key" " FROM signals a JOIN signals e" " ON e.id = COALESCE(a.advisory_event_id, a.id)" " WHERE a.symbol=? AND a.alerted_at IS NOT NULL" " AND e.advisory_outcome_status='PENDING'" " ORDER BY a.id", (symbol,)).fetchall() finally: con.close() out: List[Dict[str, Any]] = [] seen = set() for rowid, event_id, timeframe, setup_json, alerted_at, key in rows: if event_id in seen or (exclude_key and key == exclude_key): continue try: age = (now - datetime.fromisoformat(alerted_at)).total_seconds() except (TypeError, ValueError): continue if age > _open_setup_max_age_seconds(timeframe): continue setup = json.loads(setup_json) if setup_json else {} direction = setup.get("direction") if direction not in ("BUY", "SELL"): continue seen.add(event_id) out.append({ "id": int(rowid), "event_id": int(event_id), "timeframe": normalize_timeframe(timeframe), "direction": direction, "entry": _finite(setup.get("entry")), "stop_loss": _finite(setup.get("stop_loss")), "alerted_at": alerted_at, "setup_key": key, }) return out except Exception: logger.debug("signal_log: open alerted setups unavailable", exc_info=True) return [] def last_alerted_bars(path: Optional[str] = None ) -> Dict[Tuple[str, str], str]: """``{(symbol, timeframe): bar label}`` of the newest ALERTED row per symbol and timeframe (newest = highest id). The bot seeds its 5-bar entry cooldown from this at start-up, so a restart does not forget the last alert. Read-only (``mode=ro``); NEVER raises: no database, an older schema or a locked file returns ``{}``.""" try: db_path = path or app_config.signal_log_path() if not os.path.exists(db_path): return {} con = sqlite3.connect(f"file:{db_path}?mode=ro", uri=True, timeout=2.0) try: rows = con.execute( "SELECT s.symbol, s.timeframe, s.bar_ts FROM signals s" " JOIN (SELECT MAX(id) AS id FROM signals" " WHERE alerted_at IS NOT NULL" " GROUP BY symbol, timeframe) m ON m.id = s.id" ).fetchall() finally: con.close() return {(str(symbol), normalize_timeframe(timeframe)): str(bar) for symbol, timeframe, bar in rows if bar} except Exception: logger.debug("signal_log: last alerted bars unavailable", exc_info=True) return {} def reset_loggers_for_tests() -> None: """Close and drop all cached loggers (test isolation).""" with _loggers_lock: for lg in _loggers.values(): try: lg.close() except Exception: pass _loggers.clear()