diff --git a/CLAUDE.md b/CLAUDE.md index a55da2c..126208f 100644 --- a/CLAUDE.md +++ b/CLAUDE.md @@ -11,7 +11,7 @@ Every numeric threshold in `config.py` and `pair_signals/xauusd.py` was backtest ## Commands ```bash -python test_all.py # QA suite: 406 tests, 27 modules, no network (mocked) +python test_all.py # QA suite: 414 tests, 27 modules, no network (mocked) python main.py --pair XAUUSD --timeframe H1 --detail # CLI signal report (--json for full dump, # --panel to run the advisory LLM agent panel on-demand) python telegram_bot.py # the bot (refuses to boot without credentials) @@ -58,8 +58,8 @@ Signal currency everywhere is `SignalStrength` (−2..+2). DataFrames are UTC tz - **`engine.py`** — `TechnicalEngine`: 9 scored indicators (ema_trend, rsi, macd, bollinger, adx, roc, keltner, divergence, candlestick) + ATR as context. Regime classification (trending_up/down, `trending` = mixed/direction-UNKNOWN, transitioning, ranging) drives per-indicator weight multipliers and regime-aware RSI reinterpretation. Compute order matters (ATR first; MACD always computed even when not a pillar — divergence needs the histogram). Private series ride on the result as `_underscore` attrs; scorers/formatters skip anything `_`-prefixed or weight-0. - **`signal_aggregator.py`** — MTF agreement, diminishing-decay penalty system, exhaustion hard block, quality grading, then an ordered chain of post-grade gates (session, XAU SELL quality/vol/hour, regime blocks, M5 noise gates), then three setup generators: XAU ATR (A+/A), generic swing-based, plus the M5 pullback (`XAU_PB`) and chart-pattern (`XAU_CP`, `chart_patterns.py`) entry paths. The PB/CP generators run AFTER the gate chain and force quality to "B"; the CP path deliberately accepts grade-F composites and re-checks only session/day/big-candle (via warning prefixes) plus its own SELL/vol/RSI gates — it does NOT re-check exhaustion or the M5 min-score/MTF gates. **This is validated behavior, not a bug** (2026-08-04 audit: 226/586 trades in the shipped M5 12mo book are XAU_CP, +38.9R, 104 of them exhaustion-flagged and net-positive) — changing it is a rule-1 revalidation event. `TradeSetup.strategy` selects trailing params; trail params are stamped on the setup from the *producing* config so per-TF overrides reach formatters. - **`pair_signals/`** — per-pair config factories. **Always use the factory (`produce_signal` / `make_config`), never bare `PlatformConfig()`** — the factory applies the config.json overlay first, then the validated per-TF deltas (M5 hour-9 block, M15 trail arm 0.70, M15 SELL-hour block, M5 SELL-in-transitioning block, H1 all-shorts block (2026-09-03), etc.; evidence in the docstring). A test enforces factory ≡ inline-config equivalence. Confirmation map: M5→[M15,H1], M15→[M5,H1], H1→[M15,H4], H4→[H1,D1]. -- **`signal_log.py` / `advisory_outcome.py`** — SQLite signal-history logger (stdlib only). Hooked into `SignalAggregator.analyze_pair` only (the live entry point — the backtester never writes here, preserving parity), so every live scan from every surface (Telegram `telegram`, CLI `cli`, dashboard `dashboard`) lands one row in `logs/signals.db`: signal/grade/price, MTF + trade-setup + reasons JSON, full per-TF indicator snapshots, the last 14 closed OHLCV candles per fetched TF, and the news at scan time (event risk + calendar events + scored headlines, re-fetched via the NewsEngine class-level caches). Config.json `signal_log` block (`enabled`, `path`, `log_blocked_scans`; env `SIGNAL_LOG_ENABLED`/`SIGNAL_LOG_PATH`/`SIGNAL_LOG_BLOCKED_SCANS`). **Fail-safe by contract** — a logging failure can never break a scan. Panel fields arrive post-hoc: `log_signal` stashes the rowid and `agent_panel.attach_panel` writes verdict + creation/cache provenance + payload hash onto the same row. **Advisory outcomes (2026-09):** EVERY row with a trade setup is seeded at log time (not only panel rows); same-setup re-logs within 60 min from any surface (`setup_key` = tf|direction|entry|stop|strategy) are `DUPLICATE` rows pointing at the canonical row via `advisory_event_id`, so sum `advisory_r_multiple` over `RESOLVED` rows only. The versioned model advances from later stored closed-candle snapshots (next-open reanchoring, intrabar peak with close-only ratchet, SL-first, scheduled weekend/21:00-UTC closures allowed, 300-bar timeout, $0.30/$0.65 views); when a snapshot no longer overlaps the outcome's last bar it rebuilds the stream from every stored snapshot of that TF before giving up. It is a cleaned-mid evidence model, not execution P&L. **Post-gate truth columns:** `gate_reason` (the warning that removed the setup), `actionable` (grade ≥ C AND setup — the `action` text is written BEFORE the gate chain and can say "STRONG SELL — High-quality setup" on a D row), `cleaned_warnings_json` (what the XAU_CP acceptance stripped), `bar_range_median` (= the misnamed `spread` column: median High-Low, NOT a broker spread), `scan_kind` (`blocked` weekend/day early returns carry no news/candles), `config_fingerprint` (gate config hash — surfaces config drift between processes). **News is content-addressed:** `headlines_json`/`events_json` are interned into `news_snapshots` (they were 73% of a 667 MB DB); read through the `signals_with_news` view or `signal_log.read_news`. Readers must open with `mode=ro`, never `immutable=1` (blind to WAL rows). Tools: `analyze_signal_log.py` (`--stored-candles` DB-only replay with event/opportunity/one-position books + persona scorecard, `--parity` telegram-vs-dashboard divergence report), `scripts/backfill_signal_outcomes.py` (seed/dedup/advance legacy rows), `scripts/migrate_signal_log_news.py` (intern legacy blobs; `--vacuum` with both writers stopped). -- **`news_engine.py`** — Forex Factory calendar (MyFXBook RSS fallback) + RSS keyword sentiment with gold-inverse keywords. Class-level caches (fresh engine per scan). Optional DeepSeek enrichment via `llm_client.py` (requests-only, no SDK): batched headline sentiment overriding keyword scores, gold-direction on release surprises, and a market-summary blurb (`SignalReport.news_summary`) — **display/warning-only, keyword fallback on any failure, inert without a key**; prompts are static module constants (DeepSeek prefix-cache pricing) and requests send `thinking: disabled` (V4-Flash reasoning otherwise blows the timeout). By XAU-tuned default, event risk and sentiment **warn but do not gate signals** (`event_risk_caps_signals=False`, `sentiment_affects_signals=False`) — the LLM does not change that. +- **`signal_log.py` / `advisory_outcome.py`** — SQLite signal-history logger (stdlib only). Hooked into `SignalAggregator.analyze_pair` only (the live entry point — the backtester never writes here, preserving parity), so every live scan from every surface (Telegram `telegram`, CLI `cli`, dashboard `dashboard`) lands one row in `logs/signals.db`: signal/grade/price, MTF + trade-setup + reasons JSON, full per-TF indicator snapshots, the last 14 closed OHLCV candles per fetched TF, and the news at scan time (event risk + calendar events + scored headlines, re-fetched via the NewsEngine class-level caches). Config.json `signal_log` block (`enabled`, `path`, `log_blocked_scans`; env `SIGNAL_LOG_ENABLED`/`SIGNAL_LOG_PATH`/`SIGNAL_LOG_BLOCKED_SCANS`). **Fail-safe by contract** — a logging failure can never break a scan. Panel fields arrive post-hoc: `log_signal` stashes the rowid and `agent_panel.attach_panel` writes verdict + creation/cache provenance + payload hash onto the same row. **Advisory outcomes (2026-09):** EVERY row with a trade setup is seeded at log time (not only panel rows); same-setup re-logs within 60 min from any surface (`setup_key` = tf|direction|entry|stop|strategy) are `DUPLICATE` rows pointing at the canonical row via `advisory_event_id`, so sum `advisory_r_multiple` over `RESOLVED` rows only. The versioned model advances from later stored closed-candle snapshots (next-open reanchoring, intrabar peak with close-only ratchet, SL-first, scheduled weekend/21:00-UTC closures allowed, 300-bar timeout, $0.30/$0.65 views); when a snapshot no longer overlaps the outcome's last bar it rebuilds the stream from every stored snapshot of that TF before giving up. It is a cleaned-mid evidence model, not execution P&L. **Post-gate truth columns:** `gate_reason` (the warning that removed the setup), `actionable` (grade ≥ C AND setup — the `action` text is written BEFORE the gate chain and can say "STRONG SELL — High-quality setup" on a D row), `cleaned_warnings_json` (what the XAU_CP acceptance stripped), `bar_range_median` (= the misnamed `spread` column: median High-Low, NOT a broker spread), `scan_kind` (`blocked` weekend/day early returns carry no news/candles), `config_fingerprint` (gate config hash — surfaces config drift between processes). **News is content-addressed:** `headlines_json`/`events_json` are interned into `news_snapshots` (they were 73% of a 667 MB DB); read through the `signals_with_news` view or `signal_log.read_news`. Readers must open with `mode=ro`, never `immutable=1` (blind to WAL rows). Tools: `analyze_signal_log.py` (`--stored-candles` DB-only replay with event/opportunity/one-position books + persona scorecard, `--news` news-sentiment lead/lag + extremes + outcome buckets, `--parity` telegram-vs-dashboard divergence report), `scripts/backfill_signal_outcomes.py` (seed/dedup/advance legacy rows), `scripts/migrate_signal_log_news.py` (intern legacy blobs; `--vacuum` with both writers stopped). +- **`news_engine.py`** — Forex Factory calendar (MyFXBook RSS fallback) + RSS keyword sentiment with gold-inverse keywords. Class-level caches (fresh engine per scan). Optional DeepSeek enrichment via `llm_client.py` (requests-only, no SDK): batched headline sentiment overriding keyword scores, gold-direction on release surprises, and a market-summary blurb (`SignalReport.news_summary`) — **display/warning-only, keyword fallback on any failure, inert without a key**; prompts are static module constants (DeepSeek prefix-cache pricing) and requests send `thinking: disabled` (V4-Flash reasoning otherwise blows the timeout). By XAU-tuned default, event risk and sentiment **warn but do not gate signals** (`event_risk_caps_signals=False`, `sentiment_affects_signals=False`) — the LLM does not change that. **Headline pool (2026-09-04):** only headlines ≤ 4 h old with a parsable publish time (`headline_max_age_seconds=14400`, undated items dropped), and `get_fundamental_context` drops headlines whose every relevant score is inside ±0.1 (`_is_neutral_headline`, `NEUTRAL_HEADLINE_THRESHOLD`) before the per-currency mean — so `news_sentiment` from that date is a mean over directional headlines only (bolder values; log comparisons must split at the deploy date). The market summary prompt writes for someone who has never traded (no acronyms), sends headlines newest-first with the platform's own gold direction, and caps at 400 chars. **Verdict 2026-09-04** (`backtest/research/news_sentiment_value_2026-09-04.md`): `news_sentiment` is a trailing-24h echo of price (r≈0.67 with the past day, ≈0.03 with the next; flips 44 h after tops / 16 h after bottoms) — both gating flags stay False; re-check with `analyze_signal_log.py --stored-candles --news`. - **`agent_panel.py`** — ai-hedge-fund-style advisory panel: 4 DeepSeek persona calls (macro/technical/risk/contrarian, concurrent on a panel-private pool) + 1 synthesizer producing an **ADVISORY-ONLY** consensus (`AGREE/NEUTRAL/OPPOSE` + confidence + rationale) written onto `SignalReport.panel_*` fields. Attached **post-hoc** by consumers (`telegram_bot` alert path, `main --panel`, `dashboard_publisher`) — never runs inside `analyze_from_dataframes` (backtest parity, test-locked) and no gate reads its fields. Persona payloads are role-filtered: macro gets high-impact events plus only code-scored directional news; technical gets strategy/pattern, primary indicators, MTF, levels, and ATR-normalized candles; risk gets setup strategy, strategy cost assumption, trailing semantics, volatility/events/warnings (never the median candle range as spread); contrarian gets directional news, setup, timestamped latest-closed price, indicators, reasons, and warnings. Spend is bounded by exact cache, full-evidence-identity TTL reuse, and `max_panels_per_day`; changed evidence never inherits a direction-only cached verdict (test-locked — which also means TTL reuse effectively never fires live because price/candles change every bar; the daily cap is the real spend bound). Verdict provenance logs creation time, `MISS`/`EXACT_HIT`/`TTL_REUSE`, and a SHA-256 payload hash. Prompts forbid outside knowledge and outputs remain short. **OPPOSE guard (2026-09):** the 3-week signal-log scorecard showed the contrarian/risk/technical personas oppose almost every trend entry (chair said OPPOSE 73% of the time; opposed trades made +30R, PF 2.6) while the macro persona was the only informative voice (macro=support → 88% win). `_apply_oppose_guard` therefore downgrades a chair OPPOSE to NEUTRAL unless macro opposes, or technical opposes while macro is not supportive; the synthesizer prompt states the same rule. Re-score with `analyze_signal_log.py --stored-candles` (persona scorecard) before changing it again. **Verdict source (2026-09-03, owner decision):** all four personas still run and are logged (`panel_takes_json`), but `agent_panel.verdict_source` defaults to `"macro"` — the alert verdict is the macro persona's stance (support/neutral/oppose → AGREE/NEUTRAL/OPPOSE) and the synthesizer call is skipped (4 LLM calls per panel instead of 5); `"chair"` restores the four-voice synthesizer + OPPOSE guard. Re-score personas monthly with `analyze_signal_log.py --stored-candles`; weights are not to be tuned before ~300 graded outcomes with a held-out half. - **`market_outlook.py`** — hourly DeepSeek market brief for XAU/USD, using ATR-normalized M15/H1/H4 candles, visible indicator states, levels, news, and recent signal history (read `mode=ro`). It returns a written **ADVISORY-ONLY** bias/setup-forming assessment and persists it independently in `outlooks` within `logs/signals.db` together with the news, indicator/regime/level, and candle evidence sent to the model. It may be sent post-scan by Telegram or requested with `main.py --outlook`; it never imports or runs inside the signal path. Hourly dedup is by UTC hour BUCKET (the old `now − last < 3600` rule dropped every other hour because the H1 loop fires at boundary+10 s+scan time) plus its own UTC-day cap (default 12); every silent early return logs `market outlook: skipped (...)` and an empty LLM reply logs a warning — a dark feature must be visible in `logs/telegram.log`. Provider/news failures leave a safe partial brief while LLM/database failures degrade to no outlook. - **`telegram_bot.py`** — owner-locked bot + subscriber channel; candle-boundary auto-scan loops (M5/M15/H1/H4, boundary+10 s), alert quality threshold + fingerprint dedup (committed only after a successful send; the fingerprint is signal+direction — grade is NOT part of it, a grade UPGRADE re-alerts, an A↔B flicker does not), 📰 LLM market-context line on alerts when available, WebSocket level-breach monitor fed by the streaming provider (`data_providers.streaming`, falling back to the active provider; auto-disabled when the resolved provider can't stream or has no API key — Oanda does not stream, Tiingo does). The breach handler reuses this bar's M15 report (`_cached_or_scan`, keyed by closed-bar epoch) instead of rescanning — it used to log 3.5 M15 rows per candle. Rotating log: `logs/telegram.log`, 3-h rotation × 240 backups. diff --git a/analyze_signal_log.py b/analyze_signal_log.py index e20b928..c5d55f7 100644 --- a/analyze_signal_log.py +++ b/analyze_signal_log.py @@ -38,6 +38,14 @@ Three lenses: the "opportunity" (first event per tf+direction cluster) and one-position-per-timeframe views. +6. NEWS SENTIMENT (--news, with --stored-candles) — is the logged + news_sentiment score a leading or a trailing signal? Hourly mean + sentiment vs past/forward price moves at 1/4/8/24 h (split-half on + the forward number), the daily open->close read, what it said at the + window's high and low, and resolved-event R bucketed by + direction x news bucket x macro-persona stance. Warning-only feature: + nothing here gates a signal. + The DB is opened read-only (mode=ro — WAL-visible; the older immutable=1 URI silently missed the newest rows). Bar data for the default replay comes from the active provider via the backtester's fetch_history @@ -50,6 +58,7 @@ Usage: python analyze_signal_log.py --no-fetch # cache only python analyze_signal_log.py --parity # surface parity only python analyze_signal_log.py --stored-candles # DB-only replay + python analyze_signal_log.py --stored-candles --news # + news lens """ import argparse @@ -104,6 +113,8 @@ class LogRow: advisory_status: str = "" advisory_r: Optional[float] = None scan_kind: str = "" + news_sentiment: Optional[float] = None + event_risk: str = "" @dataclass @@ -132,7 +143,14 @@ _COLS = ("id, logged_at, source, timeframe, signal, score, quality," # Columns added by the 2026-09 observability work; absent on older DBs. _OPTIONAL_COLS = ("gate_reason", "actionable", "config_fingerprint", "advisory_event_id", "advisory_outcome_status", - "advisory_r_multiple", "scan_kind") + "advisory_r_multiple", "scan_kind", "news_sentiment", + "event_risk") + +# The aggregator's own "News:" reason threshold — the band outside which a +# logged sentiment score is called bullish/bearish for gold. +NEWS_BULL = 0.1 +NEWS_BEAR = -0.1 +NEWS_LAGS_H = (1, 4, 8, 24) def open_readonly(db_path: str) -> sqlite3.Connection: @@ -184,6 +202,10 @@ def load_rows(db_path: str) -> List[LogRow]: advisory_status=extra.get("advisory_outcome_status") or "", advisory_r=extra.get("advisory_r_multiple"), scan_kind=extra.get("scan_kind") or "", + news_sentiment=(float(extra["news_sentiment"]) + if extra.get("news_sentiment") is not None + else None), + event_risk=extra.get("event_risk") or "", )) return out finally: @@ -232,6 +254,12 @@ def cluster_setup_rows( return events +def _row_index(events: Sequence[SignalEvent]) -> Dict[int, SignalEvent]: + """row id -> the SignalEvent it belongs to (every row of an event + shares that event's replayed outcome).""" + return {r.id: ev for ev in events for r in ev.rows} + + # ── Trade construction + replay (backtester realism, reused code) ──── @@ -380,10 +408,7 @@ def panel_scorecard( Verdicted rows with a setup join their SignalEvent outcome; rows without a setup are scored on the +4h forward return instead. """ - by_row: Dict[int, SignalEvent] = {} - for ev in events: - for r in ev.rows: - by_row[r.id] = ev + by_row = _row_index(events) out = [] for row in rows: @@ -725,10 +750,7 @@ def persona_scorecard(rows: Sequence[LogRow], events: Sequence[SignalEvent], db_path: str) -> List[dict]: """Per persona × stance: n, mean R and win rate of the underlying event (needs panel_takes_json — read straight from the DB).""" - by_row: Dict[int, SignalEvent] = {} - for ev in events: - for r in ev.rows: - by_row[r.id] = ev + by_row = _row_index(events) con = open_readonly(db_path) agg: Dict[Tuple[str, str], List[float]] = {} try: @@ -753,6 +775,243 @@ def persona_scorecard(rows: Sequence[LogRow], events: Sequence[SignalEvent], for (p, s), v in sorted(agg.items())] +# ── News sentiment lead/lag ─────────────────────────────────────────── + + +def _hourly_sentiment(rows: Sequence[LogRow]) -> pd.Series: + """Mean logged news_sentiment per UTC hour (priced rows only).""" + buckets: Dict[pd.Timestamp, List[float]] = {} + for r in rows: + if not r.price or r.news_sentiment is None: + continue + ts = pd.Timestamp(r.logged_at) + ts = ts.tz_localize("UTC") if ts.tzinfo is None else ts.tz_convert("UTC") + buckets.setdefault(ts.floor("h"), []).append(float(r.news_sentiment)) + if not buckets: + return pd.Series(dtype=float) + return pd.Series({k: sum(v) / len(v) + for k, v in buckets.items()}).sort_index() + + +def _stream_frame(streams: Dict[str, List[dict]], tf: str) -> pd.DataFrame: + """Stored bars for one timeframe as an OHLC frame indexed by UTC ts.""" + bars = streams.get(tf) or [] + if not bars: + return pd.DataFrame() + df = pd.DataFrame(bars) + if "timestamp" not in df.columns or "close" not in df.columns: + return pd.DataFrame() + df.index = pd.to_datetime(df["timestamp"], utc=True) + keep = [c for c in ("open", "high", "low", "close") if c in df.columns] + return df[keep].astype(float).sort_index() + + +def _corr(a: pd.Series, b: pd.Series) -> Optional[float]: + """Pearson correlation of the overlapping, finite pairs (None if NaN).""" + try: + c = float(a.corr(b)) + except (TypeError, ValueError): + return None + return None if math.isnan(c) else c + + +def news_lead_lag(rows: Sequence[LogRow], streams: Dict[str, List[dict]], + tf: str = "1h") -> dict: + """Does the logged sentiment LEAD price, or only follow it? + + For each lag k (hours) the hourly mean sentiment is correlated with + the PAST move (open_t - open_{t-k}) and the FORWARD move + (close_{t+k-1} - open_t) — the scan in hour H sees the bar that + closed at H, so the forward window starts at bar H's open. A high + past-correlation with a ~zero forward-correlation means the score is + a trailing return in disguise. Split-half on the forward number + guards against reading one regime as an edge. + """ + sent = _hourly_sentiment(rows) + df = _stream_frame(streams, tf) + if sent.empty or df.empty: + return {"n_hours": 0} + joined = df.join(pd.DataFrame({"sent": sent}), how="inner").dropna( + subset=["sent"]) + n = len(joined) + if n < 8: + return {"n_hours": n} + s = joined["sent"] + half = n // 2 + first, second = joined.index[:half], joined.index[half:] + corr_past, corr_fwd, corr_split = {}, {}, {} + for k in NEWS_LAGS_H: + past = (df["open"] - df["open"].shift(k)).reindex(joined.index) + fwd = (df["close"].shift(-(k - 1)) - df["open"]).reindex(joined.index) + corr_past[k] = _corr(s, past) + corr_fwd[k] = _corr(s, fwd) + corr_split[k] = (_corr(s.loc[first], fwd.loc[first]), + _corr(s.loc[second], fwd.loc[second])) + day_sent = sent.groupby(sent.index.floor("D")).mean() + day = df.resample("1D").agg({"open": "first", "close": "last"}).dropna() + move = day["close"] - day["open"] + # .corr aligns on the index; shifting over the BAR index keeps + # "previous/next" meaning the previous/next day that actually traded. + daily = {"same": _corr(day_sent, move), + "prev": _corr(day_sent, move.shift(1)), + "next": _corr(day_sent, move.shift(-1))} + return {"n_hours": n, "corr_past": corr_past, "corr_fwd": corr_fwd, + "corr_fwd_split": corr_split, "daily": daily} + + +def news_extremes(rows: Sequence[LogRow], streams: Dict[str, List[dict]], + tf: str = "1h") -> dict: + """What the sentiment said at the window's high and low. + + Plus how long AFTER the top the first bearish hour arrived (and vice + versa at the bottom) — a turn called late is a trailing indicator. + """ + df = _stream_frame(streams, tf) + if df.empty or "high" not in df.columns or "low" not in df.columns: + return {} + sent = _hourly_sentiment(rows) + hi_ts = df["high"].idxmax() + lo_ts = df["low"].idxmin() + hi_px = float(df["high"].loc[hi_ts]) + lo_px = float(df["low"].loc[lo_ts]) + + def _at(ts): + v = sent.get(ts) + return None if v is None or (isinstance(v, float) and math.isnan(v)) \ + else float(v) + + def _first(from_ts, ref_px, sign): + for ts, val in sent.items(): + if ts < from_ts: + continue + if (val <= NEWS_BEAR) if sign < 0 else (val >= NEWS_BULL): + close = df["close"].asof(ts) + if close is None or (isinstance(close, float) + and math.isnan(close)): + continue + return {"lag_h": (ts - from_ts).total_seconds() / 3600.0, + "price_move": float(close) - ref_px} + return None + + return { + "high": {"ts": hi_ts.isoformat(), "price": hi_px, + "sentiment": _at(hi_ts)}, + "low": {"ts": lo_ts.isoformat(), "price": lo_px, + "sentiment": _at(lo_ts)}, + "bear_after_high": _first(hi_ts, hi_px, -1), + "bull_after_low": _first(lo_ts, lo_px, +1), + } + + +def news_outcome_buckets(rows: Sequence[LogRow], + events: Sequence[SignalEvent], + db_path: str) -> List[dict]: + """Resolved events bucketed by (direction, news bucket, macro stance). + + The macro persona is the only voice the 2026-09 scorecard found + informative, so it rides along as a third axis here. + """ + by_row = _row_index(events) + macro: Dict[int, str] = {} + con = open_readonly(db_path) + try: + for rid, takes in con.execute( + "SELECT id, panel_takes_json FROM signals" + " WHERE panel_takes_json IS NOT NULL ORDER BY id"): + ev = by_row.get(rid) + if ev is None or ev.representative.id in macro: + continue + try: + parsed = json.loads(takes) + except (TypeError, ValueError): + continue + stance = "none" + for t in parsed or []: + if isinstance(t, dict) and t.get("persona") == "macro": + stance = t.get("stance") or "none" + break + macro[ev.representative.id] = stance + finally: + con.close() + + agg: Dict[Tuple[str, str, str], List[float]] = {} + for ev in events: + if ev.r_headline is None: + continue + rep = ev.representative + s = rep.news_sentiment + if s is None: + bucket = "neutral" + elif s >= NEWS_BULL: + bucket = "bull" + elif s <= NEWS_BEAR: + bucket = "bear" + else: + bucket = "neutral" + direction = (rep.setup or {}).get("direction") or "?" + agg.setdefault((direction, bucket, macro.get(rep.id, "none")), + []).append(ev.r_headline) + return [{"direction": d, "bucket": b, "macro": m, "n": len(v), + "sum_r": sum(v), "mean_r": sum(v) / len(v), + "win_rate": sum(1 for x in v if x > 0) / len(v)} + for (d, b, m), v in sorted(agg.items())] + + +def _fmt_c(c: Optional[float], nd: int = 3) -> str: + return " n/a " if c is None else f"{c:+.{nd}f}" + + +def print_news(lead_lag: dict, extremes: dict, buckets: Sequence[dict], + tf: str = "1h") -> None: + print("\n── News sentiment (warning-only; hourly mean vs stored " + + tf + " bars) " + "─" * 10) + n = lead_lag.get("n_hours", 0) + if "corr_past" not in lead_lag: + print(f" only {n} joined hour(s) of logged sentiment — " + "need 8+ for a correlation read") + else: + print(f" {n} hours with a logged sentiment joined to a bar") + for k in NEWS_LAGS_H: + a, b = lead_lag["corr_fwd_split"].get(k, (None, None)) + print(f" past {k:2d}h {_fmt_c(lead_lag['corr_past'].get(k))}" + f" fwd {k:2d}h {_fmt_c(lead_lag['corr_fwd'].get(k))}" + f" (halves {_fmt_c(a, 2)} / {_fmt_c(b, 2)})") + d = lead_lag.get("daily") or {} + print(f" daily open->close: same day {_fmt_c(d.get('same'))}" + f" previous {_fmt_c(d.get('prev'))}" + f" next {_fmt_c(d.get('next'))}") + + for side, key, tag in (("high", "bear_after_high", "bearish"), + ("low", "bull_after_low", "bullish")): + info = (extremes or {}).get(side) + if not info: + continue + sv = info.get("sentiment") + head = (f" {side:4} {info['ts'][:16]}Z ${info['price']:.1f} " + f"sentiment {_fmt_c(sv, 2)}") + nxt = (extremes or {}).get(key) + if nxt: + print(f"{head} → first {tag} hour +{nxt['lag_h']:.0f} h later, " + f"price {nxt['price_move']:+.1f}") + else: + print(f"{head} → no {tag} hour logged after it") + + if buckets: + print(f" outcome buckets ({sum(b['n'] for b in buckets)} resolved " + "events):") + print(f" {'dir':5} {'news':7} {'macro':8} {'n':>3} {'total':>8} " + f"{'mean':>8} {'win':>5}") + for b in buckets: + print(f" {b['direction']:5} {b['bucket']:7} {b['macro']:8} " + f"{b['n']:3d} {_fmt_r(b['sum_r']):>8} " + f"{_fmt_r(b['mean_r']):>8} " + f"{100 * b['win_rate']:4.0f}%") + else: + print(" no resolved events to bucket yet") + print(" Caveat: sentiment ≈ trailing 24h return; buckets inherit the " + "trend confound.") + + def print_report( rows: List[LogRow], events: List[SignalEvent], @@ -892,8 +1151,15 @@ def main(argv: Optional[Sequence[str]] = None) -> int: ap.add_argument("--stored-candles", action="store_true", help="replay on the candle streams stored in the DB " "(no network) instead of provider history") + ap.add_argument("--news", action="store_true", + help="news-sentiment lead/lag + outcome buckets " + "(requires --stored-candles)") args = ap.parse_args(argv) + if args.news and not args.stored_candles: + print("--news requires --stored-candles") + return 2 + rows = load_rows(args.db) if not rows: print(f"No rows in {args.db}") @@ -923,6 +1189,10 @@ def main(argv: Optional[Sequence[str]] = None) -> int: for p in persona_scorecard(rows, events, args.db): print(f" {p['persona']:11} {p['stance']:8} n={p['n']:3d} " f"mean {p['mean_r']:+.3f}R win {100 * p['win_rate']:.0f}%") + if args.news: + print_news(news_lead_lag(rows, streams), + news_extremes(rows, streams), + news_outcome_buckets(rows, events, args.db)) print_parity(parity_report(rows)) if args.csv: export_csv(events, args.csv) diff --git a/backtest/research/news_sentiment_value_2026-09-04.md b/backtest/research/news_sentiment_value_2026-09-04.md new file mode 100644 index 0000000..9f06dff --- /dev/null +++ b/backtest/research/news_sentiment_value_2026-09-04.md @@ -0,0 +1,129 @@ +# News sentiment vs. price — does the news score carry information? — 2026-09-04 + +**Source evidence:** `logs/signals.db` 2026-08-13 → 2026-09-03 (14,683 rows, 13,947 with +a price; candle snapshots from 08-18, full-shape headline blobs from 08-19), read with +`mode=ro`. A continuous H1/M15/M5 candle stream was rebuilt from the stored 14-bar +`candles_json` snapshots (H1: 293 bars, 08-18 05:00 → 09-03 22:00). Headlines were +resolved through `news_snapshots` (4,054 distinct headlines, 2,024 gold-tagged). +Outcomes are the repo's own advisory model (`advisory_r_multiple` on the 144 `RESOLVED` +rows, R at $0.65). One-year context from the Oanda H1 parquet +(`backtest/cache/oanda_XAU_USD_1h_20250814_20260814.parquet`). No backtest: the +backtester runs with news off, and no headline history exists beyond these 3 weeks, so a +replay cannot test news. The same check reruns with +`python analyze_signal_log.py --stored-candles --news`. + +Question asked by the owner: the log reads bullish most of the time — what did the +indicators and the news say when gold fell, and is news scoring a waste? + +## Findings + +**The numeric news score is a rear-view mirror.** `news_sentiment` (= mean gold-headline +score − mean dollar-headline score over the last 12 h) tracks what gold did over the +*previous* day and says nothing stable about the next one. + +| hourly mean sentiment vs. gold return (n≈293 h) | Pearson | +|---|---| +| past 24 h | **+0.67** | +| past 8 h / 4 h / 1 h | +0.32 / +0.20 / +0.07 | +| next 1 h / 4 h / 8 h / 24 h | +0.05 / +0.10 / +0.21 / +0.14 | +| next 8 h, first half (08-18→25) / second half (08-26→09-03) | **−0.37 / +0.12** | +| next 24 h, first half / second half | −0.12 / −0.09 | +| daily mean vs same-day / previous-day / next-day move | +0.39 / +0.37 / **+0.03** | +| *change* in sentiment vs next 4 h / 8 h / 24 h | −0.09 / −0.03 / −0.01 | + +Partial correlation with the next 8 h after removing the past-24 h return is +0.26 pooled +but flips sign between halves. Alternative aggregates (gold-specific score only, dropping +price-report headlines, only headlines < 2 h old) show the same shape: past-24 h +correlation 0.59–0.82, no stable forward correlation. A naive "hold in the direction of +sentiment" rule made +$272 over 279 hours vs +$120 buy-and-hold, but trading the *change* +in sentiment lost −$200: the gain is trend persistence (sentiment ≈ trailing return), not +news. + +**Why the log reads bullish.** 62% of rows have sentiment ≥ +0.1, 21% ≤ −0.1. Component +means: gold headlines +0.158, dollar headlines +0.048. The window was a 16% August rally, +so the average headline was bullish because price had been rising. Of the gold-tagged +headlines, 42% merely restate the chart (price-up reports 489, mean +0.37; price-down 366, +mean −0.18) and 18% are long-horizon forecast pieces ("$5,400 target", "$50,000/oz"; 60% +positive, 21% of all positive score mass). The per-headline LLM labels are correct — every +regex-flagged "mis-score" was a correctly scored "yields fall → gold up" headline — which +matches the 78/78 golden set of 2026-08-06. The averaging, the 12 h window and the +headline universe are what destroy the information. + +**What news and indicators said on the way down** (top $4,697 on 08-25 00:00 UTC, low +$4,283 on 09-02 03:00; an 8.8% drop in a week): + +| moment | price | news sentiment | engine (M15) | +|---|---|---|---| +| top, 08-25 00:00 | 4,697 | +0.30, rising to **+0.44** two hours after the top (window high) | ranging, composite +5.9; 9 BUY setups that day: −7.4R, 11% win | +| 08-26/27 | −$60 | still +0.15…+0.37; first bearish hour 08-26 20:00, **44 h after the top**, price already −$102 | trending_down 134/138 rows, composite −4.6/−7.8; SELL setups 0/3 on 08-27 | +| 08-28 Warsh speech 13:00–16:00 | −$122 in 3 h, −$140 day | +0.24…+0.30 overnight, **0.00 at 13:00**, −0.17 only at 16:00; summary all morning: "a hawkish tone could pressure gold, dovish hints may lift it" | composite −6.2; SELL setups 4/4, +3.3R | +| 09-01 | −$128 day | −0.04…−0.20, "gold under pressure" all day (correct, describing the move) | trending_down 275 rows, composite −12.2, 51 SELL rows; SELL setups 7/7, +3.5R | +| low, 09-02 03:00 | 4,283 | −0.16 at the low, −0.19 six hours later; first bullish hour 09-02 19:00, **16 h after the low**, price already +$108 | SELL setups at the bottom 0/2, −2.05R | + +Sentiment peaked at the top and troughed at the bottom. The engine also lagged (it is +trend-following) but flipped regime and composite within hours; its SELL setups won on the +trend days and lost at the turns, the same as its BUYs. + +**Outcomes** (144 RESOLVED, +37.8R): BUY 121 / +43.2R / 70% win; SELL 23 / −5.4R / 48%. + +| direction × sentiment bucket (≥+0.15 bull, ≤−0.05 bear) | n | R | mean | win | +|---|---|---|---|---| +| BUY, bullish news | 81 | +32.0 | +0.40 | 73% | +| BUY, neutral | 34 | +10.6 | +0.31 | 68% | +| BUY, bearish news | 6 | +0.65 | +0.11 | 50% | +| SELL, bearish news | 14 | +0.13 | +0.01 | 71% | +| SELL, neutral | 5 | −1.46 | −0.29 | 20% | +| SELL, bullish news | 4 | −4.08 | −1.02 | 0% | + +The splits are the trend confound again ("BUY with bullish news" = "BUY in an uptrend"). +Cleanest counter-example: 08-25, the most bullish news day of the drop period (+0.25), +9 BUYs at the top lost −7.4R. corr(sentiment, R): BUY +0.10, SELL −0.45 (n=23). + +**Macro persona** (reads the same headlines plus high-impact events): macro=support 51 +BUYs → +32.6R, 90% win; macro=neutral 16 → −3.2R, 38%. Within bullish-sentiment BUYs: +support 42 → +25.2R (88%) vs neutral 7 → −2.9R (29%). On 08-25 all 7 panel rows were +macro=neutral ("awaiting PCE, mixed") while the numeric aggregate said +0.25 bullish. +Confounded with regime (64 of 84 supports were trending_up) and n≈16 on the informative +side. Not enough to promote or demote; the standing rule (re-score at ~300 outcomes) +applies. + +**Event risk** cannot be evaluated here: only 4 event windows in the sample +(08-19 17:00, 08-26 12:00, 08-28 13:00, 09-01 13:00), none produced a setup. The standing +evidence remains the 2026-07-11 audit (event gates would cost +4.6…+7.2R/yr). + +**One-year context:** daily moves ≤ −$100 happened 17 times in the year to 2026-08-14; +15 of 47 rolling 3-week windows contained a ≥ 8% drawdown. This sample is one ordinary +rally-then-drop regime. + +**Simulated 4 h window + neutral drop** (the change shipped with this note, on 11,136 +full-shape scans): never leaves a scan empty (median 19 directional headlines, 8 +gold-specific; worst hour 02:00 UTC has < 3 gold headlines 28% of the time), the aggregate +correlates 0.78 with the old one and becomes bolder (share |score| ≥ 0.3 rises 17% → 41%), +with the same lack of forward information (next-24 h corr +0.20; next-8 h halves +−0.22 / −0.10). + +## Method + +- Hourly join convention: a scan in hour H (boundary + 10 s) sees the bar closed at H, + i.e. bar H's open; forward returns run from bar H's open, past returns end there. +- Correlations are Pearson on hourly means (no scipy); halves split at 2026-08-26 00:00. +- Headline classes by regex on the title (price-up / price-down / forecast-style); the + gold-specific score is `score_by_currency.XAU` where present, else `score`. +- Outcome buckets use the representative row's `news_sentiment`; macro stance from + `panel_takes_json`. +- Everything above reproduces from the DB alone; nothing depends on live feeds. + +## Decisions + +- `sentiment_affects_signals` and `event_risk_caps_signals` stay `False`. No gate is + built in either direction (follow or fade): 3 weeks, one regime. +- The headline labelling is not the problem; the pool and the averaging are. Shipped the + same day: `headline_max_age_seconds` 12 h → 4 h with undated items dropped, and + `get_fundamental_context` drops headlines whose every relevant score is inside ±0.1 + (`_is_neutral_headline`). The market-summary prompt now writes for someone who has + never traded, sends headlines newest-first with the platform's own gold direction, and + caps at 400 chars. **`news_sentiment` rows from the deploy date onward are a mean over + directional ≤ 4 h headlines — log comparisons must split at that date.** +- Keep the macro persona running and logging; re-check with `--news` and the persona + scorecard at ~300 outcomes. +- Refuted hypothesis, do not re-propose without new data: "news sentiment leads price". diff --git a/config.py b/config.py index 335026b..635aa8f 100755 --- a/config.py +++ b/config.py @@ -172,8 +172,12 @@ class NewsConfig: # into the danger window while a stale cache says it's clear. context_cache_ttl_seconds: int = 600 # Maximum age of RSS headlines accepted for scoring. Older headlines - # are discarded before keyword analysis and LLM enrichment. - headline_max_age_seconds: int = 43200 # 12 hours + # (and headlines with no parsable publish time) are discarded before + # keyword analysis and LLM enrichment. 12 h → 4 h on 2026-09-04: the + # 3-week signal log showed the 12 h mean was a trailing echo of the + # previous day's price move (see backtest/research/ + # news_sentiment_value_2026-09-04.md). Display/warning-only either way. + headline_max_age_seconds: int = 14400 # 4 hours rss_feeds: List[str] = field(default_factory=lambda: [ "https://investinglive.com/feed/", diff --git a/deploy/README-dashboard.md b/deploy/README-dashboard.md index 9068238..ed2a09d 100644 --- a/deploy/README-dashboard.md +++ b/deploy/README-dashboard.md @@ -100,6 +100,11 @@ the public page, accepted with these guards baked into the PHP — the id must match a headline in the published news.json (arbitrary text can never reach the LLM), each headline's summary is cached forever in `ai/` (≤1 API call per headline, ever), and a file-based daily cap fails **closed** on I/O errors. +Since 2026-09-04 the engine publishes only headlines up to 4 h old whose +score leans one way (neutral items are dropped), so a headline leaves the +whitelist sooner: a dialog left open past that point returns `unknown_id` +("This headline has left the live feed"); summaries already cached in `ai/` +stay reachable. Requirements: `php` + the curl extension (`php -m | grep curl`). diff --git a/docs/beginner_guide.html b/docs/beginner_guide.html index 400e32b..b113031 100644 --- a/docs/beginner_guide.html +++ b/docs/beginner_guide.html @@ -86,6 +86,7 @@
  • Grade (A+, A, B, C): the bot's own mark for how clean the setup is. A+ is best. It never sends anything worse than C.
  • Confidence and the indicator list: extra detail for curious people. You don't need to understand it to trade the signal.
  • ⚠️ Warning lines: things the bot noticed, like high volatility. They are "be aware," not "don't trade."
    • +
  • 📰 News line: a short plain-English note on what is moving the gold price right now and why, written from the last few hours of headlines. It is background information, not a reason to trade.
  • The trade setup at the bottom is the only part you must act on.
    • diff --git a/docs/user_guide.html b/docs/user_guide.html index f0e2119..34dc232 100644 --- a/docs/user_guide.html +++ b/docs/user_guide.html @@ -69,7 +69,7 @@ Score / confidenceThe regime-weighted composite of all nine indicators. Bigger absolute score = stronger consensus. MTF alignedThe two confirmation timeframes agree with the trade direction. Signals without alignment are filtered out on M5. Warnings ⚠️Context, not veto. Backtesting shows warned trades perform about the same — strong momentum often triggers warnings. -Market context 📰An AI-written one-liner summarizing today's macro backdrop (headlines + high-impact releases) and its likely pull on gold. Informational only — it never blocks or grades a signal. +Market context 📰An AI-written note in plain English on what is moving the gold price right now, why it matters, and the next report to watch. It is built only from headlines published in the last 4 hours that lean one way (neutral items are dropped) plus today's high-impact releases. Informational only — it never blocks or grades a signal. SL / TP1 / RRStop loss sits behind the nearest swing level with an ATR buffer. TP1 is the nearest opposing level. 1R = the entry-to-SL distance = the amount you risk. Trail plan 🔄The real exit. Once price moves +0.6R in your favor, move the stop to trail 0.75R behind the best price and update it at each candle close. TP1 is advisory — let winners run. @@ -130,7 +130,7 @@ /scanRun an immediate analysis of the primary timeframe. /autoscanToggle automatic scanning (M5 / M15 / H1 / H4 at each candle close). /statusUptime, auto-scan state, WebSocket price-monitor state. -/newsNet XAU/USD news sentiment score (informational only — never gates signals). +/newsNet XAU/USD news score from the last 4 hours of headlines (neutral headlines are removed before averaging; informational only — never gates signals). /cache · /clearcacheInspect / reset the market-data cache. /helpCommand reference. diff --git a/news_engine.py b/news_engine.py index e550e53..079e539 100755 --- a/news_engine.py +++ b/news_engine.py @@ -11,6 +11,7 @@ A perfect technical setup can be destroyed by an unexpected NFP print. import hashlib import html import logging +import math import re import threading import time @@ -79,13 +80,25 @@ Reply with ONLY JSON: """ _LLM_SUMMARY_SYSTEM = """\ -You are a forex/gold market-context writer. You receive JSON with recent -"headlines" and today's high-impact "events" (released: true means the -number is out). Write a 1-2 sentence plain-English summary of the -current macro backdrop for a GOLD (XAU/USD) trader: the dominant theme -and its likely pull on gold. No advice, no hedging boilerplate, max 45 -words. +You write a short news note about the gold price for someone who has +never traded. You receive JSON with recent "headlines" (newest first; +each carries "gold": "up" or "down", the platform's own reading of how +that story pushes the gold price — trust it) and today's high-impact +"events" (released: true means the number is out). +Say three things, in this order: what is happening to gold right now, +why it matters for the gold price, and the next scheduled report to +watch if one is listed. Use only the headlines and events given; if +they are thin, say so plainly. +Style: plain everyday English, short simple sentences, no acronyms, no +abbreviations, no trader words. Write "the US central bank", not "the +Fed"; "interest rates", not "yields" or "hawkish"; "a report on US +prices", not "CPI" or "PCE"; "gold", never "XAU". No advice, no hedging +boilerplate. 2-3 sentences, max 55 words. Reply with ONLY JSON: {"summary": "..."} + +Examples: +{"summary": "Gold is falling today. The US central bank hinted it may raise interest rates, which makes the dollar stronger, and a stronger dollar usually pushes gold down. The next thing to watch is Thursday's report on US prices."} +{"summary": "Gold is climbing. Fresh fighting in the Middle East has people moving money into gold as a safe place, and a weaker dollar is helping too. No major US reports are due today."} """ @@ -215,6 +228,48 @@ class FundamentalContext: llm_summary: str = "" # DeepSeek market-context blurb (display-only) +# Headlines whose every relevant score is inside ±NEUTRAL_HEADLINE_THRESHOLD +# lean neither way: they carry no direction for display and only dilute the +# per-currency mean. Same threshold agent_panel uses for its macro payload. +NEUTRAL_HEADLINE_THRESHOLD = 0.1 + + +def _is_neutral_headline(h) -> bool: + """True when no per-currency score on the headline leans either way. + + A headline whose overall score is 0.0 but whose gold-specific score is + -0.7 is NOT neutral. Missing, empty or non-finite scores count as + neutral (they cannot be shown as leaning one way).""" + by_currency = getattr(h, "sentiment_by_currency", None) or {} + overall = getattr(h, "sentiment_score", None) + values = [] + for currency in (getattr(h, "currencies_mentioned", None) or []): + value = by_currency.get(currency, overall) + if isinstance(value, (int, float)) and math.isfinite(value): + values.append(abs(float(value))) + return not values or max(values) < NEUTRAL_HEADLINE_THRESHOLD + + +def _gold_direction_label(h) -> str: + """"up" / "down" / "flat" for the market-summary payload — the same + XAU-first, USD-inverted reading agent_panel uses for its macro persona.""" + by_currency = getattr(h, "sentiment_by_currency", None) or {} + xau = by_currency.get("XAU") + usd = by_currency.get("USD") + if isinstance(xau, (int, float)) and math.isfinite(xau): + score = float(xau) + elif isinstance(usd, (int, float)) and math.isfinite(usd): + score = -float(usd) + else: + raw = getattr(h, "sentiment_score", None) + score = float(raw) if isinstance(raw, (int, float)) and math.isfinite(raw) else 0.0 + if score > 0: + return "up" + if score < 0: + return "down" + return "flat" + + class NewsEngine: """ Economic calendar and news sentiment analyzer. @@ -375,6 +430,17 @@ class NewsEngine: # 2. News sentiment try: headlines = self._fetch_news_sentiment() + # Drop headlines that lean neither way (2026-09-04). Applied + # here rather than in the fetch so the class-level cache and + # the feed-health counts still describe what the feeds sent; + # every consumer (aggregate score, summary, dashboard, panel, + # outlook, signal log) reads context.news_headlines. + kept = [h for h in headlines if not _is_neutral_headline(h)] + if len(kept) != len(headlines): + logger.debug( + f"News: dropped {len(headlines) - len(kept)} neutral " + f"headlines, {len(kept)} kept") + headlines = kept context.news_headlines = headlines # Aggregate sentiment per currency (per-currency score where @@ -745,6 +811,7 @@ class NewsEngine: headlines = [] seen_titles = set() ok_feeds = 0 + undated = 0 for feed_url in self.config.rss_feeds: host = feed_url.split("/")[2] if "//" in feed_url else feed_url @@ -770,8 +837,14 @@ class NewsEngine: pub_date_el.text if pub_date_el is not None else None ) - # Only recent headlines (configurable: headline_max_age_seconds) - if pub_date and (datetime.now(timezone.utc) - pub_date).total_seconds() > self.config.headline_max_age_seconds: + # Only recent headlines (configurable: headline_max_age_seconds). + # An item without a parsable pubDate cannot be proven + # fresh, so it is dropped too — it used to be stamped + # "now" and never aged out (2026-09-04). + if pub_date is None: + undated += 1 + continue + if (datetime.now(timezone.utc) - pub_date).total_seconds() > self.config.headline_max_age_seconds: continue # Dedup within and across feeds so a syndicated story @@ -827,6 +900,8 @@ class NewsEngine: f"rss:{host}", False, f"{type(e).__name__}: {e}") logger.debug(f"Failed to fetch RSS from {feed_url}: {e}") + if undated: + logger.debug(f"RSS: dropped {undated} headlines without a publish time") self._record_source( "headlines", ok_feeds > 0, detail="all feeds failed" if ok_feeds == 0 else "", @@ -986,9 +1061,11 @@ class NewsEngine: return text or None def _llm_market_summary(self, context: FundamentalContext) -> str: - """1-2 sentence macro blurb for alerts; "" when unavailable. - Cached (success OR failure) for the context-cache TTL so at most - one call is made per window.""" + """2-3 sentence plain-language news note for alerts; "" when + unavailable. Headlines go newest-first with the platform's own + gold direction so the note agrees with the score badges. Cached + (success OR failure) for the context-cache TTL so at most one + call is made per window.""" with NewsEngine._cache_lock: ts, cached = NewsEngine._llm_summary_cache if time.time() - ts < self.config.context_cache_ttl_seconds: @@ -999,8 +1076,17 @@ class NewsEngine: client = get_llm_client() if not client.available: return "" + oldest = datetime.min.replace(tzinfo=timezone.utc) + recent = sorted( + context.news_headlines, + key=lambda h: getattr(h, "timestamp", None) or oldest, + reverse=True, + )[:15] result = client.chat_json(_LLM_SUMMARY_SYSTEM, { - "headlines": [h.headline for h in context.news_headlines[:15]], + "headlines": [ + {"headline": h.headline, "gold": _gold_direction_label(h)} + for h in recent + ], "events": [ {"event": e.event, "currency": e.currency, "actual": e.actual, "forecast": e.forecast, @@ -1010,7 +1096,7 @@ class NewsEngine: }) summary = "" if result: - summary = str(result.get("summary", "")).strip()[:300] + summary = str(result.get("summary", "")).strip()[:400] with NewsEngine._cache_lock: NewsEngine._llm_summary_cache = (time.time(), summary) return summary diff --git a/static/dashboard.js b/static/dashboard.js index d911954..1ba16c5 100644 --- a/static/dashboard.js +++ b/static/dashboard.js @@ -589,10 +589,10 @@ function renderWireTotal() { ${signed(mean, 2)} ${dir} ${scored.length} headlines · sum ${signed(sum, 1)}
    - ${up} bullish · ${dn} bearish · ${scored.length - up - dn} neutral     + ${up} bullish · ${dn} bearish `; - box.title = `Mean sentiment of all ${scored.length} scored headlines (−1…+1). Sum ${signed(sum, 1)}. Bullish/bearish counted at |score| > 0.05.`; + box.title = `Mean sentiment of all ${scored.length} scored headlines (−1…+1). Sum ${signed(sum, 1)}. Only headlines from the last 4 hours that lean one way are published (neutral ones are removed before scoring).`; } function renderWire() { @@ -601,7 +601,7 @@ function renderWire() { $("wire-note").textContent = news.generated_at ? `updated ${fmtAge(news.generated_at)}` : ""; renderWireTotal(); if (!items.length) { - $("wire").innerHTML = `
    No recent gold/dollar headlines.
    `; + $("wire").innerHTML = `
    No gold or dollar headlines in the last 4 hours.
    `; return; } const item = h => { diff --git a/static/news_summary.php b/static/news_summary.php index 3cb9355..76c8648 100644 --- a/static/news_summary.php +++ b/static/news_summary.php @@ -51,7 +51,10 @@ In 1-2 plain-English sentences (max 45 words) explain what this news means for the GOLD (XAU/USD) price and WHY it is bullish, bearish, or neutral. If your reading clearly contradicts xau_score, add "(the automated score disagrees)". Never write the words "xau_score" or -"score" — a general reader sees this text. +"score" — a general reader sees this text. Write for someone who has +never traded: short simple sentences, no acronyms or abbreviations +(say "the US central bank", not "the Fed"; "interest rates", not +"yields"). Gold doctrine: hawkish Fed / rate hikes / rising real yields / strong dollar are gold-bearish; easing / rate cuts / safe-haven flows / geopolitical stress / weak dollar are gold-bullish. diff --git a/test_all.py b/test_all.py index c12fafe..62faf13 100644 --- a/test_all.py +++ b/test_all.py @@ -4465,6 +4465,120 @@ class _StubLLM: return self.reply +def _m19_pubdate(hours_ago=0.5): + """RFC-2822 pubDate bytes `hours_ago` in the past (RSS fixtures must be + dated since 2026-09-04: undated items are dropped).""" + from email.utils import format_datetime + when = (_dt_m19.datetime.now(_dt_m19.timezone.utc) + - _dt_m19.timedelta(hours=hours_ago)) + return format_datetime(when).encode() + + +@test("NewsEngine: RSS age window is 4 h; older and undated items are dropped") +def _(): + assert NewsConfig().headline_max_age_seconds == 14400, "4 h (2026-09-04)" + cfg = NewsConfig() + cfg.rss_feeds = ["https://feed.example/rss"] + ne = NewsEngine(cfg) + saved = NewsEngine._source_health + NewsEngine._source_health = {} + rss = (b"" + b"Gold surges as Fed signals rate cuts" + b"" + _m19_pubdate(3) + b"" + b"Gold slips on strong dollar data" + b"" + _m19_pubdate(5) + b"" + b"Dollar steady before Fed minutes" + b"") + + class _Sess: + def get(self, url, timeout=None): + return type("R", (), {"status_code": 200, "content": rss})() + + ne._session = _Sess() + try: + heads = ne._fetch_news_sentiment_raw() + assert [h.headline for h in heads] == \ + ["Gold surges as Fed signals rate cuts"], heads + finally: + NewsEngine._source_health = saved + + +@test("NewsEngine: neutral headlines are dropped before the per-currency mean") +def _(): + from news_engine import _is_neutral_headline, NEUTRAL_HEADLINE_THRESHOLD + now = _dt_m19.datetime.now(_dt_m19.timezone.utc) + + def head(overall, by, cur): + return NewsSentiment(headline="h", source="s", timestamp=now, + currencies_mentioned=cur, sentiment_score=overall, + keywords=[], sentiment_by_currency=by) + + assert NEUTRAL_HEADLINE_THRESHOLD == 0.1 + assert _is_neutral_headline(head(0.0, {"XAU": 0.0}, ["XAU"])) + assert not _is_neutral_headline(head(0.0, {"XAU": -0.7}, ["XAU"])), \ + "overall 0.0 but gold -0.7 leans one way" + assert _is_neutral_headline(head(float("nan"), {"XAU": float("nan")}, ["XAU"])) + assert _is_neutral_headline(head(0.5, {}, [])), "no currencies → neutral" + assert _is_neutral_headline(head(0.05, {"USD": 0.05, "XAU": 0.09}, + ["USD", "XAU"])) + + ne = NewsEngine(NewsConfig()) + ne._fetch_economic_calendar = lambda currencies: [] + ne._fetch_news_sentiment = lambda: [ + head(0.0, {"XAU": 0.0}, ["XAU"]), # neutral → dropped + head(0.0, {"XAU": -0.7}, ["XAU"]), # gold-bearish → kept + head(0.3, {"USD": 0.3}, ["USD"]), # dollar-bullish → kept + head(float("nan"), {"USD": float("nan")}, ["USD"]), # unscored → dropped + ] + NewsEngine._llm_summary_cache = (0.0, "") + try: + lc._client = _StubLLM(None) + ctx = ne.get_fundamental_context() + finally: + lc._client = None + NewsEngine._llm_summary_cache = (0.0, "") + assert len(ctx.news_headlines) == 2, [h.sentiment_by_currency for h in ctx.news_headlines] + assert abs(ctx.news_sentiment["XAU"] - (-0.7)) < 1e-9, \ + "mean over survivors only (the 0.0 must not dilute it)" + assert abs(ctx.news_sentiment["USD"] - 0.3) < 1e-9 + + +@test("NewsEngine: summary prompt is beginner-language; payload newest-first with gold direction; 400-char cap") +def _(): + from news_engine import _LLM_SUMMARY_SYSTEM, _gold_direction_label + assert "never traded" in _LLM_SUMMARY_SYSTEM + assert "no acronyms" in _LLM_SUMMARY_SYSTEM + assert "max 55 words" in _LLM_SUMMARY_SYSTEM + assert "the US central bank" in _LLM_SUMMARY_SYSTEM # worked example carries the style + now = _dt_m19.datetime.now(_dt_m19.timezone.utc) + + def head(title, ts, by, cur, overall=0.0): + return NewsSentiment(headline=title, source="s", timestamp=ts, + currencies_mentioned=cur, sentiment_score=overall, + keywords=[], sentiment_by_currency=by) + + older = head("old gold story", now - _dt_m19.timedelta(hours=3), + {"XAU": 0.6}, ["XAU"]) + newer = head("dollar jumps", now, {"USD": 0.5}, ["USD"], overall=0.5) + assert _gold_direction_label(older) == "up" + assert _gold_direction_label(newer) == "down", "dollar-bullish → gold down" + ctx = FundamentalContext(news_headlines=[older, newer]) + NewsEngine._llm_summary_cache = (0.0, "") + try: + stub = _StubLLM({"summary": "x" * 450}) + lc._client = stub + out = NewsEngine(NewsConfig())._llm_market_summary(ctx) + assert len(out) == 400, len(out) + sent = stub.last_payload["headlines"] + assert [h["headline"] for h in sent] == ["dollar jumps", "old gold story"], \ + "newest first" + assert [h["gold"] for h in sent] == ["down", "up"] + assert set(sent[0]) == {"headline", "gold"} + finally: + NewsEngine._llm_summary_cache = (0.0, "") + lc._client = None + + @test("NewsEngine: _llm_headline_scores validates, clamps, rejects mismatch") def _(): ne = NewsEngine(NewsConfig()) @@ -4737,6 +4851,7 @@ def _(): NewsEngine._source_health = {} rss = (b"" b"Fed signals rate cuts as dollar slides" + b"" + _m19_pubdate() + b"" b"") class _Sess: @@ -4771,6 +4886,7 @@ def _(): # carrying the real publisher. rss = (b"" b"Gold surges as Fed signals rate cuts - Reuters" + b"" + _m19_pubdate() + b"" b"https://news.google.com/rss/articles/CBMiAbC?oc=5" b"<a href=\"https://x\">Gold &amp; dollar " b"diverge as the Fed turns dovish.</a>" @@ -4802,15 +4918,16 @@ def _(): saved = NewsEngine._source_health NewsEngine._source_health = {} long_link = b"https://a.example/" + b"x" * 2100 + stamp = b"" + _m19_pubdate() + b"" rss = (b"" # bare item: no link/description/source at all - b"Dollar steady before Fed minutes" + b"Dollar steady before Fed minutes" + stamp + b"" # javascript: link + description that merely echoes the title - b"Gold slips on strong dollar data" + b"Gold slips on strong dollar data" + stamp + b"javascript:alert(1)" b"Gold slips on strong dollar data" # over-long link - b"Fed officials split on rate cuts" + b"Fed officials split on rate cuts" + stamp + b"" + long_link + b"" b"") @@ -7738,7 +7855,7 @@ import analyze_signal_log as asl def _asl_row(id_, logged_at, tf="15min", source="telegram", signal="BUY", quality="B", entry=3400.0, sl=3392.0, strategy="GENERIC", bar_ts=None, action="BUY — Decent setup", fp="cfg1", - setup=True, price=None): + setup=True, price=None, news_sentiment=None): setup_d = None if setup: setup_d = {"direction": "BUY" if "BUY" in signal else "SELL", @@ -7753,7 +7870,7 @@ def _asl_row(id_, logged_at, tf="15min", source="telegram", signal="BUY", price=price if price is not None else entry, market_regime="trending_up", setup=setup_d, panel_verdict="", panel_confidence=None, action=action, bar_ts=bar_ts, - config_fingerprint=fp) + config_fingerprint=fp, news_sentiment=news_sentiment) @test("analyze_signal_log: entry_bar_open floors the log time to the TF boundary") @@ -7930,6 +8047,148 @@ def _(): assert by_sig["BUY"] > 0 and by_sig["SELL"] < 0 +@test("analyze_signal_log: news_lead_lag separates trailing sentiment from a lead") +def _(): + rng = np.random.default_rng(20260904) + n = 200 + start = pd.Timestamp("2026-08-19 00:00", tz="UTC") # Wednesday + opens = 3400.0 + np.cumsum(rng.normal(0, 2.0, n + 1)) + bars, rows = [], [] + for i in range(n): + o, c = float(opens[i]), float(opens[i + 1]) + ts = start + pd.Timedelta(hours=i) + bars.append({"timestamp": ts.isoformat(), "open": o, + "high": max(o, c) + 1.0, "low": min(o, c) - 1.0, + "close": c, "volume": 100}) + # The logged score IS a trailing 24h return, by construction. + sent = 0.0 if i < 24 else float(np.sign(opens[i] - opens[i - 24])) + rows.append(_asl_row(i + 1, ts.to_pydatetime() + timedelta(seconds=10), + tf="1h", setup=False, price=o, + news_sentiment=sent)) + ll = asl.news_lead_lag(rows, {"1h": bars}) + assert ll["n_hours"] == n + assert set(ll["corr_past"]) == set(asl.NEWS_LAGS_H) + assert set(ll["corr_fwd"]) == set(asl.NEWS_LAGS_H) + assert set(ll["corr_fwd_split"]) == set(asl.NEWS_LAGS_H) + assert ll["corr_past"][24] > 0.5, ll["corr_past"] + assert abs(ll["corr_fwd"][24]) < 0.3, ll["corr_fwd"] + assert set(ll["daily"]) == {"same", "prev", "next"} + + +@test("analyze_signal_log: news_lead_lag needs 8 joined hours") +def _(): + start = pd.Timestamp("2026-08-19 00:00", tz="UTC") + bars = [{"timestamp": (start + pd.Timedelta(hours=i)).isoformat(), + "open": 3400.0, "high": 3401.0, "low": 3399.0, + "close": 3400.0, "volume": 1} for i in range(4)] + rows = [_asl_row(i + 1, (start + pd.Timedelta(hours=i)).to_pydatetime(), + tf="1h", setup=False, price=3400.0, news_sentiment=0.2) + for i in range(4)] + assert asl.news_lead_lag(rows, {"1h": bars}) == {"n_hours": 4} + assert asl.news_lead_lag(rows, {}) == {"n_hours": 0} + + +@test("analyze_signal_log: news_extremes times the first turn call after the high/low") +def _(): + n = 100 + start = pd.Timestamp("2026-08-19 00:00", tz="UTC") + bars, rows = [], [] + for i in range(n): + ts = start + pd.Timedelta(hours=i) + bars.append({"timestamp": ts.isoformat(), "open": 3400.0, + "high": 3450.0 if i == 30 else 3405.0, + "low": 3350.0 if i == 80 else 3395.0, + "close": 3400.0, "volume": 100}) + sent = 0.3 if (i < 36 or i >= 86) else -0.3 + rows.append(_asl_row(i + 1, ts.to_pydatetime() + timedelta(seconds=10), + tf="1h", setup=False, price=3400.0, + news_sentiment=sent)) + ex = asl.news_extremes(rows, {"1h": bars}) + assert ex["high"]["price"] == 3450.0 and ex["low"]["price"] == 3350.0 + assert ex["high"]["ts"].startswith("2026-08-20T06:00") # hour 30 + assert ex["low"]["ts"].startswith("2026-08-22T08:00") # hour 80 + assert abs(ex["high"]["sentiment"] - 0.3) < 1e-9 + assert abs(ex["low"]["sentiment"] + 0.3) < 1e-9 + assert ex["bear_after_high"]["lag_h"] == 6.0 + assert ex["bull_after_low"]["lag_h"] == 6.0 + assert ex["bear_after_high"]["price_move"] == -50.0 + assert ex["bull_after_low"]["price_move"] == 50.0 + assert asl.news_extremes(rows, {}) == {} + + +@test("analyze_signal_log: news_outcome_buckets splits R by news bucket x macro stance") +def _(): + import sqlite3 + t0 = datetime(2026, 8, 19, 12, 0, 10, tzinfo=timezone.utc) + rows = [ + _asl_row(1, t0, signal="BUY", entry=3400.0, sl=3392.0, + news_sentiment=0.3), + _asl_row(2, t0 + timedelta(hours=3), signal="BUY", entry=3410.0, + sl=3402.0, news_sentiment=-0.3), + _asl_row(3, t0 + timedelta(hours=6), signal="SELL", entry=3420.0, + sl=3428.0, news_sentiment=0.0), + ] + events = asl.cluster_setup_rows(rows) + assert len(events) == 3 + for ev, r in zip(events, (2.0, -1.0, 0.5)): + ev.r_headline = r + fd, path = tempfile.mkstemp(suffix=".db") + os.close(fd) + try: + con = sqlite3.connect(path) + con.execute("CREATE TABLE signals(id INTEGER PRIMARY KEY," + " panel_takes_json TEXT)") + con.execute("INSERT INTO signals VALUES (?, ?)", + (1, json.dumps([{"persona": "technical", + "stance": "oppose"}, + {"persona": "macro", + "stance": "support"}]))) + con.commit() + con.close() + buckets = asl.news_outcome_buckets(rows, events, path) + finally: + os.remove(path) + assert sum(b["n"] for b in buckets) == 3 + by_key = {(b["direction"], b["bucket"], b["macro"]): b for b in buckets} + assert set(by_key) == {("BUY", "bull", "support"), + ("BUY", "bear", "none"), + ("SELL", "neutral", "none")} + assert by_key[("BUY", "bull", "support")]["sum_r"] == 2.0 + assert by_key[("BUY", "bull", "support")]["win_rate"] == 1.0 + assert by_key[("BUY", "bear", "none")]["sum_r"] == -1.0 + assert by_key[("SELL", "neutral", "none")]["sum_r"] == 0.5 + # print_news must survive every shape it can be handed + import contextlib + import io as _io + buf = _io.StringIO() + with contextlib.redirect_stdout(buf): + asl.print_news(asl.news_lead_lag(rows, {}), {}, buckets) + asl.print_news({"n_hours": 0}, {}, []) + out = buf.getvalue() + assert "need 8+" in out and "BUY bull support" in out + assert "no resolved events to bucket yet" in out + + +@test("analyze_signal_log: load_rows round-trips news_sentiment and event_risk") +def _(): + import signal_log as slog + fd, path = tempfile.mkstemp(suffix=".db") + os.close(fd) + try: + lg = slog.SignalLogger(path) + lg.log(_make_log_report(), source="telegram", timeframe="5min") + lg.close() + rows = asl.load_rows(path) + assert len(rows) == 1 + assert abs(rows[0].news_sentiment - (-0.4)) < 1e-9, rows[0].news_sentiment + assert rows[0].event_risk == "caution", rows[0].event_risk + finally: + slog.reset_loggers_for_tests() + for suffix in ("", "-wal", "-shm"): + if os.path.exists(path + suffix): + os.remove(path + suffix) + + # ══════════════════════════════════════════════════════════════════════ # MODULE 26: market_outlook.py (advisory hourly market brief) # ══════════════════════════════════════════════════════════════════════