DeepSeek V4-Flash news enrichment: LLM sentiment, surprise direction, summary

Three display/warning-only upgrades to the news engine, all behind the
existing keyword path as fallback — signals are never gated on LLM
output (event_risk_caps_signals / sentiment_affects_signals stay False
per the 2026-07 audit; backtester use_news=False untouched):

- Headline sentiment: one batched call per 600s cache window (capped at
  25 new headlines, sha1-cached so repeats are never re-sent) overrides
  the keyword scores per currency with the gold-inverse doctrine encoded
  in the prompt.
- Release surprises: get_event_risk_level's description now says which
  way the print pulls GOLD ("— gold bearish: strong jobs lift yields"),
  fixing the documented numeric-only gap; risk level and windows
  unchanged.
- Market summary: 1-2 sentence macro blurb on SignalReport.news_summary,
  rendered in Telegram alerts (📰) and the CLI (🧠).

New llm_client.py: requests-only OpenAI-compatible client (no SDK), one
attempt per call, UTC-day request cap, singleton so the cap survives
per-scan NewsEngine reconstruction. Config: deepseek block in
config.json + DEEPSEEK_API_KEY env; absent key = fully inert.

Live-measured API behavior baked in: V4-Flash is a hybrid thinking model
— requests send thinking:disabled (reasoning otherwise blew the timeout
and could exhaust max_tokens before the answer); prompts are static
module constants so DeepSeek's prefix cache prices repeat calls at the
cache-hit rate; scores use compact [s, xau] pairs; timeout 12s.

Tests 273 → 286 (new Module 19 + deepseek config tests).

Claude-Session: https://claude.ai/code/session_01Nsd8HcrV67GbCZXxZEn1Ma

news_engine.py

@@ -8,6 +8,7 @@ Key principle: High-impact news events override technical signals.
 A perfect technical setup can be destroyed by an unexpected NFP print.
 """
 
+import hashlib
 import logging
 import re
 import threading
@@ -21,10 +22,62 @@ from typing import Dict, List, Optional, Tuple
 import requests
 
 from config import NewsConfig
+from llm_client import get_llm_client
 
 logger = logging.getLogger(__name__)
 
 
+# ── DeepSeek prompts ─────────────────────────────────────────────────
+# STATIC constants on purpose: DeepSeek prices a repeated byte-identical
+# prompt prefix at the cache-hit rate (~50x cheaper), so nothing here may
+# be interpolated or timestamped. All variable content travels in the
+# user payload. LLM output is display/warning-only — it never gates
+# signals (see NewsConfig.sentiment_affects_signals).
+
+_LLM_SENTIMENT_SYSTEM = """\
+You are a forex/gold news analyst. You receive JSON:
+{"headlines": ["...", ...]}
+For EACH headline, in the same order, score:
+- "s": overall market sentiment of the headline, -1.0 (very bearish)
+  to 1.0 (very bullish), 0.0 if irrelevant or neutral.
+- "xau": the effect on the GOLD (XAU/USD) price, -1.0 (gold down)
+  to 1.0 (gold up), 0.0 if irrelevant.
+Gold doctrine: hawkish Fed / rate hikes / rising real yields / strong
+dollar are gold-bearish; easing / rate cuts / safe-haven flows /
+geopolitical stress / weak dollar are gold-bullish.
+Handle negation correctly ("no rate hike" is not hawkish).
+Reply with ONLY JSON: {"scores": [[s, xau], ...]} — exactly one
+[s, xau] pair per input headline, in the same order, nothing else.
+
+Examples:
+"Fed signals faster rate hikes as inflation stays hot" -> [0.4, -0.7]
+"Gold surges to record high on safe-haven demand" -> [0.8, 0.9]
+"Dollar slips after soft jobs report; Treasury yields fall" -> [-0.4, 0.6]
+"""
+
+_LLM_EVENT_SYSTEM = """\
+You are a forex/gold analyst. You receive one released economic event as
+JSON: {"event", "currency", "actual", "forecast", "previous"}.
+Decide the likely effect of this RELEASE (actual vs forecast) on the
+GOLD (XAU/USD) price. A stronger-than-forecast USD economy or a hawkish
+implication is usually gold-bearish; weaker data or a dovish implication
+is usually gold-bullish. Mind inverted metrics: higher unemployment or
+jobless claims = weaker economy.
+Reply with ONLY JSON:
+{"gold_direction": "bullish"|"bearish"|"neutral", "reason": "<max 10 words>"}
+"""
+
+_LLM_SUMMARY_SYSTEM = """\
+You are a forex/gold market-context writer. You receive JSON with recent
+"headlines" and today's high-impact "events" (released: true means the
+number is out). Write a 1-2 sentence plain-English summary of the
+current macro backdrop for a GOLD (XAU/USD) trader: the dominant theme
+and its likely pull on gold. No advice, no hedging boilerplate, max 45
+words.
+Reply with ONLY JSON: {"summary": "..."}
+"""
+
+
 _RELEASE_VALUE_RE = re.compile(r"(-?\d+(?:\.\d+)?)\s*([kmbt])?", re.IGNORECASE)
 _RELEASE_MULTIPLIERS = {"k": 1e3, "m": 1e6, "b": 1e9, "t": 1e12}
 
@@ -105,6 +158,7 @@ class FundamentalContext:
     news_headlines: List[NewsSentiment] = field(default_factory=list)
     event_impact_on_pairs: Dict[str, str] = field(default_factory=dict)
     calendar_available: bool = False  # True if live calendar data was fetched
+    llm_summary: str = ""  # DeepSeek market-context blurb (display-only)
 
 
 class NewsEngine:
@@ -185,6 +239,17 @@ class NewsEngine:
     _headlines_cache: Tuple[float, List[NewsSentiment]] = (0.0, [])
     _EMPTY_RESULT_TTL = 60.0  # retry sooner after a failed/empty fetch
 
+    # DeepSeek result caches. Headlines and releases are immutable, so a
+    # scored one never needs re-scoring; bounded by dropping the oldest
+    # insertion. _llm_sent_cache is only touched under _cache_lock (its
+    # writer runs inside _cached_fetch); the other two take the lock
+    # around cache access but keep HTTP calls outside it.
+    _llm_sent_cache: Dict[str, Tuple[float, float]] = {}   # sha1 → (s, xau)
+    _llm_event_cache: Dict[Tuple[str, str, str], str] = {}
+    _llm_summary_cache: Tuple[float, str] = (0.0, "")
+    _LLM_SENT_CACHE_MAX = 400
+    _LLM_EVENT_CACHE_MAX = 100
+
     def __init__(self, config: NewsConfig):
         self.config = config
         self._session = requests.Session()
@@ -264,6 +329,12 @@ class NewsEngine:
         except Exception as e:
             logger.warning(f"News sentiment fetch failed: {e}")
 
+        # 3. LLM market-context summary (display-only)
+        try:
+            context.llm_summary = self._llm_market_summary(context)
+        except Exception as e:
+            logger.debug(f"LLM market summary failed: {e}")
+
         return context
 
     @staticmethod
@@ -311,10 +382,17 @@ class NewsEngine:
                 )
                 if event.minutes_until > -window:
                     if surprised:
-                        return "caution", (
+                        desc = (
                             f"{event.currency} {event.event} released — {surprise} surprise "
                             f"(actual {event.actual} vs forecast {event.forecast})"
                         )
+                        # LLM enrichment touches the DESCRIPTION only —
+                        # `surprise` is numeric-direction and does not know
+                        # whether a beat is gold-bullish; the LLM does.
+                        gold_dir = self._llm_event_direction(event)
+                        if gold_dir:
+                            desc += f" — gold {gold_dir}"
+                        return "caution", desc
                     return "caution", f"{event.currency} {event.event} just released"
 
         return "clear", "No significant events"
@@ -596,6 +674,11 @@ class NewsEngine:
             except Exception as e:
                 logger.debug(f"Failed to fetch RSS from {feed_url}: {e}")
 
+        try:
+            self._apply_llm_sentiment(headlines)
+        except Exception as e:
+            logger.debug(f"LLM sentiment failed, keyword scores stand: {e}")
+
         return headlines
 
     def _analyze_headline_sentiment(
@@ -637,6 +720,130 @@ class NewsEngine:
 
         return (bullish_score - bearish_score) / total
 
+    # ── DeepSeek enrichment (display/warning-only, keyword fallback) ──
+
+    def _llm_headline_scores(
+        self, texts: List[str]
+    ) -> Optional[List[Tuple[float, float]]]:
+        """One batched DeepSeek call → [(overall, xau), ...] aligned with
+        `texts`, or None on any failure/mismatch (caller keeps keyword
+        scores). Tests stub this method directly."""
+        result = get_llm_client().chat_json(
+            _LLM_SENTIMENT_SYSTEM, {"headlines": texts})
+        if not result:
+            return None
+        scores = result.get("scores")
+        if not isinstance(scores, list) or len(scores) != len(texts):
+            return None
+        out = []
+        for item in scores:
+            try:
+                s = max(-1.0, min(1.0, float(item[0])))
+                x = max(-1.0, min(1.0, float(item[1])))
+            except (TypeError, KeyError, ValueError, IndexError):
+                return None
+            out.append((s, x))
+        return out
+
+    def _apply_llm_sentiment(self, headlines: List[NewsSentiment]) -> None:
+        """Overwrite keyword sentiment on `headlines` with LLM scores.
+
+        Runs inside _cached_fetch, i.e. UNDER _cache_lock — cache access
+        is direct and the (rare) HTTP call blocks sibling scans at most
+        once per cache window, matching the existing shared-fetch design.
+        Only headlines missing from the cache are sent; a failed batch
+        changes nothing.
+        """
+        if not headlines or not get_llm_client().available:
+            return
+        cache = NewsEngine._llm_sent_cache
+        keyed = [(hashlib.sha1(h.headline.encode()).hexdigest(), h)
+                 for h in headlines]
+        # Cap the batch so the reply fits the client's max_tokens — the
+        # overflow stays keyword-scored this cycle and, still uncached,
+        # goes out with the next window's batch.
+        new = [(k, h) for k, h in keyed if k not in cache][:25]
+        if new:
+            scores = self._llm_headline_scores([h.headline for _, h in new])
+            if scores is not None:
+                for (k, _), sc in zip(new, scores):
+                    cache[k] = sc
+                while len(cache) > self._LLM_SENT_CACHE_MAX:
+                    cache.pop(next(iter(cache)))
+        for k, h in keyed:
+            sc = cache.get(k)
+            if sc is None:
+                continue
+            overall, xau = sc
+            h.sentiment_score = overall
+            h.sentiment_by_currency = {
+                cur: (xau if cur == "XAU" else overall)
+                for cur in h.currencies_mentioned
+            }
+
+    def _llm_event_direction(self, event: EconomicEvent) -> Optional[str]:
+        """Gold direction for a released surprise, e.g. "bearish: strong
+        jobs lift yields". None → caller keeps the numeric description.
+        Cached per release; failures are NOT cached (retry next scan)."""
+        key = (event.event, event.currency, str(event.actual or ""))
+        with NewsEngine._cache_lock:
+            if key in NewsEngine._llm_event_cache:
+                return NewsEngine._llm_event_cache[key] or None
+        client = get_llm_client()
+        if not client.available:
+            return None
+        result = client.chat_json(_LLM_EVENT_SYSTEM, {
+            "event": event.event,
+            "currency": event.currency,
+            "actual": event.actual,
+            "forecast": event.forecast,
+            "previous": event.previous,
+        })
+        if not result:
+            return None
+        direction = result.get("gold_direction")
+        if direction not in ("bullish", "bearish", "neutral"):
+            return None
+        reason = str(result.get("reason", "")).strip()[:80]
+        text = "" if direction == "neutral" else (
+            f"{direction}: {reason}" if reason else direction)
+        with NewsEngine._cache_lock:
+            NewsEngine._llm_event_cache[key] = text
+            while len(NewsEngine._llm_event_cache) > self._LLM_EVENT_CACHE_MAX:
+                NewsEngine._llm_event_cache.pop(
+                    next(iter(NewsEngine._llm_event_cache)))
+        return text or None
+
+    def _llm_market_summary(self, context: FundamentalContext) -> str:
+        """1-2 sentence macro blurb for alerts; "" when unavailable.
+        Cached (success OR failure) for the context-cache TTL so at most
+        one call is made per window."""
+        with NewsEngine._cache_lock:
+            ts, cached = NewsEngine._llm_summary_cache
+            if time.time() - ts < self.config.context_cache_ttl_seconds:
+                return cached
+        high_impact = [e for e in context.events_today if e.is_high_impact]
+        if not context.news_headlines and not high_impact:
+            return ""
+        client = get_llm_client()
+        if not client.available:
+            return ""
+        result = client.chat_json(_LLM_SUMMARY_SYSTEM, {
+            "headlines": [h.headline for h in context.news_headlines[:15]],
+            "events": [
+                {"event": e.event, "currency": e.currency,
+                 "actual": e.actual, "forecast": e.forecast,
+                 "released": not e.is_upcoming}
+                for e in high_impact[:10]
+            ],
+        })
+        summary = ""
+        if result:
+            summary = str(result.get("summary", "")).strip()[:300]
+        with NewsEngine._cache_lock:
+            NewsEngine._llm_summary_cache = (time.time(), summary)
+        return summary
+
     def _extract_currencies(self, headline: str) -> List[str]:
         """Extract mentioned currencies from a headline."""
         headline_lower = headline.lower()