From 8521ab909ade7694dde3c5635116b2d131ab1f20 Mon Sep 17 00:00:00 2001 From: Harwinder Sekhon Date: Mon, 3 Aug 2026 13:25:02 -0400 Subject: [PATCH] DeepSeek V4-Flash news enrichment: LLM sentiment, surprise direction, summary MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Three display/warning-only upgrades to the news engine, all behind the existing keyword path as fallback — signals are never gated on LLM output (event_risk_caps_signals / sentiment_affects_signals stay False per the 2026-07 audit; backtester use_news=False untouched): - Headline sentiment: one batched call per 600s cache window (capped at 25 new headlines, sha1-cached so repeats are never re-sent) overrides the keyword scores per currency with the gold-inverse doctrine encoded in the prompt. - Release surprises: get_event_risk_level's description now says which way the print pulls GOLD ("— gold bearish: strong jobs lift yields"), fixing the documented numeric-only gap; risk level and windows unchanged. - Market summary: 1-2 sentence macro blurb on SignalReport.news_summary, rendered in Telegram alerts (šŸ“°) and the CLI (🧠). New llm_client.py: requests-only OpenAI-compatible client (no SDK), one attempt per call, UTC-day request cap, singleton so the cap survives per-scan NewsEngine reconstruction. Config: deepseek block in config.json + DEEPSEEK_API_KEY env; absent key = fully inert. Live-measured API behavior baked in: V4-Flash is a hybrid thinking model — requests send thinking:disabled (reasoning otherwise blew the timeout and could exhaust max_tokens before the answer); prompts are static module constants so DeepSeek's prefix cache prices repeat calls at the cache-hit rate; scores use compact [s, xau] pairs; timeout 12s. Tests 273 → 286 (new Module 19 + deepseek config tests). Co-Authored-By: Claude Fable 5 Claude-Session: https://claude.ai/code/session_01Nsd8HcrV67GbCZXxZEn1Ma --- CLAUDE.md | 8 +- app_config.py | 38 +++++- config.json.example | 7 + docs/user_guide.html | 1 + llm_client.py | 185 ++++++++++++++++++++++++++ main.py | 3 + news_engine.py | 209 ++++++++++++++++++++++++++++- signal_aggregator.py | 3 + telegram_bot.py | 2 + test_all.py | 310 ++++++++++++++++++++++++++++++++++++++++++- 10 files changed, 759 insertions(+), 7 deletions(-) create mode 100644 llm_client.py diff --git a/CLAUDE.md b/CLAUDE.md index 10f1a01..15f95d1 100644 --- a/CLAUDE.md +++ b/CLAUDE.md @@ -11,7 +11,7 @@ Every numeric threshold in `config.py` and `pair_signals/xauusd.py` was backtest ## Commands ```bash -python test_all.py # QA suite: 273 tests, 18 modules, no network (mocked) +python test_all.py # QA suite: 286 tests, 19 modules, no network (mocked) python main.py --pair XAUUSD --timeframe H1 --detail # CLI signal report (--json for full dump) python telegram_bot.py # the bot (refuses to boot without credentials) python oanda_daemon.py # auto-execution daemon on Oanda practice account @@ -30,7 +30,7 @@ Python 3.12; a `venv/` exists in-repo. Dependencies in `requirements.txt` (numpy ## Configuration & credentials -Precedence for every credential/deployment setting: **env var → config.json → refuse to run** (no hardcoded fallbacks). A gitignored repo-root `.env` is loaded at `app_config` import via `os.environ.setdefault`. `config.json` (gitignored; schema in `config.json.example`) holds secrets plus deployment settings: `data_providers.active` (the live-feed switch), `data_providers.streaming` (optional tick-stream provider for the level-breach monitor), `scanning.pairs`/`timeframes`, per-pair `blocked_hours_utc`, `log_level`. Key env vars: `TELEGRAM_TOKEN`, `TELEGRAM_CHAT_ID`, `TELEGRAM_CHANNEL_ID`, `TWELVE_DATA_API_KEY`, `TIINGO_API_KEY`, `OANDA_ACCOUNT_ID`/`OANDA_API_TOKEN`/`OANDA_BASE_URL`. +Precedence for every credential/deployment setting: **env var → config.json → refuse to run** (no hardcoded fallbacks). A gitignored repo-root `.env` is loaded at `app_config` import via `os.environ.setdefault`. `config.json` (gitignored; schema in `config.json.example`) holds secrets plus deployment settings: `data_providers.active` (the live-feed switch), `data_providers.streaming` (optional tick-stream provider for the level-breach monitor), `scanning.pairs`/`timeframes`, per-pair `blocked_hours_utc`, `log_level`. Key env vars: `TELEGRAM_TOKEN`, `TELEGRAM_CHAT_ID`, `TELEGRAM_CHANNEL_ID`, `TWELVE_DATA_API_KEY`, `TIINGO_API_KEY`, `OANDA_ACCOUNT_ID`/`OANDA_API_TOKEN`/`OANDA_BASE_URL`, `DEEPSEEK_API_KEY` (news LLM; the `deepseek` config.json block holds model/enabled/timeout/daily cap). ## Architecture @@ -53,8 +53,8 @@ Signal currency everywhere is `SignalStrength` (āˆ’2..+2). DataFrames are UTC tz - **`engine.py`** — `TechnicalEngine`: 9 scored indicators (ema_trend, rsi, macd, bollinger, adx, roc, keltner, divergence, candlestick) + ATR as context. Regime classification (trending_up/down, `trending` = mixed/direction-UNKNOWN, transitioning, ranging) drives per-indicator weight multipliers and regime-aware RSI reinterpretation. Compute order matters (ATR first; MACD always computed even when not a pillar — divergence needs the histogram). Private series ride on the result as `_underscore` attrs; scorers/formatters skip anything `_`-prefixed or weight-0. - **`signal_aggregator.py`** — MTF agreement, diminishing-decay penalty system, exhaustion hard block, quality grading, then an ordered chain of post-grade gates (session, XAU SELL quality/vol/hour, regime blocks, M5 noise gates), then three setup generators: XAU ATR (A+/A), generic swing-based, plus the M5 pullback (`XAU_PB`) and chart-pattern (`XAU_CP`, `chart_patterns.py`) entry paths. `TradeSetup.strategy` selects trailing params; trail params are stamped on the setup from the *producing* config so per-TF overrides reach formatters. - **`pair_signals/`** — per-pair config factories. **Always use the factory (`produce_signal` / `make_config`), never bare `PlatformConfig()`** — the factory applies the config.json overlay first, then the validated per-TF deltas (M5 hour-9 block, M15 trail arm 0.70, M15 SELL-hour block, etc.; evidence in the docstring). A test enforces factory ≔ inline-config equivalence. Confirmation map: M5→[M15,H1], M15→[M5,H1], H1→[M15,H4], H4→[H1,D1]. -- **`news_engine.py`** — Forex Factory calendar (MyFXBook RSS fallback) + RSS keyword sentiment with gold-inverse keywords. Class-level caches (fresh engine per scan). By XAU-tuned default, event risk and sentiment **warn but do not gate signals** (`event_risk_caps_signals=False`, `sentiment_affects_signals=False`). -- **`telegram_bot.py`** — owner-locked bot + subscriber channel; candle-boundary auto-scan loops (M5/M15/H1/H4, boundary+10 s), alert quality threshold + fingerprint dedup (committed only after a successful send), WebSocket level-breach monitor fed by the streaming provider (`data_providers.streaming`, falling back to the active provider; auto-disabled when the resolved provider can't stream or has no API key — Oanda does not stream, Tiingo does). Rotating log: `logs/telegram.log`, 3-h rotation Ɨ 240 backups. +- **`news_engine.py`** — Forex Factory calendar (MyFXBook RSS fallback) + RSS keyword sentiment with gold-inverse keywords. Class-level caches (fresh engine per scan). Optional DeepSeek enrichment via `llm_client.py` (requests-only, no SDK): batched headline sentiment overriding keyword scores, gold-direction on release surprises, and a market-summary blurb (`SignalReport.news_summary`) — **display/warning-only, keyword fallback on any failure, inert without a key**; prompts are static module constants (DeepSeek prefix-cache pricing) and requests send `thinking: disabled` (V4-Flash reasoning otherwise blows the timeout). By XAU-tuned default, event risk and sentiment **warn but do not gate signals** (`event_risk_caps_signals=False`, `sentiment_affects_signals=False`) — the LLM does not change that. +- **`telegram_bot.py`** — owner-locked bot + subscriber channel; candle-boundary auto-scan loops (M5/M15/H1/H4, boundary+10 s), alert quality threshold + fingerprint dedup (committed only after a successful send), šŸ“° LLM market-context line on alerts when available, WebSocket level-breach monitor fed by the streaming provider (`data_providers.streaming`, falling back to the active provider; auto-disabled when the resolved provider can't stream or has no API key — Oanda does not stream, Tiingo does). Rotating log: `logs/telegram.log`, 3-h rotation Ɨ 240 backups. - **`oanda_daemon.py` / `oanda_executor.py`** — practice-account auto-execution. Book-parity rules (see daemon docstring): no broker-side TP (trailing model — TP1/TP2 advisory), 5-bar per-TF cooldown, trail ratchets at bar close exactly once per bar. Accepted divergence: one position total. - **`backtest/gpu_backtest.py`** — bar-by-bar replay through the real `analyze_from_dataframes` (news off). Realism model (do not regress): next-bar-open entries re-anchored to the raw price path, round-trip `--spread` charged per trade, trade resolution on RAW bars (cleaning applies only to signal-side data), same-bar SL+TP tie → SL first, gap fills at bar open, trail ratchet at bar close only. Indicators are precomputed once per TF (CuPy GPU or NumPy fallback — check the `Backend:` log line); replay fans out over worker processes and the 5-bar cooldown is applied in an ordered pass afterwards, so parallel and sequential books are identical. Parquet cache in `backtest/cache/` keyed by provider/symbol/TF/window. `backtest/*.csv` are gitignored pinned trade books from past campaigns — used for manual before/after comparison. `backtest/instructions_for_backtest.txt` lists what every backtest report must record. diff --git a/app_config.py b/app_config.py index 72e639b..2709487 100644 --- a/app_config.py +++ b/app_config.py @@ -85,11 +85,15 @@ _VALID_LOG_LEVELS = ("DEBUG", "INFO", "WARNING", "ERROR", "CRITICAL") # AUTOSCAN_CONFIRMATIONS entries in telegram_bot.py). _SCANNABLE_TFS = ("M5", "M15", "H1", "H4") -_KNOWN_TOP = {"telegram", "data_providers", "scanning", "pairs", "log_level", "oanda"} +_KNOWN_TOP = {"telegram", "data_providers", "scanning", "pairs", "log_level", + "oanda", "deepseek"} _KNOWN_OANDA = { "account_id", "api_token", "base_url", "lot_size", "quality_gate", "max_hold_bars", } +_KNOWN_DEEPSEEK = { + "api_key", "model", "enabled", "timeout_seconds", "daily_request_cap", +} _KNOWN_TELEGRAM = {"token", "chat_id", "channel_id", "alert_min_quality"} _KNOWN_SCANNING = {"pairs", "timeframes"} _KNOWN_PAIR_KEYS = {"blocked_hours_utc"} @@ -113,6 +117,7 @@ class AppConfig: Timeframe.M5, Timeframe.M15, Timeframe.H1, Timeframe.H4]) pair_overrides: Dict[str, Dict[str, Any]] = field(default_factory=dict) oanda_settings: Dict[str, Any] = field(default_factory=dict) + deepseek_settings: Dict[str, Any] = field(default_factory=dict) log_level: str = "INFO" @@ -248,6 +253,29 @@ def load_app_config(path: str = CONFIG_PATH) -> AppConfig: "config.json: oanda.max_hold_bars must be a positive integer") app.oanda_settings = dict(oanda) + deepseek = raw.get("deepseek", {}) + _check_keys(deepseek, _KNOWN_DEEPSEEK, '"deepseek"') + if "enabled" in deepseek and not isinstance(deepseek["enabled"], bool): + raise AppConfigError( + "config.json: deepseek.enabled must be true or false") + if "timeout_seconds" in deepseek: + ts = deepseek["timeout_seconds"] + if isinstance(ts, bool) or not isinstance(ts, (int, float)) or ts <= 0: + raise AppConfigError( + "config.json: deepseek.timeout_seconds must be a positive " + "number") + if "daily_request_cap" in deepseek: + cap = deepseek["daily_request_cap"] + if isinstance(cap, bool) or not isinstance(cap, int) or cap < 1: + raise AppConfigError( + "config.json: deepseek.daily_request_cap must be a positive " + "integer") + if "model" in deepseek: + if not isinstance(deepseek["model"], str) or not deepseek["model"]: + raise AppConfigError( + "config.json: deepseek.model must be a non-empty string") + app.deepseek_settings = dict(deepseek) + level = raw.get("log_level", app.log_level) if level not in _VALID_LOG_LEVELS: raise AppConfigError( @@ -339,6 +367,14 @@ def oanda_base_url(app: Optional[AppConfig] = None) -> str: app.oanda_settings.get("base_url", "https://api-fxpractice.oanda.com")) +# ── DeepSeek resolution helper (env → config.json → "") ────────────── + +def deepseek_api_key(app: Optional[AppConfig] = None) -> str: + app = app or get_app_config() + return os.environ.get("DEEPSEEK_API_KEY") or str( + app.deepseek_settings.get("api_key", "")) + + # ── PlatformConfig overlay ──────────────────────────────────────────── def apply_pair_overrides( diff --git a/config.json.example b/config.json.example index 785607a..0e4eba6 100644 --- a/config.json.example +++ b/config.json.example @@ -33,5 +33,12 @@ "quality_gate": "C", "max_hold_bars": 300 }, + "deepseek": { + "api_key": "REPLACE_WITH_YOUR_DEEPSEEK_API_KEY_OR_EMPTY", + "model": "deepseek-v4-flash", + "enabled": true, + "timeout_seconds": 12, + "daily_request_cap": 500 + }, "log_level": "INFO" } diff --git a/docs/user_guide.html b/docs/user_guide.html index 5fd8793..2db2087 100644 --- a/docs/user_guide.html +++ b/docs/user_guide.html @@ -69,6 +69,7 @@ Score / confidenceThe regime-weighted composite of all nine indicators. Bigger absolute score = stronger consensus. MTF alignedThe two confirmation timeframes agree with the trade direction. Signals without alignment are filtered out on M5. Warnings āš ļøContext, not veto. Backtesting shows warned trades perform about the same — strong momentum often triggers warnings. +Market context šŸ“°An AI-written one-liner summarizing today's macro backdrop (headlines + high-impact releases) and its likely pull on gold. Informational only — it never blocks or grades a signal. SL / TP1 / RRStop loss sits behind the nearest swing level with an ATR buffer. TP1 is the nearest opposing level. 1R = the entry-to-SL distance = the amount you risk. Trail plan šŸ”„The real exit. Once price moves +0.6R in your favor (+0.8R for pullback setups), move the stop to trail 0.75R behind the best price and update it at each candle close. TP1 is advisory — let winners run. diff --git a/llm_client.py b/llm_client.py new file mode 100644 index 0000000..4588120 --- /dev/null +++ b/llm_client.py @@ -0,0 +1,185 @@ +""" +ForexBot2 — DeepSeek LLM client +================================ +Thin requests-based client for DeepSeek's OpenAI-compatible chat API, +used by `news_engine` for display/warning-only enrichment (headline +sentiment, event surprise direction, market summary). Signals are NEVER +gated on LLM output — every caller has a non-LLM fallback and treats a +`None` return as "use the fallback". + +Design constraints: +- One attempt per call, no retries: news runs inside scan cycles; the + news engine's 600 s context cache is the natural retry cadence. +- A UTC-day request cap bounds spend even if a bug causes call storms. +- Cost: DeepSeek caches identical prompt PREFIXES server-side (cache-hit + input is ~50x cheaper), so callers pass a STATIC system prompt (no + timestamps or interpolation) and put all variable content in the user + payload. + +This is a module-level singleton (`get_llm_client`) because the daily +cap counter must survive the per-scan reconstruction of `NewsEngine`. +""" + +import datetime as _dt +import json +import logging +import threading +from typing import Any, Dict, Optional + +import requests + +from app_config import deepseek_api_key, get_app_config + +logger = logging.getLogger(__name__) + +DEEPSEEK_URL = "https://api.deepseek.com/chat/completions" +DEFAULT_MODEL = "deepseek-v4-flash" +# 8s timed out on a measured cold-prefix-cache headline batch (2026-08-03); +# warm-cache calls come back well under that. 12s buys cold-start headroom +# while still bounding worst-case scan latency (≤3 calls per cache window). +DEFAULT_TIMEOUT = 12.0 +DEFAULT_DAILY_CAP = 500 + + +class DeepSeekClient: + """JSON-in/JSON-out chat wrapper. All failures return None.""" + + def __init__( + self, + api_key: str, + model: str = DEFAULT_MODEL, + timeout: float = DEFAULT_TIMEOUT, + daily_cap: int = DEFAULT_DAILY_CAP, + enabled: bool = True, + ): + self._api_key = api_key or "" + self._model = model + self._timeout = timeout + self._daily_cap = daily_cap + self._enabled = enabled + self._session = requests.Session() + self._session.headers.update({ + "Content-Type": "application/json", + "Authorization": f"Bearer {self._api_key}", + }) + self._lock = threading.Lock() + self._count_day: Optional[_dt.date] = None + self._count = 0 + self._warned: set = set() + + @property + def available(self) -> bool: + return self._enabled and bool(self._api_key) + + def _warn_once(self, kind: str, msg: str, *args: Any) -> None: + if kind in self._warned: + logger.debug(msg, *args) + else: + self._warned.add(kind) + logger.warning(msg, *args) + + def _under_cap(self) -> bool: + """Count a request against the UTC-day cap; False = don't call.""" + today = _dt.datetime.now(_dt.timezone.utc).date() + with self._lock: + if self._count_day != today: + self._count_day = today + self._count = 0 + self._warned.discard("cap") + if self._count >= self._daily_cap: + self._warn_once( + "cap", "DeepSeek daily request cap (%d) exhausted — " + "falling back to keyword analysis until UTC midnight", + self._daily_cap) + return False + self._count += 1 + return True + + def chat_json(self, system_prompt: str, + user_payload: Dict[str, Any]) -> Optional[Dict[str, Any]]: + """One chat call; returns the model's JSON object or None.""" + if not self.available or not self._under_cap(): + return None + body = { + "model": self._model, + "messages": [ + {"role": "system", "content": system_prompt}, + {"role": "user", + "content": json.dumps(user_payload, sort_keys=True)}, + ], + "response_format": {"type": "json_object"}, + "temperature": 0, + # V4-Flash is a hybrid thinking model; without this flag it + # burns hundreds of reasoning tokens per call — measured + # 2026-08-03: a 25-headline batch blew straight through the + # timeout AND could exhaust max_tokens before the answer, + # returning empty content. The doctrine + few-shot examples + # in the system prompts carry the quality instead. + "thinking": {"type": "disabled"}, + # Must fit the largest reply: a full sentiment batch is + # ~10 tokens per headline; 1000 truncated mid-JSON on a real + # RSS cycle (2026-08-03). + "max_tokens": 2000, + } + try: + resp = self._session.post( + DEEPSEEK_URL, json=body, timeout=self._timeout) + except requests.RequestException as e: + self._warn_once("network", "DeepSeek call failed: %s", e) + return None + if resp.status_code != 200: + self._warn_once( + f"http_{resp.status_code}", + "DeepSeek HTTP %s: %s", resp.status_code, resp.text[:200]) + return None + try: + parsed = self._parse_response(resp.json()) + except ValueError: + parsed = None + if parsed is None: + self._warn_once("parse", "DeepSeek response was not the " + "expected JSON object — ignoring") + return parsed + + @staticmethod + def _parse_response(body: Any) -> Optional[Dict[str, Any]]: + """Extract the assistant's JSON object from a chat completion.""" + try: + content = body["choices"][0]["message"]["content"] + except (KeyError, IndexError, TypeError): + return None + try: + obj = json.loads(content) + except (TypeError, ValueError): + return None + return obj if isinstance(obj, dict) else None + + +# ── Process-wide singleton ──────────────────────────────────────────── + +_client: Optional[DeepSeekClient] = None +_client_lock = threading.Lock() + + +def get_llm_client() -> DeepSeekClient: + global _client + with _client_lock: + if _client is None: + settings = get_app_config().deepseek_settings + _client = DeepSeekClient( + api_key=deepseek_api_key(), + model=str(settings.get("model", DEFAULT_MODEL)), + timeout=float(settings.get("timeout_seconds", + DEFAULT_TIMEOUT)), + daily_cap=int(settings.get("daily_request_cap", + DEFAULT_DAILY_CAP)), + enabled=settings.get("enabled", True) is not False, + ) + return _client + + +def reset_llm_client() -> None: + """Drop the singleton (tests / config reloads).""" + global _client + with _client_lock: + _client = None diff --git a/main.py b/main.py index 89d1142..c49a7e1 100755 --- a/main.py +++ b/main.py @@ -161,6 +161,8 @@ def print_signal_report(report: SignalReport, detail: bool = False): print(f" {Colors.YELLOW}šŸ“…{Colors.RESET} {report.event_description}") if report.news_description and report.news_description != "Neutral news sentiment": print(f" {Colors.MAGENTA}šŸ“°{Colors.RESET} {report.news_description}") + if report.news_summary: + print(f" {Colors.MAGENTA}🧠{Colors.RESET} {report.news_summary}") # ── 6. DETAIL (only with --detail) ───────────────────────────── if detail and report.primary_analysis: @@ -206,6 +208,7 @@ def report_to_dict(report: SignalReport) -> dict: "event_description": report.event_description, "news_sentiment": round(report.news_sentiment, 3), "news_description": report.news_description, + "news_summary": report.news_summary, "bullish_reasons": report.bullish_reasons, "bearish_reasons": report.bearish_reasons, "warnings": report.warnings, diff --git a/news_engine.py b/news_engine.py index 21109f0..291ebda 100755 --- a/news_engine.py +++ b/news_engine.py @@ -8,6 +8,7 @@ Key principle: High-impact news events override technical signals. A perfect technical setup can be destroyed by an unexpected NFP print. """ +import hashlib import logging import re import threading @@ -21,10 +22,62 @@ from typing import Dict, List, Optional, Tuple import requests from config import NewsConfig +from llm_client import get_llm_client logger = logging.getLogger(__name__) +# ── DeepSeek prompts ───────────────────────────────────────────────── +# STATIC constants on purpose: DeepSeek prices a repeated byte-identical +# prompt prefix at the cache-hit rate (~50x cheaper), so nothing here may +# be interpolated or timestamped. All variable content travels in the +# user payload. LLM output is display/warning-only — it never gates +# signals (see NewsConfig.sentiment_affects_signals). + +_LLM_SENTIMENT_SYSTEM = """\ +You are a forex/gold news analyst. You receive JSON: +{"headlines": ["...", ...]} +For EACH headline, in the same order, score: +- "s": overall market sentiment of the headline, -1.0 (very bearish) + to 1.0 (very bullish), 0.0 if irrelevant or neutral. +- "xau": the effect on the GOLD (XAU/USD) price, -1.0 (gold down) + to 1.0 (gold up), 0.0 if irrelevant. +Gold doctrine: hawkish Fed / rate hikes / rising real yields / strong +dollar are gold-bearish; easing / rate cuts / safe-haven flows / +geopolitical stress / weak dollar are gold-bullish. +Handle negation correctly ("no rate hike" is not hawkish). +Reply with ONLY JSON: {"scores": [[s, xau], ...]} — exactly one +[s, xau] pair per input headline, in the same order, nothing else. + +Examples: +"Fed signals faster rate hikes as inflation stays hot" -> [0.4, -0.7] +"Gold surges to record high on safe-haven demand" -> [0.8, 0.9] +"Dollar slips after soft jobs report; Treasury yields fall" -> [-0.4, 0.6] +""" + +_LLM_EVENT_SYSTEM = """\ +You are a forex/gold analyst. You receive one released economic event as +JSON: {"event", "currency", "actual", "forecast", "previous"}. +Decide the likely effect of this RELEASE (actual vs forecast) on the +GOLD (XAU/USD) price. A stronger-than-forecast USD economy or a hawkish +implication is usually gold-bearish; weaker data or a dovish implication +is usually gold-bullish. Mind inverted metrics: higher unemployment or +jobless claims = weaker economy. +Reply with ONLY JSON: +{"gold_direction": "bullish"|"bearish"|"neutral", "reason": ""} +""" + +_LLM_SUMMARY_SYSTEM = """\ +You are a forex/gold market-context writer. You receive JSON with recent +"headlines" and today's high-impact "events" (released: true means the +number is out). Write a 1-2 sentence plain-English summary of the +current macro backdrop for a GOLD (XAU/USD) trader: the dominant theme +and its likely pull on gold. No advice, no hedging boilerplate, max 45 +words. +Reply with ONLY JSON: {"summary": "..."} +""" + + _RELEASE_VALUE_RE = re.compile(r"(-?\d+(?:\.\d+)?)\s*([kmbt])?", re.IGNORECASE) _RELEASE_MULTIPLIERS = {"k": 1e3, "m": 1e6, "b": 1e9, "t": 1e12} @@ -105,6 +158,7 @@ class FundamentalContext: news_headlines: List[NewsSentiment] = field(default_factory=list) event_impact_on_pairs: Dict[str, str] = field(default_factory=dict) calendar_available: bool = False # True if live calendar data was fetched + llm_summary: str = "" # DeepSeek market-context blurb (display-only) class NewsEngine: @@ -185,6 +239,17 @@ class NewsEngine: _headlines_cache: Tuple[float, List[NewsSentiment]] = (0.0, []) _EMPTY_RESULT_TTL = 60.0 # retry sooner after a failed/empty fetch + # DeepSeek result caches. Headlines and releases are immutable, so a + # scored one never needs re-scoring; bounded by dropping the oldest + # insertion. _llm_sent_cache is only touched under _cache_lock (its + # writer runs inside _cached_fetch); the other two take the lock + # around cache access but keep HTTP calls outside it. + _llm_sent_cache: Dict[str, Tuple[float, float]] = {} # sha1 → (s, xau) + _llm_event_cache: Dict[Tuple[str, str, str], str] = {} + _llm_summary_cache: Tuple[float, str] = (0.0, "") + _LLM_SENT_CACHE_MAX = 400 + _LLM_EVENT_CACHE_MAX = 100 + def __init__(self, config: NewsConfig): self.config = config self._session = requests.Session() @@ -264,6 +329,12 @@ class NewsEngine: except Exception as e: logger.warning(f"News sentiment fetch failed: {e}") + # 3. LLM market-context summary (display-only) + try: + context.llm_summary = self._llm_market_summary(context) + except Exception as e: + logger.debug(f"LLM market summary failed: {e}") + return context @staticmethod @@ -311,10 +382,17 @@ class NewsEngine: ) if event.minutes_until > -window: if surprised: - return "caution", ( + desc = ( f"{event.currency} {event.event} released — {surprise} surprise " f"(actual {event.actual} vs forecast {event.forecast})" ) + # LLM enrichment touches the DESCRIPTION only — + # `surprise` is numeric-direction and does not know + # whether a beat is gold-bullish; the LLM does. + gold_dir = self._llm_event_direction(event) + if gold_dir: + desc += f" — gold {gold_dir}" + return "caution", desc return "caution", f"{event.currency} {event.event} just released" return "clear", "No significant events" @@ -596,6 +674,11 @@ class NewsEngine: except Exception as e: logger.debug(f"Failed to fetch RSS from {feed_url}: {e}") + try: + self._apply_llm_sentiment(headlines) + except Exception as e: + logger.debug(f"LLM sentiment failed, keyword scores stand: {e}") + return headlines def _analyze_headline_sentiment( @@ -637,6 +720,130 @@ class NewsEngine: return (bullish_score - bearish_score) / total + # ── DeepSeek enrichment (display/warning-only, keyword fallback) ── + + def _llm_headline_scores( + self, texts: List[str] + ) -> Optional[List[Tuple[float, float]]]: + """One batched DeepSeek call → [(overall, xau), ...] aligned with + `texts`, or None on any failure/mismatch (caller keeps keyword + scores). Tests stub this method directly.""" + result = get_llm_client().chat_json( + _LLM_SENTIMENT_SYSTEM, {"headlines": texts}) + if not result: + return None + scores = result.get("scores") + if not isinstance(scores, list) or len(scores) != len(texts): + return None + out = [] + for item in scores: + try: + s = max(-1.0, min(1.0, float(item[0]))) + x = max(-1.0, min(1.0, float(item[1]))) + except (TypeError, KeyError, ValueError, IndexError): + return None + out.append((s, x)) + return out + + def _apply_llm_sentiment(self, headlines: List[NewsSentiment]) -> None: + """Overwrite keyword sentiment on `headlines` with LLM scores. + + Runs inside _cached_fetch, i.e. UNDER _cache_lock — cache access + is direct and the (rare) HTTP call blocks sibling scans at most + once per cache window, matching the existing shared-fetch design. + Only headlines missing from the cache are sent; a failed batch + changes nothing. + """ + if not headlines or not get_llm_client().available: + return + cache = NewsEngine._llm_sent_cache + keyed = [(hashlib.sha1(h.headline.encode()).hexdigest(), h) + for h in headlines] + # Cap the batch so the reply fits the client's max_tokens — the + # overflow stays keyword-scored this cycle and, still uncached, + # goes out with the next window's batch. + new = [(k, h) for k, h in keyed if k not in cache][:25] + if new: + scores = self._llm_headline_scores([h.headline for _, h in new]) + if scores is not None: + for (k, _), sc in zip(new, scores): + cache[k] = sc + while len(cache) > self._LLM_SENT_CACHE_MAX: + cache.pop(next(iter(cache))) + for k, h in keyed: + sc = cache.get(k) + if sc is None: + continue + overall, xau = sc + h.sentiment_score = overall + h.sentiment_by_currency = { + cur: (xau if cur == "XAU" else overall) + for cur in h.currencies_mentioned + } + + def _llm_event_direction(self, event: EconomicEvent) -> Optional[str]: + """Gold direction for a released surprise, e.g. "bearish: strong + jobs lift yields". None → caller keeps the numeric description. + Cached per release; failures are NOT cached (retry next scan).""" + key = (event.event, event.currency, str(event.actual or "")) + with NewsEngine._cache_lock: + if key in NewsEngine._llm_event_cache: + return NewsEngine._llm_event_cache[key] or None + client = get_llm_client() + if not client.available: + return None + result = client.chat_json(_LLM_EVENT_SYSTEM, { + "event": event.event, + "currency": event.currency, + "actual": event.actual, + "forecast": event.forecast, + "previous": event.previous, + }) + if not result: + return None + direction = result.get("gold_direction") + if direction not in ("bullish", "bearish", "neutral"): + return None + reason = str(result.get("reason", "")).strip()[:80] + text = "" if direction == "neutral" else ( + f"{direction}: {reason}" if reason else direction) + with NewsEngine._cache_lock: + NewsEngine._llm_event_cache[key] = text + while len(NewsEngine._llm_event_cache) > self._LLM_EVENT_CACHE_MAX: + NewsEngine._llm_event_cache.pop( + next(iter(NewsEngine._llm_event_cache))) + return text or None + + def _llm_market_summary(self, context: FundamentalContext) -> str: + """1-2 sentence macro blurb for alerts; "" when unavailable. + Cached (success OR failure) for the context-cache TTL so at most + one call is made per window.""" + with NewsEngine._cache_lock: + ts, cached = NewsEngine._llm_summary_cache + if time.time() - ts < self.config.context_cache_ttl_seconds: + return cached + high_impact = [e for e in context.events_today if e.is_high_impact] + if not context.news_headlines and not high_impact: + return "" + client = get_llm_client() + if not client.available: + return "" + result = client.chat_json(_LLM_SUMMARY_SYSTEM, { + "headlines": [h.headline for h in context.news_headlines[:15]], + "events": [ + {"event": e.event, "currency": e.currency, + "actual": e.actual, "forecast": e.forecast, + "released": not e.is_upcoming} + for e in high_impact[:10] + ], + }) + summary = "" + if result: + summary = str(result.get("summary", "")).strip()[:300] + with NewsEngine._cache_lock: + NewsEngine._llm_summary_cache = (time.time(), summary) + return summary + def _extract_currencies(self, headline: str) -> List[str]: """Extract mentioned currencies from a headline.""" headline_lower = headline.lower() diff --git a/signal_aggregator.py b/signal_aggregator.py index b72974b..4bf9bba 100755 --- a/signal_aggregator.py +++ b/signal_aggregator.py @@ -79,6 +79,7 @@ class SignalReport: event_description: str = "" news_sentiment: float = 0.0 news_description: str = "" + news_summary: str = "" # LLM market-context blurb (display-only) final_signal: SignalStrength = SignalStrength.NEUTRAL final_score: float = 0.0 @@ -832,11 +833,13 @@ class SignalAggregator: sentiment, sent_desc = news.get_pair_sentiment(symbol, context) report.news_sentiment = sentiment report.news_description = sent_desc + report.news_summary = context.llm_summary else: report.event_risk = "clear" report.event_description = "" report.news_sentiment = 0.0 report.news_description = "" + report.news_summary = "" # ── 5. Spread estimate from primary data (avoids extra API call) ─ try: diff --git a/telegram_bot.py b/telegram_bot.py index 802c237..41813e6 100755 --- a/telegram_bot.py +++ b/telegram_bot.py @@ -395,6 +395,8 @@ def format_signal_message(report: SignalReport) -> str: extra.append(f"šŸ“‹ {ind_summary}") for w in report.warnings: extra.append(f"āš ļø {_escape_md(w)}") + if report.news_summary: + extra.append(f"šŸ“° {_escape_md(report.news_summary)}") if extra: lines += [""] + extra diff --git a/test_all.py b/test_all.py index d552584..dc6edb4 100644 --- a/test_all.py +++ b/test_all.py @@ -26,7 +26,7 @@ sys.path.insert(0, os.path.dirname(__file__)) import json import app_config for _k in ("TELEGRAM_TOKEN", "TELEGRAM_CHAT_ID", "TELEGRAM_CHANNEL_ID", - "TWELVE_DATA_API_KEY"): + "TWELVE_DATA_API_KEY", "DEEPSEEK_API_KEY"): os.environ.pop(_k, None) _TEST_APP = app_config.AppConfig( provider_settings={"twelve_data": {"api_key": "test-key"}} @@ -3044,6 +3044,42 @@ def _(): pass +@test("app_config: deepseek block parses, validates, defaults empty") +def _(): + app = load_app_config(_write_json({"deepseek": { + "api_key": "sk-x", "model": "deepseek-v4-flash", "enabled": False, + "timeout_seconds": 5, "daily_request_cap": 100}})) + assert app.deepseek_settings["api_key"] == "sk-x" + assert app.deepseek_settings["enabled"] is False + assert load_app_config(_write_json({})).deepseek_settings == {} + for payload in ( + {"deepseek": {"api_keyy": "typo"}}, # unknown key + {"deepseek": {"enabled": "yes"}}, # not a bool + {"deepseek": {"timeout_seconds": 0}}, # not positive + {"deepseek": {"timeout_seconds": True}}, # bool is not a number + {"deepseek": {"daily_request_cap": 2.5}}, # not an int + {"deepseek": {"model": ""}}, # empty model + {"deepseek": "oops"}, # not an object + ): + try: + load_app_config(_write_json(payload)) + assert False, f"should have raised for {payload}" + except AppConfigError: + pass + + +@test("app_config: deepseek_api_key resolves env → config.json → empty") +def _(): + from app_config import deepseek_api_key + app = load_app_config(_write_json({"deepseek": {"api_key": "json-ds"}})) + with patch.dict(os.environ, {}, clear=True): + assert deepseek_api_key(app) == "json-ds" + with patch.dict(os.environ, {"DEEPSEEK_API_KEY": "env-ds"}): + assert deepseek_api_key(app) == "env-ds" + with patch.dict(os.environ, {}, clear=True): + assert deepseek_api_key(AppConfig()) == "" + + @test("config.py: hardcoded twelve_data_api_key field is gone") def _(): assert not hasattr(PlatformConfig(), "twelve_data_api_key"), \ @@ -4657,6 +4693,278 @@ def _(): assert hits == ["M5_SELL_QUALITY", "SELL_HOUR", "TREND_EXHAUSTION"], hits +# ══════════════════════════════════════════════════════════════════════ +# MODULE 19: DeepSeek news enrichment (llm_client + news_engine seams) +# ══════════════════════════════════════════════════════════════════════ + +print("\n═══ Module 19: DeepSeek news enrichment ═══") + +import datetime as _dt_m19 +import llm_client as lc +from llm_client import DeepSeekClient +from news_engine import (EconomicEvent, FundamentalContext, NewsEngine, + NewsSentiment) + + +def _stub_resp(payload, status=200): + return type("R", (), { + "status_code": status, + "text": json.dumps(payload), + "json": lambda self: payload, + })() + + +def _chat_body(content_obj): + return {"choices": [{"message": {"content": json.dumps(content_obj)}}]} + + +def _counting_session(payload, status=200): + class S: + calls = 0 + + def post(self, url, json=None, timeout=None): + S.calls += 1 + return _stub_resp(payload, status) + return S() + + +@test("llm_client: response parsing and availability gating") +def _(): + parse = DeepSeekClient._parse_response + assert parse(_chat_body({"a": 1})) == {"a": 1} + assert parse({"choices": []}) is None + assert parse({"error": "x"}) is None + assert parse({"choices": [{"message": {"content": "not json"}}]}) is None + assert parse({"choices": [{"message": {"content": "[1,2]"}}]}) is None + + # keyless / disabled clients never touch the network + for client in (DeepSeekClient(api_key=""), + DeepSeekClient(api_key="k", enabled=False)): + assert client.available is False + s = _counting_session(_chat_body({"a": 1})) + client._session = s + assert client.chat_json("sys", {"q": 1}) is None + assert type(s).calls == 0 + + +@test("llm_client: daily cap blocks requests and resets on UTC date roll") +def _(): + client = DeepSeekClient(api_key="k", daily_cap=2) + s = _counting_session(_chat_body({"ok": True})) + client._session = s + assert client.chat_json("sys", {}) == {"ok": True} + assert client.chat_json("sys", {}) == {"ok": True} + assert client.chat_json("sys", {}) is None, "third call must hit the cap" + assert type(s).calls == 2, "capped call must make no HTTP request" + # UTC date roll resets the counter + client._count_day = (_dt_m19.datetime.now(_dt_m19.timezone.utc).date() + - _dt_m19.timedelta(days=1)) + assert client.chat_json("sys", {}) == {"ok": True} + assert type(s).calls == 3 + + +@test("llm_client: HTTP error and malformed body return None") +def _(): + client = DeepSeekClient(api_key="k") + client._session = _counting_session({"detail": "quota"}, status=429) + assert client.chat_json("sys", {}) is None + client._session = _counting_session({"unexpected": "shape"}) + assert client.chat_json("sys", {}) is None + + +class _StubLLM: + """Drop-in for the llm_client singleton.""" + def __init__(self, reply): + self.reply = reply + self.calls = 0 + available = True + + def chat_json(self, system_prompt, user_payload): + self.calls += 1 + return self.reply + + +@test("NewsEngine: _llm_headline_scores validates, clamps, rejects mismatch") +def _(): + ne = NewsEngine(NewsConfig()) + try: + lc._client = _StubLLM({"scores": [[5, -3], [0.2, 0.1]]}) + assert ne._llm_headline_scores(["a", "b"]) == [(1.0, -1.0), (0.2, 0.1)] + lc._client = _StubLLM({"scores": [[0.1, 0.1]]}) + assert ne._llm_headline_scores(["a", "b"]) is None, "length mismatch" + lc._client = _StubLLM({"scores": [["hot", 0], [0.3]]}) + assert ne._llm_headline_scores(["a", "b"]) is None, "non-numeric/short" + lc._client = _StubLLM(None) + assert ne._llm_headline_scores(["a"]) is None, "failed call" + finally: + lc._client = None + + +@test("NewsEngine: LLM sentiment overrides keyword scores per currency") +def _(): + now = _dt_m19.datetime.now(_dt_m19.timezone.utc) + ne = NewsEngine(NewsConfig()) + NewsEngine._llm_sent_cache = {} + try: + lc._client = _StubLLM({"ok": True}) # available, unused (stubbed below) + h = NewsSentiment( + headline="Fed signals rate hike", source="s", timestamp=now, + currencies_mentioned=["USD", "XAU"], sentiment_score=1.0, + keywords=["rate hike"], + sentiment_by_currency={"USD": 1.0, "XAU": -1.0}) + ne._llm_headline_scores = lambda texts: [(0.4, -0.7)] * len(texts) + ne._apply_llm_sentiment([h]) + assert h.sentiment_score == 0.4 + assert h.sentiment_by_currency == {"USD": 0.4, "XAU": -0.7} + assert h.keywords == ["rate hike"], "keywords stay keyword-derived" + finally: + NewsEngine._llm_sent_cache = {} + lc._client = None + + +@test("NewsEngine: LLM failure keeps keyword sentiment untouched") +def _(): + now = _dt_m19.datetime.now(_dt_m19.timezone.utc) + ne = NewsEngine(NewsConfig()) + NewsEngine._llm_sent_cache = {} + try: + lc._client = _StubLLM({"ok": True}) + h = NewsSentiment( + headline="Fed signals rate hike", source="s", timestamp=now, + currencies_mentioned=["USD"], sentiment_score=0.6, + keywords=[], sentiment_by_currency={"USD": 0.6}) + ne._llm_headline_scores = lambda texts: None + ne._apply_llm_sentiment([h]) + assert h.sentiment_score == 0.6 + assert h.sentiment_by_currency == {"USD": 0.6} + # unavailable client: not even the scores call happens + lc._client = None + ne._llm_headline_scores = lambda texts: (_ for _ in ()).throw( + AssertionError("must not be called when unavailable")) + ne._apply_llm_sentiment([h]) + finally: + NewsEngine._llm_sent_cache = {} + lc._client = None + + +@test("NewsEngine: LLM sentiment cache only sends new headlines") +def _(): + now = _dt_m19.datetime.now(_dt_m19.timezone.utc) + ne = NewsEngine(NewsConfig()) + NewsEngine._llm_sent_cache = {} + sent_batches = [] + + def make(h): + return NewsSentiment( + headline=h, source="s", timestamp=now, + currencies_mentioned=["XAU"], sentiment_score=0.0, + keywords=[], sentiment_by_currency={"XAU": 0.0}) + + try: + lc._client = _StubLLM({"ok": True}) + + def scores(texts): + sent_batches.append(list(texts)) + return [(0.5, 0.5)] * len(texts) + + ne._llm_headline_scores = scores + ne._apply_llm_sentiment([make("headline one"), make("headline two")]) + ne._apply_llm_sentiment([make("headline one"), make("headline two"), + make("headline three")]) + assert sent_batches == [["headline one", "headline two"], + ["headline three"]], sent_batches + finally: + NewsEngine._llm_sent_cache = {} + lc._client = None + + +@test("NewsEngine: LLM enriches surprise description, numeric fallback intact") +def _(): + now = _dt_m19.datetime.now(_dt_m19.timezone.utc) + ne = NewsEngine(NewsConfig()) + NewsEngine._llm_event_cache = {} + released = EconomicEvent( + time=now - _dt_m19.timedelta(minutes=10), currency="USD", + impact="high", event="Nonfarm Payrolls", + forecast="200K", actual="300K") + ctx = FundamentalContext(events_today=[released]) + try: + # failure first: description must equal today's numeric form + lc._client = _StubLLM(None) + risk, desc = ne.get_event_risk_level("XAU/USD", ctx) + assert risk == "caution" + assert desc.endswith("(actual 300K vs forecast 200K)"), desc + # then success: LLM direction appended to the description only + lc._client = _StubLLM({"gold_direction": "bearish", + "reason": "strong jobs lift yields"}) + risk, desc = ne.get_event_risk_level("XAU/USD", ctx) + assert risk == "caution" + assert desc.endswith("— gold bearish: strong jobs lift yields"), desc + # bogus direction → numeric fallback + NewsEngine._llm_event_cache = {} + lc._client = _StubLLM({"gold_direction": "sideways"}) + _, desc = ne.get_event_risk_level("XAU/USD", ctx) + assert desc.endswith("(actual 300K vs forecast 200K)"), desc + finally: + NewsEngine._llm_event_cache = {} + lc._client = None + + +@test("NewsEngine: market summary cached across engine instances") +def _(): + now = _dt_m19.datetime.now(_dt_m19.timezone.utc) + NewsEngine._llm_summary_cache = (0.0, "") + ctx = FundamentalContext(news_headlines=[NewsSentiment( + headline="Gold steady", source="s", timestamp=now, + currencies_mentioned=["XAU"], sentiment_score=0.0, keywords=[])]) + try: + stub = _StubLLM({"summary": "Soft dollar supports gold."}) + lc._client = stub + assert NewsEngine(NewsConfig())._llm_market_summary(ctx) == \ + "Soft dollar supports gold." + # second engine instance within the TTL: served from cache, no call + assert NewsEngine(NewsConfig())._llm_market_summary(ctx) == \ + "Soft dollar supports gold." + assert stub.calls == 1 + # failure caches "" so a broken API is retried next window, not per scan + NewsEngine._llm_summary_cache = (0.0, "") + lc._client = _StubLLM(None) + assert NewsEngine(NewsConfig())._llm_market_summary(ctx) == "" + assert NewsEngine._llm_summary_cache[1] == "" + finally: + NewsEngine._llm_summary_cache = (0.0, "") + lc._client = None + + +@test("SignalReport.news_summary flows to Telegram, resets when news off") +def _(): + import telegram_bot as tb + config = PlatformConfig() + config.primary_timeframe = Timeframe.H1 + config.confirmation_timeframes = [Timeframe.M15, Timeframe.H4] + agg = SignalAggregator(config) + dfs = { + Timeframe.H1: make_ohlcv(300, seed=42), + Timeframe.M15: make_ohlcv(300, seed=43), + Timeframe.H4: make_ohlcv(300, seed=44), + } + ctx = FundamentalContext(llm_summary="Test macro blurb.") + report = agg.analyze_from_dataframes( + "XAU/USD", dfs, context=ctx, use_news=True) + assert report.news_summary == "Test macro blurb." + assert "Test macro blurb." in tb.format_signal_message(report) + report = agg.analyze_from_dataframes( + "XAU/USD", dfs, context=ctx, use_news=False) + assert report.news_summary == "" + + +@test("news defaults: LLM never gates signals (2026-07 audit lock)") +def _(): + cfg = NewsConfig() + assert cfg.event_risk_caps_signals is False + assert cfg.sentiment_affects_signals is False + + # ══════════════════════════════════════════════════════════════════════ # SUMMARY # ══════════════════════════════════════════════════════════════════════