news: DeepSeek validation, vs_forecast fix, /status source health
Validation (llm_news_eval.py harness + 78-headline golden CSV + verdict doc): headline scoring 78/78 vs keyword baseline 24/78 — validated; event gold-direction calls misread close K/M numeric comparisons ~11% of the time (both claims cases in sample, JOLTS 8.79M vs 8.84M). Fix: event payload now carries code-computed "vs_forecast" (from EconomicEvent.surprise) and the prompt gains an explicit inverted- metrics rule + worked claims example — iteration 1 (field alone) fixed the arithmetic but exposed a masked inversion error. 18/18 on live re-verification (4 misreads fixed, 11 regression cases clean, 3 fresh both-side probes). /status: new "News & AI" section — per-source health for the calendar chain (which of FF/MyFXBook served), each RSS feed, and DeepSeek (daily count, last error kind), written passively by the existing fetch paths under _cache_lock. Failures WARN once on transition (repeats DEBUG, recovery INFO) so the rotating log carries outage stories without 10-min spam. test_all.py: +5 Module 19 tests (351 total); also carries the Module 22 fixed-RR-TP tests from the prior research session (shared file). Replay artifacts dir gitignored like commit_ladder. Claude-Session: https://claude.ai/code/session_017mbXs9HRawZhD1uZ3ze7zQ
This commit is contained in:
Harwinder Sekhon
+98
-8
@@ -57,12 +57,22 @@ Examples:
|
||||
|
||||
_LLM_EVENT_SYSTEM = """\
|
||||
You are a forex/gold analyst. You receive one released economic event as
|
||||
JSON: {"event", "currency", "actual", "forecast", "previous"}.
|
||||
Decide the likely effect of this RELEASE (actual vs forecast) on the
|
||||
GOLD (XAU/USD) price. A stronger-than-forecast USD economy or a hawkish
|
||||
implication is usually gold-bearish; weaker data or a dovish implication
|
||||
is usually gold-bullish. Mind inverted metrics: higher unemployment or
|
||||
jobless claims = weaker economy.
|
||||
JSON: {"event", "currency", "actual", "forecast", "previous",
|
||||
"vs_forecast"}. "vs_forecast" states whether the actual print came in
|
||||
above or below the forecast; it is computed programmatically — trust it
|
||||
over your own comparison of the numbers. If it is "unknown", compare
|
||||
actual and forecast yourself.
|
||||
Decide the likely effect of this RELEASE on the GOLD (XAU/USD) price.
|
||||
A stronger-than-forecast USD economy or a hawkish implication is usually
|
||||
gold-bearish; weaker data or a dovish implication is usually
|
||||
gold-bullish.
|
||||
Inverted metrics — for unemployment rate and unemployment/jobless
|
||||
claims a LOWER number means a STRONGER labor market: below forecast =
|
||||
strong economy = usually gold-bearish; above forecast = weak economy =
|
||||
usually gold-bullish.
|
||||
Example: {"event": "Unemployment Claims", "actual": "202K", "forecast":
|
||||
"217K", "vs_forecast": "below forecast"} ->
|
||||
{"gold_direction": "bearish", "reason": "fewer claims show strong labor market"}
|
||||
Reply with ONLY JSON:
|
||||
{"gold_direction": "bullish"|"bearish"|"neutral", "reason": "<max 10 words>"}
|
||||
"""
|
||||
@@ -250,6 +260,14 @@ class NewsEngine:
|
||||
_LLM_SENT_CACHE_MAX = 400
|
||||
_LLM_EVENT_CACHE_MAX = 100
|
||||
|
||||
# Per-source fetch health for status displays (telegram /status).
|
||||
# name → {"ok": ts|None, "err": ts|None, "detail": str, "extra": str}.
|
||||
# All writers run under _cache_lock (fetches happen inside
|
||||
# _cached_fetch); readers render without the lock — worst case a
|
||||
# display line is one cycle stale. A source is "failing" when its
|
||||
# last error is newer than its last success.
|
||||
_source_health: Dict[str, dict] = {}
|
||||
|
||||
def __init__(self, config: NewsConfig):
|
||||
self.config = config
|
||||
self._session = requests.Session()
|
||||
@@ -453,6 +471,36 @@ class NewsEngine:
|
||||
"canada": "CAD", "new zealand": "NZD",
|
||||
}
|
||||
|
||||
@classmethod
|
||||
def _record_source(cls, name: str, ok: bool, detail: str = "",
|
||||
extra: str = "") -> None:
|
||||
"""Track one dependency's fetch outcome for status displays.
|
||||
Logs state TRANSITIONS so the log file carries the story without
|
||||
a line every 10-min cycle: first failure WARNING (with detail),
|
||||
repeat failures DEBUG, recovery INFO."""
|
||||
h = cls._source_health.setdefault(
|
||||
name, {"ok": None, "err": None, "detail": "", "extra": ""})
|
||||
was_failing = h["err"] is not None and (
|
||||
h["ok"] is None or h["err"] > h["ok"])
|
||||
if ok:
|
||||
h["ok"] = time.time()
|
||||
h["extra"] = extra
|
||||
if was_failing:
|
||||
logger.info("Source recovered: %s", name)
|
||||
else:
|
||||
h["err"] = time.time()
|
||||
h["detail"] = detail
|
||||
if was_failing:
|
||||
logger.debug("Source still failing: %s (%s)", name, detail)
|
||||
else:
|
||||
logger.warning("Source failing: %s (%s)", name, detail)
|
||||
|
||||
@staticmethod
|
||||
def source_is_failing(entry: Optional[dict]) -> bool:
|
||||
"""True when the entry's last error is newer than its last ok."""
|
||||
return bool(entry and entry["err"] is not None and
|
||||
(entry["ok"] is None or entry["err"] > entry["ok"]))
|
||||
|
||||
def _cached_fetch(self, cache_attr: str, fetch_fn) -> list:
|
||||
"""Return cached data for `cache_attr` or refetch when the TTL
|
||||
expired. Successful results live for context_cache_ttl_seconds;
|
||||
@@ -485,14 +533,27 @@ class NewsEngine:
|
||||
Returns an empty list (with a log warning) if unavailable —
|
||||
we never generate fake events since that causes constant false alarms.
|
||||
"""
|
||||
for fetcher in (self._try_fetch_calendar_api, self._try_fetch_myfxbook_rss):
|
||||
sources = (
|
||||
(self._try_fetch_calendar_api, "calendar:forexfactory",
|
||||
"ForexFactory"),
|
||||
(self._try_fetch_myfxbook_rss, "calendar:myfxbook", "MyFXBook"),
|
||||
)
|
||||
for fetcher, src, label in sources:
|
||||
try:
|
||||
events = fetcher()
|
||||
if events:
|
||||
self._record_source(src, True)
|
||||
self._record_source(
|
||||
"calendar", True,
|
||||
extra=f"{len(events)} events via {label}")
|
||||
return events
|
||||
self._record_source(src, False, "no events returned")
|
||||
except Exception as e:
|
||||
self._record_source(
|
||||
src, False, f"{type(e).__name__}: {e}")
|
||||
logger.debug(f"Calendar fetch exception ({fetcher.__name__}): {e}")
|
||||
|
||||
self._record_source("calendar", False, "all sources failed")
|
||||
logger.info(
|
||||
"Economic calendar data unavailable. "
|
||||
"Event risk checks will be skipped for this cycle."
|
||||
@@ -639,11 +700,16 @@ class NewsEngine:
|
||||
"""Fetch and analyze news headlines for sentiment."""
|
||||
headlines = []
|
||||
seen_titles = set()
|
||||
ok_feeds = 0
|
||||
|
||||
for feed_url in self.config.rss_feeds:
|
||||
host = feed_url.split("/")[2] if "//" in feed_url else feed_url
|
||||
kept_before = len(headlines)
|
||||
try:
|
||||
resp = self._session.get(feed_url, timeout=10)
|
||||
if resp.status_code != 200:
|
||||
self._record_source(
|
||||
f"rss:{host}", False, f"HTTP {resp.status_code}")
|
||||
continue
|
||||
|
||||
root = ET.fromstring(resp.content)
|
||||
@@ -689,9 +755,21 @@ class NewsEngine:
|
||||
},
|
||||
))
|
||||
|
||||
ok_feeds += 1
|
||||
self._record_source(
|
||||
f"rss:{host}", True,
|
||||
extra=f"{len(headlines) - kept_before} kept")
|
||||
except Exception as e:
|
||||
self._record_source(
|
||||
f"rss:{host}", False, f"{type(e).__name__}: {e}")
|
||||
logger.debug(f"Failed to fetch RSS from {feed_url}: {e}")
|
||||
|
||||
self._record_source(
|
||||
"headlines", ok_feeds > 0,
|
||||
detail="all feeds failed" if ok_feeds == 0 else "",
|
||||
extra=(f"{len(headlines)} headlines from "
|
||||
f"{ok_feeds}/{len(self.config.rss_feeds)} feeds"))
|
||||
|
||||
try:
|
||||
self._apply_llm_sentiment(headlines)
|
||||
except Exception as e:
|
||||
@@ -802,7 +880,13 @@ class NewsEngine:
|
||||
def _llm_event_direction(self, event: EconomicEvent) -> Optional[str]:
|
||||
"""Gold direction for a released surprise, e.g. "bearish: strong
|
||||
jobs lift yields". None → caller keeps the numeric description.
|
||||
Cached per release; failures are NOT cached (retry next scan)."""
|
||||
Cached per release; failures are NOT cached (retry next scan).
|
||||
|
||||
The actual-vs-forecast comparison is precomputed and sent as
|
||||
"vs_forecast": the 2026-08-06 validation found the model reads
|
||||
close K/M-suffixed pairs backwards ~11% of the time (e.g. claims
|
||||
202K vs 217K called "higher than forecast") when left to compare
|
||||
the raw numbers itself."""
|
||||
key = (event.event, event.currency, str(event.actual or ""))
|
||||
with NewsEngine._cache_lock:
|
||||
if key in NewsEngine._llm_event_cache:
|
||||
@@ -810,12 +894,18 @@ class NewsEngine:
|
||||
client = get_llm_client()
|
||||
if not client.available:
|
||||
return None
|
||||
vs_forecast = {
|
||||
"positive": "above forecast",
|
||||
"negative": "below forecast",
|
||||
"inline": "in line with forecast",
|
||||
}.get(event.surprise or "", "unknown")
|
||||
result = client.chat_json(_LLM_EVENT_SYSTEM, {
|
||||
"event": event.event,
|
||||
"currency": event.currency,
|
||||
"actual": event.actual,
|
||||
"forecast": event.forecast,
|
||||
"previous": event.previous,
|
||||
"vs_forecast": vs_forecast,
|
||||
})
|
||||
if not result:
|
||||
return None
|
||||
|
||||
Reference in New Issue
Block a user