harry/ForexBot2
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

news: beginner-language summary, 4 h headline window, drop neutral headlines; --news log report

Browse Source 
Verdict from the 3-week signal log (backtest/research/news_sentiment_value_2026-09-04.md):
news_sentiment is a trailing-24h echo of price (r +0.67 with the past day, +0.03 with
the next; most bullish 2 h after the 08-25 top, bearish 44 h later; most bearish at the
09-02 low, bullish 16 h later). Per-headline LLM labels are correct; the 12 h pool and
the averaging are what destroy the information. Both gating flags stay False.

- news_engine: _LLM_SUMMARY_SYSTEM rewritten for someone who has never traded (no
  acronyms, two worked examples); summary payload newest-first with the platform's own
  gold direction; reply cap 300 -> 400 chars.
- config: headline_max_age_seconds 43200 -> 14400; RSS items without a parsable
  pubDate are dropped (they used to be stamped "now" and never aged out).
- news_engine.get_fundamental_context drops headlines whose every relevant score is
  inside +-0.1 (_is_neutral_headline) before the per-currency mean; news_sentiment
  from this date is a mean over directional <=4 h headlines (log comparisons split here).
- static: dashboard copy for the 4 h pool; news_summary.php prompt gets the same style rule.
- analyze_signal_log: --news (under --stored-candles) prints sentiment lead/lag with
  split halves, sentiment at the window high/low with flip lags, and outcome buckets by
  direction x news x macro stance; LogRow gains news_sentiment/event_risk; _row_index shared.
- tests: 406 -> 414 (age window, undated drop, neutral filter, summary prompt/payload/cap,
  --news helpers, load_rows round-trip); RSS fixtures now carry a pubDate.
- docs: user/beginner guides, CLAUDE.md, deploy README.

Claude-Session: https://claude.ai/code/session_0169BgFtWjp1SEAa5392Q9jH
This commit is contained in:
Harwinder Sekhon
2026-09-04 15:14:58 -04:00
parent 593ddfe81f
commit a3d7594f0c
11 changed files with 795 additions and 38 deletions
Download Patch File Download Diff File Expand all files Collapse all files
news_engine.py
+99 -13
View File
@@ -11,6 +11,7 @@ A perfect technical setup can be destroyed by an unexpected NFP print.
import hashlib
import html
import logging
import math
import re
import threading
import time
@@ -79,13 +80,25 @@ Reply with ONLY JSON:
"""
_LLM_SUMMARY_SYSTEM = """\
You are a forex/gold market-context writer. You receive JSON with recent
"headlines" and today's high-impact "events" (released: true means the
number is out). Write a 1-2 sentence plain-English summary of the
current macro backdrop for a GOLD (XAU/USD) trader: the dominant theme
and its likely pull on gold. No advice, no hedging boilerplate, max 45
words.
You write a short news note about the gold price for someone who has
never traded. You receive JSON with recent "headlines" (newest first;
each carries "gold": "up" or "down", the platform's own reading of how
that story pushes the gold price — trust it) and today's high-impact
"events" (released: true means the number is out).
Say three things, in this order: what is happening to gold right now,
why it matters for the gold price, and the next scheduled report to
watch if one is listed. Use only the headlines and events given; if
they are thin, say so plainly.
Style: plain everyday English, short simple sentences, no acronyms, no
abbreviations, no trader words. Write "the US central bank", not "the
Fed"; "interest rates", not "yields" or "hawkish"; "a report on US
prices", not "CPI" or "PCE"; "gold", never "XAU". No advice, no hedging
boilerplate. 2-3 sentences, max 55 words.
Reply with ONLY JSON: {"summary": "..."}
Examples:
{"summary": "Gold is falling today. The US central bank hinted it may raise interest rates, which makes the dollar stronger, and a stronger dollar usually pushes gold down. The next thing to watch is Thursday's report on US prices."}
{"summary": "Gold is climbing. Fresh fighting in the Middle East has people moving money into gold as a safe place, and a weaker dollar is helping too. No major US reports are due today."}
"""
@@ -215,6 +228,48 @@ class FundamentalContext:
llm_summary: str = "" # DeepSeek market-context blurb (display-only)
# Headlines whose every relevant score is inside ±NEUTRAL_HEADLINE_THRESHOLD
# lean neither way: they carry no direction for display and only dilute the
# per-currency mean. Same threshold agent_panel uses for its macro payload.
NEUTRAL_HEADLINE_THRESHOLD = 0.1
def _is_neutral_headline(h) -> bool:
"""True when no per-currency score on the headline leans either way.
A headline whose overall score is 0.0 but whose gold-specific score is
-0.7 is NOT neutral. Missing, empty or non-finite scores count as
neutral (they cannot be shown as leaning one way)."""
by_currency = getattr(h, "sentiment_by_currency", None) or {}
overall = getattr(h, "sentiment_score", None)
values = []
for currency in (getattr(h, "currencies_mentioned", None) or []):
value = by_currency.get(currency, overall)
if isinstance(value, (int, float)) and math.isfinite(value):
values.append(abs(float(value)))
return not values or max(values) < NEUTRAL_HEADLINE_THRESHOLD
def _gold_direction_label(h) -> str:
""""up" / "down" / "flat" for the market-summary payload — the same
XAU-first, USD-inverted reading agent_panel uses for its macro persona."""
by_currency = getattr(h, "sentiment_by_currency", None) or {}
xau = by_currency.get("XAU")
usd = by_currency.get("USD")
if isinstance(xau, (int, float)) and math.isfinite(xau):
score = float(xau)
elif isinstance(usd, (int, float)) and math.isfinite(usd):
score = -float(usd)
else:
raw = getattr(h, "sentiment_score", None)
score = float(raw) if isinstance(raw, (int, float)) and math.isfinite(raw) else 0.0
if score > 0:
return "up"
if score < 0:
return "down"
return "flat"
class NewsEngine:
"""
Economic calendar and news sentiment analyzer.
@@ -375,6 +430,17 @@ class NewsEngine:
# 2. News sentiment
try:
headlines = self._fetch_news_sentiment()
# Drop headlines that lean neither way (2026-09-04). Applied
# here rather than in the fetch so the class-level cache and
# the feed-health counts still describe what the feeds sent;
# every consumer (aggregate score, summary, dashboard, panel,
# outlook, signal log) reads context.news_headlines.
kept = [h for h in headlines if not _is_neutral_headline(h)]
if len(kept) != len(headlines):
logger.debug(
f"News: dropped {len(headlines) - len(kept)} neutral "
f"headlines, {len(kept)} kept")
headlines = kept
context.news_headlines = headlines
# Aggregate sentiment per currency (per-currency score where
@@ -745,6 +811,7 @@ class NewsEngine:
headlines = []
seen_titles = set()
ok_feeds = 0
undated = 0
for feed_url in self.config.rss_feeds:
host = feed_url.split("/")[2] if "//" in feed_url else feed_url
@@ -770,8 +837,14 @@ class NewsEngine:
pub_date_el.text if pub_date_el is not None else None
)
# Only recent headlines (configurable: headline_max_age_seconds)
if pub_date and (datetime.now(timezone.utc) - pub_date).total_seconds() > self.config.headline_max_age_seconds:
# Only recent headlines (configurable: headline_max_age_seconds).
# An item without a parsable pubDate cannot be proven
# fresh, so it is dropped too — it used to be stamped
# "now" and never aged out (2026-09-04).
if pub_date is None:
undated += 1
continue
if (datetime.now(timezone.utc) - pub_date).total_seconds() > self.config.headline_max_age_seconds:
continue
# Dedup within and across feeds so a syndicated story
@@ -827,6 +900,8 @@ class NewsEngine:
f"rss:{host}", False, f"{type(e).__name__}: {e}")
logger.debug(f"Failed to fetch RSS from {feed_url}: {e}")
if undated:
logger.debug(f"RSS: dropped {undated} headlines without a publish time")
self._record_source(
"headlines", ok_feeds > 0,
detail="all feeds failed" if ok_feeds == 0 else "",
@@ -986,9 +1061,11 @@ class NewsEngine:
return text or None
def _llm_market_summary(self, context: FundamentalContext) -> str:
"""1-2 sentence macro blurb for alerts; "" when unavailable.
Cached (success OR failure) for the context-cache TTL so at most
one call is made per window."""
"""2-3 sentence plain-language news note for alerts; "" when
unavailable. Headlines go newest-first with the platform's own
gold direction so the note agrees with the score badges. Cached
(success OR failure) for the context-cache TTL so at most one
call is made per window."""
with NewsEngine._cache_lock:
ts, cached = NewsEngine._llm_summary_cache
if time.time() - ts < self.config.context_cache_ttl_seconds:
@@ -999,8 +1076,17 @@ class NewsEngine:
client = get_llm_client()
if not client.available:
return ""
oldest = datetime.min.replace(tzinfo=timezone.utc)
recent = sorted(
context.news_headlines,
key=lambda h: getattr(h, "timestamp", None) or oldest,
reverse=True,
)[:15]
result = client.chat_json(_LLM_SUMMARY_SYSTEM, {
"headlines": [h.headline for h in context.news_headlines[:15]],
"headlines": [
{"headline": h.headline, "gold": _gold_direction_label(h)}
for h in recent
],
"events": [
{"event": e.event, "currency": e.currency,
"actual": e.actual, "forecast": e.forecast,
@@ -1010,7 +1096,7 @@ class NewsEngine:
})
summary = ""
if result:
summary = str(result.get("summary", "")).strip()[:300]
summary = str(result.get("summary", "")).strip()[:400]
with NewsEngine._cache_lock:
NewsEngine._llm_summary_cache = (time.time(), summary)
return summary