harry/ForexBot2
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

news corpus for a local scorer + agent panel v2 + dashboard read-only (2026-09-17); books byte-identical

Browse Source 
Agent panel v2: personas return p_win anchored to the book base rates with a
code-computed decision context; verdict = equal-weight mean (verdict_source
"mean"); only the macro persona runs live (agent_panel.personas ["macro"]);
prompt version and exact payloads logged; offline replay harness
scripts/panel_replay.py; base-rate table agent_panel_base_rates.json
(scripts/build_panel_base_rates.py). Backtester `strategy` column now carries
the generator's own stamp (label-only; five reference books re-pinned, all
62 other columns identical).

News corpus (after evaluating the owner's ML-NEWS scikit model against
DeepSeek: golden set 39/78 vs 78/78, out-of-sample agreement 64% against a
90% self-agreement ceiling; verdict: fallback only, not a replacement):
- NewsSentiment.xau_score + archive column xau_score keep DeepSeek's gold
  score for every headline (930 of the first 1,546 DeepSeek rows had none);
  an existing newsdata.db gains the column in place on first open
- headlines naming no tracked currency are scored and archived
  (currencies_json []) but never returned to the platform
- _apply_llm_sentiment sends every new headline in its cycle, 25 per call;
  a failed call ends the cycle's sending

Dashboard publisher never writes a database: main() calls
app_config.set_db_writes_for_process(False), honoured by signal_log_enabled()
and news_archive_enabled() (owner decision: the bot is the single writer).

Test suite: the hermetic app config now points both writers at a
suite-private temp dir — the RSS fixture tests were writing fixture
headlines into the real logs/newsdata.db — with a Module 32 test pinning it.
529 tests.
This commit is contained in:
Harwinder Sekhon
2026-09-17 19:49:29 -04:00
parent 183df7832b
commit ad14ea4fed
21 changed files with 3803 additions and 259 deletions
Download Patch File Download Diff File Expand all files Collapse all files
news_engine.py
+86 -48
View File
@@ -226,6 +226,15 @@ class NewsSentiment:
# artifact). Without it the two are indistinguishable in the archive
# and a model trained on the mix inherits the keyword scorer's errors.
score_source: str = "keyword"
# DeepSeek's gold-specific score for THIS headline, whatever currencies
# it names. sentiment_by_currency["XAU"] carries it only when the
# headline matched a gold keyword; for a pure Fed / dollar / yields
# headline it used to be discarded, which left 930 of the first 1,546
# archived DeepSeek rows without a gold label (eval 2026-09-17).
# Nothing displayed or averaged reads it — the archive stores it as
# training data for a local scorer. None until DeepSeek has scored
# the headline (the keyword scorer never sets it).
xau_score: Optional[float] = None
@dataclass
@@ -371,6 +380,9 @@ class NewsEngine:
_llm_summary_cache: Tuple[float, str] = (0.0, "")
_LLM_SENT_CACHE_MAX = 400
_LLM_EVENT_CACHE_MAX = 100
#: Headlines per DeepSeek sentiment call — the reply must fit the
#: client's max_tokens (~10 tokens per headline, 2000 max).
_LLM_BATCH_CAP = 25
# Per-source fetch health for status displays (telegram /status).
# name → {"ok": ts|None, "err": ts|None, "detail": str, "extra": str}.
@@ -811,7 +823,8 @@ class NewsEngine:
def _fetch_news_sentiment_raw(self) -> List[NewsSentiment]:
"""Fetch and analyze news headlines for sentiment."""
headlines = []
headlines = [] # names a tracked currency: what the platform uses
unmatched = [] # names none: scored and archived, never shown
seen_titles = set()
ok_feeds = 0
undated = 0
@@ -861,38 +874,48 @@ class NewsEngine:
sentiment = self._analyze_headline_sentiment(title)
currencies = self._extract_currencies(title)
link_el = item.find("link")
desc_el = item.find("description")
src_el = item.find("source")
# itertext() survives CDATA and stray child
# elements where .text stops at the first child.
desc = _clean_html_text(
"".join(desc_el.itertext())
if desc_el is not None else None)
# Google News descriptions are often just the
# headline again inside an anchor — worthless.
if desc and desc.strip() == title.strip():
desc = None
publisher = None
if src_el is not None:
publisher = (src_el.text or "").strip() or None
entry = NewsSentiment(
headline=title,
source=feed_url,
timestamp=pub_date or datetime.now(timezone.utc),
currencies_mentioned=currencies,
sentiment_score=sentiment,
keywords=self._extract_keywords(title),
sentiment_by_currency={
cur: self._analyze_headline_sentiment(title, cur)
for cur in currencies
},
link=_safe_link(
link_el.text if link_el is not None else None),
description=desc,
publisher=publisher,
)
# A headline naming no tracked currency stays invisible
# to the platform (no per-currency mean, no display,
# no panel) but is scored and archived since
# 2026-09-17: it was 23% of a day's feed, and DeepSeek
# gave 10 of those 24 a gold direction — the Iran /
# Hormuz stories were the strongest gold headlines of
# the day. Training data only.
if currencies:
link_el = item.find("link")
desc_el = item.find("description")
src_el = item.find("source")
# itertext() survives CDATA and stray child
# elements where .text stops at the first child.
desc = _clean_html_text(
"".join(desc_el.itertext())
if desc_el is not None else None)
# Google News descriptions are often just the
# headline again inside an anchor — worthless.
if desc and desc.strip() == title.strip():
desc = None
publisher = None
if src_el is not None:
publisher = (src_el.text or "").strip() or None
headlines.append(NewsSentiment(
headline=title,
source=feed_url,
timestamp=pub_date or datetime.now(timezone.utc),
currencies_mentioned=currencies,
sentiment_score=sentiment,
keywords=self._extract_keywords(title),
sentiment_by_currency={
cur: self._analyze_headline_sentiment(title, cur)
for cur in currencies
},
link=_safe_link(
link_el.text if link_el is not None else None),
description=desc,
publisher=publisher,
))
headlines.append(entry)
else:
unmatched.append(entry)
ok_feeds += 1
self._record_source(
@@ -905,14 +928,21 @@ class NewsEngine:
if undated:
logger.debug(f"RSS: dropped {undated} headlines without a publish time")
if unmatched:
logger.debug(f"RSS: {len(unmatched)} headlines name no tracked "
"currency (scored and archived, not shown)")
self._record_source(
"headlines", ok_feeds > 0,
detail="all feeds failed" if ok_feeds == 0 else "",
extra=(f"{len(headlines)} headlines from "
f"{ok_feeds}/{len(self.config.rss_feeds)} feeds"))
# Score and archive EVERYTHING fetched; hand back only the
# currency-matched headlines so display, per-currency means, the
# panel and the summary see exactly what they saw before.
scored = headlines + unmatched
try:
self._apply_llm_sentiment(headlines)
self._apply_llm_sentiment(scored)
except Exception as e:
logger.debug(f"LLM sentiment failed, keyword scores stand: {e}")
@@ -927,7 +957,7 @@ class NewsEngine:
# effect — it must never cost us the headlines we just fetched.
try:
news_archive.archive_headlines(
headlines,
scored,
teacher_model=getattr(get_llm_client(), "_model", None),
teacher_prompt_sha1=_LLM_SENTIMENT_PROMPT_SHA1)
except Exception as e:
@@ -1003,27 +1033,34 @@ class NewsEngine:
"""Overwrite keyword sentiment on `headlines` with LLM scores.
Runs inside _cached_fetch, i.e. UNDER _cache_lock — cache access
is direct and the (rare) HTTP call blocks sibling scans at most
is direct and the (rare) HTTP calls block sibling scans at most
once per cache window, matching the existing shared-fetch design.
Only headlines missing from the cache are sent; a failed batch
changes nothing.
Only headlines missing from the cache are sent, in batches of
_LLM_BATCH_CAP; a failed batch changes nothing and ends the
cycle's sending.
"""
if not headlines or not get_llm_client().available:
return
cache = NewsEngine._llm_sent_cache
keyed = [(hashlib.sha1(h.headline.encode()).hexdigest(), h)
for h in headlines]
# Cap the batch so the reply fits the client's max_tokens — the
# overflow stays keyword-scored this cycle and, still uncached,
# goes out with the next window's batch.
new = [(k, h) for k, h in keyed if k not in cache][:25]
if new:
scores = self._llm_headline_scores([h.headline for _, h in new])
if scores is not None:
for (k, _), sc in zip(new, scores):
cache[k] = sc
while len(cache) > self._LLM_SENT_CACHE_MAX:
cache.pop(next(iter(cache)))
# Every new headline goes out this cycle, one call per batch.
# Until 2026-09-17 only the first 25 went and the overflow stayed
# keyword-scored (31 of the first 1,577 archived headlines never
# got a DeepSeek label that way). A failed call ends the loop: a
# timeout or the daily cap would fail the next one too, and what
# is left stays keyword-scored and, still uncached, goes out with
# the next window.
new = [(k, h) for k, h in keyed if k not in cache]
for start in range(0, len(new), self._LLM_BATCH_CAP):
batch = new[start:start + self._LLM_BATCH_CAP]
scores = self._llm_headline_scores([h.headline for _, h in batch])
if scores is None:
break
for (k, _), sc in zip(batch, scores):
cache[k] = sc
while len(cache) > self._LLM_SENT_CACHE_MAX:
cache.pop(next(iter(cache)))
for k, h in keyed:
sc = cache.get(k)
if sc is None:
@@ -1034,6 +1071,7 @@ class NewsEngine:
cur: (xau if cur == "XAU" else overall)
for cur in h.currencies_mentioned
}
h.xau_score = xau
h.score_source = "llm"
def _llm_event_direction(self, event: EconomicEvent) -> Optional[str]: