From 15a4ed7831f23d507551dc977358bb0cb82a92d3 Mon Sep 17 00:00:00 2001 From: Harwinder Sekhon Date: Fri, 7 Aug 2026 09:08:17 -0400 Subject: [PATCH] =?UTF-8?q?dashboard:=20clickable=20news=20=E2=80=94=20det?= =?UTF-8?q?ail=20dialogs=20+=20on-demand=20AI=20summary=20endpoint?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit - news_engine: RSS parser captures link/description/publisher (three new optional NewsSentiment fields); _clean_html_text/_safe_link helpers - news.json: headlines carry id = sha1(headline) + enrichment fields - static/news_summary.php (NEW): on-demand DeepSeek gold-impact summary, whitelist-gated against news.json ids, cache-forever in ai/, fail-closed file-based daily cap (150/day), key at /etc/gold-dashboard/deepseek.key; installed to the docroot by install_static - dashboard.html: newswire + calendar detail dialogs (score tiles, source description, article link, AI summary with loading/error states); delegated on stable containers, dialog keyed by id across polls - tests: +6 (357 total); docs: README-dashboard §6 one-time setup (key file, apache-owned ai/, SELinux httpd_can_network_connect), CLAUDE.md Co-Authored-By: Claude Fable 5 Claude-Session: https://claude.ai/code/session_01Aq1jZ6jBsBdEC2xT3u46SA --- CLAUDE.md | 4 +- dashboard_publisher.py | 37 ++++-- deploy/README-dashboard.md | 76 ++++++++++-- news_engine.py | 63 ++++++++++ static/dashboard.html | 223 +++++++++++++++++++++++++++++++-- static/news_summary.php | 245 +++++++++++++++++++++++++++++++++++++ test_all.py | 130 ++++++++++++++++++++ 7 files changed, 745 insertions(+), 33 deletions(-) create mode 100644 static/news_summary.php diff --git a/CLAUDE.md b/CLAUDE.md index 5ac033f..85748c9 100644 --- a/CLAUDE.md +++ b/CLAUDE.md @@ -11,7 +11,7 @@ Every numeric threshold in `config.py` and `pair_signals/xauusd.py` was backtest ## Commands ```bash -python test_all.py # QA suite: 338 tests, 21 modules, no network (mocked) +python test_all.py # QA suite: 359 tests, 22 modules, no network (mocked) python main.py --pair XAUUSD --timeframe H1 --detail # CLI signal report (--json for full dump) python telegram_bot.py # the bot (refuses to boot without credentials) python oanda_daemon.py # auto-execution daemon on Oanda practice account @@ -61,7 +61,7 @@ Signal currency everywhere is `SignalStrength` (−2..+2). DataFrames are UTC tz - **`news_engine.py`** — Forex Factory calendar (MyFXBook RSS fallback) + RSS keyword sentiment with gold-inverse keywords. Class-level caches (fresh engine per scan). Optional DeepSeek enrichment via `llm_client.py` (requests-only, no SDK): batched headline sentiment overriding keyword scores, gold-direction on release surprises, and a market-summary blurb (`SignalReport.news_summary`) — **display/warning-only, keyword fallback on any failure, inert without a key**; prompts are static module constants (DeepSeek prefix-cache pricing) and requests send `thinking: disabled` (V4-Flash reasoning otherwise blows the timeout). By XAU-tuned default, event risk and sentiment **warn but do not gate signals** (`event_risk_caps_signals=False`, `sentiment_affects_signals=False`) — the LLM does not change that. - **`telegram_bot.py`** — owner-locked bot + subscriber channel; candle-boundary auto-scan loops (M5/M15/H1/H4, boundary+10 s), alert quality threshold + fingerprint dedup (committed only after a successful send), 📰 LLM market-context line on alerts when available, WebSocket level-breach monitor fed by the streaming provider (`data_providers.streaming`, falling back to the active provider; auto-disabled when the resolved provider can't stream or has no API key — Oanda does not stream, Tiingo does). Rotating log: `logs/telegram.log`, 3-h rotation × 240 backups. - **`oanda_daemon.py` / `oanda_executor.py`** — practice-account auto-execution. Book-parity rules (see daemon docstring): no broker-side TP (trailing model — TP1/TP2 advisory), 5-bar per-TF cooldown (wall-clock approximation, so it expires over weekends where the backtester would still count bars), trail ratchets at bar close once per *observed* bar (a missed/failed cycle skips those bars entirely — the intra-gap high/low never enters the peak). Accepted divergence: one position total; orders send `positionFill=OPEN_ONLY` so a state/broker divergence cancels the order instead of silently netting. State/broker re-sync runs at boot and after any suspicious fill (`_reconcile_positions`). -- **`dashboard_publisher.py` / `static/dashboard.html`** — web dashboard as a **static publisher** (live LAMP server): standalone daemon scans at candle closes via `produce_signal` and atomically writes `snapshot.json`/`news.json`/`dashboard.html` to `$DASHBOARD_OUT_DIR` (Apache serves them; page is public by owner decision). **Hard independence rule: never imports, and is never imported by, `telegram_bot`/`oanda_daemon`** — confirmation map and boundary math are deliberate local copies, so a web-side failure can only mean a stale page. Due-TF classification uses the RAW wake time (flooring wake−buffer skips every H1/H4 scan — 2026-08-02 bug). News panel reads `NewsEngine.get_fundamental_context()` plus the additive `get_week_events()` accessor (display-only; no signal path calls it); as a separate process it has its own news caches and DeepSeek daily cap. Deployment checklist: `deploy/README-dashboard.md`. +- **`dashboard_publisher.py` / `static/dashboard.html`** — web dashboard as a **static publisher** (live LAMP server): standalone daemon scans at candle closes via `produce_signal` and atomically writes `snapshot.json`/`news.json`/`dashboard.html` to `$DASHBOARD_OUT_DIR` (Apache serves them; page is public by owner decision). **Hard independence rule: never imports, and is never imported by, `telegram_bot`/`oanda_daemon`** — confirmation map and boundary math are deliberate local copies, so a web-side failure can only mean a stale page. Due-TF classification uses the RAW wake time (flooring wake−buffer skips every H1/H4 scan — 2026-08-02 bug). News panel reads `NewsEngine.get_fundamental_context()` plus the additive `get_week_events()` accessor (display-only; no signal path calls it); as a separate process it has its own news caches and DeepSeek daily cap. News/calendar rows are clickable (detail dialog); headlines carry `id` = sha1(headline) in news.json, and exactly one on-demand endpoint exists: `static/news_summary.php` (installed by `install_static`), which serves a DeepSeek gold-impact summary per headline — whitelist-gated against news.json ids, cache-forever in `ai/`, own fail-closed daily cap (150/day, file-based, separate from the publisher's), key at `/etc/gold-dashboard/deepseek.key`, never reads repo config. Deployment checklist: `deploy/README-dashboard.md`. - **`backtest/gpu_backtest.py`** — bar-by-bar replay through the real `analyze_from_dataframes` (news off). Realism model (do not regress): next-bar-open entries re-anchored to the raw price path, round-trip `--spread` charged per trade, trade resolution on RAW bars (cleaning applies only to signal-side data), same-bar SL+TP tie → SL first, gap fills at bar open, trail ratchet at bar close only. Indicators are precomputed once per TF (CuPy GPU or NumPy fallback — check the `Backend:` log line); replay fans out over worker processes and the 5-bar cooldown is applied in an ordered pass afterwards, so parallel and sequential books are identical. Parquet cache in `backtest/cache/` keyed by provider/symbol/TF/window. `backtest/*.csv` are gitignored pinned trade books from past campaigns — used for manual before/after comparison. `backtest/instructions_for_backtest.txt` lists what every backtest report must record. ## Rules for making changes (read before editing signal logic) diff --git a/dashboard_publisher.py b/dashboard_publisher.py index 20b4cd4..1bce95d 100644 --- a/dashboard_publisher.py +++ b/dashboard_publisher.py @@ -37,6 +37,7 @@ Out dir precedence: --out → DASHBOARD_OUT_DIR env → /var/www/trade import argparse import json +import hashlib import logging import math import numbers @@ -76,6 +77,11 @@ SCAN_BUFFER_SECONDS = 10 SNAPSHOT_FILE = "snapshot.json" NEWS_FILE = "news.json" PAGE_FILE = "dashboard.html" +SUMMARY_PHP_FILE = "news_summary.php" +# Repo files copied into the out dir at daemon start (git pull + restart +# updates them). The PHP endpoint reads news.json in place, so it lives +# next to the page. +STATIC_FILES = [PAGE_FILE, SUMMARY_PHP_FILE] # ══════════════════════════════════════════════════════════════════ @@ -180,9 +186,15 @@ def event_to_dict(ev) -> dict: def headline_to_dict(h) -> dict: + # id uses the same sha1(headline) scheme as NewsEngine._llm_sent_cache; + # it is the join key the PHP summary endpoint whitelists against. return { + "id": hashlib.sha1(h.headline.encode("utf-8")).hexdigest(), "headline": h.headline, "source": h.source, + "link": h.link, + "description": h.description, + "publisher": h.publisher, "timestamp": h.timestamp.astimezone(timezone.utc).isoformat() if h.timestamp else None, "currencies": list(h.currencies_mentioned), @@ -324,16 +336,21 @@ class DashboardPublisher: return self._news_engine def install_static(self) -> None: - """Copy the repo's dashboard.html into the out dir (atomic), so - git pull + restart also updates the page.""" - src = os.path.join(os.path.dirname(os.path.abspath(__file__)), - "static", PAGE_FILE) - if not os.path.exists(src): - logger.warning(f"{src} missing — page not installed") - return - dst = os.path.join(self.out_dir, PAGE_FILE) - shutil.copyfile(src, dst + ".tmp") - os.replace(dst + ".tmp", dst) + """Copy the repo's static files (page + PHP summary endpoint) + into the out dir (atomic), so git pull + restart also updates + them. The ai/ cache dir is NOT created here — the daemon runs as + a different user than Apache; that dir is a one-time deploy step + (deploy/README-dashboard.md).""" + static_dir = os.path.join( + os.path.dirname(os.path.abspath(__file__)), "static") + for name in STATIC_FILES: + src = os.path.join(static_dir, name) + if not os.path.exists(src): + logger.warning(f"{src} missing — not installed") + continue + dst = os.path.join(self.out_dir, name) + shutil.copyfile(src, dst + ".tmp") + os.replace(dst + ".tmp", dst) def scan_tf(self, tf: Timeframe) -> None: started = time.time() diff --git a/deploy/README-dashboard.md b/deploy/README-dashboard.md index 7107fe0..9068238 100644 --- a/deploy/README-dashboard.md +++ b/deploy/README-dashboard.md @@ -2,10 +2,12 @@ The dashboard is a **static publisher**: `dashboard_publisher.py` runs as its own daemon, scans XAU/USD at candle closes, and atomically writes -`dashboard.html` + `snapshot.json` + `news.json` into a folder Apache already -serves. Apache does all web serving — no new port, no Python web server, and -the telegram bot process is completely unaffected by anything that happens -here (see the module docstring for the independence rules). +`dashboard.html` + `news_summary.php` + `snapshot.json` + `news.json` into a +folder Apache already serves. Apache does all web serving — no new port, no +Python web server, and the telegram bot process is completely unaffected by +anything that happens here (see the module docstring for the independence +rules). The one dynamic piece is `news_summary.php` (§6): an on-demand +DeepSeek summary for a clicked headline, whitelist-gated against news.json. All steps below run **on the live server** (`/home/Forex/ForexBot2`). @@ -22,9 +24,10 @@ No new Python dependencies are required. ## 2. Make the web folder writable The dashboard publishes straight into the portal docroot `/var/www/trade/` -(owner decision 2026-08-03) — it adds exactly three files there: -`dashboard.html`, `snapshot.json`, `news.json`. The daemon user must be able -to write them: +(owner decision 2026-08-03) — it adds exactly four files there: +`dashboard.html`, `news_summary.php`, `snapshot.json`, `news.json` (plus the +apache-owned `ai/` cache dir from §6). The daemon user must be able to write +them: ```bash # if the portal files can be owned by the daemon user: @@ -72,8 +75,9 @@ every 3 h (240 backups), same scheme as the bot. ## 5. Optional Apache polish -The page cache-busts its JSON fetches (`?t=`), so no Apache config is -strictly needed. To be explicit anyway: +The page polls its JSON with conditional GETs (`cache: "no-cache"` → Apache +answers unchanged files with 304), so no Apache config is strictly needed. To +be explicit anyway: ```apache @@ -87,15 +91,67 @@ The page is deliberately public (owner decision 2026-08-03). To restrict it later, add standard Apache basic auth (`htpasswd` + `Require valid-user`) on that directory — no code changes needed. +## 6. AI summary endpoint (PHP) — one-time setup + +Clicking a newswire headline opens a detail dialog that fetches an on-demand +DeepSeek gold-impact summary from `news_summary.php` (installed next to the +page by the daemon). Owner decision 2026-08-07: a real on-demand endpoint on +the public page, accepted with these guards baked into the PHP — the id must +match a headline in the published news.json (arbitrary text can never reach +the LLM), each headline's summary is cached forever in `ai/` (≤1 API call per +headline, ever), and a file-based daily cap fails **closed** on I/O errors. + +Requirements: `php` + the curl extension (`php -m | grep curl`). + +```bash +# 1. SELinux: allow httpd outbound HTTPS — WITHOUT this every summary call +# dies as a curl timeout even with perfect perms (RHEL-family default). +sudo setsebool -P httpd_can_network_connect 1 + +# 2. API key OUTSIDE the docroot, readable by Apache only: +sudo install -d -o root -g root -m 755 /etc/gold-dashboard +sudo sh -c 'umask 137; echo "sk-..." > /etc/gold-dashboard/deepseek.key' +sudo chown root:apache /etc/gold-dashboard/deepseek.key # mode 640 + +# 3. apache-owned cache dir (the daemon runs as Forex and can't create it): +sudo install -d -o apache -g apache -m 755 /var/www/trade/ai +sudo semanage fcontext -a -t httpd_sys_rw_content_t '/var/www/trade/ai(/.*)?' +sudo restorecon -Rv /var/www/trade/ai +``` + +Knobs (Apache `SetEnv` in the vhost, both optional): +`AI_SUMMARY_DAILY_CAP` (default 150), `DEEPSEEK_KEY_FILE` (default +`/etc/gold-dashboard/deepseek.key`). + +Verify from the shell — expect 400 / 404 / 200 / 200-from-cache: + +```bash +curl -s "http:///news_summary.php?id=zzz" # {"error":"bad_id"} +curl -s "http:///news_summary.php?id=$(printf 'a%.0s' {1..40})" # unknown_id +ID=$(python3 -c "import json;print(json.load(open('/var/www/trade/news.json'))['headlines'][0]['id'])") +curl -s "http:///news_summary.php?id=$ID" # real summary (2-12 s) +curl -s "http:///news_summary.php?id=$ID" # instant, from ai/.json +``` + ## Operational notes - **Updating**: `git pull` + `systemctl restart gold-dashboard`. The daemon - re-copies `static/dashboard.html` into the web folder at startup. + re-copies `static/dashboard.html` and `static/news_summary.php` into the + web folder at startup. - **If the daemon dies**, systemd restarts it in 30 s; worst case the page shows stale data with its own "updated N min ago" evidence. Apache and the telegram bot are unaffected. - **DeepSeek**: the publisher process has its own daily request cap (`deepseek.daily_request_cap`, default 500) separate from the bot's — expect roughly up to ~430 extra calls/day worst case, V4-Flash pricing. +- **Two independent DeepSeek meters** hit the same API key: the publisher's + in-process cap above, and the PHP endpoint's file cap (`ai/.ht_cap.json`, + default 150/day). They don't communicate — worst-case spend is their sum; + DeepSeek-side spend alerting is the true backstop. Kept separate on + purpose (independence rule: the PHP never imports/reads bot code or + config.json). +- **Restart window**: for the first minutes after a daemon restart the new + page can meet a pre-upgrade news.json without headline ids — those rows + simply render non-clickable until the first news publish. - **Credentials**: uses the same env/.env/config.json resolution as the rest of the platform (Oanda candles feed; news needs no key; DeepSeek optional). diff --git a/news_engine.py b/news_engine.py index ab495de..7dc7498 100755 --- a/news_engine.py +++ b/news_engine.py @@ -9,6 +9,7 @@ A perfect technical setup can be destroyed by an unexpected NFP print. """ import hashlib +import html import logging import re import threading @@ -103,6 +104,44 @@ def _parse_release_value(raw: Optional[str]) -> Optional[float]: return float(match.group(1)) * multiplier +_DESC_MAX_CHARS = 500 +_MAX_LINK_CHARS = 2048 + + +def _clean_html_text(raw: Optional[str], + limit: int = _DESC_MAX_CHARS) -> Optional[str]: + """RSS → plain text: unescape entities, strip tags, + collapse whitespace, truncate. Unescape and strip both run twice + because feeds routinely double-encode (&lt;p&gt; wrapping + real markup) — the second unescape can reveal fresh tags.""" + if not raw: + return None + text = html.unescape(raw) + text = re.sub(r"<[^>]+>", " ", text) + text = html.unescape(text) + text = re.sub(r"<[^>]+>", " ", text) + text = re.sub(r"\s+", " ", text).strip() + if not text: + return None + if len(text) > limit: + text = text[:limit].rstrip() + "…" + return text + + +def _safe_link(raw: Optional[str]) -> Optional[str]: + """Article URL for display: http(s) only (an RSS is untrusted + and lands in an href), bounded length (Google News redirect URLs run + 1-2 KB; anything past that is bloat, not a link).""" + if not raw: + return None + link = raw.strip() + if not link.lower().startswith(("http://", "https://")): + return None + if len(link) > _MAX_LINK_CHARS: + return None + return link + + @dataclass class EconomicEvent: """Single economic calendar event.""" @@ -155,6 +194,11 @@ class NewsSentiment: # Per-currency score: monetary-policy keywords flip sign for XAU # (hawkish is dollar-bullish but gold-bearish). sentiment_by_currency: Dict[str, float] = field(default_factory=dict) + # Display-only enrichment (dashboard detail dialog). Optional with + # defaults so existing constructor calls keep working. + link: Optional[str] = None # article URL (Google News = redirect) + description: Optional[str] = None # plain text, tags stripped, ≤500 chars + publisher: Optional[str] = None # Google News text, else None @dataclass @@ -742,6 +786,21 @@ class NewsEngine: currencies = self._extract_currencies(title) if currencies: + link_el = item.find("link") + desc_el = item.find("description") + src_el = item.find("source") + # itertext() survives CDATA and stray child + # elements where .text stops at the first child. + desc = _clean_html_text( + "".join(desc_el.itertext()) + if desc_el is not None else None) + # Google News descriptions are often just the + # headline again inside an anchor — worthless. + if desc and desc.strip() == title.strip(): + desc = None + publisher = None + if src_el is not None: + publisher = (src_el.text or "").strip() or None headlines.append(NewsSentiment( headline=title, source=feed_url, @@ -753,6 +812,10 @@ class NewsEngine: cur: self._analyze_headline_sentiment(title, cur) for cur in currencies }, + link=_safe_link( + link_el.text if link_el is not None else None), + description=desc, + publisher=publisher, )) ok_feeds += 1 diff --git a/static/dashboard.html b/static/dashboard.html index b94d8b6..7d4e4d9 100644 --- a/static/dashboard.html +++ b/static/dashboard.html @@ -322,8 +322,8 @@ details.more .reasons { margin-top: 4px; } .rail-chip { cursor: pointer; } .rail-chip:hover, .rail-chip:focus-visible { border-color: var(--accent); } -/* ── Timeframe detail dialog ──────────────────────────────────── */ -#tf-detail { +/* ── Detail dialogs (timeframe + news) ────────────────────────── */ +#tf-detail, #news-detail { /* the global `* { margin: 0 }` reset wipes the UA stylesheet's `margin: auto`, which is what centres a modal dialog — restore it explicitly. */ position: fixed; inset: 0; margin: auto; @@ -333,9 +333,10 @@ details.more .reasons { margin-top: 4px; } background: var(--card); color: var(--ink); font-family: var(--sans); font-size: 16px; } -#tf-detail::backdrop { background: rgba(var(--ink-rgb),0.5); } -#tf-detail[open] { display: flex; flex-direction: column; } -#detail-head { +#news-detail { width: min(680px, 94vw); } +#tf-detail::backdrop, #news-detail::backdrop { background: rgba(var(--ink-rgb),0.5); } +#tf-detail[open], #news-detail[open] { display: flex; flex-direction: column; } +#detail-head, #nd-head { flex: 0 0 auto; display: flex; align-items: center; gap: 10px; flex-wrap: wrap; padding: 14px 18px; background: var(--card2); @@ -347,13 +348,17 @@ details.more .reasons { margin-top: 4px; } } #detail-head .d-price { font-family: var(--mono); font-size: 19px; font-weight: 600; } #detail-head .d-sub { font-size: 13px; color: var(--muted); } -#detail-close { +#detail-close, #nd-close { margin-left: auto; font: inherit; font-size: 15px; line-height: 1; padding: 7px 13px; border-radius: 6px; cursor: pointer; background: var(--card); color: var(--ink2); border: 1px solid var(--line-strong); } -#detail-close:hover { border-color: var(--accent); color: var(--accent-bright); } -#detail-body { flex: 1 1 auto; min-height: 0; overflow-y: auto; padding: 4px 18px 20px; } +#detail-close:hover, #nd-close:hover { border-color: var(--accent); color: var(--accent-bright); } +#detail-body, #nd-body { flex: 1 1 auto; min-height: 0; overflow-y: auto; padding: 4px 18px 20px; } +#nd-title { + font-size: 12px; letter-spacing: 0.12em; text-transform: uppercase; + color: var(--muted); font-weight: 600; +} .d-sec { margin-top: 20px; } .d-sec > h3 { @@ -435,6 +440,12 @@ details.more .reasons { margin-top: 4px; } .d-note b { color: var(--ink); } .sub-h { font-size: 12px; color: var(--muted); font-weight: 600; margin: 11px 0 5px; } +/* news detail dialog */ +.nd-headline { font-size: 18px; font-weight: 650; line-height: 1.35; margin-top: 14px; } +.nd-meta { font-size: 12.5px; color: var(--muted); margin-top: 5px; } +a.d-action { display: block; text-decoration: none; color: var(--accent-bright); } +a.d-action:hover { border-color: var(--accent); } + /* ── News section ─────────────────────────────────────────────── */ #news-wrap { display: grid; gap: 14px; grid-template-columns: minmax(0, 3fr) minmax(0, 2fr); } @media (max-width: 1100px) { #news-wrap { grid-template-columns: 1fr; } } @@ -467,6 +478,8 @@ details.more .reasons { margin-top: 4px; } } #cal-table tr.past { opacity: 0.55; } #cal-table tr.next-up td:first-child { box-shadow: inset 3px 0 0 var(--accent); padding-left: 7px; } +#cal-table tr[data-ev-key] { cursor: pointer; } +#cal-table tr[data-ev-key]:hover td { background: var(--accent-wash); } #cal-table .t { white-space: nowrap; color: var(--ink2); } #cal-table .cur { font-family: var(--mono); color: var(--muted); } #cal-table .ev { color: var(--ink); } @@ -514,6 +527,8 @@ details.more .reasons { margin-top: 4px; } padding: 7px 0; border-bottom: 1px solid var(--line); } .wire-item:last-child { border-bottom: none; } +.wire-item[data-news-id] { cursor: pointer; padding: 7px 6px; margin: 0 -6px; border-radius: 6px; } +.wire-item[data-news-id]:hover { background: var(--accent-wash); } .wire-score { font-family: var(--mono); font-size: 13px; font-weight: 700; padding: 2px 7px; border-radius: 5px; white-space: nowrap; @@ -569,6 +584,14 @@ footer {
+ +
+ News detail + +
+
+ +
Fundamentals
@@ -1056,7 +1079,8 @@ function renderCalendar() { ? `${ e.surprise === "positive" ? "▲" : e.surprise === "negative" ? "▼" : "="}` : ""; - rows += ` + rows += ` ${esc(time)} ${esc(e.currency)} ${e.impact === "medium" ? "Med" : @@ -1109,11 +1133,14 @@ function renderWire() { const scoreTxt = s == null ? "·" : (s > 0 ? "+" : "") + s.toFixed(1); let host = ""; try { host = new URL(h.source).hostname.replace(/^www\./, ""); } catch {} - return `
+ /* data-news-id only when the payload carries an id — after a daemon + restart the new page briefly meets a pre-upgrade news.json */ + const open = h.id ? ` data-news-id="${h.id}" tabindex="0" role="button" aria-haspopup="dialog"` : ""; + return `
${scoreTxt} ${esc(h.headline)} - ${esc(host)}${h.timestamp ? " · " + fmtAge(h.timestamp) : ""}
`; + ${esc(h.publisher || host)}${h.timestamp ? " · " + fmtAge(h.timestamp) : ""}
`; }; const head = items.slice(0, 12).map(item).join(""); const rest = items.slice(12); @@ -1122,6 +1149,179 @@ function renderWire() {
${rest.map(item).join("")}
` : ""); } +/* ── news / calendar detail dialog ────────────────────────────── */ +const ndlg = $("news-detail"); +let ndState = null; // {kind:"wire", id, snap} | {kind:"cal", key, snap} +const aiMemo = new Map(); // headline id -> {summary, generated_at} (successes only) +const aiErr = new Map(); // headline id -> error code (cleared on reopen → retry) +const aiPending = new Set(); + +/* stable across polls — the array index is not (new headlines shift it) */ +const evKey = e => `${e.time}|${e.currency}|${e.event}`; +const findHeadline = id => (news?.headlines || []).find(h => h.id === id); +const findEvent = key => (news?.events || []).find(e => evKey(e) === key); + +const AI_ERR_MSG = { + unknown_id: "This headline has left the live feed — no summary available.", + cap_exceeded: "Daily AI budget reached — try again tomorrow.", + no_key: "AI summaries are not configured on this server.", +}; + +async function fetchAiSummary(id) { + aiPending.add(id); + try { + const res = await fetch("news_summary.php?id=" + encodeURIComponent(id)); + const data = await res.json().catch(() => null); + if (res.ok && data?.summary) aiMemo.set(id, data); + else aiErr.set(id, data?.error || `http_${res.status}`); + } catch { + aiErr.set(id, "network"); + } finally { + aiPending.delete(id); + if (ndState?.kind === "wire" && ndState.id === id) renderNewsDlg(); + } +} + +function scoreTiles(h) { + const cls = v => v > 0.05 ? "buy" : v < -0.05 ? "sell" : ""; + let tiles = tile("Overall sentiment", signed(h.score, 2), cls(h.score)); + const xau = h.score_by_currency?.XAU; + if (xau != null) tiles += tile("Gold (XAU) effect", signed(xau, 2), cls(xau)); + for (const [cur, v] of Object.entries(h.score_by_currency || {})) { + if (cur !== "XAU") tiles += tile(cur, signed(v, 2), cls(v)); + } + let out = `
${tiles}
`; + if (h.keywords?.length) { + out += `
Matched keywords: ${esc(h.keywords.join(", "))}
`; + } + out += `
Scores range −1 (bearish) to +1 (bullish); the gold + score flips monetary-policy keywords (hawkish = dollar-bullish, gold-bearish).
`; + return out; +} + +function aiSection(id) { + if (aiMemo.has(id)) { + const m = aiMemo.get(id); + return `
${esc(m.summary)}
` + (m.generated_at + ? `
generated ${fmtAge(m.generated_at)}
` : ""); + } + if (aiPending.has(id)) return `
Generating gold-impact summary…
`; + if (aiErr.has(id)) { + return `
${esc(AI_ERR_MSG[aiErr.get(id)] || "AI summary unavailable right now.")}
`; + } + return `
Generating gold-impact summary…
`; +} + +function wireDlgHTML() { + const fresh = findHeadline(ndState.id); + const h = fresh || ndState.snap; + if (!h) return ""; + let host = ""; + try { host = new URL(h.source).hostname.replace(/^www\./, ""); } catch {} + const when = h.timestamp + ? `${new Date(h.timestamp).toLocaleString(undefined, + {dateStyle: "medium", timeStyle: "short"})} · ${fmtAge(h.timestamp)}` : ""; + let out = `
${esc(h.headline)}
+
${esc(h.publisher || host)}${when ? " · " + when : ""}
`; + if (!fresh) out += `
No longer in the live feed — showing saved copy.
`; + out += sec("Score", scoreTiles(h)); + if (h.description) out += sec("From the source", `
${esc(h.description)}
`); + out += sec("AI gold-impact summary", aiSection(ndState.id)); + if (h.link && /^https?:\/\//.test(h.link)) { + out += `Read full article ↗`; + } + return out; +} + +function calDlgHTML() { + const fresh = findEvent(ndState.key); + const e = fresh || ndState.snap; + if (!e) return ""; + const t = Date.parse(e.time) / 1000; + const when = new Date(e.time).toLocaleString(undefined, {dateStyle: "full", timeStyle: "short"}); + const impact = `${e.impact === "medium" ? "Med" + : e.impact.charAt(0).toUpperCase() + e.impact.slice(1)}`; + let out = `
${esc(e.event)}
+
${esc(e.currency)} · ${when}
`; + /* live countdown only for FUTURE events — tick()'s modulo wrap on a + passed epoch would show a bogus restarting timer for released ones */ + const timing = t > nowServer() + ? `releases in --:--` + : `released ${fmtAge(e.time)}`; + out += sec("Timing", `
${impact} impact · ${timing}
`); + const surpTxt = {positive: "actual above forecast", negative: "actual below forecast", + inline: "actual matches forecast"}[e.surprise]; + out += sec("Numbers", `
${ + tile("Forecast", esc(e.forecast ?? "—")) + + tile("Previous", esc(e.previous ?? "—")) + + tile("Actual", esc(e.actual ?? "—"), + e.surprise === "positive" ? "buy" : e.surprise === "negative" ? "sell" : "") + }
` + (surpTxt + ? `
${surpTxt} (numeric only — not gold direction)
` : "")); + return out; +} + +function renderNewsDlg() { + if (!ndState || !ndlg.open) return; + const scroll = $("nd-body").scrollTop; + $("nd-title").textContent = ndState.kind === "wire" ? "Newswire" : "Economic calendar"; + $("nd-body").innerHTML = ndState.kind === "wire" ? wireDlgHTML() : calDlgHTML(); + $("nd-body").scrollTop = scroll; + tick(); +} + +function openNewsDetail(id) { + const h = findHeadline(id); + if (!h) return; + ndState = {kind: "wire", id, snap: h}; + aiErr.delete(id); // failures retry on reopen; successes are memoised + if (!ndlg.open) ndlg.showModal(); + if (!aiMemo.has(id) && !aiPending.has(id)) fetchAiSummary(id); + renderNewsDlg(); + $("nd-body").scrollTop = 0; + $("nd-close").focus(); +} + +function openCalDetail(key) { + const e = findEvent(key); + if (!e) return; + ndState = {kind: "cal", key, snap: e}; + if (!ndlg.open) ndlg.showModal(); + renderNewsDlg(); + $("nd-body").scrollTop = 0; + $("nd-close").focus(); +} + +$("nd-close").addEventListener("click", () => ndlg.close()); +ndlg.addEventListener("close", () => { ndState = null; }); +ndlg.addEventListener("click", e => { + if (e.target !== ndlg) return; + const b = ndlg.getBoundingClientRect(); + if (e.clientX < b.left || e.clientX > b.right || e.clientY < b.top || e.clientY > b.bottom) ndlg.close(); +}); + +/* Delegated on the stable containers (renderWire/renderCalendar replace + innerHTML every poll). Deliberately NOT wireOpeners(): its details/ + summary guard would swallow the 28 wire items nested in the + "more headlines"
. A click on that toggle itself finds no + [data-news-id] ancestor and falls through to the native behaviour. */ +function wireNewsOpeners(host, attr, open) { + host.addEventListener("click", e => { + const el = e.target.closest(`[${attr}]`); + if (el) open(el.getAttribute(attr)); + }); + host.addEventListener("keydown", e => { + if (e.key !== "Enter" && e.key !== " ") return; + const el = e.target.closest?.(`[${attr}]`); + if (!el || e.target !== el) return; // let inner controls keep their own keys + e.preventDefault(); + open(el.getAttribute(attr)); + }); +} +wireNewsOpeners($("wire"), "data-news-id", openNewsDetail); +wireNewsOpeners($("cal-body"), "data-ev-key", openCalDetail); + /* ── polling ──────────────────────────────────────────────────── */ async function fetchJSON(name) { /* no-cache = revalidate with the server every time (If-Modified-Since); @@ -1150,6 +1350,7 @@ async function poll() { $("banner-fail").style.display = "none"; renderHero(); renderContext(); renderCards(); renderCalendar(); renderWire(); renderDetail(); // keep an open detail view in sync with the new scan + renderNewsDlg(); // same for an open news/calendar dialog tick(); // countdowns appear immediately, not after the first 1 s tick /* one-shot refetch shortly after the next expected publish, so a diff --git a/static/news_summary.php b/static/news_summary.php new file mode 100644 index 0000000..3cb9355 --- /dev/null +++ b/static/news_summary.php @@ -0,0 +1,245 @@ + and returns {"summary": ..., "generated_at": ...}. + * + * Abuse model (the page is public): + * - WHITELIST: the id must match a headline in the local news.json — + * arbitrary text can never reach the LLM through this endpoint. + * - CACHE FOREVER: headlines are immutable, so each id costs at most + * one DeepSeek call, ever (ai/.json). Cache is checked BEFORE + * the whitelist on purpose: summaries stay reachable after the + * headline ages out of the 24 h feed window. + * - DAILY CAP: file-based UTC-day counter, FAIL CLOSED on counter I/O + * errors (a perms mistake must not mean unmetered spend). + * + * DeepSeek doctrine (mirrors llm_client.py — deliberate local copy, the + * dashboard never imports bot code and this file never reads the repo's + * config.json): + * - STATIC byte-identical system prompt (prefix-cache pricing). + * - "thinking": disabled is MANDATORY for V4-Flash (else it burns the + * timeout reasoning), temperature 0, json response_format. + * - One attempt, no retries; every failure returns a JSON error the + * page renders as a graceful fallback. + * + * Key file lives OUTSIDE the docroot: /etc/gold-dashboard/deepseek.key + * (root:apache 640). Override paths/caps via Apache SetEnv: + * DEEPSEEK_KEY_FILE, AI_SUMMARY_DAILY_CAP. + */ + +const NEWS_FILE = __DIR__ . '/news.json'; +const CACHE_DIR = __DIR__ . '/ai'; // apache-owned; created at deploy time +const API_URL = 'https://api.deepseek.com/chat/completions'; +const MODEL = 'deepseek-v4-flash'; +const TIMEOUT = 12; +const MAX_TOKENS = 200; +const SUMMARY_MAX_CHARS = 400; + +// .ht_ prefix: Apache's default rule denies these even +// though they sit in the public docroot. +const CAP_FILE = CACHE_DIR . '/.ht_cap.json'; + +const SYSTEM_PROMPT = <<<'PROMPT' +You are a forex/gold news analyst. You receive one news item as JSON: +{"headline", "description", "score", "xau_score", "keywords", "currencies"}. +"xau_score" is the platform's automated gold-effect score for this item, +-1.0 (gold down) to 1.0 (gold up); "score" is overall market sentiment. +Both are automated and can be wrong — judge the news yourself. +In 1-2 plain-English sentences (max 45 words) explain what this news +means for the GOLD (XAU/USD) price and WHY it is bullish, bearish, or +neutral. If your reading clearly contradicts xau_score, add "(the +automated score disagrees)". Never write the words "xau_score" or +"score" — a general reader sees this text. +Gold doctrine: hawkish Fed / rate hikes / rising real yields / strong +dollar are gold-bearish; easing / rate cuts / safe-haven flows / +geopolitical stress / weak dollar are gold-bullish. +Reply with ONLY JSON: {"summary": "..."} +PROMPT; + +function respond(int $status, array $obj, string $cache = 'no-store'): void +{ + http_response_code($status); + header('Content-Type: application/json; charset=utf-8'); + header('Cache-Control: ' . $cache); + echo json_encode($obj, JSON_UNESCAPED_SLASHES | JSON_UNESCAPED_UNICODE); + exit; +} + +function atomic_write(string $path, string $data): bool +{ + $tmp = $path . '.tmp.' . getmypid(); + if (@file_put_contents($tmp, $data) === false) { + return false; + } + if (!@rename($tmp, $path)) { + @unlink($tmp); + return false; + } + return true; +} + +// ── 1. id validation ──────────────────────────────────────────────── +$id = $_GET['id'] ?? ''; +if (!preg_match('/^[a-f0-9]{40}$/', $id)) { + respond(400, ['error' => 'bad_id']); +} + +// ── 2. cache first (immutable headlines → summaries never change) ─── +$cachePath = CACHE_DIR . '/' . $id . '.json'; +$cached = @file_get_contents($cachePath); +if ($cached !== false && $cached !== '') { + http_response_code(200); + header('Content-Type: application/json; charset=utf-8'); + header('Cache-Control: max-age=3600'); + echo $cached; + exit; +} + +// ── 3. whitelist against the published feed ───────────────────────── +$news = json_decode((string)@file_get_contents(NEWS_FILE), true); +if (!is_array($news)) { + respond(503, ['error' => 'no_feed']); +} +$hit = null; +foreach ($news['headlines'] ?? [] as $h) { + // Fallback sha1(): a publisher that predates the id field still + // serves a valid whitelist during the restart window. + $hid = $h['id'] ?? sha1($h['headline'] ?? ''); + if ($hid === $id) { + $hit = $h; + break; + } +} +if ($hit === null) { + respond(404, ['error' => 'unknown_id', + 'message' => 'Headline is no longer in the feed.']); +} + +// ── 4. daily cap (fail closed) ────────────────────────────────────── +$dailyCap = (int)(getenv('AI_SUMMARY_DAILY_CAP') ?: 150); +$capFh = @fopen(CAP_FILE, 'c+'); +if ($capFh === false || !flock($capFh, LOCK_EX)) { + respond(503, ['error' => 'cap_io']); +} +$today = gmdate('Y-m-d'); +$cap = json_decode((string)stream_get_contents($capFh), true); +if (!is_array($cap) || ($cap['day'] ?? '') !== $today) { + $cap = ['day' => $today, 'count' => 0]; +} +if ($cap['count'] >= $dailyCap) { + flock($capFh, LOCK_UN); + fclose($capFh); + respond(429, ['error' => 'cap_exceeded']); +} +// Increment BEFORE the LLM call — attempts count against the cap +// (same semantics as llm_client._under_cap). +$cap['count']++; +rewind($capFh); +if (ftruncate($capFh, 0) === false + || fwrite($capFh, json_encode($cap)) === false) { + flock($capFh, LOCK_UN); + fclose($capFh); + respond(503, ['error' => 'cap_io']); +} +fflush($capFh); +flock($capFh, LOCK_UN); +fclose($capFh); + +// ── 5. API key ────────────────────────────────────────────────────── +$keyFile = getenv('DEEPSEEK_KEY_FILE') ?: '/etc/gold-dashboard/deepseek.key'; +$apiKey = trim((string)@file_get_contents($keyFile)); +if ($apiKey === '') { + respond(503, ['error' => 'no_key']); +} + +// ── 6. per-id stampede lock, then re-check the cache ──────────────── +$lockFh = @fopen(CACHE_DIR . '/.ht_lock_' . $id, 'c'); +if ($lockFh !== false) { + flock($lockFh, LOCK_EX); // blocks while a concurrent winner generates + $cached = @file_get_contents($cachePath); + if ($cached !== false && $cached !== '') { + flock($lockFh, LOCK_UN); + fclose($lockFh); + http_response_code(200); + header('Content-Type: application/json; charset=utf-8'); + header('Cache-Control: max-age=3600'); + echo $cached; + exit; + } +} + +// ── 7. DeepSeek call (exact chat_json body shape) ─────────────────── +$userPayload = json_encode([ + 'headline' => $hit['headline'] ?? '', + 'description' => $hit['description'] ?? null, + 'score' => $hit['score'] ?? null, + 'xau_score' => $hit['score_by_currency']['XAU'] ?? null, + 'keywords' => $hit['keywords'] ?? [], + 'currencies' => $hit['currencies'] ?? [], +], JSON_UNESCAPED_UNICODE); + +$body = json_encode([ + 'model' => MODEL, + 'messages' => [ + ['role' => 'system', 'content' => SYSTEM_PROMPT], + ['role' => 'user', 'content' => $userPayload], + ], + 'response_format' => ['type' => 'json_object'], + 'temperature' => 0, + 'thinking' => ['type' => 'disabled'], + 'max_tokens' => MAX_TOKENS, +]); + +$ch = curl_init(API_URL); +curl_setopt_array($ch, [ + CURLOPT_POST => true, + CURLOPT_POSTFIELDS => $body, + CURLOPT_HTTPHEADER => ['Content-Type: application/json', + 'Authorization: Bearer ' . $apiKey], + CURLOPT_RETURNTRANSFER => true, + CURLOPT_TIMEOUT => TIMEOUT, +]); +$raw = curl_exec($ch); +$httpCode = curl_getinfo($ch, CURLINFO_RESPONSE_CODE); +curl_close($ch); + +// ── 8. parse (any failure → 502, never cached) ────────────────────── +$summary = null; +if ($raw !== false && $httpCode === 200) { + $resp = json_decode($raw, true); + $content = $resp['choices'][0]['message']['content'] ?? ''; + $parsed = json_decode((string)$content, true); + if (is_array($parsed) && is_string($parsed['summary'] ?? null) + && trim($parsed['summary']) !== '') { + $summary = trim($parsed['summary']); + if (mb_strlen($summary) > SUMMARY_MAX_CHARS) { + $summary = mb_substr($summary, 0, SUMMARY_MAX_CHARS) . '…'; + } + } +} +if ($lockFh !== false && $summary === null) { + flock($lockFh, LOCK_UN); + fclose($lockFh); +} +if ($summary === null) { + respond(502, ['error' => 'llm_failed']); +} + +// ── 9. cache + respond ────────────────────────────────────────────── +$out = json_encode(['summary' => $summary, 'generated_at' => gmdate('c')], + JSON_UNESCAPED_SLASHES | JSON_UNESCAPED_UNICODE); +if (!atomic_write($cachePath, $out)) { + // Degrade to per-click generation rather than a broken feature — + // the daily cap still bounds spend. + error_log('news_summary.php: cache write failed for ' . $cachePath); +} +if ($lockFh !== false) { + flock($lockFh, LOCK_UN); + fclose($lockFh); +} +http_response_code(200); +header('Content-Type: application/json; charset=utf-8'); +header('Cache-Control: max-age=3600'); +echo $out; diff --git a/test_all.py b/test_all.py index 67ce91a..acb562a 100644 --- a/test_all.py +++ b/test_all.py @@ -5295,6 +5295,97 @@ def _(): NewsEngine._source_health = saved +@test("NewsEngine: RSS parser captures link/description/publisher") +def _(): + cfg = NewsConfig() + cfg.rss_feeds = ["https://news.google.com/rss/search?q=gold"] + ne = NewsEngine(cfg) + saved = NewsEngine._source_health + NewsEngine._source_health = {} + # Google-News-shaped item: redirect , HTML-escaped , + # carrying the real publisher. + rss = (b"" + b"Gold surges as Fed signals rate cuts - Reuters" + b"https://news.google.com/rss/articles/CBMiAbC?oc=5" + b"<a href=\"https://x\">Gold &amp; dollar " + b"diverge as the Fed turns dovish.</a>" + b"Reuters" + b"") + + class _Sess: + def get(self, url, timeout=None): + return type("R", (), {"status_code": 200, "content": rss})() + + ne._session = _Sess() + try: + heads = ne._fetch_news_sentiment_raw() + assert len(heads) == 1, heads + h = heads[0] + assert h.link == "https://news.google.com/rss/articles/CBMiAbC?oc=5" + # tags stripped, entities (incl. double-encoded &amp;) decoded + assert h.description == "Gold & dollar diverge as the Fed turns dovish." + assert h.publisher == "Reuters" + finally: + NewsEngine._source_health = saved + + +@test("NewsEngine: enrichment fields default None / reject unsafe values") +def _(): + cfg = NewsConfig() + cfg.rss_feeds = ["https://feed.example/rss"] + ne = NewsEngine(cfg) + saved = NewsEngine._source_health + NewsEngine._source_health = {} + long_link = b"https://a.example/" + b"x" * 2100 + rss = (b"" + # bare item: no link/description/source at all + b"Dollar steady before Fed minutes" + # javascript: link + description that merely echoes the title + b"Gold slips on strong dollar data" + b"javascript:alert(1)" + b"Gold slips on strong dollar data" + # over-long link + b"Fed officials split on rate cuts" + b"" + long_link + b"" + b"") + + class _Sess: + def get(self, url, timeout=None): + return type("R", (), {"status_code": 200, "content": rss})() + + ne._session = _Sess() + try: + heads = ne._fetch_news_sentiment_raw() + assert len(heads) == 3, heads + by_title = {h.headline: h for h in heads} + bare = by_title["Dollar steady before Fed minutes"] + assert bare.link is None and bare.description is None \ + and bare.publisher is None + js = by_title["Gold slips on strong dollar data"] + assert js.link is None # non-http(s) scheme rejected + assert js.description is None # identical to title → dropped + assert by_title["Fed officials split on rate cuts"].link is None + finally: + NewsEngine._source_health = saved + + +@test("NewsEngine: _clean_html_text strips/unescapes/truncates") +def _(): + from news_engine import _clean_html_text, _safe_link + assert _clean_html_text(None) is None + assert _clean_html_text("") is None + assert _clean_html_text("

") is None + assert _clean_html_text("

Fed holds\n rates

") == "Fed holds rates" + assert _clean_html_text("&lt;b&gt;x&lt;/b&gt; & y") == "x & y" + long = _clean_html_text("a" * 600) + assert len(long) == 501 and long.endswith("…") + assert _safe_link(" https://a.example/x ") == "https://a.example/x" + assert _safe_link("ftp://a.example/x") is None + assert _safe_link("javascript:alert(1)") is None + assert _safe_link("https://" + "x" * 2100) is None + assert _safe_link(None) is None + + @test("/status renders news/calendar/AI health with ⚠️ degradations") def _(): import telegram_bot as tb @@ -5540,6 +5631,45 @@ def _(): assert payload["headlines"][1]["score"] is None # NaN → None assert payload["events"][0]["event"] == "ISM" assert payload["sentiment"] == {"XAU": 0.6, "USD": -0.1} + # Every headline carries the 40-hex id the PHP endpoint whitelists on. + import re as _re + assert all(_re.fullmatch(r"[0-9a-f]{40}", h["id"]) + for h in payload["headlines"]) + + +@test("dashboard: headline_to_dict id matches LLM cache scheme + enrichment") +def _(): + import hashlib + h = NewsSentiment( + headline="Gold surges", source="http://feed/b", + timestamp=datetime(2026, 8, 7, 10, 0, tzinfo=timezone.utc), + currencies_mentioned=["XAU"], sentiment_score=0.6, + keywords=["surge"], sentiment_by_currency={"XAU": 0.6}, + link="https://x.example/a", description="Body.", publisher="Reuters") + d = dpub.headline_to_dict(h) + # Same sha1(headline) scheme as NewsEngine._llm_sent_cache — the id is + # the join key news_summary.php validates against. + assert d["id"] == hashlib.sha1(b"Gold surges").hexdigest() + assert d["link"] == "https://x.example/a" + assert d["description"] == "Body." + assert d["publisher"] == "Reuters" + # Pre-enrichment NewsSentiment (defaults) still serializes cleanly. + bare = NewsSentiment( + headline="Old", source="s", timestamp=None, + currencies_mentioned=[], sentiment_score=0.0, keywords=[]) + d2 = dpub.headline_to_dict(bare) + assert d2["link"] is None and d2["description"] is None \ + and d2["publisher"] is None + json.dumps([d, d2], allow_nan=False) + + +@test("dashboard: install_static installs page + PHP endpoint atomically") +def _(): + with tempfile.TemporaryDirectory() as tmp: + pub = dpub.DashboardPublisher(out_dir=tmp) + pub.install_static() + assert sorted(os.listdir(tmp)) == \ + ["dashboard.html", "news_summary.php"] # both installed, no .tmp @test("dashboard: publish is atomic and leaves no temp file")