harry/ForexBot2
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

Daily-trend veto on M15 + lot-size / already-open lines (campaign 2026-09-30), and the 2026-10-01 whole-codebase bug review

Browse Source 
Two bodies of work that touch the same files, so one commit.

Direction and setup-size campaign (2026-09-30 → 10-01, owner's RISK-FIRST rule,
written before the runs):
- SIGNAL: M15 daily-trend veto (factory override 7, xau_strategy.daily_trend_veto
  "both"). A finished setup against a clear daily trend (closed D1 bars, Sunday
  stubs dropped; close vs its 50-bar EMA AND vs the close 60 kept days earlier)
  is removed as the last step of the core. One definition
  (signal_aggregator.daily_trend_series) for live, the bar replay and the twin's
  host side; D1 is a required frame of the M15 scan; mirrored in
  cuda_signal_kernels.py K2. W3 824 / +205.12R / PF 1.73 / max DD 12.4R vs
  1105 / +195.79R / 1.49 / 20.9R; 2019 → 2026 1473 / +223.33R / 1.42 / 41.5R vs
  2091 / +192.53R / 1.24 / 64.8R; unseen holdout −4.62R vs −6.63R.
- FAILED, not shipped: the veto on M5 (W3 drawdown 10.1R → 15.2R) and on H1
  (−9.7R), buy-only and sell-only on M15, the wide-stop cap
  (xau_strategy.wide_stop_cap_atr stays 0.0, a research knob).
- New M15 pins: W2 dfc647ee…, OOS e87b6de1…, W3 ad3b0dbd…. M5 / H1 / H4 pins
  unchanged. --set xau_strategy.daily_trend_veto=off reproduces the previous M15
  pins byte for byte. Twin and reference script MD5-equal on every window;
  recorder and feature-export parity re-proved; base-rate table rebuilt.
- DISPLAY (books byte-identical): exposure.py — the lot size for
  trade_size.risk_per_trade (new config.json block, off by default, private to
  the owner) and the count of alerted same-direction plans still open
  (signals.alerted_at, read-only open_alerted_setups); the daily trend in words
  on Telegram, the CLI and the dashboard; signals.daily_trend logged.
- providers/oanda: a date window with no bars is an empty page, not an error
  (a D1 pad starting on a Saturday could not be fetched).
- Reports only, nothing shipped: seven context screens (no state nominated) and
  eight outside data sources (none adds a forecast to gold's own daily trend).
- Independent verification (report section 5.13): every figure reproduced on
  both scripts, scoring identical to the previous commit on all four
  timeframes, no look-ahead. Caveats recorded there: this is a risk filter, not
  a more accurate direction call; nine tenths of the 2019 → 2026 gain is 2022;
  the neighbour clause is thin (a simple 100-day average misses the R line by
  1.2R); "never adds a trade" is an observation, not a guarantee.

Bug review of the whole codebase (2026-10-01). Signal path clean. 17 defects
elsewhere, plus 4 found by an independent review of the fixes; all fixed, each
with a test; no trade book changed:
- telegram_bot: one _deliver_alert for the candle scan and the level break. The
  row is stamped alerted_at BEFORE the send, on a one-worker alert-db pool, and
  un-stamped when no chat received it or the task is cancelled mid-stamp, so
  two alerts at one candle close count each other and no SQLite write runs on
  the event loop. The 5-bar cooldown counts bars of the scan's own frame
  (== gpu_backtest._apply_cooldown; clock time released the 22:00 UTC bar after
  the settlement pause) and is restored from signals.db at start-up. The
  lot-size line goes to the owner's own chat only (a group the owner ran
  /autoscan in used to receive it). /status gains a news-model line and escapes
  failure details.
- signal_log: mark_alerted returns the stamp it wrote; unmark_alerted;
  last_alerted_bars (read-only).
- advisory_outcome: candle continuity is judged bar by bar inside the walk; a
  hole after a trade's exit no longer voids it. The stored-candle replay
  resolves 108 of 111 logged setups instead of 99, all equal to the live
  resolver's R.
- providers/oanda: an empty date window is accepted only when it is 5 days or
  shorter and no malformed candle was skipped; a longer one is retried, then
  refused, instead of being cached as history with a hole.
- news_engine: blank RSS titles skipped; one unreadable headline no longer
  costs the local model its whole batch (checked on the real bundle); model
  health recorded; no DeepSeek audit call in a process that does not archive;
  event-direction cache keyed per release; peek_week_events never fetches.
- llm_client: chat_json never raises and never logs the API key (requests
  quotes the Authorization value in InvalidHeader; error bodies are redacted).
- signal_aggregator: the scan's news block is guarded; the replay path
  (use_news=False) assigns exactly the values it did before.
- agent_panel: PANEL_PROMPT_VERSION v2.2 — same prompt text, corrected input
  (event timing over the cached calendar week, headlines newest first).
  scripts/panel_replay.py replays the logged payload per persona;
  scripts/label_scan_outcomes.py counts with an ISO cutoff.
- test_all.py: no ALTER TABLE … DROP COLUMN (the live box has SQLite 3.34);
  tests that write Parquet print a notice and return without pyarrow.

Verification after the last code change: suite 632 passed, 0 failed (also with
both Parquet engines blocked and FB2_BACKEND=cpu, the live box's view); the
five reference MD5s reproduce on both backtesters, 0 cache fallbacks; Module 37
green; static Python 3.9 / SQLite 3.34 check clean on every touched file.

Evidence: backtest/research/daily_trend_campaign_2026-09-30.md,
context_screens_2026-10.md, new_inputs_2026-10.md; CLAUDE.md sections 2, 4, 6,
8, 9, 10 and 11; pair_signals/xauusd.py docstring (M15 override 7).
This commit is contained in:
Harwinder Sekhon
2026-10-01 10:28:28 -04:00
parent 42025da0b0
commit 2cc5915b00
31 changed files with 6021 additions and 286 deletions
Download Patch File Download Diff File Expand all files Collapse all files
news_engine.py
+74 -7
View File
@@ -384,7 +384,7 @@ class NewsEngine:
# writer runs inside _cached_fetch); the other two take the lock
# around cache access but keep HTTP calls outside it.
_llm_sent_cache: Dict[str, Tuple[float, float]] = {} # sha1 → (s, xau)
_llm_event_cache: Dict[Tuple[str, str, str], str] = {}
_llm_event_cache: Dict[Tuple[str, str, str, str, str], str] = {}
_llm_summary_cache: Tuple[float, str] = (0.0, "")
_LLM_SENT_CACHE_MAX = 400
_LLM_EVENT_CACHE_MAX = 100
@@ -609,6 +609,23 @@ class NewsEngine:
key=lambda e: e.time,
)
def peek_week_events(
self, currencies: Optional[List[str]] = None
) -> List[EconomicEvent]:
"""get_week_events WITHOUT the refresh: whatever calendar the
class-level cache holds right now, however old, filtered and sorted
the same way. Never fetches and never waits on the cache lock, for
a caller that must not add a network wait: the agent panel runs
while an alert is waiting to go out, and a refetch with both
calendar sources down takes about 40 seconds."""
if currencies is None:
currencies = self.config.tracked_currencies
_stamp, events = NewsEngine._calendar_cache
return sorted(
(e for e in events if e.currency in currencies),
key=lambda e: e.time,
)
# ── Private Methods ────────────────────────────────────────────
# Maps country names in MyFXBook titles to currency codes
@@ -871,6 +888,11 @@ class NewsEngine:
continue
title = title_el.text or ""
# A blank title carries nothing to score, show or
# archive, and one such item made the local model
# reject the whole batch (2026-10-01).
if not title.strip():
continue
pub_date = self._parse_rss_date(
pub_date_el.text if pub_date_el is not None else None
)
@@ -1173,11 +1195,27 @@ class NewsEngine:
is re-sent to DeepSeek (fallback mode retries, audit sample);
F. only score_source == "keyword" rows change; an llm row stands.
"""
bundle = news_model.get_news_model()
if bundle is None or not headlines:
if not headlines:
return 0
todo = [h for h in headlines if h.score_source == "keyword"]
bundle = news_model.get_news_model()
if bundle is None:
self._record_source("news_model", False,
"not loaded in this process")
return 0
# score_batch cleans every text first and raises on one that is
# empty afterwards (" " is), which used to cost the whole
# batch its scores; such a row keeps its keyword score instead.
todo = [h for h in headlines if h.score_source == "keyword"
and self._model_can_score(h.headline)]
if not todo:
# Nothing for the model this cycle (fallback mode with DeepSeek
# healthy, or no readable headline). A failure an earlier batch
# recorded says nothing about now and must not sit on /status
# as if it were current; the next batch records its own outcome
# and warns again if it fails.
if self.source_is_failing(
NewsEngine._source_health.get("news_model")):
NewsEngine._source_health.pop("news_model", None)
return 0
try:
results = news_model.score_batch(bundle, [h.headline for h in todo])
@@ -1185,8 +1223,13 @@ class NewsEngine:
raise ValueError(
f"{len(results)} results for {len(todo)} headlines")
except Exception as e:
logger.debug(f"news model scoring failed, keyword scores stand: {e}")
# Keyword scores stand for the cycle. Loud once (WARNING on the
# first failure, INFO on recovery) and shown by /status: in
# primary mode nothing else scores these headlines.
self._record_source("news_model", False,
f"{type(e).__name__}: {e}")
return 0
self._record_source("news_model", True, extra=f"{len(todo)} scored")
for h, r in zip(todo, results):
raw = float(r["score"]) # archive value
shown = raw if r["relevant"] else 0.0 # display value
@@ -1198,6 +1241,15 @@ class NewsEngine:
h.score_source = "model"
return len(todo)
@staticmethod
def _model_can_score(text: str) -> bool:
"""True when the local model's own cleaner accepts the text."""
try:
news_model.clean_headline(text)
return True
except Exception:
return False
def _audit_with_deepseek(
self, headlines: List[NewsSentiment]
) -> List[NewsSentiment]:
@@ -1207,10 +1259,15 @@ class NewsEngine:
numbers. One batch per cycle at most; the pairs go into
_llm_sent_cache so a headline is never audited twice. This is the
only place DeepSeek is called for headlines in primary mode; with
the cap at 0 (the shipped value) it returns [] without a call."""
the cap at 0 (the shipped value) it returns [] without a call.
The copies exist only to be archived, so a process that does not
write the archive (the dashboard daemon, NEWS_ARCHIVE_ENABLED=0)
makes no call either."""
cap = app_config.news_model_audit_per_day()
if cap <= 0 or not headlines or not get_llm_client().available:
return []
if not app_config.news_archive_enabled():
return []
today = datetime.now(timezone.utc).date().isoformat()
if NewsEngine._audit_day != today:
NewsEngine._audit_day, NewsEngine._audit_sent_today = today, 0
@@ -1253,7 +1310,17 @@ class NewsEngine:
close K/M-suffixed pairs backwards ~11% of the time (e.g. claims
202K vs 217K called "higher than forecast") when left to compare
the raw numbers itself."""
key = (event.event, event.currency, str(event.actual or ""))
# Forecast and release time are part of the key: the direction is
# a reading of actual AGAINST forecast, and next month's release
# can print the same actual against a different forecast. The time
# is keyed in UTC so one instant is one key whatever offset the
# feed printed it with.
try:
when = event.time.astimezone(timezone.utc).isoformat()
except Exception:
when = str(event.time)
key = (event.event, event.currency, str(event.actual or ""),
str(event.forecast or ""), when)
with NewsEngine._cache_lock:
if key in NewsEngine._llm_event_cache:
return NewsEngine._llm_event_cache[key] or None