harry/ForexBot2
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

Audit fixes: cache boundary staleness, 6e session bypass, GPU engine parity

Browse Source 
Fixes all high+medium findings from the 2026-07-16 full-codebase audit.

High:
- data_provider: cache now rejects snapshots missing the latest closed
  bar (_has_latest_closed_bar, weekend-exempt). The wall-clock TTL served
  mid-candle snapshots at the next boundary, so live M15/H1 scans ran one
  full bar behind and the bar-close session check evaluated the wrong hour.
- signal_aggregator: gate 6e (SELL quality) only acts on actionable
  grades. It was re-grading session-blocked F reports to D, which
  re-opened the M5 pullback path (entry check: quality != "F") on
  session-blocked bars.
- gpu_backtest: port the live EMA-ribbon compression/expansion logic
  (+10 conf on expansion) into the GPU engine. Its absence diverged
  composite score/conf on 7-9% of bars and flipped signals on ~0.2%.

Medium:
- gpu_backtest: Bollinger precompute uses the exact pandas rolling calls
  (cumsum E[x2]-E[x]2 lost precision at gold price levels; bb_position
  error up to 7.6e-3 crossed signal thresholds, worst during squeezes).
- gpu_backtest: --confirm-tfs defaults to the pair factory's live
  confirmation map instead of a hardcoded 1h,4h.
- gpu_backtest: entries re-anchor to the first RAW bar after the signal
  bar, so >10-sigma spike bars dropped by cleaning are no longer skipped
  at entry (favorable bias).
- gpu_backtest: per-TF warmup pad — D1 confirms get >=210 bars at window
  start (62-day pad gave ~61 bars: no EMA200, unlike live).
- data_provider: thread-safety locks for cache + throttle (shared
  provider across the bot's 4-thread pool); outlier filter keeps the
  snap-back bar after a spike instead of punching a two-bar hole.
- telegram_bot: alert fingerprint gains a risk-relative entry bucket
  (materially-moved setups re-alert) and is committed only after a
  successful send (failed sends no longer consume the alert for 1h).
- engine: disabled_indicators=["macd"] now removes the MACD pillar from
  scoring (histogram still computed for divergence).

Verification: test_all.py 187/187 (new Module 11: 8 audit-fix
regressions incl. a live-vs-GPU parity sweep with ribbon coverage);
real-data parity sweep over 813 slices of 14mo XAU M15: zero divergence
(pre-fix: 7-9% of bars diverged); both original bug repros no longer
reproduce; cached end-to-end backtest run passes with the live confirm
map. Books shift slightly vs pre-fix runs — comparability note added to
CLAUDE.md; pinned tuned numbers need re-validation under the fixed
backtester.
This commit is contained in:
Harwinder Sekhon
2026-07-16 10:24:39 -04:00
parent 4d54fddf80
commit fe7d4c5d4a
7 changed files with 483 additions and 134 deletions
Download Patch File Download Diff File Expand all files Collapse all files
data_provider.py
+63 -24
View File
@@ -6,6 +6,7 @@ Fetches and validates OHLCV via the official `twelvedata` Python SDK.
import logging
import os
import threading
import time
from collections import deque
from typing import Dict, Optional
@@ -75,6 +76,10 @@ class ForexDataProvider:
)
self._cache: Dict[str, pd.DataFrame] = {}
self._cache_timestamps: Dict[str, float] = {}
# The bot shares one provider across a thread pool; cache eviction
# and the throttle deque are check-then-act and need locking.
self._cache_lock = threading.Lock()
self._throttle_lock = threading.Lock()
self._request_times: deque = deque()
self._td: Optional[TDClient] = (
TDClient(apikey=self._api_key) if self._api_key else None
@@ -103,18 +108,22 @@ class ForexDataProvider:
now = time.time()
cache_ttl = self.CANDLE_SECONDS.get(timeframe, self.config.refresh_interval)
if (
cache_key in self._cache_timestamps
and (now - self._cache_timestamps[cache_key]) < cache_ttl
and cache_key in self._cache
):
with self._cache_lock:
cached = self._cache.get(cache_key)
if cached is not None and not cached.empty:
if self._is_data_fresh(cached, symbol, timeframe):
return cached
logger.warning(f"{symbol} {timeframe.value}: cached data is stale, refetching")
del self._cache[cache_key]
del self._cache_timestamps[cache_key]
cached_at = self._cache_timestamps.get(cache_key)
if (
cached is not None
and cached_at is not None
and (now - cached_at) < cache_ttl
and not cached.empty
):
if (self._is_data_fresh(cached, symbol, timeframe)
and self._has_latest_closed_bar(cached, timeframe)):
return cached
logger.warning(f"{symbol} {timeframe.value}: cached data is stale, refetching")
with self._cache_lock:
self._cache.pop(cache_key, None)
self._cache_timestamps.pop(cache_key, None)
outputsize = max(
300,
@@ -130,10 +139,27 @@ class ForexDataProvider:
f"last bar {df.index[-1]} is too old. Signal quality may be degraded."
)
self._cache[cache_key] = df
self._cache_timestamps[cache_key] = now
with self._cache_lock:
self._cache[cache_key] = df
self._cache_timestamps[cache_key] = now
return df
def _has_latest_closed_bar(self, df: pd.DataFrame, timeframe: Timeframe) -> bool:
"""Cached data must include the most recent fully closed bar.
The TTL alone is wall-clock based: a snapshot fetched mid-candle is
still within TTL at the next candle boundary but is missing the bar
that just closed, so every boundary scan would analyze — and run the
bar-close session check against — the previous bar.
"""
now = pd.Timestamp.now(tz="UTC")
if _is_forex_weekend(now):
return True
candle_seconds = self.CANDLE_SECONDS.get(timeframe, 3600)
dur = pd.Timedelta(seconds=candle_seconds)
expected_label = now.floor(dur) - dur
return df.index[-1] >= expected_label
def _drop_forming_candle(self, df: pd.DataFrame, timeframe: Timeframe) -> pd.DataFrame:
"""Drop the still-forming last candle so analysis only sees closed bars.
@@ -209,19 +235,21 @@ class ForexDataProvider:
def _throttle(self, credits: int = 1) -> None:
"""Pro tier: 55 credits / minute. Block until budget allows."""
while True:
now = time.time()
while self._request_times and now - self._request_times[0] >= self.RATE_WINDOW:
self._request_times.popleft()
with self._throttle_lock:
now = time.time()
while self._request_times and now - self._request_times[0] >= self.RATE_WINDOW:
self._request_times.popleft()
available = self.RATE_LIMIT - len(self._request_times)
if available >= credits:
for _ in range(credits):
self._request_times.append(time.time())
return
available = self.RATE_LIMIT - len(self._request_times)
if available >= credits:
for _ in range(credits):
self._request_times.append(time.time())
return
wait = (self._request_times[0] + self.RATE_WINDOW) - now + 0.2
used = len(self._request_times)
wait = (self._request_times[0] + self.RATE_WINDOW) - now + 0.2
logger.info(
f"Rate limit: {len(self._request_times)}/{self.RATE_LIMIT} credits used "
f"Rate limit: {used}/{self.RATE_LIMIT} credits used "
f"— waiting {wait:.1f}s..."
)
time.sleep(wait)
@@ -324,7 +352,18 @@ class ForexDataProvider:
returns = df["Close"].pct_change()
std_ret = returns.std()
if std_ret > 0:
outlier_mask = (returns - returns.mean()).abs() > 10 * std_ret
dev = returns - returns.mean()
outlier_mask = dev.abs() > 10 * std_ret
# A spike bar makes the NEXT bar's return look extreme too (it is
# measured against the spike). If the previous bar is flagged and
# this bar reverses direction, it is the legitimate snap-back —
# keep it, or every spike would punch a two-bar hole in the data.
recovery = (
outlier_mask
& outlier_mask.shift(1, fill_value=False)
& (dev * dev.shift(1) < 0)
)
outlier_mask = outlier_mask & ~recovery
if outlier_mask.any():
logger.warning(f"{symbol}: Removing {outlier_mask.sum()} extreme outlier bars")
df = df[~outlier_mask]