harry/ForexBot2
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

Tighten signal pipeline: yfinance-only XAU/XAG, leaner Telegram /scan

Browse Source 
- XAU/XAG always route to Yahoo Finance and skip Twelve Data entirely
  (excluded from batch prefetch, served per-symbol via yfinance).
- Daily chart cleanly skipped for gold/silver — engine drops it from MTF.
- Telegram /scan limited to EURUSD, USDJPY, GBPUSD, XAUUSD.
- SCAN_TIMEFRAMES reduced to single H1 primary; engine still pulls
  M15+H1+D1 internally via confirmation TFs (halves API credits per scan).
- Default confirmation TFs swapped H4 -> D1 in config and main.py.
- mtf_agreement_threshold kept at 2-of-3.
- CLAUDE.md cleaned up — old XAUUSD/synthetic-data bug reports removed.

Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
This commit is contained in:
Harwinder
2026-05-05 19:52:41 -04:00
parent 4c1d839a1a
commit 1100b9eaf2
5 changed files with 45 additions and 70 deletions
Download Patch File Download Diff File Expand all files Collapse all files
CLAUDE.md
+4 -58
View File
@@ -1,64 +1,10 @@
# ForexBot2 — Known Bugs & Dev Notes
## Bug 1 — Concurrent TF scans exhaust API credits instantly
**Symptom:**
```
Batch prefetch failed for 1wk: You have run out of API credits for the current minute.
9 API credits were used, with the current limit at https://twelvedata.com/pricing
```
Warnings appear for every timeframe (1h, 4h, 1d, 1wk) in rapid succession.
**Root cause:**
`telegram_bot.py` runs all 4 `SCAN_TIMEFRAMES` concurrently via `ThreadPoolExecutor`.
Each `SignalAggregator` fires its own batch prefetch simultaneously.
Free tier limit is **8 credits/minute** — 4 parallel scans hit this in under a second.
**Impact:**
Confirmation timeframes fail and fall back to synthetic data.
Primary TF still has real data (first request wins the credits).
**Fix (not yet implemented):**
Run timeframes sequentially with a 60-second wait between each, or reduce
`SCAN_TIMEFRAMES` to a single primary TF and use the engine's built-in
multi-timeframe confirmation (which already batches efficiently in 3 calls).
---
## Bug 2 — XAUUSD (Gold) shows wrong price (~1.01 instead of ~3300)
**Symptom:**
```
Entry: 1.01037
Take Profit: 1.00767 (+0.3 pips)
Stop Loss: 1.01172 (-0.1 pips)
```
**Root cause:**
Two compounding issues:
1. XAUUSD is not available on the Twelve Data **free tier** — API returns no data.
2. `data_provider._generate_synthetic_data()` uses a `pair_params` dict for base
prices. XAUUSD either falls through to a wrong default (~1.0) instead of the
correct ~3300 baseline.
Check `data_provider.py` → `_generate_synthetic_data()` → `pair_params` dict.
Verify `"XAUUSD=X"` has a correct `"price"` entry around `3300.0`.
**Impact:**
All indicator values, entry, TP, and SL for XAUUSD are meaningless when
running on the free tier (synthetic data with wrong base price).
**Fix (not yet implemented):**
- Correct the base price in `pair_params` for `XAUUSD=X`.
- Add a bot-level guard: if `report.current_price < 100` for XAUUSD,
suppress the trade setup and show a "data unavailable" message instead.
---
# ForexBot2 — Dev Notes
## Notes
- Free tier: 8 credits/min, 800 credits/day. Each full scan = ~21 credits.
- Supported pairs (live data on free tier): EURUSD, GBPUSD, USDJPY, USDCHF,
AUDUSD, USDCAD, NZDUSD, EURGBP, EURJPY, GBPJPY, AUDJPY, EURCHF, CADJPY.
- All other symbols (XAU, XAG, commodities, indices) fall back to synthetic GBM data.
- XAU/XAG and other unsupported symbols fall back to yfinance.
- No synthetic data fallback — if both Twelve Data and yfinance fail, the
request errors out and no signal is generated.
config.py
+1 -1
View File
@@ -187,7 +187,7 @@ class PlatformConfig:
# Analysis timeframes (primary + confirmation)
primary_timeframe: Timeframe = Timeframe.H1
confirmation_timeframes: List[Timeframe] = field(
default_factory=lambda: [Timeframe.M15, Timeframe.H4]
default_factory=lambda: [Timeframe.M15, Timeframe.D1]
)
# Data source
data_provider.py
+27 -3
View File
@@ -157,6 +157,24 @@ class ForexDataProvider:
if cached is not None and not cached.empty:
return cached
# Symbols not on Twelve Data (gold, silver) always use yfinance.
# D1 is intentionally skipped for these — yfinance daily for futures
# is unreliable and the engine drops missing confirmation TFs cleanly.
if symbol in _YF_FALLBACK_MAP:
if timeframe == Timeframe.D1:
raise DataUnavailableError(
f"Daily chart skipped for {symbol} — gold/silver use M15/H1 only."
)
yf_df = self._fetch_yfinance_fallback(symbol, timeframe)
if yf_df is None:
raise DataUnavailableError(
f"Yahoo Finance unavailable for {symbol} ({timeframe.value})."
)
df = self._validate_and_clean(yf_df, symbol, timeframe)
self._cache[cache_key] = df
self._cache_timestamps[cache_key] = now
return df
df = self._fetch_twelve_data(symbol, timeframe, lookback_days)
df = self._validate_and_clean(df, symbol, timeframe)
@@ -181,6 +199,12 @@ class ForexDataProvider:
if lookback_days is None:
lookback_days = self.config.data_lookback_days
# XAU/XAG aren't on Twelve Data — exclude them from the batch call,
# they'll be fetched individually via yfinance on demand.
td_symbols = [s for s in symbols if s not in _YF_FALLBACK_MAP]
if not td_symbols:
return
now = time.time()
for tf in timeframes:
ttl = self._cache_ttl(tf)
@@ -188,13 +212,13 @@ class ForexDataProvider:
f"{s}_{tf.value}" in self._cache_timestamps
and (now - self._cache_timestamps[f"{s}_{tf.value}"]) < ttl
and not self._cache.get(f"{s}_{tf.value}", pd.DataFrame()).empty
for s in symbols
for s in td_symbols
)
if cached:
logger.debug(f"Skipping prefetch for {tf.value} — all {len(symbols)} symbols cached (TTL {ttl}s)")
logger.debug(f"Skipping prefetch for {tf.value} — all {len(td_symbols)} symbols cached (TTL {ttl}s)")
continue
try:
self._batch_fetch_timeframe(symbols, tf, lookback_days)
self._batch_fetch_timeframe(td_symbols, tf, lookback_days)
time.sleep(1.0) # brief pause between TF requests
except Exception as e:
logger.warning(f"Batch prefetch failed for {tf.value}: {e}")
main.py
+6 -5
View File
@@ -403,17 +403,18 @@ def run_analysis(args) -> List[SignalReport]:
}
config.primary_timeframe = tf_map[args.timeframe]
# Configure confirmation timeframes based on primary
# Configure confirmation timeframes based on primary.
# H1 uses M15 + D1 for high-conviction confluence (3-of-3 threshold).
tf_confirmations = {
"M5": [Timeframe.M1, Timeframe.M15],
"M15": [Timeframe.M5, Timeframe.H1],
"M30": [Timeframe.M15, Timeframe.H1],
"H1": [Timeframe.M15, Timeframe.H4],
"M15": [Timeframe.H1, Timeframe.D1],
"M30": [Timeframe.H1, Timeframe.D1],
"H1": [Timeframe.M15, Timeframe.D1],
"H4": [Timeframe.H1, Timeframe.D1],
"D1": [Timeframe.H4, Timeframe.W1],
}
config.confirmation_timeframes = tf_confirmations.get(
args.timeframe, [Timeframe.M15, Timeframe.H4]
args.timeframe, [Timeframe.M15, Timeframe.D1]
)
# Configure pairs
telegram_bot.py
+7 -3
View File
@@ -40,7 +40,11 @@ logger = logging.getLogger(__name__)
# ── Config ──────────────────────────────────────────────────────────
SCAN_TIMEFRAMES = ["M15", "H1"]
SCAN_TIMEFRAMES = ["H1"] # engine pulls M15 + H1 + D1 via confirmations
# Pairs analyzed by the /scan command. Limited to the most liquid majors
# plus gold to keep credits low and signal quality high.
SCAN_PAIRS = ["EURUSD=X", "USDJPY=X", "GBPUSD=X", "XAUUSD=X"]
_TF_MAP = {
"M5": Timeframe.M5, "M15": Timeframe.M15,
@@ -268,7 +272,7 @@ class ForexTelegramBot:
def _build_config(self, tf: str, pairs: List[str]) -> PlatformConfig:
cfg = PlatformConfig()
cfg.primary_timeframe = _TF_MAP[tf]
cfg.confirmation_timeframes = _TF_CONFIRMATIONS.get(tf, [Timeframe.M15, Timeframe.H4])
cfg.confirmation_timeframes = _TF_CONFIRMATIONS.get(tf, [Timeframe.M15, Timeframe.D1])
cfg.watch_pairs = list(pairs)
return cfg
@@ -418,7 +422,7 @@ class ForexTelegramBot:
pass
try:
results_by_tf = await self._analyze_all_tfs(list(MAJOR_PAIRS), progress=progress)
results_by_tf = await self._analyze_all_tfs(list(SCAN_PAIRS), progress=progress)
now_str = _now_est()
# Best grade per pair across all TFs