harry/ForexBot2
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

Migrate to yfinance and parallelize analysis

Browse Source
This commit is contained in:
Harwinder
2026-05-05 16:37:11 -04:00
parent 4c1d839a1a
commit 3737cf4e71
8 changed files with 183 additions and 395 deletions
Download Patch File Download Diff File Expand all files Collapse all files
CLAUDE.md
+9 -52
View File
@@ -1,64 +1,21 @@
# ForexBot2 — Known Bugs & Dev Notes
## Bug 1 — Concurrent TF scans exhaust API credits instantly
## [RESOLVED] Bug 1 — Concurrent TF scans exhaust API credits instantly
**Symptom:**
```
Batch prefetch failed for 1wk: You have run out of API credits for the current minute.
9 API credits were used, with the current limit at https://twelvedata.com/pricing
```
Warnings appear for every timeframe (1h, 4h, 1d, 1wk) in rapid succession.
**Root cause:**
`telegram_bot.py` runs all 4 `SCAN_TIMEFRAMES` concurrently via `ThreadPoolExecutor`.
Each `SignalAggregator` fires its own batch prefetch simultaneously.
Free tier limit is **8 credits/minute** — 4 parallel scans hit this in under a second.
**Impact:**
Confirmation timeframes fail and fall back to synthetic data.
Primary TF still has real data (first request wins the credits).
**Fix (not yet implemented):**
Run timeframes sequentially with a 60-second wait between each, or reduce
`SCAN_TIMEFRAMES` to a single primary TF and use the engine's built-in
multi-timeframe confirmation (which already batches efficiently in 3 calls).
**Status:** Resolved
**Resolution:** Replaced the Twelve Data API with `yfinance` in `data_provider.py`. Since `yfinance` has no strict per-minute rate limiting, concurrent scans no longer fail. Furthermore, the `signal_aggregator.py` has been updated to use `ThreadPoolExecutor` to speed up pair analysis in parallel.
---
## Bug 2 — XAUUSD (Gold) shows wrong price (~1.01 instead of ~3300)
## [RESOLVED] Bug 2 — XAUUSD (Gold) shows wrong price (~1.01 instead of ~3300)
**Symptom:**
```
Entry: 1.01037
Take Profit: 1.00767 (+0.3 pips)
Stop Loss: 1.01172 (-0.1 pips)
```
**Root cause:**
Two compounding issues:
1. XAUUSD is not available on the Twelve Data **free tier** — API returns no data.
2. `data_provider._generate_synthetic_data()` uses a `pair_params` dict for base
prices. XAUUSD either falls through to a wrong default (~1.0) instead of the
correct ~3300 baseline.
Check `data_provider.py` → `_generate_synthetic_data()` → `pair_params` dict.
Verify `"XAUUSD=X"` has a correct `"price"` entry around `3300.0`.
**Impact:**
All indicator values, entry, TP, and SL for XAUUSD are meaningless when
running on the free tier (synthetic data with wrong base price).
**Fix (not yet implemented):**
- Correct the base price in `pair_params` for `XAUUSD=X`.
- Add a bot-level guard: if `report.current_price < 100` for XAUUSD,
suppress the trade setup and show a "data unavailable" message instead.
**Status:** Resolved
**Resolution:** By transitioning entirely to `yfinance`, XAUUSD is now natively mapped to `GC=F` via the `_YF_FALLBACK_MAP`. This means we get the real, correct OHLCV data directly without needing synthetic fallbacks.
---
## Notes
- Free tier: 8 credits/min, 800 credits/day. Each full scan = ~21 credits.
- Supported pairs (live data on free tier): EURUSD, GBPUSD, USDJPY, USDCHF,
AUDUSD, USDCAD, NZDUSD, EURGBP, EURJPY, GBPJPY, AUDJPY, EURCHF, CADJPY.
- All other symbols (XAU, XAG, commodities, indices) fall back to synthetic GBM data.
- **Data Provider**: The primary data provider is now `yfinance`. `Twelve Data` has been completely removed along with all its throttle logic and API Key requirements.
- **Parallelization**: Pair analysis is now multithreaded using `ThreadPoolExecutor` in `signal_aggregator.py`, maximizing the speed of data fetching and processing.
- **Supported pairs**: All forex pairs (EURUSD, GBPUSD, USDJPY, etc.) are fetched with an `=X` suffix in Yahoo Finance. Commodities like Gold (XAUUSD=X) use their futures symbols (`GC=F`).