Remove Tiingo spread monitor (indicative two-tier feed, unvalidatable)
Tiingo's free /tiingo/fx/top publishes only a two-tier indicative spread for gold ($0.10 liquid hours / $0.70 rollover+news; verified on 273 production samples + live probes) — mid ± a posted increment, not executable depth, so it can never validate the $0.30 backtest cost assumption. Removed one day after shipping: tiingo_provider.py, the sampler/quote plumbing in telegram_bot.py, the channel_spread_detail config knob and tiingo_api_key resolver, and Module 14's Tiingo tests (221 -> 210). CLAUDE.md documents the verdict: do not re-add a spread display from Tiingo's free FX feed. Claude-Session: https://claude.ai/code/session_01BoZKi67Ct6QTeiFJmybXSw
This commit is contained in:
@@ -14,9 +14,8 @@ This file is the authoritative developer/agent reference. The human-facing docs
|
||||
|------|-------|---------|
|
||||
| `config.py` | ~374 | All config dataclasses, `PAIR_INFO` registry, per-pair strategy configs. **`PlatformConfig` defaults ARE the tuned XAU config.** |
|
||||
| `app_config.py` | ~290 | config.json loader/validation, env-precedence resolvers, repo-root `.env` loader (setdefault at import), `apply_pair_overrides` deployment overlay |
|
||||
| `config.json.example` | 27 | committed schema template; real gitignored config.json holds secrets + deployment settings |
|
||||
| `config.json.example` | 24 | committed schema template; real gitignored config.json holds secrets + deployment settings |
|
||||
| `data_provider.py` | ~347 | Twelve Data SDK wrapper: fetch, rate-limit (55 credits/min), in-memory cache, validation/cleaning, freshness checks |
|
||||
| `tiingo_provider.py` | ~160 | Tiingo `/tiingo/fx/top` bid/ask quotes (supplementary, NOT an OHLCV source): spread visibility on alerts (warn-only) + 5-min sampler → `logs/spread_log.csv`. No key → feature off |
|
||||
| `engine.py` | ~1173 | `TechnicalEngine`: computes all indicators on one DataFrame → `TechnicalAnalysis` (composite score, regime, key levels) |
|
||||
| `chart_patterns.py` | ~280 | Chart-pattern detectors for the XAU_CP archetype (pure functions, ATR-unit geometry, first-close-beyond triggers) |
|
||||
| `signal_aggregator.py` | ~1447 | `SignalAggregator`: MTF analysis, news integration, gates/penalties, quality grade, trade-setup generation → `SignalReport` |
|
||||
@@ -25,13 +24,13 @@ This file is the authoritative developer/agent reference. The human-facing docs
|
||||
| `pair_signals/__init__.py` | 52 | `PAIR_PRODUCERS` registry + `produce_signal()` |
|
||||
| `pair_signals/xauusd.py` | 71 | XAU config factory + per-timeframe overrides (docstring = validation record) |
|
||||
| `main.py` | ~348 | CLI: `python main.py --pair XAUUSD --timeframe H1 --detail/--json` |
|
||||
| `test_all.py` | ~3100 | QA suite: **215 tests in 14 modules**, no live API needed. Must stay green. |
|
||||
| `test_all.py` | ~3100 | QA suite: **210 tests in 14 modules**, no live API needed. Must stay green. |
|
||||
| `backtest/gpu_backtest.py` | ~2231 | Bar-replay backtester (CuPy→NumPy fallback) using the real `SignalAggregator` |
|
||||
| `backtest/diagnose_filters.py` | 609 | Filter-diagnosis tooling |
|
||||
| `backtest/research/` | — | Calendar research infra (`ff_calendar_scraper.py`, `calendar_store.py`) + result CSVs |
|
||||
| `backtest/*.csv` | — | Verification trade books from past tuning campaigns (`xau_*.csv`, …). **Gitignored, local-only** — used for manual before/after comparison, not by tests |
|
||||
| `backtest/cache/` | — | Parquet OHLCV cache (raw bars) |
|
||||
| `logs/telegram.log` | — | Bot log (3-hour rotation × 240 backups = 30-day retention); `logs/spread_log.csv` = Tiingo spread samples |
|
||||
| `logs/telegram.log` | — | Bot log (3-hour rotation × 240 backups = 30-day retention) |
|
||||
|
||||
---
|
||||
|
||||
@@ -273,10 +272,10 @@ Run: `python telegram_bot.py` (credentials resolve env `TELEGRAM_TOKEN`/`TELEGRA
|
||||
- **Alert gating**: quality threshold (`_alert_min_quality`, default "C", menu cycles C→B→A) + fingerprint dedup per (symbol, TF) on (signal, quality, direction) with 1-hour re-arm — a persisting identical signal doesn't re-send every candle, but a setup whose entry moved ≥ half the last-ALERTED risk distance (new pullback/swing anchor; compared against the stored last-alert entry/risk, a stable reference immune to per-candle ATR drift) re-alerts. The fingerprint is committed only **after** a message reaches ≥1 target (`_commit_alert`), so a failed Telegram send doesn't consume the alert.
|
||||
- **WebSocket monitor**: subscribes every `scan_pairs` symbol to TD quotes WS; 10 s heartbeat; exponential reconnect (≤60 s). On tick crossing any cached key level (`swing_high/low_1..3`, `ema_trend`, `bb_upper/lower`, `kc_upper/lower`; cache TTL 1 h) → immediate M15 rescan → "LEVEL BREAK" alert if actionable+changed (same gates as auto-scan), 5-min per-pair cooldown.
|
||||
- **Commands**: `/start` (menu), `/scan` (buttons: per pair — M5/M15/H1/H4 + Scan All TFs), `/autoscan` (toggle DM copy), `/status` (loops, WS, live prices, level cache, cooldowns), `/cache` (age per pair/TF), `/clearcache`, `/help`.
|
||||
- **Tiingo spread monitor** (optional; on only when a Tiingo key resolves — see Environment variables): a 5-min sampler task fetches top-of-book bid/ask per scanned pair (`tiingo_provider.py`), logs to `logs/spread_log.csv` (skipping the weekend-closed window), and caches the latest quote; alert sends refresh the quote first, and signal messages then show live bid/ask + a warning when spread ≥ 2× the pair's assumed round-trip cost (`WIDE_SPREAD_MULTIPLE`). **Warn-only BY DESIGN** — no historical bid/ask exists, so a suppressing gate could never be validated (live == backtest rule); signals never consume bid/ask. `/status` shows the live spread per pair. **License (free tier is internal-use-only):** raw bid/ask/spread numbers go to owner DMs always; the channel gets them ONLY while config.json `telegram.channel_spread_detail=true` (the channel is internal/testing as of 2026-07-22 — **flip to false before the channel goes public**), otherwise a number-free "spread unusually wide" caution via `format_signal_message(report, spread_detail=False)` — routing in `_spread_detail_for`, test-enforced, committed default false. **Rate limits (free tier 50 req/h, 1000/day, checked 2026-07-22):** sampler 12/h + alert-batch refreshes ≤17/h + breach refreshes ≤12/h = worst case ≈41/h with ONE pair; adding pairs multiplies this — recheck before growing `scan_pairs`.
|
||||
- **Tiingo spread monitor: REMOVED 2026-07-23** (shipped 2026-07-22, removed next day). Tiingo's free `/tiingo/fx/top` publishes a two-tier *indicative* spread for gold — exactly $0.10 in liquid hours and exactly $0.70 in the 21:00–00:00 UTC rollover window / news moments (verified: 273-sample production log + live probes; only those two values ever occur). Quotes are mid ± a posted increment, not executable depth, so the feature could never validate the $0.30 backtest cost assumption. Do not re-add a spread display from Tiingo's free FX feed.
|
||||
- Signal messages include the full indicator breakdown, warnings, trade setup, **trailing-stop management plan** (arm price + trail distance, read from the `trail_arm_r`/`trail_dist_r` stamped on the setup by the producing per-TF config), and a risk disclaimer on every outbound message.
|
||||
- Blocking scan work runs in `ThreadPoolExecutor(max_workers=4)`; one shared `ForexDataProvider` across all scans (cache reuse).
|
||||
- **Logging (hardened 2026-07-22)**: root-logger handlers (`force=True`) → console + `logs/telegram.log` (3-hour rotation, 240 backups = 30 days), full-date timestamps; every module logger (data_provider, tiingo_provider, news_engine, websockets, telegram.ext) lands in the same file. Failure surfaces all covered: candle loops and the spread sampler catch-and-continue per cycle (one bad scan can't kill a loop), breach tasks log exceptions via done-callback, PTB `add_error_handler` registered, and `sys.excepthook`/`threading.excepthook` (installed in `main()`) route uncaught crashes to the log. WS monitor keeps its exponential-backoff reconnect + `logger.exception`.
|
||||
- **Logging (hardened 2026-07-22)**: root-logger handlers (`force=True`) → console + `logs/telegram.log` (3-hour rotation, 240 backups = 30 days), full-date timestamps; every module logger (data_provider, news_engine, websockets, telegram.ext) lands in the same file. Failure surfaces all covered: candle loops catch-and-continue per cycle (one bad scan can't kill a loop), breach tasks log exceptions via done-callback, PTB `add_error_handler` registered, and `sys.excepthook`/`threading.excepthook` (installed in `main()`) route uncaught crashes to the log. WS monitor keeps its exponential-backoff reconnect + `logger.exception`.
|
||||
|
||||
---
|
||||
|
||||
@@ -321,7 +320,7 @@ Data: paginated TD fetches (5000-bar chunks) cached as parquet in `backtest/cach
|
||||
|
||||
## test_all.py — QA suite
|
||||
|
||||
`python test_all.py` — **215 tests, 14 modules** (config, data_provider, engine, news_engine, signal_aggregator, main, telegram_bot, integration, pair_signals registry + engine-bug regressions, 2026-07-16 audit-fix regressions, chart patterns, app_config, tiingo_provider). No network needed (mocked). Uses a custom `@test("name")` decorator that runs immediately (NOT pytest — don't grep for `def test_`). Exit code non-zero on failure. Includes: the xauusd-factory ≡ inline-config equivalence test, engine bug-fix regression tests (Module 9b), a live-vs-GPU engine parity sweep incl. the EMA-ribbon path (Module 11), config.json loading/precedence/validation coverage (Module 13), and Tiingo quote/spread-monitor + `.env` loader coverage (Module 14). **Any signal-logic change must keep this green and should add tests.**
|
||||
`python test_all.py` — **210 tests, 14 modules** (config, data_provider, engine, news_engine, signal_aggregator, main, telegram_bot, integration, pair_signals registry + engine-bug regressions, 2026-07-16 audit-fix regressions, chart patterns, app_config, .env loader + bot resilience). No network needed (mocked). Uses a custom `@test("name")` decorator that runs immediately (NOT pytest — don't grep for `def test_`). Exit code non-zero on failure. Includes: the xauusd-factory ≡ inline-config equivalence test, engine bug-fix regression tests (Module 9b), a live-vs-GPU engine parity sweep incl. the EMA-ribbon path (Module 11), config.json loading/precedence/validation coverage (Module 13), and `.env` loader + bot-resilience coverage (Module 14). **Any signal-logic change must keep this green and should add tests.**
|
||||
|
||||
---
|
||||
|
||||
@@ -335,7 +334,6 @@ Precedence for every credential/deployment setting: **env var → config.json
|
||||
| `TELEGRAM_TOKEN` | bot token | none — falls through to config.json `telegram.token`, then refuses to boot |
|
||||
| `TELEGRAM_CHAT_ID` | owner chat id (boot auto-scan + owner lock) | none — falls through to config.json `telegram.chat_id`, then refuses to boot |
|
||||
| `TELEGRAM_CHANNEL_ID` | subscriber channel (`-100…`); `""` disables the feed | none — falls through to config.json `telegram.channel_id` |
|
||||
| `TIINGO_API_KEY` (or `TIINGO`) | Tiingo bid/ask quote key for the spread monitor — optional | none — falls through to config.json `data_providers.tiingo.api_key`; absent = spread monitor off (never fatal) |
|
||||
| `FB2_EMA_LO` | EMA200 "near" penalty threshold in % | `0.5` |
|
||||
|
||||
Dependencies (`requirements.txt`): numpy, pandas, requests, twelvedata, python-telegram-bot ≥20, websockets. Python 3.12. A `venv/` exists in-repo.
|
||||
|
||||
Reference in New Issue
Block a user