Tiingo bid/ask spread monitor + logging hardening
- tiingo_provider.py: /tiingo/fx/top quotes (10s TTL cache, graceful None on any failure); spread log writer. Signals never consume bid/ask; warn-only by design (no historical bid/ask to validate a gate against — live == backtest rule). - app_config: repo-root .env loader (setdefault; shell env still wins), tiingo_api_key resolver (TIINGO_API_KEY/TIINGO → config.json), telegram.channel_spread_detail flag (default false). - telegram_bot: 5-min spread sampler → logs/spread_log.csv (weekend- aware), alert-time quote refresh, bid/ask + wide-spread warning in signal messages, /status spread line. Raw numbers to owner DM always; channel only while channel_spread_detail=true (Tiingo free tier is internal-use-only; flip before public launch). Free-tier limit math: worst case ~41 req/h vs 50/h cap, one pair. - Logging: 3h rotation x 240 backups (30-day retention), full-date stamps, force=True root handlers; candle loops + sampler survive failed cycles; breach-task exceptions logged via done-callback; PTB error handler; sys/threading excepthooks in main(). - test_all.py: Module 14 (tiingo, dotenv, spread routing, loop resilience, rotation) — 215 tests green. Claude-Session: https://claude.ai/code/session_01HVRmy6KspMs96o5PTPYFAB
This commit is contained in:
Harwinder Sekhon
@@ -13,9 +13,10 @@ This file is the authoritative developer/agent reference. The human-facing docs
|
||||
| Path | Lines | Purpose |
|
||||
|------|-------|---------|
|
||||
| `config.py` | ~374 | All config dataclasses, `PAIR_INFO` registry, per-pair strategy configs. **`PlatformConfig` defaults ARE the tuned XAU config.** |
|
||||
| `app_config.py` | ~244 | config.json loader/validation, env-precedence resolvers, `apply_pair_overrides` deployment overlay |
|
||||
| `config.json.example` | 24 | committed schema template; real gitignored config.json holds secrets + deployment settings |
|
||||
| `app_config.py` | ~290 | config.json loader/validation, env-precedence resolvers, repo-root `.env` loader (setdefault at import), `apply_pair_overrides` deployment overlay |
|
||||
| `config.json.example` | 27 | committed schema template; real gitignored config.json holds secrets + deployment settings |
|
||||
| `data_provider.py` | ~347 | Twelve Data SDK wrapper: fetch, rate-limit (55 credits/min), in-memory cache, validation/cleaning, freshness checks |
|
||||
| `tiingo_provider.py` | ~160 | Tiingo `/tiingo/fx/top` bid/ask quotes (supplementary, NOT an OHLCV source): spread visibility on alerts (warn-only) + 5-min sampler → `logs/spread_log.csv`. No key → feature off |
|
||||
| `engine.py` | ~1173 | `TechnicalEngine`: computes all indicators on one DataFrame → `TechnicalAnalysis` (composite score, regime, key levels) |
|
||||
| `chart_patterns.py` | ~280 | Chart-pattern detectors for the XAU_CP archetype (pure functions, ATR-unit geometry, first-close-beyond triggers) |
|
||||
| `signal_aggregator.py` | ~1447 | `SignalAggregator`: MTF analysis, news integration, gates/penalties, quality grade, trade-setup generation → `SignalReport` |
|
||||
@@ -24,13 +25,13 @@ This file is the authoritative developer/agent reference. The human-facing docs
|
||||
| `pair_signals/__init__.py` | 52 | `PAIR_PRODUCERS` registry + `produce_signal()` |
|
||||
| `pair_signals/xauusd.py` | 71 | XAU config factory + per-timeframe overrides (docstring = validation record) |
|
||||
| `main.py` | ~348 | CLI: `python main.py --pair XAUUSD --timeframe H1 --detail/--json` |
|
||||
| `test_all.py` | ~2800 | QA suite: **200 tests in 13 modules**, no live API needed. Must stay green. |
|
||||
| `test_all.py` | ~3100 | QA suite: **215 tests in 14 modules**, no live API needed. Must stay green. |
|
||||
| `backtest/gpu_backtest.py` | ~2231 | Bar-replay backtester (CuPy→NumPy fallback) using the real `SignalAggregator` |
|
||||
| `backtest/diagnose_filters.py` | 609 | Filter-diagnosis tooling |
|
||||
| `backtest/research/` | — | Calendar research infra (`ff_calendar_scraper.py`, `calendar_store.py`) + result CSVs |
|
||||
| `backtest/*.csv` | — | Verification trade books from past tuning campaigns (`xau_*.csv`, …). **Gitignored, local-only** — used for manual before/after comparison, not by tests |
|
||||
| `backtest/cache/` | — | Parquet OHLCV cache (raw bars) |
|
||||
| `logs/telegram.log` | — | Bot log (midnight rotation, 30 backups) |
|
||||
| `logs/telegram.log` | — | Bot log (3-hour rotation × 240 backups = 30-day retention); `logs/spread_log.csv` = Tiingo spread samples |
|
||||
|
||||
---
|
||||
|
||||
@@ -272,8 +273,10 @@ Run: `python telegram_bot.py` (credentials resolve env `TELEGRAM_TOKEN`/`TELEGRA
|
||||
- **Alert gating**: quality threshold (`_alert_min_quality`, default "C", menu cycles C→B→A) + fingerprint dedup per (symbol, TF) on (signal, quality, direction, entry bucket) with 1-hour re-arm — a persisting identical signal doesn't re-send every candle, but a setup that moved ≥ half its risk distance (new pullback/swing anchor) re-alerts. The fingerprint is committed only **after** a message reaches ≥1 target (`_commit_alert`), so a failed Telegram send doesn't consume the alert.
|
||||
- **WebSocket monitor**: subscribes every `scan_pairs` symbol to TD quotes WS; 10 s heartbeat; exponential reconnect (≤60 s). On tick crossing any cached key level (`swing_high/low_1..3`, `ema_trend`, `bb_upper/lower`, `kc_upper/lower`; cache TTL 1 h) → immediate M15 rescan → "LEVEL BREAK" alert if actionable+changed (same gates as auto-scan), 5-min per-pair cooldown.
|
||||
- **Commands**: `/start` (menu), `/scan` (buttons: per pair — M5/M15/H1/H4 + Scan All TFs), `/autoscan` (toggle DM copy), `/status` (loops, WS, live prices, level cache, cooldowns), `/cache` (age per pair/TF), `/clearcache`, `/help`.
|
||||
- **Tiingo spread monitor** (optional; on only when a Tiingo key resolves — see Environment variables): a 5-min sampler task fetches top-of-book bid/ask per scanned pair (`tiingo_provider.py`), logs to `logs/spread_log.csv` (skipping the weekend-closed window), and caches the latest quote; alert sends refresh the quote first, and signal messages then show live bid/ask + a warning when spread ≥ 2× the pair's assumed round-trip cost (`WIDE_SPREAD_MULTIPLE`). **Warn-only BY DESIGN** — no historical bid/ask exists, so a suppressing gate could never be validated (live == backtest rule); signals never consume bid/ask. `/status` shows the live spread per pair. **License (free tier is internal-use-only):** raw bid/ask/spread numbers go to owner DMs always; the channel gets them ONLY while config.json `telegram.channel_spread_detail=true` (the channel is internal/testing as of 2026-07-22 — **flip to false before the channel goes public**), otherwise a number-free "spread unusually wide" caution via `format_signal_message(report, spread_detail=False)` — routing in `_spread_detail_for`, test-enforced, committed default false. **Rate limits (free tier 50 req/h, 1000/day, checked 2026-07-22):** sampler 12/h + alert-batch refreshes ≤17/h + breach refreshes ≤12/h = worst case ≈41/h with ONE pair; adding pairs multiplies this — recheck before growing `scan_pairs`.
|
||||
- Signal messages include the full indicator breakdown, warnings, trade setup, **trailing-stop management plan** (arm price + trail distance, from Signal/pb params by `strategy`), and a risk disclaimer on every outbound message.
|
||||
- Blocking scan work runs in `ThreadPoolExecutor(max_workers=4)`; one shared `ForexDataProvider` across all scans (cache reuse). Logs to `logs/telegram.log`.
|
||||
- Blocking scan work runs in `ThreadPoolExecutor(max_workers=4)`; one shared `ForexDataProvider` across all scans (cache reuse).
|
||||
- **Logging (hardened 2026-07-22)**: root-logger handlers (`force=True`) → console + `logs/telegram.log` (3-hour rotation, 240 backups = 30 days), full-date timestamps; every module logger (data_provider, tiingo_provider, news_engine, websockets, telegram.ext) lands in the same file. Failure surfaces all covered: candle loops and the spread sampler catch-and-continue per cycle (one bad scan can't kill a loop), breach tasks log exceptions via done-callback, PTB `add_error_handler` registered, and `sys.excepthook`/`threading.excepthook` (installed in `main()`) route uncaught crashes to the log. WS monitor keeps its exponential-backoff reconnect + `logger.exception`.
|
||||
|
||||
---
|
||||
|
||||
@@ -318,13 +321,13 @@ Data: paginated TD fetches (5000-bar chunks) cached as parquet in `backtest/cach
|
||||
|
||||
## test_all.py — QA suite
|
||||
|
||||
`python test_all.py` — **200 tests, 13 modules** (config, data_provider, engine, news_engine, signal_aggregator, main, telegram_bot, integration, pair_signals registry + engine-bug regressions, 2026-07-16 audit-fix regressions, chart patterns, app_config). No network needed (mocked). Uses a custom `@test("name")` decorator that runs immediately (NOT pytest — don't grep for `def test_`). Exit code non-zero on failure. Includes: the xauusd-factory ≡ inline-config equivalence test, engine bug-fix regression tests (Module 9b), a live-vs-GPU engine parity sweep incl. the EMA-ribbon path (Module 11), and config.json loading/precedence/validation coverage (Module 13). **Any signal-logic change must keep this green and should add tests.**
|
||||
`python test_all.py` — **215 tests, 14 modules** (config, data_provider, engine, news_engine, signal_aggregator, main, telegram_bot, integration, pair_signals registry + engine-bug regressions, 2026-07-16 audit-fix regressions, chart patterns, app_config, tiingo_provider). No network needed (mocked). Uses a custom `@test("name")` decorator that runs immediately (NOT pytest — don't grep for `def test_`). Exit code non-zero on failure. Includes: the xauusd-factory ≡ inline-config equivalence test, engine bug-fix regression tests (Module 9b), a live-vs-GPU engine parity sweep incl. the EMA-ribbon path (Module 11), config.json loading/precedence/validation coverage (Module 13), and Tiingo quote/spread-monitor + `.env` loader coverage (Module 14). **Any signal-logic change must keep this green and should add tests.**
|
||||
|
||||
---
|
||||
|
||||
## Environment variables
|
||||
|
||||
Precedence for every credential/deployment setting: **env var → config.json → none**. Hardcoded fallbacks (previously in `config.py` / `telegram_bot.py`) were **REMOVED 2026-07-21**; the bot now refuses to boot (`telegram_bot.main`) and data fetches fail with a named error (`data_provider`) if a secret resolves to neither source. See `app_config.py` and `config.json.example`.
|
||||
Precedence for every credential/deployment setting: **env var → config.json → none**. A repo-root `.env` file (gitignored) is loaded at `app_config` import via `os.environ.setdefault` — it behaves like real env vars except a real shell export always wins. Hardcoded fallbacks (previously in `config.py` / `telegram_bot.py`) were **REMOVED 2026-07-21**; the bot now refuses to boot (`telegram_bot.main`) and data fetches fail with a named error (`data_provider`) if a secret resolves to neither source. See `app_config.py` and `config.json.example`.
|
||||
|
||||
| Variable | Purpose | Default |
|
||||
|---|---|---|
|
||||
@@ -332,6 +335,7 @@ Precedence for every credential/deployment setting: **env var → config.json
|
||||
| `TELEGRAM_TOKEN` | bot token | none — falls through to config.json `telegram.token`, then refuses to boot |
|
||||
| `TELEGRAM_CHAT_ID` | owner chat id (boot auto-scan + owner lock) | none — falls through to config.json `telegram.chat_id`, then refuses to boot |
|
||||
| `TELEGRAM_CHANNEL_ID` | subscriber channel (`-100…`); `""` disables the feed | none — falls through to config.json `telegram.channel_id` |
|
||||
| `TIINGO_API_KEY` (or `TIINGO`) | Tiingo bid/ask quote key for the spread monitor — optional | none — falls through to config.json `data_providers.tiingo.api_key`; absent = spread monitor off (never fatal) |
|
||||
| `FB2_EMA_LO` | EMA200 "near" penalty threshold in % | `0.5` |
|
||||
|
||||
Dependencies (`requirements.txt`): numpy, pandas, requests, twelvedata, python-telegram-bot ≥20, websockets. Python 3.12. A `venv/` exists in-repo.
|
||||
|
||||
+66
-4
@@ -9,9 +9,12 @@ alert-quality boot default, log level. Signal logic, weights,
|
||||
thresholds and gates stay in config.py / pair_signals.
|
||||
|
||||
Precedence for every setting: env var → config.json → code default.
|
||||
Secrets have NO code default — with neither env nor config.json the
|
||||
bot refuses to start (telegram_bot.main) and data fetches fail with a
|
||||
named error (data_provider).
|
||||
A repo-root .env file (gitignored) is loaded at import via setdefault,
|
||||
so it behaves exactly like real environment variables except that a
|
||||
real shell export always wins. Secrets have NO code default — with
|
||||
neither env nor config.json the bot refuses to start
|
||||
(telegram_bot.main) and data fetches fail with a named error
|
||||
(data_provider).
|
||||
|
||||
Parity contract (test-enforced): with no config.json, or one holding
|
||||
the config.json.example defaults, every produced PlatformConfig is
|
||||
@@ -34,6 +37,40 @@ logger = logging.getLogger(__name__)
|
||||
|
||||
CONFIG_PATH = os.path.join(
|
||||
os.path.dirname(os.path.abspath(__file__)), "config.json")
|
||||
ENV_PATH = os.path.join(
|
||||
os.path.dirname(os.path.abspath(__file__)), ".env")
|
||||
|
||||
|
||||
def load_dotenv(path: str = ENV_PATH) -> None:
|
||||
"""Minimal .env loader (no python-dotenv dependency).
|
||||
|
||||
KEY=VALUE lines land in os.environ via setdefault — a variable
|
||||
already exported in the shell always wins, keeping the
|
||||
env → config.json precedence intact (.env is just a convenient
|
||||
place to put env vars). Comments, blank lines, ``export `` prefixes
|
||||
and single/double quotes are handled; anything else is skipped.
|
||||
"""
|
||||
try:
|
||||
with open(path, encoding="utf-8") as fh:
|
||||
lines = fh.read().splitlines()
|
||||
except OSError:
|
||||
return
|
||||
for line in lines:
|
||||
line = line.strip()
|
||||
if not line or line.startswith("#") or "=" not in line:
|
||||
continue
|
||||
if line.startswith("export "):
|
||||
line = line[len("export "):]
|
||||
key, _, value = line.partition("=")
|
||||
key, value = key.strip(), value.strip()
|
||||
if len(value) >= 2 and value[0] == value[-1] and value[0] in "\"'":
|
||||
value = value[1:-1]
|
||||
if key:
|
||||
os.environ.setdefault(key, value)
|
||||
|
||||
|
||||
# Module import == process boot, before any resolver can run.
|
||||
load_dotenv()
|
||||
|
||||
_VALID_QUALITIES = ("A+", "A", "B", "C")
|
||||
_VALID_LOG_LEVELS = ("DEBUG", "INFO", "WARNING", "ERROR", "CRITICAL")
|
||||
@@ -42,7 +79,8 @@ _VALID_LOG_LEVELS = ("DEBUG", "INFO", "WARNING", "ERROR", "CRITICAL")
|
||||
_SCANNABLE_TFS = ("M5", "M15", "H1", "H4")
|
||||
|
||||
_KNOWN_TOP = {"telegram", "data_providers", "scanning", "pairs", "log_level"}
|
||||
_KNOWN_TELEGRAM = {"token", "chat_id", "channel_id", "alert_min_quality"}
|
||||
_KNOWN_TELEGRAM = {"token", "chat_id", "channel_id", "alert_min_quality",
|
||||
"channel_spread_detail"}
|
||||
_KNOWN_SCANNING = {"pairs", "timeframes"}
|
||||
_KNOWN_PAIR_KEYS = {"blocked_hours_utc"}
|
||||
|
||||
@@ -57,6 +95,11 @@ class AppConfig:
|
||||
telegram_chat_id: str = ""
|
||||
telegram_channel_id: str = "" # "" = channel feed disabled
|
||||
alert_min_quality: str = "C"
|
||||
# True = the channel is INTERNAL (testing) and may receive raw Tiingo
|
||||
# bid/ask numbers. Default False: Tiingo's free tier is licensed
|
||||
# internal-use-only, so a public/subscriber channel must only ever
|
||||
# get the number-free wide-spread caution. Flip back before launch.
|
||||
channel_spread_detail: bool = False
|
||||
active_provider: str = "twelve_data"
|
||||
provider_settings: Dict[str, Dict[str, Any]] = field(default_factory=dict)
|
||||
scan_pairs: List[str] = field(default_factory=lambda: [XAUUSD])
|
||||
@@ -105,6 +148,11 @@ def load_app_config(path: str = CONFIG_PATH) -> AppConfig:
|
||||
f'config.json: telegram.alert_min_quality "{quality}" '
|
||||
f"not in {_VALID_QUALITIES}")
|
||||
app.alert_min_quality = quality
|
||||
spread_detail = tg.get("channel_spread_detail", app.channel_spread_detail)
|
||||
if not isinstance(spread_detail, bool):
|
||||
raise AppConfigError(
|
||||
"config.json: telegram.channel_spread_detail must be a boolean")
|
||||
app.channel_spread_detail = spread_detail
|
||||
|
||||
dp = raw.get("data_providers")
|
||||
if dp is not None:
|
||||
@@ -227,6 +275,20 @@ def provider_api_key(app: Optional[AppConfig] = None) -> str:
|
||||
return str(block.get("api_key", ""))
|
||||
|
||||
|
||||
def tiingo_api_key(app: Optional[AppConfig] = None) -> str:
|
||||
"""Key for the SUPPLEMENTARY Tiingo bid/ask quote provider (spread
|
||||
monitor) — never a data_providers.active candidate; Twelve Data
|
||||
stays the OHLCV source. Empty string = feature disabled.
|
||||
Env TIINGO_API_KEY (or short TIINGO, e.g. from .env) →
|
||||
config.json data_providers.tiingo.api_key → ""."""
|
||||
app = app or get_app_config()
|
||||
env = os.environ.get("TIINGO_API_KEY") or os.environ.get("TIINGO")
|
||||
if env:
|
||||
return env
|
||||
block = app.provider_settings.get("tiingo", {})
|
||||
return str(block.get("api_key", ""))
|
||||
|
||||
|
||||
# ── PlatformConfig overlay ────────────────────────────────────────────
|
||||
|
||||
def apply_pair_overrides(
|
||||
|
||||
+5
-1
@@ -3,12 +3,16 @@
|
||||
"token": "123456789:REPLACE_WITH_YOUR_BOT_TOKEN",
|
||||
"chat_id": "REPLACE_WITH_OWNER_CHAT_ID",
|
||||
"channel_id": "-100REPLACE_WITH_CHANNEL_ID_OR_EMPTY_STRING",
|
||||
"alert_min_quality": "C"
|
||||
"alert_min_quality": "C",
|
||||
"channel_spread_detail": false
|
||||
},
|
||||
"data_providers": {
|
||||
"active": "twelve_data",
|
||||
"twelve_data": {
|
||||
"api_key": "REPLACE_WITH_YOUR_TWELVE_DATA_API_KEY"
|
||||
},
|
||||
"tiingo": {
|
||||
"api_key": ""
|
||||
}
|
||||
},
|
||||
"scanning": {
|
||||
|
||||
+204
-13
@@ -43,6 +43,7 @@ import logging
|
||||
import logging.handlers
|
||||
import os
|
||||
import sys
|
||||
import threading
|
||||
import time
|
||||
from concurrent.futures import ThreadPoolExecutor
|
||||
from datetime import datetime, timezone
|
||||
@@ -66,9 +67,12 @@ from config import (
|
||||
)
|
||||
from app_config import (
|
||||
get_app_config, provider_api_key, telegram_token, telegram_chat_id,
|
||||
telegram_channel_id,
|
||||
telegram_channel_id, tiingo_api_key,
|
||||
)
|
||||
from data_provider import ForexDataProvider, make_data_provider
|
||||
from tiingo_provider import (
|
||||
Quote, TiingoQuoteProvider, WIDE_SPREAD_MULTIPLE, log_quote,
|
||||
)
|
||||
from pair_signals import PAIR_PRODUCERS
|
||||
from signal_aggregator import SignalAggregator, SignalReport
|
||||
|
||||
@@ -81,17 +85,24 @@ if hasattr(sys.stdout, "reconfigure"):
|
||||
_LOG_DIR = os.path.join(os.path.dirname(os.path.abspath(__file__)), "logs")
|
||||
os.makedirs(_LOG_DIR, exist_ok=True)
|
||||
_log_formatter = logging.Formatter(
|
||||
"%(asctime)s [%(levelname)s] %(name)s: %(message)s", datefmt="%H:%M:%S"
|
||||
"%(asctime)s [%(levelname)s] %(name)s: %(message)s",
|
||||
datefmt="%Y-%m-%d %H:%M:%S",
|
||||
)
|
||||
_console_handler = logging.StreamHandler()
|
||||
_console_handler.setFormatter(_log_formatter)
|
||||
# 3-hour rotation × 240 backups = 30 days retention. Handlers sit on the
|
||||
# ROOT logger (force=True guarantees ownership even if something logged
|
||||
# first), so every module logger — data_provider, tiingo_provider,
|
||||
# news_engine, signal_aggregator, websockets, telegram.ext — lands in
|
||||
# the same file.
|
||||
_file_handler = logging.handlers.TimedRotatingFileHandler(
|
||||
os.path.join(_LOG_DIR, "telegram.log"),
|
||||
when="midnight", backupCount=30, encoding="utf-8",
|
||||
when="h", interval=3, backupCount=240, encoding="utf-8",
|
||||
)
|
||||
_file_handler.setFormatter(_log_formatter)
|
||||
logging.basicConfig(level=getattr(logging, _APP.log_level, logging.INFO),
|
||||
handlers=[_console_handler, _file_handler])
|
||||
handlers=[_console_handler, _file_handler],
|
||||
force=True)
|
||||
logging.getLogger("httpx").setLevel(logging.WARNING)
|
||||
logging.getLogger("httpcore").setLevel(logging.WARNING)
|
||||
logger = logging.getLogger("telegram_bot")
|
||||
@@ -202,6 +213,32 @@ def _get_shared_dp() -> ForexDataProvider:
|
||||
return _shared_dp
|
||||
|
||||
|
||||
# ── Tiingo bid/ask spread monitor (optional; no key → feature off) ───
|
||||
# Warn-only BY DESIGN: there is no historical bid/ask to replay, so a
|
||||
# suppressing gate could never be validated (live == backtest rule).
|
||||
# Signals never consume bid/ask — this is execution-cost visibility.
|
||||
|
||||
_TIINGO_KEY = tiingo_api_key()
|
||||
_tiingo: Optional[TiingoQuoteProvider] = (
|
||||
TiingoQuoteProvider(_TIINGO_KEY) if _TIINGO_KEY else None
|
||||
)
|
||||
_last_quotes: Dict[str, Quote] = {}
|
||||
_QUOTE_MAX_AGE = 600 # cached quote stays displayable this long
|
||||
_SPREAD_SAMPLE_SECONDS = 300 # sampler cadence (→ logs/spread_log.csv)
|
||||
|
||||
# config.json telegram.channel_spread_detail: True while the channel is
|
||||
# internal/testing → it gets raw bid/ask like the owner DM. MUST be
|
||||
# flipped back to false before the channel goes public (Tiingo free
|
||||
# tier is internal-use-only).
|
||||
_CHANNEL_SPREAD_DETAIL = _APP.channel_spread_detail
|
||||
|
||||
|
||||
def _spread_detail_for(chat_id: int) -> bool:
|
||||
"""Owner DMs always get raw quote numbers; the channel only while
|
||||
config.json marks it internal."""
|
||||
return chat_id != CHANNEL_ID or _CHANNEL_SPREAD_DETAIL
|
||||
|
||||
|
||||
# ── Signal formatting ─────────────────────────────────────────────────
|
||||
|
||||
_SIGNAL_EMOJI = {
|
||||
@@ -257,6 +294,37 @@ def _live_price_line(symbol: str) -> Optional[str]:
|
||||
return f"📡 Live: *{_fmt_price(price)}* ({age:.0f}s tick)"
|
||||
|
||||
|
||||
def _spread_lines(symbol: str, include_quote: bool = True) -> List[str]:
|
||||
"""Live bid/ask + wide-spread warning for signal messages.
|
||||
Empty when Tiingo is off or the cached quote is stale.
|
||||
|
||||
include_quote=False is the subscriber-channel variant: Tiingo's
|
||||
free tier is licensed internal-use-only (no redistributing the
|
||||
data), so raw bid/ask/spread numbers stay in the owner DM — the
|
||||
channel gets only a number-free wide-spread caution."""
|
||||
q = _last_quotes.get(symbol)
|
||||
if q is None or q.age_seconds() > _QUOTE_MAX_AGE:
|
||||
return []
|
||||
assumed = PAIR_INFO.get(symbol, {}).get("default_spread", 0.0)
|
||||
wide = assumed > 0 and q.spread >= WIDE_SPREAD_MULTIPLE * assumed
|
||||
if not include_quote:
|
||||
return [
|
||||
"⚠️ Spread unusually wide right now — execution cost is "
|
||||
"elevated, consider waiting for it to normalize"
|
||||
] if wide else []
|
||||
lines = [
|
||||
f"💱 Bid/Ask: {_fmt_price(q.bid)} / {_fmt_price(q.ask)}"
|
||||
f" (spread ${q.spread:.2f})"
|
||||
]
|
||||
if wide:
|
||||
lines.append(
|
||||
f"⚠️ Wide spread: ${q.spread:.2f} ≈ {q.spread / assumed:.1f}× the "
|
||||
f"${assumed:.2f} cost the strategy was validated at — "
|
||||
f"execution is expensive right now"
|
||||
)
|
||||
return lines
|
||||
|
||||
|
||||
_DISCLAIMER = (
|
||||
"⚠️ _Not financial advice. For educational and research purposes only. "
|
||||
"Trading forex/CFDs carries a high risk of loss — do your own analysis "
|
||||
@@ -264,7 +332,7 @@ _DISCLAIMER = (
|
||||
)
|
||||
|
||||
|
||||
def format_signal_message(report: SignalReport) -> str:
|
||||
def format_signal_message(report: SignalReport, spread_detail: bool = True) -> str:
|
||||
sig = report.final_signal
|
||||
emoji = _SIGNAL_EMOJI.get(sig, "⬜")
|
||||
qe = _QUALITY_EMOJI.get(report.signal_quality, "")
|
||||
@@ -285,6 +353,7 @@ def format_signal_message(report: SignalReport) -> str:
|
||||
live = _live_price_line(report.symbol)
|
||||
if live:
|
||||
lines.append(live)
|
||||
lines += _spread_lines(report.symbol, include_quote=spread_detail)
|
||||
lines += [
|
||||
f"📊 Regime: {regime_str} | Vol: {vol_str}",
|
||||
"",
|
||||
@@ -477,6 +546,24 @@ async def run_signal(
|
||||
)
|
||||
|
||||
|
||||
async def _refresh_quotes(symbols: List[str]) -> None:
|
||||
"""Fetch fresh Tiingo quotes (blocking HTTP) off the event loop —
|
||||
called just before alerts go out so the spread block reflects the
|
||||
moment of entry, and by the 5-min sampler."""
|
||||
if _tiingo is None:
|
||||
return
|
||||
loop = asyncio.get_running_loop()
|
||||
for symbol in set(symbols):
|
||||
try:
|
||||
quote = await loop.run_in_executor(
|
||||
_executor, _tiingo.get_quote, symbol)
|
||||
except Exception as exc:
|
||||
logger.warning(f"Tiingo quote refresh failed for {symbol}: {exc}")
|
||||
continue
|
||||
if quote is not None:
|
||||
_last_quotes[symbol] = quote
|
||||
|
||||
|
||||
# ── Alert gating (quality threshold + dedup) ─────────────────────────
|
||||
|
||||
_QUALITY_RANK = {"A+": 4, "A": 3, "B": 2, "C": 1}
|
||||
@@ -589,10 +676,13 @@ async def _scan_timeframe(
|
||||
if _passes_quality(r.signal_quality)
|
||||
and _should_alert(f"{r.symbol}_{tf.value}", r)
|
||||
]
|
||||
if actionable:
|
||||
await _refresh_quotes([r.symbol for r in actionable])
|
||||
sent_ok: Set[int] = set()
|
||||
for chat_id in _signal_targets():
|
||||
for i, report in enumerate(actionable):
|
||||
detail = format_signal_message(report)
|
||||
detail = format_signal_message(
|
||||
report, spread_detail=_spread_detail_for(chat_id))
|
||||
try:
|
||||
await bot.send_message(chat_id, detail, parse_mode=ParseMode.MARKDOWN)
|
||||
sent_ok.add(i)
|
||||
@@ -690,9 +780,11 @@ async def _handle_breach(bot, ws_symbol: str, pair_symbol: str, price: float, le
|
||||
f"⚡ *LEVEL BREAK* — {_escape_md(display)} crossed "
|
||||
f"*{_escape_md(label)}* @ `{_fmt_price(price)}`\n\n"
|
||||
)
|
||||
detail = header + format_signal_message(report)
|
||||
await _refresh_quotes([pair_symbol])
|
||||
sent = False
|
||||
for chat_id in _signal_targets():
|
||||
detail = header + format_signal_message(
|
||||
report, spread_detail=_spread_detail_for(chat_id))
|
||||
try:
|
||||
await bot.send_message(chat_id, detail, parse_mode=ParseMode.MARKDOWN)
|
||||
sent = True
|
||||
@@ -702,6 +794,15 @@ async def _handle_breach(bot, ws_symbol: str, pair_symbol: str, price: float, le
|
||||
_commit_alert(f"{pair_symbol}_{Timeframe.M15.value}", report)
|
||||
|
||||
|
||||
def _breach_task_done(task: asyncio.Task) -> None:
|
||||
"""Fire-and-forget breach tasks must still surface their failures —
|
||||
without this, an exception sits unretrieved until GC and never
|
||||
reaches the log file."""
|
||||
_breach_tasks.discard(task)
|
||||
if not task.cancelled() and task.exception() is not None:
|
||||
logger.error("Breach alert task failed", exc_info=task.exception())
|
||||
|
||||
|
||||
async def _ws_heartbeat(ws) -> None:
|
||||
"""Twelve Data drops idle WS connections; heartbeat every 10s keeps it
|
||||
alive through quiet periods (weekends, low-liquidity hours)."""
|
||||
@@ -764,7 +865,7 @@ async def _ws_monitor(bot) -> None:
|
||||
_handle_breach(bot, ws_symbol, pair_symbol, price, breached)
|
||||
)
|
||||
_breach_tasks.add(task)
|
||||
task.add_done_callback(_breach_tasks.discard)
|
||||
task.add_done_callback(_breach_task_done)
|
||||
|
||||
_cleanup_cooldowns()
|
||||
|
||||
@@ -791,15 +892,51 @@ async def _candle_loop(bot, tf: Timeframe, interval_minutes: int) -> None:
|
||||
if not _feed_active():
|
||||
break
|
||||
logger.info(f"{label} auto-scan: firing")
|
||||
await _scan_timeframe(
|
||||
bot, tf, AUTOSCAN_CONFIRMATIONS[tf]
|
||||
)
|
||||
try:
|
||||
await _scan_timeframe(
|
||||
bot, tf, AUTOSCAN_CONFIRMATIONS[tf]
|
||||
)
|
||||
except Exception:
|
||||
# One failed cycle must not kill the loop for good —
|
||||
# log it and wait for the next candle boundary.
|
||||
logger.exception(f"{label} auto-scan: cycle failed")
|
||||
except asyncio.CancelledError:
|
||||
pass
|
||||
finally:
|
||||
logger.info(f"{label} auto-scan loop stopped")
|
||||
|
||||
|
||||
def _weekend_closed(now: datetime) -> bool:
|
||||
"""Fri 22:00 → Sun 22:00 UTC — the window the platform treats as
|
||||
market-closed (same as the weekend filter)."""
|
||||
wd, hr = now.weekday(), now.hour
|
||||
return (wd == 4 and hr >= 22) or wd == 5 or (wd == 6 and hr < 22)
|
||||
|
||||
|
||||
async def _spread_sampler_loop() -> None:
|
||||
"""Every 5 min: fetch bid/ask for each scanned pair, cache it for
|
||||
the alert formatter, and append to logs/spread_log.csv — the
|
||||
dataset that validates (or refutes) the assumed $0.30 round-trip
|
||||
cost every backtest charges."""
|
||||
try:
|
||||
while True:
|
||||
try:
|
||||
if not _weekend_closed(datetime.now(timezone.utc)):
|
||||
await _refresh_quotes(SCAN_PAIRS)
|
||||
for symbol in SCAN_PAIRS:
|
||||
q = _last_quotes.get(symbol)
|
||||
# Age guard: never log a stale row after a failed fetch.
|
||||
if q is not None and q.age_seconds() < _SPREAD_SAMPLE_SECONDS:
|
||||
log_quote(q)
|
||||
except Exception:
|
||||
logger.exception("Spread sampler: cycle failed")
|
||||
await asyncio.sleep(_SPREAD_SAMPLE_SECONDS)
|
||||
except asyncio.CancelledError:
|
||||
pass
|
||||
finally:
|
||||
logger.info("Spread sampler stopped")
|
||||
|
||||
|
||||
_TF_INTERVAL_MINUTES = {
|
||||
Timeframe.M5: 5, Timeframe.M15: 15, Timeframe.M30: 30,
|
||||
Timeframe.H1: 60, Timeframe.H4: 240, Timeframe.D1: 1440,
|
||||
@@ -810,10 +947,11 @@ _AUTOSCAN_LOOPS = [
|
||||
|
||||
|
||||
_ws_task: Optional[asyncio.Task] = None
|
||||
_spread_task: Optional[asyncio.Task] = None
|
||||
|
||||
|
||||
def _start_scheduler(bot) -> None:
|
||||
global _ws_task
|
||||
global _ws_task, _spread_task
|
||||
alive = [t for t in _autoscan_tasks if not t.done()]
|
||||
if not alive:
|
||||
_autoscan_tasks.clear()
|
||||
@@ -828,15 +966,23 @@ def _start_scheduler(bot) -> None:
|
||||
_ws_task = asyncio.create_task(_ws_monitor(bot))
|
||||
logger.info("WebSocket level-breach monitor started")
|
||||
|
||||
if _tiingo is not None and (_spread_task is None or _spread_task.done()):
|
||||
_spread_task = asyncio.create_task(_spread_sampler_loop())
|
||||
logger.info("Tiingo spread sampler started (every %ss)",
|
||||
_SPREAD_SAMPLE_SECONDS)
|
||||
|
||||
|
||||
def _stop_scheduler() -> None:
|
||||
global _ws_task
|
||||
global _ws_task, _spread_task
|
||||
for t in _autoscan_tasks:
|
||||
t.cancel()
|
||||
_autoscan_tasks.clear()
|
||||
if _ws_task and not _ws_task.done():
|
||||
_ws_task.cancel()
|
||||
_ws_task = None
|
||||
if _spread_task and not _spread_task.done():
|
||||
_spread_task.cancel()
|
||||
_spread_task = None
|
||||
for t in list(_breach_tasks):
|
||||
t.cancel()
|
||||
_breach_tasks.clear()
|
||||
@@ -934,6 +1080,17 @@ def _status_text() -> str:
|
||||
else:
|
||||
lines.append("❌ WebSocket: *disconnected*")
|
||||
|
||||
if _tiingo is not None:
|
||||
for pair in SCAN_PAIRS:
|
||||
q = _last_quotes.get(pair)
|
||||
display = PAIR_INFO.get(pair, {}).get("display", pair)
|
||||
if q is None:
|
||||
lines.append(f"💱 Spread ({display}): waiting for first quote")
|
||||
else:
|
||||
lines.append(
|
||||
f"💱 Spread ({display}): `${q.spread:.2f}` "
|
||||
f"({q.age_seconds():.0f}s ago)")
|
||||
|
||||
if _last_prices:
|
||||
lines.append("\n*Live Prices (WebSocket):*")
|
||||
for ws_sym, pair in _WS_TO_PAIR.items():
|
||||
@@ -1266,6 +1423,38 @@ async def _post_shutdown(app: Application) -> None:
|
||||
|
||||
# ── Entry point ───────────────────────────────────────────────────────
|
||||
|
||||
async def _on_error(update: object, context: ContextTypes.DEFAULT_TYPE) -> None:
|
||||
"""python-telegram-bot routes every unhandled handler exception here;
|
||||
without a registered error handler PTB only half-logs them."""
|
||||
logger.error(
|
||||
"Unhandled error in Telegram handler (update=%s)",
|
||||
getattr(update, "update_id", update),
|
||||
exc_info=context.error,
|
||||
)
|
||||
|
||||
|
||||
def _install_excepthooks() -> None:
|
||||
"""Route uncaught exceptions from the main thread and worker threads
|
||||
into the rotating log file instead of bare stderr, so a crash right
|
||||
before exit is still recorded."""
|
||||
def hook(exc_type, exc, tb):
|
||||
if issubclass(exc_type, KeyboardInterrupt):
|
||||
sys.__excepthook__(exc_type, exc, tb)
|
||||
return
|
||||
logger.critical("Uncaught exception", exc_info=(exc_type, exc, tb))
|
||||
|
||||
def thread_hook(args):
|
||||
if issubclass(args.exc_type, SystemExit):
|
||||
return
|
||||
name = args.thread.name if args.thread else "?"
|
||||
logger.critical(
|
||||
f"Uncaught exception in thread {name}",
|
||||
exc_info=(args.exc_type, args.exc_value, args.exc_traceback))
|
||||
|
||||
sys.excepthook = hook
|
||||
threading.excepthook = thread_hook
|
||||
|
||||
|
||||
def main() -> None:
|
||||
if not TOKEN or not CHAT_ID:
|
||||
raise SystemExit(
|
||||
@@ -1273,6 +1462,7 @@ def main() -> None:
|
||||
"and telegram.chat_id in config.json, or the TELEGRAM_TOKEN and "
|
||||
"TELEGRAM_CHAT_ID environment variables.")
|
||||
|
||||
_install_excepthooks()
|
||||
logger.info("Starting ForexBot2 Telegram bot…")
|
||||
|
||||
app = (
|
||||
@@ -1292,6 +1482,7 @@ def main() -> None:
|
||||
app.add_handler(CommandHandler("clearcache", cmd_clearcache))
|
||||
app.add_handler(CallbackQueryHandler(cb_scan, pattern="^scan_"))
|
||||
app.add_handler(CallbackQueryHandler(cb_menu, pattern="^menu_"))
|
||||
app.add_error_handler(_on_error)
|
||||
|
||||
logger.info("Bot running — press Ctrl+C to stop")
|
||||
app.run_polling(allowed_updates=Update.ALL_TYPES)
|
||||
|
||||
+290
-2
@@ -25,7 +25,7 @@ sys.path.insert(0, os.path.dirname(__file__))
|
||||
import json
|
||||
import app_config
|
||||
for _k in ("TELEGRAM_TOKEN", "TELEGRAM_CHAT_ID", "TELEGRAM_CHANNEL_ID",
|
||||
"TWELVE_DATA_API_KEY"):
|
||||
"TWELVE_DATA_API_KEY", "TIINGO_API_KEY", "TIINGO"):
|
||||
os.environ.pop(_k, None)
|
||||
_TEST_APP = app_config.AppConfig(
|
||||
provider_settings={"twelve_data": {"api_key": "test-key"}}
|
||||
@@ -2607,7 +2607,7 @@ def _write_json(payload) -> str:
|
||||
|
||||
_FULL_CFG = {
|
||||
"telegram": {"token": "t0k", "chat_id": "111", "channel_id": "-100222",
|
||||
"alert_min_quality": "B"},
|
||||
"alert_min_quality": "B", "channel_spread_detail": True},
|
||||
"data_providers": {"active": "twelve_data",
|
||||
"twelve_data": {"api_key": "json-key"}},
|
||||
"scanning": {"pairs": ["XAU/USD"], "timeframes": ["M15", "H1"]},
|
||||
@@ -2623,6 +2623,7 @@ def _():
|
||||
assert app.telegram_chat_id == "111"
|
||||
assert app.telegram_channel_id == "-100222"
|
||||
assert app.alert_min_quality == "B"
|
||||
assert app.channel_spread_detail is True
|
||||
assert app.active_provider == "twelve_data"
|
||||
assert app.provider_settings["twelve_data"]["api_key"] == "json-key"
|
||||
assert app.scan_pairs == [XAUUSD]
|
||||
@@ -2676,6 +2677,7 @@ def _():
|
||||
{"data_providers": {"active": "polygon",
|
||||
"twelve_data": {"api_key": "k"}}}, # no block
|
||||
{"telegram": {"alert_min_quality": "Z"}}, # bad quality
|
||||
{"telegram": {"channel_spread_detail": "yes"}}, # not a bool
|
||||
{"log_level": "CHATTY"}, # bad level
|
||||
]
|
||||
for payload in bad_payloads:
|
||||
@@ -2857,6 +2859,292 @@ def _():
|
||||
Timeframe.H1, Timeframe.H4]
|
||||
|
||||
|
||||
# ══════════════════════════════════════════════════════════════════════
|
||||
# MODULE 14: tiingo_provider.py (bid/ask spread monitor)
|
||||
# ══════════════════════════════════════════════════════════════════════
|
||||
print("\n═══ Module 14: tiingo_provider.py ═══")
|
||||
|
||||
from tiingo_provider import (
|
||||
Quote, TiingoQuoteProvider, WIDE_SPREAD_MULTIPLE, log_quote,
|
||||
tiingo_ticker,
|
||||
)
|
||||
from app_config import load_dotenv, tiingo_api_key
|
||||
|
||||
|
||||
def _tiingo_response(payload):
|
||||
resp = MagicMock()
|
||||
resp.json.return_value = payload
|
||||
resp.raise_for_status.return_value = None
|
||||
return resp
|
||||
|
||||
|
||||
_TIINGO_ROW = [{
|
||||
"ticker": "xauusd",
|
||||
"quoteTimestamp": "2026-07-22T12:00:00+00:00",
|
||||
"bidPrice": 2400.10, "bidSize": 1000000.0,
|
||||
"askPrice": 2400.40, "askSize": 1000000.0,
|
||||
"midPrice": 2400.25,
|
||||
}]
|
||||
|
||||
|
||||
@test("tiingo: ticker mapping XAU/USD → xauusd")
|
||||
def _():
|
||||
assert tiingo_ticker("XAU/USD") == "xauusd"
|
||||
assert tiingo_ticker("EUR/USD") == "eurusd"
|
||||
|
||||
|
||||
@test("tiingo: quote parsed — spread = ask − bid, canonical symbol")
|
||||
def _():
|
||||
provider = TiingoQuoteProvider("k")
|
||||
with patch("tiingo_provider.requests.get",
|
||||
return_value=_tiingo_response(_TIINGO_ROW)) as mock_get:
|
||||
q = provider.get_quote("XAU/USD")
|
||||
assert q is not None
|
||||
assert q.symbol == "XAU/USD"
|
||||
assert abs(q.bid - 2400.10) < 1e-9 and abs(q.ask - 2400.40) < 1e-9
|
||||
assert abs(q.spread - 0.30) < 1e-9
|
||||
assert abs(q.mid - 2400.25) < 1e-9
|
||||
assert q.quote_time is not None and q.quote_time.tzinfo is not None
|
||||
assert q.age_seconds() < 5
|
||||
params = mock_get.call_args.kwargs["params"]
|
||||
assert params["tickers"] == "xauusd" and params["token"] == "k"
|
||||
|
||||
|
||||
@test("tiingo: mid falls back to (bid+ask)/2 when midPrice missing")
|
||||
def _():
|
||||
row = [dict(_TIINGO_ROW[0], midPrice=None)]
|
||||
provider = TiingoQuoteProvider("k")
|
||||
with patch("tiingo_provider.requests.get",
|
||||
return_value=_tiingo_response(row)):
|
||||
q = provider.get_quote("XAU/USD")
|
||||
assert q is not None and abs(q.mid - 2400.25) < 1e-9
|
||||
|
||||
|
||||
@test("tiingo: TTL cache dedupes calls; ttl=0 refetches")
|
||||
def _():
|
||||
provider = TiingoQuoteProvider("k")
|
||||
with patch("tiingo_provider.requests.get",
|
||||
return_value=_tiingo_response(_TIINGO_ROW)) as mock_get:
|
||||
provider.get_quote("XAU/USD")
|
||||
provider.get_quote("XAU/USD")
|
||||
assert mock_get.call_count == 1, "second call within TTL must hit cache"
|
||||
provider = TiingoQuoteProvider("k", quote_ttl=0.0)
|
||||
with patch("tiingo_provider.requests.get",
|
||||
return_value=_tiingo_response(_TIINGO_ROW)) as mock_get:
|
||||
provider.get_quote("XAU/USD")
|
||||
provider.get_quote("XAU/USD")
|
||||
assert mock_get.call_count == 2
|
||||
|
||||
|
||||
@test("tiingo: every failure path returns None (never raises)")
|
||||
def _():
|
||||
provider = TiingoQuoteProvider("k", quote_ttl=0.0)
|
||||
# network error
|
||||
with patch("tiingo_provider.requests.get",
|
||||
side_effect=Exception("boom")):
|
||||
assert provider.get_quote("XAU/USD") is None
|
||||
# HTTP error
|
||||
resp = _tiingo_response(_TIINGO_ROW)
|
||||
resp.raise_for_status.side_effect = Exception("401")
|
||||
with patch("tiingo_provider.requests.get", return_value=resp):
|
||||
assert provider.get_quote("XAU/USD") is None
|
||||
# empty / non-list / null-price payloads
|
||||
for payload in ([], {"detail": "unknown ticker"},
|
||||
[dict(_TIINGO_ROW[0], bidPrice=None)],
|
||||
[dict(_TIINGO_ROW[0], askPrice=None)]):
|
||||
with patch("tiingo_provider.requests.get",
|
||||
return_value=_tiingo_response(payload)):
|
||||
assert provider.get_quote("XAU/USD") is None, \
|
||||
f"payload {payload!r} must yield None"
|
||||
|
||||
|
||||
@test("tiingo: log_quote writes header once then appends rows")
|
||||
def _():
|
||||
q = Quote(symbol=XAUUSD, bid=2400.10, ask=2400.40, mid=2400.25,
|
||||
spread=0.30, quote_time=None, fetched_at=time.time())
|
||||
with tempfile.TemporaryDirectory() as td:
|
||||
path = os.path.join(td, "sub", "spread_log.csv")
|
||||
log_quote(q, path)
|
||||
log_quote(q, path)
|
||||
with open(path, encoding="utf-8") as fh:
|
||||
lines = fh.read().strip().splitlines()
|
||||
assert lines[0] == "ts_utc,symbol,bid,ask,mid,spread"
|
||||
assert len(lines) == 3, "header + one row per log_quote call"
|
||||
cells = lines[1].split(",")
|
||||
assert cells[1] == XAUUSD and float(cells[5]) == 0.30
|
||||
|
||||
|
||||
@test("app_config: tiingo_api_key precedence env → env-short → json → ''")
|
||||
def _():
|
||||
app = AppConfig(provider_settings={
|
||||
"twelve_data": {"api_key": "td"},
|
||||
"tiingo": {"api_key": "json-tiingo"},
|
||||
})
|
||||
with patch.dict(os.environ, {}, clear=True):
|
||||
assert tiingo_api_key(app) == "json-tiingo"
|
||||
with patch.dict(os.environ, {"TIINGO": "short-env"}, clear=True):
|
||||
assert tiingo_api_key(app) == "short-env"
|
||||
with patch.dict(os.environ, {"TIINGO": "short-env",
|
||||
"TIINGO_API_KEY": "long-env"}, clear=True):
|
||||
assert tiingo_api_key(app) == "long-env"
|
||||
with patch.dict(os.environ, {}, clear=True):
|
||||
assert tiingo_api_key(AppConfig()) == "", "no key anywhere = disabled"
|
||||
|
||||
|
||||
@test("app_config: load_dotenv parses lines, never overrides real env")
|
||||
def _():
|
||||
fh = tempfile.NamedTemporaryFile(
|
||||
"w", suffix=".env", delete=False, encoding="utf-8")
|
||||
fh.write("# comment\n\nPLAIN=abc\nexport EXPORTED=def\n"
|
||||
'QUOTED="with spaces"\nSINGLE=\'sq\'\nEXISTING=from-dotenv\n'
|
||||
"NOT A KV LINE\n")
|
||||
fh.close()
|
||||
with patch.dict(os.environ, {"EXISTING": "from-shell"}, clear=True):
|
||||
load_dotenv(fh.name)
|
||||
assert os.environ["PLAIN"] == "abc"
|
||||
assert os.environ["EXPORTED"] == "def"
|
||||
assert os.environ["QUOTED"] == "with spaces"
|
||||
assert os.environ["SINGLE"] == "sq"
|
||||
assert os.environ["EXISTING"] == "from-shell", \
|
||||
"a real shell export must beat .env"
|
||||
load_dotenv("/nonexistent/.env") # missing file is a silent no-op
|
||||
|
||||
|
||||
@test("telegram_bot: _spread_lines fresh/wide/stale/absent")
|
||||
def _():
|
||||
import telegram_bot as tb
|
||||
try:
|
||||
# fresh, normal spread → bid/ask line only
|
||||
tb._last_quotes[XAUUSD] = Quote(
|
||||
symbol=XAUUSD, bid=2400.10, ask=2400.40, mid=2400.25,
|
||||
spread=0.30, quote_time=None, fetched_at=time.time())
|
||||
lines = tb._spread_lines(XAUUSD)
|
||||
assert len(lines) == 1 and "Bid/Ask" in lines[0]
|
||||
assert "spread $0.30" in lines[0]
|
||||
# wide spread (≥ 2× the $0.30 assumption) → warning appended
|
||||
tb._last_quotes[XAUUSD] = Quote(
|
||||
symbol=XAUUSD, bid=2400.10, ask=2401.00, mid=2400.55,
|
||||
spread=0.90, quote_time=None, fetched_at=time.time())
|
||||
lines = tb._spread_lines(XAUUSD)
|
||||
assert len(lines) == 2 and "Wide spread" in lines[1]
|
||||
assert "3.0×" in lines[1]
|
||||
# stale quote → nothing
|
||||
tb._last_quotes[XAUUSD] = Quote(
|
||||
symbol=XAUUSD, bid=2400.10, ask=2400.40, mid=2400.25,
|
||||
spread=0.30, quote_time=None,
|
||||
fetched_at=time.time() - tb._QUOTE_MAX_AGE - 1)
|
||||
assert tb._spread_lines(XAUUSD) == []
|
||||
# no quote at all → nothing
|
||||
tb._last_quotes.pop(XAUUSD, None)
|
||||
assert tb._spread_lines(XAUUSD) == []
|
||||
finally:
|
||||
tb._last_quotes.pop(XAUUSD, None)
|
||||
|
||||
|
||||
@test("telegram_bot: channel variant never leaks Tiingo numbers")
|
||||
def _():
|
||||
# Tiingo free tier is internal-use-only: the subscriber channel may
|
||||
# see a number-free caution, never bid/ask/spread values.
|
||||
import telegram_bot as tb
|
||||
try:
|
||||
# normal spread → channel variant shows nothing at all
|
||||
tb._last_quotes[XAUUSD] = Quote(
|
||||
symbol=XAUUSD, bid=2400.10, ask=2400.40, mid=2400.25,
|
||||
spread=0.30, quote_time=None, fetched_at=time.time())
|
||||
assert tb._spread_lines(XAUUSD, include_quote=False) == []
|
||||
# wide spread → one caution line with no prices in it
|
||||
tb._last_quotes[XAUUSD] = Quote(
|
||||
symbol=XAUUSD, bid=2400.10, ask=2401.00, mid=2400.55,
|
||||
spread=0.90, quote_time=None, fetched_at=time.time())
|
||||
lines = tb._spread_lines(XAUUSD, include_quote=False)
|
||||
assert len(lines) == 1 and "unusually wide" in lines[0]
|
||||
assert "2400" not in lines[0] and "0.90" not in lines[0] \
|
||||
and "$" not in lines[0], "channel line must carry no data values"
|
||||
# format_signal_message plumbs the flag through
|
||||
config = PlatformConfig()
|
||||
agg = SignalAggregator(config)
|
||||
dfs = {
|
||||
Timeframe.H1: make_ohlcv(300, seed=42),
|
||||
Timeframe.M15: make_ohlcv(300, seed=43),
|
||||
Timeframe.H4: make_ohlcv(300, seed=44),
|
||||
}
|
||||
report = agg.analyze_from_dataframes("XAU/USD", dfs, use_news=False)
|
||||
owner_msg = tb.format_signal_message(report, spread_detail=True)
|
||||
channel_msg = tb.format_signal_message(report, spread_detail=False)
|
||||
assert "Bid/Ask" in owner_msg
|
||||
assert "Bid/Ask" not in channel_msg
|
||||
finally:
|
||||
tb._last_quotes.pop(XAUUSD, None)
|
||||
|
||||
|
||||
@test("telegram_bot: _spread_detail_for routes by chat + internal flag")
|
||||
def _():
|
||||
import telegram_bot as tb
|
||||
saved_channel, saved_flag = tb.CHANNEL_ID, tb._CHANNEL_SPREAD_DETAIL
|
||||
try:
|
||||
tb.CHANNEL_ID = -100222
|
||||
tb._CHANNEL_SPREAD_DETAIL = False # default: channel is public
|
||||
assert tb._spread_detail_for(111) is True, "owner DM gets raw quotes"
|
||||
assert tb._spread_detail_for(-100222) is False, \
|
||||
"public channel must get the number-free variant"
|
||||
tb._CHANNEL_SPREAD_DETAIL = True # internal-testing channel
|
||||
assert tb._spread_detail_for(-100222) is True
|
||||
assert tb._spread_detail_for(111) is True
|
||||
finally:
|
||||
tb.CHANNEL_ID, tb._CHANNEL_SPREAD_DETAIL = saved_channel, saved_flag
|
||||
|
||||
|
||||
@test("telegram_bot: log rotation is 3-hourly with 30-day retention")
|
||||
def _():
|
||||
import logging.handlers
|
||||
import telegram_bot as tb
|
||||
h = tb._file_handler
|
||||
assert isinstance(h, logging.handlers.TimedRotatingFileHandler)
|
||||
assert h.when.upper() == "H" and h.interval == 3 * 3600, \
|
||||
"rotation must fire every 3 hours"
|
||||
assert h.backupCount == 240, "240 × 3h = 30-day retention"
|
||||
|
||||
|
||||
@test("telegram_bot: candle loop survives a failing scan cycle")
|
||||
def _():
|
||||
import asyncio
|
||||
import telegram_bot as tb
|
||||
calls = {"n": 0}
|
||||
|
||||
async def boom(bot, tf, confirm):
|
||||
calls["n"] += 1
|
||||
if calls["n"] == 1:
|
||||
raise RuntimeError("scan blew up")
|
||||
tb._autoscan_chats.clear() # let the loop exit after call 2
|
||||
|
||||
tb._autoscan_chats.add(1)
|
||||
try:
|
||||
with patch.object(tb, "_scan_timeframe", boom), \
|
||||
patch.object(tb, "_seconds_until_boundary",
|
||||
lambda interval, buffer=10: 0.0):
|
||||
asyncio.run(tb._candle_loop(None, Timeframe.M15, 15))
|
||||
assert calls["n"] == 2, \
|
||||
"loop must keep firing after an exception, not die on it"
|
||||
finally:
|
||||
tb._autoscan_chats.clear()
|
||||
|
||||
|
||||
@test("telegram_bot: no Tiingo key → provider off, message unchanged")
|
||||
def _():
|
||||
import telegram_bot as tb
|
||||
assert tb._tiingo is None, "hermetic env has no Tiingo key"
|
||||
config = PlatformConfig()
|
||||
agg = SignalAggregator(config)
|
||||
dfs = {
|
||||
Timeframe.H1: make_ohlcv(300, seed=42),
|
||||
Timeframe.M15: make_ohlcv(300, seed=43),
|
||||
Timeframe.H4: make_ohlcv(300, seed=44),
|
||||
}
|
||||
report = agg.analyze_from_dataframes("XAU/USD", dfs, use_news=False)
|
||||
msg = tb.format_signal_message(report)
|
||||
assert "Bid/Ask" not in msg and "Wide spread" not in msg
|
||||
|
||||
|
||||
# ══════════════════════════════════════════════════════════════════════
|
||||
# SUMMARY
|
||||
# ══════════════════════════════════════════════════════════════════════
|
||||
|
||||
@@ -0,0 +1,157 @@
|
||||
"""
|
||||
tiingo_provider.py — Tiingo top-of-book bid/ask quotes
|
||||
======================================================
|
||||
|
||||
Supplementary LIVE-QUOTE provider, NOT an OHLCV source: Twelve Data
|
||||
remains the signal/backtest data provider. Tiingo's forex endpoint
|
||||
(GET /tiingo/fx/top) returns real bid/ask, which mid-price OHLCV
|
||||
cannot provide. Two uses:
|
||||
|
||||
1. Spread visibility on alerts — signal messages show the live
|
||||
bid/ask and warn when the spread is ≥ WIDE_SPREAD_MULTIPLE × the
|
||||
assumed round-trip cost (PAIR_INFO ``default_spread``, $0.30 on
|
||||
XAU) that every backtest charges.
|
||||
|
||||
Warn-only BY DESIGN: no historical bid/ask exists to replay, so a
|
||||
suppressing gate could never be validated (live == backtest rule).
|
||||
This sits in the same safety category as the staleness check.
|
||||
|
||||
2. Spread logging (logs/spread_log.csv, 5-min sampler in the bot) —
|
||||
builds the dataset to validate or refute the $0.30 round-trip
|
||||
assumption against reality.
|
||||
|
||||
Signals NEVER consume bid/ask — indicators stay on mid OHLCV. Every
|
||||
failure path returns None (alerts go out without the spread block
|
||||
rather than being delayed or dropped).
|
||||
|
||||
Key resolution: env TIINGO_API_KEY (or TIINGO, e.g. from .env) →
|
||||
config.json data_providers.tiingo.api_key → "" (feature disabled).
|
||||
See app_config.tiingo_api_key.
|
||||
"""
|
||||
|
||||
import logging
|
||||
import os
|
||||
import threading
|
||||
import time
|
||||
from dataclasses import dataclass
|
||||
from datetime import datetime, timezone
|
||||
from typing import Dict, Optional
|
||||
|
||||
import requests
|
||||
|
||||
logger = logging.getLogger(__name__)
|
||||
|
||||
_TIINGO_FX_TOP_URL = "https://api.tiingo.com/tiingo/fx/top"
|
||||
|
||||
# Spread ≥ this multiple of the pair's assumed round-trip cost is
|
||||
# flagged as "wide" on alerts (2× the $0.30 XAU assumption = $0.60).
|
||||
WIDE_SPREAD_MULTIPLE = 2.0
|
||||
|
||||
# A quote younger than this is served from cache — dedupes bursts
|
||||
# (alert refresh right after a sampler tick) without hiding real moves.
|
||||
QUOTE_TTL_SECONDS = 10.0
|
||||
|
||||
SPREAD_LOG_PATH = os.path.join(
|
||||
os.path.dirname(os.path.abspath(__file__)), "logs", "spread_log.csv")
|
||||
_CSV_HEADER = "ts_utc,symbol,bid,ask,mid,spread\n"
|
||||
|
||||
|
||||
def tiingo_ticker(symbol: str) -> str:
|
||||
"""Canonical '"XAU/USD"' form → Tiingo's 'xauusd'."""
|
||||
return symbol.replace("/", "").lower()
|
||||
|
||||
|
||||
@dataclass
|
||||
class Quote:
|
||||
"""One top-of-book snapshot. ``spread`` (ask − bid) IS the
|
||||
round-trip cost a market order pays."""
|
||||
symbol: str # canonical form, e.g. "XAU/USD"
|
||||
bid: float
|
||||
ask: float
|
||||
mid: float
|
||||
spread: float
|
||||
quote_time: Optional[datetime] # venue timestamp (UTC) if provided
|
||||
fetched_at: float # time.time() at fetch
|
||||
|
||||
def age_seconds(self) -> float:
|
||||
return time.time() - self.fetched_at
|
||||
|
||||
|
||||
class TiingoQuoteProvider:
|
||||
"""Minimal /tiingo/fx/top client with a short TTL cache.
|
||||
|
||||
Lock held across the fetch (news_engine convention) so concurrent
|
||||
scan threads share one request instead of racing the API.
|
||||
"""
|
||||
|
||||
def __init__(self, api_key: str, quote_ttl: float = QUOTE_TTL_SECONDS):
|
||||
self.api_key = api_key
|
||||
self.quote_ttl = quote_ttl
|
||||
self._cache: Dict[str, Quote] = {}
|
||||
self._lock = threading.Lock()
|
||||
|
||||
def get_quote(self, symbol: str) -> Optional[Quote]:
|
||||
with self._lock:
|
||||
cached = self._cache.get(symbol)
|
||||
if cached is not None and cached.age_seconds() < self.quote_ttl:
|
||||
return cached
|
||||
quote = self._fetch(symbol)
|
||||
if quote is not None:
|
||||
self._cache[symbol] = quote
|
||||
return quote
|
||||
|
||||
def _fetch(self, symbol: str) -> Optional[Quote]:
|
||||
try:
|
||||
resp = requests.get(
|
||||
_TIINGO_FX_TOP_URL,
|
||||
params={"tickers": tiingo_ticker(symbol),
|
||||
"token": self.api_key},
|
||||
timeout=10,
|
||||
)
|
||||
resp.raise_for_status()
|
||||
payload = resp.json()
|
||||
if not isinstance(payload, list) or not payload:
|
||||
logger.warning(
|
||||
f"Tiingo: unexpected payload for {symbol}: {payload!r:.200}")
|
||||
return None
|
||||
row = payload[0]
|
||||
bid, ask = row.get("bidPrice"), row.get("askPrice")
|
||||
if not isinstance(bid, (int, float)) or not isinstance(ask, (int, float)):
|
||||
logger.warning(f"Tiingo: no bid/ask for {symbol}: {row!r:.200}")
|
||||
return None
|
||||
mid = row.get("midPrice")
|
||||
if not isinstance(mid, (int, float)):
|
||||
mid = (bid + ask) / 2
|
||||
quote_time: Optional[datetime] = None
|
||||
raw_ts = row.get("quoteTimestamp")
|
||||
if isinstance(raw_ts, str):
|
||||
try:
|
||||
quote_time = datetime.fromisoformat(
|
||||
raw_ts.replace("Z", "+00:00"))
|
||||
except ValueError:
|
||||
pass
|
||||
return Quote(
|
||||
symbol=symbol, bid=float(bid), ask=float(ask),
|
||||
mid=float(mid), spread=float(ask) - float(bid),
|
||||
quote_time=quote_time, fetched_at=time.time(),
|
||||
)
|
||||
except Exception as exc:
|
||||
logger.warning(f"Tiingo quote fetch failed for {symbol}: {exc}")
|
||||
return None
|
||||
|
||||
|
||||
def log_quote(quote: Quote, path: str = SPREAD_LOG_PATH) -> None:
|
||||
"""Append one row to the spread log (creates dir + header on first
|
||||
write). ~15 KB/day at 5-min sampling — no rotation needed."""
|
||||
try:
|
||||
os.makedirs(os.path.dirname(path), exist_ok=True)
|
||||
is_new = not os.path.exists(path)
|
||||
with open(path, "a", encoding="utf-8") as fh:
|
||||
if is_new:
|
||||
fh.write(_CSV_HEADER)
|
||||
ts = datetime.now(timezone.utc).strftime("%Y-%m-%dT%H:%M:%SZ")
|
||||
fh.write(
|
||||
f"{ts},{quote.symbol},{quote.bid:.5f},{quote.ask:.5f},"
|
||||
f"{quote.mid:.5f},{quote.spread:.5f}\n")
|
||||
except OSError as exc:
|
||||
logger.warning(f"Spread log write failed: {exc}")
|
||||
Reference in New Issue
Block a user