remove the AI market-outlook feature (delivery-side only; books byte-identical)
Owner decision 2026-09-15. Gone: market_outlook.py, the post-H1-scan Telegram
outlook DM, main.py --outlook, the outlooks table DDL/migration/writer in
signal_log.py, the market_outlook config block + MARKET_OUTLOOK_ENABLED, and
test Module 26. Last commit containing the feature: 2836024.
- app_config: "market_outlook" stays in _KNOWN_TOP as a TOLERATED, ignored
legacy key (oanda.account_id precedent) with one warning, so a live
config.json that still carries the block keeps booting.
- signal_log: existing DBs keep their orphan outlooks table and rows (no
DROP TABLE); fresh DBs never create it.
- tests: Module 26's 4-thread concurrent legacy-DB migration test (the
suite's only one) moved to Module 31; new test for the tolerated block.
519 -> 509 tests, 34 -> 33 modules, 0 failed.
- docs: CLAUDE.md usage/config/file-map lines and a dated state note.
Verification: W2 books (2024-09-05 -> 2026-09-05, $0.65, NumPy CPU, 0 cache
fallbacks) byte-identical before/after — M5 29120a1ee32979431d062f210d41a9d1
(781 / +146.07R / PF 1.52), M15 d185f099482a07f29502cb5a371a9bbf
(931 / +146.14R / PF 1.43).
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
This commit is contained in:
Harwinder Sekhon
@@ -12,11 +12,10 @@ Every numeric threshold in `config.py` and `pair_signals/xauusd.py` was backtest
|
||||
|
||||
```bash
|
||||
python test_all.py # QA suite, no network (mocked); prints its own count
|
||||
# (519 tests / 34 modules as of 2026-09-13)
|
||||
# (509 tests / 33 modules as of 2026-09-15)
|
||||
python main.py --pair XAUUSD --timeframe H1 --detail # CLI signal report (--json for full dump,
|
||||
# --panel for the advisory LLM agent panel, --outlook for the
|
||||
# hourly AI market brief; the violent-move watch line appears
|
||||
# when the `violent_watch` config block is enabled;
|
||||
# --panel for the advisory LLM agent panel; the violent-move watch
|
||||
# line appears when the `violent_watch` config block is enabled;
|
||||
# --timeframe accepts M5/M15/H1/H4 only and the confirmation
|
||||
# TFs always come from the pair factory's map)
|
||||
python telegram_bot.py # the bot (refuses to boot without credentials)
|
||||
@@ -45,7 +44,7 @@ Python 3.12; a `venv/` exists in-repo. Dependencies in `requirements.txt` (numpy
|
||||
|
||||
## Configuration & credentials
|
||||
|
||||
Precedence for every credential/deployment setting: **env var → config.json → refuse to run** (no hardcoded fallbacks). A gitignored repo-root `.env` is loaded at `app_config` import via `os.environ.setdefault`. `config.json` (gitignored; schema in `config.json.example`) holds secrets plus deployment settings: `data_providers.active` (the live-feed switch), `data_providers.streaming` (optional tick-stream provider for the level-breach monitor), `scanning.pairs`/`timeframes`, per-pair `blocked_hours_utc`, `log_level`. Key env vars: `TELEGRAM_TOKEN`, `TELEGRAM_CHAT_ID`, `TELEGRAM_CHANNEL_ID`, `TWELVE_DATA_API_KEY`, `TIINGO_API_KEY`, `OANDA_API_TOKEN`/`OANDA_BASE_URL` (the data feed needs no account id; a legacy `oanda.account_id` key is tolerated and ignored), `DEEPSEEK_API_KEY` (news LLM; the `deepseek` config.json block holds model/enabled/timeout/daily cap). The `news_archive` block (`enabled` default TRUE, `path` default `logs/newsdata.db`; env `NEWS_ARCHIVE_ENABLED`/`NEWS_ARCHIVE_PATH`) controls the durable headline corpus. The `agent_panel` config.json block (all-defaults = feature dark) controls the advisory LLM agent panel: `enabled`, per-surface `telegram`/`dashboard` toggles, `min_quality`, `max_panels_per_day`, `reuse_ttl_seconds`, plus the technical-payload enrichment knobs `candles_enabled`/`candle_bars_primary`/`candle_bars_confirm`. The dark-by-default `market_outlook` block controls the hourly advisory market brief: `enabled`, `telegram`, `max_per_day`, `candles_bars`; `MARKET_OUTLOOK_ENABLED` overrides its enabled setting. The dark-by-default `violent_watch` block controls the display-only violent-move watch: `enabled`, `alert_line`, `headsup`, `max_headsup_per_day`, `headsup_min_level` (`"watch"`/`"hot"`); `VIOLENT_WATCH_ENABLED` overrides its enabled setting.
|
||||
Precedence for every credential/deployment setting: **env var → config.json → refuse to run** (no hardcoded fallbacks). A gitignored repo-root `.env` is loaded at `app_config` import via `os.environ.setdefault`. `config.json` (gitignored; schema in `config.json.example`) holds secrets plus deployment settings: `data_providers.active` (the live-feed switch), `data_providers.streaming` (optional tick-stream provider for the level-breach monitor), `scanning.pairs`/`timeframes`, per-pair `blocked_hours_utc`, `log_level`. Key env vars: `TELEGRAM_TOKEN`, `TELEGRAM_CHAT_ID`, `TELEGRAM_CHANNEL_ID`, `TWELVE_DATA_API_KEY`, `TIINGO_API_KEY`, `OANDA_API_TOKEN`/`OANDA_BASE_URL` (the data feed needs no account id; a legacy `oanda.account_id` key is tolerated and ignored; so is a legacy top-level `market_outlook` block, with one warning — feature removed 2026-09-15), `DEEPSEEK_API_KEY` (news LLM; the `deepseek` config.json block holds model/enabled/timeout/daily cap). The `news_archive` block (`enabled` default TRUE, `path` default `logs/newsdata.db`; env `NEWS_ARCHIVE_ENABLED`/`NEWS_ARCHIVE_PATH`) controls the durable headline corpus. The `agent_panel` config.json block (all-defaults = feature dark) controls the advisory LLM agent panel: `enabled`, per-surface `telegram`/`dashboard` toggles, `min_quality`, `max_panels_per_day`, `reuse_ttl_seconds`, plus the technical-payload enrichment knobs `candles_enabled`/`candle_bars_primary`/`candle_bars_confirm`. The dark-by-default `violent_watch` block controls the display-only violent-move watch: `enabled`, `alert_line`, `headsup`, `max_headsup_per_day`, `headsup_min_level` (`"watch"`/`"hot"`); `VIOLENT_WATCH_ENABLED` overrides its enabled setting.
|
||||
|
||||
## Architecture
|
||||
|
||||
@@ -73,7 +72,6 @@ Signal currency everywhere is `SignalStrength` (−2..+2). DataFrames are UTC tz
|
||||
- **`news_engine.py`** — Forex Factory calendar (MyFXBook RSS fallback) + RSS keyword sentiment with gold-inverse keywords. Class-level caches (fresh engine per scan). Optional DeepSeek enrichment via `llm_client.py` (requests-only, no SDK): batched headline sentiment overriding keyword scores, gold-direction on release surprises, and a market-summary blurb (`SignalReport.news_summary`) — **display/warning-only, keyword fallback on any failure, inert without a key**; prompts are static module constants (DeepSeek prefix-cache pricing) and requests send `thinking: disabled` (V4-Flash reasoning otherwise blows the timeout). By XAU-tuned default, event risk and sentiment **warn but do not gate signals** (`event_risk_caps_signals=False`, `sentiment_affects_signals=False`) — the LLM does not change that. **Headline pool (2026-09-04):** only headlines ≤ 4 h old with a parsable publish time (`headline_max_age_seconds=14400`, undated items dropped), and `get_fundamental_context` drops headlines whose every relevant score is inside ±0.1 (`_is_neutral_headline`, `NEUTRAL_HEADLINE_THRESHOLD`) before the per-currency mean — so `news_sentiment` from that date is a mean over directional headlines only (bolder values; log comparisons must split at the deploy date). The market summary prompt writes for someone who has never traded (no acronyms), sends headlines newest-first with the platform's own gold direction, and caps at 400 chars. **Verdict 2026-09-04** (`backtest/research/news_sentiment_value_2026-09-04.md`): `news_sentiment` is a trailing-24h echo of price (r≈0.67 with the past day, ≈0.03 with the next; flips 44 h after tops / 16 h after bottoms) — both gating flags stay False; re-check with `analyze_signal_log.py --stored-candles --news`.
|
||||
- **`news_archive.py`** — durable headline corpus in its OWN database (`logs/newsdata.db`), separate from `signals.db` so a signal-log rotation cannot take the news with it (that is exactly how the 3-week corpus behind `news_sentiment_value_2026-09-04.md` was lost — 4,054 distinct headlines, unrecoverable, `logs/` is gitignored). One row per distinct headline keyed on `sha256(headline)`, carrying the text, description, link, publisher, publish time, the score, `score_source` (`keyword`/`llm`/`model` — a new `NewsSentiment` field, also mirrored into `signal_log._headline_to_dict`), and `teacher_model`/`teacher_prompt_sha1` so labels made under a changed prompt are never pooled with older ones. **Writes are one atomic UPSERT per row and upgrade-only**: an `llm` score always replaces a `keyword` one (the >25/cycle batch-overflow tail gets its DeepSeek label on the next fetch window in a long-running process — verified: 13/13 upgraded) but a `keyword` score can never overwrite an `llm` one; re-seeing a headline always bumps `seen_count`. A read-then-write pair loses increments when the bot, the dashboard publisher and the tests first see a headline together, and the WAL switch itself needs `_with_lock_retry` — both are regression-tested with 3 concurrent writers. Hooked at the END of `news_engine._fetch_news_sentiment_raw`, **before** `_is_neutral_headline` runs in `get_fundamental_context`: archiving downstream of that ±0.1 filter is why `signals.db` holds almost no neutral headlines and would teach a trained scorer that "neutral" barely exists. Never raises (fail-safe by contract, like `signal_log`). The backtester never builds a `NewsEngine`, so replays never archive (test-locked). Seeded by `scripts/backfill_news_archive.py`, which reads both DB shapes read-only and **recomputes** provenance (stored score vs the keyword scorer: differs → `llm`, exact non-zero match → `keyword`, both zero → `unknown`) because neither source recorded it. Purpose: training data for a local numpy-only headline scorer to replace the 24/78 keyword fallback — see the note below.
|
||||
- **`agent_panel.py`** — ai-hedge-fund-style advisory panel: 4 DeepSeek persona calls (macro/technical/risk/contrarian, concurrent on a panel-private pool) + 1 synthesizer producing an **ADVISORY-ONLY** consensus (`AGREE/NEUTRAL/OPPOSE` + confidence + rationale) written onto `SignalReport.panel_*` fields. Attached **post-hoc** by consumers (`telegram_bot` alert path, `main --panel`, `dashboard_publisher`) — never runs inside `analyze_from_dataframes` (backtest parity, test-locked) and no gate reads its fields. Persona payloads are role-filtered: macro gets high-impact events plus only code-scored directional news; technical gets strategy/pattern, primary indicators, MTF, levels, and ATR-normalized candles; risk gets setup strategy, strategy cost assumption, trailing semantics, volatility/events/warnings (never the median candle range as spread); contrarian gets directional news, setup, timestamped latest-closed price, indicators, reasons, and warnings. Spend is bounded by exact cache, full-evidence-identity TTL reuse, and `max_panels_per_day`; changed evidence never inherits a direction-only cached verdict (test-locked — which also means TTL reuse effectively never fires live because price/candles change every bar; the daily cap is the real spend bound). Verdict provenance logs creation time, `MISS`/`EXACT_HIT`/`TTL_REUSE`, and a SHA-256 payload hash. Prompts forbid outside knowledge and outputs remain short. **OPPOSE guard (2026-09):** the 3-week signal-log scorecard showed the contrarian/risk/technical personas oppose almost every trend entry (chair said OPPOSE 73% of the time; opposed trades made +30R, PF 2.6) while the macro persona was the only informative voice (macro=support → 88% win). `_apply_oppose_guard` therefore downgrades a chair OPPOSE to NEUTRAL unless macro opposes, or technical opposes while macro is not supportive; the synthesizer prompt states the same rule. Re-score with `analyze_signal_log.py --stored-candles` (persona scorecard) before changing it again. **Verdict source (2026-09-03, owner decision):** all four personas still run and are logged (`panel_takes_json`), but `agent_panel.verdict_source` defaults to `"macro"` — the alert verdict is the macro persona's stance (support/neutral/oppose → AGREE/NEUTRAL/OPPOSE) and the synthesizer call is skipped (4 LLM calls per panel instead of 5); `"chair"` restores the four-voice synthesizer + OPPOSE guard. Re-score personas monthly with `analyze_signal_log.py --stored-candles`; weights are not to be tuned before ~300 graded outcomes with a held-out half.
|
||||
- **`market_outlook.py`** — hourly DeepSeek market brief for XAU/USD, using ATR-normalized M15/H1/H4 candles, visible indicator states, levels, news, and recent signal history (read `mode=ro`). It returns a written **ADVISORY-ONLY** bias/setup-forming assessment and persists it independently in `outlooks` within `logs/signals.db` together with the news, indicator/regime/level, and candle evidence sent to the model. It may be sent post-scan by Telegram or requested with `main.py --outlook`; it never imports or runs inside the signal path. Hourly dedup is by UTC hour BUCKET (the old `now − last < 3600` rule dropped every other hour because the H1 loop fires at boundary+10 s+scan time) plus its own UTC-day cap (default 12); every silent early return logs `market outlook: skipped (...)` and an empty LLM reply logs a warning — a dark feature must be visible in `logs/telegram.log`. Provider/news failures leave a safe partial brief while LLM/database failures degrade to no outlook.
|
||||
- **`violent_watch.py`** — **display-only** violent-move watch, dark by default (`violent_watch` config block: `enabled`, `alert_line`, `headsup`, `max_headsup_per_day`, `headsup_min_level`; env `VIOLENT_WATCH_ENABLED`). Flags when a "violent bar" (range ≥ 3× prior ATR14) is unusually likely and stamps `SignalReport.watch_line` **post-hoc** via `attach_watch` (telegram alert path, `main.py`, `dashboard_publisher`) — same contract as the agent panel: never imported by `signal_aggregator`/`engine`/`backtest/`, never raises, never read by a gate (test-locked, Module 30). Triggers: a high-impact **USD** calendar release within ±30 min (hot), hot clock slots (weekday 12–14 UTC, weekday 01 UTC, Sunday reopen 22–23 UTC), and **price-extreme** indicator readings (outside the Bollinger bands, RSI <30 / ≥70, outside the Keltner channel). Level = hot when the calendar fires or a clock slot plus ≥1 price extreme; watch on a clock slot alone or ≥2 price extremes. Context readings (ADX slope >3, volatility "extreme", regime "transitioning", MACD STRONG_BUY/STRONG_SELL) are printed but **never raise the level** — the first draft that let them count flagged 27% of bars "hot" at only ~2× odds; the shipped rule flags 8.6% at 3–5× (evidence: `backtest/research/sizing_campaign_2026-09.md` addenda). Direction is called DOWN on bearish readings (70–78% on M5 and M15) and UP only on M5. Telegram also sends a standalone heads-up on the fastest scanned TF (`_HEADSUP_TF`), rate-limited per UTC hour+trigger-set and per day; it explicitly tells the owner **not** to touch open trades — every protection tested lost money, the stop is the protection.
|
||||
- **`telegram_bot.py`** — owner-locked bot + subscriber channel; candle-boundary auto-scan loops (M5/M15/H1/H4, boundary+10 s), alert quality threshold + fingerprint dedup (committed only after a successful send; the fingerprint is signal+direction — grade is NOT part of it, a grade UPGRADE re-alerts, an A↔B flicker does not), 📰 LLM market-context line on alerts when available, WebSocket level-breach monitor fed by the streaming provider (`data_providers.streaming`, falling back to the active provider; auto-disabled when the resolved provider can't stream or has no API key — Oanda does not stream, Tiingo does). The breach handler reuses this bar's M15 report (`_cached_or_scan`, keyed by closed-bar epoch) instead of rescanning — it used to log 3.5 M15 rows per candle. **Alerts require a trade setup (2026-09-08):** `_is_alert_candidate` (shared by the scan and breach paths) needs non-neutral + grade ≥ threshold + `trade_setup`; a signal every generator declined used to go out anyway (a B-grade no-setup SELL under a "costs would eat the edge" warning). **Sends are logged:** every delivered alert / breach alert / violent-move heads-up writes an `… sent:` INFO line carrying the signal-log row id, and a no-setup candidate writes `alert skipped: … (gate reason)` — before this only failures were logged and a review had to reconstruct sends from panel rows. Rotating log: `logs/telegram.log`, one file per server-local day × 30 backups (was 3-h × 240 until 2026-09-08; the old `_HH`-suffixed files on the live box are not matched by the daily cleanup and need one manual delete).
|
||||
- **`dashboard_publisher.py` / `static/dashboard.html`** — web dashboard as a **static publisher** (live LAMP server): standalone daemon scans at candle closes via `produce_signal` and atomically writes `snapshot.json`/`news.json`/`dashboard.html` to `$DASHBOARD_OUT_DIR` (Apache serves them; page is public by owner decision). **Hard independence rule: never imports, and is never imported by, `telegram_bot`** — confirmation map and boundary math are deliberate local copies, so a web-side failure can only mean a stale page. Due-TF classification uses the RAW wake time (flooring wake−buffer skips every H1/H4 scan — 2026-08-02 bug). News panel reads `NewsEngine.get_fundamental_context()` plus the additive `get_week_events()` accessor (display-only; no signal path calls it); as a separate process it has its own news caches and DeepSeek daily cap. News/calendar rows are clickable (detail dialog); headlines carry `id` = sha1(headline) in news.json, and exactly one on-demand endpoint exists: `static/news_summary.php` (installed by `install_static`), which serves a DeepSeek gold-impact summary per headline — whitelist-gated against news.json ids, cache-forever in `ai/`, own fail-closed daily cap (150/day, file-based, separate from the publisher's), key at `/etc/gold-dashboard/deepseek.key`, never reads repo config. Deployment checklist: `deploy/README-dashboard.md`.
|
||||
@@ -116,3 +114,4 @@ Signal currency everywhere is `SignalStrength` (−2..+2). DataFrames are UTC tz
|
||||
- 2026-09-09 weight-change test (`backtest/research/weight_test_2026-09-09.md`, first use of the recorded replay files): a drift-adjusted per-bar screen ranked 108 regime-multiplier candidates per TF; the top 3 per TF (24 h-sign filter) went through the full 2-year replay against same-harness controls plus the 2024-06→2025-03 OOS window under a rule pre-registered before any run. M5: all three MACD candidates FAIL (−10 to −46R; the screen ranked them first — never trust the per-bar screen on M5 without a replay). M15: ema_trend × 2 FAILS (PF 1.32, DD 26R, +55% trades), divergence → 0 in trending FAILS (H1 −3R, DD); **RSI multiplier × 0.5 in every regime PASSES a–f** (2-yr 793/+117.6R/PF 1.41 → 931/+146.1R/PF 1.43, halves +27.9/+0.6, OOS +40.6 → +60.7R PF 1.49, 3-yr +151.9 → +186.6R PF 1.39 but max DD 15.5 → 19.3R and 2026 −4R; added trades are mostly B-grade BUYs). SHIPPED the same day on the owner's decision as an M15-only `technical.regime_weights_override` in `pair_signals/xauusd.py` (M15 override 5: the shipped table with RSI halved in every regime — 0.5 trending family / 0.75 ranging / 0.5 transitioning; derived from the default table, pinned literally by the Module 9 equivalence test) — the factory run without `--set` reproduces the validated candidate book byte-for-byte (parity MD5 in `weight_test_2026-09-09.md`); M15 canonical book re-pinned to `backtest/oanda_m15_2yr_065_20260909.csv`. Live M15 `engine_fingerprint` changes from that deploy; signal-log comparisons across it must split at the deploy time. Suite 499 tests.
|
||||
- 2026-09-10 improvement cycle 1 (`backtest/research/cycle_2026-09-10.md`; 501 tests; no signal-core change): HEAD baseline re-established on the 3-yr window at $0.65 (M5 870/+153.6R/PF 1.50, M15 1273/+186.6/1.39, H1 253/+70.0/1.98, H4 114/+21.7/1.59) and the three pinned MD5s (M5 2-yr, M15 2-yr, M15 OOS) reproduced exactly. Robustness: the cost ladder trips no red flag (every TF keeps PF ≥ 1.3 with positive halves at $0.80; at $1.30 M5 PF 1.36, M15 1.29, H1 1.89, H4 1.53); the intraday edge is a fat right tail (top 5% of trades carry 94% of M5 R and 96% of M15 R, versus 48% on H1; M5 2024 was flat at +1.0R over 102 trades). Pre-registered bucket screen: exactly one qualifying bucket — M15 grade-A SELLs (230 / −14.4R, negative in all three years and OOS; the 2026-09-03 5-yr run said the opposite) — parked as the only cycle-2 candidate. **C1 REJECTED:** H1 divergence multiplier → 0 in every regime (the parked 2026-09 candidate on its now-sanctioned vehicle): 3-yr ΔR +1.0, second half −1.5R, PF 1.98 → 1.89 (its OOS book 88/+21.9/PF 1.92 is trade-identical to the old `disabled_indicators` run, so a 0.0 regime multiplier ≡ disabling for the composite). **C2:** the M15 RSI × 0.5 ship sits on a stable plateau (k = 0.25 / 0.75 keep 78% / 71% of its 2-yr gain, both OOS-positive; R is monotone in k across 0…1.5) — no other k proposed. **Seeding parity SHIPPED:** live fetches floor at `MIN_LIVE_BARS = 1000` (H4 380 → 1000, D1 300 → 1000; measured ema_trend flips 2/267 H4 and 1/313 D1 bars at the old depths, none ≥ 600; the live D1 bar of 2026-09-10 itself read SELL at 300 bars and NEUTRAL at 1000/5000) and the backtester pad targets the same depth (`_history_pad_days`: 62 / 234 / 1400 days) — M5/M15 3-yr books byte-identical, H1/H4 trade-identical with only window-start float noise in the exported columns, so no re-pin. Also fixed in these notes: no PDF docs exist; `pyarrow` is a dev-venv-only dependency of the backtester/recorder.
|
||||
- 2026-09-13 bug audit (plan file `~/.claude/plans/you-are-python-software-joyful-teacup.md`; 519 tests / 34 modules; the five reference books — M5 W2 `29120a1ee32979431d062f210d41a9d1`, M15 W2 `d185f099482a07f29502cb5a371a9bbf`, M15 OOS `6f8f24b49597ced28f50a19e08c3472f`, H1 W3 `3b45d12bade823178b267c13bc698458`, H4 W3 `e75adae13c839b15736f69c0114e6aa5` (the H1/H4 values are the post-pad books, `cycle_2026-09-10/w3_{1h,4h}_pad.csv`) — byte-identical before and after every change, 0 GPU cache fallbacks). Verified sound, no change: composite = Σwscore/Σeff_weight × 50 on 400/400 DB rows; scan-outcome labels reproduce later scan prices to 0.000 on 844 rows; outcome-model exits reproduced on rows 578/1372/1500; Oanda H4/D1 alignment is UTC; no look-ahead in engine/backtester/chart_patterns; no cross-bar state; parquet caches hold only holiday gaps. Fixed (all delivery/measurement side, books unchanged): (1) NEUTRAL-signal C/B setups now alert (see `_attach_core_setup`); (2) a missing confirmation frame fails the scan (`scan_kind='failed'`, warning `DATA UNAVAILABLE:`); (3) `pd.Timestamp.utcnow()` (deprecated, removed in pandas 4) replaced in `providers/base.py`/`engine.py`, `requirements.txt` pins `pandas<4`, `numpy<3`; (4) advisory outcomes keep locked-profit floors and the rebuild fires on both gap reasons; (5) scan-outcome labeler max-age/anchor bound; (6) the daily settlement pause (Mon–Thu 17:00 NY) is a scheduled closure for the freshness gates — `_is_daily_maintenance`/`_is_scheduled_closure`/`_scheduled_closed_seconds` in `providers/base.py`, the expected label walks back only past labels whose WHOLE span lies in a closure (a D1 bar labelled Sunday 00:00 UTC and an H4 bar labelled Sunday 20:00 exist), and the bot/dashboard skip scans whose just-closed label sits in the pause (61 false "stale" rows in 3 nights before); (7) `fetch_range` retries a failed page in place (3 failures on one window raise) and logs an ERROR for interior gaps of >3 unscheduled bars (holidays look identical — verify, never fatal); (8) `_replay_span` returns the per-span GPU cache-miss count, `run_backtest` sums/logs it (must be 0) and writes `cache_fallbacks` to the `--record` manifest; (9) `regime_weight_table` raises on a partial `regime_weights_override` (every regime × all nine indicators); (10) Telegram boundary loops go through `_cached_or_scan`, `_remember_scan` files a report under the bar it analysed (label + duration), a breach inside 20 s after an M15 boundary waits for the loop's scan (15 duplicate M15 rows in 5 days before), and the rotating file handler is attached in `main()` only (importing the module in tests wrote fixture alerts into `logs/telegram.log`); (11) dashboard `catch_up_timeframes` re-scans any TF whose newest closed boundary is newer than its last scanned one (missed while a cycle overran or the process was down). Semantics recorded, NOT changed (rule 1): `blocked_hours_utc` is applied to the bar LABEL hour in the core (6b) AND the bar CLOSE hour in the live wrapper/backtester, so H1 can only trade bars closing 04–09, 15, 18–20 UTC and H4 only bars closing 08:00/12:00 — parity holds and every book was validated that way. Post-deploy: `scripts/backfill_signal_outcomes.py --reseed-unresolvable` floors the two Sep-8 shorts on the live DB; the `--stored-candles` scorecard (`analyze_signal_log.replay_stored`) still calls `advance_outcome` without the floor step (owner decision pending).
|
||||
- 2026-09-15 market outlook REMOVED (owner decision; plan `~/.claude/plans/you-are-quant-software-logical-cake.md`): `market_outlook.py`, the post-H1-scan Telegram 🔭 DM, `main.py --outlook`, the `outlooks` table DDL/migration/writer in `signal_log.py`, the `market_outlook` config block + `MARKET_OUTLOOK_ENABLED`, and test Module 26 are gone; the last commit containing the feature is `2836024`. A legacy top-level `market_outlook` block in config.json is TOLERATED and ignored with one warning (kept in `_KNOWN_TOP`, the `oanda.account_id` precedent) so the live bot/dashboard keep booting — delete the block from the live config.json at deploy. Existing `logs/signals.db` files keep their orphan `outlooks` table and rows (nothing reads or drops it; drop by hand during a weekend VACUUM if wanted); fresh DBs never create it. Module 26's 4-thread concurrent legacy-DB migration test (the suite's only one) moved to Module 31. Nothing in the signal path changed: the W2 books (2024-09-05→2026-09-05, $0.65, NumPy CPU, 0 cache fallbacks) are byte-identical before/after — M5 `29120a1ee32979431d062f210d41a9d1` (781 / +146.07R / PF 1.52), M15 `d185f099482a07f29502cb5a371a9bbf` (931 / +146.14R / PF 1.43). Suite 509 tests / 33 modules.
|
||||
|
||||
+12
-35
@@ -84,7 +84,12 @@ _VALID_LOG_LEVELS = ("DEBUG", "INFO", "WARNING", "ERROR", "CRITICAL")
|
||||
_SCANNABLE_TFS = ("M5", "M15", "H1", "H4")
|
||||
|
||||
_KNOWN_TOP = {"telegram", "data_providers", "scanning", "pairs", "log_level",
|
||||
"oanda", "deepseek", "agent_panel", "market_outlook",
|
||||
"oanda", "deepseek", "agent_panel",
|
||||
# "market_outlook" is TOLERATED but ignored: the AI market
|
||||
# outlook was removed 2026-09-15 (last commit 2836024). Live
|
||||
# config.json files still carry the block and must keep
|
||||
# loading; load_app_config logs one warning when it is there.
|
||||
"market_outlook",
|
||||
"violent_watch", "signal_log", "news_archive"}
|
||||
_KNOWN_OANDA = {
|
||||
# "account_id" is TOLERATED but ignored: it belonged to the auto-
|
||||
@@ -102,7 +107,6 @@ _KNOWN_AGENT_PANEL = {
|
||||
"candles_enabled", "candle_bars_primary", "candle_bars_confirm",
|
||||
"verdict_source",
|
||||
}
|
||||
_KNOWN_MARKET_OUTLOOK = {"enabled", "telegram", "max_per_day", "candles_bars"}
|
||||
_KNOWN_VIOLENT_WATCH = {"enabled", "alert_line", "headsup",
|
||||
"max_headsup_per_day", "headsup_min_level"}
|
||||
_KNOWN_SIGNAL_LOG = {"enabled", "path", "log_blocked_scans"}
|
||||
@@ -111,13 +115,6 @@ _KNOWN_TELEGRAM = {"token", "chat_id", "channel_id", "alert_min_quality"}
|
||||
_KNOWN_SCANNING = {"pairs", "timeframes"}
|
||||
_KNOWN_PAIR_KEYS = {"blocked_hours_utc"}
|
||||
|
||||
_MARKET_OUTLOOK_DEFAULTS = {
|
||||
"enabled": False,
|
||||
"telegram": False,
|
||||
"max_per_day": 12,
|
||||
"candles_bars": 20,
|
||||
}
|
||||
|
||||
# Violent-move watch (display-only heads-up) — dark by default.
|
||||
_VIOLENT_WATCH_DEFAULTS = {
|
||||
"enabled": False,
|
||||
@@ -149,8 +146,6 @@ class AppConfig:
|
||||
oanda_settings: Dict[str, Any] = field(default_factory=dict)
|
||||
deepseek_settings: Dict[str, Any] = field(default_factory=dict)
|
||||
agent_panel_settings: Dict[str, Any] = field(default_factory=dict)
|
||||
market_outlook_settings: Dict[str, Any] = field(
|
||||
default_factory=lambda: dict(_MARKET_OUTLOOK_DEFAULTS))
|
||||
violent_watch_settings: Dict[str, Any] = field(
|
||||
default_factory=lambda: dict(_VIOLENT_WATCH_DEFAULTS))
|
||||
signal_log_settings: Dict[str, Any] = field(default_factory=dict)
|
||||
@@ -341,19 +336,12 @@ def load_app_config(path: str = CONFIG_PATH) -> AppConfig:
|
||||
'"chair"')
|
||||
app.agent_panel_settings = dict(panel)
|
||||
|
||||
outlook = raw.get("market_outlook", {})
|
||||
_check_keys(outlook, _KNOWN_MARKET_OUTLOOK, '"market_outlook"')
|
||||
for flag in ("enabled", "telegram"):
|
||||
if flag in outlook and not isinstance(outlook[flag], bool):
|
||||
raise AppConfigError(
|
||||
f"config.json: market_outlook.{flag} must be true or false")
|
||||
for key in ("max_per_day", "candles_bars"):
|
||||
if key in outlook:
|
||||
value = outlook[key]
|
||||
if isinstance(value, bool) or not isinstance(value, int) or value < 1:
|
||||
raise AppConfigError(
|
||||
f"config.json: market_outlook.{key} must be a positive integer")
|
||||
app.market_outlook_settings.update(outlook)
|
||||
if "market_outlook" in raw:
|
||||
# The AI market-outlook feature was removed 2026-09-15; older
|
||||
# config.json files still carry its block and must keep loading.
|
||||
logger.warning(
|
||||
"config.json: market_outlook block is ignored (feature removed "
|
||||
"2026-09-15) — delete it")
|
||||
|
||||
watch = raw.get("violent_watch", {})
|
||||
_check_keys(watch, _KNOWN_VIOLENT_WATCH, '"violent_watch"')
|
||||
@@ -493,17 +481,6 @@ def deepseek_api_key(app: Optional[AppConfig] = None) -> str:
|
||||
app.deepseek_settings.get("api_key", ""))
|
||||
|
||||
|
||||
# ── AI market-outlook resolution helper ──────────────────────────────
|
||||
|
||||
def market_outlook_enabled(app: Optional[AppConfig] = None) -> bool:
|
||||
"""Whether the advisory outlook feature is enabled (env wins)."""
|
||||
env = os.environ.get("MARKET_OUTLOOK_ENABLED")
|
||||
if env:
|
||||
return env.strip().lower() in ("1", "true", "yes", "on")
|
||||
app = app or get_app_config()
|
||||
return bool(app.market_outlook_settings.get("enabled", False))
|
||||
|
||||
|
||||
# ── Violent-move watch resolution helper ─────────────────────────────
|
||||
|
||||
def violent_watch_enabled(app: Optional[AppConfig] = None) -> bool:
|
||||
|
||||
@@ -48,12 +48,6 @@
|
||||
"candle_bars_confirm": 10,
|
||||
"verdict_source": "macro"
|
||||
},
|
||||
"market_outlook": {
|
||||
"enabled": false,
|
||||
"telegram": false,
|
||||
"max_per_day": 12,
|
||||
"candles_bars": 20
|
||||
},
|
||||
"violent_watch": {
|
||||
"enabled": false,
|
||||
"alert_line": true,
|
||||
|
||||
@@ -13,7 +13,6 @@ import json
|
||||
import logging
|
||||
import sys
|
||||
from datetime import datetime, timezone
|
||||
from typing import Optional
|
||||
|
||||
# Force UTF-8 output on Windows so box-drawing chars render
|
||||
if hasattr(sys.stdout, "reconfigure"):
|
||||
@@ -24,7 +23,6 @@ if hasattr(sys.stderr, "reconfigure"):
|
||||
from agent_panel import attach_panel
|
||||
from app_config import get_app_config, violent_watch_enabled
|
||||
from config import PlatformConfig, Timeframe, SignalStrength, XAUUSD
|
||||
from market_outlook import OutlookResult, maybe_run_outlook
|
||||
from pair_signals import PAIR_PRODUCERS
|
||||
from signal_aggregator import SignalAggregator, SignalReport
|
||||
from violent_watch import attach_watch
|
||||
@@ -205,24 +203,6 @@ def print_signal_report(report: SignalReport, detail: bool = False):
|
||||
print(f" {label:>20} {color}{val:.3f}{Colors.RESET}")
|
||||
|
||||
|
||||
def print_market_outlook(outlook: Optional[OutlookResult]) -> None:
|
||||
"""Render the independent advisory outlook for explicit CLI use."""
|
||||
print(f"\n {Colors.BOLD}AI Market Outlook{Colors.RESET}")
|
||||
if outlook is None:
|
||||
print(f" {Colors.GRAY}Unavailable (disabled, market closed, or no LLM response).{Colors.RESET}")
|
||||
return
|
||||
forming = {
|
||||
"buy": "possible buy forming",
|
||||
"sell": "possible sell forming",
|
||||
"none": "no clear setup forming",
|
||||
}[outlook.setup_forming]
|
||||
print(f" {outlook.bias.title()} · {forming} · {outlook.confidence:.0f}%")
|
||||
if outlook.watch:
|
||||
print(f" Watch: {outlook.watch}")
|
||||
if outlook.note:
|
||||
print(f" {Colors.GRAY}{outlook.note}{Colors.RESET}")
|
||||
|
||||
|
||||
def report_to_dict(report: SignalReport) -> dict:
|
||||
d = {
|
||||
"symbol": report.symbol,
|
||||
@@ -315,9 +295,6 @@ Examples:
|
||||
parser.add_argument("--panel", action="store_true",
|
||||
help="Run the LLM agent panel on the result "
|
||||
"(advisory only; needs a DeepSeek key)")
|
||||
parser.add_argument("--outlook", action="store_true",
|
||||
help="Run the hourly AI market outlook now "
|
||||
"(advisory only; needs a DeepSeek key)")
|
||||
parser.add_argument("--json", action="store_true",
|
||||
help="Output as JSON")
|
||||
parser.add_argument("--verbose", "-v", action="store_true",
|
||||
@@ -372,30 +349,13 @@ def run_live(args, config: PlatformConfig):
|
||||
attach_watch(report, config.primary_timeframe,
|
||||
get_app_config().violent_watch_settings)
|
||||
|
||||
outlook = None
|
||||
if getattr(args, "outlook", False):
|
||||
outlook = maybe_run_outlook(surface="cli", force=True)
|
||||
|
||||
if args.json:
|
||||
payload = report_to_dict(report)
|
||||
if getattr(args, "outlook", False):
|
||||
payload["market_outlook"] = (
|
||||
None if outlook is None else {
|
||||
"bias": outlook.bias,
|
||||
"setup_forming": outlook.setup_forming,
|
||||
"confidence": outlook.confidence,
|
||||
"watch": outlook.watch,
|
||||
"note": outlook.note,
|
||||
"created_ts": outlook.created_ts,
|
||||
}
|
||||
)
|
||||
print(json.dumps(payload, indent=2))
|
||||
return
|
||||
|
||||
print_header(f"{config.display_name} SIGNAL INTELLIGENCE")
|
||||
print_signal_report(report, detail=args.detail)
|
||||
if getattr(args, "outlook", False):
|
||||
print_market_outlook(outlook)
|
||||
print(f"\n{Colors.CYAN}{'═' * 80}{Colors.RESET}")
|
||||
print(f"{Colors.GRAY} Disclaimer: For educational/research purposes only.{Colors.RESET}")
|
||||
print(f"{Colors.CYAN}{'═' * 80}{Colors.RESET}\n")
|
||||
|
||||
@@ -1,439 +0,0 @@
|
||||
"""
|
||||
ForexBot2 — AI market outlook (advisory hourly assessment)
|
||||
===========================================================
|
||||
Builds a broad XAU/USD market brief and asks DeepSeek whether a good setup
|
||||
appears to be forming. The result is written to the signal-history database
|
||||
and may be displayed by consumers, but it is ADVISORY-ONLY: it never changes
|
||||
signal scoring, gates, grades, or trade setup generation.
|
||||
|
||||
The outlook runs outside the replayable signal path. Individual candle/news
|
||||
failures produce a partial brief; an unavailable LLM, malformed reply, or
|
||||
locked database never reaches a scan caller as an exception.
|
||||
|
||||
The static prompt is intentional: DeepSeek prices byte-identical prompt
|
||||
prefixes at its cache-hit rate, so timestamps and all other changing evidence
|
||||
live only in the JSON user payload.
|
||||
"""
|
||||
|
||||
import datetime as _dt
|
||||
import logging
|
||||
import math
|
||||
import os
|
||||
import sqlite3
|
||||
import threading
|
||||
import time
|
||||
from dataclasses import dataclass
|
||||
from typing import Any, Dict, List, Optional
|
||||
|
||||
from agent_panel import _normalize_candles
|
||||
from app_config import (get_app_config, market_outlook_enabled,
|
||||
signal_log_path)
|
||||
from config import PlatformConfig, Timeframe, XAUUSD
|
||||
from engine import TechnicalEngine
|
||||
from llm_client import get_llm_client
|
||||
from providers import base as _provider_base
|
||||
|
||||
logger = logging.getLogger(__name__)
|
||||
|
||||
DEFAULT_MAX_PER_DAY = 12
|
||||
DEFAULT_CANDLE_BARS = 20
|
||||
_OUTLOOK_TFS = (Timeframe.M15, Timeframe.H1, Timeframe.H4)
|
||||
_VALID_BIASES = ("bullish", "bearish", "neutral")
|
||||
_VALID_SETUPS = ("buy", "sell", "none")
|
||||
|
||||
|
||||
# STATIC prompt: repeated prompt prefixes are much cheaper with DeepSeek's
|
||||
# server-side cache. Variable market evidence belongs exclusively in payload.
|
||||
SYSTEM_PROMPT = """\
|
||||
You are writing an advisory market outlook for gold (XAU/USD). You receive a
|
||||
JSON market brief covering recent candle shape, indicator states, price
|
||||
levels, news, and recent technical signals. Assess what may be forming soon;
|
||||
you are not a signal generator and must not provide an entry price, stop, or
|
||||
target.
|
||||
|
||||
Gold doctrine: hawkish Fed / rate hikes / rising real yields / strong
|
||||
dollar are gold-bearish; easing / rate cuts / safe-haven flows /
|
||||
geopolitical stress / weak dollar are gold-bullish.
|
||||
|
||||
Indicator "value" units: rsi 0-100 (above 70 overbought, below 30
|
||||
oversold); adx 0-100 (above 25 = strong trend); macd = histogram in
|
||||
dollars (momentum); roc = percent; bollinger = position 0-1 (0 = lower
|
||||
band, 1 = upper band); keltner = dollar distance from the mid-line;
|
||||
ema_trend = 0-1 alignment score; divergence and candlestick = pattern
|
||||
counts. "candles" is per timeframe: {"ref_close", "atr", "bars"}; bars
|
||||
are oldest-first [open, high, low, close] in ATR units relative to the
|
||||
latest close. Read their shape, never quote them as prices.
|
||||
|
||||
Hard rules: use ONLY the JSON provided. Do not invent numbers, events, or
|
||||
price levels. Your training data about gold, markets, and current events is
|
||||
stale and OFF-LIMITS: never use outside knowledge; if a fact is not in the
|
||||
JSON, treat it as unknown. If evidence is thin or missing, answer "neutral"
|
||||
with confidence below 30. Use plain everyday English for someone unfamiliar
|
||||
with markets: short simple sentences, no acronyms or jargon. Keep "watch"
|
||||
and "note" to at most 25 words each.
|
||||
|
||||
Reply with ONLY JSON:
|
||||
{"bias": "bullish"|"bearish"|"neutral",
|
||||
"setup_forming": "buy"|"sell"|"none",
|
||||
"confidence": 0-100,
|
||||
"watch": "<what to watch for, max 25 words>",
|
||||
"note": "<plain-English summary, max 25 words>"}
|
||||
"""
|
||||
|
||||
|
||||
@dataclass
|
||||
class OutlookResult:
|
||||
bias: str
|
||||
setup_forming: str
|
||||
confidence: float
|
||||
watch: str
|
||||
note: str
|
||||
created_ts: float
|
||||
|
||||
|
||||
def _finite(x: Any) -> Optional[float]:
|
||||
"""float(x) or None for non-numeric / non-finite JSON values."""
|
||||
try:
|
||||
value = float(x)
|
||||
except (TypeError, ValueError):
|
||||
return None
|
||||
return value if math.isfinite(value) else None
|
||||
|
||||
|
||||
def _round_finite(x: Any, digits: int) -> Optional[float]:
|
||||
value = _finite(x)
|
||||
return round(value, digits) if value is not None else None
|
||||
|
||||
|
||||
def _json_safe(value: Any) -> Any:
|
||||
"""Replace any accidental NaN/inf nested in provider data with null."""
|
||||
if isinstance(value, dict):
|
||||
return {str(key): _json_safe(item) for key, item in value.items()}
|
||||
if isinstance(value, (list, tuple)):
|
||||
return [_json_safe(item) for item in value]
|
||||
if isinstance(value, bool) or value is None or isinstance(value, str):
|
||||
return value
|
||||
if isinstance(value, (int, float)):
|
||||
return _finite(value)
|
||||
return value
|
||||
|
||||
|
||||
def _clamp_confidence(raw: Any) -> Optional[float]:
|
||||
value = _finite(raw)
|
||||
return max(0.0, min(100.0, value)) if value is not None else None
|
||||
|
||||
|
||||
def _word_limit(text: Any, words: int = 25) -> str:
|
||||
"""Bound a response field by words as well as a practical char limit."""
|
||||
clean = str(text or "").strip()
|
||||
return " ".join(clean.split()[:words])[:200]
|
||||
|
||||
|
||||
def _validate_outlook(reply: Any) -> Optional[OutlookResult]:
|
||||
"""Validate only model replies that fully match the requested contract."""
|
||||
if not isinstance(reply, dict):
|
||||
return None
|
||||
bias = reply.get("bias")
|
||||
setup_forming = reply.get("setup_forming")
|
||||
confidence = _clamp_confidence(reply.get("confidence"))
|
||||
if (bias not in _VALID_BIASES or setup_forming not in _VALID_SETUPS
|
||||
or confidence is None):
|
||||
return None
|
||||
return OutlookResult(
|
||||
bias=bias,
|
||||
setup_forming=setup_forming,
|
||||
confidence=confidence,
|
||||
watch=_word_limit(reply.get("watch")),
|
||||
note=_word_limit(reply.get("note")),
|
||||
created_ts=time.time(),
|
||||
)
|
||||
|
||||
|
||||
# ── Data collection ──────────────────────────────────────────────────
|
||||
|
||||
_dp: Any = None # Patchable singleton, mirroring agent_panel._dp.
|
||||
|
||||
|
||||
def _outlook_provider() -> Any:
|
||||
"""Lazy provider singleton so its closed-bar cache stays warm hourly."""
|
||||
global _dp
|
||||
if _dp is None:
|
||||
from providers import make_data_provider
|
||||
_dp = make_data_provider(PlatformConfig())
|
||||
return _dp
|
||||
|
||||
|
||||
def _indicator_payload(analysis: Any) -> Dict[str, Dict[str, Any]]:
|
||||
"""Visible scored indicators with finite, compact values only."""
|
||||
out: Dict[str, Dict[str, Any]] = {}
|
||||
for name, indicator in (getattr(analysis, "indicators", None) or {}).items():
|
||||
if name.startswith("_") or getattr(indicator, "weight", 0) == 0:
|
||||
continue
|
||||
entry: Dict[str, Any] = {
|
||||
"signal": getattr(getattr(indicator, "signal", None), "name", ""),
|
||||
"confidence": _round_finite(
|
||||
getattr(indicator, "confidence", None), 1),
|
||||
"value": _round_finite(getattr(indicator, "value", None), 2),
|
||||
}
|
||||
out[name] = entry
|
||||
return out
|
||||
|
||||
|
||||
def _analysis_payload(analyses: Dict[str, Any]) -> Dict[str, Any]:
|
||||
"""Pure formatter for the technical portion of the market brief."""
|
||||
indicators: Dict[str, Any] = {}
|
||||
market_regime: Dict[str, Optional[str]] = {}
|
||||
volatility_regime: Dict[str, Optional[str]] = {}
|
||||
key_levels: Dict[str, Dict[str, Optional[float]]] = {}
|
||||
for tf in _OUTLOOK_TFS:
|
||||
name = tf.value
|
||||
analysis = analyses.get(name)
|
||||
indicators[name] = _indicator_payload(analysis)
|
||||
market_regime[name] = getattr(analysis, "market_regime", None)
|
||||
volatility_regime[name] = getattr(analysis, "volatility_regime", None)
|
||||
key_levels[name] = {
|
||||
level: _round_finite(value, 2)
|
||||
for level, value in (getattr(analysis, "key_levels", None) or {}).items()
|
||||
}
|
||||
return {
|
||||
"indicators": indicators,
|
||||
"market_regime": market_regime,
|
||||
"volatility_regime": volatility_regime,
|
||||
"key_levels": key_levels,
|
||||
}
|
||||
|
||||
|
||||
def _empty_news_payload() -> Dict[str, Any]:
|
||||
return {
|
||||
"headlines": None,
|
||||
"events": None,
|
||||
"gold_sentiment": None,
|
||||
"remaining_high_impact_events": None,
|
||||
}
|
||||
|
||||
|
||||
def _news_payload() -> Dict[str, Any]:
|
||||
"""Collect current news without allowing any news failure to block us."""
|
||||
try:
|
||||
from news_engine import NewsEngine
|
||||
|
||||
news = NewsEngine(PlatformConfig().news)
|
||||
context = news.get_fundamental_context()
|
||||
headlines = [str(getattr(h, "headline", ""))
|
||||
for h in (context.news_headlines or [])[:10]
|
||||
if getattr(h, "headline", None)]
|
||||
events = [
|
||||
{
|
||||
"event": getattr(event, "event", None),
|
||||
"currency": getattr(event, "currency", None),
|
||||
"actual": getattr(event, "actual", None),
|
||||
"forecast": getattr(event, "forecast", None),
|
||||
"released": not event.is_upcoming,
|
||||
}
|
||||
for event in (context.events_today or [])
|
||||
if event.is_high_impact
|
||||
][:8]
|
||||
gold_sentiment, _ = news.get_pair_sentiment(XAUUSD, context)
|
||||
remaining = sum(
|
||||
1 for event in news.get_week_events()
|
||||
if event.is_high_impact and event.is_upcoming
|
||||
)
|
||||
return {
|
||||
"headlines": headlines,
|
||||
"events": events,
|
||||
"gold_sentiment": _round_finite(gold_sentiment, 2),
|
||||
"remaining_high_impact_events": remaining,
|
||||
}
|
||||
except Exception as exc:
|
||||
logger.debug("market outlook: news context unavailable: %s", exc)
|
||||
return _empty_news_payload()
|
||||
|
||||
|
||||
def _recent_signals() -> List[Dict[str, Any]]:
|
||||
"""Read the latest signal history through SQLite's immutable URI."""
|
||||
try:
|
||||
path = os.path.abspath(signal_log_path())
|
||||
# mode=ro (not immutable=1): immutable ignores the WAL, so the
|
||||
# newest rows written by the live loggers were invisible here.
|
||||
uri = f"file:{path}?mode=ro"
|
||||
con = sqlite3.connect(uri, uri=True)
|
||||
try:
|
||||
rows = con.execute(
|
||||
"SELECT logged_at, timeframe, signal, quality, panel_verdict "
|
||||
"FROM signals ORDER BY id DESC LIMIT 10").fetchall()
|
||||
finally:
|
||||
con.close()
|
||||
return [
|
||||
{
|
||||
"logged_at": row[0],
|
||||
"timeframe": row[1],
|
||||
"signal": row[2],
|
||||
"quality": row[3],
|
||||
"panel_verdict": row[4],
|
||||
}
|
||||
for row in reversed(rows)
|
||||
]
|
||||
except Exception as exc:
|
||||
logger.debug("market outlook: signal history unavailable: %s", exc)
|
||||
return []
|
||||
|
||||
|
||||
def _build_payload(
|
||||
analyses: Dict[str, Any],
|
||||
candles: Dict[str, Any],
|
||||
news: Dict[str, Any],
|
||||
recent_signals: List[Dict[str, Any]],
|
||||
as_of: _dt.datetime,
|
||||
) -> Dict[str, Any]:
|
||||
"""Pure market-brief formatter. Every numeric field is finite or null."""
|
||||
technical = _analysis_payload(analyses)
|
||||
return _json_safe({
|
||||
"as_of": as_of.astimezone(_dt.timezone.utc).isoformat(),
|
||||
"candles": candles,
|
||||
**technical,
|
||||
"news": news,
|
||||
"recent_signals": recent_signals,
|
||||
})
|
||||
|
||||
|
||||
def _collect_payload(settings: Dict[str, Any]) -> Dict[str, Any]:
|
||||
"""Fetch closed bars and turn them into a structured market brief."""
|
||||
analyses: Dict[str, Any] = {}
|
||||
candles: Dict[str, Any] = {tf.value: None for tf in _OUTLOOK_TFS}
|
||||
bars = int(settings.get("candles_bars", DEFAULT_CANDLE_BARS))
|
||||
provider = _outlook_provider()
|
||||
engine = TechnicalEngine(PlatformConfig().technical)
|
||||
|
||||
for tf in _OUTLOOK_TFS:
|
||||
try:
|
||||
df = provider.fetch_ohlcv(XAUUSD, tf)
|
||||
analysis = engine.analyze(df, XAUUSD, tf.value)
|
||||
analyses[tf.value] = analysis
|
||||
atr = _finite(getattr(
|
||||
(getattr(analysis, "indicators", None) or {}).get("atr"),
|
||||
"value", None))
|
||||
if atr is not None and atr > 0 and df is not None and len(df) >= 2:
|
||||
candles[tf.value] = _normalize_candles(df, atr, bars)
|
||||
except Exception as exc:
|
||||
logger.debug("market outlook: %s data unavailable: %s", tf.value, exc)
|
||||
|
||||
return _build_payload(
|
||||
analyses=analyses,
|
||||
candles=candles,
|
||||
news=_news_payload(),
|
||||
recent_signals=_recent_signals(),
|
||||
as_of=_dt.datetime.now(_dt.timezone.utc),
|
||||
)
|
||||
|
||||
|
||||
# ── Orchestration, hourly dedup, and request budget ──────────────────
|
||||
|
||||
_state_lock = threading.Lock()
|
||||
_last_run_hour: Optional[int] = None # UTC hour bucket of the last attempt
|
||||
_outlook_day: Optional[_dt.date] = None
|
||||
_outlook_count = 0
|
||||
_warned_budget = False
|
||||
|
||||
|
||||
def _skip(reason: str) -> None:
|
||||
"""Heartbeat for every silent early return — a dark outlook feature
|
||||
(zero rows for 15 days, 2026-08/09) must be visible in the log."""
|
||||
logger.info("market outlook: skipped (%s)", reason)
|
||||
|
||||
|
||||
def reset_outlook_state() -> None:
|
||||
"""Drop hourly and daily state (tests / deployment config reloads)."""
|
||||
global _last_run_hour, _outlook_day, _outlook_count
|
||||
global _warned_budget
|
||||
with _state_lock:
|
||||
_last_run_hour = None
|
||||
_outlook_day = None
|
||||
_outlook_count = 0
|
||||
_warned_budget = False
|
||||
|
||||
|
||||
def _begin_attempt(cap: int, force: bool) -> bool:
|
||||
"""Atomically apply hourly dedup, then reserve a daily attempt.
|
||||
|
||||
Hourly dedup is by UTC hour BUCKET, not ``now - last < 3600``: the
|
||||
H1 loop calls this at boundary+10 s+scan time, so consecutive hours
|
||||
were routinely 3,590 s apart and every other hour was dropped."""
|
||||
global _last_run_hour, _outlook_day, _outlook_count
|
||||
global _warned_budget
|
||||
now = time.time()
|
||||
hour_bucket = int(now // 3600)
|
||||
today = _dt.datetime.now(_dt.timezone.utc).date()
|
||||
with _state_lock:
|
||||
if not force and _last_run_hour == hour_bucket:
|
||||
_skip(f"hour bucket {hour_bucket} already attempted")
|
||||
return False
|
||||
if _outlook_day != today:
|
||||
_outlook_day = today
|
||||
_outlook_count = 0
|
||||
_warned_budget = False
|
||||
if _outlook_count >= cap:
|
||||
if not _warned_budget:
|
||||
_warned_budget = True
|
||||
logger.warning(
|
||||
"market outlook: daily budget (%d) exhausted — no more "
|
||||
"outlooks until UTC midnight", cap)
|
||||
else:
|
||||
_skip("daily budget exhausted")
|
||||
return False
|
||||
_outlook_count += 1
|
||||
_last_run_hour = hour_bucket
|
||||
return True
|
||||
|
||||
|
||||
def run_outlook(settings: Dict[str, Any],
|
||||
surface: str = "live") -> Optional[OutlookResult]:
|
||||
"""Build, ask, validate, and persist one advisory outlook."""
|
||||
try:
|
||||
payload = _collect_payload(settings)
|
||||
reply = get_llm_client().chat_json(SYSTEM_PROMPT, payload)
|
||||
result = _validate_outlook(reply)
|
||||
if result is None:
|
||||
logger.warning(
|
||||
"market outlook: no outlook produced (%s)",
|
||||
"LLM returned nothing — cap/timeout/disabled" if reply is None
|
||||
else "reply failed validation")
|
||||
return None
|
||||
try:
|
||||
from signal_log import log_outlook
|
||||
log_outlook(result, surface=surface, payload=payload)
|
||||
except Exception:
|
||||
logger.debug("market outlook: persistence unavailable", exc_info=True)
|
||||
return result
|
||||
except Exception:
|
||||
logger.warning("market outlook: run failed", exc_info=True)
|
||||
return None
|
||||
|
||||
|
||||
def maybe_run_outlook(surface: str, force: bool = False) -> Optional[OutlookResult]:
|
||||
"""Best-effort public entry point. It NEVER raises to a scan caller.
|
||||
|
||||
``force`` is for explicit CLI use: it bypasses enabled, surface, and
|
||||
hourly gates, but not the independent outlook budget or shared LLM cap.
|
||||
"""
|
||||
try:
|
||||
settings = get_app_config().market_outlook_settings
|
||||
if not force:
|
||||
if not market_outlook_enabled():
|
||||
_skip("disabled")
|
||||
return None
|
||||
if surface == "telegram" and not settings.get("telegram", False):
|
||||
_skip("telegram surface off")
|
||||
return None
|
||||
if _provider_base._is_forex_weekend():
|
||||
_skip("weekend")
|
||||
return None
|
||||
cap = int(settings.get("max_per_day", DEFAULT_MAX_PER_DAY))
|
||||
if cap < 1:
|
||||
_skip("max_per_day < 1")
|
||||
return None
|
||||
if not _begin_attempt(cap, force):
|
||||
return None
|
||||
return run_outlook(settings, surface=surface)
|
||||
except Exception:
|
||||
logger.warning("market outlook: unavailable", exc_info=True)
|
||||
return None
|
||||
+1
-1
@@ -438,7 +438,7 @@ class NewsEngine:
|
||||
# here rather than in the fetch so the class-level cache and
|
||||
# the feed-health counts still describe what the feeds sent;
|
||||
# every consumer (aggregate score, summary, dashboard, panel,
|
||||
# outlook, signal log) reads context.news_headlines.
|
||||
# signal log) reads context.news_headlines.
|
||||
kept = [h for h in headlines if not _is_neutral_headline(h)]
|
||||
if len(kept) != len(headlines):
|
||||
logger.debug(
|
||||
|
||||
@@ -184,19 +184,6 @@ CREATE TABLE IF NOT EXISTS scan_outcomes (
|
||||
);
|
||||
CREATE INDEX IF NOT EXISTS idx_scan_outcomes_tf_bar
|
||||
ON scan_outcomes (timeframe, bar_ts);
|
||||
CREATE TABLE IF NOT EXISTS outlooks (
|
||||
id INTEGER PRIMARY KEY AUTOINCREMENT,
|
||||
logged_at TEXT NOT NULL,
|
||||
surface TEXT NOT NULL,
|
||||
bias TEXT,
|
||||
setup_forming TEXT,
|
||||
confidence REAL,
|
||||
watch TEXT,
|
||||
note TEXT,
|
||||
news_json TEXT,
|
||||
indicators_json TEXT,
|
||||
candles_json TEXT
|
||||
);
|
||||
"""
|
||||
|
||||
# Additive migration for columns introduced after the original schema.
|
||||
@@ -298,12 +285,6 @@ SCAN_LABEL_TIME_BUDGET = 2.0
|
||||
# often than a candle close. Explicit calls (batch script, tests) ignore it.
|
||||
SCAN_LABEL_MIN_INTERVAL = 60.0
|
||||
|
||||
_ADDITIVE_OUTLOOK_COLS = (
|
||||
("news_json", "TEXT"),
|
||||
("indicators_json", "TEXT"),
|
||||
("candles_json", "TEXT"),
|
||||
)
|
||||
|
||||
_SIGNAL_INSERT_COLS = (
|
||||
"logged_at", "source", "symbol", "timeframe", "signal", "score",
|
||||
"confidence", "quality", "action", "price", "spread", "market_regime",
|
||||
@@ -323,13 +304,6 @@ _INSERT = (
|
||||
+ ", ".join("?" for _ in _SIGNAL_INSERT_COLS) + ")"
|
||||
)
|
||||
|
||||
_OUTLOOK_INSERT = """
|
||||
INSERT INTO outlooks (
|
||||
logged_at, surface, bias, setup_forming, confidence, watch, note,
|
||||
news_json, indicators_json, candles_json
|
||||
) VALUES (?, ?, ?, ?, ?, ?, ?, ?, ?, ?)
|
||||
"""
|
||||
|
||||
|
||||
def _finite(x: Any) -> Optional[float]:
|
||||
"""float(x) or None for non-numeric / non-finite values (NaN/inf —
|
||||
@@ -851,7 +825,6 @@ class SignalLogger:
|
||||
self._with_lock_retry(
|
||||
lambda: self._conn.executescript(_SCHEMA))
|
||||
self._with_lock_retry(self._migrate_signal_columns)
|
||||
self._with_lock_retry(self._migrate_outlook_columns)
|
||||
self._with_lock_retry(self._ensure_news_view)
|
||||
|
||||
def _ensure_news_view(self) -> None:
|
||||
@@ -917,21 +890,6 @@ class SignalLogger:
|
||||
" ON signals (timeframe, bar_ts)")
|
||||
self._conn.commit()
|
||||
|
||||
def _migrate_outlook_columns(self) -> None:
|
||||
"""Add snapshot columns to pre-existing outlook tables."""
|
||||
existing = {
|
||||
r[1] for r in self._conn.execute("PRAGMA table_info(outlooks)")
|
||||
}
|
||||
for name, col_type in _ADDITIVE_OUTLOOK_COLS:
|
||||
if name not in existing:
|
||||
try:
|
||||
self._conn.execute(
|
||||
f"ALTER TABLE outlooks ADD COLUMN {name} {col_type}")
|
||||
except sqlite3.OperationalError as exc:
|
||||
if "duplicate column name" not in str(exc).lower():
|
||||
raise
|
||||
self._conn.commit()
|
||||
|
||||
def log(
|
||||
self,
|
||||
report: Any,
|
||||
@@ -1536,33 +1494,6 @@ class SignalLogger:
|
||||
self._conn.commit()
|
||||
return reseeded
|
||||
|
||||
def log_outlook(self, outlook: Any, surface: str,
|
||||
payload: Any = None) -> int:
|
||||
"""Insert one advisory market outlook; returns its rowid."""
|
||||
payload = payload if isinstance(payload, dict) else {}
|
||||
technical = {
|
||||
key: payload.get(key)
|
||||
for key in ("indicators", "market_regime",
|
||||
"volatility_regime", "key_levels")
|
||||
if key in payload
|
||||
}
|
||||
row = (
|
||||
datetime.now(timezone.utc).isoformat(),
|
||||
surface,
|
||||
getattr(outlook, "bias", None),
|
||||
getattr(outlook, "setup_forming", None),
|
||||
_finite(getattr(outlook, "confidence", None)),
|
||||
getattr(outlook, "watch", None),
|
||||
getattr(outlook, "note", None),
|
||||
_dumps(payload.get("news")) if "news" in payload else None,
|
||||
_dumps(technical) if technical else None,
|
||||
_dumps(payload.get("candles")) if "candles" in payload else None,
|
||||
)
|
||||
with self._lock:
|
||||
cur = self._conn.execute(_OUTLOOK_INSERT, row)
|
||||
self._conn.commit()
|
||||
return int(cur.lastrowid)
|
||||
|
||||
def close(self) -> None:
|
||||
with self._lock:
|
||||
self._conn.close()
|
||||
@@ -1700,28 +1631,6 @@ def log_watch_update(report: Any, watch: Any) -> None:
|
||||
exc_info=True)
|
||||
|
||||
|
||||
def log_outlook(outlook: Any, surface: str = "live",
|
||||
payload: Any = None) -> None:
|
||||
"""Persist one advisory outlook when signal logging is enabled.
|
||||
|
||||
Outlook persistence is deliberately independent of a signal row: the
|
||||
feature runs on its own hourly cadence and must never affect a scan.
|
||||
"""
|
||||
try:
|
||||
if not app_config.signal_log_enabled():
|
||||
return
|
||||
path = app_config.signal_log_path()
|
||||
with _loggers_lock:
|
||||
lg = _loggers.get(path)
|
||||
if lg is None:
|
||||
lg = SignalLogger(path)
|
||||
_loggers[path] = lg
|
||||
lg.log_outlook(outlook, surface, payload=payload)
|
||||
except Exception:
|
||||
logger.warning("signal_log: failed to persist market outlook",
|
||||
exc_info=True)
|
||||
|
||||
|
||||
def reset_loggers_for_tests() -> None:
|
||||
"""Close and drop all cached loggers (test isolation)."""
|
||||
with _loggers_lock:
|
||||
|
||||
@@ -75,7 +75,6 @@ from providers import BaseDataProvider, make_data_provider
|
||||
from providers.base import _is_daily_maintenance
|
||||
from agent_panel import attach_panel, panel_health
|
||||
from llm_client import get_llm_client
|
||||
from market_outlook import maybe_run_outlook
|
||||
from news_engine import FundamentalContext, NewsEngine
|
||||
from pair_signals import PAIR_PRODUCERS
|
||||
from signal_aggregator import SignalAggregator, SignalReport
|
||||
@@ -919,28 +918,6 @@ async def _scan_timeframe(
|
||||
_commit_alert(key, report)
|
||||
logger.info(_alert_sent_line(report, tf.value, delivered))
|
||||
|
||||
# This independent advisory call runs only after the H1 scan completes.
|
||||
# Offloading the LLM request keeps the event loop responsive.
|
||||
if tf == Timeframe.H1:
|
||||
try:
|
||||
outlook = await asyncio.get_running_loop().run_in_executor(
|
||||
_executor, maybe_run_outlook, "telegram")
|
||||
settings = get_app_config().market_outlook_settings
|
||||
if (outlook is not None and settings.get("telegram", False)
|
||||
and OWNER_ID is not None):
|
||||
forming = {
|
||||
"buy": "possible buy forming",
|
||||
"sell": "possible sell forming",
|
||||
"none": "no clear setup forming",
|
||||
}[outlook.setup_forming]
|
||||
message = (
|
||||
f"🔭 AI outlook: {outlook.bias}, {forming} "
|
||||
f"({outlook.confidence:.0f}%) — {_escape_md(outlook.note)}"
|
||||
)
|
||||
await _send_alert(bot, OWNER_ID, message)
|
||||
except Exception:
|
||||
logger.exception("Hourly AI market outlook failed")
|
||||
|
||||
|
||||
# ── WebSocket level-breach monitor ──────────────────────────────────
|
||||
|
||||
|
||||
+84
-405
@@ -3323,6 +3323,36 @@ def _():
|
||||
f"example config.json must be byte-identical to defaults on {tf}"
|
||||
|
||||
|
||||
@test("app_config: a legacy market_outlook block is tolerated and ignored")
|
||||
def _():
|
||||
# The market-outlook feature was removed 2026-09-15; config.json files
|
||||
# on the live box still carry its block and must keep loading (the
|
||||
# oanda.account_id precedent), with one warning so the owner deletes it.
|
||||
import logging
|
||||
records = []
|
||||
|
||||
class _Grab(logging.Handler):
|
||||
def emit(self, record):
|
||||
records.append(record)
|
||||
|
||||
log = logging.getLogger("app_config")
|
||||
handler = _Grab()
|
||||
log.addHandler(handler)
|
||||
path = _write_json({
|
||||
"market_outlook": {"enabled": True, "telegram": True,
|
||||
"max_per_day": 12, "candles_bars": 20},
|
||||
})
|
||||
try:
|
||||
app = load_app_config(path)
|
||||
finally:
|
||||
log.removeHandler(handler)
|
||||
os.remove(path)
|
||||
assert not hasattr(app, "market_outlook_settings")
|
||||
assert any(r.levelno == logging.WARNING
|
||||
and "market_outlook" in r.getMessage() for r in records), \
|
||||
"loading a legacy market_outlook block must log one warning"
|
||||
|
||||
|
||||
@test("providers: api key resolves env → config.json (no code default)")
|
||||
def _():
|
||||
from providers import TwelveDataProvider
|
||||
@@ -8528,409 +8558,6 @@ def _():
|
||||
os.remove(path + suffix)
|
||||
|
||||
|
||||
# ══════════════════════════════════════════════════════════════════════
|
||||
# MODULE 26: market_outlook.py (advisory hourly market brief)
|
||||
# ══════════════════════════════════════════════════════════════════════
|
||||
|
||||
print("\n═══ Module 26: AI market outlook ═══")
|
||||
|
||||
import market_outlook as mo
|
||||
|
||||
|
||||
class _OutlookStub:
|
||||
available = True
|
||||
|
||||
def __init__(self, reply=None):
|
||||
self.reply = reply or {
|
||||
"bias": "bullish", "setup_forming": "buy", "confidence": 68,
|
||||
"watch": "Look for the price to hold above its recent low.",
|
||||
"note": "The short-term chart is improving while the bigger trend remains supportive.",
|
||||
}
|
||||
self.calls = []
|
||||
|
||||
def chat_json(self, system_prompt, user_payload):
|
||||
self.calls.append((system_prompt, user_payload))
|
||||
if isinstance(self.reply, Exception):
|
||||
raise self.reply
|
||||
return self.reply
|
||||
|
||||
|
||||
class _OutlookNews:
|
||||
def __init__(self, config):
|
||||
pass
|
||||
|
||||
def get_fundamental_context(self):
|
||||
now = datetime.now(timezone.utc)
|
||||
return FundamentalContext(
|
||||
news_headlines=[NewsSentiment(
|
||||
headline="Gold firms as the dollar weakens", source="test",
|
||||
timestamp=now, currencies_mentioned=["XAU", "USD"],
|
||||
sentiment_score=0.3, keywords=[])],
|
||||
events_today=[EconomicEvent(
|
||||
time=now - timedelta(minutes=5), currency="USD", impact="high",
|
||||
event="Inflation", actual="0.2%", forecast="0.3%")],
|
||||
)
|
||||
|
||||
def get_pair_sentiment(self, symbol, context):
|
||||
return 0.25, "slightly bullish"
|
||||
|
||||
def get_week_events(self):
|
||||
return []
|
||||
|
||||
|
||||
_OUTLOOK_ON = {
|
||||
"enabled": True,
|
||||
"telegram": True,
|
||||
"max_per_day": 12,
|
||||
"candles_bars": 20,
|
||||
}
|
||||
|
||||
|
||||
@test("app_config: market_outlook defaults, env override, and validation")
|
||||
def _():
|
||||
defaults = load_app_config(_write_json({})).market_outlook_settings
|
||||
assert defaults == {
|
||||
"enabled": False, "telegram": False,
|
||||
"max_per_day": 12, "candles_bars": 20,
|
||||
}
|
||||
app = load_app_config(_write_json({"market_outlook": _OUTLOOK_ON}))
|
||||
assert app.market_outlook_settings == _OUTLOOK_ON
|
||||
old = os.environ.get("MARKET_OUTLOOK_ENABLED")
|
||||
try:
|
||||
os.environ["MARKET_OUTLOOK_ENABLED"] = "false"
|
||||
assert app_config.market_outlook_enabled(app) is False
|
||||
os.environ["MARKET_OUTLOOK_ENABLED"] = "true"
|
||||
assert app_config.market_outlook_enabled(app) is True
|
||||
finally:
|
||||
if old is None:
|
||||
os.environ.pop("MARKET_OUTLOOK_ENABLED", None)
|
||||
else:
|
||||
os.environ["MARKET_OUTLOOK_ENABLED"] = old
|
||||
for bad in (
|
||||
{"market_outlook": {"enabld": True}},
|
||||
{"market_outlook": {"enabled": "yes"}},
|
||||
{"market_outlook": {"telegram": 1}},
|
||||
{"market_outlook": {"max_per_day": 0}},
|
||||
{"market_outlook": {"max_per_day": True}},
|
||||
{"market_outlook": {"candles_bars": 2.5}},
|
||||
):
|
||||
try:
|
||||
load_app_config(_write_json(bad))
|
||||
assert False, f"should have rejected {bad}"
|
||||
except AppConfigError:
|
||||
pass
|
||||
|
||||
|
||||
@test("signal_log: market outlook table roundtrip and old-db migration")
|
||||
def _():
|
||||
import sqlite3
|
||||
import signal_log as slog
|
||||
fd, path = tempfile.mkstemp(suffix=".db")
|
||||
os.close(fd)
|
||||
try:
|
||||
con = sqlite3.connect(path)
|
||||
# Legacy pre-panel schema: it has a signals table but no outlooks
|
||||
# table. Keep the index columns from the original schema present.
|
||||
con.execute(
|
||||
"CREATE TABLE signals (id INTEGER PRIMARY KEY, logged_at TEXT,"
|
||||
" symbol TEXT, timeframe TEXT)")
|
||||
con.execute(
|
||||
"CREATE TABLE outlooks (id INTEGER PRIMARY KEY, logged_at TEXT,"
|
||||
" surface TEXT, bias TEXT, setup_forming TEXT, confidence REAL,"
|
||||
" watch TEXT, note TEXT)")
|
||||
con.commit()
|
||||
con.close()
|
||||
barrier = threading.Barrier(4)
|
||||
loggers = []
|
||||
errors = []
|
||||
|
||||
def open_logger():
|
||||
try:
|
||||
barrier.wait()
|
||||
loggers.append(slog.SignalLogger(path))
|
||||
except Exception as exc:
|
||||
errors.append(exc)
|
||||
|
||||
threads = [threading.Thread(target=open_logger) for _ in range(4)]
|
||||
for thread in threads:
|
||||
thread.start()
|
||||
for thread in threads:
|
||||
thread.join(timeout=5)
|
||||
assert not errors, f"concurrent additive migration failed: {errors}"
|
||||
assert len(loggers) == 4
|
||||
logger = loggers[0]
|
||||
columns = {
|
||||
row[1] for row in logger._conn.execute("PRAGMA table_info(signals)")
|
||||
}
|
||||
assert {
|
||||
"indicators_json", "candles_json", "panel_created_at",
|
||||
"panel_cache_status", "panel_payload_hash",
|
||||
"advisory_outcome_status", "advisory_r_multiple",
|
||||
"advisory_outcome_json",
|
||||
} <= columns
|
||||
outlook_columns = {
|
||||
row[1] for row in logger._conn.execute("PRAGMA table_info(outlooks)")
|
||||
}
|
||||
assert {"news_json", "indicators_json", "candles_json"} <= \
|
||||
outlook_columns
|
||||
result = mo.OutlookResult(
|
||||
bias="bearish", setup_forming="sell", confidence=71.5,
|
||||
watch="Wait for price to fail at resistance.",
|
||||
note="Momentum is fading after a sharp rise.", created_ts=time.time())
|
||||
payload = {
|
||||
"news": {"headlines": ["Gold firms"]},
|
||||
"indicators": {"1h": {"rsi": {"value": 55.0}}},
|
||||
"market_regime": {"1h": "trending_up"},
|
||||
"volatility_regime": {"1h": "normal"},
|
||||
"key_levels": {"1h": {"support": 4400.0}},
|
||||
"candles": {"1h": {"bars": [[-0.2, 0.1, -0.3, 0.0]]}},
|
||||
}
|
||||
rowid = logger.log_outlook(result, surface="test", payload=payload)
|
||||
for open_instance in loggers:
|
||||
open_instance.close()
|
||||
con = sqlite3.connect(path)
|
||||
row = con.execute(
|
||||
"SELECT surface, bias, setup_forming, confidence, watch, note, "
|
||||
"news_json, indicators_json, candles_json "
|
||||
"FROM outlooks WHERE id=?", (rowid,)).fetchone()
|
||||
con.close()
|
||||
assert row[:6] == ("test", "bearish", "sell", 71.5,
|
||||
"Wait for price to fail at resistance.",
|
||||
"Momentum is fading after a sharp rise.")
|
||||
assert json.loads(row[6])["headlines"] == ["Gold firms"]
|
||||
assert json.loads(row[7])["market_regime"]["1h"] == "trending_up"
|
||||
assert len(json.loads(row[8])["1h"]["bars"]) == 1
|
||||
finally:
|
||||
slog.reset_loggers_for_tests()
|
||||
os.remove(path)
|
||||
|
||||
|
||||
@test("market_outlook: payload is finite, normalized, and news-safe")
|
||||
def _():
|
||||
try:
|
||||
mo._dp = _FakeDP()
|
||||
with patch("news_engine.NewsEngine", _OutlookNews):
|
||||
payload = mo._collect_payload(_OUTLOOK_ON)
|
||||
assert mo._normalize_candles is ap._normalize_candles
|
||||
assert set(payload["candles"]) == {"15min", "1h", "4h"}
|
||||
assert all(len(block["bars"]) == 20
|
||||
for block in payload["candles"].values())
|
||||
assert payload["indicators"]["1h"]
|
||||
assert payload["news"]["headlines"] == ["Gold firms as the dollar weakens"]
|
||||
json.dumps(payload, sort_keys=True, allow_nan=False)
|
||||
|
||||
with patch("news_engine.NewsEngine", side_effect=RuntimeError("down")):
|
||||
no_news = mo._collect_payload(_OUTLOOK_ON)["news"]
|
||||
assert no_news == mo._empty_news_payload()
|
||||
finally:
|
||||
mo._dp = None
|
||||
|
||||
|
||||
@test("market_outlook: reply validation clamps and rejects malformed JSON")
|
||||
def _():
|
||||
validate = mo._validate_outlook
|
||||
assert validate(None) is None
|
||||
assert validate({"bias": "up", "setup_forming": "buy", "confidence": 50}) is None
|
||||
assert validate({"bias": "bullish", "setup_forming": "wait", "confidence": 50}) is None
|
||||
assert validate({"bias": "bullish", "setup_forming": "buy"}) is None
|
||||
assert validate({"bias": "bullish", "setup_forming": "buy", "confidence": "hot"}) is None
|
||||
result = validate({
|
||||
"bias": "neutral", "setup_forming": "none", "confidence": 150,
|
||||
"watch": "word " * 30, "note": "note " * 30,
|
||||
})
|
||||
assert result.confidence == 100.0
|
||||
assert len(result.watch.split()) == 25 and len(result.note.split()) == 25
|
||||
|
||||
|
||||
@test("market_outlook: disabled default spends no LLM request")
|
||||
def _():
|
||||
original_app = app_config.get_app_config()
|
||||
old = os.environ.pop("MARKET_OUTLOOK_ENABLED", None)
|
||||
try:
|
||||
app_config.set_app_config(AppConfig())
|
||||
stub = _OutlookStub()
|
||||
lc._client = stub
|
||||
mo.reset_outlook_state()
|
||||
assert mo.maybe_run_outlook("telegram") is None
|
||||
assert stub.calls == []
|
||||
finally:
|
||||
if old is not None:
|
||||
os.environ["MARKET_OUTLOOK_ENABLED"] = old
|
||||
lc._client = None
|
||||
mo.reset_outlook_state()
|
||||
app_config.set_app_config(original_app)
|
||||
|
||||
|
||||
@test("market_outlook: hourly dedup prevents a second request")
|
||||
def _():
|
||||
original_app = app_config.get_app_config()
|
||||
old = os.environ.pop("MARKET_OUTLOOK_ENABLED", None)
|
||||
try:
|
||||
app_config.set_app_config(AppConfig(market_outlook_settings=dict(_OUTLOOK_ON)))
|
||||
stub = _OutlookStub()
|
||||
lc._client = stub
|
||||
mo.reset_outlook_state()
|
||||
with patch("providers.base._is_forex_weekend", return_value=False), \
|
||||
patch.object(mo, "_collect_payload", return_value={}), \
|
||||
patch("signal_log.log_outlook"):
|
||||
assert mo.maybe_run_outlook("telegram") is not None
|
||||
assert mo.maybe_run_outlook("telegram") is None
|
||||
assert len(stub.calls) == 1
|
||||
finally:
|
||||
if old is not None:
|
||||
os.environ["MARKET_OUTLOOK_ENABLED"] = old
|
||||
lc._client = None
|
||||
mo.reset_outlook_state()
|
||||
app_config.set_app_config(original_app)
|
||||
|
||||
|
||||
@test("market_outlook: daily budget blocks force calls after the cap")
|
||||
def _():
|
||||
original_app = app_config.get_app_config()
|
||||
try:
|
||||
app_config.set_app_config(AppConfig(market_outlook_settings={
|
||||
**_OUTLOOK_ON, "max_per_day": 1}))
|
||||
stub = _OutlookStub()
|
||||
lc._client = stub
|
||||
mo.reset_outlook_state()
|
||||
with patch("providers.base._is_forex_weekend", return_value=False), \
|
||||
patch.object(mo, "_collect_payload", return_value={}), \
|
||||
patch("signal_log.log_outlook"):
|
||||
assert mo.maybe_run_outlook("cli", force=True) is not None
|
||||
assert mo.maybe_run_outlook("cli", force=True) is None
|
||||
assert len(stub.calls) == 1
|
||||
finally:
|
||||
lc._client = None
|
||||
mo.reset_outlook_state()
|
||||
app_config.set_app_config(original_app)
|
||||
|
||||
|
||||
@test("market_outlook: LLM and persistence failures remain fail-safe")
|
||||
def _():
|
||||
original_app = app_config.get_app_config()
|
||||
try:
|
||||
app_config.set_app_config(AppConfig(market_outlook_settings=dict(_OUTLOOK_ON)))
|
||||
lc._client = _OutlookStub(RuntimeError("LLM down"))
|
||||
mo.reset_outlook_state()
|
||||
with patch("providers.base._is_forex_weekend", return_value=False), \
|
||||
patch.object(mo, "_collect_payload", return_value={}):
|
||||
assert mo.maybe_run_outlook("telegram") is None
|
||||
|
||||
lc._client = _OutlookStub()
|
||||
mo.reset_outlook_state()
|
||||
with patch("providers.base._is_forex_weekend", return_value=False), \
|
||||
patch.object(mo, "_collect_payload", return_value={}), \
|
||||
patch("signal_log.log_outlook", side_effect=RuntimeError("db locked")):
|
||||
assert mo.maybe_run_outlook("telegram") is not None
|
||||
finally:
|
||||
lc._client = None
|
||||
mo.reset_outlook_state()
|
||||
app_config.set_app_config(original_app)
|
||||
|
||||
|
||||
@test("market_outlook: weekend gate spends nothing")
|
||||
def _():
|
||||
original_app = app_config.get_app_config()
|
||||
try:
|
||||
app_config.set_app_config(AppConfig(market_outlook_settings=dict(_OUTLOOK_ON)))
|
||||
stub = _OutlookStub()
|
||||
lc._client = stub
|
||||
mo.reset_outlook_state()
|
||||
with patch("providers.base._is_forex_weekend", return_value=True):
|
||||
assert mo.maybe_run_outlook("telegram") is None
|
||||
assert stub.calls == []
|
||||
finally:
|
||||
lc._client = None
|
||||
mo.reset_outlook_state()
|
||||
app_config.set_app_config(original_app)
|
||||
|
||||
|
||||
@test("market_outlook: hour-bucket dedup runs consecutive hours 3,590 s apart")
|
||||
def _():
|
||||
original_app = app_config.get_app_config()
|
||||
old = os.environ.pop("MARKET_OUTLOOK_ENABLED", None)
|
||||
try:
|
||||
app_config.set_app_config(AppConfig(market_outlook_settings=dict(_OUTLOOK_ON)))
|
||||
stub = _OutlookStub()
|
||||
lc._client = stub
|
||||
mo.reset_outlook_state()
|
||||
base = 1_800_000_000 - (1_800_000_000 % 3600) # an exact hour
|
||||
clock = {"t": base + 25.0} # 10:00:25
|
||||
with patch("providers.base._is_forex_weekend", return_value=False), \
|
||||
patch.object(mo, "_collect_payload", return_value={}), \
|
||||
patch("signal_log.log_outlook"), \
|
||||
patch.object(mo.time, "time", lambda: clock["t"]):
|
||||
assert mo.maybe_run_outlook("telegram") is not None
|
||||
clock["t"] = base + 1800.0 # 10:30 same bucket
|
||||
assert mo.maybe_run_outlook("telegram") is None
|
||||
clock["t"] = base + 3600.0 + 15.0 # 11:00:15 — 3,590 s later
|
||||
assert mo.maybe_run_outlook("telegram") is not None, \
|
||||
"the old `now - last < 3600` rule dropped every other hour"
|
||||
assert len(stub.calls) == 2
|
||||
finally:
|
||||
if old is not None:
|
||||
os.environ["MARKET_OUTLOOK_ENABLED"] = old
|
||||
lc._client = None
|
||||
mo.reset_outlook_state()
|
||||
app_config.set_app_config(original_app)
|
||||
|
||||
|
||||
@test("market_outlook: every silent early return leaves a heartbeat log line")
|
||||
def _():
|
||||
original_app = app_config.get_app_config()
|
||||
old = os.environ.pop("MARKET_OUTLOOK_ENABLED", None)
|
||||
records = []
|
||||
|
||||
class _Grab(logging.Handler):
|
||||
def emit(self, record):
|
||||
records.append(record.getMessage())
|
||||
|
||||
handler = _Grab()
|
||||
mo.logger.addHandler(handler)
|
||||
prev_level = mo.logger.level
|
||||
mo.logger.setLevel(logging.INFO)
|
||||
try:
|
||||
app_config.set_app_config(AppConfig(market_outlook_settings={
|
||||
**_OUTLOOK_ON, "enabled": False}))
|
||||
mo.reset_outlook_state()
|
||||
assert mo.maybe_run_outlook("telegram") is None
|
||||
assert any("skipped (disabled)" in m for m in records), records
|
||||
records.clear()
|
||||
app_config.set_app_config(AppConfig(market_outlook_settings=dict(_OUTLOOK_ON)))
|
||||
with patch("providers.base._is_forex_weekend", return_value=True):
|
||||
assert mo.maybe_run_outlook("telegram") is None
|
||||
assert any("skipped (weekend)" in m for m in records), records
|
||||
records.clear()
|
||||
# LLM returns nothing → warning, nothing persisted
|
||||
stub = _OutlookStub(reply=None)
|
||||
stub.reply = None
|
||||
lc._client = stub
|
||||
persisted = []
|
||||
with patch("providers.base._is_forex_weekend", return_value=False), \
|
||||
patch.object(mo, "_collect_payload", return_value={}), \
|
||||
patch("signal_log.log_outlook", persisted.append):
|
||||
assert mo.maybe_run_outlook("telegram") is None
|
||||
assert any("no outlook produced" in m for m in records), records
|
||||
assert persisted == []
|
||||
finally:
|
||||
mo.logger.removeHandler(handler)
|
||||
mo.logger.setLevel(prev_level)
|
||||
if old is not None:
|
||||
os.environ["MARKET_OUTLOOK_ENABLED"] = old
|
||||
lc._client = None
|
||||
mo.reset_outlook_state()
|
||||
app_config.set_app_config(original_app)
|
||||
|
||||
|
||||
@test("market_outlook isolation: signal path never imports the advisory module")
|
||||
def _():
|
||||
root = os.path.dirname(__file__)
|
||||
for name in ("engine.py", "signal_aggregator.py"):
|
||||
with open(os.path.join(root, name), encoding="utf-8") as fh:
|
||||
assert "market_outlook" not in fh.read(), name
|
||||
|
||||
|
||||
|
||||
# ══════════════════════════════════════════════════════════════════════
|
||||
# MODULE 27: signal-log observability (2026-09 audit)
|
||||
# ══════════════════════════════════════════════════════════════════════
|
||||
@@ -10536,6 +10163,59 @@ def _():
|
||||
assert weighted_agg.gate_config_fingerprint() == m5.gate_config_fingerprint()
|
||||
|
||||
|
||||
@test("signal_log: concurrent legacy-DB migration adds the panel/advisory columns")
|
||||
def _():
|
||||
# Four loggers open the same pre-panel database at once: the additive
|
||||
# ALTER TABLE migration must survive the race (_with_lock_retry / WAL).
|
||||
# Moved from the removed market-outlook module (2026-09-15); it was the
|
||||
# suite's only concurrent-migration regression.
|
||||
import sqlite3
|
||||
import signal_log as slog
|
||||
fd, path = tempfile.mkstemp(suffix=".db")
|
||||
os.close(fd)
|
||||
try:
|
||||
con = sqlite3.connect(path)
|
||||
# Legacy pre-panel schema. Keep the index columns from the original
|
||||
# schema present.
|
||||
con.execute(
|
||||
"CREATE TABLE signals (id INTEGER PRIMARY KEY, logged_at TEXT,"
|
||||
" symbol TEXT, timeframe TEXT)")
|
||||
con.commit()
|
||||
con.close()
|
||||
barrier = threading.Barrier(4)
|
||||
loggers = []
|
||||
errors = []
|
||||
|
||||
def open_logger():
|
||||
try:
|
||||
barrier.wait()
|
||||
loggers.append(slog.SignalLogger(path))
|
||||
except Exception as exc:
|
||||
errors.append(exc)
|
||||
|
||||
threads = [threading.Thread(target=open_logger) for _ in range(4)]
|
||||
for thread in threads:
|
||||
thread.start()
|
||||
for thread in threads:
|
||||
thread.join(timeout=5)
|
||||
assert not errors, f"concurrent additive migration failed: {errors}"
|
||||
assert len(loggers) == 4
|
||||
columns = {
|
||||
row[1] for row in loggers[0]._conn.execute("PRAGMA table_info(signals)")
|
||||
}
|
||||
assert {
|
||||
"indicators_json", "candles_json", "panel_created_at",
|
||||
"panel_cache_status", "panel_payload_hash",
|
||||
"advisory_outcome_status", "advisory_r_multiple",
|
||||
"advisory_outcome_json",
|
||||
} <= columns
|
||||
for open_instance in loggers:
|
||||
open_instance.close()
|
||||
finally:
|
||||
slog.reset_loggers_for_tests()
|
||||
os.remove(path)
|
||||
|
||||
|
||||
@test("signal_log: migration adds the research columns, scan_outcomes and indexes")
|
||||
def _():
|
||||
import sqlite3
|
||||
@@ -11906,8 +11586,7 @@ def _():
|
||||
# the backtester.
|
||||
banned = {"signal_log", "news_engine", "providers", "telegram_bot",
|
||||
"news_archive", "agent_panel", "violent_watch",
|
||||
"market_outlook", "gpu_backtest", "sklearn", "scipy",
|
||||
"lightgbm", "torch"}
|
||||
"gpu_backtest", "sklearn", "scipy", "lightgbm", "torch"}
|
||||
path = Path("backtest/replay_record.py")
|
||||
tree = ast.parse(path.read_text(encoding="utf-8"), str(path))
|
||||
seen, offenders = set(), []
|
||||
|
||||
+1
-1
@@ -7,7 +7,7 @@ hour may be wild. It is a WARNING LINE and a standalone heads-up
|
||||
message — nothing here ever changes a signal, grade, gate, trade setup
|
||||
or backtest result.
|
||||
|
||||
Hard independence rule (mirrors market_outlook.py / agent_panel.py):
|
||||
Hard independence rule (mirrors agent_panel.py):
|
||||
this module is NEVER imported by `signal_aggregator`, `engine`, or
|
||||
anything under `backtest/`. Surfaces (telegram_bot, main, the dashboard
|
||||
publisher) attach the result POST-HOC via `attach_watch`, exactly the
|
||||
|
||||
Reference in New Issue
Block a user