ForexBot2/CLAUDE.md

# CLAUDE.md

This file provides guidance to Claude Code (claude.ai/code) when working with code in this repository.

## What this is

Signal-intelligence platform for **XAU/USD (gold — the only supported pair)**. It fetches OHLCV from the active data provider (**Oanda v20 since 2026-07-28**; Twelve Data and Tiingo remain registered fallbacks), scores 9 technical indicators per timeframe with regime-adaptive weights, checks multi-timeframe agreement and news risk, grades the signal (A+…F), and generates a full trade setup (entry, SL, TP1/TP2, trailing plan). Delivery: Telegram bot, CLI, and a static web dashboard. A realism-focused backtester replays history through the *exact same* signal code.

Every numeric threshold in `config.py` and `pair_signals/xauusd.py` was backtested or is load-bearing for a backtested result. Validation evidence lives in the factory/module docstrings and in `~/.claude/projects/.../memory/`.

## Commands

```bash
python test_all.py                      # QA suite, no network (mocked); prints its own count
                                        #  (498 tests / 33 modules as of 2026-09-09)
python main.py --pair XAUUSD --timeframe H1 --detail   # CLI signal report (--json for full dump,
                                        #  --panel for the advisory LLM agent panel, --outlook for the
                                        #  hourly AI market brief; the violent-move watch line appears
                                        #  when the `violent_watch` config block is enabled;
                                        #  --timeframe accepts M5/M15/H1/H4 only and the confirmation
                                        #  TFs always come from the pair factory's map)
python telegram_bot.py                  # the bot (refuses to boot without credentials)
python dashboard_publisher.py --once    # web dashboard: one scan+publish cycle to $DASHBOARD_OUT_DIR
                                        # (omit --once for the candle-close daemon; deploy/README-dashboard.md)

# Backtests
python backtest/gpu_backtest.py --pair XAUUSD --days 365
python backtest/gpu_backtest.py --pair XAUUSD --start 2025-01-01 --end 2025-06-01 \
    --primary-tf 15min --confirm-tfs 1h,4h
# key flags: --spread --quality A+,A,B,C --strategies --workers --max-hold-bars
#            --warmup-bars --cooldown-bars --no-cache --export-csv out.csv --detail
#            --set PATH=JSON (research override of any PlatformConfig leaf; unknown path = hard exit)
#            --export-features (append the 65 decision-time f_* columns; default export stays 63 cols)
#            --fixed-rr-tp N (research-only: hard TP at N×risk, trail off; verdict
#            2026-08-05: NOT shipped — backtest/research/fixedtp_2026-08-05.md)
#            --record DIR [--record-bars] (research: manifest + trades/indicators/events parquet for
#            every accepted trade, and one row per analyzed bar; the book is unchanged — pair every
#            recorded run with a control run and show the MD5s equal; read with
#            backtest.replay_record.load_run, first look with backtest/replay_scorecard.py DIR --check)
```

**`test_all.py` is NOT pytest** — it uses a custom `@test("name")` decorator that runs each test immediately at import; don't grep for `def test_` or try `pytest test_all.py`. Run the whole file; it prints per-module results and exits non-zero on failure. There is no way to run a single test short of commenting modules out. Tests must stay green; every new gate/knob needs tests. Signal-log fixtures: use a WEEKDAY signal bar when a test exercises candle continuity — the outcome model treats weekend and 21:00 UTC gaps as scheduled, so Saturday-dated bars never produce a gap. `_candles_json` stores `df.tail(14)`, so a test snapshot longer than 14 bars silently drops its head. When patching weekend logic in tests, patch `providers.base._is_forex_weekend` — but know there are THREE independent signal-path weekend implementations (`providers/base.py`, `signal_aggregator.py`, and an inline copy in `backtest/gpu_backtest.py:_is_xau_blocked_by_calendar`) plus the advisory-only continuity copy in `advisory_outcome.py`; patching one does not reach the others.

Python 3.12; a `venv/` exists in-repo. Dependencies in `requirements.txt` (numpy, pandas, requests, twelvedata, python-telegram-bot ≥20, websockets). scikit-learn and lightgbm are installed in the dev `venv/` for ad-hoc research only — never add them to `requirements.txt` or import them from live code; anything shipped must be numpy-only.

## Configuration & credentials

Precedence for every credential/deployment setting: **env var → config.json → refuse to run** (no hardcoded fallbacks). A gitignored repo-root `.env` is loaded at `app_config` import via `os.environ.setdefault`. `config.json` (gitignored; schema in `config.json.example`) holds secrets plus deployment settings: `data_providers.active` (the live-feed switch), `data_providers.streaming` (optional tick-stream provider for the level-breach monitor), `scanning.pairs`/`timeframes`, per-pair `blocked_hours_utc`, `log_level`. Key env vars: `TELEGRAM_TOKEN`, `TELEGRAM_CHAT_ID`, `TELEGRAM_CHANNEL_ID`, `TWELVE_DATA_API_KEY`, `TIINGO_API_KEY`, `OANDA_API_TOKEN`/`OANDA_BASE_URL` (the data feed needs no account id; a legacy `oanda.account_id` key is tolerated and ignored), `DEEPSEEK_API_KEY` (news LLM; the `deepseek` config.json block holds model/enabled/timeout/daily cap). The `news_archive` block (`enabled` default TRUE, `path` default `logs/newsdata.db`; env `NEWS_ARCHIVE_ENABLED`/`NEWS_ARCHIVE_PATH`) controls the durable headline corpus. The `agent_panel` config.json block (all-defaults = feature dark) controls the advisory LLM agent panel: `enabled`, per-surface `telegram`/`dashboard` toggles, `min_quality`, `max_panels_per_day`, `reuse_ttl_seconds`, plus the technical-payload enrichment knobs `candles_enabled`/`candle_bars_primary`/`candle_bars_confirm`. The dark-by-default `market_outlook` block controls the hourly advisory market brief: `enabled`, `telegram`, `max_per_day`, `candles_bars`; `MARKET_OUTLOOK_ENABLED` overrides its enabled setting. The dark-by-default `violent_watch` block controls the display-only violent-move watch: `enabled`, `alert_line`, `headsup`, `max_headsup_per_day`, `headsup_min_level` (`"watch"`/`"hot"`); `VIOLENT_WATCH_ENABLED` overrides its enabled setting.

## Architecture

### End-to-end pipeline (every scan)

```
providers/<active>.fetch_ohlcv()       per TF: single-flight fetch, drop forming candle, validate/clean, cache
                                       (only Twelve Data rate-throttles; Oanda has no client throttle)
  → engine.TechnicalEngine.analyze()   per TF: 9 scored indicators + ATR context → regime → composite score
  → signal_aggregator.analyze_pair()   live entry point (weekend/day filters, fetch, staleness)
      └─ analyze_from_dataframes()     replayable core: MTF agreement → news → bonuses/penalties →
                                       final signal + quality grade → gates → trade setup → reasons
  → SignalReport                       consumed by telegram_bot.py, main.py, dashboard_publisher.py, backtester
```

Signal currency everywhere is `SignalStrength` (−2..+2). DataFrames are UTC tz-aware, oldest-first, capitalized OHLCV columns. Everything runs on **closed bars only** — the forming candle is dropped, and session/hour filters evaluate **bar-CLOSE time** (bar label + candle duration) identically in live and backtest.

### Layers

- **`providers/`** — one class per provider subclassing `BaseDataProvider` (`providers/base.py`), which owns everything the pipeline depends on: TTL+boundary cache, forming-candle drop, freshness check, validate/clean, paginated `fetch_range` (feeds the backtester from any provider), optional tick streaming. Subclasses implement only `_fetch_raw` + symbol/interval maps. **Cleaning rule (2026-09-07):** the signal-side outlier filter is a bad-print filter, not a volatility filter — a >10σ close-to-close bar is dropped only when the NEXT close snaps back ≥80% of the way (`OUTLIER_SIGMA`/`OUTLIER_REVERT_FRACTION` on the base class); persisting moves (NFP/CPI/Sunday-gap bars) and the newest bar are always kept. The previous unconditional 10σ rule deleted ~36 real M5 bars/yr in the backtester and a *different* ~24/yr live (window-dependent σ), hiding e.g. the 2026-09-04 12:30 UTC NFP bar from ATR/exhaustion math. Oanda fetches retry 401/5xx/network faults along `RETRY_DELAYS_S` (2/5/10 s) under a 45 s `RETRY_BUDGET_S`; a scan that still loses its primary frame is logged with `scan_kind='failed'`. Registry + `make_data_provider(config, name=None)` in `providers/__init__.py`; config.json `data_providers.active` selects the live candle feed and the optional `data_providers.streaming` scalar selects a different provider for the tick stream. A test enforces that subclasses don't override shared base methods (`supports_streaming`/`iter_ticks` are the sanctioned streaming override points). Tiingo streams mid-price ticks (`wss://api.tiingo.com/fx`) — do not re-add spread display from its free FX feed (two-tier indicative quotes, removed 2026-07-23); the no-spread rule extends to the WS stream.
- **`engine.py`** — `TechnicalEngine`: 9 scored indicators (ema_trend, rsi, macd, bollinger, adx, roc, keltner, divergence, candlestick) + ATR as context. Regime classification (trending_up/down, `trending` = mixed/direction-UNKNOWN, transitioning, ranging) drives per-indicator weight multipliers and regime-aware RSI reinterpretation. Compute order matters (ATR first; MACD always computed even when not a pillar — divergence needs the histogram). Private series ride on the result as `_underscore` attrs; scorers/formatters skip anything `_`-prefixed or weight-0.
- **`signal_aggregator.py`** — MTF agreement, diminishing-decay penalty system, exhaustion hard block, quality grading, then an ordered chain of post-grade gates (session, XAU SELL quality/vol/hour, regime blocks, M5 noise gates), then three setup generators: XAU ATR (A+/A), generic swing-based, plus the M5 pullback (`XAU_PB`) and chart-pattern (`XAU_CP`, `chart_patterns.py`) entry paths. Step 7 lives in `_attach_core_setup`; a final pass after the PB/CP archetypes (`_drop_stale_spread_gate_vetoes`, 2026-09-08) moves a core generator's `SPREAD GATE:` veto to `cleaned_warnings` whenever a later generator shipped a setup (metadata only — books unchanged; the exported `warnings` text column can differ; the prefix is the `_SPREAD_GATE_PREFIX` constant shared by the emitters and both cleanups). The PB/CP generators run AFTER the gate chain and force quality to "B"; the CP path deliberately accepts grade-F composites and re-checks only session/day/big-candle (via warning prefixes) plus its own SELL/vol/RSI gates — it does NOT re-check exhaustion or the M5 min-score/MTF gates. **This is validated behavior, not a bug** (2026-08-04 audit: 226/586 trades in the shipped M5 12mo book are XAU_CP, +38.9R, 104 of them exhaustion-flagged and net-positive) — changing it is a rule-1 revalidation event. `TradeSetup.strategy` selects trailing params; trail params are stamped on the setup from the *producing* config so per-TF overrides reach formatters.
- **`pair_signals/`** — per-pair config factories. **Always use the factory (`produce_signal` / `make_config`), never bare `PlatformConfig()`** — the factory applies the config.json overlay first, then the validated per-TF deltas (M5 hour-9 block, M15 trail arm 0.70, M15 SELL-hour block, M5 SELL-in-transitioning block, H1 all-shorts block (2026-09-03), etc.; evidence in the docstring). A test enforces factory ≡ inline-config equivalence. Confirmation map: M5→[M15,H1], M15→[M5,H1], H1→[M15,H4], H4→[H1,D1].
- **`signal_log.py` / `advisory_outcome.py`** — SQLite signal-history logger (stdlib only). Hooked into `SignalAggregator.analyze_pair` only (the live entry point — the backtester never writes here, preserving parity), so every live scan from every surface (Telegram `telegram`, CLI `cli`, dashboard `dashboard`) lands one row in `logs/signals.db`: signal/grade/price, MTF + trade-setup + reasons JSON, full per-TF indicator snapshots, the last 14 closed OHLCV candles per fetched TF, and the news at scan time (event risk + calendar events + scored headlines, re-fetched via the NewsEngine class-level caches). Config.json `signal_log` block (`enabled`, `path`, `log_blocked_scans`; env `SIGNAL_LOG_ENABLED`/`SIGNAL_LOG_PATH`/`SIGNAL_LOG_BLOCKED_SCANS`). **Fail-safe by contract** — a logging failure can never break a scan. Panel fields arrive post-hoc: `log_signal` stashes the rowid and `agent_panel.attach_panel` writes verdict + creation/cache provenance + payload hash onto the same row. **Advisory outcomes (2026-09):** EVERY row with a trade setup is seeded at log time (not only panel rows); same-setup re-logs within 60 min from any surface (`setup_key` = tf|direction|entry|stop|strategy) are `DUPLICATE` rows pointing at the canonical row via `advisory_event_id`, so sum `advisory_r_multiple` over `RESOLVED` rows only. The versioned model advances from later stored closed-candle snapshots (next-open reanchoring, intrabar peak with close-only ratchet, SL-first, scheduled weekend/21:00-UTC closures allowed, 300-bar timeout, $0.30/$0.65 views); when a snapshot no longer overlaps the outcome's last bar it rebuilds the stream from every stored snapshot of that TF before giving up. It is a cleaned-mid evidence model, not execution P&L. **Post-gate truth columns:** `gate_reason` (the warning that removed the setup), `actionable` (grade ≥ C AND setup — the `action` text is written BEFORE the gate chain and can say "STRONG SELL — High-quality setup" on a D row), `cleaned_warnings_json` (warnings a shipped setup superseded — the XAU_CP acceptance's stripped vetoes and, since 2026-09-08, a stale core spread-gate veto; non-empty is NOT a CP marker, use `setup_json.strategy`), `bar_range_median` (= the misnamed `spread` column: median High-Low, NOT a broker spread), `scan_kind` (`blocked` weekend/day early returns carry no news/candles; `failed` = the primary fetch failed — price 0, no signal — readers skip it like `blocked`), `config_fingerprint` (gate config hash — surfaces config drift between processes). **News is content-addressed:** `headlines_json`/`events_json` are interned into `news_snapshots` (they were 73% of a 667 MB DB); read through the `signals_with_news` view or `signal_log.read_news`. Readers must open with `mode=ro`, never `immutable=1` (blind to WAL rows). **Weight-research columns (2026-09):** `bar_ts` (the primary bar LABEL the scan closed on — same string as `indicators_json`, indexed with `timeframe`), `engine_fingerprint` (hash of the SCORING config: indicator params, thresholds, the regime weight table in force, TFs — `SignalAggregator.engine_config_fingerprint`, the gate fingerprint's twin), `code_version` (short git SHA), `penalty_tags_json`/`penalty_total`/`momentum_fading`/`exhaustion_count`, and the post-hoc `watch_level`/`watch_kinds`/`watch_line` written by `signal_log.update_watch` from `violent_watch.attach_watch` (level and kinds are stored even when the display line is off). Every `indicators_json` entry also carries `eff_weight` (base weight × regime multiplier = the weight the composite actually used; `engine.regime_weight_table` is the single source of truth) and `wscore` (its contribution) — null for skipped indicators (weight 0, `_`-prefixed); **divergence and candlestick entries exist only when they fire, so absence is "no vote", not a zero vote**. **Table `scan_outcomes`:** model-free forward moves per scan (`fwd_1h/4h/24h` in $ and ATRs, `mfe_24h_atr`/`mae_24h_atr`, `burst_1h` = a ≥3×ATR bar within the hour, `bars_used`, `complete`), measured from the row's own last closed bar on the stored candle stream; a horizon stays NULL until it has elapsed, and `complete` needs the full 24 h with no gap over 3 bars beyond the scheduled weekend/17:00-NY closures (`advisory_outcome._has_continuity`). `SignalLogger.label_scan_outcomes` runs opportunistically at the end of `log()` (at most once per 60 s per process, 150 rows and ~2 s per pass, idempotent — a row is rewritten only when its numbers change), so no daemon is needed; `scripts/label_scan_outcomes.py` is the backlog driver (`--dry-run` counts read-only). Readers must dedup by the FIRST row per (`timeframe`, `bar_ts`) — several surfaces log the same bar. Tools: `analyze_signal_log.py` (`--stored-candles` DB-only replay with event/opportunity/one-position books + persona scorecard, `--news` news-sentiment lead/lag + extremes + outcome buckets, `--parity` telegram-vs-dashboard divergence report, `--scan-outcomes` forward-move + hit-rate table by TF×signal/grade with a NEUTRAL baseline and burst rate by watch level), `scripts/backfill_signal_outcomes.py` (seed/dedup/advance legacy rows), `scripts/label_scan_outcomes.py` (label/relabel scan outcomes), `scripts/migrate_signal_log_news.py` (intern legacy blobs; `--vacuum` with both writers stopped).
- **`news_engine.py`** — Forex Factory calendar (MyFXBook RSS fallback) + RSS keyword sentiment with gold-inverse keywords. Class-level caches (fresh engine per scan). Optional DeepSeek enrichment via `llm_client.py` (requests-only, no SDK): batched headline sentiment overriding keyword scores, gold-direction on release surprises, and a market-summary blurb (`SignalReport.news_summary`) — **display/warning-only, keyword fallback on any failure, inert without a key**; prompts are static module constants (DeepSeek prefix-cache pricing) and requests send `thinking: disabled` (V4-Flash reasoning otherwise blows the timeout). By XAU-tuned default, event risk and sentiment **warn but do not gate signals** (`event_risk_caps_signals=False`, `sentiment_affects_signals=False`) — the LLM does not change that. **Headline pool (2026-09-04):** only headlines ≤ 4 h old with a parsable publish time (`headline_max_age_seconds=14400`, undated items dropped), and `get_fundamental_context` drops headlines whose every relevant score is inside ±0.1 (`_is_neutral_headline`, `NEUTRAL_HEADLINE_THRESHOLD`) before the per-currency mean — so `news_sentiment` from that date is a mean over directional headlines only (bolder values; log comparisons must split at the deploy date). The market summary prompt writes for someone who has never traded (no acronyms), sends headlines newest-first with the platform's own gold direction, and caps at 400 chars. **Verdict 2026-09-04** (`backtest/research/news_sentiment_value_2026-09-04.md`): `news_sentiment` is a trailing-24h echo of price (r≈0.67 with the past day, ≈0.03 with the next; flips 44 h after tops / 16 h after bottoms) — both gating flags stay False; re-check with `analyze_signal_log.py --stored-candles --news`.
- **`news_archive.py`** — durable headline corpus in its OWN database (`logs/newsdata.db`), separate from `signals.db` so a signal-log rotation cannot take the news with it (that is exactly how the 3-week corpus behind `news_sentiment_value_2026-09-04.md` was lost — 4,054 distinct headlines, unrecoverable, `logs/` is gitignored). One row per distinct headline keyed on `sha256(headline)`, carrying the text, description, link, publisher, publish time, the score, `score_source` (`keyword`/`llm`/`model` — a new `NewsSentiment` field, also mirrored into `signal_log._headline_to_dict`), and `teacher_model`/`teacher_prompt_sha1` so labels made under a changed prompt are never pooled with older ones. **Writes are one atomic UPSERT per row and upgrade-only**: an `llm` score always replaces a `keyword` one (the >25/cycle batch-overflow tail gets its DeepSeek label on the next fetch window in a long-running process — verified: 13/13 upgraded) but a `keyword` score can never overwrite an `llm` one; re-seeing a headline always bumps `seen_count`. A read-then-write pair loses increments when the bot, the dashboard publisher and the tests first see a headline together, and the WAL switch itself needs `_with_lock_retry` — both are regression-tested with 3 concurrent writers. Hooked at the END of `news_engine._fetch_news_sentiment_raw`, **before** `_is_neutral_headline` runs in `get_fundamental_context`: archiving downstream of that ±0.1 filter is why `signals.db` holds almost no neutral headlines and would teach a trained scorer that "neutral" barely exists. Never raises (fail-safe by contract, like `signal_log`). The backtester never builds a `NewsEngine`, so replays never archive (test-locked). Seeded by `scripts/backfill_news_archive.py`, which reads both DB shapes read-only and **recomputes** provenance (stored score vs the keyword scorer: differs → `llm`, exact non-zero match → `keyword`, both zero → `unknown`) because neither source recorded it. Purpose: training data for a local numpy-only headline scorer to replace the 24/78 keyword fallback — see the note below.
- **`agent_panel.py`** — ai-hedge-fund-style advisory panel: 4 DeepSeek persona calls (macro/technical/risk/contrarian, concurrent on a panel-private pool) + 1 synthesizer producing an **ADVISORY-ONLY** consensus (`AGREE/NEUTRAL/OPPOSE` + confidence + rationale) written onto `SignalReport.panel_*` fields. Attached **post-hoc** by consumers (`telegram_bot` alert path, `main --panel`, `dashboard_publisher`) — never runs inside `analyze_from_dataframes` (backtest parity, test-locked) and no gate reads its fields. Persona payloads are role-filtered: macro gets high-impact events plus only code-scored directional news; technical gets strategy/pattern, primary indicators, MTF, levels, and ATR-normalized candles; risk gets setup strategy, strategy cost assumption, trailing semantics, volatility/events/warnings (never the median candle range as spread); contrarian gets directional news, setup, timestamped latest-closed price, indicators, reasons, and warnings. Spend is bounded by exact cache, full-evidence-identity TTL reuse, and `max_panels_per_day`; changed evidence never inherits a direction-only cached verdict (test-locked — which also means TTL reuse effectively never fires live because price/candles change every bar; the daily cap is the real spend bound). Verdict provenance logs creation time, `MISS`/`EXACT_HIT`/`TTL_REUSE`, and a SHA-256 payload hash. Prompts forbid outside knowledge and outputs remain short. **OPPOSE guard (2026-09):** the 3-week signal-log scorecard showed the contrarian/risk/technical personas oppose almost every trend entry (chair said OPPOSE 73% of the time; opposed trades made +30R, PF 2.6) while the macro persona was the only informative voice (macro=support → 88% win). `_apply_oppose_guard` therefore downgrades a chair OPPOSE to NEUTRAL unless macro opposes, or technical opposes while macro is not supportive; the synthesizer prompt states the same rule. Re-score with `analyze_signal_log.py --stored-candles` (persona scorecard) before changing it again. **Verdict source (2026-09-03, owner decision):** all four personas still run and are logged (`panel_takes_json`), but `agent_panel.verdict_source` defaults to `"macro"` — the alert verdict is the macro persona's stance (support/neutral/oppose → AGREE/NEUTRAL/OPPOSE) and the synthesizer call is skipped (4 LLM calls per panel instead of 5); `"chair"` restores the four-voice synthesizer + OPPOSE guard. Re-score personas monthly with `analyze_signal_log.py --stored-candles`; weights are not to be tuned before ~300 graded outcomes with a held-out half.
- **`market_outlook.py`** — hourly DeepSeek market brief for XAU/USD, using ATR-normalized M15/H1/H4 candles, visible indicator states, levels, news, and recent signal history (read `mode=ro`). It returns a written **ADVISORY-ONLY** bias/setup-forming assessment and persists it independently in `outlooks` within `logs/signals.db` together with the news, indicator/regime/level, and candle evidence sent to the model. It may be sent post-scan by Telegram or requested with `main.py --outlook`; it never imports or runs inside the signal path. Hourly dedup is by UTC hour BUCKET (the old `now − last < 3600` rule dropped every other hour because the H1 loop fires at boundary+10 s+scan time) plus its own UTC-day cap (default 12); every silent early return logs `market outlook: skipped (...)` and an empty LLM reply logs a warning — a dark feature must be visible in `logs/telegram.log`. Provider/news failures leave a safe partial brief while LLM/database failures degrade to no outlook.
- **`violent_watch.py`** — **display-only** violent-move watch, dark by default (`violent_watch` config block: `enabled`, `alert_line`, `headsup`, `max_headsup_per_day`, `headsup_min_level`; env `VIOLENT_WATCH_ENABLED`). Flags when a "violent bar" (range ≥ 3× prior ATR14) is unusually likely and stamps `SignalReport.watch_line` **post-hoc** via `attach_watch` (telegram alert path, `main.py`, `dashboard_publisher`) — same contract as the agent panel: never imported by `signal_aggregator`/`engine`/`backtest/`, never raises, never read by a gate (test-locked, Module 30). Triggers: a high-impact **USD** calendar release within ±30 min (hot), hot clock slots (weekday 12–14 UTC, weekday 01 UTC, Sunday reopen 22–23 UTC), and **price-extreme** indicator readings (outside the Bollinger bands, RSI <30 / ≥70, outside the Keltner channel). Level = hot when the calendar fires or a clock slot plus ≥1 price extreme; watch on a clock slot alone or ≥2 price extremes. Context readings (ADX slope >3, volatility "extreme", regime "transitioning", MACD STRONG_BUY/STRONG_SELL) are printed but **never raise the level** — the first draft that let them count flagged 27% of bars "hot" at only ~2× odds; the shipped rule flags 8.6% at 3–5× (evidence: `backtest/research/sizing_campaign_2026-09.md` addenda). Direction is called DOWN on bearish readings (70–78% on M5 and M15) and UP only on M5. Telegram also sends a standalone heads-up on the fastest scanned TF (`_HEADSUP_TF`), rate-limited per UTC hour+trigger-set and per day; it explicitly tells the owner **not** to touch open trades — every protection tested lost money, the stop is the protection.
- **`telegram_bot.py`** — owner-locked bot + subscriber channel; candle-boundary auto-scan loops (M5/M15/H1/H4, boundary+10 s), alert quality threshold + fingerprint dedup (committed only after a successful send; the fingerprint is signal+direction — grade is NOT part of it, a grade UPGRADE re-alerts, an A↔B flicker does not), 📰 LLM market-context line on alerts when available, WebSocket level-breach monitor fed by the streaming provider (`data_providers.streaming`, falling back to the active provider; auto-disabled when the resolved provider can't stream or has no API key — Oanda does not stream, Tiingo does). The breach handler reuses this bar's M15 report (`_cached_or_scan`, keyed by closed-bar epoch) instead of rescanning — it used to log 3.5 M15 rows per candle. **Alerts require a trade setup (2026-09-08):** `_is_alert_candidate` (shared by the scan and breach paths) needs non-neutral + grade ≥ threshold + `trade_setup`; a signal every generator declined used to go out anyway (a B-grade no-setup SELL under a "costs would eat the edge" warning). **Sends are logged:** every delivered alert / breach alert / violent-move heads-up writes an `… sent:` INFO line carrying the signal-log row id, and a no-setup candidate writes `alert skipped: … (gate reason)` — before this only failures were logged and a review had to reconstruct sends from panel rows. Rotating log: `logs/telegram.log`, one file per server-local day × 30 backups (was 3-h × 240 until 2026-09-08; the old `_HH`-suffixed files on the live box are not matched by the daily cleanup and need one manual delete).
- **`dashboard_publisher.py` / `static/dashboard.html`** — web dashboard as a **static publisher** (live LAMP server): standalone daemon scans at candle closes via `produce_signal` and atomically writes `snapshot.json`/`news.json`/`dashboard.html` to `$DASHBOARD_OUT_DIR` (Apache serves them; page is public by owner decision). **Hard independence rule: never imports, and is never imported by, `telegram_bot`** — confirmation map and boundary math are deliberate local copies, so a web-side failure can only mean a stale page. Due-TF classification uses the RAW wake time (flooring wake−buffer skips every H1/H4 scan — 2026-08-02 bug). News panel reads `NewsEngine.get_fundamental_context()` plus the additive `get_week_events()` accessor (display-only; no signal path calls it); as a separate process it has its own news caches and DeepSeek daily cap. News/calendar rows are clickable (detail dialog); headlines carry `id` = sha1(headline) in news.json, and exactly one on-demand endpoint exists: `static/news_summary.php` (installed by `install_static`), which serves a DeepSeek gold-impact summary per headline — whitelist-gated against news.json ids, cache-forever in `ai/`, own fail-closed daily cap (150/day, file-based, separate from the publisher's), key at `/etc/gold-dashboard/deepseek.key`, never reads repo config. Deployment checklist: `deploy/README-dashboard.md`.
- **`backtest/gpu_backtest.py`** — bar-by-bar replay through the real `analyze_from_dataframes` (news off). Realism model (do not regress): next-bar-open entries re-anchored to the raw price path, round-trip `--spread` charged per trade, trade resolution on RAW bars (cleaning applies only to signal-side data), same-bar SL+TP tie → SL first, gap fills at bar open, trail ratchet at bar close only. Indicators are precomputed once per TF (CuPy GPU or NumPy fallback — check the `Backend:` log line); replay fans out over worker processes and the 5-bar cooldown is applied in an ordered pass afterwards, so parallel and sequential books are identical. Parquet cache in `backtest/cache/` keyed by provider/symbol/TF/window. `backtest/*.csv` are gitignored pinned trade books from past campaigns — used for manual before/after comparison. `backtest/instructions_for_backtest.txt` lists what every backtest report must record.

## Rules for making changes (read before editing signal logic)

1. **Never change a tuned value without re-validation.** Acceptance rule: positive total R in BOTH train and validation halves of a ≥12-month window, PF ≥ 1.3, at realistic cost (XAU $0.65 round trip headline; retain $0.30 for TD-era cross-book comparability), trailing exits, next-open fills. `round_trip_cost` stays $0.30 because it also drives the live TP spread gate; changing it is a separate rule-1 event. Marginal edges must additionally pass the 2024-06→2025-03 out-of-sample window — within-year splits alone have approved configs that later failed OOS. **Owner preference (2026-09-05): backtest windows go back at most 3 years** (the 5-year books are historical reference only); the 3-year baseline books (2023-09-05→2026-09-05, $0.65: M5 857/+161.3R/PF 1.53, M15 1086/+148.7R/PF 1.37, H1 255/+70.5R/PF 1.98, H4 113/+22.7R/PF 1.63) live in the gitignored `backtest/research/sizing_2026-09/` and are regenerated with the commands in `sizing_campaign_2026-09.md`.
2. **Per-timeframe overrides live in the pair factory**, never in global defaults — the same knob has opposite effects on different TFs (proven repeatedly: `tp_floor_rr`, hour-9 block, trail params). Document validation evidence in the factory docstring.
3. **Pair isolation is a hard invariant.** XAU gates in the aggregator are display-name-gated so a future pair port cannot inherit gold-calibrated logic by accident.
4. **Live == backtest parity.** If you add a live gate, add it to the replay path in the same commit; all time filters evaluate bar-close time in both paths.
5. **ROC stays weighted** even though it's correlated with MACD/Keltner/EMA — zeroing it backtested 13R worse; composite thresholds are calibrated with it.
6. **Backtester realism knobs are results-critical** (raw-path resolution, spread charging, next-open entry, SL-first ties, close-only trail). Touching them invalidates comparability with all pinned books.
7. **The forming candle must stay dropped** — live signals repaint otherwise.
8. Don't add config fields nothing reads; don't disable indicators in production paths (`disabled_indicators` is for experiments only).
9. EUR/USD was removed 2026-07-16 after a no-edge verdict (cost-to-volatility; last commit with EUR support: `7d3fd95`). Do not re-add without new data. BTC/ETH work lives on unmerged branches only.

## 2026-08-04 QA audit — owner decisions ledger

- **Weekend calendar: FIXED (owner-approved 2026-08-04).** All three signal-path `_is_forex_weekend` copies are NY-anchored (Fri 17:00 → Sun 18:00 ET; during US DST that is Fri 21:00 UTC — measured on 5yr of Oanda bars). Kills the weekly false-"STALE DATA/feed lagging" hours. Book impact at the audit HEAD: M15 byte-identical; M5 12mo dropped exactly ONE trade — a phantom whose signal fired on the final summer-Friday bar (close 21:00 UTC) and "filled" at Sunday's reopen 49 h later (−1.08R, live-impossible), leaving 555 trades / +98.64R / PF≈1.50 @ $0.65 (was 556/+97.56/1.48). The current-HEAD canonical reference is recorded below. A Module 21 test sweeps those copies plus the advisory outcome's continuity copy hourly across both DST transitions.
- **Spread in exit-trigger detection (rule 6): owner DECLINED 2026-08-04.** Backtest stop triggers stay on raw mid High/Low; the measured optimism (~4–6% of total R: −3.55R @ $0.30, −4.24R @ $0.65 on the M15 12mo book) is accepted and books stay comparable. Re-open only as a conscious re-pin campaign.
- **`FB2_EMA_LO`: HARDENED 2026-08-04 (delegated).** Research override for the EMA200-proximity band, default = tuned 0.5; an unparseable value now degrades to the default with a warning instead of crashing every scan. Documented here — it is a knob that exists.
- **H4 seeding divergence: parked.** Live H4 fetches only 380 bars vs the backtest's unbounded history — ~2% of sampled H4 bars flip an indicator signal (composite Δ up to 12.3). M5/M15/H1 clean at their real fetch depths. Fold into any future H4 campaign; H4 remains "unproven".

## Current state notes (dated per entry)

- Oanda v20 is the live feed (`data_providers.active: "oanda"`). The 2026-08-04 re-pin made the 27 `backtest/oanda_*.csv` books the baseline; all TD-era books are superseded. (2026-08-14) The re-pin M5 book `oanda_m5_12mo_065_nopb_20260814.csv` (+106.82R PF 1.51) is itself SUPERSEDED by the 2026-09-03 book in the entry below; the M15 reference is unchanged since 2026-08-04: `oanda_m15_12mo_065.csv` (397 / +73.93R / PF 1.53). See `backtest/research/repin_verification_2026-08-14.md`.
- Cost reporting is dual: $0.65 is the measured-Oanda realistic headline ($0.65-$0.72 round trip), while $0.30 remains for cross-book comparability and as the live TP spread-gate input. Raising `round_trip_cost` from $0.30 would move the TP floor from about $9 to $19.50 and requires full rule-1 revalidation. The $0.65 figure comes from the 2026-07-24 Dukascopy tick study (9M ticks over 12 months; its tooling was removed 2026-09-06 — the conclusion lives here and in memory).
- Twelve Data is signal-incompatible with Oanda (Phase 1: about 19% M5 bar-direction disagreement and a systematic ~$1 high bias; see `backtest/research/feed_divergence_2026-08-14.md`). On an Oanda candle outage, stop scanning rather than switching to TD candles.
- Tiingo is streaming-only (`data_providers.streaming: "tiingo"`), not a designated candle fallback. Oanda-Tiingo signal parity is documented in `backtest/research/signal_parity_2026-08-14.md`; Phase 4's two-week live shadow scan (`scripts/shadow_scan.py`) remains a manual operational step.
- Human-facing docs in `docs/` (`user_guide.html`, `beginner_guide.html` + PDF builds) must be kept in sync when signal behavior changes. As of 2026-09-06 the PDF builds are BEHIND the HTML (violent-move watch entry added without a PDF rebuild).
- 2026-09-03 signal-log campaign shipped two new per-TF gates (`backtest/research/signal_log_campaign_2026-09.md`): M5 `sell_blocked_regimes=["transitioning"]` and H1 `sell_min_quality=[]` (all H1 shorts blocked). Current-HEAD M5 canonical book is now `backtest/oanda_m5_12mo_065_nopb_20260903.csv` (557 / +112.21R / PF 1.56, halves +55.9/+56.4); H1 reference `backtest/oanda_h1_12mo_065_20260903.csv` (91 / +22.2R / PF 1.79; 5yr +49.1R PF 1.46; OOS +16.7R PF 1.74).
- 2026-09-06 stake-sizing campaign (`backtest/research/sizing_campaign_2026-09.md`): a walk-forward "second opinion" stake multiplier (meta-labelling on 65 decision-time features, 3-year Oanda books at $0.65) FAILED the pre-registered rule on M5 (clean null) and M15 (lift inside the label-shuffle null, explained by BUY-vs-SELL). Nothing that changes numbers shipped. Behaviour-neutral tooling remains: `position_sizing.build_features` + `--export-features` (default export stays 63 columns), `SignalReport.penalty_tags/penalty_total/momentum_fading`, `TradeSetup.stake_multiplier` (None; reserved) (the campaign's research scripts were removed 2026-09-06; git history has them). Stage-B facts: M5 stretch measures are POSITIVE in both halves (do not re-propose exhaustion sizing on M5); M15 exhaustion tag is weakly negative (≈ +3R/yr, noise); bursts cluster at 12–15 UTC and 01 UTC bar-close and do NOT hurt entries.
- Auto-execution (oanda_daemon.py / oanda_executor.py) was removed 2026-09-03 — the owner is not auto-trading; last commit containing it is 31002cb. Book-parity rules that lived in its docstring (no broker TP, 5-bar cooldown, once-per-bar trail) remain the backtester's model.
- 2026-09-07 signal-log inspection (fresh 2-day `logs/signals.db` import: NFP Friday + Monday; 5 setups, all BUY, 4 stopped out at local tops — two days is not evidence). Three sample-size-independent fixes shipped: (1) the signal-side 10σ outlier filter now drops only bars whose NEXT close snaps back ≥80% (it had been deleting ~36 real M5 news bars/yr in the backtester and a different ~24/yr live, e.g. the 2026-09-04 12:30 UTC NFP bar; 3-yr drops are now M5 2 / M15 1 / H1 0). Book impact at $0.65, 12mo: M5 557/+112.21R/PF 1.56 → 560/+108.81R/PF 1.54 (halves +55.6/+53.2), M15 397/+73.93/1.53 → 398/+74.61/1.53 (max DD 10.8→9.1R), H1 byte-identical; 3-yr (2023-09-05→2026-09-05): M5 857/+161.3/1.53 → 870/+153.6/1.50, M15 1086/+148.7/1.37 → 1092/+151.9/1.38, H1 255/+70.5/1.98 → 253/+70.0/1.98, H4 113/+22.7/1.63 → 114/+21.7/1.59 — only trades on the flagged event days move (M5 gives back a few Jan–Mar 2026 crash-period winners); the pinned M5/M15 CSVs are therefore no longer byte-reproducible (regenerate with the parity recipe when re-pinning). (2) Oanda candle fetches retry along a 2/5/10 s ladder under a 45 s budget (11 boundary scans in two days had lost their primary frame). (3) A scan that still loses its primary frame logs `scan_kind='failed'` and readers skip it. The live dashboard publisher was found dead (0 rows Friday, stopped Mon 02:20 UTC): install `deploy/gold-dashboard.service` on the live server.
- 2026-09-08 delivery-side fixes from the 3-day signal-log + telegram-log review (nothing in the signal core changed; 468 tests): (1) `_drop_stale_spread_gate_vetoes` (after every setup archetype, so the XAU_PB path is covered too) moves a core generator's stale `SPREAD GATE:` veto to `cleaned_warnings` when a later generator ships a setup — the 3-yr H1 and M15 books were regenerated before/after and are identical on every trade column (H1 253/+70.02R/PF 1.98, M15 1092/+151.90R/PF 1.38); only the exported `warnings` text differs, on 50/253 H1 and 358/1092 M15 trades, i.e. a fifth to a third of all A-grade alerts had been carrying "costs would eat the edge" beside a valid target; (2) Telegram alerts now require a trade setup (`_is_alert_candidate`, scan + breach paths; no-setup candidates log `alert skipped: … (gate reason)`); (3) every delivered alert / breach alert / violent-move heads-up logs an `… sent:` INFO line with the signal-log row id (the review had to reconstruct sends from panel rows); (4) `telegram.log` rotates daily × 30 instead of 3-hourly × 240 (old `_HH` files on the live box need one manual delete). Review facts (not evidence — three days): 8 setups, 1 right; strong readings (|score| ≥ 20) were followed by reversals in 16 of 18 episodes; hot watch level 0/25 bursts on M5 (calendar trigger never fired, 15 of 33 hot stamps were the 12–14 UTC slot on Labor Day — no holiday awareness anywhere); Oanda published no XAU bars 18:30–22:00 UTC on Labor Day and the bot scanned the stale bar 43×; the retry ladder recovered all 25 post-deploy 401 bursts (41 of 58 incidents sit on the :00 boundary); the live dashboard publisher ran only 01:00–02:20 UTC Sep 7 and is dead again.
- 2026-09-06 stale-code cleanup (behaviour-neutral; suite green, M5/H1 12-month books byte-identical after every signal-side commit): `main.py --timeframe` now accepts M5/M15/H1/H4 and takes its confirmation TFs from the pair factory (the CLI's private table had paired M5 with M1+M15); `PlatformConfig.pip_value` is gone (`PAIR_INFO` is the source); removed as dead or superseded: `_get_affected_pairs`/`event_impact_on_pairs`, `SignalLogger.advance_with_bars`/`path`/`news_for`, `reset_llm_client`, the Telegram scan-summary/keyboard helpers and duplicate confirmation map, the 2026-07 Oanda-vs-TD research pager+study, `burst_profile.py`, `hour_analysis.py`, `scripts/compare_oanda_tiingo.py`, the never-implemented LLM-provider plan and the merged config.json spec. Second pass the same day (owner directive: keep only what the system needs): every research script and data file under `backtest/research/` (book_report, book_diff, the burst tools, sizing_model, commit_ladder, llm_news_eval + golden CSV, ff_calendar_scraper, calendar_impact, watch_rule_eval, the Dukascopy spread study + CSVs, the panel dry-test JSONL), the two completed study scripts in `scripts/`, the architecture diagram + generator and the data-source test plan were deleted; the campaign `.md` reports stay as the decision ledger. Git history before `953c5ad` has all of it. Third pass (owner: "remove dead functions as well"): the write-only pb/cp rejection counters, the unreachable `_TP_QUALITY_SCALE` B/C entries, the engine's unused DataFrame argument and M1 arm, the providers' generic-symbol guesses (unknown symbols now raise), the sub-$10 price format, the `=X` pair parsing and the unread `minutes_until`/`is_high_impact` news.json keys are gone.
- 2026-09-08 news archive shipped (`news_archive.py`, 483 tests; nothing in the signal core changed, both news gating flags stay `False`, books untouched). Motivation: headlines only ever survived as a side effect of signal logging and `signals.db` was rotated around 2026-09-04, destroying the 3-week / 4,054-headline corpus the news study was written on — with `logs/` gitignored, git could not recover it and nobody noticed for four days. Day-one corpus after backfilling both DBs on disk: **1,016 distinct headlines, 846 of them DeepSeek-labelled**, growing ~200/day (~1,400/week). The point of collecting it: distil DeepSeek's headline scoring (78/78 on the 2026-08-06 golden set) into a numpy-only local model so the fallback stops being the keyword scorer (**24/78 = 31%**, which is what silently runs on no key / disabled / daily cap / timeout / HTTP or parse failure / batch overflow). **Do not train yet** — 800-odd labels gives a holdout too small to grade (SE ±0.032); the pre-registered plan is ~4,000 deduplicated headlines over ≥3 weeks, time-split with near-duplicates clustered first (the same story is re-titled by several outlets, so a random split scores memorisation), headline text ONLY (the teacher never sees `description` — `_llm_headline_scores` sends titles), graded against the golden set recoverable with `git show 1cebebf^:backtest/research/llm_news_golden.csv`. When it ships it slots in as DeepSeek → model → keyword and stays display-only. Training happens offline in `venv/`; a Module 32 test now AST-walks the root, `providers/` and `pair_signals/` asserting no `sklearn`/`scipy`/`lightgbm`/`torch` import and that none of them reach `requirements.txt` — that guard did not exist before.
- 2026-09-09 replay recording shipped (`backtest/replay_record.py`, `backtest/replay_scorecard.py`, Module 33, 498 tests; nothing in the signal core changed, books byte-identical — `backtest/research/replay_recording_2026-09-09.md`). `--record DIR` stores every ACCEPTED trade's decision evidence in Parquet: each indicator's vote / value / confidence / BASE weight / regime multiplier / EFFECTIVE weight / contribution on the primary AND both confirmation TFs (`indicators.parquet`, long; absence = the indicator did not fire, never a zero row), per-TF regime/composite, penalty tags, setup, outcome, and the Forex Factory calendar events from 4 h before entry to exit (`events.parquet`, all-day rows matched on the New-York date; scalar `ev_*`/`us_holiday` columns on `trades.parquet`); `--record-bars` adds one float32 row per ANALYZED bar with the same votes plus 1 h/4 h/24 h forward moves in ATRs (`bars.parquet`). The manifest carries the engine + gate fingerprints, the regime weight table in force and the book MD5. Calendar source is the gitignored `backtest/cache/calendar.parquet` (15,124 FF events 2023-09-01→2026-09-10, unix-UTC `dateline`; `timeLabel` is display-only); the scraper that built it is back at `scripts/ff_calendar_scraper.py` (resume + idempotent merge, ~1.6 s/request). The 2-year books (2024-09-05→2026-09-05, $0.65) live in `backtest/research/replay_2026-09/{m5,m15}/`: M5 781 / +146.07R / PF 1.52 (halves +28.0/+118.1), M15 793 / +117.62R / PF 1.41 (halves +38.2/+79.4); control-vs-recorded MD5 equal on both. First-look facts are descriptive only (Bollinger opposes 94% of M5 entries; divergence-opposed M5 trades did best again; every indicator's SELL votes were followed by moves against them on both TFs; the bar-level baseline drifts +1.9 ATR/24 h on M5 so BUY votes must be drift-adjusted first) — every candidate change stays a rule-1 rerun with a control book.