From 552199857a713d1ff437cac9ee9e5eec52f43163 Mon Sep 17 00:00:00 2001 From: Harwinder Sekhon Date: Tue, 15 Sep 2026 20:40:05 -0400 Subject: [PATCH] docs: rewrite CLAUDE.md as a plain-language developer guide Same facts, new shape: quick start; the scan pipeline from trigger to delivery to logging (A to Z) with the real thresholds and gate order; a file-by-file guide with the import graph; data providers; config; testing; the nine change rules; the validated per-timeframe overrides; reference books with MD5s and the kept cache windows; a traps list; and the decision ledger condensed to one row per dated decision with a pointer to its research report. Every claim was checked against the code by a reviewer pass; the mechanism invariants the old file carried (MACD always computed, the _-prefixed convention, PB/CP force grade B, penalties hit confidence, gate 6b label-hour vs close-hour, the news-archive hook position, the ordered cooldown pass, engine_fingerprint split rule, oppose guard, holiday blindness) are kept. The previous 58 KB version is in git history at b0b71e1. Co-Authored-By: Claude Fable 5.1 --- CLAUDE.md | 606 +++++++++++++++++++++++++++++++++++++++++++++--------- 1 file changed, 512 insertions(+), 94 deletions(-) diff --git a/CLAUDE.md b/CLAUDE.md index e082b68..c9abca2 100644 --- a/CLAUDE.md +++ b/CLAUDE.md @@ -1,118 +1,536 @@ -# CLAUDE.md +# ForexBot2 — developer guide -This file provides guidance to Claude Code (claude.ai/code) when working with code in this repository. +ForexBot2 is a signal-intelligence system for one market: **gold against the US dollar (XAU/USD)**. +It does not place trades. Every few minutes it downloads price candles, scores them with nine +technical indicators, checks that several timeframes agree, grades the result (A+ down to F), and +turns a good grade into a written trade plan (entry, stop loss, two targets, trailing rule). The +plan is delivered to a Telegram bot, a command-line report, and a public web dashboard. A +backtester replays history through the *same* signal code, so a backtest result is what the live +scanner would have said on that day. -## What this is +This file is the map of the system. Read section 2 first if you are new. Section 7 lists the rules +you must follow before changing anything that affects signals. Section 10 lists the traps. Section +11 is the history of every validated change. The research reports behind those decisions live in `backtest/research/*.md`. -Signal-intelligence platform for **XAU/USD (gold — the only supported pair)**. It fetches OHLCV from the active data provider (**Oanda v20 since 2026-07-28**; Twelve Data and Tiingo remain registered fallbacks), scores 9 technical indicators per timeframe with regime-adaptive weights, checks multi-timeframe agreement and news risk, grades the signal (A+…F), and generates a full trade setup (entry, SL, TP1/TP2, trailing plan). Delivery: Telegram bot, CLI, and a static web dashboard. A realism-focused backtester replays history through the *exact same* signal code. +--- -Every numeric threshold in `config.py` and `pair_signals/xauusd.py` was backtested or is load-bearing for a backtested result. Validation evidence lives in the factory/module docstrings and in `~/.claude/projects/.../memory/`. - -## Commands +## 1. Quick start ```bash -python test_all.py # QA suite, no network (mocked); prints its own count - # (506 tests / 33 modules as of 2026-09-15) -python main.py --pair XAUUSD --timeframe H1 --detail # CLI signal report (--json for full dump, - # --panel for the advisory LLM agent panel; the violent-move watch - # line appears when the `violent_watch` config block is enabled; - # --timeframe accepts M5/M15/H1/H4 only and the confirmation - # TFs always come from the pair factory's map) -python telegram_bot.py # the bot (refuses to boot without credentials) -python dashboard_publisher.py --once # web dashboard: one scan+publish cycle to $DASHBOARD_OUT_DIR - # (omit --once for the candle-close daemon; deploy/README-dashboard.md) +python -m venv venv && venv/bin/pip install -r requirements.txt # Python 3.12; a venv/ already exists on the dev box +cp config.json.example config.json # then fill in the keys (see section 5) -# Backtests -python backtest/gpu_backtest.py --pair XAUUSD --days 365 -python backtest/gpu_backtest.py --pair XAUUSD --start 2025-01-01 --end 2025-06-01 \ - --primary-tf 15min --confirm-tfs 1h,4h -# key flags: --spread --quality A+,A,B,C --strategies --workers --max-hold-bars -# --warmup-bars --cooldown-bars --no-cache --export-csv out.csv --detail -# --set PATH=JSON (research override of any PlatformConfig leaf; unknown path = hard exit) -# --export-features (append the 65 decision-time f_* columns; default export stays 63 cols) -# --fixed-rr-tp N (research-only: hard TP at N×risk, trail off; verdict -# 2026-08-05: NOT shipped — backtest/research/fixedtp_2026-08-05.md) -# --record DIR [--record-bars] (research: manifest + trades/indicators/events parquet for -# every accepted trade, and one row per analyzed bar; the book is unchanged — pair every -# recorded run with a control run and show the MD5s equal; read with -# backtest.replay_record.load_run, first look with backtest/replay_scorecard.py DIR --check) +venv/bin/python test_all.py # QA suite, no network; prints its own count + # (506 tests / 33 modules as of 2026-09-15) +venv/bin/python main.py --pair XAUUSD --timeframe H1 --detail # one signal report in the terminal + # --json full dump, --panel adds the advisory LLM panel, + # --timeframe M5 | M15 | H1 | H4 +venv/bin/python telegram_bot.py # the bot (refuses to start without credentials) +venv/bin/python dashboard_publisher.py --once # one scan + publish to $DASHBOARD_OUT_DIR + # (without --once it runs as a candle-close daemon) + +# Backtests (timeframe names are 5min / 15min / 1h / 4h, not M5/M15) +venv/bin/python backtest/gpu_backtest.py --pair XAUUSD --days 365 +venv/bin/python backtest/gpu_backtest.py --pair XAUUSD --start 2024-09-05 --end 2026-09-05 \ + --primary-tf 15min --spread 0.65 --workers 12 --export-csv book.csv ``` -**`test_all.py` is NOT pytest** — it uses a custom `@test("name")` decorator that runs each test immediately at import; don't grep for `def test_` or try `pytest test_all.py`. Run the whole file; it prints per-module results and exits non-zero on failure. There is no way to run a single test short of commenting modules out. Tests must stay green; every new gate/knob needs tests. Signal-log fixtures: use a WEEKDAY signal bar when a test exercises candle continuity — the outcome model treats weekend and 21:00 UTC gaps as scheduled, so Saturday-dated bars never produce a gap. `_candles_json` stores `df.tail(14)`, so a test snapshot longer than 14 bars silently drops its head. When patching weekend logic in tests, patch `providers.base._is_forex_weekend` — but know there are THREE independent signal-path weekend implementations (`providers/base.py`, `signal_aggregator.py`, and an inline copy in `backtest/gpu_backtest.py:_is_xau_blocked_by_calendar`) plus the advisory-only continuity copy in `advisory_outcome.py`; patching one does not reach the others. +Backtest flags worth knowing: `--spread` (round-trip cost in dollars; the default is 0.30, the +realistic Oanda figure is 0.65 and must be passed explicitly), `--quality A+,A,B,C`, +`--strategies`, `--workers`, `--max-hold-bars`, `--warmup-bars`, `--cooldown-bars`, `--no-cache`, +`--export-csv out.csv`, `--detail`, `--set PATH=JSON` (research override of any config value; an +unknown path is a hard exit), `--export-features` (appends the 65 decision-time columns; never use +it on a parity run because it changes the file), `--fixed-rr-tp N` (research only, not shipped), +`--record DIR [--record-bars]` (research recorder, see section 9). -Python 3.12; a `venv/` exists in-repo. Dependencies in `requirements.txt` (numpy, pandas, requests, twelvedata, python-telegram-bot ≥20, websockets). scikit-learn and lightgbm are installed in the dev `venv/` for ad-hoc research only — never add them to `requirements.txt` or import them from live code; anything shipped must be numpy-only. +`test_all.py` is **not pytest**. It uses its own `@test("name")` decorator that runs each test the +moment the file is imported. Run the whole file; it prints per-module results and exits non-zero +on any failure. There is no way to run one test on its own. Tests must stay green, and every new +gate or setting needs a test. -## Configuration & credentials +--- -Precedence for every credential/deployment setting: **env var → config.json → refuse to run** (no hardcoded fallbacks). A gitignored repo-root `.env` is loaded at `app_config` import via `os.environ.setdefault`. `config.json` (gitignored; schema in `config.json.example`) holds secrets plus deployment settings: `data_providers.active` (the live-feed switch), `data_providers.streaming` (optional tick-stream provider for the level-breach monitor), `scanning.pairs`/`timeframes`, per-pair `blocked_hours_utc`, `log_level`. Key env vars: `TELEGRAM_TOKEN`, `TELEGRAM_CHAT_ID`, `TELEGRAM_CHANNEL_ID`, `TWELVE_DATA_API_KEY`, `TIINGO_API_KEY`, `OANDA_API_TOKEN`/`OANDA_BASE_URL` (the data feed needs no account id; a legacy `oanda.account_id` key is tolerated and ignored; so is a legacy top-level `market_outlook` block, with one warning — feature removed 2026-09-15), `DEEPSEEK_API_KEY` (news LLM; the `deepseek` config.json block holds model/enabled/timeout/daily cap). The `news_archive` block (`enabled` default TRUE, `path` default `logs/newsdata.db`; env `NEWS_ARCHIVE_ENABLED`/`NEWS_ARCHIVE_PATH`) controls the durable headline corpus. The `agent_panel` config.json block (all-defaults = feature dark) controls the advisory LLM agent panel: `enabled`, per-surface `telegram`/`dashboard` toggles, `min_quality`, `max_panels_per_day`, `reuse_ttl_seconds`, plus the technical-payload enrichment knobs `candles_enabled`/`candle_bars_primary`/`candle_bars_confirm`. The dark-by-default `violent_watch` block controls the display-only violent-move watch: `enabled`, `alert_line`, `headsup`, `max_headsup_per_day`, `headsup_min_level` (`"watch"`/`"hot"`); `VIOLENT_WATCH_ENABLED` overrides its enabled setting. +## 2. How a scan works, from A to Z -## Architecture +A "scan" is one analysis of one timeframe. The bot runs one scan per timeframe at every candle +close; the CLI and the dashboard run the same scan on demand. Everything below happens inside one +scan, in this order. -### End-to-end pipeline (every scan) +**1. Trigger.** `telegram_bot.py` wakes 10 seconds after each candle boundary (M5 every 5 min, +M15 every 15 min, H1 hourly, H4 every 4 hours). `dashboard_publisher.py` has its own copy of the +same schedule. `main.py` runs once when you call it. -``` -providers/.fetch_ohlcv() per TF: single-flight fetch, drop forming candle, validate/clean, cache - (only Twelve Data rate-throttles; Oanda has no client throttle) - → engine.TechnicalEngine.analyze() per TF: 9 scored indicators + ATR context → regime → composite score - → signal_aggregator.analyze_pair() live entry point (weekend/day filters, fetch, staleness) - └─ analyze_from_dataframes() replayable core: MTF agreement → news → bonuses/penalties → - final signal + quality grade → gates → trade setup → reasons - → SignalReport consumed by telegram_bot.py, main.py, dashboard_publisher.py, backtester -``` +**2. Build the configuration.** `pair_signals/xauusd.py` (`make_config`) creates a +`PlatformConfig` from `config.py`, applies the deployment overlay from `config.json` +(`app_config.py`), then applies the validated per-timeframe overrides (section 8). Never build a +bare `PlatformConfig()` for a real scan; always go through the factory. The confirmation +timeframes come from the factory map: M5 → M15 + H1, M15 → M5 + H1, H1 → M15 + H4, H4 → H1 + D1. -Signal currency everywhere is `SignalStrength` (−2..+2). DataFrames are UTC tz-aware, oldest-first, capitalized OHLCV columns. Everything runs on **closed bars only** — the forming candle is dropped, and session/hour filters evaluate **bar-CLOSE time** (bar label + candle duration) identically in live and backtest. +**3. Fetch candles.** `signal_aggregator.analyze_pair` (the live entry point) first checks the +calendar: on the forex weekend or a blocked weekday it returns a "blocked" scan and does nothing +else. Otherwise it asks the active data provider (`providers/`, Oanda today) for the primary +timeframe and each confirmation timeframe. The provider base class drops the still-forming candle, +validates and cleans the bars, caches them until the next candle close, and checks freshness. If +any required frame cannot be fetched after the retry ladder, the scan is logged as "failed" and no +analysis runs. -### Layers +**4. Score each timeframe.** `engine.py` (`TechnicalEngine.analyze`) computes ATR first, then nine +scored indicators: EMA trend (9/21/50/200 ribbon), RSI, MACD, Bollinger bands, ADX, rate of change, +Keltner channel, divergence, and candlestick patterns. Each indicator votes on a five-step scale +(STRONG SELL, SELL, NEUTRAL, BUY, STRONG BUY) with a confidence. The engine classifies the market +regime from ADX and the EMA ribbon: `trending_up`, `trending_down`, `trending` (strong trend, +mixed direction), `transitioning` (ADX rising fast), or `ranging`. Each regime has its own table of +per-indicator weight **multipliers** (`regime_weight_table` in `engine.py` is the single source of +truth); the effective weight is the indicator's base weight × the regime multiplier. The composite +score is the confidence-weighted average of the votes (vote value × effective weight × confidence), +scaled by 50 and clipped to −100…+100. Divergence and candlestick only vote when a pattern exists; +a missing vote is "no opinion", not zero. Three engine conventions matter when editing: MACD is +always computed even when it is not a scored pillar, because divergence needs its histogram; RSI is +reinterpreted per regime after classification (`rsi_mode = "regime_aware"`; treating the mixed +`trending` regime as an uptrend was a −6.8R bias); and private helper series ride on the result as +`_underscore` attributes, which every scorer and formatter skips along with weight-0 indicators, so +a new indicator must follow that convention. -- **`providers/`** — one class per provider subclassing `BaseDataProvider` (`providers/base.py`), which owns everything the pipeline depends on: TTL+boundary cache, forming-candle drop, freshness check, validate/clean, paginated `fetch_range` (feeds the backtester from any provider), optional tick streaming. Subclasses implement only `_fetch_raw` + symbol/interval maps. **Cleaning rule (2026-09-07):** the signal-side outlier filter is a bad-print filter, not a volatility filter — a >10σ close-to-close bar is dropped only when the NEXT close snaps back ≥80% of the way (`OUTLIER_SIGMA`/`OUTLIER_REVERT_FRACTION` on the base class); persisting moves (NFP/CPI/Sunday-gap bars) and the newest bar are always kept. The previous unconditional 10σ rule deleted ~36 real M5 bars/yr in the backtester and a *different* ~24/yr live (window-dependent σ), hiding e.g. the 2026-09-04 12:30 UTC NFP bar from ATR/exhaustion math. Oanda fetches retry 401/5xx/network faults along `RETRY_DELAYS_S` (2/5/10 s) under a 45 s `RETRY_BUDGET_S`; a scan that still loses its primary frame is logged with `scan_kind='failed'`. **Live fetch depth (2026-09-10):** every live fetch requests `max(MIN_LIVE_BARS=1000, min(60 days × bars/day + 20, 5000))` bars (`_live_outputsize`) — M5/M15 5000, H1 1460, H4 1000 (was 380), D1 1000 (was 300). The engine's EMAs use `ewm(adjust=False)`, so the first bar of the frame keeps weight (1−2/201)^n in the 200-bar trend EMA; at 380/300 bars the live H4/D1 trend reading flipped on ~0.3–0.75% of bars versus the backtester's converged history (and on the live D1 bar of 2026-09-10 itself), at ≥ 600 bars on none (`backtest/research/cycle_2026-09-10.md`). Registry + `make_data_provider(config, name=None)` in `providers/__init__.py`; config.json `data_providers.active` selects the live candle feed and the optional `data_providers.streaming` scalar selects a different provider for the tick stream. A test enforces that subclasses don't override shared base methods (`supports_streaming`/`iter_ticks` are the sanctioned streaming override points). Tiingo streams mid-price ticks (`wss://api.tiingo.com/fx`) — do not re-add spread display from its free FX feed (two-tier indicative quotes, removed 2026-07-23); the no-spread rule extends to the WS stream. -- **`engine.py`** — `TechnicalEngine`: 9 scored indicators (ema_trend, rsi, macd, bollinger, adx, roc, keltner, divergence, candlestick) + ATR as context. Regime classification (trending_up/down, `trending` = mixed/direction-UNKNOWN, transitioning, ranging) drives per-indicator weight multipliers and regime-aware RSI reinterpretation. Compute order matters (ATR first; MACD always computed even when not a pillar — divergence needs the histogram). Private series ride on the result as `_underscore` attrs; scorers/formatters skip anything `_`-prefixed or weight-0. -- **`signal_aggregator.py`** — MTF agreement, diminishing-decay penalty system, exhaustion hard block, quality grading, then an ordered chain of post-grade gates (session, XAU SELL quality/vol/hour, regime blocks, M5 noise gates), then three setup generators: XAU ATR (A+/A), generic swing-based, plus the M5 pullback (`XAU_PB`) and chart-pattern (`XAU_CP`, `chart_patterns.py`) entry paths. Step 7 lives in `_attach_core_setup`, which also stamps BUY/SELL on any shipped setup whose `final_signal` was still NEUTRAL (grade C at confidence 45–50, or gate 6c's B — the signal thresholds need ≥ 50) and repairs the pre-gate action text: the backtester always traded those setups (the replay never reads `final_signal`) while Telegram's alert gate rejected NEUTRAL first (2026-09-13; 5/91 trades, +3.7R of +22.2R, in the H1 12-mo book); a final pass after the PB/CP archetypes (`_drop_stale_spread_gate_vetoes`, 2026-09-08) moves a core generator's `SPREAD GATE:` veto to `cleaned_warnings` whenever a later generator shipped a setup (metadata only — books unchanged; the exported `warnings` text column can differ; the prefix is the `_SPREAD_GATE_PREFIX` constant shared by the emitters and both cleanups). The PB/CP generators run AFTER the gate chain and force quality to "B"; the CP path deliberately accepts grade-F composites and re-checks only session/day/big-candle (via warning prefixes) plus its own SELL/vol/RSI gates — it does NOT re-check exhaustion or the M5 min-score/MTF gates. **This is validated behavior, not a bug** (2026-08-04 audit: 226/586 trades in the shipped M5 12mo book are XAU_CP, +38.9R, 104 of them exhaustion-flagged and net-positive) — changing it is a rule-1 revalidation event. `TradeSetup.strategy` selects trailing params; trail params are stamped on the setup from the *producing* config so per-TF overrides reach formatters. -- **`pair_signals/`** — per-pair config factories. **Always use the factory (`produce_signal` / `make_config`), never bare `PlatformConfig()`** — the factory applies the config.json overlay first, then the validated per-TF deltas (M5 hour-9 block, M15 trail arm 0.70, M15 SELL-hour block, M5 SELL-in-transitioning block, H1 all-shorts block (2026-09-03), M15 RSI regime-multiplier × 0.5 via `technical.regime_weights_override` (2026-09-09), etc.; evidence in the docstring). A test enforces factory ≡ inline-config equivalence. Confirmation map: M5→[M15,H1], M15→[M5,H1], H1→[M15,H4], H4→[H1,D1]. -- **`signal_log.py` / `advisory_outcome.py`** — SQLite signal-history logger (stdlib only). Hooked into `SignalAggregator.analyze_pair` only (the live entry point — the backtester never writes here, preserving parity), so every live scan from every surface (Telegram `telegram`, CLI `cli`, dashboard `dashboard`) lands one row in `logs/signals.db`: signal/grade/price, MTF + trade-setup + reasons JSON, full per-TF indicator snapshots, the last 14 closed OHLCV candles per fetched TF, and the news at scan time (event risk + calendar events + scored headlines, re-fetched via the NewsEngine class-level caches). Config.json `signal_log` block (`enabled`, `path`, `log_blocked_scans`; env `SIGNAL_LOG_ENABLED`/`SIGNAL_LOG_PATH`/`SIGNAL_LOG_BLOCKED_SCANS`). **Fail-safe by contract** — a logging failure can never break a scan. Panel fields arrive post-hoc: `log_signal` stashes the rowid and `agent_panel.attach_panel` writes verdict + creation/cache provenance + payload hash onto the same row. **Advisory outcomes (2026-09):** EVERY row with a trade setup is seeded at log time (not only panel rows); same-setup re-logs within 60 min from any surface (`setup_key` = tf|direction|entry|stop|strategy) are `DUPLICATE` rows pointing at the canonical row via `advisory_event_id`, so sum `advisory_r_multiple` over `RESOLVED` rows only. The versioned model advances from later stored closed-candle snapshots (next-open reanchoring, intrabar peak with close-only ratchet, SL-first, scheduled weekend/21:00-UTC closures allowed, 300-bar timeout, $0.30/$0.65 views); when a snapshot no longer overlaps the outcome's last bar (either gap reason) it rebuilds the stream from every stored snapshot of that TF; a gap the database cannot fill resolves at the locked trailing stop as a FLOOR (`exit_reason=GAP_LOCKED`, `result.floor=true`, `advisory_outcome.resolve_locked_floor`) when the stop had already crossed the entry, and stays `UNRESOLVABLE` otherwise (2026-09-13 — rows 915/920 of the Sep-8 shorts were +1.14R/+0.84R floors the scorecard had been dropping). It is a cleaned-mid evidence model, not execution P&L. **Post-gate truth columns:** `gate_reason` (the warning that removed the setup), `actionable` (grade ≥ C AND setup — the `action` text is written BEFORE the gate chain and can say "STRONG SELL — High-quality setup" on a D row), `cleaned_warnings_json` (warnings a shipped setup superseded — the XAU_CP acceptance's stripped vetoes and, since 2026-09-08, a stale core spread-gate veto; non-empty is NOT a CP marker, use `setup_json.strategy`), `bar_range_median` (= the misnamed `spread` column: median High-Low, NOT a broker spread), `scan_kind` (`blocked` weekend/day early returns carry no news/candles; `failed` = ANY required frame — primary, confirmation, or the regime-filter TF when that filter is on — failed to fetch: price 0, no signal, no analysis run (2026-09-13; was primary-only, and a lost confirmation frame used to run the scan with an unearned −15 MTF penalty) — readers skip it like `blocked`), `config_fingerprint` (gate config hash — surfaces config drift between processes). **News is content-addressed:** `headlines_json`/`events_json` are interned into `news_snapshots` (they were 73% of a 667 MB DB); read through the `signals_with_news` view or `signal_log.read_news`. Readers must open with `mode=ro`, never `immutable=1` (blind to WAL rows). **Weight-research columns (2026-09):** `bar_ts` (the primary bar LABEL the scan closed on — same string as `indicators_json`, indexed with `timeframe`), `engine_fingerprint` (hash of the SCORING config: indicator params, thresholds, the regime weight table in force, TFs — `SignalAggregator.engine_config_fingerprint`, the gate fingerprint's twin), `code_version` (short git SHA), `penalty_tags_json`/`penalty_total`/`momentum_fading`/`exhaustion_count`, and the post-hoc `watch_level`/`watch_kinds`/`watch_line` written by `signal_log.update_watch` from `violent_watch.attach_watch` (level and kinds are stored even when the display line is off). Every `indicators_json` entry also carries `eff_weight` (base weight × regime multiplier = the weight the composite actually used; `engine.regime_weight_table` is the single source of truth) and `wscore` (its contribution) — null for skipped indicators (weight 0, `_`-prefixed); **divergence and candlestick entries exist only when they fire, so absence is "no vote", not a zero vote**. **Table `scan_outcomes`:** model-free forward moves per scan (`fwd_1h/4h/24h` in $ and ATRs, `mfe_24h_atr`/`mae_24h_atr`, `burst_1h` = a ≥3×ATR bar within the hour, `bars_used`, `complete`), measured from the row's own last closed bar on the stored candle stream; a horizon stays NULL until it has elapsed, and `complete` needs the full 24 h with no gap over 3 bars beyond the scheduled weekend/17:00-NY closures (`advisory_outcome._has_continuity`). `SignalLogger.label_scan_outcomes` runs opportunistically at the end of `log()` (at most once per 60 s per process, 150 rows and ~2 s per pass, idempotent — a row is rewritten only when its numbers change, but its `labeled_at` is bumped so it rotates to the back; an incomplete row older than `SCAN_LABEL_MAX_AGE_H` = 72 h (measured from the newest logged row) is FINAL and never anchors the shared bar stream, and `_stream_since` warns when it hits its row cap — before 2026-09-13 the Labor-Day pile anchored the stream and the 4,000-row cap would have blinded every later row), so no daemon is needed; `scripts/label_scan_outcomes.py` is the backlog driver (`--dry-run` counts read-only). Readers must dedup by the FIRST row per (`timeframe`, `bar_ts`) — several surfaces log the same bar. Tools: `analyze_signal_log.py` (`--stored-candles` DB-only replay with event/opportunity/one-position books + persona scorecard, `--news` news-sentiment lead/lag + extremes + outcome buckets, `--parity` telegram-vs-dashboard divergence report, `--scan-outcomes` forward-move + hit-rate table by TF×signal/grade with a NEUTRAL baseline and burst rate by watch level), `scripts/backfill_signal_outcomes.py` (seed/dedup/advance legacy rows), `scripts/label_scan_outcomes.py` (label/relabel scan outcomes). The one-time news-interning migration script was removed 2026-09-15 (last commit with it `931b265`). -- **`news_engine.py`** — Forex Factory calendar (MyFXBook RSS fallback) + RSS keyword sentiment with gold-inverse keywords. Class-level caches (fresh engine per scan). Optional DeepSeek enrichment via `llm_client.py` (requests-only, no SDK): batched headline sentiment overriding keyword scores, gold-direction on release surprises, and a market-summary blurb (`SignalReport.news_summary`) — **display/warning-only, keyword fallback on any failure, inert without a key**; prompts are static module constants (DeepSeek prefix-cache pricing) and requests send `thinking: disabled` (V4-Flash reasoning otherwise blows the timeout). By XAU-tuned default, event risk and sentiment **warn but do not gate signals** (`event_risk_caps_signals=False`, `sentiment_affects_signals=False`) — the LLM does not change that. **Headline pool (2026-09-04):** only headlines ≤ 4 h old with a parsable publish time (`headline_max_age_seconds=14400`, undated items dropped), and `get_fundamental_context` drops headlines whose every relevant score is inside ±0.1 (`_is_neutral_headline`, `NEUTRAL_HEADLINE_THRESHOLD`) before the per-currency mean — so `news_sentiment` from that date is a mean over directional headlines only (bolder values; log comparisons must split at the deploy date). The market summary prompt writes for someone who has never traded (no acronyms), sends headlines newest-first with the platform's own gold direction, and caps at 400 chars. **Verdict 2026-09-04** (`backtest/research/news_sentiment_value_2026-09-04.md`): `news_sentiment` is a trailing-24h echo of price (r≈0.67 with the past day, ≈0.03 with the next; flips 44 h after tops / 16 h after bottoms) — both gating flags stay False; re-check with `analyze_signal_log.py --stored-candles --news`. -- **`news_archive.py`** — durable headline corpus in its OWN database (`logs/newsdata.db`), separate from `signals.db` so a signal-log rotation cannot take the news with it (that is exactly how the 3-week corpus behind `news_sentiment_value_2026-09-04.md` was lost — 4,054 distinct headlines, unrecoverable, `logs/` is gitignored). One row per distinct headline keyed on `sha256(headline)`, carrying the text, description, link, publisher, publish time, the score, `score_source` (`keyword`/`llm`/`model` — a new `NewsSentiment` field, also mirrored into `signal_log._headline_to_dict`), and `teacher_model`/`teacher_prompt_sha1` so labels made under a changed prompt are never pooled with older ones. **Writes are one atomic UPSERT per row and upgrade-only**: an `llm` score always replaces a `keyword` one (the >25/cycle batch-overflow tail gets its DeepSeek label on the next fetch window in a long-running process — verified: 13/13 upgraded) but a `keyword` score can never overwrite an `llm` one; re-seeing a headline always bumps `seen_count`. A read-then-write pair loses increments when the bot, the dashboard publisher and the tests first see a headline together, and the WAL switch itself needs `_with_lock_retry` — both are regression-tested with 3 concurrent writers. Hooked at the END of `news_engine._fetch_news_sentiment_raw`, **before** `_is_neutral_headline` runs in `get_fundamental_context`: archiving downstream of that ±0.1 filter is why `signals.db` holds almost no neutral headlines and would teach a trained scorer that "neutral" barely exists. Never raises (fail-safe by contract, like `signal_log`). The backtester never builds a `NewsEngine`, so replays never archive (test-locked). Seeded 2026-09-08 by a one-time backfill script (removed 2026-09-15, last commit with it `931b265`) that read both DB shapes read-only and **recomputed** provenance (stored score vs the keyword scorer: differs → `llm`, exact non-zero match → `keyword`, both zero → `unknown`) because neither source recorded it. Purpose: training data for a local numpy-only headline scorer to replace the 24/78 keyword fallback — see the note below. -- **`agent_panel.py`** — ai-hedge-fund-style advisory panel: 4 DeepSeek persona calls (macro/technical/risk/contrarian, concurrent on a panel-private pool) + 1 synthesizer producing an **ADVISORY-ONLY** consensus (`AGREE/NEUTRAL/OPPOSE` + confidence + rationale) written onto `SignalReport.panel_*` fields. Attached **post-hoc** by consumers (`telegram_bot` alert path, `main --panel`, `dashboard_publisher`) — never runs inside `analyze_from_dataframes` (backtest parity, test-locked) and no gate reads its fields. Persona payloads are role-filtered: macro gets high-impact events plus only code-scored directional news; technical gets strategy/pattern, primary indicators, MTF, levels, and ATR-normalized candles; risk gets setup strategy, strategy cost assumption, trailing semantics, volatility/events/warnings (never the median candle range as spread); contrarian gets directional news, setup, timestamped latest-closed price, indicators, reasons, and warnings. Spend is bounded by exact cache, full-evidence-identity TTL reuse, and `max_panels_per_day`; changed evidence never inherits a direction-only cached verdict (test-locked — which also means TTL reuse effectively never fires live because price/candles change every bar; the daily cap is the real spend bound). Verdict provenance logs creation time, `MISS`/`EXACT_HIT`/`TTL_REUSE`, and a SHA-256 payload hash. Prompts forbid outside knowledge and outputs remain short. **OPPOSE guard (2026-09):** the 3-week signal-log scorecard showed the contrarian/risk/technical personas oppose almost every trend entry (chair said OPPOSE 73% of the time; opposed trades made +30R, PF 2.6) while the macro persona was the only informative voice (macro=support → 88% win). `_apply_oppose_guard` therefore downgrades a chair OPPOSE to NEUTRAL unless macro opposes, or technical opposes while macro is not supportive; the synthesizer prompt states the same rule. Re-score with `analyze_signal_log.py --stored-candles` (persona scorecard) before changing it again. **Verdict source (2026-09-03, owner decision):** all four personas still run and are logged (`panel_takes_json`), but `agent_panel.verdict_source` defaults to `"macro"` — the alert verdict is the macro persona's stance (support/neutral/oppose → AGREE/NEUTRAL/OPPOSE) and the synthesizer call is skipped (4 LLM calls per panel instead of 5); `"chair"` restores the four-voice synthesizer + OPPOSE guard. Re-score personas monthly with `analyze_signal_log.py --stored-candles`; weights are not to be tuned before ~300 graded outcomes with a held-out half. -- **`violent_watch.py`** — **display-only** violent-move watch, dark by default (`violent_watch` config block: `enabled`, `alert_line`, `headsup`, `max_headsup_per_day`, `headsup_min_level`; env `VIOLENT_WATCH_ENABLED`). Flags when a "violent bar" (range ≥ 3× prior ATR14) is unusually likely and stamps `SignalReport.watch_line` **post-hoc** via `attach_watch` (telegram alert path, `main.py`, `dashboard_publisher`) — same contract as the agent panel: never imported by `signal_aggregator`/`engine`/`backtest/`, never raises, never read by a gate (test-locked, Module 30). Triggers: a high-impact **USD** calendar release within ±30 min (hot), hot clock slots (weekday 12–14 UTC, weekday 01 UTC, Sunday reopen 22–23 UTC), and **price-extreme** indicator readings (outside the Bollinger bands, RSI <30 / ≥70, outside the Keltner channel). Level = hot when the calendar fires or a clock slot plus ≥1 price extreme; watch on a clock slot alone or ≥2 price extremes. Context readings (ADX slope >3, volatility "extreme", regime "transitioning", MACD STRONG_BUY/STRONG_SELL) are printed but **never raise the level** — the first draft that let them count flagged 27% of bars "hot" at only ~2× odds; the shipped rule flags 8.6% at 3–5× (evidence: `backtest/research/sizing_campaign_2026-09.md` addenda). Direction is called DOWN on bearish readings (70–78% on M5 and M15) and UP only on M5. Telegram also sends a standalone heads-up on the fastest scanned TF (`_HEADSUP_TF`), rate-limited per UTC hour+trigger-set and per day; it explicitly tells the owner **not** to touch open trades — every protection tested lost money, the stop is the protection. -- **`telegram_bot.py`** — owner-locked bot + subscriber channel; candle-boundary auto-scan loops (M5/M15/H1/H4, boundary+10 s), alert quality threshold + fingerprint dedup (committed only after a successful send; the fingerprint is signal+direction — grade is NOT part of it, a grade UPGRADE re-alerts, an A↔B flicker does not), 📰 LLM market-context line on alerts when available, WebSocket level-breach monitor fed by the streaming provider (`data_providers.streaming`, falling back to the active provider; auto-disabled when the resolved provider can't stream or has no API key — Oanda does not stream, Tiingo does). The breach handler reuses this bar's M15 report (`_cached_or_scan`, keyed by closed-bar epoch) instead of rescanning — it used to log 3.5 M15 rows per candle. **Alerts require a trade setup (2026-09-08):** `_is_alert_candidate` (shared by the scan and breach paths) needs non-neutral + grade ≥ threshold + `trade_setup`; a signal every generator declined used to go out anyway (a B-grade no-setup SELL under a "costs would eat the edge" warning). **Sends are logged:** every delivered alert / breach alert / violent-move heads-up writes an `… sent:` INFO line carrying the signal-log row id, and a no-setup candidate writes `alert skipped: … (gate reason)` — before this only failures were logged and a review had to reconstruct sends from panel rows. Rotating log: `logs/telegram.log`, one file per server-local day × 30 backups (was 3-h × 240 until 2026-09-08; the old `_HH`-suffixed files on the live box are not matched by the daily cleanup and need one manual delete). -- **`dashboard_publisher.py` / `static/dashboard.html`** — web dashboard as a **static publisher** (live LAMP server): standalone daemon scans at candle closes via `produce_signal` and atomically writes `snapshot.json`/`news.json`/`dashboard.html` to `$DASHBOARD_OUT_DIR` (Apache serves them; page is public by owner decision). **Hard independence rule: never imports, and is never imported by, `telegram_bot`** — confirmation map and boundary math are deliberate local copies, so a web-side failure can only mean a stale page. Due-TF classification uses the RAW wake time (flooring wake−buffer skips every H1/H4 scan — 2026-08-02 bug). News panel reads `NewsEngine.get_fundamental_context()` plus the additive `get_week_events()` accessor (display-only; no signal path calls it); as a separate process it has its own news caches and DeepSeek daily cap. News/calendar rows are clickable (detail dialog); headlines carry `id` = sha1(headline) in news.json, and exactly one on-demand endpoint exists: `static/news_summary.php` (installed by `install_static`), which serves a DeepSeek gold-impact summary per headline — whitelist-gated against news.json ids, cache-forever in `ai/`, own fail-closed daily cap (150/day, file-based, separate from the publisher's), key at `/etc/gold-dashboard/deepseek.key`, never reads repo config. Deployment checklist: `deploy/README-dashboard.md`. -- **`backtest/gpu_backtest.py`** — bar-by-bar replay through the real `analyze_from_dataframes` (news off). Realism model (do not regress): next-bar-open entries re-anchored to the raw price path, round-trip `--spread` charged per trade, trade resolution on RAW bars (cleaning applies only to signal-side data), same-bar SL+TP tie → SL first, gap fills at bar open, trail ratchet at bar close only. Indicators are precomputed once per TF over the whole cached history (CuPy GPU or NumPy fallback — check the `Backend:` log line); the per-TF warm-up pad before `--start` is `_history_pad_days` (62 days for M5/M15/H1, 234 for H4, 1400 for D1 — the same 1000-bar seed the live floor guarantees; 2026-09-10, no trade in any book moved); replay fans out over worker processes and the 5-bar cooldown is applied in an ordered pass afterwards, so parallel and sequential books are identical. Parquet cache in `backtest/cache/` keyed by provider/symbol/TF/window. Pinned trade books are no longer kept on disk (owner decision 2026-09-15): every gate MD5 and its command line is recorded here or in the ledgers, so regenerate a book from the cache when a trade-level diff is needed (`backtest/*.csv` stays gitignored for that). `backtest/instructions_for_backtest.txt` lists what every backtest report must record. +**5. Multi-timeframe agreement.** `analyze_from_dataframes` compares the primary timeframe's +direction with the two confirmation timeframes. Agreement needs 2 of 3. Disagreement costs a +penalty and blocks the top grades. -## Rules for making changes (read before editing signal logic) +**6. News context (warning only).** `news_engine.py` reads the Forex Factory calendar (MyFXBook +RSS as fallback) and four RSS feeds, scores headlines for gold, and can ask DeepSeek to improve +the scores. This produces warnings and a text summary. **By design it never changes a signal.** +The backtester runs with news off, so the live decision must be purely technical too. -1. **Never change a tuned value without re-validation.** Acceptance rule: positive total R in BOTH train and validation halves of a ≥12-month window, PF ≥ 1.3, at realistic cost (XAU $0.65 round trip headline; retain $0.30 for TD-era cross-book comparability), trailing exits, next-open fills. `round_trip_cost` stays $0.30 because it also drives the live TP spread gate; changing it is a separate rule-1 event. Marginal edges must additionally pass the 2024-06→2025-03 out-of-sample window — within-year splits alone have approved configs that later failed OOS. **Owner preference (2026-09-05): backtest windows go back at most 3 years** (the 5-year books are historical reference only); the 3-year baseline books (2023-09-05→2026-09-05, $0.65: M5 857/+161.3R/PF 1.53, M15 1086/+148.7R/PF 1.37, H1 255/+70.5R/PF 1.98, H4 113/+22.7R/PF 1.63) are regenerated with the commands in `sizing_campaign_2026-09.md` (their on-disk copies were removed 2026-09-15; the current-HEAD figures are in the 2026-09-10 note). -2. **Per-timeframe overrides live in the pair factory**, never in global defaults — the same knob has opposite effects on different TFs (proven repeatedly: `tp_floor_rr`, hour-9 block, trail params). Document validation evidence in the factory docstring. -3. **Pair isolation is a hard invariant.** XAU gates in the aggregator are display-name-gated so a future pair port cannot inherit gold-calibrated logic by accident. -4. **Live == backtest parity.** If you add a live gate, add it to the replay path in the same commit; all time filters evaluate bar-close time in both paths. -5. **ROC stays weighted** even though it's correlated with MACD/Keltner/EMA — zeroing it backtested 13R worse; composite thresholds are calibrated with it. -6. **Backtester realism knobs are results-critical** (raw-path resolution, spread charging, next-open entry, SL-first ties, close-only trail). Touching them invalidates comparability with all pinned books. -7. **The forming candle must stay dropped** — live signals repaint otherwise. -8. Don't add config fields nothing reads; don't disable indicators in production paths (`disabled_indicators` is for experiments only). -9. EUR/USD was removed 2026-07-16 after a no-edge verdict (cost-to-volatility; last commit with EUR support: `7d3fd95`). Do not re-add without new data. BTC/ETH work lives on unmerged branches only. +**7. Penalties, then the signal and the grade.** When the timeframes agree, confidence gains 10 +points and the score is blended 60/40 with the mean of all timeframe scores. Then a list of +penalties (opposing timeframes, stretched price, fading momentum, and so on) is subtracted from the +**confidence** with diminishing weight: each further penalty counts half, and the total is capped +at 30 points. Penalties never reduce the score itself. If two or more "exhaustion" penalties fire +together with fading momentum, the trade is hard-blocked (confidence set to 0). Then: -## 2026-08-04 QA audit — owner decisions ledger +| Result | Rule | +|---|---| +| STRONG BUY / STRONG SELL | score ≥ 35 (≤ −35) and confidence ≥ 55 | +| BUY / SELL | score ≥ 25 (≤ −25) and confidence ≥ 50 | +| NEUTRAL | anything else | +| Grade A+ | abs score ≥ 40, confidence ≥ 65, timeframes agree | +| Grade A | abs score ≥ 33, confidence ≥ 55, timeframes agree | +| Grade B | abs score ≥ 28, confidence ≥ 50 | +| Grade C | abs score ≥ 25, confidence ≥ 45 | +| Grade D | abs score ≥ 15 | +| Grade F | below that | -- **Weekend calendar: FIXED (owner-approved 2026-08-04).** All three signal-path `_is_forex_weekend` copies are NY-anchored (Fri 17:00 → Sun 18:00 ET; during US DST that is Fri 21:00 UTC — measured on 5yr of Oanda bars). Kills the weekly false-"STALE DATA/feed lagging" hours. Book impact at the audit HEAD: M15 byte-identical; M5 12mo dropped exactly ONE trade — a phantom whose signal fired on the final summer-Friday bar (close 21:00 UTC) and "filled" at Sunday's reopen 49 h later (−1.08R, live-impossible), leaving 555 trades / +98.64R / PF≈1.50 @ $0.65 (was 556/+97.56/1.48). The current-HEAD canonical reference is recorded below. A Module 21 test sweeps those copies plus the advisory outcome's continuity copy hourly across both DST transitions. -- **Spread in exit-trigger detection (rule 6): owner DECLINED 2026-08-04.** Backtest stop triggers stay on raw mid High/Low; the measured optimism (~4–6% of total R: −3.55R @ $0.30, −4.24R @ $0.65 on the M15 12mo book) is accepted and books stay comparable. Re-open only as a conscious re-pin campaign. -- **`FB2_EMA_LO`: HARDENED 2026-08-04 (delegated).** Research override for the EMA200-proximity band, default = tuned 0.5; an unparseable value now degrades to the default with a warning instead of crashing every scan. Documented here — it is a knob that exists. -- **H4 seeding divergence: parked.** Live H4 fetches only 380 bars vs the backtest's unbounded history — ~2% of sampled H4 bars flip an indicator signal (composite Δ up to 12.3). M5/M15/H1 clean at their real fetch depths. Fold into any future H4 campaign; H4 remains "unproven". +**8. The gate chain.** A series of checks can lower the grade or remove the setup. They run in this +order (`signal_aggregator.py`, steps 6b to 6i): session and day filter (the bar-label hour and weekday +against the blocked hours and allowed weekdays; the live wrapper and the backtester apply the same +list to the bar-close time as well, see section 8); transitioning-regime boost (a B/C/D can become A or B when a +trend is emerging); exhausted-trend cap (in a trending regime with two or more exhaustion tags, a C or D is pinned +to D and loses its setup); SELL quality gate +(shorts need a grade in `sell_min_quality`; H1's list is empty, so every H1 short is blocked); +multi-timeframe opposition gate; SELL volatility gate (shorts are allowed only when volatility is high or extreme; low- or +normal-volatility shorts are blocked); +regime blocks (M15 blocks the mixed `trending` regime; M5 blocks shorts in `transitioning`); +SELL-hour gate (M15 blocks shorts at 15:00 and 17:00 UTC); big-candle gate (shipped off); and on +M5 six extra noise gates (minimum score, required agreement, confirmation strength, SELL grade +cap, chop-transition volatility, SELL RSI floor). Every gate that removes a setup writes a warning +that names itself. -## Current state notes (dated per entry) +**9. Trade setup.** Grades A+ and A get the gold ATR setup (`_generate_xau_atr_trade_setup`): stop +2 × ATR; target 1 at 2 × ATR for A+ and 1.5 × ATR for A (1 × ATR for an A carrying two or more +exhaustion tags); target 2 at 1.5 × the target-1 distance. Other tradeable grades get the generic +swing-based setup, which floors the stop at $8 (80 pips). Both are checked against the spread gate: +target 1 must be at least 30 × the round-trip cost ($9 at the $0.30 cost setting). Two extra entry paths run after the +gates: the M5 pullback (`XAU_PB`, shipped off) and the chart-pattern entry (`XAU_CP`, +`chart_patterns.py`: M5 only, buy-side patterns inverse head and shoulders, double bottom, bull +flag). Both late paths run after the gate chain and force the grade to B, so an F-composite +chart-pattern trade shows as a B in books and alerts. The chart-pattern path deliberately accepts +weak composites and re-checks the session, day and big-candle rules, the multi-timeframe-opposition +rule and the ATR floor, plus its own SELL, volatility and RSI checks; it does not re-check +exhaustion or the M5 minimum-score and confirmation gates. That is validated behaviour, not a bug +(section 11, 2026-08-04). A shipped setup always carries a BUY or SELL direction, even +when the raw signal was NEUTRAL at grade C (confidence 45–50): the direction is the sign of the +score. Every setup is stamped with its strategy name and its trailing parameters. Finally, a stale +"costs would eat the edge" veto from an earlier generator is moved to `cleaned_warnings` when a +later generator shipped a setup. -- Oanda v20 is the live feed (`data_providers.active: "oanda"`). The 2026-08-04 re-pin made the 27 `backtest/oanda_*.csv` books the baseline; all TD-era books are superseded. (2026-08-14) The re-pin M5 book `oanda_m5_12mo_065_nopb_20260814.csv` (+106.82R PF 1.51) is itself SUPERSEDED by the 2026-09-03 book in the entry below; the M15 reference `oanda_m15_12mo_065.csv` (397 / +73.93R / PF 1.53) is SUPERSEDED on 2026-09-09 by the 2-year book `oanda_m15_2yr_065_20260909.csv` (931 / +146.14R / PF 1.43; RSI × 0.5 override shipped — see the 2026-09-09 entries below). See `backtest/research/repin_verification_2026-08-14.md`. -- Cost reporting is dual: $0.65 is the measured-Oanda realistic headline ($0.65-$0.72 round trip), while $0.30 remains for cross-book comparability and as the live TP spread-gate input. Raising `round_trip_cost` from $0.30 would move the TP floor from about $9 to $19.50 and requires full rule-1 revalidation. The $0.65 figure comes from the 2026-07-24 Dukascopy tick study (9M ticks over 12 months; its tooling was removed 2026-09-06 — the conclusion lives here and in memory). -- Twelve Data is signal-incompatible with Oanda (Phase 1: about 19% M5 bar-direction disagreement and a systematic ~$1 high bias; see `backtest/research/feed_divergence_2026-08-14.md`). On an Oanda candle outage, stop scanning rather than switching to TD candles. -- Tiingo is streaming-only (`data_providers.streaming: "tiingo"`), not a designated candle fallback. Oanda-Tiingo signal parity is documented in `backtest/research/signal_parity_2026-08-14.md`; the Phase 4 shadow-scan script was removed 2026-09-15 without ever being run (last commit with it `931b265`). -- Human-facing docs in `docs/` (`user_guide.html`, `beginner_guide.html`) must be kept in sync when signal behavior changes. There are NO PDF builds in the tree (checked 2026-09-10; earlier notes about PDFs lagging the HTML are moot). The backtester and the replay recorder read/write Parquet through `pyarrow`, which is installed in the dev `venv/` but deliberately absent from `requirements.txt` (live code never touches Parquet). -- 2026-09-03 signal-log campaign shipped two new per-TF gates (`backtest/research/signal_log_campaign_2026-09.md`): M5 `sell_blocked_regimes=["transitioning"]` and H1 `sell_min_quality=[]` (all H1 shorts blocked). Current-HEAD M5 canonical book is now `backtest/oanda_m5_12mo_065_nopb_20260903.csv` (557 / +112.21R / PF 1.56, halves +55.9/+56.4); H1 reference `backtest/oanda_h1_12mo_065_20260903.csv` (91 / +22.2R / PF 1.79; 5yr +49.1R PF 1.46; OOS +16.7R PF 1.74). -- 2026-09-06 stake-sizing campaign (`backtest/research/sizing_campaign_2026-09.md`): a walk-forward "second opinion" stake multiplier (meta-labelling on 65 decision-time features, 3-year Oanda books at $0.65) FAILED the pre-registered rule on M5 (clean null) and M15 (lift inside the label-shuffle null, explained by BUY-vs-SELL). Nothing that changes numbers shipped. Behaviour-neutral tooling remains: `position_sizing.build_features` + `--export-features` (default export stays 63 columns), `SignalReport.penalty_tags/penalty_total/momentum_fading`, `TradeSetup.stake_multiplier` (None; reserved) (the campaign's research scripts were removed 2026-09-06; git history has them). Stage-B facts: M5 stretch measures are POSITIVE in both halves (do not re-propose exhaustion sizing on M5); M15 exhaustion tag is weakly negative (≈ +3R/yr, noise); bursts cluster at 12–15 UTC and 01 UTC bar-close and do NOT hurt entries. -- Auto-execution (oanda_daemon.py / oanda_executor.py) was removed 2026-09-03 — the owner is not auto-trading; last commit containing it is 31002cb. Book-parity rules that lived in its docstring (no broker TP, 5-bar cooldown, once-per-bar trail) remain the backtester's model. -- 2026-09-07 signal-log inspection (fresh 2-day `logs/signals.db` import: NFP Friday + Monday; 5 setups, all BUY, 4 stopped out at local tops — two days is not evidence). Three sample-size-independent fixes shipped: (1) the signal-side 10σ outlier filter now drops only bars whose NEXT close snaps back ≥80% (it had been deleting ~36 real M5 news bars/yr in the backtester and a different ~24/yr live, e.g. the 2026-09-04 12:30 UTC NFP bar; 3-yr drops are now M5 2 / M15 1 / H1 0). Book impact at $0.65, 12mo: M5 557/+112.21R/PF 1.56 → 560/+108.81R/PF 1.54 (halves +55.6/+53.2), M15 397/+73.93/1.53 → 398/+74.61/1.53 (max DD 10.8→9.1R), H1 byte-identical; 3-yr (2023-09-05→2026-09-05): M5 857/+161.3/1.53 → 870/+153.6/1.50, M15 1086/+148.7/1.37 → 1092/+151.9/1.38, H1 255/+70.5/1.98 → 253/+70.0/1.98, H4 113/+22.7/1.63 → 114/+21.7/1.59 — only trades on the flagged event days move (M5 gives back a few Jan–Mar 2026 crash-period winners); the pinned M5/M15 CSVs are therefore no longer byte-reproducible (regenerate with the parity recipe when re-pinning). (2) Oanda candle fetches retry along a 2/5/10 s ladder under a 45 s budget (11 boundary scans in two days had lost their primary frame). (3) A scan that still loses its primary frame logs `scan_kind='failed'` and readers skip it. The live dashboard publisher was found dead (0 rows Friday, stopped Mon 02:20 UTC): install `deploy/gold-dashboard.service` on the live server. -- 2026-09-08 delivery-side fixes from the 3-day signal-log + telegram-log review (nothing in the signal core changed; 468 tests): (1) `_drop_stale_spread_gate_vetoes` (after every setup archetype, so the XAU_PB path is covered too) moves a core generator's stale `SPREAD GATE:` veto to `cleaned_warnings` when a later generator ships a setup — the 3-yr H1 and M15 books were regenerated before/after and are identical on every trade column (H1 253/+70.02R/PF 1.98, M15 1092/+151.90R/PF 1.38); only the exported `warnings` text differs, on 50/253 H1 and 358/1092 M15 trades, i.e. a fifth to a third of all A-grade alerts had been carrying "costs would eat the edge" beside a valid target; (2) Telegram alerts now require a trade setup (`_is_alert_candidate`, scan + breach paths; no-setup candidates log `alert skipped: … (gate reason)`); (3) every delivered alert / breach alert / violent-move heads-up logs an `… sent:` INFO line with the signal-log row id (the review had to reconstruct sends from panel rows); (4) `telegram.log` rotates daily × 30 instead of 3-hourly × 240 (old `_HH` files on the live box need one manual delete). Review facts (not evidence — three days): 8 setups, 1 right; strong readings (|score| ≥ 20) were followed by reversals in 16 of 18 episodes; hot watch level 0/25 bursts on M5 (calendar trigger never fired, 15 of 33 hot stamps were the 12–14 UTC slot on Labor Day — no holiday awareness anywhere); Oanda published no XAU bars 18:30–22:00 UTC on Labor Day and the bot scanned the stale bar 43×; the retry ladder recovered all 25 post-deploy 401 bursts (41 of 58 incidents sit on the :00 boundary); the live dashboard publisher ran only 01:00–02:20 UTC Sep 7 and is dead again. -- 2026-09-06 stale-code cleanup (behaviour-neutral; suite green, M5/H1 12-month books byte-identical after every signal-side commit): `main.py --timeframe` now accepts M5/M15/H1/H4 and takes its confirmation TFs from the pair factory (the CLI's private table had paired M5 with M1+M15); `PlatformConfig.pip_value` is gone (`PAIR_INFO` is the source); removed as dead or superseded: `_get_affected_pairs`/`event_impact_on_pairs`, `SignalLogger.advance_with_bars`/`path`/`news_for`, `reset_llm_client`, the Telegram scan-summary/keyboard helpers and duplicate confirmation map, the 2026-07 Oanda-vs-TD research pager+study, `burst_profile.py`, `hour_analysis.py`, `scripts/compare_oanda_tiingo.py`, the never-implemented LLM-provider plan and the merged config.json spec. Second pass the same day (owner directive: keep only what the system needs): every research script and data file under `backtest/research/` (book_report, book_diff, the burst tools, sizing_model, commit_ladder, llm_news_eval + golden CSV, ff_calendar_scraper, calendar_impact, watch_rule_eval, the Dukascopy spread study + CSVs, the panel dry-test JSONL), the two completed study scripts in `scripts/`, the architecture diagram + generator and the data-source test plan were deleted; the campaign `.md` reports stay as the decision ledger. Git history before `953c5ad` has all of it. Third pass (owner: "remove dead functions as well"): the write-only pb/cp rejection counters, the unreachable `_TP_QUALITY_SCALE` B/C entries, the engine's unused DataFrame argument and M1 arm, the providers' generic-symbol guesses (unknown symbols now raise), the sub-$10 price format, the `=X` pair parsing and the unread `minutes_until`/`is_high_impact` news.json keys are gone. -- 2026-09-08 news archive shipped (`news_archive.py`, 483 tests; nothing in the signal core changed, both news gating flags stay `False`, books untouched). Motivation: headlines only ever survived as a side effect of signal logging and `signals.db` was rotated around 2026-09-04, destroying the 3-week / 4,054-headline corpus the news study was written on — with `logs/` gitignored, git could not recover it and nobody noticed for four days. Day-one corpus after backfilling both DBs on disk: **1,016 distinct headlines, 846 of them DeepSeek-labelled**, growing ~200/day (~1,400/week). The point of collecting it: distil DeepSeek's headline scoring (78/78 on the 2026-08-06 golden set) into a numpy-only local model so the fallback stops being the keyword scorer (**24/78 = 31%**, which is what silently runs on no key / disabled / daily cap / timeout / HTTP or parse failure / batch overflow). **Do not train yet** — 800-odd labels gives a holdout too small to grade (SE ±0.032); the pre-registered plan is ~4,000 deduplicated headlines over ≥3 weeks, time-split with near-duplicates clustered first (the same story is re-titled by several outlets, so a random split scores memorisation), headline text ONLY (the teacher never sees `description` — `_llm_headline_scores` sends titles), graded against the golden set recoverable with `git show 1cebebf^:backtest/research/llm_news_golden.csv`. When it ships it slots in as DeepSeek → model → keyword and stays display-only. Training happens offline in `venv/`; a Module 32 test now AST-walks the root, `providers/` and `pair_signals/` asserting no `sklearn`/`scipy`/`lightgbm`/`torch` import and that none of them reach `requirements.txt` — that guard did not exist before. -- 2026-09-09 replay recording shipped (`backtest/replay_record.py`, `backtest/replay_scorecard.py`, Module 33, 498 tests; nothing in the signal core changed, books byte-identical — `backtest/research/replay_recording_2026-09-09.md`). `--record DIR` stores every ACCEPTED trade's decision evidence in Parquet: each indicator's vote / value / confidence / BASE weight / regime multiplier / EFFECTIVE weight / contribution on the primary AND both confirmation TFs (`indicators.parquet`, long; absence = the indicator did not fire, never a zero row), per-TF regime/composite, penalty tags, setup, outcome, and the Forex Factory calendar events from 4 h before entry to exit (`events.parquet`, all-day rows matched on the New-York date; scalar `ev_*`/`us_holiday` columns on `trades.parquet`); `--record-bars` adds one float32 row per ANALYZED bar with the same votes plus 1 h/4 h/24 h forward moves in ATRs (`bars.parquet`). The manifest carries the engine + gate fingerprints, the regime weight table in force and the book MD5. Calendar source is the gitignored `backtest/cache/calendar.parquet` (15,124 FF events 2023-09-01→2026-09-10, unix-UTC `dateline`; `timeLabel` is display-only); the scraper that built it is back at `scripts/ff_calendar_scraper.py` (resume + idempotent merge, ~1.6 s/request). The 2-year books (2024-09-05→2026-09-05, $0.65) live in `backtest/research/replay_2026-09/{m5,m15}/`: M5 781 / +146.07R / PF 1.52 (halves +28.0/+118.1), M15 793 / +117.62R / PF 1.41 (halves +38.2/+79.4); control-vs-recorded MD5 equal on both. First-look facts are descriptive only (Bollinger opposes 94% of M5 entries; divergence-opposed M5 trades did best again; every indicator's SELL votes were followed by moves against them on both TFs; the bar-level baseline drifts +1.9 ATR/24 h on M5 so BUY votes must be drift-adjusted first) — every candidate change stays a rule-1 rerun with a control book. -- 2026-09-09 weight-change test (`backtest/research/weight_test_2026-09-09.md`, first use of the recorded replay files): a drift-adjusted per-bar screen ranked 108 regime-multiplier candidates per TF; the top 3 per TF (24 h-sign filter) went through the full 2-year replay against same-harness controls plus the 2024-06→2025-03 OOS window under a rule pre-registered before any run. M5: all three MACD candidates FAIL (−10 to −46R; the screen ranked them first — never trust the per-bar screen on M5 without a replay). M15: ema_trend × 2 FAILS (PF 1.32, DD 26R, +55% trades), divergence → 0 in trending FAILS (H1 −3R, DD); **RSI multiplier × 0.5 in every regime PASSES a–f** (2-yr 793/+117.6R/PF 1.41 → 931/+146.1R/PF 1.43, halves +27.9/+0.6, OOS +40.6 → +60.7R PF 1.49, 3-yr +151.9 → +186.6R PF 1.39 but max DD 15.5 → 19.3R and 2026 −4R; added trades are mostly B-grade BUYs). SHIPPED the same day on the owner's decision as an M15-only `technical.regime_weights_override` in `pair_signals/xauusd.py` (M15 override 5: the shipped table with RSI halved in every regime — 0.5 trending family / 0.75 ranging / 0.5 transitioning; derived from the default table, pinned literally by the Module 9 equivalence test) — the factory run without `--set` reproduces the validated candidate book byte-for-byte (parity MD5 in `weight_test_2026-09-09.md`); M15 canonical book re-pinned to `backtest/oanda_m15_2yr_065_20260909.csv`. Live M15 `engine_fingerprint` changes from that deploy; signal-log comparisons across it must split at the deploy time. Suite 499 tests. -- 2026-09-10 improvement cycle 1 (`backtest/research/cycle_2026-09-10.md`; 501 tests; no signal-core change): HEAD baseline re-established on the 3-yr window at $0.65 (M5 870/+153.6R/PF 1.50, M15 1273/+186.6/1.39, H1 253/+70.0/1.98, H4 114/+21.7/1.59) and the three pinned MD5s (M5 2-yr, M15 2-yr, M15 OOS) reproduced exactly. Robustness: the cost ladder trips no red flag (every TF keeps PF ≥ 1.3 with positive halves at $0.80; at $1.30 M5 PF 1.36, M15 1.29, H1 1.89, H4 1.53); the intraday edge is a fat right tail (top 5% of trades carry 94% of M5 R and 96% of M15 R, versus 48% on H1; M5 2024 was flat at +1.0R over 102 trades). Pre-registered bucket screen: exactly one qualifying bucket — M15 grade-A SELLs (230 / −14.4R, negative in all three years and OOS; the 2026-09-03 5-yr run said the opposite) — parked as the only cycle-2 candidate. **C1 REJECTED:** H1 divergence multiplier → 0 in every regime (the parked 2026-09 candidate on its now-sanctioned vehicle): 3-yr ΔR +1.0, second half −1.5R, PF 1.98 → 1.89 (its OOS book 88/+21.9/PF 1.92 is trade-identical to the old `disabled_indicators` run, so a 0.0 regime multiplier ≡ disabling for the composite). **C2:** the M15 RSI × 0.5 ship sits on a stable plateau (k = 0.25 / 0.75 keep 78% / 71% of its 2-yr gain, both OOS-positive; R is monotone in k across 0…1.5) — no other k proposed. **Seeding parity SHIPPED:** live fetches floor at `MIN_LIVE_BARS = 1000` (H4 380 → 1000, D1 300 → 1000; measured ema_trend flips 2/267 H4 and 1/313 D1 bars at the old depths, none ≥ 600; the live D1 bar of 2026-09-10 itself read SELL at 300 bars and NEUTRAL at 1000/5000) and the backtester pad targets the same depth (`_history_pad_days`: 62 / 234 / 1400 days) — M5/M15 3-yr books byte-identical, H1/H4 trade-identical with only window-start float noise in the exported columns, so no re-pin. Also fixed in these notes: no PDF docs exist; `pyarrow` is a dev-venv-only dependency of the backtester/recorder. -- 2026-09-13 bug audit (plan file `~/.claude/plans/you-are-python-software-joyful-teacup.md`; 519 tests / 34 modules; the five reference books — M5 W2 `29120a1ee32979431d062f210d41a9d1`, M15 W2 `d185f099482a07f29502cb5a371a9bbf`, M15 OOS `6f8f24b49597ced28f50a19e08c3472f`, H1 W3 `3b45d12bade823178b267c13bc698458`, H4 W3 `e75adae13c839b15736f69c0114e6aa5` (the H1/H4 values are the post-pad books, `cycle_2026-09-10/w3_{1h,4h}_pad.csv`) — byte-identical before and after every change, 0 GPU cache fallbacks). Verified sound, no change: composite = Σwscore/Σeff_weight × 50 on 400/400 DB rows; scan-outcome labels reproduce later scan prices to 0.000 on 844 rows; outcome-model exits reproduced on rows 578/1372/1500; Oanda H4/D1 alignment is UTC; no look-ahead in engine/backtester/chart_patterns; no cross-bar state; parquet caches hold only holiday gaps. Fixed (all delivery/measurement side, books unchanged): (1) NEUTRAL-signal C/B setups now alert (see `_attach_core_setup`); (2) a missing confirmation frame fails the scan (`scan_kind='failed'`, warning `DATA UNAVAILABLE:`); (3) `pd.Timestamp.utcnow()` (deprecated, removed in pandas 4) replaced in `providers/base.py`/`engine.py`, `requirements.txt` pins `pandas<4`, `numpy<3`; (4) advisory outcomes keep locked-profit floors and the rebuild fires on both gap reasons; (5) scan-outcome labeler max-age/anchor bound; (6) the daily settlement pause (Mon–Thu 17:00 NY) is a scheduled closure for the freshness gates — `_is_daily_maintenance`/`_is_scheduled_closure`/`_scheduled_closed_seconds` in `providers/base.py`, the expected label walks back only past labels whose WHOLE span lies in a closure (a D1 bar labelled Sunday 00:00 UTC and an H4 bar labelled Sunday 20:00 exist), and the bot/dashboard skip scans whose just-closed label sits in the pause (61 false "stale" rows in 3 nights before); (7) `fetch_range` retries a failed page in place (3 failures on one window raise) and logs an ERROR for interior gaps of >3 unscheduled bars (holidays look identical — verify, never fatal); (8) `_replay_span` returns the per-span GPU cache-miss count, `run_backtest` sums/logs it (must be 0) and writes `cache_fallbacks` to the `--record` manifest; (9) `regime_weight_table` raises on a partial `regime_weights_override` (every regime × all nine indicators); (10) Telegram boundary loops go through `_cached_or_scan`, `_remember_scan` files a report under the bar it analysed (label + duration), a breach inside 20 s after an M15 boundary waits for the loop's scan (15 duplicate M15 rows in 5 days before), and the rotating file handler is attached in `main()` only (importing the module in tests wrote fixture alerts into `logs/telegram.log`); (11) dashboard `catch_up_timeframes` re-scans any TF whose newest closed boundary is newer than its last scanned one (missed while a cycle overran or the process was down). Semantics recorded, NOT changed (rule 1): `blocked_hours_utc` is applied to the bar LABEL hour in the core (6b) AND the bar CLOSE hour in the live wrapper/backtester, so H1 can only trade bars closing 04–09, 15, 18–20 UTC and H4 only bars closing 08:00/12:00 — parity holds and every book was validated that way. Post-deploy: `scripts/backfill_signal_outcomes.py --reseed-unresolvable` floors the two Sep-8 shorts on the live DB; the `--stored-candles` scorecard (`analyze_signal_log.replay_stored`) still calls `advance_outcome` without the floor step (owner decision pending). -- 2026-09-15 market outlook REMOVED (owner decision; plan `~/.claude/plans/you-are-quant-software-logical-cake.md`): `market_outlook.py`, the post-H1-scan Telegram 🔭 DM, `main.py --outlook`, the `outlooks` table DDL/migration/writer in `signal_log.py`, the `market_outlook` config block + `MARKET_OUTLOOK_ENABLED`, and test Module 26 are gone; the last commit containing the feature is `2836024`. A legacy top-level `market_outlook` block in config.json is TOLERATED and ignored with one warning (kept in `_KNOWN_TOP`, the `oanda.account_id` precedent) so the live bot/dashboard keep booting — delete the block from the live config.json at deploy. Existing `logs/signals.db` files keep their orphan `outlooks` table and rows (nothing reads or drops it; drop by hand during a weekend VACUUM if wanted); fresh DBs never create it. Module 26's 4-thread concurrent legacy-DB migration test (the suite's only one) moved to Module 31. Nothing in the signal path changed: the W2 books (2024-09-05→2026-09-05, $0.65, NumPy CPU, 0 cache fallbacks) are byte-identical before/after — M5 `29120a1ee32979431d062f210d41a9d1` (781 / +146.07R / PF 1.52), M15 `d185f099482a07f29502cb5a371a9bbf` (931 / +146.14R / PF 1.43). Suite 509 tests / 33 modules. -- 2026-09-15 backtest-directory cleanup (owner decisions, discussion-first; nothing in the signal core or backtester changed): REMOVED `backtest/diagnose_filters.py` (+ its 2 Module 18 tests), `scripts/shadow_scan.py` (never run), `scripts/backfill_news_archive.py` and `scripts/migrate_signal_log_news.py` (one-time scripts; the interning test kept its live-path half, the prune test went) — last commit containing all four is `931b265`. The replay-recording stack (`replay_record.py`, `replay_scorecard.py`, `scripts/ff_calendar_scraper.py`, `cache/calendar.parquet`, Module 33) STAYS. Local data (gitignored, so git has none of it) was MOVED, not deleted, to `~/Development/ForexBot2_trash_2026-09-15/` (346 MB) for the owner to delete: all 35 pinned `backtest/*.csv` books, the three research output folders (`cycle_2026-09-10/`, `replay_2026-09/`, `sizing_2026-09/`), the BTC/ETH `cache/snapshots/` (167 MB, unmerged crypto branches), the 4 Tiingo parquets and 40 uncited Oanda parquets (13 scratch/duplicate windows). The cache keeps 37 Oanda files (W3 `20230705`, W2 `20240705`, OOS `20240331`, M5-12mo `20250613`, M15/H1-12mo `20250603`, 5-yr `20210531`, 7.6-yr `20181031`, the Aug-14 parity window `20250814`, and their D1/H4 pads) + `calendar.parquet`. Verification: all five gate books regenerate offline from the pruned cache with the recorded MD5s — M5 W2 `29120a1e…`, M15 W2 `d185f099…`, M15 OOS `6f8f24b4…`, H1 W3 `3b45d12b…`, H4 W3 `e75adae1…` (NumPy CPU, 0 cache fallbacks, no network). Suite 506 tests / 33 modules. Pinned books are no longer kept on disk: regenerate from the cache when a trade-level diff is needed. +**10. Delivery.** The `SignalReport` goes to the surface that asked for it: + +- *Telegram* (`telegram_bot.py`): an alert is sent only when the signal is not NEUTRAL, the grade + reaches `alert_min_quality`, **and** a setup exists. A fingerprint (signal + direction) stops + repeats; a grade upgrade re-alerts. The fingerprint is committed only after a successful send, + so a failed send stays eligible next scan. Every send is logged with the signal-log row id. The + level-breach monitor reuses the bar's cached M15 report (`_cached_or_scan`) instead of rescanning. +- *CLI* (`main.py`): prints the report; `--json` dumps everything. +- *Dashboard* (`dashboard_publisher.py`): writes `snapshot.json`, `news.json` and the static page + into the Apache folder. + +Two advisory add-ons are attached **after** the scan by the surface, never inside the signal +code: the LLM agent panel (`agent_panel.py`, four DeepSeek personas, verdict AGREE / NEUTRAL / +OPPOSE) and the violent-move watch (`violent_watch.py`, a "the next hour may be wild" line). No +gate reads either of them. + +**11. Logging.** `signal_log.py` writes one row per live scan into `logs/signals.db` (signal, +grade, price, indicators, the last 14 candles per timeframe, the news at scan time, the setup, the +gate that removed a setup, a gate-config fingerprint and an engine fingerprint). The engine +fingerprint hashes the scoring config, so it changes whenever a weight or threshold ships; signal-log +comparisons across such a deploy must be split at the deploy time (the 2026-09-09 M15 RSI change is +one). `news_archive.py` writes every headline and +its score into `logs/newsdata.db`. Both are fail-safe: a logging error can never break a scan. + +**12. Measurement, later.** Two models read the stored candles and score what happened next: +advisory outcomes (`advisory_outcome.py`: replays each logged setup with next-open entry, stop +first, close-only trailing; a cleaned-mid evidence model, not execution P&L) and scan outcomes +(model-free forward moves 1 h / 4 h / 24 h after every scan). `analyze_signal_log.py` turns the +database into reports. + +**The backtester is the same pipeline on history.** `backtest/gpu_backtest.py` loads candles from +`backtest/cache/*.parquet` (fetched once through the same provider code), precomputes every +indicator series over the whole window, and calls `analyze_from_dataframes` bar by bar with news +off. Trades are resolved on the raw price path with next-open entries, spread charged per trade, +stop-first ties, gap fills at the open, and the trailing stop moved at bar close only. Time filters +use bar-close time in both paths, so live and backtest agree. The replay fans out over worker +processes; the 5-bar entry cooldown, the only cross-bar state, is applied in an ordered pass +afterwards, which is why parallel and sequential books are identical. + +--- + +## 3. File-by-file guide + +Arrows show who imports whom (generated from the code on 2026-09-15). + +### Entry points and surfaces + +| File | Role | Uses | Used by | +|---|---|---|---| +| `main.py` | Command-line signal report (`--detail`, `--json`, `--panel`) | pair_signals, signal_aggregator, agent_panel, violent_watch, app_config | dashboard_publisher (formatting helpers) | +| `telegram_bot.py` | Owner-locked bot: candle-close scan loops, alerts, breach monitor, heads-ups, `/scan /status /news /autoscan /cache /clearcache /help /start` | pair_signals, signal_aggregator, providers, agent_panel, violent_watch, news_engine, llm_client, signal_log, app_config | nobody (a top-level process) | +| `dashboard_publisher.py` | Static web publisher daemon; writes JSON + HTML into the Apache folder. Its own process with its own news caches and its own DeepSeek daily cap | pair_signals, providers, main, agent_panel, violent_watch, news_engine, app_config | nobody. Hard rule: never imports and is never imported by `telegram_bot` | +| `static/dashboard.html`, `.css`, `.js` | The public dashboard page (reads `snapshot.json` / `news.json`) | | copied by the publisher | +| `static/news_summary.php` | The one dynamic endpoint: on-demand DeepSeek summary for a clicked headline, whitelist-gated against `news.json`, own fail-closed daily cap, key read from `/etc/gold-dashboard/deepseek.key` (never repo config) | | installed by the publisher | +| `docs/user_guide.html`, `docs/beginner_guide.html` | Human guides. Keep them in sync with signal behaviour. There are no PDF builds | | | + +### Signal core (the part the backtester replays) + +| File | Role | Uses | Used by | +|---|---|---|---| +| `config.py` | Every tunable value: `TechnicalConfig`, `SignalConfig`, `NewsConfig`, `XauStrategyConfig`, `FilterConfig`, wrapped in `PlatformConfig`; `Timeframe`, `SignalStrength`, `PAIR_INFO` | nothing | everything | +| `pair_signals/xauusd.py` | The config factory: overlay + validated per-timeframe overrides + confirmation map. Its docstring is the evidence log for every override | config, app_config, engine | main, telegram_bot, dashboard_publisher, backtester | +| `pair_signals/__init__.py` | Registry of pair producers (`PAIR_PRODUCERS`) and `produce_signal` | config, signal_aggregator, providers, xauusd | the surfaces, backtester | +| `engine.py` | Indicators, regime, composite score (`TechnicalEngine`) | config | signal_aggregator, backtester, replay_record, pair factory | +| `signal_aggregator.py` | Multi-timeframe merge, penalties, grade, gate chain, setup generators, reasons; `analyze_pair` (live) and `analyze_from_dataframes` (replayable core) | engine, chart_patterns, news_engine, providers, signal_log, config | main, telegram_bot, backtester, pair_signals, analyze_signal_log (agent_panel only for type hints) | +| `chart_patterns.py` | The XAU_CP pattern detector | | signal_aggregator | +| `position_sizing.py` | Decision-time feature builder (65 columns) and the reserved stake multiplier (always 1.0). Bounded lookbacks keep live == backtest | config | backtester (`--export-features`) | + +### Data providers + +| File | Role | +|---|---| +| `providers/base.py` | `BaseDataProvider`: cache with candle-duration TTL, forming-candle drop, validation and cleaning, freshness check, weekend and settlement-pause calendar, live fetch depth, paginated `fetch_range` for the backtester, optional tick stream | +| `providers/oanda.py` | Oanda v20 candles (the live feed since 2026-07-28); retry ladder 2 / 5 / 10 s under a 45 s budget | +| `providers/twelve_data.py` | Twelve Data candles + WebSocket quotes; registered fallback, but signal-incompatible with Oanda (see section 4) | +| `providers/tiingo.py` | Tiingo mid-price candles (research only) and the tick stream used by the breach monitor | +| `providers/__init__.py` | `PROVIDER_REGISTRY` and `make_data_provider(config, name=None)` | + +### News and LLM + +| File | Role | +|---|---| +| `news_engine.py` | Calendar + RSS headline sentiment with gold-specific keywords; class-level caches; optional DeepSeek enrichment; display and warnings only | +| `llm_client.py` | Thin `requests` client for DeepSeek (`deepseek-v4-flash`, `thinking: disabled`, one attempt, UTC-day request cap). Every caller has a non-LLM fallback | +| `news_archive.py` | Durable headline corpus in `logs/newsdata.db` (one row per distinct headline, upgrade-only scores, provenance of the scorer). Its own database on purpose: a `signals.db` rotation in 2026-09 destroyed a 3-week corpus of 4,054 headlines and `logs/` is gitignored. Hooked at the end of `news_engine._fetch_news_sentiment_raw`, before the ±0.1 neutral-headline filter, so neutral headlines are kept for training; do not move the hook | +| `agent_panel.py` | Advisory persona panel (macro / technical / risk / contrarian + optional chair). Attached post-hoc; the alert verdict is the macro persona by default. With `verdict_source: "chair"`, `_apply_oppose_guard` downgrades a chair OPPOSE to NEUTRAL unless the macro persona opposes, or technical opposes while macro is not supportive. Persona weights are not to be tuned before about 300 graded outcomes with a held-out half | +| `violent_watch.py` | Display-only violent-move warning and heads-up. Attached post-hoc | + +### Logging and evaluation + +| File | Role | +|---|---| +| `signal_log.py` | SQLite logger for every live scan (`signals`, `news_snapshots`, `scan_outcomes`; view `signals_with_news`); opportunistic scan-outcome labelling; advisory outcome seeding | +| `advisory_outcome.py` | The deterministic outcome model for logged setups | +| `analyze_signal_log.py` | Read-only reports on `signals.db`: trade replay, panel scorecard, parity between surfaces, news lead/lag, scan outcomes (`--stored-candles`, `--news`, `--parity`, `--scan-outcomes`) | +| `scripts/backfill_signal_outcomes.py` | Batch seeding / advancing of advisory outcomes on an imported database (`--dry-run`, `--reseed-unresolvable`) | +| `scripts/label_scan_outcomes.py` | Batch labelling of scan outcomes for a backlog (`--dry-run`) | + +### Backtesting and research + +| File | Role | +|---|---| +| `backtest/gpu_backtest.py` | The backtester (CuPy on a GPU, NumPy otherwise; check the `Backend:` log line) | +| `backtest/replay_record.py` | Opt-in `--record` recorder: why every accepted trade was taken, plus calendar events. Book-neutral by contract | +| `backtest/replay_scorecard.py` | Read-only reader of recorded runs (`DIR --check` verifies its invariants) | +| `scripts/ff_calendar_scraper.py` | Builds `backtest/cache/calendar.parquet` (Forex Factory events) for the recorder | +| `backtest/instructions_for_backtest.txt` | What every backtest report must record. Follow it | +| `backtest/research/*.md` | The decision ledgers (one per campaign). Rule-1 evidence | +| `backtest/cache/` (gitignored) | Candle caches keyed by provider, symbol, timeframe and padded window; only the Oanda windows listed in section 9 are kept | + +### Configuration and deployment + +| File | Role | +|---|---| +| `app_config.py` | Loads `config.json` once per process; env var → config.json → refuse; validates keys | +| `config.json.example` | The schema of `config.json` (which is gitignored) | +| `deploy/README-dashboard.md`, `deploy/gold-dashboard.service` | Live-server checklist and systemd unit for the dashboard daemon | +| `test_all.py` | The QA suite (33 modules) | + +--- + +## 4. Data providers + +All candle data goes through `providers/base.py`. Whatever the source, the pipeline sees the same +thing: a UTC, oldest-first DataFrame with capitalised OHLCV columns, closed bars only. The base +class owns the behaviour the signals depend on; a provider only implements `_fetch_raw` plus its +symbol and interval maps. A test enforces that subclasses do not override the shared methods +(`supports_streaming` and `iter_ticks` are the sanctioned streaming hooks). + +**Which provider does what today** + +| Provider | Live candles | Tick stream | Notes | +|---|---|---|---| +| Oanda v20 (`data_providers.active: "oanda"`) | yes, since 2026-07-28 | no | Needs `oanda.api_token` and `oanda.base_url`; no account id. No client-side throttle. A legacy `oanda.account_id` key is tolerated and ignored | +| Tiingo (`data_providers.streaming: "tiingo"`) | research only | yes (`wss://api.tiingo.com/fx`, mid-price) | Never show a spread from its free quote feed (two-tier indicative quotes) | +| Twelve Data | registered fallback | yes | About 19% of M5 bars disagree in direction with Oanda and highs sit about $1 too high. On an Oanda outage, stop scanning rather than switch feeds | + +**Behaviour that matters for signals** + +- The forming candle is always dropped. Live signals would repaint otherwise. +- Cleaning is a bad-print filter, not a volatility filter: a bar more than 10σ from the previous + close is removed only when the next close snaps back at least 80% of the way. Real news bars + stay, and the newest bar is always kept. (`OUTLIER_SIGMA`, `OUTLIER_REVERT_FRACTION`.) +- Every live fetch asks for at least 1,000 bars (`MIN_LIVE_BARS`): M5 and M15 5,000, H1 1,460, + H4 and D1 1,000. The 200-bar EMA needs that much history to match the backtester. +- Weekend: New-York anchored, Friday 17:00 to Sunday 18:00 New York time. There are three + independent copies of this rule on the signal path (`providers/base.py`, + `signal_aggregator.py`, and `gpu_backtest._is_xau_blocked_by_calendar`) plus one in + `advisory_outcome.py`. Patch all of them, or none; a Module 21 test sweeps all four hour by hour + across both daylight-saving transitions. +- The daily settlement pause (Monday to Thursday 17:00 New York, 21:00 UTC in summer) counts as a + scheduled closure since 2026-09-13. A "stale data" warning at any other weekday hour is a real + feed stall. +- The backtester fetches history through `fetch_range` (paginated, retries a failed page, logs an + error for interior gaps longer than 3 bars) and stores it under `backtest/cache/`. + +--- + +## 5. Configuration and credentials + +Rule for every credential and deployment setting: **environment variable → `config.json` → refuse +to run.** There are no hard-coded fallbacks. A gitignored `.env` at the repo root is loaded at +import time and behaves like real environment variables. + +`config.json` holds secrets and deployment settings only. Signal logic, weights, thresholds and +gates live in `config.py` and the pair factory, never in `config.json`. Blocks (see +`config.json.example` for the exact shape): + +| Block | What it controls | +|---|---| +| `telegram` | `token`, `chat_id`, `channel_id`, `alert_min_quality` (env `TELEGRAM_TOKEN`, `TELEGRAM_CHAT_ID`, `TELEGRAM_CHANNEL_ID`) | +| `data_providers` | `active` (live candle feed), `streaming` (tick feed), per-provider API keys (env `TWELVE_DATA_API_KEY`, `TIINGO_API_KEY`) | +| `oanda` | `api_token`, `base_url` (env `OANDA_API_TOKEN`, `OANDA_BASE_URL`) | +| `scanning` | `pairs`, `timeframes` | +| `pairs."XAU/USD".blocked_hours_utc` | The base list of blocked hours (default 0, 1, 2, 10, 11, 13, 16, 21) | +| `deepseek` | `api_key`, `model`, `enabled`, `timeout_seconds`, `daily_request_cap` (env `DEEPSEEK_API_KEY`) | +| `agent_panel` | `enabled`, `telegram`, `dashboard`, `min_quality`, `max_panels_per_day`, `reuse_ttl_seconds`, candle payload knobs, `verdict_source` (`macro` or `chair`). All defaults = feature off | +| `violent_watch` | `enabled`, `alert_line`, `headsup`, `max_headsup_per_day`, `headsup_min_level` (env `VIOLENT_WATCH_ENABLED`). Off by default | +| `signal_log` | `enabled`, `path`, `log_blocked_scans` (env `SIGNAL_LOG_ENABLED`, `SIGNAL_LOG_PATH`, `SIGNAL_LOG_BLOCKED_SCANS`) | +| `news_archive` | `enabled` (default true), `path` (env `NEWS_ARCHIVE_ENABLED`, `NEWS_ARCHIVE_PATH`) | +| `log_level` | Logging level | + +A legacy top-level `market_outlook` block is tolerated and ignored with one warning (the feature +was removed 2026-09-15). `FB2_EMA_LO` is a research-only environment knob for the EMA200 proximity +band (default 0.5; an unparseable value falls back to the default with a warning). + +Files the system writes: `logs/signals.db`, `logs/newsdata.db`, `logs/telegram.log` (one file per +day, 30 kept), `logs/dashboard.log`. Read the databases with `mode=ro`, never `immutable=1` +(that mode cannot see rows still in the write-ahead log). + +--- + +## 6. Testing + +- `venv/bin/python test_all.py` runs everything with mocked network. Expect + `RESULTS: 506 passed, 0 failed, 506 total` (2026-09-15). +- Test module numbers are cited from this file, so never renumber them. Module 10 and 26 are gaps. +- When a test needs the weekend rule, patch `providers.base._is_forex_weekend` and remember the + other copies. Signal-log fixtures that exercise candle continuity must use a weekday bar. + `_candles_json` stores the last 14 bars, so a longer fixture silently loses its head. +- A test walks the code with `ast` and forbids `sklearn`, `scipy`, `lightgbm` and `torch` in live + modules and in `requirements.txt`. Research happens in the dev `venv/`; anything shipped is + numpy-only. +- Behaviour-neutral changes are proven by regenerating the reference books (section 9) and + showing the MD5s unchanged. + +--- + +## 7. Rules for changing signal logic + +1. **Never change a tuned value without re-validation.** Acceptance: positive total R in both + halves of a window of at least 12 months, profit factor ≥ 1.3, at realistic cost ($0.65 round + trip headline; also report $0.30 for comparability with older books), trailing exits, next-open + fills. Marginal edges must also pass the 2024-06-01 → 2025-03-30 out-of-sample window. Backtest + windows go back at most 3 years (owner preference, 2026-09-05). `round_trip_cost` stays at 0.30 + because it also drives the live target floor (0.65 would move target 1's floor from about $9 to + $19.50); changing it is its own rule-1 event. +2. **Per-timeframe overrides live in the pair factory**, never in the global defaults. The same + knob has had opposite effects on different timeframes several times. Record the evidence in the + factory docstring. +3. **Pair isolation.** Gold-specific gates are keyed on the pair's display name so a future pair + cannot inherit gold-calibrated logic by accident. +4. **Live equals backtest.** A new live gate goes into the replay path in the same commit. All time + filters use bar-close time in both paths. +5. **ROC stays weighted** even though it correlates with MACD, Keltner and EMA. Removing it + backtested 13R worse; the thresholds were calibrated with it. +6. **The backtester's realism settings are results-critical** (raw-path resolution, spread + charging, next-open entry, stop-first ties, close-only trailing). Touching them breaks + comparability with every recorded book. +7. **The forming candle stays dropped.** +8. Do not add config fields nothing reads. Do not disable indicators in production paths + (`disabled_indicators` is for experiments only; a 0.0 regime multiplier in a per-timeframe + `regime_weights_override` is equivalent for the composite and is the sanctioned vehicle for an + indicator-off candidate). +9. EUR/USD was removed on 2026-07-16 after a no-edge verdict (last commit with it `7d3fd95`). + BTC/ETH work exists only on unmerged branches. Do not re-add either without new data. + +Process rules that go with them: every backtest report follows +`backtest/instructions_for_backtest.txt`; every candidate is compared with a control run on the +same code and cache, never with an old book directly; the `Backend:` line and the "GPU cache +fallbacks: 0" line are part of the evidence; keep `docs/` in sync when behaviour changes. + +--- + +## 8. Validated configuration (what is actually running) + +Defaults in `config.py` are the tuned gold configuration. The factory adds these per-timeframe +overrides (each one has its validation record in the `pair_signals/xauusd.py` docstring): + +| Timeframe | Confirmations | Overrides on top of the defaults | +|---|---|---| +| M5 | M15, H1 | hour 9 UTC blocked and hour 21 unblocked; chart-pattern entries on (buy-side only: inverse head and shoulders, double bottom, bull flag); pullback entries off; partial EMA alignment scores neutral; trailing stop arms at 0.60R and trails 0.85R; shorts blocked in the `transitioning` regime | +| M15 | M5, H1 | target floor 1R; the mixed `trending` regime blocked; shorts blocked at 15:00 and 17:00 UTC; trailing stop arms at 0.70R and trails 0.75R; RSI weight halved in every regime | +| H1 | M15, H4 | every short blocked (`sell_min_quality` empty) | +| H4 | H1, D1 | none. H4 remains "unproven" (few trades) | + +Shared defaults worth knowing: blocked hours 0, 1, 2, 10, 11, 13, 16, 21 UTC; trading days Monday +to Friday plus Sunday evening; shorts need grade A+, A or B and high-or-extreme volatility; +trailing stop arms at 0.6R and trails 0.75R; target floor 0.5R; round-trip cost $0.30 with +target 1 at least 30 × cost; the generic and chart-pattern setups floor the stop at $8; the gold +ATR setup uses a 1 × ATR target before the grade scale (2× at A+, 1.5× at A) and a 2 × ATR stop; the H4 regime filter is off; news never gates. + +One semantic to know: `blocked_hours_utc` is applied to the bar **label** hour inside the core +and to the bar **close** hour in the live wrapper and the backtester. So H1 can only trade bars +closing 04–09, 15 and 18–20 UTC and H4 only bars closing 08:00 or 12:00. Both paths agree, and +every book was validated that way. Do not "fix" one side alone. + +--- + +## 9. Reference books and how to reproduce them + +Pinned trade books are no longer kept in the tree (owner decision 2026-09-15; the old ones sit in +`~/Development/ForexBot2_trash_2026-09-15/` until the owner deletes that folder). The gate is the MD5 of +the exported 63-column CSV (`md5sum` on the whole file). Regenerate a book from the cache when a +trade-level diff is needed; each run takes seconds to a few minutes. + +Harness for every reference book: `venv/bin/python backtest/gpu_backtest.py --pair XAUUSD +--spread 0.65 --workers 12 --primary-tf --start --end --export-csv `, +factory config, no `--set`, `--quality`, `--strategies`, `--confirm-tfs` or `--export-features`. +All of these run offline from the kept cache windows (NumPy CPU backend, 0 cache fallbacks). + +| Book | Window | Trades / R / PF | MD5 of the CSV | +|---|---|---|---| +| M5 W2 | 2024-09-05 → 2026-09-05 | 781 / +146.07 / 1.52 | `29120a1ee32979431d062f210d41a9d1` | +| M15 W2 | 2024-09-05 → 2026-09-05 | 931 / +146.14 / 1.43 | `d185f099482a07f29502cb5a371a9bbf` | +| M15 OOS | 2024-06-01 → 2025-03-30 | 355 / +60.67 / 1.49 | `6f8f24b49597ced28f50a19e08c3472f` | +| H1 W3 | 2023-09-05 → 2026-09-05 | 253 / +70.0 / 1.98 | `3b45d12bade823178b267c13bc698458` | +| H4 W3 | 2023-09-05 → 2026-09-05 | 114 / +21.7 / 1.59 | `e75adae13c839b15736f69c0114e6aa5` | + +Other current-HEAD figures at $0.65: M5 W3 (3-year) 870 / +153.6R / PF 1.50; M15 W3 1273 / ++186.6R / PF 1.39; M5 12-month (2025-08-14 → 2026-08-14) 560 / +108.81R / PF 1.54; +H1 12-month (2025-08-04 → 2026-08-04) 91 / +22.2R / PF 1.79. All five MD5s above were +reproduced on 2026-09-15 after the cache prune. + +Kept cache windows (`backtest/cache/oanda_XAU_USD___.parquet`; the padded +start is the window start minus 62 days for M5/M15/H1, 234 for H4 and 1,400 for D1, so the D1 +and H4 files of one run have different start dates than its M5 file): 3-year `20230705_20260905`, +2-year `20240705_20260905`, OOS `20240331_20250330`, M5 12-month `20250613_20260814`, M15/H1 +12-month `20250603_20260804`, 5-year `20210531_20260804`, 7.6-year `20181031_20260807`, the +2026-08-14 parity window `20250814_20260814`, their D1/H4 pads, and `calendar.parquet`. + +The research recorder (`--record DIR [--record-bars]`) writes Parquet files describing every +accepted trade (indicator votes on all three timeframes, penalties, setup, outcome, calendar +events). It never changes the book; pair every recorded run with a control run and show equal +MD5s. Scorecards from it are in-book statistics, never decisions. + +--- + +## 10. Things that bite + +- Never build a bare `PlatformConfig()` for a real scan; use the factory. A test enforces factory + equivalence with the inline construction the backtester uses. +- The backtester names timeframes `5min`, `15min`, `1h`, `4h`. `M5` is a hard exit. +- `--spread` defaults to 0.30. Pass 0.65 for every headline result. +- `--export-features` changes the CSV; never use it on a parity run. +- The signal log's `spread` column is the median bar range, not a broker spread. `actionable` + means grade ≥ C and a setup exists; the `action` text is written before the gate chain and can + disagree with the final grade. `scan_kind` is `blocked` (weekend or day filter), `failed` (a + required frame was missing) or normal; readers skip the first two. Several surfaces log the + same bar, so readers dedup by the first row per (`timeframe`, `bar_ts`). Same-setup re-logs + within 60 minutes are `DUPLICATE` rows; sum advisory R over `RESOLVED` rows only. +- Headlines and events in the signal log are content-addressed into `news_snapshots`; read + through the `signals_with_news` view or `signal_log.read_news`. +- A non-empty `cleaned_warnings_json` is not a chart-pattern marker (a stale spread-gate veto lands + there too since 2026-09-08); use `setup_json.strategy`. +- The `--stored-candles` scorecard in `analyze_signal_log.py` still advances outcomes without the + locked-profit floor step that the logger and the backfill script use (owner decision pending), so + the two measurement paths can disagree on gap rows. +- There is no holiday awareness anywhere. On Labor Day 2026 Oanda published no bars for hours, the + bot rescanned the stale bar 43 times, and the violent-move watch called the 12–14 UTC slot hot. + Interior gaps in history are logged, never fatal, for the same reason. +- Existing databases still contain an empty-going-forward `outlooks` table from the removed + market-outlook feature. Nothing reads it. +- The news score is a trailing echo of the last day's price move (r ≈ 0.67 with the past day, + ≈ 0.03 with the next). Both news gating flags stay off. +- The agent panel never runs inside `analyze_from_dataframes`, and its cached verdict is never + reused when the evidence changed; the daily cap is the real spend bound. +- Live == backtest also means the backtester never writes to `signals.db` and never builds a + `NewsEngine`. +- Twelve Data candles are not a drop-in replacement for Oanda (section 4). +- `pyarrow` (Parquet) is installed in the dev venv only; live code never touches Parquet. +- The dashboard's due-timeframe check uses the raw wake time; flooring `wake − buffer` skipped + every H1 and H4 scan once (2026-08-02). + +--- + +## 11. Decision ledger + +One line per validated change or verdict, newest first. Full evidence in the named report. + +| Date | Decision | Where the evidence is | +|---|---|---| +| 2026-09-15 | Backtest folder cleanup: `diagnose_filters.py`, `shadow_scan.py`, and the two one-time news scripts removed (last commit with them `931b265`); every pinned CSV, the research output folders, the BTC/ETH cache snapshots, the Tiingo caches and 13 uncited Oanda windows moved to `~/Development/ForexBot2_trash_2026-09-15/` (346 MB) pending the owner's delete; all five reference MD5s reproduced afterwards. Suite 506 / 33 | commit `b0b71e1` | +| 2026-09-15 | AI market-outlook feature removed (module, hourly Telegram brief, `--outlook`, `outlooks` writer, config block). Legacy config block tolerated with a warning. Books byte-identical | commit `931b265` | +| 2026-09-13 | Bug audit, 11 delivery/measurement fixes, books byte-identical: NEUTRAL-signal C/B setups now alert (about 17% of the H1 12-month book's R, +3.7R of +22.2R, was never sent); a missing confirmation frame fails the scan; pandas `utcnow` replaced and `pandas<4`, `numpy<3` pinned; advisory outcomes keep locked-profit floors (`GAP_LOCKED`); scan-outcome labeler bounded to 72 h; settlement pause is a scheduled closure; `fetch_range` retries pages and audits gaps; cache-fallback count logged and must be 0; partial `regime_weights_override` raises; the bot scans each bar once and attaches its log handler in `main()` only; the dashboard catches up missed timeframes. Verified sound and unchanged: composite maths, scan-outcome labels, outcome exits, Oanda alignment, no look-ahead | plan file `~/.claude/plans/you-are-python-software-joyful-teacup.md`, commit `2836024` | +| 2026-09-10 | Improvement cycle 1: 3-year baseline re-established; cost ladder shows no red flag (PF ≥ 1.3 everywhere at $0.80); the intraday edge is a fat right tail (top 5% of trades carry 94–96% of M5/M15 R). H1 divergence-to-zero REJECTED. M15 RSI × 0.5 sits on a stable plateau. SHIPPED: live fetches floor at 1,000 bars and the backtester pads history to match (books trade-identical). Parked cycle-2 candidate: block M15 grade-A SELLs (230 trades, −14.4R, negative in every year and OOS) | `backtest/research/cycle_2026-09-10.md` | +| 2026-09-09 | Weight test on the recorded replays: M15 RSI multiplier × 0.5 in every regime PASSED (2-year +117.6R → +146.1R, OOS +40.6 → +60.7R) and SHIPPED as an M15-only override; all M5 MACD candidates FAILED; M15 EMA × 2 and divergence-off FAILED | `backtest/research/weight_test_2026-09-09.md` | +| 2026-09-09 | Replay recorder shipped (`--record`, `--record-bars`), books byte-identical. First look is descriptive only: Bollinger opposes 94% of M5 entries; divergence-opposed M5 trades did best; bar-level baseline drifts +1.9 ATR/24 h on M5, so adjust for drift before judging BUY votes | `backtest/research/replay_recording_2026-09-09.md` | +| 2026-09-08 | Delivery fixes: stale spread-gate vetoes moved to `cleaned_warnings` (books identical; only the warning text on 20–33% of A-grade alerts changed); alerts require a setup; every send logged with its row id; `telegram.log` rotates daily × 30 | commit history, 2026-09-08 | +| 2026-09-08 | News archive shipped (`logs/newsdata.db`). Do not train a local headline scorer before about 4,000 headlines over 3+ weeks, time-split, headline text only, graded against the golden set (`git show 1cebebf^:backtest/research/llm_news_golden.csv`) | `news_archive.py` docstring, `backtest/research/llm_news_eval_2026-08-06.md` | +| 2026-09-07 | Signal-side outlier filter fixed (it deleted about 36 real M5 news bars a year in the backtester and a different 24 live). Book impact at $0.65, 12 months: M5 +112.2R → +108.8R, M15 +73.9R → +74.6R, H1 unchanged. Oanda retry ladder added; a scan that loses its primary frame is `failed` | CLAUDE.md history, commit 2026-09-07 | +| 2026-09-06 | Stake-sizing campaign FAILED its pre-registered rule on M5 (clean null) and M15 (lift inside the shuffle null). Nothing changing numbers shipped; `--export-features` and the penalty tags remain. M5 stretch measures are positive in both halves (do not propose exhaustion sizing on M5); bursts cluster 12–15 UTC and 01 UTC and do not hurt entries; every open-trade burst protection lost money | `backtest/research/sizing_campaign_2026-09.md` | +| 2026-09-06 | Violent-move watch shipped dark (display only). Hot = high-impact USD release within ±30 min, or a hot clock slot plus a price extreme; flags 8.6% of bars at 3–5× odds. Context readings never raise the level | `violent_watch.py` docstring, sizing campaign addenda | +| 2026-09-06 | Stale-code cleanup: research tooling, Dukascopy study, diagrams and plans deleted (history before `953c5ad` has them); `main.py` M5 confirmations fixed to M15 + H1; dead functions removed | commits of 2026-09-06 | +| 2026-09-04 | News sentiment verdict: the score trails price, it does not lead it. Headline window 4 h, neutral headlines dropped, beginner-language summary shipped; gating flags stay off | `backtest/research/news_sentiment_value_2026-09-04.md` | +| 2026-09-03 | Signal-log campaign shipped two gates: M5 shorts blocked in `transitioning` (+7.9R/12 mo) and every H1 short blocked (5-year +16.7R, OOS +11.5R). Rejected: M15 SELL-B block, Bollinger-off. Parked: divergence-off. Panel verdict switched to the macro persona (owner decision); the chair's OPPOSE was anti-informative (opposed trades made +30R) | `backtest/research/signal_log_campaign_2026-09.md`, `signal_log_eval_2026-08-18.md` | +| 2026-09-03 | Auto-execution (Oanda daemon and executor) removed; the owner does not auto-trade (last commit with it `31002cb`) | commit `31002cb` | +| 2026-08-15 | Agent-panel technical persona gets ATR-normalised candles and raw indicator values | `backtest/research/ai_signal_enrichment_2026-08-15.md` | +| 2026-08-14 | Books re-pinned on Oanda bars; Twelve Data books superseded. Feed divergence and Oanda-Tiingo signal parity measured | `repin_verification_2026-08-14.md`, `feed_divergence_2026-08-14.md`, `signal_parity_2026-08-14.md` | +| 2026-08-06 | DeepSeek headline scoring validated (78/78 vs 24/78 for keywords); event misreads fixed with the `vs_forecast` field | `backtest/research/llm_news_eval_2026-08-06.md` | +| 2026-08-05 | Fixed-RR hard target vs trailing exit: NOT shipped (passes the letter of rule 1 on 2024+ windows but is a trend artefact; M15 negative in 2021 and 2023, drawdown doubles). `--fixed-rr-tp` stays as a research flag | `backtest/research/fixedtp_2026-08-05.md` | +| 2026-08-04 | QA audit. Weekend calendar made New-York anchored (removed one live-impossible phantom trade). Spread inside exit-trigger detection: owner DECLINED (4–6% optimism accepted, books stay comparable). `FB2_EMA_LO` hardened. M5 pullback entries disabled (PF 1.26 → 1.44 at $0.65). Partial EMA alignment set neutral on M5. XAU_CP gate bypass documented as validated behaviour (226 of 586 M5 trades, +38.9R) | memory files `qa-audit-2026-08-04`, `m5-tuning-2026-08-04`, `m5-round2-2026-08-04` | +| 2026-07-30 | M5 relaxation shipped (score 25, confidence 5, hour 21 unblocked); big-candle gate rejected (ships off) | memory `relaxation-campaign-2026-07-30` | +| 2026-07-28 | Oanda v20 becomes the live feed | commit `d2d4fb4` | +| 2026-07-24 | Real gold spread measured at about $0.65 round trip (9M Dukascopy ticks, 12 months); $0.30 kept only for cross-book comparability and the live target floor | memory `dukascopy-spread-study` | +| 2026-07-21 | M15 trailing arm 0.70 and M5 trailing distance 0.85 adopted | memory `m15-trail-grid-2026-07-21` | +| 2026-07-16 | M15 SELL-hour block (15 and 17 UTC) adopted; EUR/USD removed (no edge) | memory `xau-m15-loss-analysis`, `eur-no-edge-verdict` | +| 2026-07-11 | M15: mixed `trending` regime blocked, target floor 1R; M5: hour 9 blocked | pair factory docstring | +| 2026-07-08 | Backtester realism fixes (spread charging, gap fills, next-open entry, stop-first ties) | memory `backtester-realism-fixes` | + +Older memory notes live in `~/.claude/projects/-home-harwinder-Development-ForexBot2/memory/`. +The previous, longer version of this file is in git history before 2026-09-15.