ForexBot2/CLAUDE.md

# ForexBot2 — developer guide

ForexBot2 is a signal-intelligence system for one market: **gold against the US dollar (XAU/USD)**.
It does not place trades. Every few minutes it downloads price candles, scores them with nine
technical indicators, checks that several timeframes agree, grades the result (A+ down to F), and
turns a good grade into a written trade plan (entry, stop loss, two targets, trailing rule). The
plan is delivered to a Telegram bot, a command-line report, and a public web dashboard. A
backtester replays history through the *same* signal code, so a backtest result is what the live
scanner would have said on that day.

This file is the map of the system. Read section 2 first if you are new. Section 7 lists the rules
you must follow before changing anything that affects signals. Section 10 lists the traps. Section
11 is the history of every validated change. The research reports behind those decisions live in `backtest/research/*.md`.

---

## 1. Quick start

```bash
python -m venv venv && venv/bin/pip install -r requirements.txt   # Python 3.12; a venv/ already exists on the dev box
cp config.json.example config.json                                 # then fill in the keys (see section 5)

venv/bin/pip install cupy-cuda12x "nvidia-cuda-runtime-cu12==12.9.*"   # optional, dev box: backtester GPU backend
                                                                   #   (CuPy from pip needs that header wheel to compile kernels)
venv/bin/python test_all.py                                        # QA suite, no network; prints its own count
                                                                   #   (552 tests / 36 modules as of 2026-09-18)
venv/bin/python main.py --pair XAUUSD --timeframe H1 --detail      # one signal report in the terminal
                                                                   #   --json full dump, --panel adds the advisory LLM panel,
                                                                   #   --timeframe M5 | M15 | H1 | H4
venv/bin/python telegram_bot.py                                    # the bot (refuses to start without credentials)
venv/bin/python dashboard_publisher.py --once                      # one scan + publish to $DASHBOARD_OUT_DIR
                                                                   #   (without --once it runs as a candle-close daemon)

# Backtests (timeframe names are 5min / 15min / 1h / 4h, not M5/M15)
venv/bin/python backtest/gpu_backtest.py --pair XAUUSD --days 365
venv/bin/python backtest/gpu_backtest.py --pair XAUUSD --start 2024-09-05 --end 2026-09-05 \
    --primary-tf 15min --spread 0.65 --workers 12 --export-csv book.csv
venv/bin/python backtest/cudaBasedBacktesting.py --pair XAUUSD --start 2024-09-05 --end 2026-09-05 \
    --primary-tf 15min --spread 0.65 --export-csv book.csv   # research twin: same flags, every stage on
                                                            #   the GPU, seconds per book, same MD5 (section 9)
```

Backtest flags worth knowing: `--spread` (round-trip cost in dollars; the default is 0.30, the
realistic Oanda figure is 0.65 and must be passed explicitly), `--quality A+,A,B,C`,
`--strategies`, `--workers`, `--max-hold-bars`, `--warmup-bars`, `--cooldown-bars`, `--no-cache`,
`--export-csv out.csv`, `--detail`, `--set PATH=JSON` (research override of any config value; an
unknown path is a hard exit), `--export-features` (appends the 65 decision-time columns; never use
it on a parity run because it changes the file), `--fixed-rr-tp N` (research only, not shipped),
`--record DIR [--record-bars]` (research recorder, see section 9), `--backend auto|gpu|cpu` (`auto`
uses the CuPy GPU backend when CUDA works and falls back to NumPy with the reason in the `Backend:`
line; `gpu` refuses to fall back; `cpu` forces the NumPy twins for a parity run; the env var
`FB2_BACKEND` sets the default). Both backends produce byte-identical books (section 9).

`test_all.py` is **not pytest**. It uses its own `@test("name")` decorator that runs each test the
moment the file is imported. Run the whole file; it prints per-module results and exits non-zero
on any failure. There is no way to run one test on its own. Tests must stay green, and every new
gate or setting needs a test.

---

## 2. How a scan works, from A to Z

A "scan" is one analysis of one timeframe. The bot runs one scan per timeframe at every candle
close; the CLI and the dashboard run the same scan on demand. Everything below happens inside one
scan, in this order.

**1. Trigger.** `telegram_bot.py` wakes 10 seconds after each candle boundary (M5 every 5 min,
M15 every 15 min, H1 hourly, H4 every 4 hours). `dashboard_publisher.py` has its own copy of the
same schedule. `main.py` runs once when you call it.

**2. Build the configuration.** `pair_signals/xauusd.py` (`make_config`) creates a
`PlatformConfig` from `config.py`, applies the deployment overlay from `config.json`
(`app_config.py`), then applies the validated per-timeframe overrides (section 8). Never build a
bare `PlatformConfig()` for a real scan; always go through the factory. The confirmation
timeframes come from the factory map: M5 → M15 + H1, M15 → M5 + H1, H1 → M15 + H4, H4 → H1 + D1.

**3. Fetch candles.** `signal_aggregator.analyze_pair` (the live entry point) first checks the
calendar: on the forex weekend or a blocked weekday it returns a "blocked" scan and does nothing
else. Otherwise it asks the active data provider (`providers/`, Oanda today) for the primary
timeframe and each confirmation timeframe. The provider base class drops the still-forming candle,
validates and cleans the bars, caches them until the next candle close, and checks freshness. If
any required frame cannot be fetched after the retry ladder, the scan is logged as "failed" and no
analysis runs.

**4. Score each timeframe.** `engine.py` (`TechnicalEngine.analyze`) computes ATR first, then nine
scored indicators: EMA trend (9/21/50/200 ribbon), RSI, MACD, Bollinger bands, ADX, rate of change,
Keltner channel, divergence, and candlestick patterns. Each indicator votes on a five-step scale
(STRONG SELL, SELL, NEUTRAL, BUY, STRONG BUY) with a confidence. The engine classifies the market
regime from ADX and the EMA ribbon: `trending_up`, `trending_down`, `trending` (strong trend,
mixed direction), `transitioning` (ADX rising fast), or `ranging`. Each regime has its own table of
per-indicator weight **multipliers** (`regime_weight_table` in `engine.py` is the single source of
truth); the effective weight is the indicator's base weight × the regime multiplier. The composite
score is the confidence-weighted average of the votes (vote value × effective weight × confidence),
scaled by 50 and clipped to −100…+100. Divergence and candlestick only vote when a pattern exists;
a missing vote is "no opinion", not zero. Three engine conventions matter when editing: MACD is
always computed even when it is not a scored pillar, because divergence needs its histogram; RSI is
reinterpreted per regime after classification (`rsi_mode = "regime_aware"`; treating the mixed
`trending` regime as an uptrend was a −6.8R bias); and private helper series ride on the result as
`_underscore` attributes, which every scorer and formatter skips along with weight-0 indicators, so
a new indicator must follow that convention.

**5. Multi-timeframe agreement.** `analyze_from_dataframes` compares the primary timeframe's
direction with the two confirmation timeframes. Agreement needs 2 of 3. Disagreement costs a
penalty and blocks the top grades.

**6. News context (warning only).** `news_engine.py` reads the Forex Factory calendar (MyFXBook
RSS as fallback) and four RSS feeds, scores headlines for gold, and can ask DeepSeek to improve
the scores. This produces warnings and a text summary. **By design it never changes a signal.**
The backtester runs with news off, so the live decision must be purely technical too.

**7. Penalties, then the signal and the grade.** When the timeframes agree, confidence gains 10
points and the score is blended 60/40 with the mean of all timeframe scores. Then a list of
penalties (opposing timeframes, stretched price, fading momentum, and so on) is subtracted from the
**confidence** with diminishing weight: each further penalty counts half, and the total is capped
at 30 points. Penalties never reduce the score itself. If two or more "exhaustion" penalties fire
together with fading momentum, the trade is hard-blocked (confidence set to 0). Then:

| Result | Rule |
|---|---|
| STRONG BUY / STRONG SELL | score ≥ 35 (≤ −35) and confidence ≥ 55 |
| BUY / SELL | score ≥ 25 (≤ −25) and confidence ≥ 50 |
| NEUTRAL | anything else |
| Grade A+ | abs score ≥ 40, confidence ≥ 65, timeframes agree |
| Grade A | abs score ≥ 33, confidence ≥ 55, timeframes agree |
| Grade B | abs score ≥ 28, confidence ≥ 50 |
| Grade C | abs score ≥ 25, confidence ≥ 45 |
| Grade D | abs score ≥ 15 |
| Grade F | below that |

**8. The gate chain.** A series of checks can lower the grade or remove the setup. They run in this
order (`signal_aggregator.py`, steps 6b to 6i): session and day filter (the bar-label hour and weekday
against the blocked hours and allowed weekdays; the live wrapper and the backtester apply the same
list to the bar-close time as well, see section 8); transitioning-regime boost (a B/C/D can become A or B when a
trend is emerging); exhausted-trend cap (in a trending regime with two or more exhaustion tags, a C or D is pinned
to D and loses its setup); SELL quality gate
(shorts need a grade in `sell_min_quality`; H1's list is empty, so every H1 short is blocked);
multi-timeframe opposition gate; SELL volatility gate (shorts are allowed only when volatility is high or extreme; low- or
normal-volatility shorts are blocked);
regime blocks (M15 blocks the mixed `trending` regime; M5 blocks shorts in `transitioning`);
SELL-hour gate (M15 blocks shorts at 15:00 and 17:00 UTC); big-candle gate (shipped off); and on
M5 six extra noise gates (minimum score, required agreement, confirmation strength, SELL grade
cap, chop-transition volatility, SELL RSI floor). Every gate that removes a setup writes a warning
that names itself.

**9. Trade setup.** Grades A+ and A get the gold ATR setup (`_generate_xau_atr_trade_setup`): stop
2 × ATR; target 1 at 2 × ATR for A+ and 1.5 × ATR for A (1 × ATR for an A carrying two or more
exhaustion tags); target 2 at 1.5 × the target-1 distance. Other tradeable grades get the generic
swing-based setup, which floors the stop at $8 (80 pips). Both are checked against the spread gate:
target 1 must be at least 30 × the round-trip cost ($9 at the $0.30 cost setting). Two extra entry paths run after the
gates: the M5 pullback (`XAU_PB`, shipped off) and the chart-pattern entry (`XAU_CP`,
`chart_patterns.py`: M5 only, buy-side patterns inverse head and shoulders, double bottom, bull
flag). Both late paths run after the gate chain and force the grade to B, so an F-composite
chart-pattern trade shows as a B in books and alerts. The chart-pattern path deliberately accepts
weak composites and re-checks the session, day and big-candle rules, the multi-timeframe-opposition
rule and the ATR floor, plus its own SELL, volatility and RSI checks; it does not re-check
exhaustion or the M5 minimum-score and confirmation gates. That is validated behaviour, not a bug
(section 11, 2026-08-04). A shipped setup always carries a BUY or SELL direction, even
when the raw signal was NEUTRAL at grade C (confidence 45–50): the direction is the sign of the
score. Every setup is stamped with its strategy name and its trailing parameters. Finally, a stale
"costs would eat the edge" veto from an earlier generator is moved to `cleaned_warnings` when a
later generator shipped a setup.

**10. Delivery.** The `SignalReport` goes to the surface that asked for it:

- *Telegram* (`telegram_bot.py`): an alert is sent only when the signal is not NEUTRAL, the grade
  reaches `alert_min_quality`, **and** a setup exists. A fingerprint (signal + direction) stops
  repeats; a grade upgrade re-alerts. The fingerprint is committed only after a successful send,
  so a failed send stays eligible next scan. Every send is logged with the signal-log row id. The
  level-breach monitor reuses the bar's cached M15 report (`_cached_or_scan`) instead of rescanning.
- *CLI* (`main.py`): prints the report; `--json` dumps everything.
- *Dashboard* (`dashboard_publisher.py`): writes `snapshot.json`, `news.json` and the static page
  into the Apache folder.

Two advisory add-ons are attached **after** the scan by the surface, never inside the signal
code: the LLM agent panel (`agent_panel.py`; since 2026-09-17 only the **macro** persona runs by
default, `agent_panel.personas: ["macro"]`, one DeepSeek call per panel — the technical, risk and
contrarian personas carried no information in two live evaluations and stay available for research;
the persona returns a probability that the trade closes profitably, anchored to the setup class's
base rate from the validated books, shown as the "AI estimate"; with several personas configured
the estimate is their equal-weight mean, `verdict_source: "mean"`; the older `macro` stance and
`chair` synthesizer paths remain for comparison) and the violent-move
watch (`violent_watch.py`, a "the next hour may be wild" line). No gate reads either of them. The
live wrapper `analyze_pair` hands the fetched frames to the add-ons through a private
`report._frames` attribute so the panel's price context sees exactly the bars the engine saw; the
replayable core never sets it.

**11. Logging.** `signal_log.py` writes one row per live scan (from the bot and the CLI; the
dashboard process never writes either database, section 3) into `logs/signals.db` (signal,
grade, price, indicators, the last 14 candles per timeframe, the news at scan time, the setup, the
gate that removed a setup, a gate-config fingerprint and an engine fingerprint). The engine
fingerprint hashes the scoring config, so it changes whenever a weight or threshold ships; signal-log
comparisons across such a deploy must be split at the deploy time (the 2026-09-09 M15 RSI change is
one). `news_archive.py` writes every headline and
its score into `logs/newsdata.db`. Both are fail-safe: a logging error can never break a scan.

**12. Measurement, later.** Two models read the stored candles and score what happened next:
advisory outcomes (`advisory_outcome.py`: replays each logged setup with next-open entry, stop
first, close-only trailing; a cleaned-mid evidence model, not execution P&L) and scan outcomes
(model-free forward moves 1 h / 4 h / 24 h after every scan). `analyze_signal_log.py` turns the
database into reports.

**The backtester is the same pipeline on history.** `backtest/gpu_backtest.py` loads candles from
`backtest/cache/*.parquet` (fetched once through the same provider code), precomputes every
indicator series over the whole window, and calls `analyze_from_dataframes` bar by bar with news
off. Trades are resolved on the raw price path with next-open entries, spread charged per trade,
stop-first ties, gap fills at the open, and the trailing stop moved at bar close only. Time filters
use bar-close time in both paths, so live and backtest agree. The replay fans out over worker
processes; the 5-bar entry cooldown, the only cross-bar state, is applied in an ordered pass
afterwards, which is why parallel and sequential books are identical.

---

## 3. File-by-file guide

Arrows show who imports whom (generated from the code on 2026-09-15).

### Entry points and surfaces

| File | Role | Uses | Used by |
|---|---|---|---|
| `main.py` | Command-line signal report (`--detail`, `--json`, `--panel`) | pair_signals, signal_aggregator, agent_panel, violent_watch, app_config | dashboard_publisher (formatting helpers) |
| `telegram_bot.py` | Owner-locked bot: candle-close scan loops, alerts, breach monitor, heads-ups, `/scan /status /news /autoscan /cache /clearcache /help /start` | pair_signals, signal_aggregator, providers, agent_panel, violent_watch, news_engine, llm_client, signal_log, app_config | nobody (a top-level process) |
| `dashboard_publisher.py` | Static web publisher daemon; writes JSON + HTML into the Apache folder. Its own process with its own news caches and its own DeepSeek daily cap. Never writes `signals.db` or `newsdata.db`: `main()` calls `app_config.set_db_writes_for_process(False)`, a process-wide switch every SQLite writer honours (owner decision 2026-09-17; the bot is the single writer because the dashboard is not always running) | pair_signals, providers, main, agent_panel, violent_watch, news_engine, app_config | nobody. Hard rule: never imports and is never imported by `telegram_bot` (test-enforced) |
| `static/dashboard.html`, `.css`, `.js` | The public dashboard page (reads `snapshot.json` / `news.json`). Bootstrap 5 + jQuery + Chart.js + Font Awesome straight from a CDN, no build step; M5/M15/H1/H4 each get a card in one column grid, with a diverging Chart.js bar chart for direction and one per card for the indicator votes | | copied by the publisher |
| `static/news_summary.php` | The one dynamic endpoint: on-demand DeepSeek summary for a clicked headline, whitelist-gated against `news.json`, own fail-closed daily cap, key read from `/etc/gold-dashboard/deepseek.key` (never repo config) | | installed by the publisher |
| `docs/user_guide.html`, `docs/beginner_guide.html` | Human guides. Keep them in sync with signal behaviour. There are no PDF builds | | |

### Signal core (the part the backtester replays)

| File | Role | Uses | Used by |
|---|---|---|---|
| `config.py` | Every tunable value: `TechnicalConfig`, `SignalConfig`, `NewsConfig`, `XauStrategyConfig`, `FilterConfig`, wrapped in `PlatformConfig`; `Timeframe`, `SignalStrength`, `PAIR_INFO` | nothing | everything |
| `pair_signals/xauusd.py` | The config factory: overlay + validated per-timeframe overrides + confirmation map. Its docstring is the evidence log for every override | config, app_config, engine | main, telegram_bot, dashboard_publisher, backtester |
| `pair_signals/__init__.py` | Registry of pair producers (`PAIR_PRODUCERS`) and `produce_signal` | config, signal_aggregator, providers, xauusd | the surfaces, backtester |
| `engine.py` | Indicators, regime, composite score (`TechnicalEngine`) | config | signal_aggregator, backtester, replay_record, pair factory |
| `signal_aggregator.py` | Multi-timeframe merge, penalties, grade, gate chain, setup generators, reasons; `analyze_pair` (live) and `analyze_from_dataframes` (replayable core) | engine, chart_patterns, news_engine, providers, signal_log, config | main, telegram_bot, backtester, pair_signals, analyze_signal_log (agent_panel only for type hints) |
| `chart_patterns.py` | The XAU_CP pattern detector | | signal_aggregator |
| `position_sizing.py` | Decision-time feature builder (65 columns) and the reserved stake multiplier (always 1.0). Bounded lookbacks keep live == backtest | config | backtester (`--export-features`) |

### Data providers

| File | Role |
|---|---|
| `providers/base.py` | `BaseDataProvider`: cache with candle-duration TTL, forming-candle drop, validation and cleaning, freshness check, weekend and settlement-pause calendar, live fetch depth, paginated `fetch_range` for the backtester, optional tick stream |
| `providers/oanda.py` | Oanda v20 candles (the live feed since 2026-07-28); retry ladder 2 / 5 / 10 s under a 45 s budget |
| `providers/twelve_data.py` | Twelve Data candles + WebSocket quotes; registered fallback, but signal-incompatible with Oanda (see section 4) |
| `providers/tiingo.py` | Tiingo mid-price candles (research only) and the tick stream used by the breach monitor |
| `providers/__init__.py` | `PROVIDER_REGISTRY` and `make_data_provider(config, name=None)` |

### News and LLM

| File | Role |
|---|---|
| `news_engine.py` | Calendar + RSS headline sentiment with gold-specific keywords; class-level caches; optional DeepSeek enrichment; display and warnings only |
| `llm_client.py` | Thin `requests` client for DeepSeek (`deepseek-v4-flash`, `thinking: disabled`, one attempt, UTC-day request cap). Every caller has a non-LLM fallback |
| `news_archive.py` | Durable headline corpus in `logs/newsdata.db` (one row per distinct headline, upgrade-only scores, provenance of the scorer). Its own database on purpose: a `signals.db` rotation in 2026-09 destroyed a 3-week corpus of 4,054 headlines and `logs/` is gitignored. Hooked at the end of `news_engine._fetch_news_sentiment_raw`, before the ±0.1 neutral-headline filter, so neutral headlines are kept for training; do not move the hook. Since 2026-09-17 the corpus is complete for a gold scorer: `xau_score` holds DeepSeek's gold score for every LLM-scored row (it used to survive only for gold-keyword headlines, so 930 of the first 1,546 rows had no gold label; older rows read NULL), headlines naming no tracked currency are scored and archived too (`currencies_json` `[]`; they are never returned to the platform), and every new headline goes to DeepSeek in the cycle it arrives, 25 per call |
| `agent_panel.py` | Advisory persona panel (macro by default; technical / risk / contrarian and the chair available through `agent_panel.personas` for research). Attached post-hoc. v2 (2026-09-17, `PANEL_PROMPT_VERSION`): every persona sees a code-computed `context` block (prior move in ATR, distance from the 24 h extreme, candle size, stop/target in ATR, the class base rate from `agent_panel_base_rates.json`, event timing against the expected hold) and returns `p_win` plus one reason each way; stance is derived symmetrically (≥ 60 support, ≤ 40 oppose); the verdict is the equal-weight mean (quorum 3, `verdict_source: "mean"`, default). `macro` (stance verbatim) and `chair` (synthesizer + `_apply_oppose_guard`) remain for comparison. The prompt version is logged and hashed. Persona weights are not to be tuned before about 300 graded outcomes with a held-out half |
| `violent_watch.py` | Display-only violent-move warning and heads-up. Attached post-hoc |

### Logging and evaluation

| File | Role |
|---|---|
| `signal_log.py` | SQLite logger for every live scan (`signals`, `news_snapshots`, `scan_outcomes`; view `signals_with_news`); opportunistic scan-outcome labelling; advisory outcome seeding |
| `advisory_outcome.py` | The deterministic outcome model for logged setups |
| `analyze_signal_log.py` | Read-only reports on `signals.db`: trade replay, panel scorecard, parity between surfaces, news lead/lag, scan outcomes (`--stored-candles`, `--news`, `--parity`, `--scan-outcomes`) |
| `scripts/backfill_signal_outcomes.py` | Batch seeding / advancing of advisory outcomes on an imported database (`--dry-run`, `--reseed-unresolvable`) |
| `scripts/label_scan_outcomes.py` | Batch labelling of scan outcomes for a backlog (`--dry-run`) |
| `scripts/panel_replay.py` | Offline replay of the agent panel on logged, resolved setups: rebuilds each persona's payload (exact `panel_payloads_json` when present, else from the stored JSON blobs, interned news and the kept Oanda cache), runs the CURRENT prompts, and scores Brier / calibration / constant-voter rate / counterfactual books against the logged v1 takes and the base-rate predictor. `--dry-run`, `--max-calls`, `--limit`, `--out`. Read-only on the DB; 4 DeepSeek calls per event |
| `scripts/build_panel_base_rates.py` | Builds `agent_panel_base_rates.json` (per timeframe × strategy × grade × direction: n, win rate, mean R, TP1 reach, median hold, book MD5) from the exported reference books. Re-run after any re-pin |
| `agent_panel_base_rates.json` | The base-rate table the panel anchors its probabilities to (built 2026-09-17 from the relabelled W2/W3 books) |

### Backtesting and research

| File | Role |
|---|---|
| `backtest/gpu_backtest.py` | The backtester. The indicator precompute over the full history and the resolution of every trade run on the GPU when CUDA works (`--backend`, `Backend:` log line names the device or the fallback reason); the per-bar replay through `analyze_from_dataframes` is the live Python signal code in forked CPU workers by design and is 80–95% of wall time |
| `backtest/gpu_kernels.py` | CuPy backend state (lazy probe, never silent, sets `CUDA_PATH` to the pip header wheel) and the three CUDA kernels: EWM batch, pandas rolling mean/var replica, trade resolver. Each is a bit-for-bit twin of the NumPy/pandas code kept in `gpu_backtest.py` (`--fmad=false`, one thread per series or trade); the twins are pinned by Module 36 and by the reference MD5s |
| `backtest/cudaBasedBacktesting.py` | Research-only full-GPU twin of the backtester (2026-09-18): same CLI, same cache files, same 63-column book byte for byte, but the per-bar decision runs in CUDA kernels instead of the live Python code, so a book takes seconds (M5 two-year 2.4 s against 115 s). Reuses the loader, the precompute, the resolver, the metrics and the writers from `gpu_backtest.py`; the host only turns the kernels' numeric codes into the exact strings. No CPU path: `--backend cpu` is refused. Its numbers count as evidence only while the MD5 parity with `gpu_backtest.py` holds (section 7) |
| `backtest/cuda_signal_kernels.py` | The CUDA decision stage: K1 (`engine.py` plus the `GPUTechnicalEngine` overrides, one thread per bar), K2 (`analyze_from_dataframes` end to end with the gates, the setup generators, XAU_PB, XAU_CP, the regime filter and the replay's per-bar rules, one thread per primary bar), K3 (`position_sizing.build_features`), K4 (cooldown), K5 (prefix median), K6 (recorder bar rows); the config packer and the name lists the `#define`s are generated from. A leaf module that never imports the live code; pinned by Module 37 |
| `backtest/replay_record.py` | Opt-in `--record` recorder: why every accepted trade was taken, plus calendar events. Book-neutral by contract |
| `backtest/replay_scorecard.py` | Read-only reader of recorded runs (`DIR --check` verifies its invariants) |
| `scripts/ff_calendar_scraper.py` | Builds `backtest/cache/calendar.parquet` (Forex Factory events) for the recorder |
| `backtest/instructions_for_backtest.txt` | What every backtest report must record. Follow it |
| `backtest/research/*.md` | The decision ledgers (one per campaign). Rule-1 evidence |
| `backtest/cache/` (gitignored) | Candle caches keyed by provider, symbol, timeframe and padded window; only the Oanda windows listed in section 9 are kept |

### Configuration and deployment

| File | Role |
|---|---|
| `app_config.py` | Loads `config.json` once per process; env var → config.json → refuse; validates keys |
| `config.json.example` | The schema of `config.json` (which is gitignored) |
| `deploy/README-dashboard.md`, `deploy/gold-dashboard.service` | Live-server checklist and systemd unit for the dashboard daemon |
| `test_all.py` | The QA suite (36 modules; Module 36 is the GPU backend parity module, Module 37 the CUDA decision-stage parity module) |

---

## 4. Data providers

All candle data goes through `providers/base.py`. Whatever the source, the pipeline sees the same
thing: a UTC, oldest-first DataFrame with capitalised OHLCV columns, closed bars only. The base
class owns the behaviour the signals depend on; a provider only implements `_fetch_raw` plus its
symbol and interval maps. A test enforces that subclasses do not override the shared methods
(`supports_streaming` and `iter_ticks` are the sanctioned streaming hooks).

**Which provider does what today**

| Provider | Live candles | Tick stream | Notes |
|---|---|---|---|
| Oanda v20 (`data_providers.active: "oanda"`) | yes, since 2026-07-28 | no | Needs `oanda.api_token` and `oanda.base_url`; no account id. No client-side throttle. A legacy `oanda.account_id` key is tolerated and ignored |
| Tiingo (`data_providers.streaming: "tiingo"`) | research only | yes (`wss://api.tiingo.com/fx`, mid-price) | Never show a spread from its free quote feed (two-tier indicative quotes) |
| Twelve Data | registered fallback | yes | About 19% of M5 bars disagree in direction with Oanda and highs sit about $1 too high. On an Oanda outage, stop scanning rather than switch feeds |

**Behaviour that matters for signals**

- The forming candle is always dropped. Live signals would repaint otherwise.
- Cleaning is a bad-print filter, not a volatility filter: a bar more than 10σ from the previous
  close is removed only when the next close snaps back at least 80% of the way. Real news bars
  stay, and the newest bar is always kept. (`OUTLIER_SIGMA`, `OUTLIER_REVERT_FRACTION`.)
- Every live fetch asks for at least 1,000 bars (`MIN_LIVE_BARS`): M5 and M15 5,000, H1 1,460,
  H4 and D1 1,000. The 200-bar EMA needs that much history to match the backtester.
- Weekend: New-York anchored, Friday 17:00 to Sunday 18:00 New York time. There are three
  independent copies of this rule on the signal path (`providers/base.py`,
  `signal_aggregator.py`, and `gpu_backtest._is_xau_blocked_by_calendar`) plus one in
  `advisory_outcome.py`. Patch all of them, or none; a Module 21 test sweeps all four hour by hour
  across both daylight-saving transitions.
- The daily settlement pause (Monday to Thursday 17:00 New York, 21:00 UTC in summer) counts as a
  scheduled closure since 2026-09-13. A "stale data" warning at any other weekday hour is a real
  feed stall.
- The backtester fetches history through `fetch_range` (paginated, retries a failed page, logs an
  error for interior gaps longer than 3 bars) and stores it under `backtest/cache/`.

---

## 5. Configuration and credentials

Rule for every credential and deployment setting: **environment variable → `config.json` → refuse
to run.** There are no hard-coded fallbacks. A gitignored `.env` at the repo root is loaded at
import time and behaves like real environment variables.

`config.json` holds secrets and deployment settings only. Signal logic, weights, thresholds and
gates live in `config.py` and the pair factory, never in `config.json`. Blocks (see
`config.json.example` for the exact shape):

| Block | What it controls |
|---|---|
| `telegram` | `token`, `chat_id`, `channel_id`, `alert_min_quality` (env `TELEGRAM_TOKEN`, `TELEGRAM_CHAT_ID`, `TELEGRAM_CHANNEL_ID`) |
| `data_providers` | `active` (live candle feed), `streaming` (tick feed), per-provider API keys (env `TWELVE_DATA_API_KEY`, `TIINGO_API_KEY`) |
| `oanda` | `api_token`, `base_url` (env `OANDA_API_TOKEN`, `OANDA_BASE_URL`) |
| `scanning` | `pairs`, `timeframes` |
| `pairs."XAU/USD".blocked_hours_utc` | The base list of blocked hours (default 0, 1, 2, 10, 11, 13, 16, 21) |
| `deepseek` | `api_key`, `model`, `enabled`, `timeout_seconds`, `daily_request_cap` (env `DEEPSEEK_API_KEY`) |
| `agent_panel` | `enabled`, `telegram`, `dashboard`, `min_quality`, `max_panels_per_day`, `reuse_ttl_seconds`, candle payload knobs, `verdict_source` (`mean`, the default since 2026-09-17; `macro` or `chair` for comparison), `personas` (default `["macro"]`; any of macro / technical / risk / contrarian; `chair` needs two or more). All defaults = feature off |
| `violent_watch` | `enabled`, `alert_line`, `headsup`, `max_headsup_per_day`, `headsup_min_level` (env `VIOLENT_WATCH_ENABLED`). Off by default |
| `signal_log` | `enabled`, `path`, `log_blocked_scans` (env `SIGNAL_LOG_ENABLED`, `SIGNAL_LOG_PATH`, `SIGNAL_LOG_BLOCKED_SCANS`) |
| `news_archive` | `enabled` (default true), `path` (env `NEWS_ARCHIVE_ENABLED`, `NEWS_ARCHIVE_PATH`) |
| `log_level` | Logging level |

A legacy top-level `market_outlook` block is tolerated and ignored with one warning (the feature
was removed 2026-09-15). `FB2_EMA_LO` is a research-only environment knob for the EMA200 proximity
band (default 0.5; an unparseable value falls back to the default with a warning).

Files the system writes: `logs/signals.db`, `logs/newsdata.db`, `logs/telegram.log` (one file per
day, 30 kept), `logs/dashboard.log`. Read the databases with `mode=ro`, never `immutable=1`
(that mode cannot see rows still in the write-ahead log).

---

## 6. Testing

- `venv/bin/python test_all.py` runs everything with mocked network. Expect
  `RESULTS: 552 passed, 0 failed, 552 total` (2026-09-18). On a box with a visible CUDA device
  Module 36 fails if the GPU backend falls back (the reason is in the message); with
  `FB2_BACKEND=cpu` or no CUDA the GPU-path tests print a notice and the NumPy twins are tested.
  Module 37 (the CUDA decision stage) adds about 85 s on the GPU because its references replay
  two synthetic fixtures through the live signal code bar by bar; without CUDA each of its device
  tests prints a notice and only the arithmetic rule, packing, flag and source-guard tests run. The suite's hermetic app config points
  both database writers at a suite-private temp dir; before 2026-09-17 the RSS fixture tests wrote
  fixture headlines into the real `logs/newsdata.db`, and a Module 32 test now pins the isolation. One Twelve Data interval-map test
  reaches the network and can time out on a slow connection; a rerun clears it.
- Test module numbers are cited from this file, so never renumber them. Module 10 and 26 are gaps.
- When a test needs the weekend rule, patch `providers.base._is_forex_weekend` and remember the
  other copies. Signal-log fixtures that exercise candle continuity must use a weekday bar.
  `_candles_json` stores the last 14 bars, so a longer fixture silently loses its head.
- A test walks the code with `ast` and forbids `sklearn`, `scipy`, `lightgbm` and `torch` in live
  modules and in `requirements.txt`. Research happens in the dev `venv/`; anything shipped is
  numpy-only.
- Behaviour-neutral changes are proven by regenerating the reference books (section 9) and
  showing the MD5s unchanged.

---

## 7. Rules for changing signal logic

1. **Never change a tuned value without re-validation.** Acceptance: positive total R in both
   halves of a window of at least 12 months, profit factor ≥ 1.3, at realistic cost ($0.65 round
   trip headline; also report $0.30 for comparability with older books), trailing exits, next-open
   fills. Marginal edges must also pass the 2024-06-01 → 2025-03-30 out-of-sample window. Backtest
   windows go back at most 3 years (owner preference, 2026-09-05). `round_trip_cost` stays at 0.30
   because it also drives the live target floor (0.65 would move target 1's floor from about $9 to
   $19.50); changing it is its own rule-1 event.
2. **Per-timeframe overrides live in the pair factory**, never in the global defaults. The same
   knob has had opposite effects on different timeframes several times. Record the evidence in the
   factory docstring.
3. **Pair isolation.** Gold-specific gates are keyed on the pair's display name so a future pair
   cannot inherit gold-calibrated logic by accident.
4. **Live equals backtest.** A new live gate goes into the replay path in the same commit. All time
   filters use bar-close time in both paths.
5. **ROC stays weighted** even though it correlates with MACD, Keltner and EMA. Removing it
   backtested 13R worse; the thresholds were calibrated with it.
6. **The backtester's realism settings are results-critical** (raw-path resolution, spread
   charging, next-open entry, stop-first ties, close-only trailing). Touching them breaks
   comparability with every recorded book.
7. **The forming candle stays dropped.**
8. Do not add config fields nothing reads. Do not disable indicators in production paths
   (`disabled_indicators` is for experiments only; a 0.0 regime multiplier in a per-timeframe
   `regime_weights_override` is equivalent for the composite and is the sanctioned vehicle for an
   indicator-off candidate).
9. EUR/USD was removed on 2026-07-16 after a no-edge verdict (last commit with it `7d3fd95`).
   BTC/ETH work exists only on unmerged branches. Do not re-add either without new data.

Process rules that go with them: every backtest report follows
`backtest/instructions_for_backtest.txt`; every candidate is compared with a control run on the
same code and cache, never with an old book directly; the `Backend:` line (since 2026-09-18 it
names the CUDA device or the fallback reason) and the "GPU cache fallbacks: 0" line are part of
the evidence; keep `docs/` in sync when behaviour changes. Any change to `engine.py`,
`signal_aggregator.py`, `chart_patterns.py`, `position_sizing.py`, the recorder's row layout or
the `GPUTechnicalEngine` overrides is also a **parity event for the CUDA research twin**
(`backtest/cudaBasedBacktesting.py`): mirror it in `backtest/cuda_signal_kernels.py` in the same
commit, keep Module 37 green, and regenerate at least the M15 OOS book with both scripts and show
equal MD5s before a number from the twin is used as evidence. Until then its books are not
evidence.

---

## 8. Validated configuration (what is actually running)

Defaults in `config.py` are the tuned gold configuration. The factory adds these per-timeframe
overrides (each one has its validation record in the `pair_signals/xauusd.py` docstring):

| Timeframe | Confirmations | Overrides on top of the defaults |
|---|---|---|
| M5 | M15, H1 | hour 9 UTC blocked and hour 21 unblocked; chart-pattern entries on (buy-side only: inverse head and shoulders, double bottom, bull flag); pullback entries off; partial EMA alignment scores neutral; trailing stop arms at 0.60R and trails 0.85R; shorts blocked in the `transitioning` regime |
| M15 | M5, H1 | target floor 1R; the mixed `trending` regime blocked; shorts blocked at 15:00 and 17:00 UTC; trailing stop arms at 0.70R and trails 0.75R; RSI weight halved in every regime |
| H1 | M15, H4 | every short blocked (`sell_min_quality` empty) |
| H4 | H1, D1 | none. H4 remains "unproven" (few trades) |

Shared defaults worth knowing: blocked hours 0, 1, 2, 10, 11, 13, 16, 21 UTC; trading days Monday
to Friday plus Sunday evening; shorts need grade A+, A or B and high-or-extreme volatility;
trailing stop arms at 0.6R and trails 0.75R; target floor 0.5R; round-trip cost $0.30 with
target 1 at least 30 × cost; the generic and chart-pattern setups floor the stop at $8; the gold
ATR setup uses a 1 × ATR target before the grade scale (2× at A+, 1.5× at A) and a 2 × ATR stop; the H4 regime filter is off; news never gates.

One semantic to know: `blocked_hours_utc` is applied to the bar **label** hour inside the core
and to the bar **close** hour in the live wrapper and the backtester. So H1 can only trade bars
closing 04–09, 15 and 18–20 UTC and H4 only bars closing 08:00 or 12:00. Both paths agree, and
every book was validated that way. Do not "fix" one side alone.

---

## 9. Reference books and how to reproduce them

Pinned trade books are no longer kept in the tree (owner decision 2026-09-15; the old ones sit in
`~/Development/ForexBot2_trash_2026-09-15/` until the owner deletes that folder). The gate is the MD5 of
the exported 63-column CSV (`md5sum` on the whole file). Regenerate a book from the cache when a
trade-level diff is needed; each run takes seconds to a few minutes.

Harness for every reference book: `venv/bin/python backtest/gpu_backtest.py --pair XAUUSD
--spread 0.65 --workers 12 --primary-tf <tf> --start <start> --end <end> --export-csv <out>`,
factory config, no `--set`, `--quality`, `--strategies`, `--confirm-tfs` or `--export-features`.
All of these run offline from the kept cache windows (0 cache fallbacks). Since 2026-09-18 the
CuPy GPU backend and `--backend cpu` both reproduce every MD5 below; earlier reports say
`NumPy CPU fallback` because CuPy could not compile kernels on the dev box until then.

| Book | Window | Trades / R / PF | MD5 of the CSV |
|---|---|---|---|
| M5 W2 | 2024-09-05 → 2026-09-05 | 781 / +146.07 / 1.52 | `49f8e602fc6ae769713c94ef7b257810` |
| M15 W2 | 2024-09-05 → 2026-09-05 | 931 / +146.14 / 1.43 | `b4aa07e33e17e23774c486a6900d673b` |
| M15 OOS | 2024-06-01 → 2025-03-30 | 355 / +60.67 / 1.49 | `9fd624fb387c36052122bd1830cf9c32` |
| H1 W3 | 2023-09-05 → 2026-09-05 | 253 / +70.0 / 1.98 | `7b848c917dcb5c371ecbb9f8f7292278` |
| H4 W3 | 2023-09-05 → 2026-09-05 | 114 / +21.7 / 1.59 | `186e677fbbd53cca735c94094c23ce05` |

Re-pinned 2026-09-17 for a **label-only** change: the exported `strategy` column now carries the
generator's own stamp (`XAU_ATR` / `XAU_PB` / `XAU_CP` / `GENERIC`) instead of a tag inferred
from the first reason's text, which had labelled every gold ATR setup `GENERIC`. All 62 other
columns of all five books are byte-identical to the previous pins (`29120a1e…`, `d185f099…`,
`6f8f24b4…`, `3b45d12b…`, `e75adae1…`); trades, R and PF are unchanged. `--strategies` accepts the
live names (`XAU` still means `XAU_ATR`).

Other current-HEAD figures at $0.65: M5 W3 (3-year) 870 / +153.6R / PF 1.50; M15 W3 1273 /
+186.6R / PF 1.39; M5 12-month (2025-08-14 → 2026-08-14) 560 / +108.81R / PF 1.54;
H1 12-month (2025-08-04 → 2026-08-04) 91 / +22.2R / PF 1.79. All five MD5s above were
reproduced on 2026-09-15 after the cache prune and on 2026-09-18 on both backends
(`backtest/research/gpu_backend_2026-09-18.md`).

The CUDA research twin (`backtest/cudaBasedBacktesting.py`, section 3) reproduced all five MD5s
above on 2026-09-18 in 0.8–2.4 s per book, plus the four 6-month acceptance books (2026-03-18 →
2026-09-18, M5 and M15 at $0.65 and $0.30: `86111c36…`, `e70d0252…`, `b8c98083…`, `e66b5fbd…`,
each equal to the reference script's), the M15 OOS `--export-features` CSV (`6239d197…`) and
`--record --record-bars` tables, and ten research-flag variants
(`backtest/research/cuda_backtester_2026-09-18.md`).

Kept cache windows (`backtest/cache/oanda_XAU_USD_<tf>_<paddedStart>_<end>.parquet`; the padded
start is the window start minus 62 days for M5/M15/H1, 234 for H4 and 1,400 for D1, so the D1
and H4 files of one run have different start dates than its M5 file): 3-year `20230705_20260905`,
2-year `20240705_20260905`, OOS `20240331_20250330`, M5 12-month `20250613_20260814`, M15/H1
12-month `20250603_20260804`, 5-year `20210531_20260804`, 7.6-year `20181031_20260807`, the
2026-08-14 parity window `20250814_20260814`, the 6-month CUDA acceptance window
`20260115_20260918` (M5/M15/H1), their D1/H4 pads, and `calendar.parquet`.

The research recorder (`--record DIR [--record-bars]`) writes Parquet files describing every
accepted trade (indicator votes on all three timeframes, penalties, setup, outcome, calendar
events). It never changes the book; pair every recorded run with a control run and show equal
MD5s. Scorecards from it are in-book statistics, never decisions.

---

## 10. Things that bite

- Never build a bare `PlatformConfig()` for a real scan; use the factory. A test enforces factory
  equivalence with the inline construction the backtester uses.
- The backtester names timeframes `5min`, `15min`, `1h`, `4h`. `M5` is a hard exit.
- `--spread` defaults to 0.30. Pass 0.65 for every headline result.
- `--export-features` changes the CSV; never use it on a parity run.
- The signal log's `spread` column is the median bar range, not a broker spread. `actionable`
  means grade ≥ C and a setup exists; the `action` text is written before the gate chain and can
  disagree with the final grade. `scan_kind` is `blocked` (weekend or day filter), `failed` (a
  required frame was missing) or normal; readers skip the first two. Several surfaces log the
  same bar, so readers dedup by the first row per (`timeframe`, `bar_ts`). Same-setup re-logs
  within 60 minutes are `DUPLICATE` rows; sum advisory R over `RESOLVED` rows only.
- Headlines and events in the signal log are content-addressed into `news_snapshots`; read
  through the `signals_with_news` view or `signal_log.read_news`.
- A non-empty `cleaned_warnings_json` is not a chart-pattern marker (a stale spread-gate veto lands
  there too since 2026-09-08); use `setup_json.strategy`.
- The `--stored-candles` scorecard in `analyze_signal_log.py` still advances outcomes without the
  locked-profit floor step that the logger and the backfill script use (owner decision pending), so
  the two measurement paths can disagree on gap rows.
- There is no holiday awareness anywhere. On Labor Day 2026 Oanda published no bars for hours, the
  bot rescanned the stale bar 43 times, and the violent-move watch called the 12–14 UTC slot hot.
  Interior gaps in history are logged, never fatal, for the same reason.
- Existing databases still contain an empty-going-forward `outlooks` table from the removed
  market-outlook feature. Nothing reads it.
- The news score is a trailing echo of the last day's price move (r ≈ 0.67 with the past day,
  ≈ 0.03 with the next). Both news gating flags stay off.
- The agent panel never runs inside `analyze_from_dataframes`, and its cached verdict is never
  reused when the evidence changed; the daily cap is the real spend bound.
- Live == backtest also means the backtester never writes to `signals.db` and never builds a
  `NewsEngine`.
- Twelve Data candles are not a drop-in replacement for Oanda (section 4).
- `pyarrow` (Parquet) is installed in the dev venv only; live code never touches Parquet.
- CuPy from pip (`cupy-cuda12x`) cannot compile a kernel without the CUDA headers; the
  `nvidia-cuda-runtime-cu12` wheel provides them and `backtest/gpu_kernels.py` points `CUDA_PATH`
  at it. Until 2026-09-18 that failure was swallowed: every book was built on NumPy while the file
  was called a GPU backtester. Now the fallback reason is in the `Backend:` line and Module 36
  fails on a box that has a visible GPU but no working backend. `cupy` is banned from the live
  roots by the same AST guard as scikit-learn.
- The CUDA kernels are bit-for-bit replicas of the CPU code: one thread per series (EWM, rolling)
  or per trade, compiled with `--fmad=false`. Do not "optimise" them with a parallel scan or a
  two-pass variance: the last bits change and a threshold can flip somewhere in 500k scans. A
  single sequential series is not faster on the GPU than pandas' C loop; the win is the batch
  across series and trades, and the replay itself stays on the CPU (same code as live).
- The replay workers are forked after the CUDA context exists and never touch the device (every
  cache array is host NumPy, asserted before the fork). Python 3.12 may warn about forking a
  multi-threaded process under `-X dev`; that is expected.
- `backtest/cudaBasedBacktesting.py` is a CUDA re-implementation of the decision stage, exempt
  from "replays go through the real signal code" only while its book MD5 equals
  `gpu_backtest.py`'s. A signal change that is not mirrored in `cuda_signal_kernels.py` makes the
  twin drift silently: no test compares it with the live code except Module 37 and the paired
  MD5 runs. It has no CPU fallback, refuses `--backend cpu`, and every description, warning and
  reason string is produced on the host from the same f-string templates as the live code; the
  kernels emit codes and numbers only.
- The kernels keep Python's evaluation order on purpose: `min`/`max` as argument-order ternaries
  (never `fmin`/`fmax`), numpy's pairwise summation for `mean`/`nanmean`, pandas' online rolling
  and EWM recurrences, Python 3.12's Neumaier-compensated `sum()` for the multi-timeframe mean,
  `round(x, 2)` through an error-free product, `atr_percentile = count / 50 * 100` (not
  `count * 2`: 15/50·100 = 30.000000000000004 flips "low" to "normal"). A cleaner formulation
  flips a threshold somewhere in 500k bars. The as-of alignment compares int64 nanoseconds; a
  mixed-unit index once returned slice length 0 for every confirmation frame.
- The dashboard's due-timeframe check uses the raw wake time; flooring `wake − buffer` skipped
  every H1 and H4 scan once (2026-08-02).
- The agent panel's price context reads the frames `analyze_pair` stashes on `report._frames`.
  A report built by hand (scripts, tests) has no frames, so pass them to `_price_context` /
  `run_panel` explicitly or the context carries geometry and base rate only. `scripts/panel_replay.py`
  rebuilds them from the kept Oanda cache; a database copied from live is never migrated (read-only),
  so the replay tolerates missing v2 columns and rebuilds payloads from the stored blobs.
- The AI estimate is a calibrated base rate, not a filter: on the 37 resolved setups of 2026-09 the
  v2 personas sat within ±2 points of the class win rate (Brier 0.262 vs 0.263 for the base rate
  alone, 0.299 for the old macro verdict) and separated nothing. Do not gate on it; do not iterate
  prompts on that sample again (`backtest/research/panel_v2_2026-09-17.md`).

---

## 11. Decision ledger

One line per validated change or verdict, newest first. Full evidence in the named report.

| Date | Decision | Where the evidence is |
|---|---|---|
| 2026-09-18 | Full-GPU research twin SHIPPED: `backtest/cudaBasedBacktesting.py` + `backtest/cuda_signal_kernels.py` (owner decisions: research only, `gpu_backtest.py` stays the base reference, every research flag supported, the one exemption from instructions item 6 recorded there). Proof: the five reference MD5s, four 6-month acceptance books (2026-03-18 → 2026-09-18, M5 and M15, $0.65 and $0.30), the M15 OOS `--export-features` CSV and `--record --record-bars` tables, and ten research-flag variants, all byte-identical to the reference script; Module 37 (16 tests) pins every stage against the live code. Books take 0.8–2.4 s instead of 13–115 s. Rule added: any signal change is a parity event for the twin | `backtest/research/cuda_backtester_2026-09-18.md` |
| 2026-09-18 | GPU backend made real: CuPy had never compiled a kernel on the dev box (`CUDA_PATH` missing for the pip layout), so every book to date was NumPy. SHIPPED: `backtest/gpu_kernels.py` (lazy, loud backend probe; `nvidia-cuda-runtime-cu12` header wheel; CUDA kernels for the EWM batch, the pandas rolling mean/var replica and the trade resolver, each a bit-for-bit twin of the CPU code), `--backend auto/gpu/cpu` + `FB2_BACKEND`, Module 36. All five reference MD5s reproduced on the GPU and with `--backend cpu`; precompute GPU == CPU on all 42 cached windows (2.34 M bars). Wall time roughly unchanged: the replay is CPU by design | `backtest/research/gpu_backend_2026-09-18.md` |
| 2026-09-17 | Local news scorer evaluated (the owner's scikit TF-IDF model in `~/Development/ML-NEWS`, trained on 606 DeepSeek gold labels): golden set 39/78 vs DeepSeek 78/78 vs keywords 24/78; out-of-sample agreement with DeepSeek 64% (DeepSeek agrees with itself 90%); cleaning / negation marking / label weighting move nothing; backfilling the 961 missing gold labels lifts macro-headline agreement 38% → 52% but only reshuffles golden-set errors. Verdict: a fallback, not a replacement. SHIPPED to grow a complete corpus: `xau_score` archived for every headline, no-currency headlines scored and archived (never shown), every new headline sent in its cycle (25 per call). Dashboard process made database-read-only (`set_db_writes_for_process`); the bot is the single writer. Books unaffected (news never enters signals) | `~/Development/ML-NEWS/eval/RESULTS_2026-09-17.md` (evaluation scripts live outside the repo by owner instruction) |
| 2026-09-17 | Live review of two weeks of `signals.db`: live == backtest bar for bar (26/29 trades matched, R within 0.14); the raw advisory −7.15R is cluster re-entries the validated books never take (cooldown view −1.6R, backtester +1.1R on the window, an ordinary fortnight); composite score inversely related to the 24 h move on all timeframes (two-week bear reading, test in-book with `--record-bars`); big-candle gate re-tested and REJECTED again (removes +32R / +60R of net-positive trades). Agent panel v2 SHIPPED: probabilities anchored to book base rates, code-computed decision context, no news doctrine, mean aggregator (`verdict_source: "mean"`), prompt version logged, offline replay harness; owner decision the same day: only the macro persona runs live (`personas: ["macro"]`, 1 call per panel; technical / risk / contrarian carried no information in either evaluation, kept for research); backtester `strategy` tag fixed (label-only, books re-pinned). Delivery findings: dashboard daemon on stale code since Sep 10, bot outage Sep 9 | `backtest/research/signal_log_review_2026-09-17.md`, `backtest/research/panel_v2_2026-09-17.md` |
| 2026-09-15 | Backtest folder cleanup: `diagnose_filters.py`, `shadow_scan.py`, and the two one-time news scripts removed (last commit with them `931b265`); every pinned CSV, the research output folders, the BTC/ETH cache snapshots, the Tiingo caches and 13 uncited Oanda windows moved to `~/Development/ForexBot2_trash_2026-09-15/` (346 MB) pending the owner's delete; all five reference MD5s reproduced afterwards. Suite 506 / 33 | commit `b0b71e1` |
| 2026-09-15 | AI market-outlook feature removed (module, hourly Telegram brief, `--outlook`, `outlooks` writer, config block). Legacy config block tolerated with a warning. Books byte-identical | commit `931b265` |
| 2026-09-13 | Bug audit, 11 delivery/measurement fixes, books byte-identical: NEUTRAL-signal C/B setups now alert (about 17% of the H1 12-month book's R, +3.7R of +22.2R, was never sent); a missing confirmation frame fails the scan; pandas `utcnow` replaced and `pandas<4`, `numpy<3` pinned; advisory outcomes keep locked-profit floors (`GAP_LOCKED`); scan-outcome labeler bounded to 72 h; settlement pause is a scheduled closure; `fetch_range` retries pages and audits gaps; cache-fallback count logged and must be 0; partial `regime_weights_override` raises; the bot scans each bar once and attaches its log handler in `main()` only; the dashboard catches up missed timeframes. Verified sound and unchanged: composite maths, scan-outcome labels, outcome exits, Oanda alignment, no look-ahead | plan file `~/.claude/plans/you-are-python-software-joyful-teacup.md`, commit `2836024` |
| 2026-09-10 | Improvement cycle 1: 3-year baseline re-established; cost ladder shows no red flag (PF ≥ 1.3 everywhere at $0.80); the intraday edge is a fat right tail (top 5% of trades carry 94–96% of M5/M15 R). H1 divergence-to-zero REJECTED. M15 RSI × 0.5 sits on a stable plateau. SHIPPED: live fetches floor at 1,000 bars and the backtester pads history to match (books trade-identical). Parked cycle-2 candidate: block M15 grade-A SELLs (230 trades, −14.4R, negative in every year and OOS) | `backtest/research/cycle_2026-09-10.md` |
| 2026-09-09 | Weight test on the recorded replays: M15 RSI multiplier × 0.5 in every regime PASSED (2-year +117.6R → +146.1R, OOS +40.6 → +60.7R) and SHIPPED as an M15-only override; all M5 MACD candidates FAILED; M15 EMA × 2 and divergence-off FAILED | `backtest/research/weight_test_2026-09-09.md` |
| 2026-09-09 | Replay recorder shipped (`--record`, `--record-bars`), books byte-identical. First look is descriptive only: Bollinger opposes 94% of M5 entries; divergence-opposed M5 trades did best; bar-level baseline drifts +1.9 ATR/24 h on M5, so adjust for drift before judging BUY votes | `backtest/research/replay_recording_2026-09-09.md` |
| 2026-09-08 | Delivery fixes: stale spread-gate vetoes moved to `cleaned_warnings` (books identical; only the warning text on 20–33% of A-grade alerts changed); alerts require a setup; every send logged with its row id; `telegram.log` rotates daily × 30 | commit history, 2026-09-08 |
| 2026-09-08 | News archive shipped (`logs/newsdata.db`). Do not train a local headline scorer before about 4,000 headlines over 3+ weeks, time-split, headline text only, graded against the golden set (`git show 1cebebf^:backtest/research/llm_news_golden.csv`) | `news_archive.py` docstring, `backtest/research/llm_news_eval_2026-08-06.md` |
| 2026-09-07 | Signal-side outlier filter fixed (it deleted about 36 real M5 news bars a year in the backtester and a different 24 live). Book impact at $0.65, 12 months: M5 +112.2R → +108.8R, M15 +73.9R → +74.6R, H1 unchanged. Oanda retry ladder added; a scan that loses its primary frame is `failed` | CLAUDE.md history, commit 2026-09-07 |
| 2026-09-06 | Stake-sizing campaign FAILED its pre-registered rule on M5 (clean null) and M15 (lift inside the shuffle null). Nothing changing numbers shipped; `--export-features` and the penalty tags remain. M5 stretch measures are positive in both halves (do not propose exhaustion sizing on M5); bursts cluster 12–15 UTC and 01 UTC and do not hurt entries; every open-trade burst protection lost money | `backtest/research/sizing_campaign_2026-09.md` |
| 2026-09-06 | Violent-move watch shipped dark (display only). Hot = high-impact USD release within ±30 min, or a hot clock slot plus a price extreme; flags 8.6% of bars at 3–5× odds. Context readings never raise the level | `violent_watch.py` docstring, sizing campaign addenda |
| 2026-09-06 | Stale-code cleanup: research tooling, Dukascopy study, diagrams and plans deleted (history before `953c5ad` has them); `main.py` M5 confirmations fixed to M15 + H1; dead functions removed | commits of 2026-09-06 |
| 2026-09-04 | News sentiment verdict: the score trails price, it does not lead it. Headline window 4 h, neutral headlines dropped, beginner-language summary shipped; gating flags stay off | `backtest/research/news_sentiment_value_2026-09-04.md` |
| 2026-09-03 | Signal-log campaign shipped two gates: M5 shorts blocked in `transitioning` (+7.9R/12 mo) and every H1 short blocked (5-year +16.7R, OOS +11.5R). Rejected: M15 SELL-B block, Bollinger-off. Parked: divergence-off. Panel verdict switched to the macro persona (owner decision); the chair's OPPOSE was anti-informative (opposed trades made +30R) | `backtest/research/signal_log_campaign_2026-09.md`, `signal_log_eval_2026-08-18.md` |
| 2026-09-03 | Auto-execution (Oanda daemon and executor) removed; the owner does not auto-trade (last commit with it `31002cb`) | commit `31002cb` |
| 2026-08-15 | Agent-panel technical persona gets ATR-normalised candles and raw indicator values | `backtest/research/ai_signal_enrichment_2026-08-15.md` |
| 2026-08-14 | Books re-pinned on Oanda bars; Twelve Data books superseded. Feed divergence and Oanda-Tiingo signal parity measured | `repin_verification_2026-08-14.md`, `feed_divergence_2026-08-14.md`, `signal_parity_2026-08-14.md` |
| 2026-08-06 | DeepSeek headline scoring validated (78/78 vs 24/78 for keywords); event misreads fixed with the `vs_forecast` field | `backtest/research/llm_news_eval_2026-08-06.md` |
| 2026-08-05 | Fixed-RR hard target vs trailing exit: NOT shipped (passes the letter of rule 1 on 2024+ windows but is a trend artefact; M15 negative in 2021 and 2023, drawdown doubles). `--fixed-rr-tp` stays as a research flag | `backtest/research/fixedtp_2026-08-05.md` |
| 2026-08-04 | QA audit. Weekend calendar made New-York anchored (removed one live-impossible phantom trade). Spread inside exit-trigger detection: owner DECLINED (4–6% optimism accepted, books stay comparable). `FB2_EMA_LO` hardened. M5 pullback entries disabled (PF 1.26 → 1.44 at $0.65). Partial EMA alignment set neutral on M5. XAU_CP gate bypass documented as validated behaviour (226 of 586 M5 trades, +38.9R) | memory files `qa-audit-2026-08-04`, `m5-tuning-2026-08-04`, `m5-round2-2026-08-04` |
| 2026-07-30 | M5 relaxation shipped (score 25, confidence 5, hour 21 unblocked); big-candle gate rejected (ships off) | memory `relaxation-campaign-2026-07-30` |
| 2026-07-28 | Oanda v20 becomes the live feed | commit `d2d4fb4` |
| 2026-07-24 | Real gold spread measured at about $0.65 round trip (9M Dukascopy ticks, 12 months); $0.30 kept only for cross-book comparability and the live target floor | memory `dukascopy-spread-study` |
| 2026-07-21 | M15 trailing arm 0.70 and M5 trailing distance 0.85 adopted | memory `m15-trail-grid-2026-07-21` |
| 2026-07-16 | M15 SELL-hour block (15 and 17 UTC) adopted; EUR/USD removed (no edge) | memory `xau-m15-loss-analysis`, `eur-no-edge-verdict` |
| 2026-07-11 | M15: mixed `trending` regime blocked, target floor 1R; M5: hour 9 blocked | pair factory docstring |
| 2026-07-08 | Backtester realism fixes (spread charging, gap fills, next-open entry, stop-first ties) | memory `backtester-realism-fixes` |

Older memory notes live in `~/.claude/projects/-home-harwinder-Development-ForexBot2/memory/`.
The previous, longer version of this file is in git history before 2026-09-15.