1793d9167a7125f8807782e5663de7c7187a8cb0
12 Commits
| Author | SHA1 | Message | Date | |
|---|---|---|---|---|
|
|
1793d9167a |
Daily-trend veto on M15 + lot-size / already-open lines (campaign 2026-09-30), and the 2026-10-01 whole-codebase bug review
Two bodies of work that touch the same files, so one commit. Direction and setup-size campaign (2026-09-30 → 10-01, owner's RISK-FIRST rule, written before the runs): - SIGNAL: M15 daily-trend veto (factory override 7, xau_strategy.daily_trend_veto "both"). A finished setup against a clear daily trend (closed D1 bars, Sunday stubs dropped; close vs its 50-bar EMA AND vs the close 60 kept days earlier) is removed as the last step of the core. One definition (signal_aggregator.daily_trend_series) for live, the bar replay and the twin's host side; D1 is a required frame of the M15 scan; mirrored in cuda_signal_kernels.py K2. W3 824 / +205.12R / PF 1.73 / max DD 12.4R vs 1105 / +195.79R / 1.49 / 20.9R; 2019 → 2026 1473 / +223.33R / 1.42 / 41.5R vs 2091 / +192.53R / 1.24 / 64.8R; unseen holdout −4.62R vs −6.63R. - FAILED, not shipped: the veto on M5 (W3 drawdown 10.1R → 15.2R) and on H1 (−9.7R), buy-only and sell-only on M15, the wide-stop cap (xau_strategy.wide_stop_cap_atr stays 0.0, a research knob). - New M15 pins: W2 dfc647ee…, OOS e87b6de1…, W3 ad3b0dbd…. M5 / H1 / H4 pins unchanged. --set xau_strategy.daily_trend_veto=off reproduces the previous M15 pins byte for byte. Twin and reference script MD5-equal on every window; recorder and feature-export parity re-proved; base-rate table rebuilt. - DISPLAY (books byte-identical): exposure.py — the lot size for trade_size.risk_per_trade (new config.json block, off by default, private to the owner) and the count of alerted same-direction plans still open (signals.alerted_at, read-only open_alerted_setups); the daily trend in words on Telegram, the CLI and the dashboard; signals.daily_trend logged. - providers/oanda: a date window with no bars is an empty page, not an error (a D1 pad starting on a Saturday could not be fetched). - Reports only, nothing shipped: seven context screens (no state nominated) and eight outside data sources (none adds a forecast to gold's own daily trend). - Independent verification (report section 5.13): every figure reproduced on both scripts, scoring identical to the previous commit on all four timeframes, no look-ahead. Caveats recorded there: this is a risk filter, not a more accurate direction call; nine tenths of the 2019 → 2026 gain is 2022; the neighbour clause is thin (a simple 100-day average misses the R line by 1.2R); "never adds a trade" is an observation, not a guarantee. Bug review of the whole codebase (2026-10-01). Signal path clean. 17 defects elsewhere, plus 4 found by an independent review of the fixes; all fixed, each with a test; no trade book changed: - telegram_bot: one _deliver_alert for the candle scan and the level break. The row is stamped alerted_at BEFORE the send, on a one-worker alert-db pool, and un-stamped when no chat received it or the task is cancelled mid-stamp, so two alerts at one candle close count each other and no SQLite write runs on the event loop. The 5-bar cooldown counts bars of the scan's own frame (== gpu_backtest._apply_cooldown; clock time released the 22:00 UTC bar after the settlement pause) and is restored from signals.db at start-up. The lot-size line goes to the owner's own chat only (a group the owner ran /autoscan in used to receive it). /status gains a news-model line and escapes failure details. - signal_log: mark_alerted returns the stamp it wrote; unmark_alerted; last_alerted_bars (read-only). - advisory_outcome: candle continuity is judged bar by bar inside the walk; a hole after a trade's exit no longer voids it. The stored-candle replay resolves 108 of 111 logged setups instead of 99, all equal to the live resolver's R. - providers/oanda: an empty date window is accepted only when it is 5 days or shorter and no malformed candle was skipped; a longer one is retried, then refused, instead of being cached as history with a hole. - news_engine: blank RSS titles skipped; one unreadable headline no longer costs the local model its whole batch (checked on the real bundle); model health recorded; no DeepSeek audit call in a process that does not archive; event-direction cache keyed per release; peek_week_events never fetches. - llm_client: chat_json never raises and never logs the API key (requests quotes the Authorization value in InvalidHeader; error bodies are redacted). - signal_aggregator: the scan's news block is guarded; the replay path (use_news=False) assigns exactly the values it did before. - agent_panel: PANEL_PROMPT_VERSION v2.2 — same prompt text, corrected input (event timing over the cached calendar week, headlines newest first). scripts/panel_replay.py replays the logged payload per persona; scripts/label_scan_outcomes.py counts with an ISO cutoff. - test_all.py: no ALTER TABLE … DROP COLUMN (the live box has SQLite 3.34); tests that write Parquet print a notice and return without pyarrow. Verification after the last code change: suite 632 passed, 0 failed (also with both Parquet engines blocked and FB2_BACKEND=cpu, the live box's view); the five reference MD5s reproduce on both backtesters, 0 cache fallbacks; Module 37 green; static Python 3.9 / SQLite 3.34 check clean on every touched file. Evidence: backtest/research/daily_trend_campaign_2026-09-30.md, context_screens_2026-10.md, new_inputs_2026-10.md; CLAUDE.md sections 2, 4, 6, 8, 9, 10 and 11; pair_signals/xauusd.py docstring (M15 override 7). Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> |
||
|
|
2836024e9d |
audit 2026-09-13: eleven delivery/measurement fixes; books byte-identical
Signal path
- A shipped setup always names a direction: grade C needs confidence
>= 45 (gate 6c grants B at >= 45) while BUY/SELL needs >= 50, so C/B
reports reached the alert gate as NEUTRAL with a full setup. The
backtester traded them (the replay never reads final_signal) while
Telegram rejected NEUTRAL first — 5/91 trades, +3.7R of +22.2R, in
the H1 12-month book. _attach_core_setup stamps the setup's direction
and repairs the pre-gate action text; no-setup reports stay NEUTRAL.
- analyze_pair fails the scan when ANY required frame is missing
("DATA UNAVAILABLE:", scan_kind='failed'); a lost confirmation frame
used to run with an unearned -15 MTF penalty.
Outcomes and labels
- advisory_outcome.resolve_locked_floor: a gap the database cannot fill
resolves at the locked trailing stop (GAP_LOCKED, result.floor) when
the stop had crossed the entry; the rebuild fires on both gap reasons
(rows 915/920: +1.14R / +0.84R floors; resolved sum -2.46R -> -0.48R).
- label_scan_outcomes: SCAN_LABEL_MAX_AGE_H = 72 h, stale rows never
anchor the shared bar stream, labeled_at bumped on re-examination,
_stream_since warns at its row cap (the Labor-Day pile would have
blinded scan_outcomes around 2026-09-14).
Providers and scheduling
- The daily settlement pause (Mon-Thu 17:00 New York) is a scheduled
closure for _has_latest_closed_bar (whole-span label walk-back) and
_is_data_fresh (closed seconds are not age); bot and dashboard skip
scans whose just-closed label sits in the pause (61 false stale rows
in three nights).
- fetch_range retries a failed page in place (three failures on one
window raise) and logs an ERROR for interior gaps of >3 unscheduled
bars.
- pd.Timestamp.utcnow() replaced in providers/base.py and engine.py;
requirements pin pandas<4, numpy<3.
- telegram_bot: boundary loops share one scan per bar with the breach
handler (_remember_scan keys on the analysed bar; breach defers 20 s
after an M15 boundary); the rotating file handler attaches in main()
only (the test suite used to write into logs/telegram.log).
- dashboard_publisher: catch_up_timeframes re-scans boundaries missed
while a cycle overran or the process was down.
Backtester and engine guards
- _replay_span returns the per-span GPU cache-miss count; run_backtest
sums, logs it (must be 0) and writes cache_fallbacks to the manifest.
- regime_weight_table raises on a partial regime_weights_override.
Verification: 519 tests; M5/M15 2-yr, M15 OOS, H1/H4 3-yr books
byte-identical before and after; live signals.db untouched (checked on
a copy). CLAUDE.md carries the dated entry and the blocked-hours union
semantics that every book was validated under.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_0139xz1J6YccndA2be6yFnEo
|
||
|
|
9942736762 |
providers: floor every live fetch at 1000 bars (H4/D1 indicator-seeding parity)
The engine's EMAs use ewm(adjust=False), so the first bar of a fetched frame keeps weight (1-2/201)^n in the 200-bar trend EMA. With the 60-day lookback the live H4 frame had 380 bars (seed weight 2.3%) and D1 300 (5%), while the backtester seeds from its whole cached history. Measured on the deepest Oanda caches (267 H4 / 313 D1 bars sampled over two years, raw and cleaned frames): ema_trend flipped on 2/267 H4 and 1/313 D1 bars at those depths (composite off by up to 9.5 points) and on none from 600 bars up. MIN_LIVE_BARS = 1000 leaves a seed weight of e^-10; M5/M15 (5000) and H1 (1460) already exceed it and are unchanged. One request either way. The backtester never calls fetch_ohlcv, so no book changes. Tests: new Module 16 depth test; the Module 21 cache test's 300-bar expectation becomes 1000. Suite 500 passed. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01NQDjW8rXYrJR4JDWMWKUij |
||
|
|
de36b104d9 |
providers: keep real news bars (snap-back bad-print rule), Oanda retry ladder, scan_kind=failed
2026-09-07 signal-log inspection follow-up — three fixes that do not depend on sample size.
- providers/base.py: the signal-side 10σ outlier filter is now a bad-print filter. A >10σ
close-to-close bar is dropped only when the NEXT close snaps back ≥80% of the way
(OUTLIER_SIGMA / OUTLIER_REVERT_FRACTION); persisting moves and the newest bar are kept.
The unconditional rule deleted ~36 real M5 news bars/yr in the backtester (whole-frame σ)
and a different ~24/yr live (fetch-window σ) — e.g. the 2026-09-04 12:30 UTC NFP bar
vanished from M5+M15 and the 12:35 scan was a byte-identical repeat of 12:30. 3-yr Oanda
drops are now M5 2 / M15 1 / H1 0.
- providers/oanda.py: transient 401/5xx/network faults retry along RETRY_DELAYS_S=(2,5,10)s
under RETRY_BUDGET_S=45s. 11 boundary scans in two days had lost their primary frame at
xx:00:22–:26 UTC; one 2 s retry could not outlast the top-of-hour blip.
- signal_aggregator/_log_signal: a scan whose primary fetch failed logs scan_kind='failed'
(price 0, no signal); analyze_signal_log.parity_report skips it like 'blocked'.
- test_all.py: 466 tests (+6): retry ladder walks all delays, 5xx blip recovers on the third
attempt, retry budget stops the ladder, persisting/newest/partially-retraced 10σ bars kept,
failed-scan row kind, parity skip.
- CLAUDE.md: cleaning rule, retry ladder, failed scan kind, dated current-state note.
Book impact at $0.65 (rule 1 still passes everywhere, both halves positive):
12mo M5 557/+112.21/PF1.56 → 560/+108.81/1.54 M15 397/+73.93/1.53 → 398/+74.61/1.53 H1 identical
3yr M5 857/+161.3/1.53 → 870/+153.6/1.50 M15 1086/+148.7/1.37 → 1092/+151.9/1.38
H1 255/+70.5/1.98 → 253/+70.0/1.98 H4 113/+22.7/1.63 → 114/+21.7/1.59
Only trades on the previously-deleted event bars move; the pinned M5/M15 CSVs are no longer
byte-reproducible and should be regenerated at the next re-pin.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01NDnBxJ2y8GRUM6ZNFJs1hZ
|
||
|
|
f7c244c6b5 |
chore: drop unreachable branches outside the signal core (no behaviour change)
- providers: _to_oanda_instrument / _to_tiingo_ticker / _to_td_symbol no longer guess a wire symbol for pairs that are not registered — the registries are the only source, so an unknown symbol raises the provider's DataIntegrityError instead of silently fetching something else. Every scanned symbol is validated against PAIR_INFO at import, so nothing reachable changes. - main / telegram_bot _fmt_price: the sub-$10 five-decimal branch could never fire for gold. - news_engine._parse_pair: BASE/QUOTE only; the Yahoo "=X" fallback was unreachable (its test goes with it). - dashboard_publisher.event_to_dict: drop minutes_until (frozen at publish time, misleading) and is_high_impact — dashboard.js never read either. - static/dashboard.html: unused id="cal-table". Tried and reverted: trimming the LogRow fields nothing in analyze_signal_log reads — the tests read them to verify the loader round-trips those DB columns, so they stay. Suite: 460 passed, 0 failed. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_0156YhogLytzj9P2iH1skGbL |
||
|
|
1f44d6bad6 |
chore: repo hygiene — stale gitignore entries, finished plans, superseded research, doc drift
Removed (all recoverable from git history):
- llm_provider_switch_plan.txt: 2026-08-08 DeepSeek/Anthropic multi-provider plan that
was never implemented (llm_client.py is DeepSeek-only, no `anthropic` anywhere)
- docs/superpowers/specs/2026-07-21-centralized-config-design.md: spec of the
config.json migration that merged the same day; its plan sibling was already
deleted in
|
||
|
|
37ff5b383d |
signal_log: persist agent panel verdict + per-persona takes to SQLite
Add panel_verdict, panel_confidence, panel_rationale, panel_takes_json columns to the signals table (additive, idempotent ALTER TABLE migration for existing DBs). _report_row now reads the panel fields attach_panel writes onto the SignalReport, so LLM panel verdicts stop evaporating on process exit. Also documents the Oanda re-pin (round_trip_cost stays $0.30 for cross-book comparability and the live TP gate; $0.65 is the measured headline) and adds the feed-divergence / parity / shadow-scan study tooling referenced by CLAUDE.md. |
||
|
|
a42ef16113 |
oanda: retry once on transient 401/5xx/network faults in candle fetch
The practice API rejects ~15-20 requests/day with a spurious 401 "Insufficient authorization" while the same token succeeds on adjacent requests; each one silently cost a full scan cycle on that timeframe. One retry after RETRY_DELAY_S (2 s) absorbs these plus transient 5xx/network faults. Real credential failures fail both attempts and raise as before; other 4xx (bad request, 429) are never retried. Tests: transient-401 recovery, persistent-401 raise, 400 not retried. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Tujtmmhxj9mxKjPFSoVjmP |
||
|
|
45ce97ed13 |
audit: NY-anchored weekend calendar (all 3 copies) + FB2_EMA_LO crash-proofing
Owner decisions from the 2026-08-04 audit report: - spread-in-exit-triggers: DECLINED (books stay comparable; ledger in CLAUDE.md records the measured ~4-6% optimism). - weekend clock: FIXED. The market week is Fri 17:00 -> Sun 18:00 New York time, not Fri/Sun 22:00 UTC — during US DST the Oanda feed stops at 21:00 UTC (measured: 159/162 summer weekends). All three weekend implementations (providers/base, signal_aggregator, gpu_backtest calendar filter) now share the NY-anchored rule; a new test sweeps them hourly across both DST transitions and asserts agreement. No NTP needed — the host clock was fine, the constant was wrong. Ends the weekly false-"STALE DATA / feed lagging" hours (summer Fri 21-22 UTC, winter Sun 22-23 UTC). - FB2_EMA_LO (delegated): kept as a research knob but crash-proofed — an unparseable value degrades to the tuned default 0.5 with a warning instead of killing every scan. Documented in CLAUDE.md. Book gates: M15 12mo pinned book byte-identical (+78.7111R, 397 trades). M5 12mo drops exactly ONE trade (556 -> 555, +97.56 -> +98.64R): a phantom whose signal fired on the final summer-Friday bar (decision 21:00 UTC = NY 17:00 market close) and was "filled" at Sunday's reopen 49 hours later, -1.08R — an entry live trading can never take. The backtest no longer contains it; live == backtest parity improved. M15 unaffected because hour 21 is in its blocked set. Suite: 341/341 green. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01UnZkzeFcTu9agFg8BDh2tn |
||
|
|
fc40ba081f |
audit: alert delivery, provider robustness, backtester NaN/cache guards (Batches C+E+F)
2026-08-04 Fable 5 QA audit. Three batches sharing one test file, all verified by runnable repros before fixing, all gated below. Batch C — telegram alert delivery (verified by h4 harness): - Failed breach rescan shrinks the cooldown to 30 s instead of silencing the symbol for 5 min; breach during cooldown is PARKED and fired on the first tick after expiry (was dropped forever). - Level cache keyed symbol→TF (M5 scans no longer wipe H1/H4 swing levels within minutes); breach labels TF-qualified. - Markdown alert sends get a plain-text fallback (mirroring _safe_edit); fingerprint commits only when a message actually landed. - Per-key asyncio.Lock closes the dedup TOCTOU between auto-scan and breach sends (forced-interleave repro produced duplicate alerts). - _start_scheduler respawns dead per-TF loops (was all-or-nothing). - Clean WS close now backs off (was a zero-delay reconnect hot loop); backoff resets only after a connection that received ticks; TD iter_ticks fires on_connect on the first RECEIVED frame (invalid key can no longer pin the backoff at minimum). - Scan-failure logs carry the real traceback (was "NoneType: None"). Batch E — providers (verified by h5 harness): - fetch_range: 3 consecutive chunk ERRORS raise instead of returning silently truncated history (verified 731-day silent gap repro; also covers 429 storms); pagination no-progress guard; head-truncation ERROR log. - Single-flight per (symbol,TF): 4 concurrent scans = 1 API call (was 4); cache records requested depth so a longer-lookback call refetches. - Multi-bar forming/future tail fully dropped (was iloc[:-1] once). - Oanda: malformed candles skipped, never 0.0-defaulted; cleaning drops non-positive-price rows BEFORE OHLC repair; inf returns neutralized so one corrupt bar can no longer disarm the whole 10-sigma filter. Batch F — backtester internals (verified by h6 harness): - NaN bar in a trailing window exits at the last valid bar (was NaN R poisoning every metric); static-path TIMEOUT NaN guard; compute_metrics excludes non-finite R loudly. Zero real-data exposure (0 NaN rows in all 10 Oanda parquets) — defensive. - fetch_history ERROR-logs pre-2026-07-08 cache files (cleaned-bars era) when loaded for raw-path resolution. Gates: suite 333/333 green; pinned M15 12mo book reproduced identically (397 trades, +78.7111R) after E and after F; M5 HEAD-config baseline reproduced identically (556 trades, +97.5604R) after F. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01UnZkzeFcTu9agFg8BDh2tn |
||
|
|
f0d76021d1 |
Tiingo WebSocket streaming: re-enable the level-breach monitor
The monitor has been dead since the Oanda feed switch (Oanda's provider does not stream). New config.json scalar data_providers.streaming selects a tick-stream provider independent of the candle feed; telegram_bot resolves it via _get_streaming_dp() (separate instance, candle cache stays on the active provider) and falls back to the active provider when unset — behavior is byte-identical to before when the key is absent. TiingoDataProvider grows the two sanctioned streaming overrides: supports_streaming=True and iter_ticks() on wss://api.tiingo.com/fx (mid price only — the no-spread rule extends to the stream). Two deliberate divergences from the TD reference implementation, both because Tiingo authenticates inside the socket: server "E" messages raise TiingoDataError (a swallowed auth error would end the generator cleanly and reconnect-hammer with zero backoff), and on_connect fires on the first received message rather than after the subscribe send. Field order of quote frames was live-verified: ask is data[7], not data[6] as the docs table suggests (data[6] is askSize). make_data_provider(config, name=None) now takes an explicit provider name; keyless streaming providers disable the monitor with one warning instead of a traceback-per-minute loop. Tests 266 → 273 (WS parser, ticker round-trip, streaming config scalar, explicit-name dispatch, streaming-dp resolution, monitor auto-disable). Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Nsd8HcrV67GbCZXxZEn1Ma |
||
|
|
c8beb7e3ae |
Provider abstraction: providers/ package with BaseDataProvider template
One class per provider (TwelveData, Oanda, Tiingo), all subclassing BaseDataProvider which owns cache, validation/cleaning, freshness + boundary guarantees, fetch_range pagination, and streaming stubs — subclasses implement only _fetch_raw. config.json data_providers.active selects the live feed; data_provider.py / oanda_provider.py stay as back-compat shims. Module 16 provider-abstraction tests (253 total green). Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Mc7XKYyPySVXiUpcN82QEK |