harry/ForexBot2
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity
216 Commits 3 Branches 0 Tags
2cc5915b006e7ca1b6175e16ef634da516743c7e
Commit Graph

6 Commits

Author SHA1 Message Date
Harwinder Sekhon 2cc5915b00 Daily-trend veto on M15 + lot-size / already-open lines (campaign 2026-09-30), and the 2026-10-01 whole-codebase bug review 
Two bodies of work that touch the same files, so one commit.

Direction and setup-size campaign (2026-09-30 → 10-01, owner's RISK-FIRST rule,
written before the runs):
- SIGNAL: M15 daily-trend veto (factory override 7, xau_strategy.daily_trend_veto
  "both"). A finished setup against a clear daily trend (closed D1 bars, Sunday
  stubs dropped; close vs its 50-bar EMA AND vs the close 60 kept days earlier)
  is removed as the last step of the core. One definition
  (signal_aggregator.daily_trend_series) for live, the bar replay and the twin's
  host side; D1 is a required frame of the M15 scan; mirrored in
  cuda_signal_kernels.py K2. W3 824 / +205.12R / PF 1.73 / max DD 12.4R vs
  1105 / +195.79R / 1.49 / 20.9R; 2019 → 2026 1473 / +223.33R / 1.42 / 41.5R vs
  2091 / +192.53R / 1.24 / 64.8R; unseen holdout −4.62R vs −6.63R.
- FAILED, not shipped: the veto on M5 (W3 drawdown 10.1R → 15.2R) and on H1
  (−9.7R), buy-only and sell-only on M15, the wide-stop cap
  (xau_strategy.wide_stop_cap_atr stays 0.0, a research knob).
- New M15 pins: W2 dfc647ee…, OOS e87b6de1…, W3 ad3b0dbd…. M5 / H1 / H4 pins
  unchanged. --set xau_strategy.daily_trend_veto=off reproduces the previous M15
  pins byte for byte. Twin and reference script MD5-equal on every window;
  recorder and feature-export parity re-proved; base-rate table rebuilt.
- DISPLAY (books byte-identical): exposure.py — the lot size for
  trade_size.risk_per_trade (new config.json block, off by default, private to
  the owner) and the count of alerted same-direction plans still open
  (signals.alerted_at, read-only open_alerted_setups); the daily trend in words
  on Telegram, the CLI and the dashboard; signals.daily_trend logged.
- providers/oanda: a date window with no bars is an empty page, not an error
  (a D1 pad starting on a Saturday could not be fetched).
- Reports only, nothing shipped: seven context screens (no state nominated) and
  eight outside data sources (none adds a forecast to gold's own daily trend).
- Independent verification (report section 5.13): every figure reproduced on
  both scripts, scoring identical to the previous commit on all four
  timeframes, no look-ahead. Caveats recorded there: this is a risk filter, not
  a more accurate direction call; nine tenths of the 2019 → 2026 gain is 2022;
  the neighbour clause is thin (a simple 100-day average misses the R line by
  1.2R); "never adds a trade" is an observation, not a guarantee.

Bug review of the whole codebase (2026-10-01). Signal path clean. 17 defects
elsewhere, plus 4 found by an independent review of the fixes; all fixed, each
with a test; no trade book changed:
- telegram_bot: one _deliver_alert for the candle scan and the level break. The
  row is stamped alerted_at BEFORE the send, on a one-worker alert-db pool, and
  un-stamped when no chat received it or the task is cancelled mid-stamp, so
  two alerts at one candle close count each other and no SQLite write runs on
  the event loop. The 5-bar cooldown counts bars of the scan's own frame
  (== gpu_backtest._apply_cooldown; clock time released the 22:00 UTC bar after
  the settlement pause) and is restored from signals.db at start-up. The
  lot-size line goes to the owner's own chat only (a group the owner ran
  /autoscan in used to receive it). /status gains a news-model line and escapes
  failure details.
- signal_log: mark_alerted returns the stamp it wrote; unmark_alerted;
  last_alerted_bars (read-only).
- advisory_outcome: candle continuity is judged bar by bar inside the walk; a
  hole after a trade's exit no longer voids it. The stored-candle replay
  resolves 108 of 111 logged setups instead of 99, all equal to the live
  resolver's R.
- providers/oanda: an empty date window is accepted only when it is 5 days or
  shorter and no malformed candle was skipped; a longer one is retried, then
  refused, instead of being cached as history with a hole.
- news_engine: blank RSS titles skipped; one unreadable headline no longer
  costs the local model its whole batch (checked on the real bundle); model
  health recorded; no DeepSeek audit call in a process that does not archive;
  event-direction cache keyed per release; peek_week_events never fetches.
- llm_client: chat_json never raises and never logs the API key (requests
  quotes the Authorization value in InvalidHeader; error bodies are redacted).
- signal_aggregator: the scan's news block is guarded; the replay path
  (use_news=False) assigns exactly the values it did before.
- agent_panel: PANEL_PROMPT_VERSION v2.2 — same prompt text, corrected input
  (event timing over the cached calendar week, headlines newest first).
  scripts/panel_replay.py replays the logged payload per persona;
  scripts/label_scan_outcomes.py counts with an ISO cutoff.
- test_all.py: no ALTER TABLE … DROP COLUMN (the live box has SQLite 3.34);
  tests that write Parquet print a notice and return without pyarrow.

Verification after the last code change: suite 632 passed, 0 failed (also with
both Parquet engines blocked and FB2_BACKEND=cpu, the live box's view); the
five reference MD5s reproduce on both backtesters, 0 cache fallbacks; Module 37
green; static Python 3.9 / SQLite 3.34 check clean on every touched file.

Evidence: backtest/research/daily_trend_campaign_2026-09-30.md,
context_screens_2026-10.md, new_inputs_2026-10.md; CLAUDE.md sections 2, 4, 6,
8, 9, 10 and 11; pair_signals/xauusd.py docstring (M15 override 7).
 2026-10-01 10:28:28 -04:00
Harwinder Sekhon d2e679cd20 providers: keep real news bars (snap-back bad-print rule), Oanda retry ladder, scan_kind=failed 
2026-09-07 signal-log inspection follow-up — three fixes that do not depend on sample size.

- providers/base.py: the signal-side 10σ outlier filter is now a bad-print filter. A >10σ
  close-to-close bar is dropped only when the NEXT close snaps back ≥80% of the way
  (OUTLIER_SIGMA / OUTLIER_REVERT_FRACTION); persisting moves and the newest bar are kept.
  The unconditional rule deleted ~36 real M5 news bars/yr in the backtester (whole-frame σ)
  and a different ~24/yr live (fetch-window σ) — e.g. the 2026-09-04 12:30 UTC NFP bar
  vanished from M5+M15 and the 12:35 scan was a byte-identical repeat of 12:30. 3-yr Oanda
  drops are now M5 2 / M15 1 / H1 0.
- providers/oanda.py: transient 401/5xx/network faults retry along RETRY_DELAYS_S=(2,5,10)s
  under RETRY_BUDGET_S=45s. 11 boundary scans in two days had lost their primary frame at
  xx:00:22–:26 UTC; one 2 s retry could not outlast the top-of-hour blip.
- signal_aggregator/_log_signal: a scan whose primary fetch failed logs scan_kind='failed'
  (price 0, no signal); analyze_signal_log.parity_report skips it like 'blocked'.
- test_all.py: 466 tests (+6): retry ladder walks all delays, 5xx blip recovers on the third
  attempt, retry budget stops the ladder, persisting/newest/partially-retraced 10σ bars kept,
  failed-scan row kind, parity skip.
- CLAUDE.md: cleaning rule, retry ladder, failed scan kind, dated current-state note.

Book impact at $0.65 (rule 1 still passes everywhere, both halves positive):
  12mo  M5 557/+112.21/PF1.56 → 560/+108.81/1.54   M15 397/+73.93/1.53 → 398/+74.61/1.53   H1 identical
  3yr   M5 857/+161.3/1.53 → 870/+153.6/1.50   M15 1086/+148.7/1.37 → 1092/+151.9/1.38
        H1 255/+70.5/1.98 → 253/+70.0/1.98   H4 113/+22.7/1.63 → 114/+21.7/1.59
Only trades on the previously-deleted event bars move; the pinned M5/M15 CSVs are no longer
byte-reproducible and should be regenerated at the next re-pin.

Claude-Session: https://claude.ai/code/session_01NDnBxJ2y8GRUM6ZNFJs1hZ
 2026-09-07 11:07:29 -04:00
Harwinder Sekhon 35f1eb17b8 chore: drop unreachable branches outside the signal core (no behaviour change) 
- providers: _to_oanda_instrument / _to_tiingo_ticker / _to_td_symbol no longer
  guess a wire symbol for pairs that are not registered — the registries are
  the only source, so an unknown symbol raises the provider's DataIntegrityError
  instead of silently fetching something else. Every scanned symbol is validated
  against PAIR_INFO at import, so nothing reachable changes.
- main / telegram_bot _fmt_price: the sub-$10 five-decimal branch could never
  fire for gold.
- news_engine._parse_pair: BASE/QUOTE only; the Yahoo "=X" fallback was
  unreachable (its test goes with it).
- dashboard_publisher.event_to_dict: drop minutes_until (frozen at publish
  time, misleading) and is_high_impact — dashboard.js never read either.
- static/dashboard.html: unused id="cal-table".

Tried and reverted: trimming the LogRow fields nothing in analyze_signal_log
reads — the tests read them to verify the loader round-trips those DB columns,
so they stay.

Suite: 460 passed, 0 failed.

Claude-Session: https://claude.ai/code/session_0156YhogLytzj9P2iH1skGbL
 2026-09-06 17:47:45 -04:00
root edefd90aaa oanda: retry once on transient 401/5xx/network faults in candle fetch 
The practice API rejects ~15-20 requests/day with a spurious 401
"Insufficient authorization" while the same token succeeds on adjacent
requests; each one silently cost a full scan cycle on that timeframe.
One retry after RETRY_DELAY_S (2 s) absorbs these plus transient
5xx/network faults. Real credential failures fail both attempts and
raise as before; other 4xx (bad request, 429) are never retried.

Tests: transient-401 recovery, persistent-401 raise, 400 not retried.

Claude-Session: https://claude.ai/code/session_01Tujtmmhxj9mxKjPFSoVjmP
 2026-08-06 23:06:29 -04:00
Harwinder Sekhon 99eb268cf3 audit: alert delivery, provider robustness, backtester NaN/cache guards (Batches C+E+F) 
2026-08-04 Fable 5 QA audit. Three batches sharing one test file, all
verified by runnable repros before fixing, all gated below.

Batch C — telegram alert delivery (verified by h4 harness):
- Failed breach rescan shrinks the cooldown to 30 s instead of
  silencing the symbol for 5 min; breach during cooldown is PARKED and
  fired on the first tick after expiry (was dropped forever).
- Level cache keyed symbol→TF (M5 scans no longer wipe H1/H4 swing
  levels within minutes); breach labels TF-qualified.
- Markdown alert sends get a plain-text fallback (mirroring _safe_edit);
  fingerprint commits only when a message actually landed.
- Per-key asyncio.Lock closes the dedup TOCTOU between auto-scan and
  breach sends (forced-interleave repro produced duplicate alerts).
- _start_scheduler respawns dead per-TF loops (was all-or-nothing).
- Clean WS close now backs off (was a zero-delay reconnect hot loop);
  backoff resets only after a connection that received ticks; TD
  iter_ticks fires on_connect on the first RECEIVED frame (invalid key
  can no longer pin the backoff at minimum).
- Scan-failure logs carry the real traceback (was "NoneType: None").

Batch E — providers (verified by h5 harness):
- fetch_range: 3 consecutive chunk ERRORS raise instead of returning
  silently truncated history (verified 731-day silent gap repro; also
  covers 429 storms); pagination no-progress guard; head-truncation
  ERROR log.
- Single-flight per (symbol,TF): 4 concurrent scans = 1 API call (was
  4); cache records requested depth so a longer-lookback call refetches.
- Multi-bar forming/future tail fully dropped (was iloc[:-1] once).
- Oanda: malformed candles skipped, never 0.0-defaulted; cleaning drops
  non-positive-price rows BEFORE OHLC repair; inf returns neutralized so
  one corrupt bar can no longer disarm the whole 10-sigma filter.

Batch F — backtester internals (verified by h6 harness):
- NaN bar in a trailing window exits at the last valid bar (was NaN R
  poisoning every metric); static-path TIMEOUT NaN guard;
  compute_metrics excludes non-finite R loudly. Zero real-data exposure
  (0 NaN rows in all 10 Oanda parquets) — defensive.
- fetch_history ERROR-logs pre-2026-07-08 cache files (cleaned-bars era)
  when loaded for raw-path resolution.

Gates: suite 333/333 green; pinned M15 12mo book reproduced identically
(397 trades, +78.7111R) after E and after F; M5 HEAD-config baseline
reproduced identically (556 trades, +97.5604R) after F.

Claude-Session: https://claude.ai/code/session_01UnZkzeFcTu9agFg8BDh2tn
 2026-08-04 20:02:57 -04:00
Harwinder Sekhon 083a4e8d63 Provider abstraction: providers/ package with BaseDataProvider template 
One class per provider (TwelveData, Oanda, Tiingo), all subclassing
BaseDataProvider which owns cache, validation/cleaning, freshness +
boundary guarantees, fetch_range pagination, and streaming stubs —
subclasses implement only _fetch_raw. config.json data_providers.active
selects the live feed; data_provider.py / oanda_provider.py stay as
back-compat shims. Module 16 provider-abstraction tests (253 total green).

Claude-Session: https://claude.ai/code/session_01Mc7XKYyPySVXiUpcN82QEK
 2026-07-28 20:09:00 -04:00