Commit Graph

113 Commits

Author SHA1 Message Date
Harwinder Sekhon ec33f8b718 Overhaul news engine: fix false signals, add surprise detection + caching
Improvements from a live-verified audit of the news/calendar pipeline:

- Currency tagging: keyword matching now uses letter-boundary regexes
  and drops ambiguous "us "/"eu "/"uk " tokens — "versus" no longer
  tags headlines as USD (confirmed live false positive polluting the
  XAU/USD sentiment average).

- Fetch caching: calendar events and headlines are cached class-level
  (10 min TTL, 60s retry after empty/failed fetch, lock-guarded for
  concurrent scans). Stops per-scan refetching that was tripping the
  Forex Factory rate limit; TTL stays under the 30-min pre-event
  buffer so the danger window can't go stale.

- Surprise detection: EconomicEvent.surprise is computed from parsed
  actual vs forecast (handles %, K/M/B/T suffixes, negatives). A
  surprise print widens the post-release caution window to 45 min;
  inline releases keep the original 15 min.

- Gold-aware sentiment: headlines are scored per currency, with
  monetary-policy keywords (hawkish/rate hike/tightening vs dovish/
  rate cut/easing/stimulus) inverted for XAU — "hawkish Fed" headlines
  scored +1.0 for gold before, -1.0 now (verified on live feed).

- New sentiment sources: Google News gold-focused query and CNBC
  Economy RSS (verified live; 40 → 74 unique headlines). Added
  cross-feed title dedup so syndicated stories count once.

- events_today now buckets by UTC date (feed timestamps carry US-
  Eastern offsets); --json output includes news_description.

- Removed dead NewsConfig.min_impact and unused HIGH_IMPACT_EVENTS.

Tests: 142 passed (7 new covering boundary matching, surprise
classification, post-event windows, cache reuse, gold inversion).

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
2026-07-10 09:43:11 -04:00
harwinder dad2432d4d Add M5 pullback entry archetype (XAU_PB): +140R combined on 12mo
Second entry pattern alongside the Keltner-breakout: enter on the
retrace to EMA21/KC-mid in an established trend (EMA >=3/4 aligned,
ADX>=20, retrace 0.8-6.0 ATR, resumption bar closed back in trend
direction, RSI in the pullback zone not the extended zone). Detector
lives in the aggregator so live and GPU-backtest paths share one
implementation. Own gates (composite floor 15, confirmation-TF
agreement, MTF opposition, vol regimes) with a diagnostic funnel.

PB stops anchor to the pullback swing (~$8 vs ~$25 breakout), so PB
trades get their own trail arm (0.8R vs 0.6R global) via per-trade
trail overrides in the backtester and strategy-aware Telegram alerts.

12mo M5 validation: PB alone +38.3R/254 trades, positive both halves
and all four full quarters; core breakout trades undisturbed
(+101.9R vs +105.0 baseline). Combined book: 549 trades (~10.6/wk),
+140.1R, PF 1.78, DD 11.3R vs 311 trades/+105.0R before.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
2026-07-09 14:21:59 -04:00
harwinder 3ac36525b0 Add pullback entry pattern implementation plan
Design doc for a second M5 entry archetype: enter on retrace to
EMA21/KC-mid in an established trend instead of at Keltner-breakout
extension. Includes detector spec, geometry, validation protocol
(both-halves + quarterly + displacement checks), and the +105R
baseline to beat. Session/frequency knobs (blocked hours, cooldown)
were tested 2026-07-09 and confirmed already optimal — this is the
remaining evidence-backed entry-side direction.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
2026-07-09 13:51:00 -04:00
harwinder 987ec5bbb0 Add M5 gates 5-6 and trailing-stop exits (+48R → +105R on 12mo)
Gates 5-6 in the M5 filter: block 'high' vol regime entries (chop
transition — 61% WR, negative total while low/normal/extreme all
positive) and block SELLs with RSI<25 (dip already extended, mean-
reverts before target). 365d: 419→311 trades, +41→+48.4R, DD 11.2→8.0R.

Replace the fixed RR-0.5 take-profit with a trailing stop: arm at
+0.6R favorable excursion, trail 0.75R behind best price, stop levels
update at bar close (conservative). The RR-0.5 target was the binding
constraint — winners kept running after TP. Whole trail parameter grid
(arm 0.4-1.2 × dist 0.5-1.25) lands +90-103R, a plateau not a spike;
also +91.7R on the pre-gate 419-trade population. TP1 stays advisory;
Telegram alerts include the arm price and trail distance. max_hold
200→300 bars (trailing exits need room; +10R).

12mo M5, same 311 trades: +105.0R total, PF 2.27, DD 7.9R, 61% WR,
train/val +54.6/+50.4, every quarter positive.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
2026-07-09 13:34:57 -04:00
harwinder db16f209bb Fix M5 signal quality: ATR ordering bug, noise gates, blocked-hour leak (+41R)
ATR was computed after EMAs so ribbon compression detection was dead code
on all timeframes — moved ATR first. Extended M5 swing S/R lookback from
50 to 120 bars. Added four M5-specific filters: min score, MTF agreement,
confirmation TF conviction, and SELL quality cap (A-grade M5 shorts enter
at dip extremes and lose). Fixed backtester blocked-hour check to use bar
close time matching the live bot. Added trade geometry config (spread gate,
SL floor, TP floor RR) and primary_tf mismatch guard in backtester.

Backtest: 1133 trades/+11.5R → 419 trades/+41.0R, PF 1.03→1.37, DD 21→11R.
Train/val split positive in both halves (+19.8R/+21.3R).

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
2026-07-09 12:48:36 -04:00
harwinder d95d31a852 Disable TP tightening: tuned pre-spread, halves total R under real fills
A/B on M15 2025-03-30→2026-05-31 with $0.30 spread + gap-through fills
(511 identical entries, only exits differ):
  factor 0.8 / min_rr 0.4: +36.1R (avg +0.071R, SELL +1.2R)
  factor 1.0 / min_rr 0.5: +66.8R (avg +0.131R, SELL +9.6R)
Worse in both train and validation halves and both directions. The 20%
tighter targets buy +1.8pt win rate but shrink every win while the fixed
spread stays constant. The knob remains for re-tuning with costs included.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
2026-07-08 21:35:59 -04:00
harwinder 5a0af663c1 Add WebSocket dual-connection test script
Probes whether Twelve Data allows two simultaneous WS connections
for one symbol (SDK-based, 30s run, prints per-connection verdict).

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
2026-07-08 21:32:46 -04:00
harwinder b191cee27e Telegram bot: WS heartbeat, breach alerts, alert dedup, interactive menu
WebSocket:
- Send heartbeat every 10s — Twelve Data drops idle connections, so the
  monitor previously died on weekends/quiet hours and churned reconnects
- Log failed symbol subscriptions from subscribe-status events
- Stamp fresh live tick price + age into every alert message

Alerts:
- Re-enable level-breach alerts: WS breach triggers M15 rescan and sends
  the report when actionable (was scanning and discarding the result)
- Dedup: identical (signal, quality, direction) per pair/TF is sent once,
  re-armed after 1h — no more repeat alerts every candle
- Configurable minimum alert quality (C+/B+/A), cycled from the menu
- Trade setup shows dollar risk/reward alongside pips

Interactive menu (/start, /scan):
- Timeframe scans, Scan All, auto-scan ON/OFF toggle, alert-quality
  cycle, status, and help as inline buttons
- Markdown-safe edits: 'message not modified' swallowed, parse failures
  fall back to plain text; exception text escaped

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
2026-07-08 21:32:46 -04:00
harwinder 6a96ed6006 Fix live/backtest parity, SELL-side losses, and indicator repairs (+42% total R)
Live pipeline:
- Drop still-forming candle before analysis (signals no longer repaint)
- Direction-aware MTF agreement + opposition gate (no trade against 2-of-3 TF consensus)
- SELL volatility gate: block shorts in low/normal vol regimes (config sell_vol_regimes)
- Divergence: 40-bar lookback, price/indicator swings must align within ±2 bars
  (fired on 0.5% of trades before, 29% after), trending weight 2.0→1.0
- RSI regime_aware by default; transitioning regime only signals at extremes
- ADX ±DM tie no longer nudges bearish; TP2 clamped beyond TP1

Backtester realism:
- Charge $0.30 round-trip spread (--spread), gap-through SL/TP fills at bar open
- Resolve trades on RAW price path (cache stores raw bars, cleaning runs on load)
- Weekend/session filters use bar-close decision time to match live wall clock

Validated per-step on M15 2025-03-30→2026-05-31 under identical pricing:
+46.9R → +66.8R, win rate 69.9% → 73.0%, SELL side -2.3R → +9.6R.
Tested-and-reverted: ROC weight removal (-13R), BB fade suppression (-11.6R),
candlestick de-weight (-2R) — composite thresholds are co-tuned with weight mass.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
2026-07-08 21:32:46 -04:00
harwinder 998f5e8616 Fix trade setup generation broken by TP tightening factor
The TP floor calculations used raw min_rr (0.4) but after 20% tightening
the effective RR dropped to ~0.32, failing the min_rr check and returning
None for all trade setups. Pre-compensate the floor with min_rr/tf so
post-tightening RR still meets the minimum.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
2026-05-24 18:22:57 -04:00
harwinder a731b7c8df Tighten TP targets by 20% for faster hits (configurable tp_tightening_factor=0.8)
Applied uniformly across all 3 trade setup functions. Lowered min_rr_ratio
0.5→0.4 proportionally to preserve trade acceptance rate. SL unchanged.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
2026-05-22 10:44:52 -04:00
Harwinder 643952a8e2 Enable M5 auto-scan alerts and remove live dashboard
M5 already broadcast through the shared _scan_timeframe path but was
documented as data-only (docstring, CLAUDE.md, DOCUMENTATION.md, plus a
phantom `if tf == M5: return` step that never existed in code). Align all
docs/comments so M5 alerting at C+ grade is the intended, documented behavior.

Remove the live dashboard entirely per request: the 30s refresh loop,
per-scan refresh, boot send, and the dashboard-only stats layer
(_signal_stats / _stats_updated / _update_signal_stats and its call sites,
_format_stats_dashboard and its helper constants). Drop now-unused ZoneInfo
import and dashboard mentions from help/autoscan/boot messages and the test
suite. The WebSocket level-breach monitor and level cache are untouched.

Tests: 136 passed, 0 failed.

Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
2026-05-20 12:45:27 -04:00
harwinder b61818330b Fix weekend filter: use precise forex market hours (Fri 22:00–Sun 22:00 UTC)
The old `weekday() >= 5` check blocked all of Sunday, but the forex
market reopens Sunday 22:00 UTC (Sydney open). Now uses proper time-based
check so the bot fetches data and generates signals on Sunday evening.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
2026-05-17 19:23:21 -04:00
harwinder 0e6f5d74a7 Add backtest scripts to version control
Track gpu_backtest.py, diagnose_filters.py, and instructions.
Data files (CSVs, parquet cache) remain gitignored.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
2026-05-15 19:49:30 -04:00
harwinder c505829dfb Reduce A-grade TP when exhausted: RR 0.75→0.50 on 2+ exhaustion tags
A-grade trades entered exhausted moves with TP too far (0.75R), causing
57.7% WR and +0.25R over 30 days. Store exhaustion_count on SignalReport
and scale TP to B-level (1.0×) when A-grade has 2+ exhaustion tags.
Backtest: A-grade 73.1% WR / +3.25R, B/C unchanged.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
2026-05-15 19:46:31 -04:00
harwinder 08742c5063 Align FilterConfig with XAU strategy: blocked hours [0,1,2], session 3-23 UTC, Mon-Fri
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
2026-05-15 15:36:03 -04:00
harwinder 3f7d24734d Add M5 to Telegram scan buttons and Scan All TFs
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
2026-05-15 15:26:00 -04:00
harwinder 5047601d46 Reduce blocked hours, enable M5 alerts, show NY time on dashboard
- Blocked hours reduced to 0-2 UTC only (daily rollover)
- M5 signals now broadcast to Telegram like M15/H1/H4
- Live dashboard shows New York time (auto EDT/EST via zoneinfo)

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
2026-05-15 15:23:48 -04:00
harwinder 8ed6454ea5 Refocus on XAU/USD: add momentum indicators, tune TP, exhaustion block
Strip EUR/USD, GBP/USD, USD/JPY pairs and strategies from entire codebase.
Add 3 new scored indicators (ROC, Keltner/TTM Squeeze, ADX slope) and
enable MACD-as-pillar + divergence detection for early trend entry.
Introduce "transitioning" regime (ADX < 25, slope > 3.0) to catch trend
births before ADX confirms. Gold-specific composite weights per regime.

Tune TP multipliers: quality-tiered scaling (A+/A get 1.5-2.0x ATR via
XAU ATR method, B/C use swing-level targeting). Backtest: +24R total,
65% WR, PF 1.12 over 365 days.

Add momentum-confirmed exhaustion block: hard-reject trades when 2+
static exhaustion signals fire AND ROC/MACD confirm fading momentum.
Prevents late trend entries without killing strong trend trades.

Telegram: simplify buttons to timeframe-only labels, add "Scan All TFs"
multi-timeframe summary, add /scan command, fix BB/KC level breach keys,
fix price formatting (.2f for gold), fix deprecated asyncio call, fix
"1 pairs" grammar and 60s/30s dashboard refresh mismatch.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
2026-05-15 10:51:51 -04:00
harwinder 98daf71a52 Add web dashboard (Redis + FastAPI + WebSocket), scope XAU penalties
Signal daemon (signal_web.py): candle-boundary scanner mirroring
telegram_bot.py, publishes serialized SignalReport JSON to Redis.
Web API (web/app.py): thin FastAPI reader behind Apache, serves
/api/signals REST + /ws WebSocket from Redis pub/sub.
Frontend (web/static/index.html): 3-pair grid, color-coded signals,
indicator breakdown per timeframe, trade setup display.

Also: XAU-specific penalty scoping in signal_aggregator (softer
confidence penalties for gold, original strict values for forex),
pair-aware session filter routing XAU to its own blocked hours.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
2026-05-14 23:15:37 -04:00
harwinder 57769967d1 Fix news engine (keywords, calendar fallback, RSS URL), remove Telegram guards
News engine: expand sentiment keywords (18%→60% headline coverage), add
MyFXBook RSS as fallback calendar source when Forex Factory returns 429,
add retry/backoff on calendar API rate limits, update stale ForexLive URL
to investinglive.com, add date formats without seconds for MyFXBook RSS.

Telegram bot: remove flood control, rate limiter, and dashboard failure
tracking — all sends go directly through bot.send_message/edit_message_text.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
2026-05-14 13:42:32 -04:00
harwinder bf9609708c Fix duplicate-index crash in data validation, add Telegram flood control
- Move index dedup before OHLC consistency fix in _validate_and_clean
  to prevent crash when df.loc[dup_idx] returns Series instead of scalar
- Add global flood backoff: detect RetryAfter, pause all sends/edits
  until ban expires instead of retrying and escalating the ban
- Stop dashboard edit-fail → new-message spam cascade: track consecutive
  failures, only resend after 3 failed edits
- Increase dashboard refresh from 5s to 30s, min send interval to 1.5s
- Add comprehensive QA test suite (140 tests across all 7 modules)

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
2026-05-14 13:04:28 -04:00
harwinder 235241f95a Harden Telegram bot, add EUR/GBP strategies, parallelize scans
Telegram reliability: rate-limited sends (1 msg/s), dashboard lock,
WebSocket JSON/price error handling, markdown escaping, level cache
TTL, cooldown cleanup, breach task tracking, graceful shutdown.

Telegram speed: parallelize 3-pair scans with asyncio.gather(),
increase thread pool 2→4, parallelize boot-up initial scan.

Strategy: add EUR/USD and GBP/USD pullback-buy strategies with
per-pair configs, optimize XAU TP to 0.9×ATR, block hours 8/9
for XAU and hour 17 for GBP.

Backtest: add --max-hours flag to evaluate.py for time-exit support.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
2026-05-13 21:07:49 -04:00
Harwinder 1f3f69c21d Fix dashboard TF key mismatch, increase refresh to 5s
Timeframe keys stored as enum values (15min/1h/4h) but dashboard
looked for M15/1h/4h — GBP/USD showed no price or signal text.
Now uses Timeframe enum directly with display labels. Reduced
dashboard refresh from 60s to 5s with throttle guard.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
2026-05-13 18:51:02 -04:00
Harwinder 439e179128 Add live dashboard with 60s auto-refresh, disable breakout alerts
Replace per-scan Telegram messages with a single dashboard message
that edits in-place every 60 seconds. Each pair shows all TFs
horizontally with short signal labels (Buy/Sell/Neutral). WebSocket
live prices feed into the dashboard. Level-breach alerts disabled
(WS still updates cache silently). Removed /stats command.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
2026-05-13 18:34:44 -04:00
harwinder 6b1ffff66a Fix live session filter overriding XAU strategy signals
The general session filter ran after analyze_from_dataframes() and
wiped out XAU oversold bounce signals at most hours. Now skips the
session filter when the XAU strategy has already fired — the strategy
has its own backtested hour filter.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
2026-05-13 17:31:46 -04:00
harwinder d63544b75d Add XAU/USD oversold-bounce strategy (79.7% WR, PF 2.06)
BUY-only strategy for gold: enters when M15 RSI drops below oversold
threshold while price remains above EMA200 on all three timeframes
(M15, H1, H4). Uses ATR-based TP/SL (0.7x/2.0x) sized from M15 ATR.
Isolated to XAU/USD only — other pairs completely unaffected.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
2026-05-13 17:25:49 -04:00
harwinder 1e5047e20f Backtest tuning, data freshness guard, Telegram improvements
- Drop USD/JPY from scan pairs (zero signals across all timeframes)
- Disable divergence indicator (zero impact in 14-config lab test)
- Parallelize evaluate.py with ProcessPoolExecutor (16 cores)
- Add --pair CLI filter to evaluate.py for single-pair backtests
- Add data freshness guard: stale cache auto-invalidated, primary TF
  stale data suppresses signals to prevent trading on old data
- Add /cache command: shows fetch age and last bar per pair/TF
- Add /clearcache command: wipes in-memory cache for fresh API fetch
- Remove BB from WebSocket breach alerts (keep structural levels only)
- Show live prices only for WS-monitored pairs (EUR/USD, XAU/USD)
- Replace arrow emojis with text tags in indicator lines

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
2026-05-12 22:56:55 -04:00
harwinder 6fe1a3e0f0 Add project documentation and MT5 integration plan
DOCUMENTATION.md: Two-section guide covering the full project —
plain-English trader guide (indicators, scoring, trade setups) and
developer reference (architecture, data flow, every threshold/weight).

mt5.md: Architecture plan for automated order execution via MT5 bridge
(REST API on Windows, async client on Linux, risk-based lot sizing).

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
2026-05-11 19:50:21 -04:00
harwinder 4db6121ca7 Fix trade setup generation, M5 data-only mode, WS symbol limits, command menu
- Trade setup TP now scales to SL distance ensuring min RR is met when
  no viable swing target exists; skip swing targets with poor RR
- Add float tolerance to RR gate to prevent edge-case rejection
- M5 auto-scan runs for level cache only, no Telegram broadcast
- WebSocket limited to EUR/USD + XAU/USD (Grow plan constraint)
- Register bot command menu via set_my_commands on startup

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
2026-05-11 18:58:40 -04:00
harwinder 5e12307b1f Suppress Telegram messages when no actionable signals found
Auto-scan still runs and updates the level cache every cycle,
but only sends to Telegram when quality A+/A/B/C signals exist.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
2026-05-11 18:03:49 -04:00
harwinder 4be7077c0a QA fixes: H4 timing bug, /status + /help commands, WS cold-start
- Fix H4 boundary calculation using total minutes (hour*60+min) not just minute
- Add /help command listing all bot commands
- Add /status command showing WS connection, live prices, level cache, cooldowns
- Run initial M15 scan on boot to populate level cache before WS starts
- Update /start message to mention new commands

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
2026-05-11 17:16:07 -04:00
harwinder 736e33302b Tune engine from backtesting: wider SL, capped TP, counter-trend filter
Backtest-driven fixes (0% win → 33% win, PF 0.00 → 1.05 over 7 days):
- SL: swing buffer 0.5→1.0 ATR, min distance 1.0→1.5 ATR (more room to breathe)
- TP: cap swing targets at ATR×3.0, fallback ATR×2.0 (was targeting unreachable levels)
- Mature-move TP: when score>30 (A/B grade), tighten TP to ATR×1.5 (move already extended)
- Counter-trend penalty: -15 confidence when signal fights regime direction
- EMA200 penalties halved: -20→-10 (was killing EUR/USD signals)
- Score thresholds lowered: BUY/SELL ±15→±12, STRONG ±40→±35
- Quality grades relaxed, C-grade now gets trade setups and Telegram alerts
- min_rr_ratio: 1.5→1.2
- Gitignore backtest/ directory

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
2026-05-11 17:00:12 -04:00
harwinder a7bed99caa Buy signal included 2026-05-11 16:37:45 -04:00
harwinder dc9c76a716 Pro tier upgrade: quant rearchitect + WebSocket level-breach monitor
- Restore XAU/USD (Pro plan), drop USD/CAD, raise rate limit to 55/min
- Reduce signal engine to 4 uncorrelated pillars (EMA, RSI, BB, Candlestick)
  with regime-adaptive weights (trending boosts EMA, ranging boosts RSI+BB)
- Add structure-based S/R (swing highs/lows) for trade setup targeting
- Fix confidence death spiral: all adjustments now additive, not multiplicative
- Dynamic R:R trade setup: SL behind swing level, TP at next structure level,
  reject if R:R < 1.5
- Fix news engine for all 4 pairs (was hardcoded to USDJPY)
- Add WebSocket monitor: streams live ticks, triggers M15 scan on level breach
  with 5-min cooldown per pair — catches breakouts between candle boundaries
- Add correlation filter (EUR/USD+GBP/USD, XAU/USD+USD/JPY exposure warnings)

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
2026-05-11 16:33:23 -04:00
harwinder a15dc40050 Replace XAU/USD with USD/CAD — TwelveData has no gold data
XAU/USD is not available in TwelveData's symbol list.
Swapped to USD/CAD across config, scan pairs, Telegram buttons,
and added CAD to tracked news currencies.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
2026-05-11 15:50:48 -04:00
harwinder b3e0da7455 Add 4-pair candle-boundary auto-scanner (free tier safe)
Expand from 2 to 4 pairs (EUR/USD, USD/JPY, GBP/USD, XAU/USD) with
automatic scanning at M15 and H1 candle close boundaries. Scheduler
staggers H1 by 60s to stay within 8 credits/min; daily budget is
480 of 800 allowed. Cache TTL is now candle-duration-aware so cross-TF
confirmation data is reused without extra API calls. Spread estimate
uses primary-TF data instead of fetching M5 separately.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
2026-05-11 15:30:11 -04:00
Harwinder 037dec643e Add EUR/USD support and 4-button Telegram scanner
Add EURUSD pair with dynamic pair info lookup. Telegram bot now shows
4 scan buttons: EUR/USD 15M, EUR/USD 1H, USD/JPY 15M, USD/JPY 1H.
Each indicator displays directional arrows in the message output.
Price formatting adapts per pair (5dp for EUR, 3dp for JPY).

Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
2026-05-10 23:49:36 -04:00
Harwinder 3a09dfd62f Strip backtesting, rebalance indicator weights toward leading signals
Remove backtest.py, backtest_gpu.py, metrics.py, local CSV data,
Dukascopy loader, and all backtest CLI modes from main.py. Rebalance
engine weights: leading indicators (divergence, RSI, stochastic, BB)
now carry 66% of signal weight vs 34% for lagging (EMA, MACD). Add
ws_twelvedata.py for live tick streaming. Drop pyarrow and tqdm deps.

Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
2026-05-10 23:40:15 -04:00
Harwinder 353ff05267 Add USAGE_LINUX.md — bot and CLI run instructions for Linux server
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
2026-05-07 23:12:55 -04:00
Harwinder a596e7cab2 Add async Telegram bot — live scan button with trade setup
Introduces telegram_bot.py: python-telegram-bot v20+ async bot that
presents a persistent Scan Signal inline button. Each press fetches a
live USD/JPY signal via SignalAggregator and formats it with emoji,
MTF breakdown, top bull/bear reasons, and warnings. STRONG_BUY /
STRONG_SELL signals additionally render the full trade setup (entry,
SL, TP1, TP2, RR). Adds python-telegram-bot>=20.0 to requirements.txt.

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
2026-05-07 23:05:01 -04:00
Harwinder c72e8e669b Round-2 weight optimisation — reduce EMA trend, boost stochastic
GPU sweep (36 combos, RTX 4070 SUPER) on H4 2022-2026 relative to the
round-1 baseline revealed EMA trend was over-weighted at 3.6. Halving
it and nudging stochastic up yields cleaner, higher-quality signals.

Weight changes (backtest_gpu.py _BASE_W + engine.py):
  ema_trend:  3.6 -> 1.8  (over-weighted; halving tightens signal filter)
  stochastic: 1.6 -> 2.4  (x1.5; improved WR to 40.4% in sweep)
  macd:       3.3          (unchanged — dominant signal, kept as anchor)
  adx/ichimoku/rsi/bollinger: unchanged

Combined result vs round-1 (H4 local data 2022-2026):
  Trades:      160 -> 73   (-87 — filters out marginal setups)
  Win Rate:   38.1% -> 43.8%  (+5.7pp)
  Profit Factor: 1.19 -> 1.52
  Return:    +21.30% -> +23.86%  (+2.57pp)
  Max DD:     -8.52% -> -9.32%
  Sharpe:      0.47  ->  0.71   (+51%)
  Sortino:     1.91  ->  3.06   (+60%)

Fewer trades at much higher quality. The lower EMA weight means only
strong, multi-indicator confluences clear the grade threshold — the
73 remaining trades have 43.8% WR vs 38.1% and expectancy nearly 3x.

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
2026-05-07 22:24:48 -04:00
Harwinder 3d9f4dc201 Enable GPU scoring via CuPy — RTX 4070 SUPER confirmed 6.3x speedup
Install nvidia pip packages for CUDA 12 runtime DLLs (no full toolkit needed):
  nvidia-cuda-nvrtc-cu12, nvidia-cuda-runtime-cu12,
  nvidia-curand-cu12, nvidia-cublas-cu12

Add _register_nvidia_dlls() in backtest_gpu.py that auto-discovers and
registers nvidia package DLL directories (site-packages/nvidia/*/bin/)
via os.add_dll_directory() before CuPy is imported. Suppresses the
spurious "CUDA path could not be detected" warning from cupy._environment.

Backend line now reads: CuPy 14.0.1 (RTX GPU)
Benchmark: scoring pass on 100k bars — NumPy 1.1ms vs CuPy 0.18ms = 6.3x.
Results are identical to CPU path (deterministic).

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
2026-05-07 22:22:21 -04:00
Harwinder e30535f5e9 GPU backtester, optimised weights, local CSV data pipeline
## What changed

### backtest_gpu.py (new — ~1,300 lines)
Standalone GPU-accelerated backtester completely separate from backtest.py.
Computes all indicator series in one vectorised O(N) pass over the full price
history, then replays positions with a sequential state machine.

Key design:
- VectorIndicators: pandas ewm/rolling (C-optimised) for all 7 indicators
- CuPy GPU backend with automatic NumPy fallback (no hard dependency)
- Two-phase API: prepare_arrays() once, score_and_run() many times
- --optimize flag: 36-combo weight sweep (1 baseline + 7 ablations + 28
  multiplier runs) — calls prepare_arrays() once, replays all combos in seconds
- Full CLI: --tf, --local, --start/--end, --regime-filter, --regime-tf,
  --session-filter, --optimize, --journal, --journal-csv

O(N) vs O(N^2): backtest.py recomputes all indicators per bar (O(N^2));
backtest_gpu.py computes each indicator series once and reads bar T in O(1).
On 100,000 H1 bars: ~4 seconds vs >10 minutes.

### data_provider.py
Added load_local_csv() method — reads MetaTrader tab-delimited CSV exports
from market_data/ directory. Format: Time|Open|High|Low|Close|Volume|Spread
(spread column dropped on load, timestamps localised to UTC). Applies the same
_validate_and_clean() pipeline as the Twelve Data path. LOCAL_CSV_MAP maps
each Timeframe enum to its filename.

market_data/ files added (MetaTrader exports, UTC):
  USDJPY_M15.csv  100,000 bars  2022-05-04 to 2026-05-08
  USDJPY_H1.csv   100,000 bars  2010-04-23 to 2026-05-08
  USDJPY_H4.csv    25,855 bars  2010-04-22 to 2026-05-08
  USDJPY_D1.csv     5,020 bars  2010-04-22 to 2026-05-08

16 years of H1/H4/D1 vs ~6 weeks of H1 on Twelve Data free plan. Use --local
for all backtests; Twelve Data reserved for live signal feed.

### engine.py — optimised indicator weights
36-combo sweep on H4 2022-2026 found MACD and EMA trend are the dominant
signals; RSI and Bollinger are net drag on this trending pair.

Old → new weights:
  ema_trend:  1.2 → 3.6  (×3 — strongest directional signal)
  macd:       1.1 → 3.3  (×3 — ablating it is the single worst move: -42%)
  stochastic: 0.8 → 1.6  (×2 — clean timing, 40% WR at this level)
  adx:        1.0 → 0.5  (÷2 — partially redundant with EMA stack)
  ichimoku:   1.1 → 0.55 (÷2 — lags too much at H4, hurts in ranging)
  rsi:        0.3 → 0.0  (ablated — net drag across all test windows)
  bollinger:  0.9 → 0.0  (ablated — counter-trend noise on trending pair)

RSI and Bollinger are still computed and displayed in live output for
reference; weight=0 means they contribute nothing to the composite score.

Combined result vs baseline (H4 2022-2026, local data):
  Return:   +7.12% → +21.30%  (+14.2pp)
  Max DD:   -9.05% → -8.52%   (tighter — gains did not come at expense of risk)
  Sharpe:    0.23  →  0.47    (2x improvement)
  Sortino:   1.38  →  1.91
  Win Rate: 36.3%  → 38.1%

### backtest.py
Added 16-thread CPU parallelism for signal pre-computation via
ThreadPoolExecutor. _precompute_all_signals() dispatches each bar to a worker;
pandas/NumPy C extensions release the GIL enabling true parallel execution.
Results are collected and replayed in the main loop, replacing the per-bar
recompute. Also added --regime-tf CLI flag so regime filter timeframe can be
specified independently of the primary TF.

### signal_aggregator.py
EMA200 distance gate: confidence *= 0.55 when price is within 0.5% of EMA200
(chop zone) or beyond 1.5% (overextended). Sweet spot is 0.5-1.5% from EMA200
where trending momentum is cleanest. Strongest single filter found: lifted
cumulative return +13.88% to +35.50% in ablation window.

### ENGINE_LOGIC.md (new)
Full reference document covering: architecture, data pipeline, all indicator
logic, composite scoring formula, optimised weights with rationale, regime
classification, EMA200 gate, MTF alignment, grade system, backtesting results,
filter catalogue, and configuration reference.

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
2026-05-07 22:12:43 -04:00
Harwinder a687db9413 Add EMA200 distance gate — strongest single signal-quality filter
Across 2024 trend / 2023 chop / 2026 hostile windows, distance from the
H1 EMA200 was empirically the most discriminative variable. Sweet spot
0.5-1.5% from EMA200; closer = false-breakout chop, farther = chase.

Outside the band, confidence is multiplied by 0.55 so the trade falls
below the B-grade threshold and is rejected.

Cumulative return across the 3 windows: +13.88% baseline -> +35.50%.
Win rate improved in every regime; 2023 chop turned positive (-2.56% to
+14.94%).

Also extends the trade journal with per-timeframe MTF scores, RSI at
entry, and EMA200 distance for future filter mining; surfaces ema_trend
in key_levels so the gate can read it.
2026-05-07 19:28:19 -04:00
Harwinder b7d03a9b61 Document session filter and auto-toggle flags in USAGE
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
2026-05-07 18:44:41 -04:00
Harwinder b38ed17c96 Add session filter + auto-toggle, extend trade journal diagnostics
Session filter restricts trades to a UTC hour window (default 07-16,
London + NY) — opt-in via --session-filter, with --session-start /
--session-end overrides. Wired into both the backtester (cheap hour
check next to the regime gate) and live mode (downgrades signal to
NEUTRAL/F when current UTC hour is outside the window).

Auto-toggle (--auto-session-filter) tracks rolling mean pnl_pct over
the last N closed trades; when it falls below threshold, the session
filter activates, releasing once expectancy recovers. Default lookback
10 trades, threshold -0.3%.

Trade dataclass extended with diagnostic fields captured at entry:
entry_hour, entry_dow, ADX, ATR%, market_regime, volatility_regime,
mtf_agreement. These let --journal-csv exports surface loss-pattern
clusters when a window underperforms.

Validated on 2024 trend / 2023 chop / 2026 recent windows. Auto-toggle
preserves ~50% of trend edge while still cutting ~25% of recent-regime
losses — pure regime filter is still best on cumulative, so both
session flags ship default-off.

Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
2026-05-07 18:38:49 -04:00
Harwinder dcd98fffa2 Swap raw requests for Twelve Data Python SDK
data_provider now uses TDClient.time_series().as_pandas() instead of
hand-rolled REST calls. Throttle, validation, and parquet caching are
unchanged. Adds twelvedata>=1.2.0 to requirements.txt.

Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
2026-05-07 18:38:38 -04:00
Harwinder 70990532a3 Compact hierarchical live signal output
Restructure print_signal_report so decision (signal · grade · score ·
confidence · MTF) sits at the top, followed by trade setup, context line,
MTF breakdown, top supporting/opposing reasons in two columns, warnings,
and an optional detail block under --detail. Adds _compact_reason helper
that strips known indicator name prefixes via _INDICATOR_PREFIXES so
reason lines stay short.

Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
2026-05-07 17:53:50 -04:00
Harwinder 53e6e07829 Wire regime filter into live mode
--regime-filter previously affected only backtests. Live mode now also
checks higher-TF ADX after analysis: signals are downgraded to NEUTRAL
with a warning when ADX < threshold, and confirming when above.

This brings live behavior into parity with what the backtester measures.

Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
2026-05-07 17:40:39 -04:00