sql-integration
13 Commits
| Author | SHA1 | Message | Date | |
|---|---|---|---|---|
 Harwinder Sekhon
|
2cc5915b00 |
Daily-trend veto on M15 + lot-size / already-open lines (campaign 2026-09-30), and the 2026-10-01 whole-codebase bug review
Two bodies of work that touch the same files, so one commit. Direction and setup-size campaign (2026-09-30 → 10-01, owner's RISK-FIRST rule, written before the runs): - SIGNAL: M15 daily-trend veto (factory override 7, xau_strategy.daily_trend_veto "both"). A finished setup against a clear daily trend (closed D1 bars, Sunday stubs dropped; close vs its 50-bar EMA AND vs the close 60 kept days earlier) is removed as the last step of the core. One definition (signal_aggregator.daily_trend_series) for live, the bar replay and the twin's host side; D1 is a required frame of the M15 scan; mirrored in cuda_signal_kernels.py K2. W3 824 / +205.12R / PF 1.73 / max DD 12.4R vs 1105 / +195.79R / 1.49 / 20.9R; 2019 → 2026 1473 / +223.33R / 1.42 / 41.5R vs 2091 / +192.53R / 1.24 / 64.8R; unseen holdout −4.62R vs −6.63R. - FAILED, not shipped: the veto on M5 (W3 drawdown 10.1R → 15.2R) and on H1 (−9.7R), buy-only and sell-only on M15, the wide-stop cap (xau_strategy.wide_stop_cap_atr stays 0.0, a research knob). - New M15 pins: W2 dfc647ee…, OOS e87b6de1…, W3 ad3b0dbd…. M5 / H1 / H4 pins unchanged. --set xau_strategy.daily_trend_veto=off reproduces the previous M15 pins byte for byte. Twin and reference script MD5-equal on every window; recorder and feature-export parity re-proved; base-rate table rebuilt. - DISPLAY (books byte-identical): exposure.py — the lot size for trade_size.risk_per_trade (new config.json block, off by default, private to the owner) and the count of alerted same-direction plans still open (signals.alerted_at, read-only open_alerted_setups); the daily trend in words on Telegram, the CLI and the dashboard; signals.daily_trend logged. - providers/oanda: a date window with no bars is an empty page, not an error (a D1 pad starting on a Saturday could not be fetched). - Reports only, nothing shipped: seven context screens (no state nominated) and eight outside data sources (none adds a forecast to gold's own daily trend). - Independent verification (report section 5.13): every figure reproduced on both scripts, scoring identical to the previous commit on all four timeframes, no look-ahead. Caveats recorded there: this is a risk filter, not a more accurate direction call; nine tenths of the 2019 → 2026 gain is 2022; the neighbour clause is thin (a simple 100-day average misses the R line by 1.2R); "never adds a trade" is an observation, not a guarantee. Bug review of the whole codebase (2026-10-01). Signal path clean. 17 defects elsewhere, plus 4 found by an independent review of the fixes; all fixed, each with a test; no trade book changed: - telegram_bot: one _deliver_alert for the candle scan and the level break. The row is stamped alerted_at BEFORE the send, on a one-worker alert-db pool, and un-stamped when no chat received it or the task is cancelled mid-stamp, so two alerts at one candle close count each other and no SQLite write runs on the event loop. The 5-bar cooldown counts bars of the scan's own frame (== gpu_backtest._apply_cooldown; clock time released the 22:00 UTC bar after the settlement pause) and is restored from signals.db at start-up. The lot-size line goes to the owner's own chat only (a group the owner ran /autoscan in used to receive it). /status gains a news-model line and escapes failure details. - signal_log: mark_alerted returns the stamp it wrote; unmark_alerted; last_alerted_bars (read-only). - advisory_outcome: candle continuity is judged bar by bar inside the walk; a hole after a trade's exit no longer voids it. The stored-candle replay resolves 108 of 111 logged setups instead of 99, all equal to the live resolver's R. - providers/oanda: an empty date window is accepted only when it is 5 days or shorter and no malformed candle was skipped; a longer one is retried, then refused, instead of being cached as history with a hole. - news_engine: blank RSS titles skipped; one unreadable headline no longer costs the local model its whole batch (checked on the real bundle); model health recorded; no DeepSeek audit call in a process that does not archive; event-direction cache keyed per release; peek_week_events never fetches. - llm_client: chat_json never raises and never logs the API key (requests quotes the Authorization value in InvalidHeader; error bodies are redacted). - signal_aggregator: the scan's news block is guarded; the replay path (use_news=False) assigns exactly the values it did before. - agent_panel: PANEL_PROMPT_VERSION v2.2 — same prompt text, corrected input (event timing over the cached calendar week, headlines newest first). scripts/panel_replay.py replays the logged payload per persona; scripts/label_scan_outcomes.py counts with an ISO cutoff. - test_all.py: no ALTER TABLE … DROP COLUMN (the live box has SQLite 3.34); tests that write Parquet print a notice and return without pyarrow. Verification after the last code change: suite 632 passed, 0 failed (also with both Parquet engines blocked and FB2_BACKEND=cpu, the live box's view); the five reference MD5s reproduce on both backtesters, 0 cache fallbacks; Module 37 green; static Python 3.9 / SQLite 3.34 check clean on every touched file. Evidence: backtest/research/daily_trend_campaign_2026-09-30.md, context_screens_2026-10.md, new_inputs_2026-10.md; CLAUDE.md sections 2, 4, 6, 8, 9, 10 and 11; pair_signals/xauusd.py docstring (M15 override 7). |
||
|
Harwinder Sekhon
|
ad14ea4fed |
news corpus for a local scorer + agent panel v2 + dashboard read-only (2026-09-17); books byte-identical
Agent panel v2: personas return p_win anchored to the book base rates with a code-computed decision context; verdict = equal-weight mean (verdict_source "mean"); only the macro persona runs live (agent_panel.personas ["macro"]); prompt version and exact payloads logged; offline replay harness scripts/panel_replay.py; base-rate table agent_panel_base_rates.json (scripts/build_panel_base_rates.py). Backtester `strategy` column now carries the generator's own stamp (label-only; five reference books re-pinned, all 62 other columns identical). News corpus (after evaluating the owner's ML-NEWS scikit model against DeepSeek: golden set 39/78 vs 78/78, out-of-sample agreement 64% against a 90% self-agreement ceiling; verdict: fallback only, not a replacement): - NewsSentiment.xau_score + archive column xau_score keep DeepSeek's gold score for every headline (930 of the first 1,546 DeepSeek rows had none); an existing newsdata.db gains the column in place on first open - headlines naming no tracked currency are scored and archived (currencies_json []) but never returned to the platform - _apply_llm_sentiment sends every new headline in its cycle, 25 per call; a failed call ends the cycle's sending Dashboard publisher never writes a database: main() calls app_config.set_db_writes_for_process(False), honoured by signal_log_enabled() and news_archive_enabled() (owner decision: the bot is the single writer). Test suite: the hermetic app config now points both writers at a suite-private temp dir — the RSS fixture tests were writing fixture headlines into the real logs/newsdata.db — with a Module 32 test pinning it. 529 tests. |
||
|
Harwinder Sekhon
|
9adaee65d2 |
backtest cleanup: drop four research/one-time scripts; books and stale cache moved out (owner decisions 2026-09-15)
Discussion-first cleanup of backtest/ and scripts/. Nothing in the signal
core or the backtester changed.
Removed (last commit containing them:
|
||
|
Harwinder Sekhon
|
daa2369ccb |
backtest: --record/--record-bars Parquet recorder + 2-year M5/M15 replay with indicator weights and calendar
Adds an opt-in recorder to the backtester so a replay keeps the evidence each trade setup was decided on: every indicator's vote, value, confidence, base weight, regime multiplier, effective weight and contribution on the primary and both confirmation timeframes, penalties, the setup and outcome, and the Forex Factory calendar events around the trade (4 h before entry to exit, all-day rows on the New-York date). --record-bars adds one float32 row per analyzed bar with the same votes plus 1 h/4 h/24 h forward moves in ATRs. Output is Parquet in a folder per run with a manifest carrying the engine/gate fingerprints, the regime weight table in force and the book MD5. Nothing in the signal core changes: the trade book is byte-identical with recording on or off (control-vs-recorded MD5 equal on both 2-year runs, M5 781 / +146.07R / PF 1.52, M15 793 / +117.62R / PF 1.41 at $0.65). Module 33 adds 12 tests (498 total): snapshot reproduces the engine's composite, absence is no vote, calendar join edges/phases/holiday, write/load round-trip, _replay_span parity with recording on, CLI flags, import and source guards. Also: backtest/replay_scorecard.py (first-look scorecards, calendar buckets, in-book and per-bar what-ifs, --check invariants), scripts/ff_calendar_scraper.py restored from git history (the only way to regenerate the gitignored calendar file), report in backtest/research/replay_recording_2026-09-09.md. Claude-Session: https://claude.ai/code/session_01Nc9Ninzyi18QkWezrxeBvB |
||
|
Harwinder Sekhon
|
e5d0260814 |
news archive: keep every headline + its DeepSeek score in its own DB
Headlines only ever survived as a side effect of signal logging, and logs/signals.db was rotated around 2026-09-04 — destroying the 3-week, 4,054-headline corpus news_sentiment_value_2026-09-04.md was written on. logs/ is gitignored, so git could not recover it and nobody noticed for four days. There is no way to backfill a fetch window once it passes. news_archive.py writes to its OWN database (logs/newsdata.db) so a signal-log rotation cannot take the news with it. One row per distinct headline (sha256 key) with the text, description, link, publisher, publish time, the score, and — new — which scorer produced that score. Provenance is the load-bearing part. DeepSeek scores 78/78 on the 2026-08-06 golden set; the keyword scorer it overrides scores 24/78 (31%), and that is what silently stands on no key, disabled, daily cap, timeout, HTTP/parse failure, or the >25-new-headlines-per-cycle batch overflow. Until now both landed in the same column, indistinguishable, so a model trained on the mix would inherit the keyword scorer's errors. NewsSentiment.score_source now records it, mirrored into signals.db via signal_log._headline_to_dict, and teacher_model/teacher_prompt_sha1 keep labels made under a changed prompt from being pooled with older ones. Writes are one atomic UPSERT per row, upgrade-only: an llm score always replaces a keyword one (the batch-overflow tail gets labelled on the next window in a long-running process — verified 13/13), a keyword score never overwrites an llm one, and re-seeing a headline always counts. A read-then-write pair lost increments when the bot, the dashboard publisher and the tests first saw a headline together, and the WAL switch itself needed _with_lock_retry; both are regression-tested with three concurrent writers. The hook sits at the end of _fetch_news_sentiment_raw, BEFORE _is_neutral_headline runs in get_fundamental_context. Archiving downstream of that ±0.1 filter is why signals.db holds almost no neutral headlines and would have taught a trained scorer that "neutral" barely exists. scripts/backfill_news_archive.py seeds it from both DB shapes on disk, read-only and idempotent, recomputing provenance by comparing each stored score against the keyword scorer (neither source recorded it): 1,016 distinct headlines, 846 DeepSeek-labelled, growing ~1,400/week. Nothing in the signal core changed. Both news gating flags stay False, no gate reads any of this, and the backtester never builds a NewsEngine so replays never archive (test-locked). 468 -> 483 tests, including a new guard that AST-walks live code for sklearn/scipy/lightgbm imports — that check did not exist before. Do not train yet: ~800 labels gives a holdout too small to grade (SE +/-0.032). Target is ~4,000 deduplicated headlines over >=3 weeks. Claude-Session: https://claude.ai/code/session_01Bzv4ETVapFNauFDQht7QtA |
||
|
Harwinder Sekhon
|
c004e3f30b |
chore: keep only what the system needs — drop research tooling, study scripts, diagram, plan
Owner directive 2026-09-06: Dukascopy and similar research material is irrelevant to the running system; keep only code and files with a role in system functioning. Deleted (all in git history before this commit): - backtest/research/ tooling and data: book_report, book_diff, burst_map, burst_precursors, burst_indicator_map, open_trade_bursts, calendar_impact, watch_rule_eval, sizing_model, commit_ladder, llm_news_eval + llm_news_golden.csv, ff_calendar_scraper, duka_spread + duka_spread_by_hour.csv (the Dukascopy spread study), feed_divergence_oanda_vs_tiingo.csv, the panel dry-test JSONL, __init__.py - scripts/feed_divergence_study.py, scripts/signal_parity_study.py (completed studies; their reports keep the results) - docs/architecture.png + architecture_diagram.py (developer illustration) - docs/data_source_change_test_plan.md (Phase 4 shadow scan stays noted in CLAUDE.md) Kept: the campaign .md reports in backtest/research/ as the decision ledger CLAUDE.md cites (rule 1 evidence); scripts/shadow_scan.py (pending operational step); backtest/diagnose_filters.py (test-locked backtester diagnostic). Local, untracked material removed on this box: .superpowers/, walkforward/, baseline/, the commit-ladder and LLM-eval result folders, the Dukascopy boundary CSV; the sizing_2026-09 folder is pruned to the four 3-year baseline books CLAUDE.md names. .gitignore no longer lists the removed research outputs; the backtest instructions and CLAUDE.md no longer point at deleted tools. Suite: 461 passed, 0 failed (nothing imported the removed files). Claude-Session: https://claude.ai/code/session_0156YhogLytzj9P2iH1skGbL |
||
|
Harwinder Sekhon
|
c00ed4471b |
chore: repo hygiene — stale gitignore entries, finished plans, superseded research, doc drift
Removed (all recoverable from git history):
- llm_provider_switch_plan.txt: 2026-08-08 DeepSeek/Anthropic multi-provider plan that
was never implemented (llm_client.py is DeepSeek-only, no `anthropic` anywhere)
- docs/superpowers/specs/2026-07-21-centralized-config-design.md: spec of the
config.json migration that merged the same day; its plan sibling was already
deleted in
|
||
|
Harwinder Sekhon
|
985ec83ef3 |
chore: remove dead functions and write-only state (no behaviour change)
Zero callers anywhere (tests included), verified by grep/AST: - dashboard_publisher.DashboardPublisher.publish_all (run_cycle inlines both publishes) - signal_log.SignalLogger.advance_with_bars (the backfill script uses backfill_seeds / resolve_from_store / reseed_unresolvable) and the never-read SignalLogger.path property plus its write-only self._path - llm_client.reset_llm_client (tests reset the singleton by assignment) - market_outlook._last_run_ts, residue of the old "now - last < 3600" dedup; _last_run_hour (the UTC hour bucket) is the live key - news_engine FundamentalContext.event_impact_on_pairs: written once, read by nothing; its only feeder NewsEngine._get_affected_pairs and that helper's unit test go with it - position_sizing.LIVE_DEPTH_BARS (referenced only by its own docstring, which now states the depths inline), the unused Iterable import and the unused atr_ok parameter of _fill_geometry - analyze_signal_log: unused `field` import and the unused `rows` parameter of persona_scorecard / news_outcome_buckets (both read the DB directly) The two backfill/label scripts' --db help text now names the config.json setting it actually falls back to. Suite: 463 passed, 0 failed (one test removed with its subject). Claude-Session: https://claude.ai/code/session_0156YhogLytzj9P2iH1skGbL |
||
|
Harwinder Sekhon
|
c94f78e0b0 |
chore: drop unused imports and a dead constant (no behaviour change)
- backtest/diagnose_filters: unused BacktestConfig import - pair_signals/__init__: unused PlatformConfig import - scripts/shadow_scan: unused pandas import - research burst_precursors / calendar_impact / duka_spread: unused sys / COST / struct - test_all: MagicMock, StringIO, Colors and two function-local imports that nothing in the test bodies used - telegram_bot: ViolentWatch is only a type hint -> imported under TYPE_CHECKING compare_oanda_tiingo.py and oanda_td_divergence.py are left alone here because the hygiene commit deletes them. Suite: 464 passed, 0 failed (unchanged). Claude-Session: https://claude.ai/code/session_0156YhogLytzj9P2iH1skGbL |
||
|
Harwinder Sekhon
|
fc55da17db |
signal_log: weight-research data points, scan_outcomes forward labels, watch provenance
New additive columns on every logged scan: bar_ts (primary closed-bar label, indexed with timeframe), engine_fingerprint (hash of the scoring config incl. the regime weight table in force - SignalAggregator.engine_config_fingerprint, twin of the gate fingerprint), code_version (short git SHA), penalty_tags_json / penalty_total / momentum_fading / exhaustion_count, and post-hoc watch_level / watch_kinds / watch_line written by violent_watch.attach_watch via signal_log.update_watch. Each indicators_json entry now also carries eff_weight (base weight x regime multiplier, the weight the composite actually used) and wscore. engine: IndicatorSignal.effective_weight / weighted_score are stamped inside _compute_composite (record-keeping only); regime_weight_table(config) is the single source of truth for the multiplier table (override honoured). Composite output byte-identical (120-report hash proof). New table scan_outcomes: model-free forward moves per scan (fwd_1h/4h/24h in $ and ATRs, mfe/mae over 24h, burst_1h, bars_used, complete) measured on the stored candle stream with the advisory model's continuity rule; labelled opportunistically at the end of log() (time-boxed, throttled, idempotent) - no daemon; scripts/label_scan_outcomes.py drives the backlog (--dry-run is read-only). analyze_signal_log --scan-outcomes: forward move + hit rate by TF x signal/grade with a NEUTRAL baseline and burst rate by watch level. Readers dedup by the first row per (timeframe, bar_ts). Module 31 (10 tests). Suite: 463 passed. Backtester untouched (test-locked). Claude-Session: https://claude.ai/code/session_01RWV2yLt3YXHHGVYHSQU2EE |
||
|
Harwinder Sekhon
|
bed8b549ac |
audit: fix 9 defects from the 2026-09-04 hidden-bug sweep
None of these touch the signal path. The 12-month M15 book (2024-03-31..2025-03-30, $0.65, cached Oanda bars) is byte-identical before and after (md5 c59d3e7e685c3c880b614640adfea74b, 375 trades), and signal/indicator logging to logs/signals.db is unchanged — verified end to end: per-TF indicators for all 3 TFs, 14 candles each, MTF and reasons all still persisted. agent_panel: the >=2-take quorum guards the CHAIR's synthesizer spend, but ran BEFORE the macro branch — so the shipped default (verdict_source="macro", which never calls the synthesizer) silently dropped a good macro verdict whenever the other three personas were slow. Quorum moved below the macro path. analyze_signal_log: _forward_return clamped to the last available bar when the horizon had not elapsed, reporting a near-zero "forward move" for every row logged within `hours` of the end of the data — on every run, blended into the hit-rate/mean-move aggregate. Now returns None. analyze_signal_log: persona_scorecard iterated raw rows through an index that maps EVERY member row of an event, so a re-logged scan appended the same event R twice. One vote per event, matching news_outcome_buckets. Measured inflation on the live DB: 1.03x. signal_log: dedup required a non-negative logged_at gap, but logged_at is stamped before the row wins the write lock (busy_timeout 5 s), so two processes logging one setup at a boundary can invert timestamp order against id order and seed a second event. Now abs(gap); same in backfill_seeds. Zero occurrences in the current 366 setup rows. advisory_outcome: the catch-all turned real evaluator faults into the same UNRESOLVABLE used for legitimate weekend gaps, with nothing in the log. Still fail-safe, now logged. llm_client: only the "cap" warn-once flag was ever reset, so in a daemon the second and every later network/http/parse outage logged at DEBUG only. Cleared on UTC-day roll and re-armed by a recovered call. dashboard.js: renderCards() redraws the selector for every timeframe but was gated on the SELECTED one changing, leaving the other buttons stale (up to 4 h on H4) while renderHero's rail already showed the new value. news_summary.php: atomic_write treated a short write (disk full) as success, publishing a truncated body the cache-first branch then served forever; and the daily cap was charged before the no-key check and before the stampede lock re-checked the cache, so a click burst could exhaust the 150/day budget with zero LLM calls. migrate_signal_log_news: logged_at is ISO with 'T' while SQLite's datetime() uses a space, so string-comparing them put the whole cutoff calendar day on the wrong side and it never pruned. NOTE: this now deletes one day more than before — it only runs when invoked by hand. Tests 414 -> 422 (new Module 28, one per fix). The existing "<2 valid takes" test was retargeted at the chair — the case it actually protects — rather than deleted, with new coverage for macro-alone. Not fixed, recorded for the owner: - Gate 6c's boost condition is textually identical to the C-grade definition, so every C in a "transitioning" regime is promoted to B before gate 6e reads it, making sell_min_quality's exclusion of "C" a no-op on M15/H4 (H1 and M5 are immune by other means). The backtester runs the same code, so every pinned book already includes this — changing it is a rule-1 revalidation event. - Safe-haven/geopolitical headlines score exactly 0.00 in the keyword fallback and are now dropped entirely by the ±0.1 neutral filter, contradicting the platform's own gold doctrine. Bites only when DeepSeek is capped/disabled/failing. Claude-Session: https://claude.ai/code/session_01Ah5EB1goi5inQmSd37Ne6a |
||
|
Harwinder Sekhon
|
b1e880be4f |
signal_log: outcomes for every setup, event dedup, interned news; fix over-scan, dead outlook, panel OPPOSE bias
- seed an advisory outcome on every setup row; same-setup re-logs within 60 min are DUPLICATE rows (advisory_event_id); resolver rebuilds bar streams across gaps - headlines/events interned into news_snapshots (DB 667 MB -> 262 MB); readers use signals_with_news / read_news and mode=ro (immutable=1 missed WAL rows) - post-gate truth columns: gate_reason, actionable, cleaned_warnings, scan_kind, config_fingerprint; SELL-hour gates fail closed - telegram: breach rescan reuses this bar's report; grade dropped from alert fingerprint (upgrade-only re-alert) - market_outlook: hour-bucket dedup (old rule skipped alternate hours), skip heartbeats - agent_panel: macro-anchored OPPOSE guard (opposed trades were +30R over 3 weeks) - analyze_signal_log --stored-candles/--parity; backfill + news migration scripts - tests: Module 25 (analyze_signal_log) + 27 (observability); 429 total Claude-Session: https://claude.ai/code/session_01FC4s6VCURFte4dub4xKxYL |
||
|
Harwinder Sekhon
|
a979dcba4b |
signal_log: persist agent panel verdict + per-persona takes to SQLite
Add panel_verdict, panel_confidence, panel_rationale, panel_takes_json columns to the signals table (additive, idempotent ALTER TABLE migration for existing DBs). _report_row now reads the panel fields attach_panel writes onto the SignalReport, so LLM panel verdicts stop evaporating on process exit. Also documents the Oanda re-pin (round_trip_cost stays $0.30 for cross-book comparability and the live TP gate; $0.65 is the measured headline) and adds the feed-divergence / parity / shadow-scan study tooling referenced by CLAUDE.md. |