Harwinder Sekhon
1f44d6bad6
Removed (all recoverable from git history):
- llm_provider_switch_plan.txt: 2026-08-08 DeepSeek/Anthropic multi-provider plan that
was never implemented (llm_client.py is DeepSeek-only, no `anthropic` anywhere)
- docs/superpowers/specs/2026-07-21-centralized-config-design.md: spec of the
config.json migration that merged the same day; its plan sibling was already
deleted in b936d44. app_config's docstring pointer goes with it.
- scripts/compare_oanda_tiingo.py: Phase 0 recent-tail smoke test, superseded by the
Phase 1/2 study scripts (result recorded in the test plan)
- backtest/research/fetch_oanda_m15.py + oanda_td_divergence.py + its by-hour CSV:
the 2026-07-24 Oanda-vs-Twelve-Data study, superseded by the 2026-08-14 feed
divergence report (which keeps a one-line citation)
- backtest/research/burst_profile.py and hour_analysis.py: superseded by burst_map.py
and book_report.py
- backtest/research/panel_payload_drytest_2026-08-15.py: one-off live DeepSeek probe;
its .jsonl evidence stays
Added to git: the three 2026-08-14 research reports (feed divergence, re-pin
verification, signal parity), their CSV, and docs/data_source_change_test_plan.md -
CLAUDE.md cites them as standing evidence and their driver scripts were already
committed. The test plan no longer refers to auto-execution.
Hygiene:
- .gitignore: drop oanda_state.json (daemon removed 2026-09-03) and the vanished
backtest/trailgrid dirs; ignore .coverage
- docs/architecture_diagram.py only writes the PNG when run as a script (it used
to plot on import); test box refreshed to 461 tests / 31 modules; PNG regenerated
(the committed one still drew the removed daemon)
- backtest/instructions_for_backtest.txt: manual-journal head replaced by the
actual reporting checklist; the 2026-09-06 addendum is unchanged
- doc drift: README/publisher list all six published files; news_summary.php says
4 h (not 24 h) and drops the pre-id sha1 fallback whose window closed;
analyze_signal_log lists six lenses and --scan-outcomes; agent_panel cost model
says 4 requests by default; tiingo comment cites the NY-anchored weekend;
Module 21 title no longer says "execution lifecycle"
Suite: 461 passed, 0 failed.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_0156YhogLytzj9P2iH1skGbL
117 lines
6.3 KiB
Markdown
117 lines
6.3 KiB
Markdown
# AI signal enrichment — analysis + build record (2026-08-15)
|
||
|
||
Owner questions:
|
||
1. If the AI panel also received indicator values and (limited) candle data, would the system be more accurate?
|
||
2. If AI reviewed every candle alongside the local score system, would prediction accuracy justify the token burn — or is there a middle ground?
|
||
|
||
## What the panel already received (pre-change baseline)
|
||
|
||
The four personas (`agent_panel.py`) already got a *distilled* picture:
|
||
per-indicator signal+confidence, MTF signals/scores, key levels,
|
||
exhaustion count, full trade-setup math, warnings, headlines/events/
|
||
sentiment. Missing: raw indicator **values** and any **candle** data.
|
||
|
||
Cost guardrails already in place: 5 requests/panel, `max_panels_per_day`
|
||
20 (≤100 requests/day), 3600 s reuse-TTL, per-bar exact cache, shared
|
||
500 req/day DeepSeek cap, static prompts on the ~50× prefix cache.
|
||
|
||
## Evidence used
|
||
|
||
- **LLM is strong at text, weak at raw numerics** (measured in-repo,
|
||
`llm_news_eval_2026-08-06.md`): headline sentiment 100% (78/78) vs
|
||
keyword scorer 31%; but ~11% of event-direction calls misread close
|
||
numeric comparisons until code pre-computed the comparison
|
||
(`vs_forecast`) and the model only mapped it to doctrine.
|
||
- **The book any filter would act on** (M5 canonical,
|
||
`oanda_m5_12mo_065_nopb_20260814.csv`): 577 trades, +106.82R,
|
||
PF 1.51, win rate 52.9%; A-grade 59.5%, B-grade 51.4% (+67.8R of the
|
||
total), C-grade 43.8% (−0.23R, 16 trades). Average edge ≈ 0.185R/trade.
|
||
|
||
## Answer 1 — enrich the existing panel? YES (cheap, safe)
|
||
|
||
- A compact candle is ~15–25 tokens; 20 primary-TF + 10 confirm-TF bars
|
||
≈ 500–1,500 tokens per persona call → cents/day at the 20-panel cap.
|
||
**Tokens are not the constraint; the request cap is.**
|
||
- Measured after the build (dry-test wire log): enriched technical
|
||
payload with 20 candles ≈ **~380 tokens**; replies ≈ 25–35 tokens
|
||
under the new 15-word note cap.
|
||
- Caveats: (a) the panel is advisory-only — richer payloads improve the
|
||
human's context, not the book's P&L; (b) raw prices are the model's
|
||
weakest input, so candles are sent **ATR-normalized** (code does the
|
||
arithmetic, LLM reads the shape) — the same pattern that fixed the
|
||
event-direction misreads.
|
||
|
||
## Answer 2 — AI on every candle? NO
|
||
|
||
- **Feasibility**: M5+M15+H1+H4 ≈ 414 bars/day → one call/bar nearly
|
||
exhausts the shared 500/day cap (news needs it too); a panel per bar
|
||
≈ 2,070/day = 4× over. Most of that spend reviews HOLD bars.
|
||
- **Parity (rule 4)**: an AI gate cannot be replayed by the backtester
|
||
(news off; no recorded verdicts) — every pinned book becomes
|
||
non-comparable, and model-version drift silently mutates the strategy.
|
||
- **Edge**: the engine already compresses 380 bars into 9 tuned
|
||
indicators + regime + MTF validated over 5 years; an LLM's generic
|
||
prior on raw candles won't beat fitted thresholds, and the margin it
|
||
would filter is ~0.185R/trade. B-grade holds +67.8R at 51.4% win —
|
||
a blunt veto is net-negative.
|
||
|
||
## The middle ground (adopted)
|
||
|
||
1. **Enrich payloads** (built today, see below) — near-zero cost, no
|
||
rule-1 exposure.
|
||
2. **Log panel verdicts to signals.db** (built today) — creates the
|
||
"does OPPOSE predict losers?" dataset with zero behavior change.
|
||
3. **After 4–8 weeks**: join logged verdicts to outcomes; simulate an
|
||
OPPOSE-veto against the pinned book offline. Only then consider any
|
||
gating, as a full rule-1 revalidation.
|
||
|
||
## Build record (2026-08-15, owner-approved)
|
||
|
||
- `agent_panel.py`
|
||
- Technical persona payload now carries indicator **values** (units
|
||
legend added to the prompt) and **ATR-normalized candles** per TF:
|
||
`{"ref_close", "atr", "bars": [[o,h,l,c]…]}` oldest-first, each price
|
||
as `(price − ref_close) / atr`, 1 decimal. Window: 20 bars primary /
|
||
10 per confirmation TF.
|
||
- New config keys (validated in `app_config.py`, in
|
||
`config.json.example`): `candles_enabled` (default true),
|
||
`candle_bars_primary` (20), `candle_bars_confirm` (10).
|
||
- Panel-private provider singleton (`_dp`, patchable in tests);
|
||
TTL cache keeps fetches near-free; any failure → candles silently
|
||
omitted, panel unaffected.
|
||
- Prompt hardening: training data declared stale and OFF-LIMITS;
|
||
"if a fact is not in the JSON, treat it as unknown"; thin evidence
|
||
→ neutral < 30 (unchanged); note cap 25→**15 words**, rationale
|
||
40→**30 words** (output-token savings).
|
||
- `attach_panel` now calls `signal_log.log_panel_update(report)` after
|
||
every successful attach (fresh, exact-cache, and reuse-TTL paths).
|
||
- `signal_log.py`
|
||
- `log()` returns the rowid; `log_signal` stashes it on the report as
|
||
`_signal_log_rowid`; new `log_panel_update()` fills the 4 panel
|
||
columns on that row post-hoc. No rowid (backtest replay, logging
|
||
disabled, other process) → silent no-op. Fail-safe contract kept.
|
||
- Tests: 381 → **388 passed** (7 new: candle normalization math, payload
|
||
values/candles, context windows + ATR gating + disable flag, attach
|
||
injection (technical persona only), prompt-contract assertions,
|
||
panel row update, rowid hand-off + no-op paths). Existing attach tests
|
||
stub `ap._dp` to stay off the network.
|
||
- **Live dry test** (`panel_payload_drytest_2026-08-15.py` — script removed 2026-09-06, raw replies kept in the `.jsonl` — fabricated
|
||
numbers only, 13 real DeepSeek requests total): **7/7 probes pass** —
|
||
coherent-bullish → support@70; bearish-vs-BUY → oppose@85; genuinely
|
||
mixed → neutral@30; empty payload → neutral@20 (contract); no invented
|
||
numbers in any note; all notes ≤ 15 words; synthesizer AGREE with a
|
||
21-word rationale. Raw wire log:
|
||
`backtest/research/panel_payload_drytest_2026-08-15.jsonl`.
|
||
- Notable: an early P3 variant with H1 SELL against a BUY brief got
|
||
oppose@60 — *correct* per the prompt (higher-TF disagreement is a
|
||
listed conflict), so the probe was tightened to a genuinely mixed
|
||
scenario. Documented so nobody "fixes" the prompt for it later.
|
||
|
||
## What this deliberately did NOT change
|
||
|
||
- No gate reads panel fields; the panel remains advisory-only.
|
||
- No backtest-path change (`analyze_from_dataframes` untouched; parity
|
||
test still locks panel fields empty in replay).
|
||
- No verdict-based filtering — that decision waits for the signals.db
|
||
dataset (4–8 weeks) and an offline veto replay against the pinned book.
|