Harwinder Sekhon
86 lines
5.5 KiB
Markdown
86 lines
5.5 KiB
Markdown
# Session progress: 2026-09-27
|
||
|
||
Continues `PROGRESS_2026-09-22.md`. Full numbers in `eval/RESULTS_2026-09-27.md`.
|
||
|
||
DeepSeek calls this session: **560** (20 pilot + 200 full run + 140 precise top-up + 200 growth set; 15
|
||
failed), approved by the user in-session. 13,625 crawled headlines labelled; history labels now total
|
||
14,625.
|
||
|
||
## Done, in order
|
||
|
||
1. **Precise crawl resumed** (`crawl_google_news.py --query-set precise`). It ran in the background the
|
||
whole session and was still running at the end, at about 860 windows done today and 3,300 to go. Its
|
||
log (`logs/crawl_precise.log`) is block-buffered and lags; use `pgrep -f crawl_google_news` and the
|
||
`crawl_log` table.
|
||
2. **Trained the transformer on the 3,620 labelled rows available before any DeepSeek call**, swept 8
|
||
configs, exported, evaluated: golden 49/78, unseen days 65.7%. Failed the golden bar of 55. Written up
|
||
as the "3,620 rows" column in the results file.
|
||
3. **Pilot, 20 calls, precise pool only:** 52% gold-relevant vs 7% for the broad pool. No hindsight bias
|
||
worth worrying about. Log: `logs/label_pilot2.log`.
|
||
4. **Full run, 200 calls, 30% broad share:** 4,750 headlines. Log: `logs/label_full_20260927.log`.
|
||
5. **Retrained on 8,870 rows, swept 5 configs** (all within 1.2 val points of each other). Chosen on dev:
|
||
lr 5e-5, 4 epochs, batch 64, best epoch 3.
|
||
- Golden 62/78, distractors 8/8, Sep 18–22 agreement 71.9%, latency 19.0 ms. **All four promotion
|
||
criteria pass.**
|
||
6. **`train.py --final`** on all 8,870 rows for 3 epochs, exported from `torch_final`. Shipped bundle
|
||
`models/xau_transformer/{model.onnx,tokenizer.json,meta.json}`: golden 65/78, 0 decision flips, 18.9 ms.
|
||
7. Wrote `eval/RESULTS_2026-09-27.md`.
|
||
8. **Round 2, at the user's request:** 140 more calls labelled the remaining 3,400 precise headlines.
|
||
Retrained on 12,270 rows (default config, no sweep). Non-final: golden 64/78, Sep 18–22 71.8%. Final on
|
||
all rows, 4 epochs: **golden 67/78**, 18.9 ms. Re-exported; this is the shipped bundle now. The live-feed
|
||
metric did not move (72.0 → 71.7%), so it has saturated on history labels.
|
||
9. **Round 3, "teach it the two weak spots":** added a `growth` query set to the crawler (20 `intitle:`
|
||
queries: GDP, retail sales, ISM, PMI, jobless claims, consumer confidence/spending, manufacturing,
|
||
"record high", "risk appetite", "Wall Street", "S&P 500", ceasefire, "trade deal", equities). Paused
|
||
the precise crawl, crawled 1,880 windows (6,183 headlines), resumed the precise crawl. Added
|
||
`--query-set` and a non-Latin filter to `label_history.py`. 200 calls labelled 4,975 growth headlines.
|
||
Retrained on 17,245 rows: non-final golden 71/78, Sep 18–22 71.5%; **final golden 72/78**, strong_usd
|
||
7/7, risk_on 4/4, 18.8 ms. Re-exported; this is the shipped bundle now.
|
||
|
||
## Where things are
|
||
|
||
| Path | What |
|
||
|---|---|
|
||
| `models/xau_transformer/` | shipped ONNX bundle from round 3 `torch_final` (17,245 rows, golden 72/78) |
|
||
| `models/xau_transformer/torch/` | round 3 non-final weights (val numbers belong to these) |
|
||
| `models/xau_transformer/torch_final/` | round 3 final weights |
|
||
| session scratchpad `sweep2/*_8870rows`, `sweep2/*_12270rows` | round 1 and round 2 weights |
|
||
| `models/xauusd_sentiment.joblib` | TF-IDF, unchanged, still what ForexBot2 would load if it loaded anything |
|
||
| session scratchpad `sweep/`, `sweep2/` | all other sweep runs; may not survive a reboot |
|
||
| `eval/results/golden_20260927_2315_transformer.*` | round 3 final model golden eval (72/78) |
|
||
| `eval/results/archive_20260927_2313_transformer.json` | round 3 non-final archive eval |
|
||
| `eval/results/golden_20260927_1949_*`, `archive_20260927_1947_*` | round 2 equivalents |
|
||
| `eval/results/golden_20260927_1931_transformer.*`, `archive_20260927_1927_*` | round 1 equivalents |
|
||
|
||
## Known issues
|
||
|
||
- `export_onnx.py` prints `PARITY FAILED` on max |diff| (0.06–0.08 vs tolerance 0.05) even with 0 flips.
|
||
Loosen the tolerance or check flips only.
|
||
- 14 of 340 label batches failed (DeepSeek timeouts / one 502); those headlines are simply unlabelled and
|
||
will be picked up by the next run. About 570 precise headlines were unlabelled at the end, plus whatever
|
||
the still-running crawl adds.
|
||
- 2025 labels have a mean xau of +0.12, higher than other years. Possible hindsight in DeepSeek's labels
|
||
for the rally year.
|
||
|
||
## Next
|
||
|
||
See "Next, revised" in the results file. The weak spots are fixed on the golden set. The step that now
|
||
produces new information is shadow mode in ForexBot2; more history labels will not. Watch the gold-key
|
||
live-feed number (83 → 79%) as history labels come to dominate the training set. The precise crawl is
|
||
still running in the background and is free to leave alone.
|
||
|
||
## Git
|
||
|
||
ML-NEWS is tracked in git since this session: `ssh://git@192.168.0.11:222/harry/News-ML-Decoder.git`,
|
||
branch `main`. Ignored: both venvs, `data/history.db`, and the transformer weights (`model.pt`, `*.onnx`).
|
||
The DeepSeek label cache and `newsdata.db` are tracked. Rebuild the weights with `transformer/train.py`
|
||
then `transformer/export_onnx.py`.
|
||
|
||
## ForexBot2 integration: specified, not built
|
||
|
||
`docs/FOREXBOT2_INTEGRATION_SPEC.md` is a full implementation spec for running the ONNX bundle inside
|
||
ForexBot2 (vendored runtime module, `news_model` config section, `news_engine` call site and the
|
||
gold-only scoring rules, archive provenance, tests, ops checklist for the bot and the dashboard daemon).
|
||
Owner decisions on 2026-09-27: mode `primary` with no DeepSeek for headline scoring, model files outside
|
||
the ForexBot2 tree. No ForexBot2 code was changed.
|