Initial import: XAUUSD news sentiment models, evals, and data tooling
TF-IDF baseline (news_sentiment.py, models/xauusd_sentiment.joblib) and the two-head DeBERTa-v3-small transformer (transformer/, news_transformer.py) with its Google News crawler and DeepSeek labeller (data_collect/), the golden-set and archive evals (eval/), and the 2026-09-22 and 2026-09-27 results and progress notes. Transformer weights (model.pt, model.onnx) and the crawl database are not tracked; rebuild with transformer/train.py and transformer/export_onnx.py. The DeepSeek label cache in data/ is tracked. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
This commit is contained in:
Harwinder Sekhon
@@ -0,0 +1,70 @@
|
||||
# Session progress: 2026-09-27
|
||||
|
||||
Continues `PROGRESS_2026-09-22.md`. Full numbers in `eval/RESULTS_2026-09-27.md`.
|
||||
|
||||
DeepSeek calls this session: **560** (20 pilot + 200 full run + 140 precise top-up + 200 growth set; 15
|
||||
failed), approved by the user in-session. 13,625 crawled headlines labelled; history labels now total
|
||||
14,625.
|
||||
|
||||
## Done, in order
|
||||
|
||||
1. **Precise crawl resumed** (`crawl_google_news.py --query-set precise`). It ran in the background the
|
||||
whole session and was still running at the end, at about 860 windows done today and 3,300 to go. Its
|
||||
log (`logs/crawl_precise.log`) is block-buffered and lags; use `pgrep -f crawl_google_news` and the
|
||||
`crawl_log` table.
|
||||
2. **Trained the transformer on the 3,620 labelled rows available before any DeepSeek call**, swept 8
|
||||
configs, exported, evaluated: golden 49/78, unseen days 65.7%. Failed the golden bar of 55. Written up
|
||||
as the "3,620 rows" column in the results file.
|
||||
3. **Pilot, 20 calls, precise pool only:** 52% gold-relevant vs 7% for the broad pool. No hindsight bias
|
||||
worth worrying about. Log: `logs/label_pilot2.log`.
|
||||
4. **Full run, 200 calls, 30% broad share:** 4,750 headlines. Log: `logs/label_full_20260927.log`.
|
||||
5. **Retrained on 8,870 rows, swept 5 configs** (all within 1.2 val points of each other). Chosen on dev:
|
||||
lr 5e-5, 4 epochs, batch 64, best epoch 3.
|
||||
- Golden 62/78, distractors 8/8, Sep 18–22 agreement 71.9%, latency 19.0 ms. **All four promotion
|
||||
criteria pass.**
|
||||
6. **`train.py --final`** on all 8,870 rows for 3 epochs, exported from `torch_final`. Shipped bundle
|
||||
`models/xau_transformer/{model.onnx,tokenizer.json,meta.json}`: golden 65/78, 0 decision flips, 18.9 ms.
|
||||
7. Wrote `eval/RESULTS_2026-09-27.md`.
|
||||
8. **Round 2, at the user's request:** 140 more calls labelled the remaining 3,400 precise headlines.
|
||||
Retrained on 12,270 rows (default config, no sweep). Non-final: golden 64/78, Sep 18–22 71.8%. Final on
|
||||
all rows, 4 epochs: **golden 67/78**, 18.9 ms. Re-exported; this is the shipped bundle now. The live-feed
|
||||
metric did not move (72.0 → 71.7%), so it has saturated on history labels.
|
||||
9. **Round 3, "teach it the two weak spots":** added a `growth` query set to the crawler (20 `intitle:`
|
||||
queries: GDP, retail sales, ISM, PMI, jobless claims, consumer confidence/spending, manufacturing,
|
||||
"record high", "risk appetite", "Wall Street", "S&P 500", ceasefire, "trade deal", equities). Paused
|
||||
the precise crawl, crawled 1,880 windows (6,183 headlines), resumed the precise crawl. Added
|
||||
`--query-set` and a non-Latin filter to `label_history.py`. 200 calls labelled 4,975 growth headlines.
|
||||
Retrained on 17,245 rows: non-final golden 71/78, Sep 18–22 71.5%; **final golden 72/78**, strong_usd
|
||||
7/7, risk_on 4/4, 18.8 ms. Re-exported; this is the shipped bundle now.
|
||||
|
||||
## Where things are
|
||||
|
||||
| Path | What |
|
||||
|---|---|
|
||||
| `models/xau_transformer/` | shipped ONNX bundle from round 3 `torch_final` (17,245 rows, golden 72/78) |
|
||||
| `models/xau_transformer/torch/` | round 3 non-final weights (val numbers belong to these) |
|
||||
| `models/xau_transformer/torch_final/` | round 3 final weights |
|
||||
| session scratchpad `sweep2/*_8870rows`, `sweep2/*_12270rows` | round 1 and round 2 weights |
|
||||
| `models/xauusd_sentiment.joblib` | TF-IDF, unchanged, still what ForexBot2 would load if it loaded anything |
|
||||
| session scratchpad `sweep/`, `sweep2/` | all other sweep runs; may not survive a reboot |
|
||||
| `eval/results/golden_20260927_2315_transformer.*` | round 3 final model golden eval (72/78) |
|
||||
| `eval/results/archive_20260927_2313_transformer.json` | round 3 non-final archive eval |
|
||||
| `eval/results/golden_20260927_1949_*`, `archive_20260927_1947_*` | round 2 equivalents |
|
||||
| `eval/results/golden_20260927_1931_transformer.*`, `archive_20260927_1927_*` | round 1 equivalents |
|
||||
|
||||
## Known issues
|
||||
|
||||
- `export_onnx.py` prints `PARITY FAILED` on max |diff| (0.06–0.08 vs tolerance 0.05) even with 0 flips.
|
||||
Loosen the tolerance or check flips only.
|
||||
- 14 of 340 label batches failed (DeepSeek timeouts / one 502); those headlines are simply unlabelled and
|
||||
will be picked up by the next run. About 570 precise headlines were unlabelled at the end, plus whatever
|
||||
the still-running crawl adds.
|
||||
- 2025 labels have a mean xau of +0.12, higher than other years. Possible hindsight in DeepSeek's labels
|
||||
for the rally year.
|
||||
|
||||
## Next
|
||||
|
||||
See "Next, revised" in the results file. The weak spots are fixed on the golden set. The step that now
|
||||
produces new information is shadow mode in ForexBot2; more history labels will not. Watch the gold-key
|
||||
live-feed number (83 → 79%) as history labels come to dominate the training set. The precise crawl is
|
||||
still running in the background and is free to leave alone.
|
||||
Reference in New Issue
Block a user