Initial import: XAUUSD news sentiment models, evals, and data tooling
TF-IDF baseline (news_sentiment.py, models/xauusd_sentiment.joblib) and the two-head DeBERTa-v3-small transformer (transformer/, news_transformer.py) with its Google News crawler and DeepSeek labeller (data_collect/), the golden-set and archive evals (eval/), and the 2026-09-22 and 2026-09-27 results and progress notes. Transformer weights (model.pt, model.onnx) and the crawl database are not tracked; rebuild with transformer/train.py and transformer/export_onnx.py. The DeepSeek label cache in data/ is tracked.
This commit is contained in:
Harwinder Sekhon
@@ -0,0 +1,206 @@
|
||||
# Session progress: 2026-09-22 → 09-23 (UTC)
|
||||
|
||||
This session had two parts:
|
||||
1. Retrain the TF-IDF XAUUSD model on the new `newsdata.db`. **Done and promoted.**
|
||||
2. Build a two-stage transformer model: is the headline gold-related, and if so, bullish or bearish? **The
|
||||
tooling is built and tested. Historical data collection is partly done. The model is not trained on
|
||||
history yet.**
|
||||
|
||||
DeepSeek calls made this session: **40**, all in the history labelling pilot. The user asked for no more
|
||||
DeepSeek calls on 2026-09-23. All background jobs are stopped.
|
||||
|
||||
---
|
||||
|
||||
## Important facts to carry forward
|
||||
|
||||
- **The labels come from DeepSeek, not from traders.** The `newsdata.db` handed over as "results from
|
||||
professional traders" has 2,661 rows scored by `deepseek-v4-flash` (prompt sha1 `d32e04c6`) and 4 rows
|
||||
scored by keywords. The user confirmed we train on these. Every model here learns to copy DeepSeek; it does
|
||||
not learn price moves.
|
||||
- **The golden set is the only independent truth:** `eval/data/llm_news_golden.csv`, 78 reviewer-labelled
|
||||
headlines. DeepSeek scores 78/78. It is excluded from every training set.
|
||||
- **The DeepSeek ceiling:** on re-scoring, DeepSeek agrees with its own earlier answer about 90% of the
|
||||
time. No student model can beat that.
|
||||
- **The deployment rule from 2026-09-08 still applies:** about 4,000 labels over 3+ weeks before any live
|
||||
use.
|
||||
- **Nothing in ForexBot2 changed, and ForexBot2 doesn't import any ML-NEWS model.**
|
||||
|
||||
---
|
||||
|
||||
## Part 1: TF-IDF retrain (done)
|
||||
|
||||
Full write-up: `eval/RESULTS_2026-09-22.md`.
|
||||
|
||||
- `newsdata.db` was copied into `ML-NEWS/newsdata.db`. The original is still untracked in
|
||||
`~/Development/SSH-Client/`; delete it there when convenient.
|
||||
- `news_sentiment.py` now takes each label from the first source that has one:
|
||||
1. the `xau_score` column,
|
||||
2. the JSON `XAU` key,
|
||||
3. the backfill file given with `--extra-labels`.
|
||||
|
||||
It also has a new `--scope all|gold` option and drops rows from test feeds.
|
||||
- Eval scripts gained `--model`, `--extra-labels` and `--train-scope`. They also get label source and gold
|
||||
flags from `load_archive_rows`.
|
||||
- **Promoted:** `--scope all` with the backfill labels, 2,620 headlines from Sep 8–22.
|
||||
- The old bundle is in `models/archive/xauusd_sentiment_20260917.*`.
|
||||
- The candidates are in `models/candidates/`.
|
||||
|
||||
Results on the Sep 18–22 days, which none of these models trained on:
|
||||
|
||||
| Model | All | Gold headlines | Macro headlines | Golden set |
|
||||
|---|---|---|---|---|
|
||||
| Old 09-17 model | 45.9% | 68.2% | 36.2% | 39/78 |
|
||||
| Gold-only retrain | 48.3% | 73.8% | 37.1% | 40/78 |
|
||||
| **All-news retrain (promoted)** | **62.8%** | 71.9% | **58.8%** | **47/78** |
|
||||
|
||||
- **The promotion rule, taken literally, failed.** It required a chronological-holdout macro-F1 of at least
|
||||
0.671, and the new model scored 0.659. The two holdouts contain different kinds of headline, so the
|
||||
numbers don't compare. On identical unseen rows the new model wins clearly. The user was told this.
|
||||
- **Known weak spots:** dovish news and falling yields read as bearish. September was a hiking regime, so
|
||||
there are almost no easing examples. Risk-on scores 0/4 and negation 1/8.
|
||||
- **Version mismatch:** ForexBot2's venv has sklearn 1.9.0 but the bundle was pickled with 1.9.1, which
|
||||
raises a warning. The old bundle had the same problem.
|
||||
|
||||
---
|
||||
|
||||
## Part 2: two-stage transformer (in progress)
|
||||
|
||||
The plan is in `~/.claude/plans/you-are-machine-learning-cozy-teapot.md`. The user chose:
|
||||
- Training on the GPU, but the trained model must run on CPU.
|
||||
- More data from DeepSeek labelling historical headlines.
|
||||
- "Gold-related" means the headline moves the gold price, not that it mentions gold.
|
||||
|
||||
### Design
|
||||
|
||||
- **Model:** `microsoft/deberta-v3-small` encoder with mean pooling and two single-logit heads.
|
||||
- **Relevance head:** target is 1 if |xau| ≥ 0.25 and 0 if |xau| ≤ 0.10. Rows in between carry no loss.
|
||||
- **Direction head:** target is the sign of xau, on relevant rows only.
|
||||
- **Input:** the headline only, at most 64 tokens.
|
||||
- **Output:** `p_relevant` and `p_bullish`, combined as `score = p_relevant × (2·p_bullish − 1)`. The label
|
||||
is `neutral` when `p_relevant` is below the tuned threshold.
|
||||
|
||||
### Files added or changed
|
||||
|
||||
| File | What it does |
|
||||
|---|---|
|
||||
| `data_collect/crawl_google_news.py` | Google News RSS crawler with date windows. It can resume, handles 429/503 with backoff, and shuffles the windows so a partial crawl covers every year. It has two query sets, `--query-set broad` and `--query-set precise` (see below). |
|
||||
| `data_collect/label_history.py` | DeepSeek labeller with a hard `--max-calls` stop and its own per-process daily cap. It samples round-robin over (quarter × query) strata from two pools, with `--broad-share 0.3` by default. It excludes the golden set and `newsdata.db` overlaps, and prints a label report per year. |
|
||||
| `transformer/train.py` | Two-head fine-tuning. Val is newsdata from 2026-09-18 on and is never used to choose anything. Dev is a random 5%, used to pick the epoch and the relevance threshold. `--final` retrains on everything into `torch_final/`. |
|
||||
| `transformer/export_onnx.py` | ONNX export plus int8 on the embedding table only. Checks tokenizer and probability parity against PyTorch and measures CPU latency. |
|
||||
| `news_transformer.py` | CPU runtime needing only onnxruntime, tokenizers and numpy. Provides `load_model`, `score_batch`, `score_news` and a CLI `score --headline/--input`. |
|
||||
| `eval/eval_common.py` | Adds `TransformerScorer` and `make_scorer`, an `oos_from` field on scorers, and `daily_cap` for `install_research_client` and `DeepSeekScorer`. |
|
||||
| `eval/eval_golden.py`, `eval/eval_archive.py` | New `--scorer tfidf|transformer`. The archive eval reports relevance precision, recall and F1 plus direction accuracy on out-of-sample rows. |
|
||||
| `requirements.txt` | Adds onnxruntime and tokenizers for the CPU runtime; installed into `.venv`. |
|
||||
| `requirements-train.txt` | New GPU environment `.venv-train` (6 GB): torch 2.14 with CUDA, transformers 5.17, onnx, onnxruntime and pandas. |
|
||||
|
||||
Backups of every file edited in Part 1 (from before the edits) were saved to the session scratchpad at
|
||||
`/tmp/claude-1000/.../scratchpad/ml-news-backup/`. That directory may not survive a reboot.
|
||||
|
||||
### Technical findings
|
||||
|
||||
1. **transformers 5 loads DeBERTa-v3-small in fp16 by default,** because that is the dtype stored in the
|
||||
checkpoint. Training fp16 weights with AdamW gives a NaN loss.
|
||||
- Fix: `AutoModel.from_pretrained(base, dtype=torch.float32)`, with bf16 autocast for compute. This is in
|
||||
`train.py`.
|
||||
2. **The dynamo ONNX exporter output breaks onnxruntime quantisation** with a shape-inference error (768
|
||||
vs 1). Using the TorchScript exporter (`dynamo=False`, opset 17) fixes it.
|
||||
3. **Dynamic int8 quantisation of DeBERTa's MatMuls ruins accuracy.**
|
||||
|
||||
| Variant | Max prob diff | Flips (of 600 decisions) | Size | Latency, 1 thread |
|
||||
|---|---|---|---|---|
|
||||
| fp32 ONNX | 0.000 | 0 | 566 MB | 18.2 ms |
|
||||
| int8 all / per-channel / MatMul-only / reduce-range / FFN-only | 0.55–0.72 | 81–96 | 172–467 MB | 7–11 ms |
|
||||
| **int8 embedding table only (chosen)** | 0.04–0.05 | about 1% | 271 MB | 18–20 ms |
|
||||
|
||||
Because of this, the export tolerance is 0.05 max difference and at most 2% of decisions flipping. The
|
||||
plan had said 0.03.
|
||||
4. **Latency on this machine:** about 18–20 ms per headline on 1 thread and about 9 ms on 4 threads. 16
|
||||
threads is slower (about 13 ms) from oversubscription. When wiring it in, pass `threads=4`.
|
||||
5. **Smoke test:** trained on only 1,402 live headlines from before Sep 18, 3 epochs, no history data.
|
||||
- Val (Sep 18–22), three-class agreement: 57.1%. The TF-IDF model trained on the same period gets 45.9%
|
||||
on those days.
|
||||
- Relevance F1: 0.73.
|
||||
- Direction accuracy on relevant rows: 80.7%, and 86% on gold headlines.
|
||||
- Golden set: 37/78.
|
||||
- It still calls Bitcoin headlines relevant. The historical data is meant to fix this.
|
||||
|
||||
The smoke artefacts were in the scratchpad only. **No transformer model is in `models/` yet.**
|
||||
|
||||
### Historical data status (`data/history.db`, 24 MB)
|
||||
|
||||
- **Broad crawl** (keyword plus `site:reuters.com / cnbc.com / marketwatch.com`, weekly windows,
|
||||
2019-01 → 2026-08): stopped at 568 of 6,000 requests, with about 21k unique headlines across all years.
|
||||
- **The broad crawl has low precision.** Google matches the query word anywhere in the article, not only in
|
||||
the headline.
|
||||
- The 40-call pilot labelled 1,000 broad headlines. Only 7% were gold-relevant (|xau| ≥ 0.25), against 43%
|
||||
in the live feed.
|
||||
- DeepSeek's labels themselves looked correct. For example, "Red Lobster endless shrimp losses" scored 0.
|
||||
- A keyword filter on the headline text only lifts the relevant share to 16%. Using the smoke model's
|
||||
relevance head as a filter gets 15%.
|
||||
- The broad pool is still useful as the not-relevant side.
|
||||
- **Precise crawl** (`intitle:<term> site:<publisher>`, 21 queries, 2-week windows): stopped at 161 of
|
||||
4,200 requests. That gives about 1,100 precise headlines so far, and they look strongly on-topic in spot
|
||||
checks: gold, Fed, rates, FX.
|
||||
- **Total:** 23,103 unique headlines, of which 1,000 are labelled (`labels` table). The labels come from the
|
||||
pilot, so almost all are from the broad pool.
|
||||
- DeepSeek answers are also cached in `data/history_deepseek_cache.json`, so no headline is paid for twice.
|
||||
- **Stray years:** Google misdates some items to 2009–2018. That is harmless; train/val is split by source
|
||||
and date.
|
||||
|
||||
---
|
||||
|
||||
## Next steps (in order)
|
||||
|
||||
Run all commands from `~/Development/ML-NEWS`.
|
||||
|
||||
1. **Finish the precise crawl.** It's free and needs no DeepSeek. It takes about 2.7 hours and resumes where
|
||||
it stopped.
|
||||
```
|
||||
python3 data_collect/crawl_google_news.py --query-set precise > logs/crawl_precise.log 2>&1
|
||||
```
|
||||
Optional: add more broad negatives with `--query-set broad`, which is 5,400 requests left and resumable.
|
||||
|
||||
2. **Second pilot, 20 calls (needs the user's OK for DeepSeek spend).** Check that the precise pool gives a
|
||||
high relevant share.
|
||||
```
|
||||
~/Development/ForexBot2/venv/bin/python data_collect/label_history.py --max-calls 20 --broad-share 0
|
||||
```
|
||||
Then look at the per-year report. Also check for hindsight bias: does the mean xau per year track gold's
|
||||
actual yearly moves too closely?
|
||||
|
||||
3. **Full labelling (needs the user's OK).** About 800–1,200 calls, a few US dollars.
|
||||
```
|
||||
~/Development/ForexBot2/venv/bin/python data_collect/label_history.py --max-calls 1000
|
||||
```
|
||||
Keep `--broad-share 0.3`. The run can be split across days; progress is saved per chunk.
|
||||
|
||||
4. **Train, export, evaluate:**
|
||||
```
|
||||
.venv-train/bin/python transformer/train.py # -> models/xau_transformer/torch/
|
||||
.venv-train/bin/python transformer/export_onnx.py --src models/xau_transformer/torch
|
||||
cd eval
|
||||
../.venv-train/bin/python eval_golden.py --no-network --scorer transformer
|
||||
../.venv-train/bin/python eval_archive.py --no-network --scorer transformer \
|
||||
--db ../newsdata.db --extra-labels results/backfill_labels.jsonl
|
||||
```
|
||||
Run the evals with `.venv-train`, which has pandas, onnxruntime and the ForexBot2 imports, so that
|
||||
ForexBot2's venv stays untouched.
|
||||
|
||||
5. **Promotion criteria, fixed in advance:**
|
||||
- Golden set ≥ 55/78 (TF-IDF has 47).
|
||||
- Distractors ≥ 6/8.
|
||||
- Sep 18–22 three-class agreement ≥ 62.8%.
|
||||
- CPU latency ≤ 20 ms.
|
||||
|
||||
If all pass: run `train.py --final`, export from `torch_final`, and write `eval/RESULTS_<date>.md`.
|
||||
|
||||
6. **Afterwards:**
|
||||
- Update the README and WORKFLOW doc for the transformer.
|
||||
- Decide separately whether and how ForexBot2 should use it, in shadow mode first.
|
||||
- Align ForexBot2's sklearn version with the training environment.
|
||||
|
||||
## Housekeeping
|
||||
|
||||
- ML-NEWS is not a git repo, so nothing was committed.
|
||||
- `~/Development/SSH-Client/newsdata.db` is an untracked copy; delete it there.
|
||||
- `.venv-train` takes 6 GB and is only needed for training and export.
|
||||
Reference in New Issue
Block a user