News-ML-Decoder/PROGRESS_2026-09-22.md

# Session progress: 2026-09-22 → 09-23 (UTC)

This session had two parts:
1. Retrain the TF-IDF XAUUSD model on the new `newsdata.db`. **Done and promoted.**
2. Build a two-stage transformer model: is the headline gold-related, and if so, bullish or bearish? **The
   tooling is built and tested. Historical data collection is partly done. The model is not trained on
   history yet.**

DeepSeek calls made this session: **40**, all in the history labelling pilot. The user asked for no more
DeepSeek calls on 2026-09-23. All background jobs are stopped.

---

## Important facts to carry forward

- **The labels come from DeepSeek, not from traders.** The `newsdata.db` handed over as "results from
  professional traders" has 2,661 rows scored by `deepseek-v4-flash` (prompt sha1 `d32e04c6`) and 4 rows
  scored by keywords. The user confirmed we train on these. Every model here learns to copy DeepSeek; it does
  not learn price moves.
- **The golden set is the only independent truth:** `eval/data/llm_news_golden.csv`, 78 reviewer-labelled
  headlines. DeepSeek scores 78/78. It is excluded from every training set.
- **The DeepSeek ceiling:** on re-scoring, DeepSeek agrees with its own earlier answer about 90% of the
  time. No student model can beat that.
- **The deployment rule from 2026-09-08 still applies:** about 4,000 labels over 3+ weeks before any live
  use.
- **Nothing in ForexBot2 changed, and ForexBot2 doesn't import any ML-NEWS model.**

---

## Part 1: TF-IDF retrain (done)

Full write-up: `eval/RESULTS_2026-09-22.md`.

- `newsdata.db` was copied into `ML-NEWS/newsdata.db`. The original is still untracked in
  `~/Development/SSH-Client/`; delete it there when convenient.
- `news_sentiment.py` now takes each label from the first source that has one:
  1. the `xau_score` column,
  2. the JSON `XAU` key,
  3. the backfill file given with `--extra-labels`.

  It also has a new `--scope all|gold` option and drops rows from test feeds.
- Eval scripts gained `--model`, `--extra-labels` and `--train-scope`. They also get label source and gold
  flags from `load_archive_rows`.
- **Promoted:** `--scope all` with the backfill labels, 2,620 headlines from Sep 8–22.
  - The old bundle is in `models/archive/xauusd_sentiment_20260917.*`.
  - The candidates are in `models/candidates/`.

Results on the Sep 18–22 days, which none of these models trained on:

| Model | All | Gold headlines | Macro headlines | Golden set |
|---|---|---|---|---|
| Old 09-17 model | 45.9% | 68.2% | 36.2% | 39/78 |
| Gold-only retrain | 48.3% | 73.8% | 37.1% | 40/78 |
| **All-news retrain (promoted)** | **62.8%** | 71.9% | **58.8%** | **47/78** |

- **The promotion rule, taken literally, failed.** It required a chronological-holdout macro-F1 of at least
  0.671, and the new model scored 0.659. The two holdouts contain different kinds of headline, so the
  numbers don't compare. On identical unseen rows the new model wins clearly. The user was told this.
- **Known weak spots:** dovish news and falling yields read as bearish. September was a hiking regime, so
  there are almost no easing examples. Risk-on scores 0/4 and negation 1/8.
- **Version mismatch:** ForexBot2's venv has sklearn 1.9.0 but the bundle was pickled with 1.9.1, which
  raises a warning. The old bundle had the same problem.

---

## Part 2: two-stage transformer (in progress)

The plan is in `~/.claude/plans/you-are-machine-learning-cozy-teapot.md`. The user chose:
- Training on the GPU, but the trained model must run on CPU.
- More data from DeepSeek labelling historical headlines.
- "Gold-related" means the headline moves the gold price, not that it mentions gold.

### Design

- **Model:** `microsoft/deberta-v3-small` encoder with mean pooling and two single-logit heads.
  - **Relevance head:** target is 1 if |xau| ≥ 0.25 and 0 if |xau| ≤ 0.10. Rows in between carry no loss.
  - **Direction head:** target is the sign of xau, on relevant rows only.
- **Input:** the headline only, at most 64 tokens.
- **Output:** `p_relevant` and `p_bullish`, combined as `score = p_relevant × (2·p_bullish − 1)`. The label
  is `neutral` when `p_relevant` is below the tuned threshold.

### Files added or changed

| File | What it does |
|---|---|
| `data_collect/crawl_google_news.py` | Google News RSS crawler with date windows. It can resume, handles 429/503 with backoff, and shuffles the windows so a partial crawl covers every year. It has two query sets, `--query-set broad` and `--query-set precise` (see below). |
| `data_collect/label_history.py` | DeepSeek labeller with a hard `--max-calls` stop and its own per-process daily cap. It samples round-robin over (quarter × query) strata from two pools, with `--broad-share 0.3` by default. It excludes the golden set and `newsdata.db` overlaps, and prints a label report per year. |
| `transformer/train.py` | Two-head fine-tuning. Val is newsdata from 2026-09-18 on and is never used to choose anything. Dev is a random 5%, used to pick the epoch and the relevance threshold. `--final` retrains on everything into `torch_final/`. |
| `transformer/export_onnx.py` | ONNX export plus int8 on the embedding table only. Checks tokenizer and probability parity against PyTorch and measures CPU latency. |
| `news_transformer.py` | CPU runtime needing only onnxruntime, tokenizers and numpy. Provides `load_model`, `score_batch`, `score_news` and a CLI `score --headline/--input`. |
| `eval/eval_common.py` | Adds `TransformerScorer` and `make_scorer`, an `oos_from` field on scorers, and `daily_cap` for `install_research_client` and `DeepSeekScorer`. |
| `eval/eval_golden.py`, `eval/eval_archive.py` | New `--scorer tfidf|transformer`. The archive eval reports relevance precision, recall and F1 plus direction accuracy on out-of-sample rows. |
| `requirements.txt` | Adds onnxruntime and tokenizers for the CPU runtime; installed into `.venv`. |
| `requirements-train.txt` | New GPU environment `.venv-train` (6 GB): torch 2.14 with CUDA, transformers 5.17, onnx, onnxruntime and pandas. |

Backups of every file edited in Part 1 (from before the edits) were saved to the session scratchpad at
`/tmp/claude-1000/.../scratchpad/ml-news-backup/`. That directory may not survive a reboot.

### Technical findings

1. **transformers 5 loads DeBERTa-v3-small in fp16 by default,** because that is the dtype stored in the
   checkpoint. Training fp16 weights with AdamW gives a NaN loss.
   - Fix: `AutoModel.from_pretrained(base, dtype=torch.float32)`, with bf16 autocast for compute. This is in
     `train.py`.
2. **The dynamo ONNX exporter output breaks onnxruntime quantisation** with a shape-inference error (768
   vs 1). Using the TorchScript exporter (`dynamo=False`, opset 17) fixes it.
3. **Dynamic int8 quantisation of DeBERTa's MatMuls ruins accuracy.**

   | Variant | Max prob diff | Flips (of 600 decisions) | Size | Latency, 1 thread |
   |---|---|---|---|---|
   | fp32 ONNX | 0.000 | 0 | 566 MB | 18.2 ms |
   | int8 all / per-channel / MatMul-only / reduce-range / FFN-only | 0.55–0.72 | 81–96 | 172–467 MB | 7–11 ms |
   | **int8 embedding table only (chosen)** | 0.04–0.05 | about 1% | 271 MB | 18–20 ms |

   Because of this, the export tolerance is 0.05 max difference and at most 2% of decisions flipping. The
   plan had said 0.03.
4. **Latency on this machine:** about 18–20 ms per headline on 1 thread and about 9 ms on 4 threads. 16
   threads is slower (about 13 ms) from oversubscription. When wiring it in, pass `threads=4`.
5. **Smoke test:** trained on only 1,402 live headlines from before Sep 18, 3 epochs, no history data.
   - Val (Sep 18–22), three-class agreement: 57.1%. The TF-IDF model trained on the same period gets 45.9%
     on those days.
   - Relevance F1: 0.73.
   - Direction accuracy on relevant rows: 80.7%, and 86% on gold headlines.
   - Golden set: 37/78.
   - It still calls Bitcoin headlines relevant. The historical data is meant to fix this.

   The smoke artefacts were in the scratchpad only. **No transformer model is in `models/` yet.**

### Historical data status (`data/history.db`, 24 MB)

- **Broad crawl** (keyword plus `site:reuters.com / cnbc.com / marketwatch.com`, weekly windows,
  2019-01 → 2026-08): stopped at 568 of 6,000 requests, with about 21k unique headlines across all years.
- **The broad crawl has low precision.** Google matches the query word anywhere in the article, not only in
  the headline.
  - The 40-call pilot labelled 1,000 broad headlines. Only 7% were gold-relevant (|xau| ≥ 0.25), against 43%
    in the live feed.
  - DeepSeek's labels themselves looked correct. For example, "Red Lobster endless shrimp losses" scored 0.
  - A keyword filter on the headline text only lifts the relevant share to 16%. Using the smoke model's
    relevance head as a filter gets 15%.
  - The broad pool is still useful as the not-relevant side.
- **Precise crawl** (`intitle:<term> site:<publisher>`, 21 queries, 2-week windows): stopped at 161 of
  4,200 requests. That gives about 1,100 precise headlines so far, and they look strongly on-topic in spot
  checks: gold, Fed, rates, FX.
- **Total:** 23,103 unique headlines, of which 1,000 are labelled (`labels` table). The labels come from the
  pilot, so almost all are from the broad pool.
- DeepSeek answers are also cached in `data/history_deepseek_cache.json`, so no headline is paid for twice.
- **Stray years:** Google misdates some items to 2009–2018. That is harmless; train/val is split by source
  and date.

---

## Next steps (in order)

Run all commands from `~/Development/ML-NEWS`.

1. **Finish the precise crawl.** It's free and needs no DeepSeek. It takes about 2.7 hours and resumes where
   it stopped.
   ```
   python3 data_collect/crawl_google_news.py --query-set precise > logs/crawl_precise.log 2>&1
   ```
   Optional: add more broad negatives with `--query-set broad`, which is 5,400 requests left and resumable.

2. **Second pilot, 20 calls (needs the user's OK for DeepSeek spend).** Check that the precise pool gives a
   high relevant share.
   ```
   ~/Development/ForexBot2/venv/bin/python data_collect/label_history.py --max-calls 20 --broad-share 0
   ```
   Then look at the per-year report. Also check for hindsight bias: does the mean xau per year track gold's
   actual yearly moves too closely?

3. **Full labelling (needs the user's OK).** About 800–1,200 calls, a few US dollars.
   ```
   ~/Development/ForexBot2/venv/bin/python data_collect/label_history.py --max-calls 1000
   ```
   Keep `--broad-share 0.3`. The run can be split across days; progress is saved per chunk.

4. **Train, export, evaluate:**
   ```
   .venv-train/bin/python transformer/train.py                      # -> models/xau_transformer/torch/
   .venv-train/bin/python transformer/export_onnx.py --src models/xau_transformer/torch
   cd eval
   ../.venv-train/bin/python eval_golden.py  --no-network --scorer transformer
   ../.venv-train/bin/python eval_archive.py --no-network --scorer transformer \
       --db ../newsdata.db --extra-labels results/backfill_labels.jsonl
   ```
   Run the evals with `.venv-train`, which has pandas, onnxruntime and the ForexBot2 imports, so that
   ForexBot2's venv stays untouched.

5. **Promotion criteria, fixed in advance:**
   - Golden set ≥ 55/78 (TF-IDF has 47).
   - Distractors ≥ 6/8.
   - Sep 18–22 three-class agreement ≥ 62.8%.
   - CPU latency ≤ 20 ms.

   If all pass: run `train.py --final`, export from `torch_final`, and write `eval/RESULTS_<date>.md`.

6. **Afterwards:**
   - Update the README and WORKFLOW doc for the transformer.
   - Decide separately whether and how ForexBot2 should use it, in shadow mode first.
   - Align ForexBot2's sklearn version with the training environment.

## Housekeeping

- ML-NEWS is not a git repo, so nothing was committed.
- `~/Development/SSH-Client/newsdata.db` is an untracked copy; delete it there.
- `.venv-train` takes 6 GB and is only needed for training and export.