News-ML-Decoder/PROGRESS_2026-09-22.md

Session progress: 2026-09-22 → 09-23 (UTC)

This session had two parts:

1. Retrain the TF-IDF XAUUSD model on the new newsdata.db. Done and promoted.
2. Build a two-stage transformer model: is the headline gold-related, and if so, bullish or bearish? The tooling is built and tested. Historical data collection is partly done. The model is not trained on history yet.

DeepSeek calls made this session: 40, all in the history labelling pilot. The user asked for no more DeepSeek calls on 2026-09-23. All background jobs are stopped.

---

Important facts to carry forward

• The labels come from DeepSeek, not from traders. The newsdata.db handed over as "results from professional traders" has 2,661 rows scored by deepseek-v4-flash (prompt sha1 d32e04c6) and 4 rows scored by keywords. The user confirmed we train on these. Every model here learns to copy DeepSeek; it does not learn price moves.
• The golden set is the only independent truth: eval/data/llm_news_golden.csv, 78 reviewer-labelled headlines. DeepSeek scores 78/78. It is excluded from every training set.
• The DeepSeek ceiling: on re-scoring, DeepSeek agrees with its own earlier answer about 90% of the time. No student model can beat that.
• The deployment rule from 2026-09-08 still applies: about 4,000 labels over 3+ weeks before any live use.
• Nothing in ForexBot2 changed, and ForexBot2 doesn't import any ML-NEWS model.

---

Part 1: TF-IDF retrain (done)

Full write-up: eval/RESULTS_2026-09-22.md.

• newsdata.db was copied into ML-NEWS/newsdata.db. The original is still untracked in ~/Development/SSH-Client/; delete it there when convenient.

• news_sentiment.py now takes each label from the first source that has one:

  1. the xau_score column,
  2. the JSON XAU key,
  3. the backfill file given with --extra-labels.

  It also has a new --scope all|gold option and drops rows from test feeds.

• Eval scripts gained --model, --extra-labels and --train-scope. They also get label source and gold flags from load_archive_rows.

• Promoted: --scope all with the backfill labels, 2,620 headlines from Sep 8–22.

  • The old bundle is in models/archive/xauusd_sentiment_20260917.*.
  • The candidates are in models/candidates/.

Results on the Sep 18–22 days, which none of these models trained on:

Model	All	Gold headlines	Macro headlines	Golden set
Old 09-17 model	45.9%	68.2%	36.2%	39/78
Gold-only retrain	48.3%	73.8%	37.1%	40/78
All-news retrain (promoted)	62.8%	71.9%	58.8%	47/78

• The promotion rule, taken literally, failed. It required a chronological-holdout macro-F1 of at least 0.671, and the new model scored 0.659. The two holdouts contain different kinds of headline, so the numbers don't compare. On identical unseen rows the new model wins clearly. The user was told this.
• Known weak spots: dovish news and falling yields read as bearish. September was a hiking regime, so there are almost no easing examples. Risk-on scores 0/4 and negation 1/8.
• Version mismatch: ForexBot2's venv has sklearn 1.9.0 but the bundle was pickled with 1.9.1, which raises a warning. The old bundle had the same problem.

---

Part 2: two-stage transformer (in progress)

The plan is in ~/.claude/plans/you-are-machine-learning-cozy-teapot.md. The user chose:

• Training on the GPU, but the trained model must run on CPU.
• More data from DeepSeek labelling historical headlines.
• "Gold-related" means the headline moves the gold price, not that it mentions gold.

Design

• Model: microsoft/deberta-v3-small encoder with mean pooling and two single-logit heads.
  • Relevance head: target is 1 if |xau| ≥ 0.25 and 0 if |xau| ≤ 0.10. Rows in between carry no loss.
  • Direction head: target is the sign of xau, on relevant rows only.
• Input: the headline only, at most 64 tokens.
• Output: p_relevant and p_bullish, combined as score = p_relevant × (2·p_bullish − 1). The label is neutral when p_relevant is below the tuned threshold.

Files added or changed

File	What it does
data_collect/crawl_google_news.py	Google News RSS crawler with date windows. It can resume, handles 429/503 with backoff, and shuffles the windows so a partial crawl covers every year. It has two query sets, --query-set broad and --query-set precise (see below).
data_collect/label_history.py	DeepSeek labeller with a hard --max-calls stop and its own per-process daily cap. It samples round-robin over (quarter × query) strata from two pools, with --broad-share 0.3 by default. It excludes the golden set and newsdata.db overlaps, and prints a label report per year.
transformer/train.py	Two-head fine-tuning. Val is newsdata from 2026-09-18 on and is never used to choose anything. Dev is a random 5%, used to pick the epoch and the relevance threshold. --final retrains on everything into torch_final/.
transformer/export_onnx.py	ONNX export plus int8 on the embedding table only. Checks tokenizer and probability parity against PyTorch and measures CPU latency.
news_transformer.py	CPU runtime needing only onnxruntime, tokenizers and numpy. Provides load_model, score_batch, score_news and a CLI score --headline/--input.
eval/eval_common.py	Adds TransformerScorer and make_scorer, an oos_from field on scorers, and daily_cap for install_research_client and DeepSeekScorer.
eval/eval_golden.py, eval/eval_archive.py	New `--scorer tfidf
requirements.txt	Adds onnxruntime and tokenizers for the CPU runtime; installed into .venv.
requirements-train.txt	New GPU environment .venv-train (6 GB): torch 2.14 with CUDA, transformers 5.17, onnx, onnxruntime and pandas.

Backups of every file edited in Part 1 (from before the edits) were saved to the session scratchpad at /tmp/claude-1000/.../scratchpad/ml-news-backup/. That directory may not survive a reboot.

Technical findings

1. transformers 5 loads DeBERTa-v3-small in fp16 by default, because that is the dtype stored in the checkpoint. Training fp16 weights with AdamW gives a NaN loss.

   • Fix: AutoModel.from_pretrained(base, dtype=torch.float32), with bf16 autocast for compute. This is in train.py.

2. The dynamo ONNX exporter output breaks onnxruntime quantisation with a shape-inference error (768 vs 1). Using the TorchScript exporter (dynamo=False, opset 17) fixes it.

3. Dynamic int8 quantisation of DeBERTa's MatMuls ruins accuracy.

   Variant	Max prob diff	Flips (of 600 decisions)	Size	Latency, 1 thread
   fp32 ONNX	0.000	0	566 MB	18.2 ms
   int8 all / per-channel / MatMul-only / reduce-range / FFN-only	0.55–0.72	81–96	172–467 MB	7–11 ms
   int8 embedding table only (chosen)	0.04–0.05	about 1%	271 MB	18–20 ms

   Because of this, the export tolerance is 0.05 max difference and at most 2% of decisions flipping. The plan had said 0.03.

4. Latency on this machine: about 18–20 ms per headline on 1 thread and about 9 ms on 4 threads. 16 threads is slower (about 13 ms) from oversubscription. When wiring it in, pass threads=4.

5. Smoke test: trained on only 1,402 live headlines from before Sep 18, 3 epochs, no history data.

   • Val (Sep 18–22), three-class agreement: 57.1%. The TF-IDF model trained on the same period gets 45.9% on those days.
   • Relevance F1: 0.73.
   • Direction accuracy on relevant rows: 80.7%, and 86% on gold headlines.
   • Golden set: 37/78.
   • It still calls Bitcoin headlines relevant. The historical data is meant to fix this.

   The smoke artefacts were in the scratchpad only. No transformer model is in models/ yet.

Historical data status (data/history.db, 24 MB)

• Broad crawl (keyword plus site:reuters.com / cnbc.com / marketwatch.com, weekly windows, 2019-01 → 2026-08): stopped at 568 of 6,000 requests, with about 21k unique headlines across all years.
• The broad crawl has low precision. Google matches the query word anywhere in the article, not only in the headline.
  • The 40-call pilot labelled 1,000 broad headlines. Only 7% were gold-relevant (|xau| ≥ 0.25), against 43% in the live feed.
  • DeepSeek's labels themselves looked correct. For example, "Red Lobster endless shrimp losses" scored 0.
  • A keyword filter on the headline text only lifts the relevant share to 16%. Using the smoke model's relevance head as a filter gets 15%.
  • The broad pool is still useful as the not-relevant side.
• Precise crawl (intitle:<term> site:<publisher>, 21 queries, 2-week windows): stopped at 161 of 4,200 requests. That gives about 1,100 precise headlines so far, and they look strongly on-topic in spot checks: gold, Fed, rates, FX.
• Total: 23,103 unique headlines, of which 1,000 are labelled (labels table). The labels come from the pilot, so almost all are from the broad pool.
• DeepSeek answers are also cached in data/history_deepseek_cache.json, so no headline is paid for twice.
• Stray years: Google misdates some items to 2009–2018. That is harmless; train/val is split by source and date.

---

Next steps (in order)

Run all commands from ~/Development/ML-NEWS.

1. Finish the precise crawl. It's free and needs no DeepSeek. It takes about 2.7 hours and resumes where it stopped.

   python3 data_collect/crawl_google_news.py --query-set precise > logs/crawl_precise.log 2>&1
   

   Optional: add more broad negatives with --query-set broad, which is 5,400 requests left and resumable.

2. Second pilot, 20 calls (needs the user's OK for DeepSeek spend). Check that the precise pool gives a high relevant share.

   ~/Development/ForexBot2/venv/bin/python data_collect/label_history.py --max-calls 20 --broad-share 0
   

   Then look at the per-year report. Also check for hindsight bias: does the mean xau per year track gold's actual yearly moves too closely?

3. Full labelling (needs the user's OK). About 800–1,200 calls, a few US dollars.

   ~/Development/ForexBot2/venv/bin/python data_collect/label_history.py --max-calls 1000
   

   Keep --broad-share 0.3. The run can be split across days; progress is saved per chunk.

4. Train, export, evaluate:

   .venv-train/bin/python transformer/train.py                      # -> models/xau_transformer/torch/
   .venv-train/bin/python transformer/export_onnx.py --src models/xau_transformer/torch
   cd eval
   ../.venv-train/bin/python eval_golden.py  --no-network --scorer transformer
   ../.venv-train/bin/python eval_archive.py --no-network --scorer transformer \
       --db ../newsdata.db --extra-labels results/backfill_labels.jsonl
   

   Run the evals with .venv-train, which has pandas, onnxruntime and the ForexBot2 imports, so that ForexBot2's venv stays untouched.

5. Promotion criteria, fixed in advance:

   • Golden set ≥ 55/78 (TF-IDF has 47).
   • Distractors ≥ 6/8.
   • Sep 18–22 three-class agreement ≥ 62.8%.
   • CPU latency ≤ 20 ms.

   If all pass: run train.py --final, export from torch_final, and write eval/RESULTS_<date>.md.

6. Afterwards:

   • Update the README and WORKFLOW doc for the transformer.
   • Decide separately whether and how ForexBot2 should use it, in shadow mode first.
   • Align ForexBot2's sklearn version with the training environment.

Housekeeping

• ML-NEWS is not a git repo, so nothing was committed.
• ~/Development/SSH-Client/newsdata.db is an untracked copy; delete it there.
• .venv-train takes 6 GB and is only needed for training and export.