Harwinder Sekhon
1d72e4006a
TF-IDF baseline (news_sentiment.py, models/xauusd_sentiment.joblib) and the two-head DeBERTa-v3-small transformer (transformer/, news_transformer.py) with its Google News crawler and DeepSeek labeller (data_collect/), the golden-set and archive evals (eval/), and the 2026-09-22 and 2026-09-27 results and progress notes. Transformer weights (model.pt, model.onnx) and the crawl database are not tracked; rebuild with transformer/train.py and transformer/export_onnx.py. The DeepSeek label cache in data/ is tracked. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
125 lines
5.7 KiB
Markdown
125 lines
5.7 KiB
Markdown
# Retrain on the Sep 8–22 archive, 2026-09-22
|
||
|
||
Data: `newsdata.db` copied from the 2026-09-22 archive, with 2,665 headlines.
|
||
- 2,661 rows are `score_source='llm'`, scored by `deepseek-v4-flash` with prompt `d32e04c6`.
|
||
- 4 rows are keyword-scored and not used.
|
||
- **Every label is DeepSeek's, not a human's or a trader's.** The 78-headline golden set is still the only
|
||
independent truth.
|
||
|
||
The ForexBot2 archive patch is live: `xau_score` is filled for every LLM row from 2026-09-18 on (1,064 rows).
|
||
Where a row also has a JSON `XAU` key (319 rows), the two values match on all of them.
|
||
|
||
DeepSeek spend for this run: 0 calls (`--no-network`, everything came from the cache).
|
||
|
||
## What changed in the code
|
||
|
||
- **`news_sentiment.py`:** the label is taken from `xau_score`, then the JSON `XAU` key, then `--extra-labels`
|
||
(backfill JSONL keyed by hash).
|
||
- Rows from test feeds are dropped.
|
||
- `--scope all|gold` was added.
|
||
- The metrics now record `label_sources`, `scope` and `published_range`.
|
||
- The model format is unchanged.
|
||
- **`eval_common.load_archive_rows`:** uses the same label order and sets `xau_src` and `gold` (whether the row
|
||
has a JSON XAU key) on each row.
|
||
- **`eval_golden.py`:** new `--model` option.
|
||
- **`eval_archive.py`:** new `--model`, `--extra-labels` and `--train-scope` options.
|
||
- Part A now also scores rows published after the bundle's `trained_at`. For the 09-17 bundle those rows
|
||
are out of sample.
|
||
- Part B pools its results by gold rows, macro rows and days on or after 09-18.
|
||
- In Part B the test day is always every labelled row, so both train scopes are graded on the same rows.
|
||
|
||
## Candidates
|
||
|
||
| Model | Train rows | Labels (bear / neu / bull) | Stratified acc / F1 | Chronological acc / F1 |
|
||
|---|---|---|---|---|
|
||
| shipped 09-17 (gold only, Sep 8–17) | 606 | 250 / 135 / 221 | 73.0% / 0.709 | 69.7% / 0.671 |
|
||
| `--scope gold` (Sep 8–22) | 945 | 359 / 202 / 384 | 80.4% / 0.793 | 74.6% / 0.734 |
|
||
| **`--scope all` + backfill (Sep 8–22)** | **2,620** | 909 / 893 / 818 | 69.3% / 0.689 | 66.6% / 0.659 |
|
||
|
||
You can't compare the built-in holdouts across rows of this table:
|
||
- The `gold` and 09-17 holdouts contain only headlines that mention gold, which are the easy ones.
|
||
- The `all` holdout is 64% macro headlines.
|
||
|
||
The next two sections grade every model on the same rows.
|
||
|
||
## Same unseen rows: Sep 18–22 (1,022–1,066 headlines, DeepSeek label at ±0.05)
|
||
|
||
| Model | All | Gold-key | Macro |
|
||
|---|---|---|---|
|
||
| shipped 09-17 bundle (never saw these days) | 45.9% (F1 0.455) | 68.2% | 36.2% |
|
||
| `gold` recipe, retrained daily on days < d | 48.3% (F1 0.478) | **73.8%** | 37.1% |
|
||
| **`all` recipe, retrained daily on days < d** | **62.8% (F1 0.629)** | 71.9% | **58.8%** |
|
||
|
||
Pooled over every daily fold from Sep 9–22 (2,566 / 2,530 rows):
|
||
|
||
| Recipe | 3-class | Macro-F1 | Gold-key | Macro | Keyword fallback |
|
||
|---|---|---|---|---|---|
|
||
| `all` | 58.2% | 0.578 | 65.1% | 54.3% | 44.7% |
|
||
| `gold` | 48.3% | 0.479 | 67.8% | 37.3% | 44.7% |
|
||
|
||
## Golden set (78 reviewer-labelled headlines, argmax label)
|
||
|
||
| Model | 3-class | McNemar vs keyword | Spearman vs DeepSeek |
|
||
|---|---|---|---|
|
||
| shipped 09-17 | 39/78 (50.0%) | 28 vs 13, p = 0.028 | +0.305 |
|
||
| `gold` | 40/78 (51.3%) | 29 vs 13, p = 0.020 | +0.310 |
|
||
| **`all`** | **47/78 (60.3%)** | 31 vs 8, p = 0.0003 | +0.423 |
|
||
| DeepSeek | 78/78 | | |
|
||
|
||
Hits by category, shipped → `all`:
|
||
|
||
| Category | Shipped | `all` |
|
||
|---|---|---|
|
||
| hawkish_fed | 3 | 8 |
|
||
| strong_usd | 2 | 5 |
|
||
| distractors | 1 | 5 |
|
||
| negation_traps | 0 | 1 |
|
||
| weak_usd | 4 | 4 |
|
||
| real_yields | 3 | 3 |
|
||
| inverted_metrics | 3 | 3 |
|
||
| dovish_fed | 6 | 4 |
|
||
| gold_direct | 8 | 7 |
|
||
| safe_haven | 8 | 7 |
|
||
| risk_on | 1 | 0 |
|
||
|
||
The `all` model's weak spots:
|
||
- **Easing and falling yields read as bearish.** "Fed cuts rates 25bp" → bearish 0.78, and "Treasury yields sink"
|
||
→ bearish 0.78. Nearly all of the September macro data comes from a hiking regime, so the model links
|
||
"Fed" and "yields" with bearish and has almost no examples of easing.
|
||
- **Risk-on headlines score 0/4.** "Stocks roar to record highs" → bullish.
|
||
- **Negation still fails** (1/8).
|
||
|
||
## Verdict
|
||
|
||
**Promoted `--scope all`.** It is now `models/xauusd_sentiment.joblib`. The 09-17 bundle is kept in
|
||
`models/archive/xauusd_sentiment_20260917.*`, and both candidates are in `models/candidates/`.
|
||
|
||
- **Against the shipped model:** on the same unseen days it wins by 17 points overall and 22 points on
|
||
macro headlines.
|
||
- **Against the gold-only recipe:** it gives up 1.9 points on gold headlines and gains 21.7 points on macro.
|
||
- **Golden set:** 39 → 47/78.
|
||
|
||
The promotion rule written before the run said "chronological macro-F1 ≥ shipped". **Taken literally, that
|
||
test fails:** 0.659 against 0.671. The two numbers come from different test sets, one gold-only and one
|
||
mostly macro. On the same unseen rows the retrained model wins, and that is the comparison the rule was
|
||
meant to capture.
|
||
|
||
This is still not a DeepSeek replacement:
|
||
- It is 31 golden headlines behind DeepSeek.
|
||
- It learns only what DeepSeek says, not how price moves.
|
||
- It has 2,620 labels over 15 days, which is short of the "about 4,000 labels over 3+ weeks" rule. Keep it in
|
||
shadow mode with DeepSeek as the answer.
|
||
|
||
Nothing in ForexBot2 imports the model, so the promotion changes nothing live.
|
||
|
||
`ForexBot2/venv` has sklearn 1.9.0 and the bundle was pickled with 1.9.1. That raises an
|
||
`InconsistentVersionWarning`, which the eval scripts suppress. The 09-17 bundle had the same mismatch. Align
|
||
the two venvs before any live use.
|
||
|
||
## Next
|
||
|
||
1. Keep collecting. Every new day adds about 300 labels from `xau_score`. Retrain weekly and rerun both
|
||
evals.
|
||
2. Easing and falling yields are the obvious gap. Around 20–40 DeepSeek-labelled dovish, yields-down and
|
||
risk-on headlines would test whether this gap comes from the regime or from the model.
|