Harwinder Sekhon
1d72e4006a
TF-IDF baseline (news_sentiment.py, models/xauusd_sentiment.joblib) and the two-head DeBERTa-v3-small transformer (transformer/, news_transformer.py) with its Google News crawler and DeepSeek labeller (data_collect/), the golden-set and archive evals (eval/), and the 2026-09-22 and 2026-09-27 results and progress notes. Transformer weights (model.pt, model.onnx) and the crawl database are not tracked; rebuild with transformer/train.py and transformer/export_onnx.py. The DeepSeek label cache in data/ is tracked. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
289 lines
16 KiB
Markdown
289 lines
16 KiB
Markdown
# Two-head transformer trained on history + archive, 2026-09-27
|
||
|
||
**Outcome: the transformer passes all four promotion criteria and is exported to `models/xau_transformer/`.**
|
||
The TF-IDF bundle `models/xauusd_sentiment.joblib` is untouched, and nothing in ForexBot2 imports either
|
||
model, so nothing changed live.
|
||
|
||
DeepSeek spend today: **560 calls** in four runs (20 pilot, 200 full run, 140 precise top-up, 200 growth
|
||
set; 15 batches failed and were skipped), labelling 13,625 crawled headlines. Every eval ran with
|
||
`--no-network`.
|
||
|
||
**The shipped bundle is from round 3** (see the last section): 17,245 training rows, golden **72/78**,
|
||
with the two weak categories fixed. Round 1 (8,870 rows, 65/78) and round 2 (12,270 rows, 67/78) are kept
|
||
below because round 1's val numbers are the cleanest comparison against TF-IDF.
|
||
|
||
## Data
|
||
|
||
### Labelling the crawl
|
||
|
||
The precise crawl (`intitle:<term> site:<publisher>`) was resumed and ran in the background all session.
|
||
Two DeepSeek runs labelled its pool, plus a 30% share of the broad pool for the not-relevant side:
|
||
|
||
| Pool | Labelled | Gold-relevant (|xau| ≥ 0.25) | Bull / bear among relevant |
|
||
|---|---|---|---|
|
||
| precise (`intitle:`) | 2,979 | 1,390 (47%) | 870 / 520 |
|
||
| broad (keyword anywhere) | 3,271 | 320 (10%) | 253 / 67 |
|
||
| **history total** | **6,250** | 1,710 | |
|
||
|
||
- The 20-call pilot on the precise pool alone gave 52% relevant against 7% for the broad pilot on
|
||
2026-09-23. Spot checks looked right ("Dollar slumps as risk appetite rebounds" +0.50, "This is what a
|
||
$52 million Hamptons beach house looks like" 0.00).
|
||
- Relevant share by query term: "rate cut" 87%, "safe haven" 88%, payrolls 80%, "rate hike" 72%, yields
|
||
65%, Fed 61%, gold 49%, dollar 44%, inflation 38%, Powell 31%.
|
||
- **Hindsight check:** mean xau per year is +0.02 to +0.06 for 2019–2024, +0.12 for 2025 and 0.00 for 2026.
|
||
2025 was the big gold rally, so there may be some hindsight in DeepSeek's 2025 labels, but the effect is
|
||
small and the direction head is trained on sign only.
|
||
- About 1,900 precise headlines are still unlabelled because the crawl kept adding them; it needs roughly
|
||
80 more calls.
|
||
|
||
### Training set (`transformer/train.py`)
|
||
|
||
| | Rows |
|
||
|---|---|
|
||
| history labels | 6,250 |
|
||
| newsdata Sep 8–22 + backfill | 2,637 |
|
||
| **unique after dedupe, golden removed** | **8,870** |
|
||
|
||
| Split | n | relevant | irrelevant | grey | bull | bear |
|
||
|---|---|---|---|---|---|---|
|
||
| train | 7,460 | 2,332 | 4,492 | 636 | 1,290 | 1,042 |
|
||
| dev (random 5%, picks epoch + threshold) | 392 | 119 | 245 | 28 | 72 | 47 |
|
||
| val (newsdata ≥ 2026-09-18, never used to choose) | 1,018 | 378 | 501 | 139 | 171 | 207 |
|
||
|
||
## Sweep on the enlarged set (`microsoft/deberta-v3-small`, lr 5e-5 unless noted)
|
||
|
||
| Run | Dev 3-class | Best epoch / thr | Val 3-class | Val macro-F1 | Val rel F1 | Val dir acc |
|
||
|---|---|---|---|---|---|---|
|
||
| 4 ep, bs 32 | 0.824 | 4 / 0.25 | 0.708 | 0.704 | 0.830 | 0.907 |
|
||
| **4 ep, bs 64 (chosen)** | **0.824** | 3 / 0.60 | **0.720** | **0.715** | 0.841 | **0.942** |
|
||
| 6 ep, bs 32 | 0.814 | 4 / 0.20 | 0.713 | 0.708 | 0.842 | 0.897 |
|
||
| 8 ep, bs 64 | 0.821 | 8 / 0.40 | 0.709 | 0.704 | 0.836 | 0.905 |
|
||
| 6 ep, bs 32, lr 3e-5 | 0.821 | 6 / 0.20 | 0.714 | 0.711 | 0.834 | 0.926 |
|
||
|
||
The two 4-epoch runs tie on dev; bs 64 has the higher dev relevance F1 (0.869 vs 0.865) and is the script
|
||
default, so it was chosen. All five runs land within 1.2 points on val: the data did the work, not the
|
||
hyperparameters.
|
||
|
||
For reference, the same sweep on the 3,620-row set before labelling (earlier today) gave dev 0.72–0.78 and
|
||
val 0.62–0.68, and the chosen model then scored 49/78 golden. deberta-v3-base was also tried on the small
|
||
set: half a dev point better, but 38.6 ms per headline on 1 thread, over the 20 ms limit.
|
||
|
||
## Same unseen rows: Sep 18–22 (DeepSeek label at ±0.05, non-final model)
|
||
|
||
| Model | All (n=1,022) | Gold-key (313) | Macro (709) | Rel P / R / F1 | Dir acc on relevant |
|
||
|---|---|---|---|---|---|
|
||
| TF-IDF `all` 09-22 (promoted) | 62.8% | 71.9% | 58.8% | | |
|
||
| Transformer on 3,620 rows | 65.7% (F1 0.656) | 75.1% | 61.5% | 0.685 / 0.895 / 0.776 | 89.8% |
|
||
| **Transformer on 8,870 rows** | **71.9% (F1 0.714)** | **83.7%** | **66.7%** | 0.802 / 0.885 / 0.841 | **94.2%** |
|
||
|
||
The DeepSeek self-agreement ceiling is about 90%, so 72% is 18 points short of "as good as DeepSeek".
|
||
|
||
## Golden set (78 reviewer-labelled headlines, argmax label)
|
||
|
||
| Model | 3-class | McNemar vs keyword | Spearman vs DeepSeek |
|
||
|---|---|---|---|
|
||
| TF-IDF `all` 09-22 | 47/78 (60.3%) | 31 vs 8, p = 0.0003 | +0.423 |
|
||
| Transformer on 3,620 rows | 49/78 (62.8%) | 30 vs 5 | +0.556 |
|
||
| Transformer on 8,870 rows, non-final | 62/78 (79.5%) | 40 vs 2 | +0.676 |
|
||
| **Transformer, `--final` (shipped)** | **65/78 (83.3%)** | 42 vs 1 | +0.775 |
|
||
| DeepSeek | 78/78 | | |
|
||
|
||
The final model also trained on dev and val, but never on the golden set, so its golden score is a fair
|
||
number. It is 3 headlines above the non-final model, which is within noise for 78 rows.
|
||
|
||
Hits by category:
|
||
|
||
| Category | TF-IDF | 3,620 rows | 8,870 rows | final |
|
||
|---|---|---|---|---|
|
||
| distractors | 5/8 | 7/8 | 8/8 | 8/8 |
|
||
| safe_haven | 7/8 | 8/8 | 7/8 | 8/8 |
|
||
| dovish_fed | 4/8 | 5/8 | 7/8 | 8/8 |
|
||
| real_yields | 3/6 | 4/6 | 5/6 | 6/6 |
|
||
| negation_traps | 1/8 | 3/8 | 6/8 | 7/8 |
|
||
| gold_direct | 7/8 | 7/8 | 7/8 | 7/8 |
|
||
| hawkish_fed | 8/8 | 7/8 | 7/8 | 7/8 |
|
||
| weak_usd | 4/7 | 5/7 | 6/7 | 6/7 |
|
||
| inverted_metrics | 3/6 | 2/6 | 6/6 | 4/6 |
|
||
| strong_usd | 5/7 | 1/7 | 3/7 | 3/7 |
|
||
| risk_on | 0/4 | 0/4 | 0/4 | 1/4 |
|
||
|
||
Remaining weak spots of the shipped model:
|
||
- **Risk-on is 1/4.** "Stocks roar to record highs as trade deal signed" and "Global equities rally on strong
|
||
earnings and easing tensions" are bullish at 0.7–0.8. The model reads "rally" as gold rallying.
|
||
- **Strong US data is 3/7.** "Consumer confidence surges to post-pandemic high" → bullish 0.83, "ISM
|
||
services index jumps" → bullish 0.52. Good-news-for-the-economy still doesn't map to bad-for-gold
|
||
reliably. The precise crawl has no query for growth data (GDP, ISM, retail sales, confidence); adding one
|
||
is the obvious next crawl.
|
||
- "FOMC minutes reveal growing support for policy easing" is still bearish at 0.63.
|
||
|
||
## Export (`transformer/export_onnx.py`, from `torch_final`)
|
||
|
||
- `models/xau_transformer/model.onnx`, 271 MB, int8 embedding table, fp32 compute. `tokenizer.json`,
|
||
`meta.json` (`final: true`, threshold 0.6, `export_checks` inside).
|
||
- Parity over 500 val headlines: tokenizer 500/500, mean |diff| 0.003–0.004, **0 decision flips**, max
|
||
|diff| 0.075 on `p_relevant` and 0.059 on `p_bullish`. The script prints `PARITY FAILED` because the max
|
||
is over its 0.05 tolerance. With zero flips the tolerance is the problem, not the export; it should be
|
||
loosened to about 0.15, or the check should be flips only.
|
||
- CPU latency: 18.9 ms on 1 thread, 9.6 ms on 16 threads, about 8.5 ms in the evals (4 threads).
|
||
|
||
## Promotion criteria (fixed on 2026-09-22)
|
||
|
||
| Criterion | Target | Non-final | Final | |
|
||
|---|---|---|---|---|
|
||
| Golden set | ≥ 55/78 | 62/78 | 65/78 | pass |
|
||
| Distractors | ≥ 6/8 | 8/8 | 8/8 | pass |
|
||
| Sep 18–22 three-class agreement | ≥ 62.8% | 71.9% | (in-sample) | pass |
|
||
| CPU latency, 1 thread | ≤ 20 ms | 19.0 ms | 18.9 ms | pass |
|
||
|
||
**All four pass.** `train.py --final` was run (8,870 rows, 3 epochs) and the shipped ONNX comes from
|
||
`torch_final`. The non-final weights (the ones the val numbers above belong to) are in
|
||
`models/xau_transformer/torch/`.
|
||
|
||
What this does and does not mean:
|
||
- It is a better DeepSeek imitation than TF-IDF: +18 golden headlines, +9 points on unseen days, +35 points
|
||
of Spearman.
|
||
- It still learns DeepSeek's opinion, not price moves, and DeepSeek itself only agrees with itself 90% of
|
||
the time.
|
||
- The "about 4,000 labels over 3+ weeks" deployment rule is met on count (8,870) but the live-feed part is
|
||
still one fortnight; the history labels are a different distribution (Reuters/CNBC/MarketWatch titles,
|
||
not the bot's feeds). Shadow mode first.
|
||
|
||
## Next
|
||
|
||
1. **Crawl growth data:** add `intitle:` queries for GDP, "retail sales", ISM, "consumer confidence",
|
||
"jobless claims" and `stocks OR equities` (for risk-on), then label them (about 100–200 calls).
|
||
2. **Finish labelling the precise pool:** about 1,900 headlines left, roughly 80 calls, plus whatever the
|
||
crawl adds. The crawl is still running; it has about 3,300 windows to go.
|
||
3. Retrain weekly as the live archive grows; the whole train → export → eval loop takes under 5 minutes.
|
||
4. Loosen the export tolerance or make it flips-only.
|
||
5. Wire into ForexBot2 in shadow mode with `threads=4`, DeepSeek still the answer. Decide the
|
||
promotion rule for that separately.
|
||
|
||
## Round 2: labelling the rest of the precise pool (same day)
|
||
|
||
The crawl kept running, so after round 1 there were about 3,500 unlabelled precise headlines. A 140-call
|
||
run with `--broad-share 0` labelled 3,400 of them (4 batches failed). History labels: 9,650, of which
|
||
6,185 precise (47% relevant, 1,955 bull / 941 bear) and 3,465 broad (12% relevant). About 570 precise
|
||
headlines were still unlabelled at the end because the crawl was still adding them.
|
||
|
||
Training set: 12,270 unique headlines. Train 10,690 / dev 562 / val 1,018 (val unchanged).
|
||
|
||
Default config only (lr 5e-5, 4 epochs, batch 64), no sweep: round 1 showed the hyperparameters don't
|
||
matter at this size. Dev picked epoch 4, threshold 0.2.
|
||
|
||
| | Round 1 (8,870 rows) | Round 2 (12,270 rows) |
|
||
|---|---|---|
|
||
| Val 3-class (unseen Sep 18–22) | 72.0% | 71.7% |
|
||
| Val macro-F1 | 0.715 | 0.715 |
|
||
| Val relevance P / R / F1 | 0.803 / 0.884 / 0.841 | 0.759 / 0.926 / 0.834 |
|
||
| Val direction acc on relevant | 94.2% | 93.4% |
|
||
| Archive eval, Sep 18–22 all / gold / macro | 71.9 / 83.7 / 66.7% | 71.8 / 83.1 / 66.9% |
|
||
| Golden, non-final | 62/78 | 64/78 |
|
||
| **Golden, final (shipped)** | 65/78 | **67/78** |
|
||
| Spearman vs DeepSeek, final | +0.775 | +0.765 |
|
||
| Export flips / max diff, final | 0 / 0.075 | 4 (0.4%) / 0.104 |
|
||
| CPU latency, 1 thread | 18.9 ms | 18.9 ms |
|
||
|
||
- **The live-feed metric has saturated** at about 72% with this kind of data. The extra 3,400 history
|
||
labels changed nothing there. More gains on the live feed will have to come from live-feed labels.
|
||
- **The golden set still moved,** 65 → 67 final. Hawkish Fed went 7 → 8, inverted metrics 4 → 6.
|
||
- Per category, final: distractors 8/8, dovish_fed 8/8, hawkish_fed 8/8, safe_haven 8/8, inverted_metrics
|
||
6/6, negation_traps 7/8, gold_direct 7/8, weak_usd 6/7, real_yields 5/6, strong_usd 3/7, risk_on 1/4.
|
||
- **Strong USD (3/7) and risk-on (1/4) did not move,** as expected: the crawl has no queries for growth
|
||
data or equities. That is the next crawl, not the next labelling run.
|
||
|
||
Promotion criteria, round 2 non-final: golden 64/78, distractors 8/8, Sep 18–22 71.8%, latency 18.9 ms.
|
||
**All pass.** `train.py --final` ran on all 12,270 rows for 4 epochs; the shipped `model.onnx` is from that
|
||
`torch_final`. Round 1's weights are in the session scratchpad (`sweep2/small_e4_bs64_8870rows`,
|
||
`sweep2/torch_final_8870rows`).
|
||
|
||
## Round 3: teaching the two weak spots (same day)
|
||
|
||
Strong-US-data (3/7) and risk-on (1/4) had not moved through rounds 1 and 2 because no crawl query covered
|
||
growth data or equities. Fix:
|
||
|
||
1. **New crawler query set `growth`** (`crawl_google_news.py --query-set growth --window-days 30`): 20
|
||
`intitle:` queries on reuters.com and cnbc.com for GDP, retail sales, ISM, PMI, consumer confidence,
|
||
jobless claims, consumer spending, manufacturing, "record high", "risk appetite", "Wall Street",
|
||
"S&P 500", ceasefire, "trade deal", equities. 1,880 requests, about 70 minutes, 6,183 new headlines
|
||
(the precise crawl was paused meanwhile so Google saw one crawler, then resumed).
|
||
2. **Labeller changes** (`label_history.py`): a `--query-set` option restricts labelling to headlines a
|
||
given crawler set found (the growth queries had also re-tagged thousands of old broad headlines), and
|
||
`is_latin()` skips Reuters' Chinese and Japanese wire items that `intitle:GDP` and `intitle:PMI` match.
|
||
3. **200 calls, growth set only**, 4,975 headlines labelled, 1 failed. DeepSeek called 19% relevant, 734
|
||
bullish vs 216 bearish: "Wall Street falls" and "ceasefire talks collapse" are risk-off, and
|
||
`"record high"` also catches gold's own records. The bearish side is the pattern the model lacked:
|
||
"Yields rise on improving risk appetite" −0.50, "US dollar rises after jobless claims data" −0.50.
|
||
|
||
| Query | Labelled | Relevant | Bear / bull |
|
||
|---|---|---|---|
|
||
| "S&P 500" | 692 | 16% | 36 / 74 |
|
||
| GDP | 690 | 12% | 17 / 67 |
|
||
| "Wall Street" | 651 | 18% | 12 / 105 |
|
||
| "record high" | 624 | 14% | 23 / 67 |
|
||
| manufacturing | 457 | 6% | 6 / 20 |
|
||
| ceasefire | 379 | 58% | 11 / 208 |
|
||
| "trade deal" | 307 | 4% | 4 / 9 |
|
||
| PMI | 304 | 6% | 3 / 15 |
|
||
| "retail sales" | 293 | 18% | 23 / 29 |
|
||
| equities | 205 | 28% | 14 / 44 |
|
||
| "jobless claims" | 126 | 57% | 35 / 37 |
|
||
| "risk appetite" | 72 | 72% | 23 / 29 |
|
||
| consumer spending / confidence, ISM | 175 | 15–56% | 9 / 30 |
|
||
|
||
Training set: 17,245 unique headlines. Train 15,416 / dev 811 / val 1,018 (val unchanged). Default config,
|
||
dev picked epoch 4, threshold 0.2.
|
||
|
||
| | Round 2 (12,270) | Round 3 (17,245) |
|
||
|---|---|---|
|
||
| Val 3-class (unseen Sep 18–22) | 71.7% | 71.2% |
|
||
| Archive eval, Sep 18–22 all / gold / macro | 71.8 / 83.1 / 66.9% | 71.5 / 78.6 / 68.4% |
|
||
| Val relevance P / R / F1 | 0.759 / 0.926 / 0.834 | 0.724 / 0.929 / 0.813 |
|
||
| Val direction acc on relevant | 93.4% | 89.4% |
|
||
| Golden, non-final | 64/78 | **71/78** |
|
||
| **Golden, final (shipped)** | 67/78 | **72/78** |
|
||
| Spearman vs DeepSeek, final | +0.765 | +0.860 |
|
||
| McNemar vs DeepSeek, final | DS-only-right 11, p = 0.001 | DS-only-right 6, p = 0.031 |
|
||
| Export flips / max diff, final | 4 / 0.104 | 2 / 0.077 |
|
||
| CPU latency, 1 thread | 18.9 ms | 18.8 ms |
|
||
|
||
Golden by category, final: **strong_usd 3/7 → 7/7, risk_on 1/4 → 4/4**, inverted_metrics 6/6,
|
||
distractors 8/8, dovish_fed 8/8, safe_haven 8/8, negation_traps 7/8, hawkish_fed 7/8, weak_usd 6/7,
|
||
real_yields 5/6, gold_direct 6/8.
|
||
|
||
The six remaining misses:
|
||
- "Powell says restrictive policy needed for longer as inflation proves sticky" → bullish 0.83 (hawkish
|
||
read as bullish; the only hawkish miss).
|
||
- "Rising real interest rates dim appeal of non-yielding assets" → bullish 0.95 (no "gold" in the text;
|
||
"non-yielding assets" is unseen vocabulary).
|
||
- "Bullion demand weakens as Indian jewellery buying stalls" → neutral (relevance 0.0; physical demand
|
||
is absent from the training feeds).
|
||
- "Miners struggle to meet gold demand as supply deficit widens" → bearish 0.32 ("miners struggle" reads
|
||
as bad news).
|
||
- "Greenback weakens as US retail sales unexpectedly contract" → neutral (relevance just under 0.2).
|
||
- "No safe-haven bid for gold despite equity wobble" → bullish 0.15, low confidence.
|
||
|
||
Trade-offs to note honestly:
|
||
- **The live-feed numbers slipped a little** (gold-key 83.1 → 78.6%, direction on relevant 93.4 → 89.4%)
|
||
while macro rose (66.9 → 68.4%). The all-rows figure is flat at 71.5%. The history labels now outnumber
|
||
the live-feed labels 9 to 1, and they are a different distribution (wire headlines, no descriptions).
|
||
If the live feed is what matters, weight it or add live labels; the 4-day val set is too small to say
|
||
whether this is real.
|
||
- Golden differences of 1–2 headlines are noise; 67 → 72 with the two target categories going 4/11 → 11/11
|
||
is not.
|
||
|
||
Promotion criteria, round 3 non-final: golden 71/78, distractors 8/8, Sep 18–22 71.5%, latency 18.8 ms.
|
||
**All pass.** Shipped `model.onnx` is from `torch_final` (17,245 rows, 4 epochs). Round 2 weights are in
|
||
the session scratchpad (`sweep2/small_e4_bs64_12270rows`, `sweep2/torch_final_12270rows`).
|
||
|
||
### Next, revised
|
||
|
||
1. **Shadow mode in ForexBot2** with `threads=4`, DeepSeek still the answer. Compare per headline for a
|
||
few weeks. This is now the step that produces new information; more history labels will not.
|
||
2. **Live-feed labels.** The bot produces about 300 a day. Retrain weekly; consider up-weighting newsdata
|
||
rows or a second fine-tuning pass on them if the gold-key live number keeps slipping.
|
||
3. The precise crawl is still running in the background (free). Label its remainder only if a specific gap
|
||
shows up in shadow mode.
|
||
4. Loosen the export `--tolerance` or make it flips-only; it has printed FAILED on every export today with
|
||
0–11 flips of 1,000.
|