# Two-head transformer trained on history + archive, 2026-09-27

**Outcome: the transformer passes all four promotion criteria and is exported to `models/xau_transformer/`.**
The TF-IDF bundle `models/xauusd_sentiment.joblib` is untouched, and nothing in ForexBot2 imports either
model, so nothing changed live.

DeepSeek spend today: **560 calls** in four runs (20 pilot, 200 full run, 140 precise top-up, 200 growth
set; 15 batches failed and were skipped), labelling 13,625 crawled headlines. Every eval ran with
`--no-network`.

**The shipped bundle is from round 3** (see the last section): 17,245 training rows, golden **72/78**,
with the two weak categories fixed. Round 1 (8,870 rows, 65/78) and round 2 (12,270 rows, 67/78) are kept
below because round 1's val numbers are the cleanest comparison against TF-IDF.

## Data

### Labelling the crawl

The precise crawl (`intitle:<term> site:<publisher>`) was resumed and ran in the background all session.
Two DeepSeek runs labelled its pool, plus a 30% share of the broad pool for the not-relevant side:

| Pool | Labelled | Gold-relevant (|xau| ≥ 0.25) | Bull / bear among relevant |
|---|---|---|---|
| precise (`intitle:`) | 2,979 | 1,390 (47%) | 870 / 520 |
| broad (keyword anywhere) | 3,271 | 320 (10%) | 253 / 67 |
| **history total** | **6,250** | 1,710 | |

- The 20-call pilot on the precise pool alone gave 52% relevant against 7% for the broad pilot on
  2026-09-23. Spot checks looked right ("Dollar slumps as risk appetite rebounds" +0.50, "This is what a
  $52 million Hamptons beach house looks like" 0.00).
- Relevant share by query term: "rate cut" 87%, "safe haven" 88%, payrolls 80%, "rate hike" 72%, yields
  65%, Fed 61%, gold 49%, dollar 44%, inflation 38%, Powell 31%.
- **Hindsight check:** mean xau per year is +0.02 to +0.06 for 2019–2024, +0.12 for 2025 and 0.00 for 2026.
  2025 was the big gold rally, so there may be some hindsight in DeepSeek's 2025 labels, but the effect is
  small and the direction head is trained on sign only.
- About 1,900 precise headlines are still unlabelled because the crawl kept adding them; it needs roughly
  80 more calls.

### Training set (`transformer/train.py`)

| | Rows |
|---|---|
| history labels | 6,250 |
| newsdata Sep 8–22 + backfill | 2,637 |
| **unique after dedupe, golden removed** | **8,870** |

| Split | n | relevant | irrelevant | grey | bull | bear |
|---|---|---|---|---|---|---|
| train | 7,460 | 2,332 | 4,492 | 636 | 1,290 | 1,042 |
| dev (random 5%, picks epoch + threshold) | 392 | 119 | 245 | 28 | 72 | 47 |
| val (newsdata ≥ 2026-09-18, never used to choose) | 1,018 | 378 | 501 | 139 | 171 | 207 |

## Sweep on the enlarged set (`microsoft/deberta-v3-small`, lr 5e-5 unless noted)

| Run | Dev 3-class | Best epoch / thr | Val 3-class | Val macro-F1 | Val rel F1 | Val dir acc |
|---|---|---|---|---|---|---|
| 4 ep, bs 32 | 0.824 | 4 / 0.25 | 0.708 | 0.704 | 0.830 | 0.907 |
| **4 ep, bs 64 (chosen)** | **0.824** | 3 / 0.60 | **0.720** | **0.715** | 0.841 | **0.942** |
| 6 ep, bs 32 | 0.814 | 4 / 0.20 | 0.713 | 0.708 | 0.842 | 0.897 |
| 8 ep, bs 64 | 0.821 | 8 / 0.40 | 0.709 | 0.704 | 0.836 | 0.905 |
| 6 ep, bs 32, lr 3e-5 | 0.821 | 6 / 0.20 | 0.714 | 0.711 | 0.834 | 0.926 |

The two 4-epoch runs tie on dev; bs 64 has the higher dev relevance F1 (0.869 vs 0.865) and is the script
default, so it was chosen. All five runs land within 1.2 points on val: the data did the work, not the
hyperparameters.

For reference, the same sweep on the 3,620-row set before labelling (earlier today) gave dev 0.72–0.78 and
val 0.62–0.68, and the chosen model then scored 49/78 golden. deberta-v3-base was also tried on the small
set: half a dev point better, but 38.6 ms per headline on 1 thread, over the 20 ms limit.

## Same unseen rows: Sep 18–22 (DeepSeek label at ±0.05, non-final model)

| Model | All (n=1,022) | Gold-key (313) | Macro (709) | Rel P / R / F1 | Dir acc on relevant |
|---|---|---|---|---|---|
| TF-IDF `all` 09-22 (promoted) | 62.8% | 71.9% | 58.8% | | |
| Transformer on 3,620 rows | 65.7% (F1 0.656) | 75.1% | 61.5% | 0.685 / 0.895 / 0.776 | 89.8% |
| **Transformer on 8,870 rows** | **71.9% (F1 0.714)** | **83.7%** | **66.7%** | 0.802 / 0.885 / 0.841 | **94.2%** |

The DeepSeek self-agreement ceiling is about 90%, so 72% is 18 points short of "as good as DeepSeek".

## Golden set (78 reviewer-labelled headlines, argmax label)

| Model | 3-class | McNemar vs keyword | Spearman vs DeepSeek |
|---|---|---|---|
| TF-IDF `all` 09-22 | 47/78 (60.3%) | 31 vs 8, p = 0.0003 | +0.423 |
| Transformer on 3,620 rows | 49/78 (62.8%) | 30 vs 5 | +0.556 |
| Transformer on 8,870 rows, non-final | 62/78 (79.5%) | 40 vs 2 | +0.676 |
| **Transformer, `--final` (shipped)** | **65/78 (83.3%)** | 42 vs 1 | +0.775 |
| DeepSeek | 78/78 | | |

The final model also trained on dev and val, but never on the golden set, so its golden score is a fair
number. It is 3 headlines above the non-final model, which is within noise for 78 rows.

Hits by category:

| Category | TF-IDF | 3,620 rows | 8,870 rows | final |
|---|---|---|---|---|
| distractors | 5/8 | 7/8 | 8/8 | 8/8 |
| safe_haven | 7/8 | 8/8 | 7/8 | 8/8 |
| dovish_fed | 4/8 | 5/8 | 7/8 | 8/8 |
| real_yields | 3/6 | 4/6 | 5/6 | 6/6 |
| negation_traps | 1/8 | 3/8 | 6/8 | 7/8 |
| gold_direct | 7/8 | 7/8 | 7/8 | 7/8 |
| hawkish_fed | 8/8 | 7/8 | 7/8 | 7/8 |
| weak_usd | 4/7 | 5/7 | 6/7 | 6/7 |
| inverted_metrics | 3/6 | 2/6 | 6/6 | 4/6 |
| strong_usd | 5/7 | 1/7 | 3/7 | 3/7 |
| risk_on | 0/4 | 0/4 | 0/4 | 1/4 |

Remaining weak spots of the shipped model:
- **Risk-on is 1/4.** "Stocks roar to record highs as trade deal signed" and "Global equities rally on strong
  earnings and easing tensions" are bullish at 0.7–0.8. The model reads "rally" as gold rallying.
- **Strong US data is 3/7.** "Consumer confidence surges to post-pandemic high" → bullish 0.83, "ISM
  services index jumps" → bullish 0.52. Good-news-for-the-economy still doesn't map to bad-for-gold
  reliably. The precise crawl has no query for growth data (GDP, ISM, retail sales, confidence); adding one
  is the obvious next crawl.
- "FOMC minutes reveal growing support for policy easing" is still bearish at 0.63.

## Export (`transformer/export_onnx.py`, from `torch_final`)

- `models/xau_transformer/model.onnx`, 271 MB, int8 embedding table, fp32 compute. `tokenizer.json`,
  `meta.json` (`final: true`, threshold 0.6, `export_checks` inside).
- Parity over 500 val headlines: tokenizer 500/500, mean |diff| 0.003–0.004, **0 decision flips**, max
  |diff| 0.075 on `p_relevant` and 0.059 on `p_bullish`. The script prints `PARITY FAILED` because the max
  is over its 0.05 tolerance. With zero flips the tolerance is the problem, not the export; it should be
  loosened to about 0.15, or the check should be flips only.
- CPU latency: 18.9 ms on 1 thread, 9.6 ms on 16 threads, about 8.5 ms in the evals (4 threads).

## Promotion criteria (fixed on 2026-09-22)

| Criterion | Target | Non-final | Final | |
|---|---|---|---|---|
| Golden set | ≥ 55/78 | 62/78 | 65/78 | pass |
| Distractors | ≥ 6/8 | 8/8 | 8/8 | pass |
| Sep 18–22 three-class agreement | ≥ 62.8% | 71.9% | (in-sample) | pass |
| CPU latency, 1 thread | ≤ 20 ms | 19.0 ms | 18.9 ms | pass |

**All four pass.** `train.py --final` was run (8,870 rows, 3 epochs) and the shipped ONNX comes from
`torch_final`. The non-final weights (the ones the val numbers above belong to) are in
`models/xau_transformer/torch/`.

What this does and does not mean:
- It is a better DeepSeek imitation than TF-IDF: +18 golden headlines, +9 points on unseen days, +35 points
  of Spearman.
- It still learns DeepSeek's opinion, not price moves, and DeepSeek itself only agrees with itself 90% of
  the time.
- The "about 4,000 labels over 3+ weeks" deployment rule is met on count (8,870) but the live-feed part is
  still one fortnight; the history labels are a different distribution (Reuters/CNBC/MarketWatch titles,
  not the bot's feeds). Shadow mode first.

## Next

1. **Crawl growth data:** add `intitle:` queries for GDP, "retail sales", ISM, "consumer confidence",
   "jobless claims" and `stocks OR equities` (for risk-on), then label them (about 100–200 calls).
2. **Finish labelling the precise pool:** about 1,900 headlines left, roughly 80 calls, plus whatever the
   crawl adds. The crawl is still running; it has about 3,300 windows to go.
3. Retrain weekly as the live archive grows; the whole train → export → eval loop takes under 5 minutes.
4. Loosen the export tolerance or make it flips-only.
5. Wire into ForexBot2 in shadow mode with `threads=4`, DeepSeek still the answer. Decide the
   promotion rule for that separately.

## Round 2: labelling the rest of the precise pool (same day)

The crawl kept running, so after round 1 there were about 3,500 unlabelled precise headlines. A 140-call
run with `--broad-share 0` labelled 3,400 of them (4 batches failed). History labels: 9,650, of which
6,185 precise (47% relevant, 1,955 bull / 941 bear) and 3,465 broad (12% relevant). About 570 precise
headlines were still unlabelled at the end because the crawl was still adding them.

Training set: 12,270 unique headlines. Train 10,690 / dev 562 / val 1,018 (val unchanged).

Default config only (lr 5e-5, 4 epochs, batch 64), no sweep: round 1 showed the hyperparameters don't
matter at this size. Dev picked epoch 4, threshold 0.2.

| | Round 1 (8,870 rows) | Round 2 (12,270 rows) |
|---|---|---|
| Val 3-class (unseen Sep 18–22) | 72.0% | 71.7% |
| Val macro-F1 | 0.715 | 0.715 |
| Val relevance P / R / F1 | 0.803 / 0.884 / 0.841 | 0.759 / 0.926 / 0.834 |
| Val direction acc on relevant | 94.2% | 93.4% |
| Archive eval, Sep 18–22 all / gold / macro | 71.9 / 83.7 / 66.7% | 71.8 / 83.1 / 66.9% |
| Golden, non-final | 62/78 | 64/78 |
| **Golden, final (shipped)** | 65/78 | **67/78** |
| Spearman vs DeepSeek, final | +0.775 | +0.765 |
| Export flips / max diff, final | 0 / 0.075 | 4 (0.4%) / 0.104 |
| CPU latency, 1 thread | 18.9 ms | 18.9 ms |

- **The live-feed metric has saturated** at about 72% with this kind of data. The extra 3,400 history
  labels changed nothing there. More gains on the live feed will have to come from live-feed labels.
- **The golden set still moved,** 65 → 67 final. Hawkish Fed went 7 → 8, inverted metrics 4 → 6.
- Per category, final: distractors 8/8, dovish_fed 8/8, hawkish_fed 8/8, safe_haven 8/8, inverted_metrics
  6/6, negation_traps 7/8, gold_direct 7/8, weak_usd 6/7, real_yields 5/6, strong_usd 3/7, risk_on 1/4.
- **Strong USD (3/7) and risk-on (1/4) did not move,** as expected: the crawl has no queries for growth
  data or equities. That is the next crawl, not the next labelling run.

Promotion criteria, round 2 non-final: golden 64/78, distractors 8/8, Sep 18–22 71.8%, latency 18.9 ms.
**All pass.** `train.py --final` ran on all 12,270 rows for 4 epochs; the shipped `model.onnx` is from that
`torch_final`. Round 1's weights are in the session scratchpad (`sweep2/small_e4_bs64_8870rows`,
`sweep2/torch_final_8870rows`).

## Round 3: teaching the two weak spots (same day)

Strong-US-data (3/7) and risk-on (1/4) had not moved through rounds 1 and 2 because no crawl query covered
growth data or equities. Fix:

1. **New crawler query set `growth`** (`crawl_google_news.py --query-set growth --window-days 30`): 20
   `intitle:` queries on reuters.com and cnbc.com for GDP, retail sales, ISM, PMI, consumer confidence,
   jobless claims, consumer spending, manufacturing, "record high", "risk appetite", "Wall Street",
   "S&P 500", ceasefire, "trade deal", equities. 1,880 requests, about 70 minutes, 6,183 new headlines
   (the precise crawl was paused meanwhile so Google saw one crawler, then resumed).
2. **Labeller changes** (`label_history.py`): a `--query-set` option restricts labelling to headlines a
   given crawler set found (the growth queries had also re-tagged thousands of old broad headlines), and
   `is_latin()` skips Reuters' Chinese and Japanese wire items that `intitle:GDP` and `intitle:PMI` match.
3. **200 calls, growth set only**, 4,975 headlines labelled, 1 failed. DeepSeek called 19% relevant, 734
   bullish vs 216 bearish: "Wall Street falls" and "ceasefire talks collapse" are risk-off, and
   `"record high"` also catches gold's own records. The bearish side is the pattern the model lacked:
   "Yields rise on improving risk appetite" −0.50, "US dollar rises after jobless claims data" −0.50.

| Query | Labelled | Relevant | Bear / bull |
|---|---|---|---|
| "S&P 500" | 692 | 16% | 36 / 74 |
| GDP | 690 | 12% | 17 / 67 |
| "Wall Street" | 651 | 18% | 12 / 105 |
| "record high" | 624 | 14% | 23 / 67 |
| manufacturing | 457 | 6% | 6 / 20 |
| ceasefire | 379 | 58% | 11 / 208 |
| "trade deal" | 307 | 4% | 4 / 9 |
| PMI | 304 | 6% | 3 / 15 |
| "retail sales" | 293 | 18% | 23 / 29 |
| equities | 205 | 28% | 14 / 44 |
| "jobless claims" | 126 | 57% | 35 / 37 |
| "risk appetite" | 72 | 72% | 23 / 29 |
| consumer spending / confidence, ISM | 175 | 15–56% | 9 / 30 |

Training set: 17,245 unique headlines. Train 15,416 / dev 811 / val 1,018 (val unchanged). Default config,
dev picked epoch 4, threshold 0.2.

| | Round 2 (12,270) | Round 3 (17,245) |
|---|---|---|
| Val 3-class (unseen Sep 18–22) | 71.7% | 71.2% |
| Archive eval, Sep 18–22 all / gold / macro | 71.8 / 83.1 / 66.9% | 71.5 / 78.6 / 68.4% |
| Val relevance P / R / F1 | 0.759 / 0.926 / 0.834 | 0.724 / 0.929 / 0.813 |
| Val direction acc on relevant | 93.4% | 89.4% |
| Golden, non-final | 64/78 | **71/78** |
| **Golden, final (shipped)** | 67/78 | **72/78** |
| Spearman vs DeepSeek, final | +0.765 | +0.860 |
| McNemar vs DeepSeek, final | DS-only-right 11, p = 0.001 | DS-only-right 6, p = 0.031 |
| Export flips / max diff, final | 4 / 0.104 | 2 / 0.077 |
| CPU latency, 1 thread | 18.9 ms | 18.8 ms |

Golden by category, final: **strong_usd 3/7 → 7/7, risk_on 1/4 → 4/4**, inverted_metrics 6/6,
distractors 8/8, dovish_fed 8/8, safe_haven 8/8, negation_traps 7/8, hawkish_fed 7/8, weak_usd 6/7,
real_yields 5/6, gold_direct 6/8.

The six remaining misses:
- "Powell says restrictive policy needed for longer as inflation proves sticky" → bullish 0.83 (hawkish
  read as bullish; the only hawkish miss).
- "Rising real interest rates dim appeal of non-yielding assets" → bullish 0.95 (no "gold" in the text;
  "non-yielding assets" is unseen vocabulary).
- "Bullion demand weakens as Indian jewellery buying stalls" → neutral (relevance 0.0; physical demand
  is absent from the training feeds).
- "Miners struggle to meet gold demand as supply deficit widens" → bearish 0.32 ("miners struggle" reads
  as bad news).
- "Greenback weakens as US retail sales unexpectedly contract" → neutral (relevance just under 0.2).
- "No safe-haven bid for gold despite equity wobble" → bullish 0.15, low confidence.

Trade-offs to note honestly:
- **The live-feed numbers slipped a little** (gold-key 83.1 → 78.6%, direction on relevant 93.4 → 89.4%)
  while macro rose (66.9 → 68.4%). The all-rows figure is flat at 71.5%. The history labels now outnumber
  the live-feed labels 9 to 1, and they are a different distribution (wire headlines, no descriptions).
  If the live feed is what matters, weight it or add live labels; the 4-day val set is too small to say
  whether this is real.
- Golden differences of 1–2 headlines are noise; 67 → 72 with the two target categories going 4/11 → 11/11
  is not.

Promotion criteria, round 3 non-final: golden 71/78, distractors 8/8, Sep 18–22 71.5%, latency 18.8 ms.
**All pass.** Shipped `model.onnx` is from `torch_final` (17,245 rows, 4 epochs). Round 2 weights are in
the session scratchpad (`sweep2/small_e4_bs64_12270rows`, `sweep2/torch_final_12270rows`).

### Next, revised

1. **Shadow mode in ForexBot2** with `threads=4`, DeepSeek still the answer. Compare per headline for a
   few weeks. This is now the step that produces new information; more history labels will not.
2. **Live-feed labels.** The bot produces about 300 a day. Retrain weekly; consider up-weighting newsdata
   rows or a second fine-tuning pass on them if the gold-key live number keeps slipping.
3. The precise crawl is still running in the background (free). Label its remainder only if a specific gap
   shows up in shadow mode.
4. Loosen the export `--tolerance` or make it flips-only; it has printed FAILED on every export today with
   0–11 flips of 1,000.