News-ML-Decoder/eval/RESULTS_2026-09-27.md

Two-head transformer trained on history + archive, 2026-09-27

Outcome: the transformer passes all four promotion criteria and is exported to models/xau_transformer/. The TF-IDF bundle models/xauusd_sentiment.joblib is untouched, and nothing in ForexBot2 imports either model, so nothing changed live.

DeepSeek spend today: 560 calls in four runs (20 pilot, 200 full run, 140 precise top-up, 200 growth set; 15 batches failed and were skipped), labelling 13,625 crawled headlines. Every eval ran with --no-network.

The shipped bundle is from round 3 (see the last section): 17,245 training rows, golden 72/78, with the two weak categories fixed. Round 1 (8,870 rows, 65/78) and round 2 (12,270 rows, 67/78) are kept below because round 1's val numbers are the cleanest comparison against TF-IDF.

Data

Labelling the crawl

The precise crawl (intitle:<term> site:<publisher>) was resumed and ran in the background all session. Two DeepSeek runs labelled its pool, plus a 30% share of the broad pool for the not-relevant side:

| Pool | Labelled | Gold-relevant (|xau| ≥ 0.25) | Bull / bear among relevant | |---|---|---|---| | precise (intitle:) | 2,979 | 1,390 (47%) | 870 / 520 | | broad (keyword anywhere) | 3,271 | 320 (10%) | 253 / 67 | | history total | 6,250 | 1,710 | |

• The 20-call pilot on the precise pool alone gave 52% relevant against 7% for the broad pilot on 2026-09-23. Spot checks looked right ("Dollar slumps as risk appetite rebounds" +0.50, "This is what a $52 million Hamptons beach house looks like" 0.00).
• Relevant share by query term: "rate cut" 87%, "safe haven" 88%, payrolls 80%, "rate hike" 72%, yields 65%, Fed 61%, gold 49%, dollar 44%, inflation 38%, Powell 31%.
• Hindsight check: mean xau per year is +0.02 to +0.06 for 2019–2024, +0.12 for 2025 and 0.00 for 2026. 2025 was the big gold rally, so there may be some hindsight in DeepSeek's 2025 labels, but the effect is small and the direction head is trained on sign only.
• About 1,900 precise headlines are still unlabelled because the crawl kept adding them; it needs roughly 80 more calls.

Training set (transformer/train.py)

	Rows
history labels	6,250
newsdata Sep 8–22 + backfill	2,637
unique after dedupe, golden removed	8,870

Split	n	relevant	irrelevant	grey	bull	bear
train	7,460	2,332	4,492	636	1,290	1,042
dev (random 5%, picks epoch + threshold)	392	119	245	28	72	47
val (newsdata ≥ 2026-09-18, never used to choose)	1,018	378	501	139	171	207

Sweep on the enlarged set (microsoft/deberta-v3-small, lr 5e-5 unless noted)

Run	Dev 3-class	Best epoch / thr	Val 3-class	Val macro-F1	Val rel F1	Val dir acc
4 ep, bs 32	0.824	4 / 0.25	0.708	0.704	0.830	0.907
4 ep, bs 64 (chosen)	0.824	3 / 0.60	0.720	0.715	0.841	0.942
6 ep, bs 32	0.814	4 / 0.20	0.713	0.708	0.842	0.897
8 ep, bs 64	0.821	8 / 0.40	0.709	0.704	0.836	0.905
6 ep, bs 32, lr 3e-5	0.821	6 / 0.20	0.714	0.711	0.834	0.926

The two 4-epoch runs tie on dev; bs 64 has the higher dev relevance F1 (0.869 vs 0.865) and is the script default, so it was chosen. All five runs land within 1.2 points on val: the data did the work, not the hyperparameters.

For reference, the same sweep on the 3,620-row set before labelling (earlier today) gave dev 0.72–0.78 and val 0.62–0.68, and the chosen model then scored 49/78 golden. deberta-v3-base was also tried on the small set: half a dev point better, but 38.6 ms per headline on 1 thread, over the 20 ms limit.

Same unseen rows: Sep 18–22 (DeepSeek label at ±0.05, non-final model)

Model	All (n=1,022)	Gold-key (313)	Macro (709)	Rel P / R / F1	Dir acc on relevant
TF-IDF all 09-22 (promoted)	62.8%	71.9%	58.8%
Transformer on 3,620 rows	65.7% (F1 0.656)	75.1%	61.5%	0.685 / 0.895 / 0.776	89.8%
Transformer on 8,870 rows	71.9% (F1 0.714)	83.7%	66.7%	0.802 / 0.885 / 0.841	94.2%

The DeepSeek self-agreement ceiling is about 90%, so 72% is 18 points short of "as good as DeepSeek".

Golden set (78 reviewer-labelled headlines, argmax label)

Model	3-class	McNemar vs keyword	Spearman vs DeepSeek
TF-IDF all 09-22	47/78 (60.3%)	31 vs 8, p = 0.0003	+0.423
Transformer on 3,620 rows	49/78 (62.8%)	30 vs 5	+0.556
Transformer on 8,870 rows, non-final	62/78 (79.5%)	40 vs 2	+0.676
Transformer, --final (shipped)	65/78 (83.3%)	42 vs 1	+0.775
DeepSeek	78/78

The final model also trained on dev and val, but never on the golden set, so its golden score is a fair number. It is 3 headlines above the non-final model, which is within noise for 78 rows.

Hits by category:

Category	TF-IDF	3,620 rows	8,870 rows	final
distractors	5/8	7/8	8/8	8/8
safe_haven	7/8	8/8	7/8	8/8
dovish_fed	4/8	5/8	7/8	8/8
real_yields	3/6	4/6	5/6	6/6
negation_traps	1/8	3/8	6/8	7/8
gold_direct	7/8	7/8	7/8	7/8
hawkish_fed	8/8	7/8	7/8	7/8
weak_usd	4/7	5/7	6/7	6/7
inverted_metrics	3/6	2/6	6/6	4/6
strong_usd	5/7	1/7	3/7	3/7
risk_on	0/4	0/4	0/4	1/4

Remaining weak spots of the shipped model:

• Risk-on is 1/4. "Stocks roar to record highs as trade deal signed" and "Global equities rally on strong earnings and easing tensions" are bullish at 0.7–0.8. The model reads "rally" as gold rallying.
• Strong US data is 3/7. "Consumer confidence surges to post-pandemic high" → bullish 0.83, "ISM services index jumps" → bullish 0.52. Good-news-for-the-economy still doesn't map to bad-for-gold reliably. The precise crawl has no query for growth data (GDP, ISM, retail sales, confidence); adding one is the obvious next crawl.
• "FOMC minutes reveal growing support for policy easing" is still bearish at 0.63.

Export (transformer/export_onnx.py, from torch_final)

• models/xau_transformer/model.onnx, 271 MB, int8 embedding table, fp32 compute. tokenizer.json, meta.json (final: true, threshold 0.6, export_checks inside).
• Parity over 500 val headlines: tokenizer 500/500, mean |diff| 0.003–0.004, 0 decision flips, max |diff| 0.075 on p_relevant and 0.059 on p_bullish. The script prints PARITY FAILED because the max is over its 0.05 tolerance. With zero flips the tolerance is the problem, not the export; it should be loosened to about 0.15, or the check should be flips only.
• CPU latency: 18.9 ms on 1 thread, 9.6 ms on 16 threads, about 8.5 ms in the evals (4 threads).

Promotion criteria (fixed on 2026-09-22)

Criterion	Target	Non-final	Final
Golden set	≥ 55/78	62/78	65/78	pass
Distractors	≥ 6/8	8/8	8/8	pass
Sep 18–22 three-class agreement	≥ 62.8%	71.9%	(in-sample)	pass
CPU latency, 1 thread	≤ 20 ms	19.0 ms	18.9 ms	pass

All four pass. train.py --final was run (8,870 rows, 3 epochs) and the shipped ONNX comes from torch_final. The non-final weights (the ones the val numbers above belong to) are in models/xau_transformer/torch/.

What this does and does not mean:

• It is a better DeepSeek imitation than TF-IDF: +18 golden headlines, +9 points on unseen days, +35 points of Spearman.
• It still learns DeepSeek's opinion, not price moves, and DeepSeek itself only agrees with itself 90% of the time.
• The "about 4,000 labels over 3+ weeks" deployment rule is met on count (8,870) but the live-feed part is still one fortnight; the history labels are a different distribution (Reuters/CNBC/MarketWatch titles, not the bot's feeds). Shadow mode first.

Next

1. Crawl growth data: add intitle: queries for GDP, "retail sales", ISM, "consumer confidence", "jobless claims" and stocks OR equities (for risk-on), then label them (about 100–200 calls).
2. Finish labelling the precise pool: about 1,900 headlines left, roughly 80 calls, plus whatever the crawl adds. The crawl is still running; it has about 3,300 windows to go.
3. Retrain weekly as the live archive grows; the whole train → export → eval loop takes under 5 minutes.
4. Loosen the export tolerance or make it flips-only.
5. Wire into ForexBot2 in shadow mode with threads=4, DeepSeek still the answer. Decide the promotion rule for that separately.

Round 2: labelling the rest of the precise pool (same day)

The crawl kept running, so after round 1 there were about 3,500 unlabelled precise headlines. A 140-call run with --broad-share 0 labelled 3,400 of them (4 batches failed). History labels: 9,650, of which 6,185 precise (47% relevant, 1,955 bull / 941 bear) and 3,465 broad (12% relevant). About 570 precise headlines were still unlabelled at the end because the crawl was still adding them.

Training set: 12,270 unique headlines. Train 10,690 / dev 562 / val 1,018 (val unchanged).

Default config only (lr 5e-5, 4 epochs, batch 64), no sweep: round 1 showed the hyperparameters don't matter at this size. Dev picked epoch 4, threshold 0.2.

	Round 1 (8,870 rows)	Round 2 (12,270 rows)
Val 3-class (unseen Sep 18–22)	72.0%	71.7%
Val macro-F1	0.715	0.715
Val relevance P / R / F1	0.803 / 0.884 / 0.841	0.759 / 0.926 / 0.834
Val direction acc on relevant	94.2%	93.4%
Archive eval, Sep 18–22 all / gold / macro	71.9 / 83.7 / 66.7%	71.8 / 83.1 / 66.9%
Golden, non-final	62/78	64/78
Golden, final (shipped)	65/78	67/78
Spearman vs DeepSeek, final	+0.775	+0.765
Export flips / max diff, final	0 / 0.075	4 (0.4%) / 0.104
CPU latency, 1 thread	18.9 ms	18.9 ms

• The live-feed metric has saturated at about 72% with this kind of data. The extra 3,400 history labels changed nothing there. More gains on the live feed will have to come from live-feed labels.
• The golden set still moved, 65 → 67 final. Hawkish Fed went 7 → 8, inverted metrics 4 → 6.
• Per category, final: distractors 8/8, dovish_fed 8/8, hawkish_fed 8/8, safe_haven 8/8, inverted_metrics 6/6, negation_traps 7/8, gold_direct 7/8, weak_usd 6/7, real_yields 5/6, strong_usd 3/7, risk_on 1/4.
• Strong USD (3/7) and risk-on (1/4) did not move, as expected: the crawl has no queries for growth data or equities. That is the next crawl, not the next labelling run.

Promotion criteria, round 2 non-final: golden 64/78, distractors 8/8, Sep 18–22 71.8%, latency 18.9 ms. All pass. train.py --final ran on all 12,270 rows for 4 epochs; the shipped model.onnx is from that torch_final. Round 1's weights are in the session scratchpad (sweep2/small_e4_bs64_8870rows, sweep2/torch_final_8870rows).

Round 3: teaching the two weak spots (same day)

Strong-US-data (3/7) and risk-on (1/4) had not moved through rounds 1 and 2 because no crawl query covered growth data or equities. Fix:

1. New crawler query set growth (crawl_google_news.py --query-set growth --window-days 30): 20 intitle: queries on reuters.com and cnbc.com for GDP, retail sales, ISM, PMI, consumer confidence, jobless claims, consumer spending, manufacturing, "record high", "risk appetite", "Wall Street", "S&P 500", ceasefire, "trade deal", equities. 1,880 requests, about 70 minutes, 6,183 new headlines (the precise crawl was paused meanwhile so Google saw one crawler, then resumed).
2. Labeller changes (label_history.py): a --query-set option restricts labelling to headlines a given crawler set found (the growth queries had also re-tagged thousands of old broad headlines), and is_latin() skips Reuters' Chinese and Japanese wire items that intitle:GDP and intitle:PMI match.
3. 200 calls, growth set only, 4,975 headlines labelled, 1 failed. DeepSeek called 19% relevant, 734 bullish vs 216 bearish: "Wall Street falls" and "ceasefire talks collapse" are risk-off, and "record high" also catches gold's own records. The bearish side is the pattern the model lacked: "Yields rise on improving risk appetite" −0.50, "US dollar rises after jobless claims data" −0.50.

Query	Labelled	Relevant	Bear / bull
"S&P 500"	692	16%	36 / 74
GDP	690	12%	17 / 67
"Wall Street"	651	18%	12 / 105
"record high"	624	14%	23 / 67
manufacturing	457	6%	6 / 20
ceasefire	379	58%	11 / 208
"trade deal"	307	4%	4 / 9
PMI	304	6%	3 / 15
"retail sales"	293	18%	23 / 29
equities	205	28%	14 / 44
"jobless claims"	126	57%	35 / 37
"risk appetite"	72	72%	23 / 29
consumer spending / confidence, ISM	175	15–56%	9 / 30

Training set: 17,245 unique headlines. Train 15,416 / dev 811 / val 1,018 (val unchanged). Default config, dev picked epoch 4, threshold 0.2.

	Round 2 (12,270)	Round 3 (17,245)
Val 3-class (unseen Sep 18–22)	71.7%	71.2%
Archive eval, Sep 18–22 all / gold / macro	71.8 / 83.1 / 66.9%	71.5 / 78.6 / 68.4%
Val relevance P / R / F1	0.759 / 0.926 / 0.834	0.724 / 0.929 / 0.813
Val direction acc on relevant	93.4%	89.4%
Golden, non-final	64/78	71/78
Golden, final (shipped)	67/78	72/78
Spearman vs DeepSeek, final	+0.765	+0.860
McNemar vs DeepSeek, final	DS-only-right 11, p = 0.001	DS-only-right 6, p = 0.031
Export flips / max diff, final	4 / 0.104	2 / 0.077
CPU latency, 1 thread	18.9 ms	18.8 ms

Golden by category, final: strong_usd 3/7 → 7/7, risk_on 1/4 → 4/4, inverted_metrics 6/6, distractors 8/8, dovish_fed 8/8, safe_haven 8/8, negation_traps 7/8, hawkish_fed 7/8, weak_usd 6/7, real_yields 5/6, gold_direct 6/8.

The six remaining misses:

• "Powell says restrictive policy needed for longer as inflation proves sticky" → bullish 0.83 (hawkish read as bullish; the only hawkish miss).
• "Rising real interest rates dim appeal of non-yielding assets" → bullish 0.95 (no "gold" in the text; "non-yielding assets" is unseen vocabulary).
• "Bullion demand weakens as Indian jewellery buying stalls" → neutral (relevance 0.0; physical demand is absent from the training feeds).
• "Miners struggle to meet gold demand as supply deficit widens" → bearish 0.32 ("miners struggle" reads as bad news).
• "Greenback weakens as US retail sales unexpectedly contract" → neutral (relevance just under 0.2).
• "No safe-haven bid for gold despite equity wobble" → bullish 0.15, low confidence.

Trade-offs to note honestly:

• The live-feed numbers slipped a little (gold-key 83.1 → 78.6%, direction on relevant 93.4 → 89.4%) while macro rose (66.9 → 68.4%). The all-rows figure is flat at 71.5%. The history labels now outnumber the live-feed labels 9 to 1, and they are a different distribution (wire headlines, no descriptions). If the live feed is what matters, weight it or add live labels; the 4-day val set is too small to say whether this is real.
• Golden differences of 1–2 headlines are noise; 67 → 72 with the two target categories going 4/11 → 11/11 is not.

Promotion criteria, round 3 non-final: golden 71/78, distractors 8/8, Sep 18–22 71.5%, latency 18.8 ms. All pass. Shipped model.onnx is from torch_final (17,245 rows, 4 epochs). Round 2 weights are in the session scratchpad (sweep2/small_e4_bs64_12270rows, sweep2/torch_final_12270rows).

Next, revised

1. Shadow mode in ForexBot2 with threads=4, DeepSeek still the answer. Compare per headline for a few weeks. This is now the step that produces new information; more history labels will not.
2. Live-feed labels. The bot produces about 300 a day. Retrain weekly; consider up-weighting newsdata rows or a second fine-tuning pass on them if the gold-key live number keeps slipping.
3. The precise crawl is still running in the background (free). Label its remainder only if a specific gap shows up in shadow mode.
4. Loosen the export --tolerance or make it flips-only; it has printed FAILED on every export today with 0–11 flips of 1,000.