News-ML-Decoder/eval/RESULTS_2026-09-22.md

Retrain on the Sep 8–22 archive, 2026-09-22

Data: newsdata.db copied from the 2026-09-22 archive, with 2,665 headlines.

• 2,661 rows are score_source='llm', scored by deepseek-v4-flash with prompt d32e04c6.
• 4 rows are keyword-scored and not used.
• Every label is DeepSeek's, not a human's or a trader's. The 78-headline golden set is still the only independent truth.

The ForexBot2 archive patch is live: xau_score is filled for every LLM row from 2026-09-18 on (1,064 rows). Where a row also has a JSON XAU key (319 rows), the two values match on all of them.

DeepSeek spend for this run: 0 calls (--no-network, everything came from the cache).

What changed in the code

• news_sentiment.py: the label is taken from xau_score, then the JSON XAU key, then --extra-labels (backfill JSONL keyed by hash).
  • Rows from test feeds are dropped.
  • --scope all|gold was added.
  • The metrics now record label_sources, scope and published_range.
  • The model format is unchanged.
• eval_common.load_archive_rows: uses the same label order and sets xau_src and gold (whether the row has a JSON XAU key) on each row.
• eval_golden.py: new --model option.
• eval_archive.py: new --model, --extra-labels and --train-scope options.
  • Part A now also scores rows published after the bundle's trained_at. For the 09-17 bundle those rows are out of sample.
  • Part B pools its results by gold rows, macro rows and days on or after 09-18.
  • In Part B the test day is always every labelled row, so both train scopes are graded on the same rows.

Candidates

Model	Train rows	Labels (bear / neu / bull)	Stratified acc / F1	Chronological acc / F1
shipped 09-17 (gold only, Sep 8–17)	606	250 / 135 / 221	73.0% / 0.709	69.7% / 0.671
--scope gold (Sep 8–22)	945	359 / 202 / 384	80.4% / 0.793	74.6% / 0.734
--scope all + backfill (Sep 8–22)	2,620	909 / 893 / 818	69.3% / 0.689	66.6% / 0.659

You can't compare the built-in holdouts across rows of this table:

• The gold and 09-17 holdouts contain only headlines that mention gold, which are the easy ones.
• The all holdout is 64% macro headlines.

The next two sections grade every model on the same rows.

Same unseen rows: Sep 18–22 (1,022–1,066 headlines, DeepSeek label at ±0.05)

Model	All	Gold-key	Macro
shipped 09-17 bundle (never saw these days)	45.9% (F1 0.455)	68.2%	36.2%
gold recipe, retrained daily on days < d	48.3% (F1 0.478)	73.8%	37.1%
all recipe, retrained daily on days < d	62.8% (F1 0.629)	71.9%	58.8%

Pooled over every daily fold from Sep 9–22 (2,566 / 2,530 rows):

Recipe	3-class	Macro-F1	Gold-key	Macro	Keyword fallback
all	58.2%	0.578	65.1%	54.3%	44.7%
gold	48.3%	0.479	67.8%	37.3%	44.7%

Golden set (78 reviewer-labelled headlines, argmax label)

Model	3-class	McNemar vs keyword	Spearman vs DeepSeek
shipped 09-17	39/78 (50.0%)	28 vs 13, p = 0.028	+0.305
gold	40/78 (51.3%)	29 vs 13, p = 0.020	+0.310
all	47/78 (60.3%)	31 vs 8, p = 0.0003	+0.423
DeepSeek	78/78

Hits by category, shipped → all:

Category	Shipped	all
hawkish_fed	3	8
strong_usd	2	5
distractors	1	5
negation_traps	0	1
weak_usd	4	4
real_yields	3	3
inverted_metrics	3	3
dovish_fed	6	4
gold_direct	8	7
safe_haven	8	7
risk_on	1	0

The all model's weak spots:

• Easing and falling yields read as bearish. "Fed cuts rates 25bp" → bearish 0.78, and "Treasury yields sink" → bearish 0.78. Nearly all of the September macro data comes from a hiking regime, so the model links "Fed" and "yields" with bearish and has almost no examples of easing.
• Risk-on headlines score 0/4. "Stocks roar to record highs" → bullish.
• Negation still fails (1/8).

Verdict

Promoted --scope all. It is now models/xauusd_sentiment.joblib. The 09-17 bundle is kept in models/archive/xauusd_sentiment_20260917.*, and both candidates are in models/candidates/.

• Against the shipped model: on the same unseen days it wins by 17 points overall and 22 points on macro headlines.
• Against the gold-only recipe: it gives up 1.9 points on gold headlines and gains 21.7 points on macro.
• Golden set: 39 → 47/78.

The promotion rule written before the run said "chronological macro-F1 ≥ shipped". Taken literally, that test fails: 0.659 against 0.671. The two numbers come from different test sets, one gold-only and one mostly macro. On the same unseen rows the retrained model wins, and that is the comparison the rule was meant to capture.

This is still not a DeepSeek replacement:

• It is 31 golden headlines behind DeepSeek.
• It learns only what DeepSeek says, not how price moves.
• It has 2,620 labels over 15 days, which is short of the "about 4,000 labels over 3+ weeks" rule. Keep it in shadow mode with DeepSeek as the answer.

Nothing in ForexBot2 imports the model, so the promotion changes nothing live.

ForexBot2/venv has sklearn 1.9.0 and the bundle was pickled with 1.9.1. That raises an InconsistentVersionWarning, which the eval scripts suppress. The 09-17 bundle had the same mismatch. Align the two venvs before any live use.

Next

1. Keep collecting. Every new day adds about 300 labels from xau_score. Retrain weekly and rerun both evals.
2. Easing and falling yields are the obvious gap. Around 20–40 DeepSeek-labelled dovish, yields-down and risk-on headlines would test whether this gap comes from the regime or from the model.