News-ML-Decoder/eval/RESULTS_2026-09-17.md

ML-NEWS scikit model vs DeepSeek — first evaluation, 2026-09-17

Model under test: models/xauusd_sentiment.joblib (TF-IDF word + char, logistic regression, trained 2026-09-17 on 606 DeepSeek-labelled gold headlines from Sep 8–17). Teacher: DeepSeek deepseek-v4-flash through ForexBot2's live prompt (sha1 d32e04c6). Scripts and raw outputs: this folder (results/*_20260917_22*). DeepSeek spend for the whole exercise: 11 calls.

Verdict

The model is a partial copy of DeepSeek, not a replacement. It has learned what gold-specific headline words mean and nothing else: perfect on direct gold news and safe-haven stories, near random on macro doctrine (hawkish/dovish Fed, dollar, yields, risk-on), zero on negation, and it assigns a direction to irrelevant headlines. DeepSeek is still 78/78 on the independent set today.

Test	Truth	Model	DeepSeek	Keyword fallback
Golden set, 78 reviewer-labelled headlines, 3-class	human	39/78 = 50.0%	78/78 = 100%	24/78 = 30.8%
Golden set, directional rows only (70), right sign	human	38/70 = 54.3%	70/70	16/70
Archive, expanding daily folds, pooled OOF (567 rows), 3-class	DeepSeek	64.4%	(is the label)	44.4%
Archive OOF, sign agreement when both lean	DeepSeek	74.8%		86.8% (on 152 rows only)
Live feed, last 24 h, 66 headlines not in training, 3-class	DeepSeek	57.6% (66.7% headline-only)		39.4%
Live feed, gold-mentioning and not in training (37)	DeepSeek	70.3% (75.7% headline-only)		35.1%
DeepSeek re-scored 100 archived headlines (teacher self-agreement)	DeepSeek (earlier)		90.0% 3-class, 98.6% sign

Paired McNemar on the golden set: model vs DeepSeek 0 vs 39 discordant (p < 0.0001); model vs keyword 28 vs 13 (p = 0.028). The model beats the keyword fallback and loses to DeepSeek by a wide margin.

Golden set by category (hits / n): model | DeepSeek | keyword

distractors         1/8  |  8/8  |  8/8     irrelevant news gets a direction
dovish_fed          6/8  |  8/8  |  7/8
gold_direct         8/8  |  8/8  |  4/8     learned
hawkish_fed         3/8  |  8/8  |  2/8     "Hot CPI fuels Fed tightening" -> bullish
inverted_metrics    3/6  |  6/6  |  0/6
negation_traps      0/8  |  8/8  |  2/8     "signals no rate hike" -> bearish 0.85 conf
real_yields         3/6  |  6/6  |  1/6
risk_on             1/4  |  4/4  |  0/4
safe_haven          8/8  |  8/8  |  0/8     learned
strong_usd          2/7  |  7/7  |  0/7     "Greenback rallies" -> bullish
weak_usd            4/7  |  7/7  |  0/7

The model's confidence does not flag its errors on this set: the negation-trap miss above carries 0.85 confidence and the "consumer confidence surges" miss 0.82.

Archive time-split (retrain on days < d, score day d)

day        train test | full acc  F1   | head acc  F1   | keyword
09-10        49   87  |  62.1%  0.626  |  51.7%  0.521  |  47.1%
09-11       136   91  |  52.7%  0.524  |  53.8%  0.525  |  39.6%
09-14       227  130  |  76.9%  0.714  |  72.3%  0.656  |  58.5%
09-15       357   84  |  69.0%  0.656  |  61.9%  0.605  |  44.0%
09-16       441  115  |  55.7%  0.545  |  54.8%  0.531  |  33.0%
09-17       556   60  |  68.3%  0.643  |  75.0%  0.703  |  40.0%
pooled      —    567  |  64.4%  0.636  |  61.4%  0.607  |  44.4%

"full" = headline + description (the shipped recipe), "head" = headline only. Agreement does not rise with more training days yet; the day-to-day swing (53–77%) is larger than any trend. In-sample the shipped bundle scores 97.4%, so the gap to 64% is generalisation, not capacity. The description helps a little on the archive folds and hurts on the live sample (60.5% vs 68.4%); both differences are inside the noise of these sample sizes.

Ceiling: DeepSeek disagrees with itself

Re-sending 100 archived headlines today (same prompt, same model, temperature 0, different batch neighbours): 90% same class, 93% directional, 98.6% same sign, but only 42% the identical number, mean |difference| 0.09. A student cannot exceed about 90% class agreement on this label. Where DeepSeek changed its class (9 rows) the OOF model matched one of its two answers on 7.

Confidence routing (pooled OOF, shipped recipe)

conf >=  kept share  kept agreement  hybrid agreement (rest to DeepSeek)
 0.60      44%          80.9%            91.5%
 0.70      28%          87.4%            96.5%
 0.80      17%          95.8%            99.3%
 0.90       6.5%       100.0%           100.0%

The WORKFLOW doc's 0.70 / 0.20 starting point keeps 28% of headlines local at 87% agreement with DeepSeek. That saves roughly a quarter of the calls, which are 56 ms and a fraction of a cent per headline in batches of 25 anyway. The golden set shows the caveat: high confidence on a macro or negated headline is not a reliable signal there.

Structural findings

1. Training covers gold-mentioning headlines only. The archive stores DeepSeek's XAU score only when the headline matches a gold keyword; for USD-only headlines only the overall score is kept and DeepSeek's xau is discarded (_apply_llm_sentiment). So the model never saw a Fed, dollar or yields headline that did not also say "gold", which is exactly where it fails. Live, 29 of the 76 headlines were USD-only; the model was at 60% on them versus 72–77% on gold-mentioning ones. Archiving DeepSeek's xau for every headline would fix the training set at no extra cost.
2. Nine days of data spanning one regime (a Fed hiking cycle in September 2026). The 2026-09-08 rule stands: do not deploy before about 4,000 labelled headlines over three or more weeks.
3. Negation and doctrine are not learnable from 606 bag-of-words examples; the regex domain markers catch "dollar ... strengthen" but not "greenback rallies" or "dollar index hits high".
4. Latency: model 1 ms per headline in-process; DeepSeek 56 ms per headline (1.4 s per batch of 25). Latency is not the constraint.

Addendum: backfilling the missing gold labels (same day)

backfill_labels.py sent the 961 archive headlines without a gold label (930 DeepSeek-scored without a gold key, 31 never scored) through the live DeepSeek path: 38 calls, all labelled, kept in results/backfill_labels.jsonl. Macro labels at ±0.05: 350 bearish, 228 bullish, 383 neutral (40% neutral versus 22% on gold headlines).

Training set	Gold-headline OOF	Macro-headline OOF	Golden set
gold rows only (616, shipped data)	64.4%	38.4%	39/78
gold + macro rows (1,577)	61.6%	51.8%	42/78

Adding the macro headlines lifts macro agreement by 13 points but costs 3 points on gold headlines, and the golden-set change is a reshuffle, not a fix: distractors 1→6, hawkish 3→5, strong-USD 2→4, weak-USD 4→6, but dovish 6→3, safe-haven 8→4, risk-on 1→0. On nine days of one regime the linear model cannot hold both worlds at once. The missing-label problem was real, but repairing it on the existing corpus does not produce a usable model; it makes the corpus complete so that weeks of further collection can.

Side finding for the live system: on the 930 macro headlines, the gold reading the platform displays for a non-gold headline (the inverted overall score, _gold_direction_label) matches DeepSeek's own gold score only 61% of the time (sign 77%). Display-only, but the macro persona and the summary read that label.

experiment_recipe.py (same day): stripping the publisher suffix, marking negated words and weighting confident labels moved OOF agreement by less than a point and the golden set to 40–42/78; negation traps went from 0/8 to 2/8. Cleaning is not the bottleneck.

After the archive patch (drafted the same day in ForexBot2)

Once the ForexBot2 patch is live, logs/newsdata.db gains a xau_score column holding DeepSeek's gold score for every LLM-scored row, and headlines naming no tracked currency arrive with currencies_json = []. news_sentiment.py still reads its label from score_by_currency_json["XAU"], so it will keep seeing only the gold-keyword rows until its loader is switched to xau_score (fall back to the JSON key for rows older than the column). Skip rows whose source is a test feed (feed.example, good.example): the ForexBot2 test suite used to write fixture headlines into the real archive; they are keyword-scored, so the score_source = 'llm' filter already excludes them.

How to keep measuring

eval_live.py appends every fresh headline to results/live_shadow.jsonl; run it daily and the out-of-sample sample grows for free (cached DeepSeek answers are never re-bought). Re-run eval_golden.py after any retrain; it is the only independent truth.