# ML-NEWS scikit model vs DeepSeek — first evaluation, 2026-09-17 Model under test: `models/xauusd_sentiment.joblib` (TF-IDF word + char, logistic regression, trained 2026-09-17 on 606 DeepSeek-labelled gold headlines from Sep 8–17). Teacher: DeepSeek `deepseek-v4-flash` through ForexBot2's live prompt (sha1 `d32e04c6`). Scripts and raw outputs: this folder (`results/*_20260917_22*`). DeepSeek spend for the whole exercise: 11 calls. ## Verdict The model is a partial copy of DeepSeek, not a replacement. It has learned what gold-specific headline words mean and nothing else: perfect on direct gold news and safe-haven stories, near random on macro doctrine (hawkish/dovish Fed, dollar, yields, risk-on), zero on negation, and it assigns a direction to irrelevant headlines. DeepSeek is still 78/78 on the independent set today. | Test | Truth | Model | DeepSeek | Keyword fallback | |---|---|---|---|---| | Golden set, 78 reviewer-labelled headlines, 3-class | human | **39/78 = 50.0%** | **78/78 = 100%** | 24/78 = 30.8% | | Golden set, directional rows only (70), right sign | human | 38/70 = 54.3% | 70/70 | 16/70 | | Archive, expanding daily folds, pooled OOF (567 rows), 3-class | DeepSeek | 64.4% | (is the label) | 44.4% | | Archive OOF, sign agreement when both lean | DeepSeek | 74.8% | | 86.8% (on 152 rows only) | | Live feed, last 24 h, 66 headlines not in training, 3-class | DeepSeek | 57.6% (66.7% headline-only) | | 39.4% | | Live feed, gold-mentioning and not in training (37) | DeepSeek | 70.3% (75.7% headline-only) | | 35.1% | | DeepSeek re-scored 100 archived headlines (teacher self-agreement) | DeepSeek (earlier) | | 90.0% 3-class, 98.6% sign | | Paired McNemar on the golden set: model vs DeepSeek 0 vs 39 discordant (p < 0.0001); model vs keyword 28 vs 13 (p = 0.028). The model beats the keyword fallback and loses to DeepSeek by a wide margin. ## Golden set by category (hits / n): model | DeepSeek | keyword ``` distractors 1/8 | 8/8 | 8/8 irrelevant news gets a direction dovish_fed 6/8 | 8/8 | 7/8 gold_direct 8/8 | 8/8 | 4/8 learned hawkish_fed 3/8 | 8/8 | 2/8 "Hot CPI fuels Fed tightening" -> bullish inverted_metrics 3/6 | 6/6 | 0/6 negation_traps 0/8 | 8/8 | 2/8 "signals no rate hike" -> bearish 0.85 conf real_yields 3/6 | 6/6 | 1/6 risk_on 1/4 | 4/4 | 0/4 safe_haven 8/8 | 8/8 | 0/8 learned strong_usd 2/7 | 7/7 | 0/7 "Greenback rallies" -> bullish weak_usd 4/7 | 7/7 | 0/7 ``` The model's confidence does not flag its errors on this set: the negation-trap miss above carries 0.85 confidence and the "consumer confidence surges" miss 0.82. ## Archive time-split (retrain on days < d, score day d) ``` day train test | full acc F1 | head acc F1 | keyword 09-10 49 87 | 62.1% 0.626 | 51.7% 0.521 | 47.1% 09-11 136 91 | 52.7% 0.524 | 53.8% 0.525 | 39.6% 09-14 227 130 | 76.9% 0.714 | 72.3% 0.656 | 58.5% 09-15 357 84 | 69.0% 0.656 | 61.9% 0.605 | 44.0% 09-16 441 115 | 55.7% 0.545 | 54.8% 0.531 | 33.0% 09-17 556 60 | 68.3% 0.643 | 75.0% 0.703 | 40.0% pooled — 567 | 64.4% 0.636 | 61.4% 0.607 | 44.4% ``` "full" = headline + description (the shipped recipe), "head" = headline only. Agreement does not rise with more training days yet; the day-to-day swing (53–77%) is larger than any trend. In-sample the shipped bundle scores 97.4%, so the gap to 64% is generalisation, not capacity. The description helps a little on the archive folds and hurts on the live sample (60.5% vs 68.4%); both differences are inside the noise of these sample sizes. ## Ceiling: DeepSeek disagrees with itself Re-sending 100 archived headlines today (same prompt, same model, temperature 0, different batch neighbours): 90% same class, 93% directional, 98.6% same sign, but only 42% the identical number, mean |difference| 0.09. A student cannot exceed about 90% class agreement on this label. Where DeepSeek changed its class (9 rows) the OOF model matched one of its two answers on 7. ## Confidence routing (pooled OOF, shipped recipe) ``` conf >= kept share kept agreement hybrid agreement (rest to DeepSeek) 0.60 44% 80.9% 91.5% 0.70 28% 87.4% 96.5% 0.80 17% 95.8% 99.3% 0.90 6.5% 100.0% 100.0% ``` The WORKFLOW doc's 0.70 / 0.20 starting point keeps 28% of headlines local at 87% agreement with DeepSeek. That saves roughly a quarter of the calls, which are 56 ms and a fraction of a cent per headline in batches of 25 anyway. The golden set shows the caveat: high confidence on a macro or negated headline is not a reliable signal there. ## Structural findings 1. **Training covers gold-mentioning headlines only.** The archive stores DeepSeek's XAU score only when the headline matches a gold keyword; for USD-only headlines only the overall score is kept and DeepSeek's xau is discarded (`_apply_llm_sentiment`). So the model never saw a Fed, dollar or yields headline that did not also say "gold", which is exactly where it fails. Live, 29 of the 76 headlines were USD-only; the model was at 60% on them versus 72–77% on gold-mentioning ones. Archiving DeepSeek's xau for every headline would fix the training set at no extra cost. 2. **Nine days of data spanning one regime** (a Fed hiking cycle in September 2026). The 2026-09-08 rule stands: do not deploy before about 4,000 labelled headlines over three or more weeks. 3. **Negation and doctrine are not learnable from 606 bag-of-words examples**; the regex domain markers catch "dollar ... strengthen" but not "greenback rallies" or "dollar index hits high". 4. **Latency**: model 1 ms per headline in-process; DeepSeek 56 ms per headline (1.4 s per batch of 25). Latency is not the constraint. ## Addendum: backfilling the missing gold labels (same day) `backfill_labels.py` sent the 961 archive headlines without a gold label (930 DeepSeek-scored without a gold key, 31 never scored) through the live DeepSeek path: 38 calls, all labelled, kept in `results/backfill_labels.jsonl`. Macro labels at ±0.05: 350 bearish, 228 bullish, 383 neutral (40% neutral versus 22% on gold headlines). | Training set | Gold-headline OOF | Macro-headline OOF | Golden set | |---|---|---|---| | gold rows only (616, shipped data) | 64.4% | 38.4% | 39/78 | | gold + macro rows (1,577) | 61.6% | 51.8% | 42/78 | Adding the macro headlines lifts macro agreement by 13 points but costs 3 points on gold headlines, and the golden-set change is a reshuffle, not a fix: distractors 1→6, hawkish 3→5, strong-USD 2→4, weak-USD 4→6, but dovish 6→3, safe-haven 8→4, risk-on 1→0. On nine days of one regime the linear model cannot hold both worlds at once. The missing-label problem was real, but repairing it on the existing corpus does not produce a usable model; it makes the corpus complete so that weeks of further collection can. Side finding for the live system: on the 930 macro headlines, the gold reading the platform displays for a non-gold headline (the inverted overall score, `_gold_direction_label`) matches DeepSeek's own gold score only 61% of the time (sign 77%). Display-only, but the macro persona and the summary read that label. `experiment_recipe.py` (same day): stripping the publisher suffix, marking negated words and weighting confident labels moved OOF agreement by less than a point and the golden set to 40–42/78; negation traps went from 0/8 to 2/8. Cleaning is not the bottleneck. ## After the archive patch (drafted the same day in ForexBot2) Once the ForexBot2 patch is live, `logs/newsdata.db` gains a `xau_score` column holding DeepSeek's gold score for every LLM-scored row, and headlines naming no tracked currency arrive with `currencies_json = []`. `news_sentiment.py` still reads its label from `score_by_currency_json["XAU"]`, so it will keep seeing only the gold-keyword rows until its loader is switched to `xau_score` (fall back to the JSON key for rows older than the column). Skip rows whose `source` is a test feed (`feed.example`, `good.example`): the ForexBot2 test suite used to write fixture headlines into the real archive; they are keyword-scored, so the `score_source = 'llm'` filter already excludes them. ## How to keep measuring `eval_live.py` appends every fresh headline to `results/live_shadow.jsonl`; run it daily and the out-of-sample sample grows for free (cached DeepSeek answers are never re-bought). Re-run `eval_golden.py` after any retrain; it is the only independent truth.