Harwinder Sekhon
01baef48ff
TF-IDF baseline (news_sentiment.py, models/xauusd_sentiment.joblib) and the two-head DeBERTa-v3-small transformer (transformer/, news_transformer.py) with its Google News crawler and DeepSeek labeller (data_collect/), the golden-set and archive evals (eval/), and the 2026-09-22 and 2026-09-27 results and progress notes. Transformer weights (model.pt, model.onnx) and the crawl database are not tracked; rebuild with transformer/train.py and transformer/export_onnx.py. The DeepSeek label cache in data/ is tracked.
172 lines
8.6 KiB
Markdown
172 lines
8.6 KiB
Markdown
# ML-NEWS scikit model vs DeepSeek — first evaluation, 2026-09-17
|
||
|
||
Model under test: `models/xauusd_sentiment.joblib` (TF-IDF word + char,
|
||
logistic regression, trained 2026-09-17 on 606 DeepSeek-labelled
|
||
gold headlines from Sep 8–17). Teacher: DeepSeek `deepseek-v4-flash`
|
||
through ForexBot2's live prompt (sha1 `d32e04c6`). Scripts and raw
|
||
outputs: this folder (`results/*_20260917_22*`). DeepSeek spend for the
|
||
whole exercise: 11 calls.
|
||
|
||
## Verdict
|
||
|
||
The model is a partial copy of DeepSeek, not a replacement. It has
|
||
learned what gold-specific headline words mean and nothing else:
|
||
perfect on direct gold news and safe-haven stories, near random on
|
||
macro doctrine (hawkish/dovish Fed, dollar, yields, risk-on), zero on
|
||
negation, and it assigns a direction to irrelevant headlines. DeepSeek
|
||
is still 78/78 on the independent set today.
|
||
|
||
| Test | Truth | Model | DeepSeek | Keyword fallback |
|
||
|---|---|---|---|---|
|
||
| Golden set, 78 reviewer-labelled headlines, 3-class | human | **39/78 = 50.0%** | **78/78 = 100%** | 24/78 = 30.8% |
|
||
| Golden set, directional rows only (70), right sign | human | 38/70 = 54.3% | 70/70 | 16/70 |
|
||
| Archive, expanding daily folds, pooled OOF (567 rows), 3-class | DeepSeek | 64.4% | (is the label) | 44.4% |
|
||
| Archive OOF, sign agreement when both lean | DeepSeek | 74.8% | | 86.8% (on 152 rows only) |
|
||
| Live feed, last 24 h, 66 headlines not in training, 3-class | DeepSeek | 57.6% (66.7% headline-only) | | 39.4% |
|
||
| Live feed, gold-mentioning and not in training (37) | DeepSeek | 70.3% (75.7% headline-only) | | 35.1% |
|
||
| DeepSeek re-scored 100 archived headlines (teacher self-agreement) | DeepSeek (earlier) | | 90.0% 3-class, 98.6% sign | |
|
||
|
||
Paired McNemar on the golden set: model vs DeepSeek 0 vs 39 discordant
|
||
(p < 0.0001); model vs keyword 28 vs 13 (p = 0.028). The model beats
|
||
the keyword fallback and loses to DeepSeek by a wide margin.
|
||
|
||
## Golden set by category (hits / n): model | DeepSeek | keyword
|
||
|
||
```
|
||
distractors 1/8 | 8/8 | 8/8 irrelevant news gets a direction
|
||
dovish_fed 6/8 | 8/8 | 7/8
|
||
gold_direct 8/8 | 8/8 | 4/8 learned
|
||
hawkish_fed 3/8 | 8/8 | 2/8 "Hot CPI fuels Fed tightening" -> bullish
|
||
inverted_metrics 3/6 | 6/6 | 0/6
|
||
negation_traps 0/8 | 8/8 | 2/8 "signals no rate hike" -> bearish 0.85 conf
|
||
real_yields 3/6 | 6/6 | 1/6
|
||
risk_on 1/4 | 4/4 | 0/4
|
||
safe_haven 8/8 | 8/8 | 0/8 learned
|
||
strong_usd 2/7 | 7/7 | 0/7 "Greenback rallies" -> bullish
|
||
weak_usd 4/7 | 7/7 | 0/7
|
||
```
|
||
|
||
The model's confidence does not flag its errors on this set: the
|
||
negation-trap miss above carries 0.85 confidence and the "consumer
|
||
confidence surges" miss 0.82.
|
||
|
||
## Archive time-split (retrain on days < d, score day d)
|
||
|
||
```
|
||
day train test | full acc F1 | head acc F1 | keyword
|
||
09-10 49 87 | 62.1% 0.626 | 51.7% 0.521 | 47.1%
|
||
09-11 136 91 | 52.7% 0.524 | 53.8% 0.525 | 39.6%
|
||
09-14 227 130 | 76.9% 0.714 | 72.3% 0.656 | 58.5%
|
||
09-15 357 84 | 69.0% 0.656 | 61.9% 0.605 | 44.0%
|
||
09-16 441 115 | 55.7% 0.545 | 54.8% 0.531 | 33.0%
|
||
09-17 556 60 | 68.3% 0.643 | 75.0% 0.703 | 40.0%
|
||
pooled — 567 | 64.4% 0.636 | 61.4% 0.607 | 44.4%
|
||
```
|
||
|
||
"full" = headline + description (the shipped recipe), "head" =
|
||
headline only. Agreement does not rise with more training days yet;
|
||
the day-to-day swing (53–77%) is larger than any trend. In-sample the
|
||
shipped bundle scores 97.4%, so the gap to 64% is generalisation, not
|
||
capacity. The description helps a little on the archive folds and
|
||
hurts on the live sample (60.5% vs 68.4%); both differences are inside
|
||
the noise of these sample sizes.
|
||
|
||
## Ceiling: DeepSeek disagrees with itself
|
||
|
||
Re-sending 100 archived headlines today (same prompt, same model,
|
||
temperature 0, different batch neighbours): 90% same class, 93%
|
||
directional, 98.6% same sign, but only 42% the identical number, mean
|
||
|difference| 0.09. A student cannot exceed about 90% class agreement on
|
||
this label. Where DeepSeek changed its class (9 rows) the OOF model
|
||
matched one of its two answers on 7.
|
||
|
||
## Confidence routing (pooled OOF, shipped recipe)
|
||
|
||
```
|
||
conf >= kept share kept agreement hybrid agreement (rest to DeepSeek)
|
||
0.60 44% 80.9% 91.5%
|
||
0.70 28% 87.4% 96.5%
|
||
0.80 17% 95.8% 99.3%
|
||
0.90 6.5% 100.0% 100.0%
|
||
```
|
||
|
||
The WORKFLOW doc's 0.70 / 0.20 starting point keeps 28% of headlines
|
||
local at 87% agreement with DeepSeek. That saves roughly a quarter of
|
||
the calls, which are 56 ms and a fraction of a cent per headline in
|
||
batches of 25 anyway. The golden set shows the caveat: high confidence
|
||
on a macro or negated headline is not a reliable signal there.
|
||
|
||
## Structural findings
|
||
|
||
1. **Training covers gold-mentioning headlines only.** The archive
|
||
stores DeepSeek's XAU score only when the headline matches a gold
|
||
keyword; for USD-only headlines only the overall score is kept and
|
||
DeepSeek's xau is discarded (`_apply_llm_sentiment`). So the model
|
||
never saw a Fed, dollar or yields headline that did not also say
|
||
"gold", which is exactly where it fails. Live, 29 of the 76
|
||
headlines were USD-only; the model was at 60% on them versus 72–77%
|
||
on gold-mentioning ones. Archiving DeepSeek's xau for every headline
|
||
would fix the training set at no extra cost.
|
||
2. **Nine days of data spanning one regime** (a Fed hiking cycle in
|
||
September 2026). The 2026-09-08 rule stands: do not deploy before
|
||
about 4,000 labelled headlines over three or more weeks.
|
||
3. **Negation and doctrine are not learnable from 606 bag-of-words
|
||
examples**; the regex domain markers catch "dollar ... strengthen"
|
||
but not "greenback rallies" or "dollar index hits high".
|
||
4. **Latency**: model 1 ms per headline in-process; DeepSeek 56 ms per
|
||
headline (1.4 s per batch of 25). Latency is not the constraint.
|
||
|
||
## Addendum: backfilling the missing gold labels (same day)
|
||
|
||
`backfill_labels.py` sent the 961 archive headlines without a gold
|
||
label (930 DeepSeek-scored without a gold key, 31 never scored) through
|
||
the live DeepSeek path: 38 calls, all labelled, kept in
|
||
`results/backfill_labels.jsonl`. Macro labels at ±0.05: 350 bearish,
|
||
228 bullish, 383 neutral (40% neutral versus 22% on gold headlines).
|
||
|
||
| Training set | Gold-headline OOF | Macro-headline OOF | Golden set |
|
||
|---|---|---|---|
|
||
| gold rows only (616, shipped data) | 64.4% | 38.4% | 39/78 |
|
||
| gold + macro rows (1,577) | 61.6% | 51.8% | 42/78 |
|
||
|
||
Adding the macro headlines lifts macro agreement by 13 points but
|
||
costs 3 points on gold headlines, and the golden-set change is a
|
||
reshuffle, not a fix: distractors 1→6, hawkish 3→5, strong-USD 2→4,
|
||
weak-USD 4→6, but dovish 6→3, safe-haven 8→4, risk-on 1→0. On nine
|
||
days of one regime the linear model cannot hold both worlds at once.
|
||
The missing-label problem was real, but repairing it on the existing
|
||
corpus does not produce a usable model; it makes the corpus complete
|
||
so that weeks of further collection can.
|
||
|
||
Side finding for the live system: on the 930 macro headlines, the gold
|
||
reading the platform displays for a non-gold headline (the inverted
|
||
overall score, `_gold_direction_label`) matches DeepSeek's own gold
|
||
score only 61% of the time (sign 77%). Display-only, but the macro
|
||
persona and the summary read that label.
|
||
|
||
`experiment_recipe.py` (same day): stripping the publisher suffix,
|
||
marking negated words and weighting confident labels moved OOF
|
||
agreement by less than a point and the golden set to 40–42/78;
|
||
negation traps went from 0/8 to 2/8. Cleaning is not the bottleneck.
|
||
|
||
## After the archive patch (drafted the same day in ForexBot2)
|
||
|
||
Once the ForexBot2 patch is live, `logs/newsdata.db` gains a
|
||
`xau_score` column holding DeepSeek's gold score for every LLM-scored
|
||
row, and headlines naming no tracked currency arrive with
|
||
`currencies_json = []`. `news_sentiment.py` still reads its label from
|
||
`score_by_currency_json["XAU"]`, so it will keep seeing only the
|
||
gold-keyword rows until its loader is switched to `xau_score`
|
||
(fall back to the JSON key for rows older than the column). Skip rows
|
||
whose `source` is a test feed (`feed.example`, `good.example`): the
|
||
ForexBot2 test suite used to write fixture headlines into the real
|
||
archive; they are keyword-scored, so the `score_source = 'llm'` filter
|
||
already excludes them.
|
||
|
||
## How to keep measuring
|
||
|
||
`eval_live.py` appends every fresh headline to
|
||
`results/live_shadow.jsonl`; run it daily and the out-of-sample sample
|
||
grows for free (cached DeepSeek answers are never re-bought). Re-run
|
||
`eval_golden.py` after any retrain; it is the only independent truth.
|