Harwinder Sekhon
1d72e4006a
TF-IDF baseline (news_sentiment.py, models/xauusd_sentiment.joblib) and the two-head DeBERTa-v3-small transformer (transformer/, news_transformer.py) with its Google News crawler and DeepSeek labeller (data_collect/), the golden-set and archive evals (eval/), and the 2026-09-22 and 2026-09-27 results and progress notes. Transformer weights (model.pt, model.onnx) and the crawl database are not tracked; rebuild with transformer/train.py and transformer/export_onnx.py. The DeepSeek label cache in data/ is tracked. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
55 lines
3.0 KiB
Markdown
55 lines
3.0 KiB
Markdown
# ML-NEWS model vs DeepSeek — evaluation scripts
|
||
|
||
Research scripts only. They live here, outside the ForexBot2 git tree,
|
||
and nothing in ForexBot2 imports them. They read ForexBot2's live news
|
||
code and its `logs/newsdata.db` read-only, and they never write to it.
|
||
|
||
Run every script with the ForexBot2 venv (it has `requests`, `pandas`
|
||
and the live news engine; the model loads there with identical output):
|
||
|
||
```bash
|
||
cd /home/harwinder/Development/ML-NEWS/eval
|
||
/home/harwinder/Development/ForexBot2/venv/bin/python eval_golden.py
|
||
/home/harwinder/Development/ForexBot2/venv/bin/python eval_archive.py --reprobe 100
|
||
/home/harwinder/Development/ForexBot2/venv/bin/python eval_live.py --max-age-hours 24
|
||
./run_all.sh # the three in order
|
||
```
|
||
|
||
Add `--no-network` to any of them to report from the DeepSeek cache
|
||
without spending. DeepSeek answers are cached in
|
||
`results/deepseek_cache.json`, keyed by headline, prompt sha1 and model,
|
||
so a headline is paid for once; a prompt change invalidates the cache
|
||
by itself.
|
||
|
||
## What each script measures
|
||
|
||
| Script | Truth | Question it answers |
|
||
|---|---|---|
|
||
| `eval_golden.py` | 78 reviewer-labelled headlines (`data/llm_news_golden.csv`, the 2026-08-06 set where DeepSeek scored 78/78 and keywords 24/78) | Is the model *right*, per category, including negation traps and inverted metrics; and is DeepSeek still 78/78 today |
|
||
| `eval_archive.py` | DeepSeek's own archived XAU scores | How well the recipe copies DeepSeek on days it never saw (expanding daily folds, headline+description vs headline only, keyword fallback for context); which confidence/margin thresholds keep how many headlines local at what agreement; how often DeepSeek agrees with itself on re-scoring (`--reprobe`), which is the ceiling for any student |
|
||
| `eval_live.py` | DeepSeek live, on today's feed | Shadow mode: agreement on fresh headlines from the real feeds, split by gold-mentioning / not-in-training; accumulates in `results/live_shadow.jsonl` across days |
|
||
|
||
Why three: the model was trained on DeepSeek's labels, so agreement on
|
||
the archive says how well the student copies the teacher, not whether
|
||
either is right. The golden set is the only independent truth. The live
|
||
log grows the out-of-sample set over time.
|
||
|
||
## Grading rules
|
||
|
||
* Golden set: a continuous XAU score is bullish at ≥ +0.15, bearish at
|
||
≤ −0.15, neutral in between (the August rule). The model is graded by
|
||
its native argmax label and, separately, by its score through the
|
||
same band.
|
||
* Archive and live: DeepSeek xau → class at ±0.05, the ML-NEWS trainer's
|
||
own label rule; model = argmax label.
|
||
* `dir` = expected non-neutral rows predicted with the right sign (a
|
||
neutral prediction is a miss). `sign` = same sign when both lean.
|
||
McNemar is exact and two-sided on discordant pairs.
|
||
|
||
## Outputs
|
||
|
||
`results/golden_<ts>.{json,csv}`, `results/archive_<ts>.json`,
|
||
`results/archive_oof_<ts>.csv` (every out-of-fold prediction),
|
||
`results/archive_reprobe_<ts>.csv`, `results/live_<ts>.json`,
|
||
`results/live_shadow.jsonl` (append-only, deduplicated by headline hash).
|