Harwinder Sekhon
01baef48ff
TF-IDF baseline (news_sentiment.py, models/xauusd_sentiment.joblib) and the two-head DeBERTa-v3-small transformer (transformer/, news_transformer.py) with its Google News crawler and DeepSeek labeller (data_collect/), the golden-set and archive evals (eval/), and the 2026-09-22 and 2026-09-27 results and progress notes. Transformer weights (model.pt, model.onnx) and the crawl database are not tracked; rebuild with transformer/train.py and transformer/export_onnx.py. The DeepSeek label cache in data/ is tracked.
3.0 KiB
ML-NEWS model vs DeepSeek — evaluation scripts
Research scripts only. They live here, outside the ForexBot2 git tree,
and nothing in ForexBot2 imports them. They read ForexBot2's live news
code and its logs/newsdata.db read-only, and they never write to it.
Run every script with the ForexBot2 venv (it has requests, pandas
and the live news engine; the model loads there with identical output):
cd /home/harwinder/Development/ML-NEWS/eval
/home/harwinder/Development/ForexBot2/venv/bin/python eval_golden.py
/home/harwinder/Development/ForexBot2/venv/bin/python eval_archive.py --reprobe 100
/home/harwinder/Development/ForexBot2/venv/bin/python eval_live.py --max-age-hours 24
./run_all.sh # the three in order
Add --no-network to any of them to report from the DeepSeek cache
without spending. DeepSeek answers are cached in
results/deepseek_cache.json, keyed by headline, prompt sha1 and model,
so a headline is paid for once; a prompt change invalidates the cache
by itself.
What each script measures
| Script | Truth | Question it answers |
|---|---|---|
eval_golden.py |
78 reviewer-labelled headlines (data/llm_news_golden.csv, the 2026-08-06 set where DeepSeek scored 78/78 and keywords 24/78) |
Is the model right, per category, including negation traps and inverted metrics; and is DeepSeek still 78/78 today |
eval_archive.py |
DeepSeek's own archived XAU scores | How well the recipe copies DeepSeek on days it never saw (expanding daily folds, headline+description vs headline only, keyword fallback for context); which confidence/margin thresholds keep how many headlines local at what agreement; how often DeepSeek agrees with itself on re-scoring (--reprobe), which is the ceiling for any student |
eval_live.py |
DeepSeek live, on today's feed | Shadow mode: agreement on fresh headlines from the real feeds, split by gold-mentioning / not-in-training; accumulates in results/live_shadow.jsonl across days |
Why three: the model was trained on DeepSeek's labels, so agreement on the archive says how well the student copies the teacher, not whether either is right. The golden set is the only independent truth. The live log grows the out-of-sample set over time.
Grading rules
- Golden set: a continuous XAU score is bullish at ≥ +0.15, bearish at ≤ −0.15, neutral in between (the August rule). The model is graded by its native argmax label and, separately, by its score through the same band.
- Archive and live: DeepSeek xau → class at ±0.05, the ML-NEWS trainer's own label rule; model = argmax label.
dir= expected non-neutral rows predicted with the right sign (a neutral prediction is a miss).sign= same sign when both lean. McNemar is exact and two-sided on discordant pairs.
Outputs
results/golden_<ts>.{json,csv}, results/archive_<ts>.json,
results/archive_oof_<ts>.csv (every out-of-fold prediction),
results/archive_reprobe_<ts>.csv, results/live_<ts>.json,
results/live_shadow.jsonl (append-only, deduplicated by headline hash).