harry/News-ML-Decoder
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

Initial import: XAUUSD news sentiment models, evals, and data tooling

Browse Source 
TF-IDF baseline (news_sentiment.py, models/xauusd_sentiment.joblib) and the
two-head DeBERTa-v3-small transformer (transformer/, news_transformer.py)
with its Google News crawler and DeepSeek labeller (data_collect/), the
golden-set and archive evals (eval/), and the 2026-09-22 and 2026-09-27
results and progress notes.

Transformer weights (model.pt, model.onnx) and the crawl database are not
tracked; rebuild with transformer/train.py and transformer/export_onnx.py.
The DeepSeek label cache in data/ is tracked.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
This commit is contained in:
Harwinder Sekhon
2026-09-27 19:22:30 -04:00
commit 1d72e4006a
98 changed files with 1719757 additions and 0 deletions
Download Patch File Download Diff File Expand all files Collapse all files
eval/RESULTS_2026-09-22.md
+124
View File
@@ -0,0 +1,124 @@
# Retrain on the Sep 8–22 archive, 2026-09-22
Data: `newsdata.db` copied from the 2026-09-22 archive, with 2,665 headlines.
- 2,661 rows are `score_source='llm'`, scored by `deepseek-v4-flash` with prompt `d32e04c6`.
- 4 rows are keyword-scored and not used.
- **Every label is DeepSeek's, not a human's or a trader's.** The 78-headline golden set is still the only
independent truth.
The ForexBot2 archive patch is live: `xau_score` is filled for every LLM row from 2026-09-18 on (1,064 rows).
Where a row also has a JSON `XAU` key (319 rows), the two values match on all of them.
DeepSeek spend for this run: 0 calls (`--no-network`, everything came from the cache).
## What changed in the code
- **`news_sentiment.py`:** the label is taken from `xau_score`, then the JSON `XAU` key, then `--extra-labels`
(backfill JSONL keyed by hash).
- Rows from test feeds are dropped.
- `--scope all|gold` was added.
- The metrics now record `label_sources`, `scope` and `published_range`.
- The model format is unchanged.
- **`eval_common.load_archive_rows`:** uses the same label order and sets `xau_src` and `gold` (whether the row
has a JSON XAU key) on each row.
- **`eval_golden.py`:** new `--model` option.
- **`eval_archive.py`:** new `--model`, `--extra-labels` and `--train-scope` options.
- Part A now also scores rows published after the bundle's `trained_at`. For the 09-17 bundle those rows
are out of sample.
- Part B pools its results by gold rows, macro rows and days on or after 09-18.
- In Part B the test day is always every labelled row, so both train scopes are graded on the same rows.
## Candidates
| Model | Train rows | Labels (bear / neu / bull) | Stratified acc / F1 | Chronological acc / F1 |
|---|---|---|---|---|
| shipped 09-17 (gold only, Sep 8–17) | 606 | 250 / 135 / 221 | 73.0% / 0.709 | 69.7% / 0.671 |
| `--scope gold` (Sep 8–22) | 945 | 359 / 202 / 384 | 80.4% / 0.793 | 74.6% / 0.734 |
| **`--scope all` + backfill (Sep 8–22)** | **2,620** | 909 / 893 / 818 | 69.3% / 0.689 | 66.6% / 0.659 |
You can't compare the built-in holdouts across rows of this table:
- The `gold` and 09-17 holdouts contain only headlines that mention gold, which are the easy ones.
- The `all` holdout is 64% macro headlines.
The next two sections grade every model on the same rows.
## Same unseen rows: Sep 18–22 (1,022–1,066 headlines, DeepSeek label at ±0.05)
| Model | All | Gold-key | Macro |
|---|---|---|---|
| shipped 09-17 bundle (never saw these days) | 45.9% (F1 0.455) | 68.2% | 36.2% |
| `gold` recipe, retrained daily on days < d | 48.3% (F1 0.478) | **73.8%** | 37.1% |
| **`all` recipe, retrained daily on days < d** | **62.8% (F1 0.629)** | 71.9% | **58.8%** |
Pooled over every daily fold from Sep 9–22 (2,566 / 2,530 rows):
| Recipe | 3-class | Macro-F1 | Gold-key | Macro | Keyword fallback |
|---|---|---|---|---|---|
| `all` | 58.2% | 0.578 | 65.1% | 54.3% | 44.7% |
| `gold` | 48.3% | 0.479 | 67.8% | 37.3% | 44.7% |
## Golden set (78 reviewer-labelled headlines, argmax label)
| Model | 3-class | McNemar vs keyword | Spearman vs DeepSeek |
|---|---|---|---|
| shipped 09-17 | 39/78 (50.0%) | 28 vs 13, p = 0.028 | +0.305 |
| `gold` | 40/78 (51.3%) | 29 vs 13, p = 0.020 | +0.310 |
| **`all`** | **47/78 (60.3%)** | 31 vs 8, p = 0.0003 | +0.423 |
| DeepSeek | 78/78 | | |
Hits by category, shipped → `all`:
| Category | Shipped | `all` |
|---|---|---|
| hawkish_fed | 3 | 8 |
| strong_usd | 2 | 5 |
| distractors | 1 | 5 |
| negation_traps | 0 | 1 |
| weak_usd | 4 | 4 |
| real_yields | 3 | 3 |
| inverted_metrics | 3 | 3 |
| dovish_fed | 6 | 4 |
| gold_direct | 8 | 7 |
| safe_haven | 8 | 7 |
| risk_on | 1 | 0 |
The `all` model's weak spots:
- **Easing and falling yields read as bearish.** "Fed cuts rates 25bp" → bearish 0.78, and "Treasury yields sink"
→ bearish 0.78. Nearly all of the September macro data comes from a hiking regime, so the model links
"Fed" and "yields" with bearish and has almost no examples of easing.
- **Risk-on headlines score 0/4.** "Stocks roar to record highs" → bullish.
- **Negation still fails** (1/8).
## Verdict
**Promoted `--scope all`.** It is now `models/xauusd_sentiment.joblib`. The 09-17 bundle is kept in
`models/archive/xauusd_sentiment_20260917.*`, and both candidates are in `models/candidates/`.
- **Against the shipped model:** on the same unseen days it wins by 17 points overall and 22 points on
macro headlines.
- **Against the gold-only recipe:** it gives up 1.9 points on gold headlines and gains 21.7 points on macro.
- **Golden set:** 39 → 47/78.
The promotion rule written before the run said "chronological macro-F1 ≥ shipped". **Taken literally, that
test fails:** 0.659 against 0.671. The two numbers come from different test sets, one gold-only and one
mostly macro. On the same unseen rows the retrained model wins, and that is the comparison the rule was
meant to capture.
This is still not a DeepSeek replacement:
- It is 31 golden headlines behind DeepSeek.
- It learns only what DeepSeek says, not how price moves.
- It has 2,620 labels over 15 days, which is short of the "about 4,000 labels over 3+ weeks" rule. Keep it in
shadow mode with DeepSeek as the answer.
Nothing in ForexBot2 imports the model, so the promotion changes nothing live.
`ForexBot2/venv` has sklearn 1.9.0 and the bundle was pickled with 1.9.1. That raises an
`InconsistentVersionWarning`, which the eval scripts suppress. The 09-17 bundle had the same mismatch. Align
the two venvs before any live use.
## Next
1. Keep collecting. Every new day adds about 300 labels from `xau_score`. Retrain weekly and rerun both
evals.
2. Easing and falling yields are the obvious gap. Around 20–40 DeepSeek-labelled dovish, yields-down and
risk-on headlines would test whether this gap comes from the regime or from the model.