Harwinder Sekhon
01baef48ff
TF-IDF baseline (news_sentiment.py, models/xauusd_sentiment.joblib) and the two-head DeBERTa-v3-small transformer (transformer/, news_transformer.py) with its Google News crawler and DeepSeek labeller (data_collect/), the golden-set and archive evals (eval/), and the 2026-09-22 and 2026-09-27 results and progress notes. Transformer weights (model.pt, model.onnx) and the crawl database are not tracked; rebuild with transformer/train.py and transformer/export_onnx.py. The DeepSeek label cache in data/ is tracked.
137 lines
4.2 KiB
Markdown
137 lines
4.2 KiB
Markdown
# XAUUSD News Sentiment Model
|
||
|
||
See [`WORKFLOW_AND_USAGE.md`](WORKFLOW_AND_USAGE.md) for the complete
|
||
integration, LLM fallback, deployment, and retraining workflow.
|
||
|
||
This project trains a scikit-learn model to answer one specific question:
|
||
|
||
> Does this news imply bullish, neutral, or bearish sentiment for XAUUSD?
|
||
|
||
- `bullish`: supportive for gold versus the US dollar, so XAUUSD tends upward
|
||
- `bearish`: negative for gold versus the US dollar, so XAUUSD tends downward
|
||
- `neutral`: no clear directional implication or conflicting signals
|
||
|
||
Neutral is retained because forcing ambiguous news into a bullish or bearish
|
||
class creates false signals.
|
||
|
||
The model uses headline and description word/character TF-IDF features with
|
||
logistic regression. It also derives transparent domain features for explicit
|
||
gold movement, US dollar direction, Treasury-yield direction, and
|
||
hawkish/dovish policy language. Its numeric score is in `[-1, 1]`:
|
||
|
||
```text
|
||
P(XAUUSD bullish) - P(XAUUSD bearish)
|
||
```
|
||
|
||
## Training Labels
|
||
|
||
Labels are DeepSeek's (`deepseek-v4-flash`) XAU score for each headline, not
|
||
human or trader labels. The general article `score` is never used. For each
|
||
row the trainer takes the first of:
|
||
|
||
1. the `xau_score` column (stored for every LLM-scored headline since
|
||
2026-09-18),
|
||
2. `score_by_currency_json["XAU"]` (older rows that mention gold),
|
||
3. `--extra-labels` JSONL keyed by headline hash (for example
|
||
`eval/results/backfill_labels.jsonl`, DeepSeek labels for older macro
|
||
headlines).
|
||
|
||
By default only `score_source = 'llm'` rows are used, and test-feed rows
|
||
(`feed.example`, `good.example`) are skipped. `--scope gold` limits training to
|
||
rows with a JSON `XAU` key, which was the behaviour before 2026-09-22. The
|
||
included model (trained 2026-09-22, `--scope all`) used 2,620 deduplicated
|
||
headlines from Sep 8–22 (1,064 `xau_score`, 640 JSON, 933 backfill):
|
||
|
||
- 909 bearish
|
||
- 893 neutral
|
||
- 818 bullish
|
||
|
||
## Setup
|
||
|
||
```bash
|
||
python3 -m venv .venv
|
||
. .venv/bin/activate
|
||
python -m pip install -r requirements.txt
|
||
```
|
||
|
||
## Train
|
||
|
||
```bash
|
||
python news_sentiment.py train --db newsdata.db \
|
||
--extra-labels eval/results/backfill_labels.jsonl
|
||
```
|
||
|
||
This creates:
|
||
|
||
- `models/xauusd_sentiment.joblib`: fitted model and metadata
|
||
- `models/xauusd_sentiment.metrics.json`: training and evaluation report
|
||
|
||
## Evaluation
|
||
|
||
The included model's built-in holdouts:
|
||
|
||
- Stratified random holdout: 69.3% accuracy and 0.689 macro-F1
|
||
- Newest chronological holdout (Sep 21–22): 66.6% accuracy and 0.659 macro-F1
|
||
|
||
These are lower than the 2026-09-17 model's 73.0% / 69.7%. That is because the
|
||
test sets now include macro headlines (Fed, dollar, yields), which are harder.
|
||
The fair comparison is on the same unseen rows: on Sep 18–22 the 09-17 model
|
||
agreed with DeepSeek on 45.9% of headlines (36.2% on macro). This recipe,
|
||
retrained daily, reached 62.8% (58.8% on macro). On the independent 78-headline
|
||
golden set it scores 47/78; the 09-17 model scored 39/78 and DeepSeek 78/78. See
|
||
`eval/RESULTS_2026-09-22.md`.
|
||
|
||
The data still covers only 15 days of a single market regime. More labelled
|
||
history across different regimes is the most valuable next improvement.
|
||
|
||
## Score News
|
||
|
||
```bash
|
||
python news_sentiment.py score \
|
||
--headline "Gold gains as the dollar weakens"
|
||
```
|
||
|
||
Example output from the included model:
|
||
|
||
```json
|
||
{
|
||
"target": "XAUUSD",
|
||
"label": "bullish",
|
||
"score": 0.94748,
|
||
"confidence": 0.971834,
|
||
"probabilities": {
|
||
"bearish": 0.024354,
|
||
"neutral": 0.003812,
|
||
"bullish": 0.971834
|
||
}
|
||
}
|
||
```
|
||
|
||
For batch scoring, pass newline-delimited JSON. Each object can contain
|
||
`headline` (or `title`/`text`) and an optional `description`:
|
||
|
||
```bash
|
||
python news_sentiment.py score --input articles.jsonl
|
||
python news_sentiment.py score --input - < articles.jsonl
|
||
```
|
||
|
||
The command writes JSONL to standard output and adds a `sentiment` object to
|
||
each input record.
|
||
|
||
## Python API
|
||
|
||
```python
|
||
from news_sentiment import load_model, score_news
|
||
|
||
model = load_model("models/xauusd_sentiment.joblib")
|
||
result = score_news(
|
||
model,
|
||
"Gold slides as Treasury yields rise",
|
||
"A stronger US dollar adds pressure to bullion.",
|
||
)
|
||
print(result)
|
||
```
|
||
|
||
This model estimates XAUUSD sentiment expressed by news. It does not predict
|
||
the size or timing of a future price move and is not trading advice.
|