Initial import: XAUUSD news sentiment models, evals, and data tooling

TF-IDF baseline (news_sentiment.py, models/xauusd_sentiment.joblib) and the
two-head DeBERTa-v3-small transformer (transformer/, news_transformer.py)
with its Google News crawler and DeepSeek labeller (data_collect/), the
golden-set and archive evals (eval/), and the 2026-09-22 and 2026-09-27
results and progress notes.

Transformer weights (model.pt, model.onnx) and the crawl database are not
tracked; rebuild with transformer/train.py and transformer/export_onnx.py.
The DeepSeek label cache in data/ is tracked.

README.md

@@ -0,0 +1,136 @@
+# XAUUSD News Sentiment Model
+
+See [`WORKFLOW_AND_USAGE.md`](WORKFLOW_AND_USAGE.md) for the complete
+integration, LLM fallback, deployment, and retraining workflow.
+
+This project trains a scikit-learn model to answer one specific question:
+
+> Does this news imply bullish, neutral, or bearish sentiment for XAUUSD?
+
+- `bullish`: supportive for gold versus the US dollar, so XAUUSD tends upward
+- `bearish`: negative for gold versus the US dollar, so XAUUSD tends downward
+- `neutral`: no clear directional implication or conflicting signals
+
+Neutral is retained because forcing ambiguous news into a bullish or bearish
+class creates false signals.
+
+The model uses headline and description word/character TF-IDF features with
+logistic regression. It also derives transparent domain features for explicit
+gold movement, US dollar direction, Treasury-yield direction, and
+hawkish/dovish policy language. Its numeric score is in `[-1, 1]`:
+
+```text
+P(XAUUSD bullish) - P(XAUUSD bearish)
+```
+
+## Training Labels
+
+Labels are DeepSeek's (`deepseek-v4-flash`) XAU score for each headline, not
+human or trader labels. The general article `score` is never used. For each
+row the trainer takes the first of:
+
+1. the `xau_score` column (stored for every LLM-scored headline since
+   2026-09-18),
+2. `score_by_currency_json["XAU"]` (older rows that mention gold),
+3. `--extra-labels` JSONL keyed by headline hash (for example
+   `eval/results/backfill_labels.jsonl`, DeepSeek labels for older macro
+   headlines).
+
+By default only `score_source = 'llm'` rows are used, and test-feed rows
+(`feed.example`, `good.example`) are skipped. `--scope gold` limits training to
+rows with a JSON `XAU` key, which was the behaviour before 2026-09-22. The
+included model (trained 2026-09-22, `--scope all`) used 2,620 deduplicated
+headlines from Sep 8–22 (1,064 `xau_score`, 640 JSON, 933 backfill):
+
+- 909 bearish
+- 893 neutral
+- 818 bullish
+
+## Setup
+
+```bash
+python3 -m venv .venv
+. .venv/bin/activate
+python -m pip install -r requirements.txt
+```
+
+## Train
+
+```bash
+python news_sentiment.py train --db newsdata.db \
+  --extra-labels eval/results/backfill_labels.jsonl
+```
+
+This creates:
+
+- `models/xauusd_sentiment.joblib`: fitted model and metadata
+- `models/xauusd_sentiment.metrics.json`: training and evaluation report
+
+## Evaluation
+
+The included model's built-in holdouts:
+
+- Stratified random holdout: 69.3% accuracy and 0.689 macro-F1
+- Newest chronological holdout (Sep 21–22): 66.6% accuracy and 0.659 macro-F1
+
+These are lower than the 2026-09-17 model's 73.0% / 69.7%. That is because the
+test sets now include macro headlines (Fed, dollar, yields), which are harder.
+The fair comparison is on the same unseen rows: on Sep 18–22 the 09-17 model
+agreed with DeepSeek on 45.9% of headlines (36.2% on macro). This recipe,
+retrained daily, reached 62.8% (58.8% on macro). On the independent 78-headline
+golden set it scores 47/78; the 09-17 model scored 39/78 and DeepSeek 78/78. See
+`eval/RESULTS_2026-09-22.md`.
+
+The data still covers only 15 days of a single market regime. More labelled
+history across different regimes is the most valuable next improvement.
+
+## Score News
+
+```bash
+python news_sentiment.py score \
+  --headline "Gold gains as the dollar weakens"
+```
+
+Example output from the included model:
+
+```json
+{
+  "target": "XAUUSD",
+  "label": "bullish",
+  "score": 0.94748,
+  "confidence": 0.971834,
+  "probabilities": {
+    "bearish": 0.024354,
+    "neutral": 0.003812,
+    "bullish": 0.971834
+  }
+}
+```
+
+For batch scoring, pass newline-delimited JSON. Each object can contain
+`headline` (or `title`/`text`) and an optional `description`:
+
+```bash
+python news_sentiment.py score --input articles.jsonl
+python news_sentiment.py score --input - < articles.jsonl
+```
+
+The command writes JSONL to standard output and adds a `sentiment` object to
+each input record.
+
+## Python API
+
+```python
+from news_sentiment import load_model, score_news
+
+model = load_model("models/xauusd_sentiment.joblib")
+result = score_news(
+    model,
+    "Gold slides as Treasury yields rise",
+    "A stronger US dollar adds pressure to bullion.",
+)
+print(result)
+```
+
+This model estimates XAUUSD sentiment expressed by news. It does not predict
+the size or timing of a future price move and is not trading advice.