Initial import: XAUUSD news sentiment models, evals, and data tooling

TF-IDF baseline (news_sentiment.py, models/xauusd_sentiment.joblib) and the two-head DeBERTa-v3-small transformer (transformer/, news_transformer.py) with its Google News crawler and DeepSeek labeller (data_collect/), the golden-set and archive evals (eval/), and the 2026-09-22 and 2026-09-27 results and progress notes. Transformer weights (model.pt, model.onnx) and the crawl database are not tracked; rebuild with transformer/train.py and transformer/export_onnx.py. The DeepSeek label cache in data/ is tracked. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
2026-09-27 19:22:30 -04:00
commit 1d72e4006a
98 changed files with 1719757 additions and 0 deletions
@@ -0,0 +1,422 @@
+# XAUUSD Sentiment Workflow and Usage Guide
+
+## Purpose
+
+This model answers one question:
+
+> Does a news headline imply bullish, neutral, or bearish sentiment for
+> XAUUSD?
+
+- **Bullish** means the news supports a higher XAUUSD price.
+- **Bearish** means the news supports a lower XAUUSD price.
+- **Neutral** means the direction is unclear, weak, or conflicting.
+
+The model runs locally with scikit-learn. It does not require an API call and
+can be placed before an existing LLM news-analysis step to reduce latency and
+LLM usage.
+
+## Files
+
+| Path | Purpose |
+| --- | --- |
+| `news_sentiment.py` | Training, scoring, and Python API |
+| `newsdata.db` | SQLite training data |
+| `models/xauusd_sentiment.joblib` | Trained model artifact |
+| `models/xauusd_sentiment.metrics.json` | Training and evaluation metrics |
+| `requirements.txt` | Runtime dependencies |
+
+## Recommended Workflow
+
+```mermaid
+flowchart LR
+    A[New article] --> B[Validate headline and description]
+    B --> C[XAUUSD local model]
+    C --> D{Prediction confident?}
+    D -->|Yes| E[Use local prediction]
+    D -->|No| F[Call existing LLM]
+    E --> G[Store result and source]
+    F --> G
+    G --> H[Retain reviewed or LLM labels]
+    H --> I[Periodic model retraining]
+```
+
+Start in **shadow mode**: run the local model and LLM together, but continue
+using the LLM result. Log both outputs for comparison. After validating the
+model on live news, enable the confidence-based LLM fallback workflow.
+
+## Environment Setup
+
+From this repository:
+
+```bash
+python3 -m venv .venv
+. .venv/bin/activate
+python -m pip install -r requirements.txt
+```
+
+The saved model was built with the exact versions in `requirements.txt`.
+
+If the existing Python project is in another directory, make this repository
+importable when starting it:
+
+```bash
+export PYTHONPATH="/home/harwinder/Development/ML-NEWS:$PYTHONPATH"
+export XAUUSD_MODEL_PATH="/home/harwinder/Development/ML-NEWS/models/xauusd_sentiment.joblib"
+python /path/to/existing/project/main.py
+```
+
+Add the same environment variables to the service manager, Docker container,
+or deployment configuration used in production.
+
+## Command-Line Usage
+
+Score one headline:
+
+```bash
+.venv/bin/python news_sentiment.py score \
+  --headline "Gold gains as the dollar weakens"
+```
+
+Score a headline and description:
+
+```bash
+.venv/bin/python news_sentiment.py score \
+  --headline "Gold slides as Treasury yields rise" \
+  --description "A stronger US dollar adds pressure to bullion."
+```
+
+The output has this structure:
+
+```json
+{
+  "target": "XAUUSD",
+  "label": "bearish",
+  "score": -0.82,
+  "confidence": 0.87,
+  "probabilities": {
+    "bearish": 0.87,
+    "neutral": 0.08,
+    "bullish": 0.05
+  }
+}
+```
+
+The score is calculated as:
+
+```text
+P(XAUUSD bullish) - P(XAUUSD bearish)
+```
+
+Use `label` as the primary classification. A neutral prediction can still have
+a positive or negative score when the model sees a secondary directional
+lean.
+
+## Batch JSONL Usage
+
+Input records must be newline-delimited JSON. Each record can contain
+`headline`, `title`, or `text`, plus an optional `description`:
+
+```jsonl
+{"id": 1, "headline": "Gold rises as the dollar falls"}
+{"id": 2, "headline": "Gold holds steady ahead of CPI"}
+{"id": 3, "title": "Gold falls as Treasury yields climb"}
+```
+
+Score the file:
+
+```bash
+.venv/bin/python news_sentiment.py score --input articles.jsonl > scored.jsonl
+```
+
+Each output record preserves the input fields and adds a `sentiment` object.
+
+## Python Integration
+
+Load the model once when the application process starts. Do not load it for
+every article.
+
+```python
+import os
+from pathlib import Path
+
+from news_sentiment import load_model, score_news
+
+
+MODEL_PATH = Path(
+    os.environ.get(
+        "XAUUSD_MODEL_PATH",
+        "/home/harwinder/Development/ML-NEWS/models/xauusd_sentiment.joblib",
+    )
+)
+
+XAUUSD_MODEL = load_model(MODEL_PATH)
+
+
+def analyze_xauusd_news(news: dict) -> dict:
+    prediction = score_news(
+        XAUUSD_MODEL,
+        headline=news["headline"],
+        description=news.get("description"),
+    )
+
+    return {
+        "target": prediction["target"],
+        "sentiment": prediction["label"],
+        "score": prediction["score"],
+        "confidence": prediction["confidence"],
+        "probabilities": prediction["probabilities"],
+        "score_source": "xauusd_sklearn_v3",
+    }
+```
+
+Example:
+
+```python
+news = {
+    "headline": "Gold rallies as the US dollar weakens",
+    "description": "Falling Treasury yields provided additional support.",
+}
+
+result = analyze_xauusd_news(news)
+print(result)
+```
+
+## Connecting to the Existing LLM Workflow
+
+Assume the existing project currently calls:
+
+```python
+result = analyze_news_with_llm(headline, description)
+```
+
+For a full local replacement:
+
+```python
+result = analyze_xauusd_news(
+    {
+        "headline": headline,
+        "description": description,
+    }
+)
+```
+
+For the recommended hybrid workflow, call the LLM only when the local result
+is uncertain:
+
+```python
+def analyze_news(news: dict) -> dict:
+    try:
+        ml_result = analyze_xauusd_news(news)
+    except Exception:
+        llm_result = analyze_news_with_llm(
+            news["headline"], news.get("description")
+        )
+        llm_result["score_source"] = "llm_model_error_fallback"
+        return llm_result
+
+    ordered_probabilities = sorted(
+        ml_result["probabilities"].values(), reverse=True
+    )
+    probability_margin = ordered_probabilities[0] - ordered_probabilities[1]
+
+    uncertain = (
+        ml_result["confidence"] < 0.70
+        or probability_margin < 0.20
+    )
+
+    if not uncertain:
+        return ml_result
+
+    llm_result = analyze_news_with_llm(
+        news["headline"], news.get("description")
+    )
+    llm_result["score_source"] = "llm_low_confidence_fallback"
+    return llm_result
+```
+
+The `0.70` confidence and `0.20` margin are initial operating thresholds, not
+universal constants. Tune them using results from the project's live news
+stream. Higher thresholds make more LLM calls but reduce reliance on uncertain
+local predictions.
+
+## Input Guidance
+
+Use English RSS-style input matching the training data:
+
+- Provide the original headline.
+- Provide a short description or summary when available.
+- Avoid adding an LLM-generated interpretation to the input.
+- Avoid passing an entire long article as the description.
+- Preserve terms such as `XAUUSD`, `gold`, `USD`, `Fed`, and `Treasury yields`.
+- Reject records where both headline and description are empty.
+
+## Persisting Predictions
+
+Always record how a score was produced. Do not silently mix model and LLM
+scores without a source field.
+
+Recommended fields are:
+
+```text
+article_hash
+target
+label
+score
+confidence
+probabilities_json
+score_source
+model_version
+predicted_at
+```
+
+If the existing `headlines` table must be used, a model result can be mapped as
+follows:
+
+```python
+import json
+import sqlite3
+
+
+prediction = analyze_xauusd_news(news)
+
+with sqlite3.connect("newsdata.db") as database:
+    database.execute(
+        """
+        UPDATE headlines
+        SET score = ?,
+            score_by_currency_json = ?,
+            score_source = ?
+        WHERE hash = ? AND score IS NULL
+        """,
+        (
+            prediction["score"],
+            json.dumps({"XAU": prediction["score"]}),
+            prediction["score_source"],
+            news["hash"],
+        ),
+    )
+```
+
+The `score IS NULL` condition prevents overwriting existing LLM training
+labels. A separate predictions table is preferable when confidence,
+probabilities, prediction history, or multiple model versions must be retained.
+
+## Retraining Workflow
+
+The trainer opens the SQLite database read-only and uses rows with:
+
+```text
+score_source = "llm"
+label = xau_score, else score_by_currency_json["XAU"], else --extra-labels
+source is not a test feed (feed.example, good.example)
+```
+
+Retrain after adding a meaningful amount of new labeled data:
+
+```bash
+.venv/bin/python news_sentiment.py train --db newsdata.db \
+  --extra-labels eval/results/backfill_labels.jsonl
+```
+
+Use `--scope gold` to train only on headlines that mention gold (the pre-2026-09-22
+data), and `--model models/candidates/<name>.joblib` to train a candidate without
+replacing the deployed model. Grade candidates with
+`eval/eval_golden.py --model ...` and
+`eval/eval_archive.py --db ../newsdata.db --extra-labels results/backfill_labels.jsonl`.
+
+Training performs the following steps:
+
+1. Reads XAU sentiment labels (`xau_score`, JSON `XAU`, extra labels).
+2. Normalizes and deduplicates article text.
+3. Maps negative, zero, and positive XAU scores to classes.
+4. Evaluates a stratified random holdout.
+5. Evaluates a chronological holdout on the newest articles.
+6. Fits the final model on all eligible examples.
+7. Writes the model and metrics report.
+
+Before deploying a retrained model, inspect:
+
+```bash
+python -m json.tool models/xauusd_sentiment.metrics.json
+```
+
+Prioritize the chronological holdout result because it better represents
+future news. Do not deploy solely because random holdout accuracy improved.
+
+## Production Rollout
+
+1. Run the model in shadow mode alongside the LLM.
+2. Log disagreements, confidence, latency, and final accepted output.
+3. Review obvious directional cases and low-confidence predictions.
+4. Enable local predictions only above conservative thresholds.
+5. Keep the LLM as a fallback for uncertain or failed predictions.
+6. Add reviewed LLM results to the training set.
+7. Retrain and compare chronological metrics periodically.
+
+Load one model per application process. A process restart is the simplest way
+to load a newly deployed model artifact. Only load trusted `.joblib` files,
+because joblib model files can execute code during deserialization.
+
+## Monitoring
+
+Track at least:
+
+- Number of articles scored
+- Bullish, neutral, and bearish distribution
+- Average confidence and probability margin
+- Percentage routed to the LLM
+- ML versus LLM agreement rate
+- Errors and model-loading failures
+- Results by publisher and news source
+- Current model version
+
+Sudden changes in class distribution or confidence can indicate data drift,
+changed feed formatting, or a new market regime.
+
+## Current Limitations
+
+- Training contains 2,620 deduplicated XAU-labeled examples, all DeepSeek labels.
+- The data covers only 15 days (2026-09-08 to 09-22) of one regime.
+- The model estimates news sentiment, not future returns or trade profitability.
+- Confidence values are useful for routing but are not guaranteed to be fully
+  calibrated probabilities.
+- Unusual wording and economic conditions absent from training can reduce
+  accuracy.
+
+The current chronological holdout accuracy is 66.6% (macro-F1 0.659). Continue using the LLM
+fallback until live shadow-mode results justify broader local-model usage.
+
+## Troubleshooting
+
+### Model not found
+
+Use an absolute model path or set `XAUUSD_MODEL_PATH`:
+
+```bash
+export XAUUSD_MODEL_PATH="/home/harwinder/Development/ML-NEWS/models/xauusd_sentiment.joblib"
+```
+
+### Cannot import `news_sentiment`
+
+Add the repository to `PYTHONPATH` before starting the existing project:
+
+```bash
+export PYTHONPATH="/home/harwinder/Development/ML-NEWS:$PYTHONPATH"
+```
+
+### scikit-learn version warning
+
+Install the pinned dependencies from this repository:
+
+```bash
+python -m pip install -r /home/harwinder/Development/ML-NEWS/requirements.txt
+```
+
+### Empty article error
+
+At least one of `headline` or `description` must contain text.
+
+### Predictions look unrelated to XAUUSD
+
+Confirm that the loaded artifact is `models/xauusd_sentiment.joblib` and that
+the result contains `"target": "XAUUSD"`.