News-ML-Decoder/WORKFLOW_AND_USAGE.md

# XAUUSD Sentiment Workflow and Usage Guide

## Purpose

This model answers one question:

> Does a news headline imply bullish, neutral, or bearish sentiment for
> XAUUSD?

- **Bullish** means the news supports a higher XAUUSD price.
- **Bearish** means the news supports a lower XAUUSD price.
- **Neutral** means the direction is unclear, weak, or conflicting.

The model runs locally with scikit-learn. It does not require an API call and
can be placed before an existing LLM news-analysis step to reduce latency and
LLM usage.

## Files

| Path | Purpose |
| --- | --- |
| `news_sentiment.py` | Training, scoring, and Python API |
| `newsdata.db` | SQLite training data |
| `models/xauusd_sentiment.joblib` | Trained model artifact |
| `models/xauusd_sentiment.metrics.json` | Training and evaluation metrics |
| `requirements.txt` | Runtime dependencies |

## Recommended Workflow

```mermaid
flowchart LR
    A[New article] --> B[Validate headline and description]
    B --> C[XAUUSD local model]
    C --> D{Prediction confident?}
    D -->|Yes| E[Use local prediction]
    D -->|No| F[Call existing LLM]
    E --> G[Store result and source]
    F --> G
    G --> H[Retain reviewed or LLM labels]
    H --> I[Periodic model retraining]
```

Start in **shadow mode**: run the local model and LLM together, but continue
using the LLM result. Log both outputs for comparison. After validating the
model on live news, enable the confidence-based LLM fallback workflow.

## Environment Setup

From this repository:

```bash
python3 -m venv .venv
. .venv/bin/activate
python -m pip install -r requirements.txt
```

The saved model was built with the exact versions in `requirements.txt`.

If the existing Python project is in another directory, make this repository
importable when starting it:

```bash
export PYTHONPATH="/home/harwinder/Development/ML-NEWS:$PYTHONPATH"
export XAUUSD_MODEL_PATH="/home/harwinder/Development/ML-NEWS/models/xauusd_sentiment.joblib"
python /path/to/existing/project/main.py
```

Add the same environment variables to the service manager, Docker container,
or deployment configuration used in production.

## Command-Line Usage

Score one headline:

```bash
.venv/bin/python news_sentiment.py score \
  --headline "Gold gains as the dollar weakens"
```

Score a headline and description:

```bash
.venv/bin/python news_sentiment.py score \
  --headline "Gold slides as Treasury yields rise" \
  --description "A stronger US dollar adds pressure to bullion."
```

The output has this structure:

```json
{
  "target": "XAUUSD",
  "label": "bearish",
  "score": -0.82,
  "confidence": 0.87,
  "probabilities": {
    "bearish": 0.87,
    "neutral": 0.08,
    "bullish": 0.05
  }
}
```

The score is calculated as:

```text
P(XAUUSD bullish) - P(XAUUSD bearish)
```

Use `label` as the primary classification. A neutral prediction can still have
a positive or negative score when the model sees a secondary directional
lean.

## Batch JSONL Usage

Input records must be newline-delimited JSON. Each record can contain
`headline`, `title`, or `text`, plus an optional `description`:

```jsonl
{"id": 1, "headline": "Gold rises as the dollar falls"}
{"id": 2, "headline": "Gold holds steady ahead of CPI"}
{"id": 3, "title": "Gold falls as Treasury yields climb"}
```

Score the file:

```bash
.venv/bin/python news_sentiment.py score --input articles.jsonl > scored.jsonl
```

Each output record preserves the input fields and adds a `sentiment` object.

## Python Integration

Load the model once when the application process starts. Do not load it for
every article.

```python
import os
from pathlib import Path

from news_sentiment import load_model, score_news


MODEL_PATH = Path(
    os.environ.get(
        "XAUUSD_MODEL_PATH",
        "/home/harwinder/Development/ML-NEWS/models/xauusd_sentiment.joblib",
    )
)

XAUUSD_MODEL = load_model(MODEL_PATH)


def analyze_xauusd_news(news: dict) -> dict:
    prediction = score_news(
        XAUUSD_MODEL,
        headline=news["headline"],
        description=news.get("description"),
    )

    return {
        "target": prediction["target"],
        "sentiment": prediction["label"],
        "score": prediction["score"],
        "confidence": prediction["confidence"],
        "probabilities": prediction["probabilities"],
        "score_source": "xauusd_sklearn_v3",
    }
```

Example:

```python
news = {
    "headline": "Gold rallies as the US dollar weakens",
    "description": "Falling Treasury yields provided additional support.",
}

result = analyze_xauusd_news(news)
print(result)
```

## Connecting to the Existing LLM Workflow

Assume the existing project currently calls:

```python
result = analyze_news_with_llm(headline, description)
```

For a full local replacement:

```python
result = analyze_xauusd_news(
    {
        "headline": headline,
        "description": description,
    }
)
```

For the recommended hybrid workflow, call the LLM only when the local result
is uncertain:

```python
def analyze_news(news: dict) -> dict:
    try:
        ml_result = analyze_xauusd_news(news)
    except Exception:
        llm_result = analyze_news_with_llm(
            news["headline"], news.get("description")
        )
        llm_result["score_source"] = "llm_model_error_fallback"
        return llm_result

    ordered_probabilities = sorted(
        ml_result["probabilities"].values(), reverse=True
    )
    probability_margin = ordered_probabilities[0] - ordered_probabilities[1]

    uncertain = (
        ml_result["confidence"] < 0.70
        or probability_margin < 0.20
    )

    if not uncertain:
        return ml_result

    llm_result = analyze_news_with_llm(
        news["headline"], news.get("description")
    )
    llm_result["score_source"] = "llm_low_confidence_fallback"
    return llm_result
```

The `0.70` confidence and `0.20` margin are initial operating thresholds, not
universal constants. Tune them using results from the project's live news
stream. Higher thresholds make more LLM calls but reduce reliance on uncertain
local predictions.

## Input Guidance

Use English RSS-style input matching the training data:

- Provide the original headline.
- Provide a short description or summary when available.
- Avoid adding an LLM-generated interpretation to the input.
- Avoid passing an entire long article as the description.
- Preserve terms such as `XAUUSD`, `gold`, `USD`, `Fed`, and `Treasury yields`.
- Reject records where both headline and description are empty.

## Persisting Predictions

Always record how a score was produced. Do not silently mix model and LLM
scores without a source field.

Recommended fields are:

```text
article_hash
target
label
score
confidence
probabilities_json
score_source
model_version
predicted_at
```

If the existing `headlines` table must be used, a model result can be mapped as
follows:

```python
import json
import sqlite3


prediction = analyze_xauusd_news(news)

with sqlite3.connect("newsdata.db") as database:
    database.execute(
        """
        UPDATE headlines
        SET score = ?,
            score_by_currency_json = ?,
            score_source = ?
        WHERE hash = ? AND score IS NULL
        """,
        (
            prediction["score"],
            json.dumps({"XAU": prediction["score"]}),
            prediction["score_source"],
            news["hash"],
        ),
    )
```

The `score IS NULL` condition prevents overwriting existing LLM training
labels. A separate predictions table is preferable when confidence,
probabilities, prediction history, or multiple model versions must be retained.

## Retraining Workflow

The trainer opens the SQLite database read-only and uses rows with:

```text
score_source = "llm"
label = xau_score, else score_by_currency_json["XAU"], else --extra-labels
source is not a test feed (feed.example, good.example)
```

Retrain after adding a meaningful amount of new labeled data:

```bash
.venv/bin/python news_sentiment.py train --db newsdata.db \
  --extra-labels eval/results/backfill_labels.jsonl
```

Use `--scope gold` to train only on headlines that mention gold (the pre-2026-09-22
data), and `--model models/candidates/<name>.joblib` to train a candidate without
replacing the deployed model. Grade candidates with
`eval/eval_golden.py --model ...` and
`eval/eval_archive.py --db ../newsdata.db --extra-labels results/backfill_labels.jsonl`.

Training performs the following steps:

1. Reads XAU sentiment labels (`xau_score`, JSON `XAU`, extra labels).
2. Normalizes and deduplicates article text.
3. Maps negative, zero, and positive XAU scores to classes.
4. Evaluates a stratified random holdout.
5. Evaluates a chronological holdout on the newest articles.
6. Fits the final model on all eligible examples.
7. Writes the model and metrics report.

Before deploying a retrained model, inspect:

```bash
python -m json.tool models/xauusd_sentiment.metrics.json
```

Prioritize the chronological holdout result because it better represents
future news. Do not deploy solely because random holdout accuracy improved.

## Production Rollout

1. Run the model in shadow mode alongside the LLM.
2. Log disagreements, confidence, latency, and final accepted output.
3. Review obvious directional cases and low-confidence predictions.
4. Enable local predictions only above conservative thresholds.
5. Keep the LLM as a fallback for uncertain or failed predictions.
6. Add reviewed LLM results to the training set.
7. Retrain and compare chronological metrics periodically.

Load one model per application process. A process restart is the simplest way
to load a newly deployed model artifact. Only load trusted `.joblib` files,
because joblib model files can execute code during deserialization.

## Monitoring

Track at least:

- Number of articles scored
- Bullish, neutral, and bearish distribution
- Average confidence and probability margin
- Percentage routed to the LLM
- ML versus LLM agreement rate
- Errors and model-loading failures
- Results by publisher and news source
- Current model version

Sudden changes in class distribution or confidence can indicate data drift,
changed feed formatting, or a new market regime.

## Current Limitations

- Training contains 2,620 deduplicated XAU-labeled examples, all DeepSeek labels.
- The data covers only 15 days (2026-09-08 to 09-22) of one regime.
- The model estimates news sentiment, not future returns or trade profitability.
- Confidence values are useful for routing but are not guaranteed to be fully
  calibrated probabilities.
- Unusual wording and economic conditions absent from training can reduce
  accuracy.

The current chronological holdout accuracy is 66.6% (macro-F1 0.659). Continue using the LLM
fallback until live shadow-mode results justify broader local-model usage.

## Troubleshooting

### Model not found

Use an absolute model path or set `XAUUSD_MODEL_PATH`:

```bash
export XAUUSD_MODEL_PATH="/home/harwinder/Development/ML-NEWS/models/xauusd_sentiment.joblib"
```

### Cannot import `news_sentiment`

Add the repository to `PYTHONPATH` before starting the existing project:

```bash
export PYTHONPATH="/home/harwinder/Development/ML-NEWS:$PYTHONPATH"
```

### scikit-learn version warning

Install the pinned dependencies from this repository:

```bash
python -m pip install -r /home/harwinder/Development/ML-NEWS/requirements.txt
```

### Empty article error

At least one of `headline` or `description` must contain text.

### Predictions look unrelated to XAUUSD

Confirm that the loaded artifact is `models/xauusd_sentiment.joblib` and that
the result contains `"target": "XAUUSD"`.