Harwinder Sekhon
2cc5915b00
Two bodies of work that touch the same files, so one commit. Direction and setup-size campaign (2026-09-30 → 10-01, owner's RISK-FIRST rule, written before the runs): - SIGNAL: M15 daily-trend veto (factory override 7, xau_strategy.daily_trend_veto "both"). A finished setup against a clear daily trend (closed D1 bars, Sunday stubs dropped; close vs its 50-bar EMA AND vs the close 60 kept days earlier) is removed as the last step of the core. One definition (signal_aggregator.daily_trend_series) for live, the bar replay and the twin's host side; D1 is a required frame of the M15 scan; mirrored in cuda_signal_kernels.py K2. W3 824 / +205.12R / PF 1.73 / max DD 12.4R vs 1105 / +195.79R / 1.49 / 20.9R; 2019 → 2026 1473 / +223.33R / 1.42 / 41.5R vs 2091 / +192.53R / 1.24 / 64.8R; unseen holdout −4.62R vs −6.63R. - FAILED, not shipped: the veto on M5 (W3 drawdown 10.1R → 15.2R) and on H1 (−9.7R), buy-only and sell-only on M15, the wide-stop cap (xau_strategy.wide_stop_cap_atr stays 0.0, a research knob). - New M15 pins: W2 dfc647ee…, OOS e87b6de1…, W3 ad3b0dbd…. M5 / H1 / H4 pins unchanged. --set xau_strategy.daily_trend_veto=off reproduces the previous M15 pins byte for byte. Twin and reference script MD5-equal on every window; recorder and feature-export parity re-proved; base-rate table rebuilt. - DISPLAY (books byte-identical): exposure.py — the lot size for trade_size.risk_per_trade (new config.json block, off by default, private to the owner) and the count of alerted same-direction plans still open (signals.alerted_at, read-only open_alerted_setups); the daily trend in words on Telegram, the CLI and the dashboard; signals.daily_trend logged. - providers/oanda: a date window with no bars is an empty page, not an error (a D1 pad starting on a Saturday could not be fetched). - Reports only, nothing shipped: seven context screens (no state nominated) and eight outside data sources (none adds a forecast to gold's own daily trend). - Independent verification (report section 5.13): every figure reproduced on both scripts, scoring identical to the previous commit on all four timeframes, no look-ahead. Caveats recorded there: this is a risk filter, not a more accurate direction call; nine tenths of the 2019 → 2026 gain is 2022; the neighbour clause is thin (a simple 100-day average misses the R line by 1.2R); "never adds a trade" is an observation, not a guarantee. Bug review of the whole codebase (2026-10-01). Signal path clean. 17 defects elsewhere, plus 4 found by an independent review of the fixes; all fixed, each with a test; no trade book changed: - telegram_bot: one _deliver_alert for the candle scan and the level break. The row is stamped alerted_at BEFORE the send, on a one-worker alert-db pool, and un-stamped when no chat received it or the task is cancelled mid-stamp, so two alerts at one candle close count each other and no SQLite write runs on the event loop. The 5-bar cooldown counts bars of the scan's own frame (== gpu_backtest._apply_cooldown; clock time released the 22:00 UTC bar after the settlement pause) and is restored from signals.db at start-up. The lot-size line goes to the owner's own chat only (a group the owner ran /autoscan in used to receive it). /status gains a news-model line and escapes failure details. - signal_log: mark_alerted returns the stamp it wrote; unmark_alerted; last_alerted_bars (read-only). - advisory_outcome: candle continuity is judged bar by bar inside the walk; a hole after a trade's exit no longer voids it. The stored-candle replay resolves 108 of 111 logged setups instead of 99, all equal to the live resolver's R. - providers/oanda: an empty date window is accepted only when it is 5 days or shorter and no malformed candle was skipped; a longer one is retried, then refused, instead of being cached as history with a hole. - news_engine: blank RSS titles skipped; one unreadable headline no longer costs the local model its whole batch (checked on the real bundle); model health recorded; no DeepSeek audit call in a process that does not archive; event-direction cache keyed per release; peek_week_events never fetches. - llm_client: chat_json never raises and never logs the API key (requests quotes the Authorization value in InvalidHeader; error bodies are redacted). - signal_aggregator: the scan's news block is guarded; the replay path (use_news=False) assigns exactly the values it did before. - agent_panel: PANEL_PROMPT_VERSION v2.2 — same prompt text, corrected input (event timing over the cached calendar week, headlines newest first). scripts/panel_replay.py replays the logged payload per persona; scripts/label_scan_outcomes.py counts with an ISO cutoff. - test_all.py: no ALTER TABLE … DROP COLUMN (the live box has SQLite 3.34); tests that write Parquet print a notice and return without pyarrow. Verification after the last code change: suite 632 passed, 0 failed (also with both Parquet engines blocked and FB2_BACKEND=cpu, the live box's view); the five reference MD5s reproduce on both backtesters, 0 cache fallbacks; Module 37 green; static Python 3.9 / SQLite 3.34 check clean on every touched file. Evidence: backtest/research/daily_trend_campaign_2026-09-30.md, context_screens_2026-10.md, new_inputs_2026-10.md; CLAUDE.md sections 2, 4, 6, 8, 9, 10 and 11; pair_signals/xauusd.py docstring (M15 override 7).
175 lines
7.6 KiB
Markdown
175 lines
7.6 KiB
Markdown
# Deploying the web dashboard (live LAMP server)
|
|
|
|
The dashboard is a **static publisher**: `dashboard_publisher.py` runs as its
|
|
own daemon, scans XAU/USD at candle closes, and atomically writes
|
|
`dashboard.html` + `news_summary.php` + `snapshot.json` + `news.json` into a
|
|
folder Apache already serves. Apache does all web serving — no new port, no
|
|
Python web server, and the telegram bot process is completely unaffected by
|
|
anything that happens here (see the module docstring for the independence
|
|
rules). The one dynamic piece is `news_summary.php` (§6): an on-demand
|
|
DeepSeek summary for a clicked headline, whitelist-gated against news.json.
|
|
|
|
All steps below run **on the live server** (`/home/Forex/ForexBot2`).
|
|
|
|
## 1. Get the code
|
|
|
|
```bash
|
|
cd /home/Forex/ForexBot2
|
|
git pull
|
|
venv/bin/python test_all.py # optional sanity: suite must be green
|
|
```
|
|
|
|
Two optional Python packages (`onnxruntime`, `tokenizers`) arrive with the local news model;
|
|
`venv/bin/pip install -r requirements.txt` after the pull. See `deploy/README-news-model.md`,
|
|
which also says when to keep this daemon on DeepSeek (`NEWS_MODEL_MODE=off` in the unit).
|
|
|
|
## 2. Make the web folder writable
|
|
|
|
The dashboard publishes straight into the portal docroot `/var/www/trade/`
|
|
(owner decision 2026-08-03) — it adds exactly six files there:
|
|
`dashboard.html`, `dashboard.css`, `dashboard.js`, `news_summary.php`,
|
|
`snapshot.json`, `news.json` (plus the
|
|
apache-owned `ai/` cache dir from §6). The daemon user must be able to write
|
|
them:
|
|
|
|
```bash
|
|
# if the portal files can be owned by the daemon user:
|
|
sudo chown Forex: /var/www/trade
|
|
# otherwise grant write without changing ownership:
|
|
sudo setfacl -m u:Forex:rwx /var/www/trade
|
|
```
|
|
|
|
(A different path is one edit away: the unit's `Environment=DASHBOARD_OUT_DIR=`
|
|
line, or `--out` for one-off runs.)
|
|
|
|
SELinux (RHEL/Alma/CentOS enforcing): directories under `/var/www` normally
|
|
already carry `httpd_sys_content_t`, and new files inherit the parent label, so
|
|
no action is usually needed. If Apache 403s despite correct permissions:
|
|
|
|
```bash
|
|
sudo restorecon -R /var/www/trade
|
|
```
|
|
|
|
## 3. Smoke-test once
|
|
|
|
```bash
|
|
venv/bin/python dashboard_publisher.py --once
|
|
ls /var/www/trade # dashboard.html/.css/.js news_summary.php snapshot.json news.json added next to the portal
|
|
```
|
|
|
|
Then open `http://<server>/dashboard.html` (path relative to however the
|
|
vhost maps `/var/www/trade`) — all four timeframe cards, the calendar, and
|
|
the newswire should populate.
|
|
|
|
## 4. Install the daemon
|
|
|
|
```bash
|
|
sudo cp deploy/gold-dashboard.service /etc/systemd/system/
|
|
# edit User= if the repo owner is not "Forex"
|
|
sudo systemctl daemon-reload
|
|
sudo systemctl enable --now gold-dashboard
|
|
systemctl status gold-dashboard
|
|
tail -f logs/dashboard.log
|
|
```
|
|
|
|
The daemon scans all four timeframes at boot, then wakes at every M5
|
|
boundary (+10 s) and rescans whichever timeframes just closed. Logs rotate
|
|
every 3 h (240 backups); the bot's own `telegram.log` rotates daily (30
|
|
backups) since 2026-09-08.
|
|
|
|
Since 2026-09-30 each card also shows the daily trend, and a trade plan can
|
|
carry an "Already open" line. That line is one **read-only** query on the
|
|
bot's `logs/signals.db` (`mode=ro`; this process still never writes either
|
|
database), so the daemon user needs read access to that file and its `-wal`
|
|
/ `-shm` companions. If it cannot read them the line is simply absent. The
|
|
lot-size line is NOT published here unless `config.json` sets
|
|
`"trade_size": {"dashboard": true}`: the page is public and the size shows
|
|
the owner's risk per trade. The M15 scan now needs the daily (D1) candles as
|
|
a required frame, so a D1 fetch failure fails that card's scan like any other
|
|
missing frame.
|
|
|
|
## 5. Optional Apache polish
|
|
|
|
The page polls its JSON with conditional GETs (`cache: "no-cache"` → Apache
|
|
answers unchanged files with 304), so no Apache config is strictly needed. To
|
|
be explicit anyway:
|
|
|
|
```apache
|
|
<Directory "/var/www/trade">
|
|
<FilesMatch "\.json$">
|
|
Header set Cache-Control "no-cache"
|
|
</FilesMatch>
|
|
</Directory>
|
|
```
|
|
|
|
The page is deliberately public (owner decision 2026-08-03). To restrict it
|
|
later, add standard Apache basic auth (`htpasswd` + `Require valid-user`) on
|
|
that directory — no code changes needed.
|
|
|
|
## 6. AI summary endpoint (PHP) — one-time setup
|
|
|
|
Clicking a newswire headline opens a detail dialog that fetches an on-demand
|
|
DeepSeek gold-impact summary from `news_summary.php` (installed next to the
|
|
page by the daemon). Owner decision 2026-08-07: a real on-demand endpoint on
|
|
the public page, accepted with these guards baked into the PHP — the id must
|
|
match a headline in the published news.json (arbitrary text can never reach
|
|
the LLM), each headline's summary is cached forever in `ai/` (≤1 API call per
|
|
headline, ever), and a file-based daily cap fails **closed** on I/O errors.
|
|
Since 2026-09-04 the engine publishes only headlines up to 4 h old whose
|
|
score leans one way (neutral items are dropped), so a headline leaves the
|
|
whitelist sooner: a dialog left open past that point returns `unknown_id`
|
|
("This headline has left the live feed"); summaries already cached in `ai/`
|
|
stay reachable.
|
|
|
|
Requirements: `php` + the curl extension (`php -m | grep curl`).
|
|
|
|
```bash
|
|
# 1. SELinux: allow httpd outbound HTTPS — WITHOUT this every summary call
|
|
# dies as a curl timeout even with perfect perms (RHEL-family default).
|
|
sudo setsebool -P httpd_can_network_connect 1
|
|
|
|
# 2. API key OUTSIDE the docroot, readable by Apache only:
|
|
sudo install -d -o root -g root -m 755 /etc/gold-dashboard
|
|
sudo sh -c 'umask 137; echo "sk-..." > /etc/gold-dashboard/deepseek.key'
|
|
sudo chown root:apache /etc/gold-dashboard/deepseek.key # mode 640
|
|
|
|
# 3. apache-owned cache dir (the daemon runs as Forex and can't create it):
|
|
sudo install -d -o apache -g apache -m 755 /var/www/trade/ai
|
|
sudo semanage fcontext -a -t httpd_sys_rw_content_t '/var/www/trade/ai(/.*)?'
|
|
sudo restorecon -Rv /var/www/trade/ai
|
|
```
|
|
|
|
Knobs (Apache `SetEnv` in the vhost, both optional):
|
|
`AI_SUMMARY_DAILY_CAP` (default 150), `DEEPSEEK_KEY_FILE` (default
|
|
`/etc/gold-dashboard/deepseek.key`).
|
|
|
|
Verify from the shell — expect 400 / 404 / 200 / 200-from-cache:
|
|
|
|
```bash
|
|
curl -s "http://<server>/news_summary.php?id=zzz" # {"error":"bad_id"}
|
|
curl -s "http://<server>/news_summary.php?id=$(printf 'a%.0s' {1..40})" # unknown_id
|
|
ID=$(python3 -c "import json;print(json.load(open('/var/www/trade/news.json'))['headlines'][0]['id'])")
|
|
curl -s "http://<server>/news_summary.php?id=$ID" # real summary (2-12 s)
|
|
curl -s "http://<server>/news_summary.php?id=$ID" # instant, from ai/<id>.json
|
|
```
|
|
|
|
## Operational notes
|
|
|
|
- **Updating**: `git pull` + `systemctl restart gold-dashboard`. The daemon
|
|
re-copies `static/dashboard.html`, `dashboard.css`, `dashboard.js` and
|
|
`news_summary.php` into the web folder at startup.
|
|
- **If the daemon dies**, systemd restarts it in 30 s; worst case the page
|
|
shows stale data with its own "updated N min ago" evidence. Apache and the
|
|
telegram bot are unaffected.
|
|
- **DeepSeek**: the publisher process has its own daily request cap
|
|
(`deepseek.daily_request_cap`, default 500) separate from the bot's —
|
|
expect roughly up to ~430 extra calls/day worst case, V4-Flash pricing.
|
|
- **Two independent DeepSeek meters** hit the same API key: the publisher's
|
|
in-process cap above, and the PHP endpoint's file cap (`ai/.ht_cap.json`,
|
|
default 150/day). They don't communicate — worst-case spend is their sum;
|
|
DeepSeek-side spend alerting is the true backstop. Kept separate on
|
|
purpose (independence rule: the PHP never imports/reads bot code or
|
|
config.json).
|
|
- **Credentials**: uses the same env/.env/config.json resolution as the rest
|
|
of the platform (Oanda candles feed; news needs no key; DeepSeek optional).
|