Harwinder Sekhon
2cc5915b00
Two bodies of work that touch the same files, so one commit. Direction and setup-size campaign (2026-09-30 → 10-01, owner's RISK-FIRST rule, written before the runs): - SIGNAL: M15 daily-trend veto (factory override 7, xau_strategy.daily_trend_veto "both"). A finished setup against a clear daily trend (closed D1 bars, Sunday stubs dropped; close vs its 50-bar EMA AND vs the close 60 kept days earlier) is removed as the last step of the core. One definition (signal_aggregator.daily_trend_series) for live, the bar replay and the twin's host side; D1 is a required frame of the M15 scan; mirrored in cuda_signal_kernels.py K2. W3 824 / +205.12R / PF 1.73 / max DD 12.4R vs 1105 / +195.79R / 1.49 / 20.9R; 2019 → 2026 1473 / +223.33R / 1.42 / 41.5R vs 2091 / +192.53R / 1.24 / 64.8R; unseen holdout −4.62R vs −6.63R. - FAILED, not shipped: the veto on M5 (W3 drawdown 10.1R → 15.2R) and on H1 (−9.7R), buy-only and sell-only on M15, the wide-stop cap (xau_strategy.wide_stop_cap_atr stays 0.0, a research knob). - New M15 pins: W2 dfc647ee…, OOS e87b6de1…, W3 ad3b0dbd…. M5 / H1 / H4 pins unchanged. --set xau_strategy.daily_trend_veto=off reproduces the previous M15 pins byte for byte. Twin and reference script MD5-equal on every window; recorder and feature-export parity re-proved; base-rate table rebuilt. - DISPLAY (books byte-identical): exposure.py — the lot size for trade_size.risk_per_trade (new config.json block, off by default, private to the owner) and the count of alerted same-direction plans still open (signals.alerted_at, read-only open_alerted_setups); the daily trend in words on Telegram, the CLI and the dashboard; signals.daily_trend logged. - providers/oanda: a date window with no bars is an empty page, not an error (a D1 pad starting on a Saturday could not be fetched). - Reports only, nothing shipped: seven context screens (no state nominated) and eight outside data sources (none adds a forecast to gold's own daily trend). - Independent verification (report section 5.13): every figure reproduced on both scripts, scoring identical to the previous commit on all four timeframes, no look-ahead. Caveats recorded there: this is a risk filter, not a more accurate direction call; nine tenths of the 2019 → 2026 gain is 2022; the neighbour clause is thin (a simple 100-day average misses the R line by 1.2R); "never adds a trade" is an observation, not a guarantee. Bug review of the whole codebase (2026-10-01). Signal path clean. 17 defects elsewhere, plus 4 found by an independent review of the fixes; all fixed, each with a test; no trade book changed: - telegram_bot: one _deliver_alert for the candle scan and the level break. The row is stamped alerted_at BEFORE the send, on a one-worker alert-db pool, and un-stamped when no chat received it or the task is cancelled mid-stamp, so two alerts at one candle close count each other and no SQLite write runs on the event loop. The 5-bar cooldown counts bars of the scan's own frame (== gpu_backtest._apply_cooldown; clock time released the 22:00 UTC bar after the settlement pause) and is restored from signals.db at start-up. The lot-size line goes to the owner's own chat only (a group the owner ran /autoscan in used to receive it). /status gains a news-model line and escapes failure details. - signal_log: mark_alerted returns the stamp it wrote; unmark_alerted; last_alerted_bars (read-only). - advisory_outcome: candle continuity is judged bar by bar inside the walk; a hole after a trade's exit no longer voids it. The stored-candle replay resolves 108 of 111 logged setups instead of 99, all equal to the live resolver's R. - providers/oanda: an empty date window is accepted only when it is 5 days or shorter and no malformed candle was skipped; a longer one is retried, then refused, instead of being cached as history with a hole. - news_engine: blank RSS titles skipped; one unreadable headline no longer costs the local model its whole batch (checked on the real bundle); model health recorded; no DeepSeek audit call in a process that does not archive; event-direction cache keyed per release; peek_week_events never fetches. - llm_client: chat_json never raises and never logs the API key (requests quotes the Authorization value in InvalidHeader; error bodies are redacted). - signal_aggregator: the scan's news block is guarded; the replay path (use_news=False) assigns exactly the values it did before. - agent_panel: PANEL_PROMPT_VERSION v2.2 — same prompt text, corrected input (event timing over the cached calendar week, headlines newest first). scripts/panel_replay.py replays the logged payload per persona; scripts/label_scan_outcomes.py counts with an ISO cutoff. - test_all.py: no ALTER TABLE … DROP COLUMN (the live box has SQLite 3.34); tests that write Parquet print a notice and return without pyarrow. Verification after the last code change: suite 632 passed, 0 failed (also with both Parquet engines blocked and FB2_BACKEND=cpu, the live box's view); the five reference MD5s reproduce on both backtesters, 0 cache fallbacks; Module 37 green; static Python 3.9 / SQLite 3.34 check clean on every touched file. Evidence: backtest/research/daily_trend_campaign_2026-09-30.md, context_screens_2026-10.md, new_inputs_2026-10.md; CLAUDE.md sections 2, 4, 6, 8, 9, 10 and 11; pair_signals/xauusd.py docstring (M15 override 7).
7.6 KiB
Deploying the web dashboard (live LAMP server)
The dashboard is a static publisher: dashboard_publisher.py runs as its
own daemon, scans XAU/USD at candle closes, and atomically writes
dashboard.html + news_summary.php + snapshot.json + news.json into a
folder Apache already serves. Apache does all web serving — no new port, no
Python web server, and the telegram bot process is completely unaffected by
anything that happens here (see the module docstring for the independence
rules). The one dynamic piece is news_summary.php (§6): an on-demand
DeepSeek summary for a clicked headline, whitelist-gated against news.json.
All steps below run on the live server (/home/Forex/ForexBot2).
1. Get the code
cd /home/Forex/ForexBot2
git pull
venv/bin/python test_all.py # optional sanity: suite must be green
Two optional Python packages (onnxruntime, tokenizers) arrive with the local news model;
venv/bin/pip install -r requirements.txt after the pull. See deploy/README-news-model.md,
which also says when to keep this daemon on DeepSeek (NEWS_MODEL_MODE=off in the unit).
2. Make the web folder writable
The dashboard publishes straight into the portal docroot /var/www/trade/
(owner decision 2026-08-03) — it adds exactly six files there:
dashboard.html, dashboard.css, dashboard.js, news_summary.php,
snapshot.json, news.json (plus the
apache-owned ai/ cache dir from §6). The daemon user must be able to write
them:
# if the portal files can be owned by the daemon user:
sudo chown Forex: /var/www/trade
# otherwise grant write without changing ownership:
sudo setfacl -m u:Forex:rwx /var/www/trade
(A different path is one edit away: the unit's Environment=DASHBOARD_OUT_DIR=
line, or --out for one-off runs.)
SELinux (RHEL/Alma/CentOS enforcing): directories under /var/www normally
already carry httpd_sys_content_t, and new files inherit the parent label, so
no action is usually needed. If Apache 403s despite correct permissions:
sudo restorecon -R /var/www/trade
3. Smoke-test once
venv/bin/python dashboard_publisher.py --once
ls /var/www/trade # dashboard.html/.css/.js news_summary.php snapshot.json news.json added next to the portal
Then open http://<server>/dashboard.html (path relative to however the
vhost maps /var/www/trade) — all four timeframe cards, the calendar, and
the newswire should populate.
4. Install the daemon
sudo cp deploy/gold-dashboard.service /etc/systemd/system/
# edit User= if the repo owner is not "Forex"
sudo systemctl daemon-reload
sudo systemctl enable --now gold-dashboard
systemctl status gold-dashboard
tail -f logs/dashboard.log
The daemon scans all four timeframes at boot, then wakes at every M5
boundary (+10 s) and rescans whichever timeframes just closed. Logs rotate
every 3 h (240 backups); the bot's own telegram.log rotates daily (30
backups) since 2026-09-08.
Since 2026-09-30 each card also shows the daily trend, and a trade plan can
carry an "Already open" line. That line is one read-only query on the
bot's logs/signals.db (mode=ro; this process still never writes either
database), so the daemon user needs read access to that file and its -wal
/ -shm companions. If it cannot read them the line is simply absent. The
lot-size line is NOT published here unless config.json sets
"trade_size": {"dashboard": true}: the page is public and the size shows
the owner's risk per trade. The M15 scan now needs the daily (D1) candles as
a required frame, so a D1 fetch failure fails that card's scan like any other
missing frame.
5. Optional Apache polish
The page polls its JSON with conditional GETs (cache: "no-cache" → Apache
answers unchanged files with 304), so no Apache config is strictly needed. To
be explicit anyway:
<Directory "/var/www/trade">
<FilesMatch "\.json$">
Header set Cache-Control "no-cache"
</FilesMatch>
</Directory>
The page is deliberately public (owner decision 2026-08-03). To restrict it
later, add standard Apache basic auth (htpasswd + Require valid-user) on
that directory — no code changes needed.
6. AI summary endpoint (PHP) — one-time setup
Clicking a newswire headline opens a detail dialog that fetches an on-demand
DeepSeek gold-impact summary from news_summary.php (installed next to the
page by the daemon). Owner decision 2026-08-07: a real on-demand endpoint on
the public page, accepted with these guards baked into the PHP — the id must
match a headline in the published news.json (arbitrary text can never reach
the LLM), each headline's summary is cached forever in ai/ (≤1 API call per
headline, ever), and a file-based daily cap fails closed on I/O errors.
Since 2026-09-04 the engine publishes only headlines up to 4 h old whose
score leans one way (neutral items are dropped), so a headline leaves the
whitelist sooner: a dialog left open past that point returns unknown_id
("This headline has left the live feed"); summaries already cached in ai/
stay reachable.
Requirements: php + the curl extension (php -m | grep curl).
# 1. SELinux: allow httpd outbound HTTPS — WITHOUT this every summary call
# dies as a curl timeout even with perfect perms (RHEL-family default).
sudo setsebool -P httpd_can_network_connect 1
# 2. API key OUTSIDE the docroot, readable by Apache only:
sudo install -d -o root -g root -m 755 /etc/gold-dashboard
sudo sh -c 'umask 137; echo "sk-..." > /etc/gold-dashboard/deepseek.key'
sudo chown root:apache /etc/gold-dashboard/deepseek.key # mode 640
# 3. apache-owned cache dir (the daemon runs as Forex and can't create it):
sudo install -d -o apache -g apache -m 755 /var/www/trade/ai
sudo semanage fcontext -a -t httpd_sys_rw_content_t '/var/www/trade/ai(/.*)?'
sudo restorecon -Rv /var/www/trade/ai
Knobs (Apache SetEnv in the vhost, both optional):
AI_SUMMARY_DAILY_CAP (default 150), DEEPSEEK_KEY_FILE (default
/etc/gold-dashboard/deepseek.key).
Verify from the shell — expect 400 / 404 / 200 / 200-from-cache:
curl -s "http://<server>/news_summary.php?id=zzz" # {"error":"bad_id"}
curl -s "http://<server>/news_summary.php?id=$(printf 'a%.0s' {1..40})" # unknown_id
ID=$(python3 -c "import json;print(json.load(open('/var/www/trade/news.json'))['headlines'][0]['id'])")
curl -s "http://<server>/news_summary.php?id=$ID" # real summary (2-12 s)
curl -s "http://<server>/news_summary.php?id=$ID" # instant, from ai/<id>.json
Operational notes
- Updating:
git pull+systemctl restart gold-dashboard. The daemon re-copiesstatic/dashboard.html,dashboard.css,dashboard.jsandnews_summary.phpinto the web folder at startup. - If the daemon dies, systemd restarts it in 30 s; worst case the page shows stale data with its own "updated N min ago" evidence. Apache and the telegram bot are unaffected.
- DeepSeek: the publisher process has its own daily request cap
(
deepseek.daily_request_cap, default 500) separate from the bot's — expect roughly up to ~430 extra calls/day worst case, V4-Flash pricing. - Two independent DeepSeek meters hit the same API key: the publisher's
in-process cap above, and the PHP endpoint's file cap (
ai/.ht_cap.json, default 150/day). They don't communicate — worst-case spend is their sum; DeepSeek-side spend alerting is the true backstop. Kept separate on purpose (independence rule: the PHP never imports/reads bot code or config.json). - Credentials: uses the same env/.env/config.json resolution as the rest of the platform (Oanda candles feed; news needs no key; DeepSeek optional).