Fix playbook bugs from review, add Telegram alerts to health and service checks

Review fixes:
- Availability plays: "ping_result is succeeded" is true for unreachable
  hosts, so offline hosts were added to reachable_hosts and failed the
  next play. Test reachability explicitly (7 playbooks).
- bootstrap_lxc: vim-enhanced and mtr-tiny package names, enable CRB next
  to EPEL, add the root key without replacing authorized_keys, validate
  sshd_config edits and restart sshd only on change.
- backup_docker: the command module does not expand $(...); fix rsync
  filter order and add --exclude=*; restart only the containers that were
  stopped, even when rsync fails; read the storage address from the
  inventory; keep semaphore and semaphore-db running.
- new_lxc: community.general.proxmox no longer exists, use
  community.proxmox; node is mini-pc; valid hostname; password or token
  auth; let the module pick the VMID; longer timeout.
- proxmox-backup: surface the vzdump exit code, let vzdump prune with
  keep-last, drop the rename/prune steps that globbed *.zstd.
- time-region: chrony regex no longer comments out its own server line;
  /etc/localtime as a symlink; reuse facts across plays.
- update_pihole_blocklists: FQCN, become: true, changed_when.
- ansible.cfg gathering = smart; update_system only pings ubuntu/rocky.

Telegram alerts:
- health_check: filesystems above disk_warn_pct (default 80), excluding
  efivarfs, and offline hosts.
- service_check: services not running.
- One message per run from localhost, credentials from Semaphore
  (TELEGRAM_BOT_TOKEN / TELEGRAM_CHAT_ID), no_log on the send task,
  prints the alert text when Telegram is not configured.

Claude-Session: https://claude.ai/code/session_01XEizHawueRXUcxQELRZ8kq
This commit is contained in:
Harwinder Sekhon
2026-09-06 23:06:18 -04:00
parent 0b7e136b24
commit 5aaa7020ac
21 changed files with 515 additions and 259 deletions
+32 -10
View File
@@ -42,20 +42,20 @@ address, the first Ubuntu VM, and the service lists for `webserver`, `bot` and
| Playbook | Hosts | Purpose / extra vars |
|----------|-------|----------------------|
| `ping.yml` | all | connectivity check |
| `health_check.yml` | all | disk and RAM report, warns above 85 % disk |
| `health_check.yml` | all | disk and RAM report; Telegram alert for filesystems above `disk_warn_pct` (default 80) and offline hosts |
| `update_system.yml` | ubuntu, rocky | apt dist-upgrade / dnf update |
| `bootstrap_lxc.yml` | ubuntu, rocky | base packages, root SSH key, sshd basics |
| `bootstrap_lxc.yml` | ubuntu, rocky | base packages (EPEL and CRB enabled on Rocky), Ansible root key, sshd basics |
| `add_root_key.yml` | all | add a personal public key to root |
| `time-region.yml` | ubuntu, rocky | timezone, NTP on VMs, NTP disabled in LXC |
| `new_lxc.yml` | localhost | create a Rocky container via the Proxmox API |
| `backup_docker.yml` | docker | rsync compose files and volumes to storage |
| `proxmox-backup.yml` | proxmox | vzdump selected guests, keep last N |
| `time-region.yml` | ubuntu, rocky | timezone, NTP on VMs, NTP disabled in LXC. `timezone_region`, `ntp_server` |
| `new_lxc.yml` | localhost | create a Rocky container via the Proxmox API. `proxmox_api_user` plus `proxmox_api_password` or `proxmox_api_token_id`/`proxmox_api_token_secret`; `lxc_hostname` |
| `backup_docker.yml` | docker | rsync compose files and volumes to storage; containers in `backup_skip_containers` (default semaphore, semaphore-db) stay up |
| `proxmox-backup.yml` | proxmox | vzdump selected guests, vzdump keeps the last `keep_last`. `guests`, `backup_mode` |
| `discover_guests.yml` | proxmox | list every guest on the host and flag those missing from the inventory; `adopt=true` pushes the Ansible key into unmanaged running containers |
| `update_pihole_blocklists.yml` | pihole | `pihole -up` and gravity update |
| `ssh_harden.yml` | ubuntu, rocky | sshd drop-in (key-only root login) and fail2ban sshd jail. `ssh_disable_password_auth`, `ssh_max_auth_tries` |
| `reboot_check.yml` | ubuntu, rocky | report reboot-required; reboot when `do_reboot=true`, skipping `reboot_exclude` (default `[docker]`) |
| `auto_updates.yml` | ubuntu, rocky | unattended-upgrades / dnf-automatic for security updates, no auto reboot |
| `service_check.yml` | ubuntu, rocky | check `critical_services` per host; `restart_stopped=true` starts stopped ones |
| `service_check.yml` | ubuntu, rocky | check `critical_services` per host; `restart_stopped=true` starts stopped ones; Telegram alert for services not running |
| `vm_guest_agent.yml` | vms | install and enable qemu-guest-agent |
| `user_setup.yml` | ubuntu, rocky | non-root admin with sudo and SSH keys. `admin_user`, `admin_sudo_nopasswd`, `admin_ssh_keys` |
@@ -76,6 +76,26 @@ as the first play of anything that touches many hosts, ASCII-only reports.
- `auto_updates.yml` and `update_system.yml` overlap. Use the first for
hands-off security patches, the second for full upgrades on demand.
## Telegram alerts
`health_check.yml` and `service_check.yml` send one Telegram message per run
when there is something to report (filesystem above the threshold, host
offline, service not running). Without credentials they print the same text
in the run log instead, so nothing breaks before the bot exists.
1. Create a bot with `@BotFather` and copy the token.
2. Send the bot a message (or add it to a group), then open
`https://api.telegram.org/bot<TOKEN>/getUpdates` and read `chat.id`.
3. In Semaphore, create a variable group with the two values as secret
environment variables `TELEGRAM_BOT_TOKEN` and `TELEGRAM_CHAT_ID` (extra
vars `telegram_bot_token` / `telegram_chat_id` work too) and attach it to
the Health check and Service check templates.
The send task runs with `no_log`, so the token never appears in the run log;
a rejected call fails the run with the Telegram error text only. To add the
alert to another playbook, copy the final `Telegram alert` play from
`health_check.yml` and change the lines the template builds.
## Semaphore schedules
One template per row. Cron times are in Semaphore's own timezone (UTC inside
@@ -112,9 +132,11 @@ Monthly (drift check, also run after adding a host):
On demand only: `bootstrap_lxc.yml`, `add_root_key.yml`, `time-region.yml`,
`new_lxc.yml`, `user_setup.yml`, `vm_guest_agent.yml`, `ping.yml`.
Before enabling: `backup_docker.yml` stops every container on the docker host
while it copies volumes. If Semaphore itself runs as a container there, move
it or exclude it from the stop step first.
`backup_docker.yml` stops the containers on the docker host while it copies
volumes, except those in `backup_skip_containers` (default `semaphore` and
`semaphore-db`, because Semaphore runs there; their volumes are copied live).
`proxmox-backup.yml` has the same trap: never list the guest that runs
Semaphore with `backup_mode: stop`, use `snapshot` for it.
## Adding a new container