Fix playbook bugs from review, add Telegram alerts to health and service checks
Review fixes: - Availability plays: "ping_result is succeeded" is true for unreachable hosts, so offline hosts were added to reachable_hosts and failed the next play. Test reachability explicitly (7 playbooks). - bootstrap_lxc: vim-enhanced and mtr-tiny package names, enable CRB next to EPEL, add the root key without replacing authorized_keys, validate sshd_config edits and restart sshd only on change. - backup_docker: the command module does not expand $(...); fix rsync filter order and add --exclude=*; restart only the containers that were stopped, even when rsync fails; read the storage address from the inventory; keep semaphore and semaphore-db running. - new_lxc: community.general.proxmox no longer exists, use community.proxmox; node is mini-pc; valid hostname; password or token auth; let the module pick the VMID; longer timeout. - proxmox-backup: surface the vzdump exit code, let vzdump prune with keep-last, drop the rename/prune steps that globbed *.zstd. - time-region: chrony regex no longer comments out its own server line; /etc/localtime as a symlink; reuse facts across plays. - update_pihole_blocklists: FQCN, become: true, changed_when. - ansible.cfg gathering = smart; update_system only pings ubuntu/rocky. Telegram alerts: - health_check: filesystems above disk_warn_pct (default 80), excluding efivarfs, and offline hosts. - service_check: services not running. - One message per run from localhost, credentials from Semaphore (TELEGRAM_BOT_TOKEN / TELEGRAM_CHAT_ID), no_log on the send task, prints the alert text when Telegram is not configured. Claude-Session: https://claude.ai/code/session_01XEizHawueRXUcxQELRZ8kq
This commit is contained in:
@@ -42,20 +42,20 @@ address, the first Ubuntu VM, and the service lists for `webserver`, `bot` and
|
||||
| Playbook | Hosts | Purpose / extra vars |
|
||||
|----------|-------|----------------------|
|
||||
| `ping.yml` | all | connectivity check |
|
||||
| `health_check.yml` | all | disk and RAM report, warns above 85 % disk |
|
||||
| `health_check.yml` | all | disk and RAM report; Telegram alert for filesystems above `disk_warn_pct` (default 80) and offline hosts |
|
||||
| `update_system.yml` | ubuntu, rocky | apt dist-upgrade / dnf update |
|
||||
| `bootstrap_lxc.yml` | ubuntu, rocky | base packages, root SSH key, sshd basics |
|
||||
| `bootstrap_lxc.yml` | ubuntu, rocky | base packages (EPEL and CRB enabled on Rocky), Ansible root key, sshd basics |
|
||||
| `add_root_key.yml` | all | add a personal public key to root |
|
||||
| `time-region.yml` | ubuntu, rocky | timezone, NTP on VMs, NTP disabled in LXC |
|
||||
| `new_lxc.yml` | localhost | create a Rocky container via the Proxmox API |
|
||||
| `backup_docker.yml` | docker | rsync compose files and volumes to storage |
|
||||
| `proxmox-backup.yml` | proxmox | vzdump selected guests, keep last N |
|
||||
| `time-region.yml` | ubuntu, rocky | timezone, NTP on VMs, NTP disabled in LXC. `timezone_region`, `ntp_server` |
|
||||
| `new_lxc.yml` | localhost | create a Rocky container via the Proxmox API. `proxmox_api_user` plus `proxmox_api_password` or `proxmox_api_token_id`/`proxmox_api_token_secret`; `lxc_hostname` |
|
||||
| `backup_docker.yml` | docker | rsync compose files and volumes to storage; containers in `backup_skip_containers` (default semaphore, semaphore-db) stay up |
|
||||
| `proxmox-backup.yml` | proxmox | vzdump selected guests, vzdump keeps the last `keep_last`. `guests`, `backup_mode` |
|
||||
| `discover_guests.yml` | proxmox | list every guest on the host and flag those missing from the inventory; `adopt=true` pushes the Ansible key into unmanaged running containers |
|
||||
| `update_pihole_blocklists.yml` | pihole | `pihole -up` and gravity update |
|
||||
| `ssh_harden.yml` | ubuntu, rocky | sshd drop-in (key-only root login) and fail2ban sshd jail. `ssh_disable_password_auth`, `ssh_max_auth_tries` |
|
||||
| `reboot_check.yml` | ubuntu, rocky | report reboot-required; reboot when `do_reboot=true`, skipping `reboot_exclude` (default `[docker]`) |
|
||||
| `auto_updates.yml` | ubuntu, rocky | unattended-upgrades / dnf-automatic for security updates, no auto reboot |
|
||||
| `service_check.yml` | ubuntu, rocky | check `critical_services` per host; `restart_stopped=true` starts stopped ones |
|
||||
| `service_check.yml` | ubuntu, rocky | check `critical_services` per host; `restart_stopped=true` starts stopped ones; Telegram alert for services not running |
|
||||
| `vm_guest_agent.yml` | vms | install and enable qemu-guest-agent |
|
||||
| `user_setup.yml` | ubuntu, rocky | non-root admin with sudo and SSH keys. `admin_user`, `admin_sudo_nopasswd`, `admin_ssh_keys` |
|
||||
|
||||
@@ -76,6 +76,26 @@ as the first play of anything that touches many hosts, ASCII-only reports.
|
||||
- `auto_updates.yml` and `update_system.yml` overlap. Use the first for
|
||||
hands-off security patches, the second for full upgrades on demand.
|
||||
|
||||
## Telegram alerts
|
||||
|
||||
`health_check.yml` and `service_check.yml` send one Telegram message per run
|
||||
when there is something to report (filesystem above the threshold, host
|
||||
offline, service not running). Without credentials they print the same text
|
||||
in the run log instead, so nothing breaks before the bot exists.
|
||||
|
||||
1. Create a bot with `@BotFather` and copy the token.
|
||||
2. Send the bot a message (or add it to a group), then open
|
||||
`https://api.telegram.org/bot<TOKEN>/getUpdates` and read `chat.id`.
|
||||
3. In Semaphore, create a variable group with the two values as secret
|
||||
environment variables `TELEGRAM_BOT_TOKEN` and `TELEGRAM_CHAT_ID` (extra
|
||||
vars `telegram_bot_token` / `telegram_chat_id` work too) and attach it to
|
||||
the Health check and Service check templates.
|
||||
|
||||
The send task runs with `no_log`, so the token never appears in the run log;
|
||||
a rejected call fails the run with the Telegram error text only. To add the
|
||||
alert to another playbook, copy the final `Telegram alert` play from
|
||||
`health_check.yml` and change the lines the template builds.
|
||||
|
||||
## Semaphore schedules
|
||||
|
||||
One template per row. Cron times are in Semaphore's own timezone (UTC inside
|
||||
@@ -112,9 +132,11 @@ Monthly (drift check, also run after adding a host):
|
||||
On demand only: `bootstrap_lxc.yml`, `add_root_key.yml`, `time-region.yml`,
|
||||
`new_lxc.yml`, `user_setup.yml`, `vm_guest_agent.yml`, `ping.yml`.
|
||||
|
||||
Before enabling: `backup_docker.yml` stops every container on the docker host
|
||||
while it copies volumes. If Semaphore itself runs as a container there, move
|
||||
it or exclude it from the stop step first.
|
||||
`backup_docker.yml` stops the containers on the docker host while it copies
|
||||
volumes, except those in `backup_skip_containers` (default `semaphore` and
|
||||
`semaphore-db`, because Semaphore runs there; their volumes are copied live).
|
||||
`proxmox-backup.yml` has the same trap: never list the guest that runs
|
||||
Semaphore with `backup_mode: stop`, use `snapshot` for it.
|
||||
|
||||
## Adding a new container
|
||||
|
||||
|
||||
Reference in New Issue
Block a user