ansible/README.md

# Homelab Ansible

Playbooks for a Proxmox homelab: Rocky Linux 9 and Ubuntu LXC containers,
Ubuntu VMs and the Proxmox host itself. Everything is run unattended from
Semaphore.

## Layout

```
ansible.cfg                  defaults for local runs (Semaphore passes its own inventory/key)
collections/requirements.yml collections Semaphore installs before each run
inventories/all.yml          the homelab inventory (groups below)
inventories/new-lxc.yml      throwaway inventory for a freshly created container
playbooks/*.yml              one self-contained playbook per task, no roles
```

## Inventory groups

| Group     | Members                     | Notes |
|-----------|-----------------------------|-------|
| `ubuntu`  | Ubuntu containers and VMs   | apt plays |
| `rocky`   | Rocky Linux 9 containers    | dnf plays |
| `proxmox` | `pve`                       | the Proxmox host |
| `lxc`     | containers only             | subset of ubuntu/rocky |
| `vms`     | QEMU VMs only               | subset of ubuntu/rocky, currently empty |

A host may sit in several groups. `ansible_host` is defined once under the OS
group. `critical_services` on a host entry feeds `service_check.yml`.

Placeholders to fill in (`grep PLACEHOLDER inventories/all.yml`): the Proxmox
address, the first Ubuntu VM, and the service lists for `webserver`, `bot` and
`storage`.

## Semaphore

- Inventory type File, path `inventories/all.yml`.
- Collections in `collections/requirements.yml` are installed automatically.
- One task template per playbook. Parameters go in as extra vars (JSON).

## Playbooks

| Playbook | Hosts | Purpose / extra vars |
|----------|-------|----------------------|
| `ping.yml` | all | connectivity check |
| `health_check.yml` | all | disk and RAM report; Telegram alert for filesystems above `disk_warn_pct` (default 80) and offline hosts |
| `update_system.yml` | ubuntu, rocky | apt dist-upgrade / dnf update |
| `bootstrap_lxc.yml` | ubuntu, rocky | base packages (EPEL and CRB enabled on Rocky), Ansible root key, sshd basics |
| `add_root_key.yml` | all | add a personal public key to root |
| `time-region.yml` | ubuntu, rocky | timezone, NTP on VMs, NTP disabled in LXC. `timezone_region`, `ntp_server` |
| `new_lxc.yml` | localhost | create a Rocky container via the Proxmox API. `proxmox_api_user` plus `proxmox_api_password` or `proxmox_api_token_id`/`proxmox_api_token_secret`; `lxc_hostname` |
| `backup_docker.yml` | docker | rsync compose files and volumes to storage; containers in `backup_skip_containers` (default semaphore, semaphore-db) stay up |
| `proxmox-backup.yml` | proxmox | vzdump selected guests, vzdump keeps the last `keep_last`. `guests`, `backup_mode` |
| `discover_guests.yml` | proxmox | list every guest on the host and flag those missing from the inventory; `adopt=true` pushes the Ansible key into unmanaged running containers |
| `update_pihole_blocklists.yml` | pihole | `pihole -up` and gravity update |
| `ssh_harden.yml` | ubuntu, rocky | sshd drop-in (key-only root login) and fail2ban sshd jail. `ssh_disable_password_auth`, `ssh_max_auth_tries` |
| `reboot_check.yml` | ubuntu, rocky | report reboot-required; reboot when `do_reboot=true`, skipping `reboot_exclude` (default `[docker]`) |
| `auto_updates.yml` | ubuntu, rocky | unattended-upgrades / dnf-automatic for security updates, no auto reboot |
| `service_check.yml` | ubuntu, rocky | check `critical_services` per host; `restart_stopped=true` starts stopped ones; Telegram alert for services not running |
| `vm_guest_agent.yml` | vms | install and enable qemu-guest-agent |
| `user_setup.yml` | ubuntu, rocky | non-root admin with sudo and SSH keys. `admin_user`, `admin_sudo_nopasswd`, `admin_ssh_keys` |

Conventions: FQCN module names, `become: true` at play level, OS handled by
targeting the `ubuntu` / `rocky` groups in separate plays, reachability check
as the first play of anything that touches many hosts, ASCII-only reports.

### Notes

- `ssh_harden.yml` only disables password login on hosts where root already
  has an authorized key. It writes `/etc/ssh/sshd_config.d/00-ansible.conf`,
  which takes precedence over `sshd_config` and cloud-init drop-ins.
- `reboot_check.yml` never reboots hosts in `reboot_exclude`; the default
  excludes `docker` because Semaphore most likely runs there.
- `discover_guests.yml` is the answer to "did I forget to add a container". Run it after
  creating anything in the Proxmox UI; with `adopt=true` it drops the Ansible key into
  the new container so bootstrap can reach it. Paste the printed snippet into the inventory.
- `auto_updates.yml` and `update_system.yml` overlap. Use the first for
  hands-off security patches, the second for full upgrades on demand.

## Telegram alerts

`health_check.yml` and `service_check.yml` send one Telegram message per run
when there is something to report (filesystem above the threshold, host
offline, service not running). Without credentials they print the same text
in the run log instead, so nothing breaks before the bot exists.

1. Create a bot with `@BotFather` and copy the token.
2. Send the bot a message (or add it to a group), then open
   `https://api.telegram.org/bot<TOKEN>/getUpdates` and read `chat.id`.
3. In Semaphore, create a variable group with the two values as secret
   environment variables `TELEGRAM_BOT_TOKEN` and `TELEGRAM_CHAT_ID` (extra
   vars `telegram_bot_token` / `telegram_chat_id` work too) and attach it to
   the Health check and Service check templates.

The send task runs with `no_log`, so the token never appears in the run log;
a rejected call fails the run with the Telegram error text only. To add the
alert to another playbook, copy the final `Telegram alert` play from
`health_check.yml` and change the lines the template builds.

## Semaphore schedules

One template per row. Cron times are in Semaphore's own timezone (UTC inside
the container unless `TZ` is set). Reports run in the morning, anything that
restarts services or reboots runs at night. Semaphore does not chain templates,
so the gaps between related runs are deliberate.

Daily:

| Template | Playbook | Cron | Extra vars |
|----------|----------|------|------------|
| Health check | `health_check.yml` | `0 7 * * *` | |
| Service check | `service_check.yml` | `5 7 * * *` | |
| Discover guests | `discover_guests.yml` | `10 7 * * *` | |
| Reboot check (report) | `reboot_check.yml` | `15 7 * * *` | |
| Docker backup | `backup_docker.yml` | `0 2 * * *` | |

Weekly:

| Template | Playbook | Cron | Extra vars |
|----------|----------|------|------------|
| System update | `update_system.yml` | `0 3 * * 0` | |
| Reboot check (reboot) | `reboot_check.yml` | `30 3 * * 0` | `{"do_reboot": true}` |
| Proxmox backup | `proxmox-backup.yml` | `0 1 * * 6` | |
| Pi-hole update | `update_pihole_blocklists.yml` | `0 4 * * 0` | |

Monthly (drift check, also run after adding a host):

| Template | Playbook | Cron | Extra vars |
|----------|----------|------|------------|
| SSH hardening | `ssh_harden.yml` | `0 5 1 * *` | |
| Auto updates | `auto_updates.yml` | `30 5 1 * *` | |

On demand only: `bootstrap_lxc.yml`, `add_root_key.yml`, `time-region.yml`,
`new_lxc.yml`, `user_setup.yml`, `vm_guest_agent.yml`, `ping.yml`.

`backup_docker.yml` stops the containers on the docker host while it copies
volumes, except those in `backup_skip_containers` (default `semaphore` and
`semaphore-db`, because Semaphore runs there; their volumes are copied live).
`proxmox-backup.yml` has the same trap: never list the guest that runs
Semaphore with `backup_mode: stop`, use `snapshot` for it.

## Adding a new container

1. Create it in the Proxmox UI (or with `new_lxc.yml`).
2. Run the Discover guests template with `{"adopt": true}`. It pushes the
   Ansible key into the container and prints an inventory snippet.
3. Paste the snippet into `inventories/all.yml` under the group it names, and
   add the name under `lxc` (or `vms`).
4. Run `bootstrap_lxc.yml`, then `ssh_harden.yml` and `auto_updates.yml`.

## Local checks

```
uv venv .venv && uv pip install --python .venv/bin/python ansible ansible-lint yamllint
.venv/bin/ansible-galaxy collection install -r collections/requirements.yml
.venv/bin/ansible-inventory --graph
.venv/bin/ansible-playbook --syntax-check playbooks/*.yml
.venv/bin/yamllint . && .venv/bin/ansible-lint
```