ansible/README.md

# Homelab Ansible

Playbooks for a Proxmox homelab: Rocky Linux 9 and Ubuntu LXC containers,
Ubuntu VMs and the Proxmox host itself. Everything is run unattended from
Semaphore.

## Layout

```
ansible.cfg                  defaults for local runs (Semaphore passes its own inventory/key)
collections/requirements.yml collections Semaphore installs before each run
inventories/all.yml          the homelab inventory (groups below)
inventories/new-lxc.yml      throwaway inventory for a freshly created container
playbooks/*.yml              one self-contained playbook per task, no roles
```

## Inventory groups

| Group     | Members                     | Notes |
|-----------|-----------------------------|-------|
| `ubuntu`  | Ubuntu containers and VMs   | apt plays |
| `rocky`   | Rocky Linux 9 containers    | dnf plays |
| `proxmox` | `pve`                       | the Proxmox host |
| `lxc`     | containers only             | subset of ubuntu/rocky |
| `vms`     | QEMU VMs only               | subset of ubuntu/rocky, currently empty |

A host may sit in several groups. `ansible_host` is defined once under the OS
group. `critical_services` on a host entry feeds `service_check.yml`.

Placeholders to fill in (`grep PLACEHOLDER inventories/all.yml`): the Proxmox
address, the first Ubuntu VM, and the service lists for `webserver`, `bot` and
`storage`.

## Semaphore

- Inventory type File, path `inventories/all.yml`.
- Collections in `collections/requirements.yml` are installed automatically.
- One task template per playbook. Parameters go in as extra vars (JSON).

## Playbooks

| Playbook | Hosts | Purpose / extra vars |
|----------|-------|----------------------|
| `ping.yml` | all | connectivity check |
| `health_check.yml` | all | disk and RAM report, warns above 85 % disk |
| `update_system.yml` | ubuntu, rocky | apt dist-upgrade / dnf update |
| `bootstrap_lxc.yml` | ubuntu, rocky | base packages, root SSH key, sshd basics |
| `add_root_key.yml` | all | add a personal public key to root |
| `time-region.yml` | ubuntu, rocky | timezone, NTP on VMs, NTP disabled in LXC |
| `new_lxc.yml` | localhost | create a Rocky container via the Proxmox API |
| `backup_docker.yml` | docker | rsync compose files and volumes to storage |
| `proxmox-backup.yml` | proxmox | vzdump selected guests, keep last N |
| `discover_guests.yml` | proxmox | list every guest on the host and flag those missing from the inventory; `adopt=true` pushes the Ansible key into unmanaged running containers |
| `update_pihole_blocklists.yml` | pihole | `pihole -up` and gravity update |
| `ssh_harden.yml` | ubuntu, rocky | sshd drop-in (key-only root login) and fail2ban sshd jail. `ssh_disable_password_auth`, `ssh_max_auth_tries` |
| `reboot_check.yml` | ubuntu, rocky | report reboot-required; reboot when `do_reboot=true`, skipping `reboot_exclude` (default `[docker]`) |
| `auto_updates.yml` | ubuntu, rocky | unattended-upgrades / dnf-automatic for security updates, no auto reboot |
| `service_check.yml` | ubuntu, rocky | check `critical_services` per host; `restart_stopped=true` starts stopped ones |
| `vm_guest_agent.yml` | vms | install and enable qemu-guest-agent |
| `user_setup.yml` | ubuntu, rocky | non-root admin with sudo and SSH keys. `admin_user`, `admin_sudo_nopasswd`, `admin_ssh_keys` |

Conventions: FQCN module names, `become: true` at play level, OS handled by
targeting the `ubuntu` / `rocky` groups in separate plays, reachability check
as the first play of anything that touches many hosts, ASCII-only reports.

### Notes

- `ssh_harden.yml` only disables password login on hosts where root already
  has an authorized key. It writes `/etc/ssh/sshd_config.d/00-ansible.conf`,
  which takes precedence over `sshd_config` and cloud-init drop-ins.
- `reboot_check.yml` never reboots hosts in `reboot_exclude`; the default
  excludes `docker` because Semaphore most likely runs there.
- `discover_guests.yml` is the answer to "did I forget to add a container". Run it after
  creating anything in the Proxmox UI; with `adopt=true` it drops the Ansible key into
  the new container so bootstrap can reach it. Paste the printed snippet into the inventory.
- `auto_updates.yml` and `update_system.yml` overlap. Use the first for
  hands-off security patches, the second for full upgrades on demand.

## Semaphore schedules

One template per row. Cron times are in Semaphore's own timezone (UTC inside
the container unless `TZ` is set). Reports run in the morning, anything that
restarts services or reboots runs at night. Semaphore does not chain templates,
so the gaps between related runs are deliberate.

Daily:

| Template | Playbook | Cron | Extra vars |
|----------|----------|------|------------|
| Health check | `health_check.yml` | `0 7 * * *` | |
| Service check | `service_check.yml` | `5 7 * * *` | |
| Discover guests | `discover_guests.yml` | `10 7 * * *` | |
| Reboot check (report) | `reboot_check.yml` | `15 7 * * *` | |
| Docker backup | `backup_docker.yml` | `0 2 * * *` | |

Weekly:

| Template | Playbook | Cron | Extra vars |
|----------|----------|------|------------|
| System update | `update_system.yml` | `0 3 * * 0` | |
| Reboot check (reboot) | `reboot_check.yml` | `30 3 * * 0` | `{"do_reboot": true}` |
| Proxmox backup | `proxmox-backup.yml` | `0 1 * * 6` | |
| Pi-hole update | `update_pihole_blocklists.yml` | `0 4 * * 0` | |

Monthly (drift check, also run after adding a host):

| Template | Playbook | Cron | Extra vars |
|----------|----------|------|------------|
| SSH hardening | `ssh_harden.yml` | `0 5 1 * *` | |
| Auto updates | `auto_updates.yml` | `30 5 1 * *` | |

On demand only: `bootstrap_lxc.yml`, `add_root_key.yml`, `time-region.yml`,
`new_lxc.yml`, `user_setup.yml`, `vm_guest_agent.yml`, `ping.yml`.

Before enabling: `backup_docker.yml` stops every container on the docker host
while it copies volumes. If Semaphore itself runs as a container there, move
it or exclude it from the stop step first.

## Adding a new container

1. Create it in the Proxmox UI (or with `new_lxc.yml`).
2. Run the Discover guests template with `{"adopt": true}`. It pushes the
   Ansible key into the container and prints an inventory snippet.
3. Paste the snippet into `inventories/all.yml` under the group it names, and
   add the name under `lxc` (or `vms`).
4. Run `bootstrap_lxc.yml`, then `ssh_harden.yml` and `auto_updates.yml`.

## Local checks

```
uv venv .venv && uv pip install --python .venv/bin/python ansible ansible-lint yamllint
.venv/bin/ansible-galaxy collection install -r collections/requirements.yml
.venv/bin/ansible-inventory --graph
.venv/bin/ansible-playbook --syntax-check playbooks/*.yml
.venv/bin/yamllint . && .venv/bin/ansible-lint
```