Add discover_guests playbook and README with Semaphore scheduling guide

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01JEczyvRjfakkiDNkbZA7PR

README.md

@@ -0,0 +1,136 @@
+# Homelab Ansible
+
+Playbooks for a Proxmox homelab: Rocky Linux 9 and Ubuntu LXC containers,
+Ubuntu VMs and the Proxmox host itself. Everything is run unattended from
+Semaphore.
+
+## Layout
+
+```
+ansible.cfg                  defaults for local runs (Semaphore passes its own inventory/key)
+collections/requirements.yml collections Semaphore installs before each run
+inventories/all.yml          the homelab inventory (groups below)
+inventories/new-lxc.yml      throwaway inventory for a freshly created container
+playbooks/*.yml              one self-contained playbook per task, no roles
+```
+
+## Inventory groups
+
+| Group     | Members                     | Notes |
+|-----------|-----------------------------|-------|
+| `ubuntu`  | Ubuntu containers and VMs   | apt plays |
+| `rocky`   | Rocky Linux 9 containers    | dnf plays |
+| `proxmox` | `pve`                       | the Proxmox host |
+| `lxc`     | containers only             | subset of ubuntu/rocky |
+| `vms`     | QEMU VMs only               | subset of ubuntu/rocky, currently empty |
+
+A host may sit in several groups. `ansible_host` is defined once under the OS
+group. `critical_services` on a host entry feeds `service_check.yml`.
+
+Placeholders to fill in (`grep PLACEHOLDER inventories/all.yml`): the Proxmox
+address, the first Ubuntu VM, and the service lists for `webserver`, `bot` and
+`storage`.
+
+## Semaphore
+
+- Inventory type File, path `inventories/all.yml`.
+- Collections in `collections/requirements.yml` are installed automatically.
+- One task template per playbook. Parameters go in as extra vars (JSON).
+
+## Playbooks
+
+| Playbook | Hosts | Purpose / extra vars |
+|----------|-------|----------------------|
+| `ping.yml` | all | connectivity check |
+| `health_check.yml` | all | disk and RAM report, warns above 85 % disk |
+| `update_system.yml` | ubuntu, rocky | apt dist-upgrade / dnf update |
+| `bootstrap_lxc.yml` | ubuntu, rocky | base packages, root SSH key, sshd basics |
+| `add_root_key.yml` | all | add a personal public key to root |
+| `time-region.yml` | ubuntu, rocky | timezone, NTP on VMs, NTP disabled in LXC |
+| `new_lxc.yml` | localhost | create a Rocky container via the Proxmox API |
+| `backup_docker.yml` | docker | rsync compose files and volumes to storage |
+| `proxmox-backup.yml` | proxmox | vzdump selected guests, keep last N |
+| `discover_guests.yml` | proxmox | list every guest on the host and flag those missing from the inventory; `adopt=true` pushes the Ansible key into unmanaged running containers |
+| `update_pihole_blocklists.yml` | pihole | `pihole -up` and gravity update |
+| `ssh_harden.yml` | ubuntu, rocky | sshd drop-in (key-only root login) and fail2ban sshd jail. `ssh_disable_password_auth`, `ssh_max_auth_tries` |
+| `reboot_check.yml` | ubuntu, rocky | report reboot-required; reboot when `do_reboot=true`, skipping `reboot_exclude` (default `[docker]`) |
+| `auto_updates.yml` | ubuntu, rocky | unattended-upgrades / dnf-automatic for security updates, no auto reboot |
+| `service_check.yml` | ubuntu, rocky | check `critical_services` per host; `restart_stopped=true` starts stopped ones |
+| `vm_guest_agent.yml` | vms | install and enable qemu-guest-agent |
+| `user_setup.yml` | ubuntu, rocky | non-root admin with sudo and SSH keys. `admin_user`, `admin_sudo_nopasswd`, `admin_ssh_keys` |
+
+Conventions: FQCN module names, `become: true` at play level, OS handled by
+targeting the `ubuntu` / `rocky` groups in separate plays, reachability check
+as the first play of anything that touches many hosts, ASCII-only reports.
+
+### Notes
+
+- `ssh_harden.yml` only disables password login on hosts where root already
+  has an authorized key. It writes `/etc/ssh/sshd_config.d/00-ansible.conf`,
+  which takes precedence over `sshd_config` and cloud-init drop-ins.
+- `reboot_check.yml` never reboots hosts in `reboot_exclude`; the default
+  excludes `docker` because Semaphore most likely runs there.
+- `discover_guests.yml` is the answer to "did I forget to add a container". Run it after
+  creating anything in the Proxmox UI; with `adopt=true` it drops the Ansible key into
+  the new container so bootstrap can reach it. Paste the printed snippet into the inventory.
+- `auto_updates.yml` and `update_system.yml` overlap. Use the first for
+  hands-off security patches, the second for full upgrades on demand.
+
+## Semaphore schedules
+
+One template per row. Cron times are in Semaphore's own timezone (UTC inside
+the container unless `TZ` is set). Reports run in the morning, anything that
+restarts services or reboots runs at night. Semaphore does not chain templates,
+so the gaps between related runs are deliberate.
+
+Daily:
+
+| Template | Playbook | Cron | Extra vars |
+|----------|----------|------|------------|
+| Health check | `health_check.yml` | `0 7 * * *` | |
+| Service check | `service_check.yml` | `5 7 * * *` | |
+| Discover guests | `discover_guests.yml` | `10 7 * * *` | |
+| Reboot check (report) | `reboot_check.yml` | `15 7 * * *` | |
+| Docker backup | `backup_docker.yml` | `0 2 * * *` | |
+
+Weekly:
+
+| Template | Playbook | Cron | Extra vars |
+|----------|----------|------|------------|
+| System update | `update_system.yml` | `0 3 * * 0` | |
+| Reboot check (reboot) | `reboot_check.yml` | `30 3 * * 0` | `{"do_reboot": true}` |
+| Proxmox backup | `proxmox-backup.yml` | `0 1 * * 6` | |
+| Pi-hole update | `update_pihole_blocklists.yml` | `0 4 * * 0` | |
+
+Monthly (drift check, also run after adding a host):
+
+| Template | Playbook | Cron | Extra vars |
+|----------|----------|------|------------|
+| SSH hardening | `ssh_harden.yml` | `0 5 1 * *` | |
+| Auto updates | `auto_updates.yml` | `30 5 1 * *` | |
+
+On demand only: `bootstrap_lxc.yml`, `add_root_key.yml`, `time-region.yml`,
+`new_lxc.yml`, `user_setup.yml`, `vm_guest_agent.yml`, `ping.yml`.
+
+Before enabling: `backup_docker.yml` stops every container on the docker host
+while it copies volumes. If Semaphore itself runs as a container there, move
+it or exclude it from the stop step first.
+
+## Adding a new container
+
+1. Create it in the Proxmox UI (or with `new_lxc.yml`).
+2. Run the Discover guests template with `{"adopt": true}`. It pushes the
+   Ansible key into the container and prints an inventory snippet.
+3. Paste the snippet into `inventories/all.yml` under the group it names, and
+   add the name under `lxc` (or `vms`).
+4. Run `bootstrap_lxc.yml`, then `ssh_harden.yml` and `auto_updates.yml`.
+
+## Local checks
+
+```
+uv venv .venv && uv pip install --python .venv/bin/python ansible ansible-lint yamllint
+.venv/bin/ansible-galaxy collection install -r collections/requirements.yml
+.venv/bin/ansible-inventory --graph
+.venv/bin/ansible-playbook --syntax-check playbooks/*.yml
+.venv/bin/yamllint . && .venv/bin/ansible-lint
+```

playbooks/discover_guests.yml

@@ -0,0 +1,214 @@
+---
+# Lists every container and VM on the Proxmox host and reports which ones are
+# missing from inventories/all.yml. A guest counts as managed when its LAN IP
+# matches an inventory ansible_host, or (if its IP is unknown, e.g. stopped)
+# when its name matches an inventory hostname.
+#
+# With adopt=true the Ansible SSH key is pushed into every unmanaged RUNNING
+# container straight from the host (pct exec), so a container created by hand
+# in the Proxmox UI becomes reachable without knowing its root password.
+# Afterwards: add the printed inventory snippet, then run bootstrap_lxc.yml.
+#
+# Extra vars (optional):
+#   adopt         - default false
+#   lan_prefix    - prefix of the LAN, used to pick the right address (default 192.168.2.)
+#   adopt_ssh_key - public key to push (default: the ansible-key from bootstrap_lxc.yml)
+
+- name: Discover Proxmox guests not in inventory
+  hosts: proxmox
+  become: true
+  gather_facts: false
+
+  vars:
+    adopt: false
+    lan_prefix: "192.168.2."
+    lan_regex: "^{{ lan_prefix | regex_escape }}"
+    adopt_ssh_key: "ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAINegitHaQG8i+s412h3wTWf/KvpeX1CEpYFva9MhHR+7 ansible-key"
+    # pct "ostype" -> inventory group
+    group_map:
+      centos: rocky
+      ubuntu: ubuntu
+      debian: ubuntu
+
+  tasks:
+    - name: Initialise working facts
+      ansible.builtin.set_fact:
+        guest_ips: {}
+        guest_ostypes: {}
+        guest_table: []
+        inventory_by_ip: {}
+
+    - name: List all guests on the host
+      ansible.builtin.command: pvesh get /cluster/resources --type vm --output-format json
+      register: resources
+      changed_when: false
+
+    - name: Parse guest list
+      ansible.builtin.set_fact:
+        guests: "{{ resources.stdout | from_json | rejectattr('template', 'defined') | sort(attribute='vmid') | list
+                    + (resources.stdout | from_json | selectattr('template', 'defined') | selectattr('template', 'equalto', 0) | list) }}"
+
+    - name: Read container OS type
+      ansible.builtin.command: pct config {{ item.vmid }}
+      loop: "{{ guests | selectattr('type', 'equalto', 'lxc') | list }}"
+      loop_control:
+        label: "{{ item.vmid }} {{ item.name | default('') }}"
+      register: lxc_configs
+      changed_when: false
+      failed_when: false
+
+    - name: Record container OS types
+      ansible.builtin.set_fact:
+        guest_ostypes: "{{ guest_ostypes | combine({(item.item.vmid | string): ostype}) }}"
+      vars:
+        ostype: "{{ (item.stdout | default('') | regex_search('(?m)^ostype: (\\S+)', '\\1')) | default(['-'], true) | first }}"
+      loop: "{{ lxc_configs.results | default([]) }}"
+      loop_control:
+        label: "{{ item.item.vmid }}"
+
+    - name: Read container IP addresses
+      ansible.builtin.command: pct exec {{ item.vmid }} -- hostname -I
+      loop: "{{ guests | selectattr('type', 'equalto', 'lxc') | selectattr('status', 'equalto', 'running') | list }}"
+      loop_control:
+        label: "{{ item.vmid }} {{ item.name | default('') }}"
+      register: lxc_ips
+      changed_when: false
+      failed_when: false
+
+    - name: Record container IP addresses
+      ansible.builtin.set_fact:
+        guest_ips: "{{ guest_ips | combine({(item.item.vmid | string): lan_ip}) }}"
+      vars:
+        all_ips: "{{ item.stdout | default('') | split }}"
+        lan_ip: "{{ all_ips | select('match', lan_regex) | first | default(all_ips | first | default('-')) }}"
+      loop: "{{ lxc_ips.results | default([]) }}"
+      loop_control:
+        label: "{{ item.item.vmid }}"
+
+    - name: Read VM IP addresses via the guest agent
+      ansible.builtin.command: qm guest cmd {{ item.vmid }} network-get-interfaces
+      loop: "{{ guests | selectattr('type', 'equalto', 'qemu') | selectattr('status', 'equalto', 'running') | list }}"
+      loop_control:
+        label: "{{ item.vmid }} {{ item.name | default('') }}"
+      register: qemu_ips
+      changed_when: false
+      failed_when: false
+
+    - name: Record VM IP addresses
+      ansible.builtin.set_fact:
+        guest_ips: "{{ guest_ips | combine({(item.item.vmid | string): lan_ip}) }}"
+      vars:
+        agent_ok: "{{ item.rc == 0 and (item.stdout | default('') | trim) is match('^\\[') }}"
+        all_ips: >-
+          {{ ((item.stdout | from_json) if agent_ok else [])
+             | map(attribute='ip-addresses', default=[]) | flatten
+             | selectattr('ip-address-type', 'equalto', 'ipv4')
+             | map(attribute='ip-address') | reject('match', '^127\\.') | list }}
+        lan_ip: "{{ all_ips | select('match', lan_regex) | first | default(all_ips | first | default('-')) }}"
+      loop: "{{ qemu_ips.results | default([]) }}"
+      loop_control:
+        label: "{{ item.item.vmid }}"
+
+    - name: Index inventory hosts by address
+      ansible.builtin.set_fact:
+        inventory_by_ip: "{{ inventory_by_ip | combine({hostvars[item].ansible_host: item}) }}"
+      loop: "{{ groups['all'] }}"
+      when: hostvars[item].ansible_host is defined
+
+    - name: Assemble guest table
+      ansible.builtin.set_fact:
+        guest_table: "{{ guest_table + [guest_entry] }}"
+      vars:
+        vmid: "{{ item.vmid | string }}"
+        guest_name: "{{ item.name | default('-') }}"
+        guest_ip: "{{ guest_ips[vmid] | default('-') }}"
+        guest_ostype: "{{ guest_ostypes[vmid] | default('-') }}"
+        inventory_match: "{{ inventory_by_ip[guest_ip] | default(guest_name if guest_name in groups['all'] else '') }}"
+        guest_entry:
+          vmid: "{{ vmid }}"
+          type: "{{ item.type }}"
+          status: "{{ item.status }}"
+          name: "{{ guest_name }}"
+          ip: "{{ guest_ip }}"
+          group: "{{ (group_map[guest_ostype] | default(guest_ostype)) if item.type == 'lxc' else 'ubuntu (verify) + vms' }}"
+          inventory_name: "{{ inventory_match }}"
+      loop: "{{ guests }}"
+      loop_control:
+        label: "{{ item.vmid }} {{ item.name | default('') }}"
+
+    - name: Collect unmanaged guests
+      ansible.builtin.set_fact:
+        unmanaged: "{{ guest_table | selectattr('inventory_name', 'equalto', '') | list }}"
+
+    - name: Print guest report
+      ansible.builtin.debug:
+        msg: |
+          ==========================================================================================
+           PROXMOX GUESTS vs INVENTORY  (host: {{ inventory_hostname }})
+          ==========================================================================================
+           VMID  TYPE  STATUS   NAME               IP               GROUP     INVENTORY
+          {% for g in guest_table %}
+           {{ '%-5s %-5s %-8s %-18s %-16s %-9s %s' | format(g.vmid, g.type, g.status, g.name, g.ip, g.group.split(' ')[0],
+                ('managed as ' ~ g.inventory_name) if g.inventory_name else 'NOT IN INVENTORY') }}
+          {% endfor %}
+          ------------------------------------------------------------------------------------------
+           TOTAL: {{ guest_table | length }}   MANAGED: {{ (guest_table | length) - (unmanaged | length) }}   UNMANAGED: {{ unmanaged | length }}
+          ==========================================================================================
+
+    - name: Warn on unmanaged guests
+      ansible.builtin.debug:
+        msg: "WARNING: {{ item.type }} {{ item.vmid }} ({{ item.name }}, {{ item.status }}, ip {{ item.ip }}) is not in inventories/all.yml"
+      loop: "{{ unmanaged }}"
+      loop_control:
+        label: "{{ item.vmid }} {{ item.name }}"
+
+    - name: Suggested inventory entries for unmanaged guests
+      ansible.builtin.debug:
+        msg: |
+          Add to inventories/all.yml under the OS group shown, and list the name under lxc or vms:
+          {% for g in unmanaged %}
+            # {{ g.type }} {{ g.vmid }} -> group: {{ g.group }}
+            {{ g.name }}:
+              ansible_host: {{ g.ip if g.ip != '-' else lan_prefix ~ 'XXX   # start the guest and rerun to discover its address' }}
+              critical_services: []
+          {% endfor %}
+      when: unmanaged | length > 0
+
+    - name: Push the Ansible SSH key into unmanaged running containers
+      ansible.builtin.shell: >-
+        pct exec {{ item.vmid }} -- sh -c
+        'mkdir -p /root/.ssh &&
+        chmod 700 /root/.ssh &&
+        touch /root/.ssh/authorized_keys &&
+        chmod 600 /root/.ssh/authorized_keys &&
+        if ! grep -qxF "{{ adopt_ssh_key }}" /root/.ssh/authorized_keys; then
+        echo "{{ adopt_ssh_key }}" >> /root/.ssh/authorized_keys && echo KEY_ADDED; fi;
+        command -v sshd >/dev/null 2>&1 && echo SSHD_OK || echo NO_SSHD'
+      loop: "{{ unmanaged | selectattr('type', 'equalto', 'lxc') | selectattr('status', 'equalto', 'running') | list }}"
+      loop_control:
+        label: "{{ item.vmid }} {{ item.name }}"
+      register: adopt_result
+      changed_when: "'KEY_ADDED' in adopt_result.stdout"
+      when: adopt | bool
+
+    - name: Adoption summary
+      ansible.builtin.debug:
+        msg: >-
+          {{ item.item.vmid }} ({{ item.item.name }}) -
+          key {{ 'ADDED' if 'KEY_ADDED' in item.stdout else 'already present' }},
+          {{ 'sshd present' if 'SSHD_OK' in item.stdout else 'WARNING: no sshd in the container - install openssh-server before Ansible can reach it' }}
+          - next: add it to inventories/all.yml and run bootstrap_lxc.yml
+      loop: "{{ adopt_result.results | default([]) }}"
+      loop_control:
+        label: "{{ item.item.vmid }}"
+      when:
+        - adopt | bool
+        - item.rc is defined
+
+    - name: Note about unmanaged VMs
+      ansible.builtin.debug:
+        msg: "NOTE: {{ item.type }} {{ item.vmid }} ({{ item.name }}) is a VM; adopt only handles containers. Add its key via cloud-init or the console."
+      loop: "{{ unmanaged | selectattr('type', 'equalto', 'qemu') | list }}"
+      loop_control:
+        label: "{{ item.vmid }} {{ item.name }}"
+      when: adopt | bool