discover_guests: treat any 192.168.x address as LAN, the lab now spans several subnets

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01JEczyvRjfakkiDNkbZA7PR

.ansible-lint

@@ -0,0 +1,9 @@
+---
+profile: basic
+
+exclude_paths:
+  - .claude/
+  - collections/ansible_collections/
+
+skip_list:
+  - yaml[line-length]

.gitignore

@@ -0,0 +1,7 @@
+*.retry
+__pycache__/
+*.pyc
+.vault_pass*
+.ansible/
+collections/ansible_collections/
+.claude/

.yamllint

@@ -0,0 +1,23 @@
+---
+extends: default
+
+ignore: |
+  .claude/
+  collections/ansible_collections/
+
+rules:
+  line-length:
+    max: 160
+  truthy:
+    allowed-values: ['true', 'false']
+  document-start:
+    level: warning
+  braces:
+    min-spaces-inside: 0
+    max-spaces-inside: 1
+  comments:
+    min-spaces-from-content: 1
+  comments-indentation: disable
+  octal-values:
+    forbid-implicit-octal: true
+    forbid-explicit-octal: true

README.md

@@ -0,0 +1,136 @@
+# Homelab Ansible
+
+Playbooks for a Proxmox homelab: Rocky Linux 9 and Ubuntu LXC containers,
+Ubuntu VMs and the Proxmox host itself. Everything is run unattended from
+Semaphore.
+
+## Layout
+
+```
+ansible.cfg                  defaults for local runs (Semaphore passes its own inventory/key)
+collections/requirements.yml collections Semaphore installs before each run
+inventories/all.yml          the homelab inventory (groups below)
+inventories/new-lxc.yml      throwaway inventory for a freshly created container
+playbooks/*.yml              one self-contained playbook per task, no roles
+```
+
+## Inventory groups
+
+| Group     | Members                     | Notes |
+|-----------|-----------------------------|-------|
+| `ubuntu`  | Ubuntu containers and VMs   | apt plays |
+| `rocky`   | Rocky Linux 9 containers    | dnf plays |
+| `proxmox` | `pve`                       | the Proxmox host |
+| `lxc`     | containers only             | subset of ubuntu/rocky |
+| `vms`     | QEMU VMs only               | subset of ubuntu/rocky, currently empty |
+
+A host may sit in several groups. `ansible_host` is defined once under the OS
+group. `critical_services` on a host entry feeds `service_check.yml`.
+
+Placeholders to fill in (`grep PLACEHOLDER inventories/all.yml`): the Proxmox
+address, the first Ubuntu VM, and the service lists for `webserver`, `bot` and
+`storage`.
+
+## Semaphore
+
+- Inventory type File, path `inventories/all.yml`.
+- Collections in `collections/requirements.yml` are installed automatically.
+- One task template per playbook. Parameters go in as extra vars (JSON).
+
+## Playbooks
+
+| Playbook | Hosts | Purpose / extra vars |
+|----------|-------|----------------------|
+| `ping.yml` | all | connectivity check |
+| `health_check.yml` | all | disk and RAM report, warns above 85 % disk |
+| `update_system.yml` | ubuntu, rocky | apt dist-upgrade / dnf update |
+| `bootstrap_lxc.yml` | ubuntu, rocky | base packages, root SSH key, sshd basics |
+| `add_root_key.yml` | all | add a personal public key to root |
+| `time-region.yml` | ubuntu, rocky | timezone, NTP on VMs, NTP disabled in LXC |
+| `new_lxc.yml` | localhost | create a Rocky container via the Proxmox API |
+| `backup_docker.yml` | docker | rsync compose files and volumes to storage |
+| `proxmox-backup.yml` | proxmox | vzdump selected guests, keep last N |
+| `discover_guests.yml` | proxmox | list every guest on the host and flag those missing from the inventory; `adopt=true` pushes the Ansible key into unmanaged running containers |
+| `update_pihole_blocklists.yml` | pihole | `pihole -up` and gravity update |
+| `ssh_harden.yml` | ubuntu, rocky | sshd drop-in (key-only root login) and fail2ban sshd jail. `ssh_disable_password_auth`, `ssh_max_auth_tries` |
+| `reboot_check.yml` | ubuntu, rocky | report reboot-required; reboot when `do_reboot=true`, skipping `reboot_exclude` (default `[docker]`) |
+| `auto_updates.yml` | ubuntu, rocky | unattended-upgrades / dnf-automatic for security updates, no auto reboot |
+| `service_check.yml` | ubuntu, rocky | check `critical_services` per host; `restart_stopped=true` starts stopped ones |
+| `vm_guest_agent.yml` | vms | install and enable qemu-guest-agent |
+| `user_setup.yml` | ubuntu, rocky | non-root admin with sudo and SSH keys. `admin_user`, `admin_sudo_nopasswd`, `admin_ssh_keys` |
+
+Conventions: FQCN module names, `become: true` at play level, OS handled by
+targeting the `ubuntu` / `rocky` groups in separate plays, reachability check
+as the first play of anything that touches many hosts, ASCII-only reports.
+
+### Notes
+
+- `ssh_harden.yml` only disables password login on hosts where root already
+  has an authorized key. It writes `/etc/ssh/sshd_config.d/00-ansible.conf`,
+  which takes precedence over `sshd_config` and cloud-init drop-ins.
+- `reboot_check.yml` never reboots hosts in `reboot_exclude`; the default
+  excludes `docker` because Semaphore most likely runs there.
+- `discover_guests.yml` is the answer to "did I forget to add a container". Run it after
+  creating anything in the Proxmox UI; with `adopt=true` it drops the Ansible key into
+  the new container so bootstrap can reach it. Paste the printed snippet into the inventory.
+- `auto_updates.yml` and `update_system.yml` overlap. Use the first for
+  hands-off security patches, the second for full upgrades on demand.
+
+## Semaphore schedules
+
+One template per row. Cron times are in Semaphore's own timezone (UTC inside
+the container unless `TZ` is set). Reports run in the morning, anything that
+restarts services or reboots runs at night. Semaphore does not chain templates,
+so the gaps between related runs are deliberate.
+
+Daily:
+
+| Template | Playbook | Cron | Extra vars |
+|----------|----------|------|------------|
+| Health check | `health_check.yml` | `0 7 * * *` | |
+| Service check | `service_check.yml` | `5 7 * * *` | |
+| Discover guests | `discover_guests.yml` | `10 7 * * *` | |
+| Reboot check (report) | `reboot_check.yml` | `15 7 * * *` | |
+| Docker backup | `backup_docker.yml` | `0 2 * * *` | |
+
+Weekly:
+
+| Template | Playbook | Cron | Extra vars |
+|----------|----------|------|------------|
+| System update | `update_system.yml` | `0 3 * * 0` | |
+| Reboot check (reboot) | `reboot_check.yml` | `30 3 * * 0` | `{"do_reboot": true}` |
+| Proxmox backup | `proxmox-backup.yml` | `0 1 * * 6` | |
+| Pi-hole update | `update_pihole_blocklists.yml` | `0 4 * * 0` | |
+
+Monthly (drift check, also run after adding a host):
+
+| Template | Playbook | Cron | Extra vars |
+|----------|----------|------|------------|
+| SSH hardening | `ssh_harden.yml` | `0 5 1 * *` | |
+| Auto updates | `auto_updates.yml` | `30 5 1 * *` | |
+
+On demand only: `bootstrap_lxc.yml`, `add_root_key.yml`, `time-region.yml`,
+`new_lxc.yml`, `user_setup.yml`, `vm_guest_agent.yml`, `ping.yml`.
+
+Before enabling: `backup_docker.yml` stops every container on the docker host
+while it copies volumes. If Semaphore itself runs as a container there, move
+it or exclude it from the stop step first.
+
+## Adding a new container
+
+1. Create it in the Proxmox UI (or with `new_lxc.yml`).
+2. Run the Discover guests template with `{"adopt": true}`. It pushes the
+   Ansible key into the container and prints an inventory snippet.
+3. Paste the snippet into `inventories/all.yml` under the group it names, and
+   add the name under `lxc` (or `vms`).
+4. Run `bootstrap_lxc.yml`, then `ssh_harden.yml` and `auto_updates.yml`.
+
+## Local checks
+
+```
+uv venv .venv && uv pip install --python .venv/bin/python ansible ansible-lint yamllint
+.venv/bin/ansible-galaxy collection install -r collections/requirements.yml
+.venv/bin/ansible-inventory --graph
+.venv/bin/ansible-playbook --syntax-check playbooks/*.yml
+.venv/bin/yamllint . && .venv/bin/ansible-lint
+```

ansible.cfg

@@ -0,0 +1,10 @@
+[defaults]
+inventory = inventories/all.yml
+host_key_checking = False
+retry_files_enabled = False
+interpreter_python = auto_silent
+callback_result_format = yaml
+forks = 10
+
+[ssh_connection]
+pipelining = True

collections/requirements.yml

@@ -0,0 +1,7 @@
+---
+# Semaphore installs these automatically before each run
+# (it only looks at collections/requirements.yml and roles/requirements.yml).
+# Unpinned on purpose until the Semaphore ansible-core version is known.
+collections:
+  - name: community.general
+  - name: ansible.posix

inventories/all.yml

@@ -1,3 +1,17 @@
+# Homelab inventory.
+#
+# Group layout:
+#   ubuntu   - Ubuntu LXC containers and VMs
+#   rocky    - Rocky Linux 9 LXC containers
+#   proxmox  - the Proxmox VE host
+#   lxc      - the subset of ubuntu/rocky hosts that are LXC containers
+#   vms      - the subset of ubuntu/rocky hosts that are QEMU VMs
+#
+# A host may belong to several groups. Define ansible_host once, under the OS
+# group; lxc/vms list names only. Entries marked PLACEHOLDER need real values.
+#
+# critical_services is read by playbooks/service_check.yml.
+---
 all:
   vars:
     ansible_user: root
@@ -6,15 +20,39 @@ all:
       hosts:
         webserver:
           ansible_host: 192.168.55.10
+          critical_services:            # PLACEHOLDER: verify service names
+            - nginx
         docker:
           ansible_host: 192.168.55.11
+          critical_services:
+            - docker
+            - containerd
+        # ubuntu-vm1:                    # PLACEHOLDER: first Ubuntu VM, also list under vms
+        #   ansible_host: 192.168.2.XXX
+        #   critical_services: []
     rocky:
       hosts:
         pihole:
           ansible_host: 192.168.2.2
+          critical_services:
+            - pihole-FTL
         bot:
           ansible_host: 192.168.2.174
+          critical_services: []         # PLACEHOLDER: add the bot's service name
         storage:
           ansible_host: 192.168.2.115
-        # nextcloud:
+          critical_services: []         # PLACEHOLDER: e.g. nfs-server, smb
-        #   ansible_host: 192.168.2.120
+    proxmox:
+      hosts:
+        pve:
+          ansible_host: 192.168.2.254   # PLACEHOLDER: verify
+    lxc:
+      hosts:
+        webserver:
+        docker:
+        pihole:
+        bot:
+        storage:
+    vms:
+      hosts:
+        # ubuntu-vm1:                    # PLACEHOLDER: uncomment together with the ubuntu entry above

new_lxc.yml

@@ -1,75 +0,0 @@
----
-- name: Build Dynamic Rocky Linux LXC Container
-  hosts: localhost
-  gather_facts: false
-
-  vars:
-    # --- Container Specifications ---
-    lxc_hostname: "new_lxc"
-    lxc_template: "local:vztmpl/rockylinux-9-default_20240912_amd64.tar.xz" 
-    lxc_cores: 2
-    lxc_memory: 1024
-    lxc_storage: "TB"
-    lxc_disk_size: 10
-    lxc_bridge: "vmbr0"
-    lxc_dns: "192.168.2.2"
-    target_node: "pve" 
-    
-    # --- Security & Access ---
-    # The public key to inject into the /root/.ssh/authorized_keys of the new container
-    lxc_ssh_key: "ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAINegitHaQG8i+s412h3wTWf/KvpeX1CEpYFva9MhHR+7 ansible-key"
-
-  tasks:
-    - name: Get the next available VMID from the Proxmox cluster
-      ansible.builtin.uri:
-        url: "https://{{ proxmox_api_host }}:8006/api2/json/cluster/nextid"
-        method: GET
-        headers:
-          Authorization: "PVEAPIToken {{ proxmox_api_user }}={{ proxmox_api_password }}"
-        validate_certs: false
-        return_content: true
-      register: nextid_response
-
-    - name: Extract the dynamic VMID from the API response
-      ansible.builtin.set_fact:
-        dynamic_vmid: "{{ nextid_response.json.data }}"
-
-    - name: Create the Rocky Linux LXC container
-      community.general.proxmox:
-        api_host: "{{ proxmox_api_host }}"
-        api_user: "{{ proxmox_api_user }}"
-        api_password: "{{ proxmox_api_password }}"
-        
-        node: "{{ target_node }}"
-        vmid: "{{ dynamic_vmid }}"
-        hostname: "{{ lxc_hostname }}"
-        ostemplate: "{{ lxc_template }}"
-        cores: "{{ lxc_cores }}"
-        memory: "{{ lxc_memory }}"
-        storage: "{{ lxc_storage }}"
-        disk: "{{ lxc_disk_size }}"
-        
-        # Configure network for DHCP on the specified bridge
-        netif: '{"net0":"name=eth0,bridge={{ lxc_bridge }},ip=dhcp"}'
-        
-        # Explicitly set the DNS nameserver
-        nameserver: "{{ lxc_dns }}"
-        
-        # Inject the SSH public key
-        pubkey: "{{ lxc_ssh_key }}"
-        
-        # Highly recommended security flag for Proxmox containers
-        unprivileged: true
-        state: present
-
-    - name: Ensure the new LXC container is powered on
-      community.general.proxmox:
-        api_host: "{{ proxmox_api_host }}"
-        api_user: "{{ proxmox_api_user }}"
-        api_password: "{{ proxmox_api_password }}"
-        vmid: "{{ dynamic_vmid }}"
-        state: started
-        
-    - name: Output the assigned VMID to the Semaphore UI logs
-      ansible.builtin.debug:
-        msg: "Successfully built {{ lxc_hostname }} (Rocky Linux 9) with dynamic ID: {{ dynamic_vmid }}"

playbooks/add_root_key.yml

@@ -1,15 +1,15 @@
 ---
 - name: Add plain text public SSH key to root user across mixed OS
   hosts: all
-  become: yes  # Elevates privileges to root via sudo
+  become: true
-  
+
   vars:
     # Paste your PUBLIC key string inside the quotes below
-    my_plain_text_public_key: "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQDbFKxd6W+gHcNrugkxMwvixGmk3LxtsMEnSdaFp4L78WZ4VhdZrsnSJQ/rViT+vuc6K5SeugtggfQ90L25UyyMv2BhLZaoMS2dpoVJEZon/Y2Hemm11vdEApAK/u0kPTTM5JB0/TZfpoP9sr4pmRIUruRXBhLDvtUV9tJvKSzbhChtB+vBg+MpywG9VvDP6eCVpFaP43RcAFtJR7irJmexd5m4CVhIFaWNeTbbe8YI08i2gDinYl95ygu9i/TorQKLWme2v9ejxQFfJTWGaXHnQ18W0X94Ns4a4OccGcvJ8JmvYsaAPRAOSveq1Fpdw1VXHSKigTmRl6pHWd9/rq6K6Ojctbjo1kHiNHdLKLZbLDeM0OQlzNl74snsImmZddY59wnOKyHtfY0ZI7sQ+7A/0cSaMY6h7lt2xK0WmayuoT8UtKZlItZHCkOA5bV29arI/HTz+yR/LW8SihPPFpVv5aDiOUs0PYd9j0+CtULBJeYpMQJpkD93g06vgEJjhUOBZ6JmilxVDihDARU0x0xiRCQJcq4Nhk/x3GzPe2kkrTI+nZFaQn1K8hcBdSyA12/xAG8Fh1SMCxLwJdCNuL3gQoYd6XPzKqn6BIqAgiyq5DbHhO6mehOye4Rro6i/H1UkZe862QH/R5uH+8Z2/FYRpiCPzQyLwrqx/eVfT7JMFQ== harwinder@harwinder.net"
+    my_plain_text_public_key: "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQDbFKxd6W+gHcNrugkxMwvixGmk3LxtsMEnSdaFp4L78WZ4VhdZrsnSJQ/rViT+vuc6K5SeugtggfQ90L25UyyMv2BhLZaoMS2dpoVJEZon/Y2Hemm11vdEApAK/u0kPTTM5JB0/TZfpoP9sr4pmRIUruRXBhLDvtUV9tJvKSzbhChtB+vBg+MpywG9VvDP6eCVpFaP43RcAFtJR7irJmexd5m4CVhIFaWNeTbbe8YI08i2gDinYl95ygu9i/TorQKLWme2v9ejxQFfJTWGaXHnQ18W0X94Ns4a4OccGcvJ8JmvYsaAPRAOSveq1Fpdw1VXHSKigTmRl6pHWd9/rq6K6Ojctbjo1kHiNHdLKLZbLDeM0OQlzNl74snsImmZddY59wnOKyHtfY0ZI7sQ+7A/0cSaMY6h7lt2xK0WmayuoT8UtKZlItZHCkOA5bV29arI/HTz+yR/LW8SihPPFpVv5aDiOUs0PYd9j0+CtULBJeYpMQJpkD93g06vgEJjhUOBZ6JmilxVDihDARU0x0xiRCQJcq4Nhk/x3GzPe2kkrTI+nZFaQn1K8hcBdSyA12/xAG8Fh1SMCxLwJdCNuL3gQoYd6XPzKqn6BIqAgiyq5DbHhO6mehOye4Rro6i/H1UkZe862QH/R5uH+8Z2/FYRpiCPzQyLwrqx/eVfT7JMFQ== harwinder@harwinder.net"  # yamllint disable-line rule:line-length
 
   tasks:
     - name: Ensure the public key is present in root's authorized_keys
       ansible.posix.authorized_key:
         user: root
         state: present
-        key: "{{ my_plain_text_public_key }}"
+        key: "{{ my_plain_text_public_key }}"

playbooks/auto_updates.yml

@@ -0,0 +1,144 @@
+---
+# Enables automatic security updates: unattended-upgrades on Ubuntu and
+# dnf-automatic on Rocky Linux. Neither reboots automatically; use
+# reboot_check.yml for that.
+
+- name: Check Host Availability
+  hosts: ubuntu:rocky
+  gather_facts: false
+  ignore_unreachable: true
+
+  tasks:
+    - name: Ping hosts to check availability
+      ansible.builtin.ping:
+      register: ping_result
+      ignore_errors: true
+      ignore_unreachable: true
+
+    - name: Add reachable hosts to group
+      ansible.builtin.group_by:
+        key: reachable_hosts
+      when: ping_result is succeeded
+
+    - name: Report offline hosts
+      ansible.builtin.debug:
+        msg: "{{ inventory_hostname }} - OFFLINE"
+      when: ping_result is not succeeded
+      delegate_to: localhost
+
+- name: Auto Updates - Ubuntu
+  hosts: ubuntu:&reachable_hosts
+  become: true
+  gather_facts: false
+
+  tasks:
+    - name: Install unattended-upgrades
+      ansible.builtin.apt:
+        name: unattended-upgrades
+        state: present
+        update_cache: true
+        cache_valid_time: 3600
+
+    - name: Enable periodic package list refresh and upgrades
+      ansible.builtin.copy:
+        dest: /etc/apt/apt.conf.d/20auto-upgrades
+        content: |
+          // Managed by Ansible (playbooks/auto_updates.yml)
+          APT::Periodic::Update-Package-Lists "1";
+          APT::Periodic::Unattended-Upgrade "1";
+          APT::Periodic::AutocleanInterval "7";
+        owner: root
+        group: root
+        mode: '0644'
+
+    - name: Configure unattended-upgrades behaviour
+      ansible.builtin.copy:
+        dest: /etc/apt/apt.conf.d/52unattended-upgrades-local
+        content: |
+          // Managed by Ansible (playbooks/auto_updates.yml)
+          // Allowed origins (security / ESM) come from the stock 50unattended-upgrades.
+          Unattended-Upgrade::Remove-Unused-Dependencies "true";
+          Unattended-Upgrade::Automatic-Reboot "false";
+        owner: root
+        group: root
+        mode: '0644'
+
+    - name: Enable apt timers
+      ansible.builtin.systemd:
+        name: "{{ item }}"
+        enabled: true
+        state: started
+      loop:
+        - apt-daily.timer
+        - apt-daily-upgrade.timer
+
+    - name: Dry-run unattended-upgrade
+      ansible.builtin.command: unattended-upgrade --dry-run -v
+      register: uu_dry_run
+      changed_when: false
+      failed_when: false
+
+    - name: List apt timers  # noqa: command-instead-of-module (read-only listing)
+      ansible.builtin.command: systemctl list-timers apt-daily* --no-pager --all
+      register: apt_timers
+      changed_when: false
+
+    - name: Print report
+      ansible.builtin.debug:
+        msg: |
+          ======================================================
+           HOST: {{ inventory_hostname }} ({{ ansible_host }})
+          ======================================================
+           TIMERS:
+          {{ apt_timers.stdout }}
+          ------------------------------------------------------
+           DRY RUN (last lines):
+          {{ (uu_dry_run.stdout_lines + uu_dry_run.stderr_lines)[-6:] | join('\n') }}
+          ======================================================
+
+- name: Auto Updates - Rocky Linux
+  hosts: rocky:&reachable_hosts
+  become: true
+  gather_facts: false
+
+  tasks:
+    - name: Install dnf-automatic
+      ansible.builtin.dnf:
+        name: dnf-automatic
+        state: present
+
+    - name: Configure dnf-automatic for security updates
+      community.general.ini_file:
+        path: /etc/dnf/automatic.conf
+        section: "{{ item.section }}"
+        option: "{{ item.option }}"
+        value: "{{ item.value }}"
+        mode: '0644'
+      loop:
+        - { section: commands, option: upgrade_type, value: security }
+        - { section: commands, option: download_updates, value: "yes" }
+        - { section: commands, option: apply_updates, value: "yes" }
+        - { section: emitters, option: emit_via, value: stdio }
+      loop_control:
+        label: "{{ item.section }}/{{ item.option }} = {{ item.value }}"
+
+    - name: Enable dnf-automatic timer
+      ansible.builtin.systemd:
+        name: dnf-automatic.timer
+        enabled: true
+        state: started
+
+    - name: List dnf-automatic timers  # noqa: command-instead-of-module (read-only listing)
+      ansible.builtin.command: systemctl list-timers dnf-automatic* --no-pager --all
+      register: dnf_timers
+      changed_when: false
+
+    - name: Print report
+      ansible.builtin.debug:
+        msg: |
+          ======================================================
+           HOST: {{ inventory_hostname }} ({{ ansible_host }})
+          ======================================================
+           TIMERS:
+          {{ dnf_timers.stdout }}
+          ======================================================

playbooks/bootstrap_lxc.yml

@@ -1,3 +1,4 @@
+---
 - name: Bootstrap - Install packages on Ubuntu
   hosts: ubuntu
   become: true
@@ -78,7 +79,7 @@
         update_cache: true
 
 - name: Bootstrap - Configure SSH access
-  hosts: all
+  hosts: ubuntu:rocky
   become: true
 
   tasks:
@@ -114,6 +115,7 @@
         line: 'PubkeyAuthentication yes'
         state: present
 
+    # NOTE: playbooks/ssh_harden.yml overrides this via /etc/ssh/sshd_config.d/00-ansible.conf
     - name: Ensure PasswordAuthentication is enabled
       ansible.builtin.lineinfile:
         path: /etc/ssh/sshd_config

playbooks/discover_guests.yml

@@ -0,0 +1,215 @@
+---
+# Lists every container and VM on the Proxmox host and reports which ones are
+# missing from inventories/all.yml. A guest counts as managed when its LAN IP
+# matches an inventory ansible_host, or (if its IP is unknown, e.g. stopped)
+# when its name matches an inventory hostname.
+#
+# With adopt=true the Ansible SSH key is pushed into every unmanaged RUNNING
+# container straight from the host (pct exec), so a container created by hand
+# in the Proxmox UI becomes reachable without knowing its root password.
+# Afterwards: add the printed inventory snippet, then run bootstrap_lxc.yml.
+#
+# Extra vars (optional):
+#   adopt         - default false
+#   lan_prefix    - address prefix that counts as the LAN; used to pick the right address
+#                   when a guest has several (default 192.168., i.e. any lab subnet)
+#   adopt_ssh_key - public key to push (default: the ansible-key from bootstrap_lxc.yml)
+
+- name: Discover Proxmox guests not in inventory
+  hosts: proxmox
+  become: true
+  gather_facts: false
+
+  vars:
+    adopt: false
+    lan_prefix: "192.168."
+    lan_regex: "^{{ lan_prefix | regex_escape }}"
+    adopt_ssh_key: "ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAINegitHaQG8i+s412h3wTWf/KvpeX1CEpYFva9MhHR+7 ansible-key"
+    # pct "ostype" -> inventory group
+    group_map:
+      centos: rocky
+      ubuntu: ubuntu
+      debian: ubuntu
+
+  tasks:
+    - name: Initialise working facts
+      ansible.builtin.set_fact:
+        guest_ips: {}
+        guest_ostypes: {}
+        guest_table: []
+        inventory_by_ip: {}
+
+    - name: List all guests on the host
+      ansible.builtin.command: pvesh get /cluster/resources --type vm --output-format json
+      register: resources
+      changed_when: false
+
+    - name: Parse guest list
+      ansible.builtin.set_fact:
+        guests: "{{ resources.stdout | from_json | rejectattr('template', 'defined') | sort(attribute='vmid') | list
+                    + (resources.stdout | from_json | selectattr('template', 'defined') | selectattr('template', 'equalto', 0) | list) }}"
+
+    - name: Read container OS type
+      ansible.builtin.command: pct config {{ item.vmid }}
+      loop: "{{ guests | selectattr('type', 'equalto', 'lxc') | list }}"
+      loop_control:
+        label: "{{ item.vmid }} {{ item.name | default('') }}"
+      register: lxc_configs
+      changed_when: false
+      failed_when: false
+
+    - name: Record container OS types
+      ansible.builtin.set_fact:
+        guest_ostypes: "{{ guest_ostypes | combine({(item.item.vmid | string): ostype}) }}"
+      vars:
+        ostype: "{{ (item.stdout | default('') | regex_search('(?m)^ostype: (\\S+)', '\\1')) | default(['-'], true) | first }}"
+      loop: "{{ lxc_configs.results | default([]) }}"
+      loop_control:
+        label: "{{ item.item.vmid }}"
+
+    - name: Read container IP addresses
+      ansible.builtin.command: pct exec {{ item.vmid }} -- hostname -I
+      loop: "{{ guests | selectattr('type', 'equalto', 'lxc') | selectattr('status', 'equalto', 'running') | list }}"
+      loop_control:
+        label: "{{ item.vmid }} {{ item.name | default('') }}"
+      register: lxc_ips
+      changed_when: false
+      failed_when: false
+
+    - name: Record container IP addresses
+      ansible.builtin.set_fact:
+        guest_ips: "{{ guest_ips | combine({(item.item.vmid | string): lan_ip}) }}"
+      vars:
+        all_ips: "{{ item.stdout | default('') | split }}"
+        lan_ip: "{{ all_ips | select('match', lan_regex) | first | default(all_ips | first | default('-')) }}"
+      loop: "{{ lxc_ips.results | default([]) }}"
+      loop_control:
+        label: "{{ item.item.vmid }}"
+
+    - name: Read VM IP addresses via the guest agent
+      ansible.builtin.command: qm guest cmd {{ item.vmid }} network-get-interfaces
+      loop: "{{ guests | selectattr('type', 'equalto', 'qemu') | selectattr('status', 'equalto', 'running') | list }}"
+      loop_control:
+        label: "{{ item.vmid }} {{ item.name | default('') }}"
+      register: qemu_ips
+      changed_when: false
+      failed_when: false
+
+    - name: Record VM IP addresses
+      ansible.builtin.set_fact:
+        guest_ips: "{{ guest_ips | combine({(item.item.vmid | string): lan_ip}) }}"
+      vars:
+        agent_ok: "{{ item.rc == 0 and (item.stdout | default('') | trim) is match('^\\[') }}"
+        all_ips: >-
+          {{ ((item.stdout | from_json) if agent_ok else [])
+             | map(attribute='ip-addresses', default=[]) | flatten
+             | selectattr('ip-address-type', 'equalto', 'ipv4')
+             | map(attribute='ip-address') | reject('match', '^127\\.') | list }}
+        lan_ip: "{{ all_ips | select('match', lan_regex) | first | default(all_ips | first | default('-')) }}"
+      loop: "{{ qemu_ips.results | default([]) }}"
+      loop_control:
+        label: "{{ item.item.vmid }}"
+
+    - name: Index inventory hosts by address
+      ansible.builtin.set_fact:
+        inventory_by_ip: "{{ inventory_by_ip | combine({hostvars[item].ansible_host: item}) }}"
+      loop: "{{ groups['all'] }}"
+      when: hostvars[item].ansible_host is defined
+
+    - name: Assemble guest table
+      ansible.builtin.set_fact:
+        guest_table: "{{ guest_table + [guest_entry] }}"
+      vars:
+        vmid: "{{ item.vmid | string }}"
+        guest_name: "{{ item.name | default('-') }}"
+        guest_ip: "{{ guest_ips[vmid] | default('-') }}"
+        guest_ostype: "{{ guest_ostypes[vmid] | default('-') }}"
+        inventory_match: "{{ inventory_by_ip[guest_ip] | default(guest_name if guest_name in groups['all'] else '') }}"
+        guest_entry:
+          vmid: "{{ vmid }}"
+          type: "{{ item.type }}"
+          status: "{{ item.status }}"
+          name: "{{ guest_name }}"
+          ip: "{{ guest_ip }}"
+          group: "{{ (group_map[guest_ostype] | default(guest_ostype)) if item.type == 'lxc' else 'ubuntu (verify) + vms' }}"
+          inventory_name: "{{ inventory_match }}"
+      loop: "{{ guests }}"
+      loop_control:
+        label: "{{ item.vmid }} {{ item.name | default('') }}"
+
+    - name: Collect unmanaged guests
+      ansible.builtin.set_fact:
+        unmanaged: "{{ guest_table | selectattr('inventory_name', 'equalto', '') | list }}"
+
+    - name: Print guest report
+      ansible.builtin.debug:
+        msg: |
+          ==========================================================================================
+           PROXMOX GUESTS vs INVENTORY  (host: {{ inventory_hostname }})
+          ==========================================================================================
+           VMID  TYPE  STATUS   NAME               IP               GROUP     INVENTORY
+          {% for g in guest_table %}
+           {{ '%-5s %-5s %-8s %-18s %-16s %-9s %s' | format(g.vmid, g.type, g.status, g.name, g.ip, g.group.split(' ')[0],
+                ('managed as ' ~ g.inventory_name) if g.inventory_name else 'NOT IN INVENTORY') }}
+          {% endfor %}
+          ------------------------------------------------------------------------------------------
+           TOTAL: {{ guest_table | length }}   MANAGED: {{ (guest_table | length) - (unmanaged | length) }}   UNMANAGED: {{ unmanaged | length }}
+          ==========================================================================================
+
+    - name: Warn on unmanaged guests
+      ansible.builtin.debug:
+        msg: "WARNING: {{ item.type }} {{ item.vmid }} ({{ item.name }}, {{ item.status }}, ip {{ item.ip }}) is not in inventories/all.yml"
+      loop: "{{ unmanaged }}"
+      loop_control:
+        label: "{{ item.vmid }} {{ item.name }}"
+
+    - name: Suggested inventory entries for unmanaged guests
+      ansible.builtin.debug:
+        msg: |
+          Add to inventories/all.yml under the OS group shown, and list the name under lxc or vms:
+          {% for g in unmanaged %}
+            # {{ g.type }} {{ g.vmid }} -> group: {{ g.group }}
+            {{ g.name }}:
+              ansible_host: {{ g.ip if g.ip != '-' else lan_prefix ~ 'XXX   # start the guest and rerun to discover its address' }}
+              critical_services: []
+          {% endfor %}
+      when: unmanaged | length > 0
+
+    - name: Push the Ansible SSH key into unmanaged running containers
+      ansible.builtin.shell: >-
+        pct exec {{ item.vmid }} -- sh -c
+        'mkdir -p /root/.ssh &&
+        chmod 700 /root/.ssh &&
+        touch /root/.ssh/authorized_keys &&
+        chmod 600 /root/.ssh/authorized_keys &&
+        if ! grep -qxF "{{ adopt_ssh_key }}" /root/.ssh/authorized_keys; then
+        echo "{{ adopt_ssh_key }}" >> /root/.ssh/authorized_keys && echo KEY_ADDED; fi;
+        command -v sshd >/dev/null 2>&1 && echo SSHD_OK || echo NO_SSHD'
+      loop: "{{ unmanaged | selectattr('type', 'equalto', 'lxc') | selectattr('status', 'equalto', 'running') | list }}"
+      loop_control:
+        label: "{{ item.vmid }} {{ item.name }}"
+      register: adopt_result
+      changed_when: "'KEY_ADDED' in adopt_result.stdout"
+      when: adopt | bool
+
+    - name: Adoption summary
+      ansible.builtin.debug:
+        msg: >-
+          {{ item.item.vmid }} ({{ item.item.name }}) -
+          key {{ 'ADDED' if 'KEY_ADDED' in item.stdout else 'already present' }},
+          {{ 'sshd present' if 'SSHD_OK' in item.stdout else 'WARNING: no sshd in the container - install openssh-server before Ansible can reach it' }}
+          - next: add it to inventories/all.yml and run bootstrap_lxc.yml
+      loop: "{{ adopt_result.results | default([]) }}"
+      loop_control:
+        label: "{{ item.item.vmid }}"
+      when:
+        - adopt | bool
+        - item.rc is defined
+
+    - name: Note about unmanaged VMs
+      ansible.builtin.debug:
+        msg: "NOTE: {{ item.type }} {{ item.vmid }} ({{ item.name }}) is a VM; adopt only handles containers. Add its key via cloud-init or the console."
+      loop: "{{ unmanaged | selectattr('type', 'equalto', 'qemu') | list }}"
+      loop_control:
+        label: "{{ item.vmid }} {{ item.name }}"
+      when: adopt | bool

playbooks/health_check.yml

@@ -59,4 +59,4 @@
       ansible.builtin.debug:
         msg: "WARNING: {{ item.split()[-1] }} usage on {{ item.split()[0] }} of {{ inventory_hostname }}"
       loop: "{{ disk_usage.stdout_lines }}"
-      when: (item.split()[-1] | replace('%','') | int) > 85
+      when: (item.split()[-1] | replace('%', '') | int) > 85

playbooks/nextcloud-upgrade.yml

@@ -1,140 +0,0 @@
----
-- name: Safe Nextcloud Upgrade Playbook
-  hosts: nextcloud
-  become: yes
-  vars:
-    nc_path: /var/www/nextcloud
-    php_bin: /usr/bin/php
-
-  tasks:
-
-  - name: Pause playbook for 3 seconds
-    pause:
-      seconds: 3
-      prompt: "Press Ctrl+C then A to abort if no backups exist."
-
-  ###########################################################################
-  # 1 - CHECK CURRENT INSTALLED VERSION
-  ###########################################################################
-  - name: 1.1 - Get current NC status
-    command: "{{ php_bin }} occ status"
-    args:
-      chdir: "{{ nc_path }}"
-    register: nc_status
-
-  - name: 1.2 - Extract installed version
-    set_fact:
-      installed_version: "{{ nc_status.stdout | regex_search('versionstring: (.*)', '\\1') }}"
-
-  - name: 1.3 - Show installed version
-    debug:
-      msg: "Current installed version: {{ installed_version }}"
-
-  ###########################################################################
-  # 2 - CHECK LATEST AVAILABLE VERSION (via updater.phar)
-  ###########################################################################
-  - name: 2.1 - Check for available update
-    command: "{{ php_bin }} updater/updater.phar --check"
-    args:
-      chdir: "{{ nc_path }}"
-    register: update_check
-    failed_when: false  # we want to parse output even if exit code != 0
-
-  - name: 2.2 - Parse latest available version
-    set_fact:
-      available_version: "{{ update_check.stdout | regex_search('Update to version (.*) available', '\\1') }}"
-    when: update_check.stdout is search('Update to version')
-
-  - name: 2.3 - Show available version
-    debug:
-      msg: "Latest available version: {{ available_version | default('NONE') }}"
-
-  ###########################################################################
-  # 2.4 - SKIP IF ALREADY UP TO DATE
-  ###########################################################################
-  - name: 2.4 - Skip upgrade if already on latest version
-    debug:
-      msg: "Nextcloud is already up to date ({{ installed_version }}). Skipping upgrade."
-    when: available_version is not defined
-    tags: skip_upgrade
-    changed_when: false
-
-  - name: End play if up to date
-    meta: end_play
-    when: available_version is not defined
-
-  ###########################################################################
-  # 3 - DISABLE APPS + MAINTENANCE MODE
-  ###########################################################################
-  - name: 3.1 - Enable maintenance mode
-    command: "{{ php_bin }} occ maintenance:mode --on"
-    args:
-      chdir: "{{ nc_path }}"
-
-  ###########################################################################
-  # 3.2 - RUN UPDATER WITH RETRIES
-  ###########################################################################
-  - name: 3.2 - Run Nextcloud CLI updater (retry)
-    command: "{{ php_bin }} updater/updater.phar --no-interaction"
-    args:
-      chdir: "{{ nc_path }}"
-    register: updater_output
-    retries: 3
-    delay: 5
-    until: updater_output.rc == 0
-    failed_when: updater_output.rc != 0
-
-  ###########################################################################
-  # 3.3 - OCC UPGRADE (DB MIGRATIONS)
-  ###########################################################################
-  - block:
-    - name: 3.3 - Run occ upgrade
-      command: "{{ php_bin }} occ upgrade"
-      args:
-        chdir: "{{ nc_path }}"
-      register: upgrade_run
-      failed_when: >
-        upgrade_run.rc != 0 and
-        "already in process" not in upgrade_run.stdout and
-        "maintenance mode" not in upgrade_run.stdout
-
-    rescue:
-      - name: 3.3R - Handle locked upgrade
-        debug:
-          msg: "Upgrade appears stuck. Removing maintenance lock and retrying."
-
-      - name: Remove maintenance mode flag from config.php
-        lineinfile:
-          path: "{{ nc_path }}/config/config.php"
-          regexp: "'maintenance' => true"
-          line: "  'maintenance' => false,"
-          backrefs: yes
-
-      - name: Retry OCC upgrade
-        command: "{{ php_bin }} occ upgrade"
-        args:
-          chdir: "{{ nc_path }}"
-        register: upgrade_retry
-        failed_when: upgrade_retry.rc != 0
-
-  ###########################################################################
-  # 4 - ALWAYS DISABLE MAINTENANCE MODE
-  ###########################################################################
-  - name: ALWAYS disable maintenance mode
-    command: "{{ php_bin }} occ maintenance:mode --off"
-    args:
-      chdir: "{{ nc_path }}"
-    ignore_errors: true
-
-  ###########################################################################
-  # 5 - FINAL RESULT
-  ###########################################################################
-  - name: Show final upgrade result
-    debug:
-      msg: |
-        ✔ Nextcloud upgrade completed.
-        Installed: {{ installed_version }}
-        Updated to: {{ available_version }}
-        Check logs if issues appear:
-         - {{ nc_path }}/data/updater.log
-         - {{ nc_path }}/data/nextcloud.log

playbooks/ping.yml

@@ -1,7 +1,7 @@
+---
 - name: Ping all servers
   hosts: all
-  gather_facts: no
+  gather_facts: false
   tasks:
     - name: Test connectivity with ping
       ansible.builtin.ping:
-

playbooks/reboot_check.yml

@@ -0,0 +1,146 @@
+---
+# Reports which Ubuntu/Rocky guests need a reboot and, only when asked,
+# reboots them. LXC containers restart via Proxmox when they reboot themselves.
+#
+# Extra vars (optional):
+#   do_reboot      - default false (report only)
+#   reboot_exclude - hosts never rebooted by this playbook (default: [docker],
+#                    which most likely runs Semaphore itself)
+
+- name: Check Host Availability
+  hosts: ubuntu:rocky
+  gather_facts: false
+  ignore_unreachable: true
+
+  tasks:
+    - name: Ping hosts to check availability
+      ansible.builtin.ping:
+      register: ping_result
+      ignore_errors: true
+      ignore_unreachable: true
+
+    - name: Add reachable hosts to group
+      ansible.builtin.group_by:
+        key: reachable_hosts
+      when: ping_result is succeeded
+
+    - name: Report offline hosts
+      ansible.builtin.debug:
+        msg: "{{ inventory_hostname }} - OFFLINE"
+      when: ping_result is not succeeded
+      delegate_to: localhost
+
+- name: Reboot Check - Ubuntu
+  hosts: ubuntu:&reachable_hosts
+  become: true
+  gather_facts: true
+
+  tasks:
+    - name: Check for the reboot-required flag
+      ansible.builtin.stat:
+        path: /var/run/reboot-required
+      register: reboot_flag
+
+    - name: Read the packages that requested a reboot
+      ansible.builtin.slurp:
+        src: /var/run/reboot-required.pkgs
+      register: reboot_pkgs
+      when: reboot_flag.stat.exists
+      failed_when: false
+
+    - name: Record reboot status
+      ansible.builtin.set_fact:
+        reboot_needed: "{{ reboot_flag.stat.exists }}"
+        reboot_reason: "{{ (reboot_pkgs.content | default('') | b64decode | trim | replace('\n', ', ')) if reboot_flag.stat.exists else '' }}"
+
+- name: Reboot Check - Rocky Linux
+  hosts: rocky:&reachable_hosts
+  become: true
+  gather_facts: true
+
+  tasks:
+    - name: Ensure needs-restarting is available
+      ansible.builtin.dnf:
+        name: yum-utils
+        state: present
+
+    # needs-restarting -r compares package install times against the kernel boot
+    # time, which inside an LXC container is the host's boot time, so the flag
+    # would never clear. Containers use the process-based check instead.
+    - name: Check whether a reboot is required (VM)
+      ansible.builtin.command: needs-restarting -r
+      register: needs_reboot
+      failed_when: needs_reboot.rc not in [0, 1]
+      changed_when: false
+      when: ansible_virtualization_type != "lxc"
+
+    - name: Check for processes using updated libraries (LXC)
+      ansible.builtin.command: needs-restarting
+      register: stale_procs
+      failed_when: stale_procs.rc not in [0, 1]
+      changed_when: false
+      when: ansible_virtualization_type == "lxc"
+
+    - name: Record reboot status
+      ansible.builtin.set_fact:
+        reboot_needed: >-
+          {{ (needs_reboot.rc == 1) if ansible_virtualization_type != 'lxc'
+             else ((stale_procs.stdout | trim | length) > 0) }}
+        reboot_reason: >-
+          {{ (needs_reboot.stdout | default('') | trim | replace('\n', '; ')) if ansible_virtualization_type != 'lxc'
+             else ((stale_procs.stdout_lines | default([]) | length | string) ~ ' process(es) using updated libraries') }}
+
+- name: Reboot Check - Report
+  hosts: reachable_hosts
+  gather_facts: false
+
+  tasks:
+    - name: Print reboot status
+      ansible.builtin.debug:
+        msg: >-
+          {{ inventory_hostname }} ({{ ansible_virtualization_type | default('unknown') }}) -
+          REBOOT NEEDED: {{ 'YES' if reboot_needed | default(false) | bool else 'no' }}
+          {{ ('- ' ~ reboot_reason) if (reboot_needed | default(false) | bool and reboot_reason | default('') | length > 0) else '' }}
+
+- name: Reboot Check - Reboot hosts that need it
+  hosts: reachable_hosts
+  become: true
+  gather_facts: false
+
+  vars:
+    do_reboot: false
+    reboot_exclude:
+      - docker
+
+  tasks:
+    - name: Report hosts skipped because they are excluded
+      ansible.builtin.debug:
+        msg: "SKIPPED: {{ inventory_hostname }} needs a reboot but is listed in reboot_exclude"
+      when:
+        - do_reboot | bool
+        - reboot_needed | default(false) | bool
+        - inventory_hostname in reboot_exclude
+
+    # Inside a container the default boot_id belongs to the host and never
+    # changes, so the reboot module would wait forever. PID 1's start time does
+    # change when the container restarts.
+    - name: Reboot LXC container
+      ansible.builtin.reboot:
+        boot_time_command: "awk '{print $22}' /proc/1/stat"
+        reboot_timeout: 300
+        msg: "Reboot initiated by Ansible (reboot_check.yml)"
+      when:
+        - do_reboot | bool
+        - reboot_needed | default(false) | bool
+        - inventory_hostname not in reboot_exclude
+        - ansible_virtualization_type == "lxc"
+
+    - name: Reboot VM
+      ansible.builtin.reboot:
+        reboot_timeout: 300
+        msg: "Reboot initiated by Ansible (reboot_check.yml)"
+      when:
+        - do_reboot | bool
+        - reboot_needed | default(false) | bool
+        - inventory_hostname not in reboot_exclude
+        - ansible_virtualization_type != "lxc"

playbooks/service_check.yml

@@ -0,0 +1,82 @@
+---
+# Verifies the critical services listed per host in the inventory
+# (critical_services) and optionally starts any that are stopped.
+#
+# Extra vars (optional):
+#   restart_stopped - default false; when true, start services found stopped
+
+- name: Check Host Availability
+  hosts: ubuntu:rocky
+  gather_facts: false
+  ignore_unreachable: true
+
+  tasks:
+    - name: Ping hosts to check availability
+      ansible.builtin.ping:
+      register: ping_result
+      ignore_errors: true
+      ignore_unreachable: true
+
+    - name: Add reachable hosts to group
+      ansible.builtin.group_by:
+        key: reachable_hosts
+      when: ping_result is succeeded
+
+    - name: Report offline hosts
+      ansible.builtin.debug:
+        msg: "{{ inventory_hostname }} - OFFLINE"
+      when: ping_result is not succeeded
+      delegate_to: localhost
+
+- name: Service Check
+  hosts: reachable_hosts
+  become: true
+  gather_facts: false
+
+  vars:
+    restart_stopped: false
+
+  tasks:
+    - name: Gather service facts
+      ansible.builtin.service_facts:
+
+    - name: Evaluate critical services
+      ansible.builtin.set_fact:
+        service_states: >-
+          {{ service_states | default([])
+             + [{'name': item, 'state': (ansible_facts.services[item ~ '.service'].state | default('not-found'))}] }}
+      loop: "{{ critical_services | default([]) }}"
+
+    - name: Print report
+      ansible.builtin.debug:
+        msg: |
+          ======================================================
+           HOST: {{ inventory_hostname }} ({{ ansible_host }})
+          ======================================================
+          {% if service_states | default([]) | length == 0 %}
+           no critical services defined in inventory
+          {% else %}
+          {% for svc in service_states %}
+           {{ '%-24s' | format(svc.name) }} {{ 'RUNNING' if svc.state == 'running' else ('NOT FOUND' if svc.state == 'not-found' else svc.state | upper) }}
+          {% endfor %}
+          {% endif %}
+          ======================================================
+
+    - name: Warn on services that are not running
+      ansible.builtin.debug:
+        msg: "WARNING: {{ item.name }} is {{ 'not installed' if item.state == 'not-found' else item.state }} on {{ inventory_hostname }}"
+      loop: "{{ service_states | default([]) }}"
+      loop_control:
+        label: "{{ item.name }}"
+      when: item.state != 'running'
+
+    - name: Start stopped services
+      ansible.builtin.service:
+        name: "{{ item.name }}"
+        state: started
+      loop: "{{ service_states | default([]) }}"
+      loop_control:
+        label: "{{ item.name }}"
+      when:
+        - restart_stopped | bool
+        - item.state not in ['running', 'not-found']

playbooks/ssh_harden.yml

@@ -0,0 +1,209 @@
+---
+# Hardens sshd on every reachable Ubuntu/Rocky guest and turns on a fail2ban
+# sshd jail. Settings go into /etc/ssh/sshd_config.d/00-ansible.conf, which
+# sorts first and therefore wins over anything in sshd_config or the
+# cloud-init drop-in (sshd keeps the first value it sees for each option).
+#
+# Lockout guard: password authentication is only disabled on hosts where
+# root already has at least one authorized key.
+#
+# Extra vars (optional):
+#   ssh_disable_password_auth - default true
+#   ssh_max_auth_tries        - default 4
+
+- name: Check Host Availability
+  hosts: ubuntu:rocky
+  gather_facts: false
+  ignore_unreachable: true
+
+  tasks:
+    - name: Ping hosts to check availability
+      ansible.builtin.ping:
+      register: ping_result
+      ignore_errors: true
+      ignore_unreachable: true
+
+    - name: Add reachable hosts to group
+      ansible.builtin.group_by:
+        key: reachable_hosts
+      when: ping_result is succeeded
+
+    - name: Report offline hosts
+      ansible.builtin.debug:
+        msg: "{{ inventory_hostname }} - OFFLINE"
+      when: ping_result is not succeeded
+      delegate_to: localhost
+
+- name: SSH Hardening - Install fail2ban on Ubuntu
+  hosts: ubuntu:&reachable_hosts
+  become: true
+  gather_facts: false
+
+  tasks:
+    - name: Install fail2ban and nftables via apt
+      ansible.builtin.apt:
+        name:
+          - fail2ban
+          - nftables
+        state: present
+        update_cache: true
+        cache_valid_time: 3600
+
+- name: SSH Hardening - Install fail2ban on Rocky Linux
+  hosts: rocky:&reachable_hosts
+  become: true
+  gather_facts: false
+
+  tasks:
+    - name: Ensure EPEL repo is enabled
+      ansible.builtin.dnf:
+        name: epel-release
+        state: present
+
+    - name: Install fail2ban and nftables via dnf
+      ansible.builtin.dnf:
+        name:
+          - fail2ban
+          - nftables
+        state: present
+
+- name: SSH Hardening - Configure sshd and fail2ban
+  hosts: reachable_hosts
+  become: true
+  gather_facts: false
+
+  vars:
+    ssh_disable_password_auth: true
+    ssh_max_auth_tries: 4
+    sshd_dropin_path: /etc/ssh/sshd_config.d/00-ansible.conf
+
+  tasks:
+    - name: Count public keys in root's authorized_keys
+      ansible.builtin.command: grep -cE '^(ssh-|ecdsa-|sk-)' /root/.ssh/authorized_keys
+      register: key_count
+      failed_when: false
+      changed_when: false
+
+    - name: Decide whether password authentication can be disabled
+      ansible.builtin.set_fact:
+        has_keys: "{{ key_count.rc == 0 and (key_count.stdout | int) > 0 }}"
+
+    - name: Warn when root has no authorized keys
+      ansible.builtin.debug:
+        msg: "WARNING: no authorized keys for root on {{ inventory_hostname }} - password authentication stays enabled"
+      when: not has_keys
+
+    - name: Ensure sshd_config.d directory exists
+      ansible.builtin.file:
+        path: /etc/ssh/sshd_config.d
+        state: directory
+        owner: root
+        group: root
+        mode: '0755'
+
+    - name: Ensure sshd_config includes the drop-in directory
+      ansible.builtin.lineinfile:
+        path: /etc/ssh/sshd_config
+        regexp: '^Include\s+/etc/ssh/sshd_config\.d/\*\.conf'
+        line: 'Include /etc/ssh/sshd_config.d/*.conf'
+        insertbefore: BOF
+        backup: true
+      register: sshd_include
+
+    - name: Write sshd hardening drop-in
+      ansible.builtin.copy:
+        dest: "{{ sshd_dropin_path }}"
+        content: |
+          # Managed by Ansible (playbooks/ssh_harden.yml) - do not edit by hand
+          PermitRootLogin prohibit-password
+          PubkeyAuthentication yes
+          PasswordAuthentication {{ 'no' if (has_keys and ssh_disable_password_auth | bool) else 'yes' }}
+          KbdInteractiveAuthentication no
+          MaxAuthTries {{ ssh_max_auth_tries }}
+          X11Forwarding no
+        owner: root
+        group: root
+        mode: '0600'
+        validate: /usr/sbin/sshd -t -f %s
+      register: sshd_dropin
+
+    - name: Record whether sshd needs a restart
+      ansible.builtin.set_fact:
+        sshd_changed: "{{ sshd_include is changed or sshd_dropin is changed }}"
+
+    - name: Configure fail2ban sshd jail
+      ansible.builtin.copy:
+        dest: /etc/fail2ban/jail.d/sshd.local
+        content: |
+          # Managed by Ansible (playbooks/ssh_harden.yml)
+          [sshd]
+          enabled = true
+          backend = systemd
+          banaction = nftables-multiport
+          maxretry = 5
+          findtime = 10m
+          bantime = 1h
+        owner: root
+        group: root
+        mode: '0644'
+      register: f2b_jail
+
+    - name: Ensure fail2ban is enabled and running
+      ansible.builtin.systemd:
+        name: fail2ban
+        enabled: true
+        state: "{{ 'restarted' if f2b_jail is changed else 'started' }}"
+
+- name: SSH Hardening - Restart SSH on Ubuntu
+  hosts: ubuntu:&reachable_hosts
+  become: true
+  gather_facts: false
+
+  tasks:
+    - name: Restart ssh service
+      ansible.builtin.service:
+        name: ssh
+        state: restarted
+      when: sshd_changed | bool
+
+- name: SSH Hardening - Restart SSH on Rocky Linux
+  hosts: rocky:&reachable_hosts
+  become: true
+  gather_facts: false
+
+  tasks:
+    - name: Restart sshd service
+      ansible.builtin.service:
+        name: sshd
+        state: restarted
+      when: sshd_changed | bool
+
+- name: SSH Hardening - Verify access
+  hosts: reachable_hosts
+  become: true
+  gather_facts: false
+
+  tasks:
+    - name: Drop the cached SSH connection so the next task performs a fresh login
+      ansible.builtin.meta: reset_connection
+
+    - name: Confirm key-based login still works
+      ansible.builtin.ping:
+
+    - name: Read fail2ban sshd jail status
+      ansible.builtin.command: fail2ban-client status sshd
+      register: f2b_status
+      changed_when: false
+      failed_when: false
+
+    - name: Summary
+      ansible.builtin.debug:
+        msg: |
+          ======================================================
+           HOST: {{ inventory_hostname }} ({{ ansible_host }})
+          ======================================================
+           PASSWORD AUTH: {{ 'disabled' if (has_keys and ssh_disable_password_auth | default(true) | bool) else 'ENABLED' }}
+           ROOT KEYS:     {{ key_count.stdout if key_count.rc == 0 else 0 }}
+           SSHD:          {{ 'restarted' if sshd_changed | bool else 'unchanged' }}
+           FAIL2BAN:      {{ 'sshd jail active' if f2b_status.rc == 0 else 'WARNING: jail not reporting (' ~ (f2b_status.stderr | default('') | trim) ~ ')' }}
+          ======================================================

playbooks/time-region.yml

@@ -1,6 +1,6 @@
 ---
 - name: Configure timezone on all hosts
-  hosts: all
+  hosts: ubuntu:rocky
   become: true
   gather_facts: true
 
@@ -101,7 +101,7 @@
         enabled: true
 
 - name: Verify time configuration
-  hosts: all
+  hosts: ubuntu:rocky
   become: true
 
   tasks:
@@ -112,4 +112,4 @@
 
     - name: Summary
       ansible.builtin.debug:
-        msg: "{{ inventory_hostname }} ({{ ansible_virtualization_type | default('unknown') }}) - {{ check_time.stdout }}"
+        msg: "{{ inventory_hostname }} ({{ ansible_virtualization_type | default('unknown') }}) - {{ check_time.stdout }}"

playbooks/update_system.yml

@@ -1,3 +1,4 @@
+---
 - name: Check Host Availability
   hosts: all
   gather_facts: false
@@ -43,4 +44,4 @@
       ansible.builtin.dnf:
         name: "*"
         state: latest
-        update_cache: true
+        update_cache: true

playbooks/user_setup.yml

@@ -0,0 +1,104 @@
+---
+# Creates a non-root admin user with sudo and the homelab SSH keys on every
+# reachable Ubuntu/Rocky guest. First step toward running Ansible as a
+# non-root user.
+#
+# Extra vars (optional):
+#   admin_user          - default harry
+#   admin_sudo_nopasswd - default true (required for unattended become)
+#   admin_ssh_keys      - list of public keys (defaults to the ansible-key and
+#                         the personal key already used by add_root_key.yml)
+
+- name: Check Host Availability
+  hosts: ubuntu:rocky
+  gather_facts: false
+  ignore_unreachable: true
+
+  tasks:
+    - name: Ping hosts to check availability
+      ansible.builtin.ping:
+      register: ping_result
+      ignore_errors: true
+      ignore_unreachable: true
+
+    - name: Add reachable hosts to group
+      ansible.builtin.group_by:
+        key: reachable_hosts
+      when: ping_result is succeeded
+
+    - name: Report offline hosts
+      ansible.builtin.debug:
+        msg: "{{ inventory_hostname }} - OFFLINE"
+      when: ping_result is not succeeded
+      delegate_to: localhost
+
+- name: User Setup - Create admin user on Ubuntu
+  hosts: ubuntu:&reachable_hosts
+  become: true
+  gather_facts: false
+
+  vars:
+    admin_user: harry
+
+  tasks:
+    - name: Create admin user in the sudo group - {{ admin_user }}
+      ansible.builtin.user:
+        name: "{{ admin_user }}"
+        shell: /bin/bash
+        groups: sudo
+        append: true
+        create_home: true
+
+- name: User Setup - Create admin user on Rocky Linux
+  hosts: rocky:&reachable_hosts
+  become: true
+  gather_facts: false
+
+  vars:
+    admin_user: harry
+
+  tasks:
+    - name: Create admin user in the wheel group - {{ admin_user }}
+      ansible.builtin.user:
+        name: "{{ admin_user }}"
+        shell: /bin/bash
+        groups: wheel
+        append: true
+        create_home: true
+
+- name: User Setup - Authorized keys and sudo
+  hosts: reachable_hosts
+  become: true
+  gather_facts: false
+
+  vars:
+    admin_user: harry
+    admin_sudo_nopasswd: true
+    admin_ssh_keys:
+      - "ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAINegitHaQG8i+s412h3wTWf/KvpeX1CEpYFva9MhHR+7 ansible-key"
+      - "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQDbFKxd6W+gHcNrugkxMwvixGmk3LxtsMEnSdaFp4L78WZ4VhdZrsnSJQ/rViT+vuc6K5SeugtggfQ90L25UyyMv2BhLZaoMS2dpoVJEZon/Y2Hemm11vdEApAK/u0kPTTM5JB0/TZfpoP9sr4pmRIUruRXBhLDvtUV9tJvKSzbhChtB+vBg+MpywG9VvDP6eCVpFaP43RcAFtJR7irJmexd5m4CVhIFaWNeTbbe8YI08i2gDinYl95ygu9i/TorQKLWme2v9ejxQFfJTWGaXHnQ18W0X94Ns4a4OccGcvJ8JmvYsaAPRAOSveq1Fpdw1VXHSKigTmRl6pHWd9/rq6K6Ojctbjo1kHiNHdLKLZbLDeM0OQlzNl74snsImmZddY59wnOKyHtfY0ZI7sQ+7A/0cSaMY6h7lt2xK0WmayuoT8UtKZlItZHCkOA5bV29arI/HTz+yR/LW8SihPPFpVv5aDiOUs0PYd9j0+CtULBJeYpMQJpkD93g06vgEJjhUOBZ6JmilxVDihDARU0x0xiRCQJcq4Nhk/x3GzPe2kkrTI+nZFaQn1K8hcBdSyA12/xAG8Fh1SMCxLwJdCNuL3gQoYd6XPzKqn6BIqAgiyq5DbHhO6mehOye4Rro6i/H1UkZe862QH/R5uH+8Z2/FYRpiCPzQyLwrqx/eVfT7JMFQ== harwinder@harwinder.net"  # yamllint disable-line rule:line-length
+
+  tasks:
+    - name: Add authorized keys for {{ admin_user }}
+      ansible.posix.authorized_key:
+        user: "{{ admin_user }}"
+        key: "{{ item }}"
+        state: present
+      loop: "{{ admin_ssh_keys }}"
+      loop_control:
+        label: "{{ item.split()[-1] }}"
+
+    - name: Grant sudo to {{ admin_user }}
+      ansible.builtin.copy:
+        dest: "/etc/sudoers.d/90-{{ admin_user }}"
+        content: "{{ admin_user }} ALL=(ALL) {{ 'NOPASSWD:' if admin_sudo_nopasswd | bool else '' }}ALL\n"
+        owner: root
+        group: root
+        mode: '0440'
+        validate: /usr/sbin/visudo -cf %s
+
+    - name: Summary
+      ansible.builtin.debug:
+        msg: >-
+          {{ inventory_hostname }} - user {{ admin_user }} ready
+          (sudo {{ 'NOPASSWD' if admin_sudo_nopasswd | bool else 'with password' }}, {{ admin_ssh_keys | length }} key(s))

playbooks/vm_guest_agent.yml

@@ -0,0 +1,39 @@
+---
+# Installs and enables the QEMU guest agent on VMs (group "vms") so Proxmox
+# can query IPs, quiesce filesystems for snapshots and shut the VM down cleanly.
+# The VM also needs the agent enabled on the Proxmox side:
+#   qm set <vmid> --agent enabled=1   (then power-cycle the VM once)
+
+- name: VM Guest Agent - Install and enable qemu-guest-agent
+  hosts: vms
+  become: true
+  gather_facts: true
+
+  tasks:
+    - name: Install qemu-guest-agent
+      ansible.builtin.package:
+        name: qemu-guest-agent
+        state: present
+
+    - name: Check whether the guest agent virtio port is present
+      ansible.builtin.stat:
+        path: /dev/virtio-ports/org.qemu.guest_agent.0
+      register: agent_port
+
+    - name: Enable and start qemu-guest-agent
+      ansible.builtin.systemd:
+        name: qemu-guest-agent
+        enabled: true
+        state: started
+      when: agent_port.stat.exists
+
+    - name: Warn when the agent device is missing
+      ansible.builtin.debug:
+        msg: "WARNING: {{ inventory_hostname }} has no guest agent device - run 'qm set <vmid> --agent enabled=1' on the Proxmox host and power-cycle the VM"
+      when: not agent_port.stat.exists
+
+    - name: Summary
+      ansible.builtin.debug:
+        msg: >-
+          {{ inventory_hostname }} ({{ ansible_virtualization_type | default('unknown') }}) -
+          qemu-guest-agent {{ 'RUNNING' if agent_port.stat.exists else 'NOT AVAILABLE' }}