Report updated vs already up-to-date hosts in update_system.yml

Check each host for pending packages before upgrading, then print a
per-host status (updated with the package list, up to date, or failed)
and one consolidated summary play with totals.

Failed upgrades are caught per host so they are reported instead of
dropping out of the run, and a final fail task keeps the run non-zero.
The read-only cache refresh and pending checks run with check_mode:
false, so --check works as a dry run.

.ansible-lint

@@ -0,0 +1,10 @@
+---
+profile: basic
+
+exclude_paths:
+  - .claude/
+  - .venv/
+  - collections/ansible_collections/
+
+skip_list:
+  - yaml[line-length]

.gitignore

@@ -0,0 +1,8 @@
+*.retry
+__pycache__/
+*.pyc
+.vault_pass*
+.ansible/
+collections/ansible_collections/
+.claude/
+.venv/

.yamllint

@@ -0,0 +1,24 @@
+---
+extends: default
+
+ignore: |
+  .claude/
+  .venv/
+  collections/ansible_collections/
+
+rules:
+  line-length:
+    max: 160
+  truthy:
+    allowed-values: ['true', 'false']
+  document-start:
+    level: warning
+  braces:
+    min-spaces-inside: 0
+    max-spaces-inside: 1
+  comments:
+    min-spaces-from-content: 1
+  comments-indentation: disable
+  octal-values:
+    forbid-implicit-octal: true
+    forbid-explicit-octal: true

README.md

@@ -0,0 +1,158 @@
+# Homelab Ansible
+
+Playbooks for a Proxmox homelab: Rocky Linux 9 and Ubuntu LXC containers,
+Ubuntu VMs and the Proxmox host itself. Everything is run unattended from
+Semaphore.
+
+## Layout
+
+```
+ansible.cfg                  defaults for local runs (Semaphore passes its own inventory/key)
+collections/requirements.yml collections Semaphore installs before each run
+inventories/all.yml          the homelab inventory (groups below)
+inventories/new-lxc.yml      throwaway inventory for a freshly created container
+playbooks/*.yml              one self-contained playbook per task, no roles
+```
+
+## Inventory groups
+
+| Group     | Members                     | Notes |
+|-----------|-----------------------------|-------|
+| `ubuntu`  | Ubuntu containers and VMs   | apt plays |
+| `rocky`   | Rocky Linux 9 containers    | dnf plays |
+| `proxmox` | `pve`                       | the Proxmox host |
+| `lxc`     | containers only             | subset of ubuntu/rocky |
+| `vms`     | QEMU VMs only               | subset of ubuntu/rocky, currently empty |
+
+A host may sit in several groups. `ansible_host` is defined once under the OS
+group. `critical_services` on a host entry feeds `service_check.yml`.
+
+Placeholders to fill in (`grep PLACEHOLDER inventories/all.yml`): the Proxmox
+address, the first Ubuntu VM, and the service lists for `webserver`, `bot` and
+`storage`.
+
+## Semaphore
+
+- Inventory type File, path `inventories/all.yml`.
+- Collections in `collections/requirements.yml` are installed automatically.
+- One task template per playbook. Parameters go in as extra vars (JSON).
+
+## Playbooks
+
+| Playbook | Hosts | Purpose / extra vars |
+|----------|-------|----------------------|
+| `ping.yml` | all | connectivity check |
+| `health_check.yml` | all | disk and RAM report; Telegram alert for filesystems above `disk_warn_pct` (default 80) and offline hosts |
+| `update_system.yml` | ubuntu, rocky | apt dist-upgrade / dnf update; reports per host whether it was updated (with the package list) or already up to date, plus a final summary. Supports `--check` as a dry run |
+| `bootstrap_lxc.yml` | ubuntu, rocky | base packages (EPEL and CRB enabled on Rocky), Ansible root key, sshd basics |
+| `add_root_key.yml` | all | add a personal public key to root |
+| `time-region.yml` | ubuntu, rocky | timezone, NTP on VMs, NTP disabled in LXC. `timezone_region`, `ntp_server` |
+| `new_lxc.yml` | localhost | create a Rocky container via the Proxmox API. `proxmox_api_user` plus `proxmox_api_password` or `proxmox_api_token_id`/`proxmox_api_token_secret`; `lxc_hostname` |
+| `backup_docker.yml` | docker | rsync compose files and volumes to storage; containers in `backup_skip_containers` (default semaphore, semaphore-db) stay up |
+| `proxmox-backup.yml` | proxmox | vzdump selected guests, vzdump keeps the last `keep_last`. `guests`, `backup_mode` |
+| `discover_guests.yml` | proxmox | list every guest on the host and flag those missing from the inventory; `adopt=true` pushes the Ansible key into unmanaged running containers |
+| `update_pihole_blocklists.yml` | pihole | `pihole -up` and gravity update |
+| `ssh_harden.yml` | ubuntu, rocky | sshd drop-in (key-only root login) and fail2ban sshd jail. `ssh_disable_password_auth`, `ssh_max_auth_tries` |
+| `reboot_check.yml` | ubuntu, rocky | report reboot-required; reboot when `do_reboot=true`, skipping `reboot_exclude` (default `[docker]`) |
+| `auto_updates.yml` | ubuntu, rocky | unattended-upgrades / dnf-automatic for security updates, no auto reboot |
+| `service_check.yml` | ubuntu, rocky | check `critical_services` per host; `restart_stopped=true` starts stopped ones; Telegram alert for services not running |
+| `vm_guest_agent.yml` | vms | install and enable qemu-guest-agent |
+| `user_setup.yml` | ubuntu, rocky | non-root admin with sudo and SSH keys. `admin_user`, `admin_sudo_nopasswd`, `admin_ssh_keys` |
+
+Conventions: FQCN module names, `become: true` at play level, OS handled by
+targeting the `ubuntu` / `rocky` groups in separate plays, reachability check
+as the first play of anything that touches many hosts, ASCII-only reports.
+
+### Notes
+
+- `ssh_harden.yml` only disables password login on hosts where root already
+  has an authorized key. It writes `/etc/ssh/sshd_config.d/00-ansible.conf`,
+  which takes precedence over `sshd_config` and cloud-init drop-ins.
+- `reboot_check.yml` never reboots hosts in `reboot_exclude`; the default
+  excludes `docker` because Semaphore most likely runs there.
+- `discover_guests.yml` is the answer to "did I forget to add a container". Run it after
+  creating anything in the Proxmox UI; with `adopt=true` it drops the Ansible key into
+  the new container so bootstrap can reach it. Paste the printed snippet into the inventory.
+- `auto_updates.yml` and `update_system.yml` overlap. Use the first for
+  hands-off security patches, the second for full upgrades on demand.
+
+## Telegram alerts
+
+`health_check.yml` and `service_check.yml` send one Telegram message per run
+when there is something to report (filesystem above the threshold, host
+offline, service not running). Without credentials they print the same text
+in the run log instead, so nothing breaks before the bot exists.
+
+1. Create a bot with `@BotFather` and copy the token.
+2. Send the bot a message (or add it to a group), then open
+   `https://api.telegram.org/bot<TOKEN>/getUpdates` and read `chat.id`.
+3. In Semaphore, create a variable group with the two values as secret
+   environment variables `TELEGRAM_BOT_TOKEN` and `TELEGRAM_CHAT_ID` (extra
+   vars `telegram_bot_token` / `telegram_chat_id` work too) and attach it to
+   the Health check and Service check templates.
+
+The send task runs with `no_log`, so the token never appears in the run log;
+a rejected call fails the run with the Telegram error text only. To add the
+alert to another playbook, copy the final `Telegram alert` play from
+`health_check.yml` and change the lines the template builds.
+
+## Semaphore schedules
+
+One template per row. Cron times are in Semaphore's own timezone (UTC inside
+the container unless `TZ` is set). Reports run in the morning, anything that
+restarts services or reboots runs at night. Semaphore does not chain templates,
+so the gaps between related runs are deliberate.
+
+Daily:
+
+| Template | Playbook | Cron | Extra vars |
+|----------|----------|------|------------|
+| Health check | `health_check.yml` | `0 7 * * *` | |
+| Service check | `service_check.yml` | `5 7 * * *` | |
+| Discover guests | `discover_guests.yml` | `10 7 * * *` | |
+| Reboot check (report) | `reboot_check.yml` | `15 7 * * *` | |
+| Docker backup | `backup_docker.yml` | `0 2 * * *` | |
+
+Weekly:
+
+| Template | Playbook | Cron | Extra vars |
+|----------|----------|------|------------|
+| System update | `update_system.yml` | `0 3 * * 0` | |
+| Reboot check (reboot) | `reboot_check.yml` | `30 3 * * 0` | `{"do_reboot": true}` |
+| Proxmox backup | `proxmox-backup.yml` | `0 1 * * 6` | |
+| Pi-hole update | `update_pihole_blocklists.yml` | `0 4 * * 0` | |
+
+Monthly (drift check, also run after adding a host):
+
+| Template | Playbook | Cron | Extra vars |
+|----------|----------|------|------------|
+| SSH hardening | `ssh_harden.yml` | `0 5 1 * *` | |
+| Auto updates | `auto_updates.yml` | `30 5 1 * *` | |
+
+On demand only: `bootstrap_lxc.yml`, `add_root_key.yml`, `time-region.yml`,
+`new_lxc.yml`, `user_setup.yml`, `vm_guest_agent.yml`, `ping.yml`.
+
+`backup_docker.yml` stops the containers on the docker host while it copies
+volumes, except those in `backup_skip_containers` (default `semaphore` and
+`semaphore-db`, because Semaphore runs there; their volumes are copied live).
+`proxmox-backup.yml` has the same trap: never list the guest that runs
+Semaphore with `backup_mode: stop`, use `snapshot` for it.
+
+## Adding a new container
+
+1. Create it in the Proxmox UI (or with `new_lxc.yml`).
+2. Run the Discover guests template with `{"adopt": true}`. It pushes the
+   Ansible key into the container and prints an inventory snippet.
+3. Paste the snippet into `inventories/all.yml` under the group it names, and
+   add the name under `lxc` (or `vms`).
+4. Run `bootstrap_lxc.yml`, then `ssh_harden.yml` and `auto_updates.yml`.
+
+## Local checks
+
+```
+uv venv .venv && uv pip install --python .venv/bin/python ansible ansible-lint yamllint
+.venv/bin/ansible-galaxy collection install -r collections/requirements.yml
+.venv/bin/ansible-inventory --graph
+.venv/bin/ansible-playbook --syntax-check playbooks/*.yml
+.venv/bin/yamllint . && .venv/bin/ansible-lint
+```

ansible.cfg

@@ -0,0 +1,12 @@
+[defaults]
+inventory = inventories/all.yml
+host_key_checking = False
+retry_files_enabled = False
+interpreter_python = auto_silent
+callback_result_format = yaml
+forks = 10
+# Gather facts once per host per run; later plays reuse them.
+gathering = smart
+
+[ssh_connection]
+pipelining = True

collections/requirements.yml

@@ -0,0 +1,8 @@
+---
+# Semaphore installs these automatically before each run
+# (it only looks at collections/requirements.yml and roles/requirements.yml).
+# Unpinned on purpose until the Semaphore ansible-core version is known.
+collections:
+  - name: community.general
+  - name: community.proxmox   # the Proxmox modules left community.general in 12.0.0
+  - name: ansible.posix

inventories/all.yml

@@ -1,20 +1,63 @@
+# Homelab inventory.
+#
+# Group layout:
+#   ubuntu   - Ubuntu LXC containers and VMs
+#   rocky    - Rocky Linux 9 LXC containers
+#   proxmox  - the Proxmox VE host
+#   lxc      - the subset of ubuntu/rocky hosts that are LXC containers
+#   vms      - the subset of ubuntu/rocky hosts that are QEMU VMs
+#
+# A host may belong to several groups. Define ansible_host once, under the OS
+# group; lxc/vms list names only. Entries marked PLACEHOLDER need real values.
+#
+# critical_services is read by playbooks/service_check.yml.
+---
 all:
-  hosts:
+  vars:
-    webserver:
+    ansible_user: root
-      ansible_host: 192.168.2.15
+  children:
-      ansible_user: root
+    ubuntu:
-    nextcloud:
+      hosts:
-      ansible_host: 192.168.2.120
+        # webserver:
-      ansible_user: root
+        #   ansible_host: 192.168.55.10
-    pihole:
+        #   critical_services:            # PLACEHOLDER: verify service names
-      ansible_host: 192.168.2.2
+        #     - nginx
-      ansible_user: root
+        docker:
-    docker:
+          ansible_host: 192.168.0.11
-      ansible_host: 192.168.2.10
+          critical_services:
-      ansible_user: root
+            - docker
-    # eve:
+            - containerd
-    #   ansible_host: 192.168.2.12
+        # ubuntu-vm1:                    # PLACEHOLDER: first Ubuntu VM, also list under vms
-    #   ansible_user: root
+        #   ansible_host: 192.168.2.XXX
-    storage:
+        #   critical_services: []
-      ansible_host: 192.168.2.115
+    rocky:
-      ansible_user: root
+      hosts:
+        courses-lxc:
+          ansible_host: 192.168.0.31
+          critical_services:
+            - httpd
+            - mariadb
+        pihole:
+          ansible_host: 192.168.0.2
+          critical_services:
+            - pihole-FTL
+        bot:
+          ansible_host: 192.168.0.174
+          critical_services: [myapp]         # PLACEHOLDER: add the bot's service name
+        storage:
+          ansible_host: 192.168.0.115
+          critical_services: [smb]         # PLACEHOLDER: e.g. nfs-server, smb
+    proxmox:
+      hosts:
+        pve:
+          ansible_host: 192.168.0.254   # PLACEHOLDER: verify
+    lxc:
+      hosts:
+        # webserver:
+        docker:
+        pihole:
+        bot:
+        storage:
+    vms:
+      hosts:
+        # ubuntu-vm1:                    # PLACEHOLDER: uncomment together with the ubuntu entry above

inventories/new-lxc.yml

@@ -1,7 +1,8 @@
+---
 all:
   children:
     newlxcs:
       hosts:
         new:
           ansible_host: 192.168.2.185
-          ansible_user: root
+          ansible_user: root

new_lxc.yml

@@ -1,75 +0,0 @@
----
-- name: Build Dynamic Rocky Linux LXC Container
-  hosts: localhost
-  gather_facts: false
-
-  vars:
-    # --- Container Specifications ---
-    lxc_hostname: "new_lxc"
-    lxc_template: "local:vztmpl/rockylinux-9-default_20240912_amd64.tar.xz" 
-    lxc_cores: 2
-    lxc_memory: 1024
-    lxc_storage: "TB"
-    lxc_disk_size: 10
-    lxc_bridge: "vmbr0"
-    lxc_dns: "192.168.2.2"
-    target_node: "pve" 
-    
-    # --- Security & Access ---
-    # The public key to inject into the /root/.ssh/authorized_keys of the new container
-    lxc_ssh_key: "ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAINegitHaQG8i+s412h3wTWf/KvpeX1CEpYFva9MhHR+7 ansible-key"
-
-  tasks:
-    - name: Get the next available VMID from the Proxmox cluster
-      ansible.builtin.uri:
-        url: "https://{{ proxmox_api_host }}:8006/api2/json/cluster/nextid"
-        method: GET
-        headers:
-          Authorization: "PVEAPIToken {{ proxmox_api_user }}={{ proxmox_api_password }}"
-        validate_certs: false
-        return_content: true
-      register: nextid_response
-
-    - name: Extract the dynamic VMID from the API response
-      ansible.builtin.set_fact:
-        dynamic_vmid: "{{ nextid_response.json.data }}"
-
-    - name: Create the Rocky Linux LXC container
-      community.general.proxmox:
-        api_host: "{{ proxmox_api_host }}"
-        api_user: "{{ proxmox_api_user }}"
-        api_password: "{{ proxmox_api_password }}"
-        
-        node: "{{ target_node }}"
-        vmid: "{{ dynamic_vmid }}"
-        hostname: "{{ lxc_hostname }}"
-        ostemplate: "{{ lxc_template }}"
-        cores: "{{ lxc_cores }}"
-        memory: "{{ lxc_memory }}"
-        storage: "{{ lxc_storage }}"
-        disk: "{{ lxc_disk_size }}"
-        
-        # Configure network for DHCP on the specified bridge
-        netif: '{"net0":"name=eth0,bridge={{ lxc_bridge }},ip=dhcp"}'
-        
-        # Explicitly set the DNS nameserver
-        nameserver: "{{ lxc_dns }}"
-        
-        # Inject the SSH public key
-        pubkey: "{{ lxc_ssh_key }}"
-        
-        # Highly recommended security flag for Proxmox containers
-        unprivileged: true
-        state: present
-
-    - name: Ensure the new LXC container is powered on
-      community.general.proxmox:
-        api_host: "{{ proxmox_api_host }}"
-        api_user: "{{ proxmox_api_user }}"
-        api_password: "{{ proxmox_api_password }}"
-        vmid: "{{ dynamic_vmid }}"
-        state: started
-        
-    - name: Output the assigned VMID to the Semaphore UI logs
-      ansible.builtin.debug:
-        msg: "Successfully built {{ lxc_hostname }} (Rocky Linux 9) with dynamic ID: {{ dynamic_vmid }}"

playbooks/add_root_key.yml

@@ -1,15 +1,15 @@
 ---
 - name: Add plain text public SSH key to root user across mixed OS
   hosts: all
-  become: yes  # Elevates privileges to root via sudo
+  become: true
-  
+
   vars:
     # Paste your PUBLIC key string inside the quotes below
-    my_plain_text_public_key: "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQDbFKxd6W+gHcNrugkxMwvixGmk3LxtsMEnSdaFp4L78WZ4VhdZrsnSJQ/rViT+vuc6K5SeugtggfQ90L25UyyMv2BhLZaoMS2dpoVJEZon/Y2Hemm11vdEApAK/u0kPTTM5JB0/TZfpoP9sr4pmRIUruRXBhLDvtUV9tJvKSzbhChtB+vBg+MpywG9VvDP6eCVpFaP43RcAFtJR7irJmexd5m4CVhIFaWNeTbbe8YI08i2gDinYl95ygu9i/TorQKLWme2v9ejxQFfJTWGaXHnQ18W0X94Ns4a4OccGcvJ8JmvYsaAPRAOSveq1Fpdw1VXHSKigTmRl6pHWd9/rq6K6Ojctbjo1kHiNHdLKLZbLDeM0OQlzNl74snsImmZddY59wnOKyHtfY0ZI7sQ+7A/0cSaMY6h7lt2xK0WmayuoT8UtKZlItZHCkOA5bV29arI/HTz+yR/LW8SihPPFpVv5aDiOUs0PYd9j0+CtULBJeYpMQJpkD93g06vgEJjhUOBZ6JmilxVDihDARU0x0xiRCQJcq4Nhk/x3GzPe2kkrTI+nZFaQn1K8hcBdSyA12/xAG8Fh1SMCxLwJdCNuL3gQoYd6XPzKqn6BIqAgiyq5DbHhO6mehOye4Rro6i/H1UkZe862QH/R5uH+8Z2/FYRpiCPzQyLwrqx/eVfT7JMFQ== harwinder@harwinder.net"
+    my_plain_text_public_key: "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQDbFKxd6W+gHcNrugkxMwvixGmk3LxtsMEnSdaFp4L78WZ4VhdZrsnSJQ/rViT+vuc6K5SeugtggfQ90L25UyyMv2BhLZaoMS2dpoVJEZon/Y2Hemm11vdEApAK/u0kPTTM5JB0/TZfpoP9sr4pmRIUruRXBhLDvtUV9tJvKSzbhChtB+vBg+MpywG9VvDP6eCVpFaP43RcAFtJR7irJmexd5m4CVhIFaWNeTbbe8YI08i2gDinYl95ygu9i/TorQKLWme2v9ejxQFfJTWGaXHnQ18W0X94Ns4a4OccGcvJ8JmvYsaAPRAOSveq1Fpdw1VXHSKigTmRl6pHWd9/rq6K6Ojctbjo1kHiNHdLKLZbLDeM0OQlzNl74snsImmZddY59wnOKyHtfY0ZI7sQ+7A/0cSaMY6h7lt2xK0WmayuoT8UtKZlItZHCkOA5bV29arI/HTz+yR/LW8SihPPFpVv5aDiOUs0PYd9j0+CtULBJeYpMQJpkD93g06vgEJjhUOBZ6JmilxVDihDARU0x0xiRCQJcq4Nhk/x3GzPe2kkrTI+nZFaQn1K8hcBdSyA12/xAG8Fh1SMCxLwJdCNuL3gQoYd6XPzKqn6BIqAgiyq5DbHhO6mehOye4Rro6i/H1UkZe862QH/R5uH+8Z2/FYRpiCPzQyLwrqx/eVfT7JMFQ== harwinder@harwinder.net"  # yamllint disable-line rule:line-length
 
   tasks:
     - name: Ensure the public key is present in root's authorized_keys
       ansible.posix.authorized_key:
         user: root
         state: present
-        key: "{{ my_plain_text_public_key }}"
+        key: "{{ my_plain_text_public_key }}"

playbooks/auto_updates.yml

@@ -0,0 +1,238 @@
+---
+# Enables automatic security updates: unattended-upgrades on Ubuntu and
+# dnf-automatic on Rocky Linux. Neither reboots automatically; use
+# reboot_check.yml for that.
+
+# Play 1: probe every ubuntu/rocky host and sort them into a "reachable_hosts"
+# group so the two update plays below can skip anything that's offline.
+- name: Check Host Availability
+  hosts: ubuntu:rocky
+  gather_facts: false
+  ignore_unreachable: true
+
+  tasks:
+    # Ping each host; failures/unreachables are swallowed so the play keeps
+    # going instead of aborting the whole run.
+    - name: Ping hosts to check availability
+      ansible.builtin.ping:
+      register: ping_result
+      ignore_errors: true
+      ignore_unreachable: true
+
+    # Dynamically creates/populates the "reachable_hosts" group, used by the
+    # Ubuntu and Rocky plays' host patterns (ubuntu:&reachable_hosts etc.).
+    - name: Add reachable hosts to group
+      ansible.builtin.group_by:
+        key: reachable_hosts
+      when: ping_result is reachable and ping_result is succeeded
+
+    # Print a message for any host that failed the ping, so offline hosts
+    # are visible in the run output.
+    - name: Report offline hosts
+      ansible.builtin.debug:
+        msg: "{{ inventory_hostname }} - OFFLINE"
+      when: ping_result is unreachable or ping_result is failed
+      delegate_to: localhost
+
+# Play 2: install and configure unattended-upgrades on Ubuntu hosts, then
+# report its status. Does not enable automatic reboots.
+- name: Auto Updates - Ubuntu
+  hosts: ubuntu:&reachable_hosts
+  become: true
+  gather_facts: false
+
+  tasks:
+    # Make sure the unattended-upgrades package is present.
+    - name: Install unattended-upgrades
+      ansible.builtin.apt:
+        name: unattended-upgrades
+        state: present
+        update_cache: true
+        cache_valid_time: 3600
+
+    # Turn on apt's periodic package-list refresh, unattended-upgrade run,
+    # and weekly autoclean.
+    - name: Enable periodic package list refresh and upgrades
+      ansible.builtin.copy:
+        dest: /etc/apt/apt.conf.d/20auto-upgrades
+        content: |
+          // Managed by Ansible (playbooks/auto_updates.yml)
+          APT::Periodic::Update-Package-Lists "1";
+          APT::Periodic::Unattended-Upgrade "1";
+          APT::Periodic::AutocleanInterval "7";
+        owner: root
+        group: root
+        mode: '0644'
+
+    # Local overrides layered on top of the stock 50unattended-upgrades
+    # config: clean up unused deps, but never reboot automatically.
+    - name: Configure unattended-upgrades behaviour
+      ansible.builtin.copy:
+        dest: /etc/apt/apt.conf.d/52unattended-upgrades-local
+        content: |
+          // Managed by Ansible (playbooks/auto_updates.yml)
+          // Allowed origins (security / ESM) come from the stock 50unattended-upgrades.
+          Unattended-Upgrade::Remove-Unused-Dependencies "true";
+          Unattended-Upgrade::Automatic-Reboot "false";
+        owner: root
+        group: root
+        mode: '0644'
+
+    # Enable and start the systemd timers that actually trigger apt's
+    # periodic update/upgrade cycle.
+    - name: Enable apt timers
+      ansible.builtin.systemd:
+        name: "{{ item }}"
+        enabled: true
+        state: started
+      loop:
+        - apt-daily.timer
+        - apt-daily-upgrade.timer
+
+    # Simulate an unattended-upgrade run (no changes made) so the report
+    # below shows what would actually be upgraded.
+    - name: Dry-run unattended-upgrade
+      ansible.builtin.command: unattended-upgrade --dry-run -v
+      register: uu_dry_run
+      changed_when: false
+      failed_when: false
+
+    # Pull the package list out of the dry-run's "Packages that will be
+    # upgraded: ..." line, used for the per-host and final summary reports.
+    - name: Extract pending package list
+      ansible.builtin.set_fact:
+        pending_update_packages: "{{ (uu_dry_run.stdout | regex_findall('Packages that will be upgraded:\\s*(.*)') | first | default('')).split() }}"
+
+    - name: Compute pending update count
+      ansible.builtin.set_fact:
+        pending_update_count: "{{ pending_update_packages | length }}"
+
+    # Grab the current state of the apt-daily timers for the report.
+    - name: List apt timers  # noqa: command-instead-of-module (read-only listing)
+      ansible.builtin.command: systemctl list-timers apt-daily* --no-pager --all
+      register: apt_timers
+      changed_when: false
+
+    # Summarize timer status, pending update count, and the tail of the
+    # dry-run output per host.
+    - name: Print report
+      ansible.builtin.debug:
+        msg: |
+          ======================================================
+           HOST: {{ inventory_hostname }} ({{ ansible_host }})
+          ======================================================
+           TIMERS:
+          {{ apt_timers.stdout }}
+          ------------------------------------------------------
+           PENDING UPDATES: {{ pending_update_count }}
+          {{ pending_update_packages | join(', ') }}
+          ------------------------------------------------------
+           DRY RUN (last lines):
+          {{ (uu_dry_run.stdout_lines + uu_dry_run.stderr_lines)[-6:] | join('\n') }}
+          ======================================================
+
+# Play 3: install and configure dnf-automatic on Rocky Linux hosts (security
+# updates only), then report its status.
+- name: Auto Updates - Rocky Linux
+  hosts: rocky:&reachable_hosts
+  become: true
+  gather_facts: false
+
+  tasks:
+    # Make sure the dnf-automatic package is present.
+    - name: Install dnf-automatic
+      ansible.builtin.dnf:
+        name: dnf-automatic
+        state: present
+
+    # Set dnf-automatic to download and apply security updates only, and
+    # log output to stdout (captured by the systemd journal).
+    - name: Configure dnf-automatic for security updates
+      community.general.ini_file:
+        path: /etc/dnf/automatic.conf
+        section: "{{ item.section }}"
+        option: "{{ item.option }}"
+        value: "{{ item.value }}"
+        mode: '0644'
+      loop:
+        - { section: commands, option: upgrade_type, value: security }
+        - { section: commands, option: download_updates, value: "yes" }
+        - { section: commands, option: apply_updates, value: "yes" }
+        - { section: emitters, option: emit_via, value: stdio }
+      loop_control:
+        label: "{{ item.section }}/{{ item.option }} = {{ item.value }}"
+
+    # Enable and start the systemd timer that triggers dnf-automatic runs.
+    - name: Enable dnf-automatic timer
+      ansible.builtin.systemd:
+        name: dnf-automatic.timer
+        enabled: true
+        state: started
+
+    # Grab the current state of the dnf-automatic timer for the report.
+    - name: List dnf-automatic timers  # noqa: command-instead-of-module (read-only listing)
+      ansible.builtin.command: systemctl list-timers dnf-automatic* --no-pager --all
+      register: dnf_timers
+      changed_when: false
+
+    # Check what security updates are pending. rc 100 means updates are
+    # available, rc 0 means none, so only other codes count as a failure.
+    - name: Check for pending security updates
+      ansible.builtin.command: dnf check-update --security
+      register: dnf_check
+      changed_when: false
+      failed_when: dnf_check.rc not in [0, 100]
+
+    # Pull the "name.arch  version  repo" lines out of the check-update
+    # output, used for the per-host and final summary reports.
+    - name: Extract pending package list
+      ansible.builtin.set_fact:
+        pending_update_packages: "{{ dnf_check.stdout_lines | select('match', '^\\S+\\.\\S+\\s+\\S+\\s+\\S+$') | map('regex_replace', '^(\\S+)\\s.*$', '\\1') | list }}"
+
+    - name: Compute pending update count
+      ansible.builtin.set_fact:
+        pending_update_count: "{{ pending_update_packages | length }}"
+
+    # Summarize timer status and pending update count per host.
+    - name: Print report
+      ansible.builtin.debug:
+        msg: |
+          ======================================================
+           HOST: {{ inventory_hostname }} ({{ ansible_host }})
+          ======================================================
+           TIMERS:
+          {{ dnf_timers.stdout }}
+          ------------------------------------------------------
+           PENDING UPDATES: {{ pending_update_count }}
+          {{ pending_update_packages | join(', ') }}
+          ======================================================
+
+# Play 4: after both OS-specific plays have run, print one consolidated
+# summary of which hosts have pending updates and how many packages.
+- name: Summary - Pending Updates
+  hosts: localhost
+  gather_facts: false
+
+  tasks:
+    - name: Print hosts with pending updates
+      ansible.builtin.debug:
+        msg: "{{ item }} - {{ hostvars[item].pending_update_count }} package(s) pending: {{ hostvars[item].pending_update_packages | join(', ') }}"
+      loop: "{{ groups['reachable_hosts'] | default([]) }}"
+      when: hostvars[item].pending_update_count | default(0) | int > 0
+
+    - name: Print hosts fully up to date
+      ansible.builtin.debug:
+        msg: "{{ item }} - up to date"
+      loop: "{{ groups['reachable_hosts'] | default([]) }}"
+      when: hostvars[item].pending_update_count is defined and hostvars[item].pending_update_count | int == 0
+
+    - name: Print totals
+      ansible.builtin.debug:
+        msg: >-
+          {{ groups['reachable_hosts'] | default([]) | length }} host(s) reachable,
+          {{ groups['reachable_hosts'] | default([])
+             | map('extract', hostvars, 'pending_update_count') | select('defined') | map('int') | select('>', 0) | list | length }}
+          with pending updates,
+          {{ groups['reachable_hosts'] | default([])
+             | map('extract', hostvars, 'pending_update_count') | select('defined') | map('int') | sum }}
+          package(s) total pending.

playbooks/backup_docker.yml

@@ -0,0 +1,120 @@
+---
+# Backs up the docker host to the storage container: compose files and
+# configs from compose_dir while everything runs, then the Docker volumes
+# with the containers stopped so the copy is consistent.
+#
+# Containers in backup_skip_containers are never stopped. Semaphore runs on
+# this host, so it and its database stay up (their volumes are copied live).
+# root on the docker host needs an SSH key that is authorized on backup_host.
+#
+# Extra vars (optional):
+#   backup_host            - inventory name of the destination (default storage)
+#   backup_root            - directory on backup_host (default /mnt/storage/Docker_backups)
+#   backup_skip_containers - default [semaphore, semaphore-db]
+
+- name: Backup Docker compose files and volumes to storage
+  hosts: docker
+  become: true
+  gather_facts: false
+
+  vars:
+    backup_host: storage
+    backup_root: /mnt/storage/Docker_backups
+    backup_addr: "{{ hostvars[backup_host].ansible_host }}"
+    backup_skip_containers:
+      - semaphore
+      - semaphore-db
+    compose_dir: /home/harry
+    volumes_dir: /var/lib/docker/volumes
+    rsync_ssh: ssh -o StrictHostKeyChecking=accept-new
+
+  tasks:
+    - name: Ensure backup destination directories exist
+      ansible.builtin.file:
+        path: "{{ backup_root }}/{{ item }}"
+        state: directory
+        mode: '0755'
+      loop: [compose, volumes]
+      delegate_to: "{{ backup_host }}"
+
+    # rsync applies the first matching rule: excludes first, then the rule
+    # that lets it descend into directories, then the wanted files, then
+    # everything else is dropped.
+    - name: Rsync compose files and configs (safe while running)
+      ansible.builtin.command:
+        argv:
+          - rsync
+          - -az
+          - --delete
+          - --prune-empty-dirs
+          - -e
+          - "{{ rsync_ssh }}"
+          - --exclude=node_modules/
+          - --exclude=.git/
+          - --exclude=__pycache__/
+          - --include=*/
+          - --include=docker-compose.yml
+          - --include=compose.yml
+          - --include=.env
+          - --include=*.conf
+          - --include=*.yml
+          - --include=*.yaml
+          - --include=*.json
+          - --exclude=*
+          - "{{ compose_dir }}/"
+          - "root@{{ backup_addr }}:{{ backup_root }}/compose/"
+      changed_when: true
+
+    - name: List running containers
+      ansible.builtin.command:
+        argv: [docker, ps, --format, "{{ '{{' }}.Names{{ '}}' }}"]
+      register: running_containers
+      changed_when: false
+
+    - name: Work out which containers to stop
+      ansible.builtin.set_fact:
+        containers_to_stop: "{{ running_containers.stdout_lines | difference(backup_skip_containers) }}"
+
+    - name: Copy the volumes with the containers stopped
+      block:
+        - name: Stop containers
+          ansible.builtin.command:
+            argv: "{{ ['docker', 'stop'] + containers_to_stop }}"
+          when: containers_to_stop | length > 0
+
+        - name: Rsync Docker volumes to storage
+          ansible.builtin.command:
+            argv:
+              - rsync
+              - -az
+              - --delete
+              - -e
+              - "{{ rsync_ssh }}"
+              - "{{ volumes_dir }}/"
+              - "root@{{ backup_addr }}:{{ backup_root }}/volumes/"
+          changed_when: true
+
+      always:
+        - name: Start the containers that were running before
+          ansible.builtin.command:
+            argv: "{{ ['docker', 'start'] + containers_to_stop }}"
+          when: containers_to_stop | length > 0
+
+    - name: Verify containers are running
+      ansible.builtin.command:
+        argv: [docker, ps, --format, "{{ '{{' }}.Names{{ '}}' }} - {{ '{{' }}.Status{{ '}}' }}"]
+      register: container_status
+      changed_when: false
+
+    - name: Backup summary
+      ansible.builtin.debug:
+        msg:
+          - "Destination: {{ backup_host }} ({{ backup_addr }}):{{ backup_root }}"
+          - "Stopped for the volume copy: {{ containers_to_stop | join(', ') if containers_to_stop else 'none' }}"
+          - "Kept running: {{ backup_skip_containers | join(', ') }}"
+          - "Containers running: {{ container_status.stdout_lines | length }} now, {{ running_containers.stdout_lines | length }} before"
+
+    - name: Warn when fewer containers are running than before
+      ansible.builtin.debug:
+        msg: "WARNING: {{ item }} did not come back up on {{ inventory_hostname }}"
+      loop: "{{ running_containers.stdout_lines | difference(container_status.stdout_lines | map('regex_replace', ' - .*$', '')) }}"

playbooks/bootstrap_lxc.yml

@@ -1,11 +1,16 @@
-- name: Bootstrap new Linux servers
+---
-  hosts: all
+# Base packages, the Ansible root key and sshd basics for every Ubuntu/Rocky
-  become: yes
+# guest. Run it once after a container is created (and adopted, see
+# discover_guests.yml). ssh_harden.yml then tightens sshd further through
+# /etc/ssh/sshd_config.d/00-ansible.conf.
+
+- name: Bootstrap - Install packages on Ubuntu
+  hosts: ubuntu
+  become: true
 
   tasks:
-
+    - name: Install packages via apt
-    - name: Install packages on Debian/Ubuntu
+      ansible.builtin.apt:
-      apt:
         name:
           - vim
           - htop
@@ -20,109 +25,114 @@
           - zip
           - git
           - sudo
-          - tree
-          - traceroute
-          - dnsutils
-          - ca-certificates
-          - python3
-          - python3-pip
-          - software-properties-common
-          - rsync
-          - apt-transport-https
-          - sysstat
-          - screen
-          - mtr
-          - tar
-        state: present
-        update_cache: yes
-      when: ansible_facts['os_family'] == "Debian"
-
-    - name: Ensure EPEL repo is enabled (RHEL/CentOS/AlmaLinux)
-      yum:
-        name: epel-release
-        state: present
-      when: ansible_facts['os_family'] == "RedHat"
-
-    - name: Install packages on RHEL/CentOS/AlmaLinux
-      yum:
-        name:
-          - tar
-          - vim
-          - htop
-          - nmap
-          - curl
-          - mtr
-          - wget
-          - iperf3
-          - fail2ban
-          - net-tools
-          - lsof
-          - unzip
-          - zip
-          - git
-          - sudo
           - tmux
           - tree
           - bash-completion
           - traceroute
-          - bind-utils        # provides dig, nslookup
+          - dnsutils
+          - ca-certificates
+          - python3
+          - python3-pip
+          - software-properties-common
+          - rsync
+          - sysstat
+          - screen
+          - mtr-tiny             # plain "mtr" pulls in GTK on Ubuntu
+          - tar
+        state: present
+        update_cache: true
+        cache_valid_time: 3600
+
+- name: Bootstrap - Install packages on Rocky Linux
+  hosts: rocky
+  become: true
+
+  tasks:
+    # epel-release ships in the "extras" repo on Rocky/Alma/CentOS Stream.
+    # dnf-plugins-core provides "dnf config-manager" for the next task.
+    - name: Ensure EPEL release package and dnf plugins are installed
+      ansible.builtin.dnf:
+        name:
+          - epel-release
+          - dnf-plugins-core
+        state: present
+
+    # EPEL packages may depend on CRB (CodeReady Builder). EPEL's install
+    # guide enables it together with epel-release.
+    - name: Enable the CRB repo
+      community.general.dnf_config_manager:
+        name: crb
+        state: enabled
+
+    - name: Install packages via dnf
+      ansible.builtin.dnf:
+        name:
+          # BaseOS / AppStream
+          - tar
+          - vim-enhanced         # "vim" is only a virtual provide on RHEL
+          - nmap
+          - curl
+          - mtr
+          - wget
+          - iperf3
+          - net-tools
+          - lsof
+          - unzip
+          - zip
+          - git
+          - sudo
+          - tmux
+          - tree
+          - bash-completion
+          - traceroute
+          - bind-utils
           - ca-certificates
           - python3
           - python3-pip
           - rsync
           - sysstat
+          # EPEL
+          - htop
+          - fail2ban
           - screen
         state: present
-        update_cache: yes
+        update_cache: true
-      when: ansible_facts['os_family'] == "RedHat"
 
-    - name: Ensure root's .ssh directory exists
+- name: Bootstrap - Configure SSH access
-      file:
+  hosts: ubuntu:rocky
-        path: /root/.ssh
+  become: true
-        state: directory
+  gather_facts: false
-        owner: root
-        group: root
-        mode: '0700'
 
-    - name: Set authorized_keys for root
+  vars:
-      copy:
+    bootstrap_ssh_key: "ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAINegitHaQG8i+s412h3wTWf/KvpeX1CEpYFva9MhHR+7 ansible-key"
-        dest: /root/.ssh/authorized_keys
+    sshd_service: "{{ 'ssh' if inventory_hostname in groups['ubuntu'] else 'sshd' }}"
-        content: |
-          ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAINegitHaQG8i+s412h3wTWf/KvpeX1CEpYFva9MhHR+7 ansible-key
-        owner: root
-        group: root
-        mode: '0600'
 
-    - name: Permit root login via public key
+  tasks:
-      lineinfile:
+    - name: Add the Ansible key to root's authorized_keys
-        path: /etc/ssh/sshd_config
+      ansible.posix.authorized_key:
-        regexp: '^#?PermitRootLogin\s+'
+        user: root
-        line: 'PermitRootLogin prohibit-password'
+        key: "{{ bootstrap_ssh_key }}"
-        state: present
-        backup: yes
-
-    - name: Ensure PubkeyAuthentication is enabled
-      lineinfile:
-        path: /etc/ssh/sshd_config
-        regexp: '^#?PubkeyAuthentication\s+'
-        line: 'PubkeyAuthentication yes'
         state: present
 
-    - name: Ensure PasswordAuthentication is enabled
+    # PasswordAuthentication stays on so a fresh container is still reachable
-      lineinfile:
+    # by password; ssh_harden.yml turns it off once keys are in place.
+    - name: Allow key-based root login
+      ansible.builtin.lineinfile:
         path: /etc/ssh/sshd_config
-        regexp: '^#?PasswordAuthentication\s+'
+        regexp: '^#?{{ item.option }}\s'
-        line: 'PasswordAuthentication yes'
+        line: "{{ item.option }} {{ item.value }}"
-        state: present
+        validate: /usr/sbin/sshd -t -f %s
+        backup: true
+      loop:
+        - { option: PermitRootLogin, value: prohibit-password }
+        - { option: PubkeyAuthentication, value: "yes" }
+        - { option: PasswordAuthentication, value: "yes" }
+      loop_control:
+        label: "{{ item.option }} {{ item.value }}"
+      notify: Restart sshd
 
-    - name: Restart SSH service (Debian)
+  handlers:
-      service:
+    - name: Restart sshd
-        name: ssh
+      ansible.builtin.service:
+        name: "{{ sshd_service }}"
         state: restarted
-      when: ansible_facts['os_family'] == "Debian"
-
-    - name: Restart SSH service (RHEL)
-      service:
-        name: sshd
-        state: restarted
-      when: ansible_facts['os_family'] == "RedHat"

playbooks/discover_guests.yml

@@ -0,0 +1,215 @@
+---
+# Lists every container and VM on the Proxmox host and reports which ones are
+# missing from inventories/all.yml. A guest counts as managed when its LAN IP
+# matches an inventory ansible_host, or (if its IP is unknown, e.g. stopped)
+# when its name matches an inventory hostname.
+#
+# With adopt=true the Ansible SSH key is pushed into every unmanaged RUNNING
+# container straight from the host (pct exec), so a container created by hand
+# in the Proxmox UI becomes reachable without knowing its root password.
+# Afterwards: add the printed inventory snippet, then run bootstrap_lxc.yml.
+#
+# Extra vars (optional):
+#   adopt         - default false
+#   lan_prefix    - address prefix that counts as the LAN; used to pick the right address
+#                   when a guest has several (default 192.168., i.e. any lab subnet)
+#   adopt_ssh_key - public key to push (default: the ansible-key from bootstrap_lxc.yml)
+
+- name: Discover Proxmox guests not in inventory
+  hosts: proxmox
+  become: true
+  gather_facts: false
+
+  vars:
+    adopt: false
+    lan_prefix: "192.168."
+    lan_regex: "^{{ lan_prefix | regex_escape }}"
+    adopt_ssh_key: "ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAINegitHaQG8i+s412h3wTWf/KvpeX1CEpYFva9MhHR+7 ansible-key"
+    # pct "ostype" -> inventory group
+    group_map:
+      centos: rocky
+      ubuntu: ubuntu
+      debian: ubuntu
+
+  tasks:
+    - name: Initialise working facts
+      ansible.builtin.set_fact:
+        guest_ips: {}
+        guest_ostypes: {}
+        guest_table: []
+        inventory_by_ip: {}
+
+    - name: List all guests on the host
+      ansible.builtin.command: pvesh get /cluster/resources --type vm --output-format json
+      register: resources
+      changed_when: false
+
+    - name: Parse guest list
+      ansible.builtin.set_fact:
+        guests: "{{ resources.stdout | from_json | rejectattr('template', 'defined') | sort(attribute='vmid') | list
+                    + (resources.stdout | from_json | selectattr('template', 'defined') | selectattr('template', 'equalto', 0) | list) }}"
+
+    - name: Read container OS type
+      ansible.builtin.command: pct config {{ item.vmid }}
+      loop: "{{ guests | selectattr('type', 'equalto', 'lxc') | list }}"
+      loop_control:
+        label: "{{ item.vmid }} {{ item.name | default('') }}"
+      register: lxc_configs
+      changed_when: false
+      failed_when: false
+
+    - name: Record container OS types
+      ansible.builtin.set_fact:
+        guest_ostypes: "{{ guest_ostypes | combine({(item.item.vmid | string): ostype}) }}"
+      vars:
+        ostype: "{{ (item.stdout | default('') | regex_search('(?m)^ostype: (\\S+)', '\\1')) | default(['-'], true) | first }}"
+      loop: "{{ lxc_configs.results | default([]) }}"
+      loop_control:
+        label: "{{ item.item.vmid }}"
+
+    - name: Read container IP addresses
+      ansible.builtin.command: pct exec {{ item.vmid }} -- hostname -I
+      loop: "{{ guests | selectattr('type', 'equalto', 'lxc') | selectattr('status', 'equalto', 'running') | list }}"
+      loop_control:
+        label: "{{ item.vmid }} {{ item.name | default('') }}"
+      register: lxc_ips
+      changed_when: false
+      failed_when: false
+
+    - name: Record container IP addresses
+      ansible.builtin.set_fact:
+        guest_ips: "{{ guest_ips | combine({(item.item.vmid | string): lan_ip}) }}"
+      vars:
+        all_ips: "{{ item.stdout | default('') | split }}"
+        lan_ip: "{{ all_ips | select('match', lan_regex) | first | default(all_ips | first | default('-')) }}"
+      loop: "{{ lxc_ips.results | default([]) }}"
+      loop_control:
+        label: "{{ item.item.vmid }}"
+
+    - name: Read VM IP addresses via the guest agent
+      ansible.builtin.command: qm guest cmd {{ item.vmid }} network-get-interfaces
+      loop: "{{ guests | selectattr('type', 'equalto', 'qemu') | selectattr('status', 'equalto', 'running') | list }}"
+      loop_control:
+        label: "{{ item.vmid }} {{ item.name | default('') }}"
+      register: qemu_ips
+      changed_when: false
+      failed_when: false
+
+    - name: Record VM IP addresses
+      ansible.builtin.set_fact:
+        guest_ips: "{{ guest_ips | combine({(item.item.vmid | string): lan_ip}) }}"
+      vars:
+        agent_ok: "{{ item.rc == 0 and (item.stdout | default('') | trim) is match('^\\[') }}"
+        all_ips: >-
+          {{ ((item.stdout | from_json) if agent_ok else [])
+             | map(attribute='ip-addresses', default=[]) | flatten
+             | selectattr('ip-address-type', 'equalto', 'ipv4')
+             | map(attribute='ip-address') | reject('match', '^127\\.') | list }}
+        lan_ip: "{{ all_ips | select('match', lan_regex) | first | default(all_ips | first | default('-')) }}"
+      loop: "{{ qemu_ips.results | default([]) }}"
+      loop_control:
+        label: "{{ item.item.vmid }}"
+
+    - name: Index inventory hosts by address
+      ansible.builtin.set_fact:
+        inventory_by_ip: "{{ inventory_by_ip | combine({hostvars[item].ansible_host: item}) }}"
+      loop: "{{ groups['all'] }}"
+      when: hostvars[item].ansible_host is defined
+
+    - name: Assemble guest table
+      ansible.builtin.set_fact:
+        guest_table: "{{ guest_table + [guest_entry] }}"
+      vars:
+        vmid: "{{ item.vmid | string }}"
+        guest_name: "{{ item.name | default('-') }}"
+        guest_ip: "{{ guest_ips[vmid] | default('-') }}"
+        guest_ostype: "{{ guest_ostypes[vmid] | default('-') }}"
+        inventory_match: "{{ inventory_by_ip[guest_ip] | default(guest_name if guest_name in groups['all'] else '') }}"
+        guest_entry:
+          vmid: "{{ vmid }}"
+          type: "{{ item.type }}"
+          status: "{{ item.status }}"
+          name: "{{ guest_name }}"
+          ip: "{{ guest_ip }}"
+          group: "{{ (group_map[guest_ostype] | default(guest_ostype)) if item.type == 'lxc' else 'ubuntu (verify) + vms' }}"
+          inventory_name: "{{ inventory_match }}"
+      loop: "{{ guests }}"
+      loop_control:
+        label: "{{ item.vmid }} {{ item.name | default('') }}"
+
+    - name: Collect unmanaged guests
+      ansible.builtin.set_fact:
+        unmanaged: "{{ guest_table | selectattr('inventory_name', 'equalto', '') | list }}"
+
+    - name: Print guest report
+      ansible.builtin.debug:
+        msg: |
+          ==========================================================================================
+           PROXMOX GUESTS vs INVENTORY  (host: {{ inventory_hostname }})
+          ==========================================================================================
+           VMID  TYPE  STATUS   NAME               IP               GROUP     INVENTORY
+          {% for g in guest_table %}
+           {{ '%-5s %-5s %-8s %-18s %-16s %-9s %s' | format(g.vmid, g.type, g.status, g.name, g.ip, g.group.split(' ')[0],
+                ('managed as ' ~ g.inventory_name) if g.inventory_name else 'NOT IN INVENTORY') }}
+          {% endfor %}
+          ------------------------------------------------------------------------------------------
+           TOTAL: {{ guest_table | length }}   MANAGED: {{ (guest_table | length) - (unmanaged | length) }}   UNMANAGED: {{ unmanaged | length }}
+          ==========================================================================================
+
+    - name: Warn on unmanaged guests
+      ansible.builtin.debug:
+        msg: "WARNING: {{ item.type }} {{ item.vmid }} ({{ item.name }}, {{ item.status }}, ip {{ item.ip }}) is not in inventories/all.yml"
+      loop: "{{ unmanaged }}"
+      loop_control:
+        label: "{{ item.vmid }} {{ item.name }}"
+
+    - name: Suggested inventory entries for unmanaged guests
+      ansible.builtin.debug:
+        msg: |
+          Add to inventories/all.yml under the OS group shown, and list the name under lxc or vms:
+          {% for g in unmanaged %}
+            # {{ g.type }} {{ g.vmid }} -> group: {{ g.group }}
+            {{ g.name }}:
+              ansible_host: {{ g.ip if g.ip != '-' else lan_prefix ~ 'XXX   # start the guest and rerun to discover its address' }}
+              critical_services: []
+          {% endfor %}
+      when: unmanaged | length > 0
+
+    - name: Push the Ansible SSH key into unmanaged running containers
+      ansible.builtin.shell: >-
+        pct exec {{ item.vmid }} -- sh -c
+        'mkdir -p /root/.ssh &&
+        chmod 700 /root/.ssh &&
+        touch /root/.ssh/authorized_keys &&
+        chmod 600 /root/.ssh/authorized_keys &&
+        if ! grep -qxF "{{ adopt_ssh_key }}" /root/.ssh/authorized_keys; then
+        echo "{{ adopt_ssh_key }}" >> /root/.ssh/authorized_keys && echo KEY_ADDED; fi;
+        command -v sshd >/dev/null 2>&1 && echo SSHD_OK || echo NO_SSHD'
+      loop: "{{ unmanaged | selectattr('type', 'equalto', 'lxc') | selectattr('status', 'equalto', 'running') | list }}"
+      loop_control:
+        label: "{{ item.vmid }} {{ item.name }}"
+      register: adopt_result
+      changed_when: "'KEY_ADDED' in adopt_result.stdout"
+      when: adopt | bool
+
+    - name: Adoption summary
+      ansible.builtin.debug:
+        msg: >-
+          {{ item.item.vmid }} ({{ item.item.name }}) -
+          key {{ 'ADDED' if 'KEY_ADDED' in item.stdout else 'already present' }},
+          {{ 'sshd present' if 'SSHD_OK' in item.stdout else 'WARNING: no sshd in the container - install openssh-server before Ansible can reach it' }}
+          - next: add it to inventories/all.yml and run bootstrap_lxc.yml
+      loop: "{{ adopt_result.results | default([]) }}"
+      loop_control:
+        label: "{{ item.item.vmid }}"
+      when:
+        - adopt | bool
+        - item.rc is defined
+
+    - name: Note about unmanaged VMs
+      ansible.builtin.debug:
+        msg: "NOTE: {{ item.type }} {{ item.vmid }} ({{ item.name }}) is a VM; adopt only handles containers. Add its key via cloud-init or the console."
+      loop: "{{ unmanaged | selectattr('type', 'equalto', 'qemu') | list }}"
+      loop_control:
+        label: "{{ item.vmid }} {{ item.name }}"
+      when: adopt | bool

playbooks/health_check.yml

@@ -0,0 +1,148 @@
+---
+# Disk and RAM report for every reachable host. Filesystems above
+# disk_warn_pct and hosts that did not answer are printed as warnings and,
+# when Telegram credentials are present, sent as one Telegram message per run.
+#
+# Extra vars (optional):
+#   disk_warn_pct      - default 80
+#   telegram_bot_token / TELEGRAM_BOT_TOKEN - extra var or environment variable
+#   telegram_chat_id / TELEGRAM_CHAT_ID     - extra var or environment variable
+# Without both Telegram values the playbook only prints the report.
+
+- name: Check Host Availability
+  hosts: all
+  gather_facts: false
+  ignore_unreachable: true
+
+  tasks:
+    - name: Ping hosts to check availability
+      ansible.builtin.ping:
+      register: ping_result
+      ignore_errors: true
+      ignore_unreachable: true
+
+    - name: Add reachable hosts to group
+      ansible.builtin.group_by:
+        key: reachable_hosts
+      when: ping_result is reachable and ping_result is succeeded
+
+    - name: Report offline hosts
+      ansible.builtin.debug:
+        msg: "{{ inventory_hostname }} - OFFLINE"
+      when: ping_result is unreachable or ping_result is failed
+      delegate_to: localhost
+
+- name: Disk and RAM Usage Report
+  hosts: reachable_hosts
+  become: true
+  gather_facts: true
+
+  vars:
+    disk_warn_pct: 80
+
+  tasks:
+    - name: Get disk usage (excluding snap, overlay, tmpfs)
+      ansible.builtin.shell: df -h --output=target,size,used,avail,pcent -x tmpfs -x devtmpfs -x squashfs -x overlay -x efivarfs | tail -n +2
+      register: disk_usage
+      changed_when: false
+
+    - name: Calculate memory usage
+      ansible.builtin.set_fact:
+        mem_total_mb: "{{ ansible_memtotal_mb }}"
+        mem_used_mb: "{{ ansible_memory_mb.real.used }}"
+        mem_pct: "{{ ((ansible_memory_mb.real.used / ansible_memtotal_mb) * 100) | round(1) }}"
+        swap_total_mb: "{{ ansible_swaptotal_mb }}"
+        swap_used_mb: "{{ ansible_swaptotal_mb - ansible_swapfree_mb }}"
+
+    - name: Print report
+      ansible.builtin.debug:
+        msg: |
+          ======================================================
+           HOST: {{ inventory_hostname }} ({{ ansible_host }})
+           OS:   {{ ansible_distribution }} {{ ansible_distribution_version }}
+          ======================================================
+           RAM:  {{ mem_used_mb }}MB / {{ mem_total_mb }}MB ({{ mem_pct }}%)
+           SWAP: {{ swap_used_mb }}MB / {{ swap_total_mb }}MB
+          ------------------------------------------------------
+           DISK:
+          {{ disk_usage.stdout }}
+          ======================================================
+
+    - name: Collect filesystems above {{ disk_warn_pct }}%
+      ansible.builtin.set_fact:
+        disk_alerts: "{{ disk_alerts | default([]) + [item.split()[0] ~ ' ' ~ item.split()[-1]] }}"
+      loop: "{{ disk_usage.stdout_lines }}"
+      when: (item.split()[-1] | replace('%', '') | int) > disk_warn_pct | int
+
+    - name: Warn on high disk usage
+      ansible.builtin.debug:
+        msg: "WARNING: {{ item }} used on {{ inventory_hostname }} (threshold {{ disk_warn_pct }}%)"
+      loop: "{{ disk_alerts | default([]) }}"
+
+- name: Telegram alert
+  hosts: localhost
+  connection: local
+  gather_facts: false
+
+  vars:
+    disk_warn_pct: 80
+    # Accepted from a Semaphore variable group as extra vars (telegram_bot_token
+    # or TELEGRAM_BOT_TOKEN) or as environment variables (TELEGRAM_BOT_TOKEN).
+    telegram_bot_token: "{{ TELEGRAM_BOT_TOKEN | default(lookup('env', 'TELEGRAM_BOT_TOKEN'), true) | string }}"
+    telegram_chat_id: "{{ TELEGRAM_CHAT_ID | default(lookup('env', 'TELEGRAM_CHAT_ID'), true) | string }}"
+    telegram_configured: "{{ telegram_bot_token | string | length > 0 and telegram_chat_id | string | length > 0 }}"
+    offline_hosts: "{{ groups['all'] | difference(groups['reachable_hosts'] | default([])) }}"
+
+  tasks:
+    - name: Build alert text
+      ansible.builtin.set_fact:
+        alert_text: "{{ alert_body | default('', true) | trim }}"   # an empty template renders as None
+      vars:
+        alert_body: |-
+          {% for h in groups['reachable_hosts'] | default([]) %}
+          {% for fs in hostvars[h].disk_alerts | default([]) %}
+          DISK {{ h }}: {{ fs }} used (threshold {{ disk_warn_pct }}%)
+          {% endfor %}
+          {% endfor %}
+          {% for h in offline_hosts %}
+          OFFLINE {{ h }} did not answer
+          {% endfor %}
+
+    - name: Nothing to alert
+      ansible.builtin.debug:
+        msg: "Every host reachable and all filesystems at or below {{ disk_warn_pct }}% - no alert"
+      when: alert_text | length == 0
+
+    - name: Print alerts when Telegram is not configured
+      ansible.builtin.debug:
+        msg: |-
+          ALERTS (Telegram not configured - bot token {{ 'set' if telegram_bot_token | string | length > 0 else 'MISSING' }},
+          chat id {{ 'set' if telegram_chat_id | string | length > 0 else 'MISSING' }};
+          pass TELEGRAM_BOT_TOKEN and TELEGRAM_CHAT_ID through the variable group attached to this template)
+          {{ alert_text }}
+      when:
+        - alert_text | length > 0
+        - not telegram_configured
+
+    - name: Send Telegram alert
+      ansible.builtin.uri:
+        url: "https://api.telegram.org/bot{{ telegram_bot_token }}/sendMessage"
+        method: POST
+        body_format: json
+        body:
+          chat_id: "{{ telegram_chat_id }}"
+          text: "Homelab health check {{ now().strftime('%Y-%m-%d %H:%M') }}\n{{ alert_text }}"
+        status_code: [200]
+      register: telegram_result
+      failed_when: false
+      no_log: true          # the bot token is part of the URL
+      when:
+        - alert_text | length > 0
+        - telegram_configured
+
+    - name: Telegram delivery status
+      ansible.builtin.assert:
+        that: telegram_result.status == 200
+        fail_msg: "Telegram API returned {{ telegram_result.status }}: {{ telegram_result.json.description | default('no response') }}"
+        success_msg: "Telegram alert sent ({{ alert_text.split('\n') | length }} line(s))"
+      when: telegram_result is not skipped

playbooks/new_lxc.yml

@@ -1,73 +1,74 @@
 ---
-- name: Build Dynamic Rocky Linux LXC Container
+# Creates a Rocky Linux 9 container through the Proxmox API and starts it.
+# Runs on the Semaphore host (localhost); proxmoxer must be importable by the
+# Python that runs Ansible there.
+#
+# Extra vars (Semaphore environment / variable group):
+#   proxmox_api_user      - e.g. root@pam            (required)
+#   proxmox_api_password  - that user's password     (this, or a token)
+#   proxmox_api_token_id / proxmox_api_token_secret  (API token instead of a password)
+#   proxmox_api_host      - defaults to the pve entry in the inventory
+#
+# Optional: lxc_hostname, lxc_template, lxc_cores, lxc_memory, lxc_storage,
+# lxc_disk_size, lxc_bridge, lxc_dns, target_node.
+
+- name: Build a Rocky Linux LXC container
   hosts: localhost
+  connection: local
   gather_facts: false
 
   vars:
-    # --- Container Specifications ---
+    proxmox_api_host: "{{ hostvars['pve'].ansible_host }}"
-    lxc_hostname: "new_lxc"
+    target_node: mini-pc
-    lxc_template: "local:vztmpl/rockylinux-9-default_20240912_amd64.tar.xz" 
+    lxc_hostname: new-lxc              # underscores are not valid in a hostname
+    lxc_template: local:vztmpl/rockylinux-9-default_20240912_amd64.tar.xz
     lxc_cores: 2
     lxc_memory: 1024
-    lxc_storage: "TB"
+    lxc_storage: TB
     lxc_disk_size: 10
-    lxc_bridge: "vmbr0"
+    lxc_bridge: vmbr0
-    lxc_dns: "192.168.2.2"
+    lxc_dns: "{{ hostvars['pihole'].ansible_host }}"
-    target_node: "pve" 
-    
-    # --- Security & Access ---
     lxc_ssh_key: "ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAINegitHaQG8i+s412h3wTWf/KvpeX1CEpYFva9MhHR+7 ansible-key"
 
+  module_defaults:
+    community.proxmox.proxmox:
+      api_host: "{{ proxmox_api_host }}"
+      api_user: "{{ proxmox_api_user }}"
+      api_password: "{{ proxmox_api_password | default(omit) }}"
+      api_token_id: "{{ proxmox_api_token_id | default(omit) }}"
+      api_token_secret: "{{ proxmox_api_token_secret | default(omit) }}"
+      validate_certs: false
+      timeout: 300               # template extraction takes longer than the 30 s default
+
   tasks:
-    # ========================================================================
+    - name: Ensure proxmoxer is available to the controller's Python
-    # NEW FIX: Auto-install missing dependencies into Semaphore's isolated venv
+      ansible.builtin.command: "{{ ansible_playbook_python }} -m pip install proxmoxer requests"
-    # ========================================================================
+      register: pip_install
-    - name: Ensure Proxmox API dependencies are installed in Semaphore
+      changed_when: "'Successfully installed' in pip_install.stdout"
-      ansible.builtin.shell: "{{ ansible_playbook_python }} -m pip install proxmoxer requests"
-      changed_when: false
-    # ========================================================================
 
-    - name: Get the next available VMID from the Proxmox cluster
+    - name: Create the container (Proxmox assigns the next free VMID)
-      ansible.builtin.uri:
+      community.proxmox.proxmox:
-        url: "https://{{ proxmox_api_host }}:8006/api2/json/cluster/nextid"
-        method: GET
-        headers:
-          Authorization: "PVEAPIToken {{ proxmox_api_user }}={{ proxmox_api_password }}"
-        validate_certs: false
-        return_content: true
-      register: nextid_response
-
-    - name: Extract the dynamic VMID from the API response
-      ansible.builtin.set_fact:
-        dynamic_vmid: "{{ nextid_response.json.data }}"
-
-    - name: Create the Rocky Linux LXC container
-      community.general.proxmox:
-        api_host: "{{ proxmox_api_host }}"
-        api_user: "{{ proxmox_api_user }}"
-        api_password: "{{ proxmox_api_password }}"
-        
         node: "{{ target_node }}"
-        vmid: "{{ dynamic_vmid }}"
         hostname: "{{ lxc_hostname }}"
         ostemplate: "{{ lxc_template }}"
         cores: "{{ lxc_cores }}"
         memory: "{{ lxc_memory }}"
         disk: "{{ lxc_storage }}:{{ lxc_disk_size }}"
-        netif: '{"net0":"name=eth0,bridge={{ lxc_bridge }},ip=dhcp"}'
+        netif:
+          net0: "name=eth0,bridge={{ lxc_bridge }},ip=dhcp"
         nameserver: "{{ lxc_dns }}"
         pubkey: "{{ lxc_ssh_key }}"
         unprivileged: true
         state: present
+      register: lxc_create
 
-    - name: Ensure the new LXC container is powered on
+    - name: Start the container
-      community.general.proxmox:
+      community.proxmox.proxmox:
-        api_host: "{{ proxmox_api_host }}"
+        vmid: "{{ lxc_create.vmid }}"
-        api_user: "{{ proxmox_api_user }}"
-        api_password: "{{ proxmox_api_password }}"
-        vmid: "{{ dynamic_vmid }}"
         state: started
-        
+
-    - name: Output the assigned VMID to the Semaphore UI logs
+    - name: Report
       ansible.builtin.debug:
-        msg: "Successfully built {{ lxc_hostname }} (Rocky Linux 9) with dynamic ID: {{ dynamic_vmid }}"
+        msg: >-
+          Built {{ lxc_hostname }} (Rocky Linux 9) as VMID {{ lxc_create.vmid }} on {{ target_node }}.
+          Next: run discover_guests.yml with adopt=true, add it to inventories/all.yml, then bootstrap_lxc.yml.

playbooks/nextcloud-upgrade.yml

@@ -1,140 +0,0 @@
----
-- name: Safe Nextcloud Upgrade Playbook
-  hosts: nextcloud
-  become: yes
-  vars:
-    nc_path: /var/www/nextcloud
-    php_bin: /usr/bin/php
-
-  tasks:
-
-  - name: Pause playbook for 3 seconds
-    pause:
-      seconds: 3
-      prompt: "Press Ctrl+C then A to abort if no backups exist."
-
-  ###########################################################################
-  # 1 - CHECK CURRENT INSTALLED VERSION
-  ###########################################################################
-  - name: 1.1 - Get current NC status
-    command: "{{ php_bin }} occ status"
-    args:
-      chdir: "{{ nc_path }}"
-    register: nc_status
-
-  - name: 1.2 - Extract installed version
-    set_fact:
-      installed_version: "{{ nc_status.stdout | regex_search('versionstring: (.*)', '\\1') }}"
-
-  - name: 1.3 - Show installed version
-    debug:
-      msg: "Current installed version: {{ installed_version }}"
-
-  ###########################################################################
-  # 2 - CHECK LATEST AVAILABLE VERSION (via updater.phar)
-  ###########################################################################
-  - name: 2.1 - Check for available update
-    command: "{{ php_bin }} updater/updater.phar --check"
-    args:
-      chdir: "{{ nc_path }}"
-    register: update_check
-    failed_when: false  # we want to parse output even if exit code != 0
-
-  - name: 2.2 - Parse latest available version
-    set_fact:
-      available_version: "{{ update_check.stdout | regex_search('Update to version (.*) available', '\\1') }}"
-    when: update_check.stdout is search('Update to version')
-
-  - name: 2.3 - Show available version
-    debug:
-      msg: "Latest available version: {{ available_version | default('NONE') }}"
-
-  ###########################################################################
-  # 2.4 - SKIP IF ALREADY UP TO DATE
-  ###########################################################################
-  - name: 2.4 - Skip upgrade if already on latest version
-    debug:
-      msg: "Nextcloud is already up to date ({{ installed_version }}). Skipping upgrade."
-    when: available_version is not defined
-    tags: skip_upgrade
-    changed_when: false
-
-  - name: End play if up to date
-    meta: end_play
-    when: available_version is not defined
-
-  ###########################################################################
-  # 3 - DISABLE APPS + MAINTENANCE MODE
-  ###########################################################################
-  - name: 3.1 - Enable maintenance mode
-    command: "{{ php_bin }} occ maintenance:mode --on"
-    args:
-      chdir: "{{ nc_path }}"
-
-  ###########################################################################
-  # 3.2 - RUN UPDATER WITH RETRIES
-  ###########################################################################
-  - name: 3.2 - Run Nextcloud CLI updater (retry)
-    command: "{{ php_bin }} updater/updater.phar --no-interaction"
-    args:
-      chdir: "{{ nc_path }}"
-    register: updater_output
-    retries: 3
-    delay: 5
-    until: updater_output.rc == 0
-    failed_when: updater_output.rc != 0
-
-  ###########################################################################
-  # 3.3 - OCC UPGRADE (DB MIGRATIONS)
-  ###########################################################################
-  - block:
-    - name: 3.3 - Run occ upgrade
-      command: "{{ php_bin }} occ upgrade"
-      args:
-        chdir: "{{ nc_path }}"
-      register: upgrade_run
-      failed_when: >
-        upgrade_run.rc != 0 and
-        "already in process" not in upgrade_run.stdout and
-        "maintenance mode" not in upgrade_run.stdout
-
-    rescue:
-      - name: 3.3R - Handle locked upgrade
-        debug:
-          msg: "Upgrade appears stuck. Removing maintenance lock and retrying."
-
-      - name: Remove maintenance mode flag from config.php
-        lineinfile:
-          path: "{{ nc_path }}/config/config.php"
-          regexp: "'maintenance' => true"
-          line: "  'maintenance' => false,"
-          backrefs: yes
-
-      - name: Retry OCC upgrade
-        command: "{{ php_bin }} occ upgrade"
-        args:
-          chdir: "{{ nc_path }}"
-        register: upgrade_retry
-        failed_when: upgrade_retry.rc != 0
-
-  ###########################################################################
-  # 4 - ALWAYS DISABLE MAINTENANCE MODE
-  ###########################################################################
-  - name: ALWAYS disable maintenance mode
-    command: "{{ php_bin }} occ maintenance:mode --off"
-    args:
-      chdir: "{{ nc_path }}"
-    ignore_errors: true
-
-  ###########################################################################
-  # 5 - FINAL RESULT
-  ###########################################################################
-  - name: Show final upgrade result
-    debug:
-      msg: |
-        ✔ Nextcloud upgrade completed.
-        Installed: {{ installed_version }}
-        Updated to: {{ available_version }}
-        Check logs if issues appear:
-         - {{ nc_path }}/data/updater.log
-         - {{ nc_path }}/data/nextcloud.log

playbooks/ping.yml

@@ -1,7 +1,7 @@
+---
 - name: Ping all servers
   hosts: all
-  gather_facts: no
+  gather_facts: false
   tasks:
     - name: Test connectivity with ping
       ansible.builtin.ping:
-

playbooks/proxmox-backup.yml

@@ -1,60 +1,81 @@
 ---
+# vzdump of selected guests on the Proxmox host. vzdump prunes old dumps of
+# each guest itself (keep-last), so nothing else touches the dump directory.
+# Every run is appended to {{ backup_dir }}/proxmox_backup.log on the host.
+#
+# backup_mode "stop" shuts the guest down for the whole backup. Never list
+# the guest that runs Semaphore with mode stop: the job would stop itself
+# half way through. Use "snapshot" for that one (falls back to suspend on
+# storage without snapshot support).
+#
+# Extra vars (optional):
+#   guests       - list of { id, type, name }
+#   keep_last    - dumps kept per guest (default 3)
+#   backup_mode  - stop | snapshot | suspend (default stop)
+#   storage      - target PVE storage id (default local)
+#   compression  - zstd | gzip | lzo (default zstd)
+
 - name: Backup selected Proxmox VMs and LXCs
   hosts: proxmox
   become: true
+  gather_facts: false
+
   vars:
-    backup_dir: "/var/lib/vz/dump"
+    backup_dir: /var/lib/vz/dump
-    storage: "local"
+    storage: local
-    compression: "zstd"
+    compression: zstd
-    maxfiles: 3
+    keep_last: 3
+    backup_mode: stop
     guests:
-      #- { id: 104, type: "lxc", name: "jellyfin" }
+      # - { id: 104, type: lxc, name: courses }
-      - { id: 108, type: "lxc", name: "docker" }
+      - { id: 108, type: lxc, name: docker }
-      #- { id: 101, type: "qemu", name: "pnetlab" }
+      # - { id: 101, type: qemu, name: pnetlab }
 
   tasks:
-    - name: Ensure backup directory exists
+    - name: Ensure the log directory exists
-      file:
+      ansible.builtin.file:
         path: "{{ backup_dir }}"
         state: directory
         mode: '0755'
 
-    - name: Run vzdump backup with logging
+    - name: Run vzdump
-      shell: |
+      ansible.builtin.shell: |
+        set -o pipefail
         LOGFILE="{{ backup_dir }}/proxmox_backup.log"
         START=$(date +%s)
-        echo "[$(date '+%Y-%m-%d %H:%M:%S')] Starting backup of {{ item.name }} ({{ item.type }}) [ID: {{ item.id }}]" >> "$LOGFILE"
+        echo "[$(date '+%Y-%m-%d %H:%M:%S')] START   {{ item.name }} ({{ item.type }} {{ item.id }}) mode={{ backup_mode }}" >> "$LOGFILE"
-
+        OUTPUT=$(vzdump {{ item.id }} --mode {{ backup_mode }} --compress {{ compression }} \
-        OUTPUT=$(vzdump {{ item.id }} --mode stop --compress {{ compression }} --storage {{ storage }} --remove 0 2>&1)
+                 --storage {{ storage }} --prune-backups keep-last={{ keep_last }} 2>&1)
         STATUS=$?
-
+        DURATION=$(( $(date +%s) - START ))
-        END=$(date +%s)
+        if [ "$STATUS" -eq 0 ]; then
-        DURATION=$((END - START))
+          echo "[$(date '+%Y-%m-%d %H:%M:%S')] SUCCESS {{ item.name }} ({{ item.id }}) ${DURATION}s" >> "$LOGFILE"
-
+          echo "$OUTPUT" | grep -E 'archive file size|pruned|removing backup' || true
-        if [ $STATUS -eq 0 ]; then
-          echo "[$(date '+%Y-%m-%d %H:%M:%S')] ✅ SUCCESS - {{ item.name }} [{{ item.id }}] - Duration: ${DURATION}s" >> "$LOGFILE"
         else
-          echo "[$(date '+%Y-%m-%d %H:%M:%S')] ❌ FAILURE - {{ item.name }} [{{ item.id }}] - Duration: ${DURATION}s" >> "$LOGFILE"
+          echo "[$(date '+%Y-%m-%d %H:%M:%S')] FAILURE {{ item.name }} ({{ item.id }}) ${DURATION}s" >> "$LOGFILE"
-          echo "---- ERROR OUTPUT ----" >> "$LOGFILE"
+          printf -- '---- vzdump output ----\n%s\n-----------------------\n' "$OUTPUT" >> "$LOGFILE"
-          echo "$OUTPUT" >> "$LOGFILE"
+          echo "$OUTPUT" | tail -n 20
-          echo "----------------------" >> "$LOGFILE"
         fi
+        exit "$STATUS"
       args:
         executable: /bin/bash
       loop: "{{ guests }}"
+      loop_control:
+        label: "{{ item.name }} ({{ item.type }} {{ item.id }})"
+      changed_when: true
 
-    - name: Rename backup files to include guest name
+    - name: List the dumps kept for each guest
-      shell: |
+      ansible.builtin.command: pvesm list {{ storage }} --content backup --vmid {{ item.id }}
-        latest_file=$(ls -t {{ backup_dir }}/vzdump-{{ item.type }}-{{ item.id }}-*.{{ compression }} | head -n 1)
-        new_file="{{ backup_dir }}/{{ item.name }}-{{ item.type }}-{{ item.id }}-{{ '%Y%m%d-%H%M%S' | strftime(ansible_date_time.epoch | int) }}.{{ compression }}"
-        cp "$latest_file" "$new_file"
-      args:
-        executable: /bin/bash
       loop: "{{ guests }}"
+      loop_control:
+        label: "{{ item.name }} ({{ item.id }})"
+      register: dump_lists
+      changed_when: false
 
-    - name: Prune old backups (keep last {{ maxfiles }})
+    - name: Backup summary
-      shell: |
+      ansible.builtin.debug:
-        ls -tp {{ backup_dir }}/{{ item.name }}-{{ item.type }}-{{ item.id }}-*{{ compression }} | grep -v '/$' | tail -n +{{ maxfiles + 1 }} | xargs -r rm --
+        msg: |
-      args:
+          {% for r in dump_lists.results %}
-        executable: /bin/bash
+          {{ r.item.name }} ({{ r.item.type }} {{ r.item.id }}) - mode {{ backup_mode }}, keeping the last {{ keep_last }}:
-      loop: "{{ guests }}"
+          {{ r.stdout }}
+          {% endfor %}

playbooks/reboot_check.yml

@@ -0,0 +1,146 @@
+---
+# Reports which Ubuntu/Rocky guests need a reboot and, only when asked,
+# reboots them. LXC containers restart via Proxmox when they reboot themselves.
+#
+# Extra vars (optional):
+#   do_reboot      - default false (report only)
+#   reboot_exclude - hosts never rebooted by this playbook (default: [docker],
+#                    which most likely runs Semaphore itself)
+
+- name: Check Host Availability
+  hosts: ubuntu:rocky
+  gather_facts: false
+  ignore_unreachable: true
+
+  tasks:
+    - name: Ping hosts to check availability
+      ansible.builtin.ping:
+      register: ping_result
+      ignore_errors: true
+      ignore_unreachable: true
+
+    - name: Add reachable hosts to group
+      ansible.builtin.group_by:
+        key: reachable_hosts
+      when: ping_result is reachable and ping_result is succeeded
+
+    - name: Report offline hosts
+      ansible.builtin.debug:
+        msg: "{{ inventory_hostname }} - OFFLINE"
+      when: ping_result is unreachable or ping_result is failed
+      delegate_to: localhost
+
+- name: Reboot Check - Ubuntu
+  hosts: ubuntu:&reachable_hosts
+  become: true
+  gather_facts: true
+
+  tasks:
+    - name: Check for the reboot-required flag
+      ansible.builtin.stat:
+        path: /var/run/reboot-required
+      register: reboot_flag
+
+    - name: Read the packages that requested a reboot
+      ansible.builtin.slurp:
+        src: /var/run/reboot-required.pkgs
+      register: reboot_pkgs
+      when: reboot_flag.stat.exists
+      failed_when: false
+
+    - name: Record reboot status
+      ansible.builtin.set_fact:
+        reboot_needed: "{{ reboot_flag.stat.exists }}"
+        reboot_reason: "{{ (reboot_pkgs.content | default('') | b64decode | trim | replace('\n', ', ')) if reboot_flag.stat.exists else '' }}"
+
+- name: Reboot Check - Rocky Linux
+  hosts: rocky:&reachable_hosts
+  become: true
+  gather_facts: true
+
+  tasks:
+    - name: Ensure needs-restarting is available
+      ansible.builtin.dnf:
+        name: yum-utils
+        state: present
+
+    # needs-restarting -r compares package install times against the kernel boot
+    # time, which inside an LXC container is the host's boot time, so the flag
+    # would never clear. Containers use the process-based check instead.
+    - name: Check whether a reboot is required (VM)
+      ansible.builtin.command: needs-restarting -r
+      register: needs_reboot
+      failed_when: needs_reboot.rc not in [0, 1]
+      changed_when: false
+      when: ansible_virtualization_type != "lxc"
+
+    - name: Check for processes using updated libraries (LXC)
+      ansible.builtin.command: needs-restarting
+      register: stale_procs
+      failed_when: stale_procs.rc not in [0, 1]
+      changed_when: false
+      when: ansible_virtualization_type == "lxc"
+
+    - name: Record reboot status
+      ansible.builtin.set_fact:
+        reboot_needed: >-
+          {{ (needs_reboot.rc == 1) if ansible_virtualization_type != 'lxc'
+             else ((stale_procs.stdout | trim | length) > 0) }}
+        reboot_reason: >-
+          {{ (needs_reboot.stdout | default('') | trim | replace('\n', '; ')) if ansible_virtualization_type != 'lxc'
+             else ((stale_procs.stdout_lines | default([]) | length | string) ~ ' process(es) using updated libraries') }}
+
+- name: Reboot Check - Report
+  hosts: reachable_hosts
+  gather_facts: false
+
+  tasks:
+    - name: Print reboot status
+      ansible.builtin.debug:
+        msg: >-
+          {{ inventory_hostname }} ({{ ansible_virtualization_type | default('unknown') }}) -
+          REBOOT NEEDED: {{ 'YES' if reboot_needed | default(false) | bool else 'no' }}
+          {{ ('- ' ~ reboot_reason) if (reboot_needed | default(false) | bool and reboot_reason | default('') | length > 0) else '' }}
+
+- name: Reboot Check - Reboot hosts that need it
+  hosts: reachable_hosts
+  become: true
+  gather_facts: false
+
+  vars:
+    do_reboot: false
+    reboot_exclude:
+      - docker
+
+  tasks:
+    - name: Report hosts skipped because they are excluded
+      ansible.builtin.debug:
+        msg: "SKIPPED: {{ inventory_hostname }} needs a reboot but is listed in reboot_exclude"
+      when:
+        - do_reboot | bool
+        - reboot_needed | default(false) | bool
+        - inventory_hostname in reboot_exclude
+
+    # Inside a container the default boot_id belongs to the host and never
+    # changes, so the reboot module would wait forever. PID 1's start time does
+    # change when the container restarts.
+    - name: Reboot LXC container
+      ansible.builtin.reboot:
+        boot_time_command: "awk '{print $22}' /proc/1/stat"
+        reboot_timeout: 300
+        msg: "Reboot initiated by Ansible (reboot_check.yml)"
+      when:
+        - do_reboot | bool
+        - reboot_needed | default(false) | bool
+        - inventory_hostname not in reboot_exclude
+        - ansible_virtualization_type == "lxc"
+
+    - name: Reboot VM
+      ansible.builtin.reboot:
+        reboot_timeout: 300
+        msg: "Reboot initiated by Ansible (reboot_check.yml)"
+      when:
+        - do_reboot | bool
+        - reboot_needed | default(false) | bool
+        - inventory_hostname not in reboot_exclude
+        - ansible_virtualization_type != "lxc"

playbooks/service_check.yml

@@ -0,0 +1,151 @@
+---
+# Verifies the critical services listed per host in the inventory
+# (critical_services) and optionally starts any that are stopped. Services
+# found not running are sent as one Telegram message per run when Telegram
+# credentials are present.
+#
+# Extra vars (optional):
+#   restart_stopped    - default false; when true, start services found stopped
+#   telegram_bot_token / TELEGRAM_BOT_TOKEN - extra var or environment variable
+#   telegram_chat_id / TELEGRAM_CHAT_ID     - extra var or environment variable
+
+- name: Check Host Availability
+  hosts: ubuntu:rocky
+  gather_facts: false
+  ignore_unreachable: true
+
+  tasks:
+    - name: Ping hosts to check availability
+      ansible.builtin.ping:
+      register: ping_result
+      ignore_errors: true
+      ignore_unreachable: true
+
+    - name: Add reachable hosts to group
+      ansible.builtin.group_by:
+        key: reachable_hosts
+      when: ping_result is reachable and ping_result is succeeded
+
+    - name: Report offline hosts
+      ansible.builtin.debug:
+        msg: "{{ inventory_hostname }} - OFFLINE"
+      when: ping_result is unreachable or ping_result is failed
+      delegate_to: localhost
+
+- name: Service Check
+  hosts: reachable_hosts
+  become: true
+  gather_facts: false
+
+  vars:
+    restart_stopped: false
+
+  tasks:
+    - name: Gather service facts
+      ansible.builtin.service_facts:
+
+    - name: Evaluate critical services
+      ansible.builtin.set_fact:
+        service_states: >-
+          {{ service_states | default([])
+             + [{'name': item, 'state': (ansible_facts.services[item ~ '.service'].state | default('not-found'))}] }}
+      loop: "{{ critical_services | default([]) }}"
+
+    - name: Print report
+      ansible.builtin.debug:
+        msg: |
+          ======================================================
+           HOST: {{ inventory_hostname }} ({{ ansible_host }})
+          ======================================================
+          {% if service_states | default([]) | length == 0 %}
+           no critical services defined in inventory
+          {% else %}
+          {% for svc in service_states %}
+           {{ '%-24s' | format(svc.name) }} {{ 'RUNNING' if svc.state == 'running' else ('NOT FOUND' if svc.state == 'not-found' else svc.state | upper) }}
+          {% endfor %}
+          {% endif %}
+          ======================================================
+
+    - name: Warn on services that are not running
+      ansible.builtin.debug:
+        msg: "WARNING: {{ item.name }} is {{ 'not installed' if item.state == 'not-found' else item.state }} on {{ inventory_hostname }}"
+      loop: "{{ service_states | default([]) }}"
+      loop_control:
+        label: "{{ item.name }}"
+      when: item.state != 'running'
+
+    - name: Start stopped services
+      ansible.builtin.service:
+        name: "{{ item.name }}"
+        state: started
+      loop: "{{ service_states | default([]) }}"
+      loop_control:
+        label: "{{ item.name }}"
+      when:
+        - restart_stopped | bool
+        - item.state not in ['running', 'not-found']
+
+- name: Telegram alert
+  hosts: localhost
+  connection: local
+  gather_facts: false
+
+  vars:
+    restart_stopped: false
+    # Accepted from a Semaphore variable group as extra vars (telegram_bot_token
+    # or TELEGRAM_BOT_TOKEN) or as environment variables (TELEGRAM_BOT_TOKEN).
+    telegram_bot_token: "{{ TELEGRAM_BOT_TOKEN | default(lookup('env', 'TELEGRAM_BOT_TOKEN'), true) | string }}"
+    telegram_chat_id: "{{ TELEGRAM_CHAT_ID | default(lookup('env', 'TELEGRAM_CHAT_ID'), true) | string }}"
+    telegram_configured: "{{ telegram_bot_token | string | length > 0 and telegram_chat_id | string | length > 0 }}"
+
+  tasks:
+    - name: Build alert text
+      ansible.builtin.set_fact:
+        alert_text: "{{ alert_body | default('', true) | trim }}"   # an empty template renders as None
+      vars:
+        alert_body: |-
+          {% for h in groups['reachable_hosts'] | default([]) %}
+          {% for svc in hostvars[h].service_states | default([]) if svc.state != 'running' %}
+          {% set note = ' - start attempted' if (restart_stopped | bool and svc.state != 'not-found') else '' %}
+          SERVICE {{ h }}: {{ svc.name }} {{ 'not installed' if svc.state == 'not-found' else svc.state }}{{ note }}
+          {% endfor %}
+          {% endfor %}
+
+    - name: Nothing to alert
+      ansible.builtin.debug:
+        msg: "All critical services running - no alert"
+      when: alert_text | length == 0
+
+    - name: Print alerts when Telegram is not configured
+      ansible.builtin.debug:
+        msg: |-
+          ALERTS (Telegram not configured - bot token {{ 'set' if telegram_bot_token | string | length > 0 else 'MISSING' }},
+          chat id {{ 'set' if telegram_chat_id | string | length > 0 else 'MISSING' }};
+          pass TELEGRAM_BOT_TOKEN and TELEGRAM_CHAT_ID through the variable group attached to this template)
+          {{ alert_text }}
+      when:
+        - alert_text | length > 0
+        - not telegram_configured
+
+    - name: Send Telegram alert
+      ansible.builtin.uri:
+        url: "https://api.telegram.org/bot{{ telegram_bot_token }}/sendMessage"
+        method: POST
+        body_format: json
+        body:
+          chat_id: "{{ telegram_chat_id }}"
+          text: "Homelab service check {{ now().strftime('%Y-%m-%d %H:%M') }}\n{{ alert_text }}"
+        status_code: [200]
+      register: telegram_result
+      failed_when: false
+      no_log: true          # the bot token is part of the URL
+      when:
+        - alert_text | length > 0
+        - telegram_configured
+
+    - name: Telegram delivery status
+      ansible.builtin.assert:
+        that: telegram_result.status == 200
+        fail_msg: "Telegram API returned {{ telegram_result.status }}: {{ telegram_result.json.description | default('no response') }}"
+        success_msg: "Telegram alert sent ({{ alert_text.split('\n') | length }} line(s))"
+      when: telegram_result is not skipped

playbooks/ssh_harden.yml

@@ -0,0 +1,209 @@
+---
+# Hardens sshd on every reachable Ubuntu/Rocky guest and turns on a fail2ban
+# sshd jail. Settings go into /etc/ssh/sshd_config.d/00-ansible.conf, which
+# sorts first and therefore wins over anything in sshd_config or the
+# cloud-init drop-in (sshd keeps the first value it sees for each option).
+#
+# Lockout guard: password authentication is only disabled on hosts where
+# root already has at least one authorized key.
+#
+# Extra vars (optional):
+#   ssh_disable_password_auth - default true
+#   ssh_max_auth_tries        - default 4
+
+- name: Check Host Availability
+  hosts: ubuntu:rocky
+  gather_facts: false
+  ignore_unreachable: true
+
+  tasks:
+    - name: Ping hosts to check availability
+      ansible.builtin.ping:
+      register: ping_result
+      ignore_errors: true
+      ignore_unreachable: true
+
+    - name: Add reachable hosts to group
+      ansible.builtin.group_by:
+        key: reachable_hosts
+      when: ping_result is reachable and ping_result is succeeded
+
+    - name: Report offline hosts
+      ansible.builtin.debug:
+        msg: "{{ inventory_hostname }} - OFFLINE"
+      when: ping_result is unreachable or ping_result is failed
+      delegate_to: localhost
+
+- name: SSH Hardening - Install fail2ban on Ubuntu
+  hosts: ubuntu:&reachable_hosts
+  become: true
+  gather_facts: false
+
+  tasks:
+    - name: Install fail2ban and nftables via apt
+      ansible.builtin.apt:
+        name:
+          - fail2ban
+          - nftables
+        state: present
+        update_cache: true
+        cache_valid_time: 3600
+
+- name: SSH Hardening - Install fail2ban on Rocky Linux
+  hosts: rocky:&reachable_hosts
+  become: true
+  gather_facts: false
+
+  tasks:
+    - name: Ensure EPEL repo is enabled
+      ansible.builtin.dnf:
+        name: epel-release
+        state: present
+
+    - name: Install fail2ban and nftables via dnf
+      ansible.builtin.dnf:
+        name:
+          - fail2ban
+          - nftables
+        state: present
+
+- name: SSH Hardening - Configure sshd and fail2ban
+  hosts: reachable_hosts
+  become: true
+  gather_facts: false
+
+  vars:
+    ssh_disable_password_auth: true
+    ssh_max_auth_tries: 4
+    sshd_dropin_path: /etc/ssh/sshd_config.d/00-ansible.conf
+
+  tasks:
+    - name: Count public keys in root's authorized_keys
+      ansible.builtin.command: grep -cE '^(ssh-|ecdsa-|sk-)' /root/.ssh/authorized_keys
+      register: key_count
+      failed_when: false
+      changed_when: false
+
+    - name: Decide whether password authentication can be disabled
+      ansible.builtin.set_fact:
+        has_keys: "{{ key_count.rc == 0 and (key_count.stdout | int) > 0 }}"
+
+    - name: Warn when root has no authorized keys
+      ansible.builtin.debug:
+        msg: "WARNING: no authorized keys for root on {{ inventory_hostname }} - password authentication stays enabled"
+      when: not has_keys
+
+    - name: Ensure sshd_config.d directory exists
+      ansible.builtin.file:
+        path: /etc/ssh/sshd_config.d
+        state: directory
+        owner: root
+        group: root
+        mode: '0755'
+
+    - name: Ensure sshd_config includes the drop-in directory
+      ansible.builtin.lineinfile:
+        path: /etc/ssh/sshd_config
+        regexp: '^Include\s+/etc/ssh/sshd_config\.d/\*\.conf'
+        line: 'Include /etc/ssh/sshd_config.d/*.conf'
+        insertbefore: BOF
+        backup: true
+      register: sshd_include
+
+    - name: Write sshd hardening drop-in
+      ansible.builtin.copy:
+        dest: "{{ sshd_dropin_path }}"
+        content: |
+          # Managed by Ansible (playbooks/ssh_harden.yml) - do not edit by hand
+          PermitRootLogin prohibit-password
+          PubkeyAuthentication yes
+          PasswordAuthentication {{ 'no' if (has_keys and ssh_disable_password_auth | bool) else 'yes' }}
+          KbdInteractiveAuthentication no
+          MaxAuthTries {{ ssh_max_auth_tries }}
+          X11Forwarding no
+        owner: root
+        group: root
+        mode: '0600'
+        validate: /usr/sbin/sshd -t -f %s
+      register: sshd_dropin
+
+    - name: Record whether sshd needs a restart
+      ansible.builtin.set_fact:
+        sshd_changed: "{{ sshd_include is changed or sshd_dropin is changed }}"
+
+    - name: Configure fail2ban sshd jail
+      ansible.builtin.copy:
+        dest: /etc/fail2ban/jail.d/sshd.local
+        content: |
+          # Managed by Ansible (playbooks/ssh_harden.yml)
+          [sshd]
+          enabled = true
+          backend = systemd
+          banaction = nftables-multiport
+          maxretry = 5
+          findtime = 10m
+          bantime = 1h
+        owner: root
+        group: root
+        mode: '0644'
+      register: f2b_jail
+
+    - name: Ensure fail2ban is enabled and running
+      ansible.builtin.systemd:
+        name: fail2ban
+        enabled: true
+        state: "{{ 'restarted' if f2b_jail is changed else 'started' }}"
+
+- name: SSH Hardening - Restart SSH on Ubuntu
+  hosts: ubuntu:&reachable_hosts
+  become: true
+  gather_facts: false
+
+  tasks:
+    - name: Restart ssh service
+      ansible.builtin.service:
+        name: ssh
+        state: restarted
+      when: sshd_changed | bool
+
+- name: SSH Hardening - Restart SSH on Rocky Linux
+  hosts: rocky:&reachable_hosts
+  become: true
+  gather_facts: false
+
+  tasks:
+    - name: Restart sshd service
+      ansible.builtin.service:
+        name: sshd
+        state: restarted
+      when: sshd_changed | bool
+
+- name: SSH Hardening - Verify access
+  hosts: reachable_hosts
+  become: true
+  gather_facts: false
+
+  tasks:
+    - name: Drop the cached SSH connection so the next task performs a fresh login
+      ansible.builtin.meta: reset_connection
+
+    - name: Confirm key-based login still works
+      ansible.builtin.ping:
+
+    - name: Read fail2ban sshd jail status
+      ansible.builtin.command: fail2ban-client status sshd
+      register: f2b_status
+      changed_when: false
+      failed_when: false
+
+    - name: Summary
+      ansible.builtin.debug:
+        msg: |
+          ======================================================
+           HOST: {{ inventory_hostname }} ({{ ansible_host }})
+          ======================================================
+           PASSWORD AUTH: {{ 'disabled' if (has_keys and ssh_disable_password_auth | default(true) | bool) else 'ENABLED' }}
+           ROOT KEYS:     {{ key_count.stdout if key_count.rc == 0 else 0 }}
+           SSHD:          {{ 'restarted' if sshd_changed | bool else 'unchanged' }}
+           FAIL2BAN:      {{ 'sshd jail active' if f2b_status.rc == 0 else 'WARNING: jail not reporting (' ~ (f2b_status.stderr | default('') | trim) ~ ')' }}
+          ======================================================

playbooks/time-region.yml

@@ -1,120 +1,121 @@
 ---
-- name: Configure timezone and NTP (LXC-Aware)
+# Sets the timezone on every Ubuntu/Rocky guest. LXC containers share the
-  hosts: all
+# host's clock, so NTP is disabled inside them; VMs sync against ntp_server.
-  become: yes
+#
+# Extra vars (optional):
+#   timezone_region - default America/New_York
+#   ntp_server      - default time.google.com
+
+- name: Configure timezone
+  hosts: ubuntu:rocky
+  become: true
 
   vars:
-    timezone_region: "America/New_York"
+    timezone_region: America/New_York
-    ntp_server: "time.google.com"
 
   tasks:
-  # ----------------------------------------------------------
+    - name: Link /etc/localtime to the zoneinfo file
-  # SETUP FACTS
+      ansible.builtin.file:
-  # ----------------------------------------------------------
-    - name: Ensure virtualization facts are gathered
-      ansible.builtin.setup:
-        gather_subset:
-          - virtual
-
-  # ----------------------------------------------------------
-  # TIMEZONE (Required for ALL systems, LXC and VMs)
-  # ----------------------------------------------------------
-    - name: Copy timezone file
-      ansible.builtin.copy:
         src: "/usr/share/zoneinfo/{{ timezone_region }}"
         dest: /etc/localtime
-        remote_src: yes
+        state: link
-      register: tz_copy
+        force: true
 
     - name: Write /etc/timezone
       ansible.builtin.copy:
         dest: /etc/timezone
         content: "{{ timezone_region }}\n"
-      register: tz_write
+        owner: root
+        group: root
+        mode: '0644'
 
-  # ----------------------------------------------------------
+# The plays below reuse the facts gathered above instead of collecting them again.
-  # LXC CONTAINER HANDLING (Disable NTP)
+
-  # ----------------------------------------------------------
+- name: NTP on Ubuntu
-    - name: Stop and mask timesyncd on LXC (Ubuntu/Debian)
+  hosts: ubuntu
+  become: true
+  gather_facts: false
+
+  vars:
+    ntp_server: time.google.com
+
+  tasks:
+    - name: Stop and mask timesyncd in LXC containers
       ansible.builtin.systemd:
         name: systemd-timesyncd
         state: stopped
-        enabled: no
+        enabled: false
-        masked: yes
+        masked: true
-      when: 
+      when: ansible_virtualization_type == "lxc"
-        - ansible_os_family == "Debian"
-        - ansible_virtualization_type == "lxc"
 
-    - name: Stop and mask chronyd on LXC (RHEL-based)
+    - name: Point timesyncd at ntp_server on VMs
-      ansible.builtin.systemd:
-        name: chronyd
-        state: stopped
-        enabled: no
-        masked: yes
-      when: 
-        - ansible_os_family == "RedHat"
-        - ansible_virtualization_type == "lxc"
-
-  # ----------------------------------------------------------
-  # FULL VM / HARDWARE HANDLING (Configure NTP)
-  # ----------------------------------------------------------
-    - name: Configure timesyncd on VMs (Ubuntu/Debian)
       ansible.builtin.lineinfile:
         path: /etc/systemd/timesyncd.conf
         regexp: '^#?NTP='
         line: "NTP={{ ntp_server }}"
-        state: present
+      when: ansible_virtualization_type != "lxc"
-      when: 
-        - ansible_os_family == "Debian"
-        - ansible_virtualization_type != "lxc"
       notify: Restart timesyncd
 
-    - name: Disable default chrony pools on VMs (RHEL-based)
-      ansible.builtin.replace:
-        path: /etc/chrony.conf
-        regexp: '^(pool|server) .*'
-        replace: '# \1 removed by Ansible'
-      when: 
-        - ansible_os_family == "RedHat"
-        - ansible_virtualization_type != "lxc"
-
-    - name: Configure chrony on VMs (RHEL-based)
-      ansible.builtin.lineinfile:
-        path: /etc/chrony.conf
-        line: "server {{ ntp_server }} iburst"
-        insertafter: EOF
-        state: present
-      when: 
-        - ansible_os_family == "RedHat"
-        - ansible_virtualization_type != "lxc"
-      notify: Restart chronyd
-
-  # ----------------------------------------------------------
-  # FINAL SUMMARY
-  # ----------------------------------------------------------
-    - name: Verify current system time
-      ansible.builtin.command: date
-      register: check_time
-      changed_when: false
-
-    - name: Final summary
-      ansible.builtin.debug:
-        msg: 
-          - "Setup complete on {{ inventory_hostname }} (Environment: {{ ansible_virtualization_type | default('unknown') }})"
-          - "System Time: {{ check_time.stdout }}"
-
-  # ----------------------------------------------------------
-  # HANDLERS (Only triggers for VMs)
-  # ----------------------------------------------------------
   handlers:
     - name: Restart timesyncd
       ansible.builtin.systemd:
         name: systemd-timesyncd
         state: restarted
-        enabled: yes
+        enabled: true
 
+- name: NTP on Rocky Linux
+  hosts: rocky
+  become: true
+  gather_facts: false
+
+  vars:
+    ntp_server: time.google.com
+
+  tasks:
+    - name: Stop and mask chronyd in LXC containers
+      ansible.builtin.systemd:
+        name: chronyd
+        state: stopped
+        enabled: false
+        masked: true
+      when: ansible_virtualization_type == "lxc"
+
+    # The lookahead keeps the server line added below from being commented
+    # out again on the next run.
+    - name: Disable the default chrony pools on VMs
+      ansible.builtin.replace:
+        path: /etc/chrony.conf
+        regexp: '^(pool|server)\s+(?!{{ ntp_server | regex_escape }}(\s|$)).*'
+        replace: '# \g<0>  (disabled by Ansible)'
+      when: ansible_virtualization_type != "lxc"
+      notify: Restart chronyd
+
+    - name: Point chrony at ntp_server on VMs
+      ansible.builtin.lineinfile:
+        path: /etc/chrony.conf
+        regexp: '^server\s+{{ ntp_server | regex_escape }}(\s|$)'
+        line: "server {{ ntp_server }} iburst"
+      when: ansible_virtualization_type != "lxc"
+      notify: Restart chronyd
+
+  handlers:
     - name: Restart chronyd
       ansible.builtin.systemd:
         name: chronyd
         state: restarted
-        enabled: yes
+        enabled: true
+
+- name: Verify time configuration
+  hosts: ubuntu:rocky
+  become: true
+  gather_facts: false
+
+  tasks:
+    - name: Check current system time
+      ansible.builtin.command: date
+      register: check_time
+      changed_when: false
+
+    - name: Summary
+      ansible.builtin.debug:
+        msg: "{{ inventory_hostname }} ({{ ansible_virtualization_type | default('unknown') }}) - {{ check_time.stdout }}"

playbooks/update_pihole_blocklists.yml

@@ -1,18 +1,22 @@
 ---
 - name: Update Pi-hole core and blocklists
   hosts: pihole
-  become: yes
+  become: true
+  gather_facts: false
+
   tasks:
     - name: Update Pi-hole core components
-      command: pihole -up
+      ansible.builtin.command: pihole -up
       register: update_out
+      changed_when: "'Everything is up to date' not in update_out.stdout"
 
     - name: Update Pi-hole blocklists
-      command: pihole -g
+      ansible.builtin.command: pihole -g
       register: gravity_out
+      changed_when: true
 
     - name: Display update summary
-      debug:
+      ansible.builtin.debug:
         msg:
           - "Core update: {{ update_out.stdout }}"
           - "Gravity update: {{ gravity_out.stdout }}"

playbooks/update_system.yml

@@ -1,53 +1,257 @@
+---
+# Full package upgrade for every reachable host: apt dist-upgrade on Ubuntu,
+# dnf update on Rocky Linux. Each host is checked for pending packages before
+# the upgrade runs, so the per-host and final reports show which systems were
+# actually updated (and with what) and which were already up to date.
+#
+# Nothing reboots here; use reboot_check.yml for that.
+
+# Play 1: probe every ubuntu/rocky host and sort them into a "reachable_hosts"
+# group so the two upgrade plays below can skip anything that's offline.
 - name: Check Host Availability
-  hosts: all
+  hosts: ubuntu:rocky
   gather_facts: false
   ignore_unreachable: true
 
   tasks:
+    # Ping each host; failures/unreachables are swallowed so the play keeps
+    # going instead of aborting the whole run.
     - name: Ping hosts to check availability
       ansible.builtin.ping:
       register: ping_result
       ignore_errors: true
       ignore_unreachable: true
 
+    # Dynamically creates/populates the "reachable_hosts" group, used by the
+    # upgrade plays' host patterns (ubuntu:&reachable_hosts etc.).
     - name: Add reachable hosts to group
       ansible.builtin.group_by:
         key: reachable_hosts
-      when: ping_result is succeeded
+      when: ping_result is reachable and ping_result is succeeded
 
+    # Print a message for any host that failed the ping, so offline hosts
+    # are visible in the run output and in the final summary.
     - name: Report offline hosts
       ansible.builtin.debug:
-        msg: "Host {{ inventory_hostname }} is OFFLINE and will be skipped"
+        msg: "{{ inventory_hostname }} - OFFLINE and will be skipped"
-      when: ping_result is not succeeded
+      when: ping_result is unreachable or ping_result is failed
       delegate_to: localhost
 
-- name: Update and Upgrade Packages
+# Play 2: upgrade Ubuntu hosts and record what each one needed.
-  hosts: reachable_hosts
+- name: Update and Upgrade - Ubuntu
+  hosts: ubuntu:&reachable_hosts
   become: true
-  gather_facts: true
+  gather_facts: false
 
   tasks:
-    - name: Update & Upgrade - Debian-based systems
+    # Refresh the package lists first so the pending check below is accurate.
+    # A cache refreshed within the last hour is reused.
+    - name: Update apt cache
       ansible.builtin.apt:
         update_cache: true
         cache_valid_time: 3600
-        upgrade: dist
+      changed_when: false   # refreshing the lists is not a change worth reporting
-      when: ansible_os_family == "Debian"
+      check_mode: false
 
-    - name: Update & Upgrade - Modern RHEL-based systems (DNF)
+    # Simulate the upgrade (no changes made) to learn which packages are
+    # pending. Each candidate shows up as an "Inst <name> ..." line.
+    - name: Check which packages are pending  # noqa: command-instead-of-module (dry run, the apt module cannot simulate)
+      ansible.builtin.command: apt-get --just-print dist-upgrade
+      register: apt_simulate
+      changed_when: false
+      check_mode: false
+      environment:
+        DEBIAN_FRONTEND: noninteractive
+
+    - name: Record pending package list
+      ansible.builtin.set_fact:
+        updated_packages: "{{ apt_simulate.stdout | regex_findall('(?m)^Inst (\\S+)') }}"
+
+    # The actual upgrade. Hosts with nothing pending come back unchanged.
+    # A failure is caught so this host still shows up in the reports below;
+    # the final play turns any failure back into a non-zero exit.
+    - name: Update & Upgrade via apt
+      block:
+        - name: Run apt dist-upgrade
+          ansible.builtin.apt:
+            update_cache: false
+            upgrade: dist
+          register: upgrade_result
+      rescue:
+        - name: Record the upgrade error
+          ansible.builtin.set_fact:
+            update_error: "{{ upgrade_result.msg | default('see the task output above') }}"
+
+    # update_status drives both the per-host report and the final summary:
+    #   updated     - packages were installed
+    #   up to date  - nothing was pending
+    #   failed      - the upgrade task itself errored out
+    - name: Record update status
+      ansible.builtin.set_fact:
+        update_status: >-
+          {{ 'failed' if upgrade_result is failed
+             else ('updated' if upgrade_result is changed else 'up to date') }}
+
+    - name: Print report
+      ansible.builtin.debug:
+        msg: |
+          ======================================================
+           HOST: {{ inventory_hostname }} ({{ ansible_host }}) - Ubuntu
+          ======================================================
+           STATUS: {{ update_status | upper }}{{ ' (CHECK MODE - nothing was applied)' if ansible_check_mode else '' }}
+          {% if update_status == 'updated' %}
+          ------------------------------------------------------
+           PACKAGES UPDATED: {{ updated_packages | length }}
+          {{ updated_packages | join(', ') | default('(list unavailable)', true) }}
+          {% elif update_status == 'failed' %}
+          ------------------------------------------------------
+           ERROR: {{ update_error | default('see the task output above') }}
+           PACKAGES STILL PENDING: {{ updated_packages | length }}
+          {{ updated_packages | join(', ') | default('(list unavailable)', true) }}
+          {% endif %}
+          ======================================================
+
+# Play 3: upgrade Rocky Linux hosts and record what each one needed.
+- name: Update and Upgrade - Rocky Linux
+  hosts: rocky:&reachable_hosts
+  become: true
+  gather_facts: false
+
+  tasks:
+    # Refresh the metadata first so the pending check below is accurate.
+    - name: Update dnf cache
       ansible.builtin.dnf:
-        name: "*"
-        state: latest
         update_cache: true
-      when:
+      changed_when: false   # refreshing the metadata is not a change worth reporting
-        - ansible_os_family == "RedHat"
+      check_mode: false
-        - ansible_pkg_mgr == "dnf"
 
-    - name: Update & Upgrade - Legacy RHEL-based systems (YUM)
+    # rc 100 means updates are available, rc 0 means none, so only other
-      ansible.builtin.yum:
+    # codes count as a failure.
-        name: "*"
+    - name: Check which packages are pending
-        state: latest
+      ansible.builtin.command: dnf check-update
-        update_cache: true
+      register: dnf_check
-      when:
+      changed_when: false
-        - ansible_os_family == "RedHat"
+      check_mode: false
-        - ansible_pkg_mgr == "yum"
+      failed_when: dnf_check.rc not in [0, 100]
+
+    # Pull the "name.arch  version  repo" lines out of the check-update
+    # output. Anything under "Obsoleting Packages" is a duplicate listing,
+    # so the output is truncated there first.
+    - name: Record pending package list
+      ansible.builtin.set_fact:
+        updated_packages: >-
+          {{ dnf_check.stdout.split('Obsoleting Packages')[0].splitlines()
+             | select('match', '^\S+\.\S+\s+\S+\s+\S+$')
+             | map('regex_replace', '^(\S+)\s.*$', '\1') | list }}
+
+    # The actual upgrade. Hosts with nothing pending come back unchanged.
+    # A failure is caught so this host still shows up in the reports below;
+    # the final play turns any failure back into a non-zero exit.
+    - name: Update & Upgrade via dnf
+      block:
+        - name: Run dnf update
+          ansible.builtin.dnf:
+            name: "*"
+            state: latest
+            update_cache: false
+          register: upgrade_result
+      rescue:
+        - name: Record the upgrade error
+          ansible.builtin.set_fact:
+            update_error: "{{ upgrade_result.msg | default('see the task output above') }}"
+
+    # Same three states as the Ubuntu play above.
+    - name: Record update status
+      ansible.builtin.set_fact:
+        update_status: >-
+          {{ 'failed' if upgrade_result is failed
+             else ('updated' if upgrade_result is changed else 'up to date') }}
+
+    - name: Print report
+      ansible.builtin.debug:
+        msg: |
+          ======================================================
+           HOST: {{ inventory_hostname }} ({{ ansible_host }}) - Rocky Linux
+          ======================================================
+           STATUS: {{ update_status | upper }}{{ ' (CHECK MODE - nothing was applied)' if ansible_check_mode else '' }}
+          {% if update_status == 'updated' %}
+          ------------------------------------------------------
+           PACKAGES UPDATED: {{ updated_packages | length }}
+          {{ updated_packages | join(', ') | default('(list unavailable)', true) }}
+          {% elif update_status == 'failed' %}
+          ------------------------------------------------------
+           ERROR: {{ update_error | default('see the task output above') }}
+           PACKAGES STILL PENDING: {{ updated_packages | length }}
+          {{ updated_packages | join(', ') | default('(list unavailable)', true) }}
+          {% endif %}
+          ======================================================
+
+# Play 4: after both upgrade plays have run, print one consolidated summary of
+# which hosts were updated, which were already current, and which were missed.
+- name: Summary - Updated Hosts
+  hosts: localhost
+  connection: local
+  gather_facts: false
+
+  vars:
+    all_targets: "{{ groups['ubuntu'] | default([]) | union(groups['rocky'] | default([])) }}"
+    reachable: "{{ groups['reachable_hosts'] | default([]) }}"
+    # update_status of every host that got far enough to set it.
+    statuses: "{{ reachable | map('extract', hostvars, 'update_status') | select('defined') | list }}"
+    status_counts:
+      "updated": "{{ statuses | select('equalto', 'updated') | list | length }}"
+      "up to date": "{{ statuses | select('equalto', 'up to date') | list | length }}"
+      "failed": "{{ statuses | select('equalto', 'failed') | list | length }}"
+    failed_hosts: "{{ reachable | map('extract', hostvars) | selectattr('update_status', 'defined')
+                      | selectattr('update_status', 'equalto', 'failed')
+                      | map(attribute='inventory_hostname') | list }}"
+
+  tasks:
+    - name: Print hosts that were updated
+      ansible.builtin.debug:
+        msg: >-
+          UPDATED      {{ item }} -
+          {{ hostvars[item].updated_packages | default([]) | length }} package(s):
+          {{ hostvars[item].updated_packages | default([]) | join(', ') | default('(list unavailable)', true) }}
+      loop: "{{ reachable }}"
+      when: hostvars[item].update_status | default('') == 'updated'
+
+    - name: Print hosts that were already up to date
+      ansible.builtin.debug:
+        msg: "UP TO DATE   {{ item }} - no packages pending"
+      loop: "{{ reachable }}"
+      when: hostvars[item].update_status | default('') == 'up to date'
+
+    - name: Print hosts whose upgrade failed
+      ansible.builtin.debug:
+        msg: "FAILED       {{ item }} - the upgrade task reported an error, see the play output above"
+      loop: "{{ reachable }}"
+      when: hostvars[item].update_status | default('') == 'failed'
+
+    # Covers both hosts that failed the initial ping and any host that became
+    # unreachable part way through an upgrade play.
+    - name: Print hosts that were skipped
+      ansible.builtin.debug:
+        msg: "SKIPPED      {{ item }} - offline or unreachable, not upgraded"
+      loop: "{{ all_targets }}"
+      when: hostvars[item].update_status is not defined
+
+    - name: Print totals
+      ansible.builtin.debug:
+        msg: >-
+          {{ all_targets | length }} host(s) targeted,
+          {{ status_counts['updated'] }} updated,
+          {{ status_counts['up to date'] }} already up to date,
+          {{ status_counts['failed'] }} failed,
+          {{ all_targets | reject('in', reachable) | list | length }} skipped;
+          {{ reachable | map('extract', hostvars, 'updated_packages')
+             | select('defined') | map('length') | sum }}
+          package(s) updated in total.
+          {{ '(CHECK MODE - nothing was applied.)' if ansible_check_mode else '' }}
+
+    # Keep a failed upgrade visible to Semaphore: every report above has been
+    # printed by now, so it is safe to end the run non-zero here.
+    - name: Fail the run if any host could not be upgraded
+      ansible.builtin.fail:
+        msg: "Upgrade failed on: {{ failed_hosts | join(', ') }}"
+      when: failed_hosts | length > 0

playbooks/user_setup.yml

@@ -0,0 +1,104 @@
+---
+# Creates a non-root admin user with sudo and the homelab SSH keys on every
+# reachable Ubuntu/Rocky guest. First step toward running Ansible as a
+# non-root user.
+#
+# Extra vars (optional):
+#   admin_user          - default harry
+#   admin_sudo_nopasswd - default true (required for unattended become)
+#   admin_ssh_keys      - list of public keys (defaults to the ansible-key and
+#                         the personal key already used by add_root_key.yml)
+
+- name: Check Host Availability
+  hosts: ubuntu:rocky
+  gather_facts: false
+  ignore_unreachable: true
+
+  tasks:
+    - name: Ping hosts to check availability
+      ansible.builtin.ping:
+      register: ping_result
+      ignore_errors: true
+      ignore_unreachable: true
+
+    - name: Add reachable hosts to group
+      ansible.builtin.group_by:
+        key: reachable_hosts
+      when: ping_result is reachable and ping_result is succeeded
+
+    - name: Report offline hosts
+      ansible.builtin.debug:
+        msg: "{{ inventory_hostname }} - OFFLINE"
+      when: ping_result is unreachable or ping_result is failed
+      delegate_to: localhost
+
+- name: User Setup - Create admin user on Ubuntu
+  hosts: ubuntu:&reachable_hosts
+  become: true
+  gather_facts: false
+
+  vars:
+    admin_user: harry
+
+  tasks:
+    - name: Create admin user in the sudo group - {{ admin_user }}
+      ansible.builtin.user:
+        name: "{{ admin_user }}"
+        shell: /bin/bash
+        groups: sudo
+        append: true
+        create_home: true
+
+- name: User Setup - Create admin user on Rocky Linux
+  hosts: rocky:&reachable_hosts
+  become: true
+  gather_facts: false
+
+  vars:
+    admin_user: harry
+
+  tasks:
+    - name: Create admin user in the wheel group - {{ admin_user }}
+      ansible.builtin.user:
+        name: "{{ admin_user }}"
+        shell: /bin/bash
+        groups: wheel
+        append: true
+        create_home: true
+
+- name: User Setup - Authorized keys and sudo
+  hosts: reachable_hosts
+  become: true
+  gather_facts: false
+
+  vars:
+    admin_user: harry
+    admin_sudo_nopasswd: true
+    admin_ssh_keys:
+      - "ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAINegitHaQG8i+s412h3wTWf/KvpeX1CEpYFva9MhHR+7 ansible-key"
+      - "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQDbFKxd6W+gHcNrugkxMwvixGmk3LxtsMEnSdaFp4L78WZ4VhdZrsnSJQ/rViT+vuc6K5SeugtggfQ90L25UyyMv2BhLZaoMS2dpoVJEZon/Y2Hemm11vdEApAK/u0kPTTM5JB0/TZfpoP9sr4pmRIUruRXBhLDvtUV9tJvKSzbhChtB+vBg+MpywG9VvDP6eCVpFaP43RcAFtJR7irJmexd5m4CVhIFaWNeTbbe8YI08i2gDinYl95ygu9i/TorQKLWme2v9ejxQFfJTWGaXHnQ18W0X94Ns4a4OccGcvJ8JmvYsaAPRAOSveq1Fpdw1VXHSKigTmRl6pHWd9/rq6K6Ojctbjo1kHiNHdLKLZbLDeM0OQlzNl74snsImmZddY59wnOKyHtfY0ZI7sQ+7A/0cSaMY6h7lt2xK0WmayuoT8UtKZlItZHCkOA5bV29arI/HTz+yR/LW8SihPPFpVv5aDiOUs0PYd9j0+CtULBJeYpMQJpkD93g06vgEJjhUOBZ6JmilxVDihDARU0x0xiRCQJcq4Nhk/x3GzPe2kkrTI+nZFaQn1K8hcBdSyA12/xAG8Fh1SMCxLwJdCNuL3gQoYd6XPzKqn6BIqAgiyq5DbHhO6mehOye4Rro6i/H1UkZe862QH/R5uH+8Z2/FYRpiCPzQyLwrqx/eVfT7JMFQ== harwinder@harwinder.net"  # yamllint disable-line rule:line-length
+
+  tasks:
+    - name: Add authorized keys for {{ admin_user }}
+      ansible.posix.authorized_key:
+        user: "{{ admin_user }}"
+        key: "{{ item }}"
+        state: present
+      loop: "{{ admin_ssh_keys }}"
+      loop_control:
+        label: "{{ item.split()[-1] }}"
+
+    - name: Grant sudo to {{ admin_user }}
+      ansible.builtin.copy:
+        dest: "/etc/sudoers.d/90-{{ admin_user }}"
+        content: "{{ admin_user }} ALL=(ALL) {{ 'NOPASSWD:' if admin_sudo_nopasswd | bool else '' }}ALL\n"
+        owner: root
+        group: root
+        mode: '0440'
+        validate: /usr/sbin/visudo -cf %s
+
+    - name: Summary
+      ansible.builtin.debug:
+        msg: >-
+          {{ inventory_hostname }} - user {{ admin_user }} ready
+          (sudo {{ 'NOPASSWD' if admin_sudo_nopasswd | bool else 'with password' }}, {{ admin_ssh_keys | length }} key(s))

playbooks/vm_guest_agent.yml

@@ -0,0 +1,39 @@
+---
+# Installs and enables the QEMU guest agent on VMs (group "vms") so Proxmox
+# can query IPs, quiesce filesystems for snapshots and shut the VM down cleanly.
+# The VM also needs the agent enabled on the Proxmox side:
+#   qm set <vmid> --agent enabled=1   (then power-cycle the VM once)
+
+- name: VM Guest Agent - Install and enable qemu-guest-agent
+  hosts: vms
+  become: true
+  gather_facts: true
+
+  tasks:
+    - name: Install qemu-guest-agent
+      ansible.builtin.package:
+        name: qemu-guest-agent
+        state: present
+
+    - name: Check whether the guest agent virtio port is present
+      ansible.builtin.stat:
+        path: /dev/virtio-ports/org.qemu.guest_agent.0
+      register: agent_port
+
+    - name: Enable and start qemu-guest-agent
+      ansible.builtin.systemd:
+        name: qemu-guest-agent
+        enabled: true
+        state: started
+      when: agent_port.stat.exists
+
+    - name: Warn when the agent device is missing
+      ansible.builtin.debug:
+        msg: "WARNING: {{ inventory_hostname }} has no guest agent device - run 'qm set <vmid> --agent enabled=1' on the Proxmox host and power-cycle the VM"
+      when: not agent_port.stat.exists
+
+    - name: Summary
+      ansible.builtin.debug:
+        msg: >-
+          {{ inventory_hostname }} ({{ ansible_virtualization_type | default('unknown') }}) -
+          qemu-guest-agent {{ 'RUNNING' if agent_port.stat.exists else 'NOT AVAILABLE' }}