docs: dpkg-interrupted runbook + apt-fleet ops + non-apt-binary pattern + Claude Code settings + chat-sudo gotcha

Bundles the documentation slice of the apt-sweep backlog:

* operations/runbook-dpkg-interrupted.md (NEW) — Path A (ssh-reachable
  recovery) + Path B (PVE web Shell when the netbird mesh broke
  alongside the dpkg state, as happened during Wave 6 on hubris).
  Closes B2.

* operations/commands.md — new "Fleet apt operations" section
  documenting `homelab apt-audit` and `homelab apt-upgrade`
  (--status / --safe / --force). Adds the dpkg-interrupted runbook to
  Related.

* operations/agent-enrollment.md —
  - new "Claude Code permissions for fleet ops" section with the
    `permissions.allow` snippet (`Bash(ssh -p 22022 *)`,
    `Bash(homelab *)`) for `~/.claude/settings.json`. Closes A3.
  - two new Troubleshooting rows: chat-mode `!` sudo no-tty gotcha
    (G2) and the cosmetic netbird DNS-probe warning.

* infrastructure/auto-deploy.md — new "Custom-built binaries that
  overlap apt-managed paths" section describing the two acceptable
  patterns (epoch-versioned .deb à la caddy 1:2.11.3-hubris1; or
  apt-mark hold) and the discovery path via `homelab apt-audit`'s
  NONAPT column. Closes D3.

Remaining backlog after this commit: A4 (upstream OpenSSH/netbird mux
bug), D1 (apt-mark hold caddy in caddy-conf bootstrap — superseded
in practice by the epoch .deb), E1/E2 (LXC DNS fallback for
tailscale-managed resolv.conf), F1/F2 (vzdump fallback doc; F3 already
shipped).

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

infrastructure/auto-deploy.md

@@ -103,10 +103,23 @@ Always commit + push. Local-only edits drift. Common ones:
   every other client has** — kept fresh by `homelab-context-sync.timer`,
   not by these webhooks.
 
+## Custom-built binaries that overlap apt-managed paths
+
+If a pipeline (or any out-of-band build) drops a binary into a path that an apt package also owns — most commonly `/usr/bin/<name>` — then the next `apt upgrade` of the corresponding package will silently clobber the custom build. That's exactly how [LXC 121 caddy](../containers/121-caddy.md) went down for ~10 min on 2026-05-21: an xcaddy build with `caddy-dns/ionos` lived at `/usr/bin/caddy` and Debian's caddy 2.11.2→2.11.3 apt upgrade replaced it with a vanilla 2.11.3 that couldn't parse the Caddyfile.
+
+Two patterns are acceptable, pick one when authoring a pipeline that ships a non-apt binary:
+
+1. **Ship the build as a `.deb` with an epoch-bumped version.** Use `dpkg-deb --build` (or `nfpm`) to package the binary as `Package: <name>`, `Version: 1:<upstream>-hubris<n>`. The epoch (`1:`) means it beats any non-epoch upstream version regardless of point bumps, so `apt upgrade` is a no-op for that package. Used by caddy: `caddy 1:2.11.3-hubris1` (see commit `2026-05-21` in [121-caddy.md](../containers/121-caddy.md)).
+
+2. **Hold the apt package.** `apt-mark hold <pkg>` on the LXC during pipeline install; apt will refuse to upgrade it. Simpler than `.deb` packaging but: (a) the hold flag isn't preserved by `dpkg -i` of a new version, (b) it's invisible unless you check `apt-mark showhold`, (c) you have to remember to `apt-mark unhold` when you intentionally want a new version. The new `homelab apt-audit` subcommand surfaces holds across the fleet so they don't get forgotten.
+
+If you're not sure what's already lurking, run `homelab apt-audit --fleet` and look at the `NONAPT` column — that's a count of binaries in `/usr/bin/{caddy,docker,jellyfin}` + `/usr/local/bin/*` that no apt package owns.
+
 ## Related
 - [Gitea (104)](../containers/104-gitea.md) — webhook source for all of these
 - [Caddy (121)](../containers/121-caddy.md), [apps (105)](../containers/105-apps.md), [mule-images (120)](../containers/120-mule-images.md), [claudio-bot (123)](../containers/123-claudio-bot.md), [hubris host](../hosts/hubris.md) — webhook targets
 - [Backups (disabled)](backups.md)
+- [Operations cheatsheet](../operations/commands.md) — `homelab apt-audit` / `homelab apt-upgrade` reference
 
 ## Changelog
 

operations/agent-enrollment.md

@@ -130,6 +130,37 @@ For Claude Code: start a new session — the `homelab` MCP server appears
 in `~/.claude/.mcp.json` and registers 14 tools (8 context, 5 management,
 1 secrets-metadata).
 
+## Claude Code permissions for fleet ops
+
+By default Claude Code's auto-mode classifier asks for confirmation on every
+ssh into the mesh. The bootstrap already installs the ssh ControlMaster block
+so subsequent in-session sshes multiplex, but the *first* ssh of each session
+still gets classifier-evaluated. Pre-authorize the common fleet ssh patterns
+by adding to `~/.claude/settings.json`:
+
+```json
+{
+  "permissions": {
+    "defaultMode": "auto",
+    "allow": [
+      "Bash(ssh -p 22022 *)",
+      "Bash(homelab *)"
+    ]
+  }
+}
+```
+
+The first rule covers any ssh to a mesh peer on the homelab netbird port; the
+second covers all `homelab` CLI invocations. Both are scoped tight enough that
+the classifier doesn't gate them but loose enough to handle the variety of
+arguments.
+
+If you also want the netbird `--ssh-jwt-cache-ttl` flag rationale to be
+visible to the classifier (it's not actually durable in 0.71.2, but the
+ControlMaster block is — see [runbook-dpkg-interrupted](runbook-dpkg-interrupted.md)
+for context), drop a free-text rule into `autoMode.allow` describing the
+authorization. Optional.
+
 ## Adding a new client to inventory
 
 If the hostname you want isn't yet in inventory, enrollment is a two-step
@@ -206,6 +237,8 @@ The CLI prints a follow-up checklist that the operator must do manually:
 | `homelab: no age key at /etc/age/key.txt` even after bootstrap | `/etc/age` is 0700 root, so non-root users couldn't even stat the key file; existence check returned False under regular users | Fixed in commit `df6aca8`: the CLI re-execs `sops -d` via sudo when invoked as a non-root user. On older deployments, re-link the CLI with `sudo ln -sfn /opt/homelab-context/bin/homelab /usr/local/bin/homelab` after the 5-min sync. |
 | `homelab` CLI doesn't pick up repo updates | Pre-`02db…` bootstrap copied the binary instead of symlinking | One-time migration: `sudo ln -sfn /opt/homelab-context/bin/homelab /usr/local/bin/homelab`. New bootstraps use the symlink, which auto-tracks the synced repo. |
 | `homelab-context-sync.service` journal shows `fatal: could not read Username for 'https://git.hubris.network'` | Pre-fix bootstrap set the gitea credential helper via `git config --global`, which writes to `/root/.gitconfig` — invisible to the systemd timer's git process (no HOME set). | One-time migration: `sudo git config --system credential.helper "store --file=/etc/homelab-context/git-credentials"`. New bootstraps store the helper in `/etc/gitconfig` instead. |
+| Chat-mode `!` shell can't `sudo` (`a terminal is required to read the password`) | Claude Code's `!` invocation doesn't allocate a tty, and standard `sudo` won't read its password from stdin or a non-tty pipe. | Run the sudo'd command in a real terminal outside chat. For commands the agent issues repeatedly, configure passwordless sudo for the narrow set (e.g. `/etc/sudoers.d/homelab-self` with `<user> ALL=(ALL) NOPASSWD: /usr/bin/dnf upgrade -y, /usr/bin/apt-get *`). |
+| `netbird status -d` reports `192.168.8.180:53 ... is Unavailable` but DNS actually works | netbird's UDP-53 probe times out over the relay latency (~90ms), but actual queries still flow through systemd-resolved. Cosmetic. | Ignore unless `dig @192.168.8.180 git.hubris.network` also fails — then check dnsmasq on [LXC 124](../containers/124-authentik.md). |
 
 ## Changelog
 

operations/commands.md

@@ -48,9 +48,25 @@ Run from the [hubris host](../hosts/hubris.md) as root. When working from `/root
 - `cat /sys/devices/system/cpu/cpu0/cpufreq/scaling_governor` — should be `powersave`
 - `ls /sys/fs/pstore/ /var/lib/systemd/pstore/` — panic traces from a previous crash (empty for pure hardware hangs — see [investigation](../investigations/2026-04-21-hubris-crash-loop.md))
 
+## Fleet apt operations
+
+Two `homelab` subcommands wrap the common patterns; both fan out to hubris + every LXC.
+
+| Command | What it does |
+| --- | --- |
+| `homelab apt-audit [--target HOST]` | Per-host table: dpkg-interrupted state, holds, upgradable count, non-apt binaries in system paths, DNS health. Exits nonzero if any host has dpkg-interrupted state. |
+| `homelab apt-upgrade --target HOST` | Launch `apt update && apt upgrade` inside a transient `systemd-run --collect` unit on the target. Survives ssh teardown. Apt configured with `Acquire::Retries=3` + `ForceIPv4=true`. |
+| `homelab apt-upgrade --all` | Same, fanned out across the standard targets. |
+| `homelab apt-upgrade ... --status` | Show running unit + tail `/var/log/homelab-apt-upgrade.log` on each target. |
+| `homelab apt-upgrade ... --safe` | Take a pre-upgrade snapshot per LXC first (`pct snapshot` → `vzdump` fallback for bind-mounted LXCs). Refuses if any snapshot fails unless `--force`. |
+| `homelab apt-upgrade ... --force` | Skip both the dpkg-audit gate and snapshot-failure refusal. |
+
+PVE/kernel deferral on hubris: `homelab apt-upgrade --target hubris` will try every upgrade, including kernel + `pve-*`. To skip those, `apt-mark hold` the relevant packages on hubris first; `homelab apt-audit` shows held packages so you can confirm.
+
 ## Related
 - [Hubris host](../hosts/hubris.md)
 - [Containers index](../containers/index.md)
 - [DNS](../infrastructure/dns.md)
 - [Monitoring](../infrastructure/monitoring.md)
 - [Auto-deploy](../infrastructure/auto-deploy.md)
+- [Runbook: dpkg-interrupted recovery](runbook-dpkg-interrupted.md) — what to do when apt got killed mid-transaction

operations/runbook-dpkg-interrupted.md

@@ -0,0 +1,108 @@
+# Runbook — recover from dpkg-interrupted state
+
+You're here because an apt run got killed mid-transaction and the target now
+has packages that are **unpacked but not configured**. Symptoms:
+
+- `apt` refuses to do anything new: `Error: dpkg was interrupted, you must
+  manually run 'dpkg --configure -a' to correct the problem.`
+- `dpkg --audit` lists packages with header
+  `The following packages have been unpacked but not yet configured.`
+- `homelab apt-audit` shows `DPKG: DIRTY(N)` for the host.
+
+The system is still running the **old** binaries (still in memory), but the
+**new** binaries are unpacked and waiting for their postinst to run. Two
+worst-case manifestations from the 2026-05-21 sweep:
+
+- LXC 121 caddy: leftover state from a prior aborted apt run; caddy itself was
+  still serving but the new caddy binary on disk hadn't been wired up.
+- hubris: ssh master died mid-Wave-6 → 135 packages unpacked-not-configured,
+  including `systemd`, `openssh-server`, `sudo`, `netbird`. The half-
+  configured netbird daemon dropped the mesh peer, and we got locked out
+  until we recovered from the PVE web UI Shell.
+
+**Do not reboot until dpkg is clean.** A reboot tries to start the new
+binaries' services, which may fail because postinst never ran (missing users,
+config dirs, capabilities, etc.). The system might not come back up cleanly.
+
+## Path A — target is still reachable over ssh (preferred)
+
+```
+homelab ssh <host> -- bash -c 'DEBIAN_FRONTEND=noninteractive dpkg --configure -a && apt -y -o Dpkg::Options::=--force-confold upgrade'
+```
+
+Or for an LXC by name:
+
+```
+homelab pct <lxc> exec -- bash -c 'DEBIAN_FRONTEND=noninteractive dpkg --configure -a && apt -y -o Dpkg::Options::=--force-confold upgrade'
+```
+
+When that returns, confirm:
+
+```
+homelab apt-audit --target <host>
+```
+
+Expect `DPKG: ok` and the remaining `UPGR` count to match what's intentionally
+deferred (kernel/PVE on hubris, 0 elsewhere).
+
+## Path B — target locked out (mesh broken / ssh dead)
+
+Most common for hubris when netbird itself went half-configured: the daemon
+crashed on the new binary, the mesh peer dropped, port 22022 stopped listening,
+and you can't ssh in.
+
+1. Open `https://proxmox.hubris.network` in a browser.
+2. Datacenter → node `hubris` → `>_ Shell` (or `_ Console`). That's a root
+   shell on hubris served by the PVE web UI, independent of the netbird mesh.
+3. Run the recovery one-liner:
+
+```
+DEBIAN_FRONTEND=noninteractive dpkg --configure -a \
+  && DEBIAN_FRONTEND=noninteractive apt -y -o Dpkg::Options::=--force-confold upgrade \
+  && systemctl restart netbird \
+  && dpkg --audit \
+  && echo RECOVERY_OK
+```
+
+   Wait for `RECOVERY_OK`. The `systemctl restart netbird` is the bit that
+   heals the mesh — once netbird's daemon comes back up clean, your client's
+   peer state moves from `Connecting` to `Connected` within ~30 seconds and
+   the rest of your tooling works again.
+
+4. For an **LXC** that's locked out (less common — LXCs reach the world via
+   netbird routed through hubris, so unless hubris itself is broken, you can
+   still `pct enter` from the hubris shell):
+
+   From the PVE web UI shell on hubris:
+
+   ```
+   pct enter <id>
+   DEBIAN_FRONTEND=noninteractive dpkg --configure -a && apt -y upgrade
+   exit
+   ```
+
+## Prevention
+
+The `homelab apt-upgrade` wrapper launches apt inside a `systemd-run --collect`
+unit on the target, so it survives ssh teardown — the failure mode that put
+hubris into this state in the first place is no longer reachable through the
+standard tool. If you absolutely need to run apt manually over ssh, wrap it:
+
+```
+ssh <host> systemd-run --unit=apt-recovery --collect bash -c 'apt -y upgrade'
+```
+
+Then `systemctl status apt-recovery` from a fresh ssh to check progress.
+
+## Related
+
+- [Operations cheatsheet](commands.md)
+- [Auto-deploy pipelines](../infrastructure/auto-deploy.md)
+- [Hubris host page](../hosts/hubris.md)
+
+## Changelog
+
+### 2026-05-21 — initial page
+Documents the dpkg-interrupted recovery path that came out of the
+fleet apt sweep (Wave 6 killed mid-transaction; hubris recovered via PVE
+web Shell).