docs: dpkg-interrupted runbook + apt-fleet ops + non-apt-binary pattern + Claude Code settings + chat-sudo gotcha

Bundles the documentation slice of the apt-sweep backlog:

* operations/runbook-dpkg-interrupted.md (NEW) — Path A (ssh-reachable
  recovery) + Path B (PVE web Shell when the netbird mesh broke
  alongside the dpkg state, as happened during Wave 6 on hubris).
  Closes B2.

* operations/commands.md — new "Fleet apt operations" section
  documenting `homelab apt-audit` and `homelab apt-upgrade`
  (--status / --safe / --force). Adds the dpkg-interrupted runbook to
  Related.

* operations/agent-enrollment.md —
  - new "Claude Code permissions for fleet ops" section with the
    `permissions.allow` snippet (`Bash(ssh -p 22022 *)`,
    `Bash(homelab *)`) for `~/.claude/settings.json`. Closes A3.
  - two new Troubleshooting rows: chat-mode `!` sudo no-tty gotcha
    (G2) and the cosmetic netbird DNS-probe warning.

* infrastructure/auto-deploy.md — new "Custom-built binaries that
  overlap apt-managed paths" section describing the two acceptable
  patterns (epoch-versioned .deb à la caddy 1:2.11.3-hubris1; or
  apt-mark hold) and the discovery path via `homelab apt-audit`'s
  NONAPT column. Closes D3.

Remaining backlog after this commit: A4 (upstream OpenSSH/netbird mux
bug), D1 (apt-mark hold caddy in caddy-conf bootstrap — superseded
in practice by the epoch .deb), E1/E2 (LXC DNS fallback for
tailscale-managed resolv.conf), F1/F2 (vzdump fallback doc; F3 already
shipped).

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
This commit is contained in:
2026-05-21 09:32:06 +02:00
parent 8f76338b05
commit 7a062bdb6b
4 changed files with 170 additions and 0 deletions

View File

@@ -48,9 +48,25 @@ Run from the [hubris host](../hosts/hubris.md) as root. When working from `/root
- `cat /sys/devices/system/cpu/cpu0/cpufreq/scaling_governor` — should be `powersave`
- `ls /sys/fs/pstore/ /var/lib/systemd/pstore/` — panic traces from a previous crash (empty for pure hardware hangs — see [investigation](../investigations/2026-04-21-hubris-crash-loop.md))
## Fleet apt operations
Two `homelab` subcommands wrap the common patterns; both fan out to hubris + every LXC.
| Command | What it does |
| --- | --- |
| `homelab apt-audit [--target HOST]` | Per-host table: dpkg-interrupted state, holds, upgradable count, non-apt binaries in system paths, DNS health. Exits nonzero if any host has dpkg-interrupted state. |
| `homelab apt-upgrade --target HOST` | Launch `apt update && apt upgrade` inside a transient `systemd-run --collect` unit on the target. Survives ssh teardown. Apt configured with `Acquire::Retries=3` + `ForceIPv4=true`. |
| `homelab apt-upgrade --all` | Same, fanned out across the standard targets. |
| `homelab apt-upgrade ... --status` | Show running unit + tail `/var/log/homelab-apt-upgrade.log` on each target. |
| `homelab apt-upgrade ... --safe` | Take a pre-upgrade snapshot per LXC first (`pct snapshot``vzdump` fallback for bind-mounted LXCs). Refuses if any snapshot fails unless `--force`. |
| `homelab apt-upgrade ... --force` | Skip both the dpkg-audit gate and snapshot-failure refusal. |
PVE/kernel deferral on hubris: `homelab apt-upgrade --target hubris` will try every upgrade, including kernel + `pve-*`. To skip those, `apt-mark hold` the relevant packages on hubris first; `homelab apt-audit` shows held packages so you can confirm.
## Related
- [Hubris host](../hosts/hubris.md)
- [Containers index](../containers/index.md)
- [DNS](../infrastructure/dns.md)
- [Monitoring](../infrastructure/monitoring.md)
- [Auto-deploy](../infrastructure/auto-deploy.md)
- [Runbook: dpkg-interrupted recovery](runbook-dpkg-interrupted.md) — what to do when apt got killed mid-transaction