From 7a062bdb6bec227a97c564d1fea913f9897a459b Mon Sep 17 00:00:00 2001 From: dtoro Date: Thu, 21 May 2026 09:32:06 +0200 Subject: [PATCH] docs: dpkg-interrupted runbook + apt-fleet ops + non-apt-binary pattern + Claude Code settings + chat-sudo gotcha MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Bundles the documentation slice of the apt-sweep backlog: * operations/runbook-dpkg-interrupted.md (NEW) — Path A (ssh-reachable recovery) + Path B (PVE web Shell when the netbird mesh broke alongside the dpkg state, as happened during Wave 6 on hubris). Closes B2. * operations/commands.md — new "Fleet apt operations" section documenting `homelab apt-audit` and `homelab apt-upgrade` (--status / --safe / --force). Adds the dpkg-interrupted runbook to Related. * operations/agent-enrollment.md — - new "Claude Code permissions for fleet ops" section with the `permissions.allow` snippet (`Bash(ssh -p 22022 *)`, `Bash(homelab *)`) for `~/.claude/settings.json`. Closes A3. - two new Troubleshooting rows: chat-mode `!` sudo no-tty gotcha (G2) and the cosmetic netbird DNS-probe warning. * infrastructure/auto-deploy.md — new "Custom-built binaries that overlap apt-managed paths" section describing the two acceptable patterns (epoch-versioned .deb à la caddy 1:2.11.3-hubris1; or apt-mark hold) and the discovery path via `homelab apt-audit`'s NONAPT column. Closes D3. Remaining backlog after this commit: A4 (upstream OpenSSH/netbird mux bug), D1 (apt-mark hold caddy in caddy-conf bootstrap — superseded in practice by the epoch .deb), E1/E2 (LXC DNS fallback for tailscale-managed resolv.conf), F1/F2 (vzdump fallback doc; F3 already shipped). Co-Authored-By: Claude Opus 4.7 (1M context) --- infrastructure/auto-deploy.md | 13 +++ operations/agent-enrollment.md | 33 ++++++++ operations/commands.md | 16 ++++ operations/runbook-dpkg-interrupted.md | 108 +++++++++++++++++++++++++ 4 files changed, 170 insertions(+) create mode 100644 operations/runbook-dpkg-interrupted.md diff --git a/infrastructure/auto-deploy.md b/infrastructure/auto-deploy.md index 942891f..31bc946 100644 --- a/infrastructure/auto-deploy.md +++ b/infrastructure/auto-deploy.md @@ -103,10 +103,23 @@ Always commit + push. Local-only edits drift. Common ones: every other client has** — kept fresh by `homelab-context-sync.timer`, not by these webhooks. +## Custom-built binaries that overlap apt-managed paths + +If a pipeline (or any out-of-band build) drops a binary into a path that an apt package also owns — most commonly `/usr/bin/` — then the next `apt upgrade` of the corresponding package will silently clobber the custom build. That's exactly how [LXC 121 caddy](../containers/121-caddy.md) went down for ~10 min on 2026-05-21: an xcaddy build with `caddy-dns/ionos` lived at `/usr/bin/caddy` and Debian's caddy 2.11.2→2.11.3 apt upgrade replaced it with a vanilla 2.11.3 that couldn't parse the Caddyfile. + +Two patterns are acceptable, pick one when authoring a pipeline that ships a non-apt binary: + +1. **Ship the build as a `.deb` with an epoch-bumped version.** Use `dpkg-deb --build` (or `nfpm`) to package the binary as `Package: `, `Version: 1:-hubris`. The epoch (`1:`) means it beats any non-epoch upstream version regardless of point bumps, so `apt upgrade` is a no-op for that package. Used by caddy: `caddy 1:2.11.3-hubris1` (see commit `2026-05-21` in [121-caddy.md](../containers/121-caddy.md)). + +2. **Hold the apt package.** `apt-mark hold ` on the LXC during pipeline install; apt will refuse to upgrade it. Simpler than `.deb` packaging but: (a) the hold flag isn't preserved by `dpkg -i` of a new version, (b) it's invisible unless you check `apt-mark showhold`, (c) you have to remember to `apt-mark unhold` when you intentionally want a new version. The new `homelab apt-audit` subcommand surfaces holds across the fleet so they don't get forgotten. + +If you're not sure what's already lurking, run `homelab apt-audit --fleet` and look at the `NONAPT` column — that's a count of binaries in `/usr/bin/{caddy,docker,jellyfin}` + `/usr/local/bin/*` that no apt package owns. + ## Related - [Gitea (104)](../containers/104-gitea.md) — webhook source for all of these - [Caddy (121)](../containers/121-caddy.md), [apps (105)](../containers/105-apps.md), [mule-images (120)](../containers/120-mule-images.md), [claudio-bot (123)](../containers/123-claudio-bot.md), [hubris host](../hosts/hubris.md) — webhook targets - [Backups (disabled)](backups.md) +- [Operations cheatsheet](../operations/commands.md) — `homelab apt-audit` / `homelab apt-upgrade` reference ## Changelog diff --git a/operations/agent-enrollment.md b/operations/agent-enrollment.md index 4fe215f..470924e 100644 --- a/operations/agent-enrollment.md +++ b/operations/agent-enrollment.md @@ -130,6 +130,37 @@ For Claude Code: start a new session — the `homelab` MCP server appears in `~/.claude/.mcp.json` and registers 14 tools (8 context, 5 management, 1 secrets-metadata). +## Claude Code permissions for fleet ops + +By default Claude Code's auto-mode classifier asks for confirmation on every +ssh into the mesh. The bootstrap already installs the ssh ControlMaster block +so subsequent in-session sshes multiplex, but the *first* ssh of each session +still gets classifier-evaluated. Pre-authorize the common fleet ssh patterns +by adding to `~/.claude/settings.json`: + +```json +{ + "permissions": { + "defaultMode": "auto", + "allow": [ + "Bash(ssh -p 22022 *)", + "Bash(homelab *)" + ] + } +} +``` + +The first rule covers any ssh to a mesh peer on the homelab netbird port; the +second covers all `homelab` CLI invocations. Both are scoped tight enough that +the classifier doesn't gate them but loose enough to handle the variety of +arguments. + +If you also want the netbird `--ssh-jwt-cache-ttl` flag rationale to be +visible to the classifier (it's not actually durable in 0.71.2, but the +ControlMaster block is — see [runbook-dpkg-interrupted](runbook-dpkg-interrupted.md) +for context), drop a free-text rule into `autoMode.allow` describing the +authorization. Optional. + ## Adding a new client to inventory If the hostname you want isn't yet in inventory, enrollment is a two-step @@ -206,6 +237,8 @@ The CLI prints a follow-up checklist that the operator must do manually: | `homelab: no age key at /etc/age/key.txt` even after bootstrap | `/etc/age` is 0700 root, so non-root users couldn't even stat the key file; existence check returned False under regular users | Fixed in commit `df6aca8`: the CLI re-execs `sops -d` via sudo when invoked as a non-root user. On older deployments, re-link the CLI with `sudo ln -sfn /opt/homelab-context/bin/homelab /usr/local/bin/homelab` after the 5-min sync. | | `homelab` CLI doesn't pick up repo updates | Pre-`02db…` bootstrap copied the binary instead of symlinking | One-time migration: `sudo ln -sfn /opt/homelab-context/bin/homelab /usr/local/bin/homelab`. New bootstraps use the symlink, which auto-tracks the synced repo. | | `homelab-context-sync.service` journal shows `fatal: could not read Username for 'https://git.hubris.network'` | Pre-fix bootstrap set the gitea credential helper via `git config --global`, which writes to `/root/.gitconfig` — invisible to the systemd timer's git process (no HOME set). | One-time migration: `sudo git config --system credential.helper "store --file=/etc/homelab-context/git-credentials"`. New bootstraps store the helper in `/etc/gitconfig` instead. | +| Chat-mode `!` shell can't `sudo` (`a terminal is required to read the password`) | Claude Code's `!` invocation doesn't allocate a tty, and standard `sudo` won't read its password from stdin or a non-tty pipe. | Run the sudo'd command in a real terminal outside chat. For commands the agent issues repeatedly, configure passwordless sudo for the narrow set (e.g. `/etc/sudoers.d/homelab-self` with ` ALL=(ALL) NOPASSWD: /usr/bin/dnf upgrade -y, /usr/bin/apt-get *`). | +| `netbird status -d` reports `192.168.8.180:53 ... is Unavailable` but DNS actually works | netbird's UDP-53 probe times out over the relay latency (~90ms), but actual queries still flow through systemd-resolved. Cosmetic. | Ignore unless `dig @192.168.8.180 git.hubris.network` also fails — then check dnsmasq on [LXC 124](../containers/124-authentik.md). | ## Changelog diff --git a/operations/commands.md b/operations/commands.md index 33c298a..2bfb60e 100644 --- a/operations/commands.md +++ b/operations/commands.md @@ -48,9 +48,25 @@ Run from the [hubris host](../hosts/hubris.md) as root. When working from `/root - `cat /sys/devices/system/cpu/cpu0/cpufreq/scaling_governor` — should be `powersave` - `ls /sys/fs/pstore/ /var/lib/systemd/pstore/` — panic traces from a previous crash (empty for pure hardware hangs — see [investigation](../investigations/2026-04-21-hubris-crash-loop.md)) +## Fleet apt operations + +Two `homelab` subcommands wrap the common patterns; both fan out to hubris + every LXC. + +| Command | What it does | +| --- | --- | +| `homelab apt-audit [--target HOST]` | Per-host table: dpkg-interrupted state, holds, upgradable count, non-apt binaries in system paths, DNS health. Exits nonzero if any host has dpkg-interrupted state. | +| `homelab apt-upgrade --target HOST` | Launch `apt update && apt upgrade` inside a transient `systemd-run --collect` unit on the target. Survives ssh teardown. Apt configured with `Acquire::Retries=3` + `ForceIPv4=true`. | +| `homelab apt-upgrade --all` | Same, fanned out across the standard targets. | +| `homelab apt-upgrade ... --status` | Show running unit + tail `/var/log/homelab-apt-upgrade.log` on each target. | +| `homelab apt-upgrade ... --safe` | Take a pre-upgrade snapshot per LXC first (`pct snapshot` → `vzdump` fallback for bind-mounted LXCs). Refuses if any snapshot fails unless `--force`. | +| `homelab apt-upgrade ... --force` | Skip both the dpkg-audit gate and snapshot-failure refusal. | + +PVE/kernel deferral on hubris: `homelab apt-upgrade --target hubris` will try every upgrade, including kernel + `pve-*`. To skip those, `apt-mark hold` the relevant packages on hubris first; `homelab apt-audit` shows held packages so you can confirm. + ## Related - [Hubris host](../hosts/hubris.md) - [Containers index](../containers/index.md) - [DNS](../infrastructure/dns.md) - [Monitoring](../infrastructure/monitoring.md) - [Auto-deploy](../infrastructure/auto-deploy.md) +- [Runbook: dpkg-interrupted recovery](runbook-dpkg-interrupted.md) — what to do when apt got killed mid-transaction diff --git a/operations/runbook-dpkg-interrupted.md b/operations/runbook-dpkg-interrupted.md new file mode 100644 index 0000000..5b1ff8a --- /dev/null +++ b/operations/runbook-dpkg-interrupted.md @@ -0,0 +1,108 @@ +# Runbook — recover from dpkg-interrupted state + +You're here because an apt run got killed mid-transaction and the target now +has packages that are **unpacked but not configured**. Symptoms: + +- `apt` refuses to do anything new: `Error: dpkg was interrupted, you must + manually run 'dpkg --configure -a' to correct the problem.` +- `dpkg --audit` lists packages with header + `The following packages have been unpacked but not yet configured.` +- `homelab apt-audit` shows `DPKG: DIRTY(N)` for the host. + +The system is still running the **old** binaries (still in memory), but the +**new** binaries are unpacked and waiting for their postinst to run. Two +worst-case manifestations from the 2026-05-21 sweep: + +- LXC 121 caddy: leftover state from a prior aborted apt run; caddy itself was + still serving but the new caddy binary on disk hadn't been wired up. +- hubris: ssh master died mid-Wave-6 → 135 packages unpacked-not-configured, + including `systemd`, `openssh-server`, `sudo`, `netbird`. The half- + configured netbird daemon dropped the mesh peer, and we got locked out + until we recovered from the PVE web UI Shell. + +**Do not reboot until dpkg is clean.** A reboot tries to start the new +binaries' services, which may fail because postinst never ran (missing users, +config dirs, capabilities, etc.). The system might not come back up cleanly. + +## Path A — target is still reachable over ssh (preferred) + +``` +homelab ssh -- bash -c 'DEBIAN_FRONTEND=noninteractive dpkg --configure -a && apt -y -o Dpkg::Options::=--force-confold upgrade' +``` + +Or for an LXC by name: + +``` +homelab pct exec -- bash -c 'DEBIAN_FRONTEND=noninteractive dpkg --configure -a && apt -y -o Dpkg::Options::=--force-confold upgrade' +``` + +When that returns, confirm: + +``` +homelab apt-audit --target +``` + +Expect `DPKG: ok` and the remaining `UPGR` count to match what's intentionally +deferred (kernel/PVE on hubris, 0 elsewhere). + +## Path B — target locked out (mesh broken / ssh dead) + +Most common for hubris when netbird itself went half-configured: the daemon +crashed on the new binary, the mesh peer dropped, port 22022 stopped listening, +and you can't ssh in. + +1. Open `https://proxmox.hubris.network` in a browser. +2. Datacenter → node `hubris` → `>_ Shell` (or `_ Console`). That's a root + shell on hubris served by the PVE web UI, independent of the netbird mesh. +3. Run the recovery one-liner: + +``` +DEBIAN_FRONTEND=noninteractive dpkg --configure -a \ + && DEBIAN_FRONTEND=noninteractive apt -y -o Dpkg::Options::=--force-confold upgrade \ + && systemctl restart netbird \ + && dpkg --audit \ + && echo RECOVERY_OK +``` + + Wait for `RECOVERY_OK`. The `systemctl restart netbird` is the bit that + heals the mesh — once netbird's daemon comes back up clean, your client's + peer state moves from `Connecting` to `Connected` within ~30 seconds and + the rest of your tooling works again. + +4. For an **LXC** that's locked out (less common — LXCs reach the world via + netbird routed through hubris, so unless hubris itself is broken, you can + still `pct enter` from the hubris shell): + + From the PVE web UI shell on hubris: + + ``` + pct enter + DEBIAN_FRONTEND=noninteractive dpkg --configure -a && apt -y upgrade + exit + ``` + +## Prevention + +The `homelab apt-upgrade` wrapper launches apt inside a `systemd-run --collect` +unit on the target, so it survives ssh teardown — the failure mode that put +hubris into this state in the first place is no longer reachable through the +standard tool. If you absolutely need to run apt manually over ssh, wrap it: + +``` +ssh systemd-run --unit=apt-recovery --collect bash -c 'apt -y upgrade' +``` + +Then `systemctl status apt-recovery` from a fresh ssh to check progress. + +## Related + +- [Operations cheatsheet](commands.md) +- [Auto-deploy pipelines](../infrastructure/auto-deploy.md) +- [Hubris host page](../hosts/hubris.md) + +## Changelog + +### 2026-05-21 — initial page +Documents the dpkg-interrupted recovery path that came out of the +fleet apt sweep (Wave 6 killed mid-transaction; hubris recovered via PVE +web Shell).