docs: dpkg-interrupted runbook + apt-fleet ops + non-apt-binary pattern + Claude Code settings + chat-sudo gotcha
Bundles the documentation slice of the apt-sweep backlog:
* operations/runbook-dpkg-interrupted.md (NEW) — Path A (ssh-reachable
recovery) + Path B (PVE web Shell when the netbird mesh broke
alongside the dpkg state, as happened during Wave 6 on hubris).
Closes B2.
* operations/commands.md — new "Fleet apt operations" section
documenting `homelab apt-audit` and `homelab apt-upgrade`
(--status / --safe / --force). Adds the dpkg-interrupted runbook to
Related.
* operations/agent-enrollment.md —
- new "Claude Code permissions for fleet ops" section with the
`permissions.allow` snippet (`Bash(ssh -p 22022 *)`,
`Bash(homelab *)`) for `~/.claude/settings.json`. Closes A3.
- two new Troubleshooting rows: chat-mode `!` sudo no-tty gotcha
(G2) and the cosmetic netbird DNS-probe warning.
* infrastructure/auto-deploy.md — new "Custom-built binaries that
overlap apt-managed paths" section describing the two acceptable
patterns (epoch-versioned .deb à la caddy 1:2.11.3-hubris1; or
apt-mark hold) and the discovery path via `homelab apt-audit`'s
NONAPT column. Closes D3.
Remaining backlog after this commit: A4 (upstream OpenSSH/netbird mux
bug), D1 (apt-mark hold caddy in caddy-conf bootstrap — superseded
in practice by the epoch .deb), E1/E2 (LXC DNS fallback for
tailscale-managed resolv.conf), F1/F2 (vzdump fallback doc; F3 already
shipped).
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
This commit is contained in:
@@ -103,10 +103,23 @@ Always commit + push. Local-only edits drift. Common ones:
|
||||
every other client has** — kept fresh by `homelab-context-sync.timer`,
|
||||
not by these webhooks.
|
||||
|
||||
## Custom-built binaries that overlap apt-managed paths
|
||||
|
||||
If a pipeline (or any out-of-band build) drops a binary into a path that an apt package also owns — most commonly `/usr/bin/<name>` — then the next `apt upgrade` of the corresponding package will silently clobber the custom build. That's exactly how [LXC 121 caddy](../containers/121-caddy.md) went down for ~10 min on 2026-05-21: an xcaddy build with `caddy-dns/ionos` lived at `/usr/bin/caddy` and Debian's caddy 2.11.2→2.11.3 apt upgrade replaced it with a vanilla 2.11.3 that couldn't parse the Caddyfile.
|
||||
|
||||
Two patterns are acceptable, pick one when authoring a pipeline that ships a non-apt binary:
|
||||
|
||||
1. **Ship the build as a `.deb` with an epoch-bumped version.** Use `dpkg-deb --build` (or `nfpm`) to package the binary as `Package: <name>`, `Version: 1:<upstream>-hubris<n>`. The epoch (`1:`) means it beats any non-epoch upstream version regardless of point bumps, so `apt upgrade` is a no-op for that package. Used by caddy: `caddy 1:2.11.3-hubris1` (see commit `2026-05-21` in [121-caddy.md](../containers/121-caddy.md)).
|
||||
|
||||
2. **Hold the apt package.** `apt-mark hold <pkg>` on the LXC during pipeline install; apt will refuse to upgrade it. Simpler than `.deb` packaging but: (a) the hold flag isn't preserved by `dpkg -i` of a new version, (b) it's invisible unless you check `apt-mark showhold`, (c) you have to remember to `apt-mark unhold` when you intentionally want a new version. The new `homelab apt-audit` subcommand surfaces holds across the fleet so they don't get forgotten.
|
||||
|
||||
If you're not sure what's already lurking, run `homelab apt-audit --fleet` and look at the `NONAPT` column — that's a count of binaries in `/usr/bin/{caddy,docker,jellyfin}` + `/usr/local/bin/*` that no apt package owns.
|
||||
|
||||
## Related
|
||||
- [Gitea (104)](../containers/104-gitea.md) — webhook source for all of these
|
||||
- [Caddy (121)](../containers/121-caddy.md), [apps (105)](../containers/105-apps.md), [mule-images (120)](../containers/120-mule-images.md), [claudio-bot (123)](../containers/123-claudio-bot.md), [hubris host](../hosts/hubris.md) — webhook targets
|
||||
- [Backups (disabled)](backups.md)
|
||||
- [Operations cheatsheet](../operations/commands.md) — `homelab apt-audit` / `homelab apt-upgrade` reference
|
||||
|
||||
## Changelog
|
||||
|
||||
|
||||
Reference in New Issue
Block a user