Files

dtoro 7a062bdb6b docs: dpkg-interrupted runbook + apt-fleet ops + non-apt-binary pattern + Claude Code settings + chat-sudo gotcha

Bundles the documentation slice of the apt-sweep backlog:

* operations/runbook-dpkg-interrupted.md (NEW) — Path A (ssh-reachable
  recovery) + Path B (PVE web Shell when the netbird mesh broke
  alongside the dpkg state, as happened during Wave 6 on hubris).
  Closes B2.

* operations/commands.md — new "Fleet apt operations" section
  documenting `homelab apt-audit` and `homelab apt-upgrade`
  (--status / --safe / --force). Adds the dpkg-interrupted runbook to
  Related.

* operations/agent-enrollment.md —
  - new "Claude Code permissions for fleet ops" section with the
    `permissions.allow` snippet (`Bash(ssh -p 22022 *)`,
    `Bash(homelab *)`) for `~/.claude/settings.json`. Closes A3.
  - two new Troubleshooting rows: chat-mode `!` sudo no-tty gotcha
    (G2) and the cosmetic netbird DNS-probe warning.

* infrastructure/auto-deploy.md — new "Custom-built binaries that
  overlap apt-managed paths" section describing the two acceptable
  patterns (epoch-versioned .deb à la caddy 1:2.11.3-hubris1; or
  apt-mark hold) and the discovery path via `homelab apt-audit`'s
  NONAPT column. Closes D3.

Remaining backlog after this commit: A4 (upstream OpenSSH/netbird mux
bug), D1 (apt-mark hold caddy in caddy-conf bootstrap — superseded
in practice by the epoch .deb), E1/E2 (LXC DNS fallback for
tailscale-managed resolv.conf), F1/F2 (vzdump fallback doc; F3 already
shipped).

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

2026-05-21 09:32:06 +02:00

4.3 KiB

Raw Blame History

Operations cheatsheet

Run from the hubris host as root. When working from /root on Linux you're already on hubris — don't ssh hubris / ping hubris.

Proxmox CLI

Command	Use
`pct list` / `qm list`	List LXC containers / VMs
`pct config <id>` / `qm config <id>`	Container / VM config
`pct exec <id> -- <cmd>`	Run command inside an LXC without entering it (no initgroups — see media permissions)
`pct enter <id>`	Shell into a container
`pct start <id>` / `pct stop <id>`	Boot / halt a container
`pvesm status`	Storage pools status
`pvesh get /nodes --output-format json`	Node summary as JSON
`pvesh get /nodes/hubris/lxc/<id>/status/current`	Live container status
`pvesh get /cluster/resources --type vm --output-format json`	Bulk per-LXC CPU/mem/disk (used by claudio-monitor)
`pveversion`	PVE version
`journalctl -u pve-cluster -n 100`	PVE service logs

Storage

Shared mount: /mnt/library (ext4 on lvmthin library).
Bind into a container: pct set <id> -mp<N> /mnt/library/<sub>,mp=/data
For the standard whole-tree mount: pct set <id> -mp0 /mnt/library,mp=/mnt/library. See media permissions for the GID-10000 onboarding recipe.

Reverse proxy

Caddyfile: /etc/caddy/Caddyfile on LXC 121. Tracked in dtoro/caddy-conf. Edits should be committed + pushed — see auto-deploy.
Hot reload: pct exec 121 -- systemctl reload caddy.
Validate: pct exec 121 -- caddy validate --config /etc/caddy/Caddyfile.

DNS

Split-horizon entries: /etc/dnsmasq.d/hubris-split.conf on LXC 124. Hard restart on edit: pct exec 124 -- systemctl restart dnsmasq. SIGHUP isn't reliable.
Verify: dig @192.168.8.180 +short <host>.hubris.network.
See DNS.

Web access

https://proxmox.hubris.network or https://192.168.8.77:8006 — Proxmox UI

Telemetry quick checks

ras-mc-ctl --summary — summary of any RAS events (memory / PCIe AER / thermal) since boot
ras-mc-ctl --errors — full event log
cat /sys/devices/system/cpu/cpu0/cpufreq/energy_performance_preference — should be balance_power
cat /sys/devices/system/cpu/cpu0/cpufreq/scaling_governor — should be powersave
ls /sys/fs/pstore/ /var/lib/systemd/pstore/ — panic traces from a previous crash (empty for pure hardware hangs — see investigation)

Fleet apt operations

Two homelab subcommands wrap the common patterns; both fan out to hubris + every LXC.

Command	What it does
`homelab apt-audit [--target HOST]`	Per-host table: dpkg-interrupted state, holds, upgradable count, non-apt binaries in system paths, DNS health. Exits nonzero if any host has dpkg-interrupted state.
`homelab apt-upgrade --target HOST`	Launch `apt update && apt upgrade` inside a transient `systemd-run --collect` unit on the target. Survives ssh teardown. Apt configured with `Acquire::Retries=3` + `ForceIPv4=true`.
`homelab apt-upgrade --all`	Same, fanned out across the standard targets.
`homelab apt-upgrade ... --status`	Show running unit + tail `/var/log/homelab-apt-upgrade.log` on each target.
`homelab apt-upgrade ... --safe`	Take a pre-upgrade snapshot per LXC first (`pct snapshot` → `vzdump` fallback for bind-mounted LXCs). Refuses if any snapshot fails unless `--force`.
`homelab apt-upgrade ... --force`	Skip both the dpkg-audit gate and snapshot-failure refusal.

PVE/kernel deferral on hubris: homelab apt-upgrade --target hubris will try every upgrade, including kernel + pve-*. To skip those, apt-mark hold the relevant packages on hubris first; homelab apt-audit shows held packages so you can confirm.

Hubris host
Containers index
DNS
Monitoring
Auto-deploy
Runbook: dpkg-interrupted recovery — what to do when apt got killed mid-transaction

4.3 KiB Raw Blame History