dtoro
7a062bdb6b
Bundles the documentation slice of the apt-sweep backlog:
* operations/runbook-dpkg-interrupted.md (NEW) — Path A (ssh-reachable
recovery) + Path B (PVE web Shell when the netbird mesh broke
alongside the dpkg state, as happened during Wave 6 on hubris).
Closes B2.
* operations/commands.md — new "Fleet apt operations" section
documenting `homelab apt-audit` and `homelab apt-upgrade`
(--status / --safe / --force). Adds the dpkg-interrupted runbook to
Related.
* operations/agent-enrollment.md —
- new "Claude Code permissions for fleet ops" section with the
`permissions.allow` snippet (`Bash(ssh -p 22022 *)`,
`Bash(homelab *)`) for `~/.claude/settings.json`. Closes A3.
- two new Troubleshooting rows: chat-mode `!` sudo no-tty gotcha
(G2) and the cosmetic netbird DNS-probe warning.
* infrastructure/auto-deploy.md — new "Custom-built binaries that
overlap apt-managed paths" section describing the two acceptable
patterns (epoch-versioned .deb à la caddy 1:2.11.3-hubris1; or
apt-mark hold) and the discovery path via `homelab apt-audit`'s
NONAPT column. Closes D3.
Remaining backlog after this commit: A4 (upstream OpenSSH/netbird mux
bug), D1 (apt-mark hold caddy in caddy-conf bootstrap — superseded
in practice by the epoch .deb), E1/E2 (LXC DNS fallback for
tailscale-managed resolv.conf), F1/F2 (vzdump fallback doc; F3 already
shipped).
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
73 lines
4.3 KiB
Markdown
73 lines
4.3 KiB
Markdown
# Operations cheatsheet
|
|
|
|
Run from the [hubris host](../hosts/hubris.md) as root. When working from `/root` on Linux you're already on hubris — don't `ssh hubris` / `ping hubris`.
|
|
|
|
## Proxmox CLI
|
|
|
|
| Command | Use |
|
|
| --- | --- |
|
|
| `pct list` / `qm list` | List LXC containers / VMs |
|
|
| `pct config <id>` / `qm config <id>` | Container / VM config |
|
|
| `pct exec <id> -- <cmd>` | Run command inside an LXC without entering it (no initgroups — see [media permissions](../infrastructure/media-permissions.md)) |
|
|
| `pct enter <id>` | Shell into a container |
|
|
| `pct start <id>` / `pct stop <id>` | Boot / halt a container |
|
|
| `pvesm status` | Storage pools status |
|
|
| `pvesh get /nodes --output-format json` | Node summary as JSON |
|
|
| `pvesh get /nodes/hubris/lxc/<id>/status/current` | Live container status |
|
|
| `pvesh get /cluster/resources --type vm --output-format json` | Bulk per-LXC CPU/mem/disk (used by [claudio-monitor](../infrastructure/monitoring.md)) |
|
|
| `pveversion` | PVE version |
|
|
| `journalctl -u pve-cluster -n 100` | PVE service logs |
|
|
|
|
## Storage
|
|
|
|
- Shared mount: `/mnt/library` (ext4 on lvmthin `library`).
|
|
- Bind into a container: `pct set <id> -mp<N> /mnt/library/<sub>,mp=/data`
|
|
- For the standard whole-tree mount: `pct set <id> -mp0 /mnt/library,mp=/mnt/library`. See [media permissions](../infrastructure/media-permissions.md) for the GID-10000 onboarding recipe.
|
|
|
|
## Reverse proxy
|
|
|
|
- Caddyfile: `/etc/caddy/Caddyfile` on [LXC 121](../containers/121-caddy.md). Tracked in `dtoro/caddy-conf`. Edits should be committed + pushed — see [auto-deploy](../infrastructure/auto-deploy.md).
|
|
- Hot reload: `pct exec 121 -- systemctl reload caddy`.
|
|
- Validate: `pct exec 121 -- caddy validate --config /etc/caddy/Caddyfile`.
|
|
|
|
## DNS
|
|
|
|
- Split-horizon entries: `/etc/dnsmasq.d/hubris-split.conf` on [LXC 124](../containers/124-authentik.md). Hard restart on edit: `pct exec 124 -- systemctl restart dnsmasq`. SIGHUP isn't reliable.
|
|
- Verify: `dig @192.168.8.180 +short <host>.hubris.network`.
|
|
- See [DNS](../infrastructure/dns.md).
|
|
|
|
## Web access
|
|
|
|
- `https://proxmox.hubris.network` or `https://192.168.8.77:8006` — Proxmox UI
|
|
|
|
## Telemetry quick checks
|
|
|
|
- `ras-mc-ctl --summary` — summary of any RAS events (memory / PCIe AER / thermal) since boot
|
|
- `ras-mc-ctl --errors` — full event log
|
|
- `cat /sys/devices/system/cpu/cpu0/cpufreq/energy_performance_preference` — should be `balance_power`
|
|
- `cat /sys/devices/system/cpu/cpu0/cpufreq/scaling_governor` — should be `powersave`
|
|
- `ls /sys/fs/pstore/ /var/lib/systemd/pstore/` — panic traces from a previous crash (empty for pure hardware hangs — see [investigation](../investigations/2026-04-21-hubris-crash-loop.md))
|
|
|
|
## Fleet apt operations
|
|
|
|
Two `homelab` subcommands wrap the common patterns; both fan out to hubris + every LXC.
|
|
|
|
| Command | What it does |
|
|
| --- | --- |
|
|
| `homelab apt-audit [--target HOST]` | Per-host table: dpkg-interrupted state, holds, upgradable count, non-apt binaries in system paths, DNS health. Exits nonzero if any host has dpkg-interrupted state. |
|
|
| `homelab apt-upgrade --target HOST` | Launch `apt update && apt upgrade` inside a transient `systemd-run --collect` unit on the target. Survives ssh teardown. Apt configured with `Acquire::Retries=3` + `ForceIPv4=true`. |
|
|
| `homelab apt-upgrade --all` | Same, fanned out across the standard targets. |
|
|
| `homelab apt-upgrade ... --status` | Show running unit + tail `/var/log/homelab-apt-upgrade.log` on each target. |
|
|
| `homelab apt-upgrade ... --safe` | Take a pre-upgrade snapshot per LXC first (`pct snapshot` → `vzdump` fallback for bind-mounted LXCs). Refuses if any snapshot fails unless `--force`. |
|
|
| `homelab apt-upgrade ... --force` | Skip both the dpkg-audit gate and snapshot-failure refusal. |
|
|
|
|
PVE/kernel deferral on hubris: `homelab apt-upgrade --target hubris` will try every upgrade, including kernel + `pve-*`. To skip those, `apt-mark hold` the relevant packages on hubris first; `homelab apt-audit` shows held packages so you can confirm.
|
|
|
|
## Related
|
|
- [Hubris host](../hosts/hubris.md)
|
|
- [Containers index](../containers/index.md)
|
|
- [DNS](../infrastructure/dns.md)
|
|
- [Monitoring](../infrastructure/monitoring.md)
|
|
- [Auto-deploy](../infrastructure/auto-deploy.md)
|
|
- [Runbook: dpkg-interrupted recovery](runbook-dpkg-interrupted.md) — what to do when apt got killed mid-transaction
|